Failed to refresh token or Connection timed out to cloud endpointCloud Gateway registration failed*.vmwarecloud.com, *.vmware.com)The vSphere+ Cloud Gateway is a virtual appliance deployed on-premises that establishes a persistent outbound connection to VMware Cloud. It relays inventory, health data, and lifecycle events from your vCenter to the cloud console. When connectivity breaks — due to proxy changes, firewall rule updates, DNS failures, or expired authentication tokens — the gateway can no longer sync. Since vSphere+ licensing is subscription-based and validated through the cloud, loss of connectivity can eventually cause license enforcement warnings on vCenter.
Check gateway appliance health and connectivity
# SSH into the Cloud Gateway appliance
# Check connectivity to VMware Cloud endpoints
curl -v https://console.cloud.vmware.com
curl -v https://gw.apm.vmware.com
# Check proxy configuration
cat /etc/environment | grep -i proxy
cat /opt/vmware/cloudgateway/config/gateway.properties | grep proxy
# Check DNS resolution
nslookup console.cloud.vmware.com
nslookup vcsa.vmware.com
Verify NTP synchronization and certificate validity
# Check NTP status on gateway
timedatectl status
ntpq -p
# Verify gateway certificates have not expired
openssl s_client -connect console.cloud.vmware.com:443 -servername console.cloud.vmware.com < /dev/null 2>/dev/null | openssl x509 -noout -dates
# Check local certificate store
ls -la /opt/vmware/cloudgateway/ssl/
Re-register the gateway if token is expired
# Check current registration status
/opt/vmware/cloudgateway/scripts/check-registration.sh
# If token expired, generate a new activation token from VMware Cloud Console:
# Cloud Console → vSphere+ → Infrastructure → Add Gateway → Copy Token
# Re-register with new token
/opt/vmware/cloudgateway/scripts/register-gateway.sh --token "<new-activation-token>"
Fix proxy or firewall if connectivity is blocked
# Set proxy on the gateway appliance
/opt/vmware/cloudgateway/scripts/set-proxy.sh --host proxy.corp.local --port 3128 --user proxyuser --password '***'
# Required outbound endpoints (allow in firewall):
# *.vmwarecloud.com:443
# *.vmware.com:443
# gw.apm.vmware.com:443
# sddcgw-prod-*.amazonaws.com:443
# Restart gateway services after proxy change
systemctl restart cloudgateway
systemctl status cloudgateway
Validate sync is working and check subscription license
# Check gateway sync log
tail -100 /var/log/vmware/cloudgateway/gateway-sync.log
# Verify subscription license in vCenter
# vCenter UI → Administration → Licensing → Verify vSphere+ subscription is active
# Force inventory resync
/opt/vmware/cloudgateway/scripts/force-sync.sh --full
Cannot complete the operation due to a crypto errorInvalid VM configuration on power-onCrypto key not found on destination hostCryptoManagerKmip: Failed to get key from KMS serverVM Encryption in vSphere relies on a Key Management Server (KMS) — either an external KMIP-compliant server (like HyTrust, Thales, HashiCorp Vault) or the built-in Native Key Provider (NKP). vCenter obtains Key Encryption Keys (KEKs) from the KMS and pushes them to ESXi hosts, which use them to wrap the Data Encryption Keys (DEKs) that encrypt VM files. If vCenter loses contact with the KMS, or if certificates expire, no new crypto operations can be performed. Existing running VMs continue operating (keys are cached in host memory), but any power-on, migration, or snapshot of an encrypted VM will fail.
Check KMS connectivity and certificate status
# Check KMS server reachability from vCenter (VCSA)
curl -v --connect-timeout 5 https://kms-server.corp.local:5696
# Check vCenter ↔ KMS trust certificates
# vCenter UI → Configure → Security → Key Providers → Select provider → Check Status
# View KMS certificates via MOB:
# https://vcenter.corp.local/mob/?moid=CryptoManager&doPath=kmipServers
# Check STS certificate validity (often the root cause)
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list --login administrator@vsphere.local
Re-establish KMS trust if certificates expired
# If using external KMS — re-upload vCenter client certificate to KMS:
# vCenter → Key Providers → Establish Trust → Download CSR
# Upload CSR to KMS management interface → Get signed certificate → Upload back to vCenter
# If STS certificate is the root issue, renew it first:
# (See KB #44 for STS certificate renewal steps)
# Verify KMS connection after trust re-establishment
# vCenter → Key Providers → "Check Status" should show green
For Native Key Provider — restore from backup
# Native Key Provider requires explicit backup before it can be used
# If vCenter was rebuilt and NKP backup exists:
# vCenter → Key Providers → Native Key Provider → Restore from backup file (.p12)
# Check if NKP is active
Get-KeyProvider | Format-Table Name, Type, Status
# If no backup exists, encrypted VMs are UNRECOVERABLE
# Prevention: Always backup NKP immediately after creation
# vCenter → Key Providers → NKP → Back Up → Save .p12 file securely
Push keys to hosts and validate VM crypto operations
# After restoring KMS connectivity, push keys to all hosts:
# vCenter → Host → Configure → Security Profile → Verify "Crypto State: Safe"
# Check host crypto readiness
esxcli system settings encryption get
# Expected: Mode = TPM (or None), Require Secure Boot = true/false
# Try powering on an encrypted VM
# If successful, repeat for all affected VMs
# Check vpxd logs for crypto resolution
grep -i "CryptoManager\|crypto\|kmip" /var/log/vmware/vpxd/vpxd.log | tail -30
The device is not available on hostesxcli hardware pci list but not in passthrough-eligible devicesVM has a device attached that prevents migrationEnable IOMMU/VT-d in server BIOS and verify in ESXi
# Verify IOMMU is enabled in ESXi
esxcli hardware pci list | grep -A5 "NVIDIA\|AMD.*Radeon"
# Check passthrough capability
esxcli hardware pci pcipassthru list | head -20
# Verify VT-d / IOMMU status in boot options
esxcli system settings kernel list | grep -i iommu
# Expected: iommu = true / interlIOMMU = true
# If not enabled, reboot into BIOS:
# Intel: Enable "VT-d" under CPU/Processor settings
# AMD: Enable "AMD-Vi" / "IOMMU" under CPU settings
Mark GPU for passthrough and install vGPU Manager VIB
# Mark GPU for passthrough via vCenter:
# Host → Configure → Hardware → PCI Devices → Toggle Passthrough on GPU
# Or via CLI:
esxcli hardware pci pcipassthru set -d 0000:3b:00.0 -e true
# Requires host reboot to take effect
# Install NVIDIA vGPU Manager VIB (for shared vGPU, not full passthrough)
esxcli software vib install -v /tmp/NVIDIA-vGPU-VMware-ESXi-8.0-535.154.02.vib
# Verify installation
esxcli software vib list | grep -i nvidia
nvidia-gpu-manager 535.154.02 NVIDIA VMwareCertified
# Check GPU is recognized by vGPU Manager
nvidia-smi # Shows GPU status, temperature, driver version
Assign vGPU profile to VM
# In vCenter → VM → Edit Settings → Add New Device → PCI Device:
# For DirectPath I/O (full passthrough):
# Select the physical GPU → Reserve entire device
# Note: This disables vMotion, HA restart, and DRS for this VM
# For Shared vGPU (recommended):
# Add New Device → Shared PCI Device → Select vGPU profile
# Profiles: grid_a100-4q (4GB), grid_a100-8q (8GB), etc.
# vMotion with vGPU suspend-resume IS supported on vSphere 8.x
# Ensure VM is set to EFI firmware (required for vGPU)
# VM → Edit Settings → VM Options → Boot Options → Firmware: EFI
# Set memory reservation = full (required for GPU passthrough)
# VM → Edit Settings → Resources → Memory → Reservation: Lock all guest memory
Install guest driver and validate acceleration
# Inside the guest VM:
# Windows:
# Download NVIDIA vGPU guest driver from NVIDIA Licensing Portal
# Install matching version (must match host vGPU Manager version branch)
# nvidia-smi.exe → Should show vGPU device
# Linux:
# Install kernel headers: yum install kernel-devel
# Run: sh NVIDIA-Linux-x86_64-535.154.02-grid.run
# nvidia-smi → Verify vGPU device is present
# Verify GPU acceleration in guest
# Windows: dxdiag → Display → Check "NVIDIA GRID" device
# Linux: glxinfo | grep "OpenGL renderer"
nvidia-smi shows GPU on host, vGPU profile assignable to VMs, guest driver loads with nvidia-smi showing vGPU device, and GPU-accelerated applications render correctly.
PXE-E18: Server response timeout or TFTP timeoutNo matching deploy rule for host MAC/UUIDAuto Deploy uses a PXE/iPXE boot chain: the host NIC sends a DHCP request, receives a TFTP server address (option 66) and boot filename (option 67 — typically undionly.kpxe.vmw-hardwired), then downloads the ESXi boot image from the Auto Deploy server on vCenter. Deploy rules determine which image profile and host profile to apply based on host attributes. If any link in this chain breaks — DHCP, TFTP, Auto Deploy service, deploy rules, image depot, or host profile — the host fails to boot or boots in an unconfigured state.
Verify DHCP options and TFTP reachability
# Check DHCP server scope options:
# Option 66 (Boot Server): IP address of vCenter/Auto Deploy server
# Option 67 (Bootfile Name): undionly.kpxe.vmw-hardwired
# Test TFTP from another machine
tftp <vcenter-ip>
tftp> get undionly.kpxe.vmw-hardwired
# Should download successfully
# If TFTP times out, check firewall on vCenter:
iptables -L -n | grep 69
# Ensure UDP port 69 and TCP port 6501-6502 are open
Check Auto Deploy (rbd) service on vCenter
# SSH to VCSA and check service status
vmon-cli -s rbd --status
# Should show: Running
# If stopped, start it
vmon-cli -s rbd --start
# Check Auto Deploy logs
tail -100 /var/log/vmware/rbd/rbd.log
# Look for: "Rule evaluation failed" or "Image profile not found"
# Verify deploy rules via PowerCLI
Connect-VIServer vcenter.corp.local
Get-DeployRule | Format-Table Name, Pattern, ItemList
Fix deploy rules and image profile associations
# List current deploy rules and their order
Get-DeployRule | Select-Object Name, @{N='Pattern';E={$_.PatternList}}, @{N='Items';E={$_.ItemList}}
# Create or fix a deploy rule
New-DeployRule -Name "DataCenter-Hosts" `
-Item (Get-VMHostProfile -Name "Production-Profile"), `
(Get-DeployRuleImageProfile -Name "ESXi-8.0U2-standard") `
-Pattern "vendor=Dell", "ipv4=10.10.0.0/24"
# Add rule to active ruleset
Add-DeployRule -DeployRule "DataCenter-Hosts"
# Repair the ruleset (ensures consistency)
Repair-DeployRuleSetCompliance
# Verify depot is accessible
Get-DeployRuleImageProfile | Select-Object Name, Vendor, CreationTime
Fix host profile answer file and reboot host
# Check host profile compliance
Test-VMHostProfileCompliance -VMHost esxi-01.corp.local
# Update answer file with required values (IP, license, etc.)
$hp = Get-VMHostProfile -Name "Production-Profile"
Export-VMHostProfile -Profile $hp -FilePath C:\profile-answer.csv
# Edit CSV with per-host answers, then import:
Import-VMHostProfile -FilePath C:\profile-answer.csv -Profile $hp
# Reboot the host to trigger Auto Deploy boot
Restart-VMHost -VMHost esxi-01.corp.local -Confirm:$false
# Host should PXE boot, download image, apply host profile
Component conflict: VIB X requires Y but found ZUnable to download metadata from depot URL/storage/updatemgr/Verify depot accessibility and sync status
# Check online depot connectivity from VCSA
curl -v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
# Check proxy settings for vLCM
# vCenter → Lifecycle Manager → Settings → Download Settings → Proxy
# Check local depot cache
du -sh /storage/updatemgr/
df -h /storage/
# Force depot resync
# vCenter → Lifecycle Manager → Settings → Sync Now
# Check sync logs
tail -100 /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log
Resolve VIB conflicts and component incompatibilities
# Check current host software profile
esxcli software profile get
# Shows: Name, Vendor, VIBs
# List all installed VIBs with acceptance level
esxcli software vib list | grep -i "community\|partner"
# Community-level VIBs often cause conflicts
# Remove conflicting VIB
esxcli software vib remove -n conflicting-vib-name
# Check image compliance detail via PowerCLI
$cluster = Get-Cluster "Production"
$compliance = Test-LCMClusterCompliance -Cluster $cluster
$compliance.HostComplianceResult | Select-Object Host, Status, IncompatibleComponents
Fix Hardware Support Manager connectivity
# Check HSM registration status
# vCenter → Lifecycle Manager → Settings → Hardware Support Managers
# For Dell: Verify OpenManage Integration for VMware vCenter (OMIVV) connectivity
# For HPE: Verify HPE OneView or ILO Amplifier Pack connectivity
# For Lenovo: Verify XClarity Integration
# Re-register HSM if expired
# HSM Admin UI → vCenter Connection → Re-authenticate with vCenter credentials
# Verify firmware compliance after HSM reconnection
$cluster = Get-Cluster "Production"
$image = Get-LCMClusterImage -Cluster $cluster
$image.FirmwareAddOn # Should show vendor firmware bundle
Stage and execute remediation
# Check boot device free space (min ~4GB needed for staging)
esxcli storage filesystem list | grep -i boot
# Pre-check remediation (dry run)
# vCenter → Cluster → Updates → Check Compliance → Review impact
# Stage the image (downloads to hosts first)
# vCenter → Cluster → Updates → Stage
# Remediate with sequential mode for safety
# vCenter → Cluster → Updates → Remediate → Sequential → Max parallel: 1
# Monitor remediation progress
tail -f /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log
# If stuck, check host-side logs during remediation
esxcli software sources profile list -d /var/tmp/update-staging/
esxcli software profile get matches the desired image on each host, and Hardware Support Manager shows "Connected" with firmware aligned.
Could not create a disk on the target datastoreReplication not in sync — cannot recovervSphere Replication works by tracking changed blocks (CBT) on source VM disks and asynchronously replicating deltas to a target VR appliance. The RPO (Recovery Point Objective) defines the maximum data loss window. If the replication engine cannot complete a sync cycle within the RPO window — due to bandwidth constraints, appliance overload, storage issues, or CBT resets — the RPO is violated. Repeated CBT resets (from snapshots, storage vMotion, or VM stunts) cause expensive full re-syncs that consume massive bandwidth and extend the violation window further.
Check VR appliance health and connectivity
# Access VR appliance VAMI
# https://vr-appliance.corp.local:5480 → Dashboard
# Check VR appliance disk usage
ssh root@vr-appliance.corp.local
df -h
# Critical partitions: /opt/vmware/hms (replication data), /var/log
# Check VR appliance memory
free -m
# If <2GB free, appliance is undersized — deploy larger OVA
# Check connectivity to partner site VR appliance
curl -v https://vr-target.corp.local:8043/health
# Should return 200 OK
Diagnose RPO violation root cause
# Check replication status via VR plugin in vCenter
# vCenter → vSphere Replication → Monitor → Outgoing Replications
# Sort by "RPO Status" to find violating VMs
# Check VR server logs for specific errors
tail -200 /opt/vmware/hms/logs/hms.log
# Look for: "RPO violation", "Full sync required", "CBT reset"
# Calculate required bandwidth:
# Required BW = (Daily Change Rate * RPO safety factor) / RPO window
# Example: 50GB daily change, 1hr RPO = 50GB/24hrs * 2 = ~4.2 Gbps
# Check if CBT was reset (forces full sync)
grep -i "cbt\|changedBlockTracking" /opt/vmware/hms/logs/hms.log | tail -20
Fix bandwidth, storage, and CBT issues
# If bandwidth limited, adjust RPO to realistic value:
# vCenter → vSphere Replication → Select VM → Configure Recovery → Increase RPO
# If target datastore is full, extend or migrate replicas
# Check target datastore: vCenter → Target → Datastores → Usage
# If CBT keeps resetting, check for snapshots or consolidation:
vim-cmd vmsvc/snapshot.get <vmid>
# Remove unnecessary snapshots to stabilize CBT
# Enable CBT if disabled
vim-cmd vmsvc/power.off <vmid> # only if maintenance window
# VM → Edit Settings → Advanced → ctkEnabled = TRUE
# VM → Edit Settings → Advanced → scsix:y.ctkEnabled = TRUE
# Configure traffic shaping to prioritize replication
# Network policy on replication port group → Traffic Shaping → Reserve bandwidth
Force resync and validate SRM recovery plan
# Force a full resync to re-baseline (schedule during off-peak)
# vCenter → vSphere Replication → Select VM → "Sync Now" → Force Full Sync
# Monitor sync progress
# vCenter → vSphere Replication → Monitor → Shows % complete and ETA
# After sync completes, validate SRM recovery plan
# SRM → Recovery Plans → Select Plan → Test → Run test recovery
# Verify VMs boot at target site with acceptable data state
# Set up RPO violation alarms
# vCenter → Alarms → vSphere Replication → RPO Violation → Email notification
kubectl get tkc -A shows "Creating" indefinitelykubectl commands timeout connecting to Supervisor APIPhase: Terminating and never completeswcp process in restart loopvSphere with Tanzu deploys a "Supervisor Cluster" — three control plane VMs running Kubernetes, integrated with vSphere infrastructure. These VMs are deployed by the WCP (Workload Control Plane) service on vCenter, which orchestrates NSX-T or vDS networking, load balancer configuration, storage policies, and content library image downloads. If any prerequisite is unmet — networking misconfigured, resources insufficient, images not synced, or WCP itself crashes — the enablement stalls. Since the Supervisor is the foundation for all Tanzu Kubernetes Clusters, its failure cascades to all developer workloads.
Check WCP service status and logs
# SSH to VCSA
# Check WCP service
vmon-cli -s wcp --status
# If not Running:
vmon-cli -s wcp --restart
# Check WCP logs for errors
tail -200 /var/log/vmware/wcp/wcpsvc.log
# Common errors:
# "Failed to deploy supervisor control plane VM"
# "Network configuration validation failed"
# "Insufficient resources in cluster"
# Check all Tanzu-related services
vmon-cli -s wcp --status
vmon-cli -s wcpguestcluster --status
Validate networking prerequisites
# For NSX-T networking:
# Verify T0/T1 gateways are configured and connected
# NSX Manager → Networking → Tier-1 Gateways → Check status
# Ensure edge cluster is deployed and healthy
# For vDS + HAProxy:
# Verify HAProxy load balancer is reachable
curl -v https://haproxy-ip:5556/v2/info
# Check HAProxy data plane API endpoint
# Ensure HAProxy CA certificate matches what was provided during enablement
# For NSX Advanced Load Balancer (Avi):
# Verify Avi Controller is connected
# vCenter → Workload Management → Networking → Check Load Balancer status
# Verify management, workload, and egress networks are properly segmented
Fix content library and TKR image sync
# Check content library for TKR images
# vCenter → Content Libraries → Subscribed library for Tanzu
# Verify subscription URL is reachable:
curl -v https://wp-content.vmware.com/v2/latest/lib.json
# Force sync
# Content Library → Sync Now
# Check available TKR versions
kubectl get tkr
# If no TKRs available, content library sync is the issue
# Verify storage policy used for content library
# Must have sufficient space for OVA images (~5GB each)
Check supervisor control plane VMs and resources
# Find supervisor control plane VMs
# vCenter → VMs → Filter by "SupervisorControlPlane"
# Should be 3 VMs, all Powered On
# If VMs are not deployed, check resource availability:
# Cluster needs: 3 x (4 vCPU + 16GB RAM) = 12 vCPU + 48 GB minimum
# Plus resource overhead for workload VMs
# Check events on the cluster
# vCenter → Cluster → Monitor → Events → Filter for "wcp" or "supervisor"
# If stuck in "Configuring", try disable + re-enable:
# vCenter → Workload Management → Cluster → Disable (wait 15min) → Re-enable
# Check supervisor cluster health via kubectl
kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes
kubectl --kubeconfig=/etc/kubernetes/admin.conf get pods -n kube-system
Fix TKC clusters stuck in "Creating"
# Check TKC status
kubectl get tanzukubernetescluster -A
kubectl describe tanzukubernetescluster <name> -n <namespace>
# Look at Events and Conditions sections
# Common fixes:
# 1. TKR image not available → sync content library
# 2. Network not ready → check supervisor networking
# 3. Storage not ready → verify storage policy binding to namespace
# Check cluster API objects
kubectl get cluster,machine,machinedeployment -A
# Machines stuck in "Provisioning" = VM creation issue
# Machines stuck in "Provisioned" = cloud-init/network issue
# Delete and recreate stuck TKC as last resort
kubectl delete tanzukubernetescluster <name> -n <namespace>
# Wait for cleanup, then re-apply YAML
kubectl get tkc shows clusters in "Running" state, and developer workloads can be deployed to namespaces.
Cannot complete operation across linked vCentersUnable to authenticate with the identity sourcePartner unreachable or Replication lag > thresholdCheck vmdir replication status between nodes
# SSH to VCSA (primary node)
# Check vmdir replication partner status
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus \
-h localhost -u administrator -w '<password>'
# Expected output for healthy replication:
# Partner: vcsa-02.corp.local
# Status: Available
# Replication Lag: 0 seconds
# Check vmdir service health
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartners \
-h localhost -u administrator -w '<password>'
# Verify network connectivity to partner (LDAP ports)
curl -v telnet://vcsa-02.corp.local:389
curl -v telnet://vcsa-02.corp.local:636
Verify DNS and NTP between vCenter nodes
# Forward and reverse DNS must work for ALL linked vCenters
nslookup vcsa-01.corp.local
nslookup vcsa-02.corp.local
nslookup <ip-of-vcsa-01> # Reverse lookup
nslookup <ip-of-vcsa-02>
# Check NTP synchronization on both nodes
# Node 1:
timedatectl status
# Node 2:
ssh root@vcsa-02.corp.local timedatectl status
# Time difference must be < 5 seconds
# If NTP drift detected:
/usr/lib/vmware-applmgmt/support/scripts/timesync.py get
/usr/lib/vmware-applmgmt/support/scripts/timesync.py ntp --servers pool.ntp.org
Clean up stale lookup service entries
# List all registered services in lookup service
python /usr/lib/vmidentity/tools/scripts/lstool.py list \
--url https://vcsa-01.corp.local/lookupservice/sdk \
--type cs.identity --no-check-cert
# Identify stale entries (referencing decommissioned/rebuilt vCenter)
# Note the service ID of stale entries
# Unregister stale entries
python /usr/lib/vmidentity/tools/scripts/lstool.py unregister \
--url https://vcsa-01.corp.local/lookupservice/sdk \
--id <stale-service-id> \
--user administrator@vsphere.local --password '<password>' \
--no-check-cert
# After cleanup, restart all services on both nodes
service-control --stop --all
service-control --start --all
Re-establish certificate trust if broken
# Check trusted certificates on each node
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS
# If partner certificates are missing, add them:
# 1. Download partner root CA certificate
wget --no-check-certificate https://vcsa-02.corp.local/certs/download.zip
unzip download.zip -d /tmp/partner-certs/
# 2. Add to trusted store
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish \
--cert /tmp/partner-certs/certs/lin/root.cer \
--login administrator@vsphere.local --password '<password>'
# 3. Refresh registration
# vCenter → Administration → System Configuration → Nodes → Repoint/Refresh
# 4. Verify linked mode
# vSphere Client → Should now show both vCenter inventories
vdcrepadmin shows all partners "Available" with zero lag, global permissions propagate across nodes, and cross-vCenter vMotion succeeds.
No coredump data found on hostesxcli system coredump partition get returns "No partition configured"Configure local coredump partition
# Check current coredump configuration
esxcli system coredump partition list
esxcli system coredump partition get
# If no partition configured, scan for available diagnostic partitions
esxcli system coredump partition set --enable true --smart
# Manually set a specific partition
esxcli system coredump partition set -p "naa.xxx:7" --enable true
# For hosts booting from USB/SD (no local disk):
# Use a coredump FILE on a VMFS datastore instead
esxcli system coredump file add -d datastore1 -f coredump-esxi01
esxcli system coredump file set -p /vmfs/volumes/datastore1/coredump-esxi01
esxcli system coredump file set --enable true
# Verify configuration
esxcli system coredump partition get
# Active: true, Configured: naa.xxx:7 (or file path)
Configure network dump collector (recommended for large environments)
# Deploy ESXi Dump Collector on a Linux server or VCSA
# The VCSA has a built-in dump collector (netdumper service)
# On VCSA:
vmon-cli -s netdumper --status
vmon-cli -s netdumper --start
# On each ESXi host, configure network dump target:
esxcli system coredump network set \
--interface-name vmk0 \
--server-ipv4 10.10.1.100 \
--server-port 6500
esxcli system coredump network set --enable true
# Verify network dump configuration
esxcli system coredump network get
# Enabled: true
# Network Server IP: 10.10.1.100
# Network Server Port: 6500
# VMKernel Interface: vmk0
# Test the dump collector (does NOT crash the host)
esxcli system coredump network check
Collect and extract crash dump after PSOD
# After host reboots from PSOD, collect the dump:
# Option A: Extract from local partition
vmkdump -l # List dumps on the coredump partition
vmkdump -d /dev/disks/naa.xxx:7 -o /tmp/esxi-crash.dump
# Option B: Collect via vm-support bundle
vm-support --include-core-dump
# Creates a .tgz bundle with all logs + crash dump
# Option C: Network dump — check collector server
# On dump collector server:
ls -la /var/core/netdump/
# Files: vmkernel-zdump-* (compressed crash dump)
# Extract crash dump information
vmkdump_extract /tmp/esxi-crash.dump -o /tmp/crash-analysis/
# Quick analysis of the PSOD backtrace
cat /tmp/crash-analysis/vmkernel-log.txt | grep -A20 "PCPU\|backtrace\|panic"
Initial PSOD analysis (before contacting VMware GSS)
# Read the PSOD screen (photograph it during the crash):
# Line 1: "#PF Exception 14 in world xxxxx:vmkernel" → type of crash
# Line 2: Module name and function → which driver/module crashed
# Line 3: Backtrace addresses → code execution path at time of crash
# Decode backtrace from the PSOD screen
# Match module name to known issues:
# - "nmlx5_core" → Mellanox NIC driver issue
# - "lpfc" → Emulex HBA driver issue
# - "nvme_pcie" → NVMe driver issue
# - "vmkernel" → Core kernel bug
# Check if the PSOD matches a known VMware KB:
# Search: https://kb.vmware.com with the error string
# Check /var/log/vmkernel.log for events leading to crash
grep -B50 "PSOD\|panic\|Assert\|coredump" /var/log/vmkernel.log
# Collect for VMware SR:
# 1. vm-support bundle with core dump
# 2. PSOD screenshot
# 3. Hardware event logs (iLO/iDRAC/IMM)
esxcli system coredump partition get shows Active: true with a configured destination, esxcli system coredump network check succeeds for network dump, and vm-support bundle generation completes with core dump included.
*.vmware.com cloud endpointsadministrator@vsphere.local passwordCheck collector appliance health and connectivity
# Access Skyline Collector admin UI
# https://skyline-collector.corp.local:443 → Admin Dashboard
# Check overall health status:
# - Collection Status: Should be "Active"
# - Upload Status: Should be "Successful"
# - Last Upload: Should be within last 24 hours
# SSH to collector and check connectivity
ssh root@skyline-collector.corp.local
# Test outbound connectivity to Skyline cloud
curl -v https://vcsa.vmware.com
curl -v https://skyline.vmware.com
curl -v https://vapp-updates.vmware.com
# Check collector logs
tail -100 /var/log/vmware/skyline/collector.log
# Look for: "Upload failed", "Connection refused", "Certificate error"
Update vCenter credentials and proxy settings
# If vCenter password was rotated:
# Collector Admin UI → vCenter Connections → Edit → Update password
# Or via CLI:
skyline-collector-cli vcenter update \
--hostname vcenter.corp.local \
--username administrator@vsphere.local \
--password '<new-password>'
# If proxy changed:
# Collector Admin UI → Proxy Settings → Update
# Or via CLI:
skyline-collector-cli proxy set \
--host proxy.corp.local \
--port 3128 \
--username proxyuser \
--password '***'
# Verify CEIP is enabled on vCenter (required)
# vCenter → Administration → CEIP → Must be "Joined"
# Or via API:
curl -k -u administrator@vsphere.local:'<password>' \
https://vcenter.corp.local/api/appliance/telemetry
Fix collector disk space and certificate issues
# Check disk usage
df -h
# If /var/log is full:
du -sh /var/log/vmware/skyline/*
# Rotate logs
logrotate -f /etc/logrotate.d/skyline
# If /opt is full (collected data waiting to upload):
du -sh /opt/vmware/skyline/data/*
# Force upload retry
skyline-collector-cli upload --force
# If certificate expired:
# Collector Admin UI → Certificate → Regenerate
# Or reregister collector with VMware Cloud:
skyline-collector-cli register --token "<cloud-org-token>"
# Restart collector services
systemctl restart skyline-collector
systemctl status skyline-collector
Verify data flow and review findings in Skyline Advisor
# Force a collection cycle
skyline-collector-cli collect --now
# Monitor upload progress
tail -f /var/log/vmware/skyline/collector.log | grep -i "upload\|success\|fail"
# After successful upload, check Skyline Advisor portal:
# https://skyline.vmware.com → Log in with VMware Cloud account
#
# Review:
# - Active Findings: Issues detected in your environment
# - Recommendations: Suggested patches, configuration fixes
# - Security Advisories: CVEs affecting your ESXi/vCenter versions
# - Upgrade Paths: Recommended upgrade sequences
# Configure email notifications
# Skyline Advisor → Settings → Notifications → Add email recipients
# Set severity threshold: Critical, Important, or All