VMware vSphere Knowledge Base

Phase 8 — Advanced & Extended ⚠ Break Scenarios ✓ Fix Procedures
10
Advanced Issues
5
Break Scenarios
5
Fix Procedures
Phase 8
Current Phase
71
vSphere+ Cloud Gateway — Subscription & Connectivity Issues
Cloud gateway appliance fails to connect to VMware Cloud, subscription licenses not reflecting, inventory sync stalls
Impact: vSphere+ subscription features unavailable. Cloud-based inventory, lifecycle management, and centralized dashboards in VMware Cloud Console stop receiving data. On-premises workloads continue running but lose cloud management plane visibility.
Symptoms
  • VMware Cloud Console shows gateway as "Disconnected" or "Inactive"
  • Subscription license not applied — vCenter shows evaluation mode
  • Inventory count in Cloud Console is stale or zero
  • Gateway appliance VAMI shows connectivity check failures
  • Logs show Failed to refresh token or Connection timed out to cloud endpoint
  • vCenter event log: Cloud Gateway registration failed
Root Causes
  • HTTP/HTTPS proxy not configured or proxy authentication failing on the gateway appliance
  • DNS cannot resolve VMware Cloud endpoints (*.vmwarecloud.com, *.vmware.com)
  • Firewall blocking outbound HTTPS (443) to VMware Cloud regional endpoints
  • OAuth refresh token expired — requires re-registration of the gateway
  • NTP drift between gateway appliance and vCenter causes token validation failures
  • Gateway appliance disk full — log rotation not configured

The vSphere+ Cloud Gateway is a virtual appliance deployed on-premises that establishes a persistent outbound connection to VMware Cloud. It relays inventory, health data, and lifecycle events from your vCenter to the cloud console. When connectivity breaks — due to proxy changes, firewall rule updates, DNS failures, or expired authentication tokens — the gateway can no longer sync. Since vSphere+ licensing is subscription-based and validated through the cloud, loss of connectivity can eventually cause license enforcement warnings on vCenter.

1

Check gateway appliance health and connectivity

# SSH into the Cloud Gateway appliance
# Check connectivity to VMware Cloud endpoints
curl -v https://console.cloud.vmware.com
curl -v https://gw.apm.vmware.com

# Check proxy configuration
cat /etc/environment | grep -i proxy
cat /opt/vmware/cloudgateway/config/gateway.properties | grep proxy

# Check DNS resolution
nslookup console.cloud.vmware.com
nslookup vcsa.vmware.com
2

Verify NTP synchronization and certificate validity

# Check NTP status on gateway
timedatectl status
ntpq -p

# Verify gateway certificates have not expired
openssl s_client -connect console.cloud.vmware.com:443 -servername console.cloud.vmware.com < /dev/null 2>/dev/null | openssl x509 -noout -dates

# Check local certificate store
ls -la /opt/vmware/cloudgateway/ssl/
3

Re-register the gateway if token is expired

# Check current registration status
/opt/vmware/cloudgateway/scripts/check-registration.sh

# If token expired, generate a new activation token from VMware Cloud Console:
# Cloud Console → vSphere+ → Infrastructure → Add Gateway → Copy Token

# Re-register with new token
/opt/vmware/cloudgateway/scripts/register-gateway.sh --token "<new-activation-token>"
4

Fix proxy or firewall if connectivity is blocked

# Set proxy on the gateway appliance
/opt/vmware/cloudgateway/scripts/set-proxy.sh --host proxy.corp.local --port 3128 --user proxyuser --password '***'

# Required outbound endpoints (allow in firewall):
# *.vmwarecloud.com:443
# *.vmware.com:443
# gw.apm.vmware.com:443
# sddcgw-prod-*.amazonaws.com:443

# Restart gateway services after proxy change
systemctl restart cloudgateway
systemctl status cloudgateway
5

Validate sync is working and check subscription license

# Check gateway sync log
tail -100 /var/log/vmware/cloudgateway/gateway-sync.log

# Verify subscription license in vCenter
# vCenter UI → Administration → Licensing → Verify vSphere+ subscription is active

# Force inventory resync
/opt/vmware/cloudgateway/scripts/force-sync.sh --full
Verified when: VMware Cloud Console shows gateway as "Connected" with a green status, inventory counts match on-premises vCenter, and subscription license is reflected in vCenter licensing page.
72
VM Encryption & vTPM — Key Provider & Crypto Errors
Encrypted VMs fail to power on due to KMS/key provider connectivity loss, certificate trust issues, or missing encryption keys
Impact: All encrypted VMs and vTPM-enabled VMs cannot power on, vMotion, or snapshot. Data at rest remains encrypted and inaccessible. This is a critical outage for environments using VM encryption for compliance (HIPAA, PCI-DSS, GDPR). Loss of encryption keys can mean permanent data loss.
Symptoms
  • VM power-on fails: Cannot complete the operation due to a crypto error
  • vCenter alarm: "Key provider is not accessible"
  • vTPM-enabled VMs show Invalid VM configuration on power-on
  • vMotion of encrypted VMs fails with Crypto key not found on destination host
  • Encrypted vSAN datastore objects inaccessible
  • vpxd.log: CryptoManagerKmip: Failed to get key from KMS server
Root Causes
  • External KMS (KMIP server) is unreachable — network, firewall, or service down
  • KMS client certificate expired on vCenter — trust chain broken
  • STS certificate expired — SSO token cannot be issued for KMS authentication
  • Native Key Provider (NKP) backup not taken — key material lost after vCenter rebuild
  • Key ID referenced by VM no longer exists in KMS (key deleted or rotated incorrectly)
  • vCenter restored from backup but KMS trust was established after backup date

VM Encryption in vSphere relies on a Key Management Server (KMS) — either an external KMIP-compliant server (like HyTrust, Thales, HashiCorp Vault) or the built-in Native Key Provider (NKP). vCenter obtains Key Encryption Keys (KEKs) from the KMS and pushes them to ESXi hosts, which use them to wrap the Data Encryption Keys (DEKs) that encrypt VM files. If vCenter loses contact with the KMS, or if certificates expire, no new crypto operations can be performed. Existing running VMs continue operating (keys are cached in host memory), but any power-on, migration, or snapshot of an encrypted VM will fail.

1

Check KMS connectivity and certificate status

# Check KMS server reachability from vCenter (VCSA)
curl -v --connect-timeout 5 https://kms-server.corp.local:5696

# Check vCenter ↔ KMS trust certificates
# vCenter UI → Configure → Security → Key Providers → Select provider → Check Status

# View KMS certificates via MOB:
# https://vcenter.corp.local/mob/?moid=CryptoManager&doPath=kmipServers

# Check STS certificate validity (often the root cause)
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list --login administrator@vsphere.local
2

Re-establish KMS trust if certificates expired

# If using external KMS — re-upload vCenter client certificate to KMS:
# vCenter → Key Providers → Establish Trust → Download CSR
# Upload CSR to KMS management interface → Get signed certificate → Upload back to vCenter

# If STS certificate is the root issue, renew it first:
# (See KB #44 for STS certificate renewal steps)

# Verify KMS connection after trust re-establishment
# vCenter → Key Providers → "Check Status" should show green
3

For Native Key Provider — restore from backup

# Native Key Provider requires explicit backup before it can be used
# If vCenter was rebuilt and NKP backup exists:
# vCenter → Key Providers → Native Key Provider → Restore from backup file (.p12)

# Check if NKP is active
Get-KeyProvider | Format-Table Name, Type, Status

# If no backup exists, encrypted VMs are UNRECOVERABLE
# Prevention: Always backup NKP immediately after creation
# vCenter → Key Providers → NKP → Back Up → Save .p12 file securely
Critical Warning: If Native Key Provider was never backed up and vCenter is lost, all encrypted VMs and their data are permanently unrecoverable. Always take and secure NKP backups.
4

Push keys to hosts and validate VM crypto operations

# After restoring KMS connectivity, push keys to all hosts:
# vCenter → Host → Configure → Security Profile → Verify "Crypto State: Safe"

# Check host crypto readiness
esxcli system settings encryption get
# Expected: Mode = TPM (or None), Require Secure Boot = true/false

# Try powering on an encrypted VM
# If successful, repeat for all affected VMs

# Check vpxd logs for crypto resolution
grep -i "CryptoManager\|crypto\|kmip" /var/log/vmware/vpxd/vpxd.log | tail -30
Verified when: Key Provider shows "Connected" status, all encrypted VMs can power on, vMotion of encrypted VMs succeeds between hosts, and vpxd.log shows no CryptoManager errors.
73
vGPU & DirectPath I/O — GPU Passthrough Configuration
Configuring GPU passthrough, vGPU profiles, and troubleshooting device reservation conflicts and driver mismatches
Impact: VMs requiring GPU acceleration (VDI desktops, AI/ML workloads, CAD applications) cannot start or experience degraded performance. DirectPath I/O VMs cannot vMotion. Incorrect configuration can cause host PSOD on reboot.
Symptoms
  • VM fails to power on: The device is not available on host
  • vGPU profiles not visible in VM settings — only "Shared Direct" or blank
  • Host reboots after enabling passthrough — PSOD at vmkernel load
  • GPU listed in esxcli hardware pci list but not in passthrough-eligible devices
  • vMotion grayed out: VM has a device attached that prevents migration
  • NVIDIA vGPU Manager VIB not loaded or version mismatch with guest driver
Root Causes
  • IOMMU / Intel VT-d / AMD-Vi not enabled in server BIOS
  • GPU not marked for passthrough in ESXi host configuration
  • NVIDIA vGPU Manager VIB version incompatible with ESXi version or guest driver
  • Multiple VMs attempting to reserve the same physical GPU in DirectPath mode
  • SR-IOV not supported by the specific GPU model or ESXi driver
  • EFI firmware required but VM set to BIOS boot (vGPU requires EFI)
1

Enable IOMMU/VT-d in server BIOS and verify in ESXi

# Verify IOMMU is enabled in ESXi
esxcli hardware pci list | grep -A5 "NVIDIA\|AMD.*Radeon"

# Check passthrough capability
esxcli hardware pci pcipassthru list | head -20

# Verify VT-d / IOMMU status in boot options
esxcli system settings kernel list | grep -i iommu
# Expected: iommu = true / interlIOMMU = true

# If not enabled, reboot into BIOS:
# Intel: Enable "VT-d" under CPU/Processor settings
# AMD: Enable "AMD-Vi" / "IOMMU" under CPU settings
2

Mark GPU for passthrough and install vGPU Manager VIB

# Mark GPU for passthrough via vCenter:
# Host → Configure → Hardware → PCI Devices → Toggle Passthrough on GPU

# Or via CLI:
esxcli hardware pci pcipassthru set -d 0000:3b:00.0 -e true
# Requires host reboot to take effect

# Install NVIDIA vGPU Manager VIB (for shared vGPU, not full passthrough)
esxcli software vib install -v /tmp/NVIDIA-vGPU-VMware-ESXi-8.0-535.154.02.vib

# Verify installation
esxcli software vib list | grep -i nvidia
nvidia-gpu-manager    535.154.02    NVIDIA    VMwareCertified

# Check GPU is recognized by vGPU Manager
nvidia-smi   # Shows GPU status, temperature, driver version
3

Assign vGPU profile to VM

# In vCenter → VM → Edit Settings → Add New Device → PCI Device:
# For DirectPath I/O (full passthrough):
#   Select the physical GPU → Reserve entire device
#   Note: This disables vMotion, HA restart, and DRS for this VM

# For Shared vGPU (recommended):
#   Add New Device → Shared PCI Device → Select vGPU profile
#   Profiles: grid_a100-4q (4GB), grid_a100-8q (8GB), etc.
#   vMotion with vGPU suspend-resume IS supported on vSphere 8.x

# Ensure VM is set to EFI firmware (required for vGPU)
# VM → Edit Settings → VM Options → Boot Options → Firmware: EFI

# Set memory reservation = full (required for GPU passthrough)
# VM → Edit Settings → Resources → Memory → Reservation: Lock all guest memory
4

Install guest driver and validate acceleration

# Inside the guest VM:

# Windows:
# Download NVIDIA vGPU guest driver from NVIDIA Licensing Portal
# Install matching version (must match host vGPU Manager version branch)
# nvidia-smi.exe   → Should show vGPU device

# Linux:
# Install kernel headers: yum install kernel-devel
# Run: sh NVIDIA-Linux-x86_64-535.154.02-grid.run
# nvidia-smi   → Verify vGPU device is present

# Verify GPU acceleration in guest
# Windows: dxdiag → Display → Check "NVIDIA GRID" device
# Linux: glxinfo | grep "OpenGL renderer"
Verified when: nvidia-smi shows GPU on host, vGPU profile assignable to VMs, guest driver loads with nvidia-smi showing vGPU device, and GPU-accelerated applications render correctly.
74
Auto Deploy & Stateless ESXi — PXE Boot Failures
Hosts fail to PXE boot via Auto Deploy — DHCP/TFTP issues, image profile mismatches, host profile application failures
Impact: Hosts cannot boot from the network and remain offline. In stateless deployments, hosts have no local ESXi installation — if Auto Deploy fails, the host is completely non-functional. Cluster capacity is reduced by every host that cannot boot. If the Auto Deploy server itself is down, no stateless hosts can reboot.
Symptoms
  • Host stuck at "Loading VMware ESXi Hypervisor" with no progress
  • PXE boot timeout: PXE-E18: Server response timeout or TFTP timeout
  • Host boots to ESXi but enters "Not configured" state — no networking, no datastore
  • vCenter shows host as non-compliant with host profile after boot
  • Auto Deploy log: No matching deploy rule for host MAC/UUID
  • After power loss, all stateless hosts simultaneously fail to boot
Root Causes
  • DHCP options 66 (TFTP server) and 67 (boot filename) misconfigured or missing
  • TFTP service on vCenter appliance stopped or port 69 blocked by firewall
  • Auto Deploy service (rbd) not running on vCenter
  • Deploy rule does not match the host's MAC address, UUID, or IP range
  • Image profile references a depot that is no longer accessible
  • Host profile contains answers that are missing (IP address, license key)

Auto Deploy uses a PXE/iPXE boot chain: the host NIC sends a DHCP request, receives a TFTP server address (option 66) and boot filename (option 67 — typically undionly.kpxe.vmw-hardwired), then downloads the ESXi boot image from the Auto Deploy server on vCenter. Deploy rules determine which image profile and host profile to apply based on host attributes. If any link in this chain breaks — DHCP, TFTP, Auto Deploy service, deploy rules, image depot, or host profile — the host fails to boot or boots in an unconfigured state.

1

Verify DHCP options and TFTP reachability

# Check DHCP server scope options:
# Option 66 (Boot Server): IP address of vCenter/Auto Deploy server
# Option 67 (Bootfile Name): undionly.kpxe.vmw-hardwired

# Test TFTP from another machine
tftp <vcenter-ip>
tftp> get undionly.kpxe.vmw-hardwired
# Should download successfully

# If TFTP times out, check firewall on vCenter:
iptables -L -n | grep 69
# Ensure UDP port 69 and TCP port 6501-6502 are open
2

Check Auto Deploy (rbd) service on vCenter

# SSH to VCSA and check service status
vmon-cli -s rbd --status
# Should show: Running

# If stopped, start it
vmon-cli -s rbd --start

# Check Auto Deploy logs
tail -100 /var/log/vmware/rbd/rbd.log
# Look for: "Rule evaluation failed" or "Image profile not found"

# Verify deploy rules via PowerCLI
Connect-VIServer vcenter.corp.local
Get-DeployRule | Format-Table Name, Pattern, ItemList
3

Fix deploy rules and image profile associations

# List current deploy rules and their order
Get-DeployRule | Select-Object Name, @{N='Pattern';E={$_.PatternList}}, @{N='Items';E={$_.ItemList}}

# Create or fix a deploy rule
New-DeployRule -Name "DataCenter-Hosts" `
  -Item (Get-VMHostProfile -Name "Production-Profile"), `
        (Get-DeployRuleImageProfile -Name "ESXi-8.0U2-standard") `
  -Pattern "vendor=Dell", "ipv4=10.10.0.0/24"

# Add rule to active ruleset
Add-DeployRule -DeployRule "DataCenter-Hosts"

# Repair the ruleset (ensures consistency)
Repair-DeployRuleSetCompliance

# Verify depot is accessible
Get-DeployRuleImageProfile | Select-Object Name, Vendor, CreationTime
4

Fix host profile answer file and reboot host

# Check host profile compliance
Test-VMHostProfileCompliance -VMHost esxi-01.corp.local

# Update answer file with required values (IP, license, etc.)
$hp = Get-VMHostProfile -Name "Production-Profile"
Export-VMHostProfile -Profile $hp -FilePath C:\profile-answer.csv
# Edit CSV with per-host answers, then import:
Import-VMHostProfile -FilePath C:\profile-answer.csv -Profile $hp

# Reboot the host to trigger Auto Deploy boot
Restart-VMHost -VMHost esxi-01.corp.local -Confirm:$false
# Host should PXE boot, download image, apply host profile
Verified when: Host successfully PXE boots from Auto Deploy, appears in vCenter inventory with correct image profile, host profile compliance shows "Compliant", and all VMs can be started on the host.
75
vLCM Image-Based Updates — Lifecycle Manager Deep Issues
Cluster image compliance failures, firmware update errors, depot sync issues, and vendor add-on conflicts in vLCM
Impact: Hosts cannot be patched or updated, leaving known vulnerabilities unpatched. Firmware updates stall, hardware support manager (HSM) disconnects prevent driver/firmware alignment. Cluster image compliance shows "Non-Compliant" for all hosts, blocking automated remediation.
Symptoms
  • Cluster image compliance: "Non-Compliant — Incompatible" for hosts
  • Remediation stuck at a percentage with no progress for hours
  • Hardware Support Manager shows "Disconnected" or "Unavailable"
  • Error: Component conflict: VIB X requires Y but found Z
  • Depot sync fails: Unable to download metadata from depot URL
  • Host enters maintenance mode but remediation never starts
Root Causes
  • Online depot URL unreachable — proxy, DNS, or firewall blocking HTTPS to VMware depot
  • Vendor add-on (Dell OpenManage, HPE ILO Amplifier) version incompatible with ESXi version
  • VIB dependency conflict — community or third-party VIBs conflict with desired image
  • Hardware Support Manager (HSM) from OEM not registered or authentication expired
  • Insufficient free space on ESXi boot device for staged remediation
  • VCSA disk partition full — depot cache stored in /storage/updatemgr/
1

Verify depot accessibility and sync status

# Check online depot connectivity from VCSA
curl -v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

# Check proxy settings for vLCM
# vCenter → Lifecycle Manager → Settings → Download Settings → Proxy

# Check local depot cache
du -sh /storage/updatemgr/
df -h /storage/

# Force depot resync
# vCenter → Lifecycle Manager → Settings → Sync Now

# Check sync logs
tail -100 /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log
2

Resolve VIB conflicts and component incompatibilities

# Check current host software profile
esxcli software profile get
# Shows: Name, Vendor, VIBs

# List all installed VIBs with acceptance level
esxcli software vib list | grep -i "community\|partner"
# Community-level VIBs often cause conflicts

# Remove conflicting VIB
esxcli software vib remove -n conflicting-vib-name

# Check image compliance detail via PowerCLI
$cluster = Get-Cluster "Production"
$compliance = Test-LCMClusterCompliance -Cluster $cluster
$compliance.HostComplianceResult | Select-Object Host, Status, IncompatibleComponents
3

Fix Hardware Support Manager connectivity

# Check HSM registration status
# vCenter → Lifecycle Manager → Settings → Hardware Support Managers

# For Dell: Verify OpenManage Integration for VMware vCenter (OMIVV) connectivity
# For HPE: Verify HPE OneView or ILO Amplifier Pack connectivity
# For Lenovo: Verify XClarity Integration

# Re-register HSM if expired
# HSM Admin UI → vCenter Connection → Re-authenticate with vCenter credentials

# Verify firmware compliance after HSM reconnection
$cluster = Get-Cluster "Production"
$image = Get-LCMClusterImage -Cluster $cluster
$image.FirmwareAddOn  # Should show vendor firmware bundle
4

Stage and execute remediation

# Check boot device free space (min ~4GB needed for staging)
esxcli storage filesystem list | grep -i boot

# Pre-check remediation (dry run)
# vCenter → Cluster → Updates → Check Compliance → Review impact

# Stage the image (downloads to hosts first)
# vCenter → Cluster → Updates → Stage

# Remediate with sequential mode for safety
# vCenter → Cluster → Updates → Remediate → Sequential → Max parallel: 1

# Monitor remediation progress
tail -f /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

# If stuck, check host-side logs during remediation
esxcli software sources profile list -d /var/tmp/update-staging/
Verified when: All hosts show "Compliant" in cluster image compliance, esxcli software profile get matches the desired image on each host, and Hardware Support Manager shows "Connected" with firmware aligned.
76
vSphere Replication — RPO Violations & Sync Failures
Replication status shows RPO violations, full syncs triggered repeatedly, VR appliance disconnects from vCenter
Impact: Disaster recovery readiness compromised. VMs are not replicated to the target site within the agreed RPO window. In the event of a site failure, recovery point may be hours or days old, resulting in significant data loss. SRM recovery plans may fail if replication is in an error state.
Symptoms
  • vSphere Replication status: "RPO Violation" with red exclamation
  • Full sync repeatedly triggered instead of incremental (delta) sync
  • VR appliance shows "Disconnected" in vCenter plugin
  • Replication error: Could not create a disk on the target datastore
  • SRM recovery plan test: Replication not in sync — cannot recover
  • Bandwidth utilization between sites at 100% during replication windows
Root Causes
  • WAN bandwidth insufficient for the configured RPO and VM change rate
  • Network latency spikes between source and target sites
  • VR appliance memory or disk exhausted — OVA undersized for workload
  • Changed Block Tracking (CBT) reset on VM — forces full initial sync
  • Target datastore full — no space to write replica disks
  • VR appliance certificate expired — mutual TLS between source/target fails

vSphere Replication works by tracking changed blocks (CBT) on source VM disks and asynchronously replicating deltas to a target VR appliance. The RPO (Recovery Point Objective) defines the maximum data loss window. If the replication engine cannot complete a sync cycle within the RPO window — due to bandwidth constraints, appliance overload, storage issues, or CBT resets — the RPO is violated. Repeated CBT resets (from snapshots, storage vMotion, or VM stunts) cause expensive full re-syncs that consume massive bandwidth and extend the violation window further.

1

Check VR appliance health and connectivity

# Access VR appliance VAMI
# https://vr-appliance.corp.local:5480 → Dashboard

# Check VR appliance disk usage
ssh root@vr-appliance.corp.local
df -h
# Critical partitions: /opt/vmware/hms (replication data), /var/log

# Check VR appliance memory
free -m
# If <2GB free, appliance is undersized — deploy larger OVA

# Check connectivity to partner site VR appliance
curl -v https://vr-target.corp.local:8043/health
# Should return 200 OK
2

Diagnose RPO violation root cause

# Check replication status via VR plugin in vCenter
# vCenter → vSphere Replication → Monitor → Outgoing Replications
# Sort by "RPO Status" to find violating VMs

# Check VR server logs for specific errors
tail -200 /opt/vmware/hms/logs/hms.log
# Look for: "RPO violation", "Full sync required", "CBT reset"

# Calculate required bandwidth:
# Required BW = (Daily Change Rate * RPO safety factor) / RPO window
# Example: 50GB daily change, 1hr RPO = 50GB/24hrs * 2 = ~4.2 Gbps

# Check if CBT was reset (forces full sync)
grep -i "cbt\|changedBlockTracking" /opt/vmware/hms/logs/hms.log | tail -20
3

Fix bandwidth, storage, and CBT issues

# If bandwidth limited, adjust RPO to realistic value:
# vCenter → vSphere Replication → Select VM → Configure Recovery → Increase RPO

# If target datastore is full, extend or migrate replicas
# Check target datastore: vCenter → Target → Datastores → Usage

# If CBT keeps resetting, check for snapshots or consolidation:
vim-cmd vmsvc/snapshot.get <vmid>
# Remove unnecessary snapshots to stabilize CBT

# Enable CBT if disabled
vim-cmd vmsvc/power.off <vmid>  # only if maintenance window
# VM → Edit Settings → Advanced → ctkEnabled = TRUE
# VM → Edit Settings → Advanced → scsix:y.ctkEnabled = TRUE

# Configure traffic shaping to prioritize replication
# Network policy on replication port group → Traffic Shaping → Reserve bandwidth
4

Force resync and validate SRM recovery plan

# Force a full resync to re-baseline (schedule during off-peak)
# vCenter → vSphere Replication → Select VM → "Sync Now" → Force Full Sync

# Monitor sync progress
# vCenter → vSphere Replication → Monitor → Shows % complete and ETA

# After sync completes, validate SRM recovery plan
# SRM → Recovery Plans → Select Plan → Test → Run test recovery
# Verify VMs boot at target site with acceptable data state

# Set up RPO violation alarms
# vCenter → Alarms → vSphere Replication → RPO Violation → Email notification
Verified when: All replicated VMs show "OK" RPO status in green, incremental syncs complete within RPO window, SRM recovery plan test succeeds, and VR appliance health is "Normal".
77
Tanzu / Workload Management — Supervisor Cluster Issues
Workload management enablement fails, supervisor control plane VMs not deploying, TKC clusters stuck in "Creating"
Impact: Kubernetes workloads on vSphere cannot be deployed or managed. Developer namespaces are unavailable. TKC (Tanzu Kubernetes Cluster) provisioning stalls. Existing running workloads may continue but cannot scale or self-heal. CI/CD pipelines targeting Tanzu break.
Symptoms
  • "Enable Workload Management" wizard fails at "Configuring" step for hours
  • Supervisor control plane VMs (3x) not deploying or stuck in "Powered Off"
  • TKC cluster: kubectl get tkc -A shows "Creating" indefinitely
  • kubectl commands timeout connecting to Supervisor API
  • vCenter → Workload Management shows "Config Status: Error"
  • Supervisor namespace shows Phase: Terminating and never completes
Root Causes
  • NSX-T networking prerequisites not met (missing T1 gateway, segments, or load balancer)
  • For vDS+HAProxy: HAProxy load balancer not responding or misconfigured data plane
  • Insufficient cluster resources — supervisor VMs require 4 vCPU / 16 GB RAM each (3 VMs)
  • Content Library sync failed — TKR (Tanzu Kubernetes Release) OVA images not downloaded
  • Storage policy not compatible — must be vSAN, VMFS, or NFS policy with required capabilities
  • WCP service crashed on vCenter — wcp process in restart loop

vSphere with Tanzu deploys a "Supervisor Cluster" — three control plane VMs running Kubernetes, integrated with vSphere infrastructure. These VMs are deployed by the WCP (Workload Control Plane) service on vCenter, which orchestrates NSX-T or vDS networking, load balancer configuration, storage policies, and content library image downloads. If any prerequisite is unmet — networking misconfigured, resources insufficient, images not synced, or WCP itself crashes — the enablement stalls. Since the Supervisor is the foundation for all Tanzu Kubernetes Clusters, its failure cascades to all developer workloads.

1

Check WCP service status and logs

# SSH to VCSA
# Check WCP service
vmon-cli -s wcp --status
# If not Running:
vmon-cli -s wcp --restart

# Check WCP logs for errors
tail -200 /var/log/vmware/wcp/wcpsvc.log
# Common errors:
#   "Failed to deploy supervisor control plane VM"
#   "Network configuration validation failed"
#   "Insufficient resources in cluster"

# Check all Tanzu-related services
vmon-cli -s wcp --status
vmon-cli -s wcpguestcluster --status
2

Validate networking prerequisites

# For NSX-T networking:
# Verify T0/T1 gateways are configured and connected
# NSX Manager → Networking → Tier-1 Gateways → Check status
# Ensure edge cluster is deployed and healthy

# For vDS + HAProxy:
# Verify HAProxy load balancer is reachable
curl -v https://haproxy-ip:5556/v2/info
# Check HAProxy data plane API endpoint
# Ensure HAProxy CA certificate matches what was provided during enablement

# For NSX Advanced Load Balancer (Avi):
# Verify Avi Controller is connected
# vCenter → Workload Management → Networking → Check Load Balancer status

# Verify management, workload, and egress networks are properly segmented
3

Fix content library and TKR image sync

# Check content library for TKR images
# vCenter → Content Libraries → Subscribed library for Tanzu

# Verify subscription URL is reachable:
curl -v https://wp-content.vmware.com/v2/latest/lib.json

# Force sync
# Content Library → Sync Now

# Check available TKR versions
kubectl get tkr
# If no TKRs available, content library sync is the issue

# Verify storage policy used for content library
# Must have sufficient space for OVA images (~5GB each)
4

Check supervisor control plane VMs and resources

# Find supervisor control plane VMs
# vCenter → VMs → Filter by "SupervisorControlPlane"
# Should be 3 VMs, all Powered On

# If VMs are not deployed, check resource availability:
# Cluster needs: 3 x (4 vCPU + 16GB RAM) = 12 vCPU + 48 GB minimum
# Plus resource overhead for workload VMs

# Check events on the cluster
# vCenter → Cluster → Monitor → Events → Filter for "wcp" or "supervisor"

# If stuck in "Configuring", try disable + re-enable:
# vCenter → Workload Management → Cluster → Disable (wait 15min) → Re-enable

# Check supervisor cluster health via kubectl
kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes
kubectl --kubeconfig=/etc/kubernetes/admin.conf get pods -n kube-system
5

Fix TKC clusters stuck in "Creating"

# Check TKC status
kubectl get tanzukubernetescluster -A
kubectl describe tanzukubernetescluster <name> -n <namespace>
# Look at Events and Conditions sections

# Common fixes:
# 1. TKR image not available → sync content library
# 2. Network not ready → check supervisor networking
# 3. Storage not ready → verify storage policy binding to namespace

# Check cluster API objects
kubectl get cluster,machine,machinedeployment -A
# Machines stuck in "Provisioning" = VM creation issue
# Machines stuck in "Provisioned" = cloud-init/network issue

# Delete and recreate stuck TKC as last resort
kubectl delete tanzukubernetescluster <name> -n <namespace>
# Wait for cleanup, then re-apply YAML
Verified when: Workload Management shows "Running" config status, 3 supervisor VMs are powered on and healthy, kubectl get tkc shows clusters in "Running" state, and developer workloads can be deployed to namespaces.
78
Enhanced Linked Mode — Multi-vCenter Replication Issues
Linked vCenter not visible in inventory, cross-vCenter operations fail, vmdir replication lag, SSO domain conflicts
Impact: Unified management view across vCenter instances is broken. Administrators cannot see or manage resources across linked vCenters from a single pane. Cross-vCenter vMotion fails. Global permissions and tags are not replicated. Single sign-on between linked nodes may break.
Symptoms
  • Linked vCenter not visible in the vSphere Client inventory tree
  • Cross-vCenter vMotion fails: Cannot complete operation across linked vCenters
  • Global permissions applied on one vCenter not propagating to partner
  • SSO login fails on one node: Unable to authenticate with the identity source
  • vmdir replication status: Partner unreachable or Replication lag > threshold
  • Lookup service entries stale — services registered on old/decommissioned node
Root Causes
  • vmdir replication partner unreachable — network/firewall blocking port 389/636 between vCenters
  • vmdir database corruption or replication lag exceeding threshold
  • DNS resolution failure between vCenter FQDNs (forward and reverse)
  • Certificate trust chain broken — partner vCenter certificates not in trusted store
  • NTP drift between vCenter nodes causes Kerberos/SAML token validation failures
  • Stale lookup service entries from decommissioned or rebuilt vCenter
1

Check vmdir replication status between nodes

# SSH to VCSA (primary node)
# Check vmdir replication partner status
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus \
  -h localhost -u administrator -w '<password>'

# Expected output for healthy replication:
# Partner: vcsa-02.corp.local
# Status: Available
# Replication Lag: 0 seconds

# Check vmdir service health
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartners \
  -h localhost -u administrator -w '<password>'

# Verify network connectivity to partner (LDAP ports)
curl -v telnet://vcsa-02.corp.local:389
curl -v telnet://vcsa-02.corp.local:636
2

Verify DNS and NTP between vCenter nodes

# Forward and reverse DNS must work for ALL linked vCenters
nslookup vcsa-01.corp.local
nslookup vcsa-02.corp.local
nslookup <ip-of-vcsa-01>   # Reverse lookup
nslookup <ip-of-vcsa-02>

# Check NTP synchronization on both nodes
# Node 1:
timedatectl status
# Node 2:
ssh root@vcsa-02.corp.local timedatectl status

# Time difference must be < 5 seconds
# If NTP drift detected:
/usr/lib/vmware-applmgmt/support/scripts/timesync.py get
/usr/lib/vmware-applmgmt/support/scripts/timesync.py ntp --servers pool.ntp.org
3

Clean up stale lookup service entries

# List all registered services in lookup service
python /usr/lib/vmidentity/tools/scripts/lstool.py list \
  --url https://vcsa-01.corp.local/lookupservice/sdk \
  --type cs.identity --no-check-cert

# Identify stale entries (referencing decommissioned/rebuilt vCenter)
# Note the service ID of stale entries

# Unregister stale entries
python /usr/lib/vmidentity/tools/scripts/lstool.py unregister \
  --url https://vcsa-01.corp.local/lookupservice/sdk \
  --id <stale-service-id> \
  --user administrator@vsphere.local --password '<password>' \
  --no-check-cert

# After cleanup, restart all services on both nodes
service-control --stop --all
service-control --start --all
4

Re-establish certificate trust if broken

# Check trusted certificates on each node
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS

# If partner certificates are missing, add them:
# 1. Download partner root CA certificate
wget --no-check-certificate https://vcsa-02.corp.local/certs/download.zip
unzip download.zip -d /tmp/partner-certs/

# 2. Add to trusted store
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish \
  --cert /tmp/partner-certs/certs/lin/root.cer \
  --login administrator@vsphere.local --password '<password>'

# 3. Refresh registration
# vCenter → Administration → System Configuration → Nodes → Repoint/Refresh

# 4. Verify linked mode
# vSphere Client → Should now show both vCenter inventories
Verified when: Both vCenters are visible in vSphere Client inventory, vdcrepadmin shows all partners "Available" with zero lag, global permissions propagate across nodes, and cross-vCenter vMotion succeeds.
79
ESXi Coredump & Crashdump — PSOD Analysis & Configuration
Configuring coredump destinations, setting up network dump collectors, and analyzing PSOD crash data for root cause
Purpose: Proper coredump configuration ensures that when a PSOD (Purple Screen of Death) occurs, the crash data is captured for VMware GSS analysis. Without a configured dump destination, the crash data is lost after reboot, making root cause analysis impossible. This KB covers configuration, collection, and initial analysis of crash dumps.
Symptoms (When Misconfigured)
  • After PSOD recovery: No coredump data found on host
  • VMware GSS requests crash dump but none was saved
  • esxcli system coredump partition get returns "No partition configured"
  • Network dump collector shows no received dumps after host crash
  • Repeated PSODs with no root cause — cannot open support ticket without dump
  • USB/SD boot devices: no local partition available for dump storage
Common PSOD Causes
  • Hardware failure: failed memory DIMM, PCIe card, NIC firmware bug
  • Driver incompatibility with ESXi version (common after upgrade)
  • Storage timeout — HBA driver panic on prolonged APD/PDL
  • VMkernel assertion failure — software bug in ESXi kernel module
  • NMI (Non-Maskable Interrupt) from hardware watchdog or management board
  • Heap exhaustion — kernel heap memory fully consumed
1

Configure local coredump partition

# Check current coredump configuration
esxcli system coredump partition list
esxcli system coredump partition get

# If no partition configured, scan for available diagnostic partitions
esxcli system coredump partition set --enable true --smart

# Manually set a specific partition
esxcli system coredump partition set -p "naa.xxx:7" --enable true

# For hosts booting from USB/SD (no local disk):
# Use a coredump FILE on a VMFS datastore instead
esxcli system coredump file add -d datastore1 -f coredump-esxi01
esxcli system coredump file set -p /vmfs/volumes/datastore1/coredump-esxi01
esxcli system coredump file set --enable true

# Verify configuration
esxcli system coredump partition get
# Active: true, Configured: naa.xxx:7 (or file path)
2

Configure network dump collector (recommended for large environments)

# Deploy ESXi Dump Collector on a Linux server or VCSA
# The VCSA has a built-in dump collector (netdumper service)
# On VCSA:
vmon-cli -s netdumper --status
vmon-cli -s netdumper --start

# On each ESXi host, configure network dump target:
esxcli system coredump network set \
  --interface-name vmk0 \
  --server-ipv4 10.10.1.100 \
  --server-port 6500
esxcli system coredump network set --enable true

# Verify network dump configuration
esxcli system coredump network get
# Enabled: true
# Network Server IP: 10.10.1.100
# Network Server Port: 6500
# VMKernel Interface: vmk0

# Test the dump collector (does NOT crash the host)
esxcli system coredump network check
3

Collect and extract crash dump after PSOD

# After host reboots from PSOD, collect the dump:

# Option A: Extract from local partition
vmkdump -l   # List dumps on the coredump partition
vmkdump -d /dev/disks/naa.xxx:7 -o /tmp/esxi-crash.dump

# Option B: Collect via vm-support bundle
vm-support --include-core-dump
# Creates a .tgz bundle with all logs + crash dump

# Option C: Network dump — check collector server
# On dump collector server:
ls -la /var/core/netdump/
# Files: vmkernel-zdump-* (compressed crash dump)

# Extract crash dump information
vmkdump_extract /tmp/esxi-crash.dump -o /tmp/crash-analysis/

# Quick analysis of the PSOD backtrace
cat /tmp/crash-analysis/vmkernel-log.txt | grep -A20 "PCPU\|backtrace\|panic"
4

Initial PSOD analysis (before contacting VMware GSS)

# Read the PSOD screen (photograph it during the crash):
# Line 1: "#PF Exception 14 in world xxxxx:vmkernel" → type of crash
# Line 2: Module name and function → which driver/module crashed
# Line 3: Backtrace addresses → code execution path at time of crash

# Decode backtrace from the PSOD screen
# Match module name to known issues:
# - "nmlx5_core" → Mellanox NIC driver issue
# - "lpfc" → Emulex HBA driver issue
# - "nvme_pcie" → NVMe driver issue
# - "vmkernel" → Core kernel bug

# Check if the PSOD matches a known VMware KB:
# Search: https://kb.vmware.com with the error string

# Check /var/log/vmkernel.log for events leading to crash
grep -B50 "PSOD\|panic\|Assert\|coredump" /var/log/vmkernel.log

# Collect for VMware SR:
# 1. vm-support bundle with core dump
# 2. PSOD screenshot
# 3. Hardware event logs (iLO/iDRAC/IMM)
Verified when: esxcli system coredump partition get shows Active: true with a configured destination, esxcli system coredump network check succeeds for network dump, and vm-support bundle generation completes with core dump included.
80
VMware Skyline Health — Proactive Findings & Recommendations
Skyline collector not uploading data, findings not appearing in Skyline Advisor, stale health status, collector appliance issues
Purpose: VMware Skyline is a proactive support technology that analyzes telemetry from your vSphere environment, identifies potential issues before they cause outages, and provides automated recommendations. Keeping Skyline healthy ensures you get early warnings about security vulnerabilities, configuration drift, and known product defects.
Symptoms
  • Skyline Advisor portal shows "No recent data" or last upload > 48 hours ago
  • Skyline Collector VAMI: "Upload Status: Failed"
  • Collector health shows "vCenter connection lost"
  • Findings in Skyline Advisor are stale — don't reflect recent changes
  • Collector appliance unreachable on port 443 (VAMI UI)
  • Email notifications from Skyline stopped arriving
Root Causes
  • Outbound HTTPS connectivity blocked to *.vmware.com cloud endpoints
  • Collector appliance certificate expired — cannot authenticate to VMware cloud
  • vCenter credentials rotated — collector using old administrator@vsphere.local password
  • Proxy configuration changed — collector not updated with new proxy details
  • Collector appliance disk full — /var/log or /opt consuming all space
  • CEIP (Customer Experience Improvement Program) disabled in vCenter — required for Skyline
1

Check collector appliance health and connectivity

# Access Skyline Collector admin UI
# https://skyline-collector.corp.local:443 → Admin Dashboard

# Check overall health status:
# - Collection Status: Should be "Active"
# - Upload Status: Should be "Successful"  
# - Last Upload: Should be within last 24 hours

# SSH to collector and check connectivity
ssh root@skyline-collector.corp.local

# Test outbound connectivity to Skyline cloud
curl -v https://vcsa.vmware.com
curl -v https://skyline.vmware.com
curl -v https://vapp-updates.vmware.com

# Check collector logs
tail -100 /var/log/vmware/skyline/collector.log
# Look for: "Upload failed", "Connection refused", "Certificate error"
2

Update vCenter credentials and proxy settings

# If vCenter password was rotated:
# Collector Admin UI → vCenter Connections → Edit → Update password
# Or via CLI:
skyline-collector-cli vcenter update \
  --hostname vcenter.corp.local \
  --username administrator@vsphere.local \
  --password '<new-password>'

# If proxy changed:
# Collector Admin UI → Proxy Settings → Update
# Or via CLI:
skyline-collector-cli proxy set \
  --host proxy.corp.local \
  --port 3128 \
  --username proxyuser \
  --password '***'

# Verify CEIP is enabled on vCenter (required)
# vCenter → Administration → CEIP → Must be "Joined"
# Or via API:
curl -k -u administrator@vsphere.local:'<password>' \
  https://vcenter.corp.local/api/appliance/telemetry
3

Fix collector disk space and certificate issues

# Check disk usage
df -h
# If /var/log is full:
du -sh /var/log/vmware/skyline/*
# Rotate logs
logrotate -f /etc/logrotate.d/skyline

# If /opt is full (collected data waiting to upload):
du -sh /opt/vmware/skyline/data/*
# Force upload retry
skyline-collector-cli upload --force

# If certificate expired:
# Collector Admin UI → Certificate → Regenerate
# Or reregister collector with VMware Cloud:
skyline-collector-cli register --token "<cloud-org-token>"

# Restart collector services
systemctl restart skyline-collector
systemctl status skyline-collector
4

Verify data flow and review findings in Skyline Advisor

# Force a collection cycle
skyline-collector-cli collect --now

# Monitor upload progress
tail -f /var/log/vmware/skyline/collector.log | grep -i "upload\|success\|fail"

# After successful upload, check Skyline Advisor portal:
# https://skyline.vmware.com → Log in with VMware Cloud account
# 
# Review:
# - Active Findings: Issues detected in your environment
# - Recommendations: Suggested patches, configuration fixes
# - Security Advisories: CVEs affecting your ESXi/vCenter versions
# - Upgrade Paths: Recommended upgrade sequences

# Configure email notifications
# Skyline Advisor → Settings → Notifications → Add email recipients
# Set severity threshold: Critical, Important, or All
Verified when: Collector admin UI shows "Collection: Active" and "Upload: Successful" with timestamp within last 24 hours, Skyline Advisor portal displays current findings for your environment, and proactive email notifications resume.