VMware vSAN Knowledge Base

Verify disk eligibility from ESXi Shell on each host

# List all disks visible to vSAN
esxcli vsan storage list

# Check for existing partitions blocking claim
esxcli storage core device partition list

# Identify vSAN-eligible disks
esxcli vsan storage automode getCopy

Wipe stale partitions on unclaimed disks (ESXi Shell)

# Remove existing partition table from disk (replace naa.xxx with actual device ID)
# WARNING: This destroys ALL data on the disk
partedUtil delete "/vmfs/devices/disks/naa.xxxxxxxxxxxxxxxxxxxx" 1

# For GPT-formatted disks, delete all partitions
partedUtil mklabel "/vmfs/devices/disks/naa.xxxxxxxxxxxxxxxxxxxx" gptCopy

Verify NTP sync and VMkernel config before retrying

# Check NTP synchronization
esxcli system ntp get

# Verify vSAN VMkernel adapter IP
esxcli network ip interface ipv4 get

# Confirm vSAN traffic tag on vmk adapter
esxcli vsan network listCopy

Validate ESXi build matches VCF BoM, then retry bring-up

# Check ESXi version on host
vmware -vl

# Compare output against VCF compatibility matrix
# Navigate to: SDDC Manager UI > Lifecycle Management > Bundle Management
# Ensure ESXi ISO/build matches BoM for your target VCF versionCopy

Validate host prerequisites from ESXi Shell

# Confirm ESXi version and build
vmware -vl

# Check all disks are clean
esxcli storage core device list | grep -i "Display Name"

# Verify no existing vSAN disk groups
esxcli vsan storage list

# Check NTP sync
esxcli system ntp getCopy

Verify network profile matches existing cluster hosts

# Check vmk interfaces and VLAN assignments
esxcli network ip interface list
esxcli network ip interface ipv4 get

# Verify vSAN vmk has correct traffic type
esxcli vsan network list

# Test MTU end-to-end (must match cluster — typically 9000 for jumbo)
vmkping -I vmk1 -d -s 8972 <existing-host-vSAN-IP>Copy

Commission host properly in SDDC Manager

# In SDDC Manager UI:
# 1. Navigate to Inventory > Hosts
# 2. Click "Commission Hosts"
# 3. Provide host FQDN, credentials, and storage/network pool
# 4. Wait for validation — all checks must pass (green)

# If commission fails, check SDDC Manager logs:
# SSH to SDDC Manager VM
cat /var/log/vmware/vcf/domainmanager/domainmanager.log | grep -i "commission"Copy

Retry host addition after fixing prerequisites

# In SDDC Manager UI:
# Inventory > Workload Domains > Select Domain > Select Cluster > Add Host
# Ensure "Retry" on the failed task or start fresh after decommission

# After host joins, verify vSAN membership:
esxcli vsan cluster get
esxcli vsan storage listCopy

Verify depot connectivity from SDDC Manager

# SSH to SDDC Manager appliance
curl -v https://depot.vmware.com 2>&1 | head -30

# If using a proxy, verify proxy settings
cat /etc/vmware/vcf/domainmanager/application.properties | grep proxy

# Check DNS resolution
nslookup depot.vmware.com

# Verify NTP sync (certificate validation will fail if clock is off)
timedatectl statusCopy

Check storage space and clean stale downloads

# Check disk usage on SDDC Manager
df -h /nfs/vmware/vcf/nfs-mount

# List downloaded bundles
ls -lah /nfs/vmware/vcf/nfs-mount/bundle/

# Remove stale/incomplete bundle files if space is low
# Navigate to SDDC Manager UI > Lifecycle > Bundle Management
# Delete any bundles in "Failed" state before retryingCopy

For air-gapped environments — use offline bundle transfer

# Download bundle from my.vmware.com on a connected machine
# Transfer to SDDC Manager via SCP:
scp VCF-*.tar vcf@sddc-manager:/nfs/vmware/vcf/nfs-mount/bundle/

# In SDDC Manager UI:
# Lifecycle Management > Bundle Management > Upload Bundle
# Select the transferred .tar file

# Alternatively, use the VCF Bundle Transfer Utility (BTU)
# from Broadcom support portal for structured offline transfersCopy

Verify upgrade path and retry

# In SDDC Manager UI:
# Lifecycle > Bundle Management — verify bundle order
# VCF REQUIRES sequential upgrades: e.g., 4.5 → 5.0 → 5.1 → 5.2
# You CANNOT skip intermediate versions

# Run pre-check before apply:
# Lifecycle > Updates > Select Update > Run Pre-Check
# Fix any failures, then ApplyCopy

Immediately verify vSAN VMkernel connectivity

# On each ESXi host — check vSAN vmk is reachable
vmkping -I vmk1 <other-host-vSAN-IP> -d -s 8972

# If vmk1 is down, check the port group and uplink
esxcli network ip interface list
esxcli network vswitch dvs vmware list

# Check vSAN network membership
esxcli vsan network listCopy

If vSAN vmk lost its uplink — emergency fix from ESXi Shell

# List current VDS port group assignments
esxcli network vswitch dvs vmware list

# If vmk1 has no active uplinks, check the physical NIC assignment
esxcli network nic list

# In vSphere Client (if accessible):
# Host > Configure > Networking > VMkernel adapters
# Verify vmk1 is on the correct VDS port group with active uplinks

# CRITICAL: Do NOT reboot the host — fix networking in-placeCopy

Fix the Transport Node Profile before re-applying

# In NSX Manager UI:
# System > Fabric > Profiles > Transport Node Profiles
# Edit the TNP:
#   - Verify uplink mapping: vSAN vmk must remain on a dedicated uplink
#   - TEP VLAN must NOT overlap with vSAN VLAN
#   - MTU must be ≥9000 for vSAN (if using jumbo frames)

# Recommended VDS layout for VCF:
#   Uplink 1+2: Management + vMotion + vSAN (active/standby)
#   Uplink 3+4: NSX overlay TEP + Edge uplinks
# NEVER share the same physical uplinks for TEP and vSAN without proper teamingCopy

Re-apply TNP and validate vSAN health

# After fixing TNP, re-apply from NSX Manager:
# System > Fabric > Nodes > Host Transport Nodes
# Select the failed node > Resolve or Re-apply

# Verify vSAN health post-fix:
esxcli vsan health cluster list

# In vSphere Client:
# Cluster > Monitor > vSAN > Health
# All network tests must show greenCopy

Deploy witness host and validate connectivity

# Deploy the vSAN Witness Appliance OVA (from Broadcom downloads)
# In vSphere Client: Deploy OVF Template > select witness OVA
# Assign witness to management vCenter (NOT workload vCenter)

# After deployment, SSH to witness and verify:
esxcli vsan network list
vmkping -I vmk1 <site-A-vSAN-IP> -d -s 8972
vmkping -I vmk1 <site-B-vSAN-IP> -d -s 8972Copy

Configure stretched cluster via SDDC Manager

# In SDDC Manager UI:
# Inventory > Workload Domains > Create Workload Domain
#   (or modify existing domain)
# Select "Stretched Cluster" deployment type
# Assign hosts to Site A (preferred) and Site B (secondary)
# Specify witness host FQDN and credentials

# SDDC Manager will:
# 1. Configure fault domains (preferred / secondary)
# 2. Set up witness traffic routing
# 3. Apply correct vSAN storage policies (Site Disaster Tolerance = 1)Copy

Validate health after stretch configuration

# In vSphere Client:
# Cluster > Monitor > vSAN > Health
# Check these critical tests:
#   - "Stretched cluster: Witness host within vCenter cluster" — MUST be green
#   - "Stretched cluster: Preferred fault domain healthy"
#   - "Stretched cluster: Secondary fault domain healthy"
#   - "Stretched cluster: Witness host healthy"

# From ESXi on any data host:
esxcli vsan cluster get
# Look for: "Sub-Cluster Master Mode: PREFERRED" or "SECONDARY"Copy

Configure DRS affinity rules for site awareness

# In vSphere Client:
# Cluster > Configure > VM/Host Rules
# Create rules to pin critical VMs to preferred site:
#   Rule Type: "Virtual Machines to Hosts"
#   VMs: [select VMs]
#   Host Group: "Preferred-Site-Hosts"
#   Constraint: "Should run on hosts in group" (soft) or "Must run" (hard)

# This ensures VMs read from local site for lowest latency
# HA will still failover to secondary site if preferred site is lostCopy

Check cluster image compliance

# In vSphere Client:
# Cluster > Updates > Image
# Click "Check Compliance" — wait for scan

# Review non-compliant hosts — expand to see:
#   - Missing components (drivers, firmware)
#   - Extra VIBs (manually installed, not in image)
#   - Version mismatch

# From ESXi Shell on non-compliant host:
esxcli software vib list | grep -v "VMware"
# This shows any non-VMware VIBs that may cause driftCopy

Register Hardware Support Manager for firmware management

# In vSphere Client:
# Administration > vCenter Server Settings > vSphere Lifecycle Manager
# Under "Hardware Support Managers" — register the vendor plugin:
#   - Dell: Dell OpenManage Integration for VMware vCenter (OMIVV)
#   - HPE: HPE OneView for VMware vCenter
#   - Lenovo: Lenovo XClarity Integrator for VMware vCenter

# After registration, edit the cluster image:
# Cluster > Updates > Image > Edit
# Add "Firmware and Drivers Add-on" from vendor depot
# Save and Check Compliance againCopy

Handle maintenance mode failures during remediation

# If host cannot enter maintenance mode due to vSAN:
# Check vSAN data migration mode in remediation settings:
# Cluster > Updates > Image > Remediate > Settings
# Options:
#   "Ensure data accessibility" (fastest, reduced redundancy)
#   "Full data migration" (safest, requires spare capacity)
#   "No data migration" (DANGEROUS — only for single-host test)

# Verify vSAN has capacity to evacuate:
esxcli vsan storage list
# Ensure remaining hosts have enough free space

# Check if DRS can migrate all VMs off the host:
# vSphere Client > Host > VMs tab — look for pinned VMs or anti-affinity rulesCopy

Sync SDDC Manager with vCenter image state

# If SDDC Manager shows different image than vCenter:
# SSH to SDDC Manager
cat /var/log/vmware/vcf/lcm/lcm.log | grep -i "image"

# In SDDC Manager UI:
# Lifecycle Management > check for "Configuration Drift" alerts
# If drift detected — use "Sync Inventory" to re-read from vCenter

# NEVER change cluster image directly in vCenter for VCF domains
# Always apply updates through SDDC Manager Lifecycle workflowCopy

Verify Management Pack and adapter status

# In Aria Operations UI:
# Administration > Solutions > VMware vSAN
# Verify status is "Installed" and "Collecting"

# If not installed:
# Administration > Solutions > Add Solution
# Search for "VMware vSAN" pack — install the matching version

# Check adapter instance:
# Administration > Solutions > VMware vSAN > Adapter Instances
# Verify "Collection Status: Data Receiving"Copy

Validate vCenter cloud account permissions

# In Aria Operations:
# Data Sources > Cloud Accounts > vCenter account
# Click "Test Connection" — must return success

# Required permissions on the vCenter service account:
#   - Host.Config.Storage (vSAN disk operations read)
#   - VirtualMachine.Config.Settings (VM policy read)
#   - StorageProfile.View (SPBM policy read)
#   - vSAN: Read-only at minimum (Cluster-level permission)

# In vSphere Client:
# Administration > Access Control > Roles
# Verify the Aria service account role includes vSAN privilegesCopy

Verify collector network connectivity

# SSH to Aria Operations collector node
# Test connectivity to vCenter:
curl -k https://<vcenter-fqdn>/sdk -o /dev/null -w "%{http_code}"
# Expected: 200 or 301

# Test connectivity to ESXi hosts (required for host-level metrics):
curl -k https://<esxi-host-ip>/sdk -o /dev/null -w "%{http_code}"

# If blocked — open firewall rules:
#   Aria Collector → vCenter: TCP 443
#   Aria Collector → ESXi hosts: TCP 443Copy

Configure vSAN-specific dashboards and alerts

# In Aria Operations:
# Dashboards > vSAN > enable built-in dashboards:
#   - vSAN Cluster Overview
#   - vSAN Capacity Planning
#   - vSAN Performance
#   - vSAN Resync Activity

# Configure alert policies:
# Alerts > Alert Definitions > search "vSAN"
# Enable and tune thresholds:
#   - vSAN Capacity Used > 70% (Warning), > 80% (Critical)
#   - vSAN Resync ETA > 4 hours (Warning)
#   - vSAN Object Health Degraded (Critical — immediate)Copy

Assess the damage — connect directly to ESXi hosts

# If vCenter is down, connect directly to each management host via ESXi Shell
# Check vSAN cluster status:
esxcli vsan cluster get

# Check vSAN health (local view):
esxcli vsan health cluster list 2>/dev/null

# Check disk group health:
esxcli vsan storage list

# Check for partitioned nodes:
esxcli vsan cluster get | grep "Partition"

# List all VMs and their power state:
vim-cmd vmsvc/getallvms
vim-cmd vmsvc/power.getstate <vmid>Copy

If host is down — bring it back online immediately

# If host had a PSOD or power loss:
# 1. Power on the physical server via iLO/iDRAC/IMM
# 2. Wait for ESXi to boot completely
# 3. Verify it rejoins vSAN:
esxcli vsan cluster get

# If disk group failed, recreate it:
# First identify the failed disk group:
esxcli vsan storage list | grep -A5 "Status"

# If cache disk failed — the entire disk group is lost
# New cache disk required. In vSphere Client (if available):
# Host > Configure > vSAN > Disk Management > Create Disk Group

# If vSphere Client is down, use RVC or wait for vCenter recoveryCopy

Recover vCenter first (highest priority)

# vCenter must be recovered first — all other components depend on it

# If vCenter VM is powered off, start it from ESXi host directly:
vim-cmd vmsvc/getallvms | grep -i "vcenter\|vcsa"
vim-cmd vmsvc/power.on <vcenter-vmid>

# Wait for vCenter services to start (can take 10-15 minutes)
# From vCenter SSH (once accessible):
service-control --status --all

# If vCenter cannot start due to I/O errors:
# 1. Check the vSAN datastore is accessible
# 2. Use vSAN RVC to investigate object health:
#    rvc root@localhost > vsan.check_state <cluster>Copy

Recover SDDC Manager and validate VCF state

# After vCenter is up, verify SDDC Manager VM:
vim-cmd vmsvc/getallvms | grep -i "sddc"
vim-cmd vmsvc/power.on <sddc-vmid>

# Wait for SDDC Manager services to start
# SSH to SDDC Manager:
systemctl status domainmanager
systemctl status operationsmanager
systemctl status lcm

# If services are degraded, restart:
systemctl restart domainmanager

# Verify VCF inventory sync:
# SDDC Manager UI > Dashboard
# Check for "Configuration Drift" or "Inventory Sync Failure" alertsCopy

Verify network reachability between clusters

# From a host in the CLIENT cluster, ping a host in the SERVER cluster via vSAN vmk:
vmkping -I vmk1 <server-cluster-host-vSAN-IP> -d -s 8972

# Verify both clusters have vSAN services enabled:
# On server cluster host:
esxcli vsan cluster get

# On client cluster host:
esxcli vsan cluster get

# Confirm both report "Cluster UUID" and "Local Node State: MASTER" or "AGENT"Copy

Mount the remote vSAN datastore

# In vSphere Client:
# Navigate to the CLIENT cluster
# Cluster > Configure > vSAN > Services > Datastore Sharing
# Click "Mount Remote Datastores"
# Select the SERVER cluster's vSAN datastore
# Click "Mount"

# Verify mount:
# Client Cluster > Datastores — the remote vSAN DS should appear
# Hosts in the client cluster can now place VMs on the remote datastore

# From ESXi on client cluster host:
esxcli storage filesystem list | grep -i vsanCopy

Troubleshoot mount failures

# If mount fails, check these:

# 1. Verify vSAN datastore sharing is enabled on SERVER cluster:
# Server Cluster > Configure > vSAN > Services > Datastore Sharing
# "Allow access to remote datastores" must be ENABLED

# 2. Check network connectivity between ALL host pairs:
# Every host in client must reach every host in server via vSAN vmk
vmkping -I vmk1 <each-server-host-IP> -d -s 8972

# 3. Check firewall rules on both clusters:
esxcli network firewall ruleset list | grep -i vsan

# 4. Ensure no IP conflicts on vSAN vmk subnet across clustersCopy

VCF-specific considerations

# In VCF environments, HCI Mesh must be configured via SDDC Manager:
# SDDC Manager UI > Workload Domains > Select Client Domain
# Configure > Storage > Mount Remote vSAN Datastore

# IMPORTANT VCF RULES:
# 1. Never share the management domain vSAN datastore
# 2. Both workload domains must be under the same SDDC Manager
# 3. HCI Mesh operations will appear in SDDC Manager audit log
# 4. Lifecycle updates to the server cluster may temporarily
#    affect I/O on the client cluster (plan maintenance windows)

# Monitor cross-cluster latency:
# vSphere Client > Server Cluster > Monitor > vSAN > Performance
# Check "Backend Latency" for HCI Mesh trafficCopy

Check current certificate status across VCF components

# On vCenter (VCSA SSH):
# Check VMCA root certificate expiry:
/usr/lib/vmware-vmca/bin/certool --getrootca | openssl x509 -noout -dates

# Check ESXi host certificate status:
# vSphere Client > Host > Configure > Certificate
# Verify "Valid To" date and issuer matches VMCA or custom CA

# On SDDC Manager:
# Inventory > Certificates
# Review all component certificate expiry datesCopy

Re-establish KMS trust with new certificates

# In vSphere Client:
# Cluster > Configure > Security > Key Management Servers
# Select the KMS provider
# Click "Establish Trust" — re-exchange certificates

# For KMIP-based KMS:
# 1. Export the new VMCA root cert:
/usr/lib/vmware-vmca/bin/certool --getrootca --cert=/tmp/vmca-root.pem

# 2. Import this cert into the KMS trust store
#    (procedure varies by KMS vendor — HyTrust, Thales, etc.)

# 3. Re-establish trust in vSphere Client
# 4. Verify: Cluster > Configure > Security > "KMS connection status: Connected"Copy

Verify vSAN encryption health after cert fix

# Check vSAN encryption status on each host:
esxcli vsan encryption info get

# Expected output:
#   Encryption Active: true
#   KMS Cluster Name: <your-kms-cluster>
#   DEK Generation ID: <number>

# Run vSAN health check:
esxcli vsan health cluster list | grep -i "encrypt"

# In vSphere Client:
# Cluster > Monitor > vSAN > Health
# "Encryption" section — all tests must be green:
#   - "vCenter and all hosts are connected to Key Management Servers"
#   - "CPU AES-NI is supported on all hosts"Copy

Prevent future cert rotation issues

# ALWAYS use SDDC Manager for certificate operations in VCF:
# SDDC Manager UI > Inventory > Certificates > Rotate Certificates
# This ensures ALL components are updated in the correct order

# Set up certificate expiry monitoring:
# SDDC Manager > Lifecycle > Certificate Dashboard
# Configure alerts for certificates expiring within 60 days

# Before any cert rotation:
# 1. Backup vCenter (file-based backup)
# 2. Snapshot SDDC Manager VM
# 3. Document current KMS trust state
# 4. Schedule a maintenance window — do NOT rotate certs in production hours

# After rotation, validate ALL of these:
# - vCenter accessible
# - ESXi hosts connected to vCenter
# - vSAN health green
# - KMS connected
# - NSX Manager cluster healthyCopy

Optimize DRS for vSAN

# vSphere Client → Cluster → Configure → vSphere DRS
# Set Automation Level: "Partially Automated" (recommended for vSAN)
# Set Migration Threshold: Conservative (3 out of 5)

# Advanced settings to consider:
# TryBalanceVmsPerHost = 1  (helps distribute VMs evenly)
# PercentIdleMBInMemDemand = 25  (reduce sensitivity)

# For vSAN specifically, ensure:
# "VM Overrides" for critical VMs → set to "Manual" to prevent DRS moving themCopy

Separate traffic during maintenance

# Before host maintenance:
# 1. Put DRS in manual mode temporarily
# 2. Manually vMotion VMs off the host in controlled batches
# 3. Wait for vSAN to begin data evacuation
# 4. Monitor resync progress before proceeding

# This prevents DRS from flooding the network with vMotions
# while vSAN is simultaneously trying to resync data

# After maintenance, re-enable DRS automation levelCopy

Check current controller mode

# From ESXi:
esxcli storage core adapter list
# Look for the HBA type — should show "SAS" not "RAID"

# Check with storcli/perccli (Dell) or sasicli (HPE):
/opt/lsi/storcli/storcli64 /c0 show
# "Topology" should show "JBOD" not "RAID0" or "RAID1"

# vSAN HCL check:
esxcli vsan storage list
# Verify controller model matches vSAN HCL entry for "passthrough" modeCopy

Switch controller to passthrough/JBOD mode

# ⚠ WARNING: This requires host downtime and may erase controller config
# 1. Put host in maintenance mode (ensure accessibility)
# 2. Reboot into controller BIOS/UEFI (usually Ctrl+R or F2 at POST)
# 3. Change mode from "RAID" to "HBA" or "JBOD" or "Passthrough"
# 4. Save and reboot

# For Dell PERC controllers (via lifecycle controller):
# System Setup → Device Settings → RAID Controller → Convert to HBA mode

# After reboot, verify ESXi sees individual disks:
esxcli storage core device list | grep -c "naa."
# Should show individual disks, not virtual drivesCopy

PVC stuck in "Pending" state

# Most common issue — PVC not binding to a PV
kubectl describe pvc <pvc-name> -n <namespace>
# Look at "Events" section for error messages

# Common causes:
# 1. StorageClass references a non-existent vSAN policy
kubectl get storageclass -o yaml | grep storagePolicyName
# Fix: verify the policy name matches exactly in vSphere

# 2. CSI driver pods not running
kubectl get pods -n vmware-system-csi
# All pods should be Running. Restart if CrashLoopBackOff:
kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi

# 3. vSAN capacity full — can't create new FCDs
# Check vSAN datastore capacity in vSphere ClientCopy

Configure vSAN StorageClass for Kubernetes

# Create a vSAN-backed StorageClass:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: vsan-default
provisioner: csi.vsphere.vmware.com
parameters:
  storagePolicyName: "vSAN Default Storage Policy"
  datastoreURL: "ds:///vmfs/volumes/vsan:xxxxxx/"
reclaimPolicy: Delete
allowVolumeExpansion: true

# Apply:
kubectl apply -f storageclass.yaml

# For Tanzu with Supervisor:
# Storage policies are mapped automatically via VM Classes
# Namespace → Storage → Add Storage PolicyCopy

Troubleshoot CSI driver connectivity

# CSI driver needs connectivity to vCenter:
kubectl logs deployment/vsphere-csi-controller -n vmware-system-csi -c vsphere-csi-controller
# Look for "Failed to connect to vCenter" errors

# Verify CSI secret has correct vCenter credentials:
kubectl get secret vsphere-config-secret -n vmware-system-csi -o yaml
# Decode and verify vCenter IP, user, and password

# For TKG clusters: check the vSphere Cloud Provider config
# kubeadm configmap → verify vcenter-config sectionCopy

Configure disk claiming rules in Host Profile

# vSAN disk claiming in Host Profile:
# Host Profile → Storage → vSAN → Disk Claiming
# Define rules that match your hardware:

# Rule example (by disk model):
# Cache: Model contains "INTEL_SSDPE2MX" → Tag: SSD, Group: Cache
# Capacity: Model contains "ST2000NM" → Tag: Non-SSD, Group: Capacity

# Or by disk size:
# Cache: Size < 500GB → Cache tier
# Capacity: Size >= 500GB → Capacity tier

# CRITICAL: Rules must be deterministic across all hosts
# Same hardware model must produce same disk assignmentCopy

Troubleshoot host profile compliance failures

# Check compliance:
# Host → Configure → Host Profile → Check Compliance

# Common vSAN-related failures:
# 1. "Disk claiming rules do not match" 
#    → Reference host has different disk models than target
#    → Fix: update claiming rules to be model-agnostic (use size-based)

# 2. "vSAN VMkernel not found"
#    → Host profile requires vmk with vSAN traffic tag
#    → Verify network profile includes vSAN VMkernel definition

# 3. "Cannot enable vSAN — host has no eligible disks"
#    → Boot device being claimed by vSAN rules
#    → Add exclusion rule for boot device (USB/SD/M.2)
#    → Exclude: Device path matches "mpx.vmhba*" (local boot)Copy