VMware vSAN Knowledge Base

Check storage pool device status

# From ESXi Shell:
esxcli vsan storage list
# In ESA mode, all devices show under a single "Storage Pool"
# Look for devices with "Status: Unmounted" or missing entries

esxcli vsan debug disk list
# Check for error counts and congestion values per deviceCopy

Check NVMe device health via SMART

esxcli storage core device smart get -d naa.xxxxxxxxxxxx
# Key indicators:
#   - Media and Data Integrity Errors
#   - Percentage Used (wear level) — alert at >90%
#   - Available Spare — alert when below thresholdCopy

Verify ESA mode is active

esxcli vsan cluster get
# Look for: "Storage Architecture: ESA" (not "OSA")

# From vSphere Client:
# Cluster → Configure → vSAN → Services
# "Storage Architecture" should display "Express Storage Architecture"Copy

Remove the failed device from the storage pool

# From vSphere Client:
# Host → Configure → vSAN → Disk Management → Select failed device → Remove Disk
# vSAN will automatically resync affected components to other devices

# From ESXi Shell:
esxcli vsan storage remove -d naa.FAILED_DEVICE_NAA_IDCopy

Physically replace the NVMe device

Follow the server vendor's hot-plug procedure. Verify the replacement drive is on the vSAN ESA HCL.

# After replacement, verify the new device is visible:
esxcli storage core device list | grep -B2 -A10 "naa."
# Confirm model, size, and NVMe transport typeCopy

Add the new device to the storage pool

# From vSphere Client:
# Host → Configure → vSAN → Disk Management → Claim Disks
# Select the new NVMe device → Add to existing Storage Pool

# From ESXi Shell:
esxcli vsan storage add -d naa.NEW_DEVICE_NAA_IDCopy

Monitor resync and rebalance

# From vSphere Client:
# Cluster → Monitor → vSAN → Resyncing Objects

# From ESXi Shell:
esxcli vsan debug resync summary
# Wait for all objects to return to "Compliant" stateCopy

Check file service agent VM status

# From vSphere Client:
# VMs and Templates → Find VMs named "vSAN File Service Node-*"
# Verify they are powered on and have a valid IP on the file services network

# Check health from the cluster:
# Cluster → Monitor → vSAN → Health → File ServiceCopy

Check file service agent logs

# SSH into the file service agent VM (default: root / same as vCenter SSO):
# Check service logs:
cat /var/log/vdfs/vdfs.log | tail -200

# Check if NFS/SMB daemons are running:
systemctl status nfs-server
systemctl status smbdCopy

Verify DNS and network connectivity

# From a file service agent VM:
nslookup <file-share-fqdn>
ping <dns_server_ip>

# From a client machine, verify the share IP is reachable:
# Linux: showmount -e <file_service_ip>
# Windows: net view \\<file_service_ip>Copy

Restart file service agent VMs

# From vSphere Client:
# Right-click each "vSAN File Service Node-*" VM → Power → Reset

# If the VM won't start, check the datastore for capacity:
# Cluster → Monitor → vSAN → Capacity
# File service agent VMs need space on the vSAN datastoreCopy

Redeploy file service if agents are corrupted

# From vSphere Client:
# Cluster → Configure → vSAN → File Service Shares
# Click "Disable File Services" (this removes agent VMs)
# Wait for cleanup to complete
# Click "Enable File Services" → re-enter configuration:
#   - File services domain (AD domain for SMB)
#   - DNS server IPs
#   - File services network (IP pool, subnet, gateway)
#   - DNS suffixes
# Agent VMs are redeployed automaticallyCopy

Fix AD/DNS integration for SMB shares

# Verify AD join status:
# Cluster → Configure → vSAN → File Service Shares → AD Status

# If "Not joined" or "Error":
# 1. Verify AD credentials and domain controller reachability
# 2. Re-join domain:
#    Cluster → Configure → vSAN → File Service Shares → Edit
#    Re-enter AD domain admin credentials
# 3. Verify DNS forward and reverse lookup zones have entries for file agentsCopy

Verify HCI Mesh connectivity

# From vSphere Client:
# Client Cluster → Configure → vSAN → HCI Mesh
# Check "Remote Datastores" status — should be "Mounted"

# From any client cluster host, ping the server cluster's vSAN IPs:
vmkping -I vmk1 <server_cluster_host_vsan_ip>Copy

Check server cluster health

# From vSphere Client:
# Server Cluster → Monitor → vSAN → Health
# All health checks must be green for HCI Mesh to function

# Verify the server cluster's vSAN datastore is operational:
# Server Cluster → Monitor → vSAN → Capacity
# Ensure available capacity existsCopy

Check vSAN network reachability between clusters

# Ensure vSAN VMkernel adapters on BOTH clusters can communicate:
# Run from a client cluster host:
vmkping -I vmk1 -d -s 8972 <server_host_vsan_ip>

# Verify firewall allows port 2233 (vSAN transport):
nc -zv <server_host_vsan_ip> 2233Copy

Restore network connectivity between clusters

Ensure both clusters' vSAN VMkernel adapters are on the same L2 network or have proper L3 routing configured. Verify VLAN tags match and MTU is consistent.

Unmount and remount the remote datastore

# From vSphere Client:
# Client Cluster → Configure → vSAN → HCI Mesh
# Select the failed remote datastore → "Unmount"
# Wait 30 seconds
# Click "Mount Remote Datastores" → Select the server cluster → Mount

# This forces re-establishment of the cross-cluster connectionCopy

If the server cluster had a failure — VMs auto-recover

# Once the server cluster is healthy again:
# VMs on the remote datastore will automatically resume I/O
# Check for stale VMs:
# Client Cluster → Monitor → vSAN → Virtual Objects → Filter "Remote Datastore"
# If VMs remain stale, power-cycle them (power off → power on)Copy

Pre-checks before key rotation

# From vSphere Client:
# Cluster → Monitor → vSAN → Health → Run all checks
# All checks must be GREEN before proceeding
# Verify KMS connectivity:
# Administration → Key Providers → Test connection

# Ensure no active resyncs:
# Cluster → Monitor → vSAN → Resyncing Objects
# "Objects Resyncing" must be 0Copy

Perform shallow rekey (KEK rotation)

# From vSphere Client:
# Cluster → Configure → vSAN → Services → Encryption
# Click "Generate New Encryption Keys" 
# Select "KEK only (shallow rekey)"
# Click "Apply"

# This operation:
# - Generates a new KEK on the KMS
# - Re-wraps the existing DEK with the new KEK
# - No data is rewritten — completes in minutes
# - No VM I/O impactCopy

Verify key rotation completed

# From vSphere Client:
# Cluster → Monitor → vSAN → Health → Encryption
# "Encryption health" should be green
# "KEK ID" should show the NEW key identifier

# Verify from KMS:
# Check that the new key is active and old key is marked as deactivatedCopy

When to perform deep rekey

• A DEK is suspected to be compromised
• Regulatory audit explicitly requires DEK rotation
• Physical disk was removed without secure erase

Execute deep rekey

# From vSphere Client:
# Cluster → Configure → vSAN → Services → Encryption
# Click "Generate New Encryption Keys"
# Select "DEK and KEK (deep rekey)"
# Click "Apply"

# WARNING: This triggers a ROLLING disk group re-encryption:
# - Each disk group is evacuated, reformatted with new DEK, data rewritten
# - Takes HOURS to DAYS depending on data volume
# - VM I/O will be impacted during the process
# - Requires at least 30% free capacity for data migrationCopy

Monitor deep rekey progress

# From vSphere Client:
# Cluster → Monitor → vSAN → Resyncing Objects
# Deep rekey appears as resync operations
# Track: "Bytes Remaining to Resync" should decrease over time

# Also check Recent Tasks for "Rekey" task status on each hostCopy

Update the vSAN HCL database manually

# From vSphere Client:
# Cluster → Monitor → vSAN → Health → Hardware Compatibility
# Click "Update VCG Database from file"

# Download latest HCL JSON:
# Go to: https://partnerweb.vmware.com/service/vsan/all.json
# Upload the downloaded all.json file

# If vCenter has internet access via proxy:
# Cluster → Configure → vSAN → Health → Internet Connectivity
# Configure proxy settings → Test → "Update HCL DB from Internet"Copy

Force re-run all health checks

# From vSphere Client:
# Cluster → Monitor → vSAN → Health
# Click "Retest" on the specific failing check
# Or click "Retest All" to refresh all health checks

# If checks remain stale, restart the health service on ESXi:
# SSH to each affected host:
/etc/init.d/vsanmgmtd restartCopy

Fix vCenter time synchronization

# Many health check false positives are caused by NTP skew
# From VCSA shell:
# Check current time sync:
timedatectl status

# Configure NTP:
# VCSA VAMI (https://vcsa:5480) → Time → Edit
# Set NTP servers (e.g., pool.ntp.org or internal NTP)
# Restart NTP:
systemctl restart systemd-timesyncdCopy

Reset Skyline Health plugin if persistently broken

# From VCSA shell:
# Restart the vSAN health service:
vmon-cli -r vsanHealth

# Verify service is running:
vmon-cli -s vsanHealth
# Expected: "Status: Running"

# Force plugin re-registration (last resort):
# vSphere Client → Menu → Administration → Client Plugins
# Find "vSAN" → Verify version and statusCopy

Check current space efficiency ratio

# From vSphere Client:
# Cluster → Monitor → vSAN → Capacity
# Review "Deduplication and Compression Overview":
#   - "Before" = logical data written
#   - "After" = physical space used
#   - Ratio below 1.5x may not justify the overhead

# Per-host view:
# Cluster → Monitor → vSAN → Capacity → Capacity Overview (per host breakdown)Copy

Check dedup metadata overhead

# From ESXi Shell:
esxcli vsan debug disk list
# Look at "Dedup Overhead" and "Metadata Usage" per disk

# Check vmkernel logs for dedup-related warnings:
grep -i "dedup\|LSOM.*compress" /var/log/vmkernel.log | tail -50Copy

Evaluate whether to keep dedup/compression enabled

General guidance from VMware/Broadcom:

• Good candidates: VDI desktops, template-heavy environments, test/dev with clones
• Poor candidates: Databases (Oracle, SQL), already-encrypted data, highly random I/O
• If space savings ratio is below 1.3x, the overhead likely exceeds the benefit

Disable dedup/compression if not beneficial

# From vSphere Client:
# Cluster → Configure → vSAN → Services → Space Efficiency
# Click "Edit" → Uncheck "Deduplication and Compression" → Apply

# IMPORTANT: This triggers a ROLLING disk format change:
# - Each disk group is evacuated one at a time
# - Data is rewritten WITHOUT dedup/compression
# - Requires sufficient free capacity (~30% of used space)
# - Process can take many hours

# Monitor progress:
# Cluster → Monitor → vSAN → Resyncing ObjectsCopy

Use "Compression Only" mode (vSAN 7.0 U1+)

# If dedup adds overhead but compression helps:
# Cluster → Configure → vSAN → Services → Space Efficiency
# Click "Edit" → Select "Compression Only" (dedup disabled)
# This avoids dedup metadata overhead while keeping compression savings

# Note: In vSAN ESA, compression is always enabled and significantly
# more efficient due to the single-tier architectureCopy

Check cluster capacity breakdown

# From vSphere Client:
# Cluster → Monitor → vSAN → Capacity
# Review:
#   - "Used" vs "Total Capacity"
#   - "vSAN Overhead" (metadata, checksums)
#   - "Primary VM Data" vs "Replica Overhead" (FTT copies)
#   - "Dedup/Compression" savings (if enabled)

# Per-host breakdown to identify imbalanced hosts:
# Cluster → Monitor → vSAN → Capacity → Per HostCopy

Identify large consumers

# From vSphere Client:
# Cluster → Monitor → vSAN → Virtual Objects
# Sort by "Physical Space Used" (descending) to find largest VMs

# Check for snapshot sprawl:
# Each VM → Snapshots → Snapshot Manager
# Old snapshots consume significant hidden capacityCopy

Immediate relief — delete old snapshots and unused VMs

# From vSphere Client:
# Identify VMs with snapshots:
# Menu → VMs → Filter "Has Snapshots: Yes"
# Delete all non-essential snapshots: VM → Snapshots → Delete All

# Remove orphaned VMDKs and unused templates:
# Datastore → Files → Browse for .vmdk files not attached to any VM

# Power off and delete decommissioned VMsCopy

Migrate VMs to reduce vSAN utilization

# Storage vMotion non-critical VMs to alternative datastores:
# Right-click VM → Migrate → "Change storage only"
# Select a non-vSAN datastore (NFS, iSCSI, local)

# Consider changing storage policy from RAID-1 to RAID-5/6:
# RAID-1 FTT=1 uses 2x capacity (mirror)
# RAID-5 FTT=1 uses 1.33x capacity (parity)
# Change via: VM → Policies → Edit VM Storage PolicyCopy

Add capacity to the cluster

# Option A — Add disks to existing hosts:
# Host → Configure → vSAN → Disk Management → Claim Disks
# Add new capacity disks to existing disk groups

# Option B — Add a new host to the cluster:
# Datacenter → Add Host → Configure vSAN VMkernel → Add to cluster
# Create disk group on new host:
esxcli vsan storage add -s naa.CACHE_SSD -d naa.CAP1 -d naa.CAP2

# After adding capacity, trigger rebalance:
# Cluster → Monitor → vSAN → Resyncing Objects → "Rebalance Disks"Copy

Validate the vLCM image before remediation

# From vSphere Client:
# Cluster → Updates → Image → "Check Compliance"
# Review any warnings about driver/firmware conflicts

# CRITICAL: Verify the storage controller driver in the image
# matches the vSAN HCL entry for your hardware:
# Cluster → Monitor → vSAN → Health → Hardware Compatibility
# Cross-reference controller model + driver version + firmware versionCopy

Configure Hardware Support Manager for firmware updates

# Each server vendor provides an HSM plugin:
#   Dell: Dell OpenManage Integration for VMware vCenter (OMIVV)
#   HPE: HPE OneView for VMware vCenter
#   Lenovo: Lenovo XClarity Integrator

# From vSphere Client:
# Menu → Lifecycle Manager → Settings → Hardware Support Managers
# Add the vendor's HSM → Configure firmware depot
# Cluster → Updates → Image → Edit → Add "Firmware and Drivers Add-on"Copy

Fix stuck maintenance mode during remediation

# If a host is stuck entering maintenance mode during vLCM update:

# Option 1: Change vSAN data migration mode to "Ensure Accessibility":
# Cancel the remediation task
# Right-click host → Maintenance Mode → Enter Maintenance Mode
# Select "Ensure Accessibility" instead of "Full data migration"
# Then re-run remediation

# Option 2: Free capacity first (see KB #17 above)
# Delete snapshots, migrate VMs to other datastoresCopy

Roll back if post-update disk group fails

# If after remediation a host's disk groups won't mount:
# The storage controller driver may be incompatible

# From ESXi Shell (on affected host):
esxcli software vib list | grep -i "scsi\|sas\|nvme\|raid"
# Compare driver version with vSAN HCL requirement

# Roll back to previous image:
# vSphere Client → Cluster → Updates → Image → "Stage and Remediate"
# Select "Rollback to previous image" (requires previous image saved)

# Or manually install the correct driver VIB:
esxcli software vib install -v /path/to/correct-driver.vib
# Reboot the hostCopy

Audit all VM storage policies

# From vSphere Client:
# Menu → Policies and Profiles → VM Storage Policies
# Review each policy's vSAN rules:
#   - "Failures to tolerate" (FTT)
#   - "Failure tolerance method" (RAID-1 or RAID-5/6)
#   - "Force provisioning" (should be "No" for production)
#   - "Object space reservation" (thin vs thick)

# Check which VMs use each policy:
# Click a policy → "VM Compliance" tab → See all assigned VMsCopy

Find VMs with FTT=0 (no protection)

# From vSphere Client:
# Cluster → Monitor → vSAN → Virtual Objects
# Filter by "Compliance Status" 
# Any VM with "No protection" has FTT=0

# Also check the default policy:
# Menu → Policies and Profiles → "vSAN Default Storage Policy"
# Verify FTT is NOT set to 0Copy

Create correct storage policies

# From vSphere Client:
# Menu → Policies and Profiles → VM Storage Policies → Create

# Production Policy example:
#   Name: "vSAN-Production-FTT1-RAID1"
#   Datastore rules: vSAN
#   Failures to tolerate: 1
#   Failure tolerance method: RAID-1 (Mirroring)
#   Force provisioning: No
#   Object space reservation: 0% (thin provisioned)

# Capacity-Efficient Policy example:
#   Name: "vSAN-Production-FTT1-RAID5"
#   Failures to tolerate: 1
#   Failure tolerance method: RAID-5/6 (Erasure Coding)
#   Requires 4+ hosts in clusterCopy

Reassign policies to existing VMs

# From vSphere Client:
# Select VM → Policies and Profiles → Edit VM Storage Policies
# Change from the incorrect policy to the correct one
# Click "Apply to all" to apply to all VMDKs

# For bulk re-assignment:
# Select multiple VMs → Right-click → VM Policies → Edit VM Storage Policies

# After policy change, vSAN automatically resyncs objects
# to match the new policy (adds replicas, changes RAID level)
# Monitor: Cluster → Monitor → vSAN → Resyncing ObjectsCopy

Set the correct default storage policy for the datastore

# From vSphere Client:
# Storage → vSAN Datastore → Configure → General
# "Default Storage Policy" → Edit → Select your production policy

# This ensures any new VM deployed without an explicit policy
# automatically gets FTT=1 protectionCopy

Check iSCSI target service status

# From vSphere Client:
# Cluster → Configure → vSAN → iSCSI Target Service
# Review each target's status (Online/Offline) and assigned host

# From the ESXi host running the iSCSI target:
/etc/init.d/vitd status
# Expected: "vitd is running"Copy

Verify iSCSI network connectivity

# From the initiator (physical server), verify target is reachable:
ping <iscsi_target_ip>

# From ESXi host, verify the iSCSI VMkernel IP is up:
esxcli network ip interface list
# Check the VMkernel adapter tagged for iSCSI traffic

# Verify from initiator:
iscsiadm -m discovery -t sendtargets -p <target_ip>:3260Copy

Restart the iSCSI target service

# From the ESXi host currently assigned as the iSCSI target home:
/etc/init.d/vitd restart

# Verify service is back:
/etc/init.d/vitd status

# Initiators should auto-reconnect within 30-60 seconds
# If not, restart the initiator service on the physical serverCopy

Relocate iSCSI target if the host is failed/maintenance

# From vSphere Client:
# Cluster → Configure → vSAN → iSCSI Target Service
# Select the affected target → "Edit" → Change "Home Host" to a healthy host

# The target IP will migrate to the new host
# Ensure the iSCSI network port group exists on the new hostCopy

Fix CHAP authentication issues

# From vSphere Client:
# Cluster → Configure → vSAN → iSCSI Target Service → Select target
# Click "Edit" → Authentication tab
# Verify CHAP name and secret match the initiator's configuration

# On the initiator side (Linux example):
cat /etc/iscsi/iscsid.conf | grep -i "chap"
# Ensure node.session.auth.username and password match the target configCopy

Check CLOMD status

# On each ESXi host in the cluster:
/etc/init.d/clomd status
# Should return "clomd is running"

# Check CLOMD logs:
grep -i "clomd\|CLOM" /var/log/vmkernel.log | tail -50

# Check for crash traces:
ls -la /var/core/clomd*Copy

Check object repair queue

# From RVC:
vsan.obj_status_report /dc/computers/cluster-name
# Look for objects with "Needs repair" state

# Check repair timer:
esxcli vsan debug object health summaryCopy

Restart CLOMD on the affected host

# SSH to the affected ESXi host:
/etc/init.d/clomd restart

# Wait 5 minutes, then verify repair starts:
esxcli vsan debug resync summary
# Should show non-zero "Total Bytes To Resync"Copy

If CLOMD won't start — restart all vSAN services

# Full vSAN service restart on the affected host:
/etc/init.d/vsanmgmtd restart
/etc/init.d/clomd restart
/etc/init.d/vsand restart

# If still failing, put host in maintenance mode (ensure accessibility)
# Reboot the host and let vSAN re-initializeCopy

Collect vSAN-specific bundle from ESXi

# SSH to each ESXi host — use the vSAN-specific group:
vm-support --group vsan -w /tmp/

# This collects vSAN-specific logs (smaller, faster):
# - vsantraces
# - CLOMD/DOM/LSOM logs
# - vSAN object health data
# - esxcli vsan debug outputCopy

Collect from vCenter Appliance

# vCenter Appliance (VAMI):
# https://vcenter:5480 → Support → Create Support Bundle

# Or via CLI:
# SSH to vCenter:
cd /usr/lib/vmware-vsan/bin/
./vc-support.sh

# Collect vsanmgmtd logs separately if needed:
tar czf /tmp/vsan-vc-logs.tar.gz /var/log/vmware/vsanmgmtd/Copy

Collect via vSphere Client (easiest method)

# vSphere Client → Cluster → Monitor → vSAN → Support
# Click "Export vSAN Support Bundle"
# This collects from ALL hosts + vCenter in one operation
# Select the time range around the incident

# For PSOD, also collect:
# /var/core/ — core dump files
# zdump (vmkernel crash dump from the PSOD host)Copy

Identify the failed disk group

# Check disk group status:
esxcli vsan storage list
# Failed disk group shows "Disk Group State: OFFLINE" or missing

# Identify the failed cache SSD:
esxcli vsan debug disk list
# Cache disk will show "SSD: true" and status "offline" or absent

# Check physical drive health in server management (iDRAC/iLO):
# Locate the failed drive slot for physical replacementCopy

Remove the failed disk group

# Remove from vSphere Client:
# Host → Configure → vSAN → Disk Management
# Select the failed disk group → Remove (with data migration if possible)

# Or from CLI:
esxcli vsan storage remove -s naa.FAILED_CACHE_SSD_NAA

# Wait for removal to complete before physical replacement
# This prevents accidental data loss from pulling the wrong driveCopy

Physical replacement and disk group recreation

# 1. Physically replace the cache SSD (hot-swap if supported)
# 2. Wait for ESXi to detect the new drive:
esxcli storage core device list | grep -i "new\|naa."

# 3. Claim the new SSD for vSAN cache and recreate disk group:
esxcli vsan storage add -s naa.NEW_CACHE_SSD -d naa.CAPACITY_1 -d naa.CAPACITY_2

# Or via vSphere Client:
# Host → Configure → vSAN → Disk Management → Create Disk Group
# Select the new cache SSD + existing capacity disks

# 4. Monitor resync:
esxcli vsan debug resync summary
# vSAN will rebuild components onto the new disk groupCopy

Enable the Performance Service

# vSphere Client:
# Cluster → Configure → vSAN → Services → Performance Service → Turn ON

# Requirements:
# - Uses ~255 GB on the vSAN datastore (stats object)
# - One host becomes the "stats master" (elected automatically)
# - Collects data every 5 minutes (configurable intervals)

# Verify it's running:
# Cluster → Monitor → vSAN → Performance → should show graphs
# If graphs show "No data available" — service may need time to collectCopy

Troubleshoot missing performance data

# 1. Check if stats master is healthy:
esxcli vsan debug object list | grep -i "stats"
# The stats DB object should be "Healthy"

# 2. Restart the performance service:
# vSphere Client → Cluster → Configure → vSAN → Services
# Turn OFF Performance Service → wait 30 seconds → Turn ON

# 3. If stats object is corrupted:
# Delete the performance stats object (loses history):
# Turn OFF service → it auto-deletes the stats DB
# Turn ON → creates fresh stats object
# Data collection resumes from scratch

# 4. Check disk space — stats DB needs ~255 GB free on vSANCopy