VMware vSAN Knowledge Base

1

vSAN Disk Group Goes Absent / Degraded

Entire disk group drops out of the vSAN cluster causing data unavailability

Critical Disk Group Data Loss Risk Storage

⚠ Impact: VMs on affected components go stale. If FTT=1 and a second failure occurs, data loss is possible. Rebuilds consume cluster I/O budget.

Symptoms & Root Cause

Symptoms

vSAN Health shows "Disk group – absent" or "degraded"
Objects report reduced availability in vSAN object health
Cache tier SSD latency spikes >50 ms before failure
esxcli vsan storage list shows disk group as Unmounted
vmkernel.log: LSOM: LSOMLostAccess

Root Causes

Cache-tier SSD failure (most common — kills entire disk group)
Multiple capacity disks fail simultaneously
HBA/controller firmware bug or crash
Cable or backplane connectivity issue
ESXi host PSOD causing ungraceful detach

Break — How It Happens

A vSAN disk group is anchored by its cache-tier SSD. If that device experiences a media error, the entire disk group (cache + all capacity disks) is removed from the cluster. All vSAN objects with components on that disk group lose a replica. If your storage policy uses FTT=1, you are now one failure away from data loss.

Diagnostic Commands

1

Check disk group status

esxcli vsan storage list
# Look for "In CMMDS: false" or "Is SSD: true" with Unmounted state

2

Check SMART data on the suspected disk

esxcli storage core device smart get -d naa.xxxxxxxxxxxx
# Look for Reallocated Sector Count, Media Errors, Wear Leveling

3

Review vmkernel logs for LSOM errors

grep -i "lsom\|VSAN\|diskgroup" /var/log/vmkernel.log | tail -100
# Key patterns: LSOMLostAccess, congestion, IO error

4

Verify object health from vCenter

# RVC (Ruby vSphere Console)
vsan.disks_stats /dc/computers/cluster-name
vsan.obj_status_report /dc/computers/cluster-name

Fix Procedure

1

If the cache SSD failed — remove the disk group

# From vSphere Client:
# Host → Configure → vSAN → Disk Management → Select disk group → Remove
# Choose "Migrate data to other hosts" (Full data migration) if capacity allows

# CLI alternative:
esxcli vsan storage remove -s naa.CACHE_SSD_NAA_ID

⚠ Always choose Full data migration unless the cluster has insufficient capacity. "No data migration" causes immediate data loss for objects with components only on this disk group.

2

Replace the failed physical disk

Follow your hardware vendor's hot-swap procedure. Verify the new disk is visible:

esxcli storage core device list | grep -A5 "naa."
# Confirm new device appears with correct size

3

Recreate the disk group

# From vSphere Client:
# Host → Configure → vSAN → Disk Management → Create Disk Group
# Select new cache SSD + existing capacity disks

# CLI:
esxcli vsan storage add -s naa.NEW_CACHE_SSD -d naa.CAPACITY_DISK_1 -d naa.CAPACITY_DISK_2

4

Monitor resync progress

# vSphere Client: Cluster → Monitor → vSAN → Resyncing Objects
# CLI:
esxcli vsan debug resync summary
# Wait for "Total Bytes To Resync: 0" before proceeding

🛈 Prevention: Enable vSAN Health proactive monitoring. Set up SMART-based alerts. Use FTT=1 minimum for production workloads. Keep at least 30% slack space for rebuilds.

2

vSAN Network Partition (Split-Brain)

Hosts lose vSAN network connectivity causing cluster partition and object inaccessibility

Critical Network Split Brain CMMDS

⚠ Impact: VMs become inaccessible when quorum is lost. Both partitions may attempt to own objects, leading to stale data. Storage I/O halts for affected VMs.

Symptoms & Root Cause

Symptoms

vSAN Health: "Network – All hosts have a vSAN vmkernel" → FAIL
Hosts show as partitioned in vSAN Health > Network
VMs report "File locked" or become unresponsive
esxcli vsan cluster get shows different Sub-Cluster Master UUID for different hosts
vmkernel.log: CMMDS: Node lost

Root Causes

Physical switch failure or misconfigured VLAN trunking
vSAN VMkernel adapter (vmk1) IP misconfiguration
MTU mismatch between hosts and switches (9000 vs 1500)
NIC firmware/driver bug causing silent drops
Spanning Tree topology change blocking vSAN VLAN

Break — How It Happens

vSAN relies on a dedicated VMkernel network (typically vmk1) for all cluster communication — CMMDS (Cluster Monitoring, Membership, and Directory Service), object I/O, and resync traffic. When this network is disrupted, hosts form isolated sub-clusters. Each sub-cluster elects its own master. Objects that span both partitions lose quorum and become inaccessible.

Diagnostic Commands

1

Check vSAN cluster membership from each host

# Run on EACH host to compare Sub-Cluster Master UUID:
esxcli vsan cluster get

# If hosts show DIFFERENT "Sub-Cluster Master UUID" → confirmed partition

2

Test vSAN network connectivity between hosts

# From each host, ping vSAN vmkernel of other hosts:
vmkping -I vmk1 -d -s 8972 <other_host_vsan_ip>
# -d = don't fragment, -s 8972 = test jumbo frame (MTU 9000)
# If this fails, you have an L2/MTU issue

3

Verify VMkernel adapter configuration

esxcli network ip interface list
# Confirm vmk1 exists and has vSAN traffic enabled

esxcli vsan network list
# Verify Agent Group and Multicast/Unicast mode

4

Check MTU end-to-end

esxcfg-vmknic -l
# Compare MTU across all hosts — must be identical
# Check physical switch: show interface trunk / show vlan

Fix Procedure

1

Restore physical network connectivity

Fix the switch port, VLAN configuration, or replace the failed cable/NIC. Verify at the physical layer first.

2

Fix MTU mismatch (if applicable)

# Set consistent MTU on all hosts:
esxcli network ip interface set -i vmk1 -m 9000

# Also set on the vSwitch/dvSwitch:
esxcli network vswitch standard set -v vSwitch1 -m 9000

# VERIFY on physical switch that VLAN supports jumbo frames

3

Verify vSAN cluster re-merges

# After network is restored, cluster should auto-heal within 60s
esxcli vsan cluster get
# All hosts should now show the SAME Sub-Cluster Master UUID

# Monitor resync:
esxcli vsan debug resync summary

4

Validate stale VMs

# Check for VMs still reporting inaccessible:
# vSphere Client → Cluster → Monitor → vSAN → Virtual Objects
# Filter by "Inaccessible" — if any remain, check their placement

✔ Prevention: Use redundant NICs (active/active or active/standby) for vSAN VMkernel. Use dedicated VLAN and switches. Enable vSAN health network checks. Configure unicast mode (vSAN 6.6+) instead of multicast.

3

vSAN Objects Inaccessible or Non-Compliant

VM storage objects fail to meet policy requirements — reduced availability or complete inaccessibility

Critical Objects Policy Compliance Data Availability

⚠ Impact: Non-compliant objects are at risk — they survive fewer failures than the policy requires. Inaccessible objects cause VM storage errors, snapshots fail, and vMotion is blocked.

Symptoms & Root Cause

Symptoms

vSAN Health: "vSAN object health – Reduced availability with no rebuild"
VM storage shows "Inaccessible" in Virtual Objects view
Cannot take snapshots or vMotion specific VMs
vsan.obj_status_report shows NON-COMPLIANT or INACCESSIBLE
Events: "vSAN object is non-compliant" for multiple VMs

Root Causes

Host failure exceeding FTT policy tolerance
Disk group removal or failure
Insufficient cluster capacity for object placement
Storage policy changed but objects not re-synced
CLOM unable to find compliant placement

Diagnostic Commands

1

Get full object health status

# From vSphere Client:
# Cluster → Monitor → vSAN → Virtual Objects
# Filter by "Compliance Status" to find Non-Compliant or Inaccessible objects

# From RVC (Ruby vSphere Console):
vsan.obj_status_report /dc/computers/cluster-name

2

Check specific object by UUID

# From ESXi:
esxcli vsan debug object health summary get

# Get details on a specific object:
esxcli vsan debug object list --uuid <object-uuid>

3

Check CLOM placement status

# From vmkernel.log:
grep -i "CLOM\|placement" /var/log/vmkernel.log | tail -50
# Look for: "Not enough resources" or "Cannot place component"

Fix Procedure

1

Bring failed hosts/disks back online

Most non-compliant objects self-heal once the failed component returns. Wait for automatic resync.

2

Force repair for objects stuck in "Reduced availability with no rebuild"

# Option A: Wait 60 minutes — vSAN auto-triggers rebuild after CLOM timeout

# Option B: Trigger immediate repair via RVC:
vsan.obj_status_report /dc/computers/cluster-name --fix-non-compliant

# Option C: From vSphere Client:
# Cluster → Monitor → vSAN → Resyncing Objects → "Repair Objects Immediately"

3

If cluster lacks capacity — adjust storage policy

# Temporarily lower FTT if needed (NOT recommended for production):
# VM → Policies and Profiles → Edit storage policy
# FTT=1, RAID-5 uses less capacity than RAID-1 (mirroring)

# Better: Add a host or disk group to increase capacity

4

Reapply storage policy to force rebalance

# vSphere Client:
# Select VM → Policies and Profiles → Reapply VM Storage Policy
# This triggers CLOM to find new compliant placement

🛈 Key Concept: vSAN objects have a 60-minute CLOM timeout. After a host disappears, vSAN waits 60 minutes before rebuilding. This prevents unnecessary rebuilds during planned maintenance. Use "Repair Objects Immediately" only when you're sure the host is permanently gone.

4

vSAN Resync / Rebuild Stalled or Extremely Slow

Object resync makes no progress or takes days — cluster remains at risk

High Resync Performance Rebuild

⚠ Impact: While resync is stalled, objects remain non-compliant and are vulnerable to additional failures. Extended resync also degrades VM I/O performance.

Symptoms & Root Cause

Symptoms

Resyncing Objects panel shows progress stuck at same % for hours
"Bytes to Resync" not decreasing over time
VM I/O latency elevated (>20ms) during resync
vSAN Performance tab shows resync bandwidth at 0 MB/s

Root Causes

vSAN resync throttling set too aggressively
Congested or errored vSAN network
Slow/failing disk causing I/O backup
Too many objects resyncing simultaneously
Host-level resource contention (CPU/Memory)

Diagnostic Commands

1

Check resync status and bandwidth

# From ESXi Shell:
esxcli vsan debug resync summary
# Shows: Total Objects Resyncing, Bytes Remaining, Estimated Time

# From vSphere Client:
# Cluster → Monitor → vSAN → Resyncing Objects
# Review "Bytes Remaining" and "Active Resync" counts

2

Check resync throttle settings

# On each ESXi host:
esxcli system settings advanced list -o /LSOM/lsomResyncThrottleThreshold
# Default: 50 (50% of I/O budget)  — lower = more throttling

3

Check for congested disks

esxcli vsan debug disk list
# Look for disks with high congestion values (>0)

vsish -e get /vmkModules/lsom/disks/<disk-uuid>/info
# Check latency and outstanding I/O count

Fix Procedure

1

Increase resync bandwidth (lower throttle)

# From ESXi Shell — run on EACH host in the cluster:
esxcli system settings advanced set -o /LSOM/lsomResyncThrottleThreshold -i 80

# From vSphere Client (per host):
# Host → Configure → System → Advanced System Settings
# Filter: "LSOM.lsomResyncThrottleThreshold" → Edit → Set to 80

# Verify the change:
esxcli system settings advanced list -o /LSOM/lsomResyncThrottleThreshold

⚠ Increasing this above 80 can cause noticeable latency for running VMs. Reset to 50 after resync completes.

2

Check and fix underlying network issues

# Verify no packet drops on vSAN vmkernel:
esxcli network ip interface stats get -i vmk1
# Check for TX/RX errors and dropped packets

# Test network throughput between hosts using vmkping:
vmkping -I vmk1 -d -s 8972 <other_host_vsan_ip>
# -d = don't fragment, -s 8972 = jumbo frame test

# Check vSAN network performance from vSphere Client:
# Cluster → Monitor → vSAN → Performance → Network tab

3

If a specific disk is causing I/O stalls — evacuate and replace

# Put host in maintenance mode with "Evacuate all data":
esxcli system maintenanceMode set -e true -m evacuateAllData

# Replace the disk, exit maintenance:
esxcli system maintenanceMode set -e false

5

ESXi Host Fails to Join vSAN Cluster

Host added to cluster but does not participate in vSAN — shown as "Not a vSAN member"

High Cluster Membership Configuration Networking

Symptoms & Root Cause

Symptoms

Host in cluster but vSAN shows "Not a member of vSAN cluster"
esxcli vsan cluster get returns "Not in a VSAN cluster"
vSAN disk management shows no disks claimed
Cannot create disk groups on the host
vSAN health check: "Cluster health – Host not contributing"

Root Causes

vSAN VMkernel adapter missing or misconfigured (vmk1 with vSAN tag)
vSAN service not enabled on the host
Host has incompatible ESXi version
Storage controller not on vSAN HCL — disks not visible
Firewall blocking vSAN ports (2233, 12321, 12345)

Diagnostic Commands

1

Check vSAN cluster membership

esxcli vsan cluster get
# Expected: Local Node State = AGENT (for non-master) or MASTER
# If "Not in VSAN cluster" → vSAN service is not active

2

Check vSAN VMkernel exists

esxcli vsan network list
# Should show an interface (vmk1 typically) with "Agent Group" set

esxcli network ip interface tag get -i vmk1
# Must include "VSAN" tag

3

Verify firewall rules

esxcli network firewall ruleset list | grep -i vsan
# vsanvp2p, vsanEncryption, vsanhealth should be enabled

Fix Procedure

1

Create or fix the vSAN VMkernel adapter

# If vmk1 is missing:
esxcli network ip interface add -i vmk1 -p "vSAN-PG"
esxcli network ip interface ipv4 set -i vmk1 -I 10.10.10.x -N 255.255.255.0 -t static

# Tag it for vSAN:
esxcli network ip interface tag add -i vmk1 -t VSAN

# Or via vSphere Client:
# Host → Configure → Networking → VMkernel adapters → Add
# Enable "vSAN traffic" checkbox

2

Restart vSAN agent

# Restart the vSAN service to re-attempt cluster join:
/etc/init.d/vsanmgmtd restart
/etc/init.d/vsand restart

# Verify:
esxcli vsan cluster get
# Should now show "Local Node State: AGENT"

3

Enable firewall rules if blocked

esxcli network firewall ruleset set -r vsanvp2p -e true
esxcli network firewall ruleset set -r vsanEncryption -e true
esxcli network firewall ruleset set -r vsanhealth -e true

4

Claim disks and create disk group

# After host joins vSAN, create disk group:
esxcli vsan storage add -s naa.CACHE_SSD -d naa.CAPACITY_1 -d naa.CAPACITY_2

# Or auto-claim via vSphere Client:
# Host → Configure → vSAN → Disk Management → Claim Disks

6

vSAN Stretched Cluster — Witness Host Failure

Witness appliance becomes unreachable, leaving stretched cluster unable to resolve quorum

Critical Stretched Cluster Witness Quorum

⚠ Impact: Without the witness, if one site goes down, the surviving site cannot prove quorum. All VMs with components in the failed site become inaccessible. No automatic failover.

Symptoms & Root Cause

Symptoms

Witness host shows disconnected in vSphere inventory
vSAN Health: "Witness host not found"
Objects show "Reduced availability" — witness component absent
Fault domain view shows only 2 of 3 fault domains active
Stretched cluster cannot survive a site failure in this state

Root Causes

Witness appliance VM crashed or its host failed
WAN link between data sites and witness site is down
Witness ESXi version incompatible after upgrade
Witness VMkernel IP changed or network misconfigured
Witness disk full — thin-provisioned disk ran out of datastore space

Fix Procedure

1

Restore connectivity to the witness

# Ping witness from both sites:
vmkping -I vmk1 <witness_vsan_ip>

# If witness VM is down, power it on from its managing vCenter
# If WAN is down, work with network team to restore

2

If witness appliance is permanently lost — replace it

# Deploy new witness appliance OVA from VMware
# Configure networking (vmk0 = management, vmk1 = vSAN witness traffic)
# In vSphere Client:
# Cluster → Configure → vSAN → Fault Domains → Replace Witness Host
# Select the new witness appliance

3

Emergency — force preferred site if witness + one site are down

# LAST RESORT — Only if preferred site is up but cannot access witness:
# This forces the preferred site to assume quorum

# From any host in the preferred site:
esxcli vsan cluster preferredfaultdomain set --preferred-fault-domain-name "Preferred-Site"

# Then from RVC, force object accessibility:
# rvc: vsan.obj_status_report /dc/computers/cluster --force-accessible
# WARNING: This can cause data inconsistency if the non-preferred site had newer writes

⚠ Forcing quorum is a destructive operation. Only use when both witness AND secondary site are permanently failed. Engage VMware Support (GSS) before performing this.

7

vSAN Encryption — KMS Connectivity Lost

Key Management Server unreachable — encrypted vSAN datastore becomes locked after host reboot

Critical Encryption KMS Security

⚠ Impact: Hosts that reboot while KMS is unreachable cannot unlock their disk groups. The vSAN datastore on that host becomes inaccessible. If multiple hosts reboot, cluster-wide outage is possible.

Symptoms & Root Cause

Symptoms

After host reboot, disk groups show "Locked" in vSAN Disk Management
vSAN Health: "Encryption – KMS connection" → FAIL
vCenter: "Cannot connect to KMS cluster" alarm
vmkernel.log: VSAN: Failed to get key from KMS
All VMs on locked disk groups become inaccessible

Root Causes

KMS server VM or appliance is down
Network/firewall change blocking port 5696 (KMIP)
KMS certificate expired or trust broken
KMS cluster configuration changed without updating vCenter
vCenter HA failover lost KMS trust anchor

Fix Procedure

1

Restore KMS connectivity

# From vCenter: Administration → Key Providers
# Check KMS cluster status — test connection

# Verify network:
# From ESXi host:
nc -zv <kms_ip> 5696
# From vCenter:
curl -k https://<kms_ip>:5696 2>&1 | head

2

If KMS cert expired — re-establish trust

# In vCenter:
# Administration → Key Providers → KMS Cluster → Edit
# Re-upload the KMS server certificate
# Re-establish trust by exchanging certificates

# Verify trust:
# Key Providers → KMS Cluster → "Establish Trust"

3

Unlock disk groups after KMS is restored

# Once KMS is reachable, disk groups should auto-unlock
# If not, restart vSAN services on the affected host:
/etc/init.d/vsand restart

# Or put the host in/out of maintenance mode:
esxcli system maintenanceMode set -e true -m noAction
esxcli system maintenanceMode set -e false

✔ Prevention: Deploy KMS in HA (at least 2 nodes). Use vSphere Native Key Provider (vSphere 7.0 U2+) as backup — it stores keys in the ESXi TPM, surviving KMS outages. Never reboot all hosts simultaneously when KMS is unreachable. Back up KMS keys regularly.

8

vSAN Performance Degradation — High Latency

VMs experience high storage latency (>10ms), low throughput, or intermittent I/O stalls

High Performance Latency Congestion

Symptoms & Root Cause

Symptoms

VM disk latency >10ms in vSAN Performance tab
vSAN Health: "Physical disk – Congestion" warnings
vSAN Performance: Backend latency spikes
Applications (SQL, Exchange) report timeout errors
SSD wear indicator approaching 100%

Root Causes

Cache tier SSD nearing end of life (high wear)
Capacity tier overloaded — disk group >80% used
Mismatched disk performance (1 slow disk bottlenecks group)
Network congestion on vSAN VMkernel
Active resync consuming I/O budget
Dedup/Compression overhead on all-flash cluster

Diagnostic Commands

1

Check per-disk congestion and latency

# vSAN Performance service must be enabled:
# Cluster → Monitor → vSAN → Performance → Physical Disks tab

# CLI:
esxcli vsan debug disk list
# Look for: "Congestion" value > 0 (scale: 0-255)

2

Check SSD wear level

esxcli storage core device smart get -d naa.xxxxxxxxxxxx
# "Wear Leveling Count" — threshold is typically 90-95%

# From vSphere Client:
# Cluster → Monitor → vSAN → Physical Disks → Filter by "Wear Level"

3

Check disk space utilization

esxcli vsan storage list
# Check "Capacity" vs "Used" per disk

# Overall:
# Cluster → Monitor → vSAN → Capacity
# Warning threshold: >80% used

Fix Procedure

1

Replace worn SSDs proactively

If cache SSD wear >80%, plan replacement during a maintenance window. Remove disk group with full data migration, replace SSD, recreate disk group.

2

Add capacity to reduce utilization

# Add new disk group or capacity disks:
esxcli vsan storage add -s naa.NEW_CACHE -d naa.NEW_CAP1 -d naa.NEW_CAP2

# Or add a new host to the cluster to distribute load

3

Evaluate and tune dedup/compression

# Check dedup/compression savings:
# Cluster → Configure → vSAN → Services → Space Efficiency

# If savings are minimal (<1.5x) and latency is high:
# Consider disabling — this requires a rolling evacuation
# vSphere Client → Cluster → Configure → vSAN → Services → Edit

⚠ Disabling dedup/compression triggers a full data rewrite (rolling format change). This takes hours and requires sufficient free capacity. Plan during low-I/O windows.

4

Rebalance objects across the cluster

# Proactive rebalance (vSAN 6.7+):
# Cluster → Monitor → vSAN → Resyncing Objects → "Rebalance Disks"

# CLI (trigger manual rebalance):
esxcli vsan debug object rebalance

9

vSAN Upgrade Failure / On-Disk Format Incompatibility

vSAN on-disk format upgrade fails or cluster runs mixed versions after ESXi upgrade

High Upgrade On-Disk Format Compatibility

Symptoms & Root Cause

Symptoms

vSAN Health: "vSAN disk format version" shows mixed versions
On-disk format upgrade stuck or fails partway through
New features (dedup, encryption, file services) cannot be enabled
ESXi upgrade succeeded but vSAN format remains at old version
Error: "On-disk format upgrade failed – Insufficient resources"

Root Causes

On-disk format upgrade was not triggered after ESXi upgrade
Insufficient free capacity to evacuate a disk group during format change
One host in cluster still on old ESXi version
Pre-check failures (health check not green before upgrade)
Controller/driver not updated to compatible version

Fix Procedure

1

Verify all hosts are on the same ESXi version

# From vSphere Client:
# Cluster → Hosts → Review "Version" column for all hosts
# All hosts MUST be at the same ESXi version before on-disk format upgrade

# From ESXi Shell (run on each host):
vmware -vl
# Shows: VMware ESXi x.x.x build-XXXXXXX

2

Run pre-upgrade health checks

# Ensure ALL vSAN health checks are green:
# Cluster → Monitor → vSAN → Health → Run all checks

# Key checks:
# ✓ Network connectivity
# ✓ Disk health  
# ✓ Object health — no inaccessible objects
# ✓ Capacity — at least 30% free

3

Trigger on-disk format upgrade

# vSphere Client:
# Cluster → Configure → vSAN → Disk Management → "Upgrade Pre-check"
# If pre-check passes → "Upgrade" button

# This performs a ROLLING upgrade:
# - Evacuates one disk group at a time
# - Reformats to new on-disk version
# - Re-adds disk group
# - Moves to next disk group

🛈 On-disk format upgrade is non-reversible. Back up critical VMs before proceeding. The process can take several hours depending on data volume and cluster size.

4

If upgrade fails midway — recover

# Check which disk groups completed and which are pending:
esxcli vsan storage list
# "vSAN Disk Group Format Version" shows current version per group

# If a specific disk group fails:
# 1. Remove the failed disk group (Full Data Migration)
# 2. Recreate it — it will use the new format automatically
# 3. Continue upgrade for remaining groups

10

Host Stuck Entering / Exiting vSAN Maintenance Mode

Host cannot enter maintenance mode due to vSAN data evacuation constraints or exits with errors

High Maintenance Mode Evacuation Operations

Symptoms & Root Cause

Symptoms

Host shows "Entering Maintenance Mode" for hours with no progress
Task stuck at specific percentage (often 0% or 99%)
Error: "Cannot evacuate data — insufficient resources"
Error: "There are VMs with vSAN policy that cannot be satisfied"
Pre-check reports objects that cannot be moved

Root Causes

Not enough capacity on remaining hosts for "Full data migration"
Objects already degraded — cannot afford another component loss
VM with local-only storage policy (FTT=0) blocks evacuation
Orphaned or stranded objects on the host
vSAN resync already in progress consuming resources

Diagnostic Commands

1

Check maintenance mode pre-check

# vSphere Client:
# Right-click host → Maintenance Mode → "Pre-Check vSAN Data Migration"
# This shows EXACTLY which objects cannot be migrated and why

# CLI:
esxcli vsan debug object list --filter-on-host
# Shows objects that would be affected

2

Check for inaccessible/orphaned objects

# RVC:
vsan.check_state /dc/computers/cluster-name

# Look for orphaned objects (no parent VM):
vsan.obj_status_report /dc/computers/cluster-name | grep -i orphan

Fix Procedure

1

Use "Ensure Accessibility" instead of "Full Data Migration"

# If full migration is not possible due to capacity:
esxcli system maintenanceMode set -e true -m ensureObjectAccessibility

# This only moves objects where the host has the ONLY copy
# Objects with replicas on other hosts stay (reduced redundancy during maintenance)
# Much faster and uses less capacity

🛈 Maintenance Mode Options:
Full data migration: Moves all data off host. Safest but needs capacity.
Ensure accessibility: Only moves data where host is the sole owner. Fast, less capacity needed.
No data migration: Leaves data in place. Risk of data loss if another failure occurs during maintenance.

2

Cancel stuck maintenance and retry

# Cancel from vSphere Client:
# Recent Tasks → Right-click "Enter Maintenance Mode" → Cancel

# If task is orphaned and cancel doesn't work:
# Restart management agents:
/etc/init.d/hostd restart
/etc/init.d/vpxa restart

# Then retry with a less aggressive migration mode

3

Remove orphaned objects blocking evacuation

# Identify orphaned objects:
# RVC:
vsan.obj_status_report /dc/computers/cluster-name --filter-type orphan

# Delete orphaned objects (careful — verify they are truly orphaned):
# Contact VMware/Broadcom Support (GSS) for safe orphan removal
# Never delete objects attached to running VMs

4

Free capacity by migrating VMs to other storage (if available)

# Storage vMotion large VMs to non-vSAN datastore temporarily:
# This frees vSAN capacity for evacuation

# From vSphere Client:
# Right-click VM → Migrate → Change storage only
# Select a non-vSAN datastore (local/NFS/iSCSI) → Finish
# Repeat for largest VMs until sufficient vSAN space is freed

71

vSAN Health Service Crash / Stale Health Data

Health checks show stale or incorrect data, health service not running, or health DB corrupted

High Health Service Monitoring Operations

⚠ Impact: Without a functioning health service, administrators lose visibility into cluster health. Real problems go undetected. Stale data can mislead troubleshooting, causing admins to chase phantom issues.

Symptoms & Root Cause

Symptoms

vSAN Health shows "Last checked: X days ago" or never updates
Health checks return "Unknown" or blank results
"Retest" button does nothing or spins indefinitely
vCenter logs: VsanVcClusterHealthSystem error
vSAN Performance Service shows no data after a specific date

Root Causes

vSAN health DB corruption (stats.db on vCenter appliance)
vCenter VMODL version mismatch after partial upgrade
vsanmgmtd service crashed and didn't restart
Disk space full on vCenter appliance blocking DB writes
ESXi host running mismatched vSAN build version

Diagnostic Commands

1

Check vSAN health service status on vCenter

# SSH to vCenter Appliance:
vmon-cli --status vsanmgmtd
# Should return "RUNNING"

# Check the service logs:
tail -200 /var/log/vmware/vsanmgmtd/vsanmgmtd.log | grep -i "error\|exception\|fail"

2

Check health service on each ESXi host

# SSH to ESXi:
/etc/init.d/vsanmgmtd status
# Should return "vsanmgmtd is running"

# Check per-host health:
esxcli vsan health cluster list

Fix Procedure

1

Restart vSAN health service on vCenter

# SSH to vCenter:
vmon-cli --restart vsanmgmtd

# Wait 2-3 minutes, then retest health in vSphere Client
# If still stale, restart the entire vSAN management stack:
vmon-cli --restart vsanmgmtd
vmon-cli --restart vpxd

2

Reset health DB if corrupted

# On vCenter appliance — backup first:
cp /storage/vsanmgmt/statsdb/stats.db /storage/vsanmgmt/statsdb/stats.db.bak

# Remove corrupted DB (service will recreate it):
rm /storage/vsanmgmt/statsdb/stats.db
vmon-cli --restart vsanmgmtd

# Historical data will be lost but health checks resume immediately

⚠ Removing stats.db deletes all historical performance data. Only do this after confirming the DB is corrupted. Always back up first.

3

Restart vSAN health on individual ESXi hosts

# SSH to the affected ESXi host:
/etc/init.d/vsanmgmtd restart

# Verify:
/etc/init.d/vsanmgmtd status
esxcli vsan health cluster list

🛈 Prevention: Monitor vCenter disk space (health DB can grow large). After vCenter upgrades, always verify vSAN health service is running. Set up Skyline Health alerts for proactive notification of health service issues.

72

Stale / Orphaned vSAN Objects Consuming Capacity

Deleted VMs leave behind orphaned components that silently consume datastore space

High Orphan Objects Capacity Cleanup

⚠ Impact: Orphaned objects can consume 5–15% of cluster capacity over time. They are invisible in the vSphere Client VM inventory but occupy real disk space. Clusters slowly run out of capacity with no apparent cause.

Symptoms & Root Cause

Symptoms

vSAN capacity usage higher than sum of all VM sizes
Capacity doesn't decrease after deleting large VMs
vSAN Health: "Invalid state objects" or orphan warnings
RVC shows objects with no associated VM
Maintenance mode pre-check references unknown objects

Root Causes

VM deleted from inventory but storage objects not fully cleaned up
Failed snapshot consolidation leaving orphaned delta disks
Interrupted Storage vMotion leaving partial copies
CLOMD crash during object deletion sequence
Template conversion failure leaving duplicate objects

Diagnostic Commands

1

Identify orphaned objects

# RVC (Ruby vSphere Console) from vCenter:
rvc administrator@vsphere.local@localhost
vsan.obj_status_report /dc/computers/cluster-name

# Look for "Orphaned" or "Inaccessible" objects
# Note the object UUIDs for cleanup

2

Check total orphan capacity waste

# From ESXi:
esxcli vsan debug object list | grep -c "orphan"

# Python script from vCenter (PyVmomi):
# Use vsan.QueryObjectIdentities() to enumerate all objects
# Cross-reference with VM inventory to find orphans

Fix Procedure

1

Safe orphan cleanup via vSphere Client

# vSphere Client (8.0u2+):
# Cluster → Monitor → vSAN → Health → Data
# "Invalid state objects" test shows orphans
# Use "Repair" action which safely deletes confirmed orphans

# Alternatively from RVC:
vsan.purge_inaccessible_vswp_objects /dc/computers/cluster-name

⚠ Critical: Never delete objects manually with objtool unless directed by VMware/Broadcom Support. Deleting the wrong object causes permanent data loss for running VMs.

2

Prevent future orphans

# Ensure proper VM deletion workflow:
# 1. Power off VM
# 2. Right-click → "Delete from Disk" (not "Remove from Inventory")
# 3. Verify vSAN object count decreased

# After snapshot operations, always verify consolidation:
# VM → Snapshots → Consolidate if "Consolidation needed" appears

85

APD / PDL — All Paths Down & Permanent Device Loss

Storage path failure triggers APD or PDL state, causing VM stuns, HA events, and potential data unavailability

Critical APD PDL Storage Paths

⚠ Impact: APD (All Paths Down) means storage is temporarily unreachable — ESXi keeps retrying for up to 140 seconds. PDL (Permanent Device Loss) means the device is gone forever — ESXi gives up immediately. The difference in ESXi behavior is dramatic and understanding it is critical to avoiding data loss.

APD vs PDL — Key Differences

APD (All Paths Down)

Storage is temporarily unreachable (network blip, switch reboot)
ESXi retries I/O for 140 seconds (default APD timeout)
VMs are stunned (frozen) — not crashed
If paths recover within timeout, VMs resume automatically
SCSI sense code: no response (timeout)
vmkernel.log: APD Timeout: Not Responded

PDL (Permanent Device Loss)

Storage device is permanently gone (disk removed, LUN destroyed)
ESXi immediately marks device as lost — no retry
VMs receive I/O errors and may crash or BSOD
SCSI sense code: 0x5/0x25 (Logical Unit Not Supported)
vmkernel.log: PDL: Permanent Device Loss
HA can restart VMs on other hosts (if configured)

Symptoms on vSAN

1

vSAN-specific APD/PDL scenarios

# vSAN can trigger APD when:
# - vSAN VMkernel network goes down (all paths to remote components lost)
# - A cache SSD fails (entire disk group enters APD-like state)
# - Host running DOM owner becomes unreachable

# vSAN can trigger PDL when:
# - A capacity disk is physically removed or fails with SMART errors
# - Controller firmware marks a drive as permanently failed
# - Disk group is manually removed while objects still reference it

# Check current APD/PDL state:
esxcli storage core device list | grep -A2 "Is Perennially Reserved\|Status"
# Look for "dead" or "off" status

Fix Procedure

1

Configure APD/PDL response policies

# vSphere HA settings for APD:
# Cluster → Configure → vSphere HA → Failures and Responses
# → Datastore with APD:
#   Response for VM: "Power off and restart VMs (conservative)"
#   Delay: 140 seconds (match APD timeout)
#   Reset period: 180 seconds

# For PDL:
# → Datastore with PDL:
#   Response: "Power off and restart VMs"
#   (HA will immediately restart VMs on healthy hosts)

# vSAN-specific: Set APD timeout
esxcfg-advcfg -s 140 /Misc/APDTimeout
# Default is 140 seconds — increase for environments with slow failover

2

Handle active APD on vSAN

# If APD is currently occurring:
# 1. Check network connectivity to vSAN VMkernel:
vmkping -I vmk1 <other-host-vsan-ip>

# 2. If network is fine, check disk health:
esxcli vsan debug disk list
esxcli storage core device list -d naa.xxx

# 3. If a single disk caused APD, the disk group will auto-recover
#    after the disk is replaced or the path is restored

# 4. If APD timeout expires and VMs are stunned:
#    HA will restart them on other hosts (if configured)
#    Or manually: vim-cmd vmsvc/power.off <vmid>
#    Then power on elsewhere

⚠ Never force-kill a VM during APD unless HA is configured to handle restart. Forcing power-off during APD may leave vSAN objects in an inconsistent state. Wait for the APD timeout or fix the underlying path issue.

🛈 Best Practice: Always configure HA APD/PDL responses explicitly. The defaults may leave VMs stunned indefinitely. For vSAN, set das.config.fdm.storageAllPathsDownTimeoutEnabled = true to ensure HA acts on storage failures.