VMware vSAN Knowledge Base

Identify congestion using esxtop (real-time)

# SSH to each ESXi host and run esxtop:
esxtop

# Press 'u' for disk device view (vSAN)
# Key columns:
#   DAVG — Device Average latency (should be <2ms for SSD, <10ms for HDD)
#   KAVG — Kernel Average latency (includes vSAN overhead, <5ms normal)
#   GAVG — Guest Average latency (what the VM sees)
#   QUED — Queue depth (high = saturation)

# If KAVG >> DAVG, the bottleneck is in the vSAN stack (congestion)
# If DAVG is high, the physical device is the bottleneck

# For vSAN-specific view, press 'v' for vSAN stats:
#   Look for "CONGESTION" column — any value > 0 is active throttlingCopy

Check vSAN Performance Service for congestion history

# In vSphere Client:
# Cluster > Monitor > vSAN > Performance

# Key charts:
#   "Congestion" — shows congestion value over time (0-255 scale)
#      0 = no congestion
#      1-30 = light throttling (may be imperceptible)
#      31-60 = moderate (applications notice)
#      61+ = severe (applications time out)

#   "Latency" — Read/Write latency graph
#   "IOPS" — drops during congestion (throttled)
#   "Throughput" — drops correlate with congestion spikes

# Per-host breakdown:
# Cluster > Monitor > vSAN > Performance > Host
# Identify which host(s) are the congestion sourceCopy

Identify the root cause tier

# Check OSA write buffer fill level (OSA clusters only):
esxcli vsan storage list
# Look for "Write Buffer Free %" — if below 20%, write buffer is causing congestion

# Check capacity disk utilization:
esxtop
# Press 'd' — look at %BUSY for capacity devices
# If HDD at 100% busy — capacity tier overload

# Check active resync:
# Cluster > Monitor > vSAN > Resyncing Objects
# If resync is active, it competes with VM I/O — see KB #52 for throttling

# Check network RTT between hosts:
vmkping -I vmk1 <other-host-vSAN-IP>
# RTT should be <1ms on same rack, <5ms cross-rack
# If >5ms — investigate physical network

# Check snapshot chains:
# VM > Snapshots > Manage Snapshots
# If chain depth > 3 — consolidate immediatelyCopy

Immediate relief actions

# Action 1: If resync is causing congestion — throttle it:
# Cluster > Configure > vSAN > Services > Resync Throttle → "Low"
# This prioritizes VM I/O over resync

# Action 2: If a single VM is generating excessive I/O:
# VM > Edit Settings > Hard Disk > Shares: "Low" (temporary)
# Or set IOPS Limit in the VM storage policy

# Action 3: If capacity disk is saturated (OSA with HDD):
# Long-term: Replace HDD capacity with SSD
# Short-term: Migrate heavy I/O VMs to hosts with less-loaded disk groups

# Action 4: If write buffer is full (OSA):
# Check cache SSD health — SMART wear level approaching 100% = replace
esxcli vsan storage list | grep -A5 "Cache Tier"
# Replace the cache SSD with a higher endurance model

# Action 5: For ESA clusters:
# ESA rarely has congestion — if it does, check NVMe SMART health
# A failing NVMe causes latency spikes before it fully failsCopy

Check current resync status and throttle setting

# In vSphere Client:
# Cluster > Monitor > vSAN > Resyncing Objects
# Shows:
#   - Total objects being resynced
#   - Bytes remaining
#   - Estimated time to completion

# Current throttle setting:
# Cluster > Configure > vSAN > Services
# "Resync throttle" setting: Low / Medium / High

# From ESXi Shell:
esxcli vsan health cluster list | grep -i "resync"

# Check resync bandwidth consumption:
# Cluster > Monitor > vSAN > Performance > "Resync"
# Shows MB/s dedicated to resync operationsCopy

Adjust resync throttle during business hours

# During production hours — minimize VM impact:
# Cluster > Configure > vSAN > Services
# Set Resync Throttle to: "Low"
# This limits resync to ~30% of available bandwidth

# During maintenance windows — maximize resync speed:
# Set Resync Throttle to: "High"
# This allows resync to consume up to ~90% of available bandwidth

# Medium (default): ~60% bandwidth for resync

# There is no automated schedule — you must change it manually
# Consider creating a scheduled task or alarm action to switchCopy

Adjust the 60-minute repair delay timer

# The default 60-minute delay before full rebuild can be changed:
# This is useful if you know a host will return quickly (planned reboot)

# In vSphere Client:
# Cluster > Configure > vSAN > Advanced Options
# Setting: "ObjectRepairTimer"
# Default: 60 (minutes)
# Range: 0-120 minutes

# Set to 0: Immediate rebuild (risky — waste of I/O if host returns soon)
# Set to 120: Wait 2 hours (use if host replacement takes time but you have
#              sufficient fault tolerance in the meantime)

# For ESA clusters, the repair timer works the same way
# ESA resyncs are faster due to NVMe speed — shorter delay is less riskyCopy

Monitor resync impact on VM workloads

# During active resync, monitor these in parallel:
# Cluster > Monitor > vSAN > Performance

# Watch for:
#   Congestion > 0 → resync is impacting VMs
#   VM Latency > 5ms → applications may notice
#   Resync MB/s → how fast data is rebuilding

# If congestion appears during resync:
# 1. Switch throttle to "Low" immediately
# 2. Identify which VMs are most impacted:
#    VM > Monitor > Performance > Virtual Disk > Latency
# 3. Consider temporarily migrating critical VMs to a less-impacted host

# Resync completion estimate:
# Cluster > Monitor > vSAN > Resyncing Objects
# "ETA" column — plan your throttle changes around thisCopy

Check current capacity status

# In vSphere Client:
# Cluster > Monitor > vSAN > Capacity
# Key metrics:
#   Total Raw Capacity
#   Used Capacity (after dedup/compress for ESA)
#   Free Capacity
#   Dedup/Compression savings (ESA)

# Per-host usage breakdown:
# Cluster > Monitor > vSAN > Capacity > "Used Capacity Breakdown"
# Shows: VM data, vSAN overhead, Swap objects, Performance mgmt objects

# From ESXi Shell:
esxcli vsan storage list
# Shows per-device: Total Capacity, Used, FreeCopy

Configure capacity threshold alarms

# In vSphere Client:
# Cluster > Configure > Alarm Definitions
# Find: "vSAN Datastore Usage on Disk" (or create custom)

# Recommended thresholds:
#   Warning: 70% used (start planning expansion)
#   Critical: 80% used (immediate action required)
#   Emergency: 90% used (VMs at risk — add disks NOW)

# Custom alarm creation:
# Cluster > Configure > Alarm Definitions > Add
# Trigger: Metric > vSAN > Capacity > "Percentage Used"
# Warning: > 70%
# Critical: > 80%
# Action: Send email notification

# vSAN Health also monitors capacity:
# Cluster > Monitor > vSAN > Health > "Capacity"
# "Space efficiency usage health" — flags underperforming dedup ratioCopy

Calculate growth rate and forecast exhaustion

# In vSphere Client:
# Cluster > Monitor > vSAN > Capacity
# Set time range to 30/60/90 days
# Look at the capacity trend graph:
#   - Linear growth → predictable exhaustion date
#   - Exponential growth → faster action needed

# Manual calculation:
#   Current Free: 10 TB
#   30-day growth rate: 500 GB/month
#   Time to exhaustion: 10TB ÷ 0.5TB/month = 20 months
#   Time to 80% threshold: calculate accordingly

# Aria Operations (if available):
# Dashboards > vSAN Capacity Planning
# Shows automated "Days Remaining" forecast
# Much more accurate with ML-based trending

# IMPORTANT: Account for one host failure scenario
# If a host fails, its capacity is lost AND its objects rebuild on remaining hosts
# This can consume 2x the host's used capacity temporarilyCopy

Emergency: cluster approaching 95%+ used

# IMMEDIATE ACTIONS when capacity is critical:

# 1. Identify and delete unnecessary snapshots:
# Cluster > VMs > sort by "Snapshot" column
# Delete all non-essential snapshots immediately

# 2. Identify orphaned VMDKs:
# Datastore Browser > look for .vmdk files not attached to any VM
# Delete orphaned disks after verification

# 3. Storage vMotion VMs to external storage (if available):
# Right-click VM > Migrate > Change Storage Only
# Move to NFS/iSCSI/FC datastore

# 4. Reduce VM swap overhead:
# Set memory reservation on VMs = allocated memory
# This eliminates the swap file (saves GB per VM)
# VM > Edit Settings > Resources > Memory Reservation = full

# 5. Add capacity disks:
# If hardware available, hot-add NVMe/SSD to hosts
# Host > Configure > vSAN > Disk Management > Claim Unused Disks

# 6. Change storage policy to more efficient type:
# RAID-1 → RAID-5 saves 33% capacity (requires policy change per VM)Copy

Disk Device view — identify storage bottleneck

# SSH to ESXi host:
esxtop

# Press 'd' for disk device view
# Key columns (press 'f' to select fields):

# DEVICE — physical device ID (naa.xxx for local, vSAN for vSAN devices)
# CMDS/s — commands per second (IOPS)
# READS/s — read IOPS
# WRITES/s — write IOPS
# MBREAD/s — read throughput
# MBWRTN/s — write throughput
# DAVG/cmd — Device Average latency per command (ms)
#   SSD: should be <1ms
#   NVMe: should be <0.5ms
#   HDD: should be <15ms
# KAVG/cmd — Kernel Average latency (includes vSAN overhead)
# GAVG/cmd — Guest Average latency (what the VM sees)
# QUED — Commands queued (saturation indicator)

# CRITICAL RULE:
# GAVG = DAVG + KAVG
# If KAVG >> DAVG → vSAN stack bottleneck (congestion, network)
# If DAVG is high → physical device bottleneck (replace disk)Copy

Virtual Machine disk view — per-VM I/O analysis

# In esxtop, press 'v' for VM storage view
# This shows per-virtual-disk metrics:

# VMDK — virtual disk file
# CMDS/s — IOPS for this specific VMDK
# GAVG/cmd — guest-visible latency for this VMDK
# KAVG/cmd — kernel overhead for this VMDK

# To filter for a specific VM:
# Press 'V' (shift+v) to toggle VM name display
# Look for your VM name in the list

# To capture to CSV for later analysis:
esxtop -b -d 5 -n 120 > /tmp/esxtop-output.csv
# -b = batch mode
# -d 5 = sample every 5 seconds
# -n 120 = 120 samples (10 minutes)
# Import CSV into Excel/Google Sheets for graphingCopy

Network view — vSAN VMkernel throughput

# In esxtop, press 'n' for network view
# Key columns:

# PORT-ID — virtual port identifier
# UPLINK — physical NIC
# MbTX/s — transmit throughput (MB/s)
# MbRX/s — receive throughput (MB/s)
# DRPTX/s — dropped transmit packets (should be 0!)
# DRPRX/s — dropped receive packets (should be 0!)

# Identify the vSAN VMkernel (vmk1 typically):
# Look for vmk1 PORT-ID — check MbTX/s and MbRX/s
# If near NIC speed limit (e.g., 10000 Mb/s for 10GbE) → network saturated

# Dropped packets on vSAN vmk = CRITICAL issue
# Any DRPTX or DRPRX > 0 causes retransmits, adding latency
# Fix: increase ring buffer, check physical switch, enable jumbo framesCopy

Advanced: vSAN-specific counters

# vSAN DOM (Distributed Object Manager) stats:
# In esxtop, some vSAN-specific devices appear as:
#   vSAN — aggregated vSAN datastore metrics
#   Individual component UUIDs

# For deeper vSAN diagnostics beyond esxtop:
# Use vSAN Observer (web-based, built into RVC):
# SSH to vCenter:
rvc root@localhost
# Navigate to cluster:
cd /datacenter/computers/cluster
# Run vSAN observer:
vsan.observer . --run-webserver

# vSAN Observer provides:
#   Per-component latency breakdown
#   DOM client/owner/component stats
#   LSOM stats (disk I/O per device)
#   Network I/O between hosts
#   Access at: https://<vcenter>:8010/observer.htmlCopy

Check write buffer fill level and cache SSD health

# Check cache SSD health and write buffer:
esxcli vsan storage list
# Look for each "Cache Tier" device:
#   "Write Buffer Free %" — should be >30%
#   If <20% — destage is not keeping up

# Check cache SSD wear level:
esxcli vsan storage list | grep -A10 "Cache Tier"
# Look for: "Wear Level" — percentage of write endurance consumed
# If >90% — the SSD is slowing down, replace immediately

# Check SMART data for the cache SSD:
esxcli storage core device smart get -d naa.xxxxxxxxxxxxxxxxxxxx
# Key indicators:
#   Percentage Used Endurance — should be <80%
#   Reallocated Sector Count — should be 0
#   Program Fail Count — should be 0Copy

Identify the destage bottleneck with esxtop

# Run esxtop and press 'd' for disk view:
esxtop

# Compare cache device vs capacity device metrics:
# Cache SSD (naa.xxx — Cache Tier):
#   DAVG should be <0.5ms
#   If DAVG >2ms — cache SSD is degraded (wear/firmware issue)

# Capacity devices (naa.xxx — Capacity Tier):
#   DAVG for HDD: normally 5-15ms, problematic at >20ms
#   DAVG for SSD: normally <1ms, problematic at >3ms
#   %BUSY — if 100% — capacity device is saturated

# The destage rate depends on capacity tier speed:
#   HDD: ~100-200 IOPS destage rate
#   SSD: ~10,000+ IOPS destage rate
# If VM write rate exceeds destage rate, buffer fills upCopy

Immediate relief actions

# Action 1: Move write-intensive VMs to a different disk group or host:
# Identify the heavy writer:
# VM > Monitor > Performance > Virtual Disk > Write IOPS
# Storage vMotion this VM to a less-loaded disk group

# Action 2: If dedup/compression is enabled and causing destage overhead:
# Evaluate if savings justify the performance hit
# Cluster > Configure > vSAN > Services > Deduplication and Compression
# If savings ratio <1.5x — consider disabling (OSA only, requires data evacuation)

# Action 3: Reduce VM swap file overhead:
# VM > Edit Settings > Resources > Memory Reservation = full
# This eliminates the swap file, reducing write I/O to vSAN

# Action 4: Check for runaway writes inside guest OS:
# Windows: Resource Monitor > Disk tab > sort by Write (B/sec)
# Linux: iotop -o (shows processes with active I/O)
# Common culprits: log rotation, backup agents, indexing servicesCopy

Long-term fix: upgrade capacity tier or migrate to ESA

# Option A: Replace HDD capacity with SSD capacity (hybrid → all-flash OSA)
# 1. Add new SSDs to the server
# 2. Create new all-flash disk group
# 3. Migrate objects from HDD disk group to SSD disk group
# 4. Remove old HDD disk group

# Option B: Add more disk groups to distribute destage load
# More disk groups = more parallel destage paths
# Each new disk group needs its own cache SSD

# Option C: Replace worn cache SSD with higher-endurance model
# Use enterprise-grade write-intensive SSDs (3+ DWPD)
# Consumer SSDs (0.3 DWPD) wear out quickly in vSAN cache tier

# Option D: Migrate to ESA (recommended long-term)
# ESA eliminates the cache destage bottleneck entirely
# See KB #43 for OSA → ESA migration workflow

# For new clusters: always choose ESA if hardware qualifiesCopy

Measure cache hit rate

# vSAN Performance Service:
# Cluster > Monitor > vSAN > Performance > Host > Select Host
# Look at "Read Cache Hit Rate" metric:
#   >90% = healthy (most reads served from cache)
#   70-90% = moderate misses (noticeable on HDD capacity)
#   <70% = severe misses (performance degraded)

# From ESXi Shell:
esxcli vsan storage list
# Per-device stats show cache utilization

# esxtop disk view:
esxtop
# Press 'd' — compare:
#   Cache SSD READS/s vs Capacity disk READS/s
#   If capacity disk READS/s is high → cache misses are going to capacityCopy

Identify the working set size vs cache capacity

# Calculate your working set:
# Working set = data actively being read by VMs at any given time
# This is NOT total VMDK size — it's the "hot" data subset

# Rule of thumb for cache sizing:
#   Read cache = 70% of cache SSD capacity
#   E.g., 800GB cache SSD → 560GB read cache
#   If working set > 560GB → cache misses are expected

# Check total VM disk allocation:
# Cluster > VMs tab > sort by "Storage Used"
# If total active VM data significantly exceeds cache size → undersized

# For database VMs: working set ≈ actively queried tables + indexes
# For VDI VMs: working set ≈ (OS + apps) × linked-clone count

# If working set > cache capacity → options:
#   1. Upgrade to larger cache SSD
#   2. Replace HDD capacity with SSD (eliminates the problem)
#   3. Migrate to ESA (eliminates cache concept entirely)Copy

Mitigate cache miss impact

# Action 1: Schedule backups outside business hours
# Backup software reads entire VMDKs, polluting the read cache
# Schedule backups for off-peak when cache misses don't impact users

# Action 2: Use Content-Based Read Cache (CBRC) for VDI
# For Horizon VDI deployments:
# vSphere Client > Host > Configure > Advanced System Settings
# Set: CBRC.Enable = 1
# This adds a RAM-based read cache layer for linked-clone reads

# Action 3: Move random-read-heavy VMs to all-flash disk groups
# If you have mixed disk groups (hybrid + all-flash on same host):
# Storage vMotion the database VM to the all-flash disk group

# Action 4: Upgrade capacity tier to SSD (long-term)
# All-flash OSA: cache miss still hits SSD capacity → still fast (<1ms)
# Hybrid OSA: cache miss hits HDD → slow (5-15ms)
# Replacing HDD with SSD eliminates the cache miss penaltyCopy

Create an IOPS-limited storage policy

# In vSphere Client:
# Menu > Policies and Profiles > VM Storage Policies
# Click "Create"

# Name: "vSAN-RAID5-IOPS-5000"
# Rules:
#   vSAN > Failures to Tolerate: 1
#   vSAN > Failure Tolerance Method: RAID-5
#   vSAN > IOPS limit for object: 5000
#     (this means 5000 normalized 32KB IOPS per VMDK)

# Common limit values:
#   VDI desktop: 200-500 IOPS
#   Dev/test VM: 1000-2000 IOPS
#   General production: 5000-10000 IOPS
#   High-performance DB: No limit (or 20000+ IOPS)Copy

Apply the policy and monitor throttling

# Apply to a VM:
# VM > Configure > Policies > Edit VM Storage Policies
# Select "vSAN-RAID5-IOPS-5000" for the VM's disks
# Click OK

# Monitor if the VM is being throttled:
# VM > Monitor > Performance > Virtual Disk
# Watch "Normalized Latency" — if it suddenly increases when
# IOPS reaches the limit, the policy is throttling

# In vSAN Performance:
# Cluster > Monitor > vSAN > Performance > VM
# Compare VM IOPS before and after policy change
# IOPS should plateau at the limit value

# To see current policy assignments:
# VM > Configure > Policies
# Shows policy name and compliance status for each VMDKCopy

Troubleshoot unexpected throttling

# If a VM is slower than expected, check for accidental IOPS limits:
# VM > Configure > Policies
# Click on the policy name — review "IOPS limit for object"
# If set to a low value, this is why the VM is slow

# Also check if the DEFAULT storage policy has an IOPS limit:
# Menu > Policies and Profiles > VM Storage Policies
# Look at "vSAN Default Storage Policy"
# If it has an IOPS limit, ALL VMs without explicit policy are throttled

# To remove the limit:
# VM > Configure > Policies > Edit
# Change to a policy without IOPS limit
# Or set IOPS limit to 0 (means unlimited)

# Note: Changing the policy triggers a vSAN object reconfiguration
# This is non-disruptive but may take a few minutesCopy

Determine compute requirements

# Calculate total CPU and RAM needed:
# Sum across all VMs:
#   Total vCPU = Σ (VM vCPU count)
#   Total RAM = Σ (VM RAM allocation)

# Apply consolidation ratio:
#   Physical cores needed = Total vCPU ÷ consolidation ratio
#   Consolidation ratio: 3:1 (conservative) to 6:1 (aggressive)
#   Example: 200 vCPUs ÷ 4:1 = 50 physical cores needed

# Per-host calculation:
#   If using dual-socket servers with 24-core CPUs:
#   48 cores per host → 50 ÷ 48 = 2 hosts for compute
#   RAM: Total RAM ÷ Per-host RAM = host count for RAM

# The higher of compute/RAM host count is your minimum
# Then apply N+1 for HA: minimum + 1 = production host countCopy

Determine storage requirements

# Calculate total raw storage needed:
# Sum across all VMs:
#   Total VMDK = Σ (VM disk provisioned size)
#
# Apply overhead multipliers:
#   After FTT:
#     RAID-1 FTT=1: Total VMDK × 2.0
#     RAID-5 FTT=1: Total VMDK × 1.33
#     RAID-6 FTT=2: Total VMDK × 1.5
#
#   After slack space (30% reserve):
#     After FTT ÷ 0.70
#
#   After overhead (2% metadata):
#     ÷ 0.98
#
# Example: 50TB VMs, RAID-5, 30% slack:
#   50 × 1.33 = 66.5TB → ÷ 0.70 = 95TB → ÷ 0.98 = 97TB raw needed
#
# For ESA with dedup (conservative 2x ratio):
#   97TB ÷ 2 = 48.5TB raw NVMe needed

# Per-host: 97TB ÷ 4 hosts = 24.25TB per host
# Select ReadyNode with ≥25TB capacity disks per hostCopy

Select ReadyNode from VMware HCL

# Navigate to VMware Compatibility Guide:
# https://www.vmware.com/resources/compatibility
# Select: vSAN > ReadyNode

# Filter by:
#   Architecture: ESA or OSA
#   Vendor: Dell, HPE, Lenovo, Supermicro, etc.
#   Server Model: based on your procurement agreements
#   Drive Configuration: NVMe count and capacity matching your sizing

# Verify the EXACT configuration:
#   - Storage controller model and firmware version
#   - NVMe/SSD model numbers and firmware versions
#   - NIC model and driver version
#   - Boot device type (M.2, SD, USB — check support status)

# IMPORTANT: The ReadyNode profile defines SPECIFIC disk models
# Substituting a different SSD/NVMe model voids the validation
# Check the ReadyNode profile PDF for exact part numbersCopy

Validate sizing before purchase

# Use the VMware vSAN Sizer Tool (free):
# https://vsansizer.vmware.com
# Input:
#   - VM profiles (count, vCPU, RAM, storage per VM)
#   - Protection level (FTT, RAID type)
#   - Architecture (ESA vs OSA)
#   - Growth projection (months/years)
# Output:
#   - Recommended host count
#   - Disk configuration per host
#   - Matched ReadyNode profiles

# Cross-check sizer output with your manual calculation
# If they differ significantly, review input assumptions

# Final checklist before ordering:
# ✓ Host count meets N+1 (or N+2 for production)
# ✓ ReadyNode profile matches vSAN HCL exactly
# ✓ Network: 25GbE minimum for ESA (10GbE minimum for OSA)
# ✓ Boot device: Check ESXi 8 boot device requirements
# ✓ Spare disks: Order 1-2 spare NVMe/SSD per host for replacementsCopy

Check witness appliance resource utilization

# SSH to the witness appliance:
# Check CPU and memory:
esxtop
# Or from vSphere Client: select the witness host > Monitor > Performance

# Check disk space on the witness:
esxcli storage filesystem list
# The witness vSAN partition should have >20% free

# Check how many objects the witness is serving:
esxcli vsan debug object list | wc -l
# Witness objects are small (metadata only) but each consumes some space

# Check witness vSAN health:
esxcli vsan health cluster list

# Verify witness connectivity from data hosts:
vmkping -I vmk1 <witness-vSAN-IP>
# RTT must be <200ms (500ms max for stretched cluster)Copy

Right-size the witness appliance

# Witness appliance deployment profiles:
#   Tiny:   2 vCPU,  8GB RAM,  15GB + 10GB storage — up to 10 VMs
#   Medium: 2 vCPU, 16GB RAM,  15GB + 350GB storage — up to 500 VMs
#   Large:  2 vCPU, 32GB RAM,  15GB + 1TB storage — up to 2000 VMs
#   X-Large: 4 vCPU, 48GB RAM, 15GB + 2.6TB storage — stretched cluster (many VMs)

# If the witness is undersized:
# 1. Power off the witness appliance
# 2. Edit Settings: increase RAM and disk to match the correct profile
# 3. Power on and verify

# OR: Deploy a new witness appliance with the correct profile
# Then reconfigure the cluster to use the new witness:
# vSphere Client > Cluster > Configure > vSAN > Fault Domains
# Replace witness host reference

# For shared witness (serving multiple clusters):
# Count total VMs across all clusters using this witness
# Size the witness for the total VM count (use Medium or Large)Copy

Clean up witness datastore if full

# SSH to witness appliance:
# Check disk usage:
df -h

# If the vSAN data partition is full:
# Check for stale objects from decommissioned clusters:
esxcli vsan debug object list
# Look for objects belonging to clusters that no longer exist

# In vSphere Client:
# Select Witness Host > Configure > vSAN > Disk Management
# Check disk utilization

# If the witness has stale data from old clusters:
# The cleanest approach is to redeploy the witness:
# 1. Remove the witness from all cluster configurations
# 2. Delete the old witness VM
# 3. Deploy a fresh witness OVA with correct profile
# 4. Re-add to cluster fault domain configurationCopy

Verify TRIM/UNMAP support in your environment

# Check VM hardware version (must be vmx-11+):
# vSphere Client > VM > Summary
# "Compatibility: ESXi 6.0 and later" = vmx-11 (minimum for UNMAP)

# Check virtual SCSI controller type:
# VM > Edit Settings > SCSI Controller
# Must be "VMware Paravirtual" (PVSCSI) or "LSI Logic SAS"
# NOT "LSI Logic Parallel" or "IDE"

# Check VMDK provisioning type:
# VM > Edit Settings > Hard Disk
# "Type: Thin Provision" — UNMAP works
# "Type: Thick Provision Lazy Zeroed" — UNMAP does NOT work
# "Type: Thick Provision Eager Zeroed" — UNMAP does NOT work

# vSAN automatic UNMAP:
# vSAN 6.7U3+ automatically processes UNMAP from guests
# No additional configuration needed on the vSAN sideCopy

Enable TRIM/UNMAP in guest operating systems

# Windows (10/2016+): TRIM is enabled by default
# Verify: Open CMD as Admin:
#   fsutil behavior query DisableDeleteNotify
#   Result: DisableDeleteNotify = 0 (TRIM is ON)
#   If = 1: fsutil behavior set DisableDeleteNotify 0

# Linux: Enable discard in filesystem mount options
# Edit /etc/fstab — add "discard" option:
#   /dev/sda1 /data ext4 defaults,discard 0 0
# OR run manual TRIM:
#   fstrim -av

# For Linux with LVM:
#   Add "issue_discards = 1" to /etc/lvm/lvm.conf

# After enabling TRIM, verify it reaches vSAN:
# vSphere Client > VM > Monitor > Performance > Virtual Disk
# Check "UNMAP" operations counter — should be non-zero after guest deleteCopy

Convert thick-provisioned VMDKs to thin

# Thick VMDKs do not support UNMAP — you must convert to thin:

# Method: Storage vMotion (online, non-disruptive):
# VM > Migrate > Change storage only
# Select the same vSAN datastore
# In disk format, select "Thin Provision"
# This migrates the VMDK from thick to thin on-the-fly

# After conversion, verify:
# VM > Edit Settings > Hard Disk
# "Type" should now show "Thin Provision"

# The conversion will initially consume the same space
# But future guest UNMAP operations will reclaim space

# For newly created VMs:
# Always use thin provisioning — set in VM storage policy:
# Object Space Reservation: 0% (thin)
# ESA: All objects are thin by default — this is a non-issueCopy

Force space reclamation for immediate capacity recovery

# If you need to reclaim space NOW:

# Inside Windows VMs:
# 1. Delete unnecessary files
# 2. Open Disk Cleanup: cleanmgr /d C:
# 3. Run: Optimize-Volume -DriveLetter C -ReTrim -Verbose
#    (PowerShell — forces TRIM on all free space)

# Inside Linux VMs:
# 1. Delete unnecessary files
# 2. Run: fstrim -av
#    Output shows how much space was trimmed per mount point

# Verify capacity recovery on vSAN:
# Cluster > Monitor > vSAN > Capacity
# Watch "Used" decrease after TRIM operations propagate
# Note: May take 5-15 minutes for vSAN to reflect reclaimed space

# For dedup/compression environments (ESA):
# TRIM is especially important — it allows dedup to operate on
# truly free blocks instead of wasting CPU on stale dataCopy

Check per-host component count

# From each ESXi host:
esxcli vsan debug object list | wc -l

# More precise count:
esxcli vsan debug disk list
# "Components" column shows count per disk

# Maximum limits:
# OSA: 9,000 components per host
# ESA: 14,000+ components per host (varies by build)
# Witness: 45,000 witness components per appliance (large)Copy

Reduce component count

# Quick wins:
# 1. Delete stale snapshots (each snapshot = additional components)
# 2. Remove orphaned objects (see KB #72)
# 3. Consolidate small VMs or migrate some to another cluster

# Reduce FTT where appropriate:
# VMs with FTT=2 RAID-1 create 3x components vs FTT=1
# Evaluate if all VMs truly need the higher protection levelCopy

Scale out the cluster

# Adding hosts distributes components more evenly
# After adding a host, vSAN automatically rebalances components
# Monitor rebalance:
esxcli vsan debug resync summary

# For long-term planning:
# Calculate: (Total VMs x avg components per VM) / number of hosts
# Keep each host below 70% of max component countCopy

Check current capacity distribution

# Per-disk usage across the cluster:
esxcli vsan debug disk list
# Compare "Used" percentages — if variance > 30%, rebalance is needed

# vSAN Health check:
# Cluster → Monitor → vSAN → Health → Capacity
# "Disk usage on a vSAN cluster" test shows distributionCopy

Enable proactive rebalance

# From vSphere Client (7.0+):
# Cluster → Configure → vSAN → Services → Proactive Rebalance → Enable

# Or via esxcli on any cluster host:
esxcli vsan debug disk rebalance
# This triggers an immediate one-time rebalance

# To set the automatic threshold (default 30%):
# Advanced settings on cluster → VSAN.ProactiveRebalanceThreshold = 25Copy

Monitor rebalance progress

# Rebalance shows as resync activity:
esxcli vsan debug resync summary

# Track in vSphere Client:
# Cluster → Monitor → vSAN → Resyncing Objects
# Filter by "Rebalance" type

# Rebalance uses the same throttle as repair resync
# To speed up (at cost of VM performance):
# Set VSAN.ResyncThrottleThreshold higher (e.g., 80)Copy

Deploy new witness appliance

# 1. Download the vSAN Witness Appliance OVA (same version as cluster)
# 2. Deploy OVA to a non-vSAN host/cluster at the witness site
# 3. Select appliance size:
#    - Tiny:   10 VMs, 1-5 hosts (2 vCPU, 8 GB RAM)
#    - Medium: up to 39 nodes (2 vCPU, 16 GB RAM)
#    - Large:  up to 64 nodes (2 vCPU, 32 GB RAM)
# 4. Configure management + witness (vSAN) VMkernel networks
# 5. Power on and verify it's accessible:
ping <new-witness-management-ip>Copy

Swap the witness in vSphere Client

# Add the new witness to vCenter inventory as a standalone host:
# Hosts and Clusters → Data Center → Add Host → new witness IP

# Replace witness on the stretched/2-node cluster:
# Cluster → Configure → vSAN → Fault Domains and Stretched Cluster
# Click "Change" on the witness host assignment
# Select the new witness host from inventory
# Click OK

# vSAN will:
# 1. Move witness components from old to new witness
# 2. This happens via resync — monitor progress:
#    Cluster → Monitor → vSAN → Resyncing Objects

# After resync completes, remove old witness from vCenter inventoryCopy

Avoid double encryption

# BAD: VM Encryption + vSAN Encryption on the same VM
# Results in double encryption → double CPU overhead, zero benefit
# vSAN dedup/compression is defeated by VM encryption layer

# RECOMMENDED configurations:
# Option A: vSAN Encryption only (most common)
#   - Cluster → Configure → vSAN → Services → Encryption → ON
#   - All data encrypted at rest, dedup/compression still effective
#   - Single KMS configuration for entire cluster

# Option B: VM Encryption only (selective compliance)
#   - Only sensitive VMs get encrypted via storage policy
#   - Other VMs benefit from dedup/compression on plaintext data
#   - Per-VM key management

# Option C: Both (ONLY if compliance requires it)
#   - Some regulations require both layers (rare)
#   - Accept the performance penalty and loss of dedupCopy