VMware vSAN Knowledge Base

Capture the PSOD screen

# BEFORE rebooting the host, photograph or record the PSOD screen
# Key information to capture:
#   - Exception type (e.g., "#PF Exception 14" or "#GP Exception 13")
#   - Module name (e.g., "nmlx5_core", "lsi_mr3", "vmkernel")
#   - Stack trace addresses
#   - Build number shown on screen

# This information is critical for VMware/Broadcom GSS to identify the root causeCopy

Reboot the host

# If the host is stuck on PSOD screen:
# Use IPMI/iLO/iDRAC remote console to perform a hard reset
# Or press the physical reset button on the server

# The host will reboot and rejoin the vSAN cluster automatically
# vSAN objects will resync once the host is back onlineCopy

Collect the core dump for root cause analysis

# After reboot, the core dump is saved automatically
# From ESXi Shell:
ls -la /var/core/
# Look for: vmkernel-zdump.* files

# Check the vmkernel log for the crash backtrace:
grep -A 50 "PSOD\|Panic\|Exception" /var/log/vmkernel.log

# Extract the core dump for VMware Support:
# From vSphere Client:
# Host → Monitor → Logs → Export System Logs
# Select "Include core dump files"

# Or via ESXi shell:
vm-support -w /vmfs/volumes/datastore1/
# This creates a support bundle with the core dump includedCopy

Verify the host rejoined the vSAN cluster

# From ESXi Shell (on the recovered host):
esxcli vsan cluster get
# Verify: "Local Node State: AGENT" or "MASTER"
# Verify: "Sub-Cluster Master UUID" matches other hosts

# Check disk groups are mounted:
esxcli vsan storage list
# All disk groups should show "In CMMDS: true"Copy

Monitor resync and object health

# From vSphere Client:
# Cluster → Monitor → vSAN → Resyncing Objects
# Objects should be resyncing back to compliant state

# Cluster → Monitor → vSAN → Health → Run All Checks
# All checks should return to green after resync completesCopy

Identify VMs with snapshots

# From vSphere Client:
# Menu → VMs and Templates → Add column "Needs Consolidation"
# Filter: "Needs Consolidation = Yes"

# Check snapshot depth per VM:
# Right-click VM → Snapshots → Snapshot Manager
# Count the depth of the snapshot chain

# Find large snapshot delta files:
# VM → Files tab → Look for *-delta.vmdk and *-sesparse.vmdk files
# Sort by size to identify the largest snapshotsCopy

Check snapshot disk usage from ESXi Shell

# SSH to the host running the VM:
# Find the VM's directory on the vSAN datastore:
find /vmfs/volumes/vsan*/ -name "*.vmdk" -type f | xargs ls -lh

# Look for delta files:
ls -lh /vmfs/volumes/vsanDatastore/<vm-folder>/*delta*
ls -lh /vmfs/volumes/vsanDatastore/<vm-folder>/*sesparse*

# Each delta file represents a snapshot level
# Large delta files = lots of writes accumulated since snapshot creationCopy

Delete snapshots safely (for chains ≤3 deep)

# From vSphere Client:
# Right-click VM → Snapshots → Snapshot Manager
# Select the OLDEST snapshot → "Delete"
# This merges the snapshot data into its parent

# For all snapshots: Click "Delete All"
# This consolidates the entire chain back to the base disk

# IMPORTANT: Schedule during low-I/O window
# Consolidation causes temporary I/O spikeCopy

Handle deep chains (>3 deep) carefully

# For deep snapshot chains, consolidate ONE snapshot at a time:
# 1. Delete the OLDEST snapshot first (smallest merge)
# 2. Wait for the consolidation task to complete
# 3. Verify VM is responsive
# 4. Delete the next oldest snapshot
# 5. Repeat until chain is clean

# If consolidation fails with "Unable to access file":
# Retry with: Right-click VM → Snapshots → "Consolidate"
# This is a separate operation from "Delete" that retries the mergeCopy

Emergency: VM stunned during consolidation

# If the VM stuns (pauses) during snapshot consolidation:
# DO NOT power off the VM — let the consolidation complete
# The stun duration depends on snapshot delta size and I/O rate

# Monitor progress from ESXi Shell:
tail -f /var/log/vmware/hostd.log | grep -i "consolidat\|snapshot"

# If consolidation is completely stuck (>1 hour no progress):
# 1. Cancel the consolidation task from Recent Tasks
# 2. Power off the VM gracefully (guest shutdown)
# 3. Consolidate while VM is powered off (much faster, no stun risk)
# 4. Power the VM back onCopy

Fix backup software leaving orphaned snapshots

# Check backup software logs for snapshot removal failures
# Common fixes:
# - Verify backup proxy has permissions to remove snapshots
# - Ensure CBT (Changed Block Tracking) is enabled on the VM:
#   VM → Edit Settings → VM Options → Advanced → Edit Configuration
#   Check: "ctkEnabled = true"
# - Reset CBT if corrupted:
#   1. Power off VM
#   2. Edit Settings → Remove all *-ctk.vmdk files
#   3. Set ctkEnabled = false, save, then set to true, save
#   4. Power on VMCopy

Verify quorum on the preferred site

# From any host on the PREFERRED site:
esxcli vsan cluster get
# Verify: "Sub-Cluster Member Count" shows preferred site hosts + witness
# Verify: "Local Node State: MASTER" or "AGENT"

# Check object accessibility:
# Cluster → Monitor → vSAN → Virtual Objects
# Objects should show "Degraded - Reduced Availability" (NOT "Inaccessible")Copy

Verify HA restarted VMs

# From vSphere Client:
# Cluster → Monitor → vSphere HA → Summary
# Check "VM Restarts" — all VMs from the failed site should be restarted

# If VMs did not restart:
# Cluster → Configure → vSphere HA → Edit
# Verify: "Host Monitoring Status: Enabled"
# Verify: "VM Restart Priority: Medium or High"Copy

Wait for vSAN to rebuild (or not)

# vSAN will wait 60 minutes before rebuilding components
# from the failed site onto the surviving site
# This is controlled by the CLOM repair timer

# If the site is expected to return soon, do NOT force rebuild
# Objects remain accessible (degraded) from the surviving site

# If the site is permanently lost:
# Cluster → Monitor → vSAN → Resyncing Objects → "Repair Objects Immediately"
# This starts rebuilding all components onto the surviving siteCopy

LAST RESORT — Force quorum on the surviving site

# This should ONLY be done when:
# - The other site AND witness are CONFIRMED permanently unavailable
# - You accept potential data inconsistency risk
# - VMware/Broadcom GSS has been contacted

# From vSphere Client (if vCenter is on the surviving site):
# Cluster → Configure → vSAN → Fault Domains and Stretched Cluster
# Click "Preferred Fault Domain" → verify the surviving site is preferred

# From ESXi Shell on the surviving site (if vCenter is down):
esxcli vsan cluster unicastagent add -a <local_host_ip> -p 12321
# Force the local hosts to form a cluster without the witnessCopy

When the failed site comes back online

# Once the failed site's hosts boot up and rejoin the cluster:
# vSAN will automatically resync stale components
# Monitor: Cluster → Monitor → vSAN → Resyncing Objects

# If the witness was also replaced:
# Cluster → Configure → vSAN → Fault Domains → Replace Witness Host
# Select the new witness appliance

# Verify full cluster health:
# Cluster → Monitor → vSAN → Health → Run All Checks
# All checks should return to green after full resyncCopy

Verify ESA mode and native snapshot support

# From vSphere Client:
# Cluster → Configure → vSAN → Services
# Confirm: "Storage Architecture: Express Storage Architecture (ESA)"
# Confirm: "Data Protection: Enabled" (or enable it)

# Native snapshots are NOT available on OSA clusters
# OSA clusters continue to use traditional VMware snapshotsCopy

Create and configure a protection group

# From vSphere Client:
# Cluster → Configure → vSAN → Data Protection → Protection Groups
# Click "Create Protection Group"
#   - Name: "Production-Hourly" (example)
#   - Schedule: Every 1 hour
#   - Retention: Keep last 24 snapshots
#   - Assign VMs: Select VMs to protect

# Verify protection group status:
# Cluster → Monitor → vSAN → Data Protection
# Status should show "Compliant" for all assigned VMsCopy

Fix snapshot creation failures

# If "Insufficient space":
# Check capacity: Cluster → Monitor → vSAN → Capacity
# Native snapshots consume space proportional to the CHANGE RATE
# High-write VMs consume more snapshot space

# Reduce retention or increase cluster capacity:
# Edit Protection Group → Reduce "Retention Count"
# Or add capacity per KB #17 (Phase 2)

# If protection group is "Non-compliant":
# Cluster → Monitor → vSAN → Health → Run All Checks
# Fix any underlying vSAN health issues first
# Then: Protection Group → "Reapply" to re-evaluate complianceCopy

Restore a VM from a native snapshot

# From vSphere Client:
# Cluster → Monitor → vSAN → Data Protection → Select VM
# Browse available snapshots by timestamp
# Select the desired recovery point → "Restore"

# Options:
#   - Restore in-place (overwrites current VM state)
#   - Restore as new VM (creates a clone from the snapshot point)

# Native snapshot restores are near-instant (metadata-level operation)
# The VM resumes from the exact point-in-time of the snapshotCopy

Check for checksum errors in health dashboard

# From vSphere Client:
# Cluster → Monitor → vSAN → Health → Data
# Look for checksum error counts

# Review vmkernel log for checksum events:
grep -i "checksum\|csum\|integrity" /var/log/vmkernel.log | tail -50Copy

Identify the affected disk

# From ESXi Shell:
esxcli vsan debug disk list
# Look for disks with non-zero "Checksum Errors" or "Read Errors"

# Check SMART data for the suspected disk:
esxcli storage core device smart get -d naa.xxxxxxxxxxxx
# Key indicators: "Uncorrectable Error Count", "Reallocated Sector Count"Copy

For isolated checksum errors — vSAN auto-heals

# If checksum errors are infrequent (a few per week):
# vSAN has already auto-healed these from redundant copies
# No action needed — monitor the trend

# Verify auto-heal occurred:
# Cluster → Monitor → vSAN → Health → Data
# "Checksum errors corrected" count should match "detected" countCopy

For persistent/increasing errors — replace the disk

# If a specific disk shows increasing checksum errors:
# The physical media is degrading — proactive replacement needed

# From vSphere Client:
# Host → Configure → vSAN → Disk Management
# Select the failing disk → "Remove Disk with Data Migration"
# Wait for full data evacuation
# Physically replace the disk
# Add the new disk: Disk Management → Claim Disks

# Or if the entire disk group is affected (cache SSD):
# Remove the disk group with full data migration
# Replace the SSD → Recreate the disk groupCopy

Verify and configure proactive scrubbing

# Proactive scrubbing is enabled by default but can be verified:
# From ESXi Shell:
esxcli system settings advanced list -o /VSAN/ObjectScrubEnabled
# Expected: "Int Value: 1" (enabled)

# Scrub frequency (default: every 365 days for full scrub):
esxcli system settings advanced list -o /VSAN/ObjectScrubInterval

# To make scrubbing more frequent (e.g., every 90 days):
esxcli system settings advanced set -o /VSAN/ObjectScrubInterval -i 90

# From vSphere Client:
# Cluster → Configure → vSAN → Services → Advanced Options
# "Object Repair Timer" — keep at default unless advised by GSSCopy

Verify vSAN is still operational via ESXi direct management

# Connect directly to each ESXi host via SSH or Host Client (https://host-ip/ui)
# From ESXi Shell:
esxcli vsan cluster get
# Should show cluster membership is intact

esxcli vsan storage list
# All disk groups should be mounted

esxcli vsan debug resync summary
# Verify no unexpected resyncsCopy

Recover VCSA from file-based backup

# If you have VCSA file-based backups configured:
# 1. Deploy a fresh VCSA from the ISO installer
# 2. During deployment, choose "Restore" instead of "Install"
# 3. Point to the backup location (FTP/FTPS/HTTP/SCP/NFS/SMB)
# 4. The restore process recovers the full vCenter database and config

# VCSA backup location check (from a working VCSA):
# VAMI (https://vcsa:5480) → Backup → Backup Schedule
# Backup protocol and location are shown here

# IMPORTANT: Deploy the restored VCSA on a host's local datastore FIRST
# Then Storage vMotion it to vSAN after vCenter is operationalCopy

If no backup exists — fresh VCSA and re-register hosts

# Deploy a new VCSA on a host's local datastore
# Create a new datacenter and cluster in vCenter
# Add hosts to the cluster:
#   Cluster → Add Host → Enter each host's IP/FQDN and root credentials

# CRITICAL: When adding hosts, vSAN configuration is PRESERVED on the hosts
# vCenter will discover the existing vSAN cluster configuration
# VMs and data remain intact on the vSAN datastore

# After all hosts are added:
# Cluster → Configure → vSAN → verify the existing config is detected
# Run health checks to confirm everything is operationalCopy

Restart vCenter VM from ESXi Host Client (if it just crashed)

# If the VCSA VM is on vSAN but just powered off/crashed:
# Connect to the host managing it: https://<esxi-host-ip>/ui
# Log in with root credentials
# Find the VCSA VM → Power On

# If the VCSA VM is not registered on any host:
# Browse the vSAN datastore from Host Client:
# Storage → vsanDatastore → Browse → Find VCSA folder → .vmx file
# Right-click .vmx → Register VM → Power OnCopy

Configure Host Isolation Response

# From vSphere Client:
# Cluster → Configure → vSphere HA → Edit

# VM Monitoring tab:
# "Host Isolation Response" → Set to "Disabled"
# Reason: On vSAN, the isolated host's VMs still have LOCAL
# access to vSAN storage. Powering them off is unnecessary.

# The VMs will continue running on the isolated host.
# When the management network recovers, vCenter reconnects
# and no VM disruption occurred.Copy

Configure APD and PDL responses for vSAN

# In the same HA settings:
# "Datastore with APD" (All Paths Down):
#   Response: "Issue events" (conservative) or "Power off and restart VMs (aggressive)"
#   APD Timeout: 140 seconds (default — do not reduce)

# "Datastore with PDL" (Permanent Device Loss):
#   Response: "Power off and restart VMs"
#   Reason: PDL means the datastore is PERMANENTLY gone — restart elsewhere

# For vSAN specifically:
# APD occurs when the host loses ALL vSAN network connectivity
# PDL is rare on vSAN — it means the object is permanently inaccessibleCopy

Configure isolation addresses

# Add a reliable isolation address that's always reachable:
# Cluster → Configure → vSphere HA → Advanced Options → Add:

# das.isolationaddress0 = <default_gateway_ip>
# das.isolationaddress1 = <dns_server_ip>
# das.usedefaultisolationaddress = false

# These IPs are used to determine if a host is truly isolated
# or if just the vCenter communication is disrupted
# Use infrastructure IPs that are ALWAYS UP (gateway, DNS, NTP)Copy

Recover VMs if they were incorrectly powered off

# If VMs were powered off due to incorrect isolation response:
# 1. Fix the management network issue first
# 2. vCenter reconnects to the host automatically
# 3. Power on the VMs that were affected:
#    Select VMs → Right-click → Power → Power On

# Prevent recurrence:
# Set "Host Isolation Response" to "Disabled" (step 1 above)
# Configure proper isolation addresses (step 3 above)Copy

Install the hardware health provider

# Each server vendor provides a vCenter plugin:
#   Dell: Dell OpenManage Integration for VMware vCenter (OMIVV)
#   HPE: HPE OneView for VMware vCenter
#   Lenovo: Lenovo XClarity Integrator for VMware vCenter

# Install the vendor plugin via vCenter:
# Menu → Administration → Client Plugins → Add plugin
# Follow the vendor's deployment guide

# Verify the health provider is registered:
# Host → Monitor → Hardware Health
# Should show component-level health (CPU, Memory, Storage, Power, Fans)Copy

Enable Proactive HA on the cluster

# From vSphere Client:
# Cluster → Configure → vSphere Availability → Proactive HA → Edit

# Settings:
#   Proactive HA: Enabled
#   Remediation: "Mixed Mode" or "Maintenance Mode"
#     - Mixed Mode: Migrates VMs (DRS) for moderate issues,
#       enters maintenance for severe issues
#     - Maintenance Mode: Always evacuates all VMs
#   Automation Level: "Automated" (recommended)
#     - VMs are vMotioned automatically without admin intervention

# Select the health provider(s) to use:
# Check the vendor plugin(s) installed in step 1Copy

Configure failure conditions and severity mapping

# Proactive HA maps vendor health alerts to two severity levels:

# "Moderately Degraded": Non-critical component issue
#   Example: Single PSU failed (redundant PSU still running)
#   Action: DRS migrates VMs to healthier hosts (preventive)

# "Severely Degraded": Critical component about to fail
#   Example: Memory DIMM correctable errors exceeding threshold
#   Action: Host enters maintenance mode, all VMs evacuated

# Fine-tune per provider:
# Cluster → Configure → Proactive HA → Provider-specific settings
# Map vendor alert categories to Moderate or SevereCopy

If Proactive HA is not triggering on hardware issues

# Verify the health provider is communicating:
# Host → Monitor → Hardware Health → Check for recent updates

# Check vCenter logs for Proactive HA events:
# Menu → Events → Filter by "Proactive HA"

# Common issues:
# - Vendor plugin lost connection to the vendor management server
# - Plugin version incompatible with current vCenter version
# - IPMI/iLO/iDRAC credentials changed — plugin cannot poll hardware

# Fix: Re-configure the vendor plugin with correct credentials
# and verify connectivity to the management controller (iLO/iDRAC/IMM)Copy

Step 1: Shut down all non-essential VMs

# From vSphere Client:
# Shut down all workload VMs gracefully:
# Select VMs → Right-click → Power → Shut Down Guest OS

# Leave these VMs running for now:
#   - vCenter Server (VCSA)
#   - Witness appliance (if stretched cluster)
#   - Any infrastructure VMs needed for vCenter operation (AD, DNS)Copy

Step 2: Shut down vCenter Server

# From VCSA VAMI (https://vcsa-ip:5480):
# Actions → Shutdown → Confirm

# Or from VCSA SSH:
shutdown -h now

# IMPORTANT: After this point, you manage hosts directly via
# ESXi Host Client (https://host-ip/ui) or SSHCopy

Step 3: Put each host in maintenance mode (No data migration)

# For a full cluster shutdown, use "No data migration"
# because ALL hosts are shutting down — there's nowhere to migrate TO

# From ESXi Shell on each host:
esxcli system maintenanceMode set -e true -m noAction

# Or from ESXi Host Client:
# Host → Actions → Enter Maintenance Mode
# Select "No data migration" (since entire cluster is shutting down)

# Wait for each host to enter maintenance mode before proceedingCopy

Step 4: Shut down ESXi hosts

# From ESXi Shell on each host:
esxcli system shutdown poweroff -r "Datacenter maintenance"

# Or from ESXi Host Client:
# Host → Actions → Shut Down

# Shut down hosts in any order (all should be in maintenance mode already)
# Last: shut down the witness host (if stretched cluster)Copy

Step 1: Power on ALL ESXi hosts (and witness)

# Power on all hosts simultaneously (or as quickly as possible)
# Use IPMI/iLO/iDRAC to remote power-on

# CRITICAL: Power on ALL hosts before exiting maintenance mode on ANY host
# This ensures vSAN can form the complete cluster and all objects are accessible

# Wait for all hosts to finish booting:
# Connect via SSH or Host Client to verify
# Each host should show ESXi login promptCopy

Step 2: Verify vSAN cluster formed

# From ESXi Shell (on any host):
esxcli vsan cluster get
# Verify: All hosts show the SAME "Sub-Cluster Master UUID"
# Verify: "Sub-Cluster Member Count" matches total host count

# Check disk groups are mounted:
esxcli vsan storage list
# All disk groups should show "In CMMDS: true"Copy

Step 3: Exit maintenance mode on all hosts

# From ESXi Shell on each host:
esxcli system maintenanceMode set -e false

# Or from ESXi Host Client on each host:
# Host → Actions → Exit Maintenance Mode

# vSAN will verify object health and start any necessary resyncs
# Wait for resync to complete (should be minimal after graceful shutdown)Copy

Step 4: Power on vCenter, then workload VMs

# Power on vCenter VCSA first:
# From ESXi Host Client (on the host where VCSA is registered):
# Find VCSA VM → Power On → Wait for services to start (5-10 minutes)

# Verify vCenter is operational:
# Browse to https://vcsa-ip/ui — login should work

# Then power on workload VMs:
# From vSphere Client: Select VMs → Power On
# Or use VM startup order if configured:
# Host → Configure → Virtual Machines → VM Startup/ShutdownCopy

Audit current fault domain configuration

# From vSphere Client:
# Cluster → Configure → vSAN → Fault Domains
# Review which hosts are in which fault domain
# Cross-reference with your physical rack layout

# If "Fault Domains" shows empty — fault domains are NOT configured
# vSAN is treating each individual host as a fault domainCopy

Create fault domains matching physical topology

# From vSphere Client:
# Cluster → Configure → vSAN → Fault Domains
# Click "+" to create a new fault domain
# Name it after the physical rack: "Rack-A", "Rack-B", "Rack-C"
# Drag hosts from "Unassigned" into the correct fault domain

# Example for a 6-host, 3-rack cluster:
#   Rack-A: esxi-01, esxi-02
#   Rack-B: esxi-03, esxi-04
#   Rack-C: esxi-05, esxi-06

# After configuration, vSAN's CLOM engine will ensure replicas
# are placed across DIFFERENT fault domainsCopy

Trigger rebalance to fix existing object placement

# After creating fault domains, existing objects may not comply
# vSAN does NOT automatically rebalance existing objects

# To fix non-compliant objects:
# Cluster → Monitor → vSAN → Virtual Objects
# Filter by "Compliance Status: Non-Compliant"

# For each non-compliant VM:
# Right-click → VM Policies → Reapply VM Storage Policy
# This triggers CLOM to re-evaluate placement with fault domain awareness

# Or trigger cluster-wide rebalance:
# Cluster → Monitor → vSAN → Resyncing Objects → "Rebalance Disks"Copy

Verify fault domain compliance

# After rebalance completes:
# Cluster → Monitor → vSAN → Virtual Objects
# All objects should show "Compliant"

# Verify replicas are spread across fault domains:
# Select any VM object → "Physical Disk Placement" tab
# Component replicas should be on hosts in DIFFERENT fault domains

# Run health check:
# Cluster → Monitor → vSAN → Health → Run All
# "Cluster – Fault domain configuration" should be greenCopy

Replace boot device and reinstall ESXi

# If USB/SD failed completely:
# 1. Replace with a new enterprise SSD/NVMe (recommended) or M.2 boot device
# 2. Install ESXi fresh from ISO
# 3. Apply the SAME build/patch level as other cluster hosts

# After install, reconfigure:
esxcli network vswitch standard list
esxcli vsan storage tag add --tag capacityFlash -d naa.xxx  # if neededCopy

Restore host configuration from backup

# If you backed up host config (best practice):
vim-cmd hostsvc/firmware/restore_config /tmp/configBundle-hostname.tgz

# If no backup exists, reconfigure manually:
# - Set management VMkernel IP
# - Tag vSAN VMkernel
# - Configure NTP, DNS, syslog
# - Add host back to vCenter and clusterCopy

Reclaim vSAN disk groups

# After host rejoins the cluster, existing vSAN disks should be auto-detected
# If not, manually re-add the disk group:
esxcli vsan storage list
# Verify existing disks show with vSAN partition

# If disks show but aren't claimed:
esxcli vsan storage add -s naa.CACHE_SSD -d naa.CAPACITY_1 -d naa.CAPACITY_2Copy

Reset CBT for affected VMs

# Power off the VM (or use hot-add if supported):
# Edit VM Settings → Options → Advanced → Configuration Parameters
# Set: ctkEnabled = FALSE
# Remove all ctk files from the VM's folder
# Re-enable: ctkEnabled = TRUE

# PowerCLI bulk reset for all VMs:
Get-VM | ForEach-Object {
    $snap = New-Snapshot -VM $_ -Name "CBT-Reset" -Memory:$false -Quiesce:$false
    Remove-Snapshot -Snapshot $snap -Confirm:$false
}
# This forces CBT to recalculate from scratchCopy

Limit concurrent backup snapshots

# In your backup software (Veeam, Commvault, etc.):
# Reduce concurrent tasks per host to 2-3 (not 5+)
# vSAN must create/commit snapshots which involves I/O

# Set backup window during low-I/O periods
# Avoid running backups during vSAN resync operations

# For Veeam specifically:
# Set "Limit maximum concurrent snapshots per datastore" = 2Copy

Use vSAN 8 native snapshots (ESA) for better backup performance

# ESA native snapshots don't have the performance penalty of OSA redo logs
# If on ESA, configure backup software to use native snapshot API
# This eliminates most snapshot-related backup failures

# Verify native snapshots are active:
esxcli vsan debug object list --filter-type snapshotCopy

Witness connectivity loss

# 2-node clusters REQUIRE the witness for quorum
# If witness is unreachable:
esxcli vsan cluster get
# Check "Sub-Cluster Member Count" — should be 3 (2 data + 1 witness)

# Common causes:
# - Witness VM powered off or suspended
# - Firewall blocking witness ↔ data host traffic (ports 2233, 12321)
# - Witness on wrong network (must reach both data hosts)

# Fix: ensure witness VMkernel can ping both data hosts:
vmkping -I vmk0 <witness-vsan-ip>Copy

Maintenance mode in 2-node cluster

# ALWAYS use "Ensure accessibility" mode:
esxcli system maintenanceMode set -e true -m ensureObjectAccessibility

# "Full data migration" is IMPOSSIBLE — there's only 1 other host
# and it already holds its own data + needs capacity headroom

# Before maintenance:
# 1. Verify witness is healthy
# 2. Check no objects are already degraded
# 3. Reduce FTT=0 for non-critical VMs temporarily (optional)

# During maintenance:
# All objects on the maintenance host go to "Reduced availability"
# If the remaining host fails during this window → DATA LOSSCopy

Configure vSphere Replication for vSAN-to-vSAN DR

# 1. Deploy vSphere Replication Appliance at both sites
# 2. Pair the vCenters in SRM
# 3. Configure replication per VM:
#    - Right-click VM → vSphere Replication → Configure Replication
#    - Target: vSAN datastore at recovery site
#    - RPO: 5 minutes (minimum) to 24 hours

# Key vSAN-specific settings:
# - Use vSAN storage policy at target for FTT protection
# - Network: replication traffic can use vSAN VMkernel or dedicated vmk
# - Compression: enable replication network compression for WAN linksCopy

Troubleshoot replication failures

# Common issues:
# 1. RPO violations — replication can't keep up
#    Fix: increase RPO, reduce VM change rate, increase bandwidth
#    Check: vSphere Replication → Monitor → Outgoing Replications

# 2. "Replication target datastore full"
#    vSAN target needs space for: VM + snapshots + replication overhead
#    Rule of thumb: 1.5x the VM size on target vSAN datastore

# 3. SRM test failover leaves placeholder VMs
#    Always use "Cleanup" after test failover in SRM recovery plan
#    Orphaned test VMs consume vSAN capacity at recovery site

# 4. Network timeout during initial sync (large VMs)
#    Use "Replication Seeds" — copy VMDK to target offline
#    SRM syncs only delta after seedingCopy