VMware vSAN Knowledge Base

Phase 5 — ESA Deep Dive ⚠ Break Scenarios ✓ Fix Procedures
11
ESA-Specific Issues
5
Break Scenarios
6
Fix Procedures
Phase 5
Current Phase
41
ESA vs OSA — Architecture Differences & Decision Criteria
Understanding the Express Storage Architecture single-tier design vs. the Original Storage Architecture cache+capacity model
High ESA OSA Architecture
🛈 Context: vSAN 8 introduced ESA (Express Storage Architecture) — a completely redesigned storage stack that eliminates the cache tier. All NVMe devices participate in a single flat storage pool. This is a fundamental departure from OSA (Original Storage Architecture) where a dedicated cache SSD fronts capacity devices.
Architecture Comparison
OSA — Original Storage Architecture
  • Disk Groups: 1 cache SSD + up to 7 capacity devices per group
  • Cache Tier: 70% read cache + 30% write buffer
  • Media Types: SSD cache + HDD or SSD capacity
  • RAID: RAID-1 mirroring or RAID-5/6 erasure coding at object level
  • Dedup/Compress: All-flash required, post-process
  • Single Point of Failure: Cache disk loss = entire disk group lost
  • Max disks per host: 5 disk groups × 8 devices = 40
ESA — Express Storage Architecture
  • Storage Pool: All NVMe devices in a single flat pool — no disk groups
  • No Cache Tier: Log-structured object manager handles writes directly
  • Media Types: NVMe only (TLC or QLC NVMe SSDs on HCL)
  • RAID: RAID-5/6 erasure coding at per-object-stripe level — far more efficient
  • Dedup/Compress: Always-on, real-time, inline — no performance penalty
  • No SPOF: Single NVMe loss does NOT lose a disk group — just reduces pool capacity
  • Native Snapshots: Efficient block-level snapshots replace legacy redo logs
Verification Commands
1

Determine current cluster architecture (ESA or OSA)

# From ESXi Shell on any cluster host:
esxcli vsan cluster get
# Look for "vSAN ESA Enabled: true" or "vSAN ESA Enabled: false"

# Check storage pool (ESA) vs. disk groups (OSA):
esxcli vsan storage list
# ESA: Shows "Storage Pool" entries
# OSA: Shows "Disk Group" entries with Cache Tier and Capacity Tier

# In vSphere Client:
# Cluster > Configure > vSAN > Disk Management
# ESA shows "Storage Pool" with all NVMe in flat list
# OSA shows "Disk Groups" with Cache + Capacity hierarchy
2

Verify NVMe devices meet ESA requirements

# List all NVMe devices:
esxcli nvme device list

# Check device model against vSAN HCL:
esxcli nvme device get -A <vmhbaX>
# Note: Model Number, Firmware Revision, Capacity

# Verify NVMe driver version:
esxcli software vib list | grep -i nvme

# ESA minimum requirements:
#   - All storage devices must be NVMe (no SAS/SATA)
#   - Minimum 4 NVMe devices per host (2 minimum for testing only)
#   - ESXi 8.0+ with vSAN 8 license
#   - Devices must be on the vSAN ESA HCL (separate from OSA HCL)
⚠ Key Decision: ESA requires ALL NVMe and ESXi 8.0+. You cannot mix OSA and ESA hosts in the same cluster. Migration from OSA to ESA requires building a new cluster and migrating workloads (no in-place conversion).
42
ESA NVMe Device Failure — Storage Pool Degradation & Recovery
An NVMe device fails in an ESA cluster — understanding impact, automatic recovery, and replacement procedures
Critical ESA NVMe Failure Storage Pool
⚠ Impact: Unlike OSA (where a cache disk failure kills the entire disk group), an ESA NVMe failure only reduces the storage pool capacity on that host. Objects with components on the failed device are rebuilt from redundant copies. However, if multiple devices fail simultaneously, data loss is possible.
Symptoms & Impact Assessment
Symptoms
  • vSAN Health: "Physical disk health — Device error"
  • Storage pool shows reduced capacity on affected host
  • vSphere Client: disk icon shows red warning in Disk Management
  • ESXi vmkernel.log: "NVMe device naa.xxx has been marked as permanently failed"
  • Automatic resync starts immediately — objects rebuild on remaining devices
ESA Advantage Over OSA
  • No disk group loss: Only the failed NVMe is affected — other devices continue operating
  • Faster rebuild: ESA's log-structured engine resyncs at higher throughput
  • RAID-5/6 efficiency: Erasure coding per stripe means less data to reconstruct
  • No cache starvation: No cache tier means no cascading performance impact
  • Hot-plug support: Replace NVMe without host reboot (if hardware supports)
Resolution — Step by Step
1

Identify the failed device

# List storage pool devices and their health:
esxcli vsan storage list

# Look for device with Status: "Error" or "Permanent Loss"
# Note the device NAA ID and slot location

# Get detailed NVMe device info:
esxcli nvme device list
# Cross-reference the failed NAA with physical slot number

# Check SMART health data (if device still responds):
esxcli nvme device log smart get -A <vmhbaX>
# Look for: Percentage Used, Available Spare, Media Errors
2

Monitor automatic resync progress

# Check resync status:
esxcli vsan health cluster list | grep -i "resync"

# In vSphere Client:
# Cluster > Monitor > vSAN > Resyncing Objects
# Shows: Objects to sync, Bytes remaining, ETA

# Monitor resync impact on performance:
# Cluster > Monitor > vSAN > Performance > Cluster
# Watch "Resync Read Latency" and "Resync Write Latency"

# The resync rate can be tuned (but usually not needed for ESA):
# vSphere Client > Cluster > Configure > vSAN > Services
# "Resync throttle" — default is balanced
3

Remove the failed device from the storage pool

# In vSphere Client:
# Host > Configure > vSAN > Disk Management
# Select the failed device (red icon)
# Click "Remove" — choose "Evacuate data" if any components remain

# From ESXi Shell (if UI is not available):
esxcli vsan storage remove -d naa.xxxxxxxxxxxxxxxxxxxx

# Confirm removal:
esxcli vsan storage list
# The failed device should no longer appear
4

Install replacement NVMe and add to storage pool

# Hot-plug the new NVMe device (if server supports hot-plug)
# Or schedule a brief host maintenance for cold-swap

# Verify new device is visible:
esxcli nvme device list
# Confirm new NAA ID appears

# Add to storage pool:
# vSphere Client > Host > Configure > vSAN > Disk Management
# Click "Claim Unused Disks" — select the new NVMe — click "Create"

# From ESXi Shell:
esxcli vsan storage add -d naa.xxxxxxxxxxxxxxxxxxxx

# Verify the device is now in the pool:
esxcli vsan storage list | grep -A3 "naa.xxxxxxxxxxxxxxxxxxxx"
🛈 Best Practice: Monitor NVMe device SMART data proactively. Set vSAN health alerts for "Percentage Used" approaching 90% and "Available Spare" below 10%. Replace devices before they fail using predictive analytics in vSphere Client > Cluster > Monitor > vSAN > Health > Physical Disk Health.
43
OSA to ESA Migration — Complete Workflow & Pre-Checks
Migrating workloads from a vSAN OSA cluster to a vSAN ESA cluster — planning, execution, and validation
Critical Migration OSA → ESA Workflow
⚠ Important: There is NO in-place upgrade from OSA to ESA. You must build a new ESA cluster and migrate workloads via vMotion/Storage vMotion. The new cluster requires all NVMe hardware on the vSAN ESA HCL.
Pre-Migration Assessment
ESA Hardware Requirements
  • All storage devices: NVMe SSDs (SAS/SATA not supported)
  • Minimum 4 NVMe devices per host
  • Minimum 512 GB RAM per host (recommended for ESA efficiency)
  • Server model on vSAN ESA HCL (separate from OSA HCL)
  • ESXi 8.0 Update 1+ required (8.0 U2+ recommended)
  • vSAN 8 Enterprise or Enterprise Plus license
Migration Methods
  • Method A — Cross-vMotion: Both clusters in same vCenter; live vMotion VMs across
  • Method B — Cross-vCenter vMotion: Clusters in different vCenters; use xVC-vMotion
  • Method C — Backup/Restore: For environments that cannot vMotion (use vSAN native snapshots on ESA side)
  • Timeline: Depends on data volume and network speed; plan for 50-100 TB/day on 25GbE
Migration — Step by Step
1

Validate ESA hardware readiness on new hosts

# On each new ESA host:
# Verify ESXi 8.0 U1+:
vmware -vl

# Confirm all NVMe devices are visible:
esxcli nvme device list

# Check device count (minimum 4):
esxcli nvme device list | grep -c "vmhba"

# Verify devices are on ESA HCL:
esxcli nvme device get -A <vmhbaX>
# Note Model Number — cross-check against:
# https://www.vmware.com/resources/compatibility (select vSAN ESA)
2

Build the ESA cluster

# In vSphere Client:
# Right-click Datacenter > New Cluster
# Enable vSAN — select "Express Storage Architecture (ESA)"
# Add the new ESXi 8.0 hosts to this cluster
# Claim NVMe devices into the storage pool:
# Cluster > Configure > vSAN > Disk Management > Claim Disks

# Verify ESA is active:
esxcli vsan cluster get
# Confirm: "vSAN ESA Enabled: true"

# Verify storage pool:
esxcli vsan storage list
# All NVMe devices should appear under "Storage Pool"
3

Migrate VMs from OSA cluster to ESA cluster

# In vSphere Client:
# Right-click VM on OSA cluster > Migrate
# Select "Change both compute resource and storage"
# Target: ESA cluster + ESA vSAN datastore
# Network: map source port groups to destination port groups
# Storage Policy: Select ESA-compatible policy (RAID-5/6 recommended)

# For bulk migration, use vSphere Client multi-select:
# Select multiple VMs > Right-click > Migrate
# Up to 8 concurrent vMotions per host (default)

# Monitor migration progress:
# vSphere Client > Tasks & Events
# Watch for any failures — common issues:
#   - Insufficient bandwidth (saturated vMotion vmk)
#   - Storage policy incompatibility
#   - EVC mode mismatch between clusters
4

Post-migration validation on ESA cluster

# Verify all VMs are running on ESA datastore:
# Cluster > VMs tab — confirm all VMs show ESA vSAN datastore

# Run vSAN health check:
esxcli vsan health cluster list

# Verify storage policies are applied:
# VM > Configure > Policies > check assigned policy matches ESA-recommended

# Validate inline dedup/compression is active:
# Cluster > Monitor > vSAN > Capacity
# "Deduplication and Compression" should show savings ratio

# Confirm native snapshots work (ESA-specific):
# VM > Snapshots > Take Snapshot
# Verify snapshot is created instantly (ESA native, not legacy redo-log)
✓ Post-Migration: After all workloads are migrated, decommission the OSA cluster. Reclaim the hardware or repurpose OSA hosts. Do NOT attempt to join OSA hosts to an ESA cluster — they are architecturally incompatible.
44
ESA RAID-5/6 Erasure Coding — Policy Design & Efficiency
Understanding ESA's per-stripe erasure coding and designing optimal storage policies for capacity and performance
High ESA RAID-5/6 Storage Policy
🛈 Key Difference: In OSA, RAID-5/6 erasure coding operates at the object level across hosts (3+1 or 4+2 stripes across hosts). In ESA, erasure coding operates at the per-object-stripe level — individual data stripes within an object are erasure-coded. This means ESA RAID-5 can work with fewer hosts and provides better capacity efficiency.
ESA Erasure Coding vs OSA
OSA RAID-5/6 (Legacy)
  • RAID-5 (FTT=1): 3 data + 1 parity across 4 hosts — 1.33x overhead
  • RAID-6 (FTT=2): 4 data + 2 parity across 6 hosts — 1.5x overhead
  • Minimum hosts: 4 for RAID-5, 6 for RAID-6
  • Performance impact: write amplification for parity calculation
  • Not recommended for mixed HDD/SSD (too slow for parity rebuild)
ESA RAID-5/6 (Express)
  • RAID-5 (FTT=1): Per-stripe 3+1 — works with 3 hosts minimum
  • RAID-6 (FTT=2): Per-stripe 4+2 — works with 4 hosts minimum
  • Capacity overhead identical but usable with fewer hosts
  • Performance: NVMe speed makes parity overhead negligible
  • ESA RAID-5 is the recommended default for all ESA workloads
Storage Policy Configuration
1

Create an ESA-optimized storage policy

# In vSphere Client:
# Menu > Policies and Profiles > VM Storage Policies
# Click "Create" > Name: "ESA-RAID5-Standard"

# Rules:
#   vSAN > Storage Type: vSAN ESA
#   Failures to Tolerate (FTT): 1
#   Failure Tolerance Method (FTM): RAID-5 (Erasure Coding)
#   Storage Tier: All Flash (auto for ESA)

# Advanced (optional):
#   Object Space Reservation: 0% (thin provision, recommended)
#   IOPS limit: No limit (unless multi-tenant)
#   Disable object checksum: No (keep enabled)
2

Create a high-resilience policy for critical VMs

# For mission-critical VMs, use RAID-6 (FTT=2):
# Policies and Profiles > Create:
#   Name: "ESA-RAID6-Critical"
#   FTT: 2
#   FTM: RAID-6 (Erasure Coding)

# Capacity impact:
#   RAID-5 (FTT=1): 100GB VM uses ~133GB raw (1.33x)
#   RAID-6 (FTT=2): 100GB VM uses ~150GB raw (1.5x)
#   RAID-1 (FTT=1): 100GB VM uses ~200GB raw (2x) — mirroring

# ESA recommendation: Use RAID-5 as default, RAID-6 for critical only
# RAID-1 mirroring is still available but rarely needed in ESA
3

Apply policies and verify compliance

# Assign policy to VM:
# VM > Configure > Policies > Edit VM Storage Policies
# Select "ESA-RAID5-Standard" for all VM disks
# Click OK — vSAN will reconfigure object layout

# Verify policy compliance:
# VM > Monitor > Policies
# Status should show "Compliant" (green)

# If "Non-compliant" — check:
# 1. Enough hosts to satisfy FTT (3 for RAID-5, 4 for RAID-6)
# 2. Enough capacity in the storage pool
# 3. No host in maintenance mode reducing available fault domains

# From ESXi Shell:
esxcli vsan debug object health summary get
# Shows count of compliant vs non-compliant objects
✓ Capacity Tip: ESA RAID-5 + inline dedup/compression typically achieves 3-5x effective capacity compared to raw. A 10TB raw ESA cluster can often store 30-50TB of effective data depending on workload data patterns.
45
ESA Native Snapshots — Failures & Troubleshooting
ESA replaces legacy redo-log snapshots with efficient block-level native snapshots — common issues and resolution
High ESA Native Snapshots Troubleshooting
🛈 ESA Native Snapshots: ESA introduces block-level native snapshots that replace the legacy redo-log mechanism. These snapshots have near-zero performance overhead, support deeper chains, and eliminate the "snapshot stun" problem during consolidation. However, they require ESA-specific configuration and have their own failure modes.
Symptoms & Root Cause
Common Failures
  • Snapshot create fails: "Native snapshot not supported for this VM"
  • Snapshot chain exceeds limit: "Maximum snapshot depth reached"
  • Snapshot delete hangs for hours with no progress
  • VM shows legacy redo-log snapshots on an ESA cluster
  • Backup software fails to quiesce ESA native snapshots
Root Causes
  • VM was migrated from OSA with existing legacy snapshots — ESA inherits them
  • VM hardware version too old — ESA native snapshots require vmx-20+
  • Storage policy does not support native snapshots (check policy config)
  • Backup software using legacy snapshot API instead of VADP ESA-aware API
  • Capacity exhaustion — native snapshots still consume space for changed blocks
Resolution — Step by Step
1

Verify native snapshot support for the VM

# Check VM hardware version:
# vSphere Client > VM > Summary
# "VM Hardware: Version vmx-20" or higher is required

# If older version:
# VM > Actions > Compatibility > Upgrade VM Compatibility
# Select "ESXi 8.0 and later" (vmx-20 or vmx-21)
# Note: VM must be powered off for hardware upgrade

# Verify the VM is on an ESA-compliant storage policy:
# VM > Configure > Policies
# Policy must specify "vSAN ESA" storage type
2

Check for legacy snapshots inherited from OSA migration

# In vSphere Client:
# VM > Snapshots > Manage Snapshots
# If you see delta disks (legacy format), they must be consolidated first:

# VM > Snapshots > Consolidate
# This converts legacy redo-log snapshots to ESA native format

# If consolidation hangs, check:
# 1. Sufficient capacity on ESA datastore
# 2. VM I/O is not preventing quiesce
# 3. No locks on the VMDK files

# Monitor progress:
# vSphere Client > VM > Tasks > "Consolidate virtual machine disk files"
3

Troubleshoot snapshot deletion that is stuck

# Check if a snapshot delete task is running:
# vSphere Client > VM > Tasks & Events
# Look for "Remove Snapshot" or "Consolidate" task

# If stuck, check for file locks on the host where VM runs:
# SSH to ESXi host:
vmkvsitools lsof | grep <vm-name>

# Check vSAN object health for the VM's namespace:
esxcli vsan debug object list | grep -i "<vm-name>"

# If the task has been running for 24+ hours with no progress:
# 1. Power off the VM gracefully (guest OS shutdown)
# 2. Retry snapshot consolidation while VM is off
# 3. Power VM back on after consolidation completes

# Verify all snapshots are cleared:
# VM > Snapshots > Manage Snapshots — should show "You are here" only
4

Configure backup software for ESA native snapshots

# ESA native snapshots require updated backup agents:
# - Veeam: v12.1+ with vSAN 8 ESA awareness
# - Commvault: v2024+ with VADP 8.0 support
# - Dell Avamar/NetWorker: Latest version with ESA support

# Verify backup integration:
# 1. Test a single VM backup + restore on ESA
# 2. Confirm backup uses native snapshot (not legacy):
#    - During backup, check VM > Snapshots
#    - Native snapshots do NOT create delta VMDK files
#    - They appear as lightweight block-level refs

# In vSphere Client:
# VM > Monitor > Events
# During backup, look for "Create snapshot" event
# Type should indicate "vSAN native snapshot"
⚠ Important: ESA native snapshots still consume space for changed blocks. Monitor snapshot space via Cluster > Monitor > vSAN > Capacity. Set alerts for snapshot space exceeding 20% of total capacity. Always delete old snapshots after backup verification.
46
ESA Inline Dedup & Compression — Monitoring & Optimization
ESA enables always-on inline deduplication and compression with zero performance penalty — how to monitor and maximize savings
High ESA Dedup Compression
🛈 ESA Innovation: In OSA, dedup & compression was optional, required all-flash, and was post-process (added latency). In ESA, dedup & compression is always enabled, inline (real-time), and has zero performance overhead thanks to the log-structured object manager and NVMe speed. You cannot disable it — and you don't want to.
How ESA Dedup/Compression Works
ESA Process
  • Inline compression: Every write is compressed before hitting NVMe (LZ4 algorithm)
  • Inline dedup: Block fingerprinting identifies duplicates in real-time
  • No destage penalty: No cache tier means no dedup destage overhead
  • Per-object dedup scope: Dedup operates within each object's block space
  • Savings ratio: Typically 2-5x for general workloads, 5-10x for VDI
OSA Process (for comparison)
  • Dedup/compression was opt-in and required all-flash
  • Post-process: Data written to cache, then deduped during destage to capacity
  • Destage overhead: 10-30% latency increase during heavy writes
  • Cluster-wide dedup scope (across disk groups on same host)
  • Could be disabled if performance impact was unacceptable
Monitoring & Validation
1

Check dedup/compression savings ratio

# In vSphere Client:
# Cluster > Monitor > vSAN > Capacity
# Look for "Deduplication and Compression Overview":
#   - Before Dedup/Compression: XXX TB
#   - After Dedup/Compression: YYY TB
#   - Savings Ratio: X.Xx

# From ESXi Shell:
esxcli vsan debug resync summary get
# Shows "Data Movement" which reflects deduplicated writes

# Per-host breakdown:
# Cluster > Monitor > vSAN > Capacity > "Dedup and Compression" column
# Shows savings per host — helpful to identify imbalanced data
2

Understand capacity reporting with dedup

# IMPORTANT: ESA capacity numbers can be confusing
# Key terms:
#   "Used" — raw bytes consumed on NVMe after dedup/compress
#   "Logical Used" — what VMs think they're using (before dedup)
#   "Savings" — difference between Logical Used and Used

# In vSphere Client:
# Cluster > Monitor > vSAN > Capacity
# "Capacity Overview" shows:
#   Total Raw Capacity → Usable Capacity → Used → Free
#   Usable = Raw minus overhead (metadata, checksums, slack)

# For accurate VM-level capacity:
# VM > Summary > Storage
# "Provisioned" = VMDK file size
# "Used" = actual space on ESA datastore (after dedup/compress)
3

Maximize dedup/compression efficiency

# Workload tips for better dedup ratios:
# 1. Use thin-provisioned VMDKs (default in ESA)
#    VM > Edit Settings > Hard Disk > Type: "Thin Provision"
#
# 2. Zero-fill free space inside guest OS:
#    Windows: sdelete -z C:\
#    Linux: cat /dev/zero > /tmp/zero.fill; rm -f /tmp/zero.fill
#    This allows ESA to dedup zero blocks efficiently
#
# 3. Use the same OS template for VDI/similar VMs
#    ESA dedup will identify common OS blocks across VMs
#
# 4. Avoid pre-encrypted data at the guest level
#    Guest-level encryption (BitLocker/LUKS) prevents dedup
#    Use vSAN encryption instead (encrypts after dedup)

# Monitor dedup efficiency over time:
# Cluster > Monitor > vSAN > Capacity
# Track the savings ratio weekly — it should stabilize after initial data load
✓ Capacity Planning: When sizing ESA clusters, assume a conservative 2x dedup/compression ratio for general workloads. For VDI, use 4x. Never size based on best-case ratios — build headroom for workloads with low dedup potential (databases, media files).
47
ESA Storage Pool Imbalance — Capacity Skew Across Hosts
vSAN ESA storage pool utilization becomes heavily skewed, causing premature capacity warnings and uneven performance
High ESA Rebalance Capacity
⚠ Impact: When storage pool utilization is unevenly distributed, one or more hosts may hit capacity limits while others have ample free space. This triggers premature "cluster capacity critical" alarms, blocks new VM provisioning, and can cause resync failures. Performance degrades on the overloaded hosts.
Symptoms & Root Cause
Symptoms
  • vSAN Health: "Disk balance" shows imbalance across hosts
  • One host at 85% used, others at 40-50%
  • VM provisioning fails: "Insufficient disk space on datastore"
  • vSAN alarm: "vSAN disk balance — Imbalance detected"
  • Higher latency on VMs whose objects land on the overloaded host
Root Causes
  • Hosts have different NVMe device counts or sizes (heterogeneous config)
  • A host was removed and re-added, causing data concentration on remaining hosts
  • Large VM with thick-provisioned VMDK placed predominantly on one host
  • Automatic rebalance threshold not reached (default: 30% variance triggers rebalance)
  • Rebalance process was interrupted (host went to maintenance mode during rebalance)
Resolution — Step by Step
1

Check current balance state

# In vSphere Client:
# Cluster > Monitor > vSAN > Capacity
# Review "Host Capacity Usage" — look for outliers

# Cluster > Monitor > vSAN > Health
# Expand "Data" section > "Disk balance"
# Shows: Variance %, Most used host, Least used host

# From ESXi Shell:
esxcli vsan health cluster list | grep -i "balance"

# Get per-device usage:
esxcli vsan storage list
# Compare "Used Capacity" across all devices on all hosts
2

Trigger manual rebalance if automatic has not kicked in

# vSAN automatically rebalances when variance exceeds 30%
# To trigger a proactive rebalance:

# In vSphere Client:
# Cluster > Monitor > vSAN > Health
# Click on "Disk balance" health check
# Click "Rebalance Disks" button

# This starts a background proactive rebalance operation
# Data moves from over-utilized hosts to under-utilized hosts

# Monitor rebalance progress:
# Cluster > Monitor > vSAN > Resyncing Objects
# Rebalance operations appear as "Rebalance" type resyncs
3

Address root cause — heterogeneous device configuration

# If hosts have different NVMe counts/sizes, consider:
# 1. Add NVMe devices to under-provisioned hosts to match capacity
# 2. Replace smaller NVMe with larger ones during maintenance windows

# Check per-host device count and sizes:
# For each host:
esxcli nvme device list
esxcli vsan storage list | grep "Capacity"

# Ideal ESA config: ALL hosts have identical NVMe device count and size
# Example: 4 hosts × 6 × 3.84TB NVMe each = 92.16TB raw per host

# If adding a new device to an existing host:
# Host > Configure > vSAN > Disk Management > Claim Unused Disks
# vSAN will gradually place new object components on the fresh device
🛈 Best Practice: Use identical hardware across all ESA hosts (same NVMe model, count, and capacity). This ensures even data distribution, predictable performance, and avoids rebalance overhead. If you must mix, keep the variance within 20% of average host capacity.
48
ESA NVMe Firmware Updates — Rolling Upgrade via vLCM
Safely updating NVMe device firmware on ESA hosts using vSphere Lifecycle Manager without data loss
Medium ESA NVMe Firmware vLCM
🛈 Context: NVMe firmware updates are critical for ESA stability. Firmware bugs can cause device drops, PSOD, or data corruption. vLCM (vSphere Lifecycle Manager) with a Hardware Support Manager (HSM) can orchestrate rolling firmware updates across the ESA cluster with zero downtime.
Pre-Update Checks
Requirements
  • vLCM image-based management enabled on the cluster
  • Hardware Support Manager registered for your server vendor (Dell OMIVV, HPE OneView, Lenovo XClarity)
  • Firmware depot accessible from vCenter (online or offline)
  • vSAN cluster healthy — all objects compliant, no active resyncs
  • Sufficient capacity for maintenance mode data evacuation
Risks
  • Firmware update requires host reboot — one host at a time (rolling)
  • During reboot, host's objects are in "Reduced Availability"
  • If firmware update fails, host may not boot — have iLO/iDRAC access ready
  • Some NVMe firmware updates require cold boot (full power cycle)
  • Estimated per-host time: 20-45 minutes (reboot + resync verification)
Update Procedure — Step by Step
1

Check current NVMe firmware versions

# On each ESXi host:
esxcli nvme device list
# Note: Firmware Revision for each device

# Or from vSphere Client:
# Host > Configure > vSAN > Disk Management
# Each NVMe device shows firmware version

# Check vLCM compliance:
# Cluster > Updates > Image > Check Compliance
# Firmware tab shows which hosts need updates
2

Validate vSAN health before starting

# ALL of these must be green before proceeding:
esxcli vsan health cluster list

# In vSphere Client:
# Cluster > Monitor > vSAN > Health
# Critical checks:
#   - "Cluster health — All hosts contributing to cluster"
#   - "Data — vSAN object health"
#   - "Network — All hosts have unicast connectivity"

# Verify no active resyncs:
# Cluster > Monitor > vSAN > Resyncing Objects
# Must show 0 resyncing objects

# Check capacity headroom for maintenance mode evacuation:
# Cluster > Monitor > vSAN > Capacity
# Free space must accommodate one host's worth of data
3

Initiate rolling remediation via vLCM

# In vSphere Client:
# Cluster > Updates > Image > Remediate
# Select all non-compliant hosts
# Settings:
#   - Maintenance Mode: "Ensure data accessibility" (recommended for vSAN)
#   - Rolling: Yes (one host at a time — sequential)
#   - Retry on failure: Yes (1 retry)

# vLCM will for each host:
# 1. Migrate VMs off (DRS)
# 2. Enter maintenance mode (vSAN data migration)
# 3. Apply firmware update
# 4. Reboot host
# 5. Exit maintenance mode
# 6. Wait for vSAN resync to complete
# 7. Move to next host

# Monitor progress:
# Cluster > Updates > Image > "Remediation" tab
# Shows per-host status and current phase
4

Post-update validation

# After all hosts are remediated:

# Verify firmware updated:
esxcli nvme device list
# Compare Firmware Revision with target version

# Run full vSAN health check:
esxcli vsan health cluster list

# Verify all objects are healthy:
esxcli vsan debug object health summary get

# Check NVMe device health (SMART data):
esxcli nvme device log smart get -A <vmhbaX>
# Confirm no new errors post-update

# Verify cluster compliance:
# Cluster > Updates > Image > Check Compliance
# All hosts should show "Compliant"
⚠ Never update all hosts simultaneously. Always use rolling remediation. If a firmware bug is introduced, you want the majority of hosts still on the known-good firmware to maintain data availability. Test firmware on one host first if the update is not from a VMware-validated bundle.
49
ESA Log-Structured Object Manager (LSOM2) — Crash & Recovery
ESA's new log-structured object manager encounters metadata corruption, object I/O failures, or service crashes
Critical ESA LSOM2 Object Manager
⚠ Impact: The ESA LSOM2 (Log-Structured Object Manager v2) is the core storage engine. If it crashes or encounters metadata corruption, all vSAN objects on the affected host become inaccessible. VMs with components on this host will experience I/O errors. This is typically caused by firmware bugs, power loss during writes, or known ESXi issues.
Symptoms & Root Cause
Symptoms
  • VMs on ESA datastore report disk I/O errors — reads/writes fail
  • ESXi vmkernel.log: "LSOM2: metadata corruption detected on device naa.xxx"
  • vSAN Health: "Physical disk health — Metadata health" fails
  • Host-local vSAN service crashes and restarts repeatedly
  • Objects on this host marked as "Absent" in vSAN object health
  • In extreme cases: host PSOD with LSOM2-related stack trace
Root Causes
  • NVMe firmware bug causing silent data corruption on metadata sectors
  • Unexpected power loss during active write operations (no UPS)
  • Known ESXi bug — check VMware patch advisories for LSOM2 fixes
  • Memory error (ECC fault) corrupting in-flight metadata before NVMe write
  • NVMe device approaching end-of-life (wear level critical)
Emergency Recovery — Step by Step
1

Assess the scope of the failure

# Check which objects are affected:
esxcli vsan debug object health summary get
# Note count of: Inaccessible, Reduced Availability, Absent

# Check which device(s) have metadata errors:
esxcli vsan storage list
# Look for devices with "Health: Error" or "Metadata Error"

# Check vmkernel log for LSOM2 errors:
grep -i "LSOM2\|lsom2\|metadata corruption" /var/log/vmkernel.log | tail -50

# Check if vSAN daemon is running:
/etc/init.d/vsand status
# If "not running" — it may be crash-looping
2

Isolate the affected device — do NOT reboot yet

# If a single NVMe device has metadata corruption:
# Remove it from the storage pool to stop further damage:
esxcli vsan storage remove -d naa.xxxxxxxxxxxxxxxxxxxx

# This will mark all components on this device as "Absent"
# vSAN will begin rebuilding those components from replicas on other hosts

# Verify the rebuild starts:
# vSphere Client > Cluster > Monitor > vSAN > Resyncing Objects
# Objects should begin rebuilding automatically

# If vsand is crash-looping, try restarting it after removing the bad device:
/etc/init.d/vsand restart

# If vsand still won't start, the host may need a reboot:
# Only reboot AFTER verifying vSAN cluster has quorum without this host
3

Check for ESXi patches that address LSOM2 bugs

# Check current ESXi build:
vmware -vl

# Review VMware/Broadcom patch advisories:
# https://support.broadcom.com > VMware vSAN
# Search for "LSOM2" or "metadata corruption" in known issues

# Common fixes:
#   ESXi 8.0 U2a: Fixed LSOM2 crash with specific NVMe controller models
#   ESXi 8.0 U3: Fixed metadata corruption during power-loss recovery
#   Check your specific build against the release notes

# If a patch is available:
# Plan to update the ESXi host via vLCM after the immediate crisis is resolved
# Do NOT apply patches during an active outage — stabilize first
4

Replace the failed device and restore full health

# After the corrupted device is removed and resync completes:

# 1. Replace the physical NVMe device (hot-swap or cold-swap)
# 2. Verify new device is visible:
esxcli nvme device list

# 3. Check new device firmware matches the validated version:
esxcli nvme device get -A <vmhbaX>

# 4. Add new device to storage pool:
esxcli vsan storage add -d naa.xxxxxxxxxxxxxxxxxxxx

# 5. Run full health check:
esxcli vsan health cluster list

# 6. Verify all objects are healthy (no absent/degraded):
esxcli vsan debug object health summary get
# Expected: Healthy = 100%, all other counts = 0

# 7. Apply ESXi patch if one is available for the LSOM2 bug
⚠ Prevention: Always use NVMe devices with power-loss protection (PLP). Run ESA clusters on UPS-protected servers. Keep ESXi patched to the latest validated build. Monitor NVMe SMART data for early warning signs (Media Errors, Unsafe Shutdowns counters incrementing).
50
ESA Performance Tuning & Benchmarking Best Practices
Measuring and optimizing vSAN ESA performance — native tools, expected baselines, and common tuning knobs
High ESA Performance Tuning
🛈 ESA Performance Expectations: ESA on NVMe typically delivers: Read latency: 100-200μs, Write latency: 150-300μs, IOPS: 100K-500K+ per host (depending on NVMe count and workload pattern). These numbers significantly exceed OSA due to eliminated cache tier bottleneck and direct NVMe access.
Performance Monitoring — Native Tools
1

Monitor cluster-wide performance via vSphere Client

# In vSphere Client:
# Cluster > Monitor > vSAN > Performance

# Key metrics to watch:
#   - IOPS: Read/Write operations per second (cluster and per-host)
#   - Throughput: MB/s read and write
#   - Latency: Average read/write latency (target: <500μs for ESA)
#   - Congestion: Should be 0 — any value >0 indicates backpressure
#   - Outstanding I/O: Queue depth to vSAN — high values indicate saturation

# Per-VM performance:
# VM > Monitor > Performance > Advanced > Virtual Disk
# Select: Read Latency, Write Latency, IOPS, Throughput

# Enable vSAN Performance Service (if not already):
# Cluster > Configure > vSAN > Services > Performance Service > Enable
2

Check per-device NVMe performance from ESXi Shell

# Real-time NVMe device stats:
esxcli nvme device log smart get -A <vmhbaX>
# Key metrics:
#   Data Units Read / Written — lifetime I/O volume
#   Host Read/Write Commands — lifetime command count
#   Temperature — must be below thermal throttle threshold

# vSAN per-device I/O stats:
esxcli vsan storage list
# Shows: Read Cache Hit Rate (ESA: N/A — no cache tier)
# Shows: Write Buffer Fill % (ESA: handled by LSOM2 log buffer)

# Monitor device queue depth:
esxtop
# Press 'd' for disk view
# Look at QUED (queue depth) and DAVG (device average latency)
# ESA NVMe DAVG should be <200μs for healthy devices
3

Tune vSAN performance parameters

# ESA generally requires LESS tuning than OSA, but key knobs:

# 1. Resync throttle — balance between resync speed and VM I/O:
# Cluster > Configure > vSAN > Services
# Resync throttle: Low / Medium / High
# Use "Low" during business hours, "High" during maintenance windows

# 2. Verify NVMe queue depth is not artificially limited:
esxcli system settings advanced list -o /Disk/QFullSampleSize
esxcli system settings advanced list -o /Disk/QFullThreshold
# Default values are usually correct for NVMe — do not change unless advised

# 3. Check vSAN network adapter ring buffer size:
esxcli network nic ring current get -n vmnic0
# For 25GbE+: RX should be 4096, TX should be 4096
# If lower, increase:
esxcli network nic ring current set -n vmnic0 -r 4096 -t 4096

# 4. Verify Jumbo Frames are enabled on vSAN vmk:
vmkping -I vmk1 -d -s 8972 <other-host-vSAN-IP>
# If fails: MTU not set to 9000 on all components (vmk, VDS, physical switch)
4

Baseline performance testing (non-production only)

# Use VMware's built-in vSAN HCL Benchmark tool (Proactive Tests):
# In vSphere Client:
# Cluster > Monitor > vSAN > Health
# "Performance Tests" section > Run Proactive Tests

# Proactive tests measure:
#   - Sequential Read/Write throughput per host
#   - Random Read/Write IOPS per host
#   - Latency under load

# IMPORTANT: Run proactive tests ONLY during maintenance windows
# They generate significant I/O load

# Expected ESA baselines (4 NVMe per host):
#   Sequential Read: 3-6 GB/s per host
#   Sequential Write: 2-4 GB/s per host
#   Random 4K Read IOPS: 200K-400K per host
#   Random 4K Write IOPS: 100K-200K per host
#   Average Latency: 100-300μs

# If results are significantly below these, investigate:
#   - NVMe firmware version (check for known performance bugs)
#   - Network MTU / bandwidth
#   - Host CPU utilization (LSOM2 is CPU-intensive)
✓ Key Insight: ESA's biggest performance advantage over OSA is the elimination of the write buffer (cache tier) bottleneck. In OSA, a single cache SSD could saturate at 50-100K IOPS. In ESA, writes are distributed across all NVMe devices in the storage pool, providing linear scalability.
77
DOM Owner Unreachable — Objects Inaccessible
DOM owner host is down or network-partitioned, making objects inaccessible even though data components are healthy
Critical DOM Object Owner Accessibility
⚠ Impact: Every vSAN object has a DOM owner host that coordinates I/O. If that host becomes unreachable, the object is inaccessible — even if ALL data components are perfectly healthy on other hosts. VMs freeze until DOM ownership is re-elected (typically 60–140 seconds).
Symptoms & Root Cause
Symptoms
  • VMs pause/freeze for 60+ seconds during host failure
  • vSAN Health: "Objects with DOM owner unreachable"
  • esxcli vsan debug object list shows owner on failed host
  • VM I/O latency spikes to >5000ms before recovery
  • After recovery, some objects remain in "Reduced availability"
Root Causes
  • ESXi host crash (PSOD or hardware failure)
  • Network partition isolating the DOM owner host
  • Host reboot without proper maintenance mode
  • vSAN VMkernel interface failure on the owner host
  • Bug preventing DOM ownership re-election
Diagnostic Commands
1

Identify objects with unreachable owners

# From a healthy ESXi host:
esxcli vsan debug object health summary
# Look for "Objects with unreachable owner" count

# Detailed list:
esxcli vsan debug object list | grep -A5 "DOM Owner"
# Cross-reference owner UUID with host UUIDs:
esxcli vsan cluster get
Fix Procedure
1

Wait for automatic DOM re-election (preferred)

# vSAN automatically re-elects DOM owners when a host becomes unreachable
# Default timeout: 60 seconds for absent components, up to 140 seconds total
# Monitor progress:
watch -n 5 'esxcli vsan debug object health summary'

# Objects should transition from "unreachable owner" back to "Healthy"
# VMs will auto-resume I/O after re-election completes
2

If re-election is stuck — force repair

# From RVC on vCenter:
vsan.obj_status_report /dc/computers/cluster-name --filter-incomplete
# Identify stuck objects

# Force object repair (re-creates missing components):
# vSphere Client → Cluster → Monitor → vSAN → Resyncing Objects
# If no resync is happening for stuck objects, restart CLOMD on the
# host that should be the new owner:
/etc/init.d/clomd restart
🛈 Architecture note: DOM owner is typically the host where the VM is running (client host). You can influence this with vsan.dom.owner.forceWarmCache advanced setting. In vSAN 8, DOM ownership tracking is improved for faster failover.