VMware vSphere Knowledge Base

Phase 6 — Storage & Datastores ⚠ Break Scenarios ✓ Fix Procedures
10
Storage Issues
5
Break Scenarios
5
Fix Procedures
Phase 6
Current Phase
51
VMFS Datastore Unmount / Detach Issues
Can't unmount datastore — VMs registered, active I/O, or HA heartbeat still bound to the volume
High VMFS Unmount Detach Datastore
⚠ Impact: Unable to decommission or migrate storage LUNs. Datastore remains locked across all cluster hosts, preventing storage array maintenance and capacity reclamation.
Symptoms & Root Cause
Symptoms
  • Unmount operation fails with "The resource is in use"
  • vSphere Client shows "Cannot unmount datastore" error
  • Datastore appears greyed out but cannot be removed
  • Storage array reports LUN still actively accessed
  • HA heartbeat datastore icon persists on the volume
Root Causes
  • VMs still registered (even powered off) on the datastore
  • Active I/O from snapshots, swap files, or ISO images mounted
  • vSphere HA using the datastore as a heartbeat volume
  • Datastore not disconnected from all hosts in the cluster
  • Stale VMFS lock held by a disconnected or crashed host
Break — How It Happens

VMFS datastores use on-disk locking (ATS/SCSI reservations) to prevent concurrent access conflicts. When you attempt to unmount, ESXi checks for any active consumers — registered VMs, open VMDKs, swap files, ISO images, HA heartbeats, and SIOC configurations. If any of these exist on any host in the cluster, the unmount is blocked. A common scenario is forgetting to Storage vMotion a powered-off VM's config files or removing HA heartbeat assignment before decommissioning.

Diagnostic Commands
1

Check for active VMs and open files on the datastore

# List all files open on the datastore:
esxcli storage filesystem list | grep -i datastore1

# Check for registered VMs on this datastore:
vim-cmd vmsvc/getallvms | grep "datastore1"

# Check for active VMDK locks:
vmkfstools -D /vmfs/volumes/datastore1/vm/vm.vmdk
2

Verify HA heartbeat and SIOC status

# Check if datastore is an HA heartbeat volume:
# vSphere Client → Cluster → Configure → vSphere Availability
# → Heartbeat Datastores — remove the target datastore

# Check Storage I/O Control:
esxcli storage filesystem list
# Look for "Mounted: true" and usage indicators
3

Unmount and detach from all hosts

# Step 1: Unmount the VMFS filesystem:
esxcli storage filesystem unmount -l datastore1

# Step 2: Detach the underlying device (LUN):
esxcli storage core device set --state=off -d naa.xxxxxxxxxxxx

# Verify unmount succeeded:
esxcli storage filesystem list | grep datastore1
# Should return empty or show "Mounted: false"
🛈 Best Practice: Before unmounting, evacuate all VMs (including powered-off), remove ISO mounts, delete swap file references, and remove the datastore from HA heartbeat selection. Unmount from every host in the cluster before detaching the LUN. Never detach a LUN without unmounting the filesystem first — this can cause metadata corruption.
52
APD / PDL on VMFS Datastores
All Paths Down (APD) or Permanent Device Loss (PDL) — VMs stunned or receiving I/O errors on storage failure
Critical APD PDL VMFS Storage Failure VMCP
⚠ Impact: CRITICAL — VMs are stunned (frozen) during APD or receive fatal I/O errors during PDL. Applications crash, databases corrupt, and production workloads halt. APD default timeout is 140 seconds of total VM freeze.
Symptoms & Root Cause
Symptoms
  • VMs become unresponsive / stunned — no ping, no console
  • vSphere Client shows datastore with red exclamation
  • ESXi logs: ScsiDeviceIO: ... APD Timeout
  • ESXi logs: Permanent device loss ... sense key 0x5
  • HA events: "VMCP has detected APD/PDL on datastore"
Root Causes
  • APD: All storage paths down — fabric switch failure, cable pull, array controller reboot
  • PDL: LUN destroyed, removed from array, or permanently inaccessible
  • Zoning changes removing ESXi host from SAN fabric
  • Storage array firmware upgrade causing temporary path loss
  • iSCSI network partition or NFS server crash
Break — How It Happens

APD (All Paths Down) is a temporary condition — ESXi loses all paths to a storage device but the device has not reported permanent loss. ESXi retries I/O for 140 seconds (APD timeout), during which VMs are stunned (completely frozen). If paths recover within 140s, VMs resume. If not, VMs may be killed or restarted by HA VMCP.

PDL (Permanent Device Loss) is an immediate, non-recoverable condition — the storage array explicitly reports the LUN is gone (SCSI sense code). VMs receive I/O errors immediately and will crash. There is no retry period for PDL.

Diagnostic & Response Commands
1

Identify APD / PDL state on the device

# Check device path state:
esxcli storage core device list | grep -A 5 "naa.xxxx"
# Look for: "Status: dead" (PDL) or "Status: dead-timeout" (APD)

# Check path status for all devices:
esxcli storage core path list | grep -i "state\|device"

# Review vmkernel log for APD/PDL events:
grep -i "APD\|PDL\|permanent device loss\|all paths down" /var/log/vmkernel.log | tail -30
2

Check and configure APD timeout and VMCP

# Check current APD timeout (default 140 seconds):
esxcli system settings advanced list -o /Misc/APDTimeout

# Configure HA VMCP (VM Component Protection):
# vSphere Client → Cluster → Configure → vSphere Availability
# → Failure Conditions → Datastore with APD:
#   Response: "Power off and restart VMs" (aggressive) or "Issue events" (conservative)
# → Datastore with PDL:
#   Response: "Power off and restart VMs" (recommended)

# Verify VMCP is enabled:
esxcli system settings advanced list -o /das/ | grep -i vmcp
3

Recover VMs after APD/PDL event

# If storage has recovered, rescan all HBAs:
esxcli storage core adapter rescan --all

# Verify device is back online:
esxcli storage core device list -d naa.xxxx | grep Status
# Should show: "Status: on"

# If VMs are still stunned, answer the VM question:
# vSphere Client → VM → Monitor → Questions
# Or power cycle the VM:
vim-cmd vmsvc/power.off <vmid>
vim-cmd vmsvc/power.on <vmid>
⚠ Best Practice: Always enable HA VMCP for both APD and PDL responses. For PDL, set response to "Power off and restart VMs" — the LUN is gone, so restarting VMs on surviving datastores is the only option. For APD, consider a conservative 180s delay before restart to allow transient storage issues to self-heal. Test VMCP responses in a non-production cluster first.
53
Multipathing (PSP) Configuration & Troubleshooting
Path Selection Policy tuning — Fixed, MRU, Round Robin for optimal storage performance and failover
High Multipathing PSP Round Robin NMP SATP
⚠ Impact: Wrong PSP can cause suboptimal I/O distribution, path thrashing on active-passive arrays, or single-path bottlenecks. Round Robin misconfiguration can trigger path failovers on ALUA arrays.
Symptoms & Root Cause
Symptoms
  • Storage latency higher than expected despite multiple paths
  • Only one path active, others show standby
  • Path failover events in vmkernel.log during normal operation
  • esxtop shows I/O concentrated on a single HBA/path
  • Array reports path thrashing between controllers
Resolution Path
  • Set Round Robin for active-active arrays (ALUA preferred)
  • Set Fixed for active-passive arrays (single active controller)
  • Use MRU when path persistence matters after failover
  • Tune Round Robin IOPS parameter per workload type
  • Verify SATP claim rules match your storage vendor
Fix Procedure
1

Check current multipathing configuration per device

# List all devices with their current PSP and SATP:
esxcli storage nmp device list

# Check a specific device:
esxcli storage nmp device list -d naa.xxxxxxxxxxxx
# Look for:
#   Path Selection Policy: VMW_PSP_RR (Round Robin)
#   Storage Array Type Plugin: VMW_SATP_ALUA

# View all paths for a device:
esxcli storage nmp path list -d naa.xxxxxxxxxxxx
2

Change PSP to Round Robin for active-active arrays

# Set Round Robin on a specific device:
esxcli storage nmp device set -d naa.xxxxxxxxxxxx --psp=VMW_PSP_RR

# Set Round Robin IOPS (switch path after N I/Os):
esxcli storage nmp psp roundrobin deviceconfig set \
  -d naa.xxxxxxxxxxxx --iops=1 --type=iops
# IOPS=1:  best for latency-sensitive workloads (databases)
# IOPS=1000: best for throughput-heavy workloads (backup, bulk I/O)

# Create a SATP claim rule to auto-apply RR for your array vendor:
esxcli storage nmp satp rule add -s VMW_SATP_ALUA \
  -P VMW_PSP_RR -V "VENDOR" -M "MODEL" \
  -O "iops=1" -e "Custom RR rule for VENDOR array"
3

Verify path distribution and performance

# Verify Round Robin is active and paths are distributing:
esxcli storage nmp path list -d naa.xxxxxxxxxxxx
# All paths should show "State: active"

# Use esxtop to verify I/O distribution across paths:
# esxtop → press 'u' for device view
# Check CMDS/s per path — should be roughly equal with RR

# Rescan to apply changes:
esxcli storage core adapter rescan --all
🛈 Best Practice: Always check your storage vendor's VMware compatibility guide for the recommended PSP and SATP. Create SATP claim rules so new LUNs automatically get the correct PSP. For ALUA arrays, Round Robin with IOPS=1 is the most common recommendation. After changing PSP on one host, replicate the SATP claim rule across all hosts using Host Profiles or PowerCLI.
54
NFS Connectivity & Performance Issues
NFS datastore inaccessible, slow performance, or intermittent disconnects affecting VM availability
High NFS Datastore Network Performance
⚠ Impact: VMs on NFS datastores become unresponsive or experience severe I/O latency. NFS disconnects trigger APD-like behavior, stunning VMs until the NFS server recovers. VMs may crash if the disconnect exceeds the NFS timeout.
Symptoms & Root Cause
Symptoms
  • NFS datastore shows "Inactive" or "Not accessible" in vSphere Client
  • VMs become stunned; console shows frozen screen
  • vmkernel.log: NFS: ... server not responding
  • Slow VM disk I/O — DAVG/cmd > 20ms in esxtop
  • Mount fails: "Unable to connect to NFS server"
Root Causes
  • NFS server overloaded or unresponsive
  • Network misconfiguration — wrong VLAN, missing VMkernel NFS tag
  • Firewall blocking NFS ports (TCP 2049, 111)
  • NFS export permissions deny ESXi host IP
  • Exceeding NFS max volumes limit (256 per host)
Break — How It Happens

ESXi connects to NFS datastores over standard TCP/IP networking. Unlike block storage (FC/iSCSI), NFS has no multipathing built into the protocol — it depends entirely on the network layer. When the NFS server becomes unreachable (network partition, server crash, firewall change), ESXi enters a retry loop. VMs are stunned for the duration of the retry. If the VMkernel port used for NFS traffic is misconfigured (wrong VLAN, no IP, or missing NFS traffic type tag), the mount silently fails or operates over a suboptimal path.

Diagnostic & Fix Commands
1

Verify NFS mount status and network connectivity

# List all NFS datastores and their status:
esxcli storage nfs list
# Check: Share, Host (NFS server IP), Accessible (true/false)

# Test network connectivity to NFS server:
vmkping -I vmk1 192.168.10.50
# Use the VMkernel interface tagged for NFS traffic

# Check for NFS errors in vmkernel log:
grep -i "NFS\|nfs" /var/log/vmkernel.log | tail -30
2

Verify VMkernel NFS traffic tag and NFS server exports

# Check VMkernel adapter services (ensure NFS traffic is tagged):
esxcli network ip interface tag get -i vmk1
# Should include "NFS" in enabled services

# On the NFS server, verify exports allow ESXi host:
# Linux NFS: cat /etc/exports
# Example: /nfs_datastore  192.168.10.0/24(rw,no_root_squash,sync)

# Check current NFS mount count vs. limit:
esxcli storage nfs list | wc -l
# Max NFS mounts per ESXi host: 256 (NFS 3) / 256 (NFS 4.1)
3

Remount NFS datastore and tune timeouts

# Remount an inaccessible NFS datastore:
esxcli storage nfs remove -v NFS_Datastore1
esxcli storage nfs add -H 192.168.10.50 -s /nfs_datastore -v NFS_Datastore1

# Tune NFS heartbeat and timeout settings:
esxcli system settings advanced list -o /NFS/HeartbeatFrequency
# Default: 12 seconds — increase for unstable networks

esxcli system settings advanced set -o /NFS/HeartbeatTimeout -i 5
esxcli system settings advanced set -o /NFS/HeartbeatMaxFailures -i 10
🛈 Best Practice: Dedicate a VMkernel port group with NFS traffic type for storage. Use jumbo frames (MTU 9000) end-to-end for NFS performance. Consider NFS 4.1 for multipathing (session trunking) and Kerberos authentication. Monitor NFS server CPU and network utilization — NFS is CPU-intensive on the server side. Set NFS heartbeat failures to 10+ to tolerate brief network blips without unmounting.
55
VMFS Heap Exhaustion
"Cannot open the disk" or "Failed to lock file" errors due to VMFS heap memory depletion
Critical VMFS Heap Memory VMDK Lock
⚠ Impact: CRITICAL — VMs fail to power on, snapshots cannot be created or deleted, and running VMs may lose disk access. The VMFS heap is a finite kernel memory region; once exhausted, all VMFS operations on the host fail.
Symptoms & Root Cause
Symptoms
  • VMs fail to power on: "Cannot open the disk"
  • Snapshot operations fail: "Failed to lock the file"
  • vmkernel.log: Heap vmfs3 already at its maximum size
  • New VMDKs cannot be created on the datastore
  • Running VMs suddenly lose access to virtual disks
Root Causes
  • Too many open files (VMDKs, snapshots, descriptor files) on VMFS volumes
  • Large number of small VMs on a single VMFS datastore
  • Snapshot chains consuming heap for each delta disk
  • VMFS heap max size insufficient for workload density
  • Orphaned .lck files consuming heap resources
Break — How It Happens

ESXi allocates a fixed kernel memory region (the VMFS heap) to track all open files, locks, and metadata on VMFS datastores. Each open VMDK, snapshot delta, descriptor file, and lock file consumes heap memory. When you pack many VMs (especially with snapshots) onto VMFS datastores, the heap can be exhausted. Once the heap is full, ESXi cannot open any new files on any VMFS datastore — VMs fail to power on, snapshots fail, and even running VMs may lose disk connectivity.

Diagnostic & Fix Commands
1

Check VMFS heap usage and configuration

# Check current VMFS heap size and maximum:
esxcli system settings advanced list -o /VMFS3/MaxHeapSizeMB
# Default: 640 MB (ESXi 7.x), 256 MB (ESXi 6.x)

# Check heap usage via vsish (advanced):
vsish -e get /system/heaps/vmfs3-*/stats
# Look for: "Current heap size" vs "Max heap size"

# Count open files per datastore:
find /vmfs/volumes/datastore1/ -name "*.vmdk" | wc -l
find /vmfs/volumes/datastore1/ -name "*.lck" -type d | wc -l
2

Reduce heap consumption — consolidate and clean up

# Consolidate all VM snapshots (reduces delta disk count):
# vSphere Client → VM → Snapshots → Delete All

# PowerCLI — find VMs with snapshots on the host:
Get-VM -Location (Get-VMHost esxi01.local) |
  Get-Snapshot | Select VM, Name, SizeMB, Created |
  Sort-Object SizeMB -Descending | Format-Table -AutoSize

# Remove orphaned lock files (ONLY if VM is confirmed off):
# find /vmfs/volumes/datastore1/vmname/ -name "*.lck" -type d
# rm -rf /vmfs/volumes/datastore1/vmname/*.lck  # CAUTION!
3

Increase VMFS heap (temporary) and redistribute VMs

# Increase VMFS heap max (requires reboot):
esxcli system settings advanced set -o /VMFS3/MaxHeapSizeMB -i 768

# Long-term: redistribute VMs across more datastores
# Use Storage vMotion to spread VMs:
# PowerCLI:
Get-VM "vm-name" | Move-VM -Datastore "datastore2"

# Monitor heap after changes:
esxcli system settings advanced list -o /VMFS3/MaxHeapSizeMB
⚠ Best Practice: Avoid packing more than 30-40 VMs per VMFS datastore. Consolidate snapshots regularly — each snapshot delta disk consumes heap. Use vSphere 7.x which has a larger default heap (640 MB). Monitor heap usage proactively with vRealize Operations or custom scripts. If you're hitting heap limits, consider migrating to vSAN or NFS which don't have this limitation.
56
Storage vMotion Failure
VM disk migration fails mid-transfer — insufficient space, locked files, snapshot conflicts, or CBT errors
High Storage vMotion Migration VMDK Snapshot CBT
⚠ Impact: VM disk migration fails mid-transfer, leaving partial VMDK copies on the target datastore. The VM remains on the source but may have inconsistent CBT state, causing backup failures. Wasted storage capacity from orphaned partial copies.
Symptoms & Root Cause
Symptoms
  • Storage vMotion task fails at xx% with timeout or I/O error
  • vpxd.log: relocate disk ... failed or No space left on device
  • Orphaned VMDK files on the target datastore
  • VM backup fails after aborted Storage vMotion (CBT corruption)
  • vSphere Client: "A general system error occurred" during migration
Root Causes
  • Insufficient free space on target datastore (need 2x VMDK for swap)
  • Active snapshots creating delta disks that grow during migration
  • CBT (Changed Block Tracking) inconsistency blocking disk copy
  • File lock held by backup software or another process
  • Network or storage I/O timeout during long migration
Break — How It Happens

Storage vMotion copies VMDK data from source to target datastore while the VM is running. It uses a mirror driver that writes to both source and target simultaneously, then switches. If the target datastore runs out of space (thin-to-thick conversion, snapshot growth), the mirror fails. If CBT is corrupted, the incremental copy logic fails. Snapshots add complexity because each delta disk must also be migrated, and active snapshots continue growing during the copy, creating a race condition.

Diagnostic & Fix Commands
1

Pre-flight checks before Storage vMotion

# Check target datastore free space:
esxcli storage filesystem list | grep target_datastore
# Ensure free space > 2x the VMDK size (for thin provisioned)

# Check VM snapshot state — consolidate first:
vim-cmd vmsvc/snapshot.get <vmid>

# If snapshots exist, consolidate before migrating:
vim-cmd vmsvc/snapshot.removeall <vmid>

# Check for file locks on VMDK:
vmkfstools -D /vmfs/volumes/source_ds/vm/vm-flat.vmdk
2

Fix CBT issues and clean up orphaned files

# Reset CBT on the VM (requires stun — brief pause):
# PowerCLI:
$vm = Get-VM "vm-name"
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.ChangeTrackingEnabled = $false
$vm.ExtensionData.ReconfigVM($spec)
# Power cycle or create+delete snapshot to apply
$spec.ChangeTrackingEnabled = $true
$vm.ExtensionData.ReconfigVM($spec)

# Clean up orphaned VMDKs on target (verify not in use first):
# ls -la /vmfs/volumes/target_ds/vm-name/
# Remove only files confirmed as orphaned partial copies
3

Retry Storage vMotion with best practices

# Use PowerCLI for more control over migration:
Get-VM "vm-name" | Move-VM -Datastore "target_ds" `
  -DiskStorageFormat Thin -VMotionPriority High

# For large VMs, consider migrating disks individually:
$vm = Get-VM "vm-name"
$disk = Get-HardDisk -VM $vm -Name "Hard disk 1"
Move-HardDisk -HardDisk $disk -Datastore "target_ds" `
  -StorageFormat Thin

# Monitor migration progress:
Get-Task | Where { $_.Name -eq "RelocateVM_Task" } |
  Select Name, PercentComplete, State
🛈 Best Practice: Always consolidate snapshots before Storage vMotion. Ensure target datastore has at least 2x the VMDK size in free space. Disable backup jobs during migration to avoid CBT conflicts. For very large VMs (>2 TB), schedule migrations during low I/O periods. After failed migrations, always check for and clean up orphaned VMDKs on the target datastore.
57
Thin Provisioning & Space Reclamation (UNMAP)
Thin-provisioned disks grow but never shrink — enabling UNMAP for automatic space reclamation on storage arrays
Medium Thin Provisioning UNMAP Space Reclamation VAAI
⚠ Impact: Storage arrays report datastores as full even after VMs are deleted or data is removed inside guest OS. Without UNMAP, deleted blocks are never returned to the storage array, wasting physical capacity and increasing storage costs.
Symptoms & Root Cause
Symptoms
  • Storage array shows LUN utilization much higher than vSphere reports
  • Thin-provisioned VMDKs never shrink after data deletion in guest
  • Datastore free space doesn't increase after VM deletion
  • Array thin pool approaching capacity alarm despite logical space available
  • Manual UNMAP command shows "VAAI not supported"
Resolution Path
  • Enable automated UNMAP on VMFS 6 datastores (default since vSphere 6.5)
  • Run manual UNMAP for VMFS 5 datastores
  • Verify storage array supports VAAI UNMAP primitive
  • Set UNMAP priority (low/medium/high) based on I/O impact tolerance
  • For guest-level reclamation, use fstrim (Linux) or Optimize-Volume (Windows)
Fix Procedure
1

Check UNMAP support and current status

# Check if VMFS datastore supports UNMAP:
esxcli storage vmfs unmap get -l datastore1
# Look for: "Reclaim Priority" and "Reclaim Granularity"

# Verify VAAI UNMAP support on the device:
esxcli storage core device vaai status get -d naa.xxxxxxxxxxxx
# "Delete Status: supported" = UNMAP is supported

# Check VMFS version (UNMAP auto only on VMFS 6):
esxcli storage filesystem list | grep datastore1
# VMFS 6.x = automated UNMAP supported
2

Enable and configure automated UNMAP (VMFS 6)

# Set automated UNMAP priority on VMFS 6 datastore:
esxcli storage vmfs unmap set -l datastore1 -n low
# Options: none, low (default), medium, high
# "low" = minimal I/O impact, reclaims slowly
# "high" = faster reclaim but more I/O overhead

# Run manual UNMAP on VMFS 5 or for immediate reclaim:
esxcli storage vmfs unmap -l datastore1
# This scans all free blocks and sends UNMAP to the array

# For VMFS 5, use the legacy reclaim command:
esxcli storage vmfs unmap --volume-label=datastore1 \
  --reclaim-unit=200
3

Guest-level space reclamation and verification

# Linux guest — trim deleted blocks (passes UNMAP to VMDK):
sudo fstrim -av

# Windows guest — optimize and trim:
Optimize-Volume -DriveLetter C -ReTrim -Verbose

# Verify space was reclaimed on the array:
# Check storage array management console for LUN utilization
# vSphere: Datastore → Monitor → Capacity
# Array-side thin pool utilization should decrease

# Enable automatic guest TRIM passthrough (VM setting):
# VM → Edit Settings → VM Options → Advanced → Configuration Parameters
# disk.EnableUUID = TRUE (for consistent device naming)
🛈 Best Practice: Use VMFS 6 for all new datastores — automated UNMAP is built-in. Set priority to "low" for production datastores to minimize I/O impact. Schedule manual UNMAP during maintenance windows for VMFS 5. Ensure the full stack supports thin provisioning: guest OS → VMDK (thin) → VMFS → array thin LUN. Monitor array-side thin pool utilization, not just vSphere datastore capacity.
58
iSCSI / FC HBA Troubleshooting
Software iSCSI adapter, FC HBA link issues, CHAP authentication, discovery targets, and zoning errors
High iSCSI FC HBA Zoning CHAP
⚠ Impact: ESXi cannot discover or connect to storage LUNs. New datastores cannot be created, and existing LUNs may become inaccessible after fabric or network changes. VMs on affected datastores will experience APD.
Symptoms & Root Cause
Symptoms
  • No LUNs visible after rescan — "No devices found"
  • iSCSI login fails: CHAP authentication failed
  • FC HBA shows "Link Down" or "Speed: Unknown"
  • Partial LUN visibility — some hosts see LUNs, others don't
  • Storage adapter shows "Offline" in vSphere Client
Resolution Path
  • Verify iSCSI discovery/static targets and CHAP credentials
  • Check FC HBA link status, speed negotiation, and zoning
  • Validate VLAN configuration for iSCSI network
  • Rescan adapters after storage configuration changes
  • Verify LUN masking/mapping on the storage array
Fix Procedure
1

Diagnose iSCSI adapter and connectivity

# List iSCSI adapters:
esxcli iscsi adapter list
# Should show vmhba6x (software iSCSI) as "Online"

# Check discovery targets:
esxcli iscsi adapter discovery sendtarget list -A vmhba65
# Verify target IP and port (default: 3260)

# Add a discovery target:
esxcli iscsi adapter discovery sendtarget add \
  -A vmhba65 -a 192.168.20.100:3260

# Add a static target:
esxcli iscsi adapter target portal add \
  -A vmhba65 -a 192.168.20.100 -n iqn.2024.com.storage:lun01

# Verify CHAP authentication settings:
esxcli iscsi adapter auth chap get -A vmhba65
2

Diagnose FC HBA link status and zoning

# List FC HBAs and their status:
esxcli storage san fc list
# Check: PortState (Online), Speed (16/32 Gbps), NodeWWN, PortWWN

# If link is down, check physical connections:
# - SFP seated properly
# - Fiber cable connected to correct switch port
# - Switch port enabled and in correct VSAN/zone

# Verify FC targets visible to the HBA:
esxcli storage san fc list -A vmhba2
# Target list should show storage array controller WWPNs

# Check zoning on the FC switch:
# Brocade: zoneshow
# Cisco MDS: show zone active
3

Rescan and verify LUN visibility

# Rescan all storage adapters:
esxcli storage core adapter rescan --all

# Rescan a specific adapter:
esxcli storage core adapter rescan -A vmhba65

# Verify LUNs are now visible:
esxcli storage core device list | grep -c "naa\."
# Compare count across all cluster hosts — should be equal

# Check VMFS volumes discovered:
esxcli storage filesystem list

# PowerCLI — rescan all hosts in cluster:
Get-Cluster "ClusterName" | Get-VMHost |
  Get-VMHostStorage -RescanAllHba -RescanVmfs
🛈 Best Practice: Use separate VLANs for iSCSI traffic with jumbo frames (MTU 9000). Configure port binding for software iSCSI to use dedicated VMkernel ports. For FC, always use redundant fabrics (Fabric A + Fabric B) with separate zones. After any SAN fabric change (zoning, LUN masking), rescan all ESXi hosts in the cluster. Document WWPN-to-host mapping for troubleshooting.
59
VAAI Offload & Storage Acceleration
Enabling and verifying VAAI primitives — Atomic Test & Set, Clone Blocks, Zero Blocks, and UNMAP offload
Medium VAAI Offload ATS XCOPY Hardware Acceleration
✓ Benefit: VAAI offloads storage-intensive operations (cloning, zeroing, locking) from ESXi CPU to the storage array firmware. This dramatically reduces VM provisioning time, ESXi CPU usage during storage operations, and network/SAN bandwidth consumption.
Symptoms & Root Cause
Symptoms (VAAI Not Working)
  • VM cloning takes extremely long (copying block-by-block)
  • Thick Eager Zeroed disk creation is very slow
  • VMFS locking issues — SCSI reservation conflicts
  • esxtop shows high KAVG/cmd during clone/zero operations
  • Device VAAI status shows "unsupported" for primitives
VAAI Primitives
  • ATS (Atomic Test & Set): Hardware-assisted VMFS locking — replaces SCSI reservations
  • Clone Blocks (XCOPY): Array-side full copy — no data traverses ESXi
  • Zero Blocks (Write Same): Array-side block zeroing for thick eager provisioning
  • UNMAP (Delete/Trim): Reclaim deleted blocks on thin-provisioned LUNs
Fix Procedure
1

Check VAAI support status per device

# Check VAAI status for a specific device:
esxcli storage core device vaai status get -d naa.xxxxxxxxxxxx
# Expected output:
#   ATS Status: supported
#   Clone Status: supported
#   Zero Status: supported
#   Delete Status: supported

# Check all devices at once:
esxcli storage core device vaai status get
# Any "unsupported" = that primitive is not offloaded

# Verify VAAI is enabled globally:
esxcli system settings advanced list -o /VMFS3/HardwareAcceleratedLocking
esxcli system settings advanced list -o /DataMover/HardwareAcceleratedMove
esxcli system settings advanced list -o /DataMover/HardwareAcceleratedInit
2

Enable VAAI primitives if disabled

# Enable Hardware Accelerated Locking (ATS):
esxcli system settings advanced set \
  -o /VMFS3/HardwareAcceleratedLocking -i 1

# Enable Hardware Accelerated Move (XCOPY/Clone):
esxcli system settings advanced set \
  -o /DataMover/HardwareAcceleratedMove -i 1

# Enable Hardware Accelerated Init (Zero/Write Same):
esxcli system settings advanced set \
  -o /DataMover/HardwareAcceleratedInit -i 1

# These are enabled by default (value=1)
# If set to 0, VAAI was explicitly disabled — investigate why
3

Verify VAAI is being used during operations

# Clone a VM and monitor VAAI usage in vmkernel log:
grep -i "vaai\|xcopy\|ATS\|WriteSame" /var/log/vmkernel.log | tail -20

# During clone, check esxtop for offload indicators:
# esxtop → press 'u' for device view
# APTS/s (ATS operations/sec) > 0 = ATS working
# MBWR/s should be low if XCOPY is offloading the copy

# Test VAAI with a quick thick eager zero disk create:
vmkfstools -c 10G -d eagerzeroedthick \
  /vmfs/volumes/datastore1/test/test.vmdk
# With VAAI: completes in seconds
# Without VAAI: takes minutes (block-by-block zeroing)
🛈 Best Practice: Verify VAAI support before purchasing storage arrays — check the VMware HCL for VAAI primitive support. ATS is the most impactful primitive — it eliminates SCSI reservation storms on VMFS volumes with many VMs. If XCOPY is not supported, cloning will fall back to ESXi-side copy (slow). Some arrays require firmware updates to support all VAAI primitives. NFS VAAI requires the array vendor's NAS plugin (vSphere APIs for Storage Awareness).
60
Datastore Cluster & SDRS Troubleshooting
Storage DRS recommendations not applying — affinity rules, I/O latency thresholds, and SDRS automation issues
Medium SDRS Datastore Cluster Storage DRS Affinity Automation
⚠ Impact: Storage imbalance across datastores in a cluster — some datastores overfull while others have ample space. I/O hotspots develop, causing latency for VMs on overloaded datastores. Manual intervention required for VM placement.
Symptoms & Root Cause
Symptoms
  • SDRS recommendations generated but not applied
  • Datastores in cluster show highly uneven utilization
  • VM placement fails: "No compatible datastore found"
  • SDRS faults in Datastore Cluster → Monitor → Recommendations
  • Storage vMotion never triggered despite imbalance
Resolution Path
  • Set SDRS automation level to Fully Automated
  • Adjust space utilization threshold (default 80%)
  • Configure I/O latency threshold (default 15ms)
  • Review VM anti-affinity rules blocking migrations
  • Enable I/O metric for SDRS if using I/O-based balancing
Fix Procedure
1

Check SDRS configuration and automation level

# vSphere Client → Datastore Cluster → Configure → SDRS
# Verify:
#   SDRS Automation: "Fully Automated" (not "Manual" or "No automation")
#   Space Utilization Threshold: 80% (default)
#   I/O Latency Threshold: 15ms (default)
#   I/O Metric: Enabled (required for I/O-based balancing)

# PowerCLI — check SDRS settings:
Get-DatastoreCluster | Select Name,
  @{N='SDRSAutomation';E={$_.SdrsAutomationLevel}},
  @{N='IOLatencyThreshold';E={$_.IOLatencyThresholdMillisecond}},
  @{N='SpaceThreshold';E={$_.SpaceUtilizationThresholdPercent}},
  @{N='IOMetric';E={$_.IOLoadBalanceEnabled}} |
  Format-Table -AutoSize
2

Check affinity rules and SDRS faults

# Check SDRS VM anti-affinity rules:
# Datastore Cluster → Configure → Rules
# Rules keeping VM disks together or apart may block migrations

# PowerCLI — list SDRS rules:
Get-DatastoreCluster "DS-Cluster-01" |
  Get-SdrsRule | Select Name, Enabled, RuleType, VM |
  Format-Table -AutoSize

# Check SDRS recommendations and faults:
# Datastore Cluster → Monitor → SDRS → Recommendations
# Look for faulted recommendations — hover for details

# Check SDRS history:
# Datastore Cluster → Monitor → SDRS → History
# Verify migrations are happening (or why they're blocked)
3

Tune SDRS thresholds and enable full automation

# PowerCLI — set SDRS to fully automated:
Set-DatastoreCluster "DS-Cluster-01" `
  -SdrsAutomationLevel FullyAutomated

# Adjust space threshold (trigger migration at 75%):
Set-DatastoreCluster "DS-Cluster-01" `
  -SpaceUtilizationThresholdPercent 75

# Enable I/O load balancing:
Set-DatastoreCluster "DS-Cluster-01" `
  -IOLoadBalanceEnabled $true `
  -IOLatencyThresholdMillisecond 10

# Set SDRS invocation schedule (default: every 8 hours):
# Datastore Cluster → Configure → SDRS → Runtime Settings
# Reduce to 4 hours for more responsive balancing
🛈 Best Practice: Use datastore clusters with SDRS for any environment with 4+ datastores. Set automation to "Fully Automated" to avoid manual recommendation approval. Keep VMDK affinity rules (keep VM disks together) unless you have a specific reason to split. Enable I/O metrics for balanced performance, not just capacity. Monitor SDRS recommendations weekly — frequent "faulted" recommendations indicate configuration issues. Don't mix NFS and VMFS datastores in the same datastore cluster.