VMware vSAN Knowledge Base

Verify VMkernel adapter configuration on each host

# SSH to the affected ESXi host:

# List all VMkernel adapters and their service tags:
esxcli network ip interface tag get
# Look for:
#   vmk1 (or whichever is for vSAN):
#     Tags: VSAN
# If "VSAN" tag is missing — that's the problem

# List VMkernel adapter IPs and subnets:
esxcli network ip interface ipv4 get
# Verify:
#   All hosts' vSAN VMkernel IPs are on the SAME subnet
#   No IP duplicates across hosts
#   Subnet mask is consistent (e.g., all /24 or all /25)

# Check which port group the VMkernel is on:
esxcli network ip interface list
# "Portgroup" column shows the associated port group/vDS portgroup
# Verify it is the correct vSAN-designated port groupCopy

Add or fix the vSAN service tag

# If the vSAN tag is missing from the correct VMkernel adapter:

# Add vSAN traffic tag to vmk1:
esxcli vsan network ipv4 add -i vmk1

# Verify the tag was applied:
esxcli vsan network list
# Should show:
#   Interface: vmk1
#   VsanVmknicUuid: (a UUID)
#   IP Protocol: IP

# If the wrong VMkernel has the vSAN tag (e.g., vmk0 instead of vmk1):
# Remove from wrong adapter:
esxcli vsan network ipv4 remove -i vmk0
# Add to correct adapter:
esxcli vsan network ipv4 add -i vmk1

# From vSphere Client (non-CLI alternative):
# Host > Configure > Networking > VMkernel adapters
# Select the correct vmk > Edit > check "vSAN" under Services
# Uncheck "vSAN" from any incorrect vmk adaptersCopy

Verify L2/L3 connectivity between all vSAN VMkernel interfaces

# From the affected host, ping every other host's vSAN VMkernel IP:
vmkping -I vmk1 <host2-vSAN-IP> -d -s 8972
vmkping -I vmk1 <host3-vSAN-IP> -d -s 8972
vmkping -I vmk1 <host4-vSAN-IP> -d -s 8972

# -I vmk1: use the vSAN VMkernel adapter
# -d: set don't-fragment bit
# -s 8972: test jumbo frames (9000 MTU - 28 byte header)

# If ping fails:
#   Check physical switch VLAN configuration
#   Check vDS/vSS port group VLAN ID matches physical switch
#   Check physical switch port is trunking the correct VLAN
#   If L3 (routed): verify gateway and routing between subnets

# After fixing, run vSAN Health check:
# Cluster > Monitor > vSAN > Health > Retest
# "Network" section should turn greenCopy

Test end-to-end MTU from every host to every other host

# SSH to each ESXi host and test jumbo frames to all other vSAN hosts:

# Test with 8972 byte payload (9000 MTU - 28 bytes IP+ICMP header):
vmkping -I vmk1 <host2-vSAN-IP> -d -s 8972
vmkping -I vmk1 <host3-vSAN-IP> -d -s 8972
vmkping -I vmk1 <host4-vSAN-IP> -d -s 8972

# -d = Don't Fragment flag (critical — forces end-to-end MTU validation)
# -s 8972 = payload size

# SUCCESS: "8972 bytes from x.x.x.x: ..."
# FAILURE: "PING x.x.x.x: sendto() No buffer space available"
#      or: "Request timeout" or "Packet needs to be fragmented"

# If ANY host pair fails — there is an MTU mismatch on that path
# You must test EVERY combination: host1→host2, host1→host3, etc.
# A single broken link can be between specific switch ports onlyCopy

Audit MTU at every layer — end to end

# Layer 1: Physical NIC MTU on each ESXi host:
esxcli network nic list
# Shows: Name, MTU, Link Status, Speed
# Every NIC carrying vSAN traffic must show MTU: 9000

# Layer 2: vSwitch / vDS MTU:
esxcli network vswitch dvs vmware list
# Check: MTU per vDS — must be 9000
# For standard vSwitch:
esxcli network vswitch standard list
# Check: MTU per vSS — must be 9000

# Layer 3: VMkernel adapter MTU:
esxcli network ip interface list
# Check: MTU for vmk1 (vSAN) — must be 9000

# Layer 4: Physical switch (from switch CLI):
# Cisco: show interface <port> | include MTU
# Arista: show interfaces <port> | include MTU
# Dell: show interfaces <port> status
# Verify EVERY port connected to ESXi hosts AND inter-switch links

# Common miss: inter-switch link / spine-leaf uplink at default 1500
# This breaks jumbo frames between hosts on different switchesCopy

Fix MTU — set 9000 end-to-end

# Fix VMkernel adapter MTU (if wrong):
# vSphere Client > Host > Configure > Networking > VMkernel adapters
# Select vmk1 > Edit > set MTU to 9000

# Fix vDS MTU:
# vSphere Client > Networking > select vDS > Configure > Settings
# Edit > MTU: 9000

# Fix standard vSwitch MTU (from ESXi Shell):
esxcli network vswitch standard set -v vSwitch1 -m 9000

# Fix physical switch (example Cisco NX-OS):
# interface Ethernet1/1
#   mtu 9216
#   (use 9216 on physical switches — slightly larger for L2 headers)

# After fixing all layers, retest:
vmkping -I vmk1 <host2-vSAN-IP> -d -s 8972

# Then rerun vSAN Health:
# Cluster > Monitor > vSAN > Health > Retest
# "Network > vSAN: MTU check" should turn green

# IMPORTANT: If you cannot set 9000 everywhere, set ALL to 1500
# Mixed MTU is worse than consistent 1500 MTU everywhereCopy

Check current teaming policy for vSAN port group

# For vDS (Distributed Switch):
# vSphere Client > Networking > vDS > Port Groups
# Select the vSAN port group > Edit Settings > Teaming and Failover

# Shows:
#   Load Balancing: (current policy)
#   Network Failure Detection: Link status / Beacon probing
#   Failover Order: Active / Standby uplinks

# From ESXi Shell (for vSS):
esxcli network vswitch standard policy failover get -v vSwitch1
# Shows: Load Balancing Policy, Active/Standby NICs

# Verify which physical NIC is currently carrying vSAN traffic:
esxcli network vswitch dvs vmware lacp status get
# Or check via esxtop (press 'n' for network view):
# Look at MbTX/s and MbRX/s per uplink — the active one has trafficCopy

Configure recommended teaming for vSAN on vDS

# Option A: Route based on physical NIC load (RECOMMENDED for vDS):
# vDS > vSAN Port Group > Edit Settings > Teaming and Failover
# Load Balancing: "Route based on physical NIC load"
# Network Failure Detection: "Link status only"
# Failover Order: Both uplinks Active

# This dynamically moves VMkernel port bindings between uplinks
# based on 30-second utilization sampling — best bandwidth utilization

# Option B: LACP (requires switch configuration):
# vDS > Configure > LACP
# New LAG > 2 ports > Mode: Active > Hash: L3+L4
# vSAN Port Group > Teaming: "Route based on IP hash"
# Failover Order: LAG1 Active, individual uplinks Unused

# Physical switch (Cisco NX-OS example):
# interface port-channel10
#   switchport mode trunk
#   switchport trunk allowed vlan 100
#   mtu 9216
#   vpc 10
# interface Ethernet1/1-2
#   channel-group 10 mode active

# IMPORTANT: LACP requires both sides configured simultaneously
# Misconfigured LACP causes complete traffic lossCopy

Test failover behavior

# After configuring teaming, test NIC failover:

# 1. Start a continuous ping from another host to verify connectivity:
# (On host2):
vmkping -I vmk1 <host1-vSAN-IP> -c 100 -i 1

# 2. On host1, simulate uplink failure by disabling one NIC:
esxcli network nic down -n vmnic2

# 3. Verify traffic fails over:
# Ping should see 1-3 lost packets, then resume
# If more than 5 seconds of loss — failover is too slow

# 4. Re-enable the NIC:
esxcli network nic up -n vmnic2

# 5. Verify failback:
# Traffic should return to original distribution

# Check vSAN Health after test:
# Cluster > Monitor > vSAN > Health > Network
# All checks should be green

# IMPORTANT: Test failover during a maintenance window
# Do NOT test during production hours on a live vSAN clusterCopy

Check if your cluster uses multicast or unicast

# From any ESXi host in the cluster:
esxcli vsan cluster get
# Look for:
#   "Sub-Cluster Master UUID" — if present, cluster is formed
#   "Multicast" or "Unicast" in the member list type

# Check vSAN version — determines communication mode:
esxcli vsan maintenancemode cancelrecoverytask
# If vSAN 7.0+ — always unicast (multicast not used)
# If vSAN 6.6-6.7 — unicast by default, multicast as fallback

# In vSphere Client:
# Cluster > Configure > vSAN > General
# If you see "Advanced Options" with multicast address — legacy mode

# vSAN on-disk format version:
esxcli vsan storage list | grep "On-Disk Format"
# Format 10+ = vSAN 7.0+ = always unicast
# Format 6-9 = vSAN 6.x = may use multicastCopy

Resolve legacy multicast issues (vSAN 6.x environments)

# If you are on vSAN 6.6+ and experiencing multicast-related partition:

# Option A: Upgrade to vSAN 7.0+ (recommended)
# This eliminates multicast entirely
# Follow standard vSAN upgrade path via vLCM

# Option B: Verify multicast on physical switches (if stuck on 6.x):
# Check IGMP snooping is enabled on the vSAN VLAN:
# Cisco: show ip igmp snooping vlan <vsan-vlan-id>
# Arista: show ip igmp snooping

# Check multicast group membership:
# Cisco: show ip igmp snooping groups vlan <vsan-vlan-id>
# The vSAN multicast address (default 224.1.2.3) should be listed

# If multicast is not working:
# Ensure IGMP querier exists on the VLAN
# Cisco: ip igmp snooping querier (on SVI for that VLAN)

# For L3 routed vSAN (across subnets):
# PIM must be configured on all L3 interfaces in the path
# This is complex — another reason to upgrade to 7.0+ for unicastCopy

Post-upgrade: clean up legacy multicast switch config

# After upgrading to vSAN 7.0+ (unicast), clean up switch configs:

# 1. IGMP snooping on the vSAN VLAN is no longer needed for vSAN
#    (keep it if other services on the same VLAN use multicast)

# 2. PIM configuration for vSAN multicast group — remove:
#    no ip pim sparse-mode (on vSAN VLAN SVI)
#    no ip igmp snooping querier (if only used for vSAN)

# 3. Multicast group reservations on switches — remove:
#    no ip igmp snooping group 224.1.2.3

# Verify vSAN cluster is healthy after switch changes:
# Cluster > Monitor > vSAN > Health > Retest all
# Network checks should all be green

# IMPORTANT: Make switch changes during maintenance window
# Even though vSAN 7.0+ doesn't use multicast, removing switch config
# during production is still a change control itemCopy

Detect the partition — identify which hosts can see which

# SSH to each ESXi host and check cluster membership:

# On Host 1:
esxcli vsan cluster get
# Note: Sub-Cluster Master UUID and member list

# On Host 2:
esxcli vsan cluster get
# Compare member lists

# If Host1 sees {Host1, Host2} but Host3 sees {Host3, Host4}
# → You have a 2+2 partition

# Quick connectivity matrix — from Host1:
vmkping -I vmk1 <host2-vSAN-IP>  # Should succeed (same sub-cluster)
vmkping -I vmk1 <host3-vSAN-IP>  # Will FAIL (other sub-cluster)
vmkping -I vmk1 <host4-vSAN-IP>  # Will FAIL (other sub-cluster)

# Check vSAN health from each side:
esxcli vsan health cluster list
# Each sub-cluster thinks the other hosts are "absent"

# CRITICAL: Do NOT reboot hosts or restart services
# This makes recovery harder. Fix the network first.Copy

Identify and fix the network break

# The partition is ALWAYS caused by a network issue. Find it:

# Check physical NIC link status on each host:
esxcli network nic list
# If "Link Status: Down" on vSAN NIC — cable/SFP/switch port issue

# Check switch port status (from switch CLI):
# Cisco: show interface status | include connected
# Look for err-disabled, notconnect, or down ports

# Check spanning tree:
# Cisco: show spanning-tree vlan <vsan-vlan-id>
# If a port is in "Blocking" state — STP is isolating the path

# Check inter-switch links:
# Cisco: show port-channel summary
# If the port-channel to another switch is "down" — that's the break

# Check for ACLs/firewall rules:
# If vSAN is on an L3 network, verify no firewall/ACL blocks:
#   TCP/UDP 2233 (vSAN transport)
#   TCP 443 (vSAN management)

# Common fix actions:
#   - Reseat SFP/cable on failed inter-switch link
#   - Re-enable err-disabled port: (Cisco) shut / no shut
#   - Fix VLAN trunk: switchport trunk allowed vlan add <vsan-vlan>
#   - Fix spanning tree: clear spanning-tree detected-protocolsCopy

Pre-migration checklist

# Before starting migration, document current config:

# Record VMkernel config per host:
esxcli network ip interface list
esxcli network ip interface ipv4 get
esxcli network ip interface tag get
# Save: vmk name, IP, subnet, MTU, VLAN, service tags (vSAN, vMotion, etc.)

# Record vSS config:
esxcli network vswitch standard list
# Save: MTU, port groups, uplink NICs, failover policy

# Record physical switch port config:
# Document which switch port connects to which NIC on each host

# Pre-create the vDS:
# vSphere Client > Networking > New Distributed Switch
# Set MTU: 9000
# Create port groups matching vSS configuration:
#   vSAN port group — same VLAN ID as vSS vSAN port group
#   Management port group — same VLAN ID
# Set teaming policy on vSAN port group (Route based on physical NIC load)Copy

Migrate one host at a time using the vDS wizard

# In vSphere Client:
# Networking > vDS > Add and Manage Hosts > Add Hosts
# Select ONE host

# Step 1 — Assign uplinks:
# Move ONLY ONE physical NIC (e.g., vmnic2) from vSS to vDS
# Leave vmnic3 on vSS as a safety net

# Step 2 — Migrate VMkernel adapters:
# Map the vSAN VMkernel (vmk1) from vSS port group → vDS vSAN port group
# The wizard preserves IP, subnet, and service tags

# Step 3 — Complete and verify:
# After migration completes:
esxcli network ip interface tag get
# Verify vmk1 still has "VSAN" tag

vmkping -I vmk1 <other-host-vSAN-IP> -d -s 8972
# Verify jumbo frame connectivity

# If connectivity fails — immediately roll back:
# Move vmk1 back to vSS port group
# Move vmnic2 back to vSS
# Investigate the issue before retryingCopy

Complete migration and move second uplink

# After vmk1 is verified on vDS with one uplink:

# Move the second NIC (vmnic3) from vSS to vDS:
# vDS > Add and Manage Hosts > Manage Hosts
# Select the host > Assign vmnic3 to vDS uplink2

# Verify both uplinks are active:
esxcli network vswitch dvs vmware list
# Should show both uplinks assigned

# Verify teaming is working:
# esxtop > 'n' > check both uplinks show traffic

# Repeat for each host — ONE HOST AT A TIME
# Always verify vSAN health after each host migration:
# Cluster > Monitor > vSAN > Health > Retest

# After ALL hosts are migrated:
# Remove the old vSS (if no longer needed):
# Host > Configure > Networking > Virtual switches
# Delete vSwitch1 (only if all port groups are migrated)

# Final verification:
# Cluster > Monitor > vSAN > Health > Full retest
# ALL network checks must be greenCopy

Verify hardware compatibility and enable RDMA on NICs

# Check if NICs support RDMA:
esxcli rdma device list
# Should show RDMA-capable devices:
#   Name: vmrdma0
#   Driver: mlx5_core (Mellanox) or bnxt_re (Broadcom)
#   State: Active

# Verify NIC firmware supports RoCE v2:
esxcli network nic get -n vmnic0
# Check firmware version against VMware HCL RDMA compatibility list

# If no RDMA devices show — NIC doesn't support it or driver is wrong
# Check VMware Compatibility Guide for the specific NIC model + firmware + driver

# Verify ESA is enabled:
# Cluster > Configure > vSAN > General
# Architecture: "Express Storage Architecture" must be activeCopy

Configure lossless Ethernet on physical switches

# This must be done by the network team on physical switches.
# Example: Cisco Nexus 9000 configuration for RoCE v2:

# Enable PFC on traffic class 3 (for RDMA):
# interface Ethernet1/1
#   priority-flow-control mode on
#   priority-flow-control priority 3 no-drop

# Enable ECN:
# policy-map type queuing rdma-policy
#   class type queuing c-out-8q-q3
#     bandwidth percent 50
#     random-detect minimum-threshold 150 kbytes maximum-threshold 3000 kbytes
#     congestion-control ecn

# Enable DCBX:
# interface Ethernet1/1
#   dcbx version auto

# IMPORTANT: Every switch in the path must be configured for lossless
# If any hop is lossy, RDMA performance will be worse than TCP

# Verify PFC counters (after enabling):
# show interface Ethernet1/1 priority-flow-control
# Watch for "Rx Pause" and "Tx Pause" — some pauses are normal
# High pause counts indicate congestion in the fabricCopy

Enable RDMA transport in vSAN

# In vSphere Client:
# Cluster > Configure > vSAN > Services
# Network > Transport: change from "TCP" to "RDMA"

# This change is non-disruptive — vSAN switches transport protocol
# on all hosts in the cluster

# After enabling, verify RDMA is active:
esxcli vsan network list
# Look for: "Transport: RDMA"

# Check RDMA device binding:
esxcli rdma device stats get -d vmrdma0
# Should show active connections and data transfer

# Monitor RDMA performance vs TCP baseline:
# Cluster > Monitor > vSAN > Performance
# Compare latency before and after — expect:
#   TCP: 200-500µs average latency
#   RDMA: 50-150µs average latency (2-4x improvement)

# If performance does NOT improve or gets worse:
# Check for packet drops in the fabric:
esxcli rdma device stats get -d vmrdma0
# Look for "RNR Retry" or "Transport Retry" — indicates lossy fabricCopy

Validate inter-site link latency and bandwidth

# Test inter-site latency from ESXi host at Site A to Site B:
vmkping -I vmk1 <siteB-host-vSAN-IP> -c 100
# Average RTT should be ≤5ms
# Jitter (variation) should be <1ms — high jitter causes I/O inconsistency

# Test MTU across the ISL:
vmkping -I vmk1 <siteB-host-vSAN-IP> -d -s 8972
# Must succeed — if it fails, MTU is not 9000 end-to-end

# Test witness connectivity from both sites:
# From Site A host:
vmkping -I vmk1 <witness-vSAN-IP> -c 20
# From Site B host:
vmkping -I vmk1 <witness-vSAN-IP> -c 20
# RTT should be ≤200ms from both sites

# Bandwidth test (schedule during maintenance):
# Use vSAN Performance Service after cluster is running:
# Cluster > Monitor > vSAN > Performance > "Backend"
# Check inter-site throughput during peak I/OCopy

Configure site awareness and fault domains

# In vSphere Client:
# Cluster > Configure > vSAN > Fault Domains and Stretched Cluster

# Create fault domains:
#   Preferred: Site A (hosts at primary site)
#   Secondary: Site B (hosts at secondary site)
#   Witness: Witness host at third location

# Assign hosts to their respective fault domains:
# Drag Host1, Host2, Host3 → "Site A" domain
# Drag Host4, Host5, Host6 → "Site B" domain
# Assign witness appliance → "Witness" domain

# Set site preference:
# "Preferred fault domain": Site A
# This determines which site keeps running if the ISL fails:
#   - Site A (preferred) keeps objects accessible
#   - Site B VMs become inaccessible until ISL restores

# IMPORTANT: Preferred site + witness must be able to communicate
# for quorum. If both ISL and witness link fail, all objects are lost.Copy

Size inter-site bandwidth for your workload

# Formula: ISL Bandwidth = VM Write Rate × FTT Multiplier + Resync Headroom

# Step 1: Measure current VM write throughput:
# Cluster > Monitor > vSAN > Performance > "Backend"
# Peak Write Throughput: e.g., 2 GB/s aggregate across all VMs

# Step 2: Calculate ISL requirement:
#   Stretched cluster with FTT=1 (site-level mirror):
#   Every write goes to BOTH sites synchronously
#   ISL bandwidth needed = Peak Write Rate = 2 GB/s = 16 Gbps
#   + 20% resync headroom = ~20 Gbps

# Step 3: Add non-vSAN traffic if ISL is shared:
#   vMotion, replication, management traffic: add 2-4 Gbps

# Total ISL: 20 + 4 = 24 Gbps → need 25GbE or dual 10GbE bonded

# IMPORTANT: vSAN writes are SYNCHRONOUS across the ISL
# Write latency = local SSD latency + ISL RTT
# If ISL RTT = 3ms → every write adds 3ms latency
# This is why ≤1ms RTT is recommended for performance-sensitive workloadsCopy

Check ESXi firewall rules for vSAN

# List all ESXi firewall rules related to vSAN:
esxcli network firewall ruleset list | grep -i vsan
# Should show:
#   vsanvp         true    (vSAN VASA vendor provider)
#   vsanEncryption true    (vSAN encryption KMS traffic)
#   vsanhealth     true    (vSAN health check)
#   vSAN           true    (vSAN data transport)

# If any vSAN ruleset is "false" (disabled):
esxcli network firewall ruleset set -r vSAN -e true
esxcli network firewall ruleset set -r vsanvp -e true
esxcli network firewall ruleset set -r vsanhealth -e true

# Verify firewall is allowing vSAN traffic:
esxcli network firewall ruleset rule list -r vSAN
# Should show TCP 2233 allowed

# Check if the ESXi firewall is enabled at all:
esxcli network firewall get
# "Enabled: true" is recommended (don't disable the entire firewall)Copy

Test port connectivity between hosts

# Test TCP 2233 (vSAN transport) between hosts:
# From Host1, test connectivity to Host2 on port 2233:
nc -z <host2-vSAN-IP> 2233
# Or use:
esxcli network diag connection test --host=<host2-vSAN-IP> --port=2233

# If connection fails — something is blocking port 2233:
#   1. Physical firewall between hosts
#   2. Switch ACL on the vSAN VLAN
#   3. ESXi firewall on the destination host

# Test UDP 12321 (CMMDS):
nc -zu <host2-vSAN-IP> 12321

# Test from vCenter to hosts:
# Ensure vCenter can reach all hosts on TCP 443 and TCP 8006

# For stretched clusters — test ports across the ISL:
nc -z <siteB-host-vSAN-IP> 2233
nc -z <witness-IP> 2233

# If ports are blocked by physical firewall:
# Request the network team to open these ports on the vSAN VLAN/subnetCopy

Resolve and verify

# After opening blocked ports:

# 1. Restart vSAN health service to re-check connectivity:
# From ESXi Shell:
/etc/init.d/vsanmgmtd restart

# 2. Run vSAN Health checks:
# Cluster > Monitor > vSAN > Health > Retest
# "Network health" section should show all green:
#   ✓ All hosts have vSAN vmknic configured
#   ✓ All hosts have matching subnets
#   ✓ vSAN MTU check (ping with large packet size)
#   ✓ vSAN cluster members network connectivity

# 3. If a host was partitioned due to blocked ports:
# After ports are opened, the host should rejoin automatically
# Check: esxcli vsan cluster get
# Member list should show all hosts

# 4. Document the required ports in your firewall change request:
# Port matrix:
#   Host ↔ Host: TCP 2233, UDP 12321, UDP 12345, UDP 23451
#   Host ↔ vCenter: TCP 443, TCP 8006
#   Host ↔ Witness: TCP 2233, UDP 12321, UDP 12345
#   ICMP: Allow between all vSAN hosts (for health checks)Copy

Select NICs from the vSAN HCL

# Check VMware Compatibility Guide for supported 25/100GbE NICs:
# https://www.vmware.com/resources/compatibility
# Filter: I/O Devices > Network > vSAN

# Popular validated NICs for ESA:
#   Mellanox ConnectX-6 Dx — 2×25GbE or 2×100GbE (supports RDMA)
#   Broadcom BCM57508 — 2×100GbE (P2100G)
#   Intel E810 — 2×25GbE or 4×25GbE

# Key selection criteria:
#   - On the vSAN HCL with your specific ESXi version
#   - RDMA support (RoCE v2) if you plan to use RDMA transport
#   - Matching firmware and driver version per HCL
#   - Dual-port minimum for redundancy

# Verify NIC after installation:
esxcli network nic list
# Check: Speed, MTU, Driver, Firmware Version
# Cross-reference with HCL entryCopy

Design the physical topology

# Recommended: Spine-Leaf with dual ToR per rack

# Per-host cabling (2×25GbE example):
#   NIC Port 1 (vmnic0) → Leaf Switch A, Port X
#   NIC Port 2 (vmnic1) → Leaf Switch B, Port Y
# This gives full redundancy — either switch can fail

# Switch configuration per port:
#   MTU 9216 (jumbo frames)
#   Trunk: vSAN VLAN + Management VLAN + vMotion VLAN
#   Spanning tree: edge port (portfast)
#   Storm control: enable on access ports

# Leaf-to-Spine uplinks:
#   Minimum: 4×25GbE or 2×100GbE from each leaf to each spine
#   Oversubscription: calculate per rack:
#     40 hosts × 25 Gbps = 1000 Gbps downlink
#     Uplink to spine: 400 Gbps = 2.5:1 oversubscription (acceptable)

# vDS configuration on ESXi:
#   vDS MTU: 9000
#   vSAN Port Group: VLAN X, teaming = Route based on physical NIC load
#   vMotion Port Group: VLAN Y
#   Management Port Group: VLAN ZCopy

Validate network throughput after deployment

# After cluster deployment, validate actual throughput:

# Check link speed on each host:
esxcli network nic list
# Verify: Speed = 25000 Mb/s (25GbE) or 100000 Mb/s (100GbE)
# If showing 10000 — SFP or autoneg issue

# Verify jumbo frames end-to-end:
vmkping -I vmk1 <other-host-vSAN-IP> -d -s 8972
# Must succeed on all host pairs

# Monitor actual vSAN throughput:
# Cluster > Monitor > vSAN > Performance > "Backend"
# Check:
#   Throughput: should scale beyond 10 Gbps during heavy I/O
#   If capped at ~10 Gbps with 25GbE NICs — check NIC teaming
#   If capped at ~1.2 GB/s — likely only one NIC active (teaming misconfigured)

# Check for network-related vSAN congestion:
# Cluster > Monitor > vSAN > Performance
# If congestion appears but disk latency is low → network bottleneck
# See KB #51 for congestion diagnosis

# Monitor NIC utilization via esxtop:
esxtop
# Press 'n' — check MbTX/s and MbRX/s per uplink
# If one NIC at 25 Gbps and other at 0 → teaming is active/standby, not load-balancedCopy

Check time sync status on all hosts

# On each ESXi host:
esxcli system time get
esxcli network firewall ruleset list --ruleset-id ntpClient

# Check NTP daemon status:
/etc/init.d/ntpd status

# Compare time across all hosts (run from vCenter via PowerCLI):
Get-VMHost | Select Name, @{N="Time";E={(Get-View $_.Id).ConfigManager |
    ForEach-Object { (Get-View $_.DateTimeSystem).QueryDateTime() }}} | Sort TimeCopy

Fix NTP configuration on all hosts

# Set NTP server (use at least 2 external/infrastructure NTP sources):
esxcli system ntp set --server=ntp1.corp.local --server=ntp2.corp.local
esxcli system ntp set --enabled=true

# Restart NTP service:
/etc/init.d/ntpd restart

# Force immediate sync (if drift is large):
/etc/init.d/ntpd stop
esxcli system time set --hour=HH --min=MM --sec=SS  # set manually first
/etc/init.d/ntpd start

# Set NTP to start on boot:
chkconfig ntpd onCopy

Verify CMMDS recovers after time fix

# After syncing time, verify cluster membership:
esxcli vsan cluster get
# All hosts should show the SAME "Sub-Cluster Master UUID"
# and "Sub-Cluster Member Count" should equal total hosts

# If partition persists, restart vSAN clustering on the affected host:
esxcli vsan cluster leave
esxcli vsan cluster join -c <cluster-uuid>Copy

Verify vSAN VMkernel tagging

# Check if VMkernel has vSAN traffic enabled:
esxcli vsan network list
# Should show vmk interface with "VsanVmknicUuid"

# If empty, re-tag the VMkernel:
esxcli vsan network ipv4 add -i vmk1
# (use the correct vmk interface for your setup)Copy

Rejoin the vSAN cluster

# Get the cluster UUID from a working host:
# On a healthy host: esxcli vsan cluster get → note "Sub-Cluster UUID"

# On the affected host:
esxcli vsan cluster join -c <cluster-uuid>

# Verify:
esxcli vsan cluster get
# Should now show cluster membership

# Refresh vCenter:
# Right-click host → Connection → ReconnectCopy

Verify datastore reappears and VMs recover

# After rejoining, vSAN datastore should appear within 1-2 minutes
# Verify disk groups are claimed:
esxcli vsan storage list

# If VMs still show "Inaccessible":
# Right-click VM → Remove from Inventory (don't delete!)
# Then: Datastore → Browse → find .vmx → Register VMCopy

Enable data-in-transit encryption

# vSphere Client:
# Cluster → Configure → vSAN → Services → Data-in-Transit Encryption → ON

# Three levels:
# 1. Disabled (default) — no network encryption
# 2. Preferred — encrypt when both hosts support it
# 3. Required — all vSAN traffic must be encrypted (fail if not possible)

# Note: does NOT require KMS (unlike data-at-rest)
# Uses self-signed certificates managed by vCenter

# Verify encryption is active:
esxcli vsan debug resync summary
# In-transit encrypted traffic shows in Performance Service dashboardsCopy

Performance impact and troubleshooting

# Performance impact:
# - Adds ~5-10% CPU overhead for encryption/decryption per host
# - Minimal latency increase (~0.1-0.3ms per I/O)
# - More noticeable during heavy resync operations
# - AES-NI acceleration reduces impact on modern CPUs

# Verify CPU supports AES-NI:
grep -c aes /proc/cpuinfo
# Any number > 0 means AES-NI is available

# If performance is a concern:
# Use "Preferred" mode instead of "Required"
# This allows resync traffic to use unencrypted path when needed

# Common issue: all hosts must be on compatible vSAN version
# If one host is too old, "Required" mode blocks communication with itCopy