VMware vSphere Knowledge Base

Phase 4 — DRS & Resource Management ⚠ Break Scenarios ✓ Fix Procedures
10
DRS & Resource Issues
5
Break Scenarios
5
Fix Procedures
Phase 4
Current Phase
31
DRS Affinity / Anti-Affinity Rule Conflicts
Conflicting rules prevent DRS from migrating VMs — rule conflict faults and migration deadlocks
Impact: DRS cannot generate valid placement recommendations. VMs remain on suboptimal hosts, and new VM power-on requests may fail with "no suitable host" errors. Conflicting must-rules create an unresolvable deadlock.
Symptoms
  • DRS faults: "Rule conflict" in Monitor → Faults
  • VMs fail to power on: "No suitable host found"
  • DRS recommendations appear but are immediately cancelled
  • Cluster shows yellow/red DRS status in Configure tab
  • vMotion blocked with "Would violate affinity rule"
Root Causes
  • Two must-rules that contradict each other (e.g., keep VM-A with VM-B AND separate VM-A from VM-B)
  • VM-Host affinity rule conflicts with VM-VM anti-affinity
  • Too many must-rules with limited host count
  • Legacy rules left from decommissioned VMs/hosts
  • HA failover constrained by mandatory placement rules

DRS evaluates all affinity and anti-affinity rules before generating migration recommendations. Must-rules are mandatory constraints — DRS will never violate them. When two must-rules conflict (e.g., "VM-A must run on Host-1" and "VM-A must be separated from VM-B" but VM-B is also pinned to Host-1), DRS enters a deadlock state. It generates faults instead of recommendations, and VMs cannot be placed or migrated. Should-rules are soft constraints that DRS will violate if necessary — but must-rules always override should-rules.

1

List all DRS rules and identify conflicts

# PowerCLI — List all DRS rules in the cluster:
$cluster = Get-Cluster "ClusterName"
Get-DrsRule -Cluster $cluster | Format-Table Name, Type, Enabled, Mandatory -AutoSize

# Check VM-Host rules:
Get-DrsVMHostRule -Cluster $cluster | Format-Table Name, Type, Enabled, VMGroup, VMHostGroup -AutoSize

# List VM groups and Host groups:
Get-DrsClusterGroup -Cluster $cluster | Format-Table Name, GroupType, Member -AutoSize
2

Check DRS faults and failed recommendations

# PowerCLI — Get DRS faults:
Get-DrsRecommendation -Cluster $cluster | Format-Table -AutoSize

# vSphere Client path:
# Cluster → Monitor → vSphere DRS → Recommendations
# Cluster → Monitor → vSphere DRS → Faults

# Check for rule conflicts in events:
Get-VIEvent -Entity $cluster -MaxSamples 100 |
  Where { $_.FullFormattedMessage -match "rule|conflict|affinity" } |
  Select CreatedTime, FullFormattedMessage | Format-Table -AutoSize
3

Resolve conflicting rules

# Option 1: Disable conflicting must-rule temporarily:
Get-DrsRule -Cluster $cluster -Name "ConflictingRuleName" |
  Set-DrsRule -Enabled:$false

# Option 2: Change must-rule to should-rule:
# vSphere Client → Cluster → Configure → VM/Host Rules
# Edit rule → Change "Must" to "Should"

# Option 3: Remove obsolete rules:
Get-DrsRule -Cluster $cluster -Name "ObsoleteRule" | Remove-DrsRule -Confirm:$false

# Verify DRS health after changes:
Get-Cluster $cluster | Select Name, DrsEnabled, DrsAutomationLevel
🛈 Best Practice: Prefer should-rules over must-rules unless regulatory or licensing requirements demand strict enforcement. Audit DRS rules quarterly to remove stale entries. Document every must-rule with a business justification. In clusters with fewer than 4 hosts, minimize must-rules to avoid deadlocks.
32
Resource Pool Misconfiguration — Nested Pools
Resources trapped in nested pools — VMs starved due to reservation/share imbalance and excessive hierarchy
Impact: VMs in deeply nested pools are starved of CPU and memory resources. Production workloads experience severe performance degradation while resources sit idle in parent pool reservations that are never consumed.
Symptoms
  • VMs show high CPU ready time (>5%) despite cluster having available capacity
  • Memory ballooning in VMs inside nested pools while parent pools show unused reservations
  • DRS cannot migrate VMs across pool boundaries effectively
  • New VM power-on fails: "Insufficient resources"
  • Resource pool hierarchy is 3+ levels deep
Root Causes
  • Excessive nesting (used as folder organization instead of resource control)
  • Fixed reservations at parent level that lock resources
  • Expandable reservation disabled on child pools
  • Share values not adjusted — child pools get equal shares regardless of workload priority
  • No one audits or cleans up pool hierarchy over time

Resource pools are designed for resource management, not for organizing VMs like folders. When administrators create deeply nested pools (Dev → Team-A → Project-X → Sprint-1), each level adds reservation overhead. A parent pool with a fixed 32 GB memory reservation locks that memory even if its child VMs only use 8 GB. Meanwhile, VMs in sibling pools starve because the cluster appears fully reserved. The Expandable Reservation checkbox (enabled by default) allows child pools to borrow from the parent — disabling it creates hard silos.

1

Audit resource pool hierarchy and reservations

# PowerCLI — List all resource pools with reservations:
Get-Cluster "ClusterName" | Get-ResourcePool | Sort Name |
  Select Name, Parent,
    @{N='CpuReservationMHz';E={$_.CpuReservationMHz}},
    @{N='MemReservationGB';E={[math]::Round($_.MemReservationGB,2)}},
    @{N='CpuExpandable';E={$_.CpuExpandableReservation}},
    @{N='MemExpandable';E={$_.MemExpandableReservation}},
    @{N='CpuShares';E={$_.NumCpuShares}},
    @{N='MemShares';E={$_.NumMemShares}} |
  Format-Table -AutoSize

# Count nesting depth:
Get-ResourcePool | ForEach {
    $depth = 0; $p = $_.Parent
    while ($p -is [VMware.VimAutomation.ViCore.Types.V1.Inventory.ResourcePool]) {
        $depth++; $p = $p.Parent
    }
    [PSCustomObject]@{ Pool=$_.Name; Depth=$depth }
} | Sort Depth -Descending | Format-Table -AutoSize
2

Check actual VM resource consumption vs pool reservations

# PowerCLI — Compare pool reservation to actual VM usage:
$pools = Get-Cluster "ClusterName" | Get-ResourcePool
foreach ($pool in $pools) {
    $vms = Get-VM -Location $pool -NoRecursion
    $usedCpu = ($vms | Measure-Object -Property UsedSpaceGB -Sum).Sum
    $usedMem = ($vms | ForEach { $_.MemoryGB }) | Measure-Object -Sum
    [PSCustomObject]@{
        Pool = $pool.Name
        VMCount = $vms.Count
        ReservedMemGB = [math]::Round($pool.MemReservationGB,1)
        ActualMemGB = [math]::Round($usedMem.Sum,1)
        Waste = [math]::Round($pool.MemReservationGB - $usedMem.Sum,1)
    }
} | Format-Table -AutoSize
3

Flatten pool hierarchy and fix shares

# Move VMs out of deeply nested pools:
# Use vSphere Client → Drag VMs to top-level resource pool

# PowerCLI — Reset expandable reservations:
Get-ResourcePool -Name "NestedPool" |
  Set-ResourcePool -MemExpandableReservation:$true -CpuExpandableReservation:$true

# Remove empty nested pools:
Get-ResourcePool | Where { (Get-VM -Location $_ -NoRecursion).Count -eq 0 } |
  Where { $_.Name -ne "Resources" } |
  Remove-ResourcePool -Confirm:$false

# Best practice: Use VM folders for organization, resource pools only for resource control
Warning: Never use resource pools as organizational folders. Use VM Folders for logical grouping. Resource pools should only exist when you need to enforce resource guarantees (reservations) or priorities (shares) between groups of workloads. One level of pools is usually sufficient.
33
DRS Migration Threshold Tuning
Optimizing DRS aggressiveness — balancing load distribution against vMotion overhead
🛈 Context: DRS migration threshold controls how aggressively DRS moves VMs to balance load. The right setting depends on your environment's tolerance for vMotion storms vs. resource imbalance.
Threshold Scale
  • Level 1 (Conservative): Only mandatory moves (host entering maintenance, affinity rules)
  • Level 2: Apply recommendations with significant cluster-wide benefit
  • Level 3 (Default): Apply recommendations for good improvements in load balance
  • Level 4: Apply recommendations for moderate improvements
  • Level 5 (Aggressive): Apply recommendations for even small improvements
Trade-offs
  • Aggressive (4-5): Better balance, more vMotion traffic, higher network overhead
  • Conservative (1-2): Fewer migrations, less overhead, potential hot spots
  • Default (3): Balanced approach for most environments
  • Each vMotion consumes ~1-3 seconds of CPU and network bandwidth
  • Latency-sensitive workloads suffer from frequent migrations
1

Check current DRS settings and migration history

# PowerCLI — Check DRS automation level and threshold:
Get-Cluster "ClusterName" | Select Name, DrsEnabled, DrsAutomationLevel,
  @{N='MigrationThreshold';E={
    $_.ExtensionData.Configuration.DrsConfig.VmotionRate
  }}

# Review recent DRS migration events:
Get-VIEvent -Entity (Get-Cluster "ClusterName") -MaxSamples 200 |
  Where { $_.FullFormattedMessage -match "DRS|migrat|vMotion" } |
  Group { $_.CreatedTime.ToString("yyyy-MM-dd") } |
  Select Name, Count | Sort Name
2

Adjust DRS migration threshold

# PowerCLI — Set DRS to Fully Automated with threshold level 3:
Set-Cluster "ClusterName" -DrsAutomationLevel FullyAutomated -Confirm:$false

# vSphere Client path:
# Cluster → Configure → vSphere DRS → Edit
# Automation Level: Fully Automated
# Migration Threshold slider: Position 3 (default, recommended)

# For performance-sensitive clusters, use Level 2:
# This reduces unnecessary migrations while still balancing significant imbalances
3

Set per-VM DRS overrides for sensitive workloads

# For latency-sensitive VMs, set individual DRS behavior:
# vSphere Client → Cluster → Configure → VM Overrides → Add
# Select VM → DRS Automation Level: "Partially Automated" or "Manual"

# PowerCLI — Set per-VM DRS override:
$cluster = Get-Cluster "ClusterName"
$vm = Get-VM "LatencySensitiveVM"
$spec = New-Object VMware.Vim.ClusterConfigSpecEx
$vmSpec = New-Object VMware.Vim.ClusterDrsVmConfigSpec
$vmSpec.Operation = "add"
$vmSpec.Info = New-Object VMware.Vim.ClusterDrsVmConfigInfo
$vmSpec.Info.Key = $vm.ExtensionData.MoRef
$vmSpec.Info.Behavior = "partiallyAutomated"
$spec.DrsVmConfigSpec += $vmSpec
$cluster.ExtensionData.ReconfigureComputeResource($spec, $true)
Best Practice: Use Level 3 (default) for general-purpose clusters. Use Level 2 for clusters hosting latency-sensitive or database workloads. Use per-VM overrides instead of lowering the cluster-wide threshold. Monitor DRS migration count — if exceeding 50+ migrations/hour consistently, consider reducing aggressiveness.
34
EVC Mode Mismatch — CPU Compatibility
vMotion fails with "CPU incompatible with EVC mode" — host CPU generation doesn't match cluster baseline
Impact: vMotion between hosts fails, preventing DRS load balancing and HA failover. VMs are pinned to their current host. Maintenance mode operations blocked — host cannot evacuate VMs.
Symptoms
  • vMotion fails: "The host's CPU is incompatible with the cluster's EVC mode"
  • Host cannot join cluster: "CPU mismatch"
  • DRS fault: "Cannot migrate VM — CPU features not available"
  • Host maintenance mode hangs — VMs cannot be evacuated
  • Cluster EVC status shows host non-compliance
Root Causes
  • New host has older CPU generation than EVC baseline
  • EVC set to a CPU level that newer host doesn't support (rare)
  • BIOS/firmware update changed CPU feature flags
  • VM was created/migrated from a non-EVC cluster with exposed CPU features
  • EVC mode was never enabled on a mixed-CPU cluster

EVC (Enhanced vMotion Compatibility) masks advanced CPU features to create a common baseline across all hosts in a cluster. When EVC is set to "Intel Skylake" baseline, all hosts present the same CPU feature set regardless of their actual hardware generation. If a new host has an older CPU (e.g., Haswell) that lacks features required by the EVC baseline, it cannot join the cluster. Similarly, if EVC is not enabled and hosts have different CPU generations, vMotion fails because the VM's CPU instructions differ between source and destination.

1

Check current EVC mode and host CPU compatibility

# PowerCLI — Check cluster EVC mode:
Get-Cluster "ClusterName" | Select Name,
  @{N='EVCMode';E={$_.ExtensionData.Summary.CurrentEVCModeKey}},
  @{N='EVCEnabled';E={$_.EVCMode -ne $null}}

# List host CPU models:
Get-Cluster "ClusterName" | Get-VMHost | Select Name,
  @{N='CPUModel';E={$_.ProcessorType}},
  @{N='CpuFeatureLevel';E={$_.MaxEVCMode}},
  @{N='State';E={$_.ConnectionState}} |
  Format-Table -AutoSize

# vSphere Client:
# Cluster → Configure → VMware EVC → View current baseline and compliance
2

Identify the lowest common CPU denominator

# PowerCLI — Find the lowest EVC mode across all hosts:
$hosts = Get-Cluster "ClusterName" | Get-VMHost
$hosts | Select Name, ProcessorType, MaxEVCMode |
  Sort MaxEVCMode | Format-Table -AutoSize

# The EVC baseline must be ≤ the lowest MaxEVCMode in the cluster

# Check if a specific host can join at current EVC level:
# Compare host's MaxEVCMode with cluster's CurrentEVCModeKey
# If host.MaxEVCMode < cluster.EVCMode → host CANNOT join
3

Fix EVC mode or remove incompatible host

# Option 1: Lower EVC mode to accommodate the oldest CPU
# REQUIREMENT: All VMs must be powered off or migrated first
# vSphere Client → Cluster → Configure → VMware EVC → Change EVC Mode
# Select the lowest common baseline (e.g., "Intel Haswell")

# Option 2: Remove the incompatible host from the cluster
Remove-VMHost -VMHost "incompatible-host.domain.com" -Confirm:$false

# Option 3: Enable EVC on a cluster that has none (all VMs must be off):
Set-Cluster "ClusterName" -EVCMode "intel-broadwell" -Confirm:$false

# Verify after change:
Get-Cluster "ClusterName" | Select Name,
  @{N='EVCMode';E={$_.ExtensionData.Summary.CurrentEVCModeKey}}
Best Practice: Always enable EVC when creating a new cluster — it's much harder to enable later with running VMs. Set EVC to the lowest CPU generation you plan to use in the cluster's lifecycle. When purchasing new hosts, verify their CPU family matches or exceeds the cluster EVC baseline before racking them. Per-VM EVC (vSphere 6.7+) allows individual VM CPU masking without cluster-wide constraints.
35
VM-Host Affinity Rules & Licensing
Pinning VMs to specific hosts for licensing compliance and hardware affinity — should vs. must rules
🛈 Context: Some software licenses (Oracle, SQL Server, SAP) require VMs to run on specific hosts or a limited set of sockets. VM-Host affinity rules enforce this at the DRS level — but misconfiguration can break HA failover.
Common Use Cases
  • Oracle licensing: Pin Oracle VMs to hosts with purchased licenses (per-socket)
  • SQL Server: Restrict to licensed hosts for Standard Edition
  • GPU passthrough: Pin VMs to hosts with specific GPU hardware
  • Data locality: Keep VMs close to local SSD storage
  • Compliance: Ensure VMs stay within a geographic boundary
Should vs. Must Rules
  • Should run on: DRS prefers these hosts but can violate during HA failover
  • Must run on: Hard constraint — HA will NOT restart VM on non-member hosts
  • Should NOT run on: Soft anti-affinity to hosts
  • Must NOT run on: Hard exclusion — VM cannot power on these hosts
  • Warning: "Must" rules + insufficient hosts = VM stays offline after failure
1

Create VM Group and Host Group

# PowerCLI — Create a VM group for Oracle VMs:
$cluster = Get-Cluster "ClusterName"
$oracleVMs = Get-VM "OracleDB-01","OracleDB-02","OracleDB-03"

New-DrsClusterGroup -Cluster $cluster -Name "Oracle-VMs" -VM $oracleVMs

# Create a Host group for licensed hosts:
$licensedHosts = Get-VMHost "esxi-01.domain.com","esxi-02.domain.com"

New-DrsClusterGroup -Cluster $cluster -Name "Oracle-Licensed-Hosts" -VMHost $licensedHosts

# Verify groups:
Get-DrsClusterGroup -Cluster $cluster | Format-Table Name, GroupType, Member -AutoSize
2

Create the VM-Host affinity rule

# PowerCLI — Create "Should run on" rule (recommended for HA safety):
New-DrsVMHostRule -Cluster $cluster -Name "Oracle-Licensing-Affinity" `
  -VMGroup "Oracle-VMs" -VMHostGroup "Oracle-Licensed-Hosts" `
  -Type ShouldRunOn -Enabled:$true

# For strict licensing compliance, use "Must run on":
# WARNING: If all hosts in the group fail, VMs will NOT be restarted by HA
New-DrsVMHostRule -Cluster $cluster -Name "Oracle-Strict-Licensing" `
  -VMGroup "Oracle-VMs" -VMHostGroup "Oracle-Licensed-Hosts" `
  -Type MustRunOn -Enabled:$true

# vSphere Client path:
# Cluster → Configure → VM/Host Rules → Add → VM-Host Affinity Rule
3

Validate HA behavior with affinity rules

# Check that HA can still protect VMs with "Must" rules:
# Ensure the host group has N+1 hosts for HA to work

# PowerCLI — Verify rule compliance:
Get-DrsVMHostRule -Cluster $cluster | Select Name, Type, Enabled,
  VMGroup, VMHostGroup | Format-Table -AutoSize

# Simulate host failure impact:
$hostGroup = Get-DrsClusterGroup -Cluster $cluster -Name "Oracle-Licensed-Hosts"
$hostCount = ($hostGroup.Member).Count
Write-Host "Licensed host count: $hostCount"
Write-Host "HA can tolerate $($hostCount - 1) host failure(s) for 'Must' rules"

# If hostCount = 1 and rule is "Must", HA CANNOT protect these VMs!
Warning: "Must run on" rules override HA failover. If all hosts in the affinity group fail simultaneously, VMs will remain powered off with no automatic recovery. Always use "Should run on" unless your licensing audit explicitly requires strict host pinning. Ensure at least N+1 hosts in every "Must" host group.
36
NUMA Optimization & CPU Scheduling
VM performance degrades when vCPUs span NUMA nodes — optimizing CPU topology for maximum throughput
Impact: VMs spanning multiple NUMA nodes experience 10-30% higher memory latency due to remote memory access. CPU-intensive workloads (databases, analytics) show measurable throughput degradation and inconsistent response times.
Symptoms
  • VM shows high %RDY and inconsistent CPU performance
  • esxtop shows VM vCPUs on different NUMA home nodes
  • Memory latency spikes in database/analytics workloads
  • VM has more vCPUs than physical cores per NUMA node
  • Performance counters show high remote NUMA memory access
Optimization Goals
  • Keep VM vCPUs within a single NUMA node
  • VM vCPU count ≤ cores per NUMA node
  • Memory allocated from local NUMA node for lowest latency
  • Avoid vNUMA misconfiguration with wrong cores-per-socket
  • Right-size VMs to minimize NUMA spanning
1

Identify NUMA topology and check VM placement

# SSH to ESXi — Check NUMA topology:
vsish -e get /hardware/numa/config
# Shows: number of NUMA nodes, cores per node, memory per node

# esxtop — Check VM NUMA placement:
# Press 'm' for memory view
# Look for NHN (NUMA Home Node) column
# VMs should ideally have all vCPUs on the same NHN

# Alternative — sched-stats:
vsish -e get /sched/groups/<vmGroupID>/stats
# Check numa.local and numa.remote memory access counters

# PowerCLI — Get host NUMA info:
Get-VMHost "esxi-01.domain.com" | Select Name,
  @{N='NumaNodes';E={$_.ExtensionData.Hardware.NumaInfo.NumNodes}},
  @{N='CoresPerNode';E={
    $_.NumCpu / $_.ExtensionData.Hardware.NumaInfo.NumNodes
  }}
2

Right-size VMs for NUMA boundaries

# Rule: VM vCPU count ≤ physical cores per NUMA node
# Example: Host with 2 NUMA nodes, 18 cores each → VM max 18 vCPUs for single-node

# Check VMs exceeding NUMA node size:
$hosts = Get-Cluster "ClusterName" | Get-VMHost
foreach ($h in $hosts) {
    $coresPerNode = $h.NumCpu / $h.ExtensionData.Hardware.NumaInfo.NumNodes
    Get-VM -Location $h | Where { $_.NumCpu -gt $coresPerNode } |
      Select Name, NumCpu, @{N='CoresPerNode';E={$coresPerNode}},
        @{N='SpansNUMA';E={"YES"}}
} | Format-Table -AutoSize

# Fix: Reduce vCPU count to fit within a single NUMA node
# Or configure cores-per-socket to match NUMA topology:
# VM → Edit Settings → CPU → Cores per Socket = CoresPerNode
3

Advanced NUMA tuning parameters

# VM Advanced Settings for NUMA optimization:
# VM → Edit Settings → VM Options → Advanced → Configuration Parameters

# Force NUMA node affinity (use with caution):
# numa.nodeAffinity = 0        # Pin to NUMA node 0

# Automatic NUMA sizing (default, recommended):
# numa.autosize = true

# Prefer Hyper-Threading for NUMA scheduling:
# numa.vcpu.preferHT = true    # Use HT siblings before spanning nodes

# vNUMA settings (VMs with >8 vCPUs auto-expose vNUMA):
# Set cores-per-socket to match physical NUMA topology
# Example: 16 vCPU VM on host with 18 cores/node
# → Set 1 socket × 16 cores (fits in 1 NUMA node)
# → NOT 2 sockets × 8 cores (forces 2 vNUMA nodes)
Best Practice: Size VMs with vCPU count ≤ cores per NUMA node. Set cores-per-socket to match the NUMA topology (prefer 1 socket with N cores over N sockets with 1 core). Let numa.autosize handle placement automatically. Only use manual NUMA pinning for extreme latency-sensitive workloads after thorough testing.
37
Memory Overcommitment & Ballooning Crisis
VMs aggressively ballooning and swapping due to host memory pressure — performance collapse under overcommit
Impact: VMs experience severe performance degradation. Guest OS sees memory being reclaimed (balloon driver inflates), forcing applications to page to disk. When host enters "Hard" or "Low" memory state, VMs swap to .vswp files — performance drops by 100-1000x compared to physical memory.
Symptoms
  • esxtop: MCTLSZ > 0 (balloon driver active, reclaiming memory)
  • esxtop: SWCUR > 0 (host swapping VM memory to .vswp file)
  • Guest OS: high page fault rates, applications OOM killed
  • VM performance drops dramatically — disk I/O spikes from swapping
  • vCenter alarm: "Host memory usage" exceeds threshold
Memory State Thresholds
  • High (>6% free): Normal — TPS active, no ballooning
  • Clear (6% free): TPS aggressive — beginning memory pressure
  • Soft (4% free): Balloon driver activated, reclaims from VMs
  • Hard (2% free): Swapping begins — .vswp files used
  • Low (1% free): Memory compression + aggressive swap — critical

ESXi is designed to overcommit memory — the total configured VM memory can exceed physical RAM. This works because VMs rarely use 100% of their allocated memory. ESXi uses a hierarchy of memory reclamation techniques: TPS (Transparent Page Sharing) → Balloon driver → Memory compression → Swapping. Under normal conditions, TPS and ballooning are invisible to VMs. But when too many VMs actively consume memory simultaneously, the host enters progressively worse memory states. Once the host hits "Hard" state, it swaps VM pages to the .vswp file on datastore — this is essentially disk-speed memory access and causes catastrophic performance loss.

1

Check host memory state and VM ballooning/swapping

# SSH to ESXi — Check memory state:
esxtop
# Press 'm' for memory view
# Key columns:
# MCTLSZ — Balloon size (MB reclaimed by balloon driver)
# SWCUR  — Current swap usage (MB swapped to .vswp)
# MEMSZ  — Configured memory size
# GRANT  — Memory granted to VM
# TCHSZ  — Touched (actively used) memory

# Check host-level memory state:
vsish -e get /memory/comprehensive
# Look for: state (high/clear/soft/hard/low)

# PowerCLI — Quick memory audit:
Get-VMHost | Select Name,
  @{N='TotalMemGB';E={[math]::Round($_.MemoryTotalGB,1)}},
  @{N='UsedMemGB';E={[math]::Round($_.MemoryUsageGB,1)}},
  @{N='FreeMemGB';E={[math]::Round($_.MemoryTotalGB - $_.MemoryUsageGB,1)}},
  @{N='FreePct';E={[math]::Round(($_.MemoryTotalGB - $_.MemoryUsageGB)/$_.MemoryTotalGB*100,1)}} |
  Format-Table -AutoSize
2

Identify over-provisioned VMs and memory waste

# PowerCLI — Find VMs with large memory that are barely using it:
Get-VM | Where { $_.PowerState -eq "PoweredOn" } | Select Name,
  @{N='ConfiguredGB';E={$_.MemoryGB}},
  @{N='HostMemUsageGB';E={
    [math]::Round(($_.ExtensionData.Summary.QuickStats.HostMemoryUsage)/1024,1)
  }},
  @{N='BalloonMB';E={$_.ExtensionData.Summary.QuickStats.BalloonedMemory}},
  @{N='SwapMB';E={$_.ExtensionData.Summary.QuickStats.SwappedMemory}} |
  Sort BalloonMB -Descending | Format-Table -AutoSize

# Find VMs actively being ballooned or swapped:
Get-VM | Where { $_.PowerState -eq "PoweredOn" } | ForEach {
    $stats = $_.ExtensionData.Summary.QuickStats
    if ($stats.BalloonedMemory -gt 0 -or $stats.SwappedMemory -gt 0) {
        [PSCustomObject]@{
            VM = $_.Name
            BalloonMB = $stats.BalloonedMemory
            SwapMB = $stats.SwappedMemory
            Host = $_.VMHost.Name
        }
    }
} | Format-Table -AutoSize
3

Resolve memory pressure

# Immediate relief:
# 1. vMotion VMs to hosts with available memory
# 2. Power off non-critical VMs
# 3. Reduce memory allocation on over-provisioned VMs

# Long-term fixes:
# Add physical memory to hosts
# Right-size VMs — reduce configured memory to actual usage + 20% headroom
# Remove unnecessary memory reservations:
Get-VM | Where { $_.ExtensionData.ResourceConfig.MemoryAllocation.Reservation -gt 0 } |
  Select Name, @{N='MemReservMB';E={
    $_.ExtensionData.ResourceConfig.MemoryAllocation.Reservation
  }} | Format-Table -AutoSize

# Enable TPS (Transparent Page Sharing) — salted since vSphere 6.0:
# Intra-VM TPS is on by default. Inter-VM TPS requires same salt:
# Host Advanced Setting: Mem.ShareForceSalting = 0  (enables inter-VM TPS)
# WARNING: Security implications — shared pages can be side-channel attacked

# Set memory limits to prevent individual VMs from consuming too much:
# Use resource pools with memory shares instead of per-VM limits
Critical Rule: If SWCUR > 0 on ANY VM in production, treat it as a P1 incident. Host-level swapping means the VM's memory pages are being served from datastore disk — this is 1000x slower than RAM. Immediately migrate VMs or add resources. Never allow production hosts to remain in "Hard" or "Low" memory state.
38
Storage DRS & Datastore Clusters
Automated storage load balancing — configuring SDRS thresholds, affinity rules, and datastore clusters
🛈 Context: Storage DRS (SDRS) automates datastore load balancing by migrating VM disk files (Storage vMotion) across datastores in a cluster. It balances both space utilization and I/O latency to prevent hot spots.
SDRS Thresholds
  • Space utilization: Default 80% — triggers recommendation when datastore exceeds this
  • I/O latency: Default 15ms — triggers recommendation for high-latency datastores
  • I/O imbalance threshold: Default 5% — minimum imbalance before SDRS acts
  • Evaluation period: Every 8 hours (configurable)
  • Automation level: No automation / Fully automated
Common Issues
  • SDRS recommendations blocked by VMDK affinity rules
  • Datastore maintenance mode stalls due to no available target
  • SDRS cannot balance RDMs or independent-persistent disks
  • Excessive Storage vMotion during peak hours degrades I/O
  • Mixed datastore sizes cause uneven space distribution
1

Create and configure a Datastore Cluster with SDRS

# vSphere Client:
# Storage view → Right-click Datacenter → New Datastore Cluster
# Enable Storage DRS → Set automation level
# Add datastores to the cluster

# PowerCLI — Check existing datastore cluster configuration:
Get-DatastoreCluster | Select Name,
  @{N='SdrsEnabled';E={$_.SdrsAutomationLevel -ne 'Disabled'}},
  @{N='AutomationLevel';E={$_.SdrsAutomationLevel}},
  @{N='SpaceThreshold';E={$_.SpaceUtilizationThresholdPercent}},
  @{N='IOLatencyThresholdMs';E={$_.IOLatencyThresholdMillisecond}},
  @{N='DatastoreCount';E={(Get-Datastore -Location $_).Count}} |
  Format-Table -AutoSize

# List datastores in the cluster with utilization:
Get-DatastoreCluster "DSCluster-01" | Get-Datastore |
  Select Name,
    @{N='CapacityGB';E={[math]::Round($_.CapacityGB,1)}},
    @{N='FreeGB';E={[math]::Round($_.FreeSpaceGB,1)}},
    @{N='UsedPct';E={[math]::Round((1-$_.FreeSpaceGB/$_.CapacityGB)*100,1)}} |
  Sort UsedPct -Descending | Format-Table -AutoSize
2

Tune SDRS thresholds and check recommendations

# PowerCLI — Adjust SDRS settings:
Set-DatastoreCluster "DSCluster-01" -SdrsAutomationLevel FullyAutomated `
  -SpaceUtilizationThresholdPercent 80 `
  -IOLatencyThresholdMillisecond 15

# Check pending SDRS recommendations:
Get-SdrsRecommendation -DatastoreCluster "DSCluster-01" |
  Format-Table -AutoSize

# vSphere Client path:
# Datastore Cluster → Configure → Storage DRS → Edit Settings
# Space: Utilized space threshold (default 80%)
# I/O: I/O latency threshold (default 15ms)
# Advanced: SDRS invocation frequency, No-Recommendation period
3

Configure SDRS affinity and anti-affinity rules

# VMDK affinity: Keep all disks of a VM on the same datastore
# (Default behavior — recommended for most workloads)

# VMDK anti-affinity: Spread disks across different datastores
# Use for: I/O-intensive VMs that benefit from parallel datastore access

# VM anti-affinity: Keep specific VMs on different datastores
# Use for: HA pairs, replicated VMs

# vSphere Client:
# Datastore Cluster → Configure → Rules → Add
# Types: VMDK Affinity, VMDK Anti-Affinity, VM Anti-Affinity

# Common fix for "SDRS recommendations not applied":
# Check if VMDK affinity rules are blocking the recommendation
# Check if VM has snapshots (SDRS cannot move VMs with snapshots)
# Check if VM disks include RDMs (not supported by SDRS)
Best Practice: Use fully automated SDRS for general-purpose datastores. Keep datastore sizes uniform within a cluster for even distribution. Set I/O latency threshold based on your storage backend (15ms for HDD, 5ms for SSD/NVMe). Schedule SDRS runs during off-peak hours using the automation schedule feature. Exclude template datastores and ISO libraries from SDRS.
39
DRS Faults & Recommendations Not Applied
DRS generates faults instead of actionable recommendations — underlying issues blocking automated balancing
Impact: DRS cannot balance the cluster. VMs remain on overloaded hosts while other hosts sit idle. Automated load management is effectively disabled despite DRS being enabled. Manual intervention required for every migration.
Symptoms
  • DRS tab shows faults instead of recommendations
  • vMotion attempts fail: "Migration to host failed"
  • DRS recommendations appear but immediately disappear or fail
  • Cluster imbalance score remains high (>20%) despite fully automated DRS
  • Events log: "DRS did not find any hosts to migrate VM"
Root Causes
  • VM-level DRS override set to "Disabled" or "Manual"
  • vMotion network misconfigured or down between hosts
  • VM disk too large for destination datastore
  • CPU compatibility issue (no EVC, different CPU generations)
  • VM has connected USB device, passthrough, or latency-sensitive config
  • Insufficient resources on all destination hosts

DRS runs every 5 minutes and generates recommendations based on cluster load balance analysis. Before recommending a migration, DRS performs a pre-check for each VM: Is vMotion feasible? Does the destination have enough resources? Are there compatibility issues? If the pre-check fails, DRS generates a fault rather than a recommendation. Common blockers include: vMotion VMkernel port group mismatch, connected devices that prevent migration (USB, PCI passthrough), disk space constraints on the target datastore, and per-VM DRS overrides that suppress automation.

1

Check DRS faults and recommendations

# PowerCLI — Get DRS recommendations and faults:
$cluster = Get-Cluster "ClusterName"
Get-DrsRecommendation -Cluster $cluster | Format-Table -AutoSize

# Check DRS events for fault details:
Get-VIEvent -Entity $cluster -MaxSamples 200 |
  Where { $_.FullFormattedMessage -match "DRS|fault|cannot.*migrat|vMotion.*fail" } |
  Select CreatedTime, FullFormattedMessage |
  Sort CreatedTime -Descending | Select -First 20 | Format-Table -AutoSize

# vSphere Client:
# Cluster → Monitor → vSphere DRS → Recommendations (check for faults)
# Cluster → Monitor → vSphere DRS → Faults (detailed fault list)
2

Check VM-level DRS overrides and migration blockers

# PowerCLI — Find VMs with DRS overrides:
$cluster = Get-Cluster "ClusterName"
$cluster.ExtensionData.Configuration.DrsVmConfig | ForEach {
    $vm = Get-VM -Id ("VirtualMachine-" + $_.Key.Value)
    [PSCustomObject]@{
        VM = $vm.Name
        DRSBehavior = $_.Behavior
        Enabled = $_.Enabled
    }
} | Format-Table -AutoSize

# Check for connected devices blocking vMotion:
Get-VM | ForEach {
    $usb = ($_.ExtensionData.Config.Hardware.Device |
      Where { $_ -is [VMware.Vim.VirtualUSB] })
    $pci = ($_.ExtensionData.Config.Hardware.Device |
      Where { $_ -is [VMware.Vim.VirtualPCIPassthrough] })
    if ($usb -or $pci) {
        [PSCustomObject]@{
            VM = $_.Name; USB = [bool]$usb; PCIPassthrough = [bool]$pci
        }
    }
} | Format-Table -AutoSize

# Verify vMotion network on all hosts:
Get-VMHost | Get-VMHostNetworkAdapter -VMKernel |
  Where { $_.VMotionEnabled } |
  Select VMHost, Name, IP, SubnetMask | Format-Table -AutoSize
3

Resolve DRS faults

# Fix 1: Remove per-VM DRS overrides:
# vSphere Client → Cluster → Configure → VM Overrides
# Remove entries with "Disabled" or "Manual" that shouldn't be there

# Fix 2: Verify vMotion connectivity between all hosts:
# Test vMotion from each host to every other host:
$hosts = Get-Cluster "ClusterName" | Get-VMHost
foreach ($src in $hosts) {
    foreach ($dst in $hosts | Where { $_ -ne $src }) {
        Write-Host "Testing vMotion: $($src.Name) → $($dst.Name)"
        # Manual test: try to migrate a small test VM
    }
}

# Fix 3: Disconnect USB devices or PCI passthrough on blocked VMs:
# VM → Edit Settings → Remove USB Controller (if not needed)

# Fix 4: Enable EVC for CPU compatibility (see KB #34)
# Fix 5: Ensure destination datastores have enough free space
# Fix 6: Check and fix vMotion VMkernel port group configuration

# After fixing, force DRS to re-evaluate:
# Right-click Cluster → Run DRS Now (vSphere 7.0+)
🛈 Best Practice: Regularly audit the DRS Faults tab — it's often overlooked. Remove VM-level DRS overrides unless documented with a business reason. Test vMotion between all host pairs quarterly. Use "Run DRS Now" (vSphere 7+) after resolving faults to immediately rebalance. Monitor DRS migration success rate in vRealize Operations or vCenter performance charts.
40
Latency Sensitivity & Performance Tuning
Configuring low-latency VM settings for trading, VoIP, and real-time applications
🛈 Context: vSphere's latency sensitivity feature optimizes the hypervisor scheduler for applications that require consistently low response times — financial trading systems, VoIP gateways, HPC workloads, and real-time control systems.
What "High" Latency Sensitivity Enables
  • Exclusive CPU scheduling: vCPUs get dedicated pCPUs (no co-scheduling)
  • Disables memory ballooning: Balloon driver will not inflate
  • Disables interrupt coalescing: Network/storage interrupts processed immediately
  • Disables memory compression: No compression overhead
  • Pass-through virtual devices: Reduced virtualization overhead for NIC/disk
Requirements & Trade-offs
  • Full memory reservation is required (no overcommit for this VM)
  • Reduces host consolidation ratio — dedicated resources locked
  • DRS migration may be less frequent due to overhead avoidance
  • VM consumes exact configured resources even if idle
  • Best combined with NUMA pinning and CPU affinity
1

Enable latency sensitivity on the VM

# vSphere Client:
# VM → Edit Settings → VM Options → Advanced → Latency Sensitivity
# Set to: "High"

# PowerCLI — Enable latency sensitivity:
$vm = Get-VM "TradingEngine-01"
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.LatencySensitivity = New-Object VMware.Vim.LatencySensitivity
$spec.LatencySensitivity.Level = "high"
$vm.ExtensionData.ReconfigVM($spec)

# Verify:
Get-VM "TradingEngine-01" | Select Name,
  @{N='LatencySensitivity';E={
    $_.ExtensionData.Config.LatencySensitivity.Level
  }}
2

Configure required memory reservation

# Latency Sensitivity = High REQUIRES full memory reservation
# VM → Edit Settings → Resources → Memory → Reservation = Max

# PowerCLI — Set full memory reservation:
$vm = Get-VM "TradingEngine-01"
$vm | Get-VMResourceConfiguration |
  Set-VMResourceConfiguration -MemReservationGB $vm.MemoryGB

# Verify reservation equals configured memory:
Get-VM "TradingEngine-01" | Select Name, MemoryGB,
  @{N='MemReservGB';E={
    [math]::Round($_.ExtensionData.ResourceConfig.MemoryAllocation.Reservation/1024,1)
  }}

# Without full reservation, the VM will fail to power on with:
# "Insufficient resources to satisfy configured failover level"
3

Additional performance tuning for low-latency VMs

# Combine with NUMA pinning for best results:
# VM → Edit Settings → VM Options → Advanced → Configuration Parameters
# numa.nodeAffinity = 0          # Pin to specific NUMA node
# sched.cpu.affinity = 0-7       # Pin to specific pCPUs (use with caution)

# Network optimization for low latency:
# Use VMXNET3 adapter (lowest virtualization overhead)
# Enable SR-IOV for near-native network performance
# Disable TCP Segmentation Offload if using UDP-heavy workloads

# Storage optimization:
# Use PVSCSI controller for lowest storage latency
# Place VM on local SSD/NVMe datastore if possible
# Enable TRIM/UNMAP for thin-provisioned disks

# Host-level tuning:
# Disable power management: esxcli system settings advanced set -o /Power/CpuPolicy -s "High Performance"
# Disable C-States in BIOS for consistent CPU frequency
# Disable Hyper-Threading if workload is CPU-cache sensitive

# Verify latency improvements with esxtop:
# Check %RDY (should be <1%), %CSTP, and %MLMTD
Warning: Latency-sensitive VMs consume dedicated resources and cannot share them with other VMs. Only enable "High" latency sensitivity for workloads that genuinely require sub-millisecond response consistency. Over-using this feature reduces your cluster's consolidation ratio and wastes resources. Benchmark before and after to confirm actual latency improvement.