VMware NSX-T KB — Phase 5: Edge Services

Load Balancer Health Monitor Failure

LB pool members marked DOWN — health monitors failing due to misconfigured checks or network issues

HighLoad BalancerHealth Monitor

⚠Impact: All pool members marked DOWN causes load balancer to return errors to clients. Application appears offline even though backend servers are healthy.

Resolution

Check pool member status and health monitor configuration

# Check LB pool status:
GET https://<nsx-manager>/policy/api/v1/infra/tier-1s/<t1-id>/lb-services/<lb-id>/pools/<pool-id>/status

# Check individual member health:
GET https://<nsx-manager>/policy/api/v1/infra/tier-1s/<t1-id>/lb-services/<lb-id>/pools/<pool-id>/status/detail

# Verify monitor config:
GET https://<nsx-manager>/policy/api/v1/infra/lb-monitor-profiles/<monitor-id>

# Common health monitor issues:
# 1. HTTP monitor expects 200 but app returns 301/302
# 2. Monitor URL path incorrect
# 3. TCP monitor port mismatch with actual service port
# 4. Monitor timeout too short for slow backends

# Fix: Update expected status codes
PATCH https://<nsx-manager>/policy/api/v1/infra/lb-monitor-profiles/<monitor-id>
{
  "response_status_codes": [200, 301, 302]
}

L7 Load Balancer SSL Certificate Binding

Configure SSL certificate binding for HTTPS virtual servers on NSX-T LB

HighSSLCertificateL7 LB

Resolution

Import certificate and bind to virtual server

# Import certificate:
POST https://<nsx-manager>/api/v1/trust-management/certificates?action=import
{
  "pem_encoded": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----"
}

# Create SSL profile:
PUT https://<nsx-manager>/policy/api/v1/infra/lb-client-ssl-profiles/WebApp-SSL
{
  "display_name": "WebApp-SSL",
  "default_certificate_path": "/infra/certificates/<cert-id>",
  "ssl_suite": "BALANCED"
}

# Attach to Virtual Server:
PATCH https://<nsx-manager>/policy/api/v1/infra/tier-1s/<t1-id>/lb-services/<lb-id>/virtual-servers/<vs-id>
{
  "client_ssl_profile_binding": {
    "ssl_profile_path": "/infra/lb-client-ssl-profiles/WebApp-SSL",
    "default_certificate_path": "/infra/certificates/<cert-id>"
  }
}

IPSec VPN Tunnel Flapping

IPSec VPN constantly renegotiating — IKE phase failures, proposal mismatches, DPD timeouts

CriticalIPSecVPNFlapping

⚠Impact: Site-to-site connectivity drops intermittently. Users experience random disconnections. Data replication between sites fails.

Resolution

Debug IKE negotiation failures

# Check IPSec tunnel status:
GET https://<nsx-manager>/policy/api/v1/infra/tier-0s/<t0-id>/ipsec-vpn-services/<svc-id>/sessions/<session-id>/statistics

# On Edge — check IKE status:
get ipsec-vpn session
get ipsec-vpn tunnel

# Common flapping causes:
# 1. IKE proposal mismatch (encryption/hash/DH group)
# 2. Pre-shared key mismatch
# 3. DPD (Dead Peer Detection) timeout too aggressive
# 4. NAT-T issues (UDP 4500 blocked)
# 5. Certificate expiration

# Verify proposals match remote peer:
GET https://<nsx-manager>/policy/api/v1/infra/ipsec-vpn-ike-profiles/<profile-id>
# Check: ike_version, encryption_algorithms, digest_algorithms, dh_groups

# Adjust DPD to be less aggressive:
PATCH https://<nsx-manager>/policy/api/v1/infra/ipsec-vpn-dpd-profiles/<dpd-id>
{
  "dpd_probe_interval": 60,
  "retry_count": 5
}

Edge Node CPU/Memory Exhaustion

Edge node running out of resources — services degrading, control plane disconnections

CriticalEdgeCPUMemory

⚠Impact: North-south traffic degradation or complete failure. BGP sessions drop due to keepalive misses. NAT table overflow. Edge services unresponsive.

Resolution

Identify resource consumption on Edge

# SSH to Edge node:
get edge-cluster status

# Check CPU:
get system stats
# Look for high CPU percentage

# Check memory:
get system memory
# If used memory > 90% — critical

# Identify top processes:
get processes
# Look for edge-datapath, edge-routing, nginx

# Check NAT session count (common resource hog):
get nat sessions count
# If > 500K sessions — need larger Edge or split load

# Check IPSec tunnel count
# Check number of LB connections

# Emergency: Restart specific service without full Edge reboot:
restart service <service-name>

Edge Datapath Failure

Edge datapath process crash — all traffic forwarding stops on the affected Edge node

CriticalEdgeDatapathCrash

⚠Impact: Complete north-south traffic failure for all SRs hosted on the affected Edge. HA failover should trigger to standby Edge within seconds if configured.

Resolution

Verify HA failover and recover datapath

# Check if HA failover occurred:
GET https://<nsx-manager>/api/v1/edge-clusters/<cluster-id>/status
# Look for member status — ACTIVE vs STANDBY roles may have swapped

# On Edge — check datapath status:
get datapath
# If datapath process is DOWN:

# Try restart:
restart service dataplane

# If restart fails — check logs:
get log-file syslog | grep datapath
get log-file syslog | grep "fatal\|panic\|segfault"

# If persistent crash — may need Edge redeployment
# Collect support bundle first:
copy support-bundle scp://user@server:/path/

# Force failover to partner Edge:
# This is automatic with HA — confirm standby took over

DNS Forwarder Cache Issues

DNS forwarder on T0/T1 serving stale records or failing to forward to upstream resolvers

MediumDNSForwarderCache

Resolution

Clear DNS cache and verify upstream connectivity

# Check DNS forwarder statistics:
GET https://<nsx-manager>/policy/api/v1/infra/tier-1s/<t1-id>/dns-forwarder/statistics

# Clear the DNS cache:
POST https://<nsx-manager>/policy/api/v1/infra/tier-1s/<t1-id>/dns-forwarder?action=clear_cache

# Verify upstream DNS resolver reachability from Edge:
# On Edge:
nslookup google.com <upstream-dns-ip>

# Check conditional forwarder zones:
GET https://<nsx-manager>/policy/api/v1/infra/dns-forwarder-zones

# Verify zone configuration points to correct upstream servers

DHCP Relay Configuration

Configure DHCP relay on T1 gateways for segments that use external DHCP servers

HighDHCPRelayT1 Gateway

Resolution

Configure DHCP relay on T1 gateway

# Create DHCP relay profile:
PUT https://<nsx-manager>/policy/api/v1/infra/dhcp-relay-configs/External-DHCP
{
  "display_name": "External-DHCP-Relay",
  "server_addresses": ["10.1.1.50", "10.1.1.51"]
}

# Attach to T1 segment:
PATCH https://<nsx-manager>/policy/api/v1/infra/tier-1s/<t1-id>/segments/<segment-id>
{
  "dhcp_config_path": "/infra/dhcp-relay-configs/External-DHCP"
}

# Verify DHCP is working:
# VM should get IP from external DHCP server
# Check that DHCP traffic can route from Edge to DHCP server
# Verify no DFW rules blocking UDP 67/68

Edge VM Anti-Affinity Rules

Ensure Edge cluster members run on separate physical hosts for HA

HighEdgeAnti-AffinityDRS

Resolution

Configure DRS anti-affinity for Edge VMs

# In vCenter — create DRS anti-affinity rule:
# Cluster → Configure → VM/Host Rules → Add Rule
# Type: "Separate Virtual Machines"
# Members: Edge-01, Edge-02 (all Edge cluster members)

# Verify Edge VMs are on different hosts:
# vCenter → Edge-01 → Summary → Host
# vCenter → Edge-02 → Summary → Host
# Must be different physical ESXi hosts

# If using 4-node Edge cluster:
# Create rule: Edge-01, Edge-02, Edge-03, Edge-04 must separate
# Minimum 4 ESXi hosts required in the cluster

# Also consider:
# 1. Edge VMs on dedicated Edge cluster (best practice)
# 2. Resource reservation for Edge VMs (CPU/Memory)
# 3. Latency-sensitive mode for Edge datastore (optional)

Gateway Firewall vs DFW Precedence

Understanding and configuring the interaction between Gateway Firewall (GFW) and Distributed Firewall (DFW)

MediumGFWDFWPrecedence

Resolution

Understand traffic flow through both firewalls

# Packet processing order for north-south traffic:
# 1. Packet arrives at T0 SR (Edge)
# 2. Gateway Firewall (GFW) processes on Edge — first checkpoint
# 3. Packet routed through overlay to destination host
# 4. DFW processes on destination host vNIC — second checkpoint

# Both must ALLOW for traffic to succeed
# GFW ALLOW + DFW DROP = traffic dropped at host
# GFW DROP + DFW ALLOW = traffic dropped at Edge (never reaches host)

# East-west traffic (same host):
# Only DFW applies (no Edge involvement)

# East-west traffic (different hosts):
# Only DFW applies on source and destination hosts
# GFW only applies for T0/T1 gateway-routed traffic

# Best practice:
# Use GFW for zone-level (N-S) enforcement
# Use DFW for microsegmentation (E-W) enforcement
# Avoid duplicating rules in both — creates confusion

Edge Node Redeployment

Safely redeploy a failed Edge node without disrupting active services

HighEdgeRedeployRecovery

Resolution

Redeploy Edge node procedure

# Step 1: Ensure HA is healthy — standby Edge is active
GET https://<nsx-manager>/api/v1/edge-clusters/<cluster-id>/status
# Confirm partner Edge has taken over all SRs

# Step 2: Remove failed Edge from cluster
POST https://<nsx-manager>/api/v1/edge-clusters/<cluster-id>?action=replace_transport_node
{
  "transport_node_id": "<failed-edge-tn-id>",
  "new_transport_node_id": "<new-edge-tn-id>"
}

# Alternative: Deploy fresh Edge VM
# System → Fabric → Nodes → Edge Transport Nodes → Add Edge VM
# Configure same form factor, uplinks, TEP IP pool

# Step 3: Add new Edge to cluster
# The new Edge automatically receives SR configuration
# HA rebalances SRs between the two Edge members

# Step 4: Verify health
GET https://<nsx-manager>/api/v1/transport-nodes/<new-edge-tn-id>/status