VMware NSX-T Knowledge Base

Check transport node state from NSX Manager

# NSX Manager CLI:
get transport-nodes status

# NSX API — Check specific transport node:
GET https://<nsx-manager>/api/v1/transport-nodes/<tn-id>/state

# Look for "state": "failed" or "in_progress"Copy

Verify VIB installation status on ESXi host

# On the ESXi host via SSH:
esxcli software vib list | grep -i nsx

# Expected VIBs:
# nsx-esx-datapath
# nsx-host
# nsx-vdpi
# nsx-exsi-transportnode-* (varies by version)

# Check for failed VIB install:
cat /var/log/esxupdate.log | grep -i "error\|fail"Copy

Verify NSX agent health on the host

# Check nsx-cfgagent status:
/etc/init.d/nsx-cfgagent status

# Check nsx-proxy (MPA) agent:
/etc/init.d/nsx-proxy status

# View agent logs:
tail -100 /var/log/nsx-cfgagent.log
tail -100 /var/log/nsx-proxy.logCopy

Verify ESXi host compatibility and disk space

# Check ESXi version:
esxcli system version get

# Check available disk space:
vdf -h

# Ensure /locker has at least 200MB free
# Ensure /tmp has at least 100MB freeCopy

Remove conflicting VIBs and retry preparation

# List all third-party VIBs:
esxcli software vib list | grep -v VMware | grep -v VMW

# Remove conflicting VIB (if identified):
esxcli software vib remove -n <conflicting-vib-name>

# Retry transport node preparation from NSX Manager:
# NSX API:
POST https://<nsx-manager>/api/v1/transport-nodes/<tn-id>?action=retry_prepareCopy

Restart NSX agents and verify connectivity

# Restart NSX messaging proxy agent:
/etc/init.d/nsx-proxy restart

# Restart NSX config agent:
/etc/init.d/nsx-cfgagent restart

# Verify connectivity to NSX Manager (port 1235):
nc -z <nsx-manager-ip> 1235

# Check ESXi firewall rules:
esxcli network firewall ruleset list | grep nsxCopy

Force re-realize the transport node configuration

# From NSX Manager CLI:
# Re-deploy host preparation
POST https://<nsx-manager>/api/v1/transport-nodes/<tn-id>?action=restore_cluster_config

# Monitor the state transition:
GET https://<nsx-manager>/api/v1/transport-nodes/<tn-id>/state
# Wait for "state": "success"Copy

Check gateway realization status from NSX Manager

# NSX API — Get T0 gateway status:
GET https://<nsx-manager>/api/v1/logical-routers/<t0-id>/status

# NSX Manager CLI:
get logical-routers
get logical-router <lr-id> interfaces

# Check edge allocation:
get logical-router <lr-id> service-routerCopy

Verify Edge uplink and BGP status

# On Edge node CLI:
get interfaces
# Check uplink interfaces — should show "admin up" and "link up"

get bgp neighbor summary
# Check if neighbors are in "Established" state

get route-table
# Verify default route and expected prefixes existCopy

Test connectivity from Service Router context

# Enter the T0 Service Router VRF on the Edge:
vrf <t0-sr-id>

# Ping the upstream physical gateway:
ping <upstream-gateway-ip>

# Check ARP table:
get arp

# Exit VRF:
exitCopy

Verify and fix Edge uplink connectivity

# Check Edge VM NIC status from vSphere:
# Ensure the Edge uplink port-group VLAN matches the physical switch config

# On Edge CLI — verify interface state:
get interfaces
# If interface shows "link down", check:
# 1. vSphere port-group connectivity
# 2. Physical switch port status
# 3. VLAN tagging consistency

# Reset the interface if needed:
set interface <interface-name> state down
set interface <interface-name> state upCopy

Re-establish BGP peering

# Verify BGP configuration on Edge:
get bgp neighbor summary
get bgp config

# If ASN is wrong — fix via NSX Manager UI:
# Networking → Tier-0 Gateways → Edit → BGP → Neighbors

# Clear BGP session to force re-establish:
clear bgp neighbor <neighbor-ip>

# Verify convergence:
get bgp neighbor <neighbor-ip> detailCopy

Force SR reallocation if edge is healthy but SR missing

# Check if SR is allocated:
get logical-routers
# Look for the T0 SR entry — if missing:

# From NSX API — trigger reallocation:
POST https://<nsx-manager>/api/v1/logical-routers/<t0-id>?action=reallocate

# Or force Edge cluster failover:
POST https://<nsx-manager>/api/v1/edge-clusters/<cluster-id>?action=replace_transport_nodeCopy

Check Edge cluster status and member health

# NSX Manager CLI:
get edge-cluster status

# NSX API:
GET https://<nsx-manager>/api/v1/edge-clusters/<cluster-id>/status

# On each Edge node:
get high-availability statusCopy

Verify inter-Edge BFD sessions

# On Edge node:
get bfd-sessions

# Check for "Up" status between cluster members
# If Down — check overlay/underlay connectivity between Edge nodes

# Verify internal transit VLAN between Edge members:
get interfaces | grep -i transitCopy

Check service router allocation

# From NSX Manager:
get logical-routers

# Look for SR entries and their edge-node allocation:
# If SR shows "standby" on both nodes — split-brain condition

# API check:
GET https://<nsx-manager>/api/v1/logical-routers?router_type=SERVICE_ROUTER_TIER0Copy

Fix inter-Edge connectivity for BFD

# Verify overlay TEP connectivity between Edge nodes:
# On Edge-1:
ping <Edge-2-TEP-IP> source <Edge-1-TEP-IP>

# If using VLAN-backed HA interface:
get interfaces
# Check the HA interface (typically fp-ethX) link state

# Ensure DRS anti-affinity is configured:
# vSphere → VMs → Edge-1, Edge-2 → Create Anti-Affinity RuleCopy

Disable preemptive failover if causing flapping

# NSX Manager UI:
# Networking → Tier-0 Gateways → Edit → Edge Cluster
# Set "Failover Mode" to "Non-Preemptive"

# API equivalent:
PUT https://<nsx-manager>/policy/api/v1/infra/tier-0s/<t0-id>
{
  "failover_mode": "NON_PREEMPTIVE"
}Copy

Force failover to recover from stuck state

# If Edge is in degraded state — force failover:
# NSX API — Force failover to specific member:
POST https://<nsx-manager>/api/v1/edge-clusters/<cluster-id>?action=replace_transport_node
{
  "transport_node_id": "<failed-edge-tn-id>",
  "new_transport_node_id": "<healthy-edge-tn-id>"
}

# Or from Edge CLI — restart HA:
restart service dataplaneCopy

Check DFW realization status

# NSX API — Check policy realization:
GET https://<nsx-manager>/policy/api/v1/infra/realized-state/status

# Check specific security policy:
GET https://<nsx-manager>/policy/api/v1/infra/domains/default/security-policies/<policy-id>/realized-state

# Look for "publish_status": "REALIZED" vs "ERROR"Copy

Verify DFW rules on the ESXi host

# SSH to the ESXi host, enter nsxcli:
nsxcli -c "get firewall rules"

# Check DFW filter on a specific VM vNIC:
nsxcli -c "get firewall <vm-vnic-uuid> rules"

# List all DFW filters:
summarize-dvfilter

# Check if filter is attached to the VM:
vsipioctl getfiltersCopy

Check controller connectivity from host

# On ESXi host via nsxcli:
nsxcli -c "get controllers"

# Should show controller IP with status "Connected"
# If "Disconnected" — DFW rules cannot be updated

# Check messaging channel:
nsxcli -c "get control-plane-stats"Copy

Restore controller connectivity

# Restart nsx-proxy on the host (reconnects to controllers):
/etc/init.d/nsx-proxy restart

# Verify connection re-establishes:
nsxcli -c "get controllers"
# Wait up to 60 seconds for "Connected" status

# If still disconnected — check NSX Manager cluster health:
get cluster statusCopy

Force DFW rule re-push from NSX Manager

# Re-publish the security policy:
POST https://<nsx-manager>/policy/api/v1/infra/domains/default/security-policies/<policy-id>?action=revise

# Or force full re-sync for a transport node:
POST https://<nsx-manager>/api/v1/transport-nodes/<tn-id>/state?action=resyncCopy

Restart DFW on the host if filters are missing

# WARNING: This briefly interrupts DFW enforcement on ALL VMs on this host
# Schedule during maintenance window

# Restart the firewall daemon:
/etc/init.d/nsx-cfgagent restart

# Wait for filters to reattach (monitor):
nsxcli -c "get firewall rules" | wc -l
# Should return expected rule count within 60-120 secondsCopy

Check current BFD timers and session state

# On Edge node CLI:
get bfd-sessions
get bfd-config

# Check the negotiated timer values:
# Tx Interval, Rx Interval, Detect Multiplier
# Default: 500ms interval, 3x multiplier = 1.5s detectionCopy

Increase BFD timers to reduce false positives

# NSX Manager UI:
# Networking → Tier-0 → BGP → BFD Configuration
# Set:
#   BFD Interval: 1000 (ms)
#   BFD Multiplier: 4
# This gives 4-second detection time — reduces false flaps

# API:
PUT https://<nsx-manager>/policy/api/v1/infra/tier-0s/<t0-id>
{
  "bfd_config": {
    "interval": 1000,
    "multiple": 4
  }
}Copy

Ensure QoS marking for BFD traffic on physical network

# BFD uses UDP port 3784 (single-hop) or 4784 (multi-hop)
# On physical switches — mark BFD as high priority:
# Example Cisco IOS-XE:
# class-map match-any BFD-TRAFFIC
#   match access-group name BFD-ACL
# policy-map MARK-BFD
#   class BFD-TRAFFIC
#     set dscp ef

# Verify from Edge — check for packet drops:
get dataplane-stats interface <uplink-interface>Copy

Check cluster status from each manager node

# SSH to each NSX Manager node and run:
get cluster status

# Expected output for healthy cluster:
# Cluster Id: ...
# Cluster Status: STABLE
# Node Status: UP (for all 3 nodes)

# If DEGRADED — check which node is unhealthy:
get cluster status verboseCopy

Check Corfu database health

# Check Corfu service status:
get service corfu

# Check Corfu cluster health:
get corfu-cluster status

# Look for replication lag or epoch divergence:
# /var/log/corfu/corfu.log
tail -100 /var/log/corfu/corfu.log | grep -i "error\|exception\|quorum"Copy

Verify inter-node connectivity

# From each manager, ping the other two:
ping <manager-2-ip>
ping <manager-3-ip>

# Check if cluster port (7777 for Corfu) is reachable:
nc -z <manager-2-ip> 7777
nc -z <manager-3-ip> 7777

# Check NTP synchronization:
get ntp-statusCopy

Restore network connectivity between manager nodes

# Identify the isolated node:
get cluster status
# The node with "UNKNOWN" status is likely isolated

# Verify the network path:
traceroute <isolated-manager-ip>

# Fix: Ensure manager VMs are on the same L2 network or
# that routing between them is reliable.
# Check vSphere DRS did not separate VMs to different sites.Copy

Fix NTP drift and restart services

# On each manager — fix NTP:
set ntp-server <ntp-server-ip>
get ntp-status

# If drift was the issue — restart cluster services:
restart service cluster-manager

# Wait 2-3 minutes for consensus to re-establish:
get cluster statusCopy

If split-brain persists — detach and re-join the divergent node

# CAUTION: Only perform if standard recovery fails
# Identify the minority node (the one that diverged):

# On the healthy majority (2 nodes):
detach node <divergent-node-uuid>

# Wait for cluster to stabilize with 2 nodes:
get cluster status
# Should show STABLE (2/3 nodes)

# Re-join the detached node:
join <divergent-node-ip> cluster-id <cluster-id> username adminCopy

Verify TEP interface and IP on host

# On ESXi host:
esxcli network ip interface list | grep -A5 "vmk10\|vmk50"

# Check TEP IP address:
esxcli network ip interface ipv4 address list | grep vmk10

# Verify the TEP vmkernel is on the correct VLAN:
esxcli network vswitch dvs vmware listCopy

Test TEP-to-TEP connectivity with proper MTU

# Ping remote TEP with GENEVE overhead accounted for:
vmkping ++netstack=vxlan -d -s 1572 <remote-tep-ip>
# -d = don't fragment
# -s 1572 = 1572 + 28 (IP+ICMP header) = 1600 = minimum MTU for GENEVE

# If this fails but standard ping works:
vmkping ++netstack=vxlan <remote-tep-ip>
# → MTU issue on physical networkCopy

Check tunnel status from NSX Manager

# NSX API — Get tunnel status:
GET https://<nsx-manager>/api/v1/transport-nodes/<tn-id>/tunnels

# On host via nsxcli:
nsxcli -c "get logical-switch <ls-uuid> vtep-table"
nsxcli -c "get logical-switch <ls-uuid> mac-table"Copy

Fix physical network MTU (must be ≥1600)

# GENEVE overhead = 54 bytes
# Minimum physical MTU = 1500 (inner) + 54 (GENEVE) + ~50 (safety) = 1600+
# Recommended: 9000 (jumbo frames end-to-end)

# Verify on ESXi uplink:
esxcli network vswitch dvs vmware lacp config get
esxcli network nic get -n vmnic0 | grep MTU

# Set MTU on N-VDS/VDS from vCenter:
# Networking → DVS → Configure → Properties → MTU = 9000

# On physical switch (example Cisco):
# interface range GigabitEthernet1/0/1-48
#   mtu 9216Copy

Verify TEP VLAN is trunked on physical switch

# Identify TEP VLAN from NSX Manager:
# System → Fabric → Profiles → Uplink Profiles → VLAN (Transport VLAN)

# On physical switch — ensure VLAN is trunked:
# Example Cisco:
# interface GigabitEthernet1/0/1
#   switchport trunk allowed vlan add <tep-vlan-id>

# Verify from ESXi:
esxcli network vswitch dvs vmware list
# Check the transport VLAN ID matchesCopy

Resolve duplicate TEP IP conflict

# Check TEP IP pool allocation:
GET https://<nsx-manager>/api/v1/pools/ip-pools/<pool-id>/allocations

# If duplicate found — release and reallocate:
# 1. Put host in maintenance mode
# 2. Delete transport node config
# 3. Remove the conflicting IP from pool
# 4. Re-add transport node

# Verify no ARP conflicts:
vmkping ++netstack=vxlan -I vmk10 <suspected-duplicate-ip>
# If responds from wrong MAC — duplicate confirmedCopy

Check BGP neighbor state on Edge

# On Edge node CLI:
get bgp neighbor summary

# Detailed neighbor info:
get bgp neighbor <neighbor-ip> detail

# Check BGP configuration:
get bgp config

# Verify local AS and remote AS match expectationsCopy

Verify TCP 179 reachability to BGP peer

# From Edge Service Router context:
vrf <t0-sr-id>

# Test BGP port connectivity:
# Note: nc may not be available — use ping first:
ping <bgp-peer-ip>

# Check if TCP 179 is being blocked:
get dataplane-stats interface <uplink>
# Look for dropped packets on the uplink interfaceCopy

Fix ASN and neighbor configuration

# NSX Manager UI:
# Networking → Tier-0 Gateways → <T0> → BGP
# Verify:
#   Local AS Number matches what physical router expects as "remote-as"
#   Neighbor IP is the correct interface IP of the physical router
#   Remote AS matches the physical router's actual AS number

# API — Update BGP neighbor:
PATCH https://<nsx-manager>/policy/api/v1/infra/tier-0s/<t0-id>/locale-services/<ls-id>/bgp/neighbors/<neighbor-id>
{
  "remote_as_num": "<correct-remote-asn>",
  "neighbor_address": "<peer-ip>"
}Copy

Fix MD5 authentication password

# Set password on NSX side to match physical router:
# NSX Manager UI:
# Networking → Tier-0 → BGP → Neighbors → Edit
# Password field — enter the exact same password as physical router

# On physical router (Cisco example):
# router bgp 65000
#   neighbor <edge-ip> password <same-password>

# After fixing, clear and re-establish:
# Edge CLI:
clear bgp neighbor <peer-ip>Copy

Configure eBGP multi-hop if needed

# If BGP peers are not directly connected (L3 hop between them):
# NSX Manager → BGP Neighbor → Advanced Settings
# Set "Hop Limit" to appropriate value (e.g., 2 or 3)

# On physical router:
# router bgp 65000
#   neighbor <edge-ip> ebgp-multihop 3
#   neighbor <edge-ip> update-source Loopback0Copy

Verify and restart NSX proxy agent on host

# Check agent status:
/etc/init.d/nsx-proxy status

# If not running or in bad state:
/etc/init.d/nsx-proxy restart

# Wait 30 seconds, then verify:
nsxcli -c "get controllers"
# Should show "Connected"Copy

Check firewall and network connectivity

# Verify ESXi firewall allows NSX traffic:
esxcli network firewall ruleset list | grep nsx

# Enable NSX firewall rules if disabled:
esxcli network firewall ruleset set --ruleset-id=nsx-manager --enabled=true

# Test port connectivity to all 3 NSX Manager nodes:
nc -z <nsx-mgr-1> 1235
nc -z <nsx-mgr-2> 1235
nc -z <nsx-mgr-3> 1235Copy

Verify CCP health on NSX Manager side

# On NSX Manager CLI:
get service proton
get service local-controller

# If services are not running:
restart service proton
restart service local-controller

# Check manager cluster is healthy:
get cluster statusCopy

Check and restart Upgrade Coordinator service

# On NSX Manager:
get service upgrade-coordinator

# If not running:
restart service upgrade-coordinator

# Wait 60 seconds, then verify:
get service upgrade-coordinator
# Should show "running"

# Access UC UI:
# https://<nsx-manager>/upgrade/index.htmlCopy

Check disk space and clean up if needed

# Check disk usage on NSX Manager:
df -h

# If /opt/vmware is full — clean old bundles:
ls -la /opt/vmware/nsx/upgrade-coordinator/bundles/
# Remove old/failed bundle files

# Alternatively, clean repository:
GET https://<nsx-manager>/api/v1/upgrade/bundles
# DELETE old bundles via APICopy

Fix pre-check failures before retrying

# Run pre-checks manually:
POST https://<nsx-manager>/api/v1/upgrade?action=execute_pre_upgrade_checks

# Review pre-check results:
GET https://<nsx-manager>/api/v1/upgrade/pre-upgrade-checks

# Common fixes:
# 1. Resolve all transport node failures first
# 2. Ensure all manager nodes are healthy
# 3. Take NSX backup before proceeding
# 4. Verify compute manager (vCenter) connectivityCopy

Resume a stuck upgrade

# If upgrade is stuck on a host group:
GET https://<nsx-manager>/api/v1/upgrade/status-summary

# Identify the failed unit:
GET https://<nsx-manager>/api/v1/upgrade/upgrade-unit-groups

# Skip the failed host (if non-critical) or retry:
POST https://<nsx-manager>/api/v1/upgrade?action=continue
# Or:
POST https://<nsx-manager>/api/v1/upgrade?action=retryCopy