VMware NSX-T KB — Phase 8: Troubleshooting & CLI

Central CLI Reference Guide

NSX Manager central-cli commands for remote troubleshooting of hosts and edges without SSH

MediumCLIcentral-cliReference

🛈Purpose: Central CLI allows running commands on remote transport nodes directly from NSX Manager without SSH access. Essential for environments where host SSH is disabled.

Resolution

Essential central-cli commands

# From NSX Manager CLI — run commands on remote hosts:

# List all transport nodes:
get transport-nodes

# Run command on specific transport node:
get transport-node <tn-uuid> interfaces
get transport-node <tn-uuid> tunnels
get transport-node <tn-uuid> logical-switches

# Edge-specific commands:
get edge-cluster <ec-uuid> status
get edge-node <en-uuid> interfaces
get edge-node <en-uuid> route-table
get edge-node <en-uuid> bgp neighbors
get edge-node <en-uuid> firewall rules

# Host-specific commands:
get host-node <hn-uuid> firewall rules
get host-node <hn-uuid> logical-switch <ls-uuid> mac-table
get host-node <hn-uuid> vtep-table

# Useful debug commands:
get transport-node <tn-uuid> tunnel-status
get transport-node <tn-uuid> controller-connection

Packet Capture & Traceflow

Capture packets and trace traffic flow through NSX-T overlay for connectivity troubleshooting

HighPacket CaptureTraceflowDebug

Resolution

NSX Traceflow (non-intrusive)

# Traceflow injects a tagged packet and reports each hop:
# NSX Manager UI → Plan & Troubleshoot → Traceflow

# API — Start traceflow:
POST https://<nsx-manager>/api/v1/traceflow
{
  "timeout": 15000,
  "packet": {
    "resource_type": "FieldsPacketData",
    "frame_size": 128,
    "ip_header": {
      "src_ip": "10.1.1.10",
      "dst_ip": "10.2.2.20",
      "protocol": 6
    },
    "transport_header": {
      "src_port": 12345,
      "dst_port": 443
    }
  },
  "source_id": "<source-port-uuid>"
}

# Get traceflow results:
GET https://<nsx-manager>/api/v1/traceflow/<traceflow-id>/observations
# Shows: Delivered, Dropped (with reason), Forwarded at each hop

Live packet capture on ESXi host

# On ESXi — capture overlay traffic:
pktcap-uw --switchport <vm-port-id> --dir 0 -o /tmp/capture.pcap
# --dir 0 = ingress, --dir 1 = egress, --dir 2 = both

# Capture on VTEP interface (encapsulated GENEVE):
pktcap-uw --uplink vmnic0 --vni <vni-id> -o /tmp/overlay.pcap

# Capture with filter:
pktcap-uw --switchport <port-id> --ip <target-ip> --proto 0x06 -o /tmp/tcp.pcap

# On Edge — packet capture:
start capture interface <interface-name> file pcap1
# Wait for traffic
stop capture interface <interface-name>
# Copy capture: copy file pcap1 url scp://user@server:/path/

NSX Intelligence Flow Analysis

Use NSX Intelligence for network flow visualization, security recommendations, and anomaly detection

MediumIntelligenceFlowVisualization

Resolution

Leverage NSX Intelligence for troubleshooting

# NSX Intelligence provides:
# 1. Real-time flow visualization
# 2. Security policy recommendations
# 3. Microsegmentation planning assistance
# 4. Suspicious traffic detection

# Access: NSX Manager → Plan & Troubleshoot → Discover & Take Action

# View flow topology:
# Shows all VM-to-VM communication flows
# Identifies: allowed flows, blocked flows, unprotected flows

# Get security recommendations:
GET https://<nsx-manager>/policy/api/v1/infra/domains/default/groups/recommendations

# Use for DFW rule planning:
# 1. Enable flow collection on target groups
# 2. Wait 7-14 days for representative traffic patterns
# 3. Review recommended rules
# 4. Apply recommended security policies

# Troubleshooting use:
# "Why can't VM-A reach VM-B?"
# → Check flow analysis for blocked/dropped flows
# → Identify which DFW rule is dropping traffic

Logical Switch Diagnostics

CLI commands for diagnosing logical switch state, VNI mapping, and port status

MediumLogical SwitchDiagnosticsCLI

Resolution

Logical switch diagnostic commands

# On ESXi host — nsxcli commands:

# List all logical switches on this host:
nsxcli -c "get logical-switches"

# Get details of specific logical switch:
nsxcli -c "get logical-switch <ls-uuid>"

# View VNI to UUID mapping:
nsxcli -c "get logical-switch vni <vni-number>"

# List ports on a logical switch:
nsxcli -c "get logical-switch <ls-uuid> ports"

# View MAC table for a switch:
nsxcli -c "get logical-switch <ls-uuid> mac-table"

# View ARP table:
nsxcli -c "get logical-switch <ls-uuid> arp-table"

# View VTEP table (which remote hosts have VMs on this switch):
nsxcli -c "get logical-switch <ls-uuid> vtep-table"

# Check replication mode:
nsxcli -c "get logical-switch <ls-uuid> replication-mode"
# Modes: MTEP (unicast), Source (head-end replication)

# Statistics:
nsxcli -c "get logical-switch <ls-uuid> stats"

Support Bundle Collection

Collect comprehensive NSX-T support bundles for VMware GSS troubleshooting

HighSupport BundleLogsGSS

Resolution

Collect support bundle via API and CLI

# Via API — collect from all nodes:
POST https://<nsx-manager>/api/v1/administration/support-bundles?action=collect
{
  "nodes": ["<manager-node-id>", "<edge-node-id>"],
  "log_age_limit": 48,
  "core_limit": 2
}

# Check collection status:
GET https://<nsx-manager>/api/v1/administration/support-bundles

# Download completed bundle:
GET https://<nsx-manager>/api/v1/administration/support-bundles/<bundle-id>/<filename>

# Via CLI on NSX Manager:
get support-bundle file support-bundle.tgz

# Via CLI on Edge:
copy support-bundle scp://admin@10.1.1.100:/tmp/edge-bundle.tgz

# On ESXi host:
vm-support -w /tmp/
# Or for NSX-specific logs:
nsxcli -c "get support-bundle file /tmp/nsx-host-bundle.tgz"

# What's included:
# - All NSX service logs
# - Configuration database dump
# - System state (CPU, memory, disk)
# - Network state (routes, interfaces, tunnels)
# - Core dumps (if any crashes)

Edge Datapath CLI Commands

Essential Edge CLI commands for debugging routing, NAT, firewall, and interface issues

HighEdge CLIDatapathDebug

Resolution

Essential Edge datapath commands

# SSH to Edge node:

# View all logical routers (VRFs) on this Edge:
get logical-routers
# Lists SR and DR instances with VRF IDs

# Enter a specific VRF:
vrf <vrf-id>

# Inside VRF — routing:
get route-table          # Full routing table
get route <ip-prefix>   # Lookup specific route
get bgp neighbors       # BGP peer status
get bgp neighbor <ip> received-routes
get bgp neighbor <ip> advertised-routes

# Interfaces:
get interfaces           # All interfaces in this VRF
get interface <name> stats  # Per-interface counters

# NAT:
get nat rules            # Active NAT rules
get nat sessions         # Active NAT translations
get nat sessions count   # Total session count
get nat statistics       # NAT hit counters

# Firewall (Gateway FW):
get firewall rules       # GFW rules on Edge
get firewall stats       # Rule hit counts

# IPSec:
get ipsec-vpn session    # VPN tunnel status
get ipsec-vpn tunnel     # Tunnel details with SAs

# Exit VRF:
exit

API Rate Limiting & Throttling

NSX-T API returning 429 errors — automation scripts hitting rate limits, blocking legitimate operations

HighAPIRate Limit429

⚠Impact: Automation tools (Terraform, Ansible, custom scripts) fail with 429 Too Many Requests. NSX Manager UI may become slow. Other API consumers blocked.

Resolution

Identify and resolve API throttling

# Check current rate limit settings:
GET https://<nsx-manager>/api/v1/node/services/http?show_sensitive_data=true
# Look for "client_api_rate_limit" and "client_api_concurrency_limit"

# Default limits:
# - 100 requests/second per client
# - 40 concurrent requests per client

# Identify which client is hitting limits:
# Check /var/log/proton/nsxapi.log for 429 responses:
get log-file nsxapi.log | grep "429\|rate_limit"

# Solutions:
# 1. Add retry logic with exponential backoff to automation:
#    Sleep: 1s → 2s → 4s → 8s on 429

# 2. Increase rate limits (temporary, during bulk operations):
PUT https://<nsx-manager>/api/v1/node/services/http
{
  "client_api_rate_limit": 200,
  "client_api_concurrency_limit": 80
}

# 3. Use batch APIs where available:
POST https://<nsx-manager>/policy/api/v1/infra
# Hierarchical API — apply many changes in single request

# 4. Distribute requests across all 3 manager nodes (not just VIP)

Proactive Health Monitoring

Set up proactive health checks and alarm monitoring for NSX-T environment

HighHealth CheckMonitoringProactive

Resolution

Essential health check APIs to monitor

# 1. NSX Manager cluster health:
GET https://<nsx-manager>/api/v1/cluster/status
# Check: "mgmt_cluster_status": "STABLE"
# All nodes: "status": "CONNECTED"

# 2. All transport node status:
GET https://<nsx-manager>/api/v1/transport-nodes/status
# UP count should equal total transport node count

# 3. Edge cluster status:
GET https://<nsx-manager>/api/v1/edge-clusters/<id>/status
# All members should be UP

# 4. Active alarms:
GET https://<nsx-manager>/api/v1/alarms?status=OPEN&severity=CRITICAL,HIGH

# 5. Controller connectivity:
GET https://<nsx-manager>/api/v1/cluster/status?source=controller
# All controllers should be CONNECTED

# 6. Certificate expiration:
GET https://<nsx-manager>/api/v1/trust-management/certificates
# Check expiry_date — alert if < 30 days

# 7. Backup status:
GET https://<nsx-manager>/api/v1/cluster/backups/history
# Verify last successful backup < 24 hours ago

# Automation: Run these checks every 5 minutes
# Alert on any non-healthy response

Host Transport Node Troubleshooting Workflow

Systematic approach to diagnosing host transport node configuration and connectivity issues

CriticalTransport NodeWorkflowTroubleshooting

🛈Purpose: Systematic workflow for diagnosing host TN issues. Follow these steps in order to quickly identify the root cause of host connectivity or preparation failures.

Resolution

Transport Node diagnostic workflow

# Step 1: Check TN status from Manager:
GET https://<nsx-manager>/api/v1/transport-nodes/<tn-id>/status
# Expected: "node_deployment_state": "success"
# Expected: "host_node_deployment_status": "INSTALL_SUCCESSFUL"

# Step 2: Check controller connectivity (on host):
nsxcli -c "get controllers"
# All controllers should show "connected: true"

# Step 3: Check tunnel status:
nsxcli -c "get tunnels"
# Should show tunnels to other hosts — status "Up"

# Step 4: Check TEP interface:
nsxcli -c "get vteps"
# TEP should have IP, link should be UP
# Verify TEP can reach other TEPs:
vmkping ++netstack=vxlan -d -s 1572 <remote-tep>

# Step 5: Check NSX agent services:
/etc/init.d/nsx-cfgagent status
/etc/init.d/nsx-proxy status

# Step 6: If agent failed — restart:
/etc/init.d/nsx-cfgagent restart
/etc/init.d/nsx-proxy restart

# Step 7: Check MPA (Message Bus) connectivity:
nsxcli -c "get node-connection status"
# Should show connected to management plane

# Step 8: If all above pass but issues persist:
# Collect support bundle and escalate to VMware GSS

NSX-T Metrics & External Monitoring

Integrate NSX-T with external monitoring (vROps, vRNI, SNMP, Syslog) for operational visibility

MediumMonitoringMetricsIntegration

Resolution

Configure external monitoring integrations

# 1. Syslog export:
PUT https://<nsx-manager>/api/v1/node/services/syslog/exporters
{
  "exporter_name": "SIEM",
  "server": "10.1.1.200",
  "port": 514,
  "protocol": "TCP",
  "level": "INFO",
  "facilities": ["DAEMON", "USER"]
}

# 2. SNMP configuration:
PUT https://<nsx-manager>/api/v1/node/services/snmp
{
  "v3_targets": [{
    "server": "10.1.1.201",
    "port": 162,
    "user": "nsxmon"
  }]
}

# 3. vRealize Network Insight (vRNI / Aria Operations for Networks):
# Add NSX Manager as data source in vRNI
# Provides: flow analysis, path visualization, intent verification

# 4. vRealize Operations (vROps / Aria Operations):
# Install NSX-T management pack
# Provides: capacity planning, alerting, dashboards

# 5. Custom API polling (Prometheus/Grafana):
# Poll key health endpoints every 60 seconds:
# /api/v1/cluster/status
# /api/v1/transport-nodes/status
# /api/v1/edge-clusters/{id}/status
# /api/v1/alarms?status=OPEN