VMware NSX-T Knowledge Base

Phase 2 — Day-2 Operations⚠ 1 Break✓ 9 Fix
10
Total Issues
1
Break Scenarios
9
Fix Procedures
Phase 2
Current Phase
11
Certificate Rotation & Expiry
NSX certificates approaching expiry — proactive rotation procedures for API, node, and cluster certificates
Impact: Expired certificates cause API failures, management plane disconnection, and inter-node communication breakdowns. Hosts may lose controller connectivity if cluster certificates expire.
Symptoms
  • NSX Manager UI shows certificate expiry warning banner
  • API calls fail with SSL/TLS handshake error
  • Inter-node cluster communication breaks after cert expiry
  • Transport nodes report "Certificate validation failed"
  • Alarms: "Certificate is about to expire"
Root Causes
  • Default self-signed certificates have 825-day validity (not renewed)
  • No certificate monitoring or alerting configured
  • CA-signed certificates not renewed before expiry date
  • Certificate rotation procedure not part of operational runbook
1

Check certificate expiry dates

# NSX Manager CLI:
get certificate-chain

# NSX API — List all certificates:
GET https://<nsx-manager>/api/v1/trust-management/certificates

# Check specific certificate expiry:
GET https://<nsx-manager>/api/v1/trust-management/certificates/<cert-id>
# Look for "not_after" field
2

Generate new self-signed certificate

# Generate CSR for NSX Manager API certificate:
POST https://<nsx-manager>/api/v1/trust-management/csrs
{
  "subject": {
    "attributes": [
      {"key": "CN", "value": "nsx-manager.example.com"},
      {"key": "O", "value": "MyOrg"},
      {"key": "OU", "value": "NSX"}
    ]
  },
  "key_size": "2048",
  "algorithm": "RSA"
}

# Self-sign if no CA available:
POST https://<nsx-manager>/api/v1/trust-management/csrs/<csr-id>?action=self_sign
{
  "days_valid": 825
}
3

Apply new certificate to NSX Manager

# Replace the API certificate:
POST https://<nsx-manager>/api/v1/trust-management/certificates/<new-cert-id>?action=apply_certificate
{
  "service_type": "API",
  "node_id": "<node-id>"
}

# Verify the new certificate is active:
GET https://<nsx-manager>/api/v1/trust-management/certificates?type=API

# Repeat for each manager node in the cluster
# Verify certificate validity:
get certificate-chain
# Should show new expiry date (825 days from now)

# Test API connectivity:
GET https://<nsx-manager>/api/v1/cluster/status
# Should return 200 OK without TLS errors

# Verify all nodes use new certificate:
GET https://<nsx-manager>/api/v1/trust-management/certificates?type=API
12
Backup & Restore Procedures
NSX Manager backup configuration, validation, and restore procedures for disaster recovery
🛈Impact: Without valid backups, NSX configuration loss requires full rebuild. Restore allows recovery of DFW policies, routing config, and all manager state. Critical before any upgrade or maintenance.
1

Configure backup target (SFTP server)

# NSX API — Configure backup:
PUT https://<nsx-manager>/api/v1/cluster/backups/config
{
  "backup_enabled": true,
  "backup_schedule": {
    "resource_type": "WeeklyBackupSchedule",
    "days_of_week": ["MONDAY", "THURSDAY"],
    "hour": 2,
    "minute": 0
  },
  "remote_file_server": {
    "server": "<sftp-server-ip>",
    "port": 22,
    "protocol": {"name": "sftp"},
    "directory_path": "/nsx-backups",
    "authentication_scheme": {
      "scheme_name": "password",
      "username": "backup-user",
      "password": "<password>"
    }
  },
  "passphrase": "<encryption-passphrase>"
}
2

Trigger manual backup before maintenance

# NSX API — Initiate backup now:
POST https://<nsx-manager>/api/v1/cluster?action=backup_to_remote

# Check backup status:
GET https://<nsx-manager>/api/v1/cluster/backups/status

# List available backups:
GET https://<nsx-manager>/api/v1/cluster/backups/history
3

Restore from backup (disaster recovery)

# IMPORTANT: Restore requires a fresh NSX Manager deployment
# 1. Deploy new NSX Manager appliance (same version as backup)
# 2. During initial setup, choose "Restore from Backup"

# Or via API on fresh manager:
POST https://<nsx-manager>/api/v1/cluster/restore
{
  "remote_file_server": {
    "server": "<sftp-server-ip>",
    "port": 22,
    "protocol": {"name": "sftp"},
    "directory_path": "/nsx-backups",
    "authentication_scheme": {
      "scheme_name": "password",
      "username": "backup-user",
      "password": "<password>"
    }
  },
  "passphrase": "<encryption-passphrase>",
  "backup_id": "<backup-id>"
}

# Monitor restore progress:
GET https://<nsx-manager>/api/v1/cluster/restore/status
13
N-VDS to VDS Migration
Migrate host switching from NSX-managed N-VDS to vSphere Distributed Switch (VDS 7.0+)
Impact: Migration consolidates switching management under vCenter. Required for VCF 4.x+ deployments. Incorrect migration can disconnect hosts from overlay. Plan maintenance window.
1

Pre-checks before migration

# Verify VDS version supports NSX (7.0+):
# vCenter → Networking → DVS → Summary → Version

# Check current N-VDS configuration:
GET https://<nsx-manager>/api/v1/transport-nodes/<tn-id>
# Note the N-VDS UUID, uplink mapping, and transport zone

# Ensure host is on vSphere 7.0+ with VDS 7.0+
# Ensure NSX-T 3.0+ (N-VDS to VDS migration support)

# Verify no active DFW or routing issues on the host
2

Perform the migration via NSX Manager

# NSX Manager UI:
# System → Fabric → Nodes → Host Transport Nodes
# Select host → Actions → "Migrate to VDS"

# API equivalent:
POST https://<nsx-manager>/api/v1/transport-nodes/<tn-id>?action=migrate_to_vds
{
  "vds_name": "Production-VDS",
  "vmware_distributed_switch_uuid": "<vds-uuid>"
}

# Monitor migration status:
GET https://<nsx-manager>/api/v1/transport-nodes/<tn-id>/state
3

Verify post-migration connectivity

# On host — verify TEP is now on VDS:
esxcli network vswitch dvs vmware list

# Verify overlay connectivity:
vmkping ++netstack=vxlan -d -s 1572 <remote-tep-ip>

# Verify VMs are connected and communicating:
nsxcli -c "get logical-switch <ls-uuid> vtep-table"

# Check transport node status:
GET https://<nsx-manager>/api/v1/transport-nodes/<tn-id>/state
# Should show "success"
14
Edge Node Replacement
Replace a failed or degraded Edge transport node with a new deployment
🛈Impact: Replacing a failed Edge node restores cluster redundancy. During replacement, services failover to remaining members. Ensure at least one healthy Edge remains active.
1

Remove failed Edge from cluster

# Identify the failed Edge node:
get edge-cluster status

# Remove failed member from Edge cluster:
POST https://<nsx-manager>/api/v1/edge-clusters/<cluster-id>?action=replace_transport_node
{
  "transport_node_id": "<failed-edge-tn-id>",
  "new_transport_node_id": "<new-edge-tn-id>"
}

# Or remove without immediate replacement:
PUT https://<nsx-manager>/api/v1/edge-clusters/<cluster-id>
# Remove the failed member from the members array
2

Deploy new Edge node

# Deploy via NSX Manager UI:
# System → Fabric → Nodes → Edge Transport Nodes → Add Edge VM
# Configure: Name, Size (Large recommended for production),
#   Management IP, Compute Manager, Cluster, Datastore
#   Transport Zone assignments, Uplink profile, IP Pool

# Wait for deployment to complete (~10-15 minutes)
# Verify new Edge is "Success":
GET https://<nsx-manager>/api/v1/transport-nodes/<new-edge-tn-id>/state
3

Add new Edge to cluster and verify

# Add to Edge cluster:
PUT https://<nsx-manager>/api/v1/edge-clusters/<cluster-id>
{
  "members": [
    {"transport_node_id": "<existing-edge-tn-id>"},
    {"transport_node_id": "<new-edge-tn-id>"}
  ]
}

# Verify cluster health:
get edge-cluster status

# Verify SR allocation rebalances:
get logical-routers
15
Transport Zone Configuration Drift
Transport zone membership or configuration becomes inconsistent across hosts — causes segment isolation
Impact: Hosts in different transport zones cannot exchange overlay traffic. VMs on mismatched hosts are isolated even on the same logical segment. Segment spans only hosts in the correct TZ.
1

Audit transport zone membership

# List all transport zones:
GET https://<nsx-manager>/api/v1/transport-zones

# Check which hosts are in each TZ:
GET https://<nsx-manager>/api/v1/transport-zones/<tz-id>/transport-node-status

# Identify hosts NOT in the expected transport zone:
# Compare against expected list from design document
2

Fix host transport zone assignment

# Update transport node to include correct TZ:
PUT https://<nsx-manager>/api/v1/transport-nodes/<tn-id>
{
  "transport_zone_endpoints": [
    {"transport_zone_id": "<overlay-tz-id>"},
    {"transport_zone_id": "<vlan-tz-id>"}
  ]
  // ... keep other config the same
}

# Verify state after update:
GET https://<nsx-manager>/api/v1/transport-nodes/<tn-id>/state
16
LDAP/AD Identity Source Issues
Configure and troubleshoot Active Directory / LDAP integration for NSX role-based access
🛈Impact: AD/LDAP users cannot log into NSX Manager. Role-based access control (RBAC) non-functional. Only local admin account works.
1

Configure LDAP identity source

# NSX API — Add LDAP identity source:
POST https://<nsx-manager>/api/v1/node/aaa/auth-policy
# Check current auth policy first

# Add LDAP via UI:
# System → Settings → User Management → LDAP
# Type: Active Directory over LDAP
# Domain: example.com
# Base DN: DC=example,DC=com
# Bind DN: CN=nsx-bind,OU=Service Accounts,DC=example,DC=com
# LDAP Server: ldaps://dc01.example.com:636

# Test connectivity:
POST https://<nsx-manager>/api/v1/aaa/ldap-identity-sources?action=probe
2

Troubleshoot LDAP connectivity

# From NSX Manager CLI — test LDAP reachability:
ping <ldap-server-ip>

# Test LDAPS port:
# openssl s_client -connect <ldap-server>:636

# Check NSX Manager logs for LDAP errors:
tail -100 /var/log/proton/nsxapi.log | grep -i ldap

# Common issues:
# 1. CA certificate not imported (for LDAPS)
# 2. Bind DN incorrect format
# 3. Base DN too narrow (missing OUs)
# 4. Firewall blocking port 636/389
17
Alarm & Event Correlation
Configure alarm thresholds, suppress false positives, and correlate events for effective monitoring
1

View and manage NSX alarms

# List all active alarms:
GET https://<nsx-manager>/api/v1/alarms

# Filter by severity:
GET https://<nsx-manager>/api/v1/alarms?severity=CRITICAL

# Suppress a specific alarm type:
POST https://<nsx-manager>/api/v1/alarms?action=suppress_alarm
{
  "alarm_id": "<alarm-id>",
  "suppress_duration": 3600
}

# Resolve an alarm:
POST https://<nsx-manager>/api/v1/alarms/<alarm-id>?action=resolve
2

Configure alarm thresholds

# NSX Manager UI:
# System → Settings → Alarm Definitions
# Adjust thresholds for:
#   - Edge CPU Utilization (default 80%, adjust to 90%)
#   - Transport Node Tunnel Down
#   - Certificate Expiry Warning (30 days before)

# Example: Modify edge CPU alarm threshold via API
# This prevents false alarms during legitimate traffic spikes
18
Syslog & Log Collection Configuration
Configure remote syslog forwarding for NSX Manager, Edge nodes, and host transport nodes
1

Configure syslog on NSX Manager

# NSX Manager CLI:
set logging-server <syslog-ip> proto udp level info facility local0

# Verify:
get logging-servers

# API equivalent:
POST https://<nsx-manager>/api/v1/node/services/syslog/exporters
{
  "server": "<syslog-ip>",
  "port": 514,
  "protocol": "UDP",
  "level": "INFO",
  "facilities": ["KERN", "USER", "LOCAL0"]
}
2

Configure syslog on Edge nodes

# On Edge node CLI:
set logging-server <syslog-ip> proto udp level info

# For all edges via API:
POST https://<nsx-manager>/api/v1/transport-nodes/<edge-tn-id>/node/services/syslog/exporters
{
  "server": "<syslog-ip>",
  "port": 514,
  "protocol": "UDP",
  "level": "INFO"
}

# Verify on Edge:
get logging-servers
19
MP/CCP Cluster Certificate Regeneration
Regenerate management plane and central control plane cluster certificates after expiry or corruption
Impact: Expired cluster certificates break inter-node communication. CCP certificates affect host-to-controller connectivity. All transport nodes may show "Controller Disconnected".
1

Identify expired certificates

# Check all certificate expiry:
GET https://<nsx-manager>/api/v1/trust-management/certificates

# Check cluster certificate specifically:
get certificate-chain cluster

# Look for certificates with "not_after" in the past
# Categories: API, CLUSTER, MP-CLUSTER, CCP
2

Regenerate cluster certificates

# Regenerate MP cluster certificate (self-signed):
POST https://<nsx-manager>/api/v1/trust-management/certificates?action=generate_self_signed
{
  "cert_type": "CLUSTER",
  "key_size": "2048",
  "algorithm": "RSA",
  "subject": {
    "attributes": [{"key":"CN","value":"NSX-MP-Cluster"}]
  },
  "days_valid": 825
}

# Apply the new cluster certificate:
POST https://<nsx-manager>/api/v1/trust-management/certificates/<new-cert-id>?action=apply_certificate
{
  "service_type": "CLUSTER_CERTIFICATE"
}

# This will restart inter-node communication
# Wait 2-3 minutes for cluster to stabilize
3

Re-establish CCP trust with transport nodes

# After cluster cert change, hosts need to re-trust the new cert
# This usually happens automatically via nsx-proxy reconnection

# If hosts remain disconnected:
# On each ESXi host:
/etc/init.d/nsx-proxy restart

# Verify controller connectivity:
nsxcli -c "get controllers"

# Force certificate re-sync (if needed):
POST https://<nsx-manager>/api/v1/transport-nodes/<tn-id>?action=resync
20
NSX Manager Node Decommission
Safely remove an NSX Manager node from the cluster for replacement or maintenance
Impact: Running on 2 nodes temporarily reduces cluster fault tolerance. If another node fails during this window, the cluster loses quorum. Complete the procedure promptly.
1

Verify cluster health before decommission

# Ensure all 3 nodes are healthy:
get cluster status
# Must show STABLE with all nodes UP

# Take a backup before proceeding:
POST https://<nsx-manager>/api/v1/cluster?action=backup_to_remote

# Identify the node to remove:
GET https://<nsx-manager>/api/v1/cluster/nodes
# Note the node UUID to decommission
2

Detach the node from cluster

# Gracefully remove the node:
POST https://<nsx-manager>/api/v1/cluster/<node-id>?action=remove_node

# Or from the node being removed:
detach node <self-node-id>

# Monitor cluster status (should become STABLE with 2 nodes):
get cluster status

# Wait for data rebalancing to complete (~5 minutes)
3

Deploy replacement and join cluster

# Deploy new NSX Manager appliance (same version)
# After deployment, join to existing cluster:

# On the new node:
join <existing-manager-ip> cluster-id <cluster-id> username admin password <password>

# Verify 3-node cluster is restored:
get cluster status
# Should show STABLE with 3 nodes UP

# Power off and delete the old manager VM from vCenter