VMware NSX-T Knowledge Base

Phase 3 — Distributed Firewall⚠ 6 Break✓ 4 Fix
10
Total Issues
6
Break Scenarios
4
Fix Procedures
Phase 3
Current Phase
21
DFW Rule Hit Count Analysis
Analyze DFW rule statistics to identify unused rules and optimize policy
🛈Purpose: Identify DFW rules with zero hit counts for cleanup. Reduces rule table size, improves DFW performance, and simplifies security posture management.
1

Retrieve DFW rule statistics

# NSX API — Get rule statistics for a section:
GET https://<nsx-manager>/api/v1/firewall/sections/<section-id>/rules/stats

# Policy API equivalent:
GET https://<nsx-manager>/policy/api/v1/infra/domains/default/security-policies/<policy-id>/statistics

# On host — check per-rule hit count:
nsxcli -c "get firewall rules" | grep -E "rule_id|hit_count"

# Look for rules with hit_count = 0 over 30+ days
2

Identify and mark unused rules for removal

# Export all rules with stats:
GET https://<nsx-manager>/policy/api/v1/infra/domains/default/security-policies

# Filter for rules with zero packets:
# Rules with packet_count: 0 and byte_count: 0
# Consider time period — rules may be seasonal

# Best practice: Disable before deleting
# Set rule to "disabled" for 7 days
# If no impact → safe to delete

PATCH https://<nsx-manager>/policy/api/v1/infra/domains/default/security-policies/<policy-id>/rules/<rule-id>
{
  "disabled": true
}
22
DFW Microsegmentation Design Patterns
Implement effective microsegmentation using NSX tags, groups, and tiered security policies
1

Design policy hierarchy (Emergency → Infrastructure → Environment → Application)

# NSX DFW policy processing order (top-down):
# Category 1: Emergency — block known threats, quarantine
# Category 2: Infrastructure — shared services (DNS, NTP, AD, backup)
# Category 3: Environment — zone isolation (Prod/Dev/DMZ)
# Category 4: Application — app-tier microsegmentation

# Create infrastructure allow policy:
POST https://<nsx-manager>/policy/api/v1/infra/domains/default/security-policies/Infra-Allow
{
  "display_name": "Infrastructure-Services",
  "category": "Infrastructure",
  "rules": [
    {
      "display_name": "Allow-DNS",
      "source_groups": ["ANY"],
      "destination_groups": ["/infra/domains/default/groups/DNS-Servers"],
      "services": ["/infra/services/DNS-UDP", "/infra/services/DNS-TCP"],
      "action": "ALLOW"
    }
  ]
}
2

Create tag-based dynamic groups

# Create group based on NSX tags:
PUT https://<nsx-manager>/policy/api/v1/infra/domains/default/groups/Web-Tier
{
  "display_name": "Web-Tier",
  "expression": [
    {
      "resource_type": "Condition",
      "member_type": "VirtualMachine",
      "key": "Tag",
      "operator": "EQUALS",
      "value": "web-tier|app-name"
    }
  ]
}

# Tag VMs for automatic group membership:
# NSX Manager UI → Inventory → Virtual Machines → Select VM → Tags → Add
# Or via API:
POST https://<nsx-manager>/api/v1/fabric/virtual-machines/<vm-id>/tags
{
  "tags": [{"scope": "app-name", "tag": "web-tier"}]
}
23
DFW Exclusion List Not Working
VMs on DFW exclusion list still have firewall rules applied — filter not removed
Impact: VMs that should bypass DFW (e.g., NSX Manager, vCenter, physical appliances) may have traffic blocked. Critical infrastructure VMs may lose connectivity.
Symptoms
  • VM added to exclusion list but DFW rules still enforce
  • Packet captures show DFW drops for excluded VM traffic
  • vsipioctl getfilters still shows filter on excluded VM vNIC
  • Exclusion list shows VM but realized state is pending
Root Causes
  • VM was powered off when added to exclusion (filter not removed until power cycle)
  • Transport node lost controller connectivity — cannot receive exclusion update
  • VM has multiple vNICs — only one was processed
  • NSX Manager realized state lag — configuration not pushed yet
1

Verify exclusion list membership

# Check current exclusion list:
GET https://<nsx-manager>/policy/api/v1/infra/settings/firewall/security/exclude-list

# Add VM to exclusion list if missing:
PUT https://<nsx-manager>/policy/api/v1/infra/settings/firewall/security/exclude-list
{
  "members": [
    {"path_expression": "/infra/domains/default/groups/DFW-Exclude-Group"}
  ]
}

# Verify the group contains the VM:
GET https://<nsx-manager>/policy/api/v1/infra/domains/default/groups/DFW-Exclude-Group/members/virtual-machines
2

Force filter removal on host

# On ESXi host — check if filter is still attached:
vsipioctl getfilters

# Check specific VM's filter:
nsxcli -c "get firewall <vm-vnic-uuid> rules"
# If rules still present — the exclusion hasn't been applied

# Power cycle the VM (vMotion will NOT trigger re-apply)
# Or restart nsx-cfgagent:
/etc/init.d/nsx-cfgagent restart

# After restart — verify filter is removed:
vsipioctl getfilters | grep <vm-name>
# Should show no DFW filter for excluded VM
24
Rule Publish Failure
DFW security policy fails to publish — realized state shows error, rules not pushed to hosts
Impact: New security rules are not enforced. Policy changes saved in intent but not realized on the data plane. Security posture is inconsistent between intended and actual state.
1

Check realized state and identify errors

# Check overall DFW realization:
GET https://<nsx-manager>/policy/api/v1/infra/realized-state/realized-entities?intent_path=/infra/domains/default/security-policies/<policy-id>

# Check specific alarms:
GET https://<nsx-manager>/api/v1/alarms?feature_name=firewall

# On host — check for partial rules:
nsxcli -c "get firewall rules" | wc -l
# Compare with expected rule count from NSX Manager
2

Resolve common publish failures

# Error: "Group member resolution failed"
# → Group references a deleted or invalid object
# Fix: Edit the group, remove invalid members

# Error: "Service not found"
# → Rule references a deleted service definition
# Fix: Update rule to use valid service

# Error: "Maximum rule limit exceeded"
# → Host DFW table full (max ~100K rules per host)
# Fix: Consolidate rules, use broader groups

# Force re-publish:
POST https://<nsx-manager>/policy/api/v1/infra?enforce_revision_check=false
# Re-save the policy with no changes to trigger re-push
25
Identity-Based Firewall (IDFW) Issues
IDFW rules not matching user sessions — AD event log integration, Guest Introspection issues
Impact: User-based DFW rules cannot enforce policy. Users from restricted AD groups access resources they shouldn't. Identity-to-IP mapping is stale or missing.
1

Verify IDFW configuration and AD event source

# Check IDFW is enabled:
GET https://<nsx-manager>/policy/api/v1/infra/settings/firewall/idfw/cluster/<cluster-id>

# Verify AD event log scraper:
GET https://<nsx-manager>/api/v1/idfw/idfw-compute-collections

# Check Active Directory event source:
# NSX UI → Security → Distributed Firewall → IDFW → AD Event Sources
# Verify connection to domain controllers

# On host — check Guest Introspection service:
nsxcli -c "get ids status"
2

Verify user-to-IP mapping

# Check current IDFW user sessions:
GET https://<nsx-manager>/api/v1/idfw/user-session-data

# Check if specific user is mapped:
GET https://<nsx-manager>/api/v1/idfw/user-session-data?user_name=<username>

# If mapping is stale — user logged in but mapping shows old IP:
# Verify Guest Introspection VM is deployed on the cluster
# Verify NSX Tools (VMware Tools with thin agent) are installed in guest VMs

# Force refresh:
POST https://<nsx-manager>/api/v1/idfw/compute-collections/<cc-id>?action=refresh
26
Time-Based Rules Not Triggering
Scheduled DFW rules don't activate or deactivate at configured times
1

Verify NTP synchronization across all components

# On NSX Manager:
get ntp-status
# Verify synchronized with NTP source

# On ESXi hosts:
esxcli system time get
ntpq -p

# If time is skewed — fix NTP:
# On NSX Manager:
set ntp-server <ntp-server-ip>

# Time-based rules use the NSX Manager's timezone
# Verify timezone setting:
get timezone
2

Verify time window configuration

# Check time window definitions:
GET https://<nsx-manager>/policy/api/v1/infra/domains/default/security-policies/<policy-id>/rules/<rule-id>

# Look for "time_window" field in the rule
# Verify start_time, end_time, days_of_week

# Common issues:
# 1. Timezone mismatch (UTC vs local)
# 2. Days of week not including current day
# 3. Rule was modified — time window removed accidentally
27
DFW Performance Degradation
DFW causing latency or packet drops — excessive rules, connection table exhaustion, CPU pressure on hosts
Impact: VM-to-VM latency increases. Packet drops under high connection rates. Application timeouts. May be mistaken for network or storage issues.
1

Check DFW connection table utilization

# On ESXi host:
nsxcli -c "get firewall connection-stats"

# Check connection table size vs limit:
nsxcli -c "get firewall thresholds"
# Default: 65536 connections per vNIC

# If connection count approaches limit:
# Option 1: Increase connection limit
# Option 2: Reduce connection timeout for idle sessions

# Check per-VM connection stats:
nsxcli -c "get firewall <vnic-uuid> connection-stats"
2

Optimize DFW rule table

# Check total rule count on host:
nsxcli -c "get firewall rules" | wc -l

# Best practices for performance:
# 1. Use "Applied To" to limit rule scope (don't apply to all VMs)
# 2. Consolidate overlapping rules
# 3. Use IP Sets instead of individual IPs in rules
# 4. Move inactive rules to disabled state
# 5. Use security policy categories efficiently

# Monitor DFW CPU impact:
# esxtop → press 'n' for network → check %DRPS column
28
Security Group Membership Stale
Dynamic security group membership not updating — VMs not added or removed correctly
Impact: New VMs not protected by DFW rules (not in correct group). Decommissioned VMs still consuming rule entries. Security posture inconsistent with intent.
1

Verify group membership and criteria

# Check effective members of a group:
GET https://<nsx-manager>/policy/api/v1/infra/domains/default/groups/<group-id>/members/virtual-machines

# Check group criteria (what should match):
GET https://<nsx-manager>/policy/api/v1/infra/domains/default/groups/<group-id>

# Verify VM tags match the criteria:
GET https://<nsx-manager>/api/v1/fabric/virtual-machines?display_name=<vm-name>
# Check "tags" field matches group expression

# Force group membership recalculation:
POST https://<nsx-manager>/policy/api/v1/infra/domains/default/groups/<group-id>?action=recompute
2

Fix tag assignment for dynamic groups

# Add missing tag to VM:
PUT https://<nsx-manager>/api/v1/fabric/virtual-machines/<vm-ext-id>/tags
{
  "tags": [
    {"scope": "environment", "tag": "production"},
    {"scope": "app-tier", "tag": "web"}
  ]
}

# Verify group membership updates within 60 seconds:
GET https://<nsx-manager>/policy/api/v1/infra/domains/default/groups/<group-id>/members/virtual-machines

# Check consolidated effective members:
GET https://<nsx-manager>/policy/api/v1/infra/domains/default/groups/<group-id>/members/consolidated-effective-ip-addresses
29
Applied To Scope Best Practices
Optimize DFW performance using "Applied To" to limit rule distribution scope
🛈Purpose: "Applied To" determines which hosts receive the DFW rules. By default, rules are pushed to ALL hosts. Using Applied To reduces rule table size on hosts that don't need those rules.
1

Configure Applied To at the policy level

# Set Applied To on a security policy:
PATCH https://<nsx-manager>/policy/api/v1/infra/domains/default/security-policies/<policy-id>
{
  "scope": ["/infra/domains/default/groups/Web-Servers"]
}

# This means the rules in this policy are ONLY pushed to hosts
# where members of "Web-Servers" group reside

# Best practices:
# 1. Set Applied To = target group for application-specific policies
# 2. Leave Infrastructure policies as "DFW" (all hosts) since they're universal
# 3. Use per-rule Applied To for mixed policies

# NSX Manager UI:
# Security → DFW → Policy → Applied To column → Click to edit
2

Verify rule distribution after Applied To change

# Check which hosts have the rules:
# On a host that SHOULD have the rules:
nsxcli -c "get firewall rules" | grep <rule-id-keyword>
# Should find the rule

# On a host that should NOT have the rules:
nsxcli -c "get firewall rules" | grep <rule-id-keyword>
# Should return empty — rule not distributed here

# This reduces total rules per host → better performance
30
DFW Rule Export/Import
Export DFW configuration for backup, migration, or replication to another NSX environment
1

Export DFW configuration via API

# Export all security policies:
GET https://<nsx-manager>/policy/api/v1/infra/domains/default/security-policies

# Export specific policy with all rules:
GET https://<nsx-manager>/policy/api/v1/infra/domains/default/security-policies/<policy-id>/rules

# Export all groups (needed for rule references):
GET https://<nsx-manager>/policy/api/v1/infra/domains/default/groups

# Export all services:
GET https://<nsx-manager>/policy/api/v1/infra/services

# Save all responses as JSON files for backup
2

Import DFW configuration to target environment

# Import order matters:
# 1. Create Services first
# 2. Create Groups (modify member criteria for new environment)
# 3. Create Security Policies with rules

# Import groups:
PUT https://<target-nsx>/policy/api/v1/infra/domains/default/groups/<group-id>
{...exported group JSON...}

# Import policy with rules:
PUT https://<target-nsx>/policy/api/v1/infra/domains/default/security-policies/<policy-id>
{...exported policy JSON with rules...}

# Verify realization:
GET https://<target-nsx>/policy/api/v1/infra/realized-state/status