VMware vSphere KB — Phase 5: vCenter Server & Platform Services

41

VCSA Disk Space Full — Partition Cleanup

vCenter services stop when VCSA partitions fill up — /storage/log, /storage/db, /storage/seat overflow

Critical VCSA Disk Space Partition VAMI

⚠ Impact: vCenter services refuse to start when disk partitions exceed 95%. The vSphere Client becomes inaccessible, all management operations halt, and automated workflows (DRS, HA) lose their orchestration layer.

Symptoms & Root Cause

Symptoms

vSphere Client returns 503 Service Unavailable
VAMI shows partition utilization at 95-100%
SSH login succeeds but service-control --status shows services STOPPED
Logs: No space left on device in /var/log/messages
VCSA sends email alerts: "Storage Health Alarm"

Root Causes

/storage/log — verbose logging, old log files not rotated
/storage/db — PostgreSQL WAL files, stats bloat
/storage/seat — inventory service data growth
VCSA deployed with Tiny sizing for a large environment
Core dumps accumulating in /storage/core

Diagnostic Commands

1

Identify which partitions are full

# SSH to VCSA as root:
df -h

# Check top consumers per partition:
du -sh /storage/log/* | sort -rh | head -20
du -sh /storage/db/* | sort -rh | head -10
du -sh /storage/seat/* | sort -rh | head -10

2

Check service status and identify stopped services

# List all service states:
service-control --status

# Check for core dumps consuming space:
ls -lh /storage/core/
du -sh /storage/core/

Fix Procedure

1

Clean old logs and rotate current logs

# Remove old compressed logs:
find /storage/log -name "*.gz" -mtime +7 -delete
find /storage/log -name "*.log.*" -mtime +3 -delete

# Force log rotation:
logrotate -f /etc/logrotate.conf

# Clean old core dumps:
rm -f /storage/core/core.*

2

Restart services after freeing space

# Verify space recovered:
df -h

# Start all vCenter services:
service-control --start --all

# Verify all services running:
service-control --status

3

Expand disk or resize VCSA for long-term fix

# Option A — Expand disk via VAMI:
# https://vcsa:5480 → Storage → Increase storage size

# Option B — Resize VCSA deployment size:
# Shut down VCSA → Increase CPU/RAM/Disk in vSphere
# Boot → VAMI detects new size automatically

# Option C — Move to larger VCSA size:
# Backup → Deploy new VCSA with correct sizing → Restore

🛈 Best Practice: Monitor VCSA disk usage with vCenter alarms set at 80% threshold. Configure log rotation in /etc/logrotate.d/ for all services. Use the VCSA sizing guide — Tiny handles up to 10 hosts, choose Small or Medium for production. Schedule weekly cleanup scripts via cron.

42

vpxd Service Crash & Recovery

VMware VirtualCenter Server service crashes or fails to start — DB pool exhaustion, certificate issues, or plugin conflicts

Critical vpxd Service Crash vCenter Core Recovery

⚠ Impact: vpxd is the core vCenter daemon. When it crashes, the entire vSphere Client is down, all API/SDK operations fail, DRS/HA orchestration stops, and no management tasks can be performed on the cluster.

Symptoms & Root Cause

Symptoms

vSphere Client: "Cannot connect to vCenter Server"
service-control --status vmware-vpxd shows STOPPED
vpxd.log: Panic: Unhandled exception or ODBC error
Service restarts in a loop — starts then crashes within seconds
vpxd-profiler.log shows DB connection pool exhausted

Root Causes

PostgreSQL connection pool max reached (default 60)
Expired or mismatched machine SSL certificate
Corrupted plugin extension blocking startup
Insufficient memory — vpxd OOM killed by systemd
Orphaned tasks or sessions locking DB tables

Diagnostic Commands

1

Check vpxd logs for crash reason

# Primary vpxd log:
tail -200 /var/log/vmware/vpxd/vpxd.log | grep -i "error\|panic\|abort\|fatal"

# Profiler log for performance/DB issues:
tail -100 /var/log/vmware/vpxd/vpxd-profiler.log | grep -i "pool\|connection\|timeout"

# Check if OOM killed:
journalctl -u vmware-vpxd --since "1 hour ago" | grep -i "oom\|killed"

2

Verify database connectivity

# Test PostgreSQL connection:
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "SELECT 1;"

# Check active connections:
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB \
  -c "SELECT count(*) FROM pg_stat_activity;"

Fix Procedure

1

Restart vpxd service cleanly

# Stop vpxd and its dependencies:
service-control --stop vmware-vpxd

# Wait 10 seconds, then start:
service-control --start vmware-vpxd

# Monitor startup in real-time:
tail -f /var/log/vmware/vpxd/vpxd.log

2

Increase DB connection pool if exhausted

# Edit vpxd configuration:
vi /etc/vmware-vpxd/vpxd.cfg

# Find and increase maxDbConnections:
# <maxDbConnections>80</maxDbConnections>

# Restart vpxd after change:
service-control --stop vmware-vpxd
service-control --start vmware-vpxd

3

Remove problematic plugin if blocking startup

# List registered extensions via MOB:
# https://vcsa/mob → content → ExtensionManager → extensionList

# Or via CLI — query registered extensions:
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB \
  -c "SELECT ext_key FROM vpx_ext WHERE enabled = true;"

# Disable a problematic extension:
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB \
  -c "UPDATE vpx_ext SET enabled = false WHERE ext_key = 'com.vendor.plugin';"

⚠ Caution: Never force-kill vpxd with kill -9 — this can corrupt in-flight DB transactions. Always use service-control for clean shutdown. If vpxd is in a crash loop, check certificate validity before anything else: expired machine SSL certs are the #1 cause of persistent vpxd failures.

43

vCenter Database (PostgreSQL) Troubleshooting

Embedded PostgreSQL issues — high connections, table bloat, locks, and performance degradation

High PostgreSQL VCDB Database Performance

⚠ Impact: Database issues cause slow vSphere Client responsiveness, vpxd timeouts, failed task operations, and in severe cases "connection refused" errors that make vCenter completely unresponsive.

Symptoms & Root Cause

Symptoms

vSphere Client extremely slow — pages take 30+ seconds
vpxd.log: ODBC connection timeout or connection refused
Tasks hang in "Queued" state indefinitely
/storage/db partition usage climbing rapidly
PostgreSQL log: too many connections for role "vc"

Root Causes

Connection pool saturation — max_connections exceeded
Table bloat — autovacuum not running or too slow
Long-running queries holding locks
WAL (Write-Ahead Log) files accumulating
Stats collector data growing unbounded

Diagnostic Commands

1

Connect to PostgreSQL and check connection stats

# Connect to embedded PostgreSQL:
/opt/vmware/vpostgres/current/bin/psql -U postgres VCDB

# Check active connections:
SELECT datname, usename, state, query_start, query
FROM pg_stat_activity
WHERE datname = 'VCDB'
ORDER BY query_start;

# Connection count vs max:
SELECT count(*) AS active, 
       (SELECT setting FROM pg_settings WHERE name='max_connections') AS max
FROM pg_stat_activity;

2

Check table bloat and vacuum status

# Find largest tables:
SELECT schemaname, relname, 
       pg_size_pretty(pg_total_relation_size(relid)) AS total_size,
       n_dead_tup, last_autovacuum
FROM pg_stat_user_tables
ORDER BY pg_total_relation_size(relid) DESC
LIMIT 20;

# Check WAL file accumulation:
SELECT count(*) FROM pg_ls_waldir();

Fix Procedure

1

Kill long-running queries and run vacuum

# Kill idle-in-transaction connections older than 1 hour:
SELECT pg_terminate_backend(pid) 
FROM pg_stat_activity 
WHERE state = 'idle in transaction' 
AND query_start < now() - interval '1 hour';

# Run manual vacuum on bloated tables:
VACUUM (VERBOSE, ANALYZE) vpx_event;
VACUUM (VERBOSE, ANALYZE) vpx_event_arg;
VACUUM (VERBOSE, ANALYZE) vpx_task;
VACUUM (VERBOSE, ANALYZE) vpx_task_arg;

2

Purge old events and tasks

# Check event/task counts:
SELECT count(*) FROM vpx_event;
SELECT count(*) FROM vpx_task;

# Delete events older than 30 days (adjust as needed):
DELETE FROM vpx_event_arg WHERE event_id IN 
  (SELECT event_id FROM vpx_event WHERE create_time < now() - interval '30 days');
DELETE FROM vpx_event WHERE create_time < now() - interval '30 days';

# Reclaim space after large deletes:
VACUUM FULL vpx_event;
VACUUM FULL vpx_event_arg;

3

Tune PostgreSQL settings for vCenter workload

# Key tuning parameters in postgresql.conf:
# /storage/db/vpostgres/postgresql.conf

# Increase max connections if needed:
max_connections = 100

# Tune autovacuum to be more aggressive:
autovacuum_vacuum_cost_delay = 10
autovacuum_vacuum_cost_limit = 400
autovacuum_max_workers = 4

# Restart PostgreSQL after changes:
service-control --stop vmware-vpostgres
service-control --start vmware-vpostgres

🛈 Best Practice: Set vCenter's event/task retention to 30 days (Administration → vCenter Settings → Database). Monitor /storage/db usage and set alarms at 70%. Schedule weekly VACUUM ANALYZE via cron. Never run VACUUM FULL during business hours — it locks tables exclusively.

44

SSO / STS Certificate Expiry & Renewal

STS signing certificate expires — all logins fail, API/SDK operations blocked, complete vCenter lockout

Critical SSO STS Certificate Authentication

⚠ Impact: When the STS signing certificate expires, NO user can log in to vCenter — web UI, API, SDK, PowerCLI all fail. ALL vCenter operations are blocked. This is one of the most severe vCenter outage scenarios.

Symptoms & Root Cause

Symptoms

vSphere Client login: "Unable to authenticate" or "A general system error occurred"
PowerCLI: Connect-VIServer returns 401 Unauthorized
REST API: com.vmware.vapi.std.errors.unauthenticated
SSO log: Token is not valid: expired certificate
All services appear running but authentication fails

Root Causes

STS signing certificate expired (default 2-year validity)
Machine SSL certificate expired
VMCA root certificate expired (10-year validity)
Clock skew between VCSA and ESXi hosts > 5 minutes
vmdir replication failure causing stale cert data

Diagnostic Commands

1

Check STS certificate expiry date

# Check all trusted certificates:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list --login administrator@vsphere.local

# For each cert ID, check expiry:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id <cert-id> --outcert /tmp/cert.pem
openssl x509 -in /tmp/cert.pem -noout -dates

# Quick check — STS signing cert in VMDIR:
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store STS_INTERNAL_SSL_CERT

2

Check machine SSL and solution user certificates

# Machine SSL cert expiry:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT \
  | openssl x509 -noout -dates

# List all certificate stores:
/usr/lib/vmware-vmafd/bin/vecs-cli store list

# Check each store for expired certs:
for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do
  echo "=== $store ==="
  /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store "$store" 2>/dev/null
done

Fix Procedure

1

Renew STS signing certificate using fixsts script

# VMware KB 79248 — fix expired STS certificate:
# Download fixsts.sh from VMware KB or use built-in:

# Option A — Use the fixsts.sh script:
chmod +x /tmp/fixsts.sh
/tmp/fixsts.sh

# Option B — Use certificate-manager (interactive):
/usr/lib/vmware-vmca/bin/certificate-manager
# Select Option 8: Reset all certificates

# After renewal, restart all services:
service-control --stop --all
service-control --start --all

2

Verify certificate renewal and test login

# Verify new STS cert dates:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list --login administrator@vsphere.local

# Test authentication:
curl -k -X POST "https://localhost/rest/com/vmware/cis/session" \
  -u "administrator@vsphere.local:PASSWORD"

# Check all services healthy:
service-control --status

# Verify from vSphere Client login page

3

Fix clock skew if present

# Check VCSA time:
date
timedatectl status

# Configure NTP on VCSA:
# VAMI → Time → Edit NTP servers

# Or via CLI:
/usr/lib/vmware-applmgmt/bin/timesync.py set --mode NTP
/usr/lib/vmware-applmgmt/bin/timesync.py set --servers "ntp1.example.com ntp2.example.com"

⚠ Caution: Always take a VCSA snapshot/backup before renewing certificates. STS cert renewal affects ALL connected vCenters in Enhanced Linked Mode. After renewal, ESXi hosts may need to be reconnected. Monitor STS cert expiry proactively — set calendar reminders 30 days before expiry.

45

Content Library Sync Failures

Subscribed library not syncing from publisher — HTTPS, proxy, certificate trust, and bandwidth issues

High Content Library Sync Templates ISO

⚠ Impact: VM templates and ISO images are not available at remote sites. New VM deployments fail if they depend on subscribed library content. Patching workflows using Content Library for update staging are blocked.

Symptoms & Root Cause

Symptoms

Subscribed library shows "Last sync: Failed"
Items show as "Not synchronized" with red icons
vCenter events: Content library sync session failed
Sync starts but hangs at 0% or fails mid-transfer
Error: SSL certificate problem: unable to get local issuer certificate

Resolution Targets

Verify HTTPS connectivity between subscriber and publisher
Update SSL thumbprint after publisher cert renewal
Configure proxy settings if required
Check firewall rules for port 443 between sites
Validate publisher URL is accessible and correct

Diagnostic Commands

1

Test connectivity from subscriber vCenter to publisher URL

# SSH to subscriber VCSA — test HTTPS connectivity:
curl -vk https://publisher-vcsa.example.com:443/cls/vcsp/lib/<library-id>/lib.json

# Check DNS resolution:
nslookup publisher-vcsa.example.com

# Test port connectivity:
python3 -c "import socket; s=socket.create_connection(('publisher-vcsa.example.com', 443), timeout=5); print('OK'); s.close()"

2

Verify publisher SSL certificate thumbprint

# Get current publisher certificate thumbprint:
echo | openssl s_client -connect publisher-vcsa.example.com:443 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256

# Compare with thumbprint configured in subscribed library:
# vSphere Client → Content Libraries → Select subscribed library → Settings

# Check Content Library logs:
tail -100 /var/log/vmware/content-library/cls.log | grep -i "error\|fail\|ssl"

Fix Procedure

1

Update SSL thumbprint and resync

# In vSphere Client:
# Content Libraries → Select subscribed library → Actions → Edit Settings
# Click "Probe" to fetch new publisher cert → Accept thumbprint
# Click "Sync Now"

# Or via REST API:
curl -k -X POST "https://vcsa/rest/com/vmware/content/subscribed-library/id:<lib-id>?~action=sync" \
  -H "vmware-api-session-id: <session-id>"

2

Configure proxy settings if needed

# Set proxy for VCSA outbound connections:
# VAMI (https://vcsa:5480) → Networking → Proxy Settings

# Or via CLI:
/opt/vmware/share/vami/vami_set_proxy https proxy.example.com 8080

# Verify proxy is applied:
/opt/vmware/share/vami/vami_get_proxy

# Add no-proxy exceptions for internal publisher:
# No proxy: localhost,127.0.0.1,publisher-vcsa.example.com

3

Recreate library if persistent corruption

# If sync continues to fail after all fixes:
# 1. Delete the broken subscribed library
# 2. Re-create it with the correct publisher URL
# 3. Accept the SSL thumbprint
# 4. Set sync schedule (immediate or on-demand)

# Verify sync completes:
# Content Libraries → Select library → check "Last Successful Sync" timestamp

🛈 Best Practice: Use "on-demand" sync for large libraries to save bandwidth — items are downloaded only when needed. Set automatic sync schedules during off-peak hours. After publisher certificate renewal, always update the thumbprint on all subscriber libraries. Monitor Content Library datastore free space to avoid sync failures due to insufficient storage.

46

vCenter Convergence — External PSC to Embedded

Migrate from deprecated external PSC topology to embedded — simplified single-appliance architecture

High PSC Convergence Migration Topology

⚠ Impact: External PSC topology is deprecated since vSphere 7.0. Convergence to embedded is required for future upgrades and eliminates a single point of failure. The process is non-disruptive when performed correctly.

Pre-Checks & Planning

Prerequisites

vCenter Server 6.7 Update 1+ or 7.0+
All PSC replication partners healthy and in sync
All vCenter services running on both PSC and VC
Fresh file-based backup of VCSA and PSC
Downtime window: ~30 minutes per vCenter

Benefits

Single appliance to manage instead of PSC + VC
Eliminates PSC as a single point of failure
Required for vSphere 8.x upgrade path
Simplified certificate and backup management
Reduced resource consumption (decommission PSC VMs)

Diagnostic — Verify Replication Health

1

Verify vmdir replication status across all nodes

# Check replication status:
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartners \
  -h localhost -u administrator -w 'PASSWORD'

# Check replication agreements:
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showservers \
  -h localhost -u administrator -w 'PASSWORD'

# Verify no replication lag:
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus \
  -h localhost -u administrator -w 'PASSWORD'

2

Verify all services are healthy before convergence

# On the vCenter:
service-control --status

# On the external PSC:
service-control --status

# Verify VCHA (if configured) is disabled before convergence:
# vSphere Client → vCenter HA → Edit → Turn Off vCenter HA

Fix Procedure — Convergence

1

Run the convergence tool on vCenter

# SSH to the vCenter Server (not the PSC):
# Run the convergence CLI tool:
/usr/lib/vmware-vmca/bin/cdc-converge

# The tool will:
# 1. Validate current topology
# 2. Move SSO/vmdir data into the embedded vCenter
# 3. Reconfigure endpoints
# 4. Restart all services

# Monitor progress — convergence takes 15-30 minutes

2

Verify convergence success and decommission PSC

# After convergence — verify topology:
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showservers \
  -h localhost -u administrator -w 'PASSWORD'
# Should show only vCenter nodes, no external PSC

# Verify all services running on vCenter:
service-control --status

# Test login to vSphere Client
# Test SSO, permissions, roles

# Decommission external PSC:
# 1. Take final backup of PSC
# 2. Power off PSC VM
# 3. After 7 days of stable operation, delete PSC VM

🛈 Best Practice: Always converge one vCenter at a time in Enhanced Linked Mode environments. Take snapshots of both PSC and VC before starting. Update DNS records after convergence if the PSC had a separate FQDN used by other systems. After convergence, verify that all third-party integrations (SRM, NSX, vROps) still connect properly.

47

VAMI Backup Schedule & Automated Recovery

Configure automated VCSA backups via VAMI or API — schedule, retention, protocols, and restore procedures

High VAMI Backup Restore DR

✓ Objective: Establish a reliable backup and recovery plan for vCenter Server Appliance. File-based backups capture the VCSA configuration, inventory, and database needed for full recovery.

Backup Options & Protocols

Supported Protocols

FTP — Basic FTP transfer (not recommended)
FTPS — FTP over SSL/TLS
HTTP/HTTPS — Web server target
SCP — Secure copy over SSH (recommended)
SMB — Windows file share (CIFS)

Backup Contents

Common — Config, inventory, tasks, events (small, fast)
Seat data — Stats, alarms, tags (medium size)
Full — Everything including historical performance data (largest)
Backup size: ~2-10 GB depending on inventory
Duration: 15-60 minutes depending on data size

Configure Automated Backup

1

Configure backup schedule via VAMI

# Access VAMI:
# https://vcsa:5480 → Backup → Configure Backup Schedule

# Settings:
# Protocol: SCP (recommended)
# Server: backup-server.example.com
# Path: /backups/vcsa/
# Username: backup-user
# Schedule: Daily at 02:00
# Retention: Keep last 5 backups
# Backup type: Full (with stats)
# Encryption: Enable with strong passphrase

2

Configure backup via REST API (for automation)

# Create a backup schedule via API:
curl -k -X POST "https://vcsa/api/appliance/recovery/backup/schedules" \
  -H "vmware-api-session-id: $SESSION" \
  -H "Content-Type: application/json" \
  -d '{
    "schedule_id": "daily-backup",
    "spec": {
      "location": "scp://backup-server.example.com/backups/vcsa",
      "location_user": "backup-user",
      "location_password": "PASSWORD",
      "enable": true,
      "recurrence_info": {
        "minute": 0, "hour": 2,
        "days": ["MONDAY","TUESDAY","WEDNESDAY","THURSDAY","FRIDAY","SATURDAY","SUNDAY"]
      },
      "retention_info": { "max_count": 5 },
      "backup_password": "ENCRYPTION_PASSPHRASE"
    }
  }'

3

Trigger manual backup and verify

# Trigger on-demand backup via VAMI:
# https://vcsa:5480 → Backup → Backup Now

# Or via API:
curl -k -X POST "https://vcsa/api/appliance/recovery/backup/jobs" \
  -H "vmware-api-session-id: $SESSION" \
  -H "Content-Type: application/json" \
  -d '{
    "piece": { "location": "scp://backup-server.example.com/backups/vcsa",
               "location_user": "backup-user",
               "location_password": "PASSWORD",
               "backup_password": "ENCRYPTION_PASSPHRASE",
               "parts": ["common","seat"] }
  }'

# Monitor backup job:
curl -k "https://vcsa/api/appliance/recovery/backup/jobs" \
  -H "vmware-api-session-id: $SESSION"

Restore Procedure

1

Restore VCSA from file-based backup

# Full restore process:
# 1. Mount VCSA ISO on a workstation with network access
# 2. Launch the VCSA Installer
# 3. Select "Restore" instead of "Install"
# 4. Stage 1: Deploy new VCSA appliance (fresh OVA)
# 5. Stage 2: Select "Restore from file-based backup"
#    - Protocol: SCP/FTP/etc.
#    - Backup location: scp://backup-server/backups/vcsa/
#    - Select the backup to restore
#    - Enter encryption passphrase
# 6. Wait for restore to complete (30-90 minutes)
# 7. Verify vSphere Client access and all services

🛈 Best Practice: Test restore procedures quarterly in a lab environment. Use SCP for backups — it's the most reliable protocol. Enable backup encryption always — backups contain sensitive credential data. Store the encryption passphrase in a secure vault (not on the vCenter itself). Keep at least 5 daily backups and 2 weekly backups. Monitor backup job success via vCenter alarms or API polling.

48

vCenter Plugin & UI Extension Issues

Plugins causing slow UI, errors, blank pages — troubleshooting and cleanup via MOB and logs

Medium Plugins UI MOB Extensions

⚠ Impact: Broken or outdated plugins can cause the vSphere Client to load slowly, display blank sections, throw JavaScript errors, or even prevent login. Common offenders include SRM, NSX, vSAN health, and vROps plugins after version upgrades.

Symptoms & Root Cause

Symptoms

vSphere Client takes 60+ seconds to load
Blank white sections or missing menu items
Browser console: JavaScript errors from plugin modules
Error banner: "An internal error has occurred"
Specific plugin pages throw 500 errors

Root Causes

Plugin version incompatible with current vCenter version
Plugin's backend server decommissioned but registration remains
Failed plugin upgrade left orphaned extension data
Plugin server SSL certificate expired or untrusted
Multiple versions of same plugin registered simultaneously

Diagnostic Commands

1

Check vSphere Client UI logs for plugin errors

# vSphere Client (HTML5) logs:
tail -200 /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log \
  | grep -i "error\|exception\|plugin\|extension"

# Check plugin loading times:
grep -i "plugin.*load\|extension.*register" \
  /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log | tail -30

# Check for connectivity issues to plugin backends:
grep -i "connect.*refused\|timeout\|unreachable" \
  /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log | tail -20

2

List all registered extensions

# Via MOB (Managed Object Browser):
# https://vcsa/mob → content → ExtensionManager → extensionList
# Review each extension's key, description, and server URL

# Via PowerCLI:
Connect-VIServer vcsa.example.com
$ext = Get-View ExtensionManager
$ext.ExtensionList | Select-Object Key, 
  @{N='Company';E={$_.Description.Label}}, 
  @{N='Version';E={$_.Version}} | Format-Table -AutoSize

Fix Procedure

1

Unregister broken plugin via MOB

# 1. Navigate to MOB:
#    https://vcsa/mob → content → ExtensionManager
# 2. Click "UnregisterExtension" method
# 3. Enter the extension key (e.g., "com.vmware.vrops.install")
# 4. Click "Invoke Method"

# Common plugin keys to check:
# com.vmware.vrops.install — vROps
# com.vmware.vcHms — NSX
# com.vmware.vcDr — SRM
# com.vmware.vsan.health — vSAN Health

2

Clear vSphere Client cache and restart

# Stop the vSphere Client service:
service-control --stop vsphere-ui

# Clear the client cache:
rm -rf /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/*
rm -rf /etc/vmware/vsphere-ui/cm-init-status

# Restart vSphere Client:
service-control --start vsphere-ui

# Monitor startup (takes 2-5 minutes):
tail -f /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log

3

Re-register plugin with correct version

# After fixing the plugin backend (upgrade, cert renewal, etc.):
# Re-register from the product's management interface:

# For NSX: NSX Manager → vCenter Registration → Re-register
# For SRM: SRM appliance → Reconfigure → Register vCenter plugin
# For vROps: vROps → Administration → vCenter Adapter → Re-register

# Verify in MOB that the new version is registered:
# https://vcsa/mob → content → ExtensionManager → extensionList

🛈 Best Practice: Before upgrading vCenter, check VMware's Interoperability Matrix to verify all plugins are compatible with the target version. After vCenter upgrade, immediately check for plugin errors. Keep a list of all registered extensions and their expected versions. Decommission products properly — always unregister the plugin before removing the backend server.

49

Lookup Service & Component Registration

Services fail to register with Lookup Service — stale endpoints, missing solution users, broken service discovery

High Lookup Service Registration SSO Endpoints

⚠ Impact: The Lookup Service is the backbone of vCenter component discovery. When registrations are broken, services can't find each other — vSphere Client may not connect to inventory, ESXi hosts may appear disconnected, and third-party tools lose connectivity.

Symptoms & Root Cause

Symptoms

vSphere Client shows "Cannot connect to one or more vCenter Server systems"
ESXi hosts appear disconnected intermittently
Service startup fails: Failed to connect to lookup service
Third-party products: "Unable to find vCenter endpoint"
Enhanced Linked Mode between vCenters is broken

Root Causes

Stale Lookup Service endpoints from decommissioned components
Solution user certificates expired or mismatched
vmdir replication failure causing inconsistent endpoint data
IP/FQDN change without re-registering endpoints
Lookup Service database corruption

Diagnostic Commands

1

List all Lookup Service registrations

# Use lstool to list all registered endpoints:
python /usr/lib/vmware-lookupsvc/tools/lstool.py list \
  --url https://localhost/lookupservice/sdk \
  --no-check-cert

# Filter for specific service types:
python /usr/lib/vmware-lookupsvc/tools/lstool.py list \
  --url https://localhost/lookupservice/sdk \
  --no-check-cert --type vcenterserver

# Check for stale entries (services that no longer exist):
python /usr/lib/vmware-lookupsvc/tools/lstool.py list \
  --url https://localhost/lookupservice/sdk \
  --no-check-cert 2>&1 | grep -i "serviceUrl\|nodeId"

2

Check solution user status and certificates

# List solution users:
/usr/lib/vmware-vmafd/bin/dir-cli service list \
  --login administrator@vsphere.local

# Check specific solution user certificate:
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension

# Verify Lookup Service is healthy:
service-control --status vmware-lookupsvc
tail -50 /var/log/vmware/lookupsvc/lookupsvc.log | grep -i "error\|fail"

Fix Procedure

1

Remove stale Lookup Service entries

# Unregister a stale service endpoint:
python /usr/lib/vmware-lookupsvc/tools/lstool.py unregister \
  --url https://localhost/lookupservice/sdk \
  --no-check-cert \
  --id "<service-id-from-list>" \
  --user "administrator@vsphere.local" \
  --password "PASSWORD"

# After cleanup, restart affected services:
service-control --stop --all
service-control --start --all

2

Re-register solution users

# Re-register vpxd solution user:
/usr/lib/vmware-vpxd/bin/vpxd_servicecfg fix

# Or use certificate-manager to refresh solution user certs:
/usr/lib/vmware-vmca/bin/certificate-manager
# Option 6: Replace solution user certificates with VMCA certificates

# Restart all services after re-registration:
service-control --stop --all
service-control --start --all

# Verify endpoints are registered:
python /usr/lib/vmware-lookupsvc/tools/lstool.py list \
  --url https://localhost/lookupservice/sdk --no-check-cert

3

Fix Enhanced Linked Mode registration

# If ELM is broken between vCenters:
# Verify vmdir replication is working:
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus \
  -h localhost -u administrator -w 'PASSWORD'

# Re-point the Lookup Service if FQDN changed:
cmsso-util repoint --repoint-psc psc.example.com \
  --username administrator@vsphere.local \
  --passwd 'PASSWORD'

# Verify cross-vCenter connectivity:
python /usr/lib/vmware-lookupsvc/tools/lstool.py list \
  --url https://remote-vcsa/lookupservice/sdk --no-check-cert

🛈 Best Practice: After any vCenter FQDN/IP change, always re-register Lookup Service endpoints. Keep a documented list of all Lookup Service registrations for audit purposes. In multi-vCenter environments, verify vmdir replication health weekly. Before decommissioning any VMware product, always unregister it from the Lookup Service first.

50

vCenter Performance & Sizing Guidelines

VCSA sizing recommendations, performance tuning, and capacity monitoring for optimal vCenter operations

Medium Sizing Performance Tuning Capacity

🛈 Objective: Right-size the VCSA deployment and tune vpxd parameters to ensure responsive vSphere Client performance, efficient task processing, and stable operations as your environment scales.

VCSA Sizing Matrix

Deployment Sizes

Tiny: 2 vCPU, 12 GB RAM — up to 10 hosts, 100 VMs
Small: 4 vCPU, 19 GB RAM — up to 100 hosts, 1,000 VMs
Medium: 8 vCPU, 28 GB RAM — up to 400 hosts, 4,000 VMs
Large: 16 vCPU, 37 GB RAM — up to 2,000 hosts, 35,000 VMs
X-Large: 24 vCPU, 56 GB RAM — up to 2,500 hosts, 45,000 VMs

Storage Requirements

Tiny: 415 GB (default) — 1,700 GB (large stats)
Small: 480 GB (default) — 1,765 GB (large stats)
Medium: 700 GB (default) — 1,985 GB (large stats)
Large: 1,065 GB (default) — 2,350 GB (large stats)
X-Large: 1,295 GB (default) — 2,580 GB (large stats)

Performance Monitoring

1

Monitor vpxd task and session metrics

# Check active vpxd tasks:
/opt/vmware/vpostgres/current/bin/psql -U postgres VCDB \
  -c "SELECT count(*) AS active_tasks FROM vpx_task WHERE state = 'running';"

# Check concurrent sessions:
/opt/vmware/vpostgres/current/bin/psql -U postgres VCDB \
  -c "SELECT count(*) AS active_sessions FROM vpx_session WHERE logged_out IS NULL;"

# Monitor vpxd CPU and memory usage:
top -bn1 | grep vpxd
ps aux | grep vpxd | grep -v grep

2

Check inventory scale and growth

# Count managed objects:
/opt/vmware/vpostgres/current/bin/psql -U postgres VCDB -c "
  SELECT 'Hosts' AS type, count(*) FROM vpx_host
  UNION ALL
  SELECT 'VMs', count(*) FROM vpx_vm
  UNION ALL
  SELECT 'Clusters', count(*) FROM vpx_compute_resource WHERE type='CLUSTER'
  UNION ALL
  SELECT 'Datastores', count(*) FROM vpx_datastore;
"

# Check VCSA resource utilization via VAMI:
# https://vcsa:5480 → Monitor → CPU, Memory, Network, Storage

Tuning Recommendations

1

Tune vpxd configuration for large environments

# Edit vpxd.cfg:
vi /etc/vmware-vpxd/vpxd.cfg

# Key tuning parameters:
# <maxQueryResult>
#   Default: 1000 — increase to 5000 for large inventories
# </maxQueryResult>

# config.vpxd.stats.maxQueryMetrics
#   Default: 64 — increase for heavy monitoring tools
#   Set via Advanced Settings in vSphere Client:
#   vCenter → Configure → Advanced Settings → Add:
#   config.vpxd.stats.maxQueryMetrics = 256

# Restart vpxd after changes:
service-control --stop vmware-vpxd
service-control --start vmware-vpxd

2

Optimize task and event retention

# Reduce event/task retention to lower DB load:
# vSphere Client → vCenter → Configure → General → Database
# Task retention: 30 days (default)
# Event retention: 30 days (default)

# For large environments, consider:
# Task retention: 15 days
# Event retention: 15 days

# Disable stats levels not needed:
# vSphere Client → vCenter → Configure → General → Statistics
# Level 1 (Basic) for intervals > 1 day
# Level 2 (Detailed) only for 5-minute interval

3

Scale up VCSA when approaching limits

# Check if current sizing is sufficient:
# VAMI → Monitor → check CPU/RAM sustained utilization
# If CPU > 80% sustained or RAM > 85% — scale up

# Scale-up procedure:
# 1. Take backup via VAMI
# 2. Shut down VCSA gracefully
# 3. Edit VM settings — increase vCPU and RAM to next size tier
# 4. Power on VCSA
# 5. VAMI will detect new resources automatically

# Note: Cannot downsize — only scale up
# Note: Storage can be expanded while VCSA is running (VAMI → Storage)

🛈 Best Practice: Always deploy one size larger than your current needs to accommodate growth. Monitor vCenter performance weekly using VAMI dashboards. Keep maxQueryMetrics at -1 (unlimited) only if absolutely required by monitoring tools — it significantly increases DB load. Review stats collection levels annually and disable unnecessary detail levels. For environments with >500 hosts, consider distributed vCenter architectures with Enhanced Linked Mode.