Troubleshooting vCenter error: 503 Service unavailable failed to connect to endpoint vpxd webserver pipe

Hello vCommunity,

Hope you re doing all great, In this opportunity I am sharing with you some troubleshooting tips when encountering the 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe).

I know, this issue can mean tons of things and out there are millions of posts with different solutions,

Here are some of the things you can try when troubleshooting this issue:

 

• DNS not resolving.

For external PSC: Make sure vCenter <-> PSC can resolve to each other’s domain names, this is a common situation that does not allow the communication between the solution causing you a lot of troubles.

Try: ping from vCenter to PSC DN and vice-versa, find what is causing the communication error (wrong DNS).

 

•  Masked services

I suggest to trying the DNS not resolving issue steps first if no DNS issue is found, then try to see if the services are masked.

the mask is a stronger version of disabling. Using disable all symlinks of the specified unit file are removed. If using the mask the units will be linked to /dev/null. This will be displayed if you check e.g. by systemctl status halt.service.

Run systemctl list-unit-files | grep vmware to see if there are masked services

Example:

[email protected] [ ~ ]# systemctl list-unit-files | grep vmware  
vmware-bigsister.service               static    
vmware-cm.service                      masked    
vmware-content-library.service         masked    
vmware-eam.service                     masked    
vmware-firewall.service                enabled   
vmware-imagebuilder.service            masked    
vmware-mbcs.service                    masked    
vmware-netdump.service                 masked    
vmware-perfcharts.service              masked    
vmware-rbd-watchdog.service            masked    
vmware-rhttpproxy.service              masked    
vmware-sca.service                     masked    
vmware-sps.service                     masked    
vmware-statsmonitor.service            masked    
vmware-updatemgr.service               masked    
vmware-vapi.service                    masked    
vmware-vcha.service                    masked    
vmware-vmon.service                    masked    
vmware-vmonapi.service                 masked    
vmware-vpostgres.service               masked    
vmware-vpxd-svcs.service               masked    
vmware-vpxd.service                    masked    
vmware-vsan-health.service             masked    
vmware-vsm.service                     masked    
vmware-bigsister.timer                 disabled

You can remove all masked files from the /etc/systemd/system directory (Just #rm vmware*) or you can try this:

# List all disabled services for removal.   
find /etc/systemd/system/ -lname '/dev/null' -exec ls {} \;    
  
# Automatically remove them (or rm each file)  
find /etc/systemd/system/ -lname '/dev/null' -exec rm {} \;   
  
# Relaod systemctl daemon  
systemctl daemon-reload   
  
# Start services or Reboot  
service-control --start --all

This will be fixed on vCenter 6.5 UPDATE2 so keep you vCenter up-to-date.

• Regenerating all SSL certificates

Run the certification manger: /usr/lib/vmware-vmca/bin/certificate-manager 

Option 3 to regenerate all SSL certificates.

Follow KB How to use vSphere 6.x Certificate Manager (2097936) to do this task:

 

• Root password expired

After you set the initial password for the root user, this expires after 365 days by default, make sure to avoid this from happening,

Follow KB Change the Password and Password Expiration Settings of the Root User to change this setting.

 

• Restart all services

Yep, that simple, p

service-control –start –all

Hope this works!

• Duplicate key in PostgreSQL

Make sure to take a snapshot or have a backup before trying this.

[email protected] [/var/log]# grep -r already * | grep device_key
UTC 583a9c81.5814 127750 VCDB vc DETAIL:  Key (id, device_key)=(361, 3002) already exists.

To remove the duplicate key run the below command:

[email protected] [/var/log]# /opt/vmware/postgress/current/bin/psql -d VCDB -U postgress
VCDB=# DELETE FROM vc.vpx_vm_virtual_device where id='361' and device_key='3002';

Save and Exit \q

Hope you enjoy this!