https://cliffcahill.com

This weekend I spent upgrading our production environment within the DellEMC CPSD Vblock Logical Engineering group. We use vCenter extensively to run a lot of our in-house created automation workflows which help us logically build and configure Vblocks to our customer specifications and it also manages a Horizon View instance with some of our remote teams use during the build process.  With the inclusion of vSphere 6.5 into our RCM Trains I needed to upgrade our vCenters to ensure could manage ESXi 6.5 hosts.

I built this system initially with vSphere 5.1 and have been upgrading ever since with each release hence the reason I still have a Windows External PSC.

The Upgrade procedure is fairly straight forward and has 5 Steps

• Exporting Data
• This Step just zipped by so i missed it  
• Removing Existing product
• Installing Components
• Importing data

Before you start the upgrade do what I did and take a snapshot of your PSC VM. The upgrade process is quite disruptive and there is no rollback feature. If it goes awry you can just go back to where you started with no harm was done.

During the upgrade process, everything looked smooth enough but during step “4 of 5 Installing Component:VCServiceManager”  just as all the services were coming up I was prompted with this Fatal Error.

“An error occurred while starting service “vmonapi” (Failed to start VMware Service lifecycle manager API service)”

Not Great. Luckily enough I was able to roll back due to the snapshot and everything was fine.

So rather than retrying an in-place upgrade, I thought I would try the migration process and move to a new appliance based PSC. Unfortunately, the same error occurred during and I got a notice saying the appliance filed during first boot and when I investigated it was the same issue. Again I was able to power off the failed PSC and roll back the changes on my windows server with little impact to the VC instance.

A Quick read of the VMware vSphere 6.5 Release Notes  shows:

Upgrading to vCenter Server 6.5 may fail at vmon-api firstboot phase because of an invalid IPv6 address in the SAN field of the SSL certificate
The vCenter Server SSL certificate takes an IPv6 address in the SAN field when you install vCenter Server and enable both IPv4 and IPv6. If you disable IPv6 after the installation and then attempt to upgrade vCenter Server to version 6.5, the upgrade fails at vmon-api firstboot phase.

Workaround: Verify that the source vCenter Server SSL certificate SAN field contains the valid IP address of the source vCenter Server instance.

But I’m not using IPv6 but seems to be a good clue there. A naming mismatch on the certificate. So i went about replacing the Certificate on the PSC.

Remember to edit the drive letter that corresponds with your own individual install location.

Step1  – open the certool.cfg file. The detail should resemble what I have below.

notepad "E:\Program Files\VMware\vCenter Server\vmcad\certool.cfg"

Save the file in it same directory.

Step 2 – Generate a new private and public Key.

"E:\Program Files\VMware\vCenter Server\vmcad\certool.exe" --server localhost --genkey --privkey=E:\ssl\ssoserver.key --pubkey=E:\ssl\ssoserver.pub

Step 3 – Create a new cert using the detail that we saved in the certool.cfg file.

"E:\Program Files\VMware\vCenter Server\vmcad\certool.exe" --gencert --cert=E:\ssl\ssoserver.cer --privkey=E:\ssl\ssoserver.key --config="E:\Program Files\VMware\vCenter Server\vmcad\certool.cfg"

Step 4 – Create the P12 File. I was concerned with the warning message  highlight in the graphic but needn’t have as the rest of the process ran through without any issue.

"E:\Program Files\VMware\vCenter Server\openSSL\openssl.exe" pkcs12 -export -in E:\ssl\ssoserver.cer -inkey E:\ssl\ssoserver.key -name "ssoserver" -passout pass:changeme -out E:\ssl\ssoserver.p12

Step 5 – Backup your existing P12 cert.

"C:\ProgramData\VMware\vCenterServer\runtime\VMwareSTSService\conf\ssoserver.p12" "C:\ProgramData\VMware\vCenterServer\runtime\VMwareSTSService\conf\ssoserver.p12.backup"

Step 6 – Copy in the new  P12 cert.

copy "E:\ssl\ssoserver.p12" "C:\ProgramData\VMware\vCenterServer\runtime\VMwareSTSService\conf\ssoserver.p12"

Step 7 – Stop all PSC Services.

"E:\Program Files\VMware\vCenter Server\bin\service-control.bat" --stop --all

Step 8 – Start all PSC Services.

"E:\Program Files\VMware\vCenter Server\bin\service-control.bat" --start --all

Step 9 – Restart the vCenter Service on any vCenter instances that are registered with the PSC

Step 10 –  Restart your PSC upgrade and Success 🙂