If you require to use Certificates from a Public CA or exclusively use your own Internal CA and not leverage VMCA.
Initially I had a whole post explaining the generation of all the CSRs and Keys from the Certificate-Manager Utility but I realised the process and utility itself are not very intuitive for this process so I decided to script the process myself.
Download the hytr-certs.sh script from HERE.
DISCLAIMER: The script provided here comes with zero support from VMware or myself. It is my own utility. Use at your own risk. It can be run on a VC or PSC 6.0 Appliance.
The process is the same no matter if you are using a VCSA w/ Embedded PSC or a VCSA w/ External PSC. You can use this process on all your vSphere 6.0 Appliances.
Firstly let me explain the small lab environment I will use.
- I have a Root CA on my domain controller (dc.domain.com)
- I have an Intermediate CA (interca.domain.com)
- I have a Platform Services Controller 6.0 U2 Appliance (psc.domain.com)
- I have a vCenter Server 6.0 U2 Appliance (vc.domain.com)
Create the Certificate Templates
In this guide we are using a Microsoft Certificate Authority.
Review https://kb.vmware.com/kb/2112009 and perform the steps outlines in the sections ‘Creating a new template for vSphere 6.0 to use for Machine SSL and Solution User certificates‘ and also ‘Adding a new template to certificate templates’
Generate Certificate Signing Requests (CSR)
1. SSH to the Appliance.
- Enable the BASH shell and set it to the default shell (we’ll need that when uploading/downloading files)
shell.set --enabled True shell chsh -s /bin/bash root
Reference: https://kb.vmware.com/kb/2100508
3. Create a new directory /certs on the Appliance where we will store our certificate files for the process.
mkdir /certs cd /certs
4. Download the hytr-certs.sh script and upload this to /certs
- Ensure you have execution permissions on the script
chmod +x hytr-certs.sh
6. Run the hytr-certs.sh script from the /certs directory
It will present you with a menu
vc:/certs # ./hytr-certs.sh ~~~~~~~~~~~~~~~~~~~~~ M A I N - M E N U ~~~~~~~~~~~~~~~~~~~~~ 1. Generate CSRs for Custom Certificates 2. Install Custom Solution User Certificates 3. Generate and Install VMCA Solution User Certificates 4. Exit Enter choice [ 1 - 4]
For this task we will choose Option 1 Generate CSRs for Custom Certificates
- The script will prompt you for your FQDN (it will try auto-detect) and your Certificate Subject Information or you can accept the defaults.
+++++++++++++++++++++++++++++++++++++++++++ Define your Fully Qualified Domain Name +++++++++++++++++++++++++++++++++++++++++++ Please enter the FQDN of the VC/PSC Enter FQDN (Detected vc.domain.com): vc.domain.com You entered: vc.domain.com +++++++++++++++++++++++++++++++++++++++++++ Define your Certificate Subject Information +++++++++++++++++++++++++++++++++++++++++++ Please enter the Country Code (Default IE): US You entered: US Please enter the State (Default Munster): California You entered: California Please enter the City (Default Cork): Palo Alto You entered: Palo Alto Please enter the Organization (Default VMware): Corp You entered: Corp Please enter the Org Unit (Default GSS): Corp Unit You entered: Corp Unit ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ This script will generate CSR's and Key's for custom Certificate Replacement Ensure you have a backup and/or snapshot of this Appliance before proceeding ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Are you sure you want to proceed? [Y/N]
- When you are ready to proceed and hit Y the script will begin it’s work
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + Generating CSRs and Keys for MachineSSL and Solution Users + ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Generating a 2048 bit RSA private key ..............+++ .............................................................+++ writing new private key to 'machineSSL.key' ----- Generating a 2048 bit RSA private key .............+++ ..........................................................................................................+++ writing new private key to 'machine.key' ----- Generating a 2048 bit RSA private key .......................................+++ .......+++ writing new private key to 'vsphere-webclient.key' ----- Generating a 2048 bit RSA private key ............+++ ...........................................................+++ writing new private key to 'vpxd.key' ----- Generating a 2048 bit RSA private key .....................................................................................................+++ ...................+++ writing new private key to 'vpxd-extension.key' ----- +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + Certificate Requests and Keys for this embedded node vc.domain.com have been generated in /certs/ + +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
- Copy each of the csr files off the Appliance and provide to your Certificate Authority or follow the next section on using an Internal Microsoft CA.
Obtain the signed Certificates from a Microsoft CA
- Open a browser to your certificate authority web interface. i.e. http://interca.domain.com/certsrv
-
Select ‘Request a certificate‘ then select ‘advanced certificate request‘
-
Open the machineSSL.csr in a text editor like notepad and copy and paste the entire contents into the ‘Saved Request‘ text field and choose your vSphere6 certificate template from ‘Certificate Template‘ drop down and then hit ‘Submit‘
- Select ‘Base 64 encoded‘ and then hit ‘Download certificate chain‘. You will be prompted to download and save a certnew.p7b file.
-
Open the certnew.p7b file in Windows and drill down to the Certificates container. You should see the new Machine SSL Certificate as well as your Root MS CA and any Intermediate MS CA certificates.
- Export each of these certificates. In my lab I exported root.crt, inter.crt and MachineSSL.crt
Right-Click > All Tasks > Export Next > Base-64 encoded X.509 > File Path > Next > Finish
- Open a Command Prompt to the location you exported the above certificates. Next we need to concatenate them into a single certificate chain, with the new Machine SSL on top, followed by the Intermediate CA, followed by the Root CA.
From the command prompt use the more command to concatenate. We will concatenate into a file called machineSSL_chain.crt
more machineSSL.crt >> machineSSL_chain.crt more inter.crt >> machineSSL_chain.crt more root.crt >> machineSSL_chain.crt
Also create a chain of issuing CA certificates by concatenating the inter.crt and root.crt into a single file we will call cachain.crt
more inter.crt >> cachain.crt more root.crt >> cachain.crt
- Repeat the previous steps 1- 4 for each of the Solution User csr files – but do not choose ‘Download certificate chain’. Choose Download Certificate.
-
You should now have several files
machineSSL_chain.crt cachain.crt machine.crt vsphere-webclient.crt vpxd.crt vpxd-extension.crt
- Copy the these files up to /certs/ on the Appliance.
Note: An external PSC will not have a vpxd.crt or vpxd-extension.crt file.
Implement the signed Certificate for the Machine SSL
- Open a Putty Session open to the Appliance and launch the Certificate-Manager.
/usr/lib/vmware-vmca/bin/certificate-manager
- Select Option 1, then Option 2
-
Provide the full path to the new Machine Certificate Chain we created and then the Key and lastly the CA Chain.
Please provide valid custom certificate for Machine SSL. File : /certs/machineSSL_chain.crt Please provide valid custom key for Machine SSL. File : /certs/machineSSL.key Please provide the signing certificate of the Machine SSL certificate File : /certs/cachain.cer
- Ensure you have entered all the information correctly and also ensure you have taken a snapshot of the PSC and VC machines before answering ‘Y‘ to continue
You are going to replace Machine SSL Cert using custom certificate Continue operation : Option[Y/N] ? : Y
- The Certificate-Manager will then replace the Machine SSL certificate.
It will also update the relevant service endpoints with the new Machine SSL certificate.
Lastly it will restart all the services on the Appliance.
You are going to replace Machine SSL Cert using custom cert Continue operation : Option[Y/N] ? : Y Get site nameCompleted [Replacing Machine SSL Cert...] cork Lookup all services Get service cork:586c2bb6-9078-4bd7-8ba9-ddc411798c1b Update service cork:586c2bb6-9078-4bd7-8ba9-ddc411798c1b; spec: /tmp/svcspec_xVqq3K Get service cork:fd3833cd-d96a-454e-9ccc-ea25d0befdfc // Snip // Get service 79b06bf4-a18d-4f9e-b5a9-04affc35d2a4_com.vmware.vsan.health Don't update service 79b06bf4-a18d-4f9e-b5a9-04affc35d2a4_com.vmware.vsan.health Get service a82770ef-91ec-4e97-ba5a-b48fa3d5a371 Don't update service a82770ef-91ec-4e97-ba5a-b48fa3d5a371 Get service 7137e349-af59-4d0e-a03f-d235c9406740 Don't update service 7137e349-af59-4d0e-a03f-d235c9406740 Get service 3996a8c7-40a7-47e6-8912-bc4688dbbc59 Don't update service 3996a8c7-40a7-47e6-8912-bc4688dbbc59 Get service 29e3287a-f635-4b2f-aded-5241cbf25395 Don't update service 29e3287a-f635-4b2f-aded-5241cbf25395 Updated 28 service(s) Status : 100% Completed [All tasks completed successfully]
Note: Depending on the node type you will see a different number of updated services.
6. Now, rather than use the Certificate-Manager for the Solution User certificates I have also scripted that process within my Utility.
(The reason I didn’t script the Machine SSL is that we need to leverage the Certificate Manager Utility to update the Service Endpoint registrations)
- Jump back to the hytr-certs.sh script at /certs on the Appliance
- Run the hytr-certs.sh script from the /certs directory
vc:/certs # ./hytr-certs.sh ~~~~~~~~~~~~~~~~~~~~~ M A I N - M E N U ~~~~~~~~~~~~~~~~~~~~~ 1. Generate CSRs for Custom Certificates 2. Install Custom Solution User Certificates 3. Generate and Install VMCA Solution User Certificates 4. Exit Enter choice [ 1 - 4]
- Select Option 2. Install Custom Solution User Certificates
-
When you are ready to proceed and hit Y the script will begin it’s work
It will install the Solution User certificates into VECS and update each Solution User. It will also update EAM and AutoDeploy services with the vpxd-extension certificate if running an Appliance containing vCenter Server services.
+++++++++++++++++++++++++++++++++++++++ + Deleting machine cert entry in VECS... + +++++++++++++++++++++++++++++++++++++++ Deleted entry with alias [machine] in store [machine] successfully +++++++++++++++++++++++++++++++++++++++++ + Recreating machine cert entry in VECS... + +++++++++++++++++++++++++++++++++++++++++ Entry with alias [machine] in store [machine] was created successfully Service [machine-a004fb3d-ffd0-4156-b855-73fdbf522bf3] updated successfully +++++++++++++++++++++++++++++++++++++++++++++++++ + Deleting vsphere-webclient cert entry in VECS... + +++++++++++++++++++++++++++++++++++++++++++++++++ Deleted entry with alias [vsphere-webclient] in store [vsphere-webclient] successfully +++++++++++++++++++++++++++++++++++++++++++++++++++ + Recreating vsphere-webclient cert entry in VECS... + +++++++++++++++++++++++++++++++++++++++++++++++++++ Entry with alias [vsphere-webclient] in store [vsphere-webclient] was created successfully Service [vsphere-webclient-a004fb3d-ffd0-4156-b855-73fdbf522bf3] updated successfully +++++++++++++++++++++++++++++++++++++++ + Deleting vpxd cert entry in VECS... + +++++++++++++++++++++++++++++++++++++++ Deleted entry with alias [vpxd] in store [vpxd] successfully +++++++++++++++++++++++++++++++++++++++++ + Recreating vpxd cert entry in VECS... + +++++++++++++++++++++++++++++++++++++++++ Entry with alias [vpxd] in store [vpxd] was created successfully Service [vpxd-a004fb3d-ffd0-4156-b855-73fdbf522bf3] updated successfully +++++++++++++++++++++++++++++++++++++++++++++++++ + Deleting vpxd-extension cert entry in VECS... + +++++++++++++++++++++++++++++++++++++++++++++++++ Deleted entry with alias [vpxd-extension] in store [vpxd-extension] successfully +++++++++++++++++++++++++++++++++++++++++++++++++++ + Recreating vpxd-extension cert entry in VECS... + +++++++++++++++++++++++++++++++++++++++++++++++++++ Entry with alias [vpxd-extension] in store [vpxd-extension] was created successfully Service [vpxd-extension-a004fb3d-ffd0-4156-b855-73fdbf522bf3] updated successfully +++++++++++++++++++++++++++++++++++++++++++++++++++++ + Updating EAM with new vpxd-extension cert and key + +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2016-09-06T12:49:08.690Z Updating certificate for "com.vmware.vim.eam" extension 2016-09-06T12:49:08.856Z Successfully updated certificate for "com.vmware.vim.eam" extension 2016-09-06T12:49:08.968Z Verified login to vCenter Server using certificate="vpxd-extension.crt" is successful +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + Updating Auto Deploy with new vpxd-extension cert and key + +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2016-09-06T12:49:09.495Z Updating certificate for "com.vmware.rbd" extension 2016-09-06T12:49:09.648Z Successfully updated certificate for "com.vmware.rbd" extension 2016-09-06T12:49:09.760Z Verified login to vCenter Server using certificate="vpxd-extension.crt" is successful +++++++++++++++++++++++++++++++++++++++++++++++ + Stopping Services after cert replacement... + +++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++ + Starting services again after cert replacement... + +++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Solution User Certificates for this embedded Node vc.domain.com have been replaced with Custom Certificates +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
- That concludes the replacement of the Machine SSL and Solution Users for a vSphere 6.0 Appliance. You would perform this process on all PSC/VC/Embedded instances you wish.
have you tried reinstalling? 
Dear Féidhlim O’Leary,
Amazing document, fully explained and detailed.
But the dropbox link doesnt work. Can u upload again?
Thx a lot
LikeLike
Hi Lucas, updated the link in the article. https://www.dropbox.com/s/sbz02193mycs9ev/hytr-certs-1.0-2016-09-12.zip
Thanks
LikeLike
Hi,
I have followed your instructions but got the following error:
New MACHINE_SSL_CERT does not contain pnid in the san field
LikeLike
This was great, thank you
LikeLike
Hi there! Good document to use; using VMware documentation wasn’t intuitive, and pretty much obscured.
My question is, does these instructions differ if your CA is DoD, and they issue the certificate? We request certificates, and I believe we can download the .p7b file.
Thanks!
LikeLike
Hi – yeah DoD certificates should be fine. You can open a .p7b file (double-click on windows at least) and export the individual certs.
LikeLike
https://haveyoutriedreinstalling.com/2016/03/25/caution-solution-user-certificates-in-vsphere-6-0/#more-160
I have used your script to generate the CSR, however it was creating the same Common Name for the Solution User certificates. According to your other article, the Name should be changed to “solution User -FQDN” for the certificates to work.
echo organizationalUnitName = vpxd-$machineID >> vpxd.cfg
echo commonName = vpxd-$FQDN >> vpxd.cfg
I have one question though, why do you add the line : vpxd-$machineID and make it unique for the OU and not the commName ?
Anyways, the reason i mentioned this is that after updating the certificates using the scripts and following the instructions, SSL stopped working, and VMware TechSupport told me that we have the same CN names for all the “solution user” certificates. After reading your other article, and reviewing the script, i was able to make some changes and make it work.
Thanks,
Michael
LikeLike
Féidhlim
I cannot thank you enough for your detailed write-up. This saved me more than a day’s headache.
One followup for those only replacing the machine SSL certificate: lighttpd’s certificate is not updated automatically, which means you will see vCSA’s self-signed cert when you access the appliance management UI on port 5480. This is VMware’s acknowledgement and their simple workaround:
https://kb.vmware.com/s/article/2136693
Cheers,
David
LikeLike
Dear Féidhlim,
thank you very much for providing this useful tool.
I have been able to replace machine SSL certificates for four different vCenters during the last days, all trouble-free thanks to your efforts. Much appreciated!
LikeLike