In vSphere 6.0 the VMCA (VMware Certificate Authority) was introduced. This is basically vSphere’s own CA and it’s purpose is to simplify certificate generation and implementation in vSphere, in conjunction with VECS (VMware Endpoint Certificate Store)
While I do agree it does simplify the whole process, it’s not without its limitations and known issues. Hopefully this guide will help you avoid those pitfalls.
Firstly let me explain the small lab environment I will use.
- I have a Root CA on my domain controller (dc.domain.com)
- I have an Intermediate CA (interca.domain.com)
- I have a Platform Services Controller 6.0 U2 Appliance (psc.domain.com)
- I have a vCenter Server 6.0 U2 Appliance (vc.domain.com)
Create the Certificate Templates
In this guide we are using a Microsoft Certificate Authority.
Review https://kb.vmware.com/kb/2112009 and perform the steps outlines in the sections ‘Creating a new template for vSphere 6.0 to use for VMCA as a Subordinate CA’ and also ‘Adding a new template to certificate templates’
Generate VMCA Certificate Signing Request (CSR)
SSH to the Platform Services Controller (or vCenter Server if using VC w/ Embedded PSC)
Enable the BASH shell and set it to the default shell (we’ll need that when uploading the new certificate files)
shell.set --enabled True shell chsh -s /bin/bash root
1. Launch the certificate-manager utility from /usr/lib/vmware-vmca/bin/certificate-manager
2. Select Option 2. Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates.
3. The certificate-manager will ask a question about generating all certificates using configuration files. We will answer Y to this question
4. Provide the email@example.com credentials
5. Next we will be asked to configure the configuration file for MACHINE_SSL_CERT. The important bit here is to provide the FQDN for ‘Hostname‘ and I also recommended providing the FQDN for the ‘Name‘ field too.
6. Next we will be asked to configure the machine solution user configuration file. Same as the Machine_SSL_CERT in the previous step, provide the FQDN for the ‘Hostname‘ field. For the ‘Name‘ field use the format <solution_user>-fqdn. i.e. machine-psc.domain.com
Important: It is crucial that the subject information of all solution user certificates in this environment are kept unique. You can read more about that in my earlier post Caution: Solution User Certificates in vSphere 6.0
7. Next we will be asked to configure the vsphere-webclient solution user configuration file. Repeat the same process as step 6 above. i.e. Make the ‘Name‘ field vsphere-webclient-psc.domain.com
8. If this was a vCenter Server with Embedded PSC there would be two more solution user configuration files to configure in the same manner; vpxd and vpxd-extension. Since this is an external PSC only two solution users exist.
9. Select Option 1: Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate
10. Provide a path to save the resulting csr and key file. In this example I just use the /tmp directory.
11. Next we will be asked to configure yet another configuration file called certool.cfg. This will control the information that our new VMCA certificate will contain. Again, provide the FQDN for the ‘Hostname‘ field and for the ‘Name‘ field use the format of VMCA-FQDN. i.e. VMCA-psc.domain.com
12. The certificate-manager will then generate the vmca_issued_csr.csr and vmca_issued_key.key in the location you specified earlier.
13. Using WinSCP or similar, copy the vmca_issued_csr.csr off the PSC Appliance.
14. Provide this csr to your certificate team to be issued with the new certificate. I will walk through this process using a Microsoft CA
Important: There is a known issue in 6.0 U2 where the CSR created for the VMCA as a Subordinate does not have the required attributes. Most Microsoft CA’s will force the inclusion of the attributes based on the certificate template used.
Check the certificate you just created from your CA by running the following OpenSSL Command:
openssl x509 -in vmca.cer -noout -text
The attributes we need present are CA:TRUE, Digital Signature, Certificate Sign, CRL Sign
X509v3 Basic Constraints: critical CA:TRUE X509v3 Key Usage: critical Digital Signature, Certificate Sign, CRL Sign
If it is missing these attributes then you will need to create the CSR a different way by following this KB Article http://kb.vmware.com/kb/2145544
Obtain the signed Certificates from a Microsoft CA
1. Open a browser to your certificate authority web interface. i.e. http://interca.domain.com/certsrv
2. Select ‘Request a certificate‘ then select ‘advanced certificate request‘
3. Open the vmca_issued_csr.csr in a text editor like notepad and copy and paste the entire contents into the ‘Saved Request‘ text field and choose your VMCA Root certificate template from ‘Certificate Template‘ drop down and then hit ‘Submit‘
4. Select ‘Base 64 encoded‘ and then hit ‘Download certificate chain‘. You will be prompted to download and save a certnew.p7b file.
5. Open the certnew.p7b file in Windows and drill down to the Certificates container. You should see the new VMCA Root Certificates as well as your Root MS CA and any Intermediate MS CA certificates.
6. Export each of these certificates. In my lab I exported root.cer, inter.cer and vmca.cer
Right-Click > All Tasks > Export Next > Base-64 encoded X.509 > File Path > Next > Finish
7. Open a Command Prompt to the location you exported the above certificates. Next we need to concatenate them into a single certificate chain, with the new VMCA on top, followed by the Intermediate CA, followed by the Root CA.
From the command prompt use the more command to concatenate. We will concatenate into a file called vmca_issued_cer.cer to keep with the naming convention of the files generated by the Certificate-Manager.
more vmca.cer >> vmca_issued_cer.cer more inter.cer >> vmca_issued_cer.cer more root.cer >> vmca_issued_cer.cer
8. Copy the vmca_issued_cer.cer up to /tmp/ on the PSC
Implement the signed Certificate as a new VMCA
1. Return to the Putty Session open to the PSC.
If you have closed this, return to the certificate-manager by running
Select Option 2, then Option 1
2. Select Option 1: Continue to importing Custom certificate(s) and key(s) for VMCA Root Signing certificate
3. Provide the full path to the new VMCA Certificate Chain we created and then the Key that was created earlier
Please provide valid custom certificate for Root. File : /tmp/vmca_issued_cer.cer Please provide valid custom key for Root. File : /tmp/vmca_issued_key.key
4. Ensure you have entered all the information correctly and also ensure you have taken a snapshot of the PSC and VC machines before answering ‘Y‘ to continue
You are going to replace Root Certificate with custom certificate and regenerate all other certificates Continue operation : Option[Y/N] ? : Y
5. The Certificate-Manager will then replace the VMCA with your new Subordinate CA certificate and replace the certificates on the PSC (MACHINE_SSL_CERT, machine and vsphere-webclient solution user certificates)
It will also update the relevant service endpoints with the new MACHINE_SSL_CERT.
Lastly it will restart all the services on the PSC.
You are going to replace Root Certificate with custom certificate and regenerate all other certificates Continue operation : Option[Y/N] ? : Y Get site nameCompleted [Replacing Machine SSL Cert...] cork Lookup all services Get service cork:586c2bb6-9078-4bd7-8ba9-ddc411798c1b Update service cork:586c2bb6-9078-4bd7-8ba9-ddc411798c1b; spec: /tmp/svcspec_xVqq3K Get service cork:fd3833cd-d96a-454e-9ccc-ea25d0befdfc // Snip // Get service 79b06bf4-a18d-4f9e-b5a9-04affc35d2a4_com.vmware.vsan.health Don't update service 79b06bf4-a18d-4f9e-b5a9-04affc35d2a4_com.vmware.vsan.health Get service a82770ef-91ec-4e97-ba5a-b48fa3d5a371 Don't update service a82770ef-91ec-4e97-ba5a-b48fa3d5a371 Get service 7137e349-af59-4d0e-a03f-d235c9406740 Don't update service 7137e349-af59-4d0e-a03f-d235c9406740 Get service 3996a8c7-40a7-47e6-8912-bc4688dbbc59 Don't update service 3996a8c7-40a7-47e6-8912-bc4688dbbc59 Get service 29e3287a-f635-4b2f-aded-5241cbf25395 Don't update service 29e3287a-f635-4b2f-aded-5241cbf25395 Updated 9 service(s) Status : 100% Completed [All tasks completed successfully]
Note: If this is a vCenter Server with Embedded PSC you would be complete and you would see more Updated Services.
If your vCenter Server is External then proceed to the next section.
Replace the Machine SSL Certificate on the vCenter Server from the new VMCA
1. Open a Putty session to the vCenter Server
2. Restart the vCenter Server service first before proceeding.
service-control --stop --all service-control --start --all
3. Launch the Certificate-Manager
4. Select Option 3 Replace Machine SSL certificate with VMCA Certificate
5. Provide the firstname.lastname@example.org credentials
6. Since this is a vCenter Server with an external PSC you will be asked to provide the IP Address of the PSC.
Performing operation on distributed setup, Please provide valid Infrastructure Server IP. Server : 192.168.2.101
7. Similar to before, we will be asked to configure the configuration file for MACHINE_SSL_CERT. The important bit here is to provide the FQDN for ‘Hostname‘ and I also recommended providing the FQDN for the ‘Name‘ field too.
8. Ensure the information entered is correct before answering ‘Y‘ to continue. Also ensure you have taken a new snapshot of the PSC and VC Machines.
You are going to regenerate Machine SSL cert using VMCA Continue operation : Option[Y/N] ? : Y
9. The vCenter Server MACHINE_SSL_CERT will be replaced with a new one issued by the VMCA as a Subordinate, the relevant service endpoints with the new MACHINE_SSL_CERT and the services will be restarted.
You are going to regenerate Machine SSL cert using VMCA Continue operation : Option[Y/N] ? : Y Get site nameompleted [Replacing Machine SSL Cert...] cork Lookup all services Get service cork:586c2bb6-9078-4bd7-8ba9-ddc411798c1b Don't update service cork:586c2bb6-9078-4bd7-8ba9-ddc411798c1b Get service cork:fd3833cd-d96a-454e-9ccc-ea25d0befdfc Don't update service cork:fd3833cd-d96a-454e-9ccc-ea25d0befdfc Get service cork:a46b5ede-2566-4d5a-9749-e83cbb644449 Don't update service cork:a46b5ede-2566-4d5a-9749-e83cbb644449 Get service be040cf2-5668-46fc-a013-3571745c7e12 Don't update service be040cf2-5668-46fc-a013-3571745c7e12 Get service f73219a5-730e-4429-81c2-4de727d16f73 // Snip // Update service 08970f6b-2b84-4e84-81f6-04d87ba44e49_kv; spec: /tmp/svcspec_nIUxUp Get service 4d30cfc1-d4eb-4303-8bfc-c2189cd1e7a1 Update service 4d30cfc1-d4eb-4303-8bfc-c2189cd1e7a1; spec: /tmp/svcspec_lkUrH5 Get service f371971c-4689-4f87-ae67-7db58f4d56df Update service ac4d2f3f-2cd4-4a92-8ac4-8cadfb2ddb30; spec: /tmp/svcspec_Qej_nM Update service a82770ef-91ec-4e97-ba5a-b48fa3d5a371; spec: /tmp/svcspec_M7zwgb Get service 7137e349-af59-4d0e-a03f-d235c9406740 Update service 7137e349-af59-4d0e-a03f-d235c9406740; spec: /tmp/svcspec_BmOy3p Get service 3996a8c7-40a7-47e6-8912-bc4688dbbc59 Update service 3996a8c7-40a7-47e6-8912-bc4688dbbc59; spec: /tmp/svcspec_b2H6M2 Get service 29e3287a-f635-4b2f-aded-5241cbf25395 Update service 29e3287a-f635-4b2f-aded-5241cbf25395; spec: /tmp/svcspec_OlFACZ Updated 21 service(s) Status : 100% Completed [All tasks completed successfully]
Replace the Solution User Certificates on the vCenter Server from the new VMCA
To be honest, replacing any solution user certificate is completely optional as they are not directly exposed to the end user. But if you require them to also be replaced by your new VMCA Subordinate then the following should be performed.
1. Launch the Certificate-Manager
2. Select Option 6 Replace Solution user certificates with VMCA certificates
4. Provide the email@example.com credentials
5. Since this is a vCenter Server with an external PSC you will be asked to provide the IP Address of the PSC.
6. Next we will be asked to configure the machine solution user configuration file. Provide the FQDN for the ‘Hostname’ field. For the ‘Name’ field use the format <solution_user>-fqdn. i.e. machine-vc.domain.com
Configure the vsphere-webclient, vpxd, vpxd-extension and solution user configuration files in the same manner. Ensuring that every solution user in the SSO Domain is unique.
If you follow my advice and make the ‘Name’ value <solution_user>-<fqdn> this will ensure that.
7. Ensure the information entered is correct before answering ‘Y‘ to continue. Also ensure you have taken a new snapshot of the PSC and VC Machines.
You are going to regenerate Solution User Certificates using VMCA Continue operation : Option[Y/N] ? : Y
8. The machine, vsphere-webclient, vpxd and vpxd-extension solution user certificates will be replaced with new ones issued by the new VMCA and the services will be restarted on the vCenter Server.