vSphere 6.x Certificates – Just because you can, doesn’t mean you should.

As I work closely with VMware Support, it’s clear that issues and confusion around vSphere 6.x certificates are still very much a pain-point for customers.

I’ve spoken a bit about this topic in the past (but have been meaning to get back to it). You can see my previous posts below: (Note: even though they say 6.0 they are applicable for 6.5 too)

• Caution: Solution User Certificates in vSphere 6.0
• vSphere 6.x SSL Trust Anchors
• vSphere 6.0 Certificate Series

What I want to achieve by this post is to hopefully dispel some of the confusion. First, repeat the title of this post to yourself – “Just because you can, doesn’t mean you should.”

Just because you can replace any and all certificates in a vSphere environment, doesn’t mean necessarily should.

The only question you need to be able to answer is – “What problem am I trying to solve?”

tl;dr

Long story short, for the majority of use cases, replacing the Machine SSL certificate on each vCenter / PSC should be sufficient. Keep reading for more information.

---

I’m going to focus on three potential answers to that question.

1. I want to get rid of the annoying browser warning when I access the Web Client

2. Management Interfaces (i.e. Web Client) must run with trusted CA signed certificates.

3. Company Policy mandates that all certificates must be replaced with trusted CA signed certificates.

In my experience, the vast majority of vSphere customers fall into category 1 or 2.

Before we delve into all that, lets run through all the certificates present on a vCenter 6.x system and what they are used for.

Diagrams

Embedded PSC

In this first diagram we illustrate the certificates for a vCenter Server with Embedded PSC

External PSC

In this diagram we illustrate the certificates when using an external PSC. You can extrapolate for larger environments as more PSCs and vCenter’s are added.

---

Machine SSL Certificate

This the main certificate and the only one you should care about if you answered 1 or 2 to the question above.

It is presented from the server on port 443 via the reverse proxy service and it is what you hit when you access the vSphere Web Client, the HTML5 Web Client (6.5), the PSC UI, the VAMI, use the C# Client (6.0), or use PowerCLI to connect to vCenter.

It is the only user-exposed certificate. If all you care about is not seeing any “untrusted certificate” warnings from any of your interfaces, then this and only this is the certificate you need to change on your Embedded vCenter, External PSC or vCenter machines.

Solution User Certificates

There are four solution user types and you can check the link above for more detail on Solution User certificates.

vCenter and PSC services need to be able to communicate with the VMware Directory Service within an SSO Domain . You have your admin user administrator@vsphere.local and that has a user-defined password.

There are also the solution users, for example, vpxd-machine-uuid@vsphere.local but they don’t have a traditional password. So for these users to log into VMdir, they use certificate based authentication. These certificates are not user-exposed. You should never hit them in your Browser.

VMCA

The VMCA certificate is just that, a CA (certificate authority). It is capable of generating and signing new certificates. It is what issues all the certificates we talk about on a new installation of vSphere 6.x

Trusted Roots

Stores all Issuer Certificates (Root CAs and Intermediate CAs) that have issued Machine SSL Certificates and Solution User Certificates. If running custom certificates, then whatever CA issued those certificates needs to be present here.

Lookupservice Certificate

In 6.5 this certificate is automatically always the same as the Machine SSL so for 6.5 it’s non-issue and you don’t need to worry. In 6.0 this is it’s own certificate and can be seen if you point your browser to the lookupservice address, but only if you specify port 7444 (https://fqdn:7444/lookupservice/sdk).

Unless you have some older vCenters running 5.5 or older versions of NSX and vSphere Replication, you shouldn’t have any products accessing the lookupservice directly on port 7444. All products should be using port 443 to access the lookupservice and as a result they will receive the Machine SSL via the reverse proxy service.

https://kb.vmware.com/kb/2118939

VMDir Certificate

The VMdir certificate is used between PSCs for replication and also between VCs and a PSC, primarily during install when the VC (vmafd service) is registering with the PSC in VMdir.

Like the Solution User certificates, it’s not something that is user-exposed. The only requirement to change this certificate comes when enabling PSC HA and the latest scripts for both 6.0 and 6.5 to enable PSC HA take care of that.

Replace the VMware Directory Service Certificate – VMware Docs

STS Certificate

The Secure Token Signing certificate is almost self-explanatory. When we, a client,  (or any solution) logs into an SSO Domain the client is given a session token and that token has been generated and signed by the STS certificate. Again, this is not a user-exposed certificate. Messing with this certificate can result in you getting locked out of your vSphere Environment. I do not advise even attempting to replace this certificate.

---

What do I need to do?

Back to our question – “What problem am I trying to solve?” and our potential answers

1. I want to get rid of the annoying browser warning when I access the Web Client

2. Management Interfaces (i.e. Web Client) must run with trusted CA signed certificates.

3. Company Policy mandates that all certificates must be replaced with trusted CA signed certificates.

 

I hope with the explanation about what certificate is used for what purpose, things are clearer as to what options you need to take to satisfy the problem you’re trying to solve.

To solve #1 you don’t even need to replace anything. You can download the VMCA Root certificate and install it in your local machine (or push via Group Policy) and voila, no more annoying “untrusted certificate” warnings. See KB2108294 for more.

If that isn’t enough and you must use a certificate from a trusted CA (either internal CA or commercial CA) and satisfy #2, then you just replace the Machine SSL Certificate(s). Leave the solution users, etc, alone.

Unless it’s absolutely necessary, you shouldn’t bother going to all the pain and effort in replacing any more certificates in the vSphere Environment. If you do, and fall into #3, my advice to you is – do it in a throwaway test VCSA first.

The elephant in the room might be “what about VMCA as a Subordinate?”. This option isn’t recommended by VMware going forward and you can read up on that in a blogpost by vSphere Senior Technical Marketing Guru Adam Eckerle here or by Security Senior Technical Marketing Guru Mike Foley here

I echo what both gents state in their respective blog posts and advocate that only the Machine SSL certificate be replaced unless it is absolutely necessary to replace more certificates.