Caution: Solution User Certificates in vSphere 6.0

In vSphere 6.0 you have Solution Users that internal vCenter/PSC services use to interact. These Solution Users use certificates to log into services and components instead of maintaining passwords.

You have the option to replace these certificates with your own certificates or use VMCA issued certificates.

To solve a separate problem, the ability to control the Certificate Subject information in the Solution Users was added in an update to the vSphere Certificate-Manager with 6.0 U1b that allows the user to specify the Subject information for each Solution User.

Update: vSphere 6.0 U3 has made improvements to the Certificate-Manager to prevent you from getting into this issue. You will be only asked to complete one cfg file and the tool will automatically make a value unique using the Solution User ID.

If you answer ‘Y’ to the above question you will then be presented later with the options to configure the machine.cfg, vsphere-webclient.cfg, vpxd.cfg and vpxd-solution.cfg files.

The problem is that this makes it easy for a user to end up with Solution User certificates that have identical Subject information.

This causes several issues in vCenter Server, including but not limited to, the following:

The vSphere Web Client may display one of the following errors:

A server error occurred.
 No error message is available. :-(
 Check the vSphere Web Client server logs for details.

 A server error occurred.
 [500]SSO error: null
 Check the vSphere Web Client server logs for details.

The PSC UI may display the error:

HTTP Status 400 - An error occurred while sending an authentication request to the PSC Single Sign-On server - null
 type Status report
 message An error occurred while sending an authentication request to the PSC Single Sign-On server - null
 description The request sent by the client was syntactically incorrect.

The psc-client.log contains similar to the following:

[2016-02-19 11:04:40.499] [ERROR] tomcat-http--5 com.vmware.vim.sso.client.impl.SoapBindingImpl - SOAP faultjavax.xml.ws.soap.SOAPFaultException: Error occured looking for solution user :: More than one solution user found

The vsphere_client_virgo.log contains similar to the following:

[2016-02-19T11:20:19.725Z] [INFO ] ing.timer.TimerFactoryBean#0 c.v.v.s.c.impl.SecurityTokenServiceImpl$RequestResponseProcessor Failed trying to retrieve token: ns0:RequestFailed: Error occured looking for solution user :: More than one solution user found
[2016-02-19T11:20:19.726Z] [ERROR] ing.timer.TimerFactoryBean#0 com.vmware.vise.vim.security.sso.impl.NgcSolutionUser Login as solution user failed. com.vmware.vim.sso.client.exception.InternalError: Failed trying to retrieve token::ns0:RequestFailed: Error occured looking for solution user :: More than one solution user found

Other vSphere 6.0 logs such as the license.log and inv-svc.log may also contain errors containing “More than one solution user found“.

You can quickly check your Solution User certificates and review their Subjects if you think you may be impacted. In the below example you can see that the Subject is identical for each Solution User.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine --text | grep Subject:
 Subject: CN=Acme, C=IE, ST=Cork, L=Cork, O=GSS, OU=VMware

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd --text | grep Subject:
 Subject: CN=Acme, C=IE, ST=Cork, L=Cork, O=GSS, OU=VMware

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension --text | grep Subject:
 Subject: CN=Acme, C=IE, ST=Cork, L=Cork, O=GSS, OU=VMware

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient --text | grep Subject:
 Subject: CN=Acme, C=IE, ST=Cork, L=Cork, O=GSS, OU=VMware

To resolve this issue you will need to re-issue or re-generate new Solution User certificates, ensuring that the Subject for each certificate is unique.

The usual method I recommend to achieve this is via the “Name” attribute and advise setting it to “Solution User – FQDN”. For example “vpxd-vcenter.domain.com”

---

A vCenter Server with an Embedded PSC will have 4 Solution Users:

• vpxd
• vpxd-extension
• machine
• vsphere-webclient

A vCenter with an External PSC will have 4 Solution Users:

• vpxd
• vpxd-extension
• machine
• vsphere-webclient

An External PSC will have 2 Solution Users:

• machine
• vsphere-webclient