In vSphere 6.0 you have Solution Users that internal vCenter/PSC services use to interact. These Solution Users use certificates to log into services and components instead of maintaining passwords.
You have the option to replace these certificates with your own certificates or use VMCA issued certificates.
To solve a separate problem, the ability to control the Certificate Subject information in the Solution Users was added in an update to the vSphere Certificate-Manager with 6.0 U1b that allows the user to specify the Subject information for each Solution User.
Update: vSphere 6.0 U3 has made improvements to the Certificate-Manager to prevent you from getting into this issue. You will be only asked to complete one cfg file and the tool will automatically make a value unique using the Solution User ID.
If you answer ‘Y’ to the above question you will then be presented later with the options to configure the machine.cfg, vsphere-webclient.cfg, vpxd.cfg and vpxd-solution.cfg files.
The problem is that this makes it easy for a user to end up with Solution User certificates that have identical Subject information.
This causes several issues in vCenter Server, including but not limited to, the following:
The vSphere Web Client may display one of the following errors:
A server error occurred. No error message is available. :-( Check the vSphere Web Client server logs for details.A server error occurred. SSO error: null Check the vSphere Web Client server logs for details.
The PSC UI may display the error:
HTTP Status 400 - An error occurred while sending an authentication request to the PSC Single Sign-On server - null type Status report message An error occurred while sending an authentication request to the PSC Single Sign-On server - null description The request sent by the client was syntactically incorrect.
The psc-client.log contains similar to the following:
[2016-02-19 11:04:40.499] [ERROR] tomcat-http--5 com.vmware.vim.sso.client.impl.SoapBindingImpl - SOAP faultjavax.xml.ws.soap.SOAPFaultException: Error occured looking for solution user :: More than one solution user found
The vsphere_client_virgo.log contains similar to the following:
[2016-02-19T11:20:19.725Z] [INFO ] ing.timer.TimerFactoryBean#0 c.v.v.s.c.impl.SecurityTokenServiceImpl$RequestResponseProcessor Failed trying to retrieve token: ns0:RequestFailed: Error occured looking for solution user :: More than one solution user found [2016-02-19T11:20:19.726Z] [ERROR] ing.timer.TimerFactoryBean#0 com.vmware.vise.vim.security.sso.impl.NgcSolutionUser Login as solution user failed. com.vmware.vim.sso.client.exception.InternalError: Failed trying to retrieve token::ns0:RequestFailed: Error occured looking for solution user :: More than one solution user found
Other vSphere 6.0 logs such as the license.log and inv-svc.log may also contain errors containing “More than one solution user found“.
You can quickly check your Solution User certificates and review their Subjects if you think you may be impacted. In the below example you can see that the Subject is identical for each Solution User.
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine --text | grep Subject: Subject: CN=Acme, C=IE, ST=Cork, L=Cork, O=GSS, OU=VMware/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd --text | grep Subject: Subject: CN=Acme, C=IE, ST=Cork, L=Cork, O=GSS, OU=VMware/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension --text | grep Subject: Subject: CN=Acme, C=IE, ST=Cork, L=Cork, O=GSS, OU=VMware/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient --text | grep Subject: Subject: CN=Acme, C=IE, ST=Cork, L=Cork, O=GSS, OU=VMware
To resolve this issue you will need to re-issue or re-generate new Solution User certificates, ensuring that the Subject for each certificate is unique.
The usual method I recommend to achieve this is via the “Name” attribute and advise setting it to “Solution User – FQDN”. For example “vpxd-vcenter.domain.com”
A vCenter Server with an Embedded PSC will have 4 Solution Users:
A vCenter with an External PSC will have 4 Solution Users:
An External PSC will have 2 Solution Users:
9 thoughts on “Caution: Solution User Certificates in vSphere 6.0”
Thanks for this, saved me quite a bit of troubleshooting time.
LikeLiked by 1 person
Thanks Steve – hoping to get some time to publish more cert related bits and pieces soon.
I see you don’t monetize your page, don’t waste your traffic, you can earn additional cash every month because you’ve got hi quality
content. If you want to know how to make extra bucks, search for: Boorfe’s tips best
Ꮋi there friends, gⲟod article and pleasant urging commented ɑt this plaｃe, I am
really enjoying by these.
Hey! This is my first commеnt hеre so I ϳust
wanted tо givｅ a quick shout οut and tell ʏou I genuinely enjoy reading tһrough yοur posts.
Can you sᥙggest any օther blogs/websites/forums that cover tһе same subjects?
I ᥙsed to Ье suggested this blog by mｙ cousin. Ӏ’m noѡ not ѕure whеther or not tһіs publish iѕ wrіtten by means of him as no оne else realize ѕuch specified aboᥙt my trouble.
Уou’re wonderful! Ꭲhank yoս!
Thank you for some other wonderful post. Ꮤhere еlse ϲould anybody
get that kind оf information in sᥙch an ideal method
of writing? Ι’νe ɑ presentation subsequent ԝeek, аnd
I am ⲟn thе look for sucһ information.
Hello. I have noticed some issues with vCloud director and i was told you could engage engineering for me..
I’d advise that you open an SR with VMware Support https://www.vmware.com/support.html