vRops 7.5 has been out little over two weeks now and from the looks of it, its an epic release with so many improvements and a great deal of new and exciting features.

I have upgraded a few environments with no problems at all and then last week yet another one got upgraded. As it does most times the upgraded went fine and the pre-upgrade bundle had no changes which would affect the customers environment.

I then went to the login page, to try to login to vRops. Selected the Active Directory source, typed my credentials and hit the login button. “Incorrect user name/password”. I tried a few times more, thinking I might have fat fingered the password or username, but no dice. I logged in as local admin in vRops and looked at the vRops version, it had upgraded to 7.5. Check authentication sources under administration->access, and compared it to the customers second environment, it is all an exact copy of the environment I had issues with, only difference was the other environment had not been upgraded yet, so it is still running vRops 7.0.

Log Insight to the rescue

Lucky, I thought, they have Log Insight so that might help me troubleshoot me issue. I set the source to the vRops node and search for my username.

It worked, but the messages were a bit cryptic, messages like:

com.vmware.vcops.auth.server.util.AuthUtils.searchUserByNameAndSourceId – Search for user by name: Michael and sourceId: 3262cfdb-e8f4-4f7d-af92-0d291ca8fd88 returned 0 entries. Must return only one entry. Something wrong

and

com.vmware.vcops.platform.gemfire.GemfireFunction.invokeHandlerMethod – Failed to invoke the method ‘UserAuthentication.authenticateUser’ : InvalidCredentialsException: No user exists with userName: Michael and sourceId: 3262cfdb-e8f4-4f7d-af92-0d291ca8fd88

I looked at authentication sources and access control lots of times, before I noticed something, which I thought might be a clue. In access control the Active Directory users username is in User Principle Name (UPN) style format. Meaning username@domain.com was shown as the username, but since I had always logged in with just the username, SAM Account Name style, I had only tested that way of logging in. Could this be why the logs stated it could not find my username?

I quickly logout of vRops and login with the UPN style naming convention. I works! So something has been changed. I reach out on twitter and slack, nobody has noticed any login issues after upgrading to vRops 7.5.

The problem

I scratch my head. As I already have upgrade my homelab vRops and had not noticed anything in that either. I logon to my homelab to verify if I can replicate the issue. Login just works, I scratch me head somewhat harder. Then I notice the UPN suffix is different from the domain name to which the customer is using.

Back into the homelab, logon to the domain controller, open domain and trusts, right click on the root and properties and add the UPN of vmware.com, what better UPN to test with.

Then I go into users and computers, find my user, right click properties, account and change the UPN suffix to @vmware.com.

The old login still works. So I go to access control and see the UPN is still the old one of MichaelRyomDK.root. Under authentication sources I click on the update icon, to synchronize the authentication source.

This time under access control, the UPN suffix has changed to @vmware.com.

So I once again logout and try to login again, SAM style meaning only using my username. Hurry, it failed to login! I successfully replicated the error.

So while trying to troubleshoot this issue one of the things that struck me as a bit odd was that under authentication sources, if I edit my source and looked under details the common name is set to userPrincipleName. This by the way is default when you add a new authentication source. This could explain why I was force to use UPN style logins, but it does not explain why if I am part of the original domain name or UPN suffix, I can do without and login with just my username.

I change the common name to samAccountName, hit ok and synced the authentication source again. Now the users under access control, showed up with no UPN suffix. I logout of vRops and back in with just my username. It worked!

Conclusion

So it seems that there has been a change in the way vRops 7.5 authenticates it users, so if you are using multiple or non default UPN suffix’s, what you might see after upgrading to vRops 7.5 is issues with authenticating users.

The quick fix is to under authentication source change the common name to samAccountName.

I have been in contact with Sunny Dua which is looking into the problem with vRops team.

With that said this was the only issue I had when upgrading to vRops 7.5 and for now there is a quick workaround to get this working again. So now you are truly ready to upgrade to vRops 7.5.

Thanks and take care.