Unable to connect to Inventory Service

Johnjones

So I've setup two vCenter's in different remote sites we'll call them North and South. I've linked them together. They both show up fine in through the vsphere thick client. However, when logging into the webclient I received the error in the thread title. Also, in the thick client the inventory search feature doesn't work for "South".

I've tried resetting/rebuilding the database, reinstalling the inventory service and webclient, etc. Nothing seems to work. I don't understand it. This defeats the purpose of linked mode if I can't manage from the webclient.

joelsfood

Check that the accounts match and mentioned firewall ports are open

VMware KB: Logging in to vCenter Server in linked mode using the vSphere Web Client fails with the error: Unable to connect to vCenter Inventory Service on xxxxxxxx

Johnjones

Thanks for reply. I checked that KB previously and the accounts match. Windows FW has been completely turned off. Could this possibly be a network layer firewall issue?

EDIT - Another thing I've noticed is on "South" when I add my AD account with admin permission...I've being told I've entered incorrect login information. It works when I click the "use windows credentials" and it's same account!

I can type in my AD credentials on "NORTH" and it works perfectly! My mind = blown.

Deathmage

1. As the two seperate location's 'north' and 'south' in two different AD forests?

2. Are the two locations in different IP ranges?

3. If their in two different IP ranges, have they been added to the Sites and Services area for each forest's AD or solo AD forest?

4. Can clients in location 'north' managed hosts in location 'south? and vis-versa?

5. if these are indeed remote sites, do they have a VPN tunnel between each location?

6. This tied in sites and services, but can you resolve the vCenter's in each location with DNS? - if their not setup in sites and services, DNS won't 'know' to ping across WAN's even if a VPN tunnel is established.

7. Do you have proper network routes between each location so each network knows where to do when requests for network X from network Y? ...vis-versa.

8. if you have a firewall is the device blocking ICMP/DNS/Domain traffic from traversing the WAN's?

let me brainstorm some more here, check these out and we can go from here.

Essendon

SSO mode? Name of SSO domain the same for both vCenters?

Johnjones

Deathmage wrote: »

1. As the two seperate location's 'north' and 'south' in two different AD forests? Nope.

2. Are the two locations in different IP ranges? Yep

3. If their in two different IP ranges, have they been added to the Sites and Services area for each forest's AD or solo AD forest? Yep

4. Can clients in location 'north' managed hosts in location 'south? and vis-versa? Yep

5. if these are indeed remote sites, do they have a VPN tunnel between each location? Believe so...I'll need to double check with our network team.

6. This tied in sites and services, but can you resolve the vCenter's in each location with DNS? - if their not setup in sites and services, DNS won't 'know' to ping across WAN's even if a VPN tunnel is established. Yep

7. Do you have proper network routes between each location so each network knows where to do when requests for network X from network Y? ...vis-versa. Again, I dunno.

8. if you have a firewall is the device blocking ICMP/DNS/Domain traffic from traversing the WAN's? Dunno.

let me brainstorm some more here, check these out and we can go from here.

See my responses above. I was unable to log into "South" from one of the North servers. Incorrect credentials. It's like South isn't able to communicate traffic out...

Johnjones

Essendon wrote: »

SSO mode? Name of SSO domain the same for both vCenters?

I believe it's vCenter Single Sign-On for an additional vCenter Server with a new site. However, when I checked the first vCenter(North) it only has one string in the LS_ServiceID file, but on "South" it has like 6 different strings or lines. They are all different from the line in North. I don't know it this matters at all?

Essendon

The strings will be different when you deploy SSO with a new site. That may not be the issue.

What port did you use for SSO? Is that port open in both directions? To me, at this stage, this looks like a firewall issue. Telnet in both directions to determine this.

Johnjones

I left it on the default which I think is 7444. I am able to telnet into both. I'm with ya on it being a network issue. I've had an issue before with these guys and spent hours troubleshooting, but it ended up being an issue on their end.

Deathmage

As Essendon has mentioned and was asked above I truly feel it's a networking issue.

Simple test:

From the north side do a 'mstsc /admin' into the South vCenter server with the administrator information you using to login. If it works then AD user authentication is working across networks. If it doesn't work then port 3389 is being blocked and if not explicitly set 3389 would be in an explicit deny. I'd be pretty safe to presume the port for SSO, if kept default, is also being blocked.

To me honestly it sounds like a firewall or an explicit deny from an ACL blocking the SSO port.

Do you manage the network aspect of the companies network?

Johnjones

I'm sure AD authentication is working between networks because I can log into servers all day long in North or South. Unfortunately, I don't manage our network. It's weird that when I click "use windows credentials" I'm able to log in, but when I manually enter the SAME creds it won't let me in.

Essendon

Johnjones wrote: »

It's weird that when I click "use windows credentials" I'm able to log in, but when I manually enter the SAME creds it won't let me in.

Not that weird. SSO isnt working. When you enter your credentials in, they are passed on to SSO which in turn checks with AD, returns you a token if you are authorized to log in. Check the SSO logs at \ProgramData\VMware\CIS. Get into the habit of scouring through logs or get Log Insight.

Johnjones

Yep. I see a ton of "Identity Manager Failed to find group" and I see a bunch of our AD groups. Also, I see "Failed to authenticate principal" for my login information. Now I just need to figure out why this is happening and it's only occurring on the South vCenter. Could this still possibility be a firewall problem?

Deathmage

Johnjones wrote: »

Yep. I see a ton of "Identity Manager Failed to find group" and I see a bunch of our AD groups. Also, I see "Failed to authenticate principal" for my login information. Now I just need to figure out why this is happening and it's only occurring on the South vCenter. Could this still possibility be a firewall problem?

Yup, figured this much.

Sounds like networking with AD authentication. Again, do you have the different networks in sites and services or are they in different AD forests?

Additionally are the vCenter Servers where SSO is located in DNS that is linked to AD?

Essendon

Are you able to log in with the administrator@vsphere.local account in both instances? Check your domain shows up as a valid identity source in both SSO machines.

Copy and paste the exact error here please?

Yes, I still think this is a f/w issue.

Johnjones

So I still don't think AD because after logging into vSphere client and trying to add a permission I see our domain and I'm able to search for users/groups and add them. This is working on both vCenters.

I am able to use the admin@vsphere.local to login for both.

I can't copy and paste exact error, but it's this: 2013-11-04 10:48:23,372 INFO [IdentityManager] Failed to find group [Remote Desktop Users@corp.hit.ads] as FSP group in tenant [vsphere.local]

Also - 2013-11-04 10:48:23,154 INFO [IdentityManager] Failed to find principal [mailto:adm_sbiswas@corp.hit.ads as FSP user in tenant [vsphere.local]

Essendon

Check this > VMware KB: In a mixed vSphere 5.1/5.5 environment, logging in to the vSphere Web Client fails with the error: Client is not authenticated to VMware Inventory Service

Johnjones

Yeah, I was looking at that earlier. We are all 5.5e. I've hit up most of the KB's and had VM support take a look but they were lost as well.

I was reading this VMware KB: Logging into the vSphere Web Client 5.5 fails with the error: Provided credentials are not valid. but it was already configured to our domain so I dunno.

Essendon

Re-install everything - might just be quicker, provided you are allowed to do it. Other than this, check with the networks team first.

Johnjones

I think something is wrong with SSO on South. Is it possible to remove it and then reinstall it by itself?

Essendon

Probably could, I haven't done it though. Verify networking first.

Deathmage

Well I'm now convinced it's a networking issue, actually quite curious now what you find as the root cause. If I was there with you we'd probably figure it out, right now it's just speculation.

Essendon

I'm not convinced yet, he said he could telnet successfully, but there could be other bits that aren't allowed through the firewall. I think a reinstall is in order. Again, check with networks.

Johnjones

Those same "failed to find group" errors are also on North. So I dunno if it's reinstall. Regardless, I'm going to check with our net guys are go from there.

Really appreciate the help guys!

Deathmage

something just came to mind. I remember I dealt with something similar, but did you setup LDAP for the SSO in the ESXi 5.5 SSO section of the 5.5a Web Client?


https://virtualizationreview.com/articles/2014/05/28/vsphere-ad-authentication.aspx

https://communities.vmware.com/message/2375836?tstart=0

Johnjones

I'm not using the appliance. Also, I am able to browse users/groups through SSO/configuration on the webclient. I've tried removing it, adding it back and setting as default but nothing. Thanks again.

Essendon

Try a fresh attempt at logging in. When it bombs out, check the SSO logs immediately. Copy and paste some of those logs here?

Deathmage

Johnjones wrote: »

I'm not using the appliance. Also, I am able to browse users/groups through SSO/configuration on the webclient. I've tried removing it, adding it back and setting as default but nothing. Thanks again.

it doesn't just apply to the appliance good sir. SSO is for the 2008 R2/2012 R2 version of vCenter and the Appliance, what matters is you still need to configure some parts of SSO from the Web Client than can't be done from the vSphere Thick Client.

Also, I think I remember seeing you mention it before but did you join the ESXi host's to the domain, is your vCenter on a VM or is it a physical box?

if all else fails, something else to try, your AD records for your vCenter box could be corrupt in AD. See if you leave the domain and join WORKGROUP, reboot, and then rejoin the domain. You could have a AD breakdown with the vCenter server.

Johnjones

No I understand what you're saying, but it's configured through SSO on webclient. The ESXi hosts are not joined to the domain...I don't think I'll have to double check that. Both vCenter's are on physical machines.

I'm going to back off until tomorrow...I don't want to spend my day off troubleshooting this when it's probably on the network side of the house.

I'll post more details tomorrow.

Essendon

Deathmage wrote: »

Also, I think I remember seeing you mention it before but did you join the ESXi host's to the domain, is your vCenter on a VM or is it a physical box?

if all else fails, something else to try, your AD records for your vCenter box could be corrupt in AD. See if you leave the domain and join WORKGROUP, reboot, and then rejoin the domain. You could have a AD breakdown with the vCenter server.

I know you are trying to think outside the square, but these points are largely irrelevant to the problem at hand. Hosts joined to the domain or not wont influence why you cannot logon to the web client. vCenter being physical or virtual wouldn't matter at all (he's confirmed he can log into the thick client with no issues). Again, if the vCenter object was corrupt in AD, he'd be on P1 call with VMware and/or Microsoft, not asking us questions here. Terrible things happen when AD goes outa whack.

Re-install, I think.

Deathmage

Essendon wrote: »

I know you are trying to think outside the square, but these points are largely irrelevant to the problem at hand. Hosts joined to the domain or not wont influence why you cannot logon to the web client. vCenter being physical or virtual wouldn't matter at all (he's confirmed he can log into the thick client with no issues). Again, if the vCenter object was corrupt in AD, he'd be on P1 call with VMware and/or Microsoft, not asking us questions here. Terrible things happen when AD goes outa whack.

Re-install, I think.

Indeedio, right now it's speculation. I'd have to see it to troubleshoot it. It could be so many different things, as you know.