Nested lab on for a few days gives error?

Asif Dasl

As the title says, I've left a nested lab on a few times over a few days (1 to 3 days) and I get this error:

Virtual machine creation may fail because agent is unable to retrieve VM creation options from the host (vmodl.fault.HostCommunication).

I found this blog, and I pinged everything. I can ping from the vCenter Server to the host when I get this error but I can't ping from host to the vCenter Server. Strange.

I loose all communication with the host for no reason, DNS & Reverse are fine. I have to do a reboot to restore communication... restarting the services doesn't help.

I don't get this with a physical lab... it's 5.5 U1 but it was the same with 5.5 GA. VMware Tools didn't help either.

Anybody running a nested lab over a few days or are you all turning it off once you've done what you wanted to do?

gc8dc95

Interesting. I have never left mine on, but maybe I should and see what happens.

Essendon

Any time-outs in Workstation that can be tweaked?

jibbajabba

I installed a nested lab yesterday which always runs 24/7 in the datacenter (now 5.5u1) , that is three virtual hosts with 2vCPU, 12GB of RAM each, and it will be running vcloud director, but at the moment only one linux vm to test the networking.

I can report back in a few days and see how it behaved by then ..

Asif Dasl

Well I noticed now that ICMP is disabled in Server 2012 Non-R2 but re-enabled in Server 2012 R2. I am using Non-R2 so the pings weren't allowed through the firewall, I have now disabled the DC & vCenter firewalls altogether to try it out again...

@gc8dc95 - Thanks for letting me know you turn yours off because I suspect most people do. If you try it out, let me know...

@Essendon - There is no timeouts that I'm aware of in Workstation - I had a bit of a Google as well just in case but found nothing. I disabled all power management on the DC & vCenter too.

@JibbaJabba - Thanks, I'd be interested in hearing how you get on with it. It's ESXi on ESXi? It's probably more stable than Workstation TBH.

If it happens again and I can ping both ways and a "services.sh restart" doesn't bring it back to life then I will move the lab to a physical 24/7 lab for VCAP-DCA because it's more stable.

Thanks again!

jibbajabba

Indeed ESXi inside ESXi ...

jibbajabba

Still alive and kicking "I am afraid" - including the hosted / nested vCloud infrastructure on it ... No disconnects in the logs either, just the usual noise ...

Asif Dasl

Thanks for trying, it's what I expected on ESXi - I'd give you some rep but it says I have to spread it around!

I tried it again without firewalls (not that I expected a different result..) and the same thing happened. I guess I am using it in a way it was not designed for. I am going to change my lab to a physical setup as planned...

jibbajabba

Workstation networking doesn't seem to be too stable. Keep reading that people got random issues.

jibbajabba

I wonder if any funky powersaving setting in your bios causes you issues.

Not DIRECTLY related, but maybe worth checking anyway :

https://communities.vmware.com/thread/473272

Asif Dasl

Thanks JibbaJabba - I would not have even thought it might have had an impact using Haswell processors...

However, I tried turning the C6 thingy off, the other one wasn't there but it made things worse. The host never gave me the error as listed in the original post, it went offline totally - no warning, nothing.

So I will turn a few other CPU performance settings off and see if that makes any difference.

Thanks again!

Asif Dasl

Just to answer my own question and in case it helps anyone else with the same error. I was using Workstation and 8 cores for the ESXi hosts but when I changed it to a single CPU with a single core my problem disappeared (ESXi hosts stayed up for 5 days consecutively) so I tried 1 CPU with 4 cores and so far so good. It didn't like me using 8 cores for some reason, even though my 4770T CPU has 8 cores?

jibbajabba

You say hosts with 8 Cores EACH running on your local PC with 8 cores total ?

Do I understand this right ?

Asif Dasl

When I setup VMs within Workstation I usually give it the full 8 cores of the CPU, Windows & Linux VMs are fine but ESXi doesn't seem to like it. The ESXi hosts don't seem to have HyperThreading enabled (it's labeled inactive), it is enabled in the BIOS of the Workstation computer though, so maybe that's why 4 cores seems to work OK. If this were ESXi on ESXi I would give it 1 CPU and let it do its magic.

Asif Dasl

Actually I am getting the same error now with 4 cores! it just popped up on my computer... back to 1 CPU and 1 core it is - at least that worked for 5 days for me which is more than enough time.

gc8dc95

This is interesting.

I have left my setup running for the last week or so and it is still working fine.

-Workstation 10
-2x ESXi 5.5
-vCenter
-Server 2K12
-Win 7 Pro

Asif Dasl

How many CPUs and cores are you using?

gc8dc95

I believe I have the ESXi hosts at 1 CPU 2 cores and all others are 1 CPU 1 core.

Asif Dasl

Cool, thanks. I will try 2 cores and see if it stays up for a few days. I have a physical lab to play with too, but that's running fine - it's just Workstation which seems to be falling over on me for some reason.

gc8dc95

I have a server at home also, but I tend to use it less now that I have setup a nested lab.

Mine is still going strong. Any more issues?

Asif Dasl

I've just got my new laptop (an Asus G750JM) so have been busy upgrading that, I haven't had the chance to play around with the nested lab the past few days. I will post an update over the weekend and will test the laptop out to see if it has the same problem as the desktop...

Asif Dasl

Still having problems with this... even on dual-cores it's having problems (albeit after 3 days). I will be mainly relying on my physical lab for VCAP-DCA once I get my physical lab re-arranged. I had to do stuff with the laptop so didn't get a chance to test this out on it yet. I will post an update once I leave it on for a few days - I will try it with dual-cores and then 8 cores if it stays up for 5 days. Again, I am probably using it in a way that it wasn't meant to be used but it should still work... it works with Server/Linux OS's

JeanM

Asif Dasl - the CPU you have doesn't have 8 cores though, it's got 4 - ARK | Intel® Core

Asif Dasl

It's kind of a grey area (I think!) because it's hyperthreaded which in Windows/Linux doubles the cores. It still doesn't work on either 2, 4, or 8 cores anyway. Only a single CPU & single core has worked for 5 days consecutively... the real test should be on the laptop as I will admit the 4770T is a rare-ish processor. The laptop processor is a i7 4700HQ, we'll see what happens on Windows 8.1

ARK | Intel® Core

jibbajabba

Hyper-Threading Gotcha with Virtual Machine vCPU Sizing | Wahl Network

Asif Dasl

Thanks for the link JibbaJabba, it was an interesting read and I will make sure to follow it in the future.

However, I finally, finally got it work on the laptop for 5 days (with 8 cores!). My desktops used for dual-booting to vSphere/Win7 - well it's the motherboard (Asus Z87-PRO) or the CPU (Intel i7 4770T) which is causing the problem as far as I can see. I'm going to use the laptop in future anyways and have the desktops running remotely so... problem solved!

Kinet1c

Might not be a fix but have you thought about scheduling a reboot of the hosts on a daily basis?

Asif Dasl

Probably not, the real reason I wanted a lab to be up for like 5 days was to do the VCAP-DCA on a nested lab while still having access to Windows 7. Which I thought should be stable enough.

When my desktops were used for the physical vSphere lab I didn't have access to Windows 7 and the way everything was set up there.

But then I figured I would run a remote physical lab and a nested lab on the laptop. I didn't investigate it as I was doing the EMC course, but now that is over I am back on the case.

I have no idea what's causing it TBH but I'm using the laptop now and it doesn't happen there so I can some proper study done now.