Bad VMware Performance

Mishra

I've been installing some VMs at home to start messing with again. I have a Dell T110 with 4GB of RAM, quad core machine. 1 Sata hard drive.

I'm installing Windows on 1 machine, and I have a Linux box on another machine. It's just dragging the host because I'm installing patches on the Windows box... But I'm not seeing any number problem except maybe disk. Can you see an issue?

CPU 25% usage:


Time
Usage in MHz
Usage - 0
Usage - 1
Usage - 2
Usage - 3
Usage


4:10:40 PM
1377
14.53
11.25
16.44
15.3
14.38



Memory 50% usage:



Time
Swap used
Active
Consumed
Balloon
Granted
Shared common


12/8/2011 4:10:40 PM
1730616
2417552
3903672
702020
2864680
83320



Disk Datastore usage, 2 and 7 MS doesn't seem bad... :



Time
Write latency -
Read latency -


12/8/2011 4:10:40 PM
2
7






Disk Usage, 2076kbs definitely not hitting the 150MB SATA rate:
















Time
Write rate -
Read rate
Write rate
Usage
Read rate -


12/8/2011 4:10:40 PM
1863
213
1863
2076
213

meadIT

How many vCPUs on each?

Can you ssh to your host, run esxtop and post the CPU and memory screens?

meadIT

On second look, it looks like you're having memory issues. The balloon driver is activating and reclaiming RAM from your VMs. You are also having to swap memory to disk. Ideally, you want your swap to be zero.

blargoe

meadIT wrote: »

On second look, it looks like you're having memory issues. The balloon driver is activating and reclaiming RAM from your VMs. You are also having to swap memory to disk. Ideally, you want your swap to be zero.

Edit: I had posted something that I realized was not correct on my way to work this morning (had a different issue in mind from my job).

If you have limits set on the VM that are less than allocated memory, that could cause swapping as well, but it doesn't sound like you have done that.

Mishra

I assumed it was RAM, but I had to reboot the entire host in order for it to act properly. Even with only 2 VMs with 1 gig each running on a 4 gig host, it was swapping and ballooning. Did the same thing even after rebooting the VMs.

Weird..

scott28tt

You don't have enough RAM in the physical host. When you only have 4GB of RAM and you see that 1.7GB has been written to VMkernel swap and an additional 0.7GB of memory has been ballooned, that's a defnite sign that you don't have enough RAM.

Mishra

scott28tt wrote: »

You don't have enough RAM in the physical host. When you only have 4GB of RAM and you see that 1.7GB has been written to VMkernel swap and an additional 0.7GB of memory has been ballooned, that's a defnite sign that you don't have enough RAM.

Thanks, but I was experiencing it even when I turned off machines that could only take a max of 2gig from my 4gig VMware host.

Forsaken_GA

Mishra wrote: »

Thanks, but I was experiencing it even when I turned off machines that could only take a max of 2gig from my 4gig VMware host.

If you were patching, and you've got a single spindle, I'm not at all surprised that it started dragging. I use a Synology DS1511+ for my storage on the backend, and even with 5 spindles, when I'm patching 3 or 4 boxes at once, it slows down *every* thing.

instant000

Mishra wrote: »

Thanks, but I was experiencing it even when I turned off machines that could only take a max of 2gig from my 4gig VMware host.

This is my understanding:
1 - Paging is used when you run out of RAM, and have to use HDD
2 - Ballooning is used when the host is trying to re-allocate RAM from the VM so that it can share it with other VMs

From all the documentation I've read, ballooning isn't a big deal, as long as the VMs have plenty of RAM. Considering the amount it appears to be paging, that doesn't appear to be the case.

If you want to test, you could try running less RAM-intensive guest OS's, and see if the problem goes away, or not. You've given no indication of what guest OS's you're running, or how many guests you're running.

Also, as Forsaken_GA said, only one spindle can't help your performance much, especially if you're having to write a lot.

If you're going to restrict yourself to relying on paging, try installing additional drives, and dedicating volumes to paging, but before all that, I'd try running less guests or less resource-intensive guests or increasing the RAM in the host. I'm not sure of the RAM costs you'd run into for your box, or even if it'd be less to get a box with more RAM versus upgrading this one, not too familiar with your hardware setup there.

Mishra

Yes, as I said in the beginning I think, I felt like it was probably disk. With only 1 spindle, it is probably not going to do well. However, the disk stats didn't seem to be saying otherwise.

I had 2 VMs, 1 Linux and 1 Windows machine. Both VMs were set at a gig. I added a third Windows 2008 VM set at 4gig (I like to give it resources when creating them) and noticed my speed. I took it down to 1gig and still saw the problem. Then I powered only 2-out-of-3 VMs on only giving 2gig max to the host an still saw the issue.

It wasn't until I rebooted the whole host that it went away.

instant000 wrote: »

This is my understanding:
1 - Paging is used when you run out of RAM, and have to use HDD
2 - Ballooning is used when the host is trying to re-allocate RAM from the VM so that it can share it with other VMs

From all the documentation I've read, ballooning isn't a big deal, as long as the VMs have plenty of RAM. Considering the amount it appears to be paging, that doesn't appear to be the case.

If you want to test, you could try running less RAM-intensive guest OS's, and see if the problem goes away, or not. You've given no indication of what guest OS's you're running, or how many guests you're running.

Also, as Forsaken_GA said, only one spindle can't help your performance much, especially if you're having to write a lot.

If you're going to restrict yourself to relying on paging, try installing additional drives, and dedicating volumes to paging, but before all that, I'd try running less guests or less resource-intensive guests or increasing the RAM in the host. I'm not sure of the RAM costs you'd run into for your box, or even if it'd be less to get a box with more RAM versus upgrading this one, not too familiar with your hardware setup there.

scott28tt

Ballooning is triggered when the host has less than 4% free RAM, and swapping at less than 2% free. So I stand by my statement that you didn't/don't have enough RAM in the host.

Scott.

blargoe

scott28tt wrote: »

Ballooning is triggered when the host has less than 4% free RAM, and swapping at less than 2% free. So I stand by my statement that you didn't/don't have enough RAM in the host.

Scott.

And, it could take a bit for the memory to swap back in, even if you freed up some host memory by getting rid of the 4GB VM. Which would explain why you still saw some slow performance after you got rid of it.