VM Total capacity vs. Available capacity vs. Memory Usage

JeanM

Hello,

Is Memory Usage = memory used by the host? 906mb (out of 3582 capacity)

If the capacity is 3582, and memory used by host is 906mb, why is available capacity 1597mb when NO vm's are running?

VM's also do not have any memory reservation set....

Essendon

Yes, memory usage is the memory in use by the host. The available capacity is the what's left taking out what's already been used (host usage + VM usage + overhead)

JeanM

Ok, that's what I was thinking about the memory used by host (906).

Then where is the rest of it going? 3582 - 906 = 2676 vs 1597 ?

Essendon

ESXi doesnt flush out memory pages that have once been allocated to a VM. So if a Windows VM's was once allocated 8GB RAM, ESXi remembers that it once handed that memory out and doesnt allocate it to another VM. Why does it do this? Because it's much quicker for it to dish out the same memory pages to the VM than having to flush them out, assign them to another VM and then when the first VM requests the same pages assign new pages to it.

Add to that the virtualization overhead and you should be pretty close to the difference between 2676 and 1597 (about 1GB)

While I'm at it, I'll recommend you and others reading this - read this > VMware KB: VMware vSphere 5 Memory Management and Monitoring diagram

JeanM

Essendon - I was starting to think it had to do with memory reservation possibly, as I did use that feature in order to enable directpath I/O feature on a nic.

I'll watch the video and check if it talks about how to maybe flush out the memory pages, there must be a way for that right ?

Essendon

Yeah, if you reserved memory in the past and removed it later - I believe the pages arent flushed till a reboot of the host takes place. When you reserve memory for a VM, ESXi will back the virtual machine memory with physical memory and lock it to that VM. It'll not be reclaimed. CPU reservations behave differently. CPU reservations are more opportunistic, if a VM needs more grunt ESXi will give it what it asks for, but if doesnt need CPU cycles any longer they are handed over to other VM's.

Yeah, check the video and the accompanying poster. The poster's really good.

santaowns

Jean you still out in mather? If so what kind of memory do you have, if it is server ddr2 I might have some.

JeanM

santaowns wrote: »

Jean you still out in mather? If so what kind of memory do you have, if it is server ddr2 I might have some.

I work in that area yeah I did already order 8gb of ddr2 on eBay just the other day to max out this board. This one is just to get my feet wet, until I build a new i7 32gb host with ddr3. I should have bought more ram last month went it was CHEAP!!!

JeanM

Essendon wrote: »

Yeah, if you reserved memory in the past and removed it later - I believe the pages arent flushed till a reboot of the host takes place. When you reserve memory for a VM, ESXi will back the virtual machine memory with physical memory and lock it to that VM. It'll not be reclaimed. CPU reservations behave differently. CPU reservations are more opportunistic, if a VM needs more grunt ESXi will give it what it asks for, but if doesnt need CPU cycles any longer they are handed over to other VM's.

Yeah, check the video and the accompanying poster. The poster's really good.

I just got home, and did the following.

Checked each VM to make sure none are set for memory reservation.
Went ahead and set each one to 256mb as a test.
Rebooted the host.

After it came back up, no change, it still shows 1595 MB as available capacity. And shows 908 MB memory usage , out of 3582.29 MB capacity.

Watching the video in the link now ..

JeanM

Removed pool and all virtual machines, rebooted.

With no Resource Pool and no VMs -
Host Summary tab shows 905 MB used, 3582.29 MB Capacity
Resource Allocation tab shows Total Capacity 1597 MB, Reserved Capacity 0 MB, Available Capacity 1597 MB.

Where is the 3582 -905 - 1597 = 1080 ?

JeanM

Essendon any ideas?

tstrip007

did you "remove from inventory" or "delete from disk"?

JeanM

Pretty sure I did both, the files are gone from the datastore. And the pool does not list the VM's....

Essendon

Open ssh access to the host and use esxtop with the memory view to take a look at what's holding on to the memory. It may well be in use or somehow reserved for something by the kernel. The stats in vCenter/client arent always exactly revealing as to what's really happening. esxtop's values are accurate.

JeanM

Essendon wrote: »

Open ssh access to the host and use esxtop with the memory view to take a look at what's holding on to the memory. It may well be in use or somehow reserved for something by the kernel. The stats in vCenter/client arent always exactly revealing as to what's really happening. esxtop's values are accurate.

Saw the esxtop in the video, and was going to do that next. Does the host need to be in maint. mode first?

Essendon

Nope...

JeanM

Here we go, how do I re-claim this memory and make it available again? There is no pool and no vm's in the host...

JeanM

Basically, it looks like 1971MB of memory is reserved by a pool that I've deleted? Is there a command that I can run to wipe any/old vm/pool config data to re-claim the memory back?

I've tried resetting the host back to default config, rebooted, re-configured management ip , same deal.

Essendon

Interesting. While I'm not aware of such a command, let's do a test. Spin up a VM or two and assign them more memory than is supposedly available and see if they power on. If the memory is indeed reserved then the VM's shouldnt be able to power on.

What version of ESXi is this? I have 5.5 running on my systems and dont see such a thing.

JeanM

It's ESXi 5.0.0 - 469512-standard
I'll give it a shot tonight.

Essendon

JeanM wrote: »

Pretty sure I did both, the files are gone from the datastore. And the pool does not list the VM's....

Hang on! Did you say pool? So is there a resource pool? Though it looks like you've said a few posts above that there's no resource pool anymore. Wouldnt hurt to double-check.

Post a screengrab of the memory tab too (the one in the resource allocation tab, similar to the second screengrab in your opening post)

JeanM

Essendon wrote: »

Hang on! Did you say pool? So is there a resource pool? Though it looks like you've said a few posts above that there's no resource pool anymore. Wouldnt hurt to double-check.

There is no pool anymore, i removed it hoping that it would free up the memory but it didn't.

JeanM

Ok, so it would not let me spin up a VM or VM's past the 1597 MB capacity..... I went ahead and wiped the host and reloaded.

All is good, I'll try to "dupe" the issue again , maybe it's a bug, no idea...

Asif Dasl

I'm interest in this thread as I have a similar problem on the latest 5.5 build. I've got 32GB of RAM - 1580 MB used and then it shows me 28674 MB free. No VMs are on at all, yet I am missing an extra 2.5 gigs of RAM for some unknown reason. Well it says reserved but nothing shows up in ESXTOP that I can see.

As an aside why did VMware dumb down the memory screen? It used to more useful than it is now, like on the PDF Essendon linked to. - EDIT: nevermind, I was looking at the host summary not the VM resource allocation tab, the PDF makes it look like it's under host summary, I even checked vSphere Client 4.1 and it's the same!

JeanM

My 4x 2gb sticks arrive tomorrow for this test (aka low budget esxi host), so I'll get to do more testing between 4.0 and 5.0 versions, I don't have 5.5.

Asif Dasl

JeanM can you check the System Resource Allocation? I've attached the screenshot of my machine and that is where the missing 2.5Gigs of my RAM is going. I can't change it to 0 MB either, under vSphere Client or the Web Client.. Looks like I'll have to live with it.

JeanM

Sure, give me about half hour. I am making some changes to the host, and working on site to site vpn config over the internet.

I'll post a screenshot.

Essendon

Here's a screenshot of mine, though there's the Edit button to the right of the screen where it lets me adjust the reserved resources.



Here's another screenshot, this is from a nested ESXi machine. It does look like 1.55GB is reserved too, but there's the Edit button to the right that lets me adjust those settings.



Interesting nevertheless, though I guess this is for situations when there's serious memory overcommitment, the reserved memory ensures (to some extent at least) that the host doesnt have to swap to disk.

JeanM

Here is mine after a fresh reload.