VM Swap File Location: Pro's/Con's

Deathmage

Hey all,

I'm sure this gets well above VCP level knowledge, but would there be a benefit to storing a VM's swap files on say a DEll R610's local RAID 6/10 array compared to being stored on a SAN/NAS array that is connected, in this instance a 1 Gbit, to the server?

My logical, is if the VM is residing in memory, and also if the VM is residing in memory on that host where the local RAID array is located and not on another host (which at the point this question would be voided) then it seems logical that having the swap file on the local array would be way faster than going over a iSCSI or maybe even a FCoE fabric.

Does anyone follow my logic and/or where I'm going with this? - I'm just curious if storing the swap file other than the default with the VM which is on the storage array (in this case a SAN/NAS) - an example would be a IOPS sensitive application like SQL, I can see the benefit of having swap for that application locally allocated. I'm pondering if Exchange would be another good example since Exchange likes to 'eat' as much memory as possible over time but the only problem I see this is if the host dies, then the swap file is royally screwed...

I'm sure this goes beyond VCP but I'm quite curious.

Essendon

Which swap file are you referring to? The .vwsp file or the VM's guest OS swap file? Assuming you are talking about the .vwsp file, placing it on shared storage will:

- make vMotion quicker
- easier management. Create one dedicated .vswp file datastore to avoid having to replicate these files to your DR location. This file will be recreated when the VM's restarted over on the DR site, so why replicate it.

Placing it on DAS storage will:

- force the hypervisor to vMotion this file over too resulting in longer vMotions.
- When the environment is under contention though, it may be slightly better to have it sitting on DAS storage.

In addition, you need to read up on what this file's for. In one sentence, this file is for hypervisor swapping when it's under duress and in a well-designed environment, it's not normally used. I reckon this is VCP material.

tomtom1

Essendon wrote: »

- force the hypervisor to vMotion this file over too resulting in longer vMotions.
- When the environment is under contention though, it may be slightly better to have it sitting on DAS storage.

But, you could configure the swap files to be stored on local disks to remove that burden from the SAN. Imagine the scenario where the VSWP files are stored on local SSD's for fast swapping to ensure memory contention isn't as big a burden. However, in doing so, you're basically providing the same functionallity as host cache.

dave330i

Essendon wrote: »

Placing it on DAS storage will:

- force the hypervisor to vMotion this file over too resulting in longer vMotions.
- When the environment is under contention though, it may be slightly better to have it sitting on DAS storage.

Unless they changed this recently with share nothing vMotion, vswap file isn't vMotioned to the new host. It's recreated on the new host.

Essendon

@tom - agreed, pros and cons of both, and then not everyone has the luxury of local SSD (blades).

@dave - I always believed the .vwsp file was copied over during a vMotion, it does look like Maish, Duncan Epping and Frank Denneman think the same. Is there a VMware link to suggest otherwise mate?

Deathmage

dave330i wrote: »

Unless they changed this recently with share nothing vMotion, vswap file isn't vMotioned to the new host. It's recreated on the new host.

Well considering this is for the 5.1 exam, if it hasn't changed by 5.5 I bet it didn't in 5.1,

tomtom1 wrote: »

But, you could configure the swap files to be stored on local disks to remove that burden from the SAN. Imagine the scenario where the VSWP files are stored on local SSD's for fast swapping to ensure memory contention isn't as big a burden. However, in doing so, you're basically providing the same functionallity as host cache.

rofl, must be nice to think in the context of 5.5 ... yes I know I'm studying for the 5.1 exam and not the 5.5... there is a method to my madness.... I like learning since 5.5 is now only like what 2 years old, most people still use 5.1 or earlier so if I build off 5.1 and upward I'll know W-T-F to do in a scenerio using older ESXi clusters....

...however host cache seems really sweet, will love to learn that in 5.5..

Essendon wrote: »

Which swap file are you referring to? The .vwsp file or the VM's guest OS swap file? Assuming you are talking about the .vwsp file, placing it on shared storage will:

- make vMotion quicker
- easier management. Create one dedicated .vswp file datastore to avoid having to replicate these files to your DR location. This file will be recreated when the VM's restarted over on the DR site, so why replicate it.

Placing it on DAS storage will:

- force the hypervisor to vMotion this file over too resulting in longer vMotions.
- When the environment is under contention though, it may be slightly better to have it sitting on DAS storage.

In addition, you need to read up on what this file's for. In one sentence, this file is for hypervisor swapping when it's under duress and in a well-designed environment, it's not normally used. I reckon this is VCP material.

...just got back from changing my oil.... thanks for the responses guys!

Hmm... see that's what I was looking for, see my thinking is this, in a well-designed environment it's so true contention won't happen but I'm always thinking ahead so say its IDK 3 am in the morning and some user left a SQL query running at 4:45 pm and somehow made it just run, and run, and run. So by 3 am the SQL server has been running up the memory for a few hours and now it's paging swap. - (now I know if alarms are setup and such the chances of a host going into contention shouldn't happen) my thought process is this with having the VM's O/S swap file being on local storage. But see now that I think about it, I'm not sure if it's a real benefit or not...

....if the SQL server VM or the host I guess is the better part of the equation doesn't report memory being used like a siv, if the swap from the VM's O/S was on say the SAN and then datastore was thin (if it was thick I'm sure it would have the same ill-effect), then as the memory increased the datastore would increase to too the point of completely using up the datastore's allocated space. So would it be better to run into contention on the swap file, in this scenerio, on the local array or on the SAN/NAS that also has the datastores for the other VM's. MY thinking is this, if swap follow the same pattern as swap on a Windows physical server over time, the IOPS on a array in contention from the growing swap would bring down the performance of that drive that has other VM's and there swap files on it... We all know even though its a datastore logical, physically the SAN has 10 to 15k RPM drives and if you have a memory intensive IOPS like swap IDC how fast those drives are it will make the IOPS on that storage array sh*t bricks....

I wouldn't use this swap for a normal server like DHCP/ DNS/Terminal Server's but for program that CAN eat memory like a siv like Exchange, SQL, that's where my thinking is going...

it's more like a "ooo snap" buffer in-case a alararm doesn't go off and it's like a DR situation for like 8 hours while I'm sleeping... don't you all hate 4 am phone calls of server not working...

Does this make any sense?

Essendon

It was a little hard to follow your last post Trev and I think you've got the .vswp file and the guest OS swap mixed up?

Essendon

The .vswp file is vMotioned to the destination host, here's a VMware kb article > VMware KB: Storing a virtual machine swap file in a location other than the default in ESX/ESXi

Deathmage

Essendon wrote: »

It was a little hard to follow your last post Trev and I think you've got the .vswp file and the guest OS swap mixed up?

possibly, it could have happen.

I'm thinking if the memory in the O/S gets used up and lets just say this:

SQL server: 32 GB's of physical ram and 48 GB of swap.

My question is this, would it be better to keep the 48 GB swap file with the VM (this VM would reside on the datastore on the NAS/SAN) or make the O/S's swap file sit on the local storage array this way if it does page it's paging locally on the host and not having to go out the storage fabric to the SAN and they bring the IOPS on the NAS/SAN down in performance by all the memory swapping.

Does this make sense?

See maybe I'm getting the .vswp and the swap file inside the guest O/S confused.....

To me, the Guest O/S swap is for the memory inside of the OS, so if you make a VM get 32 GB's it gets 32 GB's but inside of windows it's swap file is 12 GB's or 1.5 of the physical whatever that number equates too..it's that swap file inside of windows that I'm referring too...

to me, the VM's swap file (.vswp) is the swap file for the memory allocated to it inside of the ESXi host or from within vCenter. IE if a VM has 32 GB's of ram with a 16 GB's reservation then the swap file would be 16 GB's in size...

with this being said I'm pretty sure I'm referring to the guest O'S memory or maybe I still think in the physical worl and not the virtual world....

Now this is why my thinking is convoluted since while the vswp file is the swap for the memory being used by the VM, does it also make sense to make the vswp file locally too not just the swap file inside of the O/S.

-Trevor

Essendon

It's best to have the OS's page file on shared storage, gotta get ready for work and I'll hopefully write a longer response when I reach my workplace this morning. Going by your last post, it does look like you have the .vswp file and the OS page file mixed up.

Deathmage

Essendon wrote: »

It's best to have the OS's page file on shared storage, gotta get ready for work and I'll hopefully write a longer response when I reach my workplace this morning. Going by your last post, it does look like you have the .vswp file and the OS page file mixed up.

Fair enough, will go re-read those a bit.

After reading them over, I'm referring to the .vswp file. As I see it now....

I'm still converting from physical administration to virtual so my mind thinks in the ol' school manner.

Cause if I thikn about it, it I made the O/S swap file inside of windows, if it was on a physical server it would be easy, but in a virtual world I had to mount it and then link the LUN's and maybe I'm thinking too much into it but that seems like a ton of administrative overhead....

so with that being said since in a VMware world the memory is on the host's then it seem my question is directed at the vswp file, if I was studying for 5.5 can see host cache on a SSD being a viable option. is it even possible to setup swap onto a SSD in 5.1?

I'm probably thinking to much into this, going back the blueprints, lol!

dave330i

Essendon wrote: »

@tom - agreed, pros and cons of both, and then not everyone has the luxury of local SSD (blades).

@dave - I always believed the .vwsp file was copied over during a vMotion, it does look like Maish, Duncan Epping and Frank Denneman think the same. Is there a VMware link to suggest otherwise mate?

Here's Frank:

(Alternative) VM swap file locations Q&A - frankdenneman.nl

If you look at your Frank Denneman link in DRS section, he mentions swap being recreated.

If you look at the VMware 5.5 docs it says copied:

http://pubs.vmware.com/vsphere-55/index.jsp#com.vmware.vsphere.vcenterhost.doc/GUID-B8F9F952-EEE7-4C23-A8F6-35E55232263F.html?resultof=%2522%2573%2577%2561%2570%2522%2520

which is odd. With share nothing vMotion in 5.1+ you could copy local disk info, but pre-5.1 you couldn't so copy wasn't possible.

Essendon

I'll test and report back Dave. I'll stick with the copying idea for now, here's another link that says it's copied > vXpress: Changing Swap file Location of VMware vSphere Virtual Machines

Deathmage

Essendon wrote: »

I'll test and report back Dave. I'll stick with the copying idea for now, here's another link that says it's copied > vXpress: Changing Swap file Location of VMware vSphere Virtual Machines

Doh! ..... that article is exactly what I was trying to ask....

See I'm confident some other fool in the world asks the same questions I ask all the time...

Essendon

We all gotta learn somehow! vSphere is a deep black hole, there's no end to the learning, ever.

dave330i

Essendon wrote: »

I'll test and report back Dave. I'll stick with the copying idea for now, here's another link that says it's copied > vXpress: Changing Swap file Location of VMware vSphere Virtual Machines

I think I know where the confusion is coming from. If the vswap file is in use, then it is copied during vMotion (makes sense, since there's active data in swap file). If the vswap file is not being used it's simply recreated on the destination host. It's the recreating the swap file that slows down vMotion.

Essendon

Yeah if the VM's powered on, the .vswp file is copied over. If the VM's powered off and restarted at another host/site, the file's recreated. Puts to rest our discussion.

Deathmage

Essendon wrote: »

Yeah if the VM's powered on, the .vswp file is copied over. If the VM's powered off and restarted at another host/site, the file's recreated. Puts to rest our discussion.

darkerosxx

Essendon wrote: »

Yeah if the VM's powered on, the .vswp file is copied over. If the VM's powered off and restarted at another host/site, the file's recreated. Puts to rest our discussion.

I read all the docs as saying if the vswp file is located on a common datastore that both hosts have access to and use as their swap location, then they don't move/copy/create it. Is that incorrect?

Essendon

That's not incorrect either. The key takeaway is - if the VM's powered on, the .vswp file is copied. If the VM's powered off and restarted at another host/site, the file's recreated.

darkerosxx

If both hosts use the same datastore for swap, I can't see why it would do either of those things.

Essendon

If the VMs powered off, the hypervisor deletes the file.

Before:



After:

dave330i

darkerosxx wrote: »

If both hosts use the same datastore for swap, I can't see why it would do either of those things.

It won't. On a shared datastore vmdk don't move, so nothing happens to vswap file.

darkerosxx

dave330i wrote: »

It won't. On a shared datastore vmdk don't move, so nothing happens to vswap file.

That's what I thought. Is Essendon saying something I'm not understanding?

Essendon

Probably should've worded it better.

1). When VM's powered up:

- On a shared datastore vmdk's don't move, so nothing happens to .vswp file.
- On a local datastore though, the .vswp file is copied.

2). When VM's powered down:

- The .vswp file is recreated.

darkerosxx

Yeah, okay, we're on the same page. Making sure I didn't misunderstand the way that worked.