Your Daily VMware quiz!

My take on guest RAM reservations:

I steer away from them as much as possible, but there are cases where for political/business reasons you have to have them (SLA's) . After all, if a critical workload needs a guarantee of RAM in times of contention - you have to be able to give it what it needs. The key there lies in the term "contention". Contention shouldnt normally occur in a well-designed and well looked-after environment. Ironically, I've seen contention being caused by memory reservations themselves! Ill-informed admins keep dishing out RAM reservations only to discover ballooning beginning to occur (this first starts showing up on SQL VM's and heavily used app VM's). So use them very sparingly. Use them on a case-by-case basis. Evaluate every RAM reservation request closely.

Most environments would almost certainly employ RAM reservations, and if you do you need to use %age reserved for failover as your admission control policy. This policy doesnt suffer from the very restrictive slot-based policy (which is the default). The only downside to the %age reserved policy is the need to evaluate your %ages from time-to-time and especially when you put add a new host to the cluster. You MUST change the %age when a host is added/removed.

While we are at it, there are 2 more things I'd like to add. The more the RAM you allocate to a VM, the more the:

- size of its .vswp file.
- size of its memory overhead.

You may think - Oh, it cant be too bad. Err, no. If you have say a VM with 32GB RAM, it's going to need a 32 GB .vswp file (which can sit in a datastore of your choice). You have 10 VM's like that across your environment, there goes 320GB of your swap datastore. Right, now let's reserve all their RAM, so there's no .vswp files for these 10 VM's. BUT, now your cluster has 320GB less RAM to play with. Other VM's which may need RAM are now going to be starved of it. The size of the memory overhead isnt insignificant either, a 32GB RAM and 4vCPU VM is going to have a fair overhead and with 10 or so of these VM's and you'll see like 2GB RAM (probably more) taken away by overheads alone.

Moral of the story - Employ reservations, but very sparingly. And you must use %age reserved for failover whether or not you use reservations.

To stir the discussion up a bit further: What is, in general, your (everybody's) take on .vswp files in general. vSphere 5 also added a VMX-SWAP to let the VM world swap to a file. More and more swapping seems to occur. I've taken a quick peek at our production cluster (small managed hosting provider) and we're running about 123 GB of VM's. I managed to keep away from reservations, but at the same time, that means that 123 GB of our SAN storage is being "wasted" on swap files, currently not being used because the environment is not (and hope never will be) under contention.

Used this nice PowerCLI script found on the VMware forums for this calculation, which shows you the memory provisioned for the entire cluster:

Get-Cluster Production | Get-VMhost | Select Name,@{N="Memory used MB";E={$_ | Get-VM | %{$_.ExtensionData.Summary.QuickStats.HostMemoryUsage} | Measure-Object -Sum | Select -ExpandProperty Sum
}}

You have a few options when configuring swap files:

Leave it at the default, which is store in the virtual machine working directory. This is default, but as in our situation, we've lost 123 GB to .vswp files.
Create a dedicated .vswp datastore based on shared storage. Pro's: you can ensure that the .vswp files end up on cheap storage. Cons: Requires configuration
Create a dedicated .vswp datastore based on local storage. Pro's: Easy to use a single disk (either SSD or local slow storage). Cons: Provides more vMotion overhead as the .vswp file needs to be copied around at every vMotion operation. Used in conjunction with DRS, this can create some significant overhead.

I'm currently leaning towards the second option for our production cluster, since we use a mixture of RAID-5 and RAID-10 volume groups, and since we've got no contention it's really a waste of space to have .vswp files on the more expensive RAID-10 storage sets.

Your thoughts?

Great discussion topic this Tom. Like with most things, there's a trade-off. I'd go with Option 2 every time, there's some manual configuration but the benefits of a separate datastore far outweigh the slight disadvantage.

1. Array replication is probably the most notable, by having a separate datastore that you do not replicate - you save on bandwidth and capacity.

2. Like Tom's said, you can put the .vswp files on cheap storage, we have ours on RAID 6 storage.

3. At the DR site, I'd recommend you dont dedicate a separate datastore for .vswp files - this datastore will always be sitting vacant, waiting for a site failover to occur. Just let the VM's .vswp file be with the VM's config file (default setting). .vswp files are recreated when a VM starts up at another site.

4. I'd never put swap files on local storage as much as possible. vMotion performance is affected because the swap files will also need to be moved to the destination host.

However, if you are not replicating to another site i.e. you are not using vSphere Replication or Array-based Replication, it's probably better to store the swap files with the VM. Same applies for SRM, if you dont have SRM, it's okay to store the swap files with the VM. It's all about how your environment's setup and what your future plans are.

kj0 wrote: »

HAHA... Yeah, vApp is what I meant, Heads all over he shop at the moment with all this study. vMotion and DRS at the moment.

When I get a second I'll do what I was originally going to. Do up some screenshots of the answer with vApps.

Inside your vApps you can set your boot priority for the order of which your VMs will start up in. 120seconds between each is generally the ballpark.

You can then set your reserves for the host memory for the VMs inside the vApp so that when your start your VM, it is guaranteed that Memory and can hold it.

I think that's right.

Gotta be careful about resource pools too bud. They are a PITA as far as I am concerned. The need to keep them going (manual adjustment of the values of CPU/RAM/disk) is enough to put me off of them. Consider the following:

1. In this case, there are 4 VM's in the High Resource Pool and 8 in the Normal Resource Pool. You've split up the your cluster into these two pools having allocated more resources to the High pool and fewer to the Normal pool. All's well till the number of VM's in the pools stays more or less constant.

2. As time goes on, and as things follow their course - you have admins that put VM's in the High pool at a system owner/manager's request or they are just careless and just chuck every VM in the high pool thinking there's plenty to go around.

Now though, there are more VM's in the same High pool. So more VM's sharing the same pie which means there's now less to go around - less resources for more VM's. Duncan Epping's called this the Resource Pie Paradox. More consumers, less resources.

So in essence, steer away from these resource pools, if you need to have them around, visit their resource allocations regularly to ensure your VM's arent being starved.

Hope this helps.

Pie1.JPG

Pie2.JPG

tomtom1 wrote: »

You mean a vApp. One thing I would most definitely stay away from is guest VM reservations, since it messes up HA slot sizes and not in a good way. Even if the company is currently not leveraging HA, it might in the future.

A vApp has some nasty implications though, if you decide to use one, you should understand the impact it has on the shares in times of resource contention. If you leave the shares at the default, and you start coming close to a saturated host, you might run into some problems with the calculation behind shares.

And HA doesnt respect any vApp startup order, you'll find HA will power up VM's in a random manner without regard to your vApp's power-up order.

Question 15:

You're still working for the same old company you helped with the Equallogic storage PSP. Since you took over from another admin, you've made some nice improvements to the environment. You notice, that a lot of the VM's are currently in snapshot mode. Some even go back as far as 2012!

You know from your studies, that this provides the following risks:

Increased capacity on the storage LUNs.
Decreased performance (increased read overhead)

Which tools could you use to identify VM's that are in snapshot mode. Multiple valid answers exist.

- vCOps
- Alan Renouf's vCheck is particularly awesome!
- Storage Views tab at the DC, Cluster, host and VM level
- Presence of a vm-00001.vmdk file in the datastore the VM resides in, perhaps you can do this via some esxcli command too, IDK
- Snapshot Manager, but looking at each VM will be like watching paint dry
- Get-VM | Get-Snapshot | ...

Off the top of my head, anyone have more ways? Great question

this turns into a Q&A session between TomTom1 and Essendon

jibbajabba wrote: »

this turns into a Q&A session between TomTom1 and Essendon

Everybody can post their take on these questions though, in fact, I (we) 'd love to see more participants! How we monitor snapshots is by using an alarm in vCenter, configured like this.

The pro of this solution is that it leverages the vCenter itself, so no need for external tools or scripts to add complexity to the environment. Smaller VM's will fall through the limits at the start, but they will be in warning or error state quick enough.

Screen Shot 2014-03-31 at 17.54.18.jpg

Screen Shot 2014-03-31 at 17.54.08.jpg

Question 16

You somehow continue working for the same company (as in the previous question). You rock up one morning and see that several VM's in several datastores are in the "paused" state. Upon furiously clicking around and scratching your head, you discover the datastores the VM's were on are full. These datastores are thin-provisioned at the vSphere layer and thick-provisioned at the storage layer.

a). How can resume your VM's? (multiple options exist)
b). How can you prevent this issue in the future?
c). Discuss the pro's and con's of

- thin provisioning at both the storage layer and the vSphere layer
- thin provisioning at the storage layer and thick-provisioning at the vSphere layer
- thick provisioning at the storage layer and thin-provisioning at the vSphere layer
- thick provisioning at both the storage layer and the vSphere layer

jibbajabba wrote: »

this turns into a Q&A session between TomTom1 and Essendon

LOL! Certainly not the intention!

I encourage everyone to participate, just type out your answer - no one's going to ridicule you. No one knows everything and I'm out here to learn too. So let's make this a good learning method. I do realize the difficulty of questions needs to be lowered for greater participation, but harder questions enhance the learning experience, dont they?!

Essendon wrote: »

Question 16

You somehow continue working for the same company (as in the previous question). You rock up one morning and see that several VM's in several datastores are in the "paused" state. Upon furiously clicking around and scratching your head, you discover the datastores the VM's were on are full. These datastores are thin-provisioned at the vSphere layer and thick-provisioned at the storage layer.

a). How can resume your VM's? (multiple options exist)
b). How can you prevent this issue in the future?
c). Discuss the pro's and con's of

- thin provisioning at both the storage layer and the vSphere layer
- thin provisioning at the storage layer and thick-provisioning at the vSphere layer
- thick provisioning at the storage layer and thin-provisioning at the vSphere layer
- thick provisioning at both the storage layer and the vSphere layer

Datastores are thin provisioned at vSphere Layer ? You mean the VMDKs are thin provisioned surely

a.1) Extend the LUN
a.2) Add an extend to the LUN
a.3) Remove Un-needed VMs
a.4) Move Powered OFF VMs via SSH to another LUN (Storage vMotion is unlikely to work as it still requires storage available on the LUN to work, same with removing snapshots.)

b.1) Set appropriate alerts / notifications
b.2) Check for snapshots regularly
b.3) Use Thick provision so remaining space is immediately apparent

As for pro and cons of either. I have done this over and over again and to be honest, that is too much for me to write at the moment as I have to jump into a meeting, but a good article about this can be found >> Here <<

jibbajabba wrote: »

Datastores are thin provisioned at vSphere Layer ? You mean the VMDKs are thin provisioned surely

a.1) Extend the LUN

Should be, increase the LUN, or add an extent to it. Otherwise great answer.

tomtom1 wrote: »

Should be, increase the LUN, or add an extent to it. Otherwise great answer.

I pull my "I am German" card on that one

jibbajabba wrote: »

I pull my "I am German" card on that one

I'm not a native speaker, so go right ahead, but allow me to use that one too

tomtom1 wrote: »

I'm not a native speaker, so go right ahead, but allow me to use that one too

Always .. great card to have ... I usually shoot myself when people don't know the meaning of "whom" or don't know the difference between lose and loose or their and they're

jibbajabba wrote: »

As for pro and cons of either. I have done this over and over again and to be honest, that is too much for me to write at the moment as I have to jump into a meeting, but a good article about this can be found >> Here <<

Going forth on this one, what do you use? In my environments, I mainly do thin (VMDK's) on thick (storage devices) because that gives me the best compatibillity with stuff like SDRS. I have situation where I'm doing thin on thin, but if I were to do a redesign, I'd go with thin on thick here. Since we design and manage smaller clusters, the vSphere and storage administrator are the same person (mostly me).

From a personal point of view, I'd like to spend less time on monitoring storage usage on my SAN (hence the disk storage devices here) and more in my VMware environment, as I can leverage SDRS and SvMotion to compensate for the lack of storage tiering.

Love to hear your take too, jibba.

tomtom1 wrote: »

Love to hear your take too, jibba.

We use Thin on Thin ... I don't know who made that decision .. I am the guy who supposed to support the infrastructure but I was on holiday when it was implemented. Anyway, I don't like Thin on Thin simply because I have seen customers out of a sudden using a heck of a lot more storage than anticipated. If your pool on the SAN runs low there is usually not much you can do to fix it unless you buy another shelf.

So yes, alerting / monitoring is very important (and staff not ignoring the mails for weeks *sigh*). As for thin provisioned LUNs. We use 4TB LUNs (again, for no real reason I don't think) and I personally think it is pointless to have thin provisioned LUNs at that size given the average VM size in our environment (we got 250GB - 1TB VMs) ... In cases like that I would either

a. Use 4TB LUNs - Thick provisioned VMDKs
b. Use larger LUNs - Thin provisioned VMDKs

and in both cases Thick LUNs

Simply I like to know where I am at with the storage. The problem I think is the business in most cases. Buy small, sell big, hope it works out

I was "Beta Testing" Dell Equallogics years back and I managed to make the Dell guy speechless. He was all over the Thin on Thin thing. So we received an early model (pre Production 4000 Series it was I think) so he left it with us (not leaving our office for the day obviously) and let me play with it. Took me 15 minutes to "blow it up".

I left the setup as is - at that point 500GB LUNs thin on thin .. so we had 10 VMs or so .. So I kicked off some clones ... some snapshots, some removals, some more clones, some uploads and deployed some OVFs.

I "overrun" the system to a point where the alerting was somewhat overrun and the whole thing locked up lol ..

Needless to say the Dell guy lost the colour of his face ... Was not recoverable ..

Ok, it was a pre-release model and it was probably unlikely to happen in production anyway. My point is, you rely very heavily on proper alerting when using thin on thin and in my experience, even thin on thick, caused problems eventually.

I would LOVE to hear from an architect about environments implemented where either implementation made perfect sense. VCDX anyone ?

By the way - speaking of thin provision and storage vmotion - depending on your SAN you may see that you have to keep reclaiming space using VAAI because of all those svmotions ...

In my environment, I have a mix of thin-on-thin, thin-on-thick and thick-on-thick. Boot LUN's are thick, the swap file LUN's are thick on storage and so are the vmdk's. There are several thin (storage) LUN's with thin vmdks' and with VAAI support not being the best with our HP 3PAR SAN, there have been a few instances where there have been out-of-storage issues. Oh and dead space reclamation with the 3PAR SAN is like trying to wake up a dead person, it just wont work. There's plenty of free space yet on the SAN, well over 2 times free storage than used. Thin-on-thick for me too, if I could do this all over again, I inherited this storage. My team needs to keep an eye on the SAN management console at all times, the alerting isnt all that great either.

My employer goes with what jibba said too - just hope it works out. Buy small, promise big and hope for the best!

Essendon wrote: »

Thin-on-thick for me too, if I could do this all over again, I inherited this storage. My team needs to keep an eye on the SAN management console at all times, the alerting isnt all that great either.

Weird, since 3PAR profiles to be an enterprise grade SAN..

Question 17:

Your company is leveraging the DvSwitch since it has Enterprise Plus licensing and creating every port group on every host is just a pain in the ***. However, you have noticed that traffic isn't spread very equally. What would be a nice way to ensure some loadbalancing on the active uplinks? If it's at all possible, you don't want to involve the network team.

Well yeah they claim 3PAR, Data Protector and Virtual Connect to be enterprise grade but in reality they are far from it. They make great hardware if you ask me, the software has generally been just as shocking. Dont get me started on the error messages in Data Protector, my favourite is - "An unknown critical error has occurred." But I digress, lets stick to our QOTD here.

tomtom1 wrote: »

Question 17:

Your company is leveraging the DvSwitch since it has Enterprise Plus licensing and creating every port group on every host is just a pain in the ***. However, you have noticed that traffic isn't spread very equally. What would be a nice way to ensure some loadbalancing on the active uplinks? If it's at all possible, you don't want to involve the network team.

A vDS would fit the bill here. The first pain point is having to create port groups over and over on multiple hosts. While doing this, you run the slight risk of human error resulting in different port group names across hosts and problems with vMotion can ensue. Host profiles are a good way to fix this particular problem, you create the host profile from a reference host and apply them to every other host and they all get the same settings (keep in mind a host needs to be in maintenance for a host profile to be applied to it).

Coming back to the question though, you need to be able load balance across active uplinks. A vDS with the load balancing policy - based on physical NIC load - should be used here. This particular policy requires very little config and certainly requires no involvement from the networks team, so no config's needed on your pSwitches. When ESXi detects that utilization of a NIC exceeds 75% over a 30s interval, it moves traffic over to another NIC.

Your only requirement is Enterprise Plus licensing, this level of licensing gives you the use of the vDS. Load balancing based on pNIC load is not available on a vSS.

Essendon wrote: »

Your only requirement is Enterprise Plus licensing, this level of licensing gives you the use of the vDS. Load balancing based on pNIC load is not available on a vSS.

You will still have to talk to your network team in case there are port channels setup.

For example, if you currently use IP Hash because of portchannels, changing to Physical NIC Load might disconnect you from the network until you disabled all but one port in the portchannel.

@jibba - agreed. Bloody Networks team!

Question 18:

You are the Virtualization Architect for your company. Your company is embarking on a new virtualization initiative in which they wish to virtualize a remote office. The remote office runs a mix of physical Windows and Linux servers all running critical applications that generate revenue for them.

The physical Windows servers are HP DL380 G7’s 3Ghz 2-way (8 cores per proc.) with 96GB RAM and 2 x 10GbE cards in them, with hyperthreading enabled in the BIOS. There are 4 of these servers.

The physical Linux servers are also HP DL380G7’s 3Ghz 2-way (8 cores per proc.) with 96GB RAM and 2 x 10GbE cards in them, with hyperthreading enabled in the BIOS. There are 4 of these servers too.

A detailed capacity analysis of this environment over a period of 2 weeks has yielded the following info:

- The average RAM utilization on the Windows servers is 12GB and the peak utilization is 36GB.
- The average CPU utilization on the Windows servers is .5Ghz and the peak utilization is 1.5Ghz.
- The average RAM utilization on the Linux servers is 8GB and the peak utilization is 32GB.
- The average CPU utilization on the Linux servers is .5Ghz and the peak utilization is 1.5Ghz.

Constraints:

- The company wishes to reuse the physical servers as ESXi servers.

Requirements:

- In the first phase of this project, the first VM’s you create must be able to support the existing workload
- In the second phase of the project, you will also be needed to build more Windows VM’s with the following specs:

o 20 VM’s each with 16GB RAM and 2 vCPU’s
o 10 VM’s each with 8GB RAM and 1 vCPU
o 1 monster VM with 96GB RAM and 12 vCPU’s

How many ESXi hosts will you need all up? Leave some room for overhead and spike in usage (upto 25%)

How many clusters will you create?

What admission control policy will you use? You need to ensure that the company makes maximum use of its hosts but still manages to restart all VM’s on a host if it fails.

** The quotes are just to identify the questions. **

The company will rebuild the applications on these existing physical servers from backups. For the purposes of this question, ignore how they are going to do this.

Allright, here goes:

Requirements gathering:
-> The platform should support the existing workload.
-> The platform should account for new virtual machines that need to be deployed. Workload unknown at this point.

Assumptions:
-> New workloads should have a maximum workload of 50% of the configuration.

Constraints
-> Existing hardware should be reused in this project.

Capacity Analysis - existing workload:
Current number of workload: 8 (based on the existing 8 servers)

Average CPU capacity = 3 GHz * 16 = 48 GHz.
Average CPU capacity peak = 8 * 1,5 GHz = 12 GHz
Capacity necessary to support current physical infrastructure: 12 GHz

Average Memory capacity = 96 GB
Average Memory capacity peak =8 * 34 GB (32 + 36 / 2) = 272 GB
Capacity necessary to support current physical infrastructure: 272 GB.

Capacity Analysis - new workload:
Expected growth: 31 VM's (based on the information given). The assumption is made that the VM’s will not exceed 50% workload.

Maximum CPU capacity Large VM’s = 1vCPU = 3 GHz * 20 = 30 GHz.
Maximum CPU capacity Small VM’s = 0,5vCPU = 1,5 GHz * 10 = 15 GHz.
Maximum CPU capacity Monster VM = 6 vCPU = 3 GHz * 1 = 18 GHz.
Capacity necessary to support new workload: 63 GHz

Maximum Memory capacity Large VM’s = 8 GB * 20 = 160 GB.
Maximum Memory capacity Small VM’s = 4 * 10 = 40 GB.
Maximum Memory capacity Monster VM = 48 GB.
Capacity necessary to support new workload: 248 GB.

Total CPU capacity necessary: 75 GHz
Total Memory capacity necessary: 520 GB

Total CPU workload available: 48 GHz * 8 = 384 / 100 * 75 = 288 GHz
Total Memory workload available: 96 * 8 = 768 / 100 * 75 = 576 GB

We can handle the current and new workload with the resources we have available, but once we go into the necessary redundancy (N+1) we have too few resources (memory based) to provide.

N.B. Based on the assumption that the new workload will not exceed 50% capacity and the fact that memory reclamation techniques (specifically TPS) are taken out of the calculation, we indeed have insufficient memory to support necessary redundancy. When we lower the estimated capacity, sufficient resources will be available.

Since we have homogenous hosts I would create one cluster, and only deviate from this if there is a specific requirement laid out from the business.

As admission control, I would go with the percentage based approach as this will allow for different VM’s (sizes) in the cluster and allows the business to leverage the best possible out of the vSphere cluster.

Reserved memory resources: 13%
Reserved CPU resources: 13%

When sizing the VM’s correctly, the performance degradation should be non-existent to slim when a HA event occurs.

Nice write-up .. But since the hosts are are same in size, surely you could still go with admission control n+1 ?

What % would you set your admission control ? 12.5% ? 25% ?

jibbajabba wrote: »

Nice write-up .. But since the hosts are are same in size, surely you could still go with admission control n+1 ?

What % would you set your admission control ? 12.5% ? 25% ?

Admission control percentage at 13%, like I said, which is basically 12,5% rounded up. With 25 you receive an N+2 solution. I understand what you mean (thought of that), and I remember to have looked at a table somewhere that said, with X number of hosts in the cluster, it's best to have Y number of failover hosts. Been looking for that table though.

What if we like, build this scenario out? Do something which will relate to all areas of an infrastructure design:

-> Compute
-> Network
-> Storage
-> Management
-> Guest VM

Perhaps that's an idea? We can discuss pro's and con's per (high level) solution?