High Performance Vsphere 5 LAB

DexterPark

Hello fellow virtualization enthusiasts! I am in the process of building a high preforming Vsphere home lab and would like the opinion/advice of those who are wiser than me when it comes to VMWare.

I will have two ESXi servers, one will be 4 because as far as I know ESXi 5 does not support CUCM 8.6 (CallManager), Unity, etc...
Cisco Unified Communications will be a big part of my virtual lab, some Linux VM's, and I will be hosting a sizable Windows Lab domain as well. All-in-All I will have somewhere in the ballpark of 20-30 VM's.

Here is my new system build that I am ordering:
Antec DF-85 Black Steel / Plastic ATX Full Tower Computer Case
G.SKILL Ripjaws X Series 16GB (2 x 8GB) 240-Pin DDR3 SDRAM DDR3 2133 (just bought 4 times which equals 64GB's of RAM!)
ASUS P9X79 DELUXE LGA 2011 Intel X79 SATA 6Gb/s USB 3.0 ATX Intel Motherboard with UEFI BIOS
Intel Core i7-3930K Sandy Bridge-E 3.2GHz (3.8GHz Turbo) LGA 2011 130W Six-Core CPU
OCZ Vertex 3 VTX3-25SAT3-120G 2.5" 120GB SATA III SSD
Antec High Current Pro HCP-1200 1200W ATX12V


As you can see I am dropping $2000 on this box, I have an older ESXi 4 server which I built 3 years ago, but the mobo died. It has an Antec 1200 case, a first gen core i7 processor, 16GB DDR3 RAM (1600Mhz), and 2x 300GB WD VelociRaptor Hard drives.

I was thinking of buying the above build 2x and just put the new mobo/CPU/RAM/SSD into the old Antec 1200 case and re purposing the older i7 CPU, 16GB RAM & 10K RPM HDD for a (Rack mountable) Data store. I am using an IoMega ix200 cloud edition as an iscsi data store for my old VM lab but I would like something with a little more Disk I/O as It will be hammered by my ESXi boxes.

So after all that setup my question is this: What would you do if you where in my shoes? What ideas do you have for the Datastore, and is there any cool builds that you have run across which would be helpful to model after? THANKS!

P.S. I know all this is way overkill for a lab so I just wanted to say that I am using all this to move away from "the cloud" I will be hosing my own lab, email, domain, call routing system, storage and just about everything else we trust third party's with. So this will be a 24/7 live system that will be in "production" as well.

mapletune

I'm also thinking about building some VMware labs. However, mine will be modest, min requirements, to run labs smoothly =p

So, it'll be interesting watching others go and build high performance gear. hehe, I'll be watching this thread!

DexterPark

mapletune wrote: »

I'm also thinking about building some VMware labs. However, mine will be modest, min requirements, to run labs smoothly =p

So, it'll be interesting watching others go and build high performance gear. hehe, I'll be watching this thread!

I'll make sure to take some pictures for you then. Should have it together by next month, I have to piece it out so that I don't exceed my monthly budget.

Essendon

Those specs will make an awesome system for sure, and if you are going to be running all those VM's - you need some serious grunt. But do you really need all this? Looks like you have money to spend, but do you need to? Most of the time VM's idle away in a lab environment, you may be able to scale down on some components of the lab. For example, a domain controller isnt going to need 4GB of RAM and lots of storage, even if you had all the roles on the same VM. Exchange will need decent amounts of RAM, especially if your going to be running DAG's, CAS's and all the rest of it. But are you working towards getting yourself an Exchange certification? If not, I dont see the point in setting up Exchange - other than bragging rights!

Think about what you want to achieve with the lab, you may end up spending money on stuff you dont need. Maybe spare some cash for some vendor workbooks/mock labs/practice exams or something else from which you may benefit more. These are just my thoughts about your lab, but if you do end up building these systems, I'll envy ya!

Here's what I have for my ESXi lab. Aint as much as yours, but serves me well and there's plenty of room for expansion.

A physical HP DL380 G5 with 32GB RAM and 300GB x 8 SAS disks. This machine runs ESXi 5 with the following VM's.

- A 3GB RAM VM running vCenter. This VM also runs my iSCSI Starwind SAN for shared storage for the vESXi hosts. This VM has lots of disk space so I can split it up multiple datastores for the vESXi hosts.
- A 1GB RAM VM running Active Directory and DNS. Only domain controller in the forest.
- 3 x 8GB RAM VM's running ESXi 5. These vESXi hosts run varying number of VM's. Shared storage is given to these hosts from the Starwind SAN. Each of these vESXi hosts runs on their own physical SAS disk, which makes for quick boot time, usually 40-50 seconds.
- A 512MB RAM VM running vMA for command line management of the vESXi hosts.

This setup makes for a decent vSphere lab, and to top it off I was able to snag all this for $150 AUD (about $170 USD). But then again I dont need to run Exchange or CUCM. Enough to get me by, think of the extra electricity you are going to burn. It's all about sustainability!

nhan.ng

You have a good budget, but you're over-spending it on things that are not important. Like do you really need to spend that much on that case? It's not like it's a gaming machine. Why not just get a rack-mountable case? They're cheap and since it's a lab, most of the time it'll just be hiding out of sight anyway.

Second, do you really need 1200watts PSU? Totally overkill. You're not running 2/3 vid cards on it. Each of my Esxi host is powered by a 380watts Antec CPU. I have i5-2400 plus 16gig in each of them. All this inside a 2U rack mount case.

If i was in your shoe, I'd budget it for 2 Esxi systems, a few SSD and a nice managed switch.

Network_Engineer

Here are the specs on my new server for my VMware Home Lab:

2U Server 2x Intel Xeon X5470 3.3Ghz Quad Core 16GB RAM 4x 250GB S5000PSL AC Power Supply

Chassis:
2U Rackable Systems C2004 Half Depth 4 SATA Bay

CPU:
2x Intel Xeon X5470 3.3Ghz Quad Core Processors super fast

Memory:
16GB RAM (8x 2GB)

Hard Drives:
4 x 80GB

Motherboard:
Intel S5000PSL
Intel S5000PSL Will Accept 5400 CPU's
Dual socket LGA771
8 Slots DDR2 FB Memory
Dual Gigabit Ethernet ports onboard
4x USB ports
PS2 mouse and keyboard

RAID:
Onboard Intel RAID Controller
LSI MR 8888-ELP RAID CARD

Power Supply:
450W AC Rackable Systems Power Supply

mapletune

I've seen references pointing to the use of 6 gigabit ports for redundancy. 2x port per each, vmotion or vswitch, or whatever (sorry, i don't know anything about vmware, but planning to learn!).

But yea, anyone want to elaborate on that? 6 gigabit ports per server.

That's like, 2x onboard, + 1x quad, or + 2x dual gigabit. =D

jibbajabba

I usually go for 2xGbE for Management, 2xGbE for traffic (usually vlan tagging on trunk), 2xGbE iSCSI .. Nowadays we go more and more for 2x10GbE for the lot (we usually separate things like vMotion using VLANs on the Management vSwitch).

sratakhin

We have 4 ports bundled together for VMs. However, only one GbE for management and one for SAN. Don't ask me about the last one It was set up this way and I'm not yet allowed to touch it.

DexterPark

Essendon wrote: »

Those specs will make an awesome system for sure, and if you are going to be running all those VM's - you need some serious grunt. But do you really need all this? Looks like you have money to spend, but do you need to? Most of the time VM's idle away in a lab environment, you may be able to scale down on some components of the lab. For example, a domain controller isnt going to need 4GB of RAM and lots of storage, even if you had all the roles on the same VM. Exchange will need decent amounts of RAM, especially if your going to be running DAG's, CAS's and all the rest of it. But are you working towards getting yourself an Exchange certification? If not, I dont see the point in setting up Exchange - other than bragging rights!

Think about what you want to achieve with the lab, you may end up spending money on stuff you dont need. Maybe spare some cash for some vendor workbooks/mock labs/practice exams or something else from which you may benefit more. These are just my thoughts about your lab, but if you do end up building these systems, I'll envy ya!

Here's what I have for my ESXi lab. Aint as much as yours, but serves me well and there's plenty of room for expansion.

A physical HP DL380 G5 with 32GB RAM and 300GB x 8 SAS disks. This machine runs ESXi 5 with the following VM's.

- A 3GB RAM VM running vCenter. This VM also runs my iSCSI Starwind SAN for shared storage for the vESXi hosts. This VM has lots of disk space so I can split it up multiple datastores for the vESXi hosts.
- A 1GB RAM VM running Active Directory and DNS. Only domain controller in the forest.
- 3 x 8GB RAM VM's running ESXi 5. These vESXi hosts run varying number of VM's. Shared storage is given to these hosts from the Starwind SAN. Each of these vESXi hosts runs on their own physical SAS disk, which makes for quick boot time, usually 40-50 seconds.
- A 512MB RAM VM running vMA for command line management of the vESXi hosts.

This setup makes for a decent vSphere lab, and to top it off I was able to snag all this for $150 AUD (about $170 USD). But then again I dont need to run Exchange or CUCM. Enough to get me by, think of the extra electricity you are going to burn. It's all about sustainability!

Thanks for your input! I agree that I should scale down a little bit. I did want to point out that I am going to study for a lot of Microsoft certs, and that I won't have one Exchange VM but four. Same thing for A.D, SharePoint, etc... I have ran all of those standalone and now I am ready to move into the advanced setup's/labs. The main reason I am buying 2x of these is because I want almost every MS, VMWare & Linux cert that are related to infrastructure after I finish all my Cisco's certs.

What do you think would make a great High performance Datastore? Thanks!

mapletune

Oh, I remember, I heard it from Packet Pushers: Show 115 – What the FEX ? Cisco UCS

at about 26 minutes. actually, for vmware stuff go to 42 minute mark.

They were talking about Cisco UCS, and Fabric Extenders (FEX), and how every blade has an A side and a B side data path to the fabric interconnect up-link. And how VICs can present to the baremetal os one nic for each a/b path. If one path fails, upstream management will update the mac address tables transparently so that the host uses the backup nic, but the host never knows about the switch.

But yea, as jibba said, used to be 2x gigabit nics for each, but now best practice is moving onwards to 10gbE with virtual nics.

Good stuff, thanks for clarification Jibba =D

DexterPark

nhan.ng wrote: »

You have a good budget, but you're over-spending it on things that are not important. Like do you really need to spend that much on that case? It's not like it's a gaming machine. Why not just get a rack-mountable case? They're cheap and since it's a lab, most of the time it'll just be hiding out of sight anyway.

Second, do you really need 1200watts PSU? Totally overkill. You're not running 2/3 vid cards on it. Each of my Esxi host is powered by a 380watts Antec CPU. I have i5-2400 plus 16gig in each of them. All this inside a 2U rack mount case.

If i was in your shoe, I'd budget it for 2 Esxi systems, a few SSD and a nice managed switch.

Great Idea, I was going to hook these up to my 2x 2960's (All Gig ports) and have some sort of gateway redundancy. I am going a little overboard, but not because I think I need to. The only reason I go all out is because it makes me happy. The main reason I picked that case is because of the cooling capability, and the easily removable dust filters. It also has an external HDD/SSD dock so that I could pop in a drive with windows 7 on it and use it as a gaming machine while my VM's run off the other ESXi box. I do want a rack mountable datastore (Even though I am running out of rack space) do you have any idea's on high performance power supplies that have energy saving capabilities? Thanks!

mapletune

DexterPark wrote: »

do you have any idea's on high performance power supplies that have energy saving capabilities? Thanks!

I use enermax 80PLUS Gold. I'm really happy with it. Dead quiet, but still a bit warm. Maybe 200V 80PLUS Titanium can run like ice... but i dunno~

but yea, for standard atx power source, look for 80PLUS or similar designation.

nhan.ng

DexterPark wrote: »

Great Idea, I was going to hook these up to my 2x 2960's (All Gig ports) and have some sort of gateway redundancy. I am going a little overboard, but not because I think I need to. The only reason I go all out is because it makes me happy.

That was what i did too. Bought all 3-i7 2600 until i realized i'd run out of memory before cpu power, so i sold it for the 2400.

The main reason I picked that case is because of the cooling capability, and the easily removable dust filters. It also has an external HDD/SSD dock so that I could pop in a drive with windows 7 on it and use it as a gaming machine while my VM's run off the other ESXi box. I do want a rack mountable datastore (Even though I am running out of rack space) do you have any idea's on high performance power supplies that have energy saving capabilities? Thanks!

If you buy 2 rackmount, you're saving tons of space

Like other poster has commented, get 80plus power supply. Mine is a bronze. You can always spend some on the Platinum to cut down on the power usages.

Newegg.com - Computer Hardware, Power Supplies, Power Supplies, 80 PLUS PLATINUM Certified

joehalford01

This thread is pretty timely. I'm thinking about building a better server as well. Have you looked at the Supermicro barebones kits? They have some serious specs and expandability. You probably be out about $2000 to buy the kit (about $1000) and then fill it with hard drives, some memory, and hard-drives. With a LOT of room to grow.

Newegg.com - SUPERMICRO SYS-7047R-TRF 4U Rackmountable / Tower Server Barebone Dual LGA 2011 Intel C602 DDR3 1600/1333/1066/800

DexterPark

joehalford01 wrote: »

This thread is pretty timely. I'm thinking about building a better server as well. Have you looked at the Supermicro barebones kits? They have some serious specs and expandability. You probably be out about $2000 to buy the kit (about $1000) and then fill it with hard drives, some memory, and hard-drives. With a LOT of room to grow.

Newegg.com - SUPERMICRO SYS-7047R-TRF 4U Rackmountable / Tower Server Barebone Dual LGA 2011 Intel C602 DDR3 1600/1333/1066/800

Got to admit that is very tempting! The only real reason not to is because of the Motherboard only supporting up to 1600Mhz. On the plus side it does allow for up to 512GB of ram which I could never hope to fill it up with.

The Mobo I picked has speeds up to 2400Mhz and supports up to 64GB of RAM:
ASUS P9X79 DELUXE LGA 2011 Intel X79 SATA 6Gb/s USB 3.0 ATX Intel Motherboard

It has a ton of over features too but not many more related to VMWare.

reppgoa

Tinkertry.com

Read and learn.

Your storage solution can be beefed up significantly. Read about his use of the LSI raid card and SSD caching.

jibbajabba

Here is my lab setup with minimum power consumption.

1x HP Microserver N36L, 8GB of Ram providing two tier storage (powered by an Adaptec 3805)

4x 2TB Raid 6
4x 300 GB SAS 2.5" 15k Raid 10 (in special drive caddy from Supermicro)

Running CentOS, presenting the Raid 6 as NFS and Raid 10 as iSCSI. Also acts as VPN Server, Router and has an old Oracle instance from an old vCloud Director Setup.

4x HP Microserver N40L (3x 16GB, 1x8GB of Ram)

8GB Microserver runs 2008R2, vCenter, AD and Veeam B&R (mapped storage from #1)
16GB Microserver run ESXi from USB / Stateless with LUNs mapped from #1

Each server has an additional dual nic.

It is not the fastest storage, but I can easily run 10+ VMs.

The Raid10 is just a test because I used to run Raid0 to give me the I/O and used Veeam to protect the VMs, might revert to this again or just throw them out and use SSDs instead.

I think this is the cheapest solution, especially in terms of power consumption.

Granted,I had the raid card, ram and SAS disks spares, but you have to buy storage and ram anyway, and Raid card should really be used anyway (there's always eBay).

pumbaa_g

If its 24 by 7 reliability that you are looking for then you may want to explore the server components due to reliability & power options. Other then that if its just a lab then this build will work fine.
I have a budget that I need to work on so going for the following. This will be my workstation with Win 7 64 bit and VMWare Workstation 8
AMD Zambezi FX-8120 8 Core Processor
Asus 970X Motherboard
32 GB G Skill RAM 1600 Mhz
Kingston 120 GB SSD
3 TB WD Green Drive x 2
GT550 Ti Video Card (Not sure about this one yet)
430 Watts PSU (preferably something quiet)
2nd Machine my old Desktop will be the ESXi Server
AMD Phenom 2 Quad Core
Gigabyte Motherboard
6 GB of DDR2 RAM (one module has failed so will replace that sometime in the future)
1 TB Hard Disk
I am doing my VCP currently, exam on Nov 29th. Any suggestions will be welcomed, I am on a tight budget and not able to spend as much as I would like as I still have to spring for my ICM Classes plus not able to get my hands on a decent budget server build where I am located.

Essendon

@pumbaa, that kind of machine will definitely get you by mate. I got by with a machine running an E6550 (the cheapest processor that supports VT-x), a 2TB disk and 8GB RAM. Granted it wasnt the quickest system by any stretch of the imagination, but was enough for most labs. I was on a tight budget and went with the bare minimum. Max out the RAM on your current machine, if you can get it upto 8GB RAM and you throw in an SSD, you'll be laughing. Where are you doing the ICM class? I did it for $1000 AUD from UCSC, cant beat that price really. Make sure you get tons of hands-on, it isnt any easy exam. Work on your storage and networkin as much as you can, you'll do just fine.

pumbaa_g

I am in India got a few quotes from some VMWare education partners and still trying to negotiate a better deal. As of now, I am getting ICM for approx USD 600. Trying to get some corporate discounts as well. I am worried about Storage & Networking as well so started reading through the Storage+ material which seems to be helping.
I have ordered the "Mastering vSphere 5" and reading up "Vmware Certified Professional on Vsphere 5 Study Guide" by Brian Atkinson as of now but thinking of springing for "Vmware Vsphere 5 Clustering Technical Deepdive" Book. Do you think it will be useful to cover Storage & Networking?
Thanks for your inputs!

networkjutsu

Nice, $2K for VMware lab.

Here's mine:

Lian Li PC-V351B
Supermico MBD-X9SCL+-F
Intel Xeon E3-1230 V2 Ivy Bridge
Antec EA-380D Green
2 x Kingston 8GB DDR3 SDRAM ECC Unbuffered
SanDisk Cruzer Blade 4GB USB Flash Drive

This whitebox is connected to my Synology DS1812+ with NFS share.

Essendon

pumbaa_g wrote: »

I am in India got a few quotes from some VMWare education partners and still trying to negotiate a better deal. As of now, I am getting ICM for approx USD 600. Trying to get some corporate discounts as well. I am worried about Storage & Networking as well so started reading through the Storage+ material which seems to be helping.
I have ordered the "Mastering vSphere 5" and reading up "Vmware Certified Professional on Vsphere 5 Study Guide" by Brian Atkinson as of now but thinking of springing for "Vmware Vsphere 5 Clustering Technical Deepdive" Book. Do you think it will be useful to cover Storage & Networking?
Thanks for your inputs!

The $600 price tag is awesome, go for it. The Mastering vSphere 5 book is worth its weight in gold, I reckon it's the one of the best technical books ever. I havent read the other book you mentioned, but the Mastering book is enough to get you past the test. The Technical Deepdive is also an awesome resource, great for reference purposes, a bit too much for the VCP. But it wouldn't hurt to go through it, the more knowledge you have, the better you are prepared for the test. It's only 300 pages and the most thorough resource on HA/DRS and all that jazz. If you Google VMware vSphere storage guide and vSphere networking guide, you'll come across 2 guides from VMware. I used both these guides extensively for my test, in fact I read them thrice cover to cover to make up for my lack of storage experience, helped to drill home the concepts.

Lab, lab and lab, then lab some more. And thou shalt not falter!

ITMonkey

Just a small FYI: make sure you get the C2 version of the Intel 9390K. The C1 version does not have VT-d support, which as you know is manditory for Hyper-V. (You mentioned you are going for Microsoft certs.) Here's Intel's datasheet: ARK | Intel® Core

And, although your whitebox machine will probably work, there are two similar powerstation motherboards that Asus officially support Server 2003, Server 2008 and Server 2008 R2 (and since they also support Windows 8, presumably they will also support Server 2012). They are the P9X79 WS and P8Z77 WS. Here is the Asus link for the P8X79, which is a near identical mobo to your P9X79 Deluxe. ASUS - Motherboards- ASUS P9X79 WS (Select the OS pulldown and see the OSs supported. Do the same with the Deluxe, and you'll notice none of the server products are listed under the Deluxe.)

Like I wrote, the Deluxe will probably work. Sometimes, though, it pays to be a little cautious. Asus's newest top-of-the-line gaming board, spanking new releases of Server 8 and Windows 8 client ... you might find yourself in a place you would rather not be (... at least for the first few months, anyway).

I'll be watching this topic over the coming weeks too. I am scouting out parts for a new lab PC build. Just like you, my interests are sliding towards Hyper-V and the new Microsoft offerings (and of course ... the certs!).

DexterPark

Thanks ITMonkey! I appreciate your advice!