So you want to build a VSphere 5 lab

Introduction:
I've decided to beging studying for the VCP 5 exam since it is what we are using at work. After alot of research I found many of the information available on building a whitebox Vsphere 5 was limited. Considering vsphere 5 has only been out for a few months it is understandable. I wanted to create a post to help others looking to build a lab with desktop hardware that may not be on VMWare's HCL. I took many of the ideas from Julian Regel's blog: Living on the Cloud: Building a vLab Part 1: The Design
Goal:
To build a virtualization server to handel my needs for VCP and other certifications. To minimize the footprint by using one box still be able to run nested VM's so I could work with vmotion, vcenter and shared storage.
Hardware Requirements:
Processor:
There were some gotcha's when piecing my build together. I learned you want a processor which has VT-D.
I/O MMU virtualization (AMD-Vi and VT-d)Main article: IOMMU
An input/output memory management unit (IOMMU) enables guest virtual machines to directly use peripheral devices, such as Ethernet, accelerated graphics cards, and hard-drive controllers, through DMA and interrupt remapping. This is sometimes called PCI passthrough.[30] Both AMD and Intel have released specifications:
AMD's I/O Virtualization Technology, "AMD-Vi", originally called "IOMMU".[31]
Intel's "Virtualization Technology for Directed I/O" (VT-d).[32]
This elimitnate the K-series of Intel processors. The i5 2500 and i7 2600 processors have this feature and have been confirmed to work with certain motherboards.
Motherboard:
For VT-D to work, you must find a motherboard that not only has this feature but allows you to enable it. Several boards have been reported to have it greyed out, but turned on later with a bios update. After reading about Tinkertry's vZilla project “vZilla†Z68/Core i7/LSI 9265-8i – built for virtualization, storage, and speed | Tinkertry and the different motherboard's he tested with I opted for a ASRock Z68 Extreme3 Gen3. To me it had the best bang for the buck. The onboard NIC works with vSphere. I did not see the need to spend another 60 dollars for an extra nic and more sata\usb ports.
Ram:
I picked up 2x4GB sticks of Blue Kingston ram for $30AR. I plan on getting another 8GB in the next few weeks.
Storage
There are several options available. You can buy a NAS device or build one. Some of the options are: OpenFiler, FreeNAS, and Starwind. You can build a dedicated NAS on spare pc hardware or you can use a vm appliance. They allow you to use software raid and support Iscsi. I have not setup my storage pool at this time. I already have several 500gb, 1TB and 2TB drives I am able to use.
USB Drive: I am running Vsphere off a 8GB thumb drive I had already. A smaller drive will also work as ESXi is only using 900mb.
Case:
I have a 20u rack for my cisco lab and decided to get a 4u rackmount case.
I had Vsphere up and running in less than an hour. I do not have a DVD ROM so I created a bootable usb drive to install from. I found the instructions here: How to make a bootable VMware vSphere 5 USB installation media | satheesh.net
Cost:
It cost me about $560.00. Not a bad price in my book.
ASRock Z68 Extreme3 Gen3 Motherboard : 132.00
Intel i7 2600 processor: 270.00
16 GB Ram: 60.00
Rosewill RD400-2-DB 400W PS 25.00
Rosewill RSV-L4000 Black Metal 75.00
I've decided to beging studying for the VCP 5 exam since it is what we are using at work. After alot of research I found many of the information available on building a whitebox Vsphere 5 was limited. Considering vsphere 5 has only been out for a few months it is understandable. I wanted to create a post to help others looking to build a lab with desktop hardware that may not be on VMWare's HCL. I took many of the ideas from Julian Regel's blog: Living on the Cloud: Building a vLab Part 1: The Design
Goal:
To build a virtualization server to handel my needs for VCP and other certifications. To minimize the footprint by using one box still be able to run nested VM's so I could work with vmotion, vcenter and shared storage.
Hardware Requirements:
Processor:
There were some gotcha's when piecing my build together. I learned you want a processor which has VT-D.
I/O MMU virtualization (AMD-Vi and VT-d)Main article: IOMMU
An input/output memory management unit (IOMMU) enables guest virtual machines to directly use peripheral devices, such as Ethernet, accelerated graphics cards, and hard-drive controllers, through DMA and interrupt remapping. This is sometimes called PCI passthrough.[30] Both AMD and Intel have released specifications:
AMD's I/O Virtualization Technology, "AMD-Vi", originally called "IOMMU".[31]
Intel's "Virtualization Technology for Directed I/O" (VT-d).[32]
This elimitnate the K-series of Intel processors. The i5 2500 and i7 2600 processors have this feature and have been confirmed to work with certain motherboards.
Motherboard:
For VT-D to work, you must find a motherboard that not only has this feature but allows you to enable it. Several boards have been reported to have it greyed out, but turned on later with a bios update. After reading about Tinkertry's vZilla project “vZilla†Z68/Core i7/LSI 9265-8i – built for virtualization, storage, and speed | Tinkertry and the different motherboard's he tested with I opted for a ASRock Z68 Extreme3 Gen3. To me it had the best bang for the buck. The onboard NIC works with vSphere. I did not see the need to spend another 60 dollars for an extra nic and more sata\usb ports.
Ram:
I picked up 2x4GB sticks of Blue Kingston ram for $30AR. I plan on getting another 8GB in the next few weeks.
Storage
There are several options available. You can buy a NAS device or build one. Some of the options are: OpenFiler, FreeNAS, and Starwind. You can build a dedicated NAS on spare pc hardware or you can use a vm appliance. They allow you to use software raid and support Iscsi. I have not setup my storage pool at this time. I already have several 500gb, 1TB and 2TB drives I am able to use.
USB Drive: I am running Vsphere off a 8GB thumb drive I had already. A smaller drive will also work as ESXi is only using 900mb.
Case:
I have a 20u rack for my cisco lab and decided to get a 4u rackmount case.
I had Vsphere up and running in less than an hour. I do not have a DVD ROM so I created a bootable usb drive to install from. I found the instructions here: How to make a bootable VMware vSphere 5 USB installation media | satheesh.net
Cost:
It cost me about $560.00. Not a bad price in my book.
ASRock Z68 Extreme3 Gen3 Motherboard : 132.00
Intel i7 2600 processor: 270.00
16 GB Ram: 60.00
Rosewill RD400-2-DB 400W PS 25.00
Rosewill RSV-L4000 Black Metal 75.00
Currently reading: Start with Why: How Great Leaders Inspire Everyone to Take Action
Comments
The Intel K series CPUs are meant for overclocking, which is not something you need to do when setting up a lab to learn virtualization. I don't see why you used a desktop CPU for what you're trying to build here. A Xeon E3-1230 would have been cheaper, and better suited for this task. Although you would have needed a different motherboard for that.
What you've really built here is a budget gaming PC. Nothing wrong with that. I use my main PC for everything.
Got two of those in my personal lab
But my favorite bit in those are these little beauties
ATP Electronics, Inc. > Product > SIP eUSB SSD
GetCertified4Less - discounted vouchers for certs
AGC1, CLC1, GAC1, INC1, CTV1, INT1, BVC1, TBP1, TCP1, QLT1, HHT1, QBT1, BBC1 (39 CUs), (0 CUs) (0 CUs)
WFV1, BNC1, EAV1, EBV1, COV1 | MGC1, IWC1 | CQV1, CNV1, IWT1, RIT1 | DRV1, DSV1, TPV1, CVV1 | EUP1, EUC1, DHV1| CUV1, C173 | BOV1, CJV1, TXP1, TXC1 | TYP1, TYC1, SBT1, RGT1 (84 CUs) DONE!
I have used two cross over cables and NIC cards on the client machine for practice. The set up is wicked fast, and making a nice space heater under my desk so I don't even need central air.
AGC1, CLC1, GAC1, INC1, CTV1, INT1, BVC1, TBP1, TCP1, QLT1, HHT1, QBT1, BBC1 (39 CUs), (0 CUs) (0 CUs)
WFV1, BNC1, EAV1, EBV1, COV1 | MGC1, IWC1 | CQV1, CNV1, IWT1, RIT1 | DRV1, DSV1, TPV1, CVV1 | EUP1, EUC1, DHV1| CUV1, C173 | BOV1, CJV1, TXP1, TXC1 | TYP1, TYC1, SBT1, RGT1 (84 CUs) DONE!
AMD FX-8120 8-core Bulldozer
16GB Corsair Memory
3x2TB Samsung F4 disks
Is nice.
So does that give you 16 virtual processors in vmware?
What motherboard did you use?
1 box with 1 power supply is all it takes.
Hardware is
i7-950
24GB ram
1 x SSD and 3 x regular HDDs in varying sizes.
Running Win 7 Utimate
Have VMware workstation 7 installed.
Installed 2 x ESXi inside of vmware Workstation.
A couple of Openfiler VMS outside of ESXi
A few guest MS Servers including vCenter inside ESXi.
That's all I need to test out everything. Plus, I can do anything else I need to with my PC and leaving all these running in the background.
The downside is, if I need to reboot my box, I had to reboot everything inside the lab.
Since my server is a first gen version, there are no internal USB ports. The solution I found was to get a SanDisk Cruzer Fit 8GB low-profile flash drive, and plug it into one of the front USB ports, while still being able to put the front faceplate back on with the flash drive inserted. Also installed the vSphere client on my desktop machine.
AGC1, CLC1, GAC1, INC1, CTV1, INT1, BVC1, TBP1, TCP1, QLT1, HHT1, QBT1, BBC1 (39 CUs), (0 CUs) (0 CUs)
WFV1, BNC1, EAV1, EBV1, COV1 | MGC1, IWC1 | CQV1, CNV1, IWT1, RIT1 | DRV1, DSV1, TPV1, CVV1 | EUP1, EUC1, DHV1| CUV1, C173 | BOV1, CJV1, TXP1, TXC1 | TYP1, TYC1, SBT1, RGT1 (84 CUs) DONE!
The T3500 uses standard DDR3 so I got 24GB for about $70 a month back. The T5500 uses the more $ ECC DDR3 but overall it's still affordable. The PE series are the real deal, but I can't stand the loud fans.
I hadn't looked at them, but I had looked the DELL XPS desktops as it was reported they ran properly with ESXi5. When it got down to it price was also a key factor. At this point I have four virtual esxi hosts, a virtual router, two windows 2k8 venters, a DC, freeNas vm, and a couple of hosts all running at the same time on a physical ESXi host. It meets all my labbing need within my budget and is close to what the VCP class I will be taking this month's setup will be.
Down the road, I'd like to get away from the nested enviroment and get a couple physical hosts for the lab setup.