So you want to build a VSphere 5 lab

odysseyelite

Introduction:
I've decided to beging studying for the VCP 5 exam since it is what we are using at work. After alot of research I found many of the information available on building a whitebox Vsphere 5 was limited. Considering vsphere 5 has only been out for a few months it is understandable. I wanted to create a post to help others looking to build a lab with desktop hardware that may not be on VMWare's HCL. I took many of the ideas from Julian Regel's blog: Living on the Cloud: Building a vLab Part 1: The Design


Goal:
To build a virtualization server to handel my needs for VCP and other certifications. To minimize the footprint by using one box still be able to run nested VM's so I could work with vmotion, vcenter and shared storage.

Hardware Requirements:

Processor:

There were some gotcha's when piecing my build together. I learned you want a processor which has VT-D.

I/O MMU virtualization (AMD-Vi and VT-d)Main article: IOMMU
An input/output memory management unit (IOMMU) enables guest virtual machines to directly use peripheral devices, such as Ethernet, accelerated graphics cards, and hard-drive controllers, through DMA and interrupt remapping. This is sometimes called PCI passthrough.[30] Both AMD and Intel have released specifications:

AMD's I/O Virtualization Technology, "AMD-Vi", originally called "IOMMU".[31]
Intel's "Virtualization Technology for Directed I/O" (VT-d).[32]


This elimitnate the K-series of Intel processors. The i5 2500 and i7 2600 processors have this feature and have been confirmed to work with certain motherboards.

Motherboard:

For VT-D to work, you must find a motherboard that not only has this feature but allows you to enable it. Several boards have been reported to have it greyed out, but turned on later with a bios update. After reading about Tinkertry's vZilla project “vZilla” Z68/Core i7/LSI 9265-8i – built for virtualization, storage, and speed | Tinkertry and the different motherboard's he tested with I opted for a ASRock Z68 Extreme3 Gen3. To me it had the best bang for the buck. The onboard NIC works with vSphere. I did not see the need to spend another 60 dollars for an extra nic and more sata\usb ports.

Ram:
I picked up 2x4GB sticks of Blue Kingston ram for $30AR. I plan on getting another 8GB in the next few weeks.

Storage
There are several options available. You can buy a NAS device or build one. Some of the options are: OpenFiler, FreeNAS, and Starwind. You can build a dedicated NAS on spare pc hardware or you can use a vm appliance. They allow you to use software raid and support Iscsi. I have not setup my storage pool at this time. I already have several 500gb, 1TB and 2TB drives I am able to use.

USB Drive: I am running Vsphere off a 8GB thumb drive I had already. A smaller drive will also work as ESXi is only using 900mb.

Case:
I have a 20u rack for my cisco lab and decided to get a 4u rackmount case.

I had Vsphere up and running in less than an hour. I do not have a DVD ROM so I created a bootable usb drive to install from. I found the instructions here: How to make a bootable VMware vSphere 5 USB installation media | satheesh.net


Cost:
It cost me about $560.00. Not a bad price in my book.

ASRock Z68 Extreme3 Gen3 Motherboard : 132.00
Intel i7 2600 processor: 270.00
16 GB Ram: 60.00
Rosewill RD400-2-DB 400W PS 25.00
Rosewill RSV-L4000 Black Metal 75.00

odysseyelite

Future use

odysseyelite

For future use

vCole

Great post! I will be creating a vSphere5 lab shortly, so I will be following this closely.

cyberguypr

Superb. I'm also redoing my vSphere lab and will contribute soon.

Everyone

I need to update mine to 5, I did a big series doing this with 4.1... you can basically transfer the same principle over to 5. (See: Budget Laboratory | Fix the Exchange!)

The Intel K series CPUs are meant for overclocking, which is not something you need to do when setting up a lab to learn virtualization. I don't see why you used a desktop CPU for what you're trying to build here. A Xeon E3-1230 would have been cheaper, and better suited for this task. Although you would have needed a different motherboard for that.

What you've really built here is a budget gaming PC. Nothing wrong with that. I use my main PC for everything.

jibbajabba

I like the idea.

Got two of those in my personal lab



But my favorite bit in those are these little beauties


ATP Electronics, Inc. > Product > SIP eUSB SSD

odysseyelite

Correct, the K series are for overclocking but they disable the VT-D. Yes, I could have gone Xeon, but the server boards were more expensive. I also looked at getting some Dell 8300 pc's as several people have used those for ESXI labs. I also could have done AMD and got more cores for the buck but I wanted to get the lab running fairly quick so I'm sure there are other ways of building a whitebox. Some people will also have these parts laying around from their old gaming pc's and they may want to know if they will work. I also tend to repurpose my machine's over the year. My dual AMD 939 has been a gaming pc, a media center pc and now a whs. The main points to pull from the write up to make sure you have a motherboard and processor capable of VT-D.

Bl8ckr0uter

+Rep. Excellent thread. I plan to use it in the future.

ChooseLife

Thank you for sharing your experience

LinuxRacr

Excellent. I am in the process of building a lab myself, and this thread gives some insight.

odysseyelite

I'm back on track for my VCP studies and will be updating the thread shortly. Tonight I managed to get two ESX virtual hosts running, a DC, a windows 2008 box for Vcenter, and a virtual router setup with networking.

forestgiant

I got two used Dell T5500's from Dell Outlet $520 each shipped that included 8GB, 80GB HDD, and a VT-D compatible CPU. Altogether after adding 24GB DDR3 ECC RAM and 2x1TB HDD, I paid around $1500, but it's nice being able to practice with vmotion and another prebuilt SAN.

I have used two cross over cables and NIC cards on the client machine for practice. The set up is wicked fast, and making a nice space heater under my desk so I don't even need central air.

LinuxRacr

I picked up a Dell PowerEdge 2950 for $300. It has 2 Intel Xenon 5150 dual-core 2.66Ghz processors (virtualization enabled), and DRAC5 card. It also has a PERC5 card. I'm hoping to be able to get some decent performance out of this one once I get 16 GB or ram, and decide which drives I'm going to use.

gorebrush

Just built me a VMWare Server: -

AMD FX-8120 8-core Bulldozer
16GB Corsair Memory
3x2TB Samsung F4 disks

Is nice.

odysseyelite

gorebrush wrote: »

Just built me a VMWare Server: -

AMD FX-8120 8-core Bulldozer
16GB Corsair Memory
3x2TB Samsung F4 disks

Is nice.

So does that give you 16 virtual processors in vmware?
What motherboard did you use?

xxxooxxx

My primary PC is also my vsphere lab. I don't really care about performance but it is something for me to explore all the features.

1 box with 1 power supply is all it takes.

Hardware is
i7-950
24GB ram
1 x SSD and 3 x regular HDDs in varying sizes.
Running Win 7 Utimate

Have VMware workstation 7 installed.

Installed 2 x ESXi inside of vmware Workstation.
A couple of Openfiler VMS outside of ESXi
A few guest MS Servers including vCenter inside ESXi.

That's all I need to test out everything. Plus, I can do anything else I need to with my PC and leaving all these running in the background.

The downside is, if I need to reboot my box, I had to reboot everything inside the lab.

LinuxRacr

I updated the BIOS on my PE 2950 to version 2.7.0 today and installed ESXi 5.0 on a flash drive.
Since my server is a first gen version, there are no internal USB ports. The solution I found was to get a SanDisk Cruzer Fit 8GB low-profile flash drive, and plug it into one of the front USB ports, while still being able to put the front faceplate back on with the flash drive inserted. Also installed the vSphere client on my desktop machine.

forestgiant

Have you considered a Dell T workstation? I actually have a T3500 and T5500 that I use with VMware ESXi 5 for labbing.

The T3500 uses standard DDR3 so I got 24GB for about $70 a month back. The T5500 uses the more $ ECC DDR3 but overall it's still affordable. The PE series are the real deal, but I can't stand the loud fans.

odysseyelite

forestgiant wrote: »

Have you considered a Dell T workstation? I actually have a T3500 and T5500 that I use with VMware ESXi 5 for labbing.

The T3500 uses standard DDR3 so I got 24GB for about $70 a month back. The T5500 uses the more $ ECC DDR3 but overall it's still affordable. The PE series are the real deal, but I can't stand the loud fans.

I hadn't looked at them, but I had looked the DELL XPS desktops as it was reported they ran properly with ESXi5. When it got down to it price was also a key factor. At this point I have four virtual esxi hosts, a virtual router, two windows 2k8 venters, a DC, freeNas vm, and a couple of hosts all running at the same time on a physical ESXi host. It meets all my labbing need within my budget and is close to what the VCP class I will be taking this month's setup will be.

Down the road, I'd like to get away from the nested enviroment and get a couple physical hosts for the lab setup.