Home Lab

paadams

Hey guys, new guy here. I signed up a while back and have read more than I posted. I currently updated my old VCP4 to VCP5-DCV. I'm beginning to work on VCAP5-DCD and VCAP5-DCA and I'm wanting to build a home lab. I've been using VMware Workstation but I'm not a fan of nesting ESXi and I'm running in to disk issues.

I was wondering what everyone is doing currently. I've read up on different blog posts I've come across and I've specd out some builds on an Intel E3 or E5 but each host is in the $1k range. I was really wanting 3 hosts but I'm not willing to spend $3k right now.

I've looked into the Intel NUC because I've seen builds in the $600/host range. I really hate the limitation of 16GB of RAM and 1 NIC though.

I have 32GB of RAM in my desktop but have hit RAM limitations as well when trying to run everything necessary for some test scenarios with vCAC etc.

I was wanting to build a whitebox that I could keep quiet and efficient. Prices on used Dell and HP servers are pretty decent but their noise and power consumption just won't allow their use in my home office.

I'm hoping someone has an idea that I haven't thought of.

Essendon

Go nested. 32GB RAM's enough for - 4 nested ESXi hosts (6GB RAM each), 4GB RAM for vCenter, 512MB RAM each for a DC and a Starwind iSCSI SAN, there's still some left over. You'll have disk issues before you run outa RAM here. I had 32GB RAM on my machine when I was doing the DCA and of course it wasnt the fastest thing ever but with 8 x 146GB SAS disks you can carefully place your hosts and other machines so there's minimum contention (at least disk wise), I easily got by. As the hosts will be nested, you can give them as many NICs as the physical machine will allow (32?) and play with just about every aspect of networking.

joelsfood

I use a combination of nested and physical with one small host. Between the host and my desktop, I have 10 cores, 48 gigs of ram, 4-5 TB of disk, ranging from software RAID1 7.2 SATA to hardware RAID10 SATA to SSD. Way more than enough for study. Of course, nothing beats hands on experience, but the lab is great when you already know what you're doing and can just fill in the corner cases you haven't tried. Someone just released a vapp with 3 node nested VSAN too (I have the link somewhere if you want it, I retweeted and bookmarked, though I haven't deployed it yet), which is great for me as that's one of few things I've only read on, haven't had a chance to use yet.

dave330i

I use VMware Hands on Lab. It's free and I can break it as much as I like.

paadams

I ran through piecing something together again today and came up with a decent build at $830/node but only 16GB RAM. Was thinking I could upgrade later.

After some discussions I'm thinking of building a single node with 32GB and running a nested solution.

I can run a DC, vCenter and something things from VMware Workstation and allow the full 32GB of RAM to the hosts. I was looking at a Xeon E3-1220V3 but may change to something that supports HT. Also may look at an E5 that supports more RAM.

I would be interested in that link regarding vSAN if you can find it.

joelsfood

OVF template for creating Nested ESXi 3 or 32 node VSAN Cluster | virtuallyGhetto

The E3-1220V3 would not be a bad choice at all for a small home lab. I only have the V2 (2 core), but I think you'll be fine with 4, particularly if you run some things in workstation. That's what I try to do, kind of balance out disk/cpu load between workstation and physical host. IE, workstation has more cores, physical host has more available ram and more space

Essendon

Whichever CPU you get just make sure it runs EPT or you wouldnt be able to nest 5.5 inside 5.5. As for a link for the Starwind software iSCSI SAN, here it is > https://www.starwindsoftware.com/starwind-virtual-san-free

joelsfood

I have 5.5 in 6 right now on the E2-1220V2, so you should be alright, but essendon is right, double check.

And yes, I have the Starwind SAN installed as part of my setup here too. Thanks, Essendon

paadams

Started looking at E5 options and it starts to get to expensive if I ever decide to add another node. I think I'll stick with the E3-1220v3 for now.

I have an i7-3770k and 32GB of RAM as my main desktop, so with that and a single ESXi host I think I'll be ok for a while. Starwind SAN looks promising as well. Thanks for the link.

I'm used to supporting a large environment so I was hoping to have at least 3 physical nodes but dang it gets expensive when I'm the one having to write the check.

paadams

I just ordered all the parts. Ended up going with a Intel Xeon E3-1231V3 since it has HT and was only $50 more.

Hopefully this will work well for what I need.

Thanks for the replies.

jibbajabba

Essendon wrote: »

Whichever CPU you get just make sure it runs EPT or you wouldnt be able to nest 5.5 inside 5.5. As for a link for the Starwind software iSCSI SAN, here it is > https://www.starwindsoftware.com/starwind-virtual-san-free

Do you install an extra VM for that? The reason I am asking is because it always surprises me that people choose starwind over the built-in iSCSI target shipped with Windows.

kj0

jibbajabba wrote: »

Do you install an extra VM for that? The reason I am asking is because it always surprises me that people choose starwind over the built-in iSCSI target shipped with Windows.

I had issues with the Built-in. but then again, i do prefer to use a NAS with OpenFiler or, better still, a proper NAS like a Synology. Planning on getting a Synology DS4xxxSlim.

Xavor

My lab is 2 hosts with E3-1230 v3,32GB, 4 port nics, and Supermicro motherboards. I built them over time as my needs grew. If I had a garage I would have bought a used server on ebay instead.

I looked at the NUCs, but the pricepoint is too high. Also looked at the HP microservers, but it seems you have to wait for holidays sales to get them at a decent price.

I'd like to build a third host, but it would be more of a want than a need. I'm leaning toward non-VMware tech currently so my lab is changing.

Essendon

jibbajabba wrote: »

Do you install an extra VM for that? The reason I am asking is because it always surprises me that people choose starwind over the built-in iSCSI target shipped with Windows.

Yeah you do install an extra VM for it but you could install Starwind on your vCenter machine too. For me, I've used Starwind for so long it's now just a habit. From what the other blogs out there say, performance is better with a Starwind iSCSI SAN vs a Windows iSCSI target. I haven't worked with the Windows iSCSI target so I cannot vouch for the comparison. But, yeah whatever's easier and works I guess!

kj0

Xavor wrote: »

My lab is 2 hosts with E3-1230 v3,32GB, 4 port nics, and Supermicro motherboards. I built them over time as my needs grew. If I had a garage I would have bought a used server on ebay instead.

Picked up a similar unit as a tester for a project at work - I haven't tested yet, but keen to see if it runs a nested environment well. (With EPT of course).

paadams

Thanks for the Starwinds tip. Makes configuring shared storage with a nested environment pretty darn easy.

Deathmage

I paid $310 each for my Dell R610's with 32 GB's of RAM each. with the $200 VMUG Advantage yearly sub I can license all three of them with Enteprise Plus with Operations Management w/ vCenter Standard. So $930 for 3 servers + $200 for VMware licensing and another $150 for a HP Procurve 2910al 48G-PoE+ L3 switch with a 172 Gbps throughput and $75 off ebay for a HP 2848 L2+ switch for iSCSI/vMotion, finally $600 total for a QNAP TS-420 NAS with 12 TB's of Space in a 4 bay RAID 5 on a bonded Gigabit uplink.

So my total expense:

Servers: $310 x 3 w/ Dual Quad Gigabit Nics= $930
VMware Licensing: $200
Switches: $225
NAS w/ iSCSI: $600
Technet Keys: $300
Cabling/Rack: $350
Firewall w/ licensing: $225
Cisco Lab: $1500

Total Home-lab cost: $4105~

cost to ownership....gratification that what you do with your home-lab can translate to successful deployments in a production environment after months and months of breaking your home-lab till it worked....PRICELESS!

ehnde

Wow that's quite a lab. How's your power bill though?

I'm taking the opposite approach - virtualize everything on one device: my desktop. This is perfectly fine for VCP-DCV studies and I'm going to try it for CCNA-Data Center when I'm done with the VCP. This is with an i5-2500k cpu and 24gb ram, 500GB SSD, 256GB SSD, and 64GB SSD.

Not sure if I'm doing the VCAP or CCNP-Data Center after those two certs, but either way I'm upgrading to a Xeon E5-2603v3 in my desktop so I'll have VT-d and a couple more cpu cores. And a motherboard that will support 128gb of ram. I'll start out with 64GB of ram and see how that goes.

The xeon E3 posted above is a good budget buy, but only supports 32GB ram.

Xavor

My lab is around 250w-300w on average at the wall. The CPUs are largely idle and it's more about RAM usage.

1 storage server with drives
2 esxi nodes
2 HP 1810 24 port switches
1 diy pfsense firewall
1 wireless router
1 motorolla cable modem
all servers have 4 port nics in them


It's overkill, but when you're learning fresh out of school it forces you to learn certain gotchas that a fully virtualized lab won't. Specifically: subnetting, cabling, vlans, and general design concepts. Separating services into vms instead of your router ( dhcp, dns). Labeling your cables and naming interfaces a certain way, etc. Oh, and moving and having to rebuild it all is fun too.

I'd like to replace my intel gigabit nics with fiber channel and/or infiniband to mess with at some point.

If you have years in the field and just want to learn VMware, do a nested setup and call it a day.