Bulding an ESXi Home Server

lordy

Dear all,

in an effort to reduce the hardware I have and getting some more exposure to VMware I was thinking about building a home server with ESXi 5 on it. I have come across a parts lists called the "Baby Dragon II" which seems like a good place to start. This lists recommends the following:

CPU: Intel Xeon E3-1230 “Sandy Bridge” – 3.2GHz, 4 Cores, 8 Threads, 8MB (Newegg)
Motherboard: Supermicro X9SCM-F – Intel C204, Dual GigE, IPMI w/Virtual Media, 2x SATA-3, 4x SATA-2 (Newegg)
RAM: 2 x Kingston 8GB (2 x 4GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 (PC3 10600) Server Memory Model (Newegg)

The whole list can be found here -> Building ESXi 5 Whitebox Home Lab Servers « Wahl Network

What do you think? Does this sound like a good box? Do you have other recommendations?

Regards,
Lordy

shnelson

How many VMs do you plan on running? Config will work great, but you'll burn the memory up in a hurry and likely wont touch the processing power.

16gb of ram seems to be a happy medium, I'm running it with a 2.4ghz xeon and a dozen vm's (windows/linux mix) with a little bit of breathing room yet.

lordy

I forgot, I would probably up the memory to 16 GB, as it is not that expensive (any more).

I may be running somewhere around 4 to 6 VMs but none of them running under high load constantly. What do you think?

Essendon

Go with the 16GB RAM, I have 8GB on my system and sometimes it can be limiting. You can probably get by too, but 16GB would be a nice-to-have. If you can spare the cash, grab an SSD for your virtual machines. You'll see a noticeable difference, like the VM's booting up in under 30 seconds (even less maybe).

MentholMoose

The specs look good but seem overkill for 4-6 lab VMs. Do you plan on eventually using more VMs, or high-performance VMs, and need room for growth? Also, you care about server-class features such as built-in remote KVM/management and ECC RAM?

lordy

@Essendon:
I thought about adding SATA disks and one SSD and use some Tier-Storage approach. Most VMs would run on the SATA disks and get a slice of the SSD if necessary, e.g. for the Postgres-DB running on one of the VMs.

@MentholMoose:
Since my current lab has remote console, power control and stuff I am kind of used to the luxury And I have made the experience, that once you have a new resource at hand you use it more and more so I want to avoid building a new box next year. Plus, the price is not that expensive. CPU, RAM and Motherboard would cost me roughly 500 EUR (~ 630 USD). I unused SATA disks (2x 1TB) and a rack-mountable chassis lying around.

Any more input / opinions ?

onesaint

Those specs will make for a good system. If you intend to raid the disks for speed, make sure your controller is supported by ESXi.

I put together a shuttle PC with almost the same specs (i7 2600, 16gb ram, etc.) with a few SSDs and found that my raid controller wasn't supported. I've run 6 VMs with ESXi and 6 with KVM/Openstack with no issue.

MentholMoose

lordy wrote: »

Since my current lab has remote console, power control and stuff I am kind of used to the luxury

Understood. I definitely prefer server hardware myself (even for labbing), particularly for the remote access, just keep in mind you are paying a premium over desktop-grade stuff. For your current requirements of 4-6 VMs, you wouldn't need to spend much at all. I recently built a low-power system with an Asus C60M1-I motherboard/CPU combo ($80) and 8 GB RAM ($40) which I intend to use for a NAS, but for fun I installed ESXi 5.0 u1 on it. I've found that it works quite nicely, and I'm thinking about using it for labbing for the time being since I need to study/lab for a few Citrix certs and could use a super-low power lab machine (my main lab has a few servers with quad core Xeon CPUs that are a few generations old, so they double as space heaters).

So far I have six Server 2003 VMs on it (a DC, CA, IAS, MS SQL, vCenter, and WSUS) and a couple Server 2008 VMs (for XenApp) and it's still going strong (storage is a 256 GB Crucial M4 SSD - $200). Obviously the dual-core 1 GHz CPU is a bottleneck at times, but it's not bad considering it's so cheap (about $350 total including the SSD and a cheap case and PSU), uses very little power, and is silent (only has a PSU fan and a low RPM case fan). I wouldn't recommend this setup since it sounds like you want room to expand, but just wanted to give an example of what you can do on a very low budget.

lordy wrote: »

Any more input / opinions ?

The other question would be are you planning to run it 24/7, and do you care about your electricity bill? Having the latest Xeon is great if you need the best performance, and they are as efficient as ever, but there are cheaper options if you don't need the speed, and power utilization won't be insignificant if you run it 24/7.

My latest server-grade build was an AMD based system that I built with an eye toward low power utilization, low cost, and reasonably high VM density. Specs were Supermicro H8SCM-F ($210 - built-in IPMI / remote KVM), AMD Opteron 4162 EE ($120 - hex-core 1.7 GHz and extremely lower power utilization), and 32 GB RAM ($240 - 4x 8 GB Kingston registered ECC). So far it's at 15 VMs with no hint of slowing down (4x 120 GB SSDs in RAID 10 kind of helps ). I'm expecting it to last several years so I hope to double that eventually, if necessary (might need more storage, though).

lordy

I don't plan on running it 24/7, no. I have put some efforts into reducing my power bill and I intend to keep it down (As a sidenote, the price for 1 kWh in .de is roughly 0.27 $)

So, Supermicro boards seem to be a good starting point. RAM pricing will only vary with size and storage is flexible anyway. That leaves the CPU as one of the deciding factors for power consumption and sizing. I have seen that there are some low-power versions (45, compared to 70 W) of the XEON E3s availbale. The Core i7 doesn't seem to offer much better efficiency. Unfortunately, the Opteron you mentioned seems to be outdated so I can't really compare between them.

When I looked at the VMware website the list of supported hardware seemed quite difficult to work with. Is there a better overview or a comprehensive list of supported chipsets and CPU out there?

jibbajabba

Try this one :

ESX / ESXi 4.0 Whitebox HCL

paul78

@lordy - sounds like you have a preference for server hardware. I don't run ESX but XEN instead. I had looked at the Supermicro boards as well. But I ended up going with desktop-class motherboards and I opted for AMD processors. But with your kWh expense (wow that's twice my average) - you are better off sticking with Intel which I recall being more power efficient.

MentholMoose

lordy wrote: »

I don't plan on running it 24/7, no. I have put some efforts into reducing my power bill and I intend to keep it down

In that case, power efficiency probably isn't as big of a factor, so I wouldn't worry about it. For my non-24/7 machines I tend to go for more performance myself.

lordy wrote: »

So, Supermicro boards seem to be a good starting point.

Yes, definitely. I've had great experience with Supermicro over the years. My first one was actually a Superserver - dual socket 604 that I populated with low power Xeons, which ran for years with no problem until I replaced it with the previously-mentioned Opteron build.

lordy wrote: »

That leaves the CPU as one of the deciding factors for power consumption and sizing. I have seen that there are some low-power versions (45, compared to 70 W) of the XEON E3s availbale..

The thing about the low power Xeons is that Intel charges a premium for them, which I don't like. I want low power and low cost.

lordy wrote: »

Unfortunately, the Opteron you mentioned seems to be outdated so I can't really compare between them.

The 4162 EE has a very low TDP per core (35w for six cores) ratings of any server CPU. That, plus the core count (hex-core is nice for my needs) and the extremely low price were the factors that sold me on it. For raw performance I would not expect it to beat a Xeon.

lordy wrote: »

When I looked at the VMware website the list of supported hardware seemed quite difficult to work with. Is there a better overview or a comprehensive list of supported chipsets and CPU out there?

For my DIY builds I pretty much just assume Supermicro server motherboards will work. That strategy hasn't failed me yet, though I wouldn't expect all Supermicro boards to work. You can try searching Google or the various white box sites but you may not be able to find anyone reporting their results with ESXi on a particular board you are interested in.

jcarnat

MentholMoose wrote: »

My latest server-grade build was an AMD based system that I built with an eye toward low power utilization, low cost, and reasonably high VM density. Specs were Supermicro H8SCM-F ($210 - built-in IPMI / remote KVM), AMD Opteron 4162 EE ($120 - hex-core 1.7 GHz and extremely lower power utilization), and 32 GB RAM ($240 - 4x 8 GB Kingston registered ECC). So far it's at 15 VMs with no hint of slowing down (4x 120 GB SSDs in RAID 10 kind of helps ). I'm expecting it to last several years so I hope to double that eventually, if necessary (might need more storage, though).

Can you tell me which case you use with your Supermicro H8SCM-F ?

I've just discovered that MB and thought about the 4162 EE too to build an ESXi.
I just couldn't find any fan-less case.

TIA.

NotHackingYou

I think I'll throw in the outlier and say you need at least 16GB of RAM. My MacBook Pro has a 256GB Crucial M4, 2.2ghz quad core i7 and 16GB of ram - it's not enough and I only run 2VMs. Sometimes I run out of RAM with only 1 VM running.


We have several servers that are acting as VM hosts and the extra RAM makes a huge difference. 16GB will get you by but less would be frustrating, IMO.

Edited to Add: I run Lion as a base OS and a Windows 7 (64 bit) and Windows Server 2008 (64 bit) VMs.

jibbajabba

WTH ? You run out of RAM with a single VM and 16GB of RAM? My HP MicroServer has 8GB and runs 3 VMs - Server 2008R2 / AD / Exchange 2010, W7 with Office 2010 joined to the domain and CentOS LAMP and if anything then I am running out of I/O.

QHalo

Only two VMs with that configuration? What are those VMs doing because that doesn't seem right.

onesaint

Nah, it's VMware Fusion. It's not very effective at running multiple VMs. If it were ESXi he would be able to get much more out of that configuration.

SephStorm

i was going to say, im running a laptop with to server k8 dcs and 3 w7 clients in workstation that is on 8gb ram. (esx was giving me crap)

networkjutsu

What do you guys think about Mac Mini Server as ESXi host? It seems like everything works with a customized ESXi image. I know that Mac Mini Server is around $1K but man the form factor is pretty sweet.

MentholMoose

jcarnat wrote: »

Can you tell me which case you use with your Supermicro H8SCM-F ?

I've just discovered that MB and thought about the 4162 EE too to build an ESXi.
I just couldn't find any fan-less case.

I've got it installed in a standard desktop case. It is micro ATX so it should fit in any case that can take micro ATX or larger motherboard. I don't need it to be silent so I have case fans, though my case takes 120mm fans so I can use large, slow fans that are relatively quiet.

NotHackingYou

jibbajabba wrote: »

WTH ? You run out of RAM with a single VM and 16GB of RAM? My HP MicroServer has 8GB and runs 3 VMs - Server 2008R2 / AD / Exchange 2010, W7 with Office 2010 joined to the domain and CentOS LAMP and if anything then I am running out of I/O.

Windows 7 with 4GB (Depending on the day) and Windows Server 2008 64 bit with 2GB of RAM. I'm using Parallels, which is not as good (nor is it the same class of product) as ESXi. Lion is pretty memory hungry IMO - the kernal usually takes 2GB of RAM by itself. Then you figure flash takes 500MB, Chrome render takes 400MB, Chrome takes 400MB, WindowServer is 400MB.

On top of that I keep probably 40 tabs open in Chrome and a few in safari, textedit, adobe reader, preview, google drive, antivirus, and all the rest of the little stuff. It adds up fast.

Notice I have 1.27GB of free RAM. This is with only the Win7 VM running at 4GB of RAM

MiikeB

Ive got ESXI running on my Lenovo W520 laptop and I LOVE it. I took the CDRom out for another HDD so I have two 750GB HDDs, i7 processor and 16GB of Ram.

ESXi boots off the SD card then I have plenty of storage to work with. I also have a 64GB MSata drive with a Windows 7 install on it in case I ever need to use the laptop as a regular laptop.

Its great because I can take it and another laptop anywhere and have a full lab, its quiet and eats up almost no space. Cost me ~1400 total.

tonningp

I ordered , almost verbatim, the system listed on Chris' site

Building ESXi 5 Whitebox Home Lab Servers « Wahl Network


I initially ordered the wrong RAM as I needed unbuffered RAM. I have the following MBD and CPU:

SUPERMICRO MBD-X9SCM-F-O LGA 1155 Intel C204 Micro ATX Intel Xeon E3 Server Motherboard

Intel Xeon E3-1230 Sandy Bridge 3.2GHz 4 x 256KB L2 Cache 8MB L3 Cache LGA 1155 80W Quad-Core Server Processor BX80623E31230

Currently I am waiting for my RAM, but I would think that this system would give me the error code of 5 beeps and 1 long beep if there is no memory installed. It doesn't. Am I right in assuming this, or should I just sit tight and wait for the RAM to come and see if this will fix the problem of no beep, no POST, and no Video? I also noticed that the MBD on the wahl site is a X9SCM-F. The only thing that newegg had was the X9SCM-F-O and they are claiming that this is compatible with the CPU that I purchased. Do any of you have advice?

Thanks,
Paul

disco277

MiikeB wrote: »

Ive got ESXI running on my Lenovo W520 laptop and I LOVE it. I took the CDRom out for another HDD so I have two 750GB HDDs, i7 processor and 16GB of Ram.

ESXi boots off the SD card then I have plenty of storage to work with. I also have a 64GB MSata drive with a Windows 7 install on it in case I ever need to use the laptop as a regular laptop.

Its great because I can take it and another laptop anywhere and have a full lab, its quiet and eats up almost no space. Cost me ~1400 total.

Hi MiikeB,

Just curious if you ran into issues installing ESXi on your W520. I'm trying to install ESXi 5 and it hangs on "initializing acpi". I've disabled most of the on board stuff that isn't need and fiddled with drives settings.

Any lessons learned to get esxi installed?

Thanks!!!

NotHackingYou

Looking over the baby dragon and thinking about building one for myself. Looks like you could buy 2 of these kits and wind up with 32GB RAM in this thing. (32 GB is the stated max at newegg: Newegg.com - SUPERMICRO MBD-X9SCM-F-O LGA 1155 Intel C204 Micro ATX Intel Xeon E3 Server Motherboard)

CT2KIT102472BD1339 - 16GB kit (8GBx2), 240-pin DIMM , DDR3 PC3-10600 from Crucial.com

robert.berger

Hi Mike,

MiikeB wrote: »

Ive got ESXI running on my Lenovo W520 laptop and I LOVE it. I took the CDRom out for another HDD so I have two 750GB HDDs, i7 processor and 16GB of Ram.

ESXi boots off the SD card then I have plenty of storage to work with. I also have a 64GB MSata drive with a Windows 7 install on it in case I ever need to use the laptop as a regular laptop.

Its great because I can take it and another laptop anywhere and have a full lab, its quiet and eats up almost no space. Cost me ~1400 total.

I would like to get a Lenovo W520 laptop and ESXI on it to be able to have a "mobile" lab. What version of ESXi did you use? Are there any special tweaks needed?

Thanks for your reply!

Regards,

Robert

robert.berger

Hi Mike,

I am especially curious if and how you got the network up and running.

Regards,

Robert

neilperry

Seeing a lot of people go with the Supermicro board (9XSCM-F-O) and a Xeon. I did myself. Just wanted to share what I did for RAM as it seems like this may be the cheapest at $54 a stick for 8GB. https://www.superbiiz.com/detail.php?name=W1333EB8GM

I was running 2 of these sticks in my host up since July until yesterday when the two more sticks I ordered came. Have had ZERO problems with it so far. The setup is a beast.

QHalo

robert.berger wrote: »

Hi Mike,



I would like to get a Lenovo W520 laptop and ESXI on it to be able to have a "mobile" lab. What version of ESXi did you use? Are there any special tweaks needed?

Thanks for your reply!

Regards,

Robert

You could run all that in VMware workstation and have just as good of performance and not sacrifice using the machine for other things. I have a W530 with 32GB of RAM and 1 128GB SSD and 1 512GB SSD. I can run the following in VMware Workstation:

2x nested ESXi hosts
vCenter
2x Vyatta Routers - scenario testing of a WAN
2x NetApp ONTAP Simulators
3x Nested VMs within the nested hosts

I could run more because this thing doesn't even sweat when running this. Other than the heft of it, it's a beast and a great machine overall. Mine of course cost a bit more than ~1400 though. Can't recommend an SSD enough though.