Building out a lab, need some info and hardware advice

reppgoa

Hey all,

I am in the process of building out a lab that will basically be my hacklab, networking lab, and anything else I want lab. One of the key things I need to figure out is, can I virtualize Cisco routers and switches? What I mean is I want to emulate IOS. I would be doing this in an effort to not have to buy actual physical routers and switches. Is the possible?

Next, as far as hardware, I want to keep it at or around 2000.00. I am currently looking at getting a dual socket mobo, 2 of the E5 series Xeons (the $600 Xeons, not the $1000 version), 32gb of ram, and possibly some smaller SSD's. Is this the best route for a PC that will be dedicated to virtualizing everything from vulnerable hosts to full on networks?

I am starting to get into the point where I need to advance my knowledge, and I want something that is going to be able to do anything I want it to. Thanks for the input as always, and I look forward to your responses!

odysseyelite

There are two roads you can follow: buy or build. There are plenty of treads on building home ESX labs, including one I started. Other options are a nested enviroment or having two physical boxes with NAS. It gets down to cost and how fast you want your lab to be.

As for virtual cisco routers you can use GNS3. It can do some switching but not everything. You would need real switches since there isn't a end all simulator yet.

Before you buy equipment, I would read through the forum. You don't want to buy something that is not going to work with the virtualization hypervisor you choose.

reppgoa

Can you link your post? I went through you recent posts and couldn't find it. Thanks!

I am still unsure the direction I want to go. If I still have to buy routers and switches, then it is going to change things. I want to have a decent number of hosts of my machine, and yes I will be using nested VM's. I will be using the following article to build my vsphere lab:

Building the Ultimate vSphere Lab – Part 1: The Story « Boerlowie's Blog

After I get that all setup, I want to make hosts that are attackable for my hack lab as I am primarily learning security. I am thinking 10+hosts, mixed environment's. A few win7 hosts, some linux hosts, winXP, server2003 and 2008. It may end up being more than 10. What kind of horsepower do you think I will need for something like this? They wont be doing much computing at all, just sitting there to be attacked so I am thinking they will have a decently small footprint.

After doing some research, I am leaning towards getting Newegg.com - Intel Core i7-3960X Extreme Edition Sandy Bridge-E 3.3GHz (3.9GHz Turbo) LGA 2011 130W Six-Core Desktop Processor BX80619i73960X rather than the Xeons, but I am still not sure. I would get that processor with 32gb ram, and 2 120gb SSD's to through in raid for my VM's.

If you guys have opinions, please, dont hesitate to drop them in, I want to most information possible before I make this buy. Thanks!

odysseyelite

http://www.techexams.net/forums/virtualization/71793-so-you-want-build-vsphere-5-lab.html#post580020

Ram and hard drives are your largest bottlenecks. I did Intel because it was confirmed at the time to work. If I had more time I would have researched more and gone with AMD. Why AMD? because you can get 6 cores cheaper then Intel Quad core.

In the end, I haven't peaked my CPU yet, only ram. 32 GB of ram is my next move.

reppgoa

how many VM's are you running currently?

odysseyelite

reppgoa wrote: »

how many VM's are you running currently?

I have a nested enviroment. So I have 9 Infranstuture VMs.
4 ESXi 5 Hosts
freeNas
Windows 2008 Domain Controller
Vcenter
VCSA
Vyatta for routing

Then running on my virtual ESX hosts I've had various amounts of vm's but 8 at this time.

reppgoa

and with your i7, you arent capping the CPU? What is your usage under full load while labbing?

MentholMoose

Since you mentioned you want to lab Cisco gear, you can look into GNS3, which lets you virtualize such devices (GNS3 is a front-end for dynamips and dynagen which handle the actual virtualization).
Graphical Network Simulator - GNS3

reppgoa wrote: »

and with your i7, you arent capping the CPU? What is your usage under full load while labbing?

IMO, the CPU speed/performance is usually least important factor in virtualization, especially when it comes to labbing. I've been using virtualization for labbing for a while, going back to the days of single-core CPUs, and rarely have I maxed out a CPU. My experience supports odysseyelite's assertion that RAM and disk are the typical bottlenecks. The exception would be if you are running workloads that are particularly dependent on CPU performance. Video encoding and scientific computing come to mind. GNS3 might require a decent CPU if you are running a significant number of virtual routers.

For a lab machine for learning a variety of things, including virtualization (e.g. vSphere), systems adminstration (Linux/Microsoft), and networking (Cisco), I'd recommend building a fast desktop PC rather than a server. For running variety of workloads, buying or building a server with an expensive server-grade CPU is not going to help much, if at all.

odysseyelite

MentholMoose wrote: »

Since you mentioned you want to lab Cisco gear, you can look into GNS3, which lets you virtualize such devices (GNS3 is a front-end for dynamips and dynagen which handle the actual virtualization).
Graphical Network Simulator - GNS3


IMO, the CPU speed/performance is usually least important factor in virtualization, especially when it comes to labbing. I've been using virtualization for labbing for a while, going back to the days of single-core CPUs, and rarely have I maxed out a CPU. My experience supports odysseyelite's assertion that RAM and disk are the typical bottlenecks. The exception would be if you are running workloads that are particularly dependent on CPU performance. Video encoding and scientific computing come to mind. GNS3 might require a decent CPU if you are running a significant number of virtual routers.

For a lab machine for learning a variety of things, including virtualization (e.g. vSphere), systems adminstration (Linux/Microsoft), and networking (Cisco), I'd recommend building a fast desktop PC rather than a server. For running variety of workloads, buying or building a server with an expensive server-grade CPU is not going to help much, if at all.

^^^^^I think I'm at 30% but I am not running SQL or heavy loads. I'm making sure I can clone and vmotion servers. My speed problem is because of using freenas as shared storage. The machine I built is just a gaming machine in teh end running server software. When I upgrade I'll make it my desktop PC.

reppgoa

so what is the ideal disk setup? SSD's or many smaller HDD's? My plan as of right now is to get 2 120gb SSD's and hardware raid them together and use linked clones to conserve space. Thoughts?

odysseyelite

What OS are you using? ESXi is picking on what hardware raid you use.

dave330i

reppgoa wrote: »

so what is the ideal disk setup? SSD's or many smaller HDD's? My plan as of right now is to get 2 120gb SSD's and hardware raid them together and use linked clones to conserve space. Thoughts?

It's a home lab. Go for the cheapest with good capacity.

ptilsen

dave330i wrote: »

It's a home lab. Go for the cheapest with good capacity.

I actually have to disagree with this. I chugged along on RAID 0 with 7,200 RPM SATA drives, and later 10K SAS drives. Trying to boot more than three VMs at once (which is pretty much a must for 70-643) got painful. It's enough that I, for one, would get discouraged and distracted, and either take hours extra to finish my lab or simply not do it. Having an SSD or two makes a huge difference. There's no sense in putting them in RAID, though. They will perform better and give better capacity standalone due to TRIM.

On the other hand, if the lab is just for GNS3, that all seems completely excessive. If OP is going to boot more than 3 or 4 VMs, SSDs are just worth it to me. Time is valuable, and those seconds and minutes add up over years of labbing dozens or hundreds of scenarios.

edit: This is also why I advocate buying the appropriate hardware used, rather than purchasing and self-building a new desktop to use with VirtualBox or similar. A G6 Proliant or similar Poweredge is normally not more expensive than building the same desktop-quality hardware, and gives you a dedicated hypervisor system.

dave330i

ptilsen wrote: »

I actually have to disagree with this. I chugged along on RAID 0 with 7,200 RPM SATA drives, and later 10K SAS drives. Trying to boot more than three VMs at once (which is pretty much a must for 70-643) got painful. It's enough that I, for one, would get discouraged and distracted, and either take hours extra to finish my lab or simply not do it. Having an SSD or two makes a huge difference. There's no sense in putting them in RAID, though. They will perform better and give better capacity standalone due to TRIM.

On the other hand, if the lab is just for GNS3, that all seems completely excessive. If OP is going to boot more than 3 or 4 VMs, SSDs are just worth it to me. Time is valuable, and those seconds and minutes add up over years of labbing dozens or hundreds of scenarios.

edit: This is also why I advocate buying the appropriate hardware used, rather than purchasing and self-building a new desktop to use with VirtualBox or similar. A G6 Proliant or similar Poweredge is normally not more expensive than building the same desktop-quality hardware, and gives you a dedicated hypervisor system.

Interesting. My experience is the opposite. Were your VMs right size for test environment (i.e. 512 MB RAM allocated/VM)?

odysseyelite

The requirements are not clear enough. If you want a end all lab it could cost $$$. Is this a lab for ESX, Hyper-V, Cisco emulation?

You are not going to get it all in one box. If money was not an issue that ideally, two hosts with lots of ram with shared storage and some physical cisco switches.

I had limited funds at the time so 600.00 bucks I made whitebox with the goal of doign nested VM's to pass the VCP and to do some test powercli scripts for work. Its not as fast as I want, but it provides me what I need to get the concepts down.

ptilsen is right as time is money. I spent alot of time waiting for progress bars last night. I plan on getting proper hardware when money becomes available.

The point I am trying to make is everyone has different needs based on 1. what they are learning 2. money 3.space 4. Noise

ptilsen

dave330i wrote: »

Interesting. My experience is the opposite. Were your VMs right size for test environment (i.e. 512 MB RAM allocated/VM)?

My VMs were and are the appropriate size for what I'm doing. Before SSD, booting all of them (we are talking 4-8 VMs here) would take me a few minutes. Now, I can boot and reboot whatever I want and they come back in seconds. Eight VMs reboot all once takes less than 15 seconds.

Keep in mind, I'm not operating a hypervisor here; I know from experience I could comfortably do what I'm doing off of a RAID 10 of SATA drives on roughly the same hardware using ESXi. Which, again, is where purchasing a used server outright is nice. A few 10K or 15K SAS drives in a used G6 Proliant will easily handle this type of lab environment.

So really, I'm giving somewhat conflicting advice. If you're going to buy a used server and use a hypervisor, the SAS drives it most likely comes with will be plenty. If you're going to go the desktop route, SSDs are preferred. In my case, purchasing 200GB worth of SSDs was no problem, and my desktop already had plenty of hardware (6-core CPU, 16GB of RAM).

reppgoa

odysseyelite wrote: »

The requirements are not clear enough. If you want a end all lab it could cost $$$. Is this a lab for ESX, Hyper-V, Cisco emulation?

You are not going to get it all in one box. If money was not an issue that ideally, two hosts with lots of ram with shared storage and some physical cisco switches.

I had limited funds at the time so 600.00 bucks I made whitebox with the goal of doign nested VM's to pass the VCP and to do some test powercli scripts for work. Its not as fast as I want, but it provides me what I need to get the concepts down.

ptilsen is right as time is money. I spent alot of time waiting for progress bars last night. I plan on getting proper hardware when money becomes available.

The point I am trying to make is everyone has different needs based on 1. what they are learning 2. money 3.space 4. Noise

What I want this lab to be is everything, or as much of it as possible. My degree is in networking and security but I dont want to be limited to that. I want to be able to use this lam to learn VMware, M$, RH, firewalls, routers, switches. I also want to be able to use it as a hack lab. I want a sophisticated hack lab, where I can simulate attacking a full blown network. It needs to be able to run many VM's in a nested environment. My plan as of now (can change, and probably will) is to run a base win7 64 bit install in which I will install workstation 8, and then from there install my nested environment consisting of 2 esxi 5 installs etc. A link for what I basically want to do is here Building the Ultimate vSphere Lab – Part 1: The Story « Boerlowie's Blog.

Now I know this may not be the ideal setup, and am open to suggestions. My main goal is to build my lab though. I am headed to microcenter tomorrow afternoon and would like to have what I want nailed down. In my mind I am looking at a mobo with 8 DIMMS and using the 2011 socket. That would mean I have a very limited choice on processors. I will probably go with Micro Center - Intel Core i7 3820 LGA 2011 Boxed Processor 221184 initially, and if I need more horsepower, I can up it to the extreme edition. I will be using 32gb ram, expandable to 64. My only real question at this point is disks. Some people say raid them together, some people say dont. I will be buying SSD's, the question is, how many and of what size. I would like to stay reasonable of course, but am willing to make the investment. I am thinking 2 or 3 120gb Agility 3's and a 2TB spinpoint that I already have should do nicely.

Thanks for all of the responses, they have pushed me to research even farther. Keep em coming!

ptilsen

If the budget's there four SSDs in RAID 10 will be great performance. Otherwise, I would take a hybrid approach. Get SSDs and don't RAID them, but back them up to a RAID 1 of larger HDDs and/or use HDDs for storage-intense systems you don't need speed on.

Core i7 with 32GB of RAM sounds like plenty. I have an X6 with 16GB of RAM and I feel like it's more than enough, most of the time. If you can, getting the 6-core i7 with HT is your best bet. Make sure it has hardware virtualization support, though; some of the i7s don't.

reppgoa

yea, what I am seeing is that very few of the newer i7's have VT-D. I am beginning to wonder how important VT-D really is. Can anyone shed some light on this for me.

EDIT* After doing some research, I dont think VT-D is that important for what I want. I wont be needing to assign any hardware devices to specific VM's. I could see it being cool if you wanted to use your main PC to lab with as well, and then having a gaming VM with passthru on the video card, but past that, I don't see myself needing it. Please correct me if I am wrong guys.

Keep in mind, you can only assign a PCI device to 1 VM.

odysseyelite

reppgoa wrote: »

yea, what I am seeing is that very few of the newer i7's have VT-D. I am beginning to wonder how important VT-D really is. Can anyone shed some light on this for me.

EDIT* After doing some research, I dont think VT-D is that important for what I want. I wont be needing to assign any hardware devices to specific VM's. I could see it being cool if you wanted to use your main PC to lab with as well, and then having a gaming VM with passthru on the video card, but past that, I don't see myself needing it. Please correct me if I am wrong guys.

Keep in mind, you can only assign a PCI device to 1 VM.

i7 2600 has it, the K series do not. ESXi complains about not having VT-D, I'm not even sure if it will install if it is not enabled. You need a processor and a motherboard that supports VT-D