Ultimate Hybrid Dynamips CCIE Lab

yuriz43

Hello my fellow networking peoples. I thought it would be fun to share a project that I've been working on. After passing the CCIE written, I decided it was time to get serious and upgrade my lab. For the last 2 weeks I've been working like a mad scientist buying and waiting for various components to arrive, and today I received the last piece!

I plan on writing a very detailed guide on how to set all of this up. Everything from hardware specifications to installing Linux and configuring each application listed below. If you have any specific questions let me know.



The Lab:

Pictures:
Flickr: yuribank's Photostream

1x 16U Skeletek Rack
2x Cisco 3550
2x Cisco 3560
1x 3U D-Storm Chassis ( I got this idea from a blog I read - Very very cool!!)
- Quad-Core Phenom II
- 8GB DDR3 Memory
- 3x Quad PCI Network cards
- 4x USB Serial Adapters
- 2x 250GB Drives ( In Linux-MD Raid1 )


OS:
Debian 6. Squeeze

Applications:
- Dynamips
- Dynagen / GNS3
- qemu-kvm ( for virtualization - good for JunOS emulation and host emulation )
- Wireshark / tcpdump
- tftpd-hpa
- Apache
- Tacacs+
- FreeRadius
- dhcp3-server

Remote Management & Access:
- NX-Protocol ( neatx - see google project. This is the best!)
- OpenVPN ( tunnel between my home & work )




Stay tuned for more details!

ehnde

Yes, I have a few questions for you:

1. Are you using a breakout switch, or just the quad NICs inside your dynamips box for connectivity to your switches, or both? Nevermind this question, I checked out the pics of your lab.

2. Why usb to serial? It sounds messy. I haven't been able to get my $1 chinese no name serial cards to work in Ubuntu, so maybe I'll try the usb-to-serial method

3. I'm assuming you're using the INE topology. How much cpu do you wind up using when your topology is fully loaded? And how much ram? I'm having trouble tweaking the idle PC value with 14 routers (can't get it lower than 58% cpu)

4. How the heck do you get Tacacs+? I want to practice with it for CCNA:Security, but it's not downloadable.

Note: I'm using vnc over an ssh tunnel on a non-standard port complete with dynamic dns to manage my lab box.

I'll be subscribing to this thread. Any info you can provide on your progress would be extremely useful.

Oh...one more thing...could you give an example of the configuration you're using to connect your switches (the .net file) and your /etc/network/interfaces ?

I'm sorry, just very, very curious about the whole process!! Your lab is awesome.

instant000

yuriz43 wrote: »

If you have any specific questions let me know.

How much did that setup cost, is the main question that comes to mind right now.

Were the 3560 the most expensive piece?

yuriz43

ehnde wrote: »

Yes, I have a few questions for you:
2. Why usb to serial? It sounds messy. I haven't been able to get my $1 chinese no name serial cards to work in Ubuntu, so maybe I'll try the usb-to-serial method

I like the idea of having my Linux box also serving as the console server. It also means one less device on my rack! The PL2302 chipset works fine in Linux.

ehnde wrote: »

3. I'm assuming you're using the INE topology. How much cpu do you wind up using when your topology is fully loaded? And how much ram? I'm having trouble tweaking the idle PC value with 14 routers (can't get it lower than 58% cpu)

Full INE Toplogy uses around 10% CPU of my core, and around 1.2GB of memory ( This is after tweaking the idlepc & idlemax values).

ehnde wrote: »

4. How the heck do you get Tacacs+? I want to practice with it for CCNA:Security, but it's not downloadable.

There has been a Linux implementation of tacacs for a long time. You can download the source here: ftp://ftp.shrubbery.net/pub/tac_plus/
Most Linux distributions include a pre-compiled binary in their package management repositories. Debian or Ubuntu: apt-get install tacacs+

ehnde wrote: »

Note: I'm using vnc over an ssh tunnel on a non-standard port complete with dynamic dns to manage my lab box.

To be blunt, VNC sucks compared to NX. Neatx is an optimized version of NX, which is X11 over SSH with added compression and caching. Try it out, you will never go back.

ehnde wrote: »

Oh...one more thing...could you give an example of the configuration you're using to connect your switches (the .net file) and your /etc/network/interfaces ?

.NET file with NIO interfaces configured - very basic ( root priv required ).
http://yuri.easytospell.net/labs/test.net.txt

/etc/network/interfaces file
http://yuri.easytospell.net/labs/interfaces.txt



cheers

yuriz43

instant000 wrote: »

How much did that setup cost, is the main question that comes to mind right now.

Were the 3560 the most expensive piece?

Yes. The 3560s were the most expensive part. $600 each. I probably could have found a better deal, but I didn't feel like spending so much time on Ebay.

3550s - $200 (each)

Linux/Dynabox - $550

Total Price: $2150 +

(I also bought some cat5, mounting brackets, and other odds -N- ends that are not included)

gorebrush

Wow 3560's are cheap.

I need to get my Skeletek out of my shed.

The legs for mine are in the attic. I hope my nuts/bolts are up there too.

Awesome job on your lab. You've got me drooling about setting up my own again..

ehnde

I had no idea there was an open implementation of tacacs. I'll get right on using that!

I'll try out neatx, but I'm not able to install software at work, so I wind up using a portable vnc client.

And now you have me going back and tweaking my topology because I know it can run better Thanks for all of the tips, and I hope you enjoy your labbing!

instant000

yuriz43 wrote: »

Full INE Toplogy uses around 10% CPU of my core, and around 1.2GB of memory ( This is after tweaking the idlepc & idlemax values).

Yeah, I saw the idlemax values on the gns3.net forums one time. combined with the right timeout values, they work quite well.

This is a very strong tip, as idlepc is mentioned in the gns3 setup documentation, but not idlemax (that I can remember)

It's not a new feature as far as I can tell, I can find references to idlemax all the way back to 2006.

Here is an example of the effect that toggling this one setting has on your CPU:
Drastically decreasing CPU load in Dynamips Daniels quest for CCIE

For a lot of people, idlepc is enough.

Hope this helps!

instant000

yuriz43 wrote: »

Yes. The 3560s were the most expensive part. $600 each. I probably could have found a better deal, but I didn't feel like spending so much time on Ebay.

3550s - $200 (each)

Linux/Dynabox - $550

Total Price: $2150 +

(I also bought some cat5, mounting brackets, and other odds -N- ends that are not included)

I know it's a bit of a chunk of change, but the 3560 have really come down a good bit. I remember a few years ago, there was no way I'd be looking at buying a 3550 switch, and they're on ebay at "buy it now" for less than $150.

I even bid on a couple just now, but I hope that doesn't inspire someone on here to outbid me on one I just bid on, LOL. Would break my heart, but I am trying to get one for under $100, LOL.

If I come home this evening and that one fails on me (can't do ebay from work, and don't want to get into those auto-programs, seems a bit shady to me, to be honest ... I feel the person who sits there and watches it needs it a lot more than I do, LOL.)

EDIT: Realized I didn't finish the sentence.

If that bid fails on me, I'll just do a "buy it now" not worth the hassle to try to save $20 or $30 bucks to me. If I could shop from work, it wouldn't be an issue, but since I can't, it's easier for me to do a "buy it now". (Also, in defense of my employer, I really shouldn't be shopping while at work, LOL!)

I would understand during lunch break, but that's about it.)

ipSpace

What 3x Quad PCI Network cards did you buy ?

alan2308

yuriz43 wrote: »

There has been a Linux implementation of tacacs for a long time. You can download the source here: ftp://ftp.shrubbery.net/pub/tac_plus/
Most Linux distributions include a pre-compiled binary in their package management repositories. Debian or Ubuntu: apt-get install tacacs+

There's also a build for Windows for those who are allergic to Linux

Centralizing Logins with TACACS+

QHalo

inSecure.ro wrote: »

What 3x Quad PCI Network cards did you buy ?

I bought three of these from Ebay and Linux has native drivers.

HP A5506-60102
http://forum.internetworkexpert.com/forums/p/9742/106215.aspx

These are also other alternatives
http://ciscoschool.net/2010/05/13/quad-nics-pci-cards-for-dynamips/

yuriz43

I've gone through a lot of different Quad Network Cards.

I'm using:

2x D-Link 570tx Quad - PCI
1x Intel 82571EB Quad - PCI-E


Originally I had 3x SUN 501-4366 Quad FastEthernet cards. These were from my old box, and worked fine. However they didn't come without their problems. First of all, they are PCI-X, and although they work in standard PCI slots, they take up a LOT of space. Secondly, udev has issues configuring the card. I had to manually specify the PCI busID for each port on the card in my udev rules.
Ultimately, the massive size was the deal breaker for my new build because a heat sink sitting behind the PCI slots obstructs the extra length of the PCI-X interface.

CCIEWANNABE

nice man! good to see more people going the "hybrid" route. In addition to a kick butt lab, you are helping save the environment by not providing power to 10 + routers

I think that qualifies for a gov't tax rebate