CSR1000v

gorebrush

Anyone else tried this? I've just loaded it into my ESX server and I have to admit it seems pretty good. There are a number of positive posts out there about using it and some of them suggested that it is pretty much *as good* as using real equipment.

I have two very capable quad core machines with 16GB of RAM and I can easily run ESX on both!

Each instance of the CSR needs 2.5GB so with about 30GB of RAM I can run 12 of them between the two machines and I can get 6 on one. Might be enough to do most of the routing toplogies.

My concern is the switching because I've only got 2 x 3750's and a pair of 3550's but the switching really doesn't concern me to the point that I'd need to spend more money on buying more equipment. We'll see. Maybe I'll just convert one of my 3750's into a breakout and fudge the topologies a bit, and then use the 4 switches when I'm doing switch specific things.

Anyway, I've got lots of options. I think I've read INE are going to have up to 20 routers in their v5 workbook, so it will be interesting to see how people manage to get all of this running. I know it's only for the sake of learning - I guess after June we'll start hearing officially how many routers make up the topology.

EDIT: - some more thoughts...

I've been thinking more about this. What about if I overprovisioned the RAM and put the page files onto an SSD...? Would be interesting. If the routers themselves aren't doing a lot then even though they've got 2GB each, I could just get it swapped onto some fast storage. For routers I probably don't need to care.

Just like the CSR licensing itself - without a "premium" license, it's limited to 50Mbit for 60 days, then goes to 2.5Mbit. Even then that's probably fast enough, and then it would be dead easy to just redeploy the original OVA's for another 60 days worth

networkjutsu

You don't even need to activate the premium license. At least that's what I got from Brian's post. On that thread that I posted few days ago, Brian said that you can overprovision stuff as long as it's on SSD. Anything less would be slow. I have an ESXi server but I have about 9-12GB worth of "production" stuff that I use so my 32GB RAM wont cut it. I am thinking about buying used HP or Dell workstation with two quad and 64GB RAM but I am still thinking about it. I need to run few instances to see if I can swing it by just buying one quad core with HT.

gorebrush

I've got a 60Gb SSD lying around - that will do it. Only 250MB/s read/writes though but for labbing I'm sure it will be fine.

gorebrush

OK so apparently you can't use hard disks that are connected to USB as storage. Never mind.

I've started creating a boatload of CSR's on my ESX Server, despite it only having 16GB RAM. I've got 48GB of SSD set aside for RAM (so that's enough for 21 instances of CSR to swap into).

I've created 4 so far.

The key memory usage element to watch out for is "Memory Active". So far I've got 4 running and the active memory is sat at 3GB. I'm just going to keep throwing CSR's at it until we're full.

I've got 2 x 2 port HP NIC's - I can throw one in here. I think Windows 7 ought to support it, and then I should be able to run CSR on here in VMWare workstation and wire that into my lab. That'll add a few more.

gorebrush

Well I appear to be running 15 of these CSR's quite happily on my ESX server.

Using 10GB of swap. I've ordered 16GB more RAM to counter that and then I think I can get up to the 20 required for the labbing.

If I can do that successfully, I'll consider selling off some kit.

networkjutsu

I might have to scale out. I am still looking for more options. I don't really like to buy a big workstation with dual Xeons like HP XW6800 or T7500. I'd rather build another machine with the same form factor or smaller than my current ESXi box. Once you try with 32GB and running full scale labs, can you please report back? It'd be nice to know if I can swing it with 32GB RAM. Thinking of buying Xeon 1230v3 and Supermicro board. My current one is Xeon 1230v2 and Supermicro.

gorebrush

Well I've got 20 instances on there, I just need the RAM. I am not 100% that it'll run all 20 by itself, but if I can get away with it then that would be amazing.

I've nearly built my rack now, bit of a change from before! I've re-used the serial cables as point to point links, just because. I'll post up the physical topology when I'm done with it all!

networkjutsu

Yea, someone said that he was able to run 20 instances with i5 and 32gb RAM. Just a bit concern with swap though I think it'll be bearable if I am using SSD. I wonder if I can put the swap on a separate volume from the install. I am planning to use USB flash for the install and use a dedicated SSD for swap and all vmdk files in my NAS.

gorebrush

And yes of course, full updates on my experience with it as and when.

networkjutsu

With 10GB worth of swap, how's the performance with actual configs? Did you save the files locally or do you have a centralized storage? The setup that I am thinking is use my NAS for storage.

gorebrush

Well I am waiting for my extra memory to turn up (here tomorrow) and also waiting for the workbooks.

Will let you know when I'm rocking and rolling.

I have 64GB of SSD storage set aside for "host cache" which means swapping, but I think that's a bit generous. I'll probably cut that down to 48GB and then try and squeeze the actual CSR's onto the SSD as well so at least they'll boot quickly as well as run as fast as possible.

networkjutsu

I think I am going to combine both new build and my current ESXi server and just share the load so no need for SSD. 12 routers on the new build and 8 routers with my current build. I just need to pull the trigger. I have it in the cart already. I may delay it for a while though until I get rid of my lab. It'll help with expenses and make room to the new build. With 8 routers, hopefully, I can run the technology labs.

gorebrush

Yeah you won't need 20 routers for everything. I can probably run 12-14 comfortably on one box and my other i7 rig I can run a load on too will just need to get another NIC and wire it up. If I can run the full 20 and it's reliable I may sell my more valuable pieces of rack.

gorebrush

OK so the memory is in and I've got 10 CSR instances all up and running.

I have got some host cache space set aside but as it stands I am not using any. I have booted all 10 instances and of course the memory usage has shot up so I'm not sure if I'll be able to squeeze many more instances on here without resorting to swapping memory. At least I know it works and I have at least 13 routers (if you include my physical ones) that are in my rack. Any more routers that I need to supply - I will need to get another cable over from my desktop machine and a seperate NIC in there so I can **** the CSR's onto seperate networks. I could go VMWare Workstation route with those but I may end up putting a seperate partition to use and just run ESX on that as well - but we shall see.

For now I'll go with the 10 routers and start labbing things up. I kinda regret taking apart my v4 lab topology now though as I can't really practice the stuff that INE haven't updated for the v5 workbooks. However, that's not *really* a problem - I can just make my own topologies now instead and just use the workbook to follow along and create my own scenarios.

They need to hurry up and get the full practice labs created though. I'm not ready for the lab yet, but I need to start practicing larger multiprotocol things. Again, I suppose I can just mix and match and mash it all up myself.

EDIT - It started swapping memory now I have 12 instances running. So any more will need more swap space. I've set aside 24GB of SSD space for swapping, but I can always move the CSR's off the SSD and into HDD and free up space for swapping.

In fact I may well do this long term because I don't believe the CSR's will benefit from the SSD other than for booting up - so once I start seeing the need for more than 12 instances of CSR - I'll move the rest off the SSD and increase the host cache space to allow for more instances. I'd love to see all 20 running on here, but that's an experiment for another day. For now I'm just very pleased to have the 10 routers on there that I need for the INE v5 workbook. It's a start. I'm going to need to find some avenue for practicing full labs, however.

gorebrush

So I'm up to 16 instances, and it's only swapping 4GB of RAM.

I actually think I can run all 20 on here.

i5 with 32GB RAM, btw.

Well I got to 18 and wussed out, mainly because I could do without completely killing my ESX server while I'm not near it

Fun though! I have plenty of infrastructure to go on with that's for sure.

gorebrush

So right now I have 10 routers running. Don't even have a physical switch connected off the vSwitch - I can lab purely with just routers.

Right now I've just setup the topology that was in the free DMVPN series on YouTube (INE)

CPU is about 25% and memory is Active for 6GB RAM and I've got 3 other VM's (Windows) running too.

I'm very impressed. It takes an absolute age to boot up my 1840's and 2600's in comparison.

I'm just disappointed I didn't think of it before

Never mind - I'll decide what to do with my old routers later.

gorebrush

So other than bringing the ESX server down to its knees last night, I've been practicing more DMVPN this morning.

Top tips for those of you looking to use these in your lab: -

1. Don't use ESX 5.5 as your hypervisor. When you import the OVA it creates it as version 10. You can't then edit it without the web client. Setting up the web client requires vCenter Server, a Windows Domain, Two servers (because you then can't install vCenter on a Domain Controller). I found this out the *hard* way. In the end I actually installed ESX5.1 in VMWare Workstation and then imported the OVA that way. Once I had one successfully imported I just copied the files out to my desktop and then uploaded all that to my ESX 5.5

Though if someone knows how to edit the VM version afterwards then that would be interesting. I initially built my CSRs with Thick provisioned disks. I think this was a mistake because each CSR is 8GB when fully provisioned and making 20 copies of this is time consuming. When they were thin provisioned I believe they only consumed 108Mb initially so setting up new clones of this would be much quicker. I think I'll go create more in a while.

2. no ip domain-lookup <--- speeds up traceroute because these routers obviously do domain lookups now whereas plain old IOS didn't (at least not for me)

3. license boot level premium <--- activates all the features for 60 days. I would create copies of your CSRs BEFORE issuing this command. I have a CSR image set aside for 2 months time when I need to re-create it. Once you have activated this then you get all the security features etc.

More tips as I work them out.

pitviper

Good stuff man, thanks for the updates!

I have 2 Cisco C200m2 servers running ESX5.1 in the lab. I'm going to throw some CSR1000vs on there too!

gorebrush

Do it, they're awesome. They have a decent version so feature wise INE reckon it's good for the lab.

R10#sh ver
Cisco IOS XE Software, Version 03.12.00.S - Standard Support Release
Cisco IOS Software, CSR1000V Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(2)S, RELEASE SOFTWARE (fc2)

gorebrush

Moderators...

If I were to edit the top post and condense all my findings into one post, can you clean up the rest of this thread please? Or can I delete posts myself?

I think I can just delete out the useless ones myself, actually...

EDIT: Ah-ha I can delete my own posts. I will try and condense my notes.

networkjutsu

gorebrush wrote: »

1. Don't use ESX 5.5 as your hypervisor. When you import the OVA it creates it as version 10. You can't then edit it without the web client. Setting up the web client requires vCenter Server, a Windows Domain, Two servers (because you then can't install vCenter on a Domain Controller). I found this out the *hard* way. In the end I actually installed ESX5.1 in VMWare Workstation and then imported the OVA that way. Once I had one successfully imported I just copied the files out to my desktop and then uploaded all that to my ESX 5.5

Yea, the ESXi 5.5 screwed me up as well. You can install vCenter appliance to do the web client. You don't have to use the full version of vCenter. Once installed, you can edit all of them and turn it off. No wasted resources. Another option is to download the vmx file and edit it using Wordpad and add the serial settings. Another option is to ssh to the ESXi box and use VI to edit the vmx file. Add the following:

serial0.fileType = "network"
serial0.fileName = "telnet://192.168.1.151:2000"
serial0.yieldOnMsrRead = "TRUE"
serial0.present = "TRUE"

I consolidated my VMs to reduce my RAM usage and I was able to squeeze 10 CSR1000v and still have 2-3GB of RAM left. If I can reliably run CSR1000v with my current setup then I don't need to buy a new ESXi server yet. Still trying to get rid of my routers. No takers yet. May have to sell it for cheap to just get rid of it.

gorebrush

Yes I have since got the appliance - I'll have to get that running. As I've got so much memory I can run 10 as they are. I'm really very happy with it.

I've got about £450 worth of routers to shift now. Too tempting. I might build another ESX server instead with that money.

gorebrush

Heh so 9 physical routers can buy me another racked i5 with 32Gb = 16 CSR's. But I don't need 32

Probably don't need 1841's either though.

networkjutsu

gorebrush wrote: »

3. license boot level premium <--- activates all the features for 60 days. I would create copies of your CSRs BEFORE issuing this command. I have a CSR image set aside for 2 months time when I need to re-create it. Once you have activated this then you get all the security features etc.

I was under the impression that you don't need to activate the license with INE's workbook? If you do activate it, I was under the impression that you still get all the features but at 2.5 Mbps rate.

gorebrush wrote: »

Heh so 9 physical routers can buy me another racked i5 with 32Gb = 16 CSR's. But I don't need 32

Probably don't need 1841's either though.

16 CSR with swap? Or did you lower your RAM to 2GB?

tldees

I can confirm that you don't have all the features (e.g. DMVPN) unless you run license boot level premium. I'm not sure what happens after the 8 weeks though, I read that just the speed downgrades to 2.5 Mbps as you stated.

I recommend creating a snapshot prior to running the above command, and if the features stop working in 8 weeks then you can revert back to the snapshot and run the command again.

networkjutsu

tldees wrote: »

I can confirm that you don't have all the features (e.g. DMVPN) unless you run license boot level premium. I'm not sure what happens after the 8 weeks though, I read that just the speed downgrades to 2.5 Mbps as you stated.

Thanks!

I recommend creating a snapshot prior to running the above command, and if the features stop working in 8 weeks then you can revert back to the snapshot and run the command again.

Yea, it was suggested in the other thread too. Makes sense. Though, you don't really want to leave snapshot for a long time though. If I am not mistaken, it takes up a lot of storage when you leave it for a long time. Probably revert back after each sessions?

gorebrush

I just have a master copy that has the serial port setup and has been set to run the virtual console. I just copied that a bunch of times and modified serial port address to suit.

keenon

you can run csr with 2gb ram, this has been verified anything less would be uncivilized, I was able to boot 7 devices with 2 each I was working my way up to 10. I only have 12gb ram on pc these are running on

networkjutsu

gorebrush wrote: »

I just have a master copy that has the serial port setup and has been set to run the virtual console. I just copied that a bunch of times and modified serial port address to suit.

That's what I did too. I usually have a base image that I can redeploy over and over anyways. So creating a base image of the CSR1000v was out of the habit. Though, it kinda saves time to just use the snapshot manager.

keenon wrote: »

you can run csr with 2gb ram, this has been verified anything less would be uncivilized, I was able to boot 7 devices with 2 each I was working my way up to 10. I only have 12gb ram on pc these are running on

Yea, I did see that on the thread that you can run 2GB RAM. Just wasn't sure if it'll do the job for INE's workbook.

gorebrush

Ah of course! I hadn't thought of snapshotting them. Bit too late now for my current instances but I'll take that into account the next time.

Jackace

So if I'm reading this all correctly using the CSR1000v virtual machine doesn't have the bugs and quirks of IOU?