Lesson learned: Connected beats Static every day of the week
I thought I should share this little piece of experience that I gained today with you.
The idea was pretty simple: Break up a /24 into two /25, route one /25 to the new datacenter and build out new hosts there. So we set up everything, a multilayer switch, a ha-firewall and two test hosts. We also set up a private transfer network between the current router, the new multilayer switch and the firewalls.
It looked quite simple:
10.x.x.1 -> first new multilayer switch
10.x.x.2 -> second new multilayer switch (not yet there)
10.x.x.3 -> firewall #1
10.x.x.4 -> firewall #2
10.x.x.5 -> HSRP address of the switches
10.x.x.6 -> CARP address of the firewalls
10.x.x.7 -> old router to be decommissioned
I hopped on the router and put in:
ip address 10.x.x.7 255.255.255.0 secondary
and all was well. Router, Switches and Firewall could ping each other. Next I put in the route:
ip route 195.x.x.128 255.255.255.128 10.x.x.6
I verified with the "show ip route 195.x.x.198" (test hosts) that it was right in there. But it just wouldn't work. I could not reach the hosts behind the new firewalls. But I knew they were fine. They could ping the hosts. The hosts were also set up correctly. I was stunned. After drinking a big can of Red Bull I took another look at the routers config and it hit me like a hammer. There it was:
interface GigabitEthernet0/0
ip address 195.x.x.1 255.255.255.0 secondary
I was jumping off the couch and giving a speech to my buddy why this was the problem and how this prevented things from working. There obviously was a directly connected route that beat up my pretty static route. So I updated the config:
ip address 195.x.x.1 255.255.255.128 secondary
Then I ping the hosts, and ta-daa: NOTHING. What?! This should work now! I was, again, stunned. It took a few minutes (and probably more sugar) before it came to me: ARP cache, you SOB. When there was still a directly connected route the router had tried to ARP for the hosts which now resided on a different ethernet segment. But it had cached that information. So here we go:
clear arp-cache interface GigabitEthernet 0/0
and voila, it pings. Everything works. Lesson learned: Check static routes, check dynamic routes but ALSO check connected routes!
The idea was pretty simple: Break up a /24 into two /25, route one /25 to the new datacenter and build out new hosts there. So we set up everything, a multilayer switch, a ha-firewall and two test hosts. We also set up a private transfer network between the current router, the new multilayer switch and the firewalls.
It looked quite simple:
10.x.x.1 -> first new multilayer switch
10.x.x.2 -> second new multilayer switch (not yet there)
10.x.x.3 -> firewall #1
10.x.x.4 -> firewall #2
10.x.x.5 -> HSRP address of the switches
10.x.x.6 -> CARP address of the firewalls
10.x.x.7 -> old router to be decommissioned
I hopped on the router and put in:
ip address 10.x.x.7 255.255.255.0 secondary
and all was well. Router, Switches and Firewall could ping each other. Next I put in the route:
ip route 195.x.x.128 255.255.255.128 10.x.x.6
I verified with the "show ip route 195.x.x.198" (test hosts) that it was right in there. But it just wouldn't work. I could not reach the hosts behind the new firewalls. But I knew they were fine. They could ping the hosts. The hosts were also set up correctly. I was stunned. After drinking a big can of Red Bull I took another look at the routers config and it hit me like a hammer. There it was:
interface GigabitEthernet0/0
ip address 195.x.x.1 255.255.255.0 secondary
I was jumping off the couch and giving a speech to my buddy why this was the problem and how this prevented things from working. There obviously was a directly connected route that beat up my pretty static route. So I updated the config:
ip address 195.x.x.1 255.255.255.128 secondary
Then I ping the hosts, and ta-daa: NOTHING. What?! This should work now! I was, again, stunned. It took a few minutes (and probably more sugar) before it came to me: ARP cache, you SOB. When there was still a directly connected route the router had tried to ARP for the hosts which now resided on a different ethernet segment. But it had cached that information. So here we go:
clear arp-cache interface GigabitEthernet 0/0
and voila, it pings. Everything works. Lesson learned: Check static routes, check dynamic routes but ALSO check connected routes!
Working on CCNP: [X] SWITCH --- [ ] ROUTE --- [ ] TSHOOT
Goal for 2014: RHCA
Goal for 2015: CCDP
Goal for 2014: RHCA
Goal for 2015: CCDP