Rate my Corp Network, Bad design or not?

Futura

Funny thing is that this may be good or it may be bad? trying to explain this as basic as poss. It was designed by a major telecoms company here in the UK, and I'm trying to take ownership of it.

3 hub sites all with L3 switches installed. Wan links between all sites, Looks like a triangle

Each site has two switch ports assigned to vlan 800 conected to each of the other two sites and an IP Address in the same range on the vlan interface 800. All the WAN link are 100mbps

Obviously 1 of the ports is blocked due to stp.

this is the core of the network btw, I have about 60 site all linked off each of the cores.

Wonder what you think of this design, is this ok? or could it be improved?

Nate--IRL--

I'm far from knowledgeable about this stuff - but that strikes me as odd. IMO it would be better if the core operated at L3, rather than just L2, with each link in the core as a separate subnet. That way each link is routed, and will be utilised. As it is now, the root bridge is handling all of the traffic in the core.

Nate

fluk3d

Futura wrote: »

Funny thing is that this may be good or it may be bad? trying to explain this as basic as poss. It was designed by a major telecoms company here in the UK, and I'm trying to take ownership of it.

3 hub sites all with L3 switches installed. Wan links between all sites, Looks like a triangle

Each site has two switch ports assigned to vlan 800 conected to each of the other two sites and an IP Address in the same range on the vlan interface 800. All the WAN link are 100mbps

Obviously 1 of the ports is blocked due to stp.

this is the core of the network btw, I have about 60 site all linked off each of the cores.

Wonder what you think of this design, is this ok? or could it be improved?

Do you have a set of business requirements that you have to meet or is this design just something you put together?

Futura

Nate--IRL-- wrote: »

I'm far from knowledgeable about this stuff - but that strikes me as odd. IMO it would be better if the core operated at L3, rather than just L2, with each link in the core as a separate subnet. That way each link is routed, and will be utilised. As it is now, the root bridge is handling all of the traffic in the core.

Nate

This is exactly what I was thinking, can't understand why it would be layer 2 on the core, and wasn't sure if this was a common way of working?

fluk3d wrote: »

Do you have a set of business requirements that you have to meet or is this design just something you put together?

It was put together by (not my words) an 'Expert', anyways, the only thing I can think of is that stp will converge if one link goes down, thats the only good thing I can think of. I'm sure there are other ways of having redundancy though?

Nate--IRL--

Futura wrote: »

It was put together by (not my words) an 'Expert', anyways, the only thing I can think of is that stp will converge if one link goes down, thats the only good thing I can think of. I'm sure there are other ways of having redundancy though?

Routing would be be more efficient than STP, as it allows the use of all links, with redundancy.

Nate

sratakhin

How many VLANs do you have? You could use Per VLAN Spanning Tree to load balance some of the traffic.

Edited: Oops, nevermind. One VLAN won't do any good

mapletune

This is interesting as I was just reading all about 802.1q trunking, 802.1ad QinQ tunneling, 802.1ah Provider Backbone Bridge, and the latest 802.1Qay PBB-TE.

I only understand QinQ; and what that allows you to do is to create a tunnel through your provider and connect your LAN/VLAN to another location.

The reason why QinQ and all the rest in relevant to this Thread, is this question: What's the advantage of a large cross-location LAN/VLAN? They created all these technologies to specifically link Layer 2 networks together, instead of routing at Layer 3. So, there has to be a specific need in some part of the industry.

The previous "expert" who linked those three locations together on Layer 2 basis might know the answer!

JuniperGuy

Only if you have protocols that need L2 function between them or want to keep and OSPF domain across WAN links. Routing is more efficient though for sure. If it's the core around these three sites, you could probably do some MPLS as well.

mapletune

I see, thanks Juniperguy.

How about, price of a L2 WAN link vs price of a L3 WAN link?

Futura

JuniperGuy wrote: »

Only if you have protocols that need L2 function between them or want to keep and OSPF domain across WAN links. Routing is more efficient though for sure. If it's the core around these three sites, you could probably do some MPLS as well.

This is interesting!, let me expand, at each of these three core sites there are around another 30 wan links expanding to small branch offices. each switch port at the core is connected to wan link with its own vlan number say 801 - 831 assigned to the port its connected to and a vlan interface is created . At the other end of the link the branch, 1 port is connected to the vlan number 801 with a address on the same subnet, usually a /30. the rest of the ports are connected to vlan 2 for the users machines.

Again, seems odd, rather than having it routed on both ends, both ends are on a vlan with SVI's on each end also.

OSPF is enabled on all the switches and is working, and the routing table is as long as my arm,

networker050184

I wouldn't go with L2 links with SVIs unless there is a reason for it.

MickQ

Very odd. Unless there's a specific business reason for it being L2, I'd have L3 links across the WAN. Nate mentioned some of the reasons.
We have L3 MPLS between our sites, and are running OSPF.

How far apart are the three sites? Why is it able to run at 100mb (bidirectional?)? What's the L2 and L1 connectivity technology?

Forsaken_GA

Futura wrote: »

This is exactly what I was thinking, can't understand why it would be layer 2 on the core, and wasn't sure if this was a common way of working?

If you're running virtualization technology between the sites, and need to move VM's around for high availability reasons, then having your data center interconnects at layer 2 is very common. Virtualization is the driving reason for moving back to flat layer 2 networks instead of routed layer 3 links.

That's the primary use case for layer 2 interconnects, but there are others. Whether it's a good idea or not depends entirely on the rest of the topology and what kind of traffic is going across the networks, as well as the businesses defined needs, none of which you should be willing to reveal on a public forum

Futura

Forsaken_GA wrote: »

If you're running virtualization technology between the sites, and need to move VM's around for high availability reasons, then having your data center interconnects at layer 2 is very common. Virtualization is the driving reason for moving back to flat layer 2 networks instead of routed layer 3 links.

That's the primary use case for layer 2 interconnects, but there are others. Whether it's a good idea or not depends entirely on the rest of the topology and what kind of traffic is going across the networks, as well as the businesses defined needs, none of which you should be willing to reveal on a public forum

Fantastic, this makes sense now, we have VMWare running with mirrored netapp volumes at two of the sites, we also have Call Manager 8.5 running on VMWare also with HA.

Many thanks for clearing this up for me, funny thing is that the design was created before the VMWare and Netapp infrastructure was in place. Extremely lucky forward thinking there I think.

mapletune

Thanks OP for sharing an interesting case =p also thanks to all contributors for shedding some light on L2 links =D learned a lot.

Forsaken_GA

Now the question y'all should be taking away from this is 'why does virtualization require layer 2 adjacency?'. Getting a firm understanding of that is actually a pretty good grounding in how layer 2 and layer 3 interoperate

instant000

Forsaken_GA wrote: »

Now the question y'all should be taking away from this is 'why does virtualization require layer 2 adjacency?'. Getting a firm understanding of that is actually a pretty good grounding in how layer 2 and layer 3 interoperate

Because I want the flexibility to move a host around within the L2 space, I can retain its MAC address, and any communications to it.

If I move this host around in the L3 space, then I have to re-address the host, and the MAC to it would change (due to having to ARP for a gateway to reach the host) and then any communications sessions with this host would get nerfed and have to be rebuilt.

So, due to vm mobility (that is, a vm could show up here, then in a litlte bit, show up somewhere else --which is one of the main benefits of virtualization), you wind up with a flat L2 network design in an attempt to compensate.

if you move the host within the same L2 space via HA, you can have a communication session going to it, and it won't break. if you try the same across L3, it's guaranteed to break, as the session is defined by a particular IP/port ... and the IP changed. (Most people reach resources via DNS, and if the L3 address changes, you just introduced issues with that, also.)

Well, that's my theory around it, based on the problem case of the VM having to move around from one host machine to another host machine. If it's within the same broadcast domain, any communications sent to its MAC address will still reach it. If it switches broadcast domains, then the problem of MAC addresses being restricted to broadcast domains thus requires communications through a default gateway, which then requires an ARP, which means that your seamless high availability just became un-seamless.

Roguetadhg

I found this thread insanely interesting. Thank you, Futura. Rep for knowledge of what outside is like

Forsaken_GA

instant000 wrote: »

Well, that's my theory around it, based on the problem case of the VM having to move around from one host machine to another host machine. If it's within the same broadcast domain, any communications sent to its MAC address will still reach it. If it switches broadcast domains, then the problem of MAC addresses being restricted to broadcast domains thus requires communications through a default gateway, which then requires an ARP, which means that your seamless high availability just became un-seamless.

You hit the nail on the head. VM HA requires layer 2 adjacency due to the nature of how layer 2 and layer 3 interact via ARP. Which leads to flat layer 2 networks, which leads to scaling issues due to the number of mac addresses the switches involved have to retain in their CAM tables, especially if you're running a multi-tenant data center.