CCNP Route Hybrid VPN question

bencmcb

The CCNP Route Cisco Press book mentions that due to Hybrid VPNs using more headers and decreasing the amount of data you can carry in a single packet, the MTU size should be lowered.

This doesn't make sense to me. Shouldn't the MTU size be increased or at least left alone so that the amount of data in a single packet will not be decreased some more?

Niqriol

Hi bencmcb,

As per the link (http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6000-series-switches/24048-148.html):

• MTU: MTU is short for Maximum Transmission Unit, the largest physical packet size, measured in bytes, that a network can transmit. Any messages larger than the MTU are divided into smaller packets before transmission.

I'm reading the same book you're reading. My understanding of this statement is that the default MTU size that a Cisco device handles is around 1500. This MTU includes the data and the required headers. Hence, if you configure say encryption, or VPN, this introduces more headers. As a result, the packet that is sent will have to reduce the space allocated to the data to accommodate the additional headers, and still keep the MTU at 1500, failure to which the packet would become too large for transmission across the link. My understanding from all this is that a packet that is too large will be dropped.

In practical terms, assume that the new headers will introduce an overhead of 150 bytes. You'd need to subtract that from the 1500, and configure the MTU size on the interface as 1350. So the router will know to limit data to the MTU size of 1350, with the rest of the space reserved for the headers. Note that you can actually tell the router what MTU size to use when it dispatches it's data across the WAN link.

Also check the following links:

Cisco IOS IP Command Reference, Volume 1 of 3: Addressing and Services, Release 12.2 - IP Services Commands: ip mtu Through transit-interface [Cisco IOS Software Releases 12.2 Mainline] - Cisco

https://supportforums.cisco.com/discussion/11276161/changing-mtu-cisco-1841-router

Just my two cents ....

What do the other Cisco gurus think?!!

Hondabuff

When using VPNs such as GRE, This will add a header to a packet. So if your packet is 1500 bytes (max size for tcp) and you add VPN headers GRE/IPsec it will cause the packet to fragment. This will cause your SA's to reset and sometimes the VPN tunnel will drop when under load. Adjusting the MTU and MSS is best practice. What ever you set the mtu to you will subtract 40 for the mss.

Resolve IP Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPSEC - Cisco


CORP-R1(config)#int tun 0
*Mar 1 00:00:39.459: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
CORP-R1(config-if)#ip mtu 1400
CORP-R1(config-if)#ip tcp adjust-mss 1360

fredrikjj

bencmcb wrote: »

The CCNP Route Cisco Press book mentions that due to Hybrid VPNs using more headers and decreasing the amount of data you can carry in a single packet, the MTU size should be lowered.

This doesn't make sense to me. Shouldn't the MTU size be increased or at least left alone so that the amount of data in a single packet will not be decreased some more?

Your thinking is pretty much correct, and changing the MTU on a tunnel interface actually isn't really changing the MTU. It's more like a logical MTU restriction.

MTU isn't really about the IP packet itself. An IPv4 packet can carry 65,535 bytes, but this limitation is simply about the length field in the header being 16 bits, and 65,535 is the decimal representation of 1111 1111 1111 1111. An IPv6 packet can theoretically be gigabit sized (if you use a certain extension header). I'm telling you this simply to point out that a computer can create whatever packet size it wants. However, actually sending it is different matter.

The MTU is a characteristic of the layer 1/2 transport, like Ethernet. It's called maximum transmission unit, and it's the largest size of a single frame's payload. In Ethernet, that's around 1500 bytes. What's different about the frame size vs. the packet size that we discussed earlier is that you can't just increase the MTU willy nilly. If you wanted to send your 50 megabyte IPv6 packet to another computer connected to the switch, you can't just increase the MTU to the 50 megabyte range and deliver it in a single frame. Instead, the packet must be split and delivered in many 1500 byte frames. This is called IP fragmentation, and you typically want to avoid that since it uses unnecessary resources. It's an especially big problem in certain scenarios involving encryption, but we'll get to that later in this post.

So far we know that IP packets can be more or less any size, and that the MTU of the transport is fixed to a specific amount that can't be changed (this isn't completely true, we'll also get to that later). To avoid fragmentation you therefore want to make sure that your IP packets are small enough that they fit into the layer 2 frames. The IP packets must be no bigger than the MTU for this to be the case.

Now consider how the end host could determine how large of a packet that it can send while still being within the MTU. One option could be that the end host decides that since the general Ethernet MTU is 1500 it will always use say 1000 byte packets since that will always fit into the MTU. Something like this happens with some applications where they create a steady stream of identically sized small segments that are well below the assumed MTU. Voice over IP calls are an example this, and you'll never have MTU issues with these since each packet is small. That is one way to overcome any potential MTU issues, but it's perhaps not very efficient to not send as much data as you can in each packet.

Another option is for the host to look at the MTU of its own interface card, which will be an Ethernet card with 1500 MTU. If the application knows this, it knows that it can create a TCP segment that's around 1460 bytes because the IPv4 and TCP headers add 40 bytes. When that packet gets to the NIC, it can be sent in one frame and we don't have the fragmentation problem discussed earlier, which is good.

Now consider what happens when that packet gets to the next hop, because we obviously can't assume that the destination is directly connected. The next hop is some kind of router. It receives the packet just fine since the end host create it to fit the MTU of the Ethernet link between them. The router then looks at the MTU of its outgoing interface and decides if it can send it, or if the packet requires fragmentation. Let's assume that the outgoing interface is a normal Ethernet interface with 1500 MTU, which means that the router can send the packet since it was originally create to fit that MTU. If every router along the path to the destination uses Ethernet, and the packet isn't modified, you'll have no MTU issues at all.

Think about what your VPN tunnel does as it adds encapsulation to your packet; it increases the total packet size. If that first hop router for example has a tunnel interface as its outgoing interface for your packet, it might add a new IP header and a GRE header which increases the packet size by 24 bytes (20 byte IP header, 4 byte GRE header). If your packet previously was at 1500 bytes, it's now 1524.

After this tunnel encapsulation, the packet has to leave the router through a physical interface. This interface still has an MTU of 1500, and your packet is now too big and must be fragmented. You see here how a tunnel interface can add extra size to a packet and make it so that it can't be delivered in one piece the same way as it could before.

Ok, so after this overview we've reached the conclusion that the tunnel encapsulation adds headers to the packet that makes it too big and causes it to be fragmented before being sent to the next hop. Let's now look at what happens if we add this configuration to the tunnel interface on our router:

interface tunnel0
ip mtu 1400

The end host is still creating 1500 byte packets since that's what its link MTU is, and the end host has no awareness of this MTU command on the router's tunnel interface (yet). The difference now compared to earlier when we didn't have this “MTU” adjustment in place is that the packet will reach the tunnel interface and will be fragmented before its encapsulated. Before, it was fragmented after the encapsulation.

Doing the fragmentation before the encapsulation is significant because it's the end host that reassembles packets. An encapsulated packet's 'end host' is the tunnel end point, not a host computer. This means that if we fragment after we've added the GRE and IPsec headers, the tunnel end point router will have to reassemble these before removing the encapsulation. This kills throughput if non-fragmented encrypted packets are handled with hardware assistance, and reassembly must be done in some kind of 'software path'. If fragmentation is done before the encapsulation, the tunnel end point router can just remove the encapsulation and send the fragments on towards the end host computer that then does the reassembly.

However, we still do not want fragmentation, so this point was made just to highlight the fact that changing the MTU on a tunnel interface is kind of a signal to the router rather than an actual change of an MTU (remember, MTU is a layer 2 concept).

We've now learned that if the packet is too big, we'll get fragmentation, and that fragmentation happens regardless of a change in the ip mtu parameter on the tunnel interface. There are three mechanisms that can fix this problem. We'll go through them one by one.

1. Jumbo frames

The obvious solution to the MTU being too small is to just increase it. This of course requires that your layer 2 device supports larger MTUs, often called “jumbo frames”. You enter some commands into the switch and, magically, it can now carry frames in the 9000 byte range. Now if the end host sends its 1500 byte packets in our scenario, they will always be small enough no matter how much extra encapsulation is added. The problem with this solution is that it must be enabled end-to-end which makes it suitable for something like a data center where you have full control over every device, but not when packets travel across multiple administrative domains.

The main reason why you would use this feature is that it allows servers to send larger than 1500 byte packets and increase throughput due to simply having to process less packets to transport a given amount of data. It's not really seen as an MTU solution per se as far as I know.

2. MSS clamping

The TCP SYN sent during the TCP three way hand shake contains an option field where the Maximum Segment Size can be specified. A segment is the TCP payload, and if the interface MTU is 1500, the host knows that the maximum segment size is 1460 because the IP and TCP header adds 20 bytes each. Both hosts involved in setting up the session set their own MSS, but its the host with the smallest value that decides on what the MSS should be.

“MSS clamping” is the process whereby a router intercepts the TCP SYN packets and modifies the MSS field. That way the hosts will be tricked into using a lower maximum segment size than they might otherwise do. If we've decided to set the ip mtu on the tunnel interface to 1400, we could decide to set the MSS to 1360. The maximum packet size will then be 1400 since IP and TCP headers at 40 bytes.

We configure mss clamping with the ip tcp mss-adjust command:

interface tunnel0
ip tcp mss-adjust 1360

Just setting mss clamping should by itself resolve most issues since it will force TCP packets to be small enough.

3. Path MTU Discovery

This post is way too long, and I don't have time to write a longer explanation, but the basic idea is that the host (or a router if it's a tunnel end point and tunnel path mtu is set) will set the 'don't fragment' bit in its packets, and adjust the segment size downwards if packets are dropped. This is where the ip mtu tunnel interface command comes into play again. In our simple scenario with a first hop router that has a tunnel interface with ip mtu 1400, it will drop packets with the df-bit set if they are above 1400.

The drop will trigger an icmp message back to the host that can then adjust the packet size down. The obvious problem with relying on this is that the first large packet always will be dropped (which is why you do mss clamping as well), and there are also issues with icmp messages potentially getting lost, etc. It also becomes tricky in scenarios where a packet goes through the first tunnel, but then you have another tunnel further down the path that require further segment size reduction. In a worst case scenario you'll need several packets in the path mtu discovery process before the correct packet size is found.

Overall MTU is kind of a tricky subject in my opinion, and there are also some difference in IPv6 vs. the IPv4 picture that I've painted in this post. The most notable difference is that only end hosts fragment in v6, but routers do become end hosts when they do encapsulation. I have to admit that I haven't really looked into how all of this plays out with IPv6 only.