Maximum Transmission Unit (MTU)

I was asked, this week by a colleague of mine, to explain to him some things regarding a Maximum Transmission Unit (MTU) setting at a customers site, where the customer told him, that he had to set his MTU to 1472 on an endpoint device in order to accommodate for Dot1Q VLAN tagging in his network (which I doubt he actually needed to do, but some times you need to choose your battles, when explaining network stuff to people who are not well-versed in such).

As I was explaining the meaning and workings of MTU size and when you would want to change it versus when not to, I discovered I actually didn't have all the answers he sought after - mainly because I did not have much of the specifics from this particular setup we were discussing, but also because I didn't actually remember all of the header sizes of IP and Ethernet - I knew the theory of it, but it is hard to convey that without specific numbers, which I feel like I shouldn't be struggling with as a CCIE candidate. So, I sat down and looked it all up, threw some pings and other traffic around in the lab and captured some packets - and now I am writing this down so that I may gain an even better grasp of these concepts myself - and also not have to spend time explaining it to someone, when I can just send them a link to this post.

Alright, on with it!

Maximum Transmission Unit, or MTU, defines the maximum size of an Ethernet frame transmitted on the network. In most cases, this would default to 1500 bytes. Testing the MTU is usually done using ICMP ping requests with a packet size set somewhere between 1300 and 1500 (depending on what you want to test) and the DF-bit set. The DF-bit means that the packet should not be fragmented and if it encounters a point on the path from A to B, that requires fragmentation, an ICMP packet will be returned stating that it needed fragmentation, but the DF-bit was set. This reply is generated by the device, which would otherwise have fragmented the packet - had the DF-bit not been set. When testing the MTU size, you should know how the OS handles ICMP sizes.

Let's do a sample with Windows 7 pinging its local gateway, which is set with an IP MTU of 1500 on its interface.

R1#show ip interface GigabitEthernet1
GigabitEthernet1 is up, line protocol is up
  Internet address is 172.16.0.1/24
  <output omitted>
  MTU is 1500 bytes
  <output omitted>

Let's try a ping size of 1500 with the DF-bit set and see what happens.

Win7 ping size 1500 w/ df-bit set

Okay, that didn't work - but why ?! We said 1500 in the length of the packet and that is what is allowed on the router interface. So, why are we needing to fragment the packet? Well, simply put, the packet is too large for the host to send out its interface without fragmenting it. The reason is that Windows 7 takes the -l 1500 command as meaning 1500 bytes of payload - then it adds the various headers (we'll get to the headers in a second). This makes the packet too big to be transmitted without fragmentation.

Now, lets try lowering the length of the packet to 1472 and see if we can get a packet through.

Win7 ping size 1472 w/ df-bit set

So, the packet was sent and a reply was received. Now, why can we only send with 1472, even though the client and the network is configured for 1500 bytes?

If we were to replicate this on a router, pinging another device with the size set to 1500 it would work - as seen in the output below, where router R2 pings R1 on the same layer 2 subnet (172.16.0.0/24).

R2#ping 172.16.0.1 size 1500 df-bit repeat 1
Type escape sequence to abort.
Sending 1, 1500-byte ICMP Echos to 172.16.0.1, timeout is 2 seconds:
Packet sent with the DF bit set
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 4/4/4 ms

The simple explanation here is, that the packet length is interpreted differently between a router and a Windows 7 machine. The router takes the 1500 bytes value as meaning a 1500 byte Ethernet frame (excluding the Ethernet encapsulation itself), whereas the Windows machine means 1500 bytes of payload - then comes the layer 3 encapsulation and subsequently the layer 2 encapsulation.

Let's break down the packet that makes it through on the Windows machine.
Ping towards 172.16.0.1 with the df-bit set and a payload of 1472.

• Payload: 1472 bytes
• ICMP encapsulation: 8 bytes (1480 bytes)
• IP encapsulation: 20 bytes (1500 bytes)
• Ethernet encapsulation: 18 bytes (1518 bytes)
• Transmit on wire

So, as seen above, the packet reaches 1500 bytes after it is encapsulated at layer 3 (IP). This is where the IP source and destination addresses are added (among other things). After that it as handed off to layer 2 and encapsulated in an Ethernet frame containing the source and destination MAC addresses. What you will most likely see when capturing packets is a a frame size of 14 bytes, which is not wrong, but as you can see above I wrote 18 bytes - the 4 extra bytes are from the FCS (frame check sequence), which most network cards don't capture in Wireshark.

Below I have a few MTU size examples. First we will look at a TCP example as seen in a Wireshark capture that doesn't include the Ethernet FCS (most network cards don't include this value).

Alright, next up we have the same example as above, but this time we include the FCS and the result is actually how packets today are sent (by default) on an Ethernet network.

The layer 2 MTU size is now 1518 bytes and this is what counts as the packet sent on the wire.

Now, let's take it a little further and add 802.1Q VLAN tags into the mix. The packet, as seen above, is transmitted to a switch, that then transmits it to another switch over a trunk link - adding a VLAN tag to the 1518 byte long frame. This VLAN tag is worth 4 bytes, which now makes the frame a whopping 1522 bytes!

In 1998, the max-packet was changed, in 802.3, from 1518 to 1522 to allow for the 4 bytes VLAN tagging. So, the new standard size in Ethernet networks today is actually considered to be 1522 - sizes above that would be considered jumbo frames.

This would be standard Ethernet network allowing for 1500 byte packets. However, the tale is not done quite yet. The Ethernet frame may be 1522 (when using VLAN tagging), but the bytes transmitted on the wire for a packet includes a little more than just the frame itself - this is unrelated to the MTU, but still a nice little fun-fact to know about.

Before an Ethernet frame is sent, the sender makes sure the coast is clear by sending a preamble and the SFD (start-of-frame-delimiter). This is known as CSMA/CD (Carrier Sense Multiple Access Collision Detection), which is used in 802.3 networks. This takes up 8 bytes on the wire. And finally, at the end of a transmission there is a silence called the Interframe Gap, which adds another 12 bytes. So, a frame of 1522 bytes becomes a total of 1542 bytes transmitted on the wire. Keep in mind that the preamble, SFD and Interframe Gap does not count towards the MTU size - neither does the Ethernet frame encapsulation including the FCS and VLAN tag (if any).

Address Resolution Protocol

Address Resolution Protocol (ARP - defined in RFC 826) is today mainly used in multi-access IP networks to identify the layer 2 address (MAC address) associated with a specific layer 3 address (IP address).

ARP request and ARP reply

This is done by sending a layer 2 broadcast requesting the MAC address for a specific IP address. To the right is a part of a packet capture of a client (172.17.6.72) broadcasting a request and then the server (172.17.6.5) responding to that request.

The ARP protocol is, strictly speaking a layer 2 protocol, but it carries information to facilitate layer 3 functions. As a layer 2 broadcast it is not routable, but by means of a proxy it can function between clients on different subnets.

Proxy ARP
Proxy ARP can facilitate communication between clients in a scenario where you wish to segment a larger IPv4 scope, but wish to change the addresses of hosts on the subnet gradually.

For example: you have a subnet of 192.168.0.0 /21. This network allows for more than 2000 hosts on the same broadcast domain. Say you would like to segment this /21 subnet into smaller /24 subnets, but some of the hosts include printers that have statically configure IP addresses and subnet masks on the devices. You configure your subnets so that clients are located on the subnet 192.168.1.0 /24 and the printers are in the subnet 192.168.2.0 /24. The clients acquire their address from a DHCP server so they will get the new /24 subnet mask automatically, but the printers are all statically configured with a subnet mask of /21. This means that the clients trying to contact a printer knows that the printers are on another subnet and therefore sends the traffic to their default gateway, but the printers will think that the clients are on the same layer 2 segment as them and will attempt to look them up using an ARP request.
This is where proxy ARP comes into play. It is configured on the interface of the router connected to the printer network so that if it receives an ARP request on the printer interface, requesting a MAC address of a client, it will go ahead and reply to the ARP request with its own MAC address in the reply. The result: the client routes traffic as it was supposed to and the printer sends the traffic to the MAC address of the ARP proxy router, thinking it is the MAC address of the client machine.
Now, the administrator in charge of changing the IP address (or at least the subnet mask) on the printers can do so at his or her own pace.

I am not saying this is a recommended scenario - just that it is possible given the proper circumstances.

There are many other uses for Proxy ARP, which is turned on by default on Cisco devices. You can turn it off on a per interface basis with the command no ip proxy-arp.

Gratuitous ARP
The gratuitous ARP is a special form of ARP request that functions like a reply, but a reply that no one asked for - hence the gratuitous part. It is an ARP request and not an ARP reply mainly because it would be the type of ARP message most likely to be processed by the largest audience of endpoints. More can be read about the specific reasoning here.

The following is true about a Gratuitous ARP packet.

• The destination MAC address is the layer 2 broadcast (ff:ff:ff:ff:ff:ff)
• The source and destination IP address is that of the host sending the gratuitous ARP packet
• No reply is expected

Gratuitous ARP is used to inform hosts on a subnet that a MAC address has changed. This feature is used by HSRP and VRRP to notify the network (mainly the switches) of where to forward packets to their virtual IP address.

ARP Probes
A feature in IPv4 uses ARP probes as a means of Address Conflict Detection (defined in RFC 5227) by using ARP requests to check if an IP address is already in use in the particular broadcast domain.

Reverse ARP
Reverse ARP is where an ARP request is sent out on a subnet with the source and destination MAC address being that of the sender and the source and destination IP address being all zeros (0.0.0.0). This allows a Reverse ARP server to reply to this request with a reply to the MAC address of the sender with an IP address, which needs to be manually defined on the Reverse ARP server beforehand. Essentially making an entry stating which IP address is associated with which MAC address. This is a means of dealing out IP addresses to hosts without statically configuring them on the hosts themselves. This is rarely used today (I have never encountered a network using Reverse ARP for addressing purposes) because of the cumbersome act of manually configuring the database with IP to MAC mappings and the need for the Reverse ARP server to be reachable on the clients layer 2 subnet.