Cisco IOS devices have for years offered a “datagram size” option as part of an extended ping. Juniper similarly offers a ‘size’ option for the ping command. The big difference between them – and quite an important one – is that Cisco’s “datagram size” means “size of the entire datagram including IP and ICMP headers”, where the Juniper ‘size’ option mean “size of the payload only.”
Confusing, but at least consistent. Then along comes NXOS to spoil the party.
Before we go any further this is probably a good time to agree on some terminology, because it will be important throughout. Here are a couple of quick (and significantly hacked up) definitions so we know we’re talking about the same things:
|Packet||OSI Layer 3. A datagram (hah!). There will be some kind of Network Layer header plus higher level protocol data.|
|Frame||OSI Layer 2. In this case, an Ethernet frame which will be comprised of a Link-Layer Header plus the attached packet data.|
In other words, if I mention a “frame”, you know it has a layer 2 header on it. If I say “packet”, I’m talking about layer 3. That becomes relevant when considering packet sizes, which is next.
There are a few useful sizes to remember as they’ll appear repeatedly throughout the post. These are the number of bytes required for the protocol headers for Ethernet, IP and ICMP which, for a frame containing an ICMP message, are all going to be present.
|Protocol||Header Size (bytes)|
So a typical ICMP message conceptually looks like this by the time it hits the wire:
As is common in these discussions (and indeed in the MTU values used by the routers), we have cunningly left out the Ethernet preamble and frame checksum, which would add another 12 bytes to the final frame. Given that, the minimum ICMP IP Packet size is (20 + 8) = 28 bytes (Layer 3, so excluding the Ethernet header), plus the payload.
Ok, I get totally twisted up trying to remember this and get it correct, so I’ll create a quick table to compare IOS and Junos behaviors regarding MTU and ping sizes.
Cisco’s physical MTU (the number you configure as
mtu XXXX) excludes the Ethernet header already so on a Cisco platform, the IP MTU and Physical MTU are in apparently identical. Juniper on the other hand (rightly, I think) does not include the 14 bytes of Ethernet header in the physical MTU so for a 1514 MTU, the “protocol MTU” (i.e. how much is left for the IP datagram) is (1514 – 14) = 1500 bytes.
|Protocol (IP) MTU||
|Maximum Ping payload||
As a side note, what this means is that if you are configuring a link between a Junos and IOS or NXOS device, the IOS device should have the physical MTU set 14 bytes smaller than the Junos device. For reference here are the physical / protocol MTU commands that I’m talking about. All commands are at the interface/subinterface/unit level:
|OS||Physical MTU||Protocol MTU|
Additional ICMP Spin
I’ve listed the maximum ping payload for both platforms as 1472 bytes, which it is – if we define the “payload” as the block of data that’s included after the ICMP header. By that logic, the size of an ICMP frame should be 1514 bytes:
1472 bytes of payload + 8 bytes ICMP header + 20 bytes IP header + 14 bytes Ethernet header ==== 1514 bytes
So then, when you define the “size” option for a ping, what are you actually specifying? Again, IOS and Junos have different ideas about that. Cisco’s IOS size represents the size of the ICMP message in an IP packet, and includes the IP and ICMP headers. Juniper’s size option, on the other hand, refers only to the payload size. Consequently on a standard 1514 (or is that 1500?) byte MTU link, the size parameter (for valid frames that can be sent) runs like this:
|OS||Min “Size”||Max “Size”|
Both will produce the same final frame size, but the command will be different. Annoying, right?
So we’ve learned that Cisco IOS and Juniper Junos achieve the same thing in two different ways:
ping 184.108.40.206 df-bit size 1500
ping 220.127.116.11 do-not-fragment size 1472
To be fair to Cisco when you do an extended ping interactively, the size option is called “Datagram size”, which at least implies that the size might represent the whole datagram (i.e. the IP packet), which would explain why Cisco’s size option includes the IP and ICMP headers.
switch#ping Protocol [ip]: Target IP address: 18.104.22.168 Repeat count : Datagram size : [...]
Testing in IOU, inline help suggests that the minimum payload must be 8-bytes:
switch#ping 22.214.171.124 size ? <36-18024> Datagram size
So far so good; as I said earlier, confusing but consistent.
Along Came NXOS
Just when you think you’ve got it all straight (and I’m sure my rambling hasn’t helped much), NXOS throws an inconsistent kind of wrench in the works. Thankfully NXOS approaches MTU the same as IOS, so it’s the ping command where things change.
Firstly, not for the first time, the syntax has changed:
IOS: ping 126.96.36.199 df-bit size 1000 NXOS: ping 188.8.131.52 df-bit packet-size 1000
Well, at least now the option is more clearly named to represent the fact that this is the overall size of the packet, not the size of the payload. If it were the payload, I’m sure Cisco would have named the option
payload-size instead, right? To be sure, let’s check a NXOS command reference:
Well now I’m confused because that sounds awfully like a payload size range (1 byte and up). It certainly isn’t a feasible size range for a full ICMP packet, which would have to be a bare minimum of 28 bytes just for the headers. So what Cisco appears to have done here is to rename the size option to something more accurately descriptive, while simultaneously totally changing what the option now does such that the new option name is, um, utterly inaccurate. Really, I despair.
To make sure, I’ll try some pings:
nxos# ping 184.108.40.206 packet-size 1 PING 220.127.116.11 (18.104.22.168): 1 data bytes 9 bytes from 22.214.171.124: icmp_seq=0 ttl=255 9 bytes from 126.96.36.199: icmp_seq=1 ttl=255 9 bytes from 188.8.131.52: icmp_seq=2 ttl=255 [...]
The 9 bytes being returned are the 1 byte payload plus the 8 byte ICMP header I would guess. Try the same on a Junos box:
junos> ping 184.108.40.206 size 1 PING 220.127.116.11 (18.104.22.168): 1 data bytes 9 bytes from 22.214.171.124: icmp_seq=0 ttl=255 time=2.545 ms 9 bytes from 126.96.36.199: icmp_seq=1 ttl=255 time=1.424 ms [...]
So if the
packet-size option is new and unimproved, what happens when you run an interactive extended ping in NXOS?
nxos# ping Vrf context to use [default] : No user input: using default context Target IP address or Hostname: 188.8.131.52 Repeat count  : Datagram size  : 1 <<<<<<<<<<<<<<<<< [...] Sending 5, 1-bytes ICMP Echos to 184.108.40.206 Timeout is 2 seconds, data pattern is 0xABCD 9 bytes from 220.127.116.11: icmp_seq=0 ttl=63 9 bytes from 18.104.22.168: icmp_seq=1 ttl=63 [...]
Here NXOS has retained the “Datagram size” option name, but predictably the true meaning is payload size.
I say we petition Cisco to change NXOS so that the size option in both locations is changed to say something along the lines of
payload-size. That way it might actually make sense.
Next up, perhaps we could change the MTU behavior so that there’s a difference shown between Physical and protocol MTU since there really is one? I hate to say that Juniper got it right on this one but, uh, they got it right on this one.
If all that made any sense – and I really hope it did – hopefully it will be useful to you at some point. I needed to get some of this down on (e)paper just so I had a good reference for it all.