Tie The Nxoose, I’m Ready To Hang Myself

Cisco IOS devices have for years offered a “datagram size” option as part of an extended ping. Juniper similarly offers a ‘size’ option for the ping command. The big difference between them – and quite an important one – is that Cisco’s “datagram size” means “size of the entire datagram including IP and ICMP headers”, where the Juniper ‘size’ option mean “size of the payload only.”

Confusing, but at least consistent. Then along comes NXOS to spoil the party.

Terminology Check

Before we go any further this is probably a good time to agree on some terminology, because it will be important throughout. Here are a couple of quick (and significantly hacked up) definitions so we know we’re talking about the same things:

Term		Definition
Packet		OSI Layer 3. A datagram (hah!). There will be some kind of Network Layer header plus higher level protocol data.
Frame		OSI Layer 2. In this case, an Ethernet frame which will be comprised of a Link-Layer Header plus the attached packet data.

In other words, if I mention a “frame”, you know it has a layer 2 header on it. If I say “packet”, I’m talking about layer 3. That becomes relevant when considering packet sizes, which is next.

Reference

There are a few useful sizes to remember as they’ll appear repeatedly throughout the post. These are the number of bytes required for the protocol headers for Ethernet, IP and ICMP which, for a frame containing an ICMP message, are all going to be present.

Protocol	Header Size (bytes)
Ethernet	14
IP	20
ICMP	8

So a typical ICMP message conceptually looks like this by the time it hits the wire:

As is common in these discussions (and indeed in the MTU values used by the routers), we have cunningly left out the Ethernet preamble and frame checksum, which would add another 12 bytes to the final frame. Given that, the minimum ICMP IP Packet size is (20 + 8) = 28 bytes (Layer 3, so excluding the Ethernet header), plus the payload.

Head Spin

Ok, I get totally twisted up trying to remember this and get it correct, so I’ll create a quick table to compare IOS and Junos behaviors regarding MTU and ping sizes.

Cisco’s physical MTU (the number you configure as mtu XXXX) excludes the Ethernet header already so on a Cisco platform, the IP MTU and Physical MTU are in apparently identical. Juniper on the other hand (rightly, I think) does not include the 14 bytes of Ethernet header in the physical MTU so for a 1514 MTU, the “protocol MTU” (i.e. how much is left for the IP datagram) is (1514 – 14) = 1500 bytes.

	IOS	Junos
Physical MTU	1500	1514
Protocol (IP) MTU	1500	1500
Maximum Ping payload	1472	1472

As a side note, what this means is that if you are configuring a link between a Junos and IOS or NXOS device, the IOS device should have the physical MTU set 14 bytes smaller than the Junos device. For reference here are the physical / protocol MTU commands that I’m talking about. All commands are at the interface/subinterface/unit level:

OS		Physical MTU		Protocol MTU
IOS/NXOS		mtu 1500		ip mtu 1500
Junos		set mtu 1514		set family inet mtu 1500

Additional ICMP Spin

I’ve listed the maximum ping payload for both platforms as 1472 bytes, which it is – if we define the “payload” as the block of data that’s included after the ICMP header. By that logic, the size of an ICMP frame should be 1514 bytes:

1472 bytes of payload
+  8 bytes ICMP header
+ 20 bytes IP header
+ 14 bytes Ethernet header
====
1514 bytes

So then, when you define the “size” option for a ping, what are you actually specifying? Again, IOS and Junos have different ideas about that. Cisco’s IOS size represents the size of the ICMP message in an IP packet, and includes the IP and ICMP headers. Juniper’s size option, on the other hand, refers only to the payload size. Consequently on a standard 1514 (or is that 1500?) byte MTU link, the size parameter (for valid frames that can be sent) runs like this:

OS	Min “Size”	Max “Size”
Junos	1	1472
IOS	29	1500

Both will produce the same final frame size, but the command will be different. Annoying, right?

Command Lines

So we’ve learned that Cisco IOS and Juniper Junos achieve the same thing in two different ways:

Cisco

ping 1.2.3.4 df-bit size 1500

Juniper

ping 1.2.3.4 do-not-fragment size 1472

To be fair to Cisco when you do an extended ping interactively, the size option is called “Datagram size”, which at least implies that the size might represent the whole datagram (i.e. the IP packet), which would explain why Cisco’s size option includes the IP and ICMP headers.

switch#ping
Protocol [ip]:
Target IP address: 1.2.3.4
Repeat count [5]:
Datagram size [100]:
[...]

Testing in IOU, inline help suggests that the minimum payload must be 8-bytes:

switch#ping 1.2.3.4 size ?
  <36-18024>  Datagram size

So far so good; as I said earlier, confusing but consistent.

Along Came NXOS

Just when you think you’ve got it all straight (and I’m sure my rambling hasn’t helped much), NXOS throws an inconsistent kind of wrench in the works. Thankfully NXOS approaches MTU the same as IOS, so it’s the ping command where things change.

Firstly, not for the first time, the syntax has changed:

 IOS: ping 1.2.3.4 df-bit size 1000
NXOS: ping 1.2.3.4 df-bit packet-size 1000 

Well, at least now the option is more clearly named to represent the fact that this is the overall size of the packet, not the size of the payload. If it were the payload, I’m sure Cisco would have named the option payload-size instead, right? To be sure, let’s check a NXOS command reference:

Well now I’m confused because that sounds awfully like a payload size range (1 byte and up). It certainly isn’t a feasible size range for a full ICMP packet, which would have to be a bare minimum of 28 bytes just for the headers. So what Cisco appears to have done here is to rename the size option to something more accurately descriptive, while simultaneously totally changing what the option now does such that the new option name is, um, utterly inaccurate. Really, I despair.

Double Check

To make sure, I’ll try some pings:

nxos# ping 1.2.3.4 packet-size 1
PING 1.2.3.4 (1.2.3.4): 1 data bytes
9 bytes from 1.2.3.4: icmp_seq=0 ttl=255
9 bytes from 1.2.3.4: icmp_seq=1 ttl=255
9 bytes from 1.2.3.4: icmp_seq=2 ttl=255
[...]

The 9 bytes being returned are the 1 byte payload plus the 8 byte ICMP header I would guess. Try the same on a Junos box:

junos> ping 1.2.3.4 size 1
PING 1.2.3.4 (1.2.3.4): 1 data bytes
9 bytes from 1.2.3.4: icmp_seq=0 ttl=255 time=2.545 ms
9 bytes from 1.2.3.4: icmp_seq=1 ttl=255 time=1.424 ms
[...]

So if the packet-size option is new and unimproved, what happens when you run an interactive extended ping in NXOS?

nxos# ping
Vrf context to use [default] :
No user input: using default context
Target IP address or Hostname: 1.2.3.4
Repeat count [5] :
Datagram size [56] : 1    <<<<<<<<<<<<<<<<<
[...]
Sending 5, 1-bytes ICMP Echos to 1.2.3.4
Timeout is 2 seconds, data pattern is 0xABCD

9 bytes from 1.2.3.4: icmp_seq=0 ttl=63
9 bytes from 1.2.3.4: icmp_seq=1 ttl=63
[...]

Here NXOS has retained the “Datagram size” option name, but predictably the true meaning is payload size.

My Vote

I say we petition Cisco to change NXOS so that the size option in both locations is changed to say something along the lines of payload-size. That way it might actually make sense.

Next up, perhaps we could change the MTU behavior so that there’s a difference shown between Physical and protocol MTU since there really is one? I hate to say that Juniper got it right on this one but, uh, they got it right on this one.

If all that made any sense – and I really hope it did – hopefully it will be useful to you at some point. I needed to get some of this down on (e)paper just so I had a good reference for it all.