I’ve interviewed many network engineering candidates over the last few years, and one theme that keeps on coming up over and over again is a total lack of understanding of what I consider to be “the basics.”
Do younger engineers really have no in depth knowledge?
First let me say that I am conscious that there may be an implicit bias here in terms of what these “basics” would be, based on my own experience. Maybe my idea of what’s a basic fact is just out of whack with reality; you can tell me so in the comments if so. But let me share a couple of examples of what I’m talking about.
How ARP Works
If two IP hosts want to communicate with each other over Ethernet, how do they obtain the layer 2 addresses necessary to populate the ethernet frames correctly if they’re on the same IP subnet? What about if they’re on different IP subnets? To what address is the ARP sent (at layer 2), and who answers it?
Even in the last few months, I’ve seen problems caused by ARP after network failovers. If you don’t understand ARP, how can you reasonably troubleshoot an IP network?
I have run out of fingers and toes on which to count the number of times I have been told that the ARP packet is forwarded to the destination host, and the destination host responds with its MAC address. When I extend the question so that the ‘remote’ host is google.com, only about 25% of those who took this stance stop and say “Uh, wait, that can’t be right.”
ARP Table vs CAM Table
Here are some terms that are often thrown into conversation and used interchangeably as the situation merits. And I don’t mind for most of them, but ARP Table?
- ARP Table
- CAM Table
- MAC Address Table
- Switching Table
- Bridging Table
I have been assured that switches maintain ARP tables and answer on behalf of devices on the same subnet. The ARP table is apparently populated by devices broadcasting their ARP and the switch hears it and adds the IP to its lookup table. It also tells the router all about it so the router will know too. By this point in the explanation I’m tightening the noose around my neck so I lose track slightly.
One of the problems with computer-based testing is that it rewards people who can memorize definitions and facts by rote, rather than proving that they necessarily understand the technology itself.
In “Surely You’re Joking, Mr Feynman” Physicist Richard Feynman recounts teaching a Physics class at a Brazilian university (it’s long, but stick with it):
I discovered a very strange phenomenon: I could ask a question, which the students would answer immediately. But the next time I would ask the question – the same subject, and the same question, as far as I could tell – they couldn’t answer it at all! For instance, one time I was talking about polarized light, and I gave them all some strips of polaroid.
Polaroid passes only light whose electric vector is in a certain direction, so I explained how you could tell which way the light is polarized from whether the polaroid is dark or light.
We first took two strips of polaroid and rotated them until they let the most light through. From doing that we could tell that the two strips were now admitting light polarized in the same direction – what passed through one piece of polaroid could also pass through the other. But then I asked them how one could tell the absolute direction of polarization, for a single piece of polaroid.
They hadn’t any idea.
I knew this took a certain amount of ingenuity, so I gave them a hint: “Look at the light reflected from the bay outside.”
Nobody said anything.
Then I said, “Have you ever heard of Brewster’s Angle?”
“Yes, sir! Brewster’s Angle is the angle at which light reflected from a medium with an index of refraction is completely polarized.”
“And which way is the light polarized when it’s reflected?”
“The light is polarized perpendicular to the plane of reflection, sir.” Even now, I have to think about it; they knew it cold! They even knew the tangent of the angle equals the index!
I said, “Well?”
Still nothing. They had just told me that light reflected from a medium with an index, such as the bay outside, was polarized; they had even told me which way it was polarized.
I said, “Look at the bay outside, through the polaroid. Now turn the polaroid.”
“Ooh, it’s polarized!” they said.
After a lot of investigation, I finally figured out that the students had memorized everything, but they didn’t know what anything meant. When they heard “light that is reflected from a medium with an index,” they didn’t know that it meant a material such as water. They didn’t know that the “direction of the light” is the direction in which you see something when you’re looking at it, and so on. Everything was entirely memorized, yet nothing had been translated into meaningful words. So if I asked, “What is Brewster’s Angle?” I’m going into the computer with the right keywords. But if I say, “Look at the water,” nothing happens – they don’t have anything under “Look at the water”!
This sums up with depressing accuracy the state of so many technical interviews. I might ask somebody to describe Spanning Tree Protocol, and they will give a definition of which Radia Perlman would have been proud. Ask them what happens when I connect two links between two switches without using some kind of port aggregation protocol and they have no idea.
Don’t get me wrong, I do also speak to some people who really do understand these things. But I know I’m not alone in having found that the two examples I gave above are depressingly common in candidates interviewing for quite senior roles.
Why Is This?
Why is this happening? People are exiting high school with an understanding of IP that used to be the exclusive realm of the consultant. And perhaps that’s the problem; the basics aren’t being taught, they’re being learned along the way. Then when these young whippersnappers move on to college and beyond, perhaps nobody goes back and makes sure that they actually understand this “IP” thing they seem to know about already beyond typing IP addresses and default gateways into an operating system. Am I being unfair?
Maybe these candidates are a victim of abstraction. IP tends to work pretty well, and while it does, who needs to know anything about how it really works? I mean, I don’t know how my TV really works in detail – I turn it on and it works (most of the time). Similarly, when I code in Perl, I’m abstracted from the underlying machine code necessary to drive the CPUs, and I wouldn’t have the first clue where to start to optimize code to work for a particular processor (assuming I even could with Perl!). Are we all in our own way so focused on the higher level capabilities that we take the underlying functionality for granted?
Am I wrong to expect a senior network engineer to understand these things? Or should I roll with it and accept that it’s the higher level protocols that are more important? I’d love to hear your opinions.