SDN – The Natives are Getting RESTful

Oddly enough my last post (which was, to be fair, filled with enough hyperbole to sink a million or so ships) generated some rather strong responses, both in the comments, on Twitter and on Reddit.

I’m not going to tell you which was my favorite feedback (though much of it raised a smile), but I’m going to take a moment to follow up on that post with a few responses and a few more thoughts because even once you put aside the deliberately black and white opinion I put forward, there’s still something to think about here.

I’d say that if you aren’t considering how the move to SDN might affect your role (if not your job) in the future, you’re walking around with your eyes firmly closed.

Assumptions

I made some very intentional assumptions with my post, so let’s talk about those briefly. The biggest one is that of the penetration of SDN throughout the industry. After all, it’s not like every data center will convert to SDN in the next five years, and many – especially smaller DCs – may never make the change. SDN adoption will be tempered by things like:

• Suitability for a particular environment
• Successful products from vendors, especially in the controller space
• Budgetary constraints
• Benefits being seen by early adopters (and people wanting to imitate)

So for example, while SDN may be a very good fit for a large, highly-virtualized datacenter desiring a leaf/spine architecture, it may be less suitable for a service provider (in some ways at least) or a smaller enterprise, for whom the cost and complexity may just be too much for them, or the gains just wouldn’t justify the effort to change. There are exceptions, of course.

Let’s assume though for the rest of this article that we are looking at an environment where SDN is a good fit.

The Network Team Is the Bottleneck

When I say that the network team is the bottleneck, it’s not just corporate process that gets in the way (even though many companies will allow scripts to make changes 24/7 while restricting human changes to planned maintenance windows). I gave the example of turning up a switch port and a VLAN, which perhaps sounds like a trivial task for a network engineer, which it is – until you step back an look at the bigger picture. Many networks today need to have the same VLAN running on multiple access switches, connected via the aggregation (distribution layer). Unless you choose to run VTP (ugh!) or choose to just trunk all VLANs on all trunk ports rather than pruning them (in which case I’d ask you what on earth you are thinking), turning a switch port up may involve steps such as:

• Checking that the port is actually available
• Checking that the VLAN exists on the access switch already; if not, create it.
• Checking that the VLAN exists on the aggregation layer; if not, create it on each aggregation layer device and, if needed, trunk it between aggregation layer devices.
• Checking that the VLAN is trunked from the access switch to the aggregation layer
• Checking that the VLAN is trunked to that access switch from the aggregation layer
• Any decent data center will have a dual-homed host, so repeat much of the above again.

Those steps alone can generate an irritating amount of configuration and, once that configuration has been peer reviewed and approved (you DO have an approval process for scripts, right?), then the scheduling process can take its toll. And all this is for a simple switch port. I suppose you could just write a few lines to configure the access port and hope for the best, but that doesn’t sound like a particularly thorough process to me.

Of course, before we even get to the point of configuring a switch port, somebody presumably allocated a VLAN and a host IP (both automatable), and the owner will have determined which services need to run on the device and to whom they needs to communicate (automatable security via templated profiles). Load balancing? Why can’t that be done in SDN as well? After all, it’s just a set of rules, and computers just love rules.

Suitably intelligent front-end software can take the requirement to enable an access port (even a trunk), determine the dependencies across the network, and generate the necessary reconfigurations to be pushed out via whatever mechanism is most appropriate.

The Human Element

I’m not entirely sure I agree with it, but most companies are hesitant to trust human change implementations. If I had a dollar for every time I’ve heard “We shouldn’t need to schedule an access port turn up during a maintenance window,” – because it’s such a simple change – I’d have a few more bucks than I do right now. This attitude usually lasts until, inevitably, one of those “simple” changes gets screwed up horrifically and takes down the business for a while (and don’t even start me on attaching blade server switches). For some reason, people trust computer-driven changes far more than they do human changes, even though we’re then reliant on somebody coding things correctly, which isn’t always smart. Still, my guess is that if you can create a front end system that will allow users to input information based on application or server profiles and generate the appropriate configurations throughout the network, that tool will be allowed to run any time of day or night. In fact, if you want to look at the idea of dynamically moving servers around, it’s pretty much a necessity to integrate the network changes into the server orchestration processes.

And that’s where I see things going in many networks; changes that used to be worked on manually by network engineers will become automated (which is a good thing) by systems that use templates to determine the network configuration requirements and provision them automatically and move them around with the virtual host. And you think that firewall rules can’t be implemented by a script? You must be kidding me. The increased requirements for security are actually met better by a consistently deployed and verified process than by humans who screw up as often as not (remember: “when it’s broken, it’s always the firewall”).

We Still Need Experts!

Sure we do. We won’t all be out of a job, and indeed I’d argue that most likely, we network engineers will have to migrate our role to what’s needed at the time. But for all those things that currently take up our day today, there are a few ways to look at how things might work out once most of our current daily tasks can be automated by somebody choosing a profile on a web front end:

1. We will all be kept on, and will be able to find more exciting things to do with our day.
2. The company will hire tier–1 engineers to manage the onerous task of entering requirements into a web page and will retain a few high level engineers to architect the network and troubleshoot when things go wrong.
3. The network will get so intelligent and well monitored (consider Cisco’s Insieme monitoring capabilities) that only a few Tier 3/4 engineers will be retained for overall architectural control and troubleshooting; provisioning functions will be moved to the server team (as it will be integrated with their orchestration solutions).
4. We will all become Python and Ruby bitches.

I’m sure there are other options, but consider that:

• Up to a point, the efficiencies of SDN will mean that the average network team goes from being wildly under-resourced to being appropriately resourced. Hooray!
• After a point, the efficiencies of SDN and the migration of certain key functions to other teams and processes lead to the network team being over-resourced and thus thinned out. Maybe people aren’t actually fired, but their jobs will have to change significantly, because what you’re doing today will not be what you’re doing in five years in many cases.

Why The Long Face?

Too pessimistic? Yes, perhaps. People will still need to design networks, manage capacity, create the magical templates, and undoubtedly tweak the controllers that manage the network, and there will likely be a new skill set arising around those tasks. We still have WANs to deal with too; SDN really only works within your administrative control – it’s not so useful once you hand off your data to a service provider, and yes, we will still need people who understand eBGP for example. Will we need to understand OSPF? That’s a good question. Many of the current solutions that are “SDN” seem to involve automatic assimilation of new nodes and automatic routing. We may simply need to understand how to feed routes into a fabric, and get them out on the other side, and let the fabric do the rest automatically using unicorn dust and fairy magic. How well that works will be determined largely by how amazing the controllers end up being, or on the particular solution being used.

Career Suicide

So with those caveats stated a little more clearly, let’s look ahead. I was fascinated that three people expressed concern in the comments on my last post that there would be a wide gap between the tier 1 engineers and the tier 3 engineers, and no way to get those tier 1 engineers the experience necessary to move to tier 2 and 3 because the network they are working on is so abstracted and automagical that they never really get to dig into it. Perhaps we will end up in a situation where engineers come in at Tier 1 for a while, move out to a non-SDN environment to train for Tier–2 and Tier–3, then come back to SDN environments as architects and gurus?

In five years? Probably not; forgive me for being deliberately aggressive there. However, the landscape is changing. As much as network engineers can expand their scope to do programming and server provisioning, the server engineers can expand their scope to do the network engineer job too. And if you’re in the middle of that resource flattening, I can only imagine that people will be squeezed out in one direction or another.

Aww, Poor Baby

Not me; this isn’t really a selfish whine about my own career, despite how I positioned the previous article. I’m actually fairly confident about what I’ll do if and when these changes come to fruition, and I’m good enough at programming that I’m not concerned, ultimately, if that becomes a key part of my job (perhaps not ideal, but it doesn’t scare me). How about you? How about your colleagues? If your environment gets automated, what are your plans? If nothing else, I encourage you to think about the future. It’s not a negative thing – I think we’re in the most amazing phase of networking since IP became a thing and the Internet developed into something ubiquitous.

I’ve been lucky enough to ride on the wave of IP. I worked in the mobility space and rode the wave of next generation smart phones, from 2G through to LTE. I believe that the SDN wave could cause bigger ripples than either of those, and I’m looking forward to getting wet. But whenever waves are involved, some people get hit from behind and go under. So keep your eyes open, and grab onto a boogie board – we’re all in for a heck of a ride.