I’m stating the obvious here perhaps, but every time a temporary solution to a network problem is proposed, I feel morally obliged to laugh and say “And by ‘Temporary’, you mean ‘Permanent’.” This usually leads to a round of indignation about how of course it doesn’t mean that, and in a few weeks they will go back and clean up whatever the temporary solution was and put in the more complex long term solution.
But you and I both know that won’t happen, don’t we?
We’re all put in the unfortunate situation every now and again where we’re asked to just make something work quickly, regardless of how we achieve it, because the business apparently needs this problem fixed yesterday. The problem comes when, in order to fix the problem quickly, the only option is to do something that you know is wrong, inconsistent with how you normally do things, or just so architecturally nasty that while it might work, you know it’s a disaster waiting to happen – a network hack. This rather neatly wraps up two of the three elements of the consultant’s paradigm, perhaps better known as the Project Management Triangle:
Put succinctly, given the three constraints of Time (fast), Cost (cheap) and Quality (good), you can pick any two of the three. The more you focus on any of them, the more you move away from the others. i.e. A project can be delivered quickly and cheaply, but probably won’t be done well. Conversely if you delivery ultimate quality in a short time frame, it will probably cost a lot. So applying this to our situation, it’s clear that in this case we’re battling between Fast and Good – i.e. You must fix it now; “doing it right” be damned.
So Why Do We Do It?
The business pays the bills, and our wages, so ultimately most of us will put the hack in because that’s what we do in order to keep our employer in business and thus keep our pay check coming. In the past I’ve tried putting my foot down, but unfortunately project managers and executive management usually don’t get past the first half of the first sentence where they hear “Look, this solution might work short term, but…” – and they’re done. All that matters is that it’ll work, so stop wasting time and do it already.
The Five Stages of Network Hacks
And so, I present to you, the Five Stages of Network Hacks (click to see full size):
If it’s hard to read in the browser, you can try The Five Stages of Network Hacks (PDF) instead.
The Sixth Stage
I think we never truly accept the hack as being right, but sometimes, gosh darn it, you just have to move on and deal with bigger issues. This does however lead on to the missing Sixth Stage of Network Hacks which should likely be “Mockery”. This is what happens when a new engineer (or the external consultant) joins the team, looks at how something was implemented, and says “What idiot decided to do this that way?” Usually this results in a conversation over a lunch that you offer to pay for, where you try to explain how this all came about, and you attempt to rebuild what little respect the newbie has left for you.
Confessions and Coping Strategies
So how do you cope in this situation? It sounds spineless to bow to the the corporation’s demands when you know the solution is technically wrong, but sometimes you just don’t have a choice. Or do you? Let me know how you avoid this situation.
If you have had to put network hacks in, perhaps in the Comments you can confess (without naming names) to a hack that you’re particularly ashamed of and wish you’d never had to do. I cannot absolve your sin, but I can at least give a sympathetic pat on the shoulder and tell you it’s ok.
Have at it!
Ha, great read John! Fast/Good/Cheap is spot on. Though more often than not I feel like it is all a hack anymore.
cheers!
What about disillusionment ?
Oooh, a 7th stage? We may be redefining a standard model here, in a rather effective way!
This is hilarious. When I first started out in the campus, half the ports on a blade in a 4006 went bad. Due to the remoteness of the site and cost getting out there and some other reasons the hack I was instructed to implement was to put tape across the bad ports and admin them down with description. The tape on those ports has always stuck with me as something that was just wrong and needed to be fixed – even if they were not or need to be used.
Great post John, this is a fantastic summary. I still give people the ‘how could you be so nieve look’ when they describe a temporary fix. The experienced managers look back with a look that says ‘I know, but we’re doing this dance, and you know I’ll say what ever it takes to get you to fix it now’ .
The best I can do is highlight the cost (hours and money) to undo the hack, which never works. But it’s not just the managers though. I lie to myself also. The hacks are too numerous to mention. PBR, *temporary* ACLs, static routes, single homed routers. partially configured switches. The list goes on. Thanks for the PDF, I’ll be posting it on my wall in work.