My CCIE Story – Part 4

In Part 4 of my CCIE Story, I bring you the thrilling conclusion of my first attempt at the CCIE lab, as I continue with Day 2 of the exam in Brussels. If you are just joining us, it’s worth checking out Part 1, Part 2 and Part 3 “on demand” as it were.

Will I pass? Or will I fail, thus opening the door for Part 5 where I tell you about my next attempt? Read on to find out!

CCIE Story – Part 4

At the end of Part 3, you may recall that I had just gone to sleep after getting 42 out of 45 on Day 1 of the lab exam, and I was feeling pretty good about things when I woke up. On Day 2, I arrived at Cisco ready to attack whatever they could throw at me, knowing that I only needed 13 points out of the 30 available in order to be allowed to do the Troubleshooting section.

Day 2 – Morning

I sat down at my desk in the lab room and opened the exam paper. Oh joy – it’s all Appletalk. DECnet thank goodness had been removed from the lab exam the month before I arrived as I recall, but Appletalk would remain in the lab exam for a few months yet. I started working through the questions, and again much of it involved low-point basic setup work which was a no-brainer.

The higher-point trick questions though were a real pain. In particular, one part required me to do something like filtering a server advertisement so that it was only seen in a particular zone or something like that – but there was more to it than just that, and it wasn’t as simple as just filtering the name. I know I tried and tried to get it right, and just could not get it to do what was requested. Another question stumped me similarly – it honestly just went beyond my hands on knowledge of Appletalk (hardly a shock when you think about it – how many Appletalk networks have you had hands on with?).

Suddenly I realized that my 17 point cushion from Day 1 was looking a lot weaker than I had thought.

Day 2 – Morning Grading

The end of the morning session arrived, and once more we filed into the break room to gulp down some lunch in the optimistic hope that we would be asked to come back into the lab for Troubleshooting. The Proctor called my name and I went back to the lab room to run through the morning’s work with him. It was about as bad as I had feared – I had totally dropped the points on the questions I had messed up, and worse I had missed some other items too. This was a nightmare – I was about to be sent home over a dead protocol…

The Proctor made a few quick calculations and gave me my score for the morning’s work. From memory – and honestly, it’s all a bit of a fuzz as I was in a frightful state at the time – I scored 17 out of 30. I had wrecked the safety margin I had built up, and scraped through to troubleshooting by 4 points. Thank goodness I had aced Day 1 somehow, because otherwise I would have been going home right then. After the proctor had finished grading all the candidates and we had finished our lunch and dosed up on caffeine, the three of us that remained were invited to come back to the lab room for Troubleshooting.

Day 2 – Troubleshooting

The Troubleshooting section of the CCIE Lab had a reputation among CCIE candidates for being the bit that would make you fail – if you even got that far, that is. It was spoken of in fearful tones as being a nightmare; a total impossibility. Worse, since I was running at a score of 59 out of 75, I would have to get 21 out of 25 on Troubleshooting to pass the lab in total. Not good odds, all things considered.

So what is the troubleshooting? Oddly, there wasn’t much information out there about it (apart from the general FUD that was floating around) and I had found very few specifics. As it turned out, what you were being asked to troubleshoot was the very same network you had just spent a day and a half building, so it was actually a familiar architecture! I was being asked to ‘fix’ a whole bunch of the features and configurations I had already created. So what’s the problem?

I sat down at my desk and opened up the console terminal windows to my routers again. Holy cow … every window was scrolling pages and pages of syslog errors. The messages were coming constantly, and were so frequent that you could barely type a command and see the result. So that’s the first problem – being able to see what you need to do. I looked at my list of ‘things to fix’, and then looked back at the terminal windows again. I realized that before I could even start on the list, I needed to fix the problems in the network.

The best way I can describe what was done to break the network in the Troubleshooting section is that somebody takes each configuration you have created, puts it in a cement mixer for a few minutes, then pours it carelessly back into the device from which it originated. To paraphrase Eric Morecambe in the Grieg Piano Concerto sketch with Andre Previn (probably a reference more recognizable in the UK), “I had all the right config. But not necessarily in the right order.” Things looked kind of right, but many things were not working. Subnet masks had been changed. IP addresses had been changed. OSPF network statements had been tweaked. Apple process IDs had been duplicated. Correct config had been moved from the right interface to a wrong interface (particularly sneaky, I thought), and random interfaces had been shutdown.

There’s only one way to attack this kind of thing, and that’s methodically and slowly. I turned off the logging (except when I needed to get more information) and went through the steps of validating each protocol (IP, IPX, Appletalk) – interface addressing, confirm connectivity, check routing protocols are working, and so forth. On one router, even though I moved the configuration from the wrong interface to the right interface, it still wouldn’t come up. I had another router with a dead interface too, and after troubleshooting for a while I concluded the switch was screwed up. I opened a terminal to the switch console and tried to log in. Nope – wrong password. Hmm. I double checked with the Proctor that this was an intentional part of the test, and he simply smiled. Yeah, ok then, password recovery it is.

If you recall the layout of the exam room, the lab equipment was in an adjacent room:

The password recovery procedure for a Catalyst 5000 switch involves power cycling it, then at the password prompt hitting ENTER, then setting the password and enable password to something new, all in 30 seconds. Cue me, in the equipment room, flipping the switch on my Cat5k, and practically running back to my desk so that I could be at the terminal in time to hit ENTER and complete the configuration required. Ugh. Still, I managed to get control of the switch again, checked my ports and found them disabled. Alright – easy stuff! I re-enabled them and checked the router ports again. Still down. Digging further I discovered that for kicks they had also enabled port security for me. Thanks, guys! A few commands later and the ports were back up and I was in business.

I was half way through my troubleshooting time now (from memory there were 2 hours for this section, and I had burned up about 1 hour and 10 minutes), and I had yet to work on a single element that would actually score me points – I had spent all the time just normalizing the network so that the underlying protocols were in good shape again.

I spent most of the rest of my time rushing to fix the features that were on my list, and was hoping I was earning points at last! I went back and tested what I could with the remaining time, and was happy that I had done a fair job with the tasks at hand. When you think about it, although it was counter-intuitive to put aside the actual point-bearing questions and spend more than half the allotted time cleaning up the network, that decision really was critical. If I had not fixed everything that was wrong with the underlying connectivity, it would have been hard to tell whether any problems re-establishing the ‘features’ they wanted were as a result of my bad configuration, or something broken “underneath” that I had chosen to skip fixing initially. As spur of the moment decisions go, I think it was a good one.

Time’s up – this is it.

Day 2 – Troubleshooting Grading

Back again to the break room where the three of us exchanged uneasy glances as we drank sodas and waited nervously for the Proctor to give us news. Finally he came in and asked me to follow him back to the lab room.

We sat at his desk and again ran through the results of the troubleshooting. He turned his screen slightly and brought up some of my configuration, and for the second time in 24 hours I found myself saying “Oh for heaven’s sake… please tell me I didn’t forget to <x>!”. “Yep,” he said, “you sure did. Again.” Idiot. I had stupidly duplicated the exact same mistake as I had made on Day 1, and I knew this time he would not be so forgiving. Arrrgh.

“So you work for Lucent then?” he asked.

This question caught me totally off guard. Yes, I did – but given that Lucent was not exactly Cisco’s best buddy (we couldn’t even attend Cisco events per Cisco’s anti-competition policy), I had booked the lab using my home address and paid on a personal credit card. How did he know?! He had apparently looked me up on CCO – I don’t know whether he found me, or if I had to sign in at some point, but either way I was busted. “Yes, I do,” I replied, wondering if that was the right answer. “Interesting. Do you enjoy it?” he inquired. Where the heck is this going? “Sure – I still get to work on Cisco kit every day, so I’m happy enough!” I responded.

Day 2 – The Result

The Proctor – who had been facing me – turned away for a moment and reached for a baseball cap. “Here you are,” he said. Ok, well a guess a baseball cap is a nice souvenir of my experience here, and while it wouldn’t make up for failing, at least I wasn’t going home empty handed. I took the cap and held it on my lap. He then grabbed a CD in a gatefold cardboard case – a CCO documentation CD – and handed it to me. “This may be useful too,” he said. Oh great. Now we’ve moved on from gifts to sarcasm. Let me guess – this is just the documentation for Appletalk configuration, right? I really didn’t get where this was going. I stared blankly at the CD, almost in trance-like in my dazed state.

And then my eyes focused and I spotted it. Hand written at the top of the CD case – “#6727“. At this point – I know… I’m very slow – I realized that it was not a CCO documentation CD, but was in fact the CCIE Logo Usage Program. I looked again at the baseball cap which I had thoughtlessly shoved in my lap – on the front, a CCIE logo. I was slow to pick up on it, but I had passed!

“Congratulations!” he said. When I had recovered from the shock, I asked about my final score. 83 out of 100. Scrape? Yes. Pass? Hell yes. I will take 83% in a heartbeat! That meant I had scored 24 out of 25 on Troubleshooting, and I knew exactly where I had lost the single, stupid, point.

I put my on my new baseball cap proudly, grabbed my possessions, and strutted back to the hotel, Bart Simpson style:

I have to believe that CCIE candidates walking between the test center and the nearest hotel are obvious to the local workers. They come in two flavors – I’m guessing that the candidates who do not pass return to the hotel quietly, feeling quite down. I’m pretty sure I would have had my head down and I would be muttering to myself the whole way about what went wrong. The ones who passed though are skipping along and occasionally shouting “YEAAAAH!” to nobody in particular and punching the air. Or maybe that was just me.

I was still having trouble believing that I had passed first time. The stat I had heard being thrown around for the Brussels CCIE lab was that there was an 85% failure rate for first attempt, and I had also heard more recently it had gone up to 95%. Either way, I was absolutely thrilled (and shocked) with the result, and I made up for the lack of alcohol the previous night and celebrated my new certification with a nice dinner at the hotel. I definitely slept well that night!

 

In Part 5 (The Aftermath), I’ll look at some of the lessons I learned, some comments on what it means to have passed the CCIE lab and what it would have meant if I had failed, and I’ll share a few tips that seem to work both in and out of the lab.