Ok, it’s another f5 post and if you’re not using f5 you might think this is irrelevant to you. However, I beg you to read on because the issue I’m describing today has a relationship to SDN and network automation, and why they are such a pain to do in so many cases.

f5 SSL Profiles

The day began simply enough: news had broken about the “Poodle” SSLv3 vulnerability, and like the majority of network and server nerds we needed to disable or block SSLv3 as quickly as possible in order to remove that particular attack vector. My job was to look at the f5 load balancers, and to do so I realized that I needed to understand what SSL we had out there, and I’d also need to determine the exact change I would be making.

I wrote a couple of scripts to analyze our f5 configurations, and soon enough I had a spreadsheet showing all the SSL client profiles that were in use on each load balancer. It’s important, at this point, to understand how the f5 configures SSL profiles. Fundamentally, a custom profile inherits all of its settings from a “parent” profile, unless you specifically choose to override it. In fact, apart from the name of the profile, everything else is inherited and thus grayed out:

In this lab device example, the profile is inheriting its SSL ciphers (“DEFAULT”) from the parent profile called clientssl, a global template that is installed with the software. We can see this in the configuration file too:

ltm profile client-ssl no_sslv3_profile {
    app-service none
    defaults-from clientssl
}

The only configuration of note on this profile so far is a reference to the clientssl parent profile. In order to change the ciphers, for example, it’s necessary to click on the checkbox on the right hand side, then make the desired changes, for example:

Once this has been submitted, the underlying configuration reveals the change:

ltm profile client-ssl no_sslv3_profile {
    app-service none
    ciphers DEFAULT:!SSLv3
    defaults-from clientssl
}

The profile now contains a ‘ciphers’ value, absence of which was being interpreted as “go get it from the parent profile”.

Of course, the whole point of using parent profiles is precisely that you don’t have to edit every individual client ssl profile; any profiles that use clientssl as a parent profile and that have not overridden the ciphers will automatically inherit any changes we make to clientssl. So rather than adding !SSLv3 to this profile, let’s leave it inheriting its ciphers from clientssl and change that instead. Unchecking the checkbox in the Custom column of the new profile returns it to inheriting from its parent, and our configuration once more reflects the lack of ciphers:

ltm profile client-ssl no_sslv3_profile {
    app-service none
    defaults-from clientssl
}

Changing the ciphers on the clientssl profile now implicitly changes the ciphers on no_sslv3_profile as well.

The Problem

The clientssl profile starts off with ciphers set to DEFAULT, and most profiles are configured to use clientssl as their parent. However, many of the profiles I looked at had a Custom cipher configured, and that cipher is “DEFAULT”. Correct; we have set the exact same ciphers on the profiles using a Custom setting rather than just allowing them to be inherited from the identical underlying parent profile. For these profiles, changing clientssl will make no different whatsoever, and they will continue to use DEFAULT ciphers.

The Plan

The plan was that rather than individually editing every profile to add “:!SSLv3” we would instead find all the profiles with a customer cipher set to DEFAULT and a parent profile of clientssl, and first unset the Custom profile. Then when we changed clientssl all the profiles would change at the same time. This would also make future changes much simpler, so it was a win-win.

To change hundreds of profiles, using the web interface just isn’t appropriate, so I decided to generate the changes in the form of a script that I could then execute using the f5 CLI (well, tmsh).

The Problem

Starting with our client profile “no_sslc3_profile” which had a custom cipher of “DEFAULT”, this is what happens when I tried to remove the custom cipher:

(tmos)# delete ltm profile client-ssl no_sslv3_profile ciphers DEFAULT
Syntax Error: "ciphers" read-only property

Wait, what? This is not a readonly property from the web interface! Maybe I can find another way around this:

(tmos)# modify ltm profile client-ssl no_sslv3_profile ciphers ""
(tmos)# list ltm profile client-ssl no_sslv3_profile
ltm profile client-ssl no_sslv3_profile {
    app-service none
    ciphers none
    defaults-from clientssl
}

Well, that’s a step in the right direction; the ciphers are now set to “none”, but now I’m worried that this means that no ciphers will be allowed, in which case my Virtual Servers may not be taking traffic! Good news, my connections are still succeeding. Bad news, they are also succeeding using SSLv3, even though my custom ciphers say “none” and my parent profile says “DEFAULT:!SSLv3” now.

Isn’t that just dandy? You cannot remove the custom setting using tmsh, but you can using the web interface. So do I go through and manually remove the custom checkbox from hundreds of profiles? Absolutely not. I will write the tmsh lines to modify all the existing cipher entries instead. How ridiculous though. Why would there be a function in the WebUI that isn’t in the CLI?

Back to SDN and Network Automation

That’s exactly the kind of problem I’ve faced before with (for example) Cisco NXOS XML RPC calls – commands in the CLI that have not been exposed via the NETCONF API. The same is true of A10 load balancers (though they promise that won’t happen again from version 4.0 onwards). How can we write tools to automate the network when we can’t do everything we need from the interface we choose to use? We will, I’m sure, never see a unified configuration language for network equipment because there are too many vendor-specific tweaks and features. However, it would be nice if we could consider standardizing on one management protocol at some point, so I’m not bouncing between JSON and XML constantly, if we could agree on at least a very basic “Here’s what I am” functionality (see my previous post on NETCONF), and let’s agree that a lack of functional equivalency between the various interfaces and APIs within a single product should be considered a P0 bug that needs to be fixed before release. I hate that updating APIs seems to be an optional extra task for many vendors, even today.

Vendors, you know who you are. Pretty please? And a special rap on the knuckles for f5 because they made my job harder. #firstworldproblems

 

30 Blogs in 30 Days

This post is part of my participation in Etherealmind’s 30 Blogs in 30 Days challenge.