Infuriating Inconsistent Interfaces; F5 on the stand.

F5 Logo

Ok, it’s another f5 post and if you’re not using f5 you might think this is irrelevant to you. However, I beg you to read on because the issue I’m describing today has a relationship to SDN and network automation, and why they are such a pain to do in so many cases.

f5 SSL Profiles

The day began simply enough: news had broken about the “Poodle” SSLv3 vulnerability, and like the majority of network and server nerds we needed to disable or block SSLv3 as quickly as possible in order to remove that particular attack vector. My job was to look at the f5 load balancers, and to do so I realized that I needed to understand what SSL we had out there, and I’d also need to determine the exact change I would be making.

I wrote a couple of scripts to analyze our f5 configurations, and soon enough I had a spreadsheet showing all the SSL client profiles that were in use on each load balancer. It’s important, at this point, to understand how the f5 configures SSL profiles. Fundamentally, a custom profile inherits all of its settings from a “parent” profile, unless you specifically choose to override it. In fact, apart from the name of the profile, everything else is inherited and thus grayed out:

f5 New SSL ProfileIn this lab device example, the profile is inheriting its SSL ciphers (“DEFAULT”) from the parent profile called clientssl, a global template that is installed with the software. We can see this in the configuration file too:

The only configuration of note on this profile so far is a reference to the clientssl parent profile. In order to change the ciphers, for example, it’s necessary to click on the checkbox on the right hand side, then make the desired changes, for example:

f5 No-SSLv3-ProfileOnce this has been submitted, the underlying configuration reveals the change:

The profile now contains a ‘ciphers’ value, absence of which was being interpreted as “go get it from the parent profile”.

Of course, the whole point of using parent profiles is precisely that you don’t have to edit every individual client ssl profile; any profiles that use clientssl as a parent profile and that have not overridden the ciphers will automatically inherit any changes we make to clientssl. So rather than adding !SSLv3 to this profile, let’s leave it inheriting its ciphers from clientssl and change that instead. Unchecking the checkbox in the Custom column of the new profile returns it to inheriting from its parent, and our configuration once more reflects the lack of ciphers:

Changing the ciphers on the clientssl profile now implicitly changes the ciphers on no_sslv3_profile as well.

The Problem

The clientssl profile starts off with ciphers set to DEFAULT, and most profiles are configured to use clientssl as their parent. However, many of the profiles I looked at had a Custom cipher configured, and that cipher is “DEFAULT”. Correct; we have set the exact same ciphers on the profiles using a Custom setting rather than just allowing them to be inherited from the identical underlying parent profile. For these profiles, changing clientssl will make no different whatsoever, and they will continue to use DEFAULT ciphers.

The Plan

The plan was that rather than individually editing every profile to add “:!SSLv3” we would instead find all the profiles with a customer cipher set to DEFAULT and a parent profile of clientssl, and first unset the Custom profile. Then when we changed clientssl all the profiles would change at the same time. This would also make future changes much simpler, so it was a win-win.

To change hundreds of profiles, using the web interface just isn’t appropriate, so I decided to generate the changes in the form of a script that I could then execute using the f5 CLI (well, tmsh).

The Problem

Starting with our client profile “no_sslc3_profile” which had a custom cipher of “DEFAULT”, this is what happens when I tried to remove the custom cipher:

Wait, what? This is not a readonly property from the web interface! Maybe I can find another way around this:

Well, that’s a step in the right direction; the ciphers are now set to “none”, but now I’m worried that this means that no ciphers will be allowed, in which case my Virtual Servers may not be taking traffic! Good news, my connections are still succeeding. Bad news, they are also succeeding using SSLv3, even though my custom ciphers say “none” and my parent profile says “DEFAULT:!SSLv3” now.

Isn’t that just dandy? You cannot remove the custom setting using tmsh, but you can using the web interface. So do I go through and manually remove the custom checkbox from hundreds of profiles? Absolutely not. I will write the tmsh lines to modify all the existing cipher entries instead. How ridiculous though. Why would there be a function in the WebUI that isn’t in the CLI?

Back to SDN and Network Automation

That’s exactly the kind of problem I’ve faced before with (for example) Cisco NXOS XML RPC calls – commands in the CLI that have not been exposed via the NETCONF API. The same is true of A10 load balancers (though they promise that won’t happen again from version 4.0 onwards). How can we write tools to automate the network when we can’t do everything we need from the interface we choose to use? We will, I’m sure, never see a unified configuration language for network equipment because there are too many vendor-specific tweaks and features. However, it would be nice if we could consider standardizing on one management protocol at some point, so I’m not bouncing between JSON and XML constantly, if we could agree on at least a very basic “Here’s what I am” functionality (see my previous post on NETCONF), and let’s agree that a lack of functional equivalency between the various interfaces and APIs within a single product should be considered a P0 bug that needs to be fixed before release. I hate that updating APIs seems to be an optional extra task for many vendors, even today.

Vendors, you know who you are. Pretty please? And a special rap on the knuckles for f5 because they made my job harder. #firstworldproblems

 

30 Blogs in 30 Days

This post is part of my participation in Etherealmind’s 30 Blogs in 30 Days challenge.

6 Comments on Infuriating Inconsistent Interfaces; F5 on the stand.

  1. A real pain this one and not something I’ve ever noticed. I agree on your thoughts around this but I decided to test my new found ‘bash-foo’ and find a solution (that others might find useful). Seems it was pretty easy, just do this in bash when in the /config directory (searches for string: ciphers DEFAULT at the end of any line and deletes the whole line);

    sed -i ‘/ciphers DEFAULT$/d’ bigip.conf #skip to -i option to just test

    The verify the changes manually and also using tmsh;

    tmsh load sys config verify

    If you are happy, load the new config from disk;

    tm load sys config

  2. Hi John… I am very interested in knowing a) the BIG-IP version number of what you’re running, as it’ll help me find this bug in my hardware too and b) the automation you wrote to run the modify and delete commands, if you can share it with me without revealing your internal automation details…

    Would help me a lot in automating this task… 🙂

    • All v11 BIGIP boxes. I can look up the specific code rev if that helps.

      Automation was just tmsh – a /bin/sh script with alternating lines of setting values then getting the values back to check that “!SSLv3” was now present so I could easily identify which lines, if any, failed to take.

      • John,

        I just ran this on the 11.5.1 box I have and I see you’re right, one can’t delete the ciphers. But from what I see, there’s two things –

        1. You can still modify it. So in your case, what you really needed to do was ‘modify ltm profile client-ssl no_sslv3_profile ciphers “DEFAULT:!SSLv3″‘ and not ‘delete ltm profile client-ssl no_sslv3_profile ciphers’. There’s your missing automation step.

        2. You’re absolutely right about that one can’t delete a cipher. Maybe the error message should have been different though…

        • Hi Nitin. The point of deleting the ciphers was to force the profile to inherit its settings from the parent profile (clientssl) rather than have to continue to manually update each profile individually next time there’s a change. I agree that I can modify, but that’s not equivalent to deleting the ciphers.

          Thanks!

  3. Just so you are aware, F5 dropped syncronisation of the clientssl profile between cluster units in 11.2, I think it has since returned around 11.5 or maybe 11.6.

    Best to make your own custom client-ssl profile and use this to ensure it syncs correctly.

    FYI my own frustration today, 11.6 can create ECRSA private keys, awesome!

    However you can’t use these in your client profile, gui gives an error saying it needs a key which it has.

    Not until you build the profile via TMSH does it complain it needs a RSA key.

Leave a Reply

Your email address will not be published.


*


 

This site uses Akismet to reduce spam. Learn how your comment data is processed.