You’ve set up your website and secured it with an SSL certificate that you bought through your ISP. Everything works fine and the chain of trust is just fine in your browser, but when you try accessing your secured site using a command line tool, the connection fails. Why? There’s a good chance that you are not sending your intermediate certificate(s) along with the server certificate.

PKI Trust Review

As a quick reminder, the whole point of SSL certificates and the Public Key Infrastructure is to prove that the site you connected to is the one it says it is. How do we know? The server sends you a certificate with its name in it, digitally signed by an Issuer. If you choose to trust that Issuer’s honesty and believe that they made sure they issued to the right site, you implicitly trust that the end site is the right one; it’s a “Chain of Trust.”

In reality, we don’t typically trust many Issuers. Look in the Trusted Root certificates for your browser, or on a Mac, open Keychain Access and look in System Roots, and you’ll see that for Yosemite in this case, globally – to establish SSL trust for the entire Internet – there are only 214 trusted Issuers, with many Issuers owning more than one of those (e.g. Thawte is responsible for 12 of those 214). For the system to be scalable, many other companies will issue the actual server certificates; but since these Issuers are not on my Trusted list, why would I trust any server certificate that they sign? The answer is that they in turn have their certificate signed by a Trusted Root. Here’s a quick analogy of this logic?

In reality there can be multiple intermediate signing authorities in between the server certificate and a trusted root, but having two or three intermediates is probably most common. From the client’s perspective, the problem is that Chris knows a lot of people and is meeting more every day, so I don’t have all their certificates already. In an ideal world, the server (Alyssa in the example above) would send me Bob’s certificate too, and possibly even Chris’ certificate (even though I already have that). Alyssa couldn’t fake any of the information, because the signatures cover all the information and if any of the information is changed, the signature won’t look right so I’ll know that chain of trust has been broken.

The Case of the Missing Intermediate Cert

So what happens when Alyssa sends her certificate but does not send Bob’s certificate? I don’t have Bob’s certificate, but somehow my browser still says the site is trusted. How is that?

The answer turns out to be buried within Alyssa’s certificate. If I decode the certificate, near the bottom I might see something like this:

        Authority Information Access:
            CA Issuers - URI:http://www.bob.com/pkicerts/bob.crt

Alyssa can (optionally) give you the pointer to Bob’s certificate without actually having to send it. Browsers are smart enough (and kind enough) to follow that link and grab Bob’s certificate. In a production environment this is probably not what you want; you are slowing down the SSL negotiation process by forcing the client browser to go to another site (bob.com in this case) and get the certificate. It’s much more efficient to send it along with the server certificate. Still, on the off chance that you screwed up your certificates, it’s a good fallback position. At least, it would be, but command line tools might not choose to use the Authority Information Access (AIA) field when verifying the certificates. In that instance, the command line tool will fail to verify that the site is trusted, and your connection will fail.

An Example

For fun, let’s look at www.microsoft.com and see that chain of trust in action. I should comment that https://microsoft.com/ uses a different chain of trust (interesting!), so if you play along at home, make sure you add the ‘www’ bit. Safari shows the certificate chain here:

We have www.microsoft.com signed by Symantec, which is signed by the Verisign Class 3 CA, which is then signed by the Verisign root CA. The Verisign root has nobody above it to sign it, so it signs its own certificate. Effectively PKI ends with a human saying “I trust this issuer to do the right thing”, which makes the security of Trusted Root certificates incredibly important.

In reality, things get a little bit more complex than they seem because although you’d think we should trust the Verisign root CA at the top (Class 3 Public Primary Certification Authority), in fact what we see is that the Issuing CA (Verisign Class 3 Public Primary Certification Authority – G5) is in our trusted root store:

The browser can shortcut the full chain of trust and instead anything from the Verisign Class 3 CA downward can be trusted, so effectively – in theory at least – it doesn’t need to check the signature of the true Verisign root. This also means that the main Verisign root does not have to see the light of day to sign certificates al the time. It’s brought out to pretty much only to sign an intermediate CA’s certificate, then locked up again which means that it can be kept offline and thus (hopefully) secure. Security is taken seriously, and you can see some of that here in a Key Signing Ceremony:

Back to www.microsoft.com. If I just look at the www.microsoft.com root certificate, we can see the Issuer is Symantec, and there’s an AIA field too:

        Issuer: C=US, O=Symantec Corporation, OU=Symantec Trust Network, CN=Symantec Class 3 EV SSL CA - G3
        Authority Information Access:
            OCSP - URI:http://sr.symcd.com
            CA Issuers - URI:http://sr.symcb.com/sr.crt

If I then go to the “CA Issuers” URI, I get the Symantec intermediate certificate. This one does not have an AIA (remember I said it’s optional):

Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc. - For authorized use only, CN=VeriSign Class 3 Public Primary Certification Authority - G5

My computer actually already has this certificate and trusts it. As discussed above though, it’s really an intermediate certificate, so to be certain, www.microsoft.com very kindly supplies the server cert, the Symantec G3 cert and the Verisign C3 G5 cert when you connect, so there’s actually no need to use the AIA fields in this case. However, should that first cert be missing, you’re in luck as long as your users are browser-based and the browser is smart enough to work around the missing certificate. On the other hand, better just to get it right in the first place, eh?

My 10 Bits

As you can imagine, I found out the hard way what happens when the wrong (or incomplete) intermediate certificates are configured on a server or, more commonly perhaps, on a load balancer. In the past, my typical test after a certificate installation for an SSL VIP would be to spin up a web browser and open the site, and if successful I’d move on. However, having discovered a VIP on a load balancer where somebody had associated the wrong intermediate certificate, I think I’ll also use openssl to test from now on. If the intermediate certificate is missing or incorrect, it’s quite vocal about the problems its seeing. A command like this does the trick:

openssl s_client -showcerts -CAfile RootCerts.pem  -connect www.microsoft.com:443

There’s always more we can do to test our installations, and this is one case where too much intelligence in the usual test tool (a web browser) helped to disguise a configuration error. Lesson learned.

Postscript

Why did I have to specify a file containing RootCerts in the openssl command above, and where do they come from? I’ll explain in a short future post.