Troubleshooting SSL certificates and connections? Here are five handy openssl commands that every network engineer should be able to use. Bookmark this – you never know when it will come in handy!
1. Check the Connection
openssl s_client -showcerts -connect www.microsoft.com:443
This command opens an SSL connection to the specified site and displays the entire certificate chain as well. Here’s an abridged version of the sample output:
MBP$ openssl s_client -showcerts -connect www.microsoft.com:443 CONNECTED(00000003) depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5 verify error:num=20:unable to get local issuer certificate verify return:0 --- Certificate chain 0 s:/22.214.171.124.4.1.3126.96.36.199.3=US/188.8.131.52.4.1.3184.108.40.206.2= Washington/businessCategory=Private Organization/ serialNumber=600413485/C=US/postalCode=98052/ST=Washington/ L=Redmond/street=1 Microsoft Way/O=Microsoft Corporation/ OU=MSCOM/CN=www.microsoft.com i:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/ CN=Symantec Class 3 EV SSL CA - G3 -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE----- 1 s:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/ CN=Symantec Class 3 EV SSL CA - G3 i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5 -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE----- 2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5 i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE----- --- Server certificate subject=/220.127.116.11.4.1.318.104.22.168.3=US/22.214.171.124.4.1.3126.96.36.199.2= Washington/businessCategory=Private Organization/ serialNumber=600413485/C=US/postalCode=98052/ST=Washington/ L=Redmond/street=1 Microsoft Way/O=Microsoft Corporation/OU=MSCOM /CN=www.microsoft.com issuer=/C=US/O=Symantec Corporation/OU=Symantec Trust Network/ CN=Symantec Class 3 EV SSL CA - G3 --- No client certificate CA names sent --- SSL handshake has read 4170 bytes and written 456 bytes --- New, TLSv1/SSLv3, Cipher is AES256-SHA Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1 Cipher : AES256-SHA Session-ID: BF66575AD2B1E71E74796085FBB60BE1A472818DF0 0BC27378B26AEF9E924F52 Session-ID-ctx: Master-Key: 88A9E8281A5B66DC2E21D4BAA6B85CB0C6F64522FFDFE1AD 1DAC8BF3186CDBD6824EDD669C9B4CD5EDCA42AC6361FDB4 Key-Arg : None Start Time: 1425836408 Timeout : 300 (sec) Verify return code: 0 (ok) ---
I’ve reformatted the lines to make things a little clearer, and be grateful that I removed the blocks of Base64-encoded text that represent the certificates or it would be even more painful to read. There are a couple of things to note, however.
I Only Want to See the Server Certificate
Fine then; remove the
-showcerts argument, and your wish will be fulfilled.
error:num=20:unable to get local issuer certificate
depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5 verify error:num=20:unable to get local issuer certificate
There’s an error here. Depth 2 means which certificate in the chain; in this case the third one as they are numbered 0, 1 and 2, and this error means that openssl was unable to find a certificate for the issuer of certificate 2 whose Common Name(CN) is “VeriSign Class 3 Public Primary Certification Authority – G5”. That’s because the issuer is a root certificate and openssl does not know where the root certificates are. This can be fixed by adding the -CAfile option pointing to a file containing all the trusted root certificates, but where to get those? That’s coming soon in another post. For now what we need to know is that we have three certificates in a chain and at least up to certificate 2, things are verifying correctly.
Certificate Subject and Issuer
Each certificate is presented as a Subject and an Issuer. The Subject is the thing the certificate is supposed to represent, and the Issuer is the issuing Certificate Authority. For example here’s certificate 0 (the server certificate) from this chain:
0 s:/188.8.131.52.4.1.3184.108.40.206.3=US/220.127.116.11.4.1.318.104.22.168.2= Washington/businessCategory=Private Organization/serialNumber= 600413485/C=US/postalCode=98052/ST=Washington/L=Redmond/ street=1 Microsoft Way/O=Microsoft Corporation/OU=MSCOM /CN=www.microsoft.com i:/C=US/O=Symantec Corporation/OU=Symantec Trust Network /CN=Symantec Class 3 EV SSL CA - G3
You can see the certificate number (zero) then s: meaning Subject: and i: meaning Issuer. It follows then that the Issuer of certificate 0 should be the Subject of certificate 1, as we want to verify if the Issuer is valid; and so it is:
1 s:/C=US/O=Symantec Corporation/OU=Symantec Trust Network /CN=Symantec Class 3 EV SSL CA - G3 i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
This of course continues up the chain.
error:num=21:unable to verify the first certificate
If you see this when you run this command, it means exactly what it says … that chain of trust is broken right from the start. Typically it might happen if you fail to include intermediate certificates, or if you supply the wrong intermediate certificate.
This Opens a Connection
Really. It might look like the openssl command has hung, but actually it did exactly what we asked it to and opened a connection. It’s waiting for you to send something now. To quit, either Ctrl-C, or hit Enter a couple of times or – if you’re testing for a response – try typing some basic HTTP commands, e.g.:
[...] Start Time: 1425837372 Timeout : 300 (sec) Verify return code: 0 (ok) --- GET / HTTP/1.1 <<< Typed in Host: www.microsoft.com <<< Typed in <<< Typed in HTTP/1.1 200 OK [...delivers response...]
You’ll still have to break out after that using Ctrl-C, but meanwhile, what fun! Don’t forget that for most sites (particularly HTTP but usually HTTPS as well) you have to use the Host: directive so that the web server knows which site you were trying to contact. When you think about it, most hosting companies have tens or hundreds of web sites served by a single server and IP. Supplying a Host: is essential.
2. Decoding a Base64 Certificate (e.g. PEM)
The output from the previous command will display the raw certificate data between the “—–BEGIN CERTIFICATE—–” and “—–END CERTIFICATE—–” tags. I removed it from the output above so that I could hit you with one now as an example:
-----BEGIN CERTIFICATE----- MIIFmjCCBIKgAwIBAgIKNfMBNgABAAB+LzANBgkqhkiG9w0BAQUFADCBgDETMBEG CgmSJomT8ixkARkWA2NvbTEZMBcGCgmSJomT8ixkARkWCW1pY3Jvc29mdDEUMBIG CgmSJomT8ixkARkWBGNvcnAxFzAVBgoJkiaJk/IsZAEZFgdyZWRtb25kMR8wHQYD VQQDExZNU0lUIE1hY2hpbmUgQXV0aCBDQSAyMB4XDTEzMDYyMDIwMjkyOFoXDTE1 MDYyMDIwMjkyOFowGDEWMBQGA1UEAxMNbWljcm9zb2Z0LmNvbTCCASIwDQYJKoZI hvcNAQEBBQADggEPADCCAQoCggEBANV/NeoVpoco0OnLeGxUEIoXKRNj6T/r8QGa NvKRVWKR/msN8mPeWstdzKu3c5e44HnSGw74F+pDilvNxURIAVT15Plfs717+2M7 6eCWL0dvg+epNoDxx6ncMZ0U5+yPvv8rSyPldIBq4KACgSLZF4EvOBUmn/JGUwzw wHc9MI9lbvBoYoMdOm3ugIgSQJojxi5HMu0VjKbRfmnxlWuDJKcxsBc5qrWG322v mloroq94NAodqxA0mrB2Ktozm8tGvlm3C3nR9F7x53892dl2KbhiiQmtIxsvN/iK hLJfuUA+wdfE5MbdB2XF/FhACA/ELO02u225eOJ67WKN1jTs22ECAwEAAaOCAnsw ggJ3MB0GA1UdDgQWBBSJKGUqqRL75TOfAHBohT5I3Wq4yTALBgNVHQ8EBAMCBLAw HwYDVR0jBBgwFoAU69sRXvgJntjWYpz9Yp3jhEoo4Scwge4GA1UdHwSB5jCB4zCB 4KCB3aCB2oZPaHR0cDovL21zY3JsLm1pY3Jvc29mdC5jb20vcGtpL21zY29ycC9j cmwvTVNJVCUyME1hY2hpbmUlMjBBdXRoJTIwQ0ElMjAyKDEpLmNybIZNaHR0cDov L2NybC5taWNyb3NvZnQuY29tL3BraS9tc2NvcnAvY3JsL01TSVQlMjBNYWNoaW5l JTIwQXV0aCUyMENBJTIwMigxKS5jcmyGOGh0dHA6Ly9jb3JwcGtpL2NybC9NU0lU JTIwTWFjaGluZSUyMEF1dGglMjBDQSUyMDIoMSkuY3JsMIGtBggrBgEFBQcBAQSB oDCBnTBVBggrBgEFBQcwAoZJaHR0cDovL3d3dy5taWNyb3NvZnQuY29tL3BraS9t c2NvcnAvTVNJVCUyME1hY2hpbmUlMjBBdXRoJTIwQ0ElMjAyKDEpLmNydDBEBggr BgEFBQcwAoY4aHR0cDovL2NvcnBwa2kvYWlhL01TSVQlMjBNYWNoaW5lJTIwQXV0 aCUyMENBJTIwMigxKS5jcnQwPwYJKwYBBAGCNxUHBDIwMAYoKwYBBAGCNxUIg8+J Ta3yAoWhnwyC+sp9geH7dIFPg8LthQiOqdKFYwIBZAIBCjAdBgNVHSUEFjAUBggr BgEFBQcDAgYIKwYBBQUHAwEwJwYJKwYBBAGCNxUKBBowGDAKBggrBgEFBQcDAjAK BggrBgEFBQcDATANBgkqhkiG9w0BAQUFAAOCAQEAtv7pra/09qguqOGjZvsSKHrH UX/BORRqq8nFGYaE3qMe9NY3l6LVc3+ei8K+sR8ni2XtsuvSMq+7k6fawXITKBd+ ptfoAMw4w7QrmkUFBPb4e+CcN0TOmnqj/eczuw/zo8WmD6OHM3QyLot3iR16fFAb TG6oYc1w3lQZzc94Fkun0FA4gEEvi9u1XzNbjqO2tU9FxJjuCni6CwPw7maOShWY DObGVa3ZebPBq+R60eUfNicgEGgX4O2VstrOpPYYIV5ZiJDqljNh9MI2NKvOp1dl 0fN7gzHLr++OCrh4xTBzYHHCepxRcBB8MIVA8f16l/ShhEbUNjHG0lVPFOqzfg== -----END CERTIFICATE-----
Certificates can be in a variety of formats (yay for standardization), but the output from OpenSSL (like above) will be Base64 encoded and basically unreadable. Thankfully, the openssl command can help you view those in a format that is human readable and formatted nicely. Take the Base64 text (including the BEGIN and END lines) of the certificate you are interested in, and save it to a file. Then run this command (in my case with a file called
openssl x509 -noout -text -in cert-microsoft.pem
This tells openssl to read the file
cert-microsoft.pem , display it in a textual format, and not to create any kind of output certificate. The result is exactly what you asked for:
MBP$ openssl x509 -noout -text -in cert-microsoft.pem Certificate: Data: Version: 3 (0x2) Serial Number: 35:f3:01:36:00:01:00:00:7e:2f Signature Algorithm: sha1WithRSAEncryption Issuer: DC=com, DC=microsoft, DC=corp, DC=redmond, CN=MSIT Machine Auth CA 2 Validity Not Before: Jun 20 20:29:28 2013 GMT Not After : Jun 20 20:29:28 2015 GMT Subject: CN=microsoft.com Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public Key: (2048 bit) Modulus (2048 bit): [removed for brevity] Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: 89:28:65:2A:A9:12:FB:E5:33:9F:00:70:68:85:3E: 48:DD:6A:B8:C9 X509v3 Key Usage: Digital Signature, Key Encipherment, Data Encipherment X509v3 Authority Key Identifier: keyid:EB:DB:11:5E:F8:09:9E:D8:D6:62:9C:FD: 62:9D:E3:84:4A:28:E1:27 X509v3 CRL Distribution Points: URI:http://mscrl.microsoft.com/pki/mscorp/crl/ MSIT%20Machine%20Auth%20CA%202(1).crl URI:http://crl.microsoft.com/pki/mscorp/crl/ MSIT%20Machine%20Auth%20CA%202(1).crl URI:http://corppki/crl/MSIT%20Machine%20Auth%20 CA%202(1).crl Authority Information Access: CA Issuers - URI:http://www.microsoft.com/pki/ mscorp/MSIT%20Machine%20Auth%20CA%202(1).crt CA Issuers - URI:http://corppki/aia/MSIT%20Machine %20Auth%20CA%202(1).crt 22.214.171.124.4.1.311.21.7: 00.(+.....7.....M..........}...t.O.........c..d.. X509v3 Extended Key Usage: TLS Web Client Authentication, TLS Web Server Authentication 126.96.36.199.4.1.311.21.10: 0.0 ..+.......0 ..+....... Signature Algorithm: sha1WithRSAEncryption [removed for brevity]
And, since we have the other certificates in the chain, I could check them too.
Does This Work With Binary Certificate Files?
DER is a binary certificate format, but the content is basically the same underneath it all. When discussing the AIA field in a previous post, I casually skipped over the fact that this file in my experience seems to be supplied in DER format rather than PEM (I don’t know if this is a standard, just that it’s what I’ve seen. Maybe it’s to keep the transfer shorter and thus faster?). All openssl asks is that you tell if you want to supply it with a DER instead of a PEM (Base64) certificate. Personally I would have thought that the absence of “—–BEGIN CERTIFICATE” was sufficient clue for openssl to make an educated guess, but apparently that’s not the case. Instead, you have to use the command line option
-inform der. For example, to view a binary certificate as text you’d do this:
openssl x509 -noout -text -inform der -in cert_symantec.der
By the way,
-inform is short for “input format”; you’re not really “informing” openssl about anything. If you were wondering, yes, there is an
-outform command as well, and on that note:
3. Convert Certificate From DER to PEM Format
In the examples above, we asked openssl not to create an output certificate using the
-nout command line argument. However, openssl is very helpful at converting certificates between formats, so let’s try converting DER to PEM:
openssl x509 -inform der -in cert_symantec.der -out cert_symantec.pem
This command specifies that the input format is DER, the input file is
cert_symantec.der and that we want an output format of PEM saved to an output file called
cert_symantec.pem. The observant will have noted that the command actually did not specify the output format of PEM. PEM is the default input and output format, so it does not need to be specified. However, if you like to remove ambiguity in a totally harmless and logical fashion, the full command would be:
openssl x509 -inform der -in cert_symantec.der -outform pem -out cert_symantec.pem
Easy peasy. The added benefit of understanding how to do this is that you now don’t have to use somebody else’s website to convert you internal certificates between formats.
4. Checking Your Own Chain of Trust
You’re ready to deploy a certificate for a website, and you have been given a ZIP file containing the public server cert and a file purporting to contain the necessary intermediate certificate(s). How can you check that you have the correct certificates without actually installing them? Why, openssl, of course!
MBP$ openssl verify -verbose cert-www-microsoft.pem cert-www-microsoft.pem: /188.8.131.52.4.1.3184.108.40.206.3=US/ 220.127.116.11.4.1.318.104.22.168.2=Washington/businessCategory=Private Organization/serialNumber=600413485/C=US/postalCode=98052/ ST=Washington/L=Redmond/street=1 Microsoft Way/O=Microsoft Corporation/OU=MSCOM/CN=www.microsoft.com error 20 at 0 depth lookup:unable to get local issuer certificate
What happened here? Error 20 was mentioned above; it means that the intermediate certificate (or at least, the certificate for the Issuer of the server certificate) is missing. Well of course it is; we didn’t supply it! The www.microsoft.com site uses a certificate from Symantec, so let’s use that and tell openssl about it:
MBP$ openssl verify -untrusted cert-symantec cert-www-microsoft.pem cert-www-microsoft.pem: /C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 EV SSL CA - G3 error 20 at 1 depth lookup:unable to get local issuer certificate OK
That’s progress; we still have an error 20, but now it has moved to “1 depth” – that is, the error is no longer on the server certificate (0 depth) but now we can’t find the issuer certificate for the Symantec cert. In a previous post, we discovered that the Symantec cert was issued by a Verisign entity that is in our trusted root store. So now I’ll add a link to the root store as well to complete the chain:
MBP$ openssl verify -untrusted cert-symantec -CAfile ./RootCerts.pem cert-www-microsoft.pem cert-www-microsoft.pem: OK
That’s it! We have confirmed that we have a full chain of trust from a trusted root cert all the way down to the www.microsoft.com server certificate. Even for a Mac user, this is a good thing.
What About Multiple Intermediate Certificates?
If you have more than a single Intermediate Certificate between the server and a trusted root certificate, you need to make them all available to the client. That’s easily done by creating a certificate bundle, which is a fancy way of saying “add all the certificates together in a single file.” Really. If you have two files each containing an intemediate certificate and need to bundle them, in *nix / OS X you do this:
$ cat intermediate1.pem intermediate2.pem > intermediatebundle.pem
It’s that easy. In any GUI environment you can just paste them one after another in Notepad and save them out. Remember to include the BEGIN and END lines. Now in your command line just change the argument to
-untrusted intermediatebundle.pem and you’re good.
5. Testing for SSLv3 Using OpenSSL
This one is pretty easy. Using the
s_client function again, we can ask openssl to try to connect using SSLv3. A site that supports SSLv3 (naughty naughty) will look like this:
MBP$ openssl s_client -ssl3 -connect microsoft.com:443 CONNECTED(00000003) [...certificate stuff removed for brevity...] SSL-Session: Protocol : SSLv3 Cipher : RC4-SHA Session-ID: 33410000536... Session-ID-ctx: Master-Key: F88FCD7DF64CFB48... Key-Arg : None Start Time: 1425840399 Timeout : 7200 (sec) Verify return code: 0 (ok) ---
A site that does NOT support SSLv3 will look like this:
MBP$ openssl s_client -showcerts -ssl3 -connect www.microsoft.com:443 CONNECTED(00000003) 5688:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:/SourceCache/OpenSSL098/OpenSSL098-52.10.1/ src/ssl/s3_pkt.c:1143:SSL alert number 40 5688:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:/SourceCache/OpenSSL098/OpenSSL098-52.10.1/src/ssl/ s3_pkt.c:564: MBP$
There are other error codes you may see, but generally speaking if you get dumped back to a prompt with an error, chances are this means that SSLv3 is not allowed.
I thought it was interesting, by the way, that
microsoft.com allowed an SSLv3 connection, but
www.microsoft.com did not. The former uses a different certificate chain and redirects to the latter, so perhaps it all comes out in the wash. It’s actually a missed opportunity in some ways for Microsoft not to detect SSLv3 in some way, then pop up a web page saying “Hello IE6 user – why not upgrade now?”, but they don’t do that.
My 10 Bits
I have had to use all of these commands recently in order to troubleshoot a few different issues that were popping up with some web sites. I confess to being terrible at remembering commands in detail, so I’m going to bookmark my own page for reference even if you don’t! Openssl does plenty more that can be useful, but this is a great start when it comes to certificates and ciphers.
Thank you, that was interesting. Now that free certificates will be available (here: https://letsencrypt.org/) I will try to add https to my sites as well.
Thank you so much for taking the time to write this article. It was invaluable to me in resolving some coding issues with OpenSSL. Using the command line tools was a much easier way of determining and resolving the issues. By the way, I hit another basic problem that I thought was worth mentioning – when passing PEM files to OpenSSL, it is very fussy about the character encoding. Cut & pasting your Microsoft example did not work at first, as my text editor saved (by default) with the wrong character encoding – resaving as ASCII instead of UTF-8 resolved the problem. Just FYI.