Five Essential OpenSSL Troubleshooting Commands

OpenSSL LogoTroubleshooting SSL certificates and connections? Here are five handy openssl commands that every network engineer should be able to use. Bookmark this – you never know when it will come in handy!

1. Check the Connection

 openssl s_client -showcerts -connect www.microsoft.com:443

This command opens an SSL connection to the specified site and displays the entire certificate chain as well. Here’s an abridged version of the sample output:

MBP$ openssl s_client -showcerts -connect www.microsoft.com:443
CONNECTED(00000003)
depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006
 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public
 Primary Certification Authority - G5
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=
     Washington/businessCategory=Private Organization/
     serialNumber=600413485/C=US/postalCode=98052/ST=Washington/
     L=Redmond/street=1 Microsoft Way/O=Microsoft Corporation/
     OU=MSCOM/CN=www.microsoft.com
   i:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/
     CN=Symantec Class 3 EV SSL CA - G3
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
 1 s:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/
     CN=Symantec Class 3 EV SSL CA - G3
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006
     VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3
     Public Primary Certification Authority - G5
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
 2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006
     VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3
     Public Primary Certification Authority - G5
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification
      Authority
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
---
Server certificate
subject=/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=
Washington/businessCategory=Private Organization/
serialNumber=600413485/C=US/postalCode=98052/ST=Washington/
L=Redmond/street=1 Microsoft Way/O=Microsoft Corporation/OU=MSCOM
/CN=www.microsoft.com
issuer=/C=US/O=Symantec Corporation/OU=Symantec Trust Network/
CN=Symantec Class 3 EV SSL CA - G3
---
No client certificate CA names sent
---
SSL handshake has read 4170 bytes and written 456 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: BF66575AD2B1E71E74796085FBB60BE1A472818DF0
      0BC27378B26AEF9E924F52
    Session-ID-ctx:
    Master-Key: 88A9E8281A5B66DC2E21D4BAA6B85CB0C6F64522FFDFE1AD
      1DAC8BF3186CDBD6824EDD669C9B4CD5EDCA42AC6361FDB4
    Key-Arg   : None
    Start Time: 1425836408
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

I’ve reformatted the lines to make things a little clearer, and be grateful that I removed the blocks of Base64-encoded text that represent the certificates or it would be even more painful to read. There are a couple of things to note, however.

I Only Want to See the Server Certificate

Fine then; remove the -showcerts argument, and your wish will be fulfilled.

error:num=20:unable to get local issuer certificate

depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c)
2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3
Public Primary Certification Authority - G5
verify error:num=20:unable to get local issuer certificate

There’s an error here. Depth 2 means which certificate in the chain; in this case the third one as they are numbered 0, 1 and 2, and this error means that openssl was unable to find a certificate for the issuer of certificate 2 whose Common Name(CN) is “VeriSign Class 3 Public Primary Certification Authority – G5”. That’s because the issuer is a root certificate and openssl does not know where the root certificates are. This can be fixed by adding the -CAfile option pointing to a file containing all the trusted root certificates, but where to get those? That’s coming soon in another post. For now what we need to know is that we have three certificates in a chain and at least up to certificate 2, things are verifying correctly.

Certificate Subject and Issuer

Each certificate is presented as a Subject and an Issuer. The Subject is the thing the certificate is supposed to represent, and the Issuer is the issuing Certificate Authority. For example here’s certificate 0 (the server certificate) from this chain:

 0 s:/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=
     Washington/businessCategory=Private Organization/serialNumber=
     600413485/C=US/postalCode=98052/ST=Washington/L=Redmond/
     street=1 Microsoft Way/O=Microsoft Corporation/OU=MSCOM
     /CN=www.microsoft.com
   i:/C=US/O=Symantec Corporation/OU=Symantec Trust Network
     /CN=Symantec Class 3 EV SSL CA - G3

You can see the certificate number (zero) then s: meaning Subject: and i: meaning Issuer. It follows then that the Issuer of certificate 0 should be the Subject of certificate 1, as we want to verify if the Issuer is valid; and so it is:

 1 s:/C=US/O=Symantec Corporation/OU=Symantec Trust Network
     /CN=Symantec Class 3 EV SSL CA - G3
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006
     VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3
     Public Primary Certification Authority - G5

This of course continues up the chain.

error:num=21:unable to verify the first certificate

If you see this when you run this command, it means exactly what it says … that chain of trust is broken right from the start. Typically it might happen if you fail to include intermediate certificates, or if you supply the wrong intermediate certificate.

This Opens a Connection

Really. It might look like the openssl command has hung, but actually it did exactly what we asked it to and opened a connection. It’s waiting for you to send something now. To quit, either Ctrl-C, or hit Enter a couple of times or – if you’re testing for a response – try typing some basic HTTP commands, e.g.:

    [...]
    Start Time: 1425837372
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
GET / HTTP/1.1                        <<< Typed in
Host: www.microsoft.com               <<< Typed in
                                      <<< Typed in
HTTP/1.1 200 OK
[...delivers response...]

You’ll still have to break out after that using Ctrl-C, but meanwhile, what fun! Don’t forget that for most sites (particularly HTTP but usually HTTPS as well) you have to use the Host: directive so that the web server knows which site you were trying to contact. When you think about it, most hosting companies have tens or hundreds of web sites served by a single server and IP. Supplying a Host: is essential.

2. Decoding a Base64 Certificate (e.g. PEM)

The output from the previous command will display the raw certificate data between the “—–BEGIN CERTIFICATE—–” and “—–END CERTIFICATE—–” tags. I removed it from the output above so that I could hit you with one now as an example:

-----BEGIN CERTIFICATE-----
MIIFmjCCBIKgAwIBAgIKNfMBNgABAAB+LzANBgkqhkiG9w0BAQUFADCBgDETMBEG
CgmSJomT8ixkARkWA2NvbTEZMBcGCgmSJomT8ixkARkWCW1pY3Jvc29mdDEUMBIG
CgmSJomT8ixkARkWBGNvcnAxFzAVBgoJkiaJk/IsZAEZFgdyZWRtb25kMR8wHQYD
VQQDExZNU0lUIE1hY2hpbmUgQXV0aCBDQSAyMB4XDTEzMDYyMDIwMjkyOFoXDTE1
MDYyMDIwMjkyOFowGDEWMBQGA1UEAxMNbWljcm9zb2Z0LmNvbTCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBANV/NeoVpoco0OnLeGxUEIoXKRNj6T/r8QGa
NvKRVWKR/msN8mPeWstdzKu3c5e44HnSGw74F+pDilvNxURIAVT15Plfs717+2M7
6eCWL0dvg+epNoDxx6ncMZ0U5+yPvv8rSyPldIBq4KACgSLZF4EvOBUmn/JGUwzw
wHc9MI9lbvBoYoMdOm3ugIgSQJojxi5HMu0VjKbRfmnxlWuDJKcxsBc5qrWG322v
mloroq94NAodqxA0mrB2Ktozm8tGvlm3C3nR9F7x53892dl2KbhiiQmtIxsvN/iK
hLJfuUA+wdfE5MbdB2XF/FhACA/ELO02u225eOJ67WKN1jTs22ECAwEAAaOCAnsw
ggJ3MB0GA1UdDgQWBBSJKGUqqRL75TOfAHBohT5I3Wq4yTALBgNVHQ8EBAMCBLAw
HwYDVR0jBBgwFoAU69sRXvgJntjWYpz9Yp3jhEoo4Scwge4GA1UdHwSB5jCB4zCB
4KCB3aCB2oZPaHR0cDovL21zY3JsLm1pY3Jvc29mdC5jb20vcGtpL21zY29ycC9j
cmwvTVNJVCUyME1hY2hpbmUlMjBBdXRoJTIwQ0ElMjAyKDEpLmNybIZNaHR0cDov
L2NybC5taWNyb3NvZnQuY29tL3BraS9tc2NvcnAvY3JsL01TSVQlMjBNYWNoaW5l
JTIwQXV0aCUyMENBJTIwMigxKS5jcmyGOGh0dHA6Ly9jb3JwcGtpL2NybC9NU0lU
JTIwTWFjaGluZSUyMEF1dGglMjBDQSUyMDIoMSkuY3JsMIGtBggrBgEFBQcBAQSB
oDCBnTBVBggrBgEFBQcwAoZJaHR0cDovL3d3dy5taWNyb3NvZnQuY29tL3BraS9t
c2NvcnAvTVNJVCUyME1hY2hpbmUlMjBBdXRoJTIwQ0ElMjAyKDEpLmNydDBEBggr
BgEFBQcwAoY4aHR0cDovL2NvcnBwa2kvYWlhL01TSVQlMjBNYWNoaW5lJTIwQXV0
aCUyMENBJTIwMigxKS5jcnQwPwYJKwYBBAGCNxUHBDIwMAYoKwYBBAGCNxUIg8+J
Ta3yAoWhnwyC+sp9geH7dIFPg8LthQiOqdKFYwIBZAIBCjAdBgNVHSUEFjAUBggr
BgEFBQcDAgYIKwYBBQUHAwEwJwYJKwYBBAGCNxUKBBowGDAKBggrBgEFBQcDAjAK
BggrBgEFBQcDATANBgkqhkiG9w0BAQUFAAOCAQEAtv7pra/09qguqOGjZvsSKHrH
UX/BORRqq8nFGYaE3qMe9NY3l6LVc3+ei8K+sR8ni2XtsuvSMq+7k6fawXITKBd+
ptfoAMw4w7QrmkUFBPb4e+CcN0TOmnqj/eczuw/zo8WmD6OHM3QyLot3iR16fFAb
TG6oYc1w3lQZzc94Fkun0FA4gEEvi9u1XzNbjqO2tU9FxJjuCni6CwPw7maOShWY
DObGVa3ZebPBq+R60eUfNicgEGgX4O2VstrOpPYYIV5ZiJDqljNh9MI2NKvOp1dl
0fN7gzHLr++OCrh4xTBzYHHCepxRcBB8MIVA8f16l/ShhEbUNjHG0lVPFOqzfg==
-----END CERTIFICATE-----

Certificates can be in a variety of formats (yay for standardization), but the output from OpenSSL (like above) will be Base64 encoded and basically unreadable. Thankfully, the openssl command can help you view those in a format that is human readable and formatted nicely. Take the Base64 text (including the BEGIN and END lines) of the certificate you are interested in, and save it to a file. Then run this command (in my case with a file called cert-microsoft.pem):

openssl x509 -noout -text -in cert-microsoft.pem

This tells openssl to read the file cert-microsoft.pem , display it in a textual format, and not to create any kind of output certificate. The result is exactly what you asked for:

MBP$ openssl x509 -noout -text -in cert-microsoft.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            35:f3:01:36:00:01:00:00:7e:2f
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: DC=com, DC=microsoft, DC=corp, DC=redmond, 
          CN=MSIT Machine Auth CA 2
        Validity
            Not Before: Jun 20 20:29:28 2013 GMT
            Not After : Jun 20 20:29:28 2015 GMT
        Subject: CN=microsoft.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    [removed for brevity]
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                89:28:65:2A:A9:12:FB:E5:33:9F:00:70:68:85:3E:
                48:DD:6A:B8:C9
            X509v3 Key Usage:
                Digital Signature, Key Encipherment, Data
                Encipherment
            X509v3 Authority Key Identifier:
                keyid:EB:DB:11:5E:F8:09:9E:D8:D6:62:9C:FD:
                62:9D:E3:84:4A:28:E1:27

            X509v3 CRL Distribution Points:
                URI:http://mscrl.microsoft.com/pki/mscorp/crl/
                  MSIT%20Machine%20Auth%20CA%202(1).crl
                URI:http://crl.microsoft.com/pki/mscorp/crl/
                  MSIT%20Machine%20Auth%20CA%202(1).crl
                URI:http://corppki/crl/MSIT%20Machine%20Auth%20
                  CA%202(1).crl

            Authority Information Access:
                CA Issuers - URI:http://www.microsoft.com/pki/
                  mscorp/MSIT%20Machine%20Auth%20CA%202(1).crt
                CA Issuers - URI:http://corppki/aia/MSIT%20Machine
                  %20Auth%20CA%202(1).crt

            1.3.6.1.4.1.311.21.7:
                00.(+.....7.....M..........}...t.O.........c..d..

            X509v3 Extended Key Usage:
                TLS Web Client Authentication, TLS Web Server
                Authentication
            1.3.6.1.4.1.311.21.10:
                0.0
..+.......0
..+.......
    Signature Algorithm: sha1WithRSAEncryption
                    [removed for brevity]

And, since we have the other certificates in the chain, I could check them too.

Does This Work With Binary Certificate Files?

DER is a binary certificate format, but the content is basically the same underneath it all. When discussing the AIA field in a previous post, I casually skipped over the fact that this file in my experience seems to be supplied in DER format rather than PEM (I don’t know if this is a standard, just that it’s what I’ve seen. Maybe it’s to keep the transfer shorter and thus faster?). All openssl asks is that you tell if you want to supply it with a DER instead of a PEM (Base64) certificate. Personally I would have thought that the absence of “—–BEGIN CERTIFICATE” was sufficient clue for openssl to make an educated guess, but apparently that’s not the case. Instead, you have to use the command line option -inform der. For example, to view a binary certificate as text you’d do this:

openssl x509 -noout -text -inform der -in cert_symantec.der

By the way, -inform is short for “input format”; you’re not really “informing” openssl about anything. If you were wondering, yes, there is an -outform command as well, and on that note:

3. Convert Certificate From DER to PEM Format

In the examples above, we asked openssl not to create an output certificate using the -nout command line argument. However, openssl is very helpful at converting certificates between formats, so let’s try converting DER to PEM:

openssl x509 -inform der -in cert_symantec.der -out cert_symantec.pem

This command specifies that the input format is DER, the input file is cert_symantec.der and that we want an output format of PEM saved to an output file called cert_symantec.pem. The observant will have noted that the command actually did not specify the output format of PEM. PEM is the default input and output format, so it does not need to be specified. However, if you like to remove ambiguity in a totally harmless and logical fashion, the full command would be:

openssl x509 -inform der -in cert_symantec.der -outform pem -out cert_symantec.pem

Easy peasy. The added benefit of understanding how to do this is that you now don’t have to use somebody else’s website to convert you internal certificates between formats.

4. Checking Your Own Chain of Trust

You’re ready to deploy a certificate for a website, and you have been given a ZIP file containing the public server cert and a file purporting to contain the necessary intermediate certificate(s). How can you check that you have the correct certificates without actually installing them? Why, openssl, of course!

MBP$ openssl verify -verbose cert-www-microsoft.pem
cert-www-microsoft.pem: /1.3.6.1.4.1.311.60.2.1.3=US/
  1.3.6.1.4.1.311.60.2.1.2=Washington/businessCategory=Private
  Organization/serialNumber=600413485/C=US/postalCode=98052/
  ST=Washington/L=Redmond/street=1 Microsoft Way/O=Microsoft
  Corporation/OU=MSCOM/CN=www.microsoft.com
error 20 at 0 depth lookup:unable to get local issuer certificate

What happened here? Error 20 was mentioned above; it means that the intermediate certificate (or at least, the certificate for the Issuer of the server certificate) is missing. Well of course it is; we didn’t supply it! The www.microsoft.com site uses a certificate from Symantec, so let’s use that and tell openssl about it:

MBP$ openssl verify -untrusted cert-symantec cert-www-microsoft.pem
cert-www-microsoft.pem: /C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 EV SSL CA - G3
error 20 at 1 depth lookup:unable to get local issuer certificate
OK

That’s progress; we still have an error 20, but now it has moved to “1 depth” – that is, the error is no longer on the server certificate (0 depth) but now we can’t find the issuer certificate for the Symantec cert. In a previous post, we discovered that the Symantec cert was issued by a Verisign entity that is in our trusted root store. So now I’ll add a link to the root store as well to complete the chain:

MBP$ openssl verify -untrusted cert-symantec 
       -CAfile ./RootCerts.pem cert-www-microsoft.pem
cert-www-microsoft.pem: OK

That’s it! We have confirmed that we have a full chain of trust from a trusted root cert all the way down to the www.microsoft.com server certificate. Even for a Mac user, this is a good thing.

What About Multiple Intermediate Certificates?

If you have more than a single Intermediate Certificate between the server and a trusted root certificate, you need to make them all available to the client. That’s easily done by creating a certificate bundle, which is a fancy way of saying “add all the certificates together in a single file.” Really. If you have two files each containing an intemediate certificate and need to bundle them, in *nix / OS X you do this:

$ cat intermediate1.pem intermediate2.pem > intermediatebundle.pem

It’s that easy. In any GUI environment you can just paste them one after another in Notepad and save them out. Remember to include the BEGIN and END lines. Now in your command line just change the argument to -untrusted intermediatebundle.pem and you’re good.

5. Testing for SSLv3 Using OpenSSL

This one is pretty easy. Using the s_client function again, we can ask openssl to try to connect using SSLv3. A site that supports SSLv3 (naughty naughty) will look like this:

MBP$ openssl s_client -ssl3 -connect microsoft.com:443
CONNECTED(00000003)
[...certificate stuff removed for brevity...]
SSL-Session:
    Protocol  : SSLv3
    Cipher    : RC4-SHA
    Session-ID: 33410000536...
    Session-ID-ctx:
    Master-Key: F88FCD7DF64CFB48...
    Key-Arg   : None
    Start Time: 1425840399
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

A site that does NOT support SSLv3 will look like this:

MBP$ openssl s_client -showcerts -ssl3 -connect www.microsoft.com:443
CONNECTED(00000003)
5688:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert
  handshake failure:/SourceCache/OpenSSL098/OpenSSL098-52.10.1/
  src/ssl/s3_pkt.c:1143:SSL alert number 40
5688:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake
  failure:/SourceCache/OpenSSL098/OpenSSL098-52.10.1/src/ssl/
  s3_pkt.c:564:
MBP$

There are other error codes you may see, but generally speaking if you get dumped back to a prompt with an error, chances are this means that SSLv3 is not allowed.

I thought it was interesting, by the way, that microsoft.com allowed an SSLv3 connection, but www.microsoft.com did not. The former uses a different certificate chain and redirects to the latter, so perhaps it all comes out in the wash. It’s actually a missed opportunity in some ways for Microsoft not to detect SSLv3 in some way, then pop up a web page saying “Hello IE6 user – why not upgrade now?”, but they don’t do that.

My 10 Bits

I have had to use all of these commands recently in order to troubleshoot a few different issues that were popping up with some web sites. I confess to being terrible at remembering commands in detail, so I’m going to bookmark my own page for reference even if you don’t! Openssl does plenty more that can be useful, but this is a great start when it comes to certificates and ciphers.

2 Comments on Five Essential OpenSSL Troubleshooting Commands

  1. Thank you so much for taking the time to write this article. It was invaluable to me in resolving some coding issues with OpenSSL. Using the command line tools was a much easier way of determining and resolving the issues. By the way, I hit another basic problem that I thought was worth mentioning – when passing PEM files to OpenSSL, it is very fussy about the character encoding. Cut & pasting your Microsoft example did not work at first, as my text editor saved (by default) with the wrong character encoding – resaving as ASCII instead of UTF-8 resolved the problem. Just FYI.

Leave a Reply

Your email address will not be published.


*


 

This site uses Akismet to reduce spam. Learn how your comment data is processed.