OpenSSL doesn’t come with its own trusted root certificates; you have to tell it where to find them. This should be straightforward – and it is – but Apple have found a way to make it trickier.

Normal *nix Systems

On a normal unix system, openssl is pretty good at locating the root certificates, but it still doesn’t automatically reference them. For example running Ubuntu:

john@ubuntu:~$ openssl s_client -connect www.microsoft.com:443
CONNECTED(00000003)
depth=2 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network,
  OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN =
  VeriSign Class 3 Public Primary Certification Authority - G5
verify error:num=20:unable to get local issuer certificate
verify return:0
[...removed for brevity...]
    PSK identity hint: None
    SRP username: None
    Start Time: 1425842365
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

Openssl is unable to validate the Verisign certificate. So where are the trusted root certificates stored? Actually, Openssl will tell us:

john@ubuntu:~$ openssl version -d
OPENSSLDIR: "/usr/lib/ssl"

Add that into the command as the -CApath parameter, and:

john@ubuntu:~$ openssl s_client -CApath /usr/lib/ssl -connect 
  www.microsoft.com:443
CONNECTED(00000003)
depth=3 C = US, O = "VeriSign, Inc.", OU = Class 3 Public Primary
  Certification Authority
verify return:1
depth=2 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network,
  OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN =
  VeriSign Class 3 Public Primary Certification Authority - G5
verify return:1
depth=1 C = US, O = Symantec Corporation, OU = Symantec Trust
  Network, CN = Symantec Class 3 EV SSL CA - G3
verify return:1
depth=0 1.3.6.1.4.1.311.60.2.1.3 = US, 1.3.6.1.4.1.311.60.2.1.2 =
  Washington, businessCategory = Private Organization, serialNumber
  = 600413485, C = US, postalCode = 98052, ST = Washington, L =
  Redmond, street = 1 Microsoft Way, O = Microsoft Corporation,
  OU = MSCOM, CN = www.microsoft.com
verify return:1
[...removed for brevity...]
    PSK identity hint: None
    SRP username: None
    Start Time: 1425842499
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

All is well again, and the certificates are considered trusted. This Ubuntu system runs “OpenSSL 1.0.1 14 Mar 2012”, by the way.

Now on OS X

Let’s try the www.microsoft.com check again in OS X:

MBP$ openssl s_client -connect www.microsoft.com:443
CONNECTED(00000003)
depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
verify error:num=20:unable to get local issuer certificate
verify return:0
    [...removed for brevity...]
    Key-Arg   : None
    Start Time: 1425842726
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

So hang on a moment; we still get the error 20 indicating that the Verisign certificate couldn’t be validated, yet the final return code is “0 (ok)”. I intentionally didn’t bring this up in my other posts on this topic as I didn’t want to get sidetracked but what on earth is going on here? Surely this should (like Ubuntu) carry the error 20 down to the final return code?

I’ll have to think on that, but meanwhile let’s find the trusted root certificates:

john-mbp-wlan:~ john$ openssl version -d
OPENSSLDIR: "/System/Library/OpenSSL"

Looks promising, so let’s try referencing the path:

MBP$ openssl s_client -CApath /System/Library/OpenSSL -connect www.microsoft.com:443
    CONNECTED(00000003)
    depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
    verify error:num=20:unable to get local issuer certificate
    verify return:0

Oh, it didn’t work. On Ubuntu, /usr/lib/ssl/certs is a symlink to /etc/ssl/certs/ which contains lots of .pem files representing the root certificates. On OS X:

MBP$ ls -al /System/Library/OpenSSL/certs
total 0
drwxr-xr-x  2 root  wheel   68 Sep  9 18:39 .
drwxr-xr-x  6 root  wheel  204 Oct 18 09:45 ..
MBP$

Oh, it’s empty. Well that might explain why adding this as the CApath fails. I’ve confirmed the failure in both Yosemite and Mavericks (which I have available for testing), but I suspect that this has always been the case because OS X (and even MacOS before that) manage root certificates as part of the Keychain system instead.

Keychain Access

OSX doesn’t store the trusted certificates in a directory like Ubuntu does (at least not that I could find), so there’s no simple way to reference them. However, they are available if you use the Keychain Access tool in the GUI. I gather that there are CLI equivalents, but I’ll leave that to somebody else to find I think. So how do we reference the root certs? We have to export them. Thankfully this is very simple. Open Keychain Access and choose to view the System Roots:

Click on any certificate, then select all (either using CMD-A or Edit->Select All). To export all the certificates, either use File->Export Items, right-click and choose “Export NNN Items” or use Shift-CMD-E. Change the filename and location as necessary and keep the format as PEM (openssl likes that, remember!).

Click Save and all the trusted root certificates will be exported into a single file (a bundle) and are ready for use. Since we’re now referencing a single file rather than a directory full of files, we use the -CAfile option instead of -CApath:

MBP$ openssl s_client -CAfile RootCerts.pem -connect 
       www.microsoft.com:443
CONNECTED(00000003)
depth=3 /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary 
  Certification Authority
verify return:1
depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 
  2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class
  3 Public Primary Certification Authority - G5
verify return:1
depth=1 /C=US/O=Symantec Corporation/OU=Symantec Trust Network/
  CN=Symantec Class 3 EV SSL CA - G3
verify return:1
depth=0 /1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=
  Washington/businessCategory=Private Organization/
  serialNumber=600413485/C=US/postalCode=98052/ST=Washington/
  L=Redmond/street=1 Microsoft Way/O=Microsoft Corporation/
  OU=MSCOM/CN=www.microsoft.com
verify return:1
---
[...removed for brevity...]
Key-Arg   : None
Start Time: 1425843723
Timeout   : 300 (sec)
Verify return code: 0 (ok)
---

Now, we see that the certificates have expanded from depth=2 to depth=3 because the full chain is actually made up of four certificates. The entire chain was validated and there are no errors logged. So now you know.

My 10 Bits

I can’t help feeling that it would be useful if OS X found some way to expose the root certificates as a file or directory in some way for use by command line tools. While it’s easy to export the certificates from Keychain Access, it also means that a new export is required whenever there’s an update to the root certificates. Do you know when that happens? I don’t.