Obtaining, reissuing and installing SSL certificates across various pieces of Artsy infrastructure (read about our tech stack here) is a mind-numbing excercise dreaded by every platform engineer. Few people really understand what’s going on, installing SSL certificates is right up there in the opaque world of user-unfriendliness, along with onboarding a new iOS developer in an organization with Xcode.
Still, it can be done - below is our cookbook.
Buying an SSL Certificate
We buy our SSL certificates from rapidsslonline.com. You’ll be immediately overwhelmed and confused by the many brand names, including Symantec, RapidSSL, Thawte and GeoTrust and the mentions of “no extra cost”, “cheap” and “quality”. We buy two kinds of GeoTrust certificates: GeoTrust True BusinessID with Extended Validation and GeoTrust True BusinessID Wildcard. We typically get those for a at least 2 years, trying to balance the pain of installing them with the time it takes our short collective memory to forget how to obtain and deploy an SSL cert.
When buying extended validation certificates you will need an administrative and a technical contact, a working company phone number and a public entry such as yellow pages. You’ll need to begin by making a CSR, which is explained below.
GeoTrust SAN True BusinessID with Extended Validation
This certificate lets us have the green Artsy Inc. pictured above and to list multiple domains - SAN stands for Subject Alternative Names. To obtain it your company must have a public record with the Better Business Bureau or YellowPages.com with a working phone number. You will have to fill out an Acknowledgement Agreement, that will make everything seem more legit. This certificate requires you to list hostnames. You can purchase additional names when needed and reissue the certificate - we typically include www.artsy.net, m.artsy.net, etc.
GeoTrust True BusinessID Wildcard
This certificate can be installed on anything.example.com which is great for both test and production systems - we use SSL everywhere.
Install OpenSSL (typically via
brew install openssl).
Organizing Files and Keeping it Safe
You will want to organize your certificate files into folders, by name, type and date. We typically place everything in a shared Dropbox folder that is only accessible to engineering operations. Ours looks like this:
For the examples below we’ll use an example.com domain.
The first step is to generate a Certificate Signing Request (CSR). The important information in the CSR is your location and the e-mail address.
$ openssl req -new -newkey rsa:2048 -nodes -keyout example.com.key -out example.com.csr Generating a 2048 bit RSA private key ....................+++ ..............+++ writing new private key to 'example.com.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:New York Locality Name (eg, city) :New York Organization Name (eg, company) [Internet Widgits Pty Ltd]:Example, Inc. Organizational Unit Name (eg, section) : Common Name (eg, YOUR name) :*.example.com Email Address :email@example.com Please enter the following 'extra' attributes to be sent with your certificate request A challenge password : An optional company name :
The above creates two files: example.com.csr (the CSR) and example.com.key (a private key to sign the CSR and the future certificate).
This is slightly different than for a wildcard certificate. From a new folder, copy the OpenSSL configuration file.
$ cp /System/Library/OpenSSL/openssl.cnf .
Edit the file as follows.
Inside the v3_req section uncomment req_extensions. This will tell OpenSSL to include this section in the CSR.
[req] req_extensions = v3_req
Create the v3_req section with the following information along with the DNS names that will be used for the EV certificate. This can be changed later, and additional names can be bought, but will require re-issuing the certificate.
[ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = example.com DNS.2 = www.example.com DNS.3 = m.example.com DNS.4 = api.example.com
Generate the CSR.
$ openssl req -new -out san.example.com.csr -newkey rsa:2048 -nodes -keyout san.example.com.key -config openssl.cnf Country Name (2 letter code) [US]:US State or Province Name (full name) [New York]:New York Locality Name (eg, city) :New York Organization Name (eg, company) [Example, Inc.]:Example, Inc. Organizational Unit Name (eg, section) : Common Name (eg, YOUR name) :example.com Email Address :firstname.lastname@example.org Please enter the following 'extra' attributes to be sent with your certificate request A challenge password : An optional company name :
This creates san.example.com.csr and san.example.com.key. Check that it includes the correct information.
$ openssl req -text -noout -in san.example.com.csr Subject: C=US, ST=New York, L=New York, O=Example, Inc./emailAddressemail@example.com Requested Extensions: X509v3 Basic Constraints: CA:FALSE X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment X509v3 Subject Alternative Name: DNS:example.com, DNS:www.example.com, DNS:m.example.com, DNS:api.example.com
Buying the Certificate
When buying a certificate you will be asked to copy-paste the contents of the CSR generated above. There will be some back-and-forth for the verification process over e-mail and phone. The GeoTrust authenticator will be happy to hear that your yellow pages listing’s phone number matches their records.
Once issued you will receive an e-mail with the certificate. It contains the web server certificate (save it as example.com.crt) and an intermediate certificate (save it as example.com.intermediate.crt).
For RapidSSL you will also need more intermediate CAs, currently obtained from here. You can get them as a bundle (we save it as intermediate.crt) or separately (we save these as intermediate-primary.crt and intermediate-secondary.crt).
You need the SSL endpoint addon to upload the certificate to Heroku. The certificate must be a concatenation of your certificate and all intermediaries. For our RapidSSL certificates this is usually made as follows.
$ cat example.com.crt intermediate.crt > concatenated.crt $ heroku certs:add concatenated.crt example.com.key --app example-app Adding SSL endpoint to example-app... done example-app now served by tokyo-7171.herokussl.com Certificate details: subject: /serialNumber=xyz/OU=GT65834870/OU=See www.rapidssl.com/resources/cps (c)13/OU=Domain Control Validated - RapidSSL(R)/CN=*.example.com start date: 2013-01-13 18:21 EST expire date: 2014-01-17 01:55 EST common name(s): *.example.com, example.com issuer: /serialNumber=yz/OU=GT65834870/OU=See www.rapidssl.com/resources/cps (c)13/OU=Domain Control Validated - RapidSSL(R)/CN=*.example.com SSL certificate is verified by a root authority.
If you have a certificate already installed, update it.
$ heroku certs:update concatenated.crt example.com.key --app example-app
Heroku is known to patch common certificate issues during this process, such as certificates listed out of order.
OpsWorks offers 3 SSL fields in the application configuration (choose Apps, then Edit the app):
The SSL Certificate should be the same contents of the concatenated.crt we made above.
The SSL Certificate key should be the contents of example.com.key.
The SSL Certificates of Certification Authorities should be left empty.
The OpsWorks application will need to be redeployed for the new certificates to be installed.
Login to AWS, go to EC2 Instances, click on Load Balancers, find your load-balancer and choose the Listeners tab. You can upload or choose a certificate here.
The Private Key should be example.com.key.
The Public Key Certificate should be the contents of example.com.crt.
The Certificate Chain is the most problematic, it needs to list the entire chain, in our case it’s a concatenation of example.com.intermediate.crt, then intermediate-primary.crt, and the root certificate for issuer of the latter, which is the Equifax Secure Certificate Authority from here.
When the chain is incorrect the generic error message is “Unable to validate the certificate chain.” It took us a few tries and a bit of help from the AWS forums.
Reissuing a Certificate
Recently we had to upgrade our certificates from those signed using a SHA-1 hash algorithm to those signed with the SHA-256 algorithm, and had to change DNS names in the SAN certificate.
Find the order on RapidSSLOnline (you will likely have to change the start date of your search), and follow the link to reissue the certificate at the bottom. Create a new CSR, as for any new certificate. No need to choose a specific web server type and make sure to check SHA-256 for the signature method. You will need to go through all the same motions for a reissued certificate as for a new one.
Verifying a Certificate
Verify that your certificate is correct and contains all the right intermediaries.
$ openssl s_client -CApath /etc/ssl/certs/ -connect example.com:443 SSL-Session: Protocol : TLSv1 Cipher : AES256-SHA Session-ID: A746CC85733F16630CA9CBD12FD1A0379CEC22649C1558D12B575529815F8E67 Session-ID-ctx: Master-Key: 09EBFF52D4F673FA0FA5B6BD1151722CF96197711430BCA35B01D6AA2803F8904D834F4EEB151E6354601FEC55CDFDCA Key-Arg : None Start Time: 1428507204 Timeout : 300 (sec) Verify return code: 0 (ok)
You want the Verify return code: 0 (ok) message at the bottom. Pay attention to other errors, a common one may be verify error:num=19:self signed certificate in certificate chain, which can be ignored and verify error:num=20:unable to get local issuer certificate, which is a real problem and means you’re missing an intermediate certificate.