Tips & tricks for installing and running IBM products

Letsencrypt certificates for my own test servers

Tom Bosmans  26 June 2018 14:02:15
Yes, it's a bit over the top to use Let's Encrypt certificates for test systems, where a self-signed certificate would serve a similar purpose.  Furthermore, a Let's Encrypt certificate has a short lifietime and needs to be replaced every 3 months.

But since Let's Encrypt brought wild-card certificates to us fairly recently (march 2018), there is an advantage here.  You only need this single certificate and you can use it on all your systems.  Of course, in most case  you don't want to use wild-card certificates, but for my case (non-production test systems) , this is perfectly valid.
https://community.letsencrypt.org/t/acme-v2-and-wildcard-certificate-support-is-live/55579

You also must use the DNS challenge (instead of the more traditional https challenge that Let's encrypt uses for verify ) .  The mechanism that is used, is similar to other verification mechanisms like DKIM for smtp (see DKIM deployed on my mail servers )

My usecase here, is a test environment running ISAM - IBM Security Access Manager (https://www.ibm.com/us-en/marketplace/access-management).  Not having to trust the signer each time I access a page with a sefl-signed certificate , is a huge plus when demoing a solution :-)

1. Prerequisites
  • You need a recent version of certbot (that has the support for dns challenge, and the support for ACMEv2) , I'm using certbot 0.24.0
  • This certbot needs to run on a system with Internet access (outbound only, it needs to connect to the letsencrypt systems)
  • You also need a public dns domain, because Let's Encrypt uses DNS for the verification.   The only thing that needs to be in the domain records, is a TXT record btw.  You don't need to configure anything else.

2. DNS Preparation

I ordered a dns domain from my prefered DNS provider (https://www.domaindiscount24.com) .
I could get a EU domain for something like 3 Euro for the first year .

There is nothing to configure for now , the configuration is done during the certbot action.

3. Certbot


Run certbot with the option --preferred-challenges dns , and define your domain as *.. (mine is *.tombosmans.eu) .
You can also use certbot-auto, and you can use a single commandline here, but I used this method :


[root@system ~]# certbot certonly --manual --server
https://acme-v02.api.letsencrypt.org/directory --preferred-challenges dns
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Plugins selected: Authenticator manual, Installer None

Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org

Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c'

to cancel): *.tombosmans.eu

Obtaining a new certificate

Performing the following challenges:

dns-01 challenge for tombosmans.eu


-------------------------------------------------------------------------------

NOTE: The IP of this machine will be publicly logged as having requested this

certificate. If you're running certbot in manual mode on a machine that is not

your server, please ensure you're okay with that.


Are you OK with your IP being logged?

-------------------------------------------------------------------------------

(Y)es/(N)o: Y


-------------------------------------------------------------------------------

Please deploy a DNS TXT record under the name

_acme-challenge.tombosmans.eu with the following value:


9zE0cU5V1hiYo5HJWY-Zx6FW74gl1gd5P9dnS0G8cYw


Before continuing, verify the record is deployed.

-------------------------------------------------------------------------------

Press Enter to Continue



So now you need to go to your DNS provider, and create a TXT DNS record for _acme-challenge. , in my case, _acme-challenge.tombosmans.eu

_acme-challenge 28800 IN TXT 9zE0cU5V1hiYo5HJWY-Zx6FW74gl1gd5P9dnS0G8cYw


In the interface of my DNS provider, it looks like this :  I need to create a new subdomain, named _acme-challenge.tombosmans.eu .

Image:Letsencrypt certificates for my own test servers
In the next step , I can then enter the value that certbot provided , in a TXT field.

Now once you saved your DNS entry, DO NOT continue immediately.
 

Give it at least 1 minute, so you're certain the DNS entry is available, or even better, verify that your nameserver is up-to-date by performing a dns lookup, for instance using dig .

In my case, I can use this command, to use the nameserver of my provider.  Do this on another system you have your certbot command running, or open a new session .

dig -t txt +short @ns1.domaindiscount24.net _acme-challenge.tombosmans.eu

"9zE0cU5V1hiYo5HJWY-Zx6FW74gl1gd5P9dnS0G8cYw"



It needs to return the value of the TXT record.  As long as it doesn't, DO NOT continue in the certbot session, because it will fail and you need to start over.

But if it does return the key, continue.  



Waiting for verification...

Cleaning up challenges


IMPORTANT NOTES:

- Congratulations! Your certificate and chain have been saved at:

/etc/letsencrypt/live/tombosmans.eu/fullchain.pem

Your key file has been saved at:

/etc/letsencrypt/live/tombosmans.eu/privkey.pem

Your cert will expire on 2018-09-24. To obtain a new or tweaked

version of this certificate in the future, simply run certbot

again. To non-interactively renew *all* of your certificates, run

"certbot renew"

- If you like Certbot, please consider supporting our work by:


Donating to ISRG / Let's Encrypt:  
https://letsencrypt.org/donate
Donating to EFF:                    
https://eff.org/donate-le



Now the chain and certificate files are in the standard Let's Encrypt locations described (/etc/letsencrypt/live/./
Since this is a wildcard certificate, you likely want to copy it elsewhere and distribute across your systems.

4. Let's Encrypt keys and ISAM


IBM Security Access Manager expects pkcs12 certificates , so we first need to use openssl to convert the letsencrypt certificates to a .p12 .
I'm using ISAM 9.0.5, as OVA .


openssl pkcs12 -export -out tombosmans.eu.p12 \

-inkey /etc/letsencrypt/live/tombosmans.eu/privkey.pem \

-in /etc/letsencrypt/live/tombosmans.eu/cert.pem \

-certfile /etc/letsencrypt/live/tombosmans.eu/chain.pem


Enter Export Password:

Verifying - Enter Export Password:



Use a strong password to protect your key !
( https://community.letsencrypt.org/t/combining-key-and-certificate-into-a-pkcs12-file/21113/3 )

Now get the certificate to a system where you can upload it to ISAM .

In the LMI, I want it in 2 places :
- the management certificate
- the default certificate for the reverse proxies


hosts file on ISAM

Add the ip addresses for the interfaces you want to use in the hosts file on ISAM.  We could use DNS as well (since we have the public dns domain), but since this is internal, I am not going to do that and use simple hosts files .
Image:Letsencrypt certificates for my own test servers

Also, I use the following hosts file on my local machine to access my environment :

192.168.42.42  isam.tombosmans.eu

192.168.42.100
frontend.tombosmans.eu


management certificate

Go to Manage System Settings/System Settings/Management SSL Certificate

Image:Letsencrypt certificates for my own test servers
The LMI will be restarted after this.

pdsrv keydb


Edit the pdsrv keydb, go to Personal certificates and select "Import"
Image:Letsencrypt certificates for my own test servers
Then select the "Let's Encrypt" certificate , click "Edit" and set it as the default certificate.

Image:Letsencrypt certificates for my own test servers


The DST ROOT CA is missing from the IBM provided keydbs (this is actually a bug in my opinion -  see this link :
https://community.letsencrypt.org/t/dst-root-missing-from-p12/48648/4 ), so you need to add it to the Signer Certificates in all key databases basically.  You can download it from the website in the link below, or you can export it from any modern browser (for example FireFox, below).  

Image:Letsencrypt certificates for my own test servers

Note that the reverse proxy can't handle the missing root CA, while the LMI does not seem to require it.   In each case, any server you want to protect using ISAM that would use TLS/SSL and a Let's Encrypt certificate would also require you to add this DST Root CA X3.


You need to restart the reverse proxies after saving and deploying this.

5. End result


I can now access the LMI on this url :

Image:Letsencrypt certificates for my own test servers

... and the reverse proxy (using Chrome this time round)
Image:Letsencrypt certificates for my own test servers

Everything is green, so everything is OK (at least , OK enough for my test environment).

Additional information


ISAM automation

To automate all these manual actions, I actually should use automation tooling like Ansible .
Fortunately , there is a publicly available repository with ansible-roles and playbooks for ISAM .  It would be relatively straightforward to automate the management of the certificates here (generate a new one, use openssl to convert it, upload it to ISAM for the reverse proxies and for the management interface).
https://github.com/IBM-Security

My zonefile, for your information


This zonefile is obviously pretty specific to my DNS provider and to my situation, but still, it may serve as an example for what you would need to have to make this work .
It's the _acme-challenge entry that does the trick .


$ORIGIN tombosmans.eu.

@ 28800 IN SOA ns1.domaindiscount24.net. tech.key-systems.net. 2018062619 10800 3600 604800 28800

@ 28800 IN NS ns1.domaindiscount24.net.

@ 28800 IN NS ns2.domaindiscount24.net.

@ 28800 IN NS ns3.domaindiscount24.net.

_acme-challenge 28800 IN TXT 9zE0cU5V1hiYo5HJWY-Zx6FW74gl1gd5P9dnS0G8cYw