Introduction

tl;dr

This post covers setting up Caddy as a reverse proxy, and configure it to perform client certificate validation (a.k.a. mTLS). In addition to this, the post covers storing the client certificate and the private key securely on a Yubikey (acting as a virtual PIV smart card).

I wasn’t able to find a similar quickstart tutorial on the internet, therefore I’m writing this one. I have absolutely zero affiliation with Yubico or Caddy; I just enjoy mucking about with their products and tools as they are really one step closer to ItJustWorks(TM) than several other solutions.

Background

You have a self-hosted web service that you want to expose to the Internet - not because you want it to be public, but because you want to reach it from everywhere. You’ve considered various tunnel solutions including VPNs, but refrain from using them due to operational complexity and inevitable IP addressing conflicts.

You’ve heard of the possibility to use client certificates in the browser, acting as an authentication mechanism operating on the TLS layer. By doing so, you get a similar tunnel as you would have with several TLS based VPN solutions.

You decide to go for it, but when threat modeling the setup, you quickly realize that letting your users store the client certificate and its private key directly on their devices may be insecure for a number of reasons. You want to enforce- and govern protection of the user’s private key. You’ve heard that smartcards can help out here, by acting as a hardware token that electronically is designed in a way that enables RSA operations inside of the hardware, but does not allow for the private key to be read out or extracted from the card once it has been loaded onto it.

You quickly realize that smartcards, although functional, come with one major problem: not all laptops have readers, and you don’t want to roll out external ones. Enter YubiKeys!

Yubico’s YubiKeys offer the functionality of a virtual PIV smartcard, which is what we will use in this tutorial to safely store and protect the client certificate and its belonging private key.

In addition to setup and configure the YubiKey, we’ll also cover setting up a webserver that performs the client certificate validation as part of the TLS handshake, and configure Firefox to present a client certificate from a YubiKey when accessing the webserver.

Outline

  • Setup a PKI using Easy-RSA
  • Setup Caddy as a reverse proxy that performs client certificate validation
  • Generate a client certificate and store it in the Yubikey
  • Setup Firefox to use the Yubikey through PKCS11

Disclaimer

Do not hold me accountable for any misunderstandings or misconfigurations. If you have a data breach after having followed this post - it’s on you, not me.

Also, I have no affiliation with Yubico, Let’s Encrypt, Caddy, Firefox or any other tech mentioned in this post.

Finally, please excuse any misspellings or just overall bad English.

Tutorial

Note: All steps in the tutorial are written for Ubuntu unless stated differently.

Setting up the PKI

First of all, we need to set up a CA that will issue the client certificates. In this tutorial, we will use Easy-RSA to simplify this step. Easy-RSA is a well-established and maintained shell script around the OpenSSL CLI, provided by the OpenVPN project (https://github.com/OpenVPN/easy-rsa).

You can download and install Easy-RSA directly from Github Releases into your CA directory, or you can use the Ubuntu provided package:

$> sudo apt-get install easy-rsa

The Ubuntu package also provides a script to symlink in Easy-RSA to your Easy-RSA directory:

$> make-cadir my-mtls-pki
$> cd my-mtls-pki

Now that you have an Easy-RSA directory, it’s time to set up the CA by initializing the folder structure, and then build a CA (including generating the private key and the self-signed CA certificate). Easy-RSA will prompt you for a number of attributes for the CA, as well as a password to encrypt your CA’s private key.

$> ./easyrsa init-pki
$> ./easyrsa build-ca

Note: Do not lose control of this directory as it holds the private key of the CA, which is the root of trust in your PKI.

Setting up Caddy

Spin up a VPS somewhere and SSH to it. Make sure it has port 80 and port 443 accessible from the Internet (80 needed for ACME challenge), and point a DNS A record to it.

In this tutorial we will be using Caddy for one reason: simplicity. Caddy can automatically retrieve certificates from Let’s Encrypt (no need for a separate certbot or similar), it has support for basic client certificate validation and is very easy to configure, which makes it perfect for a tutorial. For a production scenario, you might want to consider another webserver with support for OCSP or CRLs, in the event that you want to revoke a client certificate.

Install Caddy by downloading a DPKG package from Github Releases:

$> wget https://github.com/caddyserver/caddy/releases/download/v2.4.3/caddy_2.4.3_linux_amd64.deb
$> sudo dpkg -i caddy_2.4.3_linux_amd64.deb

Install your recently generated CA certificate to the server by copying the contents of my-mtls-pki/pki/ca.crt to /etc/caddy/ca.crt

Edit the Caddy configuration to both retrieve a Let’s Encrypt server certificate and to require- and verify client certificates from the browser against your CA certificate:

mtls.mydomain.com { # <--- FQDN that Caddy should retrieve a certificate for
    reverse_proxy localhost:8080 # <--- Where to proxy incoming requests
    tls {
        client_auth {
            mode require_and_verify # <--- Require browser to present certificate
            trusted_ca_cert_file /etc/caddy/ca.crt # <--- Validate against our CA
        }
    }
}

Restart Caddy, and verify that it’s running:

$> sudo systemctl restart caddy
$> sudo systemctl status caddy

Install a web service of your choice, listening on localhost:8080. I will use nginx’s sample docker container:

$> docker run -P -d nginxdemos/hello

Issuing a client certificate

Connect your Yubikey. Verify that your computer recognizes it and can communicate with it by calling ykman info. Specifically, you should verify that the CCID USB interface is enabled, and that the PIV application is Enabled.

$> ykman info
Device type: YubiKey 5C
Serial number: 666666
Firmware version: 5.2.7
Form factor: Keychain (USB-C)
Enabled USB interfaces: CCID

Applications
OTP     	Disabled
FIDO U2F	Disabled
OpenPGP 	Enabled
PIV     	Enabled
OATH    	Enabled
FIDO2   	Disabled

Generate a private RSA key on the YubiKey, and generate a CSR from it by following the steps below. By generating the RSA key on the actual YubiKey and not importing it, you make sure that the private key don’t get lost.

$> ykman piv generate-key 9a /tmp/alice.pub
$> ykman piv generate-csr -s alice 9a /tmp/alice.pub /tmp/alice.csr

You will be prompted for a management key. The management key protects various management operations on the virtual smartcard of the YubiKey. You will also be prompted for a PIN which will be used to protect the virtual smartcard. Both the MGM and the PIN key have default values, and you can for now just hit enter to use the defaults. In a production scenario, you most likely want to change those!

In the commands above, 9a refers to a slot on the virtual smartcard. 9a in the PIV standard is used for authentication. See https://developers.yubico.com/PIV/Introduction/ for more information on the Yubikey’s PIV application and the different keys.

Now, use your previously setup Easy-RSA PKI to issue a client certificate for the recently generated CSR.

$> cd my-mtls-pki
$> ./easyrsa import-req /tmp/alice.csr alice
$> ./easyrsa siqn-req client alice

You will be promted for the CA private key password that you entered when the PKI and the CA was set up. The certificate will be available under the pki/issued/ directory with name alice.crt.

Finally, import the recently issued certificate back onto the YubiKey.

$> ykman piv import-certificate 9a my-mtls-pki/pki/issued/alice.crt

Like before, you’ll be prompted for the management key.

Configure Firefox to use YubiKey certificate

Connect your Yubikey. Install the official YubiKey PKCS11 module:

$> sudo apt-get install ykcs11

Configure Firefox to use the PKCS11 module by:

  1. Go into Settings → Privacy & Security → Certificates → Security Devices.
  2. Click Load.
  3. Enter a name (e.g. YKCS11) and browse for the module filename to /usr/lib/x86_64-linux-gnu/libykcs11.so.
  4. Restart Firefox for the changes to take effect properly.
  5. Repeat step 1 - you should now see the YubiKey as a hardware device.
  6. Go into Settings → Privacy & Security → Certificates → View Certificates. You’ll be prompted for your PIN. After having entered it, you should see your client certificate under Your Certificates.

Now, open a tab in Firefox, and hit the Caddy server you set up earlier. You’ll be prompted for your PIN again, see below:

PIN prompt

After having entered your PIN and hit enter, your YubiKey performs the client part of the TLS handshake, using the private key that was generated on it, and which will never and can not ever leave that YubiKey!

Enjoy your secure web service!

Read more