Certificate management on Kubernetes

When exposing services it's generally a good idea to follow the industry standard and use HTTPS protocol. HTTPS requires a certificate issued by a trusted third party, called a Certificate Authority (or CA for short). There are several ways to acquire one, but a simple and effective method is to use Let's Encrypt (a CA) by way of the ACME protocol. The ACME protocol is a communication protocol for interacting with CAs that makes it possible to automate the request and issuance of certificates. The idea is that manual certificate management can easily result in expired certificates, which usually translate to a non-working website and/or services. For this reason, you should be able to configure your infrastructure once, and let it handle certificate renewals automatically. To ensure that you automate this process, certificates issued by Let's Encrypt are valid for only 90 days, since it's widely believed that 90 days is frequent enough for users to automate the handling of renewals .

The ACME protocol

What you need to know about the ACME protocol is that it involves proving that you control the domains present in the Certificate Signing Request (CSR). This is done by solving challenges (one for each domain). You can usually choose between several challenge types, which vary depending on the CA and the domains involved. The two most common challenge types are HTTP-01 and DNS-01.

HTTP-01 challenge

This challenge type is solved by replying to a specific HTTP request with an appropriate response. The request is a GET request to a url of the form http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN> and the response is a combination of the token and your ACME account key. This is probably the simplest challenge to automate, but it can't be used for wildcard domains (for example *.example.com). For wildcard domains you have to solve the DNS-01 challenge.

DNS-01 challenge

Solving the DNS-01 challenge can be done by putting a specific value in a TXT record for the given domain. As the main idea behind the ACME protocol is automation, this challenge type only makes sense if your DNS provider has an API. There are several ACME clients which can handle the submitting of CSRs as well as solving the required challenges. One such client is certbot which can handle "legacy" environments (Apache, Nginx, etc.). If you are running your services in a Kubernetes cluster, your best bet is to use cert-manager. Let's take a look at how it works.

cert-manager

Cert-manager is the complete package when it comes to handling multiple certificate issuer types (ACME, self-signed, CA among others). It can acquire and automatically renew certificates before expiry. If you are using Kubernetes Ingress to route your ingress traffic, cert-manager can automatically solve HTTP-01 challenges. It does this by spinning up a pod for each challenge, then applying the necessary routing changes to your Ingress config. When you interact with cert-manager you'll be using a couple of custom resource types: Issuer, ClusterIssuer, and Certificate. There are a few other custom resource types involved, which aren't necessary to know about but which can be helpful in tracking down errors. These are the CertificateRequest, Order and Challenge custom resources. Let's see what each one is used for.

User facing custom resources

Issuer, ClusterIssuer resources

An issuer is an entity that can generate signed certificates. There are several supported issuers built into cert-manager, and it can be extended with new ones if necessary. An Issuer or ClusterIssuer resource describes one issuer entity. You will need at least one such resource in your cluster. We will be focusing on the ACME Issuer type. ACME is the protocol implemented by Let's Encrypt. The difference between Issuer and ClusterIssuer is that the Issuer's use is restricted to the namespace it's created in; it's not possible to reference it from a Certificate resource in another namespace. A ClusterIssuer, however, is global, usable from any Certificate resource in the cluster.

Certificate resource

A Certificate resource describes the intention to acquire a certificate for one or more of your domains. Each certificate must reference an issuer resource. This will be the issuer used for acquiring the desired certificate.

When cert-manager notices a new Certificate resource it proceeds with the creation of a CertificateRequest.

Internal custom resources

CertificateRequest resource

This resource represents one request for a certificate. It contains the certificate signing request, which is itself encoded in PEM format, and the certificate if it was already received, in which case Ready will be set to True.

The absence of this resource doesn't mean the certificate hasn't yet been requested. Cert-manager will not recreate this resource if the certificate has already been acquired.

Order resource

This resource type comes into play when a certificate is requested from an ACME issuer. An Order resource represents the order for a certificate to be issued.

In its status you can find the challenges to be solved. There can be several challenge types for a domain, but only one of those need be solved. Cert-manager will select one challenge type for each of the domains, then proceed with the creation of Challenge resources.

Challenge resource

Not unlike the Order resource type, this resource type plays a role when the certificate is requested from an ACME issuer. These resources describe the challenges cert-manager selected to solve.

Now that we've covered the parts of cert-manager, let's see it in action!

Try it out!

You will need a kubernetes cluster and domain to play with.

Create cluster

1. Create a Kubernetes cluster.

   If you need a hand with that, you can create a cluster with the Banzai Clouds Pipeline platform on five different clouds or on-premise. Pipeline is available online, for free at Try Pipeline.

2. Point KUBECONFIG at your cluster.
3. Make sure you have an Ingress controller in your cluster. If you used Banzai Cloud's Pipeline platform to create the cluster, you already have one along with a LoadBalancer-type Service:

   $ kubectl get -n pipeline-system deploy/ingress-traefik svc/ingress-traefik
     NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
     deployment.apps/ingress-traefik   1/1     1            1           3m19s
   
     NAME                      TYPE           CLUSTER-IP     EXTERNAL-IP                                                               PORT(S)                      AGE
     service/ingress-traefik   LoadBalancer   10.10.231.18   a015671f691794291bd991eb635ed907-1982155256.us-east-2.elb.amazonaws.com   443:32271/TCP,80:30543/TCP   3m20s

   The service should have port 80 and 443 open. Port 80 is for the ACME HTTP-01 challenge, and port 443 is for HTTPS traffic.
4. Install cert-managerIt can be as simple as:

   kubectl apply -f https://github.com/jetstack/cert-
   manager/releases/download/v0.15.1/cert-manager.yaml
   

   There are other options and more details at https://cert-manager.io/docs/installation/kubernetes/
5. Point your domain to the external IP of your LoadBalancer typed Service on which you want external traffic to enter your cluster. "External IP" might be a DNS name (as it is in the example above), in which case you will have to create a CNAME record instead of an A record.Set the DOMAIN environment variable to your domain (example.com will not work):

   $ DOMAIN=example.com
   

6. Deploy httpbin for testing purposes

   $ kubectl apply -f - <<EOF
   apiVersion: v1
   kind: Service
   metadata:
     name: httpbin
     namespace: default
   spec:
     selector:
       app: httpbin
     ports:
       - port: 8080
         protocol: TCP
         targetPort: 80
   ---
   apiVersion: v1
   kind: Pod
   metadata:
     name: httpbin
     namespace: default
     labels:
       app: httpbin
   spec:
     containers:
       - image: kennethreitz/httpbin:latest
         name: httpbin
         ports:
           - containerPort: 80
             protocol: TCP
   EOF

   You can check if it's up and running through port-forwarding and curl:

   $ kubectl port-forward svc/httpbin 8080 &
   $ curl localhost:8080/get
   

7. Create an issuer

   Note: we will be using the staging provider to avoid hitting rate limits.

   First, set the EMAIL environment variable to your email address. Let's Encrypt will use this to contact you about expiring certificates and other issues related to your account.

   $ EMAIL=some@email.address

   Now, you're ready to create the issuer for Let's Encrypt:

   $ kubectl apply -f - <<EOF
   apiVersion: cert-manager.io/v1alpha2
   kind: ClusterIssuer
   metadata:
     name: letsencrypt-staging
   spec:
     acme:
       email: ${EMAIL}
       server: https://acme-staging-v02.api.letsencrypt.org/directory
       privateKeySecretRef:
         # Secret resource that will be used to store the account's private key.
         name: example-issuer-account-key
       solvers:
         - http01:
             ingress:
               name: test-ingress
   EOF

8. Create an ingress

   $ kubectl apply -f - <<EOF
   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     name: test-ingress
     namespace: default
   spec:
     backend:
       serviceName: httpbin
       servicePort: 8080
     tls:
       - hosts:
           - ${DOMAIN}
         secretName: test-ingress-cert
   EOF

   This ingress is not functional yet, because the secret being referenced does not exist.

   If you remove the tls config, the service should be reachable on HTTP protocol:

   $ curl ${DOMAIN}/get

   If it's not working, try the external IP of your ingress service to see if the problem is in your cluster or with the DNS resolution. If it works with the external IP, you might need to wait a couple of minutes for the DNS changes to propagate.

9. Acquire a certificateAs I have mentioned, we have a non-working ingress because of the missing secret. We have a couple options to acquire a certificate and make it work:
   1. Create the ingress without tls config. This way, the ingress can serve HTTP traffic to complete the ACME HTTP-01 challenge. After the certificate is issued, the tls config can be added back in.
   2. Create a self-signed certificate initially, and then acquire a properly signed one:
      1. Create a self-signedIssuer and use it in your Certificate resource
      2. Wait until the certificate is ready (check it using curl --insecure https://${DOMAIN}/get)
      3. Edit your Certificate resource and change the issuer to the Let's Encrypt Issuer
      4. The self-signed certificate will then be replaced by a newly acquired, properly signed certificate
   3. The easiest solution: annotate the Certificate resource to instruct cert-manager to create a self signed certificate before attempting to request the properly signed one. This is basically the same as method 2, but automated.
   Let's go with method 3 and annotate the Certificate resource with cert-manager.io/issue-temporary-certificate: "true".

   $ kubectl apply -f - <<EOF
   apiVersion: cert-manager.io/v1alpha2
   kind: Certificate
   metadata:
     name: test-ingress
     namespace: default
     labels:
       cert: test-ingress
     annotations:
       cert-manager.io/issue-temporary-certificate: "true"
   spec:
     secretName: test-ingress-cert
     dnsNames:
       - ${DOMAIN}
     issuerRef:
       name: letsencrypt-staging
       kind: ClusterIssuer
   EOF

   If everything is set up correctly, you should receive a certificate in a few minutes. You can check its progress by inspecting the Certificate, CertificateRequest, Order and Challenge resources. Take a look at the status and events sections.

   $ kubectl describe certificate -l cert=test-ingress
   $ kubectl describe certificaterequest -l cert=test-ingress
   $ kubectl describe order -l cert=test-ingress
   $ kubectl describe challenge
   

   Note: as of cert-manager v0.15.1, Challenge resources don't inherit the labels the way other resources do, so it's not as easy to select the challenges that correspond to your Certificate resource.

   Try adding one or more entries (for example foo.< your-domain>) to dnsNames in your Certificate resource, which you have not pointed to your ingress. Challenges for these domains won't be solvable, so you'll have time to inspect the Challenge resources. You can also see the changes it made to your Ingress resource.

   The received certificate will be placed in the secret with key tls.crt and the corresponding private key with key tls.key. You can check it with the following command:

   $ kubectl describe secret test-ingress-cert
   

   Now we are ready to see the issued certificate in action (the --insecure flag is necessary when using the staging provider, which we did):

   $ curl -v --insecure https://${DOMAIN}/get
   ...
   * Server certificate:
   *  subject: CN=<your-domain>
   *  start date: Jul  3 19:57:58 2020 GMT
   *  expire date: Oct  1 19:57:58 2020 GMT
   *  issuer: CN=Fake LE Intermediate X1
   ...

   Keeping cert-manager and your Certificate resources in your cluster will ensure that all your certificates are renewed before expiry.

Wrap-up

It's important to secure your services with TLS; cert-manager makes it easy to do that in a Kubernetes environment. You should use it! If you are interested in certificate management on Istio, check out our next post, Certificate management on Istio as well.