Certificate management on Istio

Recently, we blogged about certificate management on Kubernetes. Today, we'll be returning to that topic, but we'll be focusing on the differences an Istio service mesh makes. The primary difference is the method of solving the ACME HTTP-01 challenge. Solving this challenge involves routing an HTTP request from the ACME server (the Certificate Authority) to the cert-manager challenge solver pod.

Cert management with Istio Ingress support

As we saw in our previous blog post, you can route such a challenge request by using a Kubernetes Ingress gateway. When using Istio, that same method is still viable, provided that Kubernetes Ingress support is enabled in your Istio config. If it's enabled, and you're using Kubernetes Ingress resources for routing the traffic entering your cluster, everything should work just fine. However, you will be limited to using one ingress gateway for all your ingress traffic. This can be a security issue, and it also prevents you from setting up separate ingress gateways for your different teams or products. Additionally, this will cause you to miss out on some Istio features. Considering these limitations, it usually makes sense to use native Istio routing.

Cert management using Istio

Using native Istio features for ingress gives you more freedom and flexibility:

• you can direct traffic for different domains to separate ingress gateways
• you can set up routing for HTTP and HTTPS traffic on non-standard ports
• you also get to use the familiar Istio features (circuit breaker, metrics, etc.)

While technically it's possible to use Kubernetes Ingress and Istio resources together to configure the same Istio ingress gateway, conflicts between them can easily cause difficult to diagnose problems; you should stick to using one or the other for a given ingress gateway. All in all, if you have Istio running in your cluster, it's a good idea to use it for your ingress gateways, too. So, what does this mean in terms of cert management automation? Well, cert-manager doesn't support Istio at the moment. There are solutions, though. When using an ACME issuer with challenge type HTTP-01, you will end up with an Ingress resource for routing the ACME challenge request. If you disabled Kubernetes Ingress support in Istio (or you specified a non-existent ingress class in the Issuer resource), then no ingress controller will react to the Ingress resource, therefore, no HTTP request will be routed to the ACME solver pod(s). What can we do? Two possible solutions: translate the Ingress resource into Gateway and VirtualService resources, or add Istio support to cert-manager. Obviously, adding Istio support to cert-manager would be the perfect solution, but it's a more involved process. Nonetheless, at Banzai Cloud, we're already working on just such an implementation. In fact, we have a functional solution, although it needs some more love before it's production ready (cert-manager#3011).

Translating Ingress resources

While we're waiting for direct Istio support in cert-manager to be released, we can make do with another solution: replicating the Ingress resource with Istio resources. It's possible to do this by hand, but usually the reason we use the ACME protocol is automation, so introducing an error prone and repetitive manual step into that process sort of defeats the point. When properly automated, this can serve as an effective solution, though. The good news is, if you're a Backyards (now Cisco Service Mesh Manager) user, you can try it out right now, because Backyards 1.3 includes this feature. If not, then feel free to use the evaluation version!

---

Want to know more? Get in touch with us, or delve into the details of the latest release. Or just take a look at some of the Istio features that Backyards automates and simplifies for you, and which we've already blogged about.

---

Try it out!

1. open the Backyards dashboard:

   $ backyards dashboard

2. select Gateways from the menu on the left
3. select the gateway you want secured. You can select whichever you like, but make sure the "Service Type" is "LoadBalancer". This selection determines the load balancer and, therefore, the IP address(es) that will be used for the ACME HTTP-01 challenge. In our case, it's istio-ingressgateway.
4. point your domain name to the IP address or DNS name found in the ADDRESS field.
5. in the Ports & Hosts section, click on create new in the upper right corner
6. fill in the fields:
   1. you will probably want HTTPS protocol on the standard port
   2. enter your domain name(s) for hosts. You can enter several by hitting enter after each one.You can also set up a domain under .banzaicloud.io for testing purposes if you don't want to use your own domain name. Just click on the checkbox to the right of use .banzaicloud.io and a new domain name will be generated for you, then add a subdomain under the generated domain name.
   3. click on the checkbox Use Let's Encrypt for TLS to get a certificate for your domain from Let's Encrypt
   4. enter your email address. This address will be for ACME account management, so will be sent to Let's Encrypt.
7. finally, click Create

As you can see, two more items have appeared in the Ports & Hosts list, both for the host frontpage.qahqab.backyards.banzaicloud.io. One of them on port 443, and the other on port 80. The latter one is for solving the ACME HTTP-01 challenge created by translating the Ingress resource. While the certificate is being acquired, you can look up and see the contents of related Ingress, Gateway and VirtualService resources:

1. The Ingress resource created by cert-manager:

   apiVersion: extensions/v1beta1
   kind: Ingress
   metadata:
     annotations:
       kubernetes.io/ingress.class: istio
     labels:
       acme.cert-manager.io/http-domain: "1270878147"
       acme.cert-manager.io/http-token: "1101532839"
       acme.cert-manager.io/http01-solver: "true"
     name: cm-acme-http-solver-nnpg2
     namespace: istio-system
   spec:
     rules:
       - host: frontpage.qahqab.backyards.banzaicloud.io
         http:
           paths:
             - backend:
                 serviceName: cm-acme-http-solver-rrg8c
                 servicePort: 8089
               path: /.well-known/acme-challenge/XMMsPcDx1KkqfG1DE6xsgjyoh9r697I9BXkgwnr_45s

2. The Gateway and VirtualService created by Backyards from the above Ingress resource:

   apiVersion: networking.istio.io/v1beta1
   kind: Gateway
   metadata:
     labels:
       acme.backyards.banzaicloud.io/http-domain: "1270878147"
       acme.backyards.banzaicloud.io/http-token: "1101532839"
       acme.backyards.banzaicloud.io/http01-solver: "true"
     name: cm-acme-http-solver-nnpg2-czvnv
     namespace: istio-system
   spec:
     selector:
       app: istio-ingressgateway
       istio: ingressgateway
     servers:
       - hosts:
           - istio-system/frontpage.qahqab.backyards.banzaicloud.io
         port:
           name: http
           number: 80
           protocol: HTTP

   apiVersion: networking.istio.io/v1beta1
   kind: VirtualService
   metadata:
     labels:
       acme.backyards.banzaicloud.io/http-domain: "1270878147"
       acme.backyards.banzaicloud.io/http-token: "1101532839"
       acme.backyards.banzaicloud.io/http01-solver: "true"
     name: cm-acme-http-solver-nnpg2-tv22w
     namespace: istio-system
   spec:
     exportTo:
       - "*"
     gateways:
       - istio-system/cm-acme-http-solver-nnpg2-czvnv
     hosts:
       - frontpage.qahqab.backyards.banzaicloud.io
     http:
       - match:
           - uri:
               exact: /.well-known/acme-challenge/XMMsPcDx1KkqfG1DE6xsgjyoh9r697I9BXkgwnr_45s
         route:
           - destination:
               host: cm-acme-http-solver-rrg8c.istio-system.svc.cluster.local
               port:
                 number: 8089

As you can see, these two resources describe the same routing path as the Ingress resource. After a short while the item with port 80 and protocol HTTP will disappear, and a green checkmark will appear next to HTTPS. This signals that the certificate was issued and is being used for securing your domain: Let's see if that's really the case:

$ curl -v https://frontpage.qahqab.backyards.banzaicloud.io
...
* Server certificate:
*  subject: CN=frontpage.qahqab.backyards.banzaicloud.io
*  start date: Jul 14 10:42:43 2020 GMT
*  expire date: Oct 12 10:42:43 2020 GMT
*  subjectAltName: host "frontpage.qahqab.backyards.banzaicloud.io" matched cert's "frontpage.qahqab.backyards.banzaicloud.io"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
...
< HTTP/2 404
< date: Tue, 14 Jul 2020 11:44:09 GMT
< server: istio-envoy

As you can see, the connection is secured with a certificate from Let's Encrypt. Did you notice that 404, though? That's because we haven't set up any routing yet. We can do that now, so head over to the routes tab, click create new in the upper right corner and fill in the fields. At the very least, the gateway, host and port number need to be filled in, but, depending on your requirements, you might want to specify other values and settings, too. When you're ready, click Create. Now that routing is set up for our demo service, let's see if it works:

$ curl https://frontpage.qahqab.backyards.banzaicloud.io
frontpage

You can also try it in your browser to inspect the certificate more easily.

Wrap up

As we have seen in this blog post, while cert-manager doesn't directly support Istio, it's possible to make it work. And with Backyards, it's not only possible but seamless, too.