Istio 1.8 has just been released and is one of the best Istio releases so far. The new version contains exciting experimental features, numerous enhancements, as well as deprecations and removals. The core focus of the release, however, is to increase operational stability. In this post, we'll review what's new in Istio 1.8, and highlight a few potential snags to look out for when upgrading to the latest version.

If you're looking for a production ready Istio distribution based on the new Istio 1.8 release and with support for a safe and automated canary > > upgrade flow, check out the new Backyards > > 1.5 release.

Istio 1.8 changes

Caveat: The Istio 1.8.0 release has a few known issues pointed out in the official change notes. Either make sure you won't be affected by them or wait a few patch releases before upgrading your cluster to make sure these issues are sorted out.

First of all, it's important that we point out that the supported Kubernetes versions for Istio 1.8 are 1.16, 1.17, 1.18 and 1.19. If you are running Istio 1.7 in your environment, you should already be on at least Kubernetes 1.16 (as that is also the oldest supported K8s version for Istio 1.7). As a result, you should be ready to upgrade to Istio 1.8 without having to upgrade your K8s cluster. High impact changes (can cause issues when upgrading the mesh):

• Inbound cluster name format changed
• Protocol detection timeout disabled by default
• AuthorizationPolicy has new remoteIpBlocks/notRemoteIpBlocks fields
• Trust Domain Validation enforced by default

Removals/deprecations:

• Completely removes Mixer
• Removes support to install add-ons with istioctl
• Istio CoreDNS Plugin Deprecated
• Envoy filter names deprecated

Experimental features:

• Smart DNS proxying
• Auto registration for VMs
• Integrates with external CAs with K8s CSR API

Under the radar changes (smaller changes, but which we believe are interesting/useful):

• Istiod to handle gateway certificates
• EnvoyFilter enhancements
• Debug archive option added

Other notable changes:

• Helm3 Istio installation
• Observed generation added for Istio resource statuses
• Wasm Grafana dashboards added
• Metrics properly labeled when destination not reached

For the full list of changes please refer to the change notes.

High impact changes

Previously, the Envoy inbound cluster name format looked like this: inbound|<service_port_number>|<service_port_name>|<service_hostname>. An example cluster name with this format was: inbound|80|http|httpbin.default.svc.cluster.local. For Istio 1.8.0 this format has changed, the service port name and the service's hostname are now omitted, the new format looks like this: inbound|<service_port_number>||. For the example above the new format is simply: inbound|80|| There were issues when multiple services selected the same container port in the same pod and this change is an attempt to solve those issues. The offical release note states that: "For most users, this is an implementation detail, and will only impact debugging or tooling that directly interacts with Envoy configuration." Well, from what we've seen so far this change causes undesirable behaviour in the following scenario. When there are two services, which use the same service port number and select the same pod, but they are targeting different port numbers inside the pod. Here's an example when a backend and a frontend container are in the same pod:

apiVersion: v1
kind: Service
metadata:
  name: backyards-web
  namespace: backyards-system
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http-web
  selector:
    app.kubernetes.io/name: backyards
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  name: backyards
  namespace: backyards-system
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http-app
  selector:
    app.kubernetes.io/name: backyards
  type: ClusterIP

With this setup you'll see the following warning in the istiod logs when using the upstream docker.io/istio/pilot:1.8.0 docker image:

"pilot_duplicate_envoy_clusters": {
        "inbound|80||": {
            "proxy": "backyards-8cdfc77b5-4jw44.backyards-system",
            "message": "Duplicate cluster inbound|80|| found while pushing CDS"
        }
    }

And the issue will be that when calling either the backyards-web or the backyards service that in both cases the same target port will be called inside the pod. So it will be impossible to reach the other container port through its service, which is definitely undesired behaviour and seems to be a bug. We're planning to open an issue about this with more details to sort out this issue upstream. A workaround for this problem can be to use different service port numbers to avoid such conflicts. What we did in Backyards is that we reverted this pilot change in our own docker image, which we use in Backyards (now Cisco Service Mesh Manager), so that this issue can never happen.

UPDATE: The upstream issue can be tracked here. Kudos to John Howard, who picked up the issue and fixed it very quickly. The patch should already be in the Istio 1.8.1 release.

In the past, Istio introduced a detection timeout for certain "server first" protocols, such as the protocol used by MySQL, which triggers when there are no initial bytes of traffic to sniff. This protocol detection timeout has been disabled by default to avoid potential traffic failures on slow connections or in certain cases increased latency. If you have misconfigured "server first" protocols, this change will cause immediate connection timeouts. To avoid issues involving "server first" protocols, make sure to explicitly specify the protocol for your service through one of the following methods:

• By the name of the port: name: <protocol>[-<suffix>].
• In Kubernetes 1.18+, by the appProtocol field: appProtocol: <protocol>.

Here are examples of two possible declarations:

kind: Service
metadata:
  name: httpbin
spec:
  ports:
    - number: 443
      name: https-web
    - number: 3306
      name: db
      appProtocol: mysql

Trust Domain Validation enforced by default

By default, the Trust Domain Validation feature will reject any incoming requests to sidecars, if the request is neither from the same trust domain nor in the TrustDomainAliases list. If you want to allow traffic to your mesh from different trust domains on Istio 1.8, you need to add it to your TrustDomainAliases list, otherwise it will be rejected.

Removals/deprecations

Mixer has been the control plane component to which application sidecar proxies reported their telemetry data since the early days of Istio. Mixer had the potential to cause resource consumption and latency issues in larger environments, which is why a new Mixerless telemetry model was introduced:

• first, as an experimental feature in Istio 1.3
• then in Istio 1.4 the feature was made available under the name Telemetry V2
• in Istio 1.5 Telemetry V2 was turned on by default
• in Istio 1.6 the original Mixer-based telemetry model was deprecated in favor of the new Telemetry V2 model

Now, in Istio 1.8, all Mixer-related features and functionality have been removed. So, if you haven't already, you need to adapt to the Telemetry V2 model before upgrading to 1.8.

Remove support to install add-ons with istioctl

Open-source software that enhances Istio's capabilities (like Prometheus, Grafana, Zipkin, Jaeger, and Kiali) can no longer be installed with istioctl, and it is recommended that they be installed separately.

If you want the easiest way to install all these integrated components - with a production-ready setup and with many additional features - check out Backyards (now Cisco Service Mesh Manager).

The Istio sidecar proxy now provides native support for DNS resolution with ServiceEntries, which is why the Istio CoreDNS plugin is deprecated in Istio 1.8 and will be removed in a future release. For the EnvoyFilter API, it is recommended that you use the new Envoy filter names, as some filter names were deprecated and will be removed in a future release.

Experimental features

DNS is essential for any cloud native application, but the way it works in Kubernetes results in a few challenges in Istio:

• It can be difficult for VMs outside the Kubernetes cluster to provide DNS resolution to route services inside the cluster
• If services do not have unique virtual IPs and are on the same port (e.g. databases), then DNS servers cannot distinguish between them
• In multi-cluster meshes we cannot resolve the IP addresses of a service on another cluster (this is why stub services are likewise created on remote clusters)

To solve all these issues for Istio 1.8, there is a DNS proxy implemented in the Istio sidecar agent, written in GO. This DNS proxy is dynamically updated by Istiod (based on services and ServiceEntries on the cluster) and it acts as a cache. It will try to resolve the required hostname based on its cache, and only turn to external DNS servers when the cache does not have the hostname registered. This is an exciting new feature, which can be tried right now, but is not enabled by default. To read an in-depth explanation on this new feature, make sure to check out this blog post. To register VMs for the Istio mesh, so far the only way was to manually create a WorkloadEntry resource. This was previously a manual step which registered new VMs, in contrast to how the same process was automated for nodes in a Kubernetes cluster. It could cause issues, for example, in case of auto scaling VMs, a circumstance in which it's is not easy to register new VMs automatically. There is a pre-alpha feature that solves this problem, making it possible to auto register VMs. This is done by a new WorkloadGroup resource, which is responsible for automatically creating the WorkloadEntry resources when new VMs are available. The Istio 1.8 release laid the groundwork for a wide variety of features around VM support, which should become apparent in future releases as it matures. Beginning with Kubernetes 1.18, there is a CSR API feature, which automates the request and retrieval of certificates from a Certificate Authority (CA). In Istio 1.8, experimental support has been added to allow Istio to integrate with external CAs, using the Kubernetes CSR API. This feature only lays the foundation necessary to work with external CAs, now only istiod is supported (and Google CA somewhat), but new implementations should come in future releases as this feature matures both on the Istio and Kubernetes side.

Under the radar changes

Previously, the gateways read the certificates for external communications from Kubernetes secrets. These gateways were usually public facing entities, and, if they were compromised, the certificates could be read from the Kubernetes secrets that compromised the whole cluster. In Istio 1.8 gateways fetch the certs from Istiod. They do not even have the privileges they need to read secrets from the cluster to reduce potential collateral damage. This feature is interesting because it paves the way for us to be able to fetch certificates from external secret stores (e.g. Vault or cloud key stores) in future releases, which would be convenient for many Istio users already using other secret stores. Furthermore, this is a totally backward compatible change, and does not require any user interaction. To specify the order of HTTP_ROUTE specs for the Envoy configurations,  new  INSERT_FIRST, INSERT_BEFORE, INSERT_AFTER operations were added to the EnvoyFilter API. To be able to override http and network filters, a new REPLACE operation was added to the EnvoyFilter API. A new istioctl bug-report command was added to be able to get an archive of the Istio cluster - mainly for debugging purposes.

Other notable changes

Almost two years ago now, when we open sourced the Banzai Cloud Istio operator, we did so partly because the only option for installing Istio was through Helm, which had its issues. After a while, istioctl, and then the official Istio operator, were introduced, and, slowly, the method of installing by Helm became unsupported. It turned out, however, that there was still demand among Istio users for a way of installing and upgrading Istio with Helm (mostly for users who had already deployed all their apps with Helm), and because of that, Helm 3 support was added for Istio 1.8. As a result, it is possible to install and upgrade Istio with Helm 3, but I would note here that this is NOT recommended. Only in-place upgrades are supported with Helm, whilst the canary upgrade model is the recommended flow today. Since the 1.6 release, there has been an interesting alpha feature in Istio, in which various useful pieces of information about Istio resources are exposed in their Status field. These fields include, but are not limited to resource readiness, the number of data plane instances associated with the resource, and validation messages. In Istio 1.8 a new observed generation field is also present, which, when it matches the generation in the resource's metadata, indicates that all Istio updates have been completed. This is useful for detecting when requested changes to an Istio configuration have been served and are ready to receive traffic. As WebAssembly (Wasm) gains more and more traction in Istio, it is also gradually becoming required in order to monitor it properly. Therefore, it can come handy to use the newly added Grafana dashboard on Wasm related metrics. Previously, there were cases in which requests did not reach a destination proxy (e.g. fault injection, circuit breaking, no proxy injected etc.), and there were no destination labels to identify what would have been the end target of the request. This is remedied in 1.8, which is a welcome bug fix.

Takeaway

Istio 1.8 largely increases operational stability and also lays the foundation for exciting new features in future releases. If you’d like to kickstart your Istio experience, try out Backyards (now Cisco Service Mesh Manager), our Istio distribution. Backyards (now Cisco Service Mesh Manager) operationalizes the service mesh to bring deep observability, convenient management, and policy-based security for modern container-based applications.

Want to know more? Get in touch with us, or delve into the details of the latest release. Or just take a look at some of the Istio features that Backyards automates and simplifies for you, and which we've already blogged about.