Create Multi-cluster Istio Service Meshes The Easy Way

Two months ago we announced the release of Backyards (now Cisco Service Mesh Manager), Banzai Cloud's multi- and hybrid-cloud enabled service mesh built on top of our Istio operator. One of [Backyards'] hallmarks is its ability to simplify building a production-ready Istio deployment down to a single command: backyards install -a - complete with enterprise grade security, monitoring, tracing, logs, audit, and features like canary releases, traffic management, circuit breaking and lots more, either through a convenient UI, CLI or a GraphQL API. Nevertheless, one feature was missing from Backyards (now Cisco Service Mesh Manager): the option to build an Istio service mesh that spans multiple clusters. While the Banzai Cloud open source Istio operator has long since supported such a feature (check out the Istio operator multi-cluster scenarios documentation), it was missing from Backyards (now Cisco Service Mesh Manager).

Want to know more? Get in touch with us, or delve into the details of the latest release. Or just take a look at some of the Istio features that Backyards automates and simplifies for you, and which we've already blogged about.

Previously, we have also made multi-cluster deployments, service meshes, federation and other features available in Pipeline, Banzai Cloud's container management platform for building multi- and hybrid-clouds. We are happy to announce that multi-cluster management will be baked into the next major version of Backyards (now Cisco Service Mesh Manager). And, in this post, we are going to go into detail about just how easy it will be to manage a multi-cluster service mesh with Backyards.

Typical multi-cluster-based patterns are single mesh - combine multiple clusters into one unit managed by one Istio control plane. Mesh federation is when multiple clusters act as individual management domains, and the service exposure between those domains is done selectively. Single mesh scenarios are best suited to use cases wherein clusters are configured together, sharing resources and are generally treated as one infrastructural component within an organization.

Showtime

Install Backyards CLI

Register for an evaluation version and run the following command to install the CLI tool: Register for the free tier version of Cisco Service Mesh Manager (formerly called Banzai Cloud Backyards) and follow the Getting Started Guide for up-to-date instructions on the installation.

Create two clusters

For this demo we'll need two Kubernetes cluster.

I created two Kubernetes cluster on AWS, using Banzai Cloud's lightweight, CNCF-certified Kubernetes distribution, PKE, via the Pipeline platform. If you'd like to do likewise, go ahead and create your clusters on any of the several cloud providers we support, or on-premise, using Pipeline for free.

Install Backyards to one of the clusters

In a typical single mesh scenario, a single Istio control plane exists on a cluster that receives information about service and pod states from its peers. To accomplish this, the kubeconfig of each peer cluster must be added to the cluster where the control plane is running, in the form of a k8s secret. The following command will install Backyards and deploy a service mesh to the selected cluster.

❯ backyards install -a
INFO[0002] customresourcedefinition.apiextensions.k8s.io:istios.istio.banzaicloud.io configured
INFO[0002] customresourcedefinition.apiextensions.k8s.io:remoteistios.istio.banzaicloud.io configured
INFO[0002] customresourcedefinition.apiextensions.k8s.io:istios.istio.banzaicloud.io - pending
INFO[0002] customresourcedefinition.apiextensions.k8s.io:istios.istio.banzaicloud.io - ok
...
...
NFO[0038] gateway.networking.istio.io:backyards-system/backyards-ingressgateway - pending
INFO[0038] gateway.networking.istio.io:backyards-system/backyards-ingressgateway - ok
INFO[0038] virtualservice.networking.istio.io:backyards-system/backyards-ingressgateway - pending
INFO[0038] virtualservice.networking.istio.io:backyards-system/backyards-ingressgateway - ok

The status of the mesh can be checked via the following commands.

❯ backyards istio overview
Mesh overview – metrics time span 60 seconds

Clusters  Services  in mesh  Workloads  in mesh  Pods  in mesh  Error rate  Latency  RPS
1         30        4        33         3        46    3        -1          0.01075  0

❯ backyards istio cluster status
Clusters in the mesh

Name            Type  Status       Gateway Address               Message
mesh            Host  Available    [13.48.172.40 13.53.167.219]

Attach a peer cluster to the mesh

A peer cluster is any participant cluster in a single mesh. Backyards (now Cisco Service Mesh Manager) automates the process of creating the resources necessary for the peer cluster, generates and sets up the kubeconfig for that cluster, and attaches the cluster to the mesh. The only other thing we need to do is make sure the kubeconfig for the peer cluster has the requisite RBAC permissions.

❯ backyards istio cluster attach ~/kubeconfigs/waynz0r-by-114.yaml
? Are you sure to use the following context? kubernetes-admin@waynz0r-by-114 (API Server: https://13.48.216.202:6443) Yes
INFO[0002] creating service account and rbac permissions
INFO[0002] namespace:istio-system created
INFO[0002] serviceaccount:istio-system/istio-operator created
INFO[0002] clusterrole.rbac.authorization.k8s.io:istio-operator configured
INFO[0002] clusterrolebinding.rbac.authorization.k8s.io:istio-operator configured
INFO[0002] retrieving service account token
INFO[0002] attaching cluster 'waynz0r-by-114' is started successfully. Use `backyards istio cluster status` to follow the progress.

It may take some time to attach the peer cluster, because it needs the ingress gateway address to work

Check the status of the mesh with the following command:

❯ backyards istio overview
Mesh overview – metrics time span 60 seconds

Clusters  Services  in mesh  Workloads  in mesh  Pods  in mesh  Error rate  Latency  RPS
2         30        4        37         3        77    3        -1          0.00475  0

❯ backyards istio cluster status
Clusters in the mesh

Name            Type  Status       Gateway Address               Message
mesh            Host  Available    [13.48.172.40 13.53.167.219]
waynz0r-by-114  Peer  Available    [13.48.15.76 13.48.198.212]

Backyards (now Cisco Service Mesh Manager) comes with a built-in demo application for demonstration purposes. As there are multiple clusters in the mesh, the microservices that compose the mess should span these clusters. The following command will deploy some of the services onto the host cluster:

❯ backyards demoapp install -s frontpage,catalog,bookings
INFO[0001] namespace:backyards-demo created
INFO[0001] service:backyards-demo/analytics created
INFO[0001] service:backyards-demo/bookings created
...
...
INFO[0017] virtualservice.networking.istio.io:backyards-demo/movies - pending
INFO[0017] virtualservice.networking.istio.io:backyards-demo/movies - ok

The rest of the application can be deployed to the peer cluster via the following command:

❯ backyards -c ~/kubeconfigs/waynz0r-by-114.yaml demoapp install -s movies,payments,notifications,analytics --peer
INFO[0001] namespace:backyards-demo created
INFO[0001] service:backyards-demo/analytics created
INFO[0001] service:backyards-demo/bookings created
INFO[0001] service:backyards-demo/catalog created
...
...
INFO[0011] deployment.apps:backyards-demo/notifications-v1 - pending
INFO[0016] deployment.apps:backyards-demo/notifications-v1 - ok
INFO[0016] deployment.apps:backyards-demo/payments-v1 - pending
INFO[0016] deployment.apps:backyards-demo/payments-v1 - ok

Backyards (now Cisco Service Mesh Manager) has a built-in load tester tool, which you can use to seamlessly generate traffic to the demo application. After the installation of each component has finished, send some traffic and open the Backyards UI. You should be able to see that communication is taking place between the microservices of the demo applications that span the two clusters.

❯ backyards demoapp load
INFO[0000] Sending load to demo application              duration=30 rps=10
INFO[0030] loader stopped
INFO[0030]                                               requestCount=300 responseCode=200

❯ backyards dashboard
INFO[0001] Logged in as kubernetes-admin
INFO[0001] Opening Backyards UI at http://127.0.0.1:50500

Cleanup

❯ backyards -c ~/kubeconfigs/waynz0r-by-114.yaml demoapp uninstall
❯ backyards istio cluster detach waynz0r-by-114
❯ backyards uninstall -a

Wrap-up

We still believe that, while it may be hard to navigate the hype, expanding marketplace, and increasing complexity that surrounds service mesh, it's one of the next big things. Our intention is to inject some clarity into this situation by providing a product that leverages and integrates everything our customers need, and which will make the adoption and use of the service mesh as easy as possible. So try out Backyards (now Cisco Service Mesh Manager) today by downloading it here. While you're at it, we'd love if you gave us feedback on what features you'd like to see, and what we should improve.