❯ kubectl -n istio-system get istio mesh

NAME   STATUS      ERROR   GATEWAYS                        AGE
mesh   Available           [18.184.240.108 18.196.72.62]   15m

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: frontpage
  namespace: backyards-demo
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "frontpage.18.184.240.108.xip.io"
        - "frontpage.18.196.72.62.xip.io"

Create VirtualService resource

The following VirtualService resource configures routing for the external hosts within the mesh.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: frontpage
  namespace: backyards-demo
spec:
  hosts:
    - "frontpage.18.184.240.108.xip.io"
    - "frontpage.18.196.72.62.xip.io"
  gateways:
    - frontpage
  http:
    - route:
        - destination:
            port:
              number: 8080
            host: frontpage.backyards-demo.svc.cluster.local

 

Access the service on the external address

Try to access the service on the external address you just configured, on host frontpage.18.184.240.108.xip.io. If everything is set correctly, the following command will return an HTTP 200 status code.

❯ curl -i frontpage.18.184.240.108.xip.io

HTTP/1.1 200 OK
content-type: text/plain
date: Sun, 02 Feb 2020 19:15:14 GMT
content-length: 9
x-envoy-upstream-service-time: 16
server: istio-envoy

frontpage

Create and use multiple ingress gateways

Having one ingress and egress gateway to handle incoming and outgoing traffic from the mesh is part of a basic Istio installation and has been supported by the Banzai Cloud Istio operator from day one, but in large enterprise deployments our customers typically use Backyards (now Cisco Service Mesh Manager) with multiple ingress or egress gateways.

The Banzai Cloud Istio operator and multiple gateways

The Banzai Cloud Istio operator has an Istio custom resource that defines mesh configurations. The main ingress/egress gateways are part of the specifications of that resource.

spec:
  ...
  gateways:
    egress:
      enabled: true
      maxReplicas: 1
      minReplicas: 1
      ports:
      - name: http2
        port: 80
        protocol: TCP
        targetPort: 80
      - name: https
        port: 443
        protocol: TCP
        targetPort: 443
      replicaCount: 1
      sds:
        enabled: false
        image: docker.io/istio/node-agent-k8s:1.4.3
      serviceType: ClusterIP
    enabled: true
    ingress:
      enabled: true
      maxReplicas: 1
      minReplicas: 1
      ports:
      - name: status-port
        port: 15020
        protocol: TCP
        targetPort: 15020
      - name: http2
        port: 80
        protocol: TCP
        targetPort: 80
      - name: https
        port: 443
        protocol: TCP
        targetPort: 443
      - name: tls
        port: 15443
        protocol: TCP
        targetPort: 15443
      replicaCount: 1
      sds:
        enabled: false
        image: docker.io/istio/node-agent-k8s:1.4.3
      serviceType: LoadBalancer
    ...

One way to support multiple gateways would have been to add support for specifying them in the existing custom resource. But we chose a radically different approach for the following reasons:

• a separate controller should reconcile gateways, as there could be multiple gateways in multiple namespaces
• RBAC: having a separate CR allows us to properly control who can manage gateways, without having permissions to modify other parts of the Istio mesh configuration

Thus, we have added a new CRD to the Banzai Cloud Istio operator, called the MeshGateway, that can be used to add and configure a new Istio ingress or egress gateway into the mesh. When you create a new MeshGateway CR, the Banzai Cloud Istio operator will take care of configuring and reconciling the necessary resources, including the Envoy deployment and its related Kubernetes service.

Create a new service in the default namespace and expose it

To demonstrate how to create and use multiple ingress gateways, let's add a simple service to the default namespace.

❯ kubectl apply -f https://raw.githubusercontent.com/banzaicloud/istio-operator/release-1.4/docs/federation/multimesh/echo-service.yaml

deployment.extensions/echo created
service/echo created

It would be possible to expose this echo service through the existing ingress gateway, similar to the way we would for the frontpage service, but let's assume we need to expose this service on port 8000, without modifying the existing ingress gateway.

Create a new ingress gateway using the MeshGateway resource

Apply the following resource and the operator will create a new ingress gateway deployment, and a corresponding service.

apiVersion: istio.banzaicloud.io/v1beta1
kind: MeshGateway
metadata:
  name: echo-ingress
  namespace: default
spec:
  maxReplicas: 1
  minReplicas: 1
  ports:
    - name: http
      port: 8000
      protocol: TCP
      targetPort: 8000
  serviceType: LoadBalancer
  type: ingress

The MeshGateway resource automatically labels the created Service and Deployment resources with the gateway-name and gateway-type labels and their corresponding values.

Get the echo ingress gateway IP

❯ kubectl -n default get meshgateways echo-ingress

NAME           TYPE      SERVICE TYPE   STATUS      INGRESS IPS                     ERROR   AGE
echo-ingress   ingress   LoadBalancer   Available   [18.185.173.38 18.197.110.20]           29m

Create Gateway and VirtualService resources

Just like in the first example, the following Gateway and VirtualService resources are necessary to configure listening ports on the matching gateway deployment.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: echo
  namespace: default
spec:
  selector:
    gateway-name: echo-ingress
    gateway-type: ingress
  servers:
    - port:
        number: 8000
        name: http
        protocol: HTTP
      hosts:
        - "echo.18.197.110.20.xip.io"
        - "echo.18.185.173.38.xip.io"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: echo
  namespace: default
spec:
  hosts:
    - "echo.18.197.110.20.xip.io"
    - "echo.18.185.173.38.xip.io"
  gateways:
    - echo
  http:
    - route:
        - destination:
            port:
              number: 80
            host: echo.default.svc.cluster.local

Access the echo service on the external address

The service should be accessible on host echo.18.197.110.20.xip.io and port 8000.

❯ curl -i echo.18.197.110.20.xip.io:8000

HTTP/1.1 200 OK
date: Sun, 02 Feb 2020 19:22:15 GMT
content-type: text/plain
server: istio-envoy
x-envoy-upstream-service-time: 1

Hostname: echo-68578cf9d9-874rz
...

Our ability to easily create ingress gateways gives you fine-grained control over how services are exposed to the outside world. That way, teams can manage the exposure of their own services without running the risk of misconfiguring the services of other teams. Another way of tackling this potential issue is to have separate load balancer configurations with, for example, different port level settings.

Accessing External Services

Direct outbound traffic

Any traffic that's outbound from a pod with an Istio sidecar will also pass through that sidecar's container, or, more precisely, through Envoy. Therefore, the accessibility of external services depends on the configuration of that Envoy proxy. By default, Istio configures the Envoy proxy to passthrough requests for unknown services. Although this provides a convenient way of getting started with Istio, its generally a good idea to put stricter controls in place.

To read more about the Sidecar object configuration, check out this informative blog post:.

Check the default outbound traffic policy

❯ kubectl -n istio-system get istio mesh -o jsonpath='{@.spec.outboundTrafficPolicy.mode}{"\n"}'

ALLOW_ANY

This traffic policy should be set to ALLOW_ANY by default.

Make an HTTP request to httpbin.org from the echo service container in the default namespace

❯ kubectl -n default exec -ti deploy/echo -- curl -sD /dev/stderr http://httpbin.org -o/dev/null |head -5

Defaulting container name to echo-service.
Use 'kubectl describe pod/echo-68578cf9d9-874rz -n default' to see all of the containers in this pod.
HTTP/1.1 200 OK
date: Sun, 02 Feb 2020 20:43:40 GMT
content-type: text/html; charset=utf-8
content-length: 9593
server: envoy

This should work fine, since, by default, every sidecar sends traffic towards unknown services through its passhtrough proxy.

Change the default outbound traffic policy to block unknown services

Now we're going to demonstrate a more controlled way of enabling access to external services. Change the spec.outboundTrafficPolicy.mode option from the ALLOW_ANY mode to the REGISTRY_ONLY mode in the mesh Istio resource in the istio-system namespace.

❯ kubectl -n istio-system patch istio mesh -p '{"spec":{"outboundTrafficPolicy":{"mode":"REGISTRY_ONLY"}}}' --type='merge'

istio.istio.banzaicloud.io/mesh patched

Make another HTTP request to httpbin.org from the echo service container in the default namespace

❯ kubectl -n default exec -ti deploy/echo -- curl -sD /dev/stderr http://httpbin.org -o/dev/null |head -5

HTTP/1.1 502 Bad Gateway
location: http://httpbin.org/
date: Sun, 02 Feb 2020 20:53:44 GMT
server: envoy
content-length: 0

Now we're getting a 502 response code, since now the traffic towards external services is blocked and it is going through Envoy's blackhole cluster.

Create a ServiceEntry to allow HTTP access to httpbin.org

ServiceEntry resources enable adding additional entries into Istio’s internal service registry, so that auto-discovered services in the mesh can access/route to these manually specified services. A service entry describes the properties of a service (DNS name, VIPs, ports, protocols, endpoints). These services could be external to the mesh (for example, web APIs) or mesh-internal services that are not part of the platform’s service registry. For more information about the ServiceEntry resource, see the Istio documentation.

Apply the following ServiceEntry to allow for HTTP access to httpbin.org

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-ext
  namespace: default
spec:
  hosts:
    - httpbin.org
  ports:
    - number: 80
      name: http
      protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL

Make another HTTP request to httpbin.org from the echo service container in the default namespace

❯ kubectl -n default exec -ti deploy/echo -- curl -sD /dev/stderr http://httpbin.org -o/dev/null |head -5

Defaulting container name to echo-service.
Use 'kubectl describe pod/echo-68578cf9d9-874rz -n default' to see all of the containers in this pod.
HTTP/1.1 200 OK
date: Sun, 02 Feb 2020 21:13:40 GMT
content-type: text/html; charset=utf-8
content-length: 9593
server: envoy

Outbound traffic through an egress gateway

As you probably recall from earlier in this blogpost, egress gateways are exit points from the mesh that allow us to apply Istio features. This includes applying features like monitoring and route rules to traffic that's exiting the mesh.

Use cases

Let's take a quick look at some use cases. Consider an organization which requires some, or all, outbound traffic to go through dedicated nodes. These nodes could be separated from the rest of the nodes for the purposes of monitoring and policy enforcement. Now imagine a cluster where the application nodes don’t have public IPs, so the in-mesh services that run on them cannot access the internet directly. Defining an egress gateway and routing egress traffic through it, then allocating public IPs to the gateway nodes would allow for controlled access to external services.

Create an egress gateway with the MeshGateway resource

Apply the following resource and the Istio operator will create a new egress gateway deployment and a corresponding service.

apiVersion: istio.banzaicloud.io/v1beta1
kind: MeshGateway
metadata:
  name: egress
  namespace: default
spec:
  maxReplicas: 1
  minReplicas: 1
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
  serviceType: ClusterIP
  type: egress

Create a Gateway resource for the egress gateway

Similar to the ingress gateway configuration, a Gateway resource must be created that will be a bridge between Istio configuration resources and the deployment of a matching gateway. Apply the following Gateway resource to configure the outbound port, 80, on the egress gateway that was just defined in the previous step.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: egress
  namespace: default
spec:
  selector:
    gateway-name: egress
    gateway-type: egress
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

Define a VirtualService resource to direct traffic from the sidecars to the egress gateway

Apply the following VirtualService to direct traffic from the sidecars to the egress gateway and also from the egress gateway to the external service.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-egress
spec:
  hosts:
    - httpbin.org
  gateways:
    - egress
    - mesh
  http:
    - match:
        - gateways:
            - mesh
          port: 80
      route:
        - destination:
            host: egress.default.svc.cluster.local
            port:
              number: 80
    - match:
        - gateways:
            - egress
          port: 80
      route:
        - destination:
            host: httpbin.org
            port:
              number: 80

Resend an HTTP request to httpbin.org from the echo service container in the default namespace

❯ kubectl -n default exec -ti deploy/echo -- curl -sD /dev/stderr http://httpbin.org/headers -o/dev/null |head -5

Defaulting container name to echo-service.
Use 'kubectl describe pod/echo-68578cf9d9-874rz -n default' to see all of the containers in this pod.
HTTP/1.1 200 OK
date: Sun, 02 Feb 2020 22:20:28 GMT
content-type: application/json
content-length: 1850
server: envoy

Output should be the same as earlier, but if we check the logs of the egress gateway, it shows that the request actually went through the egress gateway.

❯ kubectl -n default logs -l gateway-name=egress,gateway-type=egress -c istio-proxy | tail -1

[2020-02-02T22:20:29.191Z] "GET /headers HTTP/1.1" 200 - "-" "-" 0 1850 173 173 "10.20.5.1" "curl/7.47.0" "bf26bf45-4a0b-4d78-a968-98515a890a91" "httpbin.org" "52.202.2.199:80" outbound|80||httpbin.org - 10.20.5.16:80 10.20.5.1:58584 - -

Takeaway

Ingress and egress gateways are core concepts of a service mesh. Although Istio itself provides the basic building blocks, having an easy and simple way to create and manage multiple mesh gateways is a must. The Banzai Cloud Istio operator provides support with a new CRD called MeshGateway. Give it a try, and quickstart your Istio experience with Backyards (now Cisco Service Mesh Manager)!

About Banzai Cloud Istio operator

Banzai Cloud Istio operator is a simple way to deploy, manage and maintain Istio service meshes, even in multi-cluster topologies.

About Backyards

Banzai Cloud’s Backyards (now Cisco Service Mesh Manager) is a multi and hybrid-cloud enabled service mesh platform for constructing modern applications. Built on Kubernetes and our Istio operator, it gives you flexibility, portability, and consistency across on-premise datacenters and cloud environments. Use our simple, yet extremely powerful UI and CLI, and experience automated canary releases, traffic shifting, routing, secure service communication, in-depth observability and more, for yourself.

About Banzai Cloud

Banzai Cloud is changing how private clouds are built: simplifying the development, deployment, and scaling of complex applications, and putting the power of Kubernetes and Cloud Native technologies in the hands of developers and enterprises, everywhere. #multicloud #hybridcloud #BanzaiCloud