Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
Share
INSIGHTS
12 min read
Share
K8s secrets
, which is the standard way in which applications consume secrets and credentials on Kubernetes. Any secret that is securely stored in Vault and then unsealed for consumption will eventually end up as a K8s secret, and with much less protection and security than we'd like. K8s secrets use base64 encoding that, while better than nothing, does not satisfy our standards, and likely fails to satisfy the standards of most enterprise clients as well. As a result, we've developed a solution wherein we can bypass the K8s secrets mechanism and inject secrets directly into Pods from Vault.
channable/vaultenv
, hashicorp/envconsul
), but one thing that makes it unique is that it is a daemonless solution.
First, the Kubernetes webhook checks if a container has environment variables with values that correspond to a specific schema. Then it reads the values for those variables directly from Vault at start-up:
env: - name:
AWS_SECRET_ACCESS_KEY value:
"vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY"
After that, the init-container is injected into the Pod, and a small binary called vault-env
is attached to it as an in-memory volume. That volume is mounted to all containers with the appropriate environment variable definitions.
The init-container also changes the command
of the container to run vault-env
, instead of running the application directly. vault-env
starts up, connects to Vault (using the Kubernetes Auth method), checks that the environment variables have a reference to a value stored in Vault (vault:secret/....
) and replaces that with a corresponding value from Vault's secrets backend. Afterward, vault-env
executes the original process (with syscall.Exec()
), which uses the secret that was originally stored in Vault.
Using this solution prevents secrets stored in Vault from landing in Kubernetes secrets (and in etcd).
vault-env
was designed to work on Kubernetes, but there's nothing stopping it from being used outside of Kubernetes as well. It can be configured with the standard Vault client's environment variables, since there's a standard Go Vault client underneath.
Currently, the Kubernetes Service Account-based Vault authentication mechanism is used by vault-env
, which requests a Vault token in return for the Service Account of the container it's being injected into. But our implementation is going to change in order to allow the use of the Vault Agent's Auto-Auth feature very soon. This will allow users to request tokens in init-containers with all the authentication mechanisms supported by Vault Agent, so they won't be handcuffed to the Kubernetes Service Account-based method.
kubectl exec
-ing in running containers. If you do so, no one will be able to hijack injected environment variables from a process.
Additionally, there is no persistent connection with Vault, and any Vault token used to read environment variables is flushed from memory before the application starts, in order to minimize attack surface.
# These examples require Helm 3 and kubectl:
# Add the Banzai Cloud Helm repository
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
# Create a namespace for the bank-vaults components called vault-infra
# Namespace labeling is required, because the webhook's mutation is based on label selectors
kubectl create namespace vault-infra
kubectl label namespace vault-infra name=vault-infra
# Install the vault-operator to the vault-infra namespace
helm upgrade --namespace vault-infra --install vault-operator banzaicloud-stable/vault-operator --wait
# Clone the bank-vaults project
git clone git@github.com:banzaicloud/bank-vaults.git
cd bank-vaults
# Create a Vault instance with the operator which has the Kubernetes auth method configured
kubectl apply -f operator/deploy/rbac.yaml
kubectl apply -f operator/deploy/cr.yaml
# Now you have a fully functional Vault installation on top of Kubernetes,
# orchestrated by the `banzaicloud/vault-operator` and `banzaicloud/bank-vaults`.
# Next, install the mutating webhook with Helm into its own namespace (to bypass the catch-22 situation of self mutation)
helm upgrade --namespace vault-infra --install vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook --wait
# Set the Vault token from the Kubernetes secret
# (strictly for demonstrative purposes, we have K8s unsealing in cr.yaml)
export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode)
# Tell the CLI that the Vault Cert is signed by a custom CA
kubectl get secret vault-tls -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
export VAULT_CACERT=$PWD/vault-ca.crt
# Tell the CLI where Vault is listening (the certificate has 127.0.0.1 as well as alternate names)
export VAULT_ADDR=https://127.0.0.1:8200
# Forward the TCP connection from your Vault pod to localhost (in the background)
kubectl port-forward service/vault 8200 &
# Write a secret into Vault, which will be injected as an environment variable
vault kv put secret/accounts/aws AWS_SECRET_ACCESS_KEY=s3cr3t
# Apply the deployment with special environment variables
# It will be mutated by the webhook
kubectl apply -f deploy/test-deployment.yaml
The deployment will be mutated by the webhook, because it has at least one environment variable that has a value that is a reference to a path in Vault. Here's what the original deployment looks like:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-secrets
spec:
replicas: 1
selector:
matchLabels:
app: hello-secrets
template:
metadata:
labels:
app: hello-secrets
annotations:
vault.security.banzaicloud.io/vault-addr: "https://vault:8200"
vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
spec:
serviceAccountName: default
containers:
- name: alpine
image: alpine
command:
[
"sh",
"-c",
"echo $AWS_SECRET_ACCESS_KEY && echo going to
sleep... && sleep 10000",
]
env:
- name: AWS_SECRET_ACCESS_KEY
value: "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY"
It produces Pods like so (only the relevant parts are shown here, Pods are mutated directly):
apiVersion: v1
kind: Pod
metadata:
name: hello-secrets-575554499f-26894
labels:
app: hello-secrets
annotations:
vault.banzaicloud.io/vault-addr: "https://vault:8200"
vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
spec:
initContainers:
- name: copy-vault-env
command:
- sh
- -c
- cp /usr/local/bin/vault-env /vault/
image: banzaicloud/vault-env:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /vault/
name: vault-env
containers:
- name: alpine
command:
- /vault/vault-env
args:
- sh
- -c
- echo $AWS_SECRET_ACCESS_KEY $ && echo going to
sleep... && sleep 10000
image: alpine
imagePullPolicy: Always
env:
- name: AWS_SECRET_ACCESS_KEY
value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
- name: VAULT_ADDR
value: https://vault:8200
- name: VAULT_SKIP_VERIFY
value: "false"
- name: VAULT_PATH
value: kubernetes
- name: VAULT_ROLE
value: default
- name: VAULT_IGNORE_MISSING_SECRETS
value: "false"
- name: VAULT_CACERT
value: /vault/tls/ca.crt
volumeMounts:
- mountPath: /vault/
name: vault-env
- mountPath: /vault/tls/ca.crt
name: vault-tls
subPath: ca.crt
volumes:
- emptyDir:
medium: Memory
name: vault-env
- name: vault-tls
secret:
secretName: vault-tls
As you can see, none of the original environment variables in the definiition have been touched, and the sensitive value of the AWS_SECRET_ACCESS_KEY
variable is only visible inside the alpine
container.
imagePullSecrets
section, or in the Pod's ServiceAccount.
NOTE: Future improvement: on AWS and GKE and other cloud providers get a credential dynamically with the cloud-specific SDK
# Put the MySQL passwords into Vault
vault kv put secret/mysql MYSQL_ROOT_PASSWORD=s3cr3t MYSQL_PASSWORD=3xtr3ms3cr3t
# Install the MySQL chart with root and a user password sourced from Vault
helm upgrade --install mysql stable/mysql \
--set mysqlRootPassword=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD \
--set mysqlPassword=vault:secret/data/mysql#MYSQL_PASSWORD \
--set "podAnnotations.vault\.security\.banzaicloud\.io/vault-addr"=https://vault:8200 \
--set "podAnnotations.vault\.security\.banzaicloud\.io/vault-tls-secret"=vault-tls \
--wait
# Open a connection towards the MySQL Service
kubectl port-forward service/mysql 3306 &
# Read the MySQL user password's secret from Vault
# Make sure you still have the port-forward from the previous example
vault read secret/data/mysql
Key Value
--- -----
data map[MYSQL_PASSWORD:3xtr3ms3cr3t MYSQL_ROOT_PASSWORD:s3cr3t]
metadata map[created_time:2019-09-05T13:03:42.980780517Z deletion_time: destroyed:false version:1]
# Open up the MySQL shell with the root user and its corresponding password from Vault `s3cr3t`
mysql -h 127.0.0.1 -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 84
Server version: 5.7.14 MySQL Community Server (GPL)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
# And here's the magic: there are still no secrets in the env vars!
kubectl exec -it mysql-749cfddc67-5slcb bash
root@mysql-749cfddc67-5slcb:/\# env | grep MYSQL.*PASSWORD
MYSQL_PASSWORD=vault:secret/data/mysql#MYSQL_PASSWORD
MYSQL_ROOT_PASSWORD=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD
Of course, if you exec into the Pod and rerun vault-env
it will fork a new process which will have those environment variables correctly set:
/vault/vault-env env | grep MYSQL.*PASSWORD
2019/09/05 13:42:09 Received new Vault token
2019/09/05 13:42:09 Initial Vault token arrived
MYSQL_PASSWORD=3xtr3ms3cr3t
MYSQL_ROOT_PASSWORD=s3cr3t
"pods/exec"
Kubernetes RBAC rule snippet for users you don't want doing this:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: default
name: pod-execer
rules:
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
apiVersion: v1
kind: Secret
metadata:
name: sample-secret
annotations:
vault.security.banzaicloud.io/vault-addr: "https://vault.default.svc.cluster.local:8200"
vault.security.banzaicloud.io/vault-role: "default" # In case of Secrets the webhook's ServiceAccount
is used
vault.security.banzaicloud.io/vault-skip-verify: "true"
vault.security.banzaicloud.io/vault-path: "kubernetes"
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: eyJhdXRocyI6eyJodHRwczovL2RvY2tlci5pbyI6eyJ1c2VybmFtZSI6InZhdWx0OnNlY3JldC9kYXRhL2RvY
In the example above, the secret type is kubernetes.io/dockerconfigjson, and the webhook is capable of getting credentials from vault. The base64 encoded data contains vault paths for usernames and passwords for docker repositories. You can create it with the following commands:
kubectl create secret docker-registry dockerhub --docker-username="vault:secret/data/dockerrepo#DOCKER_REPO_USER" --docker-password="vault:secret/data/dockerrepo#DOCKER_REPO_PASSWORD"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-addr="https://vault.default.svc.cluster.local:8200"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-role="default"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-skip-verify="true"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-path="kubernetes"
vault-env
supports reading Values from the KV backend, but we have added support for dynamic secrets as well - database URLs with temporary usernames and passwords for batch or scheduled jobs, for example. This feature is implemented with consul-template's Vault component and is based on the work of Jürgen Weber. It deserved its own blog-post, and is described in detail in Vault webhook - complete secret support with consul-template.
integrated service
(we used to call them posthooks) of the Pipeline platform, which means that you can install it on a Kubernetes clusters via Pipeline (with the UI or with the CLI). It will then configure Vault with the authentications and policies necessary for it to work with our webhook - track this issue here.
For more information, or if you're interested in contributing, check out the Bank-Vaults repo - the Vault Swiss army knife and operator for Kubernetes, and/or give us a GitHub star if you think the project deserves it!
Learn more about Bank-Vaults:
- Secret injection webhook improvements
- Backing up Vault with Velero
- Vault replication across multiple datacenters
- Vault secret injection webhook and Istio
- HSM support
- Injecting dynamic configuration with templates
- OIDC issuer discovery for Kubernetes service accounts
- Show all posts related to Bank-Vaults
Get emerging insights on emerging technology straight to your inbox.
Discover why security teams rely on Panoptica's graph-based technology to navigate and prioritize risks across multi-cloud landscapes, enhancing accuracy and resilience in safeguarding diverse ecosystems.
The Shift is Outshift’s exclusive newsletter.
Get the latest news and updates on cloud native modern applications, application security, generative AI, quantum computing, and other groundbreaking innovations shaping the future of technology.