Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
Share
INSIGHTS
7 min read
Share
If you are familiar with Bank-Vaults and its offerings (if not, we recommend to look around on our Github and the docs page), you know that one of its popular features is the ability to inject secrets from Hashicorp Vault directly into Kubernetes Pods via our mutating webhook. This way you can avoid the use of Kubernetes Secrets if you consider them a security risk, or just want to skip the steps of creating and mounting secrets to containers to provide the right environment variables and other sensitive or configuration data.
However, vault-secrets-webhook can only perform its secret-injection during pod creation. This poses a challenge: what happens when the secret it relies on changes in Vault? Up until now, the Bank-Vaults ecosystem lacked an automated solution to update workloads when a Vault secret is modified.
This is where our new Vault Secrets Reloader project comes into play! With its recent alpha release, it provides an easily configurable Kubernetes Controller that can trigger a new rollout for watched workloads if a secret they use has an updated version in Vault, leaving the rest of the work to the Webhook. The frequency of collecting and reloading can be configured separately depending on your requirements in the current design of the Reloader. You can find a more detailed description of its current features and limitations in the project's README, together with the installation guide and configuration options in the README for the Helm chart. Here is a short summary of how it works within the Bank-Vaults ecosystem:
To gain experience in this tool, and to get familiar with the potential of the Bank-Vaults ecosystem, in this guide we will:
Clone the repo and cd into it. With only a few make commands, you will have a kind cluster running with the Bank-Vaults ecosystem, including the Reloader:
# install dependencies
make deps
# start a kind cluster with Bank-Vaults operator, a Vault instance and Vault Secrets Webhook
make up-kind
# build the Vault Secrets Reloader image
make container-image
# deploy Vault Secrets Reloader
make deploy-kind
The last command will install the Reloader Helm chart with the following configuration:
helm upgrade --install vault-secrets-reloader deploy/charts/vault-secrets-reloader \
--set image.tag=dev \
--set collectorSyncPeriod=30s \
--set reloaderRunPeriod=1m \
--set env.VAULT_ROLE=reloader \
--set env.VAULT_ADDR=https://vault.default.svc.cluster.local:8200 \
--set env.VAULT_TLS_SECRET=vault-tls \
--set env.VAULT_TLS_SECRET_NS=bank-vaults-infra \
--namespace bank-vaults-infra
Two important set of configurations are being set here:
export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode)
kubectl get secret vault-tls -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
export VAULT_CACERT=$PWD/vault-ca.crt
export VAULT_ADDR=https://127.0.0.1:8200
kubectl port-forward service/vault 8200 &
Now that we have the Bank-Vaults ecosystem running in our kind cluster, we can deploy some workloads:
# deploy some workloads
kubectl apply -f e2e/deploy/workloads
Looking at the manifest of one of the deployments, the only difference from one that is prepared to work with the Bank-Vaults Webhook with all the annotations starting with vault.security.banzaicloud.io and the env values starting with vault: is the presence of the new alpha.vault.security.banzaicloud.io/reload-on-secret-change: "true" annotation telling the Reloader to collect secrets and reload it if necessary.
apiVersion: apps/v1
kind: Deployment
metadata:
name: reloader-test-deployment-to-be-reloaded
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: reloader-test-deployment-to-be-reloaded
template:
metadata:
labels:
app.kubernetes.io/name: reloader-test-deployment-to-be-reloaded
annotations:
vault.security.banzaicloud.io/vault-addr: "https://vault:8200"
vault.security.banzaicloud.io/vault-tls-secret: vault-tls
alpha.vault.security.banzaicloud.io/reload-on-secret-change: "true"
spec:
initContainers:
- name: init-ubuntu
image: ubuntu
command: ["sh", "-c", "echo $AWS_SECRET_ACCESS_KEY && echo $MYSQL_PASSWORD && echo initContainers ready"]
env:
- name: AWS_SECRET_ACCESS_KEY
value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
- name: MYSQL_PASSWORD
value: vault:secret/data/mysql#${.MYSQL_PASSWORD}
resources:
limits:
memory: "128Mi"
cpu: "100m"
containers:
- name: alpine
image: alpine
command:
- "sh"
- "-c"
- "echo $AWS_SECRET_ACCESS_KEY && echo $MYSQL_PASSWORD && echo going to sleep... && sleep 10000"
env:
- name: AWS_SECRET_ACCESS_KEY
value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
- name: MYSQL_PASSWORD
value: vault:secret/data/mysql#${.MYSQL_PASSWORD}
resources:
limits:
memory: "128Mi"
cpu: "100m"
To see the Reloader in action, first of all take a look at the logs to see information about which workload secrets are being collected and if any of them needs to be reloaded.
# watch reloader logs
kubectl logs -n bank-vaults-infra -l app.kubernetes.io/name=vault-secrets-reloader --follow
Now everything is set to try some things out with the Reloader:
vault kv patch secret/mysql MYSQL_PASSWORD=totallydifferentsecret
vault kv patch secret/accounts/aws AWS_SECRET_ACCESS_KEY=s3cr3t2
vault kv patch secret/dockerrepo DOCKER_REPO_PASSWORD=dockerrepopassword2
# check the reload count after the new rollout has been completed
kubectl get po -l app.kubernetes.io/name=reloader-test-daemonset -o jsonpath='{ .items[*].metadata.annotations.alpha\.vault\.security\.banzaicloud\.io/secret-reload-count }'
# delete MYSQL_PASSWORD from the initContainer and the container as well
kubectl edit deployment reloader-test-deployment-to-be-reloaded
vault kv patch secret/mysql MYSQL_PASSWORD=totallydifferentsecret2
vault kv metadata delete secret/mysql
# watch reloader logs, there should be similar error message soon:
# time=xxx level=ERROR msg="Vault secret path secret/data/mysql not found" app=vault-secrets-reloader worker=reloader
kubectl logs -n bank-vaults-infra -l app.kubernetes.io/name=vault-secrets-reloader --follow
You can tear down the test cluster with make down once you finished.
This is an early alpha version and breaking changes are expected. As such, it is not recommended for usage in production. We are actively working on adding new features to the Reloader. You can support us with your feedback, bug reports, and feature requests.
Get emerging insights on emerging technology straight to your inbox.
Discover why security teams rely on Panoptica's graph-based technology to navigate and prioritize risks across multi-cloud landscapes, enhancing accuracy and resilience in safeguarding diverse ecosystems.
The Shift is Outshift’s exclusive newsletter.
Get the latest news and updates on cloud native modern applications, application security, generative AI, quantum computing, and other groundbreaking innovations shaping the future of technology.