Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
Share
8 min read
Share
Would you like to see Velero in action, and see how easy is to backup and recover Kubernetes clusters? Check out Pipeline, Banzai Cloud's container management platform for hybrid clouds.
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
kubectl create namespace velero
bank-vaults-velero
in the Stockholm region to encrypt and store backup data. Create your own bucket and key, or, if you're using a different provider, their equivalent.BUCKET=bank-vaults-velero
REGION=eu-north-1
KMS_KEY_ID=alias/bank-vaults-velero
SECRET_FILE=~/.aws/credentials
helm upgrade --install velero --namespace velero \
--set configuration.provider=aws \
--set-file credentials.secretContents.cloud=${SECRET_FILE} \
--set deployRestic=true \
--set configuration.backupStorageLocation.name=aws \
--set configuration.backupStorageLocation.bucket=${BUCKET} \
--set configuration.backupStorageLocation.config.region=${REGION} \
--set configuration.backupStorageLocation.config.kmsKeyId=${KMS_KEY_ID} \
--set configuration.volumeSnapshotLocation.name=aws \
--set configuration.volumeSnapshotLocation.config.region=${REGION} \
--set "initContainers[0].name"=velero-plugin-for-aws \
--set "initContainers[0].image"=velero/velero-plugin-for-aws:v1.0.0 \
--set "initContainers[0].volumeMounts[0].mountPath"=/target \
--set "initContainers[0].volumeMounts[0].name"=plugins \
vmware-tanzu/velero
helm upgrade --install vault-operator banzaicloud-stable/vault-operator
kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/cr-raft.yaml
cr-raft.yaml
has a special flag called veleroEnabled
. You can set this flag to true
to make the operator "Velero-aware" for this Vault instance. This is particularly useful when working with file-based Vault storage backends (file
, raft
) that write to disks. Velero can freeze those disks before taking a PersistentVolume snapshot, then unfreeze them once the snapshot is complete. This is all made possible through Velero hooks. And freezing the file system ensures that all pending disk I/O operations have been completed before the snapshot is taken.
# Add Velero fsfreeze sidecar container and supporting hook annotations to Vault Pods:
# https://velero.io/docs/v1.2.0/hooks/
veleroEnabled: true
After enabling this flag, run the kubectl get pods
command.
$ k get pods
NAME READY STATUS RESTARTS AGE
vault-0 4/4 Running 0 29m
vault-1 4/4 Running 0 29m
vault-2 4/4 Running 0 29m
vault-configurer-8774986d7-gp744 1/1 Running 0 29m
You can see that:
velero-fsfreeze
sidecar container (4 instead of 3) attached.exec
into this container later on. Note that these sidecar containers have only minor resource requirements:
velero-fsfreeze:
Container ID: docker://994f43364b36e34835427976e869031f9ff003ebda52eb9f5b5afcb1db2e05e3
Image: velero/fsfreeze-pause:latest
Image ID: docker-pullable://velero/fsfreeze-pause@sha256:08fe6d510c250c8bafe0d2af7daf9d6403d9d7902db8aa60c25df412ac489e23
Port: <none>
Host Port: <none>
State: Running
Started: Fri, 31 Jan 2020 09:51:56 +0100
Ready: True
Restart Count: 0
Limits:
cpu: 50m
memory: 32Mi
Requests:
cpu: 50m
memory: 32Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from vault-token-64nxw (ro)
/vault/file from vault-raft (rw)
exec
into this privileged
container. To do that, disable the already deprecated DenyEscalatingExec admission controller on a given Kubernetes cluster. Otherwise the hooks will fail (in which case Velero works in a best-effort mode and continues to create backups).kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/examples/backup/backup.yaml
To use the Velero CLI, complete the following steps:
velero backup create --selector vault_cr=vault vault-1
velero backup describe --details vault-1
Name: vault-1
Namespace: velero
Labels: velero.io/backup=vault-1
velero.io/pv=pvc-6eb4d9c1-25cd-4a28-8868-90fa9d51503a
velero.io/storage-location=default
Annotations: <none>
Phase: Completed
Namespaces:
Included: *
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: vault_cr=vault
Storage Location: default
Snapshot PVs: auto
TTL: 720h0m0s
Hooks: <none>
Backup Format Version: 1
Started: 2020-01-29 14:17:41 +0100 CET
Completed: 2020-01-29 14:17:45 +0100 CET
Expiration: 2020-02-28 14:17:41 +0100 CET
CAUTION: Hazard of data loss Make absolutely sure that you are running the following commands on your test cluster. We want to simulate a disaster not cause one.
$ kubectl delete vault -l vault_cr=vault
vault.vault.banzaicloud.com "vault" deleted
$ kubectl delete pvc -l vault_cr=vault
persistentvolumeclaim "vault-raft-vault-0" deleted
persistentvolumeclaim "vault-raft-vault-1" deleted
persistentvolumeclaim "vault-raft-vault-2" deleted
kubectl scale deployment vault-operator --replicas 0
velero restore create --from-backup vault-1
velero restore get
Expected output:
NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR
vault1-20200129142409 vault1 Completed 0 0 2020-01-29 14:24:09 +0100 CET <none>
Running
:
kubectl get pods
Expected output:
NAME READY STATUS RESTARTS AGE
vault-0 4/4 Running 0 1m42s
vault-1 4/4 Running 0 1m42s
vault-2 4/4 Running 0 1m42s
vault-configurer-8774986d7-gp744 1/1 Running 0 1m42s
kubectl scale deployment vault-operator --replicas 1
Since every resource created by the operator was fully restored from the backup, the operator has nothing to do, and will just keep watching for changes.velero backup delete vault-1
Update: since releasing Bank-Vaults 1.0, we also provide commercial support for Bank-Vaults. If you're interested in commercial support, or anything else from our suite of products, make sure you get in touch with us, here.If you're interested in contributing, check out the Bank-Vaults repository, or give us a GitHub star.
Learn more about Bank-Vaults:
- Secret injection webhook improvements
- Backing up Vault with Velero
- Vault replication across multiple datacenters
- Vault secret injection webhook and Istio
- HSM support
- Injecting dynamic configuration with templates
- OIDC issuer discovery for Kubernetes service accounts
- Show all posts related to Bank-Vaults
Get emerging insights on emerging technology straight to your inbox.
Discover why security teams rely on Panoptica's graph-based technology to navigate and prioritize risks across multi-cloud landscapes, enhancing accuracy and resilience in safeguarding diverse ecosystems.
The Shift is Outshift’s exclusive newsletter.
Get the latest news and updates on generative AI, quantum computing, and other groundbreaking innovations shaping the future of technology.