Skip to content

Operating the central OpenBao (CAPVCO)

A single, persistent OpenBao instance provides:

  1. the transit key-backend the CSI driver's encrypted StorageClass (be.cloudbuilders.csi-encrypted-vault) uses to wrap/unwrap LUKS volume keys, and
  2. a transit auto-unseal KMS that other Vault/OpenBao instances can seal against.

It runs on a dedicated VM in its own cloudspace (capvco-openbao, customer geert_bv_2, location qt-gen-dc01-003), not managed by Cluster API — so no cluster teardown affects it.

Current instance (deployed 2026-07-08): https://10.101.109.245:8200 — running, uninitialized and sealed, awaiting bao operator init. The self-signed CA is at /etc/openbao/tls/ca.pem on the VM; deploy the driver binary is at /usr/local/bin/bao (v2.5.5). VM access is via the portal console (or add an ingress port-forward to :22). OpenBao dropped mlock support, so the config has no disable_mlock line and swap is disabled on the VM instead.

  • Endpoint: https://<public-ip>:8200 (L4 LB → VM:8200, TLS terminated by OpenBao)
  • Storage: file backend at /opt/openbao/data, on a dedicated data disk (survives reboots)
  • TLS: self-signed CA + server cert (SAN = public IP). Distribute openbao-ca.pem to clients, or set BAO_SKIP_VERIFY=true / tls_skip_verify = true.

Client setup

export BAO_ADDR=https://<public-ip>:8200
export BAO_CACERT=/path/to/openbao-ca.pem      # or export BAO_SKIP_VERIFY=true
bao status                                     # shows Sealed / Initialized

First-time initialization (once)

bao operator init -key-shares=5 -key-threshold=3
# --> SAVE the 5 unseal keys and the initial root token in a password manager / offline.
#     Losing >3 unseal keys OR the data disk = permanent data loss. There is no recovery.

bao operator unseal <key1>
bao operator unseal <key2>
bao operator unseal <key3>                      # threshold reached -> unsealed

bao login <root-token>

Configure the transit engine

bao secrets enable transit

# Key used by the CSI encrypted StorageClass (keyBackendTransitKey: csi-volume-keys):
bao write -f transit/keys/csi-volume-keys

# Key used by downstream Vault/OpenBao instances for auto-unseal:
bao write -f transit/keys/autounseal

Grant a consumer access (policy + auth)

Transit least-privilege policy (per key):

# csi-transit.hcl
path "transit/encrypt/csi-volume-keys" { capabilities = ["update"] }
path "transit/decrypt/csi-volume-keys" { capabilities = ["update"] }
path "transit/rewrap/csi-volume-keys"  { capabilities = ["update"] }

bao policy write csi-transit csi-transit.hcl

For the CSI driver (Kubernetes auth): 

bao auth enable kubernetes
# TokenReview endpoint of the WORKLOAD cluster + its CA + a reviewer token:
bao write auth/kubernetes/config \
  kubernetes_host="https://<workload-apiserver>:6443" \
  kubernetes_ca_cert=@workload-ca.crt \
  token_reviewer_jwt=@reviewer-token
bao write auth/kubernetes/role/csi-controller \
  bound_service_account_names=csi-provisioner,csi-attacher,csi-driver-registrar \
  bound_service_account_namespaces=csi-driver \
  policies=csi-transit ttl=1h
(The encrypted StorageClass / CSI KEY_BACKEND_* config points KEY_BACKEND_ADDR at this endpoint, KEY_BACKEND_TRANSIT_MOUNT=transit, KEY_BACKEND_K8S_AUTH_PATH=kubernetes, KEY_BACKEND_K8S_ROLE=csi-controller / csi-node.)

For auto-unseal consumers, in the downstream instance's config: 

seal "transit" {
  address    = "https://<public-ip>:8200"
  token      = "<token-with-transit/autounseal-access>"
  key_name   = "autounseal"
  mount_path = "transit/"
  tls_ca_cert = "/etc/openbao/openbao-ca.pem"
}

Day-2 operations

Unseal after a reboot

OpenBao starts sealed after any restart. Re-run the unseal quorum: 

bao operator unseal <key1>; bao operator unseal <key2>; bao operator unseal <key3>
This central instance is the trust root, so it is intentionally manually unsealed (no chicken -and-egg auto-unseal). Keep the unseal keys accessible to whoever restarts it.

Service control (on the VM)

systemctl status openbao
journalctl -u openbao -f

Backup / restore

The file backend is the data disk at /opt/openbao/data. - Backup: stop or snapshot — either take a VCO disk snapshot of the data disk, or systemctl stop openbao && tar czf openbao-data.tgz -C /opt/openbao data && systemctl start openbao. - Restore: restore the disk/tarball to /opt/openbao/data, start, and unseal. - Also back up the unseal keys + root token and the CA/cert separately — the data is useless without the unseal keys.

Rotate / rekey

  • Rotate a transit key (new version, old ciphertext still decryptable): bao write -f transit/keys/csi-volume-keys/rotate
  • Rekey unseal shares: bao operator rekey -init -key-shares=5 -key-threshold=3
  • Rotate the root token: bao token create -policy=root then revoke the old one.

Upgrade

bao is a single binary at /usr/local/bin/bao. To upgrade: replace the binary, systemctl restart openbao, then unseal.

Health

bao status (sealed/initialized), GET /v1/sys/health (200 = unsealed+active, 503 = sealed). The LB/ingress can health-check the TCP port; app-level health needs the /v1/sys/health path.

Security notes

  • The unseal keys and root token are the crown jewels — store them offline / in a password manager, never in the cluster.
  • The endpoint is public (behind the ingress) with a self-signed cert; restrict access at the ingress/firewall to known consumers where possible, and always verify the CA (avoid BAO_SKIP_VERIFY outside first-time setup).
  • Give each consumer its own policy + auth role scoped to just the transit key it needs.