etcd backup & restore

etcd holds all your cluster state — every object, every Secret. Lose it and the cluster is gone. CAPVCO gives you encrypted, WORM-protected etcd snapshots on a schedule, and — just as important — a restore procedure that has actually been exercised, not just written down.

Turn it on

Use cluster-template-backup.yaml, or add spec.backup to a VCOCluster. You need a BYO S3 bucket with Object Lock + versioning enabled (see the prerequisite below), and an S3 credentials Secret.

kubectl create secret generic my-cluster-backup-s3 \
  --from-literal=accessKeyID=<key> \
  --from-literal=secretAccessKey=<secret>

VCO_LOCATION=<loc> \
VCO_IMAGE_ID=<prebaked-image-id-in-that-location> \
VCO_BACKUP_S3_BUCKET=my-etcd-backups \
VCO_BACKUP_S3_ENDPOINT=<s3-endpoint> \
clusterctl generate cluster my-cluster \
  --kubernetes-version v1.36.4 \
  --infrastructure vco --flavor backup \
  --worker-machine-count 1 > my-cluster.yaml
kubectl apply -f my-cluster.yaml

Create the bucket with Object Lock at creation time

S3 Object Lock can only be enabled when a bucket is created, never added later. Create the bucket up front with Object Lock + versioning on. CAPVCO validates this at reconcile and raises BackupReady=False rather than silently running without WORM protection — it only ever writes, lists, and (when retention expires) deletes.

The spec.backup fields

spec:
  backup:
    s3:
      endpoint: <s3-endpoint>          # empty = AWS S3
      region: us-east-1
      bucket: my-etcd-backups          # required
      prefix: clusters
      forcePathStyle: true             # required by most non-AWS S3
      credentialsSecretRef: my-cluster-backup-s3   # keys: accessKeyID, secretAccessKey
    schedule: "0 * * * *"              # hourly (finest tier cadence)
    retention: {hourly: 24, daily: 7, weekly: 4, monthly: 6}
    objectLock: {enabled: true, mode: compliance}    # or governance
    encryption: {mode: age, recipient: ""}           # empty = provider-generated key

Pin the backup image for a hardened deployment

The snapshot CronJob runs as root on a control-plane node — kubeadm writes the etcd key 0600 root:root, so the snapshot can't read it otherwise — with the host etcd PKI mounted read-only and the S3 credentials in its env. That blast radius is deliberately tightened (the pod drops all Linux capabilities, disables privilege escalation, and runs the default seccomp profile), but the image it runs is the provider's own manager image, which defaults to the mutable :latest tag. For a reproducible / hardened deployment, set BACKUP_IMAGE on the manager to a digest-pinned reference (…/cluster-api-provider-vco@sha256:…) so a tag re-point can't change what runs with that access on your next snapshot.

Grandfather-father-son retention

The snapshot job runs at the finest cadence (hourly). At upload time each snapshot is classified into the tiers it "opens" — if there's no snapshot yet for the current hour / day / ISO-week / month, this one becomes that tier's keeper (one snapshot can open several tiers at once). You keep the last N of each tier:

Tier	Default keep	Variable
hourly	24	VCO_BACKUP_KEEP_HOURLY
daily	7	VCO_BACKUP_KEEP_DAILY
weekly	4	VCO_BACKUP_KEEP_WEEKLY
monthly	6	VCO_BACKUP_KEEP_MONTHLY

A count of 0 disables a tier. This tier-classification approach (rather than "keep last N objects") is deliberate — it plays correctly with Object Lock, where you can't delete early.

Object Lock (WORM)

With objectLock.enabled: true, each snapshot is written with a retain-until date equal to the lifetime of the longest tier it belongs to, so it can't be deleted (or ransomware-encrypted) before it legitimately expires:

• compliance (default) — cannot be shortened or bypassed by anyone, including the account root. Ransomware-grade. Use this.
• governance — allows privileged bypass. The weaker option.

Encryption (age)

etcd contains every Secret, so snapshots are client-side encrypted with age before they leave the node — S3 never sees plaintext.

• BYO recipient (recommended): set encryption.recipient to your age public key (age1…). CAPVCO encrypts to it and never holds a private key — you own the identity, off cluster.
• Provider-generated (fallback): leave recipient empty and CAPVCO generates a keypair, storing the identity as a management-cluster Secret <cluster>-etcd-backup-age. The KeyEscrowed condition stays amber until you copy that identity somewhere safe and acknowledge it:

# export and store the private identity OFF the management cluster:
kubectl get secret <cluster>-etcd-backup-age -o jsonpath='{.data.identity}' | base64 -d > backup-identity.txt
# then acknowledge escrow:
kubectl annotate vcocluster <cluster> \
  capvco.infrastructure.cluster.x-k8s.io/backup-key-escrowed=true --overwrite

No identity, no restore

The private age identity must outlive the workload cluster. If you lose it, your backups are unrecoverable ciphertext. BYO-recipient is the safest posture because the key never touches the cluster at all.

Restore

There is no push-button restore in this release — deliberately. Restoring etcd is destructive and situational, so it's a guided runbook backed by safety-gated helper commands (capvco-backup fetch, and the restore-datadir helper). The full step-by-step, with the safety gates and the geo specifics, is the etcd restore runbook.

The shape of it:

1. Confirm the cluster is genuinely unrecoverable (not just slow or partitioned).
2. Pause Cluster API so KCP doesn't remediate into your restore:

   kubectl annotate cluster <cluster> cluster.x-k8s.io/paused=true --overwrite
   

3. Pick a snapshot (newest good, or the last one before corruption).
4. capvco-backup fetch — download + age-decrypt + verify checksum.
5. Restore into a staged dir and atomically swap it into /var/lib/etcd on one fresh control-plane node; kubeadm init against it.
6. Re-point the endpoint; bring the other control-plane nodes up empty to join fresh (never restore multiple members in parallel).
7. Verify quorum + workloads, then un-pause CAPI.

The helpers refuse to touch a data-dir unless etcd is provably dead (or you pass --force), verify the checksum before restoring, and always stage + atomic-swap rather than overwrite in place.

Practise it

A restore path you've never run is a wish, not a plan. Do a dry run against a throwaway cluster so the runbook is muscle memory before you need it for real. The fetch → decrypt → etcdctl snapshot status path has been validated against real snapshots.

---

Next: make nodes boot fast and offline → Pre-baked images.