etcd restore runbook (CAPVCO, plan §27.1)¶

Restoring etcd is the most destructive, irreversible operation in the provider: a restore run at the wrong moment overwrites a healthy cluster with a stale snapshot. CAPVCO therefore ships no push-button restore — this guided runbook, backed by the safety-gated capvco-backup helper subcommands, is the supported path. Drive it deliberately; do not skip the confirmation gates.

Restore is orchestrated from the management cluster, which holds the kubeconfig, the CAPI pause lever, and (for provider-generated keys) the age identity.

0. Prerequisites¶

The backup S3 bucket + credentials.
The age identity that decrypts the snapshots:
BYO recipient: your escrowed identity.
Provider-generated: kubectl -n <ns> get secret <cluster>-etcd-backup-age -o jsonpath='{.data.identity}' | base64 -d (this is exactly why that identity must be escrowed — if both the workload and management clusters are gone and it was never exported, the WORM-durable backups are unrecoverable).

1. Confirm the cluster is genuinely unrecoverable¶

Do not restore over a cluster that is merely partitioned or slow. Confirm etcd is truly dead/corrupted and cannot be recovered by less drastic means (member restart, single-member recovery). Only proceed once you are certain.

2. Pause CAPI reconciliation¶

Stop KCP from remediating/scaling into the restore:

kubectl annotate cluster <cluster> cluster.x-k8s.io/paused=true --overwrite

3. Choose the snapshot¶

List candidates (newest last). Pick the newest good one — if the incident was data corruption, pick the last snapshot before the corruption event.

# object keys are <prefix>/<cluster>/<UTC-timestamp>.db.age and sort chronologically

4. Fetch + decrypt (helper)¶

SNAPSHOT_KEY=<prefix>/<cluster>/<timestamp>.db.age \
OUT=/restore/snapshot.db \
AGE_IDENTITY_FILE=/restore/identity.txt \
S3_ENDPOINT=... S3_REGION=... S3_BUCKET=... \
S3_ACCESS_KEY_ID=... S3_SECRET_ACCESS_KEY=... S3_FORCE_PATH_STYLE=true \
capvco-backup fetch

This downloads the object and age-decrypts it to OUT, yielding a plain etcdctl-restorable snapshot. Verify the file is non-empty.

5. Restore into ONE fresh control-plane node¶

Bring up a single fresh CP VM (a new cloudspace if the old one is gone). On that node, with the workload paused:

etcdctl snapshot restore /restore/snapshot.db --data-dir /var/lib/etcd-restore (staged directory — never restore in place).
Stop the kubelet/static etcd pod, atomically swap the restored data dir into /var/lib/etcd, and kubeadm init against the restored data so the single member comes up as the new cluster.

Safety gates the helpers enforce: refuse if etcd is provably alive (unless --force), checksum-verify the snapshot before touching a data dir, restore into a staged dir + atomic swap, and restore exactly one member. Never restore N members from the same snapshot in parallel — the others join fresh.

6. Re-point the control-plane endpoint¶

Update the DNS/portforward endpoint to the new CP so clients and the other CPs reach it (for a geo cluster, this is the endpoint DNS name; §22 M8).

7. Let the remaining control-plane nodes rejoin¶

Bring the other CPs up empty and let them join the restored member (they must NOT carry old etcd data). For a geo cluster this rebuilds the S2S mesh as the sites reconnect.

8. Verify, then resume CAPI¶

Confirm the API server, etcd membership/quorum, and workloads are healthy, then:

kubectl annotate cluster <cluster> cluster.x-k8s.io/paused-

CAPI resumes reconciliation against the restored control plane.

Later (not v1): once this runbook is battle-tested across the disaster shapes, the clean total-loss rebuild path (steps 4–7) is promoted to a one-shot Job with the gates built in (auto-pause, --i-understand-this-is-destructive, refuse-if-alive).