HA control plane¶

A single control-plane node is fine for learning. For anything you care about, run three — so one node (or one host) failing doesn't take the cluster's brain offline. CAPVCO makes this a one-line change, and does the hard parts (etcd quorum, safe rolls, self-healing) for you.

Turn it on¶

Set the control-plane replica count to an odd number ≥ 3 when you render a cluster:

CONTROL_PLANE_MACHINE_COUNT=3 VCO_LOCATION=<your-location> \
VCO_IMAGE_ID=<stock-ubuntu-image-id-in-that-location> \
clusterctl generate cluster my-cluster \
  --kubernetes-version v1.35.8 --infrastructure vco \
  --worker-machine-count 1 > my-cluster.yaml

That's it. Everything below is what happens under the hood — worth understanding, nothing to configure.

Why odd numbers?

etcd needs a majority to make progress. 3 nodes tolerate 1 failure; 5 tolerate 2. An even count buys you nothing (4 nodes still only tolerate 1 failure) and just costs more. Use 3 for almost everything; 5 only for very large or very critical clusters.

What you get¶

Stacked etcd with quorum protection¶

Each control-plane node runs an etcd member alongside the API server ("stacked" etcd). Three nodes = a 3-member etcd that survives one member going down.

KubeadmControlPlane (KCP, the upstream control-plane provider) owns the safety here, and CAPVCO leans on it rather than reinventing it:

Control-plane nodes are rolled one at a time — scale up to the new node, wait for it to be healthy and its etcd member to join, then remove the old one. etcd stays quorate throughout.
KCP's remediation refuses to act if doing so would break quorum, and won't remediate while another control-plane node is still provisioning. You can't accidentally reprovision yourself below majority.

The control-plane endpoint and load balancer¶

The API server is published on :6443 through a whitesky.cloud L4 load balancer (the default controlPlaneExposure: loadbalancer). The LB and its public IP are allocated before any node exists, so the control-plane certificates can be minted against a stable endpoint from the start.

CAPVCO manages the LB's backend pool by health, because the platform's L4 LB has no health checks of its own:

A control-plane node is added to the pool only once its API server is actually healthy (tracked via KCP's per-machine APIServerPodHealthy condition), and removed when it stops serving.
The first node is added while the pool is still empty (bootstrap exception), and the controller never removes the last remaining host — so you can't black-hole your own control plane.
Each machine only ever adds or removes its own pool entry.

This is why a joining or dead control-plane node doesn't poison the endpoint: it simply isn't in the pool until it's ready.

Single control plane: portforward

For a one-node control plane you can set controlPlaneExposure: portforward, which publishes :6443 via a port-forward to the sole node instead of an LB. It's a documented minimal-setup fallback — use the load balancer for anything HA.

Self-healing (MachineHealthCheck)¶

The base and prebaked templates ship two MachineHealthCheck objects — one for workers, one for the control plane — that reprovision a node that goes unhealthy. Defaults:

Setting	Workers	Control plane	Variable
Node-startup timeout	900 s (15 m)	1200 s (20 m)	`MHC_WORKER_STARTUP_SECONDS` / `MHC_CP_STARTUP_SECONDS`
`Ready=False` / `Unknown` timeout	300 s	300 s	(fixed)
Max unhealthy (circuit breaker)	40 %	40 %	`MHC_WORKER_MAX_UNHEALTHY` / `MHC_CP_MAX_UNHEALTHY`

The 40 % circuit breaker matters: if more than 40 % of a pool goes unhealthy at once, CAPI stops remediating. That's almost always a systemic problem (a bad image, a network partition), and reprovisioning into it just digs the hole deeper.

The startup timeouts are generous on purpose — a whitesky.cloud VM create + mesh + kubeadm join is slower than a hyperscaler, and you don't want a slow-but-fine node killed mid-boot.

MHCs are on base/prebaked only

The ccm, backup, airgapped and geo templates don't ship MachineHealthChecks by default (they're single-CP or have site-specific remediation concerns). Add them back if you want self-healing there — copy the two MachineHealthCheck blocks from cluster-template.yaml.

Rolling Kubernetes upgrades¶

Upgrades are machine replacements, never in-place: bump the version and KCP rolls new nodes in and old nodes out, one at a time, preserving etcd quorum.

# control plane first…
kubectl patch kubeadmcontrolplane my-cluster-control-plane --type merge \
  -p '{"spec":{"version":"v1.36.4"}}'
# …then the workers (respect the one-minor-version skew rule: CP before workers)
kubectl patch machinedeployment my-cluster-md-0 --type merge \
  -p '{"spec":{"template":{"spec":{"version":"v1.36.4"}}}}'

On pre-baked images, also make sure an image for the new version exists in the cluster's location(s) and is referenced (VCO_IMAGE_ID / spec.geo.locationImages).

The v1.30 → v1.31 control-plane upgrade skew fix

During a control-plane upgrade a freshly-joined v1.31 kubelet can be routed by the L4 LB to a still-running v1.30 API server, whose older API rejects a v1.31-only field selector the kubelet uses. The kubelet's Service informer never syncs, pods stall in CreateContainerConfigError: services have not yet been read, the node never goes Ready, and the roll deadlocks.

The fix points each control-plane kubelet at its own local API server instead of the shared LB VIP, removing the skew entirely. It ships today in cluster-template-prebaked.yaml and cluster-template-geo.yaml (a postKubeadmCommands block). If you build control-plane nodes from a different template, carry that block across until the fix moves into the controller cloud-init — see known limitations.

Upstream now does this too. The example above is the v1.30 → v1.31 upgrade where we first hit it; both of those versions are long EOL. Cluster API sets kubeadm's ControlPlaneKubeletLocalMode feature gate automatically for Kubernetes v1.31–v1.35, and from v1.36 the behaviour is GA and built into kubeadm, so on the versions CAPVCO now ships the block is very likely redundant — it rewrites a value kubeadm already set. It is kept because it is idempotent and the deadlock it prevents was expensive; it should be removed once an upgrade e2e across the 1.35 → 1.36 boundary passes without it.

Next: stretch this control plane across datacenters → Geo-redundant clusters.