Skip to content

Geo-redundant clusters

This is the headline feature: one logical Kubernetes cluster with its control plane spread across multiple cloudspaces, so a whole cloudspace — or a whole datacenter — can go dark and the cluster keeps serving. Very few Cluster API providers offer this out of the box.

It's also the most involved feature. This guide starts with the mental model, gives you a working cluster, then goes deep on how it stays alive.

What "geo" actually means here

A normal cluster lives in one cloudspace in one location. A geo-redundant cluster spreads its control plane across multiple cloudspaces — it:

  • creates one cloudspace per site,
  • connects them into a single routed network over an automatically-built site-to-site VPN mesh,
  • places one control-plane node per site with a single stretched etcd, and
  • exposes the API server as a health-aware DNS name that always points at a live control plane.

There is no separate "single-location" vs "multi-location" flavor. A site is just a cloudspace pinned to a location; whether two sites share a location or sit in different datacenters is only a per-site property. Put every site in the same location for rack/host-level redundancy inside one DC, or in different locations to survive a full datacenter outage — same template, same cluster shape either way.

flowchart TB
  dns["<cluster>.<zone>  (health-aware DNS endpoint)"]
  subgraph A["Site 1 (primary)"]
    csa["Cloudspace 1"] --> cpa["control-plane + etcd"]
    csa --> wka["workers"]
  end
  subgraph B["Site 2"]
    csb["Cloudspace 2"] --> cpb["control-plane + etcd"]
    csb --> wkb["workers"]
  end
  csa <-->|"S2S VPN mesh"| csb
  dns -.->|resolves to a live CP| cpa
  dns -.->|fails over| cpb

One template does it all: cluster-template-geo.yaml. It declares three sites (one control-plane node each, so etcd quorum survives one site failure); you decide same-DC vs cross-datacenter purely by which location you give each site. The sites use positional keys — site-1, site-2, site-3 — matching the per-site DNS names <cluster>-site-N.<zone>. Two sites is the minimum geo but gives no quorum tolerance, so 3 is the default; adding a 4th (or 5th…) site is documented in the template's header comment.

Create a geo cluster

Prerequisites specific to geo

  • A pre-baked image in each site's location — VM images are location-scoped and not portable (see pre-baked images). You supply the per-site ids. Sites that share a location share an image id.
  • A managed DNS zone on your account (the endpoint lives under it).
  • The CNI add-on applied once to the management cluster — cni.yaml from the release (every cluster carries cni: calico, which already tunes Calico's veth_mtu to the cross-site tunnel MTU). See Install the provider → CNI add-on.
  • Management-cluster DNS forwarding configured (see Fast failover below) — a documented requirement, not optional for production.

Each of the three sites takes three location-scoped values — GEO_SITEn_LOCATION, GEO_SITEn_IMAGE_ID, GEO_SITEn_EXTNET_ID. Give every site the same location for an all-in-one-location geo cluster, or distinct locations to stretch across datacenters:

GEO_DNS_ZONE=<your-managed-zone> \
GEO_SITE1_LOCATION=<loc-a> GEO_SITE1_IMAGE_ID=<prebaked-id-in-loc-a> GEO_SITE1_EXTNET_ID=<extnet-in-loc-a> \
GEO_SITE2_LOCATION=<loc-b> GEO_SITE2_IMAGE_ID=<prebaked-id-in-loc-b> GEO_SITE2_EXTNET_ID=<extnet-in-loc-b> \
GEO_SITE3_LOCATION=<loc-c> GEO_SITE3_IMAGE_ID=<prebaked-id-in-loc-c> GEO_SITE3_EXTNET_ID=<extnet-in-loc-c> \
clusterctl generate cluster geo1 \
  --kubernetes-version v1.36.4 \
  --infrastructure vco --flavor geo \
  --control-plane-machine-count 3 > geo1.yaml

kubectl apply -f geo1.yaml

Watch it build — you'll see one CloudspaceReady per site, then MeshReady, then EndpointReady:

clusterctl describe cluster geo1
kubectl get vcocluster geo1 -o wide     # ENDPOINT column shows <cluster>.<zone>

The spec.geo fields

Everything geo-specific lives under VCOCluster.spec.geo:

Field Meaning
locations The list of site locations, one cloudspace each (≥ 2; the template ships 3). spec.location is site 1's location (the primary, which seeds the endpoint) and must appear here. Repeat a location for several sites in it; use distinct locations to stretch.
privateNetworks Optional per-site private CIDRs (positional with locations). Must be mutually non-overlapping and disjoint from pod/service CIDRs. Leave unset and the controller auto-assigns disjoint /24s.
etcdPlacement distributed (default) or concentrated — see below.
locationImages Per-location {location, imageId} — the template sets one per site (GEO_SITEn_IMAGE_ID), since images are location-scoped and aren't portable. Overrides the machine's spec.imageId per site. Sites sharing a location use the same id.
locationExternalNetworks Per-location external-network pinning — the template sets one per site (GEO_SITEn_EXTNET_ID); every site must be routable, since the endpoint is health-aware and probes all of them.
images Version-keyed image resolution for upgrades ({kubernetesVersion, source?, locationImageIds?}) — publish or pin the image for a given Kubernetes version, per location.
imagePublishCredentialSecretRef Operator credential used only when publishing images from a source (image create is operator-privileged).

There is no spec.geo.mesh / dns / failover

The VPN mesh, DNS endpoint and failover are entirely controller-driven — you don't configure them. They surface in status (MeshReady condition, status.dnsZone, status.controlPlaneDnsTarget).

etcd placement

spec:
  geo:
    etcdPlacement: distributed   # or concentrated
  • distributed (default) — a stacked etcd member on every site. The cluster survives a full location outage as long as etcd keeps quorum (2 of 3). This is the point of geo. Pair it with a physical etcd disk for latency.
  • concentrated — all control-plane nodes (and their stacked etcd) live in the primary location only. Cheaper and simpler, but a primary-location outage takes the whole cluster down — so it gives you worker-spread, not control-plane DR.

Inter-site latency matters for distributed

Stretched etcd is sensitive to round-trip time. The geo templates tune etcd's --heartbeat-interval (ETCD_HEARTBEAT_MS, default 250) and --election-timeout (ETCD_ELECTION_MS, default 2500) for a WAN. Keep inter-site RTT low (single-digit to low-tens of ms); very high latency will make etcd unstable no matter the tuning.

How it stays alive (the deep end)

The VPN mesh

For N cloudspaces the controller builds a full mesh — N·(N−1)/2 connected-cloudspace links, created in both directions — using the platform's connected-cloudspace (wireguard S2S) feature. Each cloudspace's private network is a distinct, non-overlapping CIDR.

Provisioning is all-or-nothing: if any cloudspace, mesh link, or control plane can't come up, the controller tears down everything it created. You never end up with a half-meshed cluster.

The CNI is already tunnel-safe

The tunnel path MTU is ~1400; the CNI must use veth_mtu: 1350 or pods intermittently can't talk across sites. The single calico CNI pins that MTU for every cluster, so the geo templates just set the …/cni: calico label like any other template — nothing geo-specific to change. See Networking.

The health-aware DNS endpoint

Instead of one load balancer (which deadlocks if the bootstrap control-plane node lands in a different cloudspace than the endpoint), the API endpoint is a DNS name:

  • <cluster>.<zone> is the endpoint clients and kubeconfigs use.
  • Each cloudspace also gets a <cluster>-site-N.<zone> name pointing at that site's control plane via a :6443 port-forward on its external IP.
  • The controller probes each site's :6443/healthz directly and points <cluster>.<zone> at a live control plane (primary-preferred). No cross-mesh dependency, so bootstrap works regardless of where the first node lands.

The certificate SANs cover the endpoint name, all per-site names, and 127.0.0.1.

Fast failover and the 1-hour TTL

Here's the honest caveat, because it changes how you deploy geo:

The platform hard-sets a 3600 s (1 hour) DNS TTL

whitesky.cloud's DNS API doesn't expose a TTL parameter — every record is 1 hour. So when the controller repoints <cluster>.<zone> to a healthy site, external clients can take up to an hour to notice if the primary's external IP is permanently lost.

  • Transient control-plane recovery is fine: the replacement node reuses the same external IP, so the DNS name keeps resolving (~5 min recovery).
  • Fast failover to a different site is defeated by the TTL for external clients.

The mitigation (a required production step) is to make the resolvers that matter bypass the TTL: point the management cluster's resolver at the zone's authoritative nameservers with caching disabled, so the management plane (KCP, the CAPVCO manager) always resolves fresh and fails over in seconds. A ready-made CoreDNS config lives in config/mgmt-dns (forward <zone> to ns01/02/03, no cache). KCP is upstream software — its resolver isn't something CAPVCO can patch in code, so this is deployment config.

The apiserver-proxy add-on

The geo template enables the …/apiserver-proxy: enabled add-on. It solves in-cluster API routing across sites:

  • Control-plane nodes talk to their local API server (127.0.0.1:6443).
  • Worker nodes run a tiny per-node HAProxy (127.0.0.1:6444) that prefers the control plane in their own cloudspace and treats remote control planes as backups.
  • A watcher DaemonSet (workers only) keeps each node's HAProxy backend list current by watching the default/kubernetes endpoints.

This exists because the platform's ingress LB has no per-host weighting or health checks and VM IPs are DHCP, so the backend list has to be maintained at runtime rather than baked in. It's fail-safe: a repoint rolls back if the new target is unhealthy.

Workers spread across sites

Important CAPI detail: KCP spreads control-plane nodes across failure domains automatically, but the MachineSet controller does not spread workers — it copies failureDomain verbatim. So the geo template ships one MachineDeployment per site, each pinned to a site's failure domain (site-1/site-2/site-3). If you add worker pools by hand, pin each to a site or they'll all land in the primary.

Day-2 and teardown

  • Upgrades roll one cloudspace at a time; resolve the right per-location image for the new version via spec.geo.locationImages / spec.geo.images.
  • Teardown first un-meshes all VPN links, then deletes cloudspaces in parallel (VMs first). Un-mesh is best-effort — a link that never fully came up may 400/500 on delete; the controller tolerates that and the cloudspace delete cascades the connections away regardless. Teardown of a geo cluster is noticeably slower than a single-cloudspace one.

Next: give it fast, durable storage → Persistent storage and Node disks.