Skip to content

Day-2 operations

Operating a running cluster over its lifetime — upgrades, scaling, rotation, observability, and the troubleshooting war stories worth knowing before they happen to you.

Getting the kubeconfig

Cluster API writes the workload kubeconfig to a Secret in the management cluster:

clusterctl get kubeconfig <cluster> > <cluster>.kubeconfig
kubectl --kubeconfig <cluster>.kubeconfig get nodes

Scaling

  • Workers: scale the MachineDeployment (kubectl scale machinedeployment <cluster>-md-0 --replicas=N), or edit WORKER_MACHINE_COUNT and re-render. For geo, scale each per-site MachineDeployment — a single unpinned pool lands entirely in the primary location.
  • Control plane: change KubeadmControlPlane.spec.replicas (odd numbers only). KCP rolls one member at a time, preserving etcd quorum. See HA control plane.

Kubernetes version upgrades

Bump KubeadmControlPlane.spec.version first, then the MachineDeployment version (respect the one-minor-version control-plane-before-workers skew rule), and make sure the matching image exists in the cluster's location(s). KCP rolls control-plane machines one at a time, preserving etcd quorum; for geo it rolls one cloudspace at a time. Version→image resolution can be handled by spec.geo.images (version-keyed).

The control-plane upgrade skew fix

A rolling control-plane upgrade can deadlock on kubelet↔apiserver skew through the shared LB. The fix (kubelet → local apiserver) ships in the prebaked and geo templates and needs carrying to other templates until it moves into the controller. On Kubernetes ≥ 1.36 kubeadm does this itself, so the block is probably redundant on current versions — details and the condition for removing it: HA control plane.

Certificate & credential rotation

  • Control-plane PKI auto-rotates: KCP rolls the control plane before the 1-year kubeadm PKI expires, controlled by spec.rollout.before.certificatesExpiryDays (template default 90). Keep it below the certificate validity or KCP rolls continuously.
  • The whitesky.cloud credential: update the credentialSecretRef Secret and CAPVCO picks it up automatically (cache invalidates on resourceVersion), re-pushing the credentials injected into the workload cluster. The JWT refresh preserves scope and can't elevate.

etcd backup & restore

Enable scheduled snapshots with spec.backup — see the backup & restore guide. Restore is deliberately not push-button; follow the etcd restore runbook, which pauses Cluster API, fetches + decrypts a snapshot, and restores into one fresh control-plane node before the others rejoin.

Auto-remediation (MachineHealthCheck)

The base/prebaked templates ship default MachineHealthChecks (opt-out). A NotReady/unreachable worker is remediated (Machine deleted → VM torn down → MachineDeployment reprovisions); the maxUnhealthy circuit breaker (40 %) stops mass remediation during a systemic outage. Control-plane remediation is KCP-driven and quorum-safe. See HA control plane.

Disk growth

If a machine template enables diskGrowthPolicy, the controller grows node boot/etcd disks before DiskPressure, capped by maxBootDiskGiB / maxEtcdDiskGiB. Watch the DiskHealthy condition and status.lastDiskGrowthTime. This is for the nodes' own disks — CSI PersistentVolume expansion is separate (edit the PVC).

Pausing reconciliation

To stop Cluster API acting on a cluster (during a restore or manual surgery):

kubectl annotate cluster <cluster> cluster.x-k8s.io/paused=true --overwrite
# …work…
kubectl annotate cluster <cluster> cluster.x-k8s.io/paused-

Observability

The manager exposes Prometheus metrics on /metrics — controller-runtime metrics plus capvco_vco_api_requests_total / _duration_seconds and capvco_drift_orphans. VCOCluster carries real conditions (Ready, CloudspaceReady, MeshReady, EndpointReady, BackupReady, DriftDetected, …) and emits Events on key transitions. A starter Grafana dashboard + Prometheus alerts live in config/observability/.

Drift detection & orphan GC

spec.driftPolicy (opt-in) periodically compares actual whitesky.cloud resources against desired. Report (default) only surfaces orphans (a DriftDetected condition + the capvco_drift_orphans metric); GC deletes them, gated behind per-resource-type opt-in + dryRun. See Configuration.

Troubleshooting — war stories

  • FailureReason=CIDROverlap. The pod/service CIDR overlaps a cloudspace private network. Choose disjoint CIDRs — the CNI would otherwise hijack the node LAN. → Networking
  • Pods intermittently can't reach across a geo cluster. The tunnel MTU is wrong — the cluster isn't running the tunnel-tuned Calico (veth_mtu: 1350). Make sure the cni.yaml add-on is applied; it pins that MTU for every cluster. → Networking
  • Nodes stuck tainted uninitialized. Kubelet has --cloud-provider=external but no CCM is running to clear the taint. Enable the CCM add-on, or don't set the flag. → Load balancing
  • Stuck in Provisioning on a stock image. The node installs Kubernetes on first boot (needs egress to pkgs.k8s.io) and can race apt locks / slow mirrors. Prefer pre-baked images for production.
  • Slow provisioning generally. Cloudspace-create takes minutes; DNS record creation is ~30 s. Watch clusterctl describe cluster conditions rather than assuming a hang.
  • BackupReady=False. Usually the BYO backup bucket lacks Object Lock — it can only be enabled at bucket creation, so recreate the bucket with Object Lock + versioning. → Backup
  • KeyEscrowed stuck amber. You're using a provider-generated backup key and haven't escrowed + acknowledged it. → Backup encryption
  • ExternalDNS records lag or webhook times out. whitesky.cloud DNS writes are ~30 s each and the TTL is a fixed hour; the shipped timeouts absorb this, but expect DNS to trail reality. → ExternalDNS
  • Workload endpoint unreachable from your laptop but fine in-cluster. The endpoint IP may be provider-network-routable (RFC1918), not internet-routable. → Known limitations

For the full list of edges and platform constraints, see Known limitations.