Skip to content

Day-2 operations

Operating a running cluster over its lifetime — upgrades, scaling, rotation, observability, and the troubleshooting war stories worth knowing before they happen to you.

Getting the kubeconfig

Cluster API writes the workload kubeconfig to a Secret in the management cluster:

clusterctl get kubeconfig <cluster> > <cluster>.kubeconfig
kubectl --kubeconfig <cluster>.kubeconfig get nodes

Scaling

  • Workers: scale the MachineDeployment (kubectl scale machinedeployment <cluster>-md-0 --replicas=N), or edit WORKER_MACHINE_COUNT and re-render. For geo, scale each per-site MachineDeployment — a single unpinned pool lands entirely in the primary location.
  • Control plane: change KubeadmControlPlane.spec.replicas (odd numbers only). KCP rolls one member at a time, preserving etcd quorum. See HA control plane.

Kubernetes version upgrades

Bump KubeadmControlPlane.spec.version first, then the MachineDeployment version (respect the one-minor-version control-plane-before-workers skew rule), and make sure the matching image exists in the cluster's location(s). KCP rolls control-plane machines one at a time, preserving etcd quorum; for geo it rolls one cloudspace at a time. Version→image resolution can be handled by spec.geo.images (version-keyed).

The control-plane upgrade skew fix

A rolling control-plane upgrade can deadlock on kubelet↔apiserver skew through the shared LB. The fix (kubelet → local apiserver) ships in the prebaked and geo templates and needs carrying to other templates until it moves into the controller. On Kubernetes ≥ 1.36 kubeadm does this itself, so the block is probably redundant on current versions — details and the condition for removing it: HA control plane.

Certificate & credential rotation

  • Control-plane PKI auto-rotates: KCP rolls the control plane before the 1-year kubeadm PKI expires, controlled by spec.rollout.before.certificatesExpiryDays (template default 90). Keep it below the certificate validity or KCP rolls continuously.
  • The whitesky.cloud credential: update the credentialSecretRef Secret and CAPVCO picks it up automatically (cache invalidates on resourceVersion), re-pushing the credentials injected into the workload cluster. The JWT refresh preserves scope and can't elevate.

etcd backup & restore

Enable scheduled snapshots with spec.backup — see the backup & restore guide. Restore is deliberately not push-button; follow the etcd restore runbook, which pauses Cluster API, fetches + decrypts a snapshot, and restores into one fresh control-plane node before the others rejoin.

Auto-remediation (MachineHealthCheck)

The base/prebaked templates ship default MachineHealthChecks (opt-out). A NotReady/unreachable worker is remediated (Machine deleted → VM torn down → MachineDeployment reprovisions); the maxUnhealthy circuit breaker (40 %) stops mass remediation during a systemic outage. Control-plane remediation is KCP-driven and quorum-safe. See HA control plane.

Deletion timeouts — why remediation needs them

Remediation deletes a Machine, and CAPI drains the node before it does. In CAPI v1beta2 an unset drain timeout means wait forever — and the node being remediated is, by definition, one that may be unreachable. Its pods can never confirm termination, so the drain never finishes and the replacement is never created. Nothing errors; the Machine simply sits in DrainingNode.

The shipped templates therefore set both timeouts explicitly on the KubeadmControlPlane and every MachineDeployment:

deletion:
  nodeDrainTimeoutSeconds: ${NODE_DRAIN_TIMEOUT_SECONDS:=120}
  nodeVolumeDetachTimeoutSeconds: ${NODE_VOLUME_DETACH_TIMEOUT_SECONDS:=900}

nodeVolumeDetachTimeoutSeconds is deliberately well above Kubernetes' ~6-minute force-detach for an unhealthy node. Set it lower and CAPI deletes the VM while a CSI disk is still attached — see the disk-ownership rule for what that used to cost. Both are overridable at render time, and CAPI propagates them to existing Machines in place, with no rollout:

kubectl patch machinedeployment <cluster>-md-0 --type merge \
  -p '{"spec":{"template":{"spec":{"deletion":{"nodeDrainTimeoutSeconds":120,"nodeVolumeDetachTimeoutSeconds":900}}}}}'
kubectl patch kcp <cluster>-control-plane --type merge \
  -p '{"spec":{"machineTemplate":{"spec":{"deletion":{"nodeDrainTimeoutSeconds":120,"nodeVolumeDetachTimeoutSeconds":900}}}}}'

A Machine stuck in DrainingNode with Pod …: deletionTimestamp set, but still not removed from the Node is this, on a cluster created before the timeouts were in the templates.

VM teardown and CSI volumes

Deleting a VM on the backend destroys every disk still attached to it unless each one is detached first — destroyed outright, not recycled, and not recoverable from the recycle bin. So on teardown CAPVCO destroys only the disks it can prove it created, and detaches everything else:

Disk Proof of ownership Teardown
Boot disk created by the VM create (disk_type: BOOT) destroyed with the VM
etcd disk status.etcdDiskId destroyed with the VM
spec.dataDisks status.dataDiskIds destroyed with the VM
Anything else — CSI PersistentVolumes none detached and left intact

A detached disk stays attachable, so Kubernetes' attach/detach controller moves the volume to the replacement node on its own and the pod comes back with its data.

Two consequences worth knowing:

  • Unprovable ownership leaks rather than destroys. A disk CAPVCO cannot account for is left behind — visible, billed and recoverable. That is the deliberate choice over the alternative. Machines created before status.dataDiskIds existed fall in this bucket: their data disks are detached and left, not deleted.
  • A failed detach aborts the delete. The VM is not removed and the finalizer stays. A stuck finalizer is recoverable; a destroyed volume is not.

This matters most during auto-remediation: nothing on a powered-off node can unpublish its volumes, so a CSI disk is still attached at exactly the moment the VM gets deleted.

Disk growth

If a machine template enables diskGrowthPolicy, the controller grows node boot/etcd disks before DiskPressure, capped by maxBootDiskGiB / maxEtcdDiskGiB. Watch the DiskHealthy condition and status.lastDiskGrowthTime. This is for the nodes' own disks — CSI PersistentVolume expansion is separate (edit the PVC).

Pausing reconciliation

To stop Cluster API acting on a cluster (during a restore or manual surgery):

kubectl annotate cluster <cluster> cluster.x-k8s.io/paused=true --overwrite
# …work…
kubectl annotate cluster <cluster> cluster.x-k8s.io/paused-

Observability

The manager exposes Prometheus metrics on /metrics — controller-runtime metrics plus capvco_vco_api_requests_total / _duration_seconds and capvco_drift_orphans. VCOCluster carries real conditions (Ready, CloudspaceReady, MeshReady, EndpointReady, BackupReady, DriftDetected, …) and emits Events on key transitions. A starter Grafana dashboard + Prometheus alerts live in config/observability/.

Drift detection & orphan GC

spec.driftPolicy (opt-in) periodically compares actual whitesky.cloud resources against desired. Report (default) only surfaces orphans (a DriftDetected condition + the capvco_drift_orphans metric); GC deletes them, gated behind per-resource-type opt-in + dryRun. See Configuration.

Troubleshooting — war stories

  • FailureReason=CIDROverlap. The pod/service CIDR overlaps a cloudspace private network. Choose disjoint CIDRs — the CNI would otherwise hijack the node LAN. → Networking
  • Pods intermittently can't reach across a geo cluster. The tunnel MTU is wrong — the cluster isn't running the tunnel-tuned Calico (veth_mtu: 1350). Make sure the cni.yaml add-on is applied; it pins that MTU for every cluster. → Networking
  • Nodes stuck tainted uninitialized. Kubelet has --cloud-provider=external but no CCM is running to clear the taint. Enable the CCM add-on, or don't set the flag. → Load balancing
  • Stuck in Provisioning on a stock image. The node installs Kubernetes on first boot (needs egress to pkgs.k8s.io) and can race apt locks / slow mirrors. Prefer pre-baked images for production.
  • Slow provisioning generally. Cloudspace-create takes minutes; DNS record creation is ~30 s. Watch clusterctl describe cluster conditions rather than assuming a hang.
  • BackupReady=False. Usually the BYO backup bucket lacks Object Lock — it can only be enabled at bucket creation, so recreate the bucket with Object Lock + versioning. → Backup
  • KeyEscrowed stuck amber. You're using a provider-generated backup key and haven't escrowed + acknowledged it. → Backup encryption
  • ExternalDNS records lag or webhook times out. whitesky.cloud DNS writes are ~30 s each and the TTL is a fixed hour; the shipped timeouts absorb this, but expect DNS to trail reality. → ExternalDNS
  • Workload endpoint unreachable from your laptop but fine in-cluster. The endpoint IP may be provider-network-routable (RFC1918), not internet-routable. → Known limitations

For the full list of edges and platform constraints, see Known limitations.