Skip to content

Node disks: etcd disk & auto-expansion

Two node-disk features, both configured on the machine spec (i.e. on a VCOMachineTemplate):

  • a dedicated physical disk for etcd, so the control plane's write-heavy database isn't fighting everything else for I/O; and
  • automatic disk growth, so a node doesn't wedge on a full root disk at 3 a.m.

Neither is on by default. This is separate from CSI persistent volumes — that's for your workloads; this is for the nodes themselves.

Dedicated etcd disk

etcd is latency-sensitive: it fsyncs its write-ahead log on every commit. On a busy cluster, sharing the boot disk's I/O makes etcd slow and flaky. Giving it a dedicated physical disk mounted at /var/lib/etcd fixes that.

Set etcdDisk on the control-plane VCOMachineTemplate (it's ignored on workers):

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: VCOMachineTemplate
metadata:
  name: my-cluster-control-plane
spec:
  template:
    spec:
      imageId: 164
      vcpus: 2
      memoryMiB: 4096
      bootDiskGiB: 40
      etcdDisk:
        sizeGiB: 20
        type: physical      # physical (default) | data

The base and geo templates carry this as a commented-out block (VCO_CP_ETCD_DISK_GIB, default 20) — uncomment it to enable.

type Backing When
physical (default) Location-scoped physical disk on low-latency local storage The point of the feature — use this
data Software-defined (SDS) data disk Fallback for locations without spare physical capacity

What the controller does

Because a physical disk can't be requested inline at VM-create, the controller sequences it carefully:

  1. create the physical disk in the machine's location,
  2. create the VM stopped,
  3. attach the disk,
  4. start the VM — so /var/lib/etcd exists before kubeadm runs.

On teardown it detaches and deletes the disk. Status records etcdDiskId and etcdDiskAttached for idempotency.

Reboot-safe mounting

The disk is formatted once and mounted by filesystem LABEL (etcd-data), not by device name — because /dev/vdX ordering isn't stable across reboots. A bootcmd (which runs on every boot, before kubeadm) formats-if-needed, writes an fstab entry with nofail, and mounts. Your etcd data survives reboots and device reordering.

It just works, but know the shape

You don't configure any of the mount mechanics. If you're debugging a control-plane node, /var/lib/etcd being its own mount (check mount | grep etcd) confirms the physical disk is in play.

Automatic disk expansion

Turn this on and the controller grows a node's disk before it fills up — and grows the in-node filesystem to match, online, no reboot.

Set diskGrowthPolicy on a VCOMachineTemplate (control-plane or worker):

spec:
  template:
    spec:
      # …sizing…
      diskGrowthPolicy:
        enabled: true
        thresholdPercent: 80     # grow at/above this usage (default 80)
        incrementGiB: 10         # grow by this much per step (default 10)
        maxBootDiskGiB: 200      # cap; 0 disables boot-disk growth
        maxEtcdDiskGiB: 100      # cap; 0 disables etcd-disk growth

How it decides

The controller reads each node's real filesystem usage (root fs via the kubelet Summary API; the dedicated etcd disk via cadvisor metrics, matched by capacity) and the DiskPressure condition. When usage is at/above thresholdPercent (or DiskPressure fires — always a hard backstop), and the disk is under its cap, and a cooldown has elapsed since the last growth, it:

  1. grows the whitesky.cloud vDisk by one incrementGiB step, then
  2. an injected growfs systemd timer grows the partition + filesystem inside the node (online).

It records the observed sizes (status.bootDiskGiB / status.etcdDiskGiB) and status.lastDiskGrowthTime, and reports a DiskHealthy condition (BelowThreshold / Grew / AtCap / CoolingDown).

Caps and cooldown are your safety rails

The per-step increment, the hard caps, and the cooldown between steps exist so a runaway log-spewing pod can't balloon a disk without bound. Set maxBootDiskGiB / maxEtcdDiskGiB to sane ceilings. A cap of 0 disables growth for that disk.

Best-effort by design

If the controller can't read a node's usage (node down, kubeconfig not ready), it simply skips — disk growth never blocks machine reconciliation. DiskPressure remains the backstop.

Edge case: same-size disks

The etcd disk is matched among the node's filesystems by capacity. If the boot disk and etcd disk are the same size, that match is ambiguous. Give the etcd disk a different size than the boot disk (they usually differ anyway) to keep growth targeting unambiguous.


Next: expose your Services to the world → Load balancing & Services.