Node disks: etcd disk & auto-expansion¶
Two node-disk features, both configured on the machine spec (i.e. on a VCOMachineTemplate):
- a dedicated physical disk for etcd, so the control plane's write-heavy database isn't fighting everything else for I/O; and
- automatic disk growth, so a node doesn't wedge on a full root disk at 3 a.m.
Neither is on by default. This is separate from CSI persistent volumes — that's for your workloads; this is for the nodes themselves.
Dedicated etcd disk¶
etcd is latency-sensitive: it fsyncs its write-ahead log on every commit. On a busy cluster, sharing
the boot disk's I/O makes etcd slow and flaky. Giving it a dedicated physical disk mounted at
/var/lib/etcd fixes that.
Set etcdDisk on the control-plane VCOMachineTemplate (it's ignored on workers):
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: VCOMachineTemplate
metadata:
name: my-cluster-control-plane
spec:
template:
spec:
imageId: 164
vcpus: 2
memoryMiB: 4096
bootDiskGiB: 40
etcdDisk:
sizeGiB: 20
type: physical # physical (default) | data
The base and geo templates carry this as a commented-out block (VCO_CP_ETCD_DISK_GIB,
default 20) — uncomment it to enable.
type |
Backing | When |
|---|---|---|
physical (default) |
Location-scoped physical disk on low-latency local storage | The point of the feature — use this |
data |
Software-defined (SDS) data disk | Fallback for locations without spare physical capacity |
What the controller does¶
Because a physical disk can't be requested inline at VM-create, the controller sequences it carefully:
- create the physical disk in the machine's location,
- create the VM stopped,
- attach the disk,
- start the VM — so
/var/lib/etcdexists before kubeadm runs.
On teardown it detaches and deletes the disk. Status records etcdDiskId and etcdDiskAttached for
idempotency.
Reboot-safe mounting¶
The disk is formatted once and mounted by filesystem LABEL (etcd-data), not by device name
— because /dev/vdX ordering isn't stable across reboots. A bootcmd (which runs on every boot,
before kubeadm) formats-if-needed, writes an fstab entry with nofail, and mounts. Your etcd data
survives reboots and device reordering.
It just works, but know the shape
You don't configure any of the mount mechanics. If you're debugging a control-plane node,
/var/lib/etcd being its own mount (check mount | grep etcd) confirms the physical disk is in
play.
Automatic disk expansion¶
Turn this on and the controller grows a node's disk before it fills up — and grows the in-node filesystem to match, online, no reboot.
Set diskGrowthPolicy on a VCOMachineTemplate (control-plane or worker):
spec:
template:
spec:
# …sizing…
diskGrowthPolicy:
enabled: true
thresholdPercent: 80 # grow at/above this usage (default 80)
incrementGiB: 10 # grow by this much per step (default 10)
maxBootDiskGiB: 200 # cap; 0 disables boot-disk growth
maxEtcdDiskGiB: 100 # cap; 0 disables etcd-disk growth
How it decides¶
The controller reads each node's real filesystem usage (root fs via the kubelet Summary API; the
dedicated etcd disk via cadvisor metrics, matched by capacity) and the DiskPressure condition.
When usage is at/above thresholdPercent (or DiskPressure fires — always a hard backstop), and the
disk is under its cap, and a cooldown has elapsed since the last growth, it:
- grows the whitesky.cloud vDisk by one
incrementGiBstep, then - an injected
growfssystemd timer grows the partition + filesystem inside the node (online).
It records the observed sizes (status.bootDiskGiB / status.etcdDiskGiB) and
status.lastDiskGrowthTime, and reports a DiskHealthy condition
(BelowThreshold / Grew / AtCap / CoolingDown).
Caps and cooldown are your safety rails
The per-step increment, the hard caps, and the cooldown between steps exist so a runaway
log-spewing pod can't balloon a disk without bound. Set maxBootDiskGiB / maxEtcdDiskGiB to
sane ceilings. A cap of 0 disables growth for that disk.
Best-effort by design
If the controller can't read a node's usage (node down, kubeconfig not ready), it simply skips —
disk growth never blocks machine reconciliation. DiskPressure remains the backstop.
Edge case: same-size disks¶
The etcd disk is matched among the node's filesystems by capacity. If the boot disk and etcd disk are the same size, that match is ambiguous. Give the etcd disk a different size than the boot disk (they usually differ anyway) to keep growth targeting unambiguous.
Next: expose your Services to the world → Load balancing & Services.