Automatic worker scaling¶

A worker pool that grows when pods cannot be scheduled and shrinks when nodes sit idle, without anyone running kubectl scale.

CAPVCO does not implement an autoscaler. It uses the upstream cluster-autoscaler with its clusterapi cloud provider, which scales a MachineDeployment by writing .spec.replicas — and CAPVCO already turns a replica change into a VM. What this guide covers is the three things that are ours: the template flavour that marks a pool as autoscalable, the deployment shape that keeps credentials where they belong, and the capacity CAPVCO publishes so a pool can come back from zero.

The short version¶

# 1. A cluster whose worker pool is autoscalable (min 1, max 5).
CLUSTER_NAME=demo \
VCO_LOCATION=<location> VCO_IMAGE_ID=<id> VCO_EXTNET_ID=<id> \
KUBERNETES_VERSION=v1.35.8 \
WORKER_MIN_COUNT=1 WORKER_MAX_COUNT=5 \
  clusterctl generate cluster demo --flavor autoscaling > demo.yaml
kubectl apply -f demo.yaml

# 2. The autoscaler for it, in the MANAGEMENT cluster.
CLUSTER_NAME=demo make autoscaler | kubectl apply -f -

Both halves are required. The template alone changes nothing: the annotations it stamps are inert until an autoscaler reads them.

No checkout? Render it from the release

make autoscaler needs the repository. The same manifest ships as a release asset with its placeholders intact, so it can be rendered in place — the same convention as the CSI add-on:

curl -fsSL https://gitlab.whitesky.cloud/api/v4/projects/730/packages/generic/capvco/v0.6.1/autoscaler.yaml \
  | CLUSTER_NAME=demo NAMESPACE=default \
    AUTOSCALER_IMAGE=registry.k8s.io/autoscaling/cluster-autoscaler:v1.35.2 \
    AUTOSCALER_SCALE_DOWN_UNNEEDED_TIME=10m \
    AUTOSCALER_SCALE_DOWN_DELAY_AFTER_ADD=10m \
    AUTOSCALER_SCALE_DOWN_UTILIZATION=0.5 \
    AUTOSCALER_MAX_NODE_PROVISION_TIME=30m \
    AUTOSCALER_LOG_LEVEL=2 \
    envsubst '$CLUSTER_NAME $NAMESPACE $AUTOSCALER_IMAGE $AUTOSCALER_SCALE_DOWN_UNNEEDED_TIME $AUTOSCALER_SCALE_DOWN_DELAY_AFTER_ADD $AUTOSCALER_SCALE_DOWN_UTILIZATION $AUTOSCALER_MAX_NODE_PROVISION_TIME $AUTOSCALER_LOG_LEVEL' \
  | kubectl apply -f -

Every variable must be given a value: envsubst renders an unset one as empty, which here produces things like --v= or a container with no image. make autoscaler supplies these defaults for you, which is why it is the recommended route when you have a checkout.

What the flavour actually changes¶

--flavor autoscaling is cluster-template.yaml with two differences on the worker MachineDeployment:

metadata:
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "1"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
spec:
  # no `replicas:` — deliberately

The annotations are how cluster-autoscaler discovers a node group at all. WORKER_MAX_COUNT is your cost ceiling: the autoscaler will never go past it, so size it against your backend quota rather than your optimism.

The missing replicas is not an oversight. replicas is the field the autoscaler writes. If the manifest also declared it, every re-apply of the rendered YAML — a GitOps sync, a kubectl apply to change something unrelated — would reset the pool and the autoscaler would scale it again. Nodes flap, and nothing in either system reports a problem. Leaving it out hands ownership over cleanly.

An unset replicas is well-defined: Cluster API's own webhook defaults it from these annotations — min-size for a new MachineDeployment, and otherwise the previous value clamped into [min, max]. So the pool starts at WORKER_MIN_COUNT.

--worker-machine-count does nothing on this flavour

clusterctl generate cluster --worker-machine-count 3 sets WORKER_MACHINE_COUNT, which this flavour does not use. Set WORKER_MIN_COUNT / WORKER_MAX_COUNT instead.

Where the autoscaler runs, and why it matters¶

It runs in the management cluster, in the workload cluster's namespace — not as a ClusterResourceSet add-on like csi or gpu.

That is a security decision, not a packaging one. The autoscaler needs two connections:

Flag	Cluster	What it does there
`--cloud-config`	management	reads and scales `MachineDeployment`s
`--kubeconfig`	workload	watches Pending pods and idle Nodes

Running it inside the workload cluster is possible, and it would fit our add-on pattern neatly — but it means placing a management-cluster credential inside a tenant cluster, one that can scale and delete Machines for every cluster the management cluster owns. So it runs on the management side, where the workload kubeconfig it needs is the one Cluster API already mints (<cluster>-kubeconfig). CAPVCO creates no credential of its own for this.

Passing only --kubeconfig looks like it works

If --cloud-config is omitted it does not fall back to in-cluster config — it falls back to the --kubeconfig value. The autoscaler then looks for MachineDeployments inside the workload cluster, finds none, and sits there scaling nothing, with no error. make autoscaler passes both explicitly.

The rendered manifest is one ServiceAccount, a namespaced Role (the CAPI node groups plus read access to vcomachinetemplates), a ConfigMap holding a kubeconfig that points at the pod's own ServiceAccount, and a single-replica Deployment. Leader election is off and the strategy is Recreate, because two autoscalers scaling one pool is worse than a few seconds of none.

Scaling from zero¶

WORKER_MIN_COUNT=0 is allowed, and it is the interesting case: a pool that costs nothing while idle. It needs one thing that only the infrastructure provider can supply.

With at least one replica running, the autoscaler learns a pool's shape by reading a real Node. At zero replicas there is no Node, so it cannot tell whether adding one would help the pending pod — and a pool at zero would never come back. The Cluster API contract's answer is for the provider to publish the shape on the machine template, which CAPVCO's manager does:

$ kubectl get vcomachinetemplate demo-md-0 -o jsonpath='{.status}' | jq
{
  "capacity": {
    "cpu": "2",
    "memory": "4Gi",
    "ephemeral-storage": "40Gi",
    "pods": "110"
  },
  "nodeInfo": { "architecture": "amd64", "operatingSystem": "linux" }
}

It is derived from the template's own vcpus / memoryMiB / bootDiskGiB — no backend call, no credential.

Two honest caveats:

This is the VM's capacity, not the node's allocatable. A real node reports less, because the kernel, the kubelet and system reservations take their cut first. So the autoscaler models a slightly roomier node than it gets, which can cost one wasted scale-up for a pod that only just fits. Every Cluster API provider has this wart; understating it instead would refuse scale-ups that would have worked.
A GPU pool only scales from zero when the vendor is named. The autoscaler needs the extended resource name the device plugin will advertise (nvidia.com/gpu), and that is vendor-specific. CAPVCO publishes it when spec.gpu.type names the vendor; when the pool is pinned by profileId, or asks for a vendor not yet supported (#36), the GPU is deliberately left out of the capacity. The pool then simply does not scale up from zero for a GPU pod — a visible no-op, rather than repeatedly building nodes the pod still cannot use.

Scale-down, and the ways it can hurt¶

Scale-up is the easy direction. Scale-down removes a node that is currently running pods, after cordoning and draining it, and that is where the sharp edges are.

The shipped defaults are conservative:

Flag	Default here	Why
`--scale-down-unneeded-time`	`10m`	a whitesky.cloud VM takes minutes to build; churn is expensive
`--scale-down-delay-after-add`	`10m`	do not trim a pool that just grew
`--max-node-provision-time`	`30m`	the upstream 15m gives up while the VM is still legitimately booting, and the autoscaler then marks the pool unhealthy
`--skip-nodes-with-local-storage`	`true`	never evict a pod whose data is on the node
`--skip-nodes-with-system-pods`	`true`	never remove a node running a non-DaemonSet kube-system pod

Override any of them via the make autoscaler variables (AUTOSCALER_*).

A pool cannot shrink past the kube-system pods it hosts¶

--skip-nodes-with-system-pods=true means the autoscaler will never remove a node running a non-DaemonSet pod in kube-system. That is the right default — the alternative lets it evict cluster infrastructure at will — but it has a consequence that surprises people:

In a CAPI cluster the control plane is tainted, so coredns can only run on workers. At its default two replicas with anti-affinity it lands one per worker, which makes every worker unremovable and means the pool never shrinks at all.

So before expecting scale-down to do anything:

# Confine coredns so it pins one worker instead of all of them.
kubectl -n kube-system scale deploy/coredns --replicas=1

and for a pool that must reach zero, coredns (and anything else like it) has to be able to live somewhere else entirely — a small always-on pool, or a control-plane toleration. A pool hosting kube-system Deployments has a hard floor of one node no matter what WORKER_MIN_COUNT says, and the only symptom is an autoscaler log line saying the node is unremovable.

The same applies to any workload of your own in kube-system.

Pod disruption budgets are your responsibility¶

The autoscaler drains through the Eviction API, so a PodDisruptionBudget is honoured — and a workload without one can be evicted to zero replicas mid-drain. If an application must not lose all of its pods at once, give it a PDB. The autoscaler will refuse to remove a node rather than violate one.

RWO volumes: read this before enabling scale-down on a stateful cluster¶

A pod with a ReadWriteOnce CSI volume is tied to the node its disk is attached to. Scale-down evicts it, the disk must detach from the removed VM, and the rescheduled pod must attach it elsewhere.

On this platform that path is not yet safe, for reasons tracked in #22: provisioned PVs carry no nodeAffinity, and the CSI driver reports no PUBLISH_UNPUBLISH_VOLUME, so Kubernetes' attach/detach tracking never engages. A rescheduled pod can hang on a volume the old node has not cleanly released, with a low-signal mount error rather than a clean Multi-Attach failure.

Until #22 is fixed, for a pool running stateful workloads either:

keep --skip-nodes-with-local-storage=true (the default) and give stateful pods a PDB that prevents eviction, or
run stateful workloads in a separate, non-autoscaled MachineDeployment — the autoscaler only touches pools carrying the annotations.

MachineHealthChecks¶

The flavour keeps the worker MachineHealthCheck from the base template. The two do not conflict — the MHC remediates unhealthy nodes, the autoscaler removes idle ones — but note that the MHC's unhealthyLessThanOrEqualTo: 40% is a percentage of a pool whose size now moves. On a pool that has scaled down to one node, a single unhealthy node is 100% and remediation stops, by design.

Picking the autoscaler image¶

cluster-autoscaler is released in lockstep with Kubernetes: use the image whose minor matches your workload cluster.

CLUSTER_NAME=demo \
AUTOSCALER_IMAGE=registry.k8s.io/autoscaling/cluster-autoscaler:v1.36.1 \
  make autoscaler | kubectl apply -f -

The default is pinned in the Makefile. Note that at the time of writing there is no 1.37 release, so a Kubernetes 1.37 cluster should run the 1.36 image.

Checking it works¶

$ kubectl -n default logs deploy/demo-autoscaler | grep -i 'node group'
I0915 ... Found multiple availability zones... node group: MachineDeployment/default/demo-md-0 (min: 1, max: 5, replicas: 1)

A wedged autoscaler still reports Ready

The provider builds an informer for every node-group kind on the API server — MachineDeployments, MachineSets, Machines and MachinePools — then blocks waiting for all of them to sync. If the ServiceAccount cannot list one of them, that reflector retries forever, the main loop never starts, and nothing scales. Meanwhile the pod is Running, 1/1, the Deployment is Available and kubectl rollout status passes. The only evidence is the log stopping at:

clusterapi_controller.go:228] waiting for caches to sync

So when nothing is scaling, check for denials first:

kubectl -n default logs deploy/demo-autoscaler | grep 'is forbidden'

The shipped Role covers this; a hand-edited one may not.

If no node group is found, in order of likelihood: the annotations are missing (wrong flavour), the --node-group-auto-discovery cluster name or namespace does not match, or the Role is missing the machinedeployments/scale subresource — a scale-up then fails with a 403 that appears only in this log.

To watch a scale-up:

kubectl get machinedeployment demo-md-0 -w
kubectl --kubeconfig demo.kubeconfig get nodes -w

What is validated¶

The autoscale scenario in hack/e2e-feature-matrix.sh runs on every merge request against a real backend: it provisions an autoscaling cluster, asserts the pool starts at its annotated minimum and that the VCOMachineTemplate publishes capacity, deploys the shipped autoscaler manifest (not a test copy), makes a pod Pending for lack of CPU, and asserts the pool grows, the new worker joins and runs the pod, and then shrinks back to its minimum with the Node removed.

It then drives the pool to zero and back: the floor is moved to 0 live, coredns is evicted so nothing pins the last worker, and a Pending pod has to bring the pool back from nothing. That leg is the one that actually exercises the provider-side contract — with no Node left to measure, the only thing that can tell the autoscaler a new node would fit is VCOMachineTemplate.status.capacity. A wrong resource name or capacity that is never published passes every other assertion and fails only there.

Still uncovered: geo clusters have a worker pool per site, each needing its own annotations, and that combination is untested.