GPUs¶

A worker pool can ask for GPUs by type — NVIDIA A100, not an instance id. CAPVCO resolves that type against the pool's location, reserves that many vGPU instances, attaches them before the node's first boot, and releases them when the machine is deleted.

Why by type¶

The platform's GPU API is instance-oriented: you reserve a named vGPU out of a GPU profile, then attach that specific id to a VM. Nobody wants a particular GPU — they want a pod to get an A100. CAPVCO does that translation, so nothing instance-shaped appears in your cluster config.

Naming the type¶

The type is matched against the profile's vendor and model as the platform reports them, not against a marketing name. A profile's description is not consulted. So on a platform that reports its cards as vendor Physical Nvidia, model 0x27b8:

spec.gpu.type: "Physical Nvidia"          # vendor          -> matches
spec.gpu.type: "0x27b8"                   # model           -> matches
spec.gpu.type: "Physical Nvidia 0x27b8"   # vendor + model  -> matches
spec.gpu.type: "NVIDIA L4"                # description     -> does NOT match

Matching is case- and hyphen-insensitive, and falls back to a substring match on "<vendor> <model>". A name that matches nothing fails the machine loudly and lists what the location actually offers, which is the quickest way to discover the right string:

no GPU type matches "NVIDIA L4" in this location;
available: "Physical Nvidia" (2 card(s), 1 free, e.g. model "0x27b8")

Read it with kubectl get vcomachine <machine> -o jsonpath='{.status.failureMessage}'.

A GPU worker pool¶

CLUSTER_NAME=gpu-demo \
KUBERNETES_VERSION=v1.35.8 \
VCO_LOCATION=<location-with-GPUs> \
VCO_IMAGE_ID=<image-id-in-that-location> \
VCO_EXTNET_ID=<external-network-id> \
VCO_GPU_TYPE="NVIDIA A100" \
VCO_GPU_COUNT=1 \
VCO_GPU_DRIVER_VERSION=550 \
clusterctl generate cluster gpu-demo --flavor gpu --worker-machine-count 1 \
  | kubectl apply -f -

The GPU request lives on the worker VCOMachineTemplate:

spec:
  template:
    spec:
      gpu:
        type: "NVIDIA A100"   # or profileId: <exact profile>
        count: 1
        driverVersion: "550"  # empty for a pre-baked GPU image

type is matched case- and space-insensitively against what the location offers, most specific first: a profile id, then an exact vendor+model / model / vendor, then a substring. So NVIDIA A100, nvidia a100 and a100 all work.

The platform lists one profile per physical card, with the card's PCI address in the model — for example vendor Dummy, model Dummy GPU 0000:f1:e1.0. A type therefore matches several profiles, which is expected: they differ only by which card, and CAPVCO picks the one with the most free capacity so reservations spread across cards. Use profileId when you want to pin one specific card.

A wrong type fails the machine, loudly

A type that does not exist, or whose cards have no room for count, puts the machine into GPUTypeUnavailable with what the location actually offers in the message — vendor, an example model, and the free count. It never quietly falls back to a different GPU. Read it with:

kubectl get vcomachine <machine> -o jsonpath='{.status.failureMessage}'

Getting a driver onto the node¶

An attached GPU is useless without a driver, a container runtime hook, and the device plugin. Three paths:

	`driverVersion`	Needs a prepared image	Needs egress	Boot time
Stock install on a prepared base image (recommended)	set it, e.g. `"570"`	yes — the GPU base image	yes	slower — installs on first boot
Stock install on a plain image	set it	no	yes	slower — and see the warning below
Fully pre-baked GPU image	leave empty	yes — built by you	no	fast

A plain stock image is not enough for a passthrough datacenter card. The driver will install perfectly and then refuse the device, leaving a node that is Ready with an unusable GPU. Two things have to be true at boot, and both are properties of the image, not of cloud-init — see What a GPU node image must do. CAPVCO ships a builder for exactly this: image-builder/build-gpu-base-image.sh.

The stock path has cloud-init install nvidia-driver-<branch>-server, the container toolkit, and a containerd nvidia runtime before kubeadm starts the kubelet, then runs nvidia-smi -L. If that fails the boot step fails loudly — deliberately, because a Ready node whose GPU nothing can use is worse than a node that never joined. (cloud-init runs each runcmd entry independently, so the node still joins; the log is the evidence.) It needs egress to Ubuntu's archive and NVIDIA's apt repo, so an air-gapped cluster must bake the driver in.

Which driver branch you need depends on what your platform hands out, and there is a fork worth settling before you rely on either path:

Full GPU passthrough — the guest sees the real PCI device. Ubuntu's archive driver works, so driverVersion is all you need. Check with lspci -nn | grep -i nvidia on a GPU-attached VM.
Mediated vGPU (GRID) — the guest sees a slice. That needs NVIDIA's licensed guest driver, which is not in Ubuntu's archive at any version. The stock install cannot work, so GPU workloads require a self-built image carrying that driver, matched to the host's vGPU Manager branch.

CAPVCO cannot tell which you have, which is why the driver is configuration rather than something it picks.

What a GPU node image must do¶

Two things must be true before the kernel reaches userspace. Neither can be done from cloud-init, because both need a reboot to take effect and runcmd cannot reboot without abandoning the kubeadm step that follows it. Both are baked by image-builder/build-gpu-base-image.sh:

bash image-builder/build-gpu-base-image.sh          # ~2 min, ~650MB qcow2
capvco-imagepublish -name capvco-ubuntu-2404-gpu-base \
  -file dist/capvco-ubuntu-2404-gpu-base.qcow2 \
  -boot-type bios -locations <your-gpu-location>

Then point VCO_IMAGE_ID at the published id and set driverVersion as usual. The image carries no Kubernetes version, no driver and no vendor, so it is one image per OS rather than a vendor x OS x k8s x driver matrix, and it works with the ordinary gpu flavour.

1. nouveau must never bind the card¶

On a stock image nouveau claims NVIDIA GPUs during early boot. On Ada-generation cards (L4, L40, RTX 40xx) it goes further and bootstraps the card's GSP firmware, after which the proprietary driver cannot take the device over — and unloading nouveau afterwards does not recover it, because the device needs a bus reset.

The image therefore blacklists it in /etc/modprobe.d and rebuilds the initramfs, which is the part that is easy to miss: modprobe.d is only honoured during early boot if the initramfs carries a copy of it.

2. the card's BAR must be reachable¶

A datacenter GPU exposes a multi-GiB BAR — an L4's BAR1 is 32GiB. The platform's firmware may advertise a 64-bit window for it at an address the guest CPU cannot reach (observed: 0x12000000000, ~1.25TiB, against a guest with 40 physical address bits = 1TiB). The kernel then discards the window and gives up:

acpi PNP0A03:00: host bridge window [mem 0x11ec0000000-...] (ignored, not CPU addressable)
pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]   <- 32-bit only
pci 0000:00:07.0: BAR 1 [mem size 0x800000000 64bit pref]: can't assign; no space
NVRM: BAR1 is 0M @ 0x0 (PCI:0000:00:07.0)
nvidia: probe of 0000:00:07.0 failed with error -1

The fix is two kernel parameters, which the image adds to the GRUB command line:

pci=nocrs — ignore the ACPI _CRS host-bridge windows (one of which is unreachable) and use the full CPU-addressable range as the root bus resource.
pci=realloc — reassign the BAR the firmware could not place.

Together the BAR lands somewhere reachable and the driver binds. Verified on real NVIDIA L4 hardware: without them, BAR1 is 0M; with them, a pod runs nvidia-smi.

Do not reach for bootType: uefi here. UEFI is the firmware that can map a large BAR, and on a platform that supports it properly it is the cleaner answer — but it depends on host support that may be absent. On the whitesky.cloud environment tested (2026-09), a GPU-attached VM with bootType: uefi never boots at all: VM status RUNNING, guest OS never starts, console stuck at "Guest has not initialized the display (yet)". The same image and boot type boots fine with no GPU attached. The gpu flavour therefore defaults to bios. If you do change it, the vm-image must be republished with a matching -boot-type — the platform rejects a VM whose boot type differs from its image's with 400: "Selected boot type can not be different from image boot type" — and it applies to the control plane too, since one image serves both.

Choosing a branch¶

Newer is not automatically more compatible: NVIDIA drops older architectures over time (R580 dropped Maxwell, Pascal and Volta), so an older LTS branch often covers more of a mixed fleet than the newest one. 535 is a reasonable default for broad hardware coverage; 570+ for recent datacenter cards.

The trade-off is CUDA. A workload image built against CUDA 12.4+ needs a driver new enough for it, or the forward-compatibility libraries — so pinning an old branch narrows which runtimes work natively.

driverVersion is a branch request, not a pin. Ubuntu keeps a retired branch's metapackage as a transitional package that depends on the supported one, so asking for a retired branch installs a newer one. On Ubuntu 24.04 (checked 2026-09) every branch but the newest forwards:

nvidia-driver-535-server -> 580
nvidia-driver-550-server -> 570 -> 580
nvidia-driver-570-server -> 580
nvidia-driver-580-server -> the actual driver

so any value resolves to 580 there. The boot log reports the branch that actually loaded whenever it differs from the request — look for capvco: NOTE: requested NVIDIA driver branch .... If you need a specific branch to hold, bake it into an image; the apt path cannot guarantee it.

Because driverVersion sits on the machine template, one cluster can run two GPU pools on different branches: an older branch for older cards, a newer one for new. That is usually better than compromising on a single branch.

If the GPU still is not usable¶

The node will be Ready and look healthy; the GPU is the only broken part. Check in this order, on the node (kubectl debug node/<name> --profile=sysadmin -- chroot /host ...):

Symptom	Cause
`nvidia-smi` exits 9, `lsmod` shows `nouveau`	the image does not blacklist nouveau
`NVRM: BAR1 is 0M @ 0x0`, `can't assign; no space`	the image lacks `pci=realloc pci=nocrs`
device plugin CrashLoops on `NVML: Driver Not Loaded`	either of the above — the plugin is fine, the driver never bound
GPU pods fail `no runtime for "nvidia" is configured`	the containerd `nvidia` runtime is not registered
`nvidia-smi` works, no `nvidia.com/gpu` on the node	the device-plugin add-on is not installed (`.../gpu` label)

Both image-level causes are what image-builder/build-gpu-base-image.sh exists to prevent — see What a GPU node image must do.

On the containerd one: the config schema differs by major version (1.x wants version = 2 with plugins."io.containerd.grpc.v1.cri", 2.x wants version = 3 with plugins."io.containerd.cri.v1.runtime"), and a drop-in written for the wrong one is silently ignored. CAPVCO registers the runtime with NVIDIA's own nvidia-ctk, which knows the local schema, and ensures the main config imports the drop-in directory — nvidia-ctk writes a drop-in by default, and nothing reads it otherwise.

Requesting a GPU in a pod¶

The gpu flavour carries the …/gpu: nvidia label, which ships the NVIDIA device plugin. That is what turns an attached GPU into an allocatable resource:

spec:
  runtimeClassName: nvidia
  containers:
    - name: train
      image: nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ["nvidia-smi", "-L"]
      resources:
        limits:
          nvidia.com/gpu: 1

Check the node is advertising them:

kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu

Mixed clusters¶

Each GPU node is labelled with its model, so a pod can target one type when a cluster has several:

nodeSelector:
  capvco.infrastructure.cluster.x-k8s.io/gpu-model: nvidia-a100

What the provider records¶

kubectl get vcomachine <machine> -o jsonpath='{.status.gpus}' | jq

[{"vgpuId":"…","name":"capvco-<machine>-gpu-0","profileId":"…","model":"NVIDIA A100","attached":true}]

The reservation name is deterministic, so a reservation is always traceable to the machine that owns it — and a reconcile that lost its response adopts the existing reservation instead of making a second one.

Teardown order¶

Deleting a machine deletes its VM first (with the platform's detach_gpu), and only then releases the reservation — the backend refuses to remove a vGPU from a running VM. The finalizer is held until every reservation recorded in status.gpus is gone, so a machine stuck in Deleting is worth checking against that list before assuming the VM is the problem.

Reservations are billed whether attached or not

A vGPU reservation costs money from the moment it exists, so CAPVCO records each id before anything else can fail, and releases every one on machine deletion. If you ever suspect a leak, list what the customer actually holds:

capvco-gpucheck -location <location> -list -name-prefix capvco-

Stub GPU profiles¶

Some locations offer a placeholder GPU profile for testing rather than real hardware. Attach/release works normally against one, which is enough to exercise the provider side — but there is no driver to install and no device to expose, so:

leave driverVersion empty (the install ends in nvidia-smi -L and would fail the node's boot), and
set VCO_GPU_ADDON=none when rendering, to leave the device plugin off. Its DaemonSet needs the containerd nvidia runtime, which only exists once a real driver is installed, so it would otherwise sit unschedulable.

The node still comes up with the GPU attached, status.gpus still records the reservation, and teardown still releases it. What you do not get is an allocatable nvidia.com/gpu or a pod that can use one.

Geo clusters¶

GPU profile ids are location-scoped, exactly like image ids. Each site resolves the requested type in its own location, so a geo GPU pool needs the type to exist in every site it spans.

GPUs commonly exist in only some locations. Because image and external-network ids are location-scoped too, moving a pool to the location that has GPUs means moving all three together — the same constraint the geo guide describes. The image there is an ordinary one; needing a GPU does not by itself mean needing a special image, unless you are taking the pre-baked driver route.

Limits¶

The type is resolved when a machine is created. Changing spec.gpu rolls the pool, as with any other machine-template change.
Scoping is per GPU type, not per performance tier within a type — the platform exposes profiles, and that is the granularity available.