Skip to content

SR-IOV networking

Pods can get a secondary interface on a whitesky.cloud external network — the path to an SR-IOV NIC for high-throughput or low-latency workloads.

How it works, and why it is per pod

SR-IOV is not a distinct concept in the platform API: an SR-IOV NIC is a NIC on an external network whose backing happens to be SR-IOV. So there is nothing SR-IOV-specific to ask for — you name an external network, and whether it yields a VF is a property of that network.

NICs can be attached and detached while a VM runs, so CAPVCO attaches them per pod, at CNI time, rather than pre-attaching them to every node in a pool:

  • VFs are scarce. Pre-attaching N per node wastes them on nodes whose pods never ask, and caps SR-IOV pods per node at N.
  • One worker pool serves both SR-IOV and ordinary workloads.
  • Adding a network to a live cluster needs no machine rollout.

Two components do it. A node agent (DaemonSet, workers only) holds the credential, serialises attaches per VM, and reclaims NICs left by pods that died without a teardown. A CNI plugin (vco-extnic) runs on the host, finds the interface that appeared, moves it into the pod's namespace, and configures the address the backend assigned. Multus routes the per-pod request to it.

Not the upstream SR-IOV stack

sriov-network-device-plugin discovers VFs the host already has. Here nothing exists until a pod asks, so discovery has nothing to find — the attach is part of pod setup, which is why this is a whitesky-specific plugin.

Enabling it

The add-on needs two things: the label, and a credential you supply.

# 1. A credential for the agent, scoped to this cluster's cloudspaces.
kubectl create secret generic sriov-agent-credential \
  --from-literal=jwt="<cloudspace-scoped JWT>" \
  --from-literal=customer_id="<customer>"

# 2. Point the cluster at it, and enable the add-on.
kubectl patch vcocluster my-cluster --type=merge \
  -p '{"spec":{"sriov":{"credentialSecretRef":"sriov-agent-credential"}}}'
kubectl label cluster my-cluster \
  capvco.infrastructure.cluster.x-k8s.io/sriov=enabled

The credential is not defaulted, on purpose

Every other add-on falls back to the cluster credential. This one does not: the agent runs on every worker, and a Secret mounted on a node is readable by anything with node-level access. Defaulting would put customer-wide access on every worker. Scope this credential to the cluster's cloudspaces, and read known limitations before you deploy it — the residual node-level exposure is real and documented.

Giving a pod an interface

Declare the network once:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-net
spec:
  config: '{"cniVersion":"1.0.0","name":"sriov-net","type":"vco-extnic","externalNetworkId":6371}'

Then annotate pods that want it:

apiVersion: v1
kind: Pod
metadata:
  name: fast-app
  annotations:
    k8s.v1.cni.cncf.io/networks: sriov-net
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ["sh","-c","ip -o addr show; sleep 3600"]

kubectl exec fast-app -- ip -o addr show should list eth0 (the cluster network, via Calico) and a second interface holding an address from the external network. Addressing comes from the backend at attach time, so there is no IPAM plugin to configure.

The secondary interface deliberately does not take the pod's default route — that stays with the primary CNI, or cluster networking would break.

Operational notes

  • Which NIC vendors work. The agent identifies a new interface by the MAC the backend reports, not by driver or interface name, so Mellanox (mlx5_core), Intel (iavf) and Broadcom (bnxt_en) all work — provided the node image carries that vendor's VF module.
  • Pod start is slower. CNI ADD waits on a VM API call plus guest device enumeration. Exhaustion of the external network shows up as a pod stuck in ContainerCreating; check the agent's logs:
kubectl -n whitesky-sriov-cni logs ds/capvco-sriov-agent
  • Leaks are reclaimed automatically. A pod that dies without a CNI teardown leaves the NIC attached. The agent sweeps every 5 minutes, detaching NICs whose pod namespace no longer exists. Allocations with no recorded namespace are never swept — detaching from a running pod would be worse.
  • Control-plane nodes do not run the agent. It exists for pods, and keeping the credential off control-plane nodes narrows where it sits.

Limits

  • CAPVCO cannot verify that a given external network is SR-IOV-backed. A wrong id yields an ordinary NIC, not an error.
  • The agent must be running before a pod can get an interface; a pod scheduled during a rollout of the DaemonSet retries until it is.