Skip to content

Known limitations

CAPVCO is honest about its edges. This page collects the current caveats, platform constraints, and deferred work in one place, so you can plan around them. Each is cross-linked from the relevant guide.

Storage & encryption

  • CSI disk-create needs a location-scoped credential. The driver creates disks via a location-scoped API path, so a purely cloudspace-scoped credential can attach/snapshot but not create disks (a genuine 403). A CSI-enabled cluster therefore needs a location grant, which spans every cloudspace in that location — slightly widening the blast radius beyond one cloudspace. This is the one isolation trade-off in the design. → Storage
  • Injected CSI credential = tenant blast radius. The workload cluster's own admin can read the injected csi-driver-secret and use the credential directly against the whitesky.cloud API. Scope the per-cluster credential accordingly.
  • Encrypted (LUKS2) volumes need cryptsetup in the driver image. All provider-side wiring (transit login, encrypt/decrypt, the encrypted StorageClass) is validated, but the reference …/csi-driver:latest image historically shipped without cryptsetup, in which case encrypted volumes fail to format. → Storage
  • No CSI volume snapshots yet. A VolumeSnapshotClass / snapshotter is not shipped (it needs the snapshot CRDs + snapshot-controller). Use etcd backup for cluster-level DR.
  • managed vault mode is deferred. Only none and external encryption vault modes are wired; managed (CAPVCO deploys + operates the vault) is future work. Per-cluster vault kubernetes-auth wiring is currently a manual step. → OpenBao operations

Geo & DNS

  • Fast failover is bounded by a hard 1-hour DNS TTL. The platform hard-sets a 3600 s TTL with no API override. External clients can take up to an hour to fail over to a different control-plane site on permanent primary-IP loss (transient recovery reusing the same IP is ~5 min). Mitigate for the management plane with the no-cache resolver config in config/mgmt-dns — a required production step. → Geo failover
  • whitesky.cloud DNS writes are slow (~30 s each). A multi-record ExternalDNS changeset can take minutes; webhook/external-dns timeouts are set generously to absorb this. → ExternalDNS
  • VM images are location-scoped and non-portable. Geo clusters need a per-location image id (spec.geo.locationImages); every site's location must have the matching image published; a same-location geo cluster just needs it in that one location. → Pre-baked images
  • Geo Service load balancing isn't site-aware yet. The Service-LB controller uses the first node's cloudspace; per-cloudspace LBs for geo are future work. → Load balancing

Load balancing

  • The platform L4 LB has no health checks, failover, or per-host weighting. CAPVCO compensates with controller-driven, health-aware backend-pool membership (and, for geo, the apiserver-proxy). LB-level health checks and session affinity aren't available as LB features. → Load balancing
  • Per-Service dedicated public IPs aren't wired. The frontend uses the cloudspace external-network IP; per-Service public IPs need an attached external network / public-IP quota.

Air-gapped

  • Fully private cloudspaces (spec.cloudspace.private: true) are rejected today. The endpoint has nowhere public to live and the management cluster can't yet reach an internal-only endpoint. The air-gapped path runs on a normal cloudspace with an external endpoint but zero workload egress. → Air-gapped

GPUs

  • Reservations are reclaimed by name, so renaming one orphans it. A vGPU is billed from the moment it is reserved — detaching one does not stop the charge, it only makes the reservation unassigned — and the platform API has no owner field on a reservation. So the provider encodes ownership in the name:
capvco_<namespace>_<cluster>_<machine>_gpu_<n>

Underscore is the separator because a Kubernetes namespace, cluster or machine name cannot contain one, which makes ownership an exact test rather than a prefix guess. Cluster teardown releases every reservation it owns — attached or not, including ones left by a machine that was force-deleted with kubectl delete vcomachine --force — and touches nothing else.

The consequence: rename a reservation in the portal and the provider can no longer see it as its own. It will then survive teardown and keep billing. Find any such stragglers with:

capvco-gpucheck -location <location> -orphans

which lists every unattached capvco* reservation and exits non-zero if there are any. Release one with DELETE /customers/<customer>/locations/<location>/vgpus/<id>?permanently=true.

  • No GPU node image is shipped. A GPU needs a driver, the container toolkit and a containerd nvidia runtime on the node. spec.gpu.driverVersion installs them on first boot from Ubuntu's archive and NVIDIA's apt repo, which needs egress — so an air-gapped GPU cluster requires an image you build yourself with image-builder/. CAPVCO ships node images without GPU drivers. → GPUs
  • Mediated vGPU needs a licensed driver CAPVCO cannot install. If the platform hands out vGPU slices rather than full passthrough, the guest needs NVIDIA's licensed guest driver, which is not in any public archive. driverVersion cannot help; the only path is a self-built image carrying that driver, matched to the host's vGPU Manager branch. Confirm which mode you have with lspci -nn | grep -i nvidia on a GPU-attached VM before planning GPU workloads.
  • A stub GPU profile attaches but cannot run anything. Reserve, attach, status.gpus and release all work against a placeholder profile — enough to validate the provider — but there is no driver to install and no device to expose, so leave driverVersion empty and set VCO_GPU_ADDON=none. A driver version set against a stub installs fine and then fails nvidia-smi, which fails the node's boot. → GPUs

SR-IOV networking

  • The SR-IOV agent's credential is readable from the node it runs on. SR-IOV NICs are attached per pod at CNI time, so the agent runs on every worker — beside tenant workloads — and reads its credential from a mounted Secret. A Secret mounted into a pod lands on the node's filesystem, which namespaces do not partition: anything with node-level access reads it, including node root and any pod that can use hostPath or run privileged, from any namespace. The agent's own namespace (whitesky-sriov-cni) plus RBAC stops ordinary workloads reading it through the API, and is worth having, but it does not close this path. Enforce PodSecurity restricted outside the agent's namespace, and do not grant tenants the ability to run privileged or hostPath pods.
  • The agent's credential is operator-supplied, and its scope is the operator's responsibility. spec.sriov.credentialSecretRef is required when the add-on is enabled and is deliberately not defaulted to the cluster credential — defaulting would put customer-wide access on every worker. Scope it to the cluster's cloudspaces. CAPVCO cannot verify how narrow it is: a customer-wide credential placed here works exactly as well and exposes exactly as much.
  • How narrow "narrow" can get. The platform's role model has no per-action granularity — permissions are create/read/update/delete/execute, and scoping comes from the resource grant (customer, location, cloudspace, objectspace, vm, disk). So "may attach NICs" is not expressible. A cloudspace-scoped credential is the practical floor: if it leaks, the holder controls that cluster's cloudspaces and nothing beyond them — no other cluster, no other location, not the customer account.
  • Compare with the CCM. The Service-LB / CCM / ExternalDNS credential is a copy of the customer credential, but that component is pinned to control-plane nodes, away from tenant pods. The SR-IOV agent cannot be, which is why it takes a separate credential instead of reusing that one.
  • NIC drivers in the guest. SR-IOV VFs surface as whatever the host's NIC vendor provides; supported targets are Mellanox (mlx5_core), Intel (iavf) and Broadcom (bnxt_en). The agent identifies a newly attached interface by the MAC the backend reports, not by driver or interface name, so a vendor CAPVCO has not seen still works — but the node image must carry that vendor's VF module.

Operator privileges

  • Image publishing and the SSL-certificate store are operator-privileged (403 for a per-cluster customer credential). Use a separate operator credential (spec.geo.imagePublishCredentialSecretRef, spec.registry.certCredentialSecretRef) or have your whitesky.cloud operator perform those out-of-band. → Configuration

Upgrades

  • The control-plane upgrade skew fix lives only in the prebaked and geo templates. The local-apiserver postKubeadmCommands fix that avoided the v1.30→v1.31 deadlock needs moving into the controller cloud-init so every control-plane machine (stock/backup/ccm/airgapped) gets it. Until then, carry that block if you build control-plane nodes from another template.

    Note this may resolve itself: kubeadm's ControlPlaneKubeletLocalMode is GA from Kubernetes 1.36, so on the versions CAPVCO now ships the workaround is probably unnecessary and the right outcome is to delete it rather than propagate it. Confirm with an upgrade e2e across the 1.35 → 1.36 boundary before doing either. → HA control plane

Pivot (clusterctl move)

  • Two VCOClusters with the same namespace/name will both adopt the same backend cloudspace, and either one's deletion destroys it. Adoption by deterministic name is deliberate — it is what lets a cluster recover its own infrastructure after a move drops status — but the backend has no way to tell a legitimate recovery from a duplicate object created by a GitOps mis-render or a second management cluster. The ambiguity is inherent to name-based adoption and is shared with the upstream CAPI providers. The provider does not block it; it records status.cloudspaces[].adopted and emits an AdoptedLiveCloudspace Warning event when the cloudspace it adopted was already running VMs. Never run two management clusters against the same cluster name, and treat that event as an incident.

  • Pivot coverage is new. No CI scenario performed a clusterctl move until now, which is why #39 and #40 reached a release. The pivot feature-matrix scenario closes that: it moves a loadbalancer-exposed cluster to a second kind management cluster and asserts the server pool and load balancer are adopted (same ids, not recreated) and that no provider id changes on either side. It runs two kind clusters at once, so the runner needs roughly double the inotify headroom.

  • A GitOps-installed provider has no clusterctl Provider inventory, so clusterctl move refuses until those records are registered on the target. Not a defect — the inventory is a clusterctl-only concept — but it surprises everyone installing through Flux, Argo or Sylva. → Installing with GitOps

Autoscaling

  • Scale-down of a node holding an RWO volume is not safe yet. The autoscaler drains and removes an idle node; a pod with a ReadWriteOnce CSI volume then has to reattach its disk elsewhere. That path is blocked by the same defect as everything else RWO here — PVs carry no nodeAffinity and the driver reports no PUBLISH_UNPUBLISH_VOLUME, so the rescheduled pod can hang on a volume the old node has not cleanly released. Keep stateful workloads out of an autoscaled pool, or rely on --skip-nodes-with-local-storage plus a PodDisruptionBudget. → Autoscaling
  • A pool that hosts kube-system Deployments has a hard floor of one node. coredns cannot run on the tainted control plane, so it lands on workers, and --skip-nodes-with-system-pods (on by default, and the safe setting) makes any node carrying it unremovable. At its default two replicas it pins one per worker and the pool never shrinks at all. Reaching zero needs coredns living somewhere else. → Autoscaling
  • A GPU pool only scales from zero when the requested type names its vendor. Pinning a pool with spec.gpu.profileId, or asking for a vendor whose device plugin does not ship yet (#36), leaves the GPU out of the published capacity — the pool then does not scale up from zero for a GPU pod.
  • Per-site autoscaling on geo clusters is untested. A geo cluster has one worker pool per site; each needs its own annotations, and that combination has never been run.
  • cluster-autoscaler has no 1.37 release yet, so a Kubernetes 1.37 cluster runs the 1.36 image.

Operational

  • Teardown is non-atomic and geo un-mesh is best-effort. Cloudspace delete requires sweeping VMs first (handled defensively); geo VPN-link deletes can 400/500 and are tolerated (the cloudspace delete cascade is authoritative). Geo teardown is noticeably slow.
  • The "public" endpoint IP may be RFC1918 on some instances (provider-network-routable, not internet-routable). Management→workload reachability then depends on network position — a truly public endpoint needs a floating/public IP or the management cluster on the same routable network.

None of these block the core promise — geo-redundant, backed-up, storage-enabled clusters — they're the honest edges. Most are platform constraints CAPVCO works around rather than bugs. If one blocks you, check the plan or open an issue.