Skip to content

Releasing & installing CAPVCO

Release flow (GitLab CI)

Pushing a semver tag vX.Y.Z runs the tag pipeline (.gitlab-ci.yml):

  1. test — lint-test: gofmt, go vet, go test.
  2. build — container-release builds the manager image (kaniko) and pushes $CI_REGISTRY_IMAGE:vX.Y.Z + :latest (→ ghub.gig.tech/whitesky-oss/cluster-api-provider-vco).
  3. package (everything a release needs, built before the Release exists) — two jobs:
  4. release-manifests renders + cross-compiles the bundle into out/ and uploads every file to the generic package registry under capvco/vX.Y.Z/: make release-manifests → infrastructure-components.yaml (image pinned to vX.Y.Z) + metadata.yaml + the cluster templates; make release-binaries → the CLIs capvco-imagepublish / capvco-backup (CGO_ENABLED=0, linux/darwin × amd64/arm64).
  5. build-node-image builds a pre-baked node image for every version in KUBERNETES_VERSIONS (.gitlab-ci.yml) on the kvm runner and uploads each qcow2 to capvco-images/node-images/. It is required on tags — see below.
  6. release — depends on both package jobs, so it runs only if the bundle and all node images succeeded. Creates a GitLab Release with one asset link per file (using direct_asset_path), giving stable URLs — e.g.:
  7. …/-/releases/vX.Y.Z/downloads/infrastructure-components.yaml
  8. …/-/releases/vX.Y.Z/downloads/metadata.yaml
  9. …/-/releases/vX.Y.Z/downloads/cluster-template-geo.yaml (and the other flavors)
  10. …/-/releases/vX.Y.Z/downloads/capvco-imagepublish_linux_amd64 (and the other CLIs/arches)
  11. …/-/releases/vX.Y.Z/downloads/images/capvco-ubuntu-2404-cloudimg-v1.36.4.qcow2 (and each other version)

Anything dropped into out/ is uploaded and linked automatically, so adding a template or CLI needs no .gitlab-ci.yml change; adding a node image is one entry in KUBERNETES_VERSIONS. 5. pages — builds + deploys the docs site.

Branches/MRs run test, build, and a per-commit image (:$CI_COMMIT_SHORT_SHA) only; build-node-image is available there as a manual dev build that just produces the qcow2 artifacts (never on MR pipelines).

Prereqs: the tag pipeline uses the built-in $CI_REGISTRY_* and $CI_JOB_TOKEN; no extra secrets. (build-node-image additionally needs a kvm runner — see image-builder/.)

Installing with clusterctl

CAPVCO is an out-of-tree infrastructure provider. clusterctl consumes a GitLab provider from the project's Package Registry API (over the gitlab.whitesky.cloud host), so register it in ~/.cluster-api/clusterctl.yaml:

providers:
  - name: vco
    type: InfrastructureProvider
    url: "https://gitlab.whitesky.cloud/api/v4/projects/730/packages/generic/capvco/v0.6.1/infrastructure-components.yaml"

Then:

clusterctl init --infrastructure vco

clusterctl fetches infrastructure-components.yaml and metadata.yaml from that release and installs the CRDs + manager into capvco-system. Provide the manager's instance config (VCO_PORTAL_URL, VCO_IAM_REFRESH_URL) via the deployment env / a patch; per-cluster credentials come from each VCOCluster's credentialSecretRef (plan §18), not the manager.

To pull a newer release, bump the url version and clusterctl upgrade apply.

Local dev (no release)

  • make docker-build IMG=… + push, or run the manager out-of-cluster against a kind management cluster (as used throughout M1–M5 validation).
  • make release-manifests RELEASE_TAG=vX.Y.Z produces out/ locally to inspect what a release would publish.

Node VM images in a release

A release ships a pre-baked node image (qcow2) for every Kubernetes version listed in the KUBERNETES_VERSIONS variable (.gitlab-ci.yml; currently v1.35.8 v1.36.4 v1.37.0) — this is the single source of truth. On a tag pipeline the build-node-image job runs automatically on the kvm runner, builds each image (each in its own throwaway image-builder checkout, so versions don't cross-contaminate), and uploads them to capvco-images/node-images/; the release job then attaches each at …/-/releases/vX.Y.Z/downloads/images/capvco-<os-target>-<version>.qcow2.

It is required: if any image fails to build, release never runs and no Release is published — a release always ships every node image (all-or-nothing). The trade-off is time: the multi-GB KVM builds dominate the pipeline (~25 min each, run serially), so a three-version release takes ~1.5 hours. Add or drop a shipped version by editing KUBERNETES_VERSIONS.

To build images without cutting a release, trigger build-node-image manually on a branch pipeline (it just produces the qcow2 artifacts), or run image-builder/build-node-image.sh locally on a KVM box.

Keeping KUBERNETES_VERSIONS current

Upstream Kubernetes patches roughly the three or four newest minors, each for about 14 months. KUBERNETES_VERSIONS should track that window: add a new minor when it ships, drop one when it goes EOL. Shipping an EOL version means shipping something that receives no security patches — and it has drifted before: the set sat at v1.30.14 v1.31.5 until 2026-09, by which point both were 10 and 14 months dead.

Check the window with the upstream schedule at https://kubernetes.io/releases/ (or curl -s https://endoflife.date/api/kubernetes.json, whose latest and eol fields give the newest patch and the retirement date per minor). Prefer a minor with plenty of runway: one that reaches EOL within a couple of months is not worth a multi-GB image build.

The version string is never the only thing that moves. Work down this list:

What Why it is coupled
KUBERNETES_VERSIONS the single source of truth: drives build-node-image, the release assets, and the docs
build-node-image matrix (.gitlab-ci.yml) must list the same versions — it is a literal list, not derived
KUBERNETES_VERSION / KUBERNETES_UPGRADE_TO the e2e default and the upgrade scenario's pair; must be two adjacent supported minors
e2e-getting-started matrix one cell per supported version
clusterctl / CAPI each CAPI minor supports a bounded Kubernetes range — check https://cluster-api.sigs.k8s.io/reference/versions. Kubernetes 1.37 needs CAPI >= v1.13.6; an older pin simply cannot bootstrap it
kind the kind cluster is the management cluster, and CAPI supports a narrower range for management than for workload clusters (older minors are workload-only). kind's node images must cover a version CAPI accepts as management
kubectl (fetched in CI) kubectl supports +/-1 minor, so pin the middle of the set, not the newest
cert-manager a CAPI dependency; check the CAPI release notes when bumping CAPI
docs + templates example versions are copy-pasted by readers — leaving them stale hands people an EOL cluster

Two traps worth knowing, both hit in practice:

  • image-builder main tracks new Kubernetes, not old. Its kubeadm.yml template hardcodes kubeadm.k8s.io/v1beta4, which kubeadm rejects before 1.31. Raising the floor fixes this; pinning IMAGE_BUILDER_REF to keep an EOL version buildable is the wrong direction (#38).
  • Behaviour changes at minor boundaries can make our workarounds redundant. ControlPlaneKubeletLocalMode went GA in 1.36, so the control-plane-kubelet fix in cluster-template-prebaked.yaml / -geo.yaml is probably unnecessary on the current floor. Read the CAPI release notes for the minors you are adding, and prefer deleting a workaround over carrying it — but confirm with the upgrade scenario first.

After changing the set: run bash hack/check-e2e-scenarios.sh, go test ./..., and let a full MR pipeline build and publish the new node images (#16's skip-if-published makes every run after the first cheap).

Consuming a released image

Download the qcow2 (or copy its release URL) and publish it into your whitesky.cloud location(s) with the released capvco-imagepublish CLI — the platform imports directly from the URL, so there's no S3 round-trip:

capvco-imagepublish -url "https://gitlab.whitesky.cloud/api/v4/projects/730/packages/generic/capvco-images/node-images/<file>.qcow2" \
  -name "k8s-v1.35.8" -locations qt-gen-dc01-003

Then reference the resulting image_id in VCOMachineTemplate.spec.imageId (or VCO_IMAGE_ID when rendering a template). See pre-baked images for the full workflow.