Private clusters¶

By default a CAPVCO cluster publishes its Kubernetes API on the cloudspace's external IP. Anyone on the internet can reach <external-ip>:6443; only Kubernetes authentication stands between them and the API. Many organisations don't allow that, whatever the authentication.

A private cluster publishes the API on the cloudspace's private network instead. Nothing listens on the external IP — a TCP scan of it finds nothing at all.

Decide how you will reach the cluster before you create it

A private endpoint is reachable only from inside the cloudspace, or over a link into it. A kind cluster on your laptop cannot reach one. Read Reaching a private cluster first — this is the part that bites.

How it works¶

The cloudspace's virtual gateway (VGW) owns the first host of the private network. That address is the one place the platform will let an ingress load balancer listen on the private side, so it is where the API endpoint goes:

spec:
  controlPlaneExposure: private
  cloudspace:
    privateNetwork: 192.168.150.0/24   # endpoint becomes 192.168.150.1:6443

flowchart LR
  subgraph cs["Cloudspace"]
    gw["VGW 192.168.150.1:6443"]
    cp["control-plane nodes"]
    gw --> cp
  end
  ext["External IP"]
  x["nothing published"]
  ext -.-> x

spec.controlPlaneEndpoint.host becomes the gateway address, kubeadm adds it to the API server certificate SANs automatically, and nodes join over the private network rather than hairpinning out through the public IP and back.

The gateway address is not a free choice

The backend refuses an LB frontend on any other private address — Ip address 192.168.150.10 not available on any attached external network — because the VGW holds the only real address on that network. CAPVCO derives the gateway from the CIDR; you don't pick it.

Egress is unchanged¶

This feature is about inbound. Nodes still reach the internet through the VGW to install packages and pull images, and the cloudspace still attaches an external network.

To remove egress as well, combine a private endpoint with pre-baked images and the air-gapped in-cloudspace registry, then add private: true:

spec:
  controlPlaneExposure: private
  cloudspace:
    private: true          # no external network at all

cloudspace.private is only accepted together with controlPlaneExposure: private — no other mode has anywhere to publish its endpoint, so the combination is refused with a clear message rather than producing a cluster that never becomes reachable.

It also cannot be combined with managementAccess.connectedCloudspace: the site-to-site VPN pairs the two cloudspaces by their external addresses, and this one has none. Run the management cluster inside the cloudspace in that case.

Reaching a private cluster¶

The management cluster must reach <private-network>.1:6443. Pick one:

Management cluster in the cloudspace (simplest)Connected cloudspaces (S2S)Your own VPN

Run the management cluster on a VM in the same cloudspace. It reaches the gateway address directly, with no VPN and no connection to build — leave managementAccess.connectedCloudspace.cloudspaceId empty.

If you bootstrap with kind on your laptop, you must clusterctl move onto that in-cloudspace management cluster — a kind cluster cannot reach a private endpoint, so it cannot keep managing the cluster after creation.

Run the management cluster in its own cloudspace and let CAPVCO connect the two:

spec:
  managementAccess:
    connectedCloudspace:
      cloudspaceId: <management cloudspace id>

CAPVCO joins every cloudspace it creates for the cluster to that one over the platform's site-to-site VPN — the same mechanism geo clusters already use between their own sites — and removes the connection again on delete. See Management access.

This is what the private e2e scenario runs: a k3s management cluster on a VM in its own cloudspace, managing a private workload cluster in another over the connection CAPVCO built.

Any VPN that terminates in the cloudspace private network works. CAPVCO does not manage it.

Management access¶

spec.managementAccess.connectedCloudspace is the only one of the three options CAPVCO builds itself. Give it the id of the cloudspace your management cluster runs in:

spec:
  controlPlaneExposure: private
  cloudspace:
    privateNetwork: 192.168.150.0/24
  managementAccess:
    connectedCloudspace:
      cloudspaceId: cs-7f2a…        # the MANAGEMENT cluster's cloudspace

What the provider does with it:

connects each cloudspace it created for this cluster to the management cloudspace, in both directions (one direction is not a tunnel),
records each connection in status.managementConnections, and
removes exactly those connections when the cluster is deleted.

That last point is the reason the connections are recorded rather than recomputed. Deleting the workload cloudspace takes its own half of the tunnel with it, but the management cloudspace is not CAPVCO's to delete — without an explicit removal it would collect a dead connection for every cluster that ever existed.

ManagementAccessReady on the VCOCluster reports the state. The condition is absent when no connection was asked for.

Non-overlapping CIDRs are not optional here

The tunnel routes by CIDR. An overlap between the management cloudspace's private network and this cluster's private network, pod CIDR or service CIDR does not fail an API call — it blackholes traffic, and surfaces much later as a management cluster that cannot reach a cluster that looks perfectly healthy from the inside. CAPVCO refuses to provision on an overlap.

Leave cloudspaceId empty when the management cluster is already in this cluster's own cloudspace: it is on the private network already, and nothing needs building.

Internal Service load balancers¶

A private API doesn't help if workloads publish themselves. Service type=LoadBalancer can put its frontend on the private gateway too.

Per Service, by annotation:

metadata:
  annotations:
    vcolb.infrastructure.cluster.x-k8s.io/internal: "true"

or by class, which is immutable and so can't be edited away later:

spec:
  loadBalancerClass: vco.whitesky.cloud/internal

The Service gets status.loadBalancer.ingress[0].ip = <private-network>.1.

For a fully private cluster, flip the default instead, so a Service has to ask to be public. That is a separate add-on bundle, config/addons/service-lb-internal, which runs the same vcolb controller with --default-internal-load-balancers. Apply it to the management cluster and label the workload Cluster for it:

metadata:
  labels:
    capvco.infrastructure.cluster.x-k8s.io/service-lb: vco-internal

vco-internal and vco are two values of the same label, so a cluster can never end up with both bundles. cluster-template-private.yaml sets vco-internal for you; you still have to apply the bundle, exactly as with any other add-on.

An individual Service then opts out with vcolb.infrastructure.cluster.x-k8s.io/internal: "false" or loadBalancerClass: vco.whitesky.cloud/public.

Coexisting with other load balancer implementations¶

CAPVCO now honours spec.loadBalancerClass. A Service carrying another implementation's class (MetalLB, kube-vip, …) is ignored completely — not reconciled, not finalized — so the two controllers no longer fight over status.loadBalancer. A Service with no class is still handled by CAPVCO, which is what every pre-existing Service looks like.

CIDR planning¶

The private network is now load-bearing: it carries the API endpoint and every internal Service. It must not overlap

the pod CIDR or the service CIDR (CAPVCO refuses to provision on an overlap — the CNI would otherwise hijack the node LAN), or
any network you connect it to, including the management cloudspace.

Pin it explicitly rather than letting the platform choose, so the routes you need on the other side are known before the cluster exists.

Limitations¶

Not supported with spec.geo. The geo endpoint is a DNS name pointed at each site's public per-site endpoint; making that private needs a private path between sites that doesn't exist yet. The combination is refused.
CAPVCO builds the S2S connection, not a VPN. managementAccess.connectedCloudspace is handled; reaching the cluster from a laptop or from a management cluster outside the platform is still yours to arrange.
No WireGuard integration. The platform has a per-cloudspace WireGuard API, but CAPVCO does not drive it.
cloudspace.private: true cannot be combined with managementAccess.connectedCloudspace. The connected-cloudspaces API pairs the two sides by their external addresses, and a cloudspace with no external network has none. The combination is refused rather than retried forever. A fully private cloudspace therefore needs a management cluster inside it, or a VPN you run yourself.
The ccm add-on needs its internal variant. The cloud-controller-manager runs the same Service load-balancer controller as vcolb, and the default bundle puts frontends on the cloudspace external IP — so on a private cluster every Service that forgets the annotation would publish itself, undoing the one property the cluster exists for. …/ccm: enabled is therefore refused on a private cluster; use …/ccm: internal, which is the same controller started with --default-internal-load-balancers. See internal Service load balancers.
controlPlaneExposure cannot be changed on a live cluster. This is intended, not a gap; see below.

Changing the exposure of an existing cluster¶

You can't, and CAPVCO refuses the edit rather than half-applying it. Change controlPlaneExposure on a cluster whose control-plane load balancer already exists and the VCOCluster goes to FailureReason: ControlPlaneExposureChanged, with nothing on the backend touched.

The immediate reason is that the VCO API can create and delete a load balancer but not move its frontend. The deeper one is that the endpoint address is not an implementation detail:

every kubeconfig issued for the cluster names it,
every node's kubelet bootstrap config names it,
the certificate SANs minted at kubeadm init cover it.

So even with a backend call to re-front the LB, the change would strand every existing node and every existing kubeconfig. It is a re-key of the cluster's identity, not a configuration tweak.

To recover a cluster that is wedged in this state, put the field back. The refusal is not sticky: it is recomputed each pass from the published endpoint, so once the spec agrees with what the LB actually fronts, reconciliation resumes where it left off. Nothing needs cleaning up.

If you genuinely need the other exposure, create a new cluster. You are re-issuing kubeconfigs and rejoining nodes either way, which is the bulk of the work, and a fresh cluster gets correct certificate SANs for the new endpoint for free.

Create one¶

CLUSTER_NAME=private-demo \
VCO_LOCATION=<location> \
VCO_IMAGE_ID=<image-id> \
VCO_EXTNET_ID=<external-network-id> \
VCO_PRIVATE_NETWORK=192.168.150.0/24 \
KUBERNETES_VERSION=v1.35.8 \
clusterctl generate cluster private-demo \
  --from ./templates/cluster-template-private.yaml \
  --worker-machine-count 1 > /tmp/private-demo.yaml

Verify it really is private — this is the acceptance test, and worth running:

# the endpoint is the private gateway, not the external IP
kubectl get vcocluster private-demo -o jsonpath='{.spec.controlPlaneEndpoint.host}'

# nothing listens on the cloudspace external IP
nmap -Pn -p- <cloudspace-external-ip>