Kubernetes Provider

The Kubernetes provider runs nested cluster nodes as pods inside an existing host Kubernetes cluster. It exposes the nested cluster’s API server through a stable, server-side endpoint so kubectl keeps working after the ksail process exits — no long-lived port-forward is required.

KSail picks the exposure mechanism automatically, in order of preference:

1. Gateway API (TCPRoute) — used when spec.provider.kubernetes.gatewayClassName is set and the host cluster has a TCPRoute-capable Gateway controller.
2. LoadBalancer Service — used when the host cluster has a LoadBalancer controller that assigns an external address.
3. NodePort Service — the universal fallback; the nested API is reached at https://<node-address>:<nodePort>.

The resolved address is written into the nested cluster’s kubeconfig, and is added to the nested API server’s certificate SANs so TLS verification succeeds.

When to Use the Kubernetes Provider

The Kubernetes provider is ideal when you:

• Already have a Kubernetes cluster and want to run nested clusters inside it
• Need isolated development clusters without Docker Desktop
• Want to share a host cluster across a team, each running their own nested cluster
• Run CI/CD pipelines in a Kubernetes environment (e.g., Tekton, Argo Workflows)

Note

The Kubernetes provider supports Vanilla, K3s, Talos, VCluster, and KWOK. See the Support Matrix for component compatibility.

Prerequisites

1. Host Kubernetes cluster — any distribution (cloud-managed, on-prem, or local)

2. Privileged pod policy — nested cluster pods require privileged: true security context

3. A reachable exposure address — the resolved Gateway/LoadBalancer/NodePort address must be routable from the machine running ksail. This is automatic for cloud host clusters and for local hosts that publish service addresses to the host (e.g. Docker Desktop, or cloud-provider-kind). On local Docker-based hosts on macOS/Windows, node and LoadBalancer IPs are frequently not routable from the host — prefer Docker Desktop’s localhost LoadBalancer, cloud-provider-kind, or a routable Gateway address.

Optional — Gateway API exposure (preferred tier; used when spec.provider.kubernetes.gatewayClassName is set):

4. Gateway API experimental CRDs (TCPRoute is in the experimental channel) — install from gateway-api.sigs.k8s.io:

   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/experimental-install.yaml

5. Gateway controller with TCPRoute support — e.g. Envoy Gateway, Cilium, or Istio. The controller must implement TCPRoute (v1alpha2), not only HTTPRoute.
6. GatewayClass — must exist on the host cluster (created by the Gateway controller)

When gatewayClassName is empty (the default), KSail falls back to a LoadBalancer Service, then a NodePort Service.

Configuration

Set provider: Kubernetes in ksail.yaml and configure the host cluster connection under spec.provider.kubernetes:

apiVersion: ksail.io/v1alpha1
kind: Cluster
metadata:
  name: my-nested-cluster
spec:
  cluster:
    distribution: K3s          # or Vanilla, Talos, VCluster
    provider: Kubernetes
  provider:
    kubernetes:
      kubeconfig: ~/.kube/config        # host cluster kubeconfig
      context: my-host-context          # host cluster context (optional)
      gatewayClassName: eg              # GatewayClass on the host cluster
      podCidr: "10.64.0.0/16"          # nested pod CIDR (must not overlap host)
      serviceCidr: "10.128.0.0/16"     # nested service CIDR (must not overlap host)
      persistence:
        enabled: false                  # true for PVC-backed persistent clusters
        storageClassName: ""            # empty = cluster default StorageClass
        size: "20Gi"                    # PVC size when persistence is enabled

Host Cluster Authentication

Field	Env Var	Default	Description
kubeconfig	KSAIL_HOST_KUBECONFIG	~/.kube/config	Path to host cluster kubeconfig
context	KSAIL_HOST_CONTEXT	current context	Host cluster kubeconfig context

Caution

spec.provider.kubernetes.kubeconfig and spec.provider.kubernetes.context configure the host cluster connection. spec.cluster.connection.kubeconfig and spec.cluster.connection.context configure the nested cluster connection.

CIDR Configuration

Nested cluster CIDRs must not overlap with the host cluster’s pod or service CIDRs. The defaults below are chosen to avoid conflicts with common host cluster ranges.

Default	Range	Purpose
10.64.0.0/16	Pod CIDR	IP addresses for nested cluster pods
10.128.0.0/16	Service CIDR	ClusterIP addresses for nested cluster services

Common host cluster CIDR ranges to avoid:

• 10.244.0.0/16 — default pod CIDR (Flannel, Calico)
• 10.96.0.0/12 — default service CIDR (kubeadm)

Persistence

By default, nested clusters are ephemeral — all state is lost on pod restart. This matches KSail’s GitOps-first philosophy where cluster state is reconstructible from Git.

Enable persistence for long-lived environments:

provider:
  kubernetes:
    persistence:
      enabled: true
      size: "20Gi"    # includes etcd + containerd image cache

Architecture

Each nested cluster runs in a dedicated namespace on the host cluster. The namespace prefix depends on the distribution:

Distribution	Namespace	Execution Mode	Description
K3s	k3k-<name>	Direct Pod	k3k operator creates K3s server in privileged pod; K3s binary runs directly
Vanilla (Kind)	ksail-<name>	Docker-in-Docker	Docker daemon sidecar, Kind SDK operates against it
Talos	ksail-<name>	Docker-in-Docker	Docker daemon sidecar, Talos SDK operates against it
VCluster	vcluster-<name>	Helm release	VCluster deployed via Helm driver on host cluster

graph TB
    subgraph "Host Kubernetes Cluster"
        subgraph "ksail-my-cluster namespace"
            CP["Control Plane Pod<br/>(privileged)"]
            W1["Worker Pod 1<br/>(privileged)"]
            SVC["ClusterIP Service<br/>→ port 6443"]
        end
        GW["Gateway<br/>TCP listener :6443"]
        ROUTE["TCPRoute<br/>→ ClusterIP Service"]
        GWCTRL["Gateway Controller"]
    end

    USER["kubectl → LB:6443"] --> GW
    GW --> ROUTE --> SVC --> CP
    GWCTRL -.-> GW

API Server Exposure

The nested cluster’s API server is exposed through the first mechanism that yields a stable address:

1. Gateway API (when gatewayClassName is set) — a ClusterIP Service targeting the control-plane pod(s), a Gateway (TCP listener on the nested API server port), and a TCPRoute routing the listener to the Service. The Gateway controller’s LoadBalancer provides the external address.
2. LoadBalancer Service — a Service of type LoadBalancer targeting the control-plane pod(s); the host LoadBalancer controller assigns the external address.
3. NodePort Service — a Service of type NodePort; KSail resolves a reachable node address (preferring a node ExternalIP, then the host cluster’s API address, then a node InternalIP).

In every case the resolved address is added to the nested API server’s certificate SANs and written to the nested cluster’s kubeconfig, so kubectl verifies TLS and connects directly — the connection survives the ksail process exiting.

Operations

Start / Stop

Start and stop are not supported for the Kubernetes provider. Nested cluster pods are managed by their respective operators/controllers and cannot be independently stopped/started. Use delete and create to manage cluster lifecycle.

Delete

ksail cluster delete   # Delete namespace and all resources

List Clusters

ksail cluster list --provider Kubernetes

This lists all nested clusters discovered on the host cluster. The host kubeconfig is resolved from the KSAIL_HOST_KUBECONFIG environment variable, or ~/.kube/config by default. The context is resolved from KSAIL_HOST_CONTEXT.

Troubleshooting

Gateway exposure was requested but a NodePort/LoadBalancer was used instead — When spec.provider.kubernetes.gatewayClassName is set but the Gateway tier cannot be configured (e.g. the experimental Gateway API CRDs or a TCPRoute-capable controller are missing, or the GatewayClass does not exist), KSail prints a warning (Gateway API exposure failed (...); falling back to LoadBalancer/NodePort) and falls back to the next tier rather than failing the create. To force Gateway exposure, install the experimental CRDs (kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/experimental-install.yaml), ensure a TCPRoute-capable controller is running, and set gatewayClassName to a class that exists (kubectl get gatewayclasses). To check which tier was selected, inspect the created resources: kubectl get svc,gateway,tcproute -n <namespace>.

kubectl times out connecting to the nested cluster — The resolved exposure address may not be routable from your machine. This is common with NodePort/LoadBalancer addresses on local Docker-based host clusters on macOS/Windows, where service addresses live inside the Docker VM. Use Docker Desktop’s localhost LoadBalancer, install cloud-provider-kind, or configure a TCPRoute-capable Gateway whose address is routable. Inspect the exposure: kubectl get svc,gateway,tcproute -n <namespace>.

nested cluster CIDR overlaps — Change spec.provider.kubernetes.podCidr and/or spec.provider.kubernetes.serviceCidr to ranges that don’t overlap with your host cluster. Check host CIDRs: kubectl cluster-info dump | grep -m 1 service-cluster-ip-range

Pods stuck in Pending — The host cluster may not allow privileged pods. Check PodSecurityStandard: kubectl get ns <namespace> -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}'

Pod CrashLoopBackOff — Check pod logs: kubectl logs -n ksail-<cluster-name> <pod-name>. Common causes: cgroup misconfiguration (ensure cgroup v2), insufficient resources, or AppArmor blocking nested container operations.