Advanced Usage
This document provides advanced usage information for k3k, including detailed use cases and explanations of the Cluster resource fields for customization.
Customizing the Cluster Resource
The Cluster resource provides a variety of fields for customizing the behavior of your virtual clusters. See CRD documentation for the full specifications.
Most of these customization options can also be configured using the k3kcli tool. Refer to k3kcli documentation for more details.
|
This example creates a "shared" mode K3k cluster with:
-
3 servers
-
K3s version v1.31.3-k3s1
-
Custom network configuration
-
Deployment on specific nodes with the
nodeSelector -
kube-apiexposed using an ingress -
Custom K3s
serverArgs -
ETCD data persisted using a
PVC
apiVersion: k3k.io/v1beta1
kind: Cluster
metadata:
name: my-virtual-cluster
namespace: my-namespace
spec:
mode: shared
version: v1.31.3-k3s1
servers: 3
tlsSANs:
- my-cluster.example.com
nodeSelector:
disktype: ssd
expose:
ingress:
ingressClassName: nginx
annotations:
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/ssl-redirect: "HTTPS"
clusterCIDR: 10.42.0.0/16
serviceCIDR: 10.43.0.0/16
clusterDNS: 10.43.0.10
serverArgs:
- --tls-san=my-cluster.example.com
persistence:
type: dynamic
storageClassName: local-pathmode
The mode field specifies the cluster provisioning mode, which can be either shared or virtual. The default mode is shared.
-
sharedmode: In this mode, the virtual cluster shares the host cluster’s resources and networking. This mode is suitable for lightweight workloads and development environments where isolation is not a primary concern. -
virtualmode: In this mode, the virtual cluster runs as a separate K3s cluster within the host cluster. This mode provides stronger isolation and is suitable for production workloads or when dedicated resources are required.
version
The version field specifies the Kubernetes version to be used by the virtual nodes. If not specified, K3k will use the same K3s version as the host cluster. For example, if the host cluster is running Kubernetes v1.31.3, K3k will use the corresponding K3s version (e.g., v1.31.3-k3s1).
servers
The servers field specifies the number of K3s server nodes to deploy for the virtual cluster. The default value is 1.
agents
The agents field specifies the number of K3s agent nodes to deploy for the virtual cluster. The default value is 0.
In shared mode, this field is ignored, as the Virtual Kubelet acts as the agent, and there are no K3s worker nodes.
|
nodeSelector
The nodeSelector field allows you to specify a node selector that will be applied to all server/agent pods. In shared mode, the node selector will also be applied to the workloads.
expose
The expose field contains options for exposing the API server of the virtual cluster. By default, the API server is only exposed as a ClusterIP, which is relatively secure but difficult to access from outside the cluster.
You can use the expose field to enable exposure via NodePort, LoadBalancer, or Ingress.
TLS passthrough is required
The K3s API server authenticates clients with mTLS certificates, so the TLS connection must reach the API server untouched. An ingress controller that terminates TLS will break both kubectl and agent authentication. Any controller used with expose.ingress must therefore support TLS passthrough.
expose.ingress also requires at least one DNS name in spec.tlsSANs: the SANs are used as the Ingress hosts, and the Ingress API does not accept IP addresses there. If tlsSANs is empty or contains only IPs, the generated Ingress is rejected by the API server and the cluster fails to provision.
Nginx
Supported directly through expose.ingress. The ingress controller has to be started with the --enable-ssl-passthrough flag, and the annotations enabling passthrough must be set on the Ingress:
spec:
tlsSANs:
- my-cluster.example.com
expose:
ingress:
ingressClassName: nginx
annotations:
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/ssl-redirect: "HTTPS"Traefik
Traefik cannot perform layer 4 TLS passthrough with a standard Ingress, so expose.ingress does not work with it.
Use expose.loadBalancer or expose.nodePort instead, or leave expose unset and create a Traefik IngressRouteTCP pointing at the cluster’s ClusterIP service:
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
name: my-virtual-cluster
namespace: my-namespace
spec:
entryPoints:
- websecure
routes:
- match: HostSNI(`my-cluster.example.com`)
services:
- name: k3k-my-virtual-cluster-service # k3k-<cluster-name>-service
port: 443
tls:
passthrough: trueSee How-to: Expose Virtual Cluster Kubernetes API for more ways to expose the API server.
clusterCIDR
The clusterCIDR field specifies the CIDR range for the pods of the cluster. The default value is 10.42.0.0/16 in shared mode, and 10.52.0.0/16 in virtual mode.
serviceCIDR
The serviceCIDR field specifies the CIDR range for the services in the cluster. The default value is 10.43.0.0/16 in shared mode, and 10.53.0.0/16 in virtual mode.
In shared mode, the serviceCIDR should match the host cluster’s serviceCIDR to prevent conflicts and in virtual mode both serviceCIDR and clusterCIDR should be different than the host cluster.
|
Using the cli
You can check the k3kcli documentation for the full specs.