Kubernetes Cluster
The k8s_cluster resource creates Kubernetes clusters running in Docker containers using K3s. These clusters provide isolated Kubernetes environments for hands-on learning and testing scenarios.
Use Cases
Section titled “Use Cases”As a lab author, you can use Kubernetes clusters to provide container orchestration environments:
- Kubernetes Learning: Provide hands-on experience with kubectl, deployments, services, and other Kubernetes concepts
- Application Deployment: Deploy and manage complex multi-tier applications using Kubernetes manifests
- Cluster Administration: Learn cluster management, RBAC, networking policies, and resource management
Kubernetes clusters enable realistic container orchestration scenarios within controlled lab environments.
HCL Syntax
Section titled “HCL Syntax”Basic Syntax
Section titled “Basic Syntax”resource "k8s_cluster" "dev" { network { target = resource.network.main }}Full Syntax
Section titled “Full Syntax”resource "k8s_cluster" "staging" { image { name = "rancher/k3s:v1.31.4-k3s1" }
network { target = resource.network.main ip_address = "10.0.20.100" }
resources { cpu = 2000 # 2 CPUs memory = 4096 # 4GB }
copy_image { name = "myapp/web:latest" }
port { local = 80 host = 8180 }
volume { source = "./config/k8s" destination = "/etc/config" type = "bind" }
environment = { CLUSTER_INIT = "true" }
config { docker { no_proxy = ["internal.registry.com"] insecure_registries = ["internal.registry.com:5000"] } }}Fields
| Field | Type | Required | Description |
|---|---|---|---|
network | block | — | Connect this resource to a specific network so it can communicate with other containers or servicesrepeatable |
image | block | — | Override the image the cluster nodes run. Leave empty to use the default K3s image. |
client_ | number | — | Add worker nodes for labs that teach scheduling, placement, or failover. 0 (default) is a single all-in-one node; 1 or more makes the server a dedicated control plane with that many workers. Each node adds startup time. |
volume | block | — | Share files from the lab repository into the cluster nodes, or persist data written by themrepeatable |
copy_ | block | — | Preload images into the cluster so pods start without pulling from a registry. Speeds up the lab and lets it run offline.repeatable |
port | block | — | Expose specific ports so users can reach services running in the clusterrepeatable |
port_ | block | — | Specify a range of ports to open for applications that require multiple connections, such as node ports or multi-service environmentsrepeatable |
resources | block | — | Cap the CPU, memory, and disk the cluster can consume |
environment | map(string) | — | Set environment variables to pass configuration values, API keys, or secrets to the cluster nodes |
config | block | — | Low-level settings for the container runtime backing the cluster nodes |
Network
Network connections for the cluster
| Field | Type | Required | Description |
|---|---|---|---|
id | string | — | ID of the network to attach the container |
target | referencenetwork | — | The network to attach the container to |
ip_ | string | — | Static IP address to assign container for the network, the ip address must be within range defined by the network subnet. If this parameter is omitted an IP address will be automatically assigned. ConstraintsMust be a valid IPv4 address (e.g., 192.168.1.10) |
aliases | list(string) | — | Aliases allow alternate names to specified for the container |
Image
Custom container image for the cluster nodes
| Field | Type | Required | Description |
|---|---|---|---|
name | string | ✓ | Full image reference, including the registry and an optional tag. Include a specific tag to ensure consistent lab environments. |
username | string | — | Only required for private registries. Leave empty for public images. |
password | string | — | Only required for private registries. Leave empty for public images. |
Volume
Volume mounts for the cluster
| Field | Type | Required | Description |
|---|---|---|---|
source | string | ✓ | The path to mount into the cluster nodes. Can be relative to the file declaring the cluster (`./`) or absolute (`/usr/local/bin`). |
destination | string | ✓ | Absolute path inside the cluster nodes where the source is mounted |
type | string | — | The type of the mount, can be one of the following values: - bind: bind the source path to the destination path in the node - volume: source is a Docker volume - tmpfs: create a temporary filesystem |
read_ | bool | — | Prevent the cluster nodes from writing to this mount |
bind_ | string | — | Configures bind propagation for Docker volume mounts, only applies to bind mounts, can be one of the following values: - shared - slave - private - rslave - rprivate For more information please see the Docker documentation |
bind_ | bool | — | Mount only the source directory itself, without the mounts nested below it. Leave off unless nested host mounts are leaking into the node. |
selinux_ | string | — | Configures SELinux relabeling for the mount (usually specified as :z or :Z) and can be one of the following values: - shared (Equivalent to :z) - private (Equivalent to :Z) |
Copy Image
Container images to pre-load into the cluster
| Field | Type | Required | Description |
|---|---|---|---|
name | string | ✓ | Full image reference, including the registry and an optional tag. Include a specific tag to ensure consistent lab environments. |
username | string | — | Only required for private registries. Leave empty for public images. |
password | string | — | Only required for private registries. Leave empty for public images. |
Port
Port mappings for the cluster
| Field | Type | Required | Description |
|---|---|---|---|
local | string | ✓ | The port the service listens on inside the cluster |
host | string | — | The host port to map the local port to |
protocol | string | — | The protocol to use when exposing the port |
Port Range
Port range mappings for the cluster
| Field | Type | Required | Description |
|---|---|---|---|
range | string | ✓ | The port range to expose, e.g. `8080-8082` would expose the ports `8080`, `8081`, `8082` |
enable_ | bool | — | Expose the port range on the host |
protocol | string | — | The protocol to use when exposing the port |
Resources
Resource constraints for the cluster
| Field | Type | Required | Description |
|---|---|---|---|
cpu | number | — | Limit how much CPU power the cluster can use to optimize performance and cost. 1 CPU core = 1000. |
cpu_ | list(number) | — | Pin the cluster to specific CPU cores for more predictable performance or isolation |
memory | number | — | Specify how much memory the cluster can use to prevent crashes or resource contention |
disk | number | — | Cap the disk space the cluster can consume |
Config
Cluster configuration
| Field | Type | Required | Description |
|---|---|---|---|
docker | block | — | Configure the Docker daemon inside the cluster nodes |
Docker
Docker daemon configuration
| Field | Type | Required | Description |
|---|---|---|---|
no_ | list(string) | — | Hosts that should bypass the proxy, for example an internal registry reachable directly |
insecure_ | list(string) | — | Registries the nodes may pull from over plain HTTP or with an untrusted certificate |
KubeConfig
Kubernetes configuration details
Computed Attributes
These attributes are set by the system and are read-only.
| Attribute | Type | Description |
|---|---|---|
network[].name | string | Name will equal the name of the network as created by jumppad |
network[].assigned_address | string | AssignedAddress will equal if IPAddress is set, else it will be the value automatically assigned from the network |
copy_image[].id | string | Image ID from local registry |
api_port | number | Kubernetes API server port |
connector_port | number | Connector service port |
container_name | string | FQDN of the cluster container |
client_container_name | list(string) | Fully qualified docker addresses for the client nodes |
external_ip | string | External IP address of the Docker host |
kube_config.path | string | Path to kubeconfig file |
kube_config.ca | string | Certificate authority data |
kube_config.client_certificate | string | Client certificate data |
kube_config.client_key | string | Client key data |
Image Caching
Section titled “Image Caching”Kubernetes clusters use a global image cache to optimize bandwidth and performance:
- Each cluster node has its own Docker image cache
- Images are pulled through a shared cache proxy
- After the first pull, subsequent pulls come from the cache
- Use
copy_imageblocks to pre-populate cluster with local images
Examples
Section titled “Examples”Basic Single-Node Cluster
Section titled “Basic Single-Node Cluster”resource "network" "main" { subnet = "10.0.20.0/24"}
resource "k8s_cluster" "dev" { network { target = resource.network.main }}Multi-Node Production-Like Cluster
Section titled “Multi-Node Production-Like Cluster”resource "k8s_cluster" "production" { # A control-plane server and two worker nodes client_nodes = 2
network { target = resource.network.main }
# Ensure adequate resources for multi-node clusters resources { cpu = 4000 # 4 CPUs per node memory = 8192 # 8GB per node }
volume { source = "./config/k8s" destination = "/etc/kubernetes/manifests-extra" type = "bind" read_only = true }
environment = { K3S_TOKEN = "my-cluster-token" }}Cluster with Custom Image and Registry
Section titled “Cluster with Custom Image and Registry”resource "k8s_cluster" "custom" { image { name = "rancher/k3s:v1.25.3-k3s1" username = "registry_user" password = "registry_pass" }
network { target = resource.network.main }
# Configure registry access for private images config { docker { insecure_registries = ["internal.company.com:5000"] no_proxy = ["internal.company.com"] } }
copy_image { name = "internal.company.com:5000/myapp:v1.2.3" }}Cluster with Application Pre-loading
Section titled “Cluster with Application Pre-loading”resource "k8s_cluster" "app" { network { target = resource.network.main }
# Pre-load application images copy_image { name = "nginx:1.27" }
copy_image { name = "redis:7-alpine" }
copy_image { name = "postgres:15" }
# Expose common ports port { local = 80 host = 8080 }
port { local = 443 host = 8443 }}Terminal Access
Section titled “Terminal Access”Since terminals cannot directly target k8s_cluster resources, use a container with kubectl:
resource "k8s_cluster" "dev" { network { target = resource.network.main }}
# Container with kubectl configuredresource "container" "kubectl" { image { name = "bitnami/kubectl:latest" }
network { target = resource.network.main }
volume { source = resource.k8s_cluster.dev.kube_config.path destination = "/root/.kube/config" type = "bind" }
command = ["sleep", "infinity"]}
# Terminal accessing the kubectl containerresource "terminal" "k8s_terminal" { target = resource.container.kubectl working_directory = "/root"}Service Access
Section titled “Service Access”The common way to expose cluster workloads is an ingress resource or a proxy container:
resource "ingress" "dashboard" { port = 8445
target { resource = resource.k8s_cluster.dev port = 443
config = { service = "kubernetes-dashboard" namespace = "kubernetes-dashboard" } }}Best Practices
Section titled “Best Practices”- Resource Planning: Allocate sufficient CPU and memory based on expected workload
- Image Pre-loading: Use
copy_imagefor applications that will be deployed - Network Isolation: Create dedicated networks for cluster communication
- Configuration Management: Use volumes to provide custom configurations
- Registry Configuration: Configure insecure registries for internal/development use
- Access Patterns: Use proxy containers for terminal and service access
Common Issues
Section titled “Common Issues”Resource Constraints
Section titled “Resource Constraints”# Ensure adequate resources for multi-node clustersresources { cpu = 4000 # 4 CPUs per node memory = 8192 # 8GB per node}Image Pull Issues
Section titled “Image Pull Issues”# Configure registry access for private imagesconfig { docker { insecure_registries = ["internal.company.com:5000"] no_proxy = ["internal.company.com"] }}Network Connectivity
Section titled “Network Connectivity”# Ensure cluster is on the same network as accessing resourcesresource "container" "client" { image { name = "ubuntu:24.04" }
network { target = resource.network.main # Same network as cluster }}