Kubernetes Cluster

Defined insandboxes.hclAlso known askubernetes_cluster

The kubernetes_cluster resource creates Kubernetes clusters running in Docker containers using K3s. These clusters provide isolated Kubernetes environments for hands-on learning and testing scenarios.

Use Cases

As a lab author, you can use Kubernetes clusters to provide container orchestration environments:

• Kubernetes Learning: Provide hands-on experience with kubectl, deployments, services, and other Kubernetes concepts
• Application Deployment: Deploy and manage complex multi-tier applications using Kubernetes manifests
• Cluster Administration: Learn cluster management, RBAC, networking policies, and resource management

Kubernetes clusters enable realistic container orchestration scenarios within controlled lab environments.

HCL Syntax

Basic Syntax

resource "kubernetes_cluster" "name" {
  network {
    id = resource.network.main.meta.id
  }
}

Full Syntax

resource "kubernetes_cluster" "name" {
  image {
    name = "rancher/k3s:latest"
  }

  network {
    id         = resource.network.main.meta.id
    ip_address = "10.0.0.100"
  }

  resources {
    cpu    = 2000  # 2 CPUs
    memory = 4096  # 4GB
  }

  copy_image {
    name = "myapp:latest"
  }

  port {
    local = 80
    host  = 8080
  }

  volume {
    source      = "./config"
    destination = "/etc/config"
    type        = "bind"
  }

  environment = {
    CLUSTER_INIT = "true"
  }

  config {
    docker {
      no_proxy            = ["internal.registry.com"]
      insecure_registries = ["internal.registry.com:5000"]
    }
  }
}

Fields

Field	Type	Required	Description
network	block	—	Connect your container to a specific network to enable communication between containers or servicesrepeatable
image	block	—	Override the image the cluster nodes run. Leave empty to use the default K3s image.
volume	block	—	Share files from the lab repository into the cluster nodes, or persist data written by themrepeatable
copy_image	block	—	Preload images into the cluster so pods start without pulling from a registry. Speeds up the lab and lets it run offline.repeatable
port	block	—	Expose specific ports so learners can reach services running in the clusterrepeatable
port_range	block	—	Specify a range of ports to open for applications that require multiple connections, such as node ports or multi-service environmentsrepeatable
resources	block	—	Cap the CPU, memory, and disk the cluster can consume
environment	map(string)	—	Set environment variables to pass configuration values, API keys, or secrets to the cluster nodes
config	block	—	Low-level settings for the container runtime backing the cluster nodes

Network

k8s_cluster→Network

Network connections for the cluster

Field	Type	Required	Description
id	string	✓	ID of the network to attach the container
ip_address	string	—	Static IP address to assign container for the network, the ip address must be within range defined by the network subnet. If this parameter is omitted an IP address will be automatically assigned.
aliases	list(string)	—	Aliases allow alternate names to specified for the container

Image

k8s_cluster→Image

Custom container image for the cluster nodes

Field	Type	Required	Description
name	string	✓	Full image reference, including the registry and an optional tag. Include a specific tag to ensure consistent lab environments.
username	string	—	Only required for private registries. Leave empty for public images.
password	string	—	Only required for private registries. Leave empty for public images.

Volume

k8s_cluster→Volume

Volume mounts for the cluster

Field	Type	Required	Description
source	string	✓	The path to mount into the cluster nodes. Can be relative to the file declaring the cluster (`./`) or absolute (`/usr/local/bin`).
destination	string	✓	Absolute path inside the cluster nodes where the source is mounted
type	string	—	The type of the mount, can be one of the following values: - bind: bind the source path to the destination path in the node - volume: source is a Docker volume - tmpfs: create a temporary filesystem
read_only	bool	—	Prevent the cluster nodes from writing to this mount
bind_propagation	string	—	Configures bind propagation for Docker volume mounts, only applies to bind mounts, can be one of the following values: - shared - slave - private - rslave - rprivate For more information please see the Docker documentation
bind_propagation_non_recursive	bool	—	Mount only the source directory itself, without the mounts nested below it. Leave off unless nested host mounts are leaking into the node.
selinux_relabel	string	—	Configures SELinux relabeling for the mount (usually specified as :z or :Z) and can be one of the following values: - shared (Equivalent to :z) - private (Equivalent to :Z)

Copy Image

k8s_cluster→Copy Image

Container images to pre-load into the cluster

Field	Type	Required	Description
name	string	✓	Full image reference, including the registry and an optional tag. Include a specific tag to ensure consistent lab environments.
username	string	—	Only required for private registries. Leave empty for public images.
password	string	—	Only required for private registries. Leave empty for public images.

Port

k8s_cluster→Port

Port mappings for the cluster

Field	Type	Required	Description
local	string	✓	The port the service listens on inside the cluster
host	string	—	The host port to map the local port to
protocol	string	—	The protocol to use when exposing the port

Port Range

k8s_cluster→Port Range

Port range mappings for the cluster

Field	Type	Required	Description
range	string	✓	The port range to expose, e.g. `8080-8082` would expose the ports `8080`, `8081`, `8082`
enable_host	bool	—	Expose the port range on the host
protocol	string	—	The protocol to use when exposing the port

Resources

k8s_cluster→Resources

Resource constraints for the cluster

Field	Type	Required	Description
cpu	number	—	Limit how much CPU power the cluster can use to optimize performance and cost. 1 CPU core = 1000.
cpu_pin	list(number)	—	Pin the cluster to specific CPU cores for more predictable performance or isolation
memory	number	—	Specify how much memory the cluster can use to prevent crashes or resource contention
disk	number	—	Cap the disk space the cluster can consume
gpu	block	—	Give the cluster access to GPU hardware for accelerated workloads

GPU

k8s_cluster→Resources→GPU

GPU resource constraints

Field	Type	Required	Description
driver	string	—	Select the GPU driver to use for hardware acceleration or GPU-based workloads
device_ids	list(string)	—	Specify which GPU devices the cluster can access when running GPU-based tasks

Config

k8s_cluster→Config

Cluster configuration

Field	Type	Required	Description
docker	block	—	Configure the Docker daemon inside the cluster nodes

Docker

k8s_cluster→Config→Docker

Docker daemon configuration

Field	Type	Required	Description
no_proxy	list(string)	—	Hosts that should bypass the proxy, for example an internal registry reachable directly
insecure_registries	list(string)	—	Registries the nodes may pull from over plain HTTP or with an untrusted certificate

KubeConfig

k8s_cluster→KubeConfig

Kubernetes configuration details

Computed Attributes

These attributes are set by the system and are read-only.

Attribute	Type	Description
network[].name	string	Name will equal the name of the network as created by jumppad
network[].assigned_address	string	AssignedAddress will equal if IPAddress is set, else it will be the value automatically assigned from the network
copy_image[].id	string	Image ID from local registry
api_port	number	Kubernetes API server port
connector_port	number	Connector service port
container_name	string	FQDN of the cluster container
external_ip	string	External IP address of the Docker host
kube_config.path	string	Path to kubeconfig file
kube_config.ca	string	Certificate authority data
kube_config.client_certificate	string	Client certificate data
kube_config.client_key	string	Client key data

Blocks

Network Block

kubernetes_cluster → network

Attaches the cluster to a network for connectivity.

Field	Required	Type	Description
id	✓	reference to network	Reference to the ID of a network resource
ip_address		string	Static IP address for the cluster. Auto-assigned if not specified.
aliases		list(string)	Network aliases. Defaults to empty list.

Image Block

kubernetes_cluster → image

Specifies the container image for cluster nodes.

Field	Required	Type	Description
name	✓	string	Docker image name and tag
username		string	Username for private registry
password		string	Password for private registry

Resources Block

kubernetes_cluster → resources

Defines resource constraints for the cluster.

Field	Required	Type	Description
cpu		number	CPU limit in MHz (1000 = 1 CPU)
cpu_pin		list(number)	Pin to specific CPU cores
memory		number	Memory limit in MB
gpu		block	GPU configuration

Copy Image Block

kubernetes_cluster → copy_image

Defines local Docker images to copy into the cluster.

Field	Required	Type	Description
name	✓	string	Docker image name and tag to copy from local cache

Port Block

kubernetes_cluster → port

Maps container ports to host ports.

Field	Required	Type	Description
local	✓	string	Container port number
host		string	Host port number. Auto-assigned if not specified.

Port Range Block

kubernetes_cluster → port_range

Maps ranges of container ports to host ports.

Field	Required	Type	Description
range	✓	string	Port range (e.g. “8000-9000”)
enable_host		bool	Enable host port mapping. Defaults to false.

Volume Block

kubernetes_cluster → volume

Mounts volumes into the cluster container.

Field	Required	Type	Description
source	✓	string	Source path on host
destination	✓	string	Destination path in container
type		string	Volume type: “bind”, “volume”, or “tmpfs”. Defaults to “bind”.
read_only		bool	Mount as read-only. Defaults to false.

GPU Block

kubernetes_cluster → resources → gpu

GPU configuration for the cluster.

Field	Required	Type	Description
driver	✓	string	GPU driver to use
device_ids	✓	list(string)	GPU device IDs to assign

Config Block

kubernetes_cluster → config

Cluster-specific configuration settings.

Field	Required	Type	Description
docker		block	Docker daemon configuration

Docker Config Block

kubernetes_cluster → config → docker

Docker daemon settings for the cluster.

Field	Required	Type	Description
no_proxy		list(string)	Registries to exclude from proxy. Defaults to empty list.
insecure_registries		list(string)	Insecure registries to allow. Defaults to empty list.

Image Caching

Kubernetes clusters use a global image cache to optimize bandwidth and performance:

• Each cluster node has its own Docker image cache
• Images are pulled through a shared cache proxy
• After the first pull, subsequent pulls come from the cache
• Use copy_image blocks to pre-populate cluster with local images

Examples

Basic Single-Node Cluster

resource "network" "k8s_net" {
  subnet = "10.0.0.0/24"
}

resource "kubernetes_cluster" "basic" {
  network {
    id = resource.network.k8s_net.meta.id
  }
}

Multi-Node Production-Like Cluster

resource "kubernetes_cluster" "production" {
  network {
    id = resource.network.cluster_network.meta.id
  }

  resources {
    cpu    = 4000  # 4 CPUs per node
    memory = 8192  # 8GB per node
  }

  volume {
    source      = "./k8s-config"
    destination = "/etc/kubernetes"
    type        = "bind"
    read_only   = true
  }

  environment = {
    K3S_TOKEN = "my-cluster-token"
  }
}

Cluster with Custom Image and Registry

resource "kubernetes_cluster" "custom" {
  image {
    name     = "rancher/k3s:v1.25.3-k3s1"
    username = "registry_user"
    password = "registry_pass"
  }

  network {
    id = resource.network.secure_network.meta.id
  }

  config {
    docker {
      insecure_registries = ["internal.company.com:5000"]
      no_proxy           = ["internal.company.com"]
    }
  }

  copy_image {
    name = "internal.company.com:5000/myapp:v1.2.3"
  }
}

Cluster with Application Pre-loading

resource "kubernetes_cluster" "app_cluster" {
  network {
    id = resource.network.app_network.meta.id
  }

  # Pre-load application images
  copy_image {
    name = "nginx:1.21"
  }

  copy_image {
    name = "redis:7-alpine"
  }

  copy_image {
    name = "postgres:14"
  }

  # Expose common ports
  port {
    local = 80
    host  = 8080
  }

  port {
    local = 443
    host  = 8443
  }
}

Terminal Access

Since terminals cannot directly target kubernetes_cluster resources, use a container with kubectl:

resource "kubernetes_cluster" "k8s" {
  network {
    id = resource.network.main.meta.id
  }
}

resource "container" "kubectl" {
  image {
    name = "bitnami/kubectl:latest"
  }

  network {
    id = resource.network.main.meta.id
  }

  volume {
    source      = resource.kubernetes_cluster.k8s.kube_config.path
    destination = "/root/.kube/config"
    type        = "bind"
  }

  command = ["sleep", "infinity"]
}

resource "terminal" "k8s_terminal" {
  target = resource.container.kubectl
}

Service Access

Services cannot directly target kubernetes_cluster resources either. Use ingress resources or proxy containers:

resource "ingress" "k8s_dashboard" {
  port = 8080

  target {
    resource = resource.kubernetes_cluster.k8s
    port     = 80

    config = {
      service   = "kubernetes-dashboard"
      namespace = "kubernetes-dashboard"
    }
  }
}

Best Practices

1. Resource Planning: Allocate sufficient CPU and memory based on expected workload
2. Image Pre-loading: Use copy_image for applications that will be deployed
3. Network Isolation: Create dedicated networks for cluster communication
4. Configuration Management: Use volumes to provide custom configurations
5. Registry Configuration: Configure insecure registries for internal/development use
6. Access Patterns: Use proxy containers for terminal and service access

Common Issues

Resource Constraints

## Ensure adequate resources for multi-node clusters
resources {
  cpu    = 2000  # Minimum 2 CPUs for production-like workloads
  memory = 4096  # Minimum 4GB for realistic scenarios
}

Image Pull Issues

# Configure registry access for private images
config {
  docker {
    insecure_registries = ["your-registry:5000"]
  }
}

Network Connectivity

# Ensure cluster is on the same network as accessing resources
resource "container" "client" {
  network {
    id = resource.network.main.meta.id  # Same network as cluster
  }
}