Kubernetes Cluster

Defined insandboxes.hclAlso known askubernetes_cluster

The kubernetes_cluster resource creates Kubernetes clusters running in Docker containers using K3s. These clusters provide isolated Kubernetes environments for hands-on learning and testing scenarios.

Use Cases

As a lab author, you can use Kubernetes clusters to provide container orchestration environments:

• Kubernetes Learning: Provide hands-on experience with kubectl, deployments, services, and other Kubernetes concepts
• Application Deployment: Deploy and manage complex multi-tier applications using Kubernetes manifests
• Cluster Administration: Learn cluster management, RBAC, networking policies, and resource management

Kubernetes clusters enable realistic container orchestration scenarios within controlled lab environments.

HCL Syntax

Basic Syntax

resource "kubernetes_cluster" "name" {
  network {
    id = resource.network.main.meta.id
  }
}

Full Syntax

resource "kubernetes_cluster" "name" {
  nodes = 3

  image {
    name = "rancher/k3s:latest"
  }

  network {
    id         = resource.network.main.meta.id
    ip_address = "10.0.0.100"
  }

  resources {
    cpu    = 2000  # 2 CPUs
    memory = 4096  # 4GB
  }

  copy_image {
    name = "myapp:latest"
  }

  port {
    local = 80
    host  = 8080
  }

  volume {
    source      = "./config"
    destination = "/etc/config"
    type        = "bind"
  }

  environment = {
    CLUSTER_INIT = "true"
  }

  config {
    docker {
      no_proxy            = ["internal.registry.com"]
      insecure_registries = ["internal.registry.com:5000"]
    }
  }
}

Fields

Field	Type	Required	Description
network	block	—	Connect your container to a specific network to enable communication between containers or services.repeatable
image	block	—	Custom container image for the cluster nodes
nodes	number	—	Number of cluster nodes
volume	block	—	Volume mounts for the clusterrepeatable
copy_image	block	—	Container images to pre-load into the clusterrepeatable
port	block	—	Port mappings for the clusterrepeatable
port_range	block	—	Port range mappings for the clusterrepeatable
resources	block	—	Resource constraints for the cluster
environment	map(string)	—	Environment variables for the cluster
config	block	—	Cluster configuration

Network

k8s_cluster→Network

Network connections for the cluster

Field	Type	Required	Description
id	string	✓	ID of the network to attach the container
ip_address	string	—	Static IP address to assign container for the network, the ip address must be within range defined by the network subnet. If this parameter is omitted an IP address will be automatically assigned.
aliases	list(string)	—	Aliases allow alternate names to specified for the container

Image

k8s_cluster→Image

Custom container image for the cluster nodes

Field	Type	Required	Description
name	string	✓	Container image name
username	string	—	Registry username for authentication
password	string	—	Registry password for authentication

Volume

k8s_cluster→Volume

Volume mounts for the cluster

Field	Type	Required	Description
source	string	✓	Source path on the host
destination	string	✓	Mount path inside the cluster
type	string	—	Volume type
read_only	bool	—	Mount as read-only
bind_propagation	string	—	Bind propagation mode
bind_propagation_non_recursive	bool	—	Use non-recursive bind propagation
selinux_relabel	string	—	SELinux relabeling option

Copy Image

k8s_cluster→Copy Image

Container images to pre-load into the cluster

Field	Type	Required	Description
name	string	✓	Container image name
username	string	—	Registry username for authentication
password	string	—	Registry password for authentication

Port

k8s_cluster→Port

Port mappings for the cluster

Field	Type	Required	Description
local	string	✓	Port inside the cluster
remote	string	—	Remote port
host	string	—	Port on the host
protocol	string	—	Protocol (tcp/udp)
open_in_browser	string	—	URL path to open in browser

Port Range

k8s_cluster→Port Range

Port range mappings for the cluster

Field	Type	Required	Description
range	string	✓	Port range (e.g. 8000-8100)
enable_host	bool	—	Enable host port mapping
protocol	string	—	Protocol (tcp/udp)

Resources

k8s_cluster→Resources

Resource constraints for the cluster

Field	Type	Required	Description
cpu	number	—	CPU limit where 1 CPU = 1000
cpu_pin	list(number)	—	Pin to specific CPU cores
memory	number	—	Max memory in MB
disk	number	—	Disk space limit in MB
gpu	block	—	GPU resource constraints

GPU

k8s_cluster→Resources→GPU

GPU resource constraints

Field	Type	Required	Description
driver	string	—	GPU driver to use
device_ids	list(string)	—	Specific GPU device IDs to expose

Config

k8s_cluster→Config

Cluster configuration

Field	Type	Required	Description
docker	block	—	Docker daemon configuration

Docker

k8s_cluster→Config→Docker

Docker daemon configuration

Field	Type	Required	Description
no_proxy	list(string)	—	Addresses excluded from proxy
insecure_registries	list(string)	—	Registries to allow insecure connections to

KubeConfig

k8s_cluster→KubeConfig

Kubernetes configuration details

Computed Attributes

These attributes are set by the system and are read-only.

Attribute	Type	Description
network[].name	string	Name will equal the name of the network as created by jumppad
network[].assigned_address	string	AssignedAddress will equal if IPAddress is set, else it will be the value automatically assigned from the network
copy_image[].id	string	Image ID from local registry
api_port	number	Kubernetes API server port
connector_port	number	Connector service port
container_name	string	FQDN of the cluster container
external_ip	string	External IP address of the Docker host
kube_config.path	string	Path to kubeconfig file
kube_config.ca	string	Certificate authority data
kube_config.client_certificate	string	Client certificate data
kube_config.client_key	string	Client key data

Blocks

Network Block

kubernetes_cluster → network

Attaches the cluster to a network for connectivity.

Field	Required	Type	Description
id	✓	reference to network	Reference to the ID of a network resource
ip_address		string	Static IP address for the cluster. Auto-assigned if not specified.
aliases		list(string)	Network aliases. Defaults to empty list.

Image Block

kubernetes_cluster → image

Specifies the container image for cluster nodes.

Field	Required	Type	Description
name	✓	string	Docker image name and tag
username		string	Username for private registry
password		string	Password for private registry

Resources Block

kubernetes_cluster → resources

Defines resource constraints for the cluster.

Field	Required	Type	Description
cpu		number	CPU limit in MHz (1000 = 1 CPU)
cpu_pin		list(number)	Pin to specific CPU cores
memory		number	Memory limit in MB
gpu		block	GPU configuration

Copy Image Block

kubernetes_cluster → copy_image

Defines local Docker images to copy into the cluster.

Field	Required	Type	Description
name	✓	string	Docker image name and tag to copy from local cache

Port Block

kubernetes_cluster → port

Maps container ports to host ports.

Field	Required	Type	Description
local	✓	string	Container port number
host		string	Host port number. Auto-assigned if not specified.

Port Range Block

kubernetes_cluster → port_range

Maps ranges of container ports to host ports.

Field	Required	Type	Description
range	✓	string	Port range (e.g. “8000-9000”)
enable_host		bool	Enable host port mapping. Defaults to false.

Volume Block

kubernetes_cluster → volume

Mounts volumes into the cluster container.

Field	Required	Type	Description
source	✓	string	Source path on host
destination	✓	string	Destination path in container
type		string	Volume type: “bind”, “volume”, or “tmpfs”. Defaults to “bind”.
read_only		bool	Mount as read-only. Defaults to false.

GPU Block

kubernetes_cluster → resources → gpu

GPU configuration for the cluster.

Field	Required	Type	Description
driver	✓	string	GPU driver to use
device_ids	✓	list(string)	GPU device IDs to assign

Config Block

kubernetes_cluster → config

Cluster-specific configuration settings.

Field	Required	Type	Description
docker		block	Docker daemon configuration

Docker Config Block

kubernetes_cluster → config → docker

Docker daemon settings for the cluster.

Field	Required	Type	Description
no_proxy		list(string)	Registries to exclude from proxy. Defaults to empty list.
insecure_registries		list(string)	Insecure registries to allow. Defaults to empty list.

Image Caching

Kubernetes clusters use a global image cache to optimize bandwidth and performance:

• Each cluster node has its own Docker image cache
• Images are pulled through a shared cache proxy
• After the first pull, subsequent pulls come from the cache
• Use copy_image blocks to pre-populate cluster with local images

Examples

Basic Single-Node Cluster

resource "network" "k8s_net" {
  subnet = "10.0.0.0/24"
}

resource "kubernetes_cluster" "basic" {
  network {
    id = resource.network.k8s_net.meta.id
  }
}

Multi-Node Production-Like Cluster

resource "kubernetes_cluster" "production" {
  nodes = 3

  network {
    id = resource.network.cluster_network.meta.id
  }

  resources {
    cpu    = 4000  # 4 CPUs per node
    memory = 8192  # 8GB per node
  }

  volume {
    source      = "./k8s-config"
    destination = "/etc/kubernetes"
    type        = "bind"
    read_only   = true
  }

  environment = {
    K3S_TOKEN = "my-cluster-token"
  }
}

Cluster with Custom Image and Registry

resource "kubernetes_cluster" "custom" {
  image {
    name     = "rancher/k3s:v1.25.3-k3s1"
    username = "registry_user"
    password = "registry_pass"
  }

  network {
    id = resource.network.secure_network.meta.id
  }

  config {
    docker {
      insecure_registries = ["internal.company.com:5000"]
      no_proxy           = ["internal.company.com"]
    }
  }

  copy_image {
    name = "internal.company.com:5000/myapp:v1.2.3"
  }
}

Cluster with Application Pre-loading

resource "kubernetes_cluster" "app_cluster" {
  nodes = 2

  network {
    id = resource.network.app_network.meta.id
  }

  # Pre-load application images
  copy_image {
    name = "nginx:1.21"
  }

  copy_image {
    name = "redis:7-alpine"
  }

  copy_image {
    name = "postgres:14"
  }

  # Expose common ports
  port {
    local = 80
    host  = 8080
  }

  port {
    local = 443
    host  = 8443
  }
}

Terminal Access

Since terminals cannot directly target kubernetes_cluster resources, use a container with kubectl:

resource "kubernetes_cluster" "k8s" {
  network {
    id = resource.network.main.meta.id
  }
}

resource "container" "kubectl" {
  image {
    name = "bitnami/kubectl:latest"
  }

  network {
    id = resource.network.main.meta.id
  }

  volume {
    source      = resource.kubernetes_cluster.k8s.kube_config.path
    destination = "/root/.kube/config"
    type        = "bind"
  }

  command = ["sleep", "infinity"]
}

resource "terminal" "k8s_terminal" {
  target = resource.container.kubectl
}

Service Access

Services cannot directly target kubernetes_cluster resources either. Use ingress resources or proxy containers:

resource "ingress" "k8s_dashboard" {
  port = 8080

  target {
    resource = resource.kubernetes_cluster.k8s
    port     = 80

    config = {
      service   = "kubernetes-dashboard"
      namespace = "kubernetes-dashboard"
    }
  }
}

Best Practices

1. Resource Planning: Allocate sufficient CPU and memory based on expected workload
2. Multi-Node Setup: Use multiple nodes for realistic cluster scenarios
3. Image Pre-loading: Use copy_image for applications that will be deployed
4. Network Isolation: Create dedicated networks for cluster communication
5. Configuration Management: Use volumes to provide custom configurations
6. Registry Configuration: Configure insecure registries for internal/development use
7. Access Patterns: Use proxy containers for terminal and service access

Common Issues

Resource Constraints

## Ensure adequate resources for multi-node clusters
resources {
  cpu    = 2000  # Minimum 2 CPUs for production-like workloads
  memory = 4096  # Minimum 4GB for realistic scenarios
}

Image Pull Issues

# Configure registry access for private images
config {
  docker {
    insecure_registries = ["your-registry:5000"]
  }
}

Network Connectivity

# Ensure cluster is on the same network as accessing resources
resource "container" "client" {
  network {
    id = resource.network.main.meta.id  # Same network as cluster
  }
}