Kubernetes Cluster

The kubernetes_cluster resource creates Kubernetes clusters running in Docker containers using K3s. These clusters provide isolated Kubernetes environments for hands-on learning and testing scenarios.

Use Cases

As a lab author, you can use Kubernetes clusters to provide container orchestration environments:

• Kubernetes Learning: Provide hands-on experience with kubectl, deployments, services, and other Kubernetes concepts
• Application Deployment: Deploy and manage complex multi-tier applications using Kubernetes manifests
• Cluster Administration: Learn cluster management, RBAC, networking policies, and resource management

Kubernetes clusters enable realistic container orchestration scenarios within controlled lab environments.

HCL Syntax

Basic Syntax

resource "kubernetes_cluster" "name" {
  network {
    id = resource.network.main.meta.id
  }
}

Full Syntax

resource "kubernetes_cluster" "name" {
  nodes = 3

  image {
    name = "rancher/k3s:latest"
  }

  network {
    id         = resource.network.main.meta.id
    ip_address = "10.0.0.100"
  }

  resources {
    cpu    = 2000  # 2 CPUs
    memory = 4096  # 4GB
  }

  copy_image {
    name = "myapp:latest"
  }

  port {
    local = 80
    host  = 8080
  }

  volume {
    source      = "./config"
    destination = "/etc/config"
    type        = "bind"
  }

  environment = {
    CLUSTER_INIT = "true"
  }

  config {
    docker {
      no_proxy            = ["internal.registry.com"]
      insecure_registries = ["internal.registry.com:5000"]
    }
  }
}

Fields

Field	Required	Type	Description
network	✓	block	Network attachments (repeatable)
nodes		number	Number of nodes in the cluster. Defaults to 1.
image		block	Container image for cluster nodes. Defaults to latest K3s.
resources		block	CPU and memory constraints
copy_image		block	Local Docker images to copy to cluster (repeatable)
port		block	Port mappings (repeatable)
port_range		block	Port range mappings (repeatable)
volume		block	Volume mounts (repeatable)
environment		map(string)	Environment variables. Defaults to empty map.
config		block	Cluster configuration settings

Blocks

Network Block

kubernetes_cluster → network

Attaches the cluster to a network for connectivity.

Field	Required	Type	Description
id	✓	reference to network	Reference to the ID of a network resource
ip_address		string	Static IP address for the cluster. Auto-assigned if not specified.
aliases		list(string)	Network aliases. Defaults to empty list.

Image Block

kubernetes_cluster → image

Specifies the container image for cluster nodes.

Field	Required	Type	Description
name	✓	string	Docker image name and tag
username		string	Username for private registry
password		string	Password for private registry

Resources Block

kubernetes_cluster → resources

Defines resource constraints for the cluster.

Field	Required	Type	Description
cpu		number	CPU limit in MHz (1000 = 1 CPU)
cpu_pin		list(number)	Pin to specific CPU cores
memory		number	Memory limit in MB
gpu		block	GPU configuration

Copy Image Block

kubernetes_cluster → copy_image

Defines local Docker images to copy into the cluster.

Field	Required	Type	Description
name	✓	string	Docker image name and tag to copy from local cache

Port Block

kubernetes_cluster → port

Maps container ports to host ports.

Field	Required	Type	Description
local	✓	string	Container port number
host		string	Host port number. Auto-assigned if not specified.

Port Range Block

kubernetes_cluster → port_range

Maps ranges of container ports to host ports.

Field	Required	Type	Description
range	✓	string	Port range (e.g. “8000-9000”)
enable_host		bool	Enable host port mapping. Defaults to false.

Volume Block

kubernetes_cluster → volume

Mounts volumes into the cluster container.

Field	Required	Type	Description
source	✓	string	Source path on host
destination	✓	string	Destination path in container
type		string	Volume type: “bind”, “volume”, or “tmpfs”. Defaults to “bind”.
read_only		bool	Mount as read-only. Defaults to false.

GPU Block

kubernetes_cluster → resources → gpu

GPU configuration for the cluster.

Field	Required	Type	Description
driver	✓	string	GPU driver to use
device_ids	✓	list(string)	GPU device IDs to assign

Config Block

kubernetes_cluster → config

Cluster-specific configuration settings.

Field	Required	Type	Description
docker		block	Docker daemon configuration

Docker Config Block

kubernetes_cluster → config → docker

Docker daemon settings for the cluster.

Field	Required	Type	Description
no_proxy		list(string)	Registries to exclude from proxy. Defaults to empty list.
insecure_registries		list(string)	Insecure registries to allow. Defaults to empty list.

Computed Attributes

These attributes are available after cluster creation:

Field	Type	Description
kube_config	block	Kubernetes configuration details
api_port	number	Kubernetes API server port
connector_port	number	Internal connector port
container_name	string	Docker container name for the cluster
external_ip	string	External IP address

KubeConfig Block

kubernetes_cluster → kube_config

Kubernetes configuration information for cluster access.

Field	Type	Description
path	string	Path to kubeconfig file
ca	string	Base64-encoded CA certificate
client_certificate	string	Base64-encoded client certificate
client_key	string	Base64-encoded client key

Image Caching

Kubernetes clusters use a global image cache to optimize bandwidth and performance:

• Each cluster node has its own Docker image cache
• Images are pulled through a shared cache proxy
• After the first pull, subsequent pulls come from the cache
• Use copy_image blocks to pre-populate cluster with local images

Examples

Basic Single-Node Cluster

resource "network" "k8s_net" {
  subnet = "10.0.0.0/24"
}

resource "kubernetes_cluster" "basic" {
  network {
    id = resource.network.k8s_net.meta.id
  }
}

Multi-Node Production-Like Cluster

resource "kubernetes_cluster" "production" {
  nodes = 3

  network {
    id = resource.network.cluster_network.meta.id
  }

  resources {
    cpu    = 4000  # 4 CPUs per node
    memory = 8192  # 8GB per node
  }

  volume {
    source      = "./k8s-config"
    destination = "/etc/kubernetes"
    type        = "bind"
    read_only   = true
  }

  environment = {
    K3S_TOKEN = "my-cluster-token"
  }
}

Cluster with Custom Image and Registry

resource "kubernetes_cluster" "custom" {
  image {
    name     = "rancher/k3s:v1.25.3-k3s1"
    username = "registry_user"
    password = "registry_pass"
  }

  network {
    id = resource.network.secure_network.meta.id
  }

  config {
    docker {
      insecure_registries = ["internal.company.com:5000"]
      no_proxy           = ["internal.company.com"]
    }
  }

  copy_image {
    name = "internal.company.com:5000/myapp:v1.2.3"
  }
}

Cluster with Application Pre-loading

resource "kubernetes_cluster" "app_cluster" {
  nodes = 2

  network {
    id = resource.network.app_network.meta.id
  }

  # Pre-load application images
  copy_image {
    name = "nginx:1.21"
  }

  copy_image {
    name = "redis:7-alpine"
  }

  copy_image {
    name = "postgres:14"
  }

  # Expose common ports
  port {
    local = 80
    host  = 8080
  }

  port {
    local = 443
    host  = 8443
  }
}

Terminal Access

Since terminals cannot directly target kubernetes_cluster resources, use a container with kubectl:

resource "kubernetes_cluster" "k8s" {
  network {
    id = resource.network.main.meta.id
  }
}

resource "container" "kubectl" {
  image {
    name = "bitnami/kubectl:latest"
  }

  network {
    id = resource.network.main.meta.id
  }

  volume {
    source      = resource.kubernetes_cluster.k8s.kube_config.path
    destination = "/root/.kube/config"
    type        = "bind"
  }

  command = ["sleep", "infinity"]
}

resource "terminal" "k8s_terminal" {
  target = resource.container.kubectl
}

Service Access

Services cannot directly target kubernetes_cluster resources either. Use ingress resources or proxy containers:

resource "ingress" "k8s_dashboard" {
  port = 8080

  target {
    resource = resource.kubernetes_cluster.k8s
    port     = 80

    config = {
      service   = "kubernetes-dashboard"
      namespace = "kubernetes-dashboard"
    }
  }
}

Best Practices

1. Resource Planning: Allocate sufficient CPU and memory based on expected workload
2. Multi-Node Setup: Use multiple nodes for realistic cluster scenarios
3. Image Pre-loading: Use copy_image for applications that will be deployed
4. Network Isolation: Create dedicated networks for cluster communication
5. Configuration Management: Use volumes to provide custom configurations
6. Registry Configuration: Configure insecure registries for internal/development use
7. Access Patterns: Use proxy containers for terminal and service access

Common Issues

Resource Constraints

## Ensure adequate resources for multi-node clusters
resources {
  cpu    = 2000  # Minimum 2 CPUs for production-like workloads
  memory = 4096  # Minimum 4GB for realistic scenarios
}

Image Pull Issues

# Configure registry access for private images
config {
  docker {
    insecure_registries = ["your-registry:5000"]
  }
}

Network Connectivity

# Ensure cluster is on the same network as accessing resources
resource "container" "client" {
  network {
    id = resource.network.main.meta.id  # Same network as cluster
  }
}