Cluster

Cluster

The kubernetes_cluster resource allows you to create immutable Kubernetes clusters running in Docker containers using K3s.

resource "kubernetes_cluster" "name" {
  ...
}

Image Caching

Kubernetes clusters do not share the local machines Docker image cache. Each node in a cluster has it's own unqiue cache.

To save bandwidth all containers launched in the Kubernetes cluster pulled through an image cache that runs in Docker. After the first pull all images are subsequently pulled from the image cache not the public internet. This cache is global to all Nomad and Kubernetes clusters created with Jumppad.

For more information on the image cache see the container_registry resource.

Attributes

Attribute	Description
Networks network type: []block NetworkAttachment	Attach to the correct network // only when Image is specified Network attaches the container to an existing network defined in a separate stanza. This block can be specified multiple times to attach the container to multiple networks. network { id = resource.network.main.meta.id }
Image image type: block Image	Image defines a Docker image to use when creating the container. By default the kubernetes cluster resource will be created using the latest Jumppad container image. image { name = "example/kubernetes:latest" }
Resources resources type: block Resources	Define resource constraints for the cluster resources { cpu = 100 memory = 1024 }
Nodes nodes type: int	The number of nodes to create in the cluster. nodes = 3
Volumes volume type: []block Volume	Additional volume to mount to the server and client nodes. volume { source = "./mydirectory" destination = "/path_in_container" }
CopyImages copy_image type: []block Image	Docker image in the local Docker image cache to copy to the cluster on creation. This image is added to the Kubernetes clients docker cache enabling jobs to use images that may not be in the local registry. Jumppad tracks changes to copied images, should the image change running jumppad up would push any changes to the cluster automatically. copy_image { name = "mylocalimage:version" }
Ports port type: []block Port	A `port` stanza allows you to expose container ports on the local network or host. This stanza can be specified multiple times. port { local = 80 host = 8080 }
PortRanges port_range type: []block PortRange	A `port_range` stanza allows you to expose a range of container ports on the local network or host. This stanza can be specified multiple times. The following example would create 11 ports from 80 to 90 (inclusive) and expose them to the host machine. port { range = "80-90" enable_host = true }
Environment environment type: map[string]string	environment variables to set when starting the container An env stanza allows you to set environment variables in the container. This stanza can be specified multiple times. env { key = "PATH" value = "/usr/local/bin" }
Config config type: block ClusterConfig	Specifies the configuration for the Kubernetes cluster.
APIPort api_port type: int	Port to expose the Kubernetes API on the host. By default this uses the standard api port `443`; however, if you are running multiple kubernetes instances you will need to override this value.
ContainerName container_name type: string	The fully qualified resource name for the Kubernetes cluster, this value can be used to address the server from the Docker network. It is also the name of the Docker container. server.name.k8s-cluster.local.jmpd.in

Computed Attributes

These attributes are computed when the config is parsed and applied, and are therefore only known at parsetime or runtime.

Attribute	Description
Meta ID meta.id string	The full ID of the resource e.g. `resource.type.name`. This is computed from the full resource path: // given the following resource resource "container" "ubuntu" { ... } // the resulting id will be resource.container.ubuntu
Meta Type meta.type string	The type of the resource. This taken from the type label of the resource definition. // given the following resource resource "container" "ubuntu" { ... } // the resulting type will be container
Meta Name meta.name string	The name of the resource. This taken from the name label of the resource definition. // given the following resource resource "container" "ubuntu" { ... } // the resulting name will be ubuntu
KubeConfig kube_config type: KubeConfig	Details for the Kubenetes config file that can be used to interact with the cluster.
ConnectorPort connector_port type: int	The port where the Jumppad connector is exposed to the host, this property is requied by the ingress resource and is not generally needed when building blueprints.
ExternalIP external_ip type: string	Local IP address of the Nomad server, this property can be used to set the NOAMD_ADDR on the Jumppad client. output "K8S_ADDR" { value = "https://${resource.kubernetes_cluster.dev.external_ip}:${resource.kubernetes_cluster.dev.api_port}" }

---

Image

Image defines a Docker image used when creating this container. An Image can be stored in a public or a private repository.

container {
  image {
    ...
  }
}

Attributes

Attribute	Description
Name name required type: string	Name of the image to use when creating the container, can either be the full canonical name or short name for Docker official images. e.g. `consul:v1.6.1` or `docker.io/consul:v1.6.1`. name = "redis:latest"
Username username type: string	Username to use when connecting to a private image repository username = "my_username"
Password password type: string	Password to use when connecting to a private image repository, for both username and password interpolated environment variables can be used in place of static values. password = "my_password"

Computed Attributes

These attributes are computed when the config is parsed and applied, and are therefore only known at parsetime or runtime.

Attribute	Description
ID id type: string	ID is the unique identifier for the image, this is independent of tag and changes each time the image is built. An image that has been tagged multiple times also shares the same ID. ID string `hcl:"id,optional" json:"id,omitempty"`

---

Resources

A resources type allows you to configure the maximum resources which can be consumed.

container {
  resources {
    ...
  }
}

Attributes

Attribute	Description
CPU cpu type: int	Set the maximum CPU which can be consumed by the container in MHz, 1 CPU == 1000MHz. cpu = 100
CPUPin cpu_pin type: []int	Pin the container CPU consumption to one or more logical CPUs. For example to pin the container to the core 1 and 4. cpi_pin = [1,4]
Memory memory type: int	Maximum memory which a container can consume, specified in Megabytes. memory = 2048
GPU gpu type: block GPU	GPU settings to pass through to container gpu { driver = "nvidia" device_ids = ["0", "1"] }

Computed Attributes

These attributes are computed when the config is parsed and applied, and are therefore only known at parsetime or runtime.

---

NetworkAttachment

Network attachment defines a network to which the container is attached.

network {
  ...
}

Attributes

Attribute	Description
ID id required type: string	ID of the network to attach the container to, specified in reference format. e.g. to attach to a network called `cloud`. id = resource.network.main.meta.id
IPAddress ip_address type: string	Static IP address to assign container for the network, the ip address must be within range defined by the network subnet. If this parameter is omitted an IP address will be automatically assigned. ip_address = "10.0.5.24"
Aliases aliases type: []string	Aliases allow alternate names to specified for the container. Aliases can be used to reference a container across the network, the container will respond to ping and other network resolution using the primary assigned name `[name].container.shipyard.run` and the aliases. aliases = [ "alt1.container.local.jmpd.in", "alt2.container.local.jmpd.in" ]

Computed Attributes

These attributes are computed when the config is parsed and applied, and are therefore only known at parsetime or runtime.

Attribute	Description
Name name type: string	Name will equal the name of the network as created by jumppad.
AssignedAddress assigned_address type: string	`assigned_address` will equal the assigned IP address for the network. This will equal ip_address if set; otherwise, this is the automatically assigned IP address.

---

Port

A port stanza defines host to container communications

container {
  port {
    ...
  }
}

Attributes

Attribute	Description
Local local required type: string	The local port in the container. local = 80
Host host type: string	The host port to map the local port to. host = 8080
Protocol protocol type: string	The protocol to use when exposing the port, can be "tcp", or "udp". protocol = "tcp"

Computed Attributes

These attributes are computed when the config is parsed and applied, and are therefore only known at parsetime or runtime.

---

PortRange

A port_range stanza defines host to container communications by exposing a range of ports for the container.

container {
  port_range {
    ...
  }
}

Attributes

Attribute	Description
Range range required type: string	The port range to expose, e.g, `8080-8082` would expose the ports `8080`, `8081`, `8082`. range = "8080-8082"
EnableHost enable_host type: bool	Expose the port range on the host. enable_host = true
Protocol protocol type: string	The protocol to use when exposing the port, can be "tcp", or "udp". protocol = "tcp"

Computed Attributes

These attributes are computed when the config is parsed and applied, and are therefore only known at parsetime or runtime.

---

Volume

A volume type allows the specification of an attached volume.

container {
  volume {
    ...
  }
}

Attributes

Attribute	Description
Source source required type: string	The source volume to mount in the container, can be specified as a relative `./` or absolute path `/usr/local/bin`. Relative paths are relative to the file declaring the container. source = "./files/nginx"
Destination destination required type: string	The destination in the container to mount the volume to, must be an absolute path. destination = "/etc/nginx"
Type type type: string	The type of the mount, can be one of the following values: bind: bind the source path to the destination path in the container volume: source is a Docker volume tmpfs: create a temporary filesystem type = "bind"
ReadOnly read_only type: bool	Whether or not the volume is read only. read_only = true
BindPropagation bind_propagation type: string	Configures bind propagation for Docker volume mounts, only applies to bind mounts, can be one of the following values: shared slave private rslave rprivate For more information please see the Docker documentation https://docs.docker.com/storage/bind-mounts/#configure-bind-propagation bind_propagation = "shared"
BindPropagationNonRecursive bind_propagation_non_recursive type: bool	Configures recursiveness of the bind mount. By default Docker mounts with the equivalent of mount --rbind meaning that mounts below the the source directory are visible in the container. or instance running docker run --rm --mount type=bind,src=/,target=/host,readonly busybox will make /run of the host available as/host/run in the container. To make matters even worse it will be writable (since only the toplevel bind is set readonly, not the children). If bind_propagation_non_recursive is set to true then the container will only see an empty /host/run, meaning thetmpfs which is typically mounted to /run on the host is not propagated into the container. bind_propagation_non_recursive = true
SelinuxRelabel selinux_relabel type: string	Configures Selinux relabeling for the container (usually specified as :z or :Z) and can be one of the following values: shared (Equivalent to :z) private (Equivalent to :Z) selinux_relabel = "shared"

Computed Attributes

These attributes are computed when the config is parsed and applied, and are therefore only known at parsetime or runtime.

---

ClusterConfig

Specifies the configuration for the Kubernetes cluster.

resource "kubernetes_cluster" "cluster" {
  config {
    ...
  }
}

Attributes

Attribute	Description
DockerConfig docker type: block DockerConfig	Docker configuration for the Kubernetes cluster. docker { no_proxy = ["insecure.container.local.jmpd.in"] insecure_registries = ["insecure.container.local.jmpd.in:5003"] }

Computed Attributes

These attributes are computed when the config is parsed and applied, and are therefore only known at parsetime or runtime.

---

DockerConfig

Specifies the configuration for the Docker engine in the cluster.

resource "kubernetes_cluster" "cluster" {
  config {
    docker {
      ...
    }
  }
}

Attributes

Attribute	Description
NoProxy no_proxy type: []string	A list of docker registries that should not be proxied. no_proxy = ["insecure.container.local.jmpd.in"]
InsecureRegistries insecure_registries type: []string	A list of insecure docker registries. insecure_registries = ["insecure.container.local.jmpd.in:5003"]

Computed Attributes

These attributes are computed when the config is parsed and applied, and are therefore only known at parsetime or runtime.

---

KubeConfig

Details for the Kubenetes config file that can be used to interact with the cluster.

Attributes

Computed Attributes

These attributes are computed when the config is parsed and applied, and are therefore only known at parsetime or runtime.

Attribute	Description
ConfigPath path type: string	The path to the kubeconfig file
CA ca type: string	The base64 encoded ca certificate
ClientCertificate client_certificate type: string	The base64 encoded client certificate
ClientKey client_key type: string	The base64 encoded client key

Examples

Simple cluster

resource "network" "cloud" {
  subnet = "10.5.0.0/16"
}

resource "kubernetes_cluster" "cluster" {
  network {
    id = resource.network.cloud.meta.id
  }
}

output "KUBECONFIG" {
  value = resource.kubernetes_cluster.cluster.kube_config.path
}

Full Example

resource "network" "cloud" {
  subnet = "10.5.0.0/16"
}

resource "kubernetes_cluster" "cluster" {
  network {
    id = resource.network.cloud.meta.id
  }

  copy_image {
    name = "shipyardrun/connector:v0.1.0"
  }
}

resource "k8s_config" "fake_service" {
  cluster = resource.kubernetes_cluster.cluster

  paths = ["./fake_service.yaml"]

  health_check {
    timeout = "240s"
    pods    = ["app.kubernetes.io/name=fake-service"]
  }
}

resource "helm" "vault" {
  cluster = resource.kubernetes_cluster.cluster

  repository {
    name = "hashicorp"
    url  = "https://helm.releases.hashicorp.com"
  }

  chart   = "hashicorp/vault"
  version = "v0.18.0"

  values = "./helm/vault-values.yaml"

  health_check {
    timeout = "240s"
    pods    = ["app.kubernetes.io/name=vault"]
  }
}

resource "ingress" "vault_http" {
  port = 18200

  target {
    resource = resource.kubernetes_cluster.cluster
    port = 8200

    config = {
      service   = "vault"
      namespace = "default"
    }
  }
}

resource "ingress" "fake_service" {
  port = 19090

  target {
    resource = resource.kubernetes_cluster.cluster
    port = 9090

    config = {
      service   = "fake-service"
      namespace = "default"
    }
  }
}

output "VAULT_ADDR" {
  value = "http://${resource.ingress.vault_http.local_address}"
}

output "KUBECONFIG" {
  value = resource.kubernetes_cluster.cluster.kube_config.path
}