K3S Cluster

Description

This terraform config will create a multi-node k3s cluster using k3d command-line inside a separate docker network and can add a private docker registry to push docker images.

graph LR;
host
subgraph docker-network: k3d
  k3d-devinfra-server-0
  k3d-devinfra-agent-0
  k3d-devinfra-agent-1
  registry-localhost
  k3d-devinfra-serverlb
end
host --> k3d-devinfra-serverlb
k3d-devinfra-serverlb-->k3d-devinfra-server-0
k3d-devinfra-serverlb-->k3d-devinfra-agent-0
k3d-devinfra-serverlb-->k3d-devinfra-agent-1
k3d-devinfra-server-0 -.uses.-> registry-localhost
k3d-devinfra-agent-0 -.uses.-> registry-localhost
k3d-devinfra-agent-1 -.uses.-> registry-localhost

Installation

Make sure docker is up and running.

Create the cluster

Note

When using WSL2 (Windows Subsystem for Linux), you need to copy the context configuration to your .kubeconfig file on the WSL2 home directory or the other way round, depending on which shell you use to run the terraform code.

cp /mnt/c/users/$(whoami)/.kube/config ~/.kube/config

or

cp ~/.kube/config /mnt/c/users/$(whoami)/.kube/config

Create a .env file to store your DOCKER_HOST location. If left empty it'll take the default. Make sure there is no newline at the end of the file.

To find the value of your current DOCKER_HOST

docker context inspect --format='{{.Endpoints.docker.Host}}'

Next, just apply the terraform code as usual.

cd ./src/clusters/k3s
echo "DOCKER_HOST=" >> .env
terraform init --upgrade
terraform apply --auto-approve

The cluster creation automatically add configuration to your kubectl contexts connect to so you can connect to the new cluster

kubectl cluster-info --context k3d-devinfra

Destroy the cluster

terraform destroy --auto-approve

Terraform

Requirements

Name	Version
terraform	>= 1.0
docker	~> 3.0.1

Providers

Name	Version
docker	3.0.1
local	2.4.0
null	3.2.1

Modules

No modules.

Resources

Name	Type
docker_network.k3d_network	resource
local_file.cluster-config	resource
null_resource.cluster_create	resource
null_resource.cluster_delete	resource

Inputs

Name	Description	Type	Default	Required
cluster-name	Name of the cluster being created	`string`	`"devinfra"`	no
expose_azurite	Expose Azurite externally	`bool`	`false`	no
expose_jaeger	Expose Jaeger externally	`bool`	`false`	no
expose_loki	Expose Loki externally	`bool`	`false`	no
expose_nosql_mongodb	Expose MongoDb externally	`bool`	`false`	no
expose_opentelemetry	Expose OpenTelemetry Collector externally	`bool`	`false`	no
expose_rabbitmq	Expose rabbitmq externally	`bool`	`false`	no
expose_rds_mariadb	Expose MariaDb externally	`bool`	`false`	no
expose_rds_mssql	Expose SQL Server externally	`bool`	`false`	no
expose_rds_mysql	Expose Mysql externally	`bool`	`false`	no
expose_rds_oracle	Expose Oracle externally	`bool`	`false`	no
expose_rds_postgres	Expose Postgres externally	`bool`	`false`	no
expose_redis	Expose Redis externally	`bool`	`false`	no
expose_seq	Expose Seq externally	`bool`	`false`	no
install_private_registry	Whether to start a private registry	`bool`	`true`	no
k3d-cluster-domain	The default domain name used in the cluster	`string`	n/a	yes
k3d-cluster-ip	The Cluster IP	`string`	`"0.0.0.0"`	no
k3d-cluster-name	K3DS cluster prefix	`list(string)`	[ "k3d" ]	no
k3d-cluster-port	The port for the k8s API server	`number`	`6445`	no
k3d-network-gateway	The network gateway for the cluster	`string`	`"172.19.0.1"`	no
k3d-network-subnet	The subnet for the cluster	`string`	`"172.19.0.0/16"`	no
k3s-version	K3S version to use for the k3s containers	`string`	`"latest"`	no
server-node-count	The amount of server nodes	`number`	`1`	no
worker-node-count	The amount of worker nodes	`number`	`0`	no

Outputs

Name	Description
clusters_created	The named of the created cluster
expose_azurite	The cluster is listen on Azurite ports (10000-10002)
expose_jaeger	The cluster is listen on Jaeger ports (6831-6832)
expose_loki	The cluster is listen on Loki port (3100)
expose_nosql_mongodb	The cluster is listen on MongoDB port (27017)
expose_opentelemetry	The cluster is listen on OpenTelemetry Collector ports(4317-4318)
expose_rabbitmq	The cluster is listen on Rabbitmq port (5672)
expose_rds_mariadb	The cluster is listen on MariaDB port (3307) (trying not to conflict with MySQL)
expose_rds_mssql	The cluster is listen on SQL Server port (1433)
expose_rds_mysql	The cluster is listen on MySQL port (3306)
expose_rds_oracle	The cluster is listen on Oracle port (1521)
expose_rds_postgres	The cluster is listen on Postgres port (5432)
expose_seq	The cluster is listen on Seq port (5341)
k3d_api_port	Cluster port
k3d_cluster_ip	Cluster IP