Setup Highly Available Kubernetus cluster with Hetzner Cloud and Terraform

5 min readDec 28, 2020

First of all — thanks Niclas Mietz and all contributors for this awesome github repo

In this implementation you can set up more that 1 master, but it doesn’t work HA cluster at all(((

So I’ve started to dig more and more into this and learn how terraform works in general and how it interact with Hetzner cloud API. Full doc for Hetzner cloud terraform provider you can find here. My fork on github https://github.com/aslubsky/terraform-k8s-hcloud

Load Balancer

We go to official docs and see first step “Create load balancer for kube-apiserver”. Add Load Balancer resource in terraform:

resource "hcloud_load_balancer" "kube_load_balancer" {
   name       = "kube-lb"
   load_balancer_type = "lb11"
   location   = var.location
}resource "hcloud_load_balancer_service" "kube_load_balancer_service" {
   load_balancer_id = hcloud_load_balancer.kube_load_balancer.id
   protocol = "tcp"
   listen_port = 6443
   destination_port = 6443
}resource "hcloud_load_balancer_target" "load_balancer_target" {
  count       = var.master_count
  depends_on  = [hcloud_server.master]
  type = "server"
  server_id = hcloud_server.master[count.index].id
  load_balancer_id = hcloud_load_balancer.kube_load_balancer.id
}

Here we have egg/chicken dilemma, because on terraform you can add server resource to LB resource only after creation, but you can’t bootstrap kubeadmin without LB. So I’ve split provision on 3 phases:

01-main.tf — create all infrastructure resources

02-kube-init.tf — setup Kubernetes HA cluster

03-kube-post-init.tf — setup CNI, CSI and etc

01-main.tf

It’s basically same terraform manifaest as `main.tf` in original repository. It setup X master nodes, Y worker nodes, LB for kubernetes control plane, join all resources in Hetzner private network.

Cluster joined in Hetnzer Private Network

02-kube-init.tf

We should call `kubeadm init` only once, in other case it’ll generate new certificates for each master node. So I’ve splited masters init process to 2 phases:
1) run `kubeadm init` on 1st master and save `join command` output
2) run `join command` on all next masters

echo "
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
  podSubnet: 10.244.0.0/16
controlPlaneEndpoint: "$LB_IP:6443"
controllerManagerExtraArgs:
  address: 0.0.0.0
schedulerExtraArgs:
  address: 0.0.0.0
controllerManager:
  extraArgs:
    bind-address: 0.0.0.0
scheduler:
  extraArgs:
    address: 0.0.0.0
    bind-address: 0.0.0.0
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
metricsBindAddress: 0.0.0.0:10249
" > /tmp/kubeadm.ymlecho "Initialize Cluster"
if [[ -n "$FEATURE_GATES" ]]; then
  kubeadm init --upload-certs --feature-gates "$FEATURE_GATES" --config /tmp/kubeadm.yml
else
  kubeadm init --upload-certs --config /tmp/kubeadm.yml
fi# used to join nodes to the cluster
kubeadm token create --print-join-command >/tmp/kubeadm_joinkubeadm init phase upload-certs --upload-certs >/tmp/cert.key
export CERT_KEY="$(tail -1 /tmp/cert.key)"
kubeadm token create --print-join-command --certificate-key $CERT_KEY >/tmp/kubeadm_control_plane_joinmkdir -p "$HOME/.kube"
cp /etc/kubernetes/admin.conf "$HOME/.kube/config"systemctl enable docker kubelet

and

if (($MASTER_INDEX == 0)); then
  echo "Skip"
else
  echo "Join to Cluster"
  eval "$(cat /tmp/kubeadm_control_plane_join)"
  systemctl enable docker kubelet
fi

In nodes part we should generate join token each time, because when we will scale our cluster after some time join tokens from `${path.module}/secrets/kubeadm_join` may be expired. So we change it and run

kubeadm token create --print-join-command > /tmp/kubeadm_join_${count.index + 1}

on each worker node during `init_workers` job.

Kube-controller

Also I’ve changed 10-kubeadm.conf file, adding cloud-provider flag:

Environment="KUBELET_EXTRA_ARGS=--cloud-provider=external"

Cilium

Unfortunately Hetzner Cloud doesn’t have any firewall services:

In our Cloud there is unfortunately no Firewall you can use in front to disable public access. You would need to do that either over the OS Firewall of the Server, or not configure the Public Interface at all. However we do not recommend the second option as cloud init communicates with our metadata service over the the public ip.

But, since 1.9 Cillium CNI has feature hostFirewall So we can protect out nodes from external access. Setup:

KUBECONFIG=secrets/admin.conf helm install cilium cilium/cilium --version 1.9.1 --namespace kube-system --set prometheus.enabled=true --set devices='{eth0}' --set hostFirewall=true

And apply next policy:

apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: "lock-down-all-nodes"
spec:
  description: "Allow a minimum set of required ports on ingress of worker nodes"
  nodeSelector:
    matchLabels:
      k8s:topology.kubernetes.io/region: hel1
  ingress:
    - fromEntities:
        - cluster
        - health
    - toPorts:
        - ports:
            - port: "6443"
              protocol: TCP
            - port: "22"
              protocol: TCP
            - port: "2379"
              protocol: TCP
            - port: "4240"
              protocol: TCP
            - port: "8472"
              protocol: UDP

Topology

In general we have next topology with 2 public IP address:

Kubernetes control plane load balancer
Http/https ingress traffic load balancer

Results

Let’s deploy some ingress and demo app:

KUBECONFIG=secrets/admin.conf helm install -n kube-system tmp-ingress ingress-nginx/ingress-nginx -f demo-ingress.yml
KUBECONFIG=secrets/admin.conf kubectl apply -f demo-app.yml

Kubernetes will create a new LoadBalancer for nginx ingress with name `tmp-http-lb` from annotation `load-balancer.hetzner.cloud/name: “tmp-http-lb”`

Now we can make http requests and see repsonces from different app instancers:

aslubsky@aslubsky-work:~$ curl http://1.2.3.4/testpath
Hello, world!
Version: 1.0.0
Hostname: tmp-web-5cf74bc8c8-8tzcd
aslubsky@aslubsky-work:~$ curl http://1.2.3.4/testpath
Hello, world!
Version: 1.0.0
Hostname: tmp-web-5cf74bc8c8-5lz65
aslubsky@aslubsky-work:~$ curl http://1.2.3.4/testpath
Hello, world!
Version: 1.0.0
Hostname: tmp-web-5cf74bc8c8-5lz65
aslubsky@aslubsky-work:~$ curl http://1.2.3.4/testpath
Hello, world!
Version: 1.0.0
Hostname: tmp-web-5cf74bc8c8-8tzcd
aslubsky@aslubsky-work:~$

Here 1.2.3.4 — public IP of our http LoadBalancer

Conclusions

Using Terraform and Kubernetes you can set up full-fledged cluster on Hetzner Cloud It will cost from 40,97€ per month (9.8€ for 2 LB11 LoadBalancers, 10.47€ for 3 masters CPX11 type, 20.7€ for 3 nodes CPX21 type). It’s probably cheapest Kubernetes cluster for now.