Setting up my K3S Cluster on Proxmox

After my week working with Docker, it’s now time to setup my initial Kubernetes cluster. I will be using k3s as it’s a nice lightweight kubernetes distribution.

The Virtual Machines

I am using my Proxmox machine for this, and my VM setup will look like this:

k3s-1: A control plane with 2CPUs, 2GB RAM, 16GB disk
k3s-2 and k3s-3: 2 worker nodes with 1CPU, 2GB RAM, 16GB disk each.

I’m keeping this lean on purpose to avoid having oversized nodes hiding my potential mistakes. Each node will be running on Unbuntu Server.

During the installation, I also had to expose IPs and setup ssh. This was similar to what I did for my previous Proxmox VM.

Installing k3s

as recommended in the k3s guide, I disabled ufw:

sudo ufw disable

on k3s-1, I ran the install script:

curl -sfL https://get.k3s.io | sh -

Then I grabbed the node token:

sudo cat /var/lib/rancher/k3s/server/node-token

on k3s-2 and k3s-3, I installed the agent nodes and added them to the cluster

curl -sfL https://get.k3s.io | K3S_URL=https://k3s-1:6443 K3S_TOKEN=mynodetoken sh -

curl -sfL https://get.k3s.io | K3S_URL=https://k3s-1:6443 K3S_TOKEN=K1031970dd0f1cc0dd45a12e791195c13bc28b5bf91e219cf04d300b6ffea7d71a1::server:f07d89c5588a6dbf0f0e2992b16e12d9 sh -

On k3s-1, I could verify that all nodes show up

sudo kubectl get node

# Output:
NAME    STATUS   ROLES           AGE     VERSION
k3s-1   Ready    control-plane   9m18s   v1.34.5+k3s1
k3s-2   Ready    <none>          2m13s   v1.34.5+k3s1
k3s-3   Ready    <none>          112s    v1.34.5+k3s1

Setting up the k3s nodes to use the private docker registry

On each node, I added a registries.yaml file to add my private registry setup. I also copied the ca file, similar as what was done in my previous post.

mirrors:
  "docker-vm-1:5000":
    endpoint:
      - "https://docker-vm-1:5000"
configs:
  "docker-vm-1:5000":
    tls:
      ca_file: /etc/ssl/certs/mkcert-ca.pem

I restarted k3s to make the change apply:

sudo systemctl restart k3s # on k3s-1
sudo systemctl restart k3s # on k3s-2 and k3s-3

Controlling the k3s cluster from my laptop

I grabbed the k3s config file from /etc/rancher/k3s/k3s.yaml to my ~/.kube folder, and edited it to have the proper server location and context name. I added that file to my kubeconfig:

export KUBECONFIG=~/.kube/config:~/.kube/k3s.yaml

Using kubectx, I can now switch context between clusters.

Restricting workloads to be applied on the Control Plane node

In production Kubernetes, the control plane node doesn’t have workloads applied on it. To mimic that in my k3s setup, I added a taint

kubectl taint nodes k3s-1 node-role.kubernetes.io/control-plane:NoSchedule

Snapshots

After checking that the connection was working fine, I took snapshots of each k3s node through proxmoxI took snapshots of each k3s node through proxmoxI took snapshots of each k3s node through proxmoxI took snapshots of each k3s node through proxmox. This way I can easily go back to a fresh state if needed.

Summary

At that point, I have a simple 3 node k3s cluster running on my proxmox setup, with private docker registry ready to use. Perfect place to start practicing my Kubernetes skills!