5.9 KiB
Title: K8s on NixOS - Chapter 0: Preface Date: 2025-03-14 Category: software Tags: k8s, nix, nixos, ipv6, flakes, server Slug: k8s-on-nixos-chapter0-preface Summary: Setting up a Kubernetes cluster can be quite a challange. Let set the stage and define the rules.
IPv6 only
The whole setup will be IPv6 only. I will not configure any IPv4 because I don't need legacy IP support. If you have legacy workloads that require IPv4 connectivity, you will have to think about adding a SIIT + NAT64 setup with something like JOOL or building everything dual stack. Both ways complicate the setup in different ways. You decide how you want to solve that issue. Maybe I'll add a SIIT-DC setup later just for fun. It's probably the easiest to get rid of your IPv4-only software anyway. If you want to create an IPv4 only setup you should be able to follow this guide and simply replace every mentioned IPv6 thing with your IPv4 equivalent. But honestly, why would you do that? It's 2025, IPv6 is now over 30 years old. There should be no reason to not support it as first class citizen. Do this first and think about legacy support later when you actually need it.
There will be an exception for the Kubernetes Ingress Controller. It needs to be reachable by public IPv4 addresses so that legacy users can still reach my public services. They are punished enough by their ISPs, let's not add more to that.
Public addresses
I will use public IPv6 addresses for many components. That means services are available to the public internet. Firewall rules will need to make sure that only our components talk to non-public endpoints. This will be done with nftables firewall rules that block outside traffic to our Pod and Service networks.
I'm still undecided if I should use ULA addresses for the Pod and Service networks or Global Unicast. There are many ways to set this up and none of them are wrong per se. Let's see how thing evolve when we get to the CNI plugin setup.
Combined control plane and worker nodes
In a production setup you would usually separate kubernetes worker nodes and control plane nodes. This is better for security and it also gives you the ability to easier scale them independent from each other. Because I only have 3 dedicated nodes at the moment and I want to simplify the setup I will combine these roles in this guide. This doesn't stop you from adding more nodes with dedicated roles in the future. In fact once this setup works I will migrate it from 3 VMs to 4 hardware nodes with one being only a worker node.
A control plane node would only host etcd, apiserver, scheduler, controller manager, addon manager and proxy services.
A worker node would only host a container runtime, kubelet and proxy services. Depending on your network plugin it would
probably also be installed on both node roles. In NixOS this is handled by the services.kubernetes.roles option. It
overwrites the enabled option of kubernetes components. For that reason we will not use that and take fine grained
control over our services.
Basic setup
We will put everything we do in a single git repository. So let's set this up now.
Let’s start with a standard gitignore file. The most important thing is the direnv part that keeps us from accidentally including our nix flake outputs. I will use PyCharm and Vim and both of them will generate some files that you don't want so I'll exclude these too. And finally we will exclude all OS level stuff.
curl -sL "gitignore.io/api/linux,windows,macos,direnv,pycharm+all,vim" > ./.gitignore
We will have a pki directory which consists of our certificate autorities and certificates. In there we need a
another gitignore file, so add a ./pki/.gitignore like this:
# don't include unencrypted certificates
*/*.pem
This will prevent you from accidentally making your precious certificates public.
We will also have a services directory which holds our Nix configuration of the services we will setup. This makes it
easy to include them only on the nodes where we need them.
The next direcotry is secrets. It will hold our age encrypted secrets. So no worries this should be plenty
secure to put them into git. At least for now. &xF609;
Lastly we will have a deployments directory with our Kubernetes manifests. I will probably come up with a better way
of handling those at some point but for now they live there.
Let's put a simple flake in there too.
{
inputs = {
nixpkgs.url = github:NixOS/nixpkgs/nixos-24.11;
agenix = {
url = github:ryantm/agenix;
inputs = {
nixpkgs.follows = "nixpkgs";
};
};
};
outputs = { self, nixpkgs, agenix }:
let
system = "x86_64-linux";
pkgs = import nixpkgs { inherit system; };
in
{
formatter.x86_64-linux = pkgs.nixpkgs-fmt;
devShells.x86_64-linux.default = pkgs.mkShell {
packages = with pkgs; [
agenix.packages.x86_64-linux.default
age # secrets debugging
gnumake # automation
];
};
}
}
Let's also add a helping Makefile that doesn't do much for now:
.DEFAULT_GOAL := help
.PHONY: all
all: update-flake ## Update flake inputs
.PHONY: update-flake
update-flake: ## Update nix flake
nix flake update
.PHONY: help
help: ## Display this help
@grep -h -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
This is the final layout so far:
.
├── Makefile
├── README.md
├── deployments/
├── flake.lock
├── flake.nix
├── pki
│ └── Makefile
├── secrets/
│ └── secrets.nix
└── services/
We will add more stuff later on.
Good bye for now!
*[SIIT]: Stateless IP/ICMP Translation *[NAT64]: Network Address Translation from IPv6 to IPv4 *[CNI]: Container Network Interface