diff --git a/content/software/k8s-on-nixos-part0.md b/content/software/k8s-on-nixos-part0.md new file mode 100644 index 0000000..2fd2ee9 --- /dev/null +++ b/content/software/k8s-on-nixos-part0.md @@ -0,0 +1,163 @@ +--- +Title: K8s on NixOS - Chapter 0: Preface +Date: 2025-03-14 +Category: software +Tags: k8s, nix, nixos, ipv6, flakes, server +Slug: k8s-on-nixos-chapter0-preface +Summary: Setting up a Kubernetes cluster can be quite a challange. Let set the stage and define the rules. +--- + +# IPv6 only + +The whole setup will be IPv6 only. I will not configure any IPv4 because I don't need legacy IP support. If you have +legacy workloads that require IPv4 connectivity, you will have to think about adding a SIIT + NAT64 setup with something +like [JOOL][jool] or building everything dual stack. Both ways complicate the setup in different ways. You decide how +you want to solve that issue. Maybe I'll add a SIIT-DC setup later just for fun. +It's probably the easiest to get rid of your IPv4-only software anyway. If you want to create an IPv4 only setup you +should be able to follow this guide and simply replace every mentioned IPv6 thing with your IPv4 equivalent. But +honestly, why would you do that? It's 2025, [IPv6][ipv6rfc] is now over 30 years old. There should be no reason to not +support it as first class citizen. Do this first and think about legacy support later when you actually need it. + +There will be an exception for the Kubernetes Ingress Controller. It needs to be reachable by public IPv4 addresses so +that legacy users can still reach my public services. They are punished enough by their ISPs, let's not add more to +that. + +# Public addresses + +I will use public IPv6 addresses for many components. That means services are available to the public internet. +Firewall rules will need to make sure that only our components talk to non-public endpoints. This will be done with +nftables firewall rules that block outside traffic to our Pod and Service networks. + +I'm still undecided if I should use ULA addresses for the Pod and Service networks or Global Unicast. There are many +ways to set this up and none of them are wrong per se. Let's see how thing evolve when we get to the CNI plugin setup. + +# Combined control plane and worker nodes + +In a production setup you would usually separate kubernetes worker nodes and control plane nodes. This is better for +security and it also gives you the ability to easier scale them independent from each other. Because I only have 3 +dedicated nodes at the moment and I want to simplify the setup I will combine these roles in this guide. This doesn't +stop you from adding more nodes with dedicated roles in the future. In fact once this setup works I will migrate it +from 3 VMs to 4 hardware nodes with one being only a worker node. + +A control plane node would only host etcd, apiserver, scheduler, controller manager, addon manager and proxy services. +A worker node would only host a container runtime, kubelet and proxy services. Depending on your network plugin it would +probably also be installed on both node roles. In NixOS this is handled by the `services.kubernetes.roles` option. It +overwrites the `enabled` option of kubernetes components. For that reason we will not use that and take fine grained +control over our services. + +# Basic setup + +We will put everything we do in a single git repository. So let's set this up now. + +Let’s start with a standard gitignore file. The most important thing is the _direnv_ part that keeps us from +accidentally including our nix flake outputs. I will use PyCharm and Vim and both of them will generate some files that +you don't want so I'll exclude these too. And finally we will exclude all OS level stuff. + +```bash +curl -sL "gitignore.io/api/linux,windows,macos,direnv,pycharm+all,vim" > ./.gitignore +``` + +We will have a `pki` directory which consists of our certificate autorities and certificates. In there we need a +another gitignore file, so add a `./pki/.gitignore` like this: + +```gitignore +# don't include unencrypted certificates +*/*.pem +``` + +This will prevent you from accidentally making your precious certificates public. + +We will also have a `services` directory which holds our Nix configuration of the services we will setup. This makes it +easy to include them only on the nodes where we need them. + +The next direcotry is `secrets`. It will hold our [age][age] encrypted secrets. So no worries this should be plenty +secure to put them into git. At least for now. &xF609; + +Lastly we will have a `deployments` directory with our Kubernetes manifests. I will probably come up with a better way +of handling those at some point but for now they live there. + +Let's put a simple flake in there too. + +```nix +{ + inputs = { + nixpkgs.url = github:NixOS/nixpkgs/nixos-24.11; + agenix = { + url = github:ryantm/agenix; + inputs = { + nixpkgs.follows = "nixpkgs"; + }; + }; + }; + outputs = { self, nixpkgs, agenix }: + let + system = "x86_64-linux"; + pkgs = import nixpkgs { inherit system; }; + in + { + formatter.x86_64-linux = pkgs.nixpkgs-fmt; + devShells.x86_64-linux.default = pkgs.mkShell { + packages = with pkgs; [ + agenix.packages.x86_64-linux.default + age # secrets debugging + gnumake # automation + ]; + }; + } +} +``` + +Let's also add a helping `Makefile` that doesn't do much for now: + +```make +.DEFAULT_GOAL := help + +.PHONY: all +all: update-flake k8s ## Update and deploy k8s cluster + +.PHONY: update-flake +update-flake: ## Update nix flake + nix flake update + +.PHONY: k8s +k8s: k8s-master k8s-worker ## Deploy k8s cluster + +.PHONY: k8s-master +k8s-master: ## Deploy k8s control plane + +.PHONY: k8s-worker +k8s-worker: ## Deploy k8s workers + +.PHONY: help +help: ## Display this help + @grep -h -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' +``` + +This is the final layout so far: + +``` +. +├── Makefile +├── README.md +├── deployments/ +├── flake.lock +├── flake.nix +├── pki +│   └── Makefile +├── secrets/ +│   └── secrets.nix +└── services/ +``` + +We will add more stuff later on. + +Good bye for now! + + +[jool]: https://www.jool.mx/en/index.html +[ipv6rfc]: https://datatracker.ietf.org/doc/html/rfc1883 +[age]: https://age-encryption.org/ + +*[SIIT]: Stateless IP/ICMP Translation +*[NAT64]: Network Address Translation from IPv6 to IPv4 +*[CNI]: Container Network Interface