k8s on nixos chapter 0
This commit is contained in:
Ricardo (XenGi) Band
parent
19b5502329
commit
ff51b70fe5
1 changed files with 163 additions and 0 deletions
163
content/software/k8s-on-nixos-part0.md
Normal file
163
content/software/k8s-on-nixos-part0.md
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
---
|
||||
Title: K8s on NixOS - Chapter 0: Preface
|
||||
Date: 2025-03-14
|
||||
Category: software
|
||||
Tags: k8s, nix, nixos, ipv6, flakes, server
|
||||
Slug: k8s-on-nixos-chapter0-preface
|
||||
Summary: Setting up a Kubernetes cluster can be quite a challange. Let set the stage and define the rules.
|
||||
---
|
||||
|
||||
# IPv6 only
|
||||
|
||||
The whole setup will be IPv6 only. I will not configure any IPv4 because I don't need legacy IP support. If you have
|
||||
legacy workloads that require IPv4 connectivity, you will have to think about adding a SIIT + NAT64 setup with something
|
||||
like [JOOL][jool] or building everything dual stack. Both ways complicate the setup in different ways. You decide how
|
||||
you want to solve that issue. Maybe I'll add a SIIT-DC setup later just for fun.
|
||||
It's probably the easiest to get rid of your IPv4-only software anyway. If you want to create an IPv4 only setup you
|
||||
should be able to follow this guide and simply replace every mentioned IPv6 thing with your IPv4 equivalent. But
|
||||
honestly, why would you do that? It's 2025, [IPv6][ipv6rfc] is now over 30 years old. There should be no reason to not
|
||||
support it as first class citizen. Do this first and think about legacy support later when you actually need it.
|
||||
|
||||
There will be an exception for the Kubernetes Ingress Controller. It needs to be reachable by public IPv4 addresses so
|
||||
that legacy users can still reach my public services. They are punished enough by their ISPs, let's not add more to
|
||||
that.
|
||||
|
||||
# Public addresses
|
||||
|
||||
I will use public IPv6 addresses for many components. That means services are available to the public internet.
|
||||
Firewall rules will need to make sure that only our components talk to non-public endpoints. This will be done with
|
||||
nftables firewall rules that block outside traffic to our Pod and Service networks.
|
||||
|
||||
I'm still undecided if I should use ULA addresses for the Pod and Service networks or Global Unicast. There are many
|
||||
ways to set this up and none of them are wrong per se. Let's see how thing evolve when we get to the CNI plugin setup.
|
||||
|
||||
# Combined control plane and worker nodes
|
||||
|
||||
In a production setup you would usually separate kubernetes worker nodes and control plane nodes. This is better for
|
||||
security and it also gives you the ability to easier scale them independent from each other. Because I only have 3
|
||||
dedicated nodes at the moment and I want to simplify the setup I will combine these roles in this guide. This doesn't
|
||||
stop you from adding more nodes with dedicated roles in the future. In fact once this setup works I will migrate it
|
||||
from 3 VMs to 4 hardware nodes with one being only a worker node.
|
||||
|
||||
A control plane node would only host etcd, apiserver, scheduler, controller manager, addon manager and proxy services.
|
||||
A worker node would only host a container runtime, kubelet and proxy services. Depending on your network plugin it would
|
||||
probably also be installed on both node roles. In NixOS this is handled by the `services.kubernetes.roles` option. It
|
||||
overwrites the `enabled` option of kubernetes components. For that reason we will not use that and take fine grained
|
||||
control over our services.
|
||||
|
||||
# Basic setup
|
||||
|
||||
We will put everything we do in a single git repository. So let's set this up now.
|
||||
|
||||
Let’s start with a standard gitignore file. The most important thing is the _direnv_ part that keeps us from
|
||||
accidentally including our nix flake outputs. I will use PyCharm and Vim and both of them will generate some files that
|
||||
you don't want so I'll exclude these too. And finally we will exclude all OS level stuff.
|
||||
|
||||
```bash
|
||||
curl -sL "gitignore.io/api/linux,windows,macos,direnv,pycharm+all,vim" > ./.gitignore
|
||||
```
|
||||
|
||||
We will have a `pki` directory which consists of our certificate autorities and certificates. In there we need a
|
||||
another gitignore file, so add a `./pki/.gitignore` like this:
|
||||
|
||||
```gitignore
|
||||
# don't include unencrypted certificates
|
||||
*/*.pem
|
||||
```
|
||||
|
||||
This will prevent you from accidentally making your precious certificates public.
|
||||
|
||||
We will also have a `services` directory which holds our Nix configuration of the services we will setup. This makes it
|
||||
easy to include them only on the nodes where we need them.
|
||||
|
||||
The next direcotry is `secrets`. It will hold our [age][age] encrypted secrets. So no worries this should be plenty
|
||||
secure to put them into git. At least for now. &xF609;
|
||||
|
||||
Lastly we will have a `deployments` directory with our Kubernetes manifests. I will probably come up with a better way
|
||||
of handling those at some point but for now they live there.
|
||||
|
||||
Let's put a simple flake in there too.
|
||||
|
||||
```nix
|
||||
{
|
||||
inputs = {
|
||||
nixpkgs.url = github:NixOS/nixpkgs/nixos-24.11;
|
||||
agenix = {
|
||||
url = github:ryantm/agenix;
|
||||
inputs = {
|
||||
nixpkgs.follows = "nixpkgs";
|
||||
};
|
||||
};
|
||||
};
|
||||
outputs = { self, nixpkgs, agenix }:
|
||||
let
|
||||
system = "x86_64-linux";
|
||||
pkgs = import nixpkgs { inherit system; };
|
||||
in
|
||||
{
|
||||
formatter.x86_64-linux = pkgs.nixpkgs-fmt;
|
||||
devShells.x86_64-linux.default = pkgs.mkShell {
|
||||
packages = with pkgs; [
|
||||
agenix.packages.x86_64-linux.default
|
||||
age # secrets debugging
|
||||
gnumake # automation
|
||||
];
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Let's also add a helping `Makefile` that doesn't do much for now:
|
||||
|
||||
```make
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
.PHONY: all
|
||||
all: update-flake k8s ## Update and deploy k8s cluster
|
||||
|
||||
.PHONY: update-flake
|
||||
update-flake: ## Update nix flake
|
||||
nix flake update
|
||||
|
||||
.PHONY: k8s
|
||||
k8s: k8s-master k8s-worker ## Deploy k8s cluster
|
||||
|
||||
.PHONY: k8s-master
|
||||
k8s-master: ## Deploy k8s control plane
|
||||
|
||||
.PHONY: k8s-worker
|
||||
k8s-worker: ## Deploy k8s workers
|
||||
|
||||
.PHONY: help
|
||||
help: ## Display this help
|
||||
@grep -h -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
|
||||
```
|
||||
|
||||
This is the final layout so far:
|
||||
|
||||
```
|
||||
.
|
||||
├── Makefile
|
||||
├── README.md
|
||||
├── deployments/
|
||||
├── flake.lock
|
||||
├── flake.nix
|
||||
├── pki
|
||||
│ └── Makefile
|
||||
├── secrets/
|
||||
│ └── secrets.nix
|
||||
└── services/
|
||||
```
|
||||
|
||||
We will add more stuff later on.
|
||||
|
||||
Good bye for now!
|
||||
|
||||
|
||||
[jool]: https://www.jool.mx/en/index.html
|
||||
[ipv6rfc]: https://datatracker.ietf.org/doc/html/rfc1883
|
||||
[age]: https://age-encryption.org/
|
||||
|
||||
*[SIIT]: Stateless IP/ICMP Translation
|
||||
*[NAT64]: Network Address Translation from IPv6 to IPv4
|
||||
*[CNI]: Container Network Interface
|
||||
Loading…
Reference in a new issue