xengi/www.xengi.de
xengi/www.xengi.de
Watch
1
0
Fork
You've already forked www.xengi.de
 0
Code Activity Actions

missing updates?

Browse source
This commit is contained in:
Ricardo (XenGi) Band 2025-04-06 01:00:06 +02:00
commit 8b9e197b17
No known key found for this signature in database
4 changed files with 458 additions and 132 deletions
Download patch file Download diff file Expand all files Collapse all files

312
content/software/k8s-on-nixos-part1.md
View file

@ -1,13 +1,14 @@
2001:db8:0:c::/64---
---
Title: K8s on NixOS - Chapter 1: Getting the nodes ready
Date: 2025-03-14
Date: 2025-03-17
Category: software
Tags: nix, nixos, flakes, server, qemu, libvirt, ssh, make
Slug: k8s-on-nixos-chapter1-nodes
Summary: In this chapter we will setup 3 nodes running NixOS. They will host our future Kubernetes cluster.
Status: Draft
---
If you missed the first part of this guide find it here: [K8s on NixOS - Chapter 0: Preface]({filename}/software/k8s-on-nixos-part0.md)
# IPAM
Before we start, let's write down the IP addresses we will use so we don't get them confused. I will use documentation
@ -15,32 +16,32 @@ prefixes throughout this guide. Replace them with your own ranges accordingly.
| IP address/network | Usage |
|:------------------------------- | ----- |
| 2606:4700:4700::1111/128 | Cloudflare DNS server |
| 2620:fe::fe/128 | Quad9 DNS server |
| 2606:4700:4700::1111/128 | [Cloudflare][cloudflare] DNS server |
| 2620:fe::fe/128 | [Quad9][quad9] DNS server |
| 2001:db8::/56 | Provided prefix by ISP/Hoster |
| ├── 2001:db8::1/64 | Default Gateway |
| │   ├── 2001:db8:0:a::/64 | Nodes |
| │   │ ├── 2001:db8:0:a::1/128 | Node-01 |
| │   │ ├── 2001:db8:0:a::2/128 | Node-02 |
| │   │ └── 2001:db8:0:a::3/128 | Node-03 |
| │   ├── 2001:db8:0:b::/112 | Kubernetes services (65536 service IPs) |
| │   └── 2001:db8:0:c::/64 | Kubernetes pods |
| │   ├── 2001:db8:0:c:1::/80 | Kubernetes pods on node-01 (256k pod IPs) |
| │   ├── 2001:db8:0:c:2::/80 | Kubernetes pods on node-02 (256k pod IPs) |
| │   └── 2001:db8:0:c:3::/80 | Kubernetes pods on node-03 (256k pod IPs) |
| └── 2001:db8:ff::1:2:3/128 | Example IP for a jump host or client |
| 1.1.1.1/32 | Cloudflare fallback DNS server |
| 9.9.9.9/32 | Quad9 fallback DNS server |
| ├─ 2001:db8::1/64 | Default Gateway |
| │ ├─ 2001:db8:0:a::/64 | Nodes |
| │ │ ├─ 2001:db8:0:a::1/128 | Node-01 |
| │ │ ├─ 2001:db8:0:a::2/128 | Node-02 |
| │ │ └─ 2001:db8:0:a::3/128 | Node-03 |
| │ ├─ 2001:db8:0:b::/112 | Kubernetes services (65536 service IPs) |
| │ └─ 2001:db8:0:c::/64 | Kubernetes pods |
| │   ├─ 2001:db8:0:c:1::/80 | Kubernetes pods on node-01 (256k pod IPs) |
| │   ├─ 2001:db8:0:c:2::/80 | Kubernetes pods on node-02 (256k pod IPs) |
| │   └─ 2001:db8:0:c:3::/80 | Kubernetes pods on node-03 (256k pod IPs) |
| └─ 2001:db8:ff::1:2:3/128 | Example IP for a jump host or client |
| 1.1.1.1/32 | [Cloudflare][cloudflare] fallback DNS server |
| 9.9.9.9/32 | [Quad9][quad9] fallback DNS server |
| 192.0.2.0/24 | Node network |
| ├── 192.0.2.254/32 | Default Gateway |
| ├── 192.0.2.1/32 | Node-01 |
| ├── 192.0.2.2/32 | Node-02 |
| └── 192.0.2.3/32 | Node-03 |
| ├─ 192.0.2.254/32 | Default Gateway |
| ├─ 192.0.2.1/32 | Node-01 |
| ├─ 192.0.2.2/32 | Node-02 |
| └─ 192.0.2.3/32 | Node-03 |
| 198.51.100.123/32 | Example IP for a jump host or client |
# Getting the nodes ready
For simplicity, all nodes in this guide are QEMU VMs. In reality these can be anything, like hardware servers or
For simplicity, all nodes in this guide are [QEMU][qemu] VMs. In reality these can be anything, like hardware servers or
Raspberry Pis. Just adapt the setup to your needs. Especially when you use ARM SBCs, check out
[flake-utils](https://github.com/numtide/flake-utils), which was mentioned in the last chapter.
@ -57,8 +58,8 @@ Create a new VM with the wizard like this:
2. Use the ISO and choose `NixOS 24.11` or `NixOS Unstable` as OS
3. Use at least 4GB of memory and 4 cores
4. Create at least a 64GB disk to have enough storage for the Nix store and container images
5. Call it `k8s-node-01` and hit `Finish` or check the `Customize configuration before install` to add more as options
as needed
5. Call it `k8s-node-01` and hit `Finish` or check the `Customize configuration before install` to review and add more
options
I usually customize the config like this:
@ -70,7 +71,7 @@ I usually customize the config like this:
- Boot options:
- Start VM on host boot up
Boot the NixOS installer and setup partitions like this:
Boot the NixOS installer and setup partitions like this for a UEFI setup:
```shell
sudo cfdisk /dev/vda
@ -109,6 +110,9 @@ vda 253:0 0 64G 0 disk
└─vda3 253:3 0 59G 0 part /mnt
```
If you want to use a traditional BIOS setup, simply use a `DOS` partition table in cfdisk and skip the creation of a ESP
partition. So you'll end up with just the root partition or root and swap.
Now generate the NixOS config:
```shell
@ -122,75 +126,79 @@ Create a minimal nix configuration like this:
{ config, lib, pkgs, modulesPath, ... }:
{
imports = [
(modulesPath + "/profiles/qemu-guest.nix")
./hardware-configuration.nix
];
imports = [
(modulesPath + "/profiles/qemu-guest.nix")
./hardware-configuration.nix
];
nix.settings.experimental-features = [ "nix-command" "flakes" ];
nix.settings.experimental-features = [ "nix-command" "flakes" ];
boot.loader = {
systemd-boot.enable = true;
efi.canTouchEfiVariables = true;
};
boot.loader = {
systemd-boot.enable = true;
efi.canTouchEfiVariables = true;
};
networking = {
hostName = "node-01";
firewall.enable = true;
useDHCP = true;
useNetworkd = true;
dhcpcd.enable = false;
nftables.enable = true;
};
services.resolved.enable = true;
networking = {
hostName = "node-01";
firewall.enable = true;
useDHCP = true;
useNetworkd = true;
dhcpcd.enable = false;
nftables.enable = true;
};
services.resolved.enable = true;
time.timeZone = "Europe/Berlin";
time.timeZone = "Europe/Berlin";
i18n.defaultLocale = "en_US.UTF-8";
console = {
font = "Lat2-Terminus16";
useXkbConfig = true;
};
i18n.defaultLocale = "en_US.UTF-8";
console = {
font = "Lat2-Terminus16";
useXkbConfig = true;
};
users.users.root.openssh.authorizedKeys.keys = [
"ssh-ed25519 Put-your-ssh-keys-in-here"
];
users.users.root.openssh.authorizedKeys.keys = [
"ssh-ed25519 Put-your-ssh-keys-in-here"
];
environment.systemPackages = with pkgs; [
vim # or any other editor you like
git
];
environment.systemPackages = with pkgs; [
vim # or any other editor you like
git
];
programs.vim = { # or any other editor you like
enable = true;
defaultEditor = true;
};
programs.vim = { # or any other editor you like
enable = true;
defaultEditor = true;
};
services.openssh.enable = true;
services.openssh.enable = true;
system.stateVersion = "24.11"; # never touch that unless you know what you're doing
system.stateVersion = "24.11"; # never touch that unless you know what you're doing
}
```
Tip: Get your SSH keys from github or gitlab with:
Tip: Get your SSH keys from GitHub or GitLab with:
```shell
curl -sL github.com/your-username.keys
curl -sL gitlab.com/your-username.keys
```
Start installation:
Start the installation:
```shell
cd /mnt
sudo nixos-install
```
The installer will ask you for a root password. Choose a strong one. Then `reboot` and run the installation on the
remaining VMs.
In the end, the installer will ask you for a root password. Choose a strong one. Then `reboot` and run the installation
on the remaining VMs.
Yes, we could reuse the image from the first machine for the others but for that we have to prepare some thing to make
them truly unique. As we only have 3 VMs it is probably faster to install them one by one.
Yes, we could reuse the image from the first machine for the others but for that we have to prepare some other things to
make them truly unique and work properly. As we only have 3 VMs it is probably faster to install them one by one than
setting up a proper template from the first image. But feel free to do so if you want to scale out more.
I won't do it here to keep the guide shorter. If you're interested in this topic have a look at
[Packer](https://www.packer.io/) or [nixos-generators](https://github.com/nix-community/nixos-generators).
## Update our flake
@ -202,9 +210,7 @@ After all VMs are installed and ready we will update them from our flake. Here i
nixpkgs.url = github:NixOS/nixpkgs/nixos-24.11;
agenix = {
url = github:ryantm/agenix;
inputs = {
nixpkgs.follows = "nixpkgs";
};
inputs.nixpkgs.follows = "nixpkgs";
};
};
@ -230,18 +236,36 @@ After all VMs are installed and ready we will update them from our flake. Here i
ip6Address = "2001:db8:0:a::1";
ip4Address = "192.0.2.1";
podCidr = "2001:db8:0:c:1::/80";
root-uuid = "237cca24-aaaa-bbbb-cccc-b33ce7e1c60c";
boot-uuid = "3AA9-ABCD";
swap-uuid = "2acde6ef-aaaa-bbbb-cccc-e5dbf51b098e";
stateVersion = "24.11";
}
{
hostName = "node-02";
ip6Address = "2001:db8:0:a::2";
ip4Address = "192.0.2.2";
podCidr = "2001:db8:0:c:2::/80";
root-uuid = "764670d1-aaaa-bbbb-cccc-2f1f0345c9ec";
boot-uuid = "62F5-ABCD";
swap-uuid = "48e980b8-aaaa-bbbb-cccc-4ccb3e753f56";
stateVersion = "24.11";
}
{
hostName = "node-03";
ip6Address = "2001:db8:0:a::3";
ip4Address = "192.0.2.3";
podCidr = "2001:db8:0:c:3::/80";
root-uuid = "1927f79a-aaaa-bbbb-cccc-54f619bd5466";
boot-uuid = "F67C-ABCD";
swap-uuid = "09893a08-aaaa-bbbb-cccc-c9b76fac5d94";
stateVersion = "24.11";
}
];
cidrs = {
@ -271,8 +295,7 @@ After all VMs are installed and ready we will update them from our flake. Here i
{
age.secrets = {} // k8s_secrets;
}
./common.nix
./node-01.nix
./configuration.nix
];
};
"node-02" =
@ -293,8 +316,7 @@ After all VMs are installed and ready we will update them from our flake. Here i
{
age.secrets = {} // k8s_secrets;
}
./common.nix
./node-02.nix
./configuration.nix
];
};
"node-03" =
@ -315,8 +337,7 @@ After all VMs are installed and ready we will update them from our flake. Here i
{
age.secrets = {} // k8s_secrets;
}
./common.nix
./node-03.nix
./configuration.nix
];
};
};
@ -325,13 +346,19 @@ After all VMs are installed and ready we will update them from our flake. Here i
```
The important additions are the `nixosConfigurations` for our 3 nodes. Each of them will receive some secrets we will
define later on. The shared configuration will be put in `common.nix` and the node specific config is in `node-##.nix`.
define later on. I combined the `configuration.nix` and `hardware-configuration.nix` that the installer generates.
### Common configuration
All node specific variables are defined in `clusterNodes`. The `cidrs` variable holds the networks and the `node`
variable will only hold the object from `clusterNodes` of the node we're currently generating.
I combined the `configuration.nix` and `hardware-configuration.nix` that the installer generates.
So for node specific config the `node` variable can be used and if information from all cluster members are needed we
can iterate over the `clusterNodes` variable.
Here is the new `common.nix` file:
We can also split cluster wide and node specific secrets that way.
## New configuration
Here is the new `configuration.nix` file:
```nix
{ config, lib, pkgs, modulesPath, node, ... }:
@ -341,6 +368,21 @@ Here is the new `common.nix` file:
(modulesPath + "/profiles/qemu-guest.nix")
];
fileSystems = {
"/" = {
device = "/dev/disk/by-uuid/${node.root-uuid}";
fsType = "ext4";
options = [ "noatime" ];
};
"/boot" = {
device = "/dev/disk/by-uuid/${node.boot-uuid}";
fsType = "vfat";
options = [ "noatime" "fmask=0022" "dmask=0022" ];
};
};
swapDevices = [{ device = "/dev/disk/by-uuid/${node-swap-uuid}"; }];
boot = {
loader = {
systemd-boot.enable = true;
@ -459,6 +501,8 @@ Here is the new `common.nix` file:
];
};
};
system.stateVersion = node.stateVersion;
}
```
@ -470,7 +514,7 @@ The important bits here are:
- `networking.domain`: choose/buy a nice domain or simply use your local one (e.g. fritz.box). Avoid using
`cluster.local` as this will be used by Kubernetes internally.
- `networking.nameservers`: can be set to your local router for a home lab or any other DNS server you like. I
choose the servers from [cloudflare](https://one.one.one.one/) and [quad9](https://quad9.net/) for performance
choose the servers from [cloudflare][cloudflare] and [quad9][quad9] for performance
and privacy reasons.
- `networking.useDHCP = false`: Disable DHCP for static IP addressing.
- `networking.defaultGateway6`: Set a fixed gateway.
@ -485,7 +529,7 @@ I also added some tools like `mtr` for network debugging, `htop` to monitor proc
if we accidentally kill our network connection and `sshguard` to shield us against brute-force attacks on our SSH
server.
#### IPv6 temporary addresses
### IPv6 temporary addresses
An IPv6 temporary address includes a randomly generated 64-bit number as the interface ID, instead of an interface's
MAC address. You can use temporary addresses for any interfaces on an IPv6 node that you want to keep anonymous. It
@ -495,7 +539,7 @@ We're setting up a server that doesn't need this feature. Even worse, services l
use these temporary addresses for outgoing connections and then fail to connect because we will explicitly allow only
the fixed addresses of our nodes to connect to each other.
#### Enforce DoT
### Enforce DoT
If you chose DNS servers that support it, you can encorce DNS oser TLS by setting
`services.resolved.dnsovertls = "true"`. If you do so, you also have to set valid domain names so that the certificates
@ -510,31 +554,6 @@ networking.nameservers = [
]
```
### Node specific files
```nix
{ config, ... }:
{
fileSystems = {
"/" = {
device = "/dev/disk/by-uuid/1927f79a-aaaa-bbbb-cccc-54f619bd5466";
fsType = "ext4";
options = [ "noatime" ];
};
"/boot" = {
device = "/dev/disk/by-uuid/F67C-ABCD";
fsType = "vfat";
options = [ "noatime" "fmask=0022" "dmask=0022" ];
};
};
swapDevices = [{ device = "/dev/disk/by-uuid/09893a08-aaaa-bbbb-cccc-c9b76fac5d94"; }];
system.stateVersion = "24.11";
}
```
## Update Makefile
The new `Makefile` looks like this:
@ -554,27 +573,32 @@ k8s: node-01 node-02 node-03 ## Deploy k8s cluster
.PHONY: node-01
node-01: ## Deploy node-01
nix shell "nixpkgs#nixos-rebuild" --command nixos-rebuild switch --build-host node-01 --target-host node-01 --flake ".#node-01"
nix run "nixpkgs#nixos-rebuild" -- switch --build-host node-01 --target-host node-01 --flake ".#node-01"
ssh node-01 'nix run nixpkgs#nvd -- --color=always diff $$(ls -d1v /nix/var/nix/profiles/system-*-link|tail -n 2)'
.PHONY: node-02
node-02: ## Deploy node-02
nix shell "nixpkgs#nixos-rebuild" --command nixos-rebuild switch --build-host node-02 --target-host node-02 --flake ".#node-02"
nix run "nixpkgs#nixos-rebuild" -- switch --build-host node-02 --target-host node-02 --flake ".#node-02"
ssh node-02 'nix run nixpkgs#nvd -- --color=always diff $$(ls -d1v /nix/var/nix/profiles/system-*-link|tail -n 2)'
.PHONY: node-03
node-03: ## Deploy node-03
nix shell "nixpkgs#nixos-rebuild" --command nixos-rebuild switch --build-host node-03 --target-host node-03 --flake ".#node-03"
nix run "nixpkgs#nixos-rebuild" -- switch --build-host node-03 --target-host node-03 --flake ".#node-03"
ssh node-03 'nix run nixpkgs#nvd -- --color=always diff $$(ls -d1v /nix/var/nix/profiles/system-*-link|tail -n 2)'
.PHONY: help
help: ## Display this help
@grep -h -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
```
This should work on NixOS aswell as any other OS with Nix installed. On non NisOS systems you won't have the
`nixos-rebuild` command, but you can run it from a `nix shell` command.
This should work on NixOS as well as any other OS with Nix installed. On non NisOS systems you won't have the
`nixos-rebuild` command, but you can run it from a `nix run` command. You might have to enable the experimental features
by adding `experimental-features = nix-command flakes` to your `nix.conf`.
The config will be build on the remote host, that has the advantage that the final config doesn't have to be transfered
to the remote host. On the other hand, does it also have the disadvantage that you have to build the config 3 times. Try
to remove the `--build-host` argument and test if that works better for you.
to the remote host. On the other hand, does it also have the disadvantage that you have to build the config 3 times,
because local builds from the Nix store can't be reused. Try to remove the `--build-host` argument and test if that
works better for you.
## SSH config
@ -603,13 +627,63 @@ Host node-03
IdentityFile /home/user/.ssh/id_ed25519
```
# Update nodes from local flake
_tba_
Now with everything prepared let's update our VMs by running `make all`.
This should update our flake and deploy all 3 nodes, one at a time.
As a proof of success login to one of them and run the fastfetch command so you can show it to all your reddit friends.
:P
```shell
[root@node-01:~]# fastfetch
▗▄▄▄ ▗▄▄▄▄ ▄▄▄▖ root@node-01
▜███▙ ▜███▙ ▟███▛ ------------
▜███▙ ▜███▙▟███▛ OS: NixOS 24.11 (Vicuna) x86_64
▜███▙ ▜██████▛ Host: KVM/QEMU Standard PC (Q35 + ICH9, 2009) (pc-q35-9.1)
▟█████████████████▙ ▜████▛ ▟▙ Kernel: Linux 6.13.6
▟███████████████████▙ ▜███▙ ▟██▙ Uptime: 6 days, 1 hour, 24 mins
▄▄▄▄▖ ▜███▙ ▟███▛ Packages: 379 (nix-system)
▟███▛ ▜██▛ ▟███▛ Shell: bash 5.2.37
▟███▛ ▜▛ ▟███▛ Display (Virtual-1): 1024x768
▟███████████▛ ▟██████████▙ Terminal: /dev/pts/0
▜██████████▛ ▟███████████▛ CPU: Intel(R) Xeon(R) E5-2699C v4 (16) @ 2.20 GHz
▟███▛ ▟▙ ▟███▛ GPU: Red Hat, Inc. QXL paravirtual graphic card
▟███▛ ▟██▙ ▟███▛ Memory: 1.11 GiB / 62.78 GiB (2%)
▟███▛ ▜███▙ ▝▀▀▀▀ Swap: 0 B / 4.00 GiB (0%)
▜██▛ ▜███▙ ▜██████████████████▛ Disk (/): 17.79 GiB / 57.77 GiB (31%) - ext4
▜▛ ▟████▙ ▜████████████████▛ Local IP (enp1s0): 192.0.2.1/24
▟██████▙ ▜███▙ Locale: en_US.UTF-8
▟███▛▜███▙ ▜███▙
▟███▛ ▜███▙ ▜███▙
▝▀▀▀ ▀▀▀▀▘ ▀▀▀▘
```
As a last step, check out the repo with our flake on each node. I prefer to put in under `/root/nix-config`. Then delete
the `/etc/nixos` directory and replace it with a symlink to our git repo like this:
```shell
ln -sf /root/nix-config /etc/nixos
```
With this setup you're able to also log into a VM enter the repo dir and roll out locally by running `nixos-rebuild
switch`.
This should be enough for this chapter. Lean back and enjoy your new VMs. Next time we will setup our PKI.
[cloudflare]: https://one.one.one.one
[quad9]: https://quad9.net
[qemu]: https://www.qemu.org
*[SSH]: Secure Shell
*[DNSSEC]: Domain Name System Security Extensions
*[DoT]: DNS over TLS
*[IPAM] IP Address Management
*[PKI]: Public Key Infrastructure
*[SBCs]: Single Board Computers
*[ESP]: EFI System Partition
*[EFI]: Extensible Firmware Interface
*[UEFI]: Unified Extensible Firmware Interface
*[BIOS]: Basic Input/Output System

200
content/software/k8s-on-nixos-part2.md
View file

@ -11,7 +11,10 @@ Status: Draft
# Things not covered in this guide
## Certificate Revocation lists (CRL)
In a production setup you would probably build a CRL so that you can put your old or compromised certificates on it. That way all users of your PKI know which certificates are still valid and should be trusted. It is a good improvement for security. For the etcd cluster this could be configured like this:
In a production setup you would probably build a CRL so that you can put your old or compromised certificates on it.
That way all users of your PKI know which certificates are still valid and should be trusted. It is a good improvement
for security. For the etcd cluster this could be configured like this:
```nix
services.etcd.extraConf = {
@ -20,38 +23,209 @@ services.etcd.extraConf = {
};
```
## Intermediate CAs
In a production scenario you would probably create intermediate CAs. It can be a good idea to do that, so you can keep
your root CA safe and secure in an offline location and use your intermediate for operational tasks. Maybe you have a
bigger PKI infrastructure with more use cases. In the case of a etcd and k8s setup it makes little sense to do that. The
security gain is minimal at best.
So if you want, you can create intermediate CAs, but I will skip this step. If you do you have to adapt the
`ca_constraints` to have a `max_path_len=1` for the root CAs and `max_path_len=0` for the intermediate CAs. Make sure
to also set `pathlenzero=true` for the intermediate CAs.
# Basic setup
```mermaid
flowchart TD
root(Etcd Root CA)
intermediate(Etcd Intermediate CA)
peer(Etcd Peer Certs)
client(Etcd Client Certs)
root --> intermediate
intermediate --> peer
intermediate --> client
root --> peer
root --> client
```
```mermaid
flowchart TD
root(Kubernetes Root CA)
intermediate(Kubernetes Intermediate CA)
apiserver(API Server Cert)
controller-manager(Controller Manager Cert)
kubelet-server(Kubelet Server Cert)
kubelet-client(Kubelet Client Cert)
proxy(Proxy Cert)
scheduler(Scheduler Cert)
root --> intermediate
intermediate --> apiserver
intermediate --> controller-manager
intermediate --> kubelet-server
intermediate --> kubelet-client
intermediate --> proxy
intermediate --> scheduler
root --> apiserver
root --> controller-manager
root --> kubelet-server
root --> kubelet-client
root --> proxy
root --> scheduler
```
We will use [cfssl](https://cfssl.org) to create our Certificate Authority. You could do the same with just `openssl` if
you prefer that. `cfssl` comes with baked in defaults that make it easier to get the security right. So if you're no
cryptography expert, I would suggest using it. In the `./pki` directory of our repository we'll create a
`ca-config.json` file like this:
```json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"root": {
"expiry": "87600h",
"usages": [
"signing",
"cert sign",
"crl sign",
"digital signature"
],
"ca_constraint": {
"is_ca": true,
"max_path_len": 0,
"max_path_len_zero": true
}
},
"client-server": {
"expiry": "8760h",
"usages": [
"digital signature",
"key encipherment",
"server auth",
"client auth"
]
},
"client": {
"expiry": "8760h",
"usages": [
"digital signature",
"key encipherment",
"client auth"
]
},
"server": {
"expiry": "8760h",
"usages": [
"digital signature",
"key encipherment",
"server auth"
]
}
}
}
}
```
It configures profiles for all types of certificates we're gonna need.
- Root certificate
- Client
- Server
- Client & Server
We will use eliptic curve for all of our certificates. I don't know if they're better then RSA. I just think they are
and I like that the file size is smaller. Ask a cryptography expert what you should use if you want a proper
recommendation.
## Create root Certificate Authorities (CA)
We will create two CAs. One for the etcd cluster and another one for the Kubernetes cluster. That way we avoid that some
component of our Kubernetes cluster could try acting like an etcd node. The etcd cluster will only trust peers that are
signed by the etcd CA.
Create a file `./pki/etcd-root/etcd-root-csr.json` for the root CA of the etcd cluster:
```json
{
"CN": "etcd-root-ca",
"key": {
"algo": "ecdsa",
"size": 521
},
"names": [
{
"C": "DE",
"L": "Berlin",
"O": "my-cluster",
"OU": "etcd",
"ST": "Berlin"
}
],
"ca": {
"expiry": "87600h",
"pathlen": 0,
"max_path_len_zero": true
}
}
```
Some of these settings are redundant. I'm not sure why it's needed but it only works this way. For our Kubernetes CA we
will create the file `./pki/k8s-root/k8s-root-csr.json` with this content:
```json
{
"CN": "k8s-root-ca",
"key": {
"algo": "ecdsa",
"size": 521
},
"names": [
{
"C": "DE",
"L": "Berlin",
"O": "wired",
"OU": "k8s",
"ST": "Berlin"
}
],
"ca": {
"expiry": "87600h",
"pathlen": 0,
"max_path_len_zero": true
}
}
```
## Create etcd certs
### etcd peers
Peer to peer communication need a certificate which has the key usages client and server auth.
`./pki/etcd-peer/etcd-peer-csr.json`:
```json
{
"CN": "etcd-peer",
"key": {
"algo": "ecdsa",
"size": 521
},
"names": [
{
"C": "DE",
"L": "Berlin",
"O": "wired",
"OU": "etcd",
"ST": "Berlin"
}
],
"hosts": [
"localhost",
"::1",
"127.0.0.1",
"node-01",
"2a00:1328:e101:1301::1"
]
}
```
### etcd clients
- etcd-peer
- etcd-client?

65
content/software/k8s-on-nixos-part3.md Normal file
View file

@ -0,0 +1,65 @@
---
Title: K8s on NixOS - Chapter 3: Etcd
Date: 2025-03-14
Category: software
Tags: etcd
Slug: k8s-on-nixos-chapter3-etcd
Summary: In this chapter we will setup the etcd cluster that Kubernetes needs to store it's state.
Status: Draft
---
# Building the etcd cluster
Make sure to disable IPv6 temporary addresses in your setup. We already did this by setting `networking.tempAddresses =
"disabled";` in out `common.nix` file. If not set, etcd will use temporary addresses to connect to it's peers and
connections will be rejected, because only the manually set IP addresses are configured to be trusted peers.
- https://etcd.io/docs/v3.5/op-guide/security/
- https://etcd.io/docs/v3.5/op-guide/clustering/
```nix
{ config, ip6Address, clusterNodes, ... }:
{
services.etcd = {
# opened manually for the 2a00:1328:e101:1301::/64 network
openFirewall = false;
# Name of the cluster; must be unique to identify this cluster
initialClusterToken = "k8s-etcd-cluster";
initialAdvertisePeerUrls = [ "https://[${ip6Address}]:2380" ];
listenPeerUrls = [ "https://[${ip6Address}]:2380" ];
listenClientUrls = [ "https://[${ip6Address}]:2379" "https://[::1]:2379" ];
advertiseClientUrls = [ "https://[${ip6Address}]:2379" ];
initialCluster = map (node: "${node.hostName}=https://[${node.ip6Address}]:2380") clusterNodes;
clientCertAuth = true;
# Certificate authority file to use for clients
trustedCaFile = config.age.secrets.k8s-root-crt.path;
# Cert file to use for clients
certFile = config.age.secrets.k8s-etcd-peer-crt.path;
# Key file to use for clients
keyFile = config.age.secrets.k8s-etcd-peer-key.path;
peerClientCertAuth = true;
# Certificate authority file to use for peer to peer communication
peerTrustedCaFile = config.age.secrets.k8s-root-crt.path;
# Cert file to use for peer to peer communication
peerCertFile = config.age.secrets.k8s-etcd-peer-crt.path;
# Key file to use for peer to peer communication
peerKeyFile = config.age.secrets.k8s-etcd-peer-key.path;
extraConf = {
"ETCD_AUTO_COMPACTION_RETENTION" = "1h"; # clean up old revisions after 1h
"QUOTA_BACKEND_BYTES" = "8589934592"; # maximum size of the etcd database
};
};
networking.firewall.extraInputRules = ''
# 2379/tcp etcd client requests
# 2380/tcp etcd peer communication
ip6 saddr 2a00:1328:e101:1301::/64 tcp dport { 2379, 2380 } accept comment "etcd"
'';
}
```
*[PKI]: Public Key Infrastructure
*[CA]: Certificate Authority

13
pelicanconf.py
View file

@ -116,3 +116,16 @@ PLUGINS = []
## Pelican-search Configuration ?
STORK_INPUT_OPTIONS = {"stemming": "English", "url_prefix": SITEURL}
# XenGi theme
SITEDESCRIPTION = "XenGis blog"
SOCIAL = (
# fontawesome icon class, url
("fa-brands fa-gitlab", "https://gitlab.com/XenGi"),
("fa-brands fa-github", "https://github.com/XenGi"),
("fa-brands fa-mastodon", "https://chaos.social/@xengi"),
("fa-solid fa-m", "https://matrix.to/#/@xengi:xengi.de"),
)