🔥

DOCS.md

@@ -0,0 +1,423 @@
+# ffd-indexer — documentation and reference
+
+This file holds the reference material for the **Federated Forge Discovery
+Indexer**: how scraping works, the exact request volume, the full configuration
+reference, the Prometheus metric reference and PromQL queries, the supported
+forge types and type detection, the NixOS module, and development setup.
+
+See [README.md](README.md) for the project's purpose and quick start.
+
+---
+
+## Table of contents
+
+- [Architecture](#architecture)
+- [Scrape behavior](#scrape-behavior)
+  - [Robots.txt](#robotstxt)
+  - [Listing repositories](#listing-repositories)
+  - [Enrichment](#enrichment)
+  - [Request volume](#request-volume)
+  - [Error handling](#error-handling)
+- [HTTP transport](#http-transport)
+  - [HTTP/2](#http2)
+  - [HTTP/3](#http3)
+  - [Forcing a protocol](#forcing-a-protocol)
+- [Configuration reference](#configuration-reference)
+- [Prometheus metrics](#prometheus-metrics)
+  - [Metric reference](#metric-reference)
+  - [PromQL queries](#promql-queries)
+  - [Metrics daemon](#metrics-daemon)
+- [Supported forge types](#supported-forge-types)
+  - [Type detection](#type-detection)
+  - [Detection detail](#detection-detail)
+- [NixOS module](#nixos-module)
+- [Development](#development)
+
+---
+
+## Architecture
+
+Two binaries, a SQLite store, and an external search UI:
+
+```mermaid
+flowchart TD
+    subgraph Forges["Configured forges"]
+        F1["forge A"]
+        F2["forge B"]
+    end
+
+    subgraph Indexer["ffd-indexer · one-shot scrape"]
+        IV["robots.txt check \n repo listing \n per-repo enrichment"]
+    end
+
+    subgraph Metrics["ffd-metrics · long-running daemon"]
+        MV["render store as Prometheus metrics"]
+    end
+
+    DB[("SQLite \n projects · forge_status")]
+    TS["Typesense \n search index"]
+    PROM["Prometheus"]
+    UI["Search UI"]
+
+    F1 --> IV
+    F2 --> IV
+    IV -->|writes| DB
+    IV -->|syncs index| TS
+    DB -->|reads on each scrape| MV
+    MV -->|/metrics| PROM
+    UI -->|queries| TS
+```
+
+*(GitHub renders Mermaid diagrams natively in Markdown; no image to maintain.)*
+
+- **`ffd-indexer`** — the scraper. Runs once per schedule, honors robots.txt,
+  scrapes every configured forge, records per-forge status and upserts each
+  project into SQLite, then syncs the index to Typesense.
+- **`ffd-metrics`** — a long-running daemon. Reads the SQLite store on every
+  Prometheus scrape and renders it at `/metrics`. It **never** touches the
+  forges itself.
+- **SQLite** (`db_path`) — the source of truth. Holds the `projects` table
+  (with per-project forge identity and languages) and the `forge_status` table
+  (per-forge online flag, detected type, software version, last error).
+- **Typesense** (`typesense`) — optional full-text search index over the
+  stored projects, searched by the UI.
+- **Search UI** — a frontend that queries the Typesense index (not shipped in
+  this repository); the indexer's sync keeps the collection searchable by it.
+
+### Binaries
+
+| Output | Package path | What it is |
+|---|---|---|
+| `ffd-indexer` | `git.xengi.de/ffd/indexer` | one-shot scraper + sync |
+| `ffd-metrics` | `git.xengi.de/ffd/indexer/cmd/ffd-metrics` | long-running metrics daemon |
+
+Both are built from the flake (`nix build .#default .#ffd-metrics`); the
+default package installs both binaries.
+
+---
+
+## Scrape behavior
+
+The indexer connects to each forge over HTTPS, trying to negotiate HTTP/2 via
+ALPN for lower latency and multiplexing, with HTTP/1.1 as the fallback. A
+single shared transport is used for robots checks, type detection and the
+scrape itself, so keep-alive connections are reused across a host's requests.
+(Full transport detail: [HTTP transport](#http-transport).)
+
+### Robots.txt
+
+Before any API call, the indexer fetches `GET /robots.txt` and consults the
+rules for the `User-Agent: *` group. If the instance disallows `/api/` (the
+API root), the instance is **skipped** — robots rules are honored for both
+scraping and type detection. A missing robots.txt (404) is treated as
+"allowed".
+
+### Listing repositories
+
+Public source repositories are listed via the repository search API
+(`GET /api/v1/repos/search`), paginated at 50 per page. Forks and mirrors are
+excluded.
+
+### Enrichment
+
+For each repository, the indexer:
+
+- reads the repository root directory **once** (`GET /api/v1/repos/{o}/{r}/contents/`);
+  this single listing is shared by README selection and license detection;
+- fetches topics, languages, latest commit, and latest release;
+- fetches the README (raw); and
+- detects SPDX licenses from the license files present in the repo.
+
+### Request volume
+
+**Per instance:**
+
+| Request | Count |
+|---|---|
+| `GET /robots.txt` | 1 |
+| `GET /api/v1/repos/search` (page) | `ceil(N/50)` where N = total repos |
+
+**Per repository:**
+
+| Request | Endpoint | Purpose |
+|---|---|---|
+| list root contents | `GET /api/v1/repos/{o}/{r}/contents/` | shared: README selection + license scan (1 call) |
+| languages | `GET /api/v1/repos/{o}/{r}/languages` | `Project.Languages` |
+| topics | `GET /api/v1/repos/{o}/{r}/topics` | `Project.Topics` |
+| latest commit | `GET /api/v1/repos/{o}/{r}/commits?limit=1` | `Project.LatestCommit` |
+| latest release | `GET /api/v1/repos/{o}/{r}/releases/latest` | `Project.LatestRelease` |
+| README | `GET /api/v1/repos/{o}/{r}/raw/{branch}/{file}` | `Project.Readme` |
+| license files | 1+ × `GET /api/v1/repos/{o}/{r}/raw/{branch}/{file}` | SPDX detection |
+
+So roughly **6–8 API requests per repository**, plus the paginated search
+requests and a single robots.txt check per instance.
+
+### Error handling
+
+Errors are non-fatal: an instance or a single repo that fails is skipped and
+the run continues. A failed forge is still recorded in `forge_status` with
+`online=0` and its last error, so the metrics daemon always has a row per
+configured forge. A failure to sync Typesense at the end of a run is fatal
+(loud failure), so the scheduler can retry the run.
+
+---
+
+## HTTP transport
+
+The shared transport always tries to use the **newest HTTP version available**
+and automatically falls back when it isn't available:
+
+- keep-alive connections are reused across a host's requests (pagination and
+  per-repo enrichment don't re-handshake);
+- HTTP/1.1 vs HTTP/2 is negotiated automatically by Go via ALPN
+  (`ForceAttemptHTTP2` is set because the TLS config is custom);
+- HTTP/3 is tried first (per host) and falls back to HTTP/2 or HTTP/1.1 when
+  the host doesn't have it.
+
+### HTTP/2
+
+HTTP/2 is negotiated over TLS via ALPN whenever the server supports it
+(verified against live forges), falling back to HTTP/1.1 otherwise. This gives
+multiplexing and lower latency without any per-host configuration.
+
+### HTTP/3
+
+HTTP/3 is supported via `github.com/quic-go/quic-go/http3` and is tried
+**first** on each host automatically — no manual opt-in is needed:
+
+- The first request to a host attempts HTTP/3 (newest available); if the host
+  answers over h3 it is used from then on.
+- If the h3 attempt fails (e.g. the host has no UDP/QUIC), the request
+  transparently falls back to HTTP/2 or HTTP/1.1, so an h3 failure is never
+  visible to the caller.
+- A **per-host negative cache** (TTL ~1h) remembers hosts that failed h3, so a
+  host without h3 isn't re-triggered with a QUIC handshake on every request.
+  Once the TTL expires, h3 is tried again in case the host started serving it.
+- An `Alt-Svc: h3=` response header records a positive capability (confirms
+  h3 rather than gating it — the default is already to try).
+- `scraper.EnableHTTP3(host)` is only an override that keeps trying h3 even
+  during a negative cool-down; normal operation never needs it.
+
+Note: Go 1.27's stdlib has no HTTP/3 client and `x/net/http3` is only a
+registration shim; this is why the quic-go transport provides it.
+
+### Forcing a protocol
+
+If you set an explicit `url: http://...` in the config, the indexer connects
+with plain HTTP (HTTP/1.1) instead of HTTPS. The stored identity and metrics
+labels always use the bare hostname regardless of the connection scheme.
+
+---
+
+## Configuration reference
+
+Both `ffd-indexer` and `ffd-metrics` read the same YAML file
+(`-config <path>`, default `config.yaml`).
+
+```yaml
+db_path: "./index.db"
+typesense:
+  host: "http://localhost:8108"
+  api_key: "dev-key"      # inline key, or
+  # api_key_file: /run/secrets/typesense   # a secret file (takes precedence)
+metrics:
+  listen: "127.0.0.1:9090"
+forges:
+  - url: git.example.org
+    type: forgejo
+  - url: git.example.net
+    type: auto          # default when omitted
+```
+
+| Key | Type | Description |
+|---|---|---|
+| `db_path` | string | Path to the SQLite database. |
+| `typesense.host` | string | Typesense server address. |
+| `typesense.api_key` | string | Typesense API key (inline). |
+| `typesense.api_key_file` | string | Path to a file containing the API key (e.g. sops/agenix secret). Takes precedence over `api_key`; contents are trimmed of surrounding whitespace. |
+| `metrics.listen` | string (optional) | Address `ffd-metrics` binds for `/metrics`. The whole `metrics:` block may be omitted when metrics are not used. |
+| `forges[].url` | string | Forge bare hostname (FQDN); a leading scheme is accepted to force the connection protocol. Identity is always the bare hostname. |
+| `forges[].type` | string | `forgejo` \| `gitea` (pinned, recommended) or `auto` (default, experimental). |
+
+A `type` other than `forgejo`, `gitea` or `auto` is rejected at startup. See
+[Supported forge types](#supported-forge-types) for detection behavior.
+
+### Typesense API key
+
+The API key can be given inline (`typesense.api_key`) or read from a file
+(`typesense.api_key_file`) — the file takes precedence when both are set. The
+file is read at startup and trimmed of surrounding whitespace/newlines, so a
+secret deployed via **sops** or **agenix** (e.g. at `/run/secrets/...`) can be
+referenced directly without the key ever appearing in the config or the Nix
+store.
+
+### Container
+
+- `nix build .#container` — an image running `ffd-indexer` once; expects a
+  config mounted at `/etc/ffd-indexer/config.yaml` (see README quick start).
+  `db_path` is resolved relative to the container working directory, so mount a
+  volume. The image bundles CA certificates so HTTPS to forges works.
+- `nix build .#container-metrics` — an image running the `ffd-metrics` daemon,
+  exposing `:9090`.
+- `nix run .#default -- -config config.yaml` — run the indexer from the flake.
+- `nix run .#ffd-metrics -- -config /etc/ffd-indexer/config.yaml` — run the
+  metrics daemon from the flake.
+
+---
+
+## Prometheus metrics
+
+A separate long-running binary, `ffd-metrics`, serves the state of the index
+as Prometheus metrics at `/metrics`. It never scrapes the forges itself: the
+indexer's scheduled runs record per-forge status into the SQLite database, and
+`ffd-metrics` renders that state on each scrape.
+
+Both binaries share the same config file. Configure the listen address under
+`metrics.listen` (e.g. `127.0.0.1:9090`, default).
+
+### Metric reference
+
+| Metric | Type | Labels | Description |
+|---|---|---|---|
+| `ffd_forge_info` | gauge | `forge_url`, `forge_type`, `forge_version` | Constant 1 per forge with its detected type and software version |
+| `ffd_forge_online` | gauge | `forge_url`, `forge_type` | 1 if the forge was reachable and scraped successfully on its last run, else 0 (so offline forges still appear) |
+| `ffd_repository_info` | gauge | `repo_url`, `forge_url`, `forge_type` | Constant 1 per stored repository |
+| `ffd_repository_languages` | gauge | `repo_url`, `language`, `forge_url`, `forge_type` | Constant 1 per (repository, language) pair |
+
+### PromQL queries
+
+Everything beyond the primitive metrics is derived with PromQL:
+
+- repositories per forge: `count by (forge_url) (ffd_repository_info)`
+- repositories overall: `count(ffd_repository_info)`
+- forges per type: `count by (forge_type) (ffd_forge_online)`
+- language usage: `sum by (language) (ffd_repository_languages)`
+- forge versions: `ffd_forge_info` (constant gauge, read `forge_version` label)
+- offline forges: `ffd_forge_online == 0`
+
+### Metrics daemon
+
+Run it directly:
+
+```sh
+ffd-metrics -config config.yaml
+# or from the flake
+nix run .#ffd-metrics -- -config /etc/ffd-indexer/config.yaml
+```
+
+The flake also ships `container-metrics` (`nix build .#container-metrics`), an
+image that runs the daemon on `:9090`, and the NixOS module enables a
+`ffd-metrics` systemd service alongside the one-shot indexer timer.
+
+---
+
+## Supported forge types
+
+The forge software is either pinned per forge in the config (`type: forgejo` /
+`type: gitea`) or detected on every run (`type: auto`).
+
+### Type detection
+
+We recommend pinning each forge's `type` by hand; this selects the scraper
+deterministically and requires no probing. The default is `auto`, which
+re-detects the type on every run because the hoster may have changed the
+backend (e.g. Gitea→Forgejo). `auto` is **experimental**:
+
+- robots.txt is honored — a forge that disallows `/api/` will not be probed,
+  so `auto` reports `unknown` for it;
+- detection may misclassify until more detectors land;
+- the previously detected type naturally comes first because detection starts
+  with the most specific signal.
+
+Forge types without a detector are reported as `unknown`
+(`forge_type="unknown"`) and skipped; the metrics daemon still shows their
+status row.
+
+### Detection detail
+
+| Type | Detection |
+|---|---|
+| Forgejo | Serves the Forgejo-native `GET /api/forgejo/v1/version`, which returns 200 with a version JSON. Forgejo is a Gitea fork and also serves the shared `/api/v1/version`, so this endpoint is the discriminator: a hit is a strong Forgejo signal. The version is recorded and exposed as `ffd_forge_info{forge_version=...}`. |
+| Gitea | Does not serve `/api/forgejo/v1/version` (404) but does serve the shared `GET /api/v1/version`, so an instance is classified Gitea if and only if the Forgejo probe fails and the shared endpoint answers. If other forges later serve `/api/v1/version` (e.g. Gogs), that classification will need a further discriminator. |
+
+Forgejo and Gitea share the same `/api/v1` surface (Forgejo is a fork of
+Gitea), so both use the same scraper class.
+
+---
+
+## NixOS module
+
+The flake exposes `nixosModules.default` (option namespace
+`services.federated-forge-discovery.indexer`). It enables:
+
+- a **one-shot `ffd-indexer` systemd service** and a **timer** driving it
+  (`schedule`: `daily` [default], `hourly`, or `weekly` — or any systemd
+  `OnCalendar` expression);
+- a **`ffd-metrics` systemd daemon** (`Type=simple`, restart on failure);
+- the rendered config at `/etc/ffd-indexer/config.yaml`, owned by the service
+  user `ffd-indexer`;
+- a `container`/`container-metrics` integration.
+
+Options:
+
+| Option | Default | Description |
+|---|---|---|
+| `enable` | — | Enables the service and timer. |
+| `package` | — | The package to run (`ffd-indexer` from the flake). |
+| `schedule` | `"daily"` | `daily` \| `hourly` \| `weekly`, or an `OnCalendar` string. |
+| `dbPath` | — | Path to the SQLite DB. |
+| `typesense.host` | — | Typesense address. |
+| `typesense.apiKey` | `""` | Typesense API key (inline). |
+| `typesense.apiKeyFile` | `null` | Path to a secret file (sops/agenix) containing the key; takes precedence over `apiKey`. |
+| `metrics.enable` | `false` | Enable the optional `ffd-metrics` daemon (and render the `metrics:` block). Disabled by default; set `true` to run metrics. |
+| `metrics.listen` | `"127.0.0.1:9090"` | Address for the metrics daemon. |
+| `forges` | `[]` | List of `{ url, type }`; `type` is `auto` (default), `forgejo` or `gitea`. |
+
+---
+
+## Development
+
+The dev shell (`nix develop`) provides Go, `gomod2nix`, `staticcheck`, and a
+local Typesense server for development.
+
+```sh
+nix develop          # enter the dev shell (Go toolchain, gomod2nix, etc.)
+go test ./...        # run tests (store, scraper, config)
+go vet ./...
+nix flake check      # lint (golangci-lint) + all tests inside Nix
+nix build .#default .#ffd-metrics .#container .#container-metrics
+```
+
+### Continuous integration
+
+[`.forgejo/workflows/ci.yml`](.forgejo/workflows/ci.yml) runs the pipeline on
+Forgejo Actions with **no upstream actions**: the checkout is a plain
+`git clone`, and every step calls the flake (`nix flake check`, `nix build`,
+`nix develop`) inside the `nixos/nix` container image. Four isolated jobs:
+
+| Job | Enforces | Fails on |
+|---|---|---|
+| **checks** | `nix flake check` | unit tests (`go test ./...`) or golangci-lint |
+| **build** | `nix build .#default .#ffd-metrics .#container .#container-metrics` | any package failing to build |
+| **format** | `gofmt -l` and `nix fmt` | unformatted `.go` / `.nix` files |
+| **integration** | live scrape of `git.xengi.de` sent to a **Typesense `services:` container** | a scrape or sync assertion failing |
+
+The **integration** job is a real end-to-end test run inside the workflow: a
+Typesense container (the `services:` feature) is started alongside the job,
+the indexer scrapes the repository's own forge (`git.xengi.de`, `type: auto`
+→ real detection) and syncs into that service. It then asserts the forge is
+online in sqlite and that the Typesense document count equals the sqlite
+project count. The Typesense service image (`typesense/typesense:30.2`)
+matches the flake's `typesense-server` version, so CI mirrors local dev
+(where you would use `nix run .#typesense-server`). It depends on the forge
+being up — it is a live test by design.
+
+Run all checks locally with `nix flake check` + `gofmt -l .` + `nix fmt`
+(followed by a `git diff --quiet`). The runner needs the `debian-stable` label and
+outbound access to the forge, github.com (flake inputs) and cache.nixos.org.
+
+### Dependencies
+
+Go modules are vendored into Nix via `gomod2nix.toml` (generated with
+`gomod2nix generate`). After adding or upgrading a Go dependency, regenerate
+`gomod2nix.toml` before `nix flake check`/`nix build` will reflect it.