indexer/DOCS.md

# ffd-indexer — documentation and reference

This file holds the reference material for the **Federated Forge Discovery
Indexer**: how scraping works, the exact request volume, the full configuration
reference, the Prometheus metric reference and PromQL queries, the supported
forge types and type detection, the NixOS module, and development setup.

See [README.md](README.md) for the project's purpose and quick start.

---

## Table of contents

- [Architecture](#architecture)
- [Scrape behavior](#scrape-behavior)
  - [Robots.txt](#robotstxt)
  - [Listing repositories](#listing-repositories)
  - [Enrichment](#enrichment)
  - [Request volume](#request-volume)
  - [Error handling](#error-handling)
- [HTTP transport](#http-transport)
  - [HTTP/2](#http2)
  - [HTTP/3](#http3)
  - [Forcing a protocol](#forcing-a-protocol)
- [Configuration reference](#configuration-reference)
- [Prometheus metrics](#prometheus-metrics)
  - [Metric reference](#metric-reference)
  - [PromQL queries](#promql-queries)
  - [Metrics daemon](#metrics-daemon)
- [Supported forge types](#supported-forge-types)
  - [Type detection](#type-detection)
  - [Detection detail](#detection-detail)
- [NixOS module](#nixos-module)
- [Development](#development)

---

## Architecture

Two binaries, a SQLite store, and an external search UI:

```mermaid
flowchart TD
    subgraph Forges["Configured forges"]
        F1["forge A"]
        F2["forge B"]
    end

    subgraph Indexer["ffd-indexer · one-shot scrape"]
        IV["robots.txt check \n repo listing \n per-repo enrichment"]
    end

    subgraph Metrics["ffd-metrics · long-running daemon"]
        MV["render store as Prometheus metrics"]
    end

    DB[("SQLite \n projects · forge_status")]
    TS["Typesense \n search index"]
    PROM["Prometheus"]
    UI["Search UI"]

    F1 --> IV
    F2 --> IV
    IV -->|writes| DB
    IV -->|syncs index| TS
    DB -->|reads on each scrape| MV
    MV -->|/metrics| PROM
    UI -->|queries| TS
```

*(GitHub renders Mermaid diagrams natively in Markdown; no image to maintain.)*

- **`ffd-indexer`** — the scraper. Runs once per schedule, honors robots.txt,
  scrapes every configured forge, records per-forge status and upserts each
  project into SQLite, then syncs the index to Typesense.
- **`ffd-metrics`** — a long-running daemon. Reads the SQLite store on every
  Prometheus scrape and renders it at `/metrics`. It **never** touches the
  forges itself.
- **SQLite** (`db_path`) — the source of truth. Holds the `projects` table
  (with per-project forge identity and languages) and the `forge_status` table
  (per-forge online flag, detected type, software version, last error).
- **Typesense** (`typesense`) — optional full-text search index over the
  stored projects, searched by the UI.
- **Search UI** — a frontend that queries the Typesense index (not shipped in
  this repository); the indexer's sync keeps the collection searchable by it.

### Binaries

| Output | Package path | What it is |
|---|---|---|
| `ffd-indexer` | `git.xengi.de/ffd/indexer` | one-shot scraper + sync |
| `ffd-metrics` | `git.xengi.de/ffd/indexer/cmd/ffd-metrics` | long-running metrics daemon |

Both are built from the flake (`nix build .#default .#ffd-metrics`); the
default package installs both binaries.

---

## Scrape behavior

The indexer connects to each forge over HTTPS, trying to negotiate HTTP/2 via
ALPN for lower latency and multiplexing, with HTTP/1.1 as the fallback. A
single shared transport is used for robots checks, type detection and the
scrape itself, so keep-alive connections are reused across a host's requests.
(Full transport detail: [HTTP transport](#http-transport).)

### Robots.txt

Before any API call, the indexer fetches `GET /robots.txt` and consults the
rules for the `User-Agent: *` group. If the instance disallows `/api/` (the
API root), the instance is **skipped** — robots rules are honored for both
scraping and type detection. A missing robots.txt (404) is treated as
"allowed".

### Listing repositories

Public source repositories are listed via the repository search API
(`GET /api/v1/repos/search`), paginated at 50 per page. Forks and mirrors are
excluded.

### Enrichment

For each repository, the indexer:

- reads the repository root directory **once** (`GET /api/v1/repos/{o}/{r}/contents/`);
  this single listing is shared by README selection and license detection;
- fetches topics, languages, latest commit, and latest release;
- fetches the README (raw); and
- detects SPDX licenses from the license files present in the repo.

### Request volume

**Per instance:**

| Request | Count |
|---|---|
| `GET /robots.txt` | 1 |
| `GET /api/v1/repos/search` (page) | `ceil(N/50)` where N = total repos |

**Per repository:**

| Request | Endpoint | Purpose |
|---|---|---|
| list root contents | `GET /api/v1/repos/{o}/{r}/contents/` | shared: README selection + license scan (1 call) |
| languages | `GET /api/v1/repos/{o}/{r}/languages` | `Project.Languages` |
| topics | `GET /api/v1/repos/{o}/{r}/topics` | `Project.Topics` |
| latest commit | `GET /api/v1/repos/{o}/{r}/commits?limit=1` | `Project.LatestCommit` |
| latest release | `GET /api/v1/repos/{o}/{r}/releases/latest` | `Project.LatestRelease` |
| README | `GET /api/v1/repos/{o}/{r}/raw/{branch}/{file}` | `Project.Readme` |
| license files | 1+ × `GET /api/v1/repos/{o}/{r}/raw/{branch}/{file}` | SPDX detection |

So roughly **6–8 API requests per repository**, plus the paginated search
requests and a single robots.txt check per instance.

### Error handling

Errors are non-fatal: an instance or a single repo that fails is skipped and
the run continues. A failed forge is still recorded in `forge_status` with
`online=0` and its last error, so the metrics daemon always has a row per
configured forge. A failure to sync Typesense at the end of a run is fatal
(loud failure), so the scheduler can retry the run.

---

## HTTP transport

The shared transport always tries to use the **newest HTTP version available**
and automatically falls back when it isn't available:

- keep-alive connections are reused across a host's requests (pagination and
  per-repo enrichment don't re-handshake);
- HTTP/1.1 vs HTTP/2 is negotiated automatically by Go via ALPN
  (`ForceAttemptHTTP2` is set because the TLS config is custom);
- HTTP/3 is tried first (per host) and falls back to HTTP/2 or HTTP/1.1 when
  the host doesn't have it.

### HTTP/2

HTTP/2 is negotiated over TLS via ALPN whenever the server supports it
(verified against live forges), falling back to HTTP/1.1 otherwise. This gives
multiplexing and lower latency without any per-host configuration.

### HTTP/3

HTTP/3 is supported via `github.com/quic-go/quic-go/http3` and is tried
**first** on each host automatically — no manual opt-in is needed:

- The first request to a host attempts HTTP/3 (newest available); if the host
  answers over h3 it is used from then on.
- If the h3 attempt fails (e.g. the host has no UDP/QUIC), the request
  transparently falls back to HTTP/2 or HTTP/1.1, so an h3 failure is never
  visible to the caller.
- A **per-host negative cache** (TTL ~1h) remembers hosts that failed h3, so a
  host without h3 isn't re-triggered with a QUIC handshake on every request.
  Once the TTL expires, h3 is tried again in case the host started serving it.
- An `Alt-Svc: h3=` response header records a positive capability (confirms
  h3 rather than gating it — the default is already to try).
- `scraper.EnableHTTP3(host)` is only an override that keeps trying h3 even
  during a negative cool-down; normal operation never needs it.

Note: Go 1.27's stdlib has no HTTP/3 client and `x/net/http3` is only a
registration shim; this is why the quic-go transport provides it.

### Forcing a protocol

If you set an explicit `url: http://...` in the config, the indexer connects
with plain HTTP (HTTP/1.1) instead of HTTPS. The stored identity and metrics
labels always use the bare hostname regardless of the connection scheme.

---

## Configuration reference

Both `ffd-indexer` and `ffd-metrics` read the same YAML file
(`-config <path>`, default `config.yaml`).

```yaml
db_path: "./index.db"
typesense:
  host: "http://localhost:8108"
  api_key: "dev-key"      # inline key, or
  # api_key_file: /run/secrets/typesense   # a secret file (takes precedence)
metrics:
  listen: "127.0.0.1:9090"
forges:
  - url: git.example.org
    type: forgejo
  - url: git.example.net
    type: auto          # default when omitted
```

| Key | Type | Description |
|---|---|---|
| `db_path` | string | Path to the SQLite database. |
| `typesense.host` | string | Typesense server address. |
| `typesense.api_key` | string | Typesense API key (inline). |
| `typesense.api_key_file` | string | Path to a file containing the API key (e.g. sops/agenix secret). Takes precedence over `api_key`; contents are trimmed of surrounding whitespace. |
| `metrics.listen` | string (optional) | Address `ffd-metrics` binds for `/metrics`. The whole `metrics:` block may be omitted when metrics are not used. |
| `forges[].url` | string | Forge bare hostname (FQDN); a leading scheme is accepted to force the connection protocol. Identity is always the bare hostname. |
| `forges[].type` | string | `forgejo` \| `gitea` (pinned, recommended) or `auto` (default, experimental). |

A `type` other than `forgejo`, `gitea` or `auto` is rejected at startup. See
[Supported forge types](#supported-forge-types) for detection behavior.

### Typesense API key

The API key can be given inline (`typesense.api_key`) or read from a file
(`typesense.api_key_file`) — the file takes precedence when both are set. The
file is read at startup and trimmed of surrounding whitespace/newlines, so a
secret deployed via **sops** or **agenix** (e.g. at `/run/secrets/...`) can be
referenced directly without the key ever appearing in the config or the Nix
store.

### Container

- `nix build .#container` — an image running `ffd-indexer` once; expects a
  config mounted at `/etc/ffd-indexer/config.yaml` (see README quick start).
  `db_path` is resolved relative to the container working directory, so mount a
  volume. The image bundles CA certificates so HTTPS to forges works.
- `nix build .#container-metrics` — an image running the `ffd-metrics` daemon,
  exposing `:9090`.
- `nix run .#default -- -config config.yaml` — run the indexer from the flake.
- `nix run .#ffd-metrics -- -config /etc/ffd-indexer/config.yaml` — run the
  metrics daemon from the flake.

---

## Prometheus metrics

A separate long-running binary, `ffd-metrics`, serves the state of the index
as Prometheus metrics at `/metrics`. It never scrapes the forges itself: the
indexer's scheduled runs record per-forge status into the SQLite database, and
`ffd-metrics` renders that state on each scrape.

Both binaries share the same config file. Configure the listen address under
`metrics.listen` (e.g. `127.0.0.1:9090`, default).

### Metric reference

| Metric | Type | Labels | Description |
|---|---|---|---|
| `ffd_forge_info` | gauge | `forge_url`, `forge_type`, `forge_version` | Constant 1 per forge with its detected type and software version |
| `ffd_forge_online` | gauge | `forge_url`, `forge_type` | 1 if the forge was reachable and scraped successfully on its last run, else 0 (so offline forges still appear) |
| `ffd_repository_info` | gauge | `repo_url`, `forge_url`, `forge_type` | Constant 1 per stored repository |
| `ffd_repository_languages` | gauge | `repo_url`, `language`, `forge_url`, `forge_type` | Constant 1 per (repository, language) pair |

### PromQL queries

Everything beyond the primitive metrics is derived with PromQL:

- repositories per forge: `count by (forge_url) (ffd_repository_info)`
- repositories overall: `count(ffd_repository_info)`
- forges per type: `count by (forge_type) (ffd_forge_online)`
- language usage: `sum by (language) (ffd_repository_languages)`
- forge versions: `ffd_forge_info` (constant gauge, read `forge_version` label)
- offline forges: `ffd_forge_online == 0`

### Metrics daemon

Run it directly:

```sh
ffd-metrics -config config.yaml
# or from the flake
nix run .#ffd-metrics -- -config /etc/ffd-indexer/config.yaml
```

The flake also ships `container-metrics` (`nix build .#container-metrics`), an
image that runs the daemon on `:9090`, and the NixOS module enables a
`ffd-metrics` systemd service alongside the one-shot indexer timer.

---

## Supported forge types

The forge software is either pinned per forge in the config (`type: forgejo` /
`type: gitea`) or detected on every run (`type: auto`).

### Type detection

We recommend pinning each forge's `type` by hand; this selects the scraper
deterministically and requires no probing. The default is `auto`, which
re-detects the type on every run because the hoster may have changed the
backend (e.g. Gitea→Forgejo). `auto` is **experimental**:

- robots.txt is honored — a forge that disallows `/api/` will not be probed,
  so `auto` reports `unknown` for it;
- detection may misclassify until more detectors land;
- the previously detected type naturally comes first because detection starts
  with the most specific signal.

Forge types without a detector are reported as `unknown`
(`forge_type="unknown"`) and skipped; the metrics daemon still shows their
status row.

### Detection detail

| Type | Detection |
|---|---|
| Forgejo | Serves the Forgejo-native `GET /api/forgejo/v1/version`, which returns 200 with a version JSON. Forgejo is a Gitea fork and also serves the shared `/api/v1/version`, so this endpoint is the discriminator: a hit is a strong Forgejo signal. The version is recorded and exposed as `ffd_forge_info{forge_version=...}`. |
| Gitea | Does not serve `/api/forgejo/v1/version` (404) but does serve the shared `GET /api/v1/version`, so an instance is classified Gitea if and only if the Forgejo probe fails and the shared endpoint answers. If other forges later serve `/api/v1/version` (e.g. Gogs), that classification will need a further discriminator. |

Forgejo and Gitea share the same `/api/v1` surface (Forgejo is a fork of
Gitea), so both use the same scraper class.

---

## NixOS module

The flake exposes `nixosModules.default` (option namespace
`services.federated-forge-discovery.indexer`). It enables:

- a **one-shot `ffd-indexer` systemd service** and a **timer** driving it
  (`schedule`: `daily` [default], `hourly`, or `weekly` — or any systemd
  `OnCalendar` expression);
- a **`ffd-metrics` systemd daemon** (`Type=simple`, restart on failure);
- the rendered config at `/etc/ffd-indexer/config.yaml`, owned by the service
  user `ffd-indexer`;
- a `container`/`container-metrics` integration.

Options:

| Option | Default | Description |
|---|---|---|
| `enable` | — | Enables the service and timer. |
| `package` | — | The package to run (`ffd-indexer` from the flake). |
| `schedule` | `"daily"` | `daily` \| `hourly` \| `weekly`, or an `OnCalendar` string. |
| `dbPath` | — | Path to the SQLite DB. |
| `typesense.host` | — | Typesense address. |
| `typesense.apiKey` | `""` | Typesense API key (inline). |
| `typesense.apiKeyFile` | `null` | Path to a secret file (sops/agenix) containing the key; takes precedence over `apiKey`. |
| `metrics.enable` | `false` | Enable the optional `ffd-metrics` daemon (and render the `metrics:` block). Disabled by default; set `true` to run metrics. |
| `metrics.listen` | `"127.0.0.1:9090"` | Address for the metrics daemon. |
| `forges` | `[]` | List of `{ url, type }`; `type` is `auto` (default), `forgejo` or `gitea`. |

---

## Development

The dev shell (`nix develop`) provides Go, `gomod2nix`, `staticcheck`, and a
local Typesense server for development.

```sh
nix develop          # enter the dev shell (Go toolchain, gomod2nix, etc.)
go test ./...        # run tests (store, scraper, config)
go vet ./...
nix flake check      # lint (golangci-lint) + all tests inside Nix
nix build .#default .#ffd-metrics .#container .#container-metrics
```

### Continuous integration

[`.forgejo/workflows/ci.yml`](.forgejo/workflows/ci.yml) runs the pipeline on
Forgejo Actions with **no upstream actions**: the checkout is a plain
`git clone`, and every step calls the flake (`nix flake check`, `nix build`,
`nix develop`) inside the `nixos/nix` container image. Four isolated jobs:

| Job | Enforces | Fails on |
|---|---|---|
| **checks** | `nix flake check` | unit tests (`go test ./...`) or golangci-lint |
| **build** | `nix build .#default .#ffd-metrics .#container .#container-metrics` | any package failing to build |
| **format** | `gofmt -l` and `nix fmt` | unformatted `.go` / `.nix` files |
| **integration** | live scrape of `git.xengi.de` sent to a **Typesense `services:` container** | a scrape or sync assertion failing |

The **integration** job is a real end-to-end test run inside the workflow: a
Typesense container (the `services:` feature) is started alongside the job,
the indexer scrapes the repository's own forge (`git.xengi.de`, `type: auto`
→ real detection) and syncs into that service. It then asserts the forge is
online in sqlite and that the Typesense document count equals the sqlite
project count. The Typesense service image (`typesense/typesense:30.2`)
matches the flake's `typesense-server` version, so CI mirrors local dev
(where you would use `nix run .#typesense-server`). It depends on the forge
being up — it is a live test by design.

Run all checks locally with `nix flake check` + `gofmt -l .` + `nix fmt`
(followed by a `git diff --quiet`). The runner needs the `debian-stable` label and
outbound access to the forge, github.com (flake inputs) and cache.nixos.org.

### Dependencies

Go modules are vendored into Nix via `gomod2nix.toml` (generated with
`gomod2nix generate`). After adding or upgrading a Go dependency, regenerate
`gomod2nix.toml` before `nix flake check`/`nix build` will reflect it.