indexer/DOCS.md

ffd-indexer — documentation and reference

This file holds the reference material for the Federated Forge Discovery Indexer: how scraping works, the exact request volume, the full configuration reference, the Prometheus metric reference and PromQL queries, the supported forge types and type detection, the NixOS module, and development setup.

See README.md for the project's purpose and quick start.

---

Table of contents

• Architecture
• Scrape behavior
  • Robots.txt
  • Listing repositories
  • Enrichment
  • Request volume
  • Error handling
• HTTP transport
  • HTTP/2
  • HTTP/3
  • Forcing a protocol
• Configuration reference
• Prometheus metrics
  • Metric reference
  • PromQL queries
  • Metrics daemon
• Supported forge types
  • Type detection
  • Detection detail
• NixOS module
• Development

---

Architecture

Two binaries, a SQLite store, and an external search UI:

flowchart TD
    subgraph Forges["Configured forges"]
        F1["forge A"]
        F2["forge B"]
    end

    subgraph Indexer["ffd-indexer · one-shot scrape"]
        IV["robots.txt check \n repo listing \n per-repo enrichment"]
    end

    subgraph Metrics["ffd-metrics · long-running daemon"]
        MV["render store as Prometheus metrics"]
    end

    DB[("SQLite \n projects · forge_status")]
    TS["Typesense \n search index"]
    PROM["Prometheus"]
    UI["Search UI"]

    F1 --> IV
    F2 --> IV
    IV -->|writes| DB
    IV -->|syncs index| TS
    DB -->|reads on each scrape| MV
    MV -->|/metrics| PROM
    UI -->|queries| TS

(GitHub renders Mermaid diagrams natively in Markdown; no image to maintain.)

• ffd-indexer — the scraper. Runs once per schedule, honors robots.txt, scrapes every configured forge, records per-forge status and upserts each project into SQLite, then syncs the index to Typesense.
• ffd-metrics — a long-running daemon. Reads the SQLite store on every Prometheus scrape and renders it at /metrics. It never touches the forges itself.
• SQLite (db_path) — the source of truth. Holds the projects table (with per-project forge identity and languages) and the forge_status table (per-forge online flag, detected type, software version, last error).
• Typesense (typesense) — optional full-text search index over the stored projects, searched by the UI.
• Search UI — a frontend that queries the Typesense index (not shipped in this repository); the indexer's sync keeps the collection searchable by it.

Binaries

Output	Package path	What it is
ffd-indexer	git.xengi.de/ffd/indexer	one-shot scraper + sync
ffd-metrics	git.xengi.de/ffd/indexer/cmd/ffd-metrics	long-running metrics daemon

Both are built from the flake (nix build .#default .#ffd-metrics); the default package installs both binaries.

---

Scrape behavior

The indexer connects to each forge over HTTPS, trying to negotiate HTTP/2 via ALPN for lower latency and multiplexing, with HTTP/1.1 as the fallback. A single shared transport is used for robots checks, type detection and the scrape itself, so keep-alive connections are reused across a host's requests. (Full transport detail: HTTP transport.)

Robots.txt

Before any API call, the indexer fetches GET /robots.txt and consults the rules for the User-Agent: * group. If the instance disallows /api/ (the API root), the instance is skipped — robots rules are honored for both scraping and type detection. A missing robots.txt (404) is treated as "allowed".

Listing repositories

Public source repositories are listed via the repository search API (GET /api/v1/repos/search), paginated at 50 per page. Forks and mirrors are excluded.

Enrichment

For each repository, the indexer:

• reads the repository root directory once (GET /api/v1/repos/{o}/{r}/contents/); this single listing is shared by README selection and license detection;
• fetches topics, languages, latest commit, and latest release;
• fetches the README (raw); and
• detects SPDX licenses from the license files present in the repo.

Request volume

Per instance:

Request	Count
GET /robots.txt	1
GET /api/v1/repos/search (page)	ceil(N/50) where N = total repos

Per repository:

Request	Endpoint	Purpose
list root contents	GET /api/v1/repos/{o}/{r}/contents/	shared: README selection + license scan (1 call)
languages	GET /api/v1/repos/{o}/{r}/languages	Project.Languages
topics	GET /api/v1/repos/{o}/{r}/topics	Project.Topics
latest commit	GET /api/v1/repos/{o}/{r}/commits?limit=1	Project.LatestCommit
latest release	GET /api/v1/repos/{o}/{r}/releases/latest	Project.LatestRelease
README	GET /api/v1/repos/{o}/{r}/raw/{branch}/{file}	Project.Readme
license files	1+ × GET /api/v1/repos/{o}/{r}/raw/{branch}/{file}	SPDX detection

So roughly 6–8 API requests per repository, plus the paginated search requests and a single robots.txt check per instance.

Error handling

Errors are non-fatal: an instance or a single repo that fails is skipped and the run continues. A failed forge is still recorded in forge_status with online=0 and its last error, so the metrics daemon always has a row per configured forge. A failure to sync Typesense at the end of a run is fatal (loud failure), so the scheduler can retry the run.

---

HTTP transport

The shared transport always tries to use the newest HTTP version available and automatically falls back when it isn't available:

• keep-alive connections are reused across a host's requests (pagination and per-repo enrichment don't re-handshake);
• HTTP/1.1 vs HTTP/2 is negotiated automatically by Go via ALPN (ForceAttemptHTTP2 is set because the TLS config is custom);
• HTTP/3 is tried first (per host) and falls back to HTTP/2 or HTTP/1.1 when the host doesn't have it.

HTTP/2

HTTP/2 is negotiated over TLS via ALPN whenever the server supports it (verified against live forges), falling back to HTTP/1.1 otherwise. This gives multiplexing and lower latency without any per-host configuration.

HTTP/3

HTTP/3 is supported via github.com/quic-go/quic-go/http3 and is tried first on each host automatically — no manual opt-in is needed:

• The first request to a host attempts HTTP/3 (newest available); if the host answers over h3 it is used from then on.
• If the h3 attempt fails (e.g. the host has no UDP/QUIC), the request transparently falls back to HTTP/2 or HTTP/1.1, so an h3 failure is never visible to the caller.
• A per-host negative cache (TTL ~1h) remembers hosts that failed h3, so a host without h3 isn't re-triggered with a QUIC handshake on every request. Once the TTL expires, h3 is tried again in case the host started serving it.
• An Alt-Svc: h3= response header records a positive capability (confirms h3 rather than gating it — the default is already to try).
• scraper.EnableHTTP3(host) is only an override that keeps trying h3 even during a negative cool-down; normal operation never needs it.

Note: Go 1.27's stdlib has no HTTP/3 client and x/net/http3 is only a registration shim; this is why the quic-go transport provides it.

Forcing a protocol

If you set an explicit url: http://... in the config, the indexer connects with plain HTTP (HTTP/1.1) instead of HTTPS. The stored identity and metrics labels always use the bare hostname regardless of the connection scheme.

---

Configuration reference

Both ffd-indexer and ffd-metrics read the same YAML file (-config <path>, default config.yaml).

db_path: "./index.db"
typesense:
  host: "http://localhost:8108"
  api_key: "dev-key"      # inline key, or
  # api_key_file: /run/secrets/typesense   # a secret file (takes precedence)
metrics:
  listen: "127.0.0.1:9090"
forges:
  - url: git.example.org
    type: forgejo
  - url: git.example.net
    type: auto          # default when omitted

Key	Type	Description
db_path	string	Path to the SQLite database.
typesense.host	string	Typesense server address.
typesense.api_key	string	Typesense API key (inline).
typesense.api_key_file	string	Path to a file containing the API key (e.g. sops/agenix secret). Takes precedence over api_key; contents are trimmed of surrounding whitespace.
metrics.listen	string (optional)	Address ffd-metrics binds for /metrics. The whole metrics: block may be omitted when metrics are not used.
forges[].url	string	Forge bare hostname (FQDN); a leading scheme is accepted to force the connection protocol. Identity is always the bare hostname.
forges[].type	string	forgejo | gitea (pinned, recommended) or auto (default, experimental).

A type other than forgejo, gitea or auto is rejected at startup. See Supported forge types for detection behavior.

Typesense API key

The API key can be given inline (typesense.api_key) or read from a file (typesense.api_key_file) — the file takes precedence when both are set. The file is read at startup and trimmed of surrounding whitespace/newlines, so a secret deployed via sops or agenix (e.g. at /run/secrets/...) can be referenced directly without the key ever appearing in the config or the Nix store.

Container

• nix build .#container — an image running ffd-indexer once; expects a config mounted at /etc/ffd-indexer/config.yaml (see README quick start). db_path is resolved relative to the container working directory, so mount a volume. The image bundles CA certificates so HTTPS to forges works.
• nix build .#container-metrics — an image running the ffd-metrics daemon, exposing :9090.
• nix run .#default -- -config config.yaml — run the indexer from the flake.
• nix run .#ffd-metrics -- -config /etc/ffd-indexer/config.yaml — run the metrics daemon from the flake.

---

Prometheus metrics

A separate long-running binary, ffd-metrics, serves the state of the index as Prometheus metrics at /metrics. It never scrapes the forges itself: the indexer's scheduled runs record per-forge status into the SQLite database, and ffd-metrics renders that state on each scrape.

Both binaries share the same config file. Configure the listen address under metrics.listen (e.g. 127.0.0.1:9090, default).

Metric reference

Metric	Type	Labels	Description
ffd_forge_info	gauge	forge_url, forge_type, forge_version	Constant 1 per forge with its detected type and software version
ffd_forge_online	gauge	forge_url, forge_type	1 if the forge was reachable and scraped successfully on its last run, else 0 (so offline forges still appear)
ffd_repository_info	gauge	repo_url, forge_url, forge_type	Constant 1 per stored repository
ffd_repository_languages	gauge	repo_url, language, forge_url, forge_type	Constant 1 per (repository, language) pair

PromQL queries

Everything beyond the primitive metrics is derived with PromQL:

• repositories per forge: count by (forge_url) (ffd_repository_info)
• repositories overall: count(ffd_repository_info)
• forges per type: count by (forge_type) (ffd_forge_online)
• language usage: sum by (language) (ffd_repository_languages)
• forge versions: ffd_forge_info (constant gauge, read forge_version label)
• offline forges: ffd_forge_online == 0

Metrics daemon

Run it directly:

ffd-metrics -config config.yaml
# or from the flake
nix run .#ffd-metrics -- -config /etc/ffd-indexer/config.yaml

The flake also ships container-metrics (nix build .#container-metrics), an image that runs the daemon on :9090, and the NixOS module enables a ffd-metrics systemd service alongside the one-shot indexer timer.

---

Supported forge types

The forge software is either pinned per forge in the config (type: forgejo / type: gitea) or detected on every run (type: auto).

Type detection

We recommend pinning each forge's type by hand; this selects the scraper deterministically and requires no probing. The default is auto, which re-detects the type on every run because the hoster may have changed the backend (e.g. Gitea→Forgejo). auto is experimental:

• robots.txt is honored — a forge that disallows /api/ will not be probed, so auto reports unknown for it;
• detection may misclassify until more detectors land;
• the previously detected type naturally comes first because detection starts with the most specific signal.

Forge types without a detector are reported as unknown (forge_type="unknown") and skipped; the metrics daemon still shows their status row.

Detection detail

Type	Detection
Forgejo	Serves the Forgejo-native GET /api/forgejo/v1/version, which returns 200 with a version JSON. Forgejo is a Gitea fork and also serves the shared /api/v1/version, so this endpoint is the discriminator: a hit is a strong Forgejo signal. The version is recorded and exposed as ffd_forge_info{forge_version=...}.
Gitea	Does not serve /api/forgejo/v1/version (404) but does serve the shared GET /api/v1/version, so an instance is classified Gitea if and only if the Forgejo probe fails and the shared endpoint answers. If other forges later serve /api/v1/version (e.g. Gogs), that classification will need a further discriminator.

Forgejo and Gitea share the same /api/v1 surface (Forgejo is a fork of Gitea), so both use the same scraper class.

---

NixOS module

The flake exposes nixosModules.default (option namespace services.federated-forge-discovery.indexer). It enables:

• a one-shot ffd-indexer systemd service and a timer driving it (schedule: daily [default], hourly, or weekly — or any systemd OnCalendar expression);
• a ffd-metrics systemd daemon (Type=simple, restart on failure);
• the rendered config at /etc/ffd-indexer/config.yaml, owned by the service user ffd-indexer;
• a container/container-metrics integration.

Options:

Option	Default	Description
enable	—	Enables the service and timer.
package	—	The package to run (ffd-indexer from the flake).
schedule	"daily"	daily | hourly | weekly, or an OnCalendar string.
dbPath	—	Path to the SQLite DB.
typesense.host	—	Typesense address.
typesense.apiKey	""	Typesense API key (inline).
typesense.apiKeyFile	null	Path to a secret file (sops/agenix) containing the key; takes precedence over apiKey.
metrics.enable	false	Enable the optional ffd-metrics daemon (and render the metrics: block). Disabled by default; set true to run metrics.
metrics.listen	"127.0.0.1:9090"	Address for the metrics daemon.
forges	[]	List of { url, type }; type is auto (default), forgejo or gitea.

---

Development

The dev shell (nix develop) provides Go, gomod2nix, staticcheck, and a local Typesense server for development.

nix develop          # enter the dev shell (Go toolchain, gomod2nix, etc.)
go test ./...        # run tests (store, scraper, config)
go vet ./...
nix flake check      # lint (golangci-lint) + all tests inside Nix
nix build .#default .#ffd-metrics .#container .#container-metrics

Continuous integration

.forgejo/workflows/ci.yml runs the pipeline on Forgejo Actions with no upstream actions: the checkout is a plain git clone, and every step calls the flake (nix flake check, nix build, nix develop) inside the nixos/nix container image. Four isolated jobs:

Job	Enforces	Fails on
checks	nix flake check	unit tests (go test ./...) or golangci-lint
build	nix build .#default .#ffd-metrics .#container .#container-metrics	any package failing to build
format	gofmt -l and nix fmt	unformatted .go / .nix files
integration	live scrape of git.xengi.de sent to a Typesense services: container	a scrape or sync assertion failing

The integration job is a real end-to-end test run inside the workflow: a Typesense container (the services: feature) is started alongside the job, the indexer scrapes the repository's own forge (git.xengi.de, type: auto → real detection) and syncs into that service. It then asserts the forge is online in sqlite and that the Typesense document count equals the sqlite project count. The Typesense service image (typesense/typesense:30.2) matches the flake's typesense-server version, so CI mirrors local dev (where you would use nix run .#typesense-server). It depends on the forge being up — it is a live test by design.

Run all checks locally with nix flake check + gofmt -l . + nix fmt (followed by a git diff --quiet). The runner needs the debian-stable label and outbound access to the forge, github.com (flake inputs) and cache.nixos.org.

Dependencies

Go modules are vendored into Nix via gomod2nix.toml (generated with gomod2nix generate). After adding or upgrading a Go dependency, regenerate gomod2nix.toml before nix flake check/nix build will reflect it.