indexer/README.md

ffd-indexer

Federated Forge Discovery Indexer.

Builds and serves an index of public open-source software projects, discovered from self-hosted forges you configure. The indexer scrapes each forge's public API, enriches every repository with languages, topics, licenses and READMEs, stores the result in SQLite, and exposes the state as Prometheus metrics and a Typesense search index.

Why this exists — public code is scattered across many small forges, and there is no federated view of it. This project fills that gap: point it at a list of forges, and it continuously aggregates and indexes what is publicly hosted there, so you can search and measure across all of them at once.

Status: working. Both binaries build from a Nix flake, ship as systemd services and containers, and are verified end-to-end against live forges. License: AGPL-3.0 (see below).

Two binaries ship together:

• ffd-indexer — the scraper. Runs once per schedule, scrapes every configured forge, stores the index, and syncs it to Typesense.
• ffd-metrics — a long-running daemon. Serves the stored index state as Prometheus metrics at /metrics; it never scrapes forges itself.

---

Quick start

The fastest way to try it is the flake's container image (it runs once and exits, bundling CA certificates so HTTPS to forges work out of the box):

nix build .#container
docker load < result
docker run --rm \
  -v "$PWD/config.yaml:/etc/ffd-indexer/config.yaml:ro" \
  ffd-indexer

A minimal config.yaml:

db_path: /data/index.db
forges:
  - url: git.example.org
    type: forgejo

Mount a volume for the DB so the index persists between runs:

docker run --rm \
  -v "$PWD/config.yaml:/etc/ffd-indexer/config.yaml:ro" \
  -v "$PWD/data:/data" \
  ffd-indexer

Or run from the flake directly (local dev):

nix run .#default -- -config config.yaml

The next step — point it at a second forge and check the output. See Configuration for the type field, and DOCS.md for metrics, architecture, and the NixOS module.

---

Configuration

The indexer reads a single YAML file (default config.yaml, override with -config <path>). Both binaries share the same file.

db_path: "./index.db"              # SQLite database (index + per-forge status)
typesense:                         # optional search index
  host: "http://localhost:8108"
  api_key: "dev-key"               # inline key...
  # api_key_file: /run/secrets/ts  # ...or a secret file (takes precedence)
metrics:
  listen: "127.0.0.1:9090"         # ffd-metrics binds here and serves /metrics
forges:
  - url: git.example.org
    type: forgejo
  - url: git.example.net

• url — a bare hostname (FQDN). The indexer connects over https:// by default and adds it internally. You may prefix http:// or https:// to force the connection protocol. The stored identity and all metrics labels always use the bare hostname.
• type — the forge's software. Set it by hand (forgejo or gitea) for the recommended, deterministic setup. The default auto re-detects the type on every run; it is experimental (see DOCS.md). A forge that disallows /api/ in its robots.txt is skipped either way.
• typesense.api_key / typesense.api_key_file — the Typesense API key, either inlined or read from a secret file (e.g. a sops/ agenix secret). If both are set, the file takes precedence, so the key never needs to appear in the config.

---

What it does

For each configured forge the indexer:

1. Respects robots.txt — skips the instance if /api/ or a sub-API is disallowed.
2. Lists public source repositories — paginated search; forks and mirrors are excluded.
3. Enriches each repository — fetches topics, languages, latest commit, latest release, the README, and derives SPDX licenses from the repo's files.

The indexer runs once and exits (systemd timer / container); it never stays resident. On each run it records per-forge status (online, type, version, last error) into SQLite, upserts every project, and pushes the index to Typesense.

Everything else — request volume, the HTTP transport (HTTP/2, optional HTTP/3), error handling, and the full detail — is in DOCS.md.

---

Verify it's working

The companion daemon ffd-metrics serves the index's state as Prometheus metrics — it never scrapes forges itself; it renders the SQLite state on each scrape:

ffd-metrics -config config.yaml
curl -s http://127.0.0.1:9090/metrics

You should see one series per configured forge: ffd_forge_info (type + software version) and ffd_forge_online (0/1). The full metric reference and ready-made PromQL queries are in DOCS.md.

For a production setup, the NixOS module wires the one-shot indexer timer, the ffd-metrics daemon, and the config file — see DOCS.md.

---

Contributing

Contributions are welcome. Please:

1. Run go test ./... and go vet ./... from the dev shell (nix develop), and run nix flake check (lint + tests) before opening a PR.
2. Keep the diff small and focused, and update the README/DOCS when behavior changes.
3. Open a pull request; issues and PRs are triaged.

---

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0) — see LICENSE. It is a strong-copyleft license selected for network server software: if you modify it and provide the resulting service to others over a network, those modifications must be made available under the same license.