indexer/README.md

[![CI Build Status](https://git.xengi.de/ffd/indexer/badges/workflows/ci.yaml/badge.svg?style=flat-square)](https://git.xengi.de/ffd/indexer/actions?workflow=ci.yaml)
[![Container Build Status](https://git.xengi.de/ffd/indexer/badges/workflows/container.yaml/badge.svg?style=flat-square)](https://git.xengi.de/ffd/indexer/actions?workflow=container.yaml)

# ffd-indexer

Federated Forge Discovery Indexer.

Builds and serves an index of public open-source software projects, discovered
from self-hosted forges you configure. The indexer scrapes each forge's public
API, enriches every repository with languages, topics, licenses and READMEs,
stores the result in SQLite, and exposes the state as Prometheus metrics and a
Typesense search index.

> **Why this exists** — public code is scattered across many small forges, and
> there is no federated view of it. This project fills that gap: point it at a
> list of forges, and it continuously aggregates and indexes what is publicly
> hosted there, so you can search and measure across all of them at once.

**Status:** working. Both binaries build from a Nix flake, ship as systemd
services and containers, and are verified end-to-end against live forges.
License: **AGPL-3.0** (see below).

Two binaries ship together:

- **`ffd-indexer`** — the scraper. Runs once per schedule, scrapes every
  configured forge, stores the index, and syncs it to Typesense.
- **`ffd-metrics`** — a long-running daemon. Serves the stored index state as
  Prometheus metrics at `/metrics`; it never scrapes forges itself.

---

## Quick start

The fastest way to try it is the flake's container image (it runs once and
exits, bundling CA certificates so HTTPS to forges work out of the box):

```sh
nix build .#container
docker load < result
docker run --rm \
  -v "$PWD/config.yaml:/etc/ffd-indexer/config.yaml:ro" \
  ffd-indexer
```

A minimal `config.yaml`:

```yaml
db_path: /data/index.db
forges:
  - url: git.example.org
    type: forgejo
```

Mount a volume for the DB so the index persists between runs:

```sh
docker run --rm \
  -v "$PWD/config.yaml:/etc/ffd-indexer/config.yaml:ro" \
  -v "$PWD/data:/data" \
  ffd-indexer
```

Or run from the flake directly (local dev):

```sh
nix run .#default -- -config config.yaml
```

**The next step** — point it at a second forge and check the output. See
[Configuration](#configuration) for the `type` field, and [DOCS.md](DOCS.md)
for metrics, architecture, and the NixOS module.

---

## Configuration

The indexer reads a single YAML file (default `config.yaml`, override with
`-config <path>`). Both binaries share the same file.

```yaml
db_path: "./index.db"              # SQLite database (index + per-forge status)
typesense:                         # optional search index
  host: "http://localhost:8108"
  api_key: "dev-key"               # inline key...
  # api_key_file: /run/secrets/ts  # ...or a secret file (takes precedence)
metrics:
  listen: "127.0.0.1:9090"         # ffd-metrics binds here and serves /metrics
forges:
  - url: git.example.org
    type: forgejo
  - url: git.example.net
```

- **`url`** — a bare hostname (FQDN). The indexer connects over `https://` by
  default and adds it internally. You may prefix `http://` or `https://` to
  force the connection protocol. The stored identity and all metrics labels
  always use the bare hostname.
- **`type`** — the forge's software. Set it by hand (`forgejo` or `gitea`) for
  the recommended, deterministic setup. The default `auto` re-detects the type
  on every run; it is **experimental** (see [DOCS.md](DOCS.md)). A forge that
  disallows `/api/` in its robots.txt is skipped either way.
- **`typesense.api_key` / `typesense.api_key_file`** — the Typesense API key,
  either inlined or read from a secret file (e.g. a [sops](https://github.com/getsops/sops)/
  [agenix](https://github.com/ryantm/agenix) secret). If both are set, the
  **file takes precedence**, so the key never needs to appear in the config.

---

## What it does

For each configured forge the indexer:

1. **Respects robots.txt** — skips the instance if `/api/` or a sub-API is
   disallowed.
2. **Lists public source repositories** — paginated search; forks and mirrors
   are excluded.
3. **Enriches each repository** — fetches topics, languages, latest commit,
   latest release, the README, and derives SPDX licenses from the repo's files.

The indexer runs once and exits (systemd timer / container); it never stays
resident. On each run it records per-forge status (online, type, version,
last error) into SQLite, upserts every project, and pushes the index to
Typesense.

Everything else — request volume, the HTTP transport (HTTP/2, optional
HTTP/3), error handling, and the full detail — is in [DOCS.md](DOCS.md).

---

## Verify it's working

The companion daemon `ffd-metrics` serves the index's state as Prometheus
metrics — it never scrapes forges itself; it renders the SQLite state on each
scrape:

```sh
ffd-metrics -config config.yaml
curl -s http://127.0.0.1:9090/metrics
```

You should see one series per configured forge: `ffd_forge_info` (type +
software version) and `ffd_forge_online` (0/1). The full metric reference and
ready-made PromQL queries are in [DOCS.md](DOCS.md).

For a production setup, the NixOS module wires the one-shot indexer timer, the
`ffd-metrics` daemon, and the config file — see [DOCS.md](DOCS.md).

---

## Contributing

Contributions are welcome. Please:

1. Run `go test ./...` and `go vet ./...` from the dev shell
   (`nix develop`), and run `nix flake check` (lint + tests) before opening a
   PR.
2. Keep the diff small and focused, and update the README/DOCS when behavior
   changes.
3. Open a pull request; issues and PRs are triaged.

---

## License

This project is licensed under the **GNU Affero General Public License v3.0**
(AGPL-3.0) — see [LICENSE](LICENSE). It is a strong-copyleft license selected
for network server software: if you modify it and provide the resulting
service to others over a network, those modifications must be made available
under the same license.