ffd/indexer
Watch
1
0
Fork
You've already forked indexer
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Federated Forge Discovery Indexer
3 commits 1 branch 0 tags 71 KiB
  • Go 87.4%
  • Nix 12.6%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Ricardo (XenGi) Band bde13e9b68
 scraper done; scrapes to local sqlite db
2026-09-28 15:36:42 +02:00
scraper scraper done; scrapes to local sqlite db 2026-09-28 15:36:42 +02:00
store scraper done; scrapes to local sqlite db 2026-09-28 15:36:42 +02:00
.editorconfig initial dump 2026-09-28 12:21:17 +02:00
.gitignore scraper done; scrapes to local sqlite db 2026-09-28 15:36:42 +02:00
config.go scraper done; scrapes to local sqlite db 2026-09-28 15:36:42 +02:00
flake.lock scraper done; scrapes to local sqlite db 2026-09-28 15:36:42 +02:00
flake.nix scraper done; scrapes to local sqlite db 2026-09-28 15:36:42 +02:00
go.mod scraper done; scrapes to local sqlite db 2026-09-28 15:36:42 +02:00
go.sum scraper done; scrapes to local sqlite db 2026-09-28 15:36:42 +02:00
gomod2nix.toml scraper done; scrapes to local sqlite db 2026-09-28 15:36:42 +02:00
LICENSE Initial commit 2026-09-25 13:07:17 +02:00
main.go scraper done; scrapes to local sqlite db 2026-09-28 15:36:42 +02:00
option.nix initial dump 2026-09-28 12:21:17 +02:00
README.md scraper done; scrapes to local sqlite db 2026-09-28 15:36:42 +02:00

README.md

indexer

Federated Forge Discovery Indexer

Builds an index of public open source software projects from configured forges.

Scrape behavior

For each configured forge, the indexer:

  1. Checks robots.txt (GET /robots.txt) before any API call. If the instance disallows /api/, the instance is skipped.
  2. Lists public source repositories via the repository search API (GET /api/v1/repos/search), paginated. Forks and mirrors are excluded.
  3. For each repository, enriches it via the Forgejo API: reads the root directory once, then fetches topics, languages, latest commit, latest release, the README, and detects licenses from the repo's files.

Errors are non-fatal: an instance or a single repo that fails is skipped and the run continues.

Request volume

Per instance:

Request Count
GET /robots.txt 1
GET /api/v1/repos/search (page) ceil(N/50) where N = total repos

Per repository:

Request Endpoint Purpose
list root contents GET /api/v1/repos/{o}/{r}/contents/ shared: README selection + license scan (1 call)
languages GET /api/v1/repos/{o}/{r}/languages Project.Languages
topics GET /api/v1/repos/{o}/{r}/topics Project.Topics
latest commit GET /api/v1/repos/{o}/{r}/commits?limit=1 Project.LatestCommit
latest release GET /api/v1/repos/{o}/{r}/releases/latest Project.LatestRelease
README GET /api/v1/repos/{o}/{r}/raw/{branch}/{file} Project.Readme
license files 1+ × GET /api/v1/repos/{o}/{r}/raw/{branch}/{file} SPDX detection

So roughly 6–8 API requests per repository plus the paginated search requests and a single robots.txt check per instance.