CodeSOTA is a registry, not just a leaderboard: every number carries a date, a source, and a verification tier. Community contributions are how coverage grows — researchers adding their own papers and results, readers fixing numbers that don't add up, and domain experts proposing tasks we've missed.
This page maps what you want to do to the right channel. Everything lands in one of three
places: a website form, a GitHub issue, or a pull request against registry/.
| I want to… | Fastest path |
|---|---|
| Submit a paper (mine or someone else's) | codesota.com/submit — no account needed |
| Submit a result to an existing leaderboard | codesota.com/contribute/score, or the Submit a result issue form |
| Fix a wrong number, dead link, or mislabelled model | Correction issue form, or edit the JSON in registry/ directly via PR |
| Propose a new task (e.g. Speech Enhancement, Tabular ML) | New task issue form |
| Propose a new benchmark / dataset / leaderboard | New benchmark issue form |
| Add many results at once | PR against registry/benchmark_results.json (see below) |
| Report a site bug | Bug report |
| Suggest a feature | Feature request |
- Every result needs a citable primary source — the paper, the official repo, the vendor blog, or an independent eval. Not a tweet, not a screenshot, not an LLM's recollection. Vendor self-reported numbers are accepted but labelled as such via trust signals.
- You are responsible for what you submit. Use AI tools freely for the mechanical work, but verify every number and link yourself before submitting. Unchecked model output is the bug this registry exists to fix.
The registry/ directory is a deterministic JSON export of the live database —
diffable, forkable, CC BY 4.0. To contribute data directly:
- Fork this repo and edit the relevant file — most commonly
registry/benchmark_results.json(one row per model × dataset × metric × source),registry/datasets.json, orregistry/models.json. - Follow the existing field order and ID conventions (stable string keys; foreign keys
reference the
idof the target file). Seeregistry/README.mdfor the full schema. - Include the source URL on every result row, and add a
sources.jsonentry for new sources. - Keep one logical change per PR — one benchmark, one correction, one paper's results.
A maintainer reviews the PR, reconciles accepted changes into the database, and the next export reflects them here. Your name stays in the git history, and corrections that land on a live page are credited on the page.
One maintainer currently reviews everything; turnaround is usually days, sometimes a couple of weeks. The queue is public — open issues and PRs are the backlog. Contributors with a track record of accepted, well-sourced submissions will be invited to help review.
If you claim something and can't finish, just say so on the issue — no penalty, someone else picks it up.
Registry data is CC BY 4.0: reuse it, fork it, build on it, even commercially — with attribution linking back to codesota.com. By contributing data you agree it is published under the same licence.