Measurement, export tooling, evaluation fixtures, and merged upstream fixes for extracting architectural parameters from RISC-V ISA material into riscv/riscv-unified-db.
Author: Ibteshamul Haque (@titoatwork) Context: LFX Fall 2026 Part II, Parameter SIG Spring Part I credit: @ishaan-arora-1 / UDB PRs #1765-#1832. This repository reproduces and extends that public surface and claims no Spring authorship.
| Question | Answer |
|---|---|
| What was investigated | Whether the published Part I extraction figures measure what they are read as measuring, and whether the UDB parameter data they are scored against is itself self-consistent |
| What was contributed | 9 merged upstream PRs, 9 open, 19 issues filed, review comments adopted into other contributors' merged code. Zero PRs rejected |
| What was learned | The headline recall number measures grounding, not discovery; most of it is awarded by fuzzy matching; and it moves ~10 points between byte-identical runs |
| How to check it | ./verify.sh re-derives each registered figure from the artifact it came from, and reports anything it cannot check rather than passing it. 119/119 checkable claims pass, offline, no API key |
| What comes next | docs/TERM-PLAN.md, nine weeks mapped to the five official Part II objectives |
git clone https://github.com/titoatwork/lfx-firstanalysis
cd lfx-firstanalysis
pip install -r riscv-param-extraction/requirements.txt # PyYAML, jsonschema
./verify.shTwo dependencies, both pure data handling. No credentials, no network model calls, no API key, no model SDK. Every failure prints which artifact disagreed with which published number. ./verify.sh --list prints the claim table without running the checks.
Six gates run. Five check what is published; the sixth checks what escaped being published as a claim, because a number that was never registered cannot be re-derived and will not fail. That gap is currently zero and held there, so a new figure added to any governed document without a claim behind it fails the build.
CI runs this exact path on every push, so the badge above tracks the same command a reviewer would type.
Windows, from the repository root:
$env:PYTHONPATH = "riscv-param-extraction"
python riscv-param-extraction/scripts/verify_claims.pyEvery row is checkable without trusting this page.
| Claim | Artifact | Proof |
|---|---|---|
| Published recall is grounding, not discovery | metrics.md grounding note |
Prompt injects all 185 gold names, set-identical to the pinned ground truth. Corrected in public on #2053 |
| Single-run recall is unstable | PRIMARY_RESULTS.md |
33.9% then 44.6%, same model, byte-identical prompt, temperature=0. Filed as #2163 |
| The cross-model gap is wider than headline recall shows | metrics.md §5 |
Adjusted recall 72.9% vs 32.2% (2.3x); exact-name 48.6% vs 6.2% (7.8x) |
| Gold WARL labels are largely undecidable from syntax | GOLD-CLASSIFICATION-AUDIT.md |
Of 26 gold WARL labels: 4 decidable, 4 stale, 18 undecidable. Posted on #2200 |
| Two independent audits converge on the same four names | PARAM-SCHEMA-SHAPES.md |
Schema-shape scan and IDL scan both dissent on SXLEN, UXLEN, VSXLEN, VUXLEN. The equality is gated as an invariant in verify_claims.py |
| Export produces schema-valid UDB YAML | export/ |
83/83 named, 20/20 new drafts. Structural validity only, not architectural approval |
| Findings became merged upstream fixes | docs/EVIDENCE.md |
9 merged PRs; 8 closed an issue filed from this measurement work, the ninth took up a maintainer's open request |
| The extraction task itself found a real defect | coding-challenge/ |
The Part II challenge snippet describes a cache block as a power of two. UDB's own parameter did not enforce it, until #2189 |
Part I prompts build every request through build_user_message(), which unconditionally injects the complete list of 185 gold parameter names, with the instruction "When a parameter you find matches one of these known names, use the exact name." That list is set-identical to the pinned ground truth.
So the published 72.9% and 32.2% figures measure grounding: given the catalogue, locate which entries apply to a passage and cite evidence. They do not measure whether a model can find architectural parameters without being handed the answer key. That second number is not measured in this repository, or anywhere public that I could find.
The figures are correctly measured and unchanged. What changed is the claim attached to them. I found this while building the variance experiment, corrected it in public on #2053, and every recall figure in this repository now carries the condition.
The same discipline retired a second claim of my own: a run-1 result suggesting WARL collapsed when the catalogue was removed reversed direction in run 2, so what stands is the narrower statement that the contrast sits inside run-to-run noise at N = 1.
Census 2026-08-12, re-derived from the GitHub API. Full index with links: docs/EVIDENCE.md.
| Kind | Count |
|---|---|
| Merged PRs authored | 9 |
| Open PRs authored | 9 |
| Issues authored | 19 |
| Unique issues and PRs involving this author | 54 |
| PRs rejected | 0 |
Merged: #2138 · #2146 · #2189 · #2215 · #2227 · #2256 · #2266 · #2362 · #2384
Open: #2289 · #2255 · #2212 · #2164 · #2395 · #2414 · #2419 · #2435 · #2459
Review comments carried into other people's merged PRs: #2090 (alignment defect adopted by the maintainer) · #2109 · #2197 (own earlier advice corrected after the author had built on it; the merged code follows the correction) · #2245 · #2284
Measurement threads: #2053 · #2163 · #2200 · #2251. These carry numbers, not only opinions; indexed in metrics.md §8.
riscv-param-extraction/
docs/metrics.md authoritative measured tables (§8 indexes upstream threads)
artifact_c/ preregistered dual-run variance experiment and results
analysis/ gold classification and schema-shape audits
export/ spreadsheet rows to draft UDB param YAML
manifests/ reproducible run records
workflow_slice/ evaluation fixtures and the review/export path
scripts/ claim verifier and audit scripts
docs/
EVIDENCE.md measurement limits and the upstream contribution index
FAQ.md how to read the numbers
TERM-PLAN.md nine-week plan mapped to the official objectives
coding-challenge/ Part II challenge submission: prompts, ten-model matrix, results
upstream-pr-drafts/ local drafts for filed and not-filed defects (historical archive)
verify.sh offline verification entrypoint
- Single-run recall is noisy. Treat any single figure as one sample, not a point estimate.
- Published recall is grounding, measured with the gold name list supplied in the prompt.
- Schema-valid export is structural validity. It is not a claim that a parameter is architecturally correct.
- Most of the adjusted score is awarded by inexact alignment passes, and that is where the run-to-run variance concentrates.
- Recall is a regression signal, not a coverage measure. A parameter that should exist but is absent from the gold cannot score as a miss.
- No Ruby toolchain on the development machine, so CI is the only validator for
.rbchanges and for whetheridlcaccepts new IDL. Nothing in this repository needs Ruby; the limitation applies to the UnifiedDB changes filed from it.
Fuller treatment: docs/EVIDENCE.md and docs/FAQ.md.
docs/TERM-PLAN.md maps nine weeks to the five official Part II objectives, with a Week 0 decision on which figure the term optimizes. A target hit by quietly changing the scoring condition is not a result.
This repository was developed with an AI coding assistant. Judge it on the technical artifacts and on ./verify.sh, which re-derives every registered figure from a committed artifact and reports what it cannot check rather than passing it quietly. Which claims to publish, which to withdraw, which findings to file upstream and which to leave alone were the author's calls, as were the corrections recorded in the three findings above.
See LICENSE and package-level licenses under riscv-param-extraction/.