Skip to content

feat(report): stamp reports with a reproducibility/provenance block (closes #31) - #38

Open
dchaudhari7177 wants to merge 1 commit into
bamdadd:mainfrom
dchaudhari7177:feat/provenance-block
Open

feat(report): stamp reports with a reproducibility/provenance block (closes #31)#38
dchaudhari7177 wants to merge 1 commit into
bamdadd:mainfrom
dchaudhari7177:feat/provenance-block

Conversation

@dchaudhari7177

Copy link
Copy Markdown
Contributor

Closes #31.

A card stated effect and cliff numbers but not the conditions they were measured under. Provenance now records n_seeds, model_id, repeng_version, hardware and wall_clock_s, renders at the foot of the markdown and HTML cards, and is exposed through --json.

Rendered block

## Reproducibility

| field | value |
|---|---|
| seeds per point (min) | 2 (contract: ≥ 3) |
| source model | org/demo-7b |
| repeng version | unknown |
| hardware | A100-40GB |
| sweep wall-clock | 612.5 s |

⚠️ **Measured over 2 seed(s): below the ≥ 3-seed reproducibility contract, so the
± spreads here are not a reliable error estimate.**

And in --json (paths stay flat, so existing consumers are unaffected):

"provenance": {"n_seeds": 2, "model_id": "org/demo-7b", "repeng_version": null,
               "hardware": "A100-40GB", "wall_clock_s": 612.5,
               "min_seeds": 3, "below_seed_contract": true}

Design decisions worth reviewing

  • Frozen dataclass, not pydantic. The issue suggested a pydantic model, but core declares dependencies = [] ("core stays light"), and report.py models every other type as a frozen dataclass — pydantic here would be the first core dependency and would break test_no_optional_deps_imported's spirit. Say the word and I'll switch it.
  • n_seeds is the minimum across dose and layer points, not the mean: the weakest point bounds the claim.
  • The seed count cannot be spoofed. provenance_from() always recomputes n_seeds from the parsed curves and only passes the caller's other fields through, so what is printed on an artifact can't disagree with the CSVs it came from. A test asserts a caller-supplied n_seeds=999 is overridden.
  • Unknown ≠ violation. Missing fields render unknown rather than failing, and an unknown seed count is deliberately not a contract breach — only a known count below MIN_SEEDS warns.
  • No breaking change. build_report() keeps its signature and return type; provenance is an optional keyword.

Incidental fix

--model was documented as "recorded in output; informational" but was never actually read anywhere. It now feeds model_id as the fallback when no vector is passed, with a loaded vector's own metadata taking precedence.

Also adds --hardware / --wall-clock-s for the GPU sweep to supply, and a stderr warning when a card is built below the seed contract.

Tests

8 new CPU-only tests, no model download: min-seed selection, pass-through vs. recomputation, unknown defaults, the below_seed_contract boundary (parametrised over 1/2/3/4/None), markdown + HTML rendering, the low-seed warning appearing in both, as_dict(), and that summarise_provenance() agrees with what the card renders.

pytest -q                 436 passed
ruff check .              All checks passed!
ruff format --check .     20 files already formatted
mypy src                  Success: no issues found in 5 source files

One note on mypy: with the repo's configured python_version = "3.11" it aborts early on this machine with numpy/__init__.pyi:737: Type statement is only supported in Python 3.12 and greater — reproducible on a pristine tree, so it is a local numpy-stub artefact, not from this change. The Success above is from mypy src --python-version 3.12, which gets the stub to parse so the actual source is checked.

🤖 Generated with Claude Code

)

A card stated effect and cliff numbers but not the conditions they were
measured under. Provenance now records n_seeds, model_id, repeng_version,
hardware and wall_clock_s, rendered at the foot of both the markdown and HTML
cards and exposed through --json.

Design notes:

- Used a frozen dataclass rather than the pydantic model the issue suggested.
  Core declares `dependencies = []` ("core stays light") and report.py models
  every other type as a frozen dataclass, so pydantic here would be the first
  core dependency. Easy to switch if you would rather have it.
- n_seeds is the MINIMUM across dose and layer points, not the mean - the
  weakest point bounds the claim.
- provenance_from() always recomputes n_seeds from the parsed curves and only
  passes the caller's other fields through, so the seed count printed on an
  artifact cannot disagree with the CSVs it came from. A test asserts a
  caller-supplied n_seeds=999 is overridden.
- Missing fields render as "unknown" rather than failing, and an UNKNOWN seed
  count is deliberately not a contract violation - only a known count below
  MIN_SEEDS warns.
- build_report() keeps its signature and return type; the new `provenance`
  argument is optional, so existing callers are unaffected.

--model was documented as "recorded in output" but was never actually read.
It now feeds model_id as the fallback when no vector is passed; a vector's own
metadata takes precedence over it.

Also adds --hardware and --wall-clock-s for the GPU sweep to supply, and a
stderr warning when a card is built below the seed contract.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stamp reports with a reproducibility/provenance block (seeds, model, hardware, wall-clock)

2 participants