feat(report): stamp reports with a reproducibility/provenance block (closes #31) - #38
Open
dchaudhari7177 wants to merge 1 commit into
Open
feat(report): stamp reports with a reproducibility/provenance block (closes #31)#38dchaudhari7177 wants to merge 1 commit into
dchaudhari7177 wants to merge 1 commit into
Conversation
) A card stated effect and cliff numbers but not the conditions they were measured under. Provenance now records n_seeds, model_id, repeng_version, hardware and wall_clock_s, rendered at the foot of both the markdown and HTML cards and exposed through --json. Design notes: - Used a frozen dataclass rather than the pydantic model the issue suggested. Core declares `dependencies = []` ("core stays light") and report.py models every other type as a frozen dataclass, so pydantic here would be the first core dependency. Easy to switch if you would rather have it. - n_seeds is the MINIMUM across dose and layer points, not the mean - the weakest point bounds the claim. - provenance_from() always recomputes n_seeds from the parsed curves and only passes the caller's other fields through, so the seed count printed on an artifact cannot disagree with the CSVs it came from. A test asserts a caller-supplied n_seeds=999 is overridden. - Missing fields render as "unknown" rather than failing, and an UNKNOWN seed count is deliberately not a contract violation - only a known count below MIN_SEEDS warns. - build_report() keeps its signature and return type; the new `provenance` argument is optional, so existing callers are unaffected. --model was documented as "recorded in output" but was never actually read. It now feeds model_id as the fallback when no vector is passed; a vector's own metadata takes precedence over it. Also adds --hardware and --wall-clock-s for the GPU sweep to supply, and a stderr warning when a card is built below the seed contract. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #31.
A card stated effect and cliff numbers but not the conditions they were measured under.
Provenancenow recordsn_seeds,model_id,repeng_version,hardwareandwall_clock_s, renders at the foot of the markdown and HTML cards, and is exposed through--json.Rendered block
And in
--json(paths stay flat, so existing consumers are unaffected):Design decisions worth reviewing
dependencies = []("core stays light"), andreport.pymodels every other type as a frozen dataclass — pydantic here would be the first core dependency and would breaktest_no_optional_deps_imported's spirit. Say the word and I'll switch it.n_seedsis the minimum across dose and layer points, not the mean: the weakest point bounds the claim.provenance_from()always recomputesn_seedsfrom the parsed curves and only passes the caller's other fields through, so what is printed on an artifact can't disagree with the CSVs it came from. A test asserts a caller-suppliedn_seeds=999is overridden.unknownrather than failing, and an unknown seed count is deliberately not a contract breach — only a known count belowMIN_SEEDSwarns.build_report()keeps its signature and return type;provenanceis an optional keyword.Incidental fix
--modelwas documented as "recorded in output; informational" but was never actually read anywhere. It now feedsmodel_idas the fallback when no vector is passed, with a loaded vector's own metadata taking precedence.Also adds
--hardware/--wall-clock-sfor the GPU sweep to supply, and a stderr warning when a card is built below the seed contract.Tests
8 new CPU-only tests, no model download: min-seed selection, pass-through vs. recomputation,
unknowndefaults, thebelow_seed_contractboundary (parametrised over 1/2/3/4/None), markdown + HTML rendering, the low-seed warning appearing in both,as_dict(), and thatsummarise_provenance()agrees with what the card renders.One note on
mypy: with the repo's configuredpython_version = "3.11"it aborts early on this machine withnumpy/__init__.pyi:737: Type statement is only supported in Python 3.12 and greater— reproducible on a pristine tree, so it is a local numpy-stub artefact, not from this change. TheSuccessabove is frommypy src --python-version 3.12, which gets the stub to parse so the actual source is checked.🤖 Generated with Claude Code