Repository navigation
Lead the README with a screenshot, quickstart and a real report - #3
Merged
Merged
Conversation
The README opened with method details and reached the quickstart halfway down, and its example report was hand-written. It now reads: what the tool does, a screenshot, how to run it, a table of the bundled scenarios with the risk level and findings run_checks gives for each, and then a report Claude Opus 5.5 rendered at low effort from the optimistic_site_shortcut payload, quoted verbatim. The sample payload is the same scenario's, so the screenshot, payload and report describe one table. The method sections are unchanged. The screenshot is a new offline capture of the current UI on optimistic_site_shortcut (no API key, no report). The old one showed caption text the app no longer prints and cut off two metric labels. docs/DEMO.md gains a correction: the API key field is no longer pre-filled, and an empty field uses the exported ANTHROPIC_API_KEY.
A live risk report on optimistic_site_shortcut wrote "With 98.8% of patients in both train and test". leakage_rate is the share of distinct test patients who also appear in train; on that table 990 of 2,000 patients are in both splits. Both prompts now define leakage_rate and say it is not a share of all patients.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch reorders the README so a visitor sees what the tool does, a screenshot, and how to run it before the method details, and it fixes one way the reporter misdescribed leakage.
README
optimistic_site_shortcut(no API key, no report requested), then the quickstart.run_checksgives for each.optimistic_site_shortcutpayload, quoted verbatim. It is one of three runs of that configuration in the October 2026 model comparison; each metric value in it is in the payload. The sample payload is now the same scenario's.docs/DEMO.mdnotes that the API key field is no longer pre-filled.Reporter prompts
A live report wrote "98.8% of patients in both train and test".
leakage_rateis the share of distinct test patients who also appear in train (990 of 2,000 patients are in both splits on that table). Both prompts now define it, and a test checks that the rule is present.Verification
python -m pytest -q: 59 passed.ruff check: clean.run_checksoutput, including the sample payload's values and key order and every number in the quoted report.