feat: per-script smoke timings as a standing dataset (#264) - #265
Merged
Conversation
The runner's per-entry timings existed only as `PASS <name> — <n>s` lines in job logs, recovered by hand-scraping every time a timing question came up. Since the smoke-runner delegation (#260-#263) all ten workspace runners are thin shims over `run_python.py`, so recording them routinely is one change here rather than ten repo sweeps. `RunReport.write()` now also emits `smoke_timings.json` into the report dir: one row per entry — `{entry, kind, status, seconds, cap_s, exit_code}` — plus the run metadata (project, directory, run_type, env_profile, python, ts) and a `legs` list. `seconds` is the runner's own `time.time()` measurement, the same number the console line prints, never re-derived. `cap_s` is `timeout_for`'s resolved cap, so a TIMEOUT row records the cap it hit. `ScriptResult` gained `cap_seconds` and `exit_code`, set by `build_util` at every execution site — durations were already recorded, caps and exit codes were not. They are deliberately absent from `to_dict()`: that dict is the per-run JSON PyAutoHeart's `script_timing`/`test_run` and `aggregate_results` read, and its shape stays byte-compatible. One timings file per report DIRECTORY, not per leg. A report dir receives several runner invocations (a gate's script and notebook legs; every directory of every workspace in the `run_all` mega-run), so each write merges its rows into the existing file keyed on the entry path: both legs survive, re-running a leg replaces its own rows, `legs` records every contributor. An unreadable file is replaced rather than merged. `aggregate_results` skips it by name — it is a sidecar, not a run report, and would otherwise enter `runs` as an empty phantom. An entry that never ran carries `"seconds": null`, never a fabricated zero: that is what `was_timed` is for, and it covers both a skipped entry and one listed but missing. With `$GITHUB_STEP_SUMMARY` set, each write appends a slowest-first table of its own entries to the job summary, so a run's timings are one click away in the Actions UI. Off CI nothing is appended, and a write failure is reported and swallowed — losing the summary must not fail a run whose scripts passed. Also fixes the notebook leg's surface statement: `run.py` never passed `env_profile` to its report, so every notebook report claimed "unknown" while running under a resolved profile. The timing dataset inherits that field, where an unknown surface makes two runs incomparable for exactly the reason `_surface` exists. PyAutoHeart's `smoke-tests.yml` uploads the report dir as the companion change, which is what makes the dataset outlive the job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EoDPz2LevKeBaDwqFKtZrU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #264. Phase 2 of the test-performance board arc (PyAutoLabs/PyAutoHeart#163): the runner records per-script timings routinely, so "what is slow" stops being a job-log archaeology session — one change here, all ten delegated workspace runners inherit it. Sibling PR: PyAutoLabs/PyAutoHeart#167 (artifact upload in the reusable workflow).
smoke_timings.jsonemitted fromRunReport.write()— one call site coversrun_python.py,run.pyandgenerate.py. One merged file per report directory (rows keyed on entry path, so the script and notebook legs both survive and a re-run replaces its own rows;legsrecords every contributor). Schemasmoke_timings/1: per entry{entry, kind, status, seconds, cap_s, exit_code}—secondsis the runner's own measured duration (the number the[PASS] — <n>sline prints), TIMEOUT entries carry the cap they hit, skips areseconds: null, never a fabricated 0.$GITHUB_STEP_SUMMARYis set, a slowest-first timing table per leg — every gate run's timings one click away with no artifact download.ScriptResultgainedcap_seconds/exit_codeat all 8 execution sites, deliberately absent fromto_dict()so the per-run JSONs Heart'sscript_timing/test_runandaggregate_resultsread stay byte-compatible (pinned by a test).aggregate_resultsskips the sidecar by name so the mega-run surface stays clean.run.pynever passedenv_profile, so every notebook report claimedunknown— now threaded through (negative-tested).Tests: 382 passed (+19), 14 pre-existing environmental failures unchanged (missing
ipynb-py-convert/pngquantlocally; identical set onmain). Tenant firewall green.🤖 Generated with Claude Code
https://claude.ai/code/session_01EoDPz2LevKeBaDwqFKtZrU
Generated by Claude Code