prompt: the two test-performance follow-ups — issues, lifecycle, records

draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md → complete/2026/08/script-timing-baselines-fix.md

@@ -1,3 +1,55 @@
+# script_timing baselines made real — run-identity dedup + rename-aware slugs
+
+PyAutoHeart#166 → `ee915f3`, closing PyAutoHeart#165, merged 2026-08-24 on
+branch `claude/test-performance-dashboard-y3fdy7` (restarted from main after
+the phase-1 board merge). Phase 0 of the test-performance board arc
+(`complete/2026/08/test-performance-board.md`) — the per-script trend surface
+reads what this fixes.
+
+## What shipped
+
+- **The "every history is one value repeated 7×" defect was not seeding — it
+  was re-ingestion.** `run()` re-read `run_logs/latest` on every tick and
+  re-appended the same observation until the window filled with copies of one
+  run. History entries now carry `{duration_s, run_id, ts}` (run_id = the
+  resolved timestamped run dir behind the `latest` symlink); a re-tick on the
+  same run replaces the newest entry, so windows only grow across distinct
+  runs. Legacy bare-float histories still read; a legacy window of
+  all-identical values collapses to the single observation it provably was.
+- **Classification floor**: the yellow/red ratio fires only against ≥3
+  distinct-run samples (`MIN_BASELINE_RUNS`); below that a script counts as
+  `building_count` instead of being judged against a fake-stable median.
+  `red/yellow/green_count` keep their exact meaning; `dashboard.py` untouched.
+- **Rename-aware slugs, loud orphans**: history files untouched by a scan are
+  orphans; a new slug with no history adopts an orphan on an unambiguous
+  (workspace, script-name) match — the #216 restructure scenario now heals
+  itself (`migrated_count`). Ambiguous/unmatched orphans are reported
+  (`orphaned_count`), never deleted, never silent.
+- Histories written via `atomic_write_json` (was a bare `write_text`).
+- Tests 573 → 583; `test_script_timing.py` 6 → 16.
+
+## Key traps / findings
+
+- **Four existing tests had encoded the bug itself** — they re-ran `run()` on
+  the same results dir as a stand-in for repeated runs, which is exactly the
+  defect. When a test suite's fixture shape mirrors a bug, the tests pass for
+  the wrong reason; the fixtures now create one dir per run.
+- **Legacy empty run_ids each count as a distinct run** in the floor — history
+  that predates provenance is real accumulation once the identical-window
+  collapse has run; treating it as one run would have zeroed every surviving
+  baseline.
+- Orphan reporting is scan-relative: pointing Heart at a partial results dir
+  surfaces other workspaces' histories as "orphaned" (reporting only —
+  migration matching is workspace-scoped, so nothing can be misattached).
+
+## Follow-ups
+
+- `unit_test_timing` / `workspace_testmode_timing` keep their own history
+  mechanics — if they share the re-ingestion pattern, the same dedup applies;
+  not audited here (out of the prompt's scope).
+
+## Original prompt
+
 # Heart script_timing baselines are orphaned by path moves and filled with one repeated value
 
 Type: bug
@@ -9,6 +61,7 @@ Autonomy: supervised
 Priority: medium
 Status: formalised
 Filed: 2026-08-04 (backfilled from git)
+Issued: 2026-08-24
 
 Two independent defects in `PyAutoHeart/heart/checks/script_timing.py`, both found
 while diagnosing the jax_grad smoke timeouts (PyAutoHands#226). Neither is fixed by

complete/2026/08/smoke-timings-dataset.md

@@ -0,0 +1,125 @@
+# Per-script smoke timings are a standing dataset — one runner change, ten repos inherit
+
+PyAutoHands#265 → `d2a22f4` (closing PyAutoHands#264) + PyAutoHeart#167 →
+`3df42b5`, merged 2026-08-24. Phase 2 of the test-performance board arc
+(`complete/2026/08/test-performance-board.md`); answers item 4 of
+`draft/research/ci/smoke_timing_and_profiling.md` — "should the runner record
+per-script timings routinely?" — with **yes, shipped**. The smoke-runner
+delegation (#260–#263) is what made this one change instead of ten repo
+sweeps.
+
+## What shipped
+
+- **`smoke_timings.json`** (schema `smoke_timings/1`) emitted from
+  `RunReport.write()` — one call site covers `run_python.py`, `run.py` and
+  `generate.py`, so every delegated gate inherits it with zero per-repo
+  edits. Per entry: `{entry, kind, status, seconds, cap_s, exit_code}` —
+  `seconds` is the runner's own measured duration (the number the
+  `[PASS] — <n>s` line prints), TIMEOUT entries carry the cap they hit,
+  skips are `seconds: null`, never a fabricated 0. One merged file per
+  report directory, rows keyed on entry path (script + notebook legs both
+  survive; a re-run replaces its own rows; `legs` records contributors).
+- **Step summary**: with `$GITHUB_STEP_SUMMARY` set, a slowest-first timing
+  table per leg — every gate run's timings one click away, no artifact
+  download.
+- **`ScriptResult` gained `cap_seconds`/`exit_code`** at all 8 execution
+  sites — deliberately absent from `to_dict()` so the per-run JSONs Heart's
+  `script_timing`/`test_run` and `aggregate_results` read stay
+  byte-compatible (pinned by a test); `aggregate_results` skips the sidecar
+  by name so the mega-run surface stays clean.
+- **PyAutoHeart `smoke-tests.yml`** uploads the report dir as
+  `smoke-timings-<python-version>` (`if: always()`,
+  `if-no-files-found: ignore`, no `retention-days` — full default artifact
+  retention). The path is a glob (`test-results/` + `**/smoke_timings.json`)
+  because the reusable workflow never passes `--report-dir` — each
+  workspace's `run_smoke.py` does.
+- Drive-by fix: `run.py` never passed `env_profile`, so every notebook
+  report claimed `unknown` — threaded through, negative-tested.
+- Tests: PyAutoHands 363 → 382 passed (14 pre-existing environmental
+  failures unchanged — missing `ipynb-py-convert`/`pngquant` locally,
+  identical set on main); PyAutoHeart 573 → 576.
+
+## Key traps / findings
+
+- **The notebook report leg is `run.py`, not `run_notebook.py`** — the
+  latter is the single-notebook kernel-cwd shim `build_util` shells out to
+  and has no report path. Emit from `RunReport.write()` and every leg is
+  covered.
+- **A fixed per-leg filename would clobber across directories in the
+  mega-run** — one report dir hosts many `(project, directory, run_type)`
+  invocations; hence the merged file keyed on entry path. Known limit: two
+  workspaces sharing a relative path in one mega-run dir collapse to one
+  row (documented in the docstring; that dir is not a consumer).
+- **Keep new fields out of `to_dict()` until every consumer is audited** —
+  the per-run JSON shape is load-bearing for three downstream readers;
+  extending the dataclass without extending the serialization is the
+  compatible move, pinned by a byte-compat test.
+- `aggregate_results` globs `**/*.json` — any new sidecar in the report dir
+  becomes a phantom run unless excluded by name.
+
+## Follow-ups
+
+- The Heart board ingesting these artifacts into per-script rows with
+  STALL/SLOW verdicts (reuse `retime.py`'s vocabulary; bimodality
+  first-class) once a few weeks of data exist — the deferred phase 3 named
+  in the board record.
+
+## Original prompt
+
+# Per-script smoke timings as a standing dataset — one runner change, ten repos inherit
+
+Type: feature
+Target: pyautohands
+Repos:
+- @PyAutoHands
+- @PyAutoHeart
+Difficulty: medium
+Autonomy: supervised
+Priority: high
+Status: formalised
+Filed: 2026-08-24
+Issued: 2026-08-24
+
+Phase 2 of the test-performance board
+([`../../../docs/pyautoheart/test_performance_board_assessment.md`](../docs/pyautoheart/test_performance_board_assessment.md);
+phase 1 shipped 2026-08-24 as PyAutoHeart#164 + PyAutoBrain#261). Today the
+smoke runner's per-entry timings exist only as `[PASS] <name> — <n>s` lines in
+job logs, recovered by hand-scraping; the 2026-08-23 slow-vs-stall audit and
+the jax_grad budget work both had to rebuild their datasets that way. Since
+the smoke-runner delegation (PyAutoHands#260–#263) all ten workspace runners
+are thin shims over `autohands/run_python.py` — **so recording per-script
+timings routinely is now one PyAutoHands change, not ten repo sweeps.** This
+answers item 4 of `draft/research/ci/smoke_timing_and_profiling.md`
+("should the runner record per-script timings routinely?") with yes.
+
+## Task
+
+1. **PyAutoHands** — the report machinery (`result_collector.RunReport`,
+   already mandatory in the PR gate via `--report-dir`) additionally emits a
+   consolidated `smoke_timings.json` in the report dir: one entry per
+   script/notebook — `{entry, kind, status, seconds, cap_s (the cap in force
+   from build_util.timeout_for), exit_code}` — plus run metadata (project,
+   env profile, python version). When `$GITHUB_STEP_SUMMARY` is set, append a
+   compact per-entry timing table (slowest first) so every smoke run's
+   timings are one click away in the Actions UI with no artifact download.
+2. **PyAutoHeart** — the reusable `smoke-tests.yml` uploads the report dir as
+   a run artifact (`smoke-timings-<python-version>`, `if: always()`,
+   `if-no-files-found: ignore`), so the dataset persists the full artifact
+   retention window for every gate run across all ten repos at once.
+3. Timing must come from the runner's own measurement (the same clock the
+   `[PASS] — <n>s` line prints), never re-derived; a TIMEOUT entry records
+   the cap it hit.
+
+## Acceptance
+
+- A PR-gate smoke run on any workspace produces `smoke_timings.json` with one
+  timed entry per executed script/notebook and a step-summary table, with no
+  per-repo changes.
+- TIMEOUT entries carry `cap_s`; skipped entries are absent or explicitly
+  marked, never silently timed as 0.
+- Existing report consumers (`run_all.py`, Heart `test_run`/`script_timing`)
+  are unaffected.
+
+Follow-up (not this task): the Heart board ingesting these artifacts into
+per-script rows with STALL/SLOW verdicts (reuse `retime.py`'s vocabulary)
+once a few weeks of data exist.

complete/index.md

@@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema:
 only then grep a dated bucket. Curators: edit the band between the CURATED
 markers; everything below GENERATED is rebuilt.
 
-1095 records across 7 buckets.
+1097 records across 7 buckets.
 
 <!-- CURATED:START -->
 ## Highlights
@@ -187,6 +187,7 @@ _(curate hard-won records here — survives regeneration.)_
 - [samplers-surface-autolens-tiers](2026/08/samplers-surface-autolens-tiers.md) — the samplers faculty's SamplerSurface now scans the findings
 - [save-json-numpy-scalar-typeerror](2026/08/save-json-numpy-scalar-typeerror.md) — Adds `NumpyEncoder` in `autofit/tools/util.py` (`np.ndarray` ->
 - [script-size-guard-git-based](2026/08/script-size-guard-git-based.md) — replaced the rotting `.script_sizes.json` snapshot with a git-diff truncation
+- [script-timing-baselines-fix](2026/08/script-timing-baselines-fix.md)
 - [script-title-underline-off-by-one](2026/08/script-title-underline-off-by-one.md)
 - [searches-readme-dashboard](2026/08/searches-readme-dashboard.md)
 - [should-simulate-capped-branch-reuse](2026/08/should-simulate-capped-branch-reuse.md) — small, single-repo follow-up of the closed PyAutoNerves#153
@@ -196,6 +197,7 @@ _(curate hard-won records here — survives regeneration.)_
 - [small-datasets-rmtree-committed-data](2026/08/small-datasets-rmtree-committed-data.md)
 - [smoke-runner-delegation](2026/08/smoke-runner-delegation.md)
 - [smoke-runner-jupyter-guard](2026/08/smoke-runner-jupyter-guard.md)
+- [smoke-timings-dataset](2026/08/smoke-timings-dataset.md)
 - [spawn-autonomy-log-generated](2026/08/spawn-autonomy-log-generated.md)
 - [spawn-drift-self-heal](2026/08/spawn-drift-self-heal.md)
 - [spawn-empty-body-privacy-fix](2026/08/spawn-empty-body-privacy-fix.md)

dashboard.html

@@ -123,7 +123,7 @@
 </head>
 <body>
 <header class="hero"><span class="orb">📋</span><h1>PyAuto<b>Mind</b><span class="kind">Dashboard</span></h1><p class="tag">Intent. Priority. Flow.</p></header><p class="lede">Every task the Mind is holding. Tap a task's 📋 and its <code>/start_dev</code> command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. <a href="#recent">Recent</a> is the same work by date — what has been happening rather than what to do next.</p>
-<ul class="stats"><li><b>1</b><span>In flight</span></li><li><b>3</b><span>Parked</span></li><li><b>6</b><span>Planned</span></li><li><b>155</b><span>Backlog</span></li></ul>
+<ul class="stats"><li><b>1</b><span>In flight</span></li><li><b>3</b><span>Parked</span></li><li><b>6</b><span>Planned</span></li><li><b>154</b><span>Backlog</span></li></ul>
 <p class="muted mdsrc"><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/dashboard.md">markdown version</a></p>
 <h2>Start here</h2>
 <h3>Highest priority <span class="facets">(filed as high) — showing 12 of 17</span></h3>
@@ -163,7 +163,7 @@ <h2>Planned <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/blob
 <div class="task"><button class="copy" data-cmd="/route start the planned PyAutoMind task latent-nan-guard-honest-run — its record is in planned.md" aria-label="Copy the Claude command">📋</button><p><b>latent-nan-guard-honest-run</b><span class="facets"> — planned 2026-07-22</span></p></div>
 </details>
 <h2>Backlog <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/tree/main/draft">markdown version</a></h2>
-<p class="muted">155 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.</p>
+<p class="muted">154 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.</p>
 <details>
 <summary>feature — 29</summary>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/feature/autoarray/numba_cpu_likelihood_mge_convolution_and_caching.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/feature/autoarray/numba_cpu_likelihood_mge_convolution_and_caching.md">Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix caching</a><span class="tags"><span class="pill w">✨ feature</span><span class="pill">autoarray</span><span class="pill n">medium</span><span class="pill n">supervised</span><span class="pill r">high</span></span></p></div>
@@ -197,12 +197,11 @@ <h2>Backlog <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/tree
 <div class="task"><button class="copy" data-cmd="/start_dev draft/feature/pyautomind/repos-sync-config-stamper.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/feature/pyautomind/repos-sync-config-stamper.md">Teach repos_sync --write to stamp organ config surfaces</a><span class="tags"><span class="pill w">✨ feature</span><span class="pill">pyautomind</span><span class="pill n">hard</span><span class="pill n">supervised</span><span class="pill n">low</span></span></p></div>
 </details>
 <details>
-<summary>bug — 32</summary>
+<summary>bug — 31</summary>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/health_fixes/jax_runtime_and_parity.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/health_fixes/jax_runtime_and_parity.md">Fix release JAX runtime compatibility and likelihood parity</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">health_fixes</span><span class="pill r">too-large</span><span class="pill n">supervised</span><span class="pill r">high</span></span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/health_fixes/jit_visualization_outputs.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/health_fixes/jit_visualization_outputs.md">Fix JIT quick-update visualization output regressions</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">health_fixes</span><span class="pill r">too-large</span><span class="pill n">supervised</span><span class="pill r">high</span></span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/health_fixes/samples_parameter_paths.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/health_fixes/samples_parameter_paths.md">Fix release result/sample parameter-path regressions</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">health_fixes</span><span class="pill r">too-large</span><span class="pill n">supervised</span><span class="pill r">high</span></span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/autofit/jax_011_message_log_partition_tuple_shape.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autofit/jax_011_message_log_partition_tuple_shape.md">jax 0.11 breaks beta/gamma message log_partition under jit…</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">autofit</span><span class="pill n">small</span><span class="pill n">supervised</span><span class="pill n">medium</span></span></p></div>
-<div class="task"><button class="copy" data-cmd="/start_dev draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md">Heart script_timing baselines are orphaned by path moves and filled…</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">pyautoheart</span><span class="pill n">small</span><span class="pill n">supervised</span><span class="pill n">medium</span></span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/autoarray/numba_kernel_shift_axes_swapped.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autoarray/numba_kernel_shift_axes_swapped.md">Numba PSF gathers derive the y/x kernel shifts from the wrong kernel…</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">autoarray</span><span class="pill n">low</span><span class="pill n">supervised</span><span class="pill n">medium</span></span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/autofit/loggaussian_prior_declares_own_support.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autofit/loggaussian_prior_declares_own_support.md"><code>LogGaussianPrior</code> misreports its own support as <code>(-inf, inf)</code></a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">autofit</span><span class="pill n">small</span><span class="pill n">supervised</span><span class="pill n">normal</span></span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/autofit/plot_functions_discard_kwargs.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autofit/plot_functions_discard_kwargs.md"><code>autofit.plot</code> functions accept <code>**kwargs</code> and silently discard them</a><span class="tags"><span class="pill w">🐛 bug</span><span class="pill">autofit</span><span class="pill n">small</span><span class="pill n">supervised</span><span class="pill n">normal</span></span></p></div>