diff --git a/draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md b/complete/2026/08/script-timing-baselines-fix.md similarity index 55% rename from draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md rename to complete/2026/08/script-timing-baselines-fix.md index 6ad122d7..64e8036e 100644 --- a/draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md +++ b/complete/2026/08/script-timing-baselines-fix.md @@ -1,3 +1,55 @@ +# script_timing baselines made real — run-identity dedup + rename-aware slugs + +PyAutoHeart#166 → `ee915f3`, closing PyAutoHeart#165, merged 2026-08-24 on +branch `claude/test-performance-dashboard-y3fdy7` (restarted from main after +the phase-1 board merge). Phase 0 of the test-performance board arc +(`complete/2026/08/test-performance-board.md`) — the per-script trend surface +reads what this fixes. + +## What shipped + +- **The "every history is one value repeated 7×" defect was not seeding — it + was re-ingestion.** `run()` re-read `run_logs/latest` on every tick and + re-appended the same observation until the window filled with copies of one + run. History entries now carry `{duration_s, run_id, ts}` (run_id = the + resolved timestamped run dir behind the `latest` symlink); a re-tick on the + same run replaces the newest entry, so windows only grow across distinct + runs. Legacy bare-float histories still read; a legacy window of + all-identical values collapses to the single observation it provably was. +- **Classification floor**: the yellow/red ratio fires only against ≥3 + distinct-run samples (`MIN_BASELINE_RUNS`); below that a script counts as + `building_count` instead of being judged against a fake-stable median. + `red/yellow/green_count` keep their exact meaning; `dashboard.py` untouched. +- **Rename-aware slugs, loud orphans**: history files untouched by a scan are + orphans; a new slug with no history adopts an orphan on an unambiguous + (workspace, script-name) match — the #216 restructure scenario now heals + itself (`migrated_count`). Ambiguous/unmatched orphans are reported + (`orphaned_count`), never deleted, never silent. +- Histories written via `atomic_write_json` (was a bare `write_text`). +- Tests 573 → 583; `test_script_timing.py` 6 → 16. + +## Key traps / findings + +- **Four existing tests had encoded the bug itself** — they re-ran `run()` on + the same results dir as a stand-in for repeated runs, which is exactly the + defect. When a test suite's fixture shape mirrors a bug, the tests pass for + the wrong reason; the fixtures now create one dir per run. +- **Legacy empty run_ids each count as a distinct run** in the floor — history + that predates provenance is real accumulation once the identical-window + collapse has run; treating it as one run would have zeroed every surviving + baseline. +- Orphan reporting is scan-relative: pointing Heart at a partial results dir + surfaces other workspaces' histories as "orphaned" (reporting only — + migration matching is workspace-scoped, so nothing can be misattached). + +## Follow-ups + +- `unit_test_timing` / `workspace_testmode_timing` keep their own history + mechanics — if they share the re-ingestion pattern, the same dedup applies; + not audited here (out of the prompt's scope). + +## Original prompt + # Heart script_timing baselines are orphaned by path moves and filled with one repeated value Type: bug @@ -9,6 +61,7 @@ Autonomy: supervised Priority: medium Status: formalised Filed: 2026-08-04 (backfilled from git) +Issued: 2026-08-24 Two independent defects in `PyAutoHeart/heart/checks/script_timing.py`, both found while diagnosing the jax_grad smoke timeouts (PyAutoHands#226). Neither is fixed by diff --git a/complete/2026/08/smoke-timings-dataset.md b/complete/2026/08/smoke-timings-dataset.md new file mode 100644 index 00000000..9d8472af --- /dev/null +++ b/complete/2026/08/smoke-timings-dataset.md @@ -0,0 +1,125 @@ +# Per-script smoke timings are a standing dataset — one runner change, ten repos inherit + +PyAutoHands#265 → `d2a22f4` (closing PyAutoHands#264) + PyAutoHeart#167 → +`3df42b5`, merged 2026-08-24. Phase 2 of the test-performance board arc +(`complete/2026/08/test-performance-board.md`); answers item 4 of +`draft/research/ci/smoke_timing_and_profiling.md` — "should the runner record +per-script timings routinely?" — with **yes, shipped**. The smoke-runner +delegation (#260–#263) is what made this one change instead of ten repo +sweeps. + +## What shipped + +- **`smoke_timings.json`** (schema `smoke_timings/1`) emitted from + `RunReport.write()` — one call site covers `run_python.py`, `run.py` and + `generate.py`, so every delegated gate inherits it with zero per-repo + edits. Per entry: `{entry, kind, status, seconds, cap_s, exit_code}` — + `seconds` is the runner's own measured duration (the number the + `[PASS] — s` line prints), TIMEOUT entries carry the cap they hit, + skips are `seconds: null`, never a fabricated 0. One merged file per + report directory, rows keyed on entry path (script + notebook legs both + survive; a re-run replaces its own rows; `legs` records contributors). +- **Step summary**: with `$GITHUB_STEP_SUMMARY` set, a slowest-first timing + table per leg — every gate run's timings one click away, no artifact + download. +- **`ScriptResult` gained `cap_seconds`/`exit_code`** at all 8 execution + sites — deliberately absent from `to_dict()` so the per-run JSONs Heart's + `script_timing`/`test_run` and `aggregate_results` read stay + byte-compatible (pinned by a test); `aggregate_results` skips the sidecar + by name so the mega-run surface stays clean. +- **PyAutoHeart `smoke-tests.yml`** uploads the report dir as + `smoke-timings-` (`if: always()`, + `if-no-files-found: ignore`, no `retention-days` — full default artifact + retention). The path is a glob (`test-results/` + `**/smoke_timings.json`) + because the reusable workflow never passes `--report-dir` — each + workspace's `run_smoke.py` does. +- Drive-by fix: `run.py` never passed `env_profile`, so every notebook + report claimed `unknown` — threaded through, negative-tested. +- Tests: PyAutoHands 363 → 382 passed (14 pre-existing environmental + failures unchanged — missing `ipynb-py-convert`/`pngquant` locally, + identical set on main); PyAutoHeart 573 → 576. + +## Key traps / findings + +- **The notebook report leg is `run.py`, not `run_notebook.py`** — the + latter is the single-notebook kernel-cwd shim `build_util` shells out to + and has no report path. Emit from `RunReport.write()` and every leg is + covered. +- **A fixed per-leg filename would clobber across directories in the + mega-run** — one report dir hosts many `(project, directory, run_type)` + invocations; hence the merged file keyed on entry path. Known limit: two + workspaces sharing a relative path in one mega-run dir collapse to one + row (documented in the docstring; that dir is not a consumer). +- **Keep new fields out of `to_dict()` until every consumer is audited** — + the per-run JSON shape is load-bearing for three downstream readers; + extending the dataclass without extending the serialization is the + compatible move, pinned by a byte-compat test. +- `aggregate_results` globs `**/*.json` — any new sidecar in the report dir + becomes a phantom run unless excluded by name. + +## Follow-ups + +- The Heart board ingesting these artifacts into per-script rows with + STALL/SLOW verdicts (reuse `retime.py`'s vocabulary; bimodality + first-class) once a few weeks of data exist — the deferred phase 3 named + in the board record. + +## Original prompt + +# Per-script smoke timings as a standing dataset — one runner change, ten repos inherit + +Type: feature +Target: pyautohands +Repos: +- @PyAutoHands +- @PyAutoHeart +Difficulty: medium +Autonomy: supervised +Priority: high +Status: formalised +Filed: 2026-08-24 +Issued: 2026-08-24 + +Phase 2 of the test-performance board +([`../../../docs/pyautoheart/test_performance_board_assessment.md`](../docs/pyautoheart/test_performance_board_assessment.md); +phase 1 shipped 2026-08-24 as PyAutoHeart#164 + PyAutoBrain#261). Today the +smoke runner's per-entry timings exist only as `[PASS] — s` lines in +job logs, recovered by hand-scraping; the 2026-08-23 slow-vs-stall audit and +the jax_grad budget work both had to rebuild their datasets that way. Since +the smoke-runner delegation (PyAutoHands#260–#263) all ten workspace runners +are thin shims over `autohands/run_python.py` — **so recording per-script +timings routinely is now one PyAutoHands change, not ten repo sweeps.** This +answers item 4 of `draft/research/ci/smoke_timing_and_profiling.md` +("should the runner record per-script timings routinely?") with yes. + +## Task + +1. **PyAutoHands** — the report machinery (`result_collector.RunReport`, + already mandatory in the PR gate via `--report-dir`) additionally emits a + consolidated `smoke_timings.json` in the report dir: one entry per + script/notebook — `{entry, kind, status, seconds, cap_s (the cap in force + from build_util.timeout_for), exit_code}` — plus run metadata (project, + env profile, python version). When `$GITHUB_STEP_SUMMARY` is set, append a + compact per-entry timing table (slowest first) so every smoke run's + timings are one click away in the Actions UI with no artifact download. +2. **PyAutoHeart** — the reusable `smoke-tests.yml` uploads the report dir as + a run artifact (`smoke-timings-`, `if: always()`, + `if-no-files-found: ignore`), so the dataset persists the full artifact + retention window for every gate run across all ten repos at once. +3. Timing must come from the runner's own measurement (the same clock the + `[PASS] — s` line prints), never re-derived; a TIMEOUT entry records + the cap it hit. + +## Acceptance + +- A PR-gate smoke run on any workspace produces `smoke_timings.json` with one + timed entry per executed script/notebook and a step-summary table, with no + per-repo changes. +- TIMEOUT entries carry `cap_s`; skipped entries are absent or explicitly + marked, never silently timed as 0. +- Existing report consumers (`run_all.py`, Heart `test_run`/`script_timing`) + are unaffected. + +Follow-up (not this task): the Heart board ingesting these artifacts into +per-script rows with STALL/SLOW verdicts (reuse `retime.py`'s vocabulary) +once a few weeks of data exist. diff --git a/complete/index.md b/complete/index.md index e5032cee..af18ac68 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1095 records across 7 buckets. +1097 records across 7 buckets. ## Highlights @@ -187,6 +187,7 @@ _(curate hard-won records here — survives regeneration.)_ - [samplers-surface-autolens-tiers](2026/08/samplers-surface-autolens-tiers.md) — the samplers faculty's SamplerSurface now scans the findings - [save-json-numpy-scalar-typeerror](2026/08/save-json-numpy-scalar-typeerror.md) — Adds `NumpyEncoder` in `autofit/tools/util.py` (`np.ndarray` -> - [script-size-guard-git-based](2026/08/script-size-guard-git-based.md) — replaced the rotting `.script_sizes.json` snapshot with a git-diff truncation +- [script-timing-baselines-fix](2026/08/script-timing-baselines-fix.md) - [script-title-underline-off-by-one](2026/08/script-title-underline-off-by-one.md) - [searches-readme-dashboard](2026/08/searches-readme-dashboard.md) - [should-simulate-capped-branch-reuse](2026/08/should-simulate-capped-branch-reuse.md) — small, single-repo follow-up of the closed PyAutoNerves#153 @@ -196,6 +197,7 @@ _(curate hard-won records here — survives regeneration.)_ - [small-datasets-rmtree-committed-data](2026/08/small-datasets-rmtree-committed-data.md) - [smoke-runner-delegation](2026/08/smoke-runner-delegation.md) - [smoke-runner-jupyter-guard](2026/08/smoke-runner-jupyter-guard.md) +- [smoke-timings-dataset](2026/08/smoke-timings-dataset.md) - [spawn-autonomy-log-generated](2026/08/spawn-autonomy-log-generated.md) - [spawn-drift-self-heal](2026/08/spawn-drift-self-heal.md) - [spawn-empty-body-privacy-fix](2026/08/spawn-empty-body-privacy-fix.md) diff --git a/dashboard.html b/dashboard.html index 1bf29b17..c4d18319 100644 --- a/dashboard.html +++ b/dashboard.html @@ -123,7 +123,7 @@
📋

PyAutoMindDashboard

Intent. Priority. Flow.

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

-
  • 1In flight
  • 3Parked
  • 6Planned
  • 155Backlog
+
  • 1In flight
  • 3Parked
  • 6Planned
  • 154Backlog

markdown version

Start here

Highest priority (filed as high) — showing 12 of 17

@@ -163,7 +163,7 @@

Planned

latent-nan-guard-honest-run — planned 2026-07-22

Backlog markdown version

-

155 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

+

154 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

feature — 29

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix caching✨ featureautoarraymediumsupervisedhigh

@@ -197,12 +197,11 @@

Backlog

Teach repos_sync --write to stamp organ config surfaces✨ featurepyautomindhardsupervisedlow

-bug — 32 +bug — 31

Fix release JAX runtime compatibility and likelihood parity🐛 bughealth_fixestoo-largesupervisedhigh

Fix JIT quick-update visualization output regressions🐛 bughealth_fixestoo-largesupervisedhigh

Fix release result/sample parameter-path regressions🐛 bughealth_fixestoo-largesupervisedhigh

jax 0.11 breaks beta/gamma message log_partition under jit…🐛 bugautofitsmallsupervisedmedium

-

Heart script_timing baselines are orphaned by path moves and filled…🐛 bugpyautoheartsmallsupervisedmedium

Numba PSF gathers derive the y/x kernel shifts from the wrong kernel…🐛 bugautoarraylowsupervisedmedium

LogGaussianPrior misreports its own support as (-inf, inf)🐛 bugautofitsmallsupervisednormal

autofit.plot functions accept **kwargs and silently discard them🐛 bugautofitsmallsupervisednormal

diff --git a/dashboard.md b/dashboard.md index 26e04989..c653c2fd 100644 --- a/dashboard.md +++ b/dashboard.md @@ -11,7 +11,7 @@ Every task the Mind is holding, on one page: what is in flight, what is parked, | [In flight](#in-flight) (`active/`) | 1 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 6 | -| [Backlog](#backlog) (`draft/`) | 155 | +| [Backlog](#backlog) (`draft/`) | 154 | ## Start here @@ -235,7 +235,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**155** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **25** of them belong to an epic and are listed only under [Epics](#epics) below. +**154** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **25** of them belong to an epic and are listed only under [Epics](#epics) below.
feature — 29 @@ -475,7 +475,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-bug — 32 +bug — 31
📋 Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high @@ -509,14 +509,6 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-
📋 Heart script_timing baselines are orphaned by path moves and filled… — pyautoheart · small · supervised · medium - -``` -/start_dev draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md -``` - -
-
📋 Numba PSF gathers derive the y/x kernel shifts from the wrong kernel… — autoarray · low · supervised · medium ```