Skip to content

feat: per-script smoke timings as a standing dataset (smoke_timings.json) #264

Description

@Jammy2211

Overview

The smoke runner's per-entry timings exist only as [PASS] <name> — <n>s lines in job logs, recovered by hand-scraping every time someone asks what is slow. Since the smoke-runner delegation (#260#263) all ten workspace runners are thin shims over autohands/run_python.py, so recording per-script timings routinely is one PyAutoHands change, not ten repo sweeps. Phase 2 of the test-performance board arc (PyAutoLabs/PyAutoHeart#163); answers item 4 of the Mind's draft/research/ci/smoke_timing_and_profiling.md with "yes". Mind prompt: active/smoke_timings_dataset.md.

Plan

  • The report machinery (result_collector.RunReport, already mandatory in the PR gate via --report-dir) additionally emits a consolidated smoke_timings.json: one entry per executed script/notebook — {entry, kind, status, seconds, cap_s, exit_code} — plus run metadata (project, env profile, python version).
  • When $GITHUB_STEP_SUMMARY is set, append a compact per-entry timing table (slowest first) so every gate run's timings are one click away with no artifact download.
  • PyAutoHeart's reusable smoke-tests.yml uploads the report dir as a run artifact (if: always(), if-no-files-found: ignore) — the dataset persists the artifact retention window for every gate run across all ten repos (separate small PR on PyAutoHeart).
  • Timing comes from the runner's own measurement, never re-derived; TIMEOUT entries record the cap they hit; skips are never silently timed as 0.
  • Existing report consumers (run_all.py, Heart test_run/script_timing) unaffected.
Detailed implementation plan

Affected Repositories

  • PyAutoHands (primary)
  • PyAutoHeart (one artifact-upload step in the reusable workflow, separate PR)

Branches

  • PyAutoHands: claude/test-performance-dashboard-y3fdy7 (from main @ e510de1)
  • PyAutoHeart: claude/smoke-timings-artifact-y3fdy7 (the phase-0 fix holds the main task branch there)

Implementation Steps

  1. autohands/result_collector.py — RunReport gains per-entry timing capture (reusing the durations it already records for the per-script JSONs) and a write_timings() leg emitting smoke_timings.json alongside the existing report files; step-summary table writer guarded on GITHUB_STEP_SUMMARY.
  2. autohands/run_python.py / run_notebook.py — no interface change; the report path emits the new file whenever --report-dir is passed.
  3. PyAutoHeart/.github/workflows/smoke-tests.yml — upload-artifact step for the report dir.
  4. Tests: entry shape incl. TIMEOUT carrying cap_s; skips absent/marked; step-summary table golden test; existing report consumers untouched (full suite green).

Original Prompt

Click to expand starting prompt

See PyAutoMind/draft/feature/pyautohands/smoke_timings_dataset.md (moved to active/ on issue) — full task text and acceptance criteria there.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions