Skip to content

CI: measure and reduce core test work before extending its timeout #2182

Open

Description

@lmeyerov

Problem

test-core-python runs nearly the full test suite in seven Python versions. The Python 3.14 cell also collects changed-line coverage. The current 8-minute job timeout leaves little margin on ordinary changes, and #2178 proposes raising it to 12 minutes. A timeout increase alone does not reduce the test work or release wait.

Evidence (11 Oct 2026)

Requested work

Profile the core lane with stable per-module/per-test timing and compare repeated runs on matched source/runner conditions. Identify redundant or disproportionately costly cases, fixtures, collection or coverage work, then remove or optimize measured work while preserving the same meaningful version/behavior and changed-line coverage gates. Report total runner-minutes and critical-path wall time, not just a faster shard achieved by adding parallel jobs. Reassess the 8-minute deadline on an exact integration head after the improvement; keep enough measured variance margin.

Polars coverage capacity is separately tracked by #1797. Its issue body predates the current xdist runner; refresh that evidence before choosing additional Polars changes. #2181's route-replay efficiency is separate from these core and Polars jobs.

Acceptance

  • A before/after repeatable per-module timing record identifies actual work removed or optimized.
  • Core correctness, supported Python-version coverage and changed-line coverage behavior stay green.
  • Measured runner-minutes and full workflow critical path improve; timeout value is justified by observed distribution.

Concrete first candidate (11 Oct 2026 hosted log audit)

In #2181 run 38115093218, Python 3.14 core job 114398757516 spent about 83.4 seconds on graphistry/tests/compute/gfql/cypher/test_differential_grammar_parity.py::test_full_repo_lalr_earley_differential, the largest single measured test on that worker. The dedicated cypher-frontend-differential-parity (py3.12) job 114398354515 also ran that exact test and passed (1 passed in 120.61s), finishing before the core job started. The core matrix has seven Python versions; repeated work is plausible, but sevenfold savings and workflow-wall gains have not yet been measured.

The full corpus scans every package .py, so removing core copies needs a retained gate on broad Python edits. Further audit found test-gfql-core already runs the full GFQL directory on Python 3.12 (with coverage) and 3.14 whenever generic core runs: exhaustive current-condition evaluation covered 2,048 path-filter/event states. #2183 therefore excludes the file only from the seven generic core cells and leaves the dedicated gate trigger untouched. Constructed grammar invariant tests remain in all core Python versions; the full-repository sweep no longer runs in generic core 3.8–3.11/3.13. Keep #2181's reviewed route-replay head unchanged and do not reduce timeouts before measured integration evidence.

#2183 hosted result (11 Oct 2026)

#2183 is now OPEN/non-draft at f86d2766d797f85704dc5f7bdab473badeae3e6e, with two quiet review waves and exact-head run 38123237023 complete: 79 successful checks and two expected path-gated skips. The baseline #2181 core logs directly attribute 19.86 runner-minutes to the seven repeated full-corpus sweep calls. The #2183 core matrix totaled 32.58 runner-minutes versus 40.85 (master schedule) and 41.23 (#2181 PR) in prior runs; the observed 20–21% group reduction is affected by runner variance. The workflow wall was 17m46s, but its path-triggered work differs from those baselines, so this is not a proven whole-workflow wall-time improvement.

The hosted core 3.8/3.14 logs show the sweep excluded and constructed grammar invariants retained; GFQL core 3.12/3.14 logs show the sweep passed. GFQL 3.12 per-file coverage audit reports cypher/parser.py 90.29% against 88.35% baseline. Changed-line coverage passed, but with 0 eligible changed package lines on this workflow-only PR; the retained sweep and per-file audit supply the relevant parser evidence. The version-scope trade-off and maintainer-only merge boundary are in the PR body. If #2181 lands first, rerun exact integrated CI before #2183 merge. The longer-term issue remains open for measured core work reduction beyond this one duplicate and for timeout-margin reassessment.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions