Skip to content

ci: a merge burst queues every superseded commit's gates (220 runs from 8 merges); add dispatch-safe concurrency groups #869

Open

Description

@gHashTag

What happened

Between 18:38:47Z and 18:42:09Z on 2026-10-04, eight PRs merged into main (#842, #864, #861, #857, #863, #853, #275, #510). Each merge commit started every push workflow whose paths matched. Nothing superseded anything.

Measured at 19:06–19:10Z through the Actions API:

count
push runs on main created by the burst 220 across 66 workflows
of those, still queued 111
queued runs in the repo, all branches 194
wait before a job got a runner (e.g. run 37225295138: created 18:39:50Z, job started 19:05:10Z) ~25 min

Concurrency is billed per account, so this queue also delays gHashTag/t27 and every dispatch in this repo, including the chipdb A/B runs for #846 and #862.

Why

workflows on main count
total 122
triggered by push to main 104
... with no concurrency: group 102
... of those, also workflow_dispatch 85
... also schedule 4
... also workflow_call 0

Estimate

With one group per workflow and ref, GitHub keeps at most one run active and one pending per group. A newer run replaces the pending one. The burst lasted 3.5 min and the first runs waited ~25 min for a runner, so each workflow would have kept two runs at most: the first commit's and the last. That is ≤ 97 of 220 runs (123 fewer, 56%). This is an estimate from the counts above, not a measurement.

Prior art: #647, and why it should not be rebased as is

#647 (opened 2026-08-19) adds group: ${{ github.workflow }}-${{ github.ref }} and cancel-in-progress: true to 78 push workflows. Today it is 766 commits behind main and conflicts in two files. The bee review on 2026-10-02 suggested regenerating it. Its design has two problems on today's tree, because 85 of the 102 workflows can also be dispatched:

  1. A push to main would cancel a manual workflow_dispatch run in progress on main. Both share the group <workflow>-refs/heads/main.
  2. A second dispatch would cancel the first, even with different inputs. Seed sweeps and A/B pairs are dispatched in batches.

Proposed

Add one block per workflow, at top level:

concurrency:
  group: ${{ github.workflow_ref }}-${{ github.event_name == 'workflow_dispatch' && github.run_id || github.event_name }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

Edited: the first draft keyed on github.workflow, which is the name:. Two files share a name: conformance-golden-selftests.yml and conformance-selftest.yml are both "Conformance golden self-tests", so they would have shared a group. github.workflow_ref is the file plus the ref, and it cannot collide. #647's key has the same flaw. The fix is in #870.

  • push: one run active and one pending per workflow and branch. The run that is already going finishes, so main still gets a result for the commit that started.
  • pull_request: a new commit cancels the old commit's run, whose result no longer matters.
  • workflow_dispatch: a group per run, so it is never cancelled and never cancels anything.
  • schedule: a new tick replaces only an older tick that is still pending.

Skip any workflow that already has a group (2 on main) and any reusable workflow (workflow_call, 0 today). A caller and its callee share github.workflow, so a group inside the callee would deadlock against the caller.

Refs #851 #647 #865

🤖 Generated with Claude Code

Activity

  1. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    #870 now has two commits, and its own runs show the behaviour this issue asks for.

    1. b1894fb gives the 102 ungrouped push workflows the dispatch-safe group (2/104 → 104/104).
    2. 066bfa2 adds tools/check_concurrency_groups.py and concurrency-gate.yml, so the next push workflow cannot arrive without a group. It fails on UNGROUPED, BAD_REUSABLE (a workflow_call file with its own group) and SHARED_KEY. The checker is registered in gates-can-fail, and MIN_TESTABLE goes from 11 to 12.

    Measured locally before the push:

    check result
    the branch 105 push-to-main workflows, 105 grouped, rc 0
    main at 57de4a1 (negative control) 102 UNGROUPED, rc 1
    agreement with the line-based reader used for the numbers above 122/122 files on main, 122/122 on the branch
    planted cases 57/57
    mutants of the checker caught by its test 29/29, plus one listed as equivalent (* crossing /; only main is asked about)
    gates-can-fail case clean rc 0, planted rc 1

    Seen on #870 (read 19:30–19:33Z):

    • None of the 60 runs on b1894fb ended in a workflow-file or startup error, so GitHub accepts both expressions. lut-report.yml ran to success (37227380125).
    • Pushing 066bfa2 cancelled all 37 of b1894fb's unfinished pull_request runs.
    • b1894fb's 16 push runs were not cancelled. The 10 push runs that 066bfa2 started on those same workflows sit as pending behind them, which is one active and one pending per group.

    Not measured yet:

    • the gate's own CI run (queued at 19:33Z);
    • actionlint;
    • a real burst on main with the change in place.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions