Skip to content

Governance incident: merged PRs with red/zero-job checks; reconcile required contexts and bypass actors #990

New issue
New issue
Open
Open
Governance incident: merged PRs with red/zero-job checks; reconcile required contexts and bypass actors#990
Labels
cicdCI/CD: workflows, actions, lockfiles, pins, runners, release gatesgovernancePolicy, rulesets, standards, compliance, and their enforcementrefactorRestructuring that preserves observable behaviour

Description

@hyperpolymath
hyperpolymath
opened on Oct 31, 2026

Observed facts (2026-10-31 UTC)

  • PR #988 merged after the new CICD Squabbler workflow reported startup_failure on both the cross-owner reusable and the same-owner composite action attempts. The affected Actions runs had zero jobs, so they cannot be counted as checks that ran and passed.
  • PR #989 merged at 11:17:25 UTC, although its final revision's Governance workflow reported failure. Its CICD Squabbler run did pass (job executed, pinned digest and attestation verified, chains completed 37/37 probed repositories, 290 edges, 0 blocking, 17 nonblocking).
  • The repeated Governance failure is the Allowlist Preflight job. It finds five metadatastician/standards/.github/workflows/* callers uncovered by the allowlist pinned to standards commit 333b06a01db1874ffedb2de1bfabb195640927c7. Current standards 1725bebd3773ea25627479f813283abf1c497a85 includes the missing pattern, but temporarily switching the Governance reusable to that revision produced startup_failure before jobs started. That pin switch was reverted before fix: restore a genuinely running, verified CICD Squabbler check #989's merge; main still carries the last runnable Governance pin, with visible failure.
  • GitHub connector GET /repos/hyperpolymath/hypatia/branches/main/protection returned 403 Resource not accessible by integration. This is a tooling/credential visibility gap: we cannot conclude which contexts are actually required, or which role/app bypass triggered the merges.
  • An attempt to convert fix: restore a genuinely running, verified CICD Squabbler check #989 to draft was denied by GitHub GraphQL permission; an attempt to close it occurred after it was already merged.

Urgent investigation

  1. With a principal that has Administration:read, read the repository rulesets (including inherited), classic branch protection, required check names and source app IDs, bypass actors/modes, and merge actor audit for both merges. Distinguish GitHub allowed merge, required gate was satisfied, and every required check actually ran and passed.
  2. Trace which actor merged ci: canary direct, pinned cicd-squabbler chains gate #988 and fix: restore a genuinely running, verified CICD Squabbler check #989 and whether an always bypass or auto-merge tool ignores failing contexts; restrict that path to explicit exceptional approval or remove it only after verifying an owner escape hatch and nonphantom producible contexts. Don't create a deadlocked rule.
  3. Use standards' scripts/check-required-contexts.sh and scripts/apply-branch-gates.sh with dry-run and --require-green N before changing rulesets. Derive check names from actually emitted jobs on PRs, not manually guessed names.
  4. Determine the effective Actions selected-workflow policy that blocked the newer standards Governance revision. Allow an immutable reviewed revision or use a guarded local reusable copy; don't convert the existing failing allowlist check to advisory.
  5. Re-run PR canary and default-branch push after remediation. A startup_failure or missing check must be NOT EVALUATED, never green.
  6. Only then turn squabbler into an estate-wide required gate, and only after an adoption census.

Safety boundary

No relaxation of verification, deleting bad history, bypassing required gates, or lifting the gitbot-fleet quarantine as a workaround. Squabbler is a standalone tool, not a Hypatia subcomponent.

Acceptance: repeated test PRs cannot merge when any truly required job fails, is skipped improperly, or has a zero-job startup failure; an owner can recover from legitimately deadlocked rules without granting unrestricted bypass to automata.

Activity

  1. github-actions
    added
    cicdCI/CD: workflows, actions, lockfiles, pins, runners, release gates
    governancePolicy, rulesets, standards, compliance, and their enforcement
    refactorRestructuring that preserves observable behaviour
    on Oct 31, 2026

  2. coderabbitai commented on Oct 31, 2026

    @coderabbitai
    coderabbitaiBot
    on Oct 31, 2026 – with coderabbitai
    Contributor
    🔗 Related PRs

    #988 - ci: canary the independent cicd-squabbler gate [open]
    #958 - fix(ci): re-point standards callers to metadatastician/standards [merged]

    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan
    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
  auto_enrich:
    enabled: false

    💬 Have feedback or questions? Drop into our discord!

  3. hyperpolymath commented on Oct 31, 2026

    @hyperpolymath
    hyperpolymath
    on Oct 31, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    Confirmed repository ruleset findings — 2026-10-31

    The repository rulesets API (read-only) reports four active repository-level rulesets:

    Ruleset (ID) Required contexts
    ABI-Codegen-Drift (23818931) abi-codegen-drift; zig build test (FFI + wire contract)
    Escript-Soundness-Required (24251055) Escript packaging soundness
    Secret-Scan-Floor (24309328) scan / gitleaks
    Branch-Floor (23869692) no status checks; deletion and non-fast-forward rules

    All four declare RepositoryRole, actor_id 5 (repository administrator), bypass_mode: always. Neither Governance nor CICD Squabbler appears among the listed required contexts. This explains why those checks could remain red while GitHub still permitted a merge, without establishing who actually merged.

    Classic branch protection remains unreadable via this GitHub integration (403), so this is a repository-ruleset finding, not a complete effective-policy inventory.

    Next: with Administration-capable GitHub access, read classic and inherited controls, audit actual merge actors, and then use standards' checked-in scripts/apply-branch-gates.sh with dry-run and --require-green N to derive nonphantom required contexts from observed job names. Evaluate and narrow unconditional administrator bypass only with a tested emergency recovery path. Never require a check during startup_failure or an unresolved red state. The current connected app has not demonstrated Administration:write for making these changes.

  4. hyperpolymath commented on Oct 31, 2026

    @hyperpolymath
    hyperpolymath
    on Oct 31, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    Additional verified GitHub records

    • GitHub REST pulls/988: merged_by.login=hyperpolymath, auto_merge.enabled_by.login=hyperpolymath, merged at 2026-10-31 11:09:39 UTC.
    • pulls/989: same account for merged_by and auto_merge.enabled_by, merged at 2026-10-31 11:17:25 UTC.
    • This establishes account attribution, not whether a human or automation used that account's credential. Investigate who enabled auto-merge, which actor/process submitted the branch/merge, and whether bypass was invoked.
    • On merged main SHA 3a60ee80bd5daea82d2bfc7470f2930ff00b8a45, CICD Squabbler's push run 41196238428 succeeded. It is now operational on both a PR and a default-branch push. Governance push run 41196238803 failed; Mirror to Git Forges push run 41196238787 also failed and should be triaged separately.

    Do not promote Governance to a required check while its existing allowlist preflight is red, or enable an unbounded auto-merge actor to bypass that requirement.

  5. hyperpolymath commented on Oct 31, 2026

    @hyperpolymath
    hyperpolymath
    on Oct 31, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    Exact PR timeline confirms unsolicited ready/auto-merge transitions

    GitHub's issue timeline gives precise timestamps and account attribution:

    PR Event Actor UTC
    #988 auto_merge_enabled hyperpolymath 11:07:54
    #988 merged hyperpolymath 11:09:39
    #989 ready_for_review hyperpolymath 11:16:46
    #989 auto_merge_enabled hyperpolymath 11:16:49
    #989 merged hyperpolymath 11:17:25

    The assistant using the connected GitHub tools did not issue an enable_auto_merge, mark_pull_request_ready_for_review, or merge_pull_request command on either PR. It attempted to convert #989 back to draft, but GraphQL refused permission, and later attempted to close it only after merge. Thus an additional actor/path with the hyperpolymath identity enabled auto-merge and marked #989 ready. The account identity alone cannot distinguish user activity from automation acting via user-scoped credentials.

    This violates the stated auto-merge classifier's policy for human-authored workflow changes: docs/operations/estate-automerge.adoc says new workflow files or modified triggers/permissions and human-authored PRs are never_auto, and scripts/sweeps/estate-pr-automerge.sh ordinarily limits auto-merge decisions to dependency-bot routine changes. That does not prove either module caused the violation; find the unguarded auto-merge enabler, including external apps/private-farm credentials and any CLI automation.

    Containment: until identified, temporarily disable native auto-merge for this repo or remove auto-merge authority from user-scoped automation, retaining manual exceptional merge by the owner. Do not remove a working required rule or bypass the known Governance failure.

  6. hyperpolymath commented on Oct 31, 2026

    @hyperpolymath
    hyperpolymath
    on Oct 31, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    Autonomy/actuator code trace — do not assign blame prematurely

    Read hyperpolymath/.git-private-farm/scripts/actuate.sh: its decide_action refuses change_level=meta, vetoes, any safety != arm_auto, and any CI input except green. Its write path, when FARM_ACTUATE_WRITE=true, re-fetches the live head and executes gh pr checks --required --json state, insisting the list be nonempty and all SUCCESS. This is a meaningful safeguard in code, not proof of running configuration or that this specific path actuated either merge. Note especially it checks only required checks; Governance and Squabbler are currently not required in readable repo rulesets.

    Read hyperpolymath/hypatia/scripts/sweeps/estate-pr-automerge.sh and docs/operations/estate-automerge.adoc: the script's normal eligibility restricts auto-merge to dependency bot routine changes, treating new workflows/changed triggers as owner-reviewed meta. PR #988/#989 introduced/edited workflows, so their auto-merge events are inconsistent with the stated safe-eligibility contract. The script's actuator additionally uses gh pr merge --auto, but no evidence it enabled auto-merge for these two PRs.

    Read .git-private-farm/scripts/merge-label-tooling.sh: it defaults to --admin, an important estate-wide escape/bypass concern, but it targets a particular automated/label-tooling branch and is not evidenced as responsible for #988/#989.

    The actual next proof is an authenticated audit of who invoked the ready_for_review and auto_merge_enabled transitions on these two new PRs. Timeline actor hyperpolymath alone cannot identify a process. Check authorising apps, token types, audit/session logs and outside-ChatGPT actors; narrow permissions accordingly.

  7. hyperpolymath commented on Oct 31, 2026

    @hyperpolymath
    hyperpolymath
    on Oct 31, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    Cross-repository reproduction and canonical non-destructive remedy (31 Oct, 18:46 UTC)

    The transfer/selected-actions problem is not isolated to Hypatia. On hyperpolymath/cicd-squabbler PR #146, the Governance allowlist preflight rejected seven metadatastician/standards/.github/workflows/*@8e48b1ca... callers, and the .github/workflows/standards-pipeline.yml reusable caller finished with zero jobs. The standalone Squabble Consumer Action and its new read-only census regression, however, passed. This is consistent with incomplete cross-owner standards Actions access, though actual selected-actions settings still require Administrator read access to confirm.

    The canonical standards transfer playbook explicitly warns that GitHub Actions reusable resolution does not follow repository redirects and can generate startup_failure. The existing widen-actions-allowlist.sh is dry-run by default, uses union-not-replace for patterns_allowed, preserves publisher flags and Actions posture, and verifies read-back after an --apply. The current canonical config/settings/actions-allowlist.json includes metadatastician/standards/.github/workflows/*@*.

    Recommended controlled sequence with an Administration-capable gh identity:

    cd standards
bash scripts/widen-actions-allowlist.sh hyperpolymath/hypatia
bash scripts/widen-actions-allowlist.sh hyperpolymath/cicd-squabbler
# Review proposed union. Use --apply only after reviewing exact additions
# and confirming the effective org/enterprise settings.

    Do not blindly run a broad set-allowed-actions.sh rollout: its policy posture transition is more invasive than this targeted monotone union, and its own comments document the historical zero-job outage when the payload fails after a posture change.

    Still unknown: whether the effective owner/org Actions policy, the called workflow YAML validation, or another administration restriction is responsible for a specific new-SHA startup_failure. Confirm by read-only settings/API inspection and a canary using a corrected immutable standards pin. Do not claim success simply because the GET or a checkout can follow redirects.

    Also, hyperpolymath/hypatia main currently has a successful, verified squabbler job but no required-context enforcement for it. Administrator-role bypass_mode=always remains active in the readable branch rulesets.

  8. hyperpolymath
    added a commit that references this issue on Oct 31, 2026

    feat: evidence-based read-only estate gate census (#146)

    eaeb428

  9. hyperpolymath commented on Nov 1, 2026

    @hyperpolymath
    hyperpolymath
    on Nov 1, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    Update — 2026-11-01: actual Actions-policy access, restored Governance execution

    The user's Windows/Desktop Commander session established an authenticated gh administrator-capable path. Read-only GET repos/{owner}/{repo}/actions/permissions and selected-actions confirmed that both Hypatia and cicd-squabbler have allowed_actions=selected, sha_pinning_required=true and the necessary metadatastician/standards/.github/workflows/*@* pattern. No live policy widening was performed (it was already allowed).

    Hypatia main at f6f8864e772003a2ee66b18b3d8261a5484a6d2a uses standards pin 1725beb...; Governance push run 41249464522 started 21 jobs, with exactly one failed job: Allowlist Preflight 124679746203. It now reports only google/clusterfuzzlite/actions/build_fuzzers@v1 and .../run_fuzzers@v1 as missing. All other jobs succeeded or intentionally skipped. This is the already-tracked metadatastician/standards#1380: the tree allowlist lacks two narrow subpath entries which config/settings/actions-allowlist.json already grants.

    For diagnostics, a GitHub-signed one-line backport of the old allowed-workflow pattern was authored at metadatastician/standards@8a9ba7c5f878da4e9d2f7d06f7500294b486a563 on isolated compat/governance-allowlist-backport-20261101; this backport is now superseded and not used by Hypatia, because the current standards pin is genuinely runnable. Do not roll back to it.

    A new fix branch metadatastician/standards:fix/clusterfuzzlite-allowlist-parity-20261101 was created from current main; its two-file allowlist-plus-regression fix was assembled as unlinked Git objects, but GitHub marked the candidate commit 1c523a6... unsigned. Per signature policy, the branch ref was deliberately NOT updated and no unsigned commit was pushed or merged. Completing this upstream fix requires the established GraphQL createCommitOnBranch GitHub-signed mutation or an unlocked signing identity. The remote desktop stopped responding to commands before that mutation could be submitted.

    No branch-protection rulesets were modified and the fleet quarantine remains unchanged. Do not treat successful PR merges as proof that Governance is enforced; #990's bypass and required-context audit still stands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    cicdCI/CD: workflows, actions, lockfiles, pins, runners, release gatesgovernancePolicy, rulesets, standards, compliance, and their enforcementrefactorRestructuring that preserves observable behaviour

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions