Repository navigation
feat: evidence-based read-only estate gate census #146
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
f179523
3f2f995
93ff147
c3dd740
267f6ed
22d2be2
6c92b40
14c0f26
d579ed4
20b4baf
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| // SPDX-License-Identifier: CC-BY-SA-4.0 | ||
| = Read-only estate gate census | ||
|
|
||
| The gate census inventories named Squabbler workflow files and probes the | ||
| corresponding GitHub Actions run and readable repository rulesets. It is | ||
| standalone: it does not call Hypatia and does not mutate GitHub repositories. | ||
|
|
||
| == Running | ||
|
|
||
| [source,shell] | ||
| ---- | ||
| bash scripts/estate-gate-census.sh --repo example/project | ||
| bash scripts/estate-gate-census.sh --owner example --output /tmp/squabble-census.jsonl | ||
| bash tests/estate_gate_census_test.sh | ||
| ---- | ||
|
|
||
| The owner form uses paginated `gh repo list --limit 10000`, excluding | ||
| archived repositories and forks. Provide a `gh` identity with read access | ||
| to the repositories, workflow runs and rulesets, plus `jq`. | ||
|
|
||
| == Reading results | ||
|
|
||
| Each output line is JSON. `filename_absent` means no workflow *filename* | ||
| matches Squabble; it **does not** establish that no embedded integration | ||
| exists in an unrelated workflow file. `named_workflow_found` establishes | ||
| only that a candidate workflow exists. `role=action_selftest` distinguishes | ||
| Squabble installer/consumer tests from `role=gate_candidate`; neither | ||
| role alone establishes enforcement. `run.status` is the latest | ||
| default-branch run status; `startup_failure` with `zero_jobs` is explicitly | ||
| not green. A successful run is not evidence that its job is required. | ||
|
|
||
| `policy.rulesets` distinguishes `required_by_ruleset`, | ||
| `not_matched_in_readable_first_page` and `unknown`. Required-check comparison uses | ||
| the **actual emitted job name**, not the guessed name of a workflow file. | ||
| `policy.bypass` contains the effective-looking bypass actors from the | ||
| matching repository ruleset. Classic branch protection is not inspected | ||
| and is always marked `not_inspected`; an apparent absence from readable | ||
| rulesets must not be promoted into an assertion of absent enforcement. | ||
| GitHub access errors always produce an unknown state rather than a pass. | ||
|
|
||
| == Known limits | ||
|
|
||
| This is a bounded inventory instrument, not an authorisation to amend | ||
| branch protections, grant workflow access, merge pull requests or enable | ||
| autonomous repairs. It checks named workflow files rather than searching | ||
| every workflow body, uses the first page of each Actions job/ruleset query, | ||
| and inspects default-branch runs rather than all pull-request events. A | ||
| full first page of rulesets or a missing emitted job name is not a | ||
| verified absence of required checks. | ||
| For an estate-wide enforcement decision, follow up with full pagination, | ||
| classic-branch-protection reads using Administration:read, PR-event | ||
| producibility checks, check-source identity and a separate human-reviewed | ||
| ruleset diff. No filename or status count substitutes for those checks. | ||
|
|
||
| The command exits nonzero if it cannot enumerate a requested owner; | ||
| per-repository access failure is retained as an `unknown` row to allow | ||
| the inventory to proceed and show gaps explicitly. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| #!/usr/bin/env bash | ||
| # SPDX-License-Identifier: MPL-2.0 | ||
| # Read-only squabbler deployment census. Unknown is NEVER a clean verdict. | ||
| set -euo pipefail | ||
| command -v gh >/dev/null && command -v jq >/dev/null || exit 2 | ||
| tmp="$(mktemp)" | ||
| trap 'rm -f "$tmp"' EXIT | ||
| out="" | ||
| while (($#)); do | ||
| case "$1" in | ||
| --owner) | ||
| gh repo list "$2" --limit 10000 --json nameWithOwner,isArchived,isFork \ | ||
| | jq -r '.[] | select(.isArchived != true and .isFork != true) | .nameWithOwner' >> "$tmp" || exit 3 | ||
|
Comment on lines
+12
to
+13
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Reject an owner census that reaches the repository limit. If an owner has more than 10,000 repositories, As per coding guidelines, “Fail loudly, seal soundly” and “no-silent-skip” apply to this census.
|
||
| shift 2 ;; | ||
| --repo) printf '%s\n' "$2" >> "$tmp"; shift 2 ;; | ||
| --output) out="$2"; shift 2 ;; | ||
| -h|--help) echo "Usage: $0 [--owner OWNER]... [--repo OWNER/REPO]... [--output JSONL]"; exit 0 ;; | ||
| *) echo "Bad argument: $1" >&2; exit 2 ;; | ||
| esac | ||
| done | ||
| # api <endpoint> — print the gh API response when jq -e accepts it. | ||
| # Returns 1 on request or validation failure, including a final false or null | ||
| # value; callers convert these failures to unknown evidence. Write errors propagate. | ||
| api() { | ||
| local data | ||
| data="$(gh api "$1" 2>/dev/null)" || return 1 | ||
| jq -e . >/dev/null 2>&1 <<< "$data" || return 1 | ||
| printf '%s\n' "$data" | ||
| } | ||
| if [[ -n "$out" ]]; then mkdir -p "$(dirname "$out")"; : > "$out"; fi | ||
| # emit <row> — write a newline-terminated row to stdout or append it to $out | ||
| # when non-empty. Write errors propagate. | ||
| emit() { | ||
| if [[ -n "$out" ]]; then printf '%s\n' "$1" >> "$out" | ||
| else printf '%s\n' "$1"; fi | ||
| } | ||
| sort -u "$tmp" | while IFS= read -r repo; do | ||
| [[ -n "$repo" ]] || continue | ||
| [[ "$repo" =~ ^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$ ]] || exit 2 | ||
| meta="$(api "repos/$repo")" || { | ||
| emit "$(jq -nc --arg r "$repo" '{repo:$r,state:"unknown",reason:"metadata_unreadable"}')"; continue | ||
| } | ||
| branch="$(jq -r '.default_branch // empty' <<<"$meta")" | ||
| if [[ -z "$branch" ]]; then | ||
| emit "$(jq -nc --arg r "$repo" '{repo:$r,state:"unknown",reason:"default_branch_missing"}')"; continue | ||
| fi | ||
| ref="$(jq -rn --arg x "$branch" '$x|@uri')" | ||
| workflows="$(api "repos/$repo/contents/.github/workflows?ref=$ref")" || { | ||
| emit "$(jq -nc --arg r "$repo" '{repo:$r,state:"unknown",reason:"workflow_listing_unavailable"}')"; continue | ||
|
Comment on lines
+48
to
+49
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
For a readable repository that simply has no Useful? React with 👍 / 👎. |
||
| } | ||
| if ! jq -e 'type == "array"' <<<"$workflows" >/dev/null; then | ||
| emit "$(jq -nc --arg r "$repo" '{repo:$r,state:"unknown",reason:"bad_listing"}')"; continue | ||
| fi | ||
| names="$(jq -r '.[] | select(.type=="file" and (.name|test("squabbl.*[.]ya?ml$";"i"))) | .name' <<<"$workflows")" | ||
| if [[ -z "$names" ]]; then | ||
| emit "$(jq -nc --arg r "$repo" '{repo:$r,state:"filename_absent",note:"embedded_callers_not_examined"}')"; continue | ||
| fi | ||
| rules="$(api "repos/$repo/rulesets?includes_parents=true")" || rules="" | ||
| while IFS= read -r name; do | ||
| [[ -n "$name" ]] || continue | ||
| role=gate_candidate | ||
| if [[ "$name" == *squabble-action* ]]; then role=action_selftest; fi | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The discovery regex accepts candidate filenames case-insensitively, but this role check is case-sensitive. A valid filename such as Useful? React with 👍 / 👎. |
||
| rid=""; status=unknown; job=""; job_status=unknown | ||
| runs="$(api "repos/$repo/actions/workflows/$name/runs?branch=$ref&per_page=10")" || runs="" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When a matched workflow filename contains a valid URL metacharacter such as Useful? React with 👍 / 👎. |
||
| if [[ -n "$runs" ]] && jq -e '.workflow_runs|type=="array"' <<<"$runs" >/dev/null; then | ||
| rid="$(jq -r '.workflow_runs[0].id // empty' <<<"$runs")" | ||
| status="$(jq -r '.workflow_runs[0].conclusion // .workflow_runs[0].status // "no_run"' <<<"$runs")" | ||
| if [[ -n "$rid" ]]; then | ||
| jobs="$(api "repos/$repo/actions/runs/$rid/jobs?per_page=100")" || jobs="" | ||
| if [[ -n "$jobs" ]] && jq -e '.jobs|type=="array"' <<<"$jobs" >/dev/null; then | ||
| job="$(jq -r '[.jobs[]|select((.name//"")|test("squabbl";"i"))|.name]|first//empty' <<<"$jobs")" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
For a candidate workflow whose gate job has any other display name, this filter leaves Useful? React with 👍 / 👎. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When a candidate workflow emits multiple Squabble-named jobs, only the first API result is retained. If that first job is an informational/setup job while a later Squabble job is the required hard gate, the row incorrectly reports Useful? React with 👍 / 👎. |
||
| if [[ -n "$job" ]]; then | ||
| job_status="$(jq -r --arg j "$job" '[.jobs[]|select(.name==$j)|.conclusion//.status]|first//"unknown"' <<<"$jobs")" | ||
| elif [[ "$status" == "startup_failure" ]]; then job_status=zero_jobs; fi | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When a run concludes Useful? React with 👍 / 👎. |
||
| fi | ||
| fi | ||
| fi | ||
| required=unknown; bypass=unknown | ||
| if [[ -n "$job" && -n "$rules" ]] && jq -e 'type=="array"' <<<"$rules" >/dev/null; then | ||
| # A single page is not a complete negative. Keep incomplete data explicit. | ||
| required=not_matched_in_readable_first_page; bypass='[]' | ||
| if [[ "$(jq -r 'length' <<<"$rules")" -ge 30 ]]; then | ||
| required=unknown; bypass=unknown | ||
| fi | ||
| ids="$(jq -r '.[]|select(.enforcement=="active")|.id' <<<"$rules")" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Apply ruleset branch conditions before reporting a required context. The loop selects every active ruleset, including one that applies only to another branch. If that ruleset names the job, the census reports
|
||
| while IFS= read -r id; do | ||
| [[ -n "$id" ]] || continue | ||
| detail="$(api "repos/$repo/rulesets/$id")" || { required=unknown; bypass=unknown; break; } | ||
| if [[ -n "$job" ]] && jq -e --arg j "$job" \ | ||
| '[.rules[]?|select(.type=="required_status_checks")|.parameters.required_status_checks[]?|.context]|index($j)!=null' \ | ||
| <<<"$detail" >/dev/null; then | ||
|
Comment on lines
+89
to
+91
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
For a repository whose emitted job name also appears in an active ruleset scoped only to another ref such as Useful? React with 👍 / 👎. |
||
| required=required_by_ruleset | ||
|
Comment on lines
+89
to
+92
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When an active ruleset contains this context but targets another branch, excludes the default branch, or targets tags, this code still reports Useful? React with 👍 / 👎. |
||
| bypass="$(jq -c '[.bypass_actors[]?|{actor_type,actor_id,bypass_mode}]' <<<"$detail")" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Distinguish hidden bypass actors from an empty bypass list. GitHub omits As per coding guidelines, “fail-closed” applies when the census cannot inspect bypass authority.
|
||
| fi | ||
| done <<<"$ids" | ||
| fi | ||
| emit "$(jq -nc --arg r "$repo" --arg wf "$name" --arg id "$rid" \ | ||
| --arg state "$status" --arg job "$job" --arg js "$job_status" \ | ||
| --arg req "$required" --arg by "$bypass" --arg role "$role" \ | ||
| '{repo:$r,state:"named_workflow_found",role:$role,workflow:$wf,run:{id:$id,status:$state,job:$job,job_conclusion:$js},policy:{rulesets:$req,bypass:$by,classic_protection:"not_inspected"}}')" | ||
| done <<<"$names" | ||
| done | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| #!/usr/bin/env bash | ||
| # SPDX-License-Identifier: MPL-2.0 | ||
| set -euo pipefail | ||
| root="$(cd "$(dirname "$0")/.." && pwd)" | ||
| tmp="$(mktemp -d)" | ||
| trap 'rm -rf "$tmp"' EXIT | ||
| mkdir -p "$tmp/bin" | ||
| cat > "$tmp/bin/gh" <<'MOCK' | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| [[ "$1" == api ]] || exit 2 | ||
| case "$2" in | ||
| repos/example/pass|repos/example/startup|repos/example/absent|repos/example/selftest) | ||
| printf '%s\n' '{"default_branch":"main","archived":false}';; | ||
| 'repos/example/pass/contents/.github/workflows?ref=main') | ||
| printf '%s\n' '[{"name":"cicd-squabbler.yml","type":"file"}]';; | ||
| 'repos/example/startup/contents/.github/workflows?ref=main') | ||
| printf '%s\n' '[{"name":"squabble.yml","type":"file"}]';; | ||
| 'repos/example/selftest/contents/.github/workflows?ref=main') | ||
| printf '%s\n' '[{"name":"squabble-action.yml","type":"file"}]';; | ||
| 'repos/example/absent/contents/.github/workflows?ref=main') | ||
| printf '%s\n' '[{"name":"ci.yml","type":"file"}]';; | ||
| 'repos/example/pass/actions/workflows/cicd-squabbler.yml/runs?branch=main&per_page=10') | ||
| printf '%s\n' '{"workflow_runs":[{"id":7,"conclusion":"success"}]}';; | ||
| 'repos/example/startup/actions/workflows/squabble.yml/runs?branch=main&per_page=10') | ||
| printf '%s\n' '{"workflow_runs":[{"id":8,"conclusion":"startup_failure"}]}';; | ||
| 'repos/example/pass/actions/runs/7/jobs?per_page=100') | ||
| printf '%s\n' '{"jobs":[{"name":"Squabble chains (hard gate)","conclusion":"success"}]}';; | ||
| 'repos/example/selftest/actions/workflows/squabble-action.yml/runs?branch=main&per_page=10') | ||
| printf '%s\n' '{"workflow_runs":[{"id":9,"conclusion":"success"}]}';; | ||
| 'repos/example/selftest/actions/runs/9/jobs?per_page=100') | ||
| printf '%s\n' '{"jobs":[{"name":"squabble action test","conclusion":"success"}]}';; | ||
| 'repos/example/selftest/rulesets?includes_parents=true') | ||
| printf '%s\n' '[]';; | ||
| 'repos/example/startup/actions/runs/8/jobs?per_page=100') | ||
| printf '%s\n' '{"jobs":[]}';; | ||
| 'repos/example/pass/rulesets?includes_parents=true') | ||
| printf '%s\n' '[{"id":3,"enforcement":"active"}]';; | ||
| 'repos/example/startup/rulesets?includes_parents=true') | ||
| printf '%s\n' '[]';; | ||
| repos/example/pass/rulesets/3) | ||
| printf '%s\n' '{"rules":[{"type":"required_status_checks","parameters":{"required_status_checks":[{"context":"Squabble chains (hard gate)"}]}}],"bypass_actors":[{"actor_type":"RepositoryRole","actor_id":5,"bypass_mode":"always"}]}';; | ||
| *) echo "unexpected mocked API call: $2" >&2; exit 1;; | ||
| esac | ||
| MOCK | ||
| chmod +x "$tmp/bin/gh" | ||
| bash -n "$root/scripts/estate-gate-census.sh" | ||
| PATH="$tmp/bin:$PATH" bash "$root/scripts/estate-gate-census.sh" \ | ||
| --repo example/pass --repo example/startup --repo example/absent --repo example/selftest --output "$tmp/rows.jsonl" | ||
| [[ "$(wc -l < "$tmp/rows.jsonl")" -eq 4 ]] | ||
| jq -e -s ' | ||
| (map(select(.repo=="example/pass"))[0] | .run.job_conclusion=="success" and | ||
| .policy.rulesets=="required_by_ruleset" and (.policy.bypass | contains("always"))) and | ||
| (map(select(.repo=="example/startup"))[0] | .run.status=="startup_failure" and | ||
| .run.job_conclusion=="zero_jobs") and | ||
| (map(select(.repo=="example/absent"))[0] | .state=="filename_absent") and | ||
| (map(select(.repo=="example/selftest"))[0] | .role=="action_selftest") | ||
| ' "$tmp/rows.jsonl" >/dev/null | ||
| echo "estate gate census: hermetic cases passed" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use the required SPDX identifier.
The new document declares
CC-BY-SA-4.0, but the repository instruction requiresMPL-2.0on all new files. Confirm the intended licence and update the identifier to comply.As per coding guidelines: “SPDX:
MPL-2.0on all new files.”Source: Coding guidelines