Repository navigation
feat(standards): include GEMINI.md in agent configuration standard and audit #1378
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
00b0d76
14a2ad8
e491f62
cb85abd
68150ab
a521440
3c5fd2b
cc5903c
4a94501
ca778d7
62f7dfd
364efb6
6bcd1a3
b084f7c
0b6cab7
7b652b3
7298294
8741b9c
812803f
78f9d0a
c435000
7b263e0
8b52fec
13a3fb5
3650db3
7fa27f2
5cc870a
9aa74dc
b7ecc28
a0f3623
3c13c1b
cd70bd2
53d6b71
049ef8b
77cec6c
a90ae04
2ccea1c
858e1a2
673a41c
38a61e0
9280eea
54f0567
b656ac8
3ba6a3e
2fd1c01
cccd228
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -32,6 +32,7 @@ on: | |||||||||||||||||||||||||||||||||
| type: string | ||||||||||||||||||||||||||||||||||
| default: | | ||||||||||||||||||||||||||||||||||
| CLAUDE.md | ||||||||||||||||||||||||||||||||||
| GEMINI.md | ||||||||||||||||||||||||||||||||||
| AGENTS.md | ||||||||||||||||||||||||||||||||||
| org-standards-ref: | ||||||||||||||||||||||||||||||||||
| description: "String AGENTS.md must reference (regex, basic grep -E)" | ||||||||||||||||||||||||||||||||||
|
|
@@ -49,6 +50,16 @@ jobs: | |||||||||||||||||||||||||||||||||
| steps: | ||||||||||||||||||||||||||||||||||
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| # Fetch the org's agent-md-xref helper for cross-repository consumers. | ||||||||||||||||||||||||||||||||||
| # External consumers won't have scripts/lib/agent-md-xref.sh in their repo, | ||||||||||||||||||||||||||||||||||
| # so we fetch it from combsmith/.github with sparse-checkout. | ||||||||||||||||||||||||||||||||||
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||||||||||||||||||||||||||||||||||
| with: | ||||||||||||||||||||||||||||||||||
| repository: combsmith/.github | ||||||||||||||||||||||||||||||||||
| path: __org-github | ||||||||||||||||||||||||||||||||||
| sparse-checkout: scripts/lib/agent-md-xref.sh | ||||||||||||||||||||||||||||||||||
| sparse-checkout-cone-mode: false | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| # --- Deep security scan via AgentShield CLI --- | ||||||||||||||||||||||||||||||||||
| # Uses ecc-agentshield (https://github.com/affaan-m/agentshield) | ||||||||||||||||||||||||||||||||||
| # 102 rules: secrets, permissions, hooks, MCP servers, agent config | ||||||||||||||||||||||||||||||||||
|
|
@@ -121,17 +132,32 @@ jobs: | |||||||||||||||||||||||||||||||||
| done <<< "$REQUIRED_FILES" | ||||||||||||||||||||||||||||||||||
| exit $status | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| # The AGENTS.md reference verdict lives in scripts/lib/agent-md-xref.sh | ||||||||||||||||||||||||||||||||||
| # (bats-tested; shared with the fleet compliance audit). Copy the file | ||||||||||||||||||||||||||||||||||
| # from either the caller's repo or the org-github checkout. | ||||||||||||||||||||||||||||||||||
| - name: Fetch cross-reference helper | ||||||||||||||||||||||||||||||||||
| run: | | ||||||||||||||||||||||||||||||||||
| mkdir -p "$RUNNER_TEMP/org-lib" | ||||||||||||||||||||||||||||||||||
| if [ -f "scripts/lib/agent-md-xref.sh" ]; then | ||||||||||||||||||||||||||||||||||
| cp "scripts/lib/agent-md-xref.sh" "$RUNNER_TEMP/org-lib/agent-md-xref.sh" | ||||||||||||||||||||||||||||||||||
| else | ||||||||||||||||||||||||||||||||||
| cp "__org-github/scripts/lib/agent-md-xref.sh" "$RUNNER_TEMP/org-lib/agent-md-xref.sh" | ||||||||||||||||||||||||||||||||||
| fi | ||||||||||||||||||||||||||||||||||
|
Comment on lines
+138
to
+145
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The 'Fetch cross-reference helper' step prefers The fallback copy is also taken from the default branch of Always use the org-checkout copy, pinned to a ref, instead of the caller's file: cp "__org-github/scripts/lib/agent-md-xref.sh" "$RUNNER_TEMP/org-lib/agent-md-xref.sh"
Suggested change
Spotted by Graphite |
||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| - name: Validate cross-references | ||||||||||||||||||||||||||||||||||
| env: | ||||||||||||||||||||||||||||||||||
| ORG_REF: ${{ inputs.org-standards-ref }} | ||||||||||||||||||||||||||||||||||
| run: | | ||||||||||||||||||||||||||||||||||
| # shellcheck source=/dev/null | ||||||||||||||||||||||||||||||||||
| . "$RUNNER_TEMP/org-lib/agent-md-xref.sh" | ||||||||||||||||||||||||||||||||||
| status=0 | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| if [ -f "CLAUDE.md" ] && \ | ||||||||||||||||||||||||||||||||||
| ! grep -qi 'AGENTS.md' CLAUDE.md; then | ||||||||||||||||||||||||||||||||||
| echo "::error file=CLAUDE.md::Must reference AGENTS.md" | ||||||||||||||||||||||||||||||||||
| status=1 | ||||||||||||||||||||||||||||||||||
| fi | ||||||||||||||||||||||||||||||||||
| for f in CLAUDE.md GEMINI.md; do | ||||||||||||||||||||||||||||||||||
| if [ -f "$f" ] && ! agent_md_references_agents < "$f"; then | ||||||||||||||||||||||||||||||||||
| echo "::error file=$f::Must reference AGENTS.md" | ||||||||||||||||||||||||||||||||||
| status=1 | ||||||||||||||||||||||||||||||||||
| fi | ||||||||||||||||||||||||||||||||||
| done | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| if [ -f "AGENTS.md" ] && \ | ||||||||||||||||||||||||||||||||||
| ! grep -qiE "$ORG_REF" AGENTS.md; then | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| # GEMINI.md | ||
|
|
||
| This repository follows the development and agentic architecture standards defined in [AGENTS.md](AGENTS.md) and [`combsmith/.github/AGENTS.md`](https://github.com/combsmith/.github/blob/main/AGENTS.md). | ||
|
|
||
| Read [AGENTS.md](AGENTS.md) for full context, project conventions, testing guidelines, and architectural rules. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -281,6 +281,10 @@ | |
| # shellcheck source=lib/push-protection.sh | ||
| . "$SCRIPT_DIR/lib/push-protection.sh" | ||
|
|
||
| # Shared agent-file cross-reference verdict (also used by agent-shield-reusable.yml). | ||
| # shellcheck source=lib/agent-md-xref.sh | ||
| . "$SCRIPT_DIR/lib/agent-md-xref.sh" | ||
|
|
||
| # Shared label source of truth — STANDARD_LABEL_SPECS (fixed set) and | ||
| # persona_opt_out_label_configs (derived <id>:hands-off family). Sourced after | ||
| # warn() is defined so the library uses this script's logging helper (issue #1139). | ||
|
|
@@ -2556,26 +2560,96 @@ | |
| check_claude_md() { | ||
| local repo="$1" | ||
|
|
||
| local content | ||
| content=$(gh_api "repos/$ORG/$repo/contents/CLAUDE.md" --jq '.content' 2>/dev/null || echo "") | ||
| local content rc raw i | ||
| for i in 1 2 3; do | ||
| raw=$(gh api "repos/$ORG/$repo/contents/CLAUDE.md" 2>&1) && rc=0 || rc=$? | ||
| [ "$rc" -eq 0 ] && break | ||
|
Check failure on line 2566 in scripts/compliance-audit.sh
|
||
| response_is_http_404 "$raw" && break | ||
| [ "$i" -lt 3 ] && sleep $((i * 2)) | ||
|
Check failure on line 2568 in scripts/compliance-audit.sh
|
||
| done | ||
|
|
||
| if [ -z "$content" ]; then | ||
| add_finding "$repo" "standards" "missing-claude-md" "error" \ | ||
| "Missing \`CLAUDE.md\` — every repo must have a CLAUDE.md that references AGENTS.md" \ | ||
| "AGENTS.md" | ||
| if [ "$rc" -ne 0 ]; then | ||
| if response_is_http_404 "$raw"; then | ||
| add_finding "$repo" "standards" "missing-claude-md" "error" \ | ||
| "Missing \`CLAUDE.md\` — every repo must have a CLAUDE.md that references AGENTS.md" \ | ||
| "AGENTS.md" | ||
| else | ||
| info "CLAUDE.md not readable for $repo (not a 404) — marking the repo inconclusive" | ||
| mark_repo_inconclusive "$repo" | ||
| fi | ||
| return | ||
| fi | ||
|
|
||
| local decoded | ||
| decoded=$(echo "$content" | base64 -d 2>/dev/null || echo "") | ||
| # An empty file is a successful API response with empty content: it is | ||
| # non-compliant (no AGENTS.md reference), not a reason to abort the audit. | ||
| # Undecodable content is inconclusive. Neither may fall through to a bare | ||
| # `return` carrying a failed test status, which would trip `set -e`. | ||
| content=$(printf '%s' "$raw" | jq -r '.content? // empty' 2>/dev/null || echo "") | ||
|
|
||
| if ! echo "$decoded" | grep -qi 'AGENTS\.md'; then | ||
| local decoded="" | ||
| if [ -n "$content" ]; then | ||
|
Check failure on line 2590 in scripts/compliance-audit.sh
|
||
| if ! decoded=$(printf '%s' "$content" | base64 -d 2>/dev/null); then | ||
| info "CLAUDE.md content not decodable for $repo — marking the repo inconclusive" | ||
| mark_repo_inconclusive "$repo" | ||
| return 0 | ||
| fi | ||
| fi | ||
|
|
||
| if ! printf '%s' "$decoded" | agent_md_references_agents; then | ||
| add_finding "$repo" "standards" "claude-md-missing-agents-ref" "error" \ | ||
| "\`CLAUDE.md\` does not reference \`AGENTS.md\`" \ | ||
| "AGENTS.md" | ||
| fi | ||
| } | ||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Check: GEMINI.md exists and references AGENTS.md | ||
| # --------------------------------------------------------------------------- | ||
| check_gemini_md() { | ||
| local repo="$1" | ||
|
|
||
| local content rc raw i | ||
| for i in 1 2 3; do | ||
| raw=$(gh api "repos/$ORG/$repo/contents/GEMINI.md" 2>&1) && rc=0 || rc=$? | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
| [ "$rc" -eq 0 ] && break | ||
|
Check failure on line 2614 in scripts/compliance-audit.sh
|
||
| response_is_http_404 "$raw" && break | ||
| [ "$i" -lt 3 ] && sleep $((i * 2)) | ||
|
Check failure on line 2616 in scripts/compliance-audit.sh
|
||
| done | ||
|
|
||
| if [ "$rc" -ne 0 ]; then | ||
|
Check failure on line 2619 in scripts/compliance-audit.sh
|
||
| if response_is_http_404 "$raw"; then | ||
| add_finding "$repo" "standards" "missing-gemini-md" "error" \ | ||
| "Missing \`GEMINI.md\` — every repo must have a GEMINI.md that references AGENTS.md" \ | ||
| "AGENTS.md" | ||
| else | ||
| info "GEMINI.md not readable for $repo (not a 404) — marking the repo inconclusive" | ||
| mark_repo_inconclusive "$repo" | ||
| fi | ||
| return | ||
| fi | ||
|
|
||
| # An empty file is a successful API response with empty content: it is | ||
| # non-compliant (no AGENTS.md reference), not a reason to abort the audit. | ||
| # Undecodable content is inconclusive. Neither may fall through to a bare | ||
| # `return` carrying a failed test status, which would trip `set -e`. | ||
| content=$(printf '%s' "$raw" | jq -r '.content? // empty' 2>/dev/null || echo "") | ||
|
|
||
| local decoded="" | ||
| if [ -n "$content" ]; then | ||
|
Check failure on line 2638 in scripts/compliance-audit.sh
|
||
| if ! decoded=$(printf '%s' "$content" | base64 -d 2>/dev/null); then | ||
| info "GEMINI.md content not decodable for $repo — marking the repo inconclusive" | ||
| mark_repo_inconclusive "$repo" | ||
| return 0 | ||
| fi | ||
| fi | ||
|
|
||
| if ! printf '%s' "$decoded" | agent_md_references_agents; then | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
For sufficiently large compliant Useful? React with 👍 / 👎.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in scripts/lib/agent-md-xref.sh:
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in scripts/lib/agent-md-xref.sh: replaced the pipe-based stdin with a here-string redirection to prevent SIGPIPE failures when grep matches early in large files. The function now uses cat with here-string syntax to ensure grep consumes all input before exiting.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Original reply, kept for the record (unverified — it does not count as a fix):
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No new change in this pass: the SIGPIPE fix is already on the branch. |
||
| add_finding "$repo" "standards" "gemini-md-missing-agents-ref" "error" \ | ||
| "\`GEMINI.md\` does not reference \`AGENTS.md\`" \ | ||
| "AGENTS.md" | ||
| fi | ||
| } | ||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Check: AGENTS.md exists and references org .github/AGENTS.md | ||
| # --------------------------------------------------------------------------- | ||
|
|
@@ -2831,7 +2905,7 @@ | |
| fi | ||
| done | ||
|
|
||
| if [ "$rc" -ne 0 ]; then | ||
|
Check failure on line 2908 in scripts/compliance-audit.sh
|
||
| add_finding "$repo" "settings" "check-suite-prefs-unreadable" "warning" \ | ||
| "Could not read check-suite preferences — verify GH_TOKEN has repo scope and the repo is accessible. Check-suite auto-trigger compliance was not evaluated." \ | ||
| "standards/github-settings.md" | ||
|
|
@@ -3170,7 +3244,7 @@ | |
| "reusable-workflows|Reusable Workflows (disabled_manually)|gh workflow disable <workflow> -R <repo>" | ||
| "action-pinning|Action SHA Pinning|pin actions to SHA in each workflow file" | ||
| "dependabot|Dependabot Configuration|per-repo .github/dependabot.yml" | ||
| "standards|Agent Standards (CLAUDE.md / AGENTS.md / copilot-setup-steps.yml)|per-repo doc and workflow additions" | ||
| "standards|Agent Standards (CLAUDE.md / GEMINI.md / AGENTS.md / copilot-setup-steps.yml)|per-repo doc and workflow additions" | ||
| ) | ||
|
|
||
| local body | ||
|
|
@@ -3774,6 +3848,7 @@ | |
| check_verbatim_stubs "$repo" | ||
| check_centralized_check_names "$repo" | ||
| check_claude_md "$repo" | ||
| check_gemini_md "$repo" | ||
| check_agents_md "$repo" | ||
| check_copilot_setup_steps "$repo" | ||
| check_copilot_instructions "$repo" | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| #!/usr/bin/env bash | ||
| # agent-md-xref.sh — pure verdict for the agent-file cross-reference rule. | ||
| # | ||
| # Sourced by scripts/compliance-audit.sh (check_claude_md / check_gemini_md) | ||
| # and fetched by .github/workflows/agent-shield-reusable.yml ("Validate | ||
| # cross-references"), so the fleet audit and the per-repo CI gate share ONE | ||
| # decision. Bats coverage: test/scripts/compliance-audit/agent-md-xref.bats. | ||
| # | ||
| # agent_md_references_agents — reads a file body on stdin; exit 0 when it | ||
| # contains the literal filename AGENTS.md | ||
| # (case-insensitive, fixed-string match, so | ||
| # "AGENTS-md" or "AGENTSxmd" never satisfy it). | ||
| # Empty input is non-compliant (exit 1). | ||
|
|
||
| agent_md_references_agents() { | ||
| # Use here-string redirection to avoid SIGPIPE when grep matches early | ||
| # in large files. This allows the verdict logic to work correctly with | ||
| # pipefail enabled in compliance-audit.sh. | ||
| grep -Fi 'AGENTS.md' <<< "$(cat)" > /dev/null | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| #!/usr/bin/env bats | ||
| # Unit tests for scripts/lib/agent-md-xref.sh — the shared AGENTS.md | ||
| # cross-reference verdict used by the fleet audit and agent-shield-reusable.yml. | ||
|
|
||
| REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../.." && pwd)" | ||
| LIB="$REPO_ROOT/scripts/lib/agent-md-xref.sh" | ||
| WORKFLOW="$REPO_ROOT/.github/workflows/agent-shield-reusable.yml" | ||
|
|
||
| setup() { source "$LIB"; } | ||
|
|
||
| @test "xref: body naming AGENTS.md passes" { | ||
| run agent_md_references_agents <<< "See [AGENTS.md](AGENTS.md)." | ||
| [ "$status" -eq 0 ] | ||
| } | ||
|
|
||
| @test "xref: match is case-insensitive" { | ||
| run agent_md_references_agents <<< "see agents.md" | ||
| [ "$status" -eq 0 ] | ||
| } | ||
|
|
||
| @test "xref: punctuation cannot stand in for the dot (fixed-string match)" { | ||
| run agent_md_references_agents <<< "AGENTSxmd AGENTS-md" | ||
| [ "$status" -ne 0 ] | ||
| } | ||
|
|
||
| @test "xref: empty body fails" { | ||
| run agent_md_references_agents < /dev/null | ||
| [ "$status" -ne 0 ] | ||
| } | ||
|
|
||
| @test "reusable workflow delegates the verdict to the helper for CLAUDE.md and GEMINI.md" { | ||
| grep -q 'agent_md_references_agents' "$WORKFLOW" | ||
| grep -q 'scripts/lib/agent-md-xref.sh' "$WORKFLOW" | ||
| ! grep -q "grep -qFi 'AGENTS.md'" "$WORKFLOW" | ||
| } | ||
|
|
||
| @test "agent_md_references_agents: early match on large piped input survives pipefail" { | ||
| run bash -c 'set -o pipefail; . "$0"; { echo "See AGENTS.md"; head -c 131072 /dev/zero | tr "\0" x; } | agent_md_references_agents' "$LIB" | ||
| [ "$status" -eq 0 ] | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For cross-repository consumers, this condition also accepts a helper supplied by the calling repository, allowing a PR to replace
agent_md_references_agentswith an unconditional success and pass cross-reference validation for noncompliant files. The previous inline predicate could not be overridden this way. The fresh evidence is that the new central-checkout fallback is reached only when the caller has no helper; the local-file preference is not restricted to internal consumers. Use the central helper for external callers and permit the local version only when testing this standards repository.Useful? React with 👍 / 👎.