Skip to content

test: refuse new tests asserting an AWS key ID literal is removed - #38375

Merged
bolichen97 merged 2 commits into
mainfrom
crew/mate/akia-removal-ratchet
Nov 17, 2026
Merged

bolichen97 merged 2 commits into
mainfrom
crew/mate/akia-removal-ratchet

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A new test that asserts an AWS key ID literal is removed from a value fails CI, so the testing-conventions rule from #37692 is enforced instead of only written.

#36640 planted the AWS documentation example key and asserted it was removed, while #29423 had made the redactor keep AWS key IDs. Both PRs were green on their own base and together turned main red on every platform (#37421, fixed by #37423). #37692 wrote the rule down in docs/system-specs/common/testing-conventions.md, but nothing stops the next such test.

Why it matters

The same collision can red main for every PR again, on every platform, whenever a new removal test plants an AKIA... literal and detector policy moves. A written rule alone did not stop it.

Not a goal

  • Rewriting the 70 existing sites to synthetic_credentials. They are recorded in a baseline that only shrinks.
  • Fixing the three vacuous tests listed below. Each needs its premise re-thought, which is not trivial.
  • Changing the redactor.

What changed (motivation → approach → change)

  • New test/test_redaction_literal_ratchet.py (283 lines). It AST-scans test/**/*.py for <literal> not in <value> and assertNotIn(<literal>, ...) where the literal (a str or bytes literal, f-string, or + of literals, or a name the same function, module or class body binds to one, read bare or as self.KEY) holds AKIA[0-9A-Z]{16}. Asserting the key is KEPT (in) is still allowed.
  • _BASELINE records the existing sites as exact per-file counts: 46 files, 81 sites (the "70" under Not a goal was the count before review added the bytes and self.KEY spellings). A new site fails with a message that points at the testing-conventions rule and the synthetic_credentials helpers. A removed site must lower its count, so the baseline only shrinks.
  • Detector self-tests pin the spellings it catches and the ones it allows.
  • testing-conventions.md: the rule now names the ratchet.
  • It is a separate module because test/test_flake_pattern_ratchet.py is at 1,932 lines, close to the 2,000-line cap.

Existing sites that pass by accident

The structured redactor keeps every AWS key ID (redact_credentials("key AKIA...") returns the input unchanged). I swapped the literal for a plain sentinel string in every baselined file and re-ran them: 18 tests failed, so those do depend on some detector removing the key. Of the tests that still passed, these three assert nothing real:

  • test/test_voice_reply.py::test_credential_rejoined_by_comment_strip_is_redacted: strip_markdown does not remove the inline <!-- keep-visible --> tag, so the key halves are never rejoined. The output is the input, and the assertion holds trivially.
  • test/test_webex_dispatch_cards.py::test_a_queue_receipt_scans_the_text_it_quotes: the receipt sent is ⏳ Folded into the reply in progress., which quotes no text at all.
  • test/test_imessage_renderer.py::test_credential_reassembled_through_link_grammar_is_caught: delivery text is AKIA (https://x.example)IOSFODNN7EXAMPLE. The link flattening never rejoins the key, so nothing is being caught.

They are left in the baseline. A follow-up should rebuild each with a synthetic_credentials value and a split that the code really rejoins.

Backwards compatibility

Compatible: test-only gate plus one doc sentence; no runtime code changes.

Removes nothing: adds a test and a doc sentence only.

Tests

  • test_the_detector_sees_every_removal_spelling / test_the_detector_allows_a_kept_key_and_other_values: the detector's catch and allow sets.
  • test_no_new_test_asserts_an_aws_key_id_literal_is_removed: whole test/ tree against the baseline.
  • test_the_baseline_has_no_empty_rows.

Proof: not provable -- test-only change

Ran locally (Windows): the new module and test/test_flake_pattern_ratchet.py (75 passed); black, isort, flake8, check_file_line_cap.py, check_comment_history.py, check_brand_name.py, scripts/docs-lint.sh all clean.

Manual verification

N/A -- unit coverage sufficient: the gate is itself a test, and its detector is proven on fixtures.

Related Issues

Refs #37421, #37423, #37692, #36640, #29423

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

The redactor keeps AWS key IDs, so a test that plants one and asserts it is gone tracks detector policy, not the code under test (#37421). A ratchet now AST-scans test/ for such removal assertions; the 70 existing sites in 43 files are a baseline that only shrinks.
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 17, 2026 10:01
@bolichen97
bolichen97 requested a review from buluoray November 17, 2026 10:01
@bolichen97
bolichen97 enabled auto-merge (squash) November 17, 2026 10:01
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f81089a02fbc3ac4cf029cef11bdaf9f0eba9b00 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision f81089a02fbc3ac4cf029cef11bdaf9f0eba9b00 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of f81089a02fbc3ac4cf029cef11bdaf9f0eba9b00 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This is a test-only ratchet. It turns an existing written rule into a gate whose baseline only shrinks, takes nothing away, and is the right size for the problem.

Notes

  • The description's Backwards compatibility section has both a Compatible: line and a Removes nothing: line. Keep only the Removes nothing: line.

[DESIGN-REVIEWED] f81089a

Description read

[DESCRIPTION-READ] 3f551ff79330b87aaf1c2b000f29ee3c6384a5fab2f025b4b04bcf3648b03178

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f81089a02fbc3ac4cf029cef11bdaf9f0eba9b00 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f81089a

Verdict parsed from the review's SHA-scoped output markers for commit f81089a02fbc3ac4cf029cef11bdaf9f0eba9b00.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f81089a02fbc3ac4cf029cef11bdaf9f0eba9b00 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f81089a02fbc3ac4cf029cef11bdaf9f0eba9b00 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f81089a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f81089a02fbc3ac4cf029cef11bdaf9f0eba9b00 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 force-pushed the crew/mate/akia-removal-ratchet branch from 899c48e to 9e84071 Compare November 17, 2026 11:00
Review found spellings the scan skipped (a bytes literal, self.KEY) and names it resolved too widely (a local rebinding, another class's attribute). Names now resolve the way Python resolves them: function scopes shadow, class bodies are their own scope, and self.KEY reads only the enclosing class. Baseline: 81 sites in 46 files.
@bolichen97
bolichen97 force-pushed the crew/mate/akia-removal-ratchet branch from 9e84071 to f81089a Compare November 17, 2026 11:33
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 17, 2026
@bolichen97
bolichen97 merged commit a3c8af3 into main Nov 17, 2026
100 of 105 checks passed
@bolichen97
bolichen97 deleted the crew/mate/akia-removal-ratchet branch November 17, 2026 12:50
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant