Skip to content

test: drop review tags and fix history from hold-area test comments - #37416

Merged
iamwhatever merged 2 commits into
mainfrom
crew/testoptimizer/test-quality-batch-f
Nov 15, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
crew/testoptimizer/test-quality-batch-f

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Remove review tags and before/after fix stories from comments in the sandbox, security, governance, name-grant, platform, skill and validation test files, keeping every sentence that explains a security reason, with no change to test behavior.

These tests carry the densest review history in the suite: test_name_grant.py alone opens 28 comments with "GPT 5.6 round-N:" or "Opus 4.8 round-18.", the governance tests open with "HIGH (GPT pass 2):" and "Regression (PR #14 alice):", and many comments say "Before the fix this returned ()" instead of the rule.

Why it matters

These tests are the proof behind the agent's own fences. The attack each one names is the useful part; the review round that found it is not, and it pushes the attack further down the comment.

Not a goal

  • No change to any test's code, names, assertions, fixtures or assertion messages.
  • No change to source code or to test/conftest.py, and no new comments that restate the code.
  • Every sentence that states an attack or a security reason stays; only its tag or its past tense changes.
  • Files whose subject is review findings (test_security_conductor*) or third-party product names (test_connections_registry.py), and two dense history blocks left as written (test_security_gate_liveness.py count note, test_security_posture.py alternation note).

What changed (motivation → approach → change)

Before After
"GPT 5.6 round-16, and the sharpest finding on this PR because it is the PR's own thesis turned against it. shlex discards the rest of the line after # ..." "shlex discards the rest of the line after #; bash does NOT ..."
"GPT 5.6 round-19. The dispatcher rule read the name as WRITTEN, so an ALIAS for one slipped past it ..." "A dispatcher rule that reads the name as WRITTEN lets an ALIAS for one slip past it ..."
"HIGH (GPT pass 2): a directory iterdir() OSError on a WARM store must NOT ..." "A directory iterdir() OSError on a WARM store must NOT ..."
"Before the fix this path returned permitted==True and the gate wrongly permitted the publish." "Without fail_closed this path returns permitted==True and the gate wrongly permits the publish."
"# --- deploy_artifact schema tests (item 6 R18) ---" "# --- deploy_artifact schema tests ---"

Twenty-nine files: 105 insertions, 108 deletions. goal_scan.py astcheck against the merge-base: files checked: 29 code-changed: 0 docstring-only: 0.

A second commit corrects one comment in test/test_browser_quick_ask_contract.py: it said the test pins "the two values a real ask mints", but the line sets three fields (id, expires_at, phase). With it: thirty files, code-changed 0.

Backwards compatibility

Compatible: test-only change, comments only.

Removes nothing: no code, key, path, record or function is removed or renamed.

Tests

Proof: not provable -- comment-only change; the AST check shows no code difference

The twenty-nine touched files plus test/test_flake_pattern_ratchet.py pass on this branch (pytest -n 2 -p no:randomly: 2284 passed, 24 skipped). flake8, check_comment_history.py and check_brand_name.py pass; no added line is over 100 columns.

Manual verification

N/A — comment-only change.

Related Issues

N/A — part of the test comment quality pass.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 14, 2026 23:51
@iamwhatever
iamwhatever requested a review from buluoray November 14, 2026 23:51
@iamwhatever
iamwhatever enabled auto-merge (squash) November 14, 2026 23:51
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f23238e7998648b4695c65e0d0631b1449c60a39 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision f23238e7998648b4695c65e0d0631b1449c60a39 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of f23238e7998648b4695c65e0d0631b1449c60a39 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing to check. The change only edits comments in test files, so no behavior, contract or surface changes.

[DESIGN-REVIEWED] f23238e

Description read

[DESCRIPTION-READ] 8831cb0f4e95e20717eb0e7076293e14a0d15aa639cd1832ae074ccb1b2fe7be

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f23238e7998648b4695c65e0d0631b1449c60a39 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f23238e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f23238e7998648b4695c65e0d0631b1449c60a39 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f23238e7998648b4695c65e0d0631b1449c60a39 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f23238e

Verdict parsed from the review's SHA-scoped output markers for commit f23238e7998648b4695c65e0d0631b1449c60a39.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f23238e7998648b4695c65e0d0631b1449c60a39 false-positive: <reason, 20 to 500 characters>

@iamwhatever

Copy link
Copy Markdown
Contributor Author

The Backend Tests (3.12, 1/2/8), Windows (8) and Coverage Gate reds on this head are main's, not this PR's: the same row-scrub tests fail on clean main 99dc8da (test/test_autonudge_row_scrub.py: 5 failed locally). Tracked in #37421, fix in #37423. This PR only changes comments in other files; it will re-run once #37423 lands.

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 5% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Comment-only change across 29 test files. It removes review tags and fix-history wording and keeps every security explanation, now written as a rule. No code, assertion, fixture or source changes, so it can't change test behavior. The failing CI checks are redaction assertions in files this PR doesn't touch and can't have been caused by comment edits.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 02453d8c7e · 213 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@iamwhatever
iamwhatever disabled auto-merge November 15, 2026 01:05
zejiangg and others added 2 commits November 15, 2026 01:05
Remove review-round, finding and severity tags and before/after fix
stories from comments in the sandbox, security, governance, name-grant,
platform, skill and validation test files, keeping every sentence that
explains a security reason. Comments only; no test behaviour changes.
The comment above the pinning line said two values while the line pins
three (id, expires_at, phase). The phase is a default state, not minted,
so the comment now names the two minted values and the phase. Comment
only.
@iamwhatever
iamwhatever force-pushed the crew/testoptimizer/test-quality-batch-f branch from 02453d8 to f23238e Compare November 15, 2026 01:05
@iamwhatever
iamwhatever enabled auto-merge (squash) November 15, 2026 01:06
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 15, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. On this head, a Python AST comparison of all 30 touched files against the merge-base found no differences. Only # comments change, and no code, docstrings, assertions or noqa/flake-ok markers do. Every path is under test/. No commit on main has touched these files since the merge-base, and the sibling passes already merged share no file with this PR. The body's file and line counts match the diff.

Done well: each rewrite in test_name_grant.py keeps the attack and the rule in the present tense and drops only the review round, so the security reason now comes first in each comment.

Optional:

  • test/test_skill_providers.py:83: "(not an AttributeError)" reads as if .get() on a scalar does not raise. Something like "(which must not surface as an AttributeError)" says what is meant.
  • test/test_memory_smoke.py:247: "(a fix that only a restart heals is a bug)" is unclear. Consider "(a stale cache that only a restart clears is a bug)".
  • Some tags and fix-history notes are still in the files this PR edits, for example # delivered 2 at test/test_computer_use_drag_path.py:312 and the F1-2/F2-3 tags in the governance tests. A later pass could remove them.

@iamwhatever
iamwhatever merged commit b914181 into main Nov 15, 2026
101 of 106 checks passed
@iamwhatever
iamwhatever deleted the crew/testoptimizer/test-quality-batch-f branch November 15, 2026 02:20
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants