Skip to content

test: tighten stale comments in the slack tests - #37320

Merged
iamwhatever merged 1 commit into
mainfrom
crew/testoptimizer/test-quality-slack
Nov 14, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
crew/testoptimizer/test-quality-slack

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make the comments in the test/test_slack*.py tests true and useful to someone changing the Slack code, keeping every reason and gotcha, with no change to test behavior.

Several comments in the Slack tests point at review rounds and commits a reader cannot open ("GPT 6.1 review on fb9677d", "BLOCKING (GPT #593)", "round on ed6d3b0", "Review regressions") or describe the fix instead of the rule ("Before the fix only the assistant row was pushed", "exactly as the pre-fix bug").

Why it matters

A reader needs to know what the test holds and why. Review IDs and fix history say neither, and age badly once the fix is old.

Not a goal

What changed (motivation → approach → change)

Review-round and fix-history narration is rewritten as the present-tense rule; section headers name what the tests cover.

Before After
"GPT 6.1 review on fb9677d: an unhashable origin raised TypeError in the frozenset membership test and aborted the whole listing." "An unhashable origin would raise TypeError in the frozenset membership test and abort the whole listing."
"Before the fix both rows were written together AFTER the turn, so the transcript was empty at this point ..." "The user row is written BEFORE the turn: writing both rows after it leaves the transcript empty here ..."
"# ── Review regressions ──" / "# ── GPT on ab90c37: a concurrent press on the other button ──" "# ── /kirocrew agent edge cases ──" / "# ── a concurrent press on the other button ──"
"Absent before the drain had a budget of its own; set so a regression reaches the graceful bound below instead of failing on the attribute." "A short drain budget, so the run reaches the graceful bound below quickly. The attribute exists, so raising=False changes nothing."
"rfind -> 0, 0 or CAP -> CAP pre-fix (accidentally correct), but the explicit check keeps it robust." "rfind -> 0. A bare 0 or CAP also gives CAP here by accident; the explicit check is what holds it."

Ten files: 22 insertions, 22 deletions. goal_scan.py astcheck against the merge-base: files checked: 10 code-changed: 0 docstring-only: 0.

Backwards compatibility

Compatible: test-only change, comments only.

Removes nothing: no code, key, path, record or function is removed or renamed.

Tests

Proof: not provable -- comment-only change; the AST check shows no code difference

The ten touched files plus test/test_flake_pattern_ratchet.py pass on this branch (pytest -n 2 -p no:randomly: 629 passed). flake8, check_black_formatting.py, check_comment_history.py and check_brand_name.py pass; no added line is over 100 columns.

Manual verification

N/A — comment-only change.

Related Issues

N/A — part of the test comment quality pass.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

Rewrite review-round, SHA and fix-history narration in the slack tests
as present-tense rules with their reasons. Comments only; no test
behaviour changes.
@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 14, 2026 19:46
@iamwhatever
iamwhatever enabled auto-merge (squash) November 14, 2026 19:47
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ed8bc4438e84887e123531daf4574124f99f403d touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision ed8bc4438e84887e123531daf4574124f99f403d ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ed8bc4438e84887e123531daf4574124f99f403d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

The change only rewrites comments in Slack tests. It changes no behaviour, takes nothing away and renders nothing new, so there is no design to judge.

[DESIGN-REVIEWED] ed8bc44

Description read

[DESCRIPTION-READ] d9cc59c6b84cfbdc08fe119b027147714d092ca661194350f7b74f9ecb973883

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ed8bc4438e84887e123531daf4574124f99f403d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ed8bc44

Verdict parsed from the review's SHA-scoped output markers for commit ed8bc4438e84887e123531daf4574124f99f403d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ed8bc4438e84887e123531daf4574124f99f403d false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ed8bc4438e84887e123531daf4574124f99f403d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ed8bc44

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ed8bc4438e84887e123531daf4574124f99f403d false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 3% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

I read all ten hunks in the diff. Every change rewrites a comment and none touches code, assertions, fixtures or flake-ok markers, so test behaviour can't change. That matches the description's "comments only" claim and its non-goals. I checked the comments whose accuracy relies on code. The new note in test_slack_gateway.py says "the attribute exists, so raising=False changes nothing", and that holds: slack/gateway.py:263 imports PERSISTENCE_DRAIN_GRACE_SECS. The rewritten comments in the dashboard-sync, title and rendering tests describe the same assertions the old ones did, now as present-tense rules instead of fix history. No state, lifecycle, limit or trust boundary is touched. The backend test jobs were still running when I reviewed, but a comment-only diff can't change their outcome. The Opus, GPT and Design review lanes report no findings.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit ed8bc4438e · 44 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 14, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (10 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: comment-only edits inside ten Slack test files; no test logic, assertion or fixture changes.

@iamwhatever
iamwhatever merged commit 6178815 into main Nov 14, 2026
98 checks passed
@iamwhatever
iamwhatever deleted the crew/testoptimizer/test-quality-slack branch November 14, 2026 21:05
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants