Skip to content

test: check real outcomes in nine new weak tests - #36562

Merged
iamwhatever merged 2 commits into
mainfrom
crew/testoptimizer/weak-r3
Nov 13, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
crew/testoptimizer/weak-r3

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Give thirteen weak tests added since aa08664c56 a check on a real outcome (stored state, a real store row, or the rendered output), so a bug in the code they cover makes them fail.

goal_scan.py weak went from 870 at aa08664c56 to 978 at c22557cd83. This PR covers thirteen of the new ones, in eight test files for non-security code: KAS model selection, wheel-update staging, the card renderer's caps, cron history rows, the prompt handoff, channel replies, compaction re-injection and issue-radar mention wakes. Each ended after the call with nothing checked, or only a mock's call count.

Why it matters

A test that only shows the code ran passes on a broken result. Each added check pins what the code is meant to leave behind, and a planted bug in the source now fails the test (table below).

Not a goal

  • No source change: only test files change.
  • No test is deleted or renamed.
  • No security, HOLD or claimed test area is touched.

What changed (motivation → approach → change)

Every row is a stored-result, store-row or return-value check; there are no log-line or wire checks.

Test file Tests Real outcome now checked
test/test_kas_model_select_capture.py 3 model_pin_refused, served_model and the kept _kas_served_ids after set_model
test/test_updates_never_restart_a_running_gateway.py 1 the staged restart note still names the version after a repeat cycle
test/test_channel_cards_render.py 2 at the cap, a field is sent whole and every item or button is sent
test/test_dashboard_stale_wake.py 2 the real CronHistoryStore row count and status, in place of an AsyncMock call count
test/test_prompt_handoff.py 1 a helper awaited inside its own turn leaves no handoff mark
test/test_turn_collectors_steer_ack.py 1 the reply stored in a real Channel's messages has the steering ack stripped
test/test_reply_format_once.py 1 a real SessionManager holds the one-shot re-injection flag under the turn's own session key
test/test_issue_radar_crew_runtime_wake.py 2 the stored wake checks keep no pending ask after an opt-out or the crew's own mention

goal_scan.py weak on the same base (c22557cd83):

Scan list Before After
trusted helpers as of the first count 978 965 (13 fixed)
with four more planted-bug-checked helpers 967 956 (two card tests already count through _over)

Backwards compatibility

Compatible: test-only change; no code, behaviour or contract moves.

Removes nothing: every existing check stays; two AsyncMock stand-ins for the history store are replaced by the real store.

Tests

Proof: not provable -- test-only change; the planted-bug runs below show each new check fails on a broken source while main's copy of the test passes.

Planted bug (source restored after each run) This branch main's test
a withheld KAS pin is not recorded in model_pin_refused fails passes
an applied KAS pick records auto as served fails passes
the KAS pick records an upper-cased spelling fails passes
a repeat update cycle drops the staged restart note fails passes
an at-cap card field is cut by one character (label, value, action_id) fails passes
the context block drops its last item fails passes
the actions block drops its last button fails passes
a failed quiet cron run is stored as success fails passes
every cron history row is stored as failure fails passes
a helper awaited inside its own turn leaves a handoff mark fails passes
the channel reply keeps the steering ack fails fails
the compaction restore marks a different session key fails passes
an opt-out mid-sweep keeps its pending asks on disk fails fails
the crew's own mention is not filtered out fails fails

For the last three rows, main's test already caught the bug through a mock; the branch adds the stored result, so the check does not rest on the mock alone.

Determinism: the first five files (167 node ids) pass in three seeded shuffled orders with -n0; the last three files (166 node ids) pass in three more, and the four changed tests pass five runs in a row. test/test_flake_pattern_ratchet.py passes. flake8, black and isort are clean.

Manual verification

N/A — test-only change.

Related Issues

N/A — part of the weak-test cleanup.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 13, 2026 10:14
@iamwhatever
iamwhatever enabled auto-merge (squash) November 13, 2026 10:14
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 014815b5f471889e6af051b76160435a9767a42c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 014815b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 014815b5f471889e6af051b76160435a9767a42c false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 014815b5f471889e6af051b76160435a9767a42c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. Only test files change, so there are no user-visible controls and no screenshots are needed.

[UX-REVIEWED] 014815b

Description read

[DESCRIPTION-READ] 915c853e23192bc7421c3d82948280dbfc2512a319d89b5d168d2096af466eed

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 014815b5f471889e6af051b76160435a9767a42c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Only test files change, and every added assertion checks a stored or rendered result that a planted bug would break.

Not checked: the patch's source hunks match base ca956367 (#33965), so I treated them as base drift and reviewed only the 8 test files.

Notes

  • Item 0 — framing: the title says "nine new weak tests" but the Goal says "thirteen weak tests". The diff has 13 tests in 8 files, so the title is the wrong one. Nothing ships differently because of it.

What this change ships

Inventory (8 items) — 8 justified

Intent: make weak tests fail when the code they cover breaks. This is a FIX to the test suite, and no product behaviour changes.

  1. KAS model-select tests check served_model, model_pin_refused and the kept served list — justified
  2. Wheel-update test checks the staged restart note still names 9.9.9 — justified
  3. Card cap tests check that a field at the cap is sent whole and that the item and button counts match the cap — justified
  4. Cron quiet-run tests read real CronHistoryStore rows (count and status) in place of an AsyncMock call count — justified
  5. Prompt-handoff test checks that no handoff mark is left — justified
  6. Steer-ack test reads the reply stored in a real Channel's messages — justified
  7. Compaction re-injection test uses a real session manager and checks the one-shot flag under the turn's own session key — justified
  8. Issue-radar tests check that no pending ask is stored after an opt-out or the crew's own mention — justified

[FIRST-PRINCIPLES-REVIEWED] 014815b

Description read

[DESCRIPTION-READ] 37fd8bfe35c8b32a067c172f82070311047bdbe3dfa62da2c6da8869bf92ca5a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 014815b5f471889e6af051b76160435a9767a42c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Test-only change: it adds outcome checks and swaps mocks for real stores. Nothing at design level to block.

[DESIGN-REVIEWED] 014815b

Description read

[DESCRIPTION-READ] c50d6979775fc7898caed224ec82eb2dd3068511e99261af61863181cb43b95c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 014815b5f471889e6af051b76160435a9767a42c — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 014815b

Verdict parsed from the review's SHA-scoped output markers for commit 014815b5f471889e6af051b76160435a9767a42c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 014815b5f471889e6af051b76160435a9767a42c false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Test-only PR that adds real-outcome assertions to twelve existing tests across eight files and swaps mocks for real stores: CronHistoryStore under tmp_path, an in-memory Channel, and a SessionManager via the existing _managed helper. Each new assertion was checked against the source: render.py keeps at-cap ids and values whole and puts block items under 'elements'; set_model sets _model itself, so served_model updates even with set_config_option mocked; an empty select keeps _kas_served_ids; history statuses are success/failure and get_job_history returns (rows, total); _stream_task posts only the stripped reply; mock providers make take_provider_context_reinjection return False, so the one-shot check is sound; sweep_repo's result lists only crews it wakes. No source changes, so no state, lifecycle, limit or trust surface moves. The red CI checks are API rate-limit and fail-closed infrastructure failures; the backend test shards were still pending, so whether the tests pass rests on the author's runs.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 014815b5f4 · 100 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@iamwhatever
iamwhatever disabled auto-merge November 13, 2026 10:57
The channel reply is read from a real Channel's messages, the compaction
re-injection flag from a real SessionManager, and the dropped or own
mention from the stored wake checks, instead of only mock calls.
@iamwhatever
iamwhatever enabled auto-merge (squash) November 13, 2026 11:02
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Goal line edited on request: the branch now carries a second commit (014815b) with four more weak-test fixes, so the goal and counts say thirteen tests in eight files instead of nine in five. Scope is otherwise unchanged (test-only, non-security).

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 13, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I read the full diff at 014815b. It is tests only, eight files, with no product code. Nine weak assertions now check real outcomes:

  • the rendered Slack blocks at each cap;
  • the job's real history rows instead of a mocked append count;
  • the wake-check pending map and the woken result;
  • served_model and model_pin_refused on the KAS handle;
  • the hand-off mark being cleared;
  • a real session manager's one-shot consume_needs_reinjection;
  • a real Channel's posted messages;
  • the staged-update note's version.

Each stronger check still passes on this head (CI green).

Checks: I reran the failed Dependency License Gate and the readiness signal once each. The latest run of every check on this head is now green; the remaining failed or cancelled entries are older runs superseded on the same head.

Not checked: I ran no tests locally.

@iamwhatever
iamwhatever merged commit 4d445d8 into main Nov 13, 2026
198 of 217 checks passed
@iamwhatever
iamwhatever deleted the crew/testoptimizer/weak-r3 branch November 13, 2026 13:58
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 13, 2026
KiroCrewCaptain added a commit to KiroCrewCaptain/kirocrew-app-issue-radar that referenced this pull request Nov 14, 2026
Merge every change to the built-in Issue Radar since 4e4bba39 three-way
onto this tree, so a host that adopts this repository runs what the
wheel copy runs today, and release it as 0.1.2 for the Kiro Crew pin
(kirodotdev/KiroCrew#26967).

The changes, all in kirodotdev/KiroCrew:

- kirodotdev/KiroCrew#31713: a crew never merges or arms auto-merge on
  GitLab. may_arm_auto_merge decides the one merge rule the nudge
  states, the brief defers to it, a crew update that changes
  auto_merge refreshes the armed loop's prompt, and the crew editor
  shows the toggle off and locked on GitLab with its reason.
- kirodotdev/KiroCrew#32503: every bad-body 400 carries a code,
  invalid_json or body_not_object.
- kirodotdev/KiroCrew#37301: the generic noun crew in each catalog's
  own script in bn, ja, ko, zh-CN and ru.
- Test-only: kirodotdev/KiroCrew#37262, kirodotdev/KiroCrew#36562,
  kirodotdev/KiroCrew#31863 and kirodotdev/KiroCrew#36062.

The test host moves to the Kiro Crew commit these tests expect, and
the built-in manifest fixture is the wheel manifest at 0.1.2. PORTING.md
lists each change.

Co-authored-by: KiroCrewCaptain <312312717+KiroCrewCaptain@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants