Repository navigation
test: check real outcomes in nine new weak tests - #36562
Conversation
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. Only test files change, so there are no user-visible controls and no screenshots are needed. [UX-REVIEWED] 014815b Description read[DESCRIPTION-READ] 915c853e23192bc7421c3d82948280dbfc2512a319d89b5d168d2096af466eed That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Only test files change, and every added assertion checks a stored or rendered result that a planted bug would break. Not checked: the patch's source hunks match base Notes
What this change shipsInventory (8 items) — 8 justifiedIntent: make weak tests fail when the code they cover breaks. This is a FIX to the test suite, and no product behaviour changes.
[FIRST-PRINCIPLES-REVIEWED] 014815b Description read[DESCRIPTION-READ] 37fd8bfe35c8b32a067c172f82070311047bdbe3dfa62da2c6da8869bf92ca5a That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Test-only change: it adds outcome checks and swaps mocks for real stores. Nothing at design level to block. [DESIGN-REVIEWED] 014815b Description read[DESCRIPTION-READ] c50d6979775fc7898caed224ec82eb2dd3068511e99261af61863181cb43b95c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. Test-only PR that adds real-outcome assertions to twelve existing tests across eight files and swaps mocks for real stores: CronHistoryStore under tmp_path, an in-memory Channel, and a SessionManager via the existing _managed helper. Each new assertion was checked against the source: render.py keeps at-cap ids and values whole and puts block items under 'elements'; set_model sets _model itself, so served_model updates even with set_config_option mocked; an empty select keeps _kas_served_ids; history statuses are success/failure and get_job_history returns (rows, total); _stream_task posts only the stripped reply; mock providers make take_provider_context_reinjection return False, so the one-shot check is sound; sweep_repo's result lists only crews it wakes. No source changes, so no state, lifecycle, limit or trust surface moves. The red CI checks are API rate-limit and fail-closed infrastructure failures; the backend test shards were still pending, so whether the tests pass rests on the author's runs. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
The channel reply is read from a real Channel's messages, the compaction re-injection flag from a real SessionManager, and the dropped or own mention from the stored wake checks, instead of only mock calls.
|
Goal line edited on request: the branch now carries a second commit (014815b) with four more weak-test fixes, so the goal and counts say thirteen tests in eight files instead of nine in five. Scope is otherwise unchanged (test-only, non-security). |
bolichen97
left a comment
There was a problem hiding this comment.
Approving. I read the full diff at 014815b. It is tests only, eight files, with no product code. Nine weak assertions now check real outcomes:
- the rendered Slack blocks at each cap;
- the job's real history rows instead of a mocked
appendcount; - the wake-check
pendingmap and thewokenresult; served_modelandmodel_pin_refusedon the KAS handle;- the hand-off mark being cleared;
- a real session manager's one-shot
consume_needs_reinjection; - a real
Channel's posted messages; - the staged-update note's version.
Each stronger check still passes on this head (CI green).
Checks: I reran the failed Dependency License Gate and the readiness signal once each. The latest run of every check on this head is now green; the remaining failed or cancelled entries are older runs superseded on the same head.
Not checked: I ran no tests locally.
Merge every change to the built-in Issue Radar since 4e4bba39 three-way onto this tree, so a host that adopts this repository runs what the wheel copy runs today, and release it as 0.1.2 for the Kiro Crew pin (kirodotdev/KiroCrew#26967). The changes, all in kirodotdev/KiroCrew: - kirodotdev/KiroCrew#31713: a crew never merges or arms auto-merge on GitLab. may_arm_auto_merge decides the one merge rule the nudge states, the brief defers to it, a crew update that changes auto_merge refreshes the armed loop's prompt, and the crew editor shows the toggle off and locked on GitLab with its reason. - kirodotdev/KiroCrew#32503: every bad-body 400 carries a code, invalid_json or body_not_object. - kirodotdev/KiroCrew#37301: the generic noun crew in each catalog's own script in bn, ja, ko, zh-CN and ru. - Test-only: kirodotdev/KiroCrew#37262, kirodotdev/KiroCrew#36562, kirodotdev/KiroCrew#31863 and kirodotdev/KiroCrew#36062. The test host moves to the Kiro Crew commit these tests expect, and the built-in manifest fixture is the wheel manifest at 0.1.2. PORTING.md lists each change. Co-authored-by: KiroCrewCaptain <312312717+KiroCrewCaptain@users.noreply.github.com>
Problem / Motivation
Goal: Give thirteen weak tests added since
aa08664c56a check on a real outcome (stored state, a real store row, or the rendered output), so a bug in the code they cover makes them fail.goal_scan.py weakwent from 870 ataa08664c56to 978 atc22557cd83. This PR covers thirteen of the new ones, in eight test files for non-security code: KAS model selection, wheel-update staging, the card renderer's caps, cron history rows, the prompt handoff, channel replies, compaction re-injection and issue-radar mention wakes. Each ended after the call with nothing checked, or only a mock's call count.Why it matters
A test that only shows the code ran passes on a broken result. Each added check pins what the code is meant to leave behind, and a planted bug in the source now fails the test (table below).
Not a goal
What changed (motivation → approach → change)
Every row is a stored-result, store-row or return-value check; there are no log-line or wire checks.
test/test_kas_model_select_capture.pymodel_pin_refused,served_modeland the kept_kas_served_idsafterset_modeltest/test_updates_never_restart_a_running_gateway.pytest/test_channel_cards_render.pytest/test_dashboard_stale_wake.pyCronHistoryStorerow count and status, in place of anAsyncMockcall counttest/test_prompt_handoff.pytest/test_turn_collectors_steer_ack.pyChannel's messages has the steering ack strippedtest/test_reply_format_once.pySessionManagerholds the one-shot re-injection flag under the turn's own session keytest/test_issue_radar_crew_runtime_wake.pygoal_scan.py weakon the same base (c22557cd83):_over)Backwards compatibility
Compatible: test-only change; no code, behaviour or contract moves.
Removes nothing: every existing check stays; two
AsyncMockstand-ins for the history store are replaced by the real store.Tests
Proof: not provable -- test-only change; the planted-bug runs below show each new check fails on a broken source while main's copy of the test passes.
model_pin_refusedautoas servedsuccessfailureFor the last three rows, main's test already caught the bug through a mock; the branch adds the stored result, so the check does not rest on the mock alone.
Determinism: the first five files (167 node ids) pass in three seeded shuffled orders with
-n0; the last three files (166 node ids) pass in three more, and the four changed tests pass five runs in a row.test/test_flake_pattern_ratchet.pypasses. flake8, black and isort are clean.Manual verification
N/A — test-only change.
Related Issues
N/A — part of the weak-test cleanup.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.