Skip to content

test(shared-crew): compare background audit rows without assuming write order - #29456

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/shared-question-audit-order
Nov 4, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/shared-question-audit-order

Conversation

@jeeshofone

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop the shared-crew audit tests from failing when two background audit writes finish in the opposite order.

test_the_losing_answer_is_audited_as_a_conflict failed in CI with the audit rows as conflict then applied. Three tests in test_shared_crew_action_audit.py make the same order assumption and fail the same way under a forced thread-scheduling shuffle.

Fixes #29455

Why it matters

Any PR whose Backend Tests shard picks up these tests can go red for a reason unrelated to its diff, which costs a re-run and a log read each time.

Not a goal

  • Making audit write order deterministic in the product. Nothing reads audit rows by arrival order, and serialising writes would make one slow write delay the next.
  • Changing schedule_action_audit, record_action or the security event schema.

What changed (motivation → approach → change)

  • Symptom: two audit rows asserted in a fixed order arrive reversed now and then.
  • Root cause: schedule_action_audit starts one task per record and record_action writes each through asyncio.to_thread. Its docstring says it records "without making its caller wait for the write", so it promises when a write is not awaited, not what order writes land in. Two worker threads can reach the writer in either order. Order is unspecified by design, so the fix belongs in the tests.
  • Change: the tests sort the captured rows by outcome before comparing (applied sorts before conflict). They still assert the exact rows and count, so a duplicate row from the winner's replay still fails. The record's at timestamp was rejected as a sort key because two records from one handler can share a clock reading. No sleeps or retries were added.

Backwards compatibility

Compatible: test-only change.

Tests

  • Before: with a local plugin that adds a random 0-20 ms delay inside each asyncio.to_thread call, the four affected tests failed 86 of 200 runs (--count=50).
  • After, same plugin: 200 of 200 pass. Without the plugin, both files at --count=50: 1000 of 1000 pass.
  • test/test_shared_crew_*.py, test_error_code_contract.py and test_ws_event_scoping.py: 1356 passed, 2 skipped.
  • black, flake8 and semgrep (semgrep/ rules) are clean on both changed files.

Manual verification

N/A: test-only change.

Screenshots / video

N/A: no UI change.

Sibling sweep

Searched test/ for assertions on events captured from audit._default_writer. Only the three listed assertions compare more than one row in order. The single-row assertions and test_shared_crew_recovery_bounds.py (session notes, not audits) are unaffected.

Pattern harvest

Assertions over rows written by fire-and-forget background tasks should compare a sorted or set view unless the code serialises the writes.

@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners November 4, 2026 04:24
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 74fc48c2b2bdfe86abd5bd01df5d3cf8e2842143 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 74fc48c2b2bdfe86abd5bd01df5d3cf8e2842143 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 74fc48c2b2bdfe86abd5bd01df5d3cf8e2842143 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Test-only change. Ordering is still checked through a defined sort key, and the row count and contents are still asserted exactly.

[DESIGN-REVIEWED] 74fc48c

Description read

[DESCRIPTION-READ] c812c0f2c0b43ca83d77f17617190458b6526074d581ecb377981879a4938b7b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 74fc48c2b2bdfe86abd5bd01df5d3cf8e2842143 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 74fc48c

Verdict parsed from the review's SHA-scoped output markers for commit 74fc48c2b2bdfe86abd5bd01df5d3cf8e2842143.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 74fc48c2b2bdfe86abd5bd01df5d3cf8e2842143 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 74fc48c2b2bdfe86abd5bd01df5d3cf8e2842143 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 74fc48c

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 74fc48c2b2bdfe86abd5bd01df5d3cf8e2842143 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 4, 2026
…te order

Each audit is written by its own background task through asyncio.to_thread,
so two writes can reach the writer in either order. Sort captured rows by
outcome before comparing.

Fixes #29455
@jeeshofone
jeeshofone force-pushed the fix/shared-question-audit-order branch from 7addea8 to 74fc48c Compare November 4, 2026 12:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 4, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 4, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Shard 6 is being cancelled at the runner's 60-minute limit, not by a test failure. A test-only diff runs the full test corpus (#23450): this PR's shard 6 ran 43,693 tests (job 126675848839: all passed, then cancelled at about 58 minutes) plus the leaf corpus gates over 810 files. Main's shard 6 push run selects 8,086 tests and finishes in about 45 minutes. #23461 tracks the fix. I'll rebase and re-run CI once it lands.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 4, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 4, 2026 20:06

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change: shared-crew audit assertions sort rows by outcome instead of assuming background-task write order.

@KiroCrewCaptain
KiroCrewCaptain merged commit fd9c2b8 into main Nov 4, 2026
355 of 371 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/shared-question-audit-order branch November 4, 2026 20:08
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_the_losing_answer_is_audited_as_a_conflict is flaky: audit rows asserted in write order

2 participants