Skip to content

test: wake gateway loops only after run_gateway built its orchestrator - #34589

Merged
RohanK6 merged 1 commit into
mainfrom
crew/conductor/gateway-entry-sweep-flake
Nov 10, 2026
Merged

RohanK6 merged 1 commit into
mainfrom
crew/conductor/gateway-entry-sweep-flake

Conversation

@RohanK6

@RohanK6 RohanK6 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_gateway_process_entry.py passes the same way every run, whichever executor thread finishes first.

The tests run run_gateway with a fake asyncio whose first sleep returns at once. That lets each hourly loop wake once. But the hourly sweep's session work-dir step reads orchestrator, and run_gateway binds that only after await asyncio.to_thread(cfg.load_credentials). When the sweep task woke before the credentials thread returned, the step raised NameError, its own except Exception logged and swallowed it, and the stub was never called. test_a_failing_step_is_logged_and_the_next_step_still_runs then failed on Linux CI with "Called 0 times", as #34528 records.

Why it matters

The flake fails Backend Tests (3.12, 3) on unrelated PRs (it hit #33713) and costs every hit a rerun. The issue author confirmed the race by making the credentials thread slow.

Not a goal

  • Changing process_entry.py. In production, the first sleep is an hour, so the orchestrator is always bound by then.

What changed (motivation → approach → change)

Symptom: the session work-dir stub is called 0 times. Root cause: the fake sleep did not model the real hour-long sleep, so a wake could beat boot. Change: _OneWakeAsyncio.sleep's first wake now waits on a booted event, with a 30s bound. _seams sets that event when GatewayOrchestrator is built, so every loop wakes after the orchestrator exists, as it does in production.

Backwards compatibility

Compatible: test-only. Each loop still wakes exactly once, and every assertion is unchanged.

Removes nothing: no assertion or seam is removed.

Tests

  • test/test_gateway_process_entry.py: all three tests pass.
  • Forced reproduction, run as a throwaway probe and not committed: making cfg.load_credentials take 0.5s on its thread fails the session-dir assertion on origin/main, and passes with this change.

Proof: not provable -- test-only change

Manual verification

N/A — unit coverage sufficient: the forced-slow-credentials probe above exercises the race in both directions.

Related Issues

Fixes #34528

Sibling sweep

Swept: git grep -n "class _OneWakeAsyncio" -- test/
Siblings: none -- only this test module fakes the gateway's loop sleeps

Pattern harvest

Rule candidate: review-prompt
Pattern: a fake sleep that returns at once lets a "sleeps first" loop run before the state its real interval guaranteed

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

The process-entry tests' fake sleep returned at once, so the hourly
sweep could run while run_gateway still waited on its credentials
thread. The session work-dir step then read an unbound orchestrator,
swallowed the NameError and never called its stub: a race that failed
Linux CI. The fake's first sleep now waits until the orchestrator is
built, which is what the real hour-long sleep guarantees.

Fixes #34528
@RohanK6
RohanK6 requested review from a team and KiroCrewCaptain as code owners November 10, 2026 15:21
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision ed432c14fceee852cfcd8a15686dd4a1c5d35b27 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ed432c14fceee852cfcd8a15686dd4a1c5d35b27 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ed432c14fceee852cfcd8a15686dd4a1c5d35b27 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix is test-only and ties each fake first wake to the orchestrator being built, which is the condition the real hour-long sleep guarantees.

[DESIGN-REVIEWED] ed432c1

Description read

[DESCRIPTION-READ] d851bef8caab3e80f7ec12ce6eb31c9c2e8a0fb4f620f9b65eb36c7c14661bb3

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ed432c14fceee852cfcd8a15686dd4a1c5d35b27 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ed432c1

Verdict parsed from the review's SHA-scoped output markers for commit ed432c14fceee852cfcd8a15686dd4a1c5d35b27.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ed432c14fceee852cfcd8a15686dd4a1c5d35b27 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ed432c14fceee852cfcd8a15686dd4a1c5d35b27 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ed432c1

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ed432c14fceee852cfcd8a15686dd4a1c5d35b27 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 10, 2026
@RohanK6
RohanK6 enabled auto-merge (squash) November 10, 2026 17:04

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (1 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the gateway process-entry test now wakes its loops only after run_gateway has built the orchestrator, removing a race in the test harness; test file only.

@RohanK6
RohanK6 merged commit 0807cb6 into main Nov 10, 2026
97 checks passed
@RohanK6
RohanK6 deleted the crew/conductor/gateway-entry-sweep-flake branch November 10, 2026 17:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 10, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving ed432c1. Test-only change for the flaky test_gateway_process_entry (#34528). The fake asyncio.sleep now holds a task's first wake until booted is set, and booted is set when the test's GatewayOrchestrator is built. That stands for the hour a real first sleep lasts, so the hourly sweep can no longer read orchestrator before run_gateway has bound it after its credentials thread returns. The wait is bounded at 30 s, so a regression fails rather than hangs. No product code changes, and no regex. All 97 checks are green. Auto-merge is left to the author.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_a_failing_step_is_logged_and_the_next_step_still_runs on Linux (session-dir sweep races run_gateway)

3 participants