Skip to content

Flaky: test_link_through_a_worker_attach_dial_delivers_and_finishes reports child exit 255 (Linux, shard 7) #23402

Closed

Description

@cixuuz

What

On Backend Tests (3.12, 7) (Linux, Python 3.12.15), test/test_remote_session_client.py::test_link_through_a_worker_attach_dial_delivers_and_finishes fails at line 114:

>       assert box["code"] == 0
E       assert 255 == 0

Captured stderr: crew-worker: child exited 255; worker exiting. Captured log:

WARNING asyncio:unix_events.py:1414 Unknown child process pid 22357, will report returncode 255
WARNING asyncio:unix_events.py:1424 Loop <_UnixSelectorEventLoop running=False closed=True> that handles pid 22359 is closed

Run: https://github.com/kirodotdev/KiroCrew/actions/runs/39923187276/job/120317052923 (PR #22926, head 9b3e2668a169, a frontend-only diff that touches no Python).

The same node id is red on four other heads since the test landed on main in #19833 (ab2f15c57b, 2026-10-23 07:18Z):

Why it is a flake

Five unrelated branches, one of them with no backend change at all, and 10 of 10 local runs of the node id pass (pytest -n0). The test runs the Worker on a background thread inside its own asyncio.run() loop (_worker, line 75) while the test body runs a second asyncio.run() on the main thread for the RemoteLink; both loops spawn subprocesses. The asyncio warnings say the child watcher reaped the worker's child from a loop that no longer handles it, so the worker sees exit 255 instead of the child's real status. Which loop reaps first depends on timing, so the result depends on load (the xdist worker was 27 minutes into the shard).

Suggested direction

Keep both subprocess-spawning loops in one thread (run the Worker and the RemoteLink on the same loop, or drive the worker as a separate python -m process the test waits on), so one child watcher owns every child pid the test creates. Then a reaped child always reports its real exit status.

Filed from the kirocrew-prepare-pr flake ledger step while driving #22926.

Activity

  1. added
    tier:T1Bug: behaviour does not match intent; fix keeps the design
    on Oct 23, 2026
  2. KiroCrewCaptain commented on Oct 23, 2026

    @KiroCrewCaptain
    Collaborator

    Thanks for the report. This issue has been triaged by Crew Captain and will be assigned to a maintainer, who will follow up here.

  3. buluoray commented on Oct 23, 2026

    @buluoray
    Contributor

    🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Closing as already fixed on main by #23403 (5f77c3f85c, merged 2026-10-23 12:03Z), which targets this exact test and symptom.

    Cause, as traced in src/kiro_crew/remote_session/worker.py: after the pump reads EOF from the echo child's stdout, Worker.run() saw returncode is None and called terminate(). Popen.send_signal() polls the pid first, and that poll can reap the child before asyncio's child watcher does. The watcher then gets ECHILD and reports 255 ("Unknown child process pid ..., will report returncode 255", as in the captured log). The fix calls terminate() only when the stop event fired before stdout EOF; otherwise it waits (bounded by stop_grace) for the child's own exit, so the real status is collected.

    All five runs cited here predate that merge (the PR #22926 run started 08:13Z). git merge-base --is-ancestor 5f77c3f85c origin/main passes on main 7d63942c4c.

    Possibly related to #23435 (same 255 symptom, but on the retire path in test/test_remote_session.py; I have not checked whether #23403 also covers that path). Please reopen if this node id fails again on a head that contains 5f77c3f85c.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething is not workingtier:T1Bug: behaviour does not match intent; fix keeps the design

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions