Repository navigation
Flaky CI Failure: Android Integration Tests — Run integration tests times out after 200/202 tests complete #3953
Description
Flaky CI Failure: Android Integration Tests — Run integration tests times out after 200/202 tests complete
Workflow: Test
Failed run: https://github.com/xmtp/libxmtp/actions/runs/31221165776
Commit: 1b762b9
Failed jobs: test-android / Integration Tests (Android)
Summary
The Run integration tests step in the Android integration test job timed out after hitting the 20-minute step limit. The emulator completed 200 of 202 tests (with 3 skipped and 0 failed), but the final 2 tests hung indefinitely. Tests 195–200 showed significantly slower execution (~1.5 minutes each vs. ~10 seconds per test earlier in the run), suggesting the suite was slowing down toward the end before fully stalling.
Error Details
emulator-5560 - 14 Tests 194/202 completed. (3 skipped) (0 failed) 21:58:02
emulator-5560 - 14 Tests 195/202 completed. (3 skipped) (0 failed) 21:59:02 (+60s)
emulator-5560 - 14 Tests 196/202 completed. (3 skipped) (0 failed) 22:00:02 (+60s)
emulator-5560 - 14 Tests 197/202 completed. (3 skipped) (0 failed) 22:01:32 (+90s)
emulator-5560 - 14 Tests 198/202 completed. (3 skipped) (0 failed) 22:03:02 (+90s)
emulator-5560 - 14 Tests 199/202 completed. (3 skipped) (0 failed) 22:04:32 (+90s)
emulator-5560 - 14 Tests 200/202 completed. (3 skipped) (0 failed) 22:06:12 (+100s)
##[error]The action 'Run integration tests' has timed out after 20 minutes. 22:06:45
Last named skipped tests (around position 159–162):
org.xmtp.android.library.HistorySyncTest > testSyncDeviceArchive[emulator-5560 - 14] SKIPPED
org.xmtp.android.library.HistorySyncTest > testSyncMessages[emulator-5560 - 14] SKIPPED
org.xmtp.android.library.HistorySyncTlsTest > testArchiveRoundTripOverTls[emulator-5560 - 14] SKIPPED
Analysis
The step-level 20-minute timeout was hit with 2 tests still running, 0 tests explicitly failed. The progressive slowdown (from ~10s/test to ~90s/test near the end) indicates one or more tests were waiting on a stream, consent update, or async event that never arrived rather than crashing outright. The identity of the final 2 hanging tests is not visible in the log because no completion or failure line was emitted before the timeout. This is a test/code failure (likely a flaky streaming or sync test), not an infrastructure failure.
Reported by Flaky Failure Watcher
Activity
github-actions commentedon Aug 7, 2026
Recurrence — 2026-08-07
Failed run: https://github.com/xmtp/libxmtp/actions/runs/31223948308
Commit: 3d28107
Same timeout pattern as the original report. The
Run integration testsstep hit the 20-minute limit with 201/202 tests completed (3 skipped, 0 failed). The last few tests slowed dramatically before stalling:This run got one further (201/202 vs 200/202 previously) before timing out, but the same progressive slowdown pattern is present. One test remained stuck indefinitely.
insipx commentedon Aug 7, 2026
Data point from a different cause that produced the identical signature, in case it helps narrow the mechanism.
On #3954 I hit this exact shape — 200/202 completed,
0 failed, then a step timeout — and there it was fully explained by extra per-client network I/O. That PR (transiently) had a 3-second liveness-check throttle, so every client build more than 3s after registration did an extrafetch_key_packagesround trip. The tell was in the progress lines: the run is brisk to ~194/202 (10-20 tests per 10s tick), then the last handful crawl to roughly one test per 90s before the step is killed. Removing the extra per-build network call fixed it, and that job is green again.I'm explicitly not claiming that is this issue's root cause — this one reproduces on unrelated branches, so something else is driving it. But the two share a mechanism worth checking: the tail crawl is not uniform slowness, it is the last few tests degrading sharply, which looks much more like a resource that accumulates per-test and saturates near the end (sockets/connections, emulator-side network state, or a backend the emulator talks to getting slower under accumulated load) than like any single slow test. Two concrete suggestions for whoever picks this up:
0 failed, no test is actually asserting anything wrong — the budget is being consumed elsewhere. Worth confirming whether the emulator or the shared backend is the saturating side before touching test code.Happy to be pinged if the accumulation theory pans out, but I'm not picking this up.
insipx commentedon Aug 8, 2026
Correcting my comment above — I was wrong, please disregard the causal claim.
I said my case was "fully explained by extra per-client network I/O" and that removing it made the job green. Both halves are false, and I should not have asserted them before re-running.
I removed the extra per-build
fetch_key_packageson #3954 and the Android job failed again with the identical signature: brisk to 194/202, then a tail crawl,0 failed, step timeout. It reached 201/202 instead of 200/202, which is noise, not a fix. My change neither caused this nor fixed it. Independently confirmed it is pre-existing:test-android / Integration Tests (Android)failed onmainin 4 of the last 5 failing runs, including runs31223948308and31221165776from earlier today, before my branch existed.What I do think is still worth keeping from that comment is one observation, now stated as an observation rather than a cause. The tail crawl has an exact 90-second cadence. From the run on my branch:
Six consecutive intervals at exactly 90s, to the second. That regularity argues against the resource-saturation theory I floated earlier — saturation would degrade raggedly. A precise, repeating 90s per test looks like a fixed timeout being hit and then something proceeding: a client/network operation with a 90s deadline, a retry ceiling, or a poll interval, in whichever tests happen to run last in this shard's ordering. Someone who knows the Android harness could likely map 90s onto a specific configured timeout quickly.
Apologies for the noise on the earlier comment.
github-actions commentedon Aug 20, 2026
Recurrence — 2026-08-20
Failed run: https://github.com/xmtp/libxmtp/actions/runs/32408859442
Commit: c30ac5f
Same timeout pattern as before, but this time 201/202 tests completed (vs 200/202 in the original report) before the 20-minute step limit was hit.
The slowdown pattern at the end of the suite persists — the last few tests each take 1–2 minutes, then the final test hangs indefinitely. The 0 failed count indicates no explicit test assertion failures, just a hang.
neekolas commentedon Oct 14, 2026
🤖 No close yet. The job was restructured (now with a 30-minute step timeout), but it still runs the same suite, and the cause of the 90-second-per-test tail crawl was never found. If there is no recurrence after more runs on the new job, close it then.