Repository navigation
Description
Problem
From the 25 Oct to 4 Nov review of the delivery pipeline (lesson ledger, CI history, push queue log, rented machine history): the checks catch real bugs, but several kinds of failure are found only at release time or by users, after hours of work have been built on top.
- Release-only paths cost about 6.5 hours (ledger: the release's own CHANGELOG section is read only at prepare; universal archives differ between macOS and Linux and only prepare compares them).
- A Browser crash at about 485 s escaped because no qualification keeps a session open that long (ledger 876).
- Update fails on Homes with the local AI engine: sealed llama.cpp tree cannot be replaced (2.0.1) #515 (Mac update with the local AI engine fails on the sealed engine tree) was found after 2.0.0 shipped: no installed update journey has the engine present.
- Smoke scripts that no gate runs rot:
browser-source-home-config-smoke.shalready fails on develop unnoticed; the Browser settlement smoke is run by hand only. - A Rust change passes the full Linux remote check and then runs the same full gate again on the Mac at push.
Done means
- Qualification (before Anders signs) runs the release preparation validator against the assembled artifacts, so a missing CHANGELOG section or a platform-dependent archive fails before signing, not after the seed upgrade.
- The installed Mac update journey runs with the local AI engine installed and fails on today's Update fails on Homes with the local AI engine: sealed llama.cpp tree cannot be replaced (2.0.1) #515 behaviour.
- Browser qualification keeps one session open at least 10 minutes with input, then Close and reopen, on the Mac.
- Every smoke script is either run by the gate or CI, or deleted.
- When a full Linux remote-check PASS receipt exists for exactly the pushed tree, the Mac pre-push gate runs only what differs on macOS (crates with macOS cfg code) plus the cheap steps; any tree change runs the full gate as today.
- Each item has a check that failed before its change and passes after.
Source audit — 6 November 2026
Candidate: develop f50e448b3e98c813106f0b17b2bf5d7fee8f5f18, tree 8fa9e56488d5a9b7c9968689b2b7a3da790ce9ed. Root checked a source export against all 1,972 Git blob identities and executable modes.
Two checks still fail outside test-source-gate.sh, just verify and CI's command lists:
node scripts/check-system-esp-projections.mjsstops at line 36 because it expectsbinding.executable !== truein System's removed Apps & Services inventory. The projection-helper cases after that assertion do not execute. Keep the adapter behaviour tests, and move any needed executable-binding refusal check to the current Runtime consumer.bash scripts/browser-source-home-config-smoke.shstops at line 304 because it requires a legacy Linux hibernation variable that the native transport configuration omits. Update its native Linux assertions for the current generator, retaining the Runtime networking, readiness, session and profile boundaries. Review the remaining legacy launcher/pool assertions in the same fixture before accepting it.
docs/HOME_MOBILE.md:172 still claims just verify runs the deleted home-entropy-check.mjs. Point the documentation to the actual checks and retain each accepted phone contract in a runnable check. The current AGENTS.md already names the replacement product-data checks correctly.
Root independently passed 485 Node tests with zero skips, the Home/UI/data/contract checks listed in #535, and both required Rust format checks. The separate CI release-policy suite stopped before running tests because the plain source export has no Git index; this is an audit-environment limitation. Root left builds, full pre-push, installed journeys and target tests with Main.
Next owner: Main. Repair these checks within this issue, show fail-before/pass-after, and put each retained smoke in an existing gate or delete it under the accepted criterion. This audit adds no second active lane and leaves the installed, long-session and human acceptance gates open.
Two confirmed coverage gaps on 2.0.1-rc.1
On 30b166b05e1e4e29e97597aad25a688a19380ad1, Root checked the source behind the completed coverage audit. The private stdin preflight test requires Apple silicon, the workspace CI test runs on Ubuntu, and the local pre-push test planner selects only libraries and binaries. The audit reports a direct Mac pass for the preflight test. That result does not establish gate coverage.
Main owns the gate repair. Add the named preflight test to the existing Apple silicon test path and record a completed result for that test on the change's commit. The Library performance smoke has no gate caller. The existing product-ui-browser job already installs the Playwright and Chromium it needs. Check the smoke there and retain its large-folder rendering, cache reuse and upload assertions. Validate the timing thresholds on that runner before making them blocking. Keep the reported missing-Playwright attempt and the earlier CI failures in the record.
Release speed plan (11 to 12 Nov)
2.0.1-rc.2 took 28.7 h from first freeze to signature: 5 Mac gates and 4 CI and package cycles, three of them thrown away (about 7.5 h) for problems a seconds-long check would have caught. One pass from freeze to ready-to-sign is about 2.5 h: Mac gate 38 min, CI and package about 76 min (the Mac N and N+1 build is 72 min), prepare 37 min.
- Fast-forward main to the tested commit instead of a release PR, saving about 65 min of re-run CI (
main-ff.sh, used for 2.0.1). - Preflight before the Mac gate:
preflight.shruns in about 10 s and checks whitespace, product data, cargo audit on every lockfile, the CHANGELOG[X.Y.Z]section, and the update-hop pin against the public head. It is built and tested; next are review and wiring into the push queue. - Run the Mac gate during Anders's try, not after it.
- Reuse downloaded release artifacts for the Stable prepare and the seed publish.
- Start the installed journeys automatically after publish, and reuse proofs for files whose hash is unchanged.
- Later, Anders decides: a private signed install check that would let a dependency-only security fix go straight to Stable with one signature.
Activity
andersalm commentedon Nov 4, 2026
Second flake source, seen on PR #526 (a JavaScript-only change) on 4 Nov: source-home-macos "build two actual Runtime versions" stopped with "new Runtime build failed" after exactly 900 s, the per-build limit in scripts/update-hop-compare.py:1214 (timeout=900). The same step took 5 to 24 minutes for its two builds in the last 12 green runs, so a slow Mac runner crosses the limit. With #433 on the first attempt, this one-file change needed three CI attempts. Done means here: the limit is set from measured build times (or the build reuses the cache), and a timeout says "timed out" instead of "build failed".
andersalm commentedon Nov 4, 2026
Third unrelated failure on PR #526 (4 Nov, attempt 3): source-home-linux (ubuntu-24.04) stopped at "require 16 GiB free disk" with 11.1 GB free of 75 GB (64 GB used) right after the Rust cache restore (ci.yml:782). The check is right to refuse, but the runner was already short before any work began. Done means: the free-space check runs before the cache restore and the job frees the runner's preinstalled toolchains it does not use, or the cache restore is bounded, so a hosted runner's starting disk cannot fail the job.
andersalm commentedon Nov 4, 2026
Fourth flake, on PR #529 (release scripts only) on 4 Nov: test-capsules "browser layout smokes" failed in scripts/documents-product-layout-smoke.mjs:983 ("An unsaved new document must be listed with its unsaved mark"). The same smoke passed on develop at 19e9930, which the PR head contains. The assertion reads the new draft's list title and mark right after clicking another document, without waiting for the list to update; it should wait for the expected state (waitForFunction) like the lines above it. The failure cancelled five other jobs.
andersalm commentedon Nov 4, 2026
#530 merged: the Documents layout smoke waits for UI state, hop builds get 1800 s each with a 70-minute step and report "timed out", and source-home-linux x86 frees runner disk before the cache restore. Its CI passed on the first attempt (17 of 17). Still open here: #433 (installed journey census flake) and the other Done means above.
andersalm commentedon Nov 4, 2026
Another timing flake, seen in the remote pre-push gate on 4 Nov (feat/90-jetson-browser at a8759c8, which does not touch Assistant): scripts/assistant-reply-layout-browser-smoke.mjs:206 expectNewestAboveComposer failed at "streamed reply" (newest reply 774 px above the composer, atEnd=false). The same suite passed 2,121/2,121 on that branch locally and in earlier gate runs. The assertion reads the scroll position right after the stream ends; it should wait for the scroll to settle (waitForFunction on the expected position) like #530 did for the Documents smoke.
andersalm commentedon Nov 4, 2026
On PR #532 (4 Nov, run 41912345716) the Documents layout smoke timed out in the 5-second wait #530 added (scripts/documents-product-layout-smoke.mjs:994, the new unsaved draft "Fresh" listed with its mark). The branch changes Documents by one callback (Home token renewal) and its local product-ui-browser run passed. First occurrence of this wait timing out; rerunning once. If it repeats on the rerun or on develop, the cause is real and gets investigated, not a longer timeout.
andersalm commentedon Nov 5, 2026
First CI failure of
scripts/chat-online-behavior.test.mjs:140("Community Chat resumes polling without a click after frame expiry" and "... after 401", bothAssertionError: the poll actually reached the fault, about 0.93 s each) on PR #542 run 42103677850 (head bce139f, an installer-only change merged with develop 344284e, which passed CI). The branch does not touch Chat or this test. Rerun once. A second failure of this test means an investigation, not another rerun: the test came with #532/#536 and twice this week a 'flake' was a product bug.andersalm commentedon Nov 6, 2026
New CI-only race (first occurrence):
capsules/ipfs-providertests::missing_cid_fails_on_one_local_readsaw 1 request instead of 2 in the full Linux check of feat/201-channel-choice (which does not touch ipfs-provider). Cause from the test (main.rs:4133-4174): the fixture backend stops accepting after 500 ms without a request, so under load the provider's second request (ls) can arrive after the backend has stopped. Fix is in the test: stop the backend when the test has made both calls, not on an idle timer. Briefed separately.andersalm commentedon Nov 6, 2026
First occurrence:
remote_engine_preparation_replay_close_and_late_input_keep_exact_native_owner(elastos-server lib) observed zero launches after 30 ms instead of one, once in a full Maccargo test -p elastos-server --librun on fix/201-browser-optin (3,281 passed, 1 failed) while four other jobs shared the Mac; it passed alone. The branch changes Browser setup on Linux, not the remote engine. A 30 ms observation window is load-sensitive; if it fails again, investigate (the test should wait for the product's own launch event, not a time window).andersalm commentedon Nov 6, 2026
Assistant reply-layout smoke, separate check: "Jump to latest" can show for 1 to 2 frames to a reader who never scrolled up, under load. Claude reviewer measured 5 of 30 failures on develop and 1 of 30 with fix/assistant-reply-layout (that fix does not cause it; zero guard trips in those runs). Second occurrence after the writer's two baseline flashes, so by the rule this needs an investigation of its own (product or test), not reruns.
andersalm commentedon Nov 6, 2026
Second occurrence of the installed model journey failure
stage=model_stop_get_different code=assertion_failed check=stop_real_stream(source-home-linux ubuntu-24.04): PR #544 run 42158780817, head ef1a95e (update discovery retry; it does not touch model streaming or Assistant). First was develop CI 40494663322 (lesson 797), where the rerun passed and the failing assert was not printed. #544 is rerun once; by the rule the cause is now investigated (separate branch), not rerun again.Merge pull request #553 from elastos/fix/527-dead-checks
andersalm commentedon Nov 8, 2026
Two live tests that no merge gate runs (found while proving the 2.0.1-rc.1 test evidence on 30b166b; Cursor lane report /private/tmp/cursor-runs/rc1-unproven-tests.report.md):
anonymous_private_stdin_pipe_reaches_preflight_without_any_effect(elastos/crates/elastos-vz/tests/browser_vz_engine_supervisor_process.rs:94) is cfg'd to macOS arm64. CItest-elastosrunscargo test --workspaceon Ubuntu, and the Mac pre-push unit plan does not select integration tests; onlyjust test(not a CI job) would run it. It passes on 30b166b on this Mac (elastos-heavy cargo test -p elastos-vz --test browser_vz_engine_supervisor_process ... --exact: 1 passed).scripts/library-performance-smoke.mjsis named by no gate script, workflow or just recipe. Run plainly it fails withMODULE_NOT_FOUNDplaywright (it needs the operator Playwright from test-behaviour.sh), so it is also not runnable as is.The rest of the 11 names without a found result were explained: four installer test names no longer exist in the source, the two pidfd tests run in source-gate (8 passes in the Ubuntu log),
shared_writable_rootfs_rejects_a_second_vm_owneris a Mac-only--bintest run by the Mac pre-push (passes on 30b166b), andhome-navigation-consumers-smoke.mjsruns insidehome-shell-regression-smoke.mjs(passes).Fix direction for this issue: a macOS CI job (or the Mac pre-push plan) that runs the macOS-only integration tests, and either wire the Library performance smoke into product-ui-browser or delete it.
andersalm commentedon Nov 12, 2026
Flaky on the Mac pre-push gate (11 Nov, twice, #573):
assistant-reply-layout-browser-smoke.mjs"a native zero-gutter scrollbar drag moves the reader" fails inside the full behaviour suite under load and passes alone 3 of 3. A fix to the drag timing follows in its own PR.andersalm commentedon Nov 12, 2026
Flaky in GitHub CI (11 Nov, #573 run 43130653696):
assistant-message-copy-browser-smoke.mjstimed out after 3 s waiting for the copy status on a slow Linux runner; it passed on develop for the same Assistant code. Failed jobs re-run once.andersalm commentedon Nov 13, 2026
develop 2a22643 (#584 merge), run 43338299318: test-capsules browser layout smokes failed at documents-product-layout-smoke.mjs:989 (the draft-list wait, 5 s timeout); first occurrence in the ledger.
#584 changed only the Assistant copy smoke; the run before it (9cc2f67) was green. Failed jobs rerun once; a repeat gets a product-versus-test investigation.
andersalm commentedon Nov 13, 2026
Second occurrence: PR #522 run 43368818081 (website-only change) failed at documents-product-layout-smoke.mjs:989, the same draft-list wait as develop 2a22643.
Two failures of one test: an investigation decides product race versus test timing; the #522 failed job is rerun once in the meantime.
andersalm commentedon Nov 13, 2026
Push-queue admission is blocked by one untriaged late failure and a failed infrastructure login check.
Main owns the failure triage and restoration of the existing login, then resumes the queued checks; earlier failed attempts remain recorded.
andersalm commentedon Nov 13, 2026
The 2.0.2 stage-1 candidate
264bff0be04b570ee7599c29bef58ed233cac343(tree06c626930a8321dd4fab573f0106829faf135bf9) earns a bounded source PASS over the four changed test files and six combined files. The repaired fixture passes the quiet-start choice to the controller; its checks cover first start, update, restoration and interrupted recovery through the linked path. The runner notes check survives a release cut. Syntax and candidate whitespace pass. The earlier integration compile failure remains in the record.The completed full Linux receipt binds this commit, tree and develop base
f6d819571. The log shows all ten linked-path fixture journeys passed; the nested failure is the deliberate controller interruption at CandidateReady, and recovery passes. Root inspected the source and receipt; Main ran the checks. The separate runner push was cancelled in favor of this candidate. The zip bump #601 is merged.Main owns the next candidate’s checks and the installed and human acceptance. The earlier runner regressions, Mac gate failures and #601’s skipped upstream-assets check retain their original scope. A passing Linux check does not establish Mac gate load control or prove the installed release journeys.