Repository navigation
Scope VS Code extension spawnCli fake timers to the test's async context - #21922
Conversation
sinon.useFakeTimers replaces timers, Date, nextTick and queueMicrotask for the whole extension-host process. Host callbacks that run during clock.tickAsync's real setImmediate yields can schedule onto the fake clock, so cliSpawn tests asserting clock.countTimers() === 0 failed intermittently on the Windows unit job. Add useScopedFakeTimers, which installs sinon's clock on a private global and routes the real globals to it only for the test's AsyncLocalStorage context, and use it in cliSpawn.test.ts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21922Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21922" |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
🟡 Changes recommended
The timer helper lets unrelated caller continuations inherit fake-clock scope and breaks Date subclass prototypes by dropping newTarget in its construct trap.
2 open issues
What changed in this PR
This PR addresses intermittent VS Code extension test failures caused by unrelated extension-host callbacks scheduling work on Sinon’s global fake clock.
Changes:
- Adds an async-context-aware fake-timer helper intended to preserve real timers for unrelated callbacks.
- Migrates all 13 fake-clock setups in
cliSpawn.test.tswithout changing assertions. - Adds regression tests for timer isolation and scoped time behavior.
| File | Description |
|---|---|
extension/src/test/scopedFakeTimers.test.ts |
Tests isolation from foreign callbacks and scoped timers, microtasks, and dates. |
extension/src/test/helpers/scopedFakeTimers.ts |
Adds context-based routing between real and fake timer APIs. |
extension/src/test/cliSpawn.test.ts |
Replaces global fake clocks with the scoped helper. |
Review effort: Balanced
…bclasses Replace useScopedFakeTimers (enterWith) with withScopedFakeTimers(body), which runs the body in its own AsyncLocalStorage.run scope so work the test runner queues after invoking the body can't inherit the fake clock. Forward newTarget in the routed Date construct trap so Date subclasses keep their prototype. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
|
[automated] Inconclusive for efficacy: the requested fresh before/fixed comparison is blocked. The native I inspected complete original Windows x64 unit-job logs:
The historical failures are The POSIX-named scenario also ran on Windows; the peer run's later successful rerun does not erase its first-attempt failure. All three jobs ran the full Fixed evidence covers head Scoped review: no high-confidence defect found in the three test-only files; cleanup assertions remain intact. The new regressions exercise foreign-callback and caller-context contamination, but the original logs do not prove who owned the leftover fake-clock work. CI-confidence is not applicable because this diff changes no CI/build/workflow/toolchain behavior. These are two outcome-selected historical failures and one existing fixed-suite run—not a contemporaneous baseline campaign, a measured failure rate, or proof that the flake is eliminated. |
Ella Hathaway (ellahathaway)
left a comment
There was a problem hiding this comment.
I reviewed the new withScopedFakeTimers helper end to end: the fake-timers install on the private global, the AsyncLocalStorage routers, the Date proxy, clear routing for native and fake handles, and restore. I also reviewed its tests and the 13 migrated spawnCliProcess tests against cliProcess.ts.
I didn't find any correctness issues. I left one nonblocking note about test coverage for restore.
|
AspireAgent stopped: the restore-coverage feedback is addressed in 6dda321 and CI is green. 🤖 Posted by AspireAgent (AI Agent) |
Covers bodies that resolve, reject and throw synchronously, so a regression in restore cannot leave the routers installed for the rest of the run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
James Newton-King (JamesNK)
left a comment
There was a problem hiding this comment.
Scoping the clock to the test's async context is a reasonable way to fix this, and simpler alternatives (sinon toFake subsets, relaxing countTimers) would either not fix it or lose the leak check. All converted cliSpawn assertions and tick amounts are unchanged. A couple of things on the helper inline.
|
AspireAgent stopped: the review threads are addressed, JamesNK is re-requested and Final Results passed on the new head. 🤖 Posted by AspireAgent (AI Agent) |
Route the globals through one set of routers shared by every active scope, which look up the caller's scope on each call. Settling a scope only retires that scope, and the real globals come back when the last scope settles and only where our routers are still installed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed to evaluate a cross-context compatibility risk where timer routers can break promisified Node timers in unrelated extension-host code while a scope is active.
No open issues 🎉
Review effort: Balanced
|
AspireAgent made the scoped fake-timer routers support 🤖 Posted by AspireAgent (AI Agent) |
The scoped setTimeout and setImmediate routers did not expose util.promisify.custom, so util.promisify(setTimeout) fell back to the callback form and threw ERR_INVALID_ARG_TYPE while any scope was active. Route the promisified form to the calling scope's fake clock, or to the real promisified timer when no scope owns the caller. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tests selector0 / 101 PR test projects · 2 PR jobs, from 3 changed files. Selected PR test projects (0 / 101)none — no PR-gated .NET test projects run for this change. Selected PR jobs (2)
How these were chosen — grouped by what changedJob reasons
Selection computed for commit |
…21927) ## Description **What failed.** The Windows VS Code extension unit job intermittently fails `E2E state file bridge > bounds the natural-close connection and releases its socket on timeout` on `assert.strictEqual(clock.countTimers(), 0)` with `1 !== 0`: [42338413245](https://github.com/microsoft/aspire/actions/runs/42338413245), [42289556640](https://github.com/microsoft/aspire/actions/runs/42289556640), [42137910318](https://github.com/microsoft/aspire/actions/runs/42137910318). That's 4 of 608 first attempts in 7 days, and every one passed on rerun. The failing runs came from unrelated PRs (a Dependabot actions bump and two dashboard UI changes) that don't touch the extension, so this is a flake, not a PR-caused failure. ## Failure stats (14d, attempt 1) `CI` workflow on microsoft/aspire, runs created 2026-10-25 06:28 → 2026-11-08 06:28 UTC, **attempt 1 only**. Denominator: attempt-1 jobs that ran (success or failure). Hits: the failed job's log matches the signature. Jobs that lost their runner, which have no log, are excluded. Main runs have no reruns, so their flake check is the next main run of the job. | | | |---|---| | Signature | `E2E state file bridge` › `bounds the natural-close connection and releases its socket on timeout` (`countTimers()` `1 !== 0`) | | Job | Run VS Code extension unit tests (Windows) | | main pushes: hits / runs | 0 / 158 (0.00%) | | main pushes, 30d: hits / runs | 0 / 273 (0.00%). The 16 days before the 14d window add 0 hits. | | PR runs: hits / runs | 10 / 986 (1.01%); 0 PR-caused excluded | | Combined rate | 10 / 1144 = **0.87%** | | Flake check | PR: 7/7 rerun hits passed on rerun (3 never rerun) | | First / last hit | 2026-11-05 / 2026-11-08 | | Still hitting main? | Yes, still active: last hit 2026-11-08; last main hit never | | Fix evidence | Local (real VS Code 1.131 host): the new regression test fails 3/3 with the CI signature before; after the fix the suite passes 4/4. The natural flake reproduced 1 time in ~18 suite runs. This PR's CI ([run](https://github.com/microsoft/aspire/actions/runs/42453400853)): the Windows unit job passed on attempt 1 (1/1). Attempts 1 and 2 failed only in `VS Code extension E2E (Windows, tree-actions)` with `Expected e2e-worker to expose an endpoint URL`, which is the known flake #21796 owned by #21914 (this PR touches only unit-test files). Attempt 3 passed and Final Results is green. | Failing attempt-1 jobs, most recent first (all 10 hits, all PR runs): | Date (UTC) | Branch | Job | Signature excerpt | |---|---|---|---| | [2026-11-08 06:03](https://github.com/microsoft/aspire/actions/runs/42453169210/job/128646825791) | `aspireagent/fix-vscode-folder-picker-colliding-selection` | Run VS Code extension unit tests (Windows) | `bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0` | | [2026-11-07 04:25](https://github.com/microsoft/aspire/actions/runs/42338413245/job/128301352107) | `sebros/resources-compact-view-regression-4abee4` | Run VS Code extension unit tests (Windows) | `bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0` | | [2026-11-06 20:34](https://github.com/microsoft/aspire/actions/runs/42289556640/job/128140462040) | `dependabot/github_actions/github-actions-ebbe822166` | Run VS Code extension unit tests (Windows) | `bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0` | | [2026-11-06 15:59](https://github.com/microsoft/aspire/actions/runs/42248127696/job/127997089873) | `sebros/dashboard-review-tracking-issue-f03a30` | Run VS Code extension unit tests (Windows) | `bounds the natural-close connection…: AssertionError strictly equal: 2 !== 0` | | [2026-11-06 14:50](https://github.com/microsoft/aspire/actions/runs/42237500020/job/127960139523) | `ievangelist-go-runtime-export-readiness` | Run VS Code extension unit tests (Windows) | `bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0` | | [2026-11-06 02:15](https://github.com/microsoft/aspire/actions/runs/42149206621/job/127671234198) | `spike/project-v2-csharpprogram-watch` | Run VS Code extension unit tests (Windows) | `bounds the natural-close connection…: AssertionError strictly equal: 2 !== 0` | | [2026-11-06 01:20](https://github.com/microsoft/aspire/actions/runs/42143166267/job/127650342491) | `spike/project-v2-csharpprogram-watch` | Run VS Code extension unit tests (Windows) | `bounds the natural-close connection…: AssertionError strictly equal: 2 !== 0` | | [2026-11-06 00:35](https://github.com/microsoft/aspire/actions/runs/42137910318/job/127631683952) | `ievangelist-dashboard-sidebar-auto-collapse` | Run VS Code extension unit tests (Windows) | `bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0` | | [2026-11-05 19:43](https://github.com/microsoft/aspire/actions/runs/42096580384/job/127491142502) | `damianedwards-launch-groups` | Run VS Code extension unit tests (Windows) | `bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0` | | [2026-11-05 19:01](https://github.com/microsoft/aspire/actions/runs/42089852361/job/127466229297) | `copilot/clarify-scope-of-agent-integrations` | Run VS Code extension unit tests (Windows) | `bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0` | Same root cause, other tests in this suite (not counted above; this PR migrates them too): [11-05](https://github.com/microsoft/aspire/actions/runs/42093304445/job/127478234883) `natural-close response … 2 !== 0` · [11-06](https://github.com/microsoft/aspire/actions/runs/42146938315/job/127661485652) `CDP proxy request … 2 !== 0` · [11-06](https://github.com/microsoft/aspire/actions/runs/42296965127/job/128166382811) `CDP proxy request … 1 !== 0` · [11-06](https://github.com/microsoft/aspire/actions/runs/42304086594/job/128189892984) `natural-close response … 1 !== 0`. **Verdict: MODERATE.** 10 hits since 2026-11-05 (0.9%), still hitting today, and all 7 that were rerun passed on rerun. It has **never hit on main** (0/158). The fix is validated on a modelled host callback in a real extension host, and its own CI ran the unit job once (green). <!-- /failure-stats --> **Why.** The suite calls `sandbox.useFakeTimers()`, which replaces `setTimeout` and the other timer globals for the whole extension-host process. `clock.tickAsync()` yields to the real event loop, and host or other-extension callbacks that run during those yields schedule their timers on the test's fake clock. `closeBrowserPageThroughCdp` and `proveBlazorWasmDebugging` clear every timer they create on this path. The extra timer belongs to someone else. I reproduced it locally in a real VS Code 1.131 extension host by looping the full unit suite: the leftover was a 25 ms timer with a bound native callback, created at fake time 1100 (after the tick ended) and not by the bridge code. This is the same root cause #21922 found for `spawnCliProcess`. **Fix.** Every clock in `e2eStateFileBridge.test.ts` (8 tests) now uses `withScopedFakeTimers` from #21922. Only the test body's async context sees the fake clock, and host callbacks keep real timers. The assertions are unchanged. A new regression test, `does not count extension host timers scheduled while the natural-close connection times out`, queues a host callback outside the test. The callback runs during the connection timeout's `tickAsync` and schedules a timer that's still pending when the wait ends. It stays in this suite as the bridge-level repro of the original flake, and a comment says so. The helper's own isolation guarantees are covered in `scopedFakeTimers.test.ts`. **Depends on #21922.** This branch is stacked on #21922's head (`4f84aa90`) because it uses that PR's helper, including its fixes for scopes that settle out of order and for `util.promisify` timers. It targets `main` so CI runs, which means the diff includes #21922's commits until that PR merges. After #21922 merges, I'll rebase so only this PR's two commits remain (`8d966f54`, `b36b2994`). ## Confidence High that this removes the flake. Evidence, all from a real VS Code 1.131.0 extension host under xvfb: - **Before** (same test file, with the suite's original global sinon clock): the new regression test fails 3/3 with the CI signature `AssertionError: 1 !== 0` at `countTimers()`. - **After** (`withScopedFakeTimers`): the `E2E state file bridge` suite passes 4/4 (62 passing). - Re-checked on the current stack in plain-Node mocha with a `vscode` shim, because VS Code can't be downloaded in that environment. The regression test **fails** 3/3 with `1 !== 0` at `countTimers()` when it uses the global sinon clock, and **passes** 3/3 with `withScopedFakeTimers`. The bridge, helper and `cliSpawn` suites together have the same failure set with #21922's version of this file as with this PR: 3 shim gaps unrelated to timers. This PR adds one passing test. `tsc` and `eslint` are clean. - Main remaining uncertainty: I didn't capture the creation stack of the host timer in CI. I reproduced the natural flake locally once in about 18 full-suite runs, so the regression test models it rather than replaying the exact host callback. **How we stop this recurring:** move the remaining extension suites that still install global `sinon.useFakeTimers` (as #21922 proposes) onto `withScopedFakeTimers`, and add a lint rule against global fake timers in extension tests. Refs #21600, #21922 ## Checklist - Is this feature complete? - [x] Yes. Ready to ship. - [ ] No. Follow-up changes expected. - Are you including unit tests for the changes and scenario tests if relevant? - [x] Yes - [ ] No - Did you add public API? - [ ] Yes - [x] No - Does the change make any security assumptions or guarantees? - [ ] Yes - [x] No **🤖 Posted by AspireAgent (AI Agent)** --------- Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
✅ No documentation update needed. docs_optional → test_only. No signals triggered. All 3 changed files match |

Description
What failed. The Windows VS Code extension unit job intermittently fails
spawnCliProcess testsonassert.strictEqual(clock.countTimers(), 0)(actual 1–2), e.g. 42116644663, 42147483007, 42153240896, 42280991794 (POSIX cleanup retry after confirmation timeout) and the main failure 42288258477 tracked in #21886 (taskkill deadline). 16 first-attempt hits in 7 days, all on runs whosecliSpawn.test.ts/cliProcess.tsmatched main, so these are not PR-caused.Failure stats (14d, attempt 1)
CIworkflow on microsoft/aspire, runs created 2026-10-25 06:28 → 2026-11-08 06:28 UTC, attempt 1 only. Denominator: attempt-1 jobs that ran (success or failure). Hits: the failed job's log matches the signature. Jobs that lost their runner, which have no log, are excluded. Main runs have no reruns, so their flake check is the next main run of the job.spawnCliProcess testsfailing withAssertionError … strictly equal(clock.countTimers())Failing attempt-1 jobs, most recent first (10 of 17 log-verified hits):
aspireagent/fix-vscode-folder-picker-colliding-selectionspawnCliProcess tests rejects when a POSIX process group remains alive after forced termination: AssertionError [ERR_ASS…aspireagent/resource-attach-proof-resume-aborted-inspectionspawnCliProcess tests retrying POSIX cleanup after confirmation timeout retains process-group ownership: AssertionError …aspireagent/java-debug-timeout-preserve-failurespawnCliProcess tests retrying POSIX cleanup after confirmation timeout retains process-group ownership: AssertionError …aspireagent/warn-cli-path-overrides-dnx-19713spawnCliProcess tests rejects when a POSIX process group remains alive after forced termination: AssertionError [ERR_ASS…eerhardt-update-appcontainers-dependencyspawnCliProcess tests force kills surviving POSIX descendants immediately when their leader exits: AssertionError [ERR_A…spawnCliProcess tests rejects when a Windows child does not close after successful taskkill: AssertionError [ERR_ASSERTI…sebros/resources-compact-view-regression-4abee4spawnCliProcess tests retrying POSIX cleanup after confirmation timeout retains process-group ownership: AssertionError …eerhardt-update-appcontainers-dependencyspawnCliProcess tests rejects when a Windows child does not close after successful taskkill: AssertionError [ERR_ASSERTI…ievangelist-go-starter-frontendspawnCliProcess tests rejects when a POSIX process group remains alive after forced termination: AssertionError [ERR_ASS…ievangelist-bound-ci-mirror-fallbackspawnCliProcess tests rejects when a POSIX process group remains alive after forced termination: AssertionError [ERR_ASS…A strict re-scan of the job logs confirmed 17 of the 21 hits counted above; the table lists only confirmed hits. The rates above use the original count.
Verdict: MODERATE. A real and frequent flake: 21 hits (1 on main), 1.8%, still hitting today. 14 of the 16 PR hits that were rerun passed on rerun. The fix is validated only on a model of the foreign-timer race, and its own CI is 2 runs; at a 1.8% rate, ~160 clean runs are needed for 95% confidence.
Why.
sinon.useFakeTimers()replacessetTimeout/setImmediate/Date/process.nextTick/queueMicrotaskfor the whole extension-host process.clock.tickAsync()yields through realsetImmediate, and host/other-extension callbacks that run during those yields schedule onto the fake clock, socountTimers()sees work the test never started.terminateCliProcessitself is deterministic under fake timers.Fix. New test helper
withScopedFakeTimers(async clock => ...)installs sinon's clock on a private global object, runs the body viaAsyncLocalStorage.run, and routes the real globals to the clock only from the body's async context (the body and everything it awaits or schedules). Callbacks from any other context, including the test runner that invoked the body, keep real timers andDate. Clears are routed by handle, so either kind can be cleared anywhere.Scopes do not have to be nested. When mocha times out a test, its body keeps running while the next test opens its own scope, so the two can settle in either order. One set of routers serves every active scope and looks up the caller's scope on each call. Settling a scope only retires that scope. The real globals come back when the last active scope settles, and only where our routers are still installed.
The helper fakes the same timer set the old
sinon.useFakeTimers({ shouldClearNativeTimers: true })faked for this code: timers,Date,process.nextTickandqueueMicrotask. With sinon 21 / fake-timers 13, that call'sclock.methodsalso includesnextTickandqueueMicrotask, socountTimers()counted queued jobs before this change too. It does not fakehrtime,performanceorIntl, because neithercliProcess.tsnor the tests read them.The
setTimeoutandsetImmediaterouters also carryutil.promisify.custom, routed per call like the timers. A scope's own code waits on its fake clock; any other caller gets the real promisified timer. Without it,util.promisify(setTimeout)threwERR_INVALID_ARG_TYPEwhile any scope was active.cliSpawn.test.tsnow uses it for all 13 clocks; assertions are unchanged.Confidence
scopedFakeTimers.test.tsmodels the flake: a callback queued before the test runs duringtickAsync's real yield and schedules a timer,nextTickand microtask. Paired run with the same test bodies on plainsinon.useFakeTimers({ shouldClearNativeTimers: true }): failing (countTimers()non-zero; foreign code sees fakeDate); with the scoped helper: passing on every repeat.Datesubclasses keep their prototype) fail on the first-iterationenterWithhelper and pass now.Date,Date.nowandprocess.nextTick, then asserts they are all back after a body that resolves, rejects or throws synchronously. It fails when restore is regressed to run only on success or to skipprocess.nextTick, and passes on the helper.The second scope lost its fake clock when the first settled.). A standalone probe of the previous helper also showssetTimeout,Dateandprocess.nextTickleft as stale routers after that sequence. With this change both pass.util.promisifyfrom the Copilot review: a test callspromisify(setTimeout)andpromisify(setImmediate)from outside an active scope and inside it. On4ddd5b11it fails (TypeError [ERR_INVALID_ARG_TYPE]: The "callback" argument must be of type function. Received type number (1)). With this change it passes (helper suite 7/7,cliSpawn.test.ts50 passing, 1 pending;tscandeslintclean).vscodeshim: identical failure set before and after (all shim limitations, none inwithScopedFakeTimersorspawnCliProcess), with one more passing test (the new one).tscandeslintclean on changed files.retrying POSIX cleanup ...andtaskkill ... bounded deadlinetests; Final Results green.cliSpawn.test.tstogether withprogressNotifier,appHostStopper,runSessionRegistryande2eStateFileBridge(which keep global sinon clocks) in one mocha process: identical failure set before/after, +2 passing (the new tests at the time).tscandeslintclean on changed files.vscodemodule shim; failures in that run are shim limitations and occur identically on main. Real-host coverage comes from CI. Not stress-tested with repeated Windows runs.How we stop this recurring: follow up by moving the remaining extension suites that use global
sinon.useFakeTimerstowithScopedFakeTimersand adding a lint rule against global fake timers in extension tests.Refs #21886, #21600
🤖 Posted by AspireAgent (AI Agent)