Skip to content

Scope E2E state file bridge fake timers to each test's async context - #21927

Merged
Eric Erhardt (eerhardt) merged 7 commits into
mainfrom
aspireagent/vscode-bridge-scoped-fake-timers
Nov 11, 2026
Merged

Eric Erhardt (eerhardt) merged 7 commits into
mainfrom
aspireagent/vscode-bridge-scoped-fake-timers

Conversation

@davidfowl

@davidfowl David Fowler (davidfowl) commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Description

What failed. The Windows VS Code extension unit job intermittently fails E2E state file bridge > bounds the natural-close connection and releases its socket on timeout on assert.strictEqual(clock.countTimers(), 0) with 1 !== 0: 42338413245, 42289556640, 42137910318. That's 4 of 608 first attempts in 7 days, and every one passed on rerun. The failing runs came from unrelated PRs (a Dependabot actions bump and two dashboard UI changes) that don't touch the extension, so this is a flake, not a PR-caused failure.

Failure stats (14d, attempt 1)

CI workflow on microsoft/aspire, runs created 2026-10-25 06:28 → 2026-11-08 06:28 UTC, attempt 1 only. Denominator: attempt-1 jobs that ran (success or failure). Hits: the failed job's log matches the signature. Jobs that lost their runner, which have no log, are excluded. Main runs have no reruns, so their flake check is the next main run of the job.

Signature E2E state file bridge › bounds the natural-close connection and releases its socket on timeout (countTimers() 1 !== 0)
Job Run VS Code extension unit tests (Windows)
main pushes: hits / runs 0 / 158 (0.00%)
main pushes, 30d: hits / runs 0 / 273 (0.00%). The 16 days before the 14d window add 0 hits.
PR runs: hits / runs 10 / 986 (1.01%); 0 PR-caused excluded
Combined rate 10 / 1144 = 0.87%
Flake check PR: 7/7 rerun hits passed on rerun (3 never rerun)
First / last hit 2026-11-05 / 2026-11-08
Still hitting main? Yes, still active: last hit 2026-11-08; last main hit never
Fix evidence Local (real VS Code 1.131 host): the new regression test fails 3/3 with the CI signature before; after the fix the suite passes 4/4. The natural flake reproduced 1 time in ~18 suite runs. This PR's CI (run): the Windows unit job passed on attempt 1 (1/1). Attempts 1 and 2 failed only in VS Code extension E2E (Windows, tree-actions) with Expected e2e-worker to expose an endpoint URL, which is the known flake #21796 owned by #21914 (this PR touches only unit-test files). Attempt 3 passed and Final Results is green.

Failing attempt-1 jobs, most recent first (all 10 hits, all PR runs):

Date (UTC) Branch Job Signature excerpt
2026-11-08 06:03 aspireagent/fix-vscode-folder-picker-colliding-selection Run VS Code extension unit tests (Windows) bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0
2026-11-07 04:25 sebros/resources-compact-view-regression-4abee4 Run VS Code extension unit tests (Windows) bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0
2026-11-06 20:34 dependabot/github_actions/github-actions-ebbe822166 Run VS Code extension unit tests (Windows) bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0
2026-11-06 15:59 sebros/dashboard-review-tracking-issue-f03a30 Run VS Code extension unit tests (Windows) bounds the natural-close connection…: AssertionError strictly equal: 2 !== 0
2026-11-06 14:50 ievangelist-go-runtime-export-readiness Run VS Code extension unit tests (Windows) bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0
2026-11-06 02:15 spike/project-v2-csharpprogram-watch Run VS Code extension unit tests (Windows) bounds the natural-close connection…: AssertionError strictly equal: 2 !== 0
2026-11-06 01:20 spike/project-v2-csharpprogram-watch Run VS Code extension unit tests (Windows) bounds the natural-close connection…: AssertionError strictly equal: 2 !== 0
2026-11-06 00:35 ievangelist-dashboard-sidebar-auto-collapse Run VS Code extension unit tests (Windows) bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0
2026-11-05 19:43 damianedwards-launch-groups Run VS Code extension unit tests (Windows) bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0
2026-11-05 19:01 copilot/clarify-scope-of-agent-integrations Run VS Code extension unit tests (Windows) bounds the natural-close connection…: AssertionError strictly equal: 1 !== 0

Same root cause, other tests in this suite (not counted above; this PR migrates them too): 11-05 natural-close response … 2 !== 0 · 11-06 CDP proxy request … 2 !== 0 · 11-06 CDP proxy request … 1 !== 0 · 11-06 natural-close response … 1 !== 0.

Verdict: MODERATE. 10 hits since 2026-11-05 (0.9%), still hitting today, and all 7 that were rerun passed on rerun. It has never hit on main (0/158). The fix is validated on a modelled host callback in a real extension host, and its own CI ran the unit job once (green).

Why. The suite calls sandbox.useFakeTimers(), which replaces setTimeout and the other timer globals for the whole extension-host process. clock.tickAsync() yields to the real event loop, and host or other-extension callbacks that run during those yields schedule their timers on the test's fake clock. closeBrowserPageThroughCdp and proveBlazorWasmDebugging clear every timer they create on this path. The extra timer belongs to someone else. I reproduced it locally in a real VS Code 1.131 extension host by looping the full unit suite: the leftover was a 25 ms timer with a bound native callback, created at fake time 1100 (after the tick ended) and not by the bridge code. This is the same root cause #21922 found for spawnCliProcess.

Fix. Every clock in e2eStateFileBridge.test.ts (8 tests) now uses withScopedFakeTimers from #21922. Only the test body's async context sees the fake clock, and host callbacks keep real timers. The assertions are unchanged. A new regression test, does not count extension host timers scheduled while the natural-close connection times out, queues a host callback outside the test. The callback runs during the connection timeout's tickAsync and schedules a timer that's still pending when the wait ends. It stays in this suite as the bridge-level repro of the original flake, and a comment says so. The helper's own isolation guarantees are covered in scopedFakeTimers.test.ts.

Depends on #21922. This branch is stacked on #21922's head (4f84aa90) because it uses that PR's helper, including its fixes for scopes that settle out of order and for util.promisify timers. It targets main so CI runs, which means the diff includes #21922's commits until that PR merges. After #21922 merges, I'll rebase so only this PR's two commits remain (8d966f54, b36b2994).

Confidence

High that this removes the flake. Evidence, all from a real VS Code 1.131.0 extension host under xvfb:

  • Before (same test file, with the suite's original global sinon clock): the new regression test fails 3/3 with the CI signature AssertionError: 1 !== 0 at countTimers().
  • After (withScopedFakeTimers): the E2E state file bridge suite passes 4/4 (62 passing).
  • Re-checked on the current stack in plain-Node mocha with a vscode shim, because VS Code can't be downloaded in that environment. The regression test fails 3/3 with 1 !== 0 at countTimers() when it uses the global sinon clock, and passes 3/3 with withScopedFakeTimers. The bridge, helper and cliSpawn suites together have the same failure set with Scope VS Code extension spawnCli fake timers to the test's async context #21922's version of this file as with this PR: 3 shim gaps unrelated to timers. This PR adds one passing test. tsc and eslint are clean.
  • Main remaining uncertainty: I didn't capture the creation stack of the host timer in CI. I reproduced the natural flake locally once in about 18 full-suite runs, so the regression test models it rather than replaying the exact host callback.

How we stop this recurring: move the remaining extension suites that still install global sinon.useFakeTimers (as #21922 proposes) onto withScopedFakeTimers, and add a lint rule against global fake timers in extension tests.

Refs #21600, #21922

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
    • No

🤖 Posted by AspireAgent (AI Agent)

sinon.useFakeTimers replaces timers, Date, nextTick and queueMicrotask for
the whole extension-host process. Host callbacks that run during
clock.tickAsync's real setImmediate yields can schedule onto the fake
clock, so cliSpawn tests asserting clock.countTimers() === 0 failed
intermittently on the Windows unit job.

Add useScopedFakeTimers, which installs sinon's clock on a private global
and routes the real globals to it only for the test's AsyncLocalStorage
context, and use it in cliSpawn.test.ts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…bclasses

Replace useScopedFakeTimers (enterWith) with withScopedFakeTimers(body), which
runs the body in its own AsyncLocalStorage.run scope so work the test runner
queues after invoking the body can't inherit the fake clock. Forward newTarget
in the routed Date construct trap so Date subclasses keep their prototype.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@davidfowl David Fowler (davidfowl) added the agent:working An AI agent is actively working on this issue or PR label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21927

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21927"

@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified for the scoped fake-timer changes.

No open issues 🎉

What changed in this PR

This PR addresses flaky timer-leak assertions in the VS Code extension’s E2E state-file bridge tests by isolating fake timers to each test’s async context. The stacked diff also includes the shared helper and CLI test migration from #21922.

Changes:

  • Add scoped fake timers that leave unrelated extension-host callbacks on real timers.
  • Migrate bridge and CLI process tests while preserving existing assertions.
  • Add regression coverage for timer isolation, scoped time, and Date subclasses.
File Description
extension/​src/​test/​scopedFakeTimers.test.ts Tests async-context isolation and scoped timer and Date behavior.
extension/​src/​test/​helpers/​scopedFakeTimers.ts Adds async-context timer routing and automatic restoration.
extension/​src/​test/​e2eStateFileBridge.test.ts Migrates fake clocks and adds a host-timer interference regression test.
extension/​src/​test/​cliSpawn.test.ts Migrates process-termination tests to scoped fake timers.

Review effort: Balanced

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@radical

Copy link
Copy Markdown
Member

[automated] Validation report: blocked for a fresh comparison; inconclusive for flaky-test efficacy.

Reviewed head: 79160a23a1ea1b38cf6b40698b950a882a5c6ffa.

Existing GitHub evidence confirms the original failure and a passing fixed-source suite, not a measured reduction in flakiness. These are two retrospectively selected CI jobs, not an independently dispatched before/after experiment.

Retained job, attempt 1 Original log result Target outcome
Historical unfixed Windows job 4152 passing, 1 failing, 9 pending; test command exit 1 Natural-close connection timeout test failed
This PR's Windows job 4158 passing, 9 pending; test command exit 0 Connection and response timeout tests, the new foreign-host-timer regression, and all four scoped-clock helper tests passed

Excerpt from the historical job log (timestamps removed, indentation trimmed, blank lines and the + expected - actual diff omitted):

1) E2E state file bridge
bounds the natural-close connection and releases its socket on timeout:
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
1 !== 0
at Context.<anonymous> (d:\a\aspire\aspire\extension\out\test\e2eStateFileBridge.test.js:806:20)

Per each job's runner-setup and step logs, both jobs ran on Windows Server 2025 (runner image windows-2025-vs2026, version 20261103.298.1) with tooling Node 22.23.4 x64 (from actions/setup-node; the Extension Host's embedded runtime was not separately measured), Yarn 1.22.22, and VS Code 1.131.0 win32-x64. Both ran the full extension unit suite through corepack yarn test, and .github/workflows/extension-unit-tests.yml and extension/.vscode-test.mjs are byte-identical across the two checkouts.

The historical checkout was 5a425857668a0d981df4739730fe1dc041fbcf1b, from another PR. The fixed checkout was bfcdbc36561cc277082ff85ea96bbb6bbf1c16db; its Git tree matches the reviewed head exactly. The five added regression tests are fixed-only coverage, not common before/after opportunities. Other source differences and the different execution times prevent treating these jobs as a controlled comparison. No failure-rate estimate or elimination claim is justified.

Concrete blocker: the native reproduction workflow resolves only tests/<name>.Tests/<name>.Tests.csproj or tests/Aspire.<name>.Tests/Aspire.<name>.Tests.csproj and invokes dotnet test. It cannot execute this TypeScript/VS Code workload. The Windows extension unit workflow exposes only workflow_call, not a dispatchable comparison interface.

No new workflows or runners were dispatched, and no source changes were made. A macOS run or standalone Node test would not replace the failing Windows Extension Host workload. The actual owner/creation stack of the extra CI timer remains unverified; the new test models that mechanism.

Next step: separately scope a faithful Windows Extension Host comparison lane using the existing full-suite command, explicit before/fixed source pins, one invocation per fresh runner, and retained passing/failing results. Admit its exact cohort, destination/ref, and collection plan before dispatch. This is a tooling limitation, not a finding that the PR's fix is wrong.

Covers bodies that resolve, reject and throw synchronously, so a regression
in restore cannot leave the routers installed for the rest of the run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bridge test conversion looks right: the harness uses a fake WebSocket emitter whose events fire from in-scope code, so bridge timers are still created in the scope and a genuine leak would still be caught. All assertions and tick amounts are unchanged.

Stacking: this PR carries #21922's first two commits but not 6dda321 (the restore-globals test in scopedFakeTimers.test.ts). Please merge #21922 first and then update this branch from main (rather than rebasing the duplicated commits, which can hit add/add conflicts on the helper), so the restore test isn't lost if #21922 ends up closed as superseded. The helper comments on #21922 apply here as well since the helper is identical.

@davidfowl David Fowler (davidfowl) added the agent:working An AI agent is actively working on this issue or PR label Nov 10, 2026 — with Copilot SWE Agent

David Fowler (davidfowl) commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

AspireAgent stopped: the review threads are addressed, JamesNK is re-requested and Final Results passed on the new head.

🤖 Posted by AspireAgent (AI Agent)

Route the globals through one set of routers shared by every active scope,
which look up the caller's scope on each call. Settling a scope only retires
that scope, and the real globals come back when the last scope settles and
only where our routers are still installed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@davidfowl
David Fowler (davidfowl) force-pushed the aspireagent/vscode-bridge-scoped-fake-timers branch from 79160a2 to 339b095 Compare November 10, 2026 14:41
@aspire-repo-bot
aspire-repo-bot Bot requested a balanced review from Copilot November 10, 2026 14:43
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the test-only migration to async-context-scoped fake timers.

No open issues 🎉

Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@davidfowl David Fowler (davidfowl) removed the agent:working An AI agent is actively working on this issue or PR label Nov 10, 2026
@davidfowl David Fowler (davidfowl) added the agent:working An AI agent is actively working on this issue or PR label Nov 10, 2026 — with Copilot SWE Agent

David Fowler (davidfowl) commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

AspireAgent made the scoped fake-timer routers support util.promisify (head 4f84aa900e5c8c5ec8cf9bb3e17088acc3695c54) and restacked #21927 on it (head b36b2994ce604dd13c820ff83a7c1bb3fc7ab9b4). Final Results passed on both heads. JamesNK is re-requested.

🤖 Posted by AspireAgent (AI Agent)

The scoped setTimeout and setImmediate routers did not expose
util.promisify.custom, so util.promisify(setTimeout) fell back to the
callback form and threw ERR_INVALID_ARG_TYPE while any scope was active.
Route the promisified form to the calling scope's fake clock, or to the
real promisified timer when no scope owns the caller.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The bridge suite installed sinon's clock on the extension host's globals, so a
host callback that scheduled a timer during clock.tickAsync()'s real yields was
counted by countTimers() as a timer leaked by the natural-close CDP proof.
Use withScopedFakeTimers for every clock in the suite and add a regression test
that schedules a host timer during the connection timeout.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues have been identified for the scoped fake-timer changes.

No open issues 🎉

Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

Tests selector

0 / 101 PR test projects · 2 PR jobs, from 4 changed files.

Selected PR test projects (0 / 101)

none — no PR-gated .NET test projects run for this change.

Selected PR jobs (2)

extension-e2e, extension-unit


How these were chosen — grouped by what changed

Job reasons

Job Triggered by
extension-e2e extension/src/test/cliSpawn.test.ts, extension/src/test/e2eStateFileBridge.test.ts, extension/src/test/helpers/scopedFakeTimers.ts, extension/src/test/scopedFakeTimers.test.ts
extension-unit extension/src/test/cliSpawn.test.ts, extension/src/test/e2eStateFileBridge.test.ts, extension/src/test/helpers/scopedFakeTimers.ts, extension/src/test/scopedFakeTimers.test.ts

Selection computed for commit b36b299.

@eerhardt
Eric Erhardt (eerhardt) merged commit 9bc7ff4 into main Nov 11, 2026
95 checks passed
@microsoft-github-policy-service microsoft-github-policy-service Bot added this to the 17.1 milestone Nov 11, 2026
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

✅ No documentation update needed.

docs_optional → test_only. No signals triggered. All 4 changed files match extension/src/test/** (test files and test helpers only).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants