Skip to content

Defer cross-context actions on idle actors instead of aborting - #7857

Open
jqmmes wants to merge 2 commits into
mainfrom
joaquim/fix-io-context-run
Open

jqmmes wants to merge 2 commits into
mainfrom
joaquim/fix-io-context-run

Conversation

@jqmmes

@jqmmes jqmmes commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

An actor's IoContext outlives its requests. When another IoContext settles one of its promises, or forwards an AbortSignal abort, the action is queued, and the delete queue signal task runs it with IoContext::run(). An idle actor, such as one holding hibernatable WebSockets, has no current IncomingRequest, so the run failed and the signal task aborted the IoContext, breaking the actor for all its clients. The same happened if the request ended while the run waited for a lock.

This PR changes that:

  • IoContext::run() checks for a current request when it's called and again after acquiring each lock. Without one, the callback doesn't run, and run() fails with a JavaScript Error carrying NO_CURRENT_INCOMING_REQUEST_DETAIL_ID instead of an untyped internal error. As before, it throws if there's no request when it's called, and its promise rejects if the request ends while it waits for a lock. The failure is reported as a user error, with a warning logged at most once an hour.
  • The signal task no longer calls run() while the IoContext has no request. The actions stay queued, and IncomingRequest::delivered() runs them as a waitUntil task of the next request, so that request can't finish before they've run. If a run loses its request while it waits for a lock, the actions stay queued and the IoContext is left alone. Any other failure, including an action's own, still aborts the IoContext.
  • A deferred action runs as part of the request it runs under: it counts toward that request's limits, appears in its logs and traces, and has no ordering guarantee relative to the request's handler. Actions still queued when the IoContext ends are dropped, with a warning logged at most once an hour. handle_cross_request_promise_resolution only guarantees that a promise settles in its owning IoContext, not when, so deferring keeps that contract.
  • An AbortSignal received over RPC queues at most one request to arm its subscription, so other contexts adding and removing listeners can't pile up requests in an idle receiver.
  • The signal task re-arms its signal before running the actions, so an action queued during the run still wakes the IoContext. It also takes the actions out of the queue first, so an action can queue another without deadlocking.

@jqmmes
jqmmes force-pushed the joaquim/fix-io-context-run branch 3 times, most recently from d82e59e to 463f413 Compare October 22, 2026 20:40
@jqmmes
jqmmes marked this pull request as ready for review October 23, 2026 18:00
@jqmmes
jqmmes requested review from a team as code owners October 23, 2026 18:00
@clanky-the-clanker

clanky-the-clanker Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

💬 Approved with comments — No defects found. The new behaviour is behind the DEFER_CROSS_CONTEXT_ACTIONS autogate. There is one optional documentation note, and some changed files were not fully checked.

Findings: 1 info · 1 suggestion

The decision is advisory: this comment never approves, blocks or requests changes on the pull request.


📋 Code Quality: 💬 1 info

No defects found. The gating is complete, including the new onHibernate change in worker.c++.

Recommendation: 💬 Optional Nits

  • [INFO] Not fully checked for correctness in this review: src/workerd/api/BUILD.bazel, src/workerd/api/abort-signal-test.c++, src/workerd/api/basics.c++, src/workerd/api/basics.h, src/workerd/io/incoming-request-test.c++, src/workerd/io/io-own-test.c++, src/workerd/io/io-own.c++, src/workerd/io/io-own.h, src/workerd/util/autogate.h. io-context.{h,c++} and worker.c++ were read in full.
View full review
  • Gating: requireCurrentIncomingRequestForRun() only throws the typed error with DEFER_CROSS_CONTEXT_ACTIONS on. The same goes for the delivered() hook that runs queued actions, the new signal-task loop, the DeleteQueuePtr warning, and the rpcSubscriptionRequested limit. With the gate off, the signal task runs its old loop.
  • worker.c++: delivered("onHibernate"_kj, RunQueuedActions::NO) only has an effect when the gate is on. With it off, onHibernate behaves as before. The other delivered() callers still compile against the new defaulted parameter.
  • Signal task: it re-arms the signal before each run. When there is no current request or nothing is queued, it skips the wake. In an actor it also adds the run as a waitUntil task, so the request stays open until the actions have run.
  • runQueuedActions(): the actions stay queued only when the run fails with NO_CURRENT_INCOMING_REQUEST_DETAIL_ID before taking them. Every other failure aborts the IoContext, including an action's own failure.
  • Tests: the tests cover both gate states. The negative cases use IgnoreAllAutogatesEnv::YES, so they also pass in the @all-autogates variant.

This review did not fully check some changed files; they are listed below.


📝 Documentation: 💬 1 suggestion

Optional: the abort-signal reference doesn't say what happens with the autogate off.

Recommendation: 💬 Optional Nits

  • [SUGGESTION] Optional: say what happens with the autogate off next to the autogate notes: a delivery to an IoContext with no current request fails and aborts that context. (docs/reference/detail/abort-signal.md:62)
View full review

docs/reference/detail/abort-signal.md:61-62 and 130-132 explain that with the autogate on, a delivery to an idle actor waits for its next event. They don't say what happens with the gate off: the delivery fails and aborts the IoContext. This is a reference doc for the repository's own developers, so adding it is optional.


✅ All other checks passed

🔒 Security: ✅ LGTM

  • Queue access: takeActions() moves the queue out under an exclusive lock, so the actions run without the lock held.
  • Shared abort payload: SharedAbortPayload is atomically refcounted and holds a const value, so sharing it across threads is safe.
  • Queue growth: queued payloads count as external memory. With the autogate on, each signal makes at most one arming request, so other contexts can't flood an idle receiver.
  • Logging: the new warnings are rate-limited. They log locations and stack traces, not request payloads.

📖 AGENTS.md: ✅ LGTM

The new line in src/workerd/io/AGENTS.md is accurate and follows the repository's comment guidelines.


⚡ Performance: ✅ LGTM

No performance concerns. hasQueuedActions() takes a shared lock once per delivered request or wake. takeActions() moves the whole queue out in one step. The new warnings are logged at most once an hour.


Reviewed commit: 85143edb

@ask-bonk

ask-bonk Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Since last review: 0 resolved, 0 still open, 0 new.
LGTM!


Reviewed commit: 85143edb · github run

Comment thread src/workerd/io/io-context.c++ Outdated
Comment thread src/workerd/api/basics.c++ Outdated
Comment thread src/workerd/io/io-context.c++ Outdated
@jqmmes
jqmmes force-pushed the joaquim/fix-io-context-run branch 2 times, most recently from 181cae7 to 9257f39 Compare October 23, 2026 18:43
Comment thread src/workerd/io/io-context.c++
@jqmmes
jqmmes force-pushed the joaquim/fix-io-context-run branch 2 times, most recently from da36ecd to a397ab5 Compare October 23, 2026 23:25
Comment thread src/workerd/api/basics.h Outdated
@jqmmes
jqmmes force-pushed the joaquim/fix-io-context-run branch 2 times, most recently from 33ee348 to 405637d Compare October 24, 2026 00:41
Comment thread src/workerd/api/basics.c++ Outdated
Comment thread src/workerd/api/tests/actor-cross-context-promise-test.js Outdated
@jqmmes
jqmmes force-pushed the joaquim/fix-io-context-run branch 2 times, most recently from 4e05894 to 086b8eb Compare October 24, 2026 01:19
Comment thread src/workerd/api/basics.c++ Outdated
@jqmmes
jqmmes force-pushed the joaquim/fix-io-context-run branch from 086b8eb to b7c28c8 Compare October 27, 2026 10:33
Comment thread src/workerd/api/abort-signal-test.c++
@jqmmes
jqmmes force-pushed the joaquim/fix-io-context-run branch 5 times, most recently from 2c8b392 to 58ec246 Compare October 28, 2026 13:14
An actor's IoContext outlives its requests, so IoContext::run() can
be called while the actor is idle, or lose its request while it
waits for a lock. run() then failed with an untyped internal error.

Now run() checks for a request when called and after taking each
lock. Without one, it skips the callback and fails with a JavaScript
Error carrying NO_CURRENT_INCOMING_REQUEST_DETAIL_ID, so callers can
tell this case apart. JavaScript sees its message instead of
"internal error". An aborted context fails with its abort exception
instead, as run() already does when it has a request.

To keep a trail for the cases that are bugs, a run called without a
request logs a warning, which reaches Sentry, at most once an hour.
A run that loses its request while waiting for a lock doesn't log:
the request can end during any wait, and the stack trace would only
show the event loop.

This is behind a new autogate, DEFER_CROSS_CONTEXT_ACTIONS, named
for the deferral of cross-context actions that the next commit
builds on this. With it off, run() fails as before.

The delete queue signal task still aborts the IoContext on this
error; the next commit fixes that.
Other IoContexts can queue actions on an IoContext, e.g. to settle
one of its promises or deliver an AbortSignal's abort. Running them
needs a current IncomingRequest. An idle actor, such as one holding
hibernatable WebSockets, has none, so the run failed and aborted the
IoContext, breaking the actor for all its clients.

Now actions queued while an actor has no request wait for its next
request, which runs them as a waitUntil task, counting toward its
limits. A run that loses its request while waiting for a lock also
leaves them queued. Any other failure, including an action's own,
still aborts the IoContext. Actions still queued when the IoContext
ends are dropped, with a NOSENTRY warning.

The onHibernate event doesn't run them: they would use up its
handler's CPU budget, and the cleanup after the handler would cancel
whatever they start. They stay queued until the IoContext ends.

The isolate's memory limit bounds the queue, since actions keep what
they act on alive on its heap. Two kinds of action escaped it:

- A request to arm the RPC subscription of an AbortSignal received
  over RPC holds only a weak reference. Queue at most one per
  signal.
- An abort copied its reason for each queued delivery and RPC clone,
  without counting the copies. Now the native deliveries share one
  copy and the RPC clones another, and both count as external
  memory.

The arm request also now checks its weak reference with tryAddRef(),
since a garbage collection may condemn the signal before it runs.

handle_cross_request_promise_resolution only guarantees which
IoContext a promise settles in, not when, so deferring keeps its
contract. The old behavior broke the actor, so we accept the risk of
changing it without a compatibility flag.

The DEFER_CROSS_CONTEXT_ACTIONS autogate from the previous commit
also covers the deferral, the one-per-signal limit and the external
memory accounting of the shared copies. With it off, they behave as
before. The sharing itself and tryAddRef() apply either way: sharing
an immutable copy changes no behavior, and tryAddRef() keeps the arm
request from using a condemned signal.
@jqmmes
jqmmes force-pushed the joaquim/fix-io-context-run branch from 58ec246 to 85143ed Compare October 28, 2026 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants