Skip to content

feat(runtime): add worker lifetime leases for external owners - #1303

Merged
Coldwings merged 5 commits into
mainfrom
feat/worker-lifetime-lease-1301
Oct 23, 2026
Merged

Coldwings merged 5 commits into
mainfrom
feat/worker-lifetime-lease-1301

Conversation

@Coldwings

@Coldwings Coldwings commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Description

Add an explicit worker-lifetime contract for long-lived external thread-affine
owners. Affinity remains a placement request; a lease retains an exact-owner
continuation and contributes to worker retirement accounting even when no
native I/O is pending.

Type of Change

  • New feature (non-breaking change that adds functionality)
  • Documentation (changes to documentation, comments, or examples)
  • Tests (adding or modifying tests)
  • Build/CI (focused test and example targets)
  • Breaking change

Related Issues

Closes #1301
Native ublk integration and a public low-level io_uring API are out of scope;
#1302 was closed as not planned. External owners may use independently created
raw rings, with their progress and teardown managed by the application.

Changes Made

Core Changes

  • Add runtime::lease_current_worker() and move-only external_worker_lease.
  • Serialize acquisition with scheduler shutdown and worker retirement admission;
    reuse existing logical-task pin and context active-operation accounting.
  • Preserve exact-owner routing for the acquiring logical task through shrink.
  • Safely serialize release/wakeup with context destruction. Token metadata may
    outlive forced scheduler teardown without dereferencing a dead scheduler.
  • Bound retiring task service and poll without blocking when local work remains,
    so a repeatedly yielding owner cannot starve native I/O or maintenance.

API Changes

Previously, public affinity APIs could not retain an external owner's worker.

auto owner = co_await elio::runtime::lease_current_worker();
// Drive, drain, and destroy external owner-thread resources.
owner.release(); // idempotent; RAII fallback also supported

Serialized foreign-thread release is supported. Moving a token retains the
original logical task's constraint; it does not transfer routing to another
task. Forced shutdown and the existing graceful timeout policy remain
non-graceful escape hatches. A lease does not drive external completion queues.

Migration Guide

Additive API; no migration required for existing callers. The new worker-lease
guide documents caller responsibilities and shutdown boundaries.

Testing

Unit Tests

  • Added deterministic tests for owner retention across shrink, closed
    admission, bounded yielding-owner service, move/multiple/idempotent release,
    foreign release, graceful waiting, forced teardown, off-worker rejection,
    exception unwind, and distinct scheduler generations.
  • Focused suite passes with explicit epoll and auto-detected native backend
    coverage: 19 test cases, 119 assertions; 20 repeated seeded runs pass. Explicit
    io_uring coverage is included and reports unavailability as a skip.
  • Full current-head core configuration passes 1070 test cases and 22977
    assertions with RNG seed 1301 (TLS/HTTP/HTTP2 disabled). CI's broader protocol
    and architecture matrices remain required.

Integration Tests

  • Runnable worker_lease example shrinks the pool and cleans up on worker 1.
  • Focused CTest entries pass.
cmake -S . -B build-agent -DCMAKE_BUILD_TYPE=Debug \
  -DELIO_BUILD_TESTS=ON -DELIO_BUILD_EXAMPLES=ON \
  -DELIO_ENABLE_TLS=OFF -DELIO_ENABLE_HTTP=OFF -DELIO_ENABLE_HTTP2=OFF \
  -DELIO_ENABLE_DEVELOPER_WARNINGS=ON -DELIO_WARNINGS_AS_ERRORS=ON
cmake --build build-agent --target elio_worker_lease_tests worker_lease --parallel 2
ctest --test-dir build-agent -R '^worker_lease(_example)?$' --output-on-failure

Sanitizer Testing

  • Focused suite compiled with ASAN+UBSAN and passes with leak detection and
    halt-on-error enabled (normal Debug dependency libraries).
  • Focused current routing-fix suite passes TSAN with halt-on-error enabled
    (normal Debug dependency libraries).

Checklist

Code Quality

  • Follows repository style and remains header-only.
  • Comments explain admission, bounded service, and teardown ordering.
  • No generated files or local build artifacts included.
  • Focused targets build with developer warnings treated as errors.

Documentation

  • Updated wiki API reference/contracts and added worker-lease guide.
  • Added runnable example and README links.
  • Updated CHANGELOG.

Compatibility

  • Existing APIs remain source-compatible.
  • Both backend paths retain the same lease contract.
  • Force shutdown is explicitly not made graceful by a lease.

Performance

  • No per-I/O lease work required; admission/release are owner-lifetime work.
  • Lease-free notification takes a read-only ownership check and does not
    acquire the endpoint mutex; no per-I/O lease acquisition/release is needed.

Additional Notes

Independent review found two owner-routing composition defects in the initial
revision. Both were reproduced by deterministic regressions before the routing
fix. Fresh-thread notification/cancellation, different-scheduler notification,
isolated-scope return, both directions of generator producer/consumer transfer,
blocking/timer/group completion rejection, captured blocking endpoint release,
stale foreign endpoint dispatch, and concurrent release/context destruction
are now covered. Two fresh independent read-only reviews report no blocking objections for
the current implementation, with residual coverage limits recorded in the
review comment.

The focused validation does not prove every concurrency interleaving. All
current-revision CI checks have now completed successfully, including both
architectures and build types. The README-only job was intentionally skipped
by path filtering. There are no unresolved review threads; the PR is ready for
maintainer consideration and has not been merged.

Reviewer Guidance

Focus on lock ordering, admission versus shrink/shutdown, cross-thread final
release versus context destruction, exact-owner routing for suspended tasks,
and progress of repeatedly requeued owners during drain. Check caller/library
boundaries, particularly moving a token versus transferring a logical task.

Copilot AI balanced review requested due to automatic review settings October 23, 2026 04:08

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Coldwings

Copy link
Copy Markdown
Owner Author

Two independent read-only reviews (concurrency/lifetime and API/contracts/tests)
found two blocking exact-owner composition defects in the current revision:

  1. A leased task waiting on an Elio event can resume directly on a fresh foreign
    notifier thread: the no-scheduler schedule_handle() trampoline ignores the
    retained owner. Cancellation dispatched from a foreign thread has the same
    risk. The original test's controller-thread notification masked this path.
  2. An isolated task_scope() may migrate legitimately, but its final symmetric
    transfer can resume the still-leased parent on the scope's worker instead of
    returning to the parent's retained OS thread.

Both concerns are accepted. Deterministic regressions and owner-aware routing
fixes are being added; the next revision will receive fresh review. The
acquisition/accounting and wake-slot teardown ordering had no additional
blocking objections on static inspection. Existing passing focused/sanitizer
tests do not cover these composition paths and are not a safety verdict.

@Coldwings

Copy link
Copy Markdown
Owner Author

The refreshed API/contracts review has no blocking objection to the routing-fix
revision, but the independent concurrency review found another blocking
composition path: async generator yield/exhaustion and for_each() still
symmetrically transfer into a leased consumer on the producer's thread.

The scenario was reproduced deterministically with a producer suspended on an
event, followed by worker shrink and producer resumption on a surviving worker.
All four combinations of yield/exhaustion and direct iteration/for_each()
violated consumer ownership before the fix. This is a library-managed transfer,
not a custom foreign executor outside the lease contract.

The next revision restores ownership at generator consumer transfers and keeps
the for_each() loop in its Elio caller's execution context. Fresh exact-head
review and regression validation will cover this additional change; the PR is
not ready on the strength of the earlier passes.

@Coldwings

Copy link
Copy Markdown
Owner Author

The latest revision addresses the two additional independent-review blockers:

  • Generator next() now restores an independently leased producer before
    entering its body. Yield/exhaustion still restore an independently leased
    consumer. A multi-yield producer/consumer regression failed before this fix
    and passes afterward.
  • A stale endpoint captured before final release now retains its original
    scheduler domain rather than falling back to notifier TLS. Deterministic
    capture → release → worker retirement → pool dispatch reproduced the old
    blocking-completion deadlock under an external timeout. The same regression
    now completes on a scheduler worker. Foreign-thread stale-endpoint dispatch
    also checks the scheduler domain explicitly.

Self-audit additionally reproduced and fixed leased group-join rejection
fallback and blocking/timer completion rejection paths. Normal focused
validation now passes 19 test cases and 119 assertions, including explicit
io_uring availability reporting. Updated sanitizer/full-regression validation
and fresh exact-head independent verdicts are being collected; earlier green
checks are not a verdict on this revision.

@Coldwings

Copy link
Copy Markdown
Owner Author

Two independent read-only reviews of the current revision, covering
concurrency/lifetime and API/contracts/tests, report no blocking objections.

The reviews covered admission versus retirement/shutdown, pin accounting,
cross-thread release and wake-slot teardown, scheduler-domain preservation after
final release, isolated-scope return, generator producer/consumer transfers in
both directions, and group/blocking/timer completion paths. Previously reported
blockers have deterministic regressions and have been addressed.

Current-revision local evidence:

  • Focused suite: 19 test cases, 119 assertions; 20 seeded repetitions pass.
  • Focused ASAN+UBSAN (including leak detection) and TSAN runs pass with
    halt-on-error enabled; dependency libraries use the normal Debug build.
  • Both focused CTest entries and the runnable shrink/owner-cleanup example pass.

Residual limits: the focused runs are not an exhaustive concurrency proof;
combined generator/shrink scenarios and every destruction ordering are not all
deterministically enumerated. Owner service coverage checks bounded polling but
does not independently establish every native-I/O load/fairness behavior.
Forced destruction of live suspended tasks/notifications is not newly supported;
token metadata lifetime is a separate protected guarantee.

Full current-revision regression and CI completion remain required before the
PR is ready. Review must be refreshed if the implementation changes materially.

@Coldwings

Copy link
Copy Markdown
Owner Author

The current revision also passes the complete locally configured core regression suite: 1070 test cases, 22977 assertions, RNG seed 1301, with TLS/HTTP/HTTP2 disabled. Focused ASAN+UBSAN and TSAN checks and both independent review verdicts cover this same revision. The broader CI matrices are still running and remain required.

@Coldwings

Copy link
Copy Markdown
Owner Author

All current-revision CI checks are now complete and successful, including the
x64/arm64 Debug/Release matrices, package-consumer and benchmark compilation,
CodeQL, and HTTP streaming contract checks. The README-only check was
intentionally skipped by the path filter. There are no unresolved review threads
and the PR is mergeable.

The two independent read-only no-blocker reviews, focused sanitizer results,
and the 1070-case local core regression cover this same revision. No additional
implementation changes have been made since those reviews. This PR is ready for
maintainer consideration; it has not been merged.

@Coldwings

Copy link
Copy Markdown
Owner Author

Scope is now explicitly limited to worker-lifetime leases. Native ublk integration (#1302) is closed as not planned, and no public low-level io_uring API is included.

Final validation of the unchanged implementation revision:

  • Both focused CTest entries pass (lease suite and runnable example).
  • The 19-case, 119-assertion focused suite passes ASAN+UBSAN with leak detection and TSAN with halt-on-error enabled.
  • A supplemental public-API-only smoke test creates an independent SINGLE_ISSUER raw ring on worker 1, shrinks the pool to one worker, resumes the owner from a fresh foreign notifier thread, and completes raw-ring submission, CQ consumption, and destruction on the original owner before releasing the lease. It passes ASAN+UBSAN. This is a ring-lifetime smoke test, not a ublk protocol or fairness proof.

The implementation has not changed since the two independent no-blocker reviews and successful CI. All checks remain complete (the README-only check is intentionally skipped), there are no review threads or inline objections, and the PR is mergeable. It remains unmerged for maintainer consideration. External ring progress, cancellation, fairness, and protocol recovery remain application responsibilities.

@Coldwings
Coldwings merged commit f55528d into main Oct 23, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Expose a worker-lifetime lease for external worker-local owners

2 participants