Skip to content

[Feature] Add native ublk target support integrated with the I/O runtime #1302

Closed as not planned

Description

@Coldwings

Feature Description

Add a native, header-only ublk protocol adapter integrated with Elio's I/O
runtime. Applications should be able to implement a virtual block target using
Elio coroutines without owning a second data-plane io_uring or a separate event
loop. This is a concrete ublk interface, not a public arbitrary-SQE extension.

Problem Statement

At v0.6.0 (c662f33), io_op has no ublk command and the private backend owns
SQE submission, completion demultiplexing, pending accounting, and operation
lifetime. A ublk target therefore has to maintain its own ring and progress
policy, although its backing file/network I/O already uses Elio.

An external worker lease (#1301) solves retirement of an independently owned
thread-affine resource. It does not integrate that resource's submissions,
completions, or registered resources with Elio's ring. These are independent
features, not alternatives or a mandatory dependency chain.

The original proposal was based on static inspection. Implementation and
validation are in progress; no performance superiority is claimed. Native
shared-ring integration and process-crash recovery are requirements to prove,
not completed capabilities inferred from normal shutdown tests.

Proposed Solution

Provide an optional elio::ublk layer with a bounded, explicit protocol and
lifecycle contract:

  1. Device control: create, configure geometry/capabilities, start, stop, and
    delete a device, with errors preserving kernel errno.
  2. Queue service: map descriptors, pre-arm fetches, dispatch requests to an
    application coroutine handler, and commit results/re-arm tags.
  3. Shared data plane: queue commands and normal backing I/O use the owning
    Elio worker's native backend, with no application-owned extra ring or CQ
    consumer. Keep backend submission/completion hooks private.
  4. Ownership: retain the exact queue owner OS thread for every command and
    teardown, including gaps between individual native operations and scheduler
    shrink. Queue state must remain alive through terminal completions.
  5. Explicit request/tag transitions prevent reuse before completion, duplicate
    commits, and destruction of kernel-visible buffers. Preserve request flags
    needed by handlers; handle read, write, and flush, and expose discard and
    write-zeroes only when advertised/supported by the target.
  6. Stop closes admission, aborts/drains fetches, waits for in-flight handlers and
    kernel commands, and only then unmaps/closes resources. Cancellation is
    cooperative; it must not imply asynchronous destruction of a live command.
  7. Unsupported backends, missing kernel/liburing support, incompatible ring
    configuration, and unsupported protocol modes fail explicitly. Do not fall
    back to a hidden independent data-plane ring.
  8. Shared SQ/CQ pressure and runnable handlers must not starve device commands
    or other Elio I/O. Define completion progress and backpressure explicitly.
  9. Mounted-device process-crash recovery is required in the initial feature:
    create with explicit kernel recovery/replay policy, preserve the block device
    and its identity when the serving process crashes, and let a replacement
    process attach to the existing device, re-establish queues, and complete the
    START_USER_RECOVERY / END_USER_RECOVERY protocol. Do not substitute normal
    STOP/DEL, recreation under the same number, or retention of only the char
    node for recovery of the mounted block device.
  10. Distinguish final administrative stop/remove from recoverable loss of a
    serving process. A failed recovery attempt must not silently STOP/DEL the
    retained mounted device. Keep ownership, retry boundaries, and errors
    explicit; require the application to restore its backing target context.
    Replay policy must be explicit: reissuing previously dispatched requests
    can duplicate writes and does not imply exactly-once storage semantics.

API Design

The confirmed direction is phased device control plus structured queue service,
not a sole create/serve/always-delete wrapper. Device creation, a serving session,
recovery attachment, and final removal have separate responsibility boundaries.
Queue/tag/buffer ownership remains internal; raw io_uring is not public.

The exact names will be refined against the implementation. The application
handler supplies storage behavior and returns a byte count or negative errno;
Elio supplies protocol and runtime integration, including the kernel recovery
handshake. The application persists/restores the mapping between device identity
and backing target state; Elio does not reconstruct storage metadata.

Usage Example

Conceptual flow (not an existing compilable API):

// Create/configure device; start queue service on selected Elio workers.
// Each request handler co_awaits ordinary Elio file/network I/O.
// Start the device after fetches are armed.
// Stop the device, await queue/handler settlement, then delete it.

Alternatives Considered

  • Independent ring plus worker lease: remains useful for external owners,
    but does not meet this shared-data-plane integration requirement.
  • Public raw ring access: exposes completion routing, pending accounting,
    registration, and cancellation invariants to callers; not requested.
  • Generic managed native-operation extension: possible future work, but a
    stable arbitrary-operation contract is not a prerequisite for ublk support.
  • Full block storage framework: out of scope. Image formats, caching,
    persistence policy, replication, and target-specific metadata recovery belong
    to the application. This does not exclude the required ublk kernel recovery
    protocol. Zero-copy, zoned devices, and batch/multishot modes remain explicitly
    unsupported until separately implemented, not silently accepted.

Use Cases

  1. File-backed or network-backed ublk targets using Elio for backing I/O.
  2. Replacing a separate queue-thread/ring/eventfd bridge with native coroutine
    request processing.

Implementation Considerations

  • Complexity: High: kernel protocol, exact-thread ownership, teardown,
    buffer lifetime, and shared-ring progress must be validated together.
  • Breaking Changes: None intended; optional additive module.
  • Dependencies: Linux ublk UAPI and suitable liburing/kernel support.
  • Performance Impact: No ranking claimed. Verify shared-ring behavior and
    benchmark submission/completion overhead before claiming improvement.

Validation must include deterministic protocol/state-machine tests without a
device, unsupported-capability tests, native backend operation/lifetime tests,
and opt-in real-kernel end-to-end read/write/flush/stop tests. Recovery tests must
kill the serving process without running cleanup, keep the same mounted block
device, and recover from a new process. Cover queued/in-flight I/O, replay policy,
failed replacement startup without accidental device deletion, repeated recovery,
and eventual administrative teardown. A cooperative stop or same-process handler
exception is not a substitute for this crash test. Real-device tests
must be reported as unverified when /dev/ublk-control or privileges are absent.
Update wiki/API documentation, an executable target example, build/package
gating, and CHANGELOG. Preserve existing epoll-only consumers.

Compatibility

  • C++ Standard: C++20; header-only.
  • Platform: Linux; ublk requires a supporting kernel and permissions.
  • Backward Compatibility: Existing APIs and default consumers remain valid.

Related Work

Additional Context

Keep private backend mechanisms private. A future public native-operation API
should be justified by a complete safety contract and actual independent needs,
not exposed solely as an implementation shortcut for this feature.

Checklist

  • I have searched existing issues to avoid duplicates
  • I have clearly described the problem and proposed solution
  • I have considered alternative approaches
  • I have provided usage examples
  • I have considered backward compatibility

Priority

  • Medium: Enables native integration; external-ring targets remain a
    viable alternative with explicit ownership and progress policies.

Activity

  1. Coldwings commented on Oct 23, 2026

    @Coldwings
    OwnerAuthor

    Implementation progress (not a completed feature):

    • The optional native backend configuration and bounded single-CQE command path
      now use the existing operation/completion accounting. Native resources survive
      awaiter destruction until the terminal completion; prepare rejection is kept
      distinct from a terminal kernel error.
    • Deterministic protocol/runtime tests cover capability and range validation,
      tag transitions, retained resources, cancellation admission retries, and
      independent tag-root ownership through worker shrink and foreign wakeups.
      The current focused suite passes 16 cases and 138 assertions.
    • An isolated Linux 6.18 guest completed an implementation probe using Elio's
      existing worker ring: create/configure, arm four tags, start, direct write/read
      data comparison, actual flush, stop, settle all tags, and delete. The target
      handled four reads, one write, and one flush. This is not a performance result.
    • Real-device validation exposed a pure-FLUSH descriptor whose sector is
      UINT64_MAX. Treating that undefined LBA as a data range incorrectly returned
      EPROTO; a deterministic regression failed before the decoder correction and
      now passes, as does the repeated guest flow.

    The public device/queue lifecycle API, comprehensive failure cleanup, committed
    integration coverage, example, documentation, package/CI gating, and independent
    review remain outstanding. The probe does not replace those deliverables, and
    no arbitrary-SQE public extension has been introduced.

  2. Coldwings commented on Oct 23, 2026

    @Coldwings
    OwnerAuthor

    Planning and implementation of native ublk support are deferred at maintainer request and are no longer part of the current implementation scope. Existing prototype validation does not establish a supported public API or a completed recovery implementation. Any future resumption needs explicit scope and design confirmation. The independent worker-lifetime lease feature (#1301, PR #1303) is unaffected.

  3. Coldwings commented on Oct 23, 2026

    @Coldwings
    OwnerAuthor

    Closing at maintainer request: native ublk integration and a public low-level io_uring API are not planned in the current scope. External libraries may continue to own and drive their own raw rings. The worker-lifetime lease remains the independent feature tracked by #1301 and PR #1303. This closure is a scope decision, not a claim that ublk integration or recovery was implemented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions