Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions bench/ffi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,9 +146,14 @@ frame path:
asyncio's, as BoltFFI's Python measured 10 us. This case starts a task per read; a
reader looping on one task would skip the start, and is what to measure before changing
a generator.
4. **A future per read.** A ready read costs C 155 ns over the write where Rust costs 73:
the future's three allocations and three crossings. Each language adds 160 (Swift) to 420 (Go) ns
more in its own wrapper, so the C ABI's share is not yet where a per-handle poll would pay.
4. **A future per read.** Since `<fn>_start` makes a future and polls it in one call, a
ready read costs C 126 ns over the write where Rust costs 93: one crossing and one
allocation, down from 176 with four allocations and three crossings. Measured 2026-11-13
on an AMD Ryzen 9 9950X (32 threads) at a load average of 26, 12 rounds alternating with
the commit before, so the tables above predate it. Over their writes, Swift pays 289,
Go 414, Dart 522, Python 445, and Kotlin 590 (from 381, 556, 868, 659, and 615). Go's and
Dart's include the returned frame's ownership, about 190 and 215 ns over C's object return;
the rest is each language's own await.
5. **Kotlin's record codec.** A record return costs Kotlin 113 ns wall against C's 31,
through the byte-array codec; Swift, which reads the C struct, costs 37.
6. **The handle borrow at the C ABI.** The u64 getter costs Rust 0.4 ns and C 10: the
Expand Down
21 changes: 12 additions & 9 deletions ffi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ re-exports its items instead, which export. A type no path reaches is refused.
| `Result<T, E>` | `bool f(..., T *out, E *err)`; an `E` of another crate that implements `Display`, such as `anyhow::Error` or `io::Error`, is `String`, its message |
| a returned tuple `(A, B)` | one out pointer each: `void f(..., A *out0, B *out1)` |
| a tuple inside another type, or as a field or parameter | `<prefix>_tuple_<A>_<B>`: a struct of fields `_0`, `_1`, and so on; of references only returned |
| `async fn`, or a return that implements `Future` | a future: `f` makes it, `f_poll` polls it, `f_future_free` drops it; a function that is not `async` runs at the call, as in Rust |
| `async fn`, or a return that implements `Future` | a future: `f` makes it, `f_poll` polls it, `f_future_free` drops it, and `f_start` makes it and polls it once, returning a handle only if it pends; a function that is not `async` runs at the call, as in Rust |
| a function returning `Poll<T>` that polls through a `&Context`, a `&mut Context`, or a reference to an object exactly one associated function makes from a `Waker` alone, of a type with no generic parameters, such as kio's `Waiter`, which may be the method's own `&self`, as `Waiter::poll_future`'s is | a future of `T`, as an `async fn`'s: each poll calls the function again with the poll's context, or with an object made from its waker and kept until the next poll, so what it registers stays registered; each other parameter taken by value must be `Copy`; a method polling through its own `&self` is a function without one |
| `poll_x`, when `x` exports as a future | left out: the future is its poll |
| `bool`, integers but 128-bit, floats, `char` | `bool`, `<stdint.h>` types, `float`, `double`, a `uint32_t` code point |
Expand Down Expand Up @@ -257,7 +257,10 @@ where it was while its file is unchanged.
A poll that finishes writes the result through the poll's own out pointers, so a
cancelled read never loses a value. The future reuses the last waker when given the same
one again, and owns what it borrowed: the handle it was called on stays borrowed until
it finishes or is freed.
it finishes or is freed. `_start` is the call and a first poll with no waker in one:
a future ready at once costs one call and no handle, and one that pends writes its
handle through the last parameter, which must not be null. A null waker wakes nothing
and costs no allocation, and a future is one allocation.
- **Wakers** move into Rust, which calls `drop` once it lets go of the last clone, so a
state machine polled with a waker, such as moq's `Driver`s through kio's `Waiter`, runs
from the caller's event loop. Rust may call `wake` and `drop` from any thread, as a
Expand Down Expand Up @@ -663,9 +666,9 @@ and, for `Swift.Duration`, macOS 13 or iOS 16, and builds in either language mod
as its `io::Error`'s message.
- **Futures** are `async throws` functions, which resume on the task's executor, never
inside a wake. Cancelling the task frees the future, losing nothing, and throws
`CancellationError`. Each call is its own future, so it has one poller. The call polls it
once before suspending, and makes the signal it parks on only if that poll pends; the
signal latches a wake that lands before it parks.
`CancellationError`. Each call is its own future, so it has one poller. The call makes
and polls it in one crossing, `_start`, before suspending, and makes the signal it parks
on only if that poll pends; the signal latches a wake that lands before it parks.
- A **future Swift implements** is an `@escaping @Sendable () async -> Void`, which Rust's
first poll runs as a task of its own. Rust dropping it cancels the task, and one dropped
before it starts never runs. A poll lent one, such as `Waiter.pollFuture`, is a static
Expand Down Expand Up @@ -780,8 +783,8 @@ reads a go.mod.
a reader that is also an `rt.StartSeeker`, as `io.Copy` uses an `io.WriterTo`, and fails
in any other. Their errors cross as their messages.
- **Futures** are calls taking a `context.Context` first, which block until the future is
ready. The call polls it once before it waits, and makes the waker it waits on only if
that poll pends; a wake from any thread signals the waiting goroutine, which polls
ready. The call makes and polls it in one crossing, `_start`, before it waits, and
makes the waker it waits on only if that poll pends; a wake from any thread signals the waiting goroutine, which polls
again, so a wake never polls inside Rust, and one that lands before the wait is kept. A
context done before the call, or while it waits, returns its error and frees the future,
which cancels it in Rust and loses nothing: the next call takes what it would have.
Expand Down Expand Up @@ -872,8 +875,8 @@ the library's own URI, mapped to the library by a build hook or by `--native-ass
throws aborts.
- **Futures** are `Cancelable<T>`, a `Future<T>` completed on the isolate that made it:
an extension type on Dart's own future, which `await` takes its shortest path through.
The call polls it once, returning it complete if that poll finished, and makes a waker
only if it pends. A wake, from any
The call makes and polls it in one crossing, `_start`, returning it complete if that
poll finished, and makes a waker only if it pends. A wake, from any
thread, posts to the isolate's port, and its event loop polls again, so a wake never
polls inside Rust, and one that lands during a poll is kept. `cancel()` frees the
future unless it finished, which cancels it in Rust and loses nothing: the next call
Expand Down
12 changes: 12 additions & 0 deletions ffi/fixtures/api/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1015,6 +1015,18 @@ fn on_thread(log: &(impl Log + ?Sized), lines: &[u8]) -> u32 {
})
}

/// A record named as Swift's own error a cancelled task throws, which the binding renames.
pub struct CancellationError {
pub code: u32,
}

/// Each of `0..n` to `count` before it returns its future, then `n` doubled once polled,
/// or `Overflow`: an eager, fallible future that lends a closure.
pub fn counted(n: u32, mut count: impl FnMut(u32)) -> std::future::Ready<Result<u32, Error>> {
(0..n).for_each(&mut count);
std::future::ready(n.checked_mul(2).ok_or(Error::Overflow))
}

/// Each line to `keep` on another thread, returning how many it kept: a lent closure may
/// run on any thread until the call returns.
pub fn filter(lines: Vec<u8>, mut keep: impl FnMut(u8, &str) -> bool + Send) -> u32 {
Expand Down
6 changes: 3 additions & 3 deletions ffi/moq/bench/export.c
Original file line number Diff line number Diff line change
Expand Up @@ -65,14 +65,14 @@ static moq_group_Consumer *recv_group(moq_track_Subscriber *subscriber) {
return reader;
}

/* A read ready at once: made and polled in one call, with no handle. */
static moq_frame_Frame *read_frame(moq_group_Consumer *reader) {
moq_group_Consumer_read_frame_future *reading = moq_group_Consumer_read_frame(reader);
moq_group_Consumer_read_frame_future *reading = NULL;
moq_frame_Frame *read = NULL;
moq_Error *err = NULL;
if (moq_group_Consumer_read_frame_poll(reading, waker, &read, &err) != MOQ_POLL_READY || read == NULL) {
if (moq_group_Consumer_read_frame_start(reader, &read, &err, &reading) != MOQ_POLL_READY || read == NULL) {
fail("read_frame");
}
moq_group_Consumer_read_frame_future_free(reading);
return read;
}

Expand Down
18 changes: 6 additions & 12 deletions ffi/runtime/ffi_cgo.go
Original file line number Diff line number Diff line change
Expand Up @@ -362,19 +362,13 @@ func reader_(k *cgo.Handle) cForeignReader {
return cForeignReader{read: goRead, seek: goSeek, data: unsafe.Pointer(k)}
}

// await_ polls a future until poll reports it finished, returning its error, or ctx's
// once ctx is done: then the caller frees the future unfinished, which cancels it in Rust
// and loses nothing, and the next call sees what it would have taken. Its first poll makes
// no waker, which only a future that pends needs; a wake that lands between a poll that
// pends and the wait is kept, so the wait returns at once.
// await_ polls a future whose call pended until poll reports it finished, returning its
// error, or ctx's once ctx is done: then the caller frees the future unfinished, which
// cancels it in Rust and loses nothing, and the next call sees what it would have taken.
// The call polled it first with no waker, which only a future that pends needs, and
// checked ctx before its first poll; a wake that lands between a poll that pends and the
// wait is kept, so the wait returns at once.
func await_(ctx context.Context, poll func(cWaker) (bool, error)) error {
// A context done already takes nothing, even what is ready.
if err := ctx.Err(); err != nil {
return err
}
if done, err := poll(cWaker{}); done {
return err
}
woken := make(chan struct{}, 1)
waker := waker_(func() {
select {
Expand Down
17 changes: 6 additions & 11 deletions ffi/runtime/rt.dart
Original file line number Diff line number Diff line change
Expand Up @@ -62,14 +62,9 @@ extension type Cancelable<T>._(Future<T> _future) implements Future<T> {
void cancel() => _n.calls[_future]?.cancel();
}

/// Poll `f` once with no waker, and if it pends, again with one: its future.
Cancelable<T> _call<T>(
ffi.Pointer<ffi.Void> f,
void Function(ffi.Pointer<ffi.Void>, ffi.Pointer<ffi.Void>) free,
bool Function(ffi.Pointer<ffi.Void>, _Waker, _Call<T>) poll,
) {
final call = _Call<T>._(f, free, poll);
call._step(_n.idle);
/// The future of `call`, which its start completed if its first poll finished, or else
/// left the future in Rust that pended, polled again here with a waker.
Cancelable<T> _call<T>(_Call<T> call) {
final first = call._first;
if (first != null) return Cancelable._(first);
final done = call._done = dasync.Completer<T>();
Expand All @@ -83,7 +78,8 @@ Cancelable<T> _call<T>(

/// A [Cancelable]'s future in Rust, until it finishes.
final class _Call<T> {
ffi.Pointer<ffi.Void> _f;
/// The future in Rust, once its start pended, until it finishes.
ffi.Pointer<ffi.Void> _f = ffi.nullptr;
final void Function(ffi.Pointer<ffi.Void>, ffi.Pointer<ffi.Void>) _free;

/// Poll once, completing this if it finished: whether it did.
Expand All @@ -99,7 +95,7 @@ final class _Call<T> {
bool _polling = false;
bool _cancelled = false;

_Call._(this._f, this._free, this._poll);
_Call._(this._free, this._poll);

void _wake() {
if (_f != ffi.nullptr) _step(_waker!);
Expand Down Expand Up @@ -524,7 +520,6 @@ final class _Native {
late final _Stack stack = _Stack();
late final isolate.RawReceivePort _port;
late final ffi.Pointer<ffi.Void> home;
final _Waker idle = ffi.Struct.create<_Waker>();

/// What a token's wake calls, and its drop.
final Map<int, void Function()> _wakes = {};
Expand Down
159 changes: 125 additions & 34 deletions ffi/runtime/rt.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1035,7 +1035,11 @@ impl Waker {
/// returned stays wrapped too, so its `data` names it for as long as a future may
/// compare it.
pub fn lift(self) -> std::task::Waker {
std::task::Waker::from(Arc::new(Foreign(self)))
match (self.wake, self.drop) {
// Nothing to wake or release: a ready future's first poll takes no allocation.
(None, None) => std::task::Waker::noop().clone(),
_ => std::task::Waker::from(Arc::new(Foreign(self))),
}
}

/// Wake it, as C does.
Expand Down Expand Up @@ -1308,30 +1312,53 @@ impl std::io::Seek for Reader {
}
}

/// Writes a finished future's result through the poll's out pointers, one per value it
/// returns, and its error pointer, returning the poll. The poll that finishes the future
/// calls it, so it never crosses threads.
pub type Done = Box<dyn FnOnce(&[*mut c_void], *mut c_void) -> i32>;

pub const PENDING: i32 = 0;
pub const READY: i32 = 1;
pub const ERROR: i32 = 2;

/// A future behind a handle, polled from C.
/// A future behind a handle, polled from C: the head of the one allocation holding the
/// future and the function that lowers its result, which [`Future::new`] makes.
#[repr(C)]
pub struct Future {
future: Option<Pin<Box<dyn std::future::Future<Output = Done> + Send>>>,
/// Poll the allocation, lowering the result through the poll's out pointers once ready.
poll: unsafe fn(*mut Future, Waker, &[*mut c_void], *mut c_void, &str) -> i32,
/// Drop the allocation.
drop: unsafe fn(*mut Future),
/// The last foreign waker, and the Rust waker around it.
waker: Option<(Waker, std::task::Waker)>,
polling: AtomicBool,
}

/// The allocation behind a handle: its head first, so the handle points at both.
#[repr(C)]
struct Typed<F, L> {
head: Future,
/// The future, pinned here until it finishes and drops in place.
future: Option<F>,
/// Writes the finished future's result through the poll's out pointers, one per value
/// it returns, and its error pointer, returning the poll. The poll that finishes the
/// future calls it, so it never crosses threads.
lower: L,
}

impl Future {
pub fn new(future: impl std::future::Future<Output = Done> + Send + 'static) -> *mut Self {
Box::into_raw(Box::new(Self {
future: Some(Box::pin(future)),
waker: None,
polling: AtomicBool::new(false),
}))
pub fn new<F, L>(future: F, lower: L) -> *mut Self
where
F: std::future::Future + Send + 'static,
// Send, as any poll may finish the future on another thread.
L: Fn(F::Output, &[*mut c_void], *mut c_void) -> i32 + Send + 'static,
{
let typed = Box::new(Typed {
head: Self {
poll: poll_typed::<F, L>,
drop: drop_typed::<F, L>,
waker: None,
polling: AtomicBool::new(false),
},
future: Some(future),
lower,
});
Box::into_raw(typed).cast()
}

/// Poll it once with `waker`, writing the result through `outs` and `err` when ready.
Expand All @@ -1344,29 +1371,51 @@ impl Future {
fail(what, "a future is null");
}
// SAFETY: the caller's contract; only the flag is touched before it is held.
if unsafe { &*future }.polling.swap(true, Ordering::Acquire) {
let head = unsafe { &*future };
if head.polling.swap(true, Ordering::Acquire) {
fail(what, "a future is polled concurrently");
}
// SAFETY: the flag makes this the only access.
let this = unsafe { &mut *future };
let Some(inner) = this.future.as_mut() else {
fail(what, "a future is polled after it finished");
};
if !this.waker.as_ref().is_some_and(|(last, _)| last.same(&waker)) {
this.waker = Some((waker, waker.lift()));
// SAFETY: the flag makes this the only access, and `new` made the allocation.
let result = unsafe { (head.poll)(future, waker, outs, err, what) };
// SAFETY: as above; the allocation outlives the poll.
unsafe { &*future }.polling.store(false, Ordering::Release);
result
}

/// Make the future's first poll, with no waker: on `READY` or `ERROR` the result is
/// written, the future freed, and null written through `handle`; on `PENDING` its
/// handle goes there, for C to poll with a waker and free.
///
/// # Safety
/// `future` is a handle `new` just made, `handle` is null, which aborts, or points
/// where a handle goes, and each of `outs` and `err` is null or points where the function's result
/// types go.
pub unsafe fn start(
future: *mut Self,
outs: &[*mut c_void],
err: *mut c_void,
handle: *mut *mut Self,
what: &str,
) -> i32 {
if handle.is_null() {
fail(what, "a future's handle pointer is null");
}
let (_, rust) = this.waker.as_ref().expect("set above");
let mut cx = Context::from_waker(rust);
let poll = inner.as_mut().poll(&mut cx);
let result = match poll {
std::task::Poll::Pending => PENDING,
std::task::Poll::Ready(done) => {
this.future = None;
this.waker = None;
done(outs, err)
}
let none = Waker {
wake: None,
drop: None,
data: std::ptr::null_mut(),
};
this.polling.store(false, Ordering::Release);
// SAFETY: the caller's contract.
let result = unsafe { Self::poll(future, none, outs, err, what) };
match result {
// SAFETY: the caller's contract.
PENDING => unsafe { handle.write(future) },
// SAFETY: it finished; null tells C there is no handle, and `free` takes it.
_ => unsafe {
handle.write(std::ptr::null_mut());
Self::free(future)
},
}
result
}

Expand All @@ -1377,11 +1426,53 @@ impl Future {
pub unsafe fn free(future: *mut Self) {
if !future.is_null() {
// SAFETY: the caller's contract.
drop(unsafe { Box::from_raw(future) });
unsafe { ((*future).drop)(future) };
}
}
}

/// [`Future`]'s poll of a `Typed<F, L>`.
///
/// # Safety
/// `future` is a live `Typed<F, L>` that only this call accesses, and each of `outs` and
/// `err` is null or points where the function's result types go.
unsafe fn poll_typed<F: std::future::Future, L: Fn(F::Output, &[*mut c_void], *mut c_void) -> i32>(
future: *mut Future,
waker: Waker,
outs: &[*mut c_void],
err: *mut c_void,
what: &str,
) -> i32 {
// SAFETY: the caller's contract.
let Typed { head, future, lower } = unsafe { &mut *future.cast::<Typed<F, L>>() };
let Some(inner) = future.as_mut() else {
fail(what, "a future is polled after it finished");
};
if !head.waker.as_ref().is_some_and(|(last, _)| last.same(&waker)) {
head.waker = Some((waker, waker.lift()));
}
let (_, rust) = head.waker.as_ref().expect("set above");
let mut cx = Context::from_waker(rust);
// SAFETY: the allocation never moves, and the future drops in place.
match unsafe { Pin::new_unchecked(inner) }.poll(&mut cx) {
std::task::Poll::Pending => PENDING,
std::task::Poll::Ready(value) => {
*future = None;
head.waker = None;
lower(value, outs, err)
}
}
}

/// [`Future`]'s drop of a `Typed<F, L>`.
///
/// # Safety
/// `future` is a live `Typed<F, L>`, dropped once.
unsafe fn drop_typed<F, L>(future: *mut Future) {
// SAFETY: the caller's contract; `new` boxed it.
drop(unsafe { Box::from_raw(future.cast::<Typed<F, L>>()) });
}

/// Write `value` through `out` if the caller wants it, or drop it.
///
/// # Safety
Expand Down
Loading