diff --git a/bench/ffi/README.md b/bench/ffi/README.md index a140e0dd5..360f6d6e5 100644 --- a/bench/ffi/README.md +++ b/bench/ffi/README.md @@ -405,8 +405,15 @@ frame path: instructions an object, three quarters of it: without them, a return and close took 62 to 65 ns against 176 to 245. Since kixel-ai/rust2#828, a Go object and its handle are one allocation (70 fewer instructions) and a Dart close drops the value without - an `_Arena` (136 fewer). [Batched ownership](../../quest/m1/ffi-speed/object-ownership.md) - would take the registration off objects closed soon after their return. Kotlin's + an `_Arena` (136 fewer). Since kixel-ai/rust2#838, an object closed within 64 returns + of its own skips the registration: each Go P and each Dart isolate holds its last 64 + returned objects in a nursery, registering only one pushed out unclosed, and a checked + object's close drops and frees in one crossing. Measured 2026-11-15 on **nixos** at a + load average of 4 to 24, 8 rounds alternating with main at `72880f48`, median wall ns: + Go's object return fell from 254 to 174 and Dart's from 225 to 96, Go's ready read from + 833 to 728 and Dart's from 1,020 to 786, and Dart's woken read from 2,930 to 2,274. + What Go has left is mostly its two cgo crossings, the call and the close, and the pool + that finds its P's nursery. Kotlin's 137 ns wall (210 CPU) is a phantom reference linked and unlinked under one global lock. Every returned frame, group, and future pays it. 3. **A woken read.** C's 427 ns is Rust's 329 plus a future's crossings. Kotlin's 1.6 us and diff --git a/ffi/README.md b/ffi/README.md index 6bc8d6871..0fe8d62aa 100644 --- a/ffi/README.md +++ b/ffi/README.md @@ -978,8 +978,13 @@ reads a go.mod. would, and returns an error matching `rt.ErrBorrowed` instead while a call on another goroutine borrows the value, as Rust refuses that drop, leaving the value to a later `Close()`. Of any other type, `Close()` frees the handle, and returns an error matching - `rt.ErrBorrowed` while a call holds it. A second `Close()` does nothing, and one never - closed is freed once the GC collects it, through `runtime.AddCleanup`. A call that meets + `rt.ErrBorrowed` while a call holds it; one no call holds, as soon after its return, + drops and frees in one crossing. A second `Close()` does nothing. A returned object waits + in a nursery of the last 64 its P returned, where `Close()` frees it without the GC's + registration (`runtime.AddCleanup` and its `Stop`), which costs more than the rest of a + return and close. One never closed is registered once pushed out, by 64 later returns or + with its nursery once the GC drops that from its `sync.Pool`, and freed once the GC + collects it after. A call that meets a closed object, as its receiver or anywhere in its arguments, returns an error matching `rt.ErrClosed`, so every such call returns an `error`; `String()` and `GoString()`, which return none, return the error's text. A `self` method, or passing the object by @@ -1116,12 +1121,16 @@ the library's own URI, mapped to the library by a build hook or by `--native-ass declares or a data enum's variant class, are kept apart as Kotlin's are. The glue's C names are `_dart_`, with a `_` more for each C name of the export already there. - **Objects** are final classes owning their handle. Of a checked type, `close()` drops - the value through Rust's own `_drop`, as Rust's `drop` would: a call after it throws a - `StateError` as for a value moved out, and a `close()` while a call borrows the value, - from a closure that call runs, throws one instead, as Rust refuses that drop. Of any - other type, `close()` frees the handle, and throws while a call holds it. A second - `close()` does nothing, and the handle itself is freed once the garbage collector finds - the object unreachable, through a finalizable handle of Dart's C API. Any call that meets + the value through Rust's own `_drop`, as Rust's `drop` would, and frees the handle in + the same crossing, and a `close()` while a call borrows the value, from a closure that + call runs, throws a `StateError` instead, as Rust refuses that drop. Of any other type, + `close()` frees the handle, and throws while a call holds it. A second `close()` does + nothing. A returned object waits in the isolate's nursery of the last 64 returned, where + `close()` frees it without the garbage collector's registration (a finalizable handle of + Dart's C API), which costs more than the rest of a return and close. One never closed is + registered once 64 later returns push it out, and freed once the garbage collector finds + it unreachable after; one still in the nursery when the isolate exits is freed then. Any + call that meets a closed object, as its receiver or anywhere in its arguments, throws a `StateError` before Rust sees it. A `self` method, or passing the object by value, moves its value out. `Clone` is `clone()`, and a `pub` @@ -1344,7 +1353,7 @@ just ffi shapes # round-trip every sampled type shape in every language but KN nix develop .#android --command just ffi android # the Kotlin test, shrunk by R8, on the Android device adb reaches nix develop .#bench --command just ffi bench # the binding benchmark: bench/ffi/README.md's cases from Rust, and through the export from C, C++, Kotlin, Python, Swift, Go, and Dart nix develop .#bench --command just ffi codspeed # after bench-build, each single-threaded case once under CodSpeed's CPU simulation, as the CodSpeed workflow runs it -nix develop .#sanitize --command just ffi sanitize # api's C and C++ scripts under ASan and TSan, and its C ABI under Miri +nix develop .#sanitize --command just ffi sanitize # api's C and C++ scripts under ASan and TSan, its Go under the race detector, and its C ABI under Miri ``` The `api` fixture covers each rule once, and re-exports the `dep` fixture's types, which @@ -1380,7 +1389,8 @@ It runs with nixpkgs' Swift on Linux, and nightly with Apple's on macOS. way, plus a context done before the call, a deadline, and a handle closed during a call, checks that bytes cross both ways, that a `Bytes` keeps Rust's buffer while only a sub-view or a reader reaches it and that the GC frees it once none does, that the GC frees an object -never closed, and that each kind of call meeting a closed object, including one racing +never closed once 64 later returns push it out of its nursery, that each object of many +goroutines' returns, closes, and drops is freed once, and that each kind of call meeting a closed object, including one racing `Close`, returns `rt.ErrClosed`, and breaks each runtime check, which must panic with its message; a runtime package from another build must fail its contract check. The `target` test exports a package for the host, for wasm32, and for both into one @@ -1404,8 +1414,8 @@ feature breaks the API twice for each kind of break, once in an item only wasm32 error matching `fixtures/lock/expected.txt`. `just ffi sanitize`, run nightly, rebuilds the `api` export with std under AddressSanitizer and then ThreadSanitizer, runs `tests/api.c` and `tests/api.cpp` against it, and matches -their output to the native transcript; it then drives the C ABI from `tests/miri/api.rs` -under Miri with strict provenance. +their output to the native transcript; it runs `tests/api.go` under Go's race detector; +and it drives the C ABI from `tests/miri/api.rs` under Miri with strict provenance. moq's crates export through `target/moq-ffi`, an empty library on each that `just ffi moq` writes in place of moq-ffi's remainder. The export serves the host and each target `just ffi moq-api` staged in `target/moq-api` from the same moq sources, and fails on a diff --git a/ffi/fixtures/api/src/lib.rs b/ffi/fixtures/api/src/lib.rs index bcfa58631..02ee53f1f 100644 --- a/ffi/fixtures/api/src/lib.rs +++ b/ffi/fixtures/api/src/lib.rs @@ -1806,6 +1806,27 @@ impl Logger { } } +/// A [`Logger`] its `&mut self` method makes checked, so a binding's close drops it through +/// Rust, running its implementation's close inside that drop. +pub struct Journal { + sink: Box, +} + +impl Journal { + pub fn new(sink: Box) -> Self { + Self { sink } + } + + /// Log to `sink` from now on, dropping the implementation before it. + pub fn redirect(&mut self, sink: Box) { + self.sink = sink; + } + + pub fn write(&self, line: &str) -> bool { + self.sink.log(0, line) + } +} + #[cfg(not(target_arch = "wasm32"))] /// A thread logging a line to a foreign implementation, which [`LogThread::join`] waits on. pub struct LogThread { diff --git a/ffi/justfile b/ffi/justfile index 6f514371a..b576fa26f 100644 --- a/ffi/justfile +++ b/ffi/justfile @@ -45,9 +45,9 @@ shapes: android: cargo test --locked --test fixtures -- --ignored --exact api_android -# Run the api fixture's C and C++ scripts under AddressSanitizer and ThreadSanitizer, and its glue's C ABI under Miri, in the sanitize shell: nix develop .#sanitize. +# Run the api fixture's C and C++ scripts under AddressSanitizer and ThreadSanitizer, its Go script under Go's race detector, and its glue's C ABI under Miri, in the sanitize shell: nix develop .#sanitize. sanitize: - cargo test --locked --test fixtures -- --ignored --exact api_asan api_tsan api_miri + cargo test --locked --test fixtures -- --ignored --exact api_asan api_tsan api_go_race api_miri # Apply formatters without changing program behavior. fix: diff --git a/ffi/runtime/dart.rs b/ffi/runtime/dart.rs index 6e2e0dd73..a5f0c9854 100644 --- a/ffi/runtime/dart.rs +++ b/ffi/runtime/dart.rs @@ -134,6 +134,39 @@ pub unsafe fn disown(own: *mut c_void, object: *mut c_void) { unsafe { (api().unfinalize)(own, object) } } +/// The slots of a nursery of `len` objects, which Dart fills: `len`, then each object's +/// handle, then the C function freeing it, the handle null while the slot is out. Once the +/// garbage collector finds `object`, the nursery, unreachable, as when its isolate exits, +/// each handle still in a slot is freed. +/// +/// # Safety +/// Called from Dart in its isolate, with `object` a handle of the Dart object, which writes +/// only handles the function beside each frees, and only while reachable. +pub unsafe fn nursery(object: *mut c_void, len: usize) -> *mut *mut c_void { + unsafe extern "C" fn finalize(_: *mut c_void, slots: *mut c_void) { + let slots = slots.cast::<*mut c_void>(); + // SAFETY: `nursery` made `slots`, whose first word is its length. + let len = unsafe { *slots } as usize; + // SAFETY: as above, nothing writes it once the nursery is unreachable. + let slots = unsafe { Box::from_raw(std::ptr::slice_from_raw_parts_mut(slots, 1 + 2 * len)) }; + for i in 0..len { + let raw = slots[1 + i]; + if !raw.is_null() { + // SAFETY: Dart's contract: each handle beside the function that frees it. + let free = unsafe { std::mem::transmute::<*mut c_void, unsafe extern "C" fn(*mut c_void)>(slots[1 + len + i]) }; + // SAFETY: as above, and this frees each at most once, as no Dart object owns it. + unsafe { free(raw) }; + } + } + } + let mut slots = vec![std::ptr::null_mut::(); 1 + 2 * len].into_boxed_slice(); + slots[0] = len as *mut c_void; + let slots = Box::into_raw(slots).cast::<*mut c_void>(); + // SAFETY: the caller's contract. + unsafe { own(object, slots.cast(), finalize) }; + slots +} + /// An isolate: its port, and the calls into it other threads wait on. One lives for each /// isolate that loads the library, and is never freed, since nothing tells Rust an /// isolate exited. diff --git a/ffi/runtime/rt.dart b/ffi/runtime/rt.dart index 0fa64229b..8d6ac84df 100644 --- a/ffi/runtime/rt.dart +++ b/ffi/runtime/rt.dart @@ -782,57 +782,141 @@ final class _Callback extends ffi.Struct { external ffi.Pointer data; } -/// What an object owns: its handle, or one Rust lends for a call, which Rust frees. An owned -/// one of a value Rust checks is Rust's alone to account: [close] drops the value as Rust's -/// own drop does, failing while a call borrows it, a call after it throws as Rust's check -/// of a moved value does, and the handle is freed once the garbage collector finds this -/// unreachable. Any other counts its calls: [close] frees it then, throwing while a call -/// holds it, and a call after it throws. +/// What an object owns: its handle, or one Rust lends for a call, which Rust frees. Each +/// call holds it, so the handle is freed only once no call does. An owned one of a value +/// Rust checks is Rust's to account: [close] drops the value as Rust's own drop does, +/// failing while a call borrows it, and frees the handle with it unless a call holds it. +/// Any other: [close] frees it then, throwing while a call holds it. A call after either +/// throws. One returned waits in the isolate's [_Nursery], where a close frees it without +/// the garbage collector, which frees it only once pushed out unclosed. abstract base class _Object implements ffi.Finalizable { final ffi.Pointer _raw; - /// The owner's hold until closed, and one for each call, of a counted one. + /// The owner's hold until closed, and one for each call. int _holds = 1; bool _closed = false; final bool _owned; - /// Whether it counts its calls, rather than Rust. - final bool _counted; + /// Whether Rust checks its borrows, so its close drops the value through Rust. + final bool _checked; - /// The garbage collector's hold of an owned one's handle, until freed. + /// The garbage collector's hold of an owned one's handle, from when the nursery pushed it + /// out until freed. ffi.Pointer _own = ffi.nullptr; - _Object(this._raw, bool owned, bool checked) : _owned = owned, _counted = !(owned && checked); + /// Its slot in the nursery while there, else -1. + int _slot = -1; + + _Object(this._raw, bool owned, bool checked) : _owned = owned, _checked = owned && checked; String get _type; - /// Free the handle and detach its finalizer. - void _free(); + /// Free an owned one's handle, and take back the garbage collector's hold of it. + void _free() {} + + /// Give the garbage collector a hold of the handle, as the nursery pushes it out open. + void _register() {} - /// Drop the value of one Rust checks, throwing [StateError] if a call borrows it. - void _drop() {} + /// Drop the value of one Rust checks, then, if `free`, free its handle and take back the + /// garbage collector's hold of it, in one crossing: throwing [StateError], changing + /// nothing, if a call borrows it. + void _close(bool free) {} ffi.Pointer _hold() { - if (!_counted) return _raw; if (_closed) throw StateError('$_type is closed'); _holds++; return _raw; } void _release() { - if (_counted && --_holds == 0 && _owned) _free(); + if (--_holds == 0 && _owned) { + _leave(); + _free(); + } + } + + /// Take it out of the nursery, returning whether it was there: before its free, which + /// may run Dart that returns objects into the nursery meanwhile. + bool _leave() { + final slot = _slot; + if (slot < 0) return false; + _slot = -1; + _n.nursery.leave(slot); + return true; } /// Close it as [_Object] says; again, it does nothing. void close() { - if (!_counted) return _drop(); if (_closed) return; + if (_checked) { + // Closed before Rust drops the value, which may run Dart that reaches it meanwhile. + _closed = true; + // A call holding it, from a callback or one waiting on a worker, may not have + // borrowed the value in Rust yet: the handle outlives its hold, so Rust's check + // refuses it. + final alone = _holds == 1; + final nursed = alone && _leave(); + try { + _close(alone); + } catch (_) { + _closed = false; + // Still open, and out of the nursery, so the garbage collector's. + if (nursed) _register(); + rethrow; + } + if (alone) { + _holds = 0; + } else { + _release(); + } + return; + } if (_holds != 1) throw StateError('$_type: a handle is already borrowed'); _closed = true; _release(); } } +/// How many of the latest objects returned the nursery holds: one dropped unclosed is freed +/// only once this many more were returned, so it bounds what a dropped object keeps of +/// Rust's. +const _nurseryLen = 64; + +/// The latest objects returned, oldest at [_next] once full: a close takes one out without +/// the garbage collector's hold of its handle, which costs more than the rest of a return +/// and close together, and one pushed out still open takes that hold then. Rust keeps a copy +/// of each slot's handle and the C function freeing it, and frees what is still in them once +/// the garbage collector finds the nursery unreachable, as when the isolate exits. +final class _Nursery { + final List<_Object?> _held = List<_Object?>.filled(_nurseryLen, null); + int _next = 0; + + /// What Rust keeps: the length, then each slot's handle, then each one's free function, + /// null while the slot is out. + late final ffi.Pointer> _slots = _dartNursery(this, _nurseryLen); + + /// Hold `object`, owned, whose handle `free` frees. + void push(_Object object, ffi.Pointer free) { + final i = _next; + _next = i + 1 == _nurseryLen ? 0 : i + 1; + final old = _held[i]; + if (old != null) { + old._slot = -1; + old._register(); + } + _held[i] = object; + object._slot = i; + final slots = _slots; + slots[1 + i] = object._raw; + slots[1 + _nurseryLen + i] = free; + } + + void leave(int slot) { + _held[slot] = null; + _slots[1 + slot] = ffi.nullptr; + } +} + /// Native memory each call lowers its values into, as a stack: a call takes what it /// needs from the top and gives it back once it returns, so a call Rust makes into Dart /// meanwhile takes and gives back above it. @@ -1057,6 +1141,9 @@ final class _Native { /// queued meanwhile, and the call reads it before any other can run, and a failure /// clears it. late final ffi.Pointer status = _dartAlloc(4).cast()..value = 0; + + /// The latest objects returned. + late final _Nursery nursery = _Nursery(); late final isolate.RawReceivePort _port; late final ffi.Pointer home; diff --git a/ffi/runtime/rt.go b/ffi/runtime/rt.go index 1448cd7ee..0b8e148ac 100644 --- a/ffi/runtime/rt.go +++ b/ffi/runtime/rt.go @@ -101,22 +101,130 @@ type Handle_ struct { state atomic.Int64 // Nil for a handle Rust lends, which it frees itself. free func(unsafe.Pointer) - // Non-nil for an owned one of a value Rust checks. - drop func(unsafe.Pointer) error + // Non-nil for an owned one of a value Rust checks: it drops the value, then frees the + // handle too if asked and the drop passed. + drop func(raw unsafe.Pointer, free bool) error name string - // What frees an owned handle once the GC collects its object. - Cleanup runtime.Cleanup + // The nursery an owned handle was returned into, and its slot there. + nursery *nursery_ + slot int + // What frees an owned handle once the GC collects its object, from when it leaves its + // nursery still open. + cleanup runtime.Cleanup } // Init makes h a handle of raw, which free frees, or nil for one Rust lends, and drop // drops the value of, or nil for one Rust does not check. The generated packages allocate // it with the object owning it, which each call holding it then keeps reachable: so once // the GC collects the object, no call holds the handle, and its cleanup frees it at once. -func (h *Handle_) Init(raw unsafe.Pointer, free func(unsafe.Pointer), drop func(unsafe.Pointer) error, name string) { +func (h *Handle_) Init(raw unsafe.Pointer, free func(unsafe.Pointer), drop func(unsafe.Pointer, bool) error, name string) { h.raw, h.free, h.drop, h.name = raw, free, drop, name h.state.Store(held_) } +// Own puts h, which Rust returned and its object owns, in a nursery of the latest +// returned, where a Close frees it without the GC's registration, which costs more than +// the rest of a return and close together. One that leaves it open, pushed out by later +// returns or with its nursery once the pool drops that, is registered then. For the +// generated packages. +func (h *Handle_) Own() { + r, _ := nurseries_.Get().(*ring_) + if r == nil { + r = &ring_{n: new(nursery_)} + runtime.AddCleanup(r, (*nursery_).flush, r.n) + } + r.n.push(h) + nurseries_.Put(r) +} + +// nurseries_ holds the ring each P returns its objects into; a GC drops an idle one. +var nurseries_ sync.Pool + +// ring_ is what the pool holds of a nursery, which its handles point at: so its cleanup +// runs once the pool dropped it, whatever its handles do. +type ring_ struct { + n *nursery_ +} + +// nurseryLen_ is how many of the latest objects returned a nursery holds: an object +// dropped unclosed is freed only once this many more were returned on its P, or the pool +// dropped its nursery, so it bounds what a dropped object keeps of Rust's. +const nurseryLen_ = 64 + +// nursery_ is the latest handles returned, oldest at next once full. Whichever comes first +// of a handle's Close and its eviction takes its slot, with a compare-and-swap. +type nursery_ struct { + // Kept reachable while here, so an eviction can register them. Only the ring's holder + // touches these two. + held [nurseryLen_]*Handle_ + next int + // Each slot's raw handle until closed or evicted, evicting_ while an eviction registers + // it, else nil. + raw [nurseryLen_]unsafe.Pointer +} + +// evicting_ marks a slot whose eviction is registering its handle, which no raw handle is. +var evicting_ = unsafe.Pointer(new(byte)) + +func (n *nursery_) push(h *Handle_) { + i := n.next + n.next = (i + 1) % nurseryLen_ + if old := n.held[i]; old != nil { + n.evict(i, old) + } + n.held[i] = h + h.nursery, h.slot = n, i + atomic.StorePointer(&n.raw[i], h.raw) +} + +// evict registers h, in slot i, with the GC, unless a Close took the slot first. A Close +// meanwhile waits until it is registered, so it stops the cleanup. +func (n *nursery_) evict(i int, h *Handle_) { + if !atomic.CompareAndSwapPointer(&n.raw[i], h.raw, evicting_) { + return + } + h.cleanup = runtime.AddCleanup(h, h.free, h.raw) + atomic.StorePointer(&n.raw[i], nil) +} + +// flush registers each handle still open in n, once the pool dropped its ring, and lets go +// of them all, so none waits on the others to be collected while any keeps n. +func (n *nursery_) flush() { + for i := range n.raw { + // Loaded first, as push stores it after the handle. + if raw := atomic.LoadPointer(&n.raw[i]); raw != nil && raw != evicting_ { + n.evict(i, n.held[i]) + } + n.held[i] = nil + } +} + +// leave takes h out of its nursery as it closes, before anything frees it, or else stops +// the cleanup its eviction registered: once, as only a close claims h. +func (h *Handle_) leave() { + if n := h.nursery; n != nil { + slot := &n.raw[h.slot] + for { + raw := atomic.LoadPointer(slot) + if raw == h.raw && atomic.CompareAndSwapPointer(slot, raw, nil) { + return + } + if raw != evicting_ && raw != h.raw { + break + } + runtime.Gosched() + } + } + h.cleanup.Stop() +} + +// own registers h with the GC, as an eviction does, once a close that took it out of its +// nursery failed, leaving it open. +func (h *Handle_) own() { + h.nursery = nil + h.cleanup = runtime.AddCleanup(h, h.free, h.raw) +} + // Hold is the handle, held until Release takes the Slot_, or nil once closed. func (h *Handle_) Hold() (unsafe.Pointer, Slot_) { if raw := h.Count(); raw != nil { @@ -219,11 +327,25 @@ func (h *Handle_) Unlist(s Slot_) { // Close closes h as Handle_ says; again, it does nothing. func (h *Handle_) Close() error { if h.drop != nil { + // Held by no call, as soon after its return: claimed closed so none starts, then the + // value dropped and the handle freed in one crossing. A call meanwhile fails as + // closed, as one racing any Close may, and a Drop that calls back into Go finds it + // closed rather than waiting on itself. + if h.state.CompareAndSwap(held_, closed_|closing_) { + h.leave() + if err := h.drop(h.raw, true); err != nil { + h.own() + h.state.Store(held_) + return err + } + h.state.Store(closed_) + return nil + } raw, s := h.Hold() if raw == nil { return nil } - err := h.drop(raw) + err := h.drop(raw, false) h.Release(s) if err != nil { return err @@ -256,8 +378,7 @@ func (h *Handle_) Close() error { return checkError_{h.name, "a handle is already borrowed", 3} } h.state.Store(closed_) - // A cleanup left registered until the GC slows each one registered after it. - h.Cleanup.Stop() + h.leave() if h.free != nil { h.free(h.raw) } @@ -281,8 +402,6 @@ func (h *Handle_) listed() bool { // Moved closes an owned handle a call moved the value out of, or Close dropped, freeing it // once no call holds it. func (h *Handle_) Moved() { - // A cleanup left registered until the GC slows each one registered after it. - h.Cleanup.Stop() h.abandon() } @@ -302,6 +421,7 @@ func (h *Handle_) abandon() { break } } + h.leave() // Claimed closed, so each call holding it in the table did so before: count them. if h.state.Load()&bias_ != 0 { key := h.key() diff --git a/ffi/src/dart.rs b/ffi/src/dart.rs index d4ab02849..f31479943 100644 --- a/ffi/src/dart.rs +++ b/ffi/src/dart.rs @@ -230,6 +230,13 @@ pub fn generate(abi: &Abi<'_>, packages: &str, contract: u64) -> Result p0", "void", ), + ( + "dart_nursery", + "ffi.Handle, ffi.Size", + "ffi.Pointer>", + "Object p0, int p1", + "ffi.Pointer>", + ), ("dart_abort", "_Buf", "ffi.Void", "_Buf p0", "void"), ( "status_message", @@ -310,6 +317,7 @@ pub fn generate(abi: &Abi<'_>, packages: &str, contract: u64) -> Result *mut c_void {{\n\tlet ptr = unsafe {{ malloc(len) }};\n\tif ptr.is_null() {{\n\t\tstd::alloc::handle_alloc_error(std::alloc::Layout::array::(len).expect(\"a length\"));\n\t}}\n\tptr\n}}\n\n\ #[unsafe(no_mangle)]\npub extern \"C\" fn {space}_buf(len: usize) -> rt::Buf {{\n\trt::Buf::vec(vec![0; len])\n}}\n\n\ #[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {space}_free(ptr: *mut c_void) {{\n\tunsafe {{ free(ptr) }}\n}}\n\n\ + #[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {space}_nursery(object: *mut c_void, len: usize) -> *mut *mut c_void {{\n\tunsafe {{ runtime::nursery(object, len) }}\n}}\n\n\ #[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {space}_abort(text: rt::Buf) {{\n\trt::fail(\"dart\", unsafe {{ text.str(\"{space}_abort\") }})\n}}\n\n\ #[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {space}_reader_read(data: *mut c_void, buf: *mut u8, len: usize, n: *mut usize, err: *mut rt::Buf) -> bool {{\n\tunsafe {{ runtime::reader_read(data, buf, len, n, err) }}\n}}\n\n\ #[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {space}_reader_seek(data: *mut c_void, whence: std::ffi::c_int, offset: i64, position: *mut u64, err: *mut rt::Buf) -> bool {{\n\tunsafe {{ runtime::reader_seek(data, whence, offset, position, err) }}\n}}\n\n{}", @@ -851,18 +859,18 @@ impl<'a> Gen<'a> { } /// The Dart functions owning a handle of the object `c`, whose C type is `handle`: one - /// giving the garbage collector a Dart object owning it to free it with, and one taking - /// that back and freeing it, serving the isolate meanwhile. - fn owned(&mut self, c: &str, handle: &Ty) -> (String, String) { + /// giving the garbage collector a Dart object owning it to free it with, the C function + /// freeing it, which the nursery keeps for the isolate's exit, the one freeing it, and of + /// a checked value, the one dropping it then freeing it if asked; the last two serve the + /// isolate meanwhile and take back the garbage collector's hold if it took one. + fn owned(&mut self, c: &str, handle: &Ty, checked: bool) -> Owned { let bare = self.bare(c); let space = &self.space; let own = format!("{space}_own_{bare}"); - let free = format!("{space}_free_{bare}"); let raw = rs(handle); let _ = write!( self.glue, - "#[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {own}(object: *mut c_void, raw: *mut c_void) -> *mut c_void {{\n\t// The garbage collector's free runs with no isolate to serve.\n\tunsafe extern \"C\" fn finalize(_: *mut c_void, raw: *mut c_void) {{\n\t\tunsafe {{ crate::{c}_free(raw.cast()) }}\n\t}}\n\tunsafe {{ runtime::own(object, raw, finalize) }}\n}}\n\n\ - #[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {free}(home: *const c_void, own: *mut c_void, object: *mut c_void, raw: {raw}) {{\n\tunsafe {{ runtime::disown(own, object) }};\n\tlet home = unsafe {{ &*home.cast::() }};\n\tunsafe {{ home.serve(move || crate::{c}_free(raw)) }}\n}}\n\n" + "#[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {own}(object: *mut c_void, raw: *mut c_void) -> *mut c_void {{\n\t// The garbage collector's free runs with no isolate to serve.\n\tunsafe extern \"C\" fn finalize(_: *mut c_void, raw: *mut c_void) {{\n\t\tunsafe {{ crate::{c}_free(raw.cast()) }}\n\t}}\n\tunsafe {{ runtime::own(object, raw, finalize) }}\n}}\n\n" ); self.natives.insert( own.clone(), @@ -870,13 +878,46 @@ impl<'a> Gen<'a> { "@ffi.Native Function(ffi.Handle, ffi.Pointer)>(symbol: '{own}')\nexternal ffi.Pointer _{own}(Object p0, ffi.Pointer p1);\n\n" ), ); + let freeing = format!("{space}_freeing_{bare}"); + self.natives.insert( + freeing.clone(), + format!( + "@ffi.Native)>(symbol: '{c}_free')\nexternal void _{freeing}(ffi.Pointer p0);\n\n" + ), + ); + let free = format!("{space}_free_{bare}"); + let _ = write!( + self.glue, + "#[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {free}(home: *const c_void, own: *mut c_void, object: *mut c_void, raw: {raw}) {{\n\tif !own.is_null() {{\n\t\tunsafe {{ runtime::disown(own, object) }};\n\t}}\n\tlet home = unsafe {{ &*home.cast::() }};\n\tunsafe {{ home.serve(move || crate::{c}_free(raw)) }}\n}}\n\n" + ); self.natives.insert( free.clone(), format!( "@ffi.Native, ffi.Pointer, ffi.Handle, ffi.Pointer)>(symbol: '{free}')\nexternal void _{free}(ffi.Pointer p0, ffi.Pointer p1, Object p2, ffi.Pointer p3);\n\n" ), ); - (format!("_{own}"), format!("_{free}")) + let close = checked.then(|| { + let close = format!("{space}_close_{bare}"); + // The status stays the call's own while the isolate serves others meanwhile, as a + // served call's does. + let _ = write!( + self.glue, + "#[unsafe(no_mangle)]\npub unsafe extern \"C\" fn {close}(home: *const c_void, own: *mut c_void, object: *mut c_void, raw: {raw}, free: bool, out: *mut i32) {{\n\tlet home = unsafe {{ &*home.cast::() }};\n\tlet status = unsafe {{\n\t\thome.serve(move || {{\n\t\t\tlet mut status = 0;\n\t\t\tcrate::{c}_drop(raw, &raw mut status);\n\t\t\tif status == 0 && free {{\n\t\t\t\tcrate::{c}_free(raw);\n\t\t\t}}\n\t\t\tstatus\n\t\t}})\n\t}};\n\tif status != 0 {{\n\t\tunsafe {{ *out = status }};\n\t}} else if free && !own.is_null() {{\n\t\tunsafe {{ runtime::disown(own, object) }};\n\t}}\n}}\n\n" + ); + self.natives.insert( + close.clone(), + format!( + "@ffi.Native, ffi.Pointer, ffi.Handle, ffi.Pointer, ffi.Bool, ffi.Pointer)>(symbol: '{close}')\nexternal void _{close}(ffi.Pointer p0, ffi.Pointer p1, Object p2, ffi.Pointer p3, bool p4, ffi.Pointer p5);\n\n" + ), + ); + format!("_{close}") + }); + Owned { + own: format!("_{own}"), + freeing: format!("_{freeing}"), + free: format!("_{free}"), + close, + } } /// [`Self::served`] of a C function by name, parameters, and return. @@ -1762,19 +1803,24 @@ impl<'a> Gen<'a> { name: c.clone(), constant: false, }; - let (own, free) = self.owned(&c, &owner); let mut members = vec!["close".to_string()]; let checked = abi.checked[i]; + let Owned { + own, + freeing, + free, + close, + } = self.owned(&c, &owner, checked); + let close = match close { + Some(close) => format!( + "\n\t@override\n\tvoid _close(bool free) {{\n\t\tfinal s_ = _n.status;\n\t\t{close}(_n.home, _own, this, _raw, free, s_);\n\t\t_status(s_.value, '{c}_drop');\n\t}}\n" + ), + None => String::new(), + }; + let close = format!("\t@override\n\tvoid _free() => {free}(_n.home, _own, this, _raw);\n{close}"); let mut body = vec![format!( - "\t{name}._(ffi.Pointer raw) : super(raw, true, {checked}) {{\n\t\t_own = {own}(this, raw);\n\t}}\n\n\t{name}._lent(ffi.Pointer raw) : super(raw, false, {checked});\n\n\t@override\n\tString get _type => '{name}';\n\n\t@override\n\tvoid _free() => {free}(_n.home, _own, this, _raw);\n" + "\t{name}._(ffi.Pointer raw) : super(raw, true, {checked}) {{\n\t\t_n.nursery.push(this, ffi.Native.addressOf)>>({freeing}).cast());\n\t}}\n\n\t{name}._lent(ffi.Pointer raw) : super(raw, false, {checked});\n\n\t@override\n\tString get _type => '{name}';\n\n\t@override\n\tvoid _register() => _own = {own}(this, _raw);\n\n{close}" )]; - if checked { - let what = format!("{c}_drop"); - let f = self.served_fn(&what, vec![owner.clone(), Ty::Out(Box::new(Ty::Status))], Ty::Void); - body.push(format!( - "\t@override\n\tvoid _drop() {{\n\t\tfinal s_ = _n.status;\n\t\t{f}(_n.home, _raw, s_);\n\t\t_status(s_.value, '{what}');\n\t}}\n" - )); - } self.traits(&ty, &c, object.traits, &mut body, &mut members); self.methods(&object.methods, &owner, &ty, &mut body, &mut members); self.distinct(&rust, &members); @@ -2572,6 +2618,18 @@ impl<'a> Gen<'a> { } } +/// The Dart names of what owns an object's handle, as [`Gen::owned`] makes them. +struct Owned { + /// Gives the garbage collector a hold of it. + own: String, + /// The C function freeing it. + freeing: String, + /// Frees it. + free: String, + /// Of a checked value, drops it, then frees it if asked. + close: Option, +} + /// How a function throws its error. enum Thrown { Itself, diff --git a/ffi/src/go.rs b/ffi/src/go.rs index 878dbceff..80834b1d1 100644 --- a/ffi/src/go.rs +++ b/ffi/src/go.rs @@ -620,6 +620,9 @@ impl<'a> Gen<'a> { extern bool {go}_read(void *data, uint8_t *buf, size_t len, size_t *n, {p}_buf *err);\n\ extern bool {go}_seek(void *data, int whence, int64_t offset, uint64_t *position, {p}_buf *err);\n\n\ static void {go}_bufs_free({p}_buf *bufs, size_t n) {{\n\tfor (size_t i = 0; i < n; i++) {{\n\t\t{p}_buf_free(bufs[i]);\n\t}}\n}}\n\n\ + // The header's contract, read through C: cgo built by clang reads a uint64_t macro\n\ + // with its top bit set as a negative constant.\n\ + static uint64_t {go}_header_contract(void) {{\n\treturn {upper}_CONTRACT;\n}}\n\n\ static void {go}_waker_wake({p}_waker w) {{\n\tif (w.wake != NULL) {{\n\t\tw.wake(w.data);\n\t}}\n}}\n\n\ static void {go}_waker_drop({p}_waker w) {{\n\tif (w.drop != NULL) {{\n\t\tw.drop(w.data);\n\t}}\n}}\n\ */\nimport \"C\"\n\n\ @@ -637,7 +640,7 @@ impl<'a> Gen<'a> { func cWakerDrop(w cWaker) {{\n\tC.{go}_waker_drop(w)\n}}\n\n\ // Go packages and a library from different builds must not mix: checked when the\n\ // runtime loads, before any other call.\n\ - func init() {{\n\tif C.{p}_contract() != 0x{contract:016x} || C.{upper}_CONTRACT != 0x{contract:016x} {{\n\t\tpanic(\"the Go of {p} and the library are from different builds\")\n\t}}\n\ttarget_ = uint64(C.{p}_target())\n}}\n\n\ + func init() {{\n\tif C.{p}_contract() != 0x{contract:016x} || C.{go}_header_contract() != 0x{contract:016x} {{\n\t\tpanic(\"the Go of {p} and the library are from different builds\")\n\t}}\n\ttarget_ = uint64(C.{p}_target())\n}}\n\n\ // target_ is the index of the library's target among those the packages serve.\n\ var target_ uint64\n\n\ // On_ panics unless the library's target is among those of on, as the item what is.\n\ @@ -831,6 +834,22 @@ impl<'a> Gen<'a> { (stmts, value) } + /// The C shim dropping the value of the object `c`'s handle, which Rust checks, then + /// freeing the handle if asked and the drop passed: a close in one crossing. + fn close_c(&mut self, c: &str) -> String { + let name = format!("{}GoClose_{}", self.abi.prefix, self.bare(c)); + let handle = Ty::Handle { + name: c.to_string(), + constant: false, + }; + let decl = crate::c::decl(self.abi, &handle, "self"); + let status = crate::c::decl(self.abi, &Ty::Status, ""); + self.decls.push(format!( + "static inline {status} {name}({decl}, int free) {{\n\t{status} r = 0;\n\t{c}_drop(self, &r);\n\tif (r == 0 && free) {{\n\t\t{c}_free(self);\n\t}}\n\treturn r;\n}}\n" + )); + name + } + /// The call of `sig` with `args`, as [`Self::call`] makes it. fn call_sig(&mut self, sig: &abi::Sig, args: &[String]) -> (String, String) { self.call(&sig.name, &sig.params, &sig.ret, args) @@ -2252,33 +2271,24 @@ impl<'a> Gen<'a> { let name = self.qualified(&ty); let handle = self.qualify(Packages::RT, "Handle_"); let (drop, dropping) = match self.abi.checked[i] { - true => { - let handle = Ty::Handle { - name: c.clone(), - constant: false, - }; - let params = [ - ("self".to_string(), handle), - ("status".to_string(), Ty::Out(Box::new(Ty::Status))), - ]; - let (call, _) = self.call(&format!("{c}_drop"), ¶ms, &Ty::Void, &[format!("(*C.{c})(p)")]); - let status_ = self.qualify(Packages::RT, "Status_"); - ( - format!( - "// drop_{c} drops the value of the handle p, which Rust checks.\n\ - func drop_{c}(p unsafe.Pointer) error {{\n\t{call}\n\treturn {status_}(int32(c_status_), \"{c}_drop\")\n}}\n\n" - ), - format!("drop_{c}"), - ) - } + true => ( + format!( + "// drop_{c} drops the value of the handle p, which Rust checks, then frees p if free\n\ + // and the drop passed.\n\ + func drop_{c}(p unsafe.Pointer, free bool) error {{\n\tvar f C.int\n\tif free {{\n\t\tf = 1\n\t}}\n\treturn {status_}(int32(C.{shim}((*C.{c})(p), f)), \"{c}_drop\")\n}}\n\n", + shim = self.close_c(&c), + status_ = self.qualify(Packages::RT, "Status_"), + ), + format!("drop_{c}"), + ), false => (String::new(), "nil".to_string()), }; format!( "func free_{c}(p unsafe.Pointer) {{\n\tC.{c}_free((*C.{c})(p))\n}}\n\n\ {drop}\ // own_{c} owns a handle Rust returned, or is nil for none: the object and its handle\n\ - // in one allocation, whose cleanup frees the handle.\n\ - func own_{c}(raw *C.{c}) *{name} {{\n\tif raw == nil {{\n\t\treturn nil\n\t}}\n\to := &struct {{\n\t\tv {name}\n\t\th {handle}\n\t}}{{}}\n\to.h.Init(unsafe.Pointer(raw), free_{c}, {dropping}, \"{local}\")\n\to.v.Handle_ = &o.h\n\to.h.Cleanup = runtime.AddCleanup(&o.v, free_{c}, unsafe.Pointer(raw))\n\treturn &o.v\n}}\n\n\ + // in one allocation, in the nursery Own puts it in.\n\ + func own_{c}(raw *C.{c}) *{name} {{\n\tif raw == nil {{\n\t\treturn nil\n\t}}\n\to := &struct {{\n\t\tv {name}\n\t\th {handle}\n\t}}{{}}\n\to.h.Init(unsafe.Pointer(raw), free_{c}, {dropping}, \"{local}\")\n\to.v.Handle_ = &o.h\n\to.h.Own()\n\treturn &o.v\n}}\n\n\ // lend_{c} is a handle Rust lends for a call, which it frees itself.\n\ func lend_{c}(raw *C.{c}) *{name} {{\n\to := &struct {{\n\t\tv {name}\n\t\th {handle}\n\t}}{{}}\n\to.h.Init(unsafe.Pointer(raw), nil, nil, \"{local}\")\n\to.v.Handle_ = &o.h\n\treturn &o.v\n}}\n\n\ // holdOption_{c} is v's handle, held until the call returns, or nil for none.\n\ diff --git a/ffi/tests/api.dart b/ffi/tests/api.dart index d70e884c9..f41e8769f 100644 --- a/ffi/tests/api.dart +++ b/ffi/tests/api.dart @@ -6,9 +6,11 @@ // contract, which must fail. import 'dart:async'; +import 'dart:collection'; import 'dart:convert'; import 'dart:ffi' as ffi; import 'dart:io'; +import 'dart:isolate'; import 'dart:typed_data'; import 'package:api/api.dart' as api; @@ -1004,6 +1006,41 @@ final class CountLog extends api.Log { void close() => closes++; } +/// A [CountLog] whose close, which Rust's drop of it runs, then runs `closed`. +final class ClosingLog extends CountLog { + final void Function() closed; + + ClosingLog(this.closed); + + @override + void close() { + super.close(); + closed(); + } +} + +/// A list of objects whose length, read as a call lowers it, first runs `read`. +final class ReadingList extends ListBase { + final void Function() read; + + ReadingList(this.read); + + @override + int get length { + read(); + return 0; + } + + @override + set length(int n) => throw UnsupportedError('a fixed list'); + + @override + api.Counter operator [](int i) => throw RangeError.index(i, this); + + @override + void operator []=(int i, api.Counter value) => throw UnsupportedError('a fixed list'); +} + /// The futures' semantics beyond the transcript, each checked by an assertion. Future futures() async { final channel = api.Channel(); @@ -1254,17 +1291,41 @@ Future owned() async { check(api.dropped() == before + 1, 'closing again frees nothing'); closed(() => token.idGet, 'Token', 'a closed token refuses a read'); + // Freed only once later returns push it out of the nursery. unclosed(); + for (var i = 0; i < 3; i++) { + pressure(); + } + check(api.dropped() == before + 1, 'the nursery keeps what nobody closed'); + fill(); await collect(() => api.dropped() >= before + 2, 'the garbage collector frees what nobody closed'); for (var i = 0; i < 3; i++) { pressure(); } check(api.dropped() == before + 2, 'the garbage collector frees each once'); + // Closed once pushed out of the nursery, which gave the garbage collector its hold: the + // close takes it back, so the collector frees nothing more. + final kept = api.Token(4); + fill(); + kept.close(); + check(api.dropped() == before + 3, 'closing an object pushed out of the nursery frees it at once'); + for (var i = 0; i < 3; i++) { + pressure(); + } + await step(); + check(api.dropped() == before + 3, 'the garbage collector frees nothing a close freed'); + + // An isolate's nursery frees what it still holds once the isolate exits. + final exited = api.dropped(); + await Isolate.run(unclosed); + await collect(() => api.dropped() > exited, "an isolate's exit frees what its nursery held"); + // An implementation an object the garbage collector frees held is closed from the event // loop, as no drop in the collector may call into Dart. final log = CountLog(); unclosedLogger(log); + fill(); await collect(() => log.closes == 1, 'the garbage collector frees what an object it frees holds'); targets(); print('owned ok'); @@ -1296,6 +1357,13 @@ void targets() { lacks(() => booth == api.Booth(mark: booth.mark), 'api_Booth_eq'); } +/// Return a nursery's worth of objects, each closed, pushing out every earlier one. +void fill() { + for (var i = 0; i < 64; i++) { + api.Tally().close(); + } +} + @pragma('vm:never-inline') void unclosed() { api.Token(3); @@ -1345,6 +1413,12 @@ void closedObjects() { throws(() => api.logCounts(spared, 256, open, []), 'an integer out of range after an implementation'); closed(() => api.logCounts(spared, 0, c, []), 'Counter', 'a closed object after an implementation'); closed(() => api.logCounts(spared, 0, open, [c]), 'Counter', 'a closed object in a list after an implementation'); + check(spared.closes == 0, 'a call Dart refuses gives Rust no implementation'); + // Moved out by a call, which only Rust's check sees. + final gone = api.Counter('g', api.U64(0)); + gone.intoLabel(null); + closed(() => api.logCounts(spared, 0, gone, []), 'Counter', 'a moved object after an implementation'); + closed(() => api.logCounts(spared, 0, open, [gone]), 'Counter', 'a moved object in a list after an implementation'); check(spared.closes == 2, 'a call whose check throws drops the implementation it gave Rust'); check(api.onFirstThread(), "a call after one that threw runs on the isolate's thread"); lacks(() => api.wasmLog(spared), 'api_wasm_log'); @@ -1352,6 +1426,32 @@ void closedObjects() { check(api.logCounts(spared, 0, open, []) == 1, 'an implementation a call that threw gave none of'); check(spared.lines == 1 && spared.closes == 3, 'an implementation given once is closed once'); + // Closed by Dart a call runs as it lowers its arguments, after it held the object: the + // handle outlives the call's hold, so Rust's check refuses the value the close dropped. + final early = api.Counter('early', api.U64(1)); + closed( + () => api.worth([early], ReadingList(early.close)), + 'Counter', + 'an object closed while its call lowers the rest', + ); + closed(early.get, 'Counter', 'an object closed during a call, after it'); + + // Closed again by Dart its drop runs, which finds it closed, so it is freed once. + late api.Journal journal; + var during = ''; + final nested = ClosingLog(() { + try { + journal.write('x'); + } on StateError catch (e) { + during = e.message; + } + journal.close(); + }); + journal = api.Journal(nested); + journal.close(); + check(nested.closes == 1, 'a close from inside its own drop closes nothing more'); + check(during == 'Journal is closed', 'an object is closed in Dart while Rust drops it: $during'); + // An async call. final channel = api.Channel(); channel.close(); @@ -1419,6 +1519,7 @@ Future lent() async { pressure(); } canary.clear(); + fill(); await collect(() => api.dropped() > before, 'the garbage collector frees an old object'); check(api.dropped() == before + 1, "a view keeps Rust's buffer"); check(view.length == 10 && view.every((b) => b == 2), "a view reads Rust's buffer after its list is gone"); @@ -1446,6 +1547,7 @@ Future forwarded() async { pressure(); } canary.clear(); + fill(); await collect(() => api.dropped() > before, 'the garbage collector frees an old object'); // Collections enough to free every list the relay left, had one held a copy. for (var i = 0; i < 3; i++) { diff --git a/ffi/tests/api.go b/ffi/tests/api.go index 7a355f8a6..5a7a3ac7c 100644 --- a/ffi/tests/api.go +++ b/ffi/tests/api.go @@ -1508,9 +1508,83 @@ func owned() { forward() collected(before+1, "a forwarded payload drops once nothing reaches it") uncollected(before+1, "a forwarded payload drops once") + + nursed() fmt.Println("owned ok") } +// nursed checks the nursery of the latest objects returned, which a Close leaves at once +// and the GC frees only once later returns push one out unclosed: each object is freed +// once, however Close and the nursery race. +func nursed() { + // One P, so every return below shares one nursery, which each GC finds in use, so its + // pool never drops it: only the push out of it registers the dropped token. The race + // detector's pool drops some at random, which frees it as a dropped nursery does. + procs := runtime.GOMAXPROCS(1) + before := api.Dropped() + unclosed() + deadline := time.Now().Add(10 * time.Second) + for api.Dropped() < before+1 { + check(time.Now().Before(deadline), "the GC frees an object pushed out of its nursery unclosed") + fill() + runtime.GC() + runtime.Gosched() + } + + // Closed after the push out registered it, so its cleanup is stopped. + kept := api.NewToken(4) + fill() + kept.Close() + check(api.Dropped() == before+2, "closing an object pushed out of its nursery frees it at once") + runtime.GOMAXPROCS(procs) + uncollected(before+2, "the GC frees nothing a Close freed after a push out") + + // Returned on many goroutines, each closing some of its own and of another's, as + // others' returns push them out, and dropping the rest. + const workers, each = 8, 1000 + handoff := make(chan *api.Token, workers*each) + var wg sync.WaitGroup + for w := range workers { + wg.Add(1) + go func() { + defer wg.Done() + for i := range each { + t := api.NewToken(uint32(i)) + switch i % 3 { + case 0: + t.Close() + case 1: + handoff <- t + } + if i%3 != 0 { + select { + case other := <-handoff: + other.Close() + default: + } + } + if w == 0 && i%100 == 0 { + runtime.GC() + } + } + }() + } + wg.Wait() + close(handoff) + for t := range handoff { + t.Close() + } + collected(before+2+workers*each, "the GC frees every object a nursery held unclosed") + uncollected(before+2+workers*each, "each object is freed once, however its Close and its nursery race") +} + +// fill returns a nursery's worth of objects, each closed, pushing out every earlier one. +func fill() { + for range 64 { + api.NewTally().Close() + } +} + // forward reads a payload Rust returned and writes it back, whole, sliced, and in a // record: Rust takes a second owner of its own buffer each time, so the bytes stay where // Rust put them, in a frame of its own so nothing keeps them reachable after. diff --git a/ffi/tests/fixtures.rs b/ffi/tests/fixtures.rs index d9e25c625..b48a45ebe 100644 --- a/ffi/tests/fixtures.rs +++ b/ffi/tests/fixtures.rs @@ -292,6 +292,34 @@ fn api_tsan() { sanitized("thread"); } +/// The Go script under Go's race detector, which catches a race between an object's +/// `Close` and the nursery that pushes it out on another goroutine. +#[test] +#[ignore = "slow: just ffi sanitize"] +fn api_go_race() { + let api = api(); + let dir = scratch().join("api").join("go-race"); + let module = dir.join("module"); + go_layout( + api, + &module, + &std::fs::read_to_string(manifest_dir().join("tests").join("api.go")).unwrap(), + ); + let exe = dir.join(format!("api-test{EXE_SUFFIX}")); + run(go(&module, api, &["build", "-race", "-o"]).arg(&exe)); + // Every report fails the run. + let env = [("GORACE", "halt_on_error=1")]; + let output = run(Command::new(&exe).envs(env)); + assert_eq!( + String::from_utf8_lossy(&output.stdout), + api.transcript, + "the Go transcript under the race detector differs from the native one" + ); + for case in ["futures", "owned", "closed", "bytes", "closures"] { + run(Command::new(&exe).arg(case).envs(env)); + } +} + /// Build the api fixture's glue, std included, under the `sanitizer`, and run the C and /// C++ scripts against it, built by clang under the same one. fn sanitized(sanitizer: &str) { diff --git a/quest/m1/ffi-speed/README.md b/quest/m1/ffi-speed/README.md index a4773c3bf..3f5d269ce 100644 --- a/quest/m1/ffi-speed/README.md +++ b/quest/m1/ffi-speed/README.md @@ -13,7 +13,7 @@ Nothing here holds [moq's pin](/quest/m1/ffi-export/ship.md) open. Decided with the maintainer on 2026-11-12, in the quest audit: ✅ this epic, beside [the exporter epic](/quest/m1/ffi-export/README.md), which keeps only what breaks the C ABI or a generated API. Kotlin records, -[returned-object ownership](/quest/m1/ffi-speed/object-ownership.md), and the woken read +returned-object ownership, and the woken read moved here from it. Rejected: keeping them there, where three quests that change no API held the pin open. diff --git a/quest/m1/ffi-speed/object-ownership.md b/quest/m1/ffi-speed/object-ownership.md deleted file mode 100644 index f206fcfd1..000000000 --- a/quest/m1/ffi-speed/object-ownership.md +++ /dev/null @@ -1,46 +0,0 @@ -# [L] Returned objects closed early skip the collector in Go and Dart - -## Goal - -A [rust2-ffi](/ffi/README.md) Go or Dart binding returns an object for close to what C -pays when the caller closes it soon after. [The binding benchmark](/bench/ffi/README.md) -measures an object return at 263 ns in Go and 242 in Dart against C's 25, and every -returned frame, group, and future pays it. -Ownership stays automatic: a dropped object is still freed without a call from the user. - -## Plan - -- The profile ([the benchmark's README](/bench/ffi/README.md#what-a-call-spends), item 2) - puts most of the cost in each runtime's per-object registration with its collector: Go's - `runtime.AddCleanup` and the `Cleanup.Stop` its `Close` makes take 39% of a return and - close, and Dart's `Dart_NewFinalizableHandle`, its crossing, and its finalization take - three quarters (1,600 instructions). Without the registration, Dart's return and close - took 62 to 65 ns against 176 to 245. kixel-ai/rust2#828 took what it could without - deferring the registration: one allocation for a Go object and its handle, and no - `_Arena` in a Dart close. -- Rejected, measured: Dart's `NativeFinalizer.attach` without `detach` (770 to 1,070 ns - a return), Go's `runtime.SetFinalizer` (13% fewer instructions to register and clear - than `AddCleanup` and `Stop`, but it runs every free on one goroutine and keeps each - object a collection longer), and leaving Go's cleanup registered at `Close` (290 to 450 - ns a return against 120, in a microbenchmark). -- Settled by the maintainer (2026-11-15, kixel-ai/rust2#828): a nursery of the last 64 - objects returned, which only registers an object - with the collector once it leaves the nursery still open. A `close` of an object in the - nursery frees its handle at once, with a checked value's drop in the same crossing, and - it never registers. Explicit `Close` stays, and the registration stays as the backstop. - - Dart: one ring in the isolate's `_n`, holding its objects, whose oldest is registered - as today when a return pushes it out. A checked object's handle is then freed at - `close`, so its calls check a closed flag in Dart, as a counted one's do (one load a - call). One finalizable handle on the ring frees what it still holds at isolate exit. - - Go: a ring per P, through a `sync.Pool` as `lending_` pools byte batches. Each - object points at its ring and the ring at its objects, so one `AddCleanup` on the ring - frees its open slots once all of them are unreachable, as when the pool drops it. A - compare-and-swap on the object's slot decides between its `Close` and its eviction. - - The later free is accepted: an object dropped without `Close` is freed only after up - to 64 more returns, or in Go once its ring's other objects are unreachable too, - rather than at the next collection. Rejected: keeping today's per-object - registration and deleting this quest; flushing the nursery each event-loop turn (a - timer each turn costs more than it saves when a turn returns one object, as a woken - read does). -- Re-measure the bench's object return and ready read, and check by drop counts in - `ffi/tests` that an object dropped in the nursery is freed once evicted and collected.