Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions ffi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,9 @@ where it was while its file is unchanged.
fails with `<PREFIX>_STATUS_NOT_SEND` before any Rust code runs, so such a future's poll
takes a trailing `status` too. A `_free` or `_drop` elsewhere drops nothing there: it
queues the drop to the value's thread, which runs it at its next check of one of its
values or as it exits. One freed once its thread exited leaks, printing why, since
values or as it exits. A thread running a loop that may call nothing for a while
passes `<prefix>_thread_wake` a waker, which such a free wakes so the loop schedules
`<prefix>_thread_drain` there. One freed once its thread exited leaks, printing why, since
nothing may drop it. A function that makes such a value, taking none or only in an
`Option`, has a twin `<name>_homed` of the same signature, as do its future's start,
`<name>_start_homed`, and `Default`'s, `<T>_default_homed`: with no such value passed,
Expand Down Expand Up @@ -471,7 +473,9 @@ packages place it in.
- **Checks**: a call passes no `status`, so a borrow or move Rust's rules refuse aborts
with its message, naming the function, as C's does with a NULL one: so does a use or
poll of a value that is not `Send` on another thread than the one that made it, and
one destroyed on another drops on its own thread at its next call or as it exits.
one destroyed on another drops on its own thread: a job posted to the executor of the
thread's latest `then` or `via` drops it, while that task lives and if the job runs on
the thread, else its next call or its exit does.
- **Sharing** stands in for `clone` where no call can tell the two apart but by counting
an `Arc`'s or `Rc`'s clones, which the exporter decides at compile time. A type's copies share when it is `Clone`; `Sync`,
since one that is not is borrowed on one thread at a time, which copies on two threads
Expand Down Expand Up @@ -730,7 +734,9 @@ caller imports it with no ignores.
borrows from, as for a closed object, and `RuntimeError` for any other, as for a value
that is not `Send` used on another thread than the one that made it. Freed on another,
as the collector or the last reference there frees it, such a value drops on its own
thread at its next call or as it exits.
thread: on the next turn of the event loop that ran there when a call made a value that
is not `Send`, though the loop calls nothing into Rust, else at its next call or as it
exits.
- **Records** are slotted dataclasses, compared by their fields, a float by IEEE `==` even
against itself, as Rust does, and **enums** an `Enum`
of upper-case members, or for one with data a class of a dataclass per variant,
Expand Down
2 changes: 2 additions & 0 deletions ffi/runtime/native.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ def fileno(self) -> _int: ...

def take(self, read: _bool, /) -> _list[_int]: ...

def drain(self) -> None: ...

class Node:
def __init__(self, queue: _n.Queue, /) -> None: ...

Expand Down
73 changes: 70 additions & 3 deletions ffi/runtime/python.rs
Original file line number Diff line number Diff line change
Expand Up @@ -243,12 +243,68 @@ pub fn call<T>(f: impl FnOnce() -> T) -> T {
// SAFETY: released just above.
unsafe { PyEval_RestoreThread(state) };
RUNNING.set(outer);
// First, as a drain it queues at once may be one to resume.
if rt::Thread::made() {
drains();
}
if !RESUMES.with_borrow(Vec::is_empty) {
resume();
}
result
}

/// The token of a drain, which no future or `Waker` takes.
const DRAIN: i64 = 0;

/// A call made a value that is not `Send`: give this thread a wake that queues a drain on
/// the event loop running here, if one is, so a free on another thread reaches an idle
/// loop, which its `Queue.drain` then runs.
#[cold]
fn drains() {
// SAFETY: the GIL is held. An exception the caller has set is kept aside while Python
// runs, and put back after.
unsafe {
let (mut kind, mut value, mut trace) = (null_mut(), null_mut(), null_mut());
PyErr_Fetch(&mut kind, &mut value, &mut trace);
// With no loop running here, the thread drains at its next call instead.
let running = running_loop();
let woken = DRAINS.with_borrow(|drains| {
drains.upgrade().is_some_and(|queue| queue.event_loop == running && !queue.closed.load(Ordering::Relaxed))
});
if running != 0 && !woken {
match Obj::new(PyObject_CallNoArgs(ty(T_DRAINS))).and_then(|queue| queue_of(queue.ptr())) {
Ok(queue) => drain_on(&queue),
// The thread drains at its next call instead, and an interrupt its Python ran
// into is raised again at the thread's next chance.
Err(Raised) if PyErr_ExceptionMatches(PyExc_KeyboardInterrupt) != 0 => {
PyErr_Clear();
PyErr_SetInterrupt();
}
Err(Raised) => PyErr_WriteUnraisable(ty(T_DRAINS)),
}
}
PyErr_Restore(kind, value, trace);
}
}

/// Wake this thread through `queue` for each drain.
fn drain_on(queue: &Arc<Queue>) {
let weak = Arc::downgrade(queue);
DRAINS.set(weak.clone());
rt::Thread::wake_here(Arc::new(move || {
if let Some(queue) = weak.upgrade() {
queue.push(DRAIN);
}
}));
}

/// `drain()`, on the thread running its loop: drop what other threads freed of the
/// thread's values that are not `Send`.
unsafe extern "C" fn queue_drain(_: *mut PyObject, _: *mut PyObject) -> *mut PyObject {
call(rt::Thread::drain_here);
wrap(none)
}

thread_local! {
/// The count of what waits on the event loop that last polled or took on this thread,
/// until a call finds none running: a call looks for the loop running only while
Expand All @@ -258,6 +314,8 @@ thread_local! {
static RUNNING: Cell<usize> = const { Cell::new(0) };
/// The queues woken on their loop's thread during the calls it is in.
static RESUMES: RefCell<Vec<Arc<Queue>>> = const { RefCell::new(Vec::new()) };
/// The queue this thread's wake queues a drain on.
static DRAINS: RefCell<std::sync::Weak<Queue>> = const { RefCell::new(std::sync::Weak::new()) };
}

static GET_RUNNING_LOOP: std::sync::OnceLock<Static> = std::sync::OnceLock::new();
Expand Down Expand Up @@ -479,14 +537,16 @@ static MEMBERS: Gil<Vec<Option<Vec<(i64, Obj)>>>> = Gil::new(Vec::new());

/// The runtime's entries, at the start of the registry: the awaitable a pending future
/// is, the runtime's `Duration`, `_resume`, which schedules the running loop's dispatch,
/// `Waker`, `_Pollable`, which runs an awaitable Rust polls, and `ipaddress`'s two classes.
/// `Waker`, `_Pollable`, which runs an awaitable Rust polls, `ipaddress`'s two classes, and
/// `_drains`, which returns the running loop's queue.
pub const T_FUTURE: usize = 0;
const T_DURATION: usize = 1;
const T_RESUME: usize = 2;
const T_WAKER: usize = 3;
const T_POLLABLE: usize = 4;
const T_IPV4: usize = 5;
const T_IPV6: usize = 6;
const T_DRAINS: usize = 7;

/// The class registered at `index`.
pub fn ty(index: usize) -> *mut PyObject {
Expand Down Expand Up @@ -2035,6 +2095,9 @@ pub struct Queue {
woken: AtomicBool,
/// The address of its event loop, which [`RUNNING`] matches while it runs a call.
event_loop: usize,
/// Whether its `Queue` was deallocated, as it is once its loop goes, after which
/// another loop may take the address.
closed: AtomicBool,
/// How many of its futures are parked and its `Waker`s alive, which a wake may queue.
waits: Arc<AtomicUsize>,
tx: Socket,
Expand Down Expand Up @@ -2239,6 +2302,7 @@ unsafe extern "C" fn queue_new(class: *mut PyTypeObject, args: *mut PyObject, _:
signaled: AtomicBool::new(false),
woken: AtomicBool::new(false),
event_loop,
closed: AtomicBool::new(false),
waits: Arc::new(AtomicUsize::new(0)),
tx,
rx,
Expand All @@ -2252,7 +2316,9 @@ unsafe extern "C" fn queue_new(class: *mut PyTypeObject, args: *mut PyObject, _:
unsafe extern "C" fn queue_dealloc(obj: *mut PyObject) {
// SAFETY: Python deallocates a `Queue` once.
unsafe {
drop((*obj.cast::<QueueObj>()).queue.take());
if let Some(queue) = (*obj.cast::<QueueObj>()).queue.take() {
queue.closed.store(true, Ordering::Relaxed);
}
dealloc(obj);
}
}
Expand Down Expand Up @@ -2807,10 +2873,11 @@ static FUTURE_METHODS: Methods<4> = Methods(UnsafeCell::new([
END,
]));

static QUEUE_METHODS: Methods<4> = Methods(UnsafeCell::new([
static QUEUE_METHODS: Methods<5> = Methods(UnsafeCell::new([
noargs(c"fileno", queue_fileno),
onearg(c"take", queue_take),
noargs(c"signaled", queue_signaled),
noargs(c"drain", queue_drain),
END,
]));

Expand Down
52 changes: 52 additions & 0 deletions ffi/runtime/rt.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -713,6 +713,33 @@ namespace detail {
// A future driven on an executor, which a `task` cancels.
struct task_node : node {
virtual void cancel() noexcept = 0;
// Post a drain of what other threads freed of this thread's values that are not
// `Send` to the executor.
virtual void drain() noexcept = 0;
};

// What a free elsewhere of a value of its thread that is not `Send` wakes: the thread's
// latest task, which posts a drain to its executor, until the task is gone. Rust holds it
// rather than the task, so a gone task's callback and executor are freed with it.
class drainer final : public node {
::std::mutex mutex_;
task_node *task_;

public:
explicit drainer(task_node *task) noexcept : task_(task) {}

void wake() noexcept override {
::std::lock_guard<::std::mutex> lock(mutex_);
if (task_ != nullptr) {
task_->drain();
}
}

// Called as the task goes, after which it is never woken again.
void detach() noexcept {
::std::lock_guard<::std::mutex> lock(mutex_);
task_ = nullptr;
}
};

template <class Op, class Exec, class F>
Expand Down Expand Up @@ -1038,6 +1065,8 @@ class driver final : public task_node {
// Held across each call to the executor, so `cancel` returns only once none is under
// way, and none starts after.
::std::mutex posting_;
// What its thread's frees elsewhere wake while it is the thread's latest task, if any.
drainer *drainer_ = nullptr;
// Null once the future finished or was cancelled.
typename Op::raw *raw_;
Exec exec_;
Expand Down Expand Up @@ -1118,6 +1147,11 @@ class driver final : public task_node {
}

void cancel() noexcept override {
// First, so no drain is under way or starts once this returns.
if (drainer_ != nullptr) {
drainer_->detach();
::std::exchange(drainer_, nullptr)->unref();
}
// From its own callback, the future is already gone.
if (on_.load(::std::memory_order_relaxed) != ::std::this_thread::get_id()) {
::std::lock_guard<::std::mutex> lock(mutex_);
Expand All @@ -1131,13 +1165,31 @@ class driver final : public task_node {
}
unref();
}

void drain() noexcept override {
::std::lock_guard<::std::mutex> lock(posting_);
// It drains the thread it runs on, which is this one's only if the executor runs it
// here; a job elsewhere leaves the drops to this thread's next call.
exec_([]() { c_thread_drain(); });
}

// Be its thread's latest task, which its frees elsewhere wake.
void drains() {
drainer_ = new drainer(this);
c_thread_wake(give(drainer_));
}
};

template <class Op, class Exec, class F>
void spawn(task &slot, typename Op::raw *raw, Exec exec, F f) {
auto *d = new driver<Op, Exec, F>(raw, ::std::move(exec), ::std::move(f));
slot = task();
slot.node_ = d;
// The latest executor of this thread drains what other threads free of its values
// that are not `Send`, though it calls nothing into Rust, until its task is gone.
if constexpr (drains) {
d->drains();
}
d->wake();
}

Expand Down
10 changes: 10 additions & 0 deletions ffi/runtime/rt.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,10 @@ async def _pump(self, loop: _asyncio.AbstractEventLoop, sock: _socket.socket) ->
def dispatch(self, read: _bool) -> None:
loop: _asyncio.AbstractEventLoop | None = None
for token in self.queue.take(read):
if token == 0:
# Other threads freed values of this thread that are not Send.
self.queue.drain()
continue
if token < 0:
self.wakers.pop(-token, None)
continue
Expand Down Expand Up @@ -87,6 +91,12 @@ def _resume() -> _n.Queue | None:
return w.queue


def _drains() -> _n.Queue:
"""The queue of the loop running here, which a free on another thread of a value this
thread made that is not Send queues a drain on."""
return _wakeup(_get_running_loop()).queue


def _wakeup(loop: _asyncio.AbstractEventLoop) -> _Wakeup:
w = _wakeups.get(loop)
if w is None:
Expand Down
Loading