Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions ffi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -727,7 +727,8 @@ the rest cross encoded.
## Python

`<prefix>.py` is Python over a CPython extension that `--features python` builds into the
same library on the stable ABI, so one build loads in CPython 3.11 and later. The module
same library on the stable ABI, so one build loads in CPython 3.11 and later, but not in a
free-threaded build (3.13t, 3.14t), which the module refuses at import with an `ImportError`. The module
loads the library from beside itself: `lib<prefix>.so`, `lib<prefix>.dylib`, or
`<prefix>.dll`. Its glue converts each Python value to the C value and calls the C ABI
function in-process, so a call is one crossing, which releases the GIL while Rust runs.
Expand Down Expand Up @@ -1472,7 +1473,8 @@ x86_64 emulator at API 24, the oldest moq's Android library supports, and at API
`tests/api.py` runs the script through the Python, in CPython's development mode with
warnings as errors, checks its futures the same way, what frees an object or bytes nobody
closed, and the values Python's types bound, and breaks each runtime check, which must raise or
abort with its message; the Python with another contract must refuse the library.
abort with its message; the Python with another contract must refuse the library. `api_free_threaded` imports it under
nixpkgs' free-threaded CPython, which the dev shell names in `PYTHON_FREE_THREADED`, and checks the `ImportError`.
`api_py_types` checks the Python with mypy and pyright, both strict, at CPython 3.11, as
`just ffi moq` checks moq's.
`tests/api.swift` runs the script through the Swift, built as a module in Swift 6's
Expand Down
76 changes: 62 additions & 14 deletions ffi/src/python.rs
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ use std::hash::Hasher;
use rustc_data_structures::stable_hash::StableHasher;

use crate::abi::{self, Abi, Ty, pascal};
use crate::ir::{Access, Family, Function, Int, Literal, On, Receiver, Traits, Type};
use crate::ir::{Access, Family, Function, Int, Literal, On, Param, Receiver, Traits, Type};
use crate::kotlin::{indent_by, rust_c, strip_refs};

const RUNTIME: &str = include_str!("../runtime/rt.py");
Expand Down Expand Up @@ -65,6 +65,7 @@ const TAKEN: &[&str] = &[
"_dataclass",
"_same",
"_all",
"_any",
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
"_dict",
"_float",
"_int",
Expand Down Expand Up @@ -210,6 +211,7 @@ pub(crate) fn generate(abi: &Abi<'_>, packages: &str, contract: u64) -> Result<O
from builtins import TypeError as _TypeError\n\
from builtins import ValueError as _ValueError\n\
from builtins import all as _all\n\
from builtins import any as _any\n\
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
from builtins import bool as _bool\n\
from builtins import bytearray as _bytearray\n\
from builtins import bytes as _bytes\n\
Expand Down Expand Up @@ -255,7 +257,18 @@ pub(crate) fn generate(abi: &Abi<'_>, packages: &str, contract: u64) -> Result<O
def _load() -> _object:\n\
\timport importlib.machinery\n\
\timport importlib.util\n\
\timport os\n\n\
\timport os\n\
\timport re\n\n\
\t# Loading by path skips the filename tag that keeps a GIL build's library out of a\n\
\t# free-threaded interpreter, which the stable ABI does not cover and which would crash.\n\
\t# Its extension suffixes carry the tag (`314t`, or `314td` on a POSIX debug build, after `_d` on a Windows one),\n\
\t# where sysconfig's `Py_GIL_DISABLED` follows an overridden sysconfig data file, as a\n\
\t# Nix shell sets.\n\
\tif _any(re.match(r\"(?:_d)?\\.(?:cpython-|cp)\\d+td?[-.]\", s) for s in importlib.machinery.EXTENSION_SUFFIXES):\n\
\t\traise _ImportError(\n\
\t\t\t\"{p}.py loads a library for CPython's GIL build, which a free-threaded CPython cannot use: \"\n\
\t\t\tf\"{{_sys.implementation.name}} {{_sys.version.partition(' ')[0]}}, free-threaded\"\n\
\t\t)\n\n\
\tname = {{\"win32\": \"{p}.dll\", \"darwin\": \"lib{p}.dylib\"}}.get(_sys.platform, \"lib{p}.so\")\n\
\tpath = os.path.join(os.path.dirname(os.path.abspath(__file__)), name)\n\
\tloader = importlib.machinery.ExtensionFileLoader(\"_{p}\", path)\n\
Expand Down Expand Up @@ -1368,10 +1381,10 @@ impl<'a> Gen<'a> {
crate::lock::function(&mut self.locked, method, &member);
let params: Vec<String> = std::iter::once("self".to_string())
.chain(
method
.params
.iter()
.map(|p| format!("{}: {}", param_name(&p.name), self.py_lent(&p.ty))),
param_names(&method.params)
.into_iter()
.zip(&method.params)
.map(|(name, p)| format!("{name}: {}", self.py_lent(&p.ty))),
)
.collect();
body.push(format!(
Expand Down Expand Up @@ -1704,9 +1717,8 @@ impl<'a> Gen<'a> {
let mut closures = Vec::new();
// Each parameter Rust writes back, by its argument's index, and what it holds.
let mut written: Vec<(usize, Type)> = Vec::new();
for param in &function.params {
for (param, name) in function.params.iter().zip(param_names(&function.params)) {
let (_, c) = c_params.next().expect("each parameter has its C one");
let name = param_name(&param.name);
let annotation = match &param.ty {
Type::Ref(_, inner) if param.ty.written() => format!("Mut[{}]", self.py_ty(inner)),
ty => self.py_param(ty),
Expand Down Expand Up @@ -1834,8 +1846,7 @@ impl<'a> Gen<'a> {
"\n\nRust changes a copy of this value, which it returns first, as the call left it: this one never changes.",
);
}
for param in &function.params {
let name = param_name(&param.name);
for (param, name) in function.params.iter().zip(param_names(&function.params)) {
if let Type::Closure(closure_) = &param.ty {
let lent = &closure_.params;
match closure_.kept {
Expand Down Expand Up @@ -1870,7 +1881,7 @@ impl<'a> Gen<'a> {
let property = matches!(function.access, Access::Field { set: false, .. }) && place == Place::Member;
match direct {
true => {
let names: Vec<String> = function.params.iter().map(|p| param_name(&p.name)).collect();
let names = param_names(&function.params);
let d = Direct {
member: &member,
place,
Expand Down Expand Up @@ -2078,15 +2089,30 @@ fn field_name(name: &str) -> String {
}
}

/// A parameter's Python name, apart from the runtime's names a body reads.
/// A parameter's Python name, apart from the runtime's names a body reads: not `_any`, which
/// only `_load` reads, and it has no parameter.
fn param_name(name: &str) -> String {
let name = kw(&abi::snake(name));
match TAKEN.contains(&name.as_str()) || name == "self" {
match (TAKEN.contains(&name.as_str()) && name != "_any") || name == "self" {
true => format!("{name}_"),
false => name,
}
}

/// The Python names of `params`, each apart from the others: one that an earlier parameter's
/// rename took, such as `_all_` after `_all`, gets another `_`.
fn param_names<'a>(params: impl IntoIterator<Item = &'a Param>) -> Vec<String> {
let mut out: Vec<String> = Vec::new();
for param in params {
let mut name = param_name(&param.name);
while out.contains(&name) {
name.push('_');
}
out.push(name);
}
out
}

/// The name a member's code declares, for [`Gen::distinct`].
fn member_name(code: &str) -> Option<String> {
let line = code.lines().find(|l| {
Expand Down Expand Up @@ -2163,7 +2189,29 @@ fn ieee_eq(class: &str, fields: &[String]) -> String {

#[cfg(test)]
mod tests {
use super::member;
use super::{member, param_name, param_names};
use crate::ir::{Param, Type};

#[test]
fn params_keep_apart() {
let params = |names: &[&str]| {
let params: Vec<Param> = names
.iter()
.map(|n| Param {
name: n.to_string(),
ty: Type::Bool,
interned: false,
})
.collect();
param_names(&params)
};
// `_any` is read only by `_load`, which has no parameter, so it keeps its name.
assert_eq!(param_name("_any"), "_any");
assert_eq!(param_name("_all"), "_all_");
assert_eq!(params(&["_all", "_all_"]), ["_all_", "_all__"]);
assert_eq!(params(&["_all_", "_all"]), ["_all_", "_all__"]);
assert_eq!(params(&["a", "b"]), ["a", "b"]);
}

#[test]
fn member_keeps_clear_of_the_runtime_aliases() {
Expand Down
73 changes: 73 additions & 0 deletions ffi/tests/fixtures.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1242,6 +1242,79 @@ fn api_python() {
);
}

/// A free-threaded CPython refuses the module with an `ImportError`: the library is for the
/// GIL's ABI, which it would load through its explicit path and crash in. `PYTHON_FREE_THREADED`
/// or `python3.14t` is the interpreter, which the dev shell provides.
#[test]
fn api_free_threaded() {
let python = std::env::var_os("PYTHON_FREE_THREADED").unwrap_or_else(|| "python3.14t".into());
let dir = api().out.join("python");
let probe = |code: &str| {
let mut cmd = Command::new(&python);
// `-P` keeps the working directory off the path; PYTHONPATH replaces the dev shell's,
// which names the GIL build's packages. `-X gil=0` beats an inherited `PYTHON_GIL=1`,
// which would turn the GIL on in a free-threaded build.
cmd.args(["-X", "gil=0", "-P", "-c", code]).env("PYTHONPATH", &dir);
cmd.output().unwrap_or_else(|err| panic!("running {cmd:?}: {err}"))
};
// A GIL build would pass for the refusal's test, so check the interpreter first.
let build = probe("import sys; sys.exit(sys._is_gil_enabled())");
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
assert!(
build.status.success(),
"{python:?} is not a free-threaded CPython: {}",
String::from_utf8_lossy(&build.stderr)
);
let output = probe("import api");
// Exit 1 is an uncaught exception, where a crash has no code.
assert_eq!(
output.status.code(),
Some(1),
"importing under a free-threaded CPython: {output:?}"
);
let stderr = String::from_utf8_lossy(&output.stderr);
assert!(
stderr.contains(
"ImportError: api.py loads a library for CPython's GIL build, which a free-threaded CPython cannot use"
),
"importing under a free-threaded CPython printed {stderr}"
);

// The suffixes of the platforms this host is not, appended on the GIL build, which
// loads from beside the module whatever its list holds.
let gil = |suffix: &str| {
let mut cmd = Command::new(std::env::var_os("PYTHON").unwrap_or_else(|| "python3".into()));
cmd.args(["-P", "-c"])
.arg(format!(
"import importlib.machinery as m; m.EXTENSION_SUFFIXES.append('{suffix}'); import api"
))
.env("PYTHONPATH", &dir);
cmd.output().unwrap_or_else(|err| panic!("running {cmd:?}: {err}"))
};
for suffix in [
".cp314t-win_amd64.pyd",
"_d.cp314t-win_amd64.pyd",
".cpython-313t-darwin.so",
".cpython-315t-aarch64-linux-gnu.so",
".cpython-314td-x86_64-linux-gnu.so",
] {
let output = gil(suffix);
assert_eq!(output.status.code(), Some(1), "{suffix} was not refused: {output:?}");
assert!(
String::from_utf8_lossy(&output.stderr).contains("a free-threaded CPython cannot use"),
"{suffix} printed {output:?}"
);
}
for suffix in [
".cp314-win_amd64.pyd",
"_d.cp314-win_amd64.pyd",
".cpython-314-darwin.so",
".cpython-314d-x86_64-linux-gnu.so",
] {
let output = gil(suffix);
assert!(output.status.success(), "{suffix} was refused: {output:?}");
}
}

/// Random sequences of the ownership protocol's steps through the Python, by
/// `tests/protocol.py`, each matching what the model says each step returns.
#[test]
Expand Down
3 changes: 3 additions & 0 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,9 @@
RUST2_TS_TOOLCHAIN = tsToolchain;
RUST2_TS_SYSROOT = rust2-ts-std;
KOTLINX_COROUTINES = "${kotlinx-coroutines}";
# The free-threaded CPython the generated Python must refuse. By path, since its
# python3 would shadow the GIL build's on PATH.
PYTHON_FREE_THREADED = "${pkgs.python314FreeThreading}/bin/python3.14t";
# The pinned toolchain goes on PATH ahead of everything the host had, which
# shadows the Cargo shim `mbx setup` installs. Put it back in front, as moq's dev
# shell does, so cargo builds through mbx's cache wherever the host set one up,
Expand Down
25 changes: 0 additions & 25 deletions quest/m1/ffi-free-threaded.md

This file was deleted.

9 changes: 5 additions & 4 deletions quest/m3/python-abi3t.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,11 @@ interpreter nightly.

## Plan

Decided with the maintainer on 2026-11-18: ✅ m3, since no consumer asks, after
[the refusal](/quest/m1/ffi-free-threaded.md) fixes the crash (rejected: both in m1; no
quest).
Decided with the maintainer on 2026-11-18: ✅ m3, since no consumer asks, after the
refusal fixes the crash (rejected: both in m1; no quest). The refusal is in; it checks the
extension suffixes for the free-threaded tag in `_load` (`ffi/src/python.rs`) and
`api_free_threaded` (`ffi/tests/fixtures.rs`) tests it, so this quest narrows both to
pre-3.15 free-threaded builds, which an abi3t library cannot load in.

It needs the `PyModExport` entry point with a `Py_mod_abi` slot ([PEP 793](https://peps.python.org/pep-0793/)),
which also brings the multi-phase init subinterpreters need; per-object data through
Expand All @@ -27,4 +29,3 @@ Public API: the wheels released; the Python API is unchanged.
## Required

- [Exporter](/quest/m1/ffi-export/README.md) - the Python binding's API is settled before its runtime is rewritten
- [Refuse free-threaded CPython](/quest/m1/ffi-free-threaded.md) - the crash is fixed first