diff --git a/ffi/README.md b/ffi/README.md index e10722bb7..36cbea61d 100644 --- a/ffi/README.md +++ b/ffi/README.md @@ -727,7 +727,8 @@ the rest cross encoded. ## Python `.py` is Python over a CPython extension that `--features python` builds into the -same library on the stable ABI, so one build loads in CPython 3.11 and later. The module +same library on the stable ABI, so one build loads in CPython 3.11 and later, but not in a +free-threaded build (3.13t, 3.14t), which the module refuses at import with an `ImportError`. The module loads the library from beside itself: `lib.so`, `lib.dylib`, or `.dll`. Its glue converts each Python value to the C value and calls the C ABI function in-process, so a call is one crossing, which releases the GIL while Rust runs. @@ -1472,7 +1473,8 @@ x86_64 emulator at API 24, the oldest moq's Android library supports, and at API `tests/api.py` runs the script through the Python, in CPython's development mode with warnings as errors, checks its futures the same way, what frees an object or bytes nobody closed, and the values Python's types bound, and breaks each runtime check, which must raise or -abort with its message; the Python with another contract must refuse the library. +abort with its message; the Python with another contract must refuse the library. `api_free_threaded` imports it under +nixpkgs' free-threaded CPython, which the dev shell names in `PYTHON_FREE_THREADED`, and checks the `ImportError`. `api_py_types` checks the Python with mypy and pyright, both strict, at CPython 3.11, as `just ffi moq` checks moq's. `tests/api.swift` runs the script through the Swift, built as a module in Swift 6's diff --git a/ffi/src/python.rs b/ffi/src/python.rs index 66d2a8a9d..4eedac795 100644 --- a/ffi/src/python.rs +++ b/ffi/src/python.rs @@ -27,7 +27,7 @@ use std::hash::Hasher; use rustc_data_structures::stable_hash::StableHasher; use crate::abi::{self, Abi, Ty, pascal}; -use crate::ir::{Access, Family, Function, Int, Literal, On, Receiver, Traits, Type}; +use crate::ir::{Access, Family, Function, Int, Literal, On, Param, Receiver, Traits, Type}; use crate::kotlin::{indent_by, rust_c, strip_refs}; const RUNTIME: &str = include_str!("../runtime/rt.py"); @@ -65,6 +65,7 @@ const TAKEN: &[&str] = &[ "_dataclass", "_same", "_all", + "_any", "_dict", "_float", "_int", @@ -210,6 +211,7 @@ pub(crate) fn generate(abi: &Abi<'_>, packages: &str, contract: u64) -> Result, packages: &str, contract: u64) -> Result _object:\n\ \timport importlib.machinery\n\ \timport importlib.util\n\ - \timport os\n\n\ + \timport os\n\ + \timport re\n\n\ + \t# Loading by path skips the filename tag that keeps a GIL build's library out of a\n\ + \t# free-threaded interpreter, which the stable ABI does not cover and which would crash.\n\ + \t# Its extension suffixes carry the tag (`314t`, or `314td` on a POSIX debug build, after `_d` on a Windows one),\n\ + \t# where sysconfig's `Py_GIL_DISABLED` follows an overridden sysconfig data file, as a\n\ + \t# Nix shell sets.\n\ + \tif _any(re.match(r\"(?:_d)?\\.(?:cpython-|cp)\\d+td?[-.]\", s) for s in importlib.machinery.EXTENSION_SUFFIXES):\n\ + \t\traise _ImportError(\n\ + \t\t\t\"{p}.py loads a library for CPython's GIL build, which a free-threaded CPython cannot use: \"\n\ + \t\t\tf\"{{_sys.implementation.name}} {{_sys.version.partition(' ')[0]}}, free-threaded\"\n\ + \t\t)\n\n\ \tname = {{\"win32\": \"{p}.dll\", \"darwin\": \"lib{p}.dylib\"}}.get(_sys.platform, \"lib{p}.so\")\n\ \tpath = os.path.join(os.path.dirname(os.path.abspath(__file__)), name)\n\ \tloader = importlib.machinery.ExtensionFileLoader(\"_{p}\", path)\n\ @@ -1368,10 +1381,10 @@ impl<'a> Gen<'a> { crate::lock::function(&mut self.locked, method, &member); let params: Vec = std::iter::once("self".to_string()) .chain( - method - .params - .iter() - .map(|p| format!("{}: {}", param_name(&p.name), self.py_lent(&p.ty))), + param_names(&method.params) + .into_iter() + .zip(&method.params) + .map(|(name, p)| format!("{name}: {}", self.py_lent(&p.ty))), ) .collect(); body.push(format!( @@ -1704,9 +1717,8 @@ impl<'a> Gen<'a> { let mut closures = Vec::new(); // Each parameter Rust writes back, by its argument's index, and what it holds. let mut written: Vec<(usize, Type)> = Vec::new(); - for param in &function.params { + for (param, name) in function.params.iter().zip(param_names(&function.params)) { let (_, c) = c_params.next().expect("each parameter has its C one"); - let name = param_name(¶m.name); let annotation = match ¶m.ty { Type::Ref(_, inner) if param.ty.written() => format!("Mut[{}]", self.py_ty(inner)), ty => self.py_param(ty), @@ -1834,8 +1846,7 @@ impl<'a> Gen<'a> { "\n\nRust changes a copy of this value, which it returns first, as the call left it: this one never changes.", ); } - for param in &function.params { - let name = param_name(¶m.name); + for (param, name) in function.params.iter().zip(param_names(&function.params)) { if let Type::Closure(closure_) = ¶m.ty { let lent = &closure_.params; match closure_.kept { @@ -1870,7 +1881,7 @@ impl<'a> Gen<'a> { let property = matches!(function.access, Access::Field { set: false, .. }) && place == Place::Member; match direct { true => { - let names: Vec = function.params.iter().map(|p| param_name(&p.name)).collect(); + let names = param_names(&function.params); let d = Direct { member: &member, place, @@ -2078,15 +2089,30 @@ fn field_name(name: &str) -> String { } } -/// A parameter's Python name, apart from the runtime's names a body reads. +/// A parameter's Python name, apart from the runtime's names a body reads: not `_any`, which +/// only `_load` reads, and it has no parameter. fn param_name(name: &str) -> String { let name = kw(&abi::snake(name)); - match TAKEN.contains(&name.as_str()) || name == "self" { + match (TAKEN.contains(&name.as_str()) && name != "_any") || name == "self" { true => format!("{name}_"), false => name, } } +/// The Python names of `params`, each apart from the others: one that an earlier parameter's +/// rename took, such as `_all_` after `_all`, gets another `_`. +fn param_names<'a>(params: impl IntoIterator) -> Vec { + let mut out: Vec = Vec::new(); + for param in params { + let mut name = param_name(¶m.name); + while out.contains(&name) { + name.push('_'); + } + out.push(name); + } + out +} + /// The name a member's code declares, for [`Gen::distinct`]. fn member_name(code: &str) -> Option { let line = code.lines().find(|l| { @@ -2163,7 +2189,29 @@ fn ieee_eq(class: &str, fields: &[String]) -> String { #[cfg(test)] mod tests { - use super::member; + use super::{member, param_name, param_names}; + use crate::ir::{Param, Type}; + + #[test] + fn params_keep_apart() { + let params = |names: &[&str]| { + let params: Vec = names + .iter() + .map(|n| Param { + name: n.to_string(), + ty: Type::Bool, + interned: false, + }) + .collect(); + param_names(¶ms) + }; + // `_any` is read only by `_load`, which has no parameter, so it keeps its name. + assert_eq!(param_name("_any"), "_any"); + assert_eq!(param_name("_all"), "_all_"); + assert_eq!(params(&["_all", "_all_"]), ["_all_", "_all__"]); + assert_eq!(params(&["_all_", "_all"]), ["_all_", "_all__"]); + assert_eq!(params(&["a", "b"]), ["a", "b"]); + } #[test] fn member_keeps_clear_of_the_runtime_aliases() { diff --git a/ffi/tests/fixtures.rs b/ffi/tests/fixtures.rs index 1efc53339..f0745e67a 100644 --- a/ffi/tests/fixtures.rs +++ b/ffi/tests/fixtures.rs @@ -1242,6 +1242,79 @@ fn api_python() { ); } +/// A free-threaded CPython refuses the module with an `ImportError`: the library is for the +/// GIL's ABI, which it would load through its explicit path and crash in. `PYTHON_FREE_THREADED` +/// or `python3.14t` is the interpreter, which the dev shell provides. +#[test] +fn api_free_threaded() { + let python = std::env::var_os("PYTHON_FREE_THREADED").unwrap_or_else(|| "python3.14t".into()); + let dir = api().out.join("python"); + let probe = |code: &str| { + let mut cmd = Command::new(&python); + // `-P` keeps the working directory off the path; PYTHONPATH replaces the dev shell's, + // which names the GIL build's packages. `-X gil=0` beats an inherited `PYTHON_GIL=1`, + // which would turn the GIL on in a free-threaded build. + cmd.args(["-X", "gil=0", "-P", "-c", code]).env("PYTHONPATH", &dir); + cmd.output().unwrap_or_else(|err| panic!("running {cmd:?}: {err}")) + }; + // A GIL build would pass for the refusal's test, so check the interpreter first. + let build = probe("import sys; sys.exit(sys._is_gil_enabled())"); + assert!( + build.status.success(), + "{python:?} is not a free-threaded CPython: {}", + String::from_utf8_lossy(&build.stderr) + ); + let output = probe("import api"); + // Exit 1 is an uncaught exception, where a crash has no code. + assert_eq!( + output.status.code(), + Some(1), + "importing under a free-threaded CPython: {output:?}" + ); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains( + "ImportError: api.py loads a library for CPython's GIL build, which a free-threaded CPython cannot use" + ), + "importing under a free-threaded CPython printed {stderr}" + ); + + // The suffixes of the platforms this host is not, appended on the GIL build, which + // loads from beside the module whatever its list holds. + let gil = |suffix: &str| { + let mut cmd = Command::new(std::env::var_os("PYTHON").unwrap_or_else(|| "python3".into())); + cmd.args(["-P", "-c"]) + .arg(format!( + "import importlib.machinery as m; m.EXTENSION_SUFFIXES.append('{suffix}'); import api" + )) + .env("PYTHONPATH", &dir); + cmd.output().unwrap_or_else(|err| panic!("running {cmd:?}: {err}")) + }; + for suffix in [ + ".cp314t-win_amd64.pyd", + "_d.cp314t-win_amd64.pyd", + ".cpython-313t-darwin.so", + ".cpython-315t-aarch64-linux-gnu.so", + ".cpython-314td-x86_64-linux-gnu.so", + ] { + let output = gil(suffix); + assert_eq!(output.status.code(), Some(1), "{suffix} was not refused: {output:?}"); + assert!( + String::from_utf8_lossy(&output.stderr).contains("a free-threaded CPython cannot use"), + "{suffix} printed {output:?}" + ); + } + for suffix in [ + ".cp314-win_amd64.pyd", + "_d.cp314-win_amd64.pyd", + ".cpython-314-darwin.so", + ".cpython-314d-x86_64-linux-gnu.so", + ] { + let output = gil(suffix); + assert!(output.status.success(), "{suffix} was refused: {output:?}"); + } +} + /// Random sequences of the ownership protocol's steps through the Python, by /// `tests/protocol.py`, each matching what the model says each step returns. #[test] diff --git a/flake.nix b/flake.nix index 1f524921b..e58ed7080 100644 --- a/flake.nix +++ b/flake.nix @@ -279,6 +279,9 @@ RUST2_TS_TOOLCHAIN = tsToolchain; RUST2_TS_SYSROOT = rust2-ts-std; KOTLINX_COROUTINES = "${kotlinx-coroutines}"; + # The free-threaded CPython the generated Python must refuse. By path, since its + # python3 would shadow the GIL build's on PATH. + PYTHON_FREE_THREADED = "${pkgs.python314FreeThreading}/bin/python3.14t"; # The pinned toolchain goes on PATH ahead of everything the host had, which # shadows the Cargo shim `mbx setup` installs. Put it back in front, as moq's dev # shell does, so cargo builds through mbx's cache wherever the host set one up, diff --git a/quest/m1/ffi-free-threaded.md b/quest/m1/ffi-free-threaded.md deleted file mode 100644 index d2e43775f..000000000 --- a/quest/m1/ffi-free-threaded.md +++ /dev/null @@ -1,25 +0,0 @@ -# [XS] Refuse free-threaded CPython at import - -## Goal - -Importing a generated Python module under a free-threaded CPython (3.13t, 3.14t) raises -`ImportError` naming the build, instead of segfaulting. `tests/api.py` checks it under the -flake's free-threaded interpreter. - -## Plan - -Found by the 2026-11-18 language audit, reproduced with nixpkgs' `python314FreeThreading` -on a moq export (exit 139): `_load` (`ffi/src/python.rs:255-262`) loads the library by -explicit path through `ExtensionFileLoader`, skipping the filename tag check that keeps a -regular build out of a free-threaded interpreter, and single-phase init gets no ABI check -([PEP 793](https://peps.python.org/pep-0793/), [gh-137210](https://github.com/python/cpython/issues/137210)). -Recommended: check `sysconfig.get_config_var("Py_GIL_DISABLED")` before loading. - -Decided with the maintainer on 2026-11-18: ✅ refuse now; real support is -[an abi3t build](/quest/m3/python-abi3t.md) in m3 (rejected: both in m1; the refusal -alone). - -Caused by [#407](https://github.com/kixel-ai/rust2/pull/407), which added the loader with -the Python generator. - -Public API: none; an import that crashed now raises. diff --git a/quest/m3/python-abi3t.md b/quest/m3/python-abi3t.md index 05ec46790..4ba406c88 100644 --- a/quest/m3/python-abi3t.md +++ b/quest/m3/python-abi3t.md @@ -11,9 +11,11 @@ interpreter nightly. ## Plan -Decided with the maintainer on 2026-11-18: ✅ m3, since no consumer asks, after -[the refusal](/quest/m1/ffi-free-threaded.md) fixes the crash (rejected: both in m1; no -quest). +Decided with the maintainer on 2026-11-18: ✅ m3, since no consumer asks, after the +refusal fixes the crash (rejected: both in m1; no quest). The refusal is in; it checks the +extension suffixes for the free-threaded tag in `_load` (`ffi/src/python.rs`) and +`api_free_threaded` (`ffi/tests/fixtures.rs`) tests it, so this quest narrows both to +pre-3.15 free-threaded builds, which an abi3t library cannot load in. It needs the `PyModExport` entry point with a `Py_mod_abi` slot ([PEP 793](https://peps.python.org/pep-0793/)), which also brings the multi-phase init subinterpreters need; per-object data through @@ -27,4 +29,3 @@ Public API: the wheels released; the Python API is unchanged. ## Required - [Exporter](/quest/m1/ffi-export/README.md) - the Python binding's API is settled before its runtime is rewritten -- [Refuse free-threaded CPython](/quest/m1/ffi-free-threaded.md) - the crash is fixed first