Skip to content

fix(ffi): a Python module lists its exports in __all__ - #925

Merged
kixelated merged 9 commits into
mainfrom
quest/m1/ffi-export/python-private-names
Nov 20, 2026
Merged

kixelated merged 9 commits into
mainfrom
quest/m1/ffi-export/python-private-names

Conversation

@kixelated

@kixelated kixelated commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m1/ffi-export/python-private-names.md (planned in kixel-ai/rust2#870, added by kixel-ai/rust2#900), which this deletes.

A Rust pub item whose name starts with _ now generates a Python module that pyright strict accepts. The generated module lists every export in a sorted __all__.

Changes

  • ffi/src/python.rs: the module's head now declares __all__, built from the generator's name table (Gen::names) minus the runtime's private names (TAKEN), plus the runtime's public classes (Duration, ForeignReader, Mut, Waker). Bound, each range class (BoundU64, ...), and each ArrayU8x<n> class are now recorded in that table where they are emitted, so __all__ comes from one source.
    • pyright counts a name in __all__ as accessed. So a function stub that is declared under if TYPE_CHECKING: and bound from the library in else: no longer reports reportUnusedFunction when its name is private (_deprecated_).
    • from <prefix> import * now reaches underscore-named exports and constants made by __getattr__. It no longer leaks annotations (the from __future__ feature object).
  • Names Python reserves: a module-level claim now treats every __x__ name as taken (Gen::claim), so a pub fn __all__ exports as __all___ instead of overwriting the export list. The same applies to __getattr__, __name__, __doc__, and the rest. A name ending in ___ counts as free, so the rename terminates. (Found in review round 1.)
  • _inspect (Pullfrog's finding on the last head): TAKEN now holds the module's inspect alias, so a pub fn _inspect exports as _inspect_ and can no longer rebind the alias that runtime/rt.py's awaitable check reads. The import header moved into an IMPORTS const, and a new test, module_names_taken, checks that every alias it binds and every _ name runtime/rt.py binds at module level is in TAKEN.
  • api fixture: adds pub fn __all__. It also adds pub fn _deprecated at the crate root. Its Python name is _deprecated_, because the module's own _deprecated decorator already takes _deprecated. Every binding names it apart from deprecated: C api__deprecated, Swift/Kotlin/Dart deprecated_, Go Deprecated__.
  • tests/api.py (idioms): runs from api import *, calls _deprecated_ and __all___, and checks that the imported names are exactly the module's public bindings, plus those two, plus the constants made by __getattr__.
  • ffi/README.md: one line on __all__. surface/: regenerated.

Public API

Python bindings gain __all__. This is a behavior change for import *: it now includes underscore-named exports and constants, and drops annotations. Nothing is breaking. There is no wire change and no other binding changes.

Decisions

  • ✅ (maintainer, in #900's pick) List every export in __all__ rather than drop the fixture item.
  • (mine) __all__ also includes the runtime's public classes (Duration, ForeignReader, Mut, Waker, and Bound where a range bound exists) and the array and range classes, because signatures name them and a star import should give a caller what annotations spell. Alternative: only Rust items. Rejected because a star-importing caller could not name Duration.
  • (mine) __all__ is placed after the imports, sorted, as a list. A tuple would behave the same; the list is the common convention and both checkers accept it.
  • (mine) The fixture item is _deprecated, at the crate root. Inside a module (clash::_x), the Python name gets a clash_ prefix and the bug does not reproduce. The name also collides with the module's own _deprecated, which exercises the trailing-_ rename.
  • (mine) Every __x__ name is reserved at module level, not only __all__, because Python defines the whole family and a list of known names would go stale.
  • (mine) Only Python's test calls the new item. The other bindings compile and type-check it, and each names it apart. Calling it from every language's test is outside this quest's Goal.
  • (mine) Fix Pullfrog's _inspect finding with a test over the whole class of names (every import alias and every runtime module-level _ name) rather than only adding one entry. Alternative: add the entry alone. Rejected because the list had already drifted once.

Validation

  • Reproduced first: with only the fixture item added, api_py_types failed with api.py:7398:6 - error: Function "_deprecated_" is not accessed (reportUnusedFunction). With the change, it passes under both mypy and pyright strict.
  • Independent check at runtime: the import * assertion in api.py idioms compares __all__ against the loaded module's own namespace. Its first run caught ArrayU8x4, BoundU64, and BoundsU64 missing from the name table; they were fixed before landing.
  • module_names_taken fails with "the module's _inspect is not in TAKEN" when the entry is removed, and passes with it. A separate script scan of the import header, rt.py, and native.py found no other missing name.
  • just ci passes (ffi fixtures: 73 passed, 27 ignored as nightly or sanitizer jobs).

Caused by

None: the bug is as old as the module's scheme of declaring stubs under TYPE_CHECKING, and no fixture had a top-level name starting with _ until now.

Follow-ups

  • quest/m1/ffi-export/cpp-reserved-names.md [S]: the fixture's pub fn __all__ renders in C++ as api::__all__, and C++ reserves every identifier containing __. It compiles today. Recommended fix, recorded in its Plan: escape the name and claim it through named in ffi/src/cpp.rs, then turn on -Wreserved-identifier in the strict build. No open decisions. Open PR #919 covers names that headers define as macros, not this.

Models

Size: S
Implementer: Claude, claude-opus-5-5, unknown effort (arm A)
Implementer: Claude Code, claude-opus-5-5, unknown effort (iterate, S)

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: 2b83610

  • Base: origin/main at 24ff182

  • Risk: high, Generated Python public API names change for consumers exporting reserved names, so affected bindings may require regeneration and redistribution.

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1f99c-c9ce-7521-8fc0-e3c3ce2582c9, round 4

  • Score: 10/10, The incremental change reserves the missing _inspect alias and extracts imports without changing their output, with no new findings.

  • Raised the risk to high: Generated Python public API names change for consumers exporting reserved names, so affected bindings may require regeneration and redistribution.

  • Rounds: 4; first round 1 (medium 1); findings 1 (medium 1): fixed 1, refuted 0, open 0

  • Findings: none

Summary by CodeRabbit

  • New Features
    • Exported functions with leading underscores are now available across supported language bindings, with reserved names adjusted where needed.
    • Generated Python modules now provide __all__ lists that identify their public exports.
  • Documentation
    • Clarified how Python exports are represented and documented name handling for C++ bindings.
  • Tests
    • Added coverage for Python wildcard imports and underscore-prefixed exports.

Summary by CodeRabbit

  • New Features

    • Generated bindings now expose additional public functions across supported languages, including functions with leading underscores or reserved names, using language-appropriate names.
    • Python modules now provide an explicit list of exported names, including eligible underscore-prefixed exports and public runtime classes.
  • Bug Fixes

    • Resolved conflicts between generated names and Python’s reserved module attributes so exported names remain accessible.
  • Documentation

    • Updated Python export documentation and added guidance on reserved names in C++ bindings.

@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The Python generator now emits an __all__ list and updates name claiming for module-level names and generated runtime types. The API fixture adds two underscore-named functions, with declarations added to language surfaces and checks added to Python tests.

Changes: Export Names

Layer / File(s) Summary
Python export listing
ffi/src/python.rs, ffi/surface/api/python.pyi, ffi/surface/moq/python.pyi, ffi/README.md, quest/m1/ffi-export/python-private-names.md
The generator builds and emits __all__, reserves imported and selected dunder names, and records generated runtime types. The Python documentation describes the export list, the stubs declare it, and the prior Python naming note is removed.
Underscore API surfaces
ffi/fixtures/api/src/lib.rs, ffi/surface/api/*, ffi/tests/api.py, quest/m1/ffi-export/cpp-reserved-names.md
The fixture adds _deprecated and __all__ functions. Language surfaces add corresponding declarations, and the C header contract value changes. Python tests check star-import names and function results. A C++ reserved-name note describes a name-escaping proposal.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Merge Risk: Minimal · up to 2b836

This change adds an explicit __all__ to generated Python modules and underscore-named exports to the API fixtures. I found no concrete merge-blocking risk. The C++ reserved-identifier issue is tracked as a follow-up.

Security architecture risk: Minimal · up to 2b836

The inspected change makes existing public exports explicit and protects Python-reserved names. It does not establish new privileges, weakened isolation, or a bypass of native-library compatibility checks.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The export-policy change applies to generated Python modules and their importers. Wildcard imports gain access to already-public underscore-named bindings and lazy constants; this changes import behavior rather than granting a new execution privilege.

Trust Boundaries and Controls

  • observed — Export membership comes from ABI-driven generator state, not a new runtime request input. Reserved names such as the native handle, loader, constant table, and imported aliases are excluded. Loading still selects the adjacent native library and performs the existing build-contract check; that check is compatibility enforcement, not library authentication.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title says a Python module lists its exports in all. The changeset adds a sorted all to generated Python modules, built from the generator's name table, which matches the main change. The …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title says a Python module lists its exports in all. The changeset adds a sorted all to generated Python modules, built from the generator's name table, which matches the main change. The title is a fix(ffi) conventional-commit style and is concise. It does not mention the reserved-name escaping (appending underscores to dunder-conflicting names), but that is a supporting aspect, not the primary change.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5c039372-e237-4b17-8207-c3d8446c4aec
Commits

Reviewing files that changed from the base of the PR and between 0e1fb0b and 2b83610.

Files selected for processing: 1

  • ffi/src/python.rs

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.

kixelated and others added 3 commits November 20, 2026 07:31
A Rust pub fn whose name starts with `_` declared a stub pyright strict
reported as an unused private function. Listing every export in __all__
counts each as used, and makes `from <prefix> import *` reach it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Python: a pub name starting with an underscore passes pyright strict fix(ffi): a Python module lists its exports in __all__ Nov 20, 2026
kixelated and others added 3 commits November 20, 2026 08:05
A pub fn __all__ overwrote the module's export list. Every __x__ name
now takes a trailing _, as other taken names do. Adds a follow-up for
C++, which reserves such names too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 20, 2026 16:21
@kixelated

Copy link
Copy Markdown
Collaborator Author

Maintainer decision (taken while the maintainer is away, on their instruction to choose what is strongly recommended): /quest-merge. Keep the follow-up quest/m1/ffi-export/cpp-reserved-names.md, escaping reserved names in C++ and turning on clang's -Wreserved-identifier in the strict build.

(Written by Opus 5.5)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated enabled auto-merge November 20, 2026 16:22

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — one minor gap inline.

Approach: sound. It builds __all__ from the generator's own name table rather than a second list, which fixes pyright's reportUnusedFunction at its cause, and reserving the whole __x__ family in Gen::claim is the general fix for the __all__ clash, not a special case for one name.

Reviewed changes

Reviewed the full diff at 0e1fb0b: the generator change, the fixture items, the import * test, the regenerated surfaces, and the quest files.

  • __all__ in the module head: g.names minus TAKEN, plus the runtime's Duration, ForeignReader, Mut, and Waker, all of which runtime/rt.py defines unconditionally.
  • One name table: Bound (only when a range bound exists), ArrayU8x<n>, and the range classes are now recorded in g.names where they are emitted.
  • Dunder reservation: a module-level __x__ takes a trailing _, and a name ending in ___ counts as free, so the rename terminates.
  • Fixture and test: pub fn _deprecated and pub fn __all__, plus an exec("from api import *") check against the module's real namespace. That check catches a name listed but never bound, and a name bound but not listed. For moq, pyright's reportUnsupportedDunderAll in just moq covers the first case.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Comment thread ffi/src/python.rs Resolved
@moq-bot
moq-bot Bot disabled auto-merge November 20, 2026 16:29
@moq-bot

moq-bot Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

quest land watch: ended this pull request's pick, disarming its auto-merge, since its head 0e1fb0b is held by finding by pullfrog[bot]: TAKEN misses _inspect, which the head imports (import inspect as _inspect, line 222) and runtime/rt.py:139 calls (#925 (comment)). Push a fix, or answer each by a thread reply or a comment linking its review or decision, then land it again.

(written by quest land watch)

The generated module imports inspect as _inspect for the runtime's
isawaitable check, but TAKEN missed it. Hoist the imports into IMPORTS and
test that every alias it binds, and every _ name the runtime binds at module
level, is in TAKEN.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the delta from 0e1fb0b to 2b83610, which answers the earlier _inspect note.

  • _inspect reserved: added to TAKEN, so a pub fn _inspect exports as _inspect_ and no longer rebinds the alias runtime/rt.py reads.
  • IMPORTS const: the import header moved out of the head format string with identical output.
  • module_names_taken test: checks that every import alias and every module-level _ name in rt.py is in TAKEN. I checked the current names (_Future, _Wakeup, _wakeups, _resume, _drains, _wakeup, _Pollable, _Const, _Dataclass, _Record) against the list.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 4b095a7 Nov 20, 2026
9 checks passed
@kixelated
kixelated deleted the quest/m1/ffi-export/python-private-names branch November 20, 2026 17:34
@codspeed

codspeed Bot commented Nov 20, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 3.57%

1 warning

⚠️ 27 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 1 improved benchmark
❌ 2 regressed benchmarks
✅ 82 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ stamped 1.4 s 1.6 s -16.45%
❌ woken 9.2 s 10.5 s -12.49%
⚡ loop 10.8 s 8.8 s +22.62%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing quest/m1/ffi-export/python-private-names (2b83610) with main (24ff182)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant