Skip to content

fix(ffi): write the generated crate's paths relative to its export - #917

Merged
kixelated merged 4 commits into
mainfrom
quest/m1/ffi-packaging/relocatable
Nov 20, 2026
Merged

kixelated merged 4 commits into
mainfrom
quest/m1/ffi-packaging/relocatable

Conversation

@kixelated

@kixelated kixelated commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Claims quest/m1/ffi-packaging/relocatable.md.

Problem

The generated crate's version hashes its Cargo.toml, which named the facade by absolute path, and cargo hashes that version into every symbol: two checkouts of one commit made different libraries.

Approach

glue.rs writes each path in the generated manifest relative to the export: the facade, each [patch] path, and each path crate the glue reaches (Paths). Two exports of one commit at other paths are now byte for byte equal while each keeps its position relative to its crate; an export at another depth writes another relative path and so another version hash (documented in the quest and ffi/release/README.md, after Pullfrog's finding).

It keeps the absolute path where a relative one would read wrong: an export whose own path holds a symlink (cargo joins .. lexically, the file system does not), or one on another Windows drive from the crate.

The quest is not finished, and its premise was half wrong. Cargo hashes a path package that lies outside the workspace root into its metadata by its absolute path, so the facade's own symbols still differ between checkouts, whatever the manifest says. I measured it: two copies of the repo at other paths, exports equal, cargo build --release with --remap-path-prefix on both: every symbol equal except the five statics of the facade crate. So reproduce still cannot move the source. The quest is rewritten to what remains (a fixed build path in package.py, which also covers the Windows GNU dlltool path) with a recommendation and the reason it needs CI on macOS and Windows to try.

Impact

  • Public API: none. Wire: none.
  • The generated Cargo.toml text changes (relative path = "../..."), so every export's version hash (0.0.0+<hash>) and each symbol hash changes once. No migration.
  • The tail of ffi/release/README.md's "Not yet" bullet now says what is fixed and what is not.

Validation

  • export_is_relocatable (new): exports one crate with a path dependency from two checkouts at different depths and fails unless every file of the two exports matches. Checked that it fails with the relative path switched off.
  • Unit tests for the path math, the [patch] rewrite, the crates dependency rewrite, and the symlink and .. fallbacks.
  • The api fixture from two copies of the repo at other paths, the export beside each: diff -r of the exports is empty (before, Cargo.toml and Cargo.lock differed). Built both with the remap flags: the .a files differ only in the five facade statics, as above.
  • just ci passes locally (clippy, fmt, the exporter's tests).
  • Not run: macOS, Windows (a different drive's fallback is unit-tested only on Windows), and the release workflow, which this does not touch.

Alternatives

  • Dropping the relative path (rejected by the maintainer on 2026-11-19, recorded in the quest).
  • Always relative from the physical path: wrong when the export is spelled through a symlink such as macOS /tmp, since cargo joins .. lexically; it falls back to absolute instead.

Decisions

Taken on my recommendation, the user being away:

  • Say plainly that the relocation holds only while the export keeps its position relative to the crate (Pullfrog's review finding, accepted: the old absolute path made the manifest depth-independent, the relative one does not). Rejected: making the path depth-independent, which would need the fixed-name link the quest already plans.
  • Keep the absolute path in the fallback cases above rather than refuse: the quest's own Plan says to fall back where no relative path exists.
  • Leave cli.rs's other absolute paths (--with, --baseline, the roots) alone: they are inputs of the check, never written into the export, and the test shows the export unchanged by them.
  • Land the exporter half now and rewrite the quest to the builder half, rather than hold the PR for a staging design I cannot test on Windows or macOS here. The quest is resized [M].
  • The remaining design, recommended in the quest: a fixed-name link to the common ancestor of the crate's workspace and the export, through which package.py build runs cargo. Alternative: a target directory under a fixed absolute root, which fixes only the dlltool path.

Follow-ups

None new: the quest's remaining work is rewritten in place.

Models

Size: M
Implementer: Claude Code, claude-sonnet-5-5, unknown effort (arm B)

🤖 Generated with Claude Code

(Written by Claude Sonnet 5.5)

Review

Rewritten by quest review after each round.

  • Head: 0680f68

  • Base: origin/main at dc27f60

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1f8b2-9d19-7a11-ae8f-6f86805a10ce, round 3

  • Score: 10/10, The incremental changes accurately clarify the relative-position requirement in documentation and introduce no new issues.

  • Rounds: 3; first round 2 (medium 2); findings 2 (medium 2): fixed 2, refuted 0, open 0

  • Findings: none

Summary by CodeRabbit

  • New Features

    • Generated exports now use relative paths for manifests and dependency references when paths can be safely represented. Absolute paths are retained when exports involve symlinks or paths without a shared root.
  • Documentation

    • Updated guidance explains path-related limitations: rebuilds may need the crate at the same location, and Windows GNU exports still have a fixed-directory requirement. Reproduction guidance also notes potential differences tied to checkout and target-directory paths.

Summary by CodeRabbit

  • New Features

    • Generated exports now use relative paths for the facade, patches, and reachable local crates when possible, making exports consistent across checkouts with the same relative layout.
    • Paths remain absolute when symlinks or different filesystem roots prevent relative paths.
  • Documentation

    • Clarified how export location, checkout depth, and build paths can affect reproducibility and generated symbols.

@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The export generator now writes relative facade, dependency, and patch paths when the export path permits it. Tests compare exports from different checkout depths. Documentation describes cases where absolute paths and path-dependent symbols remain.

Changes: FFI export relocation

Layer / File(s) Summary
Path normalization and relative paths
ffi/src/glue.rs
New helpers normalize paths and calculate relative paths when roots are shared and the export path has no symlinks. Tests cover lexical normalization and Windows path handling.
Manifest path relocation
ffi/src/glue.rs
Generation applies relocated paths to facade and reached-crate dependencies and patch/profile entries. Tests check relative manifest paths and absolute-path fallback.
Export validation and documentation
ffi/tests/fixtures.rs, ffi/release/README.md, ffi/src/cli.rs, quest/m1/ffi-packaging/relocatable.md
A fixture test asserts that exports from different checkout depths have identical paths and bytes. The documentation describes path relocation and remaining path-dependent symbols.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Possibly related PRs

  • kixel-ai/rust2#874: Both changes address how generated crate paths affect reproducible output.
Merge Risk: Minimal · up to 0680f

Generated FFI export manifests now use relative paths when it is safe, so exports of the same commit from different checkouts are identical. Documentation states the remaining path-dependent symbol differences. No merge-blocking risk was identified.

Security architecture risk: Minimal · up to 134ae

The inspected change preserves existing dependency sources and validation-before-write behavior. No introduced or worsened security, ownership, or failure-containment risk was established. Relocatable exports still do not guarantee reproducible compiled libraries after moving the source.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The affected security-relevant outcome is dependency code selected when a developer or CI job builds generated bindings. The inspected changes add path transformation but no new build-execution mechanism or credential authority; project and Cargo configuration remain the dependency inputs.

Trust Boundaries and Controls

  • observed — The relative-path eligibility check compares the resolved export directory with its lexical absolute spelling. When they differ, paths retain their prior spelling. The transformation itself performs path calculations and filesystem reads, not writes or command execution.

Resilience and Maintainability Implications

  • observed — Publication remains unchanged from the comparison base: stale owned files are pruned, then files are written individually using temporary-file creation and rename. Unchanged content preserves its modification time. Interruption can still leave an export-wide mixed generation; transactional publication and concurrent-writer coordination are not guarantees introduced by this PR.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title "fix(ffi): write the generated crate's paths relative to its export" matches the main change. The glue.rs change makes Crate::generate write manifest path values relative to the export, and …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title "fix(ffi): write the generated crate's paths relative to its export" matches the main change. The glue.rs change makes Crate::generate write manifest path values relative to the export, and the PR objectives state the same goal. The title is specific and uses a conventional-commit prefix. It does not mention the caveat that absolute paths remain for symlinked exports or different drives, but it does not need to cover every detail.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8606ba69-abb4-4ba2-ac48-8fcd9b799e35
Commits

Reviewing files that changed from the base of the PR and between 134aea2 and 0680f68.

Files selected for processing: 2

  • ffi/release/README.md
  • quest/m1/ffi-packaging/relocatable.md
Files skipped from review as they are similar to previous changes: 1

  • ffi/release/README.md

Limit details: You’ve used all 10 included reviews currently available.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title A release rebuilds the same from any checkout path fix(ffi): write the generated crate's paths relative to its export Nov 20, 2026
…plain

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 20, 2026 11:28
@kixelated
kixelated enabled auto-merge November 20, 2026 11:28

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

The code is sound, but the docs this PR rewrites now state a reproducibility property the change takes away: moving the export with the crate fixed no longer gives the same libraries.

Approach: sound for what it claims. Writing the generated manifest's paths relative to the export, and falling back to absolute where .. would read differently to cargo than to the file system, is the right way to make the export's own bytes independent of the checkout. But it moves the dependency on location around rather than removing it, and the docs don't say so.

Reviewed changes

This run reviewed the full PR: the relative-path writer in glue.rs, its tests, and the README and quest rewrites.

  • Paths in glue.rs: names the facade, each [patch] path, and each self.crates path dependency relative to the export. It keeps absolute paths when the export's spelled path holds a symlink, or when no relative path exists.
  • TOML round-trip: self.patch and each dependency's keys are now parsed and re-serialized rather than pasted in, so key order inside the reached-crate inline tables changes once.
  • export_is_relocatable: exports one crate with a path dependency from two checkouts at different depths, with the export at <checkout>/out in each, and compares every file.
  • Quest and README: the quest is resized to [M] and narrowed to the builder half (one fixed build path); the README's "Not yet" bullet now says what is fixed and what is not.

I checked the places that consume the manifest. package.py build and web both build export_dir/Cargo.toml in place, so relative paths resolve. The out/export/Cargo.toml copy is only compared, never built. Both CI rebuilds keep the export at the same depth relative to source/: export again, and web/export vs web-again/export. So CI is unaffected.

⚠️ The glue's version hash now depends on where the export sits relative to the crate

Before, the manifest named the facade by absolute path. Two exports of one checkout at different export paths wrote the same Cargo.toml, so they got the same 0.0.0+<hash> version and the same glue symbols (the old quest measured this on Linux and for the wheels). Now path = "../.." versus "../../../source/api" changes with the export's depth, so the version hash changes too. So does every glue symbol.

The facade's own symbols still hash its absolute path, so the built libraries gain nothing yet. The net change for binaries: "crate fixed, export anywhere" becomes "crate fixed, export at the same relative position". That is fine as a step toward the fixed-link design, but the README and quest should say it, and the quest's next implementer should not build on "the export can still move".

Technical details
# Document that the export's position relative to the crate is now part of the build's identity

## Affected sites
- quest/m1/ffi-packaging/relocatable.md:24-25: "So `reproduce` can still only move the export, as it could before." No longer true: moving the export to another depth relative to the crate changes the manifest, the version hash, and every glue symbol.
- ffi/release/README.md:199-201: "A rebuild must still keep the crate at the same path" leaves out that the export must also keep its path relative to the crate (or simply its own path, as a rerun does).

## Required outcome
- Both docs say that, until the builder builds through one fixed spelling, a rebuild keeps both the crate's path and the export's path (or at least the export's position relative to the crate).
- The quest's recommendation section still reads correctly: the fixed-name link to the common ancestor fixes the relative position too, so it covers this.

## Open questions for the human
- Is giving up "export anywhere, crate fixed" acceptable for consumers before the builder half lands? CI keeps the export at the same depth in both `reproduce` paths (`export`, and `web/export` vs `web-again/export`), so CI is unaffected.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Comment thread quest/m1/ffi-packaging/relocatable.md Resolved
@moq-bot
moq-bot Bot disabled auto-merge November 20, 2026 11:38
@moq-bot

moq-bot Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

quest land watch: ended this pull request's pick, disarming its auto-merge, since its head 134aea2 is held by finding by pullfrog[bot]: This is no longer true after this PR. With the crate fixed, an export at another depth now writes a different relative ` (#917 (comment)). Push a fix, or answer each by a thread reply or a comment linking its review or decision, then land it again.

(written by quest land watch)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the docs-only delta since the prior pullfrog review.

  • README — the "Not yet" bullet now says the relocation holds while the export keeps its position relative to the crate, and that a rebuild keeps the export in place beside it.
  • Quest — relocatable.md records that an export at another depth writes another relative path and hash, and that reproduce keeps the export's depth below the common ancestor.

This addresses the earlier finding about the export's position becoming part of the build's identity.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit c2dfc5e Nov 20, 2026
9 checks passed
@kixelated
kixelated deleted the quest/m1/ffi-packaging/relocatable branch November 20, 2026 12:30
kixelated added a commit that referenced this pull request Nov 20, 2026
… writes

#909's sdist keyed each local package by its absolute path and #917 made the generated
manifest spell them relative to the export, so the sdist refused every export on a
checkout without a symlink, and unreached_types expected the absolute spelling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Nov 20, 2026
Since #917 the generated manifest names each local package relative to the
export, which the sdist's relocation, keyed by absolute paths, refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant