silicon: the bitstream reuse key names prjxray by the commit of the enclosing t27 checkout (pip freeze), not by its sources -- and pays 1.15 s for it #8811
Description
Part of #8737 (hot reload), epic #8095. Found while profiling a reuse hit after #8808.
Problem
-
What the key reads:
t27c siliconputspip freezeof the toolchain venv into the bitstream reuse key (bitstream_reuse.t27). On this Mac that prints:-e git+https://github.com/gHashTag/t27@e804cd062af9bd4424a8579ea54b1f21381580de#egg=prjxray&subdirectory=build/fpga/openxc7/prjxray -
Where that commit comes from: prjxray is an editable install with no
.gitof its own, so pip names it by the HEAD of the git checkout that encloses it: the main t27 checkout,/Users/playom/t27, ate804cd062(2026-09-05). -
The defect cuts both ways:
- Any commit, pull or checkout in that unrelated checkout changes every bitstream key and orphans every cached bitstream, although no tool changed.
- An edit to prjxray's own Python, which
fasm2frames.pyimports (prjxray.fasm_assembler,prjxray.db, ...), never reaches the key. Only the bitstream audit would catch it, one hit in ten.
-
Cost:
pip freezealso takes 1.15 s of every reuse hit, the largest cost left after silicon: a bitstream reuse hit takes tool digests from a stat-keyed cache, 4.3-8.9 s -> 1.5-2.6 s (Closes #8805) #8808.
Change
- The venv part of the key becomes:
- the python directory and the name of every
*.dist-infoand__editable__*entry in itssite-packages, which carry each distribution's name and version; - the sha256 of each of prjxray's own
prjxray/*.pysources, by content.
- the python directory and the name of every
BIT_RECIPE1 -> 2, because the key's composition changed. Every cached bitstream misses once and is rebuilt.
Acceptance criteria
bitstream_reuse.t27names the new venv input andBIT_RECIPE == 2, with its invariant updated.- A reuse hit of
t27c silicon specs/fpga/ternary_link.t27 --top fpga/verilog/ternary_link_jtag.v --skip-hardware --pnr-seed 11runs nopipprocess and takes less time than with silicon: a bitstream reuse hit takes tool digests from a stat-keyed cache, 4.3-8.9 s -> 1.5-2.6 s (Closes #8805) #8808 alone. - On a scratch toolchain root, appending a comment to a copied
prjxray/*.pychanges the key, a MISSING decision. - Hand-written foreign lines do not grow.
Boundary
specs/verified/bitstream_reuse.t27
User Scenarios
- Given a cached bitstream exists for a design, when the user runs
t27c siliconagain without any toolchain changes, then the reuse key is computed without invokingpip freezeand the bitstream is reused (cache hit). - Given a developer edits a file under
prjxray/*.pyin the toolchain venv, whent27c siliconruns for the same design, then the reuse key changes and the bitstream is rebuilt (cache miss). - Given the user pulls new commits or checks out a different branch in the enclosing t27 repository, when
t27c siliconruns for an unchanged design, then the reuse key remains stable and the cached bitstream is reused.
Requirements
FR-001: The bitstream reuse key's venv component MUST be derived from the Python directory path, the name and version of every *.dist-info and __editable__* entry in site-packages, and the SHA-256 hash of each prjxray/*.py source file by content.
FR-002: BIT_RECIPE in specs/verified/bitstream_reuse.t27 MUST be incremented from 1 to 2 with its invariant updated to reflect the new key composition.
FR-003: A reuse hit for t27c silicon specs/fpga/ternary_link.t27 --top fpga/verilog/ternary_link_jtag.v --skip-hardware --pnr-seed 11 MUST complete without spawning any pip process and MUST take less wall-clock time than the same command with only #8808 applied.
FR-004: On a scratch toolchain root, appending a comment to a copied prjxray/*.py file MUST cause the reuse key to change, resulting in a MISSING decision and rebuild.
FR-005: The reuse key MUST NOT include any hand-written foreign lines that grow with unrelated repository activity.
Activity