Cache fingerprint: dead fast-path, fail-open reuse, adopted caches never re-validate, and PYTHONHASHSEED is recorded wrong

[adamjohnwright]

Four defects in the fingerprinting I added in #56. All verified.

1. stored == current can never be True (pathway_generator.py:127)

_write_cache_fingerprint injects provenance; _cache_fingerprint() does not produce that key. Against a cache generated minutes earlier by the same code and env:

stored keys:  ['db_identity', 'env', 'provenance', 'reactome_release', 'src_sha256']
current keys: ['db_identity', 'env', 'reactome_release', 'src_sha256']
stored == current ? False
difference: {'provenance': ('generated', None)}

So every reuse falls into the field-by-field diff, and when nothing has changed it logs something false:

INFO Cache fingerprint differs only in an unknown Reactome release; reusing the cache.

— the release is 97 on both sides. Worse, the safety property inverts: instead of "any difference rejects the cache" it becomes "only the four hand-enumerated differences reject it". Any field added to _cache_fingerprint() in future is written to disk and never compared.

2. An adopted cache never re-validates against source (:99-119, :131-137)

run1 (no fingerprint, csv present) reusable: True  -> provenance='adopted', src_sha256=None
run2 with COMPLETELY DIFFERENT source hash reusable: True
run3: True

Reachable from an ordinary Ctrl-C: any interruption between the first to_csv and the fingerprint write manufactures an adopted cache that is then immune to source-change detection forever. db_identity changes are still caught, so the hole is source-only.

3. A corrupt/partial cache with a matching fingerprint is reused silently

The fingerprint attests to code/env/db/release, not to the integrity of the CSVs, and the writes at :272 and :302-303 are plain to_csv to the final path (no tmp+rename). Truncating decomposed_uid_mapping.csv to 1 row and reaction_connections.csv to 2:

INFO Loading cached reaction connections from .../cache/reaction_connections.csv
INFO Loading cached decomposition from .../cache/decomposed_uid_mapping.csv
INFO Network contains 63 edges        (was 61)

No warning. The validator then reports 11/11 on the wrong network.

4. PYTHONHASHSEED is read from os.environ, not the interpreter (:40, :68)

Setting it after interpreter start has no effect, but the fingerprint records it anyway, so a non-reproducible run is stamped as reproducible and two such caches compare equal.

actual hash randomization active? -7152368689495947875   # differs every run
fingerprint records PYTHONHASHSEED = 0

Measured cost of the unseeded case (R-HSA-72163, LNG_COMPLEX_AS_NODE=0 LNG_HANDOFF_EDGES=1): two runs of the identical configuration differed by 1,494 canonical edges each way out of 342,365.

bin/create-pathways.py:13-15 does it correctly by re-exec; examples/generate_pathway_example.py has no pinning. Fix is sys.flags.hash_randomization / sys.hash_info, not os.environ.

Confirmed working (do not re-check): a corrupt fingerprint.json is rejected; an unknown release does not spuriously invalidate; a db_identity change rejects even an adopted cache; a fresh empty cache dir is correctly not treated as an adoption; and with the seed genuinely pinned before interpreter start, generation is reproducible and order-independent (342,365 edges, 0 differing, across separate processes and pathway orderings).

[adamjohnwright]

Correction to point 4's scoping, verified after filing.

The defect is real: _cache_fingerprint() reads PYTHONHASHSEED from os.environ, so a run that sets it in-process (after interpreter start, where it has no effect) gets stamped as reproducible when it is not. The fix — read sys.flags.hash_randomization — stands.

But the specific claim that the v97 catalog regeneration was hash-randomized is wrong. That run passed PYTHONHASHSEED=0 in the shell environment as well as setting it in-process, so the interpreter did start with it:

$ PYTHONHASHSEED=0 poetry run python -c "import sys; print(sys.flags.hash_randomization, hash('x'))"
0  -3368204716105132637     (x3, identical)

$ poetry run python -c "os.environ['PYTHONHASHSEED']='0'; ..."
1  -4474459696214245031
1   6448895106550650079     (in-process only: ineffective, as reported)

Empirical confirmation — regenerating R-HSA-69620 and comparing canonical structure (edges relabelled by stable id, since uuid4 node ids differ by design):

original v97 run : 3836 edges  canonical sha bbaa9ba68bab7180
fresh repeat run : 3836 edges  canonical sha bbaa9ba68bab7180
IDENTICAL

So the catalog at scratchpad/v97 is reproducible and safe to benchmark against. Only the fingerprint's recording of the seed is unsound.