Repository navigation
ESM3 residue-code alignments, native Stage-2 kernel and scalable clustering - #6
Merged
Merged
Conversation
DataRoot downloaded rcsb/rcsb-esm at whatever revision main pointed to, so a new upload would have silently changed the residue embeddings that the aggregator and every downstream calibration read as frozen features. Pin ESM_REPO_ID = "rcsb/rcsb-esm" at ESM_REVISION = 636acb42 (40-hex commit) and check the weights file against ESM_WEIGHTS_SHA256 = 5ead5a13 (the blob that rcsb/rcsb-esm@636acb42 and the EvolutionaryScale esm3-sm-open-v1 snapshots all resolve to). verify_esm_weights() checks the cached file by its blob name: in the Hugging Face cache layout the snapshot file is a symlink to blobs/<sha256>, so the check is free. Only when the layout hides the name (local_dir downloads, copied snapshots, filesystems without symlinks) is the 2.8 GB file hashed, once per process (about 1.5 s on an M1 Max). A file that was not in the cache before DataRoot resolved the snapshot - the download-models case - is hashed once after its download. Nothing is written to HF_HOME or anywhere else: no marker files. DataRoot keeps no half-built singleton when the check fails. FoldMatch and esm 3.2.3 fetch nothing else from Hugging Face when they load ESM3 and embed sequences or structures (instrumented on the fixture FASTA and 2uzi.cif: only rcsb/rcsb-esm and rcsb/rcsb-embedding-model). Tests: tests/test_release_pins.py EsmWeightsPinTest (pins; blob-name check without hashing; wrong blob refused; hidden layout hashed once and refused on wrong content; full hash behind a blob name; DataRoot passes the pinned revision and hashes only a fresh download; nothing written; the local cache holds the pinned blob). Behaviour change: none (the same weights as before). Review follow-up: a weights mismatch raises EsmWeightsMismatchError, a ValueError subclass (RuntimeError before), so fm-search, fm-embedding and inference report it as one "Error: ..." line with its remedy instead of a traceback. Test: EsmWeightsPinTest.test_mismatch_is_reported_as_a_clean_cli_error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
align_candidates checked that subject_db_size was given with a max_evalue, but not that it was positive. SequenceStore.total_residues() is COALESCE(SUM(length), 0), so an empty sequence store arrives as 0, slips past the "is None" guard, leaves every database E-value unset, and the filter then fails deep in the result loop with "'<=' not supported between instances of 'NoneType' and 'float'". A negative size failed inside karlin_altschul with "sequence lengths must be positive". Raise an explanatory ValueError up front instead, before any subject sequence is fetched, naming the size, the likely cause (an empty or unbuilt sequence store) and the two ways out (rebuild, or max_evalue=None / --max-evalue inf). Without a max_evalue nothing changes: a zero size still ranks hits with no database E-value. Tests: EmptyDatabaseTest (ported from js/dev-3di-alignments and extended with the up-front check). Behaviour change: error message only, except that a query against an empty sequence store with an E-value filter (the CLI default) used to return no hits silently and now refuses with this message. Review follow-up: the check is now check_evalue_search_space(max_evalue, subject_db_size), which align_candidates calls first and which callers can run on their own before any expensive work (the CLI does, before ESM3, from G9 on). Its message no longer says just "Rebuild it with 'fm-search build sequences'": a build refuses an existing database, so it says to delete the database's files and rebuild, or to pass --max-evalue inf. Tests: EmptyDatabaseTest checks that advice and test_check_evalue_search_space. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_align took identity_aln and identity_shorter from biotite's
get_sequence_identity, called twice per hit, which loops over every
alignment column in Python with an np.unique per column (about 1.1 ms
for a 300-column alignment). _derive_alignment_extras already counts
the same matches with numpy (n_ident: both residues present and equal;
the sanitized strings map 1:1 onto the protein codes), so divide that
by the alignment length ('all') and by the shorter full sequence
('shortest'), as biotite does. Both sides divide Python ints, so the
values are the same floats, not merely close ones.
Tests: IdentityVectorisedTest checks identity_aln and identity_shorter
for exact float equality against biotite's get_sequence_identity and a
naive per-column loop, on the mmseqs fixture pairs (comp-bias off, 1.0
and 0.5, which also covers the positional-query branch) and on 1,000
seeded random alignments (standard, ambiguous and sanitized letters;
mutated copies with indels and unrelated pairs).
Speed (serial align_candidates, best of 15 runs, alternating base and
patched trees on an Apple M1 Max under load; bench_identity.py): mmseqs
fixture pairs x20 1.611 -> 1.401 s (1.15x), 300 x 300-aa homologs
1.286 -> 0.970 s (1.33x), 40 x 1,000-aa homologs 1.511 -> 1.373 s
(1.10x).
Behaviour change: none. align_candidates output (every AlignmentMetrics
field at full precision, and the rendered TSV) is byte-identical to
the base on the mmseqs pairs and 400 random pairs in 45 configurations.
Review follow-up: the equality tests above also pass with
get_sequence_identity restored, so they could not catch a revert.
IdentityVectorisedTest.test_align_never_calls_biotite_identity runs
_align on mmseqs fixture pairs (comp-bias off and 1.0) with
biotite's get_sequence_identity patched to raise, so it fails if _align
calls it at all (checked: 20 of 20 subtests fail with this commit's
alignment.py change reverted).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CoreBatchWriter (and the CSV, tensor, DataFrame and JSON writers built on it) and ParquetBatchWriter unpacked each prediction as exactly (embeddings, ids). A module that returns per-item payloads after those two elements (the residue-code and query-encoding outputs planned for P5) would make every writer fail with "too many values to unpack". The writers now take prediction[0] and prediction[1] and drop anything after them, so they write exactly what a 2-tuple gives them. The Parquet schema stays id + embedding. Tests: tests/test_batch_writer.py (CoreBatchWriter 3-tuple test; 2-, 3- and 4-tuples give identical output for the core, CSV, tensor, JSON and Parquet writers). Behaviour change: none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three torch.load calls in the package relied on the PyTorch default for weights_only: the ESM3 weights and the structure encoder in utils/esm/loaders.py, and load_residue_embedding in utils/data.py (.pt residue embeddings supplied by the user). The default became True only in PyTorch 2.6, and FoldMatch does not pin torch, so on older versions these calls would unpickle arbitrary code. Pass weights_only=True explicitly, as get_aggregator_model and stream_embeddings already did. Tests: tests/test_release_pins.py TorchLoadWeightsOnlyTest (every torch.load call in the package passes the literal weights_only=True; load_residue_embedding still loads a tensor and refuses a pickled object). The static check fails on the base at utils/data.py:28. Behaviour change: none on PyTorch >= 2.6 (the default already applied); on older PyTorch, .pt files holding pickled Python objects are refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ates_ignored SequenceStore.create now records the corpus totals in a `meta` table (key, value) at the end of the build: entry_count (rows stored), total_residues (SUM(length) over the stored rows, i.e. the database search space) and duplicates_ignored (FASTA records dropped because their id repeated an earlier one). They are computed once from the stored rows. New SequenceStore.meta() returns the three values. count() and total_residues() now read them from the meta table instead of scanning every row. Stores built before this change have no meta table: meta() falls back to COUNT(*) and SUM(length) over the rows, as count() and total_residues() did before, and reports duplicates_ignored as None, because it cannot be recovered. Tests: tests/test_sequence_store.py (SequenceStoreMetaTest: counts with duplicates and min_res_n, the fixture FASTA (2 entries, 303 residues), an empty store, readers use the meta table, an old store without the table, force rebuild, missing store). Behaviour change: none (same values; old stores still readable). Review follow-up: the meta table doubles as a completion marker. create() commits the rows in batches and fills meta last, so a build that stops part way leaves a store whose meta table has no totals; meta() (and so count() and total_residues()) now raises ValueError "... is incomplete ..." for such a store instead of reporting the rows it holds as the database. Stores without a meta table (built before this commit) still fall back to the rows. Test: SequenceStoreMetaTest.test_interrupted_build_is_refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EmbeddingComputer named its shards emb_<id>-<rank>.parquet with
id = SLURM_JOB_ID or, outside SLURM, a fresh token_hex(16) drawn by each
process. Lightning launches the other ranks of a multi-GPU run as child
processes that re-run the command, so every child drew its own token,
wrote its shard under a name rank 0 never globbed, and rank 0 built the
database from its own shard only: rank > 0 embeddings were silently lost
in every non-SLURM multi-GPU run. Reproduced on the base commit with a
2-rank CPU build of the fixture FASTA: shards emb_2a78...-1 and
emb_b77e...-0, FAISS index with 1 of the 2 entries.
New foldmatch/utils/dist.py:
- launch_rank(): rank from the launch environment (RANK, else the largest
of SLURM_PROCID, NODE_RANK, LOCAL_RANK), usable before the process group
exists.
- run_id(): FOLDMATCH_RUN_ID if set (validated: [A-Za-z0-9][A-Za-z0-9._-]*);
else under SLURM {SLURM_JOB_ID}_{SLURM_STEP_ID or "batch"}_{SLURM_RESTART_COUNT or 0};
else a fresh token_hex(8), which only rank 0 may draw. A rank > 0 without
an id raises ValueError before any model is loaded.
- exported_run_id(): exports the id as FOLDMATCH_RUN_ID around
trainer.predict, so the ranks Lightning launches inherit it, then
restores the previous value (cleared when it was not set).
EmbeddingComputer:
- shards are emb_{run_id}-{rank}.parquet; shard_files() globs its own id and
keeps only exact matches (an id "abc" never reads "abc-1"'s shards);
- after a FASTA pass rank 0 checks, from the Parquet footers and before
anything is built, that the shards hold exactly one embedding per entry
passing min_res_n; a missing rank or a stale shard of an earlier run with
the same id now raises ValueError instead of being merged;
- remove_shards() deletes this run's shards.
CLI (D13): query sequences deletes its query shards after the search;
build structures and build sequences keep their shards (as before) unless
the new --cleanup-embeddings flag is given.
Tests: tests/test_embedding_computer.py
- RunIdTest: explicit id, SLURM ids with step and restart count, fresh id on
rank 0, refusal on rank > 0 (LOCAL_RANK, SLURM_PROCID, NODE_RANK, RANK),
invalid ids, launch_rank, exported_run_id restore semantics.
- ShardIsolationTest: same SLURM_JOB_ID with different steps and a requeue,
id-prefix isolation, count check (stale extra shard, missing rank),
remove_shards scope.
- RunIdSharingTest: rank 1 simulated in a subprocess with LOCAL_RANK=1 or
SLURM_PROCID=1 (inherited id), srun tasks sharing SLURM ids; rank 0 reads
both shards; a child without an id refuses and rank 0's count check fails.
- CliShardLifecycleTest: 2-rank build keeps shards, query deletes its own,
--cleanup-embeddings, a lost rank fails before the index is written.
- TwoRankDDPSmoke (slow; FOLDMATCH_SLOW_TESTS=1): real devices=2 CPU/gloo
build of the fixture FASTA through `python -m foldmatch.cli.search`;
FAISS and the sequence store hold both entries, both shards share one id.
Behaviour change: shard file names (emb_<run id>-<rank>.parquet with the
new id format); non-SLURM multi-GPU runs no longer lose rank > 0
embeddings; a rank > 0 launched without a shared id (e.g. torchrun without
FOLDMATCH_RUN_ID) refuses to start; query shards are deleted after the
search; a shard/entry count mismatch is an error; new
--cleanup-embeddings flag on build structures/sequences (default off).
Review follow-up: run ids collided inside one SLURM step. Every command
of a step got the same id, always {job}_batch_0 in the batch step of an
sbatch script, so a second build silently replaced the first build's
kept shards, and a query that followed a build deleted them (reproduced
with a fake embedding pass). Now:
- run_id() uses the SLURM form only in a task of an srun step
(srun_step_id(): SLURM_STEP_ID set and below SLURM's special step
values) and appends command_digest(), 8 hex digits of SHA-1 over
sys.argv[1:]: the tasks of a step run the same command line and still
agree, while the commands one step runs one after another get
different ids. The batch step is a single task, so only Lightning can
start more ranks there; it now takes the fresh-id path, like any run
outside SLURM, and its children inherit the id.
- EmbeddingComputer.refuse_existing_shards(): every rank refuses to start
(FileExistsError, before ESM3) when shards with its run id already
exist, e.g. the same command run twice in one srun step, or a reused
FOLDMATCH_RUN_ID. No rank can have written yet at that point: ranks
write only after all of them have joined the process group.
- The rank > 0 refusal now names the launches that need FOLDMATCH_RUN_ID:
torchrun, and one command per node with NODE_RANK set.
Tests: RunIdTest (srun id with the command digest; the batch step draws
a fresh id per command), ShardIsolationTest (two commands in one srun
step, commands in one batch step, existing shards refused before the
pass for FASTA and structure input), RunIdSharingTest (srun tasks with
the same command line share the id), CliShardLifecycleTest (a build, a
query and a second build in one batch step leave the first build's
shards alone; the same build twice in one srun step is refused before
ESM3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New foldmatch/utils/sequence.py (S15): - normalize_sequence(s): removes all ASCII whitespace (string.whitespace) and uppercases ASCII letters; every other character is kept one for one, so ESM3 still gets one token per residue. Non-ASCII input raises ValueError; allow_non_ascii=True keeps it unchanged (only ASCII letters are uppercased), for the plain paths that warn instead of refusing. - seq_sha1(s): SHA-1 of the ASCII bytes of the normalised sequence. - scan_fasta(path): the offset scan of utils.fasta.iter_fasta_offsets with the same record rules, but the length is the normalised length. Users: - EsmProtFromFasta normalises each sequence before ESM3 and applies min_res_n to the normalised length (scan_fasta). Lowercase residues used to be tokenised as <unk> and internal whitespace as extra <unk> residues. - SequenceStore.create stores normalize_sequence(seq) instead of seq.strip().upper(). - query sequences aligns the normalised query sequences in Stage 2. - EmbeddingComputer's expected-count check counts with scan_fasta, so it uses the same filter as the dataset. Measured on the base commit (C-runs/base_smoke/lower_check.py): the fixture FASTA in lowercase gave chain embeddings with cosine 0.3121 (1acb_E) and 0.3427 (2uzi_A) to the uppercase ones. With this commit they are bit-identical (torch.equal). Tests: tests/test_sequence_normalisation.py - NormalizeSequenceTest: whitespace kinds, all 128 ASCII characters one for one, non-ASCII refused or kept, idempotence, seq_sha1, scan lengths. - TokenizerInvariantTest (no weights): the 22 F-tok cases in tests/resources/esm_codes/tokenizer_cases_v1.json (raw and normalised token ids from esm's EsmSequenceTokenizer, normalised strings, length preserved, non-standard counts). - LowercaseFastaTest: a lowercase FASTA with spaces, tabs and CRLF reads as the uppercase fixture in the dataset, the store and the query path; min_res_n counts normalised residues; same tokens; chain embeddings from ESM3 on CPU are bit-identical to the uppercase fixture's. Behaviour change: FASTA records with lowercase letters or whitespace inside sequence lines are now embedded, stored and aligned as their uppercase, whitespace-free sequence (their embeddings were wrong before), and min_res_n counts residues without whitespace. Non-ASCII characters are kept as they are in the store (they were uppercased by str.upper() before, which can change the length, e.g. 'ss' for the German sharp s). Uppercase FASTA without internal whitespace is unchanged. Review follow-up: FoldMatch.sequence_embedding, the Python API's sequence input, now applies normalize_sequence (allow_non_ascii=True, as the plain FASTA paths do) after stripping FASTA header lines, so "acde fghik" no longer reaches ESM3 as <unk> tokens (S15/D12: every input), and min_res/max_res count residues, not characters. Tests: FoldMatchApiNormalisationTest (a fake ESM3 records the encoded sequence: lowercase, spaces, tabs and CRLF are normalised, also after a FASTA header; the length limits count normalised residues). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before, a FASTA problem or an existing output was only noticed after the
whole ESM3 pass: create_db and SequenceStore.create refused existing files
at the end, and duplicate ids were only reported by the store.
utils/sequence.py:
- preflight_fasta(path, min_res_n) extends the dataset's offset scan (same
record rules, normalised lengths) into one read-only pass that returns a
FastaPreflight: entries, entries_kept (>= min_res_n), residues,
duplicates, unique_residues, nonstandard_entries/residues,
non_ascii_entries, angle_bracket_entries ('<' or '>'), max_length,
example ids, and the offsets of the kept records.
- Duplicate ids are found from sorted 64-bit blake2b id hashes
(8 bytes per record), with an exact comparison of the ids whose hashes
collide, so memory stays bounded; the hash does not depend on hash()
randomisation, so every rank finds the same duplicates.
- FastaPreflight.check(strict): strict mode (for the coming residue-code
paths) raises one ValueError for duplicates, '<'/'>' and non-ASCII;
plain mode logs each as a warning and the run goes ahead as before.
EsmProtFromFasta accepts the FastaPreflight in place of the path and
reuses its offsets, so the pre-flight is the only scan of the FASTA on
each rank (test: one _scan_records call per build).
CLI (every rank, before EmbeddingComputer and ESM3):
- build sequences: refuses an existing {db}.index/.metadata/.invlists/
.sequences/.rescodes.* (FileExistsError), then runs the pre-flight
(plain). The pre-flight's entries_kept is the count rank 0 checks the
shards against (G7). After the pass, rank 0 alone creates the
SequenceStore and its meta must equal the pre-flight
(entry_count = unique entries, total_residues = unique residues,
duplicates_ignored = duplicates); otherwise the store is removed and the
build fails (the FASTA changed during the build).
- build structures: refuses an existing .index/.metadata/.invlists before
ESM3 (what create_db would have refused after the pass).
- query sequences: runs the pre-flight (plain) and passes it on.
The output check does not import FAISS, so FAISS is still first loaded
after the ESM3 pass, as before.
Measured (C-runs/bench/scan_bench.py, synthetic 200,000 records, 72.2 MB,
while a test suite ran on the same Mac): old iter_fasta_offsets scan
3.44 s, preflight_fasta 4.01 s (1.17x), peak traced memory 5.2 MB; an
all-duplicates file (2 x 100,000 records) 5.58 s.
Tests: tests/test_preflight.py
- PreflightCountsTest: every count on a FASTA with duplicates, '<',
non-ASCII, internal whitespace and an empty record; min_res_n; fixture and
empty file; forced 64-bit hash collisions resolved exactly; strict
refuses each problem, plain warns; the F-tok cases (strict_refusal,
normalised length, non-standard count); the dataset takes the offsets.
- PreflightTest: three simulated ranks (separate processes, different
PYTHONHASHSEED) raise the identical refusal; an existing output (6
suffixes, rank 0 and rank 1 env) and a strict refusal are raised before
ESM3 (spy on the ESM3 loader); structures; a plain build with problems
warns and keeps today's behaviour; one scan feeds pre-flight and dataset;
store meta must equal the pre-flight.
Behaviour change: existing outputs and FASTA problems are reported before
the ESM3 pass instead of after it; a build sequences where only a stale
.sequences or .rescodes.* file exists is now refused (before, the stale
.sequences made the build fail after the pass, and .rescodes.* did not
exist); plain builds and queries log warnings for duplicate ids, '<'/'>'
and non-ASCII sequences; a store that disagrees with the pre-flight fails
the build.
Review follow-up:
- A build whose FASTA has no entry with at least min_res_n residues
(including an empty file) is refused by the pre-flight, before ESM3;
it used to fail after the pass with "No embeddings provided to
create_database".
- query sequences and query db decide whether Stage 2 runs, and check
the sequence stores it will read, before ESM3 (query sequences) or the
FAISS search (query db): --seq-identity without a store, an
incomplete store (G6) and an E-value filter over an empty subject
database (G2's check_evalue_search_space) now fail there instead of
after the pass.
- When the store disagrees with the pre-flight, the build removes the
.index, .metadata (and .invlists) it wrote as well as the store, so
the same build can simply be run again; before, the leftover index
made the re-run fail the outputs-exist check.
- The database path split now lives in foldmatch/search/db_path.py
(parse_db_path, no FAISS import); EmbeddingDatabase and the CLI's
output check both use it instead of two copies.
Tests: PreflightTest (nothing to build is refused before ESM3; a store
mismatch leaves no database files and the build re-runs; Stage-2 store
problems are refused before ESM3 and before the db search; the output
check imports neither FAISS, torch nor esm).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The help of --min-coverage on `query sequences` and `query db` read
"drop hits whose query AND subject coverage are below this fraction",
which suggests a hit is dropped only when both sides are below. Stage 2
keeps a hit only if min(qcov, tcov) >= --min-coverage, i.e. it is dropped
when either side is below. Both options now share one help text that says
so.
Tests: tests/test_cli_help.py (the help of both commands says "both" and
"either side is below"; align_candidates drops a hit with qcov 1.0 and
tcov 0.45 at 0.8 and keeps it at 0.4).
Behaviour change: help text only.
Review follow-up: align_candidates' docstring had the same misleading
wording ("drop hits whose query *and* subject coverage are below
this"); it now says a hit is dropped when its query or subject coverage
is below the value, i.e. kept only if min(qcov, tcov) >= min_coverage.
Test: MinCoverageHelpTest.test_align_candidates_docstring_says_the_same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A release must not fetch a model at whatever revision a branch points to. ReleasePinTest scans the package source (AST) for every Hugging Face revision it declares or passes: module-level *REVISION* constants and the revision= argument of snapshot_download, hf_hub_download, try_to_load_from_cache and hf_hub_url, resolved through literals, module constants and parameter defaults. Each must be a 40-hex commit; None, a missing revision=, a branch or tag name, a short or uppercase hash, or a value that does not resolve to a literal is refused. esm's own from_pretrained (which fetches EvolutionaryScale repositories unpinned) is refused outside `if __name__ == "__main__":` blocks. Today the scan finds the ESM3 pin (loaders.ESM_REVISION, used by try_to_load_from_cache and snapshot_download) and the aggregator pin (utils/model.py REVISION, used by hf_hub_download). On the base it flags the unpinned snapshot_download that G1 fixed. Tests: tests/test_release_pins.py ReleasePinTest (scan of the package; the known pins are found; runtime values; the checker refuses None, missing, branch, short, uppercase and unresolvable revisions and unguarded from_pretrained). Behaviour change: none (test only). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
align_candidates created its worker pool with mp.Pool, which follows the interpreter's global default start method. Python 3.14 changes that default on Linux from 'fork' to 'forkserver', and an embedding application can change it with set_start_method, either of which would silently change how the Stage-2 workers start (and what they inherit). Create the pool through mp.get_context(_pool_start_method()), which returns the default CPython used up to 3.13: 'spawn' on macOS and Windows, 'fork' everywhere else (Linux). On the supported interpreters this is the method mp.Pool already used. Tests: StartMethodTest checks the platform mapping, that it equals a fresh pre-3.14 interpreter's default, and that with the global default forced to 'forkserver' the pool is still created with the pinned method (a spy on multiprocessing.pool.Pool) and returns the serial results. Behaviour change: none on Python <= 3.13 unless the caller changed the global start method; such a caller now gets the platform default for the Stage-2 pool instead of its own choice. align_candidates output is byte-identical to the base (45 configurations, full-precision metrics). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ESM3 block 0 runs geometric attention on every input. Without coordinates every frame is masked, so it adds exactly zero to the residue stream, but it still allocates an O(L^2) tensor per head, which set FoldMatch's practical length limit for FASTA input (A100-40GB out of memory at 2,750 aa, about 50 GB RSS on CPU at 3,000 aa). per_residue_embedding(model, esm_prot) in utils/esm/loaders.py is now the one ESM3 call path of ChainCompleteModule, EsmModule and AssemblyCompleteModule. It decides per protein: when esm_prot has no coordinates it switches use_geom_attn off on the blocks that have it (block 0) through geometric_attention_off() and restores the flag afterwards, also when ESM3 raises. Proteins with coordinates (structure inputs) take ESM3's path unchanged. The pre-flight length refusal is not part of this commit; it waits for the NERSC measurements. Measured (P0.7, NERSC): bit-identical per-residue embeddings on GPU for 300-2,000 aa and on CPU for 300-1,000 aa; 16,000 aa fit in about 10 GiB. Here (M1 Max, CPU, a 1,500-aa sequence): peak RSS 10.87 GiB with geometric attention vs 8.17 GiB without (the model-load peak), 55.6 s vs 10.3 s, per-residue embeddings bit-identical. Chain and residue embeddings of the fixture FASTA (chain_complete and sequence_inference predict, CPU, batch sizes 1 and 2) are byte-identical to the base. Tests: tests/test_esm_geom_attn.py GeomAttnOffTest on the fixture FASTA (CPU): only block 0 has geometric attention; per-residue embeddings with it off are bit-identical to on, geom_attn is never called and the flag is restored; a structure input (2uzi.cif) still runs geometric attention with unchanged output; the flag is restored when ESM3 raises; ChainCompleteModule and EsmModule predict_step never call geom_attn for FASTA proteins and give bit-identical residue and chain embeddings. Integration follow-up: FoldMatch.sequence_embedding in src/foldmatch/foldmatch.py, the Python API's other sequence-only ESM3 path, now calls per_residue_embedding too; its structure methods keep their coordinates and are unchanged. Test: GeomAttnOffTest.test_foldmatch_sequence_embedding_runs_without_geometric_attention (both fixture sequences: geom_attn is never called and the embeddings are bit-identical to the pre-G13 call; the test fails without this change). Behaviour change: memory and time only, for sequence-only ESM3 passes; embeddings bit-identical. Review follow-up (thread safety): geometric_attention_off flips a flag on the shared model, so a structure forward on the same model running in another thread at the same time would silently lose geometric attention (reproduced with a fake ESM3). The block now holds a re-entrant lock per model (weakly keyed; one shared lock for objects that cannot be weakly referenced), also with off=False, so every forward through per_residue_embedding takes it: forwards on one model from several threads run one at a time, forwards on different models do not wait. FoldMatch's structure methods (residue_embedding, residue_embedding_by_chain, residue_embedding_by_assembly) now call per_residue_embedding too, the same encode and forward_and_sample as before with their coordinates, so they take the lock as well. Tests: GeomAttnThreadSafetyTest (no weights: a structure forward waits for a sequence forward on the same model and sees geometric attention on; other models do not wait; FoldMatch's structure methods go through per_residue_embedding). Pre-flight length refusal: not added. The NERSC measurements (2026-10-13, after this commit) found no practical limit for sequence-only input: on an A100-40GB, 16,000 aa peaked at 10.40 GiB (4,000 aa at 6.56 GiB), and on a CPU node 4,000 aa peaked at 8.25 GiB RSS and 16,000 aa at 11.85 GiB; per-residue embeddings were byte-identical to the base on GPU (300-2,625 aa) and CPU (1,000-4,000 aa). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Apple Silicon the default accelerator is the GPU (MPS), and predict() failed there at any batch size: in eval mode nn.TransformerEncoder's fast path checks the padding mask with aten::_nested_tensor_from_mask_left_aligned before it checks the device, and MPS does not implement that operator. With PYTORCH_ENABLE_MPS_FALLBACK=1 it ran, but MPS then took the slow path, which leaves values in the padded rows, and the aggregator summed them: a padded chain at batch size 2 came out with the wrong embedding. (a) ResidueEmbeddingAggregator.forward: on devices other than CPU and CUDA (the only ones with the nested-tensor fast path) it sets transformer.use_nested_tensor = False for the call and restores it afterwards, so the unimplemented operator is never reached; and it always zeroes padded rows (masked_fill(x_mask[..., None], 0)) before the sum. After the fast path those rows are already zero, so CPU and CUDA outputs do not change. (b) ChainCompleteModule, AssemblyCompleteModule and EsmModule call release_mps_cache(self.device) (utils/model.py) after every batch: on MPS torch.mps.synchronize(); torch.mps.empty_cache(), elsewhere nothing. Without the release an ESM3 pass on MPS grew to 83 GB and slowed to 41 residues/s (P0.5). Measured on this M1 Max (fixture FASTA, predict(), no fallback flag): the base fails on MPS with NotImplementedError at batch sizes 1 and 2; with G14 MPS runs, and its chain embeddings match CPU within 8.9e-7 (max |diff| / max |cpu|) at batch sizes 1 and 2. On the real residue embeddings of 1acb.A and 2uzi.A (CPU), the aggregator with the fast path disabled matches the fast path within 2.2e-7 and batch size 1 within 8.2e-7, while the pre-G14 slow path gave the padded chain 1.05 relative error, cosine 0.9949 and 2.1x the norm. CPU outputs are byte-identical to the base for the fixture FASTA (chain embeddings at batch sizes 1 and 2, per-residue embeddings) and for the fixture mmCIFs (structure chain and assembly embeddings at batch sizes 1 and 2). CUDA is expected to be byte-identical too (same fast path); to be checked on NERSC. Tests: tests/test_mps_support.py AggregatorPaddingTest (CPU; padded batch: fast path disabled == fast path within 1e-5 relative == batch size 1, and the pre-G14 slow path is wrong; byte-identical to the pre-G14 forward for padded, single and unbatched inputs; on the meta device, which lacks the operator as MPS does, the old forward raises and the new one runs with the flag off for the call and restored; predict() on the fixture FASTA at batch sizes 1 and 2 byte-identical to the pre-G14 forward), ReleaseMpsCacheTest (release only on MPS; each ESM3 module releases once per batch), MpsSmokeTest (skipped unless MPS is available; runs here: predict() in a child process without PYTORCH_ENABLE_MPS_FALLBACK, batch sizes 1 and 2 match CPU within 1e-5 relative, and a spy sees synchronize then empty_cache at the end of every batch, and neither on CPU). Behaviour change: MPS only - predict() now runs on the Apple GPU (it failed before) and padded chains are correct (with the fallback flag they were wrong; databases built that way at --batch-size > 1 should be rebuilt). CPU byte-identical; CUDA expected byte-identical. Callers that run the aggregator outside the fast path on CPU/CUDA (training mode, or grad enabled with trainable weights) now also exclude padded rows from the sum. Checked on NERSC (2026-10-13, A100-SXM4-40GB, torch 2.11.0+cu130, on the series before its review follow-ups, which leave this commit's code unchanged): CUDA chain embeddings of 40 FASTA proteins (30-946 aa) at batch sizes 1 and 4, and of the fixture mmCIFs (5 chains, 2 assemblies) at batch sizes 1, 2 and 4, are byte-identical to the base, as are CPU chain embeddings of 6 proteins at batch sizes 1 and 4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the user-visible P1 changes (G1-G14) to the [Unreleased] section: Added: --cleanup-embeddings on build structures / build sequences (G7). Changed: ESM3 snapshot pinned to rcsb/rcsb-esm@636acb42 and weights checked against sha256 5ead5a13 (G1), including the offline note, and torch.load weights_only=True (G5); FASTA normalisation (G8); FASTA problems and existing outputs reported before ESM3 (G9); per-run shard names, FOLDMATCH_RUN_ID, query-shard deletion and the shard count check (G7); --max-evalue against an empty database (G2); --min-coverage help (G10); vectorised identity (G3); geometric attention off for sequence-only ESM3 input (G13). Fixed: lost rank > 0 embeddings in non-SLURM multi-device runs (G7); Apple Silicon GPU (MPS) support, with the advice to rebuild databases built on MPS with PYTORCH_ENABLE_MPS_FALLBACK=1 and --batch-size > 1 (G14). No version bump (P9). G4, G6, G11 and G12 change nothing users see and have no entry. Review follow-up: the entries follow the review fixes folded into G1-G14. G7: run ids per srun-step command and per batch-step command, the existing-shard refusal, and multi-node launches without SLURM next to torchrun. G8: FoldMatch.sequence_embedding is normalised too. G9: the nothing-to-build refusal, the Stage-2 store checks before ESM3 or the search, and removal of the database files on a store mismatch; new entry for the incomplete-store refusal (G6). G2: the rebuild advice and the early check. G13: CPU-node memory measured on NERSC and the per-model lock in the Python API. G14: CUDA byte-identity checked on NERSC, and the change for callers of ResidueEmbeddingAggregator outside the batched inference fast path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
foldmatch.model.residue_code_encoder implements spec S2-S5 of esm-codes-v1 with the exact research casts (D3; the whitener is not folded): - Zcw50Transform (S2: x16 = fp16(x); xs = fp32(x16[:, dims]) - offset; xw = fp32(fp16(xs @ M))), CodeEncoder (S3: lin + fc2(GELU_exact(fc1))), ResidueCodeHead.encode_target (S4: uint8 [L, 8], sign bits h >= 0 packed MSB-first on the device) and encode_query (S5: float16 [L, 64]), both under no_grad, a float32 dtype guard and strict_fp32; content_digests() (foldmatch-tensor-digest/1, as bundle.json lists them); - strict_fp32: matmul precision "highest", cuBLAS / cuDNN TF32 off, every per-backend fp32_precision "ieee" on torch >= 2.9, autocast disabled; restores both TF32 APIs exactly, also after an exception; - load_esm_codes_encoders(path, sha256=None, device): torch.load weights_only=True; refuses tampered files, bare state_dicts, unknown formats, unknown or altered architectures, conventions this module does not implement, and tensors of the wrong dtype, shape or non-finite; strict=True on the meta device (no RNG draw); eval, dropout 0, no grads. tests/test_esm_codes_encoder.py: loader refusals and weights_only, on-device packbits = np.packbits (CPU, MPS / CUDA when present), encode outputs = the spec written out with F.linear, fp16 input cast, autocast, the S4 sign rule (h = +0.0 / -0.0 gives a 1 bit), strict_fp32 in 8 subprocess scenarios; with FOLDMATCH_ESM_CODES_BUNDLE set (P4 adds the pinned download): bundle files, sha256s and content digests, the scoring npz without torch, parity on F2 / F2c / F2b (codes identical, q within 1 fp16 ulp of each row's largest value on every platform), GPU vs CPU, and NotRecoverableFromWeights (GELU-tanh, no xw fp16 cast, folded whitener, no input fp16 cast each change the codes; compared with the module's own output on the same machine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vendored C kernel for the esm-codes-v1 Stage 2 (ABI 1, DP id fmsw/1), derived from the research swkern.c (md5 438d5a98...) and swmulti.c (md5 b12fbc2c...) through the P0.4 spike fmsw_spike.c: - fixed 4-lane rowcol reductions on every ISA (AVX2 included), S7 degenerate rule (std < 1e-6 contributes exactly 0), anti-diagonal strip DP, traceback in forward order, -0 results canonicalised to +0; - fmsw_query_prep (S5, float64 norm in ascending k; status 2 on NaN/Inf); - the similarity is the caller's per-pair BLAS GEMM (D2): fmsw_score_fwd_rev and fmsw_align take per-pair contiguous [n, m_k] matrices at 64-byte-aligned offsets (refused otherwise); S_r reads the row-reversed view (no copy); - fmsw_sim_ref: fixed-order scalar similarity, test reference / golden hash only; - statuses (0 ok, 1 workspace too small, 2 bad arguments), int64 offsets, int32 lengths, size_t workspace sizes; - sse2 / avx2 (-mavx2, no FMA) / neon / generic (vector extensions) builds, fmsw_cpu_has_avx2 (CPUID + XGETBV), ABI / ISA / compiler / injected source sha256 exports. foldmatch._native locates the per-ISA libraries; python -m foldmatch._native.build compiles them in place (gitignored) with exactly -O3 -std=c11 -fPIC -shared -ffp-contract=off -fno-fast-math -fvisibility=hidden plus the ISA flag and the source sha256. Every x86_64 library also gets -march=x86-64 -mtune=generic before the ISA flag, so a compiler or wrapper that defaults to a newer -march (x86-64-v3 distribution compilers, the Cray cc wrapper's -march=znver3) cannot put AVX2 code into the sse2 or generic library. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
foldmatch.search.swkern (numpy + ctypes, never torch): - dispatch: x86_64 sse2 -> avx2 when fmsw_cpu_has_avx2, arm64 neon, else generic; FOLDMATCH_SW_ISA override validated (avx2 without AVX2 refused); ABI must be 1; a library built from another source sha256 is refused as stale; missing libraries name the rebuild commands; Kernel.info() logged once; - query_prep (S5 in C), decode_codes, similarity (S6: one numpy fp32 GEMM per pair into a 64-byte-aligned C-contiguous buffer, layout enforced), score_fwd_rev (round 1, S_r on the reversed view), align (round 2, numpy arrays, paths in forward order as written by the kernel); - blocking: per-pair matrices packed into blocks of <= block_floats (2^17) floats on the 64-byte grid, an oversize pair alone, pairs above max_pair_cells (2^28) skipped (NaN) with a once-per-run warning; - per-thread Workspace grown on demand; - path helpers (path_ops, cigar_from_path, expand_cigar, path_metrics) and float64 diagnostics (z64_pair, path_score, S16 bounds); - selftest(): DP golden hash through fmsw_sim_ref on 1,000 portable synthetic pairs (6b7974aa...), then the BLAS path within S16; logs numpy's BLAS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/esm_codes_reference.py: verbatim copy of normalize_scores and smith_waterman from esm3_needleman_wunsch.py (md5 7ec19b7e..., lines 113-125 and 138-311; block sha256s checked), S5/S7 wrappers, float64 margin helper, reference encoder and reference_evalue (reproduces F4). tests/test_swkern.py (P3 tests 1-9): F1 differential within S16 and 50/50 paths; 500 seeded property cases (invariants, path re-score, CIGAR round trip, batch-order and block-size invariance, reversed view, float64 path agreement outside near-ties); DP golden hash F1 + 1,000 synthetic pairs (30ec40c6...) on every local build; BLAS conformance (scores, paths, Q90, ln E, decisions, layout asserts, BLAS threads 1, no torch/biotite); F-deg, and the S7 cutoff pinned on hand-built similarities (std 5e-7 contributes 0, 2e-6 does not); 5,000 x 5,000 pair (FOLDMATCH_SLOW_TESTS=1) and max_pair_cells skip; ABI, dispatch, missing/stale/foreign libraries, the x86-64 baseline flags; workspace status codes; forward path regression; CIGAR round trip on 1,050 alignments. conftest.py builds the kernel in place when missing or stale (session fixture). benchmarks/stage2_esm_codes.py: kernel mode for P3 acceptance (e), FoldMatch vs the research block-GEMM path on the research workload rebuilt from L2m_B8_s0.chains.npz. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- hatch_build.py: hatchling custom build hook for the wheel target. Standard wheels compile every ISA library of the target machine into a temporary directory with foldmatch._native.build.compile_library (loaded by file path; the source sha256 is injected) and ship them via build_data["force_include"]; pure_python=False, tag py3-none-<platform> (sysconfig platform on Linux, macosx_<target>_<arch> on macOS with MACOSX_DEPLOYMENT_TARGET defaulting to 11.0, ARCHFLAGS cross builds, universal2 refused). No manylinux tag: auditwheel / delocate retag. Editable builds compile in place into src/foldmatch/_native/. FOLDMATCH_REQUIRE_NATIVE=1 makes a compile failure fatal; otherwise a warning and a pure py3-none-any wheel, also when one ISA fails after others compiled (build_data changes only once every library is built). x86_64 targets (also macOS cross builds) get the x86-64 baseline flags. - pyproject.toml: hook registration; in-place libraries excluded from every target (wheels never reuse them, even without .gitignore, e.g. in Docker); classifiers POSIX :: Linux and MacOS instead of OS Independent; sdist keeps hatchling's default file set; dependencies unchanged. - _native/build.py: macOS install name @rpath/<library> (the temporary output path was recorded in the binary); exports REINSTALL_COMMAND. - _native/__init__.py, swkern.py: a missing library in an installed package is reported as a pure-Python wheel with the reinstall command `pip install --no-cache-dir --force-reinstall --no-binary foldmatch foldmatch`; `python -m foldmatch.search.swkern` runs selftest() and prints the JSON report. - Dockerfile: COPY hatch_build.py, FOLDMATCH_REQUIRE_NATIVE=1, non-editable `pip install --no-cache-dir .`, selftest after the install (own RUN: the cleanup's `|| true` chain would have swallowed a failure) and again in the compiler-free runtime stage; /app no longer copied to the runtime stage. .dockerignore drops in-place libraries; .gitignore: /wheelhouse/ and the /.claude line the kernel commit had glued to its comment. - tests/test_packaging.py: wheel contents (expected _fmsw_* per platform, architecture, embedded source sha256 = tree, Root-Is-Purelib false, tags, classifiers), sdist contents (default set, hook, C source, fixtures, no compiled files), in-place stale-library check, selftest() from the wheel in a clean venv (every ISA; installed with --ignore-installed, so a same-version foldmatch in the test interpreter cannot shadow the wheel), pure-wheel fallback and its refusal, partial compile failure, FOLDMATCH_REQUIRE_NATIVE fatal, editable in-place build, platform tags. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- hatch-test: FOLDMATCH_REQUIRE_NATIVE=1 (self-hosted buildchain runner kept) and CC = a GCC >= 12 (gcc-12 is installed when the runner's gcc is older: fmsw.c needs GCC >= 12 or Clang >= 14). - kernel: kernel-x86 (ubuntu-latest, GCC and Clang; sse2, avx2, generic; the runner must have AVX2), kernel-arm64-linux (ubuntu-24.04-arm, GCC; neon, generic), kernel-macos (macos-14, Apple Clang; neon, generic): developer build, tests/test_swkern.py (DP golden hash for every ISA library, BLAS conformance, slow pair included) and selftest() per ISA. - kernel-sanitize: ASan + UBSan builds of the libraries, kernel tests with the ASan runtime preloaded. - packaging (every PR; ubuntu-latest and macos-14): sdist, wheel from the sdist, tests/test_packaging.py. - Release: hatch-build builds the sdist only (a plain wheel would be linux_x86_64); build-wheels runs cibuildwheel v3.1.4 (CIBW_BUILD=cp312-*, manylinux_2_28 images with CC=gcc (the image's gcc-toolset), Linux x86_64 / aarch64, macOS arm64 + x86_64 with MACOSX_DEPLOYMENT_TARGET=11.0), then checks the repaired wheels' contents and runs selftest() from a clean venv (the macOS x86_64 wheel under Rosetta); publish-to-pypi uploads the sdist and every wheel. The docker workflow call is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unreleased: - Added: the residue-code encoders and the native Smith-Waterman kernel as groundwork for the opt-in esm-codes-v1 Stage-2 scoring; nothing uses them yet (the database build and query options come later), existing commands and BLOSUM62 output are unchanged. - Changed: platform wheels (Linux x86_64 / aarch64 manylinux_2_28, macOS arm64 / x86_64) and classifiers; building from source (sdist, editable, hatch test) needs GCC >= 12 or Clang >= 14, else a pure-Python install with a warning (the coming esm-codes scoring then refuses with the reinstall command); FOLDMATCH_REQUIRE_NATIVE=1; python -m foldmatch._native.build; the Docker image (non-editable install, kernel self-test, no /app copy). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- search/scoring_system.py: frozen EsmCodesSystem dataclass, pure metadata (no torch, numpy or I/O). Fields per plan section 8 P4: identity, the bytes-determining encoding fields (ESM3 repo / weights sha256, tap, input normalisation, casts, transform and target-encoder content digests, encoder architecture and dtype, code layout, sign rule), scoring (query encoder digest, query dtype / prep and its eps, similarity "pm1_fp32_blas_gemm_per_pair", determinism "tolerance_s16_v1", rowcol-z normalisation with the S7 degenerate rule, setting D, DP id, score form), panel and statistics (panel digest, size, residues, quantile and method, the 6-centre table, low / high m rules, calibrated domain, degenerate-query rule) and the bundle (repo, revision, (name, sha256) pairs: the only place a bundle is named). - encoder_architecture and encoder_dtype are encoding fields beyond the plan's list (P4 review): the content digests cover only the weight tensors, and the same weights give other codes with GELU-tanh or another precision, so encoding_digest must cover them. - Every field is required. __post_init__ normalises (tuples, floats, sorted bundle files, nested Normalisation / CalibratedDomain) and refuses inconsistent values: 6 positive increasing centres, lambda > 0, finite numbers, gap_open >= gap_extend >= 0, quantile in (0, 1), 40-hex revision, 64-hex sha256s, code_bits = 8 code_bytes, bundle.json listed. - digest() over every field and encoding_digest() over ENCODING_FIELDS only: 16 hex of sha256(tag line + canonical JSON). to_json() / from_json() round-trip exactly and refuse unknown or missing fields at every level. - REGISTRY (read-only mapping, empty until the freeze commit), get() naming the known systems on an unknown name, DEFAULT_ESM_CODES = "esm-codes-v1". - types/api_types.py: ScoringChoice (blosum62 | esm-codes; one scoring per query run, D7) with DEFAULT_SCORING = blosum62. - tests/test_scoring_system.py: round trip, digest sensitivity for every leaf (only encoding fields flip encoding_digest, the encoder architecture and dtype among them), tuple / number normalisation, unknown and missing fields, 40 inconsistent-value cases, get() on unknown names, read-only registry, field groups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- search/code_statistics.py (numpy only): lam_lnk(m, table) linear in ln m
between the six centres via numpy.interp (the research function, so the
centres give the table's own numbers bit for bit), clamped below the
first and above the last centre (S11, D15); any other low/high-m rule is
refused, never approximated. log_evalue = lnK + ln m + ln N - lambda S'
in float64, left to right (N <= 0 refused); bit_score = (lambda S' - lnK)
/ ln 2; render_bits = floor(bits + 0.5), refusing a non-finite value or
|bits| > 2^53 on the scalar and the array path alike;
format_evalue_from_log renders %.3E from ln E without under- or overflow
(exactly f"{exp(x):.3E}" in the normal float range); q90 =
numpy.quantile(float64 S, 0.9, linear). The table is any object with
bin_centres / lam / lnK / low_m_rule / high_m_rule (an EsmCodesSystem or
its worker view).
- tests/test_code_statistics.py: all 208 F4 rows at rel <= 1e-12 (lambda,
lnK, lnE, E, bits; printed bits exact), exact values at the six centres,
the low clamp, the D15 high-m clamp (not F4's extrapolated columns),
E = m N 2^-bits, vectorised = scalar, format_evalue_from_log(-1000) =
5.076E-435 plus decimal-arithmetic and carry checks, render_bits(-2.5) =
-2 and (-0.4) = 0 and its range, q90 against numpy, refusals.
On the Mac (numpy 2.4.4) every F4 row is bit-exact (worst rel 0.0).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- utils/model.py: get_esm_codes_bundle(system) fetches every file of the system's bundle_files with hf_hub_download at system.bundle_revision (the registry entry is the only source of repo, revision and sha256s) and verifies each sha256 on every call, writing nothing (no marker files), so HF_HUB_OFFLINE=1 and read-only caches work. Only a registry entry, exactly as registered, is served: a system from elsewhere (a manifest) with a foreign repo or revision under a registered name, or an unknown name, is refused (UnknownScoringSystemError) before any Hub call. Also refused: a non-EsmCodesSystem, an unpinned revision (None, branch, short or upper-case hash), BundleUnavailableError when a file is neither cached nor downloadable (offline, cached-only, Hub unreachable, a read-only cache without the bundle, HTTP and OS errors; names download-models), BundleMismatchError for a sha256 difference (naming the cache blob to delete) or a file missing at the pinned revision. local_files_only=True never downloads (the every-rank pre-flight after one fetch). download_esm_codes_bundles(names) for every registered system or the named ones; it downloads a corrupt cached file once more (force_download) and refuses it only if the Hub's file differs from the pin. torch and ESM3 are now imported inside the model builders, so the bundle loader needs no torch. - search/scoring_system.py: registered(system); prepare(system, local_files_only=False) runs check_implemented() (fmsw/1 and swkern.SETTING_D, implemented m rules, code layout and the method values in IMPLEMENTED, the encoder architecture and dtype among them), fetches the bundle and refuses it unless bundle.json and the scoring npz meta_json equal the system exactly (DOCUMENT_FIELDS, numbers as float64), every file bundle.json lists is pinned with the same sha256, the table arrays are bit-identical, the table digest matches, and the panel's shape, offsets and content digest match. PreparedSystem.for_worker() gives a picklable WorkerSpec (paths and scalars; load_panel() re-checks the npz sha256; usable as a statistics table); load_head() builds the encoder head and checks its content digests, architecture, encoder dtype and the ESM3 pin. - utils/tensor_digest.py: the foldmatch-tensor-digest/1 implementation in numpy only; residue_code_encoder.tensor_digest delegates to it. - download-models (fm-embedding and inference): also fetches the bundles of every registered system; --system (repeatable) restricts it; unknown names exit 2 before anything is downloaded. - tests/test_release_pins.py: revision=<system>.bundle_revision is a pin exactly when every registry entry's revision is a 40-hex commit. - tests/test_scoring_system.py: synthetic bundles in a Hugging Face cache layout fetched through the real hf_hub_download: verified paths, hashing on every call without writes, sha256 / corrupt-cache / unpinned refusals, the corrupt-file re-download of download-models, only the registered entry served, cached-only use, read-only-cache and HTTP errors as refusals, the bundle only from the registry (a second system's revision is the one downloaded, mocked hub), prepare() field-by-field refusals (a table off by one ulp, 20 more edits), unsupported systems refused before any download, offline / read-only / not-cached subprocess runs without torch and with an unreachable endpoint, download-models in both CLIs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- search/scoring_system.py: REGISTRY holds the frozen esm-codes-v1 entry (registry freeze, after P0.6, P0.7 and the upload): model M8_s0 (esm-codes-mlp-v1 encoders in float32), setting D, S = S_f - S_r, UniRef50 1,000-member panel, M3t999 table; calibrated domain queries 21-966 aa, targets <= 3,000 aa, <= 1% non-standard, targets with >= 2 distinct codes; low and high m clamps (D15); degenerate queries report no hits (S13b); bundle rcsb/foldmatch-esm-codes at its upload commit with the sha256s of LICENSE, README.md, bundle.json and the encoders and scoring files. This entry is the only place in the tree that names the bundle revision. - tests/test_scoring_system_golden.py (standard library and scoring_system only): golden digest(), encoding_digest() and canonical-JSON sha256 of every registered system (any change now fails), the digests recomputed from their definition, and the same values in fresh processes with numpy, torch and huggingface_hub blocked. - .github/workflows/publish.yaml: the kernel jobs (Linux x86_64 gcc and clang, Linux arm64, macOS arm64) run the golden test, so CI checks the digests on Linux and macOS on every PR (P4 acceptance). - tests/test_scoring_system.py: the entry against the kernel's DP and setting, the ESM3 loader pin, the encoder module and the P0.8 fixtures (ship_final table bit for bit, setting, identity), bundle.json states every value but the pins, the bundle revision exists exactly once in the source tree, and on the published bundle: pinned files, prepare(), a registered entry off by one ulp and other differences refused, unregistered variants refused before any Hub call, load_head() digests, architecture and dtype, offline and read-only use of a copied cache without torch; the skip rule of the test bundle helper. - tests/test_code_statistics.py: the registry entry gives the F4 values; F1 Q90 and lnE at N = 1e9 within S16 (1e-4) and F3 Q90 within 1e-3 through the production path with the bundle panel. - tests/esm_codes_bundle.py: the bundle for tests, from the registry (get_esm_codes_bundle(get(DEFAULT_ESM_CODES))), FOLDMATCH_ESM_CODES_BUNDLE still overriding it for the encoder tests. Skipped only when offline mode is on and the bundle is not cached; with offline mode off an unreachable Hub, and always a mismatch, is a test error, never a skip. tests/test_esm_codes_encoder.py uses it, so the P2 parity tests now run by default. Golden digests (macOS arm64 and Linux x86_64 identical): esm-codes-v1 digest 81de063d13a7ef66, encoding_digest 92220d870638cb57. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unreleased: - Added: the groundwork bullet now also names the versioned scoring system (frozen registry entry esm-codes-v1 with the bundle's repo, pinned revision and file sha256s) and its E-value statistics; still no command uses them. - Changed: download-models (fm-embedding and inference) also fetches and sha256-verifies the esm-codes bundle of every registered system (about 26 MB), --system restricts it, unknown names are refused before any download, a corrupt cached file is downloaded again; later runs work with HF_HUB_OFFLINE=1 and read-only caches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ChainCompleteModule(..., residue_head=None, residue_output=None | "residue_codes" | "query_encoding"). The aggregator input (BOS and EOS included) is untouched; with a residue output, rows 1..L of each protein are encoded one protein at a time (S4 target codes, L*8 bytes; S5 query encodings, float16[L, 64] little-endian) and the step returns (chain_embeddings, names, payloads, seq_sha1s), seq_sha1 = sha1 of the normalised ASCII sequence (S15). predict_step never raises for a payload: ESM3 rows that are not L + 2, a non-ASCII sequence or an encoder failure give an empty payload (logged, counted), for the post-pass validation on rank 0. The head is a submodule, so Lightning moves it to CPU, CUDA or MPS (D18, G14). chain_complete_inference.predict(..., residue_output=None, scoring_system=None): before ESM3 loads, and alike on every rank, it refuses structure input, a non-Parquet output, a system other than the registry entry of its name (prepare() / registered()), a bundle that does not match, a query pass without the database's system, and FASTA input with duplicate ids, '<'/'>' or non-ASCII (strict pre-flight, now reachable). The head comes from prepare(system).load_head(); only the process that starts a single-node run fetches the bundle, every other rank (Lightning children, multi-task srun steps, torchrun worlds, multi-node runs) runs prepare(system, local_files_only=True) (bundle_fetch_allowed). prepare_scoring_system() and check_residue_pass() are public for the CLI pre-flight. Callers without the keyword get the same module, writer and outputs. search/residue_shards.py starts the shard contract (names, the residue-output check, payload sizes). Parquet shards with the residue columns come with P5.3; until then a residue pass to an output folder is refused by the writer constructor, before ESM3. Tests (tests/test_esm_codes_build_channel.py, no ESM3: F2c/F2b rows through a fake per_residue_embedding): aggregator input and chain embeddings bitwise identical with and without codes, 2-tuple equal to the pre-P5 step, F2c and F2b arithmetic, empty payloads instead of raises, constructor refusals, every predict() refusal before ESM3, the bundle fetch rule, and predict() without the keyword constructing the legacy module with legacy outputs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ParquetBatchWriter(..., residue_output=None, system=None, byte_budget=None, min_res_n=None, esm_revision=None). The schema is fixed at construction: builds add residue_codes: large_binary and seq_sha1: fixed_size_binary(20), queries query_encoding: large_binary and seq_sha1, with the schema metadata foldmatch.residue_output, foldmatch.system, foldmatch.encoding_digest and foldmatch.system_digest (the query encodings also depend on the query encoder, which only the full digest covers). The payload column is built from explicit 64-bit offsets, so a row group may hold more than 2 GiB. A residue shard also records its pass, foldmatch.pass (JSON, residue_shards.pass_record): min_res_n and the ESM3 revision from the constructor, then, at on_predict_start and before any row group, the device type the module runs on, its ESM3 precision (bf16-autocast on CUDA, fp32 elsewhere; esm 3.2.3), the world size, the encoder precision and the FoldMatch, torch and esm versions and platform. The code store's manifest takes its build block from it, also when it is rebuilt from the shards. chain_complete_inference.residue_shard_writer() builds the writer of a residue pass. A plain writer refuses a pass record and keeps its legacy schema and bytes. Rows are flushed on a byte budget (default 256 MiB with a residue column) as well as on the row count (16,384). A plain writer has no budget by default, which keeps its row groups and bytes exactly as before (chain embeddings stay below 256 MiB per 16,384 rows anyway); one may be given explicitly. A writer with a column refuses a prediction without it (2- or 3-tuples, mismatched lengths, a seq_sha1 that is not 20 bytes) and, at on_predict_start on every rank, a module that does not produce it. A plain writer drops extra tuple elements. search/residue_shards.py: shard_schema(), shard_metadata(), pass_record(), esm_precision(), large_binary_array(). Tests: plain writer file bytes identical to the pre-P5 writer (2- and 4-tuples, one and several row groups), the fixed schemas and metadata, the pass record (CUDA, CPU, MPS; from a real Lightning pass in predict()), round trip with an empty payload, byte-budget and row-count flushes, a mocked payload chunk over 2 GiB (17 x 128 MiB, nothing touched), missing-column and constructor refusals, and predict() writing residue shards whose id/embedding columns equal the plain shard's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
align_candidates now aligns every (query, candidate) pair with fmswi_align (swint, P9.S1-S2) instead of biotite's align_optimal. The kernel is output-equivalent to biotite 1.7.1 (same score, end cell and path), so every AlignmentMetrics field and every output column is unchanged: on the paper's Fig. 2 search (job 60557988, one Perlmutter GPU node) the Stage-2 tables of 17,733,246 rows (threshold 0.7) and 9,503,077 rows (0.8) were byte-identical to the benchmark's, and Stage 2 took 486 s instead of 11,403 s in this commit's configuration (the cell dispatch and the 'prefix' traceback below; 468-486 s over the runs with the cell dispatch). - One kernel call per task: _align_query sanitises the query and the candidates with byte tables (sanitize_sequence for non-ASCII text), builds the query profile once (biotite's BLOSUM62 rows plus the composition-bias offsets when that correction is on) and derives the metrics from the kernel's counts; bit scores and database E-values are computed once per distinct score of a task. The per-pair biotite code is gone (_to_protein, _positional_query, _derive_alignment_extras, _count_runs, _run_length_encode, _align). - No other engine and no switch. The kernel is loaded as for esm-codes: the parent calls swint.load() before it reads a sequence (a missing, stale or wrong-ABI library raises swkern.KernelError with the rebuild command; no fallback) and passes its ISA to the workers, which load that ISA in the pool initializer. - A worker whose initializer fails no longer hangs Stage 2 (review finding 5): _pool_worker_init keeps the error and every task of that worker raises it, as the esm-codes Stage 2 does; multiprocessing.Pool would otherwise start new workers for ever. The serial path raises it at once. - Dispatch by DP cells (the esm-codes Stage 2's): contiguous candidate slices of at most max(2^20, total cells / (4 x workers)) cells (a candidate above that is a slice of its own), queued longest first through imap_unordered and put back in task order, so the per-query lists and the stable sort see the same input as before. It replaces the equal-candidate chunks in input order, which left the largest queries to the end of the run: on Fig. 2 that tail took 8,459 s with biotite, 4.3 s on the kernel and 0.4 s on the kernel with the cell dispatch, which saves about 5% of the kernel's Stage 2 (473 s against 500 s; the 72 s instead of 9,864 s of the largest query is the kernel's). - Traceback strategy 'prefix' (swint.DEFAULT_TRACE): at full scale it gave the same tables in the same time as 'full', with less worker CPU and memory. - check_gap_costs(gap_open, gap_extend, significance_mode, significance, max_evalue), new and public: the gap costs are the absolute values of the arguments (the sign is ignored, as before), which must be whole numbers (whole_number(): an int or a numpy integer, not a bool, or a float such as 11.0, which biotite took as 11), else TypeError; gap_open must be between 1 and swint.MAX_GAP (2^20) and gap_extend between 0 and swint.MAX_GAP, else ValueError, with the reason of the bound (the end-cell rule; FMSWI_MAX_GAP); and with significance_mode 'default', when significance is computed (a significance column, or a finite max_evalue) the costs must be 11/1, with the messages align_candidates gave before. align_candidates runs it before it loads the kernel or reads a sequence, whatever the queries (the 11/1 refusal used to come after the fetch, and only when some query had a sequence). Changes for the CHANGELOG: gap_open = 0 is now refused (biotite accepted it); a gap cost that is not a whole number, such as 10.5, is refused (biotite truncated it to 10); and -11/-1 under 'default' significance is now 11/1 (it was refused: the calibrated lookup got the signed values while the aligner used their absolute values). - The flag buffer of a task is n_pad x its longest candidate, one byte per cell, with no cap (review finding 6, left open). biotite stays for the BLOSUM62 matrix, for the 'sampled' significance estimate and as the tests' oracle. Comments that said Stage 2 aligns with biotite (api_types, karlin_altschul) now name the kernel. Tests: - tests/blosum62_reference.py: the replaced per-pair biotite path, as a test oracle. - test_alignment.py: the cell dispatch (order, budget, floor, coverage of every candidate); a single query cut into slices runs on the pool with the same output as serial; identity against biotite's alignment and get_sequence_identity (exact floats) through the kernel path; NativeKernelOnlyTest: biotite's aligner is never called, a kernel that cannot be loaded raises KernelError before any fetch (also with no query), initializer errors are raised by the tasks, and a pool whose workers cannot load the parent's ISA fails with KernelError within a deadline; GapCostsTest: the kernel range and the reason of each bound, the sign, whole floats taken and other non-integers refused (TypeError), 11/1 under 'default' significance and its messages, the checks before any work, and -go/-ge and whole floats aligning as go/ge. - The tests that run the BLOSUM62 Stage 2 use the fmsw_libs fixture (test_alignment, test_cli_help, test_code_alignment, test_preflight). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/test_swint.py, with biotite as the oracle (fixture fmsw_libs): - The kernel against biotite's align_optimal (score, end cell and the whole trace), in both traceback strategies (which must give identical outputs and paths): constructed tie cases (gaps as early as possible, zero-sum prefixes cut, the first maximal cell, no insertion next to a deletion, B/Z/X/*), random pairs at eight gap settings (gap_extend > gap_open included), long gaps (60 residues, both directions, three gap settings), position-specific bias (random offsets and the MMseqs2 correction itself) and the 32-bit redo. - Every ISA library of the machine, not only the dispatched one, gives the same outputs (WIDE included) and paths in both strategies, at 11/1 and 20/5, on random pairs, a 32-bit pair, a biased pair and the long-gap pairs (review finding 1: a bug in the generic library's 32-bit path passed the old test). - Argument refusals (gap costs out of the kernel's range, an unknown strategy, a code outside the alphabet), score-only mode, empty targets, the byte-table sanitiser against sanitize_sequence, and 'prefix' as the default strategy. - The Stage 2 against the biotite path it replaced (tests/blosum62_reference.py): align_candidates, serial and on a pool, equals the reference field for field (heavy strings, calibrated E-values and bits, order), also with the composition bias at scales 1 and 0.5, with negative gap costs, and with the 'sampled' significance at 5/2; and _align_query (one kernel call for a task) equals it pair by pair at eight settings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review finding 2: CI ran fmswi_align only in the x86 hatch-test job, because tests/test_swint.py needs biotite; the kernel jobs (every ISA), the sanitizer job and the wheel jobs (linux-aarch64, macOS x86_64 under Rosetta) never did. - swint.golden_cases(): 129 calls, 477 pairs, numpy only and integer only (swkern's splitmix64 stream, so the same bytes on every platform and numpy version), on a synthetic symmetric 24 x 24 score table instead of BLOSUM62: random and related pairs at eight gap settings (gap_extend > gap_open included), position-specific offsets, 60-code gaps both ways, 32-bit pairs (with gaps; from the start with a profile beyond +-16,384, and with one whose only large score, a code at -40,000, 16-bit lanes would wrap to +25,536), scores just below and just above the 16-bit limit 32767 - max(s) and a column best that reaches it to the unit (11 j - 1: 32,757 at column 2,978 goes to 32 bits before 32,768 wraps), a lazy-F loop long enough to wrap 16 bits without the clamp of F at -open (gaps 1000/1000, at n = 1024 and 2048 for 8 and 16 lanes), and edge cases (n = 1, m = 1, an empty target, no positive score). - swint.selftest(kernel): every case in traceback modes 0, 1 and 2; modes 1 and 2 must give the same outputs and paths, mode 0 the same score, end cell and WIDE with every other field 0; the sha256 of the inputs and outputs must be GOLDEN_SHA256. KernelError otherwise. - swkern.selftest() runs it on the same library and reports it under "fmswi", so 'python -m foldmatch.search.swkern' (each ISA in the kernel jobs, the wheel jobs, Docker) checks the integer DP too. Under -m, swkern's __main__ block now runs the package module's main: swint imports foldmatch.search.swkern, a second copy of the module, whose KernelError the __main__ copy's main did not catch (a failed fmswi check printed a traceback instead of "fmsw selftest failed"). - GOLDEN_SHA256 is the same for Apple clang NEON and generic and GCC 14.2 NEON and generic (arm64), and Apple clang SSE2, AVX2 and generic (x86_64 under Rosetta), all built with -Wall -Wextra -Werror. - The digest catches each of the review's 11 seeded kernel bugs on the libraries they affect, including the generic library's 32-bit max (finding 1) and the 16-bit limit without max(s), and, on all five local builds, the second review's lazy-F loop without its clamp and 16-bit limit raised by 2 (which neither the digest nor the biotite tests caught before the limit and carry cases) and 16-bit lanes chosen without the profile's range (caught since the -40,000 case); 0.3-0.5 s per library on an M1 Max. Tests: - test_swkern.py FmswiGoldenTest (no biotite, so the kernel jobs and the sanitizer job run it): the digest on every ISA library of the machine; the golden set reaches the 16- and 32-bit paths, the limit to the unit, the long lazy-F loop, long gaps both ways, offsets and empty alignments; its inputs are stable (their own sha256); a changed result or disagreeing modes fail the selftest. test_selftest checks the "fmswi" part of swkern.selftest(). - test_swkern.py FmswiArgumentsTest, on every ISA library and so under the sanitizers too: fmswi_align called directly refuses with status 2 a code >= A in the query or a target, A outside 1..255, n, K, m or t_off < 0, an unknown traceback mode, a profile score beyond +-FMSWI_MAX_SCORE, gap costs outside the kernel's range and a score beyond int32; with status 1 an ops buffer too small for the next pair (both traceback modes); and handles all-empty targets, an empty query and K = 0. SelftestCommandTest: a failed fmswi check under -m is reported as a failed selftest (exit status 1, no traceback). - test_packaging.py: the wheel's selftest in a clean venv reports the fmswi digest, for the dispatched library and every other one; the sdist test requires and byte-compares every kernel source file (TREE.SOURCE_FILES, fmswi_body.h included) and requires swint.py, tests/test_swint.py and tests/blosum62_reference.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… Stage 1
Whenever Stage 2 runs (a sequence store found, or --seq-identity),
'fm-search query sequences' (--scoring blosum62) and 'fm-search query db'
now check the BLOSUM62 Stage 2 before Stage 1: before the ESM3 pass of
the queries, on every rank, and before FAISS loads the database. The
11/1 refusal used to come from align_candidates, after Stage 1; gap_open
0 was accepted (biotite took it; the kernel needs a gap to cost at least
1 to open, P9.S3).
- _check_blosum62_gap_costs, a usage error (typer.BadParameter, exit
status 2, D20) naming the options:
- --gap-open between 1 and 2^20 and --gap-extend between 0 and 2^20,
the kernel's range (swint.MAX_GAP); the sign is ignored, as it always
was (the absolute values are checked); whole numbers
(alignment.whole_number: the Python functions may pass 11.0, which
the release took, but not 10.5 or True), and a --significance-mode
the enum knows (a Python caller may pass a string);
- under --significance-mode default, when the run needs significance
(an evalue or bits column in --format-output, or a finite
--max-evalue, 1e-3 by default), the gap costs must be 11/1. The
message names what needs significance and the ways out: --gap-open 11
--gap-extend 1; --significance-mode sampled; or --max-evalue inf
and/or a --format-output without evalue and bits, whichever apply.
The rule is alignment.check_gap_costs, the check align_candidates runs
again before it reads a sequence, so the command line refuses exactly
what align_candidates would refuse (tested over a grid of values).
The costs and the mode are checked first, each refusal naming its
option, so the 11/1 message is given for the 11/1 rule only.
- _check_significance_sample_size: under --significance-mode sampled,
when the run needs significance, --significance-sample-size must be
at least 2 (from 1 or 0 alignments lambda and K came out infinite or
NaN; a negative size failed after Stage 1, in the estimator). Both
commands' option also has min=2; the check serves the Python
functions.
- _check_blosum62_kernel: swint.load(), as resolve_esm_codes_database
loads the kernel for esm-codes; a missing (an install built without a
C compiler), stale or wrong-ABI library is a RefusalError (exit status
1) carrying the rebuild command, and the message says that
--no-seq-identity runs an embedding-only search without the kernel.
- Order: the option checks (exit status 2) first, then the stores
(_check_stage2_stores, unchanged), then the kernel, as under
esm-codes. An embedding-only search checks none of them: the gap
options stay unused there, as before, and it needs no kernel.
- query db now parses --format-output before its search whenever Stage 2
runs (the significance rule needs the columns): an unknown or empty
field is a usage error (exit status 2) before the database is loaded,
as in query sequences; it was a ValueError (exit status 1) after the
search.
- --help: --gap-open "1 to 1048576 (the sign is ignored)",
--gap-extend "0 to 1048576 (the sign is ignored)" (both said
"positive").
- query sequences still imports Lightning in set_log_level before these
checks, like every other refusal of the query commands.
Tests:
- tests/test_cli_blosum62_stage2.py (new): both commands through the
command line, on a database built with the fake ESM3 pass. Refused
before Stage 1 (ESM3 and the pass not reached, the database not loaded,
no shard or output written): the kernel range at both ends, either
sign; 11/1 under default significance for each reason (the default
columns and --max-evalue, the columns only, the filter only, bits
only, -10/-1); on rank 0 and another rank; a kernel that cannot be
loaded (RefusalError from KernelError, with the rebuild command); a
usage error before the kernel refusal; query db's --format-output.
Accepted, Stage 2 run to the end (fixture fmsw_libs): --significance-
mode sampled at 5/2, --max-evalue inf without evalue/bits at 5/2,
1/0 and 2^20/2^20; negative costs give the bytes of the positive ones
(-11/-1 under default significance included); --no-seq-identity
ignores the gap options and needs no kernel; --significance-sample-
size below 2 under sampled significance, refused before Stage 1.
PythonFunctionTest: query_database_from_database called from Python
refuses 10.5, True, 0.0, an unknown mode string, 10.0 under default
significance (named 10/1) and a sample size of -5 or 1 before the
search, and runs 11.0/1.0 to the bytes of 11/1. GapCostRuleTest: the
command line and check_gap_costs agree on 24,000 combinations (whole
and other floats and a bool among the gap costs); each refusal names
its own cause (a float, a bool or an unknown mode used to get the
11/1 message with empty slots); the sample-size rule. A negative
control with both gap-cost checks made no-ops failed all 27 gap-cost
refusal subtests.
- test_alignment GapCostsTest: align_candidates refuses before the
kernel, the fetch and any worker pool, serial and pooled (10.5 with a
TypeError).
- test_sequence_normalisation patches the kernel check as it patches
the store check; test_cli_search's two real-ESM3 tests that run
Stage 2 (25, 27) get the fmsw_libs fixture (needed since P9.S3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since P9.S3 the BLOSUM62 Stage 2 aligns on the same native library as esm-codes, so these messages became wrong: - --scoring esm-codes, step 4 of resolve_esm_codes_database (the kernel cannot be used): the refusal ended with "--scoring blosum62 still works with this database.", which is not so when the kernel cannot be loaded. It now says "--scoring blosum62 needs it too, for its Stage 2." The refusal itself (RefusalError, exit status 1, before ESM3, with the rebuild command) is unchanged. - The refusals of steps 1-3, which come before the kernel is loaded (no residue codes, an unreadable, incompatible or partial code store, an unknown system), ended with the same sentence, or with "--scoring blosum62 still works." (ResidueCodeStore.check_compatible, step 3): on an install without the kernel, that way out led to a second refusal. Both now say "--scoring blosum62 can query this database (its Stage 2 needs the native kernel too)." Wording only: the steps and their order are unchanged. check_faiss_count keeps "still works": it runs on rank 0 after step 4 has loaded the kernel. - The build hook's pure-Python fallback (no C compiler, without FOLDMATCH_REQUIRE_NATIVE=1): its warning and docstring said "BLOSUM62 search works, but --scoring esm-codes will refuse to run". They now say that the Stage 2 of a search (sequence identity, under --scoring blosum62 and esm-codes) refuses, and that everything else works. test_packaging matches only the warning's first words and the reinstall command, which are unchanged. Test: test_cli_esm_codes_query RefusalTest.test_missing_kernel asserts the new sentence and that the message no longer says BLOSUM62 still works; test_built_without_residue_codes and test_incompatible_stores assert the reworded sentence of steps 1 and 3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The golden digest of P9.S5 (swint.GOLDEN_SHA256, checked on every build and ISA by test_swkern.py and swkern's selftest, without biotite) froze the outputs of a library that test_swint.py had checked against biotite, but on other inputs: its synthetic profiles were never given to biotite themselves. test_swint.py, test_the_golden_set_against_biotite: every pair of swint.golden_cases() (129 calls, 477 pairs: random and related pairs at eight gap settings with gap_extend > gap_open among them, position-specific offsets, 60-code gaps both ways, the 10 pairs of the 32-bit path (one from the start with a profile column at -40,000), the 16-bit limit to the unit, the long lazy-F loop at 1000/1000, empty alignments and an empty target) is aligned by biotite 1.7.1 align_optimal(local=True, max_number=1) with the case's own position-specific scores (a query over its positions, the profile as an n x 24 substitution matrix), and the kernel's traceback-mode-1 outputs must equal it: score, alignment length, the path, end cell, identities, gap opens and the 'D' and 'I' column counts. selftest() already requires modes 1 and 2 to be identical and mode 0 to agree with them, so the digest is now that of biotite-checked outputs. 0 differences on the NEON library of an M1 Max; 1.7 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CHANGELOG, [Unreleased], Changed: - "The BLOSUM62 Stage 2 aligns on FoldMatch's native SIMD kernel: the same output, about 23 times faster" (P9.S1-S5, S8): fmswi_align in the compiled library; output-equivalent to biotite 1.7.1 align_optimal (local, affine gaps, ties); byte-identical Stage-2 tables on the full nbt.3988 benchmark (17,733,246 rows at 0.7, 9,503,077 at 0.8); Stage 2 486 s in the shipped configuration (cell dispatch, 'prefix' traceback; 468-486 s over the cell-dispatch runs) instead of 11,403 s on one Perlmutter GPU node with 128 workers; the kernel's own gains (largest query 72 s instead of 9,864 s, the end-of-run tail 4 s instead of 8,459 s, both measured with the old whole-query dispatch) told apart from the cell dispatch's (slices of about a quarter of each worker's share of the cells, a larger pair on its own: that tail under 1 s); 359 MB of traceback flags for the largest pair against biotite's 4.87 GB; initializer errors fail instead of hang; the selftest's "fmswi" digest; biotite kept for the matrix and the sampled statistics. The numbers are those of SIMD_STAGE2_REPORT.md (sections 4-6), measured with the same kernel DP (the C of P9.S1 differs from the measured build in comments, a local variable's name and the allocator of its workspace, posix_memalign instead of aligned_alloc). It replaces the unreleased "Faster BLOSUM62 Stage 2" entry (vectorised identity from biotite's trace), whose code P9.S3 removed. - "BREAKING - the BLOSUM62 Stage 2 needs the native kernel, and its gap costs are checked before Stage 1" (P9.S3, P9.S6): no kernel, no BLOSUM62 Stage 2 (it ran on biotite before), refused before Stage 1 with the rebuild command; --gap-open >= 1 (0 was accepted), the 2^20 bounds, the sign still ignored; from Python, whole numbers only (11.0 still 11, 10.5 refused: biotite truncated it); --significance-sample- size >= 2 (1 and 0 gave infinite or NaN significance, a negative size failed after Stage 1); the 11/1 rule unchanged but refused before Stage 1 as a usage error (exit status 2, was 1 after Stage 1); -11/-1 now 11/1; query db's --format-output checked before its search. - The --format-output entry no longer says that query db is unchanged, and the "Platform wheels" entry no longer says that a pure-Python install runs everything but esm-codes. README: - Features: the Smith-Waterman runs on FoldMatch's native SIMD kernel. - Requirements: Biotite 1.7.1 (it said 1.6.0; pyproject pins 1.7.1), and a C compiler for builds from source, with what the pure-Python fallback can and cannot do. - Two-stage search: Stage 2 aligns on the kernel, reproduces biotite's alignments, and without it is refused before the queries are embedded (--no-seq-identity still works); the gap-cost rules (--gap-open >= 1, --gap-extend >= 0, at most 1,048,576; 11/1 under default significance whenever significance is needed, and the ways out), checked before Stage 1; the sign of a gap cost is ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The four follow-ups that P9.A3 left (P9_CHAIN_PIECES_REPORT.md).
max_res. FoldMatch.residue_embedding_by_chain and
residue_embedding_by_assembly ('fm-search query structure --max-res')
tested max_res against every amino-acid residue of the structure or
assembly (_protein_length: free amino acids, chains under min_res and
the pieces a chain leaves out included), even with a chain ID, so a
chain within the limit was refused in a structure above it;
residue_embedding and structure_embedding tested the same, or a
chain-ID selection, the last one in src, free amino acids included.
structure_parser.protein_chains_with_sizes yields the chains of
protein_chain_iter, each with the polymer amino-acid count that
min_res_n is tested against (one counting path; protein_chain_iter and
get_protein_chains give what they gave). FoldMatch tests max_res against
the sum over the chains it is about to embed: those with chain_id (in an
assembly, one per copy), else every protein chain of the structure or
assembly. Over the limit, a ValueError before any ESM3 pass names the
count, the limit and what was counted. With no chain to embed, each
method gives what it gives without max_res (residue_embedding and
structure_embedding their ValueError, the other two {}), where the old
count could refuse it as too long. _protein_length and its biotite
imports go. The --max-res help says what is counted.
--min-res-n. 'inference residue-embedding', 'structure-embedding' and
'chain-embedding' took --min-res-n and did nothing with it: each row of
a chain list names its chain. The three no longer take it (Typer
refuses it, exit status 2); 'complete-embedding' keeps it, with the help
of 'assembly-embedding', and passes it to its assembly step only. In the
library, esm_inference.predict and structure_inference.predict given
chains, and chain_inference.predict given residue files, Parquet or an
inference_set, refuse a min_res_n other than 0 (RefusalError, from
inference.utils.refuse_min_res_n) before anything is read; signatures
unchanged. Every other min_res_n in src has an effect: it selects
chains or sequences, or records, checks or reports the min_res_n of a
pass or a store.
Chain lists. dataset.utils.read_chain_list reads the 4-column chain
lists of residue-embedding, structure-embedding, chain-embedding and
complete-embedding, and the file or stream of EsmProtFromChain, as
EsmProtFromChain read a file (pandas, no header, empty fields ''), a
short stream row like a short line, the embedding name defaulting to
<structure name>.<chain ID> when it is missing or empty: a 3-field row
made residue-embedding stop in torch.save on the file name '.pt', and
structure-embedding write an empty id. chain-embedding now reads the
residue embedding <embedding name>.<format> and writes under that name;
it read and wrote <structure>.<chain>, whatever the 4th column said,
and skipped lines of one or two fields. It also reads the list as
residue-embedding does, where it opened it as plain text and took each
stripped line's 1st and 3rd fields: a space, a tab or other whitespace
at a line's start or end is now part of a field, quoted fields and a
byte-order mark are read as CSV reads them, and a list whose name ends
in .gz, .bz2, .zip, .xz, .zst or .tar (letter case ignored) is read as
compressed. A row of more than 4 fields is refused, before any model
is built, by the four commands and by EsmProtFromChain (RefusalError,
naming the file or the stream and the row): pandas took the leading
fields of a first such row as the index, shifting the columns of every
row, and stopped at a later one with a ParserError that named no file,
while chain-embedding ignored the fields after the 3rd. Any other
ParserError is refused the same way, with pandas' message.
adapt_csv_to_embedding_chain_stream goes.
complete-embedding refuses, before any model is built, rows named
otherwise (RefusalError: up to 5 rows, fields quoted, and a count), as
its assembly step reads <structure>.<chain>, as assembly-embedding does.
ResidueEmbeddingFromTensorFile names a missing residue file with the
row's embedding name. The chain lists' help (--src-file of the three
commands, --src-chain-file) describes the columns and the names (no
more "(switch structure-location)" or "asym_i" there).
README: the FoldMatch class (it said RcsbStructureEmbedding), what
min_res and max_res do, the shapes [L + 2, 1536] with ESM3's BOS and EOS
rows, several chains one after another, {} when no chain qualifies, the
key '0' of an asymmetric unit, and a paragraph on the inference chain
lists.
What breaks: --min-res-n on the three commands; min_res_n other than 0
in the three library calls; chain-embedding reads and writes by the
embedding name, and reads its list as residue-embedding does (a space,
a tab or other whitespace at a line's start or end kept, quoted fields
and a byte-order mark read as CSV, a list whose name ends in .gz, .bz2,
.zip, .xz, .zst or .tar read as compressed: rename a plain-text one); a
row of more than 4 fields is refused by the four commands and by
EsmProtFromChain; complete-embedding refuses lists with other names.
CHANGELOG: two BREAKING entries under Changed, after P9.B5's, and one
under Fixed, after P9.A3's.
tests/test_inference_follow_ups.py, with no ESM3 (FoldMatch and the
inference modules run with stub models): max_res on 1acb's E (241) and I
(63) with a free GLU and a 6-residue chain that must not count, a chain
of exactly max_res and one residue fewer, residues that ESM3 is given
but min_res does not count (a free GLU in I's piece, a residue alone
between numbering gaps), E alone, I alone, both, the query with
--chain-id, two copies in an assembly, the messages and no ESM3 pass
when refused, nothing to embed at any max_res; --min-res-n refused by
the three commands, taken by assembly-embedding (also through the real
chain_inference) and by complete-embedding for its assembly step only,
the help of the five commands; the library refusals before any read or
model; custom names from residue-embedding to chain-embedding, 3-field
rows in all three commands and in complete-embedding, the residue-file
recipe, conventional lists as before, quoted fields, a missing residue
file and short lines, the complete-embedding refusal before any step
(its fields quoted, so that a structure name's leading space shows),
the reader (input order, spaces kept), and a blank chain ID; with model
builders that fail if called, rows of more than 4 fields (a first row
with a trailing comma, with a 5th field of text or of 6 fields, a later
row of 5, a row of 6 after a first row of 5) refused by the reader, the
four commands and EsmProtFromChain, naming the file and the row, stream
rows of 5 and 6 by the reader and of 5 by EsmProtFromChain, and another
pandas error (an unclosed quote) by the reader, without a claim of
extra fields; and lists of at most 4 fields read as before. Against
c0a2eaf, 26 of its 29 tests fail; the conventional-list, recipe and
assembly-embedding --min-res-n tests pass on both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The assembly lists of inference assembly-embedding and complete-embedding, and of ResidueAssemblyEmbeddingFromTensorFile and ResidueAssemblyEmbeddingFromParquet, are CSV without a header of 4 fields: structure name, structure file or URL, assembly ID, embedding name. A row of more was not refused. pandas read the leading fields of a first such row as the index, which shifted the columns of every row and ended in a misleading error: in a list of two entries, a trailing comma in "Structure file source is neither a recognized URL nor file: '1'" (the assembly ID read as the file), and 6 fields in a shared-name refusal of the name '1'. A list whose every row began with an index column gave the files of the list without it. A longer later row stopped with a bare ParserError that named no file, a stream row of 5 elements with pandas' ValueError. complete-embedding read its assembly list only in its assembly step, after its residue (ESM3) and chain steps had written their files. What changes: - read_chain_list's refusal is factored into _read_list (the columns, the list's name and fields), used by read_chain_list, whose frames and messages are unchanged, and by the new read_assembly_list (exported from foldmatch.dataset.utils), which both assembly readers call: a row of more than 4 fields is refused (RefusalError, naming the file, an open file's name or the stream, and the row), and so is any other pandas ParserError, quoted. pandas' call is unchanged (default NA values kept), so lists of at most 4 fields per row give the frames they gave. - complete-embedding reads an assembly list in a regular file not under /dev (read_assembly_list and refuse_shared_names) before any model is built, next to its chain-list name check, and passes the same file to its assembly step. A list given as a pipe (a named pipe, or <(...) on macOS), or on macOS as /dev/stdin or /dev/fd/N (open shares the descriptor's offset, even on a regular file), can be read once only: read early, it would reach the assembly step empty, and nothing would be embedded, with exit status 0 (a named pipe whose writer has exited would wait for another writer instead). So such a list is read by the assembly step alone, as on 9763f44, which still refuses a malformed one (after the chain step). assembly-embedding already built its reader before any model. - The help of assembly-embedding --src-file and complete-embedding --src-assembly-file describes the columns; one README sentence. - concatenate_tensors: a comment says why a residue file that cannot be loaded is skipped with a warning and not an error (by design: an assembly whose chains include some for which ESM3 could not be computed is still embedded from the chains that were); its code is unchanged. - CHANGELOG: a BREAKING entry under Changed, as a list whose every row begins with one or more extra fields (an index column, as pandas' to_csv writes unless index=False) worked on 9763f44 and is now refused; it says to write such a list with to_csv(header=False, index=False). Item (3) of P9.A4's chain-list entry is corrected the same way: residue-embedding, structure-embedding and EsmProtFromChain read such a chain list right on c0a2eaf, and P9.A4 refuses it. Not changed: concatenate_tensors' behaviour, the assembly readers' NA handling, and the 3-column structure lists and 2-column residue-file list, which have the same pandas call (a follow-up). Tests: tests/test_assembly_lists.py, 12 tests and 133 subtests. On 9763f44 the 6 tests that pin the refusal and complete-embedding's early read fail in every one of their 78 subtests (the shared-name test has none), and so do the help test (2 subtests) and read_assembly_list's frame test (15, ImportError); the readers' frames, and complete-embedding's files on conventional lists, with its assembly list in a regular file, in a named pipe or as /dev/fd/N, are the same on both trees. The P9.A4 tests of tests/test_inference_follow_ups.py pass unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A structure list holds 3 fields per row (structure name, structure file or URL, embedding name) and a residue file list 2 (residue embedding file, embedding name). Their readers, EsmProtFromStructure, EsmAssemblyFromStructure, ResidueEmbeddingFromStructure, ResidueAssemblyDatasetFromStructure and ResidueEmbeddingFromTensorFile, did not refuse a row of more, as chain lists (P9.A4) and assembly lists (P9.A5) do. pandas read the leading fields of a first such row as the index, which shifted the columns of every row; a chain list given to a structure list reader was such a list. A list whose every row began with an index column worked, as the columns lined up. A longer later row stopped with a bare ParserError that named no file, a longer stream row with pandas' ValueError. What changes: - read_structure_list and read_residue_file_list (exported from foldmatch.dataset.utils), on _read_list, are what all 5 readers call: a row of too many fields is refused, from a file (path, open file, text buffer) and from a stream (RefusalError, naming the source and the row, saying what a row holds and what to do), and so is any other pandas ParserError, quoted, without claiming extra fields. Each reader keeps its COLUMNS, attributes, sort, refuse_shared_names and NA handling. _read_list's texts now hold for 2, 3 and 4 columns; the frames and messages of chain and assembly lists are as they were. - The residue file list in a file is read as strings, as its stream and every other list are. Without dtype=str, pandas 3 reads an index column 0, 1, 2 as a RangeIndex, which cannot be told from no index, so the refusal would miss it on pandas 3 alone; one read of the file with dtype=str holds on every pandas. It also ends the mangling of names: in a list whose names all look like numbers, or all like booleans (any of them may be empty), 007 and 1.50 give 007.csv and 1.50.csv (before, 7.0.csv and 1.5.csv), TRUE gives TRUE.csv (before, True.csv), 1 next to an empty name 1.csv (before, 1.0.csv), and a space around a number stays in the name (before, dropped). 1 and 2 name the same csv and pt files as before; such ids are strings in JSON (before, numbers or booleans), and Parquet takes them (before, ArrowTypeError, nothing written). An empty name stays NaN. - CHANGELOG: a BREAKING entry under Changed, as a list whose every row began with extra fields worked and is now refused (write it with to_csv(header=False, index=False)). Not changed: each reader reads its list once, where it did, so no entry point moves a model load and a list in a named pipe reaches the readers as before; streams reach pandas as they did, so the commands' own streams give the same frames; NA handling, FASTA and Parquet inputs, a header row (not detected), and the README and CLI help, which describe none of these lists. Tests: tests/test_structure_lists.py, 30 tests and 576 subtests. On 8ded017, 18 tests fail: the 8 of the refusals (all their 387 subtests), the 8 of the names read as strings (with their 20 subtests) and the 2 of the new functions and constants (ImportError, with 36 subtests). The 12 that pin frames, streams and the files of conventional lists pass on both trees. tests/test_inference_follow_ups.py and tests/test_assembly_lists.py pass unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Help text only: no option, default or output changes. --log-level read "Number of nodes to use for inference of embeddings." in the 5 embedding commands of inference and in fm-search stats, cluster, build structures, query structure and query db. It now reads "Logging level.", as in nearly all the other commands. inference structure-embedding named a pandas DataFrame file, an option out-df-name that does not exist and a JSON file. It writes the Parquet file <output-name>-<rank>.parquet, one per process. inference residue-embedding, chain-embedding, assembly-embedding and complete-embedding named one output format, whatever --output-format said; complete-embedding's output paths said a single JSON file, where the default is one CSV file per chain or assembly. They now name the default, and say that csv and pt write one file per item, and parquet and json one file per process. chain-embedding and assembly-embedding also name the CSV residue embeddings they read (--res-embedding-format). Tests: tests/test_inference_outputs.py runs each command in each format it takes, with stub models, and checks the files against what the help says; tests/test_cli_help.py checks the texts, and --log-level in every command of the three CLIs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two breaking changes to inference, both in CHANGELOG.md. chain-embedding reads residue embeddings and no structure file, so --structure-format did nothing. It no longer takes it: Typer refuses it with exit status 2, as for --min-res-n in P9.A4. complete-embedding keeps the option, and passes it to its residue and assembly steps only. chain_inference.predict keeps structure_format, for src_from structure. complete-embedding's --output-assembly-name defaulted to chain-inference, the default of --output-chain-name. With --output-format parquet or json and one folder for --output-chain-path and --output-assembly-path, the assembly file replaced the chain file. The default is now assembly-inference. In any folder, a script that reads the default assembly file must use the new name, or pass --output-assembly-name chain-inference. csv and pt do not use the name. Tests: tests/test_inference_outputs.py, the refusal, the steps that get the option, residue files read whatever the list's structure file says, and both files kept in one folder in parquet and json. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One run of inference, or of its library, never writes two outputs to one file, an output onto one of its own inputs, or outside its output folder: such a run is refused (RefusalError, exit status 1, one Error: line) before any model is built, on every rank alike. Files of earlier runs are not checked: a rerun into the same folders runs as before. What changes: - foldmatch.utils.output_files, the shared helpers: the file name rules (D1), the probe of how a folder compares names (a mkstemp file, its swapped-case, NFD and ss-for-sharp-s spellings tried, removed in finally; in a folder that cannot be written, the same spellings of the names it holds, nothing written: its entries, then its own name in its parent, a spelling that is a sibling skipped), names compared exactly or by canonical caseless matching (NFD, case folded, NFC; full case folding, straße as strasse, where the folder folds so, as APFS does), folders compared as files (device and inode), OutputFiles (names and prefixes looked up, so the checks stay linear in the rows), check_outputs, and the refusals. - D1: with csv and pt, an embedding name that is empty, '.', '..', holds '/' or NUL, or makes a file name over 255 bytes is refused (an integer id, from Parquet, is judged as the file name it gives); the output name is checked in every format (<name>-<rank>.parquet with a 6-digit rank reserved, and .json.gz), its refusal naming what the name itself is. Parquet and JSON ids keep any characters (Q9/1-100). - D2: no output is the list file, a local structure file of the list, or a residue embedding the run reads. chain-embedding refuses --output-folder = --res-embedding-location in one format, naming both options; assembly-embedding refuses an assembly name <structure name>.<anything> written beside the residue embeddings in their format (1acb-1 stays allowed); chain_inference.predict refuses a Parquet input in out_folder named <out_name>-<digits>.parquet (ASCII digits, as the writers name ranks). An input reached through a symbolic or hard link from another folder is compared as a file with the output files that exist (chain-embedding on links to the residue embeddings in its output folder), and so is every input when one of those files is a link (an output's file linked to a residue embedding); assembly-embedding lists its residue files for this only when its output files exist. - D3: an embedding name given to more than one row is refused by every list reader, in every format (refuse_repeated_names, rows counted from 1, 5 shown); with csv and pt, names that the output folder takes for one file too (A and a on APFS; kept on a case-sensitive folder), structure list names, which name <name>.<chain ID>, included. Such a refusal names what the folder does not tell apart (case, or NFC and NFD) and what keeps the names apart (complete-embedding, which names chains <structure name>.<chain ID>: the folder only). chain_inference.predict refuses a Parquet id held twice. - complete-embedding checks its three steps before its residue step, and refuses two steps that would write one file in one folder (the residue and chain embeddings as pt, or one name for chains and assemblies in Parquet or JSON), and an output of any step over either list or a structure file of either. An assembly list that can be read once (a pipe, /dev/fd/N, a path under /dev) is read once, into memory, before the residue step, and its text given to the assembly step, so it is checked as a list in a regular file is. - D5: every list reader reads NA, nan and null as text and a missing field as ''; a missing or empty name is <structure name>-<assembly ID> (assembly list), the structure name (structure list), the file's name without its extension (residue file list), as the chain list already gave <structure name>.<chain ID>. read_structure_list drops keep_default_na. Help of the assembly list options and README. - D8: the CHANGELOG recipe --output-assembly-name chain-inference is removed. CHANGELOG: two BREAKING entries under Changed, one under Fixed. Tests: tests/test_name_conflicts.py, 46 tests, stub models; each refusal before any model with its message, and the same command without the conflict, reruns and complete-embedding's default names in one folder included; 20,000 structures in the output folder checked in well under a second; links either way, a read-only folder (with a sibling that links to it: simulated, and real on a case-sensitive file system only), the Greek letters that only canonical caseless matching folds. On f4db76b 44 fail (outright or through subtests); integer Parquet ids that name 101.csv pass there, as they guard a run that worked, and the real sibling test is skipped on macOS. Updated for D5: tests/test_structure_lists.py (the frames and the NaN names: nan.csv, keep_default_na) and tests/test_assembly_lists.py (the frames, the help text, the docstrings of the assembly list in a pipe). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fm-embedding from-sequences wrote each record under its id, the first word of its header: a repeated id kept the last record (csv, pt) or gave two rows of one id (Parquet, JSON), an id ../x wrote outside the output folder, Q9/1-100 stopped the run with a traceback, abc and ABC were one file on macOS, and a --fasta-file named like an output was replaced by it. from-structures kept one of chains A and a of one structure on macOS. All with exit status 0 but the traceback. What changes: - D3: from-sequences residue and chain refuse a repeated id among the records kept (--min-res-n), in every format, before ESM3, from the duplicates of the residue pass's pre-flight (preflight_fasta, the check strict_fasta_preflight makes; its other strict refusals, < > and non-ASCII, are not applied). The pre-flight is passed on to the dataset, which reads its offsets, so the file is scanned once more only for csv and pt. - D1 and D3 with csv and pt: an id that cannot name a file <id>.<fmt> of the output folder, and ids that the folder takes for one file (check_outputs, records counted from 1); D2: the FASTA file among the outputs. --output-name follows the file name rules in every command (the library check of P9.N1). - D4: CsvBatchWriter and TensorBatchWriter keep the files written in the run (claim_file), compared as the folder compares names, and raise RefusalError at a second write of one, naming the file: the first file is kept, the second item not written, the run stops (straße then strasse on APFS too). A name that cannot be a file name there is refused too. Per process, so not across ranks (documented). - CHANGELOG: two BREAKING entries under Changed, one under Fixed; README. fm-search is unchanged: its FASTA passes write Parquet, and a repeated query id under BLOSUM62 stays a warning. Tests: tests/test_name_conflicts_embedding.py, 9 tests, stub models: each refusal before any model with its message, the same command without the conflict, a rerun, and residue then chain runs into one folder; chains A and a, and straße and strasse in the writers, on this Mac; the writers with a faked case-sensitive or simply folding probe. On f4db76b all 9 fail. The refusal of abc and ABC names case alone and says what keeps the ids apart (P9.N1's message). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
query structure, embedding, sequences and db, cluster and similarity-graph wrote --output-file over anything: a file of the database they had just searched (<db path>.index, .metadata, ...), which every later command then failed on, or the query FASTA, structure or embedding file, with exit status 0. query embedding kept the results by query id, so of two vectors of one id one query disappeared from the output. What changes: - D6: each of these commands refuses, before any work (RefusalError, exit status 1), an --output-file that is an input: a file of a database it reads (<db path>.index, .metadata, .invlists, .sequences, .rescodes.*: database_files), the query FASTA or structure file, or a query embedding file that query embedding reads (the files stream_embeddings selects, --file-extension included). Compared as files, os.path.samefile with the existing output (a symbolic link, and DB.INDEX on macOS, are refused). An output that does not exist is no input; one an earlier command wrote is replaced, as before, even one named <db path>.hits.tsv beside the database. - D7: query embedding refuses, before the database is loaded, query embeddings in which two vectors share an id (refuse_repeated_query_ids in foldmatch.utils.data, which reads the Parquet id columns and the .pt and .csv stems; the file selection of stream_embeddings is factored into embedding_files), naming the ids and their files. - CHANGELOG: two BREAKING entries under Changed, one under Fixed; README. Not changed: an output file of an earlier command, the default output names of cluster and similarity-graph, fm-search build. Tests: tests/test_name_conflicts_search.py, 7 tests, no model loaded (databases built from vectors, query structure's embedding a stub, query sequences stopped at its ESM3 pass): each refusal with its message and the input unchanged, and the same command into another file, run twice; query embedding, cluster and similarity-graph run twice into <db path>.<command>.tsv. On f4db76b all 7 fail (the last one at its .rescodes.* refusal only: the reruns worked there). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On GitHub Actions, where GITHUB_ACTIONS is set (also FORCE_COLOR or
PY_COLORS), typer forces a terminal and rich writes colour codes into
help and error output. The codes split the words that the tests look
for ('--min-res-n'), so 7 tests failed there although the text was
right: 4 in test_cli_split_build, 2 in test_cli_residue_codes and 1
in test_download_models. publish.yaml runs hatch test on GitHub, so
they would have failed the CI job.
tests/conftest.py now sets _TYPER_FORCE_DISABLE_TERMINAL=1 when it is
loaded, before any test runs. typer reads it once, when
typer.rich_utils is first imported, and the CLIs that tests run as
subprocesses inherit it. It also drops TERMINAL_WIDTH, which would
override the COLUMNS that the help tests give the CLI. NO_COLOR is
not set, so pytest keeps its own colours in a local terminal.
Tests: the full suite with GITHUB_ACTIONS=true, slow tests on, passes
(1206 passed, 6 skipped, 0 failed). The CLI-output tests also pass
with FORCE_COLOR=1 and PY_COLORS=1, with TERMINAL_WIDTH=60 and
COLUMNS=60, with TERM=dumb, from inside tests/ and under
pytest-randomly. Under pytest-xdist (hatch test --parallel, not used
by publish.yaml) 5 tests fail with or without this change: their
subTest arguments (a Path, a LogLevel) cannot be sent between workers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
inference (residue-embedding, structure-embedding, chain-embedding, assembly-embedding, complete-embedding) was built for the RCSB PDB weekly update, not as a user tool. The README no longer documents it: its section, with the chain and assembly list rules, is removed, and the CLI overview names the two CLIs for users, fm-embedding and fm-search. The command itself is unchanged and still installed; inference --help and the CHANGELOG still describe it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FAISS 1.14.2 to 1.15.1 search only the first nlist // T coarse cells for the queries of a slice of fewer than T queries when nlist >= max(10000, 1024 * T): silent wrong hits on 2 to 16 threads with the default nlist 16384 (facebookresearch/faiss#5824). parallel_mode 3 calls the quantizer outside the region and gives the correct result at every thread count. The defect is likely solved in the next FAISS release: the upstream fix (facebookresearch/faiss#5824) was merged on 2026-10-31, but it is not in 1.15.1, the latest release and FoldMatch's pin. Re-examine this workaround when the faiss-cpu pin moves to a release that has it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fm-search cluster and similarity-graph failed on every ivf_pq
database ("direct map not initialized"): the graph read each chain
with reconstruct(i). They now read blocks of chains with reconstruct_n,
which needs no direct map, and search each block with one FAISS call;
the edges are kept with NumPy instead of a Python loop over every
neighbour. A block holds as many chains as fit a result budget of
about 200 MB (16,384 at k = 1000), and at most 16,384 chains, so both
its results and its embeddings stay bounded at any k.
On ivf_pq the decoded query is L2-normalised before the search, as in
query db: a score is the unit decoded query against the stored PQ code.
flat and hnsw are searched as stored. The edge rule is unchanged (an
edge comes from the search of its lower-numbered chain), and so is the
edge order: the graph is the same on hnsw, and on flat up to float
rounding. Cluster sizes come from np.bincount; the graph build and the
Leiden clustering log their elapsed time. The --threshold help no
longer says that scores run from 0 to 1 with 1.0 for identical chains.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fm-search cluster and similarity-graph kept each block's edges as int64 arrays, concatenated them into one array (the copy in which the 30.4M-chain ivf_pq run was OOM-killed) and handed that array to igraph, which converts it through Python objects. The kept node ids are now int32 when the largest id fits, int64 otherwise; the blocks are not concatenated, and igraph reads the edges from a generator of int tuples that releases each block once consumed. Equal weights share one Python float object, keyed by their float32 bit pattern (0.0 and -0.0 stay apart), when the kept weights span at most 2**24 bit patterns; past that, one float an edge, as before. On Linux, glibc's malloc_trim(0) is called after the igraph build and after the weights, to hand the freed blocks back to the OS; where the C library has no malloc_trim it does nothing. The graph is unchanged: the same edges in the same order, the same weights (values and type) and names, so the same GraphML, seeded clusters and TSV at the same FAISS thread count. The new tests compare the graph bit for bit with K2's build on hnsw, flat and ivf_pq at several block sizes and edge chunks, with the int64 fallback and with the weight table and its fallback, and check how the graph is built: no concatenation, igraph pulling the edges from the generator itself, every block released before each trim, and the trim called twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fm-search cluster gains --method {leiden,greedy}, Leiden the default.
Greedy picks representatives in database order: a chain in no cluster
yet becomes a representative and takes every chain of its k-NN row
(k = min(--max-neighbors, chains), searched as the graph build searches
it, ivf_pq queries normalised) that scores at least --threshold and is
in no cluster yet. It runs in the graph build's blocks, but a block
searches only its chains in no cluster at the block's start, and each
block's results are dropped before the next: memory grows with the
chains, not with the edges. No graph, no seed, no modularity. On hnsw
the partition is the sequential rule's bit for bit at any block size.
Cluster ids are numbered by decreasing size, ties by representative
index (0 the largest, as leidenalg already numbers Leiden's). The
greedy TSV gains a 4th column, representative; Leiden's file keeps its
3 columns and bytes. export_clusters streams its rows in chunks of
_EXPORT_CHUNK chains for both methods. The statistics add, for greedy,
the chains searched, the representatives whose k-th neighbour is kept
(n_saturated, their clusters may be cut; 0 when k = n), with a WARNING
above half, and the clusters of at least --max-neighbors chains.
BREAKING: --resolution and --seed are refused with --method greedy in
one message (exit status 2); --max-neighbors below 1 is refused by
cluster and similarity-graph before the load (FAISS raised a bare
AssertionError after it); --resolution's default is stated in its help.
New tests (tests/test_clustering_greedy.py): equality with a one-query-
per-chain reference of the rule on hnsw and on flat below the BLAS
switch at blocks of 1, 7 and above n, with the searched counts; planted
clusters; the size cap and saturation; cluster ids; ivf_pq invariants
against the result rows the method used; statistics; edge cases; the
4-column rows and the Leiden file as c40d4e1 wrote it at several export
chunks; the refusals through the CLI.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cluster_leiden calls Graph.community_leiden (objective "modularity", the edge weights, the resolution, beta 0.01) for a fixed number of iterations, 10 by default (_LEIDEN_ITERATIONS), instead of leidenalg run to convergence; leidenalg is no longer a dependency (pyproject, uv.lock, README). The generator is a private random.Random(seed), so Python's global random state is untouched; python-igraph's generator is set back to the random module afterwards. Seeds go through operator.index and must be at least 0. Cluster ids are by decreasing size, ties by smallest chain index, as a NumPy array (int32 below 2**31 chains; BREAKING: a list before; n_iterations must be at least 1, -1 was the default). build_similarity_graph records the smallest kept weight with a weak reference to its graph; cluster_leiden raises ValueError for such a graph with a negative weight, and cluster --method leiden refuses it after the build (exit status 2). A negative --threshold whose kept scores are all at least 0 still works. New cluster option --leiden-iterations (default 10, at least 1, Leiden-only: refused under --method greedy with --resolution and --seed in one message). --seed below 0 is refused before the load. Tests: LeidenIgraphTest in test_clustering.py; the Leiden reference of test_clustering_greedy.py is the in-test igraph call; the route, refusal and help tests cover --leiden-iterations; test_release_pins counts 12 pinned dependencies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 0.9.0 (2026-08-27) and 0.10.0 (2026-09-15) entries were still under [Unreleased]. Move them into their own sections: --max-evalue and the new --min-coverage / align_candidates defaults in 0.9.0, compositional bias correction in 0.10.0. Text unchanged; release links added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub no longer provides the macos-14 runner image, so kernel-macos and packaging (macos-14) fail at once with "The configuration 'macos-14-arm64-us-default' is not supported", and a release would build no macOS wheels. Use macos-15 in the three places. The wheels keep MACOSX_DEPLOYMENT_TARGET=11.0, so they still support macOS 11+. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hatch test installs the direct dependencies at their pins but the transitive ones at their newest releases: CI got click 8.5.1, which writes "No such option '--x'." where 8.3.1 (uv.lock) wrote "No such option: --x", and 46 cases failed. click is pinned at the uv.lock version (uv lock --check passes with the two added lock lines), and the five assertions match either wording (tests/click_messages.py). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build_lock removed the lock at the end when the file there had its own lock's (st_dev, st_ino). Linux reuses the inode of a deleted file at once, so a lock deleted during the build and created again by another build was removed (BuildLockTest failed on CI's Linux runner). The lock now holds a random lock_id, and only the lock with it is removed. The test adds a lock rewritten in place on the same inode, which fails the old code on every system. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Typer's resolve_path made --src-assembly-file /dev/fd/N, on Linux, the file the descriptor is open on, so refusals named that file instead of the path given (CompleteEmbeddingInputTest failed on CI's Linux runner); /dev/stdin from a pipe resolved to a pipe:[N] that does not exist. complete-embedding now resolves the path itself, keeping a path under /dev as given. The test also runs with os.path.realpath patched to resolve /dev/fd/N as Linux does, which fails the old code anywhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_equal_weights_share_one_float needs equal weights in the graph, and HnswSameGraphAsK2Test's fixture had them by chance only: 1 to 4 repeated float32 scores of 2,796 edges, by platform (FAISS's SIMD code), and the test failed on CI's Linux runner. Twenty vectors are now repeated, as in the ivf_pq fixture, so the repeats' scores are equal everywhere, and the test checks that the fixture has them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
js/dev-esm-code-alignmentsbrings ESM3 residue-code alignments to FoldMatch, puts the Stage-2 alignment on a native Smith-Waterman kernel, and reworks clustering so it scales to very large collections. It also contains a long series of input-validation and robustness fixes. 112 commits onmaster(3c3625c);CHANGELOG.md[Unreleased]has the full detail and every breaking change.ESM3 residue-code alignments (G, P1–P7)
fm-search download-models.esm-codes-v1scoring system and its E-value statistics.build sequences --residue-codesandbuild residue-codes --from-shards.query sequences --scoring esm-codes: a Stage 2 that aligns residue codes, with its own output fields.Native alignment kernel (P3, P9.S)
fmsw, a C Smith-Waterman library built at install time (hatchling hook), with a ctypes wrapper, a self-test, CI jobs and wheels.Database builds
build sequencessplit into resumable steps:sequences-embed,sequences-index,sequences-seal, with run records and launch checks.faiss-cpu1.15.1, with index-phase thread settings;uv.lockin sync.--cleanup-embeddingsto delete the intermediate shards after a build.Structures, assemblies and embedding files (P9.A, P9.B, P9.H, P9.N)
max_resand--min-res-ncount the residues actually embedded.inference,fm-embeddingandfm-searchrefuse names or output files that would overwrite their own inputs or outputs.Clustering (P9.K, P9.Q, P9.T, P9.U)
parallel_mode = 3; upstream fix Fix database-parallel kNN when the parallel region gets fewer threads facebookresearch/faiss#5824, not yet released).clusterandsimilarity-graphnow work onivf_pqdatabases, and search in batches instead of one chain at a time.malloc_trim), with identical results.cluster --method greedy: streaming greedy-representative clustering whose memory grows with chains, not edges. It clustered the 30.4M-chain library in 54 min using 5.6 GiB, where the graph route ran out of memory at 476 GiB.community_leidenwith--leiden-iterations(default 10), and leidenalg is dropped. It is 8–11x faster and uses less memory; partitions change about as much as a change of seed does.Breaking changes
All are listed with migration notes in the CHANGELOG. The main ones:
fm-search cluster: Leiden partitions and labels change for the same--seed.--seedmust be at least 0.--resolution,--leiden-iterationsand--seedare refused under--method greedy.inference,fm-embeddingandfm-search.inference chain-embeddingtakes no--structure-format, and--min-res-nis accepted only where it selects chains.Other
CHANGELOG.mdnow records releases 0.9.0 and 0.10.0 in their own sections; those entries had stayed under[Unreleased].pyproject.tomlstill says0.10.0. Bump it when this is released.Testing
🤖 Generated with Claude Code