Repository navigation
fm-embedding: residue codes with --output-codes-folder; fm-search skips subfolders - #7
Merged
Merged
Conversation
'fm-search build embeddings' and 'fm-search query embedding' read the
files of --embedding-folder named *.parquet, *.pt or *.csv
(utils.data.embedding_files, Path.glob). The glob lists a folder so named
too (the dataset folder x.parquet that Spark writes, a folder codes.pt
inside an embedding folder), which the readers then refused ("... is a
folder, not a Parquet file"; a .csv one "could not be read"), so the
whole embedding folder was refused.
embedding_files now skips such an entry, a symbolic link to a folder
included (a symbolic link to a file is a file, read as before), and logs
one WARNING that names the skipped folders (the first five, then a
count) and says that the files of such a folder are read by giving that
folder itself. A folder that then holds no embedding file keeps the
"No embedding files found ..." ValueError, which names the skipped
folders too. The WARNING is logged once per command: stream_embeddings
logs it (build embeddings); query embedding logs it from its first
listing (the inputs it refuses to overwrite) and passes
warn_folders=False to refuse_repeated_query_ids' listing and to
stream_embeddings. The --help of --embedding-folder of both commands
says that subfolders are not read.
The other listings by name are not the same listing and are unchanged:
the structure folders of fm-embedding and 'fm-search build structures'
(scan_structure_folder, EmbeddingComputer: structure files), the build's
own shards emb_<run id>-<rank>.parquet (run_shard_files,
shard_run_ids), split-build records and code stores, a database's files
(database_files, the existing-output refusal) and the residue files of
an assembly run (residue_files), which name the files they want.
Tests: tests/test_embedding_folders.py (skipped folders, a link to a
folder and one to a file, the WARNING once and its text, the error of a
folder of folders, --file-extension, build and query on a folder with
such folders: the index, metadata and hits of the folder without them);
the folder cases of test_parquet_embeddings and test_csv_embeddings now
expect a skip (the reader of a .csv itself still names a folder it is
given). CHANGELOG [Unreleased], Changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The files that fm-embedding from-sequences residue|chain
--output-codes-folder (P9.X3) writes to the codes folder, one uint8 [L, 8]
table of esm-codes target codes per sequence, in the layout of each output
format:
- CsvCodeWriter, TensorCodeWriter: <id>.csv (L lines of 8 integers, no
header or index; an empty file for an empty table) and <id>.pt (a CPU
uint8 tensor [L, 8], its own storage), named as the embedding writers
name an id's file; their refusals name "the residue codes of" the id
and advise writing "the residue codes" as parquet or json
(CoreBatchWriter._what, _ITEM, _ITEMS; an embedding writer's messages
are unchanged);
- ParquetCodeWriter: <codes name>-<rank>.parquet, columns id |
residue_codes (list<fixed_size_list<uint8, 8>>) | seq_sha1
(fixed_size_binary(20)), row groups by rows or a 256 MB byte budget, no
file for a rank without rows;
- JsonCodeStorage: <codes name>-<rank>.json.gz, records {id,
residue_codes} (orient records, gzip), as JsonStorage writes.
They read predictions (embeddings, ids, payloads, seq_sha1s) of a module
with the residue_codes output (residue_code_tables: whole 8-byte codes,
20-byte seq_sha1s) and refuse, at the start of the pass, a module without
it. residue_code_writer picks one per format.
residue_shards.code_table_schema: the Parquet schema, with the metadata of
a build shard (system name, encoding and system digests, foldmatch.pass)
but the kind foldmatch.residue_output = residue_code_table
(RESIDUE_CODE_TABLE). Every shard reader keys on that kind, so such a file
is refused wherever a run's shards are read, never misread: check_shards
(and iter_residue_codes, the query-encoding index, check_codes_footers)
names it "not an embedding shard", build residue-codes --from-shards finds
no shards with residue codes, the seal and the plain-footer check refuse
it.
fm-search's embedding readers, a safety net for code files moved into an
embedding folder: a Parquet code file is refused naming it as residue
codes (_parquet_file), and refuse_repeated_query_ids leaves a Parquet file
without an embedding column to stream_embeddings, which refuses it,
instead of reporting its ids as repeated query ids. A .pt code file (a
uint8 [L, 8] tensor) is refused by its type, whatever L (a one-residue
table would otherwise read as an 8-value vector), and the refusal of a
.csv or .pt table of 8 columns of integers 0-255 adds that it may be
residue codes. Each says that residue codes belong in a folder of their
own (--output-codes-folder) and to move them out of the embedding folder.
A one-residue .csv code file cannot be told from an 8-value vector.
utils/file_kinds.py: what an existing output file holds, told positively
from its first bytes (file_kind, by the name's suffix): EMBEDDINGS (chain,
residue, structure or assembly embeddings, a build's shards included),
RESIDUE_CODES, or None. Parquet: the footer only (the kind
residue_code_table is residue codes; columns id and embedding, a list of
floats or a list of lists of floats, are embeddings). .pt: the tensor,
loaded memory-mapped and weights_only on the CPU, its values not read
(uint8 [L, 8] is residue codes, a floating-point tensor embeddings). .csv:
the first line, in the first 4 KiB (a whole line of 8 integers 0-255 is
residue codes; numbers, at least one written as a float, embeddings; a
line not ended in the bytes read is judged without its last value and is
never codes). .json.gz: the first record, from the first 16 KiB
decompressed as far as they go (the key after id, residue_codes or
embedding, a list). An empty, cut short, corrupt or foreign file is of no
kind, and a prefix of a file is of its kind or of none, never of the
other: a file being written is never taken for the other kind. The
suffix is matched in any case (a case-insensitive folder takes X.PT for
x.pt). file_kinds.remembering: a file read within it is remembered while
it is unchanged (device, inode, size, mtime), for the next check of it
outside, which takes it from memory once and forgets it (a command
checks its folders before the library checks them again); nothing is
remembered outside it, or after a refusal within it. P9.X3 refuses a run
that would replace a file of the other kind.
Tests: tests/test_residue_code_files.py (writers, ranks, row groups, the
item writers' names and refusals, the refusals of every shard and
embedding reader; file_kind of every writer's files, of their prefixes
(Parquet and .pt of no kind until whole), of foreign files, the suffix
in upper case, and read once from one check to the next).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both commands gain --output-codes-folder and --output-codes-name (one help text for both). Giving --output-codes-folder turns the residue codes on: one ESM3 pass computes each sequence's embeddings, as without it, and its residue codes, the esm-codes target codes uint8 [L, 8] (S4, f_t) of rows 1..L of the same ESM3 output, which 'fm-search build sequences --residue-codes' stores. They go to that folder in --output-format (P9.X2): <id>.csv|pt, as the embeddings are <output-folder>/<id>.csv|pt, or <name>-<rank>.parquet|json.gz, the name --output-codes-name, by default <output-name>-codes (the output name itself is allowed; like --output-name, it names no csv or pt file). The output folder holds the embedding files only, so 'fm-search build embeddings' and 'query embedding' read a chain run's output folder as a plain run's. The codes folder must exist (typer, exit 2) and must not be the output folder, compared as files (output_files.same_folder_problem: a symbolic link, '.', a case variant on a case-insensitive volume): a usage error, typer.BadParameter on --output-codes-folder (exit 2, D20), naming both options and the folder, before anything is imported. One inside the other is allowed, both ways, whatever the subfolder's name: fm-search reads the files of a folder, not its subfolders (P9.X1). --output-codes-name without --output-codes-folder is a usage error (exit 2). The help says what the codes are, that a subfolder will do, that runs on several nodes or under srun/torchrun need 'fm-embedding download-models' first, and that missing codes end the run with exit status 1. output_files.check_code_outputs: refuse_same_folder (a ValueError, as the library's other argument errors), a codes folder that is not an existing folder (the library's callers have no typer check: refused before ESM3, not when the first code file is written), then check_outputs in the codes folder, as for the embeddings in theirs: the codes name names files of it whatever the format, the ids of csv and pt as the codes folder compares names, no input among the code files; last, refuse_folder_at_output: one folder inside the other must not be named as a file of the other's output (out/<id>.csv as the codes folder of a csv run, out/inference-0.parquet of a parquet run, or the output folder as a code file of the codes folder; by its path and its real path, as that folder compares names), which the writer would meet as a folder after the pass. The CLI passes the embedding files with the ids (csv, pt), the library those of the output name (parquet, json). The refusals of check_outputs name their folder (folder=, "output folder" by default, "codes folder" here). Modules: the payload code of ChainCompleteModule moves, unchanged, to the mixin ResiduePayloads (modules/residue_payloads.py), which EsmModule now uses too. With residue_output residue_codes and the head, EsmModule returns (embeddings, names, payloads, seq_sha1s), the embeddings the same tensors. code_files=True (fm-embedding) only changes the ERROR lines (residue codes, not payloads) and keeps the first ids without codes (residue_failed_ids): a protein whose codes cannot be computed gets an empty [0, 8] table. Missing codes end the run with an error, once every file of every rank is written: ResidueCodesPass.run wraps trainer.predict, then check_after_pass. With several ranks, each rank posts its count and first ids in the store of the default process group (the rendezvous store, still up after the pass; store operations only, no NCCL collective) and waits until every rank has posted (RANK_WAIT, a day), so that no rank fails while another one still writes: a launcher (Lightning's child observer, torchrun's agent, srun with KillOnBadExit) kills the other ranks once one exits with an error. Rank 0 then raises for the whole run and the other ranks log their count and return, so the exit status is 1 whatever the launcher, and no process waits for another's exit (rank 0 only waits, at most READ_WAIT, until every rank has read the reports, as its process may hold the store). A rank whose pass raises posts an error before it re-raises and counts as having read the reports (it never will), so no rank waits for it, rank 0's READ_WAIT included; rank 0 then raises a RefusalError naming it. If rank 0's process, which holds the store under Lightning's launcher and srun, has ended, the other ranks' wait fails at once with a RefusalError that says so. Under a strategy that spawns or forks its ranks (ddp_spawn, ddp_fork, ddp_notebook), the calling process has no process group and does not learn the ranks' counts: it logs a WARNING and raises nothing (documented). The error is errors.MissingResidueCodesError, a new RefusalError subclass (exit 1; a subclass so that a caller can tell it from a pre-flight refusal, after which nothing is written), naming how many sequences got no codes, on which ranks, the first ids and where their empty tables are; it carries count and ids of the run, rank_counts, rank, world_size and the predictions, as predict would have returned them. Library: sequence_inference.predict(out_codes_folder=, out_codes_name=, scoring_system=) and chain_complete_inference.predict(out_codes_folder=, out_codes_name=), named as out_folder and out_name are for --output-folder and --output-name; out_codes_name without out_codes_folder is refused; scoring_system on sequence_inference.predict is the system whose encoders compute the codes, used only with out_codes_folder, None (the default) selecting DEFAULT_ESM_CODES, esm-codes-v1, as on chain_complete_inference.predict; residue_output keeps its meaning and is refused together with out_codes_folder; scoring_system on chain_complete_inference.predict without residue_output or out_codes_folder is refused, naming both; structure input is refused; an out_codes_folder that is out_folder is a ValueError. With out_codes_folder, each prediction (return_predictions) is (embeddings, ids, codes, seq_sha1s), codes one CPU uint8 tensor [L, 8] per id, the object that its .pt code file holds ([0, 8] without codes): the modules and the code writers exchange the raw payload bytes during the pass, and ResidueCodesPass.run converts what trainer.predict returns (code_predictions, with the writers' residue_code_tables), inside the part that tells the other ranks of an error; MissingResidueCodesError carries the same predictions. The shard pass of a build or a query (residue_output) returns its payloads as the bytes its shards store, and no internal caller asks for predictions. residue_codes_pass is the pre-flight, before ESM3, every rank alike: the codes folder and name (check_code_outputs), the strict FASTA pre-flight of a residue pass, the bundle (fetched only where bundle_fetch_allowed allows), the head. The embedding writers and module are those of a plain run; the chain Parquet stays the legacy id | embedding. CLI: _refuse_fasta_conflicts also checks, with csv and pt, each id's code file in the codes folder; a run with --output-codes-folder reports every refusal as a RefusalError (D20), as 'build sequences --residue-codes' does; a plain run calls predict and raises as before. Files of the other kind: output_files.refuse_other_kind, the last check of check_outputs (writes=EMBEDDINGS by default, RESIDUE_CODES for the codes folder from check_code_outputs), so of every fm-embedding command, the legacy inference commands and every library function that writes embeddings, before any model and any output, on every rank alike: a run refuses (RefusalError, exit 1, D20) to replace a residue-code file in its output folder, with or without --output-codes-folder (a codes folder given as --output-folder), or an embedding file in its codes folder (an earlier run's output folder given as --output-codes-folder). Only the files that the output would replace are read (OutputFiles.existing: csv and pt <id>.<format>, parquet and json <name>-<rank> of any rank), every one, by file_kinds.file_kind (P9.X2), and positively only: a file of no kind (empty, cut short, being written by another rank of the run, corrupt, foreign) is replaced as before, and a file that the run writes is of the kind it writes there, so a rerun replaces its own files, on any number of ranks. A folder that also holds files of the other kind under other names is not refused (nothing of them is lost). The message names how many files, the folder, the first five with their kind, and what to do. check_outputs(replaced_names=): with csv and pt, the library's FASTA functions pass the ids of the records kept (fasta_item_ids, one scan of the FASTA file), which only tell the files the run would replace (their checks stay fm-embedding's), to the output folder's and, through residue_codes_pass and check_code_outputs, the codes folder's checks. The commands that check folders before the library checks them again (fm-embedding from-sequences with csv and pt, the legacy chain-embedding and complete-embedding) check them within file_kinds.remembering, so each file is read once in a run, and nothing is held in memory once the library has checked. Help: --output-folder of every fm-embedding command and --output-codes-folder say it in one sentence each. Tests: tests/test_embedding_residue_codes.py. With stub ESM3 rows (F2c) and the real encoders: every format of both commands, the output folder byte for byte that of a plain run, the code files mirroring the embedding files (csv, pt), the codes F2c's and the build store's, the layouts and Parquet provenance, min_res_n, the codes name, nested folders both ways (codes.parquet, codes.pt, codes.csv inside the output folder included); 'fm-search build embeddings' and 'query embedding' on a chain run's output folder (codes elsewhere, inside it, or inside it under such a name) give the index, metadata and hits of a plain run's; the same folder by path, symbolic link, '.', relative path and case variant (exit 2, before torch is imported, in a fresh process), in the library by symbolic link (a ValueError); every other refusal before any model, a missing codes folder; missing codes (exit 1, the message, the count, ids and predictions, every file written, the embeddings those of a plain run on the same rows); a plain run unchanged (module, writers, no bundle, no strict pre-flight). Library predictions: the codes uint8 tensors [L, 8] equal to the .pt files and to the bytes of the codes Parquet rows, for both predict functions, the empty table of a sequence without codes in MissingResidueCodesError.predictions; a plain run's predictions still (embeddings, ids), not converted, and the shard pass's payloads bytes; every format (json and each csv file included); the keyword names out_codes_folder and out_codes_name in the refusals; scoring_system without the codes and an unknown system refused by both functions. RealEsm3Test: real CPU ESM3, 16 runs and a build: embedding files identical with and without codes (.json.gz but for the gzip mtime), codes equal to the build store's and to F2c (19392/19392 bits), fm-search on the chain output folders identical to a plain run's. The rank reports with a store (rank 0 raises for the run, the others log; a rank whose pass raised, counted as a reader so that rank 0 does not wait for it; rank 0's pass raised; the store gone; a rank that never reports; rank 0 waiting for the reads; the WARNING of a spawn strategy's parent); a codes folder named as an embedding file, or the output folder as a code file, in every format of both commands, by case variant and through a symbolic link, and in the library. TwoRankSmoke (slow): --devices 2 as a command, and with one sequence's ESM3 rows a row short on rank 1 or on rank 0, also with rank 0 12 s slower than rank 1, under Lightning's launcher and under torchrun: every file written, exit status 1, rank 0's message once; and a library caller on two ranks (both predict functions, pt and parquet, and a sequence without codes on rank 1): each rank's predictions hold its ids' codes as uint8 tensors, equal to the .pt files and the Parquet bytes; a rerun into the same folders on two ranks (residue parquet, chain csv and pt), with and without the codes, is not refused. Files of the other kind (OtherKindTest, stub models): an earlier run's output folder as the codes folder (pt and csv, both commands: refused, nothing written, the embeddings intact, the message exactly); a codes folder as the output folder of a plain run and of a run with codes, every format; parquet and json, a codes name equal to the earlier embeddings' (refused) or the default (a mixed folder, allowed, nothing replaced); code files of other names; a rerun in every format of both commands, every file written again; files cut short, empty or foreign, and files rewritten while the check reads them (another rank), never refused; the <name>-<rank> files of other ranks; every fm-embedding command and the legacy 'inference residue-embedding' refusing code files in the output folder, before any model; the library (both predict functions, both halves, every format; esm_inference and assembly_complete_inference; csv and pt with min_res_n, a file of the other kind under a record left out not looked at, under a record kept refused); case variants of the id and the suffix on a case-insensitive volume, refused and named as on disk; each file read once in a run of the command, nothing remembered after it. tests/test_name_conflicts.py: check_outputs' stat budget counts one stat per output file that exists (the key of its kind, read once in a run). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README's fm-embedding section: what --output-codes-folder computes, the code files of each format in the codes folder, --output-codes-name, the folder rule (not the output folder, a usage error; nested allowed both ways, as fm-search reads a folder's files, not its subfolders, unless the inner folder is named as a file of the outer folder's output), the refusals, the bundle (on several nodes, download-models first), the empty table and the exit status 1 that follows once every file is written (with several ranks, from rank 0 for the whole run, once every rank has ended its pass), that the output folder stays readable by 'fm-search build embeddings', and that fm-search refuses code files moved among embeddings. CHANGELOG [Unreleased], Added (above P9.X1's Changed): --output-codes-folder and --output-codes-name, the exit statuses, the library keywords (out_codes_folder, out_codes_name, and scoring_system on sequence_inference.predict, by default esm-codes-v1), their checks, the predictions' codes as uint8 tensors [L, 8], and MissingResidueCodesError (raised on rank 0, with rank_counts; under a spawn or fork strategy the caller only logs a WARNING); the legacy inference CLI has no FASTA command, so no equivalent. That no run replaces a file of the other kind: README one sentence (an embedding file in the codes folder, a residue-code file in the output folder, with or without --output-codes-folder, refused before ESM3, exit status 1; a rerun replaces its own files); CHANGELOG Added, in the codes entry (the codes folder half), and Changed, a bullet (every fm-embedding command, the legacy inference commands and the library refuse to replace residue-code files in their output folder). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fm-embedding from-sequences residue|chaincan now write each sequence's residue codes (the esm-codes-v1 target codes thatfm-search build sequences --residue-codesstores, 8 bytes per residue) in the same ESM3 pass as the embeddings, into a folder of their own.--output-codes-folder PATHturns the codes on. It must exist and must not be--output-folder(compared on disk: symlinks,., case variants). Otherwise it is a usage error, exit 2, before anything is loaded. Nesting either way is allowed.--output-codes-namenames the parquet/json files (default<output-name>-codes). Without--output-codes-folderit is exit 2.--output-format:<codes-folder>/<id>.csv|pt(uint8 [L, 8]);<codes-name>-<rank>.parquet(id | residue_codes | seq_sha1, with system, digests and pass record in the schema metadata);<codes-name>-<rank>.json.gz.fm-search build embeddings/query embeddingread it as before.[0, 8]table. The run writes every file, then exits 1 with a message naming the sequences. On several ranks, the ranks meet in the process group's store after the pass and rank 0 raises for the whole run, so no rank is killed while another still writes. Verified under Lightning's launcher,srunandtorchrun.fm-embeddingcommand). Reruns into the same folders still replace their own files.out_codes_folder=/out_codes_name=onsequence_inference.predictandchain_complete_inference.predict, plusscoring_system=(defaultesm-codes-v1). Predictions become(embeddings, ids, codes, seq_sha1s), with codes as CPUuint8 [L, 8]tensors.MissingResidueCodesError(aRefusalError) is raised at the end.Also changed:
fm-search build embeddings/query embeddingnow skip subfolders of--embedding-foldernamed like embedding files (e.g. a Sparkx.parquetdataset folder), with one WARNING, instead of refusing the whole folder.See
CHANGELOG.md›[Unreleased]for the full description.Commits
fm-embedding from-sequences residue|chain --output-codes-folderTesting
fm-search buildresidue-code store.srun -n 2and Lightning--devices 2:Notes
srunwithKillOnBadExit=1, a refused run's step status can be 143 instead of 1, depending on which rank exits first. Every rank still prints the error..ptfile per rank on Lustre.🤖 Generated with Claude Code