diff --git a/.github/workflows/publish.yaml b/.github/workflows/publish.yaml index 8577a73..87835ad 100644 --- a/.github/workflows/publish.yaml +++ b/.github/workflows/publish.yaml @@ -13,7 +13,7 @@ jobs: hatch-test: name: Test on Python ${{ matrix.python-version }} runs-on: ["self-hosted", "buildchain"] - timeout-minutes: 120 + timeout-minutes: 180 env: # The editable install compiles the native kernel in place (hatch_build.py); a failed compile is an error here, # never a pure-Python install or skipped kernel tests. diff --git a/CHANGELOG.md b/CHANGELOG.md index e8dac3e..c6adb4c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,96 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). While the major version is `0`, breaking changes are released in a minor bump. +## [Unreleased] + +### Added + +- **`--output-codes-folder`** on `fm-embedding from-sequences residue` and + `fm-embedding from-sequences chain`: the same ESM3 pass also computes each + sequence's residue codes, the esm-codes-v1 target codes that `fm-search + build sequences --residue-codes` stores (8 bytes per residue), and writes + them to that folder, in `--output-format`: `.csv` (L lines of 8 + integers) or `.pt` (a `uint8` tensor `[L, 8]`), as the embeddings are + `/.csv|pt`, or `-.parquet` (`id + | residue_codes | seq_sha1`, `residue_codes` a `list>`, the scoring system, its digests and the pass record in the schema + metadata) or `-.json.gz`. Row `i` of a sequence's + codes is residue `i`, row `i + 1` of its residue embeddings (`[L + 2, + 1536]`, with ESM3's BOS and EOS rows). The folder must exist and must not be + `--output-folder`, compared as folders on disk (a symbolic link, `.` or a + case variant on macOS is the same folder): a usage error, exit status 2, + before anything is loaded. One inside the other is allowed, both ways + (`fm-search` skips subfolders, below), unless the subfolder is named as a + file of the other folder's output (refused below). The + embedding files are byte for byte those of a run without the option (a + `.json.gz` but for its gzip timestamp, as between any two runs), and the + output folder holds them only: `fm-search build embeddings` and `query + embedding` read a chain run's output folder as that of a plain run. + **`--output-codes-name`** names the parquet and json files (default + `-codes`; like `--output-name`, no csv or pt file); without + `--output-codes-folder` it is refused (exit status 2). Refused before ESM3, + exit status 1: a codes name that cannot name the files, ids (csv, pt) that + the codes folder takes for one file, a `--fasta-file` among the code files, + one folder inside the other under the name of one of the other's files + (`out/.csv` as the codes folder of a csv run, or + `out/inference-0.parquet` of a parquet run; the writer would meet a folder + after the pass), FASTA records with `<`, `>` or non-ASCII sequences, an + esm-codes bundle that cannot be fetched (only the process that starts a run + fetches it; on several nodes, run `fm-embedding download-models` first), and + a run that would replace an embedding file in the codes folder (an earlier + run's output folder given as the codes folder, say; each file that the run + would replace is read as little as tells its kind, and one of no kind, + empty or cut short, is replaced as before). A + sequence whose codes cannot be computed (ESM3's rows do not match its + length) is logged as an ERROR and gets an empty table (an empty file in + csv); the run writes every file, then ends with exit status 1 and a message + that names how many sequences got no codes, the first ids and where their + empty tables are. With several ranks, each rank that ends its pass posts + its count in the process group's store and waits for the others (at most a + day), and rank 0 then fails for the whole run while the other ranks log + their count and end, so that no rank fails while another still writes + (its launcher would kill it): exit status 1 under Lightning's launcher, + `srun` and `torchrun` alike. Library: `out_codes_folder=` + and `out_codes_name=` on `foldmatch.inference.sequence_inference.predict` + and `foldmatch.inference.chain_complete_inference.predict`, with the same + checks (an `out_codes_folder` that is `out_folder` raises `ValueError`), and + a codes folder that is not an existing folder refused before ESM3; + `scoring_system=` on the former too, as on the latter (default `None`, + `esm-codes-v1`; used only with `out_codes_folder`). The predictions are then + `(embeddings, ids, codes, seq_sha1s)`, `codes` one CPU `uint8` tensor `[L, + 8]` per sequence, as its `.pt` code file holds. Missing codes raise + `foldmatch.errors.MissingResidueCodesError` (a `RefusalError`) at the end, + on rank 0, with `count` and `ids` of the run, `rank_counts`, `rank`, + `world_size` and the `predictions`; under a strategy that spawns or forks + its ranks (`ddp_spawn`, `ddp_fork`, `ddp_notebook`) the calling process + does not learn the ranks' counts and only logs a WARNING. The code + Parquet files carry the kind `residue_code_table`, which the shard readers + of builds and queries refuse; `fm-search build embeddings` and `query + embedding` refuse code files moved into an embedding folder (a code Parquet + file and a code `.pt` are named as such). The legacy `inference` CLI has no + FASTA command, so no equivalent. + +### Changed + +- **`fm-search build embeddings` and `fm-search query embedding` skip + folders in `--embedding-folder`.** They read the files of the folder named + `*.parquet`, `*.pt` or `*.csv`, and listed a folder so named too (the + dataset folder `x.parquet` that Spark writes, say), which they then refused + (`... is a folder, not a Parquet file`), so the whole embedding folder was + refused. Such a folder, or a symbolic link to one, is now skipped, not read, + with one WARNING that names the skipped folders (the first five, and how + many more); a symbolic link to a file is read as before. To read the files + of such a folder, give that folder itself. A folder that then holds no + embedding file is refused as before (`No embedding files found ...`), its + message naming the skipped folders. `--help` of both says that subfolders + are not read. +- **Every `fm-embedding` command (and the legacy `inference` commands, and + the library functions that write embeddings) refuses to replace a file of + residue codes in its output folder** (as `--output-codes-folder` writes + them: a codes folder given as `--output-folder`, say), before ESM3, exit + status 1; a rerun into the same folder replaces its own embedding files as + before. + ## [0.11.0] - 2026-11-12 ### Added diff --git a/README.md b/README.md index 58a5515..b6c6d1c 100644 --- a/README.md +++ b/README.md @@ -191,6 +191,25 @@ Assembly-level embeddings are only available under `from-structures` — there i Each FASTA record's embedding is stored under its id, the first word of its header, so `from-sequences` refuses a repeated id before ESM3 is loaded. With `--output-format csv` or `pt` (one file per record or chain), an id is a file name: one that holds `/` or cannot name a file, ids that the output folder takes for one file (`abc` and `ABC` on macOS's default file system), and a `--fasta-file` named like an output are refused too, and a run stops at a second write of one file (chains `A` and `a` of one structure on macOS). Parquet and JSON keep any id. A rerun into the same folder replaces its own files, as before. +`from-sequences residue` and `from-sequences chain` take `--output-codes-folder FOLDER`: the same ESM3 pass also computes each sequence's residue codes, the 8-byte per-residue ESM3 codes that `fm-search build sequences --residue-codes` stores (esm-codes-v1 target codes: 8 integers, 0-255, per residue), and writes them to that folder. ESM3 runs once; the embedding files are byte for byte those of a run without the option, and the output folder holds them only, so `fm-search build embeddings` and `query embedding` read it as before (for chain embeddings). The codes folder must exist and must not be `--output-folder` (compared as folders on disk: a symbolic link, `.`, or another case of the name on macOS are the same folder; a usage error, exit status 2); a subfolder of it, or a folder that holds it, will do (`fm-search` reads the files of a folder, not its subfolders), unless it is named as a file of the other folder's output. The codes follow `--output-format`, in the layout of the embeddings, one `[L, 8]` table per sequence of `L` residues, one row per residue: row `i` of the codes is residue `i`, which is row `i + 1` of the residue embeddings (`[L + 2, 1536]`, ESM3's BOS and EOS rows included): + +| Format | Residue-code files in the codes folder | Layout | +|---|---|---| +| `csv` | `.csv`, as the embedding is `/.csv` | L lines of 8 integers, no header or index | +| `pt` | `.pt` | a CPU `uint8` tensor `[L, 8]` | +| `parquet` | `-.parquet` | columns `id`, `residue_codes` (`list>`), `seq_sha1`; the schema metadata names the scoring system, its digests and the pass (`--min-res-n`, device, ranks, versions) | +| `json` | `-.json.gz` | records `{"id", "residue_codes"}`, a list of 8 integers per residue | + +`--output-codes-name NAME` names the parquet and json files (default `-codes`; the output name itself is allowed, as the folders differ); like `--output-name`, it names no csv or pt file. It is refused without `--output-codes-folder`. Before ESM3 is loaded, a run with `--output-codes-folder` refuses a codes name that cannot name those files, ids (csv, pt) that the codes folder takes for one file, a `--fasta-file` named like a code file, one folder inside the other under the name of a file that the other one gets (`out/.csv` as the codes folder of a csv run, say), and FASTA records whose sequence holds `<`, `>` or non-ASCII characters (as the build does). Nor does any `fm-embedding` run replace a file of the other kind: before ESM3 (exit status 1), it refuses to replace an embedding file in its codes folder (an earlier run's output folder given as the codes folder, say) or a residue-code file in its output folder, with or without `--output-codes-folder`; files of other names are left alone, and a rerun into the same folders replaces its own files as before. The esm-codes bundle (~26 MB) is downloaded once, as for builds; on several nodes, or ranks started by `srun` or `torchrun`, run `fm-embedding download-models` first. A sequence whose codes cannot be computed (ESM3's rows do not match its length) is logged as an ERROR and gets an empty table (an empty file in csv); the run still writes every file, then exits with status 1, naming how many sequences got no codes, the first ids and where their empty tables are. With several devices or nodes, each rank that ends its pass waits for the others, and rank 0 then fails for the whole run (the other ranks log their count), so that no rank fails while another one still writes: the exit status is 1 under Lightning's own launcher, `srun` or `torchrun` alike. The code files are neither embeddings nor embedding shards: `fm-search build embeddings`, `query embedding` and `build residue-codes --from-shards` refuse them if they are moved among embedding files. + +```bash +mkdir -p out/seq-chain out/seq-codes +fm-embedding from-sequences chain --fasta-file seqs.fasta --output-folder out/seq-chain \ + --output-format parquet --output-codes-folder out/seq-codes +# out/seq-chain/inference-0.parquet (chain embeddings), out/seq-codes/inference-codes-0.parquet (residue codes) +fm-search build embeddings --embedding-folder out/seq-chain --output-db db/chains +``` + Run `fm-embedding [from-structures|from-sequences] [command] --help` for full options (batch size, accelerator, devices, output format, distributed settings, etc.). ### `fm-search` — build and query FAISS databases diff --git a/src/foldmatch/cli/embedding.py b/src/foldmatch/cli/embedding.py index 42b69c5..630b4f4 100644 --- a/src/foldmatch/cli/embedding.py +++ b/src/foldmatch/cli/embedding.py @@ -13,11 +13,12 @@ """ import os import sys +from contextlib import contextmanager, nullcontext from pathlib import Path import typer -from typing import Annotated, Optional, List +from typing import Annotated, Optional, List, Tuple from foldmatch import __version__ from foldmatch.cli.args_utils import DOWNLOAD_MODELS_HELP, DOWNLOAD_MODELS_SYSTEM_HELP, arg_devices, \ @@ -35,6 +36,28 @@ os.environ["TOKENIZERS_PARALLELISM"] = "false" +_OUTPUT_FOLDER_HELP = ( + "Output folder to store predictions. A run that would replace a file of residue codes there (as 'fm-embedding " + "from-sequences ... --output-codes-folder' writes them) is refused." +) +_OUTPUT_CODES_FOLDER_HELP = ( + "Also compute each sequence's residue codes in the same ESM3 pass and write them to this folder, which must exist " + "and must not be --output-folder (a subfolder of it will do): the 8-byte per-residue ESM3 codes that 'fm-search " + "build sequences --residue-codes' stores, 8 integers (0-255) per residue. They are written in --output-format, as " + "the embeddings are: .csv or .pt per sequence, or -.parquet or .json.gz. The " + "embedding files are those of a run without it, and --output-folder holds them only ('fm-search build embeddings' " + "and 'query embedding' read it as before). Row i of a sequence's codes is its residue i, which is row i + 1 of the " + "residue embeddings that 'from-sequences residue' writes (ESM3's BOS and EOS rows included). Refuses FASTA " + "records whose sequence holds '<', '>' or non-ASCII characters. A sequence whose codes cannot be computed gets an " + "empty table, and the run exits with status 1 once every file is written. A run that would replace an embedding " + "file there is refused. The esm-codes bundle (~26 MB) is " + "downloaded once; on several nodes, or under srun or torchrun, run 'fm-embedding download-models' first." +) +_OUTPUT_CODES_NAME_HELP = ( + "File name for storing residue codes, with --output-codes-folder. Used when output-format is parquet or json: " + "-.parquet or -.json.gz (default -codes)." +) + app = typer.Typer( add_completion=False, pretty_exceptions_enable=False, @@ -75,7 +98,7 @@ def from_structures_residue( file_okay=False, dir_okay=True, resolve_path=True, - help='Output folder to store predictions.' + help=_OUTPUT_FOLDER_HELP )], output_format: Annotated[OutFormat, typer.Option( help='Format of the output. Options: csv, pt, parquet, json. Parquet holds one row per chain, with one list of D values per residue.' @@ -155,7 +178,7 @@ def from_structures_chain( file_okay=False, dir_okay=True, resolve_path=True, - help='Output folder to store predictions.' + help=_OUTPUT_FOLDER_HELP )], output_format: Annotated[OutFormat, typer.Option( help='Format of the output. Options: csv, pt, parquet, json.' @@ -235,7 +258,7 @@ def from_structures_assembly( file_okay=False, dir_okay=True, resolve_path=True, - help='Output folder to store predictions.' + help=_OUTPUT_FOLDER_HELP )], output_format: Annotated[OutFormat, typer.Option( help='Format of the output. Options: csv, pt, parquet, json.' @@ -322,7 +345,7 @@ def from_sequences_residue( file_okay=False, dir_okay=True, resolve_path=True, - help='Output folder to store predictions.' + help=_OUTPUT_FOLDER_HELP )], output_format: Annotated[OutFormat, typer.Option( help='Format of the output. Options: csv, pt, parquet, json. Parquet holds one row per sequence, with one list of D values per residue.' @@ -330,6 +353,16 @@ def from_sequences_residue( output_name: Annotated[str, typer.Option( help='File name for storing embeddings. Used when output-format is parquet or json.' )] = 'inference', + output_codes_folder: Annotated[Optional[Path], typer.Option( + exists=True, + file_okay=False, + dir_okay=True, + resolve_path=True, + help=_OUTPUT_CODES_FOLDER_HELP + )] = None, + output_codes_name: Annotated[Optional[str], typer.Option( + help=_OUTPUT_CODES_NAME_HELP + )] = None, min_res_n: Annotated[int, typer.Option( help='Consider only sequences with at least residues.' )] = 0, @@ -355,24 +388,27 @@ def from_sequences_residue( help='Logging level.' )] = LogLevel.info ): + codes = _codes_options(output_codes_folder, output_codes_name, output_folder, output_name) from foldmatch.inference.sequence_inference import predict set_log_level(log_level) - predict( - # Its pre-flight, which the dataset reads its records' offsets from - fasta_file=_refuse_fasta_conflicts(fasta_file, output_folder, output_format, min_res_n), - min_res_n=min_res_n, - batch_size=batch_size, - num_workers=num_workers, - num_nodes=num_nodes, - accelerator=accelerator, - devices=arg_devices(devices), - out_format=output_format, - out_name=output_name, - out_folder=output_folder, - strategy=strategy, - return_predictions=False, - ) + with _refusals(codes is not None): + predict( + # Its pre-flight, which the dataset reads its records' offsets from + fasta_file=_refuse_fasta_conflicts(fasta_file, output_folder, output_format, min_res_n, codes=codes), + min_res_n=min_res_n, + batch_size=batch_size, + num_workers=num_workers, + num_nodes=num_nodes, + accelerator=accelerator, + devices=arg_devices(devices), + out_format=output_format, + out_name=output_name, + out_folder=output_folder, + strategy=strategy, + return_predictions=False, + **_codes_keywords(codes), + ) @from_sequences_app.command( @@ -392,7 +428,7 @@ def from_sequences_chain( file_okay=False, dir_okay=True, resolve_path=True, - help='Output folder to store predictions.' + help=_OUTPUT_FOLDER_HELP )], output_format: Annotated[OutFormat, typer.Option( help='Format of the output. Options: csv, pt, parquet, json.' @@ -400,6 +436,16 @@ def from_sequences_chain( output_name: Annotated[str, typer.Option( help='File name for storing embeddings. Used when output-format is parquet or json.' )] = 'inference', + output_codes_folder: Annotated[Optional[Path], typer.Option( + exists=True, + file_okay=False, + dir_okay=True, + resolve_path=True, + help=_OUTPUT_CODES_FOLDER_HELP + )] = None, + output_codes_name: Annotated[Optional[str], typer.Option( + help=_OUTPUT_CODES_NAME_HELP + )] = None, min_res_n: Annotated[int, typer.Option( help='Consider only sequences with at least residues.' )] = 0, @@ -425,36 +471,90 @@ def from_sequences_chain( help='Logging level.' )] = LogLevel.info ): + codes = _codes_options(output_codes_folder, output_codes_name, output_folder, output_name) set_log_level(log_level) from foldmatch.inference.chain_complete_inference import predict - predict( - # Its pre-flight, which the dataset reads its records' offsets from - src_stream=_refuse_fasta_conflicts(fasta_file, output_folder, output_format, min_res_n), - src_location=SrcLocation.file, - src_from=SrcEsmFrom.fasta, - min_res_n=min_res_n, - batch_size=batch_size, - num_workers=num_workers, - num_nodes=num_nodes, - accelerator=accelerator, - devices=arg_devices(devices), - out_format=output_format, - out_folder=output_folder, - out_name=output_name, - strategy=strategy, - return_predictions=False, - ) + with _refusals(codes is not None): + predict( + # Its pre-flight, which the dataset reads its records' offsets from + src_stream=_refuse_fasta_conflicts(fasta_file, output_folder, output_format, min_res_n, codes=codes), + src_location=SrcLocation.file, + src_from=SrcEsmFrom.fasta, + min_res_n=min_res_n, + batch_size=batch_size, + num_workers=num_workers, + num_nodes=num_nodes, + accelerator=accelerator, + devices=arg_devices(devices), + out_format=output_format, + out_folder=output_folder, + out_name=output_name, + strategy=strategy, + return_predictions=False, + **_codes_keywords(codes), + ) + + +def _codes_options(output_codes_folder: Optional[Path], output_codes_name: Optional[str], output_folder: Path, + output_name: str) -> Optional[Tuple[Path, str]]: + """The codes folder and the codes name (--output-codes-name, else default_codes_name) of a run with residue codes, + None without --output-codes-folder. Before anything is imported, usage errors (typer.BadParameter, exit status 2, + D20): --output-codes-name without --output-codes-folder, and --output-codes-folder that is --output-folder, + compared as files (same_folder_problem; the library raises the same message as a ValueError).""" + if output_codes_folder is None: + if output_codes_name is not None: + raise typer.BadParameter("names the residue-code files of --output-codes-folder, which is not given.", + param_hint="--output-codes-name") + return None + from foldmatch.utils.output_files import default_codes_name, same_folder_problem + problem = same_folder_problem(output_folder, output_codes_folder, _CODES_OPTIONS) + if problem is not None: + raise typer.BadParameter(problem, param_hint="--output-codes-folder") + return output_codes_folder, default_codes_name(output_name) if output_codes_name is None else output_codes_name + + +#: The folders of a run with residue codes, as its refusals name them. +_CODES_OPTIONS = ("--output-folder", "--output-codes-folder") + + +def _codes_keywords(codes: Optional[Tuple[Path, str]]) -> dict: + """The residue-code keywords of predict: none without --output-codes-folder, so a plain run calls it as before.""" + return {} if codes is None else {"out_codes_folder": codes[0], "out_codes_name": codes[1]} + + +def _refusals(codes: bool): + """A run with residue codes (--output-codes-folder) reports every refusal of its checks as a RefusalError (D20: + exit status 1), as 'fm-search build sequences --residue-codes' does; a plain run raises exactly what it raised + before.""" + return _as_refusals() if codes else nullcontext() + + +@contextmanager +def _as_refusals(): + from foldmatch.errors import RefusalError + try: + yield + except RefusalError: + raise + except (ValueError, OSError) as exc: + raise RefusalError(str(exc)) from exc -def _refuse_fasta_conflicts(fasta_file: Path, output_folder: Path, output_format: OutFormat, min_res_n: int): +def _refuse_fasta_conflicts(fasta_file: Path, output_folder: Path, output_format: OutFormat, min_res_n: int, + codes: Optional[Tuple[Path, str]] = None): """The FASTA checks of from-sequences, before ESM3 (P9.N), on every rank alike: a record id repeated among the records kept (at least min_res_n residues), in every format, found by the residue pass's pre-flight (preflight_fasta, its duplicates); with csv and pt, ids that cannot name a file . of the output folder, - or that the folder takes for one file (abc and ABC on APFS), and the FASTA file among the output files. The - pre-flight, whose offsets the dataset reads.""" + or that the folder takes for one file (abc and ABC on APFS), the FASTA file among the output files, and residue + codes in the files that the run would replace (refuse_other_kind). With ``codes`` (--output-codes-folder, the + codes name) and csv or pt, the same of the code files . of the codes folder, as that folder compares + names (embeddings in the files that the run would replace there), and neither folder inside the other named as + one of the other's files (check_code_outputs). The files read are remembered for the library's check of the same + folders (file_kinds.remembering). The pre-flight, whose offsets the dataset reads.""" from foldmatch.errors import RefusalError - from foldmatch.utils.output_files import ITEM_FORMATS, check_outputs + from foldmatch.utils.file_kinds import remembering + from foldmatch.utils.output_files import ITEM_FORMATS, check_code_outputs, check_outputs from foldmatch.utils.sequence import preflight_fasta, scan_fasta preflight = preflight_fasta(fasta_file, min_res_n) if preflight.duplicates: @@ -469,8 +569,14 @@ def _refuse_fasta_conflicts(fasta_file: Path, output_folder: Path, output_format if min_res_n <= 0 or length >= min_res_n: ids.append(name) labels.append(f"record {k}") - check_outputs(output_folder, output_format, None, names=ids, labels=labels, source=str(fasta_file), - inputs=[("the FASTA file", fasta_file)], what="FASTA id", given="those records") + inputs = [("the FASTA file", fasta_file)] + with remembering(): + outputs = check_outputs(output_folder, output_format, None, names=ids, labels=labels, + source=str(fasta_file), inputs=inputs, what="FASTA id", given="those records") + if codes is not None: + check_code_outputs(codes[0], output_format, codes[1], out_folder=output_folder, names=ids, + labels=labels, source=str(fasta_file), inputs=inputs, what="FASTA id", + given="those records", outputs=outputs, options=_CODES_OPTIONS) return preflight diff --git a/src/foldmatch/cli/inference.py b/src/foldmatch/cli/inference.py index 6147d5b..1bc8873 100644 --- a/src/foldmatch/cli/inference.py +++ b/src/foldmatch/cli/inference.py @@ -223,6 +223,7 @@ def chain_embedding( ): from foldmatch.dataset.utils import CHAIN_LIST_DEFAULT_NAME, read_chain_list, refuse_repeated_names, row_labels from foldmatch.inference.chain_inference import predict + from foldmatch.utils.file_kinds import remembering from foldmatch.utils.output_files import check_outputs set_log_level(log_level) @@ -230,8 +231,10 @@ def chain_embedding( chains = read_chain_list(src_file) refuse_repeated_names(chains, str(src_file), CHAIN_LIST_DEFAULT_NAME) _refuse_chains_over_residues(output_folder, output_format, res_embedding_location, res_embedding_format) - check_outputs(output_folder, output_format, output_name, names=list(chains.item_name), labels=row_labels(chains), - source=str(src_file), inputs=[("the chain list", src_file)], stem_what="--output-name") + with remembering(): # predict checks the folder again + check_outputs(output_folder, output_format, output_name, names=list(chains.item_name), + labels=row_labels(chains), source=str(src_file), inputs=[("the chain list", src_file)], + stem_what="--output-name") # The residue embedding of each row is the file of its embedding name, as residue-embedding writes it predict( src_stream=[(os.path.join(res_embedding_location, f"{name}.{res_embedding_format.value}"), name) @@ -553,6 +556,7 @@ def _refuse_conflicting_steps(src_chain_file: Path, src_assembly_file, assemblie read_chain_list, refuse_repeated_names, row_labels from foldmatch.errors import RefusalError from foldmatch.inference.utils import list_inputs + from foldmatch.utils.file_kinds import remembering from foldmatch.utils.output_files import check_outputs, file_id, probe_folding, refuse_assemblies_over_residues, \ same_folder, shown res_path, chain_path, assembly_path = folders @@ -582,22 +586,23 @@ def of_list(frame, src, list_name): # Chains that differ only as the folder does not tell apart (A and a of one structure): complete-embedding names # them ., and its residue step writes pt files, so only another folder keeps them apart named = "complete-embedding names each chain ." - steps = [ - ("--output-res-path", "residue", res_path, check_outputs( - res_path, OutFormat.pt, None, names=chain_names, labels=chain_rows, source=str(src_chain_file), - inputs=inputs, folding=folding(res_path), - folded_advice=f"give --output-res-path a folder of a file system that tells {{apart}} apart ({named}, " - f"and its residue step writes pt files)")), - ("--output-chain-path", "chain", chain_path, check_outputs( - chain_path, output_format, chain_name, names=chain_names, labels=chain_rows, source=str(src_chain_file), - inputs=inputs, folding=folding(chain_path), stem_what="--output-chain-name", - folded_advice=f"give --output-chain-path a folder of a file system that tells {{apart}} apart, or use " - f"--output-format parquet or json ({named})")), - ("--output-assembly-path", "assembly", assembly_path, check_outputs( - assembly_path, output_format, assembly_name, names=assembly_names, labels=assembly_rows, - source=assembly_source, folding=folding(assembly_path), stem_what="--output-assembly-name", - inputs=inputs)), - ] + with remembering(): # each step's predict checks its folder again + steps = [ + ("--output-res-path", "residue", res_path, check_outputs( + res_path, OutFormat.pt, None, names=chain_names, labels=chain_rows, source=str(src_chain_file), + inputs=inputs, folding=folding(res_path), + folded_advice=f"give --output-res-path a folder of a file system that tells {{apart}} apart ({named}, " + f"and its residue step writes pt files)")), + ("--output-chain-path", "chain", chain_path, check_outputs( + chain_path, output_format, chain_name, names=chain_names, labels=chain_rows, source=str(src_chain_file), + inputs=inputs, folding=folding(chain_path), stem_what="--output-chain-name", + folded_advice=f"give --output-chain-path a folder of a file system that tells {{apart}} apart, or use " + f"--output-format parquet or json ({named})")), + ("--output-assembly-path", "assembly", assembly_path, check_outputs( + assembly_path, output_format, assembly_name, names=assembly_names, labels=assembly_rows, + source=assembly_source, folding=folding(assembly_path), stem_what="--output-assembly-name", + inputs=inputs)), + ] refuse_assemblies_over_residues(assembly_names, assemblies.stream_name, out_folder=assembly_path, out_format=output_format, res_folder=res_path, res_format=OutFormat.pt, source=assembly_source, labels=assembly_rows) diff --git a/src/foldmatch/cli/search.py b/src/foldmatch/cli/search.py index db6935c..8efda0a 100644 --- a/src/foldmatch/cli/search.py +++ b/src/foldmatch/cli/search.py @@ -382,7 +382,8 @@ def build_database_from_embeddings( dir_okay=True, resolve_path=True, help='Folder of pre-computed chain or assembly embeddings: .pt or .csv files of one vector each, or ' - '.parquet files (columns id and embedding). Residue embeddings are refused.' + '.parquet files (columns id and embedding). Residue embeddings are refused. Subfolders are not ' + 'read (one named like an embedding file is skipped with a warning).' )], output_db: Annotated[str, typer.Option( help='Path to save the FAISS database.' @@ -1009,7 +1010,8 @@ def query_database_from_embedding( dir_okay=True, resolve_path=True, help='Directory of pre-computed chain or assembly embeddings: .pt or .csv files of one vector each, or ' - '.parquet files (columns id and embedding). Residue embeddings are refused.' + '.parquet files (columns id and embedding). Residue embeddings are refused. Subfolders are not ' + 'read (one named like an embedding file is skipped with a warning).' )], output_file: Annotated[str, typer.Option( help=_OUTPUT_FILE_HELP @@ -1044,6 +1046,7 @@ def query_database_from_embedding( set_log_level(log_level) # The embedding files it reads are inputs (P9.N D6), and one query per id: the results are kept by query id (D7) from foldmatch.utils.data import embedding_files, refuse_repeated_query_ids + # (this listing reports the folders that the embedding listings skip, once: stream_embeddings below does not) _refuse_output_over_inputs(output_file, databases=[("the database", db_path)], files=[("the query embedding file", f) for kind in embedding_files(embedding_folder, file_extension) for f in kind]) @@ -1062,6 +1065,7 @@ def query_database_from_embedding( path=embedding_folder, file_extension=file_extension, expected_dim=embedding_db.db.dimension, + warn_folders=False, ), top_k=top_k ) diff --git a/src/foldmatch/errors.py b/src/foldmatch/errors.py index 3882730..a4452b2 100644 --- a/src/foldmatch/errors.py +++ b/src/foldmatch/errors.py @@ -20,6 +20,33 @@ class RefusalError(ValueError): """ +class MissingResidueCodesError(RefusalError): + """A pass with residue codes in files of their own (``fm-embedding from-sequences residue|chain + --output-codes-folder``) ended with sequences whose codes could not be computed (ESM3 gave rows that do not match + the sequence's length, or the encoding failed). Raised at the end, after every file is written: those sequences' + codes are empty ``[0, 8]`` tables, their embeddings and every other output as in a complete run. A + :class:`RefusalError` (exit status 1: the data cannot serve the run), raised after the pass rather than before it, + so that a caller can tell it from a pre-flight refusal, after which nothing is written. With several ranks, rank 0 + raises it for the whole run, once every rank has ended its pass; the other ranks log their count and return. + + ``count`` is the number of such sequences in the run, ``ids`` the first of their ids (up to 10, by rank), + ``rank_counts`` the count of each rank, ``rank`` and ``world_size`` the process's rank and the number of ranks, + ``predictions`` what ``predict`` would have returned on this rank (None unless the caller asked for predictions): + ``(embeddings, ids, codes, seq_sha1s)`` per batch, ``codes`` one CPU ``uint8`` tensor ``[L, 8]`` per id, as its + ``.pt`` code file holds (``[0, 8]`` for those sequences). + """ + + def __init__(self, message: str, *, count: int = 0, ids=(), rank: int = 0, world_size: int = 1, + rank_counts=(), predictions=None): + super().__init__(message) + self.count = count + self.ids = tuple(ids) + self.rank = rank + self.world_size = world_size + self.rank_counts = tuple(rank_counts) + self.predictions = predictions + + class InternalError(RuntimeError): """An internal consistency check failed: a FoldMatch bug, not a problem with the input or the database. diff --git a/src/foldmatch/inference/chain_complete_inference.py b/src/foldmatch/inference/chain_complete_inference.py index 31b58f2..ca187a0 100644 --- a/src/foldmatch/inference/chain_complete_inference.py +++ b/src/foldmatch/inference/chain_complete_inference.py @@ -1,8 +1,11 @@ +import itertools +import json import logging import os +from dataclasses import dataclass from datetime import timedelta from pathlib import Path -from typing import Optional +from typing import Any, Iterable, List, Optional, Tuple import torch from lightning.pytorch.strategies import DDPStrategy @@ -12,18 +15,21 @@ from foldmatch.dataset.esm_prot_from_fasta import EsmProtFromFasta from foldmatch.dataset.esm_prot_from_structure import EsmProtFromStructure from foldmatch.dataset.utils import list_source, row_labels +from foldmatch.errors import MissingResidueCodesError, RefusalError from foldmatch.inference.utils import is_distributed, list_inputs from foldmatch.modules.chain_complete_module import ChainCompleteModule from foldmatch.search import scoring_system as _ss -from foldmatch.search.residue_shards import QUERY_ENCODING, check_residue_output +from foldmatch.search.residue_shards import QUERY_ENCODING, RESIDUE_CODES, check_residue_output from foldmatch.types.api_types import StructureFormat, Accelerator, Devices, Strategy, \ SrcEsmFrom, FileOrStreamTuple, SrcLocation, OutFormat from foldmatch.utils.data import identity_collate from foldmatch.utils.dist import launch_rank, srun_step_id from foldmatch.utils.model import get_residue_model, get_aggregator_model -from foldmatch.utils.output_files import check_outputs -from foldmatch.utils.sequence import FastaPreflight, preflight_fasta -from foldmatch.writer.batch_writer import ParquetBatchWriter, JsonStorage, TensorBatchWriter, CsvBatchWriter +from foldmatch.utils.output_files import ITEM_FORMATS, SHOWN, check_code_outputs, check_outputs, default_codes_name, \ + format_value +from foldmatch.utils.sequence import FastaPreflight, preflight_fasta, scan_fasta +from foldmatch.writer.batch_writer import ParquetBatchWriter, JsonStorage, TensorBatchWriter, CsvBatchWriter, \ + residue_code_tables, residue_code_writer def predict( @@ -44,6 +50,8 @@ def predict( return_predictions: bool = True, residue_output: Optional[str] = None, scoring_system=None, + out_codes_folder: Optional[Path] = None, + out_codes_name: Optional[str] = None, ): """Chain embeddings of structures or FASTA sequences (ESM3 + the residue aggregator). @@ -52,33 +60,76 @@ def predict( the system's name, its :class:`~foldmatch.search.scoring_system.EsmCodesSystem` (exactly the registry entry) or a :class:`~foldmatch.search.scoring_system.PreparedSystem` from :func:`prepare_scoring_system`; a build defaults to ``DEFAULT_ESM_CODES``, a query must name the database's system. Each prediction is then - ``(chain_embeddings, ids, payloads, seq_sha1s)`` and a Parquet output gets the residue columns - (:class:`ParquetBatchWriter`). Refused before ESM3 loads, on every rank alike: structure input (the encoders - were trained on sequence-only ESM3), an output other than Parquet, a system other than the registry entry of its - name or a bundle that does not match it, and FASTA input the method cannot take (duplicate ids, ``<``/``>``, - non-ASCII; :meth:`FastaPreflight.problems`). Any accelerator works, the Apple GPU (MPS) included (D18). + ``(chain_embeddings, ids, payloads, seq_sha1s)``, each payload the raw ``bytes`` that the shards store, and a + Parquet output gets the residue columns (:class:`ParquetBatchWriter`). Refused before ESM3 loads, on every rank + alike: structure input (the encoders were trained on sequence-only ESM3), an output other than Parquet, a system + other than the registry entry of its name or a bundle that does not match it, and FASTA input the method cannot + take (duplicate ids, ``<``/``>``, non-ASCII; :meth:`FastaPreflight.problems`). Any accelerator works, the Apple + GPU (MPS) included (D18). - Without ``residue_output`` nothing changes: same module, same writer, same outputs byte for byte. + ``out_codes_folder`` (fm-embedding's ``--output-codes-folder``) also computes each sequence's residue codes in + the same ESM3 pass, the esm-codes target codes ``uint8 [L, 8]`` of ``scoring_system`` (``None``, the default, + selects ``DEFAULT_ESM_CODES``, ``esm-codes-v1``: what ``fm-search build sequences --residue-codes`` stores), and + writes them to that folder, which is not ``out_folder``, in ``out_format``: ``.csv`` or ``.pt``, else + ``-.parquet`` or ``.json.gz`` (``out_codes_name`` by default ``-codes``; + :func:`residue_codes_pass`). The chain embeddings and their files are those of a run without it (the Parquet file + the legacy ``id | embedding``), and ``out_folder`` holds embeddings only. Each prediction (``return_predictions``) + is then ``(chain_embeddings, ids, codes, seq_sha1s)``: ``codes`` one CPU ``uint8`` tensor ``[L, 8]`` per id, the + object that its ``.pt`` code file holds (``[0, 8]`` for a sequence without codes), and ``seq_sha1s`` the 20-byte + SHA-1 of each normalised sequence. Not with ``residue_output``, the shard pass of a build or a query; refused + before ESM3 loads as :func:`residue_codes_pass` says (an ``out_codes_folder`` that is ``out_folder`` is a + ValueError; a code file that the call would replace and that holds embeddings is a RefusalError). A sequence + whose codes cannot be computed gets an empty ``[0, 8]`` table, and the call raises + :class:`~foldmatch.errors.MissingResidueCodesError` at the end, once every file is written + (:meth:`ResidueCodesPass.check_after_pass`; with several ranks, rank 0 raises it for the run, once every rank has + ended its pass), its ``predictions`` those this call would have returned. Not under a strategy that spawns or + forks its ranks (``ddp_spawn``, ``ddp_fork``, or ``ddp_notebook``, Lightning's choice for several devices in a + notebook): this process then does not learn the ranks' counts, so it logs a WARNING and raises nothing; each rank + logs its sequences without codes as ERRORs. + + Without ``residue_output`` or ``out_codes_folder`` nothing changes: same module, same writer, same outputs byte + for byte, same predictions ``(chain_embeddings, ids)``. Whatever the options, a file that the call would replace + in ``out_folder`` and that holds fm-embedding's residue codes is refused before ESM3 (RefusalError; + :func:`~foldmatch.utils.output_files.refuse_other_kind`). """ logger = logging.getLogger(__name__) residue_output = check_residue_output(residue_output) + if out_codes_folder is not None and residue_output is not None: + raise ValueError(f"out_codes_folder writes the residue codes to files of their own, and residue_output " + f"({residue_output}) to the shards of a build or a query: pass one or the other") + if out_codes_folder is not None and src_from != SrcEsmFrom.fasta: + raise ValueError(_STRUCTURE_INPUT.format(what="residue codes")) residue_head = None prepared = None + codes = None + inputs = [] + outputs = None if src_from == SrcEsmFrom.fasta: - # Before any model: the output name names files of the output folder, and none of them is the FASTA file - # (P9.N; the ids, and so the files of csv and pt output, are fm-embedding's to check) + # Before any model: the output name names files of the output folder, none of them is the FASTA file (P9.N; + # the ids, and so the files of csv and pt output, are fm-embedding's to check), and none that the run would + # replace holds residue codes (the ids tell those of csv and pt output) fasta = src_stream.path if isinstance(src_stream, FastaPreflight) else src_stream - check_outputs(out_folder, out_format, out_name, source=str(fasta), - inputs=[("the FASTA file", fasta)] if isinstance(fasta, (str, os.PathLike)) else ()) - if residue_output is not None: + inputs = [("the FASTA file", fasta)] if isinstance(fasta, (str, os.PathLike)) else [] + ids = fasta_item_ids(fasta, min_res_n, out_format, out_folder, out_codes_folder) + outputs = check_outputs(out_folder, out_format, out_name, source=str(fasta), inputs=inputs, + replaced_names=ids) + if out_codes_folder is not None: + codes = residue_codes_pass(src_stream, out_codes_folder, min_res_n=min_res_n, out_folder=out_folder, + out_format=out_format, out_name=out_name, out_codes_name=out_codes_name, + scoring_system=scoring_system, num_nodes=num_nodes, inputs=inputs, + outputs=outputs, replaced_names=ids) + src_stream = codes.preflight + elif out_codes_name is not None: + raise ValueError("out_codes_name names the residue-code files of out_codes_folder, which is not given") + elif residue_output is not None: prepared = check_residue_pass(residue_output, scoring_system, src_from=src_from, out_folder=out_folder, out_format=out_format, num_nodes=num_nodes) src_stream = strict_fasta_preflight(src_stream, min_res_n) logger.info(f"{residue_output} with {prepared.system.name}: loading the residue-code head") residue_head = prepared.load_head(device=torch.device("cpu")) elif scoring_system is not None: - raise ValueError("scoring_system is only used with a residue_output") + raise ValueError("scoring_system is only used with residue_output or out_codes_folder, and neither is given") if out_folder is not None: if out_format == OutFormat.parquet: @@ -92,6 +143,9 @@ def predict( inference_writer = CsvBatchWriter(out_folder) else: inference_writer = None + callbacks = [inference_writer] if inference_writer is not None else [] + if codes is not None: + callbacks.append(codes.writer(out_format)) inference_set = EsmProtFromStructure( src_stream=src_stream, @@ -126,21 +180,30 @@ def predict( aggregator_model = get_aggregator_model( device=torch.device("cpu") ) - module = ChainCompleteModule( - res_model=res_model, - aggregator_model=aggregator_model - ) if residue_output is None else ChainCompleteModule( - res_model=res_model, - aggregator_model=aggregator_model, - residue_head=residue_head, - residue_output=residue_output, - ) + if codes is not None: + module = ChainCompleteModule( + res_model=res_model, + aggregator_model=aggregator_model, + residue_head=codes.head, + residue_output=RESIDUE_CODES, + code_files=True, + ) + else: + module = ChainCompleteModule( + res_model=res_model, + aggregator_model=aggregator_model + ) if residue_output is None else ChainCompleteModule( + res_model=res_model, + aggregator_model=aggregator_model, + residue_head=residue_head, + residue_output=residue_output, + ) logger.info(f"rcsb-esm + rcsb-aggregator module ready") if strategy == Strategy.auto and is_distributed(devices, num_nodes): strategy = DDPStrategy(timeout=timedelta(days=1)) trainer = Trainer( - callbacks=[inference_writer] if inference_writer is not None else None, + callbacks=callbacks or None, num_nodes=num_nodes, accelerator=accelerator, devices=devices, @@ -149,6 +212,8 @@ def predict( ) logger.info(f"{src_from.name}-inference starts") + if codes is not None: + return codes.run(trainer, module, inference_dataloader, out_format, return_predictions) prediction = trainer.predict( module, inference_dataloader, @@ -227,13 +292,302 @@ def check_residue_pass(residue_output, scoring_system=None, *, src_from=SrcEsmFr if residue_output is None: raise ValueError("check_residue_pass needs a residue output") if src_from != SrcEsmFrom.fasta: - raise ValueError(f"{residue_output} needs sequence (FASTA) input: the residue-code encoders were trained on " - "sequence-only ESM3 embeddings, and structure input gives ESM3 coordinates") + raise ValueError(_STRUCTURE_INPUT.format(what=residue_output)) if out_folder is not None and out_format != OutFormat.parquet: raise ValueError(f"{residue_output} is written to Parquet shards only, not {OutFormat(out_format).value}") return prepare_scoring_system(scoring_system, residue_output=residue_output, num_nodes=num_nodes) +_STRUCTURE_INPUT = ("{what} needs sequence (FASTA) input: the residue-code encoders were trained on sequence-only " + "ESM3 embeddings, and structure input gives ESM3 coordinates") + + +@dataclass(frozen=True) +class ResidueCodesPass: + """What an fm-embedding pass with residue codes (``--output-codes-folder``) has after its pre-flight + (:func:`residue_codes_pass`): the strict FASTA pre-flight (the dataset reads its offsets), the prepared scoring + system, its residue-code head, the codes folder, the codes name and ``min_res_n``.""" + preflight: FastaPreflight + prepared: Any + head: Any + out_codes_folder: Any + out_codes_name: str + min_res_n: int + + def writer(self, out_format): + """The writer of the codes in ``out_format`` to the codes folder + (:func:`foldmatch.writer.batch_writer.residue_code_writer`): csv and pt ``.``; parquet + ``-.parquet`` with the system and the pass record (``min_res_n``, the pinned ESM3 revision + that ``load_head`` checked; the writer adds the device and the ranks); json ``-.json.gz``.""" + from foldmatch.utils.esm.loaders import ESM_REVISION + return residue_code_writer(out_format, self.out_codes_folder, self.out_codes_name, self.prepared.system, + min_res_n=self.min_res_n, esm_revision=ESM_REVISION) + + def run(self, trainer, module, dataloader, out_format, return_predictions): + """``trainer.predict`` of the pass, its predictions with the codes as tensors (:func:`code_predictions`; the + writers have stored the payloads during the pass), then :meth:`check_after_pass`; the predictions, None without + ``return_predictions``. A rank whose pass raises tells the other ranks so before it re-raises, and counts as + having read their reports (:func:`_post_report`), so that no rank waits for it at the end.""" + pass_id = next(_PASSES) + try: + predictions = trainer.predict(module, dataloader, return_predictions=return_predictions) + predictions = code_predictions(predictions, self.prepared.system.code_bytes) + except BaseException as exc: + _post_report(trainer, pass_id, {"error": type(exc).__name__}) + raise + self.check_after_pass(trainer, module, out_format, predictions, pass_id=pass_id) + return predictions + + def check_after_pass(self, trainer, module, out_format, predictions=None, *, pass_id: Optional[int] = None): + """After the pass, once this rank's writers have flushed (``trainer.predict`` has returned): the ranks tell + each other how many of their sequences got no codes and their first ids (:func:`_rank_reports`, through the + process group's store, not a collective of the backend; a rank waits there until every rank has ended its + pass, so that no rank fails while another still writes, which its launcher would then kill). Then rank 0 + raises if any sequence of the run got no codes (:class:`~foldmatch.errors.MissingResidueCodesError`, naming + how many, on which ranks, the first ids and where their empty tables are) or, with none, if another rank's pass + ended with an error (a :class:`~foldmatch.errors.RefusalError`; that rank raises its own error). Only rank 0 + raises, so the run's exit status is non-zero whatever the launcher (Lightning's, srun, torchrun) and no + launcher kills a rank that has not ended; the other ranks log their count as an ERROR and return. One rank: + no store, the same error.""" + count = int(getattr(module, "residue_failures", 0)) + ids = list(getattr(module, "residue_failed_ids", ())) + rank, world_size = int(getattr(trainer, "global_rank", 0)), int(getattr(trainer, "world_size", 1)) + reports = _rank_reports(trainer, {"count": count, "ids": ids[:_KEPT_IDS]}, + next(_PASSES) if pass_id is None else pass_id) + if reports is None: # several ranks without a process group (ddp_spawn's parent): this rank's own + logging.getLogger(__name__).warning( + f"This process started its {world_size} ranks with a spawn or fork strategy and does not learn their " + f"counts of sequences without residue codes: it raises no MissingResidueCodesError for them. Each " + f"rank has logged its own as ERROR lines; their codes are empty [0, 8] tables.") + reports, rank, world_size = [{"count": count, "ids": ids[:_KEPT_IDS]}], 0, 1 + counts = [int(report.get("count", 0)) for report in reports] + total = sum(counts) + errors = [r for r, report in enumerate(reports) if "error" in report] + if not total and not errors: + return + if rank != 0: + if count: + logging.getLogger(__name__).error( + f"{count} sequence(s) on rank {rank} of {world_size} got no residue codes; rank 0 ends the run " + f"with an error that names them") + return + failed = "" + if errors: + failed = (f"rank(s) {_ranks(errors)} of {world_size} ended the pass with an error, which they report " + f"themselves; their files may be incomplete.") + if not total: + raise RefusalError(f"R{failed[1:]} The files of the other ranks are written.") + run_ids = [i for report in reports for i in report.get("ids", ())][:_KEPT_IDS] + shown = run_ids[:SHOWN] + listed = ", ".join(repr(i) for i in shown) + (f" and {total - len(shown)} more" if total > len(shown) else "") + on = [r for r, n in enumerate(counts) if n] + if world_size <= 1: + where_ranks = "" + elif len(on) == 1: + where_ranks = f" on rank {on[0]} of {world_size}" + else: + where_ranks = (f" on {len(on)} ranks of {world_size} (" + + ", ".join(f"rank {r}: {counts[r]}" for r in on) + ")") + raise MissingResidueCodesError( + f"{total} sequence(s){where_ranks} got no residue codes: {listed}. ESM3 gave rows that do not match the " + f"sequence's length, or the encoding failed (each one's reason is logged above" + f"{', by its rank' if world_size > 1 else ''}). Their codes are written as empty [0, 8] tables " + f"({self._empty_tables(out_format, on)}); their embeddings and every other file of the run are written. " + f"Check those sequences, or leave them out of the FASTA file." + (f" Also, {failed}" if failed else ""), + count=total, ids=run_ids, rank=rank, world_size=world_size, rank_counts=counts, predictions=predictions) + + def _empty_tables(self, out_format, ranks) -> str: + """Where the empty tables of the sequences without codes of ``ranks`` are, in ``out_format``.""" + fmt = format_value(out_format) + folder = os.fspath(self.out_codes_folder) + if fmt in ("parquet", "json"): + suffix = ".parquet" if fmt == "parquet" else ".json.gz" + files = [os.path.join(folder, f"{self.out_codes_name}-{rank}{suffix}") for rank in ranks] + files = files[0] if len(files) == 1 else ", ".join(files[:-1]) + " and " + files[-1] + return f"{'rows' if fmt == 'parquet' else 'records'} of {files} without residues" + return f"{os.path.join(folder, f'.{fmt}')} of each, " + ("an empty file" if fmt == "csv" else + "a uint8 tensor [0, 8]") + + +def code_predictions(predictions, code_bytes: int = 8): + """What ``predict`` returns from a pass with residue codes in files of their own: each prediction ``(embeddings, + ids, payloads, seq_sha1s)`` of the pass as ``(embeddings, ids, codes, seq_sha1s)``, ``codes`` one CPU ``uint8`` + tensor ``[L, code_bytes]`` per id, the object that its ``.pt`` code file holds + (:func:`~foldmatch.writer.batch_writer.residue_code_tables`, as the code writers read the payloads; ``[0, + code_bytes]`` for a sequence without codes). None (no ``return_predictions``) stays None. Only this pass's + predictions are converted: the shard pass of a build or a query (``residue_output``) returns its payloads as the + ``bytes`` that the shards store.""" + if predictions is None: + return None + converted = [] + for prediction in predictions: + if prediction is None: + converted.append(None) + continue + _, tables, _ = residue_code_tables(prediction, code_bytes) + converted.append((prediction[0], prediction[1], tuple(torch.from_numpy(t) for t in tables), prediction[3])) + return converted + + +#: The ids of the sequences without codes that each rank tells rank 0 (MissingResidueCodesError.ids). +_KEPT_IDS = 10 +#: How long a rank that has ended its pass waits for the other ranks (as long as the chain pass's DDP timeout). +RANK_WAIT = timedelta(days=1) +#: How long rank 0 waits for the other ranks to read the reports (they have them as soon as rank 0 has). +READ_WAIT = timedelta(minutes=10) +#: The passes of this process, numbered alike on every rank (every rank runs the same passes): their store keys. +_PASSES = itertools.count() + + +def _ranks(ranks) -> str: + return ", ".join(str(r) for r in ranks) + + +def _default_store(): + """The store of the default process group (rendezvous; ``torch.distributed``), or None without one.""" + try: + import torch.distributed as dist + if dist.is_available() and dist.is_initialized(): + return dist.distributed_c10d._get_default_store() + except Exception: # noqa: BLE001 (no store: no exchange) + pass + return None + + +def _report_key(pass_id: int, rank) -> str: + return f"foldmatch/residue-codes/{pass_id}/{rank}" + + +def _post_report(trainer, pass_id: int, report: dict): + """Tell the other ranks the report of a pass ``pass_id`` that raised on this rank ({"error"}), without waiting + (best effort: none without a process group of several ranks). This rank never reads the others' reports, so it + also counts as having read them (:func:`_count_read`): rank 0 does not wait :data:`READ_WAIT` for it.""" + world_size = int(getattr(trainer, "world_size", 1)) + store = _default_store() if world_size > 1 else None + if store is None: + return + try: + store.set(_report_key(pass_id, int(getattr(trainer, "global_rank", 0))), json.dumps(report)) + _count_read(store, pass_id, world_size) + except Exception: # noqa: BLE001 (the error being raised is what matters) + pass + + +def _count_read(store, pass_id: int, world_size: int): + """Count one more rank done with the reports of pass ``pass_id`` (read, or never to read them: its pass raised); + the last one tells rank 0 (key ``all-read``).""" + if store.add(_report_key(pass_id, "read"), 1) == world_size: + store.set(_report_key(pass_id, "all-read"), "1") + + +def _duration(wait: timedelta) -> str: + """``wait`` for a message: "1 day", "10 minutes", "200 ms".""" + seconds = wait.total_seconds() + for unit, size in (("day", 86400), ("hour", 3600), ("minute", 60), ("second", 1)): + if seconds >= size: + n = seconds / size + return f"{n:g} {unit}{'' if n == 1 else 's'}" + return f"{seconds * 1000:g} ms" + + +def _rank_reports(trainer, report: dict, pass_id: int) -> Optional[list]: + """Every rank's report of pass ``pass_id``, by rank: ``report`` is this rank's ({"count", "ids"}; a rank whose + pass raised posts {"error"}). One rank: ``[report]``. Several: each rank sets its report in the store of the + default process group (the rendezvous that every launcher sets up, here still up after the pass) and waits until + every rank has set its own, at most :data:`RANK_WAIT`. Store operations only, no collective of the backend (no + NCCL watchdog, no teardown to wait for); a rank that dies inside the pass gets the others killed by its launcher + (Lightning's observer, torchrun's agent, srun with KillOnBadExit), or tells them (:meth:`ResidueCodesPass.run`, + :func:`_post_report`; it then counts as having read the reports); rank 0's death closes the store where rank 0's + process holds it (Lightning's launcher, srun), and the others' wait fails at once, with a RefusalError that says + so. Rank 0 then waits until every rank has read the reports (or posted an error), at most :data:`READ_WAIT`, so + that it does not end before them. None for several ranks without a process group.""" + world_size, rank = int(getattr(trainer, "world_size", 1)), int(getattr(trainer, "global_rank", 0)) + if world_size <= 1: + return [report] + store = _default_store() + if store is None: + return None + keys = [_report_key(pass_id, r) for r in range(world_size)] + try: + store.set(keys[rank], json.dumps(report)) + store.wait(keys, RANK_WAIT) + reports = [json.loads(store.get(key)) for key in keys] + _count_read(store, pass_id, world_size) + except Exception as exc: # noqa: BLE001 (a store timeout, or rank 0's store gone) + missing, reachable = [], True + for r, key in enumerate(keys): + try: + if not store.check([key]): + missing.append(r) + except Exception: # noqa: BLE001 + reachable = False + break + if not reachable: + why = "the process group's store could not be reached" + ( + " (rank 0's process holds it under Lightning's launcher and srun: rank 0 has most likely ended, with " + "an error of its own)" if rank else "") + elif missing: + why = (f"rank(s) {_ranks(missing)} did not report the end of its pass (a rank waits for that at most " + f"{_duration(RANK_WAIT)})") + else: + why = "the reports could not be read" + raise RefusalError( + f"Rank {rank} of {world_size} ended its pass, with its files written, but could not learn whether the " + f"other ranks' sequences got residue codes: {why}; {type(exc).__name__}: {exc}." + ) from exc + if rank == 0: + try: + store.wait([_report_key(pass_id, "all-read")], READ_WAIT) + except Exception as exc: # noqa: BLE001 (the reports are read: only the other ranks' reading is unsure) + logging.getLogger(__name__).warning(f"rank 0 of {world_size}: not every rank read the residue-code " + f"reports within {_duration(READ_WAIT)} ({type(exc).__name__}: " + f"{exc})") + return reports + + +def residue_codes_pass(src_stream, out_codes_folder, *, min_res_n: int = 0, out_folder=None, + out_format=OutFormat.csv, out_name: str = "inference", out_codes_name: Optional[str] = None, + scoring_system=None, + num_nodes: int = 1, src_from=SrcEsmFrom.fasta, + inputs: Iterable[Tuple[str, object]] = (), outputs=None, + replaced_names: Optional[Iterable[str]] = None) -> ResidueCodesPass: + """The pre-flight of a pass with residue codes (``from-sequences residue`` and ``chain`` with + ``--output-codes-folder``), before ESM3 loads, read-only and deterministic, so every rank refuses alike, in this + order: structure input (the encoders were trained on sequence-only ESM3); the codes folder and the codes name + (``out_codes_name``, by default :func:`default_codes_name` of ``out_name``; + :func:`~foldmatch.utils.output_files.check_code_outputs`: the codes folder is not ``out_folder``, compared as + files (a ValueError), and is an existing folder, the codes name names files of it, no input is a code file, no + code file that the pass would replace holds embeddings (``replaced_names``, the ids of csv and pt output, tell + those files: :func:`fasta_item_ids`), neither folder inside the other is named as one of the other's files; the + ids of csv and pt output are fm-embedding's to check, as for the embeddings); the strict FASTA pre-flight of a + residue pass (duplicate ids, ``<``/``>``, non-ASCII); the scoring system's bundle, fetched only where + :func:`bundle_fetch_allowed` says so, else it must be cached (``download-models``); then its residue-code head, on + the CPU (Lightning moves it with the module). ``inputs`` are the run's inputs (described, path), ``outputs`` the + embedding files (:func:`check_outputs` of ``out_folder``).""" + if src_from != SrcEsmFrom.fasta: + raise ValueError(_STRUCTURE_INPUT.format(what="residue codes")) + if out_codes_name is None: + out_codes_name = default_codes_name(out_name) + check_code_outputs(out_codes_folder, out_format, out_codes_name, out_folder=out_folder, inputs=inputs, + outputs=outputs, replaced_names=replaced_names) + preflight = strict_fasta_preflight(src_stream, min_res_n) + prepared = prepare_scoring_system(scoring_system, residue_output=RESIDUE_CODES, num_nodes=num_nodes) + logging.getLogger(__name__).info(f"residue codes of {prepared.system.name}: loading the residue-code head") + head = prepared.load_head(device=torch.device("cpu")) + return ResidueCodesPass(preflight, prepared, head, out_codes_folder, out_codes_name, min_res_n) + + +def fasta_item_ids(fasta, min_res_n: int, out_format, *folders) -> Optional[List[str]]: + """The ids of the records of a FASTA file that a pass keeps (at least ``min_res_n`` residues), which name its csv + and pt files ``.`` in each of ``folders`` (its output folder, its codes folder), so that its checks + know which files it would replace (one scan of the file); None for parquet and json, without a folder, or for + input that is not a path.""" + if format_value(out_format) not in ITEM_FORMATS or all(folder is None for folder in folders) or \ + not isinstance(fasta, (str, os.PathLike)): + return None + return [name for _, name, length in scan_fasta(fasta) if min_res_n <= 0 or length >= min_res_n] + + def strict_fasta_preflight(src_stream, min_res_n: int) -> FastaPreflight: """The FASTA pre-flight of a residue pass: ``src_stream`` (a path, or its :class:`FastaPreflight`) refused if it has duplicate ids, ``<``/``>`` or non-ASCII sequences, whose residues would not line up with stored codes.""" diff --git a/src/foldmatch/inference/sequence_inference.py b/src/foldmatch/inference/sequence_inference.py index d62c77b..63aa502 100644 --- a/src/foldmatch/inference/sequence_inference.py +++ b/src/foldmatch/inference/sequence_inference.py @@ -1,6 +1,7 @@ import logging import os from pathlib import Path +from typing import Optional import torch from torch.utils.data import DataLoader @@ -8,6 +9,7 @@ from foldmatch.dataset.esm_prot_from_fasta import EsmProtFromFasta from foldmatch.modules.esm_module import EsmModule +from foldmatch.search.residue_shards import RESIDUE_CODES from foldmatch.types.api_types import Accelerator, Devices, Strategy, OutFormat from foldmatch.utils.data import identity_collate from foldmatch.utils.model import get_residue_model @@ -29,14 +31,58 @@ def predict( out_name: str = 'inference', out_folder: Path = None, return_predictions: bool = True, + out_codes_folder: Optional[Path] = None, + out_codes_name: Optional[str] = None, + scoring_system=None, ): + """Residue embeddings of FASTA sequences (ESM3): one ``[L + 2, 1536]`` table per sequence, BOS and EOS included. + Each prediction (``return_predictions``) is ``(embeddings, ids)``. + + ``out_codes_folder`` (fm-embedding's ``--output-codes-folder``) also computes each sequence's residue codes in + the same ESM3 pass, the esm-codes target codes ``uint8 [L, 8]`` that ``fm-search build sequences + --residue-codes`` stores, and writes them to that folder, which is not ``out_folder``, in ``out_format``: + ``.csv`` or ``.pt``, else ``-.parquet`` or ``.json.gz`` (``out_codes_name`` by + default ``-codes``; :func:`~foldmatch.inference.chain_complete_inference.residue_codes_pass`). Row ``i`` + of a sequence's codes is residue ``i``, row ``i + 1`` of its embeddings. Each prediction is then ``(embeddings, + ids, codes, seq_sha1s)``: ``codes`` one CPU ``uint8`` tensor ``[L, 8]`` per id, the object that its ``.pt`` code + file holds (``[0, 8]`` for a sequence without codes), and ``seq_sha1s`` the 20-byte SHA-1 of each normalised + sequence. The embeddings and their files are those of a run without it. An ``out_codes_folder`` that is + ``out_folder`` (compared as files) is a ValueError; a code file that the call would replace there and that holds + embeddings is a RefusalError, before ESM3, as is a residue-code file in ``out_folder`` with or without the codes + (:func:`~foldmatch.utils.output_files.refuse_other_kind`). A sequence whose codes cannot be computed gets an + empty ``[0, 8]`` table, and the call raises :class:`~foldmatch.errors.MissingResidueCodesError` at the end, once + every file is written (with several ranks, rank 0 raises it for the run, once every rank has ended its pass), its + ``predictions`` those this call would have returned. Not under a strategy that spawns or forks its ranks + (``ddp_spawn``, ``ddp_fork``, or ``ddp_notebook``, Lightning's choice for several devices in a notebook): this + process then does not learn the ranks' counts, so it logs a WARNING and raises nothing; each rank logs its + sequences without codes as ERRORs. + + ``scoring_system`` is the scoring system whose encoders compute the codes, used only with ``out_codes_folder`` + (otherwise a ValueError): its name, its :class:`~foldmatch.search.scoring_system.EsmCodesSystem` or a + :class:`~foldmatch.search.scoring_system.PreparedSystem` + (:func:`~foldmatch.inference.chain_complete_inference.prepare_scoring_system`). ``None``, the default, selects + ``DEFAULT_ESM_CODES``, ``esm-codes-v1``, as for ``chain_complete_inference.predict`` and the builds (fm-embedding + has no option for it: it uses the default). + """ logger = logging.getLogger(__name__) - # Before the model: the output name names files of the output folder, and none of them is the FASTA file (P9.N; - # the ids, and so the files of csv and pt output, are fm-embedding's to check) + # Before the model: the output name names files of the output folder, none of them is the FASTA file (P9.N; the + # ids, and so the files of csv and pt output, are fm-embedding's to check), and none that the run would replace + # holds residue codes (the ids tell those of csv and pt output) + from foldmatch.inference.chain_complete_inference import fasta_item_ids, residue_codes_pass fasta = fasta_file.path if isinstance(fasta_file, FastaPreflight) else fasta_file - check_outputs(out_folder, out_format, out_name, source=str(fasta), - inputs=[("the FASTA file", fasta)] if isinstance(fasta, (str, os.PathLike)) else ()) + inputs = [("the FASTA file", fasta)] if isinstance(fasta, (str, os.PathLike)) else [] + ids = fasta_item_ids(fasta, min_res_n, out_format, out_folder, out_codes_folder) + outputs = check_outputs(out_folder, out_format, out_name, source=str(fasta), inputs=inputs, replaced_names=ids) + codes = None + if out_codes_folder is not None: + codes = residue_codes_pass(fasta_file, out_codes_folder, min_res_n=min_res_n, out_folder=out_folder, + out_format=out_format, out_name=out_name, out_codes_name=out_codes_name, + scoring_system=scoring_system, num_nodes=num_nodes, inputs=inputs, + outputs=outputs, replaced_names=ids) + fasta_file = codes.preflight + elif out_codes_name is not None or scoring_system is not None: + raise ValueError("out_codes_name and scoring_system are only used with out_codes_folder, which is not given") inference_set = EsmProtFromFasta( fasta_file=fasta_file, min_res_n=min_res_n @@ -56,6 +102,11 @@ def predict( ) module = EsmModule( model=esm_model + ) if codes is None else EsmModule( + model=esm_model, + residue_head=codes.head, + residue_output=RESIDUE_CODES, + code_files=True, ) logger.info(f"rcsb-esm module ready") @@ -70,8 +121,11 @@ def predict( inference_writer = CsvBatchWriter(out_folder) else: inference_writer = None + callbacks = [inference_writer] if inference_writer is not None else [] + if codes is not None: + callbacks.append(codes.writer(out_format)) trainer = Trainer( - callbacks=[inference_writer] if inference_writer is not None else None, + callbacks=callbacks or None, num_nodes=num_nodes, accelerator=accelerator, devices=devices, @@ -80,6 +134,8 @@ def predict( ) logger.info(f"sequence-inference starts") + if codes is not None: + return codes.run(trainer, module, inference_dataloader, out_format, return_predictions) prediction = trainer.predict( module, inference_dataloader, diff --git a/src/foldmatch/modules/chain_complete_module.py b/src/foldmatch/modules/chain_complete_module.py index 43c6035..1396509 100644 --- a/src/foldmatch/modules/chain_complete_module.py +++ b/src/foldmatch/modules/chain_complete_module.py @@ -1,22 +1,16 @@ -import hashlib import logging -import numpy as np from lightning import LightningModule -from foldmatch.search.residue_shards import QUERY_ENCODING_DTYPE, RESIDUE_CODES, check_residue_output +from foldmatch.modules.residue_payloads import ResiduePayloads from foldmatch.utils.data import collate_seq_embeddings from foldmatch.utils.esm.loaders import per_residue_embedding from foldmatch.utils.model import release_mps_cache -from foldmatch.utils.sequence import normalize_sequence, seq_sha1 logger = logging.getLogger(__name__) -# Rows whose residue payload failed are logged one by one up to this many per rank, then only counted. -_LOGGED_FAILURES = 10 - -class ChainCompleteModule(LightningModule): +class ChainCompleteModule(ResiduePayloads, LightningModule): """ESM3 per-residue embeddings -> chain embeddings, optionally with a per-residue payload (plan P5.1). The aggregator always gets ESM3's per-residue embeddings unchanged, BOS and EOS included. With @@ -29,7 +23,8 @@ class ChainCompleteModule(LightningModule): ``predict_step`` never raises for a residue payload: under DDP a rank that raised would leave the others waiting for the timeout. A protein whose ESM3 rows are not ``L + 2`` (the tokenizer did not keep one token per residue), whose sequence is not ASCII, or whose encoding failed gets an empty payload instead, which the code-store - validation refuses on rank 0 after the pass. + validation refuses on rank 0 after the pass (:class:`ResiduePayloads`; with ``code_files``, fm-embedding's + ``--output-codes-folder``, it is written as an empty ``[0, 8]`` table). """ def __init__( @@ -38,27 +33,16 @@ def __init__( aggregator_model, residue_head=None, residue_output=None, + code_files=False, ): super().__init__() self.esm3 = res_model self.aggregator = aggregator_model - residue_output = check_residue_output(residue_output) - if (residue_head is None) != (residue_output is None): - raise ValueError("residue_head and residue_output go together: pass both (a residue pass) or neither") - self.residue_head = residue_head - self.residue_output = residue_output - self.residue_failures = 0 + self._init_residue_payloads(residue_head, residue_output, code_files) def on_predict_start(self): logger.info(f"ESM + Aggregator device: {self.device} | {self.trainer.global_rank + 1}/{self.trainer.world_size}") - def on_predict_end(self): - if self.residue_failures: - logger.error( - f"{self.residue_failures} protein(s) on this rank got an empty {self.residue_output} payload; " - f"the residue store validation will refuse them" - ) - def predict_step(self, prot_batch, batch_idx): prot_embeddings = [] prot_names = [] @@ -71,41 +55,6 @@ def predict_step(self, prot_batch, batch_idx): if self.residue_output is None: release_mps_cache(self.device) return chain_embeddings, tuple(prot_names) - payloads = [] - digests = [] - for (esm_prot, name), embeddings in zip(prot_batch, prot_embeddings): - payload, digest = self._residue_payload(esm_prot.sequence, name, embeddings) - payloads.append(payload) - digests.append(digest) + payloads, digests = self._residue_payloads(prot_batch, prot_embeddings) release_mps_cache(self.device) - return chain_embeddings, tuple(prot_names), tuple(payloads), tuple(digests) - - def _residue_payload(self, sequence, name, embeddings): - """``(payload bytes, 20-byte seq_sha1)`` of one protein; an empty payload instead of any exception.""" - sequence = sequence or "" - if not sequence.isascii(): - # S15 refuses non-ASCII input before the pass; the row still gets a 20-byte seq_sha1 (of the UTF-8). - digest = hashlib.sha1(normalize_sequence(sequence, allow_non_ascii=True).encode("utf-8", "surrogatepass")) - return self._failed(name, "the sequence is not ASCII"), digest.digest() - digest = seq_sha1(sequence) - if embeddings.dim() != 2 or embeddings.shape[0] != len(sequence) + 2: - return self._failed(name, f"ESM3 gave {tuple(embeddings.shape)} rows for {len(sequence)} residues " - f"(expected {len(sequence) + 2} with BOS and EOS)"), digest - try: - rows = embeddings[1:-1] # S1: residues 1..L, BOS and EOS removed (a view; the aggregator input is untouched) - if self.residue_output == RESIDUE_CODES: - payload = self.residue_head.encode_target(rows).cpu().numpy() - else: - payload = np.ascontiguousarray(self.residue_head.encode_query(rows).cpu().numpy(), - dtype=QUERY_ENCODING_DTYPE) - return payload.tobytes(), digest - except Exception as e: # noqa: BLE001 (never raise inside the distributed pass; validated after it) - return self._failed(name, f"{type(e).__name__}: {e}"), digest - - def _failed(self, name, reason): - self.residue_failures += 1 - if self.residue_failures <= _LOGGED_FAILURES: - logger.error(f"{name}: no {self.residue_output} payload ({reason}); the row is written empty") - elif self.residue_failures == _LOGGED_FAILURES + 1: - logger.error(f"more proteins without a {self.residue_output} payload; only counting them from now on") - return b"" + return chain_embeddings, tuple(prot_names), payloads, digests diff --git a/src/foldmatch/modules/esm_module.py b/src/foldmatch/modules/esm_module.py index f2dffab..b51f125 100644 --- a/src/foldmatch/modules/esm_module.py +++ b/src/foldmatch/modules/esm_module.py @@ -2,27 +2,44 @@ from lightning import LightningModule +from foldmatch.modules.residue_payloads import ResiduePayloads from foldmatch.utils.esm.loaders import per_residue_embedding from foldmatch.utils.model import release_mps_cache logger = logging.getLogger(__name__) -class EsmModule(LightningModule): +class EsmModule(ResiduePayloads, LightningModule): + """ESM3 per-residue embeddings, ``(L + 2, 1536)`` per protein with BOS and EOS: each step returns + ``(embeddings, names)``. + + With ``residue_output`` ``"residue_codes"`` and the scoring system's ``residue_head`` (fm-embedding's + ``--output-codes-folder``), each protein's residue codes are also computed from the same ESM3 rows + (:class:`ResiduePayloads`, the embeddings returned unchanged), and a step returns ``(embeddings, names, payloads, + seq_sha1s)``. + """ def __init__( self, - model + model, + residue_head=None, + residue_output=None, + code_files=False, ): super().__init__() self.esm3 = model + self._init_residue_payloads(residue_head, residue_output, code_files) def on_predict_start(self): logger.info(f"ESM device: {self.device} | {self.trainer.global_rank + 1}/{self.trainer.world_size}") def predict_step(self, prot_batch, batch_idx): embeddings = tuple([self.__compute_embeddings(esm_prot) for esm_prot, name in prot_batch]) + if self.residue_output is None: + release_mps_cache(self.device) + return embeddings, tuple([name for esm_prot, name in prot_batch]) + payloads, digests = self._residue_payloads(prot_batch, embeddings) release_mps_cache(self.device) - return embeddings, tuple([name for esm_prot, name in prot_batch]) + return embeddings, tuple([name for esm_prot, name in prot_batch]), payloads, digests def __compute_embeddings(self, esm_prot): - return per_residue_embedding(self.esm3, esm_prot) \ No newline at end of file + return per_residue_embedding(self.esm3, esm_prot) diff --git a/src/foldmatch/modules/residue_payloads.py b/src/foldmatch/modules/residue_payloads.py new file mode 100644 index 0000000..6b8ce69 --- /dev/null +++ b/src/foldmatch/modules/residue_payloads.py @@ -0,0 +1,102 @@ +"""The per-residue payload of an ESM3 pass (plan P5.1), shared by the modules that compute it from ESM3's rows.""" +import hashlib +import logging + +import numpy as np + +from foldmatch.search.residue_shards import QUERY_ENCODING_DTYPE, RESIDUE_CODES, check_residue_output +from foldmatch.utils.sequence import normalize_sequence, seq_sha1 + +logger = logging.getLogger(__name__) + +# Rows whose residue payload failed are logged one by one up to this many per rank, then only counted. +_LOGGED_FAILURES = 10 + + +class ResiduePayloads: + """A LightningModule mixin: with ``residue_output`` (``"residue_codes"`` or ``"query_encoding"``) and the + ``residue_head`` of the scoring system (:class:`foldmatch.model.residue_code_encoder.ResidueCodeHead`, a submodule, + so Lightning moves it with the module), the rows ``1..L`` of each protein's ESM3 embeddings (S1) are encoded, one + protein at a time: S4 target codes (``L * 8`` bytes) or S5 query encodings (``float16[L, 64]`` little-endian, + ``L * 128`` bytes), with ``seq_sha1``, the SHA-1 of the normalised sequence (S15). + + Never raises for a payload: under DDP a rank that raised would leave the others waiting for the timeout. A protein + whose ESM3 rows are not ``L + 2`` (the tokenizer did not keep one token per residue), whose sequence is not ASCII, + or whose encoding failed gets an empty payload instead, logged as an ERROR (the first ``_LOGGED_FAILURES`` by name) + and counted at the end of the pass (``residue_failures``; the first ids in ``residue_failed_ids``). ``code_files``: + the payloads are residue codes that fm-embedding writes to files of their own (``--output-codes-folder``), where an + empty payload is an empty ``[0, 8]`` table, and which raises after the pass when there are any + (``ResidueCodesPass.check_after_pass``); otherwise they go to a build's or a query's shards, whose validation + refuses an empty one after the pass. + """ + + def _init_residue_payloads(self, residue_head=None, residue_output=None, code_files: bool = False): + residue_output = check_residue_output(residue_output) + if (residue_head is None) != (residue_output is None): + raise ValueError("residue_head and residue_output go together: pass both (a residue pass) or neither") + if code_files and residue_output != RESIDUE_CODES: + raise ValueError(f"code files hold {RESIDUE_CODES}, not {residue_output or 'no residue output'}") + self.residue_head = residue_head + self.residue_output = residue_output + self.code_files = bool(code_files) + self.residue_failures = 0 + self.residue_failed_ids = [] + + def on_predict_end(self): + if self.residue_failures: + if self.code_files: + logger.error(f"{self.residue_failures} protein(s) on this rank got no residue codes: their codes are " + f"written as empty [0, 8] tables") + else: + logger.error( + f"{self.residue_failures} protein(s) on this rank got an empty {self.residue_output} payload; " + f"the residue store validation will refuse them" + ) + + def _residue_payloads(self, prot_batch, prot_embeddings): + """``(payloads, seq_sha1s)`` of a batch, as tuples, from its proteins' ESM3 embeddings.""" + payloads = [] + digests = [] + for (esm_prot, name), embeddings in zip(prot_batch, prot_embeddings): + payload, digest = self._residue_payload(esm_prot.sequence, name, embeddings) + payloads.append(payload) + digests.append(digest) + return tuple(payloads), tuple(digests) + + def _residue_payload(self, sequence, name, embeddings): + """``(payload bytes, 20-byte seq_sha1)`` of one protein; an empty payload instead of any exception.""" + sequence = sequence or "" + if not sequence.isascii(): + # S15 refuses non-ASCII input before the pass; the row still gets a 20-byte seq_sha1 (of the UTF-8). + digest = hashlib.sha1(normalize_sequence(sequence, allow_non_ascii=True).encode("utf-8", "surrogatepass")) + return self._failed(name, "the sequence is not ASCII"), digest.digest() + digest = seq_sha1(sequence) + if embeddings.dim() != 2 or embeddings.shape[0] != len(sequence) + 2: + return self._failed(name, f"ESM3 gave {tuple(embeddings.shape)} rows for {len(sequence)} residues " + f"(expected {len(sequence) + 2} with BOS and EOS)"), digest + try: + rows = embeddings[1:-1] # S1: residues 1..L, BOS and EOS removed (a view; the embeddings are untouched) + if self.residue_output == RESIDUE_CODES: + payload = self.residue_head.encode_target(rows).cpu().numpy() + else: + payload = np.ascontiguousarray(self.residue_head.encode_query(rows).cpu().numpy(), + dtype=QUERY_ENCODING_DTYPE) + return payload.tobytes(), digest + except Exception as e: # noqa: BLE001 (never raise inside the distributed pass; validated after it) + return self._failed(name, f"{type(e).__name__}: {e}"), digest + + def _failed(self, name, reason): + self.residue_failures += 1 + if len(self.residue_failed_ids) < _LOGGED_FAILURES: + self.residue_failed_ids.append(name) + if self.residue_failures <= _LOGGED_FAILURES: + if self.code_files: + logger.error(f"{name}: no residue codes ({reason}); its codes are written as an empty [0, 8] table") + else: + logger.error(f"{name}: no {self.residue_output} payload ({reason}); the row is written empty") + elif self.residue_failures == _LOGGED_FAILURES + 1: + if self.code_files: + logger.error("more proteins without residue codes; only counting them from now on") + else: + logger.error(f"more proteins without a {self.residue_output} payload; only counting them from now on") + return b"" diff --git a/src/foldmatch/search/residue_shards.py b/src/foldmatch/search/residue_shards.py index 395458f..d5fe0fe 100644 --- a/src/foldmatch/search/residue_shards.py +++ b/src/foldmatch/search/residue_shards.py @@ -34,6 +34,12 @@ A run's shards are ``emb_-.parquet`` (:func:`run_shard_files`, :func:`shard_run_ids`; also imported from ``foldmatch.search.embedding_computer``). +``fm-embedding from-sequences residue|chain --output-codes-folder`` writes the same residue codes to Parquet files of +another kind, which are not shards (:func:`code_table_schema`): one ``uint8 [L, 8]`` table per sequence, columns +``id: string | residue_codes: list> | seq_sha1: fixed_size_binary(20)``, with the metadata +of a shard but ``foldmatch.residue_output`` = ``residue_code_table`` (:data:`RESIDUE_CODE_TABLE`), so every shard +reader refuses such a file (it holds neither ``residue_codes`` nor ``query_encoding``). + pyarrow and numpy only: no torch, so Stage-2 code (and ``foldmatch.search.split_run``) can import it. """ from __future__ import annotations @@ -57,6 +63,9 @@ RESIDUE_CODES = "residue_codes" QUERY_ENCODING = "query_encoding" RESIDUE_OUTPUTS = (RESIDUE_CODES, QUERY_ENCODING) +#: The kind (``foldmatch.residue_output``) of fm-embedding's residue-code Parquet files: one ``[L, 8]`` table per +#: sequence, not a payload column beside an embedding. Not a residue output of a shard: the shard readers refuse it. +RESIDUE_CODE_TABLE = "residue_code_table" ID = "id" EMBEDDING = "embedding" @@ -118,8 +127,12 @@ def payload_row_bytes(residue_output: str, system) -> int: def shard_metadata(residue_output: str, system) -> dict: + return _system_metadata(check_residue_output(residue_output), system) + + +def _system_metadata(kind: str, system) -> dict: return { - META_RESIDUE_OUTPUT: check_residue_output(residue_output).encode("ascii"), + META_RESIDUE_OUTPUT: kind.encode("ascii"), META_SYSTEM: system.name.encode("ascii"), META_ENCODING_DIGEST: system.encoding_digest().encode("ascii"), META_SYSTEM_DIGEST: system.digest().encode("ascii"), @@ -185,6 +198,24 @@ def shard_schema(residue_output: Optional[str] = None, system=None, record: Opti metadata=metadata) +def code_table_type(system) -> pa.DataType: + """The ``residue_codes`` column of a residue-code file: one list of ``code_bytes`` (8) values per residue.""" + return pa.list_(pa.list_(pa.uint8(), system.code_bytes)) + + +def code_table_schema(system, record: Optional[Dict[str, Any]] = None) -> pa.Schema: + """The schema of fm-embedding's residue-code Parquet files (``--output-codes-folder``): ``id | residue_codes | + seq_sha1``, ``residue_codes`` a ``uint8 [L, 8]`` table per sequence (:func:`code_table_type`), with the metadata of + a shard of ``system`` (its name and digests) and the pass ``record`` (:func:`pass_record`), but the kind + :data:`RESIDUE_CODE_TABLE`, which no shard reader takes.""" + if system is None: + raise ValueError("a residue-code table schema needs the scoring system") + metadata = _system_metadata(RESIDUE_CODE_TABLE, system) + metadata.update(pass_metadata(pass_record() if record is None else record)) + return pa.schema([(ID, pa.string()), (RESIDUE_CODES, code_table_type(system)), (SEQ_SHA1, SEQ_SHA1_TYPE)], + metadata=metadata) + + def _concat_payloads(payloads: Sequence[bytes], total: int): """One contiguous buffer holding ``payloads`` back to back (``total`` bytes).""" return b"".join(payloads) @@ -258,6 +289,9 @@ def key(self) -> tuple: def describe(self) -> str: if self.residue_output is None: return "no residue column (a plain pass)" + if self.residue_output == RESIDUE_CODE_TABLE: + return (f"the residue-code tables of 'fm-embedding ... --output-codes-folder' ({self.system}), not an " + f"embedding shard") return (f"{self.residue_output} of {self.system} (encoding_digest {self.encoding_digest}, system_digest " f"{self.system_digest})") @@ -341,6 +375,9 @@ def check_shards(paths: Iterable, residue_output: str, system) -> List[ShardInfo "the shards of other runs or use another folder.") info = infos[0] where = f"{len(infos)} shard(s) ({info.path.name}, ...)" + if info.residue_output == RESIDUE_CODE_TABLE: + raise ResidueShardError(f"{where} hold {info.describe()}: a build's {residue_output} are read from the " + f"embedding shards of its own pass (emb_-.parquet), not from these files") if info.residue_output != residue_output: raise ResidueShardError(f"{where} hold {info.describe()}, not {residue_output}: they were written by a pass " f"without --residue-codes or for another purpose") diff --git a/src/foldmatch/utils/data.py b/src/foldmatch/utils/data.py index 18b65a0..b462862 100644 --- a/src/foldmatch/utils/data.py +++ b/src/foldmatch/utils/data.py @@ -279,10 +279,11 @@ def _file_vector(f: Path) -> np.ndarray: The file holds one vector: a dense tensor, or a CSV of numbers, with at most one axis longer than 1 ([D], [1, D], [D, 1]; a .csv of one column, as CsvBatchWriter writes a chain or assembly embedding, or of one line). Anything else is refused with a :class:`RefusalError` that names the file: a file that cannot be read, no values, NaN or - infinite values (such an embedding is never found, and as a query it finds nothing) and a table, such as residue - embeddings ([L, D], one row per residue), whose mean is not a chain embedding. Non-finite values are reported - before the shape in a CSV, where a line cut short reads as NaN (as does the empty field above the index column of - pandas' to_csv layout), and after it in a .pt. + infinite values (such an embedding is never found, and as a query it finds nothing), a table, such as residue + embeddings ([L, D], one row per residue), whose mean is not a chain embedding, and fm-embedding's residue codes in + a .pt (``--output-codes-folder``: a ``uint8 [L, 8]`` tensor, refused by its type whatever L). Non-finite values are + reported before the shape in a CSV, where a line cut short reads as NaN (as does the empty field above the index + column of pandas' to_csv layout), and after it in a .pt. """ ext = f.suffix[1:] if ext == 'pt': @@ -301,6 +302,14 @@ def _file_vector(f: Path) -> np.ndarray: f"{f} holds a {type(emb).__name__}, not a tensor. A .pt embedding holds one vector: a tensor of D " f"values saved with torch.save." ) + if emb.dtype == torch.uint8 and emb.dim() == 2 and emb.shape[1] == 8: + # fm-embedding's /.pt (--output-codes-folder), which a one-residue sequence would make a + # vector + raise RefusalError( + f"{f} holds residue codes (a uint8 tensor of {emb.shape[0]} x 8), as 'fm-embedding from-sequences ... " + f"--output-codes-folder' writes them, not an embedding vector: residue codes belong in a folder of " + f"their own (--output-codes-folder); move them out of the embedding folder." + ) try: values = emb.detach().float().numpy() except Exception as exc: # a sparse, quantized, nested or meta tensor @@ -353,6 +362,11 @@ def _table_message(f: Path, values: np.ndarray, axes: List[int]) -> str: f"NAME.CHAIN.{ext} with 'inference chain-embedding --src-file LIST --res-embedding-location DIR " f"--res-embedding-format {ext} --output-folder OUT', LIST holding one line NAME,,CHAIN per file." ) + if (values.ndim == 2 and values.shape[1] == 8 and np.all(values == np.round(values)) + and values.min() >= 0 and values.max() <= 255): + text += (f" A table of 8 columns of integers 0-255 may be residue codes, as 'fm-embedding from-sequences ... " + f"--output-codes-folder' writes them (.{ext}): they are not embeddings either; residue codes " + f"belong in a folder of their own (--output-codes-folder): move them out of the embedding folder.") if ext == 'csv': # pandas' layouts: a header line 0,1,...,C-1 (unless header=False), an index column rows, columns = values.shape if np.array_equal(values[0], np.arange(columns)): @@ -404,6 +418,12 @@ def _parquet_file(f: Path) -> pq.ParquetFile: f"leaves an unreadable file." ) from exc missing = [f"'{name}'" for name in ('id', 'embedding') if name not in pf.schema_arrow.names] + if missing and _is_code_table(pf.schema_arrow): + raise RefusalError( + f"{f} holds residue codes, as 'fm-embedding from-sequences ... --output-codes-folder' writes them (one " + f"table of 8-byte codes per sequence), not embeddings: residue codes belong in a folder of their own " + f"(--output-codes-folder); move them out of the embedding folder." + ) if missing: raise RefusalError( f"{f} has no {' or '.join(missing)} column: a Parquet embedding file holds the columns id and embedding." @@ -411,6 +431,13 @@ def _parquet_file(f: Path) -> pq.ParquetFile: return pf +def _is_code_table(schema: pa.Schema) -> bool: + """Whether a Parquet schema is that of fm-embedding's residue-code files (--output-codes-folder), which are not + embeddings.""" + from foldmatch.search.residue_shards import META_RESIDUE_OUTPUT, RESIDUE_CODE_TABLE + return (schema.metadata or {}).get(META_RESIDUE_OUTPUT) == RESIDUE_CODE_TABLE.encode("ascii") + + def _read_batches(f: Path, batches: Iterator[pa.RecordBatch]) -> Iterator[pa.RecordBatch]: """The ``batches`` of the Parquet file ``f``: a read error (a corrupted page) is a :class:`RefusalError` that names the file. An error in the caller's loop body is raised in the caller, not in this generator, so it is not caught.""" @@ -488,6 +515,8 @@ def stream_embeddings( file_extension: Optional[str] = None, batch_size: int = 32768, expected_dim: Optional[int] = None, + *, + warn_folders: bool = True, ) -> Iterator[tuple[list[str], np.ndarray]]: """Yield (ids, [B, D] float32) batches from a file or directory of .pt / .csv / .parquet. @@ -503,9 +532,11 @@ def stream_embeddings( infinite values (:func:`_file_vector`), and a .pt and a .csv of one stem (one id twice). Every embedding has one dimension: ``expected_dim`` (a query's database's) when given, else the first one's (Parquet files are read first, then .pt, then .csv). A single file is - handled as a degenerate directory-of-one; ``file_extension`` is ignored in that case. + handled as a degenerate directory-of-one; ``file_extension`` is ignored in that case. A + folder's subfolders are not read, not even one named like an embedding file, which is + skipped with a WARNING unless ``warn_folders`` is False (:func:`embedding_files`). """ - parquet_files, point_files = embedding_files(path, file_extension) + parquet_files, point_files = embedding_files(path, file_extension, warn_folders=warn_folders) by_id = {} for f in point_files: @@ -538,9 +569,16 @@ def stream_embeddings( yield ids_buf, np.ascontiguousarray(np.stack(emb_buf)) -def embedding_files(path: Path, file_extension: Optional[str] = None) -> Tuple[List[Path], List[Path]]: +def embedding_files(path: Path, file_extension: Optional[str] = None, *, + warn_folders: bool = True) -> Tuple[List[Path], List[Path]]: """The (Parquet files, .pt and .csv files) that stream_embeddings reads from a file or a folder, each list sorted - by extension then name; a ValueError for a missing path, an unsupported extension or a folder without them.""" + by extension then name; a ValueError for a missing path, an unsupported extension or a folder without them. + + A folder is read one level deep: its files named ``*.parquet``, ``*.pt`` or ``*.csv`` (a symbolic link to a file + is a file). An entry so named that is a folder (``x.parquet``, the dataset folder that Spark writes, say, or a + symbolic link to a folder) is not an embedding file: it is skipped, with a WARNING that names the skipped folders + (:func:`skipped_folders_message`; not with ``warn_folders=False``, for a listing whose folder another one reports), + and a folder that holds no embedding file but such folders is refused as a folder without embedding files.""" p = Path(path) if not p.exists(): raise ValueError(f"Embeddings path does not exist: {path}") @@ -560,12 +598,17 @@ def embedding_files(path: Path, file_extension: Optional[str] = None) -> Tuple[L files = [p] else: extensions = (file_extension,) if file_extension is not None else _SUPPORTED_EXTS - files = [] + files, folders = [], [] for ext in extensions: - files.extend(sorted(p.glob(f"*{ext}"))) + for entry in sorted(p.glob(f"*{ext}")): + (folders if entry.is_dir() else files).append(entry) + if folders and warn_folders: + logging.getLogger(__name__).warning(skipped_folders_message(path, folders)) if not files: + skipped = (f" ({len(folders)} folder(s) named like embedding files skipped: " + f"{_shown_names(folders)})" if folders else "") raise ValueError( - f"No embedding files found with extensions {list(extensions)} in {path}" + f"No embedding files found with extensions {list(extensions)} in {path}{skipped}" ) parquet_files = [f for f in files if f.suffix in _BATCH_EXTS] @@ -573,15 +616,35 @@ def embedding_files(path: Path, file_extension: Optional[str] = None) -> Tuple[L return parquet_files, point_files +#: The skipped folders that a message names, the rest counted. +_SHOWN_FOLDERS = 5 + + +def _shown_names(paths: List[Path]) -> str: + shown = ", ".join(path.name for path in paths[:_SHOWN_FOLDERS]) + return shown + (f" and {len(paths) - _SHOWN_FOLDERS} more" if len(paths) > _SHOWN_FOLDERS else "") + + +def skipped_folders_message(path, folders: List[Path]) -> str: + """The WARNING of :func:`embedding_files` for the folders of ``path`` named like embedding files, which it skips.""" + return (f"{path}: {len(folders)} folder(s) named like embedding files skipped, not read: {_shown_names(folders)}. " + f"The embedding files of a folder are read, not those of its subfolders; to read the files of such a " + f"folder (the parts of a Parquet dataset that Spark writes, say), give that folder itself.") + + def refuse_repeated_query_ids(path: Path, file_extension: Optional[str] = None): """Refuse (RefusalError) query embeddings in which two vectors share an id (P9.N D7), read before any search as stream_embeddings would read them: the ids of the Parquet files (their id column only) and the stems of the .pt and .csv files. A search's results are kept by query id, so a repeated id kept one vector's hits and dropped the - others' without notice. A Parquet file whose ids cannot be read is left to stream_embeddings, which refuses it.""" - parquet_files, point_files = embedding_files(path, file_extension) + others' without notice. A Parquet file whose ids cannot be read, or without embeddings (fm-embedding's residue + codes), is left to stream_embeddings, which refuses it. The folders it skips (:func:`embedding_files`) are not + reported here: stream_embeddings, or the caller's own listing, reports them.""" + parquet_files, point_files = embedding_files(path, file_extension, warn_folders=False) where = {} for f in parquet_files: try: + if 'embedding' not in pq.read_schema(f).names: + continue # not embeddings (fm-embedding's residue codes, say): stream_embeddings refuses it ids = pq.read_table(f, columns=["id"]).column("id").to_pylist() except Exception: continue diff --git a/src/foldmatch/utils/file_kinds.py b/src/foldmatch/utils/file_kinds.py new file mode 100644 index 0000000..d3a4e4f --- /dev/null +++ b/src/foldmatch/utils/file_kinds.py @@ -0,0 +1,202 @@ +"""What an existing output file of fm-embedding holds: embeddings, or the residue codes of ``fm-embedding +from-sequences residue|chain --output-codes-folder`` (:func:`file_kind`). + +A run refuses to replace a file of the other kind (``foldmatch.utils.output_files.refuse_other_kind``): embeddings in +its codes folder, residue codes in its output folder. Identification is positive only: a file is of a kind when its +first bytes say so, as the writers write it, and anything else (an empty file, a file cut short, one that another rank +of the run is writing, a corrupt or foreign file) is of no kind, None, and is replaced as before. A file that the run +itself writes is always of the right kind, so a run never refuses one of its own files, whatever another rank has +written of it when it is looked at. + +Each file is read as little as can tell it: a Parquet file's footer (its schema and metadata); a ``.pt`` file's +tensor, loaded memory-mapped (its type and shape; the values are not read); a ``.csv`` file's first line (its first 4 +KiB); a ``.json.gz`` file's first record's keys (its first 16 KiB, decompressed as far as they go). The suffix is +matched in any case, as a case-insensitive folder takes ``X.PT`` for the file ``x.pt`` that a run would write. A +command that checks a folder before the library checks it again remembers what it read (:func:`remembering`), and the +library's check takes each file's kind from there once (while the file is unchanged: its device, inode, size and +modification time) and forgets it: the memory is held from one check to the next only, never for the whole run. + +pyarrow, torch, gzip and json are imported only when a file of their format is read. +""" +import contextlib +import json +import os +import re +import zlib +from typing import Optional + +#: A file of FoldMatch embeddings: chain, residue, structure or assembly embeddings, in any format (Parquet shards of +#: a build included, whose ``embedding`` column holds chain embeddings). +EMBEDDINGS = "embeddings" +#: A file of fm-embedding's residue codes (``--output-codes-folder``): ``uint8 [L, 8]`` tables. +RESIDUE_CODES = "residue codes" + +#: The bytes of one residue's code in a residue-code file. +CODE_BYTES = 8 +#: How much of a .csv file is read: a code line takes at most 32 bytes, and an embedding line's first values tell it. +CSV_HEAD = 4096 +#: How much of a .json.gz file is read, compressed: the first record's id and the key after it. +JSON_HEAD = 16384 + +_INTEGER = re.compile(r"[0-9]+") +_FLOAT = re.compile(r"[+-]?(?:[0-9]+(?:\.[0-9]*)?|\.[0-9]+)(?:[eE][+-]?[0-9]+)?|[+-]?(?:inf|nan)") +_SIGNED_INTEGER = re.compile(r"[+-]?[0-9]+") + +#: (device, inode, size, mtime_ns) -> kind, of the files read within remembering() and not yet read again. +_KNOWN = {} +_remembering = 0 + + +@contextlib.contextmanager +def remembering(): + """Remember the kind of each file read within it (:func:`file_kind`), for the library's check of the same files + after it: there each is taken from memory once, and forgotten. A command checks its folders within it before it + calls the library, which checks them again (fm-embedding from-sequences, csv and pt; the legacy chain-embedding and + complete-embedding); a check that refuses ends the run, and what was remembered is forgotten.""" + global _remembering + _remembering += 1 + try: + yield + except BaseException: + _KNOWN.clear() + raise + finally: + _remembering -= 1 + + +def file_kind(path) -> Optional[str]: + """:data:`EMBEDDINGS`, :data:`RESIDUE_CODES`, or None (not positively either, or not readable), by the file's + format, its name's suffix (``.parquet``, ``.pt``, ``.csv``, ``.json.gz``): :func:`parquet_kind`, :func:`pt_kind`, + :func:`csv_kind`, :func:`json_kind`; the suffix in any case (``X.PT``). Within :func:`remembering`, the kind is + remembered; outside it, a remembered kind is taken once and forgotten.""" + name = os.fspath(path) + lowered = name.lower() + read = _KINDS.get(next((suffix for suffix in _KINDS if lowered.endswith(suffix)), None)) + if read is None: + return None + try: + stat = os.stat(name) + except (OSError, ValueError): + return None + key = (stat.st_dev, stat.st_ino, stat.st_size, stat.st_mtime_ns) + if key in _KNOWN: + return _KNOWN[key] if _remembering else _KNOWN.pop(key) + try: + kind = read(name) if stat.st_size else None + except Exception: # noqa: BLE001 - a file that cannot be read is of no kind + kind = None + if _remembering: + _KNOWN[key] = kind + return kind + + +def parquet_kind(path) -> Optional[str]: + """A Parquet file by its footer: residue codes when its metadata ``foldmatch.residue_output`` is + ``residue_code_table`` (``search.residue_shards.code_table_schema``); embeddings when it has the columns ``id`` + and ``embedding``, ``embedding`` a list of floats (chain, structure, assembly embeddings, and a build's shards) or + a list of lists of floats (residue embeddings). A file being written has no footer yet.""" + import pyarrow as pa + import pyarrow.parquet as pq + from foldmatch.search.residue_shards import EMBEDDING, ID, META_RESIDUE_OUTPUT, RESIDUE_CODE_TABLE + schema = pq.read_schema(path) + if (schema.metadata or {}).get(META_RESIDUE_OUTPUT) == RESIDUE_CODE_TABLE.encode("ascii"): + return RESIDUE_CODES + if ID not in schema.names or EMBEDDING not in schema.names: + return None + lists = (pa.types.is_list, pa.types.is_large_list, pa.types.is_fixed_size_list) + kind = schema.field(EMBEDDING).type + for _ in range(2): + if not any(is_list(kind) for is_list in lists): + return None + kind = kind.value_type + if pa.types.is_floating(kind): + return EMBEDDINGS + return None + + +def pt_kind(path) -> Optional[str]: + """A ``.pt`` file by its tensor, loaded memory-mapped on the CPU (``weights_only``; its values are not read): + residue codes when it is ``uint8 [L, 8]``, embeddings when it is a floating-point tensor. A file being written is + not yet a whole zip archive, which cannot be loaded.""" + import torch + tensor = torch.load(path, map_location="cpu", weights_only=True, mmap=True) + if not isinstance(tensor, torch.Tensor): + return None + if tensor.dtype == torch.uint8 and tensor.dim() == 2 and tensor.shape[1] == CODE_BYTES: + return RESIDUE_CODES + return EMBEDDINGS if tensor.is_floating_point() else None + + +def csv_kind(path) -> Optional[str]: + """A ``.csv`` file by its first line (in its first :data:`CSV_HEAD` bytes; no header, as the writers write it): + residue codes when it is a whole line of 8 integers 0-255; embeddings when its values are numbers, at least one + of them written as a float ('0.5', '1e-05'), as every float value is (a chain embedding is one value per line, a + residue embedding one residue per line). A line not ended in the bytes read (cut short, or long) is judged on its + values but the last, which may be cut; 8 integers are never codes then. An empty file (an empty code table, say) + is of no kind.""" + with open(path, "rb") as f: + head = f.read(CSV_HEAD) + line, ended, _ = head.partition(b"\n") + try: + text = line.decode("ascii") + except UnicodeDecodeError: + return None + fields = (text[:-1] if ended and text.endswith("\r") else text).split(",") + if ended and len(fields) == CODE_BYTES and all(_INTEGER.fullmatch(f) and int(f) <= 255 for f in fields): + return RESIDUE_CODES + if not ended: + fields = fields[:-1] + if fields and all(_FLOAT.fullmatch(f) for f in fields) and \ + not all(_SIGNED_INTEGER.fullmatch(f) for f in fields): + return EMBEDDINGS + return None + + +def json_kind(path) -> Optional[str]: + """A ``.json.gz`` file by its first record (orient records, as the writers write it: ``[{"id": ..., "": + [...]}, ...]``), decompressed from its first :data:`JSON_HEAD` bytes as far as they go: residue codes when the key + after ``id`` is ``residue_codes``, embeddings when it is ``embedding``, each a list. A file cut short is judged on + what it holds.""" + with open(path, "rb") as f: + head = f.read(JSON_HEAD) + text = zlib.decompressobj(16 + zlib.MAX_WBITS).decompress(head, 4 * JSON_HEAD).decode("utf-8", "replace") + decoder = json.JSONDecoder() + at = _skip(text, 0) + for expected in "[{": + if not text.startswith(expected, at): + return None + at = _skip(text, at + 1) + key, at = decoder.raw_decode(text, at) + if key != "id": + return None + at = _skip(text, _expect(text, _skip(text, at), ":")) + _, at = decoder.raw_decode(text, at) + at = _skip(text, _expect(text, _skip(text, at), ",")) + key, at = decoder.raw_decode(text, at) + at = _skip(text, _expect(text, _skip(text, at), ":")) + if not text.startswith("[", at): + return None + return {"embedding": EMBEDDINGS, "residue_codes": RESIDUE_CODES}.get(key) if isinstance(key, str) else None + + +def _skip(text: str, at: int) -> int: + while at < len(text) and text[at] in " \t\r\n": + at += 1 + return at + + +def _expect(text: str, at: int, character: str) -> int: + if not text.startswith(character, at): + raise ValueError(f"{character!r} expected at {at}") + return at + 1 + + +_KINDS = {".parquet": parquet_kind, ".pt": pt_kind, ".csv": csv_kind, ".json.gz": json_kind} + + +def describe(path, kind: str) -> str: + """A file of a kind, as a refusal names it: 'an embedding' (a .csv or .pt file), 'embeddings' (a Parquet or JSON + file of many), 'residue codes'.""" + if kind == EMBEDDINGS and os.fspath(path).lower().endswith((".csv", ".pt")): + return "an embedding" + return kind diff --git a/src/foldmatch/utils/output_files.py b/src/foldmatch/utils/output_files.py index 564ce7c..2e2ee46 100644 --- a/src/foldmatch/utils/output_files.py +++ b/src/foldmatch/utils/output_files.py @@ -12,6 +12,10 @@ which :func:`probe_folding` finds out per folder. Folders are compared as files (device and inode), never as strings: a symbolic link and, on macOS, ``SHARED`` and ``shared`` are one folder; and an input and an output's file that exists are one file when either is a link to the other from another folder. + +Nor does a run replace a file of the other kind (:func:`refuse_other_kind`): residue codes in a folder of embeddings, +or embeddings in the folder of fm-embedding's residue codes (``--output-codes-folder``), told by their first bytes +(:mod:`foldmatch.utils.file_kinds`). """ import glob import os @@ -22,6 +26,7 @@ from typing import Iterable, List, NamedTuple, Optional, Sequence, Tuple from foldmatch.errors import RefusalError +from foldmatch.utils.file_kinds import EMBEDDINGS, RESIDUE_CODES, describe, file_kind #: The longest file name, in bytes of UTF-8, that the usual file systems take (ext4, XFS, APFS, Lustre, GPFS). MAX_FILE_NAME_BYTES = 255 @@ -89,10 +94,11 @@ def _usable_rule(suffix: str) -> str: def refuse_unusable_names(names: Sequence, suffix: str, *, source: str, labels: Optional[Sequence] = None, - what: str = "embedding name", given: str = "those rows", pattern: Optional[str] = None): + what: str = "embedding name", given: str = "those rows", pattern: Optional[str] = None, + folder: str = "output folder"): """Refuse (RefusalError) names that cannot name the file ```` in the output folder (name_problem), naming the first ones (with their label, "row 3", say, when given); ``pattern`` is the file as the message names - it, ```` by default.""" + it, ```` by default, and ``folder`` the folder ("codes folder", say).""" bad = [] for k, name in enumerate(names): problem = name_problem(name, suffix) @@ -103,13 +109,13 @@ def describe(item): name, problem, label = item return f"{name!r}{'' if label is None else f' ({label})'} {problem}" raise RefusalError( - f"{source}: {len(bad)} {what}(s) that cannot name a file {pattern or '' + suffix} in the output " - f"folder: {shown(bad, describe)}. Each {what} names its own file there: give {given} " + f"{source}: {len(bad)} {what}(s) that cannot name a file {pattern or '' + suffix} in the {folder}: " + f"{shown(bad, describe)}. Each {what} names its own file there: give {given} " f"{'an' if what[0] in 'aeiou' else 'a'} {what} {_usable_rule(suffix)}." ) -def refuse_unusable_stem(stem, what: str = "The output name"): +def refuse_unusable_stem(stem, what: str = "The output name", folder: str = "output folder"): """Refuse (RefusalError) a name that cannot name the files ``-.parquet`` and ``-.json.gz`` of the output folder (a 6-digit rank reserved), whatever the format: name_problem's rules.""" suffix = "-" + "0" * RANK_DIGITS + max(STEM_SUFFIXES.values(), key=len) @@ -122,7 +128,7 @@ def refuse_unusable_stem(stem, what: str = "The output name"): if problem is not None: raise RefusalError( f"{what} {stem!r} {problem}: it names the files -.parquet or -.json.gz of the " - f"output folder. Give a name {_usable_rule(suffix)}." + f"{folder}. Give a name {_usable_rule(suffix)}." ) @@ -431,7 +437,8 @@ def _folded_by(names: Sequence[str], folding: Folding) -> List[str]: def repeated_files_message(source: str, repeats: List[tuple], folding: Folding, what: str = "embedding name", - given: str = "those rows", folded_advice: Optional[str] = None) -> str: + given: str = "those rows", folded_advice: Optional[str] = None, + folder: str = "output folder") -> str: """The refusal of names that give one file twice (OutputFiles.add_files' and add_prefixes' repeats). Names that differ, and that the folder only takes for one, are told so, with what can keep them apart: ``folded_advice``, whose ``{apart}`` is what the folder does not tell apart, or by default another folder, Parquet or JSON, or other @@ -447,8 +454,8 @@ def describe(repeat): folded += [part for part in _folded_by(names, folding) if part not in folded] folded.sort(key=lambda part: part.startswith("Unicode")) apart = " or ".join(folded) - note = f" (the output folder does not tell {apart} apart)" if folded else "" - found = (f"{source}: {len(repeats)} file(s) of the output folder would each be written for more than one " + note = f" (the {folder} does not tell {apart} apart)" if folded else "" + found = (f"{source}: {len(repeats)} file(s) of the {folder} would each be written for more than one " f"{what}{note}: {shown(repeats, describe)}. Each one would replace the one before") if not folded: return f"{found}: give each its own {what}." @@ -469,7 +476,8 @@ def _linked_id(path) -> Optional[Tuple[int, int]]: return (link.st_dev, link.st_ino) if link.st_nlink > 1 else None -def refuse_outputs_over_inputs(outputs: OutputFiles, inputs: Iterable[Tuple[str, object]], *, what: str = "run"): +def refuse_outputs_over_inputs(outputs: OutputFiles, inputs: Iterable[Tuple[str, object]], *, what: str = "run", + folder: str = "output folder"): """Refuse (RefusalError) an input of the run (described, path) that is in the output folder under the name of one of its outputs, or, from another folder, one file with a file of the output folder that one of its outputs would write, through a symbolic or hard link either way (compared as files, with the files that exist: an input @@ -495,7 +503,7 @@ def refuse_outputs_over_inputs(outputs: OutputFiles, inputs: Iterable[Tuple[str, found.append((described, f"{os.fspath(path)} (which is {file[0]})", file[1])) if found: raise RefusalError( - f"{len(found)} input(s) of this {what} would be replaced by its output, in the output folder " + f"{len(found)} input(s) of this {what} would be replaced by its output, in the {folder} " f"{outputs.folder}: {shown(found, lambda f: f'{f[0]} {f[1]} by {f[2]}')}. An output never replaces an " f"input of its run: write the output to another folder or under other names, or move the input." ) @@ -506,39 +514,159 @@ def check_outputs(out_folder, out_format, out_name, *, names: Optional[Sequence] separator: str = ".", inputs: Iterable[Tuple[str, object]] = (), folding: Optional[Folding] = None, stem_what: str = "The output name", what: str = "embedding name", given: str = "those rows", - folded_advice: Optional[str] = None) -> Optional[OutputFiles]: + folded_advice: Optional[str] = None, folder: str = "output folder", + writes: str = EMBEDDINGS, replaced_names: Optional[Sequence] = None) -> Optional[OutputFiles]: """The checks of one output of a run, before any model is built (RefusalError): the output name ``out_name`` names files of the folder, whatever the format (D1); for csv and pt, each of ``names`` names one file ``.`` (D1) of its own, as the folder compares names (D3), and each of ``prefixes`` names the files ``.`` of items known only once their structure is read, prefixes that the folder - takes for one refused too; and no input (described, path) is one of the output files (D2). ``stem_what`` names the - output name in a refusal, ``what`` a name and ``given`` what holds them, ``folded_advice`` what keeps names apart - that the folder takes for one (repeated_files_message); ``folding`` is the folder's, when known. The output's - files, or None without ``out_folder``.""" + takes for one refused too; no input (described, path) is one of the output files (D2); and no file that the + output would replace holds the other kind of data than it ``writes`` (:data:`EMBEDDINGS`, or the + :data:`RESIDUE_CODES` of fm-embedding's codes folder; :func:`refuse_other_kind`). ``replaced_names``, with csv and + pt, are names whose files ``.`` the output writes, whose checks are the caller's (fm-embedding's, for + the ids of a FASTA file): they only tell the files that the output would replace. ``stem_what`` names the output + name in a refusal, ``what`` a name and ``given`` what holds them, ``folded_advice`` what keeps names apart that + the folder takes for one (repeated_files_message); ``folding`` is the folder's, when known, and ``folder`` what the + refusals call it ("codes folder", say). The output's files, or None without ``out_folder``.""" if out_folder is None: return None fmt = format_value(out_format) outputs = OutputFiles(out_folder, folding) if out_name is not None: - refuse_unusable_stem(out_name, stem_what) + refuse_unusable_stem(out_name, stem_what, folder) if fmt in STEM_SUFFIXES: outputs.add_stem(out_name, STEM_SUFFIXES[fmt]) if fmt in ITEM_FORMATS: suffix = f".{fmt}" repeats = [] if names is not None: - refuse_unusable_names(names, suffix, source=source, labels=labels, what=what, given=given) + refuse_unusable_names(names, suffix, source=source, labels=labels, what=what, given=given, + folder=folder) repeats = outputs.add_files(names, suffix, labels) if prefixes is not None and not repeats: refuse_unusable_names(prefixes, separator + suffix, source=source, labels=labels, - pattern=f"{separator}{suffix}") + pattern=f"{separator}{suffix}", folder=folder) repeats = outputs.add_prefixes(prefixes, separator, suffix, labels) if repeats: - raise RefusalError(repeated_files_message(source, repeats, outputs.folding, what, given, folded_advice)) - refuse_outputs_over_inputs(outputs, inputs) + raise RefusalError(repeated_files_message(source, repeats, outputs.folding, what, given, folded_advice, + folder)) + if replaced_names is not None: + outputs.add_files(replaced_names, suffix) + refuse_outputs_over_inputs(outputs, inputs, folder=folder) + refuse_other_kind(outputs, writes, folder) return outputs +def refuse_other_kind(outputs: OutputFiles, writes: str = EMBEDDINGS, folder: str = "output folder"): + """Refuse (RefusalError) a run that would replace, in the folder of ``outputs``, a file of the other kind than it + ``writes``: residue codes where it writes embeddings (any fm-embedding command, any format), embeddings where it + writes fm-embedding's residue codes (``--output-codes-folder``). Only the files that the output would replace are + read (:meth:`OutputFiles.existing`), every one of them, each as little as tells its kind + (:func:`~foldmatch.utils.file_kinds.file_kind`). Positively only: a file of no kind (empty, cut short, being + written by another rank of the run, corrupt, foreign) is replaced as before; a file that the run itself writes is + of the kind it writes, so a rerun into the same folders replaces its own files. A folder that also holds other + files of the other kind is not refused: nothing of them is lost.""" + other = RESIDUE_CODES if writes == EMBEDDINGS else EMBEDDINGS + found = sorted(path for path, _ in outputs.existing().values() if file_kind(path) == other) + if not found: + return + listed = shown(found, lambda path: f"{path} ({describe(path, other)})") + if writes == EMBEDDINGS: + raise RefusalError( + f"{len(found)} file(s) of the {folder} {outputs.folder} that this run would replace hold residue codes, as " + f"'fm-embedding from-sequences ... --output-codes-folder' writes them, not embeddings: {listed}. The " + f"embeddings and the residue codes go to folders of their own: give an output folder without those " + f"residue-code files, or move them out of it." + ) + raise RefusalError( + f"{len(found)} file(s) of the {folder} {outputs.folder} that this run would replace hold embeddings, not " + f"residue codes: {listed}. The residue codes go to a folder of their own: give a codes folder for the codes " + f"only, without those embedding files, or move them out of it." + ) + + +# ===================================================================================================================== +# Residue codes in their own folder (fm-embedding from-sequences residue|chain --output-codes-folder) +# ===================================================================================================================== +def default_codes_name(out_name: str) -> str: + """The name of the residue-code files when none is given: ``-codes`` (the files + ``-codes-.parquet`` or ``.json.gz``; it names no file in csv and pt, as ``out_name`` does not).""" + return f"{out_name}-codes" + + +def same_folder_problem(out_folder, codes_folder, + options: Tuple[str, str] = ("out_folder", "out_codes_folder")) -> Optional[str]: + """Why ``codes_folder`` cannot hold the residue codes of a run whose embeddings go to ``out_folder``, or None: it + is ``out_folder`` itself, compared as files (device and inode: a symbolic link, '.' and its absolute path, or a + case variant on a case-insensitive volume, are one folder); ``options`` name the two folders in the message + (fm-embedding's ``--output-folder`` and ``--output-codes-folder``). One folder inside the other is allowed: + fm-search reads the files of a folder, not its subfolders (``utils.data.embedding_files`` skips them).""" + if out_folder is None or codes_folder is None or not same_folder(out_folder, codes_folder): + return None + first, second = options + return (f"{first} and {second} are one folder ({os.fspath(codes_folder)}): the residue codes go to a folder of " + f"their own, so that the embedding folder holds embeddings only ('fm-search build embeddings' and " + f"'fm-search query embedding' read every .parquet, .pt and .csv file of it). Give {second} another folder " + f"(a subfolder of {first} will do).") + + +def refuse_same_folder(out_folder, codes_folder, options: Tuple[str, str] = ("out_folder", "out_codes_folder")): + """Refuse a codes folder that is the output folder (:func:`same_folder_problem`) with a ValueError, as the library + refuses its other argument values; fm-embedding checks it first, as a usage error (typer.BadParameter, exit + status 2).""" + problem = same_folder_problem(out_folder, codes_folder, options) + if problem is not None: + raise ValueError(problem) + + +def check_code_outputs(out_codes_folder, out_format, out_codes_name: str, *, out_folder=None, + names: Optional[Sequence] = None, labels: Optional[Sequence] = None, source: str = "The input", + inputs: Iterable[Tuple[str, object]] = (), what: str = "embedding name", + given: str = "those rows", outputs: Optional[OutputFiles] = None, + options: Tuple[str, str] = ("out_folder", "out_codes_folder"), + replaced_names: Optional[Sequence] = None) -> Optional[OutputFiles]: + """The checks of the residue-code files of an fm-embedding pass (``--output-codes-folder``), before any model is + built: ``out_codes_folder`` is not ``out_folder`` (:func:`refuse_same_folder`, ``options`` naming them; a + ValueError) and is an existing folder (a RefusalError; the CLI's typer checks it; a library caller would otherwise + learn it only when the first code file is written, after ESM3); then those of :func:`check_outputs` in the codes + folder, as for the embeddings in theirs: ``out_codes_name`` names files of the folder, whatever the format + (``-.parquet`` or ``.json.gz``); with csv and pt, each of ``names`` (the ids) names one file + ``.`` of its own, as the codes folder compares names (``replaced_names`` only tell the files it + would replace, as there); no input (described, path) is a code file; and no file that the codes would replace + holds embeddings (:func:`refuse_other_kind`) (RefusalError). Last, one folder inside the other is not at the path + of a file that the other's output writes + (:func:`refuse_folder_at_output`, RefusalError): the codes folder is no file of ``outputs`` (the embedding files, + :func:`check_outputs` of ``out_folder``), and ``out_folder`` no code file. The code files, or None without + ``out_codes_folder``.""" + if out_codes_folder is None: + return None + refuse_same_folder(out_folder, out_codes_folder, options) + if not os.path.isdir(out_codes_folder): + raise RefusalError(f"{options[1]} {os.fspath(out_codes_folder)} is not a folder: give an existing folder.") + code_outputs = check_outputs(out_codes_folder, out_format, out_codes_name, names=names, labels=labels, + source=source, inputs=inputs, stem_what="The codes name", what=what, given=given, + folder="codes folder", writes=RESIDUE_CODES, replaced_names=replaced_names) + refuse_folder_at_output(out_codes_folder, outputs, "The codes folder", "the embedding files of the output folder") + refuse_folder_at_output(out_folder, code_outputs, "The output folder", "the code files of the codes folder") + return code_outputs + + +def refuse_folder_at_output(folder, outputs: Optional[OutputFiles], what: str, files: str): + """Refuse (RefusalError) a folder (``what``) that is in the folder of ``outputs`` (compared as files) under the + name of one of their files (as that folder compares names), by its path as given or by its real path: the writer + of that file would meet a folder, after the model has run. ``files`` says whose files they are.""" + if folder is None or outputs is None: + return + in_folder = InFolder(outputs.folder) + for path in dict.fromkeys((os.path.abspath(os.fspath(folder)), os.path.realpath(folder))): + output = outputs.holds(os.path.basename(path)) if in_folder(path) else None + if output is not None: + raise RefusalError( + f"{what} {os.fspath(folder)} is named as one of {files} {outputs.folder} ({output}), which could not " + f"be written over a folder: give a folder of another name, or one outside {outputs.folder}." + ) + + def refuse_assemblies_over_residues(names: Sequence, structure_names: Iterable, *, out_folder, out_format, res_folder, res_format, source: str, labels: Optional[Sequence] = None): """Refuse (RefusalError) assembly embeddings written in the format of the residue embeddings they are computed diff --git a/src/foldmatch/writer/batch_writer.py b/src/foldmatch/writer/batch_writer.py index 3cb92ed..d58b5a0 100644 --- a/src/foldmatch/writer/batch_writer.py +++ b/src/foldmatch/writer/batch_writer.py @@ -12,7 +12,8 @@ from foldmatch.errors import RefusalError from foldmatch.utils.output_files import name_key, name_problem, probe_folding from foldmatch.search.residue_shards import ( - SEQ_SHA1, SEQ_SHA1_BYTES, SEQ_SHA1_TYPE, check_residue_output, large_binary_array, pass_record, shard_schema, + ID, RESIDUE_CODES, SEQ_SHA1, SEQ_SHA1_BYTES, SEQ_SHA1_TYPE, check_residue_output, code_table_schema, + code_table_type, large_binary_array, pass_record, shard_schema, ) #: Default byte budget of a writer with a residue column or residue embeddings: buffered rows are written once they @@ -34,6 +35,15 @@ def __init__( self._written = {} self._folding = None + #: What this writer writes, in its refusals ("Each embedding of a run needs a file of its own", "Write the + #: embeddings as parquet or json"). + _ITEM = "embedding" + _ITEMS = "embeddings" + + def _what(self, dom_id) -> str: + """What this writer writes of ``dom_id``, in its refusals.""" + return f"The embedding {dom_id!r}" + def write_on_batch_end( self, trainer, @@ -72,8 +82,8 @@ def claim_file(self, dom_id) -> str: problem = name_problem(dom_id, suffix) if problem is not None: raise RefusalError( - f"The embedding {dom_id!r} cannot be written to {self.out_path}: its name {problem}, so it cannot name " - f"the file {suffix} there. Write the embeddings as parquet or json, which keep each under its " + f"{self._what(dom_id)} cannot be written to {self.out_path}: its name {problem}, so it cannot name " + f"the file {suffix} there. Write the {self._ITEMS} as parquet or json, which keep each under its " f"id, or give it another name." ) if self._folding is None: @@ -85,8 +95,8 @@ def claim_file(self, dom_id) -> str: note = f" (the folder does not tell {folded} apart)" if folded and first != dom_id else "" raise RefusalError( f"{self.file_name(first)} would be written twice by this run, for {first!r} and then for " - f"{dom_id!r}{note}: the first is kept, {dom_id!r} is not written, and the run stops. Each embedding of " - f"a run needs a file of its own: write them as parquet or json, which keep each under its id, or, " + f"{dom_id!r}{note}: the first is kept, {dom_id!r} is not written, and the run stops. Each {self._ITEM} " + f"of a run needs a file of its own: write them as parquet or json, which keep each under its id, or, " f"for names that differ only in case, to a folder of a case-sensitive file system." ) self._written[key] = dom_id @@ -403,3 +413,199 @@ def on_predict_end(self, trainer, pl_module): self._flush(trainer.global_rank) if self._writer is not None: self._writer.close() + + +# ===================================================================================================================== +# Residue codes in their own folder (fm-embedding from-sequences residue|chain --output-codes-folder) +# ===================================================================================================================== +def residue_code_tables(prediction, code_bytes: int = 8): + """The residue codes of a prediction ``(embeddings, ids, payloads, seq_sha1s)`` of a module with the + ``residue_codes`` output: ``(ids, tables, seq_sha1s)``, one writable ``uint8 [L, code_bytes]`` array per id (``[0, + code_bytes]`` for an empty payload, a protein whose codes could not be computed). ValueError for a prediction + without the payloads, or a payload that is not whole codes.""" + if prediction is None or len(prediction) < 4: + raise ValueError( + f"a residue-code writer needs predictions (embeddings, ids, payloads, seq_sha1s), got " + f"{'none' if prediction is None else f'a {len(prediction)}-tuple'}: the module computes no residue codes" + ) + ids, payloads, digests = tuple(prediction[1]), prediction[2], prediction[3] + if len(payloads) != len(ids) or len(digests) != len(ids): + raise ValueError(f"{len(ids)} ids but {len(payloads)} residue-code payloads and {len(digests)} seq_sha1s") + tables = [] + for dom_id, payload in zip(ids, payloads): + payload = bytes(payload) + if len(payload) % code_bytes: + raise ValueError(f"the residue codes of {dom_id!r} are {len(payload)} bytes, not a whole number of " + f"{code_bytes}-byte codes") + tables.append(np.frombuffer(payload, dtype=np.uint8).reshape(-1, code_bytes).copy()) + digests = [bytes(d) for d in digests] + bad = [i for i, d in zip(ids, digests) if len(d) != SEQ_SHA1_BYTES] + if bad: + raise ValueError(f"seq_sha1 must be {SEQ_SHA1_BYTES} bytes (e.g. for {bad[0]!r})") + return ids, tables, digests + + +def _check_codes_module(pl_module): + """Refuse, at the start of the pass and so on every rank, a module that does not compute residue codes.""" + produced = getattr(pl_module, "residue_output", None) + if produced != RESIDUE_CODES: + raise ValueError(f"this writer stores {RESIDUE_CODES}, but the module produces " + f"{produced or 'no residue output'}") + + +class _CodeFiles: + """The residue codes of each id to its own file, ``/.`` (as the embedding of the id is + ``/.``), by the item writer it is mixed into (csv or pt), which writes each ``uint8 [L, + 8]`` table as it writes an embedding.""" + + _ITEM = "residue-code table" + _ITEMS = "residue codes" + + def _what(self, dom_id) -> str: + return f"The residue codes of {dom_id!r}" + + def on_predict_start(self, trainer, pl_module): + _check_codes_module(pl_module) + + def write_on_batch_end(self, trainer, pl_module, prediction, batch_indices, batch, batch_idx, dataloader_idx): + if prediction is None: + return + ids, tables, _ = residue_code_tables(prediction) + deque(map(self._write_embedding, (torch.from_numpy(t) for t in tables), ids)) + + +class CsvCodeWriter(_CodeFiles, CsvBatchWriter): + """``/.csv``: an id's residue codes, L lines of 8 integers (0-255), no header or index (an empty + file for a protein without codes).""" + + def __init__(self, output_path, write_interval="batch"): + super().__init__(output_path, postfix="csv", write_interval=write_interval) + + +class TensorCodeWriter(_CodeFiles, TensorBatchWriter): + """``/.pt``: an id's residue codes, a CPU ``uint8`` tensor ``[L, 8]``.""" + + def __init__(self, output_path, write_interval="batch"): + super().__init__(output_path, postfix="pt", write_interval=write_interval, device="cpu") + + +class JsonCodeStorage(BasePredictionWriter): + """``/-.json.gz``: one record ``{"id", "residue_codes"}`` per id, its residue codes + a list of 8 integers per residue, written at the end of the pass as the embeddings' JSON (orient records, gzip).""" + + def __init__(self, output_path, codes_name, write_interval="batch"): + super().__init__(write_interval) + self.out_path = output_path + self.codes_name = codes_name + self._ids = [] + self._tables = [] + + def on_predict_start(self, trainer, pl_module): + _check_codes_module(pl_module) + + def write_on_batch_end(self, trainer, pl_module, prediction, batch_indices, batch, batch_idx, dataloader_idx): + if prediction is None: + return + ids, tables, _ = residue_code_tables(prediction) + self._ids.extend(ids) + self._tables.extend(tables) + + def on_predict_end(self, trainer, pl_module): + codes = np.empty(len(self._tables), dtype=object) + for k, table in enumerate(self._tables): + codes[k] = table + pd.DataFrame({'id': pd.Series(self._ids, dtype=object), RESIDUE_CODES: codes}).to_json( + f"{self.out_path}/{self.codes_name}-{trainer.global_rank}.json.gz", + orient='records', + compression='gzip' + ) + + +class ParquetCodeWriter(BasePredictionWriter): + """Per-rank ``/-.parquet`` of residue codes, written in row groups: columns ``id | + residue_codes | seq_sha1`` (``residue_codes`` one ``uint8 [L, 8]`` table per sequence, as + ``list>``), with the scoring system's name and digests and the pass record in the + schema metadata (:func:`foldmatch.search.residue_shards.code_table_schema`: not an embedding shard, which every + shard reader refuses). The pass record holds ``min_res_n`` and ``esm_revision`` given here, and the device type, + its ESM3 precision and the number of ranks, added at the start of the pass. Rows are buffered up to ``buffer_size`` + rows or ``byte_budget`` bytes of codes. No file for a rank without rows, as for the embeddings.""" + + def __init__(self, output_path, codes_name, system, *, min_res_n=None, esm_revision=None, write_interval="batch", + buffer_size=16384, byte_budget=DEFAULT_BYTE_BUDGET): + super().__init__(write_interval) + if system is None: + raise ValueError("a residue-code writer needs the scoring system (its schema metadata)") + if int(byte_budget) <= 0: + raise ValueError(f"byte_budget must be > 0, got {byte_budget!r}") + self.out_path = output_path + self.codes_name = codes_name + self.system = system + self.buffer_size = buffer_size + self.byte_budget = int(byte_budget) + self._pass = {"min_res_n": min_res_n, "esm_revision": esm_revision} + self._schema = code_table_schema(system, pass_record(**self._pass)) + self._writer = None + self._ids, self._tables, self._digests = [], [], [] + self._buffered_bytes = 0 + + @property + def schema(self): + return self._schema + + def on_predict_start(self, trainer, pl_module): + _check_codes_module(pl_module) + if self._writer is None: + device = getattr(getattr(pl_module, "device", None), "type", None) + world_size = getattr(trainer, "world_size", None) + self._schema = code_table_schema(self.system, pass_record( + **self._pass, accelerator=device, world_size=None if world_size is None else int(world_size))) + + def write_on_batch_end(self, trainer, pl_module, prediction, batch_indices, batch, batch_idx, dataloader_idx): + if prediction is None: + return + ids, tables, digests = residue_code_tables(prediction, self.system.code_bytes) + self._ids.extend(ids) + self._tables.extend(tables) + self._digests.extend(digests) + self._buffered_bytes += sum(t.nbytes for t in tables) + if len(self._ids) >= self.buffer_size or self._buffered_bytes >= self.byte_budget: + self._flush(trainer.global_rank) + + def _flush(self, rank): + if not self._ids: + return + width = self.system.code_bytes + offsets = np.concatenate(([0], np.cumsum([len(t) for t in self._tables]))).astype(np.int32) + values = np.concatenate(self._tables).reshape(-1) if self._tables else np.empty(0, dtype=np.uint8) + residues = pa.FixedSizeListArray.from_arrays(pa.array(values, type=pa.uint8()), width) + codes = pa.ListArray.from_arrays(pa.array(offsets, type=pa.int32()), residues) + table = pa.table({ + ID: pa.array(self._ids, type=pa.string()), + RESIDUE_CODES: codes.cast(code_table_type(self.system)), + SEQ_SHA1: pa.array(self._digests, type=SEQ_SHA1_TYPE), + }, schema=self._schema) + if self._writer is None: + self._writer = pq.ParquetWriter(f"{self.out_path}/{self.codes_name}-{rank}.parquet", self._schema, + compression='snappy') + self._writer.write_table(table) + self._ids.clear() + self._tables.clear() + self._digests.clear() + self._buffered_bytes = 0 + + def on_predict_end(self, trainer, pl_module): + self._flush(trainer.global_rank) + if self._writer is not None: + self._writer.close() + + +def residue_code_writer(out_format, codes_folder, codes_name, system, *, min_res_n=None, esm_revision=None): + """The writer of the residue codes of an fm-embedding pass (``--output-codes-folder``) in ``out_format``, to + ``codes_folder``: ``-.parquet`` or ``.json.gz``, else ``.csv`` or ``.pt`` (which + ``codes_name`` does not name).""" + fmt = getattr(out_format, "value", out_format) + if fmt == "parquet": + return ParquetCodeWriter(codes_folder, codes_name, system, min_res_n=min_res_n, esm_revision=esm_revision) + if fmt == "json": + return JsonCodeStorage(codes_folder, codes_name) + return TensorCodeWriter(codes_folder) if fmt == "pt" else CsvCodeWriter(codes_folder) diff --git a/tests/test_csv_embeddings.py b/tests/test_csv_embeddings.py index 64b8286..d34bf01 100644 --- a/tests/test_csv_embeddings.py +++ b/tests/test_csv_embeddings.py @@ -211,13 +211,22 @@ def test_a_csv_that_does_not_parse_is_named(self): self.assertLess(len(message), len(str(folder)) + 400) def test_a_csv_that_cannot_be_read_is_named(self): - """A folder named NAME.csv in an embedding folder, which the glob of .csv files matches, is refused with its - name, as a .pt that cannot be read is, not with a traceback.""" + """A .csv that cannot be read is refused with its name, as a .pt that cannot be read is, not with a traceback: + here a folder named NAME.csv given to the reader itself. In an embedding folder, such a folder, which the glob + of .csv files matches, is skipped with a WARNING (P9.X1, tests/test_embedding_folders.py), and the folder's + files are read.""" + from foldmatch.utils import data folder = self.tmp / "folder" (folder / "x.A.csv").mkdir(parents=True) - message = self.refusal(folder) + with self.assertRaises(RefusalError) as refused: + data._file_vector(folder / "x.A.csv") + message = str(refused.exception) self.assertTrue(message.startswith(f"{folder / 'x.A.csv'} could not be read ("), message) self.assertIn("Is a directory", message) + CsvBatchWriter(str(folder))._write_embedding(_vectors()["1abc-1"], "1abc-1") + with self.assertLogs("foldmatch.utils.data", "WARNING") as logs: + self.assertEqual(list(_read(folder)), ["1abc-1"]) + self.assertIn(f"{folder}: 1 folder(s) named like embedding files skipped, not read: x.A.csv.", logs.output[0]) def test_pandas_header_and_index_layouts_are_refused(self): """Files written with pandas' default header line (0,1,...,D-1) or index column are tables, refused with what diff --git a/tests/test_embedding_folders.py b/tests/test_embedding_folders.py new file mode 100644 index 0000000..3d2ff1d --- /dev/null +++ b/tests/test_embedding_folders.py @@ -0,0 +1,186 @@ +"""P9.X1: fm-search reads the embedding files of a folder, not its subfolders. + +'fm-search build embeddings' and 'query embedding' list the entries of --embedding-folder named *.parquet, *.pt and +*.csv (utils.data.embedding_files). A folder so named (the dataset folder ``x.parquet`` that Spark writes, a codes +folder ``codes.pt`` inside an embedding folder) was listed too, and refused ("is a folder, not a Parquet file"), so the +whole embedding folder was refused. Such a folder is now skipped, with one WARNING per command that names the skipped +folders; a symbolic link to a file is a file, one to a folder a folder; a folder that then holds no embedding file +keeps the "No embedding files found" error.""" +import logging +import tempfile +import unittest +from pathlib import Path +from types import SimpleNamespace +from unittest import mock + +import numpy as np +import torch +from typer.testing import CliRunner + +from foldmatch.cli import search +from foldmatch.utils import data +from foldmatch.utils.data import embedding_files, refuse_repeated_query_ids, stream_embeddings +from foldmatch.writer.batch_writer import CsvBatchWriter, ParquetBatchWriter, TensorBatchWriter + +_LOGGER = "foldmatch.utils.data" +_ADVICE = ("The embedding files of a folder are read, not those of its subfolders; to read the files of such a folder " + "(the parts of a Parquet dataset that Spark writes, say), give that folder itself.") + + +def _vectors(prefix: str, n: int = 4, dim: int = 16, seed: int = 0) -> dict: + generator = torch.Generator().manual_seed(seed) + return {f"{prefix}{i}.A": torch.randn(dim, generator=generator) for i in range(n)} + + +def _shard(folder: Path, embeddings: dict, name: str = "inference", rank: int = 0) -> Path: + """``-.parquet`` as fm-embedding writes chain embeddings.""" + writer = ParquetBatchWriter(str(folder), name) + trainer = SimpleNamespace(global_rank=rank) + writer.write_on_batch_end(trainer, None, (list(embeddings.values()), list(embeddings)), None, None, 0, 0) + writer.on_predict_end(trainer, None) + return folder / f"{name}-{rank}.parquet" + + +def _files(writer, folder: Path, embeddings: dict) -> Path: + files = writer(str(folder)) + for dom_id, embedding in embeddings.items(): + files._write_embedding(embedding, dom_id) + return folder + + +def _read(path, **kwargs): + ids, batches = [], [] + for batch_ids, batch in stream_embeddings(path, **kwargs): + ids.extend(batch_ids) + batches.append(batch) + return ids, np.concatenate(batches) + + +class EmbeddingFolderTest(unittest.TestCase): + + def setUp(self): + self.tmp = Path(self.enterContext(tempfile.TemporaryDirectory())).resolve() + + def folder(self, name: str) -> Path: + folder = self.tmp / name + folder.mkdir() + return folder + + def embeddings(self, name: str) -> Path: + """A folder of chain embeddings: a Parquet shard (c0.A..c3.A) and a .pt file (p0.A).""" + folder = self.folder(name) + _shard(folder, _vectors("c")) + _files(TensorBatchWriter, folder, _vectors("p", n=1, seed=1)) + return folder + + def with_folders(self, folder: Path) -> list: + """Adds folders named like embedding files to ``folder``: a Spark dataset folder ``dataset.parquet`` whose + part repeats the ids of the shard, a codes folder ``codes.pt``, an empty ``y.csv``, and ``link.parquet``, a + symbolic link to a folder elsewhere; and ``file-link.csv``, a symbolic link to a .csv embedding elsewhere (a + file, which is read). Returns the skipped folders' names, in the order of the listing (.pt, .csv, .parquet).""" + _shard(self.folder_in(folder, "dataset.parquet"), _vectors("c", seed=2), name="part") + _files(TensorBatchWriter, self.folder_in(folder, "codes.pt"), _vectors("p", n=1, seed=3)) + self.folder_in(folder, "y.csv") + elsewhere = self.folder(f"{folder.name}-elsewhere") + _shard(elsewhere, _vectors("c", seed=4)) + (folder / "link.parquet").symlink_to(elsewhere, target_is_directory=True) + csv = _files(CsvBatchWriter, self.folder(f"{folder.name}-csv"), _vectors("v", n=1, seed=5)) / "v0.A.csv" + (folder / "file-link.csv").symlink_to(csv) + return ["codes.pt", "y.csv", "dataset.parquet", "link.parquet"] + + @staticmethod + def folder_in(folder: Path, name: str) -> Path: + (folder / name).mkdir() + return folder / name + + def test_folders_named_like_embedding_files_are_skipped(self): + """The files are read as without the folders, a link to a file among them; one WARNING names the folders.""" + plain = self.embeddings("plain") + folder = self.embeddings("folders") + skipped = self.with_folders(folder) + with self.assertLogs(_LOGGER, "WARNING") as logs: + parquet_files, point_files = embedding_files(folder) + self.assertEqual(parquet_files, [folder / "inference-0.parquet"]) + self.assertEqual(point_files, [folder / "p0.A.pt", folder / "file-link.csv"]) + self.assertEqual(logs.output, [f"WARNING:{_LOGGER}:{folder}: 4 folder(s) named like embedding files skipped, " + f"not read: {', '.join(skipped)}. {_ADVICE}"]) + with self.assertLogs(_LOGGER, "WARNING") as logs: + ids, vectors = _read(folder) + self.assertEqual(len(logs.output), 1) + plain_ids, plain_vectors = _read(plain) + self.assertEqual(ids, plain_ids + ["file-link"]) + np.testing.assert_array_equal(vectors[:len(plain_ids)], plain_vectors) + with self.assertNoLogs(_LOGGER, "WARNING"): + self.assertEqual(embedding_files(folder, warn_folders=False), (parquet_files, point_files)) + _read(folder, warn_folders=False) + refuse_repeated_query_ids(folder) # the part of dataset.parquet repeats the shard's ids: not read + + def test_a_folder_with_only_such_folders(self): + """No embedding file: today's error, which names the skipped folders too; the WARNING as well.""" + folder = self.folder("only") + self.folder_in(folder, "dataset.parquet") + text = (f"No embedding files found with extensions ['.pt', '.csv', '.parquet'] in {folder} (1 folder(s) named " + f"like embedding files skipped: dataset.parquet)") + with self.assertLogs(_LOGGER, "WARNING") as logs, self.assertRaises(ValueError) as refused: + _read(folder) + self.assertEqual(str(refused.exception), text) + self.assertEqual(len(logs.output), 1) + with self.assertNoLogs(_LOGGER, "WARNING"), self.assertRaises(ValueError) as refused: + embedding_files(folder, warn_folders=False) + self.assertEqual(str(refused.exception), text) + + def test_many_folders_and_one_extension(self): + """The first 5 folders named, the rest counted; --file-extension lists (and skips) that extension only.""" + folder = self.embeddings("many") + for k in range(7): + self.folder_in(folder, f"d{k}.parquet") + self.folder_in(folder, "codes.pt") + with self.assertLogs(_LOGGER, "WARNING") as logs: + embedding_files(folder, ".parquet") + self.assertIn("7 folder(s) named like embedding files skipped, not read: d0.parquet, d1.parquet, d2.parquet, " + "d3.parquet, d4.parquet and 2 more.", logs.output[0]) + with self.assertLogs(_LOGGER, "WARNING") as logs: + self.assertEqual(embedding_files(folder, ".pt"), ([], [folder / "p0.A.pt"])) + self.assertIn("1 folder(s) named like embedding files skipped, not read: codes.pt.", logs.output[0]) + + def test_build_and_query_embeddings(self): + """'fm-search build embeddings' and 'query embedding' on a folder with such folders: the index, metadata and + hits of the folder without them, and one WARNING per command.""" + self.enterContext(mock.patch.object(search, "set_log_level")) + plain = self.embeddings("plain") + folder = self.embeddings("folders") + self.with_folders(folder) + (folder / "file-link.csv").unlink() # the same files as the plain folder + outputs = {} + for name, source in (("plain", plain), ("folders", folder)): + db = self.folder(f"db-{name}") / "db" + hits = db.parent / "hits.tsv" + warnings = [] + for args in (("build", "embeddings", "--embedding-folder", source, "--output-db", db), + ("query", "embedding", "--db-path", db, "--embedding-folder", source, "--output-file", hits, + "--threshold", "0")): + with self.assertLogs(_LOGGER, "INFO") as logs: + logging.getLogger(_LOGGER).info("start") + result = CliRunner().invoke(search.app, [str(a) for a in args]) + self.assertEqual(result.exit_code, 0, f"{args}: {result.output} {result.exception!r}") + warnings.append([line for line in logs.output if line.startswith("WARNING")]) + self.assertEqual([len(w) for w in warnings], [0, 0] if name == "plain" else [1, 1]) + outputs[name] = {suffix: Path(f"{db}{suffix}").read_bytes() for suffix in (".index", ".metadata")} + outputs[name]["hits"] = hits.read_bytes() + self.assertTrue(outputs["plain"]["hits"]) + self.assertEqual(outputs["folders"], outputs["plain"]) + + def test_a_skipped_folder_is_not_opened(self): + """A folder named *.parquet is never handed to the Parquet reader, which refuses a folder.""" + folder = self.embeddings("never-opened") + self.folder_in(folder, "dataset.parquet") + opened = [] + real = data._parquet_file + with mock.patch.object(data, "_parquet_file", side_effect=lambda f: opened.append(f) or real(f)), \ + self.assertLogs(_LOGGER, "WARNING"): + _read(folder) + self.assertEqual(opened, [folder / "inference-0.parquet"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_embedding_residue_codes.py b/tests/test_embedding_residue_codes.py new file mode 100644 index 0000000..cb11536 --- /dev/null +++ b/tests/test_embedding_residue_codes.py @@ -0,0 +1,2075 @@ +"""P9.X: ``fm-embedding from-sequences residue|chain --output-codes-folder``: the residue codes in the same ESM3 pass, +in a folder of their own. + +Most tests run the commands with ESM3 replaced by rows (``_esm3``: the F2c rows of the fixture FASTA, the FoldMatch CPU +fp32 ESM3 rows, and fixed pseudo-random rows for any other sequence), a stub aggregator and the real esm-codes-v1 +encoders (the registry bundle, ``esm_codes_bundle``: skipped only when offline and not cached), so the codes of the +fixture are F2c's exactly. ``RealEsm3Test`` runs the real ESM3 on the CPU: the embedding files with and without +``--output-codes-folder`` in every format of both commands, the codes against ``fm-search build sequences +--residue-codes``, and fm-search on the chain output folders. ``TwoRankSmoke`` (slow) runs two ranks as a command +and as a library caller. +""" +import functools +import gc +import gzip +import hashlib +import json +import logging +import os +import re +import shutil +import socket +import subprocess +import sys +import tempfile +import time +import unittest +import zlib +from contextlib import chdir, contextmanager +from datetime import timedelta +from pathlib import Path +from types import SimpleNamespace +from unittest import mock + +import numpy as np +import pandas as pd +import pyarrow as pa +import pyarrow.parquet as pq +import torch +from typer.testing import CliRunner + +import esm_codes_bundle +import foldmatch +from foldmatch.cli import embedding, search +from foldmatch.errors import MissingResidueCodesError, RefusalError +# Imported before any test patches their model getters +from foldmatch.inference import chain_complete_inference, sequence_inference +from foldmatch.modules import chain_complete_module, esm_module +from foldmatch.modules.chain_complete_module import ChainCompleteModule +from foldmatch.modules.esm_module import EsmModule +from foldmatch.modules.residue_payloads import ResiduePayloads +from foldmatch.search import residue_shards as rs +from foldmatch.search import scoring_system as ss +from foldmatch.search.residue_code_store import ResidueCodeStore +from foldmatch.types.api_types import Accelerator, OutFormat, SrcEsmFrom +from foldmatch.utils import data, output_files +from foldmatch.utils.output_files import EXACT, Folding, default_codes_name, same_folder + +_TEST_PATH = Path(__file__).parent +_FASTA = _TEST_PATH / "resources" / "fasta" / "test_sequences.fasta" +_F2C = _TEST_PATH / "resources" / "esm_codes" / "fasta_rows_codes_v1.npz" +_SRC = str(Path(foldmatch.__file__).resolve().parent.parent) +_CPU = ("--accelerator", "cpu", "--devices", "1") +_FORMATS = ("csv", "pt", "parquet", "json") +_COMMANDS = ("residue", "chain") +_LAUNCH_ENV = ("FOLDMATCH_RUN_ID", "SLURM_JOB_ID", "SLURM_STEP_ID", "SLURM_RESTART_COUNT", "SLURM_PROCID", + "SLURM_NTASKS", "SLURM_STEP_NUM_TASKS", "RANK", "LOCAL_RANK", "NODE_RANK", "WORLD_SIZE") +SYSTEM = ss.get(ss.DEFAULT_ESM_CODES) +#: A4_THRESHOLDS.json, gate A4-xcpu: CPU Mac vs CPU Linux code bits, also the F2c threshold. +A4_XCPU_BITS = 0.999 + + +@contextmanager +def launch_env(**values): + """Exactly ``values`` among the launch variables; the rest unset (only those variables are restored).""" + saved = {name: os.environ.get(name) for name in _LAUNCH_ENV} + try: + for name in _LAUNCH_ENV: + os.environ.pop(name, None) + os.environ.update({k: str(v) for k, v in values.items()}) + yield + finally: + for name, value in saved.items(): + if value is None: + os.environ.pop(name, None) + else: + os.environ[name] = value + + +@functools.lru_cache(maxsize=None) +def _f2c(): + """F2c: ``{sequence: ESM3 rows [L + 2, 1536] fp32}``, ``{id: reference codes [L, 8]}``, the ids and sequences.""" + z = np.load(_F2C, allow_pickle=False) + ro, offsets = z["rows_offsets"], z["offsets"] + ids = [str(i) for i in z["ids"]] + sequences = [str(s) for s in z["sequences"]] + rows = {sequences[k]: torch.from_numpy(z["rows"][ro[k]:ro[k + 1]].astype(np.float32)) for k in range(len(ids))} + codes = {ids[k]: z["codes"][offsets[k]:offsets[k + 1]] for k in range(len(ids))} + return rows, codes, ids, sequences + + +def _esm3(model, protein): + """Stands in for ESM3: the F2c rows of the fixture's sequences, fixed pseudo-random rows [L + 2, 1536] for any + other.""" + rows = _f2c()[0].get(protein.sequence) + if rows is not None: + return rows.clone() + generator = torch.Generator().manual_seed(zlib.crc32(protein.sequence.encode("utf-8", "surrogatepass"))) + return torch.randn(len(protein.sequence) + 2, 1536, generator=generator) + + +def _aggregate(x, x_mask): + """Stands in for the aggregator: the sum of each item's rows.""" + return x.masked_fill(x_mask.unsqueeze(-1), 0.0).sum(dim=1) + + +def _flat(text: str) -> str: + """Text on one line, typer's error panel borders and colour codes dropped.""" + return " ".join(re.sub(r"\x1b\[[0-9;]*m", "", text).translate({ord(c): " " for c in "│╭╮╰╯─"}).split()) + + +def _sha1(sequence: str) -> bytes: + return hashlib.sha1(sequence.encode("ascii")).digest() + + +def _case_insensitive() -> bool: + with tempfile.NamedTemporaryFile(prefix="CaseProbe-") as f: + head, name = os.path.split(f.name) + return os.path.exists(os.path.join(head, name.swapcase())) + + +_CASE_INSENSITIVE = _case_insensitive() + + +def read_codes(folder: Path, out_format: str, out_codes_name: str = "inference-codes", ids=None) -> dict: + """``{id: uint8 [L, 8]}`` of the residue-code files of a run in its codes folder ``folder``, read without + FoldMatch: ``-.parquet|json.gz``, or ``.csv|pt`` for each of ``ids``.""" + out = {} + if out_format == "parquet": + for path in sorted(folder.glob(f"{out_codes_name}-*.parquet")): + table = pq.read_table(path) + for dom_id, residues in zip(table.column("id").to_pylist(), table.column("residue_codes").to_pylist()): + out[dom_id] = np.array(residues, dtype=np.uint8).reshape(-1, 8) + elif out_format == "json": + for path in sorted(folder.glob(f"{out_codes_name}-*.json.gz")): + for record in json.loads(gzip.decompress(path.read_bytes())): + out[record["id"]] = np.array(record["residue_codes"], dtype=np.uint8).reshape(-1, 8) + else: + for dom_id in ids: + path = folder / f"{dom_id}.{out_format}" + if out_format == "pt": + tensor = torch.load(path, weights_only=True) + assert tensor.dtype == torch.uint8 and tensor.device.type == "cpu", (tensor.dtype, tensor.device) + out[dom_id] = tensor.numpy() + else: + text = path.read_text() + out[dom_id] = (np.zeros((0, 8), dtype=np.uint8) if not text else + pd.read_csv(path, header=None).to_numpy().astype(np.uint8)) + return out + + +def names(folder: Path) -> list: + return sorted(p.name for p in folder.iterdir()) + + +def same_bytes(a: Path, b: Path) -> bool: + """Whether two output files hold the same bytes; a .json.gz but for the gzip header's mtime (bytes 4-7), which + pandas sets to the time of writing.""" + x, y = a.read_bytes(), b.read_bytes() + if a.name.endswith(".gz"): + return x[:4] + x[8:] == y[:4] + y[8:] + return x == y + + +class _Case(unittest.TestCase): + """The commands with stub models (``_esm3``, ``_aggregate``) and the real encoders; ``refusing`` makes building a + model an error, so a refused run shows that it stopped before ESM3.""" + + @classmethod + def setUpClass(cls): + super().setUpClass() + esm_codes_bundle.registry_files() # unittest.SkipTest only when offline and not cached + + def setUp(self): + self.tmp = Path(self.enterContext(tempfile.TemporaryDirectory())).resolve() + self.enterContext(launch_env()) + for module in (sequence_inference, chain_complete_inference): + self.enterContext(mock.patch.object(module, "get_residue_model", self.model(torch.nn.Identity()))) + self.enterContext(mock.patch.object(chain_complete_inference, "get_aggregator_model", + self.model(_aggregate))) + for module in (esm_module, chain_complete_module): + self.enterContext(mock.patch.object(module, "per_residue_embedding", side_effect=_esm3)) + self.prepare = self.enterContext(mock.patch.object(ss, "prepare", side_effect=ss.prepare)) + self.refusing = False + + def model(self, model): + def get(**_): + if self.refusing: + raise AssertionError("a model was built") + return model + return get + + def folder(self, name: str = "out") -> Path: + return Path(tempfile.mkdtemp(prefix=f"{name}-", dir=self.tmp)) + + def fasta(self, *records, name=None) -> Path: + path = Path(name) if name else self.tmp / f"q{len(list(self.tmp.iterdir()))}.fasta" + path.write_text("".join(f">{header}\n{sequence}\n" for header, sequence in records), encoding="utf-8") + return path + + def invoke(self, *args, app=None, cpu=True): + root = logging.getLogger() + handlers, level = root.handlers[:], root.level + try: + return CliRunner().invoke(app or embedding.app, [str(a) for a in (*args, *(_CPU if cpu else ()))]) + finally: + root.handlers[:] = handlers + root.setLevel(level) + + def run_ok(self, command, fasta, out, *extra): + self.refusing = False + result = self.invoke("from-sequences", command, "--fasta-file", fasta, "--output-folder", out, *extra) + self.assertEqual(result.exit_code, 0, f"{result.output}\n{result.exception!r}") + return result + + def run_search(self, *args): + result = self.invoke(*args, app=search.app, cpu=False) + self.assertEqual(result.exit_code, 0, f"{result.output}\n{result.exception!r}") + return result + + def refused(self, *args, written=()) -> str: + """The command exits 1 with a RefusalError (D20) before any model, writing nothing to ``written``; returns + its message on one line.""" + before = {folder: sorted(p.name for p in folder.iterdir()) for folder in written} + self.refusing = True + try: + result = self.invoke(*args) + finally: + self.refusing = False + self.assertEqual(result.exit_code, 1, f"{result.output}\n{result.exception!r}") + self.assertIsInstance(result.exception, RefusalError, repr(result.exception)) + for folder, names in before.items(): + self.assertEqual(sorted(p.name for p in folder.iterdir()), names, folder) + return _flat(str(result.exception)) + + +# ===================================================================================================================== +# The options +# ===================================================================================================================== +class OptionTest(unittest.TestCase): + + def help(self, command): + result = CliRunner().invoke(embedding.app, ["from-sequences", command, "--help"], env={"COLUMNS": "200"}) + self.assertEqual(result.exit_code, 0, result.output) + return _flat(result.output) + + def test_both_commands_have_the_options_with_one_help(self): + texts = {command: self.help(command) for command in _COMMANDS} + for command, text in texts.items(): + with self.subTest(command=command): + self.assertIn("--output-codes-folder", text) + self.assertIn("--output-codes-name", text) + self.assertNotIn("--residue-codes --no-residue-codes", text) + self.assertIn("Also compute each sequence's residue codes in the same ESM3 pass and write them to " + "this folder", text) + self.assertIn("Used when output-format is parquet or json", text) + self.assertIn("(default -codes)", text) + self.assertIn("must not be --output-folder (a subfolder of it will do):", text) + self.assertNotIn(".parquet, .pt or .csv", text) + self.assertIn("A sequence whose codes cannot be computed gets an empty table, and the run exits with " + "status 1 once every file is written.", text) + self.assertIn("row i + 1 of the residue embeddings that 'from-sequences residue' writes", text) + self.assertIn("on several nodes, or under srun or torchrun, run 'fm-embedding download-models' first", + text) + self.assertIn("A run that would replace an embedding file there is refused.", text) + start = lambda text: text[text.index("--output-codes-folder", text.index("--output-name")): # noqa: E731 + text.index("--min-res-n")] + self.assertEqual(start(texts["residue"]), start(texts["chain"])) + for command in ("residue", "chain", "assembly"): + result = CliRunner().invoke(embedding.app, ["from-structures", command, "--help"], + env={"COLUMNS": "200"}) + texts[command] = _flat(result.output) + self.assertNotIn("--output-codes-folder PATH", texts[command], command) + self.assertNotIn("--output-codes-name", texts[command], command) + for command, text in texts.items(): + # Z6: every command's output folder + self.assertIn("Output folder to store predictions. A run that would replace a file of residue codes " + "there (as 'fm-embedding from-sequences ... --output-codes-folder' writes them) is refused.", + text, command) + + def test_the_codes_name_without_the_codes_folder_is_a_usage_error(self): + """V5: --output-codes-name alone is typer.BadParameter, exit status 2, before anything runs.""" + with tempfile.TemporaryDirectory() as tmp, \ + mock.patch.object(sequence_inference, "predict", side_effect=AssertionError("ran")), \ + mock.patch.object(chain_complete_inference, "predict", side_effect=AssertionError("ran")): + for command in _COMMANDS: + with self.subTest(command=command): + result = CliRunner().invoke(embedding.app, [ + "from-sequences", command, "--fasta-file", str(_FASTA), "--output-folder", tmp, + "--output-codes-name", "x"]) + self.assertEqual(result.exit_code, 2, result.output) + self.assertIn("--output-codes-name", _flat(result.output)) + self.assertIn("names the residue-code files of --output-codes-folder, which is not given.", + _flat(result.output)) + + def test_the_codes_folder_must_exist(self): + with tempfile.TemporaryDirectory() as tmp: + for codes in (Path(tmp) / "missing", _FASTA): + with self.subTest(codes=codes.name): + result = CliRunner().invoke(embedding.app, [ + "from-sequences", "chain", "--fasta-file", str(_FASTA), "--output-folder", tmp, + "--output-codes-folder", str(codes)]) + self.assertEqual(result.exit_code, 2, result.output) + self.assertIn("--output-codes-folder", _flat(result.output)) + + def test_default_codes_name(self): + self.assertEqual(default_codes_name("inference"), "inference-codes") + self.assertEqual(default_codes_name("x"), "x-codes") + + +# ===================================================================================================================== +# Without --output-codes-folder nothing changes +# ===================================================================================================================== +class PlainRunTest(_Case): + + def test_a_plain_run_is_unchanged(self): + """The legacy module and writers, no esm-codes bundle, no strict FASTA pre-flight (a '<' still runs).""" + fasta = self.fasta(("a", "MKV"), ("b", "MKVLA")) + made, writers = [], [] + real_trainer = sequence_inference.Trainer + + def trainer(**kwargs): + writers.append([type(c).__name__ for c in kwargs["callbacks"] or []]) + return real_trainer(**kwargs) + + for command, module_path, module_class in (("residue", sequence_inference, EsmModule), + ("chain", chain_complete_inference, ChainCompleteModule)): + for out_format in _FORMATS: + with self.subTest(command=command, out_format=out_format): + made.clear() + writers.clear() + construct = lambda *a, _cls=module_class, **k: made.append(k) or _cls(*a, **k) # noqa: E731 + with mock.patch.object(module_path, module_class.__name__, side_effect=construct), \ + mock.patch.object(module_path, "Trainer", side_effect=trainer), \ + mock.patch.object(chain_complete_inference, "strict_fasta_preflight") as strict, \ + mock.patch.object(chain_complete_inference, "residue_code_writer") as code_writer: + self.run_ok(command, fasta, self.folder(), "--output-format", out_format) + self.assertEqual(len(made), 1) + self.assertNotIn("residue_head", made[0]) + self.assertNotIn("code_files", made[0]) + self.assertEqual(len(writers[0]), 1, writers) + strict.assert_not_called() + code_writer.assert_not_called() + self.prepare.assert_not_called() + + +# ===================================================================================================================== +# The code files +# ===================================================================================================================== +class CodeFilesTest(_Case): + + def test_every_format_of_both_commands(self): + """U2/U4/V3/V4: the F2c codes of the fixture, one [L, 8] table per sequence, in the codes folder (csv and pt + ., mirroring the embedding files; parquet and json -codes-), and the output + folder holds the embedding files of a run without --output-codes-folder, byte for byte, and nothing else.""" + _, reference, ids, _ = _f2c() + for command in _COMMANDS: + for out_format in _FORMATS: + with self.subTest(command=command, out_format=out_format): + plain, out, codes = self.folder("plain"), self.folder("out"), self.folder("codes") + self.run_ok(command, _FASTA, plain, "--output-format", out_format) + self.run_ok(command, _FASTA, out, "--output-format", out_format, "--output-codes-folder", codes) + self.assertEqual(names(out), names(plain)) + for name in names(plain): + self.assertTrue(same_bytes(plain / name, out / name), name) + if out_format in ("csv", "pt"): + self.assertEqual(names(codes), names(plain)) + self.assertEqual(names(codes), sorted(f"{i}.{out_format}" for i in ids)) + else: + suffix = "parquet" if out_format == "parquet" else "json.gz" + self.assertEqual(names(codes), [f"inference-codes-0.{suffix}"]) + got = read_codes(codes, out_format, ids=ids) + self.assertEqual(list(got), ids) + for dom_id in ids: + np.testing.assert_array_equal(got[dom_id], reference[dom_id]) + + def test_the_layout_of_each_format(self): + _, reference, ids, sequences = _f2c() + out, codes = self.folder(), self.folder("codes") + self.run_ok("chain", _FASTA, out, "--output-format", "csv", "--output-codes-folder", codes) + lines = (codes / f"{ids[1]}.csv").read_text().splitlines() + self.assertEqual(len(lines), 58) + self.assertEqual(lines[0], ",".join(str(v) for v in reference[ids[1]][0])) + out, codes = self.folder(), self.folder("codes") + self.run_ok("residue", _FASTA, out, "--output-format", "pt", "--output-codes-folder", codes) + tensor = torch.load(codes / f"{ids[0]}.pt", weights_only=True) + self.assertEqual((tensor.dtype, tuple(tensor.shape), tensor.device.type), (torch.uint8, (245, 8), "cpu")) + self.assertEqual(tensor.untyped_storage().nbytes(), 245 * 8) + out, codes = self.folder(), self.folder("codes") + self.run_ok("residue", _FASTA, out, "--output-format", "json", "--output-codes-folder", codes) + records = json.loads(gzip.decompress((codes / "inference-codes-0.json.gz").read_bytes())) + self.assertEqual([sorted(r) for r in records], [["id", "residue_codes"]] * 2) + self.assertEqual(records[1]["residue_codes"][0], reference[ids[1]][0].tolist()) + + def test_the_parquet_layout_and_provenance(self): + _, reference, ids, sequences = _f2c() + for command in _COMMANDS: + with self.subTest(command=command): + out, codes = self.folder(), self.folder("codes") + self.run_ok(command, _FASTA, out, "--output-format", "parquet", "--output-codes-folder", codes, + "--min-res-n", "50") + schema = pq.read_schema(codes / "inference-codes-0.parquet") + self.assertEqual(schema.names, ["id", "residue_codes", "seq_sha1"]) + self.assertEqual(schema.field("residue_codes").type, pa.list_(pa.list_(pa.uint8(), 8))) + self.assertEqual(schema.field("seq_sha1").type, pa.binary(20)) + meta = schema.metadata + self.assertEqual(meta[b"foldmatch.residue_output"], b"residue_code_table") + self.assertEqual(meta[b"foldmatch.system"], SYSTEM.name.encode()) + self.assertEqual(meta[b"foldmatch.encoding_digest"], SYSTEM.encoding_digest().encode()) + self.assertEqual(meta[b"foldmatch.system_digest"], SYSTEM.digest().encode()) + record = json.loads(meta[b"foldmatch.pass"]) + from foldmatch.utils.esm.loaders import ESM_REVISION + self.assertEqual({k: record[k] for k in ("min_res_n", "accelerator", "world_size", "esm_precision", + "encoder_precision", "esm_revision")}, + {"min_res_n": 50, "accelerator": "cpu", "world_size": 1, "esm_precision": "fp32", + "encoder_precision": "fp32-no-tf32", "esm_revision": ESM_REVISION}) + table = pq.read_table(codes / "inference-codes-0.parquet") + self.assertEqual(table.column("seq_sha1").to_pylist(), [_sha1(s) for s in sequences]) + # The embeddings' Parquet is untouched: no metadata, the legacy columns for chain embeddings + self.assertEqual(names(out), ["inference-0.parquet"]) + embeddings = pq.read_schema(out / "inference-0.parquet") + self.assertIsNone(embeddings.metadata) + if command == "chain": + self.assertEqual(embeddings.names, ["id", "embedding"]) + + def test_min_res_n_filters_both(self): + _, reference, ids, _ = _f2c() + for out_format in _FORMATS: + with self.subTest(out_format=out_format): + out, codes = self.folder(), self.folder("codes") + self.run_ok("residue", _FASTA, out, "--output-format", out_format, "--output-codes-folder", codes, + "--min-res-n", "100") + self.assertEqual(list(read_codes(codes, out_format, ids=ids[:1])), ids[:1]) + if out_format in ("csv", "pt"): + self.assertEqual(names(out), [f"{ids[0]}.{out_format}"]) + self.assertEqual(names(codes), [f"{ids[0]}.{out_format}"]) + + def test_the_codes_name(self): + """V3/V4: --output-codes-name names the parquet and json files (by default -codes; the output + name itself is allowed, in another folder), and no file in csv and pt.""" + _, reference, ids, _ = _f2c() + cases = [("csv", ("--output-codes-name", "esmc"), [f"{i}.csv" for i in ids]), + ("pt", (), [f"{i}.pt" for i in ids]), + ("parquet", ("--output-name", "emb"), ["emb-codes-0.parquet"]), + ("parquet", ("--output-codes-name", "inference"), ["inference-0.parquet"]), + ("json", ("--output-codes-name", "c"), ["c-0.json.gz"])] + for out_format, extra, files in cases: + with self.subTest(out_format=out_format, extra=extra): + out, codes = self.folder(), self.folder("codes") + self.run_ok("chain", _FASTA, out, "--output-format", out_format, "--output-codes-folder", codes, + *extra) + self.assertEqual(names(codes), sorted(files)) + self.assertNotIn("esmc", " ".join(names(out))) + self.assertEqual(list(read_codes(codes, "json", "c")), ids) + + def test_nested_folders(self): + """V2: one folder inside the other is allowed, both ways.""" + _, reference, ids, _ = _f2c() + for out_format in ("csv", "parquet"): + for name in ("codes", "codes.parquet", "codes.pt", "codes.csv"): # W2: fm-search skips folders + with self.subTest(out_format=out_format, codes=f"inside the output folder, {name}"): + out = self.folder() + (out / name).mkdir() + self.run_ok("chain", _FASTA, out, "--output-format", out_format, "--output-codes-folder", + out / name) + self.assertEqual(list(read_codes(out / name, out_format, ids=ids)), ids) + with self.subTest(out_format=out_format, codes="around the output folder"): + codes = self.folder("codes") + (codes / "embeddings").mkdir() + self.run_ok("chain", _FASTA, codes / "embeddings", "--output-format", out_format, + "--output-codes-folder", codes) + self.assertEqual(list(read_codes(codes, out_format, ids=ids)), ids) + + def test_a_rerun_replaces_its_files(self): + out, codes = self.folder(), self.folder("codes") + for out_format in ("csv", "parquet"): + for _ in range(2): + self.run_ok("residue", _FASTA, out, "--output-format", out_format, "--output-codes-folder", codes) + + def test_codes_equal_the_build_store(self): + """U4: the codes are the target codes that 'fm-search build sequences --residue-codes' stores for the FASTA, + same device and min_res_n.""" + _, _, ids, _ = _f2c() + db = self.folder("db") + shards = self.folder("shards") + with launch_env(FOLDMATCH_RUN_ID="p9x-build"): + result = self.invoke("build", "sequences", "--fasta-file", _FASTA, "--output-db", db / "db", + "--tmp-embedding-folder", shards, "--residue-codes", app=search.app) + self.assertEqual(result.exit_code, 0, f"{result.output}\n{result.exception!r}") + store = ResidueCodeStore(db, "db") + store.check_compatible() + memmap = np.asarray(store.open_memmap()) + entries = store.lookup(ids) + stored = {i: memmap[entries[i][0]:entries[i][0] + entries[i][1]] for i in ids} + for command in _COMMANDS: + for out_format in _FORMATS: + with self.subTest(command=command, out_format=out_format): + codes = self.folder("codes") + self.run_ok(command, _FASTA, self.folder(), "--output-format", out_format, + "--output-codes-folder", codes) + got = read_codes(codes, out_format, ids=ids) + for dom_id in ids: + np.testing.assert_array_equal(got[dom_id], stored[dom_id]) + + def test_a_protein_without_codes_gets_an_empty_table(self): + """W4: ESM3 rows that do not match the sequence's length: an ERROR per protein and a count, an empty [0, 8] + table, the embedding written as without --output-codes-folder, every file written; then the run ends with + exit status 1, a MissingResidueCodesError (a RefusalError, D20) naming the count, the ids and where the empty + tables are.""" + _, reference, ids, sequences = _f2c() + + def one_row_short(model, protein): + rows = _esm3(model, protein) + return rows[:-1] if protein.sequence == sequences[1] else rows + + for command, module in (("residue", esm_module), ("chain", chain_complete_module)): + for out_format in _FORMATS: + with self.subTest(command=command, out_format=out_format), \ + mock.patch.object(module, "per_residue_embedding", side_effect=one_row_short): + plain = self.folder("plain") + self.run_ok(command, _FASTA, plain, "--output-format", out_format) + out, codes = self.folder(), self.folder("codes") + with self.assertLogs("foldmatch.modules.residue_payloads", "ERROR") as logs: + result = self.invoke("from-sequences", command, "--fasta-file", _FASTA, "--output-folder", + out, "--output-format", out_format, "--output-codes-folder", codes) + self.assertEqual(result.exit_code, 1, f"{result.output}\n{result.exception!r}") + self.assertIsInstance(result.exception, MissingResidueCodesError) + self.assertEqual((result.exception.count, result.exception.ids), (1, (ids[1],))) + where = {"csv": f"{codes}/.csv of each, an empty file", + "pt": f"{codes}/.pt of each, a uint8 tensor [0, 8]", + "parquet": f"rows of {codes}/inference-codes-0.parquet without residues", + "json": f"records of {codes}/inference-codes-0.json.gz without residues"}[out_format] + self.assertEqual(str(result.exception), + f"1 sequence(s) got no residue codes: {ids[1]!r}. ESM3 gave rows that do not " + f"match the sequence's length, or the encoding failed (each one's reason is " + f"logged above). Their codes are written as empty [0, 8] tables " + f"({where}); their embeddings and every other file of the run are written. Check " + f"those sequences, or leave them out of the FASTA file.") + got = read_codes(codes, out_format, ids=ids) + np.testing.assert_array_equal(got[ids[0]], reference[ids[0]]) + self.assertEqual(got[ids[1]].shape, (0, 8)) + if out_format == "csv": + self.assertEqual((codes / f"{ids[1]}.csv").read_bytes(), b"") + self.assertEqual(names(out), names(plain)) # the embeddings of a run without the option + for name in names(plain): + self.assertTrue(same_bytes(plain / name, out / name), name) + text = "\n".join(logs.output) + self.assertIn(f"{ids[1]}: no residue codes (ESM3 gave (59, 1536) rows for 58 residues", text) + self.assertIn("1 protein(s) on this rank got no residue codes: their codes are written as empty " + "[0, 8] tables", text) + + def test_many_proteins_without_codes(self): + """The first 5 ids named, the rest counted.""" + records = [(f"s{k}", "MKVLA" + "A" * k) for k in range(8)] + fasta = self.fasta(*records) + + def short(model, protein): + return _esm3(model, protein)[:-1] + + with mock.patch.object(esm_module, "per_residue_embedding", side_effect=short), \ + self.assertLogs("foldmatch.modules.residue_payloads", "ERROR"): + result = self.invoke("from-sequences", "residue", "--fasta-file", fasta, "--output-folder", + self.folder(), "--output-codes-folder", self.folder("codes")) + self.assertEqual(result.exit_code, 1, result.output) + self.assertTrue(str(result.exception).startswith( + "8 sequence(s) got no residue codes: 's0', 's1', 's2', 's3', 's4' and 3 more. "), str(result.exception)) + self.assertEqual(result.exception.ids, tuple(f"s{k}" for k in range(8))) + + +# ===================================================================================================================== +# fm-search reads the output folder of a chain run with residue codes +# ===================================================================================================================== +class FmSearchTest(_Case): + """The reason for the codes folder (SPEC2): the output folder of 'from-sequences chain --output-codes-folder' is + read by 'fm-search build embeddings' and 'query embedding' as that of a plain run: the same index bytes, the same + hits; with the codes folder elsewhere or inside the output folder (fm-search reads a folder, not its subfolders), + even under a name that ends like an embedding file's (P9.X1: fm-search skips it, with a WARNING).""" + + def test_build_and_query_embeddings(self): + for out_format in ("parquet", "csv", "pt"): + plain = self.folder("plain") + self.run_ok("chain", _FASTA, plain, "--output-format", out_format) + db = self.folder("db") / "db" + self.run_search("build", "embeddings", "--embedding-folder", plain, "--output-db", db) + want = {suffix: Path(f"{db}{suffix}").read_bytes() for suffix in (".index", ".metadata")} + self.run_search("query", "embedding", "--db-path", db, "--embedding-folder", plain, "--output-file", + db.parent / "hits.tsv", "--threshold", "0") + hits = (db.parent / "hits.tsv").read_bytes() + self.assertTrue(hits) + for where in ("elsewhere", "inside", "inside, named like an embedding file"): + with self.subTest(out_format=out_format, codes=where): + out = self.folder("out") + codes = {"elsewhere": self.folder("codes"), "inside": out / "codes"}.get( + where, out / f"codes.{out_format}") + codes.mkdir(exist_ok=True) + self.run_ok("chain", _FASTA, out, "--output-format", out_format, "--output-codes-folder", codes) + self.assertTrue(list(codes.iterdir())) + db = self.folder("db") / "db" + result = db.parent / "hits.tsv" + with self.assertLogs("foldmatch.utils.data", "INFO") as logs: + logging.getLogger("foldmatch.utils.data").info("start") + self.run_search("build", "embeddings", "--embedding-folder", out, "--output-db", db) + self.run_search("query", "embedding", "--db-path", db, "--embedding-folder", out, + "--output-file", result, "--threshold", "0") + for suffix in (".index", ".metadata"): + self.assertEqual(Path(f"{db}{suffix}").read_bytes(), want[suffix], suffix) + self.assertEqual(result.read_bytes(), hits) + skipped = [line for line in logs.output if "folder(s) named like embedding files skipped" in line] + self.assertEqual(len(skipped), 2 if codes.name.endswith(f".{out_format}") else 0, logs.output) + + +# ===================================================================================================================== +# Refusals, before ESM3 +# ===================================================================================================================== +class RefusalTest(_Case): + + def args(self, command, fasta, out, codes, *extra): + return ("from-sequences", command, "--fasta-file", fasta, "--output-folder", out, "--output-codes-folder", + codes, *extra) + + def same_folder_refused(self, out, given): + """W1: the run with --output-codes-folder ``given`` (one folder with ``out``) is a usage error, exit status 2 + (typer.BadParameter on --output-codes-folder), before anything is imported or written: the message names both + options and the folder.""" + message = None + for command in _COMMANDS: + before = names(Path(out)) + with mock.patch.object(sequence_inference, "predict", side_effect=AssertionError("ran")), \ + mock.patch.object(chain_complete_inference, "predict", side_effect=AssertionError("ran")), \ + mock.patch.object(output_files, "check_outputs", side_effect=AssertionError("checked")): + self.refusing = True + try: + result = CliRunner().invoke(embedding.app, [str(a) for a in (*self.args(command, _FASTA, out, + given), *_CPU)], + env={"COLUMNS": "2000"}) + finally: + self.refusing = False + self.assertEqual(result.exit_code, 2, f"{result.output}\n{result.exception!r}") + self.assertEqual(names(Path(out)), before) + message = _flat(result.output) + self.assertRegex(message, r"Invalid value for '?--output-codes-folder'?: --output-folder and " + r"--output-codes-folder are one folder \(.+\): the residue codes go to a folder " + r"of their own, so that the embedding folder holds embeddings only \('fm-search " + r"build embeddings' and 'fm-search query embedding' read every \.parquet, \.pt " + r"and \.csv file of it\)\. Give --output-codes-folder another folder \(a " + r"subfolder of --output-folder will do\)\.") + return message + + def test_the_codes_folder_is_the_output_folder(self): + """V2: by path, through a symbolic link, as '.' (a relative path), by a case variant on a case-insensitive + volume; compared as files.""" + out = self.folder() + self.assertIn(f"({out})", self.same_folder_refused(out, out)) + link = self.tmp / "link" + link.symlink_to(out, target_is_directory=True) + self.same_folder_refused(out, link) + self.same_folder_refused(link, out) + with chdir(out): + self.same_folder_refused(out, ".") + with chdir(out.parent): + self.same_folder_refused(out, out.name) + self.same_folder_refused(Path(out.name), out) + + @unittest.skipUnless(_CASE_INSENSITIVE, "a case-insensitive file system (macOS APFS)") + def test_the_codes_folder_is_the_output_folder_in_another_case(self): + out = self.folder("Out") + variant = out.parent / out.name.swapcase() + self.assertNotEqual(str(variant), str(out)) + self.same_folder_refused(out, variant) + + def test_the_same_folder_before_torch(self): + """W1: the usage error comes before torch is imported (a fresh process).""" + out = self.folder() + code = ("import sys; from typer.testing import CliRunner; from foldmatch.cli import embedding; " + "r = CliRunner().invoke(embedding.app, sys.argv[1:]); print(r.exit_code, 'torch' in sys.modules)") + env = dict(os.environ, PYTHONPATH=os.pathsep.join([_SRC] + [p for p in [os.environ.get("PYTHONPATH")] if p])) + for command in _COMMANDS: + with self.subTest(command=command): + run = subprocess.run([sys.executable, "-c", code, *map(str, self.args(command, _FASTA, out, out))], + env=env, capture_output=True, text=True, timeout=600) + self.assertEqual(run.stdout.split(), ["2", "False"], run.stderr[-4000:]) + + def test_a_folder_named_as_a_file_of_the_other(self): + """One folder inside the other is allowed (V2), but not under the name of a file that the other's output + writes (.csv|pt, -.parquet|json.gz), as that folder compares names: refused before ESM3 (exit + status 1), not by the writer after the pass.""" + _, _, ids, _ = _f2c() + for command in _COMMANDS: + for out_format, name in (("csv", f"{ids[0]}.csv"), ("pt", f"{ids[1]}.pt"), + ("parquet", "inference-0.parquet"), ("json", "inference-1.json.gz")): + with self.subTest(command=command, out_format=out_format, codes="inside the output folder"): + out = self.folder() + codes = out / name + codes.mkdir() + message = self.refused(*self.args(command, _FASTA, out, codes, "--output-format", out_format), + written=[out, codes]) + self.assertTrue(message.startswith(f"The codes folder {codes} is named as one of the embedding " + f"files of the output folder {out} ("), message) + self.assertTrue(message.endswith(f"), which could not be written over a folder: give a folder " + f"of another name, or one outside {out}."), message) + with self.subTest(command=command, out_format=out_format, codes="around the output folder"): + codes = self.folder("codes") + out = codes / name.replace("inference-", "inference-codes-") + out.mkdir() + message = self.refused(*self.args(command, _FASTA, out, codes, "--output-format", out_format), + written=[out, codes]) + self.assertTrue(message.startswith(f"The output folder {out} is named as one of the code files " + f"of the codes folder {codes} ("), message) + out = self.folder() + (out / f"{ids[0]}.pt").mkdir() # a file name of another format is no output file + self.run_ok("chain", _FASTA, out, "--output-format", "csv", "--output-codes-folder", out / f"{ids[0]}.pt") + if _CASE_INSENSITIVE: + with self.subTest(codes="a case variant on a case-insensitive volume"): + out = self.folder() + codes = out / f"{ids[0].swapcase()}.csv" + codes.mkdir() + message = self.refused(*self.args("chain", _FASTA, out, codes), written=[out, codes]) + self.assertIn(f"(the output {ids[0]!r})", message) + with self.subTest(codes="through a symbolic link to such a folder"): + out, elsewhere = self.folder(), self.folder("elsewhere") + (out / f"{ids[1]}.csv").mkdir() + (elsewhere / "link").symlink_to(out / f"{ids[1]}.csv", target_is_directory=True) + self.refused(*self.args("residue", _FASTA, out, elsewhere / "link"), written=[out]) + + def test_an_unusable_codes_name(self): + """Checked whatever the format, as --output-name is.""" + for command in _COMMANDS: + for out_format in _FORMATS: + for name, problem in (("a/b", "holds '/'"), ("..", "is '..'"), + ("x" * 241, "makes a file name of 256 bytes once a rank and a suffix are added")): + with self.subTest(command=command, out_format=out_format, name=name[:5]): + out, codes = self.folder(), self.folder("codes") + message = self.refused(*self.args(command, _FASTA, out, codes, "--output-format", out_format, + "--output-codes-name", name), written=[out, codes]) + self.assertEqual(message, f"The codes name {name!r} {problem}: it names the files " + f"-.parquet or -.json.gz of the codes " + f"folder. Give a name that is not empty, without '/' or NUL, other " + f"than '.' and '..', of at most 240 bytes in UTF-8.") + self.prepare.assert_not_called() + + def test_ids_that_the_codes_folder_takes_for_one_file(self): + """csv and pt: the ids are checked as the codes folder compares names, which can differ from the output + folder (a case-folding codes folder beside a case-sensitive output folder; probe_folding stubbed).""" + fasta = self.fasta(("a", "MKVLA"), ("A", "MKVL")) + for out_format in ("csv", "pt"): + with self.subTest(out_format=out_format): + out, codes = self.folder(), self.folder("codes") + folding = lambda folder: (Folding(case=True, unicode=True, full=True) # noqa: E731 + if same_folder(folder, codes) else EXACT) + with mock.patch.object(output_files, "probe_folding", side_effect=folding): + message = self.refused(*self.args("chain", fasta, out, codes, "--output-format", out_format), + written=[out, codes]) + self.assertEqual(message, f"{fasta}: 1 file(s) of the codes folder would each be written for more " + f"than one FASTA id (the codes folder does not tell upper and lower case " + f"apart): 'a' (record 1) and 'A' (record 2) give a.{out_format}. Each one " + f"would replace the one before, as the folder takes those names for one: " + f"write them to a folder of a file system that tells upper and lower case " + f"apart or as parquet or json (which keep each FASTA id as given), or give " + f"those records FASTA ids that differ otherwise.") + + def test_the_fasta_file_among_the_code_files(self): + _, _, ids, _ = _f2c() + for out_format, name in (("csv", f"{ids[0]}.csv"), ("pt", f"{ids[1]}.pt"), + ("parquet", "inference-codes-0.parquet"), ("json", "inference-codes-3.json.gz")): + with self.subTest(out_format=out_format): + out, codes = self.folder(), self.folder("codes") + fasta = codes / name + shutil.copyfile(_FASTA, fasta) + message = self.refused(*self.args("residue", fasta, out, codes, "--output-format", out_format), + written=[out, codes]) + self.assertIn(f"1 input(s) of this run would be replaced by its output, in the codes folder {codes}: " + f"the FASTA file {fasta} by", message) + + def test_fasta_the_residue_pass_cannot_take(self): + """The strict pre-flight of a residue pass, only with --output-codes-folder: before the bundle and ESM3.""" + for records, what in (((("a", "MKV"),), "contain '<' or '>'"), ((("a", "MKÉV"),), "non-ASCII")): + fasta = self.fasta(*records) + for command in _COMMANDS: + for out_format in ("csv", "parquet"): + with self.subTest(what=what, command=command, out_format=out_format): + message = self.refused(*self.args(command, fasta, self.folder(), self.folder("codes"), + "--output-format", out_format)) + self.assertIn(what, message) + self.run_ok(command, fasta, self.folder(), "--output-format", out_format) + self.prepare.assert_not_called() + + def test_a_repeated_id(self): + fasta = self.fasta(("a", "MKV"), ("a", "MKVL")) + message = self.refused(*self.args("chain", fasta, self.folder(), self.folder("codes"), "--output-format", + "parquet")) + self.assertIn("1 record(s) repeat an earlier id (e.g. a)", message) + + def test_a_bundle_that_cannot_be_had(self): + with mock.patch.object(ss, "prepare", side_effect=ss.BundleUnavailableError("not cached")): + for command in _COMMANDS: + with self.subTest(command=command): + message = self.refused(*self.args(command, _FASTA, self.folder(), self.folder("codes"))) + self.assertEqual(message, "not cached") + + def test_only_the_process_that_starts_the_run_fetches_the_bundle(self): + """The command of one process fetches the bundle; the pre-flight of every other rank finds it cached + (bundle_fetch_allowed, as in builds).""" + for command in _COMMANDS: + with self.subTest(command=command): + self.prepare.reset_mock() + self.run_ok(command, _FASTA, self.folder(), "--output-format", "parquet", "--output-codes-folder", + self.folder("codes")) + self.prepare.assert_called_once_with(SYSTEM, local_files_only=False) + for env, num_nodes, local_only in (({}, 1, False), ({}, 2, True), ({"LOCAL_RANK": "1"}, 1, True), + ({"WORLD_SIZE": "2", "RANK": "0"}, 1, True)): + with self.subTest(env=env, num_nodes=num_nodes), launch_env(**env): + self.prepare.reset_mock() + chain_complete_inference.residue_codes_pass(_FASTA, self.folder("codes"), num_nodes=num_nodes) + self.prepare.assert_called_once_with(SYSTEM, local_files_only=local_only) + + def test_a_refusal_is_a_refusal_error(self): + """D20: every check of a run with --output-codes-folder exits 1 with a RefusalError, ValueErrors of the + library included; the same FASTA without --output-codes-folder runs.""" + fasta = self.fasta(("a", "MK>V")) + result = self.invoke(*self.args("residue", fasta, self.folder(), self.folder("codes"))) + self.assertEqual(result.exit_code, 1) + self.assertIsInstance(result.exception, RefusalError) + self.assertIsInstance(result.exception.__cause__, ValueError) + + +# ===================================================================================================================== +# Files of the other kind (Z1-Z5): embeddings in the codes folder, residue codes in the output folder +# ===================================================================================================================== +_CIF = _TEST_PATH / "resources" / "pdb" / "1acb.cif" +_SUFFIX = {"csv": "csv", "pt": "pt", "parquet": "parquet", "json": "json.gz"} + + +class OtherKindTest(_Case): + """A run refuses, before ESM3 and before any file is written (RefusalError, exit status 1), to replace a file of + the other kind: an embedding file in its codes folder, a residue-code file in its output folder, with or without + --output-codes-folder, in every fm-embedding command. Only the files that the run would replace are read; files + of no kind (empty, cut short, being written) are replaced as before; a rerun replaces its own files.""" + + def setUp(self): + super().setUp() + # The models of the structure commands, never built here: every run with them is refused + for module in ("esm_inference", "assembly_complete_inference"): + self.enterContext(mock.patch(f"foldmatch.inference.{module}.get_residue_model", + side_effect=AssertionError("a model was built"))) + self.enterContext(mock.patch("foldmatch.inference.assembly_complete_inference.get_aggregator_model", + side_effect=AssertionError("a model was built"))) + + def args(self, command, fasta, out, codes, *extra): + return ("from-sequences", command, "--fasta-file", fasta, "--output-folder", out, "--output-codes-folder", + codes, *extra) + + def code_files(self, out_format, *extra): + """A codes folder that a run with --output-codes-folder filled (its output folder too), and the ids.""" + _, _, ids, _ = _f2c() + out, codes = self.folder("emb"), self.folder("codes") + self.run_ok("chain", _FASTA, out, "--output-format", out_format, "--output-codes-folder", codes, *extra) + return out, codes, ids + + def contents(self, *folders) -> dict: + return {path: path.read_bytes() for folder in folders for path in sorted(folder.iterdir())} + + def test_an_embedding_folder_as_the_codes_folder(self): + """Friday's accident: --output-codes-folder is the output folder of an earlier run (csv and pt: the code + files would be . there, its embeddings): refused, nothing written, the embeddings intact.""" + _, _, ids, _ = _f2c() + for command in _COMMANDS: + for out_format in ("pt", "csv"): + with self.subTest(command=command, out_format=out_format): + earlier, out = self.folder("earlier"), self.folder("out") + self.run_ok(command, _FASTA, earlier, "--output-format", out_format) + before = self.contents(earlier) + message = self.refused(*self.args(command, _FASTA, out, earlier, "--output-format", out_format), + written=[earlier, out]) + self.assertEqual(self.contents(earlier), before) + files = sorted(earlier / f"{i}.{out_format}" for i in ids) + self.assertEqual(message, ( + f"2 file(s) of the codes folder {earlier} that this run would replace hold embeddings, not " + f"residue codes: {files[0]} (an embedding); {files[1]} (an embedding). The residue codes go " + f"to a folder of their own: give a codes folder for the codes only, without those embedding " + f"files, or move them out of it.")) + self.prepare.assert_not_called() + + def test_a_codes_folder_as_the_output_folder(self): + """The reverse: a plain run (or one with codes) whose --output-folder holds the code files that it would + replace: refused in every format of both commands; with other names (parquet, json) the folder is mixed and + the run allowed.""" + for out_format in _FORMATS: + _, codes, ids = self.code_files(out_format, "--output-codes-name", "inference") + for command in _COMMANDS: + with self.subTest(command=command, out_format=out_format): + before = self.contents(codes) + message = self.refused("from-sequences", command, "--fasta-file", _FASTA, "--output-folder", + codes, "--output-format", out_format, written=[codes]) + self.assertEqual(self.contents(codes), before) + many = out_format in ("csv", "pt") + files = sorted(f"{codes / i}.{out_format}" for i in ids) if many else \ + [f"{codes}/inference-0.{_SUFFIX[out_format]}"] + self.assertEqual(message, ( + f"{len(files)} file(s) of the output folder {codes} that this run would replace hold residue " + f"codes, as 'fm-embedding from-sequences ... --output-codes-folder' writes them, not " + f"embeddings: {'; '.join(f'{f} (residue codes)' for f in files)}. The embeddings and the " + f"residue codes go to folders of their own: give an output folder without those residue-code " + f"files, or move them out of it.")) + other = self.folder("codes") + self.refused(*self.args(command, _FASTA, codes, other, "--output-format", out_format), + written=[codes, other]) + if not many: + self.run_ok(command, _FASTA, codes, "--output-format", out_format, "--output-name", "emb") + + def test_the_parquet_codes_name(self): + """parquet and json: --output-codes-name equal to the earlier embeddings' name in the codes folder is refused; + the default name writes beside them (a mixed folder: nothing is replaced, nothing lost).""" + _, reference, ids, _ = _f2c() + for out_format in ("parquet", "json"): + for command in _COMMANDS: + with self.subTest(command=command, out_format=out_format): + earlier, out = self.folder("earlier"), self.folder("out") + self.run_ok(command, _FASTA, earlier, "--output-format", out_format) + before = self.contents(earlier) + message = self.refused(*self.args(command, _FASTA, out, earlier, "--output-format", out_format, + "--output-codes-name", "inference"), written=[earlier, out]) + self.assertIn(f"1 file(s) of the codes folder {earlier} that this run would replace hold " + f"embeddings, not residue codes: {earlier}/inference-0.{_SUFFIX[out_format]} " + f"(embeddings).", message) + self.run_ok(command, _FASTA, out, "--output-format", out_format, "--output-codes-folder", + earlier) + self.assertEqual(names(earlier), sorted([f"inference-0.{_SUFFIX[out_format]}", + f"inference-codes-0.{_SUFFIX[out_format]}"])) + self.assertEqual({p: d for p, d in self.contents(earlier).items() if p in before}, before) + got = read_codes(earlier, out_format) + for dom_id in ids: + np.testing.assert_array_equal(got[dom_id], reference[dom_id]) + # csv: embedding files of other ids in the codes folder are not looked at + earlier, out = self.folder("earlier"), self.folder("out") + fasta = self.fasta(("other", "MKVLA")) + self.run_ok("chain", fasta, earlier, "--output-format", "csv") + self.run_ok("chain", _FASTA, out, "--output-format", "csv", "--output-codes-folder", earlier) + self.assertEqual(names(earlier), sorted(["other.csv"] + [f"{i}.csv" for i in ids])) + + def test_a_rerun_replaces_its_files(self): + """A rerun into the same folders, every format of both commands: allowed, every file written again.""" + for command in _COMMANDS: + for out_format in _FORMATS: + with self.subTest(command=command, out_format=out_format): + out, codes = self.folder("out"), self.folder("codes") + self.run_ok(command, _FASTA, out, "--output-format", out_format, "--output-codes-folder", codes) + files = sorted([*out.iterdir(), *codes.iterdir()]) + for path in files: + os.utime(path, ns=(0, 0)) + self.run_ok(command, _FASTA, out, "--output-format", out_format, "--output-codes-folder", codes) + self.assertEqual(sorted([*out.iterdir(), *codes.iterdir()]), files) + for path in files: + self.assertGreater(path.stat().st_mtime_ns, 0, path) + self.run_ok(command, _FASTA, out, "--output-format", out_format) + + def test_files_of_no_kind_are_replaced(self): + """Z4: a file cut short (as another rank of the run may be writing it) or of no kind (empty, foreign) is + replaced, never refused, in both folders; an empty code csv (a table without residues) too.""" + _, _, ids, _ = _f2c() + for out_format in _FORMATS: + out, codes, _ = self.code_files(out_format) + whole = self.contents(out, codes) + for cut in (0, 1, 7, 30, 0.5, -1): + with self.subTest(out_format=out_format, cut=cut): + for path, data in whole.items(): + size = int(len(data) * cut) if isinstance(cut, float) else \ + min(cut, len(data)) if cut >= 0 else len(data) + cut + path.write_bytes(data[:size]) + self.run_ok("chain", _FASTA, out, "--output-format", out_format, "--output-codes-folder", codes) + self.assertEqual(sorted(self.contents(out, codes)), sorted(whole)) + for out_format in ("csv", "pt"): + with self.subTest(out_format=out_format, files="foreign"): + out, codes = self.folder("out"), self.folder("codes") + for folder in (out, codes): + (folder / f"{ids[0]}.{out_format}").write_bytes(b"not a FoldMatch file\n") + (folder / f"{ids[1]}.{out_format}").write_bytes(b"") + self.run_ok("chain", _FASTA, out, "--output-format", out_format, "--output-codes-folder", codes) + + def test_another_rank_writes_while_the_check_reads(self): + """Z4: the files that another rank of the same run is writing when this rank's check reads them: of the kind + that the run writes there, or of none yet (truncated, then written in parts), so never refused. The check is + made to read each file while it is being rewritten, at several points of the write.""" + reference = {} + for out_format in _FORMATS: + out, codes, _ = self.code_files(out_format) + reference[out_format] = (out, codes, self.contents(out, codes)) + real = output_files.file_kind + for out_format, (out, codes, whole) in reference.items(): + for share in (0.0, 0.1, 0.5, 0.9, 1.0): + with self.subTest(out_format=out_format, written=share): + seen = [] + + def written_meanwhile(path, _whole=whole, _share=share): + data = _whole[Path(path)] + Path(path).write_bytes(data[:int(len(data) * _share)]) + seen.append(real(path)) + return seen[-1] + + with mock.patch.object(output_files, "file_kind", side_effect=written_meanwhile): + self.run_ok("chain", _FASTA, out, "--output-format", out_format, "--output-codes-folder", + codes) + self.assertTrue(seen) + self.assertTrue(set(seen) <= {None, "embeddings", "residue codes"}) + + def test_parquet_files_of_other_ranks(self): + """parquet and json: the files - of every rank are among the files a run would replace (another + rank's file of the other kind is refused; one being written, with no footer yet, is not).""" + out, codes, _ = self.code_files("parquet") + (codes / "inference-codes-0.parquet").rename(codes / "inference-codes-3.parquet") + shutil.copyfile(out / "inference-0.parquet", codes / "inference-codes-5.parquet") + message = self.refused(*self.args("chain", _FASTA, out, codes, "--output-format", "parquet"), + written=[out, codes]) + self.assertIn(f"1 file(s) of the codes folder {codes} that this run would replace hold embeddings, not residue " + f"codes: {codes}/inference-codes-5.parquet (embeddings).", message) + data = (out / "inference-0.parquet").read_bytes() + (codes / "inference-codes-5.parquet").write_bytes(data[:len(data) - 8]) # no footer yet + self.run_ok("chain", _FASTA, out, "--output-format", "parquet", "--output-codes-folder", codes) + + def test_every_command_refuses_code_files_in_its_output_folder(self): + """The output-folder half in every fm-embedding command that writes embedding files, and in the legacy + 'inference' command: code files under the names of the files it would write (.., + -., ., -.) are refused before any + model; code files under other names are not looked at.""" + from foldmatch.cli import inference as legacy + src = self.folder("src") + shutil.copyfile(_CIF, src / "1acb.cif") + chain_list = self.tmp / "chains.csv" + chain_list.write_text(f"1acb,{_CIF},A,x\n") + commands = { + ("from-structures", "residue"): "1acb.A", ("from-structures", "chain"): "1acb.B", + ("from-structures", "assembly"): "1acb-1", ("from-sequences", "residue"): _f2c()[2][0], + ("from-sequences", "chain"): _f2c()[2][1], ("inference", "residue-embedding"): "x", + } + for out_format in _FORMATS: + _, codes, ids = self.code_files(out_format, "--output-codes-name", "inference") + one = (codes / f"{ids[0]}.{out_format}") if out_format in ("csv", "pt") else \ + codes / f"inference-0.{_SUFFIX[out_format]}" + for (group, command), name in commands.items(): + with self.subTest(command=f"{group} {command}", out_format=out_format): + out = self.folder("out") + target = out / (f"{name}.{out_format}" if out_format in ("csv", "pt") else one.name) + shutil.copyfile(one, target) + if group == "from-structures": + args = (group, command, "--src-folder", src) + elif group == "from-sequences": + args = (group, command, "--fasta-file", _FASTA) + else: + args = (command, "--src-file", chain_list) + args += ("--output-folder", out, "--output-format", out_format) + before = self.contents(out) + self.refusing = True + try: + result = CliRunner().invoke(legacy.app if group == "inference" else embedding.app, + [str(a) for a in (*args, *_CPU)]) + finally: + self.refusing = False + self.assertEqual(result.exit_code, 1, f"{result.output}\n{result.exception!r}") + self.assertIsInstance(result.exception, RefusalError, repr(result.exception)) + self.assertTrue(str(result.exception).startswith( + f"1 file(s) of the output folder {out} that this run would replace hold residue codes"), + str(result.exception)) + self.assertIn(f"{target} (residue codes)", str(result.exception)) + self.assertEqual(self.contents(out), before) + + def test_the_library(self): + """Both predict functions refuse either kind, with the codes or without; the other library entry points that + write embeddings refuse code files in their output folder.""" + from foldmatch.inference import assembly_complete_inference, esm_inference + from foldmatch.types.api_types import SrcLocation, SrcProteinFrom + for out_format in (OutFormat.csv, OutFormat.pt, OutFormat.parquet, OutFormat.json): + fmt = out_format.value + earlier, codes, ids = self.code_files(fmt, "--output-codes-name", "inference") + self.refusing = True + files = sorted(earlier / f"{i}.{fmt}" for i in ids) if fmt in ("csv", "pt") else \ + [earlier / f"inference-0.{_SUFFIX[fmt]}"] + for predict, kwargs in ((sequence_inference.predict, {"fasta_file": _FASTA}), + (chain_complete_inference.predict, {"src_stream": _FASTA, + "src_from": SrcEsmFrom.fasta})): + with self.subTest(predict=predict.__module__, out_format=fmt): + out = self.folder("out") + before = self.contents(earlier, codes) + # Embeddings in the codes folder (the codes named as the embeddings in parquet and json) + with self.assertRaises(RefusalError) as refused: + predict(out_folder=out, out_codes_folder=earlier, out_format=out_format, + out_codes_name="inference", **kwargs) + self.assertTrue(str(refused.exception).startswith( + f"{len(files)} file(s) of the codes folder {earlier} that this run would replace hold " + f"embeddings, not residue codes: {files[0]} ("), str(refused.exception)) + # Residue codes in the output folder, with and without the codes + for extra in ({}, {"out_codes_folder": self.folder("codes")}): + with self.assertRaises(RefusalError) as refused: + predict(out_folder=codes, out_format=out_format, **extra, **kwargs) + self.assertTrue(str(refused.exception).startswith( + f"{len(files)} file(s) of the output folder {codes} that this run would replace hold " + f"residue codes"), str(refused.exception)) + self.assertEqual(self.contents(earlier, codes), before) + self.assertEqual(names(out), []) + with self.subTest(predict="esm_inference and assembly_complete_inference", out_format=fmt): + out = self.folder("out") + for name in ("1acb.A", "1acb-1"): + if fmt in ("csv", "pt"): + shutil.copyfile(codes / f"{ids[0]}.{fmt}", out / f"{name}.{fmt}") + else: + shutil.copyfile(codes / f"inference-0.{_SUFFIX[fmt]}", out / f"inference-0.{_SUFFIX[fmt]}") + stream = [("1acb", str(_CIF), "1acb")] + with self.assertRaisesRegex(RefusalError, "hold residue codes"): + esm_inference.predict(src_stream=stream, src_location=SrcLocation.stream, + src_from=SrcProteinFrom.structure, out_folder=out, out_format=out_format) + with self.assertRaisesRegex(RefusalError, "hold residue codes"): + assembly_complete_inference.predict(src_stream=stream, src_location=SrcLocation.stream, + out_folder=out, out_format=out_format) + + + def test_the_library_reads_the_files_of_the_records_kept(self): + """csv and pt: the library's checks read the files of the records that the run keeps (min_res_n), the files + it would replace: another kind under the name of a record left out is not looked at (allowed, untouched); + under the name of a record kept it is refused.""" + from foldmatch.utils import file_kinds + from foldmatch.utils.sequence import scan_fasta + lengths = {name: length for _, name, length in scan_fasta(_FASTA)} + kept, left = max(lengths, key=lengths.get), min(lengths, key=lengths.get) + min_res_n = lengths[left] + 1 + for fmt in ("csv", "pt"): + out_format = OutFormat(fmt) + earlier, codes, _ = self.code_files(fmt) + for predict, kwargs in ((sequence_inference.predict, {"fasta_file": _FASTA}), + (chain_complete_inference.predict, {"src_stream": _FASTA, + "src_from": SrcEsmFrom.fasta})): + with self.subTest(predict=predict.__module__, out_format=fmt): + out, out_codes = self.folder("out"), self.folder("codes") + shutil.copyfile(codes / f"{left}.{fmt}", out / f"{left}.{fmt}") + shutil.copyfile(earlier / f"{left}.{fmt}", out_codes / f"{left}.{fmt}") + before = self.contents(out, out_codes) + self.refusing = False + predict(out_folder=out, out_codes_folder=out_codes, out_format=out_format, min_res_n=min_res_n, + accelerator=Accelerator.cpu, devices=1, **kwargs) + self.assertEqual(names(out), sorted([f"{kept}.{fmt}", f"{left}.{fmt}"])) + self.assertEqual({p: d for p, d in self.contents(out, out_codes).items() if p in before}, before) + self.assertEqual(file_kinds.file_kind(out / f"{kept}.{fmt}"), file_kinds.EMBEDDINGS) + for folder, source, kind in ((out, codes, "residue codes"), (out_codes, earlier, "embeddings")): + shutil.copyfile(source / f"{kept}.{fmt}", folder / f"{kept}.{fmt}") + self.refusing = True + with self.assertRaisesRegex(RefusalError, f"^1 file\\(s\\) .* hold {kind}"): + predict(out_folder=out, out_codes_folder=out_codes, out_format=out_format, + min_res_n=min_res_n, **kwargs) + os.remove(folder / f"{kept}.{fmt}") + + @unittest.skipUnless(_CASE_INSENSITIVE, "a case-insensitive file system (macOS APFS)") + def test_case_variants_of_the_files(self): + """On a case-insensitive volume, the files that the folder takes for the run's (another case of the id, of + the suffix) are the files it would replace: of the other kind, refused, named as they are on disk.""" + _, _, ids, _ = _f2c() + for out_format in ("csv", "pt"): + out, codes, _ = self.code_files(out_format) + upper = out_format.upper() + with self.subTest(out_format=out_format, folder="codes"): + earlier = self.folder("earlier") + for dom_id in ids: + shutil.copyfile(out / f"{dom_id}.{out_format}", earlier / f"{dom_id.upper()}.{upper}") + before = self.contents(earlier) + message = self.refused(*self.args("chain", _FASTA, self.folder("out"), earlier, "--output-format", + out_format), written=[earlier]) + files = sorted(f"{earlier / dom_id.upper()}.{upper}" for dom_id in ids) + self.assertIn(f"2 file(s) of the codes folder {earlier} that this run would replace hold embeddings, " + f"not residue codes: {files[0]} (an embedding); {files[1]} (an embedding).", message) + self.assertEqual(self.contents(earlier), before) + with self.subTest(out_format=out_format, folder="output"): + folder = self.folder("codes") + shutil.copyfile(codes / f"{ids[0]}.{out_format}", folder / f"{ids[0].swapcase()}.{upper}") + message = self.refused("from-sequences", "chain", "--fasta-file", _FASTA, "--output-folder", folder, + "--output-format", out_format, written=[folder]) + self.assertIn(f"1 file(s) of the output folder {folder} that this run would replace hold residue " + f"codes", message) + self.assertIn(f"{folder / ids[0].swapcase()}.{upper} (residue codes)", message) + + def test_each_file_is_read_once_per_run(self): + """csv and pt: the command checks both folders before it calls the library, which checks them again; each + file is read once in all (file_kinds.remembering), and nothing is remembered once the library has checked.""" + from foldmatch.utils import file_kinds + for out_format in ("csv", "pt"): + out, codes, ids = self.code_files(out_format) + for command in _COMMANDS: + with self.subTest(command=command, out_format=out_format): + file_kinds._KNOWN.clear() + reader = mock.Mock(wraps=file_kinds._KINDS[f".{out_format}"]) + with mock.patch.dict(file_kinds._KINDS, {f".{out_format}": reader}): + self.run_ok(command, _FASTA, out, "--output-format", out_format, "--output-codes-folder", + codes) + read = sorted(Path(c.args[0]).relative_to(self.tmp) for c in reader.call_args_list) + self.assertEqual(read, sorted(Path(folder).relative_to(self.tmp) / f"{i}.{out_format}" + for folder in (out, codes) for i in ids)) + self.assertEqual(file_kinds._KNOWN, {}) + +# ===================================================================================================================== +# The library +# ===================================================================================================================== +class LibraryTest(_Case): + + def test_predictions_with_codes(self): + """Y2: with out_codes_folder each prediction is (embeddings, ids, codes, seq_sha1s), the codes one CPU uint8 + tensor [L, 8] per id, the object of its .pt code file and the bytes of its row in the codes Parquet file; the + embeddings are those of a plain run, whose predictions stay (embeddings, ids), without any conversion.""" + _, reference, ids, sequences = _f2c() + converted = mock.patch.object(chain_complete_inference, "code_predictions", + side_effect=chain_complete_inference.code_predictions) + for predict, kwargs in ((sequence_inference.predict, {"fasta_file": _FASTA}), + (chain_complete_inference.predict, {"src_stream": _FASTA, + "src_from": SrcEsmFrom.fasta})): + with converted as conversion: + plain = predict(accelerator=Accelerator.cpu, devices=1, batch_size=2, **kwargs) + conversion.assert_not_called() + self.assertEqual(len(plain), 1) + self.assertEqual(len(plain[0]), 2) + self.assertEqual(plain[0][1], tuple(ids)) + for out_format in (None, OutFormat.pt, OutFormat.parquet, OutFormat.json): + with self.subTest(predict=predict.__module__, out_format=out_format): + codes = self.folder("codes") + # Without out_folder (no embedding files) and the default format, csv; else with both folders + given = {} if out_format is None else {"out_folder": self.folder(), "out_format": out_format} + with converted as conversion: + (embeddings, names_, tables, digests), = predict( + accelerator=Accelerator.cpu, devices=1, batch_size=2, out_codes_folder=codes, **given, + **kwargs) + conversion.assert_called_once() + self.assertEqual(names_, tuple(ids)) + self.assertEqual(digests, tuple(_sha1(s) for s in sequences)) + self.assertIsInstance(tables, tuple) + for k, dom_id in enumerate(ids): + table = tables[k] + self.assertIsInstance(table, torch.Tensor) + self.assertEqual((table.dtype, table.device.type, tuple(table.shape)), + (torch.uint8, "cpu", (len(sequences[k]), 8))) + self.assertTrue(table.is_contiguous()) + np.testing.assert_array_equal(table.numpy(), reference[dom_id]) + self.assertTrue(torch.equal(plain[0][0][k], embeddings[k])) + if out_format in (None, OutFormat.json): + fmt = "csv" if out_format is None else "json" + if out_format is None: + self.assertEqual(names(codes), sorted(f"{i}.csv" for i in ids)) + stored = read_codes(codes, fmt, ids=ids) + self.assertEqual(sorted(stored), sorted(ids)) + for dom_id, table in zip(ids, tables): + np.testing.assert_array_equal(stored[dom_id], table.numpy()) + elif out_format == OutFormat.pt: + for dom_id, table in zip(ids, tables): + stored = torch.load(codes / f"{dom_id}.pt", weights_only=True) + self.assertEqual((stored.dtype, stored.shape), (table.dtype, table.shape)) + self.assertTrue(torch.equal(stored, table)) + else: + parquet = pq.read_table(codes / "inference-codes-0.parquet") + self.assertEqual(parquet.column("id").to_pylist(), list(ids)) + column = parquet.column("residue_codes").combine_chunks() + for k, table in enumerate(tables): + stored = column[k].values.flatten().to_numpy(zero_copy_only=False).tobytes() + self.assertEqual(table.numpy().tobytes(), stored) + self.assertEqual(parquet.column("seq_sha1").to_pylist(), list(digests)) + + def test_the_shard_pass_returns_its_payloads_as_bytes(self): + """Y2 leaves the shard pass of a build (residue_output) as it is: its predictions hold the raw payloads that + its shards store, bytes, not converted.""" + _, reference, ids, _ = _f2c() + out = self.folder() + with mock.patch.object(chain_complete_inference, "code_predictions") as conversion: + (_, names_, payloads, _), = chain_complete_inference.predict( + src_stream=_FASTA, src_from=SrcEsmFrom.fasta, accelerator=Accelerator.cpu, devices=1, batch_size=2, + out_folder=out, out_format=OutFormat.parquet, residue_output=rs.RESIDUE_CODES) + conversion.assert_not_called() + self.assertEqual(names_, tuple(ids)) + stored = pq.read_table(out / "inference-0.parquet").column(rs.RESIDUE_CODES).to_pylist() + for k, dom_id in enumerate(ids): + self.assertIsInstance(payloads[k], bytes) + self.assertEqual(payloads[k], reference[dom_id].tobytes()) + self.assertEqual(payloads[k], stored[k]) + + def test_library_refusals_before_esm3(self): + self.refusing = True + with tempfile.TemporaryDirectory() as tmp, tempfile.TemporaryDirectory() as codes: + with self.assertRaisesRegex(ValueError, "out_codes_name and scoring_system are only used with " + "out_codes_folder, which is not given"): + sequence_inference.predict(fasta_file=_FASTA, out_folder=tmp, out_codes_name="c") + with self.assertRaisesRegex(ValueError, "out_codes_name names the residue-code files of out_codes_folder, " + "which is not given"): + chain_complete_inference.predict(src_stream=_FASTA, src_from=SrcEsmFrom.fasta, out_folder=tmp, + out_codes_name="c") + with self.assertRaisesRegex(ValueError, "pass one or the other"): + chain_complete_inference.predict(src_stream=_FASTA, src_from=SrcEsmFrom.fasta, out_folder=tmp, + out_format=OutFormat.parquet, out_codes_folder=codes, + residue_output=rs.RESIDUE_CODES) + with self.assertRaisesRegex(ValueError, "residue codes needs sequence \\(FASTA\\) input"): + chain_complete_inference.predict(src_stream=[("1acb", str(_TEST_PATH / "resources/pdb/1acb.cif"), + "1acb")], src_location="stream", out_codes_folder=codes) + for predict, kwargs in ((sequence_inference.predict, {"fasta_file": _FASTA}), + (chain_complete_inference.predict, {"src_stream": _FASTA, + "src_from": SrcEsmFrom.fasta})): + with self.subTest(predict=predict.__module__): + # W1: a ValueError, as the library's other argument errors (the CLI's is a usage error) + with self.assertRaises(ValueError) as refused: + predict(out_folder=tmp, out_codes_folder=os.path.join(tmp, "."), **kwargs) + self.assertNotIsInstance(refused.exception, RefusalError) + self.assertTrue(str(refused.exception).startswith( + f"out_folder and out_codes_folder are one folder ({os.path.join(tmp, '.')}): the residue " + f"codes go to a folder of their own"), str(refused.exception)) + # Compared as files: a symbolic link to the output folder is that folder (the library has no + # typer resolve_path), on any file system + link = os.path.join(codes, "link") + if not os.path.islink(link): + os.symlink(tmp, link, target_is_directory=True) + with self.assertRaises(ValueError) as refused: + predict(out_folder=tmp, out_codes_folder=link, **kwargs) + self.assertNotIsInstance(refused.exception, RefusalError) + self.assertTrue(str(refused.exception).startswith( + f"out_folder and out_codes_folder are one folder ({link}):"), str(refused.exception)) + os.remove(link) + missing = os.path.join(codes, "missing") + with self.assertRaises(RefusalError) as refused: + predict(out_folder=tmp, out_codes_folder=missing, **kwargs) + self.assertEqual(str(refused.exception), + f"out_codes_folder {missing} is not a folder: give an existing folder.") + with self.assertRaises(RefusalError) as refused: + predict(out_folder=tmp, out_codes_folder=_FASTA, **kwargs) + self.assertEqual(str(refused.exception), + f"out_codes_folder {_FASTA} is not a folder: give an existing folder.") + with self.assertRaisesRegex(RefusalError, "The codes name 'a/b' holds '/'"): + predict(out_folder=tmp, out_codes_folder=codes, out_codes_name="a/b", **kwargs) + # scoring_system: refused without the codes (or a residue_output), and an unknown name refused before + # ESM3 by both functions + with self.assertRaisesRegex(ValueError, "out_codes_name and scoring_system are only used with " + "out_codes_folder, which is not given"): + sequence_inference.predict(fasta_file=_FASTA, out_folder=tmp, scoring_system="esm-codes-v1") + with self.assertRaisesRegex(ValueError, "^scoring_system is only used with residue_output or " + "out_codes_folder, and neither is given$"): + chain_complete_inference.predict(src_stream=_FASTA, src_from=SrcEsmFrom.fasta, out_folder=tmp, + scoring_system="esm-codes-v1") + with self.assertRaises(ss.UnknownScoringSystemError): + sequence_inference.predict(fasta_file=_FASTA, out_folder=tmp, out_codes_folder=codes, + scoring_system="esm-codes-v0") + with self.assertRaises(ss.UnknownScoringSystemError): + chain_complete_inference.predict(src_stream=_FASTA, src_from=SrcEsmFrom.fasta, out_folder=tmp, + out_codes_folder=codes, scoring_system="esm-codes-v0") + self.assertEqual(os.listdir(tmp), []) + self.assertEqual(os.listdir(codes), []) + + def test_missing_codes_raise_at_the_end(self): + """W4: predict raises MissingResidueCodesError once the files are written, with the count, the ids, the rank + and the predictions (None without return_predictions).""" + _, reference, ids, sequences = _f2c() + + def one_row_short(model, protein): + rows = _esm3(model, protein) + return rows[:-1] if protein.sequence == sequences[1] else rows + + for predict, module, kwargs in ((sequence_inference.predict, esm_module, {"fasta_file": _FASTA}), + (chain_complete_inference.predict, chain_complete_module, + {"src_stream": _FASTA, "src_from": SrcEsmFrom.fasta})): + for return_predictions, out_format in ((True, "pt"), (False, "csv")): + with self.subTest(predict=predict.__module__, return_predictions=return_predictions), \ + mock.patch.object(module, "per_residue_embedding", side_effect=one_row_short), \ + self.assertLogs("foldmatch.modules.residue_payloads", "ERROR"): + codes = self.folder("codes") + with self.assertRaises(MissingResidueCodesError) as raised: + predict(accelerator=Accelerator.cpu, devices=1, batch_size=2, out_codes_folder=codes, + out_format=OutFormat(out_format), return_predictions=return_predictions, **kwargs) + error = raised.exception + self.assertIsInstance(error, RefusalError) + self.assertEqual((error.count, error.ids, error.rank, error.world_size), (1, (ids[1],), 0, 1)) + self.assertIn(f"({codes}/.{out_format} of each, " + + ("an empty file)" if out_format == "csv" else "a uint8 tensor [0, 8])"), str(error)) + self.assertEqual(names(codes), sorted(f"{i}.{out_format}" for i in ids)) + if return_predictions: + # Y2: the predictions of the call, the codes as the tensors of the .pt files, [0, 8] for + # the sequence without codes + (_, names_, tables, _), = error.predictions + self.assertEqual(names_, tuple(ids)) + for dom_id, table in zip(ids, tables): + self.assertEqual(table.dtype, torch.uint8) + self.assertTrue(torch.equal(table, torch.load(codes / f"{dom_id}.pt", weights_only=True))) + self.assertEqual(tuple(tables[1].shape), (0, 8)) + np.testing.assert_array_equal(tables[0].numpy(), reference[ids[0]]) + else: + self.assertEqual((codes / f"{ids[1]}.csv").read_bytes(), b"") + self.assertIsNone(error.predictions) + + def test_the_ranks_report_to_rank_0(self): + """W4 with several ranks: each rank posts its count and first ids in the process group's store and waits for + every rank's; rank 0 raises for the run (the count, the ranks, the first ids by rank, the files of those + ranks), the other ranks log theirs and return; a rank whose pass raised is reported by rank 0 as a + RefusalError; a rank that does not report is waited for at most RANK_WAIT.""" + codes = chain_complete_inference.ResidueCodesPass(None, None, None, Path("/c"), "inference-codes", 0) + key = chain_complete_inference._report_key + + def trainer(rank, world_size=2): + return SimpleNamespace(global_rank=rank, world_size=world_size) + + def module(*ids): + return SimpleNamespace(residue_failures=len(ids), residue_failed_ids=list(ids)) + + def check(rank, own, others, out_format="parquet", world_size=2, read=True): + """check_after_pass on ``rank`` with the other ranks' reports already posted (and read).""" + store = torch.distributed.HashStore() + pass_id = 7 + for r, report in others.items(): + store.set(key(pass_id, r), json.dumps(report)) + if read: + store.add(key(pass_id, "read"), world_size - 1) + with mock.patch.object(chain_complete_inference, "_default_store", return_value=store): + result = codes.check_after_pass(trainer(rank, world_size), own, out_format, pass_id=pass_id) + return result, store + + self.assertEqual(check(0, module(), {1: {"count": 0, "ids": []}})[0], None) + with self.assertRaises(MissingResidueCodesError) as raised: + check(0, module(), {1: {"count": 2, "ids": ["c", "d"]}}) + error = raised.exception + self.assertEqual((error.count, error.ids, error.rank_counts, error.rank, error.world_size), + (2, ("c", "d"), (0, 2), 0, 2)) + self.assertEqual(str(error), "2 sequence(s) on rank 1 of 2 got no residue codes: 'c', 'd'. ESM3 gave rows that " + "do not match the sequence's length, or the encoding failed (each one's reason " + "is logged above, by its rank). Their codes are written as empty [0, 8] tables " + "(rows of /c/inference-codes-1.parquet without residues); their embeddings and " + "every other file of the run are written. Check those sequences, or leave them " + "out of the FASTA file.") + with self.assertRaises(MissingResidueCodesError) as raised: + check(0, module("a", "b"), {1: {"count": 1, "ids": ["c"]}, 2: {"count": 0, "ids": []}}, "json", 3) + self.assertEqual((raised.exception.count, raised.exception.ids, raised.exception.rank_counts), + (3, ("a", "b", "c"), (2, 1, 0))) + self.assertTrue(str(raised.exception).startswith( + "3 sequence(s) on 2 ranks of 3 (rank 0: 2, rank 1: 1) got no residue codes: 'a', 'b', 'c'. ")) + self.assertIn("(records of /c/inference-codes-0.json.gz and /c/inference-codes-1.json.gz without residues)", + str(raised.exception)) + with self.assertRaises(MissingResidueCodesError) as raised: + check(0, module(*"abc"), {1: {"count": 9, "ids": list("defghijkl")}}) + self.assertEqual(raised.exception.ids, tuple("abcdefghij")) + self.assertIn("got no residue codes: 'a', 'b', 'c', 'd', 'e' and 7 more.", str(raised.exception)) + # The other ranks log their count and return: rank 0 raises for them + with self.assertLogs(chain_complete_inference.__name__, "ERROR") as logs: + result, store = check(1, module("c"), {0: {"count": 0, "ids": []}}) + self.assertIsNone(result) + self.assertEqual(logs.output, [f"ERROR:{chain_complete_inference.__name__}:1 sequence(s) on rank 1 of 2 got " + f"no residue codes; rank 0 ends the run with an error that names them"]) + self.assertEqual(json.loads(store.get(key(7, 1))), {"count": 1, "ids": ["c"]}) + self.assertTrue(store.check([key(7, "all-read")])) + # A rank whose pass raised: rank 0 refuses the run, after its own missing codes or alone + with self.assertRaises(RefusalError) as raised: + check(0, module(), {1: {"error": "RuntimeError"}}) + self.assertNotIsInstance(raised.exception, MissingResidueCodesError) + self.assertEqual(str(raised.exception), "Rank(s) 1 of 2 ended the pass with an error, which they report " + "themselves; their files may be incomplete. The files of the other " + "ranks are written.") + with self.assertRaises(MissingResidueCodesError) as raised: + check(0, module("a"), {1: {"error": "RuntimeError"}}) + self.assertTrue(str(raised.exception).endswith(" Also, rank(s) 1 of 2 ended the pass with an error, which " + "they report themselves; their files may be incomplete.")) + self.assertIsNone(check(1, module(), {0: {"error": "RuntimeError"}})[0]) + # A rank that does not report + with mock.patch.object(chain_complete_inference, "RANK_WAIT", timedelta(milliseconds=200)), \ + self.assertRaises(RefusalError) as raised: + check(0, module(), {}) + self.assertTrue(str(raised.exception).startswith( + "Rank 0 of 2 ended its pass, with its files written, but could not learn whether the other ranks' " + "sequences got residue codes: rank(s) 1 did not report the end of its pass (a rank waits for that at " + "most 200 ms); "), str(raised.exception)) + self.assertEqual(chain_complete_inference._duration(chain_complete_inference.RANK_WAIT), "1 day") + self.assertEqual(chain_complete_inference._duration(chain_complete_inference.READ_WAIT), "10 minutes") + # The store gone (rank 0's process held it and rank 0 ended): the other ranks say so, with the store's error + gone = mock.Mock(**{f"{op}.side_effect": RuntimeError("Connection reset by peer") + for op in ("set", "wait", "get", "add", "check")}) + with mock.patch.object(chain_complete_inference, "_default_store", return_value=gone), \ + self.assertRaises(RefusalError) as raised: + codes.check_after_pass(trainer(1), module(), "parquet", pass_id=7) + self.assertEqual(str(raised.exception), "Rank 1 of 2 ended its pass, with its files written, but could not " + "learn whether the other ranks' sequences got residue codes: the " + "process group's store could not be reached (rank 0's process holds " + "it under Lightning's launcher and srun: rank 0 has most likely " + "ended, with an error of its own); RuntimeError: Connection reset by " + "peer.") + # Rank 0 waits for the others to read the reports (its process may hold the store), at most READ_WAIT + with mock.patch.object(chain_complete_inference, "READ_WAIT", timedelta(milliseconds=200)), \ + self.assertLogs(chain_complete_inference.__name__, "WARNING") as logs: + self.assertIsNone(check(0, module(), {1: {"count": 0, "ids": []}}, read=False)[0]) + self.assertIn("rank 0 of 2: not every rank read the residue-code reports within 200 ms", logs.output[0]) + # A rank whose pass raised never reads the reports: its post counts as read, so rank 0 does not wait + # READ_WAIT for it (NERSC job 61254854: rank 0 sat out the 10 minutes while NCCL held rank 1's exit) + store = torch.distributed.HashStore() + with mock.patch.object(chain_complete_inference, "_default_store", return_value=store), \ + mock.patch.object(chain_complete_inference, "READ_WAIT", timedelta(seconds=30)), \ + self.assertNoLogs(chain_complete_inference.__name__, "WARNING"): + chain_complete_inference._post_report(trainer(1), 7, {"error": "RuntimeError"}) + started = time.monotonic() + with self.assertRaises(RefusalError) as raised: + codes.check_after_pass(trainer(0), module(), "parquet", pass_id=7) + self.assertLess(time.monotonic() - started, 5) + self.assertTrue(str(raised.exception).startswith("Rank(s) 1 of 2 ended the pass with an error")) + self.assertTrue(store.check([key(7, "all-read")])) + # Rank 0's pass raised: the other ranks read its report and end; the last reader sets all-read + store = torch.distributed.HashStore() + with mock.patch.object(chain_complete_inference, "_default_store", return_value=store): + chain_complete_inference._post_report(trainer(0), 8, {"error": "RuntimeError"}) + self.assertFalse(store.check([key(8, "all-read")])) + self.assertIsNone(codes.check_after_pass(trainer(1), module(), "parquet", pass_id=8)) + self.assertTrue(store.check([key(8, "all-read")])) + # Several ranks without a process group (a spawn or fork strategy's parent): a WARNING, no error + with mock.patch.object(chain_complete_inference, "_default_store", return_value=None), \ + self.assertLogs(chain_complete_inference.__name__, "WARNING") as logs: + self.assertIsNone(codes.check_after_pass(trainer(0), module(), "parquet", pass_id=9)) + self.assertEqual(logs.output, [f"WARNING:{chain_complete_inference.__name__}:This process started its 2 ranks " + f"with a spawn or fork strategy and does not learn their counts of sequences " + f"without residue codes: it raises no MissingResidueCodesError for them. Each " + f"rank has logged its own as ERROR lines; their codes are empty [0, 8] " + f"tables."]) + + def test_a_pass_that_raises_tells_the_other_ranks(self): + """ResidueCodesPass.run: a rank whose trainer.predict raises posts {"error"} for its pass, then re-raises; + one that ends posts its count and returns its predictions with the codes as uint8 tensors [L, 8]. The pass + ids advance alike on every rank (one per pass).""" + codes = chain_complete_inference.ResidueCodesPass(None, SimpleNamespace(system=SYSTEM), None, Path("/c"), + "inference-codes", 0) + store = torch.distributed.HashStore() + store.set(chain_complete_inference._report_key(100, 0), json.dumps({"count": 0, "ids": []})) + store.add(chain_complete_inference._report_key(100, "read"), 1) + + def fails(*_, **__): + raise RuntimeError("out of memory") + + with mock.patch.object(chain_complete_inference, "_default_store", return_value=store), \ + mock.patch.object(chain_complete_inference, "_PASSES", iter([99, 100])): + with self.assertRaisesRegex(RuntimeError, "out of memory"): + codes.run(SimpleNamespace(global_rank=1, world_size=2, predict=fails), None, None, "csv", False) + self.assertEqual(json.loads(store.get(chain_complete_inference._report_key(99, 1))), + {"error": "RuntimeError"}) + # ... and counts as having read the reports (it never will), so that rank 0 does not wait for it + self.assertEqual(store.add(chain_complete_inference._report_key(99, "read"), 0), 1) + embeddings = torch.zeros(2, 4) + done = SimpleNamespace(global_rank=1, world_size=2, predict=lambda *_, **__: [ + (embeddings, ("a", "b"), (bytes(range(16)), b""), (bytes(20), bytes(20)))]) + (got,) = codes.run(done, SimpleNamespace(residue_failures=0, residue_failed_ids=[]), None, "csv", True) + self.assertIs(got[0], embeddings) + self.assertEqual(got[1], ("a", "b")) + self.assertTrue(torch.equal(got[2][0], torch.arange(16, dtype=torch.uint8).reshape(2, 8))) + self.assertEqual((got[2][1].dtype, tuple(got[2][1].shape)), (torch.uint8, (0, 8))) + self.assertEqual(got[3], (bytes(20), bytes(20))) + self.assertEqual(json.loads(store.get(chain_complete_inference._report_key(100, 1))), + {"count": 0, "ids": []}) + + def test_a_folder_named_as_a_file_of_the_other_in_the_library(self): + """A codes folder inside the output folder under the name of an embedding file (a library caller: the files + -.parquet|json.gz), or the reverse: refused before ESM3, nothing written.""" + self.refusing = True + for predict, kwargs in ((sequence_inference.predict, {"fasta_file": _FASTA}), + (chain_complete_inference.predict, {"src_stream": _FASTA, + "src_from": SrcEsmFrom.fasta})): + for out_format, name in ((OutFormat.parquet, "inference-0.parquet"), + (OutFormat.json, "inference-2.json.gz")): + with self.subTest(predict=predict.__module__, out_format=out_format): + out = self.folder() + (out / name).mkdir() + with self.assertRaises(RefusalError) as refused: + predict(out_folder=out, out_codes_folder=out / name, out_format=out_format, **kwargs) + self.assertTrue(str(refused.exception).startswith( + f"The codes folder {out / name} is named as one of the embedding files of the output folder " + f"{out} (a file -"), str(refused.exception)) + self.assertEqual(names(out), [name]) + self.assertEqual(names(out / name), []) + codes = self.folder("codes") + inner = codes / name.replace("inference-", "inference-codes-") + inner.mkdir() + with self.assertRaises(RefusalError) as refused: + predict(out_folder=inner, out_codes_folder=codes, out_format=out_format, **kwargs) + self.assertTrue(str(refused.exception).startswith( + f"The output folder {inner} is named as one of the code files of the codes folder {codes} " + f"(a file -"), str(refused.exception)) + self.assertEqual(names(inner), []) + + def test_refuse_same_folder(self): + """Folders compared as files (device and inode), not as path strings: a link and '..' are the folder, on any + file system; a missing folder is no folder; the message names the options given.""" + with tempfile.TemporaryDirectory() as tmp, tempfile.TemporaryDirectory() as other: + link = os.path.join(other, "link") + os.symlink(tmp, link, target_is_directory=True) + os.mkdir(os.path.join(tmp, "sub")) + for given in (link, os.path.join(tmp, "sub", ".."), os.path.join(link, ".")): + with self.subTest(given=given), self.assertRaisesRegex(ValueError, r"^a and b are one folder"): + output_files.refuse_same_folder(tmp, given, ("a", "b")) + self.assertTrue(output_files.same_folder_problem(tmp, given, ("a", "b")).startswith( + f"a and b are one folder ({given}): ")) + os.mkdir(os.path.join(tmp, "codes.parquet")) # W2: fm-search skips such a subfolder + for given in (other, os.path.join(tmp, "sub"), os.path.join(tmp, "missing"), + os.path.join(tmp, "codes.parquet")): + with self.subTest(given=given): + output_files.refuse_same_folder(tmp, given) + self.assertIsNone(output_files.same_folder_problem(tmp, given)) + output_files.refuse_same_folder(os.path.join(tmp, "missing"), os.path.join(tmp, "missing")) + + +# ===================================================================================================================== +# Modules and writers +# ===================================================================================================================== +class ModuleTest(unittest.TestCase): + + def test_esm_module_with_codes(self): + class Head(torch.nn.Module): + calls = [] + + def encode_target(self, rows): + self.calls.append(rows) + return torch.zeros(rows.shape[0], 8, dtype=torch.uint8) + + head = Head() + batch = [(SimpleNamespace(sequence="MKV", coordinates=None), "a"), + (SimpleNamespace(sequence="MK", coordinates=None), "b")] + rows = {"MKV": torch.randn(5, 4), "MK": torch.randn(4, 4)} + with mock.patch.object(esm_module, "per_residue_embedding", lambda m, p: rows[p.sequence]): + plain = EsmModule(None).predict_step(batch, 0) + module = EsmModule(None, residue_head=head, residue_output=rs.RESIDUE_CODES, code_files=True) + out = module.predict_step(batch, 0) + self.assertEqual(len(plain), 2) + self.assertEqual(out[1], ("a", "b")) + self.assertTrue(all(x is y for x, y in zip(out[0], plain[0]))) + self.assertEqual(out[2], (bytes(24), bytes(16))) + self.assertEqual(out[3], (_sha1("MKV"), _sha1("MK"))) + self.assertTrue(torch.equal(head.calls[0], rows["MKV"][1:-1])) + self.assertIs(dict(module.named_children())["residue_head"], head) + + def test_failures_past_the_logged_ones(self): + """After the first ten failures by name, one line, about residue codes for code files.""" + for code_files, line in ((True, "more proteins without residue codes; only counting them from now on"), + (False, "more proteins without a residue_codes payload; only counting them from now " + "on")): + with self.subTest(code_files=code_files): + payloads = ResiduePayloads() + payloads._init_residue_payloads(torch.nn.Identity(), rs.RESIDUE_CODES, code_files=code_files) + with self.assertLogs("foldmatch.modules.residue_payloads", "ERROR") as logs: + for k in range(12): + self.assertEqual(payloads._failed(f"p{k}", "a reason"), b"") + messages = [record.getMessage() for record in logs.records] + self.assertEqual(len(messages), 11) + self.assertEqual(messages[-1], line) + self.assertEqual(payloads.residue_failures, 12) + + def test_constructor_refusals(self): + with self.assertRaises(ValueError): + EsmModule(None, residue_head=torch.nn.Identity()) + with self.assertRaises(ValueError): + EsmModule(None, residue_output=rs.RESIDUE_CODES) + with self.assertRaises(ValueError): + EsmModule(None, code_files=True) + with self.assertRaises(ValueError): + ChainCompleteModule(None, None, residue_head=torch.nn.Identity(), residue_output=rs.QUERY_ENCODING, + code_files=True) + + +# ===================================================================================================================== +# 'build residue-codes --from-shards' refuses a code file +# ===================================================================================================================== +class BuildFromShardsTest(_Case): + """A code file among a run's shards is refused by 'build residue-codes --from-shards' (the readers of + tests/test_residue_code_files.py).""" + + def codes_parquet(self, command="chain"): + codes = self.folder("codes") + self.run_ok(command, _FASTA, self.folder(), "--output-format", "parquet", "--output-codes-folder", codes) + return codes + + def test_build_residue_codes_from_shards_refuses_a_code_file(self): + db, shards = self.folder("db"), self.folder("shards") + with launch_env(FOLDMATCH_RUN_ID="p9x-plain"): + result = self.invoke("build", "sequences", "--fasta-file", _FASTA, "--output-db", db / "db", + "--tmp-embedding-folder", shards, app=search.app) + self.assertEqual(result.exit_code, 0, result.output) + out = self.codes_parquet() + stray = self.folder("stray") + shutil.copyfile(out / "inference-codes-0.parquet", stray / "emb_p9x-codes-0.parquet") + root = logging.getLogger() + handlers, level = root.handlers[:], root.level + try: + result = CliRunner().invoke(search.app, ["build", "residue-codes", str(db), "db", "--from-shards", + str(stray), "--run-id", "p9x-codes"]) + finally: + root.handlers[:] = handlers + root.setLevel(level) + self.assertEqual(result.exit_code, 1, result.output) + self.assertIn("not an embedding shard", _flat(str(result.exception))) + self.assertFalse(ResidueCodeStore(db, "db").exists()) + + +# ===================================================================================================================== +# The real ESM3 (CPU) +# ===================================================================================================================== +class RealEsm3Test(unittest.TestCase): + """The real ESM3 on the CPU, loaded once: in both commands and every format, the embedding files with and without + --output-codes-folder (byte for byte; a .json.gz but for its gzip mtime), the codes against the residue-code store + of 'fm-search build sequences --residue-codes' on the same FASTA (equal) and F2c (A4-xcpu), and 'fm-search build + embeddings' and 'query embedding' on the chain output folders (the same bytes as from a plain run's).""" + + @classmethod + def setUpClass(cls): + super().setUpClass() + esm_codes_bundle.registry_files() + gc.collect() + from foldmatch.utils.model import get_residue_model + cls.esm3 = get_residue_model(torch.device("cpu")) + cls._tmp = tempfile.TemporaryDirectory() + cls.tmp = Path(cls._tmp.name).resolve() + cls.results = {} + with mock.patch.object(sequence_inference, "get_residue_model", lambda **_: cls.esm3), \ + mock.patch.object(chain_complete_inference, "get_residue_model", lambda **_: cls.esm3), \ + launch_env(FOLDMATCH_RUN_ID="p9x-real"): + for command in _COMMANDS: + for out_format in _FORMATS: + for kind in ("plain", "codes"): + out = cls.tmp / f"{command}-{out_format}-{kind}" + out.mkdir() + extra = () + if kind == "codes": + (cls.tmp / f"{command}-{out_format}-code-files").mkdir() + extra = ("--output-codes-folder", cls.tmp / f"{command}-{out_format}-code-files") + cls.results[command, out_format, kind] = cls._invoke( + embedding.app, "from-sequences", command, "--fasta-file", _FASTA, "--output-folder", out, + "--output-format", out_format, "--batch-size", "2", *extra) + (cls.tmp / "shards").mkdir() + cls.results["build"] = cls._invoke(search.app, "build", "sequences", "--fasta-file", _FASTA, + "--output-db", cls.tmp / "db" / "db", "--tmp-embedding-folder", + cls.tmp / "shards", "--residue-codes", "--batch-size", "2") + del cls.esm3 + gc.collect() + + @classmethod + def _invoke(cls, app, *args): + root = logging.getLogger() + handlers, level = root.handlers[:], root.level + try: + return CliRunner().invoke(app, [str(a) for a in (*args, *_CPU)]) + finally: + root.handlers[:] = handlers + root.setLevel(level) + + @classmethod + def tearDownClass(cls): + cls._tmp.cleanup() + super().tearDownClass() + + def setUp(self): + for key, result in self.results.items(): + self.assertEqual(result.exit_code, 0, f"{key}: {result.output[-4000:]} {result.exception!r}") + + def test_the_embedding_files_are_those_of_a_run_without_codes(self): + for command in _COMMANDS: + for out_format in _FORMATS: + with self.subTest(command=command, out_format=out_format): + plain = self.tmp / f"{command}-{out_format}-plain" + out = self.tmp / f"{command}-{out_format}-codes" + self.assertEqual(len(names(plain)), 2 if out_format in ("csv", "pt") else 1) + self.assertEqual(names(out), names(plain)) + for name in names(plain): + self.assertTrue(same_bytes(plain / name, out / name), name) + if out_format in ("csv", "pt"): + self.assertEqual(names(self.tmp / f"{command}-{out_format}-code-files"), names(plain)) + + def test_the_codes_are_the_build_store_codes(self): + _, reference, ids, _ = _f2c() + store = ResidueCodeStore(self.tmp / "db", "db") + store.check_compatible() + memmap = np.asarray(store.open_memmap()) + entries = store.lookup(ids) + stored = {i: memmap[entries[i][0]:entries[i][0] + entries[i][1]] for i in ids} + for command in _COMMANDS: + for out_format in _FORMATS: + with self.subTest(command=command, out_format=out_format): + got = read_codes(self.tmp / f"{command}-{out_format}-code-files", out_format, ids=ids) + self.assertEqual(list(got), ids) + for dom_id in ids: + np.testing.assert_array_equal(got[dom_id], stored[dom_id]) + codes = np.concatenate([stored[i] for i in ids]) + ref = np.concatenate([reference[i] for i in ids]) + equal = int((np.unpackbits(codes, axis=1) == np.unpackbits(ref, axis=1)).sum()) + print(f"\nfm-embedding --output-codes-folder F2c CPU code bits: {equal}/{ref.size * 8}", file=sys.stderr) + self.assertGreaterEqual(equal / (ref.size * 8), A4_XCPU_BITS) + + def test_fm_search_reads_the_chain_output_folder(self): + """The chain output folder of a run with --output-codes-folder: 'fm-search build embeddings' gives the index + and metadata bytes of the plain run's folder, and 'query embedding' the same hits.""" + for out_format in ("parquet", "csv", "pt"): + files = {} + for kind in ("plain", "codes"): + folder = self.tmp / f"chain-{out_format}-{kind}" + db = self.tmp / f"db-{out_format}-{kind}" + db.mkdir() + hits = db / "hits.tsv" + for args in (("build", "embeddings", "--embedding-folder", folder, "--output-db", db / "db"), + ("query", "embedding", "--db-path", db / "db", "--embedding-folder", folder, + "--output-file", hits, "--threshold", "0")): + root = logging.getLogger() + handlers, level = root.handlers[:], root.level + try: + result = CliRunner().invoke(search.app, [str(a) for a in args]) + finally: + root.handlers[:] = handlers + root.setLevel(level) + self.assertEqual(result.exit_code, 0, f"{args}: {result.output} {result.exception!r}") + files[kind] = {name: (db / name).read_bytes() for name in ("db.index", "db.metadata", "hits.tsv")} + with self.subTest(out_format=out_format): + self.assertTrue(files["plain"]["hits.tsv"]) + self.assertEqual(files["codes"], files["plain"]) + print(f"\nchain {out_format}: fm-search build embeddings / query embedding on the output folder of a " + f"run with --output-codes-folder: db.index {len(files['codes']['db.index'])} B, db.metadata " + f"{len(files['codes']['db.metadata'])} B, hits {len(files['codes']['hits.tsv'])} B, identical " + f"to the plain run's", file=sys.stderr) + + +# ===================================================================================================================== +# Two ranks (slow) +# ===================================================================================================================== +_SITECUSTOMIZE = r""" +import os +import numpy as np +import torch +from foldmatch.inference import chain_complete_inference, sequence_inference +from foldmatch.modules import chain_complete_module, esm_module +z = np.load(os.environ["FOLDMATCH_TEST_F2C"]) +ro = z["rows_offsets"] +rows = {str(z["sequences"][k]): torch.from_numpy(z["rows"][ro[k]:ro[k + 1]].astype(np.float32)) + for k in range(len(ro) - 1)} +for module in (sequence_inference, chain_complete_inference): + module.get_residue_model = lambda **_: torch.nn.Identity() +short = os.environ.get("FOLDMATCH_TEST_SHORT") # the index of a sequence whose ESM3 rows lack one row +short = None if short is None else str(z["sequences"][int(short)]) +slow = float(os.environ.get("FOLDMATCH_TEST_SLOW_RANK0") or 0) # seconds that rank 0 spends on each sequence + + +def per_residue_embedding(model, protein): + if slow and os.environ.get("LOCAL_RANK", "0") == "0": + import time + time.sleep(slow) + return (rows[protein.sequence][:-1] if protein.sequence == short else rows[protein.sequence]).clone() + + +for module in (esm_module, chain_complete_module): + module.per_residue_embedding = per_residue_embedding +""" +#: A library caller on two ranks (Lightning's launcher starts rank 1 by running this script again): predict with +#: out_codes_folder, then this rank's predictions (ids, codes, seq_sha1s) and the error it raised, if any, to +#: /rank-.pt. +_LIBRARY_DRIVER = r""" +import sys +from pathlib import Path +import torch +from foldmatch.errors import MissingResidueCodesError +from foldmatch.inference import chain_complete_inference, sequence_inference +from foldmatch.types.api_types import Accelerator, OutFormat, SrcEsmFrom + +command, fasta, out, codes, out_format, dump = sys.argv[1:7] +kwargs = dict(accelerator=Accelerator.cpu, devices=2, batch_size=1, out_folder=Path(out), + out_format=OutFormat(out_format), out_codes_folder=Path(codes)) +if command == "residue": + predict = sequence_inference.predict + kwargs["fasta_file"] = Path(fasta) +else: + predict = chain_complete_inference.predict + kwargs.update(src_stream=Path(fasta), src_from=SrcEsmFrom.fasta) +error = None +try: + predictions = predict(**kwargs) +except MissingResidueCodesError as e: + predictions, error = e.predictions, str(e) +rank = torch.distributed.get_rank() if torch.distributed.is_initialized() else 0 +torch.save({"predictions": [tuple(p[1:]) for p in predictions], "error": error}, f"{dump}/rank-{rank}.pt") +""" +_DDP_ENV_PREFIXES = ("MASTER_", "PL_", "TORCHELASTIC_") +_DDP_ENV = ("WORLD_SIZE", "LOCAL_WORLD_SIZE", "GROUP_RANK", "ROLE_RANK") + + +@unittest.skipUnless(os.environ.get("FOLDMATCH_SLOW_TESTS") == "1", + "slow: two CPU ranks as a command; set FOLDMATCH_SLOW_TESTS=1 to run") +class TwoRankSmoke(unittest.TestCase): + """``--devices 2`` (CPU, gloo; ESM3 replaced by the F2c rows in both processes): a codes file per rank in parquet, + a code file per id in csv, in the codes folder, the F2c codes; the output folder holds the embeddings only. W4: + with one sequence's ESM3 rows a row short, on rank 1 (a process that Lightning's launcher, rank 0, started) or on + rank 0, the run ends, no rank hangs, every file is written, and the exit status is 1, from rank 0, also when + rank 0 ends its pass long after rank 1 (which then waits for it, rather than failing while rank 0 still writes, + which would get rank 0 killed), under Lightning's launcher and under torchrun.""" + + def run_two_ranks(self, tmp, env, command, out_format, short=None, slow=0, torchrun=False, out=None, + codes=None): + """A run on two ranks, into new folders, or into ``out`` and ``codes`` (None: no codes) when given.""" + tag = f"{command}-{out_format}-{short}-{slow}-{'torchrun' if torchrun else 'lightning'}" + if out is None: + out, codes = tmp / tag, tmp / f"{tag}-codes" + out.mkdir() + codes.mkdir() + env = dict(env) + if short is not None: + env["FOLDMATCH_TEST_SHORT"] = str(short) + if slow: + env["FOLDMATCH_TEST_SLOW_RANK0"] = str(slow) + launcher = [sys.executable] + if torchrun: + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + launcher += ["-m", "torch.distributed.run", "--nproc-per-node", "2", "--master-addr", "127.0.0.1", + "--master-port", str(port)] + start = time.monotonic() + run = subprocess.run([*launcher, "-m", "foldmatch.cli.embedding", "from-sequences", command, + "--fasta-file", str(_FASTA), "--output-folder", str(out), "--output-format", + out_format, *(() if codes is None else ("--output-codes-folder", str(codes))), + "--accelerator", "cpu", "--devices", "2"], + env=env, capture_output=True, text=True, timeout=1800) + print(f"\nTwoRankSmoke {tag}: exit status {run.returncode} in {time.monotonic() - start:.1f} s", + file=sys.stderr) + return run, out, codes + + def env(self, tmp): + (tmp / "site").mkdir() + (tmp / "site" / "sitecustomize.py").write_text(_SITECUSTOMIZE) + env = {k: v for k, v in os.environ.items() + if k not in _LAUNCH_ENV and k not in _DDP_ENV and not k.startswith(_DDP_ENV_PREFIXES)} + env["PYTHONPATH"] = os.pathsep.join([str(tmp / "site"), _SRC] + + [p for p in [os.environ.get("PYTHONPATH")] if p]) + env["FOLDMATCH_TEST_F2C"] = str(_F2C) + return env + + def test_two_ranks(self): + esm_codes_bundle.registry_files() + _, reference, ids, _ = _f2c() + with tempfile.TemporaryDirectory() as tmp: + tmp = Path(tmp) + env = self.env(tmp) + for command, out_format in (("residue", "parquet"), ("chain", "csv"), ("chain", "pt")): + with self.subTest(command=command, out_format=out_format): + run, out, codes = self.run_two_ranks(tmp, env, command, out_format) + self.assertEqual(run.returncode, 0, run.stderr[-4000:]) + if out_format == "parquet": + self.assertEqual(names(out), ["inference-0.parquet", "inference-1.parquet"]) + self.assertEqual(names(codes), ["inference-codes-0.parquet", "inference-codes-1.parquet"]) + for rank in (0, 1): + record = json.loads(pq.read_schema(codes / f"inference-codes-{rank}.parquet") + .metadata[b"foldmatch.pass"]) + self.assertEqual(record["world_size"], 2) + else: + self.assertEqual(names(out), sorted(f"{i}.{out_format}" for i in ids)) + self.assertEqual(names(codes), names(out)) + got = read_codes(codes, out_format, ids=ids) + self.assertEqual(sorted(got), sorted(ids)) + for dom_id in ids: + np.testing.assert_array_equal(got[dom_id], reference[dom_id]) + # Z4: a rerun into the same folders on two ranks replaces its files (each rank checks the files + # of both ranks, which the other may be writing), with and without the codes + files = sorted([*out.iterdir(), *codes.iterdir()]) + for path in files: + os.utime(path, ns=(0, 0)) + rerun = self.run_two_ranks(tmp, env, command, out_format, out=out, codes=codes)[0] + self.assertEqual(rerun.returncode, 0, rerun.stderr[-4000:]) + self.assertNotIn("would replace", rerun.stderr) + self.assertEqual(sorted([*out.iterdir(), *codes.iterdir()]), files) + self.assertTrue(all(path.stat().st_mtime_ns > 0 for path in files)) + rerun = self.run_two_ranks(tmp, env, command, out_format, out=out, codes=None)[0] + self.assertEqual(rerun.returncode, 0, rerun.stderr[-4000:]) + + def test_two_ranks_return_the_codes(self): + """Y2 on two ranks, a library caller (Lightning's launcher): each rank's predictions hold its ids' codes as CPU + uint8 tensors [L, 8], those of the .pt code files and the bytes of the codes Parquet files, for both predict + functions; with a sequence without codes on rank 1, rank 0's MissingResidueCodesError carries rank 0's + predictions and rank 1 returns its own, the empty table a [0, 8] tensor equal to its .pt file.""" + esm_codes_bundle.registry_files() + _, reference, ids, sequences = _f2c() + with tempfile.TemporaryDirectory() as tmp: + tmp = Path(tmp) + env = self.env(tmp) + driver = tmp / "driver.py" + driver.write_text(_LIBRARY_DRIVER) + for command, out_format, short in (("residue", "pt", None), ("residue", "parquet", None), + ("chain", "pt", None), ("chain", "parquet", None), + ("chain", "pt", 1)): + with self.subTest(command=command, out_format=out_format, short=short): + tag = f"lib-{command}-{out_format}-{short}" + out, codes, dump = tmp / tag, tmp / f"{tag}-codes", tmp / f"{tag}-dump" + for folder in (out, codes, dump): + folder.mkdir() + run_env = dict(env) + if short is not None: + run_env["FOLDMATCH_TEST_SHORT"] = str(short) + start = time.monotonic() + run = subprocess.run([sys.executable, str(driver), command, str(_FASTA), str(out), str(codes), + out_format, str(dump)], env=run_env, capture_output=True, text=True, + timeout=1800) + print(f"\nTwoRankSmoke {tag}: exit status {run.returncode} in {time.monotonic() - start:.1f} s", + file=sys.stderr) + self.assertEqual(run.returncode, 0, run.stderr[-4000:]) + self.assertEqual(names(dump), ["rank-0.pt", "rank-1.pt"]) + returned = {} + for rank in (0, 1): + record = torch.load(dump / f"rank-{rank}.pt", weights_only=False) + if short is None or rank == 1: + self.assertIsNone(record["error"]) + else: + self.assertTrue(record["error"].startswith( + f"1 sequence(s) on rank 1 of 2 got no residue codes: {ids[short]!r}."), + record["error"]) + for names_, tables, digests in record["predictions"]: + self.assertIsInstance(tables, tuple) + for dom_id, table, digest in zip(names_, tables, digests): + returned[dom_id] = (rank, table, digest) + self.assertEqual(sorted(returned), sorted(ids)) + self.assertEqual({rank for rank, _, _ in returned.values()}, {0, 1}) + if out_format == "parquet": + stored = {} + for rank in (0, 1): + parquet = pq.read_table(codes / f"inference-codes-{rank}.parquet") + column = parquet.column("residue_codes").combine_chunks() + for k, dom_id in enumerate(parquet.column("id").to_pylist()): + stored[dom_id] = (rank, column[k].values.flatten().to_numpy(zero_copy_only=False) + .tobytes()) + for k, dom_id in enumerate(ids): + rank, table, digest = returned[dom_id] + self.assertEqual((table.dtype, table.device.type), (torch.uint8, "cpu")) + self.assertEqual(digest, _sha1(sequences[k])) + if dom_id == (ids[short] if short is not None else None): + self.assertEqual(tuple(table.shape), (0, 8)) + else: + self.assertEqual(tuple(table.shape), (len(sequences[k]), 8)) + np.testing.assert_array_equal(table.numpy(), reference[dom_id]) + if out_format == "pt": + pt = torch.load(codes / f"{dom_id}.pt", weights_only=True) + self.assertEqual((pt.dtype, pt.shape), (table.dtype, table.shape)) + self.assertTrue(torch.equal(pt, table)) + else: + self.assertEqual(stored[dom_id], (rank, table.numpy().tobytes())) + + def test_two_ranks_with_missing_codes(self): + """W4: the sequence of rank 1 (ids[1]) or of rank 0 (ids[0]) gets no codes; rank 0 also ends its pass 12 s + after rank 1 (more than the 5 s that Lightning's child observer takes to kill every process once a rank exits + with an error), under Lightning's launcher and under torchrun.""" + esm_codes_bundle.registry_files() + _, reference, ids, _ = _f2c() + with tempfile.TemporaryDirectory() as tmp: + tmp = Path(tmp) + env = self.env(tmp) + for command, out_format, short, slow, torchrun in ( + ("chain", "parquet", 1, 0, False), ("residue", "csv", 1, 0, False), + ("chain", "csv", 0, 0, False), ("residue", "parquet", 0, 0, False), + ("chain", "parquet", 1, 12, False), ("residue", "csv", 1, 12, True)): + with self.subTest(command=command, out_format=out_format, short_rank=short, slow=slow, + torchrun=torchrun): + run, out, codes = self.run_two_ranks(tmp, env, command, out_format, short, slow, torchrun) + stderr = _flat(run.stderr) + self.assertEqual(run.returncode, 1, run.stderr[-4000:]) + message = f"Error: 1 sequence(s) on rank {short} of 2 got no residue codes: {ids[short]!r}." + self.assertEqual(stderr.count(message), 1, run.stderr[-4000:]) + self.assertNotIn("Forcefully terminating", stderr) + if short == 1: + self.assertIn("1 sequence(s) on rank 1 of 2 got no residue codes; rank 0 ends the run with an " + "error that names them", stderr) + if out_format == "parquet": + self.assertEqual(names(out), ["inference-0.parquet", "inference-1.parquet"]) + self.assertEqual(names(codes), ["inference-codes-0.parquet", "inference-codes-1.parquet"]) + self.assertIn(f"(rows of {codes.resolve()}/inference-codes-{short}.parquet without " + f"residues)", stderr) + else: + self.assertEqual(names(out), sorted(f"{i}.{out_format}" for i in ids)) + self.assertEqual(names(codes), names(out)) + self.assertEqual((codes / f"{ids[short]}.csv").read_bytes(), b"") + got = read_codes(codes, out_format, ids=ids) + self.assertEqual(got[ids[short]].shape, (0, 8)) + np.testing.assert_array_equal(got[ids[1 - short]], reference[ids[1 - short]]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_name_conflicts.py b/tests/test_name_conflicts.py index a305f33..54ab50a 100644 --- a/tests/test_name_conflicts.py +++ b/tests/test_name_conflicts.py @@ -947,7 +947,8 @@ def test_an_output_file_linked_to_an_input(self): def test_no_link_among_the_output_files(self): """Where no output's file that exists is a link, an input that is not a link is not looked up beyond its - lstat: one lstat per output file and per input, and a stat per folder.""" + lstat: one lstat per output file and per input, and a stat per folder (and a stat per output file that exists, + to tell its kind once: refuse_other_kind).""" of = _output_files() out, inputs, n = self.folder(), self.folder("in"), 200 for k in range(n): @@ -958,7 +959,7 @@ def test_no_link_among_the_output_files(self): mock.patch.object(of.os, "lstat", wraps=os.lstat) as lstat: of.check_outputs(out, OutFormat.pt, None, names=[f"s{k}" for k in range(n)], folding=folding, inputs=[("the structure file", inputs / f"s{k}.cif") for k in range(n)]) - self.assertLessEqual(stat.call_count + lstat.call_count, 2 * n + 4) + self.assertLessEqual(stat.call_count + lstat.call_count, 3 * n + 4) def test_chain_embedding_into_links_to_its_residues(self): """chain-embedding into a folder whose 1acb.A.pt is a symbolic link to the residue embedding it reads diff --git a/tests/test_parquet_embeddings.py b/tests/test_parquet_embeddings.py index 9e0aa07..6651591 100644 --- a/tests/test_parquet_embeddings.py +++ b/tests/test_parquet_embeddings.py @@ -212,9 +212,9 @@ def test_a_file_that_cannot_be_read_or_lacks_a_column_is_refused(self): file: an empty file after a chain shard, a shard cut short (its first half, as an interrupted run leaves it: the footer is written last), a shard with a corrupted page (its first page header zeroed) and a shard's rows under the columns name and vector, or without id. They stopped on pyarrow's message, which names no file, or - on a traceback: an OSError, or a KeyError (iter_batches drops a missing column without a word). A folder named - *.parquet, such as the dataset folder that Spark writes, is refused as a folder, and a broken link with the - system's reason, which pyarrow's message gives after the whole path.""" + on a traceback: an OSError, or a KeyError (iter_batches drops a missing column without a word). A broken link + is refused with the system's reason, which pyarrow's message gives after the whole path. (A folder named + *.parquet, such as the dataset folder that Spark writes, is skipped: tests/test_embedding_folders.py.)""" shard = self.shard(self.folder("shard"), _vectors(64)) data, table = shard.read_bytes(), pq.read_table(shard) empty = self.folder("empty") @@ -227,8 +227,6 @@ def test_a_file_that_cannot_be_read_or_lacks_a_column_is_refused(self): renamed, no_id = folder / "renamed.parquet", folder / "no-id.parquet" pq.write_table(table.rename_columns(["name", "vector"]), renamed) pq.write_table(table.drop_columns(["id"]), no_id) - dataset = self.folder("dataset") - (dataset / "embeddings.parquet").mkdir() broken = self.folder("broken") (broken / "inference-0.parquet").symlink_to(broken / "moved.parquet") again = "Write it again: an interrupted run or a damaged copy leaves an unreadable file." @@ -241,7 +239,6 @@ def test_a_file_that_cannot_be_read_or_lacks_a_column_is_refused(self): f"thrift: TProtocolException: Invalid data). {again}"), "name and vector": (renamed, f"{renamed} has no 'id' or 'embedding' column: {columns}"), "no id": (no_id, f"{no_id} has no 'id' column: {columns}"), - "folder": (dataset, f"{dataset / 'embeddings.parquet'} is a folder, not a Parquet file."), "broken link": (broken, f"{broken / 'inference-0.parquet'} could not be opened (No such file or " f"directory).")} for name, (path, text) in cases.items(): diff --git a/tests/test_residue_code_files.py b/tests/test_residue_code_files.py new file mode 100644 index 0000000..ed3a30b --- /dev/null +++ b/tests/test_residue_code_files.py @@ -0,0 +1,432 @@ +"""P9.X: the residue-code files of ``fm-embedding from-sequences residue|chain --output-codes-folder``: their writers +(one file per id in csv and pt, one per rank in parquet and json) and the readers that must refuse them (the shard readers of +builds and queries, and the embedding readers of fm-search). + +No model: the predictions are made up, as a module with the ``residue_codes`` output returns them. +""" +import gzip +import hashlib +import json +import shutil +import tempfile +import unittest +import unittest.mock +from pathlib import Path +from types import SimpleNamespace + +import numpy as np +import pyarrow.parquet as pq +import torch + +from foldmatch.cli import search +from foldmatch.errors import RefusalError +from foldmatch.search import residue_shards as rs +from foldmatch.search import scoring_system as ss +from foldmatch.writer.batch_writer import ( + CsvBatchWriter, CsvCodeWriter, JsonCodeStorage, ParquetBatchWriter, ParquetCodeWriter, TensorBatchWriter, + TensorCodeWriter, residue_code_tables, +) + +SYSTEM = ss.get(ss.DEFAULT_ESM_CODES) + + +def _prediction(ids, lengths, seed=0): + rng = np.random.default_rng(seed) + tables = [rng.integers(0, 256, (n, 8), dtype=np.uint8) for n in lengths] + return (([torch.zeros(1)] * len(ids)), tuple(ids), tuple(t.tobytes() for t in tables), + tuple(hashlib.sha1(i.encode()).digest() for i in ids)), tables + + +_CODES_MODULE = SimpleNamespace(residue_output=rs.RESIDUE_CODES, device=torch.device("cpu")) + + +class WriterTest(unittest.TestCase): + + def setUp(self): + self.out = Path(self.enterContext(tempfile.TemporaryDirectory())) + + def test_parquet_ranks_and_row_groups(self): + for rank in (0, 3): + trainer = SimpleNamespace(global_rank=rank, world_size=4) + writer = ParquetCodeWriter(str(self.out), "c", SYSTEM, min_res_n=0, byte_budget=100) + writer.on_predict_start(trainer, _CODES_MODULE) + ids = [f"r{rank}-{k}" for k in range(4)] + prediction, tables = _prediction(ids, [10, 0, 3, 7], seed=rank) + writer.write_on_batch_end(trainer, None, prediction, None, None, 0, 0) + writer.write_on_batch_end(trainer, None, _prediction(["z"], [20])[0], None, None, 1, 0) + writer.on_predict_end(trainer, None) + path = self.out / f"c-{rank}.parquet" + self.assertEqual(pq.ParquetFile(path).metadata.num_row_groups, 2) + table = pq.read_table(path) + self.assertEqual(table.column("id").to_pylist(), ids + ["z"]) + for got, want in zip(table.column("residue_codes").to_pylist(), tables): + np.testing.assert_array_equal(np.array(got, dtype=np.uint8).reshape(-1, 8), want) + self.assertEqual(json.loads(pq.read_schema(path).metadata[b"foldmatch.pass"])["world_size"], 4) + self.assertEqual(sorted(p.name for p in self.out.iterdir()), ["c-0.parquet", "c-3.parquet"]) + + def test_no_rows_no_parquet_file(self): + writer = ParquetCodeWriter(str(self.out), "c", SYSTEM) + trainer = SimpleNamespace(global_rank=0, world_size=1) + writer.on_predict_start(trainer, _CODES_MODULE) + writer.on_predict_end(trainer, None) + self.assertEqual(list(self.out.iterdir()), []) + + def test_json_ranks(self): + for rank in (0, 1): + trainer = SimpleNamespace(global_rank=rank) + writer = JsonCodeStorage(str(self.out), "c") + prediction, tables = _prediction(["a", "b"], [2, 0]) + writer.write_on_batch_end(trainer, None, prediction, None, None, 0, 0) + writer.on_predict_end(trainer, None) + records = json.loads(gzip.decompress((self.out / f"c-{rank}.json.gz").read_bytes())) + self.assertEqual(records, [{"id": "a", "residue_codes": tables[0].tolist()}, + {"id": "b", "residue_codes": []}]) + + def test_item_writers(self): + """csv and pt: /., as the embedding of the id is /.; the + layouts; a file that the folder takes for one written twice is refused, naming the residue codes.""" + trainer = SimpleNamespace(global_rank=0) + prediction, tables = _prediction(["a", "b"], [3, 0]) + for writer, fmt in ((CsvCodeWriter, "csv"), (TensorCodeWriter, "pt")): + with self.subTest(fmt=fmt): + out = Path(tempfile.mkdtemp(dir=self.out)) + codes = writer(str(out)) + codes.on_predict_start(trainer, _CODES_MODULE) + codes.write_on_batch_end(trainer, None, prediction, None, None, 0, 0) + self.assertEqual(sorted(p.name for p in out.iterdir()), [f"a.{fmt}", f"b.{fmt}"]) + if fmt == "csv": + self.assertEqual((out / "a.csv").read_text(), + "".join(",".join(map(str, row)) + "\n" for row in tables[0])) + self.assertEqual((out / "b.csv").read_bytes(), b"") + else: + for dom_id, table in zip("ab", tables): + got = torch.load(out / f"{dom_id}.pt") + self.assertEqual((got.dtype, got.device.type), (torch.uint8, "cpu")) + np.testing.assert_array_equal(got.numpy(), table) + with self.assertRaises(RefusalError) as refused: + codes.write_on_batch_end(trainer, None, _prediction(["a"], [2])[0], None, None, 1, 0) + text = str(refused.exception) + self.assertTrue(text.startswith( + f"{out}/a.{fmt} would be written twice by this run, for 'a' and then for 'a': the first is kept"), + text) + self.assertIn("Each residue-code table of a run needs a file of its own: write them as parquet or " + "json", text) + + def test_a_code_file_name_that_cannot_be(self): + trainer = SimpleNamespace(global_rank=0) + long_id = "x" * 260 + with self.assertRaises(RefusalError) as refused: + CsvCodeWriter(str(self.out)).write_on_batch_end(trainer, None, _prediction([long_id], [3])[0], None, None, + 0, 0) + text = str(refused.exception) + self.assertTrue(text.startswith(f"The residue codes of {long_id!r} cannot be written to {self.out}"), text) + self.assertTrue(text.endswith("or give it another name."), text) + self.assertIn("Write the residue codes as parquet or json, which keep each under its id", text) + # The embedding writers' refusals are those of before + plain = CsvBatchWriter(str(self.out)) + with self.assertRaises(RefusalError) as refused: + plain.write_on_batch_end(trainer, None, ([torch.ones(2)], ("y" * 260,)), None, None, 0, 0) + self.assertTrue(str(refused.exception).startswith("The embedding 'yyy"), str(refused.exception)) + self.assertIn("Write the embeddings as parquet or json, which keep each under its id", str(refused.exception)) + self.assertTrue(str(refused.exception).endswith("or give it another name."), str(refused.exception)) + plain.write_on_batch_end(trainer, None, ([torch.ones(2)], ("e",)), None, None, 0, 0) + with self.assertRaises(RefusalError) as refused: + plain.write_on_batch_end(trainer, None, ([torch.ones(2)], ("e",)), None, None, 1, 0) + self.assertTrue(str(refused.exception).endswith( + "Each embedding of a run needs a file of its own: write them as parquet or json, which keep each under " + "its id, or, for names that differ only in case, to a folder of a case-sensitive file system."), + str(refused.exception)) + + def test_writers_refuse_a_module_without_codes(self): + trainer = SimpleNamespace(global_rank=0, world_size=1) + for writer in (ParquetCodeWriter(str(self.out), "c", SYSTEM), JsonCodeStorage(str(self.out), "c"), + CsvCodeWriter(str(self.out)), TensorCodeWriter(str(self.out))): + with self.subTest(writer=type(writer).__name__): + with self.assertRaisesRegex(ValueError, "the module produces no residue output"): + writer.on_predict_start(trainer, SimpleNamespace(residue_output=None)) + with self.assertRaisesRegex(ValueError, "the module produces query_encoding"): + writer.on_predict_start(trainer, SimpleNamespace(residue_output=rs.QUERY_ENCODING)) + + def test_residue_code_tables(self): + with self.assertRaisesRegex(ValueError, "got a 2-tuple"): + residue_code_tables(([], ())) + with self.assertRaisesRegex(ValueError, "not a whole number of 8-byte codes"): + residue_code_tables(([None], ("a",), (b"123",), (bytes(20),))) + with self.assertRaisesRegex(ValueError, "seq_sha1 must be 20 bytes"): + residue_code_tables(([None], ("a",), (bytes(8),), (b"x",))) + ids, tables, _ = residue_code_tables(([None], ("a",), (bytes(range(16)),), (bytes(20),))) + self.assertTrue(tables[0].flags.writeable) + np.testing.assert_array_equal(tables[0], np.arange(16, dtype=np.uint8).reshape(2, 8)) + + + +def write_run(folder: Path, codes_name: str = "inference-codes", embeddings: bool = True): + """The Parquet files of a chain run of ids a and b with --output-codes-folder, both in ``folder`` (as a user who + moved them there would have them): inference-0.parquet (legacy chain embeddings, unless not ``embeddings``) and + ``-0.parquet``.""" + trainer = SimpleNamespace(global_rank=0, world_size=1) + prediction, tables = _prediction(["a", "b"], [3, 5]) + if embeddings: + writer = ParquetBatchWriter(str(folder), "inference") + writer.write_on_batch_end(trainer, None, ([torch.ones(4), torch.zeros(4)],) + prediction[1:], None, None, 0, 0) + writer.on_predict_end(trainer, None) + writer = ParquetCodeWriter(str(folder), codes_name, SYSTEM, min_res_n=0) + writer.on_predict_start(trainer, _CODES_MODULE) + writer.write_on_batch_end(trainer, None, prediction, None, None, 0, 0) + writer.on_predict_end(trainer, None) + return tables + + +# ===================================================================================================================== +# The readers of shards and of embeddings refuse a code file +# ===================================================================================================================== +class ReaderTest(unittest.TestCase): + + def setUp(self): + self.out = Path(self.enterContext(tempfile.TemporaryDirectory())) + + def test_shard_readers_refuse_a_code_file(self): + shards = self.out / "shards" + shards.mkdir() + write_run(shards, codes_name="emb_p9x-run", embeddings=False) + shard = shards / "emb_p9x-run-0.parquet" + info = rs.read_shard_info(shard) + self.assertEqual(info.residue_output, rs.RESIDUE_CODE_TABLE) + for kind in rs.RESIDUE_OUTPUTS: + with self.subTest(kind=kind), self.assertRaisesRegex(rs.ResidueShardError, "not an embedding shard"): + rs.check_shards([shard], kind, SYSTEM) + with self.assertRaisesRegex(rs.ResidueShardError, "not an embedding shard"): + rs.iter_residue_codes([shard], SYSTEM) + from foldmatch.search.query_encodings import QueryEncodingIndex + with self.assertRaisesRegex(rs.ResidueShardError, "not an embedding shard"): + QueryEncodingIndex.build([shard], shards / "q.bin", system=SYSTEM) + self.assertFalse((shards / "q.bin").exists()) + with self.assertRaisesRegex(RefusalError, "None of the embedding shards .* hold residue codes"): + search._residue_code_shards(shards, None) + from foldmatch.search import split_run + with self.assertRaisesRegex(RefusalError, "not an embedding shard"): + split_run.check_codes_footers([shard], system=SYSTEM, min_res_n=0, world_size=1) + with self.assertRaisesRegex(RefusalError, "not the residue codes of"): + split_run.check_codes_footers([shard], expect={"name": SYSTEM.name, "digest": SYSTEM.digest(), + "encoding_digest": SYSTEM.encoding_digest()}, + min_res_n=0, world_size=1) + with self.assertRaisesRegex(RefusalError, "not an embedding shard|not plain|not the residue codes"): + search._seal_footers([shard], folder=shards, run_id="p9x-run", min_res_n=None) + with self.assertRaises(RefusalError): + split_run.check_plain_footers([shard]) + + def test_embedding_readers_refuse_a_code_file(self): + from foldmatch.utils.data import refuse_repeated_query_ids, stream_embeddings + out = self.out + write_run(out) + refuse_repeated_query_ids(out) # the ids of the code file are not query ids + with self.assertRaises(RefusalError) as refused: + list(stream_embeddings(out)) + self.assertEqual(str(refused.exception), ( + f"{out / 'inference-codes-0.parquet'} holds residue codes, as 'fm-embedding from-sequences ... " + f"--output-codes-folder' writes them (one table of 8-byte codes per sequence), not embeddings: residue " + f"codes belong in a folder of their own (--output-codes-folder); move them out of the embedding folder.")) + self.assertEqual([ids for ids, _ in stream_embeddings(out / "inference-0.parquet")], [["a", "b"]]) + + def test_embedding_readers_refuse_code_csv_and_pt_files(self): + """fm-search's .pt/.csv embedding reader on code files: a .pt by its type, whatever L (a one-residue code + table would otherwise be an 8-value vector); a .csv table with a word on residue codes. A one-residue .csv + cannot be told from an 8-value vector (a question for the user).""" + from foldmatch.utils.data import _file_vector + trainer = SimpleNamespace(global_rank=0) + prediction, _ = _prediction(["a", "b", "c"], [5, 1, 0]) + for writer, fmt in ((TensorCodeWriter, "pt"), (CsvCodeWriter, "csv")): + out = Path(tempfile.mkdtemp(dir=self.out)) + writer(str(out)).write_on_batch_end(trainer, None, prediction, None, None, 0, 0) + for dom_id, rows in (("a", 5), ("b", 1), ("c", 0)): + with self.subTest(fmt=fmt, rows=rows): + path = out / f"{dom_id}.{fmt}" + if fmt == "csv" and rows == 1: + self.assertEqual(_file_vector(path).shape, (8,)) # not recognisable + continue + with self.assertRaises(RefusalError) as refused: + _file_vector(path) + text = str(refused.exception) + if fmt == "pt": + self.assertIn(f"holds residue codes (a uint8 tensor of {rows} x 8), as 'fm-embedding " + f"from-sequences ... --output-codes-folder' writes them, not an embedding " + f"vector: residue codes belong in a folder of their own " + f"(--output-codes-folder); move them out of the embedding folder.", text) + elif rows == 5: + self.assertIn("holds a table of 5 x 8 values, not one embedding vector", text) + self.assertIn("A table of 8 columns of integers 0-255 may be residue codes, as " + "'fm-embedding from-sequences ... --output-codes-folder' writes them (.csv): " + "they are not embeddings either; residue codes belong in a folder of their own " + "(--output-codes-folder): move them out of the embedding folder.", text) + else: + self.assertIn("is not a CSV of numbers", text) + # Embeddings are read as before: float tables are not called codes, float vectors and an 8-value uint8 + # vector are vectors + out = Path(tempfile.mkdtemp(dir=self.out)) + torch.save(torch.rand(3, 8), out / "f.pt") + with self.assertRaises(RefusalError) as refused: + _file_vector(out / "f.pt") + self.assertNotIn("residue codes", str(refused.exception)) + torch.save(torch.arange(8, dtype=torch.uint8), out / "v.pt") + np.testing.assert_array_equal(_file_vector(out / "v.pt"), np.arange(8, dtype=np.float32)) + + +# ===================================================================================================================== +# What an output file holds (foldmatch.utils.file_kinds) +# ===================================================================================================================== +class FileKindTest(unittest.TestCase): + """Positive identification of each writer's files: embeddings of every writer and format, residue codes of every + code writer; anything else (empty, cut short, corrupt, foreign) of no kind.""" + + def setUp(self): + self.out = Path(self.enterContext(tempfile.TemporaryDirectory())) + + def written(self) -> dict: + """{file: kind} of every writer, a and b (3 and 0 residues) in each.""" + trainer = SimpleNamespace(global_rank=0, world_size=1) + prediction, _ = _prediction(["a", "b"], [3, 0]) + chain = ([torch.rand(6), torch.rand(6)],) + prediction[1:] + residue = ([torch.rand(5, 6), torch.rand(2, 6)],) + prediction[1:] + files = {} + + def run(writer, pred, folder, kind, start=None): + folder = self.out / folder + folder.mkdir() + writer.out_path = str(folder) + if start is not None: + writer.on_predict_start(trainer, start) + writer.write_on_batch_end(trainer, None, pred, None, None, 0, 0) + if hasattr(writer, "on_predict_end"): + writer.on_predict_end(trainer, None) + files.update({path: kind for path in folder.iterdir()}) + + from foldmatch.utils.file_kinds import EMBEDDINGS, RESIDUE_CODES + from foldmatch.writer.batch_writer import JsonStorage + for name, pred in (("chain", chain), ("residue", residue)): + run(CsvBatchWriter("."), pred, f"{name}-csv", EMBEDDINGS) + run(TensorBatchWriter("."), pred, f"{name}-pt", EMBEDDINGS) + run(JsonStorage(".", "e"), pred, f"{name}-json", EMBEDDINGS) + run(ParquetBatchWriter(".", "e"), chain, "chain-parquet", EMBEDDINGS) + run(ParquetBatchWriter(".", "e", residue_embeddings=True), residue, "residue-parquet", EMBEDDINGS) + run(ParquetBatchWriter(".", "e", residue_output=rs.RESIDUE_CODES, system=SYSTEM, min_res_n=0), + chain + (), "shard-parquet", EMBEDDINGS, _CODES_MODULE) + run(CsvCodeWriter("."), prediction, "codes-csv", RESIDUE_CODES, _CODES_MODULE) + run(TensorCodeWriter("."), prediction, "codes-pt", RESIDUE_CODES, _CODES_MODULE) + run(JsonCodeStorage(".", "c"), prediction, "codes-json", RESIDUE_CODES, _CODES_MODULE) + run(ParquetCodeWriter(".", "c", SYSTEM), prediction, "codes-parquet", RESIDUE_CODES, _CODES_MODULE) + # b's code table has no residue: an empty csv is of no kind, an empty [0, 8] tensor is residue codes + files[self.out / "codes-csv" / "b.csv"] = None + return files + + def test_every_writer(self): + from foldmatch.utils.file_kinds import file_kind + files = self.written() + self.assertEqual(len(files), 19) + for path, kind in files.items(): + with self.subTest(file=f"{path.parent.name}/{path.name}"): + self.assertEqual(file_kind(path), kind) + + def test_a_file_cut_short_is_never_of_the_other_kind(self): + """A prefix of a file (a file that a writer is writing, or that was cut short) is of its kind or of none: + never of the other kind; a Parquet or .pt file is of none until whole.""" + from foldmatch.utils import file_kinds + for path, kind in self.written().items(): + data = path.read_bytes() + cut = path.parent / f"cut-{path.name}" + for size in sorted({0, 1, 2, 5, 10, 20, 31, 64, 100, len(data) // 2, len(data) - 1}): + if size >= len(data): + continue + with self.subTest(file=f"{path.parent.name}/{path.name}", size=size): + cut.write_bytes(data[:size]) + got = file_kinds.file_kind(cut) + self.assertIn(got, (kind, None)) + if path.suffix in (".parquet", ".pt"): + self.assertIsNone(got) + + def test_foreign_files_are_of_no_kind(self): + from foldmatch.utils.file_kinds import file_kind + import pyarrow as pa + foreign = { + "random.pt": bytes(range(256)) * 4, "random.parquet": b"PAR1" + bytes(100) + b"PAR1", + "random.csv": bytes(range(256)), "random.json.gz": bytes(range(256)), "text.json.gz": b"[]", + "header.csv": b"a,b,c\n0.5,1.5,2.5\n", "integers.csv": b"1,2,3\n4,5,6\n", + "nine.csv": b"1,2,3,4,5,6,7,8,9\n", "big.csv": b"1,2,3,4,5,6,7,256\n", "word.csv": b"x\n", + "spaces.csv": b" 0.5, 1.5\n", "codes-no-newline.csv": b"1,2,3,4,5,6,7,8", + "other.txt": b"0.5\n", + } + for name, data in foreign.items(): + (self.out / name).write_bytes(data) + (self.out / "empty.json.gz").write_bytes(gzip.compress(b"[]")) + (self.out / "other-keys.json.gz").write_bytes(gzip.compress(b'[{"id":"a","x":[1]}]')) + (self.out / "scalar.json.gz").write_bytes(gzip.compress(b'[{"id":"a","embedding":1}]')) + torch.save({"a": torch.rand(3)}, self.out / "dict.pt") + torch.save(torch.arange(6, dtype=torch.int64), self.out / "int.pt") + torch.save(torch.zeros(3, 9, dtype=torch.uint8), self.out / "uint8-9.pt") + torch.save(torch.rand(3), self.out / "legacy.pt", _use_new_zipfile_serialization=False) + pq.write_table(pa.table({"id": ["a"], "embedding": [["x"]]}), self.out / "strings.parquet") + pq.write_table(pa.table({"id": ["a"], "vector": [[0.5]]}), self.out / "vector.parquet") + pq.write_table(pa.table({"id": ["a"], "embedding": [0.5]}), self.out / "scalar.parquet") + for path in sorted(self.out.iterdir()): + with self.subTest(file=path.name): + self.assertIsNone(file_kind(path)) + # A folder named like a file, a missing file, a link to a file is the file + (self.out / "folder.pt").mkdir() + self.assertIsNone(file_kind(self.out / "folder.pt")) + self.assertIsNone(file_kind(self.out / "missing.csv")) + torch.save(torch.rand(3), self.out / "e.pt") + (self.out / "link.pt").symlink_to(self.out / "e.pt") + self.assertEqual(file_kind(self.out / "link.pt"), "embeddings") + + def test_the_suffix_in_any_case(self): + """A case-insensitive folder takes X.PT for the file x.pt that a run would write: read as x.pt is.""" + from foldmatch.utils.file_kinds import EMBEDDINGS, RESIDUE_CODES, describe, file_kind + for path, kind in self.written().items(): + if kind is None: + continue + upper = path.with_name(path.name.upper()) + with self.subTest(file=f"{path.parent.name}/{upper.name}"): + shutil.copyfile(path, path.parent / f"x-{upper.name}") + self.assertEqual(file_kind(path.parent / f"x-{upper.name}"), kind) + self.assertEqual(describe("A.PT", EMBEDDINGS), "an embedding") + self.assertEqual(describe("A.Csv", EMBEDDINGS), "an embedding") + self.assertEqual(describe("A.PARQUET", EMBEDDINGS), "embeddings") + self.assertEqual(describe("A.PT", RESIDUE_CODES), "residue codes") + + def test_a_file_is_read_once_from_one_check_to_the_next(self): + """Within remembering() a file's kind is remembered while the file is unchanged; the next check, outside it, + takes it from memory once and forgets it; nothing is remembered outside it, or after a refusal within it.""" + from foldmatch.utils import file_kinds + path = self.out / "x.pt" + torch.save(torch.rand(3), path) + file_kinds._KNOWN.clear() + with unittest.mock.patch.dict(file_kinds._KINDS, {".pt": unittest.mock.Mock(wraps=file_kinds.pt_kind)}): + read = file_kinds._KINDS[".pt"] + # Outside remembering(): each check reads, nothing is kept + self.assertEqual(file_kinds.file_kind(path), file_kinds.EMBEDDINGS) + self.assertEqual(file_kinds.file_kind(path), file_kinds.EMBEDDINGS) + self.assertEqual((read.call_count, file_kinds._KNOWN), (2, {})) + # The command's check, then the library's: one read + with file_kinds.remembering(): + self.assertEqual(file_kinds.file_kind(path), file_kinds.EMBEDDINGS) + self.assertEqual(file_kinds.file_kind(path), file_kinds.EMBEDDINGS) + self.assertEqual(read.call_count, 3) + self.assertEqual(file_kinds.file_kind(path), file_kinds.EMBEDDINGS) + self.assertEqual((read.call_count, file_kinds._KNOWN), (3, {})) + self.assertEqual(file_kinds.file_kind(path), file_kinds.EMBEDDINGS) + self.assertEqual(read.call_count, 4) + # A file changed since it was remembered is read again + with file_kinds.remembering(): + self.assertEqual(file_kinds.file_kind(path), file_kinds.EMBEDDINGS) + torch.save(torch.zeros(4, 8, dtype=torch.uint8), path) + self.assertEqual(file_kinds.file_kind(path), file_kinds.RESIDUE_CODES) + self.assertEqual(read.call_count, 6) + # A check that refuses ends the run: what it read is forgotten + file_kinds._KNOWN.clear() + with self.assertRaises(RuntimeError), file_kinds.remembering(): + file_kinds.file_kind(path) + raise RuntimeError("refused") + self.assertEqual(file_kinds._KNOWN, {}) + + +if __name__ == "__main__": + unittest.main()