Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
118 commits
Select commit Hold shift + click to select a range
0156cc3
esm alignment resources
bioinsilico Oct 12, 2026
384fefa
G1: Pin the ESM3 snapshot and verify its weights
bioinsilico Oct 13, 2026
c88238e
G2: Explain max_evalue with an empty or unusable database size
bioinsilico Oct 13, 2026
632e174
G3: Vectorised identity from _derive_alignment_extras
bioinsilico Oct 13, 2026
d253a7b
G4: Writers tolerate 3- and 4-tuple predictions
bioinsilico Oct 13, 2026
d6a555a
G5: weights_only=True in every torch.load
bioinsilico Oct 13, 2026
a9d6f0c
G6: SequenceStore meta table with entry_count, total_residues, duplic…
bioinsilico Oct 13, 2026
840e41a
G7: Shared run id for multi-rank passes; shard isolation and count check
bioinsilico Oct 13, 2026
75c7aff
G8: normalize_sequence() for every FASTA input path
bioinsilico Oct 13, 2026
b1a7ee9
G9: Read-only FASTA pre-flight on every rank before ESM3
bioinsilico Oct 13, 2026
9f83798
G10: --min-coverage help says both qcov and tcov must reach the value
bioinsilico Oct 13, 2026
b9cdc41
G11: ReleasePinTest - every Hugging Face revision is a 40-hex commit
bioinsilico Oct 13, 2026
390e175
G12: Pin the BLOSUM Stage-2 pool start method to the platform default
bioinsilico Oct 13, 2026
37523e8
G13: ESM3 geometric attention off for sequence-only inputs
bioinsilico Oct 13, 2026
6c6f865
G14: Apple GPU (MPS) support in the aggregator and ESM3 modules
bioinsilico Oct 13, 2026
6f15f90
P1: CHANGELOG
bioinsilico Oct 13, 2026
4ee8e0e
P2.2: ESM3 residue-code encoder module
bioinsilico Oct 13, 2026
78c7b6f
P3.1: fmsw native Smith-Waterman kernel
bioinsilico Oct 13, 2026
14ee4d2
P3.3-P3.4: swkern ctypes wrapper and dispatch
bioinsilico Oct 13, 2026
a008a7c
P3.5-P3.6: kernel tests and reference oracle
bioinsilico Oct 13, 2026
fcfcb1a
P3.2: hatchling build hook and packaging
bioinsilico Oct 13, 2026
0861cb5
P3.7: CI jobs for the kernel, packaging and wheels
bioinsilico Oct 13, 2026
5de74d9
P2-P3: CHANGELOG
bioinsilico Oct 13, 2026
d6b2419
P4.1: EsmCodesSystem registry and digests
bioinsilico Oct 14, 2026
c652ae5
P4.2: E-value statistics
bioinsilico Oct 14, 2026
8ff2825
P4.3: bundle loader, prepare() and download-models
bioinsilico Oct 14, 2026
7da2cb5
P4.4: frozen registry entry esm-codes-v1 and goldens
bioinsilico Oct 14, 2026
e8b389c
P4: CHANGELOG
bioinsilico Oct 14, 2026
88655cd
P5.1-P5.2: residue-code module hook and inference plumbing
bioinsilico Oct 14, 2026
39b3ad0
P5.3: Parquet writer residue columns
bioinsilico Oct 14, 2026
602075e
P5.4: residue-code and query-encoding readers
bioinsilico Oct 14, 2026
d6babca
P5.5: residue-code store
bioinsilico Oct 14, 2026
44a7b53
P5.6: build sequences --residue-codes
bioinsilico Oct 14, 2026
77510ff
P5.7: build residue-codes --from-shards
bioinsilico Oct 14, 2026
8e15c79
P5: CHANGELOG
bioinsilico Oct 14, 2026
9da4af6
P6.0: download-models downloads only; fm-search download-models
bioinsilico Oct 19, 2026
5fc1c23
P6.4: Stage-2 code aligner
bioinsilico Oct 19, 2026
34144c1
P6.5: esm-codes output fields
bioinsilico Oct 19, 2026
066b239
P6.1-P6.2: query sequences --scoring and resolution before ESM3
bioinsilico Oct 19, 2026
0f67636
P6.3: query encodings in the query pass
bioinsilico Oct 19, 2026
e93c13b
P6: CHANGELOG
bioinsilico Oct 19, 2026
d2a745a
P7.0a: multi-rank query output in FASTA order
bioinsilico Oct 20, 2026
ff505fc
P7.0b: refuse --comp-bias-corr-scale and --significance-sample-size u…
bioinsilico Oct 20, 2026
9edd1d3
P7.0c: validate --format-output before ESM3 for every scoring
bioinsilico Oct 20, 2026
4056036
P7.0d: declare threadpoolctl
bioinsilico Oct 20, 2026
37408c2
P7.0e: Mac TMPDIR in the residue-codes CLI test
bioinsilico Oct 20, 2026
c714d64
P7.1: Stage-2 benchmark and profiling
bioinsilico Oct 20, 2026
2b862b7
P7.2a: swkern lays out a call's blocks once
bioinsilico Oct 20, 2026
81c300d
P7.2b: round 2 counts distinct codes only up to the S13 minimum
bioinsilico Oct 20, 2026
3a5ffe8
P7.2c: leaner per-hit objects in the parent
bioinsilico Oct 20, 2026
78d19f2
P7.2d: size the Stage-2 pool by the work, not by its first batch
bioinsilico Oct 20, 2026
d4912af
P7.2e: Stage 2 as a dataflow: per-query round 2 and hits, overlapped …
bioinsilico Oct 20, 2026
70e345c
P7.2f: one array lookup of a batch's candidate code rows
bioinsilico Oct 20, 2026
d2be21e
P7.2g: kernel workspace arrays on 64-byte boundaries
bioinsilico Oct 20, 2026
8d74fbf
P7.2h: benchmark the default pool and the persistent-pool sizing
bioinsilico Oct 20, 2026
870ac59
P7.2i: one sequence fetch per target per batch
bioinsilico Oct 20, 2026
dab5778
P7.2j: default Stage-2 pool: one worker per physical core
bioinsilico Oct 20, 2026
bcae26d
P7.2k: round-2 workers return the path metrics, not the paths
bioinsilico Oct 20, 2026
8b97f65
P7.2l: a candidate listed twice keeps its order in any scheduling
bioinsilico Oct 20, 2026
c07335f
P7.2m: a Stage-2 worker releases its kernel buffers after a long pair
bioinsilico Oct 20, 2026
cc55c01
P7: CHANGELOG
bioinsilico Oct 20, 2026
29b8991
P8b.1: phases of build sequences as functions; index-phase logging
bioinsilico Oct 22, 2026
14572a5
P8b.2: run record module and single-process check
bioinsilico Oct 22, 2026
d932fec
P8b.3: build sequences-embed
bioinsilico Oct 22, 2026
50abe42
P8b.4: build sequences-index
bioinsilico Oct 22, 2026
68c874c
P8b.5: build sequences-seal
bioinsilico Oct 22, 2026
7213228
P8b.6: CHANGELOG
bioinsilico Oct 22, 2026
f07ccbf
P9.F1: uv.lock in sync with pyproject.toml
bioinsilico Oct 26, 2026
94e99bd
P9.F2: faiss-cpu 1.15.1
bioinsilico Oct 26, 2026
3dd9d38
P9.F3: index-phase threads with the faiss-cpu 1.15.1 wheel
bioinsilico Oct 26, 2026
1b62954
P9.F4: README and CHANGELOG for faiss-cpu 1.15.1
bioinsilico Oct 26, 2026
b8371db
P9.F5: the esm-codes in-process BLAS limit keeps FAISS's threads
bioinsilico Oct 26, 2026
0176141
P9.I1: step 2 and the seal refuse I2 and I3 before Lightning is imported
bioinsilico Oct 27, 2026
226e0ef
P9.I2: the launch checks' advice ends with the command to run
bioinsilico Oct 27, 2026
789a639
P9.I3: CHANGELOG for the launch checks of step 2 and the seal
bioinsilico Oct 27, 2026
62b5fc7
P9.L1: --log-level warn sets up the WARNING logging
bioinsilico Oct 29, 2026
a353dfa
P9.L2: step 2 and the seal refuse before Lightning is imported
bioinsilico Oct 29, 2026
8f2d5a4
P9.L3: CHANGELOG for --log-level warn and the later refusals of step 2
bioinsilico Oct 29, 2026
c05aaac
P9.M1: the launch checks' command on a line of its own
bioinsilico Oct 29, 2026
de32bc6
P9.A1: each assembly of an entry is read as itself
bioinsilico Oct 30, 2026
57cc907
P9.B1: a chain or assembly embedding saved as CSV is one vector
bioinsilico Oct 30, 2026
711855d
P9.B2: a structure name names one file or URL
bioinsilico Oct 30, 2026
9f68540
P9.B3: a .pt or .csv embedding file holds one vector
bioinsilico Oct 30, 2026
4d42830
P9.B4: one vector per Parquet row, one dimension per database
bioinsilico Oct 30, 2026
562c718
P9.B5: residue embeddings in Parquet keep their shape
bioinsilico Oct 31, 2026
a3e5d8b
P9.A2: each copy of a chain in an assembly is read as itself
bioinsilico Oct 31, 2026
6d77bf8
P9.A3: a chain split into pieces is read whole, and once
bioinsilico Nov 1, 2026
2f68f4c
P9.S1: integer Smith-Waterman fmswi_align in the fmsw library
bioinsilico Oct 31, 2026
4a91a9a
P9.S2: swint, the ctypes wrapper of fmswi_align
bioinsilico Oct 31, 2026
1c3ca31
P9.S3: the BLOSUM62 Stage 2 aligns on the native kernel only
bioinsilico Oct 31, 2026
6752f37
P9.S4: tests of the kernel and of the Stage 2 against biotite
bioinsilico Oct 31, 2026
7a89928
P9.S5: a biotite-free golden digest of fmswi in the kernel selftest
bioinsilico Oct 31, 2026
ea98047
P9.S6: query sequences and query db check the BLOSUM62 Stage 2 before…
bioinsilico Oct 31, 2026
feea60e
P9.S7: no message offers BLOSUM62 as a way round a missing kernel
bioinsilico Oct 31, 2026
d1a5041
P9.S8: the fmswi golden set aligns as biotite aligns it
bioinsilico Oct 31, 2026
c0a2eaf
P9.S9: README and CHANGELOG for the native BLOSUM62 Stage 2
bioinsilico Oct 31, 2026
9763f44
P9.A4: max_res, --min-res-n and chain-list names, the P9.A3 follow-ups
bioinsilico Nov 1, 2026
8ded017
P9.A5: assembly lists refuse a row of more than 4 fields
bioinsilico Nov 2, 2026
0fc382f
P9.A6: structure and residue-file lists refuse too many fields
bioinsilico Nov 2, 2026
e47909b
P9.H1: the help of inference and fm-search says what they do
bioinsilico Nov 3, 2026
611225f
P9.H2: chain-embedding drops --structure-format; assembly file name
bioinsilico Nov 3, 2026
b1ce16c
P9.N1: inference refuses names that conflict in its files
bioinsilico Nov 5, 2026
d685ba1
P9.N2: fm-embedding refuses FASTA ids that conflict in its files
bioinsilico Nov 5, 2026
82d8271
P9.N3: fm-search refuses an output file that is one of its inputs
bioinsilico Nov 5, 2026
875b514
P9.G1: tests read plain CLI text whatever the environment
bioinsilico Nov 6, 2026
a9d877c
P9.R1: README drops the inference command
bioinsilico Nov 6, 2026
a958532
P9.K1: ivf_pq batch searches quantize outside OpenMP regions
bioinsilico Nov 3, 2026
10b9a9b
P9.K2: cluster and similarity-graph run on ivf_pq, in batches
bioinsilico Nov 6, 2026
0c642f8
P9.Q1: the similarity graph is built with less memory per edge
bioinsilico Nov 8, 2026
5610561
P9.T1: cluster --method greedy, a streaming clustering mode
bioinsilico Nov 9, 2026
6824f15
P9.U1: cluster's Leiden runs on python-igraph's community_leiden
bioinsilico Nov 10, 2026
2edb7d5
CHANGELOG: record releases 0.9.0 and 0.10.0
bioinsilico Nov 11, 2026
9c00ecb
CI: run the macOS jobs on macos-15
bioinsilico Nov 11, 2026
4618d2f
CI: pin click 8.3.1; tests free of click's wording
bioinsilico Nov 11, 2026
753d328
CI: tell the build lock apart by a lock_id, not its inode
bioinsilico Nov 11, 2026
e8b84f7
CI: name an assembly list under /dev as given
bioinsilico Nov 11, 2026
700186e
CI: repeated vectors in the hnsw weight-sharing fixture
bioinsilico Nov 11, 2026
1b860f3
README doc fix
bioinsilico Nov 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@
# Gitignored directories
node_modules/

# Native kernel libraries built in place (the image compiles its own)
/src/foldmatch/_native/_fmsw_*

# Directories
/dist/
/tests/
Expand Down
240 changes: 237 additions & 3 deletions .github/workflows/publish.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ jobs:
name: Test on Python ${{ matrix.python-version }}
runs-on: ["self-hosted", "buildchain"]
timeout-minutes: 120
env:
# The editable install compiles the native kernel in place (hatch_build.py); a failed compile is an error here,
# never a pure-Python install or skipped kernel tests.
FOLDMATCH_REQUIRE_NATIVE: "1"
strategy:
matrix:
python-version: ["3.12"]
Expand All @@ -26,6 +30,18 @@ jobs:
sudo apt-get update
sudo apt-get install -y build-essential pkg-config libzstd-dev

# fmsw.c needs GCC >= 12 or Clang >= 14 (older GCC stops at its #error). The self-hosted runner's gcc is not
# pinned, so an older one is replaced by gcc-12 for this job; CC reaches the editable install's build hook.
- name: C compiler for the native kernel (GCC >= 12)
run: |
cc_bin=gcc
if [ "$(gcc -dumpversion | cut -d. -f1)" -lt 12 ]; then
sudo apt-get install -y gcc-12
cc_bin=gcc-12
fi
"$cc_bin" --version
echo "CC=$cc_bin" >> "$GITHUB_ENV"

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
Expand All @@ -37,10 +53,142 @@ jobs:
- name: Run tests
run: hatch test

# Native Smith-Waterman kernel (PLAN section 7, P3.7) on every supported compiler and ISA. The DP golden hash (from
# q fp16 through the fixed-order reference similarity) must be identical in every job: tests/test_swkern.py checks
# it for every ISA library of the machine. The production BLAS path (OpenBLAS on Linux, Accelerate on macOS) must
# stay within the S16 tolerance of it: the test module checks the dispatched ISA, selftest() every other one.
kernel:
name: ${{ matrix.job }} (${{ matrix.cc }})
runs-on: ${{ matrix.os }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include:
- { job: kernel-x86, os: ubuntu-latest, cc: gcc, isas: "sse2 avx2 generic" }
- { job: kernel-x86, os: ubuntu-latest, cc: clang, isas: "sse2 avx2 generic" }
- { job: kernel-arm64-linux, os: ubuntu-24.04-arm, cc: gcc, isas: "neon generic" }
- { job: kernel-macos, os: macos-15, cc: cc, isas: "neon generic" }
env:
CC: ${{ matrix.cc }}
PYTHONPATH: src
FOLDMATCH_REQUIRE_NATIVE: "1"
FOLDMATCH_SLOW_TESTS: "1"
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Install test dependencies
run: pip install "numpy>=2,<3" pytest==9.0.2 threadpoolctl==3.6.0 importlib-metadata==8.7.1

- name: Compiler version
run: $CC --version

- name: Require AVX2 on the x86 runner (the avx2 library must run, not be skipped)
if: matrix.job == 'kernel-x86'
run: grep -qw avx2 /proc/cpuinfo

- name: Build every ISA library of this machine
run: python -m foldmatch._native.build

- name: Kernel tests
run: python -m pytest tests/test_swkern.py -rs

# The esm-codes registry golden digests (plan P4 acceptance: identical on Linux and macOS). The test module needs
# only the standard library and foldmatch.search.scoring_system, so it runs on every kernel runner.
- name: Scoring-system golden digests
run: python -m pytest tests/test_scoring_system_golden.py -rs

- name: selftest() for each ISA
run: |
for isa in ${{ matrix.isas }}; do
FOLDMATCH_SW_ISA=$isa python -m foldmatch.search.swkern
done

# The kernel test module against ASan + UBSan builds of the sse2, avx2 and generic libraries (undersized workspaces
# and bad arguments included). The sanitizer runtime is preloaded because python itself is not instrumented.
kernel-sanitize:
name: kernel-sanitize (ASan + UBSan, gcc)
runs-on: ubuntu-latest
timeout-minutes: 30
env:
CC: gcc
PYTHONPATH: src
FOLDMATCH_REQUIRE_NATIVE: "1"
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Install test dependencies
run: pip install "numpy>=2,<3" pytest==9.0.2 threadpoolctl==3.6.0 importlib-metadata==8.7.1

- name: Build the libraries with ASan and UBSan
run: |
python - <<'EOF'
from foldmatch._native import build
build.build(extra_flags=["-fsanitize=address,undefined", "-fno-sanitize-recover=all",
"-fno-omit-frame-pointer", "-g"])
EOF

- name: Kernel tests under the sanitizers
run: |
export LD_PRELOAD="$(gcc -print-file-name=libasan.so)"
export ASAN_OPTIONS=detect_leaks=0:abort_on_error=1
export UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1
python -m pytest tests/test_swkern.py -rs

# Every PR: the sdist, a wheel built from it, the wheel and sdist content tests, selftest() from the wheel in a clean
# venv, and the build-hook tests (pure-wheel fallback, FOLDMATCH_REQUIRE_NATIVE, editable build, platform tag).
packaging:
name: packaging (${{ matrix.os }})
runs-on: ${{ matrix.os }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-15]
env:
FOLDMATCH_REQUIRE_NATIVE: "1"
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Install build and test tools
run: >-
pip install build hatchling editables packaging "numpy>=2,<3" pytest==9.0.2
importlib-metadata==8.7.1

- name: Build the sdist, then the wheel from the sdist
run: python -m build --outdir dist .

- name: Packaging tests
env:
FOLDMATCH_DIST_DIR: dist
PYTHONPATH: src
run: python -m pytest tests/test_packaging.py -rs

hatch-build:
name: Build to PyPI
name: Build the sdist
needs:
- hatch-test
- kernel
- kernel-sanitize
- packaging
runs-on: ubuntu-latest
if: github.event_name == 'release'
steps:
Expand All @@ -55,18 +203,98 @@ jobs:
- name: Install Hatch
run: pip install hatch

# Wheels come from the cibuildwheel matrix below: a plain `hatch build` wheel would carry a linux_x86_64 tag,
# which PyPI refuses.
- name: Build distribution
run: hatch build
run: hatch build -t sdist

- name: Store the distribution packages
uses: actions/upload-artifact@v4
with:
name: python-package-distributions
path: dist/

# Platform wheels (D19): Linux x86_64 and aarch64 in manylinux_2_28 images (auditwheel repairs and retags them),
# macOS arm64 and x86_64 (cross-built on the arm64 runner; delocate checks and retags them). The build hook tags the
# raw wheel py3-none-<platform>; no manylinux tag is hard-coded.
build-wheels:
name: Wheels (${{ matrix.name }})
needs:
- hatch-test
- kernel
- kernel-sanitize
- packaging
if: github.event_name == 'release'
runs-on: ${{ matrix.os }}
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
include:
- { name: linux-x86_64, os: ubuntu-latest, archs: x86_64 }
- { name: linux-aarch64, os: ubuntu-24.04-arm, archs: aarch64 }
- { name: macos, os: macos-15, archs: "arm64 x86_64" }
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Build wheels
uses: pypa/cibuildwheel@v3.1.4
env:
CIBW_BUILD: "cp312-*"
CIBW_SKIP: "*-musllinux_*"
CIBW_ARCHS: ${{ matrix.archs }}
CIBW_MANYLINUX_X86_64_IMAGE: manylinux_2_28
CIBW_MANYLINUX_AARCH64_IMAGE: manylinux_2_28
CIBW_ENVIRONMENT: FOLDMATCH_REQUIRE_NATIVE=1
# gcc = the image's gcc-toolset compiler (GCC >= 12), chosen explicitly over any system cc
CIBW_ENVIRONMENT_LINUX: FOLDMATCH_REQUIRE_NATIVE=1 CC=gcc
CIBW_ENVIRONMENT_MACOS: FOLDMATCH_REQUIRE_NATIVE=1 MACOSX_DEPLOYMENT_TARGET=11.0
with:
output-dir: wheelhouse

# The wheels are checked here, not by cibuildwheel's test step, which would install the full dependency set
# (torch, esm): the kernel check needs only numpy and importlib-metadata (imported by foldmatch/__init__), so the
# wheel goes into a clean venv with --no-deps that borrows those two.
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Wheel contents and selftest() in a clean venv
env:
FOLDMATCH_DIST_DIR: wheelhouse
run: |
pip install packaging "numpy>=2,<3" pytest==9.0.2 importlib-metadata==8.7.1
python -m pytest tests/test_packaging.py -rs -k "test_wheel_contents or test_wheel_selftest_in_clean_venv"

- name: Set up an x86_64 Python 3.12 (Rosetta)
if: runner.os == 'macOS'
id: python-x64
uses: actions/setup-python@v5
with:
python-version: "3.12"
architecture: x64

- name: selftest() of the macOS x86_64 wheel under Rosetta
if: runner.os == 'macOS'
env:
FOLDMATCH_DIST_DIR: wheelhouse
run: |
PY="${{ steps.python-x64.outputs.python-path }}"
"$PY" -m pip install packaging "numpy>=2,<3" pytest==9.0.2 importlib-metadata==8.7.1
"$PY" -m pytest tests/test_packaging.py -rs -k "test_wheel_selftest_in_clean_venv"

- name: Store the wheels
uses: actions/upload-artifact@v4
with:
name: wheels-${{ matrix.name }}
path: wheelhouse/*.whl

push-image:
needs:
- hatch-build
- build-wheels
name: Push image to harbor
uses: ./.github/workflows/_workflow-docker.yaml

Expand All @@ -84,10 +312,16 @@ jobs:
id-token: write

steps:
- name: Download all the dists
- name: Download the sdist
uses: actions/download-artifact@v4
with:
name: python-package-distributions
path: dist/
- name: Download the wheels
uses: actions/download-artifact@v4
with:
pattern: wheels-*
merge-multiple: true
path: dist/
- name: Publish distribution 📦 to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
7 changes: 6 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,9 @@
/.pypi.rc
__pycache__
/tests/resources/tmp
/.claude
/.claude
# native kernel libraries built in place (python -m foldmatch._native.build)
/src/foldmatch/_native/_fmsw_*
/src/foldmatch/_native/._fmsw_*
# cibuildwheel output
/wheelhouse/
Loading
Loading