Repository navigation
Fix the index sequence count on multiz MAFs (format 3) and fetch memory growth - #9
Merged
Merged
Conversation
In MAF mode, fetchByRegion kept one chunk dict per `s` line of a region just to compute the per-element summary columns at the end. A 69 Mb region of a 45-way bird MAF was OOM-killed at 6 GB, where v0.5.0 used 307 MB. ElementLociAccumulator now folds in each chunk as it arrives. A species keeps packed start/size/block arrays only while it is single-scaffold and single-strand. The classification and gap computation are unchanged, and summarizeElementLoci is now a wrapper over the accumulator. FASTA mode holds every chunk anyway, so it still summarizes once at the end, as before. - Worker peak RSS in MAF mode: 93 Mb region 1,531 -> 94 MB, 30 Mb 420 -> 56 MB, 10 Mb 113 -> 44 MB. - Outputs, summary included, are byte-identical to v0.8.1 on a 979,442-region BED in MAF and FASTA mode. Runtime is within noise. - tests/test_loci_accumulator.py checks it against a verbatim copy of v0.8.1's code on 12,000 random elements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the first release, the index's per-block sequence count (block.idx column 6, and the seq_lines/max_seqs header totals) included i, e and q lines. Cactus MAFs have none of these lines, but UCSC multiz MAFs do. On a 45-way multiz MAF, v0.8.1 reported max_seqs=89 and 633M sequence lines; the true values are 45 and 270M. index: - Counts only s lines, reference included. - Stops with one error, and writes no index, on an unrecognized line type, an s line before the first a line, or a block with no s line. - 5-10% faster on every input measured (84 GB multiz MAF: 323 -> 290 s). Memory is unchanged. - Gives the same output otherwise: on a 42.7 GB cactus MAF (uncompressed, gz, bgzip) the index bodies are byte-identical to v0.8.1. On the multiz MAF only column 6 changes, and it matches an independent s-line count for all 8.4M blocks. Index format 3: - New indexes say format=3, because column 6 changed meaning. - stats (unless --no-block-table) and info refuse an index of format 2 or lower, or one with no header, and say how to rebuild it. validate reports the format without changing its verdict. fetch and gc never read the count, so they still accept old indexes. Tests: - New tests/test_index_seq_lines.py, on tests/multiz45way-excerpt.maf (14 real blocks with 72 s, 58 i and 12 e lines). - The 13 committed test indexes were regenerated, with bodies confirmed identical, and their headers changed from format=2 to format=3. test_validate and test_info now expect format 3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Duplicate checks in stats: assessed, but deferred over the temporary-disk cost on very large MAFs. - fetch copies i/e/q lines through trimmed blocks unchanged, so a q line no longer matches the length of its trimmed s line. - The e-line status could tell an indel from missing data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes for v0.9.0.
Index counts only
slines (index format 3)seq_lines,max_seqs) also countedi/e/qlines. It was wrong on UCSC multiz MAFs and correct on cactus MAFs, which have none of these lines.max_seqs=89.format=3. Rebuild older indexes withmafutils indexto use them withstats(unless--no-block-table) orinfo.fetchandgcstill accept them.indexnow errors on unrecognized line types instead of miscounting them.indexis 5-10% faster. Apart from the corrected count, output is identical to v0.8.1.fetch: flat memory in MAF mode
Tests: 18 new, on a real multiz excerpt; the committed test indexes now say
format=3. All 298 pass.Details and measurements are in DEVELOPMENT.md.
🤖 Generated with Claude Code