Repository navigation
perf(step2): a faster commit walk with the same output - #110
Merged
Merged
Conversation
Step 2 copied every untokenized original blob into the cregit repo one by one: inflate, hash, deflate, write a loose file. On psi-probe that was half of the consumer thread's time. blobExec --alternates now writes dst/objects/info/alternates naming src's objects before the walk, and copies no original blob. Trees that equal one of src's are not written either. The runner passes it by default. The pack after step 2 then runs git repack -a -d without -l, which copies the borrowed objects in once, sets the alternates file aside, proves dst complete with git fsck --connectivity-only, and only then deletes it. prune and commit-graph follow, as gc does. While dst borrows, the pack is required, not optional. A failed pack puts the file back (dst stays readable through src), writes a PACK-FAILED keep marker and stops the run. Any resume past step 2 packs first, also after a kill during fsck. Default on, because the objects, refs and mapping rows are the same. Opt out with --copy-objects or CREGIT_COPY_OBJECTS=1; ctp.py has no pass-through for runner flags, but the census inherits the environment. --gc none and --mode sharded always copy.
Step 2's mapping DB ran one transaction per commit with SQLite's 2 MB page cache and no mmap. On psi-probe the per-commit transaction was 28% of the consumer thread and lookups 79% of the producer; on foxhound the JVM made 7.0e9 page reads against a 2.5 GB DB. Mapping.inGroupedTx lets up to 256 commits (or 2 s of them) share one transaction. A group is all-or-nothing like the transaction it replaces: a commit's rows are one body, a throw in a body rolls the group back, inTx and close commit an open group first, and the walk commits it when it stops, by abort too. So a commit row is still never durable without its trees and blobs. A crash or a watchdog halt loses at most one group, which resume re-walks with tokens from the memo. Mapping.open also sets cache_size to 256 MiB and mmap_size to 4 GiB. The cache is private memory that fills only as pages are read, so at most the DB's size: up to 768 MiB for the census's three runs. The map reads the kernel's page cache in place; those are shared, reclaimable file pages, the ones plain reads cache anyway, so it adds no memory. Defaults on in blobExec, so the runner and ctp.py need no change. Opt out with --commits-per-transaction=1, --sqlite-cache-mb=0 and --sqlite-mmap-mb=0. Only speed changes: rows and order are the same.
Every object step 2 writes is a loose file that lives only until the pack after step 2, which re-deflates it at git's pack.compression. So deflating it at JGit's default level 6 is wasted work: the profile measured -11% consumer CPU on psi-probe with level 1. blobExec sets core.compression=1 on dst's config in memory only. The config file is not written, so git repack and steps 3+ see the same config as before, and pack.compression needs no override. Object ids hash the content, not the deflate stream, so nothing in the output can depend on the level. If JGit reloaded the file mid-walk, the only effect would be the default level again. Default on in blobExec; opt out with --loose-compression=-1.
JGit's own Java SHA-1 also runs collision detection (SHA1DC) on every object it hashes. The profile measured -5% with the JDK's intrinsic SHA-1 (org.eclipse.jgit.util.sha1.implementation=jdkNative). Main sets that property at start, unless the operator set it. JGit reads it for every new hasher. Both give the same id for every input that is not a crafted SHA-1 collision; on one of those the Java hasher throws, the JDK one does not. A test checks the ids agree, including git's empty blob. Default on; opt out with java -Dorg.eclipse.jgit.util.sha1.implementation=java.
git writes a reachability bitmap by default when it packs a bare repo. It serves fetches and clones over a transport, and nothing does either with the cregit repo: step 6 clones it locally, which hard-links the pack. On Linux the bitmap cost 1.4 h of the 14.4 h gc. Both the gc (--gc plain/aggressive) and the repack that ends borrowing now run with -c repack.writeBitmaps=false. The pack's objects are the same; only the .bitmap file is gone. The shell test checks both paths. Always on: there is nothing to opt out of.
The step-2 speed-ups in blobExec are its defaults, and ctp.py has no pass-through for runner flags. So a run that must opt out of one had no way to: the census calls the runner through ctp.py, and only the environment reaches it. CREGIT_BLOBEXEC_OPTS now adds blobExec flags (for example --commits-per-transaction=1 --loose-compression=-1) and CREGIT_BLOBEXEC_JAVA_OPTS adds JVM flags before -jar (for example -Dorg.eclipse.jgit.util.sha1.implementation=java, or -XX:ActiveProcessorCount=3 to size the tokenizer pool). Both are empty by default, so the command line is unchanged.
The borrowing path had no test for --gc aggressive, for a walk killed mid-way and resumed at step 2, or for a crash that loses the open transaction group while dst already holds that group's objects. pack_alternates.sh gains two cases: --gc aggressive borrows and its repack leaves dst whole with no bitmap; a SIGKILLed walk leaves dst borrowing, and a step-2 resume packs it and a step-6 clone reads it. AlternatesSpec rolls the DB back to the last committed group after the walk wrote the later commits' objects, drops the refs, resumes, and checks objects, refs and mapping rows against an uninterrupted walk.
…g it CREGIT_BLOBEXEC_OPTS and CREGIT_BLOBEXEC_JAVA_OPTS were expanded unquoted, so a * in a flag could turn into file names from the runner's directory. And a value exported by mistake changed every run with no line in the pipeline log to show it. Both are now read into arrays with read -ra, which splits on blanks and never globs, and each is logged when set. Unset, the command line is the same as before.
The pack of the cregit repo writes no bitmap index, and that was the one step-2 speed-up with no opt-out. CREGIT_WRITE_BITMAPS=1 now drops the -c repack.writeBitmaps=false from both the borrowing repack and git gc, so git writes its default again. ctp.py passes the environment through, so the census can use it without a new flag. The README's shell test loop also lists pack_alternates, as CI does.
Collaborator
Author
|
Review of 2026-10-28 (tool review of the token dataset build). The dataset build ran this branch (1a392cd) as a parent of 95cdc88 and of all later run commits. These items are for this PR; none of them changed a file of the dataset (the variables were not set in any run kit).
Tests that pass on this branch: 1a392cd is a parent of the run commit 95cdc88. The tag |
1 of 21 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Step 2 (tokenize: the commit walk of blobExec) is the longest step of the pipeline. This PR makes it faster with five changes. Each one has an opt-out.
objects/info/alternates) during the walk; thenpack-refs,repack -a -d, the alternates file is set aside,fsck --connectivity-only, and it is deleted. A failure restores the file, writesPACK-FAILEDand stops; a resume packs first.--copy-objectsorCREGIT_COPY_OBJECTS=1CREGIT_BLOBEXEC_OPTS="--commits-per-transaction=1 --sqlite-cache-mb=0 --sqlite-mmap-mb=0"--loose-compression=-1CREGIT_BLOBEXEC_JAVA_OPTS=-Dorg.eclipse.jgit.util.sha1.implementation=javaCREGIT_WRITE_BITMAPS=11919a3b and 9dde4ae pass
CREGIT_BLOBEXEC_OPTSandCREGIT_BLOBEXEC_JAVA_OPTSto blobExec (split without globbing, and logged). d063114 adds tests: aggressive gc, a killed walk, a lost transaction group.Evidence
--skip-html) gives the same outputs as the base on curl, starlark-rust, libexpat and psi-probe: refs, all reachable objects,commit_map,blob_map,tree_map,ref_map,meta,blame/and the Parquet sha256.tokenize_gate50,pipeline_workdir_guard79,ensure_artifacts42,retokenize_passthrough34,reblame_passthrough15,pack_alternates44,test_token_worker) pass;prove tests/t239 pass; pytest 82 pass.Behaviour changes
PACK-FAILED); a failed gc used to only warn.repackreuses src's looser deltas; no.bitmapunlessCREGIT_WRITE_BITMAPS=1.