Skip to content

perf(step2): a faster commit walk with the same output - #110

Merged
EllianCarlos merged 9 commits into
masterfrom
perf/step2-serial-path
Nov 13, 2026
Merged

EllianCarlos merged 9 commits into
masterfrom
perf/step2-serial-path

Conversation

@EllianCarlos

Copy link
Copy Markdown
Collaborator

What this changes

Step 2 (tokenize: the commit walk of blobExec) is the longest step of the pipeline. This PR makes it faster with five changes. Each one has an opt-out.

commit change opt-out
2b89b3c dst borrows src's objects (objects/info/alternates) during the walk; then pack-refs, repack -a -d, the alternates file is set aside, fsck --connectivity-only, and it is deleted. A failure restores the file, writes PACK-FAILED and stops; a resume packs first. --copy-objects or CREGIT_COPY_OBJECTS=1
c6afc4b up to 256 commits (or 2 s) per SQLite transaction; cache 256 MiB, mmap 4 GiB CREGIT_BLOBEXEC_OPTS="--commits-per-transaction=1 --sqlite-cache-mb=0 --sqlite-mmap-mb=0"
afcb15a dst's loose objects at zlib level 1 --loose-compression=-1
d6b168f JGit uses the JDK SHA-1, not its collision-detecting one CREGIT_BLOBEXEC_JAVA_OPTS=-Dorg.eclipse.jgit.util.sha1.implementation=java
0e1e5f5, 1a392cd no bitmap index when dst is packed CREGIT_WRITE_BITMAPS=1

1919a3b and 9dde4ae pass CREGIT_BLOBEXEC_OPTS and CREGIT_BLOBEXEC_JAVA_OPTS to blobExec (split without globbing, and logged). d063114 adds tests: aggressive gc, a killed walk, a lost transaction group.

Evidence

  • The full pipeline (steps 1-10, --skip-html) gives the same outputs as the base on curl, starlark-rust, libexpat and psi-probe: refs, all reachable objects, commit_map, blob_map, tree_map, ref_map, meta, blame/ and the Parquet sha256.
  • curl: step 2 CPU 1,418 s to 871 s (-39%), wall 1,135 s to 921 s (-19%). blobExec CPU: curl 1,287 to 771 s, psi-probe 55 to 34 s, starlark 70 to 55 s, libexpat 51 to 29 s.
  • A walk killed mid-run (libexpat) and a kill during the repack (starlark) both resume to equal output.
  • Tests: sbt 201 in 14 suites, 0 failed; the shell suites (tokenize_gate 50, pipeline_workdir_guard 79, ensure_artifacts 42, retokenize_passthrough 34, reblame_passthrough 15, pack_alternates 44, test_token_worker) pass; prove tests/t 239 pass; pytest 82 pass.
  • The census runs batches 2-5 on EC2 with this code (merged with the srcML fix); LineageOS step 2 took 9.6 h there.

Behaviour changes

  • The JDK SHA-1 does not detect crafted collisions (SHAttered-type) as JGit's does.
  • A failed pack now stops the run (resumable, PACK-FAILED); a failed gc used to only warn.
  • The pack is larger (libexpat +21%, curl 95 to 103 MB), because repack reuses src's looser deltas; no .bitmap unless CREGIT_WRITE_BITMAPS=1.

Step 2 copied every untokenized original blob into the cregit repo one
by one: inflate, hash, deflate, write a loose file. On psi-probe that was
half of the consumer thread's time.

blobExec --alternates now writes dst/objects/info/alternates naming
src's objects before the walk, and copies no original blob. Trees that
equal one of src's are not written either. The runner passes it by
default. The pack after step 2 then runs git repack -a -d without -l,
which copies the borrowed objects in once, sets the alternates file
aside, proves dst complete with git fsck --connectivity-only, and only
then deletes it. prune and commit-graph follow, as gc does.

While dst borrows, the pack is required, not optional. A failed pack
puts the file back (dst stays readable through src), writes a
PACK-FAILED keep marker and stops the run. Any resume past step 2
packs first, also after a kill during fsck.

Default on, because the objects, refs and mapping rows are the same.
Opt out with --copy-objects or CREGIT_COPY_OBJECTS=1; ctp.py has no
pass-through for runner flags, but the census inherits the environment.
--gc none and --mode sharded always copy.
Step 2's mapping DB ran one transaction per commit with SQLite's 2 MB
page cache and no mmap. On psi-probe the per-commit transaction was 28%
of the consumer thread and lookups 79% of the producer; on foxhound the
JVM made 7.0e9 page reads against a 2.5 GB DB.

Mapping.inGroupedTx lets up to 256 commits (or 2 s of them) share one
transaction. A group is all-or-nothing like the transaction it
replaces: a commit's rows are one body, a throw in a body rolls the
group back, inTx and close commit an open group first, and the walk
commits it when it stops, by abort too. So a commit row is still never
durable without its trees and blobs. A crash or a watchdog halt loses
at most one group, which resume re-walks with tokens from the memo.

Mapping.open also sets cache_size to 256 MiB and mmap_size to 4 GiB.
The cache is private memory that fills only as pages are read, so at
most the DB's size: up to 768 MiB for the census's three runs. The map
reads the kernel's page cache in place; those are shared, reclaimable
file pages, the ones plain reads cache anyway, so it adds no memory.

Defaults on in blobExec, so the runner and ctp.py need no change.
Opt out with --commits-per-transaction=1, --sqlite-cache-mb=0 and
--sqlite-mmap-mb=0. Only speed changes: rows and order are the same.
Every object step 2 writes is a loose file that lives only until the
pack after step 2, which re-deflates it at git's pack.compression. So
deflating it at JGit's default level 6 is wasted work: the profile
measured -11% consumer CPU on psi-probe with level 1.

blobExec sets core.compression=1 on dst's config in memory only. The
config file is not written, so git repack and steps 3+ see the same
config as before, and pack.compression needs no override. Object ids
hash the content, not the deflate stream, so nothing in the output can
depend on the level. If JGit reloaded the file mid-walk, the only effect
would be the default level again.

Default on in blobExec; opt out with --loose-compression=-1.
JGit's own Java SHA-1 also runs collision detection (SHA1DC) on every
object it hashes. The profile measured -5% with the JDK's intrinsic
SHA-1 (org.eclipse.jgit.util.sha1.implementation=jdkNative).

Main sets that property at start, unless the operator set it. JGit
reads it for every new hasher. Both give the same id for every input
that is not a crafted SHA-1 collision; on one of those the Java hasher
throws, the JDK one does not. A test checks the ids agree, including
git's empty blob.

Default on; opt out with java -Dorg.eclipse.jgit.util.sha1.implementation=java.
git writes a reachability bitmap by default when it packs a bare repo.
It serves fetches and clones over a transport, and nothing does either
with the cregit repo: step 6 clones it locally, which hard-links the
pack. On Linux the bitmap cost 1.4 h of the 14.4 h gc.

Both the gc (--gc plain/aggressive) and the repack that ends borrowing
now run with -c repack.writeBitmaps=false. The pack's objects are the
same; only the .bitmap file is gone. The shell test checks both paths.

Always on: there is nothing to opt out of.
The step-2 speed-ups in blobExec are its defaults, and ctp.py has no
pass-through for runner flags. So a run that must opt out of one had
no way to: the census calls the runner through ctp.py, and only the
environment reaches it.

CREGIT_BLOBEXEC_OPTS now adds blobExec flags (for example
--commits-per-transaction=1 --loose-compression=-1) and
CREGIT_BLOBEXEC_JAVA_OPTS adds JVM flags before -jar (for example
-Dorg.eclipse.jgit.util.sha1.implementation=java, or
-XX:ActiveProcessorCount=3 to size the tokenizer pool). Both are empty
by default, so the command line is unchanged.
The borrowing path had no test for --gc aggressive, for a walk killed
mid-way and resumed at step 2, or for a crash that loses the open
transaction group while dst already holds that group's objects.

pack_alternates.sh gains two cases: --gc aggressive borrows and its
repack leaves dst whole with no bitmap; a SIGKILLed walk leaves dst
borrowing, and a step-2 resume packs it and a step-6 clone reads it.
AlternatesSpec rolls the DB back to the last committed group after the
walk wrote the later commits' objects, drops the refs, resumes, and
checks objects, refs and mapping rows against an uninterrupted walk.
…g it

CREGIT_BLOBEXEC_OPTS and CREGIT_BLOBEXEC_JAVA_OPTS were expanded
unquoted, so a * in a flag could turn into file names from the
runner's directory. And a value exported by mistake changed every run
with no line in the pipeline log to show it.

Both are now read into arrays with read -ra, which splits on blanks
and never globs, and each is logged when set. Unset, the command line
is the same as before.
The pack of the cregit repo writes no bitmap index, and that was the
one step-2 speed-up with no opt-out. CREGIT_WRITE_BITMAPS=1 now drops
the -c repack.writeBitmaps=false from both the borrowing repack and
git gc, so git writes its default again. ctp.py passes the
environment through, so the census can use it without a new flag.

The README's shell test loop also lists pack_alternates, as CI does.
@EllianCarlos

Copy link
Copy Markdown
Collaborator Author

Review of 2026-10-28 (tool review of the token dataset build). The dataset build ran this branch (1a392cd) as a parent of 95cdc88 and of all later run commits. These items are for this PR; none of them changed a file of the dataset (the variables were not set in any run kit).

  • CREGIT_BLOBEXEC_OPTS and JAVA_OPTS accept any flag (run_pipeline_process.sh:977-984), also flags that change the output or load another jar, and only pipeline.log records them. Allow-list the flags and write them to the run record.
  • dissolve_alternates (run_pipeline_process.sh:181-218) keeps a stale absolute path after the work folder moves. Rewrite the alternates path before the repack.
  • No committed end-to-end check proves "the same output". Add the gate hashes (the libexpat gate Parquet before and after) to the PR and to a test.

Tests that pass on this branch: tests/pack_alternates.sh 44 of 44; a shell mutant of the pass-through split is killed.

1a392cd is a parent of the run commit 95cdc88. The tag dataset-build/cregit-95cdc88 keeps it on GitHub, so a squash of this PR is safe for provenance. The PR of the blame and trailer fixes starts from master and does not need this PR.

@EllianCarlos
EllianCarlos merged commit bc00a45 into master Nov 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant