Repository navigation
Conversation
`resolve_to_commit()` passes every revspec to `rev_parse_single()`, and gix treats a 40-hex id there as a prefix to disambiguate. A prefix lookup always ends in an object-store refresh: it takes the store's write lock and rescans the pack directory and alternates, whether the object was found or not. Commit-metadata extraction resolves full ids three times per commit and the range generators once more, so metadata extraction alone rescans about 57,000 times on the v7.3..v7.4-rc1 merge window. The next commit shares one store between the workers, which turns that write lock into a queue; it depends on this change. A revspec that parses as a full object id and names an object in the repository now resolves directly to that object, peeled to a commit; a tag id peels to its commit as before. Anything else, a missing id included, goes through `rev_parse_single()` unchanged. A full id of a tree or blob now fails with "'<id>' does not resolve to a commit" instead of gix's peel error. Indexing only the commits of that merge window (`semcode-index --commits`, 19,780 commits, 316 workers) took 15.1 s instead of 17.6 s and stored the same `git_commits` rows. `resolve_to_commit_takes_full_ids_directly` covers commit, tag, tree, missing, short and ancestry specs. `resolve_to_commit_full_id_does_not_refresh_the_store` reads the store's refresh counter: unchanged after resolving full ids, and moving for the same commits by short id. The git test helper moves to `git::test_git()`, which also clears `GIT_DIR`, `GIT_WORK_TREE` and `GIT_INDEX_FILE`, so a test run from a hook or `git rebase --exec` cannot touch the enclosing repository.
A `--git` run over the v7.3..v7.4-rc1 merge window peaks at 270 GiB VmRSS, and 249 GiB of that is the kernel's pack files counted once per thread. Every commit-metadata worker, range generator and tuple worker calls `gix::discover()`, and each discovered repository has an object store of its own that maps the pack and index files itself. VmRSS counts a page once per mapping, so 316 workers reading the same packs count their pages up to 316 times. The pages are shared page cache (Pss_File never exceeded 2.8 GiB), but /usr/bin/time, monitoring and the OOM killer's badness score all see the inflated number. Each pipeline now opens the repository once, converts it with `into_sync()`, and hands every thread a `to_thread_local()` handle on that one store: - `process_commits_pipeline()`: its metadata workers, whose loop moves into `commit_metadata_worker()`. `--commits` runs through it too. - `process_git_tuples_streaming()`: its generators and tuple workers. `TupleWorkerContext` carries the store instead of the repository path. - `process_git_tree_streaming()`: its tree walker and tuple workers. `process_git_range()` drops the repository it opened to list the range before the pipelines start, so it does not hold a third store. The merge window now peaks at 91.6 GiB instead of 269.6 GiB, in 690 s against 802 s, with file-backed RSS under 2.8 GiB. Indexing only its commits peaks at 9.7 GiB instead of 269.6 GiB, in 9.1 s against 17.6 s. (One run each on a 316-CPU host with other jobs running: load average 10 to 190 for these runs and the `--commits` baseline, 90 to 230 for the upstream merge-window run.) What remains is anonymous memory, mostly freed memory that glibc malloc keeps in its per-thread arenas. Rows indexed are the same in every table but `symbol_filename` and `call_edges`, which differ between runs of the unchanged code as well. The riskiest part is the state the threads now share. Their handles keep separate caches but share one slot map, read without locks, and one write lock, which an object-store refresh takes. Lookups that still refresh, such as a short id or a missing object, now queue on each other. Full commit ids stopped refreshing in the previous commit. With that change in place, indexing only the merge window's commits takes 9.1 s here against 15.1 s at the previous commit, where every worker still has a store of its own. A path that is not a repository now fails these pipelines up front with "Not in a git repository". Before, every thread logged that it could not open the repository and the run finished without reading anything. `process_lore_commits_pipeline()` keeps a store per worker; it was not measured here. Three tests hand the threads a store over a packed repository and check that the store's pack was opened and the right objects were read: - `commit_metadata_workers_read_through_the_store_they_are_handed`: the metadata workers. - `generators_and_workers_read_through_the_store_they_are_handed`: a range generator and four tuple workers. - `tree_walker_reads_through_the_store_it_is_handed`: the tree walker.
`process_git_range()` and `process_git_tree()` start one tuple worker per CPU, 316 on the measuring host. A worker that finishes a 512-file batch holds it, full function bodies included, until the depth-2 result channel takes it, so the worker count is how many parsed batches can wait ahead of the inserters. The v7.3..v7.4-rc1 merge window has 124 batches and a cold index of v7.3 about 127, both fewer than the workers, so `RESULT_CHANNEL_DEPTH` bounded nothing: in a heap profile of the merge window, parsed output waiting in workers was 11.4 GiB of a 12.9 GiB live heap. Both paths now start at most 8 tuple workers, so hosts with 8 or fewer CPUs are unaffected. Parsing parallelism does not depend on the count, because each worker parses its batch on the whole rayon pool. The merge window peaked at 83.1 GiB instead of 91.6 GiB with the previous commit alone, in 667 s against 690 s. A cold index of v7.3 with both commits peaks at 32.6 GiB against 79.3 GiB on upstream. The cap's own share of that was not isolated: an uncapped build with a shared store peaked at 33.6 GiB, within one-run noise. (One run each, load average 10 to 190 from other jobs for these runs, 90 to 230 for the upstream cold run.) Rows match as in the previous commit, except that both capped merge-window runs stored more `call_edges` rows: 3.95M and 3.90M, against 3.75M upstream and 3.81M without the cap. The per-path delete in `call_edges` (fixed in facebookexperimental#117) removes edges of every version of a path, so how many survive depends on the order batches land in; more rows means it removed fewer versions, not that edges were lost. Most of what remains is freed memory that glibc malloc keeps in per-thread arenas. In a local build with one shared store and jemalloc as the global allocator, which this change does not add, the same cap cut the merge-window peak from 20.5 to 13.6 GiB. `tuple_workers_follow_the_cpus_up_to_the_cap` checks the count at 0, 1, 7, 8, 9 and 316 CPUs.
meta-cla
Bot
added
the
CLA Signed
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR cuts the peak VmRSS of indexing kernel git history: on a
--gitrun over the v7.3..v7.4-rc1 merge window, 249 GiB of the 270 GiB peak was the kernel's pack files counted once per thread, because every thread's repository had an object store of its own. The second commit shares one object store across each git pipeline's threads, after the first stops full commit ids from refreshing that store under its write lock. The third caps tuple workers at 8, since each worker can hold a parsed batch waiting for the inserters. The merge-window peak drops to 91.6 GiB with the shared store and to 83.1 GiB with the cap, and a cold index of v7.3 peaks at 32.6 GiB against 79.3 GiB on upstream (one run each, on a 316-CPU host with other jobs running).The commits are best reviewed one at a time; their messages carry the detail.