Repository navigation
perf(reader): the named-parameter read searches a record for each id it looks for, not every four-byte window (#1335) - #1502
Closed
DrunkOnJava wants to merge 10 commits into
Closed
DrunkOnJava wants to merge 10 commits into
DrunkOnJava wants to merge 10 commits into
Conversation
… each Family and FamilySymbol record (#1335)
…f each SWall and ArcWall graph (#1335)
… hex without a string per byte and reads the catalog's document without building what it ignores (#1335)
…reader, the probe imports them, and the index leaves are named fields (#1335)
…it looks for, not every four-byte window (#1335)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #1483 (its commits come first on this branch; this PR is the last one). The read of the parameters named for a property (
filtered_parameters_only) decides which of a Revit 2015 to 2023 file's records to decode by looking for the four bytes of each of a few definition ids in the record. It scanned every four-byte window of each record; it now searches each record for each id withmemchr::memmem, and keeps the window scan for more than 16 ids. The records it decodes are the same by construction: a window equals an id's low 32 bits exactly where the id's four little-endian bytes occur. Time, byte identity and peak memory on this head: not yet measured.Motivation
#1335's profile of the final head (run 42124471783, scratch head c85c73c, shares only): the named-parameter read is 3.4% of the exports' time over the 33 samples of 2015 to 2023 (57 ms of 1,646 on 2015
rac_advanced), most of it the byte scan over the 5,608 records it selects, for 6 ids.memchris already a normal dependency of the crate (Cargo.toml, used byreader.rs,partition_curve_fields.rsand others): nothing is added to the build graph.Changes
src/research/native_document.rs:IdNeedlesholds amemchr::memmem::Finderfor each id while there are at mostMAX_FINDERS(16) of them, andoccur_inis true where any finder finds its id in the body; with more ids, or none, it is the window scan as before (occur_in_windows).changelog.d/changed/1335-needle-scan.md.Result: no gain, so this should not land
Measure run 42130086398 (
-f ref=98566dd3 -f base=4140b96c -f samples=true -f sample_years="2015 2023" -f models="-" -f scorers=none -f timing=true; this head against #1483's measured head; both shards an EPYC 7763; median of three exports each):"Every head run below every base run", from the timing jobs' raw medians of three: 0 of 39 files; every head run above every base run on 8; the runs overlap on 31. The window scan it replaces was already fast enough that the search over six ids is not measurably quicker, and my estimate of about 1.5% of the export (from the profile's 57 ms for the named-parameter read on 2015
rac_advanced, most of it assumed to be the scan) was wrong: the rest of that read, the walk over 17,101 records and the selection of 5,608, is what it costs. Peak memory is the same (largest change: +0.4 MiB, -1.6 MiB).ifc.sha256anddiagnostics.jsonof the 33 samples are equal against #1483's head (66 of 66).tools/check-local.sh: nothing is built on this Mac (no local compute); CI's fast jobs run the draft, and its test job runs the helper tests.Audit-honesty checklist
changelog.d/changed/.Breaking changes
None.
Related issues
Refs #1335.