Skip to content

perf(reader): the named-parameter read searches a record for each id it looks for, not every four-byte window (#1335) - #1502

Closed
DrunkOnJava wants to merge 10 commits into
mainfrom
perf/1335-needle-scan
Closed

DrunkOnJava wants to merge 10 commits into
mainfrom
perf/1335-needle-scan

Conversation

@DrunkOnJava

@DrunkOnJava DrunkOnJava commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

Summary

Stacked on #1483 (its commits come first on this branch; this PR is the last one). The read of the parameters named for a property (filtered_parameters_only) decides which of a Revit 2015 to 2023 file's records to decode by looking for the four bytes of each of a few definition ids in the record. It scanned every four-byte window of each record; it now searches each record for each id with memchr::memmem, and keeps the window scan for more than 16 ids. The records it decodes are the same by construction: a window equals an id's low 32 bits exactly where the id's four little-endian bytes occur. Time, byte identity and peak memory on this head: not yet measured.

Motivation

#1335's profile of the final head (run 42124471783, scratch head c85c73c, shares only): the named-parameter read is 3.4% of the exports' time over the 33 samples of 2015 to 2023 (57 ms of 1,646 on 2015 rac_advanced), most of it the byte scan over the 5,608 records it selects, for 6 ids.

memchr is already a normal dependency of the crate (Cargo.toml, used by reader.rs, partition_curve_fields.rs and others): nothing is added to the build graph.

Changes

  • src/research/native_document.rs: IdNeedles holds a memchr::memmem::Finder for each id while there are at most MAX_FINDERS (16) of them, and occur_in is true where any finder finds its id in the body; with more ids, or none, it is the window scan as before (occur_in_windows).
  • Pure-helper tests, next to the helper (the one exception to the ban on unit tests: no file, no I/O): where the two ways could differ, an id in the last four bytes, at an unaligned offset, in a body shorter than an id, a window that shares only an id's low 16 bits, no ids, an eight-byte id found by its first four bytes, and both ways on 1, 16, 17 and 40 ids, present and absent.
  • changelog.d/changed/1335-needle-scan.md.

Result: no gain, so this should not land

Measure run 42130086398 (-f ref=98566dd3 -f base=4140b96c -f samples=true -f sample_years="2015 2023" -f models="-" -f scorers=none -f timing=true; this head against #1483's measured head; both shards an EPYC 7763; median of three exports each):

class files #1483 (s) this PR (s) sum mean median per file peak memory
samples 2015 6 9.4 9.4 0.998 0.997 0.998 0.99 to 1.00 0.98 to 1.00
samples 2016 to 2018 9 10.0 10.0 1.001 1.002 1.000 0.99 to 1.01 1.00
samples 2019 to 2022 15 20.6 20.6 1.001 1.001 1.000 0.98 to 1.01 1.00
samples 2023 3 4.7 4.7 1.016 1.016 1.016 1.02 1.00
33 samples 33 44.7 44.8 1.002 1.002 1.000 0.98 to 1.02 0.98 to 1.00
reference models 6 9.1 9.1 1.006 1.006 1.005 1.00 to 1.02 1.00

"Every head run below every base run", from the timing jobs' raw medians of three: 0 of 39 files; every head run above every base run on 8; the runs overlap on 31. The window scan it replaces was already fast enough that the search over six ids is not measurably quicker, and my estimate of about 1.5% of the export (from the profile's 57 ms for the named-parameter read on 2015 rac_advanced, most of it assumed to be the scan) was wrong: the rest of that read, the walk over 17,101 records and the selection of 5,608, is what it costs. Peak memory is the same (largest change: +0.4 MiB, -1.6 MiB).

ifc.sha256 and diagnostics.json of the 33 samples are equal against #1483's head (66 of 66).

  • No gain measured: recommend closing this pull request without merging.
  • tools/check-local.sh: nothing is built on this Mac (no local compute); CI's fast jobs run the draft, and its test job runs the helper tests.

Audit-honesty checklist

Breaking changes

None.

Related issues

Refs #1335.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant