Repository navigation
perf(reader): the wall records read builds only the fields it keeps of each SWall and ArcWall graph (#1335) - #1485
Closed
DrunkOnJava wants to merge 3 commits into
Closed
DrunkOnJava wants to merge 3 commits into
DrunkOnJava wants to merge 3 commits into
Conversation
9 of 10 tasks
DrunkOnJava
added a commit
that referenced
this pull request
Nov 5, 2026
DrunkOnJava
added a commit
that referenced
this pull request
Nov 6, 2026
…alog's document build only the fields they read; the preamble does less (#1335) (#1483) ## Summary One mechanism, three cuts, for #1335: a native pass that reads a few fields of a graph now reads and checks the whole graph exactly as before and builds only the fields it reads (`native_parameters::decode_graph_projected`, `Options::graph_fields`). It is used by the parameter definitions pass (its `Family` and `FamilySymbol` records), the read of the wall records (`SWall` and `ArcWall`) and the extensible storage catalog's document; the preamble's element index and checksum hex also do less work. Folded from #1483, #1485 and #1488; the phase read (#1487) is not changed and is recorded below as a measured floor. Measured at 4140b96 (`main` merged at df96767). The pushed head, 1811752, merges `main` at 8060ecc cleanly: no conflict in the files this changes, and #1470's additions to `native_walls.rs` are in `native_walls_and_top_joins_into`, which reads in full, not in `WallReads::read`. - **Peak memory falls on 58 of 68 files, is equal on 6 and rises on 4 (by 1, 1, 1 and 8 MiB).** The catalog's document, built whole, was the peak of many files: 2024 Snowdon Towers HVAC 398 to 214 MiB (0.54), Einhoven 37 to 30, 2015 to 2019 `rst_basic` 91 to 94 down to 79 to 80, Core Interior 324 to 310, RE1 Plumbing 73 to 67, Snowdon Towers plumbing 513 to 462. The four rises: 2026 `racbasicsampleproject` 225 to 233 MiB (1.03), and RE1 Electrical, 2015 `rac_advanced` and the BIM4LCA office at building permit by 1 MiB each. Why the 8 MiB is not measured. - **Time against `main` (the same unfiltered run, paired on one runner per shard, median of three): 0.867 on the 2015 samples, 0.878 on 2016 to 2018, 0.862 on 2019 to 2022, 0.903 on 2023, 0.968 on 2024 to 2027, 0.972 on the reference models and offices;** 0.947 over the 58 samples (mean of the per-file ratios 0.914, median 0.901) and 0.953 over all 68 files. The table is below. - **Against 0.11.0 and 0.15.0** (same-runner timing runs, 2015 to 2023 samples and the six reference models): 1.175 of 0.11.0's time over the 33 samples (mean of per-file ratios 1.199, median 1.254; 2015 to 2022 about 1.2, 2023 0.894, reference models 0.772) and 0.884 of 0.15.0's (mean 0.890, median 0.894). Tables below. - **No export moves:** `ifc.sha256` equal on all 111 exports of the unfiltered run, `diagnostics.json` equal on the 93 that upload one, and the 18 held-out models' `counts.txt` and `ifc.sha256` equal; of the 2,251 result files common to the two sides, 2,233 are byte-identical and the 18 that differ are the runs' own `commit.txt`, `runner-*.txt`, `vmstat-*.log` and `score-cache-*.log`. The semantic check says "no semantic difference in 34 models". ## Motivation #1335: the definitions pass is `FamilySymbol` 207 ms decode + 110 ms freeing, `Family` 28 + 8, of its 403 ms on 2015 `rac_advanced` (profile run 42111526405, scratch head cc7f9c1, one loaded runner; shares only). Over the 33 samples of 2015 to 2023 that profile has the passes at: definitions 9,843 of 63,116 ms (15.6%), wall records 2,227 (3.5%), phase read 1,946 (3.1%), native preamble 2,499 (4.0%), record index 1,220 (1.9%). On a quiet runner the cuts are smaller than the shares: 8.1%, 2.4% and 1.2%. "The shared preamble before the native passes, about 590 ms" is not the native preamble. It is everything an export does before its first native pass (the partition strings, the arc wall scan, the 2023 record scan, the levels), which 0.11.0 does as well: profiled on both (run 42111526405 on an EPYC 9V74 and run 42112096504, scratch head 03aa8f6, on an EPYC 7763: different runners, so shares only) those steps are 65.7% of the export of 2015 `rac_advanced` at 0.11.0 and 35.5% on `main`, and among the steps both versions run each takes about the same share on both (the partition strings 35% at 0.11.0 and 29% on `main`, the 2023 record scan 17% and 18%, writing the file 15% and 18%). They are a floor shared with 0.11.0. What the native passes added is `read_preamble`: 68 ms on 2015 `rac_advanced`: schema parse 11 ms, element index parse 18 ms, storage increments 4 ms, extensible storage catalog 28 ms. The issue's concern about the definitions pass: a root-only read could ingest a record the full decode refuses today. This does not read less: every byte is read and every check made, in the same order and with the same errors, and only the values of the fields not kept are not built. A record the full decode refuses is refused with the same message. ## Changes - `src/research/native_parameters.rs`: the cursor's `pointer`, `entity_pointer`, `class` and `field` take `const BUILD: bool`. With it unset they read, check and note pointers exactly as with it set (one body, so the checks cannot differ) and return `Null` in place of the value. `decode_graph_projected(.., keep)` decodes a graph whose objects keep only the fields named in `keep`. A repeated field key is found in the built object where there is one, and by position (`ClassPlan::duplicate`, found only when projecting) where the field is not built. The full decode's path is the same statements as before. - `src/research/native_document.rs`: `Options::graph_fields`, a function from a record's root class to the fields to keep; a projected record has no `saved_metadata`. The definitions pass sets its own, whatever the caller's. - Each field list is a `pub const` beside the one function that reads the projected graph, so that a field the reader comes to read is added where it is read: `native_parameter_definitions::{FAMILY_BINDING_FIELDS, SYMBOL_BINDING_FIELDS}` and `Registry::ingest_binding_context`; `native_walls::WALL_READ_FIELDS` (`m_id`, `m_WallAttributesId`, `m_baseOrTop`) and `wall_facts`, which `WallReads::read` calls; `native_es_catalog::CATALOG_FIELDS` and `from_graph`, which `decode` calls. Doc lines at those readers and at `wall_type_id` and `is_top_joined` say so, and `docs/maintaining.md` has a section on it. - `src/research/native_index.rs`: `parse` searched the element table's layout for the leaf of each of the 7 fields it takes, at each of the file's rows (17,101 on `rac_advanced`), with a vector of candidates each time. The leaf a prefix names is now found at the first row that asks for it and kept in a named field of `IndexLeaves`, so a lookup that fails fails at the same point with the same message and an eighth lookup is a new field, not an index. - `src/research/schema_registry.rs`, `src/research/native_es_catalog.rs`, `src/research/native_document.rs`: the checksums of the schema (one per class) and of the catalog's stream are formatted with `lower_hex`, the same lowercase digits without a string for each byte. - `probes/examples/probe_1335_projected_graphs.rs`: imports the constants, decodes every record both ways under each list and under two that stress the mechanism (the identity alone, no field), and runs the real readers on both graphs: `ingest_binding_context` (the registry and the refusals), `wall_facts` (every `SWall` and `ArcWall`) and `from_graph` (the catalog and its provenance JSON) on the `Global/Latest` document in full and projected. - `changelog.d/changed/1335-*.md`: three fragments. ## What is not cut (floors) Shares of the exports' own time over the 33 samples of 2015 to 2023 at the final head (profile run 42124471783, scratch head c85c73c, an EPYC 9V74, shares only), against `main` before in run 42111526405: definitions 6.8% (15.6%), the named-parameter read 3.4% (2.9%), the phase read 3.5% (3.1%), the record index 2.3% (1.9%), the wall records 1.8% (3.5%) and wall types 0.4%, the preamble 1.7% (4.0%), area regions 0.9% (0.8%); all native passes 22.8% (33.0%). If every one cost nothing, the export would take about 0.92 of 0.11.0's time (an estimate: those shares applied to the timing run's totals), so the 1.2 against 0.11.0 is what these passes still cost. - **The phase read:** 97 ms of 2015 `rac_advanced`'s 2,207 (4.4%, run 42111526405). Projecting its `RoomElem` graphs (38 ms of it) measured 0.993 of this PR's first head's time (run 42117002359, closed as #1487): inside the run's noise, for a list kept by hand. The rest is the per-record selection and routing of `extract_records` (about 26 ms) and the Element base read of 16,981 other records (18 ms), which builds an `ElementBase` and a `Record` for each so that the same closure sees what it saw. - **The record index:** one walk of every partition that includes the first inflate of each, which 0.11.0 does as well. - **The steps before the first native pass** (the paragraph above), shared with 0.11.0. - **Area regions** (`native_area_regions.rs`, new since 0.11.0): on 2015 `rac_advanced` the `LevelRoomPlan` graphs are 9.5 ms decoding and 5.0 ms freeing and the `CurveElem` graphs 5.9 and 1.6, about 22 ms of 1,646 (1.3%) at the final head (run 42124471783), and only a file with Areas pays it; over the 33 samples the pass is 0.9% of the exports' time in all. Its reader, `plan()`, builds `m_segments`, `m_circuits`, `m_components` and `m_rooms` of the `PlanTopology`, which is the bulk of those graphs, so a projection would leave most of the 1.3%: not worth a pull request. A floor. - **Taken in follow-up pull requests, stacked on this one:** building the class plans of a decode afresh for each record (3.1% of all export time: 1,024,681 plans built over the 33 samples, 2,114 ms of 68,189, in the plan-timing profile run 42127560399 on an EPYC 7763; 47 of the definitions pass's 166 ms and 20 of the wall records' 61 ms on `rac_advanced`), and the byte scan of the named-parameter read. ## Testing Unit tests are banned here: none added. All runs are of 4140b96, against `main` at df96767 or the clean tags; Autodesk's sample projects, the reference models, the BIM4LCA offices, the Revit 2025 models and the held-out models are fetched for the runs and never stored (held-out as counts). - [x] The unfiltered Measure, run 42125670870 (`-f base=main -f nordic=true -f r2025=true -f holdout=true -f samples=true -f timing=true`, default scorers, no `ifc_flags`): byte identity of every model and the timing against `main`, above. Its `same-project gates` job wrote the check run 127599178067: `success`, "pass: 9 samples held to their earlier saves, and the Areas on 4 saves and on the 2021 save". The probe run below, dispatched on the same commit and finished 30 seconds later, wrote a `neutral` "NOT MEASURED: the scorers asked for exclude bodies_across_saves" (127599399222) that GitHub lists as the latest check of that name; the success is listed with `filter=all`. - [x] The projected decode against the full decode, run 42125686355 (`-f probe=probe_1335_projected_graphs -f samples=true -f r2025=true -f scorers=none`; every sample of 2015 to 2027, the reference models and the Revit 2025 models, 88 files): 1,128,043 records, 6,768,258 projected decodes under six lists, 0 mismatches; the 4,743 records the full decode refuses are refused alike (`FamilyInstance` 3,403, `VarSketch` 1,125, `ImageHolder` 105, `AppearanceAssetElem` 38, `RebarShape` 30, others fewer). No `Family` or `FamilySymbol` is among them: the registry the definitions pass builds is the same from both graphs on every file. `wall_facts` is the same on all 7,900 `SWall` and `ArcWall` records. The catalog's document gives the same catalog and provenance on 79 files and is refused alike on 9. Not measured: the BIM4LCA offices and the held-out models under the probe. - [x] Time against 0.11.0 and 0.15.0, runs 42125703262 and 42125717387 (timing jobs, the 33 samples of 2015 to 2023 and the six reference models, one runner per shard, an EPYC 9V74 for the samples in both and a 9V74 and a 9V45 for the models; median of three exports each; sum of medians, mean and median of the per-file ratios; Revit 2024 to 2027 samples were not timed against the tags): | class | files | 0.11.0 (s) | PR (s) | sum | mean | median | per file | |---|---:|---:|---:|---:|---:|---:|---| | samples 2015 | 6 | 6.2 | 7.7 | 1.236 | 1.237 | 1.260 | 1.07 to 1.31 | | samples 2016 to 2018 | 9 | 6.6 | 8.0 | 1.213 | 1.230 | 1.282 | 1.07 to 1.33 | | samples 2019 to 2022 | 15 | 13.7 | 16.6 | 1.216 | 1.222 | 1.254 | 1.06 to 1.33 | | samples 2023 | 3 | 4.3 | 3.8 | 0.894 | 0.914 | 0.950 | 0.79 to 1.00 | | 33 samples | 33 | 30.7 | 36.1 | 1.175 | 1.199 | 1.254 | 0.79 to 1.33 | | reference models | 6 | 9.4 | 7.3 | 0.772 | 0.811 | 0.830 | 0.73 to 0.89 | | class | files | 0.15.0 (s) | PR (s) | sum | mean | median | per file | |---|---:|---:|---:|---:|---:|---:|---| | samples 2015 | 6 | 9.0 | 7.8 | 0.870 | 0.870 | 0.868 | 0.85 to 0.89 | | samples 2016 to 2018 | 9 | 9.0 | 8.0 | 0.890 | 0.895 | 0.900 | 0.86 to 0.92 | | samples 2019 to 2022 | 15 | 19.0 | 16.7 | 0.882 | 0.889 | 0.894 | 0.86 to 0.92 | | samples 2023 | 3 | 4.3 | 3.9 | 0.913 | 0.915 | 0.914 | 0.90 to 0.93 | | 33 samples | 33 | 41.3 | 36.5 | 0.884 | 0.890 | 0.894 | 0.85 to 0.93 | | reference models | 6 | 5.8 | 5.6 | 0.961 | 0.940 | 0.936 | 0.91 to 0.98 | Peak memory against 0.11.0 is 0.61 to 0.96 on the samples (the models 0.73 to 1.03), against 0.15.0 0.86 to 1.00 on the samples (the models 0.82 to 1.00). - [x] Time against `main`, the timing job of the unfiltered run (one runner per shard; shard 1 an EPYC 7763 for the 2024 samples, shard 2 a 9V74 for the samples to 2023 and the reference models, shard 3 a 7763 for the offices and the 2027 samples, shard 4 a 9V45 for the 2025 and 2026 samples): | class | files | `main` (s) | PR (s) | sum | mean | median | per file | peak memory, PR / `main` | |---|---:|---:|---:|---:|---:|---:|---|---| | samples 2015 | 6 | 9.1 | 7.9 | 0.867 | 0.868 | 0.863 | 0.84 to 0.90 | 0.85 to 1.00 | | samples 2016 to 2018 | 9 | 9.3 | 8.2 | 0.878 | 0.883 | 0.886 | 0.85 to 0.91 | 0.86 to 1.00 | | samples 2019 to 2022 | 15 | 19.7 | 17.0 | 0.862 | 0.871 | 0.875 | 0.83 to 0.90 | 0.86 to 0.99 | | samples 2023 | 3 | 4.3 | 3.9 | 0.903 | 0.906 | 0.905 | 0.89 to 0.92 | 0.89 to 1.00 | | samples 2024 to 2027 | 25 | 160.9 | 155.7 | 0.968 | 0.963 | 0.964 | 0.89 to 1.00 | 0.54 to 1.03 | | reference models and offices | 10 | 61.1 | 59.4 | 0.972 | 0.946 | 0.950 | 0.88 to 0.98 | 0.82 to 1.01 | | 58 samples | 58 | 203.3 | 192.6 | 0.947 | 0.914 | 0.901 | 0.83 to 1.00 | 0.54 to 1.03 | | all files | 68 | 264.4 | 252.0 | 0.953 | 0.919 | 0.914 | 0.83 to 1.00 | 0.54 to 1.03 | Earlier heads of the three parts (the decoder is the same; each slice measured against the one before it): the definitions part 0.919 of `main` over the 33 samples (run 42115512932, head fa10c6d), the wall records part 0.976 (run 42116989088, head 16527ce) and the preamble part 0.988 (run 42117016111, head 76e98d7). The wall records part raised the peak of 2019 and 2020 `rac_advanced` from 168 to 173 MiB (1.03) in its run; the definitions part had taken them from 173 to 168, so they are at what `main` peaks at. - [ ] `tools/check-local.sh`: nothing is built on this Mac (no local compute); CI's fast jobs pass on this head. ## Audit-honesty checklist - [x] No capability changes; no export is meant to change, and none does in the 111 exports of the unfiltered run. Nothing here is a claim about Revit's own export. - [x] A user-visible change has its entry as a fragment under `changelog.d/changed/`. - [x] No limitation resolved or added. #1335 stays open. - [x] Opt-in costs nothing when off: the plan of a class has an empty `kept` and no `duplicate` unless the decode is projected. - [x] No emojis, no private paths. ## Breaking changes None to the stable API, the CLIs or any export. `rvt::research` (unstable): `native_document::Options` gains `graph_fields` (and `GraphFields`); `native_parameters` gains `decode_graph_projected`; `native_walls` gains `WALL_READ_FIELDS`, `WallFacts` and `wall_facts`; `native_es_catalog` gains `CATALOG_FIELDS`, `document`, `document_limits` and `from_graph`; `native_parameter_definitions` gains `FAMILY_BINDING_FIELDS`, `SYMBOL_BINDING_FIELDS` and `binding_context_fields`. ## Related issues Refs #1335, which stays open.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closed, not merged: folded into #1483. The wall records read, its field list and its measurements are in #1483, where the list is a constant beside its one reader (
native_walls::wall_facts) and the probe imports it. What follows is the text of this pull request as it was when it was closed; the figures are of head 16527ce, run 42116989088.Summary
Stacked on #1483 (its commits are the first two of this branch; this PR is the last one). The read of the wall records decodes every
SWallandArcWallgraph in full to take two things from it: the type the record names and whether aJoinToRoofGStepjoins its top. It now reads and checks the same bytes, with the same refusals, and builds only the fields it keeps. On the 33 samples of 2015 to 2023 the exports take 0.976 of #1483's time (per file 0.95 to 1.01), and all 66 hashes are identical. Peak memory is not lower: see Testing.Motivation
#1335: the wall records are 132 ms on 2015
rac_advanced(profile run 42111526405, scratch head cc7f9c1, one loaded runner; shares only): 180SWallgraphs, 89 ms decoding and 33 ms freeing them. Over the 33 samples of 2015 to 2023 the pass is 2,227 of 63,116 ms (3.5%). On a quiet runner the cut is 2.4%.A wall's top joins live in objects behind
m_geomSteps, so the graph cannot be cut to its root. Projection keeps the whole graph's reading and builds the three fields the two consumers use, which is why this is a cut and not a floor.Changes
src/research/native_walls.rs:WallReads::readsetsOptions::graph_fields(from perf(reader): the definitions pass, the wall records read and the catalog's document build only the fields they read; the preamble does less (#1335) #1483) tom_id,m_WallAttributesIdandm_baseOrTopfor anSWallor anArcWall. The consumers arewall_type_id(m_WallAttributesIdof the first object that has it) andis_top_joined(m_baseOrTopof aJoinToRoofGStep); no other field of these records is read. Every other class of record the read selects, and every other reader of walls (native_walls_and_top_joins_into, the wall cuts, the wall types), is read in full.probes/examples/probe_1335_projected_graphs.rs: the listm_id, m_WallAttributesId, m_baseOrTopis compared too.changelog.d/changed/1335-walls-projected.md.Testing
Unit tests are banned here: none added. Measure run 42116989088 (
-f ref=16527cea -f base=f5bee766 -f samples=true -f sample_years="2015 2023" -f models="-" -f scorers=none -f timing=true -f probe=probe_1335_projected_graphs, this head against #1483's head; Autodesk's sample projects, fetched for the run and never stored).tools/check-local.sh: nothing is built on this Mac (no local compute); CI's fast jobs run the draft.m_id, m_WallAttributesId, m_baseOrTopamong them: 505,203 records, 3,536,421 projected decodes, 0 mismatches; the 4,599 records the full decode refuses are refused alike. The same probe on Revit 2024 to 2027, the reference models and the Revit 2025 models (run 42117420185, at the scratch merge b005499): 622,840 records, 0 mismatches.ifc.sha256anddiagnostics.jsonof the 33 samples, head against base, 66 of 66 equal. The unfiltered run of every model is not made.Peak memory is not lower here: 2019 and 2020
rac_advancedpeak at 173 MiB against #1483's 168 (1.03), and 2015rac_advancedat 171 against 170. #1483 had taken those two files from 173 to 168 MiB, so this brings them back to whatmainpeaks at, not above it (chained from run 42115512932, not measured againstmainin one run). Why is not measured: the peak is set after the wall read, and a likely cause is that the wall read now leaves less freed memory for the later steps to reuse.Audit-honesty checklist
changelog.d/changed/.Breaking changes
None.
Related issues
Refs #1335.