Repository navigation
perf(reader): the phase read builds only the fields it keeps of each RoomElem graph (#1335) - #1487
Closed
DrunkOnJava wants to merge 3 commits into
Closed
DrunkOnJava wants to merge 3 commits into
DrunkOnJava wants to merge 3 commits into
Conversation
… each Family and FamilySymbol record (#1335)
9 of 10 tasks
DrunkOnJava
added a commit
that referenced
this pull request
Nov 6, 2026
…alog's document build only the fields they read; the preamble does less (#1335) (#1483) ## Summary One mechanism, three cuts, for #1335: a native pass that reads a few fields of a graph now reads and checks the whole graph exactly as before and builds only the fields it reads (`native_parameters::decode_graph_projected`, `Options::graph_fields`). It is used by the parameter definitions pass (its `Family` and `FamilySymbol` records), the read of the wall records (`SWall` and `ArcWall`) and the extensible storage catalog's document; the preamble's element index and checksum hex also do less work. Folded from #1483, #1485 and #1488; the phase read (#1487) is not changed and is recorded below as a measured floor. Measured at 4140b96 (`main` merged at df96767). The pushed head, 1811752, merges `main` at 8060ecc cleanly: no conflict in the files this changes, and #1470's additions to `native_walls.rs` are in `native_walls_and_top_joins_into`, which reads in full, not in `WallReads::read`. - **Peak memory falls on 58 of 68 files, is equal on 6 and rises on 4 (by 1, 1, 1 and 8 MiB).** The catalog's document, built whole, was the peak of many files: 2024 Snowdon Towers HVAC 398 to 214 MiB (0.54), Einhoven 37 to 30, 2015 to 2019 `rst_basic` 91 to 94 down to 79 to 80, Core Interior 324 to 310, RE1 Plumbing 73 to 67, Snowdon Towers plumbing 513 to 462. The four rises: 2026 `racbasicsampleproject` 225 to 233 MiB (1.03), and RE1 Electrical, 2015 `rac_advanced` and the BIM4LCA office at building permit by 1 MiB each. Why the 8 MiB is not measured. - **Time against `main` (the same unfiltered run, paired on one runner per shard, median of three): 0.867 on the 2015 samples, 0.878 on 2016 to 2018, 0.862 on 2019 to 2022, 0.903 on 2023, 0.968 on 2024 to 2027, 0.972 on the reference models and offices;** 0.947 over the 58 samples (mean of the per-file ratios 0.914, median 0.901) and 0.953 over all 68 files. The table is below. - **Against 0.11.0 and 0.15.0** (same-runner timing runs, 2015 to 2023 samples and the six reference models): 1.175 of 0.11.0's time over the 33 samples (mean of per-file ratios 1.199, median 1.254; 2015 to 2022 about 1.2, 2023 0.894, reference models 0.772) and 0.884 of 0.15.0's (mean 0.890, median 0.894). Tables below. - **No export moves:** `ifc.sha256` equal on all 111 exports of the unfiltered run, `diagnostics.json` equal on the 93 that upload one, and the 18 held-out models' `counts.txt` and `ifc.sha256` equal; of the 2,251 result files common to the two sides, 2,233 are byte-identical and the 18 that differ are the runs' own `commit.txt`, `runner-*.txt`, `vmstat-*.log` and `score-cache-*.log`. The semantic check says "no semantic difference in 34 models". ## Motivation #1335: the definitions pass is `FamilySymbol` 207 ms decode + 110 ms freeing, `Family` 28 + 8, of its 403 ms on 2015 `rac_advanced` (profile run 42111526405, scratch head cc7f9c1, one loaded runner; shares only). Over the 33 samples of 2015 to 2023 that profile has the passes at: definitions 9,843 of 63,116 ms (15.6%), wall records 2,227 (3.5%), phase read 1,946 (3.1%), native preamble 2,499 (4.0%), record index 1,220 (1.9%). On a quiet runner the cuts are smaller than the shares: 8.1%, 2.4% and 1.2%. "The shared preamble before the native passes, about 590 ms" is not the native preamble. It is everything an export does before its first native pass (the partition strings, the arc wall scan, the 2023 record scan, the levels), which 0.11.0 does as well: profiled on both (run 42111526405 on an EPYC 9V74 and run 42112096504, scratch head 03aa8f6, on an EPYC 7763: different runners, so shares only) those steps are 65.7% of the export of 2015 `rac_advanced` at 0.11.0 and 35.5% on `main`, and among the steps both versions run each takes about the same share on both (the partition strings 35% at 0.11.0 and 29% on `main`, the 2023 record scan 17% and 18%, writing the file 15% and 18%). They are a floor shared with 0.11.0. What the native passes added is `read_preamble`: 68 ms on 2015 `rac_advanced`: schema parse 11 ms, element index parse 18 ms, storage increments 4 ms, extensible storage catalog 28 ms. The issue's concern about the definitions pass: a root-only read could ingest a record the full decode refuses today. This does not read less: every byte is read and every check made, in the same order and with the same errors, and only the values of the fields not kept are not built. A record the full decode refuses is refused with the same message. ## Changes - `src/research/native_parameters.rs`: the cursor's `pointer`, `entity_pointer`, `class` and `field` take `const BUILD: bool`. With it unset they read, check and note pointers exactly as with it set (one body, so the checks cannot differ) and return `Null` in place of the value. `decode_graph_projected(.., keep)` decodes a graph whose objects keep only the fields named in `keep`. A repeated field key is found in the built object where there is one, and by position (`ClassPlan::duplicate`, found only when projecting) where the field is not built. The full decode's path is the same statements as before. - `src/research/native_document.rs`: `Options::graph_fields`, a function from a record's root class to the fields to keep; a projected record has no `saved_metadata`. The definitions pass sets its own, whatever the caller's. - Each field list is a `pub const` beside the one function that reads the projected graph, so that a field the reader comes to read is added where it is read: `native_parameter_definitions::{FAMILY_BINDING_FIELDS, SYMBOL_BINDING_FIELDS}` and `Registry::ingest_binding_context`; `native_walls::WALL_READ_FIELDS` (`m_id`, `m_WallAttributesId`, `m_baseOrTop`) and `wall_facts`, which `WallReads::read` calls; `native_es_catalog::CATALOG_FIELDS` and `from_graph`, which `decode` calls. Doc lines at those readers and at `wall_type_id` and `is_top_joined` say so, and `docs/maintaining.md` has a section on it. - `src/research/native_index.rs`: `parse` searched the element table's layout for the leaf of each of the 7 fields it takes, at each of the file's rows (17,101 on `rac_advanced`), with a vector of candidates each time. The leaf a prefix names is now found at the first row that asks for it and kept in a named field of `IndexLeaves`, so a lookup that fails fails at the same point with the same message and an eighth lookup is a new field, not an index. - `src/research/schema_registry.rs`, `src/research/native_es_catalog.rs`, `src/research/native_document.rs`: the checksums of the schema (one per class) and of the catalog's stream are formatted with `lower_hex`, the same lowercase digits without a string for each byte. - `probes/examples/probe_1335_projected_graphs.rs`: imports the constants, decodes every record both ways under each list and under two that stress the mechanism (the identity alone, no field), and runs the real readers on both graphs: `ingest_binding_context` (the registry and the refusals), `wall_facts` (every `SWall` and `ArcWall`) and `from_graph` (the catalog and its provenance JSON) on the `Global/Latest` document in full and projected. - `changelog.d/changed/1335-*.md`: three fragments. ## What is not cut (floors) Shares of the exports' own time over the 33 samples of 2015 to 2023 at the final head (profile run 42124471783, scratch head c85c73c, an EPYC 9V74, shares only), against `main` before in run 42111526405: definitions 6.8% (15.6%), the named-parameter read 3.4% (2.9%), the phase read 3.5% (3.1%), the record index 2.3% (1.9%), the wall records 1.8% (3.5%) and wall types 0.4%, the preamble 1.7% (4.0%), area regions 0.9% (0.8%); all native passes 22.8% (33.0%). If every one cost nothing, the export would take about 0.92 of 0.11.0's time (an estimate: those shares applied to the timing run's totals), so the 1.2 against 0.11.0 is what these passes still cost. - **The phase read:** 97 ms of 2015 `rac_advanced`'s 2,207 (4.4%, run 42111526405). Projecting its `RoomElem` graphs (38 ms of it) measured 0.993 of this PR's first head's time (run 42117002359, closed as #1487): inside the run's noise, for a list kept by hand. The rest is the per-record selection and routing of `extract_records` (about 26 ms) and the Element base read of 16,981 other records (18 ms), which builds an `ElementBase` and a `Record` for each so that the same closure sees what it saw. - **The record index:** one walk of every partition that includes the first inflate of each, which 0.11.0 does as well. - **The steps before the first native pass** (the paragraph above), shared with 0.11.0. - **Area regions** (`native_area_regions.rs`, new since 0.11.0): on 2015 `rac_advanced` the `LevelRoomPlan` graphs are 9.5 ms decoding and 5.0 ms freeing and the `CurveElem` graphs 5.9 and 1.6, about 22 ms of 1,646 (1.3%) at the final head (run 42124471783), and only a file with Areas pays it; over the 33 samples the pass is 0.9% of the exports' time in all. Its reader, `plan()`, builds `m_segments`, `m_circuits`, `m_components` and `m_rooms` of the `PlanTopology`, which is the bulk of those graphs, so a projection would leave most of the 1.3%: not worth a pull request. A floor. - **Taken in follow-up pull requests, stacked on this one:** building the class plans of a decode afresh for each record (3.1% of all export time: 1,024,681 plans built over the 33 samples, 2,114 ms of 68,189, in the plan-timing profile run 42127560399 on an EPYC 7763; 47 of the definitions pass's 166 ms and 20 of the wall records' 61 ms on `rac_advanced`), and the byte scan of the named-parameter read. ## Testing Unit tests are banned here: none added. All runs are of 4140b96, against `main` at df96767 or the clean tags; Autodesk's sample projects, the reference models, the BIM4LCA offices, the Revit 2025 models and the held-out models are fetched for the runs and never stored (held-out as counts). - [x] The unfiltered Measure, run 42125670870 (`-f base=main -f nordic=true -f r2025=true -f holdout=true -f samples=true -f timing=true`, default scorers, no `ifc_flags`): byte identity of every model and the timing against `main`, above. Its `same-project gates` job wrote the check run 127599178067: `success`, "pass: 9 samples held to their earlier saves, and the Areas on 4 saves and on the 2021 save". The probe run below, dispatched on the same commit and finished 30 seconds later, wrote a `neutral` "NOT MEASURED: the scorers asked for exclude bodies_across_saves" (127599399222) that GitHub lists as the latest check of that name; the success is listed with `filter=all`. - [x] The projected decode against the full decode, run 42125686355 (`-f probe=probe_1335_projected_graphs -f samples=true -f r2025=true -f scorers=none`; every sample of 2015 to 2027, the reference models and the Revit 2025 models, 88 files): 1,128,043 records, 6,768,258 projected decodes under six lists, 0 mismatches; the 4,743 records the full decode refuses are refused alike (`FamilyInstance` 3,403, `VarSketch` 1,125, `ImageHolder` 105, `AppearanceAssetElem` 38, `RebarShape` 30, others fewer). No `Family` or `FamilySymbol` is among them: the registry the definitions pass builds is the same from both graphs on every file. `wall_facts` is the same on all 7,900 `SWall` and `ArcWall` records. The catalog's document gives the same catalog and provenance on 79 files and is refused alike on 9. Not measured: the BIM4LCA offices and the held-out models under the probe. - [x] Time against 0.11.0 and 0.15.0, runs 42125703262 and 42125717387 (timing jobs, the 33 samples of 2015 to 2023 and the six reference models, one runner per shard, an EPYC 9V74 for the samples in both and a 9V74 and a 9V45 for the models; median of three exports each; sum of medians, mean and median of the per-file ratios; Revit 2024 to 2027 samples were not timed against the tags): | class | files | 0.11.0 (s) | PR (s) | sum | mean | median | per file | |---|---:|---:|---:|---:|---:|---:|---| | samples 2015 | 6 | 6.2 | 7.7 | 1.236 | 1.237 | 1.260 | 1.07 to 1.31 | | samples 2016 to 2018 | 9 | 6.6 | 8.0 | 1.213 | 1.230 | 1.282 | 1.07 to 1.33 | | samples 2019 to 2022 | 15 | 13.7 | 16.6 | 1.216 | 1.222 | 1.254 | 1.06 to 1.33 | | samples 2023 | 3 | 4.3 | 3.8 | 0.894 | 0.914 | 0.950 | 0.79 to 1.00 | | 33 samples | 33 | 30.7 | 36.1 | 1.175 | 1.199 | 1.254 | 0.79 to 1.33 | | reference models | 6 | 9.4 | 7.3 | 0.772 | 0.811 | 0.830 | 0.73 to 0.89 | | class | files | 0.15.0 (s) | PR (s) | sum | mean | median | per file | |---|---:|---:|---:|---:|---:|---:|---| | samples 2015 | 6 | 9.0 | 7.8 | 0.870 | 0.870 | 0.868 | 0.85 to 0.89 | | samples 2016 to 2018 | 9 | 9.0 | 8.0 | 0.890 | 0.895 | 0.900 | 0.86 to 0.92 | | samples 2019 to 2022 | 15 | 19.0 | 16.7 | 0.882 | 0.889 | 0.894 | 0.86 to 0.92 | | samples 2023 | 3 | 4.3 | 3.9 | 0.913 | 0.915 | 0.914 | 0.90 to 0.93 | | 33 samples | 33 | 41.3 | 36.5 | 0.884 | 0.890 | 0.894 | 0.85 to 0.93 | | reference models | 6 | 5.8 | 5.6 | 0.961 | 0.940 | 0.936 | 0.91 to 0.98 | Peak memory against 0.11.0 is 0.61 to 0.96 on the samples (the models 0.73 to 1.03), against 0.15.0 0.86 to 1.00 on the samples (the models 0.82 to 1.00). - [x] Time against `main`, the timing job of the unfiltered run (one runner per shard; shard 1 an EPYC 7763 for the 2024 samples, shard 2 a 9V74 for the samples to 2023 and the reference models, shard 3 a 7763 for the offices and the 2027 samples, shard 4 a 9V45 for the 2025 and 2026 samples): | class | files | `main` (s) | PR (s) | sum | mean | median | per file | peak memory, PR / `main` | |---|---:|---:|---:|---:|---:|---:|---|---| | samples 2015 | 6 | 9.1 | 7.9 | 0.867 | 0.868 | 0.863 | 0.84 to 0.90 | 0.85 to 1.00 | | samples 2016 to 2018 | 9 | 9.3 | 8.2 | 0.878 | 0.883 | 0.886 | 0.85 to 0.91 | 0.86 to 1.00 | | samples 2019 to 2022 | 15 | 19.7 | 17.0 | 0.862 | 0.871 | 0.875 | 0.83 to 0.90 | 0.86 to 0.99 | | samples 2023 | 3 | 4.3 | 3.9 | 0.903 | 0.906 | 0.905 | 0.89 to 0.92 | 0.89 to 1.00 | | samples 2024 to 2027 | 25 | 160.9 | 155.7 | 0.968 | 0.963 | 0.964 | 0.89 to 1.00 | 0.54 to 1.03 | | reference models and offices | 10 | 61.1 | 59.4 | 0.972 | 0.946 | 0.950 | 0.88 to 0.98 | 0.82 to 1.01 | | 58 samples | 58 | 203.3 | 192.6 | 0.947 | 0.914 | 0.901 | 0.83 to 1.00 | 0.54 to 1.03 | | all files | 68 | 264.4 | 252.0 | 0.953 | 0.919 | 0.914 | 0.83 to 1.00 | 0.54 to 1.03 | Earlier heads of the three parts (the decoder is the same; each slice measured against the one before it): the definitions part 0.919 of `main` over the 33 samples (run 42115512932, head fa10c6d), the wall records part 0.976 (run 42116989088, head 16527ce) and the preamble part 0.988 (run 42117016111, head 76e98d7). The wall records part raised the peak of 2019 and 2020 `rac_advanced` from 168 to 173 MiB (1.03) in its run; the definitions part had taken them from 173 to 168, so they are at what `main` peaks at. - [ ] `tools/check-local.sh`: nothing is built on this Mac (no local compute); CI's fast jobs pass on this head. ## Audit-honesty checklist - [x] No capability changes; no export is meant to change, and none does in the 111 exports of the unfiltered run. Nothing here is a claim about Revit's own export. - [x] A user-visible change has its entry as a fragment under `changelog.d/changed/`. - [x] No limitation resolved or added. #1335 stays open. - [x] Opt-in costs nothing when off: the plan of a class has an empty `kept` and no `duplicate` unless the decode is projected. - [x] No emojis, no private paths. ## Breaking changes None to the stable API, the CLIs or any export. `rvt::research` (unstable): `native_document::Options` gains `graph_fields` (and `GraphFields`); `native_parameters` gains `decode_graph_projected`; `native_walls` gains `WALL_READ_FIELDS`, `WallFacts` and `wall_facts`; `native_es_catalog` gains `CATALOG_FIELDS`, `document`, `document_limits` and `from_graph`; `native_parameter_definitions` gains `FAMILY_BINDING_FIELDS`, `SYMBOL_BINDING_FIELDS` and `binding_context_fields`. ## Related issues Refs #1335, which stays open.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closed, not merged: dropped. Projecting the
RoomElemgraphs of the phase read measured 0.993 of #1483's time over the 33 samples of 2015 to 2023 (run 42117002359, head 42eb54f), which is inside the run's noise, and it adds a field list kept by hand. The phase read is recorded as a measured floor in #1483: on 2015rac_advancedit is 97 ms of 2,207 (4.4%) in profile run 42111526405, of which theRoomElemgraphs are 38 ms; the rest is the per-record selection and routing ofextract_recordsand the Element base read of 16,981 other records. What follows is the text of this pull request as it was when it was closed.Summary
Stacked on #1483 (its commits come first on this branch; this PR is the last one). The phase read decodes every
RoomElemgraph in full to take three fields of it. It now reads and checks the same bytes, with the same refusals, and builds only those fields. On the 33 samples of 2015 to 2023 the exports take 0.993 of #1483's time (per file 0.97 to 1.01): a gain of 0.7%, which is inside the run's noise for most files. All 66 hashes are identical. Whether it is worth a CI round on its own is for the main session.Motivation
#1335: the phase read is 97 ms of 2015
rac_advanced's export (profile run 42111526405, scratch head cc7f9c1, one loaded runner; shares only). It decodes the 116RoomElemgraphs in full (28 ms decoding, 10 ms freeing) and builds a record for each of 16,981 other records to read their Element base (18 ms). Over the 33 samples of 2015 to 2023 the pass, without the preamble and the record index it can be the first to pay, is 1,946 of 63,116 ms (3.1%). The RoomElem graphs are a third of that, which is why the cut is small.Changes
src/research/native_phases.rs:compute_phase_filtersetsOptions::graph_fields(from perf(reader): the definitions pass, the wall records read and the catalog's document build only the fields they read; the preamble does less (#1335) #1483) tom_id,m_createdPhaseId,m_demolishedPhaseIdandm_phaseIdfor aRoomElem: the only fields the room branch of its closure reads, andm_idfor the identity check. The records of the project's phases (AllProjectPhases,ProjectPhase) are read in full as before.changelog.d/changed/1335-phase-projected.md.What is not cut, and why: the Element base read of every other record (18 ms on
rac_advanced) is the direct byte reader of #1333; what is left is building anElementBaseand aRecordfor each of those records, 16,981 times, so that the same closure sees what it saw. Taking that out needs a second callback onextract_records. Each pass over every record also pays the per-record selection and routing ofextract_records(about 26 ms onrac_advanced), which is the fail-closed path and is not touched. Both are floors of this PR.Testing
Unit tests are banned here: none added. Measure run 42117002359 (
-f ref=42eb54f1 -f base=f5bee766 -f samples=true -f sample_years="2015 2023" -f models="-" -f scorers=none -f timing=true, this head against #1483's head; Autodesk's sample projects, fetched for the run and never stored).tools/check-local.sh: nothing is built on this Mac (no local compute); CI's fast jobs run the draft.m_id, m_createdPhaseId, m_demolishedPhaseId, m_phaseId: perf(reader): the definitions pass, the wall records read and the catalog's document build only the fields they read; the preamble does less (#1335) #1483's probe, which has had that list since its first run: 505,203 records of the 33 samples (run 42115512932) and 622,840 of Revit 2024 to 2027, the reference models and the Revit 2025 models (run 42117420185), 0 mismatches.ifc.sha256anddiagnostics.jsonof the 33 samples, head against base, 66 of 66 equal. The unfiltered run of every model is not made.Audit-honesty checklist
changelog.d/changed/.Breaking changes
None.
Related issues
Refs #1335.