[DrunkOnJava]

Summary

One mechanism, three cuts, for #1335: a native pass that reads a few fields of a graph now reads and checks the whole graph exactly as before and builds only the fields it reads (native_parameters::decode_graph_projected, Options::graph_fields). It is used by the parameter definitions pass (its Family and FamilySymbol records), the read of the wall records (SWall and ArcWall) and the extensible storage catalog's document; the preamble's element index and checksum hex also do less work. Folded from #1483, #1485 and #1488; the phase read (#1487) is not changed and is recorded below as a measured floor.

Measured at 4140b96 (main merged at df96767). The pushed head, 1811752, merges main at 8060ecc cleanly: no conflict in the files this changes, and #1470's additions to native_walls.rs are in native_walls_and_top_joins_into, which reads in full, not in WallReads::read.

• Peak memory falls on 58 of 68 files, is equal on 6 and rises on 4 (by 1, 1, 1 and 8 MiB). The catalog's document, built whole, was the peak of many files: 2024 Snowdon Towers HVAC 398 to 214 MiB (0.54), Einhoven 37 to 30, 2015 to 2019 rst_basic 91 to 94 down to 79 to 80, Core Interior 324 to 310, RE1 Plumbing 73 to 67, Snowdon Towers plumbing 513 to 462. The four rises: 2026 racbasicsampleproject 225 to 233 MiB (1.03), and RE1 Electrical, 2015 rac_advanced and the BIM4LCA office at building permit by 1 MiB each. Why the 8 MiB is not measured.
• Time against main (the same unfiltered run, paired on one runner per shard, median of three): 0.867 on the 2015 samples, 0.878 on 2016 to 2018, 0.862 on 2019 to 2022, 0.903 on 2023, 0.968 on 2024 to 2027, 0.972 on the reference models and offices; 0.947 over the 58 samples (mean of the per-file ratios 0.914, median 0.901) and 0.953 over all 68 files. The table is below.
• Against 0.11.0 and 0.15.0 (same-runner timing runs, 2015 to 2023 samples and the six reference models): 1.175 of 0.11.0's time over the 33 samples (mean of per-file ratios 1.199, median 1.254; 2015 to 2022 about 1.2, 2023 0.894, reference models 0.772) and 0.884 of 0.15.0's (mean 0.890, median 0.894). Tables below.
• No export moves: ifc.sha256 equal on all 111 exports of the unfiltered run, diagnostics.json equal on the 93 that upload one, and the 18 held-out models' counts.txt and ifc.sha256 equal; of the 2,251 result files common to the two sides, 2,233 are byte-identical and the 18 that differ are the runs' own commit.txt, runner-*.txt, vmstat-*.log and score-cache-*.log. The semantic check says "no semantic difference in 34 models".

Motivation

#1335: the definitions pass is FamilySymbol 207 ms decode + 110 ms freeing, Family 28 + 8, of its 403 ms on 2015 rac_advanced (profile run 42111526405, scratch head cc7f9c1, one loaded runner; shares only). Over the 33 samples of 2015 to 2023 that profile has the passes at: definitions 9,843 of 63,116 ms (15.6%), wall records 2,227 (3.5%), phase read 1,946 (3.1%), native preamble 2,499 (4.0%), record index 1,220 (1.9%). On a quiet runner the cuts are smaller than the shares: 8.1%, 2.4% and 1.2%.

"The shared preamble before the native passes, about 590 ms" is not the native preamble. It is everything an export does before its first native pass (the partition strings, the arc wall scan, the 2023 record scan, the levels), which 0.11.0 does as well: profiled on both (run 42111526405 on an EPYC 9V74 and run 42112096504, scratch head 03aa8f6, on an EPYC 7763: different runners, so shares only) those steps are 65.7% of the export of 2015 rac_advanced at 0.11.0 and 35.5% on main, and among the steps both versions run each takes about the same share on both (the partition strings 35% at 0.11.0 and 29% on main, the 2023 record scan 17% and 18%, writing the file 15% and 18%). They are a floor shared with 0.11.0. What the native passes added is read_preamble: 68 ms on 2015 rac_advanced: schema parse 11 ms, element index parse 18 ms, storage increments 4 ms, extensible storage catalog 28 ms.

The issue's concern about the definitions pass: a root-only read could ingest a record the full decode refuses today. This does not read less: every byte is read and every check made, in the same order and with the same errors, and only the values of the fields not kept are not built. A record the full decode refuses is refused with the same message.

Changes

• src/research/native_parameters.rs: the cursor's pointer, entity_pointer, class and field take const BUILD: bool. With it unset they read, check and note pointers exactly as with it set (one body, so the checks cannot differ) and return Null in place of the value. decode_graph_projected(.., keep) decodes a graph whose objects keep only the fields named in keep. A repeated field key is found in the built object where there is one, and by position (ClassPlan::duplicate, found only when projecting) where the field is not built. The full decode's path is the same statements as before.
• src/research/native_document.rs: Options::graph_fields, a function from a record's root class to the fields to keep; a projected record has no saved_metadata. The definitions pass sets its own, whatever the caller's.
• Each field list is a pub const beside the one function that reads the projected graph, so that a field the reader comes to read is added where it is read: native_parameter_definitions::{FAMILY_BINDING_FIELDS, SYMBOL_BINDING_FIELDS} and Registry::ingest_binding_context; native_walls::WALL_READ_FIELDS (m_id, m_WallAttributesId, m_baseOrTop) and wall_facts, which WallReads::read calls; native_es_catalog::CATALOG_FIELDS and from_graph, which decode calls. Doc lines at those readers and at wall_type_id and is_top_joined say so, and docs/maintaining.md has a section on it.
• src/research/native_index.rs: parse searched the element table's layout for the leaf of each of the 7 fields it takes, at each of the file's rows (17,101 on rac_advanced), with a vector of candidates each time. The leaf a prefix names is now found at the first row that asks for it and kept in a named field of IndexLeaves, so a lookup that fails fails at the same point with the same message and an eighth lookup is a new field, not an index.
• src/research/schema_registry.rs, src/research/native_es_catalog.rs, src/research/native_document.rs: the checksums of the schema (one per class) and of the catalog's stream are formatted with lower_hex, the same lowercase digits without a string for each byte.
• probes/examples/probe_1335_projected_graphs.rs: imports the constants, decodes every record both ways under each list and under two that stress the mechanism (the identity alone, no field), and runs the real readers on both graphs: ingest_binding_context (the registry and the refusals), wall_facts (every SWall and ArcWall) and from_graph (the catalog and its provenance JSON) on the Global/Latest document in full and projected.
• changelog.d/changed/1335-*.md: three fragments.

What is not cut (floors)

Shares of the exports' own time over the 33 samples of 2015 to 2023 at the final head (profile run 42124471783, scratch head c85c73c, an EPYC 9V74, shares only), against main before in run 42111526405: definitions 6.8% (15.6%), the named-parameter read 3.4% (2.9%), the phase read 3.5% (3.1%), the record index 2.3% (1.9%), the wall records 1.8% (3.5%) and wall types 0.4%, the preamble 1.7% (4.0%), area regions 0.9% (0.8%); all native passes 22.8% (33.0%). If every one cost nothing, the export would take about 0.92 of 0.11.0's time (an estimate: those shares applied to the timing run's totals), so the 1.2 against 0.11.0 is what these passes still cost.

• The phase read: 97 ms of 2015 rac_advanced's 2,207 (4.4%, run 42111526405). Projecting its RoomElem graphs (38 ms of it) measured 0.993 of this PR's first head's time (run 42117002359, closed as perf(reader): the phase read builds only the fields it keeps of each RoomElem graph (#1335) #1487): inside the run's noise, for a list kept by hand. The rest is the per-record selection and routing of extract_records (about 26 ms) and the Element base read of 16,981 other records (18 ms), which builds an ElementBase and a Record for each so that the same closure sees what it saw.
• The record index: one walk of every partition that includes the first inflate of each, which 0.11.0 does as well.
• The steps before the first native pass (the paragraph above), shared with 0.11.0.
• Area regions (native_area_regions.rs, new since 0.11.0): on 2015 rac_advanced the LevelRoomPlan graphs are 9.5 ms decoding and 5.0 ms freeing and the CurveElem graphs 5.9 and 1.6, about 22 ms of 1,646 (1.3%) at the final head (run 42124471783), and only a file with Areas pays it; over the 33 samples the pass is 0.9% of the exports' time in all. Its reader, plan(), builds m_segments, m_circuits, m_components and m_rooms of the PlanTopology, which is the bulk of those graphs, so a projection would leave most of the 1.3%: not worth a pull request. A floor.
• Taken in follow-up pull requests, stacked on this one: building the class plans of a decode afresh for each record (3.1% of all export time: 1,024,681 plans built over the 33 samples, 2,114 ms of 68,189, in the plan-timing profile run 42127560399 on an EPYC 7763; 47 of the definitions pass's 166 ms and 20 of the wall records' 61 ms on rac_advanced), and the byte scan of the named-parameter read.

Testing

Unit tests are banned here: none added. All runs are of 4140b96, against main at df96767 or the clean tags; Autodesk's sample projects, the reference models, the BIM4LCA offices, the Revit 2025 models and the held-out models are fetched for the runs and never stored (held-out as counts).

• The unfiltered Measure, run 42125670870 (-f base=main -f nordic=true -f r2025=true -f holdout=true -f samples=true -f timing=true, default scorers, no ifc_flags): byte identity of every model and the timing against main, above. Its same-project gates job wrote the check run 127599178067: success, "pass: 9 samples held to their earlier saves, and the Areas on 4 saves and on the 2021 save". The probe run below, dispatched on the same commit and finished 30 seconds later, wrote a neutral "NOT MEASURED: the scorers asked for exclude bodies_across_saves" (127599399222) that GitHub lists as the latest check of that name; the success is listed with filter=all.
• The projected decode against the full decode, run 42125686355 (-f probe=probe_1335_projected_graphs -f samples=true -f r2025=true -f scorers=none; every sample of 2015 to 2027, the reference models and the Revit 2025 models, 88 files): 1,128,043 records, 6,768,258 projected decodes under six lists, 0 mismatches; the 4,743 records the full decode refuses are refused alike (FamilyInstance 3,403, VarSketch 1,125, ImageHolder 105, AppearanceAssetElem 38, RebarShape 30, others fewer). No Family or FamilySymbol is among them: the registry the definitions pass builds is the same from both graphs on every file. wall_facts is the same on all 7,900 SWall and ArcWall records. The catalog's document gives the same catalog and provenance on 79 files and is refused alike on 9. Not measured: the BIM4LCA offices and the held-out models under the probe.
• Time against 0.11.0 and 0.15.0, runs 42125703262 and 42125717387 (timing jobs, the 33 samples of 2015 to 2023 and the six reference models, one runner per shard, an EPYC 9V74 for the samples in both and a 9V74 and a 9V45 for the models; median of three exports each; sum of medians, mean and median of the per-file ratios; Revit 2024 to 2027 samples were not timed against the tags):

class	files	0.11.0 (s)	PR (s)	sum	mean	median	per file
samples 2015	6	6.2	7.7	1.236	1.237	1.260	1.07 to 1.31
samples 2016 to 2018	9	6.6	8.0	1.213	1.230	1.282	1.07 to 1.33
samples 2019 to 2022	15	13.7	16.6	1.216	1.222	1.254	1.06 to 1.33
samples 2023	3	4.3	3.8	0.894	0.914	0.950	0.79 to 1.00
33 samples	33	30.7	36.1	1.175	1.199	1.254	0.79 to 1.33
reference models	6	9.4	7.3	0.772	0.811	0.830	0.73 to 0.89

class	files	0.15.0 (s)	PR (s)	sum	mean	median	per file
samples 2015	6	9.0	7.8	0.870	0.870	0.868	0.85 to 0.89
samples 2016 to 2018	9	9.0	8.0	0.890	0.895	0.900	0.86 to 0.92
samples 2019 to 2022	15	19.0	16.7	0.882	0.889	0.894	0.86 to 0.92
samples 2023	3	4.3	3.9	0.913	0.915	0.914	0.90 to 0.93
33 samples	33	41.3	36.5	0.884	0.890	0.894	0.85 to 0.93
reference models	6	5.8	5.6	0.961	0.940	0.936	0.91 to 0.98

Peak memory against 0.11.0 is 0.61 to 0.96 on the samples (the models 0.73 to 1.03), against 0.15.0 0.86 to 1.00 on the samples (the models 0.82 to 1.00).

• Time against main, the timing job of the unfiltered run (one runner per shard; shard 1 an EPYC 7763 for the 2024 samples, shard 2 a 9V74 for the samples to 2023 and the reference models, shard 3 a 7763 for the offices and the 2027 samples, shard 4 a 9V45 for the 2025 and 2026 samples):

class	files	main (s)	PR (s)	sum	mean	median	per file	peak memory, PR / main
samples 2015	6	9.1	7.9	0.867	0.868	0.863	0.84 to 0.90	0.85 to 1.00
samples 2016 to 2018	9	9.3	8.2	0.878	0.883	0.886	0.85 to 0.91	0.86 to 1.00
samples 2019 to 2022	15	19.7	17.0	0.862	0.871	0.875	0.83 to 0.90	0.86 to 0.99
samples 2023	3	4.3	3.9	0.903	0.906	0.905	0.89 to 0.92	0.89 to 1.00
samples 2024 to 2027	25	160.9	155.7	0.968	0.963	0.964	0.89 to 1.00	0.54 to 1.03
reference models and offices	10	61.1	59.4	0.972	0.946	0.950	0.88 to 0.98	0.82 to 1.01
58 samples	58	203.3	192.6	0.947	0.914	0.901	0.83 to 1.00	0.54 to 1.03
all files	68	264.4	252.0	0.953	0.919	0.914	0.83 to 1.00	0.54 to 1.03

Earlier heads of the three parts (the decoder is the same; each slice measured against the one before it): the definitions part 0.919 of main over the 33 samples (run 42115512932, head fa10c6d), the wall records part 0.976 (run 42116989088, head 16527ce) and the preamble part 0.988 (run 42117016111, head 76e98d7). The wall records part raised the peak of 2019 and 2020 rac_advanced from 168 to 173 MiB (1.03) in its run; the definitions part had taken them from 173 to 168, so they are at what main peaks at.

• tools/check-local.sh: nothing is built on this Mac (no local compute); CI's fast jobs pass on this head.

Audit-honesty checklist

• No capability changes; no export is meant to change, and none does in the 111 exports of the unfiltered run. Nothing here is a claim about Revit's own export.
• A user-visible change has its entry as a fragment under changelog.d/changed/.
• No limitation resolved or added. Revit 2015–2023 exports: the remaining time against 0.11.0 after #1237 #1335 stays open.
• Opt-in costs nothing when off: the plan of a class has an empty kept and no duplicate unless the decode is projected.
• No emojis, no private paths.

Breaking changes

None to the stable API, the CLIs or any export. rvt::research (unstable): native_document::Options gains graph_fields (and GraphFields); native_parameters gains decode_graph_projected; native_walls gains WALL_READ_FIELDS, WallFacts and wall_facts; native_es_catalog gains CATALOG_FIELDS, document, document_limits and from_graph; native_parameter_definitions gains FAMILY_BINDING_FIELDS, SYMBOL_BINDING_FIELDS and binding_context_fields.

Related issues

Refs #1335, which stays open.