Memory footprint round and streamed record execution (v2.2.0)

CHANGELOG.md

@@ -4,6 +4,39 @@ Notable changes, newest first. Versions follow [semantic
 versioning](https://semver.org): from 1.0.0 the exported API is stable, and a
 breaking change means 2.0 with a new module path. See *Stability* below.
 
+## Unreleased
+
+Memory footprint work: what a tree holds and what a transform peaks at. Output
+is byte-identical; new API is additive.
+
+### Added
+
+| Change | What it does | Commit |
+|---|---|---|
+| Streamed aggregates over records | A lone `value-of`/`copy-of`/`sequence` over one record path streams too; `count`/`sum`/`min`/`max` fold per record: 1 GB summed in 33 MB RSS. Any error re-runs on the tree. | [`457aa03`][457aa03], [`9f3d815`][9f3d815], [`48b1a38`][48b1a38], [`3157566`][3157566], [`4944511`][4944511] |
+| Streamed `xsl:source-document`, first stage | A streamable `xsl:for-each`/`xsl:iterate` over a child path reads one record at a time: 1 GB totalled at 27 MB RSS, not 3.7 GB, at about 2× CPU. Other shapes build the tree. | [`3f185fe`][3f185fe], [`27757b3`][27757b3] |
+| `xdm.ParseBursts` | Parses a document handing over one selected element at a time in a tree of its own, with limits counted over the whole document; memory stays at the spine plus one record. | [`b4b6052`][b4b6052], [`80f999f`][80f999f] |
+| `xslt.FileResolver.MaxNodes`; `MaxBytes` reaches the parser | Raising `MaxBytes` past 64 MB had no effect on `fn:doc`, the parser kept its own default. Both limits now apply to what the resolver parses. | [`80f999f`][80f999f] |
+| `xdmbuild.InPlaceDocument` | A builder rooted at a new document node, for content built straight into it. | [`302c932`][302c932] |
+
+### Changed — streamability analysis
+
+| Change | Problem → solution | Commit |
+|---|---|---|
+| Six constructs escaped the §19.8 check and compiled | A nested `xsl:map-entry`, extension instructions (their `xsl:fallback`), `xsl:result-document` serialization AVTs, boolean pattern predicates, `mode="#all"` rules and rule `xsl:param` defaults are now judged: such stylesheets get XTSE3430. | [`b5703be`][b5703be], [`8c391d8`][8c391d8], [`ff0ed5e`][ff0ed5e], [`ca1b029`][ca1b029], [`268acce`][268acce], [`7920635`][7920635] |
+| A verdict could rest on a declaration nobody checked | Verdicts carry an unverified bit for streamable functions, attribute sets, accumulators, modes and other modules; streamed execution requires it clear. No refusal changes. | [`b12ecbf`][b12ecbf], [`e94cdf9`][e94cdf9] |
+
+### Changed — memory
+
+| Change | Problem → solution | Commit |
+|---|---|---|
+| Namespace nodes lived as long as their tree | `//namespace::*` on 5 MiB kept 140 MB. They are held weakly, 16 to a chunk, and freed when nothing holds them: under 0.5 MB after the query. | [`ac4973e`][ac4973e], [`fcab85a`][fcab85a], [`e1b5e1f`][e1b5e1f] |
+| Built-in shallow-copy built a document result twice | In a mode with no template rules it builds in place: parse-100mb identity peak 1091 → 930 MB, CPU −25 to −40%. | [`302c932`][302c932] |
+| Built trees stored every whitespace-only text value | They share one value per string, as a parse does: an identity result is the size of its source again. | [`c60f8f3`][c60f8f3], [`09f0c5d`][09f0c5d] |
+| A deep copy duplicated the source's text | `xsl:copy-of` of 4,096+ records borrows the source blocks it fills by half or more: XMark identity result −28%, peak −19%. | [`546af03`][546af03], [`b4cd614`][b4cd614] |
+| Every `copy-of` was kept until the transform ended | Each copied node was recorded against its original for accumulators, even with none declared: a 10 MB stream copying each record peaked at 830 MB. The table exists only when an accumulator does: 2.8 MB. Also in v2.1.1. | [`c7952b0`][c7952b0] |
+| Full record chunks were padded 6% by the malloc header | They are allocated two at a time past the 8th, 40,960 B exactly: parse-100mb retained 370 → 350.5 MB. | [`907c674`][907c674] |
+
 ## v2.1.1 — 2026-11-01
 
 Memory and CPU fixes from go-xmlsec's measurements against v2.1.0. No API
@@ -1830,3 +1863,30 @@ here so every entry in this file sits under a release.
 [9572c44]: https://github.com/knroy/go-xml/commit/9572c44
 [fd9750e]: https://github.com/knroy/go-xml/commit/fd9750e
 [5712d9f]: https://github.com/knroy/go-xml/commit/5712d9f
+[3f185fe]: https://github.com/knroy/go-xml/commit/3f185fe
+[27757b3]: https://github.com/knroy/go-xml/commit/27757b3
+[b4b6052]: https://github.com/knroy/go-xml/commit/b4b6052
+[80f999f]: https://github.com/knroy/go-xml/commit/80f999f
+[302c932]: https://github.com/knroy/go-xml/commit/302c932
+[ac4973e]: https://github.com/knroy/go-xml/commit/ac4973e
+[fcab85a]: https://github.com/knroy/go-xml/commit/fcab85a
+[e1b5e1f]: https://github.com/knroy/go-xml/commit/e1b5e1f
+[c60f8f3]: https://github.com/knroy/go-xml/commit/c60f8f3
+[09f0c5d]: https://github.com/knroy/go-xml/commit/09f0c5d
+[546af03]: https://github.com/knroy/go-xml/commit/546af03
+[b4cd614]: https://github.com/knroy/go-xml/commit/b4cd614
+[907c674]: https://github.com/knroy/go-xml/commit/907c674
+[c7952b0]: https://github.com/knroy/go-xml/commit/c7952b0
+[b5703be]: https://github.com/knroy/go-xml/commit/b5703be
+[8c391d8]: https://github.com/knroy/go-xml/commit/8c391d8
+[ff0ed5e]: https://github.com/knroy/go-xml/commit/ff0ed5e
+[ca1b029]: https://github.com/knroy/go-xml/commit/ca1b029
+[268acce]: https://github.com/knroy/go-xml/commit/268acce
+[7920635]: https://github.com/knroy/go-xml/commit/7920635
+[b12ecbf]: https://github.com/knroy/go-xml/commit/b12ecbf
+[e94cdf9]: https://github.com/knroy/go-xml/commit/e94cdf9
+[457aa03]: https://github.com/knroy/go-xml/commit/457aa03
+[9f3d815]: https://github.com/knroy/go-xml/commit/9f3d815
+[48b1a38]: https://github.com/knroy/go-xml/commit/48b1a38
+[3157566]: https://github.com/knroy/go-xml/commit/3157566
+[4944511]: https://github.com/knroy/go-xml/commit/4944511

README.md

@@ -139,7 +139,7 @@ and maintains it as a project of his own.
 | **XSD 1.1** | 99.98% instance (26,217 of 26,222); **99.97%** schema-validity (15,350 of 15,354); opt-in via `Version11` |
 | **RELAX NG** | 100.00% of James Clark's spectest (965 of 965 assertions); XML and compact syntax, from Go and from `go-xml validate -rng` |
 | **DTD** | content models, attribute defaults, enumerations, `ID`/`IDREF`; external subset, parameter entities across both subsets, conditional sections — via `dtd.Load` with a caller-supplied resolver, nothing fetched by default |
-| **Tests** | 2,960 `func Test` declarations, clean under `-race` (a few subtests skip without the corpora below) |
+| **Tests** | 2,988 `func Test` declarations, clean under `-race` (a few subtests skip without the corpora below) |
 | **Production schemas** | UBL 2.1, UN/CEFACT CII, Factur-X/ZUGFeRD, Peppol BIS 3.0 — 88 schemas load, instances validate clean |
 | **API** | 1.2; the exported surface is stable and additive over 1.1, and a breaking change means 2.0 with a new module path |
 
@@ -178,7 +178,12 @@ true here:
    each test's declared version dependency — a different kind of measurement
    from running a suite written for the version under test.
 
-   **Streamed execution is not implemented**, but the streaming cases are in
+   **Streamed execution is implemented only for two narrow shapes** (an
+   `xsl:source-document streamable="yes"` whose body is one `xsl:for-each` or
+   `xsl:iterate` over a child path, or one `xsl:value-of`/`xsl:copy-of`/
+   `xsl:sequence` over one record path such as `sum(doc/rec/@v)`, read one
+   record at a time; see
+   [known gaps](docs/known-gaps.md)). Otherwise the streaming cases are in
    scope and counted: the stylesheet-level half of streaming, the §19.8
    streamability analysis that refuses a construct with `XTSE3430`, is
    implemented and measured. What is absent is streamed *execution*, not the
@@ -1388,7 +1393,7 @@ back, is in [docs/testing.md](docs/testing.md).
 
 | method | what it catches | what it misses |
 |---|---|---|
-| **Unit tests** (2,960 `func Test` declarations) | places where a plausible implementation is quietly wrong | anything nobody thought to write a test for |
+| **Unit tests** (2,988 `func Test` declarations) | places where a plausible implementation is quietly wrong | anything nobody thought to write a test for |
 | **Spec inventories** | features absent entirely | features present but behaving wrongly |
 | **Saxon differential** | subtle behavioural divergence on real stylesheets | constructs the corpora do not use |
 | **W3C QT3 suite** | systematic conformance across 22,054 XPath and 30,517 XQuery cases | XSLT (it is an XPath suite) |

docs/conformance-gaps.md

@@ -49,7 +49,7 @@ W3C disagreements: 0 + 0 + 0 + 1 + 8 + 23 + 30 + 31 + 0 = 93. Measured 2026-09-1
 <!-- BEGIN GENERATED UNIT TEST COUNT -->
 <!-- Generated from tests/conformance/results.json and the source tree by
      tests/conformance-docs.go. Do not edit; see docs/stats.md. -->
-The unit-test suite is 2,960 tests.
+The unit-test suite is 2,988 tests.
 <!-- END GENERATED UNIT TEST COUNT -->
 
 The last two rows are not W3C suites but real-world corpora — DocBook xslTNG's

docs/known-gaps.md

@@ -128,9 +128,10 @@ Found by the third memory audit (2026-11-01) and not fixed, each with the reason
   (40k items 12.4 GB, all garbage). Appending in place is unsound while other
   references may share the prefix.
 - **Identity templates allocate about 3.5 KB a node of context copies**
-  (`withSel`, `hostWithCurrent`, the leaf-hook copy), and built-in shallow-copy
-  copies a document-level result twice; removing that copy changes an error
-  case into output.
+  (`withSel`, `hostWithCurrent`, the leaf-hook copy). Built-in shallow-copy of
+  a document node builds the result once in a mode with no template rules
+  (v2.2.0), but still builds it and copies it into a document in a mode that
+  has rules, where a rule could add an attribute that must stay `XTDE0420`.
 - **Several large text values in one document cost about 1.3× their size to
   parse** (eight 3 MiB values: 1.30×). A single value is about 1.1× at every
   size since v2.1.1; with several, the first value's buffer overshoots once
@@ -150,6 +151,83 @@ Found by the third memory audit (2026-11-01) and not fixed, each with the reason
 - **A regex program over 4,096 instructions is compiled on every call**,
   because it is not cached; the largest QT3 pattern is 4,059.
 
+### Memory: what the footprint round left open (performance)
+
+Measured on branches in 2026-11 and not adopted, each with the reason:
+
+- **A 32-byte node record** (the tree pointer read from a chunk header):
+  −11 to −22% retained, rejected because a by-value copy of an `xdm.Node`
+  would then read foreign memory instead of failing safely.
+- **Compact attribute records** (16 B, nodes made on demand): −10% retained on
+  parse-100mb, −16% on SAML, about 0 on XMark, but +10–25% CPU on
+  attribute-heavy XPath, c14n and copies (+45% worst case), and a held
+  attribute would cost 62 B instead of 40. Pointer identity of `*xdm.Node`
+  forces a real record per held node; denser storage needs a value-type node
+  handle, which is a breaking change.
+- **Template-driven shallow-copy does not share source text.** Deep copies
+  (`xsl:copy-of`, `xdm.Copy` of 4,096+ records) borrow the source's text
+  blocks; a template-built copy adds each value as a string.
+- **With any accumulator declared, every `fn:copy-of`/`fn:snapshot` copy is kept
+  until the transform ends**, with its original: §18.2.2 and §18.2.10 make the
+  copy answer with the original's accumulator values, and the copy-to-original
+  table is how it finds them. Weak references would not free them, since the
+  accumulator value cache holds every tree an accumulator was evaluated over.
+  With no accumulator in any package the table is not made.
+- **Holding millions of namespace nodes costs a sweep per collection.**
+  Namespace nodes are held weakly since v2.2.0; while a tree's table is not
+  empty, every GC cycle sweeps it.
+
+### Streamed execution covers two shapes (XSLT 3.0, performance)
+
+Since v2.2.0 an `xsl:source-document streamable="yes"` whose body is one
+`xsl:for-each` or `xsl:iterate` (optionally inside constant literal result
+elements) over a child path of plain names, with non-positional predicates on
+the last step only, and a body the gate recognises with a known and verified
+§19.8 verdict, runs record by record
+through `xdm.ParseBursts`: a 1 GB file totalled by `xsl:iterate` peaks at
+27 MB RSS against 3.7 GB, at about 2× the CPU. Everything else builds the tree,
+as before.
+
+The second shape is a body that is one `xsl:value-of`, `xsl:copy-of` or
+`xsl:sequence` (again optionally inside constant literal result elements) whose
+select reads one record path, such as `sum(doc/rec/@v)` or
+`count(doc/rec[@k = 'x'])`. The path runs per record; the rest of the select
+runs once over the values. `count`, `sum`, `min` and `max` fold per record and
+stay flat: over a 1 GB file of 25.9M records, `count` peaks at 3.4 MB heap /
+25 MB RSS and `sum` at 11 MB / 33 MB, where tree mode needs 773–1,284 MB at
+100 MB. Other consumers (`avg`, comparisons, `string-join`, a bare `value-of`
+of attributes) keep one atomic value per item, about half the tree's memory,
+still growing with the input. A select that would keep the records' nodes or
+copies (`copy-of(doc/rec)`, filters returning nodes, `head`/`tail` passed on)
+is refused and runs on the tree, because a kept record costs more there than in
+a whole tree (945 B against 208 B). Also refused: `distinct-values`, `reverse`,
+`sort`, `last()`, union/intersect/except, `let`, user functions, wildcard steps.
+
+In the W3C XSLT 3.0 run 364 cases take the burst path (9 before the second
+shape). Of the 786 single-instruction bodies over one record path with a known
+and verified verdict, 294 are admitted (261 bounded reducers, 33 pipelined).
+Also:
+
+- **The result is still built in memory**, so a large filtered copy grows with
+  its output and stops at the result-node cap.
+- **Error order can differ from tree mode, for the loop shape.** (The single-instruction
+  shape re-runs on the tree after any error, so its errors are the tree's; the
+  cost is a second read of the document, and the 1 GiB string budget charges the
+  strings both attempts built.) Tree mode reads the whole file,
+  then evaluates the whole `select`, before the body runs; bursts do each per
+  record. So a dynamic error in the body is raised before a malformed tail, a
+  `MaxNodes`/`MaxBytes` refusal or an encoding error later in the file, and
+  before a predicate's error on a later record (`a/r[xs:integer(@v) gt 0]`
+  with `1 idiv 0` in the body: FOAR0001 in bursts, FORG0001 in the tree).
+  `xsl:try` can see which code wins. A differential fuzzer (1.28M generated
+  cases, 495k on the burst path) found no other difference.
+- **`xsl:break` still reads the rest of the file**, so that a malformed tail is
+  an error as in tree mode.
+- **Only `FileResolver` streams**; another `DocumentResolver` builds the tree.
+  The streamed document is not cached, so a later `fn:doc` of it reads it again.
+- **The `FileResolver` limits apply to the whole file**: raise `MaxBytes` and
+  `MaxNodes` to stream a document past 64 MB or 10M nodes.
+
 ### `fn:replace` charges a backtracking or backreference result after building it (XPath, resource limits)
 
 The RE2 path of `fn:replace` charges `len(input) + matches × len(replacement)`
@@ -179,7 +257,8 @@ and §18.1's grounded demand at both the sites it names.
 
 All three fail in the same direction: the suite expects `XTSE3430` — *this
 construct is not guaranteed streamable* — and the transform succeeds instead.
-The engine builds a tree and streams nothing, so it produces the **right
+The engine builds a tree for these (it reads in bursts only the narrow shape
+described under *Streamed execution covers two shapes*), so it produces the **right
 answer** for all three; what it does not produce is the static refusal §19.8
 requires a streaming processor to make before running anything. These are not
 wrong answers; they are a static analysis that is not complete.
@@ -189,7 +268,13 @@ reports whether it *modelled* every construct it met separately from what it
 concluded, and an error is raised only on a fully-modelled verdict; anything
 else is "no opinion". A missing rule leaves a case failing, while a wrong rule
 would reject a valid stylesheet, and it is the second that the whole-corpus
-scan measures at zero.
+scan measures at zero. Since v2.2.0 a verdict also carries an *unverified* bit:
+it is set when the verdict rests on a streamable function, attribute set,
+accumulator or mode whose own check gave up, or on a declaration in another
+module. Refusals ignore the bit, so no XTSE3430 result moves; streamed execution
+requires it clear. In the W3C streaming stylesheets the static checks accept,
+58 known, grounded containers in 41 files are unverified (modes 28, functions
+10, cross-module 10, accumulators 7, other 3).
 
 **All three are measured, and each known fix costs more than it gains.**
 Modelling `||` refuses `si-fork-902`/`-952` and costs 18 `sx-union` cases
@@ -255,7 +340,10 @@ None is a wrong verdict.
   other arity is unmodelled.
 - `*xpath.LookupExpr` (`?`) is modelled in `expr()` (§19.8.8.18), but the
   inline-function walker `bodyMentionsStreamingParam` still lacks it, along
-  with `*xpath.StringConcat` and `*xpath.ArgumentPlaceholder`. The
+  with `*xpath.StringConcat` and `*xpath.ArgumentPlaceholder`. Since v2.2.0 the
+  walker runs on every inline function in a streamable construct (it looks for
+  `current()`), so such a body is unknown anywhere, not only in a stylesheet
+  function. The
   dispatcher half of `||` is measured and reverted; see
   [*The `||` operator*](conformance-gaps.md#the--operator--2-cases-gained-18-lost).
 - `functionParamTypes` splits a declared function type by hand: the comma in
@@ -1281,6 +1369,11 @@ dangerous than an open bug, and deleting one invites the same probe again.
 
 ### "An inline function declaration is grounded" — only if it mentions nothing streamed
 
+Since v2.2.0 two more closures are unknown rather than grounded: one over a
+quantified range variable bound to streamed nodes, and one calling `current()`
+or `current#0`. Read literally, §19.8.8.16 grounds both; withholding the verdict
+keeps execution from trusting it without refusing anything.
+
 Three §19.8.8 expression rules were proposed with code to implement them, and
 reading the Recommendation first changed two of them.