diff --git a/CHANGELOG.md b/CHANGELOG.md
index 47795402..b0d0f420 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,92 @@ Notable changes, newest first. Versions follow [semantic
versioning](https://semver.org): from 1.0.0 the exported API is stable, and a
breaking change means 2.0 with a new module path. See *Stability* below.
+## Unreleased
+
+A memory audit across parsing, XPath/XQuery, XSLT, validation and
+serialization. Output is byte-identical except where *Changed* says otherwise.
+Three new exported functions make this a minor release.
+
+### Added
+
+| Change | What it does | Commit |
+|---|---|---|
+| `xpath.SerializeJSONTo`, `xpath.SerializeAdaptiveTo` | The json and adaptive methods written to an `io.Writer` instead of returned as a string. | [`599b2f9`][599b2f9] |
+| `xdm.Node.AddNamespaces` | Sets several namespace bindings on an element in one call. | [`b42f04a`][b42f04a] |
+| `xdm.Node.Index`, `Node.SubtreeEnd`, `Tree.Len`, `Tree.NodeAt` | A node's dense, document-order position in its tree (attributes included, namespace nodes -1), stable once the node exists; size a table with `SubtreeEnd() - Index()` and look nodes up by position. | [`b37796b`][b37796b] |
+| `xdm.Node.InScopeNamespaceBindings` | Iterates an element's in-scope prefixes and URIs without making a namespace node for each. | [`73249e2`][73249e2] |
+| `xdm.ArrayItem.AppendMembers` | Appends an array's members to a slice without an intermediate copy. | [`59ac569`][59ac569] |
+| `xpath.CompileBounded`, `xpath.MaxRegexpProgram` | `regexp.Compile` that refuses a program over 100,000 instructions; used by XPath and XSD patterns. | [`c1c2bb9`][c1c2bb9] |
+| `xpath.Env.MaxDigits`, `xpath.MaxDigits` | Bounds the digits of an `xs:integer` or `xs:decimal` cast or arithmetic result; 0 is 10^6, negative is none. | [`8577906`][8577906] |
+| `xdm.AppendIntegers`, `xpath.Context.WithVars` | Appends a run of integers allocated a chunk at a time; binds several variables in one scope. | [`30c7c07`][30c7c07], [`5951f24`][5951f24] |
+| `xpath.Context.NodesRefused`, `ItemsRefused` | Whether the node budget was crossed, and whether a running item total is past the item bound, without charging either. | [`4e8f0a5`][4e8f0a5], [`21995a6`][21995a6] |
+
+### Changed
+
+| Change | What to do | Commit |
+|---|---|---|
+| `xsd` and `dtd` error messages clip document-supplied strings to 1 KB plus "..." | Nothing, unless you match messages longer than 1 KB. Message wording is not part of the contract. | [`20958aa`][20958aa], [`bd865e4`][bd865e4] |
+| json and adaptive serialization can leave partial output when an error is raised mid-document | Treat the writer's content as invalid on error, as the XML methods already required. | [`599b2f9`][599b2f9] |
+| New refusals: maps and arrays nested past 1,000 levels in json/adaptive output (XPDY0130), `deep-equal` and `map:find` (XPDY0001), with their own counter, apart from function recursion; regexes over 100,000 instructions (FORX0002, or a pattern error in XSD); picture widths over 65,536 (FOFD1340); `xsl:key` and `group-by` keys past 5,000,000 retained (XPDY0130); integers over 10^6 digits (FOCA0003, FOAR0002); sequences built by `,` or `!`, `xsl:next-iteration` values, `unparsed-text-lines` lines, `parse-json`/`json-doc` values and stored accumulator items past `MaxItems` (XPDY0130); `xsl:copy-of` of a document past `MaxNodes` (XPDY0130) | Nothing for real inputs: the largest W3C suite regex is 4,313 instructions. All wrap `xdm.ErrResourceLimit`. | [`702c6e6`][702c6e6], [`014a862`][014a862], [`c1c2bb9`][c1c2bb9], [`21995a6`][21995a6], [`8577906`][8577906], [`dcb49b2`][dcb49b2], [`c73a06b`][c73a06b], [`a3d8ef6`][a3d8ef6], [`40cd52c`][40cd52c], [`b35ec14`][b35ec14] |
+| The CLI drops its collector from `GOGC=200` to 100 once the live heap passes 128 MB | Nothing; set `GOGC` to keep a fixed value. Peak RSS −20% on DocBook's largest table, about 12% more user CPU. | [`f3449f7`][f3449f7] |
+| The `FileResolver` no longer evicts to admit a document that does not fit beside its cache | A document refused admission is parsed again on its next request, so across transforms it is not the same tree; within one transform it is. | [`f49ff92`][f49ff92] |
+| `xdm.ProcessXInclude` returns a parsed tree unchanged when it has no XInclude element | Nothing; a parsed tree is frozen. A built tree is still copied. | [`e139525`][e139525], [`9c3a983`][9c3a983] |
+
+### Fixed
+
+| Change | Problem → solution | Commit |
+|---|---|---|
+| `xsd` error messages kept attacker strings whole | A 1 MB namespace in 200 attributes kept 105 MB. `fail` and `attrName` clip to 1 KB: 0.13 MB. | [`20958aa`][20958aa] |
+| The DTD validator built every element's path, quadratic in depth | A valid 990-deep chain of 4 KB names allocated 2 GB. The path is rendered only for a failure, text unchanged. | [`8dcb07f`][8dcb07f] |
+| DTD validity messages kept `#FIXED`, enum and ID values whole | 200 mismatches on a 1 MB `#FIXED` value kept 105 MB. `fail` clips to 1 KB. | [`bd865e4`][bd865e4] |
+| The nested `fn:transform` cache kept compiles for dead resolvers | A fresh resolver per call kept 49 MB on DocBook. Resolvers are weak keys and dead entries are dropped: 0.3 MB. | [`a5b385d`][a5b385d] |
+| Sequences built by `,` or `!` were not counted against `MaxItems` | A 70-byte doubling `fold-left` built 134M items against the 5M limit, at 6.3 GB; through recursive `with-param` too. They are charged above 64 items. Also in v2.0.3. | [`dcb49b2`][dcb49b2] |
+| RELAX NG reported a valid document invalid | `(a?){13} a{13}` against 13 `` hit the derivative cap, naming a `oneOrMore` the schema lacks; `xmllint` accepts it. Choice chains merge alternative by alternative: N = 12, 13, 18, 30 validate. Also in v2.0.3. | [`44229aa`][44229aa] |
+| Serializing a sequence of atomic values was quadratic | `1 to 100000` took 21.6 s and 6.95 GB, joined by repeated concatenation. Each run is joined once: 0.056 s. Also in v2.0.3. | [`a9ad280`][a9ad280] |
+| Text lines, JSON values, DTD default values, integer digits and accumulator values grew past the limits | `unparsed-text-lines` kept 5.2 GB, 2 MB of DTD defaults 1.97 GB, a 4M-digit cast 34 GB, a 200 KB accumulator 12.4 GB. Each is charged, bounded, stored once or shared. Also in v2.0.3. | [`c73a06b`][c73a06b], [`be5afc3`][be5afc3], [`8577906`][8577906], [`40cd52c`][40cd52c] |
+| `xsl:copy-of` of a document was not counted against `MaxNodes` | It copied 5.7M nodes and exited 0, where shallow-copy is refused. It is charged before copying. | [`b35ec14`][b35ec14] |
+| Deeply nested maps and arrays crashed the process | `serialize(..., json)` or `deep-equal` on a 400,000-deep array from a 96-byte query overflowed the Go stack. They are refused with a resource-limit error; `xdm.Flatten` is iterative. Also in v2.0.3. | [`702c6e6`][702c6e6] |
+| A built-in `shallow-copy` rule returned a truncated result past `MaxNodes` with no error | A 1M-element identity copy emptied a third of its elements and exited 0. The transform now fails with XPDY0130 whenever the node budget was crossed. Also in v2.0.3. | [`4e8f0a5`][4e8f0a5] |
+| Sequential regex repeats compiled to huge programs | `\pL{1000}` repeated 150 times passes Go's 1,000-repeat check: 181 KB of patterns kept 984 MB. Programs over 100,000 instructions are refused in XPath and XSD. | [`c1c2bb9`][c1c2bb9] |
+| `fn:replace`, picture widths, `xsl:key` and `group-by` grew past their budgets | `replace` charged after building (12.8 GB for 500 KB in), `[D01,1000000000]` allocated 2.8 GB, and `use="1 to 1000"` over 100k nodes 3.2 GB. Each is charged or refused first. | [`c1c2bb9`][c1c2bb9], [`21995a6`][21995a6] |
+| Caches bounded by entries held gigabytes of long keys or documents | Regex, collation and char-atom keys over 1 KB are not cached; the `FileResolver` document cache is also bounded by 32 MB of source. | [`c1c2bb9`][c1c2bb9], [`de4a515`][de4a515] |
+
+### Changed — memory
+
+| Change | Problem → solution | Commit |
+|---|---|---|
+| Namespace URIs were tokeniser-arena strings | Each parsed tree kept a 32 KB arena block alive. URIs are copied at declaration: DocBook trees retain 31.6 → 12.3 MB. | [`18d374d`][18d374d] |
+| Every parse allocated a 32 KB string-arena block | The first block is sized to the input: DocBook corpus parse 37.6 → 19.6 MB a pass. | [`9e13f70`][9e13f70] |
+| `TrackPositions` kept the source text to count lines | Line starts are recorded as the text is read: a tracked 10 MB tree retains 50.4 → 41.5 MB. | [`ec1ac54`][ec1ac54] |
+| Parsed trees kept build-only indexes, slack and copied long text twice | Indexes dropped at freeze, the last text block trimmed, long text handed over: DocBook retains a further −10%. | [`2ad3a32`][2ad3a32], [`a5ab736`][a5ab736], [`4440665`][4440665] |
+| Each parse allocated three read windows | They are pooled: DocBook corpus parse 19.8 → 17.1 MB a pass. | [`6345114`][6345114] |
+| `isBase64Binary` copied each value into a `[]rune` | One byte pass: a 380 KB value 1.54 MB → 0 allocations. | [`5d8fceb`][5d8fceb] |
+| DTD content matching, ID bookkeeping and bound facets allocated per child or value | Two reused state sets, one ID map, bounds parsed once: `dtd` validation −47%, `xsd` instance −24%. | [`e056467`][e056467], [`f09c1a3`][f09c1a3], [`14c84b5`][14c84b5] |
+| XQuery FLWOR tuples copied names, rebuilt values and grew results by doubling | Shared names, own value, reserved results, version set once, no per-call library: XMark q1–q20 eval 194.9 → 119.0 MB. | [`ba9e7a0`][ba9e7a0], [`f890740`][f890740], [`a8d6e7b`][a8d6e7b], [`24b963c`][24b963c], [`73af7a8`][73af7a8], [`f148247`][f148247] |
+| `a to b` and `fn:distinct-values` used big arithmetic and eager promotion keys | int64 ranges, keys written once a double arrives: `distinct-values(1 to 100000)` 96.0 → 53.2 MB. | [`70ff980`][70ff980], [`ef5f064`][ef5f064] |
+| Compiled stylesheets held ~980 copies of one namespace map and pinned module trees | Interned by content; instructions keep the map, not the element: compiled CEN 9.1 → 6.1 MB, DocBook 12.4 → 9.3 MB. | [`490fc5a`][490fc5a], [`dcb15d1`][dcb15d1] |
+| XSLT calls built parameter maps, memo keys, namespace frames and error text per call | A slice of pairs, a reused key buffer, one-step frames, lazy text, shared global cells: DocBook section.001 −15%. | [`e3b6288`][e3b6288], [`b5aaa52`][b5aaa52], [`b42f04a`][b42f04a], [`b6d4e4e`][b6d4e4e], [`327b09e`][327b09e] |
+| The json, adaptive and text methods built the whole output as a string | They stream to the writer: xmark json 152.7 MB → 12.6 KB, text 41.7 MB → 33 KB. Normalized output keeps the string. | [`599b2f9`][599b2f9], [`cf65d62`][cf65d62] |
+| CDATA sections, character maps and `fn:serialize` escaping copied per node | Written straight to the sink: xmark CDATA 20.4 MB → 34 KB, character map 16.5 MB → 34 KB. | [`cf65d62`][cf65d62], [`599b2f9`][599b2f9] |
+| c14n `Bytes` and `Result.String` doubled a buffer from empty; `Serialize` allocated 32 KB a call | Output chunks are joined once and the write buffer is pooled: c14n 3.7× → 2.1× the output, a small result 33.6 → 0.7 KB. | [`f235795`][f235795], [`9496810`][9496810], [`37206b1`][37206b1], [`c501fc4`][c501fc4], [`1f18b1b`][1f18b1b] |
+| The `xslt` benchmarks read inputs no checkout has, so they always skipped | They run XRechnung and Peppol CEN from `testdata/bench`. | [`beb6072`][beb6072] |
+| `-xinclude` copied every tree, and included documents twice, even with nothing to include | A parsed tree with no XInclude element is returned as is and inclusions are built into the destination: `parse-100mb` 514 → 0 MB. | [`e139525`][e139525] |
+| RELAX NG copied its name memo on every store | Stores land in a pending map published as a snapshot, and after 8 locked hits: 1,024 stores 47.5 → under 2 MB, lookups lock-free. | [`9f0db9c`][9f0db9c], [`676ad52`][676ad52] |
+| Integer `mod`, `idiv`, negation and comparison used big arithmetic; ranges made 3 allocations an integer | int64 fast paths and chunked ranges: comparison 2 → 0 allocations, `sum(1 to 1e6)` 800k → 4.8k allocations. | [`40085f6`][40085f6], [`30c7c07`][30c7c07] |
+| Dynamic calls, FLWOR tuples and window clauses made a scope per variable; closures kept their whole scope | One scope per binding set, and a closure keeps only what its body reads: `$f($i)` over 1M 2,673 → 1,617 MB. | [`5951f24`][5951f24], [`a4f316d`][a4f316d], [`b5aaf06`][b5aaf06] |
+| The function result memo kept each key string | It keeps a 128-bit SHA-256 hash: DocBook `table-cals.049` live heap 359 → 172 MB, peak 1,150 → 533 MB. | [`15273c7`][15273c7] |
+| `fn:doc` read the file whole and copied it; `replace` copied and over-charged a non-matching input | `fn:doc` parses as it reads (16 MB file 82 → 29 MB); `replace` returns early with no match (64 MB: 256 → 0 MB). | [`a6f2eaa`][a6f2eaa], [`6bab64a`][6bab64a] |
+| Every c14n call allocated an 80 KiB buffer, and `Bytes` grew its result by doubling | The buffer is pooled (101 `Digest` calls on a tiny document 8.4 MB → 121 KB); `Bytes` and package-made node sets write into a buffer sized from the tree (about 1.1× the output, from 2.1–3.6×). | [`127b2f2`][127b2f2], [`5d6454a`][5d6454a] |
+| A tag with many unique attribute names grew its slices, name table and duplicate map step by step | They are sized in one step past 256 attributes: 10 MiB of unique attributes 99.5× → 37.6× allocated. Small documents are unchanged. | [`22d32ac`][22d32ac] |
+| Each namespace node rebuilt its element's whole scope to find its place; accumulators kept two maps per node | The slot is passed in (c14n over an XPath node set at 1 MiB 356 → 299 MB); accumulator values sit in slices indexed by `Node.Index` (about 27% less per node). | [`73249e2`][73249e2], [`b331aa1`][b331aa1] |
+| A predicate step heap-allocated its comparison, and an ancestor step grew its result one node at a time | Both are cut: `ancestor-or-self::*[@Id='x']` per node 384 B / 9.75 allocations → 304 B / 7. | [`567aa1a`][567aa1a] |
+| `xsl:sort`, `xsl:iterate`, `xsl:result-document`, built-in shallow-copy, namespace scopes and `on-empty` allocated per item | Smaller sort values in one array, parameter slices, set lookups, no focus copy without rules, frame walks, a lazy leaf walk. `result-document` is linear. | [`6ea432f`][6ea432f], [`a3d8ef6`][a3d8ef6], [`951f0d9`][951f0d9], [`f650a19`][f650a19], [`4b0b564`][4b0b564], [`469c95e`][469c95e] |
+| DTD defaults were scanned per element; XSD key scopes rebuilt their entries per level | Defaults indexed by element (188 → 5 MB); scopes extend in place (960 levels 84 → 1 MB). | [`6293e98`][6293e98], [`996b7db`][996b7db] |
+| Large regex programs were cached; `parse-json` objects and `serialize` refusals cost per key and item | Programs over 4,096 instructions are not cached; objects use `MapBuilder` (about 3.3× less); `serialize` stops at the first refusal. | [`331bc96`][331bc96], [`98811f6`][98811f6], [`6317d49`][6317d49] |
+| DTD validation built a state-set map, a name slice and an attribute map per element | Small sets dedup by scan, the slice is reused, no ATTLIST skips the map: 12,062 → 8,062 allocations for 6,001 elements. | [`5dc120c`][5dc120c] |
+| `xsdnet.HTTPResolver` built a new transport per fetch | One guarded transport is shared, so connections are reused; the SSRF guard is unchanged. | [`da05b16`][da05b16] |
+| Typed copies pinned dropped schemas; `array:append` copied twice; line index, interning key and memo key kept slack | Weak typing-name slots, one copy, clipped index, reused key buffer, oversized memo buffer dropped. | [`967dfb3`][967dfb3], [`59ac569`][59ac569], [`3b71501`][3b71501], [`69414ae`][69414ae], [`887d5d3`][887d5d3] |
+
## v2.0.3 — 2026-10-30
An XPath cost fix from an external audit. No API change.
@@ -1625,3 +1711,94 @@ here so every entry in this file sits under a release.
[b76c85d]: https://github.com/knroy/go-xml/commit/b76c85d
[8b7aab2]: https://github.com/knroy/go-xml/commit/8b7aab2
[4264de8]: https://github.com/knroy/go-xml/commit/4264de8
+[599b2f9]: https://github.com/knroy/go-xml/commit/599b2f9
+[b42f04a]: https://github.com/knroy/go-xml/commit/b42f04a
+[20958aa]: https://github.com/knroy/go-xml/commit/20958aa
+[bd865e4]: https://github.com/knroy/go-xml/commit/bd865e4
+[8dcb07f]: https://github.com/knroy/go-xml/commit/8dcb07f
+[a5b385d]: https://github.com/knroy/go-xml/commit/a5b385d
+[18d374d]: https://github.com/knroy/go-xml/commit/18d374d
+[9e13f70]: https://github.com/knroy/go-xml/commit/9e13f70
+[ec1ac54]: https://github.com/knroy/go-xml/commit/ec1ac54
+[2ad3a32]: https://github.com/knroy/go-xml/commit/2ad3a32
+[a5ab736]: https://github.com/knroy/go-xml/commit/a5ab736
+[4440665]: https://github.com/knroy/go-xml/commit/4440665
+[6345114]: https://github.com/knroy/go-xml/commit/6345114
+[5d8fceb]: https://github.com/knroy/go-xml/commit/5d8fceb
+[e056467]: https://github.com/knroy/go-xml/commit/e056467
+[f09c1a3]: https://github.com/knroy/go-xml/commit/f09c1a3
+[14c84b5]: https://github.com/knroy/go-xml/commit/14c84b5
+[ba9e7a0]: https://github.com/knroy/go-xml/commit/ba9e7a0
+[f890740]: https://github.com/knroy/go-xml/commit/f890740
+[a8d6e7b]: https://github.com/knroy/go-xml/commit/a8d6e7b
+[24b963c]: https://github.com/knroy/go-xml/commit/24b963c
+[73af7a8]: https://github.com/knroy/go-xml/commit/73af7a8
+[f148247]: https://github.com/knroy/go-xml/commit/f148247
+[70ff980]: https://github.com/knroy/go-xml/commit/70ff980
+[ef5f064]: https://github.com/knroy/go-xml/commit/ef5f064
+[490fc5a]: https://github.com/knroy/go-xml/commit/490fc5a
+[dcb15d1]: https://github.com/knroy/go-xml/commit/dcb15d1
+[e3b6288]: https://github.com/knroy/go-xml/commit/e3b6288
+[b5aaa52]: https://github.com/knroy/go-xml/commit/b5aaa52
+[b6d4e4e]: https://github.com/knroy/go-xml/commit/b6d4e4e
+[327b09e]: https://github.com/knroy/go-xml/commit/327b09e
+[cf65d62]: https://github.com/knroy/go-xml/commit/cf65d62
+[f235795]: https://github.com/knroy/go-xml/commit/f235795
+[9496810]: https://github.com/knroy/go-xml/commit/9496810
+[37206b1]: https://github.com/knroy/go-xml/commit/37206b1
+[c501fc4]: https://github.com/knroy/go-xml/commit/c501fc4
+[1f18b1b]: https://github.com/knroy/go-xml/commit/1f18b1b
+[beb6072]: https://github.com/knroy/go-xml/commit/beb6072
+[59ac569]: https://github.com/knroy/go-xml/commit/59ac569
+[c1c2bb9]: https://github.com/knroy/go-xml/commit/c1c2bb9
+[4e8f0a5]: https://github.com/knroy/go-xml/commit/4e8f0a5
+[21995a6]: https://github.com/knroy/go-xml/commit/21995a6
+[702c6e6]: https://github.com/knroy/go-xml/commit/702c6e6
+[e139525]: https://github.com/knroy/go-xml/commit/e139525
+[9c3a983]: https://github.com/knroy/go-xml/commit/9c3a983
+[de4a515]: https://github.com/knroy/go-xml/commit/de4a515
+[9f0db9c]: https://github.com/knroy/go-xml/commit/9f0db9c
+[5dc120c]: https://github.com/knroy/go-xml/commit/5dc120c
+[da05b16]: https://github.com/knroy/go-xml/commit/da05b16
+[967dfb3]: https://github.com/knroy/go-xml/commit/967dfb3
+[3b71501]: https://github.com/knroy/go-xml/commit/3b71501
+[69414ae]: https://github.com/knroy/go-xml/commit/69414ae
+[887d5d3]: https://github.com/knroy/go-xml/commit/887d5d3
+[8577906]: https://github.com/knroy/go-xml/commit/8577906
+[30c7c07]: https://github.com/knroy/go-xml/commit/30c7c07
+[5951f24]: https://github.com/knroy/go-xml/commit/5951f24
+[014a862]: https://github.com/knroy/go-xml/commit/014a862
+[dcb49b2]: https://github.com/knroy/go-xml/commit/dcb49b2
+[c73a06b]: https://github.com/knroy/go-xml/commit/c73a06b
+[a3d8ef6]: https://github.com/knroy/go-xml/commit/a3d8ef6
+[40cd52c]: https://github.com/knroy/go-xml/commit/40cd52c
+[b35ec14]: https://github.com/knroy/go-xml/commit/b35ec14
+[f3449f7]: https://github.com/knroy/go-xml/commit/f3449f7
+[f49ff92]: https://github.com/knroy/go-xml/commit/f49ff92
+[44229aa]: https://github.com/knroy/go-xml/commit/44229aa
+[a9ad280]: https://github.com/knroy/go-xml/commit/a9ad280
+[be5afc3]: https://github.com/knroy/go-xml/commit/be5afc3
+[676ad52]: https://github.com/knroy/go-xml/commit/676ad52
+[40085f6]: https://github.com/knroy/go-xml/commit/40085f6
+[a4f316d]: https://github.com/knroy/go-xml/commit/a4f316d
+[b5aaf06]: https://github.com/knroy/go-xml/commit/b5aaf06
+[15273c7]: https://github.com/knroy/go-xml/commit/15273c7
+[a6f2eaa]: https://github.com/knroy/go-xml/commit/a6f2eaa
+[6bab64a]: https://github.com/knroy/go-xml/commit/6bab64a
+[6ea432f]: https://github.com/knroy/go-xml/commit/6ea432f
+[951f0d9]: https://github.com/knroy/go-xml/commit/951f0d9
+[f650a19]: https://github.com/knroy/go-xml/commit/f650a19
+[4b0b564]: https://github.com/knroy/go-xml/commit/4b0b564
+[469c95e]: https://github.com/knroy/go-xml/commit/469c95e
+[6293e98]: https://github.com/knroy/go-xml/commit/6293e98
+[996b7db]: https://github.com/knroy/go-xml/commit/996b7db
+[331bc96]: https://github.com/knroy/go-xml/commit/331bc96
+[98811f6]: https://github.com/knroy/go-xml/commit/98811f6
+[6317d49]: https://github.com/knroy/go-xml/commit/6317d49
+[b37796b]: https://github.com/knroy/go-xml/commit/b37796b
+[73249e2]: https://github.com/knroy/go-xml/commit/73249e2
+[127b2f2]: https://github.com/knroy/go-xml/commit/127b2f2
+[5d6454a]: https://github.com/knroy/go-xml/commit/5d6454a
+[22d32ac]: https://github.com/knroy/go-xml/commit/22d32ac
+[b331aa1]: https://github.com/knroy/go-xml/commit/b331aa1
+[567aa1a]: https://github.com/knroy/go-xml/commit/567aa1a
diff --git a/README.md b/README.md
index 7a2b6837..6e6d2da2 100644
--- a/README.md
+++ b/README.md
@@ -139,7 +139,7 @@ and maintains it as a project of his own.
| **XSD 1.1** | 99.98% instance (26,217 of 26,222); **99.97%** schema-validity (15,350 of 15,354); opt-in via `Version11` |
| **RELAX NG** | 100.00% of James Clark's spectest (965 of 965 assertions); XML and compact syntax, from Go and from `go-xml validate -rng` |
| **DTD** | content models, attribute defaults, enumerations, `ID`/`IDREF`; external subset, parameter entities across both subsets, conditional sections — via `dtd.Load` with a caller-supplied resolver, nothing fetched by default |
-| **Tests** | 2,854 `func Test` declarations, clean under `-race` (a few subtests skip without the corpora below) |
+| **Tests** | 2,950 `func Test` declarations, clean under `-race` (a few subtests skip without the corpora below) |
| **Production schemas** | UBL 2.1, UN/CEFACT CII, Factur-X/ZUGFeRD, Peppol BIS 3.0 — 88 schemas load, instances validate clean |
| **API** | 1.2; the exported surface is stable and additive over 1.1, and a breaking change means 2.0 with a new module path |
@@ -669,7 +669,7 @@ exact signal.
`FileResolver`'s document cache exists because `fn:doc` is defined to return
the *same* node for the same URI within one execution, so `doc('x') is doc('x')`
-requires it. It is mutex-guarded, and now capped at 256 documents. It had been
+requires it. It is mutex-guarded, and now capped at 256 documents and 32 MB of source. It had been
unbounded on the reasoning that the directories the caller opened with
`-allow-dir` bound it in practice — true for the size of the *set*, but the
process holds every document it has ever parsed for its whole lifetime, and a
@@ -1388,7 +1388,7 @@ back, is in [docs/testing.md](docs/testing.md).
| method | what it catches | what it misses |
|---|---|---|
-| **Unit tests** (2,854 `func Test` declarations) | places where a plausible implementation is quietly wrong | anything nobody thought to write a test for |
+| **Unit tests** (2,950 `func Test` declarations) | places where a plausible implementation is quietly wrong | anything nobody thought to write a test for |
| **Spec inventories** | features absent entirely | features present but behaving wrongly |
| **Saxon differential** | subtle behavioural divergence on real stylesheets | constructs the corpora do not use |
| **W3C QT3 suite** | systematic conformance across 22,054 XPath and 30,517 XQuery cases | XSLT (it is an XPath suite) |
diff --git a/c14n/bytes_alloc_test.go b/c14n/bytes_alloc_test.go
new file mode 100644
index 00000000..55167720
--- /dev/null
+++ b/c14n/bytes_alloc_test.go
@@ -0,0 +1,47 @@
+package c14n
+
+import (
+ "runtime"
+ "testing"
+)
+
+// Bytes and BytesNodeSet grew a bytes.Buffer by doubling from empty, 3.7
+// times the canonical form in all, then kept the writer's chunks and joined
+// them, 2.1 times, and Equal paid it for its first operand. They now size
+// the result from the node set and write into it once, about 1.1 times.
+func TestBytesAllocation(t *testing.T) {
+ // 3 MB: doubling cost 3.1x the output here and 3.7x at 10 MB.
+ root := cachedDoc(t, 3<<20).FirstChild()
+ out, err := Bytes(root, Options{Algorithm: Inclusive10})
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, tc := range []struct {
+ name string
+ f func()
+ }{
+ {"Bytes", func() { _, _ = Bytes(root, Options{Algorithm: Inclusive10}) }},
+ {"BytesNodeSet", func() { _, _ = BytesNodeSet(Subtree(root), Options{Algorithm: Inclusive10}) }},
+ {"Equal", func() { _, _ = Equal(root, root, Options{Algorithm: Inclusive10}) }},
+ } {
+ got := heapAllocated(tc.f)
+ t.Logf("%s: %d bytes allocated for %d bytes of output (%.2fx)", tc.name, got, len(out), float64(got)/float64(len(out)))
+ if got > uint64(len(out))*7/5 {
+ t.Errorf("%s allocated %d bytes for %d bytes of output, want under 1.4x", tc.name, got, len(out))
+ }
+ }
+}
+
+// heapAllocated is the heap f allocates, the least of three runs so that a
+// collection landing inside one does not count against it.
+func heapAllocated(f func()) uint64 {
+ best := ^uint64(0)
+ for range 3 {
+ var a, b runtime.MemStats
+ runtime.ReadMemStats(&a)
+ f()
+ runtime.ReadMemStats(&b)
+ best = min(best, b.TotalAlloc-a.TotalAlloc)
+ }
+ return best
+}
diff --git a/c14n/c14n.go b/c14n/c14n.go
index 8a7a3046..4eda2bd0 100644
--- a/c14n/c14n.go
+++ b/c14n/c14n.go
@@ -192,9 +192,95 @@ func FormatPrefixList(prefixes []string) string {
// descendants are, with the namespace and xml:* axis treatment the
// chosen algorithm specifies for a subset apex.
func Bytes(n *xdm.Node, opts Options) ([]byte, error) {
- var b bytes.Buffer
- err := Write(&b, n, opts)
- return b.Bytes(), err
+ if n == nil || (n.Kind() != xdm.KindDocument && n.Kind() != xdm.KindElement) {
+ return nil, ErrUnsupportedNode
+ }
+ return BytesNodeSet(Subtree(n), opts)
+}
+
+// chunkWriter keeps a copy of everything written to it.
+type chunkWriter struct{ chunks [][]byte }
+
+func (c *chunkWriter) Write(p []byte) (int, error) {
+ c.chunks = append(c.chunks, bytes.Clone(p))
+ return len(p), nil
+}
+
+// bytes joins the chunks. No output is nil, as from an empty bytes.Buffer.
+func (c *chunkWriter) bytes() []byte {
+ switch len(c.chunks) {
+ case 0:
+ return nil
+ case 1:
+ return c.chunks[0]
+ }
+ return bytes.Join(c.chunks, nil)
+}
+
+// sizedWriter collects everything written to it in one buffer, sized once
+// from the tree under the node set's root at the first write (after the options were validated, so
+// a refused call allocates nothing). The estimate runs 5% to 15% over the
+// output, so the result costs about 1.1 times its size where a buffer grown
+// by doubling cost 2.1 times and joined chunks twice; a short estimate (an
+// escaped value is longer than its text) costs one more copy.
+type sizedWriter struct {
+ ns NodeSet
+ opts Options
+ buf []byte
+}
+
+func (s *sizedWriter) Write(p []byte) (int, error) {
+ if s.buf == nil {
+ s.buf = make([]byte, 0, max(estimateSize(s.ns, s.opts), len(p)))
+ }
+ if need := len(s.buf) + len(p); need > cap(s.buf) {
+ // A short estimate: grow by a quarter, not by append's doubling.
+ s.buf = append(make([]byte, 0, need+need/4), s.buf...)
+ }
+ s.buf = append(s.buf, p...)
+ return len(p), nil
+}
+
+// estimateSize adds up the markup and values of ns's members. It reads the
+// tree once, a sixth of the time canonicalizing it takes. It counts every
+// declaration an element has and no escaping, and adds an eighth for the
+// escapes, so it is an estimate. It does not ask ns which nodes are members
+// (a NodeSet is asked once per node), so it is for the sets this package
+// made: a Subtree is counted exactly, an ExcludeSubtree with what it excludes.
+func estimateSize(ns NodeSet, opts Options) int {
+ root := ns.Root()
+ if root == nil {
+ return 0
+ }
+ comments := opts.Algorithm.WithComments()
+ n := 0
+ visit := func(d *xdm.Node) {
+ switch d.Kind() {
+ case xdm.KindElement:
+ q := d.Name()
+ n += 2*(len(q.Local)+len(q.Prefix)) + 6
+ for a := range d.Attrs() {
+ q := a.Name()
+ n += len(q.Local) + len(q.Prefix) + len(a.Value()) + 5
+ }
+ for p, u := range d.DeclaredNamespaces() {
+ n += len(p) + len(u) + 9
+ }
+ case xdm.KindComment:
+ if comments {
+ n += len(d.Value()) + 8
+ }
+ case xdm.KindPI:
+ n += len(d.Name().Local) + len(d.Value()) + 8
+ case xdm.KindText:
+ n += len(d.Value())
+ }
+ }
+ visit(root)
+ for d := range root.Descendants() {
+ visit(d)
+ }
+ return n + n/8
}
// Digest canonicalizes n and writes the result into h, returning h's sum.
@@ -277,9 +363,19 @@ func WriteNodeSet(w io.Writer, ns NodeSet, opts Options) error {
// BytesNodeSet mirrors Bytes for a subset.
func BytesNodeSet(ns NodeSet, opts Options) ([]byte, error) {
- var b bytes.Buffer
- err := WriteNodeSet(&b, ns, opts)
- return b.Bytes(), err
+ switch ns.(type) {
+ case subtree, excludeSet:
+ c := sizedWriter{ns: ns, opts: opts}
+ err := WriteNodeSet(&c, ns, opts)
+ return c.buf, err
+ }
+ // Any other set is the caller's, and Contains is called once per node:
+ // the output cannot be sized without calling it again, so it arrives in
+ // chunks of about outBufSize, kept and joined once, about twice the
+ // canonical form in all.
+ var c chunkWriter
+ err := WriteNodeSet(&c, ns, opts)
+ return c.bytes(), err
}
// DigestNodeSet mirrors Digest for a subset.
diff --git a/c14n/canon.go b/c14n/canon.go
index fe6d4425..e93fc5ca 100644
--- a/c14n/canon.go
+++ b/c14n/canon.go
@@ -6,6 +6,7 @@ import (
"regexp"
"slices"
"strings"
+ "sync"
"github.com/knroy/go-xml/v2/xdm"
)
@@ -96,7 +97,10 @@ type canon struct {
const noNode = "\x00"
func canonicalize(w io.Writer, ns NodeSet, opts Options) error {
- _, err := run(w, ns, opts)
+ c, err := run(w, ns, opts)
+ if c != nil {
+ c.w.release()
+ }
return err
}
@@ -108,7 +112,6 @@ func run(w io.Writer, ns NodeSet, opts Options) (*canon, error) {
}
_, all := ns.(subtree)
c := &canon{
- w: newOutBuf(w),
set: ns,
all: all,
excl: opts.Algorithm.Exclusive(),
@@ -139,6 +142,7 @@ func run(w io.Writer, ns NodeSet, opts Options) (*canon, error) {
c.scope.m[prefix] = uri
}
}
+ c.w = newOutBuf(w)
var err error
switch root.Kind() {
case xdm.KindDocument:
@@ -150,6 +154,7 @@ func run(w io.Writer, ns NodeSet, opts Options) (*canon, error) {
c.leaf(root)
}
default:
+ c.w.release()
return nil, ErrUnsupportedNode
}
if err == nil {
@@ -170,12 +175,32 @@ func run(w io.Writer, ns NodeSet, opts Options) (*canon, error) {
// into nothing.
type outBuf struct {
buf []byte
+ pb *[]byte // the pool entry buf came from
w io.Writer
err error
}
+// outBufPool keeps the buffers between calls: a fixed 80 KiB allocated per
+// call was 81 KB of garbage for each Digest of a tiny document.
+var outBufPool = sync.Pool{New: func() any {
+ b := make([]byte, 0, outBufSize+outBufSize/4)
+ return &b
+}}
+
func newOutBuf(w io.Writer) *outBuf {
- return &outBuf{buf: make([]byte, 0, outBufSize+outBufSize/4), w: w}
+ pb := outBufPool.Get().(*[]byte)
+ return &outBuf{buf: (*pb)[:0], pb: pb, w: w}
+}
+
+// release returns the buffer to the pool unless output grew it far past its
+// size, so that one large text node does not stay allocated. o is not used
+// afterwards.
+func (o *outBuf) release() {
+ if cap(o.buf) <= 2*outBufSize {
+ *o.pb = o.buf[:0]
+ outBufPool.Put(o.pb)
+ }
+ o.buf, o.pb = nil, nil
}
// flush hands buf to w. A short write without an error is io.ErrShortWrite,
diff --git a/c14n/digest_alloc_test.go b/c14n/digest_alloc_test.go
new file mode 100644
index 00000000..f2541039
--- /dev/null
+++ b/c14n/digest_alloc_test.go
@@ -0,0 +1,27 @@
+package c14n
+
+import (
+ "crypto/sha256"
+ "testing"
+)
+
+// Every call allocated an 80 KiB output buffer, 81 KB of garbage for each
+// Digest of a tiny document. The buffers now come from a pool. The bound is half the unpooled
+// figure because the race detector makes a pool drop a quarter of its puts.
+func TestDigestSmallDocumentAllocation(t *testing.T) {
+ root := parse(t, `x`).FirstChild()
+ h := sha256.New()
+ const calls = 101
+ got := heapAllocated(func() {
+ for range calls {
+ h.Reset()
+ if _, err := Digest(h, root, Options{Algorithm: Inclusive10}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ })
+ t.Logf("%d Digest calls allocated %d bytes", calls, got)
+ if got > 4<<20 {
+ t.Errorf("%d Digest calls allocated %d bytes, want under 4 MiB (8.3 MB with a fresh buffer each)", calls, got)
+ }
+}
diff --git a/cmd/go-xml/gc_test.go b/cmd/go-xml/gc_test.go
index 99744c68..4d62594b 100644
--- a/cmd/go-xml/gc_test.go
+++ b/cmd/go-xml/gc_test.go
@@ -23,3 +23,23 @@ func TestLowerGCRespectsGOGC(t *testing.T) {
t.Errorf("with GOGC=100: GC percent %d, want the environment's 100", got)
}
}
+
+// Past 128 MB of live heap the command drops from GOGC=200 to 100; below it
+// the setting is left alone.
+func TestRelaxGCOnBigLiveHeap(t *testing.T) {
+ orig := debug.SetGCPercent(cliGCPercent)
+ defer debug.SetGCPercent(orig)
+
+ if relaxGC(cliGCBigHeap) {
+ t.Error("relaxed at exactly the threshold, want only past it")
+ }
+ if got := debug.SetGCPercent(cliGCPercent); got != cliGCPercent {
+ t.Errorf("a small heap changed GC percent to %d", got)
+ }
+ if !relaxGC(cliGCBigHeap + 1) {
+ t.Error("did not relax past the threshold")
+ }
+ if got := debug.SetGCPercent(cliGCPercent); got != 100 {
+ t.Errorf("GC percent after a big heap = %d, want 100", got)
+ }
+}
diff --git a/cmd/go-xml/main.go b/cmd/go-xml/main.go
index 379052d6..b9022508 100644
--- a/cmd/go-xml/main.go
+++ b/cmd/go-xml/main.go
@@ -11,8 +11,11 @@ import (
"os"
"path"
"path/filepath"
+ "runtime"
"runtime/debug"
+ "runtime/metrics"
"strings"
+ "sync"
"time"
"github.com/knroy/go-xml/v2/xdm"
@@ -51,12 +54,51 @@ func main() {
// never in the library, whose callers own their process's GC policy.
const cliGCPercent = 200
+// cliGCBigHeap is the live heap, in bytes, past which the command drops to
+// GOGC=100. At 200 the heap may reach three times what is live, and past this
+// size that is what costs: peak RSS on DocBook table-cals.049 was 1,160 MB at
+// 200 and 789 MB at 100, and on a 100 MB identity transform 1,840-1,956 MB
+// against 1,360-1,400, for the same wall time and 5-10% more CPU. Below it a
+// small heap is cheap to hold and the extra CPU is what 200 saves. The size of
+// the input is no guide -- DocBook's is small -- so it is the live heap that
+// is read.
+const cliGCBigHeap = 128 << 20
+
// lowerGC applies cliGCPercent unless GOGC is set, so a user who tunes the
-// collector keeps their setting.
+// collector keeps their setting, and watches the live heap to relax it.
func lowerGC() {
if os.Getenv("GOGC") == "" {
debug.SetGCPercent(cliGCPercent)
+ watchLiveHeap.Do(armGCSentinel)
+ }
+}
+
+var watchLiveHeap sync.Once
+
+type gcSentinel struct{ _ [16]byte }
+
+// armGCSentinel runs relaxGC after each collection until it has acted. A
+// finalizer on an unreachable object runs once the cycle that found it has
+// marked, so the live-heap metric it reads is that cycle's.
+func armGCSentinel() {
+ runtime.SetFinalizer(new(gcSentinel), func(*gcSentinel) {
+ s := []metrics.Sample{{Name: "/gc/heap/live:bytes"}}
+ metrics.Read(s)
+ if s[0].Value.Kind() == metrics.KindUint64 && relaxGC(s[0].Value.Uint64()) {
+ return
+ }
+ armGCSentinel()
+ })
+}
+
+// relaxGC drops to GOGC=100 when live bytes are past cliGCBigHeap, and
+// reports whether it did.
+func relaxGC(live uint64) bool {
+ if live <= cliGCBigHeap {
+ return false
}
+ debug.SetGCPercent(100)
+ return true
}
// paramFlag collects repeated -p name=value flags.
diff --git a/docs/benchmark.md b/docs/benchmark.md
index 930d0690..83605b60 100644
--- a/docs/benchmark.md
+++ b/docs/benchmark.md
@@ -472,7 +472,8 @@ Peak RSS in cold runs (median / maximum over items):
For transforms v1 uses a fifth to two thirds of a JVM's memory, and v2 an
eighth to a quarter. The CLI runs its collector at `GOGC=200` unless `GOGC`
-is set, which trades some peak memory for CPU. For large documents v1's tree
+is set, which trades some peak memory for CPU, and drops to `GOGC=100` once
+the live heap passes 128 MB. For large documents v1's tree
cost about 23 bytes of RSS per byte of input, 1.6× libxml2's; v2's costs
under 4, about a quarter of libxml2's. Only `encoding/xml`, which builds no
tree, uses less.
diff --git a/docs/c14n.md b/docs/c14n.md
index 0a27e58e..89ed7bc4 100644
--- a/docs/c14n.md
+++ b/docs/c14n.md
@@ -409,7 +409,7 @@ against `xdm`'s public API as it stands:
| R1 | The namespace declarations made *on* each element, as written | `Node.NamespaceDecls()` — the declarations on that element only, an undeclaration (`xmlns=""`) carried as an empty value | yes |
| R2 | The bindings *in scope* at an element | `Node.InScopeNamespaces()` (a prefix-to-URI map, `xml` always present) and `Node.LookupPrefix(prefix)` for one prefix | yes |
| R3 | `xml:*` attributes, for inheritance and the 1.1 `xml:base` fix-up | ordinary `Node.Attrs()` whose `Name().URI == xdm.NSXML` | yes |
-| R4 | Document order | `Node.Compare` orders any two nodes. `Node.Order()` is a per-tree index assigned by `Tree.Finalize` in one pre-order walk, which every parser entry point calls; it is strictly increasing in document order within a finalized tree but **not dense** — `Finalize` reserves a slot for every in-scope binding of each element, for the namespace-axis nodes synthesized on demand, and adds a per-tree offset. `c14n` needs neither: it walks `Children` and `Attrs` in order and sorts what the specifications tell it to sort | yes |
+| R4 | Document order | `Node.Compare` orders any two nodes. `Node.Order()` is a per-tree index assigned by `Tree.Finalize` in one pre-order walk, which every parser entry point calls; it is strictly increasing in document order within a finalized tree but **not dense** — `Finalize` reserves a slot for every in-scope binding of each element, for the namespace-axis nodes synthesized on demand, and adds a per-tree offset. `Node.Index()` is the dense position, with `Tree.NodeAt` to go back. `c14n` needs neither: it walks `Children` and `Attrs` in order and sorts what the specifications tell it to sort | yes |
| R5 | Node identity, for node-set membership | pointer equality for every node the parser builds; `Node.Is` for the general case (it differs only for synthesized namespace nodes, which never reach `c14n`) | yes |
| R6 | Character data in canonical shape | the parser resolves character and entity references, merges CDATA sections with adjacent text into one text node, normalises CRLF and a lone CR to LF, keeps a `
` reference as a literal CR (which C14N then writes as `
`), and normalises attribute whitespace — including, now, the collapse for attributes declared non-CDATA; verified by the `c14n` tests | yes |
diff --git a/docs/conformance-gaps.md b/docs/conformance-gaps.md
index b693e744..281400a0 100644
--- a/docs/conformance-gaps.md
+++ b/docs/conformance-gaps.md
@@ -49,7 +49,7 @@ W3C disagreements: 0 + 0 + 0 + 1 + 8 + 23 + 30 + 31 + 0 = 93. Measured 2026-09-1
-The unit-test suite is 2,854 tests.
+The unit-test suite is 2,950 tests.
The last two rows are not W3C suites but real-world corpora — DocBook xslTNG's
diff --git a/docs/known-gaps.md b/docs/known-gaps.md
index 680b6a0a..cd8f4c54 100644
--- a/docs/known-gaps.md
+++ b/docs/known-gaps.md
@@ -105,6 +105,57 @@ tables being copied per scope and target nodes being walked again. Found by
an external audit of v2.0.0; the figures are the `BenchmarkIdentity*`
benchmarks in `xsd/`.
+### Memory: what the third memory audit left open (resource limits, performance)
+
+Found by the third memory audit (2026-11-01) and not fixed, each with the reason:
+
+- **No bound on the total across many `fn:doc` calls.** The per-document
+ `MaxBytes` holds, but 40 distinct 500k-element documents were held at 846 MB
+ live. A correct total needs a per-evaluation set of loaded URIs (the
+ `FileResolver` cache is shared, so charging on a miss undercounts), state
+ carried into nested `fn:transform`, and every reader covered. Lower
+ `FileResolver.MaxBytes` or the allowed roots meanwhile.
+- **`Env.MaxDigits` covers casts, constructors and binary arithmetic, not
+ `CastAtomic`, `fn:sum` or `fn:avg`**, which take no context.
+- **RELAX NG's derivative cap still refuses some valid documents.** The choice
+ merge fixed `(a?){n} a{n}` up to about n = 30. At n = 50 and 100 the cap is
+ reached again, though in 20–300 ms rather than 19.9 s. Hash-consing
+ derivatives would close it.
+- **`xsl:number level="any"` is quadratic in CPU** (40k items 44 s). A cached
+ prefix count is not valid, since `count` and `from` patterns may use
+ `current()` or per-call variables.
+- **`fold-left` appending one item a step is quadratic in allocation**
+ (40k items 12.4 GB, all garbage). Appending in place is unsound while other
+ references may share the prefix.
+- **Identity templates allocate about 3.5 KB a node of context copies**
+ (`withSel`, `hostWithCurrent`, the leaf-hook copy), and built-in shallow-copy
+ copies a document-level result twice; removing that copy changes an error
+ case into output.
+- **A single text value of a few to tens of MB allocates up to about 1.9× its
+ size while parsing** (10 MiB: 1.87×; the tree keeps 1.09×; 60 MiB: 1.13×).
+ The decoder's buffer grows in steps of eight. Sizing it to the input left
+ in one step reached 1.08×, but a document of eight 3 MiB values went from
+ 1.33× to 2.04×, so it was reverted.
+- **`c14n.BytesNodeSet` on a caller's own `NodeSet` still costs about 2.1× its
+ output.** Sizing it would call `Contains` twice per node, which the API
+ promises not to do; the package's `Subtree` and `ExcludeSubtree` are sized.
+- **XQuery inline functions capture their whole scope.** The XPath and XSLT
+ closure (`InlineFunctionExpr`) keeps only what its body reads;
+ `xquery/inline.go` has its own implementation.
+- **A regex program over 4,096 instructions is compiled on every call**,
+ because it is not cached; the largest QT3 pattern is 4,059.
+
+### `fn:replace` charges a backtracking or backreference result after building it (XPath, resource limits)
+
+The RE2 path of `fn:replace` charges `len(input) + matches × len(replacement)`
+before it builds the result, so a hostile replacement is refused without being
+allocated. The two other engines, the backreference matcher and the opt-in
+backtracking one (`-backtracking-regex`), still build the result and charge it
+afterwards. Their step budgets bound the number of matches, but not the size
+of each replacement, so a long replacement string can allocate past
+`MaxBytes` before the refusal. Found by the second memory audit
+(2026-10-31), which measured only the RE2 path.
+
### §19.8 streamability analysis is partially implemented (XSLT 3.0)
**3 of the 23 XSLT 3.0 failures** — `si-fork-902`, `si-fork-952` and
diff --git a/docs/migrating-to-v2.md b/docs/migrating-to-v2.md
index 29c763c9..a741babb 100644
--- a/docs/migrating-to-v2.md
+++ b/docs/migrating-to-v2.md
@@ -177,6 +177,7 @@ doc.Finalize()
The calls are `AppendElement`, `AppendText`, `AppendComment`, `AppendPI`,
`AppendAttr` (before the element's first child), `AddNamespace`,
+`AddNamespaces` (several bindings in one call),
`AppendCopy` (a deep copy) and `AppendShallowCopy` (the node alone, for a copy
you complete yourself). A node is open from the call that made it until you
append to something that is not it or inside it; appending to a closed node,
@@ -185,7 +186,7 @@ or adding an attribute after a child, panics. `Finalize` closes everything.
What may still change after a node is made is its scalar state: `SetName`,
`SetValue`, `AppendValue` (merging adjacent text), `SetBaseURI`,
`SetDocumentURI`, the typing setters, and the namespace declarations of an
-element (`AddNamespace`, `SetNamespaceDecl`, `RemoveNamespaceDecls`) of a tree
+element (`AddNamespace`, `AddNamespaces`, `SetNamespaceDecl`, `RemoveNamespaceDecls`) of a tree
that is not a parsed document. A parsed document is frozen.
`xdm.NewNode(kind, name, value)` still makes a parentless node; it belongs to
@@ -230,7 +231,8 @@ Namespace nodes exist once per element and prefix: the namespace axis, and
agree for them.
`xdm.ProcessXInclude` returns the included document as a new tree instead of
-editing the one it is given: `tree, err = xdm.ProcessXInclude(tree, opts)`.
+editing the one it is given (a parsed tree with nothing to include comes back
+as it is): `tree, err = xdm.ProcessXInclude(tree, opts)`.
## xdmbuild setters removed
diff --git a/docs/options.md b/docs/options.md
index 954f03c6..7701fffa 100644
--- a/docs/options.md
+++ b/docs/options.md
@@ -180,7 +180,7 @@ tree, err := xdm.ParseString(src, xdm.ParseOptions{
| `BaseURI` | `string` | none | Recorded on the document node; what `fn:base-uri`, `fn:doc` and relative `xsl:include` resolve against. Set it whenever the document came from somewhere addressable. |
| `StripSpace` | `func(QName) bool` | keep all | Drops whitespace-only text nodes for the element names it returns true for. A predicate rather than a bool because `xsl:strip-space` is per-element. |
| `AllowDOCTYPE` | `bool` | refuse | Permits a `DOCTYPE` declaration. See below. |
-| `TrackPositions` | `bool` | off | Records where each element starts, so a validation error can name a line. Retains the source text for the life of the tree — about 10% more memory, no extra parse time. |
+| `TrackPositions` | `bool` | off | Records where each element starts, so a validation error can name a line. Keeps each line's start offset and each element's column, not the source text: about 10% more retained memory (parse-10mb 37.2 → 41.5 MB), no extra parse time. |
| `MaxDepth` | `int` | `DefaultMaxDepth` = 1000 | Nesting limit. Deep input is the cheapest route to stack exhaustion. |
| `MaxBytes` | `int64` | `DefaultMaxBytes` = 64 MB | Source size limit, enforced on the read. |
| `MaxNodes` | `int` | `DefaultMaxNodes` = 10,000,000 | Node count limit, counting attributes and namespaces. |
@@ -688,6 +688,7 @@ functions or `WithEnv`):
| `RegexVersion`, `LibraryVersion` | `Version` | Raise the regular-expression dialect and the function library above the expression's version, without admitting new syntax. |
| `MaxDepth` | `int` | Recursion bound. Zero is `xpath.MaxDepth` (500). |
| `MaxItems` | `int` | Items an evaluation may materialise. Zero is `xpath.MaxItems` (5,000,000); negative is no bound; a nested evaluation adopts the caller's bound and cannot raise it. |
+| `MaxDigits` | `int` | Digits of an `xs:integer` or `xs:decimal` a cast or arithmetic may produce. Zero is `xpath.MaxDigits` (10^6), `FOCA0003` or `FOAR0002` past it; negative is no bound. |
| `QualifyVar`, `MissingVar` | funcs | Host hooks for variable references (XSLT packages, XQuery cycles). |
| `MapDuplicateCode` | `string` | Error code for a duplicate key in a map constructor; `XQDY0137` by default. |
@@ -703,7 +704,11 @@ evaluation's. A nested evaluation therefore never resets or raises a budget.
`WithStaticBaseURI` and `WithStaticHost` set the defaults an expression compiled
without them uses (a one-shot `xpath.Eval` compiles at the context's version).
A host whose expressions all carry one version or static host sets it once on
-the context, so evaluating them does not copy the context to install it.
+the context, so evaluating them does not copy the context to install it. This
+matters for a host that reuses one context: an expression compiled as XPath 3.1
+evaluated against `NewContext`'s 2.0 default copies the 96-byte context on every
+call, about a quarter of a simple filter's allocation per node, until the host
+calls `ctx = ctx.WithVersion(xpath.XPath31)` once.
`StaticNamespaces()` is installed only for an expression that calls or
references `fn:format-date`, `fn:format-dateTime`, `fn:format-time` or
`fn:function-lookup`; any other expression sees its caller's (nil at the top
@@ -871,7 +876,8 @@ tree, err = xdm.ProcessXInclude(tree, xdm.XIncludeOptions{
`ProcessXInclude` returns the included document as a new tree, built
top-down as every tree is; the tree it is given is left as it was, and node
-identities do not carry over.
+identities do not carry over. A parsed tree with no XInclude element has
+nothing to replace and is returned as it is.
**The resolver is the whole of the confinement.** `xdm` has no filesystem and
no network; it can read only what a resolver hands it. `xslt.FileResolver`
diff --git a/docs/profiling.md b/docs/profiling.md
index 294d61d3..181d57e5 100644
--- a/docs/profiling.md
+++ b/docs/profiling.md
@@ -131,6 +131,9 @@ The profile is flat. The largest single functions are `QName` equality
| `Tree.alloc` (result-tree records) | — | 6.2% |
| `evalParams` (cumulative) | — | 6.4% |
+This profile predates the memory audit: `evalParams` now passes a slice of
+pairs, not two maps, and section.001 allocates 15% less (CHANGELOG, *Unreleased*).
+
No source exceeds a tenth of the objects. DocBook's cost is many small
allocations spread over the whole evaluator, and the collector work they
cause.
@@ -201,7 +204,8 @@ invoice are 20 k and 1.95 MB.
Allocations: 42% of the bytes are node-record chunks (`Tree.alloc`), 22% the
harness's input copy and 16% the text store. By count, FLWOR tuple binding
-(`tuple.bind`) is 27%, but these are small objects and about 2% of CPU. XMark
+(`tuple.bind`) was 27% before the memory audit, which shares one names
+slice per tuple stream, but these are small objects and about 2% of CPU. XMark
at factor 0.1 is a parse benchmark.
### XSD validation and typed validation
@@ -248,9 +252,10 @@ objects. By bytes, `Tree.alloc` makes 21%, the clone 17% and the typing
48% of the bytes are the tokenizer's 32 KB string arena, one per document.
`newAfterPat` makes 37% of the objects and `strings.Fields` (token
normalisation) 12%. The documents are small, so that fixed arena dominates
-the bytes and drives the GC share. Sizing the arena to the input was measured
-and rejected because of how it moved GC pacing on DocBook (see
-[rejected](#measured-and-rejected)).
+the bytes and drives the GC share. Sizing the arena to the input was first
+rejected because of how it moved GC pacing on DocBook (see
+[rejected](#measured-and-rejected)); the memory audit adopted it once
+namespace URIs stopped pinning the arena (CHANGELOG, *Unreleased*).
### Parse and Canonical XML
@@ -379,7 +384,10 @@ any dropped passing count.
the same `GOGC`. A change that cuts retained memory can cost CPU elsewhere:
starting the string arena small cut small documents' retained heap 20–37%
and cost DocBook 5% CPU (85 GC cycles against 77). Count GC cycles a pass
- in both arms.
+ in both arms. The memory audit raised DocBook xslTNG (593 inputs, one
+ process) from about 142 to 174 cycles, the same with or without the sized
+ arena, with peak RSS 1.24–1.35 GB → 1.07–1.15 GB and user CPU within noise
+ over five alternating rounds.
- **The memory profiler costs CPU.** A binary that links `runtime/pprof`
samples allocations unless `MemProfileRate` is 0. The CLI does not link it,
so a harness that does measures a slightly different program.
@@ -477,7 +485,7 @@ longer holds.
| Smaller first record chunk (2 records, V15) | Most fragments hold one node, but bytes moved −0.4% to +0.2% and allocations up to +0.6%: the next chunk comes sooner for every other tree |
| Chunked result nodes | −0.9% allocations at most, +0.4–0.7% bytes |
| String arena starting small | Retained heap −20–37% on small documents, but DocBook +5% CPU: the smaller live heap ran the collector more often (85 cycles against 77). Reverted in `629cf48` |
-| String arena sized to the input (V18) | 40 small RELAX NG documents: parse bytes −52%, CPU −20%, retained heap −72%. But DocBook ran about 7% more GC cycles (29.3 → 31.3 a pass), as above. Only the read windows were sized (`9ddd2bc6`) |
+| String arena sized to the input (V18) | 40 small RELAX NG documents: parse bytes −52%, CPU −20%, retained heap −72%. But DocBook ran about 7% more GC cycles (29.3 → 31.3 a pass), as above. Only the read windows were sized (`9ddd2bc6`). **Adopted later** in `9e13f70`: with namespace URIs copied out (`18d374d`) the arena no longer pinned trees, and an A/B with and without it gave DocBook the same GC count |
| Attribute values passed to xdm without the arena copy (V40) | Parse bytes −2.3 to −2.6%, but `regex-syntax` +2.6% bytes, XRechnung +26 to +53 allocations a pass and more GC cycles (7.2 → 7.6); no CPU gain. Reviving it needs the copy at `textStore.add`, since the values sit in a reused buffer |
| End tags checked against a stack of open names (V35) | After the tokenizer's name cache (V34), parse +0.8 to +1.7% CPU and 7 more allocations a parse |
| SWAR scan in the tokenizer's `text()` | No gain (round 3, and again on v2): it is already a tight table loop |
diff --git a/docs/security.md b/docs/security.md
index 34dbbcf6..c71a9bbf 100644
--- a/docs/security.md
+++ b/docs/security.md
@@ -610,6 +610,8 @@ what actually drives them:
the resolver. `resolverCacheMax = 256` **cleared on full** rather than
evicting, so past 257 distinct URIs it stopped being a bound and became an
amplifier; it now evicts one entry at a time, which is also fixed below.
+ Since the memory audit it is also bounded by `resolverCacheMaxBytes`, 32 MB
+ of source (about 5x that in trees); a `Preload`ed tree is charged 0.
Two further sites — `fn:transform`'s `nestedParseOptions`
(`xslt/fntransform.go:531`) and `relaxng/resolve.go:141` — have the same shape
@@ -1252,7 +1254,12 @@ itself is rejected without it. See
Go's `regexp` is RE2. `matches('aaaa…!', '^(a+)+$')` is flat at 2–6 µs from
n=24 to n=40. Go also *rejects* repeat counts over 1000, so `{1,1000000}` cannot
be used to force an expansion; the XSD pattern translator allocates nothing on
-nested quantifiers or 200-deep groups.
+nested quantifiers or 200-deep groups. That check bounds one repeat and nested
+products, **not sequential repeats**: `\pL{1000}` written 150 times compiled to
+a program of millions of instructions, and 181 KB of such patterns retained
+984 MB. Every pattern now goes through `xpath.CompileBounded`, which counts the
+program before building it and refuses past `MaxRegexpProgram` (100,000; the
+largest in the W3C suites is 4,313), in XPath and in XSD pattern facets.
**Backreferences do not change this, by default.** XPath 2.0 has them and RE2
does not, and the usual way to bridge that is a backtracking engine — which is
@@ -1721,7 +1728,8 @@ and facets still apply to the substituted type.
static context an environment of its own; the four reads are marked in the
file.
- The `xpath` regex cache is bounded at 1024, as is the backtracking engine's
- single-character-atom cache; the UCA collation cache is bounded at 256. All
+ single-character-atom cache; the UCA collation cache is bounded at 256. The
+ bound is on entries, so a key over 1 KB (`maxCachedKey`) is never cached. All
three hold their bound under concurrent use, not merely on a single goroutine:
each is a `boundedCache` (`xpath/cache.go`) that performs the full-check and
the insert under one lock hold. An earlier form checked an atomic size counter
@@ -1857,6 +1865,24 @@ Each line names the **direction** of the defect, because that is what decides
who was exposed: a *false accept* let an invalid input through, a *false
reject* refused a legal one, and *cost* produced the right answer too slowly.
+**Nineteenth pass — a third memory audit, 2026-11-01.** The findings and figures are in the audit's own write-up; the retention and allocation fixes are in CHANGELOG under *Changed — memory*.
+
+- **Sequences built by `,` and `!` were not counted against `MaxItems`** — availability, driven by a query or stylesheet. `fold-left(1 to 26,(1,2),function($s,$i){($s,$s)})` (70 bytes) built 134M items against the 5M limit at 6.3 GB, and the same doubling through a recursive `xsl:with-param` reached 4.6 GB on v2.0.3. `SequenceExpr` and `!` charge the copies beyond their widest operand above 64 items, and `xsl:next-iteration` values are checked against the bound. Pinned by `TestSequenceCopiesAreCharged`.
+- **RELAX NG rejected a valid document** — correctness. `(a?){13} a{13}` against 13 `` crossed the 100,000-node derivative cap, and the message named a `oneOrMore` the schema does not have; `xmllint` accepts it. Choice chains now merge alternative by alternative, under a 1,024-pair budget per merge so a hostile schema stays bounded. Pinned by `TestOptionalsBeforeRequiredStayValid`. Larger N still reaches the cap (docs/known-gaps.md).
+- **Serializing atomic values was quadratic** — cost, driven by result size. `xslt/seqnorm.go` joined adjacent atomics by repeated concatenation, so `1 to 100000` took 21.6 s and 6.95 GB for 588 KB of output. Each run is joined once. Pinned by `TestJoinAdjacentAtomicsIsLinear`.
+- **Several producers were uncharged or unbounded** — cost, driven by document or query data. `unparsed-text-lines`, `parse-json` and `json-doc` built tens of millions of items past `MaxItems` (5.2 GB from a 67 MB file); DTD default values were copied into every element unseen by `MaxNodes` or `MaxBytes` (2 MB retained 1.97 GB); an `xs:integer` had no size bound (a 4M-digit cast allocated 34 GB); a sequence-valued accumulator copied its value into every node (200 KB held 12.4 GB). Lines and JSON values are charged, defaults are stored once, `Env.MaxDigits` (10^6) refuses with FOCA0003 or FOAR0002, and accumulator values share their prefix and are charged.
+- **`xsl:copy-of` of a document bypassed `MaxNodes`** — the inconsistency, not data loss: it copied 5.7M nodes and exited 0 where shallow-copy is refused. It is charged before the copy.
+
+**Eighteenth pass — a memory audit, 2026-10-31.** The rest of that audit was retention and allocation with no hostile driver; it is in CHANGELOG under *Changed — memory*.
+
+- **`xsd` error messages kept document strings whole** — cost, driven by document data. `fail` and `attrName` quoted names and values in full and the validator keeps up to 100 errors, so 200 foreign attributes in a 1 MB namespace retained 105 MB, and one 15 MB value under a failing rule set retained 1.57 GB (audit 20). Strings and the formatted message are clipped to 1 KB plus "...", and the pattern and list-item errors quote through `truncate`: 0.13 MB. Pinned by `TestValidationMessagesClipDocumentStrings` in `xsd/message_clip_test.go`.
+- **The DTD validator built every element's path, quadratic in depth** — cost, driven by document data. The path string was rebuilt at each element whether or not anything failed: a *valid* 990-deep chain of 4 KB names allocated 2 GB. It is rendered from the ancestors only when a failure is recorded, and an IDREF keeps its element rather than a path. Its `#FIXED`, enum and ID values are clipped to 1 KB too. Pinned by `TestPathBuiltOnlyForFailures` and `TestValidityMessagesClipLongValues` in `dtd/`.
+- **Deeply nested maps and arrays overflowed the Go stack** — availability, driven by a query or stylesheet. `serialize(..., map{"method":"json"})`, the adaptive method, `fn:deep-equal` and `map:find` recursed once per level, so `fold-left(1 to 400000, [], function($a,$i){[$a]})` (96 bytes) ended the process with `fatal error: stack overflow`, which no `recover` can catch; v2.0.3 has it too. Serialization refuses past 1,000 levels (the `parse-json` limit) with XPDY0130, `deep-equal` and `map:find` past 1,000 levels on a counter of their own (XPDY0001), so function recursion does not lower it, and `xdm.Flatten` is iterative. Pinned by `TestDeepNestedValuesAreRefused` in `xpath/deepnest_test.go`.
+- **A built-in rule returned a truncated result tree as a success** — correctness, driven by document size. The result builder latches a `MaxNodes` refusal, but only instruction bodies checked it, so `` over 1M elements emptied a third of them and exited 0; v2.0.3 has it too. The transform now fails with XPDY0130 whenever the node budget was crossed anywhere, through `xpath.Context.NodesRefused`. Pinned by `TestBuiltInRuleResultTreeOverrunIsRefused`.
+- **`fn:replace`, picture widths, `xsl:key` and `group-by` grew past their budgets** — cost, driven by document or stylesheet data. `replace` charged its result after building it (12.8 GB for a 500 KB input and a 4 KB replacement); a `format-date` width of `[D01,1000000000]` (52 bytes) allocated 2.8 GB through `strings.Repeat`; `use="1 to 1000"` over 100k nodes built 100M key entries at 3.2 GB peak, since the item counter resets per expression. `replace` charges first on the RE2 path (the other two engines are a known gap), widths over 65,536 are FOFD1340, and keys are totalled per index and grouping against `MaxItems`.
+- **Caches bounded by entry count held gigabytes** — cost, driven by document data or the host. 1,024 cached regexes of 200 KB, 256 failed 1 MB collation URIs, or 256 large `FileResolver` documents each held hundreds of MB to GB. Keys over 1 KB are not cached, and the document cache is also bounded by 32 MB of source.
+- **The nested `fn:transform` cache kept compiles for collected resolvers** — cost, driven by the host. The key held each resolver by identity, so a service that made one per request kept up to 16 dead compiles a stylesheet: 49 MB on DocBook after 20 resolvers. Resolvers are weak keys and `put` drops entries whose resolver is gone: 0.3 MB. Pinned by `TestTransformCache/dead_resolver_dropped`.
+
**Seventeenth pass — normalize-space built a list of every token, 2026-10-30.**
- **`fn:normalize-space` allocated per whitespace token** — cost, driven by document data. It was `strings.Join(splitXMLSpace(s), " ")`, so a value of n tokens built an n-element slice and then the result; a rule set evaluating it 180 times on one 15 MB value allocated 133.5 GB in 20.6 s, and `MaxBytes` charges only the result. It and `fn:tokenize#1` now use `collapseXMLSpace`, one pass into a builder, which returns an already-collapsed value as it is: 0 GB in 2.6 s, and 2.5 GB in 4.2 s for a value that needs collapsing. Pinned by `TestCollapseXMLSpaceMatchesSplitJoin` and `TestCollapseXMLSpaceAllocation` in `xpath/xml_space_test.go`. Fixed in v2.0.3.
diff --git a/docs/stats.md b/docs/stats.md
index b9784dbe..d8837195 100644
--- a/docs/stats.md
+++ b/docs/stats.md
@@ -23,7 +23,7 @@ claim is only as good as the command behind it.
| figure | count | counted by |
|---|---:|---|
-| Unit tests | 2,854 | `grep -rn '^func Test' --include='*_test.go' . \| grep -v '/\.claude/worktrees/' \| grep -vc '^\./bench/'` |
+| Unit tests | 2,950 | `grep -rn '^func Test' --include='*_test.go' . \| grep -v '/\.claude/worktrees/' \| grep -vc '^\./bench/'` |
| Fuzz targets | 12 | `grep -rn '^func Fuzz' --include='*_test.go' . \| grep -v '/\.claude/worktrees/' \| grep -vc '^\./bench/'` |
| Limit boundary tests | 14 | `grep -hc '^func Test' ./*/limits_boundary*_test.go \| awk '{n += $1} END {print n + 0}'` |
diff --git a/docs/testing.md b/docs/testing.md
index 2eea7e20..093e85c3 100644
--- a/docs/testing.md
+++ b/docs/testing.md
@@ -21,7 +21,7 @@ let something through, and the column that matters is the last one.
| layer | count | catches | misses |
|---|---:|---|---|
-| **Unit tests** | 2,854 | a plausible implementation that is quietly wrong | anything nobody thought to write a test for |
+| **Unit tests** | 2,950 | a plausible implementation that is quietly wrong | anything nobody thought to write a test for |
| **Limit boundary tests** | 14 tests | an off-by-one or an overflow at the edge of a configurable limit | a limit nobody added to the inventory |
| **Race detector** | same tests | shared state a single-goroutine run never reveals | a data race on a path no test walks |
| **W3C conformance suites** | 152,241 cases | systematic divergence from the specification | what the suites do not ask about — see below |
diff --git a/docs/todo.md b/docs/todo.md
index 8c7001db..52ce996e 100644
--- a/docs/todo.md
+++ b/docs/todo.md
@@ -23,7 +23,7 @@ Current position:
| XSLT 3.0 | 99.80% — 11,495 of 11,518 in scope (23 failing); 3 of those need more of the §19.8 streamability analysis |
| RELAX NG | 100.00% — 965 of 965 |
| Schemas wrongly refused | 7 — 6 on XSD 1.0, 1 on 1.1 |
-| Tests | 2,854 `func Test` declarations, clean under `-race` |
+| Tests | 2,950 `func Test` declarations, clean under `-race` |
Every one of those failures, and why it is still open, is catalogued in
[known-gaps.md](known-gaps.md). This file is the forward-looking half — what
@@ -150,7 +150,8 @@ silently replaced the character — which is the corruption the MUST exists to
prevent, not merely a conformance divergence. `xslt/serialize.go` had the rule
right all along, so this was again one rule with two implementations.
-Both functions now delegate to `escapeXMLRunes`, which walks runes and decides
+Both functions now delegate to `escapeXMLRunes` (since the memory audit, content
+escaping streams through `writeEscaped`, its writer form), which walks runes and decides
by range rather than by a fixed list: adding U+0085 and U+2028 as two more
replacer entries would have closed the named half and left the ranges open.
`#x0` is deliberately not escaped — it is not a valid XML character at any
diff --git a/docs/xsd.md b/docs/xsd.md
index 9f5e85f3..93aecfb2 100644
--- a/docs/xsd.md
+++ b/docs/xsd.md
@@ -42,7 +42,9 @@ if errors.As(err, &invalid) {
The code — `cvc-complex-type.2.4.a`, `cvc-datatype-valid.1.2.1` — is what
lets a caller tell the kinds of failure apart without matching on message
-text. Message wording is not part of the contract; codes are.
+text. Message wording is not part of the contract; codes are. A name or value
+the document supplies is clipped to 1 KB plus "..." in the message, so a
+hostile input cannot make the retained errors large.
There are four ways in, differing only in where the schema comes from:
diff --git a/dtd/alloc_perelement_test.go b/dtd/alloc_perelement_test.go
new file mode 100644
index 00000000..c3c58e57
--- /dev/null
+++ b/dtd/alloc_perelement_test.go
@@ -0,0 +1,35 @@
+package dtd
+
+import (
+ "runtime"
+ "strings"
+ "testing"
+
+ "github.com/knroy/go-xml/v2/xdm"
+)
+
+// With no ATTLIST for an element, checkAttributes built a map anyway, and the
+// child-name slice was rebuilt for every element.
+func TestValidateAllocsPerElement(t *testing.T) {
+ d, err := Parse(`
+ ]>`)
+ if err != nil {
+ t.Fatal(err)
+ }
+ const n = 2000
+ tree, err := xdm.ParseString(""+strings.Repeat("
", n)+"", xdm.ParseOptions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ var m0, m1 runtime.MemStats
+ runtime.ReadMemStats(&m0)
+ err = Validate(tree.Root, d, Options{})
+ runtime.ReadMemStats(&m1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // 6001 elements: 8062 allocations with the fix, 12062 without.
+ if got := m1.Mallocs - m0.Mallocs; got > 10000 {
+ t.Errorf("%d allocations for %d elements, want at most 10000", got, 3*n+1)
+ }
+}
diff --git a/dtd/message_clip_test.go b/dtd/message_clip_test.go
new file mode 100644
index 00000000..f9de1fbe
--- /dev/null
+++ b/dtd/message_clip_test.go
@@ -0,0 +1,36 @@
+package dtd
+
+import (
+ "errors"
+ "strings"
+ "testing"
+
+ "github.com/knroy/go-xml/v2/xdm"
+)
+
+// A value quoted in a message is clipped: 200 mismatches against a 1 MB
+// #FIXED value kept 105 MB in the errors. The message keeps its prefix.
+func TestValidityMessagesClipLongValues(t *testing.T) {
+ big := strings.Repeat("v", 1<<20)
+ d, err := Parse(`
+ ]>`)
+ if err != nil {
+ t.Fatal(err)
+ }
+ tree, err := xdm.ParseString(""+strings.Repeat(``, 200)+"", xdm.ParseOptions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ var errs *Errors
+ if err := Validate(tree.Root, d, Options{}); !errors.As(err, &errs) || len(errs.Errors) == 0 {
+ t.Fatalf("expected Errors, got %v", err)
+ }
+ for _, e := range errs.Errors {
+ if len(e.Message) > 1024+len("...") {
+ t.Fatalf("message is %d bytes, want at most 1027", len(e.Message))
+ }
+ if !strings.HasPrefix(e.Message, `attribute a is #FIXED "vvvv`) || !strings.HasSuffix(e.Message, "...") {
+ t.Fatalf("message lost its prefix or its clip marker: %.200q", e.Message)
+ }
+ }
+}
diff --git a/dtd/path_alloc_test.go b/dtd/path_alloc_test.go
new file mode 100644
index 00000000..c44c1ae1
--- /dev/null
+++ b/dtd/path_alloc_test.go
@@ -0,0 +1,52 @@
+package dtd
+
+import (
+ "runtime"
+ "strings"
+ "testing"
+
+ "github.com/knroy/go-xml/v2/xdm"
+)
+
+// The path is rendered only for a recorded failure. Concatenated at every
+// element it cost the square of the depth: a valid 990-deep chain of 4 KB
+// names allocated 2 GB. The text of a reported path is unchanged, including
+// for an IDREF, which is resolved after the walk has left the element.
+func TestPathBuiltOnlyForFailures(t *testing.T) {
+ d, err := Parse(`
+ ]>`)
+ if err != nil {
+ t.Fatal(err)
+ }
+ tree, err := xdm.ParseString(``, xdm.ParseOptions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ const want = "2 validity errors:\n" +
+ " /r/a/b: attribute x is #FIXED \"0\" but is \"1\"\n" +
+ " /r/a/c: IDREF \"nope\" matches no ID in the document"
+ if err := Validate(tree.Root, d, Options{}); err == nil || err.Error() != want {
+ t.Fatalf("got\n%v\nwant\n%s", err, want)
+ }
+
+ nm := "n" + strings.Repeat("x", 4095)
+ d, err = Parse(`]>`)
+ if err != nil {
+ t.Fatal(err)
+ }
+ tree, err = xdm.ParseString(strings.Repeat("<"+nm+">", 990)+strings.Repeat("", 990), xdm.ParseOptions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ var m0, m1 runtime.MemStats
+ runtime.ReadMemStats(&m0)
+ err = Validate(tree.Root, d, Options{})
+ runtime.ReadMemStats(&m1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := m1.TotalAlloc - m0.TotalAlloc; got > 16<<20 {
+ t.Errorf("validating a valid deep chain allocated %d MB, want under 16 MB", got>>20)
+ }
+}
diff --git a/dtd/validate.go b/dtd/validate.go
index 6e6692d8..e95fb948 100644
--- a/dtd/validate.go
+++ b/dtd/validate.go
@@ -2,6 +2,7 @@ package dtd
import (
"fmt"
+ "slices"
"strings"
"github.com/knroy/go-xml/v2/xdm"
@@ -133,7 +134,8 @@ func Validate(doc *xdm.Node, d *DTD, opts Options) error {
}
}
}
- v.walk(root, "")
+ v.root = root
+ v.walk(root)
v.checkIDRefs()
if len(v.errs) == 0 {
return nil
@@ -147,6 +149,7 @@ type validator struct {
allowUndec bool
errs []*Error
matchers map[string]*xsd.SequenceMatcher
+ names []xdm.QName // child-name buffer reused across elements
// ids are every ID value seen, for uniqueness and for resolving IDREF.
ids map[string]bool
// partial records that the DTD was read from the internal subset alone
@@ -161,40 +164,77 @@ type validator struct {
// refs are IDREF values and where they appeared, checked once the whole
// document has been read — a reference may point forward.
refs []idref
+ // root is the element the walk started from, where a path begins.
+ root *xdm.Node
}
type idref struct {
value string
- path string
+ el *xdm.Node
}
-func (v *validator) fail(path, format string, args ...any) {
+func (v *validator) fail(el *xdm.Node, format string, args ...any) {
if v.max > 0 && len(v.errs) >= v.max {
return
}
- v.errs = append(v.errs, &Error{Path: path, Message: fmt.Sprintf(format, args...)})
+ // A #FIXED or enumerated value comes from the DTD or the document and
+ // may be megabytes; 200 mismatches against a 1 MB #FIXED value kept
+ // 105 MB. Arguments are clipped before formatting, the message after.
+ for i, a := range args {
+ switch x := a.(type) {
+ case string:
+ args[i] = clip(x)
+ case error:
+ args[i] = clip(x.Error())
+ }
+ }
+ v.errs = append(v.errs, &Error{Path: v.pathOf(el), Message: clip(fmt.Sprintf(format, args...))})
+}
+
+// clip shortens s to 1 KB, marking the cut with "..." as the xsd package
+// does. The concatenation copies, so the result never pins the original.
+func clip(s string) string {
+ const max = 1024
+ if len(s) <= max {
+ return s
+ }
+ return s[:max] + "..."
}
-func (v *validator) walk(el *xdm.Node, parentPath string) {
- path := parentPath + "/" + el.Name().Local
- v.checkContent(el, path)
- v.checkAttributes(el, path)
+// pathOf renders the path from the root to el, as "/root/child". It is built
+// only for a recorded failure: concatenating it at every element cost the
+// square of the depth, 2 GB allocated for a valid 990-deep chain of 4 KB names.
+func (v *validator) pathOf(el *xdm.Node) string {
+ var names []string
+ for n := el; ; n = n.Parent() {
+ names = append(names, n.Name().Local)
+ if n == v.root {
+ break
+ }
+ }
+ slices.Reverse(names)
+ return "/" + strings.Join(names, "/")
+}
+
+func (v *validator) walk(el *xdm.Node) {
+ v.checkContent(el)
+ v.checkAttributes(el)
for c := range el.Children() {
if c.Kind() == xdm.KindElement {
- v.walk(c, path)
+ v.walk(c)
}
}
}
// checkContent applies the element's content model.
-func (v *validator) checkContent(el *xdm.Node, path string) {
+func (v *validator) checkContent(el *xdm.Node) {
decl, ok := v.dtd.Elements[el.Name().Local]
if !ok {
// An undeclared element is a validity error in a DTD-validated
// document — unlike XSD, where a wildcard may admit it. A caller
// working with a deliberately partial internal subset can say so.
if !v.allowUndec {
- v.fail(path, "element %s is not declared", el.Name().Local)
+ v.fail(el, "element %s is not declared", el.Name().Local)
}
return
}
@@ -204,7 +244,7 @@ func (v *validator) checkContent(el *xdm.Node, path string) {
case ContentEmpty:
for c := range el.Children() {
if c.Kind() == xdm.KindElement || (c.Kind() == xdm.KindText && strings.TrimSpace(c.Value()) != "") {
- v.fail(path, "element %s is declared EMPTY but has content",
+ v.fail(el, "element %s is declared EMPTY but has content",
el.Name().Local)
return
}
@@ -212,7 +252,7 @@ func (v *validator) checkContent(el *xdm.Node, path string) {
case ContentMixed:
for c := range el.Children() {
if c.Kind() == xdm.KindElement && !decl.Mixed[c.Name().Local] {
- v.fail(path, "element %s is not permitted in the mixed "+
+ v.fail(el, "element %s is not permitted in the mixed "+
"content of %s", c.Name().Local, el.Name().Local)
}
}
@@ -221,28 +261,29 @@ func (v *validator) checkContent(el *xdm.Node, path string) {
// unexpected elements: "(a, b)" does not admit text between them.
for c := range el.Children() {
if c.Kind() == xdm.KindText && strings.TrimSpace(c.Value()) != "" {
- v.fail(path, "element %s has element-only content but "+
+ v.fail(el, "element %s has element-only content but "+
"contains character data", el.Name().Local)
break
}
}
m, err := v.matcherFor(decl)
if err != nil {
- v.fail(path, "content model of %s: %v", el.Name().Local, err)
+ v.fail(el, "content model of %s: %v", el.Name().Local, err)
return
}
- var names []xdm.QName
+ names := v.names[:0]
for c := range el.Children() {
if c.Kind() == xdm.KindElement {
names = append(names, xdm.QName{Local: c.Name().Local})
}
}
+ v.names = names
if ok, at := m.Match(names); !ok {
if at < len(names) {
- v.fail(path, "element %s is not permitted here in the "+
+ v.fail(el, "element %s is not permitted here in the "+
"content of %s", names[at].Local, el.Name().Local)
} else {
- v.fail(path, "the content of %s is incomplete", el.Name().Local)
+ v.fail(el, "the content of %s is incomplete", el.Name().Local)
}
}
}
@@ -261,8 +302,11 @@ func (v *validator) matcherFor(decl *Element) (*xsd.SequenceMatcher, error) {
}
// checkAttributes applies the ATTLIST declarations for an element.
-func (v *validator) checkAttributes(el *xdm.Node, path string) {
+func (v *validator) checkAttributes(el *xdm.Node) {
decls := v.dtd.Attributes[el.Name().Local]
+ if len(decls) == 0 {
+ return
+ }
present := map[string]string{}
for a := range el.Attrs() {
name := a.Name().Local
@@ -280,16 +324,16 @@ func (v *validator) checkAttributes(el *xdm.Node, path string) {
switch d.Default {
case AttrRequired:
if !have {
- v.fail(path, "required attribute %s is missing", d.Name)
+ v.fail(el, "required attribute %s is missing", d.Name)
continue
}
case AttrFixed:
if have && val != d.Value {
- v.fail(path, "attribute %s is #FIXED %q but is %q",
+ v.fail(el, "attribute %s is #FIXED %q but is %q",
d.Name, d.Value, val)
}
case AttrInvalid:
- v.fail(path, "attribute %s has an unrecognised default "+
+ v.fail(el, "attribute %s has an unrecognised default "+
"declaration %s, so its presence was not checked",
d.Name, d.Value)
}
@@ -297,7 +341,7 @@ func (v *validator) checkAttributes(el *xdm.Node, path string) {
continue
}
if len(d.Enum) > 0 && !containsString(d.Enum, val) {
- v.fail(path, "attribute %s = %q is not one of %s",
+ v.fail(el, "attribute %s = %q is not one of %s",
d.Name, val, strings.Join(d.Enum, ", "))
}
switch d.Type {
@@ -309,34 +353,34 @@ func (v *validator) checkAttributes(el *xdm.Node, path string) {
// new here, and it is a property of the DECLARATION rather than
// of this element's value — so it is reported once per
// declaration, not once per occurrence.
- v.checkNotationDecl(path, d)
+ v.checkNotationDecl(el, d)
case "ENTITY":
// §3.3.1: the value must be the name of an unparsed entity —
// declared, and declared with an NDATA notation. A parsed
// entity's name is as wrong here as an undeclared one.
- v.checkEntityValue(path, d.Name, val)
+ v.checkEntityValue(el, d.Name, val)
case "ENTITIES":
// The same rule over a whitespace-separated list. An empty list
// is not a Names, so it fails the type as surely as an unknown
// name would.
names := strings.Fields(val)
if len(names) == 0 {
- v.fail(path, "attribute %s is ENTITIES but is empty, which "+
+ v.fail(el, "attribute %s is ENTITIES but is empty, which "+
"names no unparsed entity", d.Name)
}
for _, n := range names {
- v.checkEntityValue(path, d.Name, n)
+ v.checkEntityValue(el, d.Name, n)
}
case "ID":
if v.ids[val] {
- v.fail(path, "duplicate ID %q", val)
+ v.fail(el, "duplicate ID %q", val)
}
v.ids[val] = true
case "IDREF":
- v.refs = append(v.refs, idref{val, path})
+ v.refs = append(v.refs, idref{val, el})
case "IDREFS":
for _, r := range strings.Fields(val) {
- v.refs = append(v.refs, idref{r, path})
+ v.refs = append(v.refs, idref{r, el})
}
case "CDATA", "NMTOKEN", "NMTOKENS", "ENUMERATION":
// Declared, and either unconstrained (CDATA) or constrained by
@@ -348,7 +392,7 @@ func (v *validator) checkAttributes(el *xdm.Node, path string) {
// Without this branch an unrecognised type — "IDREFF" for
// "IDREFS" — falls through to no check at all, and the caller
// cannot tell an attribute that passed from one never examined.
- v.fail(path, "attribute %s is declared with the unrecognised "+
+ v.fail(el, "attribute %s is declared with the unrecognised "+
"type %s, so its value was not checked", d.Name, d.Type)
}
}
@@ -361,7 +405,7 @@ func (v *validator) checkAttributes(el *xdm.Node, path string) {
// It is reported once per declaration rather than once per element carrying
// the attribute, because the fault is in the DTD and repeating it for every
// instance would bury the finding under its own copies.
-func (v *validator) checkNotationDecl(path string, d *Attribute) {
+func (v *validator) checkNotationDecl(el *xdm.Node, d *Attribute) {
if v.partial {
// The declarations were never all read, so a name absent from
// Notations may simply be in the half that was not fetched.
@@ -376,20 +420,20 @@ func (v *validator) checkNotationDecl(path string, d *Attribute) {
v.notationsChecked[key] = true
for _, n := range d.Enum {
if !v.dtd.Notations[n] {
- v.fail(path, "attribute %s permits the notation %s, which no "+
+ v.fail(el, "attribute %s permits the notation %s, which no "+
" declares", d.Name, n)
}
}
}
// checkEntityValue applies §3.3.1's Entity Name constraint to one name.
-func (v *validator) checkEntityValue(path, attr, name string) {
+func (v *validator) checkEntityValue(el *xdm.Node, attr, name string) {
if v.partial {
// As for notations: half a DTD cannot say an entity is undeclared.
return
}
if !v.dtd.Unparsed[name] {
- v.fail(path, "attribute %s = %q names no unparsed entity", attr, name)
+ v.fail(el, "attribute %s = %q names no unparsed entity", attr, name)
}
}
@@ -398,7 +442,7 @@ func (v *validator) checkEntityValue(path, attr, name string) {
func (v *validator) checkIDRefs() {
for _, r := range v.refs {
if !v.ids[r.value] {
- v.fail(r.path, "IDREF %q matches no ID in the document", r.value)
+ v.fail(r.el, "IDREF %q matches no ID in the document", r.value)
}
}
}
diff --git a/internal/xmltok/xmltok.go b/internal/xmltok/xmltok.go
index cf987ee6..3a85fd5e 100644
--- a/internal/xmltok/xmltok.go
+++ b/internal/xmltok/xmltok.go
@@ -162,7 +162,8 @@ type Decoder struct {
// Size, when set, is about how many bytes the stream holds. A long text
// run that outgrows the scratch sizes it from what is left, rather than
- // growing it by append's steps window by window.
+ // growing it by append's steps window by window, and the string arena's
+ // first block is no larger than it.
Size int
src io.Reader
@@ -213,17 +214,22 @@ func NewDecoder(r io.Reader) *Decoder {
return &Decoder{Strict: true, src: r, buf: make([]byte, 0, bufSize)}
}
-// NewDecoderSize is NewDecoder for a stream known to hold about size bytes:
-// its read window is no larger than the input needs, where a small document
-// otherwise paid for 4 KB. The string arena is not sized down: a smaller
-// arena retains less heap per tree, which made the collector run more often
-// (docs/profiling.md, "Measured and rejected").
-func NewDecoderSize(r io.Reader, size int) *Decoder {
- return &Decoder{Strict: true, src: r, buf: make([]byte, 0, min(max(size, minWindow), bufSize))}
+// NewDecoderBuffer is NewDecoder reading through buf, whose capacity is the
+// read window, so that a caller parsing many documents can reuse one. The
+// Decoder writes into buf until the caller is done with it and with every
+// token it returned.
+func NewDecoderBuffer(r io.Reader, buf []byte) *Decoder {
+ return &Decoder{Strict: true, src: r, buf: buf[:0]}
}
-// minWindow is the smallest read window NewDecoderSize uses.
-const minWindow = 512
+// TakeText hands the caller the CharData last returned, to keep: the Decoder
+// builds its next token in a new buffer rather than over it. It is called
+// only directly after RawToken returned a CharData.
+func (d *Decoder) TakeText() []byte {
+ b := d.tokText
+ d.scratch = nil
+ return b
+}
// IsVersion11 reports whether the document declared version="1.1". The
// version comes from the XML declaration and nowhere else, so there is no
@@ -418,6 +424,7 @@ func (d *Decoder) startTag() (Token, error) {
return nil, d.orSyntaxError("expected element name after <")
}
attrs := d.attrs[:0]
+ start := d.InputOffset()
for {
spaced := d.space()
b, ok := d.mustgetc()
@@ -463,6 +470,15 @@ func (d *Decoder) startTag() (Token, error) {
if !ok {
return nil, d.err
}
+ if d.arena.buf == nil && d.Size > 0 {
+ // A document smaller than a block gets a first block its own
+ // size; the values of a larger one, or what entities expand
+ // to, go on to full blocks.
+ d.arena.buf = make([]byte, 0, min(d.Size, arenaBlock))
+ }
+ if len(attrs) == cap(attrs) && len(attrs) >= bigTag {
+ attrs = d.growAttrs(attrs, start)
+ }
attrs = append(attrs, Attr{an, d.arena.String(v)})
}
d.attrs = attrs
@@ -470,6 +486,25 @@ func (d *Decoder) startTag() (Token, error) {
return &d.tokStart, nil
}
+// bigTag is how many attributes a start tag has before startTag stops growing
+// the slice of them by append's steps (1.25x at this size, so a chain of them
+// allocates 5x the final slice).
+const bigTag = 256
+
+// growAttrs makes room in attrs for the rest of a tag with many attributes.
+// The input left, over the bytes an attribute has taken so far, bounds how many
+// remain; ponytail: capped at 8x the slice, so that a tag early in a huge
+// document takes at most that, and at least doubling.
+func (d *Decoder) growAttrs(attrs []Attr, start int64) []Attr {
+ est := len(attrs)
+ if rest := d.Size - int(d.InputOffset()); rest > 0 {
+ if used := int(d.InputOffset() - start); used > 0 {
+ est = rest / max(used/len(attrs), 1)
+ }
+ }
+ return append(make([]Attr, 0, len(attrs)+max(len(attrs), min(est, 8*len(attrs)))), attrs...)
+}
+
// procInst reads [16] PI after "", and the body is checked
// against [2] Char.
diff --git a/internal/xpathleaf/locals.go b/internal/xpathleaf/locals.go
index 112ac438..4aec00f3 100644
--- a/internal/xpathleaf/locals.go
+++ b/internal/xpathleaf/locals.go
@@ -108,6 +108,18 @@ func (l *Locals) Frozen() *Locals {
vals: append([]xdm.Sequence(nil), l.vals[l.base:]...)}
}
+// FrozenOnly is Frozen keeping only the bindings keep accepts.
+func (l *Locals) FrozenOnly(keep func(uri, local string) bool) *Locals {
+ f := &Locals{}
+ for i := l.base; i < len(l.names); i++ {
+ if n := l.names[i]; keep(n.uri, n.local) {
+ f.names = append(f.names, n)
+ f.vals = append(f.vals, l.vals[i])
+ }
+ }
+ return f
+}
+
// Package xpath sets these in its init; ctx and base are *xpath.Context.
// WithLocals returns a child scope of ctx resolving names through l.
// WithBindingsOf returns a copy of ctx whose variable bindings are base's.
diff --git a/relaxng/choice_dedupe_test.go b/relaxng/choice_dedupe_test.go
index 780590b2..23d39cfa 100644
--- a/relaxng/choice_dedupe_test.go
+++ b/relaxng/choice_dedupe_test.go
@@ -59,3 +59,33 @@ func TestChoiceMergesEqualAlternatives(t *testing.T) {
t.Error("choice(value, value) merged two values")
}
}
+
+// (a?){n} a{n} against n is valid. The derivative is a choice of
+// choices whose chains share nearly every alternative; merging only a
+// whole chain left the copies in, and from n = 13 the pattern passed the
+// 100,000-node bound and the valid document was refused. xmllint agrees
+// that it validates for n = 12, 13, 18 and 30.
+func TestOptionalsBeforeRequiredStayValid(t *testing.T) {
+ for _, n := range []int{12, 13, 18, 30} {
+ var b strings.Builder
+ b.WriteString(``)
+ for i := 0; i < n; i++ {
+ b.WriteString(``)
+ }
+ for i := 0; i < n; i++ {
+ b.WriteString(``)
+ }
+ b.WriteString(``)
+ s, err := compileString(t, b.String())
+ if err != nil {
+ t.Fatalf("compile: %v", err)
+ }
+ doc, err := xdm.ParseString(""+strings.Repeat("", n)+"", xdm.ParseOptions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := s.Validate(doc.Root); err != nil {
+ t.Errorf("n=%d: %v", n, err)
+ }
+ }
+}
diff --git a/relaxng/derive.go b/relaxng/derive.go
index bfd26766..e848d38e 100644
--- a/relaxng/derive.go
+++ b/relaxng/derive.go
@@ -210,22 +210,56 @@ func (pb *patBuilder) choice(a, b pattern) pattern {
if patEq(a, b) || pb.inChoice(a, b) {
return a
}
+ // A choice on the right is merged alternative by alternative. Taking the
+ // derivative of a choice of choices yields chains that share most of their
+ // alternatives, and testing the whole chain against a only finds a chain
+ // that is identical: (a?){13} a{13} then grew past the pattern bound while
+ // holding a few dozen distinct alternatives.
+ //
+ // The merge compares every pair of alternatives, so it is skipped when
+ // the chains are long enough to make that dearer than the copies it
+ // saves: alternatives that never compare equal (values, data) would
+ // otherwise buy a quadratic cost per choice from a hostile document.
+ // ponytail: a fixed pair budget; a hash of the alternatives if a schema
+ // needs more.
+ if c, ok := b.(*choicePat); ok && altCount(a)*altCount(b) <= maxMergePairs {
+ return pb.choice(pb.choice(a, c.Left), c.Right)
+ }
if pb != nil {
return pb.mk(kChoice, a, b)
}
return newChoicePat(a, b)
}
+// maxMergePairs bounds the pairs of alternatives choice compares when it
+// merges one chain into another.
+const maxMergePairs = 1 << 10
+
+// altCount is the number of alternatives in a chain of choices.
+func altCount(p pattern) int {
+ n := 1
+ for {
+ c, ok := unstatic(p).(*choicePat)
+ if !ok {
+ return n
+ }
+ n += altCount(c.Right)
+ p = c.Left
+ }
+}
+
// inChoice reports whether b is already one of a's alternatives. A chain of
// choices is built leftwards, so the alternatives are the Right of each link
// and the Left of the last.
func (pb *patBuilder) inChoice(a, b pattern) bool {
+ // Equal patterns have equal sizes, so a size mismatch skips the walk.
+ sb := patternSize(b)
for {
c, ok := unstatic(a).(*choicePat)
if !ok {
- return patEq(a, b)
+ return patternSize(a) == sb && patEq(a, b)
}
- if patEq(c.Right, b) {
+ if patternSize(c.Right) == sb && patEq(c.Right, b) {
return true
}
a = c.Left
diff --git a/relaxng/namememo_test.go b/relaxng/namememo_test.go
new file mode 100644
index 00000000..4502b927
--- /dev/null
+++ b/relaxng/namememo_test.go
@@ -0,0 +1,54 @@
+package relaxng
+
+import (
+ "fmt"
+ "runtime"
+ "testing"
+
+ "github.com/knroy/go-xml/v2/xdm"
+)
+
+// store used to copy the whole map every time: 1000 stores allocated tens of
+// megabytes. Now the copies sum to O(n). Concurrent readers must stay safe
+// (run under -race) and the cap must hold.
+func TestNameMemoStoreAmortised(t *testing.T) {
+ var c nameMemo
+ var m0, m1 runtime.MemStats
+ runtime.ReadMemStats(&m0)
+ for i := 0; i < maxOpenMemo; i++ {
+ c.store(xdm.QName{Local: fmt.Sprintf("n%d", i)}, emptyPat{})
+ }
+ runtime.ReadMemStats(&m1)
+ if got := m1.TotalAlloc - m0.TotalAlloc; got > 2<<20 {
+ t.Errorf("%d stores allocated %d KB, want under 2048 KB", maxOpenMemo, got>>10)
+ }
+ if _, ok := c.load(xdm.QName{Local: "n0"}); !ok {
+ t.Error("first entry not published")
+ }
+ c.store(xdm.QName{Local: "over"}, emptyPat{})
+ c.store(xdm.QName{Local: "over2"}, emptyPat{})
+ n := len(*c.m.Load()) + len(c.pending)
+ if n > maxOpenMemo {
+ t.Errorf("%d entries, cap is %d", n, maxOpenMemo)
+ }
+}
+
+// Pending entries are published after a few locked hits, so a steady
+// lookup stream becomes lock-free instead of staying on the slow path.
+func TestNameMemoPendingPublishedOnHits(t *testing.T) {
+ var c nameMemo
+ for _, n := range []string{"a", "b"} { // 2nd store leaves one pending
+ c.store(xdm.QName{Local: n}, emptyPat{})
+ }
+ if c.np.Load() == 0 {
+ t.Fatal("setup: expected a pending entry")
+ }
+ for i := 0; i < pendingHitsPublish; i++ {
+ if _, ok := c.load(xdm.QName{Local: "b"}); !ok {
+ t.Fatal("pending entry lost")
+ }
+ }
+ if c.np.Load() != 0 || len(*c.m.Load()) != 2 {
+ t.Errorf("pending not published: np=%d snapshot=%d", c.np.Load(), len(*c.m.Load()))
+ }
+}
diff --git a/relaxng/pattern.go b/relaxng/pattern.go
index f4510ecf..df99bffc 100644
--- a/relaxng/pattern.go
+++ b/relaxng/pattern.go
@@ -175,25 +175,47 @@ type staticInfo struct {
// nameMemo is a copy-on-write map from a name to a pattern, shared by every
// validation of a schema: a lookup is one atomic load and a map read with no
-// boxing, and a store copies the map under a lock. The entries are capped at
+// boxing, and a store lands in a small pending map under a lock and is published as a
+// new snapshot once the pending entries outgrow it. The entries are capped at
// maxOpenMemo so that a document cannot grow it without bound by inventing
// names, which also bounds each copy.
type nameMemo struct {
- m atomic.Pointer[map[nameKey]pattern]
- mu sync.Mutex
+ m atomic.Pointer[map[nameKey]pattern]
+ mu sync.Mutex
+ pending map[nameKey]pattern // stored but not yet published in m; under mu
+ np atomic.Int32 // len(pending), so a load skips the lock when empty
+ hits int // locked pending hits since the last publish; under mu
}
type nameKey struct{ ns, local string }
func (c *nameMemo) load(n xdm.QName) (pattern, bool) {
- m := c.m.Load()
- if m == nil {
+ k := nameKey{n.URI, n.Local}
+ if m := c.m.Load(); m != nil {
+ if p, ok := (*m)[k]; ok {
+ return p, true
+ }
+ }
+ if c.np.Load() == 0 {
return nil, false
}
- p, ok := (*m)[nameKey{n.URI, n.Local}]
+ c.mu.Lock()
+ p, ok := c.pending[k]
+ // A pending hit costs a lock; after a few, publish so later lookups
+ // are lock-free again.
+ if ok {
+ if c.hits++; c.hits >= pendingHitsPublish {
+ c.publishLocked()
+ }
+ }
+ c.mu.Unlock()
return p, ok
}
+// pendingHitsPublish is how many locked lookups of pending entries trigger
+// their publication.
+const pendingHitsPublish = 8
+
func (c *nameMemo) store(n xdm.QName, p pattern) {
c.mu.Lock()
defer c.mu.Unlock()
@@ -201,15 +223,42 @@ func (c *nameMemo) store(n xdm.QName, p pattern) {
if o := c.m.Load(); o != nil {
old = *o
}
- if len(old) >= maxOpenMemo {
- return
+ k := nameKey{n.URI, n.Local}
+ if _, ok := c.pending[k]; !ok {
+ if len(old)+len(c.pending) >= maxOpenMemo {
+ return
+ }
+ }
+ if c.pending == nil {
+ c.pending = make(map[nameKey]pattern)
+ }
+ c.pending[k] = p
+ c.np.Store(int32(len(c.pending)))
+ // Publish once the pending entries outgrow the snapshot, so the copies
+ // sum to O(n) over n stores. A load that misses the snapshot checks the
+ // pending map.
+ if len(c.pending) > len(old) {
+ c.publishLocked()
}
- m := make(map[nameKey]pattern, len(old)+1)
+}
+
+// publishLocked folds the pending entries into a new snapshot.
+func (c *nameMemo) publishLocked() {
+ var old map[nameKey]pattern
+ if o := c.m.Load(); o != nil {
+ old = *o
+ }
+ m := make(map[nameKey]pattern, len(old)+len(c.pending))
for k, v := range old {
m[k] = v
}
- m[nameKey{n.URI, n.Local}] = p
+ for k, v := range c.pending {
+ m[k] = v
+ }
c.m.Store(&m)
+ c.pending = nil
+ c.np.Store(0)
+ c.hits = 0
}
type patBox struct {
diff --git a/relaxng/pattern_size_test.go b/relaxng/pattern_size_test.go
index 5bc3c760..8775fefc 100644
--- a/relaxng/pattern_size_test.go
+++ b/relaxng/pattern_size_test.go
@@ -110,7 +110,7 @@ func TestPlainOneOrMoreStillValidates(t *testing.T) {
// at twelve, which still finishes, rather than hanging at fourteen.
func TestAttributePatternSizeIsBounded(t *testing.T) {
const sch = `
-`
+`
schTree, err := xdm.ParseString(sch, xdm.ParseOptions{})
if err != nil {
t.Fatal(err)
diff --git a/relaxng/validate.go b/relaxng/validate.go
index d93ffcd5..0ed1518e 100644
--- a/relaxng/validate.go
+++ b/relaxng/validate.go
@@ -111,8 +111,8 @@ func (s *Schema) ValidateWithOptions(doc *xdm.Node, opts ValidateOptions) error
return &Error{
Path: v.deepPath,
Message: fmt.Sprintf(
- "the derivative pattern exceeds %d nodes; a oneOrMore nested "+
- "inside a oneOrMore grows the pattern multiplicatively in "+
+ "the derivative pattern exceeds %d nodes; nested repetition "+
+ "or optional content grows the pattern multiplicatively in "+
"the number of children", maxPattern),
}
}
diff --git a/xdm/atomic.go b/xdm/atomic.go
index f062320e..4041874d 100644
--- a/xdm/atomic.go
+++ b/xdm/atomic.go
@@ -669,3 +669,44 @@ func NewGregorian(dt *DateTime, t TypeCode) *Atomic {
func NewDuration(d *Duration, t TypeCode) *Atomic {
return &Atomic{Type: t, ext: &atomicExt{dur: d}, str: d.Lexical(t)}
}
+
+// integerChunk is how many xs:integers AppendIntegers allocates at once.
+const integerChunk = 128
+
+// AppendIntegers appends the count integers lo, lo+1, ... to out. It allocates
+// the Atomics, rationals and numerator words a chunk at a time rather than
+// three allocations per value, which is what NewInteger costs; the values
+// are indistinguishable from NewInteger's. The caller must ensure lo+count-1
+// does not overflow int64.
+func AppendIntegers(out Sequence, lo int64, count int) Sequence {
+ var (
+ atoms []Atomic
+ rats []big.Rat
+ words []big.Word
+ )
+ for i := 0; i < count; i++ {
+ v := lo + int64(i)
+ if v >= smallIntMin && v < smallIntMin+int64(len(smallInts)) {
+ out = append(out, NewInteger(v))
+ continue
+ }
+ if len(atoms) == 0 {
+ n := min(integerChunk, count-i)
+ atoms, rats, words = make([]Atomic, n), make([]big.Rat, n), make([]big.Word, n)
+ }
+ a, r := &atoms[0], &rats[0]
+ if v < 0 {
+ words[0] = big.Word(-uint64(v))
+ } else {
+ words[0] = big.Word(v)
+ }
+ r.Num().SetBits(words[:1:1])
+ if v < 0 {
+ r.Num().Neg(r.Num())
+ }
+ *a = Atomic{Type: TypeInteger, dec: r}
+ atoms, rats, words = atoms[1:], rats[1:], words[1:]
+ out = append(out, a)
+ }
+ return out
+}
diff --git a/xdm/build.go b/xdm/build.go
index 559916f5..a1401791 100644
--- a/xdm/build.go
+++ b/xdm/build.go
@@ -261,10 +261,10 @@ func (t *Tree) CopySourceFrom(src *Tree) {
}
t.CopyDTDFrom(src)
t.XMLVersion = src.XMLVersion
- if s := src.srcText(); s != "" {
- t.ownSource().src = s
+ if l := src.lineIdx(); l != nil {
+ t.ownSource().lines = l
} else if t.source != nil {
- t.source.src = ""
+ t.source.lines = nil
}
}
@@ -279,7 +279,7 @@ func CopyPosition(dst, src *Node) {
if !ok {
return
}
- if _, foreign := src.tree.foreign(src.self); !foreign && dst.tree.srcText() == src.tree.srcText() {
+ if _, foreign := src.tree.foreign(src.self); !foreign && dst.tree.lineIdx() == src.tree.lineIdx() {
dst.setOffset(src.offset())
return
}
diff --git a/xdm/clone_test.go b/xdm/clone_test.go
index 6a967d00..5f3abbbd 100644
--- a/xdm/clone_test.go
+++ b/xdm/clone_test.go
@@ -105,8 +105,8 @@ func TestCloneInternsPastSmallNames(t *testing.T) {
}
got := xdmclone.Clone(tree.Root, xdmclone.Options{})(tree.Root).(*Node)
c := got.Tree()
- if c == nil || c.nameIx == nil {
- t.Fatalf("clone has no name index to exercise (tree %v)", c)
+ if c == nil || len(c.names) <= smallNames {
+ t.Fatalf("clone has too few names to need an index (tree %v)", c)
}
old := c.intern(QName{Local: "e3"})
if c.names[old] != (QName{Local: "e3"}) {
diff --git a/xdm/concurrency_test.go b/xdm/concurrency_test.go
index 2ea88bb7..b05e790a 100644
--- a/xdm/concurrency_test.go
+++ b/xdm/concurrency_test.go
@@ -1,14 +1,15 @@
package xdm
import (
+ "fmt"
"strings"
"sync"
"testing"
)
-// Position resolution builds a line index lazily on first use, guarded by a
-// sync.Once. A parsed tree is shared across concurrent transforms, so several
-// goroutines can reach that initialisation at the same moment.
+// Position resolution reads the tree's line index. A parsed tree is shared
+// across concurrent transforms, so several goroutines resolve positions in it
+// at the same moment.
func TestConcurrentPositionLookup(t *testing.T) {
var b strings.Builder
b.WriteString("\n")
@@ -78,3 +79,31 @@ func TestConcurrentDocumentOrder(t *testing.T) {
}
wg.Wait()
}
+
+// Parses share pooled read windows (readBufs). Concurrent parses of
+// different documents, through readers that hand data over a few bytes at a
+// time, must each build their own tree; run under -race this also checks
+// that no window is used by two parses at once.
+func TestConcurrentParsesShareNoWindow(t *testing.T) {
+ var wg sync.WaitGroup
+ for g := range 8 {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ doc := fmt.Sprintf("%s", g, strings.Repeat(fmt.Sprintf("%d\r\n", g), 500))
+ want := strings.Repeat(fmt.Sprintf("%d\n", g), 500)
+ for range 20 {
+ tree, err := Parse(&chunkReader{s: doc, n: 7}, ParseOptions{})
+ if err != nil {
+ t.Error(err)
+ return
+ }
+ if got := tree.Root.StringValue(); got != want {
+ t.Errorf("goroutine %d parsed %.40q, want %.40q", g, got, want)
+ return
+ }
+ }
+ }()
+ }
+ wg.Wait()
+}
diff --git a/xdm/dtd_attnames_test.go b/xdm/dtd_attnames_test.go
index e2fd0eb9..96841dfb 100644
--- a/xdm/dtd_attnames_test.go
+++ b/xdm/dtd_attnames_test.go
@@ -45,9 +45,10 @@ func TestDTDAttributeTypingDoesNotBuildNames(t *testing.T) {
}
el := NewNode(KindElement, QName{Prefix: "schema", Local: "element-with-a-long-name-x"}, "")
el.AppendAttr(QName{Prefix: "prefix", Local: "identifier-with-a-long-name"}, "a")
+ ix := indexDecls(len(types), func(i int) string { return types[i].element })
if n := testing.AllocsPerRun(100, func() {
- normalizeAttTokens(tok, types)
- applyAttTypes(el, types)
+ normalizeAttTokens(tok, types, ix)
+ applyAttTypes(el, types, ix)
}); n != 0 {
t.Errorf("typing an element against 40 declarations allocated %.0f times, want 0", n)
}
diff --git a/xdm/dtd_defaults.go b/xdm/dtd_defaults.go
index 722c328f..e46df8c9 100644
--- a/xdm/dtd_defaults.go
+++ b/xdm/dtd_defaults.go
@@ -34,6 +34,49 @@ type attDefault struct {
entities map[string]int
}
+// declIndex maps the element name an ATTLIST declaration was written with to
+// the positions of its declarations, in order. Every element is matched
+// against the declarations, so scanning them all cost a document of n elements
+// and m declarations n*m.
+type declIndex map[string][]int
+
+func indexDecls(n int, element func(i int) string) declIndex {
+ ix := declIndex{}
+ for i := range n {
+ e := element(i)
+ ix[e] = append(ix[e], i)
+ }
+ return ix
+}
+
+// at returns the positions of the declarations that apply to an element
+// written prefix:local: those naming it as written and those naming its local
+// name alone, in declaration order.
+func (ix declIndex) at(prefix, local string) []int {
+ byLocal := ix[local]
+ if prefix == "" {
+ return byLocal
+ }
+ var buf [64]byte // stays on the stack for a name that fits
+ b := append(append(append(buf[:0], prefix...), ':'), local...)
+ lexical := ix[string(b)]
+ if len(byLocal) == 0 {
+ return lexical
+ }
+ if len(lexical) == 0 {
+ return byLocal
+ }
+ out := make([]int, 0, len(lexical)+len(byLocal))
+ for len(lexical) > 0 && len(byLocal) > 0 {
+ if lexical[0] < byLocal[0] {
+ out, lexical = append(out, lexical[0]), lexical[1:]
+ } else {
+ out, byLocal = append(out, byLocal[0]), byLocal[1:]
+ }
+ }
+ return append(append(out, lexical...), byLocal...)
+}
+
// parseAttListDefaults extracts the defaulted attributes from a DOCTYPE
// internal subset.
//
@@ -193,16 +236,14 @@ func unquote(s string) string {
// that was omitted, it does not override one that was given. Even a #FIXED
// default only constrains what may be written, and validating that constraint
// is DTD validation, which this parser does not do.
-func applyAttDefaults(t xml.StartElement, defs []attDefault) xml.StartElement {
+func applyAttDefaults(t xml.StartElement, defs []attDefault, ix declIndex) xml.StartElement {
var add []xml.Attr
- for _, d := range defs {
- // A DTD is not namespace-aware: an ATTLIST names the element as
- // written, prefix included. Matching the local name alone missed
- // "p:doc", and with it a defaulted xmlns:p the element's own name
- // needs. The local-name match stays, as applyAttTypes has it.
- if d.element != attrLexical(t.Name) && d.element != t.Name.Local {
- continue
- }
+ // A DTD is not namespace-aware: an ATTLIST names the element as
+ // written, prefix included. Matching the local name alone missed
+ // "p:doc", and with it a defaulted xmlns:p the element's own name
+ // needs. The local-name match stays, as applyAttTypes has it.
+ for _, i := range ix.at(t.Name.Space, t.Name.Local) {
+ d := defs[i]
if hasAttrNamed(t.Attr, d.name) {
continue
}
@@ -274,12 +315,10 @@ func lexicalIs(prefix, local, s string) bool {
// Recording the annotation here puts a DTD-validated document on the same
// footing as a schema-validated one, so fn:id finds the same attributes in
// both.
-func applyAttTypes(el *Node, types []attDeclaredType) {
+func applyAttTypes(el *Node, types []attDeclaredType, ix declIndex) {
en := el.Name()
- for _, t := range types {
- if !lexicalIs(en.Prefix, en.Local, t.element) && t.element != en.Local {
- continue
- }
+ for _, i := range ix.at(en.Prefix, en.Local) {
+ t := types[i]
for a := range el.Attrs() {
if an := a.Name(); !lexicalIs(an.Prefix, an.Local, t.attr) && an.Local != t.attr {
continue
diff --git a/xdm/dtd_defaults_test.go b/xdm/dtd_defaults_test.go
index bc58c06e..f32b0748 100644
--- a/xdm/dtd_defaults_test.go
+++ b/xdm/dtd_defaults_test.go
@@ -1,6 +1,11 @@
package xdm
-import "testing"
+import (
+ "fmt"
+ "runtime"
+ "strings"
+ "testing"
+)
// An ATTLIST default is supplied to every matching element, including a
// namespace declaration — "xmlns:p CDATA #FIXED '...'" is how a DTD supplies
@@ -218,3 +223,99 @@ func TestAttListFirstDeclarationBinds(t *testing.T) {
t.Errorf("s/@a = %q, want z1: the first-binds rule is per element", got)
}
}
+
+// A default is declared once and applies to every element. Its value used to
+// be copied into the tree for each element that received it, so 2 MB of input
+// retained 1.9 GB (250 attributes of 8,000 bytes on 1,000 elements). The
+// value is stored once and aliased: here 20 defaults of 4,000 bytes on 200
+// elements need 80 KB, and 16 MB when copied.
+func TestAttListDefaultValuesAreShared(t *testing.T) {
+ const attrs, elems, size = 20, 200, 4000
+ var dtd, doc strings.Builder
+ dtd.WriteString("")
+ for i := 0; i < attrs; i++ {
+ // Half are NMTOKENS, whose value is collapsed per element.
+ typ, val := "CDATA", strings.Repeat("x", size)
+ if i%2 == 1 {
+ typ, val = "NMTOKENS", strings.Repeat("x ", size/2-1)+"xx"
+ }
+ fmt.Fprintf(&dtd, ``, i, typ, val)
+ }
+ doc.WriteString("")
+ for i := 0; i < elems; i++ {
+ doc.WriteString("")
+ }
+ doc.WriteString("")
+ src := doc.String()
+ var tree *Tree
+ var err error
+ var m0, m1 runtime.MemStats
+ runtime.GC()
+ runtime.ReadMemStats(&m0)
+ tree, err = ParseString(src, ParseOptions{AllowDOCTYPE: true})
+ runtime.GC()
+ runtime.ReadMemStats(&m1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := int64(m1.HeapAlloc) - int64(m0.HeapAlloc); got > 4<<20 {
+ t.Errorf("tree retains %d KB, want under 4096 KB", got>>10)
+ }
+ n := 0
+ for r := range tree.Root.Children() {
+ for e := range r.Children() {
+ for a := range e.Attrs() {
+ if len(a.Value()) != size {
+ t.Fatalf("attribute %s has %d bytes, want %d", a.Name().Local, len(a.Value()), size)
+ }
+ n++
+ }
+ }
+ }
+ if n != attrs*elems {
+ t.Errorf("%d defaulted attributes, want %d", n, attrs*elems)
+ }
+ runtime.KeepAlive(tree)
+}
+
+// Each element was matched against every ATTLIST declaration, building the
+// element's lexical name for each: 1.4 MB of declarations and elements took
+// 11.8 s. The declarations are now indexed by element name, so the cost per
+// element does not depend on how many others are declared. Measured: 5 MB
+// for 2,000 declarations and 2,000 prefixed elements, against 188 MB scanning
+// them all.
+func TestAttListDefaultsAreIndexedByElement(t *testing.T) {
+ const n = 2000
+ var dtd, doc strings.Builder
+ for i := 0; i < n; i++ {
+ fmt.Fprintf(&dtd, ``, i, i)
+ }
+ doc.WriteString("`)
+ for i := 0; i < n; i++ {
+ fmt.Fprintf(&doc, ``, i)
+ }
+ doc.WriteString("")
+ var tree *Tree
+ var err error
+ b := allocated(func() { tree, err = ParseString(doc.String(), ParseOptions{AllowDOCTYPE: true}) })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if b > 24<<20 {
+ t.Errorf("parse allocated %d MB, want under 24", b>>20)
+ } else {
+ t.Logf("parse allocated %d KB", b>>10)
+ }
+ i := 0
+ for r := range tree.Root.Children() {
+ for e := range r.Children() {
+ if got := e.Attr("", "d"); got == nil || got.Value() != fmt.Sprintf("v%d", i) {
+ t.Fatalf("element %d: default d = %v", i, got)
+ }
+ i++
+ }
+ }
+ if i != n {
+ t.Errorf("%d elements, want %d", i, n)
+ }
+}
diff --git a/xdm/dtd_external.go b/xdm/dtd_external.go
index 310ceb37..1c3e3afb 100644
--- a/xdm/dtd_external.go
+++ b/xdm/dtd_external.go
@@ -1,6 +1,7 @@
package xdm
import (
+ "bufio"
"bytes"
"fmt"
"io"
@@ -148,7 +149,7 @@ func (t *entityTable) fetchExternal(systemID, publicID, base string) (string, st
// UTF-16 entity included by a UTF-8 document is UTF-16 still. The same
// detection the document entity goes through decodes it, and drops the
// byte order mark, which is not part of the replacement text.
- dec, err := decodeReader(bytes.NewReader(data), readWindow(len(data)))
+ dec, err := decodeReader(bufio.NewReaderSize(bytes.NewReader(data), readWindow(len(data))))
if err == nil {
data, err = io.ReadAll(dec)
}
diff --git a/xdm/encoding.go b/xdm/encoding.go
index 4faa6635..ab7d3a67 100644
--- a/xdm/encoding.go
+++ b/xdm/encoding.go
@@ -29,8 +29,7 @@ import (
// The BOM is consumed when present. A UTF-8 BOM is also removed: it is legal
// and carries no information, but encoding/xml treats it as content and a
// document beginning with one would fail to parse.
-func decodeReader(r io.Reader, window int) (io.Reader, error) {
- br := bufio.NewReaderSize(r, window)
+func decodeReader(br *bufio.Reader) (io.Reader, error) {
prefix, err := br.Peek(4)
if err != nil && err != io.EOF && !errIsShort(err) {
return nil, err
diff --git a/xdm/index_test.go b/xdm/index_test.go
new file mode 100644
index 00000000..40724f2d
--- /dev/null
+++ b/xdm/index_test.go
@@ -0,0 +1,110 @@
+package xdm
+
+import (
+ "bytes"
+ "os"
+ "path/filepath"
+ "sort"
+ "strings"
+ "testing"
+)
+
+// checkIndex walks every node of t, attributes included, in document order and
+// checks that Index counts 0, 1, 2, ... with no gap, that NodeAt inverts it,
+// that a subtree's SubtreeEnd is one past its last index, and that namespace
+// nodes have no index.
+func checkIndex(t *testing.T, tr *Tree) {
+ t.Helper()
+ next := 0
+ var walk func(n *Node)
+ walk = func(n *Node) {
+ if got := n.Index(); got != next {
+ t.Fatalf("%v %v: Index %d, want %d", n.Kind(), n.Name(), got, next)
+ }
+ if tr.NodeAt(next) != n {
+ t.Fatalf("NodeAt(%d) is not the node whose Index is %d", next, next)
+ }
+ next++
+ for ns := range n.NamespaceNodes() {
+ if ns.Index() != -1 || ns.SubtreeEnd() != -1 {
+ t.Fatalf("namespace node has Index %d", ns.Index())
+ }
+ }
+ for a := range n.Attrs() {
+ if a.Index() != next || a.SubtreeEnd() != next+1 {
+ t.Fatalf("attribute %v: Index %d End %d, want %d", a.Name(), a.Index(), a.SubtreeEnd(), next)
+ }
+ if tr.NodeAt(next) != a {
+ t.Fatalf("NodeAt(%d) is not the attribute", next)
+ }
+ next++
+ }
+ for c := range n.Children() {
+ walk(c)
+ }
+ if n.SubtreeEnd() != next {
+ t.Fatalf("%v: SubtreeEnd %d, want %d", n.Name(), n.SubtreeEnd(), next)
+ }
+ }
+ walk(tr.Root)
+ if tr.Len() != next {
+ t.Fatalf("Len %d, want %d", tr.Len(), next)
+ }
+ if tr.NodeAt(-1) != nil || tr.NodeAt(next) != nil || tr.NodeAt(next+1<<20) != nil {
+ t.Fatal("NodeAt out of range is not nil")
+ }
+}
+
+func TestNodeIndexDense(t *testing.T) {
+ var ns strings.Builder
+ ns.WriteString(``)
+ for i := 0; i < 50; i++ {
+ ns.WriteString(`t`)
+ }
+ ns.WriteString(``)
+ docs := map[string][]byte{"namespaces": []byte(ns.String())}
+ for name, glob := range map[string]string{
+ "docbook": "xsltng/src/test/resources/xml/*.xml",
+ "xmark": "bench/xmark/auction-0.01.xml",
+ } {
+ paths, _ := filepath.Glob(filepath.Join("..", "testdata", filepath.FromSlash(glob)))
+ sort.Strings(paths)
+ if len(paths) > 20 {
+ paths = paths[:20]
+ }
+ for _, p := range paths {
+ if data, err := os.ReadFile(p); err == nil {
+ docs[name+":"+filepath.Base(p)] = data
+ }
+ }
+ }
+ for name, data := range docs {
+ tr, err := Parse(bytes.NewReader(data), ParseOptions{MaxBytes: -1})
+ if err != nil {
+ continue // a corpus file that needs more than defaults
+ }
+ t.Run(name, func(t *testing.T) { checkIndex(t, tr) })
+ }
+}
+
+// A tree still being built numbers what it has so far, and taking back the
+// last subtree frees its indices.
+func TestNodeIndexBuilt(t *testing.T) {
+ root := NewNode(KindElement, QName{Local: "r"}, "")
+ a := root.AppendElement(QName{Local: "a"})
+ if root.Index() != 0 || a.Index() != 1 || root.SubtreeEnd() != 2 {
+ t.Fatalf("index %d %d end %d", root.Index(), a.Index(), root.SubtreeEnd())
+ }
+ tr := root.tree
+ if tr.NodeAt(1) != a || tr.NodeAt(2) != nil {
+ t.Fatal("NodeAt on a built tree")
+ }
+ b := root.AppendElement(QName{Local: "b"})
+ if b.Index() != 2 || root.SubtreeEnd() != 3 || tr.Len() != 3 {
+ t.Fatalf("after append: %d end %d len %d", b.Index(), root.SubtreeEnd(), tr.Len())
+ }
+ root.RemoveLastChild()
+ if tr.Len() != 2 || tr.NodeAt(2) != nil {
+ t.Fatalf("after take-back: len %d", tr.Len())
+ }
+}
diff --git a/xdm/integers_test.go b/xdm/integers_test.go
new file mode 100644
index 00000000..0708ae04
--- /dev/null
+++ b/xdm/integers_test.go
@@ -0,0 +1,33 @@
+package xdm
+
+import (
+ "math"
+ "testing"
+)
+
+// AppendIntegers gives the values NewInteger gives, including across the
+// shared small-integer range, negatives and the int64 edges, in far fewer
+// allocations.
+func TestAppendIntegers(t *testing.T) {
+ for _, lo := range []int64{-300, -17, 0, 250, 1000, math.MaxInt64 - 299, math.MinInt64} {
+ const n = 300
+ got := AppendIntegers(nil, lo, n)
+ if len(got) != n {
+ t.Fatalf("lo=%d: %d items", lo, len(got))
+ }
+ for i, it := range got {
+ want := NewInteger(lo + int64(i))
+ a := it.(*Atomic)
+ if a.Type != TypeInteger || a.String() != want.String() || a.Rat().Cmp(want.Rat()) != 0 || a.Int64() != lo+int64(i) {
+ t.Fatalf("lo=%d item %d: got %v want %v", lo, i, a, want)
+ }
+ }
+ }
+ if got := AppendIntegers(nil, math.MaxInt64, 1); got[0].(*Atomic).Int64() != math.MaxInt64 {
+ t.Error("MaxInt64")
+ }
+ chunked := testing.AllocsPerRun(5, func() { AppendIntegers(nil, 1000, 1280) })
+ if chunked > 100 { // NewInteger per value is 3 x 1280
+ t.Errorf("%v allocations for 1280 integers", chunked)
+ }
+}
diff --git a/xdm/lineend.go b/xdm/lineend.go
index 8a404336..0dd9040a 100644
--- a/xdm/lineend.go
+++ b/xdm/lineend.go
@@ -38,8 +38,9 @@ type lineEndReader struct {
err error
}
-func newLineEndReader(r io.Reader, window int) *lineEndReader {
- return &lineEndReader{src: r, scratch: make([]byte, 0, window)}
+// newLineEndReader reads r through scratch, whose capacity is the window.
+func newLineEndReader(r io.Reader, scratch []byte) *lineEndReader {
+ return &lineEndReader{src: r, scratch: scratch[:0]}
}
func (l *lineEndReader) Read(p []byte) (int, error) {
diff --git a/xdm/lineend_test.go b/xdm/lineend_test.go
index f3d86b7c..e543d843 100644
--- a/xdm/lineend_test.go
+++ b/xdm/lineend_test.go
@@ -20,7 +20,7 @@ func TestLineEndReader(t *testing.T) {
}
for _, tc := range cases {
for chunk := 1; chunk <= len(tc.in); chunk++ {
- r := newLineEndReader(&chunkReader{s: tc.in, n: chunk}, 4096)
+ r := newLineEndReader(&chunkReader{s: tc.in, n: chunk}, make([]byte, 0, 4096))
got, err := io.ReadAll(r)
if err != nil {
t.Fatal(err)
diff --git a/xdm/lineindex_clip_test.go b/xdm/lineindex_clip_test.go
new file mode 100644
index 00000000..ad3d42c4
--- /dev/null
+++ b/xdm/lineindex_clip_test.go
@@ -0,0 +1,30 @@
+package xdm
+
+import (
+ "strings"
+ "testing"
+)
+
+// The line index is built by append, which leaves its slack allocated; a
+// parse must hand back an index with none (TrackPositions on a 100 MB
+// document kept 0.9 MB unused).
+func TestTrackPositionsLineIndexHasNoSlack(t *testing.T) {
+ for _, n := range []int{1000, 5000} { // 5000 lands mid-doubling
+ src := "" + strings.Repeat("\n", n) + ""
+ opts := ParseOptions{TrackPositions: true}
+ fromString, err := ParseString(src, opts)
+ if err != nil {
+ t.Fatal(err)
+ }
+ fromReader, err := Parse(strings.NewReader(src), opts)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, tree := range []*Tree{fromString, fromReader} {
+ l := tree.lineIdx()
+ if len(l.starts) < n || cap(l.starts) != len(l.starts) {
+ t.Errorf("n=%d: line index len %d cap %d", n, len(l.starts), cap(l.starts))
+ }
+ }
+ }
+}
diff --git a/xdm/maparray.go b/xdm/maparray.go
index 4ca2093a..f7fc93ff 100644
--- a/xdm/maparray.go
+++ b/xdm/maparray.go
@@ -434,18 +434,40 @@ func (a *ArrayItem) Members() []Sequence {
return out
}
+// AppendMembers appends the members to dst and returns it. It is Members
+// without the intermediate copy, for a caller that is building a new slice
+// anyway.
+func (a *ArrayItem) AppendMembers(dst []Sequence) []Sequence {
+ return append(dst, a.members...)
+}
+
// Flatten replaces every array in a sequence with its members, recursively,
// which is what array:flatten and the function conversion rules do.
func Flatten(seq Sequence) Sequence {
out := make(Sequence, 0, len(seq))
- for _, it := range seq {
- arr, ok := it.(*ArrayItem)
- if !ok {
- out = append(out, it)
+ // An explicit stack rather than recursion: an array nested a million deep
+ // is a value a query can build, and recursing on it overflows the Go stack.
+ type frame struct {
+ seqs []Sequence // the member sequences still to walk
+ i int // next item within seqs[0]
+ }
+ stack := []frame{{seqs: []Sequence{seq}}}
+ for len(stack) > 0 {
+ f := &stack[len(stack)-1]
+ if len(f.seqs) == 0 {
+ stack = stack[:len(stack)-1]
continue
}
- for _, m := range arr.members {
- out = append(out, Flatten(m)...)
+ if f.i == len(f.seqs[0]) {
+ f.seqs, f.i = f.seqs[1:], 0
+ continue
+ }
+ it := f.seqs[0][f.i]
+ f.i++
+ if arr, ok := it.(*ArrayItem); ok {
+ stack = append(stack, frame{seqs: arr.members})
+ } else {
+ out = append(out, it)
}
}
return out
diff --git a/xdm/node.go b/xdm/node.go
index 4d0384d1..3f0fb209 100644
--- a/xdm/node.go
+++ b/xdm/node.go
@@ -4,7 +4,7 @@ import (
"errors"
"math"
"math/big"
- "sort"
+ "slices"
"strconv"
"strings"
@@ -598,7 +598,7 @@ func (n *Node) FirstElement(uri, local string) *Node {
//
// A caller that caches trees needs it: one parsed without positions cannot
// serve a request that needs them, and the only way to tell is to ask.
-func (t *Tree) HasPositions() bool { return t != nil && t.srcText() != "" }
+func (t *Tree) HasPositions() bool { return t != nil && t.lineIdx() != nil }
// positionAt converts a byte offset into a 1-based line and column.
//
@@ -606,25 +606,19 @@ func (t *Tree) HasPositions() bool { return t != nil && t.srcText() != "" }
// decoder, and a caller pointing an editor at the failure wants the same
// units the decoder used.
func (t *Tree) positionAt(off int) (line, col int, ok bool) {
- s := t.source
- if s == nil || s.src == "" || off < 0 || off > len(s.src) {
+ l := t.lineIdx()
+ if l == nil || off < 0 || off > l.size {
return 0, 0, false
}
- s.lineOnce.Do(func() {
- // Line 1 starts at offset 0; every byte after a newline starts another.
- s.lineStarts = append(s.lineStarts, 0)
- for i := 0; i < len(s.src); i++ {
- if s.src[i] == '\n' {
- s.lineStarts = append(s.lineStarts, i+1)
- }
- }
- })
// The line is the last one starting at or before off.
- i := sort.SearchInts(s.lineStarts, off+1) - 1
+ i, found := slices.BinarySearch(l.starts, int32(off))
+ if !found {
+ i--
+ }
if i < 0 {
return 0, 0, false
}
- return i + 1, off - s.lineStarts[i] + 1, true
+ return i + 1, off - int(l.starts[i]) + 1, true
}
// The six functions below are the name-only face of the type derivation
diff --git a/xdm/nsbindings.go b/xdm/nsbindings.go
new file mode 100644
index 00000000..db4df358
--- /dev/null
+++ b/xdm/nsbindings.go
@@ -0,0 +1,52 @@
+package xdm
+
+// InScopeNamespaceBindings iterates the prefix-to-URI bindings in scope on
+// an element, the same set InScopeNamespaces returns, without making a
+// namespace node for any of them or a map to hold them. Other kinds have
+// none.
+//
+// Each in-scope prefix is yielded once, with the URI the innermost
+// declaration gives it, and the xml prefix is always among them. A prefix
+// undeclared by xmlns:p="" (or a default undeclared by xmlns="") is not
+// yielded. The order is unspecified; use NamespaceNodes for the namespace
+// axis' prefix order.
+//
+// The cost of each binding is a scan of the ones yielded before it, so the
+// iteration suits a scope of the usual size.
+func (n *Node) InScopeNamespaceBindings() func(yield func(prefix, uri string) bool) {
+ return func(yield func(prefix, uri string) bool) {
+ if n.kind != uint8(KindElement) || n.flags&fSide != 0 {
+ return
+ }
+ if !yield("xml", NSXML) {
+ return
+ }
+ var buf [16]string
+ seen := buf[:0] // prefixes already decided: yielded or undeclared
+ for cur := n; cur != nil; cur = cur.Parent() {
+ if cur.kind != uint8(KindElement) || cur.flags&fSide != 0 {
+ continue
+ }
+ f := cur.frame()
+ for i := len(f) - 1; i >= 0; i-- { // the last declaration wins
+ b := f[i]
+ if b.prefix == "xml" || containsString(seen, b.prefix) {
+ continue
+ }
+ seen = append(seen, b.prefix)
+ if b.uri != "" && !yield(b.prefix, b.uri) {
+ return
+ }
+ }
+ }
+ }
+}
+
+func containsString(s []string, x string) bool {
+ for _, v := range s {
+ if v == x {
+ return true
+ }
+ }
+ return false
+}
diff --git a/xdm/nsbindings_test.go b/xdm/nsbindings_test.go
new file mode 100644
index 00000000..ee0182e4
--- /dev/null
+++ b/xdm/nsbindings_test.go
@@ -0,0 +1,80 @@
+package xdm
+
+import (
+ "maps"
+ "runtime"
+ "testing"
+)
+
+func TestInScopeNamespaceBindings(t *testing.T) {
+ tr, err := ParseString(`some text
`, 200)+``, ParseOptions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ s := &tree.text
+ b := s.blocks[s.cur-1]
+ // Rounding to the allocator's size class may leave a little.
+ if slack := cap(b) - len(b); slack > len(b)/8 {
+ t.Errorf("last text block holds %d bytes in %d", len(b), cap(b))
+ }
+}
+
+// The name and frame indexes serve only building; a parsed tree kept them
+// for its life. Measured on the DocBook corpus: 12.03 MB retained for 591
+// trees with them, 11.07 MB without. A tree changed after the parse rebuilds
+// what it needs.
+func TestFrozenTreeDropsBuildIndexes(t *testing.T) {
+ var b strings.Builder
+ b.WriteString("")
+ for i := range 2 * smallNames {
+ fmt.Fprintf(&b, ``, i, i, i)
+ }
+ b.WriteString("")
+ tree, err := ParseString(b.String(), ParseOptions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if tree.nameIx != nil || tree.nameHot != nil || tree.frameOne != nil || tree.frameMany != nil || tree.open != nil {
+ t.Errorf("a parsed tree kept its build indexes")
+ }
+ e := kids(kids(tree.Root)[0])
+ e[0].SetName(QName{Local: "e5"})
+ if e[0].name != e[5].name || tree.nameIx == nil {
+ t.Errorf("renaming to an existing name gave index %d, want %d (index rebuilt: %v)", e[0].name, e[5].name, tree.nameIx != nil)
+ }
+}
diff --git a/xdm/source_copy_test.go b/xdm/source_copy_test.go
index b35bcc73..ac26be58 100644
--- a/xdm/source_copy_test.go
+++ b/xdm/source_copy_test.go
@@ -97,31 +97,41 @@ func TestSourceCopyStillServesItsUsers(t *testing.T) {
})
}
-// TrackPositions keeps the whole source to count lines in. A reader that
-// knows its length lets that copy be allocated once at its final size,
-// rather than grown by doubling to about twice the document in all.
-func TestTrackedCopyIsSizedOnce(t *testing.T) {
- doc := "\n" + bigBody(100000) + "\n"
- parse := func(track bool) func() {
- return func() {
- if _, err := ParseString(doc, ParseOptions{TrackPositions: track, MaxBytes: -1, MaxNodes: -1}); err != nil {
- t.Fatal(err)
- }
+// TrackPositions kept the whole source text, a byte per input byte for the
+// life of the tree, to count lines in. It keeps where the lines start now:
+// four bytes a line. Measured on parse-10mb, a tracked tree went from 50.4 MB
+// to 41.5 MB.
+func TestTrackedTreeKeepsNoSourceText(t *testing.T) {
+ if raceEnabled {
+ t.Skip("heap sizes are not stable under -race")
+ }
+ doc := "\n" + bigBody(20000) + "\n"
+ parse := func(track bool) func() (*Tree, error) {
+ return func() (*Tree, error) {
+ // A reader, not ParseString, whose string the tree may share.
+ return Parse(strings.NewReader(doc), ParseOptions{TrackPositions: track, MaxBytes: -1, MaxNodes: -1})
}
}
- plain := allocated(parse(false))
- tracked := allocated(parse(true))
- extra := int64(tracked) - int64(plain)
- // Besides the copy, tracking keeps each node's source offset: four bytes
- // a record, in a column a parse without positions does not have.
- tr, err := ParseString(doc, ParseOptions{MaxBytes: -1, MaxNodes: -1})
+ plain := retainedPerTree(t, 4, parse(false))
+ tracked := retainedPerTree(t, 4, parse(true))
+ tr, err := parse(true)()
if err != nil {
t.Fatal(err)
}
- column := 4 * int64(tr.n)
- t.Logf("tracked copy cost %d bytes for a %d-byte document (offset column %d)", extra, len(doc), column)
- if extra > int64(len(doc))*5/4+column {
- t.Errorf("TrackPositions allocated %d bytes more than without (document %d bytes), want about one copy",
+ // Tracking keeps each element's source offset, four bytes a record in a
+ // column a parse without positions does not have, and the line starts.
+ column := int64(0)
+ for _, ch := range tr.chunks {
+ column += 4 * int64(cap(ch))
+ }
+ lines := 8 * int64(strings.Count(doc, "\n")+1) // a slice grown by doubling
+ extra := tracked - plain
+ t.Logf("tracking kept %d bytes for a %d-byte document (offset column %d, line starts %d)", extra, len(doc), column, lines)
+ if extra > column+lines+int64(len(doc))/8 {
+ t.Errorf("a tracked tree kept %d bytes more than an untracked one (document %d bytes), want no copy of the text",
extra, len(doc))
}
+ if line, col, ok := kids(kids(tr.Root)[0])[2*20000-1].Position(); !ok || line != 20001 || col != 1 {
+ t.Errorf("last element at line %d col %d (ok=%v), want line 20001 col 1", line, col, ok)
+ }
}
diff --git a/xdm/textgrow_test.go b/xdm/textgrow_test.go
new file mode 100644
index 00000000..c2d1432b
--- /dev/null
+++ b/xdm/textgrow_test.go
@@ -0,0 +1,62 @@
+package xdm
+
+import (
+ "runtime"
+ "strconv"
+ "strings"
+ "sync"
+ "testing"
+)
+
+// Concurrent parses of long text share nothing: each decoder's buffer is its
+// own, and a value handed to a tree is not written again.
+func TestParseLongTextConcurrent(t *testing.T) {
+ var wg sync.WaitGroup
+ for i := 0; i < 8; i++ {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ text := strings.Repeat("QUJD", (1<<20+i*997)/4)
+ tr, err := Parse(strings.NewReader(""+text+""), ParseOptions{})
+ if err != nil {
+ t.Error(err)
+ return
+ }
+ if tr.Root.StringValue() != text {
+ t.Errorf("parse %d: text differs", i)
+ }
+ }()
+ }
+ wg.Wait()
+}
+
+// A start tag of many distinct attributes sizes the structures that grow with
+// them -- the tokenizer's attribute slice, the name table and its index, the
+// duplicate check -- from the tag, not through every step of their growth.
+// 100,000 attributes allocated about 100 bytes of each byte of the tag with
+// the steps; the structures' own floor is about 40.
+func TestParseManyAttributesAllocation(t *testing.T) {
+ var sb strings.Builder
+ sb.WriteString("")
+ doc := sb.String()
+ var a, b runtime.MemStats
+ runtime.GC()
+ runtime.ReadMemStats(&a)
+ tr, err := Parse(strings.NewReader(doc), ParseOptions{})
+ runtime.ReadMemStats(&b)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := tr.Root.ChildElements()[0].NumAttrs(); got != 100000 {
+ t.Fatalf("%d attributes, want 100000", got)
+ }
+ if got := (b.TotalAlloc - a.TotalAlloc) / uint64(len(doc)); got > 60 {
+ t.Errorf("parse allocated %dx the tag's size, want at most 60x", got)
+ }
+}
diff --git a/xdm/wellformed.go b/xdm/wellformed.go
index caaa9ffe..e5ec446a 100644
--- a/xdm/wellformed.go
+++ b/xdm/wellformed.go
@@ -221,7 +221,7 @@ func validateStartElement(t xml.StartElement, ns *nsScope, xml11 bool) error {
case len(seen) < len(small):
seen = append(seen, k)
default:
- many = make(map[expandedName]bool, 2*len(small))
+ many = make(map[expandedName]bool, max(2*len(small), len(t.Attr)))
for _, s := range seen {
many[s] = true
}
@@ -281,7 +281,10 @@ func (s *nsScope) declare(prefix, uri string) {
s.prev = append(s.prev, s.top[prefix]-1)
s.top[prefix] = len(s.prev)
}
- s.b = append(s.b, nsBinding{prefix, uri})
+ // The URI is copied out of the tokeniser's arena: the tree keeps it in
+ // every name and frame it resolves, and one arena string reachable from a
+ // tree keeps its whole block alive.
+ s.b = append(s.b, nsBinding{prefix, strings.Clone(uri)})
s.own++
}
@@ -361,21 +364,25 @@ func requireQName(n xml.Name) error {
// attributes to the raw tag, ahead of validateStartElement. Namespaces in
// XML §3 binds a prefix to the attribute's *normalized* value, so xmlns:b
// declared NMTOKEN and written " urn:x " binds urn:x — and p:a, b:a collide.
-func normalizeAttTokens(t xml.StartElement, types []attDeclaredType) xml.StartElement {
+func normalizeAttTokens(t xml.StartElement, types []attDeclaredType, ix declIndex) xml.StartElement {
var attrs []xml.Attr
- for _, d := range types {
- if !lexicalIs(t.Name.Space, t.Name.Local, d.element) && d.element != t.Name.Local {
- continue
- }
+ for _, i := range ix.at(t.Name.Space, t.Name.Local) {
+ d := types[i]
for i, a := range t.Attr {
if !lexicalIs(a.Name.Space, a.Name.Local, d.attr) && a.Name.Local != d.attr {
continue
}
+ // A value already collapsed is left as it is, so that a default
+ // keeps the storage the text store shares.
+ c := strings.Join(strings.FieldsFunc(a.Value, func(r rune) bool { return r == ' ' }), " ")
+ if c == a.Value {
+ continue
+ }
if attrs == nil {
// Copy: the tokeniser's attribute slice is not ours to write.
attrs = append([]xml.Attr(nil), t.Attr...)
}
- attrs[i].Value = strings.Join(strings.FieldsFunc(a.Value, func(r rune) bool { return r == ' ' }), " ")
+ attrs[i].Value = c
}
}
if attrs != nil {
diff --git a/xdm/wellformed_scope_test.go b/xdm/wellformed_scope_test.go
index 74dddd76..ef82f078 100644
--- a/xdm/wellformed_scope_test.go
+++ b/xdm/wellformed_scope_test.go
@@ -215,8 +215,8 @@ func TestTagScopeDoesNotAllocate(t *testing.T) {
t.Fatal(err)
}
ns.pop(ns.own)
- }); n != 0 {
- t.Errorf("validateStartElement under 16 inherited bindings allocated %.0f times, want 0", n)
+ }); n > 1 { // the one is the declared URI's copy out of the arena (nsScope.declare)
+ t.Errorf("validateStartElement under 16 inherited bindings allocated %.0f times, want 1", n)
}
}
diff --git a/xdm/xinclude.go b/xdm/xinclude.go
index 3107308c..78aa0169 100644
--- a/xdm/xinclude.go
+++ b/xdm/xinclude.go
@@ -113,6 +113,12 @@ func ProcessXInclude(tree *Tree, opts XIncludeOptions) (*Tree, error) {
if base == "" {
base = tree.Root.DocumentURI()
}
+ if tree.frozen && !mentionsXInclude(tree) {
+ // Nothing to replace: the copy processTree would make is the tree.
+ // Only a parsed tree is returned as is; a built one can still be
+ // changed, and the caller is promised a tree separate from it.
+ return tree, nil
+ }
p := &includeProc{opts: opts, budget: &entityBudget{}}
if base != "" {
p.stack = append(p.stack, base)
@@ -161,6 +167,18 @@ type includeProc struct {
done map[*Node]*Node
}
+// mentionsXInclude reports whether t could hold an XInclude-namespace element.
+// Every element name is in the name table, so a table without the namespace
+// proves the tree has no xi:include or xi:fallback to process.
+func mentionsXInclude(t *Tree) bool {
+ for _, q := range t.names {
+ if q.URI == NSXInclude {
+ return true
+ }
+ }
+ return false
+}
+
// processTree builds the included form of t: a new tree with every xi:include
// replaced by what it includes.
func (p *includeProc) processTree(t *Tree, base string, depth int) (*Tree, error) {
@@ -255,18 +273,10 @@ func appendLeafKeep(dst, n *Node) *Node {
// include appends to dst what one xi:include element includes.
//
-// The inclusion is built in a holder first and only appended once it has
-// succeeded: a failure anywhere in it must leave dst as it was, so that the
-// fallback can take its place.
+// An inclusion is appended only once it has succeeded: a failure anywhere in
+// it must leave dst as it was, so that the fallback can take its place.
func (p *includeProc) include(dst, inc *Node, depth int) error {
- holder, err := p.expandInclude(inc, depth)
- if err != nil {
- return err
- }
- for n := range holder.Children() {
- appendSubtreeKeep(dst, n)
- }
- return nil
+ return p.expandInclude(dst, inc, depth)
}
// appendSubtreeKeep appends a deep copy of n, merging text as appendLeafKeep
@@ -287,11 +297,10 @@ func appendSubtreeKeep(dst, n *Node) {
// positions of what it holds survive the move.
func newHolder() *Node { return NewTree().Root }
-// expandInclude computes, in a holder, the replacement for one xi:include
-// element.
-func (p *includeProc) expandInclude(inc *Node, depth int) (*Node, error) {
+// expandInclude appends to dst the replacement for one xi:include element.
+func (p *includeProc) expandInclude(dst, inc *Node, depth int) error {
if depth >= maxIncludeDepth {
- return nil, fatalInclude{fmt.Errorf(
+ return fatalInclude{fmt.Errorf(
"resource limit exceeded: xi:include nesting exceeds %d levels: %w",
maxIncludeDepth, ErrResourceLimit)}
}
@@ -306,20 +315,20 @@ func (p *includeProc) expandInclude(inc *Node, depth int) (*Node, error) {
if c.Name().Local != "fallback" {
// Section 3.1: the content of xi:include is "(fallback?)",
// so any other XInclude-namespace child is a fatal error.
- return nil, fmt.Errorf(
+ return fmt.Errorf(
"xi:%s is not permitted as a child of xi:include", c.Name().Local)
}
seen++
}
}
if seen > 1 {
- return nil, fmt.Errorf("xi:include has more than one xi:fallback")
+ return fmt.Errorf("xi:include has more than one xi:fallback")
}
href := inc.AttrValue("href")
if href != "" {
if err := ValidateXIncludeHref(href); err != nil {
- return nil, err
+ return err
}
}
parse := inc.AttrValue("parse")
@@ -332,7 +341,7 @@ func (p *includeProc) expandInclude(inc *Node, depth int) (*Node, error) {
// something to fall back from: the attribute is the processor's
// instruction, not the resource's condition, so a fallback would be
// answering the wrong question.
- return nil, fmt.Errorf("xi:include parse=%q must be \"xml\" or \"text\"", parse)
+ return fmt.Errorf("xi:include parse=%q must be \"xml\" or \"text\"", parse)
}
xptr := inc.AttrValue("xpointer")
@@ -389,10 +398,17 @@ func (p *includeProc) expandInclude(inc *Node, depth int) (*Node, error) {
// the loop rule does not apply — a text inclusion cannot recurse.
holder, err = p.fetchText(target, base, encoding)
default:
- holder, err = p.fetch(target, base, parse, xptr, encoding, depth)
+ // fetch appends to dst itself, and only once it cannot fail.
+ err = p.fetch(dst, target, base, parse, xptr, encoding, depth)
+ if err == nil {
+ return nil
+ }
}
- if err == nil {
- return holder, nil
+ if holder != nil && err == nil {
+ for n := range holder.Children() {
+ appendSubtreeKeep(dst, n)
+ }
+ return nil
}
// Section 4.3: "If the resource ... cannot be fetched ... the processor
@@ -406,7 +422,7 @@ func (p *includeProc) expandInclude(inc *Node, depth int) (*Node, error) {
// a document could quietly get a different result by being expensive, and
// a fallback chain would become a way to keep asking after being told no.
if fatalIncludeError(err) {
- return nil, err
+ return err
}
if fb := fallbackOf(inc); fb != nil {
// The fallback's own content may itself contain xi:include elements
@@ -415,14 +431,10 @@ func (p *includeProc) expandInclude(inc *Node, depth int) (*Node, error) {
// like any other content. The depth is carried through so that a
// fallback chain cannot be used to sidestep the nesting bound.
fbBase := elementBase(fb)
- holder := newHolder()
- if err := p.copyChildren(holder, fb, fbBase, depth+1); err != nil {
- return nil, err
- }
- return holder, nil
+ return p.copyChildren(dst, fb, fbBase, depth+1)
}
// Section 4.3: "if the fallback element is absent, it is a fatal error."
- return nil, fmt.Errorf("xi:include of %q failed and has no xi:fallback: %w", target, err)
+ return fmt.Errorf("xi:include of %q failed and has no xi:fallback: %w", target, err)
}
// selectLocal resolves an xpointer against the including document, for an
@@ -540,25 +552,32 @@ func copySubtree(dst, n *Node, base string) {
}
}
-// fetch reads one resource and returns, in a holder, the nodes that replace
-// the xi:include element.
-func (p *includeProc) fetch(target, base, parse, xptr, encoding string, depth int) (*Node, error) {
+// fetch reads one resource and appends to dst the nodes that replace the
+// xi:include element. It appends only once nothing can fail any more.
+func (p *includeProc) fetch(dst *Node, target, base, parse, xptr, encoding string, depth int) error {
if parse == "text" {
- return p.fetchText(target, base, encoding)
+ holder, err := p.fetchText(target, base, encoding)
+ if err != nil {
+ return err
+ }
+ for n := range holder.Children() {
+ appendSubtreeKeep(dst, n)
+ }
+ return nil
}
if p.fetches >= maxIncludeFetches {
- return nil, fatalInclude{fmt.Errorf(
+ return fatalInclude{fmt.Errorf(
"resource limit exceeded: document performs more than %d inclusions: %w",
maxIncludeFetches, ErrResourceLimit)}
}
if p.opts.Resolver == nil {
- return nil, fmt.Errorf("no include resolver: inclusions are not permitted")
+ return fmt.Errorf("no include resolver: inclusions are not permitted")
}
p.fetches++
data, uri, err := p.opts.Resolver.ResolveInclude(target, base, encoding)
if err != nil {
- return nil, err
+ return err
}
// A cycle is checked on the URI the resolver reports rather than on the
@@ -567,7 +586,7 @@ func (p *includeProc) fetch(target, base, parse, xptr, encoding string, depth in
// without this the recursion below simply does not terminate.
for _, u := range p.stack {
if u == uri {
- return nil, fatalInclude{fmt.Errorf(
+ return fatalInclude{fmt.Errorf(
"circular xi:include loop: %q includes itself", uri)}
}
}
@@ -595,19 +614,22 @@ func (p *includeProc) fetch(target, base, parse, xptr, encoding string, depth in
// entity bomb expanded 6,291,456 bytes — six times the ceiling — and
// the document was accepted, the refusal swallowed at every level.
if errors.Is(err, ErrResourceLimit) {
- return nil, fatalInclude{err}
+ return fatalInclude{err}
}
- return nil, err
+ return err
}
// Process the included document before selecting, so that an xpointer
// may address content that an inner inclusion brought in. The stack
// grows for the duration.
p.stack = append(p.stack, uri)
- done, err := p.processTree(sub, uri, depth+1)
+ done := sub
+ if mentionsXInclude(sub) {
+ done, err = p.processTree(sub, uri, depth+1)
+ }
p.stack = p.stack[:len(p.stack)-1]
if err != nil {
- return nil, err
+ return err
}
var picked []*Node
@@ -624,22 +646,21 @@ func (p *includeProc) fetch(target, base, parse, xptr, encoding string, depth in
} else {
picked, err = selectXPointer(done.Root, xptr)
if err != nil {
- return nil, err
+ return err
}
if len(picked) == 0 {
// Section 4.4: an xpointer that "does not identify any
// resource" is an error, recoverable through fallback — which
// is what returning an error here achieves, since the caller
// consults xi:fallback on any error from this function.
- return nil, fmt.Errorf("xpointer %q selected nothing in %s", xptr, uri)
+ return fmt.Errorf("xpointer %q selected nothing in %s", xptr, uri)
}
}
- holder := newHolder()
for _, n := range picked {
- copyFixedUp(holder, n, base)
+ copyFixedUp(dst, n, base)
}
- return holder, nil
+ return nil
}
// copyFixedUp appends a copy of n, a top-level included node, with the base
diff --git a/xdm/xinclude_nocopy_test.go b/xdm/xinclude_nocopy_test.go
new file mode 100644
index 00000000..07b12134
--- /dev/null
+++ b/xdm/xinclude_nocopy_test.go
@@ -0,0 +1,49 @@
+package xdm
+
+import (
+ "runtime"
+ "strings"
+ "testing"
+)
+
+// A tree with no XInclude element has nothing to process, so neither the
+// including document nor an included one is copied (-xinclude
+// on a 100 MB document peaked 500 MB higher for no inclusion at all).
+func TestXIncludeSkipsCopyWithoutXIElements(t *testing.T) {
+ tree := parseWithBase(t, "mem:/a.xml", "text")
+ out, err := ProcessXInclude(tree, XIncludeOptions{})
+ if err != nil || out != tree {
+ t.Fatalf("plain tree: got %p (tree %p), err %v; want the tree itself", out, tree, err)
+ }
+ // A tree that is not a frozen parse can still be changed, so it gets its
+ // own copy even with nothing to include.
+ built := &Tree{Root: Copy(tree.Root)}
+ if out, err := ProcessXInclude(built, XIncludeOptions{}); err != nil || out == built {
+ t.Fatalf("built tree: got %p (tree %p), err %v; want a new tree", out, built, err)
+ }
+
+ big := "" + strings.Repeat("text", 20000) + ""
+ parseBytes := allocBytes(func() { parseWithBase(t, "mem:/big.xml", big) })
+ doc := parseWithBase(t, "mem:/a.xml", ``)
+ res := &mapResolver{files: map[string]string{"mem:/big.xml": big}}
+ incBytes := allocBytes(func() {
+ if _, err := ProcessXInclude(doc, XIncludeOptions{Resolver: res}); err != nil {
+ t.Fatal(err)
+ }
+ })
+ // Parse plus the one copy into the result is about 2.2 parses; a copy of
+ // the included document before it, or through a holder, adds 1 each.
+ if incBytes > parseBytes*13/5 {
+ t.Errorf("including a plain document allocated %d bytes; parsing it alone is %d", incBytes, parseBytes)
+ }
+ t.Logf("parse %d include %d", parseBytes, incBytes)
+}
+
+func allocBytes(f func()) uint64 {
+ var a, b runtime.MemStats
+ runtime.GC()
+ runtime.ReadMemStats(&a)
+ f()
+ runtime.ReadMemStats(&b)
+ return b.TotalAlloc - a.TotalAlloc
+}
diff --git a/xdmbuild/builder.go b/xdmbuild/builder.go
index b3defda2..3faf4c8e 100644
--- a/xdmbuild/builder.go
+++ b/xdmbuild/builder.go
@@ -777,7 +777,7 @@ func (b *Builder) setDecl(prefix, uri string) {
}
}
if b.declared == nil {
- b.declared = make([]decl, 0, 8)
+ b.declared = make([]decl, 0, 3)
}
b.declared = append(b.declared, decl{Prefix: prefix, URI: uri})
b.declShared = false // a lent list has no spare capacity, so append copied it
diff --git a/xpath/arrayappend_alloc_test.go b/xpath/arrayappend_alloc_test.go
new file mode 100644
index 00000000..b855275c
--- /dev/null
+++ b/xpath/arrayappend_alloc_test.go
@@ -0,0 +1,44 @@
+package xpath
+
+import (
+ "runtime"
+ "testing"
+ "unsafe"
+
+ "github.com/knroy/go-xml/v2/xdm"
+)
+
+// array:append, array:insert-before and array:join must copy the members once,
+// not once in Members() and again into the result.
+func TestArrayBuildersCopyMembersOnce(t *testing.T) {
+ const n, runs = 10000, 20
+ members := make([]xdm.Sequence, n)
+ for i := range members {
+ members[i] = xdm.One(xdm.NewInteger(int64(i)))
+ }
+ arr := xdm.One(xdm.NewArray(members...))
+ oneCopy := uint64(n * unsafe.Sizeof(xdm.Sequence(nil)))
+ const ns = "Q{http://www.w3.org/2005/xpath-functions/array}"
+ for _, q := range []string{
+ ns + "append($a, 1)", ns + "insert-before($a, 5, 1)", ns + "join(($a))",
+ } {
+ c, err := CompileWith(q, CompileOptions{Version: XPath31})
+ if err != nil {
+ t.Fatal(err)
+ }
+ ctx := NewContext(nil, Builtins()).WithVar(xdm.QName{Local: "a"}, arr)
+ var before, after runtime.MemStats
+ runtime.GC()
+ runtime.ReadMemStats(&before)
+ for i := 0; i < runs; i++ {
+ if _, err := c.Eval(ctx); err != nil {
+ t.Fatal(err)
+ }
+ }
+ runtime.ReadMemStats(&after)
+ per := (after.TotalAlloc - before.TotalAlloc) / runs
+ if per > oneCopy*3/2 {
+ t.Errorf("%s allocated %d bytes per call; one copy of the members is %d", q, per, oneCopy)
+ }
+ }
+}
diff --git a/xpath/ast.go b/xpath/ast.go
index 38af2182..f874db81 100644
--- a/xpath/ast.go
+++ b/xpath/ast.go
@@ -315,6 +315,16 @@ type InlineFunctionExpr struct {
// Result is the declared return type, or nil if none was written.
Result *SequenceType
Body Expr
+
+ // needs is what closureNeeds found, computed on first evaluation.
+ needs atomic.Pointer[closureVars]
+}
+
+// closureVars is the variables a closure's body reads, or all when it must
+// capture every one.
+type closureVars struct {
+ names []xdm.QName
+ all bool
}
// InlineParam is one parameter of an inline function.
diff --git a/xpath/builtins.go b/xpath/builtins.go
index 0992399d..a4713186 100644
--- a/xpath/builtins.go
+++ b/xpath/builtins.go
@@ -197,7 +197,7 @@ func registerConstructors(l *Library) {
for name, code := range types {
target := code
- l.register(xdm.NSXS, name, 1, func(_ *Context, args []xdm.Sequence) (xdm.Sequence, error) {
+ l.register(xdm.NSXS, name, 1, func(ctx *Context, args []xdm.Sequence) (xdm.Sequence, error) {
atoms := xdm.Atomize(args[0])
// A constructor applied to the empty sequence yields the empty
// sequence rather than raising, so "xs:date($optional)" is safe.
@@ -208,6 +208,9 @@ func registerConstructors(l *Library) {
if err != nil {
return nil, err
}
+ if err := ctx.checkCastDigits(it.(*xdm.Atomic), target); err != nil {
+ return nil, err
+ }
out, err := CastAtomic(it.(*xdm.Atomic), target)
if err != nil {
return nil, err
diff --git a/xpath/cache.go b/xpath/cache.go
index 588a40f3..639b43aa 100644
--- a/xpath/cache.go
+++ b/xpath/cache.go
@@ -31,6 +31,12 @@ import "sync"
// contended case that would justify an RWMutex is a long read, which this is
// not. Mutex is also the smaller and simpler primitive, which matters for a
// type instantiated three times over three different value shapes.
+// maxCachedKey is the longest key worth caching. The caches bound entries, not
+// bytes, and the key (a pattern, a collation URI) can come from document data:
+// 1024 entries of long keys, each with a compiled program behind it, retained
+// gigabytes. A key this long is simply recompiled on each use.
+const maxCachedKey = 1024
+
type boundedCache[K comparable, V any] struct {
mu sync.Mutex
items map[K]V
diff --git a/xpath/clearhostvars_test.go b/xpath/clearhostvars_test.go
new file mode 100644
index 00000000..39729c54
--- /dev/null
+++ b/xpath/clearhostvars_test.go
@@ -0,0 +1,36 @@
+package xpath
+
+import (
+ "testing"
+
+ "github.com/knroy/go-xml/v2/xdm"
+)
+
+// TestClearHostVarsOneScope: the host's cleared and marked names are bound in
+// one scope each, not one scope per name, and still resolve as before.
+func TestClearHostVarsOneScope(t *testing.T) {
+ oc, om := ClearedOnDynamicCall, MarkedOnDynamicCall
+ defer func() { ClearedOnDynamicCall, MarkedOnDynamicCall = oc, om }()
+ var cleared []xdm.QName
+ for _, n := range []string{"a", "b", "c", "d", "e", "f"} {
+ cleared = append(cleared, xdm.QName{URI: "urn:h", Local: n})
+ }
+ ClearedOnDynamicCall = cleared
+ MarkedOnDynamicCall = []xdm.QName{{URI: "urn:h", Local: "m"}}
+
+ base := NewContext(nil, Builtins()).WithVar(xdm.QName{URI: "urn:h", Local: "c"}, xdm.One(xdm.NewInteger(7)))
+ got := clearHostVars(base)
+ if v, ok := got.LookupVar(cleared[2]); !ok || len(v) != 0 {
+ t.Fatalf("cleared name resolves to %v, %v; want bound and empty", v, ok)
+ }
+ if v, ok := got.LookupVar(MarkedOnDynamicCall[0]); !ok || len(v) != 1 {
+ t.Fatalf("marked name resolves to %v, %v; want one true", v, ok)
+ }
+ depth := 0
+ for s := got; s.bind != base.bind; s = s.Parent {
+ depth++
+ }
+ if depth != 2 {
+ t.Fatalf("clearHostVars made %d scopes, want 2", depth)
+ }
+}
diff --git a/xpath/closure_capture_test.go b/xpath/closure_capture_test.go
new file mode 100644
index 00000000..127e2f31
--- /dev/null
+++ b/xpath/closure_capture_test.go
@@ -0,0 +1,41 @@
+package xpath
+
+import (
+ "runtime"
+ "testing"
+
+ "github.com/knroy/go-xml/v2/xdm"
+)
+
+// TestInlineFunctionCapturesOnlyWhatItReads: a closure whose body does not
+// mention $big must not keep it alive, and one that does must still see it.
+func TestInlineFunctionCapturesOnlyWhatItReads(t *testing.T) {
+ heldAfterGC := func(src string) (uint64, xdm.Sequence) {
+ c, err := CompileWith(src, CompileOptions{Version: XPath31})
+ if err != nil {
+ t.Fatal(err)
+ }
+ out, err := c.Eval(NewContext(nil, Builtins()))
+ if err != nil {
+ t.Fatal(err)
+ }
+ runtime.GC()
+ var m runtime.MemStats
+ runtime.ReadMemStats(&m)
+ return m.HeapAlloc, out
+ }
+ var before runtime.MemStats
+ runtime.GC()
+ runtime.ReadMemStats(&before)
+ heldIgnoring, f1 := heldAfterGC(`let $big := (1 to 300000) ! string(.) return function() { 1 }`)
+ if grown := int64(heldIgnoring) - int64(before.HeapAlloc); grown > 2<<20 {
+ t.Fatalf("closure ignoring $big retains %d bytes", grown)
+ }
+ runtime.KeepAlive(f1)
+ _, f2 := heldAfterGC(`let $big := (1 to 3) ! string(.), $k := 2 return function($i) { $big[$i] || $k }`)
+ call := f2[0].(*xdm.FunctionItem)
+ got, err := call.Invoke(NewContext(nil, Builtins()), []xdm.Sequence{xdm.One(xdm.NewInteger(2))})
+ if err != nil || renderSeq(got) != "22" {
+ t.Fatalf("closure over $big and $k gave %v, %v", got, err)
+ }
+}
diff --git a/xpath/collation.go b/xpath/collation.go
index 1fcaea97..a2f4e8b4 100644
--- a/xpath/collation.go
+++ b/xpath/collation.go
@@ -820,7 +820,9 @@ func ucaCollationFor(uri string) (Collation, error) {
return e.c, e.err
}
c, err := parseUCACollation(uri)
- ucaCache.Put(uri, ucaCacheEntry{c, err})
+ if len(uri) <= maxCachedKey {
+ ucaCache.Put(uri, ucaCacheEntry{c, err})
+ }
return c, err
}
diff --git a/xpath/context.go b/xpath/context.go
index 38a813d1..38813dc5 100644
--- a/xpath/context.go
+++ b/xpath/context.go
@@ -253,6 +253,13 @@ type Env struct {
// AdoptBudget copies it along with the shared counter, so a nested
// evaluation cannot raise the bound it is charged against.
MaxItems int
+ // MaxDigits bounds the decimal digits of an integer or decimal that a
+ // cast reads from text or an arithmetic operator produces. Zero means
+ // MaxDigits' package default, 10^6; a negative value means no bound. A
+ // cast over it is FOCA0003 and a result over it FOAR0002, both carrying
+ // xdm.ErrResourceLimit: parsing and multiplying are quadratic in the
+ // digit count, so an unbounded value is a time and memory attack.
+ MaxDigits int
// items counts the items materialised into intermediate sequences during
// this evaluation, bounding memory the way Depth bounds stack.
//
@@ -922,6 +929,35 @@ func (c *Context) WithVar(name xdm.QName, val xdm.Sequence) *Context {
return n
}
+// withVars binds every name to val in one scope. The names have non-empty
+// local parts; it is what the host registries of a dynamic call are bound with.
+func (c *Context) withVars(names []xdm.QName, val xdm.Sequence) *Context {
+ if len(names) == 0 {
+ return c
+ }
+ s := &struct {
+ n Context
+ b varBinding
+ }{n: *c}
+ s.n.Vars = nil
+ s.b = varBinding{names: names, val: val}
+ s.n.bind = &s.b
+ s.n.Parent = c
+ return &s.n
+}
+
+// WithVars is WithVar for several variables at once: one scope binds names[i]
+// to vals[i]. The names must be distinct, non-empty and as long as vals, and
+// neither slice may be written while the scope is in use.
+func (c *Context) WithVars(names []xdm.QName, vals []xdm.Sequence) *Context {
+ if len(names) == 0 {
+ return c
+ }
+ n := c.withVars(names, nil)
+ n.bind.vals = vals
+ return n
+}
+
// varBinding is the binding a scope made by WithVar adds. Never written once
// the scope is published. A scope made by WithLazyVars holds lazy instead,
// and binds every name in it; one made by xpathleaf.WithLocals holds locals,
@@ -929,6 +965,8 @@ func (c *Context) WithVar(name xdm.QName, val xdm.Sequence) *Context {
type varBinding struct {
uri, local string
val xdm.Sequence
+ names []xdm.QName // withVars: every name is bound, to vals[i] or else to val
+ vals []xdm.Sequence
lazy map[varKey]*LazyVar
locals *xpathleaf.Locals
}
@@ -954,6 +992,38 @@ func frozenLocals(c *Context) *Context {
return &n
}
+// capturedFor is frozenLocals for a closure that reads only the variables keep
+// accepts: a scope that binds one name and is not accepted is left out of the
+// chain, and a host stack of locals is cut down to the accepted names. Scopes
+// binding several names, lazily or by the Vars map stay whole.
+func capturedFor(c *Context, keep func(uri, local string) bool, leaf bool) *Context {
+ if c == nil {
+ return nil
+ }
+ if b := c.bind; b != nil && b.locals != nil {
+ n := *c
+ n.bind = &varBinding{locals: b.locals.FrozenOnly(keep)}
+ return &n
+ }
+ p := capturedFor(c.Parent, keep, false)
+ if b := c.bind; b != nil && b.names == nil && b.lazy == nil && b.local != "" &&
+ len(c.Vars) == 0 && !keep(b.uri, b.local) {
+ if !leaf {
+ return p
+ }
+ // The innermost scope also carries the environment the body runs in.
+ n := *c
+ n.bind, n.Parent = nil, p
+ return &n
+ }
+ if p == c.Parent {
+ return c
+ }
+ n := *c
+ n.Parent = p
+ return &n
+}
+
// varKey is a variable's expanded name as a map key that needs no Clark
// string built for a lookup.
type varKey struct{ uri, local string }
@@ -1240,6 +1310,15 @@ func (c *Context) lookupVarPlain(name xdm.QName) (xdm.Sequence, bool, error) {
return v, true, err
}
}
+ } else if b.names != nil {
+ for i, q := range b.names {
+ if q.Local == name.Local && q.URI == name.URI {
+ if b.vals != nil {
+ return b.vals[i], true, nil
+ }
+ return b.val, true, nil
+ }
+ }
} else if b.local == name.Local && b.uri == name.URI {
return b.val, true, nil
}
@@ -1322,6 +1401,18 @@ func (c *Context) checkDepth() error {
return nil
}
+// checkValueDepth bounds the nesting of a map or array value that deep-equal
+// or map:find descends into at maxJSONDepth, like parse-json. The depth is
+// passed down by the walkers rather than kept in the Context, whose size is
+// pinned, and so is counted apart from function recursion.
+func checkValueDepth(depth int) error {
+ if depth >= maxJSONDepth {
+ return fmt.Errorf("XPDY0001: value nests more than %d levels: %w",
+ maxJSONDepth, xdm.ErrResourceLimit)
+ }
+ return nil
+}
+
// countItems charges n items against the evaluation budget.
//
// Accumulating constructs call it as they build, so a runaway is stopped while
@@ -1333,13 +1424,86 @@ func (c *Context) countItems(n int) error {
return nil
}
if lim := c.itemLimit(); atomic.AddInt64(c.ev().items, int64(n)) > lim {
- // The code is kept -- the suites and callers read it -- and the
- // sentinel added, because this is the processor declining to
- // allocate rather than anything wrong with the expression.
- return fmt.Errorf(
- "XPDY0130: evaluation materialised more than %d items; "+
- "the expression is building a sequence too large to hold: %w",
- lim, xdm.ErrResourceLimit)
+ return itemsRefusal(lim)
+ }
+ return nil
+}
+
+// itemsRefusal is the XPDY0130 the item budget raises.
+func itemsRefusal(lim int64) error {
+ // The code is kept -- the suites and callers read it -- and the
+ // sentinel added, because this is the processor declining to
+ // allocate rather than anything wrong with the expression.
+ return fmt.Errorf(
+ "XPDY0130: evaluation materialised more than %d items; "+
+ "the expression is building a sequence too large to hold: %w",
+ lim, xdm.ErrResourceLimit)
+}
+
+// ItemsRefused reports the item-budget refusal if total exceeds the item bound
+// in force, and charges nothing.
+//
+// A host that retains items across many expression evaluations -- xsl:key's
+// index, for-each-group's groups -- keeps the running total itself, because
+// Compiled.Eval resets the shared counter for every expression and a charge
+// made through ChargeItems would be cleared by the next node's evaluation.
+func (c *Context) ItemsRefused(total int64) error {
+ if lim := c.itemLimit(); total > lim {
+ return itemsRefusal(lim)
+ }
+ return nil
+}
+
+// MaxDigits is the digit bound in force where Env.MaxDigits is zero.
+const MaxDigits = 1000000
+
+// digitLimit is the digit bound in force, or -1 for none.
+func (c *Context) digitLimit() int {
+ switch {
+ case c == nil || c.ev().MaxDigits == 0:
+ return MaxDigits
+ case c.ev().MaxDigits < 0:
+ return -1
+ }
+ return c.ev().MaxDigits
+}
+
+// checkCastDigits refuses a string cast to xs:integer or xs:decimal whose text
+// is longer than the digit bound, before it is parsed.
+func (c *Context) checkCastDigits(a *xdm.Atomic, target xdm.TypeCode) error {
+ if target != xdm.TypeInteger && target != xdm.TypeDecimal ||
+ a.Type != xdm.TypeString && a.Type != xdm.TypeUntypedAtomic {
+ return nil
+ }
+ if lim := c.digitLimit(); lim >= 0 && len(a.Str()) > lim {
+ return &xdm.Error{
+ Code: "FOCA0003",
+ Message: fmt.Sprintf("a number of more than %d digits: %s",
+ lim, xdm.ErrResourceLimit),
+ Err: xdm.ErrResourceLimit,
+ }
+ }
+ return nil
+}
+
+// checkResultDigits refuses an exact arithmetic result of more than the digit
+// bound. A value held in an int64 has no big.Rat and is always small enough.
+func (c *Context) checkResultDigits(a *xdm.Atomic) error {
+ r := a.Rat()
+ if r == nil {
+ return nil
+ }
+ lim := c.digitLimit()
+ if lim < 0 {
+ return nil
+ }
+ if bits := max(r.Num().BitLen(), r.Denom().BitLen()); bits*30103/100000 > lim {
+ return &xdm.Error{
+ Code: "FOAR0002",
+ Message: fmt.Sprintf("a result of more than %d digits: %s",
+ lim, xdm.ErrResourceLimit),
+ Err: xdm.ErrResourceLimit,
+ }
}
return nil
}
@@ -1499,21 +1663,38 @@ func (c *Context) countNodes(n int) error {
return nil
}
if atomic.AddInt64(c.ev().nodes, int64(n)) > MaxNodes {
- // XPDY0130 again, and the wording says nodes so that the three
- // refusals are not confused with one another. XPath 3.1 §2.3.1
- // sanctions exactly this: "limitations may exist on the maximum
- // numbers or sizes of various objects... An error must be raised if
- // such a limitation is exceeded [err:XPDY0130]". The sentinel is
- // added alongside so a caller can tell a refusal from a fault. See
- // xdm.ErrResourceLimit.
- return fmt.Errorf(
- "XPDY0130: evaluation constructed more than %d result-tree "+
- "nodes; the transformation is building a tree too large to "+
- "hold: %w", MaxNodes, xdm.ErrResourceLimit)
+ return nodesRefusal()
}
return nil
}
+// nodesRefusal is the XPDY0130 the node budget raises.
+func nodesRefusal() error {
+ // XPDY0130 again, and the wording says nodes so that the three
+ // refusals are not confused with one another. XPath 3.1 §2.3.1
+ // sanctions exactly this: "limitations may exist on the maximum
+ // numbers or sizes of various objects... An error must be raised if
+ // such a limitation is exceeded [err:XPDY0130]". The sentinel is
+ // added alongside so a caller can tell a refusal from a fault. See
+ // xdm.ErrResourceLimit.
+ return fmt.Errorf(
+ "XPDY0130: evaluation constructed more than %d result-tree "+
+ "nodes; the transformation is building a tree too large to "+
+ "hold: %w", MaxNodes, xdm.ErrResourceLimit)
+}
+
+// NodesRefused reports the node-budget refusal if any charge has already
+// crossed MaxNodes, and charges nothing. A host calls it where a transform
+// finishes: the builders latch a refusal per builder, so a built-in rule or a
+// temporary tree that crossed the bound would otherwise end as truncated
+// output with no error.
+func (c *Context) NodesRefused() error {
+ if c == nil || c.ev().nodes == nil || atomic.LoadInt64(c.ev().nodes) <= MaxNodes {
+ return nil
+ }
+ return nodesRefusal()
+}
+
// ChargeNodes charges n constructed result-tree nodes against the evaluation
// budget, reporting XPDY0130 when the budget is exhausted.
//
diff --git a/xpath/copycharge_test.go b/xpath/copycharge_test.go
new file mode 100644
index 00000000..85b16142
--- /dev/null
+++ b/xpath/copycharge_test.go
@@ -0,0 +1,32 @@
+package xpath
+
+import (
+ "strings"
+ "testing"
+)
+
+// A sequence constructor or a "!" that copies its operand into a larger one
+// is charged for the copies, so doubling it in a fold hits the item budget.
+// The 23 doublings build 16.7M items against the 5M limit.
+func TestSequenceCopiesAreCharged(t *testing.T) {
+ for _, q := range []string{
+ `fold-left(1 to 23, (1,2), function($s,$i){($s,$s)})`,
+ `fold-left(1 to 23, (1,2), function($s,$i){$s ! (.,.)})`,
+ } {
+ c, err := CompileWith(q, CompileOptions{Version: XPath31})
+ if err != nil {
+ t.Fatal(err)
+ }
+ _, err = c.Eval(NewContext(nil, Builtins()))
+ if err == nil || !strings.Contains(err.Error(), "XPDY0130") {
+ t.Errorf("%s: got %v, want XPDY0130", q, err)
+ }
+ }
+ // A legal modest sequence still evaluates, with and without the floor.
+ for _, q := range []string{`count((1 to 1000, 1 to 1000))`, `count(((1,2),(3,4)))`, `count((1 to 1000) ! (., .))`} {
+ c, _ := CompileWith(q, CompileOptions{Version: XPath31})
+ if _, err := c.Eval(NewContext(nil, Builtins())); err != nil {
+ t.Errorf("%s: %v", q, err)
+ }
+ }
+}
diff --git a/xpath/deepnest_test.go b/xpath/deepnest_test.go
new file mode 100644
index 00000000..6fcc5f59
--- /dev/null
+++ b/xpath/deepnest_test.go
@@ -0,0 +1,53 @@
+package xpath
+
+import (
+ "errors"
+ "testing"
+
+ "github.com/knroy/go-xml/v2/xdm"
+)
+
+// A value nested far past any real document must be refused with the
+// resource-limit error, not recurse until the Go stack overflows (fatal, and
+// not recoverable). The depth is just past the smallest bound, so a reverted
+// fix fails on the assertion rather than crashing the test binary.
+func TestDeepNestedValuesAreRefused(t *testing.T) {
+ const depth = 20000
+ nest := func(wrap func(xdm.Sequence) xdm.Item) xdm.Sequence {
+ v := xdm.One(xdm.NewInteger(1))
+ for i := 0; i < depth; i++ {
+ v = xdm.One(wrap(v))
+ }
+ return v
+ }
+ arrays := nest(func(s xdm.Sequence) xdm.Item { return xdm.NewArray(s) })
+ maps := nest(func(s xdm.Sequence) xdm.Item {
+ m := xdm.NewMap()
+ m, _ = m.Put(xdm.NewString("k"), s)
+ return m
+ })
+ for _, tc := range []struct{ name, query string }{
+ {"json array", `serialize($a, map{"method":"json"})`},
+ {"json map", `serialize($m, map{"method":"json"})`},
+ {"adaptive array", `serialize($a, map{"method":"adaptive"})`},
+ {"adaptive map", `serialize($m, map{"method":"adaptive"})`},
+ {"deep-equal array", `deep-equal($a, $a)`},
+ {"deep-equal map", `deep-equal($m, $m)`},
+ {"map:find", `Q{http://www.w3.org/2005/xpath-functions/map}find($m, "nope")`},
+ } {
+ c, err := CompileWith(tc.query, CompileOptions{Version: XPath31})
+ if err != nil {
+ t.Fatal(err)
+ }
+ ctx := NewContext(nil, Builtins())
+ ctx = ctx.WithVar(xdm.QName{Local: "a"}, arrays).WithVar(xdm.QName{Local: "m"}, maps)
+ if _, err := c.Eval(ctx); err == nil || !errors.Is(err, xdm.ErrResourceLimit) {
+ t.Errorf("%s: got %v, want an ErrResourceLimit refusal", tc.name, err)
+ }
+ }
+
+ // Flatten is iterative: the same value flattens to its one leaf.
+ if got := xdm.Flatten(arrays); len(got) != 1 {
+ t.Errorf("Flatten of %d nested arrays: %d items, want 1", depth, len(got))
+ }
+}
diff --git a/xpath/eval.go b/xpath/eval.go
index 1997eaf4..85c9d797 100644
--- a/xpath/eval.go
+++ b/xpath/eval.go
@@ -42,16 +42,36 @@ func (e *ContextItem) Eval(ctx *Context) (xdm.Sequence, error) {
// Eval implements Expr for a sequence constructor.
func (e *SequenceExpr) Eval(ctx *Context) (xdm.Sequence, error) {
var out xdm.Sequence
+ widest := 0
for _, it := range e.Items {
v, err := it.Eval(ctx)
if err != nil {
return nil, err
}
+ widest = max(widest, len(v))
out = append(out, v...)
}
+ // The widest operand was charged by whatever built it; the rest is
+ // copies, which ($s, $s) in a loop doubles without any other charge.
+ if err := ctx.chargeCopies(len(out) - widest); err != nil {
+ return nil, err
+ }
return out, nil
}
+// chargeCopyFloor is the size below which chargeCopies charges nothing, so
+// that ordinary small sequences cost no atomic add.
+const chargeCopyFloor = 64
+
+// chargeCopies charges n items a constructor copied beyond those the
+// expressions that built them already charged.
+func (c *Context) chargeCopies(n int) error {
+ if n < chargeCopyFloor {
+ return nil
+ }
+ return c.countItems(n)
+}
+
// Eval implements Expr for a single axis step.
//
// A step is evaluated against the context item alone; iterating a step over
@@ -74,22 +94,7 @@ func (e *Step) Eval(ctx *Context) (xdm.Sequence, error) {
// nothing would read.
func (e *Step) evalFrom(ctx *Context, node *xdm.Node) (xdm.Sequence, error) {
var err error
- principal := e.Axis.PrincipalKind()
- var selected xdm.Sequence
- if nt, ok := e.Test.(*NameTest); ok && (e.Axis == AxisDescendant || e.Axis == AxisDescendantOrSelf) {
- // The direct walk selects what walkAxis would, in the same order.
- if e.Axis == AxisDescendantOrSelf && nt.Matches(node, principal) {
- selected = append(selected, node)
- }
- selected = appendNamedDescendants(selected, node, nt)
- } else {
- walkAxis(node, e.Axis, func(n *xdm.Node) bool {
- if e.Test.Matches(n, principal) {
- selected = append(selected, n)
- }
- return true
- })
- }
+ selected := e.appendFrom(nil, node)
// Predicates apply in axis order, so a reverse axis numbers positions
// from the context node outwards.
@@ -105,6 +110,32 @@ func (e *Step) evalFrom(ctx *Context, node *xdm.Node) (xdm.Sequence, error) {
return xdm.SortDocumentOrder(selected), nil
}
+// appendFrom appends to selected the nodes e's axis and node test select
+// from node, in axis order, before any predicate.
+func (e *Step) appendFrom(selected xdm.Sequence, node *xdm.Node) xdm.Sequence {
+ principal := e.Axis.PrincipalKind()
+ if nt, ok := e.Test.(*NameTest); ok && (e.Axis == AxisDescendant || e.Axis == AxisDescendantOrSelf) {
+ // The direct walk selects what walkAxis would, in the same order.
+ if e.Axis == AxisDescendantOrSelf && nt.Matches(node, principal) {
+ selected = append(selected, node)
+ }
+ return appendNamedDescendants(selected, node, nt)
+ }
+ // Ancestors come in a chain, so a first match usually has more behind it:
+ // one block of four in place of the 1, 2, 4 growth.
+ chain := e.Axis == AxisAncestor || e.Axis == AxisAncestorOrSelf
+ walkAxis(node, e.Axis, func(n *xdm.Node) bool {
+ if e.Test.Matches(n, principal) {
+ if chain && selected == nil {
+ selected = make(xdm.Sequence, 0, 4)
+ }
+ selected = append(selected, n)
+ }
+ return true
+ })
+ return selected
+}
+
// Eval implements Expr for a path expression.
func (e *PathExpr) Eval(ctx *Context) (xdm.Sequence, error) {
var cur xdm.Sequence
@@ -234,6 +265,13 @@ func evalStepOver(ctx *Context, input xdm.Sequence, step Expr, last bool) (xdm.S
var v xdm.Sequence
var err error
st, fresh := step.(*Step)
+ if fresh && len(st.Predicates) == 0 && !st.Axis.IsReverse() {
+ // Nothing numbers this node's selection apart from the others,
+ // so it goes straight onto out; the sort below orders and
+ // deduplicates the whole, as it would the per-node results.
+ out = st.appendFrom(out, it.(*xdm.Node))
+ continue
+ }
if fresh {
v, err = st.evalFrom(ctx, it.(*xdm.Node))
} else if da, ok := step.(*descendantAttrs); ok {
@@ -328,8 +366,9 @@ func applyPredicate(ctx *Context, seq xdm.Sequence, pred Expr) (xdm.Sequence, er
var out xdm.Sequence
size := len(seq)
var h *comparisonHoist
- if size > 1 {
- h = newComparisonHoist(ctx, pred)
+ var hoist comparisonHoist // on the stack: one allocation less per predicate
+ if size > 1 && hoist.init(ctx, pred) {
+ h = &hoist
}
// One focus context serves every item when the predicate cannot keep
@@ -470,34 +509,36 @@ type comparisonHoist struct {
ready bool
}
-// newComparisonHoist returns nil when pred is not such a comparison, under
+// init sets up h for pred and reports true, or reports false when pred is not
+// such a comparison, under
// XPath 1.0 compatibility (whose conversions need the raw operands), and for
// "A = (M to N)", which evalGeneralComparison answers from the bounds.
-func newComparisonHoist(ctx *Context, pred Expr) *comparisonHoist {
+func (h *comparisonHoist) init(ctx *Context, pred Expr) bool {
b, ok := pred.(*BinaryOp)
if !ok {
- return nil
+ return false
}
switch b.Op {
case "eq", "ne", "lt", "le", "gt", "ge":
case "=", "!=", "<", "<=", ">", ">=":
if r, isRange := b.Right.(*BinaryOp); ctx.Compat() || (isRange && r.Op == "to") {
- return nil
+ return false
}
default:
- return nil
+ return false
}
side, left := b.Right, false
if !hoistable(side, false) {
side, left = b.Left, true
if !hoistable(side, false) {
- return nil
+ return false
}
}
if !callsBuiltins(side, ctx) {
- return nil
+ return false
}
- return &comparisonHoist{cmp: b, left: left}
+ *h = comparisonHoist{cmp: b, left: left}
+ return true
}
// holds reports whether the predicate selects sub's context item. The hoisted
diff --git a/xpath/fn_31.go b/xpath/fn_31.go
index 2b18e2e9..5122e73b 100644
--- a/xpath/fn_31.go
+++ b/xpath/fn_31.go
@@ -1179,7 +1179,7 @@ func permuteWith(state uint64, seq xdm.Sequence) xdm.Sequence {
// A map or an array compared against anything else is simply unequal rather
// than an error, which is the same rule the atomic branch follows for values
// of incomparable types.
-func deepEqualMapArray(ctx *Context, x, y xdm.Item) (eq, handled bool, err error) {
+func deepEqualMapArray(ctx *Context, x, y xdm.Item, depth int) (eq, handled bool, err error) {
xm, xIsMap := x.(*xdm.MapItem)
ym, yIsMap := y.(*xdm.MapItem)
xa, xIsArr := x.(*xdm.ArrayItem)
@@ -1190,10 +1190,10 @@ func deepEqualMapArray(ctx *Context, x, y xdm.Item) (eq, handled bool, err error
}
switch {
case xIsMap && yIsMap:
- e, err := deepEqualMaps(ctx, xm, ym)
+ e, err := deepEqualMaps(ctx, xm, ym, depth)
return e, true, err
case xIsArr && yIsArr:
- e, err := deepEqualArrays(ctx, xa, ya)
+ e, err := deepEqualArrays(ctx, xa, ya, depth)
return e, true, err
}
// One of the two is a map or an array and the other is not, so they
@@ -1202,7 +1202,11 @@ func deepEqualMapArray(ctx *Context, x, y xdm.Item) (eq, handled bool, err error
return false, true, nil
}
-func deepEqualArrays(ctx *Context, a, b *xdm.ArrayItem) (bool, error) {
+func deepEqualArrays(ctx *Context, a, b *xdm.ArrayItem, depth int) (bool, error) {
+ // Nested values recurse here; bound them like any other recursion.
+ if err := checkValueDepth(depth); err != nil {
+ return false, err
+ }
if a.Len() != b.Len() {
return false, nil
}
@@ -1211,7 +1215,7 @@ func deepEqualArrays(ctx *Context, a, b *xdm.ArrayItem) (bool, error) {
// Members are sequences, so this is the sequence comparison rather
// than the item one: an array of one two-item member is not equal to
// an array of two one-item members.
- eq, err := deepEqual(ctx, am[i], bm[i])
+ eq, err := deepEqualAt(ctx, am[i], bm[i], depth+1)
if err != nil || !eq {
return false, err
}
@@ -1219,7 +1223,11 @@ func deepEqualArrays(ctx *Context, a, b *xdm.ArrayItem) (bool, error) {
return true, nil
}
-func deepEqualMaps(ctx *Context, a, b *xdm.MapItem) (bool, error) {
+func deepEqualMaps(ctx *Context, a, b *xdm.MapItem, depth int) (bool, error) {
+ // Nested values recurse here; bound them like any other recursion.
+ if err := checkValueDepth(depth); err != nil {
+ return false, err
+ }
if a.Len() != b.Len() {
return false, nil
}
@@ -1242,7 +1250,7 @@ func deepEqualMaps(ctx *Context, a, b *xdm.MapItem) (bool, error) {
equal = false
return nil
}
- eq, err := deepEqual(ctx, value, other)
+ eq, err := deepEqualAt(ctx, value, other, depth+1)
if err != nil {
return err
}
diff --git a/xpath/fn_array.go b/xpath/fn_array.go
index 4586a541..22b76c7f 100644
--- a/xpath/fn_array.go
+++ b/xpath/fn_array.go
@@ -70,7 +70,8 @@ func registerArrayFuncs(l *Library) {
if err != nil {
return nil, err
}
- return xdm.One(xdm.NewArray(append(a.Members(), seqArg(args, 1))...)), nil
+ members := a.AppendMembers(make([]xdm.Sequence, 0, a.Len()+1))
+ return xdm.One(xdm.NewArray(append(members, seqArg(args, 1))...)), nil
})
// array:subarray($array, $start as xs:integer[, $length as xs:integer])
@@ -161,11 +162,11 @@ func registerArrayFuncs(l *Library) {
"array:insert-before: position %d is outside the array's bounds (1 to %d)",
i, a.Len()+1)
}
- old := a.Members()
- members := make([]xdm.Sequence, 0, len(old)+1)
- members = append(members, old[:i-1]...)
- members = append(members, seqArg(args, 2))
- members = append(members, old[i-1:]...)
+ members := make([]xdm.Sequence, 0, a.Len()+1)
+ members = a.AppendMembers(members)
+ members = append(members, nil)
+ copy(members[i:], members[i-1:])
+ members[i-1] = seqArg(args, 2)
return xdm.One(xdm.NewArray(members...)), nil
})
@@ -216,7 +217,7 @@ func registerArrayFuncs(l *Library) {
return nil, xdm.ErrType(
"array:join: expected an array, got %s", it.TypeName())
}
- members = append(members, a.Members()...)
+ members = a.AppendMembers(members)
}
return xdm.One(xdm.NewArray(members...)), nil
})
diff --git a/xpath/fn_json.go b/xpath/fn_json.go
index 319f3c54..a39ca1fb 100644
--- a/xpath/fn_json.go
+++ b/xpath/fn_json.go
@@ -882,6 +882,8 @@ type jsonBuilder struct {
// to; finishing one pops it and hands the finished item to its parent.
stack []*jsonFrame
opts jsonOptions
+ // ctx is charged one item for every value the parse produces.
+ ctx *Context
// result is the top-level value, once one has been produced.
result xdm.Sequence
done bool
@@ -901,6 +903,9 @@ type jsonFrame struct {
}
func (b *jsonBuilder) emit(v xdm.Sequence) error {
+ if err := b.ctx.countItems(1); err != nil {
+ return err
+ }
if len(b.stack) == 0 {
if b.done {
return errFOJS0001("more than one value at the top level")
@@ -943,15 +948,13 @@ func (b *jsonBuilder) key(k string) error {
func (b *jsonBuilder) endMap() error {
f := b.stack[len(b.stack)-1]
b.stack = b.stack[:len(b.stack)-1]
- m := xdm.NewMap()
+ mb := xdm.NewMapBuilder()
for i, k := range f.keys {
- next, err := m.Put(xdm.NewString(k), f.values[i])
- if err != nil {
+ if err := mb.Set(xdm.NewString(k), f.values[i]); err != nil {
return err
}
- m = next
}
- return b.emit(xdm.One(m))
+ return b.emit(xdm.One(mb.Build()))
}
func (b *jsonBuilder) startArray() error {
@@ -1023,7 +1026,7 @@ func jsonDocReadError(err error) error {
// parseJSONText is the shared body of fn:parse-json and fn:json-doc.
func parseJSONText(ctx *Context, text string, opts jsonOptions) (xdm.Sequence, error) {
- b := &jsonBuilder{opts: opts}
+ b := &jsonBuilder{opts: opts, ctx: ctx}
fb := jsonFallback(ctx, opts)
if err := scanJSON(text, opts, b, fb); err != nil {
return nil, err
diff --git a/xpath/fn_map.go b/xpath/fn_map.go
index 1c9ba972..72df80f4 100644
--- a/xpath/fn_map.go
+++ b/xpath/fn_map.go
@@ -182,14 +182,18 @@ func registerMapFuncs(l *Library) {
// one array. Anything that is neither a map nor an array is passed over
// rather than being a type error, which is what makes it usable on a
// json-doc whose shape is not known in advance.
- l.registerMap("find", []int{2}, func(_ *Context, args []xdm.Sequence) (xdm.Sequence, error) {
+ l.registerMap("find", []int{2}, func(ctx *Context, args []xdm.Sequence) (xdm.Sequence, error) {
key, err := argMapKey(args, 1)
if err != nil {
return nil, err
}
var found []xdm.Sequence
- var walk func(seq xdm.Sequence) error
- walk = func(seq xdm.Sequence) error {
+ var walk func(seq xdm.Sequence, depth int) error
+ walk = func(seq xdm.Sequence, depth int) error {
+ // Recursion on the value's own nesting, so bounded like any other.
+ if err := checkValueDepth(depth); err != nil {
+ return err
+ }
for _, it := range seq {
switch v := it.(type) {
case *xdm.MapItem:
@@ -203,13 +207,13 @@ func registerMapFuncs(l *Library) {
found = append(found, val)
}
if err := v.Entries(func(_ *xdm.Atomic, val xdm.Sequence) error {
- return walk(val)
+ return walk(val, depth+1)
}); err != nil {
return err
}
case *xdm.ArrayItem:
for _, mem := range v.Members() {
- if err := walk(mem); err != nil {
+ if err := walk(mem, depth+1); err != nil {
return err
}
}
@@ -217,7 +221,7 @@ func registerMapFuncs(l *Library) {
}
return nil
}
- if err := walk(args[0]); err != nil {
+ if err := walk(args[0], 0); err != nil {
return nil, err
}
return xdm.One(xdm.NewArray(found...)), nil
diff --git a/xpath/fn_misc.go b/xpath/fn_misc.go
index c7ba5b70..0ea074d8 100644
--- a/xpath/fn_misc.go
+++ b/xpath/fn_misc.go
@@ -198,7 +198,7 @@ func registerUnparsedText(l *Library, since Version) {
if err != nil {
return nil, err
}
- return xdm.Sequence(splitTextLines(text)), nil
+ return splitTextLines(ctx, text)
})
}
@@ -298,21 +298,26 @@ func readText(ctx *Context, s, enc string) (string, error) {
// The spec defines the result as the sequence obtained by splitting on \n,
// \r\n or \r, with a trailing line ending producing no final empty string —
// "a\n" is one line, not two.
-func splitTextLines(text string) []xdm.Item {
+func splitTextLines(ctx *Context, text string) (xdm.Sequence, error) {
if text == "" {
- return nil
+ return nil, nil
}
// Normalise the two-character ending first so the split is over single
// characters, then treat a lone \r as an ending too.
text = strings.ReplaceAll(text, "\r\n", "\n")
text = strings.ReplaceAll(text, "\r", "\n")
text = strings.TrimSuffix(text, "\n")
+ // Charge the lines before the strings.Split slice and the items are built:
+ // a 67 MB file of newlines is 67M items.
+ out, err := makeSequence(ctx, strings.Count(text, "\n")+1)
+ if err != nil {
+ return nil, err
+ }
parts := strings.Split(text, "\n")
- out := make([]xdm.Item, 0, len(parts))
for _, p := range parts {
out = append(out, xdm.NewString(p))
}
- return out
+ return out, nil
}
// unparsedTextArgs pulls the href and the optional encoding out of a call on
@@ -493,11 +498,16 @@ func lookupByID(ctx *Context, args []xdm.Sequence, wantID bool) (xdm.Sequence, e
// instructions are ignored when comparing element content, which is why this
// cannot simply compare serialised forms.
func deepEqual(ctx *Context, a, b xdm.Sequence) (bool, error) {
+ return deepEqualAt(ctx, a, b, 0)
+}
+
+// deepEqualAt is deepEqual for values nested depth maps or arrays down.
+func deepEqualAt(ctx *Context, a, b xdm.Sequence, depth int) (bool, error) {
if len(a) != len(b) {
return false, nil
}
for i := range a {
- eq, err := deepEqualItem(ctx, a[i], b[i])
+ eq, err := deepEqualItem(ctx, a[i], b[i], depth)
if err != nil || !eq {
return false, err
}
@@ -505,7 +515,7 @@ func deepEqual(ctx *Context, a, b xdm.Sequence) (bool, error) {
return true, nil
}
-func deepEqualItem(ctx *Context, x, y xdm.Item) (bool, error) {
+func deepEqualItem(ctx *Context, x, y xdm.Item, depth int) (bool, error) {
// An Opaque is engine-internal state, not a value the data model defines
// equality over. A stylesheet naming the internal namespace can put one
// here, so it is compared by identity rather than asserted to be atomic.
@@ -531,7 +541,7 @@ func deepEqualItem(ctx *Context, x, y xdm.Item) (bool, error) {
// before the node and atomic branches because neither is either one: the
// fallback below compares unrecognised items by identity, so two equal
// arrays built separately were answering false.
- if eq, handled, err := deepEqualMapArray(ctx, x, y); handled {
+ if eq, handled, err := deepEqualMapArray(ctx, x, y, depth); handled {
return eq, err
}
@@ -1313,6 +1323,10 @@ func defaultPresentation(comp byte) string {
return "1"
}
+// maxPictureWidth bounds a width modifier's number. The padding it asks for is
+// built before any budget sees it: [D01,1000000000] allocated 2.8 GB.
+const maxPictureWidth = 1 << 16
+
// checkWidthModifier rejects a width modifier whose syntax is wrong, which is
// XTDE1340. Both bounds must be "*" or an integer greater than zero.
func checkWidthModifier(width string) error {
@@ -1333,6 +1347,10 @@ func checkWidthModifier(width string) error {
return fmt.Errorf("FOFD1340: malformed width modifier %q", width)
}
n = n*10 + int(c-'0')
+ if n > maxPictureWidth {
+ return fmt.Errorf("FOFD1340: width modifier %q asks for more than %d "+
+ "characters: %w", width, maxPictureWidth, xdm.ErrResourceLimit)
+ }
}
if n == 0 {
return fmt.Errorf("FOFD1340: width modifier %q must be greater than zero", width)
diff --git a/xpath/fn_misc_test.go b/xpath/fn_misc_test.go
index 518208bc..8cc5cf9a 100644
--- a/xpath/fn_misc_test.go
+++ b/xpath/fn_misc_test.go
@@ -867,6 +867,10 @@ func TestDistinctValuesNumericPromotion(t *testing.T) {
{`count(distinct-values((1.2, xs:double(1.2))))`, "1"},
{`count(distinct-values((1, xs:double(1))))`, "1"},
{`count(distinct-values((1, 1.0, xs:double(1), xs:float(1))))`, "1"},
+ // The promoted keys are recorded from the first double on, for the
+ // exact values kept before it and after it alike.
+ {`count(distinct-values((1.2, xs:double(5), 1.3, xs:double(1.2), xs:double(1.3))))`, "3"},
+ {`count(distinct-values((xs:double(1.2), 1.2)))`, "1"},
// Genuinely different values are still distinguished.
{`count(distinct-values((1, 2, 3)))`, "3"},
{`count(distinct-values((1.2, 1.3)))`, "2"},
@@ -1012,6 +1016,32 @@ func TestDistinctValuesLargeHomogeneous(t *testing.T) {
if got := evalStrXSLT(t, testDoc, `count(distinct-values((1 to 1000, xs:float(1.0))))`); got != "1000" {
t.Errorf("range plus equal float = %q, want 1000", got)
}
+
+ // With no float or double among them, exact values are keyed once, not
+ // also under the double and float they promote to. 2000 integers
+ // measured 14,061 allocations with the promoted keys and 6,054 without.
+ ctx := NewContext(nil, Builtins())
+ v, err := Eval(`1000 to 2999`, ctx, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ ctx.Vars["v"] = v
+ dv := MustCompile(`distinct-values($v)`, nil)
+ var n int
+ allocs := testing.AllocsPerRun(5, func() {
+ got, err := dv.Eval(ctx)
+ if err != nil {
+ t.Fatal(err)
+ }
+ n = len(got)
+ })
+ if n != 2000 {
+ t.Fatalf("distinct-values($v): %d items, want 2000", n)
+ }
+ t.Logf("%.0f allocations", allocs)
+ if allocs > 10000 {
+ t.Errorf("distinct-values($v) allocated %.0f times, want at most 10,000", allocs)
+ }
}
// benchEvalXSLT is evalStrXSLT for a benchmark, which cannot use the *testing.T
diff --git a/xpath/fn_regex.go b/xpath/fn_regex.go
index a231d5c7..5c647b25 100644
--- a/xpath/fn_regex.go
+++ b/xpath/fn_regex.go
@@ -133,10 +133,32 @@ func registerRegexFuncs(l *Library) {
if err != nil {
return nil, err
}
- out := re.ReplaceAllString(s, goRepl)
- if err := ctx.countBytes(len(out)); err != nil {
+ // No match leaves the input as it is: nothing is built and nothing is
+ // charged.
+ if !re.MatchString(s) {
+ return strSeq(s), nil
+ }
+ // The result is charged before it is built: 500 KB of input with a 4 KB
+ // replacement allocated 12.8 GB before the old after-the-fact charge
+ // refused it. The estimate is an upper bound -- the unmatched text, one
+ // replacement per match, and for each "$" in the replacement at most
+ // the matched text again, since a group lies inside its match. What
+ // the built string comes to is then settled against it, so a bound
+ // that was generous is handed back rather than kept.
+ n, matched := 0, 0
+ re.ReplaceAllStringFunc(s, func(m string) string { n++; matched += len(m); return "" })
+ est := len(s) - matched + n*len(goRepl) + strings.Count(goRepl, "$")*matched
+ if err := ctx.countBytes(est); err != nil {
return nil, err
}
+ out := re.ReplaceAllString(s, goRepl)
+ if len(out) > est {
+ if err := ctx.countBytes(len(out) - est); err != nil {
+ return nil, err
+ }
+ } else {
+ ctx.refundBytes(est - len(out))
+ }
return strSeq(out), nil
})
@@ -299,6 +321,16 @@ func compileXPathRegexp(pattern, flags string, v Version) (*regexp.Regexp, error
// the check and the insert under one lock: doing them separately let concurrent
// callers interleave between the two and carry the table past its limit.
func storeRegex(key string, v any) {
+ if len(key) > maxCachedKey {
+ return
+ }
+ // A pattern under the key limit can still compile to megabytes (85 copies
+ // of \p{L}{1000} is about 1 KB and retains 3.95 MB), so a large program is
+ // not kept either. The largest real pattern in the corpora is about 2,593
+ // instructions.
+ if re, ok := v.(*regexp.Regexp); ok && cachedProgramTooBig(re) {
+ return
+ }
regexCache.Put(key, v)
}
@@ -391,7 +423,7 @@ func buildRegexp(pattern, flags string, v Version) (*regexp.Regexp, error) {
// XPath's fn:matches is a *containment* test, unlike XML Schema's pattern
// facet which is anchored. Go's MatchString is also containment, so no
// anchoring is added here.
- re, err := regexp.Compile(translated)
+ re, err := CompileBounded(translated)
if err != nil {
// A repeat count RE2 will not accept is still a valid XML Schema
// pattern — "a{2147483647}" simply matches nothing any input can
@@ -399,7 +431,7 @@ func buildRegexp(pattern, flags string, v Version) (*regexp.Regexp, error) {
// when it was merely enormous, so the count is clamped to one past
// RE2's limit, which matches exactly what the original would.
if clamped, ok := clampRepeatCounts(translated); ok {
- if re2, err2 := regexp.Compile(clamped); err2 == nil {
+ if re2, err2 := CompileBounded(clamped); err2 == nil {
return re2, nil
}
}
diff --git a/xpath/fn_seq.go b/xpath/fn_seq.go
index 6763672a..d014e224 100644
--- a/xpath/fn_seq.go
+++ b/xpath/fn_seq.go
@@ -352,6 +352,21 @@ func fnDistinctValues(ctx *Context, args []xdm.Sequence) (xdm.Sequence, error) {
// the pairwise scan walks neither the non-numeric results nor the exact
// numerics, which the hash key already settles.
var inexact []*xdm.Atomic
+ // A later double is compared against an exact value by promoting it to
+ // double, so each exact value kept records which double (and float) it
+ // promotes to as well. That is the key the double will look itself up
+ // under, and it is what makes xs:decimal("1.2") and xs:double(1.2) one
+ // value whichever order they arrive in. Only a float or a double looks
+ // those keys up, so they are recorded once the first one arrives, and
+ // the exact values kept until then wait in pending.
+ var pending []*big.Rat
+ promoting := false
+ promote := func(r *big.Rat) {
+ f, _ := r.Float64()
+ seen["xd\x00"+strconv.FormatFloat(f, 'g', 17, 64)] = true
+ seen["xf\x00"+strconv.FormatFloat(
+ float64(float32(f)), 'g', 17, 32)] = true
+ }
for _, it := range atoms {
a := it.(*xdm.Atomic)
@@ -397,15 +412,11 @@ func fnDistinctValues(ctx *Context, args []xdm.Sequence) (xdm.Sequence, error) {
continue
}
seen[key] = true
- // A later double is compared against this value by promoting
- // it to double, so record which double it promotes to as
- // well. That is the key the double will look itself up under,
- // and it is what makes xs:decimal("1.2") and xs:double(1.2)
- // one value whichever order they arrive in.
- f, _ := r.Float64()
- seen["xd\x00"+strconv.FormatFloat(f, 'g', 17, 64)] = true
- seen["xf\x00"+strconv.FormatFloat(
- float64(float32(f)), 'g', 17, 32)] = true
+ if promoting {
+ promote(r)
+ } else {
+ pending = append(pending, r)
+ }
out = append(out, a)
continue
}
@@ -414,7 +425,14 @@ func fnDistinctValues(ctx *Context, args []xdm.Sequence) (xdm.Sequence, error) {
// the scan. An exact value e is equal to this value under eq
// exactly when e promoted to this type equals it -- so the double
// (or float) that e promotes to is the key to look up, and each
- // exact value recorded both when it was kept.
+ // exact value records both from here on.
+ if !promoting {
+ promoting = true
+ for _, r := range pending {
+ promote(r)
+ }
+ pending = nil
+ }
var key string
if a.Type == xdm.TypeFloat {
key = "xf\x00" + strconv.FormatFloat(
diff --git a/xpath/fn_serialize.go b/xpath/fn_serialize.go
index fe9f5fae..bde6d6a9 100644
--- a/xpath/fn_serialize.go
+++ b/xpath/fn_serialize.go
@@ -1,7 +1,9 @@
package xpath
import (
+ "bufio"
"fmt"
+ "io"
"math"
"sort"
"strings"
@@ -113,6 +115,12 @@ func registerSerialize(l *Library) {
if err := serializeItem(sb, it, opts); err != nil {
return nil, err
}
+ // A refused sink drops every later write, but the loop would
+ // still walk the rest of the items and their separators. The
+ // latched field is read directly: err() settles the reserve.
+ if sb.e != nil {
+ return nil, sb.err()
+ }
prevAtomic = atomic
}
if err := sb.err(); err != nil {
@@ -902,13 +910,13 @@ func serializeNode(sb *serializeSink, n *xdm.Node, opts serializeOptions, depth
}
case xdm.KindElement:
sb.WriteString("<")
- sb.WriteString(elementName(n))
+ writeName(sb, n)
writeNamespaceDecls(sb, n, opts)
// Attributes are written in the order the document had them, which is
// what a round-trip test compares against.
for a := range n.Attrs() {
sb.WriteString(" ")
- sb.WriteString(elementName(a))
+ writeName(sb, a)
sb.WriteString(`="`)
// A URI-valued attribute under the html or xhtml method has its
// non-ASCII characters percent-escaped. XSLT 2.0
@@ -917,9 +925,9 @@ func serializeNode(sb *serializeSink, n *xdm.Node, opts serializeOptions, depth
// so the two serialisers disagreed on identical input until now.
if opts.escapesURIAttrs() && (opts.method == "html" || opts.method == "xhtml") &&
isURIAttribute(n.Name().Local, a) {
- sb.WriteString(escapeAttr(escapeURIAttribute(a.Value())))
+ writeEscaped(sb, escapeURIAttribute(a.Value()), true)
} else {
- sb.WriteString(escapeAttr(a.Value()))
+ writeEscaped(sb, a.Value(), true)
}
sb.WriteString(`"`)
}
@@ -952,7 +960,7 @@ func serializeNode(sb *serializeSink, n *xdm.Node, opts serializeOptions, depth
sb.WriteString(">")
} else {
sb.WriteString(">")
}
return
@@ -1046,10 +1054,10 @@ func serializeNode(sb *serializeSink, n *xdm.Node, opts serializeOptions, depth
writeIndent(sb, depth)
}
sb.WriteString("")
case xdm.KindText:
- sb.WriteString(escapeText(opts.normalized(n.Value())))
+ writeEscaped(sb, opts.normalized(n.Value()), false)
case xdm.KindComment:
sb.WriteString("")
+ s.writeString("")
case xdm.KindPI:
if err := s.validatePI(n.Name().Local, n.Value()); err != nil {
@@ -730,9 +772,11 @@ func (s *serializer) node(n *xdm.Node, depth int) {
return
}
s.indent(depth)
- s.writeString("" alone,
// which is the same rule the SERE0015 check above is written for: if
@@ -1154,15 +1198,12 @@ func (s *serializer) ownNamespaceDecl(n *xdm.Node, base int, prefix, uri string)
// otherwise the union is sorted by prefix, which is the order a literal
// result element copies its bindings in.
func rootNamespaces(n *xdm.Node) []nsPair {
- scope := n.InScopeNamespaces()
- own := make(map[string]bool, n.NumNamespaceDecls())
- for prefix := range n.DeclaredNamespaces() {
- own[prefix] = true
- }
- var out []nsPair
- for p, uri := range scope {
- if p != "xml" && !own[p] {
- out = append(out, nsPair{p, uri})
+ var arr [32]nsBinding
+ scope := scopeBindings(n, arr[:0])
+ out := make([]nsPair, 0, len(scope)+n.NumNamespaceDecls())
+ for _, b := range scope {
+ if !declares(n, b.prefix) {
+ out = append(out, nsPair{b.prefix, b.uri})
}
}
inherited := len(out)
@@ -1170,8 +1211,8 @@ func rootNamespaces(n *xdm.Node) []nsPair {
out = append(out, nsPair{prefix, uri})
}
if inherited > 0 {
- sort.SliceStable(out, func(i, j int) bool {
- return out[i].prefix < out[j].prefix
+ slices.SortStableFunc(out, func(a, b nsPair) int {
+ return strings.Compare(a.prefix, b.prefix)
})
}
return out
@@ -1668,30 +1709,36 @@ func (s *serializer) writeCData(text string) {
// character the encoding cannot hold — NFD splits "\u00e7" into "c" and a
// combining cedilla, and US-ASCII output must then break the section
// around the cedilla rather than write it raw.
+ //
+ // The section is written to w in runs, as escapeText writes text: the
+ // function has no error path, so nothing is gained by building it first.
text = s.normalized(text)
- var sb strings.Builder
- sb.WriteString("") {
// The section ends at the first "]]>", which the text may itself
// contain — so it is split there and the ">" begins a new one.
- sb.WriteString("]]]]>")
+ s.writeString(text[run:i])
+ s.writeString("]]]]>")
i += 3
+ run = i
continue
}
r, size := utf8.DecodeRuneInString(text[i:])
if !s.representable(r) {
- sb.WriteString("]]>")
- fmt.Fprintf(&sb, "&#%d;", r)
- sb.WriteString("")
+ s.writeCharRef(r)
+ s.writeString("")
- s.writeString(sb.String())
+ s.writeString(text[run:])
+ s.writeString("]]>")
}
// representable reports whether the declared encoding can hold this
@@ -1785,7 +1832,10 @@ func (s *serializer) mapSegments(text string) []mapSegment {
s.oneSeg[0] = mapSegment{text: s.normalized(text)}
return s.oneSeg[:]
}
- var segs []mapSegment
+ // The slice is the serializer's own, reused for the same reason as
+ // oneSeg: a fresh one per text node and attribute value was the whole
+ // allocation cost of serialising with a character map.
+ segs := s.segBuf[:0]
run := 0
for i, r := range text {
repl, ok := s.charMap[r]
@@ -1799,7 +1849,9 @@ func (s *serializer) mapSegments(text string) []mapSegment {
})
run = i + utf8.RuneLen(r)
}
- return append(segs, mapSegment{text: s.normalized(text[run:])})
+ segs = append(segs, mapSegment{text: s.normalized(text[run:])})
+ s.segBuf = segs
+ return segs
}
// mapSegment is one unmapped run of text and, where the run ended at a
@@ -2758,15 +2810,5 @@ func flattenArrays(seq xdm.Sequence) xdm.Sequence {
if !has {
return seq
}
- out := make(xdm.Sequence, 0, len(seq))
- for _, it := range seq {
- if a, ok := it.(*xdm.ArrayItem); ok {
- for _, m := range a.Members() {
- out = append(out, flattenArrays(m)...)
- }
- continue
- }
- out = append(out, it)
- }
- return out
+ return xdm.Flatten(seq)
}
diff --git a/xslt/serialize_streamalloc_test.go b/xslt/serialize_streamalloc_test.go
new file mode 100644
index 00000000..9faf39ca
--- /dev/null
+++ b/xslt/serialize_streamalloc_test.go
@@ -0,0 +1,169 @@
+package xslt
+
+import (
+ "bytes"
+ "io"
+ goruntime "runtime"
+ "strings"
+ "testing"
+
+ "github.com/knroy/go-xml/v2/xdm"
+)
+
+// streamDoc is n copies of an element holding text, a comment and a PI, so
+// that every path below has a node per copy to allocate for.
+func streamDoc(t *testing.T, n int) xdm.Sequence {
+ t.Helper()
+ var b strings.Builder
+ b.WriteString(``)
+ for range n {
+ b.WriteString(`x ]]> y z`)
+ }
+ b.WriteString(``)
+ tree, err := xdm.ParseString(b.String(), xdm.ParseOptions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ return xdm.One(tree.Root)
+}
+
+// The cdata-section-elements path built each section in a builder, a
+// character map allocated a segment slice per text node, and comments and PIs
+// were concatenated before being written. Each cost an allocation per node,
+// so the count must not grow with the number of nodes.
+func TestSerializeStreamAllocsFlat(t *testing.T) {
+ if raceEnabled {
+ t.Skip("allocation counts are not stable under -race")
+ }
+ for _, tc := range []struct {
+ name string
+ opts OutputSettings
+ cmap map[rune]string
+ }{
+ {"cdata", OutputSettings{Method: "xml", CDataElements: []xdm.QName{{Local: "a"}}}, nil},
+ {"charmap", OutputSettings{Method: "xml"}, map[rune]string{' ': " "}},
+ {"comment-pi", OutputSettings{Method: "xml"}, nil},
+ } {
+ count := func(seq xdm.Sequence) float64 {
+ return testing.AllocsPerRun(10, func() {
+ if err := Serialize(io.Discard, seq, tc.opts, tc.cmap); err != nil {
+ t.Fatal(err)
+ }
+ })
+ }
+ few, many := count(streamDoc(t, 5)), count(streamDoc(t, 500))
+ if many > few+5 {
+ t.Errorf("%s: 500 nodes %.0f allocations, 5 nodes %.0f; want the same", tc.name, many, few)
+ }
+ }
+}
+
+// The text method concatenated the whole string value before writing it, a
+// copy of the output and its builder's growth on top. It now writes each text
+// node as it comes, so what it allocates must not grow with the output.
+func TestSerializeTextMethodStreams(t *testing.T) {
+ if raceEnabled {
+ t.Skip("allocation counts are not stable under -race")
+ }
+ seq := streamDoc(t, 20000)
+ var out bytes.Buffer
+ if err := Serialize(&out, seq, OutputSettings{Method: "text"}, nil); err != nil {
+ t.Fatal(err)
+ }
+ if got := allocBytes(func() {
+ if err := Serialize(io.Discard, seq, OutputSettings{Method: "text"}, nil); err != nil {
+ t.Fatal(err)
+ }
+ }); got > uint64(out.Len())/4 {
+ t.Errorf("text method allocated %d bytes for %d bytes of output", got, out.Len())
+ }
+}
+
+// allocBytes is the heap f allocates, the least of three runs so that a
+// collection landing inside one does not count against it.
+func allocBytes(f func()) uint64 {
+ best := ^uint64(0)
+ for range 3 {
+ var a, b goruntime.MemStats
+ goruntime.ReadMemStats(&a)
+ f()
+ goruntime.ReadMemStats(&b)
+ best = min(best, b.TotalAlloc-a.TotalAlloc)
+ }
+ return best
+}
+
+// The json and adaptive methods built the whole document as a string before
+// writing it, 7 to 12 times the output in all. They now write as they render,
+// so what they allocate must not grow with the output.
+func TestSerializeJSONAdaptiveStream(t *testing.T) {
+ if raceEnabled {
+ t.Skip("allocation counts are not stable under -race")
+ }
+ seq := streamDoc(t, 20000)
+ for _, tc := range []struct {
+ name string
+ opts OutputSettings
+ cmap map[rune]string
+ }{
+ {"json", OutputSettings{Method: "json"}, nil},
+ {"adaptive", OutputSettings{Method: "adaptive"}, nil},
+ {"adaptive-charmap", OutputSettings{Method: "adaptive"}, map[rune]string{' ': " "}},
+ } {
+ var out bytes.Buffer
+ if err := Serialize(&out, seq, tc.opts, tc.cmap); err != nil {
+ t.Fatal(err)
+ }
+ if got := allocBytes(func() {
+ if err := Serialize(io.Discard, seq, tc.opts, tc.cmap); err != nil {
+ t.Fatal(err)
+ }
+ }); got > uint64(out.Len())/4 {
+ t.Errorf("%s allocated %d bytes for %d bytes of output", tc.name, got, out.Len())
+ }
+ }
+}
+
+// Result.String, SecondaryResult.String and SerializeAsXML grew a
+// strings.Builder by doubling from empty, about five times the output. They now keep the serializer's flushed
+// chunks and join them once.
+func TestResultStringAllocation(t *testing.T) {
+ if raceEnabled {
+ t.Skip("allocation counts are not stable under -race")
+ }
+ for _, n := range []int{2000, 13000} {
+ r := &Result{Nodes: streamDoc(t, n), output: OutputSettings{Method: "xml"}}
+ sr := &SecondaryResult{Nodes: r.Nodes, Output: r.output}
+ out := r.String()
+ for name, f := range map[string]func(){
+ "String": func() { _ = r.String() },
+ "SerializeAsXML": func() { _ = SerializeAsXML(r) },
+ "SecondaryResult.String": func() { _ = sr.String() },
+ } {
+ got := allocBytes(f)
+ t.Logf("%s, %d bytes: %d allocated (%.2fx)", name, len(out), got, float64(got)/float64(len(out)))
+ // The 32 KiB write buffer is a fixed cost on top.
+ if got > uint64(len(out))*5/2+64<<10 {
+ t.Errorf("%s allocated %d bytes for %d bytes of output, want under 2.5x and the buffer", name, got, len(out))
+ }
+ }
+ }
+}
+
+// Each Serialize to an unbuffered writer allocated a 32 KiB buffer, most of
+// what a small result cost. The buffers are pooled now.
+func TestSerializeBufferPooled(t *testing.T) {
+ if raceEnabled {
+ t.Skip("allocation counts are not stable under -race")
+ }
+ seq := streamDoc(t, 10)
+ f := func() {
+ if err := Serialize(io.Discard, seq, OutputSettings{Method: "xml"}, nil); err != nil {
+ t.Fatal(err)
+ }
+ }
+ f()
+ if got := allocBytes(f); got > 16<<10 {
+ t.Errorf("a small result allocated %d bytes, want under 16 KiB", got)
+ }
+}
diff --git a/xslt/sharedns_test.go b/xslt/sharedns_test.go
index 784346ee..b70509c4 100644
--- a/xslt/sharedns_test.go
+++ b/xslt/sharedns_test.go
@@ -54,3 +54,49 @@ func TestInScopeNamespacesShared(t *testing.T) {
t.Error("maps shared outside a compilation")
}
}
+
+// SchXslt redeclares one prefix on each of the CEN rules' svrl:failed-assert
+// elements. Elements that each declare the same bindings share one map, one
+// resolver and one binding list in the compiled stylesheet, rather than a copy
+// per element.
+func TestNamespacesInternedByContent(t *testing.T) {
+ const sheet = `
+
+
+
+
+`
+ doc, err := xdm.ParseString(sheet, xdm.ParseOptions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ var outs []*xdm.Node
+ for n := range doc.Root.Descendants() {
+ if n.Kind() == xdm.KindElement && n.Name().Local == "out" {
+ outs = append(outs, n)
+ }
+ }
+ setSharedNS(map[*xdm.Node]map[string]string{})
+ ptr := func(m map[string]string) uintptr { return reflect.ValueOf(m).Pointer() }
+ if ptr(inScopeNamespacesShared(outs[0])) != ptr(inScopeNamespacesShared(outs[1])) {
+ t.Error("two elements declaring the same bindings do not share a map")
+ }
+ if newNSResolver(outs[0], "") != newNSResolver(outs[1], "") {
+ t.Error("two elements with the same static context do not share a resolver")
+ }
+ setSharedNS(nil)
+
+ st, err := Compile(doc.Root, CompileOptions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ var lits []*literalElemInstr
+ for _, in := range st.named["t"].Body {
+ if l, ok := in.(*literalElemInstr); ok {
+ lits = append(lits, l)
+ }
+ }
+ if len(lits) != 2 || len(lits[0].namespaces) == 0 || &lits[0].namespaces[0] != &lits[1].namespaces[0] {
+ t.Error("two literal elements with the same bindings do not share one list")
+ }
+}
diff --git a/xslt/sortvalue_test.go b/xslt/sortvalue_test.go
new file mode 100644
index 00000000..74ef6065
--- /dev/null
+++ b/xslt/sortvalue_test.go
@@ -0,0 +1,15 @@
+package xslt
+
+import (
+ "testing"
+ "unsafe"
+)
+
+// applySorts holds one sortValue per item per key. It was 152 bytes, with the
+// collation and case-fold material inline; that material now hangs off a
+// pointer, which only the keys needing it pay for.
+func TestSortValueStaysSmall(t *testing.T) {
+ if sz := unsafe.Sizeof(sortValue{}); sz > 96 {
+ t.Fatalf("sortValue is %d bytes, want at most 96", sz)
+ }
+}
diff --git a/xslt/stylesheet.go b/xslt/stylesheet.go
index 1f23b2d6..dfb10eab 100644
--- a/xslt/stylesheet.go
+++ b/xslt/stylesheet.go
@@ -2,9 +2,11 @@ package xslt
import (
"fmt"
+ "reflect"
"sort"
"strings"
"sync"
+ "unsafe"
"github.com/knroy/go-xml/v2/xdm"
"github.com/knroy/go-xml/v2/xpath"
@@ -245,6 +247,10 @@ type Stylesheet struct {
source *xdm.Node
// nested caches the stylesheets fn:transform compiles; see nestedCache.
nested nestedCache
+ // globalsOnce builds globalsByName, the first declaration of each global
+ // by Clark name, for evalGlobals; see globalByName.
+ globalsOnce sync.Once
+ globalsByName map[string]*Variable
}
// Template is a compiled xsl:template.
@@ -700,17 +706,61 @@ func Compile(doc *xdm.Node, opts CompileOptions) (*Stylesheet, error) {
// Outside a compilation m is nil and nothing is cached. It has its own mutex
// rather than relying on compileMu because a running transform, which does
// not hold compileMu, can reach inScopeNamespacesShared too.
+//
+// byContent and lists intern the maps and the literal elements' binding lists
+// by content as well: SchXslt redeclares one prefix on each of the CEN
+// rules' ~980 svrl:failed-assert elements, so keying by the declaring
+// element alone kept that many identical copies in the compiled stylesheet.
var sharedNS struct {
sync.Mutex
- m map[*xdm.Node]map[string]string
+ m map[*xdm.Node]map[string]string
+ byContent map[string]map[string]string
+ lists map[string][]nsBinding
+ resolvers map[nsResolverKey]*nsResolver
+ keyBuf []byte // scratch for internBindings, under the lock
}
func setSharedNS(m map[*xdm.Node]map[string]string) {
sharedNS.Lock()
sharedNS.m = m
+ sharedNS.byContent, sharedNS.lists, sharedNS.resolvers = nil, nil, nil
+ if m != nil {
+ sharedNS.byContent = map[string]map[string]string{}
+ sharedNS.lists = map[string][]nsBinding{}
+ sharedNS.resolvers = map[nsResolverKey]*nsResolver{}
+ }
sharedNS.Unlock()
}
+// internBindings returns a shared copy of s for the rest of the compilation,
+// or s itself outside one. The result must not be modified.
+func internBindings(s []nsBinding) []nsBinding {
+ if len(s) == 0 {
+ return s
+ }
+ sharedNS.Lock()
+ defer sharedNS.Unlock()
+ if sharedNS.lists == nil {
+ return s
+ }
+ // Built in a buffer reused across calls and looked up as m[string(b)],
+ // which does not allocate: a strings.Builder allocated a key per call
+ // and raised CEN's compile allocation by 12-14%. Only a miss copies it.
+ b := sharedNS.keyBuf[:0]
+ for _, ns := range s {
+ b = append(b, ns.prefix...)
+ b = append(b, 0)
+ b = append(b, ns.uri...)
+ b = append(b, 0)
+ }
+ sharedNS.keyBuf = b
+ if v, ok := sharedNS.lists[string(b)]; ok {
+ return v
+ }
+ sharedNS.lists[string(b)] = s
+ return s
+}
+
// forgetSharedNS drops the cached maps, and the version memo, for a caller
// that has just changed a stylesheet tree's namespaces or parent links
// during a compilation.
@@ -741,11 +791,43 @@ func inScopeNamespacesShared(el *xdm.Node) map[string]string {
m, ok := sharedNS.m[d]
if !ok {
m = d.InScopeNamespaces()
+ // NUL cannot occur in a prefix or a URI, so it separates them.
+ prefixes := make([]string, 0, len(m))
+ for p := range m {
+ prefixes = append(prefixes, p)
+ }
+ sort.Strings(prefixes)
+ var b strings.Builder
+ for _, p := range prefixes {
+ b.WriteString(p)
+ b.WriteByte(0)
+ b.WriteString(m[p])
+ b.WriteByte(0)
+ }
+ if v, ok := sharedNS.byContent[b.String()]; ok {
+ m = v
+ } else {
+ sharedNS.byContent[b.String()] = m
+ }
sharedNS.m[d] = m
}
return m
}
+// globalByName is the first declaration of each global by Clark name. It
+// depends only on the compiled globals, so every transform shares one.
+func (s *Stylesheet) globalByName() map[string]*Variable {
+ s.globalsOnce.Do(func() {
+ s.globalsByName = make(map[string]*Variable, len(s.globals))
+ for _, g := range s.globals {
+ if _, dup := s.globalsByName[g.Name.Clark()]; !dup {
+ s.globalsByName[g.Name.Clark()] = g
+ }
+ }
+ })
+ return s.globalsByName
+}
+
// compileNestedLocked compiles a stylesheet from inside a compilation that is
// already running on this goroutine.
//
@@ -1233,7 +1315,7 @@ func newNSResolver(el *xdm.Node, defaultElementNS string) *nsResolver {
if defaultElementNS == "" {
defaultElementNS = xpathDefaultNamespaceAt(el)
}
- return &nsResolver{
+ r := &nsResolver{
bindings: inScopeNamespacesShared(el),
defaultNS: defaultElementNS,
baseURI: el.BaseURI(),
@@ -1245,6 +1327,38 @@ func newNSResolver(el *xdm.Node, defaultElementNS string) *nsResolver {
xpathVersion: xpathVersionAt(el),
xsltVersion: declaredXSLTVersion(el),
}
+ return internResolver(r)
+}
+
+// nsResolverKey is an nsResolver as a map key. The bindings map is keyed by
+// identity, which inScopeNamespacesShared has already made content identity
+// for the compilation; every other field must be listed here.
+type nsResolverKey struct {
+ bindings unsafe.Pointer
+ defaultNS, baseURI, collate string
+ compat bool
+ schema *xsd.Schema
+ pkg int
+ xpathVersion xpath.Version
+ xsltVersion float64
+}
+
+// internResolver returns the compilation's shared resolver equal to r, or r
+// itself outside a compilation. A resolver is never modified once built
+// (xsl:evaluate copies one to change it), so equal ones can be one.
+func internResolver(r *nsResolver) *nsResolver {
+ sharedNS.Lock()
+ defer sharedNS.Unlock()
+ if sharedNS.resolvers == nil {
+ return r
+ }
+ k := nsResolverKey{reflect.ValueOf(r.bindings).UnsafePointer(), r.defaultNS, r.baseURI,
+ r.collation, r.compat, r.schema, r.pkg, r.xpathVersion, r.xsltVersion}
+ if v, ok := sharedNS.resolvers[k]; ok {
+ return v
+ }
+ sharedNS.resolvers[k] = r
+ return r
}
// LookupSchemaDeclaration implements xpath.SchemaTypes.
@@ -1666,13 +1780,28 @@ func (r *nsResolver) ResolvePrefix(p string) (string, bool) {
func (r *nsResolver) DefaultElementNamespace() string { return r.defaultNS }
func (r *nsResolver) DefaultFunctionNamespace() string { return xdm.NSFN }
+// prefixLookup is an element, or the in-scope namespaces a compiled
+// instruction keeps in its place so as not to keep the stylesheet tree alive.
+type prefixLookup interface {
+ LookupPrefix(prefix string) (string, bool)
+}
+
+// nsMap is an element's in-scope namespaces, as InScopeNamespaces returns
+// them (xml included), answering LookupPrefix exactly as the element would.
+type nsMap map[string]string
+
+func (m nsMap) LookupPrefix(prefix string) (string, bool) {
+ uri, ok := m[prefix]
+ return uri, ok || prefix == ""
+}
+
// resolveQNameAttr resolves a lexical QName found in a stylesheet attribute
// against the element's namespace context.
//
// An unprefixed QName in an attribute value is in *no* namespace, unlike an
// unprefixed element name test. This is the rule that makes
// name="foo" and mean different things inside a default namespace.
-func resolveQNameAttr(el *xdm.Node, lex string) (xdm.QName, error) {
+func resolveQNameAttr(el prefixLookup, lex string) (xdm.QName, error) {
lex = strings.TrimSpace(lex)
// The EQName form Q{uri}local carries its own namespace, so no prefix has
// to be in scope for it. The suite writes it wherever a QName is
diff --git a/xslt/transform.go b/xslt/transform.go
index 1033e75e..994d45f2 100644
--- a/xslt/transform.go
+++ b/xslt/transform.go
@@ -532,7 +532,7 @@ func (s *Stylesheet) Transform(ctx context.Context, source *xdm.Node, opts Trans
}
}
if err := runTemplate(rt, t,
- opts.InitialTemplateParams, opts.InitialTemplateTunnelParams,
+ paramsOf(opts.InitialTemplateParams), paramsOf(opts.InitialTemplateTunnelParams),
out); err != nil {
return nil, err
}
@@ -620,7 +620,7 @@ func (s *Stylesheet) Transform(ctx context.Context, source *xdm.Node, opts Trans
"parameter $%s", code, opts.InitialTemplate, p.Name.Lexical())
}
if err := runTemplate(rt, t,
- opts.InitialTemplateParams, opts.InitialTemplateTunnelParams,
+ paramsOf(opts.InitialTemplateParams), paramsOf(opts.InitialTemplateTunnelParams),
out); err != nil {
return nil, err
}
@@ -699,7 +699,7 @@ func (s *Stylesheet) Transform(ctx context.Context, source *xdm.Node, opts Trans
initialMode = want
}
if err := applyInitialSelection(rt, source, opts.InitialMatchSelection,
- initialMode, opts.InitialModeParams, opts.InitialModeTunnelParams,
+ initialMode, paramsOf(opts.InitialModeParams), paramsOf(opts.InitialModeTunnelParams),
out); err != nil {
return nil, err
}
@@ -748,6 +748,14 @@ func (s *Stylesheet) Transform(ctx context.Context, source *xdm.Node, opts Trans
}
}
+ // A budget refusal is latched by the builder that met it and is only
+ // reported by execSequenceBody's loop. The built-in rules, and any
+ // temporary tree, write through a builder nobody polls, so output cut
+ // short at the bound would otherwise be returned as a success. The budget
+ // counter is the transform's, so it sees the overrun wherever it happened.
+ if err := rt.ctx.NodesRefused(); err != nil {
+ return nil, err
+ }
nodes := out.Sequence()
if haveRaw {
nodes = rawResult
@@ -1265,9 +1273,24 @@ func (s *Stylesheet) spaceDeclsFor(pkg int) (strip, preserve []xdm.QName) {
// Transform returns nil for a stylesheet whose output settings are invalid.
// Use Serialize for anything whose failure you need to see.
func (r *Result) String() string {
- var sb strings.Builder
- _ = r.Serialize(&sb)
- return sb.String()
+ return serializeString(r.Nodes, r.output, r.charMap)
+}
+
+// serializeString is serialize to a string. The output is kept in the chunks
+// the serializer's buffer flushes and joined once, about twice its size in
+// all, where a strings.Builder grown by doubling cost up to four times.
+func serializeString(seq xdm.Sequence, opts OutputSettings, charMap map[rune]string) string {
+ var c chunkWriter
+ _ = serialize(&c, seq, opts, charMap)
+ return strings.Join(c, "")
+}
+
+// chunkWriter keeps a copy of each write.
+type chunkWriter []string
+
+func (c *chunkWriter) Write(p []byte) (int, error) {
+ *c = append(*c, string(p))
+ return len(p), nil
}
// Serialize writes the result using the stylesheet's xsl:output settings.
@@ -1470,10 +1493,8 @@ func appendLeafCopy(parent, n *xdm.Node) *xdm.Node {
// and -020 stopped at ""), reporting a defect in the transform where the
// only defect was in how the comparison asked for the text.
func SerializeAsXML(r *Result) string {
- var sb strings.Builder
- _ = serialize(&sb, r.Nodes,
+ return serializeString(r.Nodes,
OutputSettings{Method: "xml", Version: r.output.Version}, r.charMap)
- return sb.String()
}
// applyInitialSelection runs the initial apply-templates over the initial
@@ -1485,7 +1506,7 @@ func SerializeAsXML(r *Result) string {
// position and size of each item have to be set the way xsl:apply-templates
// sets them, since a template rule reached this way may call fn:position().
func applyInitialSelection(rt *runtime, source *xdm.Node, sel xdm.Sequence,
- mode string, params, tunnels map[string]xdm.Sequence,
+ mode string, params, tunnels paramValues,
out *outputBuilder) error {
if sel == nil {