Memory: go-xmlsec's second report (v2.1.1)

CHANGELOG.md

@@ -4,6 +4,26 @@ Notable changes, newest first. Versions follow [semantic
 versioning](https://semver.org): from 1.0.0 the exported API is stable, and a
 breaking change means 2.0 with a new module path. See *Stability* below.
 
+## Unreleased
+
+Memory and CPU fixes from go-xmlsec's measurements against v2.1.0. No API
+change; output is byte-identical.
+
+### Fixed
+
+| Change | Problem → solution | Commit |
+|---|---|---|
+| `c14n.ParsePrefixList` was quadratic in the prefixes | It deduplicated with a scan per token: 1 MiB of distinct prefixes took 10.9 s. It uses a map, keeping first-seen order: about 0.02 s. | [`ad8c47a`][ad8c47a] |
+
+### Changed — memory
+
+| Change | Problem → solution | Commit |
+|---|---|---|
+| A large text value cost up to 3.1× its size to parse | `slices.Grow` rounded the buffer past the hand-off check, so the value was copied twice. The buffer is grown exactly and handed to the tree when no later value wants it: 1 MiB 3.10 → 1.12×, 10 MiB 1.87 → 1.08×. | [`707ca28`][707ca28] |
+| A tag with many attributes or namespace declarations peaked at 13–22× its size | Duplicates past 8 attributes are found by sorting, no name index is built for such a tag, one scope index serves both lookups and repeated URIs are interned: 800K attributes 21.5 → 12.8× peak, 100K declarations 16.5 → 6.6×. | [`5475837`][5475837], [`b08624c`][b08624c] |
+| c14n allocated a map per element of a namespace node set; the namespace axis sorted through a map | The map is reused (2,000 elements: 4,016 → 18 allocations); `scopeSlots` uses one stable sort (4 → 1 allocation a call). | [`9572c44`][9572c44], [`fd9750e`][fd9750e] |
+| `/r/l[1]` built every `l` child before the predicate ran | A step whose first predicate is an integer literal stops at that match. | [`5712d9f`][5712d9f] |
+
 ## v2.1.0 — 2026-11-01
 
 Three memory audits across parsing, XPath/XQuery, XSLT, validation,
@@ -1803,3 +1823,10 @@ here so every entry in this file sits under a release.
 [22d32ac]: https://github.com/knroy/go-xml/commit/22d32ac
 [b331aa1]: https://github.com/knroy/go-xml/commit/b331aa1
 [567aa1a]: https://github.com/knroy/go-xml/commit/567aa1a
+[ad8c47a]: https://github.com/knroy/go-xml/commit/ad8c47a
+[707ca28]: https://github.com/knroy/go-xml/commit/707ca28
+[5475837]: https://github.com/knroy/go-xml/commit/5475837
+[b08624c]: https://github.com/knroy/go-xml/commit/b08624c
+[9572c44]: https://github.com/knroy/go-xml/commit/9572c44
+[fd9750e]: https://github.com/knroy/go-xml/commit/fd9750e
+[5712d9f]: https://github.com/knroy/go-xml/commit/5712d9f

README.md

@@ -139,7 +139,7 @@ and maintains it as a project of his own.
 | **XSD 1.1** | 99.98% instance (26,217 of 26,222); **99.97%** schema-validity (15,350 of 15,354); opt-in via `Version11` |
 | **RELAX NG** | 100.00% of James Clark's spectest (965 of 965 assertions); XML and compact syntax, from Go and from `go-xml validate -rng` |
 | **DTD** | content models, attribute defaults, enumerations, `ID`/`IDREF`; external subset, parameter entities across both subsets, conditional sections — via `dtd.Load` with a caller-supplied resolver, nothing fetched by default |
-| **Tests** | 2,950 `func Test` declarations, clean under `-race` (a few subtests skip without the corpora below) |
+| **Tests** | 2,960 `func Test` declarations, clean under `-race` (a few subtests skip without the corpora below) |
 | **Production schemas** | UBL 2.1, UN/CEFACT CII, Factur-X/ZUGFeRD, Peppol BIS 3.0 — 88 schemas load, instances validate clean |
 | **API** | 1.2; the exported surface is stable and additive over 1.1, and a breaking change means 2.0 with a new module path |
 
@@ -1388,7 +1388,7 @@ back, is in [docs/testing.md](docs/testing.md).
 
 | method | what it catches | what it misses |
 |---|---|---|
-| **Unit tests** (2,950 `func Test` declarations) | places where a plausible implementation is quietly wrong | anything nobody thought to write a test for |
+| **Unit tests** (2,960 `func Test` declarations) | places where a plausible implementation is quietly wrong | anything nobody thought to write a test for |
 | **Spec inventories** | features absent entirely | features present but behaving wrongly |
 | **Saxon differential** | subtle behavioural divergence on real stylesheets | constructs the corpora do not use |
 | **W3C QT3 suite** | systematic conformance across 22,054 XPath and 30,517 XQuery cases | XSLT (it is an XPath suite) |

c14n/c14n.go

@@ -49,7 +49,6 @@ import (
 	"fmt"
 	"hash"
 	"io"
-	"slices"
 	"strings"
 
 	"github.com/knroy/go-xml/v2/xdm"
@@ -162,11 +161,13 @@ var (
 // string. Repeated and empty tokens are ignored.
 func ParsePrefixList(s string) []string {
 	var out []string
-	for _, f := range strings.Fields(s) {
+	seen := map[string]struct{}{}
+	for f := range strings.FieldsSeq(s) {
 		if f == "#default" {
 			f = ""
 		}
-		if !slices.Contains(out, f) {
+		if _, dup := seen[f]; !dup {
+			seen[f] = struct{}{}
 			out = append(out, f)
 		}
 	}
@@ -220,9 +221,10 @@ func (c *chunkWriter) bytes() []byte {
 // sizedWriter collects everything written to it in one buffer, sized once
 // from the tree under the node set's root at the first write (after the options were validated, so
 // a refused call allocates nothing). The estimate runs 5% to 15% over the
-// output, so the result costs about 1.1 times its size where a buffer grown
-// by doubling cost 2.1 times and joined chunks twice; a short estimate (an
-// escaped value is longer than its text) costs one more copy.
+// output on generated documents and averages 0.96 of it on DocBook, so the
+// result costs about 1.1 times its size where a buffer grown by doubling cost
+// 2.1 times and joined chunks twice; a short estimate (an escaped value is
+// longer than its text) costs one more copy.
 type sizedWriter struct {
 	ns   NodeSet
 	opts Options

c14n/canon.go

@@ -427,7 +427,13 @@ func (c *canon) namespaceNodes(e *xdm.Node, in bool) {
 	}
 	var axis map[string]string
 	if in {
-		axis = map[string]string{}
+		// Reuse the map a finished element left in c.axes's backing array.
+		if n := len(c.axes); n < cap(c.axes) && c.axes[:n+1][n] != nil {
+			axis = c.axes[:n+1][n]
+			clear(axis)
+		} else {
+			axis = map[string]string{}
+		}
 	}
 	hasDefault := false
 	for prefix, uri := range c.scope.m {

c14n/nsaxis_alloc_test.go

@@ -0,0 +1,38 @@
+package c14n
+
+import (
+	"io"
+	"strings"
+	"testing"
+
+	"github.com/knroy/go-xml/v2/xdm"
+)
+
+// canon.namespaceNodes made a map per element of a namespace node set; the
+// maps of finished elements are reused now.
+func TestNamespaceNodesReuseAxisMaps(t *testing.T) {
+	const kids = 2000
+	var b strings.Builder
+	b.WriteString(`<r xmlns:a="urn:a">`)
+	for range kids {
+		b.WriteString(`<e/>`)
+	}
+	b.WriteString(`</r>`)
+	tr, err := xdm.ParseString(b.String(), xdm.ParseOptions{})
+	if err != nil {
+		t.Fatal(err)
+	}
+	set, err := FromXPathFilter(tr.Root, "true()", nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	allocs := testing.AllocsPerRun(3, func() {
+		if err := WriteNodeSet(io.Discard, set, Options{Algorithm: Inclusive10}); err != nil {
+			t.Fatal(err)
+		}
+	})
+	t.Logf("%.0f allocations for %d elements", allocs, kids)
+	if allocs > kids/2 {
+		t.Errorf("%.0f allocations for %d elements, want a map reused across them", allocs, kids)
+	}
+}

c14n/prefixlist_test.go

@@ -0,0 +1,22 @@
+package c14n
+
+import (
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+)
+
+// ParsePrefixList scanned its result for every token: 1 MiB of distinct
+// prefixes took 12 s of CPU. A map makes it linear.
+func TestParsePrefixListLinear(t *testing.T) {
+	var b strings.Builder
+	for i := 0; b.Len() < 1<<20; i++ {
+		b.WriteString("p" + strconv.Itoa(i) + " ")
+	}
+	start := time.Now()
+	got := ParsePrefixList(b.String())
+	if d := time.Since(start); d > 2*time.Second || len(got) < 100000 {
+		t.Errorf("1 MiB of prefixes: %d distinct in %v, want linear time", len(got), d)
+	}
+}

docs/conformance-gaps.md

@@ -49,7 +49,7 @@ W3C disagreements: 0 + 0 + 0 + 1 + 8 + 23 + 30 + 31 + 0 = 93. Measured 2026-09-1
 <!-- BEGIN GENERATED UNIT TEST COUNT -->
 <!-- Generated from tests/conformance/results.json and the source tree by
      tests/conformance-docs.go. Do not edit; see docs/stats.md. -->
-The unit-test suite is 2,950 tests.
+The unit-test suite is 2,960 tests.
 <!-- END GENERATED UNIT TEST COUNT -->
 
 The last two rows are not W3C suites but real-world corpora — DocBook xslTNG's

docs/known-gaps.md

@@ -131,11 +131,16 @@ Found by the third memory audit (2026-11-01) and not fixed, each with the reason
   (`withSel`, `hostWithCurrent`, the leaf-hook copy), and built-in shallow-copy
   copies a document-level result twice; removing that copy changes an error
   case into output.
-- **A single text value of a few to tens of MB allocates up to about 1.9× its
-  size while parsing** (10 MiB: 1.87×; the tree keeps 1.09×; 60 MiB: 1.13×).
-  The decoder's buffer grows in steps of eight. Sizing it to the input left
-  in one step reached 1.08×, but a document of eight 3 MiB values went from
-  1.33× to 2.04×, so it was reverted.
+- **Several large text values in one document cost about 1.3× their size to
+  parse** (eight 3 MiB values: 1.30×). A single value is about 1.1× at every
+  size since v2.1.1; with several, the first value's buffer overshoots once
+  before each later value is copied exactly once.
+- **An XPath expression in 1.0 compatibility mode copies its context on every
+  evaluation** when the caller's context is not in that mode (96 bytes a call;
+  12× of an XPath 1.0 transform's allocation in go-xmlsec). The mode can be
+  set only on a `Compiled` (`WithCompatMode`), and a `*Context` is updated in
+  place during evaluation, so a cached copy would be unsafe. Needs a
+  context-side setter: new API, so a minor release.
 - **`c14n.BytesNodeSet` on a caller's own `NodeSet` still costs about 2.1× its
   output.** Sizing it would call `Contains` twice per node, which the API
   promises not to do; the package's `Subtree` and `ExcludeSubtree` are sized.

docs/security.md

@@ -1865,6 +1865,10 @@ Each line names the **direction** of the defect, because that is what decides
 who was exposed: a *false accept* let an invalid input through, a *false
 reject* refused a legal one, and *cost* produced the right answer too slowly.
 
+**Twentieth pass — go-xmlsec's measurements against v2.1.0, 2026-11-01.**
+
+- **`c14n.ParsePrefixList` was quadratic in its tokens** — cost, driven by a signed document. It parses the `PrefixList` of an `ec:InclusiveNamespaces` element, which the document under verification supplies, and deduplicated each token with a scan of those before it: 1 MiB of distinct prefixes took 10.9 s of CPU. It deduplicates with a map, keeping first-seen order: about 0.02 s. go-xmlsec caps the list at 1,024 prefixes before calling it; other callers were exposed. Pinned by `TestParsePrefixListLinear` in `c14n/prefixlist_test.go`. Fixed in v2.1.1.
+
 **Nineteenth pass — a third memory audit, 2026-11-01.** The findings and figures are in the audit's own write-up; the retention and allocation fixes are in CHANGELOG under *Changed — memory*.
 
 - **Sequences built by `,` and `!` were not counted against `MaxItems`** — availability, driven by a query or stylesheet. `fold-left(1 to 26,(1,2),function($s,$i){($s,$s)})` (70 bytes) built 134M items against the 5M limit at 6.3 GB, and the same doubling through a recursive `xsl:with-param` reached 4.6 GB on v2.0.3. `SequenceExpr` and `!` charge the copies beyond their widest operand above 64 items, and `xsl:next-iteration` values are checked against the bound. Pinned by `TestSequenceCopiesAreCharged`.

docs/stats.md

@@ -23,7 +23,7 @@ claim is only as good as the command behind it.
 
 | figure | count | counted by |
 |---|---:|---|
-| Unit tests | 2,950 | `grep -rn '^func Test' --include='*_test.go' . \| grep -v '/\.claude/worktrees/' \| grep -vc '^\./bench/'` |
+| Unit tests | 2,960 | `grep -rn '^func Test' --include='*_test.go' . \| grep -v '/\.claude/worktrees/' \| grep -vc '^\./bench/'` |
 | Fuzz targets | 12 | `grep -rn '^func Fuzz' --include='*_test.go' . \| grep -v '/\.claude/worktrees/' \| grep -vc '^\./bench/'` |
 | Limit boundary tests | 14 | `grep -hc '^func Test' ./*/limits_boundary*_test.go \| awk '{n += $1} END {print n + 0}'` |
 

docs/testing.md

@@ -21,7 +21,7 @@ let something through, and the column that matters is the last one.
 
 | layer | count | catches | misses |
 |---|---:|---|---|
-| **Unit tests** | 2,950 | a plausible implementation that is quietly wrong | anything nobody thought to write a test for |
+| **Unit tests** | 2,960 | a plausible implementation that is quietly wrong | anything nobody thought to write a test for |
 | **Limit boundary tests** | 14 tests | an off-by-one or an overflow at the edge of a configurable limit | a limit nobody added to the inventory |
 | **Race detector** | same tests | shared state a single-goroutine run never reveals | a data race on a path no test walks |
 | **W3C conformance suites** | 152,241 cases | systematic divergence from the specification | what the suites do not ask about — see below |

docs/todo.md

@@ -23,7 +23,7 @@ Current position:
 | XSLT 3.0 | 99.80% — 11,495 of 11,518 in scope (23 failing); 3 of those need more of the §19.8 streamability analysis |
 | RELAX NG | 100.00% — 965 of 965 |
 | Schemas wrongly refused | 7 — 6 on XSD 1.0, 1 on 1.1 |
-| Tests | 2,950 `func Test` declarations, clean under `-race` |
+| Tests | 2,960 `func Test` declarations, clean under `-race` |
 <!-- END GENERATED STATUS TABLE -->
 Every one of those failures, and why it is still open, is catalogued in
 [known-gaps.md](known-gaps.md). This file is the forward-looking half — what

internal/xmltok/xmltok.go

@@ -45,7 +45,6 @@ import (
 	"encoding/binary"
 	"fmt"
 	"io"
-	"slices"
 	"strconv"
 	"strings"
 	"unicode/utf8"
@@ -120,6 +119,9 @@ type Directive []byte
 // stream was read through before.
 const bufSize = 4096
 
+// textFit: see Decoder.text.
+const textFit = 32
+
 // A Decoder reads tokens from an XML byte stream.
 //
 // A returned token, and the byte and attribute slices in it, alias the
@@ -1060,10 +1062,20 @@ func (d *Decoder) text(quote byte, cdata bool) ([]byte, bool) {
 			if n > 0 {
 				// A run past a window long that outgrows the scratch is
 				// sized from what is left of the input, not by append's
-				// 1.25x steps. ponytail: capped at 8x the scratch, so that a
-				// run early in a huge document takes at most that.
+				// 1.25x steps, and by an exact make and copy so that the
+				// capacity is what was asked for. ponytail: the step is capped
+				// at 8x the scratch, so that a run early in a huge document
+				// takes at most that; but when what is left is within textFit
+				// times the scratch it is taken whole, so that a lone value ends
+				// its life fitted and not a step short of it or past it.
 				if rest := d.Size - int(d.InputOffset()); n > cap(out)-len(out) && len(out) >= bufSize && rest > n {
-					out = slices.Grow(out, min(rest, 8*cap(out)))
+					grow := min(rest, 8*cap(out))
+					if rest <= textFit*cap(out) {
+						grow = rest
+					}
+					bigger := make([]byte, len(out), len(out)+grow)
+					copy(bigger, out)
+					out = bigger
 				}
 				out = append(out, run[:n]...)
 				if n == 1 {