Repository navigation
Memory audit: cut retention and allocation across the stack - #31
Merged
Merged
Conversation
fail clips every string and error argument and the formatted message to 1 KB; attrName clips before fail sees the name; the pattern-facet and list-item errors quote their value through truncate. 200 foreign attributes in a 1 MB namespace: 105.7 MB live / 422.8 MB allocated -> 0.13 MB / 0.83 MB.
A namespace URI was an arena string, and the tree keeps it in every name it resolves, so each parsed document kept a 32 KB arena block alive. Retained heap for the DocBook corpus (591 trees): 31.6 MB -> 12.3 MB.
A fresh resolver per transform left up to nestedCacheSize dead nested compiles alive (49 MB on DocBook section.001 over 20 resolvers). The key now holds the resolvers as weak pointers and put drops entries whose resolver has been collected (3.3 MB, the live resolver's entries).
walk concatenated parentPath + "/" + name at every element, which is quadratic in depth. fail now renders the path from the element's ancestors, and an IDREF keeps its element instead of a path. Text is unchanged. Valid 990-deep chain of 4 KB names: 2015 MB -> 0.00 MB allocated.
A clause binding one name onto every tuple copied the names slice per tuple, and "for" wrapped each item in a fresh one-item sequence. Tuples of one stream now share the extended names slice, replacing a binding shares the names, and "for" and the join bind a capped subslice of the binding sequence. XMark q1-q20 eval: 194.9 -> 156.7 MB.
fail kept a #FIXED, enumerated or ID value whole in every error. It now clips string and error arguments and the message to 1 KB. 200 mismatches against a 1 MB #FIXED value: 104.6 MB live / 317 MB allocated -> ~0 / 0.49 MB.
"let $l := for ..." sent the FLWOR's result through a top-level builder, which appended every item a second time. flworNode is now a valueNode and answers with the sequence itself. XMark q1-q20 eval: 156.7 -> 142.7 MB.
Every document allocated a 32 KB arena block. It was kept at that size because a smaller retained heap ran the collector more often, but the retention was the namespace-URI pin, now gone. DocBook corpus: 37.6 -> 19.6 MB a parse pass, GC cycles 13.4 -> 6.8.
A query compiled as 3.1 under a 2.0 context made every expression evaluation copy the context to install the version. prepare installs it once. XMark q1-q20 eval: 142.7 -> 133.1 MB.
isBase64Binary copied the value into a []rune, four bytes per input byte. It now makes one byte pass with a four-byte ring for the final quantum. 380 KB value: 1,540,475 B/op, 1 alloc -> 0 allocs.
The concatenated result of the return expression grew by doubling. It is now reserved for one item per remaining tuple on the first non-empty return. XMark q1-q20 eval: 133.1 -> 125.6 MB.
…nd comments The text method concatenated the whole string value before writing it (41.7 MB for 8.2 MB of output on xmark); it now writes each text node. writeCData built every section in a builder, mapSegments allocated a segment slice per text node, and comments and PIs were concatenated: all now write straight to the serializer's writer. Output is unchanged.
Every evaluation of an expression with a lifted operand built an xpath.Library map and a closure per operand. liftedLib reads the operand index out of the xq-stepN name instead. XMark q1-q20 eval: 125.6 -> 119.4 MB.
SchXslt redeclares a prefix on each CEN svrl:failed-assert, so the compiled sheet kept ~980 copies of one map, binding list and resolvers. Intern them by content per Compile, one resolver per literal element: CEN compiled 9.14 -> 6.06 MB, DocBook 9.71 -> 8.76 MB.
matchNames made a new map-backed state set for every child. It now swaps two, clearing the reused one. dtd BenchmarkValidate: 1,295,033 -> 691,383 B/op, 14,609 -> 8,112 allocs/op.
TrackPositions kept the whole document text for the tree's life only to count lines in. The tee now records where lines start as the text goes by. Tracked parse-10mb tree: 50.4 -> 41.5 MB retained.
idOwners and ids were two maps with the same keys. They are now one map
of {owner, n}. BenchmarkValidateInstance: 213,906 -> 183,403 B/op;
20,000 IDs: 3.50 MB -> 2.63 MB allocated.
…sult evalStepOver built a selection slice per input node and then copied it into the result. A forward step with no predicates now appends onto the result, which the final sort orders and deduplicates. XMark q15 eval: 345.7 -> 244.9 KB.
evalRange made a big.Int and a big.Rat per value, and NewIntegerFromRat then narrowed each back to int64. Bounds that fit in int64 now loop as int64. distinct-values(1 to 100000) benchmark: 96.0 -> 76.8 MB/op.
xsl:element, xsl:attribute and xsl:message/@error-code kept their stylesheet element only for LookupPrefix, which pinned every imported module tree. They keep the shared in-scope map instead: DocBook compiled (own resolver) 12.36 -> 9.31 MB.
A text value of 8 KB or more was built in the decoder's buffer and copied into a block of its own. The buffer becomes the block when it is at most 1/8 larger than the value. E-invoice corpus: 4.10 -> 3.49 MB allocated a pass.
Every exact value kept also wrote the double and float keys it promotes to, though only a float or double looks them up. They are now written when the first one arrives. distinct-values(1 to 100000) benchmark: 76.8 -> 53.2 MB/op.
checkBounds parsed every min/max bound into a big.Rat for every value. The chain memo now carries the parsed bounds. BenchmarkValidateInstance: 183,403 -> 163,398 B/op, 5,036 -> 4,036 allocs/op.
Text blocks double, so a parsed tree's last one was on average a quarter empty. It is copied to its length at freeze. DocBook corpus: 12.29 -> 12.03 MB retained.
The name and namespace-frame indexes serve only building, but a parsed tree kept them. They are dropped at freeze and rebuilt by a later rename or clone. DocBook corpus: 12.03 -> 11.07 MB retained.
evalParams built a normal and a tunnel map per call, the tunnel one even when empty: 3 allocations for two parameters. A slice of pairs is 1; DocBook section.001 transform 20.1 -> 19.0 MB allocated.
Both methods built the whole document as a string before writing it, 7.5-12.6x the output size, and a node inside JSON was serialised to a string and then escaped into a second one. xpath gains SerializeJSONTo and SerializeAdaptiveTo, which write through the serializer's writer; a JSON node is escaped as it is written. Adaptive output under normalization-form keeps the string, because norm's streaming Writer panics on long combining runs split across writes. fn:serialize's text and attribute escaping and prefixed names now write to the sink in pieces instead of building a string per node.
functionCallKey built a Clark name, a growing strings.Builder and fmt boxes on every deterministic call. It appends into a per-transform buffer and looks up via m[string(b)]: a memo hit 19 -> 13 allocations, DocBook section.001 19.0 -> 17.9 MB allocated.
Bytes grew a bytes.Buffer from empty, 3.7x the canonical form at 10 MB, and Equal paid it for its first operand. Bytes now keeps the writer's 64 KiB chunks and joins them once: 37.8 MB -> 21.8 MB for 10.2 MB.
Both grew a strings.Builder by doubling from empty, about 5x the output. They now keep the serializer's 32 KiB flushes and join them once: 2.5 MB -> 1.0 MB for 481 KB of output.
Changelog rows, the nineteenth security pass, the deep-equal bound, the MaxDigits and CLI GC notes, the open items in known-gaps, and regenerated test counts.
…InScopeNamespaceBindings
…g in one step (99x to 38x at 10 MiB)
The comparison hoist now lives on the stack and an ancestor step collects its chain in one block of four instead of growing 1, 2, 4.
…llocates 1.08x not 1.87x" This reverts commit 28d41c8.
Changelog rows for the node index and namespace binding APIs, the c14n buffer and sizing, many-attribute tags, namespace nodes, accumulator slices and predicate steps; the reverted long-text change and caller node sets in known-gaps; the WithVersion advice; regenerated test counts.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the findings of the memory audit across parsing, XPath/XQuery, XSLT, validation and serialization. Successful output is byte-identical; the full table with figures is the new
## Unreleasedsection in CHANGELOG.md.Highlights
TrackPositionskeeps line starts, not the source text.xsd/dtdmessages clip document strings to 1 KB (105 MB → 0.13 MB retained); the DTD path is no longer quadratic in depth (2 GB → ~0); thefn:transformcache drops compiles of collected resolvers (49 → 0.3 MB).distinct-values(1 to 100000)96 → 53 MB.BytesandResult.Stringno longer double buffers.Round 2: a second audit of this branch
Two defects that v2.0.3 also has, both reproduced on the CLI:
serializeanddeep-equalon a 400,000-deep array overflowed the Go stack. Now refused with a resource-limit error.shallow-copyreturned a truncated tree with exit 0. A 1M-element copy emptied a third of its elements. Now XPDY0130.Also:
\pL{1000}repeats are bounded (181 KB of patterns had held 984 MB).replace(), picture widths, andxsl:key/group-bykeys.FileResolverdocument cache.xsdnettransport reuse.array:appenddouble copy, slack trims.Still open:
fn:replaceon the backreference and backtracking engines is charged after building (docs/known-gaps.md).Round 3: a third audit
Four regressions in round 2's own fixes, now corrected:
deep-equalandmap:findwere refusing values thatparse-jsonaccepts.replace()over-charged.Three defects that v2.0.3 also has, each reproduced on the CLI before and after:
with-param.xmllintaccepts it). It now validates.1 to 100000took 21.6 s and 6.95 GB. Now 0.056 s.Also:
unparsed-text-lines,parse-json, DTD defaults, integer digits (Env.MaxDigits), accumulator values andcopy-of.$f($i)over 1M went from 2.7 to 1.6 GB.fn:docstreaming.GOGC=100past 128 MB live.What round 3 left open is in docs/known-gaps.md.
Round 4: go-xmlsec's report
All seven items from go-xmlsec were verified on v2.0.3 and on this branch, then fixed:
Digestcalls allocate 121 KB instead of 8.4 MB;Bytesand package-made node sets are sized from the tree, at about 1.1x instead of 2.1–3.6x.Node.InScopeNamespaceBindings.Node.Index,Node.SubtreeEnd,Tree.LenandTree.NodeAt. Accumulator values now sit in slices built on that API, about 27% less per node.docs/options.mdnow has theWithVersionadvice for hosts that reuse one context.The one-step long-text sizing was reverted. It cut a single 10 MiB value from 1.87x to 1.08x, but raised eight 3 MiB values from 1.33x to 2.04x. A 60 MiB value is already 1.13x on this branch. See docs/known-gaps.md.
API and behaviour
xpath.SerializeJSONTo,xpath.SerializeAdaptiveTo,xdm.Node.AddNamespaces, so the next release is a minor one.xsd/dtdmessages longer than 1 KB now end in "...".Not done
xs:keyrefrework (high risk, out of scope).normalization-formkeeps the string path:x/text's normalizing writer panics on long combining runs split across writes.Verification
func Test(2,950).tests/check.shon the merged branch.