Repository navigation
Memory footprint round and streamed record execution (v2.2.0) - #35
Merged
Merged
Conversation
…; unfiltered loops skip select
Rewinding the text store between bursts left the shared whitespace table pointing into released blocks: whitespace-only attribute values read other bytes, or the parse panicked. A path predicate whose last step is not an axis step can be numeric, so positional; the gate no longer admits it. A document that fails to parse in bursts no longer counts as read for XTDE1500, as in tree mode.
# Conflicts: # xdm/bursts.go
CHANGELOG Unreleased; known-gaps: what the round left open and the one streamed shape; options: FileResolver.MaxNodes and MaxBytes reaching the parser; profiling: paired chunks against 508/511; test count 2,976.
… is pending (M4, L1)
… tree does not pay a kilobyte (M3)
…and small trees still share whitespace (M3)
ParseBursts cloned the parse tree's indexes into every burst, and they grow with the stream, so CPU was quadratic in distinct names and namespace declarations. intern and internFrame rebuild them on use.
Every burst copied them, so the cost was prolog x records. No burst body can reach the document node's other children.
The parse's name and frame tables grew with the whole stream. Past 1024 new ones beyond the spine they are rewound like the text store, clipped so the bursts that share them keep reading theirs.
deliver copied the whole parse tree while it was alive, so one large record peaked at twice the tree. The chunks past the record's first are moved into the burst's tree; truncate reaches its tables by index.
fn:copy-of, fn:snapshot and xsl:copy-of copy-accumulators=yes recorded every copied node against its original in rt.accumOrigin for the whole transform, pinning every copy and every streamed burst tree. The table is only read by accumulator-before/after, which cannot resolve a name when no package declares an accumulator (all packages file into Stylesheet.accumulators), so it is now nil then. Streamed copy-of(.) per record at 10 MB: 830 MB -> 2.8 MB peak heap.
The analysis takes several declarations at their word: a declared-streamable function by its category, a streamable attribute set, an accumulator named by accumulator-before/after, the rules of a streamable mode, and anything another module can override. Each declaration's own check stays silent when it meets an unmodelled construct, so a container could be known and grounded while resting on a declaration nobody verified. An unverified bit now travels beside known through every analyzer, set at each of those trust points unless a declTrust verifies the declaration on demand (memoised; in-progress counts as verified, per §19.8.5). Refusals read known only, so XTSE3430 results are unchanged (TestXSLT30Suite 11495/23). The burst gate now assesses the body with the module's declarations and requires known && !unverified.
….4.24) xsl:map-entry was skipped as if it were an operand of its parent, so one inside an xsl:for-each or xsl:if within xsl:map was never assessed and a streamed node could be bound into the map. It is an instruction, and is now analysed as one. TestXSLT30Suite 11495/23, unchanged.
…19.8.4.2) An element in an extension namespace was analysed as a literal result element, whose excluded xsl:fallback children were never looked at -- yet fallback is what this processor runs. Its operands are now the fallback sequence constructors, usage transmission. TestXSLT30Suite 11495/23, unchanged.
….8.4.35) The rule's comment said every serialization attribute was treated as an absorbing AVT operand; the code read href alone, so a consuming expression in item-separator, say, was a second read of the stream that went unseen. TestXSLT30Suite 11495/23, unchanged.
…treamed node §19.8.8.16 grounds every inline function that names no streaming parameter. Two closures escape that test: one over a range variable this analysis's quantified-expression data flow bound to streamed nodes, and one calling fn:current(), whose value is the streamed item current at creation. Both are now unknown -- withheld, not refused. TestXSLT30Suite 11495/23, unchanged.
….10) §19.8.10 calls a predicate positional when its static type is numeric or it calls position(), last() or function-lookup(); the classifier's numeric test is syntactic. Its false positives on comparisons were withheld from refusal, which let rec[following-sibling::rec/@k = 3] through unjudged in a rule pattern or a group-starting-with. A boolean-typed predicate calling none of the three is now non-positional and takes the motionless test like any other; the withholding goes. TestXSLT30Suite 11495/23, unchanged.
§6.6.4 applies the streamable-template conditions to every rule that includes the mode, "including by specifying #all". templateModeNames names modes and leaves #all out, so such a rule was never checked, and a comment said that was deliberate. Both the pattern and the body checks now include it. TestXSLT30Suite 11495/23, unchanged.
… (§6.6.4) §6.6.4's third condition on a template rule of a streamable mode is that every expression in a contained xsl:param is motionless. It was not checked, so a default reading a following sibling compiled. TestXSLT30Suite 11495/23, unchanged.
Three of the m10 reproducers rewritten as the one record loop the burst gate plans for, where its whitelist is the only other guard: a nested xsl:map-entry is refused, and the two closures over a range variable are unknown and stay on the tree path.
…nstructs now judged
…(stage 1b(i)) A streamable xsl:source-document whose body is one xsl:value-of, xsl:copy-of or xsl:sequence reading one record path now runs in bursts: the path is evaluated per burst and the rest of the select once over all the items, with count/sum/min/max folded per burst. Bounded and pipelined consumers only; distinct-values, reverse, sort, last(), union/intersect/except and user functions stay on the tree. Any error re-runs the instruction on the tree, so errors are the tree's.
tail(doc/rec)/@v and copy-of(doc/rec)/x put their result in document order, which across bursts is the order they were made in; refused when the record path itself made copies, which bursts make earlier than the tree would. fn:unordered returns its argument unchanged here, so it pipelines.
…izes them A record path kept as nodes holds each burst's tree, about 950 bytes a record against 208 in the tree: avg(doc/rec/@v) over 100 MB peaked at 3.8 GB heap against 1.3 GB on the tree. Where the path's own consumer atomizes it (aggregates, string functions, comparisons, arithmetic, casts) the items are atomized as each burst is read: 609 MB.
A burst node holds its burst's whole tree, 945 bytes a record against 208 in the document's tree, so copy-of(doc/rec), filters, simple maps and head/tail over the path's nodes used more memory streamed than on the tree. The gate now admits a select only where the path's items are folded, atomized by their consumer (or by xsl:value-of of the bare path), or are values or copies already; the path steps from items this made unreachable are removed.
…h made doc/rec/copy-of() kept to the end is one tree per record: at 100 MB it peaked at 4.2 GB RSS streamed against 4.0 GB on the tree. Such copies are admitted only where they are folded or atomized.
…ttributes xsl:value-of joins adjacent text nodes without its separator, so atomizing doc/rec/x/text() per burst put a space between records (si-value-of-009). Attributes are as good as their values to it; other node paths stay on the tree.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Memory footprint work for v2.2.0: what a parsed tree holds and what a transform peaks at. Output is byte-identical; the new API is additive. The full list is in CHANGELOG.md under "Unreleased".
Memory
//namespace::*on 5 MiB: 140 MB held → under 0.5 MBStreaming
xsl:source-document streamable="yes"now runs record by record for two shapes:xsl:for-eachorxsl:iterateover a child record path;xsl:value-of,xsl:copy-oforxsl:sequenceover one record path.count,sum,minandmaxfold per record.New API
xdm.ParseBurstsxdmbuild.InPlaceDocumentxslt.FileResolver.MaxNodesBehaviour changes
FileResolver.MaxBytesis now also the parser's byte limit, so raising it past 64 MB takes effect forfn:doc.Reviews
Three reviews ran against this branch, and their findings are fixed:
A differential fuzzer compared burst and tree execution:
The remaining error-order class is documented in docs/known-gaps.md.
Verification
sh tests/check.shat 1337974: every lane that ran passed. UBL and CII are skipped (licensed corpora).Race, 32-bit vet and the
w3cschemasmodule pass.