Repository navigation
feat(org): mae-org parser, semantics, renderers and WASM build (ADR-112 O1–O3) - #853
Merged
Merged
Conversation
…erives from MAE parses org in four places that disagree (KB ingest, export, the highlighter, read mode), and editor commands munge org text ad hoc. Both structure parsers accept indented stars as headings, and export publishes :noexport: sections and comment/verse/special-block content. The decision: mae-org (shared/org) — a lossless rowan tree, no I/O, WASM-capable — with MAE's KB semantics as one projection of it; the parse depends only on the text and declared configuration, so every peer and the daemon derive the same graph; Emacs-level flexibility through a per-user Scheme layer and, later, owner-signed KB-carried derivation; conformance gated on Emacs org-element with a pinned divergence register, round trip, linear scaling and fuzzing. The research (docs/research/112-*) evaluated nine existing parsers hands-on against MAE's corpus, Org's own repository and an adversarial set; none passed a maintenance gate, and none was close to linear on hostile inline input.
…2 O1)
New crate shared/org (mae-org, GPL-3.0-or-later), an editor-workspace
member. Its only dependency is rowan 0.17; it does no I/O and builds for
wasm32-unknown-unknown.
parse(text, &ParseOptions) -> Document is total and lossless: every byte
belongs to one token, so the tree prints back to the input exactly (CRLF,
BOM, no final newline, any malformed input). Inputs over u32::MAX bytes
give an empty document with a diagnostic.
Block structure follows org-element (Org 9.7.11), with element extents in
its representation (affiliated keywords and trailing blank lines inside the
element): headlines (column-0 stars + space), sections, planning, property
and generic drawers, greater/lesser/special/dynamic blocks, keywords,
affiliated keywords, babel calls, comments, fixed width, rules, footnote
definitions, plain lists with items, org and table.el tables, clock lines,
diary sexps, LaTeX environments, paragraphs. Inline content is one TEXT
token per line until O2.
A block or drawer exists only if its closing line comes before the end of
its container (for a section, the next headline); otherwise its opening
line is paragraph text plus a diagnostic. Every such forward search is a
binary search in an index built in one pass, so the parse has no rescans.
Containers are an explicit stack capped by ParseOptions::max_depth; deeper
contents become one opaque token with a diagnostic, never recursion.
TODO sequences are data: ParseOptions::todo_sequences, replaced by in-file
#+TODO/#+SEQ_TODO/#+TYP_TODO keyword elements wherever they appear.
Typed views: Headline (level, todo, priority, COMMENT, title, tags,
property drawer), PropertyDrawer::properties (canonical upper-case key,
spelling as written kept in the tree), Block (kind, name, parameters, raw
contents), Keyword (key, value). Diagnostics are {span, kind} data.
docs/CODE_MAP.{json,md} regenerated for the new workspace member. The
lockfile adds rowan, countme and text-size only.
…ister conformance/org-element-dump.el is the oracle: emacs -Q --batch emits each element's type and byte offsets (position-bytes, file read as utf-8-unix). It is self-checking: a first line records the buffer's byte count and whether org-mode left the buffer equal to the file, and the harness rejects a golden that fails either. org-inhibit-startup is bound because #+STARTUP: align realigns tables in the buffer (found on Worg), and setup files are not read because mae-org does no I/O. conformance/regen.sh runs one Emacs over conformance/corpus.txt and writes the checked-in goldens (CI needs no Emacs): all 35 synthetic fixtures and 40 bundled-corpus files chosen for element-kind coverage. tests/conformance.rs compares kind/begin/end per element through an explicit kind table. Unlisted differences fail, and so do register entries that no longer match anything. The CRLF and BOM variants of every LF file must match the shifted LF golden with no exceptions. The typed views must report the oracle's headline, keyword, node-property, drawer and special-block properties. conformance/divergences.toml lists two entries, both decoding: a fixture stored with a BOM and one stored with CRLF. The oracle reads raw bytes, so it sees the BOM and CRs as content. mae-org decodes them the way Emacs does for a user, and tests/eol_bom.rs pins that. tests/oracle_external.rs (ignored) runs the oracle live over any directory from MAE_ORG_ORACLE_DIR and prints agreement per kind without writing anything to the repository. Pure-CRLF files are compared through their LF form. The fixtures are synthetic adversarial cases written for this crate: column-0 headings inside src/example/quote blocks, unterminated blocks and drawers, empty file, lone star, BOM, CRLF and mixed EOL, #+TODO variants, list-item drawers, deep lists and blocks, affiliated keywords and orphans, property-drawer placement, footnotes, tables, bracketed keywords.
roundtrip.rs: every .org file under assets/ and tests/ (walked at test time, 340 today) and its CRLF variant; every synthetic fixture as stored, as CRLF and with a BOM; generated inputs (20k headings, 2000 nested blocks of both kinds, a 5000-deep list, a 1 MB line, `*a `x20k, `[[`x50k, newline-only and CR-only files). Each tree must print back exactly and pass support::check_tree: children tile their parent, no empty tokens or nodes, element kinds only under parents Org allows, nested headlines deeper than their parent, depth within the cap, and no UNPARSED tokens. properties.rs (proptest): 4000 org-shaped inputs built from the fragments that decide structure, 2000 arbitrary Unicode strings, and 2000 corpus files with fragments spliced in. Each case checks for no panic, the invariants above, determinism, a lossless tree under max_depth 3, and configured TODO keywords. The regression seeds are shrunk cases that broke deliberately mutated code. rules.rs pins the rules that move KB node boundaries one by one. The oracle-confirmed ones: `*\tB` and `*D` are paragraphs, ` * C` is a list item, and a column-0 headline ends an open src/example/quote block or drawer, whose opening line becomes text with a diagnostic. It also pins #+TODO from the file versus the options, affiliated keyword placement, property-drawer placement, list-item drawers, the nesting cap and diagnostic spans. scaling.rs: 14 hostile families timed at N and 8N. The release-only assertion bounds growth per input byte at x24; linear is about x8 and quadratic x64. The worst measured is x13.5. The two rescans that were quadratic (table.el candidates, runs of orphaned affiliated keywords) fail it when reintroduced.
… contents
Fixes the eight findings of the independent review of mae-org O1. Each is
reproduced by a failing test or a golden first, confirmed against Emacs, and
mutation-checked.
1. TODO matching is linear. The keywords in force are a hash set and a
headline's candidate is its first space-delimited word. Before, every
keyword was tried on every headline, x4 per doubling (14.3 s at 1.1 MB).
tests/scaling.rs gains both review shapes: N keywords + N headlines, and
N `#+TODO:` lines.
2. Line endings are decided once per document, as Emacs's decoder decides
them (measured with find-file over the whole file): any LF gives DOS iff
every LF follows a CR, otherwise unix, where every CR is content; CRs with
no LF give mac. LineEnding is public. The oracle now decodes the way
find-file does (utf-8-auto: EOL detection, BOM stripped) and maps offsets
back to the stored bytes. Its self-check reports the convention, which
must equal LineEnding::detect. The BOM and CRLF divergences are gone.
This also fixed the zeroth section's start: org-skip-whitespace treats a
content CR as blank.
3. Character classes and case folding come from Emacs.
conformance/gen-char-tables.el asks Emacs 30 inside an org-mode buffer
and writes src/chartables.rs:
- whitespace syntax: 8 ranges (NBSP, U+2003, U+3000, ZWSP and U+202F
are whitespace; VT is not);
- word syntax: 113 ranges;
- [[:alnum:]]: 761 ranges;
- the canonicalize table: 1452 pairs (ß~ẞ and σ~ς fold; Kelvin, ſ and
ı do not).
It fails if any non-ASCII char folds or upcases to ASCII, which is what
keeps ASCII-only matching of literals such as #+BEGIN_ exact. Block and
LaTeX end names are indexed by the canonical fold, tags use
[[:alnum:]], and property keys canonicalise the same way. The script
also writes per-class fixtures, and unicode-case-negative.org was added
by hand.
4. TodoSequence follows org-set-regexps-and-options:
- values split on ASCII whitespace only;
- a name is the shortest prefix whose remainder is empty or `(...)`;
- an empty or `|`-only declaration still replaces the defaults.
Tags keep interior empty tags, as org-split-string does.
5. The differential compares contents_begin/contents_end and each
element's parent as well as kind/begin/end. Document::contents exposes
org-element's contents ranges. Two real differences fell out:
- a blank-line paragraph's contents end;
- indentation width: Org uses string-width, where every tab counts 8,
not tab stops (list-tab-width.org).
The eight surviving mutants' inputs are now golden fixtures, along with
affiliated-names.org (which kills the RESNAME mutant).
6. Inside a list, Emacs reads a block name as a regexp (no regexp-quote
in org-element--list-struct). mae-org reads names literally, as Org's
block parser does. This is registered as a divergence with a pinning
test: matching it needs an Emacs-regexp engine over author text.
7. max_depth is clamped to MAX_DEPTH_CEILING (1024). The parser never
recursed (10k-deep input parses on a 256 KiB stack); rowan's Drop and
PartialEq recurse once per tree level. At the ceiling, 5,000-deep
blocks and 2,000-deep lists and headlines parse, compare and drop on a
1 MiB thread.
8. API:
- SyntaxKind is #[non_exhaustive] with pinned values (tokens from 0,
nodes from 1000; a test pins every number). DiagnosticKind is
#[non_exhaustive].
- Diagnostics and contents use the crate's own Span. rowan is
re-exported as `mae_org::rowan` rather than re-exporting TextRange
and TextSize.
- parse_fragment(text, options, level) parses a headline body exactly
as it parses under a level-L headline in a document. That is checked
over every fixture and bundled file at levels 1-3 and by proptest.
A headline that would end the body is a diagnostic.
The oracle also turns an Emacs error on one file into a #error line
instead of aborting the batch.
Fixes the re-review findings for bbc08f0. 1. parse_fragment takes a FragmentContext { level, line_ending }. The line-ending convention belongs to the whole file, and a fragment cannot detect it: a CRLF body under an LF headline is a unix file, in which `#+end_src\r` closes nothing. A body never starts its file, so a leading BOM is content, except at level 0. tests/fragment.rs now chooses the headline's line ending and the body's independently (LF, CRLF, mac and BOM variants of every fixture under all three header endings, plus 3000 proptest cases), and pins both review repros. 2. Org's affiliated-keyword collector orphans its keywords before ANY `^[ \t]*CLOCK:` line (org-clock-line-re, any case), not only before a well-formed clock line. New fixtures: clock-prefix-orphans.org and clock-prefix-lower.org. 3. Headline titles and keyword values are org-trim'd: a content CR (a unix file with CRLF lines, or `\r\r\n` in a DOS file) is not part of them. Node-property values keep it, as Org does. The typed-views gate no longer skips files with a CR or a BOM. Its comparison moved into tests/support, so oracle_external now compares views too. Across bundled, Org/Worg, fixtures and the reviewer's 5,000 generated and EOL documents it finds 0 field mismatches in about 32k compared elements. Two harness issues it surfaced are fixed: - Headline::level is the star count (Org's :true-level, which decides structure); the oracle now emits true_level for the comparison. The reduced :level under `#+STARTUP: odd` is documented as not modelled. - The drawer-name lookup took the first child node, which is an affiliated keyword when one precedes the drawer. 4. gen-char-tables.el's output is now byte-identical to the committed chartables.rs (no trailing blank line), so regenerate-and-diff works.
Checkpoint for parallel O2 work; the conformance gate is deliberately red
until every object parser lands (the goldens now include objects).
- Object kinds are pinned from 2000 (tokens MARKUP=28, LINK_PATH=29;
nodes HEADLINE_TITLE, ITEM_TAG_CONTENT, AFFILIATED_VALUE).
- src/inline: org-element--parse-objects without recursion (an explicit
stack, capped like element nesting). The lexer follows
org-element--object-regexp, and dispatch follows
org-element--object-lex's order and restriction guards.
- Emphasis uses an index of closers per mark. It is region-aware, so the
closer search Emacs repeats per candidate is a lookup.
- Bracket, angle and plain links follow org-link-make-regexps, with link
types as declared configuration (ParseOptions::link_types; the default
is emacs -Q's set).
- The oracle dumps objects ('object granularity, link type, path,
raw_link and format) and reports a secondary-string parent for
CAPTION values.
classify_link reads a link path in one pinned order: Org's path normalisation, then the scheme (id: stripped; http/https, mailto and a fixed exclude list of non-KB schemes; everything else is a KB link), then the [[T|D]] store form, then the ADR-030 query, then the fragment. The tree, not a text search, splits a bracket link's path from its description. Tests reproduce the three split orders MAE had (mae-kb's projector, the exporter, the help renderer) on the inputs where they disagree, and property-test the query: any key order, unknown keys kept in order, malformed and non-finite values defaulted, the link never dropped. semantics() projects the tree to KB nodes (file, headline, list item with its own property drawer), plain-text titles, tags, aliases, kind, canonical properties, typed links and transclusions, reading rendered regions only. An id or link found only in a comment line or block, an export block, a non-property drawer, a COMMENT or excluded subtree, or an unreferenced footnote definition yields a diagnostic and nothing else. Links belong to the innermost node; a repeated id is diagnosed, not a second node. body_span is the body alone, ending at the first child headline.
semantics() is compared with parse_org_multi_result over the bundled corpus (339 files) and 14 synthetic fixtures. Each difference gets a class from a mechanical check that it is an intended change (child links counted for an ancestor, an id in an unrendered region, plain-text titles, canonical property keys, ...) or stays unclassified; the register conformance/semantics-divergences.toml must cover every difference, every entry must still match one, and every entry names a pinning test that must exist. A second test injects a changed title, property and link source and checks they surface as unclassified. On the bundled corpus the node sets agree (338 nodes); 1124 of mae-kb's 1269 edges agree and the other 145 are links in table cells, pending the table-cell object parser; the rest are 89 property-key spellings, 18 titles read as plain text and 2 transclusion lines inside an example block. mae-kb is a dev-dependency only, for this test; the test moves into mae-kb when mae-kb depends on mae-org (O4).
…nce corpus tests/fixtures/kb/collabtest/eor-instance.org is the KB instance marker :kb-register writes (its .gitignore says never commit it). It was listed in corpus.txt with a checked-in golden, so the gate read a file that exists only on the machine that registered the KB: in a fresh checkout or CI the differential panicked on the missing file before comparing anything.
….7.11 Implements the O2 stubs (ADR-112 D1): table cells, citations and their references, entities, export snippets, footnote references, inline babel calls and src blocks, LaTeX fragments, line breaks, macros, radio targets, statistics cookies, sub/superscripts, targets and timestamps, plus radio links. Every regexp is transcribed by hand from org-element.el, org.el and ol.el of the installed Org (9.7.11); there is still no regex crate. Linear by construction. Every forward search a parser makes is a lookup in InlineIndex, built in one pass: positions per needle ($, $$, \), \], @@, )}}}, ;, ] or >, >, ), @key, and the stops of call_/src_ names), and the scan-lists pairing of every [, ( and { with its nesting depth (which is also the org-match-sexp-depth limit of sub/superscript braces). Radio links are found once per document by an Aho-Corasick automaton over the case-folded text with whitespace runs as one symbol, which for targets whose only whitespace is spaces IS org-target-link-regexp; targets with other whitespace are checked by a memoised backtracking match. The lexer keeps a per-frame cursor into the matches, so no candidate is examined twice by one frame. Objects can now have several parsed regions (a citation's common prefix and suffix are objects of the citation, outside its contents), so an object is a list of parts, and a contents range is recorded even when empty ([fn::], _{}, an empty table cell), as Org reports it. Past the nesting budget objects are plain text, so a tree never exceeds the cap. Found by the external differential (Org's repo + Worg, 339 files, and the bundled assets, 336 files: now every element and object agrees): - org-mode loads org-modules on its first call, which registers id, info, doi and nine more link types; the oracle always parses after that, so DEFAULT_LINK_TYPES is now that list of 23 (was the 11 seen before any Org buffer exists). - A plain link cannot end in `-`: [^[:punct:] \t\n] excludes it. - An item tag is greedy `(.*)[ \t]+::`, so only ONE blank before `::` is outside the tag. - Plain and angle links read a CRLF or CR line ending as one newline. Generated from Emacs, byte-identical on regeneration: - gen-entities.el writes src/entities.rs, the 412 names org-entity-get accepts (\_ with 1 to 20 spaces included). - gen-char-tables.el also writes DELIM (\s. \s( \s) \s"), ALPHA ([[:alpha:]], rx letter) and LINEBREAK (category |).
A mutation run over the semantics layer (54 mutants) left three rules unpinned, all in the list-item drawer extension: its placement, its name and the canonical spelling of its keys could each be broken with every test green. A new test breaks each condition in turn on the same item. The proptest seeds the mutants shrank to (empty store forms, NaN after a valid weight, an empty rel) are kept. Two fixes found on review: - referenced footnotes are found with a worklist; the fixed point rescanned every definition per round, quadratic on a hostile chain. - a link path's CRLF and lone CR are line breaks, as in the Emacs buffer, so a mac file's multi-line path normalises like the others. OTHER_SCHEMES carries an @ai-caution: it is an exclude list on purpose.
Fixtures. Nine invented files put every object kind under the Emacs differential, citations included (the public corpora have none): emphasis in every container and at region boundaries; links of every format and type (escapes, multi-line paths, file+app::search, id:/info:/doi:, #+LINK abbreviations); radio targets and links (case, whitespace runs, line breaks, one char at a line start, CJK borders, tab and no-break-space targets, a target containing markup, inside other objects and table cells); entities (\_ families, there4/sup/frac) and LaTeX fragments of every form; timestamps (ranges, repeaters, warnings, diary, mismatched brackets); table cells; inline src blocks and babel calls; macros; export snippets; footnote references of all three forms; statistics cookies; sub/superscripts (brace depth 3 and 4, parens, *, line start); line breaks; targets; and an adversarial file of unclosed openers, storms, nesting and objects at region ends. One file is stored with CR line endings; the LF ones are also checked as CRLF and with a BOM by the existing variant test. No divergence is registered: every element and object of the 118 goldens agrees with Org 9.7.11 except the two list-block-name entries already in the register. Writing them found: a CAPTION's dual value (#+CAPTION[short]: long) is parsed for objects too, so it is an AFFILIATED_VALUE node like the main value; and the variant test shifted the "begin" of a secondary-string parent, which has none. Mutation testing (each rule broken, a test must fail) found two rules no golden pinned, now covered: a $ right after a $ starts no fragment, and `_` tries underline before subscript. It also showed the radio search bound can be exact in chars (no byte rounding) and that the per-frame cursor needs no per-candidate bookkeeping. Link view. Link (src/views/link.rs) reads a LINK node and reports what org-element-link-parser computes: :type, :path, :raw-link, :format, :search-option, :application and :type-explicit-p, plus the description (the link's contents). Bracket links are unescaped (org-link-unescape), have each line break and its blanks read as one space, expand #+LINK abbreviations (org-link-expand-abbrev: %s, %h, appended tag, the last definition of a key wins), and are classified as file (absolute or ./ ../), a registered type, coderef, custom-id or fuzzy; angle links drop line breaks from the path; file links split file+APP and ::search and fold `\`///*\(.:\)?/`. Line endings read as one newline, so a CRLF file gives the same view as its LF twin. A bracket link's type depends on the registered link types, which the tree does not record, so Link::info takes them; LinkAbbrevs::of collects a document's abbreviations once for bulk use (Link::info_with). compare_views now checks link_type, path, raw_link and format for every link of the goldens (410, with a floor), and builds its span index once per document instead of searching the tree per golden element. Over Org's repository and Worg (339 files, 18,821 views including ~5,600 links) and the bundled assets there is no field mismatch. tests/inline.rs pins what the goldens cannot show: objects nested 5,000 deep (emphasis whose closers sit at region ends, and inline footnotes) parse on a 1 MiB stack within the nesting cap; objects past the default cap are plain text; a timestamp Org itself cannot read keeps its regexp's extent; and the link view on inputs chosen to break it, each checked against Emacs where Emacs can answer.
Twenty-two hostile inline families join gate 3 (bound x24 per byte,
release-only timing): =/~ storms, emphasis openers with no closers,
plain-link-like text, a long backslash run, and per parser an unclosed
shape repeated on one line ({{{a(, $, <1, [1, unclosed timestamps,
[fn:a:, src_x[, call_x(, @@A:, <<<, \alpha, ^{ and _{, [cite:@, \( and a
long table row), plus three radio-link shapes: many short links; a
5,000-word target whose prefix matches the whole text, which a regexp
match attempt per position makes quadratic; and objects before a run of
rejected candidates, which pins the per-frame cursor (without it the
family measures x60).
Every inline family measures x8.0-x8.7 (linear is x8). The measured table
in the doc comment is refreshed, now with every O2 parser running.
Brings in classify_link, semantics() and the differential against mae-kb's org.rs, built in parallel with the inline object parsers. Changes made to integrate the two: - Link paths have one normaliser (principle #8). link_grammar's normalize_path follows Org exactly on the buffer form: line endings as \n, and a content CR kept, as Org keeps it. The Link view now uses it instead of its own copy of unescape. semantics() classifies the buffer form (Link::path_text) and still reports `path` as written. A path split over lines is several LINK_PATH tokens; it was read as its first line only. - The `pending_table_cell` register entry is removed: table cells now parse and their links agree. The tests that waited on table cells and footnote references are un-ignored and pass. - New register class `plain_link`, pinned by plain_id_links_are_links: Org registers `id` (org-modules), so a plain id:x in prose is a link, where mae-kb read only bracket links. - Corpus walks skip gitignored files. A :kb-register instance file had made the corpus differ between machines. - green::objects (used for #+title values) is built with the merged InlineIndex and no radio targets.
…vention - the_bundled_corpus_is_exactly_the_tracked_org_files: the walk must equal `git ls-files` (a gitignored :kb-register instance file had made the corpus differ between machines). It kills a mutant that removes the filter. - a_multi_line_path_classifies_alike_in_every_convention: in DOS and mac files a path split over lines is classified from Emacs's buffer form, while in a unix file a content CR is kept. It kills a mutant that classifies the path as written, which survived every other test.
…does On some inputs org-element never terminates: `<<<x>>>\n\nx[[x]]` makes it allocate without limit (an unbounded oracle reached 16.6 GB earlier and the OOM killer took the user's editor). regen.sh and the external oracle ran one unbounded Emacs over the whole corpus. Both now run files in chunks under a virtual-memory cap (ulimit -v, through sh, no unsafe pre_exec) and a time limit (timeout, else gtimeout, else none, chosen by command -v), and retry the files of a chunk that does not finish one at a time. A file that dies under the caps gets a #nonterminating golden, counted as "oracle: Emacs did not terminate" rather than aborting the run. Out of memory is read from Emacs's message: measured, the exit status is 3, 1 or 255 depending on where exhaustion strikes. A preflight on a trivial file rules out a broken dump script being read as nontermination. The dump bound coding-system-for-read to utf-8-auto, which decodes a file with non-ASCII text before its first CRLF as unix where find-file sees dos. It now lets insert-file-contents detect the coding as find-file does, with UTF-8 preferred. All 215 fuzz files the reviewer's oracle rejected now pass the self-check; the 118 checked-in goldens are byte-identical.
…linear `timestamp.rs::repeater` rescanned the digits before the first `>` on the line once per candidate. Candidates sharing one far `>` behind a long digit run (`"<1-1-1"*M + " " + "9"*6M + "d>"`) made that quadratic: the new scaling family measured x56.5 per 8x bytes before this change, x8.0 after. The run before each `>` is now counted once when the inline index is built; runs before distinct `>`s are disjoint, so that pass is linear. A run reaching back before the third number fails the `+` bound exactly as the rescan, clipped to the third number, did.
…ripts mae-org read `\<` as "a word char not preceded by a word char". Emacs's regex engine also finds a word boundary BETWEEN two word chars when WORD_BOUNDARY_P holds (category.h): not both below 256, and word_boundary_p (category.c) says so from the chars' `char-script-table` entries and `word-combining-categories` / `word-separating-categories`. So `参见https://example.com`, `яhttp://a.b`, `ａhttp://…` were not links, and `яcall_f()`, `яsrc_py{1}` were not babel objects but subscripts. gen-char-tables.el now also emits the script runs, both category lists as Emacs holds them, and a range table per category they name (it fails if a char has no category set, the C code's NILP case). Regeneration is byte-identical when run twice. `chars::word_boundary` transcribes word_boundary_p, `chars::word_start` is `\<`, and every `\<` / `\b` in the regexps mae-org transcribes uses it: grepping org-element.el and the string regexps it reads finds only the plain-link `\<TYPE:` (object regexp and org-link-plain-re, one site here) and the `\<call_` / `\<src_` heads. Tests: a unit table of 24 pairs measured with `(looking-at "\\<")` in an org-mode buffer (Han, kana, Hangul, Cyrillic, Greek, Arabic, fullwidth, Thai, combining marks; `é`, digits, Thai+Thai, Latin-1 pairs as controls), and objects-word-boundary.org with its golden: 31 objects; reverting word_start to the old rule loses all of them and adds 19 subscripts.
Sub/superscript braces were accepted when nested at most 3 deep. The
regexp Org builds for depth 3 is L\(N\|(?:N L N R)+N\|(?:N L (?:N L N R)+N
R)+N\)R with N = [^LR]*?: the contents hold no group, only flat groups,
or only groups that each hold flat groups. So the direct child groups must
all have one depth, and `x_{a{b}c{d{e}}} y`, `x_{{{a}}b{c}} y` and
`x^(a(b)c(d(e))) y` are no objects in Emacs; mae-org made all three.
The inline index now records, per bracket pair, whether its direct
children share a depth, and script.rs requires it. objects-multibrace.org
pins positives and negatives with its golden; dropping the new check
brings back six objects Emacs does not make.
Emacs reads `[[\n]]` as a fuzzy link with path " " and `[[\n@@]]` with path " @@". The path's line ending is a NEWLINE token, so `[[\n]]` holds no LINK_PATH token, and Link::is_radio took "no LINK_PATH" to mean radio: mae-org reported a radio link with path "\n]]". path_text also started at the first LINK_PATH, dropping a leading line ending (`[[\n@@]]` gave "@@"). A radio link is now a plain-format link with no path token (bracket and angle links have their markup; a plain link's path starts with its type), and the path is every LINK_PATH and NEWLINE token after the opening markup. objects-link-eol-paths.org pins seven shapes against Emacs's fields; restoring the old is_radio fails it on link_type, path and raw_link.
The corpus wrote Org syntax in Markdown backticks, which Org does not know, so Org parsed the text inside as live markup: - Five DevPractices notes quoted link syntax in backticks: `id:`, `[[id:NAMESPACE:node][Display Text]]`, `[[id:...][...]]`, and a bash `if [[ -z "$X" ]]`. Each was a real link to Org (`id:`-prefixed made an edge to the id "`-prefixed"). Every such span under assets/ is now Org verbatim, =...=; a scripted pass converted them and a grep confirms none is left. - The manual's 18 scheme-star-*.org titles wrote `Scheme: *buffer-count*`, which is bold in Org and reads "Scheme: buffer-count". The variable name is now verbatim: `Scheme: =*buffer-count*=`. The Emacs golden of practice-mutating-script-safety.org is regenerated (regen.sh): the link it held is now a verbatim object.
Review findings F2, F10 and F12 on the one link grammar: - `org-element-link-parser` decides the type from the path's first characters, untrimmed: a file name (`/`, `~/`, `./`, `../`), then a known `TYPE:` in any case, then `(coderef)`, `#custom-id`, `*heading`. classify_link trimmed first, so `[[ id:lead]]` (Emacs: the fuzzy link " id:lead") became an id link. `[[https:x]]` is web (Emacs: type https), `Mailto:` is mail. - `[[*Heading]]`, `[[#custom-id]]` and `[[(coderef)]]` get their own kinds and are never KB edges; `[[#c]]` was a fragment of an empty target. - `[[id:X::SEARCH]]` is the id X with Org's search option (org-id-open splits at the first `::`; Org 9.7 writes this form by default). It is read after the store form and the query, before the fragment, so `[[id:x::*h?rel=r]]` keeps its relation and a `#` after `::` stays Org's custom-id search. File links split the same way. The new `LinkClass::search` holds it; mae-kb kept it in the target. - The non-KB schemes are declared data (ADR-112 D4): `LinkTypes`, with `classify_link_with`; `classify_link` uses the defaults. The default list is Org's own link types plus ftps, attachment, man and the bibliography schemes cite, zotero and roam. An unknown scheme stays a KB link. - Every trim is `org-trim` (space, tab, LF, CR): a no-break space is content, as in Emacs. Tests (each failed on the previous code): kinds_are_read_as_org_reads_types, id_and_file_links_split_their_search_option, every_non_kb_scheme_is_pinned (one by one against a list written in the test, so dropping news: or man: fails), link_types_are_declared_data, and the trim and description cases of paths_are_normalised_like_org and the chosen-order test.
Review findings on semantics(): - F1, linear on hostile input. An item's drawer check looked at only its first two elements instead of collecting every child per drawer; properties resolve through a key map; tags, exclude and select tags are sets built once; every headline's region is decided once in Rules::new. The file tags are stored once (Semantics::file_tags) and a node holds only its own (SemanticNode::own_tags, Semantics::tags_of gives Org's list): N file tags and N nodes were N^2 strings, a memory DoS from a few hundred KB. Twelve `semantics()` families join the scaling gate, timed apart from the parse; the six the review measured at x47 to x391 are now x7-x8. Restoring the linear key scan, or one file-tag set per headline, fails the gate. - F3, a body never holds another node. A node's body runs from after its front matter to the first node nested in its element, the same rule for files, headlines and items, so ADR-092's write of one body can never rewrite another node's drawer or :ID:. A headline without an id is part of the body around it, as its links already were; an item node in a headline's section used to sit inside the headline's body. Text after a nested node is in no body (one contiguous range per write; documented). - F6, Org re-adds a referenced footnote definition from a pruned region (org-export--missing-definitions): one under a noexport subtree, an unselected tree, an archived headline or a drawer is rendered. Not under a COMMENT subtree, which the exporter deletes before parsing (emacs -Q: "Definition not found"). A rendered footnote's links belong to the node of the text that first references it, not the node it sits under. - F7, links in keyword and property values are edges of the owning node, org-roam's defaults, except ROAM_REFS, TRANSCLUDE and the raw export keywords (#+HTML: and the like), all declared in SemanticsOptions. - F8, a #+TRANSCLUDE in an unrendered region is diagnosed (TransclusionInUnrenderedRegion), as ids and links already were. - F9, the missing D4 configuration: select tags (default `export`, in-file #+SELECT_TAGS), which render only the selected trees and their ancestors and hide the zeroth section's content; the ARCHIVE tag with org-export-with-archived-trees (default: headline rendered, contents not); exclusion by a file tag; and a SetupFileNotRead diagnostic. Each rule was checked against emacs -Q exporting the same text. - F10, F11, F12: Org's trim everywhere (an :ID: ending in a no-break space now agrees with properties[ID]); one precedence rule for every file keyword, document order with an empty last value clearing; the link form of #+TRANSCLUDE gives the link's target as member and keeps its args. Fixtures: external-links.org gains local targets, bibliography schemes, search options and a plain id link; items.org a mixed-case item key; titles.org a title of objects kept as written; keyword-precedence.org and keyword-before-drawer.org are new. Every fixture is now named by a test.
The F3 invariant, pinned three ways: by example for items (a sub-item node ends the outer body; text after it is in no body, its link still the outer node's), headlines (an id-less child is the parent's; an item node in the section ends the headline's body) and the file; as a proptest over 600 generated documents of headlines, items, drawers, footnotes, noexport tags and repeated ids; and over the bundled corpus and fixtures. Removing the clipping fails all five tests. The regressions file holds the seeds proptest found while that mutant was applied.
`#+CAPTION[s b]: x` with nothing after it, or before a blank line, is a paragraph in Emacs and was a keyword in mae-org. When the line after the keywords is the end of the buffer, blank, a comment, a clock line or a headline, org-element--collect-affiliated-keywords moves back and drops them; the current-element regexp then reads that first line on its own, and `#+CAPTION[s b]:` (a space inside the brackets) is not `#+KEY:`, so it is a paragraph. Keywords kept but running into the limit (last in a block) still go to the keyword parser regardless; its `looking-at` fails there and the key and value come from the affiliated regexp's match data, which the keyword view now reproduces (key CAPTION, value after `]:`). affiliated-orphan-shapes.org pins seven shapes with its golden; mapping orphans to keywords again, as before, fails it on two paragraphs.
Footnotes re-added from noexport subtrees and drawers but not from COMMENT subtrees, their attribution to the referencing node (also through a chain and for a definition before its reference), transclusion diagnostics in every unrendered region, select tags (default, in-file, configured, and by file tag), exclusion by file tag, archived headlines in all three modes and an exact tag match, SETUPFILE diagnostics, and a long tag list. Each case was first exported with emacs -Q (Org 9.7.11); each fails when its rule is removed.
F4: the differential judged a title change "plain text" by running mae-org's own plain-text reading on mae-kb's title, so a bug in plain.rs agreed with itself (plain_keep_markup, plain_link_ignores_desc and plain_drop_all_objects_text survived). conformance/semantics-titles.el computes every node's title and level from Org's own parser, and gen-semantics-titles.sh writes semantics-titles.tsv (checked in, Org 9.7.11). mae-org's title and level must equal Emacs's, and a mae-kb title difference is intended only where mae-org's equals Emacs's. Bodies are now compared with mae-kb's ingest parser through its own rewrite_links, each difference classed by rebuilding mae-kb's range from mae-org's spans (front matter included, run past a nested node). The header says exactly which fields are compared and which are not. F5: an entry for one file must name the difference (detail); a wildcard entry pins its hit count; properties compare key by key, and a key must be exactly the canonical form of mae-kb's (prop_no_canon now fails); plain_link needs a plain link making exactly that edge; a pinning test must be a #[test] function. New classes: id_search_option, local_target, title_matches_emacs, the body classes. Stale entries removed: the keyword_value and plain_link entries (keyword-value links are edges now; the corpus no longer writes link syntax in backticks) and the file-wide wildcards, replaced by one entry per difference. All named surviving mutants now fail a test; injected disagreements in a title, level, tag, body, property value, property key and link source each surface unclassified.
The consumer's exposures (opaque target anchors, undefined footnotes, link attributes, citations as unresolved) took the module from 865,332 to 868,108 bytes (sha256 df68b587…), inside the 900,000 budget. ADR-112's O3 note and test.mjs record it.
Every mae-org output channel for a document in every render mode (property_drawers x headline_ids x full resolver): body HTML and hast, text(), node_text per node, every nodes[] field and the result JSON. Markers are qz<digits>qz, scanned case-insensitively; attribute values can be stripped in HTML, JSON-escaped HTML and hast spellings.
leak-export.el exports a directory body-only with ox-html, babel never evaluating, ids untracked, writing each result as soon as it is known. An id: link exports as a link to an opaque target with its description, and a fuzzy or custom-id link that resolves nowhere as its description (MAE's KB and unresolved links), instead of failing or vanishing. emacs.rs runs it under systemd-run --scope MemoryMax=3G (else ulimit -v) and timeout, HOME under target/, with a process-group kill as backstop; after a failed run the first document without a result is recorded as the hang and the rest are exported again.
Small deterministic Org documents (SplitMix64, one seed per document)
that plant one marker per site and record each site's intent: leaf,
containers, and where MAE may show it that Org does not (a plain
property value, alias or title in metadata; an id link's #fragment in
an href). Covers inline src and calls by :exports, results, snippets,
comments, COMMENT/noexport/ARCHIVE/select/exclude, drawers, header-args
scopes, macros (arguments, $0, built-ins, undefined, circular, pairs
that open and close inline constructs and blocks, line-start structural
expansions), targets, radio targets, footnotes, citations, typed links,
property values, aliases, captions, tables, lists and blocks. A metadata
value with generated content is shown to Org by a {{{property}}} or
{{{title}}} line carrying its own marker.
tests/render_leak_oracle.rs (ignored: needs Emacs) generates the corpus, exports it through bounded Emacs in parallel chunks, and judges every MAE channel: a LEAK is a marker MAE shows that Emacs's HTML does not, outside the site's allowance; where Emacs fails, MAE must render nothing unless the register lists the error. Losses and fail-closed documents are counted, never failed. Each failure category is minimised by batched ddmin (lines, then tokens; one Emacs run per round, a run budget per repro) and printed with both outputs. Fixed cases: the confirmed leaks (a)-(e) and the consumer fixtures, each with Emacs's measured behaviour. The report reports when each known shape is first found by the generator alone. Seed, count, depth, jobs, chunk, timeouts and minimisation are environment variables; corpus, logs and report go under target/leak-oracle/.
examples/probe.rs prints every mae-org channel for a file (show), searches all of them for a needle (grep), runs replace_body and prints every channel that changed (splice FILE ID BODYFILE), and times a render (rtime); it shares the oracle's channel list. conformance/cmp.sh puts the probe's grep beside bounded Emacs's export of the same files, in POSIX sh.
The export and minimisation worker loops nested five deep inside their thread scopes, past the audit-metrics nesting ceiling of four.
Review round 6 (blocker 1): a macro glued into a line's first token
(`#+BEGIN{{{m}}}`, `:{{{e}}} `, `\begin{{{m}}}`, `#`, `#+KW`, `*`, `-`,
`|`, `1.`, `[fn:`) makes the line another element, and Org judges every
later macro inside it. GLUED adds 33 such documents to the oracle, and
the reviewer's and the round's own inputs join tests/macro-review/.
At ca22c58 the oracle finds 18 leaks (f00, f03-f06, f08, f24, f27,
f28 and nine of the inputs) and one document failing closed that Emacs
renders (r6-k6).
…buffer
Review round 6 (blocker 1): an expansion inside a line's first token
changes that line's element (`#+BEGIN{{{m}}}` a block, `:{{{e}}} ` a
fixed-width line, `\begin{{{m}}}` a LaTeX environment), and the scan
still expanded the macros Org then judges inside the new element: only an
expansion standing after blanks at a line's start was checked, by a
character heuristic, and a plain word between blanks skipped every check.
The recogniser now dispatches on one line classifier (element::line_kind,
element_start: the regexps org-element--current-element tries, in its
order), and after every expansion the scan compares the line it touches
before and after with element::same_line: the same kind with the same
names, indentation, paragraph and table.el separation, and list item
parts. Any change, or a call taking lines outside a paragraph, re-reads
the whole buffer; otherwise the unit is read again. The inert-word path
and its line-start heuristic are gone. A todo keyword or priority cookie
an expansion makes keeps the title where it started, as Emacs does
(r6-k10). What a line closes is not compared: an opener a new closer
matches would have held the macro.
Each condition is pinned by a document Emacs 30.2 exports (tests/
macro-review/r6-*): removing the kind, indentation, paragraph-end,
table.el or item-part comparison, the line-taking guard, or the text
before the unit on its line each leaks a marker.
The cost is honest now: one unit of N macros costs N reads of it. The
macros register entry states the measured limits (320 short macros on
one line, 568 on as many, 117 in 15 KB of prose; 455 lines each made a
list item), replacing two false claims; one line of 60,000 counters now
fails closed with the budget diagnostic.
Review round 6 (major 2): the budget emptied ordinary documents. A macro in a caption or a description list's tag was an `Other` unit, read with the whole buffer after each expansion, and a line-start value beginning with `[` or `-` was taken for a new element: a glossary of 250 tags, 300 captions or a changelog of 400 link-macro entries rendered nothing. Captions' values, items' tags and verse blocks' contents are now units read again alone, as paragraphs are, whenever the line the expansion touches stays the same kind of line. A unit test counts the work of 5,000 glossary tags, captions, changelog entries, numbered steps and table rows: 8 times the entries cost at most 9 times the work, under a quarter of the budget (at 07c5262 the glossary fails with the budget diagnostic); scaling.rs times the reviewer's three shapes.
Which Emacs export errors MAE may render through now comes from
conformance/render-divergences.toml, not a hard-coded list: an entry
registers the error its `org` field quotes, and MAE must carry a
diagnostic kind its `mae` field names (footnote_undefined). An
undefined footnote is exported again with that label's references
deleted (leak-export.el writes NAME.registered beside the HTML), so the
document is still compared marker by marker.
A store-form link's description ([[id:X|D]], ADR-030) is MAE's syntax
for text the author wants shown: it is allowed wherever Org shows the
text planted beside it. An exposure line counts only while it still
holds its macro call. The generator glues macros into a line's first
token (#+BEGIN{{{m}}}, #+END{{{m}}}, :{{{e}}}, \begin{{{m}}}, #+KW, *,
1., |, [fn:) in single lines and in block pairs, and never repeats the
self-referential title macros Emacs expands forever.
…arithmic Review round 6 (major 4): each subtree an expansion made COMMENT was kept in a list searched for every later macro, uncharged: N headlines made COMMENT, each with a macro below, took 2.3 s at 1 MB and 85 s at 8 MB while the budget never tripped. The subtrees are pushed in scan order and none holds another (a macro inside one is not expanded), so they are sorted and disjoint: a binary search finds the one that may hold a position. scaling.rs gains the family, sized so the quadratic term shows: x34.5 for 8 times the bytes at 29b5ef6, x5.0 now.
Shape (f), a macro glued into a line's first token, joins (a)-(e) in
the report, with a fixed case: #+BEGIN{{{bc}}} opens a comment block
whose glued closer #+END{{{bc}}} Org never expands, so the text after
it stays hidden.
…nder
Review round 6 (major 3): when either side's macros failed, both sides
were compared with nothing expanded, so an edit writing an undefined
macro or an (eval ...) template into a body of a rendering page was
accepted and the page then rendered nothing (`Gamma body with a typo
{{{nosuch}}}.`).
An edit that makes a rendering page fail is now refused with
BodyEditError::BreaksRendering { kind }, the failure's diagnostic name
(macro_undefined, macro_eval, ...). A page whose macros already fail
may still be edited, to repair them or not, compared unexpanded as
before. The test fails with the refusal removed, and with the
unexpanded comparison limited to both sides failing.
Review round 6 (minor 5): `[cite/nb/b:@A]` rendered `1`, Emacs 30.2 `(1)`; `[cite/nb/bare:@A;@b]` `1, 2`, Emacs `(1, 2)`. oc-basic's numeric style parenthesises unconditionally (`"(" prefix numbers suffix ")"`); only the author-year styles honour a bare variant. Four variants pinned against Emacs, two of which fail before.
The generative leak oracle (tests/render_leak_oracle.rs and its modules, conformance/leak-export.el, conformance/cmp.sh, examples/probe.rs): new files only, based on ca22c58.
Review round 6 (G, and the leak oracle's property-value and alias categories, 60 documents at ca22c58): a property value was read as objects and its macros rendered nothing, so a macro closing what the value opened (`src_sh[:exports none]{echo X{{{cs}}} tail`) showed X in the node metadata and the drawer, and an argument a template drops showed too. Org shows a property value only through the property macro, whose expansion is scanned again where it stands. A value with a macro is now expanded as that macro's expansion would be at its entry (the heading line and its property drawer before a paragraph holding it): the document's templates, `{{{keyword}}}` read from the document's lines, `{{{property}}}` and ITEM/TODO from the entry, then babel's results (one an inline src block or call owns removed), all charged to one budget per render linear in the source. A value whose macros cannot be expanded is withheld (nothing), with a new property_withheld diagnostic at the drawer. Eight documents pinned against Emacs 30.2's page of `P {{{property(Q)}}}`; at 9d684ed all eight differ.
Review round 6 (F): Org deletes the COMMENT subtrees before it parses and selects (org-export--delete-comment-trees, then org-export--selected-trees on the parse, archived trees and excluded ones still in it), so a select tag found only under a COMMENT headline, or under one of its descendants, selects nothing. MAE selected the COMMENT headline's ancestors, rendering their whole subtrees in every channel. Selection now reads only the headlines outside every COMMENT subtree, for the page and the semantics alike (one Rules). Four documents pinned against Emacs 30.2 (tests/pruning/f*), three of which fail at 8dbea79; the descendant case kills its mutant.
Review round 6 (H): Org deletes the COMMENT subtrees before it parses and runs babel. A footnote definition under a COMMENT headline inside an archived one was taken as the archived contents Org parses (so used: its text shown in every channel), and a named #+RESULTS: under a COMMENT headline was babel's result for its block, so the real one was shown. A COMMENT headline is now a COMMENT region before it is archived contents, and babel's tables (names, named results, calls) read the buffer through one walker that skips the COMMENT subtrees (keywords::uncommented, which the settings and templates already read). The other tables were already built from the pruned tree: targets, radio targets and names from the page's own first pass, footnote definitions from the one Rules the page and the semantics share. tests/pruning/matrix pins every table (footnotes, targets, radio targets, #+NAME, named results, babel calls, macro templates) against every pruning reason (COMMENT, :noexport:, archived, unselected, COMMENT inside archived) to Emacs 30.2: 35 documents, two failing before; removing either fix fails it.
Review round 6 (I, and the leak oracle's citation categories, about 40
documents per seed): MAE expanded macros inside citations and read the
citation from what they made, so a prefix or suffix showed text
oc-basic drops, and a key leaked (`#+MACRO: oh @@` made
`[cite:{{{oh}}} @k]` a reference `@` with `@K` its suffix).
Measured in Emacs 30.2: Org's export never expands a macro inside a
citation (org-macro-replace-all does not see it as a macro there), and
exports it as nothing; a counter there does not count
(`[cite:@k {{{n}}} suf] C{{{n}}}` gives `(??, ???? suf) C1`). The scan now
skips a macro inside a citation, in both expansions, and a citation's
affixes render with their macros as nothing instead of being dropped
whole. Ten documents pinned against Emacs (tests/citations/); at
622cd93 seven render differently and five show a marker.
…used Review round 6 (H, corrected): what each name table consults is what ox.el consults, measured, not one pruned view: ox deletes only the COMMENT subtrees before it parses, never reads a comment as a definition, and re-adds a footnote definition from any other pruned region (org-export--missing-definitions). The pruning matrix now pins 8 tables (standalone and inline footnote definitions, targets, radio targets, #+NAME, named results, babel calls, macro templates) against 10 regions (COMMENT, COMMENT inside archived, :noexport:, an #+EXCLUDE_TAGS tag, archived, unselected, a LOGBOOK, another drawer, a comment block, comment lines) to Emacs 30.2: 80 documents. One cell class differed: an inline definition `[fn❌ …]` in a :noexport:, excluded, archived or unselected subtree, or a drawer, is used by Org while MAE left the reference undefined (Notes kept only standalone definitions): six failing cells, fixed. A non-LOGBOOK drawer's contents are hidden by MAE (registered: drawers). The footnote_undefined wording (d.ts, the diagnostic's doc, the undefined-footnote entry) no longer says a :noexport: definition is undefined.
Review round 6 (J): `#+MACRO: res {{{results($1)}}}` with
`call_f[:exports foo](x="A") {{{res({{{res(t B)}}})}}}` made Emacs fail
("Undefined Org macro: res; aborting") while MAE rendered `)}}}`.
Why: the first expansion leaves `{{{results({{{res(t B)}}})}}}` (its
argument is no macro object); babel leaves a results macro in place when
its block's :exports holds a word Org does not know
(org-babel-exp-do-export returns nil); the second expansion, with only
`results` defined, replaces it by its argument and scans that again,
finding `res` undefined. MAE hides those results (privacy first,
registered exports-unknown-word / exports-no-known-word), so never ran
that expansion.
The second expansion is now also run with Org's set of removed results
(Babel::left_as_written: an unknown word in any merged :exports source)
whenever it differs from MAE's; a failure there fails the page, as
Org's export does. What renders is still the fail-closed one. Five
documents pinned against Emacs (tests/results-macros/): three fail
closed now, two render as before; at 2148007 j5 and j9 render.
Review round 6 (K): the register promises a footnote_undefined diagnostic wherever Org's export fails on an undefined footnote, but only references the page renders were checked. Measured in Emacs 30.2, Org also fails on one in a drawer it exports (any but LOGBOOK; an item's property drawer is one to Org) and on one in a definition nothing references; not on one in a LOGBOOK, a :noexport:, archived or unselected subtree, a headline's property drawer, a comment block or an example block. notes::unrendered_undefined finds those references from the semantics' regions and the labels with a usable definition (Semantics now carries them, defined_notes), and reports each as a rendered one is. Fourteen documents pinned against Emacs (tests/footnote-undefined/): three were missing their diagnostic at 7d3d862. The drawers divergence (MAE draws no drawer Org exports) was already registered.
Leak oracle (seed 1, after the round-6 fixes: the last three leaks,
all `#+TITLE: [cite:{{{id(X)}}} @k]`): the file node's title rendered
the citation with the prefix the forced title expansion made, while
Emacs's page never shows it.
Measured with the full HTML export of Emacs 30.2: Org does expand a
title's macros anywhere in its value (verbatim, link paths, inline src
included; the forcing MAE does is right), but it processes only the
parse tree's citations, so one in the title exports as nothing (an
empty <h1 class="title">). The file node's title now renders its
citations as nothing. Six titles pinned against <h1 class="title"> and
the page's {{{title}}}; at c87a615 the citation one shows its prefix.
Org shows an inline src block's code in the title (babel leaves the
title alone); MAE hides it, now registered (babel-in-titles).
Leak oracle (seed 2, after the round-6 fixes): a COMMENT headline an
expansion makes (`*{{{star}}} COMMENT :sel:`) is not deleted by Org,
which deletes the source's COMMENT subtrees from the text before it
expands anything. It is parsed: its select tag selects (Emacs then
exports nothing else), its footnote definitions are used, and it is
pruned with what it holds (org-export--skip-p). MAE treated it as a
deleted subtree, so 0e7199c's rule (a select tag under COMMENT selects
nothing) wrongly applied and the whole page leaked.
As ox.el does: expand() first cuts the source's COMMENT subtrees from
the text (one map piece each), so the expansion's parse holds only
COMMENT headlines an expansion made; its semantics (comments_deleted)
select and prune those like excluded subtrees. The source's semantics
keep the round's rule. Babel skips what is under a COMMENT or archived
heading (org-babel-exp-process-buffer), so a results macro there stays
for the second expansion and an undefined macro it completes fails the
page (seed 2's `Undefined Org macro: cat`).
Seven documents pinned against Emacs 30.2 (tests/pruning/expanded/);
five fail at 575c9a9.
Review round 6 (minor 10): ox-html converts every plain-text object's special strings (org-export-with-special-strings, on by default): `\-` a soft hyphen, `---` and `--` before a character that is not `-` an em and an en dash, `...` an ellipsis, its four regexps in order, each left to right. MAE showed them as written. Five places rendered a container's objects with their own loop (a paragraph or title, an object's contents, a link's description, an inline footnote, a table cell, a citation's affix); they now share one, Builder::parts, which hands each run of tokens to plain_text as one plain-text object: so nothing converts across objects (a hidden export snippet between two dashes keeps them, as in Emacs), inside code, verbatim or an inline src block, or in a link's path. Search text and anchors keep the source's characters. One document of ten paragraph shapes pinned line by line to Emacs 30.2's page (tests/special-strings/); at 1bfddc8 every line differs.
Round 6 (macro expansions judged by the parser's own line classifier, the name tables ox.el consults, citations parsed after expansion, special strings, replace_body refusing edits that break the page) took the module to 906,436 bytes, past the 900,000 budget. The budget guards accidental bloat in a build-time module; this growth is deliberate, so it is raised and recorded in ADR-112's O3 note.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ADR-112 phases O1–O3:
mae-org(shared/org), one lossless org parser that every MAE consumer will derive from, andmae-org-wasm(shared/org-wasm), its WASM facade with a synchronous JS loader. Taggedmae-org-v0.1.0(87fa53a), the first version an external consumer may pin.org-element, except for a test-pinned divergence register (conformance/divergences.toml).:exportsscopes, babel calls, snippets, COMMENT/noexport/select/exclude/archive pruning, macros expanded asorg-macro-replace-alldoes, and citations rendered as oc-basic renders unresolved ones. Every deliberate difference from Org is inconformance/render-divergences.toml.MAE itself does not use the crate yet. Switching ingest, export and read mode over to it is O4, which comes in a separate PR. Hence
release:none.Verification (at 87fa53a)
-D warningsis clean for debug, release and wasm32; fmt, audit-metrics, code-map and shell lint all pass.org-element: the checked-in goldens, plus 46/46 org-mode and 293/293 worg files with 0 field mismatches.tests/render_leak_oracle.rs, ignored test, run locally): seeded random documents with planted markers, checked in every output channel and render mode, attributes and metadata included. Seeds 1–3, 2000 documents each: 0 leaks. The only cases where Emacs errors and MAE renders are registered.Review history
Five independent adversarial reviews, plus an external consumer's own review, drove six fix rounds. Round 6 was verified by the gates and the generative leak differential above, not by an independent reviewer.
Notes for reviewers
org-wasm.make org-wasm,org-wasm-reproandorg-wasm-testrun viascripts/org-wasm.sh.#[ignore]d.conformance/regen.shregenerates the goldens, andconformance/cmp.shplusexamples/probe.rssupport review.replace_body, the span-based write path O4 will use, is in the crate. O4 is where it gets its real caller.