Repository navigation
fix(protocol): decide the marker terminator outside the pattern - #10084
Conversation
UX Review (Fable 5) — ✅ PASSUX-level review of This diff is entirely protocol/parser logic and tests — no control, label, string, or persistent-element state a user sees is added or changed. The reconcile table is empty: there was nothing for a blind read to look at (the report confirms no screenshots existed, correctly, since none were needed). The one behavioral surface change fails toward visible prose instead of deleted prose, which is the pre-existing failure direction and strictly improves what the user experiences. UX-Verdict: PASS Pure parser fix with no new user-visible surface; prose the old pattern silently deleted now stays on screen, failing visible rather than destructive. [UX-REVIEWED] acb8348 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of All claims verified against the repo: five backend call-site files ( First-Principles-Verdict: PASS Verify the accepted cost against live traffic: a stray unclosed Not justified as shipped
What this change shipsInventory (8 items) — 6 justifiedIntent: stop an unclosed
The defect's provenance is the new tests, which fail on base ( [FIRST-PRINCIPLES-REVIEWED] acb8348 |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Non-regular decision (bracket balance) correctly moved out of the regex, with the raw patterns made unreachable so the check can't be skipped — right root-cause shape. The one rewritten pin ( Suggestions
[DESIGN-REVIEWED] acb8348 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
18df47e to
6465131
Compare
An `[OPTIONS:` head the model never closed consumed the rest of its line whenever
the line held any later `]`:
[OPTIONS: A | B then check arr[0]
labels = ' A | B then check arr[0'
visible = ''
The only closer belongs to `arr[0]`, so the body ran on through the prose and that
`]` became the terminator. Every consumer removes the whole match -- `parse_options`
replaces, `slack.format` and `messaging.renderer` cut at `match.start()`, and
`whatsapp.turn_renderer` PERSISTS the cut turn -- so the line left the message and
came back as the pill label `B then check arr[0`.
The bare-`[` alternative is what admitted that opener, and it exists for a reason:
`[OPTIONS: Fix [x logging | Skip]` parses and is pinned on both surfaces. Nothing
about the brackets separates the two. Reduced to skeletons they are one string:
[OPTIONS: Fix | Skip [x logging] -> w|w[w]
[OPTIONS: A | B then check arr[0] -> w|w[w]
So no bracket-balance rule can accept one and refuse the other, and both are now
pinned as sharing a skeleton so a future attempt at one finds the argument.
The discriminator is where the opener sits relative to the END. The bare form is
refused exactly when nothing but ordinary text lies between it and a closer at the
end anchor: that closer is then its own partner rather than the marker's, which
means the marker is unterminated. Crossing a SEPARATOR clears the gate, because the
opener is inside a label and the list continues past it -- which is what keeps the
pinned stray-opener shape. Crossing another BRACKET clears it too, because some
other bracket form owns that closer, which is what keeps
`[OPTIONS: Fix arr[0] | Skip]` (continuation admits the `]`) and
`[OPTIONS: a[1] | b[2]]`.
Spelled twice per surface because the two bodies end differently (`$` under
MULTILINE, `\s*\Z` under DOTALL), and both carry the optional stray markdown-link
tic and a wrapper run -- otherwise the shape returns wearing one, which is asserted.
The gate resolves a second shape the grammar documented as unresolved: the
separator-tail form `Done. [OPTIONS: Merge | Wait], details in CHANGELOG[1]`. Its
comment said `], ` genuinely continues the list so no guard at the INTERNAL closer
can tell it apart, and deciding it was "a separate call". The gate reaches it from
the other end, and the line now stays whole instead of losing its tail to a pill
reading `Wait], details in CHANGELOG[1`. Its two pins are updated, one per surface,
from "unchanged" to "declined".
COST: a stray opener in the FINAL label, where no separator follows to clear the
gate -- `[OPTIONS: Fix | Skip [x logging]`. It fails toward a VISIBLE marker with
nothing removed, the direction every cost in this grammar fails in, and it is
asserted at the consumer rather than only at the regex.
ReDoS: the lookahead is tempered, its scan stopping at the first bracket or
separator, so the runs scanned from different openers are disjoint and the work
stays linear -- measured at 0.5/2.3/10.2/31.8 ms for 2k/10k/40k/120k of `a[b`
repeated on an unterminated marker, where every gate fails. It adds no quantifier
over a quantifier, and the frontend `.source` pin is extended to cover the gate
character for character so the two surfaces cannot drift.
Verified by differential rather than by inspection: every marker shape appearing in
the five marker suites on main (96 of them) was run on both grammars, and the only
behaviour changes are the two shapes above. Backend and frontend agree on all 24
gate-relevant shapes.
Fixes #10058
Review round 1, four findings, all applied. Two were holes in the gate itself:
- BLOCKING, and correct: the scan stopped at any separator, so the comma INSIDE
`dict[str, int]` halted it before the closer and the gate cleared --
`[OPTIONS: A | B then inspect dict[str, int]` still ate its line. The scan now
crosses `,` and stops only at `|`, a bracket, or the end. The asymmetry is the
point: `|` is the unambiguous option separator, while `,` is only the fallback
one (`parseOptions` reaches for it just when no `|` is present) and inside
brackets it is ordinary punctuation models write constantly.
- WATCH, also correct: the TRAILER scan excluded `\n` while its body spans them
under DOTALL, so a newline blinded the gate and
`[OPTIONS: A | B then check arr[0\nAnd that is all]` walked through. The two
scans now differ deliberately -- LINE stops at a newline because its body does,
TRAILER does not because its body does not -- and both halves are pinned so the
asymmetry cannot read as an oversight.
Cost of the comma change, enumerated with the others: a stray opener whose only
following separator is a comma (`[OPTIONS: Fix [x logging, Skip]`) joins the
given-up family. Same direction -- visible marker, nothing deleted.
Two were about keeping the file honest:
- The `_MARKER_LABEL_CONTINUES` docstring and `optionMarker.ts` both still said the
separator-tail form was "deliberately unchanged... a separate call", which this
change contradicts three screens below. Both rewritten to point at the gate.
- `OPTIONS_RE_LINE` / `OPTIONS_RE_TRAILER` now build their tails from
`_MARKER_STRAY_TIC` / `_MARKER_WRAP_RUN` instead of respelling them, so a tail
edit cannot drift from the gate that has to mirror it exactly.
Review round 2: the gate is a PARTIAL fix, and that is now recorded rather than
implied.
The blocking finding is correct: nested terminal brackets bypass it.
`[OPTIONS: A | B then inspect list[dict[str, int]]` still matches, because the scan
halts at the inner `[` and never reaches the outer opener's partner -- which is the
terminator. Measured: depth one is fixed, depth two and three behave as on main.
Encoding one nesting level into the scan only moves the boundary; depth three
defeats that, depth four the next. The condition the gate reaches for is "the
terminating closer is not nested", which is bracket BALANCE, and balance is not a
regular language at unbounded depth. So this cannot be closed inside the pattern,
and a check outside it would have to be added at all five backend call sites plus
both frontend entry points -- where any future caller using the raw regex silently
loses it, which is the drift this module exists to prevent.
What this change therefore claims: it closes the single-bracket form, which is the
common one in prose (`arr[0]`, `docs[4]`, `CHANGELOG[1]`), plus the cross-line and
separator-tail variants, and regresses nothing. The residual is pinned at the depth
where the gate stops reaching, in `TestTheResidualThisGateCannotClose`, and named in
the constant's own docstring, so the boundary is a measured cost rather than a
surprise for the next reader.
6465131 to
9c1ceb7
Compare
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/constants.py:357 -- pipe inside a terminal bracket bypasses the terminator gate |
An `[OPTIONS:` head the model never closed consumed the rest of its line whenever
the line held any later `]`:
[OPTIONS: A | B then check arr[0]
labels = ' A | B then check arr[0'
visible = ''
The only closer belongs to `arr[0]`, so the body ran on through the prose and that
`]` became the terminator. Every consumer removes the whole match -- `parse_options`
replaces, `slack.format` and `messaging.renderer` cut at `match.start()`, and
`whatsapp.turn_renderer` PERSISTS the cut turn -- so the line left the message and
came back as the pill label `B then check arr[0`.
THE PATTERN CANNOT DECIDE THIS, and three review rounds on an earlier attempt are
the evidence. The body admits a bare `[` on purpose so a stray opener does not sink
the marker (`[OPTIONS: Fix [x logging | Skip]` parses, and is pinned). Reduced to
their bracket structure the shape to refuse and the shape to accept are one string:
[OPTIONS: Fix | Skip [x logging] -> w|w[w]
[OPTIONS: A | B then check arr[0] -> w|w[w]
A lookahead can only see one nesting level, so `list[dict[str, int]]` defeats a
one-level rule and `a[b[c[d]]]` a two-level one. The condition wanted is "the
terminator is not nested" -- bracket BALANCE, not a regular language at unbounded
depth. Encoding depths is a treadmill; the check has to leave the pattern.
So it does. `_marker_terminator_is_nested` walks the captured labels with a stack:
if an opener is still open at the end, the closer the pattern took as the terminator
is really THAT opener's partner -- unless a `|` follows it, which means it sits
inside a label and the list continues past it. `,` is deliberately not enough: it is
only the fallback separator and inside brackets it is ordinary punctuation
(`dict[str, Any]`, `arr[i, j]`). An unmatched CLOSER is ignored, because a label may
legitimately carry one. That single rule decides every depth.
AND THE CHECK CANNOT BE BYPASSED, which is the other half of the fix. A check that
callers must remember to call is one a future caller forgets, and this module's
comments call the grammar a single source of truth precisely to prevent that drift.
So the compiled patterns are now private (`_RAW_OPTIONS_RE_*`, and module-scoped in
`optionMarker.ts`), and what callers get applies both halves:
- backend: `OPTIONS_RE_LINE` / `OPTIONS_RE_TRAILER` are matchers exposing the four
members anything used -- `search`, `finditer`, `sub`, `pattern` -- so all five call
sites and 94 existing test references read unchanged and every one of them gains
the check.
- frontend: `findOptionMarkers`, `findLastOptionMarker`, `stripOptionMarkers`,
`terminatorIsNested`, plus `OPTION_MARKER_PATTERN_SOURCE` as a STRING for the
shape pin. Nothing callable leaks, so nothing can be applied by halves.
Making the pattern private found two callers I had not enumerated -- `TurnBlock.tsx`
and `searchableText.ts` -- which were using the raw regex and would have kept the
old behaviour silently. That is the argument for the indirection, discovered by the
typechecker rather than by review.
Two side benefits worth naming. The g-flag `lastIndex` hazard the frontend docs
warned every caller about is gone, because the functions clone internally; its test
now asserts the stronger property that no `RegExp` is exported at all. And the
separator-tail form `Done. [OPTIONS: Merge | Wait], details in CHANGELOG[1]`, which
`constants.py` documented as unresolved ("deciding which shape loses -- a separate
call"), is decided: the `[` of `CHANGELOG[1]` is still open at the labels' end, so
the line stays whole instead of losing its tail to a pill.
COST: a stray opener in the FINAL label, where no `|` follows to make it label text
-- `[OPTIONS: Fix | Skip [x logging]`, `[OPTIONS: Fix [x logging, Skip]`. Fails
toward a VISIBLE marker with nothing removed, the direction every cost in this
grammar fails in, and asserted at the consumer rather than only at the regex.
Verified by differential, not inspection: all 96 marker shapes in main's five marker
suites run on both the old and new decision, and the only behaviour change is the
separator-tail form. Cross-surface: 32 markers compared on labels AND stripped text,
plus 10 label runs on the rule itself -- zero disagreements. 3802 backend tests,
27311 frontend. The check is one pass with a stack, pinned linear at 200k openers.
Fixes #10058
Review round 1 on the refactor: the rule is now TOTAL, with no separator hatch.
The blocking finding is correct. The hatch -- "an unmatched opener is label text if a
`|` follows it" -- is satisfied by a `|` INSIDE the unmatched bracket, so
`[OPTIONS: A | B then inspect dict[str | int]` cleared it and the line was eaten
again. That is the third defeat of the same hatch, after a separator stopping a scan
early and a newline blinding it, and each time the shape it readmitted was
structurally identical to the shape it was meant to protect. The hatch was the
defect, not its spelling.
So the rule is simply: a marker's labels must have BALANCED brackets. No depth limit,
no separator heuristic, no exceptions. `_marker_labels_have_unmatched_opener` /
`labelsHaveUnmatchedOpener` -- renamed, because the old names described the hatch.
Measured cost, over all 96 marker shapes in main's five suites: exactly ONE shape
changes beyond the intended fixes -- `[OPTIONS: Fix [x logging | Skip]`, a label
carrying an unclosed `[`, is refused and renders as visible text. That shape was a
deliberate allowance ("does not make the pair a REQUIREMENT") taken when the
alternative was unknown; the alternative is deleting a line of prose, since both
shapes hold one unmatched opener and a closer at the end anchor. Its two pins are
rewritten, one per surface, with that argument in them rather than in a commit
message.
Cross-surface differential extended to the new shapes: 34 markers on labels and
stripped text, 12 label runs on the rule -- zero disagreements. 3804 backend tests,
217 frontend marker tests.
c102852 to
acb8348
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Maintainer review (agent-assisted), pinned to acb8348. All five backend consumers were checked: search/finditer/sub/pattern only, with empty-string replacement. The matcher preserves these contracts and the frontend uses the same closer set. Owner-delegated acceptance: an unclosed '[' in a label now renders literally. Dedicated-worktree source/test review; no local suite rerun this wave; current-head readiness success verified.
…dotdev#10084) * fix(protocol): a bare opener may not be the closer that ends the marker An `[OPTIONS:` head the model never closed consumed the rest of its line whenever the line held any later `]`: [OPTIONS: A | B then check arr[0] labels = ' A | B then check arr[0' visible = '' The only closer belongs to `arr[0]`, so the body ran on through the prose and that `]` became the terminator. Every consumer removes the whole match -- `parse_options` replaces, `slack.format` and `messaging.renderer` cut at `match.start()`, and `whatsapp.turn_renderer` PERSISTS the cut turn -- so the line left the message and came back as the pill label `B then check arr[0`. The bare-`[` alternative is what admitted that opener, and it exists for a reason: `[OPTIONS: Fix [x logging | Skip]` parses and is pinned on both surfaces. Nothing about the brackets separates the two. Reduced to skeletons they are one string: [OPTIONS: Fix | Skip [x logging] -> w|w[w] [OPTIONS: A | B then check arr[0] -> w|w[w] So no bracket-balance rule can accept one and refuse the other, and both are now pinned as sharing a skeleton so a future attempt at one finds the argument. The discriminator is where the opener sits relative to the END. The bare form is refused exactly when nothing but ordinary text lies between it and a closer at the end anchor: that closer is then its own partner rather than the marker's, which means the marker is unterminated. Crossing a SEPARATOR clears the gate, because the opener is inside a label and the list continues past it -- which is what keeps the pinned stray-opener shape. Crossing another BRACKET clears it too, because some other bracket form owns that closer, which is what keeps `[OPTIONS: Fix arr[0] | Skip]` (continuation admits the `]`) and `[OPTIONS: a[1] | b[2]]`. Spelled twice per surface because the two bodies end differently (`$` under MULTILINE, `\s*\Z` under DOTALL), and both carry the optional stray markdown-link tic and a wrapper run -- otherwise the shape returns wearing one, which is asserted. The gate resolves a second shape the grammar documented as unresolved: the separator-tail form `Done. [OPTIONS: Merge | Wait], details in CHANGELOG[1]`. Its comment said `], ` genuinely continues the list so no guard at the INTERNAL closer can tell it apart, and deciding it was "a separate call". The gate reaches it from the other end, and the line now stays whole instead of losing its tail to a pill reading `Wait], details in CHANGELOG[1`. Its two pins are updated, one per surface, from "unchanged" to "declined". COST: a stray opener in the FINAL label, where no separator follows to clear the gate -- `[OPTIONS: Fix | Skip [x logging]`. It fails toward a VISIBLE marker with nothing removed, the direction every cost in this grammar fails in, and it is asserted at the consumer rather than only at the regex. ReDoS: the lookahead is tempered, its scan stopping at the first bracket or separator, so the runs scanned from different openers are disjoint and the work stays linear -- measured at 0.5/2.3/10.2/31.8 ms for 2k/10k/40k/120k of `a[b` repeated on an unterminated marker, where every gate fails. It adds no quantifier over a quantifier, and the frontend `.source` pin is extended to cover the gate character for character so the two surfaces cannot drift. Verified by differential rather than by inspection: every marker shape appearing in the five marker suites on main (96 of them) was run on both grammars, and the only behaviour changes are the two shapes above. Backend and frontend agree on all 24 gate-relevant shapes. Fixes kirodotdev#10058 Review round 1, four findings, all applied. Two were holes in the gate itself: - BLOCKING, and correct: the scan stopped at any separator, so the comma INSIDE `dict[str, int]` halted it before the closer and the gate cleared -- `[OPTIONS: A | B then inspect dict[str, int]` still ate its line. The scan now crosses `,` and stops only at `|`, a bracket, or the end. The asymmetry is the point: `|` is the unambiguous option separator, while `,` is only the fallback one (`parseOptions` reaches for it just when no `|` is present) and inside brackets it is ordinary punctuation models write constantly. - WATCH, also correct: the TRAILER scan excluded `\n` while its body spans them under DOTALL, so a newline blinded the gate and `[OPTIONS: A | B then check arr[0\nAnd that is all]` walked through. The two scans now differ deliberately -- LINE stops at a newline because its body does, TRAILER does not because its body does not -- and both halves are pinned so the asymmetry cannot read as an oversight. Cost of the comma change, enumerated with the others: a stray opener whose only following separator is a comma (`[OPTIONS: Fix [x logging, Skip]`) joins the given-up family. Same direction -- visible marker, nothing deleted. Two were about keeping the file honest: - The `_MARKER_LABEL_CONTINUES` docstring and `optionMarker.ts` both still said the separator-tail form was "deliberately unchanged... a separate call", which this change contradicts three screens below. Both rewritten to point at the gate. - `OPTIONS_RE_LINE` / `OPTIONS_RE_TRAILER` now build their tails from `_MARKER_STRAY_TIC` / `_MARKER_WRAP_RUN` instead of respelling them, so a tail edit cannot drift from the gate that has to mirror it exactly. Review round 2: the gate is a PARTIAL fix, and that is now recorded rather than implied. The blocking finding is correct: nested terminal brackets bypass it. `[OPTIONS: A | B then inspect list[dict[str, int]]` still matches, because the scan halts at the inner `[` and never reaches the outer opener's partner -- which is the terminator. Measured: depth one is fixed, depth two and three behave as on main. Encoding one nesting level into the scan only moves the boundary; depth three defeats that, depth four the next. The condition the gate reaches for is "the terminating closer is not nested", which is bracket BALANCE, and balance is not a regular language at unbounded depth. So this cannot be closed inside the pattern, and a check outside it would have to be added at all five backend call sites plus both frontend entry points -- where any future caller using the raw regex silently loses it, which is the drift this module exists to prevent. What this change therefore claims: it closes the single-bracket form, which is the common one in prose (`arr[0]`, `docs[4]`, `CHANGELOG[1]`), plus the cross-line and separator-tail variants, and regresses nothing. The residual is pinned at the depth where the gate stops reaching, in `TestTheResidualThisGateCannotClose`, and named in the constant's own docstring, so the boundary is a measured cost rather than a surprise for the next reader. * fix(protocol): decide the marker terminator outside the pattern An `[OPTIONS:` head the model never closed consumed the rest of its line whenever the line held any later `]`: [OPTIONS: A | B then check arr[0] labels = ' A | B then check arr[0' visible = '' The only closer belongs to `arr[0]`, so the body ran on through the prose and that `]` became the terminator. Every consumer removes the whole match -- `parse_options` replaces, `slack.format` and `messaging.renderer` cut at `match.start()`, and `whatsapp.turn_renderer` PERSISTS the cut turn -- so the line left the message and came back as the pill label `B then check arr[0`. THE PATTERN CANNOT DECIDE THIS, and three review rounds on an earlier attempt are the evidence. The body admits a bare `[` on purpose so a stray opener does not sink the marker (`[OPTIONS: Fix [x logging | Skip]` parses, and is pinned). Reduced to their bracket structure the shape to refuse and the shape to accept are one string: [OPTIONS: Fix | Skip [x logging] -> w|w[w] [OPTIONS: A | B then check arr[0] -> w|w[w] A lookahead can only see one nesting level, so `list[dict[str, int]]` defeats a one-level rule and `a[b[c[d]]]` a two-level one. The condition wanted is "the terminator is not nested" -- bracket BALANCE, not a regular language at unbounded depth. Encoding depths is a treadmill; the check has to leave the pattern. So it does. `_marker_terminator_is_nested` walks the captured labels with a stack: if an opener is still open at the end, the closer the pattern took as the terminator is really THAT opener's partner -- unless a `|` follows it, which means it sits inside a label and the list continues past it. `,` is deliberately not enough: it is only the fallback separator and inside brackets it is ordinary punctuation (`dict[str, Any]`, `arr[i, j]`). An unmatched CLOSER is ignored, because a label may legitimately carry one. That single rule decides every depth. AND THE CHECK CANNOT BE BYPASSED, which is the other half of the fix. A check that callers must remember to call is one a future caller forgets, and this module's comments call the grammar a single source of truth precisely to prevent that drift. So the compiled patterns are now private (`_RAW_OPTIONS_RE_*`, and module-scoped in `optionMarker.ts`), and what callers get applies both halves: - backend: `OPTIONS_RE_LINE` / `OPTIONS_RE_TRAILER` are matchers exposing the four members anything used -- `search`, `finditer`, `sub`, `pattern` -- so all five call sites and 94 existing test references read unchanged and every one of them gains the check. - frontend: `findOptionMarkers`, `findLastOptionMarker`, `stripOptionMarkers`, `terminatorIsNested`, plus `OPTION_MARKER_PATTERN_SOURCE` as a STRING for the shape pin. Nothing callable leaks, so nothing can be applied by halves. Making the pattern private found two callers I had not enumerated -- `TurnBlock.tsx` and `searchableText.ts` -- which were using the raw regex and would have kept the old behaviour silently. That is the argument for the indirection, discovered by the typechecker rather than by review. Two side benefits worth naming. The g-flag `lastIndex` hazard the frontend docs warned every caller about is gone, because the functions clone internally; its test now asserts the stronger property that no `RegExp` is exported at all. And the separator-tail form `Done. [OPTIONS: Merge | Wait], details in CHANGELOG[1]`, which `constants.py` documented as unresolved ("deciding which shape loses -- a separate call"), is decided: the `[` of `CHANGELOG[1]` is still open at the labels' end, so the line stays whole instead of losing its tail to a pill. COST: a stray opener in the FINAL label, where no `|` follows to make it label text -- `[OPTIONS: Fix | Skip [x logging]`, `[OPTIONS: Fix [x logging, Skip]`. Fails toward a VISIBLE marker with nothing removed, the direction every cost in this grammar fails in, and asserted at the consumer rather than only at the regex. Verified by differential, not inspection: all 96 marker shapes in main's five marker suites run on both the old and new decision, and the only behaviour change is the separator-tail form. Cross-surface: 32 markers compared on labels AND stripped text, plus 10 label runs on the rule itself -- zero disagreements. 3802 backend tests, 27311 frontend. The check is one pass with a stack, pinned linear at 200k openers. Fixes kirodotdev#10058 Review round 1 on the refactor: the rule is now TOTAL, with no separator hatch. The blocking finding is correct. The hatch -- "an unmatched opener is label text if a `|` follows it" -- is satisfied by a `|` INSIDE the unmatched bracket, so `[OPTIONS: A | B then inspect dict[str | int]` cleared it and the line was eaten again. That is the third defeat of the same hatch, after a separator stopping a scan early and a newline blinding it, and each time the shape it readmitted was structurally identical to the shape it was meant to protect. The hatch was the defect, not its spelling. So the rule is simply: a marker's labels must have BALANCED brackets. No depth limit, no separator heuristic, no exceptions. `_marker_labels_have_unmatched_opener` / `labelsHaveUnmatchedOpener` -- renamed, because the old names described the hatch. Measured cost, over all 96 marker shapes in main's five suites: exactly ONE shape changes beyond the intended fixes -- `[OPTIONS: Fix [x logging | Skip]`, a label carrying an unclosed `[`, is refused and renders as visible text. That shape was a deliberate allowance ("does not make the pair a REQUIREMENT") taken when the alternative was unknown; the alternative is deleting a line of prose, since both shapes hold one unmatched opener and a closer at the end anchor. Its two pins are rewritten, one per surface, with that argument in them rather than in a commit message. Cross-surface differential extended to the new shapes: 34 markers on labels and stripped text, 12 label runs on the rule -- zero disagreements. 3804 backend tests, 217 frontend marker tests. --------- Co-authored-by: Chen Tong <cttong@amazon.com>
Problem / Motivation
An
[OPTIONS:head the model never closed consumed the rest of its line whenever the line held any later]:The only closer on that line belongs to
arr[0], so the body ran on through the prose and that]became the marker's terminator.Why it matters
Every consumer removes the whole match —
parse_optionsreplaces it,slack/format.pyandmessaging/renderer.pycut atmatch.start(), andwhatsapp/turn_renderer.pypersists the cut turn. So on the channels that cannot render buttons the prose is unrecoverable, and the user is shown pills readingAandB then check arr[0. A dropped closer is an ordinary model slip; its blast radius was the rest of the line, and a paragraph under theDOTALLtrailer form.Why this is not a regex change
The body admits a bare
[on purpose, so that a stray opener inside a label did not sink the marker —[OPTIONS: Fix [x logging | Skip]was pinned as parsing on both surfaces. But that shape and the defect are the same string once reduced to their bracket structure:So no rule over brackets alone can accept one and refuse the other — which is why the allowance had to go, and it is the single accepted cost below. Separately, a lookahead inside the pattern sees only one nesting level:
list[dict[str, int]]defeats a one-level rule,a[b[c[d]]]a two-level one. The condition wanted is "the terminating closer is not nested" — bracket balance, which is not a regular language at unbounded depth. Encoding depths is a treadmill.I know that from having walked it: an earlier revision of this branch put the rule in the pattern and took three review rounds, each one a locally-correct patch that exposed the next reachable shape (a separator stopping the scan early, a newline blinding the trailer scan, then nesting). The commit history has the detail.
What changed
The decision left the pattern, and the rule is total. A marker's labels must have balanced brackets: an unmatched opener means the closer the pattern took as the terminator is really that opener's partner, so the marker was never closed. No depth limit, no separator heuristic, no exceptions.
It got there the hard way. An earlier form carried an escape hatch — "an unmatched opener is label text if a
|follows it" — and review defeated it three times: a separator stopping the scan early, a newline blinding the trailer scan, and finally a|inside the unmatched bracket (dict[str | int]) satisfying it. Each time, the shape the hatch readmitted was structurally identical to the shape it was meant to protect. The hatch was the defect, not its spelling.One thing the rule deliberately ignores: an unmatched closer. A label may legitimately carry one —
[OPTIONS: Alpha ] | Bravo ]]is supported and tested — so it says nothing about the terminator.And the check cannot be bypassed, which is the other half of the fix. A check callers must remember to call is one a future caller forgets, and this module's own comments call the grammar a single source of truth precisely to stop that drift. So the compiled patterns are private now, and what callers get applies both halves:
OPTIONS_RE_LINE/OPTIONS_RE_TRAILER— matchers exposingsearch,finditer,sub,pattern_RAW_OPTIONS_RE_LINE/_RAW_OPTIONS_RE_TRAILERfindOptionMarkers,findLastOptionMarker,stripOptionMarkers,labelsHaveUnmatchedOpener,OPTION_MARKER_PATTERN_SOURCE(a string)Those four members are exactly what anything used, so all five backend call sites and 94 existing test references read unchanged and every one of them gains the check. On the frontend nothing callable leaks, so the two halves cannot be applied separately.
Making the pattern private found two callers I had not enumerated —
TurnBlock.tsxandsearchableText.ts— both using the raw regex, both of which would have kept the old behaviour silently. The typechecker found them, not review. That is the argument for the indirection, demonstrated.Two things fall out of it
The
lastIndexhazard is gone. The frontend docs warned every caller thatmatchAllseeds its clone from a shared g-flagged regex'slastIndex, so one stray.test()anywhere would make a scan start past the marker. The functions clone internally, so there is no shared handle to corrupt; the test that used to pokelastIndexnow asserts the stronger property that the module exports noRegExpat all.A shape the grammar documented as unresolved is resolved.
constants.pysaid ofDone. [OPTIONS: Merge | Wait], details in CHANGELOG[1]:True of a guard at the internal closer. Counting from the other end decides it: the
[ofCHANGELOG[1]is still open when the labels end and no|follows, so the line stays whole instead of losing its tail to a pill readingWait], details in CHANGELOG[1. Its two pins are updated, one per surface.Accepted cost
Exactly one shape, measured across all 96 marker shapes in
main's suites: a label carrying an unclosed[.The first was a deliberate allowance — its test said the grammar "does not make the pair a REQUIREMENT" — taken when the alternative was unknown. The alternative turns out to be deleting a line of prose:
[OPTIONS: Fix [x logging | Skip]and[OPTIONS: A | B then check arr[0]both hold one unmatched opener and a closer at the end anchor, so accepting either accepts both. Its two pins are rewritten, one per surface, carrying that argument.All of them fail toward a visible marker with nothing removed — the direction every cost in this grammar fails in — and that is asserted at the consumer, not only at the pattern.
Tests
test/test_options_marker_terminator.pyandwebsite/src/test/optionsMarkerTerminator.test.ts, including the rule tested directly in isolation on both surfaces.main's five marker suites, run on both the old and the new decision. The only behaviour changes are the separator-tail form and the one accepted cost below. That method is what caught the four pinned shapes an earlier pattern-level attempt would have broken.slack,whatsapp,feishu,imessage,weixin). Frontend: 27311 passed, whole suite.Manual verification
N/A, same reasoning as #9341: the trigger is what the model happens to emit, not something a user can type. Both surfaces are exercised at the parser level, which is the only code the marker passes through, and the end-to-end assertions go through
split_options_trailer/parseOptionsrather than the regex alone.One pre-existing divergence observed, not fixed here
For
[OPTIONS: Alpha ], Bravo]the backend yields['Alpha ], Bravo']while the frontend yields['Alpha ]', 'Bravo'], becauseparseOptionspicks its delimiter aslabels.includes('|') ? '|' : ','and the backend splits on|only. Untouched by this diff and out of scope; flagged because it means the surfaces can disagree about how many options a comma-only marker has.Related Issues
Fixes #10058
Use [OPTIONS: A | B] then check arr[0]) and could not reach this one; both remain pinned.Pattern harvest
Rule candidate: review-prompt
Pattern: when a grammar has a forgiving fallback — a bare opener, an optional delimiter, a lenient escape — check what that forgiveness does at the END of the input, where the terminator lives. A fallback that is harmless mid-input can let an unterminated construct swallow everything up to the next character satisfying the end anchor, and because parse-and-strip consumers remove the whole match, "mis-parsed" becomes "deleted".
Three general follow-ons, each earned here. First: before writing the rule, reduce the shape you must REFUSE and the shape you must ACCEPT to their structural skeletons; if the skeletons collide, no rule over that structure exists and you need a different discriminator — that check is cheap and it saved a fix that would have broken four pinned shapes. Second: if the discriminator needs unbounded counting (bracket balance, nesting depth, matching quotes), a regex cannot express it, and adding one more level is a treadmill — three review rounds here, each closing one depth. Move the decision out and keep the pattern as a candidate matcher. Third: when you move a check out of a pattern, make the unchecked form unreachable in the same change — private the raw pattern and hand out functions. Otherwise the check is advice, and the typechecker will show you the callers who were never going to take it.