Skip to content

fix(protocol): decide the marker terminator outside the pattern - #10084

Merged
bolichen97 merged 2 commits into
mainfrom
fix/options-terminator-not-a-nested-opener
Sep 12, 2026
Merged

bolichen97 merged 2 commits into
mainfrom
fix/options-terminator-not-a-nested-opener

Conversation

@cixuuz

@cixuuz cixuuz commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

An [OPTIONS: head the model never closed consumed the rest of its line whenever the line held any later ]:

[OPTIONS: A | B then check arr[0]
  labels  = ' A | B then check arr[0'
  visible = ''                          ← the whole line is gone

The only closer on that line belongs to arr[0], so the body ran on through the prose and that ] became the marker's terminator.

Why it matters

Every consumer removes the whole match — parse_options replaces it, slack/format.py and messaging/renderer.py cut at match.start(), and whatsapp/turn_renderer.py persists the cut turn. So on the channels that cannot render buttons the prose is unrecoverable, and the user is shown pills reading A and B then check arr[0. A dropped closer is an ordinary model slip; its blast radius was the rest of the line, and a paragraph under the DOTALL trailer form.

Why this is not a regex change

The body admits a bare [ on purpose, so that a stray opener inside a label did not sink the marker — [OPTIONS: Fix [x logging | Skip] was pinned as parsing on both surfaces. But that shape and the defect are the same string once reduced to their bracket structure:

[OPTIONS: Fix | Skip [x logging]    ->  w|w[w]
[OPTIONS: A | B then check arr[0]   ->  w|w[w]

So no rule over brackets alone can accept one and refuse the other — which is why the allowance had to go, and it is the single accepted cost below. Separately, a lookahead inside the pattern sees only one nesting level: list[dict[str, int]] defeats a one-level rule, a[b[c[d]]] a two-level one. The condition wanted is "the terminating closer is not nested" — bracket balance, which is not a regular language at unbounded depth. Encoding depths is a treadmill.

I know that from having walked it: an earlier revision of this branch put the rule in the pattern and took three review rounds, each one a locally-correct patch that exposed the next reachable shape (a separator stopping the scan early, a newline blinding the trailer scan, then nesting). The commit history has the detail.

What changed

The decision left the pattern, and the rule is total. A marker's labels must have balanced brackets: an unmatched opener means the closer the pattern took as the terminator is really that opener's partner, so the marker was never closed. No depth limit, no separator heuristic, no exceptions.

It got there the hard way. An earlier form carried an escape hatch — "an unmatched opener is label text if a | follows it" — and review defeated it three times: a separator stopping the scan early, a newline blinding the trailer scan, and finally a | inside the unmatched bracket (dict[str | int]) satisfying it. Each time, the shape the hatch readmitted was structurally identical to the shape it was meant to protect. The hatch was the defect, not its spelling.

One thing the rule deliberately ignores: an unmatched closer. A label may legitimately carry one — [OPTIONS: Alpha ] | Bravo ]] is supported and tested — so it says nothing about the terminator.

And the check cannot be bypassed, which is the other half of the fix. A check callers must remember to call is one a future caller forgets, and this module's own comments call the grammar a single source of truth precisely to stop that drift. So the compiled patterns are private now, and what callers get applies both halves:

surface public API private
backend OPTIONS_RE_LINE / OPTIONS_RE_TRAILER — matchers exposing search, finditer, sub, pattern _RAW_OPTIONS_RE_LINE / _RAW_OPTIONS_RE_TRAILER
frontend findOptionMarkers, findLastOptionMarker, stripOptionMarkers, labelsHaveUnmatchedOpener, OPTION_MARKER_PATTERN_SOURCE (a string) the pattern is module-scoped

Those four members are exactly what anything used, so all five backend call sites and 94 existing test references read unchanged and every one of them gains the check. On the frontend nothing callable leaks, so the two halves cannot be applied separately.

Making the pattern private found two callers I had not enumerated — TurnBlock.tsx and searchableText.ts — both using the raw regex, both of which would have kept the old behaviour silently. The typechecker found them, not review. That is the argument for the indirection, demonstrated.

Two things fall out of it

The lastIndex hazard is gone. The frontend docs warned every caller that matchAll seeds its clone from a shared g-flagged regex's lastIndex, so one stray .test() anywhere would make a scan start past the marker. The functions clone internally, so there is no shared handle to corrupt; the test that used to poke lastIndex now asserts the stronger property that the module exports no RegExp at all.

A shape the grammar documented as unresolved is resolved. constants.py said of Done. [OPTIONS: Merge | Wait], details in CHANGELOG[1]:

], DOES continue the label list … so no guard applied at the internal closer can tell them apart. Resolving it means deciding which shape loses — a separate call.

True of a guard at the internal closer. Counting from the other end decides it: the [ of CHANGELOG[1] is still open when the labels end and no | follows, so the line stays whole instead of losing its tail to a pill reading Wait], details in CHANGELOG[1. Its two pins are updated, one per surface.

Accepted cost

Exactly one shape, measured across all 96 marker shapes in main's suites: a label carrying an unclosed [.

[OPTIONS: Fix [x logging | Skip]     refused   ← was pinned as parsing
[OPTIONS: Fix | Skip [x logging]     refused
[OPTIONS: Fix [x logging]            refused

The first was a deliberate allowance — its test said the grammar "does not make the pair a REQUIREMENT" — taken when the alternative was unknown. The alternative turns out to be deleting a line of prose: [OPTIONS: Fix [x logging | Skip] and [OPTIONS: A | B then check arr[0] both hold one unmatched opener and a closer at the end anchor, so accepting either accepts both. Its two pins are rewritten, one per surface, carrying that argument.

All of them fail toward a visible marker with nothing removed — the direction every cost in this grammar fails in — and that is asserted at the consumer, not only at the pattern.

Tests

  • New: test/test_options_marker_terminator.py and website/src/test/optionsMarkerTerminator.test.ts, including the rule tested directly in isolation on both surfaces.
  • Differential, not inspection. All 96 marker shapes appearing in main's five marker suites, run on both the old and the new decision. The only behaviour changes are the separator-tail form and the one accepted cost below. That method is what caught the four pinned shapes an earlier pattern-level attempt would have broken.
  • Depth is exercised, not assumed: nesting from depth 1 to 6 is refused in a loop, so a future change that reintroduces a depth-limited rule fails rather than regressing quietly.
  • Cross-surface: 34 markers compared on labels and stripped text, plus 12 label runs on the rule itself — zero disagreements. Both surfaces must agree or a marker becomes buttons on one and prose on the other.
  • Backend: 3804 passed across the marker, renderer, preview and channel suites (slack, whatsapp, feishu, imessage, weixin). Frontend: 27311 passed, whole suite.
  • Cost: one pass with a depth counter. Pinned linear at 200k openers on both surfaces, and the pattern's own shape pin still forbids a quantifier over a quantifier.

Manual verification

N/A, same reasoning as #9341: the trigger is what the model happens to emit, not something a user can type. Both surfaces are exercised at the parser level, which is the only code the marker passes through, and the end-to-end assertions go through split_options_trailer / parseOptions rather than the regex alone.

One pre-existing divergence observed, not fixed here

For [OPTIONS: Alpha ], Bravo] the backend yields ['Alpha ], Bravo'] while the frontend yields ['Alpha ]', 'Bravo'], because parseOptions picks its delimiter as labels.includes('|') ? '|' : ',' and the backend splits on | only. Untouched by this diff and out of scope; flagged because it means the surfaces can disagree about how many options a comma-only marker has.

Related Issues

Fixes #10058

Pattern harvest

Rule candidate: review-prompt

Pattern: when a grammar has a forgiving fallback — a bare opener, an optional delimiter, a lenient escape — check what that forgiveness does at the END of the input, where the terminator lives. A fallback that is harmless mid-input can let an unterminated construct swallow everything up to the next character satisfying the end anchor, and because parse-and-strip consumers remove the whole match, "mis-parsed" becomes "deleted".

Three general follow-ons, each earned here. First: before writing the rule, reduce the shape you must REFUSE and the shape you must ACCEPT to their structural skeletons; if the skeletons collide, no rule over that structure exists and you need a different discriminator — that check is cheap and it saved a fix that would have broken four pinned shapes. Second: if the discriminator needs unbounded counting (bracket balance, nesting depth, matching quotes), a regex cannot express it, and adding one more level is a treadmill — three review rounds here, each closing one depth. Move the decision out and keep the pattern as a candidate matcher. Third: when you move a check out of a pattern, make the unchecked form unreachable in the same change — private the raw pattern and hand out functions. Otherwise the check is advice, and the typechecker will show you the callers who were never going to take it.

@cixuuz
cixuuz requested review from a team and buluoray September 11, 2026 07:38
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 11, 2026
@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ✅ PASS

UX-level review of acb83483c0c6f1bc150f5c295a7986d9e6ab52dd — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

This diff is entirely protocol/parser logic and tests — no control, label, string, or persistent-element state a user sees is added or changed. The reconcile table is empty: there was nothing for a blind read to look at (the report confirms no screenshots existed, correctly, since none were needed). The one behavioral surface change fails toward visible prose instead of deleted prose, which is the pre-existing failure direction and strictly improves what the user experiences.

UX-Verdict: PASS

Pure parser fix with no new user-visible surface; prose the old pattern silently deleted now stays on screen, failing visible rather than destructive.

[UX-REVIEWED] acb8348

@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of acb83483c0c6f1bc150f5c295a7986d9e6ab52dd and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] acb8348

False positive or not applicable? A repository writer can comment:
/ai-review override gpt acb83483c0c6f1bc150f5c295a7986d9e6ab52dd: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of acb83483c0c6f1bc150f5c295a7986d9e6ab52dd — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All claims verified against the repo: five backend call-site files (preview_text.py, slack/format.py, dashboard/state.py, whatsapp/turn_renderer.py, messaging/renderer.py) use only search/finditer/sub; no caller anywhere uses a re.Pattern method the matcher lacks; the frontend's new functions have real consumers (options.ts, TurnBlock.tsx, searchableText.ts); the two rewritten pins carry in-diff evidence (the skeleton-equality assertion) for why the prior decision could not stand; the barrel never exported the regex, so no app-surface change. The review below is the deliverable.

First-Principles-Verdict: PASS

Verify the accepted cost against live traffic: a stray unclosed [ in a label — pinned as parsing until now — renders as raw marker text.

Not justified as shipped

  1. lastIndex hazard removal — rides along: harm-free fallout of item 5, declared, no surface of its own.
  2. _MARKER_STRAY_TIC/_MARKER_WRAP_RUN tail constants — undeclared: private dedup of two inline spellings; harm-free rider.

What this change ships

Inventory (8 items) — 6 justified

Intent: stop an unclosed [OPTIONS: head from silently deleting the user's prose on every channel — a FIX.

  1. An unterminated marker no longer eats its line/paragraph; it shows as text — justified
  2. A label with a stray unclosed [ no longer becomes buttons — justified
  3. Separator-tail form (…Wait], details in CHANGELOG[1]) left whole instead of truncated — justified
  4. Backend patterns private behind four-member matchers; 5 call-site files counted, only search/finditer/sub used — justified
  5. Frontend regex module-private; three functions replace it at 3 real consumer files — justified
  6. Shared-regex lastIndex corruption hazard gone — rides along
  7. labelsHaveUnmatchedOpener + OPTION_MARKER_PATTERN_SOURCE exported for the parity and ReDoS-shape pins — justified
  8. Marker-tail regex fragments named as private constants — undeclared

The defect's provenance is the new tests, which fail on base (test_the_raw_pattern_still_over_matches shows base's pattern accepts the shape). Items 2 and 3 rewrite pins whose own words deferred the call ("a separate call"); the in-diff skeleton-equality proof settles it, satisfying the deleted-pin bar. The fix sits at cause level: the decision (bracket balance) is not expressible in the pattern, so it moved out, and privatizing the raw patterns is what makes the fix total — demonstrated by the two callers the typechecker caught.

[FIRST-PRINCIPLES-REVIEWED] acb8348

@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of acb83483c0c6f1bc150f5c295a7986d9e6ab52dd — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Non-regular decision (bracket balance) correctly moved out of the regex, with the raw patterns made unreachable so the check can't be skipped — right root-cause shape.

The one rewritten pin (Fix [x logging | Skip] parsing) is treated as a reversed decision, not a gap: the description quotes the old pin's rationale, proves by skeleton collision that no bracket rule can keep it while refusing the defect shape, and fails the cost toward visible text — the direction every consumer (including WhatsApp's persisted cut) is asserted to fail in. Differential testing over all 96 prior shapes plus cross-surface agreement runs cover the compat story. All five backend call sites use only the four wrapped members, verified against the tree.

Suggestions

  • _MarkerMatcher.sub inserts repl literally, silently diverging from re.Pattern.sub escape/backref semantics for a future non-empty caller — assert repl == "" (all current callers) or expand via match.expand.

[DESIGN-REVIEWED] acb8348

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 11, 2026
@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed acb83483c0c6f1bc150f5c295a7986d9e6ab52dd — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] acb8348

Verdict parsed from the review's SHA-scoped output markers for commit acb83483c0c6f1bc150f5c295a7986d9e6ab52dd.

False positive or not applicable? A repository writer can comment:
/ai-review override fable acb83483c0c6f1bc150f5c295a7986d9e6ab52dd: <one-sentence reason>

@cixuuz
cixuuz force-pushed the fix/options-terminator-not-a-nested-opener branch from 18df47e to 6465131 Compare September 11, 2026 08:01
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 11, 2026
An `[OPTIONS:` head the model never closed consumed the rest of its line whenever
the line held any later `]`:

    [OPTIONS: A | B then check arr[0]
      labels  = ' A | B then check arr[0'
      visible = ''

The only closer belongs to `arr[0]`, so the body ran on through the prose and that
`]` became the terminator. Every consumer removes the whole match -- `parse_options`
replaces, `slack.format` and `messaging.renderer` cut at `match.start()`, and
`whatsapp.turn_renderer` PERSISTS the cut turn -- so the line left the message and
came back as the pill label `B then check arr[0`.

The bare-`[` alternative is what admitted that opener, and it exists for a reason:
`[OPTIONS: Fix [x logging | Skip]` parses and is pinned on both surfaces. Nothing
about the brackets separates the two. Reduced to skeletons they are one string:

    [OPTIONS: Fix | Skip [x logging]   ->  w|w[w]
    [OPTIONS: A | B then check arr[0]  ->  w|w[w]

So no bracket-balance rule can accept one and refuse the other, and both are now
pinned as sharing a skeleton so a future attempt at one finds the argument.

The discriminator is where the opener sits relative to the END. The bare form is
refused exactly when nothing but ordinary text lies between it and a closer at the
end anchor: that closer is then its own partner rather than the marker's, which
means the marker is unterminated. Crossing a SEPARATOR clears the gate, because the
opener is inside a label and the list continues past it -- which is what keeps the
pinned stray-opener shape. Crossing another BRACKET clears it too, because some
other bracket form owns that closer, which is what keeps
`[OPTIONS: Fix arr[0] | Skip]` (continuation admits the `]`) and
`[OPTIONS: a[1] | b[2]]`.

Spelled twice per surface because the two bodies end differently (`$` under
MULTILINE, `\s*\Z` under DOTALL), and both carry the optional stray markdown-link
tic and a wrapper run -- otherwise the shape returns wearing one, which is asserted.

The gate resolves a second shape the grammar documented as unresolved: the
separator-tail form `Done. [OPTIONS: Merge | Wait], details in CHANGELOG[1]`. Its
comment said `], ` genuinely continues the list so no guard at the INTERNAL closer
can tell it apart, and deciding it was "a separate call". The gate reaches it from
the other end, and the line now stays whole instead of losing its tail to a pill
reading `Wait], details in CHANGELOG[1`. Its two pins are updated, one per surface,
from "unchanged" to "declined".

COST: a stray opener in the FINAL label, where no separator follows to clear the
gate -- `[OPTIONS: Fix | Skip [x logging]`. It fails toward a VISIBLE marker with
nothing removed, the direction every cost in this grammar fails in, and it is
asserted at the consumer rather than only at the regex.

ReDoS: the lookahead is tempered, its scan stopping at the first bracket or
separator, so the runs scanned from different openers are disjoint and the work
stays linear -- measured at 0.5/2.3/10.2/31.8 ms for 2k/10k/40k/120k of `a[b`
repeated on an unterminated marker, where every gate fails. It adds no quantifier
over a quantifier, and the frontend `.source` pin is extended to cover the gate
character for character so the two surfaces cannot drift.

Verified by differential rather than by inspection: every marker shape appearing in
the five marker suites on main (96 of them) was run on both grammars, and the only
behaviour changes are the two shapes above. Backend and frontend agree on all 24
gate-relevant shapes.

Fixes #10058

Review round 1, four findings, all applied. Two were holes in the gate itself:

- BLOCKING, and correct: the scan stopped at any separator, so the comma INSIDE
  `dict[str, int]` halted it before the closer and the gate cleared --
  `[OPTIONS: A | B then inspect dict[str, int]` still ate its line. The scan now
  crosses `,` and stops only at `|`, a bracket, or the end. The asymmetry is the
  point: `|` is the unambiguous option separator, while `,` is only the fallback
  one (`parseOptions` reaches for it just when no `|` is present) and inside
  brackets it is ordinary punctuation models write constantly.

- WATCH, also correct: the TRAILER scan excluded `\n` while its body spans them
  under DOTALL, so a newline blinded the gate and
  `[OPTIONS: A | B then check arr[0\nAnd that is all]` walked through. The two
  scans now differ deliberately -- LINE stops at a newline because its body does,
  TRAILER does not because its body does not -- and both halves are pinned so the
  asymmetry cannot read as an oversight.

  Cost of the comma change, enumerated with the others: a stray opener whose only
  following separator is a comma (`[OPTIONS: Fix [x logging, Skip]`) joins the
  given-up family. Same direction -- visible marker, nothing deleted.

Two were about keeping the file honest:

- The `_MARKER_LABEL_CONTINUES` docstring and `optionMarker.ts` both still said the
  separator-tail form was "deliberately unchanged... a separate call", which this
  change contradicts three screens below. Both rewritten to point at the gate.

- `OPTIONS_RE_LINE` / `OPTIONS_RE_TRAILER` now build their tails from
  `_MARKER_STRAY_TIC` / `_MARKER_WRAP_RUN` instead of respelling them, so a tail
  edit cannot drift from the gate that has to mirror it exactly.

Review round 2: the gate is a PARTIAL fix, and that is now recorded rather than
implied.

The blocking finding is correct: nested terminal brackets bypass it.
`[OPTIONS: A | B then inspect list[dict[str, int]]` still matches, because the scan
halts at the inner `[` and never reaches the outer opener's partner -- which is the
terminator. Measured: depth one is fixed, depth two and three behave as on main.

Encoding one nesting level into the scan only moves the boundary; depth three
defeats that, depth four the next. The condition the gate reaches for is "the
terminating closer is not nested", which is bracket BALANCE, and balance is not a
regular language at unbounded depth. So this cannot be closed inside the pattern,
and a check outside it would have to be added at all five backend call sites plus
both frontend entry points -- where any future caller using the raw regex silently
loses it, which is the drift this module exists to prevent.

What this change therefore claims: it closes the single-bracket form, which is the
common one in prose (`arr[0]`, `docs[4]`, `CHANGELOG[1]`), plus the cross-line and
separator-tail variants, and regresses nothing. The residual is pinned at the depth
where the gate stops reaching, in `TestTheResidualThisGateCannotClose`, and named in
the constant's own docstring, so the boundary is a measured cost rather than a
surprise for the next reader.
@cixuuz
cixuuz force-pushed the fix/options-terminator-not-a-nested-opener branch from 6465131 to 9c1ceb7 Compare September 11, 2026 08:18
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging 9c1ceb78ad06123b95d7094fbd793c26a3862ae2. Adjudication produced no [GPT-ADJUDICATED] marker for 9c1ceb7: the adjudication stage FAILED TO RUN (its raw output is in the Adjudication details block -- typically an API or setup error), so no proportionality review took place and GPT's blocking verdict stands unreviewed.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/constants.py:357 -- pipe inside a terminal bracket bypasses the terminator gate
_MARKER_TERMINATOR_SCAN_LINE = rf"[^[{re.escape(MARKER_CLOSERS)}|\n]*"
[OPTIONS: A | B then inspect dict[str | None] -> both regexes match, consumers delete the line, and three bogus choices replace it.
Anchor: residual/crash-data-loss-corruption
Fix: Distinguish bracket-internal pipes from option separators in both backend and frontend gates.
[BLOCK-MERGE] 9c1ceb7
[GPT-REVIEWED] 9c1ceb7
False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9c1ceb78ad06123b95d7094fbd793c26a3862ae2: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 11, 2026
@cixuuz cixuuz changed the title fix(protocol): a bare opener may not be the closer that ends the marker fix(protocol): decide the marker terminator outside the pattern Sep 11, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 11, 2026
An `[OPTIONS:` head the model never closed consumed the rest of its line whenever
the line held any later `]`:

    [OPTIONS: A | B then check arr[0]
      labels  = ' A | B then check arr[0'
      visible = ''

The only closer belongs to `arr[0]`, so the body ran on through the prose and that
`]` became the terminator. Every consumer removes the whole match -- `parse_options`
replaces, `slack.format` and `messaging.renderer` cut at `match.start()`, and
`whatsapp.turn_renderer` PERSISTS the cut turn -- so the line left the message and
came back as the pill label `B then check arr[0`.

THE PATTERN CANNOT DECIDE THIS, and three review rounds on an earlier attempt are
the evidence. The body admits a bare `[` on purpose so a stray opener does not sink
the marker (`[OPTIONS: Fix [x logging | Skip]` parses, and is pinned). Reduced to
their bracket structure the shape to refuse and the shape to accept are one string:

    [OPTIONS: Fix | Skip [x logging]   ->  w|w[w]
    [OPTIONS: A | B then check arr[0]  ->  w|w[w]

A lookahead can only see one nesting level, so `list[dict[str, int]]` defeats a
one-level rule and `a[b[c[d]]]` a two-level one. The condition wanted is "the
terminator is not nested" -- bracket BALANCE, not a regular language at unbounded
depth. Encoding depths is a treadmill; the check has to leave the pattern.

So it does. `_marker_terminator_is_nested` walks the captured labels with a stack:
if an opener is still open at the end, the closer the pattern took as the terminator
is really THAT opener's partner -- unless a `|` follows it, which means it sits
inside a label and the list continues past it. `,` is deliberately not enough: it is
only the fallback separator and inside brackets it is ordinary punctuation
(`dict[str, Any]`, `arr[i, j]`). An unmatched CLOSER is ignored, because a label may
legitimately carry one. That single rule decides every depth.

AND THE CHECK CANNOT BE BYPASSED, which is the other half of the fix. A check that
callers must remember to call is one a future caller forgets, and this module's
comments call the grammar a single source of truth precisely to prevent that drift.
So the compiled patterns are now private (`_RAW_OPTIONS_RE_*`, and module-scoped in
`optionMarker.ts`), and what callers get applies both halves:

- backend: `OPTIONS_RE_LINE` / `OPTIONS_RE_TRAILER` are matchers exposing the four
  members anything used -- `search`, `finditer`, `sub`, `pattern` -- so all five call
  sites and 94 existing test references read unchanged and every one of them gains
  the check.
- frontend: `findOptionMarkers`, `findLastOptionMarker`, `stripOptionMarkers`,
  `terminatorIsNested`, plus `OPTION_MARKER_PATTERN_SOURCE` as a STRING for the
  shape pin. Nothing callable leaks, so nothing can be applied by halves.

Making the pattern private found two callers I had not enumerated -- `TurnBlock.tsx`
and `searchableText.ts` -- which were using the raw regex and would have kept the
old behaviour silently. That is the argument for the indirection, discovered by the
typechecker rather than by review.

Two side benefits worth naming. The g-flag `lastIndex` hazard the frontend docs
warned every caller about is gone, because the functions clone internally; its test
now asserts the stronger property that no `RegExp` is exported at all. And the
separator-tail form `Done. [OPTIONS: Merge | Wait], details in CHANGELOG[1]`, which
`constants.py` documented as unresolved ("deciding which shape loses -- a separate
call"), is decided: the `[` of `CHANGELOG[1]` is still open at the labels' end, so
the line stays whole instead of losing its tail to a pill.

COST: a stray opener in the FINAL label, where no `|` follows to make it label text
-- `[OPTIONS: Fix | Skip [x logging]`, `[OPTIONS: Fix [x logging, Skip]`. Fails
toward a VISIBLE marker with nothing removed, the direction every cost in this
grammar fails in, and asserted at the consumer rather than only at the regex.

Verified by differential, not inspection: all 96 marker shapes in main's five marker
suites run on both the old and new decision, and the only behaviour change is the
separator-tail form. Cross-surface: 32 markers compared on labels AND stripped text,
plus 10 label runs on the rule itself -- zero disagreements. 3802 backend tests,
27311 frontend. The check is one pass with a stack, pinned linear at 200k openers.

Fixes #10058

Review round 1 on the refactor: the rule is now TOTAL, with no separator hatch.

The blocking finding is correct. The hatch -- "an unmatched opener is label text if a
`|` follows it" -- is satisfied by a `|` INSIDE the unmatched bracket, so
`[OPTIONS: A | B then inspect dict[str | int]` cleared it and the line was eaten
again. That is the third defeat of the same hatch, after a separator stopping a scan
early and a newline blinding it, and each time the shape it readmitted was
structurally identical to the shape it was meant to protect. The hatch was the
defect, not its spelling.

So the rule is simply: a marker's labels must have BALANCED brackets. No depth limit,
no separator heuristic, no exceptions. `_marker_labels_have_unmatched_opener` /
`labelsHaveUnmatchedOpener` -- renamed, because the old names described the hatch.

Measured cost, over all 96 marker shapes in main's five suites: exactly ONE shape
changes beyond the intended fixes -- `[OPTIONS: Fix [x logging | Skip]`, a label
carrying an unclosed `[`, is refused and renders as visible text. That shape was a
deliberate allowance ("does not make the pair a REQUIREMENT") taken when the
alternative was unknown; the alternative is deleting a line of prose, since both
shapes hold one unmatched opener and a closer at the end anchor. Its two pins are
rewritten, one per surface, with that argument in them rather than in a commit
message.

Cross-surface differential extended to the new shapes: 34 markers on labels and
stripped text, 12 label runs on the rule -- zero disagreements. 3804 backend tests,
217 frontend marker tests.
@cixuuz
cixuuz force-pushed the fix/options-terminator-not-a-nested-opener branch from c102852 to acb8348 Compare September 11, 2026 18:43
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 11, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer review (agent-assisted), pinned to acb8348. All five backend consumers were checked: search/finditer/sub/pattern only, with empty-string replacement. The matcher preserves these contracts and the frontend uses the same closer set. Owner-delegated acceptance: an unclosed '[' in a label now renders literally. Dedicated-worktree source/test review; no local suite rerun this wave; current-head readiness success verified.

@bolichen97
bolichen97 merged commit 17e1eed into main Sep 12, 2026
72 of 79 checks passed
@bolichen97
bolichen97 deleted the fix/options-terminator-not-a-nested-opener branch September 12, 2026 04:18
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 12, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…dotdev#10084)

* fix(protocol): a bare opener may not be the closer that ends the marker

An `[OPTIONS:` head the model never closed consumed the rest of its line whenever
the line held any later `]`:

    [OPTIONS: A | B then check arr[0]
      labels  = ' A | B then check arr[0'
      visible = ''

The only closer belongs to `arr[0]`, so the body ran on through the prose and that
`]` became the terminator. Every consumer removes the whole match -- `parse_options`
replaces, `slack.format` and `messaging.renderer` cut at `match.start()`, and
`whatsapp.turn_renderer` PERSISTS the cut turn -- so the line left the message and
came back as the pill label `B then check arr[0`.

The bare-`[` alternative is what admitted that opener, and it exists for a reason:
`[OPTIONS: Fix [x logging | Skip]` parses and is pinned on both surfaces. Nothing
about the brackets separates the two. Reduced to skeletons they are one string:

    [OPTIONS: Fix | Skip [x logging]   ->  w|w[w]
    [OPTIONS: A | B then check arr[0]  ->  w|w[w]

So no bracket-balance rule can accept one and refuse the other, and both are now
pinned as sharing a skeleton so a future attempt at one finds the argument.

The discriminator is where the opener sits relative to the END. The bare form is
refused exactly when nothing but ordinary text lies between it and a closer at the
end anchor: that closer is then its own partner rather than the marker's, which
means the marker is unterminated. Crossing a SEPARATOR clears the gate, because the
opener is inside a label and the list continues past it -- which is what keeps the
pinned stray-opener shape. Crossing another BRACKET clears it too, because some
other bracket form owns that closer, which is what keeps
`[OPTIONS: Fix arr[0] | Skip]` (continuation admits the `]`) and
`[OPTIONS: a[1] | b[2]]`.

Spelled twice per surface because the two bodies end differently (`$` under
MULTILINE, `\s*\Z` under DOTALL), and both carry the optional stray markdown-link
tic and a wrapper run -- otherwise the shape returns wearing one, which is asserted.

The gate resolves a second shape the grammar documented as unresolved: the
separator-tail form `Done. [OPTIONS: Merge | Wait], details in CHANGELOG[1]`. Its
comment said `], ` genuinely continues the list so no guard at the INTERNAL closer
can tell it apart, and deciding it was "a separate call". The gate reaches it from
the other end, and the line now stays whole instead of losing its tail to a pill
reading `Wait], details in CHANGELOG[1`. Its two pins are updated, one per surface,
from "unchanged" to "declined".

COST: a stray opener in the FINAL label, where no separator follows to clear the
gate -- `[OPTIONS: Fix | Skip [x logging]`. It fails toward a VISIBLE marker with
nothing removed, the direction every cost in this grammar fails in, and it is
asserted at the consumer rather than only at the regex.

ReDoS: the lookahead is tempered, its scan stopping at the first bracket or
separator, so the runs scanned from different openers are disjoint and the work
stays linear -- measured at 0.5/2.3/10.2/31.8 ms for 2k/10k/40k/120k of `a[b`
repeated on an unterminated marker, where every gate fails. It adds no quantifier
over a quantifier, and the frontend `.source` pin is extended to cover the gate
character for character so the two surfaces cannot drift.

Verified by differential rather than by inspection: every marker shape appearing in
the five marker suites on main (96 of them) was run on both grammars, and the only
behaviour changes are the two shapes above. Backend and frontend agree on all 24
gate-relevant shapes.

Fixes kirodotdev#10058

Review round 1, four findings, all applied. Two were holes in the gate itself:

- BLOCKING, and correct: the scan stopped at any separator, so the comma INSIDE
  `dict[str, int]` halted it before the closer and the gate cleared --
  `[OPTIONS: A | B then inspect dict[str, int]` still ate its line. The scan now
  crosses `,` and stops only at `|`, a bracket, or the end. The asymmetry is the
  point: `|` is the unambiguous option separator, while `,` is only the fallback
  one (`parseOptions` reaches for it just when no `|` is present) and inside
  brackets it is ordinary punctuation models write constantly.

- WATCH, also correct: the TRAILER scan excluded `\n` while its body spans them
  under DOTALL, so a newline blinded the gate and
  `[OPTIONS: A | B then check arr[0\nAnd that is all]` walked through. The two
  scans now differ deliberately -- LINE stops at a newline because its body does,
  TRAILER does not because its body does not -- and both halves are pinned so the
  asymmetry cannot read as an oversight.

  Cost of the comma change, enumerated with the others: a stray opener whose only
  following separator is a comma (`[OPTIONS: Fix [x logging, Skip]`) joins the
  given-up family. Same direction -- visible marker, nothing deleted.

Two were about keeping the file honest:

- The `_MARKER_LABEL_CONTINUES` docstring and `optionMarker.ts` both still said the
  separator-tail form was "deliberately unchanged... a separate call", which this
  change contradicts three screens below. Both rewritten to point at the gate.

- `OPTIONS_RE_LINE` / `OPTIONS_RE_TRAILER` now build their tails from
  `_MARKER_STRAY_TIC` / `_MARKER_WRAP_RUN` instead of respelling them, so a tail
  edit cannot drift from the gate that has to mirror it exactly.

Review round 2: the gate is a PARTIAL fix, and that is now recorded rather than
implied.

The blocking finding is correct: nested terminal brackets bypass it.
`[OPTIONS: A | B then inspect list[dict[str, int]]` still matches, because the scan
halts at the inner `[` and never reaches the outer opener's partner -- which is the
terminator. Measured: depth one is fixed, depth two and three behave as on main.

Encoding one nesting level into the scan only moves the boundary; depth three
defeats that, depth four the next. The condition the gate reaches for is "the
terminating closer is not nested", which is bracket BALANCE, and balance is not a
regular language at unbounded depth. So this cannot be closed inside the pattern,
and a check outside it would have to be added at all five backend call sites plus
both frontend entry points -- where any future caller using the raw regex silently
loses it, which is the drift this module exists to prevent.

What this change therefore claims: it closes the single-bracket form, which is the
common one in prose (`arr[0]`, `docs[4]`, `CHANGELOG[1]`), plus the cross-line and
separator-tail variants, and regresses nothing. The residual is pinned at the depth
where the gate stops reaching, in `TestTheResidualThisGateCannotClose`, and named in
the constant's own docstring, so the boundary is a measured cost rather than a
surprise for the next reader.

* fix(protocol): decide the marker terminator outside the pattern

An `[OPTIONS:` head the model never closed consumed the rest of its line whenever
the line held any later `]`:

    [OPTIONS: A | B then check arr[0]
      labels  = ' A | B then check arr[0'
      visible = ''

The only closer belongs to `arr[0]`, so the body ran on through the prose and that
`]` became the terminator. Every consumer removes the whole match -- `parse_options`
replaces, `slack.format` and `messaging.renderer` cut at `match.start()`, and
`whatsapp.turn_renderer` PERSISTS the cut turn -- so the line left the message and
came back as the pill label `B then check arr[0`.

THE PATTERN CANNOT DECIDE THIS, and three review rounds on an earlier attempt are
the evidence. The body admits a bare `[` on purpose so a stray opener does not sink
the marker (`[OPTIONS: Fix [x logging | Skip]` parses, and is pinned). Reduced to
their bracket structure the shape to refuse and the shape to accept are one string:

    [OPTIONS: Fix | Skip [x logging]   ->  w|w[w]
    [OPTIONS: A | B then check arr[0]  ->  w|w[w]

A lookahead can only see one nesting level, so `list[dict[str, int]]` defeats a
one-level rule and `a[b[c[d]]]` a two-level one. The condition wanted is "the
terminator is not nested" -- bracket BALANCE, not a regular language at unbounded
depth. Encoding depths is a treadmill; the check has to leave the pattern.

So it does. `_marker_terminator_is_nested` walks the captured labels with a stack:
if an opener is still open at the end, the closer the pattern took as the terminator
is really THAT opener's partner -- unless a `|` follows it, which means it sits
inside a label and the list continues past it. `,` is deliberately not enough: it is
only the fallback separator and inside brackets it is ordinary punctuation
(`dict[str, Any]`, `arr[i, j]`). An unmatched CLOSER is ignored, because a label may
legitimately carry one. That single rule decides every depth.

AND THE CHECK CANNOT BE BYPASSED, which is the other half of the fix. A check that
callers must remember to call is one a future caller forgets, and this module's
comments call the grammar a single source of truth precisely to prevent that drift.
So the compiled patterns are now private (`_RAW_OPTIONS_RE_*`, and module-scoped in
`optionMarker.ts`), and what callers get applies both halves:

- backend: `OPTIONS_RE_LINE` / `OPTIONS_RE_TRAILER` are matchers exposing the four
  members anything used -- `search`, `finditer`, `sub`, `pattern` -- so all five call
  sites and 94 existing test references read unchanged and every one of them gains
  the check.
- frontend: `findOptionMarkers`, `findLastOptionMarker`, `stripOptionMarkers`,
  `terminatorIsNested`, plus `OPTION_MARKER_PATTERN_SOURCE` as a STRING for the
  shape pin. Nothing callable leaks, so nothing can be applied by halves.

Making the pattern private found two callers I had not enumerated -- `TurnBlock.tsx`
and `searchableText.ts` -- which were using the raw regex and would have kept the
old behaviour silently. That is the argument for the indirection, discovered by the
typechecker rather than by review.

Two side benefits worth naming. The g-flag `lastIndex` hazard the frontend docs
warned every caller about is gone, because the functions clone internally; its test
now asserts the stronger property that no `RegExp` is exported at all. And the
separator-tail form `Done. [OPTIONS: Merge | Wait], details in CHANGELOG[1]`, which
`constants.py` documented as unresolved ("deciding which shape loses -- a separate
call"), is decided: the `[` of `CHANGELOG[1]` is still open at the labels' end, so
the line stays whole instead of losing its tail to a pill.

COST: a stray opener in the FINAL label, where no `|` follows to make it label text
-- `[OPTIONS: Fix | Skip [x logging]`, `[OPTIONS: Fix [x logging, Skip]`. Fails
toward a VISIBLE marker with nothing removed, the direction every cost in this
grammar fails in, and asserted at the consumer rather than only at the regex.

Verified by differential, not inspection: all 96 marker shapes in main's five marker
suites run on both the old and new decision, and the only behaviour change is the
separator-tail form. Cross-surface: 32 markers compared on labels AND stripped text,
plus 10 label runs on the rule itself -- zero disagreements. 3802 backend tests,
27311 frontend. The check is one pass with a stack, pinned linear at 200k openers.

Fixes kirodotdev#10058

Review round 1 on the refactor: the rule is now TOTAL, with no separator hatch.

The blocking finding is correct. The hatch -- "an unmatched opener is label text if a
`|` follows it" -- is satisfied by a `|` INSIDE the unmatched bracket, so
`[OPTIONS: A | B then inspect dict[str | int]` cleared it and the line was eaten
again. That is the third defeat of the same hatch, after a separator stopping a scan
early and a newline blinding it, and each time the shape it readmitted was
structurally identical to the shape it was meant to protect. The hatch was the
defect, not its spelling.

So the rule is simply: a marker's labels must have BALANCED brackets. No depth limit,
no separator heuristic, no exceptions. `_marker_labels_have_unmatched_opener` /
`labelsHaveUnmatchedOpener` -- renamed, because the old names described the hatch.

Measured cost, over all 96 marker shapes in main's five suites: exactly ONE shape
changes beyond the intended fixes -- `[OPTIONS: Fix [x logging | Skip]`, a label
carrying an unclosed `[`, is refused and renders as visible text. That shape was a
deliberate allowance ("does not make the pair a REQUIREMENT") taken when the
alternative was unknown; the alternative is deleting a line of prose, since both
shapes hold one unmatched opener and a closer at the end anchor. Its two pins are
rewritten, one per surface, with that argument in them rather than in a commit
message.

Cross-surface differential extended to the new shapes: 34 markers on labels and
stripped text, 12 label runs on the rule -- zero disagreements. 3804 backend tests,
217 frontend marker tests.

---------

Co-authored-by: Chen Tong <cttong@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An unterminated [OPTIONS: head consumes the rest of its line if any later ] appears

2 participants