Skip to content

fix(protocol): pair each bracket opener with its own closer in OPTIONS labels - #10032

Closed
cixuuz wants to merge 1 commit into
mainfrom
fix/options-label-lookalike-openers
Closed

cixuuz wants to merge 1 commit into
mainfrom
fix/options-label-lookalike-openers

Conversation

@cixuuz

@cixuuz cixuuz commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

A closer is readmitted to an [OPTIONS:] label when an opener earlier in the same label matches it. "Matches" was decided against ASCII [ alone, while MARKER_CLOSERS already accepted three lookalikes (】 U+3011, ] U+FF3D, 〕 U+3015). A model writing a label in CJK punctuation writes both halves in it:

[OPTIONS: 见【表1】说明 | 跳过]

That 】 had no opener the rule recognised, so the body ended there, the line anchor rejected the remainder, and the whole marker reached the reader as literal text with no pills.

Why it matters

This is the affordable half of a cost that was accepted knowingly: the shape fails toward a visible marker, not toward deleted prose. So it is a rendering failure, not data loss — but it is a rendering failure that hits Chinese, Japanese and Korean output specifically, on the surface that decides whether a turn has buttons at all. It was row 3 of the four-row cost table in #9341, pinned by tests in both languages, and #9375 was filed to track paying it.

What changed

_MARKER_OPENERS = "[【[〔" is zipped positionally with MARKER_CLOSERS to emit one matched-pair branch per pair. It is private, unlike its public twin: MARKER_CLOSERS is public because messaging/renderer.py reads it, and review counted zero consumers outside constants.py for the opener set, so symmetry alone was not a reason to export it.

The pairing is the load-bearing part, not a detail

One branch accepting any opener followed by any closer would make 【表1] a pair — and that is #9284's greedy body reached by a side door. The label would end at a closer whose opener never appeared, so the body could again run past the marker's real terminator and delete the prose after it. So the mismatched shapes are asserted as explicitly as the matched ones:

input outcome
[OPTIONS: 见【表1】说明 | 跳过] parses — 【 pairs with 】
[OPTIONS: 见【表1] 说明 | 跳过] declined — CJK opener, ASCII closer
[OPTIONS: 见[表1】 说明 | 跳过] declined — ASCII opener, CJK closer
[OPTIONS: 见【表1〕 说明 | 跳过】 declined — two different lookalikes

Two consequences that deserve review in their own right

The catch-all class now excludes every bracket, not just the closers. Leaving 【 in it would give that character two parses — the pair branch and the catch-all — costing the disjointness the linearity argument rests on. TestLinearity reads the exclusion off the compiled pattern so the claim cannot drift from the comment, and exercises the shape that would blow up if it did.

MARKER_CLOSERS is no longer widenable on its own. Continuation readmits all four codepoints at once; the pair form readmits exactly one per branch, its own opener's partner. A fifth closer with no opener beside it would be admitted by continuation and by nothing else — a real shape, but not the matched one. The two constants are zipped with strict=True, so a length mismatch raises at import. That replaced a hand-written assert after review pointed out the obvious: strict=True is the language guarantee for exactly this, and an assert is stripped by python -O while the zip is not.

The frontend stays one regex literal, deliberately

Composing the pattern from named fragments reads far better and was the first attempt. But every fragment is a module-scope string literal, and eslint.i18n.strict.config.js reports those as untranslated user-visible copy — 11 findings, [added-lines] zero-tolerance. The exemption the gate offers is by shape in eslint.i18n.config.js, and that config argues at length against widening a shape for one file ("A path this narrow cannot exempt a future file"). A hole cut for regex fragments is one real copy could sit in later, so I did not cut it. A regex literal contains no string literal and needs no hole.

The duplication that costs is guarded instead: the .source pin in AssistantMessage.test.tsx is built from per-pair pieces, so the two copies of the body cannot drift from each other, or from the backend, without the test naming the difference. Every lookalike is written as a \uXXXX escape rather than the character itself — ] and ] are indistinguishable at a glance, so a literal class would be unreviewable and a wrong codepoint would read as correct.

This NARROWS one shape, and that is a cost the table has to carry

Raised by review, and correct: the pair interior excludes every bracket, both classes, so a pair whose interior holds a bracket of a different kind loses its pair parse. On base the interior excluded only [ and the closers, so 【 was ordinary interior content and these parsed:

[OPTIONS: See [a【b] ref | Skip]       base: parses     this branch: declined
[OPTIONS: See [a[b] ref | Skip]       base: parses     this branch: declined
[OPTIONS: 见【表[1】说明 | 跳过]         base: parses     this branch: declined

All three fail toward a visible marker — verified, sub leaves the text byte-identical — so it is the same affordable class as the other rows, and the same shape still parses when a separator follows the closer ([OPTIONS: See [a【b] | Skip]), because then continuation admits it. The cost is the tail, not the interior.

I kept the behaviour rather than admitting openers into the interior. Admitting 【 there would let the ASCII branch consume a character the 【 branch could also open on, so a span would have two parses — and single-parse-per-span is exactly what TestLinearity reads off the compiled pattern and what the ReDoS profile rests on. Trading a shape that degrades to visible text for an ambiguity in the body is the wrong way round.

My first draft of this description claimed the table "cannot quietly grow or shrink" while this PR grew it. That accounting error is fixed here and pinned in TestAMixedBracketInterior.

The pair interior also refuses BOTH separators, and that is about the CONSUMER

Raised by review as blocking — twice, and correct both times. Admitting a pair whole is only safe if what receives the capture can keep it whole, and neither splitter has any notion of nesting. So a pair carrying an internal separator, which is an ordinary CJK phrase, was captured and then torn apart:

[OPTIONS: 见【表1,表2】说明, 跳过]
  base:    declines  (【 was not an opener, so this never parsed)
  round 1: ['见【表1', '表2】说明', '跳过']   ← three choices, two with unbalanced brackets
  now:     declines, text intact

The corruption was introduced by this PR — base declines that shape — so it was mine to fix, not a pre-existing condition.

| alone was not enough. parseOptions picks its delimiter as labels.includes('|') ? '|' : ',' (options.ts:38), so a comma is load-bearing exactly when no pipe is present. The backend splitter has no such fallback, so a check against the backend plus a frontend shape that happened to contain a pipe reported commas as safe. They were not.

But excluding , from every interior was too much, and review caught that too. It declined [OPTIONS: Fix dict[str, Any] now | Skip] and [OPTIONS: Refactor arr[i, j] now | Skip] — labels a model writes constantly, which parse on main. So the exclusion is now per branch:

interior excludes why
ASCII […] brackets only dict[str, Any] and arr[i, j] parse today; nothing may regress
【…】, […], 〔…〕 brackets + both separators new in this PR, so no shape depends on them admitting a separator

Each half is chosen against what it would break, and both are pinned.

It is a mitigation, not a guarantee — and that matters for how the row reads. The continuation form admits a closer followed by a separator regardless of what precedes it, so [OPTIONS: 见【表1|表2】 | 跳过] reaches the splitter and splits into three. Measured on main as well as here: identical. Closing that class needs a nesting-aware splitter, not a tighter interior, and it is out of scope; the continuation-path behaviour is pinned as unchanged so the interior rule is not read as more than it is.

The mistakes worth naming, since all three were mine: round 1 asserted what the regex captured and never asked what the consumer did with it — the lesson this grammar's own pattern harvest already carried from #9341. Round 2 verified the fix against one of two consumers. Round 3 then over-corrected and regressed ordinary ASCII labels. Every cost row is now asserted at the consumer on both surfaces, and the shapes that must keep parsing are pinned alongside the ones that must not.

The remaining costs, and what is still out of scope

The cost table is now four rows, all asserted at the consumer on both surfaces so it cannot quietly change: an unmatched closer with no opener at all ([OPTIONS: Fix ]x logging | Skip]), nesting deeper than one level ([OPTIONS: Fix list[dict[str, Any]] now | S]), the mixed-bracket interior, and the separator in a lookalike interior above. The first two need bracket matching to arbitrary depth, which a regex is the wrong tool for; the last two are interior exclusions taken deliberately, each for a reason recorded beside the pair rule.

A comma outside a pair is unaffected — [OPTIONS: 见【表1】说明, 跳过] still parses to two clean choices. The separator-tail form (…| Wait], details in CHANGELOG[1]) is unchanged and still out of scope, for the reason #9341 gave: ], genuinely does continue the list.

Two cost pins flipped rather than deleted

One per surface — test_options_marker_label_closers.py and optionsMarkerLabelClosers.test.ts — each kept in its original suite, now asserting the parse instead of the decline, because the rule under test there is what decides the outcome.

Tests

  • New: test/test_options_marker_lookalike_openers.py and website/src/test/optionsMarkerLookalikeOpeners.test.ts.
  • Backend: 761 passed across the marker, option and renderer suites — test_*option*, test_*marker*, test_*renderer*, test_slack*, test_whatsapp*, test_parse_options.
  • Frontend: 170 passed across the four marker suites including the .source pin.
  • Cross-surface agreement: a differential over all 16 opener×closer combinations plus the pinned shape vocabulary — 70 shapes, zero disagreements between OPTIONS_RE_LINE and OPTION_MARKER_RE. Both surfaces have to agree or a marker parses to buttons on one and to text on the other.
  • Linearity: measured, not asserted in a comment. Lookalike-heavy adversarial input ([OPTIONS: + 【a】 | × N) stays linear — 0.5 ms at 2 KB, 1.7 ms at 8 KB, 4.1 ms at 20 KB — and a long mismatched run, the worst case for a pair form since every branch is attempted and every one fails, is pinned too.

Manual verification

N/A, and the reason is the same as #9341's: the trigger is what the model happens to emit, not an input a user can type. Both surfaces are exercised at the parser level, which is the only code the marker passes through, and the end-to-end assertion goes through split_options_trailer / parseOptions rather than the regex alone.

Related Issues

Fixes #9375

Follow-up to #9341 / #9284, which established the matched-or-continues rule and enumerated this as one of its four accepted costs.

Overlaps #6823. That PR carries MARKER_OPENERS with the same name and the same value, added for a different reason ("a citation [1] must not cancel an open marker head"). If it lands first this becomes a one-line dedupe; if this lands first, #6823's copy is redundant. #6823 also touches constants.py and optionMarker.ts and is currently labelled merge conflict, so a conflict there is expected either way and is not caused by this PR alone. Flagged rather than resolved, since that PR is its author's to sequence.

Pattern harvest

Rule candidate: review-prompt

Pattern: when a grammar admits a delimiter conditionally on some earlier context, check that the test for "earlier context" ranges over the same alphabet as the delimiter set. Widening one side alone is silent: MARKER_CLOSERS grew three codepoints while "what can match a closer" stayed ASCII, so a whole script's worth of input fell through a rule that looked complete. The tell is a constant naming one half of a pair with no constant naming the other. Two general follow-ons: when you pair two ordered constants by zip, assert their lengths at import, because zip truncates silently and the only symptom is one shape quietly failing; and when a fix adds alternatives to a regex body, re-audit the catch-all class — an alternative that can also be consumed by the catch-all gives that character two parses and silently costs whatever linearity argument the body relied on.

@cixuuz
cixuuz requested review from a team and buluoray September 11, 2026 01:11
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 11, 2026
@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ✅ PASS

UX-level review of 8d51c8f15449a94f65134689b7caffb025f13c35 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

The diff changes only parser regex constants and tests — no new UI controls, strings, components, or states. The user-facing effect is that CJK-bracket [OPTIONS:] labels now render as the existing pill buttons instead of leaking raw marker text; the narrowed shapes degrade to visible text, the same pre-existing failure mode. With no user-visible control added or changed, the missing blind read and screenshots are not evidence gaps under the review's own rules.

UX-Verdict: PASS

Pure parser fix: CJK-bracket option labels now render as the existing pills instead of raw marker text; no new control, string, or state to misread.

[UX-REVIEWED] 8d51c8f

@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 8d51c8f15449a94f65134689b7caffb025f13c35 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Positional pairing pays a tracked cost (#9375) without reopening #9284's greedy body; every trade-off is pinned at the consumer on both surfaces.

[DESIGN-REVIEWED] 8d51c8f

@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 8d51c8f15449a94f65134689b7caffb025f13c35 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All claims verified: MARKER_CLOSERS has real external consumers (messaging/renderer.py:39,491), the new opener constants are private with in-module consumers only, options.ts:38 has the delimiter fallback exactly as described, both i18n eslint configs exist and the strict one targets module-level string declarations, and the reversed test pin's own comment recorded the old behavior as an accepted cost ("hence stated explicitly"), with this PR's new tests failing on base as checkable provenance. No unfixed siblings: both grammar surfaces (backend constants.py, frontend optionMarker.ts) are fixed in lockstep, and the one residual (the continuation-path splitter) is declared and pinned as out of scope.

First-Principles-Verdict: PASS

Verify the one declared narrowing: mixed-bracket interiors like [OPTIONS: See [a【b] ref | Skip] parsed on base and now decline to visible text.

What this change ships

Inventory (8 items) — 8 justified

Intent: FIX — a label written in CJK punctuation lost its option pills because the closer set accepted lookalikes the opener rule could not match.

  1. A CJK/fullwidth bracket pair inside a label now renders as pills instead of literal text — justified
  2. A pair interior holding a bracket of another kind no longer parses; text stays visible — justified
  3. A lookalike pair carrying | or , declines whole instead of splitting into corrupt choices — justified
  4. A stray lookalike opener still reads as ordinary label text — justified
  5. Private opener/bracket constants added, deliberately unexported (zero external consumers, counted) — justified
  6. The pin recording "lookalike pair is a cost" reversed in both languages — justified
  7. Frontend stays one regex literal; a per-pair-built drift pin guards the copies — justified
  8. Crossed/mismatched pairs stay declined, now pinned explicitly — justified

[FIRST-PRINCIPLES-REVIEWED] 8d51c8f

@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging 8d51c8f15449a94f65134689b7caffb025f13c35. 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/constants.py:408 -- matched pairs can backtrack into bare openers and consume trailing prose

rf"(?:(?:{_MARKER_LABEL_PAIR})|\[(?!OPTIONS:)|{_MARKER_LOOKALIKE_OPEN_CLASS}"
[OPTIONS: A | B 【note】 then check arr[0] -> parser reinterprets [0] as bare -> trailing prose is removed and exposed as option text.
Anchor: residual/crash-data-loss-corruption
Fix: In both backend and frontend regexes, reject each bare-opener fallback when that opener can complete its matched-pair branch.
[BLOCK-MERGE] 8d51c8f
[GPT-REVIEWED] 8d51c8f

Adjudication (Opus 4.8) — is blocking on each finding proportionate?

The adjudicable block is empty (0 findings). One fenced finding, F1, to annotate.

F1 — src/kiro_crew/constants.py:408. Tracing the grammar: the label body (_MARKER_BODY_LINE, 407–411) admits a matched pair via _MARKER_LABEL_PAIR (383–389), a bare opener (\[(?!OPTIONS:) and _MARKER_LOOKALIKE_OPEN_CLASS), and an ordinary run excluding all brackets. The pair form requires only that its closer NOT be followed by a separator/closer (387), so a well-formed 【…】 pair is swallowed whole and the body continues past it into later ordinary text and a bare […], letting a trailing ] become the marker terminator — turning trailing prose into option text. The PR newly extends this to lookalike (【】) pairs; the ASCII-bracket form of the same swallowing already existed pre-PR (documented residual, 282–309). The triggering input is ordinary model-authored label prose containing brackets — precisely the shape this grammar exists to handle ("labels a model writes constantly", 371–372) — so the condition combination is not extreme, not writer-impossible, and not legacy-only. I could not execute the regex to falsify (sandbox denied), and the rarity record required for FLAG cannot be completed. Consequence is visible-but-corrupting (prose lost/echoed as a tapped reply), anchored unbounded. Cannot complete the extreme-rarity record → UPHOLD-FENCED.

[ADJUDICATION] 8d51c8f15449a94f65134689b7caffb025f13c35 total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] 8d51c8f15449a94f65134689b7caffb025f13c35
[ADJUDICATION-FENCED] 8d51c8f15449a94f65134689b7caffb025f13c35 fenced=1 flagged=0
UPHOLD-FENCED F1 src/kiro_crew/constants.py:408 -- Matched-pair swallowing past a lookalike bracket into a later closer is reachable from ordinary model-authored label prose, not an extreme or writer-impossible input, so the unbounded corruption claim stands.
[GPT-ADJUDICATED-FENCED] 8d51c8f15449a94f65134689b7caffb025f13c35

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8d51c8f15449a94f65134689b7caffb025f13c35: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 11, 2026
@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 8d51c8f15449a94f65134689b7caffb025f13c35 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8d51c8f

Verdict parsed from the review's SHA-scoped output markers for commit 8d51c8f15449a94f65134689b7caffb025f13c35.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8d51c8f15449a94f65134689b7caffb025f13c35: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 11, 2026
@cixuuz
cixuuz force-pushed the fix/options-label-lookalike-openers branch from 5a3f163 to bed4ab4 Compare September 11, 2026 01:36
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 11, 2026
@cixuuz
cixuuz force-pushed the fix/options-label-lookalike-openers branch from bed4ab4 to c16def1 Compare September 11, 2026 01:54
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 11, 2026
@cixuuz
cixuuz force-pushed the fix/options-label-lookalike-openers branch from c16def1 to 82c04a2 Compare September 11, 2026 02:12
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 11, 2026
…S labels

A closer is readmitted to an `[OPTIONS:]` label when an opener earlier in the
same label MATCHES it. "Matches" was decided against ASCII `[` alone, while
`MARKER_CLOSERS` already accepted three lookalikes -- so a label written wholly
in CJK punctuation had a `】` with no recognised opener:

    [OPTIONS: 见【表1】说明 | 跳过]

The body ended at that `】`, the line anchor rejected the remainder, and the
whole marker leaked into the message as literal text with no pills. That was an
enumerated, tested cost of the closer-continuation rule (#9284); this pays it.

`MARKER_OPENERS` is zipped POSITIONALLY with `MARKER_CLOSERS` to emit one
matched-pair branch per pair. The pairing is the load-bearing half: one branch
accepting any opener followed by any closer would make `【表1]` a pair, which is
#9284's greedy body back through a side door -- the label would end at a closer
whose opener never appeared, so the body could again run past the marker's real
terminator and delete the prose after it. The mismatched shapes are asserted as
explicitly as the matched ones.

Two consequences worth review in their own right:

- The body's catch-all class now excludes EVERY bracket, not just the closers.
  Leaving `【` in it would give that character two parses -- the pair branch and
  the catch-all -- costing the disjointness the linearity argument rests on.
- `MARKER_CLOSERS` can no longer be widened alone. Continuation readmits all
  four codepoints at once; the pair form readmits one per branch. A fifth closer
  with no opener beside it would be admitted by continuation and nothing else,
  and `zip` is silent on a length mismatch -- so the two constants are guarded by
  an import-time assert, in the style of `sandbox.py`.

The frontend pattern stays ONE regex literal, with the body spelled twice and the
four pair branches spelled out in each. Composing it from named fragments reads
far better, and was the first attempt -- but every fragment is a module-scope
STRING literal, and `eslint.i18n.strict.config.js` reports those as untranslated
user-visible copy. The exemption the gate offers is by shape, and that config
argues at length against widening a shape for one file ("A path this narrow
cannot exempt a future file"); a hole cut for regex fragments is one real copy
could sit in later. A regex literal contains no string literal, so it needs no
hole. The duplication is guarded instead by the `.source` pin in
`AssistantMessage.test.tsx`, which is BUILT from per-pair pieces, so the two
copies cannot drift from each other or from the backend silently. Every lookalike
is written as a `\uXXXX` escape, never the character: `]` and `]` are
indistinguishable at a glance, so a literal class would be unreviewable.

Two existing cost pins flipped to assert the new behaviour rather than being
deleted, one per surface, each kept in its original suite as the row that left
it. The remaining two costs (unmatched with no opener at all, nesting deeper
than one level) are asserted so the table cannot quietly grow or shrink.

Fixes #9375

Review round 1 (GPT first-principles), three findings, all applied:

- An UNDECLARED narrowing: the pair interior excludes every bracket, so a pair
  whose interior holds a different bracket kind (`[OPTIONS: See [a【b] ref |
  Skip]`) loses its pair parse where base admitted it. Verified against base by
  execution -- 4 shapes, all failing toward a VISIBLE marker. Kept the behaviour,
  because admitting `【` there would let the ASCII branch consume a character the
  `【` branch could also open on, giving a span two parses and costing the
  single-parse property the linearity argument reads off the pattern. Declared
  and pinned in `TestAMixedBracketInterior`; the cost table is three rows now.
  The description had claimed the table could not grow while this grew it.

- `zip(..., strict=True)` replaces the hand-written length assert: it is the
  language guarantee for the same invariant, and it holds under `python -O`,
  which strips asserts.

- `MARKER_OPENERS` / `MARKER_BRACKETS` are private. `MARKER_CLOSERS` is public
  because `messaging/renderer.py` reads it; the opener set has no consumer
  outside `constants.py` and its tests, so mirroring the twin was not a reason
  to export it.

Review round 2, two findings, both applied:

- BLOCKING, and correct: the pair interior admitted `|`, and
  `split_options_trailer` splits the captured label group on `|` with no notion of
  nesting. `[OPTIONS: 见【表1|表2】说明 | 跳过]` -- an ordinary CJK phrase -- was
  captured whole and then torn into ['见【表1', '表2】说明', '跳过']: three choices
  with unbalanced brackets, echoed back as the user's reply when tapped. Measured
  against base, which DECLINES that shape, so the corruption was introduced here.
  The interior now refuses `|`; `,` is not excluded, because the splitter does not
  split on commas and the exclusion tracks what the consumer actually does. The
  mistake behind it: round 1 asserted what the REGEX captured and never asked what
  the consumer did with the capture, which is the same lesson this grammar's own
  pattern-harvest already carried.

  One shape improves beyond base as a side effect: `[OPTIONS: See [a|b] c | Skip]`
  corrupted identically on base and now declines, so a pre-existing corruption
  becomes a visible marker.

- Stale documentation: `constants.py` still carried the OLD four-row residual-cost
  table, including a lookalike-PAIR row this change makes parse and an "opener set
  this grammar does not have" clause it makes false. Rewritten to the current five
  rows, and `_MARKER_LABEL_PAIR`'s opening sentence no longer says the interior
  excludes only `[` and the closers.

Review round 3, one finding, applied:

- BLOCKING, and correct again: excluding only `|` from the pair interior fixed
  half the problem. The frontend's `parseOptions` picks its delimiter as
  `labels.includes('|') ? '|' : ','` (options.ts:38), so a COMMA is load-bearing
  exactly when no pipe is present -- and `[OPTIONS: 见【表1,表2】说明, 跳过]` was
  captured whole and comma-split into ['见【表1', '表2】说明', '跳过']. Measured on
  the frontend, which is where the fallback lives; the backend splitter does not
  fall back, which is why a backend-only check said commas were safe. The interior
  now refuses both separators on both surfaces.

  Root cause of getting this wrong twice: round 2 verified the claim against the
  BACKEND consumer and a frontend shape that happened to contain a pipe, so the
  fallback branch was never exercised. When two surfaces have to agree, a claim
  about "the consumer" has to be measured on each of them.

  The excluded set is now exactly what `_MARKER_LABEL_CONTINUES` /
  `CONTINUES_LABELS_RE` treat as separators, and that correspondence is the
  invariant to hold: a character that ends a label for the continuation rule must
  not be swallowable into a pair, or the two rules disagree about where a label
  ends.

Review round 4, two advisories, both applied -- and the second reverses part of
round 3:

- Opus: excluding `,` from EVERY pair interior declined
  `[OPTIONS: Fix dict[str, Any] now | Skip]` and
  `[OPTIONS: Refactor arr[i, j] now | Skip]`, which parse on main. Ordinary labels,
  regressed for a corruption that only the lookalike branches can reach. The
  exclusion is now per-branch: lookalike interiors refuse both separators, the
  ASCII interior does not. Each half is chosen against what it would break -- the
  ASCII branch has history to protect, the lookalike branches are new here.

- GPT: the comment claimed the interior exclusion prevents the corruption. It does
  not, and the counter-example is measurable on main as well as here:
  `[OPTIONS: 见【表1|表2】 | 跳过]` is admitted by the CONTINUATION form, whose
  lookahead does not care what preceded the closer, so it reaches the splitter and
  splits into three either way. The exclusion is a mitigation on the matched-pair
  path only; closing the class needs a nesting-aware splitter, which is out of
  scope. Comment, tests and cost table now say that, and the continuation-path
  behaviour is pinned as unchanged rather than left implicit.
@cixuuz
cixuuz force-pushed the fix/options-label-lookalike-openers branch from 82c04a2 to 8d51c8f Compare September 11, 2026 03:11
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 11, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 11, 2026
@cixuuz

cixuuz commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

Closing this in favour of fixing #10058 first. The opener set is the right fix for #9375, but it cannot land safely on top of a defect it makes newly reachable.

What the review round actually found

GPT's last blocking finding was right, and chasing it down changed my understanding of the whole change. The reported shape:

[OPTIONS: A | B 【note】 then check arr[0]
  origin/main : declines
  this branch : labels = ' A | B 【note】 then check arr[0'   ← the prose is consumed

I first read that as "the matched-pair rule lets the body run on". It is not about pairs at all. Measured on origin/main, with no lookalike brackets involved:

[OPTIONS: A | B then check arr[0]     MATCHES, whole line consumed
[OPTIONS: A | B then check arr]       MATCHES, whole line consumed
[OPTIONS: A | B then check arr        declines   (no closer anywhere)
[OPTIONS: A | B] then check arr[0]    declines   (properly closed — #9284 working)

So the real defect is: an unterminated [OPTIONS: head plus any later ] on the line eats the rest of the line. That ships today, it is not documented anywhere in the residual-cost block, and it has nothing to do with this PR. Filed as #10058.

What this branch does is remove an accidental partial mitigation. On main, [OPTIONS: A | B 【note】 … declined only because 】 had no opener and broke the body early — not because the grammar handled the shape correctly. Teaching it that 【…】 is a pair removes that accident and the pre-existing defect becomes reachable for CJK input too.

Why closing rather than patching again

The trade is bad in its current form. #9375's symptom is benign: a CJK-pair label fails to a visible marker with nothing deleted. This PR converts that into a path that silently deletes prose. For the shapes involved, the fix is worse than the bug.

It is also not fixable inside the pair rule. Four review rounds each produced a locally-correct patch that exposed the next reachable path:

  1. separator inside a pair interior → captured whole, then torn into three broken choices
  2. excluding , everywhere → regressed [OPTIONS: Fix dict[str, Any] now | Skip], which parses today
  3. per-branch exclusion → fine, but the interior rule turned out to be a mitigation, not a guarantee
  4. the run-on above → not reachable by any interior rule at all

The correct terminator condition is "the closer is not nested", which is bracket balance and not something a regex expresses. That is #10058's problem to solve, and once it is solved this change becomes small and safe.

What is worth keeping from here

The branch fix/options-label-lookalike-openers is left in place rather than deleted, because several pieces are reusable once #10058 lands:

  • the positional opener/closer pairing, with zip(..., strict=True) so a length mismatch raises at import
  • the per-branch interior rule, and the measured reason the ASCII and lookalike branches differ
  • the four-row cost table, asserted at the consumer on both surfaces rather than at the regex
  • the cross-surface differential (all 16 opener×closer combinations, 70 shapes, zero disagreements)
  • the .source pin built from per-pair pieces, so the backend and frontend cannot drift silently

Three review findings from this round are also worth carrying regardless of what happens to the opener set: MARKER_OPENERS/MARKER_BRACKETS had no consumers outside constants.py and should be private; zip(strict=True) beats a hand-written length assert, which python -O strips; and every cost row needs asserting at the consumer on both surfaces, since parseOptions picks its delimiter as labels.includes('|') ? '|' : ',' while the backend splitter has no such fallback.

Corrections to this PR's own description

Two claims in earlier revisions were wrong and are worth flagging so nobody inherits them:

  • "the cost table cannot quietly grow or shrink" — while that revision grew it by one row.
  • "the interior refuses the characters the consumer cannot survive" — it refuses them only on the matched-pair path; the continuation form admits a closer followed by a separator regardless of what preceded it, so [OPTIONS: 见【表1|表2】 | 跳过] still splits into three, identically on main.

#9375 stays open, now blocked on #10058.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[OPTIONS:] labels containing a lookalike bracket pair (【…】, […], 〔…〕) are not parsed

1 participant