Repository navigation
fix(options): parse lookalike bracket pairs in [OPTIONS:] labels - #10835
iamwhatever merged 1 commit into
Conversation
UX Review (Fable 5, fork) — ✅ PASSUX-level review of UX-Verdict: PASS Verify once in the live chat that a [UX-REVIEWED] 72d0ccc |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Only removed behavior: What this change shipsInventory (8 items) — 8 justifiedIntent: FIX — a matched CJK/fullwidth bracket pair inside an
Notes a human can audit: the deleted pins are declared, and their recorded reason for existing (no safe terminator gate) is answered by [FIRST-PRINCIPLES-REVIEWED] 72d0ccc |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Confirm #9375 intended lifting the lookalike-pair "cost" now that #10058's terminator gate landed — the flipped pins rest on that sequencing claim. Suggestions
[DESIGN-REVIEWED] 72d0ccc |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed |
fdd767b to
75776cc
Compare
|
Quadratic backtracking (GPT F1, Opus candidate 2, Design BLOCK). The per-pair interiors and the body's ordinary class excluded Silent prose deletion (Opus candidate 1, BLOCKING). The #10058 balance gate Tests: the Pattern harvestRule candidate: when a regex grammar gains a new opener/closer family, EVERY negated body class and EVERY balance-counter that referenced the old opener must be widened in the same change — a class that excludes only the original opener turns a repeated new-opener run quadratic, and a counter blind to the new opener reopens the terminator/prose-deletion hole the balance gate exists to close. Both are invisible to the parse-correctness tests and need an explicit adversarial-run timing pin and a bare-new-opener decline test. |
|
Status note on |
75776cc to
86cbcd4
Compare
|
Fixed at class level on both grammars: Tests: 2084 across the options/constants suites, 138 vitest across the two options files (the linearity pins included), black/isort/flake8/mypy, tsc, eslint, i18n gate, black-formatting/comment-history/agent-sdk-boundary gates green; no |
86cbcd4 to
e4cc5a7
Compare
…odotdev#9375) ## Problem / Motivation A model that substitutes a fullwidth / CJK lookalike bracket emits a whole matched PAIR (`【...】`, `[...]`, `〔...〕`), not just a stray closer. The grammar accepted the lookalike CLOSERS (`MARKER_CLOSERS`) but had no matching OPENER set: the matched-pair body form opened only on ASCII `[` and closed on any closer. So a label like `[OPTIONS: 修复【x】 | 跳过]` read its `】` as an unmatched closer, the marker declined, and the turn silently lost its follow-up pills. ## Why it matters `[OPTIONS:]` follow-up pills are how a turn offers choices. A single lookalike-pair label makes the whole marker leak into the message as literal text with no buttons — a silent loss of the turn's interaction, common in Chinese output. ## What changed Add `MARKER_OPENERS = "[\u3010\uff3b\u3014"`, paired POSITIONALLY with `MARKER_CLOSERS = "]\u3011\uff3d\u3015"` (`[`<->`]`, `【`<->`】`, `[`<->`]`, `〔`<->`〕`). `_MARKER_LABEL_PAIR` now emits ONE matched-pair alternative per pair by zipping the two strings: each alternative opens on its own opener, its interior excludes `[` and every closer, and it closes on THAT pair's closer only. So `[OPTIONS: 【x】 | Skip]` parses exactly as `[OPTIONS: [x] | Skip]` does, while a MISMATCHED pair (`【 ... ]`) has no pair parse and declines toward a visible marker rather than deleting prose. The disjointness invariant is preserved: every opener keeps its `(?!OPTIONS:)` guard (load-bearing on `[`, a harmless no-op on the lookalikes, so "no bracket form consumes a nested head" holds by construction); each per-pair form's trailing lookahead stays the negation of `_MARKER_LABEL_CONTINUES` over the full closer class; the body stays linear (each alternative begins at a distinct opener, an unmatched lookalike opener falls to the negated-char run, and no span has two parses). The TS mirror `optionMarker.ts` is updated in agreement (both the wrapped and bare branches of `OPTION_MARKER_RE`). Explanatory comments are rewritten in present tense to describe the paired opener set. ## Tests The two suites that pinned the old cost are flipped to expect a parse, all other assertions unchanged and passing: - `test/test_options_marker_label_closers.py`: the lookalike-PAIR "cost" case now asserts a matched-pair parse; a mismatched-pair cost is added; the non-destructive-decline case swaps its now-parsing lookalike-pair input for a mismatched-pair input. A new `TestLookalikeBracketPairsParse` covers each of the three CJK/fullwidth pairs (LINE and TRAILER), a mismatched pair declining non-destructively, a citation still parsing, and the trailing-prose and bare-opener-terminator declines still holding. - `website/src/test/optionsMarkerLabelClosers.test.ts`: the lookalike-PAIR cost test is flipped to a matched-pair parse; mismatched-pair, per-pair, citation, and bare-opener-terminator cases are added. Verified from the worktree with the repo venv: options Python suites green (281 across all `test/test_*options*`), frontend options suites green (109 across the four `optionsMarker*` vitest files), `tsc -b` and `eslint` clean. Old-head check confirmed the new CJK-pair assertions fail on the base `constants.py`. black / check_black_formatting / isort / flake8 / mypy / comment-history / brand-name gates pass; zero `.github/` diff.
|
Same tree re-pushed as a new head: |
e4cc5a7 to
72d0ccc
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (5 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: root cause is that the marker grammar accepted lookalike CLOSERS but only the ASCII opener, so a model emitting a whole lookalike pair inside a label produced a closer the matched-pair form read as unmatched and the whole marker was declined, losing the option pills; MARKER_OPENERS now pairs positionally with MARKER_CLOSERS and each alternative closes on its own closer only, with mismatched pairs still refused and linearity pinned by tests. CodeQL is not applicable on this fork PR (default-setup emits no check-run); SAST coverage is Semgrep only, latest run success with 0 annotations.
…odotdev#9375) (kirodotdev#10835) ## Problem / Motivation A model that substitutes a fullwidth / CJK lookalike bracket emits a whole matched PAIR (`【...】`, `[...]`, `〔...〕`), not just a stray closer. The grammar accepted the lookalike CLOSERS (`MARKER_CLOSERS`) but had no matching OPENER set: the matched-pair body form opened only on ASCII `[` and closed on any closer. So a label like `[OPTIONS: 修复【x】 | 跳过]` read its `】` as an unmatched closer, the marker declined, and the turn silently lost its follow-up pills. ## Why it matters `[OPTIONS:]` follow-up pills are how a turn offers choices. A single lookalike-pair label makes the whole marker leak into the message as literal text with no buttons — a silent loss of the turn's interaction, common in Chinese output. ## What changed Add `MARKER_OPENERS = "[\u3010\uff3b\u3014"`, paired POSITIONALLY with `MARKER_CLOSERS = "]\u3011\uff3d\u3015"` (`[`<->`]`, `【`<->`】`, `[`<->`]`, `〔`<->`〕`). `_MARKER_LABEL_PAIR` now emits ONE matched-pair alternative per pair by zipping the two strings: each alternative opens on its own opener, its interior excludes `[` and every closer, and it closes on THAT pair's closer only. So `[OPTIONS: 【x】 | Skip]` parses exactly as `[OPTIONS: [x] | Skip]` does, while a MISMATCHED pair (`【 ... ]`) has no pair parse and declines toward a visible marker rather than deleting prose. The disjointness invariant is preserved: every opener keeps its `(?!OPTIONS:)` guard (load-bearing on `[`, a harmless no-op on the lookalikes, so "no bracket form consumes a nested head" holds by construction); each per-pair form's trailing lookahead stays the negation of `_MARKER_LABEL_CONTINUES` over the full closer class; the body stays linear (each alternative begins at a distinct opener, an unmatched lookalike opener falls to the negated-char run, and no span has two parses). The TS mirror `optionMarker.ts` is updated in agreement (both the wrapped and bare branches of `OPTION_MARKER_RE`). Explanatory comments are rewritten in present tense to describe the paired opener set. ## Tests The two suites that pinned the old cost are flipped to expect a parse, all other assertions unchanged and passing: - `test/test_options_marker_label_closers.py`: the lookalike-PAIR "cost" case now asserts a matched-pair parse; a mismatched-pair cost is added; the non-destructive-decline case swaps its now-parsing lookalike-pair input for a mismatched-pair input. A new `TestLookalikeBracketPairsParse` covers each of the three CJK/fullwidth pairs (LINE and TRAILER), a mismatched pair declining non-destructively, a citation still parsing, and the trailing-prose and bare-opener-terminator declines still holding. - `website/src/test/optionsMarkerLabelClosers.test.ts`: the lookalike-PAIR cost test is flipped to a matched-pair parse; mismatched-pair, per-pair, citation, and bare-opener-terminator cases are added. Verified from the worktree with the repo venv: options Python suites green (281 across all `test/test_*options*`), frontend options suites green (109 across the four `optionsMarker*` vitest files), `tsc -b` and `eslint` clean. Old-head check confirmed the new CJK-pair assertions fail on the base `constants.py`. black / check_black_formatting / isort / flake8 / mypy / comment-history / brand-name gates pass; zero `.github/` diff.
Problem / Motivation
A model that substitutes a fullwidth / CJK lookalike bracket emits a whole
matched PAIR (
【...】,[...],〔...〕), not just a stray closer. Thegrammar accepted the lookalike CLOSERS (
MARKER_CLOSERS) but had no matchingOPENER set: the matched-pair body form opened only on ASCII
[and closed onany closer. So a label like
[OPTIONS: 修复【x】 | 跳过]read its】as anunmatched closer, the marker declined, and the turn silently lost its follow-up
pills.
Why it matters
[OPTIONS:]follow-up pills are how a turn offers choices. A singlelookalike-pair label makes the whole marker leak into the message as literal
text with no buttons — a silent loss of the turn's interaction, common in
Chinese output.
What changed
Add
MARKER_OPENERS = "[\u3010\uff3b\u3014", paired POSITIONALLY withMARKER_CLOSERS = "]\u3011\uff3d\u3015"([<->],【<->】,[<->],〔<->〕)._MARKER_LABEL_PAIRnow emits ONE matched-pair alternative perpair by zipping the two strings: each alternative opens on its own opener, its
interior excludes
[and every closer, and it closes on THAT pair's closeronly. So
[OPTIONS: 【x】 | Skip]parses exactly as[OPTIONS: [x] | Skip]does, while a MISMATCHED pair (
【 ... ]) has no pair parse and declines towarda visible marker rather than deleting prose.
The disjointness invariant is preserved: every opener keeps its
(?!OPTIONS:)guard (load-bearing on
[, a harmless no-op on the lookalikes, so "no bracketform consumes a nested head" holds by construction); each per-pair form's
trailing lookahead stays the negation of
_MARKER_LABEL_CONTINUESover the fullcloser class; the body stays linear (each alternative begins at a distinct
opener, an unmatched lookalike opener falls to the negated-char run, and no span
has two parses). The TS mirror
optionMarker.tsis updated in agreement (boththe wrapped and bare branches of
OPTION_MARKER_RE). Explanatory comments arerewritten in present tense to describe the paired opener set.
The balance check the patterns cannot make (
_marker_labels_have_unmatched_opener/labelsHaveUnmatchedOpener) pairs openers by TYPE as well: a closer pops the innermost opener only when it is that opener's partner, and a closer of another kind is ignored exactly like an unmatched one. Under the previous single shared count[OPTIONS: A 【x] | B]parsed — the]afterxclosed the【— and a label with a half-open lookalike pair rendered as options; now the【is still open at the terminator (the bare-opener shape) and the candidate declines with no prose moved.Tests
The two suites that pinned the old cost are flipped to expect a parse, all other
assertions unchanged and passing:
test/test_options_marker_label_closers.py: the lookalike-PAIR "cost" casenow asserts a matched-pair parse; a mismatched-pair cost is added; the
non-destructive-decline case swaps its now-parsing lookalike-pair input for a
mismatched-pair input. A new
TestLookalikeBracketPairsParsecovers each ofthe three CJK/fullwidth pairs (LINE and TRAILER), a mismatched pair declining
non-destructively, a citation still parsing, and the trailing-prose and
bare-opener-terminator declines still holding.
website/src/test/optionsMarkerLabelClosers.test.ts: the lookalike-PAIR costtest is flipped to a matched-pair parse; mismatched-pair, per-pair, citation,
and bare-opener-terminator cases are added.
test_a_mismatched_pair_declines_even_when_the_closer_ends_the_label(py) anddeclines a mismatched pair even when its closer ends the label(ts) pin[OPTIONS: A 【x] | B]and its three sibling shapes to decline non-destructively while cross-kind nesting ([x【y】]) and the unmatched-closer shape still parse; both fail on the previous head's shared-count check.Verified from the worktree with the repo venv: options Python suites green
(281 across all
test/test_*options*), frontend options suites green (109across the four
optionsMarker*vitest files),tsc -bandeslintclean.Old-head check confirmed the new CJK-pair assertions fail on the base
constants.py. black / check_black_formatting / isort / flake8 / mypy /comment-history / brand-name gates pass; zero
.github/diff.Closes #9375. Lands after #10058 (merged via #10084), whose bare-opener terminator gate is what makes the opener set safe: without it, teaching the grammar that
【…】is a pair would have turned the visible-marker failure into silent prose deletion for CJK labels — the reason this fix was deliberately held.Checklist
constants.pydescribe the opener setPattern harvest
Rule candidate: when a regex grammar gains a new opener/closer family, EVERY negated body class and EVERY balance counter that referenced the old opener must be widened — and a balance counter must pair by TYPE, not by count, or the new family's closers close the old family's openers. Both must change in the same commit as the opener set. A class that excludes only the original opener turns a repeated new-opener run quadratic, and a counter blind to the new opener reopens the terminator / prose-deletion hole the balance gate exists to close. Neither is visible to parse-correctness tests; each needs its own pin — an adversarial-run cost ratio against the ASCII run, and a bare-new-opener-before-terminator decline.