Repository navigation
fix(chat): parse same-line option markers and refuse a nested one - #30465
Conversation
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change is parser-only, every shape it now refuses shows as written text, and a revert walks it back cleanly. Notes
[DESIGN-REVIEWED] 463d890 Description read[DESCRIPTION-READ] c65a541578f81c623e0de24d525373a3fd6b5f124519e2fc406310e8d3e93890 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5, fork) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. This is a parser-only change: no control, string or component is added or changed. The existing pills and text rendering simply apply to more of the marker shapes models write. [UX-REVIEWED] 463d890 Description read[DESCRIPTION-READ] e583f396104fa3b11a1197f2a73349eddff5eee8e0e9943c82f3abcaea36a957 That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Each reported shape has a reproduction test, and the fix is in the grammar itself rather than at the call sites. Three subtractions are left as notes. Notes
What this change shipsInventory (6 items) — 3 justifiedIntent: make the chat's option pills appear for every marker shape that models actually write, without hiding prose. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 463d890 Description read[DESCRIPTION-READ] 77b1266c2c65622631781ea977e165bfa31335ee07f8b37546d8c08bb1c64900 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
bolichen97
left a comment
There was a problem hiding this comment.
Thanks. The two bugs are real: a second same-line marker leaked as raw text, and a span nested in an unclosed head parsed as its own marker. The tests describe them well.
One blocking point, under our rule of no new or wider regex in product code for parsing text or detecting markers. app-sdk/protocol/optionMarker.ts grows the marker grammar as regex:
OPTION_MARKER_REis rebuilt withnew RegExpfrom new pieces:SIBLING_CHAINlookahead,LABEL_PAIRS,TAIL_LINE_WRAPPED;- new
CONTINUES_AT(sticky),HEAD_REbuilt fromMARKER_HEADS,OPENER_RE; - the escape helper
ch.replace(/[.*+?^${}()|[\]\\]/g, …); - an inline
/[\n\r\u2028\u2029]/.test(...)inacceptMarkers.
Please solve the same-line and nesting cases with a small scanner instead of a wider pattern. acceptMarkers and unclosedMarkerFlags already walk the text with a bracket stack, so they are most of the way there:
- walk each line once;
- find a head by exact comparison (
[OPTION:/[OPTIONS:, case-folded); - track bracket pairs with the existing
PAIR_OPENERS/PAIR_CLOSERSindex tables; - close a marker on its own closer when what follows is end of line or another head.
That gives the same linear bound without new regex. You don't need to touch the regex that was already on main for this; just don't widen it. The new tests should carry over unchanged.
Happy to re-review once it's up.
5a344c1 to
8b875d9
Compare
|
@bolichen97 thanks, done in 8b875d9.
|
8b875d9 to
38398eb
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Thanks, the regex point is fixed. The same-line and nesting cases are now handled by a scanner (headAt, closerContinues, afterLinkClose, markerSpanning) with no new pattern, and only main's existing OPTION_MARKER_RE and HEAD_RE are reused.
One small blocking point, Frontend Ratchets & i18n. Two zero-tolerance checks fail on lines this PR wrote ([added-lines] and [vs-base]):
app-sdk/protocol/optionMarker.ts:234 disallow literal string: ['[OPTIONS:', '[OPTION:'] [i18n-strict:all-caps-const]
The i18n gate reads an ALL_CAPS const array of literal strings as UI copy. These two marker heads are protocol tokens, so please build them the way main's MARKER_HEADS is built, from a regex literal's .source (the comment above it explains why the gate does not read that as copy), rather than as an ALL_CAPS literal array.
That push should also re-run the fork AI lanes, which could not run on this head. I'll re-review once they report.
|
Correction to my last review: please don't use a regex literal for the heads. A plain string constant works, either kept out of the ALL_CAPS const pattern the i18n gate reads or marked with the gate's own exemption comment. These are protocol tokens, not UI copy. |
38398eb to
4fcf64e
Compare
|
@bolichen97 thanks, fixed in 4fcf64e.
|
bolichen97
left a comment
There was a problem hiding this comment.
Thanks, re-checked at 4fcf64e. The i18n point is fixed with a plain string constant: the marker heads are now markerHeads = ['[options:', '[option:'], lower-case protocol tokens the i18n gate does not read as copy, and headAt folds the case per character. No regex is added: the scanner (headAt, closerContinues, afterLinkClose, markerSpanning) does the work, and only main's existing OPTION_MARKER_RE and HEAD_RE are reused.
Frontend Ratchets & i18n is green, E2E passes, and every fork lane passes on this head (Opus 5.5, GPT 6.1, Design, First Principles, UX, Security Scope, Content Scan). Only backend shards and PR Readiness are still running, and none has failed. Approving.
4fcf64e to
17be6e9
Compare
|
@bolichen97 this PR conflicted with
|
Two [OPTIONS:] markers on one line both strip and the last supplies the options; a line is accepted or refused whole. A marker inside an unclosed head is refused, a comma after a closed head no longer keeps it open, and a row whose last marker offers nothing keeps its text on screen and in search.
17be6e9 to
463d890
Compare
Code change approver recommendation: A person should decideWhy: the decider thinks the change is probably fine but estimates a 20% chance that this change introduces a defect, above the 12% limit for approving on its own. I'm approving this PR. It changes only how the web chat parses option markers. I checked each new rule against the code in optionMarker.ts:
An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approving. On head 463d890 the review compared the merge-base and head parsers on about 40 marker shapes (sibling pairs and chains, link-close tails, nested heads, the comma rule, case folding, the 256-head cap). No shape deleted prose, and every new refusal shows visible text. The PR's 72 tests pass, 8 of them fail against main's grammar as the body says, and the added tests use no sleeps, clocks or randomness. Every caller of parseOptions, searchableText and the TurnBlock predicates was checked at the head.
Done well: each sibling span must pass the same existing marker match and balance check as the line-final marker, and a line is accepted or refused as a whole. So the change cannot admit anything the grammar did not already admit.
Optional:
- Backwards compatibility could name two more effects. A closed marker followed by
|(for example[OPTION: Yes] | [OPTION: No]) now refuses the final marker. Also, the backendhas_optionsstill counts the nested and over-cap shapes as an ask, so the Waiting lane can list a session that shows no pill. A follow-up issue for the Python twin would help. - Three
closerContinuesbranches in optionMarker.ts (comma then closer,|, immediate closer) have no test that fails when they are removed. The two CONTROL rows use a single line-final marker, so the loop never reaches them. A row like[OPTIONS: Alpha ], Bravo ] [OPTIONS: C]expecting text '' and options [C] would pin them. - HEAD_CAP applies to the whole message, so 257 heads anywhere drop valid pills on other lines too. The uncapped scan measured linear, so scoping the cap to one line, or removing it, would be enough.
Problem / Motivation
Goal: Make the web
[OPTIONS:]marker parse the shapes models really write, without ever deleting prose the user should see.The web marker grammar in
website/src/app-sdk/protocol/optionMarker.tsloses pills or hides text in a few shapes:Body [OPTIONS: A] [OPTIONS: B]: only the last marker matched, so the first leaked as raw text.[OPTIONS: dropped [OPTIONS: A | B]: the inner marker parsed although the outer one never closed.[OPTIONS: A] [OPTIONS:]: an earlier marker's options vouched for a last one that renders none, so the row was emptied and showed neither prose nor pills.Why it matters
The marker is model output, so every shape here is reachable without anyone trying. A lost pill costs the user a typed reply. Hidden prose is worse: the user never learns it was there.
Not a goal
[OPTION-ACTIONS:]close chip. It stays in feat(chat): render zero-turn option action chips in the follow-up bar #6823, which builds on this grammar.What changed (motivation → approach → change)
Approach. The marker pattern on
mainis unchanged and still finds each line's final marker. A small line scan decides the rest of that line: the earlier markers on it, and whether the final one sits inside an unclosed head. No regex is added or widened. Every refused shape stays visible as written.Change, 6 files, +375/-21, all in
website/src:markersBeforewalks the line once with a stack of brackets typed by the existingPAIR_OPENERS/PAIR_CLOSERStables. It finds a head by exact, case-folded comparison against a small head table, one plain string per head. When a head closes and blanks then another head follow, that span is a sibling. Each sibling must match the same pattern and bracket-balance check as the final marker, and the siblings must run unbroken up to it. Then all of them strip and the last one supplies the options.[1]inside an open head does not close it.Use [OPTIONS: A | B], then pick [OPTIONS: Yes | No],],continues the label list only when the next bracket on that line is a closer, so the final marker keeps its pills.stripMarkersForDisplaykeeps the raw text when the last marker offers no pill.parseOptionsandsearchableTextboth call it, and search strips markers before widgets, as the transcript does.optionsOfis the one label split, shared byparseOptionsand the display rule.Backwards compatibility
No public signature changed.
optionsOfandstripMarkersForDisplayare new exports. Every shape that parsed before still parses, except a marker nested in an unclosed head, a line whose final marker does not balance, and any marker in a message with more than 256 marker heads. All three now render as written text.Tests
optionMarkerSameLine.test.ts: same-line pairs, a chain that does not reach the line end, and a row whose last marker offers nothing, on screen and in search.optionMarkerNestingDepth.test.ts: a nested marker is refused; the comma rule keeps the line-final marker, with three controls.optionsMarkerTerminator.test.ts: a same-line chain whose sibling does not balance offers nothing and deletes no prose, a broken chain does not poison another line, a 256-marker line decided in one walk, and the head cap. The stray-opener cases underwhat the check gives upstay as controls.main's grammar. Removing each scanner rule on its own (siblings, the nesting check, the comma rule) fails the tests for that rule.tsc,eslintandnpm run i18n:checkare clean, with[added-lines]and[vs-base]at 0.Manual verification
None beyond the tests: the change is parser-only and renders through the existing pills.
Screenshots / video
No UI change: the pills and the text rendering are unchanged. Only which marker text counts as a marker changed.
Related Issues
Split out of #6823 to bring it under the size cap. #6823 keeps the close chip and rebases onto this once it merges.
Sibling sweep
Swept: git grep -nE "stripOptionMarkers|findOptionMarkers|findLastOptionMarker|parseOptions(" -- website/src
Sibling: website/src/utils/searchableText.ts -- changed here
Sibling: website/src/app-sdk/protocol/options.ts -- changed here
Sibling: website/src/pages/chat/TurnBlock.tsx -- not affected -- it calls the shared helpers, which carry the fix
Pattern harvest
Not generalizable: the fix is specific to this one marker grammar, which already keeps its pattern private behind one acceptance function.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.