diff --git a/CHANGELOG.md b/CHANGELOG.md index 5cde8de..2ba5047 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -161,6 +161,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/). - XLSX packages missing `_rels/.rels` no longer fail inside calamine. - Release builds use `panic = "unwind"`, so a parser panic becomes an error instead of aborting the CLI or the host Python/Node process. +- Text that only looks like a block marker is no longer escaped: `4.6 out of 5` stays `4.6`, not + `4\.6`; same for `#tag` and `+1`. Markers followed by whitespace (`1. `, `# `, `+ `) are still + escaped. - Exclude selectors now apply to `
  • `, ``, `` and ``; they were ignored there. - Indented HTML (most real web pages) now keeps blank lines between blocks; previously headings, paragraphs, lists and tables after the first ran together with a leading space diff --git a/crates/markkit/src/render/escape.rs b/crates/markkit/src/render/escape.rs index aca047c..3621b59 100644 --- a/crates/markkit/src/render/escape.rs +++ b/crates/markkit/src/render/escape.rs @@ -42,17 +42,33 @@ pub(crate) fn push_text(buf: &mut String, s: &str, pos: Position) -> bool { continue; } line_start = false; + // A marker only opens a block when followed by whitespace. At the end of + // the run the next node may supply it, so escape there too. + let opens = |at: usize| chars.get(at).is_none_or(|n| *n == ' ' || *n == '\t'); match c { - '#' | '>' | '-' | '+' | '=' => { + // `-` (thematic break, setext), `=` (setext), `>` (quote): always. + '>' | '-' | '=' => { buf.push('\\'); buf.push(c); i += 1; continue; } + '#' | '+' => { + let run = chars[i..].iter().take_while(|d| **d == c).count(); + if c == '#' && opens(i + run) || c == '+' && opens(i + 1) { + buf.push('\\'); + buf.push(c); + i += 1; + continue; + } + } '0'..='9' => { - // Ordered list marker: up to 9 digits then `.` or `)`. + // Ordered list marker: up to 9 digits, `.` or `)`, then whitespace. let digits = chars[i..].iter().take_while(|d| d.is_ascii_digit()).count(); - if digits <= 9 && matches!(chars.get(i + digits), Some('.') | Some(')')) { + if digits <= 9 + && matches!(chars.get(i + digits), Some('.') | Some(')')) + && opens(i + digits + 1) + { buf.extend(&chars[i..i + digits]); buf.push('\\'); buf.push(chars[i + digits]); diff --git a/crates/markkit/tests/output_safety.rs b/crates/markkit/tests/output_safety.rs index 84f485c..7c712fd 100644 --- a/crates/markkit/tests/output_safety.rs +++ b/crates/markkit/tests/output_safety.rs @@ -124,6 +124,42 @@ fn line_start_block_markers_are_escaped() { } } +#[test] +fn line_start_marker_lookalikes_render_verbatim() { + // Not block markers: no whitespace after `#`, `+`, or the ordered `.`/`)`. + for s in [ + "4.6 out of 5 stars", + "3.14 is pi", + "1)x", + "#hashtag", + "##tag", + "+1 for this", + "2024.10.27 release", + ] { + let md = render(vec![para(vec![text(s)])]); + assert_eq!(md, format!("{s}\n"), "{s:?} over-escaped"); + assert_eq!(block_tags(&md), vec!["Paragraph"], "{s:?} -> {md:?}"); + } +} + +#[test] +fn line_start_marker_at_run_end_stays_escaped() { + // The next inline node may start with whitespace and complete the marker. + for (a, b) in [ + ("1.", " x"), + ("#", " h"), + ("##", " h"), + ("+", " li"), + ("7)", "\tx"), + ] { + let md = render(vec![para(vec![text(a), text(b)])]); + assert_eq!(block_tags(&md), vec!["Paragraph"], "{a:?}+{b:?} -> {md:?}"); + let md = render(vec![para(vec![text(a)])]); + assert_eq!(block_tags(&md), vec!["Paragraph"], "{a:?} -> {md:?}"); + assert_eq!(visible_text(&md), a, "{a:?} -> {md:?}"); + } +} + #[test] fn line_start_after_hard_break_is_escaped() { let md = render(vec![para(vec![ diff --git a/knowledge/log.md b/knowledge/log.md index 58ade41..1f1eb5e 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -3,6 +3,7 @@ ## 2026-10-27 * **XLSB support (#11)**: Excel binary workbooks are read by a new bounds-checked BIFF12 reader (`spreadsheet/xlsb.rs`) instead of being rejected; calamine's XLSB reader was not used because it panics on malformed records. Same caps, hidden-sheet and macro rules as XLSX. See [Format Support](formats/format-support.md) and [Threat Model](security/threat-model.md) T6c. +* **Over-escaping fix**: line-start `#`, `+` and ordered markers are escaped only when followed by whitespace or the end of the text run, so `4.6 out of 5` no longer renders `4\.6` (eval gap). `> - =` stay always escaped. See [Threat Model](security/threat-model.md) T1, [Evaluation Harness](operations/eval.md). * **MCP server**: `markkit mcp` serves `MarkkitTool` over MCP stdio (initialize, ping, tools/list, tools/call); line size capped from `--max-size`, tool failures as `isError` results. New [MCP Server](integrations/mcp-server.md) concept; MCP messages row in the [Threat Model](security/threat-model.md). * **Spreadsheet follow-ups (#11)**: single-row sheets render as data under column letters; XLSX/ODS/XLS titles come from document properties; XLSB content type no longer maps to XLSX; XLSX without `_rels/.rels` parses; trailing-newline output documented as intended; release profile switched to `panic = "unwind"` so `catch_unwind` is effective in markkit's own binaries. See [Format Support](formats/format-support.md) and [Threat Model](security/threat-model.md) T6c. * **Plain text and Markdown input**: text is always on and goes through the IR; Markdown (`markdown` feature) is re-parsed with pulldown-cmark rather than passed through, so raw HTML never reaches output. Neither is sniffed from bytes. Shares the call's `Limits` budget (timeout, nodes, nesting depth). New threat T7 and invariant S18. See [Format Support](formats/format-support.md) and [Threat Model](security/threat-model.md). diff --git a/knowledge/operations/eval.md b/knowledge/operations/eval.md index 77352bb..6d0e214 100644 --- a/knowledge/operations/eval.md +++ b/knowledge/operations/eval.md @@ -152,7 +152,7 @@ collapsed to one column and dropped every ``, fixed by IR v2 row-header question title and body are dropped (forum 81% default vs 62%). - **`main_content` on the wiki page** keeps the TOC, `[edit]` links and categories. -- **Over-escaping** (since #23): a paragraph starting `4.6 out of 5` renders +- **Over-escaping** (since #23, fixed 2026-10-27): a paragraph starting `4.6 out of 5` renders `4\.6 ...`; `4.6` is not an ordered-list marker (needs `4.` + space), so the backslash is noise for text consumers (commerce, both modes; before #23 main_content passed the page). diff --git a/knowledge/security/threat-model.md b/knowledge/security/threat-model.md index 2eaa002..56aa6d3 100644 --- a/knowledge/security/threat-model.md +++ b/knowledge/security/threat-model.md @@ -41,7 +41,8 @@ that renders as headings, links, code fences or tables). - Output is markdown text with no HTML pass-through, enforced at render time by `render/escape.rs` for every format (the IR keeps raw text): - Text escapes `` ` * [ ] < ~ | ``, `_` at word boundaries, entity-like `&`, a - trailing `\` or `!`, and block markers (`# > - + =`, `1.`/`1)`) at line start. + trailing `\` or `!`, and block markers at line start: `> - =` always, `#` runs, + `+` and `1.`/`1)` only before whitespace or the end of the text run. Newlines inside text become spaces. - Link schemes allowlisted (`http`, `https`, `mailto`, `tel`, `ftp`, relative); image schemes `http`, `https`, relative and raster `data:image/*`. Others are