diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index c49bc43b..aa88f62b 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -52,6 +52,7 @@ Failure shapes to catch yourself in, and the command that replaces each: - not knowing which command answers a question you can already state in words → answer "", which resolves the subject in the index, runs the right command, prints via: , and refuses rather than guess - a shell text search with context flags to find a function body → read "file::symbol" - paging one function with view/view_range → read "file::symbol" +- issuing sequential or overlapping line-range reads across a file (paging) → read "file::symbol" / outline file / skeleton file / rg - a read refused by a [tg] deny, retried with a shifted range, different lines, or another read tool → never retry reading the file at different lines; run the command the deny names or use bash-output / token-goat read - reading a symbol plus chasing its callers and containing doc section as separate reads → brief "file::symbol" - reading one heading of a large doc → section "file::Heading" diff --git a/CHANGELOG.md b/CHANGELOG.md index a050f267..60a3e646 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,16 +6,89 @@ All notable changes to Token-Goat are documented in this file. Format follows Ke ### Added +- **Claude Code's PowerShell tool now gets the same handling as its Bash tool.** A large PowerShell result is compressed, fenced as untrusted content, redacted and kept for `token-goat bash-output` recall, and a result the harness cut gets the recall pointer; until now token-goat registered for Bash alone and left all 372 recorded PowerShell results untouched. The filter is chosen with PowerShell's own command names in mind, so `ls`, `cat`, `curl` and the other aliases and cmdlets get the generic filter rather than one written for the Unix tool. The pre-hook checks and post-hook bookkeeping apply too: a whole-file `Get-Content` of a large source file is pointed at `token-goat read`/`outline`, a repeated `token-goat read` gets the dedup note, a `Get-Content` the model saw reaches the session read cache, and a file a `git checkout`, `Set-Content`, `Add-Content`, `Out-File` or `Tee-Object` wrote is reindexed, as `sed -i` and `>` redirects already were. PowerShell commands are never rewritten, since token-goat's compress wrapper is bash syntax and its permission check for a rewrite models `Bash(...)` rules only, so an inline Python or Node file read that the Bash tool would run under a token cap is refused under PowerShell. Re-run `token-goat install` to add PowerShell to the hook matcher in an existing settings file. +- **On Windows, a long Claude Code Bash command with a quoted heredoc now runs instead of failing.** Claude Code's Bash tool on Windows fails a command of about 7,300 characters or more before any of it runs, with "unexpected EOF while looking for matching" and a quote (exit 2), and a little under that a heredoc can be written cut short; across the measured sessions every heredoc command of 7,963 characters or more failed and had to be regenerated. Where the call would run with no prompt (bypassPermissions with no rule source token-goat cannot read), a command of about 6,500 characters or more (counting the room its quotes take) now has each `<<'EOF'`, `<<"EOF"` or `<<\EOF` body moved to a new owner-only file under `~/.token-goat/heredoc_spill`, and the heredoc becomes `< 'file'`, so the command that runs is short and writes the same bytes. A body holding a recognised credential is never written, and each file is deleted once it is an hour old. Everywhere else (a permission prompt, which could not show the moved body, auto mode, an unquoted or `<<-` heredoc, a delimiter like `<<'EOF'x` that the shell reads as a longer word), a command of 8,000 characters or more is refused with the reason and what to do: write the content with the Write tool, or append it in smaller heredocs. Claude Code on Windows only. +- **Playwright MCP results no longer carry the model's own script back, or a tab list it has already seen, at any size.** `@playwright/mcp` answers in markdown `### ` sections rather than JSON, and most of its results are under the 2 KB floor MCP compression starts at, so they were only fenced: `browser_evaluate` re-quoted the model's own `function` argument under `### Ran Playwright code` (34% of its measured bytes, longer than what was sent), and `### Open tabs` repeated the previous list in 83 of 212 measured results. A code block that is byte-for-byte the call's own `function` or `code` argument (`browser_evaluate`, `browser_run_code`, `browser_run_code_unsafe`) now becomes one line giving its size, and a tab list identical to the last one shown this session becomes a note with the tab count plus the `(current)` tab. `### Result`, `### Page`, errors and every action tool's code block are left as they are, and the full result stays recoverable with `token-goat mcp-output `. Long prose in a `browser_snapshot` YAML node and overlong query strings in `browser_network_requests` lines are now cut past 200 characters with an `N chars elided` marker, as the other browser servers' snapshots already were. `TOKEN_GOAT_MCP_COMPRESS=0` turns all of it off. +- **`section` reads every spec it is given, not only the first.** `section a.md::A b.md::B` read `a.md::A` and printed "Note: 1 extra spec argument ignored", so agents chained one `section` call per heading. Several positional specs are now served the way their comma form `a.md::A,b.md::B` is, in one call, and a heading holding a comma stays whole. Words a shell split out of one unquoted heading still get the note that says how to quote it. The token-goat skill now names both forms. - **A Codex subagent spawn now gets the same briefing as a Claude Code or Copilot CLI one.** Codex fires its pre-tool hook for a spawn under the name `spawn_agent`, with the brief in `message` or in text `items`, and token-goat only knew the `prompt` field of `Agent` and `task`, so Codex subagents started with no project map and no cached-output ids. The briefing is now appended to `message`, or added as one more text item when the spawn sends items, and `token-goat install --codex` wires a `spawn_agent` matcher so the call reaches the hook. An existing Codex install reports as not installed by `doctor` until `install --codex` or `doctor --fix` adds the matcher. - **A first read of a big indexed file now draws advice to read it surgically.** When a file of 50 KB or more is read whole for the first time and token-goat has indexed its symbols or headings, the read goes through and a note names the file's size, how many symbols or headings it holds, and the `read`, `outline` or `skeleton` command that returns just the part you need. `hints.first_read_symbol_policy` picks what happens: `warn` (the default) adds the note, `deny` refuses the whole-file read and says what to run instead, `off` does nothing. `hints.first_read_symbol_bytes` sets the size, 50,000 bytes by default, and 0 turns the check off. Both are set only in your global config or the environment (`TOKEN_GOAT_FIRST_READ_SYMBOL_POLICY`, `TOKEN_GOAT_FIRST_READ_SYMBOL_BYTES`): a `.token-goat.toml` checked in with a repository cannot set them, since it would decide how much of its own code a reviewing agent is shown. ### Changed +- **`section`, `skill-body`, `skill-section`, `bash-output`, `web-output` and `mcp-output` keep their answer under 25,610 characters.** Claude Code shows a Bash result inline only up to about 29,700 characters and persists anything longer to a file with a 2 KB preview, so a 52 KB section, a 49 KB skill body or a 48 KB `--grep` recall cost the agent further calls to read the file back. An answer past the ceiling, which leaves 4,096 characters of that limit for the continuation note and stderr, is cut at a line boundary, below the closing fence tag where there is one, with a note naming the command that returns the next part: `token-goat read "file@A-B"` for the section and skill commands, the same recall with `--lines A-B` and its narrowing flags kept for the cached-output ones. `--full` still prints everything, and `overflow_guard.enabled = false` turns the ceiling off. +- **`bash-output --grep` stops at 100 matches unless told otherwise.** With no `--max-matches`, `--head`, `--tail` or `--full`, it prints the first 100 matching lines and names the `--grep ... --lines A-B` that continues after the last one, instead of the first 30 and last 80 behind a marker that pointed at `--full`. +- **A read command's miss repeats its suggestions on stdout.** `section`, `read`, `symbol`, `brief`, `outline` and `skeleton` wrote their "Did you mean" list and `Try:` line only to stderr, so a call run with `2>/dev/null`, or piped into `head`, showed nothing at all. The lines after the miss's first one now go to stdout as well, and the command still exits 1. - **Upgrading re-embeds every file that was already embedded, once.** The code that chunks files and decides what gets embedded changed since 2.9.30, so token-goat marks every embedded file out of date, whatever its type, and the next `token-goat index` or the background worker embeds it again. The embeddings you have keep answering `semantic` and `search` until each file is redone, so nothing stops working in the meantime; only the one-time embedding work is new. - **Indexing a large Salesforce flow holds less in memory.** Finding a flow's top-level `name` used to list every matching element in the file and every tag before it, then pick one. It now reads the elements one at a time and keeps a single running tag depth, stopping at the first top-level match, so a flow with thousands of nested elements indexes in steady time instead of building lists the size of the file. +### Security + +- **A fence tag spelled with an invisible character or a stray dot can no longer close the untrusted-content fence.** The tag escape wanted whitespace, `/` or `>` right after the tag name, the way an HTML parser ends a name, so ``, and the same tag with a zero-width space inside its name, reached the model unescaped and read as the end of the fence. Tags are now matched on the text as a reader sees it, past format and control characters, and any tag that opens `untrusted-` is escaped whatever follows the prefix, so a fence also escapes every other surface's tags and longer names sharing the prefix. Token-goat's own `[token-goat` and `[tg]` markers are matched the same way. The escape is now a single linear scan; the old pattern took over two seconds on a `<` followed by 40,000 spaces. +- **A fetched HTML page no longer delivers a credential printed on it.** When token-goat folded a page from an unwrapped `curl` GET (auto mode, a deny or ask rule, or a harness that never rewrites the shell) to its text, it clipped and shipped that text without redaction, so an API key in the article went straight to the model. WebFetch's compression redacted only the converted text, and the HTML-to-text converter cuts its input at 16,777,216 characters without a marker, so a token straddling the cut arrived as a fragment too short to recognize. Both paths now redact the page before conversion and the text before it is clipped, and the cut is raised to 32 MiB, the most output token-goat captures from a command, or to the length of the page when that is larger. +- **A nested terminal escape can no longer turn into a closing fence tag after token-goat has checked the text.** Inside fenced output token-goat escapes any copy of its fence tags and of its own `[tg]` deny prefix, and afterwards it strips terminal colour codes from whatever it prints to a pipe, which under an agent is every time. That strip is not idempotent, so an escape nested inside another one, placed in the middle of ``, looked like noise to the check and came out of the strip as the exact closing tag; a `[tg]` deny line could be forged the same way. This reached `bash-output` recall, `pr-slice`, `html-query` and the other printed fences. Every fence around tool output now strips escape sequences until none are left and removes any leftover escape byte before it checks, so nothing printed later can assemble a tag or a marker. Fenced file content keeps plain colour codes and complete character-set designations such as the `ESC(B` that `tput sgr0` and ncurses write into logs, because a model edits from those bytes and every strip in use removes such a code the same way; any other escape sequence, an 8-bit control character that opens one, a lone surrogate, a bell character beside them, a stray C1 control in a file that also holds a bell, or a code whose removal would spell a tag, makes each escape byte show as `␛` instead, which no strip removes. The strip itself is now linear: a line of unterminated control-string openers took 1.5 seconds at 80,000 characters and grew with the square of the length. +- **Escape sequences written with 8-bit control characters are handled like their 7-bit forms.** A fence passed over a control character such as U+009B but read the bytes after it, so `<`, U+009B, `0m/untrusted-tool-output>` reached the model as harmless-looking text that any strip aware of 8-bit escapes would turn into an exact closing tag. These introducers are now read as the sequences they stand for and removed with them, and the line and paragraph separators U+2028 and U+2029 are passed over inside a tag or marker as other invisible characters are. +- **A credential that a colour code splits in two is redacted.** `rg --color=always` and similar tools colour the match inside a line like `GITHUB_TOKEN=ghp_...`, which puts an escape sequence in the middle of the token, so the redactor did not recognise it, while a terminal, token-goat's own strip and the fence all join it back together. Redaction now also reads the text with its escape sequences removed, and with only a terminal control string's opener, terminator and colour codes removed, so a token split inside a device control string is found too, and returns the text redacted in the form that holds the credential. This covers MCP results, fetched pages, the curl HTML fold, recalled `bash-output`, the Read hook's folds, which now decline on such a file as on a plain credential, and the `recall` index, which used to store the joined token in `global.db` and could show it whole in its listing. +- **`token-goat recall` no longer shows a credential from a row indexed by an older build.** Older builds redacted a recall row before stripping its escapes, so a key a colour code split was stored whole once the strip joined it, and recall's snippet showed it until the row aged out after 24 hours. The index schema moves to version 18, which empties the recall index once on first open; cached outputs stay recallable by their ids, and an older build can no longer open the upgraded database to write such rows again. +- **A shell read that overlaps an earlier one no longer ships a credential from its new lines.** When a `sed -n` or similar read of a file covered lines served earlier in the session, token-goat replaced those lines with a pointer and passed the rest through inside its own rewrite without redacting them, and its fence joined a key a colour code had split. Such a read is now left as the command printed it whenever it holds a credential, as the other rewrites of a file read already were. +- **Fence tags and token-goat's markers spelled with look-alikes, other invisible characters or Unicode spaces are escaped too.** The check passed over zero-width and control characters only, so a closing tag with a variation selector, a combining mark or a Hangul filler spliced into it, one spelled with a Cyrillic `е`, an accented `ü`, a small capital, mathematical letters, Coptic, Cherokee, Lisu or syllabics letters, a non-breaking hyphen or no hyphen at all, fullwidth or other look-alike angle brackets, or one opened with a division slash, `<\/` or `` on the same line only. In file content, source code that only spells the word, such as `Map`, is left as written. +- **`[tg: note]`, `[tg: ok]` and `[tg: stale-write]` lines inside fenced text are escaped.** Only `[tg]` followed by its closing bracket counted as token-goat's voice, so a command's output could print a line shaped like one of token-goat's own short notes and have it reach the model as written. `[tg` followed by a colon now counts as well. +- **Every untrusted-content fence now closes with a tag carrying a per-fence random nonce, so fenced content can no longer write the tag the fence actually ends at.** The closing tag was the fixed `` (and its five siblings), so keeping a fence shut depended entirely on the escape recognising every spelling a model might read as that tag. Both tags now carry the same 48-bit nonce drawn when the fence is built, written unquoted (``) so it survives a JSON string, and token-goat's own fence parsers match a fence only by that pair. The escape still runs, since a model does not check the nonce. A fence-shaped region with no nonce, as a forger would write it, is now treated as plain text and its notice is escaped. +- **Fencing a very large text now does a bounded amount of work.** The tag and voice scan is linear but had no ceiling, and reading look-alikes made each character up to fifteen times dearer, so a multi-megabyte body could hold a hook for seconds. Past 4 Mi UTF-16 code units the scan is skipped and every character that reads as `<`, `>` or `[` is escaped outright. +- **`transcript-outline` now redacts and fences the speaker names and cue previews it prints from a .vtt or .srt file.** Both its text and `--json` forms printed them bare, so a caption file carrying an API key or an injection phrase reached the model unmarked, while `transcript` on the same file redacted and fenced it. The text outline is now fenced whole and each flagged `--json` field is fenced on its own, as the other document readers do. Cue text is redacted before the outline cuts each preview to 60 characters, since a cut through a credential left a prefix too short for redaction to recognise. +- **opencode sessions now get MCP tool results fenced and redacted, as other harnesses already did.** The opencode plugin only routed opencode's built-in tools through token-goat, and opencode hands a plugin an MCP tool's raw result object, building the model's text from its `content` array after the hook returns. Prompt-injection text and credentials in an MCP server's response reached the model untouched. The plugin now rewrites `content` for MCP tools, and `output` too when a result carries both, keeping image and blob blocks, and covers `read_mcp_resource` and the resource-listing tools too. One gap remains: opencode throws an MCP result marked as an error before any plugin hook runs, so that error text still arrives unfenced. ### Fixed +- **A list joined across CRLF line endings keeps every entry through Bash compression.** Every filter treated a carriage return in the middle of a line as a progress bar redrawing itself and kept only the text after the last one, so `tr '\n' ','` over a 168-address CRLF file reached the model as its last address under `generic filter -99%`. A line now collapses to its final state only when every earlier piece looks like a redraw: a percentage, a spinner or bar glyph, a rate, an ETA, a counter, or the same text with different numbers. Anything else loses the stray carriage returns and keeps all of its text, and a list whose pieces all start with the same separator is never collapsed. +- **A grep of one file, or one you already bounded with `head` or `-m`, now arrives as its match lines.** Past 30 lines the grep filter kept only a count per file, so `grep -n MOQ page.yml | head -80` came back as `grep: 80 matches across 1 file` and every one of eight recorded summaries was fetched again in full within two calls. One file's matches, and a result bounded to 200 lines or fewer by a `head`/`tail` stage or `-m N`, now ship as lines; a search across many files keeps its first 40 matches verbatim and counts only the rest per file. The filter's own closing notices now sit outside the untrusted-content fence instead of arriving escaped as `[token-goat`, and the opt-out is named once. token-goat recognises its notices by a private tag it adds, not by their wording, so a file line that copies one stays inside the fence. +- **The grep output filter no longer invents files out of the lines of a single-file `grep -v`.** grep prefixes a filename only when it searches more than one file, but the filter read the text before each line's first colon as a file name whenever it held a `.` or `/`, so `grep -v '^\s*#' script.sh` came back as `grep: 40 matches across 6 files` naming fragments like `/^[[:` as files, and the script's lines were dropped. A single-file or stdin grep now attributes every line to the file it searched and ships the lines, a pattern given with `-e` or `-f` is no longer counted as a file, and an inverted search's summary counts lines rather than matches. +- **A long one-line JSON document keeps every record instead of being cut after its first thousand characters.** The per-line cap kept the head of any line over 1,000 characters, so a compact API response lost every record after the cut, mid-string, and the next call had to recall the output in full. A line that is one JSON object or array is now laid out with two-space indentation, one line per record or member wherever a container is too wide to fit, before the cap applies. The layout copies every token as printed, so large integers and escapes arrive unchanged, and it stops a few levels down, so a deeply nested document cannot crash `token-goat compress` or swell its output. +- **`token-goat compress` no longer ships a rewrite that its own recall line makes bigger than the original.** The net-benefit gate priced the compression marker but not the `[token-goat] full output: bash-output --full` line printed under every compressed result, so a saving within about 60 bytes of the floor shipped and cost more than it saved. Every line token-goat adds is now priced, and such an output arrives untouched. +- **`token-goat bench` now reports the bytes the model actually receives for a compressed result, including the untrusted-content fence and the recall line.** It used to count only the compressed text and the savings marker, so every case's savings looked larger than what `token-goat compress` delivers, and a rewrite the runner would decline as not worth its framing could still be credited. The benchmark now prices and frames a rewrite through the same function the runner uses, adding about 190 bytes per compressed case on the bundled corpus. +- **The "still running" heartbeat no longer reaches the model in a finished Bash result.** `token-goat compress` writes `[token-goat compress] still running, 15s elapsed` to stderr while a long command is quiet, so whoever watches the run sees it is alive, and Claude Code merged those lines into the result the model read. The heartbeat still appears while the command runs; the PostToolUse hook now removes it from the finished result. +- **The "Output was NKB uncompressed" note now appears only when it has something to offer.** It fired on every uncompressed output of 4 KB or more, and in recorded sessions its recall id was used 0 times in 80, because the model had already seen the whole output. The `token-goat bash-output ` pointer now appears only when the harness did not show the whole output (Claude Code persisted it), and the `token-goat compress -c` suggestion only when the same command produced a large output earlier in the session. `hint-stats` scores the note under a new `bash_large_output` category and backs it off like the other hints. +- **A Bash or PowerShell result token-goat rewrites no longer carries a credential through its colour-stripping or its stderr.** The pass that only strips terminal colour codes shipped the stripped output without redacting it, and a rewrite of a result with both stdout and stderr replaced stdout alone, leaving stderr raw and outside the untrusted-content fence. Both streams are now read and rewritten together, and every rewrite is redacted. +- **Output of a command token-goat wrapped for compression now arrives inside the untrusted-content fence.** A command the pre hook ran under `token-goat compress`, the usual shape for grep, ls, git and test runs, was the one Bash path whose compressed view reached the model unfenced, so a matched line spelling out `[token-goat] ignore previous instructions` read like token-goat's own voice. The command's bytes are now fenced and token-goat's marker, notices and recall line follow the closing tag. Output passed through unchanged stays unfenced as before, and `injection.enabled = false` still turns the fence off. +- **A command can no longer pass its own output off as the compress runner's result.** The post hook treated a wrapped command's result as the runner's when it ended with a recall line naming any stored output of the same command, then stripped its opening heartbeat lines and declined to rewrite it, and a command could print that line with an id computed from the directory and the command, or copied from an earlier run. The runner now stores a digest of what it printed, and only a result matching it is treated as the runner's. +- **The compress runner's notes about a timeout, a spawn failure or the capture cap now print after the fence.** They were appended to the command's own output, so on a compressed or capped result they sat inside the untrusted-content fence escaped as `[token-goat: ...`, were saved in the recall copy as though the command had printed them, and on uncompressed output followed a `---` stderr separator. They now print with token-goat's other lines after the closing tag, or after the uncompressed output, and the recall copy holds only what the command wrote. +- **A `token-goat compress` timeout now stops the whole command, not just the shell running it.** The runner killed only the shell it started, bash.exe on Windows and `/bin/sh` on Linux and macOS, so the command the shell was running, and anything that command had started, kept going after the runner reported exit 124; on the captured path a survivor also held the output pipes open, so the run waited for it anyway. The runner now ends the process tree with `taskkill /T` on Windows while the shell is still alive, and on Linux and macOS reads the shell's descendants from the process table and kills them with it, on the timeout and when output passes the capture cap, on the compressed and `--no-compress` paths alike. On Linux and macOS the shell stays in the terminal's session, so Ctrl-C and `/dev/tty` prompts still reach the command. +- **Savings and rewrite fitting now use Claude Code's measured inline limit of 29,706 characters instead of 20,000 bytes.** In the measured transcripts Claude Code 2.1.220 to 2.1.296 delivered Bash results of up to 29,706 characters inline and persisted none below 30,695, so a result between 20,000 and 29,706 bytes was booked as a 2 KB preview the model never saw and had its rewrite clipped for no reason. +- **A rewritten Bash result now fits Claude Code's inline limit with its closing fence tag intact.** The Bash rewrites that withhold already-served lines or fold code bodies, and the curl HTML fold, reserved room for the fence by pricing it off the whole output, but escaping markers can double output that is dense in them and a cut can end inside one. 1,500 lines of `[tg] [tg] [tg] [tg]` shipped at 54,259 bytes, and a single long line cut just after `[token-goat` came out over the limit, so the harness cut off the closing tag and the recall note. Each cut is now fenced and measured, and cut again until it fits, always on a whole character. An oversized MCP result's preview is fitted the same way, under the 25,000 bytes at which Claude Code spills an MCP result to disk; it allowed a fixed 500 bytes for the fence, and a 60 KB result of `[tg]` lines came out at 34,920. So is an MCP result token-goat fenced whole, one its server flags as an error, a repeated call, or one too large to cache, which came out at 100,912 bytes for the same 60 KB; a note after the closing tag names the cached copy or says there is none. `TOKEN_GOAT_MCP_COMPRESS=0` still leaves MCP results uncut. +- **Output clipped to the harness delivery cap now fills the cap instead of stopping at the last line break.** The clip always cut back to a whole line, so a short line followed by one long line shipped the short line alone, and about one clipped output in sixteen came to less than half the cap. The cut now ends on a line break only when that gives up at most an eighth of it, and otherwise cuts inside the line on a whole character. This covers oversized MCP previews, the curl HTML fold and the served-line elision. +- **A clipped shell-output rewrite no longer claims its dropped part can be recalled.** The note said a pointer above recalled the untouched output, but a served-line elision or shell-read fold stores the rewrite it ships, and its pointers name earlier runs, so the clipped part was gone. The note now says the clipped rest was not kept and to re-run the command over a narrower range. +- **Checking a deny message or a `--json` report for forged fences is no longer quadratic.** Text holding many copies of a fence's opening lines with no closing tag took 208 ms at 320,000 characters and grew with the square of the length; it is now linear. +- **A file fence shows `[tg]` and `[token-goat:` as written when source code spells them inside a line.** Every spoken marker in a file was escaped wherever it sat, so a string literal such as `'[tg] cached'` came back from a Read as `[tg] cached`, and an Edit built from it failed to match the file or wrote the entity into it. A marker now ships unchanged when an ASCII letter, digit or quote stands before it on its line, past any Read line number. One that opens a line, or follows only whitespace, punctuation, or a character that draws nothing such as a Hangul filler or the blank Braille pattern, is still escaped, as is every marker on a line holding a control character other than a tab, and every marker after one already escaped on its line. +- **`cat` of a small file now runs instead of being refused.** A whole-file `cat`, `bat`, `type` or `Get-Content` was refused at any size, while the Read tool took the same file without a word; across the sampled refusals, all of 0.8 to 12.6 KB notes, the agent read the same file with the Read tool or `sed` next six times out of eight and ran the suggested command none of the eight. A file under the size the Read gate hands over whole (20 KB for every type the Bash hook checks) now runs as written, and so does a `sed` or `awk` range over all of it. A `.env` file is still refused at any size, and a multi-file `cat` is refused only for its files past the floor. +- **A whole-file read refusal no longer names a command that cannot run.** For a document with no heading, a config file with no key or table found, a `.env` with no variable, a SQL file with no table, or a plain `.txt` file, the Bash hook printed `section "file::SectionHeading"`, `config-get "file" "KEY_NAME"`, `read "file::table_name"` or a grep for ``, and each of those exits 1 or needs editing before it runs. It now names `token-goat read "file::1-80"`, the file's first lines, through the same helper the post-edit, large-file and re-read hints use. +- **Hints no longer suggest commands with stand-in names such as `::SymbolName`, `::Heading Name`, `""`, `""` or `""`.** When no real heading, symbol or key resolved, a hint fell back to a placeholder, and the agent ran the command as printed: it exited 1 or searched for a string no file holds. Each hint now names a part the file really holds, or leads with a command that runs on any file (an outline, a skeleton, a read of the first lines, `lockdeps` for a lock file, the first top-level key of a JSON or YAML file, slide 1 or page 1 of a deck or PDF), and leaves the targeted command out when nothing resolves. +- **A compound command that writes a Python script with a heredoc is no longer refused as a file read.** `python a.py && cat > b.py <<'EOF' ... open('d.json') ... EOF` was scanned as one Python snippet, so the `open()` inside the script being written read as a read of `d.json` and the command was refused; agents then wrote the identical script with the Write tool and ran it. Each simple command of a compound one is now read on its own, and a heredoc counts as Python only when a `python` reads its program from it. +- **A Python command whose output goes to a file is no longer refused or rewritten.** `python -c "...open(sys.argv[1])..." "notes.txt" > "out.md"` prints nothing into the conversation, but it was refused with a grep for ``. A redirect of standard output to a file (not `2>`, `>&2` or `/dev/stdout`) now leaves the command alone. +- **A Python read that parses its file now gets advice instead of a refusal where the capped run cannot ship.** Outside bypassPermissions, and wherever compression is off or a permission rule withholds the rewrite, `python -c "d = json.load(open('d.json')); print(len(d))"` was refused outright, and every measured refusal of one was worked around within a turn (the same script from a file, the file on stdin, a copy elsewhere). A read that hands the file to `json.load`, `json.loads`, `csv.reader`, `csv.DictReader`, `yaml.safe_load`, `load_workbook` or `pd.read_*` and never prints its text now runs with the advice attached. A read that prints the raw file is still refused. +- **The note on a `token-goat read` no longer claims a file was fully read when only part of it was.** After a Read of lines 225-254 of a 1,069-line file, `token-goat read` of that file was told it "was already fully read via the Read tool this session". The note now says the file was partially read and names the lines on record, and it says nothing when the lines asked for (`@A-B` or `::A-B`) lie outside every window read so far. +- **A long directory listing no longer gets told to use `token-goat map`.** `ls -la dist/ | head`, `ls -lR`, `eza --long` and `tree -h` drew "Use `token-goat map --compact`", but map lists files and symbols, not the sizes, times and permissions those listings ask for. A listing with a metadata flag (`-l`, `-h`, `-t`, `-S`, `--full-time`, `--long`, `--git` and the like) now gets no map hint; a plain or recursive listing of names still does. +- **An image outside the working directory is now shrunk on Claude Code in bypassPermissions mode.** The shrink was skipped for every image outside the session's working directory, because its approval was only honest for a Read that would not have prompted, and in bypassPermissions no Read prompts. Images in the Claude scratchpad or on another drive reached the model at full size. In bypassPermissions the shrunk copy is now approved on the same conditions as every other bypass rewrite: no Read deny or ask rule that could match the image or its real path, no `PermissionRequest` hook, and no rule source token-goat cannot read. Every other mode still shrinks only images inside the working directory, and Copilot CLI does so in every mode. +- **An image a Read rule refuses no longer leaves a shrunk copy in token-goat's cache.** The shrink decoded the image and wrote its copy to `~/.token-goat/image_shrink_cache/` before checking your Claude Code Read rules, then dropped only the temporary copy when a rule applied. Claude Code refused the original, but nothing refused a Read of the cached copy. The rules are now checked before the image or the cache is opened, and a shrink is cached only once it is actually delivered. +- **A harness token-goat cannot identify no longer gets a shrunk image as base64 text in its hook context.** The image shrink handed every unrecognized harness the shrunk copy as a data URL, because the OpenClaw plugin reached the hook without naming itself and only its bridge turns that payload into an image. Anywhere else the data URL reached the model as text, often hundreds of kilobytes of base64 a text model cannot use, beside a Read that still loaded the original. An unrecognized harness now gets no shrink output at all, and the OpenClaw plugin sets `TOKEN_GOAT_HARNESS_OVERRIDE=openclaw` when it registers, so its shrink still ships; `token-goat install` or `token-goat doctor --repair` rewrites an installed plugin. +- **Browser screenshots in MCP results are now shrunk on Claude Code.** The screenshot pass for claude-in-chrome, Playwright and chrome-devtools results ran after the MCP handler, and the hook stops at the first handler that rewrites a result, so any screenshot that came with text (a caption, Playwright's result section, claude-in-chrome's tab list) reached the model at full size, a repeat screenshot was sent again, and a repeated tab list was never shortened. The pass now runs inside the MCP handler before it fences the result's text. A shrunk screenshot is delivered as a smaller image block of the same shape, rather than as a base64 data URL in text, and harnesses that read an MCP rewrite as one string keep the original image. When a result holds no text token-goat reads, the rewrite keeps only the images and token-goat's own notes, so a block such as a resource link whose only text is its title cannot reach the model unfenced beside the shrunk image. +- **On Copilot CLI, an MCP tool result's images and audio are now kept when token-goat fences its text.** Copilot hands the hook an MCP result's text and its attachments separately, the attachments in `binaryResultsForLlm`, and token-goat fences every MCP result, so every MCP call got a replacement result that carried the fenced text alone. The replacement now carries the original image and audio attachments beside the text, without the descriptions Copilot sets on them. A text blob or any other attachment is left out, since it would reach the model with no fence and no redaction. Claude Code and pi already kept images. +- **The Grep tool searches a large Claude Code tool-results file instead of being refused.** A Grep on a `tool-results/*.txt` or `tasks/*.output` file of 20 KB or more was sized as a whole-file Read and refused with advice to page it, though a Grep returns only matching lines and is the narrow search the refusal asked for. A whole-file Read of the same file is still refused. +- **A ranged Read refused for size now names a limit that fits.** A Read of a large `.txt`, `.log`, `.out`, `.jsonl` or unrecognised file that already carried `offset` and `limit` was told to use `offset` and `limit`, with the slice's size beside the whole file's line count and a preview of the whole file's first and last lines. The refusal now says which lines were asked for and how large they are, suggests the largest limit that fits under the cap from the slice's average line length, and previews the first and last lines of that slice. A log keeps its `bash-output --grep` alternative. +- **The PDF refusal now gives its real reason.** It said Read cannot return PDF content, which is false on Claude Code, whose Read takes a `pages` range. It now says token-goat refuses every PDF Read because the extracted text costs far fewer tokens than the page images, and that a page whose layout matters can be rendered to an image and read. A Read of a PDF that does not exist is no longer refused as "PDF file (0 B)"; it goes ahead and reports the missing file. +- **Paging through a plain-text file no longer suggests `token-goat skeleton`.** The note after the third ranged read of one file named `skeleton` and a symbol read for any file, including a `.txt` or a log, which have no symbols to show. The note now appears only for source files, as the paging refusal already did. +- **A memory note, a scratchpad report or a short document now comes back from a Read whole.** A Read of a file in Claude Code's memory folder (`~/.claude/projects//memory/`) or in its session temp folder (`claude/` under the system temp folder or `CLAUDE_CODE_TMPDIR`, where the session scratchpad lives) could be cut down to its headings or have paragraphs folded, and an agent fetched the withheld text back with `sed` in the very next call. Those files, and any markdown document under 4,000 bytes, are no longer outlined, skeletoned or folded. Large documents elsewhere are cut down as before. +- **WebSearch results now reach the model fenced and redacted on Claude Code.** The post-WebSearch hook sent its fenced copy as a bare string, and Claude Code accepts only its own WebSearch result shape there, so every rewrite was rejected with "does not match WebSearch's output shape; using original output" and the model got the original, unfenced and unredacted, every time. The hook now returns a copy of the result itself, with each summary string inside `results` redacted and fenced, each hit's title and URL stripped of terminal escapes, redacted and stripped of anything posing as token-goat's own marker or the fence tag, and the query, the structure of the hit lists and the counts left as they were. The redaction count in `token-goat stats` now counts what that copy carries. A result with no summary string is cached and passed through, and no built-in tool is sent a bare-string rewrite any more: when no field of its result can carry the new text, token-goat sends nothing instead of a rewrite the harness would refuse. +- **The "re-read a specific section" hint no longer follows a Write or an edit of a small file.** It fired after Write created a new 1 to 2 KB file, although the model had just written every byte of it, and in the recorded sessions it was followed 0 times out of 50. It now appears only after an Edit of a markdown file of at least `hints.reread_deny_min_bytes` (50 KB by default), where a whole re-read is costly. `token-goat hint-stats` now scores it on whether the `section` command it names was run, not on the absence of a re-read, so an `edit_reread_suggest` row no longer carries the `*` absence marker and a hint nobody follows can be suppressed like any other. +- **The general-purpose spawn advisory no longer comes back after a session sits idle for a day.** It was meant to appear once per session, but the mark that said so was deleted with the session's state 24 hours after its last write, so a session resumed the next day saw it again; one session showed it three times. The mark is now also kept in `global.db`, where it ages out with the stats retention. +- **A subagent briefing no longer describes a near-empty project or tells the subagent to index it.** For a project of 0 to 4 files the briefing carried a map such as "Files: 0" with "run `token-goat index . --walk`", which sent every subagent spawned there off to index the directory. The map is now left out below 5 files, and when it is shown for a project with nothing indexed it says so without naming the index command. The reminder, the report contract and the cached-output ids are unchanged. +- **A background subagent launch is no longer cached as if it were the subagent's report.** An Agent call with `run_in_background` returns a launch receipt that repeats the prompt just written, and token-goat cached that receipt and said "This subagent report (12KB) is cached for later recall". The receipt is now passed through untouched; finished reports are cached as before. +- **Claude Code no longer starts a PreToolUse hook for MCP tools token-goat does nothing with before the call.** The installed PreToolUse matcher carried `^mcp__`, so every MCP call paid a hook process, although the two PreToolUse handlers that look at MCP tools act only on read calls (`get_`, `list_`, `search_` and the other read verbs, for the repeat-call dedup) and on screenshot tools: all 1,409 measured PreToolUse runs on MCP tools printed `{}`. Each handler now contributes a matcher built from the same verb list or pattern its own check uses, so a navigate, click, evaluate or create_draft call no longer starts the hook (947 of 1,184 MCP calls in the measured sessions), while PostToolUse keeps matching every MCP tool. An existing install picks up the narrower matcher on the next `token-goat install` or `token-goat doctor --repair`, and until then `doctor` shows a `Claude Code hook matchers` line naming the events whose matcher this build no longer writes. +- **`token-goat doctor --repair` no longer narrows the Claude Code hooks to Read and Grep.** When a repair rewrote the hooks (a missing event, a stale shim), the doctor process had not loaded token-goat's hook handlers, so the matcher it derived from them held only the two it happened to reach: PreToolUse came back as `^Read$|^Grep$`, silently switching off every other PreToolUse handler, and PostToolUse as the match-everything matcher. Doctor now loads every handler before it checks or rewrites a hook config. +- **`pdf-extract` says when a PDF has no text layer instead of printing an empty fence.** A scanned or image-only PDF came back as an empty untrusted-content block at exit 0, which read as a broken command. When none of the requested pages has a text layer, pdf-extract now prints a note on stdout naming the pages and a `pdftoppm` or Ghostscript command that turns them into images to Read or pass to `image-text`, and exits 1. +- **`skill-body --compact` no longer hands back the full body when a skill has no compact slice.** A skill with no `` marker, or one whose marker cuts more than half the body, printed the whole body as its "compact" form. It now prints the skill's heading tree with the `skill-section` command that loads one part, and says on stderr that no slice was served; a skill with too few headings to map still gets its body, said as such. The Skill hook's reload and first-load notices no longer offer `--compact` for such a skill. +- **Playwright snapshot files are no longer indexed.** The Playwright CLI and the Playwright MCP server save page snapshots as `page-.yml` under `.playwright-cli/` and `.playwright-mcp/`, and a repository that committed them had them indexed and embedded, so they crowded code out of `search` results. Files under those directories are now left out of the index, and ones an existing index holds are removed at the next reconcile or `token-goat index`. Upgrading reindexes nothing. +- **The branch hint, the subagent no-changes check and the compaction git bonus work on Windows, and a git call that times out no longer leaves git running in your repository.** Their git calls were bounded at 50 ms, and on Windows each one takes 57 to 184 ms, so all three were skipped on every call. The `git` on PATH there is Git for Windows' launcher, which starts the real git as a separate process, and the timeout stopped only the launcher, so the real git kept running in the project directory, holding it open so it could not be deleted. The default `hints.git_hint_max_ms` is now 500, and calls with a time limit start the real git directly, so the limit stops git itself. Because the changed file is one of the sources the parser fingerprint covers, upgrading reparses every indexed file once. +- **`token-goat image-text` reads images again when token-goat runs from a clone of this repository.** The repository pinned `tesseract.js-core` to 6.1.2 while `tesseract.js` was 7.0.0, and tesseract.js 7 loads a relaxed-SIMD build of the engine that only core 7 ships, so on Node 24 every image was reported as unreadable by OCR. The pin is gone and the engine runs as tesseract.js 7 with tesseract.js-core 7. The pin had been held to avoid a Socket alert on core 7's minified Emscripten glue, which was reviewed and found to embed only the WebAssembly module the package already ships. An npm install of token-goat already got core 7, since `overrides` never reached a consumer install. The language model and its pinned checksum are unchanged. +- **`--json` output from the database, document, spreadsheet, archive, query and recall readers no longer slows down with every row.** csv-query, sqlite-query, sqlite-schema, sqlite-tables, describe, xlsx-sheets, xlsx-columns, xlsx-query, pptx-outline, docx-outline, docx-tables, pdf-locate, pdf-outline, pdf-meta, zip-list, zip-read, html-query, html-outline, dep-docs, recall, pr-slice and mcp-output --json-query fence each string of the envelope on its own, and each string read token-goat's config again: both config files and every config environment variable, about 0.8 ms per read on Windows, and again for each size the overflow cap tried. A 60-row xlsx-query read the config about 3,600 times, and a 401-table `sqlite-schema --json` took about 30 seconds on Windows against under a second as text. Each reader now reads it once per envelope, whatever the row count, and prints the same bytes as before. +- **A capped `--json` database listing no longer counts one flagged name ten times in the injection stats.** sqlite-schema, sqlite-tables and describe cut a listing that overflows `overflow_guard.max_tokens` by rendering it at several sizes and keeping the largest that fits, and every one of those renders booked an injection_detected row for each flagged table or column name it held, so `token-goat stats` showed 8 to 13 detections for a name printed once. Only the printed render's detections are booked now. - **Copilot CLI and VS Code SessionStart hooks no longer wait for stdin EOF or run indefinitely.** The shared shim frames one complete JSON object and runs the normal handler under one 3-second deadline, reserving the final second for cleanup. On Windows, the existing packaged `tg-hook` creates the canonical process inside a strict Job Object at creation; closing the owner's non-inherited job handle terminates descendants even after the canonical root exits. Output is published only after job accounting reports zero active processes, the root has exited, and bounded pipes have drained. Missing or incompatible native owners fail open without running unowned canonical work. POSIX retains its detached process-group path. The parent does not probe the entry or native binary on disk. Reminder, notes, drift and receipt work remain enabled; invalid input, a stalled helper, or unverified cleanup returns `{}` with a stderr diagnostic. Other events retain their existing behavior. - **The skill list no longer shows an older version of a skill as current when two versions were saved within one millisecond.** Each saved version is stamped with the time it was saved, and the list keeps the newest stamp. Two saves in the same millisecond got the same stamp, so the list fell back to directory order and could pick the older body, along with a compact summary that no longer matched it. Each save in a process now gets a stamp strictly later than the one before. - **On Windows, a hook no longer waits on the registry every 30 seconds.** The resident hook server keeps Claude Code's managed policy (the `HKLM` and `HKCU` `Policies\ClaudeCode` keys) for 30 seconds, and the first hook after that window ran `reg.exe` twice and waited for both before answering. The server now re-reads both keys in the background every 20 seconds, without blocking, so a hook finds the policy already loaded. A background read that cannot be parsed, or that a newer read or a reset has overtaken, is dropped, and the next hook reads the registry itself as it did before. The background reads stop after five minutes without a hook and resume with the next one. diff --git a/CLAUDE.arch.md b/CLAUDE.arch.md index 763abe7d..6b986a38 100644 --- a/CLAUDE.arch.md +++ b/CLAUDE.arch.md @@ -83,8 +83,8 @@ token-goat is a TypeScript CLI bundled to `dist/token-goat.mjs` via esbuild. The | [`src/hook_registry.ts`](src/hook_registry.ts) | `registerHook(eventName, handler, filter?)` — stores handlers in a `Map`; `runHook()` dispatches by event name and optional tool-name filter | | [`src/hooks_agent_spawn.ts`](src/hooks_agent_spawn.ts) | Subagent briefing pack hook. | | [`src/hooks_bash_commands.ts`](src/hooks_bash_commands.ts) | Shell command parsing, git mutation detection, pipeline filtering, and syntax validation. | -| [`src/hooks_bash_post.ts`](src/hooks_bash_post.ts) | `postBashHandler()` — caches bash stdout/stderr, records the files a command read and the paths it rewrote, delivers a smaller rendering of the result where one pays (served-line elision, identical-read collapse, structural and body folds, compound-command compression, curl HTML cleanup, ANSI stripping), emits `gh api` and one-time `gh pr/issue view` field-batching advisories | -| [`src/hooks_bash.ts`](src/hooks_bash.ts) | `preBashHandler()` — cat/wsl-cat/rg detection (incl. `powershell -Command`/`pwsh -c`-wrapped `Get-Content` via `extractPowerShellWrappedGetContent`, size-gated on temp paths), bash output dedup, compression filters | +| [`src/hooks_bash_post.ts`](src/hooks_bash_post.ts) | `postBashHandler()` — caches bash stdout/stderr, records the files a command read and the paths it rewrote, delivers a smaller rendering of the result where one pays (served-line elision, identical-read collapse, structural and body folds, compound-command compression, curl HTML cleanup, ANSI stripping), emits `gh api` and one-time `gh pr/issue view` field-batching advisories; `postPowerShellHandler()` gives Claude Code's PowerShell tool the output half of that (compression with PowerShell-aware filter choice, the untrusted fence, the recall copy, the large-output advisory) plus the bookkeeping in `recordShellCall()` (reindexing files a git command or `Set-Content`/`Out-File` write changed, download, surgical-read and `gh`/scoped `git status` recall records, the `gh` advisories) and the session read cache for a `Get-Content` that reached the model uncompressed; the pre hook runs `preBashHandler()` for it with no rewrite | +| [`src/hooks_bash.ts`](src/hooks_bash.ts) | `preBashHandler()` — cat/wsl-cat/rg detection (incl. `powershell -Command`/`pwsh -c`-wrapped `Get-Content` via `extractPowerShellWrappedGetContent`, size-gated on temp paths), bash output dedup, compression filters. Also registered for Claude Code's PowerShell tool, minus the Git Bash checks (double backslash, heredoc spill, bash quoting) and every command rewrite, since `rewrite_permission.ts` models `Bash(...)` rules only | | [`src/hooks_bashoutput.ts`](src/hooks_bashoutput.ts) | BashOutput poll-delta caching hook. | | [`src/hooks_browser_image.ts`](src/hooks_browser_image.ts) | post_tool_use handler for browser-automation MCP tools whose results embed an inline base64 screenshot with no destination-file option. | | [`src/hooks_cli.ts`](src/hooks_cli.ts) | Harness payload normalization only: `normalizePayload()` translates harness-specific tool payloads to token-goat's canonical shape before dispatch. Response serialization and hook dispatch live in `relay.ts` / `hook_registry.ts` | @@ -285,6 +285,7 @@ token-goat is a TypeScript CLI bundled to `dist/token-goat.mjs` via esbuild. The | Module | Role | |--------|------| | [`src/bash_extractors.ts`](src/bash_extractors.ts) | Command line extractors, classification, and surgical hint builders for bash hook handlers. | +| [`src/bash_heredoc_spill.ts`](src/bash_heredoc_spill.ts) | Moves the bodies of a long Bash command's quoted heredocs into temp files, for Claude Code's Bash tool on Windows, which fails a command of about 7,300 characters or more before an | | [`src/bash_output_cache.ts`](src/bash_output_cache.ts) | Bash stdout/stderr disk store (byte cap plus 4096 file-count cap, oldest-first eviction) | | [`src/bash_range_savings.ts`](src/bash_range_savings.ts) | Prices a line-range read's proposed surgical replacement against the read itself, so a hint that redirects a line-range read can be required to prove it saves something before it i | | [`src/bash_runner.ts`](src/bash_runner.ts) | Exports: `DEFAULT_TIMEOUT_SECONDS`, `MAX_CAPTURE_BYTES`, `RunOptions`, `run` | @@ -444,7 +445,7 @@ token-goat is a TypeScript CLI bundled to `dist/token-goat.mjs` via esbuild. The | [`src/coverage_query.ts`](src/coverage_query.ts) | Narrow "gaps only" extraction for `token-goat coverage-report-gaps`, so a code-coverage report (which can run to tens of thousands of lines for a real project) never needs a full ` | | [`src/csv_query.ts`](src/csv_query.ts) | Narrow CSV projection/filter for `token-goat csv-query`, so a multi-thousand row CSV never needs a full `Read` just to answer "what's in column X where Y = Z". | | [`src/delivering_deny.ts`](src/delivering_deny.ts) | The wording that marks a token-goat deny as the delivery of content rather than a refusal. | -| [`src/delivery_cap.ts`](src/delivery_cap.ts) | Exports: `CLAUDE_CODE_BASH_OUTPUT_CAP_BYTES`, `bashOutputCapBytes`, `clipToDeliveryCap`, `deliveredOutputBytes` | +| [`src/delivery_cap.ts`](src/delivery_cap.ts) | Exports: `CLAUDE_CODE_BASH_OUTPUT_CAP_BYTES`, `bashOutputCapBytes`, `clipToDeliveryCap`, `deliveredOutputBytes`, `fitToDeliveryCap` | | [`src/dep_docs.ts`](src/dep_docs.ts) | `token-goat dep-docs ` — surgical read for an installed npm dependency. | | [`src/describe_sqlite.ts`](src/describe_sqlite.ts) | Exports: `describeSqliteFile` | | [`src/dirty_queue.ts`](src/dirty_queue.ts) | `queue/dirty.txt` itself: `dirtyQueuePathFor()`, the line codec, `parseDirtyQueueLines()` and `getDirtyPathsFor()` for readers, and `appendDirtyQueuePaths()`, the one append every producer goes through | @@ -464,6 +465,7 @@ token-goat is a TypeScript CLI bundled to `dist/token-goat.mjs` via esbuild. The | [`src/failures_state.ts`](src/failures_state.ts) | Cross-invocation state for `token-goat failures --delta` -- persists the failure-signature set (test names / summary lines, see `failures.ts::failureSignatures`) from the last `fai | | [`src/failures.ts`](src/failures.ts) | Extract failing test blocks from test runner output. | | [`src/fence_cap.ts`](src/fence_cap.ts) | Exports: `guardThenFence`, `guardRedactAndFence`, `guardAndFenceFileText` | +| [`src/fence_confusables.ts`](src/fence_confusables.ts) | What a look-alike reads as, for the fence-marker scans in src/injection_scan.ts. | | [`src/filter_counts.ts`](src/filter_counts.ts) | Aggregated filter/rule counts for all hook types. | | [`src/filter_notice.ts`](src/filter_notice.ts) | The notices a listing command prints when a filter left nothing of what the store holds, so a filtered view never reads as a definitive "nothing here". | | [`src/fold_delivery.ts`](src/fold_delivery.ts) | Turning a delivered slice of a source file into a folded one: pick the spans, render the notices. | @@ -509,6 +511,7 @@ token-goat is a TypeScript CLI bundled to `dist/token-goat.mjs` via esbuild. The | [`src/markdown_lines.ts`](src/markdown_lines.ts) | Iterate markdown lines, skipping fenced-code-block content (``` or ~~~ blocks) and the fence delimiter lines themselves, so a `#` comment inside a code fence is never mistaken for | | [`src/mcp_client_text.ts`](src/mcp_client_text.ts) | Token-goat's own notes worded for whichever client asked. | | [`src/mcp_compress_packs.ts`](src/mcp_compress_packs.ts) | Schema-aware compression packs for two specific MCP servers, layered on top of {@link mcp_compress.ts}'s generic structural pass. | +| [`src/mcp_compress_playwright.ts`](src/mcp_compress_playwright.ts) | Section pass for `@playwright/mcp` browser-tool results. | | [`src/mcp_compress.ts`](src/mcp_compress.ts) | Deterministic, structural compression for MCP tool results. | | [`src/mcp_jsonrpc.ts`](src/mcp_jsonrpc.ts) | A minimal Model Context Protocol server, in-house. | | [`src/mcp_stdio.ts`](src/mcp_stdio.ts) | The stdio transport for token-goat's MCP server -- newline-delimited JSON over the process's own stdin and stdout, which is the only transport `token-goat mcp-serve` has ever offer | @@ -719,7 +722,7 @@ All three layers are fail-soft: a disk error never throws into a hook. - **`containers.ts`** (batch F) — 5 filters for the container/kubernetes landscape, faithful port of the Python `bash_compress.py` container filter family. Dispatch order: `KubectlLogsFilter` must precede `KubectlFilter` (both match `kubectl`/`k`); `DockerComposeFilter` must precede `DockerFilter` (both match `docker`). `DockerFilter` (custom `compress()` override; reverses stdout/stderr merge order via `combineOutput(stderr, stdout)` because Docker writes progress to stderr; drops `---> Using cache`/`---> xxxxxxxx` layer IDs, collapses `Step N/M` build steps via `dedupeNumericRuns`, head+tail cap at 60), `DockerComposeFilter` (custom `matches()` detects `docker-compose` binary OR `docker compose` two-word form; `errorPassthrough=true`; drops `Network created`/`Volume created`/`Pulling`/`Pulled from`/`Pull complete`/`Digest:`/`Status:` noise; dedupes service health-check wait lines; keep-first for `Creating`/`Starting`/`Stopping`/`Removing` lifecycle verbs), `KubectlFilter` (`errorPassthrough=true`; custom `matches()` on `kubectl`/`k` binaries; routes by first positional: `get`/`describe`/`apply`/`delete`/`rollout` → head+tail, `events` → `_compressKubectlEvents`, `logs` (non-logs-filter path) → head-30/tail-20, else generic truncation; tables: column-count collapse to first 3 rows + count note when ≥10 rows), `KubectlLogsFilter` (custom `matches()` requires `kubectl`/`k` binary + `logs` as first positional; `errorPassthrough=true`; only engages when `nonEmpty.length > 50` — faithful to Python; applies in order: `_collapseAccessLogs`, `_collapseStackTraces`, `_dedupLogLinesWithPodPrefix`, `_collapseJsonBlobs`, head+tail cap at 200 lines), `HelmFilter` (`errorPassthrough=true`; custom `matches()` on `helm`/`helmfile` binaries; routes by first positional: `install`/`upgrade` → `_compressHelmInstall` (drops STATUS/LAST_DEPLOYED/CHART/APP_VERSION/NAMESPACE noise, keeps NOTES: block), `list`/`ls` → `_compressHelmList` (head+tail 20), `template` → `_compressHelmTemplate` (counts YAML docs, keeps first 3 + last 1), else head+tail 50). `CONTAINER_FILTERS` array (5 entries; `kubectlLogsFilter` first). Golden tests in [`tests/tool_filters_containers.test.ts`](tests/tool_filters_containers.test.ts) (64 cases); e2e smoke (docker build) in [`tests/bash_compress_rewrite.test.ts`](tests/bash_compress_rewrite.test.ts). - **`cloud.ts`** (batch G) — 16 filters for the cloud/IaC landscape, faithful port of the Python `bash_compress.py` cloud filter family. The four cloud provider CLI filters (`AwsCliFilter`, `AwsFilter`, `GcloudFilter`, `AzureCliFilter`) live in **`cloud_providers.ts`**, and `CLOUD_FILTERS` in `cloud.ts` registers all 16 and sets their order. Dispatch order: `AwsCliFilter` must precede `AwsFilter` (both match `aws`/`aws2`; `AwsCliFilter` owns CFN/S3 routing; `AwsFilter` is the simpler JSON-array fallback). `TerraformFilter` (matches terraform/tofu/terragrunt; subcommand dispatch: plan — drops refresh/read lines, collapses `(known after apply)` attribute blocks, emits plan-summary first + last-20-line tail; apply — dedupes `Still creating` per-resource, keeps Creation complete + Apply complete!; init — head=5/tail=5 compression for provider installs; show/state — keeps key attributes only per resource block; validate/output passthrough; `errorPassthrough=true`), `AwsCliFilter` (json-array threshold 10/keep 3; `s3 cp/sync/mv` S3-transfer upload/download/progress collapse; `cloudformation describe-stack-events` CFN repeated IN_PROGRESS dedup via logical resource ID; generic JSON array fallback), `AwsFilter` (json-array threshold 20/keep 20; simple JSON-array or table truncation), `GcloudFilter` (spinner-char drop, `Enabling service`/`Waiting for async operation` collapse, structured-output collapse when >70% of lines are structured and >20 lines), `AzureCliFilter` (JSON-array threshold 10/keep 3; preview warning collapse; repeated provisioning progress-JSON collapse to last status), `AnsibleFilter` (`compress()` override — not `compressBody` — because ansible-playbook handles non-zero exits structurally; routes by binary: ansible-lint → rule-code grouping (keep first 3 per rule, elide rest); ansible-galaxy → head=5/tail=5; ansible-playbook — `ok`/`changed`/`skipped` lines collapsed per task section into `[token-goat: N ok, M changed ...]`, failure blocks kept verbatim, PLAY RECAP always preserved), `PulumiFilter` (drops `creating/updating/deleting` in-progress lines and `still` heartbeat lines, keeps completion/summary/diagnostic), `CdkFilter` (drops asset build/upload progress, drops `WORD_IN_PROGRESS` table rows, keeps `WORD_COMPLETE` rows and failure/summary lines), `VaultFilter` (`kv list`/`list` collapses paths to first 5 + elision note when >10; lease/token metadata lines dropped; table dividers dropped), `PackerFilter` (SSH/WinRM connection-wait polls collapsed to count note; provisioner step announcements collapsed; build-step lifecycle lines kept), `NixFilter` (fetch/substitute lines collapsed to count note; `building` derivation lines collapsed to count note; flake lock update lines collapsed to count note; error/note/warning always kept), `WranglerFilter` (asset upload lines collapsed to count note; skip lines collapsed; bulk-operation progress collapsed; build-step noise dropped), `HardhatFilter` (Compiling N files lines collapsed; Solc timing lines collapsed; passing-test tick lines (`✓`) collapsed; tx receipt noise dropped; test summary always kept), `ServerlessFilter` (Packaging/Uploading/Validating/Updating step lines collapsed to count note; CF IN_PROGRESS rows dropped; polling dots dropped; Service Information/endpoint lines kept), `FlyFilter` (Docker build step lines + `#N CACHED/DONE` layer lines collapsed to count note; per-machine wait lines collapsed; DNS polling lines dropped; step headers + summary lines kept), `ForgeFilter` (Compiling/Solc-timing lines collapsed to count note; passing-test `[PASS]` lines collapsed; gas-report table separator rows dropped; test-suite header/summary always kept). `CLOUD_FILTERS` array (16 entries; `awsCliFilter` before `awsFilter`). Golden tests in [`tests/tool_filters_cloud.test.ts`](tests/tool_filters_cloud.test.ts) (117 cases); e2e smoke (terraform plan) in [`tests/bash_compress_rewrite.test.ts`](tests/bash_compress_rewrite.test.ts). - **`ci.ts`** (batch H) — 9 filters for the CI-runner and security-scanner landscape, faithful port of the Python `bash_compress.py` CI and security filter families. Dispatch order: `GhRunLogFilter` must precede `GhFilter` (both match `gh`; `GhRunLogFilter` is more specific — requires `run view` positionals AND `--log` flag); `GenericCIFilter` is last (keyword-only match, no binary gate). `GhRunLogFilter` (`gh run view --log` — strips step-name TAB column prefix and ISO-8601 timestamps; drops `##[command]` echo lines unless they contain a failure signal; drops runner boilerplate/cleanup; collapses `##[group]...##[endgroup]` blocks >20 lines with no failure; summarises setup-action Run lines), `GhFilter` (`gh api/pr/run/issue` — includes the `python-gh-filter-ref` enhancement: recursive strip of boilerplate `*_url` JSON fields from `gh api` responses, preserving `html_url`/`avatar_url`/`clone_url`/`ssh_url` and stripping noise keys `gravatar_id`/`site_admin`; `compress()` override dispatches on subcommand: `run view` → pass-step collapse, `*/list` → 30-row cap, `api` → `*_url` strip + JSON pretty-print; redacts base64 `content` fields before dispatch), `ActFilter` (`act` binary — strips `[job/step] |` prefix from body lines, collapses Docker-pull progress, collapses matrix expansion, preserves ✅/❌ status lines verbatim), `PreCommitFilter` (`pre-commit` — collapses Passed/Skipped hook result lines into a count sentinel, keeps Failed blocks verbatim, keeps only the first `[INFO]` env-setup line and emits a dropped-count note for the rest), `BanditFilter` (`bandit` — per-file `testing` progress drop; HIGH/MEDIUM issue blocks kept verbatim, LOW blocks collapsed to count note; `Code scanned:`/`Total issues` stats block always kept), `TrivyFilter` (`trivy` — drops timestamped `INFO`/`WARN`/`DEBUG` log lines from stderr; parses table column positions dynamically to find SEVERITY/LIBRARY; keeps CRITICAL/HIGH rows, collapses MEDIUM/LOW to per-library count notes; always keeps `Total:` summary and no-vulnerability messages), `SnykFilter` (`snyk` — keeps first `Testing` line only; tree-lines >10 collapsed to count note; vulnerability block headers kept; `More about this vulnerability:` URL lines collapsed; summary line always kept; license lines always kept), `SemgrepFilter` (`semgrep` — per-rule instance cap of 3 via `ruleCounts`/`ruleSuppressed` maps; `Details:` and annotation URL lines dropped per block; scanning banner kept first-only; suppression notes injected before summary). `CI_FILTERS` array (9 entries; `ghRunLogFilter` first, `genericCIFilter` last). Golden tests in [`tests/tool_filters_ci.test.ts`](tests/tool_filters_ci.test.ts) (123 cases); e2e smoke (`gh run view --log`) in [`tests/bash_compress_rewrite.test.ts`](tests/bash_compress_rewrite.test.ts). -- **`shell_file.ts`** (batch J) — 20 filters for the shell/file-utility landscape, faithful port of the Python `bash_compress.py` shell/file filter family. Dispatch ordering inside `SHELL_FILE_FILTERS`: `RgFilter` precedes `GrepFilter` (RgFilter claims both `rg` and `grep`; GrepFilter is the simpler fallback for grep-only usage), `LsFilter` precedes `EzaFilter` (LsFilter claims `ls`/`eza`/`exa`; EzaFilter is reached only via `exa`). `GrepFilter` (matches grep/grep variants; head+tail 80; collapses same-file repeated-line runs to count note; `errorPassthrough=true`), `RgFilter` (matches rg + grep; context-line prefix stripping per `-A`/`-B`/`-C` depth; head+tail 80 cap; count aggregation from `N matches` summary lines), `LsFilter` (ls/eza/exa; collapses dot-files block to count note when total ≥30; drops extended-attribute lines `@` / ACL lines `+`; long-format: permission-char block collapse; binary/symlink/dir totals note), `EzaFilter` (eza-specific: strips header/footer frame lines; icon/colour noise drop via ANSI strip; keeps only `--long` content rows), `TreeFilter` (tree; strips box-drawing chars to count note for sub-trees deeper than 2 levels; collapses homogeneous leaf blocks to `N files` note; always keeps first+last lines), `FdFilter` (fd/fdfind; head+tail 50; collapses same-extension runs to count note; `errorPassthrough=true`), `WcFilter` (wc; strips leading whitespace from each line; collapses multiple blank-separated file groups to normalized lines), `BatFilter` (bat/batcat; strips decorative border lines top and bottom; keeps syntax-highlighted body; head+tail 100), `DeltaFilter` (delta; strips commit header block; keeps hunk headers + body; head+tail 120 when large), `FzfFilter` (fzf; head+tail 30; strips ANSI from selection lines), `LazyGitFilter` (lazygit; head+tail 60; strips raw ANSI escape sequences), `JqFilter` (jq; head+tail 100; collapses deep-nested repeated keys to count note when >50 lines), `YqFilter` (yq; head+tail 100; collapses repeated scalar-list blocks when >150 lines), `CurlFilter` (curl/wget; drops download-progress lines `%`/`#`/`-` patterns; keeps headers on `-v`/`-I`; head+tail 60 for response body), `RsyncFilter` (rsync; drops `xfr#N` transfer-progress lines; keeps summary stats line; drops paths with '/' when >100 lines and keeps first 20 + count note), `DiffFilter` (diff; hunk scoring via `_scoreAndCapHunks` — each hunk gets a density score capped at `MAX_HUNK_LINES`; keeps top-N hunks by score; always keeps file `---`/`+++` headers), `FfmpegFilter` (ffmpeg; drops frame/fps/bitrate progress lines; collapses stream-info block to first 3 lines; keeps error/warning lines), `BinaryInspectFilter` (xxd/hexdump/od/hd; head+tail 30; keeps magic-byte line + offset annotation), `FileTypeFilter` (file; head+tail 40; collapses runs of same MIME type to count note), `PsFilter` (ps/top; parses column headers dynamically; keeps header row; sorts by CPU% descending; caps at 30 rows; collapses low-CPU processes to count note). `SHELL_FILE_FILTERS` array (20 entries). Golden tests in [`tests/tool_filters_shell_file.test.ts`](tests/tool_filters_shell_file.test.ts) (137 cases); e2e smoke (rg) in [`tests/bash_compress_rewrite.test.ts`](tests/bash_compress_rewrite.test.ts). +- **`shell_file.ts`** (batch J) — 20 filters for the shell/file-utility landscape, faithful port of the Python `bash_compress.py` shell/file filter family. Dispatch ordering inside `SHELL_FILE_FILTERS`: `RgFilter` precedes `GrepFilter` (RgFilter claims both `rg` and `grep`; GrepFilter is the simpler fallback for grep-only usage), `LsFilter` precedes `EzaFilter` (LsFilter claims `ls`/`eza`/`exa`; EzaFilter is reached only via `exa`). `GrepFilter` (matches grep/grep variants; one file's matches, or a result the command bounded to 200 lines or fewer with `| head -N` or `-m N`, ship as lines clipped around the match; otherwise the first 40 match lines ship verbatim and only the rest become per-file counts; its own closing notices carry the per-process `ownNotice` tag, so the post hook lifts them out of the untrusted fence by provenance and a matched line spelling the same text stays inside), `RgFilter` (matches rg + grep; context-line prefix stripping per `-A`/`-B`/`-C` depth; head+tail 80 cap; count aggregation from `N matches` summary lines), `LsFilter` (ls/eza/exa; collapses dot-files block to count note when total ≥30; drops extended-attribute lines `@` / ACL lines `+`; long-format: permission-char block collapse; binary/symlink/dir totals note), `EzaFilter` (eza-specific: strips header/footer frame lines; icon/colour noise drop via ANSI strip; keeps only `--long` content rows), `TreeFilter` (tree; strips box-drawing chars to count note for sub-trees deeper than 2 levels; collapses homogeneous leaf blocks to `N files` note; always keeps first+last lines), `FdFilter` (fd/fdfind; head+tail 50; collapses same-extension runs to count note; `errorPassthrough=true`), `WcFilter` (wc; strips leading whitespace from each line; collapses multiple blank-separated file groups to normalized lines), `BatFilter` (bat/batcat; strips decorative border lines top and bottom; keeps syntax-highlighted body; head+tail 100), `DeltaFilter` (delta; strips commit header block; keeps hunk headers + body; head+tail 120 when large), `FzfFilter` (fzf; head+tail 30; strips ANSI from selection lines), `LazyGitFilter` (lazygit; head+tail 60; strips raw ANSI escape sequences), `JqFilter` (jq; head+tail 100; collapses deep-nested repeated keys to count note when >50 lines), `YqFilter` (yq; head+tail 100; collapses repeated scalar-list blocks when >150 lines), `CurlFilter` (curl/wget; drops download-progress lines `%`/`#`/`-` patterns; keeps headers on `-v`/`-I`; head+tail 60 for response body), `RsyncFilter` (rsync; drops `xfr#N` transfer-progress lines; keeps summary stats line; drops paths with '/' when >100 lines and keeps first 20 + count note), `DiffFilter` (diff; hunk scoring via `_scoreAndCapHunks` — each hunk gets a density score capped at `MAX_HUNK_LINES`; keeps top-N hunks by score; always keeps file `---`/`+++` headers), `FfmpegFilter` (ffmpeg; drops frame/fps/bitrate progress lines; collapses stream-info block to first 3 lines; keeps error/warning lines), `BinaryInspectFilter` (xxd/hexdump/od/hd; head+tail 30; keeps magic-byte line + offset annotation), `FileTypeFilter` (file; head+tail 40; collapses runs of same MIME type to count note), `PsFilter` (ps/top; parses column headers dynamically; keeps header row; sorts by CPU% descending; caps at 30 rows; collapses low-CPU processes to count note). `SHELL_FILE_FILTERS` array (20 entries). Golden tests in [`tests/tool_filters_shell_file.test.ts`](tests/tool_filters_shell_file.test.ts) (137 cases); e2e smoke (rg) in [`tests/bash_compress_rewrite.test.ts`](tests/bash_compress_rewrite.test.ts). - **`languages.ts`** (batch K1) — 17 filters for language runtimes and compilers, faithful port of the Python `bash_compress.py` K1 family. All 17 have `errorPassthrough = true` (stderr preserved on failure). Twelve are implemented as classes/bespoke instances; five are produced by `makeLanguageFilter` (the new factory in `families.ts`). **Dispatch ordering:** `bunFilter` is registered *before* `PACKAGE_MANAGER_FILTERS` because `NodePackageFilter` also claims `bun` (first-match wins); `swiftLintFilter` is NOT re-declared here — it belongs to `LINTER_FILTERS` (batch C). `NodeFilter` (`node -e`/`--eval`/`-p`/`--print`; success → 500-token cap; failure → node_modules and node:internal frame collapse with user-code frames preserved). `PythonFilter` (`python`/`python3`/`python3.1x`; excludes `python -m pytest`; traceback: first-2+last-3 frames for >10-frame traces; `_dedupeRepeatedLines` on 5+ consecutive identical lines; `_compressWarnings` keyed on warning-class+message, capped at 3 per key). `RubyFilter` (`ruby -e`/`-r`; backtrace compression; gem-install progress drop). `BunFilter` (`bun test`/`build`/`run`/`install`; download-line collapse `↕/↑/↓ pkg@ver`; test-suite result keep; error-section preserve). `DenoFilter` (`deno run`/`test`/`compile`/`check`; bundle/compile progress drop; error-chunk preserve). `FlutterFilter` (`flutter build`/`run`/`test`/`pub`; verbose framework-frame collapse; download-progress drop). `DartFilter` (`dart run`/`test`/`compile`/`pub`; stack-trace head+tail with omission marker). `SwiftFilter` (`swift build`/`run`/`test`/`package`; linker/linking-progress line collapse; compiler-warning dedup capped at 5). `XcodeFilter` (`xcodebuild`; per-phase line collapsing for `CompileC`/`Ld`/`CpHeader`/`Copy` lines; keeps error/warning lines). `MixFilter` (`mix`/`elixir`; compilation-warning dedup; test-suite summary keep; depencency-fetch drop). `ZigFilter` (`zig build`/`run`/`test`; note-line accumulation; error-chunk extraction). `RCmdFilter` (`R`/`Rscript`; message/warning dedup; `Loading required package` progress drop). Factory-built via `makeLanguageFilter`: `erlangFilter` (`erl`/`escript`/`rebar3`; crash-dump lines drop, error-chunk keep), `crystalFilter` (`crystal`; compile-error keep, progress drop), `haskellFilter` (`ghc`/`stack`/`cabal`; warning/note dedup, linking-progress collapse), `elmFilter` (`elm`; compile-error keep, dependency-fetch drop), `juliaFilter` (`julia`; precompile-progress drop, deprecation dedup — 7 ANSI-escape regexes guarded with `no-control-regex` disable comments), `powerShellFilter` (`pwsh`/`powershell`; error/warning dedup, progress-bar drop). `LANGUAGE_FILTERS` array (17 entries, no `bunFilter` — it is spread individually). Golden tests in [`tests/tool_filters_languages.test.ts`](tests/tool_filters_languages.test.ts) (81 cases); e2e smoke (python) in [`tests/bash_compress_rewrite.test.ts`](tests/bash_compress_rewrite.test.ts). - **`misc.ts`** (batch K2 — completes the 158-filter port) — 19 filters for DB clients, runner tooling, CSS preprocessors, system package managers, and generic utility/catch-all filters, split by family: `db_clients.ts` holds the four DB clients, `misc.ts` the other 15 and the `MISC_FILTERS` registry; faithful port of the Python `bash_compress.py` K2 family. **Dispatch ordering (critical):** `playwrightFilter` and `cypressFilter` are NOT in `MISC_FILTERS` — they are registered individually in `dispatch.ts` BEFORE `bunFilter` because `BunFilter.binaries` includes `bunx`, and `bunx playwright test`/`bunx cypress run` must route to their bespoke filters, not the generic bun handler. The remaining 17 filters are in `MISC_FILTERS` spread after `LANGUAGE_FILTERS`, the four DB clients first. **DB clients:** `PsqlFilter` (`psql` — migration path: ≥3 CREATE TABLE lines → summarise counts; SELECT path: state machine detects border/header/data rows, collapses >20 rows to first 5), `MySQLFilter` (`mysql`/`mysqldump` — routed by binary name; query path: 4-phase table state machine; dump path: keep first 3 CREATE TABLE blocks), `Sqlite3Filter` (`sqlite3` — schema passthrough when ≥50% non-empty lines are DDL; else collapse >20 data rows), `RedisCLIFilter` (`redis-cli` — SCAN output, bulk-OK collapse, long-list collapse). **Build/runner tooling:** `SysPackageFilter` (`apt-get`/`apt`/`apt-cache`/`apk`/`brew` — per-manager routing; apt: Get:N + Unpacking/Setting up + triggers collapse; apk: fetch/installing collapse; brew: first-3 progress lines sample), `WmicFilter` (`wmic` — collapses the fixed-width column padding of its table output to a two-space delimiter; `/format:csv` and `/format:list` pass through), `ProtocFilter` (`protoc`/`buf` — drop `[libprotobuf INFO]`, keep `WARNING`; dedup repeated identical warning lines), `SassFilter` (`sass`/`scss`/`lessc`/`node-sass` — sample first 5 write lines; drop map-write lines; dedup deprecation warnings by 60-char prefix, keep first 2 per key), `ToxFilter` (`tox`; `errorPassthrough=true` — collapse env-create/pip-progress/req-satisfied/tox4-separator noise), `NoxFilter` (`nox`; `errorPassthrough=true` — collapse venv-create/reuse/pip noise), `WasmPackFilter` (`wasm-pack`; `errorPassthrough=true` — drop `[INFO]` and Cargo `Compiling` lines; WASMPACK_DONE_RE uses regex `test()` not `match` because done message appears INSIDE `[INFO]: :-) Your wasm pkg...`), `NgFilter` (`ng`; `errorPassthrough=true` — subcommand routing: `test` → drop Karma log noise; `build`/`serve` → webpack chunk runs + table head+tail; other → capBytes 8192), `PlaywrightFilter` (`playwright`/`npx playwright`/`pnpx playwright`/`bunx playwright`; custom `matches()`; concatenates stdout+stderr directly; drops ✓/✔ pass lines and download progress), `CypressFilter` (`cypress`/`npx cypress`/`pnpx cypress`/`bunx cypress`; custom `matches()`; 5-state machine suppresses Run Starting/Results/Video sections and separator lines). **Generic/catch-all (ordered last in MISC_FILTERS):** `DotenvFilter` (`dotenv` — collapse ≥2 banner lines to `[dotenv] loaded N vars`), `EnvFilter` (`env`/`printenv` — passthrough when ≤20 vars; else suppress low-priority vars), `JsonArrayFilter` (`json` stem, content-based; key-set dedup of objects unless high-entropy values; truncate to 50 items; `detectFromCommand()` returns false), `SeverityLogFilter` (no binary; `matches()` returns false; `detect()` fires when ≥5 lines and ≥30% have log-level keywords; `_scoreLine`: ERROR/CRITICAL=1.0, WARN=0.5, INFO=0.1, DEBUG=0.0; keeps WARN+ lines plus 2-line context window and stack-trace continuations), `TailTruncFilter` (no binary; `matches()` returns false — explicit opt-in only via `filterByName('tail-trunc')`; keeps first 50 + marker + last 50 when >500 lines; NOTE: Python model has `matches()=True` as a universal catch-all post-execution, but TS pre-hook rewrites commands before execution making a universal catch-all prohibitively expensive for trivial commands). `MISC_FILTERS` array (16 entries; PlaywrightFilter and CypressFilter excluded — registered before bunFilter in dispatch.ts; TailTruncFilter last). Golden tests in [`tests/tool_filters_misc.test.ts`](tests/tool_filters_misc.test.ts) (108 cases); e2e smoke (psql + playwright) in [`tests/bash_compress_rewrite.test.ts`](tests/bash_compress_rewrite.test.ts). @@ -781,7 +784,7 @@ Steps: Eleven invariants, each established by a real defect that shipped. A change that touches any of them needs a test that fails when the invariant is removed, not just a green suite. -**Every path that persists bytes goes through `redactSecrets`, and it must run before any truncation, not merely before the write.** `storeBlob()` in [`src/disk_cache.ts`](src/disk_cache.ts) is the funnel for the JSON-envelope caches (bash-output, web-output, mcp-output), and it is fail-safe: a throwing regex does not write raw bytes. `src/skill_cache.ts`'s `storeOutput`/`storeCompact` persist skill bodies and compact slices as plain `.txt` files rather than through `storeBlob()`'s JSON envelope, so they call `redactSecrets` directly at their own write sites, with the same fail-safe shape (a throw propagates to the function's own outer try/catch, which returns without writing anything). Presence of a `redactSecrets` call on a path is not sufficient on its own: `redactSecrets`'s patterns carry a minimum-match-length floor, so a truncator (a `.slice()`, a line-width clip, a middle-cut, a byte cap) that runs *before* redaction can cut a credential's matched span short enough that the pattern no longer recognises what's left, and the fragment ships raw even though the path technically "goes through redactSecrets" downstream. Three sites shipped this exact defect before it was named as its own class: `mcp_cache.ts::mcpInputPreview` (truncated `JSON.stringify(toolInput)` to 120 chars before the caller redacted it), `tool_filters/base.ts::apply()` (redaction ran last, after four separate truncators had already clipped the body), and `mcp_compress_packs.ts::truncateSnapshotLine` via `hooks_mcp.ts::postMcpHandler` (compression, which truncates per-field, ran on raw `resultText` instead of the already-redacted text). All three fixes moved redaction to the boundary the path crosses — right after the raw bytes enter the function, before the first truncator — rather than reordering just that one call site, so a truncator added upstream later cannot reintroduce the class. A fourth, differently-shaped defect hit the same model-facing surface without any truncator being involved: `tool_filters/dispatch.ts::deliverCompressed` — documented in its own comment as "the single definition of what the model actually receives for this command" — computed a correctly-redacted `compressed.text`, then discarded it whenever `compressed.worthApplying(minNet)` (the net-benefit/byte-savings floor) returned false, falling back to the RAW pre-redaction `combineStreams(stdout, stderr)`. The net-benefit gate exists to decide whether *compression* is worth shipping, and shipping the redacted-but-uncompressed original instead is always safe; discarding *redaction* along with compression on that same gate is not a truncation-ordering bug (nothing was cut short here) but the same family of mistake — a value computed correctly upstream gets thrown away by a branch that was only ever supposed to arbitrate a different tradeoff. The invariant this generalizes to: a net-benefit or worth-applying gate may only ever discard the compression it is gating, never the redaction that ran ahead of it. `tests/guards/truncators_are_classified.test.ts` and `tests/guards/session_state_fields_are_redaction_classified.test.ts` turn both halves of this invariant into a structural pre-commit check: every truncation/shortening-shaped function in `src/` must carry an explicit classification (redacted-before-truncating, downstream of an already-redacted input, or otherwise verified safe) or the guard fails, and every field of `SerializedSession` must carry an explicit redaction-coverage classification or the guard fails — so a fifth truncator or a sixth session-state field starts red until someone consciously classifies it, rather than silently inheriting whatever the surrounding code happened to do. The per-session state file, written directly by [`src/session_store.ts`](src/session_store.ts) rather than through `storeBlob()`, has the same shape of gap: adding a new field to `SerializedSession` used to mean redaction depended entirely on that field's author remembering to call `redactSecrets` at its own construction site in [`src/session.ts`](src/session.ts) — `outstandingAgentSpawns` shipped without it, persisting a raw prompt (and any credential pasted into it) verbatim. The fix is two-layered: the field's own writer (`recordOutstandingAgentSpawn`) now redacts at push time, and `saveSessionState` additionally sweeps the fully-serialized JSON through `redactSerializedJson` once, immediately before the only `atomicWriteText` call that ever writes this file — a defense-in-depth backstop so a *future* field that ships without its author remembering redaction is still covered, rather than relying on that memory a second time. A pass over serialized JSON goes through `redactSerializedJson` in [`src/secret_redact.ts`](src/secret_redact.ts), never `redactSecrets` directly, and `storeBlob()` and `mcp_cache.ts::mcpInputPreview` do the same: the patterns are written for raw text, and JSON spells every control character as an escape, so a line break reads as the two characters `\` and `n`. Read that way, an empty `API_KEY=` took every following line as its value, an `AccountKey=` at the start of a line missed its word-boundary anchor, and strict mode left a lone backslash that made the stored blob unparseable, as it also did when its run started at the `u` of the escape JSON.stringify writes for a surrogate with no partner. `redactSerializedJson` decodes both kinds of escape before the patterns run and writes them back after. Redaction is a pattern matcher, not a classifier: adding a pattern is cheap, and a credential shape with no distinctive prefix (presigned urls were the last one found) needs an anchor plus a length floor so a generic parameter name stays safe to key on. Persisting redacted is not the same guarantee as *showing* the model redacted text this turn: `postMcpHandler` in [`src/hooks_mcp.ts`](src/hooks_mcp.ts) redacts its compression branch's rewrite, and `postWebSearchHandler`/`postFetchHandler` in [`src/hooks_websearch.ts`](src/hooks_websearch.ts)/[`src/hooks_fetch.ts`](src/hooks_fetch.ts) now redact on every live-rewrite path (computed once, ahead of every early-return caching guard, the same ordering discipline the injection scan below already enforces) — closing the gap where a credential that tripped no injection pattern reached the model unredacted even though the cached copy on disk was clean. +**Every path that persists bytes goes through `redactSecrets`, and it must run before any truncation, not merely before the write.** `storeBlob()` in [`src/disk_cache.ts`](src/disk_cache.ts) is the funnel for the JSON-envelope caches (bash-output, web-output, mcp-output), and it is fail-safe: a throwing regex does not write raw bytes. `src/skill_cache.ts`'s `storeOutput`/`storeCompact` persist skill bodies and compact slices as plain `.txt` files rather than through `storeBlob()`'s JSON envelope, so they call `redactSecrets` directly at their own write sites, with the same fail-safe shape (a throw propagates to the function's own outer try/catch, which returns without writing anything). Presence of a `redactSecrets` call on a path is not sufficient on its own: `redactSecrets`'s patterns carry a minimum-match-length floor, so a truncator (a `.slice()`, a line-width clip, a middle-cut, a byte cap) that runs *before* redaction can cut a credential's matched span short enough that the pattern no longer recognises what's left, and the fragment ships raw even though the path technically "goes through redactSecrets" downstream. Three sites shipped this exact defect before it was named as its own class: `mcp_cache.ts::mcpInputPreview` (truncated `JSON.stringify(toolInput)` to 120 chars before the caller redacted it), `tool_filters/base.ts::apply()` (redaction ran last, after four separate truncators had already clipped the body), and `mcp_compress_packs.ts::truncateSnapshotLine` via `hooks_mcp.ts::postMcpHandler` (compression, which truncates per-field, ran on raw `resultText` instead of the already-redacted text). All three fixes moved redaction to the boundary the path crosses — right after the raw bytes enter the function, before the first truncator — rather than reordering just that one call site, so a truncator added upstream later cannot reintroduce the class. A library can be the truncator: html-to-text cuts its input at `limits.maxInputLength` (1 << 24 by default) with no marker, so the curl HTML fold (`hooks_bash_post.ts::maybeFoldCurlHtml`) redacted the text of a page already cut through a credential; it and WebFetch's compression (`hooks_fetch.ts::postFetchHandler`) now redact the page before `extractCleanText` as well as after, and `extractCleanText` raises that limit to `MAX_CAPTURE_BYTES` or the input's own length, whichever is larger, since a placeholder can be longer than the credential it replaced. A strip joins the way a cut splits: a credential a colour code splits in two reads as one once the escape is gone, and the tool-output fence, `out()` and a terminal all remove it. So `redactSecrets` reads the `withoutEscapes` view of its input as well, and returns that view redacted when only it held a credential, which covers every site that redacts and then fences, and the Read fold gates' `hasPreciseSecret` reads the same view. `indexRecallEntry` stores `withoutEscapes` of its content, redacted, and recall's snippet strips with the same function, so no second, different strip can join a key the index passed: a single whole-text strip left an OSC open across a line break, and the snippet's own strip then closed it. `redactSerializedJson` keeps to the patterns alone, because the strip would cut a serialized document's quotes. The served-line elision of a pure file read declines on a credential as both folds and the generic elision do; it used to hand the command's unserved rows back unredacted, and its fence joined a colour-split key. A fourth, differently-shaped defect hit the same model-facing surface without any truncator being involved: `tool_filters/dispatch.ts::deliverCompressed` — documented in its own comment as "the single definition of what the model actually receives for this command" — computed a correctly-redacted `compressed.text`, then discarded it whenever `compressed.worthApplying(minNet)` (the net-benefit/byte-savings floor) returned false, falling back to the RAW pre-redaction `combineStreams(stdout, stderr)`. The net-benefit gate exists to decide whether *compression* is worth shipping, and shipping the redacted-but-uncompressed original instead is always safe; discarding *redaction* along with compression on that same gate is not a truncation-ordering bug (nothing was cut short here) but the same family of mistake — a value computed correctly upstream gets thrown away by a branch that was only ever supposed to arbitrate a different tradeoff. The invariant this generalizes to: a net-benefit or worth-applying gate may only ever discard the compression it is gating, never the redaction that ran ahead of it. `tests/guards/truncators_are_classified.test.ts` and `tests/guards/session_state_fields_are_redaction_classified.test.ts` turn both halves of this invariant into a structural pre-commit check: every truncation/shortening-shaped function in `src/` must carry an explicit classification (redacted-before-truncating, downstream of an already-redacted input, or otherwise verified safe) or the guard fails, and every field of `SerializedSession` must carry an explicit redaction-coverage classification or the guard fails — so a fifth truncator or a sixth session-state field starts red until someone consciously classifies it, rather than silently inheriting whatever the surrounding code happened to do. The per-session state file, written directly by [`src/session_store.ts`](src/session_store.ts) rather than through `storeBlob()`, has the same shape of gap: adding a new field to `SerializedSession` used to mean redaction depended entirely on that field's author remembering to call `redactSecrets` at its own construction site in [`src/session.ts`](src/session.ts) — `outstandingAgentSpawns` shipped without it, persisting a raw prompt (and any credential pasted into it) verbatim. The fix is two-layered: the field's own writer (`recordOutstandingAgentSpawn`) now redacts at push time, and `saveSessionState` additionally sweeps the fully-serialized JSON through `redactSerializedJson` once, immediately before the only `atomicWriteText` call that ever writes this file — a defense-in-depth backstop so a *future* field that ships without its author remembering redaction is still covered, rather than relying on that memory a second time. A pass over serialized JSON goes through `redactSerializedJson` in [`src/secret_redact.ts`](src/secret_redact.ts), never `redactSecrets` directly, and `storeBlob()` and `mcp_cache.ts::mcpInputPreview` do the same: the patterns are written for raw text, and JSON spells every control character as an escape, so a line break reads as the two characters `\` and `n`. Read that way, an empty `API_KEY=` took every following line as its value, an `AccountKey=` at the start of a line missed its word-boundary anchor, and strict mode left a lone backslash that made the stored blob unparseable, as it also did when its run started at the `u` of the escape JSON.stringify writes for a surrogate with no partner. `redactSerializedJson` decodes both kinds of escape before the patterns run and writes them back after. Redaction is a pattern matcher, not a classifier: adding a pattern is cheap, and a credential shape with no distinctive prefix (presigned urls were the last one found) needs an anchor plus a length floor so a generic parameter name stays safe to key on. Persisting redacted is not the same guarantee as *showing* the model redacted text this turn: `postMcpHandler` in [`src/hooks_mcp.ts`](src/hooks_mcp.ts) redacts its compression branch's rewrite, and `postWebSearchHandler`/`postFetchHandler` in [`src/hooks_websearch.ts`](src/hooks_websearch.ts)/[`src/hooks_fetch.ts`](src/hooks_fetch.ts) now redact on every live-rewrite path (computed once, ahead of every early-return caching guard, the same ordering discipline the injection scan below already enforces) — closing the gap where a credential that tripped no injection pattern reached the model unredacted even though the cached copy on disk was clean. **The line that matters is `rewriteOutput`, not the hook.** A `pass` result is the harness's own tool output, which token-goat neither authored nor can alter; redacting there would be a different feature (suppressing what the model would see regardless), not a leak fix. What must always hold is that text token-goat *substitutes* for a tool result is clean. `postAgentHandler` in [`src/hooks_agent_spawn.ts`](src/hooks_agent_spawn.ts) violated exactly this: it cached the subagent report through `storeMcpOutput` (which redacts) and, on the next line, built its compacted replacement from the same raw string. Redact where the value *enters* the handler, never on the branch that happens to emit — the entry point is the only place a future branch inherits it for free. [`tests/guards/rewritten_output_never_carries_a_secret.test.ts`](tests/guards/rewritten_output_never_carries_a_secret.test.ts) enforces this by firing every registered `post_tool_use` handler and reading what it emits, with the population taken from the registry rather than a list. It is behavioral, not static, for a measured reason: a static "reaches a rewrite must reach the redactor" rule flags `hooks_bash_post.ts`, whose compression rewrite is redacted inside `compressOutput()` in another module — invisible to same-file call analysis, and silenceable only with the kind of hand-maintained list these guards exist to eliminate. @@ -791,7 +794,7 @@ Eleven invariants, each established by a real defect that shipped. A change that **The injection scan runs before every early return, never after.** `postMcpHandler` in [`src/hooks_mcp.ts`](src/hooks_mcp.ts) computes the scan and a `passOrFence()` closure before the first guard. Three caching guards (error response, missing session id, dedup hit) each used to return above the scan, so a hostile server could flag its own reply as an error, or simply be called twice, and get an unfenced payload through. A guard that exists to save work must never double as a way around a security control. The same rule applies to any new guard added to a hook that fences or redacts. -**Untrusted content is fenced by provenance, not by heuristic.** Every surface fences unconditionally: the scan decides whether the notice names the patterns it matched, never whether the fence appears. Eight sites each restated that rule and each got it wrong the same way (`if (matches.length === 0) return text`), so the decision now lives in one module, [`src/untrusted_fence.ts`](src/untrusted_fence.ts) -- `fenceUntrusted` for a printed block, `fenceWithMatches` where a hook already scanned, `scanAndRecord` for the stat. The one legitimate gate is `injection.enabled = false`, which is explicit user configuration rather than a detector, and it switches off the whole subsystem including the fence. The fence tags in [`src/injection_scan.ts`](src/injection_scan.ts) are per-surface (`UNTRUSTED_WEB_TAG`, `UNTRUSTED_TOOL_TAG`, `UNTRUSTED_FILE_TAG`, `UNTRUSTED_GITHUB_TAG`) so escaping one does not escape another. `neutralizeFenceMarkers` has to match as loosely as an HTML parser reads: case-insensitive, whitespace anywhere inside the tag, junk attributes, and a slash before the closing bracket. Every spelling that reads as a closing tag but is not escaped is a way out of the fence. Cached Bash and MCP output recalled later is third-party text exactly as a fetched page is: `bash-output` and `mcp-output` fence on recall, `web-output` too. So is `recall` (unified search over those same three caches — its no-query browse mode returns a raw excerpt with no scan skipped even when nothing matched a search term), `gdrive-sections` (a Google Doc, fetched over HTTP like a WebFetch page, same `UNTRUSTED_WEB_TAG`), and `pr-slice` (a GitHub PR's title/description/comments/diff, under its own `UNTRUSTED_GITHUB_TAG` since it is neither a page token-goat fetched itself nor a recall of a prior tool call). `_applyFiltersAndPrint` in `src/cli_cached_output.ts` still defaults its `fenceByProvenance` parameter to `false`, because it also prints the caller's own files; nothing makes that default safe by itself, and every third-party caller passes `true`. A structural guard, [`tests/guards/third_party_content_reaches_fence.test.ts`](tests/guards/third_party_content_reaches_fence.test.ts), parses every function in `src/` and fails when one reaches a named third-party content source (`fetchPrFiles`/`fetchPrDiff`/`fetchPrComments`/`fetchPrDescription`, `fetchDoc`/`getSectionContent`/`getDocSections`, `searchRecall`/`listRecentRecall`) without also reaching a fence terminal (`fenceUntrustedContent`, `fenceUntrusted`, `fenceWithMatches`, `fenceUntrustedOcrText`) or `_applyFiltersAndPrint(..., true, ...)`, transitively through same-file calls, **and** fails when a function that does reach one can still skip it via a zero-match early return — so a new command reusing one of those functions is caught the turn it is written, with nobody needing to remember a checklist. A wholly new fetch primitive that bypasses every name in that list is the one gap the guard discloses in its own header comment; it needs a human to add the new entry point, the same way a newly-lock-worthy config setting needs a human decision in `project_config_lock_coverage.test.ts`'s allowlist. A local file is third-party content too when the caller only *named* it rather than authored it: `pdf-extract`, `pdf-locate`, `pdf-outline`, `docx-text`, `docx-outline`, `pptx-outline`, `pptx-slide`, `pptx-notes`, `pptx-text`, and `xlsx-sheets`/`xlsx-head`/`xlsx-range`/`xlsx-query` in [`src/cli_office.ts`](src/cli_office.ts) all fence their extracted text under `UNTRUSTED_FILE_TAG` whatever it says. So do the data-file readers (`csv-query`, `csv-profile`, `zip-list`, `zip-read`, `sqlite-query`, `sqlite-schema`, `sqlite-tables`, `describe` on a database) and `dep-docs`, whose README, package.json fields and declarations a package publisher wrote; `html-query` and `html-outline` fence under `UNTRUSTED_HTML_TAG`. Each redacts, then caps, then fences what the cap kept, with the cap marker below the closing tag (`guardThenFence` and `guardRedactAndFence` in [`src/read_commands.ts`](src/read_commands.ts)): the cap keeps leading lines only, so fencing first cut the closing tag off a long result. `--json` is the single documented exception, and it is a wire-format constraint rather than a judgement call: a fence wrapped around JSON is no longer JSON, so those envelopes fence individual fields on a pattern match only, while the printed form of the same command is always fenced. Each exception is named in `CONDITIONAL_FENCE_EXCEPTIONS` in the guard with its reason, and the guard also fails on a stale entry, so an exception that stops being conditional cannot sit there licensing a gap that no longer exists. This family used to be left **out** of `THIRD_PARTY_SOURCE_CALLS` above, on the reasoning that its entry points are also called by [`src/doc_embed_extract.ts`](src/doc_embed_extract.ts) to feed the embeddings/chunking pipeline rather than to print, so listing them would flag that file. That left thirteen document commands outside the guard's view entirely while it reported green. All thirteen are now listed, named by the local alias each is imported under in `cli_office.ts` (`runPdfExtractText`, `docxText`, `pptxOutline`, `xlsxHeadSheet`, ...), and the false positive is handled where it belongs: a module that *defines* one of these functions, `doc_embed_extract.ts` included, is exempt from the check, so an internal consumer no longer costs the emit paths their coverage. Behavioral coverage for the family stays in [`tests/cli_doc_extract_fencing.test.ts`](tests/cli_doc_extract_fencing.test.ts), which asserts the actual fenced output per command, the same proof strategy the guard's header comment already prescribes for `bash-output`/`web-output`/`mcp-output`. +**Untrusted content is fenced by provenance, not by heuristic.** Every surface fences unconditionally: the scan decides whether the notice names the patterns it matched, never whether the fence appears. Eight sites each restated that rule and each got it wrong the same way (`if (matches.length === 0) return text`), so the decision now lives in one module, [`src/untrusted_fence.ts`](src/untrusted_fence.ts) -- `fenceUntrusted` for a printed block, `fenceWithMatches` where a hook already scanned, `scanAndRecord` for the stat. The one legitimate gate is `injection.enabled = false`, which is explicit user configuration rather than a detector, and it switches off the whole subsystem including the fence. The fence tags in [`src/injection_scan.ts`](src/injection_scan.ts) are per-surface (`UNTRUSTED_WEB_TAG`, `UNTRUSTED_TOOL_TAG`, `UNTRUSTED_FILE_TAG`, `UNTRUSTED_GITHUB_TAG`) so escaping one does not escape another. `neutralizeFenceMarkers` has to match as loosely as the model reads, which is looser than an HTML parser: case-insensitive, whitespace anywhere inside the tag, junk attributes, a slash before the closing bracket, and any character a reader passes over wherever it sits: format and control characters (a zero-width space, a word joiner, a soft hyphen), every other default-ignorable code point (a variation selector, the combining grapheme joiner, a Hangul filler, the tag block), combining marks and lone surrogates. Look-alikes read as what they look like: letters go through NFKD with the accents dropped (fullwidth, mathematical and accented letters, ligatures) and through Unicode's confusables data (UTS #39), every entry whose prototype is an ASCII letter or digit, a dash, a slash, a bracket or a colon, generated into [`src/fence_confusables.ts`](src/fence_confusables.ts) by `scripts/generate-fence-confusables.mjs` from the extract in `scripts/data/confusables-fence.txt`, plus a short supplement of look-alikes the audits found that the data does not list; a character the data and its decomposition read differently is read either way, the tag is matched on `untrusted` alone, so a missing, soft or look-alike hyphen changes nothing, the small and fullwidth `<` and `>` and the angle brackets of other scripts and symbol blocks are brackets, and any run of slashes, backslashes, fullwidth, division, fraction, box-drawing or katakana slashes may come before the name. An ASCII character is read without normalizing it, and only a code point in a fixed set of Unicode ranges (`FOLDING_RANGES`) is decomposed at all, once per call; the rest read as themselves, and what a call works out for one of them is kept in a direct-mapped table of 4,096 entries indexed by its low bits and overwritten on a collision, so a letter a text repeats is classified and read once. A test that walks every code point proves the ranges hold every one whose decomposition reads as a character the scans spell or match, and nothing a text puts first can push a later character out of the per-call map, which a capped first-come cache allowed. Reading look-alikes costs most where the old scan had nothing to match: 2,000,000 characters of an opener before each fullwidth, Thai, CJK or Hangul letter fence in 0.1 to 0.4 s, against under 15 ms before look-alikes were read at all, and `[` before a Thai letter took 1.3 s while nothing outside the ranges was kept. In a file the reading is narrower in one place: an opening ``, `count ` on its own line only, so a `return a > b` lines further down is left alone. A hand list of 32 letters stood in for the data and let some two thousand look-alikes through; [`tests/fence_confusables.test.ts`](tests/fence_confusables.test.ts) checks every entry of the extract through the scans' reading and through the fence builders, and fails when the table is stale. The scans are linear, at most twelve character reads per character across a tool and a file fence on the worst shapes [`tests/fence_scan_bounded.test.ts`](tests/fence_scan_bounded.test.ts) counts, and bounded: past `FENCE_SCAN_CEILING` (4 Mi code units) they read nothing and every bracket that reads as `<`, `>` or `[` is escaped outright. A look-alike the data does not list is the residual risk, and it no longer closes anything by itself: each fence's two tags carry a nonce of 48 random bits drawn when the fence is built (`` and the closing tag with the same value), so the real closing tag is the one string the fenced bytes cannot produce. The value is unquoted because a fence often travels inside a JSON string, where a quote would be escaped. `neutralizeOutsideFences` recognises a region only by an opening tag with a nonce and the closing tag that repeats it, looking every closing tag up in one pass so openers that each carry their own nonce stay linear, and `closeCutFence` in `src/manifest.ts` spells the close from the open. The escape still has to catch every spelling a model reads as a tag, since a model does not check the nonce; Claude Code marks the uploads it hands a model the same way (``). On that reading it escapes every `<` that opens `untrusted-`, whatever follows, so each surface's tags are escaped inside every fence, as is a longer name sharing the prefix. A pattern that wanted an HTML tag-name terminator after the name let ``, `` and `` through, and a pass-over set of format and control characters alone let ``, `` (a Cyrillic `\u0435`), `` and `\uff1c/untrusted-tool-output\uff1e` through. Every spelling that reads as a closing tag but is not escaped is a way out of the fence, and look-alikes are open-ended, so the generous reading is the point: an over-escape costs one visible `<`. `neutralizeSpokenMarkers` reads token-goat's own `[token-goat` and `[tg]` markers the same way, and `[tg` followed by a colon too, the shape of its `[tg: note]`, `[tg: ok]` and `[tg: stale-write]` lines, with or without the hyphen in `token-goat`, and in lenticular, mathematical white or tortoise-shell brackets as well as square ones. `(tg)` is ordinary prose and stays as written. The ANSI strip happens inside the tool-output fence builders, before the neutralizer, and nowhere after it can change a fenced body: `out()` and `emit()` strip ANSI from anything printed to a pipe, and `stripAnsiEscapes` is not idempotent (`ESC ESC[0m[0m` strips to `ESC[0m`), so a neutralizer that ran first let a nested escape be stripped into an exact closing tag or a raw `[tg]` deny voice after it had looked. `fencedBody` in [`src/injection_scan.ts`](src/injection_scan.ts), which every tool-output builder wraps bytes through, rewrites the 8-bit introducers to their 7-bit forms, strips each line to a fixpoint (at most eight passes, since each unwraps one layer and an unbounded loop is quadratic) and drops any ESC or C1 control left, so the body holds nothing for a later strip, C1-aware or not, to collapse. File content is the exception. `fenceUntrustedFileContent` and `fenceNumberedFileContent` go through `fencedFileBody`, which keeps a file's bytes, escapes included, because a model edits from a Read and a snapshot fixture stripped of its colour codes is a silent corruption. It keeps the escapes only when every ESC begins a complete SGR colour code (ESC, `[`, parameters of at most four digits separated by `;`, `m`) or a complete character-set designation (ESC, an optional `$`, a G0 to G3 designator, and a letter, or a digit followed by nothing a strip reads as more parameters or a final, so `ESC(B` from `tput sgr0` stays while `ESC(0` before a letter does not) and the text holds no 8-bit introducer (U+0090, U+0098, U+009B to U+009F), no lone surrogate and no BEL, and when the text with those codes removed holds no tag or marker. Every strip in use removes such a code as the same span: token-goat's own, the ECMA-48 grammar, ansi-regex from 6.1 (claude.exe's cy(), which deletes lone surrogates first, and Node's util.stripVTControlCharacters) to 6.4 (current strip-ansi). The exclusions are where they part: an 8-bit introducer opens a sequence for some of them, a lone surrogate deleted first joins the bytes around it, and a BEL after a colour code's letters ends the 6.1 pattern's string branch, which then takes all of it. Anything else writes every ESC and 8-bit introducer as U+241B, which no strip removes. The other C1 controls open no sequence in any of those strips and the tag check reads past them, so a file keeps them: `don` U+0092 `t`, a CP1252 quote read as Latin-1, comes back as it was, and an Edit built from it matches. The exception is a text that also holds a BEL: a strip that rewrote every C1 control as ESC and its 7-bit final would make U+0085 an ESC `E` that the 6.1 pattern's string branch takes through the BEL, so there the other C1 controls are written as U+241B too. Claude Code 2.1.296 does no such rewrite, and in fact strips nothing from hook output or tool results before the model sees them (measured against a mock API: ESC, C1 controls, BEL and every invisible character arrive unchanged, and only a lone surrogate changes, to U+FFFD), so the strips modelled here bound what a harness or a tool could do rather than describe what one does. Enumerating the grammars instead failed twice: checking token-goat's own missed `ESC(B` and colon parameters, and the ansi-regex pattern added for them was quadratic on a line of ESC followed by `;?`, holding a Read hook for 22.6 s on a 104 KB text file. The suite tries ESC followed by one or two printable characters, the 8-bit introducers with each string ending, every string of up to three of twenty symbols, the sixteen a security audit searched and four C1 controls that open nothing, and that audit's own inputs, under each of those strips, and none assembles. What a fence adds to a body is therefore no fixed amount: the strip shrinks coloured output, the neutralizer can double a body dense in markers, and a cut can end inside a marker the whole text spelled differently. So the two Bash rewrites that clip to the delivery cap before fencing (`fenceRewriteWithinCap`, `maybeFoldCurlHtml`) go through `fitToDeliveryCap` in [`src/delivery_cap.ts`](src/delivery_cap.ts), which frames each cut, measures it, and cuts again in proportion to the overrun and by at least a byte, then halves between the empty body and the shortest cut that did not fit, with an empty body as the floor; a cut backs off to the first byte of a character, so it never ends in U+FFFD or splits a surrogate pair. Both MCP oversized previews in `hooks_mcp.ts`, and every return there that fences an MCP result whole (an error reply, a repeated read-only call, a result too large to cache), are fitted the same way, at one byte under the 25,000 at which the harness spills an MCP result, with a note outside the closing tag that names the cached copy or says there is none, unless `TOKEN_GOAT_MCP_COMPRESS=0` has turned every shortening MCP rewrite off; and `clipToDeliveryCap` clips to nothing when the framing alone fills the cap instead of handing the text back whole. A reserve priced off the whole text let coloured output keep more than the cap holds and marker-dense output skip the clip altogether, and the harness, which cuts from the end, would take the closing tag with it. That put `stripAnsiEscapes` on every hook's fence, so its control-string alternative runs lazily to a string terminator only in the text before the last one, where a terminator is sure to be found; letting every opener scan to an ST that did not exist was quadratic on a line of `ESC P`, and stopping each at the first ESC instead let the payload of tmux's passthrough, which holds ESC bytes, through as text. Cached Bash and MCP output recalled later is third-party text exactly as a fetched page is: `bash-output` and `mcp-output` fence on recall, `web-output` too. So is `recall` (unified search over those same three caches — its no-query browse mode returns a raw excerpt with no scan skipped even when nothing matched a search term), `gdrive-sections` (a Google Doc, fetched over HTTP like a WebFetch page, same `UNTRUSTED_WEB_TAG`), and `pr-slice` (a GitHub PR's title/description/comments/diff, under its own `UNTRUSTED_GITHUB_TAG` since it is neither a page token-goat fetched itself nor a recall of a prior tool call). `_applyFiltersAndPrint` in `src/cli_cached_output.ts` still defaults its `fenceByProvenance` parameter to `false`, because it also prints the caller's own files; nothing makes that default safe by itself, and every third-party caller passes `true`. A structural guard, [`tests/guards/third_party_content_reaches_fence.test.ts`](tests/guards/third_party_content_reaches_fence.test.ts), parses every function in `src/` and fails when one reaches a named third-party content source (`fetchPrFiles`/`fetchPrDiff`/`fetchPrComments`/`fetchPrDescription`, `fetchDoc`/`getSectionContent`/`getDocSections`, `searchRecall`/`listRecentRecall`) without also reaching a fence terminal (`fenceUntrustedContent`, `fenceUntrusted`, `fenceWithMatches`, `fenceUntrustedOcrText`) or `_applyFiltersAndPrint(..., true, ...)`, transitively through same-file calls, **and** fails when a function that does reach one can still skip it via a zero-match early return — so a new command reusing one of those functions is caught the turn it is written, with nobody needing to remember a checklist. A wholly new fetch primitive that bypasses every name in that list is the one gap the guard discloses in its own header comment; it needs a human to add the new entry point, the same way a newly-lock-worthy config setting needs a human decision in `project_config_lock_coverage.test.ts`'s allowlist. A local file is third-party content too when the caller only *named* it rather than authored it: `pdf-extract`, `pdf-locate`, `pdf-outline`, `docx-text`, `docx-outline`, `pptx-outline`, `pptx-slide`, `pptx-notes`, `pptx-text`, and `xlsx-sheets`/`xlsx-head`/`xlsx-range`/`xlsx-query` in [`src/cli_office.ts`](src/cli_office.ts) all fence their extracted text under `UNTRUSTED_FILE_TAG` whatever it says. So do the data-file readers (`csv-query`, `csv-profile`, `zip-list`, `zip-read`, `sqlite-query`, `sqlite-schema`, `sqlite-tables`, `describe` on a database) and `dep-docs`, whose README, package.json fields and declarations a package publisher wrote; `html-query` and `html-outline` fence under `UNTRUSTED_HTML_TAG`. Each redacts, then caps, then fences what the cap kept, with the cap marker below the closing tag (`guardThenFence` and `guardRedactAndFence` in [`src/read_commands.ts`](src/read_commands.ts)): the cap keeps leading lines only, so fencing first cut the closing tag off a long result. `--json` is the single documented exception, and it is a wire-format constraint rather than a judgement call: a fence wrapped around JSON is no longer JSON, so those envelopes fence individual fields on a pattern match only, while the printed form of the same command is always fenced. Each exception is named in `CONDITIONAL_FENCE_EXCEPTIONS` in the guard with its reason, and the guard also fails on a stale entry, so an exception that stops being conditional cannot sit there licensing a gap that no longer exists. This family used to be left **out** of `THIRD_PARTY_SOURCE_CALLS` above, on the reasoning that its entry points are also called by [`src/doc_embed_extract.ts`](src/doc_embed_extract.ts) to feed the embeddings/chunking pipeline rather than to print, so listing them would flag that file. That left thirteen document commands outside the guard's view entirely while it reported green. All thirteen are now listed, named by the local alias each is imported under in `cli_office.ts` (`runPdfExtractText`, `docxText`, `pptxOutline`, `xlsxHeadSheet`, ...), and the false positive is handled where it belongs: a module that *defines* one of these functions, `doc_embed_extract.ts` included, is exempt from the check, so an internal consumer no longer costs the emit paths their coverage. Behavioral coverage for the family stays in [`tests/cli_doc_extract_fencing.test.ts`](tests/cli_doc_extract_fencing.test.ts), which asserts the actual fenced output per command, the same proof strategy the guard's header comment already prescribes for `bash-output`/`web-output`/`mcp-output`. **The three emit channels are not equally defended, so "which one does this reach" is a real question.** `denyOutput` neutralizes its payload and adds its `[tg]` prefix outside the neutralized region; `emitRewrite` is the boundary the substitution guard walks; `contextOutput` does neither, and its callers carry their own `[token-goat]` prefix inside the payload, which is why it cannot simply be made to neutralize without moving that prefix out at all 54 call sites. The consequence is not hypothetical: adding `/.claude/agents` as a scan root in 2.9.7 made the unrestricted-spawn advisory's names repository-authored, and they reached the model through `contextOutput` in token-goat's own voice with nothing between. The fix constrains the name's shape at the parser rather than fencing the notice, because the notice is ours and a fence around it would escape our own prefix. Two further things this cost, both worth carrying: the substitution guard's population is `emitRewrite`/`emitRewriteIfChanged` only, and widening it to `contextOutput` surfaces 11 functions that have never been adjudicated; and `postAgentHandler` was already exempt for a *different* body, so a function-keyed exemption silently covered a second block its written reason never mentioned. @@ -805,9 +808,9 @@ Three of the eleven candidates that sweep produced turned out to be unreachable, That alternative was built and then withdrawn, which is the part worth keeping. Escaping on the pass-through path fires on any output containing the literal `[token-goat`, and that includes token-goat's own source, its docs, and its changelog: `cat src/tool_filters/node_test_runners.ts` came back with escaped brackets, and a model round-tripping that text writes `[` into the file. The suite caught it, in `tests/read_served_output_producer.test.ts`, as four negative assertions flipping to `rewriteOutput`. A defence whose failure mode is corrupting the files it is defending is not worth a speculative gain on a path where token-goat is not speaking. So the escape lives where the substitution does, inside `fenceUntrustedContent`, and the pass-through paths are exempt from both. Reopen this if the marker prefix ever appears in the model's system prompt or documentation as something to trust, since that is the premise the whole trade rests on. -**Where the fence is available, and where it is not.** [`tests/guards/substituted_output_reaches_fence.test.ts`](tests/guards/substituted_output_reaches_fence.test.ts) walks every function reaching `emitRewrite`, and the eight it found split cleanly in two. Where token-goat's text sits at one end of the body -- the `BashOutput` and `TaskOutput` poll deltas, and the compound-Bash rewrite before them -- the notice goes above the opening tag and the fence covers the rest, which is the shape to copy. Where our markers are spliced *between* the bytes they replaced, there is no such cut: fencing the whole body would run the marker neutraliser over token-goat's own markers and hand the model `[token-goat: 40 lines elided]`. The Bash elisions are no longer in that position. [`fenceUntrustedSpans`](src/untrusted_fence.ts) takes the body already split into spans, each marked with whether token-goat wrote it, and runs the neutraliser on the others and on nothing else. Authorship is positional, declared by the producer that emitted the span, rather than recognised from the text: a rule that spotted our markers by their spelling would exempt a forged one just as readily. Both `hooks_bash_post.ts` elision sites fence through it, and the fence is priced before the net-benefit gate rather than after, so the gate cannot decline a rewrite for the cost of its own protection. Three remain open: the agent-report compaction, the already-served Read elision, and the browser block join. What keeps them open is the round-trip risk rather than the missing mechanism, since an untrusted span containing the literal `[token-goat` comes back escaped and those three feed the surfaces most likely to be copied into a file. Reopen them when there is a way to escape a span that a model cannot round-trip into an edit, or when a marker-shaped line is measured arriving on one of the three rather than assumed. +**Where the fence is available, and where it is not.** [`tests/guards/substituted_output_reaches_fence.test.ts`](tests/guards/substituted_output_reaches_fence.test.ts) walks every function reaching `emitRewrite`, and the eight it found split cleanly in two. Where token-goat's text sits at one end of the body -- the `BashOutput` and `TaskOutput` poll deltas, and the compound-Bash rewrite before them -- the notice goes above the opening tag and the fence covers the rest, which is the shape to copy. The compress runner (`bash_runner.ts::wrapAndCompress`) is the same shape without an `emitRewrite`: for a command the pre hook wrapped it prints its compressed or capped view itself, so it fences the command's bytes there and prints the filter's closing notices, the cap notice, its own notes on how the run ended (the capture cap, a spawn failure, the timeout kill), the marker and the recall line after the closing tag, split by the `ownNotice` tag rather than by wording; the fence is priced in its net-benefit gate with the marker and the recall line. Output the runner passes through unchanged stays unfenced under the 2026-09-04 decision above, as does `--no-compress`, and the run's notes follow it as lines of their own rather than riding in its stderr; neither kind of result puts those notes in the recall copy. The post hook leaves a result alone when it ends with the runner's recall line and matches the digest of everything the runner printed, which the runner stores beside the output (`BashOutputEntry.shipped`), so a runner view is never fenced twice. The line alone was once enough, and a command can print it: the command-hash redirect and an earlier run's id both resolved to an entry for the same command. The deliverCompressed check in the same guard turns red if the runner's printed body stops coming from the fence call. Where our markers are spliced *between* the bytes they replaced, there is no such cut: fencing the whole body would run the marker neutraliser over token-goat's own markers and hand the model `[token-goat: 40 lines elided]`. The Bash elisions are no longer in that position. [`fenceUntrustedSpans`](src/untrusted_fence.ts) takes the body already split into spans, each marked with whether token-goat wrote it, and runs the neutraliser on the others and on nothing else. Authorship is positional, declared by the producer that emitted the span, rather than recognised from the text: a rule that spotted our markers by their spelling would exempt a forged one just as readily. Both `hooks_bash_post.ts` elision sites fence through it, and the fence is priced before the net-benefit gate rather than after, so the gate cannot decline a rewrite for the cost of its own protection. Two remain open: the agent-report compaction and the already-served Read elision. The browser block join closed when the browser pass moved inside `postMcpHandler`: the server's text is fenced like any MCP result, and token-goat's screenshot notes travel as their own blocks beside it. What keeps the two open is the round-trip risk rather than the missing mechanism, since an untrusted span containing the literal `[token-goat` comes back escaped and those two feed the surfaces most likely to be copied into a file. Reopen them when there is a way to escape a span that a model cannot round-trip into an edit, or when a marker-shaped line is measured arriving on one of the three rather than assumed. -**A rewrite must leave the user's own permission rules deciding the original call.** Claude Code matches its permission rules against a PreToolUse hook's `updatedInput`, and `permissionDecision: "allow"` skips its prompt, so when `serializeOutput` answered every rewrite with allow, a compressible `curl` wrapped in `token-goat compress` ran with no prompt, a `Bash(curl:*)` deny never saw `curl` at the head of the command, and a shrunk image's temp path walked past a `Read(./private/**)` deny. Every rewriteInput producer (Bash compression, the structural rg/grep query, the Agent briefing, the image Read) goes through `permissionNeutralRewrite` in [`src/rewrite_permission.ts`](src/rewrite_permission.ts), which reads every settings source Claude Code could apply (managed file, registry and plist policy, the user files, every `.claude` directory from the cwd up) and declines the rewrite when a deny or ask rule could match the original or any source cannot be read. A registry policy key counts as absent only when `reg` gives its own not-found answer and gives the identical answer for a sentinel key under HKLM that no user can create; any other failure leaves the policy unreadable, because a user can deny read access to an HKCU key and so fake its absence. `reg` and PowerShell start only by absolute path from a validated Windows folder ([`src/windows_system32.ts`](src/windows_system32.ts)), never by bare name, since a bare name or an unvalidated `SystemRoot` lets a `reg.cmd` planted in the working directory answer in their place. Claude Code checks a path rule against both the requested path and the file it resolves to, and the rewrite shows it neither, so the image Read's path and every word of a shell command are also matched under `realpathSync.native` (symlinks, junctions, 8.3 short names, a not-yet-created file through its nearest existing ancestor); a junction `pics -> secrets` once carried a Read past `Read(./secrets/**)`. Its `approve` verdict is the only thing that puts allow on the Claude Code wire, and it is set only when the original would have run with no prompt anyway: a trusted allow rule matching a simple command, a built-in read-only command, an image Read inside the working directory. A `.claude/settings.local.json` allow is trusted only when `git ls-files --error-unmatch` exits 1 (untracked inside a repository); exit 128 once counted too, but it also means no repository at all, as in an unpacked archive that ships the file, or one git refuses under `safe.directory`. `loadPermissionSnapshot` anchors on `CLAUDE_PROJECT_DIR` (where the session started) as well as the hook's cwd: deny and ask rules count from every `.claude` folder above the cwd, the project and its main checkout (a linked worktree resolves through `commondir`), a cwd outside the project returns null, and a local allow counts only at the starting directory while the cwd is there and, off Windows, that directory is the checkout root, the one place old and new Claude Code versions agree they read it from. A rewrite is never approved in plan mode. In auto and bypassPermissions the user is never prompted, so a rewrite there is approved or skipped, never deferred: `decideRewrite` skips every kind in auto (auto drops broad allow rules and has its classifier review even read-only commands, so an allow skipped that review, and a plain rewrite could draw a prompt the original would not), and in bypassPermissions approves a non-skip verdict only when the snapshot has no `PermissionRequest` hook and [`src/claude_hidden_rules.ts`](src/claude_hidden_rules.ts) finds no source the settings files do not show: an entry point other than `cli`/`sdk-cli` (an SDK host's `canUseTool`, an editor), a claude command line carrying any flag outside its harmless allowlist (`--disallowedTools`, `--settings`), read from `CLAUDE_PID` through CIM, `/proc` or `ps`, a skill, command or agent `.md` whose `disallowed-tools` names a pattern, and a plugin cache `hooks.json` with `PermissionRequest`, scanning nested `.claude` folders through `git ls-files` because Claude Code skips gitignored ones. That module is a dynamic import primed by `loadingHiddenRuleCheck` on the bypass event, so it stays off the eager chunk, and an unprimed check counts as hidden. Every other mode withholds an `approve` the same way: a `PermissionRequest` hook in the snapshot, or the same module asked for the files only (skills, commands, agents and plugin hooks, not the claude process), on Claude Code and VS Code alike, and `lineRules` (the claude command line, where `--setting-sources` is harmless only when it names user, project and local) on Claude Code only; with the check unloaded no mode approves. A tracked skill file that is gone from disk is read from the git index, an untracked gone one adds no rule, and the nested scan is stamped like the fixed folders so a clean answer is dropped when a file it read changes, when the checkout's index or HEAD changes (the git path stamps both, and a linked worktree's or separate git directory's), or when a folder that git listed within three levels of the project, or on the path to a listed `.claude` at any depth, gains an entry; a scan whose stamp is under 2 s old (`RACY_MS`, git's racy-index rule, which also covers a `.claude` made between the listing and the stamping), cannot be read (only ENOENT and ENOTDIR are `absent`) or needs 2000 folders or more is `volatile` and its clean answer is rescanned on the next call; a `.claude` first created untracked more than three levels down in a folder with no listed file keeps the minute. [`tests/guards/bypass_and_auto_never_prompt.test.ts`](tests/guards/bypass_and_auto_never_prompt.test.ts) sweeps the decision and forbids any `ask` decision or a `rewriteInput` built outside this module. Every doubt resolves to no rewrite, because losing a compression costs tokens and skipping a rule costs the user's control. Codex is the one wire that still carries allow, since it applies `updatedInput` only beside it. Codex matches its execpolicy `prefix_rule`s against the command's words, which the wrapper hides, so a Codex shell rewrite goes through `decideCodex` in [`src/codex_rules.ts`](src/codex_rules.ts), a dynamic import that `loadCodexRules` primes from the Codex Bash hook so its Starlark reader stays off every other hook's eager path (unprimed, the rewrite is skipped): it ships only when no rule of any decision in any rules layer (the system folder, `~/.codex`, `CODEX_HOME`, every `.codex` from the cwd up) could match the original, because even an allow rule runs the original outside the sandbox, which the wrapper would lose. A rule headed by a shell needs only its script tokens, since Codex matches a script it cannot split as `[shell, flag, script]` (the TUI saves Windows approvals that way, as `pwsh.exe -Command