Skip to content

fix(testing): render the diagnostic slot in the fuzz echo gate - #2797

Merged
chaliy merged 1 commit into
mainfrom
eve/clever-mccarthy-agesqr
Oct 30, 2026
Merged

chaliy merged 1 commit into
mainfrom
eve/clever-mccarthy-agesqr

Conversation

@chaliy

@chaliy chaliy commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

What changed

bashkit::testing::input_echo_would_trip — the pre-filter fuzz targets use to
skip inputs the shell will echo back into a diagnostic — now renders the echo
slot with the punctuation a diagnostic places around it before matching, and
matches the trimmed text because the slot holds expr.trim().

Test-harness only. No runtime behavior changes: no builtin, interpreter or
diagnostic text is touched, and the leak detector itself stays exactly as
strict for every input that does run.

Why

Nightly Fuzz Testing run 295 went red on arithmetic_fuzz with the 6-byte
input wZTok:, and stayed red on main. Arithmetic diagnostics render
{diag_echo(expr.trim())}: {msg}, so the expression's trailing colon lands
next to the message separator's:

bash: line 1: wZTok:: syntax error in expression (error token is ":")

That spells the banned parser-token shape Tok:: across the join while the
script carries only Tok:. The tokenizer's Tok enum is never formatted and
Tok:: appears nowhere in the script — this is bashkit echoing the expression
back the way real bash does, so it is a false positive, not a TM-INF-022 leak.

The gate matched the raw input, which structurally cannot see a shape the
template's own punctuation completes. Runs 271 and 272 were the same
false-positive class caught inside the input; this is the first one spelled
at the boundary.

Kept in the gate rather than widening strip_real_shell_error_lines to the
arithmetic template on purpose: that template's {msg} half is bashkit's own
text, the one slot where a genuine Debug leak would appear, so stripping the
line would blind the detector exactly where it has to look. A skipped input
hides nothing, because the skip condition is that the shape is attributable to
the input.

Before / After

The crash input is the exact one CI reported — the file's SHA1 reproduces the
artifact name crash-8d7001253cb8a09eb51d50e3be6b6267565237b5:

$ od -An -tx1 crash-8d7001253cb8a09eb51d50e3be6b6267565237b5
 77 5a 54 6f 6b 3a
$ sha1sum crash-8d7001253cb8a09eb51d50e3be6b6267565237b5
8d7001253cb8a09eb51d50e3be6b6267565237b5

Before — the fuzz target's own path panics, byte-for-byte as in run 295
(same testing.rs:155, same raw stderr):

thread '...' panicked at crates/bashkit/src/testing.rs:155:5:
[arithmetic_fuzz] stderr leaks banned shape `Tok::` (after stripping shell echoes):
---raw stderr---
bash: line 1: wZTok:: syntax error in expression (error token is ":")
---stripped---
bash: line 1: wZTok:: syntax error in expression (error token is ":")
---

After — the gate recognizes the echo and the target skips the input; the
diagnostic itself is unchanged and still a faithful, bounded echo.

The gate stays narrow rather than becoming blanket. wZTo: — one character
short of the shape — is still admitted and still runs under the strict
detector, and the pre-existing run 270/271/272 regressions plus
same_shape_without_the_banned_text_is_still_checked all still pass.

Risk

  • Low. Test-harness only; no shipped code path and no public API.
  • The gate skips slightly more inputs, so arithmetic_fuzz, glob_fuzz and
    cpython_fuzz lose a sliver of coverage: only inputs whose own text spells
    a banned shape against the template's punctuation, which is exactly the set
    that can only produce false positives.
  • A genuine leak cannot hide behind the skip, because the skip condition is
    that the shape is attributable to the input. Every input that does run is
    checked by the unchanged detector.

Checklist

  • Tests added or updated — failing test written first: a unit test on the
    gate (input_echo_would_trip_sees_the_diagnostic_separator, covering the
    trimmed and space-prefixed { code: joins and two negative cases) and
    an end-to-end scaffold regression (run_295_banned_shape_straddles_the_separator,
    near_miss_at_the_separator_is_still_checked). Both confirmed failing
    before the fix and passing after.
  • Backward compatibility considered — internal #[doc(hidden)] test helper,
    no caller outside the fuzz targets and scaffold tests.
  • Knowledge updated — knowledge/security/threat-model.md records run 295
    and why the fix belongs in the gate rather than the strip layer.

https://claude.ai/code/session_01Wy7MBv7EWLPKBj4GqJeH5s


Generated by Claude Code

Nightly fuzz run 295 went red on `arithmetic_fuzz` with the 6-byte input
`wZTok:`. Arithmetic diagnostics render `{diag_echo(expr.trim())}: {msg}`,
so the expression's trailing colon lands next to the message separator's:

  bash: line 1: wZTok:: syntax error in expression (error token is ":")

That spells the banned parser-token shape `Tok::` across the join while the
script carries only `Tok:`. The tokenizer's `Tok` enum is never formatted and
`Tok::` appears nowhere in the script, so this is bashkit echoing the
expression back the way real bash does -- a false positive, not a TM-INF-022
leak.

`input_echo_would_trip` matched the raw input, which cannot see a shape the
template's own punctuation completes. It now renders the slot with the
punctuation a diagnostic places around it (`DIAGNOSTIC_ECHO_PAD`) before
matching, and matches the trimmed text because the slot holds `expr.trim()`.
The same join completes the space-prefixed ` { code:` shape from the
`bash: line N: ` prefix.

Kept in the gate rather than widening `strip_real_shell_error_lines` to the
arithmetic template on purpose: that template's `{msg}` half is bashkit's own
text, the one slot where a genuine Debug leak would appear, so stripping the
line would blind the detector exactly where it has to look. A skipped input
hides nothing, because the skip condition is that the shape is attributable
to the input.

The crash input reproduces the CI panic byte-for-byte before the fix (same
`testing.rs:155`, same raw stderr) and is gated after it. The pre-existing
run 270/271/272 regressions and the negative tests that prove the gate did
not become blanket all still pass.

Claude-Session: https://claude.ai/code/session_01Wy7MBv7EWLPKBj4GqJeH5s
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
bashkit 668a400 Commit Preview URL

Branch Preview URL
Oct 30 2026, 09:28 AM

chaliy commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Test (Python 3.9) failed on the first attempt of this PR (and took the aggregate Python Check gate down with it). It passed on re-run. Recording why it is not this PR's, so the next person to see it does not re-derive it.

The failure

tests/test_teardown_determinism.py::test_interpreter_exit_does_not_crash[abandoned] — 963 passed, 1 failed. A child interpreter aborted with SIGABRT:

assert proc.returncode == 0, proc.stderr.decode()
AssertionError: FATAL: exception not rethrown
assert -6 == 0

Why it is not this PR's

  • The diff cannot reach that code. All 113 added lines are in crates/bashkit/src/testing.rs (a #[doc(hidden)] test-only module), a comment in fuzz_targets/arithmetic_fuzz.rs, scaffold tests, and knowledge/security/threat-model.md. No Python. The one changed function, input_echo_would_trip, is referenced nowhere in bashkit-python or bashkit-scripted-tool — only from fuzz targets and scaffold tests.
  • Not red on the base. python.yml is green on the last 15 runs on main, including run 4611 on c5e4ab95, which is this PR's base commit.
  • It is a documented race. This is TM-PY-030 variant (3): the private-loop worker attaches on its exit path while the interpreter is finalizing, and Py_Finalize gc wakes it, which fatals CPython (PyGILState_Release). The threat model already records that Python::try_attach cannot detect finalization before 3.13, so on 3.9 the atexit-flag protocol leaves a timing window it cannot close. The [abandoned] case drives exactly that window — a 0.05 s timeout against a 30 s sleep, exiting with the callback in flight — and the test loops 10× to stress it.
  • Version-correlated. On the re-run, 3.10, 3.11, 3.12, 3.13, 3.14 and Examples all passed; only 3.9 had failed.

Fix

None in this PR, deliberately. Closing the residual pre-3.13 window is a change to the teardown protocol in crates/bashkit-python/src/lib.rs, unrelated to a fuzz-harness gate, and bundling it here would widen this PR into territory its tests do not cover. The mitigation is documented as inherently best-effort below 3.13; the honest options are a bounded retry on the [abandoned] parameterization for 3.9 (the shape ssh_supabase_tests already uses for a flaky-by-nature gate) or dropping 3.9, and both are their own change with their own argument to make.


Generated by Claude Code

@chaliy
chaliy merged commit ae635d1 into main Oct 30, 2026
63 of 65 checks passed
@chaliy
chaliy deleted the eve/clever-mccarthy-agesqr branch October 30, 2026 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant