Summary
checkSyntaxValidity for TypeScript/JavaScript is not a parser — it is bracket
matching plus a small set of regex "common error patterns"
(packages/hallucination-guard/src/checks/syntax-validity.ts,
checkJavaScriptSyntax → checkBrackets + checkCommonPatterns).
Consequence: natural-language prose passes the syntax check, because prose
has balanced brackets and matches none of the error patterns. This is not
hypothetical — it is the dominant real failure mode observed in a benchmark run
today.
When the model answers with a clarifying question instead of code, the agent
writes that answer into the target source file. tsc then reports
TS1127: Invalid character, but the guard has already passed the content, so
nothing is blocked and nothing is rolled back.
Reproduction
Measured directly against the guard, using the two files an agent run actually
produced:
formatDate.ts guard.validateCode pass = true | blockedBy = null
head = "我需要先澄清一下：您提供的代码中没有具体说明要做什么修改。..."
TodoList.tsx guard.validateCode pass = true | blockedBy = null
head = "无法完成。您提供了原始代码和一般性的修改要求..."
Both files are pure Chinese prose. Both were written to disk. tsc --noEmit
on them:
src/utils/formatDate.ts(1,9): error TS1127: Invalid character.
src/components/TodoList.tsx(1,5): error TS1127: Invalid character.
Minimal repro:
const guard = new HallucinationGuard({ projectRoot: '/tmp' });
await guard.validateCode('这是一段说明文字，不是代码。', 'typescript', 'a.ts');
// => { pass: true }Impact
In a 5-task smoke run of the ablation benchmark (post-#402), 2 of 5 tasks failed
with exactly this signature, and validation_failed was 0 for both — the guard
never objected. PR #402 makes the executor block writes that the guard flags
and emit validation_failed; it cannot help when the checker itself returns
pass: true. So this is now the binding constraint on interception rate, not
the executor plumbing.
Note the related weakness in cleanGeneratedCode
(packages/core/src/llm/code-generation.ts): its fence stripper is
/^```[\w]*\n/m with no g flag, so it removes only the first fence. Multi-block
model output still leaves inner fences in the file — another TS1127 source that
bracket matching will not catch.
Affected Area
hallucination-guard, executor, benchmarks
Environment
Suggested direction
A real parse is the only reliable check here. The constraint is that
packages/hallucination-guard intentionally ships with zero runtime
dependencies for CLI distribution, so adding @babel/parser or typescript as
a dependency is a maintainer-level call, not something to slip in.
Cheaper options that stay dependency-free and would have caught both cases:
- Reject content whose first non-blank, non-comment line does not look like
code (no import/export/const/function/class/type/interface/{/<).
This is the same heuristic cleanGeneratedCode already uses to find
codeStartIndex — it just is not used as a rejection signal.
- Reject a
.ts/.tsx file whose ratio of CJK (or generally non-ASCII
non-comment) characters outside string literals exceeds a small threshold.
- Fix the fence stripper to be global, and reject any residual ``` in output.
Happy to implement whichever direction maintainers prefer.
Summary
checkSyntaxValidityfor TypeScript/JavaScript is not a parser — it is bracketmatching plus a small set of regex "common error patterns"
(
packages/hallucination-guard/src/checks/syntax-validity.ts,checkJavaScriptSyntax→checkBrackets+checkCommonPatterns).Consequence: natural-language prose passes the syntax check, because prose
has balanced brackets and matches none of the error patterns. This is not
hypothetical — it is the dominant real failure mode observed in a benchmark run
today.
When the model answers with a clarifying question instead of code, the agent
writes that answer into the target source file.
tscthen reportsTS1127: Invalid character, but the guard has already passed the content, sonothing is blocked and nothing is rolled back.
Reproduction
Measured directly against the guard, using the two files an agent run actually
produced:
Both files are pure Chinese prose. Both were written to disk.
tsc --noEmiton them:
Minimal repro:
Impact
In a 5-task smoke run of the ablation benchmark (post-#402), 2 of 5 tasks failed
with exactly this signature, and
validation_failedwas 0 for both — the guardnever objected. PR #402 makes the executor block writes that the guard flags
and emit
validation_failed; it cannot help when the checker itself returnspass: true. So this is now the binding constraint on interception rate, notthe executor plumbing.
Note the related weakness in
cleanGeneratedCode(
packages/core/src/llm/code-generation.ts): its fence stripper is/^```[\w]*\n/mwith nogflag, so it removes only the first fence. Multi-blockmodel output still leaves inner fences in the file — another TS1127 source that
bracket matching will not catch.
Affected Area
hallucination-guard, executor, benchmarks
Environment
develop@ db42301 (v2.2.0) + PR fix(executor): validate writes before disk, wire rollback, emit validation_failed #402 branchclaude-haiku-4-5via Claude Code CLI text backendSuggested direction
A real parse is the only reliable check here. The constraint is that
packages/hallucination-guardintentionally ships with zero runtimedependencies for CLI distribution, so adding
@babel/parserortypescriptasa dependency is a maintainer-level call, not something to slip in.
Cheaper options that stay dependency-free and would have caught both cases:
code (no
import/export/const/function/class/type/interface/{/<).This is the same heuristic
cleanGeneratedCodealready uses to findcodeStartIndex— it just is not used as a rejection signal..ts/.tsxfile whose ratio of CJK (or generally non-ASCIInon-comment) characters outside string literals exceeds a small threshold.
Happy to implement whichever direction maintainers prefer.