Skip to content

fix: close the false-positive causes the precision benchmark found - #783

Merged
orieg merged 5 commits into
mainfrom
claude/benchmark-fp-fixes-468a2d
Oct 16, 2026
Merged

orieg merged 5 commits into
mainfrom
claude/benchmark-fp-fixes-468a2d

Conversation

@orieg

@orieg orieg commented Oct 16, 2026

Copy link
Copy Markdown
Owner

Refs #777

Fixes the four false-positive causes the public precision benchmark found (#482 Part B), one commit each. The benchmark is not rerun here: the rerun follows these fixes under a new pre-registration, and docs/benchmarks/ is untouched.

Changes

Commit Gate Cause Fix
fix(dependency-delta) dependency-delta A head dependency was compared with the first base record of its name in any section, so a devDependencies workspace link read as loosening a peerDependencies range (vitejs/vite packages/plugin-legacy/package.json) Each record carries its manifest section; the base counterpart is the same name in the same section. A name new to a section is reported as added there
fix(vacuous-tests) vacuous-tests, assertion-reduction Only a bare expect(..) receiver was an assertion, so tests made of expect.poll(fn).toBe(..) were vacuous (vitejs/vite playground specs) The receiver is read from the syntax tree: expect or its poll/soft member, also split across lines. A poll with no matcher, or .poll on another object, is not an assertion
fix(stub-bodies) stub-bodies, error-swallowing A file added under a test directory and holding no test is judged as production code; a linter's input file under resources/test/fixtures/ was reported (astral-sh/ruff) A file below a fixtures directory inside a test, tests or __tests__ directory is a test's input and stays test scope, for both gates that share the rule. A fixtures directory outside a test directory is not one, and a fixture that replaces a deleted production file is still judged as production code
fix(pii) pii, shell-secrets Home paths and RFC 1918 addresses in test code, a private-key header kept as a constant, a Bearer deadbeef12346 docstring example In test code (a test path, or the lines of a test the pack finds): a conventional test user name in a home path is not reported, and a LAN address is capped at warning. A private-key header that is the whole of one string literal, and a Bearer value starting with a hexadecimal example word, are not hits

Fixture classification: no other gate reads fixtures as its own scope; pii exempts only declared test scope and golden guards tests/fixtures/**/output* as committed output. The fixture rule is therefore placed in the one shared test-scope rule that stub-bodies and error-swallowing both use, rather than in either gate.

No real-leak bypass in pii: outside test code nothing changes, a real-looking user name in test code is still an error, and a LAN address in test code is still reported (as a warning). The e2e test keeps positive controls for a LAN host, a home path with a real-looking user, a private key with its body, and a Bearer token, all in non-test code.

Not addressed: a LAN address in a JSDoc example in production code (denoland/deno ext/node/polyfills/dgram.ts) is still reported at the gate's severity.

Evidence

Mutation evidence (RUN: each mutant applied, the named tests run and observed failing, then restored with git checkout):

Mechanism Mutant Killed by
dependency-delta base counterpart matched by name only (section dropped) guards::dependency::tests::a_dependency_is_compared_with_the_base_within_its_own_section, dependency_delta_compares_a_dependency_within_its_own_section
vacuous-tests EXPECT_RECEIVER_MEMBERS emptied ast::javascript::conditional_skip_tests::expect_poll_and_soft_receivers_are_assertions, vacuous_tests_read_expect_poll_and_soft_as_assertion_receivers
vacuous-tests any object's .poll accepted expect_poll_and_soft_receivers_are_assertions (negative control)
stub-bodies test_fixture_path not consulted stub_bodies_reads_a_test_fixture_input_as_test_scope
stub-bodies test-directory ancestor not required ast::functions::tests::a_test_fixture_is_a_file_below_fixtures_inside_a_test_directory
pii placeholder user names accepted outside test code pii_reads_test_code_and_placeholder_secrets_without_opening_a_leak
pii lone quoted private-key header check removed guards::token_formats::tests::a_private_key_header_alone_in_its_string_names_the_format, pii_reads_test_code_and_placeholder_secrets_without_opening_a_leak
pii hexadecimal example words removed guards::token_formats::tests::a_bearer_value_made_of_a_hex_example_word_is_a_placeholder, pii_reads_test_code_and_placeholder_secrets_without_opening_a_leak
pii LAN warning cap removed pii_reads_test_code_and_placeholder_secrets_without_opening_a_leak
pii test-code reading always false guards::hygiene::pii::tests::test_code_is_a_test_path_or_the_lines_of_a_test, pii_reads_test_code_and_placeholder_secrets_without_opening_a_leak

Each new e2e and unit test was also run before its fix and observed failing on the reproduction (RUN).

Spot check (RUN, not a benchmark rerun): the reproduction files fetched read-only from each finding's head commit, added to a scratch repository and checked with this branch's binary. pii reports nothing on the deno libs/core/error.rs home paths, the ruff options.rs home paths, the deno webcrypto PEM header constants or the fastapi Bearer docstring; the LAN addresses in the vite, deno and bench test code are warnings; dgram.ts stays an error.

Gates (RUN): cargo fmt --check, cargo clippy --all-targets -- -D warnings, discipline docs --check, and discipline check --policy-from base with this body pass locally. The full cargo test --no-fail-fast run (per-worktree CARGO_TARGET_DIR) was still running when this was opened; its result is posted as a comment.

Behaviour Changes: one unreleased row per change in docs/ROADMAP.md; docs/GATES.md updated for pii and the test-scope rule.

orieg added 4 commits October 16, 2026 12:17
…ection

A head dependency was compared with the first base record of the same
name in any section, so a devDependencies workspace link was read
against a peerDependencies range of the same name and reported as a
loosened constraint (7 false positives in the public precision
benchmark, vitejs/vite packages/plugin-legacy/package.json).

Each record now carries the table that declares it, and the base
counterpart is the same name in the same section. A name that appears
in a section the base did not list it in is reported as added there.

Refs #777

Signed-off-by: Nicolas Brousse <nicolas@brousse.info>
…eivers

`expect.poll(fn).toBe(..)` retries its matcher until it passes (Vitest,
Playwright) and `expect.soft(x)` records a failure without stopping the
test. The JavaScript pack recognised only a bare `expect(..)` receiver,
so tests made of polls were reported as vacuous (7 false positives in
the public precision benchmark, vitejs/vite playground specs).

The receiver is now read from the syntax tree: `expect` itself or its
`poll`/`soft` member, so `expect\n  .poll(..)` across lines is the same
callee. A poll with no matcher, or `.poll` on another object, is not an
assertion.

Refs #777

Signed-off-by: Nicolas Brousse <nicolas@brousse.info>
A file the change adds in test scope by its directory alone, holding no
test, is judged as production code by stub-bodies and error-swallowing,
to catch stubs hidden under a test name. A linter's input file under
`resources/test/fixtures/` holds the stub bodies its rule tests read,
and was reported (5 false positives in the public precision benchmark,
astral-sh/ruff airflow fixtures).

A file below a `fixtures` directory that is itself below a `test`,
`tests` or `__tests__` directory is now a test's input and stays test
scope, for both gates that share the rule. A `fixtures` directory
outside a test directory is not one, and a fixture file that replaces
a deleted production file is still judged as production code.

Refs #777

Signed-off-by: Nicolas Brousse <nicolas@brousse.info>
…bypass

The public precision benchmark found 43 of 46 heuristic-class false
positives in `pii`: home paths and RFC 1918 addresses that tests feed
their code (vitejs/vite, denoland/deno, astral-sh/ruff), a private-key
header kept as a constant a PEM parser compares with (deno webcrypto
tests), and a `Bearer deadbeef12346` example in a docstring (fastapi).

- Test code is a test path or the lines of a test the language pack
  finds (a Rust `#[test]` function in a production file). It is read
  only for a line that holds a candidate hit.
- In test code, a home path whose user name is a conventional test name
  (`dev`, `test`, `alice`, `jdoe`, ...) is not reported; any other user
  name still is, and outside test code only `allowed_users` applies.
- In test code, a private LAN address is capped at warning: still shown,
  no longer blocking.
- A private-key header that is the whole of one string literal names
  the format; a header followed by a body, or on its own line, is still
  a hit. The shared token table carries this for `shell-secrets` too.
- A Bearer value starting with a hexadecimal example word (`deadbeef`,
  `cafebabe`, `feedface`, `baadf00d`) is a placeholder.

Positive controls for real-looking leaks in non-test code (a LAN host, a
home path with a real-looking user, a key with its body, a Bearer
token) are in the e2e test.

Refs #777

Signed-off-by: Nicolas Brousse <nicolas@brousse.info>
@orieg
orieg enabled auto-merge (squash) October 16, 2026 19:23
The lone-header rule read any private-key header that was the whole of
one string literal as a format name. The shell-secrets corpus holds the
case it must not cover: `echo "-----BEGIN RSA PRIVATE KEY-----" >
key.pem` writes the first line of a key. The full suite caught it (6
false negatives in shell_secrets_corpus_evaluation).

The literal must now also be bound by `=` (`const pemHeader = "..."`, a
`=== "..."` comparison); a header echoed or printed alone is a hit
again. The unit test gains the echo and print cases as positive
controls; dropping the `=` condition fails both it and the corpus test.

Refs #777

Signed-off-by: Nicolas Brousse <nicolas@brousse.info>
@orieg

orieg commented Oct 16, 2026

Copy link
Copy Markdown
Owner Author

Full local cargo test --no-fail-fast (RUN, per-worktree CARGO_TARGET_DIR) at d513317: 4224 passed, 0 failed, 1 ignored across 141 test binaries.

The first full run, at c0c4442, failed guards::shell_secrets::tests::shell_secrets_corpus_evaluation with 6 false negatives. Those were echo "-----BEGIN … PRIVATE KEY-----" > key.pem lines, which write the first line of a key. CI failed the same test on that head. d513317 narrows the lone-header rule to a literal bound by =, so a header that is echoed or printed alone is reported again.

Mutation evidence for d513317 (RUN): dropping the = condition fails guards::token_formats::tests::a_private_key_header_alone_in_its_string_names_the_format (its new echo/print positive controls) and guards::shell_secrets::tests::shell_secrets_corpus_evaluation. docs/GATES.md and the ledger row now describe the narrower rule.

@orieg
orieg merged commit 9ecbedf into main Oct 16, 2026
25 checks passed
@orieg
orieg deleted the claude/benchmark-fp-fixes-468a2d branch October 16, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant