Repository navigation
fix: close the false-positive causes the precision benchmark found - #783
Conversation
…ection A head dependency was compared with the first base record of the same name in any section, so a devDependencies workspace link was read against a peerDependencies range of the same name and reported as a loosened constraint (7 false positives in the public precision benchmark, vitejs/vite packages/plugin-legacy/package.json). Each record now carries the table that declares it, and the base counterpart is the same name in the same section. A name that appears in a section the base did not list it in is reported as added there. Refs #777 Signed-off-by: Nicolas Brousse <nicolas@brousse.info>
…eivers `expect.poll(fn).toBe(..)` retries its matcher until it passes (Vitest, Playwright) and `expect.soft(x)` records a failure without stopping the test. The JavaScript pack recognised only a bare `expect(..)` receiver, so tests made of polls were reported as vacuous (7 false positives in the public precision benchmark, vitejs/vite playground specs). The receiver is now read from the syntax tree: `expect` itself or its `poll`/`soft` member, so `expect\n .poll(..)` across lines is the same callee. A poll with no matcher, or `.poll` on another object, is not an assertion. Refs #777 Signed-off-by: Nicolas Brousse <nicolas@brousse.info>
A file the change adds in test scope by its directory alone, holding no test, is judged as production code by stub-bodies and error-swallowing, to catch stubs hidden under a test name. A linter's input file under `resources/test/fixtures/` holds the stub bodies its rule tests read, and was reported (5 false positives in the public precision benchmark, astral-sh/ruff airflow fixtures). A file below a `fixtures` directory that is itself below a `test`, `tests` or `__tests__` directory is now a test's input and stays test scope, for both gates that share the rule. A `fixtures` directory outside a test directory is not one, and a fixture file that replaces a deleted production file is still judged as production code. Refs #777 Signed-off-by: Nicolas Brousse <nicolas@brousse.info>
…bypass The public precision benchmark found 43 of 46 heuristic-class false positives in `pii`: home paths and RFC 1918 addresses that tests feed their code (vitejs/vite, denoland/deno, astral-sh/ruff), a private-key header kept as a constant a PEM parser compares with (deno webcrypto tests), and a `Bearer deadbeef12346` example in a docstring (fastapi). - Test code is a test path or the lines of a test the language pack finds (a Rust `#[test]` function in a production file). It is read only for a line that holds a candidate hit. - In test code, a home path whose user name is a conventional test name (`dev`, `test`, `alice`, `jdoe`, ...) is not reported; any other user name still is, and outside test code only `allowed_users` applies. - In test code, a private LAN address is capped at warning: still shown, no longer blocking. - A private-key header that is the whole of one string literal names the format; a header followed by a body, or on its own line, is still a hit. The shared token table carries this for `shell-secrets` too. - A Bearer value starting with a hexadecimal example word (`deadbeef`, `cafebabe`, `feedface`, `baadf00d`) is a placeholder. Positive controls for real-looking leaks in non-test code (a LAN host, a home path with a real-looking user, a key with its body, a Bearer token) are in the e2e test. Refs #777 Signed-off-by: Nicolas Brousse <nicolas@brousse.info>
The lone-header rule read any private-key header that was the whole of one string literal as a format name. The shell-secrets corpus holds the case it must not cover: `echo "-----BEGIN RSA PRIVATE KEY-----" > key.pem` writes the first line of a key. The full suite caught it (6 false negatives in shell_secrets_corpus_evaluation). The literal must now also be bound by `=` (`const pemHeader = "..."`, a `=== "..."` comparison); a header echoed or printed alone is a hit again. The unit test gains the echo and print cases as positive controls; dropping the `=` condition fails both it and the corpus test. Refs #777 Signed-off-by: Nicolas Brousse <nicolas@brousse.info>
|
Full local The first full run, at c0c4442, failed Mutation evidence for d513317 (RUN): dropping the |
Refs #777
Fixes the four false-positive causes the public precision benchmark found (#482 Part B), one commit each. The benchmark is not rerun here: the rerun follows these fixes under a new pre-registration, and
docs/benchmarks/is untouched.Changes
fix(dependency-delta)dependency-deltadevDependenciesworkspace link read as loosening apeerDependenciesrange (vitejs/vitepackages/plugin-legacy/package.json)fix(vacuous-tests)vacuous-tests,assertion-reductionexpect(..)receiver was an assertion, so tests made ofexpect.poll(fn).toBe(..)were vacuous (vitejs/vite playground specs)expector itspoll/softmember, also split across lines. A poll with no matcher, or.pollon another object, is not an assertionfix(stub-bodies)stub-bodies,error-swallowingresources/test/fixtures/was reported (astral-sh/ruff)fixturesdirectory inside atest,testsor__tests__directory is a test's input and stays test scope, for both gates that share the rule. Afixturesdirectory outside a test directory is not one, and a fixture that replaces a deleted production file is still judged as production codefix(pii)pii,shell-secretsBearer deadbeef12346docstring exampleFixture classification: no other gate reads
fixturesas its own scope;piiexempts only declared test scope andgoldenguardstests/fixtures/**/output*as committed output. The fixture rule is therefore placed in the one shared test-scope rule thatstub-bodiesanderror-swallowingboth use, rather than in either gate.No real-leak bypass in
pii: outside test code nothing changes, a real-looking user name in test code is still an error, and a LAN address in test code is still reported (as a warning). The e2e test keeps positive controls for a LAN host, a home path with a real-looking user, a private key with its body, and a Bearer token, all in non-test code.Not addressed: a LAN address in a JSDoc example in production code (denoland/deno
ext/node/polyfills/dgram.ts) is still reported at the gate's severity.Evidence
Mutation evidence (RUN: each mutant applied, the named tests run and observed failing, then restored with
git checkout):guards::dependency::tests::a_dependency_is_compared_with_the_base_within_its_own_section,dependency_delta_compares_a_dependency_within_its_own_sectionEXPECT_RECEIVER_MEMBERSemptiedast::javascript::conditional_skip_tests::expect_poll_and_soft_receivers_are_assertions,vacuous_tests_read_expect_poll_and_soft_as_assertion_receivers.pollacceptedexpect_poll_and_soft_receivers_are_assertions(negative control)test_fixture_pathnot consultedstub_bodies_reads_a_test_fixture_input_as_test_scopeast::functions::tests::a_test_fixture_is_a_file_below_fixtures_inside_a_test_directorypii_reads_test_code_and_placeholder_secrets_without_opening_a_leakguards::token_formats::tests::a_private_key_header_alone_in_its_string_names_the_format,pii_reads_test_code_and_placeholder_secrets_without_opening_a_leakguards::token_formats::tests::a_bearer_value_made_of_a_hex_example_word_is_a_placeholder,pii_reads_test_code_and_placeholder_secrets_without_opening_a_leakpii_reads_test_code_and_placeholder_secrets_without_opening_a_leakguards::hygiene::pii::tests::test_code_is_a_test_path_or_the_lines_of_a_test,pii_reads_test_code_and_placeholder_secrets_without_opening_a_leakEach new e2e and unit test was also run before its fix and observed failing on the reproduction (RUN).
Spot check (RUN, not a benchmark rerun): the reproduction files fetched read-only from each finding's head commit, added to a scratch repository and checked with this branch's binary.
piireports nothing on the denolibs/core/error.rshome paths, the ruffoptions.rshome paths, the deno webcrypto PEM header constants or the fastapi Bearer docstring; the LAN addresses in the vite, deno and bench test code are warnings;dgram.tsstays an error.Gates (RUN):
cargo fmt --check,cargo clippy --all-targets -- -D warnings,discipline docs --check, anddiscipline check --policy-from basewith this body pass locally. The fullcargo test --no-fail-fastrun (per-worktreeCARGO_TARGET_DIR) was still running when this was opened; its result is posted as a comment.Behaviour Changes: one
unreleasedrow per change indocs/ROADMAP.md;docs/GATES.mdupdated forpiiand the test-scope rule.