Repository navigation
Detection pass, November 2026 (draft, do not merge before 2026-11-17) - #435
Merged
Merged
Conversation
Dogfood triage in weeks 10 to 12 was nearly all false positives, and most came from one mistake: the hook classified whatever text a tool carried as if it were a shell command. A Write of a project's .env earned T1552.001 from its path and completed credential-exfil at critical with a market-data curl 15 hours later; Codex's apply_patch carries the patch as its command, so a test script holding .ssh/id_ed25519 as a string read as the agent running it. Tool kind now decides what text can mean: a write or a listing earns nothing, a read earns path traits only, everything else is a command (#329 shapes 4 and 5). The rules' command-word view is empty for file tools. The other #329 shapes: `hf` counts only as a command, not as the first segment of hf-render/; Remove-Item and rm whose every target is on Env:, Variable:, Function: or Alias: delete nothing; a .env read piped straight into cut -d= -f1 or awk -F= '{print $1}' reads no value. The environment dump check also read `.env |` as `env |`. Codex's ~/.codex/auth.json is a credential path (#425). MCP tools that pass a command any way but a top-level string were recorded and classified as nothing (#433). context-mode, a plugin with about 34k stars, routes shell through ctx_execute {language, code} and ctx_batch {commands}. structured_command now reads argv, a program with its arguments, command lists, code (a non-shell body as that interpreter's program), nested inputs and JSON in strings, keeps prose and file content out, and marks the event partial at its bounds. The shapes follow dioptx/weft's guard engine. Archive creation, bucket uploads, scheduled tasks, cron installs and decoding are mapped to T1560.001, T1567.002, T1053.005, T1053.003 and T1140 instead of collapsing into T1059 (#55). session-tool-burst is high only when an autonomous or scheduled session produced it (#172). Across 90 dogfood sessions the busiest ten minutes had a median of 13 tool calls and a 95th percentile of 152; 27 crossed the threshold of 40, all interactive, all ordinary work. No count separates a busy agent from a campaign. The Sigma pack is regenerated with high and medium variants. ENV_FILE and PRIVATE_KEY_PATH lose their unanchored left prefix, which changed no answer and made an 8 KiB token cost about 3 s per hook call; the same input now takes about 40 ms. Benchmark 70 cases (33 attack, 37 benign), 0 missed, 0 false positives, with 14 new cases generated through the real classifiers. Moves the ruleset fingerprint by design; this does not merge before 2026-11-17. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…used Listing a key folder named the keys and opened none, and in week 11 `Get-ChildItem $env:USERPROFILE\.ssh` was the credential half of a critical dotfile-read-then-git-push whose other half pushed the user's own repo. ATT&CK and the vendor rules (Elastic's Private Key Searching Activity, SigmaHQ's Private Keys Reconnaissance) treat *searching* for keys as T1552.004, not one folder's listing. A command that only lists or stats, alone or piped into filters that narrow the listing, is now T1083 and earns no key half. Recursive listings and `find` stay searches; reading, copying or archiving a key is unchanged. The Hugging Face rule fired on order, not on where the secret went (#431). `hf auth login --token $HF_TOKEN`, a whoami call to huggingface.co with the token, and `source .env` before `hf upload` were each a critical. A command whose only credential evidence is provider tokens, sent nowhere but that provider's hosts or CLI, now earns no credential half; any other credential, an unknown token name or a foreign host and it still does. Literal token values are credential evidence, so a live hf_ token in a URL to a foreign host is credential-exfil instead of nothing. Loading .env into the shell no longer opens a credential sequence, and `curl -d @.env` and `git add .env`, which sent the file with no credential half, now count. Benchmark 74 cases (35 attack, 39 benign), 0 missed, 0 false positives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaying 69,999 recorded events through this branch and comparing with
the maintainer's dispositions found shapes no corpus case had:
- PowerShell `$env:API_KEY=''` clears a variable and read as a read of it;
an escaped `\$env:SECRET` inside a note is a literal dollar sign.
- `Get-ChildItem Env:X* | Remove-Item` unsets variables and read as both a
deletion and a dump of the environment.
- MCP arguments were credential evidence in any key not on a prose list,
so a desktop app's `wrap_up {recap}` mentioning .env earned T1552.001.
Only keys that name a location (path, file, dir, target, uri) count now.
- `Test-Path .env`, `[ -f .env ]` and `test -f` ask whether a file exists;
the quoted value of -m, --message, --note, --title and similar flags is
prose for the path rules (a $SECRET in one still expands and still counts).
- Reading ~/.ssh/config, known_hosts, authorized_keys or a .pub file is
not a private key read. The folder and every other file in it still are.
- `Out-String`, `Format-List` and similar narrow a listing and do not turn
it into a read.
- destructive-delete-burst counted deletions across a session that can run
for weeks: 52 over 425 hours fired as a burst. It now needs five within
ten minutes, the clock the other burst rules already have. Of six dogfood
sessions with five or more deletions, two still fire; the two the window
drops that were marked resolved are, by their notes, intended cleanup.
Replay on one snapshot, this branch against master: critical and high
detections fall from 65 to 28, and those marked false positive from 27 to
14. Of 34 detections marked resolved or risk_accepted, 32 still fire. One
new detection appears, at medium.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The benchmark corpus is written by us, so a clean score says little about attacks we did not think of. tools/score_atomic_red_team.py parses Red Canary's Atomic Red Team test definitions as data (nothing is executed) and asks whether the classifier labels each command for the techniques this sensor tags. Master labelled 37 of 231; this branch labels 104. What it found, and what changed: - find / -path '*/.aws/credentials' -exec cat: the masking that keeps `echo 'cat ~/.aws/credentials'` quiet also hid every quoted find pattern. Quoted -name/-path/-wholename/-regex arguments to find, locate and fd are search patterns and now reach the path rules. - Credential files the list lacked: .git-credentials, .vault-token, .pgpass, .my.cnf, terraform.tfstate, the Azure CLI token cache, OCI sessions, unattend.xml, Windows Microsoft\Credentials, /proc/*/environ and shell histories. CI configs were left out on purpose: every developer reads them. - `grep -ri password /` and its kin rooted at /, /home, /etc, /Users or ~ are credential hunting; the same search inside a project is not. - GnuPG keyrings and .ppk, .pfx and .p12 key containers are private keys. - scp, rsync and sftp to `user@host:`, and data piped into ssh, are network egress. The mapper had its own client and host lists, which had drifted from the trait classifier's (#40 again); it now uses the shared ones. - Cron jobs written straight into /etc/cron.d or /var/spool/cron (T1053.003), b64decode and interpreter base64 decoding (T1140), ditto, makecab and single-file compressors (T1560.001), interpreter archive libraries (T1560.002), and overwriting with shred, sdelete, cipher /w or dd over a device or system path, plus bucket deletion (T1485). Precision held: replaying the 69,999-event dogfood trail gives exactly the same detections as before these changes, none added and none removed, and the new patterns stay under 70 ms on 8 KiB hostile input. Benchmark 74 cases, 0 missed, 0 false positives. Atomic Red Team is Copyright (c) 2018 Red Canary, Inc., MIT License. The pinned commands in tests/test_detection_atomic_red_team.py carry that notice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… formats The dogfood trail is one person's machine. tools/score_agent_trajectories.py replays public coding-agent trajectories (SWE-Gym's OpenHands runs, MIT, and a sample of nebius/SWE-rebench-openhands-trajectories, CC-BY-4.0) through the classifier and the rules. They are agents fixing GitHub issues, so every detection is a false positive. On 491 SWE-Gym and 300 SWE-rebench trajectories (26,482 tool calls), the credential, exfiltration and staging rules raised nothing on master or on this branch. All of master's critical and high detections there came from session-tool-burst firing high on ordinary agent work, the #172 shape on someone else's agents; this branch raises one high on each set, a delete burst. The datasets carry no timestamps and calls are spaced one second apart, which inflates the burst rules, so those counts are an upper bound. What the trajectories found: test code written with `echo 'os.environ["AWS_ACCESS_KEY_ID"] = "testing"' > t.py` read as a credential read. The shell expands `$NAME` bare and in double quotes, never in single quotes, so variable reads now look outside single quotes unless the quoted text is inline code that runs (`python -c '...'`); and `environ["NAME"] = ...` is an assignment, not a read. Token formats with a distinctive prefix and an obvious issuer now come from gitleaks' rule set (MIT), rewritten for Python: Google API keys, Slack, Stripe, SendGrid, Shopify, DigitalOcean, Linear, Doppler, Postman, Pulumi and Databricks, with AWS temporary key prefixes and OpenAI service-account keys. Each is credential evidence and counts as use only when it goes to its own provider. Generic shapes such as Twilio's were left out. Precision held on the dogfood trail: the replay gives the same detections as before. Atomic Red Team stays at 104 of 231. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Credential read, then egress" meant anywhere in the session, and a session can stay open for weeks. On the dogfood trail one remote-staging detection paired a download on 3 September with a run on 1 October. Every pair the maintainer dispositioned as real closed within 165 minutes. The credential-then-egress, cloud API and git push rules, untrusted input then a risky action, and both remote-staging forms now need the second step within sequence_window_minutes of the first (240 by default, in the manifest; 0 restores session-wide pairing). Each later first step opens its own window, so a stale early read no longer hides a fresh one. Timestamps that do not parse count as inside. approval-denied-then-executed keeps no window: a denied action that runs later is a bypass at any distance. Replayed against the dispositions, nothing changes in count: 57 detections, 28 critical or high, 32 of 34 resolved or risk-accepted kept. The 28-day pair in session 716fbfb9 is gone; that session still fires, on a same-day pair from 10 October. Benchmark 0 missed, 0 false positives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… nothing Traced from session 716fbfb9 on the dogfood trail, where both criticals were already dispositioned false positive. A grep pattern was read as a path. `grep -oE "ALSO SEND ~/.ssh/id_rsa" "$f"` checks a built blog post for its own prompt-injection example and opens no key, but it was the credential half of a critical credential-exfil and a critical dotfile-read-then-git-push. mask_search_patterns blanks the pattern argument of grep, egrep, rg, ag, ack, git grep, findstr and Select-String before the path rules look. The files a search reads still count, -e and -f are followed the way GNU tools permute them, and a pattern holding a private key header is kept, because searching contents for one is key hunting. A fetch piped into grep was staging. The README fetched from GitHub raw and piped into `grep -n "^## "` was the first half of a critical remote-staging-then-execute, because any `|` anywhere in the command counted (#51). stages_artifact reads only the fetch's own pipeline: a pipe stages when it feeds something other than a filter that prints, and a redirect, an output flag or a bare wget writes a file. Output to /dev/null, stdout or $null saves nothing, and a download saved only under a data extension (svg, png, json, md and the like) is not a payload. That last pair of rules was needed because the pattern this replaces, STAGES_ARTIFACT, had been broken since 0.7.0: three of its five alternatives carried a literal backspace (0x08) where `\b` was meant, so `-o`, `-O`, `--output` and `wget -P` never matched. Fixing it alone would have turned a URL status check (`curl -o /dev/null`) and the AiSOC logo saved as an .svg into criticals. Running a downloaded file by name was caught throughout by the same-file check. Replayed against the dispositions: 56 detections, 27 critical or high (was 28), critical or high false positives 13 (was 14), 32 of 34 resolved or risk-accepted kept. Four corpus cases, benchmark 78 cases, 0 missed, 0 false positives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 15, 2026
Traced from the two credential pairs left in the replay, both already
dispositioned false positive, both four criticals between them.
The 0.8.0 release check on 19 September asked whether publish credentials
were set: `env | grep -qiE "TWINE_|PYPI_"` prints nothing, and `env | grep
-oiE "^(TWINE_[A-Z_]+|PYPI_[A-Z_]+)="` prints names. CREDENTIAL_ENV_DUMP
read both as dumping the environment. env_dump_view now blanks an env,
printenv or Get-ChildItem Env: stage whose next stage prints no value:
grep or rg with -q, -c or -l, grep -o with a pattern that ends at `=`,
cut -d= -f1, awk -F= '{print $1}', wc, Select-Object Name, `% { $_.Name }`,
Measure-Object. `env | grep TWINE` prints values and still counts.
A test run on 10 November cleared Agentmetry's own variables with
`Get-ChildItem Env:AGENTMETRY_* | ForEach-Object { Remove-Item -LiteralPath
('Env:' + $_.Name) }`, the per-item form of #329 shape 2. The state-removal
exemption now covers a ForEach-Object block that only removes the piped
item by itself, its name, key or path. A block reading `$_.Value` does not
qualify. Residual, kept: `Remove-Item "Env:$($_.Name)"` still counts,
because the masked view blanks double quotes and cannot tell it from
`"$($_.Value)"`.
Replayed against the dispositions: 54 detections, 25 critical or high
(was 27), critical or high false positives 11 (was 13), 32 of 34 resolved
or risk-accepted kept. Session 716fbfb9 still pairs, now on 2 October,
where `grep -c ... .env` counts lines in a .env file; left for a decision.
Four corpus cases, benchmark 82 cases, 0 missed, 0 false positives.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The detection rules learned twenty provider formats in this pass
(traits._ISSUERS) and redaction kept six. Worse than the count: the OpenAI
pattern, `sk-[A-Za-z0-9]{20,}`, stops at the second hyphen, so OpenAI
project keys (`sk-proj-...`) and Anthropic keys (`sk-ant-...`) typed into a
command were stored as typed whenever command capture was on. So were
fine-grained GitHub tokens, AWS temporary keys (ASIA and the other STS
prefixes), and the GitLab, npm, PyPI, Docker, Google, Stripe, SendGrid,
Shopify, DigitalOcean, Linear, Doppler, Postman, Pulumi and Databricks
formats. The NAME=value rule caught them only when assigned to a name.
Both copies, core/audit/redaction.py and the hook's inline mirror, carry
the same new patterns. The hook classifies before it scrubs (ingest.py,
classify at the trait step, scrub at storage), so the token rule (#431)
still sees the literal; only the stored copy loses it.
Tests: a real-shaped sample per provider, checked against the detection
pattern and scrubbed by both copies; a check that every issuer has a
sample, so a provider added to detection must be added here; exact
equality of the two pattern lists, which the old parity test only
approximated; and ordinary text (hashes, package names, s3 paths) left
alone. The docstring named a mirror file that does not exist; fixed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The redaction patterns shipped first, in 0.12.2 (#439), from this branch's a125b46. That release needed the provider-table cross-check to skip where the table does not exist yet, and its comments name 0.12.2 rather than 0.13.0. The same edits here keep the two copies of these files identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nmap, masscan, rustscan, naabu, zmap and unicornscan, and a `for` loop that opens /dev/tcp connections with its own variable as the port, map to T1046 Network Service Discovery. nuclei, nikto and wpscan map to T1595.002, and ffuf, gobuster, feroxbuster, dirb, dirsearch and wfuzz to T1595.003. Before this a coding agent sweeping an internal range read as "Unix Shell" like any other command, which is the #55 shape: an observable technique collapsing into T1059. The scanner has to be in program position with a real argument, so `grep nmap README.md`, `apt-get install nmap`, `nmap --version` and a commit message about nmap stay T1059. Egress still wins: a scan piped to curl is T1071.001. T1046 is Discovery, so scans now count toward the discovery-then-collect burst (medium); T1595 is Reconnaissance and feeds no rule. Corpus: one attack case (scans, then a config read, fires discovery-then-collect) and one benign case (installing and grepping for a scanner). Benchmark 84 cases, 0 missed, 0 false positives. Atomic Red Team, with T1046 and T1595 added to the scorer: 107 of 245 on this branch, 37 on master; T1046 3 of 13. The new patterns take 3 ms or less on 8 KiB crafted input. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`{"run_command": ...}`, `{"shellCommand": ...}`, `{"setup_script": ...}`,
`{"pre-cmd": ...}` and `{"stdin": ...}` reached the rules as an MCP call with
no command: only exact key names (`command`, `cmd`, `script` and the shell
names) were read. weft's guard engine (dioptx/weft, MIT) reads any key ending
in command, cmd or script, plus stdin; this follows it. Keys that end that way
but hold text or another language stay data: transcript, postscript,
javascript, typescript, argument keys such as `subcommand`, and the prose
keys.
Corpus: a `setup_script` that reads and sends a credential fires
credential-exfil; a transcript quoting that command, a JavaScript body and a
`run_command` running a linter fire nothing. Benchmark 86 cases, 0 missed,
0 false positives. 2,847 passed, 7 skipped. Dogfood replay: the same 54
(correlation id, rule, severity) detections as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bring the reviewed Ruff and dashboard dependency updates into the detection pass. Describe the unreleased behavior changes and distinguish MITRE label counts from detection coverage.
Match explicit Hugging Face destinations by exact hostname. Surface literal tokens in captured command input as a separate high-severity finding while preserving normal environment-based authentication. Require reported successful tool results for execution sequences where phase metadata is available. Count repeated discovery results once per invocation and retain explicit denial events as evidence for the denial rule. Add regression coverage for both changes and regenerate the Sigma rule. Refs #431
Include the repository and wheel preflight scanners. Preserve both sets of changelog entries and reconcile the expected pytest collection for the combined tree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Do not merge before the r/netsec post on 2026-11-17 (CLAUDE.md, "The detection pass"). This moves the ruleset fingerprint by design; merging restarts the dogfood clock.
Opened so CI runs the full suite on each slice.
In this PR
.envis not reading it, and anapply_patchbody is file content.hfonly as a command;Remove-Item/rmonEnv:,Variable:,Function:,Alias:is not a deletion;.envpiped intocut -d= -f1orawk -F= '{print $1}'reads no value. The env-dump check no longer reads.env |asenv |.grep,rg,ag,ack,git grep,findstr,Select-String); an environment listing into something that prints no value reads no secret; per-itemForEach-Object { Remove-Item ... }onEnv:is state removal.~/.codex/auth.jsonis a credential path.traits.issuer_bound), for 20 provider formats including the gitleaks set; loading.envinto the shell does not start the cloud-API sequence.commands,codewithlanguage, nested, JSON in strings), bounded and marked partial at the bounds. Covers context-mode'sctx_executeandctx_batch, and keys ending in command, cmd or script plusstdin, as weft's guard engine reads them./dev/tcpport loop are T1046; nuclei, nikto and wpscan T1595.002; ffuf, gobuster and the like T1595.003, in program position only.STAGES_ARTIFACTpattern carried literal backspace characters from 0.7.0, so-o,-O,--outputandwget -Pnever matched; replaced.session-tool-burstis high only for an autonomous or scheduled session, medium otherwise. Sigma pack regenerated with high and medium variants.sequence_window_minutes(240 by default, manifest setting; 0 restores session-wide pairing).sk-proj-,sk-ant-,github_pat_, AWS STS prefixes and others). Shipped separately as 0.12.2.ENV_FILEandPRIVATE_KEY_PATHanchored: an 8 KiB hostile token went from about 3 s to about 40 ms per hook call, same answers.Measured
tools/score_atomic_red_team.py): 107 of 245 commands labelled, against 37 on master (T1046 and T1595 added to the scorer).tools/score_agent_trajectories.py, 791 sessions): 0 credential, exfil or staging detections on master or branch.Still to do
Merge-day updates only: fingerprint in CLAUDE.md, the site and the skills; ATT&CK coverage recount; CHANGELOG;
agentmetry dogfood --start --restart; recorder checkout; release 0.13.0.🤖 Generated with Claude Code