Skip to content

Detection pass, November 2026 (draft, do not merge before 2026-11-17) - #435

Merged
blitzcrieg1 merged 17 commits into
masterfrom
detection/pass-2026-11
Nov 18, 2026
Merged

blitzcrieg1 merged 17 commits into
masterfrom
detection/pass-2026-11

Conversation

@blitzcrieg1

@blitzcrieg1 blitzcrieg1 commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Draft. Do not merge before the r/netsec post on 2026-11-17 (CLAUDE.md, "The detection pass"). This moves the ruleset fingerprint by design; merging restarts the dogfood clock.

Opened so CI runs the full suite on each slice.

In this PR

Measured

  • Benchmark: 86 cases (41 attack, 45 benign), 0 missed, 0 false positives.
  • Atomic Red Team (tools/score_atomic_red_team.py): 107 of 245 commands labelled, against 37 on master (T1046 and T1595 added to the scorer).
  • Public coding-agent trajectories (tools/score_agent_trajectories.py, 791 sessions): 0 credential, exfil or staging detections on master or branch.
  • Replay of the maintainer's dogfood trail against its dispositions: done; figures stay private until the replay is a committed tool.

Still to do

Merge-day updates only: fingerprint in CLAUDE.md, the site and the skills; ATT&CK coverage recount; CHANGELOG; agentmetry dogfood --start --restart; recorder checkout; release 0.13.0.

🤖 Generated with Claude Code

blitzcrieg1 and others added 3 commits November 15, 2026 15:25
Dogfood triage in weeks 10 to 12 was nearly all false positives, and most
came from one mistake: the hook classified whatever text a tool carried as
if it were a shell command. A Write of a project's .env earned T1552.001
from its path and completed credential-exfil at critical with a market-data
curl 15 hours later; Codex's apply_patch carries the patch as its command,
so a test script holding .ssh/id_ed25519 as a string read as the agent
running it. Tool kind now decides what text can mean: a write or a listing
earns nothing, a read earns path traits only, everything else is a command
(#329 shapes 4 and 5). The rules' command-word view is empty for file tools.

The other #329 shapes: `hf` counts only as a command, not as the first
segment of hf-render/; Remove-Item and rm whose every target is on Env:,
Variable:, Function: or Alias: delete nothing; a .env read piped straight
into cut -d= -f1 or awk -F= '{print $1}' reads no value. The environment
dump check also read `.env |` as `env |`. Codex's ~/.codex/auth.json is a
credential path (#425).

MCP tools that pass a command any way but a top-level string were recorded
and classified as nothing (#433). context-mode, a plugin with about 34k
stars, routes shell through ctx_execute {language, code} and ctx_batch
{commands}. structured_command now reads argv, a program with its
arguments, command lists, code (a non-shell body as that interpreter's
program), nested inputs and JSON in strings, keeps prose and file content
out, and marks the event partial at its bounds. The shapes follow
dioptx/weft's guard engine.

Archive creation, bucket uploads, scheduled tasks, cron installs and
decoding are mapped to T1560.001, T1567.002, T1053.005, T1053.003 and T1140
instead of collapsing into T1059 (#55).

session-tool-burst is high only when an autonomous or scheduled session
produced it (#172). Across 90 dogfood sessions the busiest ten minutes had
a median of 13 tool calls and a 95th percentile of 152; 27 crossed the
threshold of 40, all interactive, all ordinary work. No count separates a
busy agent from a campaign. The Sigma pack is regenerated with high and
medium variants.

ENV_FILE and PRIVATE_KEY_PATH lose their unanchored left prefix, which
changed no answer and made an 8 KiB token cost about 3 s per hook call;
the same input now takes about 40 ms.

Benchmark 70 cases (33 attack, 37 benign), 0 missed, 0 false positives,
with 14 new cases generated through the real classifiers. Moves the ruleset
fingerprint by design; this does not merge before 2026-11-17.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…used

Listing a key folder named the keys and opened none, and in week 11
`Get-ChildItem $env:USERPROFILE\.ssh` was the credential half of a critical
dotfile-read-then-git-push whose other half pushed the user's own repo.
ATT&CK and the vendor rules (Elastic's Private Key Searching Activity,
SigmaHQ's Private Keys Reconnaissance) treat *searching* for keys as
T1552.004, not one folder's listing. A command that only lists or stats,
alone or piped into filters that narrow the listing, is now T1083 and earns
no key half. Recursive listings and `find` stay searches; reading, copying
or archiving a key is unchanged.

The Hugging Face rule fired on order, not on where the secret went (#431).
`hf auth login --token $HF_TOKEN`, a whoami call to huggingface.co with
the token, and `source .env` before `hf upload` were each a critical. A
command whose only credential evidence is provider tokens, sent nowhere but
that provider's hosts or CLI, now earns no credential half; any other
credential, an unknown token name or a foreign host and it still does.
Literal token values are credential evidence, so a live hf_ token in a URL
to a foreign host is credential-exfil instead of nothing. Loading .env into
the shell no longer opens a credential sequence, and `curl -d @.env` and
`git add .env`, which sent the file with no credential half, now count.

Benchmark 74 cases (35 attack, 39 benign), 0 missed, 0 false positives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaying 69,999 recorded events through this branch and comparing with
the maintainer's dispositions found shapes no corpus case had:

- PowerShell `$env:API_KEY=''` clears a variable and read as a read of it;
  an escaped `\$env:SECRET` inside a note is a literal dollar sign.
- `Get-ChildItem Env:X* | Remove-Item` unsets variables and read as both a
  deletion and a dump of the environment.
- MCP arguments were credential evidence in any key not on a prose list,
  so a desktop app's `wrap_up {recap}` mentioning .env earned T1552.001.
  Only keys that name a location (path, file, dir, target, uri) count now.
- `Test-Path .env`, `[ -f .env ]` and `test -f` ask whether a file exists;
  the quoted value of -m, --message, --note, --title and similar flags is
  prose for the path rules (a $SECRET in one still expands and still counts).
- Reading ~/.ssh/config, known_hosts, authorized_keys or a .pub file is
  not a private key read. The folder and every other file in it still are.
- `Out-String`, `Format-List` and similar narrow a listing and do not turn
  it into a read.
- destructive-delete-burst counted deletions across a session that can run
  for weeks: 52 over 425 hours fired as a burst. It now needs five within
  ten minutes, the clock the other burst rules already have. Of six dogfood
  sessions with five or more deletions, two still fire; the two the window
  drops that were marked resolved are, by their notes, intended cleanup.

Replay on one snapshot, this branch against master: critical and high
detections fall from 65 to 28, and those marked false positive from 27 to
14. Of 34 detections marked resolved or risk_accepted, 32 still fire. One
new detection appears, at medium.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
blitzcrieg1 and others added 4 commits November 15, 2026 17:03
The benchmark corpus is written by us, so a clean score says little about
attacks we did not think of. tools/score_atomic_red_team.py parses Red
Canary's Atomic Red Team test definitions as data (nothing is executed) and
asks whether the classifier labels each command for the techniques this
sensor tags. Master labelled 37 of 231; this branch labels 104.

What it found, and what changed:

- find / -path '*/.aws/credentials' -exec cat: the masking that keeps
  `echo 'cat ~/.aws/credentials'` quiet also hid every quoted find pattern.
  Quoted -name/-path/-wholename/-regex arguments to find, locate and fd are
  search patterns and now reach the path rules.
- Credential files the list lacked: .git-credentials, .vault-token, .pgpass,
  .my.cnf, terraform.tfstate, the Azure CLI token cache, OCI sessions,
  unattend.xml, Windows Microsoft\Credentials, /proc/*/environ and shell
  histories. CI configs were left out on purpose: every developer reads them.
- `grep -ri password /` and its kin rooted at /, /home, /etc, /Users or ~
  are credential hunting; the same search inside a project is not.
- GnuPG keyrings and .ppk, .pfx and .p12 key containers are private keys.
- scp, rsync and sftp to `user@host:`, and data piped into ssh, are network
  egress. The mapper had its own client and host lists, which had drifted
  from the trait classifier's (#40 again); it now uses the shared ones.
- Cron jobs written straight into /etc/cron.d or /var/spool/cron (T1053.003),
  b64decode and interpreter base64 decoding (T1140), ditto, makecab and
  single-file compressors (T1560.001), interpreter archive libraries
  (T1560.002), and overwriting with shred, sdelete, cipher /w or dd over a
  device or system path, plus bucket deletion (T1485).

Precision held: replaying the 69,999-event dogfood trail gives exactly the
same detections as before these changes, none added and none removed, and
the new patterns stay under 70 ms on 8 KiB hostile input. Benchmark 74
cases, 0 missed, 0 false positives.

Atomic Red Team is Copyright (c) 2018 Red Canary, Inc., MIT License. The
pinned commands in tests/test_detection_atomic_red_team.py carry that notice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… formats

The dogfood trail is one person's machine. tools/score_agent_trajectories.py
replays public coding-agent trajectories (SWE-Gym's OpenHands runs, MIT, and
a sample of nebius/SWE-rebench-openhands-trajectories, CC-BY-4.0) through the
classifier and the rules. They are agents fixing GitHub issues, so every
detection is a false positive.

On 491 SWE-Gym and 300 SWE-rebench trajectories (26,482 tool calls), the
credential, exfiltration and staging rules raised nothing on master or on
this branch. All of master's critical and high detections there came from
session-tool-burst firing high on ordinary agent work, the #172 shape on
someone else's agents; this branch raises one high on each set, a delete
burst. The datasets carry no timestamps and calls are spaced one second
apart, which inflates the burst rules, so those counts are an upper bound.

What the trajectories found: test code written with
`echo 'os.environ["AWS_ACCESS_KEY_ID"] = "testing"' > t.py` read as a
credential read. The shell expands `$NAME` bare and in double quotes, never
in single quotes, so variable reads now look outside single quotes unless
the quoted text is inline code that runs (`python -c '...'`); and
`environ["NAME"] = ...` is an assignment, not a read.

Token formats with a distinctive prefix and an obvious issuer now come from
gitleaks' rule set (MIT), rewritten for Python: Google API keys, Slack, Stripe,
SendGrid, Shopify, DigitalOcean, Linear, Doppler, Postman, Pulumi and
Databricks, with AWS temporary key prefixes and OpenAI service-account keys.
Each is credential evidence and counts as use only when it goes to its own
provider. Generic shapes such as Twilio's were left out.

Precision held on the dogfood trail: the replay gives the same detections
as before. Atomic Red Team stays at 104 of 231.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Credential read, then egress" meant anywhere in the session, and a session
can stay open for weeks. On the dogfood trail one remote-staging detection
paired a download on 3 September with a run on 1 October. Every pair the
maintainer dispositioned as real closed within 165 minutes.

The credential-then-egress, cloud API and git push rules, untrusted input
then a risky action, and both remote-staging forms now need the second step
within sequence_window_minutes of the first (240 by default, in the
manifest; 0 restores session-wide pairing). Each later first step opens its
own window, so a stale early read no longer hides a fresh one. Timestamps
that do not parse count as inside. approval-denied-then-executed keeps no
window: a denied action that runs later is a bypass at any distance.

Replayed against the dispositions, nothing changes in count: 57
detections, 28 critical or high, 32 of 34 resolved or risk-accepted kept.
The 28-day pair in session 716fbfb9 is gone; that session still fires, on
a same-day pair from 10 October. Benchmark 0 missed, 0 false positives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… nothing

Traced from session 716fbfb9 on the dogfood trail, where both criticals
were already dispositioned false positive.

A grep pattern was read as a path. `grep -oE "ALSO SEND ~/.ssh/id_rsa" "$f"`
checks a built blog post for its own prompt-injection example and opens no
key, but it was the credential half of a critical credential-exfil and a
critical dotfile-read-then-git-push. mask_search_patterns blanks the pattern
argument of grep, egrep, rg, ag, ack, git grep, findstr and Select-String
before the path rules look. The files a search reads still count, -e and -f
are followed the way GNU tools permute them, and a pattern holding a private
key header is kept, because searching contents for one is key hunting.

A fetch piped into grep was staging. The README fetched from GitHub raw and
piped into `grep -n "^## "` was the first half of a critical
remote-staging-then-execute, because any `|` anywhere in the command
counted (#51). stages_artifact reads only the fetch's own pipeline: a pipe
stages when it feeds something other than a filter that prints, and a
redirect, an output flag or a bare wget writes a file. Output to /dev/null,
stdout or $null saves nothing, and a download saved only under a data
extension (svg, png, json, md and the like) is not a payload.

That last pair of rules was needed because the pattern this replaces,
STAGES_ARTIFACT, had been broken since 0.7.0: three of its five
alternatives carried a literal backspace (0x08) where `\b` was meant, so
`-o`, `-O`, `--output` and `wget -P` never matched. Fixing it alone would
have turned a URL status check (`curl -o /dev/null`) and the AiSOC logo
saved as an .svg into criticals. Running a downloaded file by name was
caught throughout by the same-file check.

Replayed against the dispositions: 56 detections, 27 critical or high
(was 28), critical or high false positives 13 (was 14), 32 of 34 resolved
or risk-accepted kept. Four corpus cases, benchmark 78 cases, 0 missed,
0 false positives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
blitzcrieg1 and others added 10 commits November 15, 2026 20:14
Traced from the two credential pairs left in the replay, both already
dispositioned false positive, both four criticals between them.

The 0.8.0 release check on 19 September asked whether publish credentials
were set: `env | grep -qiE "TWINE_|PYPI_"` prints nothing, and `env | grep
-oiE "^(TWINE_[A-Z_]+|PYPI_[A-Z_]+)="` prints names. CREDENTIAL_ENV_DUMP
read both as dumping the environment. env_dump_view now blanks an env,
printenv or Get-ChildItem Env: stage whose next stage prints no value:
grep or rg with -q, -c or -l, grep -o with a pattern that ends at `=`,
cut -d= -f1, awk -F= '{print $1}', wc, Select-Object Name, `% { $_.Name }`,
Measure-Object. `env | grep TWINE` prints values and still counts.

A test run on 10 November cleared Agentmetry's own variables with
`Get-ChildItem Env:AGENTMETRY_* | ForEach-Object { Remove-Item -LiteralPath
('Env:' + $_.Name) }`, the per-item form of #329 shape 2. The state-removal
exemption now covers a ForEach-Object block that only removes the piped
item by itself, its name, key or path. A block reading `$_.Value` does not
qualify. Residual, kept: `Remove-Item "Env:$($_.Name)"` still counts,
because the masked view blanks double quotes and cannot tell it from
`"$($_.Value)"`.

Replayed against the dispositions: 54 detections, 25 critical or high
(was 27), critical or high false positives 11 (was 13), 32 of 34 resolved
or risk-accepted kept. Session 716fbfb9 still pairs, now on 2 October,
where `grep -c ... .env` counts lines in a .env file; left for a decision.
Four corpus cases, benchmark 82 cases, 0 missed, 0 false positives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The detection rules learned twenty provider formats in this pass
(traits._ISSUERS) and redaction kept six. Worse than the count: the OpenAI
pattern, `sk-[A-Za-z0-9]{20,}`, stops at the second hyphen, so OpenAI
project keys (`sk-proj-...`) and Anthropic keys (`sk-ant-...`) typed into a
command were stored as typed whenever command capture was on. So were
fine-grained GitHub tokens, AWS temporary keys (ASIA and the other STS
prefixes), and the GitLab, npm, PyPI, Docker, Google, Stripe, SendGrid,
Shopify, DigitalOcean, Linear, Doppler, Postman, Pulumi and Databricks
formats. The NAME=value rule caught them only when assigned to a name.

Both copies, core/audit/redaction.py and the hook's inline mirror, carry
the same new patterns. The hook classifies before it scrubs (ingest.py,
classify at the trait step, scrub at storage), so the token rule (#431)
still sees the literal; only the stored copy loses it.

Tests: a real-shaped sample per provider, checked against the detection
pattern and scrubbed by both copies; a check that every issuer has a
sample, so a provider added to detection must be added here; exact
equality of the two pattern lists, which the old parity test only
approximated; and ordinary text (hashes, package names, s3 paths) left
alone. The docstring named a mirror file that does not exist; fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The redaction patterns shipped first, in 0.12.2 (#439), from this branch's
a125b46. That release needed the provider-table cross-check to skip where
the table does not exist yet, and its comments name 0.12.2 rather than
0.13.0. The same edits here keep the two copies of these files identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nmap, masscan, rustscan, naabu, zmap and unicornscan, and a `for` loop that
opens /dev/tcp connections with its own variable as the port, map to T1046
Network Service Discovery. nuclei, nikto and wpscan map to T1595.002, and
ffuf, gobuster, feroxbuster, dirb, dirsearch and wfuzz to T1595.003. Before
this a coding agent sweeping an internal range read as "Unix Shell" like any
other command, which is the #55 shape: an observable technique collapsing
into T1059.

The scanner has to be in program position with a real argument, so `grep
nmap README.md`, `apt-get install nmap`, `nmap --version` and a commit
message about nmap stay T1059. Egress still wins: a scan piped to curl is
T1071.001. T1046 is Discovery, so scans now count toward the
discovery-then-collect burst (medium); T1595 is Reconnaissance and feeds no
rule.

Corpus: one attack case (scans, then a config read, fires
discovery-then-collect) and one benign case (installing and grepping for a
scanner). Benchmark 84 cases, 0 missed, 0 false positives. Atomic Red Team,
with T1046 and T1595 added to the scorer: 107 of 245 on this branch, 37 on
master; T1046 3 of 13. The new patterns take 3 ms or less on 8 KiB crafted
input.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in 0.12.2 (#439) and the CLA bot removal (#440). The pass had stopped
getting CI because #435 conflicted with master on the README's test count;
it now says 2844, what pytest collects on the merged tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`{"run_command": ...}`, `{"shellCommand": ...}`, `{"setup_script": ...}`,
`{"pre-cmd": ...}` and `{"stdin": ...}` reached the rules as an MCP call with
no command: only exact key names (`command`, `cmd`, `script` and the shell
names) were read. weft's guard engine (dioptx/weft, MIT) reads any key ending
in command, cmd or script, plus stdin; this follows it. Keys that end that way
but hold text or another language stay data: transcript, postscript,
javascript, typescript, argument keys such as `subcommand`, and the prose
keys.

Corpus: a `setup_script` that reads and sends a credential fires
credential-exfil; a transcript quoting that command, a JavaScript body and a
`run_command` running a linter fire nothing. Benchmark 86 cases, 0 missed,
0 false positives. 2,847 passed, 7 skipped. Dogfood replay: the same 54
(correlation id, rule, severity) detections as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bring the reviewed Ruff and dashboard dependency updates into the detection pass. Describe the unreleased behavior changes and distinguish MITRE label counts from detection coverage.
Match explicit Hugging Face destinations by exact hostname. Surface literal
tokens in captured command input as a separate high-severity finding while
preserving normal environment-based authentication.

Require reported successful tool results for execution sequences where phase
metadata is available. Count repeated discovery results once per invocation
and retain explicit denial events as evidence for the denial rule.

Add regression coverage for both changes and regenerate the Sigma rule.

Refs #431
Include the repository and wheel preflight scanners. Preserve both sets of changelog entries and reconcile the expected pytest collection for the combined tree.
@blitzcrieg1
blitzcrieg1 marked this pull request as ready for review November 18, 2026 13:44
@blitzcrieg1
blitzcrieg1 merged commit edbf509 into master Nov 18, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant