Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ body:
attributes:
label: Agentmetry version
description: 'Run: python -c "import agentmetry; print(agentmetry.__version__)"'
placeholder: "0.12.2"
placeholder: "0.13.0"
validations:
required: true
- type: input
Expand Down
13 changes: 10 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ separately (currently `1.2.0`) and changes additively.

## [Unreleased]

## [0.13.0]

This release improves how detection rules use successful tool outcomes and
adds offline preflight checks for repositories and Python wheels.

### Added

- **Repository execution preflight.** `agentmetry scan --preflight PATH`
Expand Down Expand Up @@ -68,8 +73,9 @@ separately (currently `1.2.0`) and changes additively.
executable (#51).
- Credential-path patterns avoid repeated scans of hostile long input tokens.

This pass changes the ruleset fingerprint and requires a new dogfood period when
it ships. The provider-token redaction fixes shipped separately in 0.12.2.
The ruleset fingerprint is `01e0a0c3339f6d80`. Start a new dogfood period
when the recorder adopts this ruleset. The provider-token redaction fixes
shipped separately in 0.12.2.

## [0.12.2] - 2026-11-16

Expand Down Expand Up @@ -3064,7 +3070,8 @@ tamper-evident JSONL trail you own.
- Agentmetry records the agents you wire in. It is not a CASB and does not see
unmanaged ChatGPT or an IDE with hooks disabled.

[Unreleased]: https://github.com/blitzcrieg1/agentmetry/compare/v0.9.9...HEAD
[Unreleased]: https://github.com/blitzcrieg1/agentmetry/compare/v0.13.0...HEAD
[0.13.0]: https://github.com/blitzcrieg1/agentmetry/compare/v0.12.2...v0.13.0
[0.9.9]: https://github.com/blitzcrieg1/agentmetry/compare/v0.9.8...v0.9.9
[0.9.8]: https://github.com/blitzcrieg1/agentmetry/compare/v0.9.7...v0.9.8
[0.9.7]: https://github.com/blitzcrieg1/agentmetry/compare/v0.9.6...v0.9.7
Expand Down
22 changes: 9 additions & 13 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,16 +58,13 @@ also scored RED, on one and three untriaged `session-tool-burst` detections, and
turned GREEN when those were triaged that day. The fingerprint did not move
during the run.

**Decided by the maintainer on 2026-11-15: the freeze ends after the r/netsec
post on 2026-11-17.** By then the gate reads 11 of 4. The trigger is outside
**The maintainer ended this freeze after confirming the r/netsec post.**
The trigger was outside
reports saying which rules matter: #431 (a Hugging Face forum critique of the
token rule), #425 (an outside test of copied agent logins), and #433, where
context-mode, a plugin with about 34k stars, routes shell commands through MCP
inputs the command detections cannot read. **Until then the list above stays
frozen**, because the post leans on the unchanged-ruleset line. Work may start
in a worktree before then; nothing merges to master, and the live recorder is
not updated, until the post is out. See "The detection pass" below for what
lands and how.
inputs the command detections could not read. The detection pass merged in
[#435](https://github.com/blitzcrieg1/agentmetry/pull/435) on 2026-11-18.

That flip is worth knowing about. **A week scored red on untriaged detections
can be rescued at any time**, because `assess` reads the disposition store
Expand Down Expand Up @@ -101,7 +98,7 @@ fingerprint has not moved before opening any PR that touches

```bash
apps/orchestrator/.venv/Scripts/python.exe -c "from agentmetry.core.audit.dogfood import ruleset_fingerprint; print(ruleset_fingerprint()[:16])"
# expect 15846a0915769d4a
# expect 01e0a0c3339f6d80 for the 0.13.0 source
```

Four of the five parked detection-precision bugs were fixed in 0.7.0 and are
Expand All @@ -128,11 +125,10 @@ dogfood trail through the new rules and compare against the dispositions, so
the false-positive drop on real traffic is measured rather than claimed. That
figure replaces the green-week streak as the number worth quoting.

On the day it merges: record the new fingerprint in place of the one above,
restart the clock with `agentmetry dogfood --start --restart`, update the
recorder checkout, and change everything that quotes `15846a0915769d4a` or the
streak (the site's `DOGFOOD` constant and copy, the README, the skills). Then
release it as its own minor version.
When rolling out 0.13.0, update the recorder and verify its active fingerprint.
Restart the clock with `agentmetry dogfood --start --restart` after the new
ruleset is running. Update the site and skills after publication. Historical
results for fingerprint `15846a0915769d4a` belong to the previous ruleset.

Files in `detection/` that are **not** frozen and are safe to edit:
`disposition.py`, `live.py`, `live_store.py`, `benchmark.py`, `yaml_rules.py`,
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,10 +158,10 @@ These are the numbers from commands, not from a slide.

Run `agentmetry benchmark` to check these expectations. The 14 rule IDs in the corpus include 13 published rules and the experimental rule. The Sigma generator also covers `host-subagent-swarm-burst` and `off-hours-activity`.

Expected collection: 2904 tests collected (`pytest --collect-only -q` from `apps/orchestrator`). The
[0.12.1 release PR's CI run](https://github.com/blitzcrieg1/agentmetry/actions/runs/43610018759)
on Python 3.12 reported 2443 passed and 19 skipped on Linux and macOS,
2456 passed and 6 skipped on Windows, and 83% line coverage on Linux
2904 tests collected (`pytest --collect-only -q` from `apps/orchestrator`). The
[detection pass CI run](https://github.com/blitzcrieg1/agentmetry/actions/runs/44312544675)
reported 2885 passed and 19 skipped on Linux, 2898 passed and 6 skipped on Windows,
and 84% line coverage on Linux
(`pytest -q --cov=agentmetry --cov-report=term`). CI fails the coverage job under 78%.

The corpus and the expectations live in [`apps/orchestrator/agentmetry/core/audit/detection/corpus/`](apps/orchestrator/agentmetry/core/audit/detection/corpus/).
Expand Down
2 changes: 1 addition & 1 deletion apps/orchestrator/agentmetry/core/version.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,4 @@

from __future__ import annotations

__version__ = "0.12.2"
__version__ = "0.13.0"
2 changes: 1 addition & 1 deletion funding.json
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
{
"guid": "agentmetry",
"name": "Agentmetry",
"description": "AI coding agents run shell commands, edit files and call third-party tools hundreds of times an hour, largely unattended. What they actually did is recorded nowhere by default. The agent transcript is a conversation rather than a record of effects, the editor keeps no durable log, and endpoint security sees only that one process spawned another, which is what a coding agent looks like when it is working correctly.\n\nAgentmetry records what an agent did at the tool boundary, correlates sequences into detections, and writes a hash-chained JSONL trail the operator owns, forwarding into whatever SIEM they already run. It is local-first: there is no vendor cloud, and no telemetry reaches the maintainer. Tool arguments are hashed inside the hook process, so plaintext never crosses the wire.\n\nVerifiable as of 2026-11-16: version 0.12.2 on PyPI, Apache-2.0, a detection benchmark that runs at zero misses and zero false positives, MITRE ATT&CK and ATLAS mappings, and five SIEM adapters. Case and test counts are deliberately not quoted here, because they move every time somebody contributes. The README prints the current ones and a test checks that it still prints them correctly.\n\nFunding would go to making the format underneath it common rather than ours: a standalone agent session record specification, mappings into Elastic Common Schema, OCSF, Splunk CIM and Google SecOps UDM, and a conformance suite so a third party can verify their own mapping without reading our Python. The success condition is other vendors adopting the fields, including ones who compete with us.",
"description": "AI coding agents run shell commands, edit files and call third-party tools hundreds of times an hour, largely unattended. What they actually did is recorded nowhere by default. The agent transcript is a conversation rather than a record of effects, the editor keeps no durable log, and endpoint security sees only that one process spawned another, which is what a coding agent looks like when it is working correctly.\n\nAgentmetry records what an agent did at the tool boundary, correlates sequences into detections, and writes a hash-chained JSONL trail the operator owns, forwarding into whatever SIEM they already run. It is local-first: there is no vendor cloud, and no telemetry reaches the maintainer. Tool arguments are hashed inside the hook process, so plaintext never crosses the wire.\n\nVerifiable as of 2026-11-18: version 0.13.0 in source, Apache-2.0, a detection benchmark that runs at zero misses and zero false positives, MITRE ATT&CK and ATLAS mappings, and five SIEM adapters. Case and test counts are deliberately not quoted here, because they move every time somebody contributes. The README prints the current ones and a test checks that it still prints them correctly.\n\nFunding would go to making the format underneath it common rather than ours: a standalone agent session record specification, mappings into Elastic Common Schema, OCSF, Splunk CIM and Google SecOps UDM, and a conformance suite so a third party can verify their own mapping without reading our Python. The success condition is other vendors adopting the fields, including ones who compete with us.",
"webpageUrl": {
"url": "https://agentmetry.ai",
"wellKnown": "https://agentmetry.ai/.well-known/funding-manifest-urls"
Expand Down
Loading