Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ body:
attributes:
label: Agentmetry version
description: 'Run: python -c "import agentmetry; print(agentmetry.__version__)"'
placeholder: "0.12.1"
placeholder: "0.12.2"
validations:
required: true
- type: input
Expand Down
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,27 @@ separately (currently `1.2.0`) and changes additively.

## [Unreleased]

## [0.12.2] - 2026-11-16

A privacy release. Both fixes apply only with command logging or full-argument
capture turned on, which are off by default; with the defaults, arguments are
stored as hashes and nothing here changes. Detection rules are unchanged; the
ruleset fingerprint is still `15846a0915769d4a`. If you ran either capture
option on an earlier version, the trail may hold the values below as typed:
rotate those credentials. Do not rewrite the trail, since that breaks its chain.

### Security

- **Provider tokens are scrubbed by format.** The OpenAI pattern,
`sk-[A-Za-z0-9]{20,}`, stopped at the second hyphen, so OpenAI project and
service keys (`sk-proj-`, `sk-svcacct-`, `sk-admin-`) and Anthropic keys
(`sk-ant-`) typed into a command were stored as typed. So were fine-grained
GitHub tokens (`github_pat_`), AWS temporary keys (`ASIA` and the other STS
prefixes), and GitLab, npm, PyPI, Docker, Google, Slack app, Stripe,
SendGrid, Shopify, DigitalOcean, Linear, Doppler, Postman, Pulumi and
Databricks tokens. Each now has its own pattern and label, in both the hook
and the recorder, and a test fails if the two pattern lists ever differ.
Found in the detection pass, which teaches the rules the same formats.
- **Inline secret assignments are scrubbed from stored commands.** With
command logging or full-argument capture on (both off by default), the
scrubber missed a secret assigned to a prefixed name, because its pattern
Expand Down
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ These are small, self-contained, and testable. The best way in:

| Area | Where |
|------|-------|
| A new **detection rule** | `apps/orchestrator/core/audit/detection/rules.py`: a pure function over a session's events, plus tests in `tests/test_detection_engine.py` |
| A new **detection rule** | `apps/orchestrator/agentmetry/core/audit/detection/rules.py`: a pure function over a session's events, plus tests in `tests/test_detection_engine.py` |
| A new **DLP pattern** | `agentmetry/policies/dlp/manifest.yaml`: regex only, no Python; add a fires/doesn't-fire test |
| A **SIEM adapter** | `apps/orchestrator/core/audit/sinks.py` |
| A **SIEM adapter** | `apps/orchestrator/agentmetry/core/audit/sinks.py` |
| A **hook adapter** for a new IDE | [docs/agentmetry-external-ingest.md](docs/agentmetry-external-ingest.md) |
| A **Sigma rule** | [docs/integrations/sigma/](docs/integrations/sigma/) |

Expand Down Expand Up @@ -61,7 +61,7 @@ is rewritten. New commits after the steps above will not add Cursor as co-author
From `apps/orchestrator`:

```bash
python -m ruff check core api tests # lint
python -m ruff check agentmetry tests # lint
python -m pytest -q # tests
```

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ These are the numbers from commands, not from a slide.

That block is what `agentmetry benchmark` printed from the PyPI 0.9.4 package and from this tree. "Rules covered 13" counts rule ids named in corpus expectations. Twelve of those are published rules. The thirteenth is the experimental rule. Two published rules have no corpus case: `host-subagent-swarm-burst` and `off-hours-activity`. They are named in `apps/orchestrator/tools/generate_sigma_pack.py` so the Sigma pack still has them.

2491 tests collected (`pytest --collect-only -q` from `apps/orchestrator`). The
2523 tests collected (`pytest --collect-only -q` from `apps/orchestrator`). The
[0.12.1 release PR's CI run](https://github.com/blitzcrieg1/agentmetry/actions/runs/43610018759)
on Python 3.12 reported 2443 passed and 19 skipped on Linux and macOS,
2456 passed and 6 skipped on Windows, and 83% line coverage on Linux
Expand Down
26 changes: 25 additions & 1 deletion apps/orchestrator/agentmetry/core/audit/redaction.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
key-based argument redaction never sees. Scrub those before storage.

Keep the pattern list in sync with the inline mirror in
`scripts/agentmetry_ingest.py` (the standalone hook cannot import this module).
`agentmetry/hooks/ingest.py` (the standalone hook cannot import this module);
`tests/test_redaction.py` fails if the two lists differ.
"""

from __future__ import annotations
Expand Down Expand Up @@ -35,6 +36,29 @@
(re.compile(r"\bgh[pousr]_[A-Za-z0-9]{20,}\b"), "<redacted-gh-token>"),
(re.compile(r"\bxox[baprs]-[A-Za-z0-9-]{10,}\b"), "<redacted-slack-token>"),
(re.compile(r"\bhf_[A-Za-z0-9]{30,}\b"), "<redacted-hf-token>"),
# The provider formats the detection rules recognise (traits._ISSUERS),
# so a token the sensor can name is a token it does not store. Until
# 0.12.2 `sk-[A-Za-z0-9]{20,}` stopped at the second hyphen, which left
# OpenAI project keys (`sk-proj-`) and Anthropic keys (`sk-ant-`) stored
# as typed, along with every format below.
(re.compile(r"\bsk-(?:ant|proj|svcacct|admin)-[A-Za-z0-9_-]{20,}"), "<redacted-key>"),
(re.compile(r"\b(?:A3T[A-Z0-9]|ASIA|ABIA|ACCA)[A-Z2-7]{16}\b"), "<redacted-aws-key>"),
(re.compile(r"\bgithub_pat_[A-Za-z0-9_]{50,}"), "<redacted-gh-token>"),
(re.compile(r"\bglpat-[A-Za-z0-9_-]{20,}"), "<redacted-gitlab-token>"),
(re.compile(r"\bnpm_[A-Za-z0-9]{36,}"), "<redacted-npm-token>"),
(re.compile(r"\bpypi-[A-Za-z0-9_-]{50,}"), "<redacted-pypi-token>"),
(re.compile(r"\bdckr_pat_[A-Za-z0-9_-]{20,}"), "<redacted-docker-token>"),
(re.compile(r"\bAIza[\w-]{35}"), "<redacted-google-key>"),
(re.compile(r"\bxapp-\d-[A-Za-z0-9]+-\d+-[A-Za-z0-9]+"), "<redacted-slack-token>"),
(re.compile(r"\b(?:sk|rk)_(?:test|live|prod)_[A-Za-z0-9]{10,99}"), "<redacted-stripe-key>"),
(re.compile(r"\bSG\.[A-Za-z0-9=_.-]{66}"), "<redacted-sendgrid-key>"),
(re.compile(r"\bshpat_[a-fA-F0-9]{32}"), "<redacted-shopify-token>"),
(re.compile(r"\bdo[po]_v1_[a-f0-9]{64}"), "<redacted-digitalocean-token>"),
(re.compile(r"\blin_api_[A-Za-z0-9]{40}"), "<redacted-linear-key>"),
(re.compile(r"\bdp\.pt\.[A-Za-z0-9]{43}"), "<redacted-doppler-token>"),
(re.compile(r"\bPMAK-[a-fA-F0-9]{24}-[a-fA-F0-9]{34}"), "<redacted-postman-key>"),
(re.compile(r"\bpul-[a-f0-9]{40}"), "<redacted-pulumi-token>"),
(re.compile(r"\bdapi[a-f0-9]{32}(?:-\d)?"), "<redacted-databricks-token>"),
]


Expand Down
2 changes: 1 addition & 1 deletion apps/orchestrator/agentmetry/core/version.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,4 @@

from __future__ import annotations

__version__ = "0.12.1"
__version__ = "0.12.2"
23 changes: 23 additions & 0 deletions apps/orchestrator/agentmetry/hooks/ingest.py
Original file line number Diff line number Diff line change
Expand Up @@ -581,6 +581,29 @@ def redact_arguments(args: Any) -> dict[str, Any]:
(re.compile(r"\bgh[pousr]_[A-Za-z0-9]{20,}\b"), "<redacted-gh-token>"),
(re.compile(r"\bxox[baprs]-[A-Za-z0-9-]{10,}\b"), "<redacted-slack-token>"),
(re.compile(r"\bhf_[A-Za-z0-9]{30,}\b"), "<redacted-hf-token>"),
# The provider formats the detection rules recognise (traits._ISSUERS),
# so a token the sensor can name is a token it does not store. Until
# 0.12.2 `sk-[A-Za-z0-9]{20,}` stopped at the second hyphen, which left
# OpenAI project keys (`sk-proj-`) and Anthropic keys (`sk-ant-`) stored
# as typed, along with every format below.
(re.compile(r"\bsk-(?:ant|proj|svcacct|admin)-[A-Za-z0-9_-]{20,}"), "<redacted-key>"),
(re.compile(r"\b(?:A3T[A-Z0-9]|ASIA|ABIA|ACCA)[A-Z2-7]{16}\b"), "<redacted-aws-key>"),
(re.compile(r"\bgithub_pat_[A-Za-z0-9_]{50,}"), "<redacted-gh-token>"),
(re.compile(r"\bglpat-[A-Za-z0-9_-]{20,}"), "<redacted-gitlab-token>"),
(re.compile(r"\bnpm_[A-Za-z0-9]{36,}"), "<redacted-npm-token>"),
(re.compile(r"\bpypi-[A-Za-z0-9_-]{50,}"), "<redacted-pypi-token>"),
(re.compile(r"\bdckr_pat_[A-Za-z0-9_-]{20,}"), "<redacted-docker-token>"),
(re.compile(r"\bAIza[\w-]{35}"), "<redacted-google-key>"),
(re.compile(r"\bxapp-\d-[A-Za-z0-9]+-\d+-[A-Za-z0-9]+"), "<redacted-slack-token>"),
(re.compile(r"\b(?:sk|rk)_(?:test|live|prod)_[A-Za-z0-9]{10,99}"), "<redacted-stripe-key>"),
(re.compile(r"\bSG\.[A-Za-z0-9=_.-]{66}"), "<redacted-sendgrid-key>"),
(re.compile(r"\bshpat_[a-fA-F0-9]{32}"), "<redacted-shopify-token>"),
(re.compile(r"\bdo[po]_v1_[a-f0-9]{64}"), "<redacted-digitalocean-token>"),
(re.compile(r"\blin_api_[A-Za-z0-9]{40}"), "<redacted-linear-key>"),
(re.compile(r"\bdp\.pt\.[A-Za-z0-9]{43}"), "<redacted-doppler-token>"),
(re.compile(r"\bPMAK-[a-fA-F0-9]{24}-[a-fA-F0-9]{34}"), "<redacted-postman-key>"),
(re.compile(r"\bpul-[a-f0-9]{40}"), "<redacted-pulumi-token>"),
(re.compile(r"\bdapi[a-f0-9]{32}(?:-\d)?"), "<redacted-databricks-token>"),
]


Expand Down
80 changes: 80 additions & 0 deletions apps/orchestrator/tests/test_redaction.py
Original file line number Diff line number Diff line change
Expand Up @@ -192,3 +192,83 @@ def test_a_long_command_scrubs_quickly(scrub):
start = time.perf_counter()
scrub(chunk)
assert time.perf_counter() - start < 1.0


# --- every token format the detection rules name is scrubbed (0.12.2) -----------

import re # noqa: E402

from agentmetry.core.audit import redaction as _redaction # noqa: E402

# The detection rules' provider table arrives with the detection pass. The
# redaction patterns ship first (0.12.2), so the cross-checks against that
# table run only once it exists; the scrubbing tests run either way.
try:
from agentmetry.core.audit.detection.traits import _ISSUERS # noqa: E402
except ImportError: # before the detection pass
_ISSUERS = None

_A, _H = "aB3dE5gH7jK9mN1pQ3sT5vW7yZ9bC2dF4", "0123456789abcdef"
ISSUER_SAMPLES = {
"huggingface": ["hf_" + _A],
"github": ["ghp_" + _A + "xyz", "github_pat_" + (_A + "_") * 2],
"gitlab": ["glpat-" + _A[:20]],
"npm": ["npm_" + _A + "xyz"],
"pypi": ["pypi-" + _A * 2],
"openai": ["sk-proj-" + _A + "_x-" + _A[:6], "sk-svcacct-" + _A],
"anthropic": ["sk-ant-api03-" + _A],
"aws": ["ASIA" + "QWERTY234567ABCD", "AKIA" + "QWERTY234567ABCD"],
"docker": ["dckr_pat_" + _A[:27]],
"google": ["AIza" + _A + "_x"],
"slack": ["xoxb-1234567890-1234567890-" + _A[:24], "xapp-1-A0123BCD-4567-" + _H * 4],
"stripe": ["sk_live_" + _A[:24], "rk_test_" + _A[:24]],
"sendgrid": ["SG." + _A[:22] + "." + (_A + _A)[:43]],
"shopify": ["shpat_" + _H * 2],
"digitalocean": ["dop_v1_" + _H * 4],
"linear": ["lin_api_" + (_A * 2)[:40]],
"doppler": ["dp.pt." + (_A * 2)[:43]],
"postman": ["PMAK-" + (_H * 2)[:24] + "-" + (_H * 3)[:34]],
"pulumi": ["pul-" + (_H * 3)[:40]],
"databricks": ["dapi" + _H * 2],
}


def test_every_issuer_has_a_redaction_sample():
"""A provider added to the detection rules must be added here, and to redaction."""
if _ISSUERS is None:
pytest.skip("the detection rules' provider table is not in this version")
assert set(ISSUER_SAMPLES) == set(_ISSUERS)


@pytest.mark.parametrize(
"provider, token", [(p, t) for p, ts in ISSUER_SAMPLES.items() for t in ts]
)
def test_issuer_tokens_are_scrubbed_by_both_copies(provider, token):
if _ISSUERS is not None:
assert re.fullmatch(_ISSUERS[provider][1], token), (provider, token)
# Not a NAME=value assignment, so only the format patterns can catch it.
command = f"curl -s https://api.example.net/v1/check -d {token} | jq ."
for scrubbed in (scrub_secrets(command), ingest.scrub_command(command)):
assert token not in scrubbed, (provider, scrubbed)
assert "api.example.net/v1/check" in scrubbed


def test_hook_mirror_carries_the_same_patterns():
server = [(p.pattern, p.flags, r) for p, r in _redaction._SECRET_PATTERNS]
client = [(p.pattern, p.flags, r) for p, r in ingest._SECRET_PATTERNS]
assert server == client


@pytest.mark.parametrize(
"command",
[
"git log --oneline -3 && pytest -q",
"sha256sum dist/agentmetry-0.13.0.tar.gz",
"git show 3f9c2a1d8e7b6a5f4e3d2c1b0a9f8e7d6c5b4a39",
"python -m pip install skl2onnx sk-video",
"aws s3 ls s3://bucket/path --region eu-west-1",
],
)
def test_ordinary_text_survives_scrubbing(command):
assert scrub_secrets(command) == command
assert ingest.scrub_command(command) == command
2 changes: 1 addition & 1 deletion funding.json
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
{
"guid": "agentmetry",
"name": "Agentmetry",
"description": "AI coding agents run shell commands, edit files and call third-party tools hundreds of times an hour, largely unattended. What they actually did is recorded nowhere by default. The agent transcript is a conversation rather than a record of effects, the editor keeps no durable log, and endpoint security sees only that one process spawned another, which is what a coding agent looks like when it is working correctly.\n\nAgentmetry records what an agent did at the tool boundary, correlates sequences into detections, and writes a hash-chained JSONL trail the operator owns, forwarding into whatever SIEM they already run. It is local-first: there is no vendor cloud, and no telemetry reaches the maintainer. Tool arguments are hashed inside the hook process, so plaintext never crosses the wire.\n\nVerifiable as of 2026-11-14: version 0.12.1 on PyPI, Apache-2.0, a detection benchmark that runs at zero misses and zero false positives, MITRE ATT&CK and ATLAS mappings, and five SIEM adapters. Case and test counts are deliberately not quoted here, because they move every time somebody contributes. The README prints the current ones and a test checks that it still prints them correctly.\n\nFunding would go to making the format underneath it common rather than ours: a standalone agent session record specification, mappings into Elastic Common Schema, OCSF, Splunk CIM and Google SecOps UDM, and a conformance suite so a third party can verify their own mapping without reading our Python. The success condition is other vendors adopting the fields, including ones who compete with us.",
"description": "AI coding agents run shell commands, edit files and call third-party tools hundreds of times an hour, largely unattended. What they actually did is recorded nowhere by default. The agent transcript is a conversation rather than a record of effects, the editor keeps no durable log, and endpoint security sees only that one process spawned another, which is what a coding agent looks like when it is working correctly.\n\nAgentmetry records what an agent did at the tool boundary, correlates sequences into detections, and writes a hash-chained JSONL trail the operator owns, forwarding into whatever SIEM they already run. It is local-first: there is no vendor cloud, and no telemetry reaches the maintainer. Tool arguments are hashed inside the hook process, so plaintext never crosses the wire.\n\nVerifiable as of 2026-11-16: version 0.12.2 on PyPI, Apache-2.0, a detection benchmark that runs at zero misses and zero false positives, MITRE ATT&CK and ATLAS mappings, and five SIEM adapters. Case and test counts are deliberately not quoted here, because they move every time somebody contributes. The README prints the current ones and a test checks that it still prints them correctly.\n\nFunding would go to making the format underneath it common rather than ours: a standalone agent session record specification, mappings into Elastic Common Schema, OCSF, Splunk CIM and Google SecOps UDM, and a conformance suite so a third party can verify their own mapping without reading our Python. The success condition is other vendors adopting the fields, including ones who compete with us.",
"webpageUrl": {
"url": "https://agentmetry.ai",
"wellKnown": "https://agentmetry.ai/.well-known/funding-manifest-urls"
Expand Down
Loading