diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 0fd2359..13869d7 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -16,7 +16,7 @@ body: attributes: label: Agentmetry version description: 'Run: python -c "import agentmetry; print(agentmetry.__version__)"' - placeholder: "0.12.1" + placeholder: "0.12.2" validations: required: true - type: input diff --git a/CHANGELOG.md b/CHANGELOG.md index a965931..3ff0242 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,8 +9,27 @@ separately (currently `1.2.0`) and changes additively. ## [Unreleased] +## [0.12.2] - 2026-11-16 + +A privacy release. Both fixes apply only with command logging or full-argument +capture turned on, which are off by default; with the defaults, arguments are +stored as hashes and nothing here changes. Detection rules are unchanged; the +ruleset fingerprint is still `15846a0915769d4a`. If you ran either capture +option on an earlier version, the trail may hold the values below as typed: +rotate those credentials. Do not rewrite the trail, since that breaks its chain. + ### Security +- **Provider tokens are scrubbed by format.** The OpenAI pattern, + `sk-[A-Za-z0-9]{20,}`, stopped at the second hyphen, so OpenAI project and + service keys (`sk-proj-`, `sk-svcacct-`, `sk-admin-`) and Anthropic keys + (`sk-ant-`) typed into a command were stored as typed. So were fine-grained + GitHub tokens (`github_pat_`), AWS temporary keys (`ASIA` and the other STS + prefixes), and GitLab, npm, PyPI, Docker, Google, Slack app, Stripe, + SendGrid, Shopify, DigitalOcean, Linear, Doppler, Postman, Pulumi and + Databricks tokens. Each now has its own pattern and label, in both the hook + and the recorder, and a test fails if the two pattern lists ever differ. + Found in the detection pass, which teaches the rules the same formats. - **Inline secret assignments are scrubbed from stored commands.** With command logging or full-argument capture on (both off by default), the scrubber missed a secret assigned to a prefixed name, because its pattern diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4422c5e..0a37315 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,9 +10,9 @@ These are small, self-contained, and testable. The best way in: | Area | Where | |------|-------| -| A new **detection rule** | `apps/orchestrator/core/audit/detection/rules.py`: a pure function over a session's events, plus tests in `tests/test_detection_engine.py` | +| A new **detection rule** | `apps/orchestrator/agentmetry/core/audit/detection/rules.py`: a pure function over a session's events, plus tests in `tests/test_detection_engine.py` | | A new **DLP pattern** | `agentmetry/policies/dlp/manifest.yaml`: regex only, no Python; add a fires/doesn't-fire test | -| A **SIEM adapter** | `apps/orchestrator/core/audit/sinks.py` | +| A **SIEM adapter** | `apps/orchestrator/agentmetry/core/audit/sinks.py` | | A **hook adapter** for a new IDE | [docs/agentmetry-external-ingest.md](docs/agentmetry-external-ingest.md) | | A **Sigma rule** | [docs/integrations/sigma/](docs/integrations/sigma/) | @@ -61,7 +61,7 @@ is rewritten. New commits after the steps above will not add Cursor as co-author From `apps/orchestrator`: ```bash -python -m ruff check core api tests # lint +python -m ruff check agentmetry tests # lint python -m pytest -q # tests ``` diff --git a/README.md b/README.md index 2e8c994..cf63255 100644 --- a/README.md +++ b/README.md @@ -156,7 +156,7 @@ These are the numbers from commands, not from a slide. That block is what `agentmetry benchmark` printed from the PyPI 0.9.4 package and from this tree. "Rules covered 13" counts rule ids named in corpus expectations. Twelve of those are published rules. The thirteenth is the experimental rule. Two published rules have no corpus case: `host-subagent-swarm-burst` and `off-hours-activity`. They are named in `apps/orchestrator/tools/generate_sigma_pack.py` so the Sigma pack still has them. -2491 tests collected (`pytest --collect-only -q` from `apps/orchestrator`). The +2523 tests collected (`pytest --collect-only -q` from `apps/orchestrator`). The [0.12.1 release PR's CI run](https://github.com/blitzcrieg1/agentmetry/actions/runs/43610018759) on Python 3.12 reported 2443 passed and 19 skipped on Linux and macOS, 2456 passed and 6 skipped on Windows, and 83% line coverage on Linux diff --git a/apps/orchestrator/agentmetry/core/audit/redaction.py b/apps/orchestrator/agentmetry/core/audit/redaction.py index b6588f3..e6fb236 100644 --- a/apps/orchestrator/agentmetry/core/audit/redaction.py +++ b/apps/orchestrator/agentmetry/core/audit/redaction.py @@ -5,7 +5,8 @@ key-based argument redaction never sees. Scrub those before storage. Keep the pattern list in sync with the inline mirror in -`scripts/agentmetry_ingest.py` (the standalone hook cannot import this module). +`agentmetry/hooks/ingest.py` (the standalone hook cannot import this module); +`tests/test_redaction.py` fails if the two lists differ. """ from __future__ import annotations @@ -35,6 +36,29 @@ (re.compile(r"\bgh[pousr]_[A-Za-z0-9]{20,}\b"), ""), (re.compile(r"\bxox[baprs]-[A-Za-z0-9-]{10,}\b"), ""), (re.compile(r"\bhf_[A-Za-z0-9]{30,}\b"), ""), + # The provider formats the detection rules recognise (traits._ISSUERS), + # so a token the sensor can name is a token it does not store. Until + # 0.12.2 `sk-[A-Za-z0-9]{20,}` stopped at the second hyphen, which left + # OpenAI project keys (`sk-proj-`) and Anthropic keys (`sk-ant-`) stored + # as typed, along with every format below. + (re.compile(r"\bsk-(?:ant|proj|svcacct|admin)-[A-Za-z0-9_-]{20,}"), ""), + (re.compile(r"\b(?:A3T[A-Z0-9]|ASIA|ABIA|ACCA)[A-Z2-7]{16}\b"), ""), + (re.compile(r"\bgithub_pat_[A-Za-z0-9_]{50,}"), ""), + (re.compile(r"\bglpat-[A-Za-z0-9_-]{20,}"), ""), + (re.compile(r"\bnpm_[A-Za-z0-9]{36,}"), ""), + (re.compile(r"\bpypi-[A-Za-z0-9_-]{50,}"), ""), + (re.compile(r"\bdckr_pat_[A-Za-z0-9_-]{20,}"), ""), + (re.compile(r"\bAIza[\w-]{35}"), ""), + (re.compile(r"\bxapp-\d-[A-Za-z0-9]+-\d+-[A-Za-z0-9]+"), ""), + (re.compile(r"\b(?:sk|rk)_(?:test|live|prod)_[A-Za-z0-9]{10,99}"), ""), + (re.compile(r"\bSG\.[A-Za-z0-9=_.-]{66}"), ""), + (re.compile(r"\bshpat_[a-fA-F0-9]{32}"), ""), + (re.compile(r"\bdo[po]_v1_[a-f0-9]{64}"), ""), + (re.compile(r"\blin_api_[A-Za-z0-9]{40}"), ""), + (re.compile(r"\bdp\.pt\.[A-Za-z0-9]{43}"), ""), + (re.compile(r"\bPMAK-[a-fA-F0-9]{24}-[a-fA-F0-9]{34}"), ""), + (re.compile(r"\bpul-[a-f0-9]{40}"), ""), + (re.compile(r"\bdapi[a-f0-9]{32}(?:-\d)?"), ""), ] diff --git a/apps/orchestrator/agentmetry/core/version.py b/apps/orchestrator/agentmetry/core/version.py index 355e556..44c35ce 100644 --- a/apps/orchestrator/agentmetry/core/version.py +++ b/apps/orchestrator/agentmetry/core/version.py @@ -10,4 +10,4 @@ from __future__ import annotations -__version__ = "0.12.1" +__version__ = "0.12.2" diff --git a/apps/orchestrator/agentmetry/hooks/ingest.py b/apps/orchestrator/agentmetry/hooks/ingest.py index 56e8750..a320f6f 100644 --- a/apps/orchestrator/agentmetry/hooks/ingest.py +++ b/apps/orchestrator/agentmetry/hooks/ingest.py @@ -581,6 +581,29 @@ def redact_arguments(args: Any) -> dict[str, Any]: (re.compile(r"\bgh[pousr]_[A-Za-z0-9]{20,}\b"), ""), (re.compile(r"\bxox[baprs]-[A-Za-z0-9-]{10,}\b"), ""), (re.compile(r"\bhf_[A-Za-z0-9]{30,}\b"), ""), + # The provider formats the detection rules recognise (traits._ISSUERS), + # so a token the sensor can name is a token it does not store. Until + # 0.12.2 `sk-[A-Za-z0-9]{20,}` stopped at the second hyphen, which left + # OpenAI project keys (`sk-proj-`) and Anthropic keys (`sk-ant-`) stored + # as typed, along with every format below. + (re.compile(r"\bsk-(?:ant|proj|svcacct|admin)-[A-Za-z0-9_-]{20,}"), ""), + (re.compile(r"\b(?:A3T[A-Z0-9]|ASIA|ABIA|ACCA)[A-Z2-7]{16}\b"), ""), + (re.compile(r"\bgithub_pat_[A-Za-z0-9_]{50,}"), ""), + (re.compile(r"\bglpat-[A-Za-z0-9_-]{20,}"), ""), + (re.compile(r"\bnpm_[A-Za-z0-9]{36,}"), ""), + (re.compile(r"\bpypi-[A-Za-z0-9_-]{50,}"), ""), + (re.compile(r"\bdckr_pat_[A-Za-z0-9_-]{20,}"), ""), + (re.compile(r"\bAIza[\w-]{35}"), ""), + (re.compile(r"\bxapp-\d-[A-Za-z0-9]+-\d+-[A-Za-z0-9]+"), ""), + (re.compile(r"\b(?:sk|rk)_(?:test|live|prod)_[A-Za-z0-9]{10,99}"), ""), + (re.compile(r"\bSG\.[A-Za-z0-9=_.-]{66}"), ""), + (re.compile(r"\bshpat_[a-fA-F0-9]{32}"), ""), + (re.compile(r"\bdo[po]_v1_[a-f0-9]{64}"), ""), + (re.compile(r"\blin_api_[A-Za-z0-9]{40}"), ""), + (re.compile(r"\bdp\.pt\.[A-Za-z0-9]{43}"), ""), + (re.compile(r"\bPMAK-[a-fA-F0-9]{24}-[a-fA-F0-9]{34}"), ""), + (re.compile(r"\bpul-[a-f0-9]{40}"), ""), + (re.compile(r"\bdapi[a-f0-9]{32}(?:-\d)?"), ""), ] diff --git a/apps/orchestrator/tests/test_redaction.py b/apps/orchestrator/tests/test_redaction.py index 31ec1ff..2e59ea3 100644 --- a/apps/orchestrator/tests/test_redaction.py +++ b/apps/orchestrator/tests/test_redaction.py @@ -192,3 +192,83 @@ def test_a_long_command_scrubs_quickly(scrub): start = time.perf_counter() scrub(chunk) assert time.perf_counter() - start < 1.0 + + +# --- every token format the detection rules name is scrubbed (0.12.2) ----------- + +import re # noqa: E402 + +from agentmetry.core.audit import redaction as _redaction # noqa: E402 + +# The detection rules' provider table arrives with the detection pass. The +# redaction patterns ship first (0.12.2), so the cross-checks against that +# table run only once it exists; the scrubbing tests run either way. +try: + from agentmetry.core.audit.detection.traits import _ISSUERS # noqa: E402 +except ImportError: # before the detection pass + _ISSUERS = None + +_A, _H = "aB3dE5gH7jK9mN1pQ3sT5vW7yZ9bC2dF4", "0123456789abcdef" +ISSUER_SAMPLES = { + "huggingface": ["hf_" + _A], + "github": ["ghp_" + _A + "xyz", "github_pat_" + (_A + "_") * 2], + "gitlab": ["glpat-" + _A[:20]], + "npm": ["npm_" + _A + "xyz"], + "pypi": ["pypi-" + _A * 2], + "openai": ["sk-proj-" + _A + "_x-" + _A[:6], "sk-svcacct-" + _A], + "anthropic": ["sk-ant-api03-" + _A], + "aws": ["ASIA" + "QWERTY234567ABCD", "AKIA" + "QWERTY234567ABCD"], + "docker": ["dckr_pat_" + _A[:27]], + "google": ["AIza" + _A + "_x"], + "slack": ["xoxb-1234567890-1234567890-" + _A[:24], "xapp-1-A0123BCD-4567-" + _H * 4], + "stripe": ["sk_live_" + _A[:24], "rk_test_" + _A[:24]], + "sendgrid": ["SG." + _A[:22] + "." + (_A + _A)[:43]], + "shopify": ["shpat_" + _H * 2], + "digitalocean": ["dop_v1_" + _H * 4], + "linear": ["lin_api_" + (_A * 2)[:40]], + "doppler": ["dp.pt." + (_A * 2)[:43]], + "postman": ["PMAK-" + (_H * 2)[:24] + "-" + (_H * 3)[:34]], + "pulumi": ["pul-" + (_H * 3)[:40]], + "databricks": ["dapi" + _H * 2], +} + + +def test_every_issuer_has_a_redaction_sample(): + """A provider added to the detection rules must be added here, and to redaction.""" + if _ISSUERS is None: + pytest.skip("the detection rules' provider table is not in this version") + assert set(ISSUER_SAMPLES) == set(_ISSUERS) + + +@pytest.mark.parametrize( + "provider, token", [(p, t) for p, ts in ISSUER_SAMPLES.items() for t in ts] +) +def test_issuer_tokens_are_scrubbed_by_both_copies(provider, token): + if _ISSUERS is not None: + assert re.fullmatch(_ISSUERS[provider][1], token), (provider, token) + # Not a NAME=value assignment, so only the format patterns can catch it. + command = f"curl -s https://api.example.net/v1/check -d {token} | jq ." + for scrubbed in (scrub_secrets(command), ingest.scrub_command(command)): + assert token not in scrubbed, (provider, scrubbed) + assert "api.example.net/v1/check" in scrubbed + + +def test_hook_mirror_carries_the_same_patterns(): + server = [(p.pattern, p.flags, r) for p, r in _redaction._SECRET_PATTERNS] + client = [(p.pattern, p.flags, r) for p, r in ingest._SECRET_PATTERNS] + assert server == client + + +@pytest.mark.parametrize( + "command", + [ + "git log --oneline -3 && pytest -q", + "sha256sum dist/agentmetry-0.13.0.tar.gz", + "git show 3f9c2a1d8e7b6a5f4e3d2c1b0a9f8e7d6c5b4a39", + "python -m pip install skl2onnx sk-video", + "aws s3 ls s3://bucket/path --region eu-west-1", + ], +) +def test_ordinary_text_survives_scrubbing(command): + assert scrub_secrets(command) == command + assert ingest.scrub_command(command) == command diff --git a/funding.json b/funding.json index 56e3cdc..e639b41 100644 --- a/funding.json +++ b/funding.json @@ -15,7 +15,7 @@ { "guid": "agentmetry", "name": "Agentmetry", - "description": "AI coding agents run shell commands, edit files and call third-party tools hundreds of times an hour, largely unattended. What they actually did is recorded nowhere by default. The agent transcript is a conversation rather than a record of effects, the editor keeps no durable log, and endpoint security sees only that one process spawned another, which is what a coding agent looks like when it is working correctly.\n\nAgentmetry records what an agent did at the tool boundary, correlates sequences into detections, and writes a hash-chained JSONL trail the operator owns, forwarding into whatever SIEM they already run. It is local-first: there is no vendor cloud, and no telemetry reaches the maintainer. Tool arguments are hashed inside the hook process, so plaintext never crosses the wire.\n\nVerifiable as of 2026-11-14: version 0.12.1 on PyPI, Apache-2.0, a detection benchmark that runs at zero misses and zero false positives, MITRE ATT&CK and ATLAS mappings, and five SIEM adapters. Case and test counts are deliberately not quoted here, because they move every time somebody contributes. The README prints the current ones and a test checks that it still prints them correctly.\n\nFunding would go to making the format underneath it common rather than ours: a standalone agent session record specification, mappings into Elastic Common Schema, OCSF, Splunk CIM and Google SecOps UDM, and a conformance suite so a third party can verify their own mapping without reading our Python. The success condition is other vendors adopting the fields, including ones who compete with us.", + "description": "AI coding agents run shell commands, edit files and call third-party tools hundreds of times an hour, largely unattended. What they actually did is recorded nowhere by default. The agent transcript is a conversation rather than a record of effects, the editor keeps no durable log, and endpoint security sees only that one process spawned another, which is what a coding agent looks like when it is working correctly.\n\nAgentmetry records what an agent did at the tool boundary, correlates sequences into detections, and writes a hash-chained JSONL trail the operator owns, forwarding into whatever SIEM they already run. It is local-first: there is no vendor cloud, and no telemetry reaches the maintainer. Tool arguments are hashed inside the hook process, so plaintext never crosses the wire.\n\nVerifiable as of 2026-11-16: version 0.12.2 on PyPI, Apache-2.0, a detection benchmark that runs at zero misses and zero false positives, MITRE ATT&CK and ATLAS mappings, and five SIEM adapters. Case and test counts are deliberately not quoted here, because they move every time somebody contributes. The README prints the current ones and a test checks that it still prints them correctly.\n\nFunding would go to making the format underneath it common rather than ours: a standalone agent session record specification, mappings into Elastic Common Schema, OCSF, Splunk CIM and Google SecOps UDM, and a conformance suite so a third party can verify their own mapping without reading our Python. The success condition is other vendors adopting the fields, including ones who compete with us.", "webpageUrl": { "url": "https://agentmetry.ai", "wellKnown": "https://agentmetry.ai/.well-known/funding-manifest-urls"