Repository navigation
Anonymous, opt-out usage reporting for the slayer CLI and servers #484
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
172f2a8
Plan usage telemetry as an OpenSpec change (collection, controls, del…
ZmeiGorynych ccac2e6
Add failing usage-telemetry test suite (payload vocabulary, scope, co…
ZmeiGorynych b75f3a1
Implement usage telemetry: recorder, spool, sender, feature extractio…
ZmeiGorynych 001796f
Unwind every server on SIGTERM so its counts are spooled (design deci…
ZmeiGorynych 33622f5
Plan: the payload-vocabulary principle lives in telemetry.arc42.md, n…
ZmeiGorynych 9bf430c
Architecture: telemetry node, its arc42 principles, regenerated diagrams
ZmeiGorynych dd2c7d5
Fix three telemetry test bugs: seeding warm-up, broken-pipe reader de…
ZmeiGorynych d98f43f
Facade: SQL that fails to tokenize is a translation/syntax error, not…
ZmeiGorynych a66d83c
Flight sentinel test: escape the hostile literal so the query is vali…
ZmeiGorynych b18d97f
Tick docs and gate tasks
ZmeiGorynych 46c4746
Merge branch 'origin/main' into egor/dev-2035-usage-tracking
ZmeiGorynych 58332bf
Telemetry extras vocabulary covers the new trino extra
ZmeiGorynych 1b9fd11
Tasks: record the DEV-2035 issue-body update
ZmeiGorynych c6a2f17
Merge branch 'origin/main' into egor/dev-2035-usage-tracking
ZmeiGorynych 81d1711
Telemetry review fixes: atomic stale-lease takeover, fail-silent send…
ZmeiGorynych 016b11f
The slayer telemetry commands are not counted
ZmeiGorynych 0f4f431
Merge branch 'origin/main' into egor/dev-2035-usage-tracking
ZmeiGorynych e2c126d
A dry run is not counted as an executed query
ZmeiGorynych 1484a2c
Rename telemetry to usage reporting
ZmeiGorynych 5a7baea
Tasks: tick 5.2 (README and CLI reference done; release notes are dra…
ZmeiGorynych 6730acb
Ship the PostHog EU project token
ZmeiGorynych ae2896f
Usage reports: person profiles, 11-month install IDs, 12-month retent…
ZmeiGorynych 63b1b82
Usage-reporting config writes under one cross-process lock; retention…
ZmeiGorynych b499e96
Merge branch 'origin/main' into egor/dev-2035-usage-tracking
ZmeiGorynych 0e3771f
Retention workflow: checkout without persisted credentials
ZmeiGorynych e962ead
Usage-reporting docs: exact retention rule, install ID as a PostHog p…
ZmeiGorynych ec28a83
Archive the dev-2035-usage-tracking OpenSpec change
ZmeiGorynych File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| name: Usage-reporting retention | ||
|
|
||
| # Weekly: delete PostHog usage reports older than the retention window (see docs/reference/usage-reporting.md). | ||
| # Manual run with an install ID: delete that install's reports now (erasure request). | ||
| on: | ||
| schedule: | ||
| - cron: "17 4 * * 1" | ||
| workflow_dispatch: | ||
| inputs: | ||
| install_id: | ||
| description: "Install ID to delete now (leave empty for the regular retention sweep)" | ||
| required: false | ||
| default: "" | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| retention: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| persist-credentials: false | ||
| - uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: "3.12" | ||
| - name: Delete expired or requested usage reports | ||
| env: | ||
| POSTHOG_PERSONAL_API_KEY: ${{ secrets.POSTHOG_PERSONAL_API_KEY }} | ||
| INSTALL_ID: ${{ inputs.install_id }} | ||
| run: python scripts/usage_reporting_retention.py |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| # usage_reporting — usage reporting | ||
|
|
||
| ## 1. Purpose & context | ||
|
|
||
| `slayer/usage_reporting` records anonymous usage counters in processes started by the | ||
| `slayer` CLI, spools them per process, and sends them to PostHog EU at most once | ||
| per 24 h. Behaviour is specified in `openspec/specs/usage-reporting/`. | ||
|
|
||
| ## 2. Building blocks | ||
|
|
||
| Child `features` turns the queries and datasources a surface has already loaded | ||
| into vocabulary tokens, and is the only part that imports other SLayer nodes | ||
| (`core`, `sql`). The rest (settings, payload vocabulary, recorder, spool, sender) | ||
| imports no SLayer node. | ||
|
|
||
| ## 3. Principles | ||
|
|
||
| 1. **Fail-silent**: no usage-reporting path raises into, writes to stdout of, or blocks | ||
| the host process beyond the one bounded send wait. | ||
| [enforced: test:tests/test_usage_reporting_isolation.py] | ||
| 2. **Usage reports carry no user content**: every value a usage report sends is | ||
| drawn from a SLayer-defined vocabulary (versions, enums, booleans, buckets, | ||
| counts keyed by built-in names); anything else is reported as `other`. | ||
| Library use never reports. | ||
| [enforced: test:tests/test_usage_reporting_payload.py] | ||
| 3. **No user-derived values**: no value supplied by a user, an MCP client or a | ||
| database leaves the process or reaches a usage-reporting file in any form: verbatim, | ||
| truncated, case-folded, hashed, encoded, or as its length. Such a value may | ||
| only pick a token from a fixed table; a miss is reported as `other` / `custom`. | ||
| The one exception is an MCP client's self-reported major version, bounded to | ||
| 0–999. | ||
| [enforced: test:tests/test_usage_reporting_payload.py] | ||
| 4. **Errors by class only**: a failure is reduced to a token by its exception | ||
| class hierarchy; usage reporting never reads an exception's message, arguments, | ||
| traceback or `repr`. | ||
| [enforced: test:tests/test_usage_reporting_isolation.py] | ||
| 5. **Sizes are bucketed**: a count of user-owned things (models, datasources) is | ||
| reported only as a bucket, never as the exact number. | ||
| [enforced: test:tests/test_usage_reporting_features.py] | ||
|
|
||
| ## 4. Rationale | ||
|
|
||
| Usage reporting runs inside every CLI process, so its own defects must never reach the | ||
| user; principles 2–5 keep user content, and anything derived from it, from | ||
| leaving the machine. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| # Usage reporting | ||
|
|
||
| SLayer collects **anonymous, aggregated usage statistics** to learn which surfaces, databases and | ||
| query features people use. It never sends your queries, data, model / column / datasource names, | ||
| memory text, arguments, error messages, file paths or credentials. | ||
|
|
||
| ## When it is active | ||
|
|
||
| Only in processes started by the `slayer` command (`slayer query`, `slayer mcp`, `slayer serve`, | ||
| `slayer flight-serve`, `slayer pg-serve`, …, and the Docker image, which runs `slayer serve`). | ||
| Using SLayer as a Python library — the query engine, the client, or an app or MCP server you | ||
| build in your own code — never records, prints a notice or sends anything. | ||
|
|
||
| ## Turning it off | ||
|
|
||
| The first rule that matches decides: | ||
|
|
||
| | # | Rule | Effect | | ||
| |---|------|--------| | ||
| | 1 | `DO_NOT_TRACK=1` (or `true` / `yes` / `on`) | off | | ||
| | 2 | `SLAYER_USAGE_REPORTING=on` / `SLAYER_USAGE_REPORTING=off` | on / off | | ||
| | 3 | `CI` is set; SLayer is an editable install; the config directory cannot be written | off | | ||
| | 4 | `slayer usage-reporting enable` / `slayer usage-reporting disable` | on / off (persisted) | | ||
| | 5 | Default | on | | ||
|
|
||
| ```bash | ||
| slayer usage-reporting status # on or off, which rule decided it, the install ID | ||
| slayer usage-reporting show # print exactly the report the next send would contain (sends nothing) | ||
| slayer usage-reporting disable # turn off for good; also deletes the install ID and unsent data | ||
| slayer usage-reporting enable | ||
| ``` | ||
|
|
||
| ## The notice | ||
|
|
||
| The first time usage reporting is on, SLayer prints a one-line notice on **stderr** (never stdout, so | ||
| stdio MCP stays clean) and records that it did. It is shown again only if the report format | ||
| changes in a major way. | ||
|
|
||
| ## What a report contains | ||
|
|
||
| One JSON document per send; every value is a version, a SLayer-defined token, a boolean, a | ||
| bucket, a date, a random ID or a count. Anything SLayer does not recognise is reported as | ||
| `other` (or `custom` for user-defined aggregations). | ||
|
|
||
| | Field | Content | | ||
| |-------|---------| | ||
| | `schema` | Report format version. | | ||
| | `install_id` | Random UUID created on first use, replaced every 11 months. | | ||
| | `batch_id` | Random UUID per send (lets the backend drop duplicates). | | ||
| | `period` | First and last **date** covered (no times). | | ||
| | `env` | SLayer version, Python `major.minor`, OS family, CPU architecture, whether it runs in a container, installed SLayer extras (read from package metadata). | | ||
| | `usage` | Per action — CLI command (except `slayer usage-reporting …`), MCP tool name, REST route template (e.g. `/models/{name}`), Flight SQL / PostgreSQL query or probe — the count of successes and of failures per error kind (a fixed list such as `value_error`, `schema_drift`, `http_4xx`). | | ||
| | `features` | Per query: counts of multistage queries, inline source models, time dimensions, filters, computed dimensions, saved-query runs, non-default `to_many_handling`; counts of each built-in aggregation and transform (`sum`, `time_shift`, …), user-defined ones as `custom`. | | ||
| | `dialects` | Queries per SQL dialect; datasources per dialect in buckets `0` / `1` / `2-5` / `6+`. | | ||
| | `context` | Per process: storage backend (`yaml` / `sqlite` / `other`), model count in buckets `0` / `1-10` / `11-50` / `51-200` / `200+`, whether the bundled demo database was queried. | | ||
| | `mcp_clients` | MCP sessions per client (Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Codex, Zed, Cline, Goose, Continue; anything else `other`) with its major version. | | ||
|
|
||
| ## How and when it is sent | ||
|
|
||
| Each process keeps its counts in memory and writes them to its own spool file when it ends — no | ||
| network at that point. A report is sent at most **once per 24 hours**, and only while SLayer is | ||
| running anyway (no heartbeat): when a `slayer` process starts, or a running server records an | ||
| action, and the last successful send was 24 hours ago or more. The send merges all spool files | ||
| and runs in a background thread; at exit SLayer waits for it at most about one second. Unsent | ||
| data stays in the spool (capped at about 100 files / 1 MB) for a later try. | ||
|
|
||
| Usage reporting never changes a command's output or exit code: every usage reporting failure is ignored. | ||
|
|
||
| Files live in the platform config directory, not in SLayer's model storage: | ||
| `$XDG_CONFIG_HOME/slayer` (default `~/.config/slayer`) on Linux, | ||
| `~/Library/Application Support/slayer` on macOS, `%APPDATA%\slayer` on Windows — | ||
| `usage-reporting.json` (setting, install ID, notice and send bookkeeping), `usage-reporting-spool/` | ||
| and two small lock files that keep concurrent `slayer` processes from overwriting each other. | ||
|
|
||
| ## Destination | ||
|
|
||
| Reports go to the PostHog EU capture API (`https://eu.i.posthog.com/i/v0/e/`) as one | ||
| `slayer_usage` event per send, keyed by the install ID, which PostHog keeps as one "person" so that | ||
| all of an install's reports can be deleted together. Your IP address is not in the report; | ||
| PostHog receives it at the network level, our project is configured to discard it, and every | ||
| event disables GeoIP lookup. `SLAYER_USAGE_REPORTING_ENDPOINT` overrides the URL. | ||
|
|
||
| ## Privacy notice | ||
|
|
||
| - **Controller**: Motley AI, the maintainer of SLayer. | ||
| - **Purpose**: understanding how SLayer is used, to prioritise its development. | ||
| - **Processor**: PostHog (EU hosting), under a data processing agreement. | ||
| - **Retention**: raw reports (keyed by install ID) are deleted within 12 months: an install ID is used | ||
| for at most 11 months, and a weekly job deletes it, with all its reports, once its first report is | ||
| 11 months and one week old. Aggregates without install IDs are kept but not published; any future | ||
| summary suppresses every figure covering fewer than 10 installs. | ||
| - **Your rights**: `slayer usage-reporting status` prints your install ID; to have its data deleted, | ||
| email privacy@motley.ai with that ID, and its reports are deleted, normally within a few days. | ||
| `slayer usage-reporting disable` stops collection and deletes the local ID and unsent data. |
2 changes: 2 additions & 0 deletions
2
openspec/changes/archive/2026-10-17-dev-2035-usage-tracking/.openspec.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| schema: spec-driven | ||
| created: 2026-10-08 |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.