Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
172f2a8
Plan usage telemetry as an OpenSpec change (collection, controls, del…
ZmeiGorynych Oct 8, 2026
ccac2e6
Add failing usage-telemetry test suite (payload vocabulary, scope, co…
ZmeiGorynych Oct 8, 2026
b75f3a1
Implement usage telemetry: recorder, spool, sender, feature extractio…
ZmeiGorynych Oct 8, 2026
001796f
Unwind every server on SIGTERM so its counts are spooled (design deci…
ZmeiGorynych Oct 9, 2026
33622f5
Plan: the payload-vocabulary principle lives in telemetry.arc42.md, n…
ZmeiGorynych Oct 9, 2026
9bf430c
Architecture: telemetry node, its arc42 principles, regenerated diagrams
ZmeiGorynych Oct 9, 2026
dd2c7d5
Fix three telemetry test bugs: seeding warm-up, broken-pipe reader de…
ZmeiGorynych Oct 9, 2026
d98f43f
Facade: SQL that fails to tokenize is a translation/syntax error, not…
ZmeiGorynych Oct 9, 2026
a66d83c
Flight sentinel test: escape the hostile literal so the query is vali…
ZmeiGorynych Oct 9, 2026
b18d97f
Tick docs and gate tasks
ZmeiGorynych Oct 9, 2026
46c4746
Merge branch 'origin/main' into egor/dev-2035-usage-tracking
ZmeiGorynych Oct 9, 2026
58332bf
Telemetry extras vocabulary covers the new trino extra
ZmeiGorynych Oct 9, 2026
1b9fd11
Tasks: record the DEV-2035 issue-body update
ZmeiGorynych Oct 9, 2026
c6a2f17
Merge branch 'origin/main' into egor/dev-2035-usage-tracking
ZmeiGorynych Oct 12, 2026
81d1711
Telemetry review fixes: atomic stale-lease takeover, fail-silent send…
ZmeiGorynych Oct 12, 2026
016b11f
The slayer telemetry commands are not counted
ZmeiGorynych Oct 12, 2026
0f4f431
Merge branch 'origin/main' into egor/dev-2035-usage-tracking
ZmeiGorynych Oct 12, 2026
e2c126d
A dry run is not counted as an executed query
ZmeiGorynych Oct 12, 2026
1484a2c
Rename telemetry to usage reporting
ZmeiGorynych Oct 12, 2026
5a7baea
Tasks: tick 5.2 (README and CLI reference done; release notes are dra…
ZmeiGorynych Oct 12, 2026
6730acb
Ship the PostHog EU project token
ZmeiGorynych Oct 16, 2026
ae2896f
Usage reports: person profiles, 11-month install IDs, 12-month retent…
ZmeiGorynych Oct 16, 2026
63b1b82
Usage-reporting config writes under one cross-process lock; retention…
ZmeiGorynych Oct 16, 2026
b499e96
Merge branch 'origin/main' into egor/dev-2035-usage-tracking
ZmeiGorynych Oct 16, 2026
0e3771f
Retention workflow: checkout without persisted credentials
ZmeiGorynych Oct 16, 2026
e962ead
Usage-reporting docs: exact retention rule, install ID as a PostHog p…
ZmeiGorynych Oct 16, 2026
ec28a83
Archive the dev-2035-usage-tracking OpenSpec change
ZmeiGorynych Oct 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
738 changes: 5 additions & 733 deletions .basedpyright/baseline.json

Large diffs are not rendered by default.

32 changes: 32 additions & 0 deletions .github/workflows/usage-reporting-retention.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: Usage-reporting retention

# Weekly: delete PostHog usage reports older than the retention window (see docs/reference/usage-reporting.md).
# Manual run with an install ID: delete that install's reports now (erasure request).
on:
schedule:
- cron: "17 4 * * 1"
workflow_dispatch:
inputs:
install_id:
description: "Install ID to delete now (leave empty for the regular retention sweep)"
required: false
default: ""

permissions:
contents: read

jobs:
retention:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Delete expired or requested usage reports
env:
POSTHOG_PERSONAL_API_KEY: ${{ secrets.POSTHOG_PERSONAL_API_KEY }}
INSTALL_ID: ${{ inputs.install_id }}
run: python scripts/usage_reporting_retention.py
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,10 @@ Now SLayer MCP will be visible in Claude Code next time you start it. Make sure

Read more on how to get started with [MCP](https://docs.motley.ai/slayer/getting-started/mcp/), [CLI](https://docs.motley.ai/slayer/getting-started/cli/), [REST API](https://docs.motley.ai/slayer/getting-started/rest-api/), [Python](https://docs.motley.ai/slayer/getting-started/python/) in the docs.

## Usage reporting

The `slayer` command sends anonymous, aggregated usage statistics (command, tool and query-feature counts, versions) at most once a day — never queries, names, data or credentials; library use sends nothing. Opt out with `SLAYER_USAGE_REPORTING=off`, `DO_NOT_TRACK=1` or `slayer usage-reporting disable`; see [Usage reporting](https://docs.motley.ai/slayer/reference/usage-reporting/) for every field.

## License

MIT
6 changes: 5 additions & 1 deletion architecture/core.arc42.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ flowchart TD
search("Search & embeddings")
memories("Agent memories")
protocols("BI wire protocols")
subgraph usage_reporting["Usage reporting"]
usage_reporting__features["Features"]
end
surfaces("User-facing surfaces")
core__query -.-> engine__syntax
core__models -.-> sql__dialects
Expand All @@ -56,8 +59,9 @@ flowchart TD
sql --> core
storage --> core
surfaces --> core
usage_reporting__features --> core
classDef leaf fill:none;
class core__query,core__models,sql__dialects,sql__sql_predicate,sql__window_detect,engine__syntax,ir,storage__migrations,importers,search,memories,protocols,surfaces leaf;
class core__query,core__models,sql__dialects,sql__sql_predicate,sql__window_detect,engine__syntax,ir,storage__migrations,importers,search,memories,protocols,usage_reporting__features,surfaces leaf;
```
*Dashed arrows: legacy edges slated to die.*
<!-- /likec4:core_focus -->
Expand Down
12 changes: 12 additions & 0 deletions architecture/model.c4
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,14 @@ model {
specs ['facade']
}
}
usage_reporting = node 'Usage reporting' {
metadata {
package 'slayer.usage_reporting'
arc42 'architecture/usage_reporting.arc42.md'
specs ['usage-reporting']
}
features = node 'Features'
}
surfaces = bucket 'User-facing surfaces' {
metadata {
packages ['slayer.api', 'slayer.mcp', 'slayer.cli', 'slayer.client', 'slayer.inspect', 'slayer.demo', 'slayer.__main__']
Expand Down Expand Up @@ -146,5 +154,9 @@ model {
surfaces -> search
surfaces -> sql
surfaces -> storage
protocols -> usage_reporting
surfaces -> usage_reporting
usage_reporting.features -> core
usage_reporting.features -> sql
}
}
6 changes: 5 additions & 1 deletion architecture/sql.arc42.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ flowchart TD
ir["Intermediate representation"]
storage["Storage backends"]
importers("Importers")
subgraph usage_reporting["Usage reporting"]
usage_reporting__features["Features"]
end
surfaces("User-facing surfaces")
core__models -.-> sql__dialects
core__models -.-> sql__sql_predicate
Expand All @@ -44,8 +47,9 @@ flowchart TD
sql --> ir
storage --> sql
surfaces --> sql
usage_reporting__features --> sql
classDef leaf fill:none;
class core__query,core__models,sql__render,sql__dialects,sql__sql_predicate,sql__window_detect,engine,ir,storage,importers,surfaces leaf;
class core__query,core__models,sql__render,sql__dialects,sql__sql_predicate,sql__window_detect,engine,ir,storage,importers,usage_reporting__features,surfaces leaf;
```
*Dashed arrows: legacy edges slated to die.*
<!-- /likec4:sql_focus -->
Expand Down
9 changes: 7 additions & 2 deletions architecture/system.arc42.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ flowchart TD
search("Search & embeddings")
memories("Agent memories")
protocols("BI wire protocols")
usage_reporting["Usage reporting"]
surfaces("User-facing surfaces")
core -.-> engine
core -.-> sql
Expand Down Expand Up @@ -66,12 +67,16 @@ flowchart TD
surfaces --> search
surfaces --> sql
surfaces --> storage
protocols --> usage_reporting
surfaces --> usage_reporting
usage_reporting --> core
usage_reporting --> sql
```
*Dashed arrows: legacy edges slated to die.*
<!-- /likec4:landscape -->

Ten nodes: precise `core`, `sql`, `ir`,
`engine`, `storage` around the query pipeline; virtual buckets `importers`,
Eleven nodes: precise `core`, `sql`, `ir`,
`engine`, `storage` around the query pipeline, and `usage_reporting`; virtual buckets `importers`,
`search`, `memories`, `protocols`, `surfaces` for the rest. Package claims and
spec mapping are node metadata in [model.c4](model.c4); the
legacy-arrow baseline is in [index.yaml](index.yaml).
Expand Down
45 changes: 45 additions & 0 deletions architecture/usage_reporting.arc42.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# usage_reporting — usage reporting

## 1. Purpose & context

`slayer/usage_reporting` records anonymous usage counters in processes started by the
`slayer` CLI, spools them per process, and sends them to PostHog EU at most once
per 24 h. Behaviour is specified in `openspec/specs/usage-reporting/`.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

## 2. Building blocks

Child `features` turns the queries and datasources a surface has already loaded
into vocabulary tokens, and is the only part that imports other SLayer nodes
(`core`, `sql`). The rest (settings, payload vocabulary, recorder, spool, sender)
imports no SLayer node.

## 3. Principles

1. **Fail-silent**: no usage-reporting path raises into, writes to stdout of, or blocks
the host process beyond the one bounded send wait.
[enforced: test:tests/test_usage_reporting_isolation.py]
2. **Usage reports carry no user content**: every value a usage report sends is
drawn from a SLayer-defined vocabulary (versions, enums, booleans, buckets,
counts keyed by built-in names); anything else is reported as `other`.
Library use never reports.
[enforced: test:tests/test_usage_reporting_payload.py]
3. **No user-derived values**: no value supplied by a user, an MCP client or a
database leaves the process or reaches a usage-reporting file in any form: verbatim,
truncated, case-folded, hashed, encoded, or as its length. Such a value may
only pick a token from a fixed table; a miss is reported as `other` / `custom`.
The one exception is an MCP client's self-reported major version, bounded to
0–999.
[enforced: test:tests/test_usage_reporting_payload.py]
4. **Errors by class only**: a failure is reduced to a token by its exception
class hierarchy; usage reporting never reads an exception's message, arguments,
traceback or `repr`.
[enforced: test:tests/test_usage_reporting_isolation.py]
5. **Sizes are bucketed**: a count of user-owned things (models, datasources) is
reported only as a bucket, never as the exact number.
[enforced: test:tests/test_usage_reporting_features.py]

## 4. Rationale

Usage reporting runs inside every CLI process, so its own defects must never reach the
user; principles 2–5 keep user content, and anything derived from it, from
leaving the machine.
11 changes: 11 additions & 0 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -467,3 +467,14 @@ slayer memory forget kb.brooklyn-pos
| `save` | `--query JSON_OR_@FILE` | Inline SLayer query (or `@path.json`). Entities are auto-extracted and the query is persisted on the memory. Mutually exclusive with `--entities`. |
| `save` | `--id ID` | User-pinned canonical memory id. Forbidden charset: `:`, `/`, `?`, `#`, whitespace, ASCII control. Omit to auto-allocate (`max(int-shaped id) + 1`). Duplicate id → unconditional upsert; `created_at` preserved. |
| `forget` | `<id>` (positional) | Memory id. Cascade-strips every `memory:<id>` reference to it from every other memory's `entities` list. |

### `slayer usage-reporting`

Show or change [usage reporting](usage-reporting.md).

```bash
slayer usage-reporting status # on/off, the rule that decided it, the install ID
slayer usage-reporting show # the report the next send would contain (sends nothing)
slayer usage-reporting disable # persist off; delete the install ID and unsent data
slayer usage-reporting enable # persist on
```
94 changes: 94 additions & 0 deletions docs/reference/usage-reporting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Usage reporting

SLayer collects **anonymous, aggregated usage statistics** to learn which surfaces, databases and
query features people use. It never sends your queries, data, model / column / datasource names,
memory text, arguments, error messages, file paths or credentials.

## When it is active

Only in processes started by the `slayer` command (`slayer query`, `slayer mcp`, `slayer serve`,
`slayer flight-serve`, `slayer pg-serve`, …, and the Docker image, which runs `slayer serve`).
Using SLayer as a Python library — the query engine, the client, or an app or MCP server you
build in your own code — never records, prints a notice or sends anything.

## Turning it off

The first rule that matches decides:

| # | Rule | Effect |
|---|------|--------|
| 1 | `DO_NOT_TRACK=1` (or `true` / `yes` / `on`) | off |
| 2 | `SLAYER_USAGE_REPORTING=on` / `SLAYER_USAGE_REPORTING=off` | on / off |
| 3 | `CI` is set; SLayer is an editable install; the config directory cannot be written | off |
| 4 | `slayer usage-reporting enable` / `slayer usage-reporting disable` | on / off (persisted) |
| 5 | Default | on |

```bash
slayer usage-reporting status # on or off, which rule decided it, the install ID
slayer usage-reporting show # print exactly the report the next send would contain (sends nothing)
slayer usage-reporting disable # turn off for good; also deletes the install ID and unsent data
slayer usage-reporting enable
```

## The notice

The first time usage reporting is on, SLayer prints a one-line notice on **stderr** (never stdout, so
stdio MCP stays clean) and records that it did. It is shown again only if the report format
changes in a major way.

## What a report contains

One JSON document per send; every value is a version, a SLayer-defined token, a boolean, a
bucket, a date, a random ID or a count. Anything SLayer does not recognise is reported as
`other` (or `custom` for user-defined aggregations).

| Field | Content |
|-------|---------|
| `schema` | Report format version. |
| `install_id` | Random UUID created on first use, replaced every 11 months. |
| `batch_id` | Random UUID per send (lets the backend drop duplicates). |
| `period` | First and last **date** covered (no times). |
| `env` | SLayer version, Python `major.minor`, OS family, CPU architecture, whether it runs in a container, installed SLayer extras (read from package metadata). |
| `usage` | Per action — CLI command (except `slayer usage-reporting …`), MCP tool name, REST route template (e.g. `/models/{name}`), Flight SQL / PostgreSQL query or probe — the count of successes and of failures per error kind (a fixed list such as `value_error`, `schema_drift`, `http_4xx`). |
| `features` | Per query: counts of multistage queries, inline source models, time dimensions, filters, computed dimensions, saved-query runs, non-default `to_many_handling`; counts of each built-in aggregation and transform (`sum`, `time_shift`, …), user-defined ones as `custom`. |
| `dialects` | Queries per SQL dialect; datasources per dialect in buckets `0` / `1` / `2-5` / `6+`. |
| `context` | Per process: storage backend (`yaml` / `sqlite` / `other`), model count in buckets `0` / `1-10` / `11-50` / `51-200` / `200+`, whether the bundled demo database was queried. |
| `mcp_clients` | MCP sessions per client (Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Codex, Zed, Cline, Goose, Continue; anything else `other`) with its major version. |

## How and when it is sent

Each process keeps its counts in memory and writes them to its own spool file when it ends — no
network at that point. A report is sent at most **once per 24 hours**, and only while SLayer is
running anyway (no heartbeat): when a `slayer` process starts, or a running server records an
action, and the last successful send was 24 hours ago or more. The send merges all spool files
and runs in a background thread; at exit SLayer waits for it at most about one second. Unsent
data stays in the spool (capped at about 100 files / 1 MB) for a later try.

Usage reporting never changes a command's output or exit code: every usage reporting failure is ignored.

Files live in the platform config directory, not in SLayer's model storage:
`$XDG_CONFIG_HOME/slayer` (default `~/.config/slayer`) on Linux,
`~/Library/Application Support/slayer` on macOS, `%APPDATA%\slayer` on Windows —
`usage-reporting.json` (setting, install ID, notice and send bookkeeping), `usage-reporting-spool/`
and two small lock files that keep concurrent `slayer` processes from overwriting each other.

## Destination

Reports go to the PostHog EU capture API (`https://eu.i.posthog.com/i/v0/e/`) as one
`slayer_usage` event per send, keyed by the install ID, which PostHog keeps as one "person" so that
all of an install's reports can be deleted together. Your IP address is not in the report;
PostHog receives it at the network level, our project is configured to discard it, and every
event disables GeoIP lookup. `SLAYER_USAGE_REPORTING_ENDPOINT` overrides the URL.

## Privacy notice

- **Controller**: Motley AI, the maintainer of SLayer.
- **Purpose**: understanding how SLayer is used, to prioritise its development.
- **Processor**: PostHog (EU hosting), under a data processing agreement.
- **Retention**: raw reports (keyed by install ID) are deleted within 12 months: an install ID is used
for at most 11 months, and a weekly job deletes it, with all its reports, once its first report is
11 months and one week old. Aggregates without install IDs are kept but not published; any future
summary suppresses every figure covering fewer than 10 installs.
- **Your rights**: `slayer usage-reporting status` prints your install ID; to have its data deleted,
email privacy@motley.ai with that ID, and its reports are deleted, normally within a few days.
`slayer usage-reporting disable` stops collection and deletes the local ID and unsent data.
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-10-08
Loading
Loading