From 4b326bc757e391eb80047c6fab19987317f2cbc9 Mon Sep 17 00:00:00 2001 From: Peter Steinberger Date: Fri, 30 Oct 2026 12:48:27 +0100 Subject: [PATCH] feat: aggregate identifier-free reporting installation counts Receiver for openclaw/openclaw#186569. Successful update answers advertise the server UTC day only when the optional INSTALLATION_COUNTS D1 binding exists; a separate empty count-only POST increments an aggregate (day, gap, reports) histogram with 90-day pruning. Production stays unbound. Recovered from stalled Team publication 3f1ba16c (push never confirmed). Co-authored-by: roboclaw-bot <309084314+roboclaw-bot@users.noreply.github.com> --- CHANGELOG.md | 1 + README.md | 40 +++- docs/installation-counts.md | 207 +++++++++++++++++++++ docs/update-results.md | 7 +- migrations/0001_installation_counts.sql | 7 + src/env.ts | 2 + src/index.ts | 72 +++++++- src/installation-count.ts | 40 ++++ src/page.ts | 7 +- test/deployment.test.ts | 7 + test/installation-count-runtime.test.mjs | 224 +++++++++++++++++++++++ test/installation-count.test.ts | 97 ++++++++++ test/public-surface.test.ts | 4 + wrangler.local.jsonc | 16 ++ 14 files changed, 715 insertions(+), 16 deletions(-) create mode 100644 docs/installation-counts.md create mode 100644 migrations/0001_installation_counts.sql create mode 100644 src/installation-count.ts create mode 100644 test/installation-count-runtime.test.mjs create mode 100644 test/installation-count.test.ts create mode 100644 wrangler.local.jsonc diff --git a/CHANGELOG.md b/CHANGELOG.md index 170108b..18784d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## Unreleased +- Accept identifier-free reporting-installation counts into an optional aggregate-only D1 `(day, gap, reports)` histogram with 90-day pruning; successful update answers advertise the server UTC day only when the binding exists, and production stays unbound. Thanks @roboclaw-bot. - Maintain public vocabulary with daily checks of immutable stable releases, retained backfill, isolated generation, constrained publication, serialized deployment, and retryable exact-commit rollout verification. - Preserve all reported public channel, provider, and plugin names instead of truncating each list to 32 before validation; keep upload and Analytics Engine byte limits covered by regression tests. - Refresh retained public names from OpenClaw 2026.9.7, including GitHub, QuickJS Code Mode, and Session Share; require a reviewed snapshot for the latest published release in the existing CI check. diff --git a/README.md b/README.md index d30276a..af17757 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,29 @@ written from a validated request. | `HEAD /api/latest-version` | Outcome capability only: empty 204 with `OpenClaw-Update-Results: 2` when its separate binding is present; otherwise empty 503 without that header. Always `Cache-Control: no-store`; no analytics, quota consumption or version lookup. | | `GET /` | Human-readable page: what is collected, how to turn it off, without a public statistics dashboard. | +## Aggregate-only reporting installation counts + +Ordinary daily GETs and valid schema-1 POSTs send **no count header**. When the +optional D1 binding is configured, successful 200 version answers advertise +`OpenClaw-Installation-Count: 1;` with +`Cache-Control: no-store`. A companion client can claim that server day once per +endpoint/state root and send one separate empty POST to the same endpoint using +`User-Agent: openclaw-installation-count/1` and +`OpenClaw-Installation-Count: 1;;`. It has no feature/version/location +fields, cannot enter raw Analytics Engine recording, and increments only a separate +D1 histogram `(day, gap, reports)`. These are **observed reporting installation-state +roots**, not unique people or engagement. Explicit telemetry off and `DO_NOT_TRACK` +suppress the count POST; it forbids redirects, has an independent three-second +timeout and never retries or discards the successful version answer. Old or +unconfigured receivers get no claim or count metadata. At most one extra count-only +POST follows an existing uncached successful 24-hour update check; no extra update +check is added. Schema-2 outcomes and HEAD neither advertise nor increment counts. +Production config deliberately omits the +optional binding and cleanup trigger, so landing cannot activate collection or +provision D1. A separately authorized rollout must configure and monitor 90-day +cleanup, review provider recovery retention, and verify delivery. See the +[wire contract, privacy boundaries, local proof and private rolling/calendar SQL](docs/installation-counts.md). + ## Identifier-free update outcomes The same POST endpoint also accepts a strict schema-2 `update_result` event from @@ -36,9 +59,10 @@ so the existing main-push deployment cannot implicitly activate outcome collecti Outcome attempts HEAD the same full configured endpoint and POST only after exact 204 plus `OpenClaw-Update-Results: 2`; old receivers (405) and unconfigured receivers (503) never receive the outcome payload. Binding presence is capability, not proof -of production delivery. The daily GET and opt-in schema-1 POST remain single requests. +of production delivery. The daily version GET or opt-in schema-1 POST remains one request; the optional +installation-count protocol above may follow it with one isolated count-only POST. See the contract for the shared timeout, no-redirect and opt-out recheck requirements. -The daily-check behavior described below is unchanged. +The daily version-check cadence and client cache described below are unchanged. ## What an install sends @@ -98,7 +122,7 @@ each field; missing or invalid values are left empty without discarding valid fi ## What is stored -Each recorded daily update check contributes one Analytics Engine data point with these columns and no others (schema-2 outcome storage is documented separately above): +Each recorded daily update check contributes one Analytics Engine data point with these columns and no others (schema-2 outcomes and the optional aggregate-only installation histogram are documented separately above): | Column | Value | | --- | --- | @@ -172,7 +196,7 @@ the edge and in validation: vocabulary. Accepted names are lowercased, deduplicated, and sorted; unknown names are dropped. Versions must match the release format or become `unknown`. Runtime catalog changes and network outages cannot widen the vocabulary or erase its history. -- **Plausibility.** Raw rows are retained, so a skew attempt appears as a discontinuity in a +- **Plausibility (legacy Analytics Engine only).** Raw rows are retained, so a skew attempt appears as a discontinuity in a dimension and can be discounted after the fact. An attacker willing to distribute traffic can still inflate counts for things that genuinely exist. @@ -188,7 +212,8 @@ get attention, not billing or security decisions. - Raw numeric UTC offsets, coordinates, postal codes, or physical-device hardware details These Analytics Engine rows contain no direct user, account, install, or device identifier. -Reports are not unique installations or users, and the service does not maintain per-install +Legacy daily rows and outcome reports are not unique installations or users. The optional +histogram estimates observed reporting installation-state roots without IDs; the service does not maintain per-install histories or retention curves. Cloudflare processes connection IP addresses, and the Worker uses them transiently for rate @@ -200,7 +225,7 @@ processing is outside those settings. | Command or setting | Effect | | --- | --- | -| `openclaw telemetry off` | Stops anonymous feature statistics. Update checks and default-on outcomes continue. | +| `openclaw telemetry off` | Stops anonymous feature statistics and the companion installation-count POST. Update checks and default-on outcomes continue. | | `DO_NOT_TRACK=1` | Same, enforced from the environment. | | `update.checkOnStart: false` | Stops automatic update requests and outcome reporting. Explicit update commands and other configured services are separate. | @@ -209,7 +234,8 @@ daily checks and schema-1 feature reports unless a replacement `OPENCLAW_TELEMET is explicitly configured. Outcome reports remain suppressed in CI even with a replacement endpoint. Disabling requests stops future automatic reports; it does not erase previously recorded rows. -The same three-month Analytics Engine retention applies. This receiver adds no backup or export job. +The same three-month Analytics Engine retention applies; the optional installation histogram +has separate 90-day cleanup described above. This receiver adds no backup or export job. `openclaw telemetry show` displays policy and a CLI-built payload preview, not the exact next Gateway payload: registry state, configuration, and collection time can differ. It cannot preview diff --git a/docs/installation-counts.md b/docs/installation-counts.md new file mode 100644 index 0000000..8d0e2f1 --- /dev/null +++ b/docs/installation-counts.md @@ -0,0 +1,207 @@ +# Identifier-free reporting installation counts + +This is an **observed reporting installation-state-root** metric, not people, +accounts, physical machines, active users, human engagement or all installations. +The counter stores aggregates only. Legacy daily Analytics Engine rows and +schema-2 update outcomes are separate, unchanged owners. + +## Capability and isolated count-only transport + +The public constants in `src/installation-count.ts` are: + +| Constant | Exact value | +| --- | --- | +| INSTALLATION_COUNT_HEADER | `OpenClaw-Installation-Count` | +| INSTALLATION_COUNT_USER_AGENT | `openclaw-installation-count/1` | + +1. The existing uncached daily update GET or opt-in schema-1 feature POST sends + **no count header**. A valid successful **200** version answer advertises + `OpenClaw-Installation-Count: 1;` **only when the optional + `INSTALLATION_COUNTS` binding is configured**. The receiver captures that day + once from its clock after the version lookup and uses `Cache-Control: no-store` + on this day-bearing response. The separate five-minute npm cache remains in use. + HEAD, schema-2 outcomes, malformed feature uploads and errors never advertise + count capability. Quota-exhausted POSTs also omit it because their body is not + read or classified. Old/unconfigured receivers advertise nothing. +2. After a fresh valid version answer, the companion client requires the exact + capability `1;` with canonical bounded integer day (0–99999999). It + atomically claims **that server day**, not its local-clock day, using its + existing SQLite owner per **endpoint + state root + day**. There is no claim + or count metadata sent to an old/unconfigured receiver. Local time is only + update bookkeeping, not the count bucket. A first install can count in this + same capability-bearing cycle; no activation warm-up or persistent capability + state is needed. +3. After claiming, the client rechecks current policy including explicit + `telemetry.enabled: false` (`openclaw telemetry off`), `DO_NOT_TRACK` and + CI restrictions for the captured endpoint. It may then send **one separate + POST to that same full endpoint**, with the fixed generic User-Agent above, + `OpenClaw-Installation-Count: 1;;` and **no body**. There are no + feature, inventory, version, geography, identifier or token fields. Redirects + are forbidden; the count POST has an independent bounded three-second timeout. + No retry, outbox or delayed queue is added. Failed or ambiguous delivery never + discards the valid cached version answer and never causes a repeated count. + +This adds at most **one count-only POST** after an existing uncached successful +24-hour update check; it does not add another update check or model request. +A client-side cache hit, including a same-day cold restart that reuses the cache, +sends neither request. Feature statistics remain off by default. + +Gap is canonical ASCII integer **1–31**: server-day distance from the previous +local claim, capped at 31; first/unknown history uses 31. No zero-gap header is +sent. Header fields accept no whitespace, extra fields or leading zeroes. The +receiver accepts the current or immediately previous authoritative UTC day, +retaining the reported original day rather than rebucketing by arrival time. +This allows delivery across midnight; future/older days and comma-joined duplicate +headers are rejected. No local client-clock comparison selects the count day. + +**Any presence** of the count header selects the isolated count path before HEAD, +outcome parsing, legacy analytics, geography or npm lookup, even if the value is +empty/malformed. The generic count User-Agent also selects this path if its header +is missing. Valid requests require POST, that exact User-Agent, a valid header and +an absent/empty body, validated synchronously from provider-trusted physical HTTP +framing: no Transfer-Encoding and either exact Content-Length: 0, or absent +Content-Length with a null body. Real workerd exposes a non-null stream even for +a correctly framed Node fetch POST with Content-Length: 0, so that framing is +accepted without probing EOF. Nonzero/invalid lengths and present chunked or +unframed bodies are rejected immediately, including stalled streams. No stream +reader is obtained and no body is read, accumulated or decoded. There is no +raw-analytics fallback. + +Count-only responses have no body and `Cache-Control: no-store`: + +| Status | Meaning | +| --- | --- | +| 204 | Accepted, or intentionally ignored for absent binding, `DNT: 1` or exhausted recording quota | +| 400 | Invalid header, wrong method/User-Agent, or invalid/nonempty/unframed body framing | +| 503 | D1 write failed; delivery may still be ambiguous, so never retry | + +Binding presence is capability, not proof of storage health or production +readiness. Count-only errors are separate from the already-successful update +answer. The client does not derive exactly-once storage from any response. + +There is no identifier to deduplicate repeated requests at the receiver. The +client's durable daily claim owns that guarantee. Separate state roots count +separately; deleted/reset/copied state, missed deliveries, server clock errors, +opt-outs, rate limits, unavailable storage and forged requests bias this metric. +Missing an early qualifying report can undercount a longer window even when a +later report arrives. It is not suitable for billing, security decisions or +precise fleet size. No public configuration knob or statistics route is added. + +## Aggregate storage and expiry + +The optional `INSTALLATION_COUNTS` D1 binding uses +[`migrations/0001_installation_counts.sql`](../migrations/0001_installation_counts.sql). +Its only application table is: + +| Column | Meaning | +| --- | --- | +| day | Report's bounded UTC ordinal day | +| gap | Days since previous local claim, capped at 31 | +| reports | Positive aggregate number of reports in this day/gap bin | + +The primary key is `(day, gap)`. All columns have bounded integer constraints. +One transactional D1 batch deletes expired bins and atomically UPSERTs the count; +there is no read/modify/write race and no request-level record. At most 90 × 31 +bins remain after cleanup. The D1 code receives only day/gap, not a Request, +User-Agent, IP, geography, token, dimension or timestamp. No raw events are +written to Analytics Engine for this counter. Count-only requests never enter +the legacy recorder, even when malformed. The separate ordinary update request +can still produce its existing documented identity/geography row; the histogram +contains none of those dimensions. + +Retention is the current UTC day and preceding 89 days. Ingestion prunes older +bins; a daily scheduled cleanup also prunes when no clients report. A separately +authorized production rollout **must configure and monitor this cleanup**. Failed +cleanup or a disabled trigger can extend physical retention until recovery; +never describe a broken schedule as guaranteed automatic expiry. No archive, +backup or raw export job is added. Provider-managed D1 recovery/Time Travel and +connection processing are separate infrastructure policies the rollout must +review. Worker observability, logs and invocation logs remain disabled. + +The existing per-IP recording limiter gates the histogram too, exactly once +after synchronous framing validation; no count body is read. IP is used only as +the existing transient limiter key, +not written to D1. Ordinary quota-exhausted update callers still receive their +version answer; quota-exhausted count-only POSTs return empty 204 without storage +or reading an unfinished body. D1 writes are awaited only by the separate count +POST; missing schema/storage errors return empty 503, never a changed update +answer. A successful update response is **not** an acknowledgement of count +storage; do not retry an ambiguous count. + +## Deployment gate + +The production `wrangler.jsonc` intentionally has **no D1 binding or cleanup +trigger**. Existing main-push automation can deploy receiver code, but cannot +provision or activate the new counter. Landing this change authorizes neither +production collection nor paid resources, infrastructure provisioning or manual +deployment. A separate authorized rollout must provision/approve D1, apply the +migration, add `INSTALLATION_COUNTS` and a daily cleanup trigger, verify storage, +expiry and provider recovery retention, then approve client rollout. Preserve +`TELEMETRY` and `UPDATE_RESULTS` ownership. No credentials are needed for local proof. +Never deploy `wrangler.local.jsonc`: its UUID is a dummy local database, not a +production identifier. + +## Private aggregate analysis + +Authorized operators can query the account-level **private D1 API or console**; +no Worker statistics route or client credential is added. Bind `:S` and `:E` +as inclusive UTC ordinal days with `0 <= E-S <= 30`, both inside retention: + +```sql +SELECT COALESCE(SUM(reports), 0) AS reporting_installation_state_roots +FROM installation_counts +WHERE day BETWEEN :S AND :E AND gap > day - :S; +``` + +A report qualifies only when its previous local claim predates the window. +Under intact client claims and complete delivery this counts the first report +per root in that window exactly once. The 31-day gap cap supports windows of up +to 31 days, not arbitrary longer periods. Attempt-before-network prevents retry +overcount, but a lost qualifying report can cause undercount even if a later +report arrives. Unknown history and state resets can overcount. + +Rolling 1/7/30 calendar-day windows ending today (today may be incomplete): + +```sql +WITH today(d) AS (SELECT CAST(unixepoch('now') / 86400 AS INTEGER)), +windows(n) AS (VALUES (1), (7), (30)) +SELECT n AS days, COALESCE(SUM(h.reports), 0) AS reporting_installation_state_roots +FROM windows CROSS JOIN today +LEFT JOIN installation_counts h ON h.day BETWEEN d-n+1 AND d AND h.gap > h.day-(d-n+1) +GROUP BY n ORDER BY n; +``` + +For a calendar month, convert its first and last included date to `S` and `E`; +months up to 31 days use the same query. Late boundary delivery can revise +counts for the previous day. These are private, biased reporting-root counts, +not public unique-user statistics; review any publication separately. + +## Reproduce with local workerd and persisted SQLite + +Use the locked dependencies, Node.js 24+ and **local-only** D1 commands: + +```sh +npm ci +npm test -- test/installation-count.test.ts test/installation-count-runtime.test.mjs test/deployment.test.ts +npx wrangler d1 migrations apply installation-counts-local --local --config wrangler.local.jsonc +WRANGLER_SEND_METRICS=false npx wrangler dev --local --config wrangler.local.jsonc --ip 127.0.0.1 --port 8797 --test-scheduled +# In another terminal: inspect capability on the ordinary update response. +curl -i http://127.0.0.1:8797/api/latest-version +# Use its advertised server day below; do not derive it from the client clock. +# Example only: day=20754 +curl -i -X POST -A openclaw-installation-count/1 -H "OpenClaw-Installation-Count: 1;$day;31" http://127.0.0.1:8797/api/latest-version +npx wrangler d1 execute installation-counts-local --local --config wrangler.local.jsonc --command 'SELECT day,gap,reports FROM installation_counts ORDER BY day,gap' +# Local simulated cron only: +curl 'http://127.0.0.1:8797/cdn-cgi/handler/scheduled?cron=0+0+*+*+*' +``` + +On hosts at their filesystem-watch limit, prefix the dev command with +`CHOKIDAR_USEPOLLING=true` instead of changing system limits. + +The tests use the production Worker over real workerd HTTP with simulated D1 +persisted to local SQLite, synthetic inputs and an npm upstream fixture. They +prove concurrent atomic increments, persistence across runtime restart, strict +invalid/duplicate header rejection, capability/legacy/outcome/HEAD separation, +zero raw count-path analytics/geography/npm access, DNT and rate limiting, +missing binding/schema availability, migration constraints and idle cleanup. +This does not establish production readiness or replace real client integration. diff --git a/docs/update-results.md b/docs/update-results.md index 55d0587..5254b8c 100644 --- a/docs/update-results.md +++ b/docs/update-results.md @@ -39,8 +39,11 @@ uses the recording limiter, writes analytics (including sample points), or looks up a version. It is a protocol capability check, **not production readiness or successful storage proof**. The client shares a three-second timeout across both requests, disallows redirects, and rechecks update-request opt-outs after the -HEAD await before sending. No retries or delayed queue are added. Daily GET and -opt-in schema-1 feature POST remain single requests and need no handshake. +HEAD await before sending. No retries or delayed queue are added. The daily +version GET or opt-in schema-1 feature POST needs no separate HEAD handshake. +When separately configured, the [installation-count protocol](installation-counts.md) +can advertise its own capability on the successful version response and follow +it with one isolated empty count-only POST; that is not a schema-2 outcome. ## Wire contract diff --git a/migrations/0001_installation_counts.sql b/migrations/0001_installation_counts.sql new file mode 100644 index 0000000..827db7f --- /dev/null +++ b/migrations/0001_installation_counts.sql @@ -0,0 +1,7 @@ +-- No identifiers, request records, timestamps, dimensions or geography. +CREATE TABLE installation_counts ( + day INTEGER NOT NULL CHECK (typeof(day) = 'integer' AND day BETWEEN 0 AND 99999999), + gap INTEGER NOT NULL CHECK (typeof(gap) = 'integer' AND gap BETWEEN 1 AND 31), + reports INTEGER NOT NULL CHECK (typeof(reports) = 'integer' AND reports BETWEEN 1 AND 9007199254740991), + PRIMARY KEY (day, gap) +) WITHOUT ROWID; diff --git a/src/env.ts b/src/env.ts index c96b5f9..25544c4 100644 --- a/src/env.ts +++ b/src/env.ts @@ -6,6 +6,8 @@ export type Env = { TELEMETRY: AnalyticsEngineDataset; /** Separate, identifier-free outcome dataset; never fall back to TELEMETRY. */ UPDATE_RESULTS?: AnalyticsEngineDataset; + /** Optional aggregate-only D1 histogram; deliberately absent in production. */ + INSTALLATION_COUNTS?: D1Database; /** Per-IP limit on recorded update checks. */ RATE_LIMIT?: RateLimiter; }; diff --git a/src/index.ts b/src/index.ts index bcd3023..8e9c74c 100644 --- a/src/index.ts +++ b/src/index.ts @@ -2,6 +2,7 @@ import { normalizeVersion } from "./allowlist.js"; import { buildDataPoint } from "./analytics.js"; import type { Env } from "./env.js"; import { readCappedBody } from "./feature-stats.js"; +import { INSTALLATION_COUNT_HEADER, INSTALLATION_COUNT_USER_AGENT, parseInstallationCount, pruneInstallationCounts, recordInstallationCount } from "./installation-count.js"; import { parseRequestGeography } from "./geography.js"; import { parseClientIdentity, parseFeatureStats } from "./payload.js"; import type { FeatureStats } from "./payload.js"; @@ -111,7 +112,10 @@ function recordRequest(request: Request, env: Env, features: FeatureStats | unde async function handleLatestVersion(request: Request, env: Env): Promise { // Check recording quota exactly once, before reading any upload. Exhausted // callers must still get a version answer even if their body never finishes. - if (!(await mayRecord(request, env))) return latestVersionResponse(); + if (!(await mayRecord(request, env))) return latestVersionResponse( + Boolean(env.INSTALLATION_COUNTS) && request.method === "GET" && + request.headers.get("user-agent") !== UPDATE_RESULT_USER_AGENT, + ); let parsed: unknown; if (request.method === "POST") { @@ -131,21 +135,79 @@ async function handleLatestVersion(request: Request, env: Env): Promise { +async function latestVersionResponse(advertiseInstallationCount = false): Promise { const latest = await fetchLatestVersion(); if (!latest) return jsonResponse({ error: "version_unavailable" }, 503); - return jsonResponse(RELEASE_NOTE ? { ...latest, note: RELEASE_NOTE } : latest, 200, VERSION_CACHE_SECONDS); + const response = jsonResponse(RELEASE_NOTE ? { ...latest, note: RELEASE_NOTE } : latest, + 200, advertiseInstallationCount ? 0 : VERSION_CACHE_SECONDS); + if (advertiseInstallationCount) { + // Capture the authoritative day once, after the version lookup. The npm + // cache still works, but this day-bearing response must never be cached. + const day = Math.floor(Date.now() / 86_400_000); + response.headers.set(INSTALLATION_COUNT_HEADER, `1;${day}`); + } + return response; +} + +function installationCountResponse(status: number): Response { + return new Response(null, { status, headers: { "cache-control": "no-store" } }); +} + +/** Validate provider-trusted HTTP framing synchronously; never read a stream. */ +function hasEmptyInstallationCountBody(request: Request): boolean { + if (request.headers.has("transfer-encoding")) return false; + const length = request.headers.get("content-length"); + // Real workerd exposes a non-null stream even for a Node fetch POST with + // Content-Length: 0. Trust that physical HTTP framing, not stream EOF. + return length === "0" || (length === null && request.body === null); +} + +async function handleInstallationCount(request: Request, env: Env): Promise { + const now = Date.now(); + const count = parseInstallationCount(request.headers.get(INSTALLATION_COUNT_HEADER), now); + if (request.method !== "POST" || !count || + request.headers.get("user-agent") !== INSTALLATION_COUNT_USER_AGENT) return installationCountResponse(400); + // Count requests never fall through to legacy analytics or a version lookup, + // even when malformed, quota-exhausted or unavailable. IP stays a limiter key. + if (!hasEmptyInstallationCountBody(request)) return installationCountResponse(400); + if (!(await mayRecord(request, env))) return installationCountResponse(204); + if (!env.INSTALLATION_COUNTS || request.headers.get("dnt") === "1") return installationCountResponse(204); + // The existing limiter await can cross midnight. Validate again at the + // write boundary and use that clock for retention pruning. + const recordedAt = Date.now(); + if (!parseInstallationCount(request.headers.get(INSTALLATION_COUNT_HEADER), recordedAt)) return installationCountResponse(400); + try { + await recordInstallationCount(env.INSTALLATION_COUNTS, count, recordedAt); + return installationCountResponse(204); + } catch { + // Delivery can be ambiguous: no retry here or in the client, even on 503. + return installationCountResponse(503); + } } export default { + async scheduled(_controller: ScheduledController, env: Env): Promise { + await pruneInstallationCounts(env.INSTALLATION_COUNTS); + }, async fetch(request: Request, env: Env): Promise { const url = new URL(request.url); if (url.pathname === "/api/latest-version") { + // Header presence, not validity, selects the isolated count-only path. + // The fixed count UA also fails closed when its header is missing. + if (request.headers.has(INSTALLATION_COUNT_HEADER) || + request.headers.get("user-agent") === INSTALLATION_COUNT_USER_AGENT) { + return handleInstallationCount(request, env); + } // Capability only: no body, geography, recording quota, analytics or npm. // Presence is not a delivery/readiness probe; never write a sample point. if (request.method === "HEAD") { diff --git a/src/installation-count.ts b/src/installation-count.ts new file mode 100644 index 0000000..45793b5 --- /dev/null +++ b/src/installation-count.ts @@ -0,0 +1,40 @@ +/** Aggregate-only histogram: never receives request metadata or an identifier. */ +export const INSTALLATION_COUNT_HEADER = "OpenClaw-Installation-Count"; +export const INSTALLATION_COUNT_USER_AGENT = "openclaw-installation-count/1"; +export const INSTALLATION_COUNT_RETENTION_DAYS = 90; +const DAY_MS = 86_400_000; +const HEADER_PATTERN = /^1;(0|[1-9][0-9]{0,7});([1-9]|[12][0-9]|3[01])$/; +export type InstallationCount = { day: number; gap: number }; + +export function parseInstallationCount(value: string | null, now = Date.now()): InstallationCount | undefined { + // Bound work first. Comma-joined duplicate headers, whitespace, extra fields, + // leading zeroes, and trailing line terminators are not protocol values. + if (!value || value.length > 13) return undefined; + const match = HEADER_PATTERN.exec(value); + if (!match || match[0] !== value) return undefined; + const day = Number(match[1]); + const today = Math.floor(now / DAY_MS); + if (day !== today && day !== today - 1) return undefined; + return { day, gap: Number(match[2]) }; +} + +export async function pruneInstallationCounts(db: D1Database | undefined, now = Date.now()): Promise { + if (!db) return; + // Scheduled cleanup failures must fail the scheduled invocation so operators + // can detect broken expiry. This handler is never on the update request path. + await db.prepare("DELETE FROM installation_counts WHERE day < ?") + .bind(Math.floor(now / DAY_MS) - INSTALLATION_COUNT_RETENTION_DAYS + 1).run(); +} + +export async function recordInstallationCount(db: D1Database, count: InstallationCount, now = Date.now()): Promise { + // D1 batch is transactional. UPSERT increments inside SQLite, not via a + // racy read/modify/write, and there is never a request-level row. + await db.batch([ + db.prepare("DELETE FROM installation_counts WHERE day < ?") + .bind(Math.floor(now / DAY_MS) - INSTALLATION_COUNT_RETENTION_DAYS + 1), + db.prepare( + "INSERT INTO installation_counts (day, gap, reports) VALUES (?, ?, 1) " + + "ON CONFLICT (day, gap) DO UPDATE SET reports = reports + 1", + ).bind(count.day, count.gap), + ]); +} diff --git a/src/page.ts b/src/page.ts index d51217f..3b008b3 100644 --- a/src/page.ts +++ b/src/page.ts @@ -54,6 +54,9 @@ footer { margin-top: 3rem; padding-top: 1.5rem; border-top: 1px solid var(--line

Interactive setup defaults to No thanks; guided Quick Start skips the question. Scripted installs do not opt in automatically. The enabled setting controls inclusion, not whether a prompt was answered.

Channels and providers describe configuration; plugins describe enabled inventory, not invocations. sessionsLast24h counts retained session-creation events timestamped in the preceding 24 hours, not active sessions or messages. Missing or unreadable local state produces zero.

+

Aggregate-only installation counts

+

Ordinary daily update requests send no count header. When its optional D1 binding is configured, a successful 200 update answer advertises OpenClaw-Installation-Count: 1;<server UTC ordinal day> with Cache-Control: no-store. A companion client can claim that server day per endpoint/state root and send one separate empty POST using User-Agent: openclaw-installation-count/1 and OpenClaw-Installation-Count: 1;<server day>;<gap 1–31>. This adds at most one count-only POST per existing uncached successful 24-hour update check, never another update check or a retry. Old/unconfigured receivers get no count claim or metadata. Explicit telemetry off and DO_NOT_TRACK suppress this count POST. Count-only requests never enter raw analytics or geography processing, even when malformed. Only an aggregate D1 histogram (day, gap, reports) is stored: no identifiers, request-level rows or dimensions. It measures observed reporting installation-state roots, not people or engagement. HEAD and update outcomes neither advertise nor increment it. Count failures never discard a valid version answer. The optional production binding and cleanup trigger are absent; separately authorized rollout must configure and monitor 90-day cleanup and review provider recovery retention before collection. See the count contract and private aggregate SQL.

+

Update outcomes

The receiver also supports identifier-free terminal update outcomes from a companion client implementation. These use a separate dataset, strict public version labels and bounded outcome categories, with no geography, install IDs, raw errors or logs. Uploads are limited to 4096 bytes. Reports are retained for three months; no public individual-report route is provided. The companion client reports outcomes on by default, like the existing update ping, under update-request policy rather than optional feature-statistics consent. update.checkOnStart: false, OPENCLAW_NO_AUTO_UPDATE=1, and Nix mode suppress outcomes; a truthy CI always suppresses outcomes, even when a replacement OPENCLAW_TELEMETRY_ENDPOINT is configured. DO_NOT_TRACK controls feature statistics, not update outcomes. Receiver support does not itself enable client reporting: deploy and verify the receiver and separate dataset before releasing the default-on client, under separately authorized rollout. See the outcome contract and collection boundaries.

@@ -69,12 +72,12 @@ footer { margin-top: 3rem; padding-top: 1.5rem; border-top: 1px solid var(--line
  • Any install ID or device ID
  • Coordinates, postal codes, or physical-device hardware details
    • -

    Reports contain no user, account, install, or device identifier. Cloudflare processes connection IP addresses, and the Worker uses them transiently for rate limiting without storing them in Analytics Engine. Worker logs are disabled; Cloudflare's separate infrastructure processing is outside those settings.

    +

    Reports contain no user, account, install, or device identifier. Cloudflare processes connection IP addresses, and the Worker uses them transiently for rate limiting without storing them in Analytics Engine or the installation histogram. Worker logs are disabled; Cloudflare's separate infrastructure processing is outside those settings.

    How to turn it off

    - +
    Command or settingEffect
    openclaw telemetry offStops anonymous feature statistics. Update checks and default-on outcomes continue.
    openclaw telemetry offStops anonymous feature statistics and the companion installation-count POST. Update checks and default-on outcomes continue.
    DO_NOT_TRACK=1Same, enforced from the environment.
    update.checkOnStart: falseStops automatic update requests and outcome reporting. Explicit updates and other configured services are separate.
    diff --git a/test/deployment.test.ts b/test/deployment.test.ts index 5b143bc..60c4583 100644 --- a/test/deployment.test.ts +++ b/test/deployment.test.ts @@ -2,6 +2,13 @@ import { describe, expect, it } from "vitest"; import { experimental_readRawConfig } from "wrangler"; describe("deployment privacy", () => { + it("keeps installation collection and retention triggers out of production", () => { + const { rawConfig } = experimental_readRawConfig({ config: "wrangler.jsonc" }); + expect(rawConfig.d1_databases ?? []).toEqual([]); + expect(rawConfig.triggers?.crons ?? []).toEqual([]); + const { rawConfig: local } = experimental_readRawConfig({ config: "wrangler.local.jsonc" }); + expect(local.d1_databases).toMatchObject([{ binding: "INSTALLATION_COUNTS", remote: false }]); + }); it("keeps outcome collection out of the automatic production deployment", () => { const { rawConfig } = experimental_readRawConfig({ config: "wrangler.jsonc" }); expect(rawConfig.analytics_engine_datasets).toEqual([ diff --git a/test/installation-count-runtime.test.mjs b/test/installation-count-runtime.test.mjs new file mode 100644 index 0000000..2a32f56 --- /dev/null +++ b/test/installation-count-runtime.test.mjs @@ -0,0 +1,224 @@ +import { createRequire } from "node:module"; +import { Agent, request as httpRequest } from "node:http"; +import { readFileSync, mkdtempSync, rmSync, mkdirSync } from "node:fs"; +import { resolve } from "node:path"; +import { DatabaseSync } from "node:sqlite"; +import { afterEach, beforeAll, describe, expect, it } from "vitest"; +import { experimental_readRawConfig } from "wrangler"; +const require = createRequire(import.meta.url); +const wranglerRequire = createRequire(require.resolve("wrangler")); +const { build } = wranglerRequire("esbuild"); +const { Miniflare, convertV4MiniflareOptions } = wranglerRequire("miniflare"); +const { rawConfig: local } = experimental_readRawConfig({ config: "wrangler.local.jsonc" }); +const migration = readFileSync("migrations/0001_installation_counts.sql", "utf8"); +const outcome = readFileSync("test/fixtures/update-result.json", "utf8"); +const day = Math.floor(Date.now() / 86_400_000); +const header = (gap = 31, d = day) => "1;" + d + ";" + gap; + +const COUNT_HEADER = "OpenClaw-Installation-Count"; +const COUNT_UA = "openclaw-installation-count/1"; +describe("capability-gated count-only HTTP and persisted D1", () => { + let script, guardScript, runtime, db, persist, outboundCalls; + beforeAll(async () => { + const options = { bundle: true, format: "esm", platform: "browser", write: false }; + script = (await build({ ...options, entryPoints: ["src/index.ts"] })).outputFiles[0].text; + guardScript = (await build({ ...options, stdin: { resolveDir: process.cwd(), contents: ` + import worker from "./src/index.ts"; + export default { async fetch(request, env) { + const calls = { daily: 0, outcomes: 0, geography: 0, bodyReads: 0 }; + Object.defineProperty(request, "cf", { get() { calls.geography++; throw new Error("unexpected geography"); } }); + const body = request.body; + if (body) Object.defineProperty(body, "getReader", { value() { calls.bodyReads++; throw new Error("unexpected body read"); } }); + const response = await worker.fetch(request, { ...env, + TELEMETRY: { writeDataPoint() { calls.daily++; } }, + UPDATE_RESULTS: { writeDataPoint() { calls.outcomes++; } }, + }); + // Test-only lost acknowledgement after the real D1 write completed. + if (request.headers.get("X-Test-Lose-Ack") === "1" && response.status === 204) return new Response(null, { status: 503 }); + const headers = new Headers(response.headers); + headers.set("X-Test-Calls", JSON.stringify(calls)); + headers.set("X-Test-Body-Shape", JSON.stringify({ nullBody: body === null, length: request.headers.get("content-length") })); + return new Response(response.body, { status: response.status, headers }); + }, scheduled: worker.scheduled }; + ` } })).outputFiles[0].text; + }); + afterEach(async () => { await runtime?.dispose(); if (persist) rmSync(persist, { recursive: true, force: true }); persist = undefined; }); + async function start({ binding = true, migrate = true, rateLimit = false, versionAvailable = true, guard = false } = {}) { + mkdirSync(".wrangler", { recursive: true }); + persist ??= mkdtempSync(resolve(".wrangler/installation-count-test-")); + outboundCalls = 0; + runtime = new Miniflare(convertV4MiniflareOptions({ + modules: true, script: guard ? guardScript : script, port: 0, cf: false, + compatibilityDate: local.compatibility_date, compatibilityFlags: local.compatibility_flags, + d1Databases: binding ? { INSTALLATION_COUNTS: local.d1_databases[0].database_id } : {}, + resourcePersistencePath: persist, logRequests: false, + analyticsEngineDatasets: { TELEMETRY: { dataset: "synthetic_daily" }, UPDATE_RESULTS: { dataset: "synthetic_outcomes" } }, + ...(rateLimit ? { ratelimits: { RATE_LIMIT: { namespace_id: "1001", simple: { limit: 1, period: 60 } } } } : {}), + outboundService: async (request) => { + outboundCalls++; + if (request.url !== "https://registry.npmjs.org/openclaw/latest") throw new Error("unexpected outbound request"); + return versionAvailable ? Response.json({ version: "2026.10.28" }) : new Response(null, { status: 503 }); + }, + })); + await runtime.ready; + if (binding) { db = await runtime.getD1Database("INSTALLATION_COUNTS"); if (migrate) await db.prepare(migration).run(); } + } + async function send(value = header(), init = {}) { + const headers = new Headers({ "user-agent": COUNT_UA, ...init.headers }); + if (value !== null) headers.set(COUNT_HEADER, value); + const response = await fetch(new URL("/api/latest-version", await runtime.ready), { method: "POST", ...init, headers, redirect: "error" }); + expect(await response.text()).toBe(""); + expect(response.headers.get("cache-control")).toBe("no-store"); + expect(response.headers.get(COUNT_HEADER)).toBeNull(); + if (response.headers.has("X-Test-Calls")) expect(JSON.parse(response.headers.get("X-Test-Calls"))).toEqual({ daily: 0, outcomes: 0, geography: 0, bodyReads: 0 }); + return response; + } + async function rows() { return (await db.prepare("SELECT day, gap, reports FROM installation_counts ORDER BY day, gap").all()).results; } + async function update(init = {}) { + const response = await fetch(new URL("/api/latest-version", await runtime.ready), init); + expect(await response.json()).toEqual({ version: "2026.10.28" }); + return response; + } + it.each([false, true])("ordinary GET/schema1 capability is fresh and storage-gated: %s", async (binding) => { + await start({ binding }); + for (const init of [{}, { method: "POST", body: JSON.stringify({ schema: 1, features: {} }) }]) { + const response = await update(init); + expect(response.status).toBe(200); + expect(response.headers.get(COUNT_HEADER)).toBe(binding ? "1;" + day : null); + expect(response.headers.get("cache-control")).toBe(binding ? "no-store" : "public, max-age=300"); + } + expect(outboundCalls).toBe(1); // Existing npm cache still works. + if (binding) expect(await rows()).toEqual([]); // A capability is not a count. + }, 30_000); + it("never advertises capability on HEAD, outcomes, invalid feature uploads or errors", async () => { + await start(); + for (const init of [{ method: "HEAD" }, { method: "POST", body: outcome }, + { method: "POST", body: "{", headers: { "user-agent": "openclaw-update-result/1" } }, + { method: "POST", body: "{" }, { headers: { "user-agent": "openclaw-update-result/1" } }]) { + const response = await fetch(new URL("/api/latest-version", await runtime.ready), init); + expect(response.headers.get(COUNT_HEADER)).toBeNull(); + await response.arrayBuffer(); + } + expect(await rows()).toEqual([]); + }, 30_000); + it("version errors cannot advertise or increment counts", async () => { + await start({ versionAvailable: false }); + const response = await fetch(new URL("/api/latest-version", await runtime.ready)); + expect(response.status).toBe(503); + expect(response.headers.get(COUNT_HEADER)).toBeNull(); + expect(await response.json()).toEqual({ error: "version_unavailable" }); + expect(await rows()).toEqual([]); + }, 30_000); + it("atomically increments count-only posts and persists only three histogram columns across restart", async () => { + await start({ guard: true }); + const responses = await Promise.all(Array.from({ length: 40 }, () => send(header(7)))); + expect(responses.every((response) => response.status === 204)).toBe(true); + await send(header(31)); await send(header(1, day - 1), { body: "" }); + const expected = [{ day: day - 1, gap: 1, reports: 1 }, { day, gap: 7, reports: 40 }, { day, gap: 31, reports: 1 }]; + expect(await rows()).toEqual(expected); + expect((await db.prepare("PRAGMA table_info(installation_counts)").all()).results.map((column) => column.name)).toEqual(["day", "gap", "reports"]); + expect(outboundCalls).toBe(0); + await runtime.dispose(); await start({ migrate: false }); expect(await rows()).toEqual(expected); + }, 30_000); + it("real Node fetch no-body POST passes zero-length framing without reading the workerd stream", async () => { + await start({ guard: true }); + const response = await send(); + expect(response.status).toBe(204); + expect(JSON.parse(response.headers.get("X-Test-Body-Shape"))).toEqual({ nullBody: false, length: "0" }); + expect(await rows()).toEqual([{ day, gap: 31, reports: 1 }]); + expect(outboundCalls).toBe(0); + }, 30_000); + it("header-selected stalled chunked upload is rejected before any body read", async () => { + await start({ guard: true }); + const url = new URL("/api/latest-version", await runtime.ready); + const agent = new Agent(); + const response = await new Promise((resolve, reject) => { + const request = httpRequest(url, { agent, method: "POST", headers: { + "user-agent": COUNT_UA, [COUNT_HEADER]: header(), "transfer-encoding": "chunked", + } }); + const timer = setTimeout(() => { request.destroy(); reject(new Error("count receiver waited for stalled body")); }, 1000); + request.on("error", (error) => { clearTimeout(timer); reject(error); }); + request.on("response", (incoming) => { + let body = ""; + incoming.setEncoding("utf8"); incoming.on("data", (chunk) => { body += chunk; }); + incoming.on("end", () => { + clearTimeout(timer); + resolve({ status: incoming.statusCode, body, calls: JSON.parse(incoming.headers["x-test-calls"]) }); + request.destroy(); + }); + }); + request.flushHeaders(); // Send no body bytes and never end the request. + }).finally(() => agent.destroy()); + expect(response).toEqual({ status: 400, body: "", calls: { daily: 0, outcomes: 0, geography: 0, bodyReads: 0 } }); + expect(await rows()).toEqual([]); expect(outboundCalls).toBe(0); + }, 30_000); + it("a lost HTTP acknowledgement after committed D1 storage does not trigger a receiver retry", async () => { + await start({ guard: true }); + const response = await fetch(new URL("/api/latest-version", await runtime.ready), { method: "POST", headers: { "user-agent": COUNT_UA, [COUNT_HEADER]: header(), "X-Test-Lose-Ack": "1" } }); + expect(response.status).toBe(503); await response.arrayBuffer(); + expect(await rows()).toEqual([{ day, gap: 31, reports: 1 }]); + expect(outboundCalls).toBe(0); + }, 30_000); + it("any header presence fails closed before raw analytics, geography, outcomes or npm", async () => { + await start({ guard: true }); + for (const value of [null, "", header(0), header(32), header(1, day + 1), header(1, day - 2), header() + ", " + header(), "2;" + day + ";1", "1;" + day + ";01", "1;" + day + ";1;id"]) expect((await send(value)).status).toBe(400); + for (const method of ["GET", "HEAD", "PUT", "DELETE", "OPTIONS"]) expect((await send(header(), { method })).status).toBe(400); + for (const agent of ["legacy-feature-agent", "openclaw-update-result/1", ""]) expect((await send(header(), { headers: { "user-agent": agent } })).status).toBe(400); + for (const body of [" ", "{", JSON.stringify({ schema: 1, features: {}, installId: "synthetic-private-id" }), outcome, "x".repeat(16_385)]) expect((await send(header(), { body })).status).toBe(400); + // No Content-Length: reject framed/present body without reading any chunk. + expect((await send(header(), { body: new ReadableStream({ start(controller) { controller.enqueue(new TextEncoder().encode("{")); controller.close(); } }), duplex: "half" })).status).toBe(400); + expect(outboundCalls).toBe(0); expect(await rows()).toEqual([]); + }, 30_000); + it("DNT and exhausted existing limiter ignore counts without update lookups", async () => { + await start({ guard: true, rateLimit: true }); + expect((await send(header(), { headers: { DNT: "1" } })).status).toBe(204); + for (let i = 0; i < 3; i++) expect((await send()).status).toBe(204); + expect(await rows()).toEqual([]); expect(outboundCalls).toBe(0); + }, 30_000); + it("existing limiter admits one count, silently drops the rest, and leaves ordinary updates available", async () => { + await start({ rateLimit: true }); + for (let i = 0; i < 3; i++) expect((await send()).status).toBe(204); + expect(await rows()).toEqual([{ day, gap: 31, reports: 1 }]); + expect((await update()).status).toBe(200); + }, 30_000); + it.each([{ binding: false, status: 204 }, { migrate: false, status: 503 }])("optional D1 unavailable stays isolated: %j", async ({ status, ...options }) => { + await start(options); expect((await send()).status).toBe(status); expect(outboundCalls).toBe(0); + expect((await update()).status).toBe(200); + }, 30_000); + it("prunes old days during ingestion and scheduled cleanup, including idle histograms", async () => { + await start(); + await db.prepare("INSERT INTO installation_counts VALUES (?, 31, 9), (?, 31, 8)").bind(day - 90, day - 89).run(); + await send(); const expected = [{ day: day - 89, gap: 31, reports: 8 }, { day, gap: 31, reports: 1 }]; expect(await rows()).toEqual(expected); + await db.prepare("INSERT INTO installation_counts VALUES (?, 31, 9)").bind(day - 90).run(); + const worker = await runtime.getWorker(); await worker.scheduled({ cron: "0 0 * * *" }); expect(await rows()).toEqual(expected); + }, 30_000); +}); + +it("private documented SQL counts roots once for 1/7/30/31-day and calendar windows", () => { + const sqlite = new DatabaseSync(":memory:"); + try { + sqlite.exec(migration); + // Root labels exist only in this synthetic oracle, never in the histogram. + const claims = [Array.from({ length: 61 }, (_, i) => day - 60 + i), + [day - 15], [day - 40, day - 5], [day - 31, day], [day - 29, day - 1], [day - 100, day - 60]]; + const insert = sqlite.prepare("INSERT INTO installation_counts VALUES (?, ?, 1) ON CONFLICT(day,gap) DO UPDATE SET reports=reports+1"); + for (const dates of claims) for (let i = 0; i < dates.length; i++) insert.run(dates[i], i ? Math.min(31, dates[i] - dates[i-1]) : 31); + const doc = readFileSync("docs/installation-counts.md", "utf8"); + const sql = [...doc.matchAll(/```sql\n([\s\S]*?)```/g)].map((match) => match[1]); + for (const [start, end] of [[day, day], [day-6, day], [day-29, day], [day-30, day], [day-30, day-3]]) { + const expected = claims.filter((dates) => dates.some((d) => d >= start && d <= end)).length; + expect(sqlite.prepare(sql[0]).get({ S: start, E: end }).reporting_installation_state_roots).toBe(expected); + } + expect(sqlite.prepare(sql[1]).all()).toEqual([1, 7, 30].map((n) => ({ days: n, reporting_installation_state_roots: claims.filter((dates) => dates.some((d) => d >= day-n+1 && d <= day)).length }))); + } finally { sqlite.close(); } +}); + +it("migration constrains integers and at most one aggregate row per day/gap", () => { + const sqlite = new DatabaseSync(":memory:"); + try { + sqlite.exec(migration); + for (const values of ["-1,1,1", "100000000,1,1", "1,0,1", "1,32,1", "1,1,0", "1,1,9007199254740992", "1.5,1,1", "1,1.5,1", "1,1,1.5"]) expect(() => sqlite.exec("INSERT INTO installation_counts VALUES (" + values + ")")).toThrow(); + sqlite.exec("INSERT INTO installation_counts VALUES (1, 1, 1)"); + expect(() => sqlite.exec("INSERT INTO installation_counts VALUES (1, 1, 1)")).toThrow(); + } finally { sqlite.close(); } +}); diff --git a/test/installation-count.test.ts b/test/installation-count.test.ts new file mode 100644 index 0000000..d3a21a6 --- /dev/null +++ b/test/installation-count.test.ts @@ -0,0 +1,97 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import worker from "../src/index.js"; +import type { Env } from "../src/env.js"; +import { INSTALLATION_COUNT_HEADER, INSTALLATION_COUNT_USER_AGENT, parseInstallationCount } from "../src/installation-count.js"; +afterEach(() => { vi.unstubAllGlobals(); vi.restoreAllMocks(); }); +const day = 20_000; +const now = day * 86_400_000; +it("ordinary update never touches optional stalled D1 and advertises one authoritative day", async () => { + vi.stubGlobal("caches", { default: { match: async () => Response.json({ version: "2026.10.28" }) } }); + const batch = vi.fn(() => new Promise(() => {})); + const db = { batch } as unknown as D1Database; + const env: Env = { TELEMETRY: { writeDataPoint() {} }, INSTALLATION_COUNTS: db }; + const request = new Request("https://telemetry.example/api/latest-version"); + const clock = vi.spyOn(Date, "now").mockReturnValue(now); + const response = await worker.fetch(request, env); + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ version: "2026.10.28" }); + expect(response.headers.get(INSTALLATION_COUNT_HEADER)).toBe("1;20000"); + expect(response.headers.get("cache-control")).toBe("no-store"); + expect(clock).toHaveBeenCalledTimes(1); + expect(batch).not.toHaveBeenCalled(); +}); +it("captures the post-lookup server day once when npm lookup crosses midnight", async () => { + let finishLookup!: () => void; + let started!: () => void; + const lookupStarted = new Promise((resolve) => { started = resolve; }); + const lookupGate = new Promise((resolve) => { finishLookup = resolve; }); + vi.stubGlobal("caches", { default: { + match: async () => { started(); await lookupGate; return undefined; }, + put: async () => undefined, + } }); + vi.stubGlobal("fetch", async () => Response.json({ version: "2026.10.28" })); + const env: Env = { TELEMETRY: { writeDataPoint() {} }, INSTALLATION_COUNTS: {} as D1Database }; + const request = new Request("https://telemetry.example/api/latest-version"); + const clock = vi.spyOn(Date, "now").mockReturnValue(now + 86_400_000 - 1); + const pending = worker.fetch(request, env); + await lookupStarted; + expect(clock).not.toHaveBeenCalled(); + clock.mockReturnValue(now + 86_400_000); + finishLookup(); + const response = await pending; + expect(response.headers.get(INSTALLATION_COUNT_HEADER)).toBe("1;20001"); + expect(response.headers.get("cache-control")).toBe("no-store"); + expect(clock).toHaveBeenCalledTimes(1); +}); + +it("count write failure is one attempt even if a commit preceded the lost acknowledgement", async () => { + const prepare = vi.fn(() => ({ bind() { return this; } })); + const batch = vi.fn().mockRejectedValue(new Error("synthetic lost acknowledgement")); + const env: Env = { TELEMETRY: { writeDataPoint: vi.fn() }, INSTALLATION_COUNTS: { prepare, batch } as unknown as D1Database }; + const request = new Request("https://telemetry.example/api/latest-version", { method: "POST", headers: { [INSTALLATION_COUNT_HEADER]: "1;20000;31", "user-agent": INSTALLATION_COUNT_USER_AGENT } }); + vi.spyOn(Date, "now").mockReturnValue(now); + const response = await worker.fetch(request, env); + expect(response.status).toBe(503); + expect(await response.text()).toBe(""); + expect(batch).toHaveBeenCalledTimes(1); + expect(env.TELEMETRY.writeDataPoint).not.toHaveBeenCalled(); +}); + +it.each([undefined, "1", "00", "invalid"])("rejects stalled/present bodies without obtaining a reader (length %s)", async (length) => { + const headers = new Headers({ [INSTALLATION_COUNT_HEADER]: "1;20000;31", "user-agent": INSTALLATION_COUNT_USER_AGENT }); + if (length !== undefined) headers.set("content-length", length); + const request = new Request("https://telemetry.example/api/latest-version", { method: "POST", headers, body: new ReadableStream({}), duplex: "half" } as RequestInit); + const getReader = vi.spyOn(request.body!, "getReader").mockImplementation(() => { throw new Error("must not read body"); }); + const prepare = vi.fn(); + const env: Env = { TELEMETRY: { writeDataPoint: vi.fn() }, INSTALLATION_COUNTS: { prepare } as unknown as D1Database }; + vi.spyOn(Date, "now").mockReturnValue(now); + expect((await worker.fetch(request, env)).status).toBe(400); + expect(getReader).not.toHaveBeenCalled(); + expect(prepare).not.toHaveBeenCalled(); + expect(env.TELEMETRY.writeDataPoint).not.toHaveBeenCalled(); +}); + +it("revalidates the server-day window after the asynchronous quota gate", async () => { + const prepare = vi.fn(); + const env: Env = { TELEMETRY: { writeDataPoint: vi.fn() }, INSTALLATION_COUNTS: { prepare } as unknown as D1Database }; + const request = new Request("https://telemetry.example/api/latest-version", { method: "POST", headers: { [INSTALLATION_COUNT_HEADER]: "1;20000;31", "user-agent": INSTALLATION_COUNT_USER_AGENT } }); + vi.spyOn(Date, "now").mockReturnValueOnce(now).mockReturnValue(now + 2 * 86_400_000); + const response = await worker.fetch(request, env); + expect(response.status).toBe(400); + expect(prepare).not.toHaveBeenCalled(); + expect(env.TELEMETRY.writeDataPoint).not.toHaveBeenCalled(); +}); + +describe("installation count header", () => { + it.each([1, 7, 30, 31])("accepts bounded gap %i on today and yesterday", (gap) => { + for (const d of [day, day - 1]) expect(parseInstallationCount(`1;${d};${gap}`, now)).toEqual({ day: d, gap }); + }); + it.each([ + null, "", "1;20000;0", "1;20000;32", "1;20000;-1", "1;20000;1.0", "1;20000;01", + "1;20000;1e1", "1;020000;1", "2;20000;1", "1;20001;1", "1;19998;1", "1;-1;1", + "1;20000;1;id", "1;20000;1, 1;20000;1", "1;20000;1\n", " 1;20000;1", "1;20000;1 ", + "1;20000;١", "1;20000;+1", "1;" + "9".repeat(100_000) + ";1", + ])("ignores malformed or out-of-window header %#", (value) => { + expect(parseInstallationCount(value, now)).toBeUndefined(); + }); +}); diff --git a/test/public-surface.test.ts b/test/public-surface.test.ts index 40244dd..d039dd8 100644 --- a/test/public-surface.test.ts +++ b/test/public-surface.test.ts @@ -31,6 +31,10 @@ it.each(["/", "/index.html"])("serves privacy information without a dashboard at const html = await response.text(); expect(html).toContain("How to turn it off"); expect(html).toContain("Approximate location"); + expect(html).toContain("Aggregate-only installation counts"); + expect(html).toContain("observed reporting installation-state roots, not people or engagement"); + expect(html).toContain("optional production binding and cleanup trigger are absent"); + expect(html).toContain("Stops anonymous feature statistics and the companion installation-count POST"); expect(html).toContain("outcomes on by default"); expect(html).toContain("DO_NOT_TRACK controls feature statistics, not update outcomes"); expect(html).toContain("Nix mode suppress outcomes"); diff --git a/wrangler.local.jsonc b/wrangler.local.jsonc new file mode 100644 index 0000000..8035337 --- /dev/null +++ b/wrangler.local.jsonc @@ -0,0 +1,16 @@ +{ + "$schema": "./node_modules/wrangler/config-schema.json", + "name": "openclaw-telemetry-local-proof", + "main": "src/index.ts", + "compatibility_date": "2026-08-01", + "compatibility_flags": ["nodejs_compat"], + "observability": { "enabled": false, "logs": { "enabled": false, "invocation_logs": false } }, + "d1_databases": [{ + "binding": "INSTALLATION_COUNTS", + "database_name": "installation-counts-local", + "database_id": "00000000-0000-0000-0000-000000000001", + "migrations_dir": "migrations", + "remote": false + }], + "triggers": { "crons": ["0 0 * * *"] } +}