Skip to content

docs(rfc): propose app usage stats for app builders - #18855

Open
FlorentLa wants to merge 1 commit into
kirodotdev:mainfrom
FlorentLa:docs/rfc-app-usage-stats
Open

FlorentLa wants to merge 1 commit into
kirodotdev:mainfrom
FlorentLa:docs/rfc-app-usage-stats

Conversation

@FlorentLa

Copy link
Copy Markdown

Problem / Motivation

Goal: Propose a design that shows the builder of a published Kiro Crew app its installs, active installs and version distribution over time, on the app's listing.

A builder gets no usage feedback after publishing; an official listing shows GitHub stars and nothing else. The only app-level signal leaving a host is the official-app install receipt, which counts install events, carries Kiro Crew's release rather than the app's, never reports for external registries, and has no consumer in this repository. #18761 asks for these stats on any registry, and its triage reply asked for an RFC under docs/request-for-change/.

Why it matters

Without these numbers a builder cannot tell whether a release was adopted, whether an old version still needs support, or when a breaking change is safe.

Not a goal

What changed (motivation → approach → change)

Adds docs/request-for-change/rfc-app-usage-stats.md as draft, and its row in the index. The document proposes:

  • a weekly presence ping per enabled app, with a token that is domain-separated from the receipt token and rotates every week, so it cannot be joined to install events or the heartbeat;
  • three fields (active, versions, series) and a small-count floor added to the ranking document feat(apps): publish install-count ranking that Discover can sort by #17958 proposes, instead of a second document;
  • a Usage section on the app detail page, read only from the listing's own registry;
  • external registries reporting only to a collector URL the owner confirmed, stored where the agent cannot write it, with a separate governance scope.

Eligibility fails closed: local, dev and legacy installs with unknown provenance never report. Phases are PR-sized with testable exit criteria, and the five decisions a maintainer must make, including who owns the official producer, gate the phases that send or render anything. Acceptance is requested from a maintainer; the status flips to accepted when one records it in the document.

Backwards compatibility

Compatible: documentation only; no code, config or wire change.
Removes nothing: the diff adds a file and a table row.

Tests

N/A: documentation only. python3 scripts/docs_lint.py passes.

Manual verification

Every claim about current code was checked against d32c084921, the commit named in the front matter.

Related Issues

Refs #18761. Related: #17520, #17958, #9490, #12883, #1440.

no linked issue: this is the design for #18761, which stays open for the implementation and for its finer-grained usage part.

Checklist

  • At most two commits, with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality (N/A, docs only)
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@FlorentLa
FlorentLa requested review from a team and KiroCrewCaptain as code owners October 10, 2026 12:53
@FlorentLa
FlorentLa requested a review from buluoray October 10, 2026 12:53
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 6440fb5e6dc99120c82ff32a463b3ca0e229a603 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ⏭️ skipped

Revision 6440fb5e6dc99120c82ff32a463b3ca0e229a603 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 6440fb5e6dc99120c82ff32a463b3ca0e229a603 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this PR adds only a draft RFC and its index row, and every phase that sends or renders anything waits on a named maintainer decision.

Notes

  • §2 sends presence to a pinned endpoint so a config.json edit cannot redirect it, but the receipt still goes to beacon_endpoint and now carries av. That leaves two different trust rules on one egress family. Decide in Phase 2 whether to pin both, not leave it optional.

[DESIGN-REVIEWED] 6440fb5

Description read

[DESCRIPTION-READ] 5c9b780da7470be72e2e8028f5311f37fe26634a6b654e52867998a8ba9ce1f6

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed df7c609f50bbcc214e5053b8c417d4c4c26c7150 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] df7c609

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 6440fb5e6dc99120c82ff32a463b3ca0e229a603 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 6440fb5

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@FlorentLa
FlorentLa force-pushed the docs/rfc-app-usage-stats branch from 10a40da to f12825f Compare October 10, 2026 13:35
@FlorentLa

Copy link
Copy Markdown
Author
  • Schema example breaks its own floor (rfc-app-usage-stats.md:148)

fixed in f12825f: windowedInstalls and newInstalls are null in the example, and 0.3.10 is hidden beside the folded other bucket in both versions maps, so no hidden count is recoverable by subtraction.

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 10, 2026
@FlorentLa

Copy link
Copy Markdown
Author
  • Send-time rule does not deliver the few-hours-a-day claim (rfc-app-usage-stats.md:111)

fixed in f12825f: the target is now drawn within the first 8 hours of cumulative weekly uptime, counted across restarts, so any gateway up 8 hours in a week reports (two hours a day reports by Thursday). The claim and the Phase 2 exit criterion state that bound, and the no-send case is explicit.

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 10, 2026
@FlorentLa

Copy link
Copy Markdown
Author
  • registry_trust.json described as agent-unreadable (rfc-app-usage-stats.md:132)

fixed in f12825f: §2 now says agent-unwritable, read-only in every sandbox mode (_CREW_READONLY_LEAVES in sandbox.py), and the Agent-driven egress bullet rests the control on write protection alone; the confirmed collector URL is not a secret.

@FlorentLa

Copy link
Copy Markdown
Author
  • Reword agent-unreadable to agent-unwritable

fixed in f12825f: both §2 and the Agent-driven egress bullet now say agent-unwritable and argue from write protection only.

@FlorentLa
FlorentLa force-pushed the docs/rfc-app-usage-stats branch from f12825f to 3b44485 Compare October 10, 2026 14:11
@FlorentLa

Copy link
Copy Markdown
Author
  • Phase 1 changes the privacy text ahead of the send it describes

fixed in 3b44485: Phase 1 now adds only the metrics.md contract, in a section marked proposed and not yet sent, and its exit criteria forbid changing README.md, the existing metrics.md sections or the privacy copy. Every text the send contradicts (README telemetry section, "Default-ON with six suppressions", the governance.py and _SECRET_FILE comments, the security_posture.py justification, and the privacyDisclosure strings in all locales) is a Phase 2 exit criterion, landing with the code.

@FlorentLa

Copy link
Copy Markdown
Author

fixed in 3b44485: Phase 3 is now marked blocked on #17958. If #17958 is closed instead, Phase 3 first lands that document's schema (schemaVersion, cumulativeInstalls, windowedInstalls) as its own, with the floor from the start. The merge-order note says Phase 1 can merge first and Phase 3 only once #17958 has.

@FlorentLa

Copy link
Copy Markdown
Author
  • Heartbeat shift breaks the 8-hour reporting guarantee (rfc-app-usage-stats.md:111)

fixed in 3b44485: the target is drawn within the first 7 hours of weekly uptime, and a target reached in the 30 minutes after a heartbeat waits for that window to close instead of moving by uptime. The guarantee is stated as 8 hours of uptime in sessions longer than 30 minutes, and the Phase 2 exit criterion matches.

@FlorentLa
FlorentLa force-pushed the docs/rfc-app-usage-stats branch from 3b44485 to 39f617e Compare October 10, 2026 14:44
@FlorentLa

Copy link
Copy Markdown
Author
  • Complementarily hidden count shown as "fewer than N" (rfc-app-usage-stats.md:142)

fixed in 39f617e: §3 now says null means hidden, never below the floor, because the second suppressed count can be at or above it. §6 renders every null as "hidden (small count)", never a number or a bound, and an all-null app as one line, "Too few installs to show". The Phase 3 exit criterion pins it.

@FlorentLa

Copy link
Copy Markdown
Author

CI note for maintainers: this PR changes only docs/request-for-change/rfc-app-usage-stats.md and one row of docs/request-for-change/README.md, so the two red backend tests on earlier heads are unrelated to it:

  • f12825f: test/test_stt_engine.py::test_close_frees_a_model_that_ran_a_sized_decode (Backend Tests 3.12, shard 2), which also failed the Coverage Gate
  • 3b44485: test/test_crew_log_emit.py::test_the_repair_is_queued_under_the_predecessor_not_the_successor (Backend Tests Windows, shard 4)

A fork cannot re-run jobs, so a rerun of the failed jobs on the current head would be appreciated if either recurs.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@FlorentLa
FlorentLa force-pushed the docs/rfc-app-usage-stats branch from 39f617e to 8c68d34 Compare October 10, 2026 15:29
@FlorentLa

Copy link
Copy Markdown
Author
  • config.json called agent-writable though the sandbox seals it (rfc-app-usage-stats.md:104)

fixed in 8c68d34: "(agent-writable)" is gone. §2 and the Agent-driven egress bullet now rest on the real distinction: config.json is read-only in the sandbox but has gateway-side writers that act on requests (dashboard config API, channel handlers, migrations), so a value in it is not an owner assertion, while registry_trust.json is written only by the owner-only route.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 12, 2026
@FlorentLa
FlorentLa force-pushed the docs/rfc-app-usage-stats branch 2 times, most recently from 8ff44bb to ee82da6 Compare October 12, 2026 10:05
@FlorentLa

Copy link
Copy Markdown
Author
  • Client-side floor listed as a privacy mitigation (Design suggestion)

fixed in ee82da6: "Small populations" now names only the producer's floor with complementary suppression and weekly granularity, and states that the client-side minimum floor (§6) only governs what the page displays, since the stats document is public JSON anyone can read without the client.

  • Send-time rule heavy for a best-effort ping (Design suggestion)

kept as is, with the trade-off for the maintainer's call: a random offset per (app, week) sent at the first tick after it drops a week whenever the gateway is not up after the offset, so a machine that runs only early in the week (or a couple of hours a day with a late offset) would miss weeks. An earlier review round measured that gap for an unconditioned draw, and closing it is what the uptime target buys. The persisted state is one cumulative-uptime counter per ISO week, reset when the week changes. If a maintainer prefers the simpler rule and accepts the weaker guarantee, it is a one-paragraph change to the Send time bullet and the Phase 2 exit criterion.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 12, 2026
@FlorentLa
FlorentLa force-pushed the docs/rfc-app-usage-stats branch from ee82da6 to e0497ab Compare October 12, 2026 15:14
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@FlorentLa
FlorentLa force-pushed the docs/rfc-app-usage-stats branch from e0497ab to 78be152 Compare October 12, 2026 16:20
@FlorentLa

Copy link
Copy Markdown
Author
  • "The client-side minimum floor (§6)" points to the wrong section (rfc-app-usage-stats.md:259)

fixed in 78be152: the reference now reads (§3), where the client floor is defined.

  • Phase 2 list misses two "only default-on egress family" texts (rfc-app-usage-stats.md:221)

fixed in 78be152: the Phase 2 exit criterion now names every text making that claim: the governance.py comment, the dashboard/handlers/feedback.py module docstring and the governance.md sentence.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 12, 2026
@FlorentLa
FlorentLa force-pushed the docs/rfc-app-usage-stats branch from 78be152 to ee831e6 Compare October 12, 2026 17:19
@FlorentLa

Copy link
Copy Markdown
Author

CI note for maintainers on 7a472a27: the two red backend shards (Backend Tests 3.12 shard 2, Backend Tests Windows shard 2) fail on the same tree-scan ratchet, test/test_wait_discipline_ratchet.py::test_no_file_gains_a_short_fixed_wait. It flags test/test_surface_deny_notice_sites.py (0 -> 1) and test/test_webhook_permission_requests.py (2 -> 3), neither of which this PR touches: the diff is docs/request-for-change/rfc-app-usage-stats.md and one row of the RFC index. The ratchet reads main's tree as of 63c155a5, so it should fail the same way on any PR rebased onto it.

@FlorentLa

Copy link
Copy Markdown
Author
  • "Nothing the agent can write … redirects a send" is false for the install receipt (rfc-app-usage-stats.md:257)

fixed in 303abf1: the Agent-driven egress bullet now claims only the sends this RFC adds. It says the official install receipt still sends to config.telemetry.beacon_endpoint (apps/install_receipt.py), so an override there redirects receipts, which now also carry av, and that pinning the receipt to the build default is a one-line Phase 2 change if a maintainer wants it. The RFC leaves the receipt's endpoint as it is.

@FlorentLa

Copy link
Copy Markdown
Author

CI note for maintainers on ce723ec6: this PR still changes only docs/request-for-change/rfc-app-usage-stats.md and one row of docs/request-for-change/README.md. The red jobs on this head are unrelated to that diff:

  • Backend Tests (3.12, shard 5): test/test_session_start_gate.py::test_a_cancelled_gate_wait_closes_the_init_scope timed out (1 failed, 25141 passed).
  • Backend Tests (Windows, shard 8): test/test_crew_log_writer.py::test_a_hung_write_holds_the_backlog_and_a_bounded_shutdown_says_so (assert False). Coverage Gate fails as a consequence of these two shards.
  • Dependency License Gate: Dependency review is not supported on this repository. Please ensure that Dependency graph is enabled, a repository setting.
  • Sensitive Change Review: Could not read the PR from the API (CalledProcessError); re-run this job.

A fork cannot re-run jobs, so a rerun of the failed jobs on this head would be appreciated.

@FlorentLa

Copy link
Copy Markdown
Author
  • versionsBelowFloor marks which null buckets are complementary cells (rfc-app-usage-stats.md:141)

fixed in 527f7a0: the field is now hiddenVersions and names every version whose count is not published, whether its bucket is null or folded into other, without counts or a reason, so the list no longer tells a below-floor count from a complementarily hidden one. The example lists 0.3.10 and 0.3.9, and the versions bullet says 0.3.9 is counted in other.

@iamwhatever

Copy link
Copy Markdown
Contributor

Holding on head df7c609 because the branch conflicts with main in docs/request-for-change/README.md (the RFC index row). Please rebase onto main and resolve the row; the review will need to be redone on the new head.

The document itself is a draft RFC with no code change, all checks on this head are green, and the earlier Opus and Design findings were answered with fixes.

Two current-head AI notes look non-blocking, and you may want to fix them while rebasing:

  • Opus 5.5 note: the Phase 2 exit criterion names "the governance.py comment" (rfc-app-usage-stats.md#L200), but that text lives in governance_parts/catalog.py#L270. This is pointer wording only.
  • Design Review note: making install-receipt endpoint pinning a Phase 2 exit criterion. The RFC already presents this as a maintainer option, which is reasonable for a draft.

— crew-pr-reviewer (r133)

Add a draft RFC for showing a published app's builder its installs,
active installs and version distribution over time: a weekly presence
ping with a rotating per-app token, fields and a small-count floor
added to the ranking document proposed in kirodotdev#17958, and a Usage section
on the app detail page. External registries report only to a collector
the owner confirmed.

Refs kirodotdev#18761
@chenmingwei23

Copy link
Copy Markdown
Contributor

Thanks a lot for this contribution, @FlorentLa. We have an RFC covering the same ground in #39107: per-app installs, updates and uninstalls by version, plus sessions and run time under each app. It already borrows from your small-count floor and version normalization ideas. Feel free to comment on that one; we would love your review there.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants