Repository navigation
feat(members): append-only crew ledger with pluggable contributors - #11964
chenmingwei23 wants to merge 2 commits into
Conversation
3200dbc to
c5abffa
Compare
A crew member's state lives in several places (trust files, activity files, in-memory registries), each read separately and reassembled by the Members page, which refetches the whole roster on every change. Nothing outside the gateway process can contribute state about a member, so any plugin or app that has something to say about a crew has to be written into the gateway itself.
Members page changes appear without a refresh and survive restarts with a reason (a patrol interrupted by a restart reads "interrupted", not "armed"). Any app or adapter, in any language, can read a member's history, record its own facts, and put a view on the member's card through one wire contract, while the gateway stays the only writer and enforces ownership and quota at the boundary.
Per-member append-only ledger and projections, piloted on the Members page (`src/kiro_crew/eventlog/`, `docs/system-specs/modules/member-event-log.md`). One writer, contiguous seq, fsync before announce, synthesized closers written on cold load, whole-value pushed frames, reconnect as truncate then reseed. Presence stays live and is joined at render time.
Contribution protocol (`docs/system-specs/modules/contribution-protocol.md`): manifest-declared `contributions` checked on every call; catch-up `GET` plus a WebSocket delta channel with the contiguity rule; namespaced appends with size and budget limits; whole-value projection publish with higher-seq-wins; teardown withdraws views and keeps events; the drawer renders any contributed view generically from a small declarative schema.
An adapter that hosts an unmodified plugin written for a foreign plugin runtime was built and validated against this protocol on a separate branch (`feat/dsh-adapter`) and is deliberately not part of this PR; the RFC keeps it as the worked example of what an adapter is.
`kirocrew app import` (`apps/plugin_import.py`, `docs/system-specs/modules/plugin-import.md`) converts a manifest-declared plugin bundle (skills, MCP servers, hooks, connectors) into an installable app; `docs/system-specs/modules/harness-plugin-mapping.md` records which foreign plugin kinds reach which extension point and which do not.
`docs/request-for-change/rfc-crew-native-append-only.md` states the model in plain language and answers the crews-managing-crews question: storage follows the subject, visibility follows grants, the tree is a view.
One platform defect fixed on the way: a builtin's manifest snapshot dropped `contributions` (the repo's own round-trip guard was red on it).
On the integrated branch: backend 893 passed / 2 skipped across the eventlog, contribution protocol, members, plugin import, app manifest and bridges, error-code contract, ws scoping and route-table suites; frontend 148 passed (members page, projection store, contributed card); `npx tsc -b` clean; adapter Node suite 48/48; flake8 clean on touched files; `scripts/docs-lint.sh` passes.
End to end in isolated pods: a star toggled outside the browser lands as one pushed frame with zero refetch; a seeded `patrol/started` plus restart writes `patrol/stopped {reason: interrupted}` exactly once; a reference contributor completes token exchange, catch-up, subscribe, three appends, fold and publish, and after being stopped while two events were appended by another client, resumes through `GET ?after=` and republishes; three real published plugin bundles import, enable and register their MCP server into the pod's agent config.
Not run: the full CI suite. That is what this draft is for.
Known gaps, listed in the RFC's last section and in each commit body: `slot/opened` fires only for member-created slots, so a member never driven by an agent folds to zeros; the app token travels in the query string and should move to a header before this leaves draft; the protocol has no version field in its handshake yet; only the `member` kind exists, and the plugins that matter most fold session events, so `session` is the next kind and should share this envelope with the ACP-replay direction in #8862. The branch is large because the protocol is only provable with a real contributor on the other side (the reference client in `test/contrib_protocol_demo.py`); it can be split into the ledger, the protocol and the importer if reviewers prefer.
c5abffa to
51ee5b0
Compare
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS New public wire contract ships with two self-acknowledged defects — query-string token, unversioned handshake — that ossify once external contributors bind to it. Watch
Suggestions
[DESIGN-REVIEWED] 619879c |
UX Review (Fable 5) — 🟡 CONCERNSUX-level review of UX-Verdict: CONCERNS Contributed cards read as content of unknown origin — the blind reader could not tell what they are, who put them there, or if they're real. Watch
[UX-REVIEWED] 619879c |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The contribution protocol ships with zero in-tree contributors — its only consumer is deliberately left on an unmerged branch — plus a kind registry holding one kind. Not justified as shipped
What this change shipsInventory (10 items, capped) — 8 justifiedIntent: make a crew member's state one append-only, tamper-fenced record that external apps can read and contribute to — an ADDITION (the "unfenced log" fixed exists only in the PR's own history, not on base).
The member-projection push shape is covered on the base by [FIRST-PRINCIPLES-REVIEWED] 619879c |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsFINDING — the §6 teardown deletion frame omits FINDING — src/kiro_crew/apps/teardown.py:169 — the deletion frame [OPUS-REVIEWED] 619879c Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
51ee5b0 to
09e6cb9
Compare
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/eventlog/grants.py:74 -- contribution authorization blocks the event loop BLOCKING -- src/kiro_crew/dashboard/handlers/eventlog.py:239 -- failed appends consume quota BLOCKING -- src/kiro_crew/eventlog/contrib.py:395 -- projection rows have no population bound BLOCKING -- src/kiro_crew/apps/plugin_import.py:834 -- malformed keywords crash the import command BLOCKING -- src/kiro_crew/eventlog/service.py:227 -- cached logs omit commits from other processes BLOCKING -- src/kiro_crew/dashboard/ws.py:428 -- disable can race a subscription grant (origin: validation) BLOCKING -- src/kiro_crew/eventlog/contrib.py:460 -- failed row deletion is published as durable (origin: validation) [BLOCK-MERGE] 619879c Adjudication (Opus 4.8) — is blocking on each finding proportionate?All four adjudicable findings cite AUTOSDE anchors that carry F1 — F2 — F3 — F7 — The rule flag outranks proportionality weighing in every case (reason code Fenced findings. F4: False positive or not applicable? A repository writer can comment: |
|
First Principles item 7, the one-kind The finding's count is right on the write side and I reproduced it: one What the subtraction would cost is on the read side, which the count does not cover. Four call sites in three modules resolve a kind through the table:
So I read this as a real observation whose remedy costs more than the generalization does. The finding's own clearing condition is "a second kind is registered by a merged change, or the registry is inlined". The RFC's Items 6 and 8 from the same review are adopted on |
09e6cb9 to
bbf91d1
Compare
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
…-log root Extracted from #11964 as the ledger half: the member event log, its projections, the Members page live updates, and the move of the per-member log under the already-fenced crew-log/ root. The contribution protocol and the plugin importer are deliberately NOT here -- they are separate PRs, which is what both reviewers asked for.
|
Closing in favour of #9907, which implements the same change. Evidence. Same title, same author, 83 of 93 files shared (64 after excluding common-surface files), including the crew-log envelope docs and the RFC itself. Why this one and not #9907. #9907 is the newer cut of the same umbrella and is the one to carry it forward. This is not a judgement on the work -- the two changes were written independently and this one is not the weaker of the pair, it is the one that does not need to land twice. If you disagree, or if #9907 turns out to miss something this branch handles, say so and I will reopen this. From a full relationship audit of every open PR against |
Same commit as #9907, raised from a branch in this repository so the review lanes can post their findings. On a fork the reviewers hold a read-only token for this repo, so their comment write fails: the check-run reports a count and the detail comment never appears, which leaves nothing to adjudicate. #9907 stays open as the original; this branch carries the identical tree at
3200dbc1f9.What is the problem?
A crew member's state lives in several places (trust files, activity files, in-memory registries), each read separately and reassembled by the Members page, which refetches the whole roster on every change. Nothing outside the gateway process can contribute state about a member, so any plugin or app that has something to say about a crew has to be written into the gateway itself.
A second problem sits inside the first, and it is what the top commit fixes. The per-member log was written to
<data home>/members/<slug>/log.jsonl, which is not fenced:is_sensitive_pathanswers False for it, and_CREW_HIDDEN_LEAVESdoes not hide it from a sandboxed subprocess. So an agent could rewrite the append-only record that dispatch trust reads. The crew log it duplicates,crew-log/log.jsonl, is fenced on both gates, which is the whole reason that root exists.Why this issue matters to the user
Members page changes appear without a refresh and survive restarts with a reason (a patrol interrupted by a restart reads "interrupted", not "armed"). Any app or adapter, in any language, can read a member's history, record its own facts, and put a view on the member's card through one wire contract, while the gateway stays the only writer and enforces ownership and quota at the boundary.
An audit record an agent can edit is not an audit record. A rewritten one cannot be detected afterwards, which is the property that makes keeping it worthwhile.
How our fix solves it
Per-member append-only ledger and projections, piloted on the Members page (
src/kiro_crew/eventlog/,docs/system-specs/modules/member-event-log.md). One writer, contiguous seq, fsync before announce, synthesized closers written on cold load, whole-value pushed frames, reconnect as truncate then reseed. Presence stays live and is joined at render time.Contribution protocol (
docs/system-specs/modules/contribution-protocol.md): manifest-declaredcontributionschecked on every call; catch-upGETplus a WebSocket delta channel with the contiguity rule; namespaced appends with size and budget limits; whole-value projection publish with higher-seq-wins; teardown withdraws views and keeps events; the drawer renders any contributed view generically from a small declarative schema.The storage half is now the crew log rather than a second mechanism beside it.
memberjoinscrewandsessionas a unit kind insrc/kiro_crew/crew_log/, so the log lands at<data home>/crew-log/members/<store name>/log.jsonland inherits the existing fence by construction instead of needing a parallel one.MemberLogbecame an adapter overLedger, usingappend,iter_fromfor the protocol's forward catch-up,pagefor history andlast_seqfor the cursor.SCHEMA_VERSIONstays 1 because the schema's own rule is that additive changes never bump it, andTYPE_OWNERSHIP[KIND_CREW]already owned the four namespaces this reserves (member,activity,slot,patrol).One behaviour changed in a shipped module rather than being added to it:
check_ownershiprejected a guestapp:type by comparing the kind againstKIND_CREW. A contributor writing to a member log needs those types, so the rule now readsKIND_SOURCE_PREFIXES[kind]: a kind accepts a guest's type exactly when it accepts that guest'ssrc. Session logs still refuse them, because they declare no guest prefix. That removes a hardcoded special case instead of adding one.kirocrew app import(apps/plugin_import.py,docs/system-specs/modules/plugin-import.md) converts a manifest-declared plugin bundle (skills, MCP servers, hooks, connectors) into an installable app;docs/system-specs/modules/harness-plugin-mapping.mdrecords which foreign plugin kinds reach which extension point and which do not.docs/request-for-change/rfc-crew-native-append-only.mdstates the model in plain language and answers the crews-managing-crews question: storage follows the subject, visibility follows grants, the tree is a view.One platform defect fixed on the way: a builtin's manifest snapshot dropped
contributions(the repo's own round-trip guard was red on it).An adapter that hosts an unmodified plugin written for a foreign plugin runtime was built and validated against this protocol on a separate branch (
feat/dsh-adapter) and is deliberately not part of this PR; the RFC keeps it as the worked example of what an adapter is.What tests we did
The fence is a test, not a claim:
is_sensitive_pathanswers True forcrew-log/members/<fold>/log.jsonland False for the oldmembers/<slug>/log.jsonlas the control, and the test goes red ifcrew-logis unnamed on the tool gate.Each translation the adapter performs is pinned by mutation: dropping the
seq - 1translation reds 7 tests, storing a contributed type without itsapp:prefix reds 1, dropping the bad-payload-to-ValueErrortranslation reds 1.Backend: 489 pass across the eventlog, contribution protocol, members, dm-thread, wiring and docs-reference suites, plus 1032 across all 16
crew_logfiles.eventlog/log.pycoverage went 81% to 96%;eventlog_hooks.py72.7% to 85%;dashboard/ws.pygained 39 covered statements, measured by running the ws suites with and without the new file (178 missing to 139). The PR's own_handle_eventlog_framehad no test at all before this.Frontend 148 pass (members page, projection store, contributed card);
npx tsc -bclean; adapter Node suite 48/48; flake8 and mypy clean on touched files;scripts/docs-lint.shpasses.End to end in isolated pods: a star toggled outside the browser lands as one pushed frame with zero refetch; a seeded
patrol/startedplus restart writespatrol/stopped {reason: interrupted}exactly once; a reference contributor completes token exchange, catch-up, subscribe, three appends, fold and publish, and after being stopped while two events were appended by another client, resumes throughGET ?after=and republishes; three real published plugin bundles import, enable and register their MCP server into the pod's agent config.Two behaviours the fold changed, both measured rather than assumed. A damaged committed line no longer kills the whole file. The store skips it inside a segment on purpose, so the two tests asserting
LogCorruptthere were asserting the old rule;LogCorruptnow covers an unparseable header, without which nothing in the file is attributable. And a bad payload arrived asLedgerErrorwhere one caller distinguishes a client's bad append from a server fault by type, so the adapter carries the store's coded refusal back asValueErrorwhile the store keeps making the judgement.Any other suggestions on the work
Two behaviour bounds this change carries, itemized here because a reader should
not have to find them in the diff. The duplicate-session probe reads the member's
own ACTIVITY_RECORD events rather than scanning a file, and that scan is
UNBOUNDED on purpose: the log is already materialised in memory by the same read,
so a cap saved no I/O and did cost a miss (a session resumed after enough later
events deduped against nothing and wrote a second participation record). And the
contributed-projection wire now carries
stateVersionon the push frame and inthe roster baseline, because the server accepts a refold whose version rose even
when its seq did not advance, and a browser applying seq-wins alone dropped
exactly that frame.
Known gaps, listed in the RFC's last section and in each commit body:
slot/openedfires only for member-created slots, so a member never driven by an agent folds to zeros; the app token travels in the query string and should move to a header before this leaves draft; the protocol has no version field in its handshake yet; the plugins that matter most fold session events, sosessionsharing this envelope with the ACP-replay direction in #8862 is the next step. The branch is large because the protocol is only provable with a real contributor on the other side (the reference client intest/contrib_protocol_demo.py); it can be split into the ledger, the protocol and the importer if reviewers prefer.Disclosed rather than widened:
eventlog/contrib.pykeeps an external projection store at<data home>/eventlog/contrib, also unfenced. It holds derived projections rather than the audit record, so it is a different class of artifact and a different decision; it is not changed here.Screenshots
All captured at the current head via the PR's own harness (
website/scripts/capture-members-patrol.mjs).The one visible behaviour this PR changes: the Crew Members drawer when a patrol was interrupted by a gateway restart. It reads a sentence ("Interrupted by a restart."), not a raw token, and the patrol still lists under Wake sources as a muted "Patrol stopped." row.
That row is the fix for a contradiction UX Review found in an earlier revision of this branch: the empty-wake branch fired whenever the patrol was not ACTIVE, so a durably-stopped patrol rendered "Patrol stopped. Interrupted by a restart." directly above "Nothing wakes this member automatically." A stopped patrol still IS a wake source, just not an armed one, so it lists muted the way a disabled job does.
The reshaped Crew Members page at rest -- roster plus the Crew summary with the Auto patrol detail:
A patrol reported armed by the wake projection before any loop record lands: the drawer reads "Patrolling" and lists the patrol as a wake source (no interval yet), instead of a lit icon over "No patrol scheduled":
The contributed-view cards the drawer renders once an app publishes on a member's log. Each is drawn from the value plus a declarative schema (no contributor code runs), and they mount BELOW the member's own Configuration so a chatty app adds to the drawer's tail rather than displacing it. A populated keyvalue card (final-segment labels,
last/total, not the rawpings.lastselector) and its empty-body state:The remaining body kinds, one still each: a badge, a text body, a list sliced past its 20-row cap with the "+N more" marker (the same honesty the member's own activity counts get), and a table with its declared columns:
Behaviour is also covered by the vitest suites (MembersPage.test.tsx, memberProjectionStore.test.ts, useMemberProjection.test.tsx, ContributedViewCard.test.tsx).