From c6c807777dfdfdc7910a559c8bb0942e2b3b30a8 Mon Sep 17 00:00:00 2001 From: Max Peterson Date: Fri, 23 Oct 2026 15:56:25 -0700 Subject: [PATCH 1/3] add per-chat and per-gadget observation tracking --- .agents/skills/write-gatekeeper/SKELETON.md | 4 + .agents/skills/write-gatekeeper/SKILL.md | 54 +- docs/observers.md | 153 +- .../gatekeeper-google/__tests__/worker.ts | 4 + .../__tests__/workerd/native-sessions.test.ts | 4 + .../__tests__/workerd/sheets-session.test.ts | 4 + .../__tests__/workerd/slides-session.test.ts | 4 + .../__tests__/workerd/worker.ts | 4 + .../workerd/conformance/gatekeeper.ts | 4 + .../__tests__/workerd/cursors.test.ts | 4 + .../__tests__/observer-tracking.test.ts | 566 +++++ .../gatekeeper-test/src/test-gatekeeper.ts | 250 ++- .../typed-storage/__tests__/index.test.ts | 17 +- packages/typed-storage/src/index.ts | 10 + .../__tests__/exports-binding.test.ts | 2 +- .../__tests__/git-binding.test.ts | 2 +- .../__tests__/observer-exclude-scope.test.ts | 2 + .../__tests__/observer-ledger.test.ts | 1861 +++++++++++++++++ .../__tests__/observer-scope-restart.test.ts | 1 + .../observer-verification-failure.test.ts | 397 +++- .../workshop-backend/src/observability.ts | 7 +- .../workshop-backend/src/observer-ledger.ts | 658 ++++++ packages/workshop-backend/src/overseer.ts | 495 +++-- .../src/storage-schema/overseer-storage.ts | 137 +- packages/workshop-backend/src/user.ts | 11 +- packages/workshop-shared/src/gatekeeper.ts | 134 +- plans/kernel-observer-tracking.md | 111 + plans/restricted-data-sharing.md | 2 - 28 files changed, 4558 insertions(+), 344 deletions(-) create mode 100644 packages/integration-tests/__tests__/observer-tracking.test.ts create mode 100644 packages/workshop-backend/__tests__/observer-ledger.test.ts create mode 100644 packages/workshop-backend/src/observer-ledger.ts create mode 100644 plans/kernel-observer-tracking.md diff --git a/.agents/skills/write-gatekeeper/SKELETON.md b/.agents/skills/write-gatekeeper/SKELETON.md index ab5bdbcafc..70008765f4 100644 --- a/.agents/skills/write-gatekeeper/SKELETON.md +++ b/.agents/skills/write-gatekeeper/SKELETON.md @@ -410,6 +410,10 @@ export class MyGatekeeperImpl extends DurableObject // - Strategy D (low-stakes): make both methods no-ops. // - Strategy C (data-set tracking): record observed sets + store verifiers, and route every read // through an authorizeSetObservation helper that sets `excludeObservers` (see SKILL.md). + // - Workshop-tracked (`supportsObserverTracking` in describe()): always throw (a redeploy can + // land between the Workshop's describe() and this call); authorize every read with + // authorizeTrackedObservation(), reporting its `subresources`, and implement + // verifyObserverAccess(observers, ids) instead, checking each observer's verifier the same way. async addObserver(_id: string, user: Fetcher): Promise { let verifier = user as unknown as Fetcher; if (!(await verifier.hasResourceAccess(/* this.ctx.props.resourceId */ "TODO"))) { diff --git a/.agents/skills/write-gatekeeper/SKILL.md b/.agents/skills/write-gatekeeper/SKILL.md index 81b11eed81..d6efa10635 100644 --- a/.agents/skills/write-gatekeeper/SKILL.md +++ b/.agents/skills/write-gatekeeper/SKILL.md @@ -21,13 +21,13 @@ Read `packages/workshop-shared/src/gatekeeper.ts` for the canonical interfaces a 3. **Fine-grained resource granting** — Enable the end user to grant access to agents at fine granularities, in addition to coarse-grained access. For example, a user may want to give an agent access to a specific Google Doc or GitHub repo, rather than granting broad access to everything they can do. This should be straightforward given a capability-based API. That said, broad access should also be allowed when it makes sense. Consider carefully which granularities are meaningful — a Jira gatekeeper might support "whole service", "project", and "issue" granularities, but it would be silly to support granting access to a single field of an issue separately. -4. **Logging & approvals** — Every action the agent or gadget performs must be logged via the `ApprovalQueue` API. Every action with an externally-visible side effect must be submitted via `submitAction()`, and must not actually be performed until `applyAction()` has been called. Read-only observations must call `authorizeObservation()` before returning data to the caller. Reads the gatekeeper makes for its own purposes, returning nothing to the caller, are not observations (see [What counts as an observation](#what-counts-as-an-observation)). +4. **Logging & approvals** — Every action the agent or gadget performs must be logged via the `ApprovalQueue` API. Every action with an externally-visible side effect must be submitted via `submitAction()`, and must not actually be performed until `applyAction()` has been called. Read-only observations must call `authorizeObservation()` (or, under the Workshop-tracked protocol, `authorizeTrackedObservation()`) before returning data to the caller. Reads the gatekeeper makes for its own purposes, returning nothing to the caller, are not observations (see [What counts as an observation](#what-counts-as-an-observation)). 5. **Caching** — When it makes sense, cache remote content in the gatekeeper's DO storage to improve performance when agents or gadgets repeatedly read the same data. Caching also enables a better TypeScript API when the service's underlying API has an inconvenient data shape. For example, Gmail's API for listing threads returns only thread IDs without metadata, requiring a callback for each thread; with caching, the gatekeeper can provide an API that returns rich thread summaries directly, reading from local content synchronized with Gmail as needed. See Phase 2 for implementation guidance. 6. **Simulation** — Actions submitted but not yet applied should be simulated as if they already occurred, to the maximum extent reasonable. If the caller reads back data, it should observe the data as if pending actions had been applied, even though they haven't yet. This allows the agent to continue working without waiting for each approval, and allows the end user to batch-approve a lot of work at once. Simulation may leverage caching (updating the cache on submit, clearing or repopulating it on reject), or it may work by storing pending actions separately and adjusting read results at query time — the latter is arguably cleaner but trickier to implement correctly. See Phase 2 for implementation guidance. -7. **Observer verification** — When a Gadget is shared, collaborators may "observe" data the Gadget previously read through the gatekeeper. The gatekeeper must ensure each collaborator could access that data themselves, via `getVerifier()` / `addObserver()` / `removeObserver()`. The interface methods are mandatory — a gatekeeper won't type-check without them — so include at least minimal versions in Phase 1; but *choosing and implementing the right strategy* is a Phase 2 security concern, like logging/approvals. See [Observer verification](#observer-verification). +7. **Observer verification** — When a Gadget is shared, collaborators may "observe" data the Gadget previously read through the gatekeeper. The gatekeeper must ensure each collaborator could access that data themselves: under the Workshop-tracked protocol (`supportsObserverTracking`) by reporting what each read depends on and answering the Workshop's access checks, or under the legacy protocol via `addObserver()` / `removeObserver()`. `getVerifier()` / `addObserver()` / `removeObserver()` are mandatory — a gatekeeper won't type-check without them — so include at least minimal versions in Phase 1; but *choosing and implementing the right strategy* is a Phase 2 security concern, like logging/approvals. See [Observer verification](#observer-verification). ## Phase 1: Core implementation @@ -175,7 +175,7 @@ Do **not** call `authorizeObservation()` for: Returning a boolean or count *is* data when that is the answer the caller asked for (`isFollowing()`, `areTracksSaved()`, a `null` from `getArea()` meaning "no area"). Log those. -**Exception: observer tracking.** Under observer strategy C (below), `authorizeObservation()` does more than log. Its `excludeObservers` and set-tracking side effects gate which collaborators may see the workspace, and `containsRestrictedData` / `ownerInvitesOnly` change workspace state. A call that carries any of these is a security control, and removing it is a security decision, not a logging cleanup (see the Drive session's "Check Google Drive folder" fences). +**Exception: observer tracking.** Some observation fields do more than log. Under the Workshop-tracked protocol, `subresources` is what collaborators are verified against; under legacy strategy C (below), `excludeObservers` and the set-tracking side effects gate which collaborators may see the workspace; and `containsRestrictedData` / `ownerInvitesOnly` change workspace state. A call that carries any of these (for `subresources`, a non-empty list) is a security control, and removing it is a security decision, not a logging cleanup (see the Drive session's "Check Google Drive folder" fences). It's critically important that you add `ApprovalQueue` to all API operations that interact with the outside world, otherwise the gatekeeper security model is broken. @@ -212,15 +212,20 @@ For concrete examples, see the Google gatekeeper's Google Docs simulation/cache This is responsibility 7. When a Gadget is shared, each non-owner collaborator becomes an **observer** of every gatekeeper bound to the Gadget, and may see data the Gadget previously read. The gatekeeper's job is to refuse — or forward-restrict — observers who couldn't access that data themselves. -Three methods implement this (full JSDoc in `gatekeeper.ts`): +Each resource type uses one of two protocols, chosen by `ResourceDescription.supportsObserverTracking` in its `describe()`: + +- **Workshop-tracked** (flag set). The gatekeeper authorizes every read with `authorizeTrackedObservation()`, reporting the IDs it depends on as `TrackedObservationDescription.subresources`, and answers the Workshop's `Gatekeeper.verifyObserverAccess()` checks using each observer's verifier. The Workshop records the IDs against the chat or gadget whose binding made the read, checks every collaborator who can see that context before authorizing the read, and checks the recorded IDs again on each collaborator's open. The gatekeeper keeps no observer state. Prefer it for new gatekeepers; every existing gatekeeper is expected to migrate to it. See [The Workshop-tracked protocol](#the-workshop-tracked-protocol). +- **Legacy** (flag absent). The gatekeeper tracks observers itself through `addObserver()` / `removeObserver()` and, for strategy C, `excludeObservers`. One workspace may hold connections on either protocol. + +Three methods are mandatory under both (full JSDoc in `gatekeeper.ts`): - `GatekeeperUser.getVerifier()` — mints a `GatekeeperUserVerifier` (a persistent service stub) representing *this* user's account. The overseer mints one per open and **only ever passes it back to a gatekeeper of the same vendor**, so the gatekeeper may trust whatever it learns from it. -- `Gatekeeper.addObserver(id, verifier)` — must **throw** if the user represented by `verifier` is not allowed to observe everything read through this gatekeeper so far. The overseer calls it on **every open by every authorized observer** (re-verification, so revoked access is caught at the next open); cache as needed if the check is expensive. `id` is an opaque, stable per-(user,gadget) string. -- `Gatekeeper.removeObserver(id)` — idempotent; drop a tracked observer. +- `Gatekeeper.addObserver(id, verifier)` — **legacy protocol.** Must **throw** if the user represented by `verifier` is not allowed to observe everything read through this gatekeeper so far. The overseer calls it on **every open by every authorized observer** (re-verification, so revoked access is caught at the next open); cache as needed if the check is expensive. `id` is an opaque, stable per-(user,gadget) string. Never called for a tracked resource. +- `Gatekeeper.removeObserver(id)` — **legacy protocol.** Idempotent; drop a tracked observer. Never called for a tracked resource. #### The verifier "non-standard method" pattern -`GatekeeperUserVerifier` has no methods of its own — it's an opaque token. To actually answer "can this observer access X?", define a vendor-specific interface that **extends `GatekeeperUserVerifier`** with your own methods, implement it on a `WorkerEntrypoint` that queries the service using the **observer's own token**, and cast the `Fetcher` back to that interface inside `addObserver`. The overseer's same-vendor guarantee is what makes the cast safe. +`GatekeeperUserVerifier` has no methods of its own — it's an opaque token. To actually answer "can this observer access X?", define a vendor-specific interface that **extends `GatekeeperUserVerifier`** with your own methods, implement it on a `WorkerEntrypoint` that queries the service using the **observer's own token**, and cast the `Fetcher` back to that interface inside `addObserver` (legacy protocol) or `verifyObserverAccess` (Workshop-tracked protocol). The overseer's same-vendor guarantee is what makes the cast safe. ```typescript // In types/impl: a verifier interface with non-standard methods. @@ -266,9 +271,33 @@ Strategy is chosen **per `Gatekeeper` DO class / binding**, not per package — The **B-vs-C decision** (the "broad binding" lens): use C only when **both** (1) the binding spans sub-resources with distinct ACLs *and* (2) there's a per-observer oracle to check each against. If one ACL covers everything → B. If there's no oracle → A or D. -#### Implementing strategy C +The strategy is the same choice under the Workshop-tracked protocol, where it decides what each observation reports as `subresources` (`addObserver()` only throws; see below): + +- **A** — an owner-only ID (for example, one naming the owner's external user) that `verifyObserverAccess()` grants only to an observer whose verifier is that same user. Keep marking truly sensitive observations `containsRestrictedData: true`. +- **B** — the binding's own resource ID, on every read. +- **C** — the ID of every sub-resource the read revealed, plus the coarse baseline (such as org or workspace membership) that the binding's reads rely on. A read that reveals no sub-resource reports the baseline alone. +- **D** — `[]`. + +#### The Workshop-tracked protocol + +Opt in by returning `supportsObserverTracking: true` from the resource's `describe()`. That field's JSDoc in `gatekeeper.ts` lists what it promises, and the JSDoc on each member it names is the contract (ID rules and limits, the answer format, export semantics). In practice: + +1. **Authorize every observation with `authorizeTrackedObservation()`**, hook deliveries included. Report as `subresources` the IDs of everything the disclosure depends on that might restrict who may see it (`[]` when nothing does), including the coarse requirement the read relies on (see the mapping above): nothing else checks it, since `addObserver()` no longer verifies anything. Make each ID carry everything your `verifyObserverAccess()` needs, and keep its meaning stable across versions; encoding a tuple avoids delimiter collisions (the integration fixture reports `JSON.stringify([resourceUrl, localId])`). If the call fails because the Workshop predates observer tracking, let it propagate: falling back to `authorizeObservation()` would disclose the read without checking anyone. +2. **Implement `verifyObserverAccess(observers, subresources)`.** Check each observer's verifier through your vendor's non-standard methods (as `addObserver()` does) and return the IDs of those whose account can read **every** ID; throw when access can't be determined. Check in bulk where the service allows; because one call covers a read's audience, the check may also read the resource's ACL once with the connection's own credentials and compare each observer's identity. Don't store the verifiers or cache verdicts by observer ID, since the same observer can come back with another account. +3. **Implement `getHistoricalSubresources()`** over the requirements your legacy tracking recorded, hook deliveries included, in the same form as `subresources`. A gatekeeper with no legacy history may return a cursor that ends at once. +4. **Make `addObserver()` throw** (any error); `removeObserver()` may stay a no-op. + +What changes for the gatekeeper: + +- **No observer state.** There are no observed sets, stored verifiers, or `excludeObservers` to keep. The Workshop attributes each read, checks its audience, and closes the races between observations and opens. +- **The switch is all at once and permanent.** Once a connection is tracked, the Workshop refuses every `authorizeObservation()` through it, whichever session made it, so every session, sub-session, cursor, and hook path must use `authorizeTrackedObservation()` before the flag ships; a read that uses `authorizeObservation()` before the Workshop has noticed the upgrade is treated as legacy. Tracking is never downgraded. +- **Registrations made before the switch keep working.** Hook controllers and other persistent stubs stored by an earlier version keep the `props` they were minted with, and their hooks keep firing after the upgrade. Choose the protocol from the binding's current state, never from props or stored fields an earlier version did not write. + +`docs/observers.md` §10 describes the Workshop's side in full. No shipping gatekeeper opts in yet; the integration fixture's tracked mode (`packages/integration-tests/fixtures/gatekeeper-test/src/test-gatekeeper.ts`) is a minimal working example. + +#### Implementing strategy C (legacy protocol) -Route **every data-revealing observation** through a helper that takes the set id(s) the observation reveals, instead of calling `authorizeObservation()` directly: +This applies only to a binding that doesn't advertise `supportsObserverTracking`; a tracked binding reports each set as a subresource instead. Route **every data-revealing observation** through a helper that takes the set id(s) the observation reveals, instead of calling `authorizeObservation()` directly: ```typescript // On the Gatekeeper DO. `setIds` are the data sets this observation reveals. @@ -308,7 +337,7 @@ Key points for C: #### `excludeObservers` semantics (why conservative is safe) -When `authorizeObservation()` is given `excludeObservers`, the overseer **blocks the observation** if any named observer is still authorized, and only lets it proceed (tearing down the observer) if they've already lost access. So erring toward listing an observer is never a leak — at worst it blocks an observation that could in principle have been allowed. The leak-relevant gate is always the live sharing graph, so stale observer state self-heals on the next open. +Legacy protocol only: a tracked connection's observations may not carry `excludeObservers`. When `authorizeObservation()` is given `excludeObservers`, the overseer **blocks the observation** if any named observer is still authorized, and only lets it proceed (tearing down the observer) if they've already lost access. So erring toward listing an observer is never a leak — at worst it blocks an observation that could in principle have been allowed. The leak-relevant gate is always the live sharing graph, so stale observer state self-heals on the next open. ## Hooks (push notifications) @@ -359,7 +388,7 @@ Reviews tend to stop at general correctness. The most important review is of the - All DO classes must appear in `cloudflare.config.ts`'s `migrations` export under `new_sqlite_classes`. - Set a self-destruct alarm in `UserAccount.setCallback()` in case the OAuth flow is never completed. - `authorizeObservation()` may be called *after* fetching data (so the description can include details about what was fetched) but must be awaited *before* returning anything to the caller. -- `getVerifier()` / `addObserver()` / `removeObserver()` are **mandatory** — the gatekeeper won't type-check without them. Even a read-only or push-only gatekeeper needs them (sharing is independent of whether the gatekeeper has actions). Pick a strategy per [Observers](#observer-verification): a low-stakes one can be A or D; otherwise B/C. +- `getVerifier()` / `addObserver()` / `removeObserver()` are **mandatory** — the gatekeeper won't type-check without them. Even a read-only or push-only gatekeeper needs them (sharing is independent of whether the gatekeeper has actions). Pick a protocol and a strategy per [Observers](#observer-verification): a low-stakes one can be A or D; otherwise B/C. A tracked resource's `addObserver()` must throw (see [The Workshop-tracked protocol](#the-workshop-tracked-protocol)), and its `removeObserver()` may be a no-op. ## Reference implementations @@ -369,4 +398,5 @@ Reviews tend to stop at general correctness. The most important review is of the - `packages/gatekeeper-gitlab/` — A **mirror** of gatekeeper-github for a provider with instance configuration (self-hosted URLs, an optional Cloudflare Access service token on upstream requests) and **rotating credentials** (expiring access tokens, single-use refresh tokens refreshed under a mutex). Reads the same `plans/worktrees.md` §3 git contract through `gatekeeper-kit`'s `git-transport`/`git-objects`/`git-diff`. **Observers:** strategy B with a role-aware probe (`membershipGrantsFullRead`): Reporter membership whatever the visibility, since a public project can still hold confidential issues and internal notes, read from `members/all?user_ids[]=` because the project's `permissions` object omits group-inherited access. Its `plans/gitlab-gatekeeper.md` divergence table is a worked example of porting one gatekeeper's API to another provider's vocabulary. - `packages/gatekeeper-supabase/` — **Observers:** strategy C with a per-session context object (`authorizeProjectObservation`) — good when sessions already hold a shared context. - `packages/gatekeeper-linear/` & `packages/gatekeeper-notion/` — **Observers:** strategy C where the page/team session impls are shared between the narrow (B) and broad (C) bindings, threading an `observe` hook through sub-sessions; both also handle one observation revealing multiple sets. -- `packages/workshop-shared/src/gatekeeper.ts` — Canonical interfaces with detailed JSDoc (`getVerifier`, `addObserver`, `removeObserver`, `GatekeeperUserVerifier`, `ObservationDescription.excludeObservers`). +- `packages/integration-tests/fixtures/gatekeeper-test/` — **Observers:** the only Workshop-tracked example: `authorizeTrackedObservation()` with qualified `subresources` on reads and hook firings, a `verifyObserverAccess()` that asks each observer's verifier through a non-standard method, and a `getHistoricalSubresources()` export of its legacy history. +- `packages/workshop-shared/src/gatekeeper.ts` — Canonical interfaces with detailed JSDoc (`getVerifier`, `addObserver`, `removeObserver`, `GatekeeperUserVerifier`, `ResourceDescription.supportsObserverTracking`, `ObservationAuthorizer.authorizeTrackedObservation`, `Gatekeeper.verifyObserverAccess`, `Gatekeeper.getHistoricalSubresources`, `TrackedObservationDescription.subresources` and `ObservationDescription.excludeObservers`). diff --git a/docs/observers.md b/docs/observers.md index 3ed258f711..e3b87ca3af 100644 --- a/docs/observers.md +++ b/docs/observers.md @@ -313,11 +313,12 @@ Logic: - Only a *first-ever* verification rolls anything back (fully — nothing referenced its registrations before the call, and the minted id would otherwise linger unresolvable). A returning observer's registrations are **all** kept, including ones this call added: their - persisted `observerId` is shared with concurrent opens, so a rollback could delete a - registration a concurrent successful open just made and persisted. De-registering is the - fail-open direction (the gatekeeper stops naming them in `excludeObservers`); a spurious - registration only blocks fail-closed until the lazy cleanup at exclusion time or a later - open re-verifies it, and the next successful `addObserver` overwrites its verifier. + persisted `observerId` stays resolvable, and sessions their earlier opens admitted may still + depend on those registrations (opens by one profile are serialized, so none is concurrent). + De-registering is the fail-open direction (the gatekeeper stops naming them in + `excludeObservers`); a spurious registration only blocks fail-closed until the lazy cleanup + at exclusion time or a later open re-verifies it, and the next successful `addObserver` + overwrites its verifier. - A denial ends only this open. Sessions the collaborator already holds are untouched, and keep the access their own opens verified until they next re-open — the lazy-revocation residual described under "Known gaps" below. @@ -354,9 +355,11 @@ collaborator (see `docs/sharing.md`) — so every client's browser reconnects an What it gates on is a **live session** of the affected role, not an entry in the sharing table: severing sessions is all a restart achieves, so a collaborator who isn't connected has nothing to -cut, and a solo workspace is never disturbed. `OverseerImpl.joinSession(kind)` counts them, called -synchronously in each client interface's constructor and released in its `[Symbol.dispose]`, with -the owner counted apart from the two collaborator roles because the owner is never an observer. +cut, and a solo workspace is not disturbed (except by an observer-tracking import, below, whose +restart also replaces client stubs to the facet it aborted). `OverseerImpl.joinSession(kind)` +counts them, called synchronously in each client interface's constructor and released in its +`[Symbol.dispose]`, with the owner counted apart from the two collaborator roles because the owner +is never an observer. Being synchronous is the point: the check can neither be skipped by a failed lookup nor land some unbounded time after the change. The count is deliberately not derived from `#presence`, which a session joins only once its `fetchProfile()` resolves — fine for a roster, fail-open for an access @@ -439,7 +442,7 @@ gadget-minted children alike: needs lifecycle authorization for gadget-authored turns; re-checking person-initiated resumes alone does not cover it. -Five events trigger it: +Six events trigger it: | Event | What grows | |---|---| @@ -448,12 +451,13 @@ Five events trigger it: | A merge that promotes a pending gadget or a pending binding edge into `use` scope | **use** scope, same reason | | `enableHook` on a vendor-backed connection not already in `use` scope | **use** scope — the hook delivers the connection's data into a gadget a `use` session can open (a hook waking a still-provisional gadget stays out of `use` scope until promotion; the merge diff reports that widening) | | Approving a creation queued by the agent's `createExternalResource` | every role's scope — the connection was published without a restart because until now it only simulated its resource, so nobody was verified against the real one. The restart is unconditional on role, since the connection may already be bound | +| A legacy connection moving to Workshop tracking (§10) — its history import | every role's scope — sessions were admitted against its legacy registrations, not the imported requirements. The restart is unconditional, even with only the owner connected: the import aborts the connection's facet, and client capabilities hold stubs to it | -The two roles widen independently, so each trigger passes the role it grew and the restart is -skipped when no collaborator holds it: a new connection is in every `build` collaborator's scope -at once but in no `use` collaborator's until a gadget binds it, and binding one enters `use` scope -having been in `build` scope since it was created. A workspace shared only the other way has nobody -with new verification requirements. +The two roles widen independently, so each trigger except the import passes the role it grew and +the restart is skipped when no collaborator holds it: a new connection is in every `build` +collaborator's scope at once but in no `use` collaborator's until a gadget binds it, and binding +one enters `use` scope having been in `build` scope since it was created. A workspace shared only +the other way has nobody with new verification requirements. The three `use`-scope triggers share one helper (`#restartIfUseScopeWidened`) that compares the effective `use` scope before and after rather than firing on any mutation: most merges promote @@ -588,7 +592,7 @@ For each id in `description.excludeObservers`: classified before anything is torn down, so a blocked observation leaves no teardown behind it; the removals are then all issued together and awaited with a single `Promise.all`. A fresh open's `addObserver` racing one of those `removeObserver` RPCs on the same (observer, - gatekeeper) pair is ordered behind it (`#withObserverGatekeeperLock`; the overseer is the only + gatekeeper) pair is ordered behind it (`#serialized`, keyed by the pair; the overseer is the only caller of either, so ordering its own calls is sufficient), so a fresh registration is never silently undone. An observer put back in scope while the teardown is in flight — a bind plus a fresh open landing inside a removal's window — is admitted for this one observation; that is @@ -891,6 +895,123 @@ This is why the broad bindings split the way they do: --- +## 10. Kernel-owned observer tracking (opt-in connections) + +A connection whose `describe()` returns `supportsObserverTracking: true` is tracked by the +Workshop itself instead of the protocol above (`src/observer-ledger.ts`). Connections without the +flag keep the legacy `addObserver`/`removeObserver`/`excludeObservers` protocol unchanged, and one +workspace can mix the two. No shipping gatekeeper opts in yet. The gatekeeper's side of the +contract (ID rules and limits, `verifyObserverAccess()` answers, the history export) is +documented on those members in `workshop-shared/src/gatekeeper.ts`, and +`plans/kernel-observer-tracking.md` records why it is shaped this way. + +- **A separate authorization method.** A tracked connection's gatekeeper authorizes every read + with `ObservationAuthorizer.authorizeTrackedObservation()`, which a Workshop that predates + observer tracking lacks, so the read fails there rather than being disclosed unchecked. + `authorizeObservation()` refuses a description carrying `subresources`, and every observation + through a tracked connection, whichever protocol its session was minted under. +- **The audit log is the requirement store.** A tracked observation's audit records keep its + `subresources`, and the Workshop strips them from every other observation, so keeping them is + what marks a requirement. The record's `caller` attributes it (`observerContextOf`): the agent's + chat, the calling gadget, the chat a slash command runs in, the gadget a firing hook wakes, or + the workspace context for a hook naming no gadget or a user acting outside any chat (a session + opened from the connection itself, a new chat's first slash command, an import). A gadget caller + persisted without a `gadgetId` is refused. The sparse multi-key `actions.byObserverRequirement` + index holds one key per subresource, `::` (context `w` + for workspace, `c` for chats, `g` for gadgets). Admission lists only the distinct keys a role + reaches through each tracked connection, never the records under them, so repeated reads of the + same data add no admission work. Nothing else stores requirements, so the action log is + append-only: deleting an entry would drop its checks. Client action-log entries omit + `subresources`. +- **Checks.** Account choices stay in `observers.accountChoices`, keyed by connection id under both + protocols. A tracked connection enters each role's verification scope exactly as a legacy one + does: "build" at creation, "use" while a gadget binds it or an enabled hook feeds it. Within that + scope, admission checks every requirement the role reaches: "build" reaches every context, "use" + the workspace and every gadget. Requirements through a connection outside a "use" collaborator's + scope (one nothing binds or hooks) are not checked against them, as under the legacy protocol. + Each observation checks its IDs for every configured observer the data reaches, online or not; + a collaborator who lost access is skipped. Either way the Workshop looks up each observer's + verifier for their chosen account (`getVerifier()` refuses a disconnected or wrong-vendor account + even when there is nothing to check) and asks the connection's `verifyObserverAccess()`: one call + per 10 observers for a read's audience, and batches of 10,000 IDs for admission, stopping at the + first batch that denies. A denial, missing method, failed call, or malformed answer fails closed + for every observer in the call, with no legacy fallback and no permission cached between + operations. +- **Bounded persistence.** Live reads and historical imports split their IDs across audit records + with a conservative 4 MiB ID-array budget (two bytes per UTF-16 code unit plus per-ID overhead), + leaving room for metadata below SQLite's row limit. Each record preserves the original caller + and metadata; admission unions every chunk. Live reads verify all IDs before committing all + chunks and workspace policy flags in one transaction. A failed write rolls everything back, + including action IDs; chat notifications happen only after the transaction commits. +- **Races.** Admissions are serialized per profile. After verifying, admission recomputes what it + must verify and runs the pass again (at most three times) if a connection in its scope appeared, + disappeared or changed protocol, or a requirement it did not check was recorded through one; + reads through connections outside its scope don't count. It rereads requirements only if its + connections changed or an action was recorded meanwhile (only a new action record can add a + requirement), so an admission nothing disturbed reads them once. An observation recomputes who + it reaches after its checks and checks whoever it gained (at most three times). Both decisions + are confirmed in the same synchronous block that publishes them (the observer record; the audit + record), so neither can miss the other. +- **Historical import.** The Workshop does not describe a legacy connection on every use. It looks + for an upgrade on two occasions: + - one of the connection's reads calls `authorizeTrackedObservation()`, which only an upgraded + gatekeeper does. The read came through a session minted under the legacy protocol, so it is + refused after the import (or, if the description doesn't advertise support, refused + outright); + - a collaborator is admitted. Before calling a legacy connection's `addObserver()`, inside the + registration lock and after account selection and verifier lookup (so an upgrade during the + prompt is caught too), admission describes it and imports it if it advertises support, then + reruns against the tracked requirements before publishing its record. A failed description + refuses the binding like any other verification failure, rather than falling back to + `addObserver()`; an upgraded gatekeeper's `addObserver()` throws, so a redeploy between the + description and the call refuses the collaborator too. + + The import is single-flight and fenced (the facet restarts, and the connection is unusable + until the import settles). It drains `getHistoricalSubresources()` to its end, then one + transaction sets `observerTracking` and, for a non-empty export, writes the workspace audit + chunks. A failed description or export commits nothing, and the next tracked read or admission + tries again; after the commit, the export is never asked for again. Once the facet has been + aborted, the workspace restarts whether or not the import commits, even with only the owner + connected: client capabilities such as a `GatekeeperClient` hold stubs to the aborted facet, and + live sessions must reopen against the imported requirements. The connection stays held back + until the restart lands, and the read or call that revealed the upgrade fails rather than being + replayed. Legacy registrations are retired best-effort, racing the restart. A connection created + while advertising support starts tracked without an export. Tracking is never downgraded. + + The residual: an upgraded gatekeeper that wrongly calls `authorizeObservation()` for a read, + before any tracked read of its own or any admission, has that read treated as legacy. +- **Pending creations.** A simulated resource from `createExternalResource` is not imported + while its creation is pending; its reads are audited and enforce workspace policy but record no + requirements. After approval it is imported like any other connection. +- **Removal.** A disconnected connection's audit records stay, but admission ignores them: its + checks go with it, even when data it read survives (edge case 8). +- **Direct contexts only.** A chat binding read records requirements only on that chat, checked + against "build" collaborators; a gadget binding read only on that gadget, checked against + "build" and "use" collaborators. Gadget session access, root RPC calls, forged restore stubs, + callbacks, spawned chats, code edits, merges, arguments and persistent writes transfer nothing + between contexts. This deliberately permits indirect disclosure, such as chat-derived data + reaching a gadget without checking its "use" collaborators against the chat's requirements. + Attribution follows the original authorizer, not wherever its capability is passed. +- **Execution fence.** Agent gatekeeper loopbacks carry their `executionId`; a session that + outlives its executeCode call cannot authorize a tracked observation. +- **Hooks.** Tracked connections support hook registration and delivery. A firing's observations + are attributed to the gadget the hook wakes, recorded as `{from: "hook", gadgetId}` from the + hook record's `gadgetId` (or `defaultGadgetId`), and to the workspace when neither names one. + That `gadgetId` is trusted even though an agent-bound hook's is inferred and its callback + carries its own restore target. While the hook is enabled, both "build" and "use" + collaborators must pass verification before delivery, and later admission checks the recorded + subresources. Firings revalidate that the hook exists, is enabled, and its connection is usable + on each call and synchronously before the observation's audit/policy commit, after asynchronous + verification. A racing disable/delete prevents delivery. Disabling or deleting a hook stops + future events and leaves delivered data's requirements in the action log, including across + workspace restarts; "build" collaborators stay verified against them, and "use" collaborators + only while something else still binds or hooks the connection. + +The integration fixture's tracked mode (`/control/observer-tracking`) exercises this end to end in +`packages/integration-tests/__tests__/observer-tracking.test.ts`. + +--- + ## Known limitations Revocations and role changes take effect within seconds -- the revocation restart lands in @@ -900,8 +1021,6 @@ place here only if its failure mode is *persistent* wrong state that outlives th The observer-side deferrals are ledgered in `plans/restricted-data-sharing.md`, each marked at its site in the code by a matching `TODO` comment: -- Observer verification is not serialized per profile, so concurrent opens by one collaborator - can overwrite each other's records. - A mid-registration observer named in `excludeObservers` is read as unknown and the observation admitted; persistent, since it lands in chat history (Step 5, item 1). Fix: a pending-id map, `observer-verification-fixes`. diff --git a/packages/gatekeeper-google/__tests__/worker.ts b/packages/gatekeeper-google/__tests__/worker.ts index 845fb09169..4697a0a604 100644 --- a/packages/gatekeeper-google/__tests__/worker.ts +++ b/packages/gatekeeper-google/__tests__/worker.ts @@ -71,6 +71,10 @@ class TestApprovalQueue extends RpcTarget implements ApprovalQueue { this.observations.push(description.description); } + async authorizeTrackedObservation(): Promise { + throw new Error("Unexpected tracked observation"); + } + async getGitCache(): Promise { throw new Error("Unexpected git cache access"); } diff --git a/packages/gatekeeper-google/__tests__/workerd/native-sessions.test.ts b/packages/gatekeeper-google/__tests__/workerd/native-sessions.test.ts index 4f6a3a6fab..83ca057967 100644 --- a/packages/gatekeeper-google/__tests__/workerd/native-sessions.test.ts +++ b/packages/gatekeeper-google/__tests__/workerd/native-sessions.test.ts @@ -108,6 +108,10 @@ class TestApprovalQueue extends RpcTarget implements ApprovalQueue { this.observations.push(description); } + async authorizeTrackedObservation(): Promise { + throw new Error("Unexpected tracked observation"); + } + async getGitCache(): Promise { throw new Error("Unexpected git cache access"); } diff --git a/packages/gatekeeper-google/__tests__/workerd/sheets-session.test.ts b/packages/gatekeeper-google/__tests__/workerd/sheets-session.test.ts index e7599ace39..d93865b57b 100644 --- a/packages/gatekeeper-google/__tests__/workerd/sheets-session.test.ts +++ b/packages/gatekeeper-google/__tests__/workerd/sheets-session.test.ts @@ -26,6 +26,10 @@ class TestApprovalQueue extends RpcTarget implements ApprovalQueue { this.observations.push(description); } + async authorizeTrackedObservation(): Promise { + throw new Error("Unexpected tracked observation"); + } + async getGitCache(): Promise { throw new Error("Unexpected git cache access"); } diff --git a/packages/gatekeeper-google/__tests__/workerd/slides-session.test.ts b/packages/gatekeeper-google/__tests__/workerd/slides-session.test.ts index e92d46b58c..aa8662e14c 100644 --- a/packages/gatekeeper-google/__tests__/workerd/slides-session.test.ts +++ b/packages/gatekeeper-google/__tests__/workerd/slides-session.test.ts @@ -19,6 +19,10 @@ class TestApprovalQueue extends RpcTarget implements ApprovalQueue { this.observations.push(description); } + async authorizeTrackedObservation(): Promise { + throw new Error("Unexpected tracked observation"); + } + async getGitCache(): Promise { throw new Error("Unexpected git cache access"); } diff --git a/packages/gatekeeper-google/__tests__/workerd/worker.ts b/packages/gatekeeper-google/__tests__/workerd/worker.ts index f964635e1e..d7a555f03c 100644 --- a/packages/gatekeeper-google/__tests__/workerd/worker.ts +++ b/packages/gatekeeper-google/__tests__/workerd/worker.ts @@ -156,6 +156,10 @@ class TestApprovalQueue extends RpcTarget { } } + async authorizeTrackedObservation(): Promise { + throw new Error("Tracked observations are not used by these tests."); + } + async submitAction(actionId: number, description: unknown): Promise { this.#submissions.push({actionId, description}); if (this.#pausedSubmission) { diff --git a/packages/gatekeeper-kit/__tests__/workerd/conformance/gatekeeper.ts b/packages/gatekeeper-kit/__tests__/workerd/conformance/gatekeeper.ts index 0a3b2aaee6..cc8c0ddb4e 100644 --- a/packages/gatekeeper-kit/__tests__/workerd/conformance/gatekeeper.ts +++ b/packages/gatekeeper-kit/__tests__/workerd/conformance/gatekeeper.ts @@ -103,6 +103,10 @@ export class FixtureQueue extends RpcTarget { observations.push(description); } + async authorizeTrackedObservation(): Promise { + throw new Error("The conformance gatekeeper does not use observer tracking."); + } + async submitAction(action: number, description: ActionDescription): Promise { submissions.push([action, description]); } diff --git a/packages/gatekeeper-kit/__tests__/workerd/cursors.test.ts b/packages/gatekeeper-kit/__tests__/workerd/cursors.test.ts index 38d3cef571..2d8b1f0759 100644 --- a/packages/gatekeeper-kit/__tests__/workerd/cursors.test.ts +++ b/packages/gatekeeper-kit/__tests__/workerd/cursors.test.ts @@ -41,6 +41,10 @@ class TestAuthorizer extends RpcTarget implements ObservationAuthorizer { this.#seen.push(description.description); } + async authorizeTrackedObservation(): Promise { + throw new Error("Tracked observations are not used in this test."); + } + async getGitCache(): Promise { throw new Error("Git cache is not used in this test."); } diff --git a/packages/integration-tests/__tests__/observer-tracking.test.ts b/packages/integration-tests/__tests__/observer-tracking.test.ts new file mode 100644 index 0000000000..5fcf1b94cd --- /dev/null +++ b/packages/integration-tests/__tests__/observer-tracking.test.ts @@ -0,0 +1,566 @@ +// Kernel-owned observer tracking (docs/observers.md §10), against the fixture gatekeeper's tracked +// mode: a binding advertises `supportsObserverTracking` once `/control/observer-tracking` says so, +// reports each read's `subresources` through `authorizeTrackedObservation()`, and its +// `verifyObserverAccess()` asks each observer's verifier for the outcomes `/control/verify-outcome` +// sets per subresource. + +import { afterAll, beforeAll, expect, it } from "vitest"; +import type { RpcStub } from "capnweb"; +import type { + AuthenticatedApi, ObserverBindingNeed, Overseer, PublicApi, WorkpieceId, +} from "@gadgets/workshop-shared/api"; +import { diffFiles } from "@gadgets/workshop-shared/code-change"; +import type { TestSession } from "../fixtures/gatekeeper-test/src/test-gatekeeper.js"; +import { + settleRestart, startTestGatekeeperHarness, TEST_VENDOR_ID, testControl, type Harness, +} from "../src/harness.js"; +import { NetworkInterceptor } from "../src/network-interceptor.js"; +import { + accountLabel, connect, listConnectedAccounts, MAX_OBSERVER_PROMPTS, nextUsernames, + ObserverConfigRecorder, restartWorkspace, signUp, stubFor, waitFor, WorkpieceRecorder, + type ConnectedAccount, RpcTarget, +} from "../src/rpc-client.js"; + +let harness: Harness; +const network = new NetworkInterceptor(); + +beforeAll(async () => { + network.install(); + harness = await startTestGatekeeperHarness({ enableGadgetExecution: true }); +}); + +afterAll(async () => { + try { + await harness?.server.close(); + expect(network.getUnmockedCalls()).toEqual([]); + } finally { + network.uninstall(); + } +}); + +const DENIED = + "This account can't access some of the data this workspace read through this service."; + +/** A resource URL no other test (or earlier run against the same storage) has used. */ +function freshThingUrl(name: string): string { + return `https://gadgets-test.example/things/${name}-${crypto.randomUUID().slice(0, 8)}`; +} + +function enableObserverTracking(resourceUrl: string): Promise { + return testControl(harness, "observer-tracking", { resourceUrl }); +} + +/** Refuse (or re-admit) `label` for one subresource of the tracked binding at `resourceUrl`. */ +function setSubresourceAccess( + label: string, resourceUrl: string, subresource: string, allow: boolean): Promise { + return testControl(harness, "verify-outcome", + { label, resourceUrl, subresource, allow, reason: "The test refused this subresource." }); +} + +async function historyExportCount(resourceUrl: string): Promise { + return (await testControl<{ count: number }>(harness, "history-export-count", { resourceUrl })) + .count; +} + +type ObserverEvent = { resourceUrl: string; type: "add" | "remove"; id: string }; + +async function observerEvents(resourceUrl: string): Promise { + return (await testControl<{ events: ObserverEvent[] }>(harness, "observer-events", { resourceUrl })) + .events; +} + +async function provisionAccount(api: RpcStub): Promise { + await api.provisionAmbientAccount(TEST_VENDOR_ID); + return waitFor("the test account to be provisioned", async () => + (await listConnectedAccounts(api)).find(a => a.vendorId === TEST_VENDOR_ID) ?? null); +} + +type Workspace = Disposable & { + gadgetId: string; + overseer: RpcStub; + account: ConnectedAccount; +}; + +async function newWorkspace(publicApi: RpcStub): Promise { + const [alice] = nextUsernames("trackalice"); + const api = await signUp(publicApi, alice); + const account = await provisionAccount(api); + const overseer = await api.newWorkspace(); + const { id: gadgetId } = await overseer.getMetadata(); + return { + gadgetId, overseer, account, + [Symbol.dispose]() { + overseer[Symbol.dispose](); + api[Symbol.dispose](); + }, + }; +} + +/** Connect `resourceUrl` in the owner's workspace and read it once, depending on `subresources`. */ +async function connectAndRead(ws: Workspace, resourceUrl: string, subresources: string[]) + : Promise { + using gatekeeper = await ws.overseer.newGatekeeper(ws.account.id, resourceUrl); + if (!gatekeeper) throw new Error(`Failed to connect ${resourceUrl}`); + using session = await gatekeeper.openSession() as RpcStub; + await expect(session.readValue(false, false, undefined, subresources)).resolves.toBe(42); + return gatekeeper.getId(); +} + +type Collaborator = Disposable & { + api: RpcStub; + account: ConnectedAccount; + label: string; +}; + +/** + * Share the workspace with a fresh collaborator, with an account of their own. They get their + * own connection, which a restart of the owner's workspace doesn't sever. + */ +async function addCollaborator(ws: Workspace, role: "build" | "use" = "build") + : Promise { + const [bob] = nextUsernames("trackbob"); + const connection = connect(harness.url); + const api = await signUp(connection, bob); + const account = await provisionAccount(api); + if (!await ws.overseer.addCollaborator(bob, role)) { + throw new Error(`Failed to share the workspace with ${bob}`); + } + return { + api, account, label: accountLabel(account), + [Symbol.dispose]() { + api[Symbol.dispose](); + connection[Symbol.dispose](); + }, + }; +} + +/** The collaborator opens the workspace, choosing their account at every prompt. */ +async function collaboratorOpens(bob: Collaborator, gadgetId: string) + : Promise<{ recorder: ObserverConfigRecorder; error?: Error }> { + const recorder = new ObserverConfigRecorder().alwaysChoose(bob.account.id, MAX_OBSERVER_PROMPTS); + const callback = stubFor(recorder); + try { + (await bob.api.openWorkspace(gadgetId, undefined, callback))[Symbol.dispose](); + return { recorder }; + } catch (error) { + return { recorder, error: error as Error }; + } finally { + callback[Symbol.dispose](); + } +} + +it.concurrent("legacy and tracked connections coexist, each verified its own way", async () => { + using publicApi = connect(harness.url); + using ws = await newWorkspace(publicApi); + const legacyUrl = freshThingUrl("coexist-legacy"); + const trackedUrl = freshThingUrl("coexist-tracked"); + await enableObserverTracking(trackedUrl); + await connectAndRead(ws, legacyUrl, []); + const trackedId = await connectAndRead(ws, trackedUrl, ["doc-1"]); + using bob = await addCollaborator(ws); + + const first = await collaboratorOpens(bob, ws.gadgetId); + expect(first.error).toBeUndefined(); + // The tracked connection is asked about under the same connection ID used by its binding. + expect(first.recorder.calls[0]!.map(need => need.gatekeeperId)).toContain(trackedId); + // The legacy connection registered Bob as an observer; the tracked one never hears of him. + expect(await observerEvents(legacyUrl)).toMatchObject([{ type: "add" }]); + expect(await observerEvents(trackedUrl)).toEqual([]); + + // Only the tracked connection's verifier check can refuse him for what it recorded. + await setSubresourceAccess(bob.label, trackedUrl, "doc-1", false); + const second = await collaboratorOpens(bob, ws.gadgetId); + expect(second.error?.message).toContain(DENIED); + expect(second.error?.message).toContain(bob.label); + // His persisted choices are re-verified first, so the opening prompt carries the failure. + const failed = second.recorder.calls[0]!.filter(need => need.failure); + expect(failed).toEqual([expect.objectContaining({ gatekeeperId: trackedId })]); + expect(failed[0]!.failure!.reason).toContain(DENIED); +}); + +it.concurrent("the same local subresource ID is checked independently in different resources", + async () => { + using publicApi = connect(harness.url); + using ws = await newWorkspace(publicApi); + const firstUrl = freshThingUrl("scope-first"); + const secondUrl = freshThingUrl("scope-second"); + const localId = 'same/local#id:"雪"'; + await enableObserverTracking(firstUrl); + await enableObserverTracking(secondUrl); + const firstId = await connectAndRead(ws, firstUrl, [localId]); + const secondId = await connectAndRead(ws, secondUrl, [localId]); + using bob = await addCollaborator(ws); + expect((await collaboratorOpens(bob, ws.gadgetId)).error).toBeUndefined(); + + await setSubresourceAccess(bob.label, firstUrl, localId, false); + const firstDenied = await collaboratorOpens(bob, ws.gadgetId); + expect(firstDenied.error?.message).toContain(DENIED); + expect(firstDenied.recorder.calls[0]!.filter(need => need.failure)).toEqual([ + expect.objectContaining({ gatekeeperId: firstId, resourceUrl: firstUrl }), + ]); + + await setSubresourceAccess(bob.label, firstUrl, localId, true); + await setSubresourceAccess(bob.label, secondUrl, localId, false); + const secondDenied = await collaboratorOpens(bob, ws.gadgetId); + expect(secondDenied.error?.message).toContain(DENIED); + expect(secondDenied.recorder.calls[0]!.filter(need => need.failure)).toEqual([ + expect.objectContaining({ gatekeeperId: secondId, resourceUrl: secondUrl }), + ]); + + await setSubresourceAccess(bob.label, secondUrl, localId, true); + expect((await collaboratorOpens(bob, ws.gadgetId)).error).toBeUndefined(); +}); + +it.concurrent("a legacy connection's history is imported at its first tracked observation", + async () => { + using publicApi = connect(harness.url); + using ws = await newWorkspace(publicApi); + const url = freshThingUrl("import"); + using gatekeeper = await ws.overseer.newGatekeeper(ws.account.id, url); + if (!gatekeeper) throw new Error(`Failed to connect ${url}`); + using session = await gatekeeper.openSession() as RpcStub; + // Read under the legacy protocol: only the binding itself remembers what it depended on. The + // fixture exports one ID per page after an empty page, so "hist-2" arrives on the last page. + await expect(session.readValue(false, false, undefined, ["hist-1", "hist-2"])).resolves.toBe(42); + using bob = await addCollaborator(ws); + + // The gatekeeper upgrades. Its first tracked observation, through a session minted under the + // legacy protocol, has the Workshop import the connection's history; the import aborts the + // facet that read is running in. It also restarts the workspace, owner's session included: let + // that land before Bob opens. + await enableObserverTracking(url); + await expect(session.readValue(false, false, undefined, ["new-1"])) + .rejects.toThrow(/restarted to move its observer tracking/); + expect(await historyExportCount(url)).toBe(1); + await settleRestart(); + + await setSubresourceAccess(bob.label, url, "hist-2", false); + const refused = await collaboratorOpens(bob, ws.gadgetId); + // The refusal can only come from the imported ID: no tracked read has succeeded. + expect(refused.error?.message).toContain(DENIED); + // Migration may retire legacy registrations, but admission must not register new ones. + expect((await observerEvents(url)).filter(event => event.type === "add")).toEqual([]); + + await setSubresourceAccess(bob.label, url, "hist-2", true); + expect((await collaboratorOpens(bob, ws.gadgetId)).error).toBeUndefined(); + // The import committed, so the history is never exported again. + expect(await historyExportCount(url)).toBe(1); +}); + +it.concurrent("imports a protocol upgrade that arrives while account selection is pending", + async () => { + using publicApi = connect(harness.url); + using ws = await newWorkspace(publicApi); + const url = freshThingUrl("prompt-upgrade"); + await connectAndRead(ws, url, ["historical-secret"]); + using bob = await addCollaborator(ws); + await setSubresourceAccess(bob.label, url, "historical-secret", false); + + let reached!: () => void; + let release!: () => void; + const prompted = new Promise(resolve => { reached = resolve; }); + const parked = new Promise(resolve => { release = resolve; }); + class DelayedAccounts extends RpcTarget { + async configure(needs: ObserverBindingNeed[]) { + reached(); + await parked; + return needs.map(need => ({ gatekeeperId: need.gatekeeperId, accountId: bob.account.id })); + } + } + using callback = stubFor(new DelayedAccounts()); + const opening = bob.api.openWorkspace(ws.gadgetId, undefined, callback).then(stub => { + stub[Symbol.dispose](); + return undefined; + }, error => error as Error); + await prompted; + try { + await enableObserverTracking(url); + } finally { + release(); + } + + expect((await opening)?.message).toContain(DENIED); + expect(await historyExportCount(url)).toBe(1); + + await settleRestart(); + await setSubresourceAccess(bob.label, url, "historical-secret", true); + expect((await collaboratorOpens(bob, ws.gadgetId)).error).toBeUndefined(); +}); + +it.concurrent("disconnect drops tracked permission checks but keeps the observation audit", + async () => { + using publicApi = connect(harness.url); + using ws = await newWorkspace(publicApi); + const url = freshThingUrl("removed"); + await enableObserverTracking(url); + const connectionId = await connectAndRead(ws, url, ["kept-1"]); + using bob = await addCollaborator(ws); + const audit = await ws.overseer.listActions({ filter: "observation" }); + expect(audit.entries).toMatchObject([ + { description: { title: "Read the test value" } }, + ]); + await setSubresourceAccess(bob.label, url, "kept-1", false); + expect((await collaboratorOpens(bob, ws.gadgetId)).error?.message).toContain(DENIED); + + { + using gatekeeper = await ws.overseer.getGatekeeperById(connectionId); + await gatekeeper.remove(); + } + expect(await ws.overseer.listActions({ filter: "observation" })).toEqual(audit); + const afterDisconnect = await collaboratorOpens(bob, ws.gadgetId); + expect(afterDisconnect.error).toBeUndefined(); + expect(afterDisconnect.recorder.calls).toEqual([]); + let restarted = false; + ws.overseer.onRpcBroken(() => { restarted = true; }); + await restartWorkspace(harness.url, ws.overseer); + await waitFor("the workspace restart", async () => restarted || null); + + // The denied resource is still denied at the vendor, but no longer belongs to this workspace. + const afterRestart = await collaboratorOpens(bob, ws.gadgetId); + expect(afterRestart.error).toBeUndefined(); + expect(afterRestart.recorder.calls).toEqual([]); +}); + +// Author a new gadget's server code through the same human-only edit/merge API as the code editor. +async function installServer(ws: Workspace, gadgetId: WorkpieceId, server: string): Promise { + const workpieces = new WorkpieceRecorder(); + using subscriber = stubFor(workpieces); + using _subscription = await ws.overseer.subscribeToWorkpieces(subscriber); + await workpieces.loaded; + const summary = workpieces.summaries.get(gadgetId); + if (summary?.type !== "gadget" || summary.commitId === undefined) { + throw new Error("The new gadget has no committed head"); + } + const chatId = await ws.overseer.newChat("Implement the server", null); + await ws.overseer.submitCodeChange(chatId, { + generation: 0, revision: 0, clientId: "observer-tracking", seq: 1, + pins: [{ gadgetId, baseCommit: summary.commitId }], + change: diffFiles(new Map(), new Map([[gadgetId, new Map([["server.js", server]])]])), + }); + expect(await ws.overseer.mergeChanges(chatId)).toEqual({ outcome: "merged" }); +} + +interface TrackedGadget extends RpcTarget { + read(subresource: string): Promise<{ value: number; subresource: string }>; +} + +const TRACKED_SERVER = `import { DurableObject } from "cloudflare:workers"; +export class Gadget extends DurableObject { + async read(subresource) { + const value = await this.env.TRACKED_THING.readValue(false, false, undefined, [subresource]); + return { value, subresource }; + } +}`; + +it.concurrent("a gadget's tracked reads enforce use collaborator admission and live access", + async () => { + using publicApi = connect(harness.url); + using ws = await newWorkspace(publicApi); + const url = freshThingUrl("gadget-tracked"); + await enableObserverTracking(url); + using tracked = await ws.overseer.newGatekeeper(ws.account.id, url); + if (!tracked) throw new Error("Failed to create the tracked connection"); + const connectionId = await tracked.getId(); + using gadget = await ws.overseer.createGadget("Tracked Gadget", undefined, "TRACKED_GADGET"); + const gadgetId = await gadget.getId(); + await gadget.bind("TRACKED_THING", connectionId); + await installServer(ws, gadgetId, TRACKED_SERVER); + + using facet = await gadget.connectToGadget() as RpcStub; + await expect(facet.read("gadget-1")).resolves.toEqual({ value: 42, subresource: "gadget-1" }); + using viewer = await addCollaborator(ws, "use"); + await setSubresourceAccess(viewer.label, url, "gadget-1", false); + const refused = await collaboratorOpens(viewer, ws.gadgetId); + expect(refused.error?.message).toContain(DENIED); + expect(refused.recorder.calls[0]).toContainEqual(expect.objectContaining({ + gatekeeperId: connectionId, resourceUrl: url, + })); + + await setSubresourceAccess(viewer.label, url, "gadget-1", true); + const recorder = new ObserverConfigRecorder().alwaysChoose(viewer.account.id, MAX_OBSERVER_PROMPTS); + using callback = stubFor(recorder); + using viewerWorkspace = await viewer.api.openWorkspace(ws.gadgetId, undefined, callback); + using viewerGadget = await viewerWorkspace.getGadget(gadgetId); + using viewerFacet = await viewerGadget.connectToGadget() as RpcStub; + await expect(viewerFacet.read("gadget-1")) + .resolves.toEqual({ value: 42, subresource: "gadget-1" }); + + // A new subresource is checked before the binding can return its value to gadget code. + await setSubresourceAccess(viewer.label, url, "live-2", false); + await expect(viewerFacet.read("live-2")) + .rejects.toThrow(/blocked because it contains data that a current collaborator/); + // A refused read did not add the inaccessible ID to the gadget's admission history. + expect((await collaboratorOpens(viewer, ws.gadgetId)).error).toBeUndefined(); + await setSubresourceAccess(viewer.label, url, "live-2", true); + await expect(viewerFacet.read("live-2")).resolves.toEqual({ value: 42, subresource: "live-2" }); + + // The successful live read joined gadget history, so revocation also denies the next admission. + await setSubresourceAccess(viewer.label, url, "live-2", false); + expect((await collaboratorOpens(viewer, ws.gadgetId)).error?.message).toContain(DENIED); + expect(await observerEvents(url)).toEqual([]); +}); + +interface TrackedHookGadget extends RpcTarget { + watch(key: string): Promise; + deliveredValues(): Promise; +} + +const TRACKED_HOOK_SERVER = `import { DurableObject, RpcTarget, restore } from "cloudflare:workers"; +export class Gadget extends DurableObject { + async [restore](params) { + if (params.type !== "tracked-hook") throw new TypeError("Unknown hook type"); + return new TrackedHook(this.ctx.storage); + } + async watch(key) { + await this.env.TRACKED_THING.watch(key, await this.ctx.restore({ type: "tracked-hook" })); + } + async deliveredValues() { return (await this.ctx.storage.get("values")) ?? []; } +} +class TrackedHook extends RpcTarget { + constructor(storage) { super(); this.storage = storage; } + async onValueRequested(value) { + const values = (await this.storage.get("values")) ?? []; + await this.storage.put("values", [...values, value]); + } +}`; + +it.concurrent("tracked hooks deliver authorized data, check both collaborator roles, and retain history after deletion", + async () => { + using publicApi = connect(harness.url); + using ws = await newWorkspace(publicApi); + const url = freshThingUrl("hook-tracked"); + const hookKey = `tracked-hook-${crypto.randomUUID()}`; + await enableObserverTracking(url); + using tracked = await ws.overseer.newGatekeeper(ws.account.id, url); + if (!tracked) throw new Error("Failed to create the tracked connection"); + const connectionId = await tracked.getId(); + using gadget = await ws.overseer.createGadget("Tracked Hook", undefined, "TRACKED_HOOK"); + const gadgetId = await gadget.getId(); + await gadget.bind("TRACKED_THING", connectionId); + await installServer(ws, gadgetId, TRACKED_HOOK_SERVER); + + using registrationFacet = await gadget.connectToGadget() as RpcStub; + await registrationFacet.watch(hookKey); + const [hook] = await ws.overseer.listHooks(); + expect(hook).toMatchObject({ gatekeeperId: connectionId, gadgetId, enabled: false }); + if (!hook) throw new Error("The gadget did not register its tracked hook"); + // The temporary binding is only needed to register. Delivery and later admission must work + // without a gadget binding keeping this connection in scope. + await gadget.unbind("TRACKED_THING"); + await ws.overseer.enableHook(hook.id); + expect(await ws.overseer.listHooks()).toMatchObject([{ id: hook.id, enabled: true }]); + using facet = await gadget.connectToGadget() as RpcStub; + + using builder = await addCollaborator(ws, "build"); + using viewer = await addCollaborator(ws, "use"); + using observers = new DisposableStack(); + for (const collaborator of [builder, viewer]) { + const recorder = new ObserverConfigRecorder() + .alwaysChoose(collaborator.account.id, MAX_OBSERVER_PROMPTS); + using callback = stubFor(recorder); + observers.use(await collaborator.api.openWorkspace(ws.gadgetId, undefined, callback)); + expect(recorder.calls[0]).toContainEqual(expect.objectContaining({ gatekeeperId: connectionId })); + } + + // No session read has seeded these IDs: the hook must authorize them before invoking the + // persistent callback, which records the delivered data in the actual gadget's storage. + const subresources = ["hook-doc-1", 'hook/local#id:"雪"']; + expect(await testControl(harness, "fire-hook", + { key: hookKey, value: 71, resourceUrl: url, subresources })).toEqual({ fired: true }); + expect(await facet.deliveredValues()).toEqual([71]); + const audit = await ws.overseer.listActions({ filter: "observation" }); + expect(audit.entries).toMatchObject([{ + type: "observation", gatekeeperId: connectionId, + description: { + title: `Hook ${hookKey} requested 71`, + }, + }]); + + for (const [index, collaborator] of [builder, viewer].entries()) { + const deniedId = `denied-hook-${index}`; + await setSubresourceAccess(collaborator.label, url, deniedId, false); + expect(await testControl(harness, "fire-hook", { + key: hookKey, value: 80 + index, resourceUrl: url, subresources: [deniedId], + })).toEqual({ error: expect.stringContaining( + "blocked because it contains data that a current collaborator") }); + expect(await facet.deliveredValues()).toEqual([71]); + expect(await ws.overseer.listActions({ filter: "observation" })).toEqual(audit); + // A failed delivery must not commit an admission requirement for its denied subresource. + expect((await collaboratorOpens(collaborator, ws.gadgetId)).error).toBeUndefined(); + } + + await ws.overseer.deleteHook(hook.id); + expect(await ws.overseer.listHooks()).toEqual([]); + expect(await testControl(harness, "fire-hook", + { key: hookKey, value: 99, resourceUrl: url, subresources })) + .toEqual({ error: "Hook has been deleted or disabled." }); + expect(await facet.deliveredValues()).toEqual([71]); + expect(await ws.overseer.listActions({ filter: "observation" })).toEqual(audit); + + // Hook removal does not erase data the callback already brought into the workspace, so build + // collaborators are still verified against it. Nothing binds or hooks the connection any more, + // so as under the legacy protocol it has left "use" collaborators' verification scope. + await setSubresourceAccess(builder.label, url, subresources[0]!, false); + const refused = await collaboratorOpens(builder, ws.gadgetId); + expect(refused.error?.message).toContain(DENIED); + expect(refused.recorder.calls[0]).toContainEqual(expect.objectContaining({ + gatekeeperId: connectionId, failure: expect.objectContaining({ reason: expect.stringContaining(DENIED) }), + })); + await setSubresourceAccess(viewer.label, url, subresources[1]!, false); + const admitted = await collaboratorOpens(viewer, ws.gadgetId); + expect(admitted.error).toBeUndefined(); + expect(admitted.recorder.calls).toEqual([]); + expect(await observerEvents(url)).toEqual([]); +}); + +// Not concurrent: with other tests running, workerd 1.20261001 dies with "Promise callback +// destroyed itself" when a workspace restarts after a hook delivery, whether this import restarts +// it or removing a collaborator does. +it("a hook enabled under the legacy protocol keeps firing after the upgrade imports its deliveries", + async () => { + using publicApi = connect(harness.url); + using ws = await newWorkspace(publicApi); + const url = freshThingUrl("hook-upgrade"); + const hookKey = `upgrade-hook-${crypto.randomUUID()}`; + using legacy = await ws.overseer.newGatekeeper(ws.account.id, url); + if (!legacy) throw new Error(`Failed to connect ${url}`); + using gadget = await ws.overseer.createGadget("Upgraded Hook", undefined, "UPGRADED_HOOK"); + const gadgetId = await gadget.getId(); + await gadget.bind("TRACKED_THING", await legacy.getId()); + await installServer(ws, gadgetId, TRACKED_HOOK_SERVER); + using facet = await gadget.connectToGadget() as RpcStub; + await facet.watch(hookKey); + const [hook] = await ws.overseer.listHooks(); + if (!hook) throw new Error("The gadget did not register its hook"); + await ws.overseer.enableHook(hook.id); + using bob = await addCollaborator(ws); + const fire = (value: number, subresources: string[]) => + testControl(harness, "fire-hook", { key: hookKey, value, resourceUrl: url, subresources }); + + // Under the legacy protocol only the gatekeeper remembers what a delivery depended on. + expect(await fire(71, ["hook-secret"])).toEqual({ fired: true }); + + // The gatekeeper upgrades. The hook's first tracked delivery has the Workshop import the + // connection's history and is refused; the restart the import starts must land before a retry. + await enableObserverTracking(url); + expect(await fire(72, ["live-1"])).toEqual({ + error: expect.stringContaining("now supports the Workshop's observer tracking"), + }); + expect(await historyExportCount(url)).toBe(1); + await settleRestart(); + // The hook's registration predates the upgrade, and it delivers under the tracked protocol. + expect(await fire(73, ["live-1"])).toEqual({ fired: true }); + + // Only the legacy delivery depended on "hook-secret", so only its export can refuse Bob. + await setSubresourceAccess(bob.label, url, "hook-secret", false); + expect((await collaboratorOpens(bob, ws.gadgetId)).error?.message).toContain(DENIED); + + await setSubresourceAccess(bob.label, url, "hook-secret", true); + const recorder = new ObserverConfigRecorder().alwaysChoose(bob.account.id, MAX_OBSERVER_PROMPTS); + using callback = stubFor(recorder); + using bobWorkspace = await bob.api.openWorkspace(ws.gadgetId, undefined, callback); + using bobGadget = await bobWorkspace.getGadget(gadgetId); + using bobFacet = await bobGadget.connectToGadget() as RpcStub; + expect(await bobFacet.deliveredValues()).toEqual([71, 73]); + expect(await historyExportCount(url)).toBe(1); +}); diff --git a/packages/integration-tests/fixtures/gatekeeper-test/src/test-gatekeeper.ts b/packages/integration-tests/fixtures/gatekeeper-test/src/test-gatekeeper.ts index e11d41a165..c7da973703 100644 --- a/packages/integration-tests/fixtures/gatekeeper-test/src/test-gatekeeper.ts +++ b/packages/integration-tests/fixtures/gatekeeper-test/src/test-gatekeeper.ts @@ -26,9 +26,9 @@ import { skipRpcValidation, validateRpc } from "capnweb-validate"; import { connectHandoffPageHtml, htmlResponse } from "@gadgets/gatekeeper-kit/connect-pages"; import type { AccountDescription, ActionDescription, ActionKind, AgentCatalog, ApprovalQueue, ConnectHandoff, - Gatekeeper, GatekeeperConnectCallback, GatekeeperUser, GatekeeperUserVerifier, HookController, - HookInitiator, HookTargetMetadata, ResourceDescription, ResourceConfiguratorFrame, - SupportedResource, VendorDescription, + Cursor, Gatekeeper, GatekeeperConnectCallback, GatekeeperUser, GatekeeperUserVerifier, + HookController, HookInitiator, HookTargetMetadata, ResourceDescription, + ResourceConfiguratorFrame, SupportedResource, VendorDescription, } from "@gadgets/workshop-shared/gatekeeper"; import type { ChatGatewayRpcTarget, GadgetResponse, SubmitExternalMessageInput, SubmitExternalMessageResult, @@ -94,7 +94,8 @@ type HookState = { disableCount: number; }; -function outcomeKey(label: string, resourceUrl?: string): string { +function outcomeKey(label: string, resourceUrl?: string, subresource?: string): string { + if (subresource !== undefined) return `outcome:${label}:${resourceUrl}#${subresource}`; return resourceUrl ? `outcome:${label}:${resourceUrl}` : `outcome:${label}`; } @@ -104,18 +105,61 @@ function newAccountLabel(): string { @validateRpc() export class TestControl extends DurableObject { - setVerifyOutcome(label: string, outcome: VerifyOutcome, resourceUrl?: string): void { - this.ctx.storage.kv.put(outcomeKey(label, resourceUrl), outcome); - } - - getVerifyOutcome(label: string, resourceUrl?: string): VerifyOutcome { - // A resource-specific outcome wins over a label-wide one. Default to admitting: a - // collaborator's first open has to be able to succeed. - return this.ctx.storage.kv.get(outcomeKey(label, resourceUrl)) + setVerifyOutcome( + label: string, outcome: VerifyOutcome, resourceUrl?: string, subresource?: string): void { + this.ctx.storage.kv.put(outcomeKey(label, resourceUrl, subresource), outcome); + } + + getVerifyOutcome(label: string, resourceUrl?: string, subresource?: string): VerifyOutcome { + // A subresource-specific outcome wins over a resource-specific one, which wins over a + // label-wide one. Default to admitting: a collaborator's first open has to be able to succeed. + return (subresource !== undefined + ? this.ctx.storage.kv.get(outcomeKey(label, resourceUrl, subresource)) + : undefined) + ?? this.ctx.storage.kv.get(outcomeKey(label, resourceUrl)) ?? this.ctx.storage.kv.get(outcomeKey(label)) ?? { allow: true }; } + /** Answers TestVerifier.deniedSubresources(): input IDs whose verification outcome denies. */ + deniedSubresources(label: string, subresources: string[]): string[] { + return subresources.filter(id => { + const [resourceUrl, subresource] = JSON.parse(id) as [string, string]; + return !this.getVerifyOutcome(label, resourceUrl, subresource).allow; + }); + } + + /** + * Makes the binding at `resourceUrl` advertise kernel observer tracking. Read on every + * describe() and observation, so this upgrades an existing legacy binding in place. + */ + enableObserverTracking(resourceUrl: string): void { + this.ctx.storage.kv.put(`observer-tracking:${resourceUrl}`, true); + } + + isObserverTracked(resourceUrl: string): boolean { + return this.ctx.storage.kv.get(`observer-tracking:${resourceUrl}`) ?? false; + } + + recordHistoryExport(resourceUrl: string): void { + this.ctx.storage.kv.put( + `history-exports:${resourceUrl}`, this.getHistoryExportCount(resourceUrl) + 1); + } + + /** Every getHistoricalSubresources() call: the Workshop should need exactly one. */ + getHistoryExportCount(resourceUrl: string): number { + return this.ctx.storage.kv.get(`history-exports:${resourceUrl}`) ?? 0; + } + + /** + * The subresources legacy hook deliveries about `resourceUrl` depended on. Deliveries run here, + * outside the binding's facet, so this object keeps them, keyed by resource URL like the rest of + * this fixture's per-binding state: every binding of the URL exports them, which only restricts. + */ + getLegacyHookHistory(resourceUrl: string): string[] { + return this.ctx.storage.kv.get(`legacy-hook-history:${resourceUrl}`) ?? []; + } + recordObserverEvent(event: ObserverEvent): void { const events = this.ctx.storage.kv.get("observer-events") ?? []; events.push(event); @@ -265,16 +309,36 @@ export class TestControl extends DurableObject { return { enabled: initiator !== undefined, target, disableCount }; } - async fireHook(key: string, value: number): Promise<{ fired: true } | { error: string }> { + /** + * Delivers `value` as an event about the binding at `resourceUrl`, as a provider's event names + * its resource: the protocol comes from that binding's current mode, never from the hook's + * registration, so hooks registered before the binding upgraded keep firing. Without + * `resourceUrl`, the delivery is a legacy observation that depends on nothing. + */ + async fireHook(key: string, value: number, resourceUrl?: string, subresources?: string[]) + : Promise<{ fired: true } | { error: string }> { const { initiator } = this.#hook(key); if (!initiator) return { error: "hook was never enabled" }; try { const { callback, approvalQueue } = await initiator.startHook(); try { - await approvalQueue.authorizeObservation({ + const qualified = (subresources ?? []).map(id => JSON.stringify([resourceUrl, id])); + const description = { title: `Hook ${key} requested ${value}`, description: "The integration test fired this hook.", - }); + }; + if (resourceUrl !== undefined && this.isObserverTracked(resourceUrl)) { + await approvalQueue.authorizeTrackedObservation({ ...description, subresources: qualified }); + } else { + // Recorded before authorizing: an import can drain the history while the authorization + // is in flight, and the delivery must not then reach the gadget unexported. + if (resourceUrl !== undefined && qualified.length > 0) { + const history = this.getLegacyHookHistory(resourceUrl); + this.ctx.storage.kv.put( + `legacy-hook-history:${resourceUrl}`, [...new Set([...history, ...qualified])]); + } + await approvalQueue.authorizeObservation(description); + } await callback.onValueRequested(value); } finally { callback[Symbol.dispose](); @@ -566,6 +630,8 @@ export class TestAccount */ export interface TestVerifierApi extends GatekeeperUserVerifier { identify(): Promise; + /** The given subresource IDs this account may not read. */ + deniedSubresources(subresources: string[]): Promise; } @validateRpc() @@ -574,6 +640,14 @@ export class TestVerifier async identify(): Promise { return this.ctx.props.label; } + + /** + * Each ID encodes its resource URL and local subresource: grant unless the test refused this + * account for that subresource, at that binding, or everywhere (`/control/verify-outcome`). + */ + async deniedSubresources(subresources: string[]): Promise { + return control(this.ctx.exports).deniedSubresources(this.ctx.props.label, subresources); + } } // --------------------------------------------------------------------------- @@ -588,10 +662,12 @@ export interface TestSession { /** * `restricted` marks the observation `containsRestrictedData`; `ownerInvitesOnly` marks it * `ownerInvitesOnly`. `excludeObservers` lists observer ids (from `/control/observer-events`) - * that must not see it. + * that must not see it. `subresources` are local IDs the read depends on: qualified with the + * resource URL before reporting them on a tracked observation (`[]` if omitted), or before + * keeping them in the legacy binding's history for `getHistoricalSubresources()` to export. */ - readValue(restricted?: boolean, ownerInvitesOnly?: boolean, excludeObservers?: string[]) - : Promise; + readValue(restricted?: boolean, ownerInvitesOnly?: boolean, excludeObservers?: string[], + subresources?: string[]): Promise; /** `incomplete` omits the `descriptionIsComplete` claim, as a summary-only gatekeeper would. */ writeValue(value: number, opts?: { autoApprovable?: boolean; incomplete?: boolean }): Promise; writeValues(values: number[]): Promise; @@ -613,21 +689,34 @@ class TestSessionTarget extends RpcTarget implements TestSession { approvalQueue: RpcStub, private readonly state: DurableObjectStub, private readonly label: string, + private readonly resourceUrl: string, private readonly ctx: DurableObjectState) { super(); this.approvalQueue = approvalQueue.dup(); } async readValue( - restricted?: boolean, ownerInvitesOnly?: boolean, excludeObservers?: string[]) - : Promise { - await this.approvalQueue.authorizeObservation({ + restricted?: boolean, ownerInvitesOnly?: boolean, excludeObservers?: string[], + subresources?: string[]): Promise { + const tracked = await this.state.isObserverTracked(this.resourceUrl); + // JSON tuples preserve both resource scope and arbitrary local IDs without delimiter collisions. + const qualified = (subresources ?? []).map(id => JSON.stringify([this.resourceUrl, id])); + const description = { title: "Read the test value", description: "Read the deterministic value exposed by the integration-test gatekeeper.", ...(restricted ? { containsRestrictedData: true } : {}), ...(ownerInvitesOnly ? { ownerInvitesOnly: true } : {}), - ...(excludeObservers ? { excludeObservers } : {}), - }); + }; + if (tracked) { + await this.approvalQueue.authorizeTrackedObservation({ ...description, subresources: qualified }); + } else { + await this.approvalQueue.authorizeObservation( + { ...description, ...(excludeObservers ? { excludeObservers } : {}) }); + } + if (!tracked && subresources) { + const history = this.ctx.storage.kv.get(HISTORY_KEY) ?? []; + this.ctx.storage.kv.put(HISTORY_KEY, [...new Set([...history, ...qualified])]); + } return 42; } @@ -675,6 +764,30 @@ class TestSessionTarget extends RpcTarget implements TestSession { const SET_VALUE_ACTION_KIND: ActionKind = { tag: "set-value", label: "Set value" }; +/** + * Where a legacy binding keeps the subresources its reads depended on, in its own storage. Its + * hooks' deliveries keep theirs in TestControl (getLegacyHookHistory()). + */ +const HISTORY_KEY = "observer-history"; + +/** + * Pages a legacy binding's history for the Workshop's import one ID at a time, after an empty + * page such as a filtered walk can yield: the Workshop must drain it to `null`, not stop early. + */ +@validateRpc() +class HistoryCursor extends RpcTarget implements Cursor { + readonly #pages: string[][]; + + constructor(ids: string[]) { + super(); + this.#pages = [[], ...ids.map(id => [id])]; + } + + async next(): Promise { + return this.#pages.shift() ?? null; + } +} + /** Polls rather than parks a promise: the release arrives on another request to TestControl. */ async function waitForApplyRelease(state: DurableObjectStub, label: string) { for (const deadline = Date.now() + 30_000; !await state.isApplyReleased(label);) { @@ -713,6 +826,7 @@ export class TestGatekeeper }; } const name = decodeURIComponent(new URL(this.ctx.props.resourceUrl).pathname.split("/").pop()!); + const tracked = await control(this.ctx.exports).isObserverTracked(this.ctx.props.resourceUrl); return { url: this.ctx.props.resourceUrl, // Distinct per binding, so a message covering two failing bindings names both. @@ -720,9 +834,29 @@ export class TestGatekeeper snippet: `The test resource ${name}.`, suggestedBindingName: "TEST_THING", tsType: "TestThing", + ...(tracked ? { supportsObserverTracking: true } : {}), }; } + /** What this binding's legacy-mode reads and hook deliveries recorded, for the one-time import. */ + async getHistoricalSubresources(): Promise> { + const state = control(this.ctx.exports); + await state.recordHistoryExport(this.ctx.props.resourceUrl); + return new HistoryCursor([ + ...this.ctx.storage.kv.get(HISTORY_KEY) ?? [], + ...await state.getLegacyHookHistory(this.ctx.props.resourceUrl), + ]); + } + + /** Admits each observer whose verifier reports none of `subresources` denied. */ + async verifyObserverAccess( + observers: { id: string; verifier: Fetcher }[], + subresources: string[]): Promise<{ allowedObserverIds: string[] }> { + const denials = await Promise.all( + observers.map(({ verifier }) => verifier.deniedSubresources(subresources))); + return { allowedObserverIds: observers.filter((_, i) => denials[i].length === 0).map(o => o.id) }; + } + async getTypeScriptTypes(): Promise { return TYPES_CODE; } @@ -732,8 +866,8 @@ export class TestGatekeeper } async startSession(approvalQueue: RpcStub): Promise { - return new TestSessionTarget( - approvalQueue, control(this.ctx.exports), this.ctx.props.label, this.ctx); + return new TestSessionTarget(approvalQueue, control(this.ctx.exports), this.ctx.props.label, + this.ctx.props.resourceUrl, this.ctx); } [restore](): ConnectionProbe { @@ -750,11 +884,15 @@ export class TestGatekeeper * * Asks the verifier who it speaks for, then consults the control state for that account. Throwing * is how a gatekeeper reports "this user may not observe what the gadget has read", and it's the - * behaviour the overseer's failure handling is built around. + * behaviour the overseer's failure handling is built around. In tracked mode it always throws, as + * `ResourceDescription.supportsObserverTracking` requires. */ async addObserver(id: string, user: Fetcher): Promise { - const label = await user.identify(); const { resourceUrl } = this.ctx.props; + if (!this.ctx.props.ambient && await control(this.ctx.exports).isObserverTracked(resourceUrl)) { + throw new Error("This resource's observers are tracked by the Workshop."); + } + const label = await user.identify(); if (this.ctx.props.ambient) { await control(this.ctx.exports).recordAmbientVerification(label); } else { @@ -870,10 +1008,12 @@ export default { } // Set what addObserver() should do for one account, either everywhere or (when `resourceUrl` - // is given) at one binding only. - // Body: {"label": "...", "allow": false, "reason": "...", "resourceUrl": "..."} + // is given) at one binding only. With `subresource` too, it sets whether the tracked binding's + // verifyObserverAccess() admits the account for that one subresource. + // Body: {"label": "...", "allow": false, "reason": "...", "resourceUrl": "...", + // "subresource": "..."} if (url.pathname === "/control/verify-outcome" && req.method === "POST") { - const { label, allow, reason, resourceUrl } = body as Record; + const { label, allow, reason, resourceUrl, subresource } = body as Record; if (!isNonEmptyString(label)) return badRequest("`label` must be a non-empty string"); if (typeof allow !== "boolean") return badRequest("`allow` must be a boolean"); if (reason !== undefined && typeof reason !== "string") { @@ -882,14 +1022,41 @@ export default { if (resourceUrl !== undefined && !isNonEmptyString(resourceUrl)) { return badRequest("`resourceUrl` must be a non-empty string when present"); } + if (subresource !== undefined && !isNonEmptyString(subresource)) { + return badRequest("`subresource` must be a non-empty string when present"); + } + if (subresource !== undefined && resourceUrl === undefined) { + return badRequest("`subresource` needs `resourceUrl`"); + } const outcome: VerifyOutcome = allow ? { allow: true } : { allow: false, reason: reason ?? "The test gatekeeper refused this account." }; - await control(ctx.exports).setVerifyOutcome(label, outcome, resourceUrl); + await control(ctx.exports).setVerifyOutcome(label, outcome, resourceUrl, subresource); + return new Response(null, { status: 204 }); + } + + // Make the binding at `resourceUrl` advertise kernel observer tracking. Set before connecting + // for a binding tracked from creation, or after to upgrade a legacy one. + // Body: {"resourceUrl": "..."} + if (url.pathname === "/control/observer-tracking" && req.method === "POST") { + const { resourceUrl } = body as Record; + if (!isNonEmptyString(resourceUrl)) { + return badRequest("`resourceUrl` must be a non-empty string"); + } + await control(ctx.exports).enableObserverTracking(resourceUrl); return new Response(null, { status: 204 }); } + // Body: {"resourceUrl": "..."} -> {"count": number}, one per getHistoricalSubresources(). + if (url.pathname === "/control/history-export-count" && req.method === "POST") { + const { resourceUrl } = body as Record; + if (!isNonEmptyString(resourceUrl)) { + return badRequest("`resourceUrl` must be a non-empty string"); + } + return Response.json({ count: await control(ctx.exports).getHistoryExportCount(resourceUrl) }); + } + // Read back the addObserver()/removeObserver() calls one binding's gatekeeper has seen, in // order. // Body: {"resourceUrl": "..."} -> {"events": [{"resourceUrl", "type", "id"}, ...]} @@ -1022,13 +1189,26 @@ export default { return Response.json({ responses: await control(ctx.exports).getGadgetResponses(messageKey) }); } - // Fire a hook through the initiator its controller's enable() stored. - // Body: {"key": "...", "value": number} -> {"fired": true} | {"error": string} + // Fire a hook through the initiator its controller's enable() stored, as an event about the + // binding at `resourceUrl` whose reads depended on `subresources` (see TestControl.fireHook()). + // Body: {"key": "...", "value": number, "resourceUrl"?: "...", "subresources"?: string[]} + // -> {"fired": true} | {"error": string} if (url.pathname === "/control/fire-hook" && req.method === "POST") { - const { key, value } = body as Record; + const { key, value, resourceUrl, subresources } = body as Record; if (!isNonEmptyString(key)) return badRequest("`key` must be a non-empty string"); if (typeof value !== "number") return badRequest("`value` must be a number"); - return Response.json(await control(ctx.exports).fireHook(key, value)); + if (resourceUrl !== undefined && !isNonEmptyString(resourceUrl)) { + return badRequest("`resourceUrl` must be a non-empty string when present"); + } + if (subresources !== undefined && + (!Array.isArray(subresources) || !subresources.every(isNonEmptyString))) { + return badRequest("`subresources` must be an array of non-empty strings"); + } + if (subresources !== undefined && resourceUrl === undefined) { + return badRequest("`subresources` needs `resourceUrl`"); + } + return Response.json( + await control(ctx.exports).fireHook(key, value, resourceUrl, subresources)); } // Call the stub a session's keepSelfStub() stored. diff --git a/packages/typed-storage/__tests__/index.test.ts b/packages/typed-storage/__tests__/index.test.ts index d39c66702d..b70f26ca85 100644 --- a/packages/typed-storage/__tests__/index.test.ts +++ b/packages/typed-storage/__tests__/index.test.ts @@ -1,6 +1,6 @@ import { expect, expectTypeOf, it, describe } from "vitest" -import { createTypedStorage, collection, singleton, Singleton, SingletonSchema, UniqueIndex, - NonUniqueIndex } +import { createTypedStorage, collection, singleton, keyString, Singleton, SingletonSchema, + UniqueIndex, NonUniqueIndex } from "../src/index.js"; import { DurableObjectListOptions, DurableObjectStorage } from "@cloudflare/workers-types/experimental"; @@ -866,6 +866,10 @@ describe("non-unique index by number", () => { expect([...index.list({limit: 2, reverse: true})]).toStrictEqual([CAROL, ALICE, BOB]); }); + it("lists integer keys as keyString() encodes them", () => { + expect([...index.listKeys()]).toStrictEqual([keyString(1), keyString(4), keyString(8)]); + }); + it("supports delete", () => { expect(index.delete(8)).toStrictEqual(2); expect(index.delete(8)).toStrictEqual(0); @@ -920,12 +924,21 @@ describe("non-unique index by array", () => { [DAVE, DAVE, CAROL, BOB, ALICE]); }); + it("supports listKeys", () => { + expect([...index.listKeys()]).toStrictEqual(["admin", "everyone", "interns"]); + expect([...index.listKeys({reverse: true})]).toStrictEqual(["interns", "everyone", "admin"]); + expect([...index.listKeys({prefix: "e"})]).toStrictEqual(["everyone"]); + expect([...index.listKeys({startAfter: "admin", limit: 1})]).toStrictEqual(["everyone"]); + }); + it("supports delete", () => { expect(index.delete("admin")).toStrictEqual(2); expect(index.delete("admin")).toStrictEqual(0); expect(index.delete("nobody")).toStrictEqual(0); expect([...index.list()]).toStrictEqual([BOB, DAVE, DAVE]); + // A key goes once its last record does. + expect([...index.listKeys()]).toStrictEqual(["everyone", "interns"]); }); }); diff --git a/packages/typed-storage/src/index.ts b/packages/typed-storage/src/index.ts index e7de1e29e9..80383383f4 100644 --- a/packages/typed-storage/src/index.ts +++ b/packages/typed-storage/src/index.ts @@ -80,6 +80,13 @@ export interface NonUniqueIndex): Iterable; list(options?: ListOptions): Iterable; + + /** + * List the distinct keys at least one record is indexed under, in key order, without reading + * the records: string keys verbatim, integer keys as keyString() encodes them. `options` ranges + * and pages over the keys; `dedupe` does not apply. + */ + listKeys(options?: ListOptions): Iterable; delete(key: Key): number; /** @@ -684,6 +691,9 @@ function createCollection< } } }, + listKeys(options: ListOptions = {}): Iterable { + return idxKv.listKeys(options); + }, delete(key: Key): number { let id = idxKv.get(key); if (id === undefined) { diff --git a/packages/workshop-backend/__tests__/exports-binding.test.ts b/packages/workshop-backend/__tests__/exports-binding.test.ts index dc97fc0dc7..cd96413d85 100644 --- a/packages/workshop-backend/__tests__/exports-binding.test.ts +++ b/packages/workshop-backend/__tests__/exports-binding.test.ts @@ -67,7 +67,7 @@ describe("gadget bindings in the agent's env", () => { expect(agentEnv).toEqual({ GIT: { type: "git" }, APP: { rpc: { type: "gadget", id: 100 }, exports: { type: "exports", id: 100 } }, - DB: { type: "gatekeeper", id: 5 }, + DB: { type: "gatekeeper", id: 5, executionId: "exec-1" }, }); // A gadget's own env keeps direct RPC stubs and offers no exports. diff --git a/packages/workshop-backend/__tests__/git-binding.test.ts b/packages/workshop-backend/__tests__/git-binding.test.ts index 011fee7da4..76343cec25 100644 --- a/packages/workshop-backend/__tests__/git-binding.test.ts +++ b/packages/workshop-backend/__tests__/git-binding.test.ts @@ -351,7 +351,7 @@ describe("env.GIT presence", () => { impl.storage.gatekeepers.put({ id: 1, resourceTitle: "Conn", class: {} as any }); expect(impl.getEnvForAgent(1, { GIT: { type: "workpiece", id: 1 } }, "exec-1")) - .toEqual({ GIT: { type: "gatekeeper", id: 1 } }); + .toEqual({ GIT: { type: "gatekeeper", id: 1, executionId: "exec-1" } }); })); it("reserves the name for new gadget bindings", () => withImpl(async impl => { diff --git a/packages/workshop-backend/__tests__/observer-exclude-scope.test.ts b/packages/workshop-backend/__tests__/observer-exclude-scope.test.ts index a23e6b4d9c..3ff7b6fdde 100644 --- a/packages/workshop-backend/__tests__/observer-exclude-scope.test.ts +++ b/packages/workshop-backend/__tests__/observer-exclude-scope.test.ts @@ -149,6 +149,7 @@ describe("excludeObservers against the observer's verification scope", () => { let releaseRemove!: () => void; let removeReached = new Promise(resolve => { impl.getGatekeeperFacet = (id: number) => ({ + describe: async () => ({}), removeObserver: async (observerId: string) => { events.push("remove:start"); resolve(); @@ -250,6 +251,7 @@ describe("a bound agent spawner's env keeps its targets in use scope", () => { bindSpawnerIntoGadget(impl); let added: string[] = []; impl.getGatekeeperFacet = (id: number) => ({ + describe: async () => ({}), addObserver: async (observerId: string) => { added.push(`${id}:${observerId}`); }, removeObserver: async () => {}, }); diff --git a/packages/workshop-backend/__tests__/observer-ledger.test.ts b/packages/workshop-backend/__tests__/observer-ledger.test.ts new file mode 100644 index 0000000000..d55a7ae093 --- /dev/null +++ b/packages/workshop-backend/__tests__/observer-ledger.test.ts @@ -0,0 +1,1861 @@ +// Kernel-owned observer tracking (observer-ledger.ts): connections that advertise +// ResourceDescription.supportsObserverTracking coexist with legacy ones, import prior reads once, +// report subresources on every real observation through authorizeTrackedObservation(), and are +// verified by the connection's own verifyObserverAccess(), handed each observer's verifier -- +// failing closed on every malformed or failed answer. +// +// Runs against a real OverseerDurableObject (the TEST_OVERSEER binding, like +// observer-verification-failure.test.ts); gatekeeper facets, User DOs, verifiers, and the sharing +// graph are the fakes. + +import { describe, expect, it } from "vitest"; +import { env } from "cloudflare:workers"; +import { abortAllDurableObjects, runInDurableObject } from "cloudflare:test"; +import type { + ObservationAuthorizer, ObservationDescription, TrackedObservationDescription, +} from "@gadgets/workshop-shared/gatekeeper"; +import type { OverseerDurableObject } from "../src/overseer.js"; +import { + observerRequirementKey, observerRequirementSubresource, type GatekeeperCaller, + type GatekeeperRecord, type ObserverContext, +} from "../src/storage-schema/overseer-storage.js"; +import { openFakeOverseer } from "./fixtures.js"; +import type { BoundHookRecord } from "../src/storage-schema/overseer-storage.js"; + +declare module "cloudflare:workers" { + interface ProvidedEnv { + TEST_OVERSEER: DurableObjectNamespace; + } +} + +const LEGACY = 1; +const TRACKED = 2; +const GADGET = 100; +const WORKSPACE: ObserverContext = { type: "workspace" }; +const chat = (chatId: number): ObserverContext => ({ type: "chat", chatId }); +const gadget = (gadgetId: number): ObserverContext => ({ type: "gadget", gadgetId }); + +type ObserverImpl = OverseerDurableObject["impl"]; +// Facet factories are replaced in these tests, so the stored gatekeeper class is never invoked. +const fakeGatekeeperClass = {} as unknown as GatekeeperRecord["class"]; + +type Check = { profileId: string, ids: string[] }; + +// A vendor's verifier for one account: answers which of `ids` it cannot read, through a +// vendor-specific method only its gatekeeper calls. +type VendorVerifier = { deniedSubresources(ids: string[]): Promise }; +type ObserverToCheck = { id: string, verifier: VendorVerifier }; +type VerifyObserverAccess = (observers: ObserverToCheck[], ids: string[]) => unknown; + +// One test's fakes: who holds which role, what each profile's verifier answers, and what was +// asked of them. +type World = { + impl: any; + instance: OverseerDurableObject; + roles: Map; + // Per profile: the whole verifier, or an Error its User DO's getVerifier() throws. + verifiers: Map; + checks: Check[]; + // Restarts the workspace scheduled. Recorded instead of run: a real one aborts this DO, so a + // test that needs the restart itself aborts the DO and opens it again. + restarts: string[]; + clientUser(profileId: string): any; +}; + +// `answer` gives the denied IDs from `ids`. +function observerVerifier(world: () => World, profileId: string, + answer: (ids: string[]) => string[] | Promise = () => []): + VendorVerifier { + return { + deniedSubresources: async (ids: string[]) => { + world().checks.push({ profileId, ids }); + return await answer(ids); + }, + }; +} + +// A tracked connection's verifyObserverAccess(): asks every observer's own verifier at once, and +// allows each one whose verifier denies none of `ids`. +async function verifyEachObserver(observers: ObserverToCheck[], ids: string[]) { + let denied = await Promise.all( + observers.map(observer => observer.verifier.deniedSubresources(ids))); + return { + allowedObserverIds: observers.filter((_, i) => denied[i].length === 0) + .map(observer => observer.id), + }; +} + +// `ids` followed by a hole. +function sparseIds(ids: string[]): string[] { + let sparse = [...ids]; + sparse.length++; + return sparse; +} + +// Answers a connection's verifyObserverAccess() may give that must refuse every observer in the +// call (null: the connection lacks the method). +const refusingAnswers: [string, VerifyObserverAccess | null][] = [ + ["an omitted observer", () => ({ allowedObserverIds: [] })], + ["missing allowed IDs", () => ({})], + ["non-array allowed IDs", observers => ({ allowedObserverIds: observers[0].id })], + ["sparse allowed IDs", + observers => ({ allowedObserverIds: sparseIds(observers.map(observer => observer.id)) })], + ["non-string allowed IDs", + observers => ({ allowedObserverIds: [...observers.map(observer => observer.id), false] })], + ["an allowed ID that was not asked about", + observers => ({ allowedObserverIds: [...observers.map(observer => observer.id), "obs-x"] })], + ["an unwrapped array", observers => observers.map(observer => observer.id)], + ["a missing method", null], + ["a failed check", () => { throw new Error("provider down"); }], +]; + +let doCounter = 0; + +async function inOverseer(name: string, fn: (world: World) => Promise): Promise { + let stub = env.TEST_OVERSEER.getByName(name); + await runInDurableObject(stub, async (instance: OverseerDurableObject) => { + // (Bracket access reaches the private field with its real type.) + let impl = instance["impl"]; + impl.ownerProfileId = "owner"; + let world: World = { + impl, + instance, + roles: new Map(), + verifiers: new Map(), + checks: [], + restarts: [], + clientUser: (profileId: string) => ({ + getVerifier: async () => { + let verifier = world.verifiers.get(profileId); + if (verifier instanceof Error) throw verifier; + return verifier ?? observerVerifier(() => world, profileId); + }, + describeConnectedAccount: async () => null, + listProvidedAccounts: async () => [], + }), + }; + world.impl.scheduleAccessRestart = async (reason: string) => { world.restarts.push(reason); }; + // Tracked connections verify an observation's audience through their facet. + world.impl.getGatekeeperFacet = async () => ({ verifyObserverAccess: verifyEachObserver }); + world.impl.getSharingManager = async () => ({ + getEffectiveRole: (id: string) => id === "owner" ? "build" : world.roles.get(id), + computeEffectiveRoles: () => new Map(world.roles), + }); + world.impl.users = { + idFromName: (name: string) => name, + get: (id: string) => world.clientUser(id), + }; + await fn(world); + }); +} + +function withWorld(fn: (world: World) => Promise): Promise { + return inOverseer(`observer-ledger-${++doCounter}`, fn); +} + +function spec(vendorId: string, id: number) { + return { + type: "gatekeeper" as const, vendorId, resourceUrl: `https://example.com/${id}`, + typeUrlPattern: "https://*", + }; +} + +function seedLegacy(impl: ObserverImpl, id = LEGACY, extra: Partial = {}): void { + impl.storage.gatekeepers.put({ + id, resourceTitle: `Legacy ${id}`, class: fakeGatekeeperClass, + creationSpec: spec("legacyvendor", id), ...extra, + }); +} + +function seedTracked(impl: ObserverImpl, id = TRACKED): void { + impl.storage.gatekeepers.put({ + id, resourceTitle: `Tracked ${id}`, class: fakeGatekeeperClass, + creationSpec: spec("trackedvendor", id), observerTracking: true, + }); +} + +// `bindings` maps binding names to connection ids; a bound connection is in "use" scope. +function seedGadget(impl: ObserverImpl, id = GADGET, bindings: Record = {}): void { + impl.storage.gadgets.put({ + type: "gadget", id, title: "G", created: new Date(0), bindingName: `G${id}`, + bindings: Object.fromEntries(Object.entries(bindings).map(([name, target]) => [name, { target }])), + }); +} + +// Requirements are tracked observation audit records, attributed to a context by their caller. +function addRows(impl: ObserverImpl, context: ObserverContext, gatekeeperId: number, ids: string[]): void { + let caller: GatekeeperCaller = context.type === "chat" ? { from: "agent", chatId: context.chatId } + : context.type === "gadget" ? { from: "gadget", gadgetId: context.gadgetId } + : { from: "user" }; + let id = impl.storage.nextActionId.get(); + impl.storage.nextActionId.put(id + 1); + impl.storage.actions.put({ + id, gatekeeperId, caller, createdAt: new Date(0), state: "approved", + type: "observation", description: trackedObservation(ids), + }); +} + +// Every subresource the requirement index holds on `context`, whether or not its connection +// still exists. +function rows(impl: ObserverImpl, context: ObserverContext): string[] { + let connections = new Set(Array.from(impl.storage.actions.list(), record => record.gatekeeperId)); + let found: string[] = []; + for (let gatekeeperId of connections) { + let prefix = observerRequirementKey(gatekeeperId, context, ""); + for (let key of impl.storage.actions.byObserverRequirement.listKeys({ prefix })) { + found.push(`${gatekeeperId}:${observerRequirementSubresource(key)}`); + } + } + return found.toSorted(); +} + +// What admission requires of `role` through every connection in the workspace. +function requirements(impl: ObserverImpl, role: "build" | "use"): Map { + return impl.observerLedger.requiredSubresources(role, Array.from(impl.storage.gatekeepers.list())); +} + +describe("byte-bounded observation persistence", () => { + for (let historical of [false, true]) { + let label = historical ? "historical import" : "live observation"; + + it(`keeps every requirement from an oversized ${label}`, () => withWorld(async world => { + let { impl } = world; + // Bound, so "use" collaborators are verified against it. + seedGadget(impl, GADGET, { C: historical ? LEGACY : TRACKED }); + let ids = Array.from({ length: historical ? 20_000 : 10_000 }, + (_, i) => String(i).padStart(512, historical ? "x" : "界")); + if (historical) { + seedLegacy(impl); + fakeFacets(impl, { + advertise: new Set([LEGACY]), exports: new Map([[LEGACY, async () => ids]]), + }); + await impl.observerLedger.initialize(LEGACY); + } else { + seedTracked(impl); + await impl.authorizeTrackedObservation(TRACKED, trackedObservation(ids), + { from: "gadget", gadgetId: GADGET }); + } + let connection = historical ? LEGACY : TRACKED; + expect(requirements(impl, "use").get(connection)?.toSorted()) + .toEqual(ids.toSorted()); + world.roles.set("carol", "use"); + world.verifiers.set("carol", observerVerifier(() => world, "carol", + batch => batch.filter(id => id === ids.at(-1)))); + await expect(impl.ensureObserver("carol", world.clientUser("carol"), "use", configureAll)) + .rejects.toThrow(/could not confirm/); + expect(impl.storage.observers.get("carol")).toBeUndefined(); + world.verifiers.delete("carol"); + await impl.ensureObserver("carol", world.clientUser("carol"), "use", configureAll); + expect(impl.storage.observers.get("carol").accountChoices).toEqual({ [connection]: connection * 10 }); + })); + + it(`rolls back every chunk and policy change after a failed ${label}`, () => + withWorld(async ({ impl }) => { + let ids = Array.from({ length: historical ? 20_000 : 10_000 }, + (_, i) => String(i).padStart(512, historical ? "x" : "界")); + if (historical) { + seedLegacy(impl); + fakeFacets(impl, { + advertise: new Set([LEGACY]), exports: new Map([[LEGACY, async () => ids]]), + }); + } else { + seedTracked(impl); + } + let sharing = await impl.getSharingManager(); + sharing.computeAffectedByOwnerInvitesOnly = () => []; + impl.getSharingManager = async () => sharing; + let before = impl.storage.nextActionId.get(); + let put = impl.storage.actions.put.bind(impl.storage.actions); + let writes = 0; + impl.storage.actions.put = (record: Parameters[0]) => { + if (++writes === 2) throw new Error("later chunk failed"); + return put(record); + }; + let run = () => historical ? impl.observerLedger.initialize(LEGACY) + : impl.authorizeTrackedObservation(TRACKED, trackedObservation(ids, { + containsRestrictedData: true, ownerInvitesOnly: true, + }), { from: "user" }); + try { + await expect(run()).rejects.toThrow("later chunk failed"); + } finally { + impl.storage.actions.put = put; + } + expect(impl.storage.nextActionId.get()).toBe(before); + expect([...impl.storage.actions.list()]).toEqual([]); + expect(requirements(impl, "build").size).toBe(0); + expect(impl.storage.containsRestrictedData.get()).toBeFalsy(); + expect(impl.storage.ownerInvitesOnly.get()).toBeFalsy(); + if (historical) expect(impl.observerLedger.isTracked(LEGACY)).toBe(false); + await run(); + expect(requirements(impl, "build").get(historical ? LEGACY : TRACKED) + ?.toSorted()).toEqual(ids.toSorted()); + if (historical) { + expect(impl.observerLedger.isTracked(LEGACY)).toBe(true); + } else { + expect(impl.storage.containsRestrictedData.get()).toBe(true); + expect(impl.storage.ownerInvitesOnly.get()).toBe(true); + } + })); + } +}); + +type HistoryCursor = { next(): Promise, [Symbol.dispose](): void }; + +// A historical export as a gatekeeper pages it: at most `pageSize` IDs per page, after an empty +// page that must not end the walk. +function historyCursor(ids: string[], pageSize = 10_000): HistoryCursor { + let pages: string[][] = [[]]; + for (let i = 0; i < ids.length; i += pageSize) pages.push(ids.slice(i, i + pageSize)); + return { next: async () => pages.shift() ?? null, [Symbol.dispose]() {} }; +} + +// Facets: legacy ones advertise nothing; tracked ones advertise support and refuse addObserver(), +// as the contract requires. An export given as IDs is paged by historyCursor(). `verify` replaces +// verifyObserverAccess() (null removes it). +function fakeFacets(impl: any, opts: { + adds?: number[], + exports?: Map Promise>, + advertise?: Set, + verify?: VerifyObserverAccess | null, +} = {}): void { + let tracked = (id: number) => + opts.advertise?.has(id) || impl.storage.gatekeepers.get(id)?.observerTracking; + impl.getGatekeeperFacet = async (id: number) => ({ + describe: async () => ({ + url: `https://example.com/${id}`, title: `Connection ${id}`, snippet: "", + suggestedBindingName: "C", tsType: "C", + ...(tracked(id) ? { supportsObserverTracking: true } : {}), + }), + getHistoricalSubresources: async () => { + let history = await (opts.exports?.get(id)?.() ?? []); + return Array.isArray(history) ? historyCursor(history) : history; + }, + ...(opts.verify === null ? {} : { verifyObserverAccess: opts.verify ?? verifyEachObserver }), + addObserver: async () => { + if (tracked(id)) throw new Error("This connection is tracked by the Workshop."); + opts.adds?.push(id); + }, + removeObserver: async () => {}, + }); +} + +const configureAll = { + configure: async (needs: { gatekeeperId: number }[]) => + needs.map(need => ({ gatekeeperId: need.gatekeeperId, accountId: need.gatekeeperId * 10 })), +} as any; + +function observation(extra: Partial = {}): ObservationDescription { + return { title: "Read", description: "A read", ...extra }; +} + +function trackedObservation(subresources: string[], + extra: Partial = {}): + TrackedObservationDescription { + return { title: "Read", description: "A read", ...extra, subresources }; +} + +// Only controller/callback capabilities are faked; the hook row, ledger, and action log use +// real DO storage. Functions cannot be structured-cloned, so attach capabilities on reads. +async function startTrackedHook(world: World, gadgetId?: number) { + let { impl, instance } = world; + let deliveries: string[] = []; + let hooks: ObserverImpl["storage"]["boundHooks"] = impl.storage.boundHooks; + let get = hooks.get.bind(hooks); + let put = hooks.put.bind(hooks); + // Inert persisted capability slots are replaced by local gatekeeper/gadget fakes below. + let storedController = {} as BoundHookRecord["controller"]; + let storedCallback = {} as BoundHookRecord["callback"]; + let controller = { disable: async () => {} } as unknown as BoundHookRecord["controller"]; + let callback = { + deliver: async (payload: string) => { deliveries.push(payload); }, + } as unknown as BoundHookRecord["callback"]; + put({ + id: 5, actionId: 999, gatekeeperId: TRACKED, vendorId: "trackedvendor", gadgetId, + controller: storedController, callback: storedCallback, enabled: true, + description: { title: "Hook", description: "Delivers events" }, + }); + hooks.get = (id: number) => { + let record = get(id); + return record && { ...record, controller, callback }; + }; + hooks.put = (record: BoundHookRecord) => put({ + ...record, controller: storedController, callback: storedCallback, + }); + // The unit-test worker has no admin-config KV binding. Replace only that external read. + let host = instance as unknown as { env: { BLUEPRINTS: { get(): Promise } } }; + host.env = { ...host.env, BLUEPRINTS: { get: async () => null } }; + fakeFacets(impl); + let firing = await instance.startHook(5); + // The firing wrapper preserves the stored callback's deliver method across its RPC type. + let deliveryCallback = firing.callback as unknown as { deliver(payload: string): Promise }; + let client = await openFakeOverseer({ boundHooks: hooks, actions: impl.storage.actions }, { + impl: { deleteHook: (id: number) => impl.deleteHook(id) }, + }); + return { + approvalQueue: firing.approvalQueue, + // Await once here: native RPC promises fork on each .then, including assertion handlers. + deliver: async (payload: string) => { await deliveryCallback.deliver(payload); }, + deliveries, + revoke: (operation: "disable" | "delete") => operation === "disable" + ? client.disableHook(5) : client.deleteHook(5), + [Symbol.dispose]() { + firing.callback[Symbol.dispose](); + (client as typeof client & Disposable)[Symbol.dispose](); + hooks.get = get; + hooks.put = put; + }, + }; +} + +describe("tracked hook observations", () => { + it.each([ + { hookGadget: undefined, defaultGadget: undefined, context: WORKSPACE }, + { hookGadget: GADGET, defaultGadget: undefined, context: gadget(GADGET) }, + { hookGadget: undefined, defaultGadget: GADGET, context: gadget(GADGET) }, + ])("delivers through the firing queue and records requirements on the gadget it wakes " + + "(hook gadgetId: $hookGadget, default gadget: $defaultGadget)", + ({ hookGadget, defaultGadget, context }) => withWorld(async world => { + let { impl } = world; + seedTracked(impl); + seedGadget(impl); + impl.defaultGadgetId = defaultGadget; + addRows(impl, gadget(GADGET), TRACKED, ["gadget-only"]); + for (let [profileId, role] of [["bob", "build"], ["carol", "use"]] as const) { + world.roles.set(profileId, role); + impl.storage.observers.put( + { profileId, observerId: `obs-${profileId}`, accountChoices: { [TRACKED]: 20 } }); + } + using firing = await startTrackedHook(world, hookGadget); + let before = impl.storage.nextActionId.get(); + + await firing.approvalQueue.authorizeTrackedObservation(trackedObservation(["hook/event"])); + await firing.deliver("evt"); + + expect(firing.deliveries).toEqual(["evt"]); + expect(world.checks).toHaveLength(2); + for (let profileId of ["bob", "carol"]) { + expect(world.checks).toContainEqual({ profileId, ids: ["hook/event"] }); + } + let onGadget = context.type === "gadget"; + expect(rows(impl, WORKSPACE)).toEqual(onGadget ? [] : ["2:hook/event"]); + expect(rows(impl, gadget(GADGET))) + .toEqual(onGadget ? ["2:gadget-only", "2:hook/event"] : ["2:gadget-only"]); + expect(impl.storage.actions.get(before)).toMatchObject({ + gatekeeperId: TRACKED, type: "observation", state: "approved", + description: { subresources: ["hook/event"] }, + }); + expect(impl.storage.actions.get(before).caller) + .toEqual(onGadget ? { from: "hook", gadgetId: GADGET } : { from: "hook" }); + expect(impl.storage.nextActionId.get()).toBe(before + 1); + })); + + it.each(["build", "use"] as const)( + "does not deliver or commit a read denied by a %s collaborator", role => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + world.roles.set("carol", role); + impl.storage.observers.put( + { profileId: "carol", observerId: "obs-c", accountChoices: { [TRACKED]: 20 } }); + world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => ids)); + using firing = await startTrackedHook(world); + let before = impl.storage.nextActionId.get(); + let deliver = async () => { + await firing.approvalQueue.authorizeTrackedObservation(trackedObservation(["denied"])); + await firing.deliver("evt"); + }; + + await expect(deliver()).rejects.toThrow(/could not be confirmed/); + + expect(world.checks).toEqual([{ profileId: "carol", ids: ["denied"] }]); + expect(firing.deliveries).toEqual([]); + expect(rows(impl, WORKSPACE)).toEqual([]); + expect(impl.storage.nextActionId.get()).toBe(before); + expect(impl.storage.actions.get(before)).toBeUndefined(); + })); + + // Requirements stay in the log either way. A disabled or deleted hook no longer makes its + // connection reachable from a "use" session, so as under the legacy protocol only "build" + // collaborators are still verified against it. + for (let operation of ["disable", "delete"] as const) { + it.each(["build", "use"] as const)( + `retains delivered hook requirements after ${operation} and a DO restart (%s admission)`, + async role => { + let name = `observer-ledger-hook-history-${++doCounter}`; + let auditId = -1; + await inOverseer(name, async world => { + let { impl } = world; + seedTracked(impl); + using firing = await startTrackedHook(world); + auditId = impl.storage.nextActionId.get(); + await firing.approvalQueue.authorizeTrackedObservation(trackedObservation(["past/event"])); + await firing.deliver("evt"); + expect(firing.deliveries).toEqual(["evt"]); + await firing.revoke(operation); + expect(impl.storage.boundHooks.get(5)?.enabled).not.toBe(true); + }); + + await abortAllDurableObjects(); + + await inOverseer(name, async world => { + let { impl } = world; + fakeFacets(impl); + world.roles.set("carol", role); + world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => ids)); + + if (role === "build") { + await expect(impl.ensureObserver("carol", world.clientUser("carol"), role, configureAll)) + .rejects.toThrow(/could not confirm/); + expect(impl.storage.observers.get("carol")).toBeUndefined(); + } else { + await impl.ensureObserver("carol", world.clientUser("carol"), role, configureAll); + expect(world.checks).toEqual([]); + } + expect(rows(impl, WORKSPACE)).toEqual(["2:past/event"]); + expect(requirements(impl, role).get(TRACKED)).toEqual(["past/event"]); + expect(impl.storage.actions.get(auditId)).toMatchObject({ + caller: { from: "hook" }, description: { subresources: ["past/event"] }, + }); + expect(impl.storage.boundHooks.get(5)?.enabled).not.toBe(true); + }); + }); + } + + it.each(["disable", "delete", "restart"] as const)( + "refuses authorization and policy/log writes when %s races verification", operation => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + world.roles.set("carol", "use"); + impl.storage.observers.put( + { profileId: "carol", observerId: "obs-c", accountChoices: { [TRACKED]: 20 } }); + let release!: () => void; + let reached!: () => void; + let parked = new Promise(resolve => { release = resolve; }); + let checking = new Promise(resolve => { reached = resolve; }); + world.verifiers.set("carol", observerVerifier(() => world, "carol", async () => { + reached(); + await parked; + return []; + })); + using firing = await startTrackedHook(world); + let before = impl.storage.nextActionId.get(); + let authorize = firing.approvalQueue.authorizeTrackedObservation(trackedObservation( + ["racing/event"], { containsRestrictedData: true, ownerInvitesOnly: true })) + .then(() => ({ authorized: true, error: "" }), + error => ({ authorized: false, error: String(error) })); + await checking; + let refusal = operation === "restart" ? /restarting/ : /deleted or disabled/; + if (operation === "restart") { + impl.restartForObserverTracking(TRACKED); + expect(impl.gatekeeperUsable(TRACKED)).toBe(false); + } else { + await firing.revoke(operation); + } + release(); + + expect(await authorize).toEqual({ + authorized: false, error: expect.stringMatching(refusal), + }); + await expect(firing.deliver("evt")).rejects.toThrow(refusal); + expect(firing.deliveries).toEqual([]); + expect(rows(impl, WORKSPACE)).toEqual([]); + expect(impl.storage.nextActionId.get()).toBe(before); + expect(impl.storage.actions.get(before)).toBeUndefined(); + expect(impl.storage.containsRestrictedData.get()).toBe(false); + expect(impl.storage.ownerInvitesOnly.get()).toBe(false); + })); +}); + +describe("legacy and tracked connections coexist", () => { + it("verifies each by its own protocol and keeps the legacy exclusion path", () => + withWorld(async world => { + let { impl } = world; + seedLegacy(impl); + seedTracked(impl); + addRows(impl, WORKSPACE, TRACKED, ["a"]); + let adds: number[] = []; + fakeFacets(impl, { adds }); + world.roles.set("carol", "build"); + + await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); + + // The legacy connection registered her; the tracked one was checked statelessly instead. + expect(adds).toEqual([LEGACY]); + expect(world.checks).toEqual([{ profileId: "carol", ids: ["a"] }]); + let record = impl.storage.observers.get("carol"); + expect(record.accountChoices).toEqual({ [LEGACY]: 10, [TRACKED]: 20 }); + + // The legacy connection still names observers to exclude, and is still blocked by them. + await expect(impl.authorizeObservation(LEGACY, + observation({ excludeObservers: [record.observerId] }), { from: "user" })) + .rejects.toThrow(/not permitted to see/); + // A tracked one may not mix the two. + let mixed = { ...trackedObservation([]), excludeObservers: [record.observerId] }; + await expect(impl.authorizeTrackedObservation(TRACKED, mixed, { from: "user" })) + .rejects.toThrow(/cannot exclude observers/); + })); + + it("imports a legacy connection at its first tracked observation, and refuses that read", () => + withWorld(async world => { + let { impl } = world; + seedLegacy(impl); + fakeFacets(impl, { + advertise: new Set([LEGACY]), + exports: new Map([[LEGACY, async () => ["history"]]]), + }); + // A session minted before the Workshop noticed the upgrade makes a tracked observation: the + // read is refused, not silently recorded or dropped, and the upgrade it reveals is imported. + await expect(impl.authorizeTrackedObservation(LEGACY, trackedObservation(["x"]), + { from: "user" })) + .rejects.toThrow(/must be set up/); + expect(impl.observerLedger.isTracked(LEGACY)).toBe(true); + expect(rows(impl, WORKSPACE)).toEqual(["1:history"]); + })); + + it("refuses subresources sent through the legacy method, recording and importing nothing", () => + withWorld(async world => { + let { impl } = world; + seedLegacy(impl); + fakeFacets(impl, { advertise: new Set([LEGACY]) }); + let before = impl.storage.nextActionId.get(); + // Only authorizeTrackedObservation() checks them: accepting them here would let the read + // through without checking anyone, as a Workshop that predates tracking would. + let misrouted = { ...observation(), subresources: ["x"] }; + await expect(impl.authorizeObservation(LEGACY, misrouted, { from: "user" })) + .rejects.toThrow(/authorizeTrackedObservation/); + expect(impl.storage.nextActionId.get()).toBe(before); + expect(impl.observerLedger.isTracked(LEGACY)).toBe(false); + })); + + it("refuses a tracked observation through a connection that doesn't advertise tracking", () => + withWorld(async world => { + let { impl } = world; + seedLegacy(impl); + fakeFacets(impl); + let before = impl.storage.nextActionId.get(); + await expect(impl.authorizeTrackedObservation(LEGACY, trackedObservation(["x"]), + { from: "user" })).rejects.toThrow(/does not advertise observer tracking/); + expect(impl.storage.nextActionId.get()).toBe(before); + expect(impl.observerLedger.isTracked(LEGACY)).toBe(false); + })); +}); + +describe("observations through a tracked connection", () => { + it("accept an explicit [] and refuse omitted IDs", () => withWorld(async world => { + let { impl } = world; + seedTracked(impl); + let before = impl.storage.nextActionId.get(); + + await impl.authorizeTrackedObservation(TRACKED, trackedObservation([]), { from: "user" }); + expect(impl.storage.nextActionId.get()).toBe(before + 1); // audited + expect(rows(impl, WORKSPACE)).toEqual([]); + + await expect(impl.authorizeObservation(TRACKED, observation(), { from: "user" })) + .rejects.toThrow(/must report its subresources/); + expect(impl.storage.nextActionId.get()).toBe(before + 1); + })); + + it("refuse IDs that are not well-formed Unicode", () => withWorld(async world => { + let { impl } = world; + seedTracked(impl); + let before = impl.storage.nextActionId.get(); + + // Storage would turn both lone surrogates into the same replacement characters. + await expect(impl.authorizeTrackedObservation(TRACKED, + trackedObservation(["doc:\uD800", "doc:\uDBFF"]), { from: "user" })) + .rejects.toThrow(/well-formed/); + expect(impl.storage.nextActionId.get()).toBe(before); + })); + + it.each(refusingAnswers)("fail closed on %s from the connection's access check", + (_name, verify) => withWorld(async world => { + let { impl } = world; + seedTracked(impl); + let registrations: number[] = []; + fakeFacets(impl, { adds: registrations, verify }); + world.roles.set("carol", "use"); + impl.storage.observers.put( + { profileId: "carol", observerId: "obs-c", accountChoices: { [TRACKED]: 20 } }); + let before = impl.storage.nextActionId.get(); + + await expect(impl.authorizeTrackedObservation(TRACKED, trackedObservation(["x", "y"]), + { from: "user" })) + .rejects.toThrow(/could not be confirmed/); + expect(rows(impl, WORKSPACE)).toEqual([]); + expect(impl.storage.nextActionId.get()).toBe(before); + expect(registrations).toEqual([]); + })); + + it("check configured observers who are offline, but not ones who lost access", () => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + world.roles.set("carol", "build"); + impl.storage.observers.put( + { profileId: "carol", observerId: "obs-c", accountChoices: { [TRACKED]: 20 } }); + impl.storage.observers.put( + { profileId: "erin", observerId: "obs-e", accountChoices: { [TRACKED]: 21 } }); + // Erin is configured but no longer in the sharing graph; her denial must not block anything. + world.verifiers.set("erin", observerVerifier(() => world, "erin", ids => ids)); + + await impl.authorizeTrackedObservation(TRACKED, trackedObservation(["x"]), { from: "user" }); + expect(world.checks).toEqual([{ profileId: "carol", ids: ["x"] }]); + expect(rows(impl, WORKSPACE)).toEqual(["2:x"]); + + // A remembered ID is not a remembered grant: the next read checks it again. + await impl.authorizeTrackedObservation(TRACKED, trackedObservation(["x"]), { from: "user" }); + expect(world.checks).toHaveLength(2); + })); + + it("refuse a read denied only by an observer past the first access-check call", () => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + let calls: string[][] = []; + fakeFacets(impl, { + verify: (observers, ids) => { + calls.push(observers.map(observer => observer.id)); + return verifyEachObserver(observers, ids); + }, + }); + let profiles = Array.from({ length: 11 }, (_, i) => `p${String(i).padStart(2, "0")}`); + for (let profileId of profiles) { + world.roles.set(profileId, "build"); + impl.storage.observers.put( + { profileId, observerId: `obs-${profileId}`, accountChoices: { [TRACKED]: 20 } }); + } + world.verifiers.set("p10", observerVerifier(() => world, "p10", ids => ids)); + let before = impl.storage.nextActionId.get(); + + await expect(impl.authorizeTrackedObservation(TRACKED, trackedObservation(["x"]), + { from: "user" })).rejects.toThrow(/could not be confirmed/); + + // Each call carries at most ten observers; the denier is alone in the second. + expect(calls.map(call => call.length)).toEqual([10, 1]); + expect(calls[1]).toEqual(["obs-p10"]); + expect(world.checks.map(check => check.profileId).toSorted()).toEqual(profiles); + expect(rows(impl, WORKSPACE)).toEqual([]); + expect(impl.storage.nextActionId.get()).toBe(before); + })); + + it("does not admit a collaborator alongside an unreadable concurrent observation", () => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + fakeFacets(impl); + world.roles.set("carol", "build"); + let sharing = await impl.getSharingManager(); + impl.getSharingManager = async () => { + // Let the read's final audience snapshot race with admission's final history check. + for (let i = 0; i < 3; i++) await Promise.resolve(); + return sharing; + }; + let read: Promise | undefined; + let clientUser = { + ...world.clientUser("carol"), + getVerifier: () => { + read ??= impl.authorizeTrackedObservation(TRACKED, trackedObservation(["x"]), + { from: "user" }).then(() => true, () => false); + return Promise.resolve(observerVerifier(() => world, "carol", ids => ids)); + }, + }; + let admitted = await impl.ensureObserver("carol", clientUser, "build", configureAll) + .then(() => true, () => false); + expect(read).toBeDefined(); + let authorized = await read; + if (admitted) { + expect(authorized).toBe(false); + expect(rows(impl, WORKSPACE)).toEqual([]); + } else { + expect(authorized).toBe(true); + expect(rows(impl, WORKSPACE)).toEqual(["2:x"]); + expect(impl.storage.observers.get("carol")).toBeUndefined(); + } + expect(world.checks).toContainEqual({ profileId: "carol", ids: ["x"] }); + })); + + it.each([true, false])( + "rechecks a replacement account before publishing (replacement readable: %s)", readable => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + world.roles.set("carol", "build"); + impl.storage.observers.put( + { profileId: "carol", observerId: "obs-c", accountChoices: { [TRACKED]: 20 } }); + let release!: () => void; + let reached!: () => void; + let parked = new Promise(resolve => { release = resolve; }); + let checking = new Promise(resolve => { reached = resolve; }); + let accounts: number[] = []; + world.clientUser = (profileId: string) => ({ + getVerifier: async (accountId: number) => { + accounts.push(accountId); + return observerVerifier(() => world, profileId, async ids => { + if (accountId === 20) { + reached(); + await parked; + } + return accountId === 20 || readable ? [] : ids; + }); + }, + }); + let before = impl.storage.nextActionId.get(); + let read = impl.authorizeTrackedObservation(TRACKED, trackedObservation(["x"]), + { from: "user" }); + await checking; + impl.storage.observers.put( + { profileId: "carol", observerId: "obs-c", accountChoices: { [TRACKED]: 21 } }); + release(); + + if (readable) await read; + else await expect(read).rejects.toThrow(/could not be confirmed/); + expect(accounts).toEqual([20, 21]); + expect(rows(impl, WORKSPACE)).toEqual(readable ? ["2:x"] : []); + expect(impl.storage.nextActionId.get()).toBe(before + (readable ? 1 : 0)); + if (!readable) expect(impl.storage.actions.get(before)).toBeUndefined(); + })); +}); + +describe("admission against tracked requirements", () => { + it.each(["build", "use"] as const)( + "admits %s collaborators against workspace and every reachable context", role => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + seedGadget(impl, GADGET, { C: TRACKED }); + seedGadget(impl, GADGET + 1); + addRows(impl, WORKSPACE, TRACKED, ["workspace"]); + addRows(impl, chat(1), TRACKED, ["chat-one"]); + addRows(impl, chat(2), TRACKED, ["chat-two"]); + addRows(impl, gadget(GADGET), TRACKED, ["gadget-one"]); + addRows(impl, gadget(GADGET + 1), TRACKED, ["gadget-two"]); + fakeFacets(impl); + world.roles.set("carol", role); + world.verifiers.set("carol", observerVerifier(() => world, "carol", + ids => ids.filter(id => role !== "build" && id.startsWith("chat-")))); + + await impl.ensureObserver("carol", world.clientUser("carol"), role, configureAll); + + let expected = ["workspace", "gadget-one", "gadget-two"]; + if (role === "build") expected.push("chat-one", "chat-two"); + expect(world.checks.map(check => ({ ...check, ids: check.ids.toSorted() }))).toEqual([ + { profileId: "carol", ids: expected.toSorted() }, + ]); + expect(impl.storage.observers.get("carol").accountChoices).toEqual({ [TRACKED]: 20 }); + })); + + // Admission shares the observation path's answer validation (see refusingAnswers), so only its + // own outcomes are repeated here: a denial, and a verifier lookup that fails. + it.each([ + ["a denial", { + verify: (() => ({ allowedObserverIds: [] })) satisfies VerifyObserverAccess, + lookupFailure: undefined, reason: /can't access some of the data/, + }] as const, + ["a wrong vendor or failed lookup", { + verify: undefined, reason: /Invalid account selection/, + lookupFailure: new Error("Invalid account selection for this service."), + }] as const, + ])("fails closed on %s, with no legacy fallback", (_name, { verify, lookupFailure, reason }) => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + addRows(impl, WORKSPACE, TRACKED, ["a"]); + let adds: number[] = []; + fakeFacets(impl, { adds, verify }); + world.roles.set("carol", "build"); + if (lookupFailure) world.verifiers.set("carol", lookupFailure); + impl.storage.observers.put( + { profileId: "carol", observerId: "obs-c", accountChoices: { [TRACKED]: 20 } }); + + let admission = impl.ensureObserver("carol", world.clientUser("carol"), "build"); + await expect(admission).rejects.toThrow(/could not confirm/); + await expect(admission).rejects.toThrow(reason); + expect(adds).toEqual([]); // never handed to the facet's addObserver + })); + + it("re-verifies when a requirement lands while the account prompt is open", () => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + addRows(impl, WORKSPACE, TRACKED, ["a"]); + fakeFacets(impl); + world.roles.set("carol", "build"); + + let release!: () => void; + let prompted!: () => void; + let promptOpen = new Promise(resolve => { prompted = resolve; }); + let parked = new Promise(resolve => { release = resolve; }); + let configureCb = { + configure: async (needs: { gatekeeperId: number }[]) => { + prompted(); + await parked; + return configureAll.configure(needs); + }, + }; + let admission = impl.ensureObserver("carol", world.clientUser("carol"), "build", configureCb); + await promptOpen; + + // The owner reads more while Carol chooses an account; she is not an observer yet, so the + // read is not checked against her -- her admission must pick it up instead. + await impl.authorizeTrackedObservation(TRACKED, trackedObservation(["b"]), { from: "user" }); + expect(world.checks).toEqual([]); + release(); + await admission; + + expect(world.checks.at(-1)).toEqual({ profileId: "carol", ids: ["a", "b"] }); + expect(impl.storage.observers.get("carol").accountChoices).toEqual({ [TRACKED]: 20 }); + })); + + it("blocks an in-flight observation that a newly admitted collaborator cannot access", () => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + world.roles.set("carol", "build"); + world.roles.set("dave", "build"); + impl.storage.observers.put( + { profileId: "carol", observerId: "obs-c", accountChoices: { [TRACKED]: 20 } }); + let release!: () => void; + let parked = new Promise(resolve => { release = resolve; }); + let reached!: () => void; + let checking = new Promise(resolve => { reached = resolve; }); + world.verifiers.set("carol", observerVerifier(() => world, "carol", async () => { + reached(); + await parked; + return []; + })); + world.verifiers.set("dave", observerVerifier(() => world, "dave", ids => ids)); + let before = impl.storage.nextActionId.get(); + + let read = impl.authorizeTrackedObservation(TRACKED, trackedObservation(["x"]), + { from: "user" }); + await checking; + // Dave's admission publishes while Carol's check is in flight. + impl.storage.observers.put( + { profileId: "dave", observerId: "obs-d", accountChoices: { [TRACKED]: 30 } }); + release(); + await expect(read).rejects.toThrow(/could not be confirmed/); + expect(rows(impl, WORKSPACE)).toEqual([]); + expect(impl.storage.nextActionId.get()).toBe(before); + expect(impl.storage.actions.get(before)).toBeUndefined(); + })); + + it("asks use collaborators for no account for a connection nothing binds or hooks", () => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + addRows(impl, chat(1), TRACKED, ["chat-only"]); + fakeFacets(impl); + world.roles.set("carol", "use"); + world.roles.set("dave", "build"); + let asked: number[][] = []; + // A direct-DO double supplying only the RPC method admission calls. + let configure = { + configure: async (needs: { gatekeeperId: number }[]) => { + asked.push(needs.map(need => need.gatekeeperId)); + return needs.map(need => ({ gatekeeperId: need.gatekeeperId, accountId: need.gatekeeperId * 10 })); + }, + } as unknown as NonNullable[3]>; + + await impl.ensureObserver("carol", world.clientUser("carol"), "use", configure); + expect(asked).toEqual([]); + expect(world.checks).toEqual([]); + + await impl.ensureObserver("dave", world.clientUser("dave"), "build", configure); + expect(asked).toEqual([[TRACKED]]); + expect(world.checks).toEqual([{ profileId: "dave", ids: ["chat-only"] }]); + })); + + it("ignores reads outside a use collaborator's scope that land while she is verified", () => + withWorld(async world => { + let { impl } = world; + let unbound = TRACKED + 1; + seedTracked(impl); + seedTracked(impl, unbound); + seedGadget(impl, GADGET, { C: TRACKED }); + addRows(impl, WORKSPACE, TRACKED, ["bound"]); + fakeFacets(impl); + world.roles.set("carol", "use"); + let reads = 0; + world.verifiers.set("carol", observerVerifier(() => world, "carol", async () => { + // The owner keeps reading through a connection nothing binds while Carol is checked. + await impl.authorizeTrackedObservation(unbound, + trackedObservation([`unbound-${++reads}`]), { from: "user" }); + return []; + })); + + await impl.ensureObserver("carol", world.clientUser("carol"), "use", configureAll); + + expect(world.checks).toEqual([{ profileId: "carol", ids: ["bound"] }]); + expect(impl.storage.observers.get("carol").accountChoices).toEqual({ [TRACKED]: 20 }); + })); + + it("verifies existing reads through a connection that enters a use scope mid-verification", () => + withWorld(async world => { + let { impl } = world; + let later = TRACKED + 1; + seedTracked(impl); + seedTracked(impl, later); + seedGadget(impl, GADGET, { C: TRACKED }); + addRows(impl, WORKSPACE, TRACKED, ["bound"]); + addRows(impl, WORKSPACE, later, ["secret"]); + fakeFacets(impl); + world.roles.set("carol", "use"); + world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => { + // Binding records no action, so only the scope's connections show that it changed. + seedGadget(impl, GADGET, { C: TRACKED, D: later }); + return ids.filter(id => id === "secret"); + })); + + await expect(impl.ensureObserver("carol", world.clientUser("carol"), "use", configureAll)) + .rejects.toThrow(/could not confirm/); + expect(world.checks).toContainEqual({ profileId: "carol", ids: ["secret"] }); + expect(impl.storage.observers.get("carol")).toBeUndefined(); + })); +}); + +describe("historical import", () => { + it.each(["allowed", "denied", "unavailable"] as const)( + "rechecks an upgrade after account selection (%s)", outcome => withWorld(async world => { + let { impl } = world; + seedLegacy(impl); + let advertise = new Set(); + let adds: number[] = []; + let exports = 0; + fakeFacets(impl, { + advertise, adds, + exports: new Map([[LEGACY, async () => { exports++; return ["restricted/history"]; }]]), + }); + let getFacet = impl.getGatekeeperFacet; + impl.getGatekeeperFacet = async (id: number) => { + let facet = await getFacet(id); + return { + ...facet, + describe: async () => { + if (advertise.has(id) && outcome === "unavailable") throw new Error("Discovery unavailable"); + return facet.describe(); + }, + }; + }; + world.verifiers.set("carol", observerVerifier(() => world, "carol", + ids => outcome === "allowed" ? [] : ids)); + let release!: () => void; + let reached!: () => void; + let parked = new Promise(resolve => { release = resolve; }); + let prompted = new Promise(resolve => { reached = resolve; }); + let configureCb = { + configure: async (needs: { gatekeeperId: number }[]) => { + reached(); + await parked; + return configureAll.configure(needs); + }, + }; + let admission = impl.ensureObserver("carol", world.clientUser("carol"), "build", configureCb) + .then(() => ({ allowed: true, error: "" }), + (error: unknown) => ({ allowed: false, error: String(error) })); + await prompted; + advertise.add(LEGACY); + release(); + + let result = await admission; + expect(result.allowed).toBe(outcome === "allowed"); + expect(adds).toEqual([]); + if (outcome === "allowed") { + expect(impl.storage.observers.get("carol").accountChoices).toEqual({ [LEGACY]: 10 }); + } else { + expect(result.error).toMatch(/could not confirm/); + expect(impl.storage.observers.get("carol")).toBeUndefined(); + } + if (outcome === "unavailable") { + expect(exports).toBe(0); + expect(rows(impl, WORKSPACE)).toEqual([]); + } else { + expect(exports).toBe(1); + expect(rows(impl, WORKSPACE)).toEqual(["1:restricted/history"]); + expect(world.checks).toContainEqual({ profileId: "carol", ids: ["restricted/history"] }); + } + })); + + it("rediscovers a legacy connection's upgrade before admitting a collaborator", () => + withWorld(async world => { + let { impl } = world; + seedLegacy(impl); + let advertise = new Set(); + let adds: number[] = []; + fakeFacets(impl, { + advertise, adds, + exports: new Map([[LEGACY, async () => ["restricted/history"]]]), + }); + await impl.observerLedger.initialize(LEGACY); + advertise.add(LEGACY); + world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => ids)); + + await expect(impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll)) + .rejects.toThrow(/could not confirm/); + expect(impl.storage.observers.get("carol")).toBeUndefined(); + expect(adds).toEqual([]); + expect(rows(impl, WORKSPACE)).toEqual(["1:restricted/history"]); + + world.verifiers.delete("carol"); + await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); + expect(impl.storage.observers.get("carol").accountChoices).toEqual({ [LEGACY]: 10 }); + })); + + it("refuses admission without registering when discovery fails, and imports once it succeeds", + () => withWorld(async world => { + let { impl } = world; + seedLegacy(impl); + let adds: number[] = []; + fakeFacets(impl, { adds }); + let fail = true; + let getFacet = impl.getGatekeeperFacet; + // A gatekeeper that can't always be described: admission must not fall back to registering + // the collaborator through its addObserver(). + impl.getGatekeeperFacet = async (id: number) => ({ + ...await getFacet(id), + describe: async () => { + if (fail) throw new Error("Discovery unavailable"); + return { supportsObserverTracking: true }; + }, + getHistoricalSubresources: async () => historyCursor(["restricted/history"]), + }); + + let prompts: (string | undefined)[][] = []; + let configure = { + configure: async (needs: { gatekeeperId: number, failure?: { reason: string } }[]) => { + prompts.push(needs.map(need => need.failure?.reason)); + return configureAll.configure(needs); + }, + }; + await expect(impl.ensureObserver("carol", world.clientUser("carol"), "build", configure)) + .rejects.toThrow(/Legacy 1 .*observer tracking protocol/); + // The failure is that binding's own, offered for repair before admission is denied. + expect(prompts).toEqual([[undefined], [expect.stringMatching(/observer tracking protocol/)]]); + expect(adds).toEqual([]); + expect(impl.storage.observers.get("carol")).toBeUndefined(); + expect(impl.observerLedger.isTracked(LEGACY)).toBe(false); + + fail = false; + world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => ids)); + await expect(impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll)) + .rejects.toThrow(/could not confirm/); + expect(adds).toEqual([]); + expect(impl.storage.observers.get("carol")).toBeUndefined(); + expect(rows(impl, WORKSPACE)).toEqual(["1:restricted/history"]); + })); + + it("describes a legacy connection once per admission, and not to open its sessions", () => + withWorld(async world => { + let { impl } = world; + seedLegacy(impl); + let adds: number[] = []; + fakeFacets(impl, { adds }); + let describes = 0; + let getFacet = impl.getGatekeeperFacet; + impl.getGatekeeperFacet = async (id: number) => ({ + ...await getFacet(id), + describe: async () => { describes++; return {}; }, + startSession: async () => ({}), + }); + + await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); + expect(adds).toEqual([LEGACY]); + expect(describes).toBe(1); + await impl.openGatekeeperSession( + LEGACY, await impl.getGatekeeperFacet(LEGACY), { from: "user" }); + expect(describes).toBe(1); + })); + + it("imports an existing legacy connection that starts advertising, even with no prior reads", + async () => { + let name = `observer-ledger-import-${++doCounter}`; + let exports = 0; + let facets = (impl: ObserverImpl) => fakeFacets(impl, { + advertise: new Set([LEGACY]), + exports: new Map([[LEGACY, async () => { exports++; return []; }]]), + }); + await inOverseer(name, async ({ impl, restarts }) => { + seedLegacy(impl); + facets(impl); + let before = impl.storage.nextActionId.get(); + + await impl.observerLedger.initialize(LEGACY); + expect(impl.observerLedger.isTracked(LEGACY)).toBe(true); + expect(exports).toBe(1); + // An empty export writes no audit record. + expect(impl.storage.nextActionId.get()).toBe(before); + expect(rows(impl, WORKSPACE)).toEqual([]); + // The import aborted the facet that clients hold stubs to, so the workspace restarts even + // with only the owner connected, and the connection is held back until it does. + expect(restarts).toHaveLength(1); + expect(impl.gatekeeperUsable(LEGACY)).toBe(false); + }); + + await abortAllDurableObjects(); + + await inOverseer(name, async world => { + let { impl } = world; + facets(impl); + world.roles.set("carol", "build"); + await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); + await impl.authorizeTrackedObservation(LEGACY, trackedObservation(["first-real-read"]), + { from: "user" }); + expect(world.checks).toEqual([ + { profileId: "carol", ids: ["first-real-read"] }, + ]); + expect(rows(impl, WORKSPACE)).toEqual(["1:first-real-read"]); + expect(exports).toBe(1); + }); + }); + + it("exports once for concurrent first callers and merges into the workspace context", () => + withWorld(async world => { + let { impl } = world; + seedLegacy(impl, 3); + let exports = 0; + fakeFacets(impl, { + advertise: new Set([3]), + exports: new Map([[3, async () => { exports++; return ["p1", "p1", "p2"]; }]]), + }); + + let before = impl.storage.nextActionId.get(); + await Promise.all([impl.observerLedger.initialize(3), impl.observerLedger.initialize(3)]); + expect(exports).toBe(1); + expect(impl.observerLedger.isTracked(3)).toBe(true); + // One workspace audit record carries the whole (deduplicated) export. + expect(impl.storage.nextActionId.get()).toBe(before + 1); + expect(impl.storage.actions.get(before)).toMatchObject({ + gatekeeperId: 3, type: "observation", caller: { from: "user" }, + description: { subresources: ["p1", "p2"] }, + }); + // Every context, existing or future, includes workspace requirements. + expect(requirements(impl, "use").get(3)).toEqual(["p1", "p2"]); + })); + + it("imports a paged history larger than one observation and checks every verification batch", + async () => { + let name = `observer-ledger-import-${++doCounter}`; + let ids = Array.from({ length: 10_001 }, (_, i) => String(i)); + let facets = (impl: ObserverImpl) => fakeFacets(impl, { + advertise: new Set([LEGACY]), + exports: new Map([[LEGACY, async () => ids]]), + }); + let required: string[] = []; + await inOverseer(name, async world => { + let { impl } = world; + seedLegacy(impl); + facets(impl); + await impl.observerLedger.initialize(LEGACY); + required = requirements(impl, "build").get(LEGACY)!; + expect(required.toSorted()).toEqual(ids.toSorted()); + world.roles.set("carol", "build"); + world.verifiers.set("carol", observerVerifier(() => world, "carol", + batch => batch.filter(id => id === required.at(-1)))); + await expect(impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll)) + .rejects.toThrow(/could not confirm/); + expect(world.checks).toContainEqual({ profileId: "carol", ids: required.slice(10_000) }); + expect(impl.storage.observers.get("carol")).toBeUndefined(); + }); + + // Reads wait for the restart the import scheduled. + await abortAllDurableObjects(); + + await inOverseer(name, async world => { + let { impl } = world; + facets(impl); + await expect(impl.authorizeTrackedObservation(LEGACY, trackedObservation(ids), + { from: "user" })).rejects.toThrow(/at most 10000/); + expect(requirements(impl, "build").get(LEGACY)).toEqual(required); + world.roles.set("carol", "build"); + await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); + expect(impl.storage.observers.get("carol").accountChoices).toEqual({ [LEGACY]: 10 }); + }); + }); + + it("refuses an export page of more than 10,000 IDs and stays legacy", () => + withWorld(async world => { + let { impl } = world; + seedLegacy(impl); + let ids = Array.from({ length: 10_001 }, (_, i) => String(i)); + fakeFacets(impl, { + advertise: new Set([LEGACY]), + exports: new Map([[LEGACY, async () => historyCursor(ids, ids.length)]]), + }); + + await expect(impl.observerLedger.initialize(LEGACY)).rejects.toThrow(/at most 10000/); + expect(impl.observerLedger.isTracked(LEGACY)).toBe(false); + expect(rows(impl, WORKSPACE)).toEqual([]); + })); + + it.each(["the export call", "a later page"])( + "retries after a failure before the commit, and never after it (failing at %s)", + async failure => { + let name = `observer-ledger-import-${++doCounter}`; + let exports = 0; + let fail = true; + let facets = (impl: any) => fakeFacets(impl, { + advertise: new Set([3]), + exports: new Map([[3, async () => { + exports++; + if (!fail) return ["p1"]; + if (failure === "the export call") throw new Error("export interrupted"); + // The first page arrives; the walk fails before the export ends. + let pages = [["p0"]]; + return { + next: async () => pages.shift() ?? Promise.reject(new Error("export interrupted")), + [Symbol.dispose]() {}, + }; + }]]), + }); + await inOverseer(name, async ({ impl, restarts }) => { + seedLegacy(impl, 3); + facets(impl); + await expect(impl.observerLedger.initialize(3)).rejects.toThrow(/export interrupted/); + // A failed export never clears workspace requirements, and leaves the connection legacy. + expect(impl.observerLedger.isTracked(3)).toBe(false); + expect(rows(impl, WORKSPACE)).toEqual([]); + // The failed import had already aborted the facet clients hold, so the workspace restarts. + expect(restarts).toHaveLength(1); + + fail = false; + await impl.observerLedger.initialize(3); + expect(exports).toBe(2); + }); + + await abortAllDurableObjects(); + + await inOverseer(name, async ({ impl }) => { + facets(impl); + await impl.observerLedger.initialize(3); + expect(exports).toBe(2); // completion is durable: the stored history is authoritative + expect(rows(impl, WORKSPACE)).toEqual(["3:p1"]); + }); + }); + + it("refuses legacy hook queues while their historical import is parked", () => + withWorld(async world => { + let { impl } = world; + seedLegacy(impl); + let release!: () => void; + let reached!: () => void; + let parked = new Promise(resolve => { release = resolve; }); + let exporting = new Promise(resolve => { reached = resolve; }); + fakeFacets(impl, { + advertise: new Set([LEGACY]), + exports: new Map([[LEGACY, async () => { + reached(); + await parked; + return ["history"]; + }]]), + }); + let before = impl.storage.nextActionId.get(); + let importing = impl.observerLedger.initialize(LEGACY); + await exporting; + + await expect(impl.authorizeObservation(LEGACY, observation(), { from: "hook" })) + .rejects.toThrow(/being imported/); + expect(rows(impl, WORKSPACE)).toEqual([]); + expect(impl.storage.nextActionId.get()).toBe(before); + expect(impl.storage.actions.get(before)).toBeUndefined(); + expect(impl.observerLedger.isTracked(LEGACY)).toBe(false); + + release(); + await importing; + expect(rows(impl, WORKSPACE)).toEqual(["1:history"]); + })); + + it("refuses a legacy observation imported while its policy lookup was in flight", () => + withWorld(async world => { + let { impl } = world; + seedLegacy(impl); + fakeFacets(impl, { advertise: new Set([LEGACY]) }); + let release!: () => void; + let reached!: () => void; + let parked = new Promise(resolve => { release = resolve; }); + let lookingUp = new Promise(resolve => { reached = resolve; }); + let sharing = await impl.getSharingManager(); + impl.getSharingManager = async () => { + reached(); + await parked; + return sharing; + }; + let before = impl.storage.nextActionId.get(); + let read = impl.authorizeObservation(LEGACY, + observation({ ownerInvitesOnly: true, containsRestrictedData: true }), { from: "hook" }); + await lookingUp; + await impl.observerLedger.initialize(LEGACY); + release(); + + // The import's restart holds the connection back, so the read commits under neither protocol. + await expect(read).rejects.toThrow(/restarting/); + expect(impl.storage.nextActionId.get()).toBe(before); + expect(impl.storage.actions.get(before)).toBeUndefined(); + expect(impl.storage.ownerInvitesOnly.get()).toBe(false); + expect(impl.storage.containsRestrictedData.get()).toBe(false); + })); + + it("refuses a legacy exclusion check that the import overtakes", () => withWorld(async world => { + let { impl } = world; + seedLegacy(impl); + fakeFacets(impl, { advertise: new Set([LEGACY]) }); + let release!: () => void; + let reached!: () => void; + let parked = new Promise(resolve => { release = resolve; }); + let excluding = new Promise(resolve => { reached = resolve; }); + let sharing = await impl.getSharingManager(); + impl.getSharingManager = async () => { + reached(); + await parked; + return sharing; + }; + let before = impl.storage.nextActionId.get(); + let read = impl.authorizeObservation(LEGACY, + observation({ excludeObservers: ["obs-unknown"] }), { from: "user" }); + await excluding; + await impl.observerLedger.initialize(LEGACY); + release(); + + await expect(read).rejects.toThrow(/restarting/); + expect(impl.storage.nextActionId.get()).toBe(before); + })); + + it("is skipped for a connection created already advertising support", () => + withWorld(async world => { + let { impl } = world; + let exports = 0; + let describe = async () => ({ + url: "https://example.com/new", title: "New", snippet: "", suggestedBindingName: "N", + tsType: "N", supportsObserverTracking: true, + }); + impl.getGatekeeperFacet = async () => ({ + describe, getHistoricalSubresources: async () => { exports++; return historyCursor(["old"]); }, + verifyObserverAccess: verifyEachObserver, + }); + + await impl.addGatekeeper({} as any, spec("trackedvendor", 9), "owner"); + let id = [...impl.storage.gatekeepers.list()][0].id; + await impl.observerLedger.initialize(id); + expect(impl.observerLedger.isTracked(id)).toBe(true); + expect(impl.storage.gatekeepers.get(id).observerTracking).toBe(true); + expect(exports).toBe(0); + expect(rows(impl, WORKSPACE)).toEqual([]); + })); + + it("audits simulated reads without external permissions, then checks the approved real resource", + async () => { + let name = `observer-ledger-creation-${++doCounter}`; + let real = false; + let exports = 0; + let gatekeeperId!: number; + let facet = () => ({ + describe: async () => ({ + url: `https://example.com/${real ? "real" : "simulated"}`, + title: real ? "Real" : "Simulated", snippet: "", suggestedBindingName: "N", tsType: "N", + supportsObserverTracking: true, + }), + getHistoricalSubresources: async () => { exports++; return historyCursor([]); }, + verifyObserverAccess: verifyEachObserver, + applyCreation: async () => { + real = true; + return { class: {}, resourceUrl: "https://example.com/real" }; + }, + startSession: async (queue: ObservationAuthorizer) => ({ + read: (description: TrackedObservationDescription) => + queue.authorizeTrackedObservation(description), + }), + }); + + await inOverseer(name, async world => { + let { impl } = world; + impl.ownerId = "owner"; + impl.traceAgentActionApproval = () => {}; + impl.wrapUserDo = (user: unknown) => user; + let sharing = await impl.getSharingManager(); + impl.getSharingManager = async () => ({ + ...sharing, computeAffectedByOwnerInvitesOnly: () => [], + }); + impl.users = { + idFromName: (id: string) => id, + idFromString: (id: string) => id, + get: (id: string) => ({ + ...world.clientUser(id), + createResourceGatekeeper: async () => ({ + class: {}, typeUrlPattern: "https://example.com/*", + action: { title: "Create", description: "Create", implementsRevert: false }, + }), + }), + }; + impl.getGatekeeperFacet = async () => facet(); + + let leave = impl.joinSession("build"); + try { + ({ gatekeeperId } = await impl.createExternalResource(1, { + vendorId: "trackedvendor", resourceUrlPattern: "https://example.com/*", title: "New", + bindingName: "N", + })); + await impl.observerLedger.initialize(gatekeeperId); + let session = await impl.openGatekeeperSession( + gatekeeperId, await impl.getGatekeeperFacet(gatekeeperId), { from: "user" }); + let before = impl.storage.nextActionId.get(); + await session.read(trackedObservation(["simulated-id"], { + containsRestrictedData: true, ownerInvitesOnly: true, + })); + expect(impl.storage.actions.get(before)).toMatchObject({ + gatekeeperId, type: "observation", state: "approved", + }); + expect(impl.storage.actions.get(before).description.subresources).toBeUndefined(); + expect(impl.storage.nextActionId.get()).toBe(before + 1); + expect(impl.storage.ownerInvitesOnly.get()).toBe(true); + expect(() => impl.getWebFetchEnv()).toThrow(/observed sensitive data/); + expect(impl.storage.gatekeepers.get(gatekeeperId).observerTracking).toBeUndefined(); + expect([...impl.storage.actions.byObserverRequirement.listKeys()]).toEqual([]); + for (let role of ["build", "use"] as const) { + expect(impl.listObserverRequirements(role)).toEqual([]); + await impl.ensureObserver("carol", world.clientUser("carol"), role); + } + expect(world.checks).toEqual([]); + expect(exports).toBe(0); + expect(world.restarts).toEqual([]); + expect(impl.gatekeeperUsable(gatekeeperId)).toBe(true); + + // Let the simulated collaborator session close so approval can be exercised in this DO + // without its normal access restart quarantining the real connection until a reset. + leave(); + let [creation] = impl.storage.actions.pendingByGatekeeper.get(gatekeeperId); + await impl.applyPendingAction(creation, + { type: "user", id: "owner", name: "Owner" }, false, + { creator: {}, configuration: null }); + expect(impl.storage.actions.get(creation.id).state).toBe("approved"); + expect([...impl.storage.gatekeepers.list()].map((record: GatekeeperRecord) => record.id)) + .toEqual([gatekeeperId]); + expect(impl.storage.gatekeepers.get(gatekeeperId)).toMatchObject({ + id: gatekeeperId, creationSpec: { resourceUrl: "https://example.com/real" }, + }); + // Approval makes it an existing connection advertising support. Its first tracked + // observation imports it, exactly once, and is refused: its session predates the import. + let staleSession = await impl.openGatekeeperSession( + gatekeeperId, await impl.getGatekeeperFacet(gatekeeperId), { from: "user" }); + await expect(staleSession.read(trackedObservation(["real-id"]))) + .rejects.toThrow(/now supports/); + expect(impl.storage.gatekeepers.get(gatekeeperId).observerTracking).toBe(true); + expect(exports).toBe(1); + expect(rows(impl, WORKSPACE)).toEqual([]); + // The import restarts the workspace, holding the connection back until it lands. + expect(world.restarts).toHaveLength(1); + expect(impl.gatekeeperUsable(gatekeeperId)).toBe(false); + } finally { + leave(); + } + }); + + await abortAllDurableObjects(); + + await inOverseer(name, async world => { + let { impl } = world; + impl.getGatekeeperFacet = async () => facet(); + let realSession = await impl.openGatekeeperSession( + gatekeeperId, await impl.getGatekeeperFacet(gatekeeperId), { from: "user" }); + + world.roles.set("carol", "build"); + await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); + world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => ids)); + let beforeDenied = impl.storage.nextActionId.get(); + await expect(realSession.read(trackedObservation(["real-id"]))) + .rejects.toThrow(/could not be confirmed/); + expect(impl.storage.nextActionId.get()).toBe(beforeDenied); + expect(rows(impl, WORKSPACE)).toEqual([]); + world.verifiers.delete("carol"); + await realSession.read(trackedObservation(["real-id"])); + expect(world.checks).toEqual([ + { profileId: "carol", ids: ["real-id"] }, + { profileId: "carol", ids: ["real-id"] }, + ]); + expect(rows(impl, WORKSPACE)).toEqual([`${gatekeeperId}:real-id`]); + expect(exports).toBe(1); + }); + }); + + it("verifies fully qualified historical and new IDs without conflating same-named resources", () => + withWorld(async world => { + let { impl } = world; + let first = "workspace-a/documents/shared"; + let second = "workspace-b/documents/shared"; + seedTracked(impl); + seedLegacy(impl, 3, { creationSpec: spec("trackedvendor", 3) }); + addRows(impl, WORKSPACE, TRACKED, [first]); + fakeFacets(impl, { + advertise: new Set([3]), + exports: new Map([[3, async () => [second]]]), + }); + await impl.observerLedger.initialize(3); + world.roles.set("carol", "build"); + world.verifiers.set("carol", observerVerifier(() => world, "carol", + ids => ids.filter(id => id !== first))); + + await expect(impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll)) + .rejects.toThrow(/could not confirm/); + expect(world.checks.some(check => check.ids.includes(second))).toBe(true); + expect(impl.storage.observers.get("carol")).toBeUndefined(); + expect(rows(impl, WORKSPACE)).toEqual([`2:${first}`, `3:${second}`]); + + world.verifiers.delete("carol"); + await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); + expect(impl.storage.observers.get("carol").accountChoices).toEqual({ 2: 20, 3: 30 }); + + // A grant for this leaf name in workspace A cannot authorize a read in workspace B. + world.verifiers.set("carol", observerVerifier(() => world, "carol", + ids => ids.filter(id => id !== first))); + let before = impl.storage.nextActionId.get(); + await expect(impl.authorizeTrackedObservation(TRACKED, trackedObservation([second]), + { from: "user" })).rejects.toThrow(/could not be confirmed/); + expect(world.checks.at(-1)).toEqual({ profileId: "carol", ids: [second] }); + expect(rows(impl, WORKSPACE)).toEqual([`2:${first}`, `3:${second}`]); + expect(impl.storage.nextActionId.get()).toBe(before); + })); +}); + +describe("connection removal", () => { + it.each(["build", "use"] as const)( + "ignores a removed connection's retained requirements at %s admission, keeping its audit history across restarts", + async role => { + let name = `observer-ledger-removal-${++doCounter}`; + let auditId = -1; + let liveAuditId = -1; + await inOverseer(name, async ({ impl }) => { + seedTracked(impl); + seedTracked(impl, 3); + seedGadget(impl); + auditId = impl.storage.nextActionId.get(); + await impl.authorizeTrackedObservation(TRACKED, + trackedObservation(["removed/first"], { title: "Retained read" }), { from: "user" }); + liveAuditId = impl.storage.nextActionId.get(); + await impl.authorizeTrackedObservation(3, trackedObservation(["live/workspace"]), + { from: "user" }); + for (let context of [WORKSPACE, chat(1), chat(99), gadget(GADGET), gadget(999)]) { + addRows(impl, context, TRACKED, ["removed/read"]); + addRows(impl, context, 3, ["live/read"]); + } + impl.bindWorkpiece(GADGET, "REMOVED", TRACKED); + impl.bindWorkpiece(GADGET, "LIVE", 3); + + impl.removeGatekeeper(TRACKED); + + expect(impl.storage.gatekeepers.get(TRACKED)).toBeUndefined(); + expect(impl.storage.gatekeepers.get(3).observerTracking).toBe(true); + // The removed connection's records stay in the audit log. + for (let context of [chat(1), chat(99), gadget(GADGET), gadget(999)]) { + expect(rows(impl, context)).toEqual(["2:removed/read", "3:live/read"]); + } + expect(impl.storage.gadgets.get(GADGET).bindings).toEqual({ LIVE: { target: 3 } }); + expect(impl.storage.actions.get(auditId)).toMatchObject({ + gatekeeperId: TRACKED, type: "observation", state: "approved", + description: { title: "Retained read", subresources: ["removed/first"] }, + }); + expect(impl.storage.actions.get(liveAuditId).gatekeeperId).toBe(3); + }); + + await abortAllDurableObjects(); + + await inOverseer(name, async world => { + let { impl } = world; + fakeFacets(impl); + world.roles.set("carol", role); + world.verifiers.set("carol", observerVerifier(() => world, "carol", + ids => ids.filter(id => !id.startsWith("live/")))); + + await impl.ensureObserver("carol", world.clientUser("carol"), role, configureAll); + + expect(world.checks).toEqual([ + { profileId: "carol", ids: expect.arrayContaining(["live/workspace", "live/read"]) }, + ]); + expect(impl.listObserverRequirements(role) + .map((need: { gatekeeperId: number }) => need.gatekeeperId)) + .toEqual([3]); + expect(impl.storage.observers.get("carol").accountChoices).toEqual({ 3: 30 }); + expect(impl.storage.gatekeepers.get(TRACKED)).toBeUndefined(); + expect(impl.storage.actions.get(auditId)).toMatchObject({ + gatekeeperId: TRACKED, description: { subresources: ["removed/first"] }, + }); + expect(impl.storage.actions.get(liveAuditId).gatekeeperId).toBe(3); + }); + }); +}); + +describe("direct context authorization", () => { + it("records chat reads only in that chat and checks build, not use, collaborators", () => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + seedGadget(impl); + addRows(impl, WORKSPACE, TRACKED, ["workspace"]); + addRows(impl, gadget(GADGET), TRACKED, ["gadget"]); + addRows(impl, chat(2), TRACKED, ["other-chat"]); + world.roles.set("bob", "build"); + world.roles.set("carol", "use"); + for (let profileId of ["bob", "carol"]) { + impl.storage.observers.put( + { profileId, observerId: `obs-${profileId}`, accountChoices: { [TRACKED]: 20 } }); + } + world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => ids)); + + impl.isLiveExecution = (chatId: number, id: string) => chatId === 1 && id === "e1"; + await impl.authorizeTrackedObservation(TRACKED, trackedObservation(["read"]), + { from: "agent", chatId: 1 }, "e1"); + + expect(world.checks).toEqual([{ profileId: "bob", ids: ["read"] }]); + expect(rows(impl, chat(1))).toEqual(["2:read"]); + expect(rows(impl, chat(2))).toEqual(["2:other-chat"]); + expect(rows(impl, gadget(GADGET))).toEqual(["2:gadget"]); + expect(rows(impl, WORKSPACE)).toEqual(["2:workspace"]); + + impl.isLiveExecution = () => false; + let before = impl.storage.nextActionId.get(); + await expect(impl.authorizeTrackedObservation(TRACKED, trackedObservation(["late"]), + { from: "agent", chatId: 1 }, "e1")) + .rejects.toThrow(/has ended/); + expect(rows(impl, chat(1))).toEqual(["2:read"]); + expect(impl.storage.nextActionId.get()).toBe(before); + })); + + it("refuses a read whose execution ends while observer verification is in flight", () => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + world.roles.set("bob", "build"); + impl.storage.observers.put( + { profileId: "bob", observerId: "obs-b", accountChoices: { [TRACKED]: 20 } }); + let release!: () => void; + let reached!: () => void; + let parked = new Promise(resolve => { release = resolve; }); + let checking = new Promise(resolve => { reached = resolve; }); + world.verifiers.set("bob", observerVerifier(() => world, "bob", async () => { + reached(); + await parked; + return []; + })); + let live = "e1"; + impl.isLiveExecution = (chatId: number, id: string) => chatId === 1 && id === live; + let before = impl.storage.nextActionId.get(); + let read = impl.authorizeTrackedObservation(TRACKED, trackedObservation(["X"]), + { from: "agent", chatId: 1 }, "e1"); + await checking; + live = "e2"; + release(); + await expect(read).rejects.toThrow(/has ended/); + expect(rows(impl, chat(1))).toEqual([]); + expect(impl.storage.nextActionId.get()).toBe(before); + expect(impl.storage.actions.get(before)).toBeUndefined(); + })); + + it("checks newly reachable collaborators when sharing changes during verification", () => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + world.roles.set("bob", "build"); + world.roles.set("carol", "use"); + for (let profileId of ["bob", "carol"]) { + impl.storage.observers.put( + { profileId, observerId: `obs-${profileId}`, accountChoices: { [TRACKED]: 20 } }); + } + let release!: () => void; + let reached!: () => void; + let parked = new Promise(resolve => { release = resolve; }); + let checking = new Promise(resolve => { reached = resolve; }); + world.verifiers.set("bob", observerVerifier(() => world, "bob", async () => { + reached(); + await parked; + return []; + })); + world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => ids)); + let before = impl.storage.nextActionId.get(); + let read = impl.authorizeTrackedObservation(TRACKED, trackedObservation(["private"]), + { from: "user", chatId: 1 }); + await checking; + world.roles.set("carol", "build"); + release(); + + await expect(read).rejects.toThrow(/could not be confirmed/); + expect(world.checks.some(check => check.profileId === "carol")).toBe(true); + expect(rows(impl, chat(1))).toEqual([]); + expect(impl.storage.nextActionId.get()).toBe(before); + expect(impl.storage.actions.get(before)).toBeUndefined(); + })); + + it("records gadget reads on the originating gadget and checks both collaborator roles", () => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + seedGadget(impl); + seedGadget(impl, GADGET + 1); + addRows(impl, WORKSPACE, TRACKED, ["workspace"]); + addRows(impl, chat(1), TRACKED, ["chat"]); + addRows(impl, gadget(GADGET), TRACKED, ["other-gadget"]); + addRows(impl, gadget(GADGET + 1), TRACKED, ["retained"]); + world.roles.set("bob", "build"); + world.roles.set("carol", "use"); + for (let profileId of ["bob", "carol"]) { + impl.storage.observers.put( + { profileId, observerId: `obs-${profileId}`, accountChoices: { [TRACKED]: 20 } }); + } + + impl.bindWorkpiece(GADGET + 1, "DB", TRACKED); + expect(impl.storage.gadgets.get(GADGET + 1).bindings.DB).toEqual({ target: TRACKED }); + impl.storage.defaultGadgetId.put(GADGET); + let caller = { from: "gadget", chatId: 1, gadgetId: GADGET + 1 }; + await impl.authorizeTrackedObservation(TRACKED, trackedObservation(["first"]), caller); + await impl.authorizeTrackedObservation(TRACKED, trackedObservation(["second"]), caller); + + expect(world.checks).toHaveLength(4); + for (let profileId of ["bob", "carol"]) { + expect(world.checks.filter(check => check.profileId === profileId)).toEqual([ + { profileId, ids: ["first"] }, + { profileId, ids: ["second"] }, + ]); + } + expect(rows(impl, gadget(GADGET + 1))).toEqual(["2:first", "2:retained", "2:second"]); + expect(rows(impl, gadget(GADGET))).toEqual(["2:other-gadget"]); + expect(rows(impl, chat(1))).toEqual(["2:chat"]); + expect(rows(impl, WORKSPACE)).toEqual(["2:workspace"]); + })); + + it.each(["build", "use"] as const)( + "does not commit a gadget read refused by a %s collaborator", role => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + seedGadget(impl); + addRows(impl, gadget(GADGET), TRACKED, ["retained"]); + world.roles.set("carol", role); + impl.storage.observers.put( + { profileId: "carol", observerId: "obs-c", accountChoices: { [TRACKED]: 20 } }); + world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => ids)); + let before = impl.storage.nextActionId.get(); + + await expect(impl.authorizeTrackedObservation(TRACKED, trackedObservation(["denied"]), + { from: "gadget", gadgetId: GADGET })) + .rejects.toThrow(/could not be confirmed/); + + expect(world.checks).toEqual([{ profileId: "carol", ids: ["denied"] }]); + expect(rows(impl, gadget(GADGET))).toEqual(["2:retained"]); + expect(rows(impl, WORKSPACE)).toEqual([]); + expect(impl.storage.nextActionId.get()).toBe(before); + expect(impl.storage.actions.get(before)).toBeUndefined(); + })); + + it("refuses tracked observations with no attributable context", () => + withWorld(async world => { + let { impl } = world; + seedTracked(impl); + fakeFacets(impl); + impl.storage.defaultGadgetId.put(GADGET); + let before = impl.storage.nextActionId.get(); + + // A gadget caller persisted before multi-gadget support has no gadgetId. Even a default + // gadget and chatId do not make that historical caller attributable. + await expect(impl.authorizeTrackedObservation(TRACKED, trackedObservation(["x"]), + { from: "gadget", chatId: 1 })).rejects.toThrow(/no attributable chat or gadget/); + + expect([...impl.storage.actions.byObserverRequirement.listKeys()]).toEqual([]); + expect(impl.storage.nextActionId.get()).toBe(before); + })); +}); diff --git a/packages/workshop-backend/__tests__/observer-scope-restart.test.ts b/packages/workshop-backend/__tests__/observer-scope-restart.test.ts index 641777cb5d..44b365d419 100644 --- a/packages/workshop-backend/__tests__/observer-scope-restart.test.ts +++ b/packages/workshop-backend/__tests__/observer-scope-restart.test.ts @@ -439,6 +439,7 @@ describe("restarting sessions when verification scope widens", () => { impl.storage.observers.put( { profileId: "alice", observerId: "obs-1", accountChoices: { 1: 10 } }); impl.getGatekeeperFacet = () => ({ + describe: async () => ({}), addObserver: async () => { throw new Error("access revoked upstream"); }, removeObserver: async () => {}, }); diff --git a/packages/workshop-backend/__tests__/observer-verification-failure.test.ts b/packages/workshop-backend/__tests__/observer-verification-failure.test.ts index b7982fee15..c84c3a6a8d 100644 --- a/packages/workshop-backend/__tests__/observer-verification-failure.test.ts +++ b/packages/workshop-backend/__tests__/observer-verification-failure.test.ts @@ -8,6 +8,7 @@ import { describe, expect, it } from "vitest"; import { env } from "cloudflare:workers"; import { runInDurableObject } from "cloudflare:test"; import type { OverseerDurableObject } from "../src/overseer.js"; +import type { GatekeeperRecord } from "../src/storage-schema/overseer-storage.js"; declare module "cloudflare:workers" { interface ProvidedEnv { @@ -15,17 +16,24 @@ declare module "cloudflare:workers" { } } +type ObserverImpl = OverseerDurableObject["impl"]; +type ObserverClientUser = Parameters[1]; +type ObserverConfigureCallback = NonNullable[3]>; + +// Tests replace the facet factory, so the stored class is never instantiated. +const fakeGatekeeperClass = {} as unknown as GatekeeperRecord["class"]; + // Seed the owner profile id, so the sharing manager needs no User DO round trip. -function seedOwner(impl: any): void { +function seedOwner(impl: ObserverImpl): void { impl.ownerProfileId = "owner"; } -function seedGatekeepers(impl: any, ids: number[] = [1, 2]): void { +function seedGatekeepers(impl: ObserverImpl, ids: number[] = [1, 2]): void { for (let id of ids) { impl.storage.gatekeepers.put({ id, resourceTitle: `Connection ${id}`, - class: {} as any, + class: fakeGatekeeperClass, creationSpec: { type: "gatekeeper", vendorId: "testvendor", @@ -36,17 +44,65 @@ function seedGatekeepers(impl: any, ids: number[] = [1, 2]): void { } } +function seedTrackedGatekeeper(impl: ObserverImpl, id: number): void { + seedGatekeepers(impl, [id]); + impl.storage.gatekeepers.put({ ...impl.storage.gatekeepers.get(id)!, observerTracking: true }); +} + +// A tracked read of `subresource` by a user outside any chat, which puts it on the workspace +// context. +function seedWorkspaceRequirement(impl: ObserverImpl, gatekeeperId: number, subresource: string): void { + impl.observerLedger.recordObservation({ + gatekeeperId, caller: { from: "user" }, createdAt: new Date(), state: "approved", + type: "observation", + description: { title: "Read", description: "A tracked read.", subresources: [subresource] }, + }); +} + +// A vendor's verifier for one account: answers which of `ids` it cannot read, through a +// vendor-specific method only its gatekeeper calls. +type VendorVerifier = { deniedSubresources(ids: string[]): Promise }; + +// Facets for every seeded connection: a legacy one registers through `hooks`, and a tracked one +// admits each observer whose verifier denies none of the IDs. +function setObserverFacets(impl: ObserverImpl, hooks: { + addObserver?(id: number, observerId: string): Promise; + removeObserver?(id: number): Promise; +} = {}): void { + const getFacet = async (id: number) => ({ + describe: async () => ({}), + addObserver: async (observerId: string) => { await hooks.addObserver?.(id, observerId); }, + removeObserver: async () => { await hooks.removeObserver?.(id); }, + verifyObserverAccess: async (observers: {id: string, verifier: VendorVerifier}[], + ids: string[]) => { + let denied = await Promise.all( + observers.map(observer => observer.verifier.deniedSubresources(ids))); + return { allowedObserverIds: observers.filter((_, i) => denied[i].length === 0) + .map(observer => observer.id) }; + }, + }); + // These direct-DO tests exercise only the observer methods admission calls, not the full RPC + // facet. + impl.getGatekeeperFacet = getFacet as unknown as ObserverImpl["getGatekeeperFacet"]; +} + +// Direct-DO test doubles supply only the RPC methods admission exercises. +const configureAll = { + configure: async (needs: {gatekeeperId: number}[]) => + needs.map(need => ({ gatekeeperId: need.gatekeeperId, accountId: need.gatekeeperId * 10 })), +} as unknown as ObserverConfigureCallback; + // A client User DO that always has the account and always mints a verifier. const fakeClientUser = { getVerifier: async () => ({}), describeConnectedAccount: async () => null, -} as any; +} as unknown as ObserverClientUser; describe("a binding that fails verification", () => { it("reports a verifier rejection as a denial, not as the raw RPC error", async () => { let stub = env.TEST_OVERSEER.getByName("observer-verify-getverifier"); await runInDurableObject(stub, async (instance: OverseerDurableObject) => { - let impl = (instance as unknown as { impl: any }).impl; + let impl = instance["impl"]; seedOwner(impl); seedGatekeepers(impl); // Already-configured coverage for both gatekeepers, as a previous successful open left it. @@ -54,10 +110,7 @@ describe("a binding that fails verification", () => { { profileId: "alice", observerId: "obs-1", accountChoices: { 1: 10, 2: 20 } }); let removed: number[] = []; - impl.getGatekeeperFacet = (id: number) => ({ - addObserver: async () => {}, - removeObserver: async () => { removed.push(id); }, - }); + setObserverFacets(impl, { removeObserver: async id => { removed.push(id); } }); // Gatekeeper 1's verifier never materializes: the client's User DO *rejects* (the // deterministic vendor-mismatch throw, or any cross-worker transport failure) rather than @@ -68,7 +121,7 @@ describe("a binding that fails verification", () => { return {}; }, describeConnectedAccount: async () => null, - } as any; + } as unknown as ObserverClientUser; // The rejection is caught per binding like any other refusal, so with no repair channel the // open is denied with the descriptive message that names what to fix -- rather than the @@ -95,7 +148,7 @@ describe("a binding that fails verification", () => { async () => { let stub = env.TEST_OVERSEER.getByName("observer-verify-keeps-registration"); await runInDurableObject(stub, async (instance: OverseerDurableObject) => { - let impl = (instance as unknown as { impl: any }).impl; + let impl = instance["impl"]; seedOwner(impl); seedGatekeepers(impl); // Alice's previous open covered gatekeeper 1 only; gatekeeper 2 is a binding added since, @@ -104,111 +157,114 @@ describe("a binding that fails verification", () => { { profileId: "alice", observerId: "obs-1", accountChoices: { 1: 10 } }); let removed: number[] = []; - impl.getGatekeeperFacet = (id: number) => ({ - // Gatekeeper 1 has revoked her access upstream: the binding she *was* admitted for is the - // one that now refuses, which is exactly the case that used to drop her registration. - addObserver: async () => { if (id === 1) throw new Error("access revoked upstream"); }, - removeObserver: async () => { removed.push(id); }, + setObserverFacets(impl, { + // The connection Alice was already admitted for has now revoked her access upstream. + addObserver: async id => { if (id === 1) throw new Error("access revoked upstream"); }, + removeObserver: async id => { removed.push(id); }, }); - let configureCb = { configure: async (needs: {gatekeeperId: number}[]) => - needs.map(need => ({ gatekeeperId: need.gatekeeperId, accountId: 20 })) } as any; - - await expect(impl.ensureObserver("alice", fakeClientUser, "build", configureCb)) + await expect(impl.ensureObserver("alice", fakeClientUser, "build", configureAll)) .rejects.toThrow(/could not confirm/); // Gatekeeper 1's registration predates this call and survives, so it keeps naming her in - // `excludeObservers`. Gatekeeper 2's was created by this call -- but her persisted - // observerId is shared with any concurrent open, which may have registered (and persisted) - // the very same gatekeeper, so it is kept too: de-registering is the fail-open direction, - // a spurious registration only blocks fail-closed until a later open re-verifies it. + // `excludeObservers`. Gatekeeper 2's was created by this call, but Alice's persisted + // observerId remains resolvable to exclusion checks too: de-registering is the fail-open + // direction, and a spurious registration only blocks fail-closed until a later open + // re-verifies it. expect(removed).toEqual([]); }); }); - it("a failed open does not roll back registrations a concurrent open persisted", async () => { + it("queues a successful open after a failed one without blocking another profile", async () => { let stub = env.TEST_OVERSEER.getByName("observer-verify-concurrent"); await runInDurableObject(stub, async (instance: OverseerDurableObject) => { - let impl = (instance as unknown as { impl: any }).impl; + let impl = instance["impl"]; seedOwner(impl); seedGatekeepers(impl, [1, 2, 3]); - // Alice's record predates connections 2 and 3, so both opens must register her with both -- - // under the one persisted observerId they share. impl.storage.observers.put( { profileId: "alice", observerId: "obs-1", accountChoices: { 1: 10 } }); let removed: number[] = []; - // Call A parks on connection 3's *verifier* -- deliberately outside the per-(observer, - // gatekeeper) serialization of the addObserver RPCs themselves, which would otherwise queue - // call B's registration behind a parked one. Connection 3 then admits only call B's - // registration (the 1st): call A's attempts (the 2nd, and the 3rd after its repair - // re-prompt) are refused. let releaseVerifier!: () => void; let verifierParked = new Promise(resolve => { releaseVerifier = resolve; }); let reached!: () => void; let verifierReached = new Promise(resolve => { reached = resolve; }); - let gk3Verifiers = 0; + let firstAttempt = true; let parkingClientUser = { getVerifier: async (accountId: number) => { - if (accountId === 30 && ++gk3Verifiers === 1) { reached(); await verifierParked; } + if (accountId === 30) { + if (firstAttempt) { + firstAttempt = false; + reached(); + await verifierParked; + } + throw new Error("no access"); + } return {}; }, describeConnectedAccount: async () => null, - } as any; - let gk3Adds = 0; - impl.getGatekeeperFacet = (id: number) => ({ - addObserver: async () => { - if (id === 3 && ++gk3Adds > 1) throw new Error("no access"); - }, - removeObserver: async () => { removed.push(id); }, - }); - let configureCb = { configure: async (needs: {gatekeeperId: number}[]) => - needs.map(need => ({ gatekeeperId: need.gatekeeperId, accountId: need.gatekeeperId * 10 })) - } as any; + } as unknown as ObserverClientUser; + setObserverFacets(impl, { removeObserver: async id => { removed.push(id); } }); - // Call A registers her with connection 2, then parks before registering with connection 3. - let callA = impl.ensureObserver("alice", parkingClientUser, "build", configureCb); - callA.catch(() => {}); // asserted below; not unhandled in the meantime + let callA = impl.ensureObserver("alice", parkingClientUser, "build", configureAll); + callA.catch(() => {}); // asserted below; not unhandled while its verifier is parked await verifierReached; - // Call B -- the same returning observer opening concurrently -- verifies everything and - // persists the record asserting all three registrations. - await impl.ensureObserver("alice", fakeClientUser, "build", configureCb); - expect(impl.storage.observers.get("alice").accountChoices).toEqual({ 1: 10, 2: 20, 3: 30 }); + let queuedPrompts = 0; + let queuedConfig = { + configure: async (needs: {gatekeeperId: number}[]) => { + queuedPrompts++; + return configureAll.configure(needs); + }, + } as unknown as ObserverConfigureCallback; + let callB = impl.ensureObserver("alice", fakeClientUser, "build", queuedConfig); + + // Bob can finish while Alice's first open is parked, but her second open cannot prompt yet. + await impl.ensureObserver("bob", fakeClientUser, "build", configureAll); + let queuedPromptsBeforeRelease = queuedPrompts; + let choicesBeforeRelease = impl.storage.observers.get("alice")?.accountChoices; - // Now connection 3 refuses call A terminally. releaseVerifier(); await expect(callA).rejects.toThrow(/could not confirm/); + await callB; - // Call A also registered her with connection 2 before failing on 3, but must not roll that - // back: the registration is keyed by the shared observerId, so removing it would delete the - // one call B just made -- while B's persisted record asserts it exists, breaking every - // future exclusion that names her. + expect(impl.storage.observers.get("bob")).toBeDefined(); + expect(queuedPromptsBeforeRelease).toBe(0); + expect(choicesBeforeRelease).toEqual({ 1: 10 }); + expect(queuedPrompts).toBe(1); expect(removed).toEqual([]); - expect(impl.storage.observers.get("alice").accountChoices).toEqual({ 1: 10, 2: 20, 3: 30 }); + expect(impl.storage.observers.get("alice")).toEqual({ + profileId: "alice", observerId: "obs-1", accountChoices: { 1: 10, 2: 20, 3: 30 }, + }); }); }); - it("a first-ever verification failure still rolls its registrations back", async () => { - let stub = env.TEST_OVERSEER.getByName("observer-verify-first-ever"); + it.each(["verifier rejection", "disconnected account", "registration refusal"])( + "rolls back a first-ever observer after %s", async failure => { + let stub = env.TEST_OVERSEER.getByName(`observer-verify-first-ever-${failure}`); await runInDurableObject(stub, async (instance: OverseerDurableObject) => { - let impl = (instance as unknown as { impl: any }).impl; + let impl = instance["impl"]; seedOwner(impl); seedGatekeepers(impl); - // No observer record: Alice has never been admitted, so the observerId minted for this call - // is discarded with the unpersisted record and anything registered under it would linger - // unresolvable. let removed: number[] = []; - impl.getGatekeeperFacet = (id: number) => ({ - addObserver: async () => { if (id === 2) throw new Error("no access"); }, - removeObserver: async () => { removed.push(id); }, + setObserverFacets(impl, { + addObserver: async id => { + if (id === 2 && failure === "registration refusal") throw new Error("no access"); + }, + removeObserver: async id => { removed.push(id); }, }); + let clientUser = { + getVerifier: async (accountId: number) => { + if (accountId === 20) { + if (failure === "verifier rejection") throw new Error("verifier unavailable"); + if (failure === "disconnected account") return null; + } + return {}; + }, + describeConnectedAccount: async () => null, + } as unknown as ObserverClientUser; - let configureCb = { configure: async (needs: {gatekeeperId: number}[]) => - needs.map(need => ({ gatekeeperId: need.gatekeeperId, accountId: need.gatekeeperId * 10 })) - } as any; - - await expect(impl.ensureObserver("alice", fakeClientUser, "build", configureCb)) + await expect(impl.ensureObserver("alice", clientUser, "build", configureAll)) .rejects.toThrow(/could not confirm/); // Both the one that verified and the one that refused are rolled back, and no record is @@ -217,4 +273,199 @@ describe("a binding that fails verification", () => { expect(impl.storage.observers.get("alice")).toBeUndefined(); }); }); + + it("rolls back registrations from earlier repaired attempts after a later failure", async () => { + let stub = env.TEST_OVERSEER.getByName("observer-verify-repaired-rollback"); + await runInDurableObject(stub, async (instance: OverseerDurableObject) => { + let impl = instance["impl"]; + seedOwner(impl); + seedGatekeepers(impl); + let attempts = new Map(); + let removed: number[] = []; + setObserverFacets(impl, { + addObserver: async id => { + let attempt = (attempts.get(id) ?? 0) + 1; + attempts.set(id, attempt); + if ((id === 1 && attempt === 1) || (id === 2 && attempt === 2)) { + throw new Error("no access"); + } + }, + removeObserver: async id => { removed.push(id); }, + }); + + await expect(impl.ensureObserver("alice", fakeClientUser, "build", configureAll)) + .rejects.toThrow(/could not confirm/); + + expect(attempts.get(1)).toBe(2); + expect(attempts.get(2)).toBe(2); + expect(removed.toSorted()).toEqual([1, 2]); + expect(impl.storage.observers.get("alice")).toBeUndefined(); + }); + }); + + it.each(["legacy-only", "mixed"])( + "reuses one observer identity for first-ever concurrent successful opens in %s workspaces", + async workspace => { + let stub = env.TEST_OVERSEER.getByName(`observer-verify-first-concurrent-${workspace}`); + await runInDurableObject(stub, async (instance: OverseerDurableObject) => { + let impl = instance["impl"]; + seedOwner(impl); + seedGatekeepers(impl); + if (workspace === "mixed") { + seedTrackedGatekeeper(impl, 2); + seedWorkspaceRequirement(impl, 2, "read"); + } + let registrations: {id: number, observerId: string}[] = []; + setObserverFacets(impl, { + addObserver: async (id, observerId) => { registrations.push({id, observerId}); }, + }); + let checks: {accountId: number, ids: string[]}[] = []; + // The same direct-DO user double supports both legacy and tracked verification. + const clientUser = { + getVerifier: async (accountId: number): Promise => ({ + deniedSubresources: async (ids: string[]) => { + checks.push({accountId, ids}); + return []; + }, + }), + describeConnectedAccount: async () => null, + } as unknown as ObserverClientUser; + let release!: () => void; + let reached!: () => void; + let parked = new Promise(resolve => { release = resolve; }); + let prompted = new Promise(resolve => { reached = resolve; }); + let firstPrompts = 0; + // These local callbacks stand in for the client RPC callbacks on the direct DO call. + const firstConfig = { + configure: async (needs: {gatekeeperId: number}[]) => { + firstPrompts++; + reached(); + await parked; + return configureAll.configure(needs); + }, + } as unknown as ObserverConfigureCallback; + let secondPrompts = 0; + const secondConfig = { + configure: async (needs: {gatekeeperId: number}[]) => { + secondPrompts++; + return configureAll.configure(needs); + }, + } as unknown as ObserverConfigureCallback; + + let callA = impl.ensureObserver("alice", clientUser, "build", firstConfig); + await prompted; + let callB = impl.ensureObserver("alice", clientUser, "build", secondConfig); + + // Another profile makes progress while Alice chooses, without opening her second prompt. + await impl.ensureObserver("bob", clientUser, "build", configureAll); + let secondPromptsBeforeRelease = secondPrompts; + let observerBeforeRelease = impl.storage.observers.get("alice"); + + release(); + await Promise.all([callA, callB]); + + let alice = impl.storage.observers.get("alice")!; + let bob = impl.storage.observers.get("bob")!; + let aliceRegistrations = registrations.filter(entry => entry.observerId !== bob.observerId); + expect(secondPromptsBeforeRelease).toBe(0); + expect(observerBeforeRelease).toBeUndefined(); + expect(firstPrompts).toBe(1); + expect(secondPrompts).toBe(0); + expect(alice.accountChoices).toEqual({ 1: 10, 2: 20 }); + expect(aliceRegistrations.length).toBeGreaterThanOrEqual(2); + expect(aliceRegistrations.every(entry => entry.observerId === alice.observerId)).toBe(true); + if (workspace === "mixed") { + expect(aliceRegistrations.every(entry => entry.id === 1)).toBe(true); + expect(checks.length).toBeGreaterThanOrEqual(2); + expect(checks.every(check => check.accountId === 20 && check.ids.join() === "read")) + .toBe(true); + } + }); + }); + + it("releases the profile queue when account selection is refused", async () => { + let stub = env.TEST_OVERSEER.getByName("observer-verify-queue-refusal"); + await runInDurableObject(stub, async (instance: OverseerDurableObject) => { + let impl = instance["impl"]; + seedOwner(impl); + seedGatekeepers(impl); + setObserverFacets(impl); + let release!: () => void; + let reached!: () => void; + let parked = new Promise(resolve => { release = resolve; }); + let prompted = new Promise(resolve => { reached = resolve; }); + // A rejected client RPC must release the same queue as a verifier refusal. + const refusingConfig = { + configure: async () => { + reached(); + await parked; + throw new Error("account selection refused"); + }, + } as unknown as ObserverConfigureCallback; + + let refused = impl.ensureObserver("alice", fakeClientUser, "build", refusingConfig); + refused.catch(() => {}); + await prompted; + let admitted = impl.ensureObserver("alice", fakeClientUser, "build", configureAll); + release(); + + await expect(refused).rejects.toThrow("account selection refused"); + await admitted; + expect(impl.storage.observers.get("alice")?.accountChoices).toEqual({ 1: 10, 2: 20 }); + }); + }); + + it("admits against live requirements when an observed legacy connection is removed during its prompt", async () => { + let stub = env.TEST_OVERSEER.getByName("observer-verify-removed-during-prompt"); + await runInDurableObject(stub, async (instance: OverseerDurableObject) => { + let impl = instance["impl"]; + seedOwner(impl); + seedGatekeepers(impl, [1]); + seedTrackedGatekeeper(impl, 2); + seedWorkspaceRequirement(impl, 2, "live/read"); + let checkedAccounts: number[] = []; + let checks: string[][] = []; + const clientUser = { + getVerifier: async (accountId: number): Promise => { + checkedAccounts.push(accountId); + if (accountId === 10) throw new Error("Removed connection must not be checked"); + return { + deniedSubresources: async (ids: string[]) => { + checks.push(ids); + return ids.filter(id => id !== "live/read"); + }, + }; + }, + describeConnectedAccount: async () => null, + } as unknown as ObserverClientUser; + let removed: number[] = []; + setObserverFacets(impl, { removeObserver: async id => { removed.push(id); } }); + let release!: () => void; + let reached!: () => void; + let parked = new Promise(resolve => { release = resolve; }); + let prompted = new Promise(resolve => { reached = resolve; }); + // This local callback keeps the real admission parked across the connection removal. + const config = { + configure: async (needs: {gatekeeperId: number}[]) => { + reached(); + await parked; + return configureAll.configure(needs); + }, + } as unknown as ObserverConfigureCallback; + + let admission = impl.ensureObserver("alice", clientUser, "build", config); + await prompted; + impl.removeGatekeeper(1); + release(); + + await admission; + expect(impl.storage.observers.get("alice")?.accountChoices[2]).toBe(20); + expect(checkedAccounts).toContain(20); + expect(checkedAccounts).not.toContain(10); + expect(checks.length).toBeGreaterThan(0); + expect(checks.every(ids => ids.length === 1 && ids[0] === "live/read")).toBe(true); + expect(impl.listObserverRequirements("build").map(need => need.gatekeeperId)).toEqual([2]); + expect(removed).toEqual([]); + }); + }); }); diff --git a/packages/workshop-backend/src/observability.ts b/packages/workshop-backend/src/observability.ts index e31410df82..c6c7a6ad91 100644 --- a/packages/workshop-backend/src/observability.ts +++ b/packages/workshop-backend/src/observability.ts @@ -1,4 +1,4 @@ -import { createObservabilityContext } from "@gadgets/observability/observability-context"; +import { createObservabilityContext, type Logger } from "@gadgets/observability/observability-context"; /** Observability fields emitted by the Workshop backend. */ export type WorkshopObservabilityFields = { @@ -53,10 +53,13 @@ export type WorkshopObservabilityFields = { vendorId: string; }; +/** A logger restricted to the Workshop backend's field vocabulary. */ +export type WorkshopLogger = Logger; + /** Ambient observability fields for one Workshop operation. */ export const obsContext = createObservabilityContext(); /** Creates a logger restricted to the Workshop backend's field vocabulary. */ -export function createWorkshopLogger(component: string) { +export function createWorkshopLogger(component: string): WorkshopLogger { return obsContext.createLogger({ component }); } diff --git a/packages/workshop-backend/src/observer-ledger.ts b/packages/workshop-backend/src/observer-ledger.ts new file mode 100644 index 0000000000..c68f0bfa72 --- /dev/null +++ b/packages/workshop-backend/src/observer-ledger.ts @@ -0,0 +1,658 @@ +// Kernel-owned observer tracking for connections that advertise +// ResourceDescription.supportsObserverTracking. docs/observers.md §10 describes the behavior and +// plans/kernel-observer-tracking.md the reasons for it; this comment covers only what the code +// here must keep true: +// +// - Requirements live only in the action log: an observation that kept `subresources` is one, +// attributed to a context by its caller (observerContextOf). A read must never commit without +// its record, nor a record land without its check. +// - Every check fails closed, and no permission is cached between operations. +// - Each decision is confirmed synchronously before its caller publishes it: an observation's +// audience before the audit record is written (authorizeTrackedObservation), and an admission's +// scope before the observer record is (OverseerImpl.ensureObserver, via observerScope()). +// - Legacy connections never reach these checks: their addObserver()/removeObserver()/ +// excludeObservers protocol is untouched, and a workspace may mix both. + +import type { CollaboratorRole, WorkpieceId } from "@gadgets/workshop-shared/api"; +import type { + Gatekeeper, GatekeeperUserVerifier, ObservationDescription, ResourceDescription, + TrackedObservationDescription, +} from "@gadgets/workshop-shared/gatekeeper"; +import { + connectionVendorId, observerContextOf, observerRequirementPrefix, observerRequirementSubresource, + type ActionRecord, type GatekeeperCaller, type GatekeeperRecord, type ObserverContext, + type ObserverRecord, type OverseerStorage, +} from "./storage-schema/overseer-storage"; +import type { SharingManager } from "./sharing"; +import type { UserDurableObject } from "./user"; +import type { WorkshopLogger } from "./observability"; + +/** + * Most subresource IDs one observation, one page of a historical export, or one + * `verifyObserverAccess()` call may carry; accumulated history has no count cap, so admission + * checks it in batches of this size. + */ +const MAX_SUBRESOURCES = 10_000; + +/** Longest subresource ID accepted. */ +const MAX_SUBRESOURCE_LENGTH = 512; + +// Budget only the ID array: two bytes per UTF-16 code unit plus serialization overhead per +// element. Leave ample room for the action's metadata below SQLite's 8 MiB row limit. +const SUBRESOURCE_CHUNK_BYTES = 4 * 1024 * 1024; + +/** + * Most observers sent to `verifyObserverAccess()` in one call, bounding the verifier fan-out each + * call asks of the gatekeeper (gatekeeper-kit caps a binding's per-read fan-out the same way). + */ +const CHECK_OBSERVER_BATCH_SIZE = 10; + +/** Most verifier lookups, or `verifyObserverAccess()` calls, in flight at once for one check. */ +const CHECK_CONCURRENCY = 4; + +/** + * How many times an admission or an observation re-runs its checks because what it must verify + * changed while they were in flight, before it gives up with a retryable error. + */ +export const MAX_VERIFICATION_PASSES = 3; + +/** Why a read was refused when a collaborator's access to it could not be confirmed. */ +const OBSERVATION_REFUSED = + "This was blocked because it contains data that a current collaborator could not be " + + "confirmed to have access to."; + +// The gatekeeper as an opted-in connection promises it (see +// ResourceDescription.supportsObserverTracking). RPC stubs can't report method presence, and +// their typing makes an optional method uncallable, hence this narrowed view. +type TrackingGatekeeper = Fetcher & + Required, "getHistoricalSubresources" | "verifyObserverAccess">>>; + +/** The observer identity and selected account reached by one observation. */ +type ObservationRecipient = Pick & { + accountId: number; + + /** Identifies this recipient's check: a changed account choice must be checked again. */ + key: string; +}; + +/** One observer to check, with the account they chose for the connection being checked. */ +type ObserverAccount = { + observerId: string; + user: DurableObjectStub; + accountId: number; +}; + +/** What a collaborator holding some role is verified against (see ObserverLedger.observerScope). */ +export type ObserverScope = { + gatekeepers: GatekeeperRecord[]; + requirements: Map; + /** + * `nextActionId` when `requirements` was read. The append-only action log is the only + * requirement store and only a new record adds one, so while this is unchanged, so are the + * requirements of unchanged connections. + */ + nextActionId: number; +}; + +// Whether two scopes hold the same connections, each on the same observer protocol. +function sameObserverConnections(a: GatekeeperRecord[], b: GatekeeperRecord[]): boolean { + let tracked = new Map(a.map(gk => [gk.id, gk.observerTracking])); + return b.length === tracked.size && + b.every(gk => tracked.has(gk.id) && tracked.get(gk.id) === gk.observerTracking); +} + +/** + * Whether a verification pass over `verified` also covers `current`: the same connections, each + * on the same observer protocol, and no tracked requirement the pass did not check. + */ +export function observerScopeCovers(verified: ObserverScope, current: ObserverScope): boolean { + if (!sameObserverConnections(verified.gatekeepers, current.gatekeepers)) return false; + return current.requirements === verified.requirements || + Array.from(current.requirements).every(([id, subresources]) => { + let checked = new Set(verified.requirements.get(id)); + return subresources.every(subresource => checked.has(subresource)); + }); +} + +/** + * What the ledger needs from the Overseer, which satisfies this structurally and passes itself + * (as with OverseerMigrationHost). + */ +export interface ObserverLedgerHost { + ctx: DurableObjectState; + storage: OverseerStorage; + logger: WorkshopLogger; + users: DurableObjectNamespace; + getSharingManager(): Promise; + getGatekeeperFacet(id: number): Promise>>; + gatekeeperFacetName(id: WorkpieceId): string; + creationPending(id: WorkpieceId): boolean; + + /** Throws (retryably) while the connection is held back pending a restart, or importing. */ + assertGatekeeperUsable(id: number): void; + + /** Whether `executionId` is the executeCode invocation still running for `chatId`. */ + isLiveExecution(chatId: number, executionId: string): boolean; + + /** + * A connection's historical import aborted its facet: restart the workspace, so that no client + * keeps a stub to the aborted facet and sessions admitted under the legacy protocol reopen + * against the imported requirements, and hold the connection back until the reset lands. + */ + restartForObserverTracking(gatekeeperId: WorkpieceId): void; + + /** Best-effort removal of every legacy observer registration on `gatekeeperId`. */ + retireLegacyObserverRegistrations(gatekeeperId: WorkpieceId): Promise; +} + +/** + * The verifier for `accountId` in `user`'s own User DO. Throws if the account is gone, with the + * reason the user sees, or belongs to another vendor (getVerifier's own error). + */ +export async function requireVerifier(user: DurableObjectStub, + accountId: number, vendorId: string) { + let verifier = await user.getVerifier(accountId, vendorId); + if (!verifier) throw new Error("This account is no longer connected."); + return verifier; +} + +/** + * Validates one observation's, or one export page's, reported subresource IDs (the RPC layer has + * checked an observation's are strings, but not an export's) and deduplicates them without + * interpreting them. Each must be well-formed Unicode: requirement keys hold them, and storage + * would replace a lone surrogate, merging distinct IDs and checking a different one at admission. + */ +export function normalizeSubresources(ids: unknown): string[] { + if (!Array.isArray(ids) || ids.some(id => typeof id !== "string")) { + throw new Error("Subresources must be a list of strings."); + } + if (ids.length > MAX_SUBRESOURCES) { + throw new Error( + `An observation or export page may report at most ${MAX_SUBRESOURCES} subresources.`); + } + for (let id of ids) { + if (id.length === 0 || id.length > MAX_SUBRESOURCE_LENGTH) { + throw new Error( + `Subresource IDs must be 1 to ${MAX_SUBRESOURCE_LENGTH} characters long.`); + } + if (!id.isWellFormed()) throw new Error("Subresource IDs must be well-formed Unicode."); + } + return [...new Set(ids)]; +} + +// Drains a connection's historical export to its end: only `null` ends it, since a page may be +// empty mid-walk. Each page is held to an observation's limits. +async function readHistoricalExport(facet: TrackingGatekeeper): Promise { + using cursor = await facet.getHistoricalSubresources(); + let history = new Set(); + for (let page = await cursor.next(); page !== null; page = await cursor.next()) { + let ids = normalizeSubresources(page); + for (let id of ids) history.add(id); + } + return [...history]; +} + +/** The observer ledger of one workspace. See the module comment. */ +export class ObserverLedger { + // In-flight initializations, one per connection (see initialize()). + #initializing = new Map>(); + + // Connections whose historical import is in flight (see isImporting). + #importing = new Set(); + + constructor(private host: ObserverLedgerHost) {} + + get #storage() { return this.host.storage; } + + /** Whether the Workshop tracks this existing connection's observations itself. */ + isTracked(gatekeeperId: WorkpieceId): boolean { + return this.#storage.gatekeepers.get(gatekeeperId)?.observerTracking === true; + } + + /** + * Whether the connection's historical import is in flight. Nothing may reach the connection + * until it commits; the Overseer's assertGatekeeperUsable() refuses it meanwhile. + */ + isImporting(gatekeeperId: WorkpieceId): boolean { + return this.#importing.has(gatekeeperId); + } + + // --------------------------------------------------------------------------------------- + // Connection lifecycle + + /** + * Called by addGatekeeper() with a connection's first description, before its record is + * published: a connection advertising support from creation has read nothing yet, so it starts + * tracked with no history. Mutates `record`; the caller puts it. + */ + adoptCreatedConnection(record: GatekeeperRecord, description: ResourceDescription): void { + if (!description.supportsObserverTracking) return; + if (!connectionVendorId(record)) { + throw new Error("Only a vendor's connections can opt into observer tracking."); + } + record.observerTracking = true; + } + + /** + * Describe a legacy connection and import it if it now advertises support. Called at admission + * before the connection's addObserver() (see OverseerImpl.ensureObserver), so a collaborator is + * never registered with an upgraded gatekeeper, and when a read through the connection calls + * authorizeTrackedObservation(). Single-flight per connection. Does nothing for a connection + * that is tracked already, vendorless, removed, still legacy, or a pending creation (simulated, + * so nothing to track until it is approved). Throws if the connection can't be described or its + * history can't be imported, so a failed discovery never falls back to the legacy protocol. + */ + initialize(gatekeeperId: WorkpieceId): Promise { + if (this.isTracked(gatekeeperId) || this.host.creationPending(gatekeeperId)) { + return Promise.resolve(); + } + let pending = this.#initializing.get(gatekeeperId); + if (pending === undefined) { + pending = this.#initialize(gatekeeperId) + .finally(() => this.#initializing.delete(gatekeeperId)); + this.#initializing.set(gatekeeperId, pending); + } + return pending; + } + + async #initialize(gatekeeperId: WorkpieceId): Promise { + let record = this.#storage.gatekeepers.get(gatekeeperId); + if (!connectionVendorId(record)) return; // vendorless or gone: never verifies anyone + + let description: ResourceDescription; + try { + description = await (await this.host.getGatekeeperFacet(gatekeeperId)).describe(); + } catch (error) { + throw new Error( + "This connection could not confirm its observer tracking protocol, so it can't be " + + "used right now. Please retry.", {cause: error}); + } + + if (!this.#storage.gatekeepers.get(gatekeeperId)) { + throw new Error("This connection has been removed from the workspace."); + } + if (description.supportsObserverTracking) await this.#import(gatekeeperId); + } + + // The fenced historical import of a legacy connection. Aborting the facet ends the sessions + // minted under the legacy protocol, and isImporting() refuses everything else until the import + // settles. The abort also invalidates every client's stub to the facet, so the workspace then + // restarts whether or not the import commits. A failure commits nothing, so the next tracked + // observation or admission exports again; after the commit, the imported history is + // authoritative and export is never asked for again. + async #import(gatekeeperId: WorkpieceId): Promise { + this.#importing.add(gatekeeperId); + let subresources: string[]; + try { + this.host.ctx.facets.abort(this.host.gatekeeperFacetName(gatekeeperId), + new Error("Connection restarted to move its observer tracking into the workspace.")); + let facet = await this.host.getGatekeeperFacet(gatekeeperId) as TrackingGatekeeper; + subresources = await readHistoricalExport(facet); + + let record = this.#storage.gatekeepers.get(gatekeeperId); + if (!record) throw new Error("This connection has been removed from the workspace."); + this.#storage.transaction(() => { + this.#storage.gatekeepers.put({...record, observerTracking: true}); + if (subresources.length > 0) this.#recordImportedHistory(record, subresources); + }); + } finally { + // In one synchronous block, so the connection is never usable between the two fences. + this.#importing.delete(gatekeeperId); + this.host.restartForObserverTracking(gatekeeperId); + } + this.host.logger.info("imported connection history into the observer ledger", { + event: "observer.ledger.import.completed", gatekeeperId, size: subresources.length, + }); + + // The restart re-admits live sessions under the ledger; retire legacy registrations + // best-effort meanwhile. + this.host.ctx.waitUntil(this.host.retireLegacyObserverRegistrations(gatekeeperId)); + } + + /** + * Persists an observation as byte-bounded requirement records, returning their action IDs. + * The caller must include this and any policy changes in one storage transaction, and publish + * notifications only after it commits. Verification must already cover the complete read. + */ + recordObservation(record: Omit, "id">): number[] { + let subresources = record.description.subresources; + let count = subresources?.length ?? 0; + let actionIds: number[] = []; + let start = 0; + do { + let end = start; + let bytes = 0; + while (end < count) { + let size = 2 * subresources![end].length + 8; + if (bytes + size > SUBRESOURCE_CHUNK_BYTES) break; + bytes += size; + end++; + } + let id = this.#storage.nextActionId.get(); + this.#storage.nextActionId.put(id + 1); + this.#storage.actions.put({ + ...record, id, + description: subresources && (start !== 0 || end !== subresources.length) + ? {...record.description, subresources: subresources.slice(start, end)} + : record.description, + }); + actionIds.push(id); + start = end; + } while (start < count); + return actionIds; + } + + // No session read the imported history, so its records belong to the workspace, not a chat. + #recordImportedHistory(record: GatekeeperRecord, subresources: string[]): void { + this.recordObservation({ + gatekeeperId: record.id, + caller: {from: "user"}, + resourceTitle: record.resourceTitle, + resourceUrl: record.resourceUrl, + createdAt: new Date(), + state: "approved", + type: "observation", + description: { + title: "Imported earlier reads", + description: "The Workshop took over observer tracking for this connection and recorded " + + "what it had read before.", + subresources, + }, + }); + } + + // --------------------------------------------------------------------------------------- + // Observation + + /** + * The ledger half of Overseer.authorizeObservation(), for the legacy protocol: calls `commit` + * synchronously, recording no requirement. Each observation is held to the connection's + * protocol when it arrives, whichever protocol its session was minted under: once tracked, every + * observation through this method is refused. So is one carrying `subresources`, which only + * authorizeTrackedObservation() enforces. + */ + authorizeObservation(gatekeeperId: WorkpieceId, description: ObservationDescription, + commit: () => undefined): void { + this.#assertUsable(gatekeeperId); + // RPC validation forwards properties the method doesn't declare, and recording these + // unchecked would let the read through without checking anyone. + if ("subresources" in description) { + throw new Error( + "This observation was blocked: subresources must be reported through " + + "authorizeTrackedObservation()."); + } + if (this.isTracked(gatekeeperId)) { + throw new Error( + "This observation was blocked: the connection must report its subresources through " + + "authorizeTrackedObservation()."); + } + return commit(); + } + + /** + * The ledger half of Overseer.authorizeTrackedObservation(). Through a legacy connection, the + * call shows that its gatekeeper upgraded, so this initializes the connection (see initialize()) + * and refuses the read, which came through a session minted under the legacy protocol (the + * import restarts its facet). For a tracked connection, attributes the read to its caller's + * context and checks the reported IDs for everyone that context reaches, then recomputes that + * audience after the awaits and checks whoever it gained: sharing changes, newly published + * observers, and changed account choices. Calls `commit` synchronously with the validated IDs at + * the final decision, so no admission can publish between the audience check and the audit + * write; resolving a promise first would let one publish before the caller writes the + * requirement. The callback must not await. A pending creation commits with undefined, + * recording no requirement. + */ + async authorizeTrackedObservation( + gatekeeperId: WorkpieceId, description: TrackedObservationDescription, + caller: GatekeeperCaller, executionId: string | undefined, + commit: (subresources: string[] | undefined) => undefined): Promise { + this.#assertUsable(gatekeeperId); + if (this.host.creationPending(gatekeeperId)) return commit(undefined); + if (!this.isTracked(gatekeeperId)) { + // Only an upgraded gatekeeper calls this, so import the connection now. The read came + // through a session minted under the legacy protocol, whose facet the import restarts, so + // it is refused either way. + await this.initialize(gatekeeperId); + throw new Error(this.isTracked(gatekeeperId) + ? "This connection now supports the Workshop's observer tracking, which must be set " + + "up before it can be used. Please retry." + : "This observation was blocked: the connection does not advertise observer tracking."); + } + + // RPC validation forwards properties the method doesn't declare. + if ("excludeObservers" in description && description.excludeObservers !== undefined) { + throw new Error( + "This observation was blocked: a connection tracked by the Workshop cannot exclude " + + "observers."); + } + // Attributed to the trusted binding authorizer's caller, not its holder or a gatekeeper- + // supplied ID: passing that capability elsewhere does not change its context. A hook firing + // is attributed to the gadget its record says it wakes. + let context = observerContextOf(caller); + if (context === undefined) { + throw new Error("This observation has no attributable chat or gadget."); + } + let subresources = normalizeSubresources(description.subresources); + if (subresources.length === 0) { + this.#assertLive(gatekeeperId, caller, executionId); + return commit(subresources); + } + + let connection = this.#storage.gatekeepers.get(gatekeeperId)!; + let sharing = await this.host.getSharingManager(); + let verified = new Set(); + for (let pass = 0; ; pass++) { + this.#assertLive(gatekeeperId, caller, executionId); + let unverified = this.#audience(sharing, context, gatekeeperId) + .filter(observer => !verified.has(observer.key)); + if (unverified.length === 0) return commit(subresources); + if (pass > MAX_VERIFICATION_PASSES) { + throw new Error( + "Collaborators' access kept changing while this was being checked. Please retry."); + } + let denied: string[]; + try { + denied = await this.#deniedObservers(connection, unverified.map(observer => ({ + observerId: observer.observerId, accountId: observer.accountId, + user: this.host.users.get(this.host.users.idFromName(observer.profileId)), + })), subresources); + } catch (error) { + this.host.logger.warn("observer check refused an observation", { + event: "observer.ledger.observation.refused", gatekeeperId, error, + }); + throw new Error(OBSERVATION_REFUSED, {cause: error}); + } + if (denied.length > 0) { + for (let observerId of denied) { + this.host.logger.warn("observer check refused an observation", { + event: "observer.ledger.observation.refused", gatekeeperId, observerId, + }); + } + throw new Error(OBSERVATION_REFUSED); + } + for (let observer of unverified) verified.add(observer.key); + } + } + + // Refuses authorization through a removed connection, or one held back or importing. + #assertUsable(gatekeeperId: WorkpieceId): void { + if (!this.#storage.gatekeepers.get(gatekeeperId)) { + throw new Error("This connection has been removed from the workspace."); + } + this.host.assertGatekeeperUsable(gatekeeperId); + } + + // #assertUsable(), and refuses an agent session whose executeCode invocation has ended. + #assertLive(gatekeeperId: WorkpieceId, caller: GatekeeperCaller, + executionId: string | undefined): void { + this.#assertUsable(gatekeeperId); + if (caller.from === "agent" && + (executionId === undefined || !this.host.isLiveExecution(caller.chatId, executionId))) { + throw new Error( + "This observation was blocked: the code execution its session belongs to has ended."); + } + } + + // Observers without a choice for this connection have had no session since it entered their + // role's verification scope (every widening restarts the workspace), and their next admission + // checks everything. A "use" collaborator outside that scope (nothing binds or hooks the + // connection) usually has no choice and goes unchecked, as under the legacy protocol; a choice + // left from an earlier binding is still checked, which can only refuse more. + #audience(sharing: SharingManager, context: ObserverContext, + gatekeeperId: WorkpieceId): ObservationRecipient[] { + let effective = sharing.computeEffectiveRoles(); + let audience: ObservationRecipient[] = []; + for (let observer of this.#storage.observers.list()) { + let role = effective.get(observer.profileId); + let accountId = observer.accountChoices[gatekeeperId]; + if (role !== undefined && (context.type !== "chat" || role === "build") && + accountId !== undefined) { + let {profileId, observerId} = observer; + audience.push({ + profileId, observerId, accountId, + key: JSON.stringify([profileId, observerId, accountId]), + }); + } + } + return audience; + } + + // --------------------------------------------------------------------------------------- + // Admission + + /** + * The deduplicated subresources a collaborator holding `role` must be able to read through each + * tracked connection in `connections`, the role's verification scope: a "build" collaborator + * reaches every context, a "use" collaborator the workspace and every gadget. Reads only the + * distinct requirement keys, never the observations recorded under them. `connections` are live + * records, so nothing through a connection since removed is read. + */ + requiredSubresources(role: CollaboratorRole, + connections: GatekeeperRecord[]): Map { + let index = this.#storage.actions.byObserverRequirement; + let contexts: ObserverContext["type"][] = + role === "build" ? ["workspace", "chat", "gadget"] : ["workspace", "gadget"]; + let required = new Map(); + for (let connection of connections) { + if (connection.observerTracking !== true) continue; + let ids = new Set(); + for (let type of contexts) { + for (let key of index.listKeys({prefix: observerRequirementPrefix(connection.id, type)})) { + ids.add(observerRequirementSubresource(key)); + } + } + if (ids.size > 0) required.set(connection.id, [...ids]); + } + return required; + } + + /** + * Everything a collaborator holding `role` is verified against: `gatekeepers`, the role's + * verification scope, and the requirements recorded through the tracked ones that the role + * reaches (see requiredSubresources()). Reuses `previous`'s requirements while neither its + * connections nor `nextActionId` have changed, so an admission nothing disturbs reads them once. + */ + observerScope(role: CollaboratorRole, gatekeepers: GatekeeperRecord[], + previous?: ObserverScope): ObserverScope { + let nextActionId = this.#storage.nextActionId.get(); + let requirements = previous?.nextActionId === nextActionId && + sameObserverConnections(previous.gatekeepers, gatekeepers) + ? previous.requirements + : this.requiredSubresources(role, gatekeepers); + return {gatekeepers, requirements, nextActionId}; + } + + /** + * Check that `accountId` (in `user`'s own User DO) can read `subresources` through + * `connection`, asking the connection about observer `observerId`. Throws, with a reason for + * the user, on any refusal: account gone, wrong vendor, missing method, failed call, malformed + * result, or denial. + */ + async checkAccess(user: DurableObjectStub, accountId: number, + observerId: string, connection: GatekeeperRecord, + subresources: string[]): Promise { + let denied = await this.#deniedObservers( + connection, [{observerId, user, accountId}], subresources); + if (denied.length > 0) { + throw new Error( + "This account can't access some of the data this workspace read through this " + + "service."); + } + } + + // Looks up each observer's verifier for their chosen account (always, so a disconnected account + // is refused even with nothing to check), then asks the connection's verifyObserverAccess() + // about `subresources` in bounded batches. Returns the IDs of the observers denied by the first + // batch that denies anyone. Throws if a verifier can't be found (getVerifier's own reason), or + // if the gatekeeper fails to answer or answers malformed. + async #deniedObservers(connection: GatekeeperRecord, observers: ObserverAccount[], + subresources: string[]): Promise { + let vendorId = connectionVendorId(connection)!; + let verifiers = new Map>(); + await boundedAll(observers, async ({observerId, user, accountId}) => { + verifiers.set(observerId, await requireVerifier(user, accountId, vendorId)); + }); + if (subresources.length === 0) return []; + + let chunks: {id: string, verifier: Fetcher}[][] = []; + for (let i = 0; i < observers.length; i += CHECK_OBSERVER_BATCH_SIZE) { + chunks.push(observers.slice(i, i + CHECK_OBSERVER_BATCH_SIZE).map(({observerId}) => ({ + id: observerId, verifier: verifiers.get(observerId)!, + }))); + } + let facet = await this.host.getGatekeeperFacet(connection.id) as TrackingGatekeeper; + let denied = new Set(); + for (let i = 0; i < subresources.length && denied.size === 0; i += MAX_SUBRESOURCES) { + let batch = subresources.slice(i, i + MAX_SUBRESOURCES); + await boundedAll(chunks, async chunk => { + for (let id of await askGatekeeper(facet, chunk, batch)) denied.add(id); + }); + } + return [...denied]; + } +} + +// One verifyObserverAccess() call, returning the IDs of the observers it denied: every one its +// answer omits. A failed call or malformed answer throws, which denies every observer in it. +async function askGatekeeper(facet: TrackingGatekeeper, + observers: {id: string, verifier: Fetcher}[], + subresources: string[]): Promise { + let result: unknown; + try { + result = await facet.verifyObserverAccess(observers, subresources); + } catch (error) { + throw new Error("This service could not confirm the account's access.", {cause: error}); + } + let allowed = typeof result === "object" && result !== null && "allowedObserverIds" in result + ? result.allowedObserverIds : undefined; + if (!Array.isArray(allowed)) throw new Error("This service returned an invalid access check."); + let asked = new Set(observers.map(observer => observer.id)); + let granted = new Set(); + // for...of visits a sparse array's holes as undefined, so they are malformed too. + for (let id of allowed) { + if (typeof id !== "string" || !asked.has(id)) { + throw new Error("This service returned an invalid access check."); + } + granted.add(id); + } + return [...asked].filter(id => !granted.has(id)); +} + +// Promise.all over `items` with at most CHECK_CONCURRENCY in flight; rejects with the first +// failure (the rest still settle, their results unused). +async function boundedAll(items: T[], fn: (item: T) => Promise): Promise { + let next = 0; + let failure: {error: unknown} | undefined; + let worker = async () => { + while (next < items.length && failure === undefined) { + let item = items[next++]; + try { + await fn(item); + } catch (error) { + failure ??= {error}; + } + } + }; + await Promise.all(Array.from({length: Math.min(CHECK_CONCURRENCY, items.length)}, worker)); + if (failure !== undefined) throw failure.error; +} diff --git a/packages/workshop-backend/src/overseer.ts b/packages/workshop-backend/src/overseer.ts index 45af94dc34..202f379241 100644 --- a/packages/workshop-backend/src/overseer.ts +++ b/packages/workshop-backend/src/overseer.ts @@ -4,7 +4,7 @@ import { Overseer, WorkspaceMetadata, UiBundle, WorkpieceId, WorkpieceSummary, W import { applyCodeChange, changedGadgets, codeChangeSerializedSize, composeCodeChange, MAX_CODE_CHANGE_SIZE, transformCodeChange, validateCodeChangeContent, validateCodeChangeSchema, requireTextFiles, type FileContent, type CodeContent, type CodeChange } from "@gadgets/workshop-shared/code-change"; -import { type AgentCatalog, Gatekeeper, GatekeeperUserVerifier, HookInitiator, ResourceDescription, ApprovalQueue, ActionDescription, ObservationAuthorizer, ObservationDescription, VendorDescription, SupportedResource, resolveRequestedResource, HookController, HookDescription, ActionKind, GitCache, GitPullHints } from "@gadgets/workshop-shared/gatekeeper"; +import { type AgentCatalog, Gatekeeper, GatekeeperUserVerifier, HookInitiator, ResourceDescription, ApprovalQueue, ActionDescription, ObservationAuthorizer, ObservationDescription, TrackedObservationDescription, VendorDescription, SupportedResource, resolveRequestedResource, HookController, HookDescription, ActionKind, GitCache, GitPullHints } from "@gadgets/workshop-shared/gatekeeper"; import { DurableObject, WorkerEntrypoint, RpcStub as NativeRpcStub, RpcTarget as NativeRpcTarget, restore, @@ -12,7 +12,8 @@ import { import { keyString } from "@gadgets/typed-storage"; import type { ListOptions } from "@gadgets/typed-storage"; import { - actionLastChangedKey, chatChangeClientKey, chatKey, chatKeyPrefix, makeOverseerStorage, + actionLastChangedKey, chatChangeClientKey, chatKey, chatKeyPrefix, connectionVendorId, + makeOverseerStorage, type ActionRecord, type ActiveAgentRecord, type AgentSpawnerBindingProps, type AiChatAgentContext, type BindingRecord, type BlueprintGadgetRecord, type BoundHookRecord, type ChatBindingEntry, @@ -20,6 +21,7 @@ import { type ExternalChatRecord, type ExternalMessageRecord, type GadgetRecord, type GatekeeperCaller, type GatekeeperClass, type GatekeeperRecord, type ObserverRecord, type OverseerStorage, type StoredAssistantMessage, type StoredChatMessage, type StoredChatMetadata, + type StoredObservationDescription, type WorkpieceRecord, type WorktreeRecord, } from "./storage-schema/overseer-storage"; import type { UserAiModelRecord, WorkspaceOutputEntry } from "./storage-schema/user-storage"; @@ -77,6 +79,9 @@ import { checkUsageAndBalance } from "./ai-gateway-billing/limits/usage-checker" import { normalizeAgentCatalog } from "./agent-catalog"; import { refreshCachedBalance } from "./ai-gateway-billing/cloudflare/connection-service"; import { SharingManager, SharingCaller, roleRank } from "./sharing"; +import { + MAX_VERIFICATION_PASSES, ObserverLedger, observerScopeCovers, requireVerifier, +} from "./observer-ledger"; import { AutoApprovalDrainer, autoApprovalRule } from "./auto-approval"; import { collectSlashCommands, invokeSlashCommand } from "./slash-commands"; import { createWorkshopLogger, obsContext } from "./observability"; @@ -265,8 +270,7 @@ function defaultBlueprintBindingTitle(record: GatekeeperRecord, bindingName?: st } function gatekeeperVendorId(record: GatekeeperRecord | undefined): string | undefined { - let spec = record?.creationSpec; - return spec && "vendorId" in spec ? spec.vendorId.toLowerCase() : undefined; + return connectionVendorId(record)?.toLowerCase(); } // Produce a valid, unused binding name from a suggested base name: sanitized to identifier @@ -340,7 +344,7 @@ function observerVendorId(record: GatekeeperRecord): string | null { "owner must remove the connection before the workspace can be shared, or start a new " + "workspace."); } - return "vendorId" in record.creationSpec ? record.creationSpec.vendorId : null; + return connectionVendorId(record) ?? null; } // Human-readable title for an observer binding -- what the user sees both in the config modal and in @@ -538,7 +542,10 @@ function actionRecordToLog(record: ActionRecord): ActionLogEntry { let gatekeeperId = record.gatekeeperId >= 0 ? record.gatekeeperId : undefined; switch (record.type) { - case "observation": + case "observation": { + // Subresource IDs are enforcement data (see ActionRecord), not display: they + // name what was read, and one observation may report thousands. + let {subresources: _, ...description} = record.description; return { id: record.id, gatekeeperId, @@ -547,8 +554,9 @@ function actionRecordToLog(record: ActionRecord): ActionLogEntry { createdAt: record.createdAt, state: record.state, type: "observation", - description: record.description, + description, }; + } case "action": return { id: record.id, @@ -735,6 +743,10 @@ class OverseerImpl implements AgentHooks { // from this via `new GitCacheImpl(...)`. readonly gitCache: WorkspaceGitCache; + // Kernel-owned observer tracking for connections that opted in (see observer-ledger.ts). + // Legacy connections keep the addObserver/excludeObservers protocol below. + readonly observerLedger: ObserverLedger; + // Per-chat in-memory state for running agents. #liveChats = new Map(); #chatSubscribers: Set> = new Set(); @@ -1229,6 +1241,7 @@ class OverseerImpl implements AgentHooks { }); this.users = this.ctx.exports.UserDurableObject; this.ownerId = this.storage.ownerId.get(); + this.observerLedger = new ObserverLedger(this); // Run any pending storage migration (see storage-schema/overseer-migrations.ts) before // anything else can touch storage. This must happen in the constructor (not just open()) @@ -1380,6 +1393,11 @@ class OverseerImpl implements AgentHooks { return this.defaultGadgetId === id ? "gadget" : `gadget${id}`; } + // Facet name for the given gatekeeper. + gatekeeperFacetName(id: WorkpieceId): string { + return `gatekeeper${id}`; + } + // Resolve an agent tool's optional workpiece reference. Absent means the workspace's default // gadget; the error when there is none tells the agent how to proceed. When `mustExist` is // set, the workpiece must currently exist in the registry (used by live file tools; history @@ -2665,7 +2683,8 @@ class OverseerImpl implements AgentHooks { env[name] = this.makeBindingLoopback( {type: "worktree", id: entry.id, executionId}, caller); } else if (this.storage.gatekeepers.get(entry.id)) { - env[name] = this.makeBindingLoopback({type: "gatekeeper", id: entry.id}, caller); + env[name] = this.makeBindingLoopback( + {type: "gatekeeper", id: entry.id, executionId}, caller); } break; } @@ -4881,7 +4900,7 @@ class OverseerImpl implements AgentHooks { // The bare facet stub behind getGatekeeperFacet(): a request made on it leaves the facet unable // to call its own ctx.restore(). #getGatekeeperFacetRaw(id: number, cls?: GatekeeperClass): Fetcher> { - return this.ctx.facets.get(`gatekeeper${id}`, async () => { + return this.ctx.facets.get(this.gatekeeperFacetName(id), async () => { let resolved = cls ?? this.storage.gatekeepers.get(id)?.class; if (!resolved) { throw new Error("no such gatekeeper?"); @@ -4945,7 +4964,7 @@ class OverseerImpl implements AgentHooks { configuration: string | null}): Promise { let id = record.gatekeeperId; // A created resource exists only once its creation applies, and that stays pending until then. - if (record.action !== "create" && this.#creationPending(id)) { + if (record.action !== "create" && this.creationPending(id)) { throw new Error("This resource doesn't exist yet. Approve its creation first."); } let gatekeeper = await this.getGatekeeperFacet(record.gatekeeperId); @@ -4996,7 +5015,7 @@ class OverseerImpl implements AgentHooks { } if (gatekeeperRecord) { // Restart the facet on the new class before anything else reaches it. - this.ctx.facets.abort(`gatekeeper${id}`, + this.ctx.facets.abort(this.gatekeeperFacetName(id), new Error("Connection restarted because its resource was created.")); // Minting skipped addGatekeeper's restart: collaborators had no real resource to be // verified against until now (see createExternalResource). @@ -5091,6 +5110,7 @@ class OverseerImpl implements AgentHooks { gatekeeperRecord.resourceTitle = description.title; gatekeeperRecord.resourceUrl = description.url; gatekeeperRecord.hasSlashCommands = description.hasSlashCommands; + this.observerLedger.adoptCreatedConnection(gatekeeperRecord, description); this.storage.gatekeepers.put(gatekeeperRecord); } catch (error) { // Still the right teardown with nothing published: it deletes the facet we just created, and @@ -5165,7 +5185,7 @@ class OverseerImpl implements AgentHooks { } } - this.ctx.facets.delete(`gatekeeper${id}`); + this.ctx.facets.delete(this.gatekeeperFacetName(id)); this.storage.gatekeepers.delete(id); this.#rebroadcastChatMeta(affectedChats); } @@ -5182,8 +5202,8 @@ class OverseerImpl implements AgentHooks { } case "gatekeeper": - return this.getGatekeeperFacet(target.id) - .then(facet => this.openGatekeeperSession(target.id, facet, caller)); + return this.getGatekeeperFacet(target.id).then(facet => + this.openGatekeeperSession(target.id, facet, caller, target.executionId)); case "worktree": { // The programmatic Worktree binding (worktree-session.ts). A worktree's uncommitted @@ -5200,8 +5220,8 @@ class OverseerImpl implements AgentHooks { if (record.chatId !== caller.chatId) { throw new Error(`No such worktree: ${target.id}`); } - let turn = this.#activeWorktreeTurns.get(caller.chatId); - if (turn === undefined || turn.executionId !== target.executionId) { + let execution = this.#activeExecutions.get(caller.chatId); + if (execution?.worktree === undefined || execution.executionId !== target.executionId) { throw new Error( "This worktree binding is no longer live; worktree bindings are usable only " + "while the executeCode call they were provided to is running."); @@ -5209,9 +5229,9 @@ class OverseerImpl implements AgentHooks { // Commits go to the turn's initiator: in a collaborative chat, a collaborator's work is // attributed to the collaborator, matching how accepted commits use the acting user's // profile. - let initiator = turn.initiator; + let initiator = execution.initiator; return Promise.resolve( - new WorktreeSessionImpl(this, target.id, turn.access, async () => initiator)); + new WorktreeSessionImpl(this, target.id, execution.worktree, async () => initiator)); } case "exports": @@ -5238,8 +5258,8 @@ class OverseerImpl implements AgentHooks { // the owner standing in for a spawner's creator. Resolved at commit time, so read-only use // of env.GIT never waits on the owner's DO. async #gitAuthorFor(caller: GatekeeperCaller): Promise { - let turn = caller.from === "agent" ? this.#activeWorktreeTurns.get(caller.chatId) : undefined; - if (turn !== undefined) return turn.initiator; + let execution = caller.from === "agent" ? this.#activeExecutions.get(caller.chatId) : undefined; + if (execution?.worktree !== undefined) return execution.initiator; return this.gadgetAuthorFor(await retryOnDoReset(() => this.ownerUserDo().whoami(), this.logger)); } @@ -5255,12 +5275,18 @@ class OverseerImpl implements AgentHooks { }; } - // The worktree turn state registered by a running executeCode, keyed by chat (one turn per - // chat, and executeCode calls within it are sequential). `executionId` names the registering - // execution: worktree loopbacks are minted with it and verified against it, so only stubs from - // the currently-running execution resolve. See executeCodeMode. - #activeWorktreeTurns = new Map(); + // The executeCode invocation running for each chat (one turn per chat, and executeCode calls + // within it are sequential), with the turn's worktree state when it has one. `executionId` + // names the registering execution: worktree and gatekeeper loopbacks are minted with it and + // verified against it, so only stubs from the currently-running execution resolve. See + // executeCodeMode. + #activeExecutions = new Map(); + + // ObserverLedgerHost: whether `executionId` is the executeCode invocation running for `chatId`. + isLiveExecution(chatId: number, executionId: string): boolean { + return this.#activeExecutions.get(chatId)?.executionId === executionId; + } // Maps chat ID to action numbers recently performed by that chat's agent. These are drained into // the chat log after the tool returns. `awaitDecision` is true if any captured action needs it. @@ -5283,9 +5309,10 @@ class OverseerImpl implements AgentHooks { async #associateAction(caller: GatekeeperCaller, actionId: number) { try { + // (A user caller's chat is a slash command's, whose own chat event already shows it.) if (caller.from === "agent") { this.#getOrCreateCapturedActions(caller.chatId).actions.push(actionId); - } else if (caller.from !== "hook" && caller.chatId !== undefined && this.ownerId) { + } else if (caller.from === "gadget" && caller.chatId !== undefined && this.ownerId) { let owner = this.users.get(this.users.idFromString(this.ownerId)); let userMeta = await owner.getChatContext(null); this.addChatMessages(caller.chatId, this.gadgetAuthorFor(userMeta.profile), @@ -5298,72 +5325,102 @@ class OverseerImpl implements AgentHooks { } } + // A hook firing's `hookId` is revalidated at the synchronous commit point, after any policy + // lookup has awaited. async authorizeObservation(gatekeeperId: number, description: ObservationDescription, - caller: GatekeeperCaller): Promise { + caller: GatekeeperCaller, hookId?: number): Promise { // Forward exclusion: the gatekeeper may name observers who must not see this observation. Since // v1 has no per-thread hiding, the only way to let such an observation proceed is if no named // observer could reach it -- either they have lost access in the sharing graph, or this // connection has left their role's verification scope. See - // observers-implementation-plan.md §5 Step 5. - if (description.excludeObservers && description.excludeObservers.length > 0) { + // observers-implementation-plan.md §5 Step 5. (Legacy connections only: the ledger below + // refuses every observation through a tracked one, including one tracked since this check + // began.) + if (description.excludeObservers && description.excludeObservers.length > 0 && + !this.observerLedger.isTracked(gatekeeperId)) { await this.#enforceExcludeObservers(gatekeeperId, description.excludeObservers); } - + await this.#commitObservation(gatekeeperId, description, caller, hookId, + commit => this.observerLedger.authorizeObservation(gatekeeperId, description, commit)); + } + + // `executionId` is an agent session's originating execution (see ApprovalQueueImpl). A hook + // firing's `hookId` is revalidated at the synchronous commit point, after any policy lookup or + // observer verification has awaited. + authorizeTrackedObservation(gatekeeperId: number, description: TrackedObservationDescription, + caller: GatekeeperCaller, executionId?: string, + hookId?: number): Promise { + return this.#commitObservation(gatekeeperId, description, caller, hookId, + commit => this.observerLedger.authorizeTrackedObservation( + gatekeeperId, description, caller, executionId, commit)); + } + + // Applies an observation's workspace policy and writes its audit record when `authorize` -- the + // ledger's decision under the observation's protocol -- calls `commit`: synchronously, at its + // final decision, before yielding to any admission or import that could invalidate the audience + // or protocol it checked. + async #commitObservation( + gatekeeperId: number, description: StoredObservationDescription, caller: GatekeeperCaller, + hookId: number | undefined, + authorize: (commit: (subresources?: string[]) => undefined) => void | Promise) + : Promise { // Setting ownerInvitesOnly narrows access to direct owner grants (see // SharingManager.computeEffectiveRoles), so the first observation to set it snapshots who had // access beforehand. The manager may need an RPC on first use; from the flag read below through // the diff after the writes, nothing awaits. let sharing = description.ownerInvitesOnly && !this.storage.ownerInvitesOnly.get() ? await this.getSharingManager() : undefined; - let baseline = sharing && !this.storage.ownerInvitesOnly.get() - ? sharing.computeEffectiveRoles() : undefined; - if (description.containsRestrictedData) { - this.storage.containsRestrictedData.put(true); - } - if (description.ownerInvitesOnly) { - this.storage.ownerInvitesOnly.put(true); - } - - let actionId = this.storage.nextActionId.get(); - this.storage.nextActionId.put(actionId + 1); - - let gatekeeper = this.storage.gatekeepers.get(gatekeeperId); - - let record: ActionRecord = { - id: actionId, - gatekeeperId, - caller, - resourceTitle: gatekeeper?.resourceTitle, - resourceUrl: gatekeeper?.resourceUrl, - createdAt: new Date(), - state: "approved", - type: "observation", - description - }; + await authorize(subresources => { + if (hookId !== undefined) requireLiveHook(this, hookId); + let baseline = sharing && !this.storage.ownerInvitesOnly.get() + ? sharing.computeEffectiveRoles() : undefined; - this.storage.actions.put(record); - this.#associateAction(caller, actionId); + let actionIds = this.storage.transaction(() => { + if (description.containsRestrictedData) { + this.storage.containsRestrictedData.put(true); + } + if (description.ownerInvitesOnly) { + this.storage.ownerInvitesOnly.put(true); + } - if (sharing && baseline) { - let affected = sharing.computeAffectedByOwnerInvitesOnly(baseline); - if (affected.length > 0) { - // Anyone who joined through a link or through another collaborator just lost access (or - // was downgraded), so sever live sessions as removeCollaborator does, after the writes - // above. The cleanup is best-effort and not awaited: the observation shouldn't wait on - // gatekeeper and User-DO round trips, and whatever the restart cuts off self-heals (see - // removeCollaborator). - this.scheduleAccessRestart( - "Gadget restarted to revoke access for people the owner did not add directly."); - this.tearDownLostObservers(affected) - .then(() => this.refreshAffectedCollaboratorListings(affected)) - .catch(err => { - this.logger.warn("failed to clean up after ownerInvitesOnly revoked access", { - event: "sharing.owner.invites.only.cleanup.failed", error: err, + // Only a tracked connection's observation keeps `subresources`, which makes it an observer + // requirement (see ActionRecord); a pending creation's simulated read must not. + let {subresources: _, ...untracked} = description; + let gatekeeper = this.storage.gatekeepers.get(gatekeeperId); + return this.observerLedger.recordObservation({ + gatekeeperId, + caller, + resourceTitle: gatekeeper?.resourceTitle, + resourceUrl: gatekeeper?.resourceUrl, + createdAt: new Date(), + state: "approved", + type: "observation", + description: subresources ? {...untracked, subresources} : untracked, + }); + }); + for (let actionId of actionIds) this.#associateAction(caller, actionId); + + if (sharing && baseline) { + let affected = sharing.computeAffectedByOwnerInvitesOnly(baseline); + if (affected.length > 0) { + // Anyone who joined through a link or through another collaborator just lost access (or + // was downgraded), so sever live sessions as removeCollaborator does, after the writes + // above. The cleanup is best-effort and not awaited: the observation shouldn't wait on + // gatekeeper and User-DO round trips, and whatever the restart cuts off self-heals (see + // removeCollaborator). + this.scheduleAccessRestart( + "Gadget restarted to revoke access for people the owner did not add directly."); + this.tearDownLostObservers(affected) + .then(() => this.refreshAffectedCollaboratorListings(affected)) + .catch(err => { + this.logger.warn("failed to clean up after ownerInvitesOnly revoked access", { + event: "sharing.owner.invites.only.cleanup.failed", error: err, + }); }); - }); + } } - } + }); } async getChatAttachmentData(chatId: number, id: string): Promise { @@ -5794,7 +5851,7 @@ class OverseerImpl implements AgentHooks { // Nobody named can reach the observation. Tear down those who have lost access entirely, since // they are no longer set up to observe at all, and de-register the rest from this gatekeeper // only. A fresh open racing one of these removals is ordered behind it by - // #withObserverGatekeeperLock, so its registration is never silently undone. + // #serialized, so its registration is never silently undone. let allGatekeeperIds = [...this.storage.gatekeepers.list()].map(gk => gk.id); let removals = unauthorized.map(observer => { this.storage.observers.delete(observer.profileId); @@ -6301,24 +6358,29 @@ class OverseerImpl implements AgentHooks { // along. #gatekeepersPendingRestart = new Set(); - // Whether `id` is NOT blocked pending a scheduled restart (see #gatekeepersPendingRestart). - // For callers that enumerate connections (listSlashCommands, prepareChatBindings' ambient - // catalogs) and silently skip a blocked one -- it reappears once the reset lands and clients - // reconnect. A caller reaching for one specific connection throws via assertGatekeeperUsable - // instead. + // Whether `id` is NOT blocked pending a scheduled restart (see #gatekeepersPendingRestart) or + // its observer-tracking import (ObserverLedger.isImporting). For callers that enumerate + // connections (listSlashCommands, prepareChatBindings' ambient catalogs) and silently skip a + // blocked one -- it reappears once the reset lands and clients reconnect. A caller reaching for + // one specific connection throws via assertGatekeeperUsable instead. gatekeeperUsable(id: number): boolean { - return !this.#gatekeepersPendingRestart.has(id); + return !this.#gatekeepersPendingRestart.has(id) && !this.observerLedger.isImporting(id); } - // Throw (retryable) if `id` is blocked pending the scheduled restart; see - // #gatekeepersPendingRestart. Called from getGatekeeperById (the mint clients pipeline on), - // openGatekeeperSession (the chokepoint every gatekeeper session passes through), the - // slash-command invoke in #prepareChatMessage, GadgetClientImpl.bindWithSuggestedName (the + // Throw (retryable) if `id` is blocked pending the scheduled restart or its observer-tracking + // import; see #gatekeepersPendingRestart. Called from getGatekeeperById (the mint clients + // pipeline on), openGatekeeperSession (the chokepoint every gatekeeper session passes through), + // the slash-command invoke in #prepareChatMessage, GadgetClientImpl.bindWithSuggestedName (the // latter two take a client-supplied gatekeeper id and reach the connection outside - // openGatekeeperSession), and startHook (the inbound delivery route, whose arming enable may - // itself be the widening that scheduled the restart). + // openGatekeeperSession), startHook (the inbound delivery route, whose arming enable may itself + // be the widening that scheduled the restart), and the ObserverLedger's authorize methods. assertGatekeeperUsable(id: number): void { - if (!this.gatekeeperUsable(id)) { + if (this.observerLedger.isImporting(id)) { + throw new Error( + "This connection's observer tracking history is being imported. Please retry in a " + + "moment."); + } + if (this.#gatekeepersPendingRestart.has(id)) { throw new Error( "The workspace is restarting to apply a connection change. Please retry."); } @@ -6326,13 +6388,15 @@ class OverseerImpl implements AgentHooks { // Every gatekeeper session -- GatekeeperClientImpl.openSession and binding loopbacks via // startGatekeeperSession alike -- passes through here, so this is where a connection still - // blocked pending a scope-widening restart is refused (see #gatekeepersPendingRestart). + // blocked pending a scope-widening restart is refused (see #gatekeepersPendingRestart). Agent + // loopbacks carry the execution their authorizers belong to. openGatekeeperSession>( - id: number, facet: Fetcher>, caller: GatekeeperCaller) - : Promise> { + id: number, facet: Fetcher>, caller: GatekeeperCaller, + executionId?: string): Promise> { this.assertGatekeeperUsable(id); let queue = this.storage.gatekeepers.get(id)?.creationSpec?.type === "agentSpawner" - ? new AgentSpawnerQueueImpl(this, id, caller) : new ApprovalQueueImpl(this, id, caller); + ? new AgentSpawnerQueueImpl(this, id, caller) + : new ApprovalQueueImpl(this, id, caller, undefined, executionId); // @ts-expect-error TODO: Remove annotation when Cap'n Web fixes cyclic type issues return facet.startSession(queue); } @@ -6367,6 +6431,17 @@ class OverseerImpl implements AgentHooks { return true; } + // ObserverLedgerHost: a historical import aborted the connection's facet, which invalidates every + // stub to it that a client capability holds (GatekeeperClientImpl keeps the facet it was minted + // with), and sessions admitted under the legacy protocol must reopen against the imported + // requirements. So restart even when only the owner is connected, and hold the connection back + // until the reset lands, as for a newly added connection. + restartForObserverTracking(gatekeeperId: WorkpieceId): void { + this.#gatekeepersPendingRestart.add(gatekeeperId); + this.scheduleAccessRestart( + "Gadget restarted to move a connection's observer tracking into the workspace."); + } + // Diff the account-requiring "use" scope against a snapshot taken just before a mutation, and // when it widened onto a live "use" session, restart -- additionally blocking each widened // connection until the reset lands (see #gatekeepersPendingRestart): the severed sessions stay @@ -6484,9 +6559,11 @@ class OverseerImpl implements AgentHooks { // Invoke slash-command requests before committing their visible event and optional generated // message. A result without a message suppresses only the generated message, not the invocation. + // `chatId` is the existing chat the command runs in, to which its reads are attributed; a new + // chat's first command runs before the chat exists, so its reads go to the workspace context. async #prepareChatMessage( message: string | SlashCommandRequest, - hasAttachments: boolean): Promise { + hasAttachments: boolean, chatId?: number): Promise { if (typeof message !== "string") { // A built-in command is handled by the Workshop, not a Gatekeeper: there is nothing to invoke // here. Committing the event is what makes the turn a compaction turn (see isCompactionTurn). @@ -6508,7 +6585,7 @@ class OverseerImpl implements AgentHooks { // Display-only, and from the browser, so a bad value is dropped rather than refused. message = {...message, commandPosition: sanitizeCommandPosition(message)}; using authorizer = new NativeRpcStub( - new SlashCommandAuthorizerImpl(this, gatekeeperId, {from: "user"})); + new SlashCommandAuthorizerImpl(this, gatekeeperId, {from: "user", chatId})); let result = await invokeSlashCommand(facet, message, authorizer); if (result.message === undefined) { return {slashCommand: message, skillName: result.skillName}; @@ -6746,7 +6823,7 @@ class OverseerImpl implements AgentHooks { using _chatMessageReservation = this.reserveChatMessagePreparation(chatId); await this.#migrateChatAttachments(chatId); let prepared = await this.#prepareChatMessage( - message, (canonicalAttachments?.length ?? 0) > 0); + message, (canonicalAttachments?.length ?? 0) > 0, chatId); let attachmentFiles = await this.#prepareWorkspaceAttachmentFiles(chatId, canonicalAttachments); let scratchWorktree = this.#scratchWorktree(chatId); @@ -9021,13 +9098,11 @@ class OverseerImpl implements AgentHooks { crypto.getRandomValues(bytes); let executionId: string = bytes.toBase64(); - // Register the turn's worktree state for the worktree binding loopbacks (see - // startGatekeeperSession's "worktree" case), for exactly this execution's duration -- the - // executionId is minted into the loopbacks below, so stubs from other executions never - // resolve against this registration. - if (worktreeTurn !== undefined) { - this.#activeWorktreeTurns.set(chatId, {access: worktreeTurn, initiator, executionId}); - } + // Register the execution, with the turn's worktree state, for the binding loopbacks (see + // startGatekeeperSession's "worktree" case and ObserverLedger.authorizeTrackedObservation), for + // exactly this execution's duration -- the executionId is minted into the loopbacks below, so + // stubs from other executions never resolve against this registration. + this.#activeExecutions.set(chatId, {executionId, initiator, worktree: worktreeTurn}); if (onOutputText) { this.#codeModeOutputSubscribers.set(executionId, onOutputText); @@ -9116,8 +9191,8 @@ class OverseerImpl implements AgentHooks { return log; } finally { // Guarded by executionId so this cleanup can never clobber a newer registration. - if (this.#activeWorktreeTurns.get(chatId)?.executionId === executionId) { - this.#activeWorktreeTurns.delete(chatId); + if (this.#activeExecutions.get(chatId)?.executionId === executionId) { + this.#activeExecutions.delete(chatId); } this.#codeModeOutputSubscribers.delete(executionId); this.#codeModeResolvers.delete(executionId); @@ -9452,7 +9527,7 @@ class OverseerImpl implements AgentHooks { } // Whether the gatekeeper's resource is still simulated: its creation is pending. - #creationPending(id: WorkpieceId): boolean { + creationPending(id: WorkpieceId): boolean { return [...this.storage.actions.pendingByGatekeeper.get(id)] .some(queued => queued.type === "action" && queued.action === "create"); } @@ -9779,33 +9854,33 @@ class OverseerImpl implements AgentHooks { // A simulated resource has no account behind it to verify anyone against. Approving its // creation restarts every session (see applyPendingAction), so verification happens then. // (Filtered after the listing: reading another index mid-listing would end it.) - return result.filter(gk => !this.#creationPending(gk.id)); + return result.filter(gk => !this.creationPending(gk.id)); } listObserverRequirements(role: CollaboratorRole): ObserverBindingNeed[] { return this.#inScopeGatekeepers(role).map(observerBindingNeed); } - // Tail of the in-flight addObserver/removeObserver chain per `${observerId}/${gatekeeperId}` - // (see #withObserverGatekeeperLock). In-memory only, entries dropped as each chain drains; the - // calls it orders are themselves re-run/repaired across DO restarts. - #observerGatekeeperLocks = new Map>(); + // Tail of the in-flight observer work chain per key (see #serialized). In-memory only, entries + // dropped as each chain drains; the work it orders is itself re-run/repaired across DO restarts. + #observerLocks = new Map>(); - // Serialize this DO's addObserver/removeObserver RPCs per (observer, gatekeeper) pair. The - // overseer is the only caller of either, so ordering our own calls is enough to close the race - // between an exclusion teardown's in-flight removeObserver and a fresh open's addObserver on - // the same pair: the add either lands first or waits for the removal and re-registers cleanly. - async #withObserverGatekeeperLock( - observerId: string, gatekeeperId: number, fn: () => Promise): Promise { - let key = `${observerId}/${gatekeeperId}`; - let prior = this.#observerGatekeeperLocks.get(key) ?? Promise.resolve(); + // Run `fn` once every earlier call with the same `key` has settled. Two kinds of key: + // - `${observerId}/${gatekeeperId}` orders this DO's addObserver/removeObserver RPCs per pair. + // The overseer is the only caller of either, so ordering our own calls is enough to close + // the race between an exclusion teardown's in-flight removeObserver and a fresh open's + // addObserver on the same pair: the add either lands first or waits for the removal and + // re-registers cleanly. + // - `admission/${profileId}` orders one profile's ensureObserver calls (see there). + async #serialized(key: string, fn: () => Promise): Promise { + let prior = this.#observerLocks.get(key) ?? Promise.resolve(); let release!: () => void; let tail = new Promise(resolve => { release = resolve; }); - this.#observerGatekeeperLocks.set(key, tail); + this.#observerLocks.set(key, tail); tail.then(() => { // Drop the entry once the chain drains; a newer tail means someone queued behind us. - if (this.#observerGatekeeperLocks.get(key) === tail) { - this.#observerGatekeeperLocks.delete(key); + if (this.#observerLocks.get(key) === tail) { + this.#observerLocks.delete(key); } }); await prior; // never rejects: each holder settles its own tail via the finally below @@ -9823,7 +9898,7 @@ class OverseerImpl implements AgentHooks { async #removeObserverFromGatekeepers(observerId: string, gatekeeperIds: number[]): Promise { await Promise.all(gatekeeperIds.map(async id => { try { - await this.#withObserverGatekeeperLock(observerId, id, + await this.#serialized(`${observerId}/${id}`, async () => (await this.getGatekeeperFacet(id)).removeObserver(observerId)); } catch (err) { this.logger.warn("failed to remove observer from gatekeeper", { @@ -9833,6 +9908,13 @@ class OverseerImpl implements AgentHooks { })); } + // ObserverLedgerHost: a connection's history moved into the ledger, so the registrations its + // legacy protocol kept are retired (best-effort, never throws). + async retireLegacyObserverRegistrations(gatekeeperId: WorkpieceId): Promise { + await Promise.all(Array.from(this.storage.observers.list(), + observer => this.#removeObserverFromGatekeepers(observer.observerId, [gatekeeperId]))); + } + // Tear down observer records for collaborators who lost access as a result of a sharing change. // For each affected collaborator who is now fully unauthorized (newRole === null) and has an // observer record: best-effort removeObserver on all gatekeeper facets, then delete the record. @@ -9917,25 +9999,33 @@ class OverseerImpl implements AgentHooks { // Bring a non-owner `profileId` into compliance as an observer for their `role`, so that they may // open the Gadget. May invoke `configureCb` to ask the user to choose connected accounts for - // gatekeeper bindings they haven't configured yet. Re-runs `addObserver` (re-verification) for - // already-configured bindings on every open, catching revocation of the user's underlying - // resource access promptly. Returns when fully verified; throws to deny access. - // - // See observers-implementation-plan.md §5 Step 3. - // - // TODO: Concurrent opens by the same profile race this method -- two calls mint two observerIds - // and the last-written record forgets the other's gatekeeper registrations -- so verification - // needs to be serialized per profile. - async ensureObserver( + // gatekeeper bindings they haven't configured yet. Re-verifies already-configured bindings on + // every open, catching revocation of the user's underlying resource access promptly: a legacy + // connection through `addObserver()`, and one the Workshop tracks itself (see observer-ledger.ts) + // by asking it to check every requirement the role reaches with the chosen account. Returns + // when fully verified; throws to deny access. Serialized per profile, so a queued open rereads + // the observer record the preceding one persisted (and they never mint two observerIds); an + // account prompt blocks only that profile. See observers-implementation-plan.md §5 Step 3. + ensureObserver( profileId: string, clientUser: DurableObjectStub, role: CollaboratorRole, configureCb?: RpcStub): Promise { - // 1. Select in-scope gatekeepers. If none require an account, there is nothing to verify and + return this.#serialized(`admission/${profileId}`, + () => this.#ensureObserver(profileId, clientUser, role, configureCb)); + } + + async #ensureObserver( + profileId: string, + clientUser: DurableObjectStub, + role: CollaboratorRole, + configureCb?: RpcStub): Promise { + // 1. Select in-scope gatekeepers, and the requirements recorded through the tracked ones (see + // ObserverLedger.observerScope). If none require an account, there is nothing to verify and // no observer record is needed (built-in gatekeepers never name observers in // excludeObservers). - let inScope = this.#inScopeGatekeepers(role); - if (inScope.length === 0) return; + let scope = this.observerLedger.observerScope(role, this.#inScopeGatekeepers(role)); + if (scope.gatekeepers.length === 0) return; // 2. Load any existing observer record, and build a working copy of its account choices. let record = this.storage.observers.get(profileId); @@ -9946,14 +10036,13 @@ class OverseerImpl implements AgentHooks { // observer's `observerId` is already persisted, so it stays resolvable no matter how this call // ends -- which is what makes keeping their registrations on a failure safe (see the catch). let returningObserver = record !== undefined; - // Gatekeepers this call touched, for a *first-time* observer's rollback only (see the catch): - // those we successfully registered the observer with, and those that refused (or whose account - // was gone) and have not verified since. - let newlyAdded = new Set(); - let invalidated = new Set(); + // Legacy gatekeepers this call tried to register the observer with, for a *first-time* + // observer's rollback only (see the catch): a refused addObserver may still have registered + // them. Tracked connections hold no registrations. + let attempted = new Set(); // Failures from the previous pass, keyed by gatekeeper id: an already-configured binding whose - // chosen account was disconnected, or which the gatekeeper refused. + // chosen account was disconnected, or which refused it. let passFailures = new Map(); // We may need to re-prompt the configuration modal when an already-configured binding fails, so @@ -9961,9 +10050,13 @@ class OverseerImpl implements AgentHooks { // misbehaving client (or an account that simply keeps failing). let reprompts = 0; const MAX_CONFIG_REPROMPTS = 1; + // Passes re-run because the scope changed while one awaited (see step 6). + let scopePasses = 0; try { while (true) { + let inScope = scope.gatekeepers; + // 3. Determine uncovered bindings: in-scope gatekeepers with no account choice yet. Ambient // bindings use the collaborator's matching provided account automatically; unlike an // ordinary connection, there is no meaningful account choice when one already exists. @@ -10027,50 +10120,71 @@ class OverseerImpl implements AgentHooks { } } - // 5. Verify all in-scope bindings (covered + newly chosen). For each, resolve the chosen - // account's verifier and hand it to the gatekeeper's addObserver(). Collect *every* + // 5. Verify all in-scope bindings (covered + newly chosen) with the chosen accounts: a + // tracked connection by asking it to check its requirements with the account's + // verifier, a legacy one by handing that verifier to its addObserver(). Collect *every* // failure rather than just the first, so a re-prompt can present them all at once. let failures = new Map(); await Promise.all(inScope.map(async gk => { + // Removed while this open awaited: step 6 re-runs the pass without it. + if (!this.storage.gatekeepers.get(gk.id)) return; let accountId = accountChoices[gk.id]; let vendorId = observerVendorId(gk); if (!vendorId) { throw new Error("An observer account was requested for a non-gatekeeper binding."); } - let fail = (reason: string, err?: unknown) => { - failures.set(gk.id, {accountId, reason}); - invalidated.add(gk.id); - this.logger.warn("observer verification failed", { - event: "gatekeeper.observer.verify.failed", - gatekeeperId: gk.id, vendorId, accountId, observerId, error: err, - }); - }; - try { - let verifier = await clientUser.getVerifier(accountId, vendorId); - if (!verifier) { - // Account gone -> the overseer authors the reason. (Wrong vendor throws above.) - fail("This account is no longer connected."); + if (gk.observerTracking) { + await this.observerLedger.checkAccess( + clientUser, accountId, observerId, gk, scope.requirements.get(gk.id) ?? []); return; } + attempted.add(gk.id); + let verifier = await requireVerifier(clientUser, accountId, vendorId); // Serialized per (observer, gatekeeper) so this registration can't land while an // exclusion teardown's removeObserver for the same pair is still in flight (which - // would delete it moments later); see #withObserverGatekeeperLock. - await this.#withObserverGatekeeperLock(observerId, gk.id, - async () => (await this.getGatekeeperFacet(gk.id)).addObserver(observerId, verifier)); - newlyAdded.add(gk.id); - // Keep `invalidated` meaning "failed and has not verified since": this binding just - // verified on a repaired pass, so the catch below must not roll its registration back. - invalidated.delete(gk.id); + // would delete it moments later); see #serialized. + await this.#serialized(`${observerId}/${gk.id}`, async () => { + // Describe the connection first, and import it if it now advertises observer + // tracking (already tracked, it was imported while this open awaited): step 6 then + // reruns the pass against its tracked requirements. A failed description refuses + // the binding rather than trusting the legacy protocol, and an upgraded + // gatekeeper's addObserver() throws, so a redeploy between the two can't admit + // anyone either. + await this.observerLedger.initialize(gk.id); + if (this.observerLedger.isTracked(gk.id)) return; + await (await this.getGatekeeperFacet(gk.id)).addObserver(observerId, verifier); + }); } catch (err) { // Either a settled denial or an operational failure (expired credentials, upstream // outage). Treat every failure as repairable and let the user try again. - fail(stringifyError(err), err); + failures.set(gk.id, {accountId, reason: stringifyError(err)}); + this.logger.warn("observer verification failed", { + event: "gatekeeper.observer.verify.failed", + gatekeeperId: gk.id, vendorId, accountId, observerId, error: err, + }); } })); + // 6. What this open must verify can change while it awaits: a requirement recorded, a + // connection added, removed, approved, or moved to Workshop tracking. Successes and + // failures alike may be stale, so re-run the pass rather than admit against them. This + // check and the publication below share one synchronous block, so nothing recorded + // after it can be missed: an observation checks every published observer. + let current = + this.observerLedger.observerScope(role, this.#inScopeGatekeepers(role), scope); + if (!observerScopeCovers(scope, current)) { + if (++scopePasses > MAX_VERIFICATION_PASSES) { + throw new Error( + "What this workspace must verify kept changing while you were being verified. " + + "Please try again."); + } + scope = current; + continue; + } + if (failures.size > 0) { // Drop the failed choices so the re-prompt asks about exactly these bindings. A failed // binding's unpersisted account choice self-corrects on the next open (re-verification @@ -10104,14 +10218,13 @@ class OverseerImpl implements AgentHooks { // the unpersisted record, so anything left behind would linger unresolvable. // // A *returning* observer's registrations are all kept -- including the ones this call added. - // Their persisted observerId is shared with concurrent opens, so a rollback here could - // delete a registration that a concurrent successful open just made and persisted. - // De-registering is the fail-open direction: the gatekeeper stops naming that observer in - // `excludeObservers`, so an observation it should have excluded them from is admitted with - // nothing left to block it. A spurious registration merely blocks fail-closed until it is - // lazily cleaned up (see #enforceExcludeObservers) or a later open re-verifies it. + // Their persisted observerId remains resolvable by exclusion checks. De-registering is the + // fail-open direction: the gatekeeper stops naming that observer in `excludeObservers`, so + // an observation it should have excluded them from is admitted with nothing left to block + // it. A spurious registration merely blocks fail-closed until it is lazily cleaned up + // (see #enforceExcludeObservers) or a later open re-verifies it. if (!returningObserver) { - await this.#removeObserverFromGatekeepers(observerId, [...newlyAdded, ...invalidated]); + await this.#removeObserverFromGatekeepers(observerId, [...attempted]); } // Only this open is denied. Sessions the collaborator already holds are left alone: they @@ -10120,8 +10233,8 @@ class OverseerImpl implements AgentHooks { throw err; } - // 6. Persist the observer record only after all addObserver calls succeed. Creating/updating - // the record is the canonical moment the user becomes a configured observer. + // 7. Persist the observer record only after every binding verified. Creating/updating the + // record is the canonical moment the user becomes a configured observer. this.storage.observers.put({profileId, observerId, accountChoices}); } @@ -10790,10 +10903,13 @@ export class OverseerDurableObject extends DurableObject { // Both returned capabilities revalidate the hook per call rather than trusting this moment: // they are held outside this DO (even across resets -- the stored callback is a persistent // stub), so this is what ties them to the firing, per the session contract documented on - // Gatekeeper.bindHook (workshop-shared/gatekeeper.ts). + // Gatekeeper.bindHook (workshop-shared/gatekeeper.ts). The firing's observations are + // attributed to the gadget the hook wakes. + let gadgetId = record.gadgetId ?? this.impl.defaultGadgetId; return { callback: makeHookFiringCallback(this.impl, hookId), - approvalQueue: new ApprovalQueueImpl(this.impl, record.gatekeeperId, {from: "hook"}, hookId), + approvalQueue: new ApprovalQueueImpl(this.impl, record.gatekeeperId, + {from: "hook", ...(gadgetId !== undefined ? {gadgetId} : {})}, hookId), }; } @@ -10831,14 +10947,20 @@ type GatekeeperLoopbackProps = { }; type BindingLoopbackTarget = { - type: "gadget" | "gatekeeper"; + type: "gadget"; id: WorkpieceId; +} | { + type: "gatekeeper"; + id: WorkpieceId; + + // Set on agent loopbacks so tracked observations cannot outlive their executeCode invocation. + executionId?: string; } | { type: "worktree"; id: WorkpieceId; // The executeCodeMode execution this loopback was minted for. A worktree binding resolves - // against the turn state registered for exactly that execution (see #activeWorktreeTurns), so + // against the turn state registered for exactly that execution (see #activeExecutions), so // a stub retained past it -- say, stored in a gadget the agent called -- fails closed instead // of coming back to life against a later execution's turn. executionId: string; @@ -13361,6 +13483,10 @@ class SlashCommandAuthorizerImpl extends NativeRpcTarget implements ObservationA return this.impl.authorizeObservation(this.gatekeeperId, description, this.caller); } + authorizeTrackedObservation(description: TrackedObservationDescription): Promise { + return this.impl.authorizeTrackedObservation(this.gatekeeperId, description, this.caller); + } + async getGitCache(): Promise { return new GitCacheImpl(this.impl.gitCache, this.gatekeeperId); } @@ -13431,14 +13557,23 @@ class ApprovalQueueImpl extends RpcTarget implements ApprovalQueue { // observations against a scope the shrink already excluded someone from (or set // containsRestrictedData). Session queues (openGatekeeperSession) pass no hookId: they are // bounded by the facet's in-DO lifetime, which the session chokepoints already gate. + // `executionId` is the agent execution an agent caller's session was minted for. constructor(protected impl: OverseerImpl, protected gatekeeperId: number, - private caller: GatekeeperCaller, private hookId?: number) { + private caller: GatekeeperCaller, private hookId?: number, + private executionId?: string) { super(); } authorizeObservation(description: ObservationDescription): Promise { if (this.hookId !== undefined) requireLiveHook(this.impl, this.hookId); - return this.impl.authorizeObservation(this.gatekeeperId, description, this.caller); + return this.impl.authorizeObservation( + this.gatekeeperId, description, this.caller, this.hookId); + } + + authorizeTrackedObservation(description: TrackedObservationDescription): Promise { + if (this.hookId !== undefined) requireLiveHook(this.impl, this.hookId); + return this.impl.authorizeTrackedObservation( + this.gatekeeperId, description, this.caller, this.executionId, this.hookId); } async getGitCache(): Promise { diff --git a/packages/workshop-backend/src/storage-schema/overseer-storage.ts b/packages/workshop-backend/src/storage-schema/overseer-storage.ts index 3c04da1be3..1ede2230a5 100644 --- a/packages/workshop-backend/src/storage-schema/overseer-storage.ts +++ b/packages/workshop-backend/src/storage-schema/overseer-storage.ts @@ -62,6 +62,13 @@ export type GatekeeperRecord = { */ bindings?: Record; + /** + * Set once the Workshop tracks this connection's observations itself (see observer-ledger.ts): + * created advertising ResourceDescription.supportsObserverTracking, or its history imported + * when it began advertising it. Never cleared. + */ + observerTracking?: true; + /** * OBSOLETE: Before we had support for multiple gadgets per workspace, the binding name and * blueprint annotation information lived on the GatekeeperRecord. These properties continue @@ -73,6 +80,16 @@ export type GatekeeperRecord = { blueprintAnnotation?: StoredBlueprintBindingAnnotation; }; +/** + * The vendor a connection belongs to, as its creationSpec names it (case preserved, as account + * selection and the User DO's getVerifier() expect it). Undefined for vendorless records + * (aiModel/agentSpawner) and legacy ones with no creationSpec. + */ +export function connectionVendorId(record: GatekeeperRecord | undefined): string | undefined { + let spec = record?.creationSpec; + return spec && "vendorId" in spec ? spec.vendorId : undefined; +} + /** * A stored blueprint annotation. Those written while the UI still offered an exclusion control * may carry `included: false`, which keeps the binding out of blueprints. Nothing writes the @@ -405,7 +422,9 @@ export type GatekeeperCaller = { /** * Which gadget made the call. Optional for backward compatibility: callers embedded in * ActionRecords persisted before multi-gadget support have no gadgetId. `defaultGadgetId` - * should be assumed when `gadgetId` is absent. + * should be assumed when `gadgetId` is absent -- except by observer tracking, which refuses + * such a caller (see observerContextOf): the default gadget can change, and attribution must + * not. */ gadgetId?: WorkpieceId; } | { @@ -413,6 +432,14 @@ export type GatekeeperCaller = { chatId?: number; } | { from: "hook"; + + /** + * The gadget the firing hook wakes: its BoundHookRecord's `gadgetId`, or the `defaultGadgetId` + * that resolved to when it fired. Observer tracking attributes the firing's observations to it + * (see observerContextOf). Absent when neither names a gadget, and on records written before + * firings carried it. + */ + gadgetId?: WorkpieceId; }; export type ActionRecord = { @@ -449,7 +476,17 @@ export type ActionRecord = { autoApproved?: boolean; // set when applied by an auto-approval rule rather than a human } | { type: "observation"; - description: ObservationDescription; + + /** + * `subresources` is kept only on observations authorized through a connection the Workshop + * tracks itself (and on the history it imported when tracking began): the Workshop writes the + * validated list and strips the field from every other observation. Kept, it makes the record an + * observer requirement, attributed by `caller` (see observerContextOf and + * ObserverLedger.requiredSubresources). Nothing else stores requirements, which is why the + * action log is append-only: deleting such a record would silently drop checks while its + * connection still exists. + */ + description: StoredObservationDescription; } | { type: "bindHook"; @@ -469,6 +506,13 @@ export type ActionRecord = { enabled: boolean; }); +/** + * An observation's description as the action log stores it: one authorized through a connection + * the Workshop tracks keeps the validated `TrackedObservationDescription.subresources` (see + * ActionRecord). + */ +export type StoredObservationDescription = ObservationDescription & {subresources?: string[]}; + /** * Key of the actions `byLastChanged` index: last state-change time, id-disambiguated because the * frozen clock makes same-instant records routine. Every mutation path stamps appliedAt (apply, @@ -484,9 +528,10 @@ export type BoundHookRecord = { gatekeeperId: WorkpieceId; /** - * The gadget whose code this hook wakes. Bookkeeping only -- used to display which gadget a - * hook belongs to and to delete a gadget's hooks when the gadget is deleted. Operationally the - * `callback` already encapsulates OverseerRestoreParams pointing at the correct gadget. + * The gadget whose code this hook wakes: shown as the hook's gadget, deleted with it, and + * trusted for its "use" scope and the attribution of its firings (see GatekeeperCaller). + * Operationally the `callback` encapsulates its own OverseerRestoreParams, which an agent-bound + * hook's inferred gadgetId (see OverseerImpl.bindHook) can rarely mismatch; that is accepted. * If omitted, use `defaultGadgetId`. */ gadgetId?: WorkpieceId; @@ -1100,6 +1145,67 @@ export type ObserverRecord = { accountChoices: { [gatekeeperId: number]: number }; }; +/** + * What an observer requirement is attached to: the workspace (requirements with no + * narrower trusted attribution, which every chat and gadget includes), one chat, or one gadget. + */ +export type ObserverContext = + | {type: "workspace"} + | {type: "chat", chatId: number} + | {type: "gadget", gadgetId: WorkpieceId}; + +/** + * The context an observation by `caller` is attributed to: its trusted binding authorizer's chat + * or gadget, the gadget a firing hook wakes, or the workspace for a hook naming no gadget and a + * user acting outside any chat. Undefined for gadget callers persisted without a gadgetId. + * + * The actions `byObserverRequirement` index stores this mapping's result when each record is + * written, so changing an existing attributed mapping requires reindexing its observations keeping + * `subresources`. + */ +export function observerContextOf(caller: GatekeeperCaller): ObserverContext | undefined { + switch (caller.from) { + case "agent": return {type: "chat", chatId: caller.chatId}; + case "user": + return caller.chatId !== undefined ? {type: "chat", chatId: caller.chatId} : {type: "workspace"}; + case "gadget": + return caller.gadgetId !== undefined ? {type: "gadget", gadgetId: caller.gadgetId} : undefined; + case "hook": + return caller.gadgetId !== undefined + ? {type: "gadget", gadgetId: caller.gadgetId} : {type: "workspace"}; + } +} + +// Leads the part of an observer requirement key naming its context's type. +const OBSERVER_CONTEXT_TAGS = {workspace: "w", chat: "c", gadget: "g"} as const; + +/** + * `listKeys()` prefix selecting the actions `byObserverRequirement` keys through `gatekeeperId` on + * every context of type `contexts`. + */ +export function observerRequirementPrefix( + gatekeeperId: WorkpieceId, contexts: ObserverContext["type"]): string { + return `${keyString(gatekeeperId)}:${OBSERVER_CONTEXT_TAGS[contexts]}`; +} + +/** + * Key of the actions `byObserverRequirement` index: one per subresource of an observer + * requirement, led by its connection and context (see observerRequirementPrefix()) and ending + * with the subresource itself (see observerRequirementSubresource()). The connection and context + * parts are colon-free and colon-terminated, so they never run into the subresource. + */ +export function observerRequirementKey( + gatekeeperId: WorkpieceId, context: ObserverContext, subresource: string): string { + let id = context.type === "chat" ? keyString(context.chatId) + : context.type === "gadget" ? keyString(context.gadgetId) : ""; + return `${observerRequirementPrefix(gatekeeperId, context.type)}${id}:${subresource}`; +} + +/** The subresource an observerRequirementKey() names: everything after its second colon. */ +export function observerRequirementSubresource(key: string): string { + return key.slice(key.indexOf(":", key.indexOf(":") + 1) + 1); +} + // ======================================================================================= // Blueprints @@ -1353,7 +1459,9 @@ export function makeOverseerStorage(storage: DurableObjectStorage) { actions: collection()({ primaryKey: "id", - // All three indexes are backfilled by the version-3 migration. + // The first three indexes are backfilled by the version-3 migration; + // byObserverRequirement needs none: a Workshop without it had no + // authorizeTrackedObservation(), so none of its observations is a requirement. uniqueIndexes: { // Resume-replay index (see subscribeToActions): keyed by last state-change time so a // reconnect replays only the records changed during the gap. @@ -1373,6 +1481,23 @@ export function makeOverseerStorage(storage: DurableObjectStorage) { byHistoryFilter(record: ActionRecord) { return record.state === "pending" ? ["pending", record.type] : record.type; }, + + // Sparse index over the observer requirements: one key per subresource of each + // observation through a real connection that kept `subresources` (see ActionRecord), + // led by the connection and the context its caller attributes it to. Admission lists + // only the distinct keys (see ObserverLedger.requiredSubresources), never the records + // under them, so its cost follows distinct requirements, not repeated reads. + byObserverRequirement(record: ActionRecord): string[] { + let subresources = record.type === "observation" && record.gatekeeperId >= 0 + ? record.description.subresources : undefined; + let context = subresources ? observerContextOf(record.caller) : undefined; + if (!subresources || !context) return []; + let keys = new Set(); + for (let id of subresources) { + keys.add(observerRequirementKey(record.gatekeeperId, context, id)); + } + return [...keys]; + }, } }), diff --git a/packages/workshop-backend/src/user.ts b/packages/workshop-backend/src/user.ts index 110fd92c20..c94cc8c78b 100644 --- a/packages/workshop-backend/src/user.ts +++ b/packages/workshop-backend/src/user.ts @@ -2217,11 +2217,12 @@ export class UserDurableObject extends DurableObject { /** * Mint a verifier from one of THIS user's connected accounts, identified by accountId. The - * overseer passes the returned verifier to a gatekeeper's `addObserver()` so the gatekeeper can - * check whether this user is allowed to observe the data read through it, or to its - * `applyCreation()` to create the resource in this account. Returns null if the - * account no longer exists (or never existed). Throws if the account belongs to a different - * vendor (not a legitimate UI state — only reachable by bypassing client-side filtering). + * overseer passes the returned verifier to a gatekeeper's `addObserver()` or + * `verifyObserverAccess()` so the gatekeeper can check whether this user is allowed to observe + * the data read through it, or to a gatekeeper's `applyCreation()` to create the resource in + * this account. Returns null if the account no longer exists (or never existed). Throws if the + * account belongs to a different vendor (not a legitimate UI state — only reachable by bypassing + * client-side filtering). * * Account *selection* (which of the user's accounts to use for a given binding) is done by the * frontend; this method validates and resolves a chosen account to its verifier. diff --git a/packages/workshop-shared/src/gatekeeper.ts b/packages/workshop-shared/src/gatekeeper.ts index 7aa224f98e..0d44de1db6 100644 --- a/packages/workshop-shared/src/gatekeeper.ts +++ b/packages/workshop-shared/src/gatekeeper.ts @@ -320,6 +320,25 @@ export type ResourceDescription = { /** Indicates that getSlashCommandProvider() is available. */ hasSlashCommands?: true; + /** + * Opts this resource into kernel-owned observer tracking. When set, the gatekeeper promises: + * - every observation goes through `ObservationAuthorizer.authorizeTrackedObservation()`, never + * `authorizeObservation()`; + * - `Gatekeeper.getHistoricalSubresources()` is implemented, so the Workshop can import the + * requirements recorded before it opted in; + * - `Gatekeeper.verifyObserverAccess()` is implemented, so the Workshop can check collaborators' + * accounts against those requirements; + * - `Gatekeeper.addObserver()` throws, so a Workshop still treating the connection as legacy + * refuses the registration rather than admitting anyone. + * + * `addObserver()` is then never called for this resource, and `removeObserver()` only to retire + * its legacy registrations. Absence keeps the legacy protocol. A resource advertising support + * when it is connected is tracked from the start; one that starts advertising it later has its + * history imported once (see `Gatekeeper.getHistoricalSubresources()`). Tracking is never + * downgraded. + */ + supportsObserverTracking?: true; + /** * Some resources implement the ability for the client to subscribe to events. The application * implements a "hook", which is a WorkerEntrypoint that implements the TypeScript interface @@ -698,6 +717,8 @@ export interface GatekeeperVendor extends WorkerEntrypoint { * account to reach the provider through, though it may still be asked to describe() itself. * Creates nothing. The Overseer queues `action` as the creation, which applies before anything * else queued on the gatekeeper, through Gatekeeper.applyCreation() once a user approves it. + * While the creation is pending, reads are audited but record no observer requirements (see + * Gatekeeper.getHistoricalSubresources()). * Present only on vendors with a `creatable` resource type. */ createResource?(resourceUrlPattern: string, title: string): Promise<{ @@ -942,10 +963,11 @@ export interface GatekeeperUser extends WorkerEntrypoint { /** * Opaque object representing one of a user's accounts. Minted by `GatekeeperUser`, and then passed - * to `Gatekeeper.addObserver()`, to verify whether the user is able to access a particular - * Gatekeeper, and to `Gatekeeper.applyCreation()`, to create a resource with the account's authority. - * Also returned from `GatekeeperUserSelectionExchange.redeem()` for the selected user's eligible - * accounts with the same vendor. + * to `Gatekeeper.addObserver()` or `Gatekeeper.verifyObserverAccess()`, to verify whether the user + * is able to access a particular Gatekeeper, and to `Gatekeeper.applyCreation()`, to create a + * resource with the account's authority. Also returned from + * `GatekeeperUserSelectionExchange.redeem()` for the selected user's eligible accounts with the + * same vendor. * * At present, this interface has no methods, because it is merely meant to be passed back to the * Gatekeeper that created it. @@ -953,10 +975,10 @@ export interface GatekeeperUser extends WorkerEntrypoint { * IMPLEMENTATION NOTE: As of this writing, there is no runtime-supported way to "unwrap" a * `Fetcher` passed back to its implementer in order to extract the underlying `props`. This will * be added eventually. For now, we recommend that the `GatekeeperUserVerifier` implement a public - * but non-standard method which the same gatekeeper's `addObserver()` and `applyCreation()` - * implementations can call. The overseer promises only to pass a `GatekeeperUserVerifier` object - * back to the same gatekeeper that created it, so they can then call that non-standard method and - * trust the results. + * but non-standard method which the same gatekeeper's `addObserver()`, `verifyObserverAccess()` + * and `applyCreation()` implementations can call. The overseer promises only to pass a + * `GatekeeperUserVerifier` object back to the same gatekeeper that created it, so they can then + * call that non-standard method and trust the results. */ export interface GatekeeperUserVerifier extends WorkerEntrypoint {} @@ -1075,6 +1097,10 @@ export interface Gatekeeper extends DurableObject { * access. However, if the Gadget itself carefully reads only emails addressed to a particular * mailing list, then it is OK to reveal those observations to any member of the mailing list. * The gatekeeper should ideally permit observers who are mailing list members. + * + * A resource advertising `ResourceDescription.supportsObserverTracking` must throw instead: a + * redeploy can land between the Workshop's describe() of a legacy connection and this call, and + * throwing keeps it from admitting a collaborator without checking anything. */ addObserver(id: string, user: Fetcher): Promise; @@ -1088,6 +1114,56 @@ export interface Gatekeeper extends DurableObject { */ removeObserver(id: string): Promise; + /** + * For a gatekeeper advertising `ResourceDescription.supportsObserverTracking`: a cursor over + * every subresource ID this gatekeeper's past observations, hook deliveries included, still + * oblige observers to be able to read -- completed and pending, plus any binding-wide or + * owner-only requirement it records itself -- in the same form as + * `TrackedObservationDescription.subresources`, which `verifyObserverAccess()` can check. Not an + * audit log or listing of today's resources. + * IDs obey the same rules as there, and each page may hold at most 10,000 of them, which keeps + * it within the RPC size limit. The complete export has no ID count cap, and may repeat an ID. + * + * The Workshop calls this once per connection, when an existing one first shows that it + * advertises support: at its first `ObservationAuthorizer.authorizeTrackedObservation()`, or + * when a collaborator is admitted (the Workshop describes a legacy connection before calling its + * `addObserver()`), including after a pending creation is approved. It drains the cursor until + * `next()` returns `null`, and stores the result in the same commit that marks tracking + * initialized. It must therefore be read-only and repeatable: an import interrupted at any page + * calls it again and drains a new cursor from the start. A cursor that ends at once is valid. + * Reads made while a creation was pending were simulated and must not be exported. A gatekeeper + * connected while already advertising support is never asked. + */ + getHistoricalSubresources?(): Promise>; + + /** + * For a gatekeeper advertising `ResourceDescription.supportsObserverTracking`: which of + * `observers` can read every one of `subresources` now. The Workshop calls this to check + * collaborators against the subresource IDs this gatekeeper reported in + * `TrackedObservationDescription.subresources` or exported from `getHistoricalSubresources()`: + * before authorizing a read, for every collaborator who can see the context that made it, and + * at admission, for everything the collaborator's role reaches. The answer must not depend on + * which of the two is asking, and the Workshop does not say. + * + * Each observer's `id` is the opaque observer ID `addObserver()` and `excludeObservers` use, + * unique within one call. It is not a cache key: the same observer may arrive later with a + * different account's verifier. Each `verifier` was minted by this gatekeeper's vendor (see + * `GatekeeperUserVerifier`) and is meant for this call only. A call carries at most 10,000 + * subresource IDs, never none; the Workshop splits larger checks. + * + * `allowedObserverIds` lists the IDs of the observers whose account can read every one of + * `subresources`; every observer it omits is denied. Throw if access cannot be determined + * (expired credentials, provider outage). A throw, a malformed result, or an ID that was not + * passed in denies every observer in the call. + * + * The Workshop may call this while one of this gatekeeper's own `authorizeObservation()` calls + * is pending, so it must not wait on anything a read holds while it awaits authorization. + */ + verifyObserverAccess?( + observers: {id: string, verifier: Fetcher}[], + subresources: string[], + ): Promise<{allowedObserverIds: string[]}>; + /** Returns the provider for describe().hasSlashCommands, if supported. */ getSlashCommandProvider?(): Promise; @@ -1224,9 +1300,28 @@ export interface ObservationAuthorizer extends RpcTarget { * remote service, so that the description can include details about the actual data. As long * as the operation is strictly read-only, and the call is made before actually returning any * data to the gadget, this is OK. + * + * A resource advertising `ResourceDescription.supportsObserverTracking` uses + * `authorizeTrackedObservation()` instead. Once its connection is tracked, this refuses every + * observation through it; and since only that method enforces `subresources`, this refuses a + * description carrying them rather than ignoring them. */ authorizeObservation(description: ObservationDescription): Promise; + /** + * `authorizeObservation()` for a resource advertising + * `ResourceDescription.supportsObserverTracking`, which must make every observation through + * this method. The first call through a connection made under the legacy protocol imports its + * history (see `Gatekeeper.getHistoricalSubresources()`) and is refused; a retry, through a new + * session, is checked. + * + * Returning normally means the Workshop recorded `description.subresources` and confirmed that + * every collaborator who can see the read may access them. A Workshop that predates observer + * tracking lacks this method, so the call fails. Let that failure propagate: falling back to + * `authorizeObservation()` would let the read through without checking anyone. + */ + authorizeTrackedObservation(description: TrackedObservationDescription): Promise; + /** * Get the workspace's git cache, scoped to this gatekeeper (see `GitCache` for the view rules). * @@ -1497,6 +1592,29 @@ export type ObservationDescription = { excludeObservers?: string[]; } +/** + * An observation through a resource advertising `ResourceDescription.supportsObserverTracking`, + * as passed to `ObservationAuthorizer.authorizeTrackedObservation()`. The Workshop checks + * collaborators itself, so it names no observers to exclude. + */ +export type TrackedObservationDescription = Omit & { + /** + * The IDs of everything this disclosure depends on that might restrict who may see it. IDs must + * be stable, non-secret, and sufficient for this gatekeeper's `verifyObserverAccess()` to check + * access; the Workshop treats them as opaque. Each ID must be well-formed Unicode (no lone + * surrogates) with a JavaScript `length` of 1–512 (UTF-16 code units), and a single observation + * may report at most 10,000 IDs before deduplication. `[]` means nothing restricts it. Include + * any coarse binding-wide requirement the read relies on. + * + * The Workshop records these IDs on the chat or gadget whose trusted binding authorizer + * approved the read (a hook firing's on the gadget the hook wakes, or the workspace for a user + * session with no known chat). + * Attribution stays with the original authorizer, including child sessions that reuse it; + * passing capabilities or data between contexts does not transfer requirements. + */ + subresources: string[]; +}; + /** The language a `text` action field is written in, named so the approver knows how it is read. */ export type ActionFieldSyntax = "markdown" | "html" | "sql"; diff --git a/plans/kernel-observer-tracking.md b/plans/kernel-observer-tracking.md new file mode 100644 index 0000000000..b7a97c96c7 --- /dev/null +++ b/plans/kernel-observer-tracking.md @@ -0,0 +1,111 @@ +# Design record: kernel-owned observer tracking + +How it works today is in `docs/observers.md` §10 and `packages/workshop-backend/src/observer-ledger.ts`. +This file records why it is shaped that way and what was deliberately left out. + +## Scope + +Kernel-only: `workshop-backend`, the shared contracts in `workshop-shared/src/gatekeeper.ts`, and the +integration test fixture. No shipping gatekeeper opts in yet; every shipping gatekeeper keeps the +legacy `addObserver`/`removeObserver`/`excludeObservers` protocol unchanged, and a workspace may +mix both. + +## Decisions + +- **Opt-in by capability, not method probing.** `ResourceDescription.supportsObserverTracking` + promises `authorizeTrackedObservation()` for every observation, `getHistoricalSubresources()`, + `verifyObserverAccess()`, and an `addObserver()` that throws. RPC stubs can't report method + presence, so the flag is read explicitly. Once a connection is tracked, it is never downgraded. +- **Tracked reads use their own authorization method.** `subresources` on `authorizeObservation()` + would be enough for a current Workshop to notice, but a gatekeeper cannot tell from a successful + call whether the Workshop enforced them: RPC validation forwards undeclared properties, so an + older Workshop accepts the call and ignores them. A migrated gatekeeper has dropped its own + checks, so that read would be disclosed unchecked, and its requirement would never reach the + import, which exports only legacy history. `authorizeTrackedObservation()` does not exist on + an older Workshop, so the call fails closed whatever the deployment order or rollback, and the + gatekeeper needs no acknowledgment to check. The cost: implementers of `ObservationAuthorizer` + (test doubles included) add the method. +- **Upgrades are detected from calls already made, plus one description at admission.** Many + gatekeepers' `describe()` calls their provider, and every binding call opens a session, so + describing on every use would add provider traffic, latency and a failure mode to every call. + Only an upgraded gatekeeper calls `authorizeTrackedObservation()`, so a legacy connection's + first such call has it described and imported. Admission describes each legacy connection in + scope before its `addObserver()` (inside the registration lock, after the account prompt), + imports one that now advertises support, and refuses the binding if it can't be described. + That keeps upgrade detection out of `addObserver()`'s error handling, for one `describe()` per + legacy connection per admission. `addObserver()` still throws for an upgraded gatekeeper, so a + redeploy between the description and the call refuses rather than admits. +- **An import restarts the workspace, even with only the owner connected.** The import aborts the + connection's facet, and client capabilities (a `GatekeeperClient`) keep the stub they were + minted with. Restarting replaces them once, at the migration, instead of resolving the facet + again on every session open forever after. It restarts after a failed import too, since the + abort has already happened. The cost: everyone connected, the owner included, reconnects once + per import attempt, and the call that revealed the upgrade fails. +- **The kernel decides who is checked and when; the connection decides access.** The Workshop + computes each check's observers and IDs and calls `Gatekeeper.verifyObserverAccess(observers, + subresources)` on the connection that reported them, handing it each observer's verifier. + `GatekeeperUserVerifier` stays opaque, as for `addObserver()` and `applyCreation()`, and the + gatekeeper may use its own props, state, and credentials (one ACL read checked against the whole + audience, say). It is a bulk call because a read's decision covers its whole audience, and + returns an allow-list (`allowedObserverIds`) so that an observer the answer omits is denied + rather than silently admitted. IDs need only mean something to the connection that reported + them, since requirements are indexed per connection; the kernel never parses them. The costs: + checks end with the connection (as they already do, see "Disconnect drops checks"), and the call + can arrive while one of the gatekeeper's own `authorizeObservation()` calls is pending, so the + gatekeeper must not wait on anything a read holds. +- **The audit log is the requirement store.** Each tracked observation's audit record already + holds its IDs and its caller, which attributes it; keeping `subresources` (stripped from every + other observation) marks it as a requirement. One representation, written in the same + synchronous block as the audit/policy commit, so a refused read records nothing and no + admission can be published between a check and its write. A multi-key index derived from those + records holds one key per (connection, context, subresource), and admission lists only its + distinct keys, never the records under them: rereading the same data makes the log grow, not + admission. The index is maintained in the same transaction as the record, so it needs no + separate projection to keep consistent. The costs: index entries grow with every recorded + subresource occurrence; audit retention is part of enforcement, so the action log must stay + append-only; IDs become storage keys, so they must be well-formed Unicode (storage would merge + lone surrogates); and the index stores the caller-to-context mapping's result per record, so + changing that mapping needs a reindexing migration. +- **Always import on upgrade.** An existing legacy connection that starts advertising support is + imported once (fenced, single-flight, repeatable until it commits). No kernel-side "did it read + anything" flag: the gatekeeper's export is authoritative, an empty export is cheap, and a flag + would fail open for requirements recorded outside `authorizeObservation`. A connection created + already advertising support starts tracked. Imported IDs land on the workspace context, since no + narrower attribution exists. +- **The export is a cursor.** History has no count cap, and one RPC result is limited to 32 MiB, + so `getHistoricalSubresources()` returns the existing `Cursor` with pages capped at an + observation's 10,000 IDs. The Workshop drains it to `null` and commits the union in one + transaction, keeping import all-or-nothing; the union is held in memory until then. +- **Disconnect drops checks.** A removed connection's audit records stay but are ignored, matching + the legacy protocol's behavior on removal (`docs/observers.md` edge case 8). +- **Same verification scope as the legacy protocol.** A tracked connection enters "build" scope at + creation and "use" scope only while a gadget binds it or an enabled hook feeds it, through the + existing scope-widening restarts. Within scope, "build" reaches every context and "use" the + workspace and every gadget. Requirements through a connection outside a "use" collaborator's + scope are not checked against them: the legacy "never-bound producers" gap, kept rather than + closed so a connection only chats read asks "use" collaborators for no account. +- **Direct contexts only; no propagation.** Attribution follows the trusted binding authorizer that + approved the read. Calls, callbacks, spawned chats, code publication, arguments, and persistent + writes transfer nothing between contexts. Indirect disclosure is knowingly outside the guarantee. +- **Hook observations use the gadget the hook wakes.** The audit caller is + `{from: "hook", gadgetId}`, from the hook record's `gadgetId` (or `defaultGadgetId` when it has + none) at firing time, and the workspace when neither names a gadget. That `gadgetId` is + inferred for an agent-bound hook and the callback carries its own restore target, so the two can + rarely disagree; it is trusted anyway, as it already is for the hook's "use" scope, rather than + writing every hook's data into the workspace context that no future per-gadget verification + could narrow. Disabling/deleting a hook stops future events without erasing requirements for + data already delivered, even after a restart; "use" collaborators stay verified against them + only while something still binds or hooks the connection. +- **Races between reads and admissions are closed by re-checking at the commit point, not by a + shared lock.** Only one profile's admissions are serialized with each other; after verifying, + admission recomputes its scope and re-runs if it grew. + It rereads requirements there only if its connections changed or `nextActionId` moved, since + only a new action record can add one, so an admission nothing disturbed reads them once. + An observation recomputes its audience after its checks and re-checks if it changed. A hook + firing also revalidates its live record and usable connection synchronously before committing + an observation, so a disable/delete during verification prevents delivery and policy/log writes. + +## Deferred + +- Message-level attribution and historical cutoffs (the log's ordered ids are a starting point). +- Migrating shipping gatekeepers and their trackers. diff --git a/plans/restricted-data-sharing.md b/plans/restricted-data-sharing.md index 68a023c776..d499297dfe 100644 --- a/plans/restricted-data-sharing.md +++ b/plans/restricted-data-sharing.md @@ -214,8 +214,6 @@ stays on this ledger only if its failure mode is *persistent* wrong state that o the window. Each item is marked in the code by a matching `TODO` comment; this ledger is the follow-up worklist. -- Observer verification is not serialized per profile, so concurrent opens by one - collaborator can overwrite each other's records and registrations. - A mid-registration observer named in `excludeObservers` is read as unknown and the observation admitted: a first-time `ensureObserver` registers the id with the gatekeepers before the record is persisted, so `#enforceExcludeObservers` cannot map it From 6fd2817b557e9405eb67f1773a4ed8f88dd4717d Mon Sep 17 00:00:00 2001 From: Max Peterson Date: Tue, 27 Oct 2026 23:05:46 -0700 Subject: [PATCH 2/3] make migration more graceful --- .agents/skills/write-gatekeeper/SKELETON.md | 4 +- .agents/skills/write-gatekeeper/SKILL.md | 6 +- docs/observers.md | 56 +-- .../__tests__/observer-tracking.test.ts | 37 +- .../__tests__/observer-ledger.test.ts | 420 ++++++++++++------ .../workshop-backend/src/observer-ledger.ts | 182 ++++---- packages/workshop-backend/src/overseer.ts | 79 ++-- .../src/storage-schema/overseer-storage.ts | 10 +- packages/workshop-shared/src/gatekeeper.ts | 36 +- plans/kernel-observer-tracking.md | 41 +- 10 files changed, 507 insertions(+), 364 deletions(-) diff --git a/.agents/skills/write-gatekeeper/SKELETON.md b/.agents/skills/write-gatekeeper/SKELETON.md index 70008765f4..82a7774b59 100644 --- a/.agents/skills/write-gatekeeper/SKELETON.md +++ b/.agents/skills/write-gatekeeper/SKELETON.md @@ -410,8 +410,8 @@ export class MyGatekeeperImpl extends DurableObject // - Strategy D (low-stakes): make both methods no-ops. // - Strategy C (data-set tracking): record observed sets + store verifiers, and route every read // through an authorizeSetObservation helper that sets `excludeObservers` (see SKILL.md). - // - Workshop-tracked (`supportsObserverTracking` in describe()): always throw (a redeploy can - // land between the Workshop's describe() and this call); authorize every read with + // - Workshop-tracked (`supportsObserverTracking` in describe()): always throw (the refusal is + // how the Workshop notices the upgrade at admission); authorize every read with // authorizeTrackedObservation(), reporting its `subresources`, and implement // verifyObserverAccess(observers, ids) instead, checking each observer's verifier the same way. async addObserver(_id: string, user: Fetcher): Promise { diff --git a/.agents/skills/write-gatekeeper/SKILL.md b/.agents/skills/write-gatekeeper/SKILL.md index d6efa10635..428fd7ee16 100644 --- a/.agents/skills/write-gatekeeper/SKILL.md +++ b/.agents/skills/write-gatekeeper/SKILL.md @@ -284,13 +284,13 @@ Opt in by returning `supportsObserverTracking: true` from the resource's `descri 1. **Authorize every observation with `authorizeTrackedObservation()`**, hook deliveries included. Report as `subresources` the IDs of everything the disclosure depends on that might restrict who may see it (`[]` when nothing does), including the coarse requirement the read relies on (see the mapping above): nothing else checks it, since `addObserver()` no longer verifies anything. Make each ID carry everything your `verifyObserverAccess()` needs, and keep its meaning stable across versions; encoding a tuple avoids delimiter collisions (the integration fixture reports `JSON.stringify([resourceUrl, localId])`). If the call fails because the Workshop predates observer tracking, let it propagate: falling back to `authorizeObservation()` would disclose the read without checking anyone. 2. **Implement `verifyObserverAccess(observers, subresources)`.** Check each observer's verifier through your vendor's non-standard methods (as `addObserver()` does) and return the IDs of those whose account can read **every** ID; throw when access can't be determined. Check in bulk where the service allows; because one call covers a read's audience, the check may also read the resource's ACL once with the connection's own credentials and compare each observer's identity. Don't store the verifiers or cache verdicts by observer ID, since the same observer can come back with another account. -3. **Implement `getHistoricalSubresources()`** over the requirements your legacy tracking recorded, hook deliveries included, in the same form as `subresources`. A gatekeeper with no legacy history may return a cursor that ends at once. -4. **Make `addObserver()` throw** (any error); `removeObserver()` may stay a no-op. +3. **Implement `getHistoricalSubresources()`** over the requirements your legacy tracking recorded, hook deliveries included, in the same form as `subresources`. Cover every read the legacy protocol authorized, including one still in flight when the new version deployed: the Workshop accepts tracked reads while it imports, so it checks nothing else from before the switch. A gatekeeper with no legacy history may return a cursor that ends at once. +4. **Make `addObserver()` throw** (any error): that refusal is how a Workshop still treating the connection as legacy learns of the upgrade during an admission. `removeObserver()` may stay a no-op. What changes for the gatekeeper: - **No observer state.** There are no observed sets, stored verifiers, or `excludeObservers` to keep. The Workshop attributes each read, checks its audience, and closes the races between observations and opens. -- **The switch is all at once and permanent.** Once a connection is tracked, the Workshop refuses every `authorizeObservation()` through it, whichever session made it, so every session, sub-session, cursor, and hook path must use `authorizeTrackedObservation()` before the flag ships; a read that uses `authorizeObservation()` before the Workshop has noticed the upgrade is treated as legacy. Tracking is never downgraded. +- **The switch is all at once and permanent.** The Workshop switches a connection when one of its reads calls `authorizeTrackedObservation()` or its `addObserver()` refuses a collaborator, then imports its history in the background without interrupting it. From then on the Workshop refuses every `authorizeObservation()` through it, whichever session made it, so every session, sub-session, cursor, and hook path must use `authorizeTrackedObservation()` before the flag ships; a read that uses `authorizeObservation()` before the Workshop has noticed the upgrade is treated as legacy. Tracking is never downgraded. - **Registrations made before the switch keep working.** Hook controllers and other persistent stubs stored by an earlier version keep the `props` they were minted with, and their hooks keep firing after the upgrade. Choose the protocol from the binding's current state, never from props or stored fields an earlier version did not write. `docs/observers.md` §10 describes the Workshop's side in full. No shipping gatekeeper opts in yet; the integration fixture's tracked mode (`packages/integration-tests/fixtures/gatekeeper-test/src/test-gatekeeper.ts`) is a minimal working example. diff --git a/docs/observers.md b/docs/observers.md index e3b87ca3af..c8da98fa80 100644 --- a/docs/observers.md +++ b/docs/observers.md @@ -952,37 +952,41 @@ documented on those members in `workshop-shared/src/gatekeeper.ts`, and it reaches after its checks and checks whoever it gained (at most three times). Both decisions are confirmed in the same synchronous block that publishes them (the observer record; the audit record), so neither can miss the other. -- **Historical import.** The Workshop does not describe a legacy connection on every use. It looks - for an upgrade on two occasions: +- **Switch and historical import.** The Workshop does not describe a legacy connection on every + use. It looks for an upgrade only when the gatekeeper itself shows one: - one of the connection's reads calls `authorizeTrackedObservation()`, which only an upgraded - gatekeeper does. The read came through a session minted under the legacy protocol, so it is - refused after the import (or, if the description doesn't advertise support, refused - outright); - - a collaborator is admitted. Before calling a legacy connection's `addObserver()`, inside the - registration lock and after account selection and verifier lookup (so an upgrade during the - prompt is caught too), admission describes it and imports it if it advertises support, then - reruns against the tracked requirements before publishing its record. A failed description - refuses the binding like any other verification failure, rather than falling back to - `addObserver()`; an upgraded gatekeeper's `addObserver()` throws, so a redeploy between the - description and the call refuses the collaborator too. - - The import is single-flight and fenced (the facet restarts, and the connection is unusable - until the import settles). It drains `getHistoricalSubresources()` to its end, then one - transaction sets `observerTracking` and, for a non-empty export, writes the workspace audit - chunks. A failed description or export commits nothing, and the next tracked read or admission - tries again; after the commit, the export is never asked for again. Once the facet has been - aborted, the workspace restarts whether or not the import commits, even with only the owner - connected: client capabilities such as a `GatekeeperClient` hold stubs to the aborted facet, and - live sessions must reopen against the imported requirements. The connection stays held back - until the restart lands, and the read or call that revealed the upgrade fails rather than being - replayed. Legacy registrations are retired best-effort, racing the restart. A connection created - while advertising support starts tracked without an export. Tracking is never downgraded. + gatekeeper does. If the description advertises support, the connection switches and the read + is checked like any tracked read; otherwise, or if it can't be described, the read is refused; + - its `addObserver()` refuses a collaborator, as an upgraded gatekeeper's does. Admission then + describes the connection, inside the registration lock; if it advertises support, the + connection switches and admission reruns against its tracked requirements. Otherwise, or if + it can't be described, the refusal stands. A collaborator the legacy protocol accepts never + depends on the description, which may need the owner's credentials. + + A switch sets `observerTracking` and `observerHistoryPending` together: from then on every + observation through the connection is checked and recorded by the Workshop, and the legacy + `authorizeObservation()` is refused, so no read goes unchecked. Nothing is aborted or + restarted. The switch starts the history import in the background: single-flight, it drains + `getHistoricalSubresources()` to its end, then one transaction clears `observerHistoryPending` + and, for a non-empty export, writes the workspace audit chunks. Reads through the connection + carry on meanwhile, recording their own requirements beside the imported ones. Admission waits + for the import before verifying anyone against the connection, and treats its completion as a + protocol change, so it reruns against the imported requirements. A failed export commits + nothing and leaves the history pending: the binding fails like any other verification failure, + and the next admission exports again; after the commit, the export is never asked for again. + Legacy registrations are then retired best-effort. A connection created while advertising + support starts tracked without an export. Tracking is never downgraded. + + Sessions admitted under the legacy protocol stay open across the switch: the legacy protocol + verified them against everything read before it, and every tracked read checks them after it. + Their next admission checks the imported history too. The export must cover every read the + legacy protocol authorized, including one still in flight when the gatekeeper upgraded. The residual: an upgraded gatekeeper that wrongly calls `authorizeObservation()` for a read, before any tracked read of its own or any admission, has that read treated as legacy. -- **Pending creations.** A simulated resource from `createExternalResource` is not imported +- **Pending creations.** A simulated resource from `createExternalResource` does not switch while its creation is pending; its reads are audited and enforce workspace policy but record no - requirements. After approval it is imported like any other connection. + requirements. After approval it switches like any other connection. - **Removal.** A disconnected connection's audit records stay, but admission ignores them: its checks go with it, even when data it read survives (edge case 8). - **Direct contexts only.** A chat binding read records requirements only on that chat, checked diff --git a/packages/integration-tests/__tests__/observer-tracking.test.ts b/packages/integration-tests/__tests__/observer-tracking.test.ts index 5fcf1b94cd..f613e1b6dc 100644 --- a/packages/integration-tests/__tests__/observer-tracking.test.ts +++ b/packages/integration-tests/__tests__/observer-tracking.test.ts @@ -12,7 +12,7 @@ import type { import { diffFiles } from "@gadgets/workshop-shared/code-change"; import type { TestSession } from "../fixtures/gatekeeper-test/src/test-gatekeeper.js"; import { - settleRestart, startTestGatekeeperHarness, TEST_VENDOR_ID, testControl, type Harness, + startTestGatekeeperHarness, TEST_VENDOR_ID, testControl, type Harness, } from "../src/harness.js"; import { NetworkInterceptor } from "../src/network-interceptor.js"; import { @@ -225,18 +225,16 @@ it.concurrent("a legacy connection's history is imported at its first tracked ob using bob = await addCollaborator(ws); // The gatekeeper upgrades. Its first tracked observation, through a session minted under the - // legacy protocol, has the Workshop import the connection's history; the import aborts the - // facet that read is running in. It also restarts the workspace, owner's session included: let - // that land before Bob opens. + // legacy protocol, switches the connection to tracking and is itself authorized; the history + // import runs in the background without disturbing the owner's session. await enableObserverTracking(url); - await expect(session.readValue(false, false, undefined, ["new-1"])) - .rejects.toThrow(/restarted to move its observer tracking/); - expect(await historyExportCount(url)).toBe(1); - await settleRestart(); + await expect(session.readValue(false, false, undefined, ["new-1"])).resolves.toBe(42); + await waitFor("the history export", async () => await historyExportCount(url) === 1 || null); + await expect(session.readValue(false, false, undefined, ["new-2"])).resolves.toBe(42); await setSubresourceAccess(bob.label, url, "hist-2", false); const refused = await collaboratorOpens(bob, ws.gadgetId); - // The refusal can only come from the imported ID: no tracked read has succeeded. + // Only the imported ID is refused; the tracked reads' own IDs remain accessible. expect(refused.error?.message).toContain(DENIED); // Migration may retire legacy registrations, but admission must not register new ones. expect((await observerEvents(url)).filter(event => event.type === "add")).toEqual([]); @@ -282,7 +280,7 @@ it.concurrent("imports a protocol upgrade that arrives while account selection i expect((await opening)?.message).toContain(DENIED); expect(await historyExportCount(url)).toBe(1); - await settleRestart(); + // The upgraded addObserver() refusal had admission switch and import the connection. await setSubresourceAccess(bob.label, url, "historical-secret", true); expect((await collaboratorOpens(bob, ws.gadgetId)).error).toBeUndefined(); }); @@ -513,9 +511,8 @@ it.concurrent("tracked hooks deliver authorized data, check both collaborator ro expect(await observerEvents(url)).toEqual([]); }); -// Not concurrent: with other tests running, workerd 1.20261001 dies with "Promise callback -// destroyed itself" when a workspace restarts after a hook delivery, whether this import restarts -// it or removing a collaborator does. +// Not concurrent: with other tests running, workerd 1.20261001 has died with "Promise callback +// destroyed itself" around hook deliveries in a workspace that restarts; keep this isolated. it("a hook enabled under the legacy protocol keeps firing after the upgrade imports its deliveries", async () => { using publicApi = connect(harness.url); @@ -540,15 +537,11 @@ it("a hook enabled under the legacy protocol keeps firing after the upgrade impo // Under the legacy protocol only the gatekeeper remembers what a delivery depended on. expect(await fire(71, ["hook-secret"])).toEqual({ fired: true }); - // The gatekeeper upgrades. The hook's first tracked delivery has the Workshop import the - // connection's history and is refused; the restart the import starts must land before a retry. + // The gatekeeper upgrades. The hook's first tracked delivery switches the connection and is + // delivered at once; the registration predates the upgrade. await enableObserverTracking(url); - expect(await fire(72, ["live-1"])).toEqual({ - error: expect.stringContaining("now supports the Workshop's observer tracking"), - }); - expect(await historyExportCount(url)).toBe(1); - await settleRestart(); - // The hook's registration predates the upgrade, and it delivers under the tracked protocol. + expect(await fire(72, ["live-1"])).toEqual({ fired: true }); + await waitFor("the history export", async () => await historyExportCount(url) === 1 || null); expect(await fire(73, ["live-1"])).toEqual({ fired: true }); // Only the legacy delivery depended on "hook-secret", so only its export can refuse Bob. @@ -561,6 +554,6 @@ it("a hook enabled under the legacy protocol keeps firing after the upgrade impo using bobWorkspace = await bob.api.openWorkspace(ws.gadgetId, undefined, callback); using bobGadget = await bobWorkspace.getGadget(gadgetId); using bobFacet = await bobGadget.connectToGadget() as RpcStub; - expect(await bobFacet.deliveredValues()).toEqual([71, 73]); + expect(await bobFacet.deliveredValues()).toEqual([71, 72, 73]); expect(await historyExportCount(url)).toBe(1); }); diff --git a/packages/workshop-backend/__tests__/observer-ledger.test.ts b/packages/workshop-backend/__tests__/observer-ledger.test.ts index d55a7ae093..ffa3661d55 100644 --- a/packages/workshop-backend/__tests__/observer-ledger.test.ts +++ b/packages/workshop-backend/__tests__/observer-ledger.test.ts @@ -214,6 +214,13 @@ function requirements(impl: ObserverImpl, role: "build" | "use"): Map { + expect(await impl.observerLedger.switchToTracking(id)).toBe(true); + await impl.observerLedger.importHistory(id); +} + describe("byte-bounded observation persistence", () => { for (let historical of [false, true]) { let label = historical ? "historical import" : "live observation"; @@ -229,7 +236,7 @@ describe("byte-bounded observation persistence", () => { fakeFacets(impl, { advertise: new Set([LEGACY]), exports: new Map([[LEGACY, async () => ids]]), }); - await impl.observerLedger.initialize(LEGACY); + await switchAndImport(impl); } else { seedTracked(impl); await impl.authorizeTrackedObservation(TRACKED, trackedObservation(ids), @@ -271,7 +278,7 @@ describe("byte-bounded observation persistence", () => { if (++writes === 2) throw new Error("later chunk failed"); return put(record); }; - let run = () => historical ? impl.observerLedger.initialize(LEGACY) + let run = () => historical ? switchAndImport(impl) : impl.authorizeTrackedObservation(TRACKED, trackedObservation(ids, { containsRestrictedData: true, ownerInvitesOnly: true, }), { from: "user" }); @@ -285,12 +292,16 @@ describe("byte-bounded observation persistence", () => { expect(requirements(impl, "build").size).toBe(0); expect(impl.storage.containsRestrictedData.get()).toBeFalsy(); expect(impl.storage.ownerInvitesOnly.get()).toBeFalsy(); - if (historical) expect(impl.observerLedger.isTracked(LEGACY)).toBe(false); + // A failed import commits nothing, leaving the connection tracked with its history pending. + if (historical) { + expect(impl.storage.gatekeepers.get(LEGACY)) + .toMatchObject({ observerTracking: true, observerHistoryPending: true }); + } await run(); expect(requirements(impl, "build").get(historical ? LEGACY : TRACKED) ?.toSorted()).toEqual(ids.toSorted()); if (historical) { - expect(impl.observerLedger.isTracked(LEGACY)).toBe(true); + expect(impl.storage.gatekeepers.get(LEGACY).observerHistoryPending).toBeUndefined(); } else { expect(impl.storage.containsRestrictedData.get()).toBe(true); expect(impl.storage.ownerInvitesOnly.get()).toBe(true); @@ -538,7 +549,18 @@ describe("tracked hook observations", () => { await parked; return []; })); - using firing = await startTrackedHook(world); + // For "restart", the hook wakes a gadget still provisional to a chat, so the connection is + // outside "use" scope until a gadget binds it below. + let pendingGadget = GADGET + 1; + if (operation === "restart") { + impl.storage.gadgets.put({ + type: "gadget", id: pendingGadget, title: "P", created: new Date(0), + bindingName: "P", bindings: {}, pending: { chatId: 1 }, + }); + seedGadget(impl); + } + using firing = await startTrackedHook(world, + operation === "restart" ? pendingGadget : undefined); let before = impl.storage.nextActionId.get(); let authorize = firing.approvalQueue.authorizeTrackedObservation(trackedObservation( ["racing/event"], { containsRestrictedData: true, ownerInvitesOnly: true })) @@ -547,7 +569,11 @@ describe("tracked hook observations", () => { await checking; let refusal = operation === "restart" ? /restarting/ : /deleted or disabled/; if (operation === "restart") { - impl.restartForObserverTracking(TRACKED); + // Binding it while a "use" session is live widens that session's scope, so the workspace + // restarts and holds the connection back until the reset lands. + using _session = { [Symbol.dispose]: impl.joinSession("use") }; + impl.bindWorkpiece(GADGET, "C", TRACKED); + expect(world.restarts).toHaveLength(1); expect(impl.gatekeeperUsable(TRACKED)).toBe(false); } else { await firing.revoke(operation); @@ -596,21 +622,35 @@ describe("legacy and tracked connections coexist", () => { .rejects.toThrow(/cannot exclude observers/); })); - it("imports a legacy connection at its first tracked observation, and refuses that read", () => + it("switches a legacy connection at its first tracked observation, and authorizes that read", () => withWorld(async world => { let { impl } = world; seedLegacy(impl); + seedGadget(impl); + let exports = 0; fakeFacets(impl, { advertise: new Set([LEGACY]), - exports: new Map([[LEGACY, async () => ["history"]]]), + exports: new Map([[LEGACY, async () => { exports++; return ["history"]; }]]), }); + world.roles.set("carol", "build"); + impl.storage.observers.put( + { profileId: "carol", observerId: "obs-c", accountChoices: { [LEGACY]: 10 } }); // A session minted before the Workshop noticed the upgrade makes a tracked observation: the - // read is refused, not silently recorded or dropped, and the upgrade it reveals is imported. - await expect(impl.authorizeTrackedObservation(LEGACY, trackedObservation(["x"]), - { from: "user" })) - .rejects.toThrow(/must be set up/); + // connection switches, and the read is checked and recorded like any other. + await impl.authorizeTrackedObservation(LEGACY, trackedObservation(["x"]), + { from: "gadget", gadgetId: GADGET }); expect(impl.observerLedger.isTracked(LEGACY)).toBe(true); + expect(world.checks).toEqual([{ profileId: "carol", ids: ["x"] }]); + expect(rows(impl, gadget(GADGET))).toEqual(["1:x"]); + + // The switch imports the connection's history into the workspace, exactly once. + await impl.observerLedger.importHistory(LEGACY); + await impl.observerLedger.importHistory(LEGACY); expect(rows(impl, WORKSPACE)).toEqual(["1:history"]); + expect(impl.storage.gatekeepers.get(LEGACY).observerHistoryPending).toBeUndefined(); + expect(exports).toBe(1); + expect(world.restarts).toEqual([]); + expect(impl.gatekeeperUsable(LEGACY)).toBe(true); })); it("refuses subresources sent through the legacy method, recording and importing nothing", () => @@ -1086,7 +1126,7 @@ describe("historical import", () => { advertise, adds, exports: new Map([[LEGACY, async () => ["restricted/history"]]]), }); - await impl.observerLedger.initialize(LEGACY); + expect(await impl.observerLedger.switchToTracking(LEGACY)).toBe(false); advertise.add(LEGACY); world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => ids)); @@ -1101,23 +1141,28 @@ describe("historical import", () => { expect(impl.storage.observers.get("carol").accountChoices).toEqual({ [LEGACY]: 10 }); })); - it("refuses admission without registering when discovery fails, and imports once it succeeds", - () => withWorld(async world => { + it("keeps an addObserver refusal when describing the connection fails, and switches once it " + + "succeeds", () => withWorld(async world => { let { impl } = world; seedLegacy(impl); - let adds: number[] = []; - fakeFacets(impl, { adds }); + fakeFacets(impl); let fail = true; + let describes = 0; + let exports = 0; let getFacet = impl.getGatekeeperFacet; - // A gatekeeper that can't always be described: admission must not fall back to registering - // the collaborator through its addObserver(). + // An upgraded gatekeeper that refuses addObserver(), but can't always be described. impl.getGatekeeperFacet = async (id: number) => ({ ...await getFacet(id), describe: async () => { + describes++; if (fail) throw new Error("Discovery unavailable"); return { supportsObserverTracking: true }; }, - getHistoricalSubresources: async () => historyCursor(["restricted/history"]), + getHistoricalSubresources: async () => { + exports++; + return historyCursor(["restricted/history"]); + }, + addObserver: async () => { throw new Error("This connection is tracked by the Workshop."); }, }); let prompts: (string | undefined)[][] = []; @@ -1128,23 +1173,53 @@ describe("historical import", () => { }, }; await expect(impl.ensureObserver("carol", world.clientUser("carol"), "build", configure)) - .rejects.toThrow(/Legacy 1 .*observer tracking protocol/); - // The failure is that binding's own, offered for repair before admission is denied. - expect(prompts).toEqual([[undefined], [expect.stringMatching(/observer tracking protocol/)]]); - expect(adds).toEqual([]); + .rejects.toThrow(/Legacy 1 .*tracked by the Workshop/); + // The addObserver refusal is that binding's own, offered for repair before admission is + // denied; each refusal describes the connection once. + expect(prompts).toEqual([[undefined], [expect.stringMatching(/tracked by the Workshop/)]]); + expect(describes).toBe(2); + expect(exports).toBe(0); expect(impl.storage.observers.get("carol")).toBeUndefined(); - expect(impl.observerLedger.isTracked(LEGACY)).toBe(false); + expect(impl.storage.gatekeepers.get(LEGACY).observerTracking).toBeUndefined(); + expect(impl.storage.gatekeepers.get(LEGACY).observerHistoryPending).toBeUndefined(); fail = false; world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => ids)); await expect(impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll)) .rejects.toThrow(/could not confirm/); - expect(adds).toEqual([]); + expect(impl.observerLedger.isTracked(LEGACY)).toBe(true); expect(impl.storage.observers.get("carol")).toBeUndefined(); + expect(world.checks).toContainEqual({ profileId: "carol", ids: ["restricted/history"] }); expect(rows(impl, WORKSPACE)).toEqual(["1:restricted/history"]); + expect(exports).toBe(1); + })); + + it("keeps an addObserver refusal from a connection that doesn't advertise tracking", () => + withWorld(async world => { + let { impl } = world; + seedLegacy(impl); + fakeFacets(impl); + let describes = 0; + let getFacet = impl.getGatekeeperFacet; + impl.getGatekeeperFacet = async (id: number) => { + let facet = await getFacet(id); + return { + ...facet, + describe: async () => { describes++; return facet.describe(); }, + addObserver: async () => { throw new Error("This account can't read the sheet."); }, + }; + }; + world.roles.set("carol", "build"); + impl.storage.observers.put( + { profileId: "carol", observerId: "obs-c", accountChoices: { [LEGACY]: 10 } }); + + await expect(impl.ensureObserver("carol", world.clientUser("carol"), "build")) + .rejects.toThrow(/Legacy 1 .*can't read the sheet/); + expect(describes).toBe(1); + expect(impl.observerLedger.isTracked(LEGACY)).toBe(false); })); - it("describes a legacy connection once per admission, and not to open its sessions", () => + it("admits a legacy collaborator through addObserver without describing the connection", () => withWorld(async world => { let { impl } = world; seedLegacy(impl); @@ -1152,61 +1227,55 @@ describe("historical import", () => { fakeFacets(impl, { adds }); let describes = 0; let getFacet = impl.getGatekeeperFacet; + // Describing may use credentials the collaborator lacks: a describe() that throws must not + // lock out a collaborator the legacy protocol admits. impl.getGatekeeperFacet = async (id: number) => ({ ...await getFacet(id), - describe: async () => { describes++; return {}; }, + describe: async () => { describes++; throw new Error("Owner credentials expired"); }, startSession: async () => ({}), }); await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); expect(adds).toEqual([LEGACY]); - expect(describes).toBe(1); + expect(impl.storage.observers.get("carol").accountChoices).toEqual({ [LEGACY]: 10 }); await impl.openGatekeeperSession( LEGACY, await impl.getGatekeeperFacet(LEGACY), { from: "user" }); - expect(describes).toBe(1); + expect(describes).toBe(0); + expect(impl.observerLedger.isTracked(LEGACY)).toBe(false); })); it("imports an existing legacy connection that starts advertising, even with no prior reads", - async () => { - let name = `observer-ledger-import-${++doCounter}`; + () => withWorld(async world => { + let { impl } = world; + seedLegacy(impl); let exports = 0; - let facets = (impl: ObserverImpl) => fakeFacets(impl, { + fakeFacets(impl, { advertise: new Set([LEGACY]), exports: new Map([[LEGACY, async () => { exports++; return []; }]]), }); - await inOverseer(name, async ({ impl, restarts }) => { - seedLegacy(impl); - facets(impl); - let before = impl.storage.nextActionId.get(); - - await impl.observerLedger.initialize(LEGACY); - expect(impl.observerLedger.isTracked(LEGACY)).toBe(true); - expect(exports).toBe(1); - // An empty export writes no audit record. - expect(impl.storage.nextActionId.get()).toBe(before); - expect(rows(impl, WORKSPACE)).toEqual([]); - // The import aborted the facet that clients hold stubs to, so the workspace restarts even - // with only the owner connected, and the connection is held back until it does. - expect(restarts).toHaveLength(1); - expect(impl.gatekeeperUsable(LEGACY)).toBe(false); - }); + let before = impl.storage.nextActionId.get(); - await abortAllDurableObjects(); + await switchAndImport(impl); + expect(impl.observerLedger.isTracked(LEGACY)).toBe(true); + expect(impl.storage.gatekeepers.get(LEGACY).observerHistoryPending).toBeUndefined(); + expect(exports).toBe(1); + // An empty export writes no audit record. + expect(impl.storage.nextActionId.get()).toBe(before); + expect(rows(impl, WORKSPACE)).toEqual([]); + // The switch neither restarts the workspace nor holds the connection back. + expect(world.restarts).toEqual([]); + expect(impl.gatekeeperUsable(LEGACY)).toBe(true); - await inOverseer(name, async world => { - let { impl } = world; - facets(impl); - world.roles.set("carol", "build"); - await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); - await impl.authorizeTrackedObservation(LEGACY, trackedObservation(["first-real-read"]), - { from: "user" }); - expect(world.checks).toEqual([ - { profileId: "carol", ids: ["first-real-read"] }, - ]); - expect(rows(impl, WORKSPACE)).toEqual(["1:first-real-read"]); - expect(exports).toBe(1); - }); - }); + world.roles.set("carol", "build"); + await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); + await impl.authorizeTrackedObservation(LEGACY, trackedObservation(["first-real-read"]), + { from: "user" }); + expect(world.checks).toEqual([ + { profileId: "carol", ids: ["first-real-read"] }, + ]); + expect(rows(impl, WORKSPACE)).toEqual(["1:first-real-read"]); + expect(exports).toBe(1); + })); it("exports once for concurrent first callers and merges into the workspace context", () => withWorld(async world => { @@ -1219,7 +1288,11 @@ describe("historical import", () => { }); let before = impl.storage.nextActionId.get(); - await Promise.all([impl.observerLedger.initialize(3), impl.observerLedger.initialize(3)]); + let switches = await Promise.all([ + impl.observerLedger.switchToTracking(3), impl.observerLedger.switchToTracking(3), + ]); + expect(switches).toEqual([true, true]); + await Promise.all([impl.observerLedger.importHistory(3), impl.observerLedger.importHistory(3)]); expect(exports).toBe(1); expect(impl.observerLedger.isTracked(3)).toBe(true); // One workspace audit record carries the whole (deduplicated) export. @@ -1233,46 +1306,35 @@ describe("historical import", () => { })); it("imports a paged history larger than one observation and checks every verification batch", - async () => { - let name = `observer-ledger-import-${++doCounter}`; + () => withWorld(async world => { + let { impl } = world; let ids = Array.from({ length: 10_001 }, (_, i) => String(i)); - let facets = (impl: ObserverImpl) => fakeFacets(impl, { + seedLegacy(impl); + fakeFacets(impl, { advertise: new Set([LEGACY]), exports: new Map([[LEGACY, async () => ids]]), }); - let required: string[] = []; - await inOverseer(name, async world => { - let { impl } = world; - seedLegacy(impl); - facets(impl); - await impl.observerLedger.initialize(LEGACY); - required = requirements(impl, "build").get(LEGACY)!; - expect(required.toSorted()).toEqual(ids.toSorted()); - world.roles.set("carol", "build"); - world.verifiers.set("carol", observerVerifier(() => world, "carol", - batch => batch.filter(id => id === required.at(-1)))); - await expect(impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll)) - .rejects.toThrow(/could not confirm/); - expect(world.checks).toContainEqual({ profileId: "carol", ids: required.slice(10_000) }); - expect(impl.storage.observers.get("carol")).toBeUndefined(); - }); - - // Reads wait for the restart the import scheduled. - await abortAllDurableObjects(); + await switchAndImport(impl); + let required = requirements(impl, "build").get(LEGACY)!; + expect(required.toSorted()).toEqual(ids.toSorted()); + world.roles.set("carol", "build"); + world.verifiers.set("carol", observerVerifier(() => world, "carol", + batch => batch.filter(id => id === required.at(-1)))); + await expect(impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll)) + .rejects.toThrow(/could not confirm/); + expect(world.checks).toContainEqual({ profileId: "carol", ids: required.slice(10_000) }); + expect(impl.storage.observers.get("carol")).toBeUndefined(); - await inOverseer(name, async world => { - let { impl } = world; - facets(impl); - await expect(impl.authorizeTrackedObservation(LEGACY, trackedObservation(ids), - { from: "user" })).rejects.toThrow(/at most 10000/); - expect(requirements(impl, "build").get(LEGACY)).toEqual(required); - world.roles.set("carol", "build"); - await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); - expect(impl.storage.observers.get("carol").accountChoices).toEqual({ [LEGACY]: 10 }); - }); - }); + // One observation is still held to the per-call limit the import paged around. + await expect(impl.authorizeTrackedObservation(LEGACY, trackedObservation(ids), + { from: "user" })).rejects.toThrow(/at most 10000/); + expect(requirements(impl, "build").get(LEGACY)).toEqual(required); + world.verifiers.delete("carol"); + await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); + expect(impl.storage.observers.get("carol").accountChoices).toEqual({ [LEGACY]: 10 }); + })); - it("refuses an export page of more than 10,000 IDs and stays legacy", () => + it("refuses an export page of more than 10,000 IDs and leaves the history pending", () => withWorld(async world => { let { impl } = world; seedLegacy(impl); @@ -1282,11 +1344,15 @@ describe("historical import", () => { exports: new Map([[LEGACY, async () => historyCursor(ids, ids.length)]]), }); - await expect(impl.observerLedger.initialize(LEGACY)).rejects.toThrow(/at most 10000/); - expect(impl.observerLedger.isTracked(LEGACY)).toBe(false); + await expect(switchAndImport(impl)).rejects.toThrow(/at most 10000/); + expect(impl.storage.gatekeepers.get(LEGACY)) + .toMatchObject({ observerTracking: true, observerHistoryPending: true }); expect(rows(impl, WORKSPACE)).toEqual([]); + expect(impl.gatekeeperUsable(LEGACY)).toBe(true); })); + // The pending history is durable: a failure, or the object resetting, leaves it for the next + // admission to export again. it.each(["the export call", "a later page"])( "retries after a failure before the commit, and never after it (failing at %s)", async failure => { @@ -1310,32 +1376,47 @@ describe("historical import", () => { await inOverseer(name, async ({ impl, restarts }) => { seedLegacy(impl, 3); facets(impl); - await expect(impl.observerLedger.initialize(3)).rejects.toThrow(/export interrupted/); - // A failed export never clears workspace requirements, and leaves the connection legacy. - expect(impl.observerLedger.isTracked(3)).toBe(false); + await expect(switchAndImport(impl, 3)).rejects.toThrow(/export interrupted/); + // A failed export commits nothing, and leaves the connection tracked and usable with its + // history still pending. + expect(impl.storage.gatekeepers.get(3)) + .toMatchObject({ observerTracking: true, observerHistoryPending: true }); expect(rows(impl, WORKSPACE)).toEqual([]); - // The failed import had already aborted the facet clients hold, so the workspace restarts. - expect(restarts).toHaveLength(1); - - fail = false; - await impl.observerLedger.initialize(3); - expect(exports).toBe(2); + expect(restarts).toEqual([]); + expect(impl.gatekeeperUsable(3)).toBe(true); + expect(exports).toBe(1); }); await abortAllDurableObjects(); - await inOverseer(name, async ({ impl }) => { + fail = false; + await inOverseer(name, async world => { + let { impl } = world; facets(impl); - await impl.observerLedger.initialize(3); - expect(exports).toBe(2); // completion is durable: the stored history is authoritative + expect(impl.storage.gatekeepers.get(3).observerHistoryPending).toBe(true); + world.roles.set("carol", "build"); + world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => ids)); + // The next admission exports again, and verifies her against what it imported. + await expect(impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll)) + .rejects.toThrow(/could not confirm/); + expect(exports).toBe(2); + expect(world.checks).toContainEqual({ profileId: "carol", ids: ["p1"] }); expect(rows(impl, WORKSPACE)).toEqual(["3:p1"]); + expect(impl.storage.gatekeepers.get(3).observerHistoryPending).toBeUndefined(); + + world.verifiers.delete("carol"); + await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); + await impl.observerLedger.importHistory(3); + expect(exports).toBe(2); // completion is durable: the stored history is authoritative }); }); - it("refuses legacy hook queues while their historical import is parked", () => - withWorld(async world => { + it.each([["the imported", "history"], ["the concurrent", "live"], ["no", null]] as const)( + "keeps a switched connection usable while its import is parked, and admits only after it " + + "(denying %s requirement)", (_label, denied) => withWorld(async world => { let { impl } = world; seedLegacy(impl); + seedGadget(impl); let release!: () => void; let reached!: () => void; let parked = new Promise(resolve => { release = resolve; }); @@ -1348,24 +1429,65 @@ describe("historical import", () => { return ["history"]; }]]), }); - let before = impl.storage.nextActionId.get(); - let importing = impl.observerLedger.initialize(LEGACY); + world.roles.set("carol", "build"); + impl.storage.observers.put( + { profileId: "carol", observerId: "obs-c", accountChoices: { [LEGACY]: 10 } }); + expect(await impl.observerLedger.switchToTracking(LEGACY)).toBe(true); await exporting; + // The legacy protocol is refused at once; tracked reads proceed without waiting. + let before = impl.storage.nextActionId.get(); await expect(impl.authorizeObservation(LEGACY, observation(), { from: "hook" })) - .rejects.toThrow(/being imported/); - expect(rows(impl, WORKSPACE)).toEqual([]); + .rejects.toThrow(/must report its subresources through authorizeTrackedObservation/); expect(impl.storage.nextActionId.get()).toBe(before); - expect(impl.storage.actions.get(before)).toBeUndefined(); - expect(impl.observerLedger.isTracked(LEGACY)).toBe(false); + await impl.authorizeTrackedObservation(LEGACY, trackedObservation(["live"]), + { from: "gadget", gadgetId: GADGET }); + expect(world.checks).toEqual([{ profileId: "carol", ids: ["live"] }]); + expect(rows(impl, gadget(GADGET))).toEqual(["1:live"]); + expect(rows(impl, WORKSPACE)).toEqual([]); + expect(impl.gatekeeperUsable(LEGACY)).toBe(true); + + // An admission waits for the history before verifying anyone against the connection. + world.roles.set("dave", "build"); + world.verifiers.set("dave", observerVerifier(() => world, "dave", + ids => ids.filter(id => id === denied))); + let { promise: prompted, resolve: promptedResolve } = Promise.withResolvers(); + let configure = { + configure: async (needs: { gatekeeperId: number }[]) => { + promptedResolve(); + return configureAll.configure(needs); + }, + }; + let settled = false; + let admission = impl.ensureObserver("dave", world.clientUser("dave"), "build", configure) + .then(() => ({ allowed: true, error: "" }), + (error: unknown) => ({ allowed: false, error: String(error) })) + .finally(() => { settled = true; }); + await prompted; + // Another full read round trip through the connection, while the admission's pass is past + // its prompt: the read still doesn't wait, and the admission still hasn't checked anyone. + await impl.authorizeTrackedObservation(LEGACY, trackedObservation(["live"]), + { from: "gadget", gadgetId: GADGET }); + expect(settled).toBe(false); + expect(world.checks.filter(check => check.profileId === "dave")).toEqual([]); release(); - await importing; + let result = await admission; expect(rows(impl, WORKSPACE)).toEqual(["1:history"]); + expect(world.checks.map(check => ({ ...check, ids: check.ids.toSorted() }))) + .toContainEqual({ profileId: "dave", ids: ["history", "live"] }); + expect(result.allowed).toBe(denied === null); + if (denied === null) { + expect(impl.storage.observers.get("dave").accountChoices).toEqual({ [LEGACY]: 10 }); + } else { + expect(result.error).toMatch(/could not confirm/); + expect(impl.storage.observers.get("dave")).toBeUndefined(); + } + expect(world.restarts).toEqual([]); })); - it("refuses a legacy observation imported while its policy lookup was in flight", () => - withWorld(async world => { + it("refuses a legacy observation whose connection switches while its policy lookup is in " + + "flight", () => withWorld(async world => { let { impl } = world; seedLegacy(impl); fakeFacets(impl, { advertise: new Set([LEGACY]) }); @@ -1383,18 +1505,20 @@ describe("historical import", () => { let read = impl.authorizeObservation(LEGACY, observation({ ownerInvitesOnly: true, containsRestrictedData: true }), { from: "hook" }); await lookingUp; - await impl.observerLedger.initialize(LEGACY); + expect(await impl.observerLedger.switchToTracking(LEGACY)).toBe(true); release(); - // The import's restart holds the connection back, so the read commits under neither protocol. - await expect(read).rejects.toThrow(/restarting/); + // The ledger checks the protocol at commit, so the read commits under neither protocol. + await expect(read).rejects.toThrow( + /must report its subresources through authorizeTrackedObservation/); expect(impl.storage.nextActionId.get()).toBe(before); expect(impl.storage.actions.get(before)).toBeUndefined(); expect(impl.storage.ownerInvitesOnly.get()).toBe(false); expect(impl.storage.containsRestrictedData.get()).toBe(false); })); - it("refuses a legacy exclusion check that the import overtakes", () => withWorld(async world => { + it("refuses a legacy exclusion check that the switch overtakes", () => + withWorld(async world => { let { impl } = world; seedLegacy(impl); fakeFacets(impl, { advertise: new Set([LEGACY]) }); @@ -1412,10 +1536,11 @@ describe("historical import", () => { let read = impl.authorizeObservation(LEGACY, observation({ excludeObservers: ["obs-unknown"] }), { from: "user" }); await excluding; - await impl.observerLedger.initialize(LEGACY); + expect(await impl.observerLedger.switchToTracking(LEGACY)).toBe(true); release(); - await expect(read).rejects.toThrow(/restarting/); + await expect(read).rejects.toThrow( + /must report its subresources through authorizeTrackedObservation/); expect(impl.storage.nextActionId.get()).toBe(before); })); @@ -1434,7 +1559,9 @@ describe("historical import", () => { await impl.addGatekeeper({} as any, spec("trackedvendor", 9), "owner"); let id = [...impl.storage.gatekeepers.list()][0].id; - await impl.observerLedger.initialize(id); + expect(await impl.observerLedger.switchToTracking(id)).toBe(true); + await impl.observerLedger.importHistory(id); + expect(impl.storage.gatekeepers.get(id).observerHistoryPending).toBeUndefined(); expect(impl.observerLedger.isTracked(id)).toBe(true); expect(impl.storage.gatekeepers.get(id).observerTracking).toBe(true); expect(exports).toBe(0); @@ -1493,7 +1620,8 @@ describe("historical import", () => { vendorId: "trackedvendor", resourceUrlPattern: "https://example.com/*", title: "New", bindingName: "N", })); - await impl.observerLedger.initialize(gatekeeperId); + // A pending creation is simulated, so there is nothing to switch to tracking yet. + expect(await impl.observerLedger.switchToTracking(gatekeeperId)).toBe(false); let session = await impl.openGatekeeperSession( gatekeeperId, await impl.getGatekeeperFacet(gatekeeperId), { from: "user" }); let before = impl.storage.nextActionId.get(); @@ -1532,17 +1660,17 @@ describe("historical import", () => { id: gatekeeperId, creationSpec: { resourceUrl: "https://example.com/real" }, }); // Approval makes it an existing connection advertising support. Its first tracked - // observation imports it, exactly once, and is refused: its session predates the import. + // observation, through a session opened before then, switches it and is checked and + // recorded like any other; its (empty) history is imported exactly once. let staleSession = await impl.openGatekeeperSession( gatekeeperId, await impl.getGatekeeperFacet(gatekeeperId), { from: "user" }); - await expect(staleSession.read(trackedObservation(["real-id"]))) - .rejects.toThrow(/now supports/); + await staleSession.read(trackedObservation(["real-id"])); expect(impl.storage.gatekeepers.get(gatekeeperId).observerTracking).toBe(true); + await impl.observerLedger.importHistory(gatekeeperId); expect(exports).toBe(1); - expect(rows(impl, WORKSPACE)).toEqual([]); - // The import restarts the workspace, holding the connection back until it lands. - expect(world.restarts).toHaveLength(1); - expect(impl.gatekeeperUsable(gatekeeperId)).toBe(false); + expect(rows(impl, WORKSPACE)).toEqual([`${gatekeeperId}:real-id`]); + expect(world.restarts).toEqual([]); + expect(impl.gatekeeperUsable(gatekeeperId)).toBe(true); } finally { leave(); } @@ -1560,17 +1688,19 @@ describe("historical import", () => { await impl.ensureObserver("carol", world.clientUser("carol"), "build", configureAll); world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => ids)); let beforeDenied = impl.storage.nextActionId.get(); - await expect(realSession.read(trackedObservation(["real-id"]))) + await expect(realSession.read(trackedObservation(["next-id"]))) .rejects.toThrow(/could not be confirmed/); expect(impl.storage.nextActionId.get()).toBe(beforeDenied); - expect(rows(impl, WORKSPACE)).toEqual([]); + expect(rows(impl, WORKSPACE)).toEqual([`${gatekeeperId}:real-id`]); world.verifiers.delete("carol"); - await realSession.read(trackedObservation(["real-id"])); + await realSession.read(trackedObservation(["next-id"])); expect(world.checks).toEqual([ { profileId: "carol", ids: ["real-id"] }, - { profileId: "carol", ids: ["real-id"] }, + { profileId: "carol", ids: ["next-id"] }, + { profileId: "carol", ids: ["next-id"] }, ]); - expect(rows(impl, WORKSPACE)).toEqual([`${gatekeeperId}:real-id`]); + expect(rows(impl, WORKSPACE)) + .toEqual([`${gatekeeperId}:next-id`, `${gatekeeperId}:real-id`]); expect(exports).toBe(1); }); }); @@ -1587,7 +1717,7 @@ describe("historical import", () => { advertise: new Set([3]), exports: new Map([[3, async () => [second]]]), }); - await impl.observerLedger.initialize(3); + await switchAndImport(impl, 3); world.roles.set("carol", "build"); world.verifiers.set("carol", observerVerifier(() => world, "carol", ids => ids.filter(id => id !== first))); diff --git a/packages/workshop-backend/src/observer-ledger.ts b/packages/workshop-backend/src/observer-ledger.ts index c68f0bfa72..cfd1dc1a4d 100644 --- a/packages/workshop-backend/src/observer-ledger.ts +++ b/packages/workshop-backend/src/observer-ledger.ts @@ -12,6 +12,9 @@ // scope before the observer record is (OverseerImpl.ensureObserver, via observerScope()). // - Legacy connections never reach these checks: their addObserver()/removeObserver()/ // excludeObservers protocol is untouched, and a workspace may mix both. +// - A connection switches to tracking before its legacy history is imported, so no read through +// it goes unchecked; until that import commits (observerHistoryPending), admission waits for it +// rather than verify anyone against an incomplete history. Reads never wait for it. import type { CollaboratorRole, WorkpieceId } from "@gadgets/workshop-shared/api"; import type { @@ -94,11 +97,17 @@ export type ObserverScope = { nextActionId: number; }; +// A connection's observer protocol as admission verifies it: legacy, tracked while its history is +// still being imported, or tracked. +function observerProtocol(gk: GatekeeperRecord): string { + if (!gk.observerTracking) return "legacy"; + return gk.observerHistoryPending ? "importing" : "tracked"; +} + // Whether two scopes hold the same connections, each on the same observer protocol. function sameObserverConnections(a: GatekeeperRecord[], b: GatekeeperRecord[]): boolean { - let tracked = new Map(a.map(gk => [gk.id, gk.observerTracking])); - return b.length === tracked.size && - b.every(gk => tracked.has(gk.id) && tracked.get(gk.id) === gk.observerTracking); + let protocols = new Map(a.map(gk => [gk.id, observerProtocol(gk)])); + return b.length === protocols.size && b.every(gk => protocols.get(gk.id) === observerProtocol(gk)); } /** @@ -125,22 +134,14 @@ export interface ObserverLedgerHost { users: DurableObjectNamespace; getSharingManager(): Promise; getGatekeeperFacet(id: number): Promise>>; - gatekeeperFacetName(id: WorkpieceId): string; creationPending(id: WorkpieceId): boolean; - /** Throws (retryably) while the connection is held back pending a restart, or importing. */ + /** Throws (retryably) while the connection is held back pending a restart. */ assertGatekeeperUsable(id: number): void; /** Whether `executionId` is the executeCode invocation still running for `chatId`. */ isLiveExecution(chatId: number, executionId: string): boolean; - /** - * A connection's historical import aborted its facet: restart the workspace, so that no client - * keeps a stub to the aborted facet and sessions admitted under the legacy protocol reopen - * against the imported requirements, and hold the connection back until the reset lands. - */ - restartForObserverTracking(gatekeeperId: WorkpieceId): void; - /** Best-effort removal of every legacy observer registration on `gatekeeperId`. */ retireLegacyObserverRegistrations(gatekeeperId: WorkpieceId): Promise; } @@ -194,11 +195,11 @@ async function readHistoricalExport(facet: TrackingGatekeeper): Promise>(); + // In-flight switches to tracking, one per connection (see switchToTracking()). + #switching = new Map>(); - // Connections whose historical import is in flight (see isImporting). - #importing = new Set(); + // In-flight historical imports, one per connection (see importHistory()). + #importing = new Map>(); constructor(private host: ObserverLedgerHost) {} @@ -209,14 +210,6 @@ export class ObserverLedger { return this.#storage.gatekeepers.get(gatekeeperId)?.observerTracking === true; } - /** - * Whether the connection's historical import is in flight. Nothing may reach the connection - * until it commits; the Overseer's assertGatekeeperUsable() refuses it meanwhile. - */ - isImporting(gatekeeperId: WorkpieceId): boolean { - return this.#importing.has(gatekeeperId); - } - // --------------------------------------------------------------------------------------- // Connection lifecycle @@ -234,30 +227,28 @@ export class ObserverLedger { } /** - * Describe a legacy connection and import it if it now advertises support. Called at admission - * before the connection's addObserver() (see OverseerImpl.ensureObserver), so a collaborator is - * never registered with an upgraded gatekeeper, and when a read through the connection calls - * authorizeTrackedObservation(). Single-flight per connection. Does nothing for a connection - * that is tracked already, vendorless, removed, still legacy, or a pending creation (simulated, - * so nothing to track until it is approved). Throws if the connection can't be described or its - * history can't be imported, so a failed discovery never falls back to the legacy protocol. + * Describe a legacy connection and, if it now advertises support, switch it to tracking at once, + * with its history pending (see importHistory(), started here in the background). Called when a + * read through it calls authorizeTrackedObservation() and when its addObserver() refuses a + * collaborator (see OverseerImpl.ensureObserver), both of which an upgraded gatekeeper does. + * Single-flight per connection. Returns whether the connection is tracked afterwards: false for + * a vendorless one, a pending creation (simulated, so nothing to track until it is approved), or + * one still legacy. Throws if it can't be described or has been removed, so a failed discovery + * never leaves it to the legacy protocol it has upgraded from. */ - initialize(gatekeeperId: WorkpieceId): Promise { - if (this.isTracked(gatekeeperId) || this.host.creationPending(gatekeeperId)) { - return Promise.resolve(); - } - let pending = this.#initializing.get(gatekeeperId); + switchToTracking(gatekeeperId: WorkpieceId): Promise { + if (this.isTracked(gatekeeperId)) return Promise.resolve(true); + let pending = this.#switching.get(gatekeeperId); if (pending === undefined) { - pending = this.#initialize(gatekeeperId) - .finally(() => this.#initializing.delete(gatekeeperId)); - this.#initializing.set(gatekeeperId, pending); + pending = this.#switch(gatekeeperId).finally(() => this.#switching.delete(gatekeeperId)); + this.#switching.set(gatekeeperId, pending); } return pending; } - async #initialize(gatekeeperId: WorkpieceId): Promise { + async #switch(gatekeeperId: WorkpieceId): Promise { let record = this.#storage.gatekeepers.get(gatekeeperId); - if (!connectionVendorId(record)) return; // vendorless or gone: never verifies anyone + if (!connectionVendorId(record) || this.host.creationPending(gatekeeperId)) return false; let description: ResourceDescription; try { @@ -267,45 +258,61 @@ export class ObserverLedger { "This connection could not confirm its observer tracking protocol, so it can't be " + "used right now. Please retry.", {cause: error}); } + if (!description.supportsObserverTracking) return false; + + record = this.#storage.gatekeepers.get(gatekeeperId); + if (!record) throw new Error("This connection has been removed from the workspace."); + // From here every observation through it is checked and recorded here, and the legacy + // protocol refuses it; the history import covers everything read before. + this.#storage.gatekeepers.put({...record, observerTracking: true, observerHistoryPending: true}); + this.host.logger.info("switched connection to observer tracking", { + event: "observer.ledger.switch.completed", gatekeeperId, + }); + this.host.ctx.waitUntil(this.importHistory(gatekeeperId).catch(error => { + this.host.logger.warn("failed to import connection history", { + event: "observer.ledger.import.failed", gatekeeperId, error, + }); + })); + return true; + } - if (!this.#storage.gatekeepers.get(gatekeeperId)) { - throw new Error("This connection has been removed from the workspace."); + /** + * Import the history of a connection that switched to tracking after creation: the + * requirements its gatekeeper recorded under the legacy protocol. Single-flight per connection; + * resolves at once when none is pending. Observations through the connection carry on + * meanwhile, recording their own requirements, so the import only adds records. A failure + * commits nothing and leaves the history pending, so the next admission that needs it exports + * again; after the commit, the export is never asked for again. + */ + importHistory(gatekeeperId: WorkpieceId): Promise { + if (!this.#storage.gatekeepers.get(gatekeeperId)?.observerHistoryPending) { + return Promise.resolve(); } - if (description.supportsObserverTracking) await this.#import(gatekeeperId); + let pending = this.#importing.get(gatekeeperId); + if (pending === undefined) { + pending = this.#import(gatekeeperId).finally(() => this.#importing.delete(gatekeeperId)); + this.#importing.set(gatekeeperId, pending); + } + return pending; } - // The fenced historical import of a legacy connection. Aborting the facet ends the sessions - // minted under the legacy protocol, and isImporting() refuses everything else until the import - // settles. The abort also invalidates every client's stub to the facet, so the workspace then - // restarts whether or not the import commits. A failure commits nothing, so the next tracked - // observation or admission exports again; after the commit, the imported history is - // authoritative and export is never asked for again. async #import(gatekeeperId: WorkpieceId): Promise { - this.#importing.add(gatekeeperId); - let subresources: string[]; - try { - this.host.ctx.facets.abort(this.host.gatekeeperFacetName(gatekeeperId), - new Error("Connection restarted to move its observer tracking into the workspace.")); - let facet = await this.host.getGatekeeperFacet(gatekeeperId) as TrackingGatekeeper; - subresources = await readHistoricalExport(facet); - - let record = this.#storage.gatekeepers.get(gatekeeperId); - if (!record) throw new Error("This connection has been removed from the workspace."); - this.#storage.transaction(() => { - this.#storage.gatekeepers.put({...record, observerTracking: true}); - if (subresources.length > 0) this.#recordImportedHistory(record, subresources); - }); - } finally { - // In one synchronous block, so the connection is never usable between the two fences. - this.#importing.delete(gatekeeperId); - this.host.restartForObserverTracking(gatekeeperId); - } + let facet = await this.host.getGatekeeperFacet(gatekeeperId) as TrackingGatekeeper; + let subresources = await readHistoricalExport(facet); + + let record = this.#storage.gatekeepers.get(gatekeeperId); + if (!record) throw new Error("This connection has been removed from the workspace."); + let {observerHistoryPending: _, ...imported} = record; + this.#storage.transaction(() => { + this.#storage.gatekeepers.put(imported); + if (subresources.length > 0) this.#recordImportedHistory(record, subresources); + }); this.host.logger.info("imported connection history into the observer ledger", { event: "observer.ledger.import.completed", gatekeeperId, size: subresources.length, }); - // The restart re-admits live sessions under the ledger; retire legacy registrations - // best-effort meanwhile. + // Admission no longer registers anyone with this connection; retire the registrations the + // legacy protocol made, best-effort. this.host.ctx.waitUntil(this.host.retireLegacyObserverRegistrations(gatekeeperId)); } @@ -391,15 +398,14 @@ export class ObserverLedger { /** * The ledger half of Overseer.authorizeTrackedObservation(). Through a legacy connection, the - * call shows that its gatekeeper upgraded, so this initializes the connection (see initialize()) - * and refuses the read, which came through a session minted under the legacy protocol (the - * import restarts its facet). For a tracked connection, attributes the read to its caller's - * context and checks the reported IDs for everyone that context reaches, then recomputes that - * audience after the awaits and checks whoever it gained: sharing changes, newly published - * observers, and changed account choices. Calls `commit` synchronously with the validated IDs at - * the final decision, so no admission can publish between the audience check and the audit - * write; resolving a promise first would let one publish before the caller writes the - * requirement. The callback must not await. A pending creation commits with undefined, + * call shows that its gatekeeper upgraded, so this first switches the connection to tracking + * (see switchToTracking()), whose history import doesn't hold up the read. Then attributes the + * read to its caller's context and checks the reported IDs for everyone that context reaches, + * then recomputes that audience after the awaits and checks whoever it gained: sharing changes, + * newly published observers, and changed account choices. Calls `commit` synchronously with the + * validated IDs at the final decision, so no admission can publish between the audience check + * and the audit write; resolving a promise first would let one publish before the caller writes + * the requirement. The callback must not await. A pending creation commits with undefined, * recording no requirement. */ async authorizeTrackedObservation( @@ -408,15 +414,11 @@ export class ObserverLedger { commit: (subresources: string[] | undefined) => undefined): Promise { this.#assertUsable(gatekeeperId); if (this.host.creationPending(gatekeeperId)) return commit(undefined); - if (!this.isTracked(gatekeeperId)) { - // Only an upgraded gatekeeper calls this, so import the connection now. The read came - // through a session minted under the legacy protocol, whose facet the import restarts, so - // it is refused either way. - await this.initialize(gatekeeperId); - throw new Error(this.isTracked(gatekeeperId) - ? "This connection now supports the Workshop's observer tracking, which must be set " + - "up before it can be used. Please retry." - : "This observation was blocked: the connection does not advertise observer tracking."); + // Only an upgraded gatekeeper calls this, so switch a legacy connection to tracking, then + // check this read like any other; the switch imports the connection's history meanwhile. + if (!this.isTracked(gatekeeperId) && !await this.switchToTracking(gatekeeperId)) { + throw new Error( + "This observation was blocked: the connection does not advertise observer tracking."); } // RPC validation forwards properties the method doesn't declare. @@ -474,7 +476,7 @@ export class ObserverLedger { } } - // Refuses authorization through a removed connection, or one held back or importing. + // Refuses authorization through a removed connection, or one held back. #assertUsable(gatekeeperId: WorkpieceId): void { if (!this.#storage.gatekeepers.get(gatekeeperId)) { throw new Error("This connection has been removed from the workspace."); diff --git a/packages/workshop-backend/src/overseer.ts b/packages/workshop-backend/src/overseer.ts index 202f379241..2eaafc08c3 100644 --- a/packages/workshop-backend/src/overseer.ts +++ b/packages/workshop-backend/src/overseer.ts @@ -6358,29 +6358,24 @@ class OverseerImpl implements AgentHooks { // along. #gatekeepersPendingRestart = new Set(); - // Whether `id` is NOT blocked pending a scheduled restart (see #gatekeepersPendingRestart) or - // its observer-tracking import (ObserverLedger.isImporting). For callers that enumerate - // connections (listSlashCommands, prepareChatBindings' ambient catalogs) and silently skip a - // blocked one -- it reappears once the reset lands and clients reconnect. A caller reaching for - // one specific connection throws via assertGatekeeperUsable instead. + // Whether `id` is NOT blocked pending a scheduled restart (see #gatekeepersPendingRestart). + // For callers that enumerate connections (listSlashCommands, prepareChatBindings' ambient + // catalogs) and silently skip a blocked one -- it reappears once the reset lands and clients + // reconnect. A caller reaching for one specific connection throws via assertGatekeeperUsable + // instead. gatekeeperUsable(id: number): boolean { - return !this.#gatekeepersPendingRestart.has(id) && !this.observerLedger.isImporting(id); + return !this.#gatekeepersPendingRestart.has(id); } - // Throw (retryable) if `id` is blocked pending the scheduled restart or its observer-tracking - // import; see #gatekeepersPendingRestart. Called from getGatekeeperById (the mint clients - // pipeline on), openGatekeeperSession (the chokepoint every gatekeeper session passes through), - // the slash-command invoke in #prepareChatMessage, GadgetClientImpl.bindWithSuggestedName (the + // Throw (retryable) if `id` is blocked pending the scheduled restart; see + // #gatekeepersPendingRestart. Called from getGatekeeperById (the mint clients pipeline on), + // openGatekeeperSession (the chokepoint every gatekeeper session passes through), the + // slash-command invoke in #prepareChatMessage, GadgetClientImpl.bindWithSuggestedName (the // latter two take a client-supplied gatekeeper id and reach the connection outside // openGatekeeperSession), startHook (the inbound delivery route, whose arming enable may itself // be the widening that scheduled the restart), and the ObserverLedger's authorize methods. assertGatekeeperUsable(id: number): void { - if (this.observerLedger.isImporting(id)) { - throw new Error( - "This connection's observer tracking history is being imported. Please retry in a " + - "moment."); - } - if (this.#gatekeepersPendingRestart.has(id)) { + if (!this.gatekeeperUsable(id)) { throw new Error( "The workspace is restarting to apply a connection change. Please retry."); } @@ -6431,17 +6426,6 @@ class OverseerImpl implements AgentHooks { return true; } - // ObserverLedgerHost: a historical import aborted the connection's facet, which invalidates every - // stub to it that a client capability holds (GatekeeperClientImpl keeps the facet it was minted - // with), and sessions admitted under the legacy protocol must reopen against the imported - // requirements. So restart even when only the owner is connected, and hold the connection back - // until the reset lands, as for a newly added connection. - restartForObserverTracking(gatekeeperId: WorkpieceId): void { - this.#gatekeepersPendingRestart.add(gatekeeperId); - this.scheduleAccessRestart( - "Gadget restarted to move a connection's observer tracking into the workspace."); - } - // Diff the account-requiring "use" scope against a snapshot taken just before a mutation, and // when it widened onto a live "use" session, restart -- additionally blocking each widened // connection until the reset lands (see #gatekeepersPendingRestart): the severed sessions stay @@ -10137,6 +10121,13 @@ class OverseerImpl implements AgentHooks { try { if (gk.observerTracking) { + // Until a switched connection's history is imported, this pass's requirements for + // it are incomplete: wait for the import, which step 6 sees as a protocol change, + // re-running the pass against the imported requirements. + if (gk.observerHistoryPending) { + await this.observerLedger.importHistory(gk.id); + return; + } await this.observerLedger.checkAccess( clientUser, accountId, observerId, gk, scope.requirements.get(gk.id) ?? []); return; @@ -10147,15 +10138,22 @@ class OverseerImpl implements AgentHooks { // exclusion teardown's removeObserver for the same pair is still in flight (which // would delete it moments later); see #serialized. await this.#serialized(`${observerId}/${gk.id}`, async () => { - // Describe the connection first, and import it if it now advertises observer - // tracking (already tracked, it was imported while this open awaited): step 6 then - // reruns the pass against its tracked requirements. A failed description refuses - // the binding rather than trusting the legacy protocol, and an upgraded - // gatekeeper's addObserver() throws, so a redeploy between the two can't admit - // anyone either. - await this.observerLedger.initialize(gk.id); - if (this.observerLedger.isTracked(gk.id)) return; - await (await this.getGatekeeperFacet(gk.id)).addObserver(observerId, verifier); + try { + await (await this.getGatekeeperFacet(gk.id)).addObserver(observerId, verifier); + } catch (err) { + // An upgraded gatekeeper's addObserver() throws, and only then is the connection + // described, so admitting a collaborator the legacy protocol accepts never + // depends on the description. A switch to tracking leaves step 6 to re-run the + // pass against the tracked requirements; otherwise the refusal stands. + let switched = await this.observerLedger.switchToTracking(gk.id).catch(error => { + this.logger.warn("failed to describe a connection that refused an observer", { + event: "gatekeeper.observer.describe.failed", + gatekeeperId: gk.id, observerId, error, + }); + return false; + }); + if (!switched) throw err; + } }); } catch (err) { // Either a settled denial or an operational failure (expired credentials, upstream @@ -10169,10 +10167,11 @@ class OverseerImpl implements AgentHooks { })); // 6. What this open must verify can change while it awaits: a requirement recorded, a - // connection added, removed, approved, or moved to Workshop tracking. Successes and - // failures alike may be stale, so re-run the pass rather than admit against them. This - // check and the publication below share one synchronous block, so nothing recorded - // after it can be missed: an observation checks every published observer. + // connection added, removed, approved, switched to Workshop tracking, or its history + // imported. Successes and failures alike may be stale, so re-run the pass rather than + // admit against them. This check and the publication below share one synchronous + // block, so nothing recorded after it can be missed: an observation checks every + // published observer. let current = this.observerLedger.observerScope(role, this.#inScopeGatekeepers(role), scope); if (!observerScopeCovers(scope, current)) { diff --git a/packages/workshop-backend/src/storage-schema/overseer-storage.ts b/packages/workshop-backend/src/storage-schema/overseer-storage.ts index 1ede2230a5..c49945aa6c 100644 --- a/packages/workshop-backend/src/storage-schema/overseer-storage.ts +++ b/packages/workshop-backend/src/storage-schema/overseer-storage.ts @@ -64,11 +64,17 @@ export type GatekeeperRecord = { /** * Set once the Workshop tracks this connection's observations itself (see observer-ledger.ts): - * created advertising ResourceDescription.supportsObserverTracking, or its history imported - * when it began advertising it. Never cleared. + * created advertising ResourceDescription.supportsObserverTracking, or switched when it was + * later seen advertising it. Never cleared. */ observerTracking?: true; + /** + * Set with `observerTracking` by a switch after creation, until the history the connection + * recorded under the legacy protocol is imported: admission waits for that import. + */ + observerHistoryPending?: true; + /** * OBSOLETE: Before we had support for multiple gadgets per workspace, the binding name and * blueprint annotation information lived on the GatekeeperRecord. These properties continue diff --git a/packages/workshop-shared/src/gatekeeper.ts b/packages/workshop-shared/src/gatekeeper.ts index 0d44de1db6..c2d280a855 100644 --- a/packages/workshop-shared/src/gatekeeper.ts +++ b/packages/workshop-shared/src/gatekeeper.ts @@ -328,14 +328,15 @@ export type ResourceDescription = { * requirements recorded before it opted in; * - `Gatekeeper.verifyObserverAccess()` is implemented, so the Workshop can check collaborators' * accounts against those requirements; - * - `Gatekeeper.addObserver()` throws, so a Workshop still treating the connection as legacy - * refuses the registration rather than admitting anyone. + * - `Gatekeeper.addObserver()` throws, which is how a Workshop still treating the connection as + * legacy learns of the upgrade, and keeps it from admitting anyone it has not checked. * * `addObserver()` is then never called for this resource, and `removeObserver()` only to retire * its legacy registrations. Absence keeps the legacy protocol. A resource advertising support - * when it is connected is tracked from the start; one that starts advertising it later has its - * history imported once (see `Gatekeeper.getHistoricalSubresources()`). Tracking is never - * downgraded. + * when it is connected is tracked from the start. One that starts advertising it later is + * switched to tracking at its first `authorizeTrackedObservation()` or refused `addObserver()`, + * and its history is imported once while it stays in use (see + * `Gatekeeper.getHistoricalSubresources()`). Tracking is never downgraded. */ supportsObserverTracking?: true; @@ -1099,8 +1100,9 @@ export interface Gatekeeper extends DurableObject { * The gatekeeper should ideally permit observers who are mailing list members. * * A resource advertising `ResourceDescription.supportsObserverTracking` must throw instead: a - * redeploy can land between the Workshop's describe() of a legacy connection and this call, and - * throwing keeps it from admitting a collaborator without checking anything. + * Workshop treating the connection as legacy only describes it, and switches it to tracking, + * once this refuses, and throwing keeps it from admitting a collaborator without checking + * anything. */ addObserver(id: string, user: Fetcher): Promise; @@ -1124,13 +1126,13 @@ export interface Gatekeeper extends DurableObject { * IDs obey the same rules as there, and each page may hold at most 10,000 of them, which keeps * it within the RPC size limit. The complete export has no ID count cap, and may repeat an ID. * - * The Workshop calls this once per connection, when an existing one first shows that it - * advertises support: at its first `ObservationAuthorizer.authorizeTrackedObservation()`, or - * when a collaborator is admitted (the Workshop describes a legacy connection before calling its - * `addObserver()`), including after a pending creation is approved. It drains the cursor until - * `next()` returns `null`, and stores the result in the same commit that marks tracking - * initialized. It must therefore be read-only and repeatable: an import interrupted at any page - * calls it again and drains a new cursor from the start. A cursor that ends at once is valid. + * The Workshop calls this once per connection, after switching an existing one to tracking: at + * its first `ObservationAuthorizer.authorizeTrackedObservation()`, or when its `addObserver()` + * refuses a collaborator. Observations through the connection carry on meanwhile; admission + * waits for the import. The Workshop drains the cursor until `next()` returns `null` and stores + * the result in one commit. It must therefore be read-only and repeatable: an import + * interrupted at any page calls it again and drains a new cursor from the start. A cursor that + * ends at once is valid. * Reads made while a creation was pending were simulated and must not be exported. A gatekeeper * connected while already advertising support is never asked. */ @@ -1311,9 +1313,9 @@ export interface ObservationAuthorizer extends RpcTarget { /** * `authorizeObservation()` for a resource advertising * `ResourceDescription.supportsObserverTracking`, which must make every observation through - * this method. The first call through a connection made under the legacy protocol imports its - * history (see `Gatekeeper.getHistoricalSubresources()`) and is refused; a retry, through a new - * session, is checked. + * this method. The first call through a connection made under the legacy protocol switches it + * to tracking, starting the import of its history (see + * `Gatekeeper.getHistoricalSubresources()`), and is then checked like any other. * * Returning normally means the Workshop recorded `description.subresources` and confirmed that * every collaborator who can see the read may access them. A Workshop that predates observer diff --git a/plans/kernel-observer-tracking.md b/plans/kernel-observer-tracking.md index b7a97c96c7..1c3f90fbe0 100644 --- a/plans/kernel-observer-tracking.md +++ b/plans/kernel-observer-tracking.md @@ -25,22 +25,29 @@ mix both. an older Workshop, so the call fails closed whatever the deployment order or rollback, and the gatekeeper needs no acknowledgment to check. The cost: implementers of `ObservationAuthorizer` (test doubles included) add the method. -- **Upgrades are detected from calls already made, plus one description at admission.** Many - gatekeepers' `describe()` calls their provider, and every binding call opens a session, so - describing on every use would add provider traffic, latency and a failure mode to every call. - Only an upgraded gatekeeper calls `authorizeTrackedObservation()`, so a legacy connection's - first such call has it described and imported. Admission describes each legacy connection in - scope before its `addObserver()` (inside the registration lock, after the account prompt), - imports one that now advertises support, and refuses the binding if it can't be described. - That keeps upgrade detection out of `addObserver()`'s error handling, for one `describe()` per - legacy connection per admission. `addObserver()` still throws for an upgraded gatekeeper, so a - redeploy between the description and the call refuses rather than admits. -- **An import restarts the workspace, even with only the owner connected.** The import aborts the - connection's facet, and client capabilities (a `GatekeeperClient`) keep the stub they were - minted with. Restarting replaces them once, at the migration, instead of resolving the facet - again on every session open forever after. It restarts after a failed import too, since the - abort has already happened. The cost: everyone connected, the owner included, reconnects once - per import attempt, and the call that revealed the upgrade fails. +- **Upgrades are detected only from calls the upgraded gatekeeper makes.** Many gatekeepers' + `describe()` calls their provider with the owner's credentials, and every binding call opens a + session, so describing on every use, or before every admission, would add provider traffic, + latency and a failure mode that has nothing to do with the collaborator being checked: an owner + whose access lapsed would lock out collaborators whose own access still works. Only an upgraded + gatekeeper calls `authorizeTrackedObservation()`, and only an upgraded gatekeeper's + `addObserver()` must throw, so the Workshop describes a legacy connection only after one of + those, and switches it if the description advertises support. A refused `addObserver()` whose + connection can't be described, or doesn't advertise support, stays a refusal; nothing falls + back to admitting. +- **A switch is the cutover; the import is a backfill.** The switch marks the connection tracked + with its history pending, in one write: from then on the Workshop checks and records every + observation through it and refuses the legacy `authorizeObservation()`, so a read either was + authorized under the legacy protocol, and is the export's to cover, or is recorded here. The + import then appends the export beside whatever reads recorded meanwhile, and only admission, + which must not verify anyone against an incomplete history, waits for it. An earlier design + fenced the import instead: it aborted the facet, refused the read that revealed the upgrade, + and restarted the workspace so clients dropped their stubs to the aborted facet and legacy + sessions reopened. Neither is needed once nothing is aborted: sessions admitted under the + legacy protocol were verified against everything it authorized, and tracked reads check them + from the switch on. The cost: the export must cover reads the legacy protocol authorized that + were still in flight when the gatekeeper upgraded, and a collaborator opening the workspace + during the import waits for it. - **The kernel decides who is checked and when; the connection decides access.** The Workshop computes each check's observers and IDs and calls `Gatekeeper.verifyObserverAccess(observers, subresources)` on the connection that reported them, handing it each observer's verifier. @@ -67,7 +74,7 @@ mix both. lone surrogates); and the index stores the caller-to-context mapping's result per record, so changing that mapping needs a reindexing migration. - **Always import on upgrade.** An existing legacy connection that starts advertising support is - imported once (fenced, single-flight, repeatable until it commits). No kernel-side "did it read + imported once (single-flight, repeatable until it commits). No kernel-side "did it read anything" flag: the gatekeeper's export is authoritative, an empty export is cheap, and a flag would fail open for requirements recorded outside `authorizeObservation`. A connection created already advertising support starts tracked. Imported IDs land on the workspace context, since no From 939573ba70dc89fe58d5b2f03b7c40421c30e65f Mon Sep 17 00:00:00 2001 From: Max Peterson Date: Tue, 27 Oct 2026 23:26:18 -0700 Subject: [PATCH 3/3] simplify historical import api --- .agents/skills/write-gatekeeper/SKILL.md | 2 +- docs/observers.md | 4 +- .../gatekeeper-test/src/test-gatekeeper.ts | 26 ++-------- .../__tests__/observer-ledger.test.ts | 52 ++++++------------- .../workshop-backend/src/observer-ledger.ts | 38 +++++--------- packages/workshop-shared/src/gatekeeper.ts | 26 +++++----- plans/kernel-observer-tracking.md | 10 ++-- 7 files changed, 56 insertions(+), 102 deletions(-) diff --git a/.agents/skills/write-gatekeeper/SKILL.md b/.agents/skills/write-gatekeeper/SKILL.md index 428fd7ee16..07d713c192 100644 --- a/.agents/skills/write-gatekeeper/SKILL.md +++ b/.agents/skills/write-gatekeeper/SKILL.md @@ -284,7 +284,7 @@ Opt in by returning `supportsObserverTracking: true` from the resource's `descri 1. **Authorize every observation with `authorizeTrackedObservation()`**, hook deliveries included. Report as `subresources` the IDs of everything the disclosure depends on that might restrict who may see it (`[]` when nothing does), including the coarse requirement the read relies on (see the mapping above): nothing else checks it, since `addObserver()` no longer verifies anything. Make each ID carry everything your `verifyObserverAccess()` needs, and keep its meaning stable across versions; encoding a tuple avoids delimiter collisions (the integration fixture reports `JSON.stringify([resourceUrl, localId])`). If the call fails because the Workshop predates observer tracking, let it propagate: falling back to `authorizeObservation()` would disclose the read without checking anyone. 2. **Implement `verifyObserverAccess(observers, subresources)`.** Check each observer's verifier through your vendor's non-standard methods (as `addObserver()` does) and return the IDs of those whose account can read **every** ID; throw when access can't be determined. Check in bulk where the service allows; because one call covers a read's audience, the check may also read the resource's ACL once with the connection's own credentials and compare each observer's identity. Don't store the verifiers or cache verdicts by observer ID, since the same observer can come back with another account. -3. **Implement `getHistoricalSubresources()`** over the requirements your legacy tracking recorded, hook deliveries included, in the same form as `subresources`. Cover every read the legacy protocol authorized, including one still in flight when the new version deployed: the Workshop accepts tracked reads while it imports, so it checks nothing else from before the switch. A gatekeeper with no legacy history may return a cursor that ends at once. +3. **Implement `getHistoricalSubresources()`** over the requirements your legacy tracking recorded, hook deliveries included, in the same form as `subresources`. Cover every read the legacy protocol authorized, including one still in flight when the new version deployed: the Workshop accepts tracked reads while it imports, so it checks nothing else from before the switch. The export must fit in one RPC result (32 MiB): condense a larger history into coarser IDs, each of which your `verifyObserverAccess()` passes only for an observer who can read everything it replaces, and never truncate it. A gatekeeper with no legacy history returns `[]`. 4. **Make `addObserver()` throw** (any error): that refusal is how a Workshop still treating the connection as legacy learns of the upgrade during an admission. `removeObserver()` may stay a no-op. What changes for the gatekeeper: diff --git a/docs/observers.md b/docs/observers.md index c8da98fa80..e164e16958 100644 --- a/docs/observers.md +++ b/docs/observers.md @@ -966,8 +966,8 @@ documented on those members in `workshop-shared/src/gatekeeper.ts`, and A switch sets `observerTracking` and `observerHistoryPending` together: from then on every observation through the connection is checked and recorded by the Workshop, and the legacy `authorizeObservation()` is refused, so no read goes unchecked. Nothing is aborted or - restarted. The switch starts the history import in the background: single-flight, it drains - `getHistoricalSubresources()` to its end, then one transaction clears `observerHistoryPending` + restarted. The switch starts the history import in the background: single-flight, it reads + `getHistoricalSubresources()` in one call, then one transaction clears `observerHistoryPending` and, for a non-empty export, writes the workspace audit chunks. Reads through the connection carry on meanwhile, recording their own requirements beside the imported ones. Admission waits for the import before verifying anyone against the connection, and treats its completion as a diff --git a/packages/integration-tests/fixtures/gatekeeper-test/src/test-gatekeeper.ts b/packages/integration-tests/fixtures/gatekeeper-test/src/test-gatekeeper.ts index c7da973703..625449c85e 100644 --- a/packages/integration-tests/fixtures/gatekeeper-test/src/test-gatekeeper.ts +++ b/packages/integration-tests/fixtures/gatekeeper-test/src/test-gatekeeper.ts @@ -26,7 +26,7 @@ import { skipRpcValidation, validateRpc } from "capnweb-validate"; import { connectHandoffPageHtml, htmlResponse } from "@gadgets/gatekeeper-kit/connect-pages"; import type { AccountDescription, ActionDescription, ActionKind, AgentCatalog, ApprovalQueue, ConnectHandoff, - Cursor, Gatekeeper, GatekeeperConnectCallback, GatekeeperUser, GatekeeperUserVerifier, + Gatekeeper, GatekeeperConnectCallback, GatekeeperUser, GatekeeperUserVerifier, HookController, HookInitiator, HookTargetMetadata, ResourceDescription, ResourceConfiguratorFrame, SupportedResource, VendorDescription, } from "@gadgets/workshop-shared/gatekeeper"; @@ -770,24 +770,6 @@ const SET_VALUE_ACTION_KIND: ActionKind = { tag: "set-value", label: "Set value" */ const HISTORY_KEY = "observer-history"; -/** - * Pages a legacy binding's history for the Workshop's import one ID at a time, after an empty - * page such as a filtered walk can yield: the Workshop must drain it to `null`, not stop early. - */ -@validateRpc() -class HistoryCursor extends RpcTarget implements Cursor { - readonly #pages: string[][]; - - constructor(ids: string[]) { - super(); - this.#pages = [[], ...ids.map(id => [id])]; - } - - async next(): Promise { - return this.#pages.shift() ?? null; - } -} - /** Polls rather than parks a promise: the release arrives on another request to TestControl. */ async function waitForApplyRelease(state: DurableObjectStub, label: string) { for (const deadline = Date.now() + 30_000; !await state.isApplyReleased(label);) { @@ -839,13 +821,13 @@ export class TestGatekeeper } /** What this binding's legacy-mode reads and hook deliveries recorded, for the one-time import. */ - async getHistoricalSubresources(): Promise> { + async getHistoricalSubresources(): Promise { const state = control(this.ctx.exports); await state.recordHistoryExport(this.ctx.props.resourceUrl); - return new HistoryCursor([ + return [ ...this.ctx.storage.kv.get(HISTORY_KEY) ?? [], ...await state.getLegacyHookHistory(this.ctx.props.resourceUrl), - ]); + ]; } /** Admits each observer whose verifier reports none of `subresources` denied. */ diff --git a/packages/workshop-backend/__tests__/observer-ledger.test.ts b/packages/workshop-backend/__tests__/observer-ledger.test.ts index ffa3661d55..dd017f62d3 100644 --- a/packages/workshop-backend/__tests__/observer-ledger.test.ts +++ b/packages/workshop-backend/__tests__/observer-ledger.test.ts @@ -310,22 +310,13 @@ describe("byte-bounded observation persistence", () => { } }); -type HistoryCursor = { next(): Promise, [Symbol.dispose](): void }; - -// A historical export as a gatekeeper pages it: at most `pageSize` IDs per page, after an empty -// page that must not end the walk. -function historyCursor(ids: string[], pageSize = 10_000): HistoryCursor { - let pages: string[][] = [[]]; - for (let i = 0; i < ids.length; i += pageSize) pages.push(ids.slice(i, i + pageSize)); - return { next: async () => pages.shift() ?? null, [Symbol.dispose]() {} }; -} - // Facets: legacy ones advertise nothing; tracked ones advertise support and refuse addObserver(), -// as the contract requires. An export given as IDs is paged by historyCursor(). `verify` replaces -// verifyObserverAccess() (null removes it). +// as the contract requires. An export resolves no sooner than a real RPC, after a macrotask, so +// switchAndImport() joins the import the switch started. `verify` replaces verifyObserverAccess() +// (null removes it). function fakeFacets(impl: any, opts: { adds?: number[], - exports?: Map Promise>, + exports?: Map Promise>, advertise?: Set, verify?: VerifyObserverAccess | null, } = {}): void { @@ -338,8 +329,8 @@ function fakeFacets(impl: any, opts: { ...(tracked(id) ? { supportsObserverTracking: true } : {}), }), getHistoricalSubresources: async () => { - let history = await (opts.exports?.get(id)?.() ?? []); - return Array.isArray(history) ? historyCursor(history) : history; + await scheduler.wait(0); + return await opts.exports?.get(id)?.() ?? []; }, ...(opts.verify === null ? {} : { verifyObserverAccess: opts.verify ?? verifyEachObserver }), addObserver: async () => { @@ -1160,7 +1151,7 @@ describe("historical import", () => { }, getHistoricalSubresources: async () => { exports++; - return historyCursor(["restricted/history"]); + return ["restricted/history"]; }, addObserver: async () => { throw new Error("This connection is tracked by the Workshop."); }, }); @@ -1305,7 +1296,7 @@ describe("historical import", () => { expect(requirements(impl, "use").get(3)).toEqual(["p1", "p2"]); })); - it("imports a paged history larger than one observation and checks every verification batch", + it("imports a history larger than one observation and checks every verification batch", () => withWorld(async world => { let { impl } = world; let ids = Array.from({ length: 10_001 }, (_, i) => String(i)); @@ -1325,7 +1316,7 @@ describe("historical import", () => { expect(world.checks).toContainEqual({ profileId: "carol", ids: required.slice(10_000) }); expect(impl.storage.observers.get("carol")).toBeUndefined(); - // One observation is still held to the per-call limit the import paged around. + // One observation is still held to its own limit. await expect(impl.authorizeTrackedObservation(LEGACY, trackedObservation(ids), { from: "user" })).rejects.toThrow(/at most 10000/); expect(requirements(impl, "build").get(LEGACY)).toEqual(required); @@ -1334,17 +1325,16 @@ describe("historical import", () => { expect(impl.storage.observers.get("carol").accountChoices).toEqual({ [LEGACY]: 10 }); })); - it("refuses an export page of more than 10,000 IDs and leaves the history pending", () => + it("refuses an export with an invalid ID and leaves the history pending", () => withWorld(async world => { let { impl } = world; seedLegacy(impl); - let ids = Array.from({ length: 10_001 }, (_, i) => String(i)); fakeFacets(impl, { advertise: new Set([LEGACY]), - exports: new Map([[LEGACY, async () => historyCursor(ids, ids.length)]]), + exports: new Map([[LEGACY, async () => ["doc:ok", "doc:\uD800"]]]), }); - await expect(switchAndImport(impl)).rejects.toThrow(/at most 10000/); + await expect(switchAndImport(impl)).rejects.toThrow(/well-formed/); expect(impl.storage.gatekeepers.get(LEGACY)) .toMatchObject({ observerTracking: true, observerHistoryPending: true }); expect(rows(impl, WORKSPACE)).toEqual([]); @@ -1353,9 +1343,7 @@ describe("historical import", () => { // The pending history is durable: a failure, or the object resetting, leaves it for the next // admission to export again. - it.each(["the export call", "a later page"])( - "retries after a failure before the commit, and never after it (failing at %s)", - async failure => { + it("retries after a failure before the commit, and never after it", async () => { let name = `observer-ledger-import-${++doCounter}`; let exports = 0; let fail = true; @@ -1363,14 +1351,8 @@ describe("historical import", () => { advertise: new Set([3]), exports: new Map([[3, async () => { exports++; - if (!fail) return ["p1"]; - if (failure === "the export call") throw new Error("export interrupted"); - // The first page arrives; the walk fails before the export ends. - let pages = [["p0"]]; - return { - next: async () => pages.shift() ?? Promise.reject(new Error("export interrupted")), - [Symbol.dispose]() {}, - }; + if (fail) throw new Error("export interrupted"); + return ["p1"]; }]]), }); await inOverseer(name, async ({ impl, restarts }) => { @@ -1553,7 +1535,7 @@ describe("historical import", () => { tsType: "N", supportsObserverTracking: true, }); impl.getGatekeeperFacet = async () => ({ - describe, getHistoricalSubresources: async () => { exports++; return historyCursor(["old"]); }, + describe, getHistoricalSubresources: async () => { exports++; return ["old"]; }, verifyObserverAccess: verifyEachObserver, }); @@ -1580,7 +1562,7 @@ describe("historical import", () => { title: real ? "Real" : "Simulated", snippet: "", suggestedBindingName: "N", tsType: "N", supportsObserverTracking: true, }), - getHistoricalSubresources: async () => { exports++; return historyCursor([]); }, + getHistoricalSubresources: async () => { exports++; return []; }, verifyObserverAccess: verifyEachObserver, applyCreation: async () => { real = true; diff --git a/packages/workshop-backend/src/observer-ledger.ts b/packages/workshop-backend/src/observer-ledger.ts index cfd1dc1a4d..40012f0492 100644 --- a/packages/workshop-backend/src/observer-ledger.ts +++ b/packages/workshop-backend/src/observer-ledger.ts @@ -31,9 +31,8 @@ import type { UserDurableObject } from "./user"; import type { WorkshopLogger } from "./observability"; /** - * Most subresource IDs one observation, one page of a historical export, or one - * `verifyObserverAccess()` call may carry; accumulated history has no count cap, so admission - * checks it in batches of this size. + * Most subresource IDs one observation or one `verifyObserverAccess()` call may carry; accumulated + * history has no count cap, so admission checks it in batches of this size. */ const MAX_SUBRESOURCES = 10_000; @@ -158,18 +157,18 @@ export async function requireVerifier(user: DurableObjectStub } /** - * Validates one observation's, or one export page's, reported subresource IDs (the RPC layer has - * checked an observation's are strings, but not an export's) and deduplicates them without - * interpreting them. Each must be well-formed Unicode: requirement keys hold them, and storage - * would replace a lone surrogate, merging distinct IDs and checking a different one at admission. + * Validates one observation's, or a historical export's, reported subresource IDs (the RPC layer + * has checked an observation's are strings, but not an export's), at most `max` of them, and + * deduplicates them without interpreting them. Each must be well-formed Unicode: requirement keys + * hold them, and storage would replace a lone surrogate, merging distinct IDs and checking a + * different one at admission. */ -export function normalizeSubresources(ids: unknown): string[] { +export function normalizeSubresources(ids: unknown, max = Infinity): string[] { if (!Array.isArray(ids) || ids.some(id => typeof id !== "string")) { throw new Error("Subresources must be a list of strings."); } - if (ids.length > MAX_SUBRESOURCES) { - throw new Error( - `An observation or export page may report at most ${MAX_SUBRESOURCES} subresources.`); + if (ids.length > max) { + throw new Error(`An observation may report at most ${max} subresources.`); } for (let id of ids) { if (id.length === 0 || id.length > MAX_SUBRESOURCE_LENGTH) { @@ -181,18 +180,6 @@ export function normalizeSubresources(ids: unknown): string[] { return [...new Set(ids)]; } -// Drains a connection's historical export to its end: only `null` ends it, since a page may be -// empty mid-walk. Each page is held to an observation's limits. -async function readHistoricalExport(facet: TrackingGatekeeper): Promise { - using cursor = await facet.getHistoricalSubresources(); - let history = new Set(); - for (let page = await cursor.next(); page !== null; page = await cursor.next()) { - let ids = normalizeSubresources(page); - for (let id of ids) history.add(id); - } - return [...history]; -} - /** The observer ledger of one workspace. See the module comment. */ export class ObserverLedger { // In-flight switches to tracking, one per connection (see switchToTracking()). @@ -298,7 +285,8 @@ export class ObserverLedger { async #import(gatekeeperId: WorkpieceId): Promise { let facet = await this.host.getGatekeeperFacet(gatekeeperId) as TrackingGatekeeper; - let subresources = await readHistoricalExport(facet); + // Unlike an observation, the export has no count cap: one RPC result bounds it. + let subresources = normalizeSubresources(await facet.getHistoricalSubresources()); let record = this.#storage.gatekeepers.get(gatekeeperId); if (!record) throw new Error("This connection has been removed from the workspace."); @@ -434,7 +422,7 @@ export class ObserverLedger { if (context === undefined) { throw new Error("This observation has no attributable chat or gadget."); } - let subresources = normalizeSubresources(description.subresources); + let subresources = normalizeSubresources(description.subresources, MAX_SUBRESOURCES); if (subresources.length === 0) { this.#assertLive(gatekeeperId, caller, executionId); return commit(subresources); diff --git a/packages/workshop-shared/src/gatekeeper.ts b/packages/workshop-shared/src/gatekeeper.ts index c2d280a855..e7f228c349 100644 --- a/packages/workshop-shared/src/gatekeeper.ts +++ b/packages/workshop-shared/src/gatekeeper.ts @@ -1117,26 +1117,26 @@ export interface Gatekeeper extends DurableObject { removeObserver(id: string): Promise; /** - * For a gatekeeper advertising `ResourceDescription.supportsObserverTracking`: a cursor over - * every subresource ID this gatekeeper's past observations, hook deliveries included, still - * oblige observers to be able to read -- completed and pending, plus any binding-wide or - * owner-only requirement it records itself -- in the same form as - * `TrackedObservationDescription.subresources`, which `verifyObserverAccess()` can check. Not an - * audit log or listing of today's resources. - * IDs obey the same rules as there, and each page may hold at most 10,000 of them, which keeps - * it within the RPC size limit. The complete export has no ID count cap, and may repeat an ID. + * For a gatekeeper advertising `ResourceDescription.supportsObserverTracking`: every subresource + * ID this gatekeeper's past observations, hook deliveries included, still oblige observers to be + * able to read -- completed and pending, plus any binding-wide or owner-only requirement it + * records itself -- in the same form as `TrackedObservationDescription.subresources`, which + * `verifyObserverAccess()` can check. Not an audit log or listing of today's resources. + * IDs obey the same rules as there, except that the export has no count cap of its own: it must + * fit in one RPC result (32 MiB serialized), and may repeat an ID. A larger history must be + * condensed into coarser IDs, each of which `verifyObserverAccess()` passes only for an observer + * who can read every subresource it replaces; it must never be truncated. * * The Workshop calls this once per connection, after switching an existing one to tracking: at * its first `ObservationAuthorizer.authorizeTrackedObservation()`, or when its `addObserver()` * refuses a collaborator. Observations through the connection carry on meanwhile; admission - * waits for the import. The Workshop drains the cursor until `next()` returns `null` and stores - * the result in one commit. It must therefore be read-only and repeatable: an import - * interrupted at any page calls it again and drains a new cursor from the start. A cursor that - * ends at once is valid. + * waits for the import. The Workshop stores the result in one commit, and refuses an export that + * breaks these rules, leaving the history pending. It must therefore be read-only and + * repeatable: an import that fails or is refused calls it again. `[]` is valid. * Reads made while a creation was pending were simulated and must not be exported. A gatekeeper * connected while already advertising support is never asked. */ - getHistoricalSubresources?(): Promise>; + getHistoricalSubresources?(): Promise; /** * For a gatekeeper advertising `ResourceDescription.supportsObserverTracking`: which of diff --git a/plans/kernel-observer-tracking.md b/plans/kernel-observer-tracking.md index 1c3f90fbe0..7b8f821f3f 100644 --- a/plans/kernel-observer-tracking.md +++ b/plans/kernel-observer-tracking.md @@ -79,10 +79,12 @@ mix both. would fail open for requirements recorded outside `authorizeObservation`. A connection created already advertising support starts tracked. Imported IDs land on the workspace context, since no narrower attribution exists. -- **The export is a cursor.** History has no count cap, and one RPC result is limited to 32 MiB, - so `getHistoricalSubresources()` returns the existing `Cursor` with pages capped at an - observation's 10,000 IDs. The Workshop drains it to `null` and commits the union in one - transaction, keeping import all-or-nothing; the union is held in memory until then. +- **The export is one RPC result.** `getHistoricalSubresources()` returns every ID in one array, + which must fit one RPC result (32 MiB serialized), so the Workshop validates the whole export + and commits it in one transaction, keeping import all-or-nothing. A gatekeeper whose history is + larger condenses it into coarser IDs, each checked as strictly as everything it replaces, + rather than truncating it. A paged cursor was dropped: it lifted the bound only to hold an + unbounded union in memory until that commit. - **Disconnect drops checks.** A removed connection's audit records stay but are ignored, matching the legacy protocol's behavior on removal (`docs/observers.md` edge case 8). - **Same verification scope as the legacy protocol.** A tracked connection enters "build" scope at