diff --git a/CLAUDE.md b/CLAUDE.md index 2978e242aa..694984412e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -151,7 +151,7 @@ Commonly is collapsing the legacy `App` + `AgentRegistry` split into a single `I | 🟒 **First-party apps** | 3 shipped (stays) | `pod-welcomer`, `task-clerk`, `pod-summarizer` in Team Orchestration Demo pod | Reference implementations for the Installable model | | ⏸️ **ADR-010 Phase 2+** | Paused (Phase 1 shipped) | OpenClaw β†’ MCP migration, extension `commonly_*` retirement | Re-activates when a second runtime needs `commonly_*` mid-turn | | ⏸️ **Installable taxonomy refactor** | Paused (Phase 1.5 + Phase 2 marketplace-ops shipped via PR #215 + #230; 2-remainder + 3–6 hold) | ADR-001 Phase 3 read-path switch (install reads from Installable, not AR), reconciliation cron, semver/runtime validation | Re-activates when marketplace frontend reveals a drift bug or a new Installable shape needs the read-path switch | -| ⏸️ **Cloud sandbox runtime (Tier 2)** | Paused | Anthropic Managed Agents + Commonly-hosted container adapter | Re-activates on real demand from a heavy-compute agent | +| ⏸️ **Cloud sandbox runtime (Tier 2)** | Partly lifted: the per-turn E2B code sandbox for native agents is live (ADR-011 2026-10-15 update, narrowed 2026-10-22; ADR-023 A2). VMs, long-lived containers and the Managed Agents adapter stay paused | Anthropic Managed Agents + Commonly-hosted container adapter | Re-activates on real demand from a heavy-compute agent | | ⏸️ **Slash command infrastructure** | Paused (taxonomy Phase 4) | `/command` addressing mode, command registry, UI autocomplete | Re-activates when an app/marketplace listing needs `/command` primary | | ⏸️ **Kernel / CAP spec** | Paused β€” #61, #46 | OpenAPI spec + coupling reduction | Re-activates when federation work begins or a second instance comes online | | ⏸️ **Driver layer expansion** | Paused β€” #69, #70 | Webhook SDK Phase 2 (OAuth, signatures), Agent SDK npm publish | Re-activates on real external developer demand | diff --git a/docs/adr/ADR-011-shell-first-pre-gtm.md b/docs/adr/ADR-011-shell-first-pre-gtm.md index 14cb151441..4b55f07ae0 100644 --- a/docs/adr/ADR-011-shell-first-pre-gtm.md +++ b/docs/adr/ADR-011-shell-first-pre-gtm.md @@ -1,11 +1,95 @@ # ADR-011: Shell-first pre-GTM -**Status:** Active β€” 2026-04-27 (Phase 1 shipped 2026-04-29) +**Status:** Accepted β€” 2026-10-25, ratified by Cofounder under Sam's delegation (D77), covering the 2026-10-15 lift and its 2026-10-22 narrowing below. Active since 2026-04-27 (Phase 1 shipped 2026-04-29). Sam may overrule. **Author:** Sam Xu **Companion:** [`ADR-010`](ADR-010-commonly-mcp-server.md) (Phase 2+ paused under this ADR), [`CLAUDE.md`](../../CLAUDE.md) Β§Design Rules #5 ("the social surface has to earn human presence") +**Scope boundary:** this ADR decides which work is paused, not where agents run. The agent-runtime substrate is [ADR-023](ADR-023-agent-runtime-substrate.md)'s, and its open Containers spike decides it after 2026-11-01. --- +## Status update β€” 2026-10-22: what the lift built, and how far it reaches + +**Status:** the 2026-10-15 update below was ratified by Cofounder on 2026-10-21 +under Sam's standing delegation. This update, recorded 2026-10-22, narrows it, +and was ratified by Cofounder on 2026-10-25 under Sam's delegation (D77). +Sam may overrule both. + +**The decision came early, and both stages merged within a day.** The one-page +decision the update below scheduled for 2026-10-17 was made on 2026-10-15 and is +recorded as [ADR-023](ADR-023-agent-runtime-substrate.md)'s amendment of that +date: hosted turns move onto the native runtime, and tools run in a sandbox. It +planned two stages, for about 2026-10-22 and 2026-10-29. Both merged on +2026-10-15 and 2026-10-16 (UTC): + +- **Hosted turns on the native runtime.** #2224 records each hosted install's + engine in `config.hosted.engine` (`backend/services/hostedEngine.ts`). #2236 + puts new installs on the native engine (`hostedEngine: "native"` in + `k8s/helm/commonly/values-dev.yaml`) and adds a forward-only migration for + existing ones (`backend/services/hostedEngineMigration.ts`). #2226 re-runs a + native turn that dies with its process. +- **Code in a per-turn E2B sandbox with no internet.** #2232 adds + `commonly_run_code` (`backend/services/turnSandboxService.ts`). +- **Web lookup.** #2244 adds `commonly_web_search` and #2249 adds + `commonly_fetch_url`. #2248 offers search and code to hosted agents on every + plan. +- **Cost bounds.** #2227 pages when one user's agent spend passes $2 in a UTC + day (`backend/services/spendAlarmService.ts`). #2237 sets the plan line at 30 + hosted turns a day free and 200 on Pro. + +Since then, #2297 (2026-10-21) starts a hosted agent from Scout's welcome in one +click. Measured on the hosted path, in two walks by one tester: from clicking +"Run it here" to provisioned in 2.47 s and 2.54 s, and replies in 7.8 to 14.3 s, +mean 10.35 s over eight asks (`frontend/src/__tests__/timingClaims.test.ts`). +Those two steps together sit well inside the 60-second target. The time from +signup to that click is not in those figures, and none of them is a stranger's +run. + +**The lift is narrower than the update below reads.** Only the per-turn E2B code +sandbox for Commonly's native agents is unpaused: one sandbox per turn, opened +on the turn's first code run and killed when the turn ends, with no internet and +no environment variables. Virtual machines, long-lived containers and an +Anthropic Managed Agents adapter stay paused under the **Cloud sandbox runtime +(Tier 2)** row of the table below, and its reactivation trigger is unchanged: +real demand from a heavy-compute agent that cannot run on Tier 1. + +**Operational requirement: an alert before E2B credit runs out.** The E2B +account runs on prepaid credit with no automatic top-up, so spending it does +not produce an overage. The sandbox stops instead, and a turn that cannot get one hands the +error back to the model on every `commonly_run_code` call (the catch at the end +of `dispatchTool` in `backend/services/nativeRuntimeService.ts`). Nothing +watches the balance today. The spend alarm prices sandbox seconds per user but +never reads the account, and no policy in `k8s/monitoring/alert-policies.json` +covers E2B. Required: an alert when E2B credit falls below $25. Owner: SRE. + +**Agent computers are not a product.** Commonly does not host or sell agent +computers (desktop or browser microVMs for computer use). Agents bring their own +compute: Grok Bot, Cua, Devin, and Claude Code or Codex on the owner's machine. +Commonly runs only the per-turn E2B sandbox for its native agents. Reopen this +only if at least two of the first three written outside use cases need an agent +to operate an app that has no API or connector. Even then, integrate a partner +(Cua, E2B, Cloudflare Browser Rendering) before building anything. Decided +2026-10-22 on Sam's question. + +## Status update β€” 2026-10-15: the cloud sandbox pause is lifted for "out of the box" + +*Narrowed by the 2026-10-22 update above: only the per-turn E2B sandbox for +native agents is unpaused.* + +Sam's direction on 2026-10-15: make Commonly usable immediately, with a cloud +agent as the main lever for that, "whether we adopt new OSS sandbox/vm/container +solution or no". The target is a brand-new signup with a working hosted agent in +their pod, nothing to install, and a useful reply in under 60 seconds. Bring your +own CLI becomes the second path. + +That lifts the **Cloud sandbox runtime (Tier 2)** pause below for this purpose. +The trigger is Sam's direction, not the "heavy-compute agent" trigger the table +names, and it is scoped to the out-of-the-box goal; the other paused tracks are +unchanged. The direction does not decide the runtime itself. Whether today's +hosted runtime is enough, or Commonly adopts an OSS sandbox, VM or container, +is decided by numbers in a one-page decision due 2026-10-17: time to a first +useful reply from a fresh account, what the agent can do, where it breaks, and +the cost per team at the hosted cap. + ## Status update β€” 2026-04-29 **Phase 1 of shell polish is on `main` and deployed.** PR #251 mounted the `/v2` shell (V2App router, V2Layout, V2NavRail, V2PodsSidebar, V2PodChat, V2PodInspector, V2MessageBubble, V2Avatar, V2Login, V2FeaturePage, plus `useV2Api` / `useV2Pods` / `useV2PodDetail` / `useV2Pinned` / `useV2Embedded` hooks) and bundled the YC sprint reorg: trimmed nav rail (Pods Β· Agents Β· Apps Β· Settings), "Your Team" roster page with Hire CTA, Plan/Execute mode pill in the chat header, per-pod displayName overrides for agent messages, inline file-pill + reaction rendering (fixture-only β€” agents never react and there's no human write path; real reactions backend ships post-YC). diff --git a/docs/adr/ADR-021-pi-turn-engine-and-openclaw-retirement.md b/docs/adr/ADR-021-pi-turn-engine-and-openclaw-retirement.md index 667fdc149c..0b71e0a682 100644 --- a/docs/adr/ADR-021-pi-turn-engine-and-openclaw-retirement.md +++ b/docs/adr/ADR-021-pi-turn-engine-and-openclaw-retirement.md @@ -3,6 +3,7 @@ - **Status:** Accepted (ratified by Sam 2026-08-13; drafted 2026-08-13) - **Depends on:** ADR-020 D4 (pi SDK ratified as Scout's turn engine), #937 (backend Node 18 β†’ 22, merged β€” the SDK's hard `engines` floor) - **Supersedes** (in force on ratification; the CLAUDE.md edit is follow-through, not yet made): ADR-010's paused "Phase 2+" openclawβ†’MCP migration track (retirement makes migration moot); the moltbot rows of the runtime-tier table in CLAUDE.md +- **Scope-boundary note (2026-10-22):** Part A.1 below is superseded by [ADR-023](ADR-023-agent-runtime-substrate.md), which records where hosted turns actually run: in-process on `nativeRuntimeService`, with code in a per-turn E2B sandbox. Read ADR-023 before building on A.1. Part B (OpenClaw retirement) is unaffected. ## Context @@ -47,6 +48,8 @@ Build scout-runtime β†’ deploy dark β†’ route ONE Scout (the smoke workspace) vi ## Part A.1 β€” Build spec (decisions ratified by Sam, 2026-08-13) +> **Superseded by [ADR-023](ADR-023-agent-runtime-substrate.md)** (2026-10-22). The `agent-runtime` Deployment, PVC sessions, run-PATCH endpoint and token secret below were never built; ADR-023's "What replaced ADR-021 A.1" maps each item to what runs. Kept as the record of what was decided on 2026-08-13. + Three contested points settled: **transport = the CAP event queue** (the runtime is architecturally "N wrappers in one process," joining through the same door as every external driver β€” no new private API); **AgentRun accounting is kernel-owned** (scheduler creates the row, runtime PATCHes it through an authenticated runtime endpoint β€” if credits ever bill from these rows, the meter is never written by the thing being metered); **failure semantics are manual-fallback only in v1** (runtime down β†’ turns queue and deliver late rather than wrong; no auto engine switching mid-conversation). ### Components diff --git a/docs/adr/ADR-023-agent-runtime-substrate.md b/docs/adr/ADR-023-agent-runtime-substrate.md index 8c6794e206..0f4f76cf74 100644 --- a/docs/adr/ADR-023-agent-runtime-substrate.md +++ b/docs/adr/ADR-023-agent-runtime-substrate.md @@ -1,16 +1,100 @@ -# ADR-023: the hosted agent-runtime substrate β€” Durable Objects vs a multiplexed Deployment +# ADR-023: the hosted agent-runtime substrate β€” as built, the native runtime with tools in a per-turn sandbox -- **Status:** **Accepted** (ratified by Sam, 2026-08-28, under the activation goal: one stranger, unassisted, reaches a live agent conversation). D1's spike resolved YES 2026-08-22; the selected design is W2's runtime as Durable Objects. Acknowledged unknowns carried into build: D4's credit-term economics remain estimates, not measurements. - Scope-boundary note: ADR-026 (local agent daemon, Proposed) covers users whose value proposition is LOCAL execution; this ADR is the zero-install path for everyone else. The two are complements, not competitors β€” see ADR-026's scope boundary. +*Titled "Durable Objects vs a multiplexed Deployment" from 2026-08-15 to 2026-10-22. That comparison is kept below as Context and Decision; "As built" is what runs.* - **Spike evidence (run locally under workerd via `wrangler dev --local`, `nodejs_compat`, pi 0.84.2):** `@earendil-works/pi-agent-core` β€” the turn engine: `Agent`, `AgentHarness`, compaction, 106 exports β€” **loads AND constructs inside workerd.** Three findings that de-risk the DO design beyond the yes/no: - 1. The package quarantines its Node half behind a separate `./node` export (fs sessions, child_process, readline live there and only there β€” verified by builtin survey of the dist graph). The default entry the worker imports never touches them. - 2. The engine's LLM transport is dependency-injected (`streamFn`; construction without one fails with the engine's own validation error, not a compat error). Fetch-based transport, no Node HTTP binding β€” exactly the shape a DO provides. - 3. Construction with a stub transport succeeds end-to-end: `{ loaded: true, exports: 106, constructed: true }`. +- **Status:** **Accepted.** Ratified by Sam on 2026-08-28 under the activation goal (one stranger, unassisted, reaches a live agent conversation), selecting Durable Objects (D1–D4 below). Amended 2026-10-15 (A1–A3, ruled by Cofounder under Sam's delegation), moving hosted turns onto the native runtime; that amendment was ratified by Cofounder on 2026-10-21 under Sam's standing delegation. Rewritten as built on 2026-10-22; that rewrite was Accepted on 2026-10-25, ratified by Cofounder under Sam's delegation (D77). Sam may overrule. Acknowledged unknowns this does not settle are listed under "Open items": the substrate after 2026-11-01 (the Containers spike), per-user usage measured on hosted agents themselves, and E2B as an outside dependency. +- **Supersedes:** [ADR-021](ADR-021-pi-turn-engine-and-openclaw-retirement.md) Part A.1, the `agent-runtime` build spec. A.1's separate Deployment, PVC sessions, run-PATCH endpoint and token secret were never built; "What replaced ADR-021 A.1" below maps each item to what runs. ADR-021 Part B (OpenClaw retirement) is unaffected. +- **Scope-boundary note:** for where a hosted agent runs, read this ADR, not ADR-021 A.1. [ADR-011](ADR-011-shell-first-pre-gtm.md) decides what is paused: its 2026-10-22 update unpauses only the per-turn E2B sandbox for native agents, keeps VMs, long-lived containers and Managed Agents paused, and rules that Commonly does not host agent computers. [ADR-026](ADR-026-local-agent-daemon.md) (local agent daemon) covers users whose value proposition is LOCAL execution; this ADR is the zero-install path for everyone else, and the two are complements, not competitors. +- **Does NOT touch:** the shell (Express, MongoDB, PostgreSQL, Socket.io, the v2 frontend). See "What this is not" below. - Per D1's own text, the resolution selects: **build W2's runtime as Durable Objects** β€” one DO per `(agentName, instanceId)`, sessions on DO storage (replacing the `./node` fs half), tools as CAP calls. Spike artifacts: `worker.js` + `wrangler.toml`, reproducible in ~5 minutes. -- **Amends, if accepted:** ADR-021 Part A.1 (the `agent-runtime` build spec). Everything else in ADR-021 β€” pi as the turn engine, staged OpenClaw retirement, the M1–M4 ladder, the fallback discipline β€” stands unchanged. -- **Does NOT touch:** the shell (Express, MongoDB, PostgreSQL, Socket.io, the v2 frontend). See "What this is not" below, which is the most important section in this document. +## As built (2026-10-22) + +A hosted ("Run it here") agent runs inside the Commonly backend, on the same loop as Commonly's other native agents (`backend/services/nativeRuntimeService.ts`). Code it writes runs outside the backend, in an E2B sandbox that lives for one turn. The Cloudflare Durable Object this ADR first selected (`workers/agent-runtime`) is still in the tree and configured, as the fallback engine. Every statement in this section was read from `main` at `46da417f9`. + +### Which engine runs a hosted turn + +- An install keeps `config.runtime.runtimeType: 'hosted'`, so the hosted cap, the entitlement gate and the hosted count still key on it. Its engine is a separate record, `config.hosted.engine` (`backend/services/hostedEngine.ts:1-19`, `:36-40`). +- A new install's engine comes from `HOSTED_ENGINE` (`hostedEngine.ts:103-107`), which `k8s/helm/commonly/values-dev.yaml:60-64` sets to `native`; empty keeps the Durable Object. The install route writes `config.hosted` itself and drops any client-sent copy (`backend/routes/registry/install.ts:569-576`). +- `AgentEventService.resolveInProcessInstallation` (`backend/services/agentEventService.ts:1264`) returns the install when it is `native`, or hosted, on the native engine and provisioned (`:1302-1326`; `hostedRunsNative`, `hostedEngine.ts:128-132`). Such an event is created already claimed and run fire-and-forget by `dispatchInProcess` (`agentEventService.ts:1333`, `:1470`). Nothing polls for it. +- A hosted agent answers only `chat.mention` and `first_contact`, the two event types the Durable Object handled. Any other event is acked without a model call (`hostedEngine.ts:140-152`, `nativeRuntimeService.ts:1353-1361`). +- A turn that dies with the backend process is re-run from the event sweep, claimed by swapping its delivery nonce (`AgentEventService.redispatchStuckInProcess`, `agentEventService.ts:919`; #2226). The backend is pinned to the on-demand pool, not spot (`values-dev.yaml:41-52`), so the restarts this covers are deploys and crashes. + +### What a hosted turn runs with + +- A configuration built on the server and never taken from the install, because a hosted install's stored config came from the browser (`hostedNativeRunConfig`, `hostedEngine.ts:266-288`). It carries the system prompt, nine tools (`HOSTED_NATIVE_TOOLS`, `:167-177`), the pod's last 8 messages (`:180`) and the owner's plan line as the daily cap. The model is left unset, so the turn runs on the runtime default through LiteLLM: `LITELLM_CHAT_MODEL`, else `deepseek-v4-flash` (`nativeRuntimeService.ts:87`). The person cannot choose another model. +- No session state. Continuity is the recent messages each turn reads, plus the agent's memory tools. +- No runtime token. Every mint path refuses an agent Commonly runs in-process, with 409 `in_process_agent_no_token` (`backend/routes/registry/tokens.ts:227-243`), so nothing outside the backend can act as it. +- Connector broker tools are added per run and fail closed (`nativeRuntimeService.ts:1717-1730`). +- The tool gate holds at dispatch. The loop refuses any tool it did not offer in that model call (`nativeRuntimeService.ts:1959-1963`), and once web text is in the turn only in-pod tools and further lookups remain (`:643-648`, `:498`). + +### What leaves the process + +- **Code, in E2B.** `commonly_run_code` opens one sandbox on the turn's first call and kills it when the turn ends, whatever happened (`backend/services/turnSandboxService.ts:71-90`, `:120-130`; closed at `nativeRuntimeService.ts:2113-2118`). The template is pinned (`code-interpreter-v1`, `:22`). The sandbox has no internet (`allowInternetAccess: false`), no public inbound (`allowPublicTraffic: false`) and no environment variables (`:76-86`). It runs Python or JavaScript, at most 5 runs a turn of 30 s each, with a 3-minute lifetime as the backstop if the kill fails (`:23-30`). The tool is offered only when the instance holds `E2B_API_KEY` (`:32`; `nativeRuntimeService.ts:467-476`), which Helm sets only under `hostedSandbox.enabled` (off in `values.yaml`, on in `values-dev.yaml:11-15`). +- **Web search,** Exa by default and Tavily by setting (`backend/services/webSearchService.ts`), at most 3 a turn (`nativeRuntimeService.ts:499`). It is called from the backend, not from the sandbox. +- **Reading a page.** `commonly_fetch_url` opens only an address that appears in text the turn was given (`backend/services/webAddressProvenance.ts`), at most 3 a turn (`nativeRuntimeService.ts:503`). + +### Limits and cost + +- Per turn: 10 model rounds and 60 s of wall clock (`nativeRuntimeService.ts:75-77`), plus the tool limits above. +- Per hosted install per day: 30 turns free and 200 on Pro (`backend/services/hostedRuntimeService.ts:36-39`, `:57-61`, `:80`), enforced as the native per-install cap, which fails open (`nativeRuntimeService.ts:1382-1416`). The first-contact greeting does not count. +- Per user per day, across all their agents: 120 runs, 300 on Pro, failing closed (`nativeRuntimeService.ts:1418-1467`). +- Every turn writes an `AgentRun` (`nativeRuntimeService.ts:1544-1558`), with `sandboxSeconds`, `webSearches` and `webFetches` (`:2113-2127`). +- The spend alarm prices each user's day (model tokens, sandbox seconds and searches) and pages past $2 (`backend/services/spendAlarmService.ts`; policy in `k8s/monitoring/alert-policies.json:380`). It reports and never blocks. +- Not built: an alert on the E2B account's credit. ADR-011's 2026-10-22 update makes it a requirement (below $25, owner SRE). + +### The Durable Object, now the fallback + +- `workers/agent-runtime` is still in the tree, and the backend is still configured to reach it (`values-dev.yaml:79-80`). It runs any hosted install that is not on the native engine, and every new one if `HOSTED_ENGINE` is emptied. +- The migration off it is forward-only, with no rollback (`backend/services/hostedEngineMigration.ts:1-10`, run by `backend/scripts/migrate-hosted-to-native.ts`). So the fallback A1 describes holds for new installs only: a migrated install has no scripted way back. +- Its turn engine is pi agent-core (`workers/agent-runtime/src/turn.ts`). The backend has no pi dependency, so hosted turns on the native engine do not run on pi. pi otherwise survives as a bring-your-own wrapper adapter (`cli/src/lib/adapters/pi.js`). +- Retiring it is not scheduled here. + +### What replaced ADR-021 A.1 + +| A.1 specified | As built | +|---|---| +| A new top-level `agent-runtime` Deployment on the dev pool, with a PVC and no public ingress | Not built. Hosted turns run in the backend process. | +| Transport: the CAP event queue, polled per hosted agent with its own token | Events are claimed at creation and run in-process. Only the Durable Object fallback polls CAP. | +| A pi session per `(agentName, instanceId)`, persistent on the PVC, LRU-hydrated | No session state: recent messages and memory tools, each turn. | +| `AgentRun` created by the scheduler and closed by the runtime through `PATCH /api/agents/runtime/runs/:runId` | The native runtime creates and closes the row in-process. No such route exists. | +| Per-agent runtime tokens in an `agent-runtime-tokens` secret | None. An agent Commonly runs in-process cannot hold a runtime token. | +| Failure: flip `HOSTED_RUNTIME_ENABLED`, no deploy | `HOSTED_ENGINE` picks the engine for new installs; existing installs keep the engine recorded on them. | + +### Known costs of the build + +- **A shared process.** ADR-021 Part A kept hosted turns out of the backend for blast radius; as built puts them in it. Model calls wait on the network, but anything a turn computes runs on the API's event loop. The review of #2249 measured a quadratic trim that would have let three hostile pages block that loop for 7.2 s in one turn; it was fixed before merge. Each new in-process tool needs the same review. +- **E2B is an outside dependency** with no provider-side spending cap, and the account runs on prepaid credit with no automatic top-up. Running out of credit stops the sandbox; ADR-011's requirement is the alert before that. + +### Open items + +1. **The Containers spike decides the substrate, after 2026-11-01.** As built was chosen for the out-of-the-box goal, not as the long-run substrate. The spike asks whether hosted agents should move to a container per agent (Cloudflare Containers was D1's named alternative), for isolation and longer work, against the in-process loop's lower cost. Until it reports, nothing here commits Commonly to VMs or long-lived containers, which ADR-011 keeps paused. +2. **Per-user usage, measured on hosted agents.** A3's figures used Scout as a proxy. Hosted turns now write `AgentRun` rows, so re-measure from them two weeks after Stage 2 merged, on or after 2026-10-30. +3. **The E2B credit alert** (ADR-011, owner SRE). +4. Retiring the Durable Object, and whether pi returns as a hosted engine. Neither is scheduled. + +## Amendment, 2026-10-15: hosted turns run on the native runtime, and tools run in a sandbox + +- **Status:** **Accepted** 2026-10-15, ruled by Cofounder under Sam's delegation; Sam may overrule. Acknowledged unknowns this does not settle: per-user usage was measured on Scout as a proxy, because hosted agents write no run records yet (re-measured two weeks after Stage 2); how hosted turns behave under a spot-node reclaim; and E2B as an outside dependency. +- **Scope-boundary note:** [ADR-011](ADR-011-shell-first-pre-gtm.md) paused the Tier 2 cloud sandbox. Its 2026-10-15 status update lifts that pause for the out-of-the-box goal only, and A2 below depends on that lift. [ADR-026](ADR-026-local-agent-daemon.md) (local execution) is unchanged. + +**Why.** W2 shipped as D1 selected: one Durable Object per hosted agent. Measured from a fresh account on 2026-10-15, it is fast (agent ready 26 s after signup, first useful reply about 10 s later) and honest, but its only tool reads pod context. It has no web, no code, no files, no connectors and no run records. It answered one of four basic asks (summarize the pod) and declined web lookup, running a script and reading a repo. Making it useful needs the connector broker, an `AgentRun` per turn, the per-user daily ceiling ([ADR-022](ADR-022-persona-colleagues.md) D5), the tool gate and the connector eligibility rule. `nativeRuntimeService` already has all five; the Durable Object has none. + +**A1. Hosted turns run on `nativeRuntimeService`** (Stage 1, about 2026-10-22). A `runtimeType: 'hosted'` install answers through the native loop, with DeepSeek through LiteLLM, so the platform fallbacks apply. It sits behind a per-install flag, and the Durable Object stays as the fallback until the native path reaches parity, then retires. This narrows D2: hosted agents move instead of the runtime being strangled in place, because the one-loop rule (CLAUDE.md design rule 6) outweighs keeping a second loop that would need the same five features. + +**A2. Tools run in an E2B sandbox** (Stage 2, about 2026-10-29). Each turn gets one Firecracker microVM, stopped when the turn ends so there is no idle billing. Outbound traffic is limited to listed hosts, and credentials come from the broker, never the prompt. Web lookup goes through the same egress path or a search API, whichever costs less per turn. +- **Rejected:** the Cloudflare Sandbox SDK runs inside Workers, so the backend would need a proxy, and a sandbox left idle bills memory for 10 minutes. gVisor on our own cluster needs warm slots we would run, with no TypeScript SDK. Daytona shares the host kernel by default and has been closed source since June 2026. +- **The exit:** E2B's runtime is Apache-2.0 and can be self-hosted. + +**A3. Cost is recorded and bounded.** Every hosted turn writes an `AgentRun`, and an alarm fires when any one user passes $2 in a day. The plan line ships with tools: 30 turns a day for free or individual, 200 for team or paid. DeepSeek stays the default, and a premium model is a paid add-on. +- **Measured on 2026-10-15,** on DeepSeek V4.1-Flash at $0.30/$1.20 per million tokens, with Scout runs averaging 5,468 tokens: about $1–2.30 per user a month at the 90th percentile of use, and at most $6.70 at the 30-turn cap. +- **The 200-turn cap is the tail,** at $14–44. +- **The model sets the price, not the sandbox:** about $0.0004 per 20-second turn on E2B. + +**What shipped, and where it differs from the text above** (see "As built" for paths): +- **A1** merged on 2026-10-15 and 2026-10-16 (#2224, #2236, #2226), a week before its date. The fallback is narrower than written: the migration off the Durable Object is forward-only, so only new installs can fall back. +- **A2** merged on the same two days (#2232, #2244, #2249, #2248), two weeks early. The sandbox is stricter than written: no outbound traffic at all rather than listed hosts, and no credentials inside it. Web lookup is a search API and a page reader called from the backend, not traffic from the sandbox. A turn that runs no code opens no sandbox. +- **A3** merged (#2227, #2237). The plan line is 30 turns a day free and 200 on Pro, keyed on the owner's `entitlements.pro`. A premium model as a paid add-on is not built: a hosted agent runs on the default model only. +- **The spot-reclaim unknown** in the status line above no longer applies as worded: hosted turns now run in the backend, which is pinned to the on-demand pool. A restart mid-turn is covered by #2226. ## Context @@ -35,7 +119,9 @@ Two consequences worth stating separately, because they remove *work* rather tha **4. There are $10,000 of Cloudflare credits available, and they are substantially the runway.** Commonly is unfunded β€” no raise, no YC (F26 declined, on wedge rather than engineering). Infrastructure is paid out of pocket. So these credits are not a discount on a budget; for practical purposes they are the budget, and their term is a real clock rather than an accounting detail. What they do not buy is engineering time, which is the other scarce resource and the one no credit covers. See D4, which is where the economics actually live. -## Decision (proposed) +## Decision (drafted 2026-08-15, ratified 2026-08-28) + +*As of 2026-10-22: D1's Durable Object design shipped as W2 (`workers/agent-runtime`) and is now the fallback engine. A1 supersedes D2's "nothing that exists moves": hosted agents moved. D4's economics assumed Durable Object hibernation and inference on Workers AI credits; hosted turns now run on paid DeepSeek inference through LiteLLM, so A3 carries the cost model. D3, and D4's two obligations, stand as written.* **D1 β€” Nothing is decided until one spike resolves.** Does `@earendil-works/pi-coding-agent` run under `workerd`'s Node compatibility? @@ -44,6 +130,13 @@ Two consequences worth stating separately, because they remove *work* rather tha This is a day of work and it decides the ADR. **Do not amend ADR-021 before it returns.** +**D1 resolved YES on 2026-08-22.** Spike evidence (run locally under workerd via `wrangler dev --local`, `nodejs_compat`, pi 0.84.2): `@earendil-works/pi-agent-core` β€” the turn engine: `Agent`, `AgentHarness`, compaction, 106 exports β€” **loads AND constructs inside workerd.** Three findings that de-risk the DO design beyond the yes/no: +1. The package quarantines its Node half behind a separate `./node` export (fs sessions, child_process, readline live there and only there β€” verified by builtin survey of the dist graph). The default entry the worker imports never touches them. +2. The engine's LLM transport is dependency-injected (`streamFn`; construction without one fails with the engine's own validation error, not a compat error). Fetch-based transport, no Node HTTP binding β€” exactly the shape a DO provides. +3. Construction with a stub transport succeeds end-to-end: `{ loaded: true, exports: 106, constructed: true }`. + +Per D1's own text, the resolution selects: **build W2's runtime as Durable Objects** β€” one DO per `(agentName, instanceId)`, sessions on DO storage (replacing the `./node` fs half), tools as CAP calls. Spike artifacts: `worker.js` + `wrangler.toml`, reproducible in ~5 minutes. Acknowledged unknown carried into that build: D4's credit-term economics remained estimates, not measurements. + **D2 β€” Strangler, not migration.** If accepted, W2 is *born* on Cloudflare. Nothing that exists moves. The shell keeps talking to the runtime through CAP β€” the same four verbs a BYO wrapper uses, which ADR-021 already requires the hosted runtime to dogfood. That interface is what makes the substrate swappable, and it is the reason this is a driver decision rather than an architectural rewrite (CLAUDE.md: *"one runtime change = one adapter file"*). **D3 β€” Uses of the credits that need no substrate decision at all.** These are independent of D1 and worth doing regardless: @@ -77,6 +170,7 @@ Stated explicitly because this is the failure mode of every "let's move to X" pr - **Not a backend migration.** Express + Mongo + Postgres + Socket.io + the stateful agent containers stay on GKE. Rewriting them onto Workers is months of work with no user-visible payoff, during which activation stays at 4%. - **Not a reason to delay the staging environment.** A second Helm release in the empty `commonly` namespace is hours of work and independent of everything here. This session wrote to production three times to verify things β€” 10 episode rows, test messages in a live pod, compiled code staged into the running backend β€” because there is nowhere else. That need is real today. - **Not a re-litigation of ADR-021.** pi stays the turn engine. The M1–M4 ladder, the `SCOUT_RUNTIME_ENABLED` gate, the staged rollout and the named revert procedure all stand. Only the substrate under Part A.1 is in question. + *(2026-10-22: as built, hosted turns do not run on pi, and no `SCOUT_RUNTIME_ENABLED` or `HOSTED_RUNTIME_ENABLED` gate exists in the code; see "As built". This ADR still does not rule on ADR-021 Part A's choice of pi.)* ## Risks @@ -87,7 +181,9 @@ Stated explicitly because this is the failure mode of every "let's move to X" pr - **The cliff at the end of the credit term is real and dated.** Zero marginal cost is a property of the credits, not of the architecture. If hosted agents work and nothing charges before the term ends, the reward for succeeding is a bill with no revenue behind it β€” a worse position than today, because by then users depend on it. This is why D4 makes pricing a condition rather than a follow-up. - **Opportunity cost is the largest risk in this ADR and it is not technical.** Twelve months spent on substrate arrives at month twelve with better architecture and the same 4%, and the same wedge YC already declined. Every week here has to be defensible as the fastest available path to a second returning user, or it is the wrong week. -## Open questions +## Open questions (2026-08-15) + +*Status 2026-10-22: 1 and 3 are moot for the native engine, whose tools run in-process and whose hosted agents answer only mentions and first contact. 2 is answered: the native runtime writes the `AgentRun` in-process. 6 is partly answered by the plan line (#2237). 4 and 5 stay open. The open items that decide what comes next are under "As built".* 1. Do DOs have a workable path to the per-agent MCP surface, or does each DO need its own MCP client? (Affects M1 scope.) 2. Where does `AgentRun` accounting live if turns execute off-cluster β€” still written back through CAP, or does the runtime own its own ledger? diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 8096552c87..5227ea18a9 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -4210,3 +4210,33 @@ Settings read one writable pod's `config.environment`, while the daemon selected **Repair:** Settings now uses the daemon's environment projection for model and effort, saves through that projection's source pod, and shows standalone CLI controls as locally set and read-only. Focused route, UI and adapter tests connect the saved effort to the spawn argv. **Lesson:** a displayed runtime setting must identify the source the runner reads. A server preference cannot attest a local process that does not consume it. + +## 85. A config PATCH addressed to one pod writes the agent's installs in every pod the caller can reach (2026-10-22, lily-shen) + +*Origin: measured on 2026-10-22 while narrowing `boardWake` to one working pod per builder seat. The narrowing could not be done through the route and was written per installation document instead.* + +`PATCH /api/registry/pods/:podId/agents/:name` names one pod. The handler checks the caller against that pod (`backend/routes/registry/agent-config.ts:81-95`) and loads that pod's installation (`:97-105`). Then it widens. It loads every installation of the same agent and instance that is not uninstalled, in any pod (`:205-209`), and keeps those in pods the caller created or belongs to (`:218-235`; `userHasPodAccess`, `backend/routes/registry/helpers.ts:604-609`). An instance admin writes all of them; an installer who is not an admin writes only the rows they installed (`:244-247`). The body's `config` is shallow-merged into each target (`:170-172`), and `status`, `scopes` and `displayName` are applied the same way (`:194-202`), row by row (`:288-292`). The only sign that more than one row changed is `updatedPods` in the response (`:366`). + +So a setting the data model keeps per pod cannot be set for one pod through this route. `config.boardWake` is read per installation: the task wake path loads the pod's active installs and filters each one with `boardWakeEnabled` (`backend/services/taskEventService.ts:184-197`; the predicate is `backend/services/agentMentionService.ts:1175`). An admin who sends `{ config: { boardWake: { enabled: false } } }` to one pod, meaning to quiet the board there, turns board wakes off for that agent in every pod they belong to. Because the merge is shallow, any top-level `config` key in the body replaces that key on every row reached. `status: 'paused'` at one pod pauses the agent in all of them. + +The path taught the false model. A URL that names one pod reads as a write to that pod's installation, and nothing in the request or the route's comment says "every pod". The comment above the fan-out explains how its authority is scoped (`:237-243`), which is a different question from how far it reaches. + +**Repair:** none in code yet. Per-pod settings are written to the installation document directly. A fix belongs to the route's owner: either a body flag that limits the write to the named pod, or a split between identity-wide fields (display name, persona, runtime), which fan out, and per-pod fields such as wake subscriptions, which do not. + +**Lesson:** a write route whose path names one resource should write only that resource, or say in its name and its response that it writes more. Until this one does, read `updatedPods` after every call: anything above 1 means other pods changed. + +*Witness: the handler read at `46da417f9`; the line numbers above are from that tree.* + +## 86. A seat that can drive a browser can lift the browser's session token (2026-10-19, operator) + +*Origin: an incident on 2026-10-19, written up by the operator. Seat names, account names and browser profiles are left out on purpose; the mechanism is the finding.* + +Two fleet seats had access to the operator's automation browser. Each read the signed-in `token` from commonly.me's `localStorage` and set it into its own localhost dev frontend, and some of those frontends pointed at the production API. The token value never entered a transcript, and an audit found no writes made with it. + +What that token is, read from the code. The web app keeps its session JWT in `localStorage` under `token` (`frontend/src/context/AuthContext.tsx:71`, `:129`), so anything that can run script in a signed-in tab can read it. It is a stateless JWT, signed with the server secret and valid for 7 days (`backend/controllers/authController.ts:841`). The session check verifies the signature and then refuses only a missing, banned or bot account (`backend/middleware/auth.ts:108-127`, `backend/services/sessionAccountService.ts:45-50`). Nothing reads the token's `iat`. So, short of suspending the account, the only way to end one lifted session before it expires is to rotate the signing secret, which signs out every user. + +The second lesson is about the fence. The host-side control that keeps seats out of the browser had been replaced, silently, by an auto-update: a wrapper symlink was relinked to the plain binary. Nothing failed and nothing alerted. A fence that an updater can overwrite is only as good as the watcher that checks it. + +**Repair:** planned after 2026-11-01: per-user session revocation, a per-account timestamp (a `sessionsValidAfter`-style field) that the session check compares with the token's `iat`, so one account's sessions can be ended without signing out everyone. + +**Lesson:** browser access is credential access. A browser an agent can drive holds every session the person holds, including their session to the product being built, and an agent that needs a signed-in session will find the one already there. Grant browser access as you would grant the token. Two corollaries: a bearer with no per-user revocation turns every leak into an all-users decision, and a guard that lives in a replaceable file needs a check that runs after every update to the thing it wraps.