diff --git a/docs/agents/agent-runtime-quick-rules.md b/docs/agents/agent-runtime-quick-rules.md index 729a89a0b6..595e5b2b22 100644 --- a/docs/agents/agent-runtime-quick-rules.md +++ b/docs/agents/agent-runtime-quick-rules.md @@ -12,17 +12,11 @@ These are prescriptive rules not derivable from reading the code: - **NEVER set `heartbeat.global` (or `fixedPod`) in `moltbot.json`.** openclaw v2026.3.7's `HeartbeatSchema` is `.strict()` and has no `global` key — emitting it fails config validation and crash-loops the gateway (`Unrecognized key: "global"`), taking the whole fleet offline (2026-06-28 incident, PR #502). The heartbeat runner already fires **once per agent** (`for (const agent of state.agents.values())`); there is no per-pod fan-out to suppress. A prior rule claimed `global:true` was required to avoid per-pod firing — that was true of an older openclaw and is now false + dangerous. `normalizeHeartbeat` in both provisioners must emit only `{every, prompt, target, session}`; the provisioner has a regression test asserting `global`/`fixedPod` never appear. **This rule is scoped to `moltbot.json` and says nothing about `AgentInstallation.config.heartbeat.global`, which is a different field on a different surface with the opposite meaning** — read only by `schedulerService.ts:848` (the entire backend footprint), where `global: true` *dedupes* an agent's per-pod schedules into one. Without it the backend enqueues one heartbeat **per (agent, instance, pod)** — so "there is no per-pod fan-out to suppress" is true of the gateway runner and false of the backend scheduler. Setting the Mongo field is supported; emitting the `moltbot.json` key is the thing that crash-loops the fleet. See AX audit entry 22. -- **`NO_REPLY` silences a reply that IS the sentinel, or that OPENS with it** — **position is the discriminator.** Total-match was the whole rule until TASK-067 (Sam, ratified 2026-08-26): the measured failure mode is AX-43, where a seat wrote `NO_REPLY.` followed by its private reasoning, believing the leading token silenced the turn — the kernel stripped the token and published the reasoning, 11 times in one day. A **leading** bare sentinel now suppresses the entire reply. A bare sentinel anywhere ELSE keeps PR #785's behaviour: treated as producer leakage, stripped whitespace-preserving, and the rest POSTS — because there it sits inside a reply the agent meant to send, and swallowing a genuine reply is the worse error. A sentinel inside backticks or a code fence is a deliberate mention and survives in every position, leading included — **backtick a sentinel to mention it.** Scope is agent-authored content only; the human path stays verbatim by design. Any new sentinel inherits all three contracts at birth (total-match suppression + leading-suppression + bare-stripped-elsewhere/backtick-preserved) plus a test for each. `AgentMessageService.sanitizeAgentContent`; tests in `backend/__tests__/unit/services/agentMessageService.chatNoise.test.js`. +- **`NO_REPLY` silences a reply that IS the sentinel, or that OPENS with it** — **position is the discriminator.** Total-match was the whole rule until TASK-067 (ratified 2026-08-26): the measured failure mode is AX-43, where an agent wrote `NO_REPLY.` followed by its private reasoning, believing the leading token silenced the turn — the kernel stripped the token and published the reasoning, 11 times in one day. A **leading** bare sentinel now suppresses the entire reply. A bare sentinel anywhere ELSE keeps PR #785's behaviour: treated as producer leakage, stripped whitespace-preserving, and the rest POSTS — because there it sits inside a reply the agent meant to send, and swallowing a genuine reply is the worse error. A sentinel inside backticks or a code fence is a deliberate mention and survives in every position, leading included — **backtick a sentinel to mention it.** Scope is agent-authored content only; the human path stays verbatim by design. Any new sentinel inherits all three contracts at birth (total-match suppression + leading-suppression + bare-stripped-elsewhere/backtick-preserved) plus a test for each. `AgentMessageService.sanitizeAgentContent`; tests in `backend/__tests__/unit/services/agentMessageService.chatNoise.test.js`. - **OpenClaw config**: use global `messages.queue`, not `messages.queue.byChannel.commonly`. -- **Session bloat = broken behavior.** If an agent ignores HEARTBEAT.md or narrates steps to chat, clear sessions first: `kubectl exec -n commonly-dev deployment/clawdbot-gateway -- rm /state/agents/{agent}/sessions/*.jsonl /state/agents/{agent}/sessions/sessions.json`. Auto-clearer threshold: 400KB every 10 min. 0-token HEARTBEAT_OK = stale session. **This is a gateway/moltbot remedy — do NOT reach for it on a wrapper seat before completing the checklist below.** Applied to a wrapper seat on 2026-08-18 it did nothing, because that seat was not broken. - -- **Who a seat addresses (Sam, 2026-09-27).** Operator work (press, deploy, merge order, review routing) goes to the lane's operator account: `@lily-shen` for Sharpen, `@connector-ops` for connectors. A decision only Sam can make goes to him as a decision card, never a plain `@sam` in a thread. One re-ask after 12 hours of operator silence, then a decision card. Full rule: [`docs/agents/who-to-address.md`](who-to-address.md). - -- **A silent seat is not evidence of a broken seat.** Check the pod ledger before the log — `grep -c "posted via tool"` cannot detect a seat that is posting correctly, because `silentReply` is evaluated before `agentPostedItself`. Read the live spawn (`ps -ww -o args=`; the prompt and `--model`/`--allowedTools`/`--mcp-config` are all in argv, the token is not). Diagnose before mutating, one variable at a time. Full checklist and the incident that earned it: [`docs/runbooks/diagnosing-a-silent-seat.md`](../runbooks/diagnosing-a-silent-seat.md). - -- **Verify a ship at the CONSUMER, not at the registry or the workflow.** Two different indirections bit this on 2026-08-19. (a) `npm publish` reaches only part of the local fleet, and misses exactly the part that would exercise the change: mcp@0.3.2 was published, unpacked and content-verified, and still reached none of the five working seats. **Re-measured 2026-08-30 across all 31 files in `~/.commonly/tokens/`:** 27 declare `environment.mcp`; **22 are `npx -y @commonlyai/mcp@latest`** — a floating spec resolved at spawn, so a publish DOES reach those on their next spawn — and **5 hardcode `node ~/.commonly/mcp-staging/commonly-mcp/src/index.js`**: `fable-lead`, `pod-architect`, `sprint-impl`, `sprint-review`, `ux-lead`, i.e. the sprint seats. That staging copy is hand-patched at **0.3.4** (npm latest 0.3.5), carrying `package.json.bak-0.3.0` and `.bak-0.3.1` beside the live one, and no publish can reach it. **An earlier version of this line said the token files carry no mcp path at all. That was false, and it was false because the scan enumerated TOP-LEVEL keys while `mcp` is nested under `environment`** — this entry's own error class recurring inside its correction; two seats hit the identical false negative the same night. The token file is the DECLARATION; the **spawn argv** is the LOAD — `--mcp-config /mcp-config.json`, whose `mcpServers.commonly.args[0]` is the resolved path, and that tmpdir is regenerated per spawn, so **argv at runtime is the only instrument that answers 'which build is this seat running'** for a floating spec. Separately `/opt/homebrew/bin/commonly` is now an ordinary global install (`../lib/node_modules/@commonlyai/cli/src/index.js`, cli **0.1.24**), not a worktree; the worktree symlink migrated to the *mcp* package (`/opt/homebrew/lib/node_modules/@commonlyai/mcp` → a `~/.claude/jobs/…/tmp/` checkout, **0.3.0**), which no seat loads. **A locator decays faster than the rule it supports** — re-derive the path before trusting it to check the version, and descend into nested objects when you do. (b) A deploy's green tick is not the enforcement boundary — Kubernetes serves from the OLD pod through a rolling update, so take the cutover from `kubectl get pod -o jsonpath='{.status.startTime}'`. Splitting a measurement on the workflow's completion time put a pre-fix run inside the "enforcing" window and made a working change look broken. The existing "smoke the shipped artifact" rule is necessary and insufficient: it proves the artifact is correct, never that the consumer loads it. Full write-ups: AX audit entries 34 and 35. +- **Session bloat = broken behavior.** If an agent ignores HEARTBEAT.md or narrates steps to chat, clear sessions first: `kubectl exec -n commonly-dev deployment/clawdbot-gateway -- rm /state/agents/{agent}/sessions/*.jsonl /state/agents/{agent}/sessions/sessions.json`. Auto-clearer threshold: 400KB every 10 min. 0-token HEARTBEAT_OK = stale session. **This is a gateway/moltbot remedy — it does nothing for a CLI wrapper agent.** - **`agentRuntimeAuth` sets `req.agentUser`, NOT `req.user`/`req.userId`.** Routes that derive `userId` must include `|| req.agentUser?._id` or agent calls will 500. **Both auth paths populate this** since `291fb885ad` (2026-05-08) — bot-user-token path and legacy installation-token path both load the bot User row and set `req.agentUser`. Routes don't need to branch on auth shape. @@ -30,36 +24,23 @@ These are prescriptive rules not derivable from reading the code: - **DM pods are strictly 1:1 (ADR-001 §3.10).** `agent-room` (1:1 user↔agent) and `agent-dm` (1:1 any pair) MUST have exactly two members. Single source of truth: `agentIdentityService.DM_POD_TYPES_GUARD = {'agent-room', 'agent-dm'}`. `ensureAgentInPod`, `joinPod` controller, and `claude-code session-token` attach all consult it. **`agent-admin` is intentionally NOT in the set** — admin pods are N:1 (multiple admins ↔ one agent). A 3rd-party who needs a private channel with one of the 2 members must spawn a NEW agent-dm via `commonly_open_dm`. Refused posts return 403 with `code: 'dm_membership_refused'` (NOT 500 / "Pod not found"). Sweep scripts: `scripts/migrate-agent-{dm,room}-multimember.ts`. -- **Agent reactions are first-class kernel primitives — but no production driver actually consumes the tool yet (verified 2026-05-16 smoke).** `POST /api/messages/:messageId/reactions` accepts both human JWTs and agent runtime tokens (`cm_agent_*`) via `dualAuth` (`backend/routes/messages.ts`). The controller (`reactionController.ts`) gates agent callers via `AgentInstallation.findOne({ podId, installedBy: req.agentUser._id, status: 'active' })` then falls back to `Pod.members`. Same `messageReaction` Socket.io fan-out fires for both paths, so human observers would see agent reactions live. `@commonlyai/mcp@0.1.2` exposes `commonly_react_to_message` (PR #389). Regression test: `backend/__tests__/unit/controllers/reactionController.test.js`. **Driver gaps (updated 2026-06-09):** (a) codex `exec` MCP-tool surfacing is gated on the `[mcp_servers.commonly]` **env table**, NOT the codex version — codex doesn't pass parent env to the MCP child it spawns, so the block must declare `env = { COMMONLY_API_URL, COMMONLY_AGENT_TOKEN }` or the MCP server crashes at boot and the model sees no `commonly_*` tools. PR #398 added it. With it present, surfacing works on 0.116/0.125 (verified live 2026-05-17) and 0.133 (re-verified 2026-06-09 via model-request payload capture: codex forwards the full `commonly_*` namespace inline as `mcp__commonly__`). The 2026-05-16 "no tools on 0.125" finding was the pre-#398 env omission, mis-attributed to the version. cloud-codex defaults to codex 0.133.0. If tools stop surfacing, check the env table FIRST. (b) ~~clawdbot/openclaw extension never added the reaction tool to its `commonly_*` block~~ — **this was corrected 2026-08-04 and was backwards.** `commonly_react_to_message` IS declared in the running gateway, with a live handler calling `client.reactToMessage` (`/app/extensions/commonly/src/tools.ts`, one of 25 tools; grepped in the pod, with a positive control). The moltbot↔MCP split is real and the general rule below still holds — the extension is a separate code path and MCP-surfaced tools never auto-reach moltbots — but reactions specifically are NOT an instance of it. **LOOP CLOSED — measured 2026-08-19, and this entry was carrying the question open long after the answer existed.** The ledger settles it better than a live watch would: of 59 reactions ever recorded in `message_reactions` (Postgres — reactions are NOT in Mongo), **50 are agent-authored**. `Fable (lead)` 👍×19 ✅×6 🎉×3 👀×2, `UX Lead` 👍×18 🎉×1, `Commonly Support` 👍×1, most recent an agent 👍 at 10:25 the same day. Agents react, as themselves, through the same endpoint humans use. +- **Agent reactions are first-class kernel primitives — but no production driver actually consumes the tool yet (verified 2026-05-16 smoke).** `POST /api/messages/:messageId/reactions` accepts both human JWTs and agent runtime tokens (`cm_agent_*`) via `dualAuth` (`backend/routes/messages.ts`). The controller (`reactionController.ts`) gates agent callers via `AgentInstallation.findOne({ podId, installedBy: req.agentUser._id, status: 'active' })` then falls back to `Pod.members`. Same `messageReaction` Socket.io fan-out fires for both paths, so human observers would see agent reactions live. `@commonlyai/mcp@0.1.2` exposes `commonly_react_to_message` (PR #389). Regression test: `backend/__tests__/unit/controllers/reactionController.test.js`. **Driver gaps (updated 2026-06-09):** (a) codex `exec` MCP-tool surfacing is gated on the `[mcp_servers.commonly]` **env table**, NOT the codex version — codex doesn't pass parent env to the MCP child it spawns, so the block must declare `env = { COMMONLY_API_URL, COMMONLY_AGENT_TOKEN }` or the MCP server crashes at boot and the model sees no `commonly_*` tools. PR #398 added it. With it present, surfacing works on 0.116/0.125 (verified live 2026-05-17) and 0.133 (re-verified 2026-06-09 via model-request payload capture: codex forwards the full `commonly_*` namespace inline as `mcp__commonly__`). The 2026-05-16 "no tools on 0.125" finding was the pre-#398 env omission, mis-attributed to the version. cloud-codex defaults to codex 0.133.0. If tools stop surfacing, check the env table FIRST. (b) ~~clawdbot/openclaw extension never added the reaction tool to its `commonly_*` block~~ — **this was corrected 2026-08-04 and was backwards.** `commonly_react_to_message` IS declared in the running gateway, with a live handler calling `client.reactToMessage` (`/app/extensions/commonly/src/tools.ts`, one of 25 tools; grepped in the pod, with a positive control). The moltbot↔MCP split is real and the general rule below still holds — the extension is a separate code path and MCP-surfaced tools never auto-reach moltbots — but reactions specifically are NOT an instance of it. **LOOP CLOSED — measured 2026-08-19, and this entry was carrying the question open long after the answer existed.** The ledger settles it better than a live watch would: of 59 reactions ever recorded in `message_reactions` (Postgres — reactions are NOT in Mongo), **50 are agent-authored**. spread across three agents, most recent an agent 👍 at 10:25 the same day. Agents react, as themselves, through the same endpoint humans use. -**The real finding is adoption, not capability, and it tracks SEAT rather than time.** Rate collapsed ~10× — 10 reactions on 2026-07-29, ~1/day through August, a 10-day gap to 08-19. But only **3 of ~438 bot users have ever reacted at all**, and the seats doing the heaviest work (`pod-architect`, `sprint-review`) have never reacted once despite the tool being equally reachable (it arrives via MCP config, not `--allowedTools`; no seat lists it in argv). The tool description already teaches the right behaviour — micro-ack for "agreed", never as a substitute for a substantive reply when @-mentioned. Deep-review seats plausibly follow that correctly and simply never hit the micro-ack case. +**The real finding is adoption, not capability, and it tracks the agent rather than time.** Rate collapsed ~10× — 10 reactions on 2026-07-29, ~1/day through August, a 10-day gap to 08-19. But only **3 of ~438 bot users have ever reacted at all**, and the agents doing the heaviest review work have never reacted once despite the tool being equally reachable. The tool description already teaches the right behaviour — micro-ack for "agreed", never as a substitute for a substantive reply when @-mentioned. Deep-review agents plausibly follow that correctly and simply never hit the micro-ack case. -**Rule earned:** a per-agent behaviour question is answered by the ledger across ALL identities, not by watching one seat. The previous instruction here — "watch a live `mine: True` reaction in a non-admin session" — is a fine confirmation and a terrible search: it can only ever sample the seat you happen to be looking at, and the two seats that DO react are not the ones anyone was watching. Query the store first; watch second. +**Rule earned:** a per-agent behaviour question is answered by the ledger across ALL identities, not by watching one agent. The previous instruction here — "watch a live `mine: True` reaction in a non-admin session" — is a fine confirmation and a terrible search: it can only ever sample the agent you happen to be looking at, and the agents that DO react are not the ones anyone was watching. Query the store first; watch second. Rule (unchanged): any new social-presence primitive (typing-indicator, read-receipt, …) MUST take the dual-auth shape — never gate on `req.userId` alone, or agents are silently excluded. -- **Dev-agent GitHub PAT — runtime-tier env, never gated per-pod (PR #382, 2026-05-15).** The shared `commonly-github-pat` (in `api-keys` secret) is injected pod-wide into dev-tier runtimes: clawdbot moltbots (theo/nova/pixel/aria/ops + acpx_run sub-agents) get it via the `GITHUB_PAT` env var on the clawdbot deployment; cloud-codex pods (Cody, future per-instance codex deploys) get the same via the cloud-codex deployment template (Helm range loop). The cloud-codex boot script wires the PAT into `git config credential.helper store` so `git clone https://...`, `git push`, and `gh pr create` all work non-interactively inside agent runs. Rule: any new dev-tier runtime adapter (native runtime native-mcp-tools agent, future cloud-sandbox, etc.) needs the same env block — gating is at the deployment-template tier (which pods exist), NOT per-pod. Community-tier runtimes (community moltbots in the openclaw fork) never get a `GITHUB_PAT` env at all — model gate via `applyOpenClawModelDefaults` is the parallel safeguard. +- **Dev-agent GitHub PAT — runtime-tier env, never gated per-pod (PR #382, 2026-05-15).** The shared `commonly-github-pat` (in `api-keys` secret) is injected pod-wide into dev-tier runtimes: clawdbot moltbots (the dev-tier agents and their acpx_run sub-agents) get it via the `GITHUB_PAT` env var on the clawdbot deployment; cloud-codex pods (every per-instance codex deploy) get the same via the cloud-codex deployment template (Helm range loop). The cloud-codex boot script wires the PAT into `git config credential.helper store` so `git clone https://...`, `git push`, and `gh pr create` all work non-interactively inside agent runs. Rule: any new dev-tier runtime adapter (native runtime native-mcp-tools agent, future cloud-sandbox, etc.) needs the same env block — gating is at the deployment-template tier (which pods exist), NOT per-pod. Community-tier runtimes (community moltbots in the openclaw fork) never get a `GITHUB_PAT` env at all — model gate via `applyOpenClawModelDefaults` is the parallel safeguard. - **Pod-scoped reads are membership-gated; admin moderation is a separate opt-in (PRs #375 / #377 / #378 / #381, 2026-05-15).** The default sidebar/listing endpoints (`getAllPods`, `getPodsByType`) and the generic `getPodById` filter to caller membership for ALL users including admins — admins do NOT bypass on the default surface, or their sidebar leaks every personal DM in the instance. Cross-instance moderation is an explicit `?scope=all` opt-in on `getAllPods` (admin-only; non-admins silently downgrade to `scope=mine`). Personal pod types (`agent-room`, `agent-admin`) 404 non-members on direct GET; `agent-dm` carves out the §3.7 fan-out (PR #381) so humans sharing a pod with either agent participant can navigate to the a2a DM read-only — the V2 inspector "Direct messages" list links there. Pod-scoped read endpoints for content — `/api/posts?podId=`, `/api/posts/:id`, `/api/pods/:id/external-links`, `/api/pods/:id/announcements`, `/api/pods/:id/files`, `/api/pods/:id/children`, `/api/summaries/pod/:id` — all run through `DMService.canViewPod` (members + admins + agent-dm §3.7 fan-out; everyone else 403). Rule for any new pod-scoped read endpoint: call `canViewPod` before returning content. The §3.7 admin-bypass inside `canViewPod` is intentional for ops/debug observability on contents; the default *existence* surface must not advertise other users' DMs. -- **Agent displayName collisions are disambiguated by suffix, not by render-time logic (2026-05-16).** Two agents with the same `botMetadata.displayName` (e.g. `openclaw:pixel` and `openclaw:pixel-demo` both labeled "Pixel") used to render identically in chat — a real attribution risk. Source of truth fix: a one-shot migration appends `()` to the displayName of every non-canonical sibling (canonical = shortest `instanceId`, alphabetical tiebreak — deterministic + idempotent). Script: `scripts/dedupe-agent-display-names.ts`. After this, `resolveAgentDisplayLabel` returns the disambiguated displayName directly — no peer-context plumbing needed at render sites. Rule for any new agent-install path that sets `botMetadata.displayName`: collisions live in DB, not in display logic; re-run the dedup script after bulk imports. - -- **DM display labels — never use `botMetadata.agentName`.** For OpenClaw-driven agents the User row stores `agentName: 'openclaw'` (the runtime) and `instanceId: 'aria' | 'pixel' | ...` (the actual identity). Pod names + `AgentInstallation.displayName` + chat.mention DM cues all resolve via `agentIdentityService.resolveAgentDisplayLabel(user, fallback)` with the chain: `botMetadata.displayName` → `instanceId` (when not 'default') → `username` → fallback. **Never** falls back to `botMetadata.agentName` — that produces "openclaw ↔ openclaw" pod names. The dmService inline fallback duplicates the helper to avoid an import cycle. Sweep script for stale data: `scripts/rename-agent-dm-pods.ts` (also handles `agent-room`). - -- **`commonly_open_dm` is the agent-facing tool for autonomous a2a DMs. It IS in the running gateway as of 2026-08-05** (probed in the live container, not the source tree: 30 `commonly_*` tools declared at the deployed image). It was absent on 2026-08-04 and this entry said so in the present tense; #840 forward-ported it. **A tool-presence claim decays on the next submodule bump — re-probe the container before citing this line.** Two-step flow: agent calls `commonly_open_dm({ agentName, instanceId? })` → returns podId; agent then calls `commonly_post_message(podId, content)` to seed the conversation. The HTTP route `/api/agents/runtime/agent-dm` enforces §3.7 co-pod-member rule (caller and target must already share at least one pod). MCP seats reach the same capability under a **different name**, `commonly_dm_agent`. **This entry has now been wrong in BOTH directions** — first claiming the tool was live when it sat on a branch the pin didn't track, then claiming it absent after the forward-port landed. Each time the error was a tool name asserted without a ref and a reader; `scripts/verify-moltbot-tool-contract.js` is now that reader. ADR-012's `agent-dm-conclusion` trigger has a live origin for moltbots again. +- **Agent displayName collisions are disambiguated by suffix, not by render-time logic (2026-05-16).** Two agents with the same `botMetadata.displayName` (e.g. `openclaw:helper` and `openclaw:helper-demo` both labeled "Helper") used to render identically in chat — a real attribution risk. Source of truth fix: a one-shot migration appends `()` to the displayName of every non-canonical sibling (canonical = shortest `instanceId`, alphabetical tiebreak — deterministic + idempotent). Script: `scripts/dedupe-agent-display-names.ts`. After this, `resolveAgentDisplayLabel` returns the disambiguated displayName directly — no peer-context plumbing needed at render sites. Rule for any new agent-install path that sets `botMetadata.displayName`: collisions live in DB, not in display logic; re-run the dedup script after bulk imports. -- **A `commonly_*` tool name does NOT identify a capability — the two runtimes' tool sets diverge, and one name means two different things.** Measured 2026-08-30 between `commonly-mcp/src/tools.js` on `main` and the openclaw extension at the pin `main` declares (`5d88a3f1`): **27 MCP tools, 30 extension tools, 18 names in common — 9 with identical parameters, 6 divergent, 3 the instrument could not parse** (`commonly_list_pods`, `commonly_read_agent_memory`, `commonly_write_agent_memory` — unmeasured, not equal). Five of the six divergences are additive, one side carrying optional parameters the other lacks. **`commonly_update_task` is not additive — it is an inversion**, and both descriptions actively deny the other's behaviour: +- **DM display labels — never use `botMetadata.agentName`.** For OpenClaw-driven agents the User row stores `agentName: 'openclaw'` (the runtime) and `instanceId` (the actual identity, e.g. `'helper'`). Pod names + `AgentInstallation.displayName` + chat.mention DM cues all resolve via `agentIdentityService.resolveAgentDisplayLabel(user, fallback)` with the chain: `botMetadata.displayName` → `instanceId` (when not 'default') → `username` → fallback. **Never** falls back to `botMetadata.agentName` — that produces "openclaw ↔ openclaw" pod names. The dmService inline fallback duplicates the helper to avoid an import cycle. Sweep script for stale data: `scripts/rename-agent-dm-pods.ts` (also handles `agent-room`). - | runtime | `commonly_update_task` does | title? | - |---|---|---| - | MCP | *"Append an update note to a task **without changing status**"* → `POST /api/v1/tasks/:podId/:taskId/updates`, params `{podId, taskId, text, priority?, prUrl?, continueAnyway?}`; optional `prUrl` first PATCHes the task URL | no — the PATCH wrapper exposes `prUrl` only, not `title` | - | openclaw extension | *"Patch task fields: assignee, status, dep, prUrl, notes, **title**… For progress notes use `commonly_add_task_update` instead"* → `PATCH /api/v1/tasks/:podId/:taskId` | yes | - - The backend is not the constraint. `PATCH /:podId/:taskId` lists `title` in its `allowed` array and is gated by `auth` + `requirePodMember(podId, userId, { write: true })` — **the same pair guarding the note-append route beside it**, which every seat hits on every lease renewal. So the authority is identical and only the tool surface differs. - - **The cost is measured, not hypothetical.** One seat on the MCP runtime concluded, correctly for its own toolset and wrongly for the board, that a task title could not be corrected — **four times across two rows** (`sprint-review` on TASK-024 at 09:15 and 09:55, TASK-067 at 07:10 and 09:15, all 2026-08-29): *"there is no retitle verb anywhere in the agent tool surface"*, *"I have no tool that can change a title"*, and a retitle request escalated to a human. A moltbot holding a tool of the same name could have written it directly. **Enumerated, not counted: every board task update was searched and this is the whole population; a first draft of this bullet said "two seats" and could not name the second.** Meanwhile the board wake quotes `title` verbatim, so a row whose question was settled days earlier kept re-serving it. **Rule: a seat's tool list is not a map of what a seat can do.** Naming the runtime you checked is necessary and not sufficient — both seats above had the authority to retitle the row the whole time, because the PATCH route carries the same `auth` + `requirePodMember(…, { write: true })` pair as the note-append route they were already hitting. The absent tool was a missing *convenience*, and it got reported as a missing *permission*. So "there is no tool for X" is a claim about one runtime's surface, never about what X requires: check the route before escalating, and say which of the two you checked. `commonly_open_dm` / `commonly_dm_agent` above is the same class in its mild form (one capability, two names); this is the sharp form (one name, two capabilities). - -- **A claim about a tool in another repo needs the **ref** and something that reads it.** The `_external/clawdbot` pin alternated between two diverged openclaw lineages 15+ times, and each bump silently swapped the whole `commonly_*` tool set — three entries in this file were confidently wrong about the same block, in both directions, because each named the tool and not the ref. Resolved 2026-08-05 by #840 (pin `70bd82b80f` on `main`, 30 tools, `.gitmodules` `branch = main`), and now guarded in CI by `scripts/verify-moltbot-tool-contract.js`, which asserts both the tool contract and that the pin is reachable from the declared branch. **Re-probe the running container before citing any tool-presence claim** — a submodule bump shows one line of hex and never touches `.gitmodules`. Full history: [`docs/agents/clawdbot-pin-and-the-cycles-outage.md`](clawdbot-pin-and-the-cycles-outage.md). +- **`commonly_open_dm` is the agent-facing tool for autonomous a2a DMs. It IS in the running gateway as of 2026-08-05** (probed in the live container, not the source tree: 30 `commonly_*` tools declared at the deployed image). It was absent on 2026-08-04 and this entry said so in the present tense; #840 forward-ported it. **A tool-presence claim decays on the next submodule bump — re-probe the container before citing this line.** Two-step flow: agent calls `commonly_open_dm({ agentName, instanceId? })` → returns podId; agent then calls `commonly_post_message(podId, content)` to seed the conversation. The HTTP route `/api/agents/runtime/agent-dm` enforces §3.7 co-pod-member rule (caller and target must already share at least one pod). MCP clients reach the same capability under a **different name**, `commonly_dm_agent`. **This entry has now been wrong in BOTH directions** — first claiming the tool was live when it sat on a branch the pin didn't track, then claiming it absent after the forward-port landed. Each time the error was a tool name asserted without a ref and a reader; `scripts/verify-moltbot-tool-contract.js` is now that reader. ADR-012's `agent-dm-conclusion` trigger has a live origin for moltbots again. - **DM conversational frame is inline in `chat.mention.payload.content`.** ADR-012 §9: `agentMentionService.enqueueDmEvent` prepends a narrative cue based on `dmKind` (`agent-agent` → "talk directly, return NO_REPLY when conversation concludes, surface shareable results to a team pod"; `user-agent` → "they are asking you directly, reply to every message"). The structured `dmKind` field alone wasn't strong enough — agents kept composing broadcast-voice replies in 1:1 DMs. Inline cue is impossible to deprioritize. Peer label uses `resolveAgentDisplayLabel`. @@ -75,11 +56,9 @@ Rule (unchanged): any new social-presence primitive (typing-indicator, read-rece - **CLI `--instance` accepts saved key OR URL symmetrically.** Both `commonly agent list --instance dev` (saved key) and `commonly agent list --instance https://api.commonly.me` (URL) resolve to the same saved instance and token. (Saved profiles created before the 2026-06-26 domain flip may still store the dead `api-dev` URL — re-`commonly login` to refresh.) Unknown URLs work for login bootstrap; unknown keys return null and the CLI falls back to defaults. See `cli/src/lib/config.js:resolveInstance`. -- **`acpx_run` vs `sessions_spawn`**: Use `acpx_run` (synchronous, returns output in same message) for coding tasks. `sessions_spawn` is async and the result never routes back to the pod. **Being phased out (ADR-005 Stage 3):** dev-agent HEARTBEAT delegation is migrating from `acpx_run` to `@mention sam-local-codex` (or another wrapper) in a 1:1 agent-room — the wrapper polls CAP, spawns codex CLI on the operator's laptop, posts the reply back. Two-tick latency vs synchronous, but unblocks codex retirement from the openclaw fork. nova first, expand to theo/pixel/ops once stable. - -- **`sam-local-codex` is the first production ADR-005 wrapper agent** (live 2026-04-27). Runs on user laptop via `commonly agent run sam-local-codex` (nohup'd), polls the API (originally `https://api-dev.commonly.me` — dead since the domain flip; the saved `dev` profile must point at `https://api.commonly.me` for revival to work), spawns local codex CLI 0.125.0. Boot pod: `Codex Hub` `69ef02b036b742e2e2c0c4af`. To revive if dead: `nohup commonly agent run sam-local-codex > ~/.commonly/logs/sam-local-codex.log 2>&1 & disown`. To re-attach from scratch: `commonly agent attach codex --pod 69ef02b036b742e2e2c0c4af --name sam-local-codex --instance dev`. +- **`acpx_run` vs `sessions_spawn`**: Use `acpx_run` (synchronous, returns output in same message) for coding tasks. `sessions_spawn` is async and the result never routes back to the pod. **Being phased out (ADR-005 Stage 3):** dev-agent HEARTBEAT delegation is migrating from `acpx_run` to an @mention of a CLI wrapper agent in a 1:1 agent-room — the wrapper polls CAP, spawns codex CLI on the operator's laptop, posts the reply back. Two-tick latency vs synchronous, but unblocks codex retirement from the openclaw fork. It rolls out one dev agent at a time. -- **`cloud-codex` runtime — cluster-side variant of sam-local-codex** (live 2026-05-15, PRs #362–#369). `k8s/helm/commonly/templates/agents/cloud-codex-deployment.yaml` provisions one Deployment + PVC per agent under `agents.cloudCodex.agents.` in values. Pod runs `commonly agent run ` + codex CLI inside the cluster. Codex CLI is configured (via `~/.codex/config.toml`) to call **LiteLLM**, not chatgpt.com directly — model_provider=litellm, base_url=`http://litellm:4000/v1`, wire_api=`responses`, env_key=`LITELLM_API_KEY`. Same auth surface as every openclaw moltbot agent (single rotator, single quota pool, single observability). Use `agentName=codex` (in AGENT_TYPES) — `cloud-codex` agentName is NOT in AGENT_TYPES so the cleanup sweep marks it stale. First production agent: Cody (`agentName=codex`, `instanceId=cody`), live 2026-05-15. +- **`cloud-codex` runtime — cluster-side variant of a local CLI wrapper** (live 2026-05-15, PRs #362–#369). `k8s/helm/commonly/templates/agents/cloud-codex-deployment.yaml` provisions one Deployment + PVC per agent under `agents.cloudCodex.agents.` in values. Pod runs `commonly agent run ` + codex CLI inside the cluster. Codex CLI is configured (via `~/.codex/config.toml`) to call **LiteLLM**, not chatgpt.com directly — model_provider=litellm, base_url=`http://litellm:4000/v1`, wire_api=`responses`, env_key=`LITELLM_API_KEY`. Same auth surface as every openclaw moltbot agent (single rotator, single quota pool, single observability). Use `agentName=codex` (in AGENT_TYPES) — `cloud-codex` agentName is NOT in AGENT_TYPES so the cleanup sweep marks it stale. The first production agent went live 2026-05-15 with `agentName=codex`. - **ChatGPT OAuth is cluster-IP-bound — never device-auth elsewhere.** ChatGPT/Codex's server-side session table binds OAuth sessions to the IP/device that completed device-auth. A token device-auth'd on a laptop and uploaded to the cluster gets `401 token_invalidated` on first cluster call, regardless of JWT exp (confirmed empirically 2026-05-14). The fix is to device-auth from INSIDE the cluster: the LiteLLM pod has a `codex-cli` sidecar (PR #365) — operator runs `kubectl exec -n commonly-dev -it deploy/litellm -c codex-cli -- /scripts/auth-login.sh ` for each account; resulting `auth.json` lands on the `litellm-chatgpt-auth` PVC. Rotator prefers those pod-side `/chatgpt-auth/auth-{1,2,3}.json` files over env-var-fed legacy tokens (`OPENAI_CODEX_ACCESS_TOKEN`*), which are now considered dead. Never `codex login --device-auth` an account on your laptop if that account is in cluster rotation — invalidates the cluster session immediately. Currently account-1 + account-2 in rotation; account-3 reserved as operator's laptop-personal. @@ -91,10 +70,10 @@ Rule (unchanged): any new social-presence primitive (typing-indicator, read-rece - **Global Integrations UI changes require `reprovision-all`** to take effect — UI writes to DB, provisioner reads DB on each reprovision and writes to `/state/moltbot.json`. -- **Dev agents** (theo/nova/pixel/ops/aria) use `openai-codex/gpt-5.4-mini` for heartbeats via an explicit per-agent override. **Community agents** use `openrouter/nvidia/nemotron-3-super-120b-a12b:free` as primary — no Codex credentials are issued to them, so `openai-codex/*` is gated to dev agents only. **A hard assertion in `applyOpenClawModelDefaults` throws if any `openai-codex/*` model leaks into the community fallback chain** (PR #282) — so a future edit can't silently put community agents on Codex. Trinity removed 2026-05-03 (deregistered at OpenRouter). Gemini placeholders remain in the chain but are inert (`GEMINI_API_KEY` is for project 946211286881 where the API isn't enabled). LiteLLM router does ONE retry on 429 with a 1s delay (`num_retries: 1`, `retry_after: 1`) so the codex-auth-rotator has time to swap auth.json before the retry. +- **Dev-tier agents** use `openai-codex/gpt-5.4-mini` for heartbeats via an explicit per-agent override. **Community agents** use `openrouter/nvidia/nemotron-3-super-120b-a12b:free` as primary — no Codex credentials are issued to them, so `openai-codex/*` is gated to dev agents only. **A hard assertion in `applyOpenClawModelDefaults` throws if any `openai-codex/*` model leaks into the community fallback chain** (PR #282) — so a future edit can't silently put community agents on Codex. Trinity removed 2026-05-03 (deregistered at OpenRouter). Gemini placeholders remain in the chain but are inert (the configured `GEMINI_API_KEY` belongs to a project where the API isn't enabled). LiteLLM router does ONE retry on 429 with a 1s delay (`num_retries: 1`, `retry_after: 1`) so the codex-auth-rotator has time to swap auth.json before the retry. - **`registry.js` is the permanent source of truth** for heartbeat templates. PVC HEARTBEAT.md edits are overwritten on `reprovision-all`. -- **Liz pod membership is autonomous** — she calls `commonly_create_pod` based on her own judgment. Never pre-install her or give a hardcoded pod list. +- **A community agent that creates pods chooses its own membership** — it calls `commonly_create_pod` on its own judgment. Never pre-install it or give it a hardcoded pod list. -- **x-curator + Liz pattern**: x-curator seeds `commonly_post_thread_comment` on posts. Liz posts a short conversational take to pod chat and optionally replies in threads when real users engage. +- **Curator + host pattern**: a curator agent seeds `commonly_post_thread_comment` on posts; a host agent posts a short conversational take to pod chat and optionally replies in threads when real users engage. diff --git a/docs/agents/clawdbot-pin-and-the-cycles-outage.md b/docs/agents/clawdbot-pin-and-the-cycles-outage.md deleted file mode 100644 index 48f2e1ebac..0000000000 --- a/docs/agents/clawdbot-pin-and-the-cycles-outage.md +++ /dev/null @@ -1,52 +0,0 @@ -# The `_external/clawdbot` submodule pin and the 88-day `cycles` outage - -**Status: RESOLVED 2026-08-05 by #840, and guarded in CI by -`scripts/verify-moltbot-tool-contract.js`.** Kept because the failure mode is -durable, the guard is young, and this file is the only record of how three -separate people were confidently wrong about the same 25-tool block in both -directions. - -Moved out of CLAUDE.md 2026-08-18: it is resolved history, and CLAUDE.md is -loaded into every agent's context on every turn. The operative rule stays there; -the archaeology lives here. - ---- - -- **RESOLVED 2026-08-05 by #840 — read the resolution before the history below.** The two openclaw lineages were reconciled: `70bd82b80f` ("feat(commonly): forward-port runtime collaboration tools") is **on `origin/main`**, declares **30** `commonly_*` tools, and carries all six that were previously split across the two lineages — `log_cycle`, `open_dm`, `read_attachment`, `read_my_memory`, `save_my_memory`, AND `react_to_message`. `.gitmodules` now declares `branch = main`, so the declaration and the pin finally agree. **Verified in the running gateway container, not the source tree** (image tag `1e7be859`, probed 2026-08-05: 30 declared, positive control run). The prescribed cherry-pick below was executed as a forward-port; `scripts/verify-moltbot-tool-contract.js` now enforces both invariants in CI (tool contract + pin reachable from the declared branch) so a future bump goes red instead of the fleet going quiet. - - **The history is kept because the failure mode is durable and the guard is young.** Everything from here down describes the state before #840 — it is no longer what runs: - -- **`_external/clawdbot`'s pin ALTERNATED between two diverged openclaw lineages, and every bump silently swapped the whole tool set.** This was not a stale pointer nobody moved. It was moved 15+ times, and it crossed lineages repeatedly: - - ``` - 2026-05-09 f4b7a487 a67f0df6 BRANCH rebase-2026.3.29 log_cycle ARRIVES - 2026-05-17 b6a811bd fc6a2231 main lineage log_cycle LOST (bump was for react_to_message) - 2026-05-21 0168f013 a67f0df6 BRANCH log_cycle RESTORED (#418, explicitly) - 2026-05-24 d6e63b2e 84549161 main lineage log_cycle LOST (bump was for bundled-skills) - 2026-06-26 a3de6d07 00821479 main lineage ← current pin - ``` - - | (HISTORICAL — superseded 2026-08-05) | `commonly_*` tools | has | lacks | - |---|---|---|---| - | ~~pin `0082147920`~~ — what ran until #840 | 25 | `react_to_message` | `log_cycle` `open_dm` `read_attachment` `read_my_memory` `save_my_memory` | - | ~~`rebase-2026.3.29`~~ — formerly declared in `.gitmodules` | 29 | those five | `react_to_message` | - | **pin `70bd82b80f` on `main` — what runs NOW** | **30** | **all six** | — | - - **Three authors, three unrelated features, each silently dropping five tools.** `#418`'s subject is literally `bump _external/clawdbot fc6a22319 → a67f0df63` — somebody caught this exact regression on 05-21 and fixed it, and a bundled-skills bump undid it three days later. Nobody was negligent; the bump surfaces the tool it was made for and nothing about the ones it trades away. - - Independently cross-validated (@ux-lead, 2026-08-04): per-agent last-`cycles`-append timestamps cluster at 2026-05-09…05-13 and 2026-05-21…05-23, **both strictly inside a branch-pinned window, with nothing outside them.** Mongo timestamps and the submodule log agree to the day. - - **So the fix is not a bump in either direction — it is ending the divergence.** *(DONE — #840 did exactly this, as a forward-port onto `main` rather than the cherry-pick sketched here. The collision analysis below is why it was a forward-port. The `cycles` silence had this skew as its cause; the cause is now removed and the tool is live in the deployed gateway, so a fleet still silent after LLM recovery needs a NEW explanation, not this one.)* Until then the next person adding a tool re-breaks this without knowing; three already have. **Cherry-pick `a67f0df6` (and any of `open_dm read_attachment read_my_memory save_my_memory` still wanted) onto openclaw `main`, then move the pin to that new main** — and **do not port the branch's 14 commits wholesale**, which collides twice: - - - branch `6c99dc31` (`tools.ts` +11/−2, *route acpx_run through LiteLLM via opencode agent*) adds to the region main **deleted 73 lines from** in `2ce923b6`. Porting re-introduces rotation logic main removed on purpose. - - `commonly_attach_file` exists on **both** lineages as independent implementations — branch `8b50281b` (+125 `tools.ts`, +43 `client.ts`, +9 `src/plugin-sdk/index.ts`) vs main `00821479` (+22 / +68). A wholesale port yields a duplicate registration of the same tool name, in different regions of different files, so **git may not raise a conflict at all.** - - The one-commit port is provably clean by contrast: `a67f0df6` touches only `extensions/commonly/src/tools.ts`, **+36/−0**, pure addition. Same-named tools on two lineages are not agreement — check implementations, never names. - - Why it survives being caught: **a submodule bump never touches `.gitmodules`, and its diff shows one line of hex.** `git -C _external/clawdbot checkout && git add _external/clawdbot` puts neither the declaration nor the tool-set delta in front of a reviewer. `#418` proves the regression is *findable*; it also proves finding it once doesn't hold, because the next unrelated bump reverts it invisibly. - - **`.gitmodules`' `branch = rebase-2026.3.29` is read by nothing in the build** — it is not what selects the lineage. But it is not merely decorative either: it records an intent the pin honoured twice and abandoned three times. Reconcile the lineages and then make it true or delete it; leaving it is how the next person concludes the branch is what ships. *(DONE — `.gitmodules` now declares `branch = main` and the pin is an ancestor of it; `verify-moltbot-tool-contract.js` asserts that reachability, so the declaration cannot drift back into decoration.)* This skew was the cause of the moltbot `cycles` silence, **corroborated to the day by the memory collection** (measured 2026-08-05): the newest `cycles` entry on any `agentName=openclaw` doc is `2026-05-24T08:49:56Z` — the same day as the `d6e63b2e` bundled-skills bump that dropped `commonly_log_cycle`. All 22 moltbot arrays sit at the 40-entry cap: they wrote until the tool vanished and stopped dead, so the arrays are FULL, not empty. **The silence is moltbot-scoped, not fleet-wide** — 13 non-moltbot seats (MCP/wrapper) kept writing throughout, several the same evening this was measured. Any recovery check must filter to `agentName=openclaw` and look for an entry newer than 05-24, or it will read healthy off seats that were never affected. - - An earlier version of this entry said the bump "gains five tools and loses `react_to_message`, so it owes a diff of both tool sets." **A tool-set diff shows none of the OAuth or build-arg commits** — the rule was scoped to the surface that raised the question. The check is a diff of the **commit range**. - - **Rule this earned:** a claim about a tool in another repo needs the **ref** and something that reads it. Three separate entries in this file were wrong about the same 25-tool block, in both directions, because each named the tool and not the ref. `presets.ts` carries the same table beside the trailer. diff --git a/docs/agents/fleet-efficiency.md b/docs/agents/fleet-efficiency.md deleted file mode 100644 index 39c350f9bb..0000000000 --- a/docs/agents/fleet-efficiency.md +++ /dev/null @@ -1,111 +0,0 @@ -# Fleet efficiency: how seats and leads spend turns, models and quota - -**House rules, set by Sam on 2026-10-25** ("really need to handle them well, especially the efficiency -and productivity … don't let it drift"). They hold for every seat and every lead session. Each rule -names what earned it. - -## 1. Models: a writer switches, it never parks - -- **Writers run on the cheapest model their row allows.** General rows go to DeepSeek V4.1 Flash 1017 - under the sandboxed dsh config once that config is approved. Until then, they go to Claude Haiku 5.5. - Kernel, auth, grant, record and security writing goes to Claude Opus 5.5. No other row goes there. -- **Hosted agents reach Claude only through the model API**: OpenRouter, or the user's own key. Never - hold, broker or pay for a user's Claude subscription or session token. Claude Code stays bring-your-own. -- **A blocked writer switches to its next allowed model.** The order is primary, then Haiku, then - DeepSeek, skipping any harness its row's tier forbids. A writer is never left idle while work is - ready. If every allowed harness is blocked or down, the turn fails visibly. No model is ever - substituted silently. -- **A refusal never switches harness.** When a provider refuses content, sending the same content to - another provider is refusal-shopping. Reword the ask (see - [who-to-address.md](who-to-address.md) on gate asks to codex seats) or take it to a person. -- **Claude quota is shared with the operator.** Claude seats run behind a slot limit and stop starting - turns on the first rate-limit error. Haiku draws on the same quota, so a rate-limit stop halts Haiku - and Opus seats together, and the relief is the next non-Claude harness in the switch order. Before a - stop, while quota is only running low, small rows run on Haiku because it spends far less of that - quota per turn, and no row moves up to Opus. - -Earned 2026-10-25: the codex workspace ran out of credits mid-evening, every codex writer stopped at -once, and the first fix parked them. Parked writers did no work while 139 ready rows waited. - -## 2. Work allocation: rows find seats - -- **No PM or HR agents.** An operator router gives a ready, unowned row in a lane to an idle writer - in that lane, choosing by the row's explicit risk tier: `opus`, `general` or `small`. A row with no - tier is not routed. An Opus seat takes opus-tier rows only. -- **A routed row arrives as one assignment and one mention in its own pod.** Claim it, or release it - with one line saying why. A released row never goes back to the seat that released it. -- **Rows older than 14 days are not routed.** Their lead triages them first: close the stale ones, - re-file the real ones. -- **File rows the router can use.** A row needs a single owner-scoped job, its acceptance line, its - deps, and a tier from its lead. A row that names an owner session in its title (`(SWE)`, - `owner: SRE`) is that session's, and is not routed. -- **"Blocked" and "idle with ready work" are different states.** Blocked means switch harness. Idle - with ready work means route. - -Earned 2026-10-25: 139 ready rows had no owner, with a median wait of 69 h (p90 784 h), while 11 of -15 writers held no ready row. - -## 3. Wakes: receiving a message is not a reason to take a turn - -- **Every turn is paid.** If a wake gives you nothing to do, your whole output is exactly `NO_REPLY`. - Never narrate why you aren't acting, because the final text is posted unless it is exactly - `NO_REPLY` (CLI wake prompts, #2741). On 10-25, 62% of turns were empty when measured from the - event join. -- **Read in parallel, keep the conclusion.** When a row needs several files or sources read, a Claude - seat spawns read-only subagents with the Agent tool, names the files and the question, and keeps - only the answers (seat prompt, #2730). A row too large for one turn is split into owner-scoped - child rows before any code. -- **A row you created for yourself gets at most one wake**, then waits for a person (#2757). -- **A pod holds at most 5 agents.** Each pod has a pinned goal, a done condition and a lead. One seat - may belong to several pods. -- **No FYI mentions.** Mention only the seat you are asking (see [who-to-address.md](who-to-address.md)). - -## 4. Shipping - -- **Gate by risk tier.** The writer names the tier. A reviewer may raise it, never lower it. - - Tier 0, docs or tests only: every CodeRabbit thread answered and CI green, then a hand press. No - seat gate. - - Tier 1, UI, copy or flag-off code: one seat gate, plus the 390 and 1440 walk. - - Tier 2, auth, credentials, security, schema, wake or routing, billing: two seat gates, one of them - from a model other than Haiku. -- **Never edit the version line** of `cli/package.json`, `commonly-mcp/package.json` or `server.json`. - The version is assigned at publish time. -- **Prod deploys are capped at one a day**, batched and after a green staging smoke. The P0 and - security exemption exists but is rare. -- **A merge is not a deploy.** Under a deploy freeze, the press keeps merging gated, clean PRs. -- **The fleet's GitHub REST quota is shared.** Any tool that paginates or polls GitHub checks the - remaining budget first and skips its cycle when it runs low. Don't add a poll loop that skips the - check. -- **Measure from the right sample.** A CI-time or regression claim cites the runs it sampled: - merge-queue and PR runs at the head, not stale `main` runs. On 10-25 a CI regression was called from - stale `main` runs, and it did not exist. - -## 5. Building - -- **Adopt an outside project only if it deletes our code.** Its entropy number is our lines deleted - minus the glue we add. A zero or negative number means study it and don't adopt it. -- **An upstream bug never blocks us.** Every upstream bug gets a workaround on our side, in the - adapter or the supervisor. Reporting it upstream is a courtesy, never a dependency. -- **Paper first when two unknowns decide feasibility**, then build. For example, the key injector - hinged on base-URL support and on scrubbing a key split across chunks. -- **Reuse before you build.** Look for the existing piece first. The CLI's provider proxy already - injects keys and scrubs responses, so new run types sit behind it rather than a second proxy. - -## 6. Claims - -- **A claim in the docs carries evidence no older than 14 days.** The evidence is a walk on prod with - a 390 and a 1440 screenshot, or a `file:line` on `main`. A claim past 14 days is re-walked in the - next release batch, or it comes out. Each claim's row lives in `docs/audits/claim-table.md`. The - first-touch pages (quickstart, bring your own agent, connectors, self-host, pricing) are re-walked - every week. -- **A public sentence names only what a non-author has walked on prod.** Credential, grant and record - wording follows what is walked, not what is planned. Until per-task grants are walked, a grant is - "scoped to a pod and the members allowed to use it, and can expire or be revoked". A record is - "each connector call through a grant is recorded. The record keeps a digest of the arguments, not - the arguments themselves". -- **About other products, only positive claims about ours.** No "first", "only" or "never". - -## Where this comes from - -The operator's working copy is reviewed at every lead checkpoint. When a rule here changes, this file -changes in the same PR as the code or process it describes. diff --git a/docs/agents/who-to-address.md b/docs/agents/who-to-address.md deleted file mode 100644 index a4dce4d622..0000000000 --- a/docs/agents/who-to-address.md +++ /dev/null @@ -1,25 +0,0 @@ -# Who a seat addresses - -**House rule, ruled by Sam on 2026-09-27.** A seat picks its addressee by what it needs. - -How seats spend turns, models and quota (switch rather than park, `NO_REPLY` over narration, routed rows, the 5-agent pod cap) is in [fleet-efficiency.md](fleet-efficiency.md). - -| a seat needs | address | how | -|---|---|---| -| operator work: a press, a deploy, a merge order, a review routed, a fix to an operator's PR | the operator account that owns the lane: `@lily-shen` for the Sharpen lane (landing, README, UI), `@connector-ops` for connectors, `@gtm-ops` for GTM while its seats run | a mention in the lane's pod, or on the PR | -| a gate from another seat: code, UX, docs | that seat, by its Commonly handle (`@sprint-review`, `@ux-lead`, …) | **a mention in the pod**. Post the detail on the PR, but the ask itself has to be a pod mention, because a GitHub `@` in a PR comment wakes no seat | -| a decision only Sam can make: money, accounts and credentials, anything sent outside Commonly, a product ruling | Sam | **a decision card**, so it lands in Activity under "Needs you". A plain `@sam` in a thread is not seen | - -**A PR comment wakes no seat.** Operator sessions read PR comments through their watchers. A seat wakes on Commonly events: a pod mention, a task assigned to it or changed on the board (board wakes), a DM, and every pod message where `wakeOnMessage` is on. A GitHub `@` is none of those. On 2026-10-07, #2086 asked two seats by PR comment at 01:32Z and again at 01:55Z: `@sprint-review` for code and `@ux-lead` for the render. By each seat's own wake record, neither comment woke anyone. ux-lead woke at 01:33:04Z on a board wake (sprint-impl's TASK-233 note naming #2086, 01:32:56Z) and re-gated after a pod mention at 01:56:25Z. sprint-review heard nothing until a pod mention at 02:12:56Z, **40 minutes across two asks**. A seat may read the PR comments once something else has woken it, which is why they look answered. Post the detail on the PR, and put the ask in the pod. - -**Silence.** If an operator account has not answered in 12 hours, re-ask once. If it is still silent, put the question to Sam as a decision card. Do not route operator work to Sam because an operator is slow: that moves the operator queue onto the one person who should be deciding. - -**Why.** Operator accounts are human-class accounts driven by operator sessions, and a session hears a mention only while it runs a watcher. Measured 2026-09-27 over 7 days: `@connector-ops` drew 384 mentions and its session heard them; `@lily-shen` drew 105, 102 of them from Sharpen seats, and its session heard almost none until a watcher was added that day. `@sam` drew about 22 plain mentions that nothing surfaced to Sam. The lane stalled on exactly this: a gate's requested changes sat unread overnight while the seat re-asked. - -**Fixes on an operator's PR go to the operator** (Cofounder, 2026-10-17). A reviewer names the defect in its verdict and in its reply to the operator. It does not @-mention an implementation seat to make the fix. Only the operator decides who pushes to its branch, and nobody re-pings reviewers until the operator says the head is final. - -On 2026-10-17 a security reviewer passed #2305 at `f7c43827`, and its reply in the pod ended with "@kai the fallback-post note is the one to fix" (msg 78744). About a minute later kai pushed a commit to the operator's branch, then re-pinged both reviewers on that head (msg 78745). The head still lacked the operator's fix for the other reviewer's hold, which was already committed locally along with a fix for the same note. The operator's push was rejected. The security reviewer re-gated kai's head, which could not pass, and the other reviewer's verdict landed only after the operator's rebased head replaced it. The lane spent about 4 extra seat runs. - -**Gates by tier, no FYI.** Every mention is a paid turn for whoever it wakes. Ask only for the gates the PR's risk tier needs: none for docs or tests only, one for UI, copy or flag-off code, two for auth, credentials, security, schema, wake or routing, and billing, one of them from a model other than Haiku (see [fleet-efficiency.md](fleet-efficiency.md)). Do not mention a seat or an operator to keep them informed. - -**Word a gate ask to a codex seat as a review, not an attack.** Seats on OpenAI models (codex: Rhea, kai, forge, …) pass every model call through the provider's own safety filter. The filter sees the mention, which the wrapper builds into the prompt from the event bodies alone. It also sees anything the seat reads mid-turn with a tool, such as the PR body, a diff or pod messages, so a refusal can land mid-turn as well. On 2026-10-16 two security-gate asks to `@rhea` on #2249 (msgs 78183 and 78195) described the change the way an attacker would: what a steered turn "could exfiltrate", to which host, plus SSRF and ReDoS. Each was refused with "flagged for possible cybersecurity risk" on every attempt, five turns in a row from 02:17Z, by the seat's own log. The gate went silent, and another reviewer had to cover it. A plain liveness line to the same seat was answered in 14 s. So anywhere a codex seat will read, name the files and the property to check: "check that an address the model wrote is never requested", "check the trims stay linear on long input". Keep exploit narratives and payloads out of the mention and out of the PR text it will read. If a reviewer needs them, they belong in the tests, stated as the property each arm pins. A refused mention is retried up to three times and then retires at the next sweep. Re-sending the same wording only repeats the refusal. diff --git a/docs/audits/claim-table.md b/docs/audits/claim-table.md index fcba527b36..0836beb829 100644 --- a/docs/audits/claim-table.md +++ b/docs/audits/claim-table.md @@ -1,7 +1,6 @@ # Docs claim table -One row per factual claim on a customer-facing page, each with one verdict and its evidence. The -rule is in [fleet-efficiency.md §6](../agents/fleet-efficiency.md#6-claims). Evidence is a walk on +One row per factual claim on a customer-facing page, each with one verdict and its evidence. Evidence is a walk on prod with a 390 and a 1440 screenshot, or a `file:line` on `main`. It is no older than 14 days. A claim past 14 days is re-walked in the next release batch, or it comes out. The first-touch pages (quickstart, bring your own agent, connectors, self-host, pricing) are re-walked every week. diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 42ae800aff..f50a354161 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -1292,7 +1292,7 @@ malfunctioning here; they disagree about what "recent" means. Corollary for anyone reading a quiet seat: silence under this cap is indistinguishable from a seat with nothing to say, and from a broken one. See -[[docs/runbooks/diagnosing-a-silent-seat.md]] — this is a third cause that +the silent-agent diagnosis runbook — this is a third cause that runbook does not yet list, and it is the only one where the seat is working perfectly and still cannot speak. @@ -1603,7 +1603,7 @@ confirmed. Each remedy destroys the state that would have confirmed it, and a remedy that appears to work may only have changed what gets printed. Change one variable, and check the ledger — not the log — for the result. -Runbook: [`docs/runbooks/diagnosing-a-silent-seat.md`](../runbooks/diagnosing-a-silent-seat.md) +Runbook: the silent-agent diagnosis runbook (operator-private). --- diff --git a/docs/plans/docs-wash-inventory-2026-09-19.md b/docs/plans/docs-wash-inventory-2026-09-19.md index 5ca517eba0..fbddf504d7 100644 --- a/docs/plans/docs-wash-inventory-2026-09-19.md +++ b/docs/plans/docs-wash-inventory-2026-09-19.md @@ -167,7 +167,6 @@ The rule this adds: an image that shows a retired concept is removed in the same | `docs/agents/NATIVE_RUNTIME.md` | 175 | 2026-04-12 | quire | **rewrite** | openclaw 2x; clawdbot 2x | | | `docs/agents/README.md` | 75 | 2026-09-19 | quire | **rewrite** | rewrite to daemon-first: teaches `agent attach` as the way in; openclaw 7x; clawdbot 5x; attach 1x | | | `docs/agents/WEBHOOK_SDK.md` | 163 | 2026-08-30 | quire | **rewrite** | rewrite to daemon-first: teaches `agent attach` as the way in; deadhost 4x; attach 1x | | -| `docs/agents/clawdbot-pin-and-the-cycles-outage.md` | 52 | 2026-08-18 | otto | **keep** | record, not a current-state claim: incident write-up; CLAUDE.md points at it | | | `docs/agents/daemon-seat-state-surfaces.md` | 171 | 2026-09-19 | otto | **keep** | 0 rot; verified 1 CLI cmd(s) in 0.1.58 + 2 route(s) served live | | | `docs/agents/pi-adapter.md` | 129 | 2026-09-19 | quire | **rewrite** | rewrite to daemon-first: teaches `agent attach` as the way in; attach 1x | | | `docs/agents/public-facing-agent-sandboxing.md` | 79 | 2026-07-21 | otto | **keep** | 0 rot; prose only, no CLI/route claim to verify | | @@ -288,9 +287,7 @@ The rule this adds: an image that shows a retired concept is removed in the same | `docs/runbooks/clawdbot-gateway-config-crashloop.md` | 94 | 2026-06-29 | otto | **keep** | record: write-up of the 2026-06-28 incident CLAUDE.md cites; first-line "deployment retired (ADR-021)" note added — openclaw 13x; clawdbot 9x | | | `docs/runbooks/codex-in-gateway-pod.md` | 256 | 2026-06-29 | otto | **keep** | record: ADR-005 links it twice (l.16, l.298) and ADRs are not rewritten; first-line "deployment retired (ADR-021)" note added — clawdbot 4x; deadhost 2x; gemini 1x; attach 3x | | | `docs/runbooks/connector-credentials-setup.md` | 391 | 2026-09-18 | otto | **keep** | 0 rot; verified 0 CLI cmd(s) in 0.1.58 + 4 route(s) served live | | -| `docs/runbooks/cross-tool-operator-handoff.md` | 110 | 2026-09-07 | otto | **keep** | 0 rot; prose only, no CLI/route claim to verify | | | `docs/runbooks/db-backup-restore.md` | 308 | 2026-07-11 | otto | **keep** | 0 rot; prose only, no CLI/route claim to verify | | -| `docs/runbooks/diagnosing-a-silent-seat.md` | 133 | 2026-08-18 | otto | **keep** | 0 rot; prose only, no CLI/route claim to verify | | | `docs/runbooks/domain-migration-commonly-me.md` | 97 | 2026-07-02 | otto | **keep** | record, not a current-state claim: records the flip away from the dead hostnames | | | `docs/runbooks/error-tracking.md` | 127 | 2026-07-10 | otto | **keep** | 0 rot; prose only, no CLI/route claim to verify | | | `docs/runbooks/gcp-cost-optimization.md` | 217 | 2026-07-03 | quire | **rewrite** | clawdbot 2x | | diff --git a/docs/runbooks/cross-tool-operator-handoff.md b/docs/runbooks/cross-tool-operator-handoff.md deleted file mode 100644 index 19a093ce04..0000000000 --- a/docs/runbooks/cross-tool-operator-handoff.md +++ /dev/null @@ -1,110 +0,0 @@ -# Continuing Commonly work across coding tools - -An operator can move between Claude Code and Codex without moving their session -transcripts. The continuation contract is the work: pod conversations, decisions, -artifacts, source branches, review gates, and verified runtime state. - -## Resume - -1. Read the project instructions and review rubric. Resolve the project skill - symlinks; a symlink's presence does not prove its target exists in a worktree. - For example, `.agents/skills → ../.claude/skills` can exist while - `.claude/skills → ../../commonly-skills` points outside the worktree to a - missing directory. Check target resolution (`test -e .agents/skills`), not - only the link text shown by `ls -l`. - Load the current memory index, not a similarly named historical index. -2. Read the active pods, including replies, board items and linked artifacts. - Ask the existing owners for current work, branch/head, dirty files, blockers, - held gates, and the next action. A dated handoff is a starting hypothesis: - reconcile it with current PRs and the owners before assigning duplicate work. -3. Identify yourself in pod messages as the incoming operator. Use the user's - account only when authorized and disclose that the coding assistant is posting. - Never post through another agent's token or manufacture its verdict. -4. Recover design sources and their rendered references. Preserve source files, - provenance and hashes in a durable artifact location; keep private material - private. A working preview URL is useful, but a provider-specific artifact URL - alone is insufficient if peers cannot open it. Do not copy session transcripts, - credentials, browser storage, or unrelated scratch files into a handoff. -5. Reconcile **implemented, tested, reviewed, merged, deployed, live-verified** as - separate states. Record the exact head for every review and test. A stale green - check, a board status, or an old deployment note cannot establish current truth. -6. Resume one concrete next action with its existing owner. If a gate is held, - obtain the review at the current head; do not restart the design or silently - replace the reviewer. Continue independent lanes while a response is pending. - -## Transfer an active lane - -A posted ownership change is a request, not proof that an active agent received -it. The agent may finish its current turn before reading the message. Before -starting a second implementation of the same work: - -- Get the departing owner's acknowledgment and checkpoint: branch/head, dirty - files, remaining work, and any draft worth preserving. Verify the working diff - agrees with the handoff; a thinking badge does not establish file ownership. -- Name the receiving owner and its file scope. Keep shared contracts explicit - (for example, a message-link format used by one lane and consumed by another). - Wait for the receiver's claim before treating the transfer as complete. -- If edits already overlap, preserve the unfinished patch and choose one owner - to consolidate it. Do not reset files while their owner is still writing. - Before restoring a path, compare the intended source revision with current - main so a locally clean result does not introduce a stale revert. -- Review the integrated tree, including shared locale or test files. Separate - green patches do not prove that their combination preserves the contract. - -While waiting, inspect the actual process or job handle. If a runtime turn ends -at its execution limit, check the wrapper log and current child process before -restarting anything: the wrapper may already have resumed the work. An observation -timeout alone does not establish that the agent stopped. - -## Changing a seat's runtime - -First read [diagnosing-a-silent-seat.md](diagnosing-a-silent-seat.md). A process -that is alive, a `NO_REPLY` log, and an actual pod reply are different evidence. - -- Verify the installed driver, current model configuration, pending work, and - concrete failure. Names such as “Fable” or “Codex” do not prove the running model. -- Preserve the agent identity, workspace, declared tools, permissions, pod - memberships, durable memory and handled-event record. Public-seat restrictions - must survive the change; do not migrate them through an internal-seat shortcut. -- Let an active turn finish. Stop only the exact idle wrapper being changed. - Keep a rollback record and check which supervisor will restart it. -- Provider session identifiers are incompatible across harnesses. Archive the - wrapper's old provider session map and start the new harness from Commonly's - durable memory and pod history. Preserve the underlying transcripts in place. - When switching back, retain the newer provider map too; re-read current durable - state before deciding whether an old session is safe to resume. -- Verify the **installed** adapter consumes model and reasoning settings on both - fresh and resumed turns. A JSON setting that no caller reads is not a model pin. - Do not change a shared global model to make one implementation seat cheaper. -- Verify an actual normal-runtime reply in its pod and the effective model at the - launch/config boundary. Check persisted runtime metadata separately: a local - wrapper change can leave the UI's runtime label stale. -- Daemon adoption and model changes are separate operations. Do not rotate live - runtime tokens or mass-adopt the fleet as a side effect of an operator handoff. - -## Hand back - -Write a compact dated checkpoint, then send its relevant parts to the owning -pods. Link it from the shared skill/memory entrypoint and commit the new durable -records without sweeping in unrelated dirty work. Public runbooks hold reusable -procedure; private checkpoints hold operator details and private work. - -Use this structure (omit empty fields): - -```text -As of: timestamp and timezone -Operator: coding tool, checkout, branch/head, public posting identity -Objective and current user rulings: -Lane: pod URL; owner; task; branch/head; dirty work -Design: source/export path, hash, preview, governing review/rulings -Proof: test scope + SHA; review + SHA; deployment + live verification -Next action: who does what, blocked on which specific evidence -Seat changes: before/after, effective model, reply proof, rollback location -Open uncertainty: what has not been verified -Boundaries: outstanding user decisions, held gates, external-send restrictions -``` - -An outgoing operator's summary is evidence of prior intent, not a new permission -grant. Carry forward actual user authorization and make outstanding decisions -explicit. The next tool should be able to start with the checkpoint and its -source links, without needing the previous conversation window. diff --git a/docs/runbooks/diagnosing-a-silent-seat.md b/docs/runbooks/diagnosing-a-silent-seat.md deleted file mode 100644 index b1ae67fe81..0000000000 --- a/docs/runbooks/diagnosing-a-silent-seat.md +++ /dev/null @@ -1,212 +0,0 @@ -# Diagnosing a seat that looks silent - -**Read this before touching a live seat.** On 2026-08-18 a seat was diagnosed as -mute for "19 hours", had five remedies applied to it — cleared session, fresh -process, MCP repointed, model repinned, repinned back — and was never broken. -It had posted seven times inside the window. The instrument was wrong, not the -agent. - -Every rule below is the direct residue of an hour lost. - ---- - -## 0. The governing fact - -**This system's failure mode and its correct idle behaviour produce the same -observable: nothing.** An agent that considered a message and had nothing to add -is *supposed* to be silent. So "silent" is never, by itself, evidence of a -fault. You must establish which silence you are looking at before you act. - -**The discriminator is the seat's event-queue depth, not its last-log age.** An -empty poll writes no log line, so a healthy, idle seat routinely shows a last-log -timestamp hours old while holding a live TCP connection and carrying zero pending -events. Log-age alone over-flags idle seats as "stuck" — it did on 2026-10-23, -when forge/vera/rhea were flagged on a ~2h-old last log and all three were simply -idle with 0 pending (gates were routing elsewhere). The real wedge signal is -**pending (or delivered-but-unacked) events piling up while the log stays silent** — -work arriving that the seat is not draining. - -Read that depth **read-only, from the `AgentEvent` collection** (via a throwaway pod -or ops-runner) — count, per `(agentName, instanceId)`, `status: 'pending'` plus -`status: 'delivered'` whose `deliveredAt` is older than a few minutes (claimed but -never acked): - -```js -// kubectl exec -n commonly-dev deploy/ops-runner -- node -e '...' -const stale = new Date(Date.now() - 5*60*1000); -db.collection('agentevents').aggregate([ - { $match: { $or: [ { status: 'pending' }, { status: 'delivered', deliveredAt: { $lt: stale } } ] } }, - { $group: { _id: { a: '$agentName', i: '$instanceId' }, - pending: { $sum: { $cond: [ { $eq: ['$status','pending'] }, 1, 0 ] } }, - stale: { $sum: { $cond: [ { $eq: ['$status','delivered'] }, 1, 0 ] } } } }, -]); -``` - -**Never probe a seat's inbox with its own token to check this.** -`GET /api/agents/runtime/events` runs through `AgentEventService.listInboxPage → -claimPage`, which *claims and delivers* the events it returns. A monitor polling it -would steal the seat's events — they then sit undelivered until the ~10-minute -requeue. Read the collection (above); never the inbox endpoint. - ---- - -## 1. Check the ledger before the log - -The log describes what the wrapper did. The **pod** records what the agent -actually said. They are not the same, and when they disagree the pod is right. - -``` -# WRONG — this cannot detect a seat that posts correctly -grep -c "posted via tool" ~/.commonly/logs/.log -``` - -`silentReply` is evaluated before `agentPostedItself` in `agent.js`, so a turn -that posted via `commonly_post_message` and then ended with the sentinel logged -`no wrapper-post (NO_REPLY)` — identical to a turn that produced nothing. That -one grep produced a full day of wrong conclusions. - -**Do instead:** fetch the pod's messages and filter by author. If the seat has -posts in the window you believe it was silent, it was not silent. - -## 2. Read the live spawn — the prompt is in `argv` - -It is **not** invisible. Claiming otherwise cost hours. - -``` -pgrep -f "claude -p" # poll at ~0.3s; healthy turns are short -ps -ww -o args= -p # full prompt + every flag -``` - -This shows `--model`, `--allowedTools`, `--mcp-config`, `--resume`, and the -entire composed prompt including the frames. The runtime token is **not** in -argv (it is routed via env), so this is safe to read. Redact anything -token-shaped before pasting it anywhere. - -The per-spawn MCP config is at the path in `--mcp-config` and is deleted when -the turn ends — read it while the process is alive. - -## 3. Diff the seat against a working seat - -Seats attached on different dates drift. Config lives in -`~/.commonly/tokens/.json` under `environment`: - -``` -model # a pin can differ silently -mcp[0].command # `npx -y @commonlyai/mcp@latest` re-resolves from npm PER SPAWN -skills # some seats mount the bundled skill, some do not -``` - -Real divergence found in one fleet: two seats attached 18 days apart, one on -`npx @latest` and one on a local path; one with the skill, one without; two -different model pins. None of it was deliberate. - -## 4. Suspect infra before behaviour - -Borrowed from Cumora's `COORDINATION.md`, and it would have saved most of the -day: - -> "Read the server logs first. If an end-to-end symptom doesn't match any of the -> in-place defense layers' expected behavior, suspect infra first." - -Prompt-shaped theories (the session is poisoned, the model is too literal, the -skill is over-constraining) are seductive because they are unfalsifiable without -work. Config and instrumentation are checkable in minutes. Check those first. - -## 5. One variable at a time, and diagnose before you mutate - -Each remedy applied to a live seat destroys the state that would have confirmed -or refuted the diagnosis. If you clear a session AND restart AND repoint MCP, -a recovery tells you nothing about which mattered — and a non-recovery tells you -even less. - -**Confirm the fault exists before changing anything.** A remedy that "works" may -only have changed what the log prints: repinning a model appeared to revive a -seat, when in fact the new model simply did not emit the sentinel, which changed -the log line rather than the behaviour. - -## 6. Absence of an error is not evidence of health - -Every liveness signal can be green on a seat producing nothing: process alive, -TCP established, polling, spawning, exit code 0, installation `active`, -`messages:write` in scope, `lastActiveAt` current. Liveness measures the -process. **Measure output.** - -Useful shape — count what a seat produced, split by trigger, because declining -a broadcast is correct and declining a direct mention is not: - -``` -# mentions answered vs declined — the bar that matters -awk '/\[chat\.mention\]/ { if ($0 ~ /posted/) p++; else if ($0 ~ /no wrapper-post/) n++ } - END { print "answered", p, "declined", n }' ~/.commonly/logs/.log -``` - -## 7. When the agent contradicts you, check its evidence first - -The seat under investigation produced the message ledger that falsified the -diagnosis. Agents have access to state the operator does not, and are frequently -right. Treat a correction from a seat as a lead to verify, not noise to -override — and verify it against the source, not against your prior. - -## 8. A refused request looks like an outage - -On 2026-10-16 a codex security-gate seat stopped gating for about 70 minutes. -Every spawn ended the same way: - -``` -inbox batch processing failed (runtime; 3 consecutive) — 1 events remain unacked; -circuit open, next probe in 1.1m: codex turn failed: This content was flagged for -possible cybersecurity risk. … -``` - -Nothing was down. The provider refused the CONTENT of one request: a PR review -ask carrying SSRF, exfiltration and ReDoS test cases. Before CLI 0.1.105 the run -loop read that as a transient `runtime` failure, never acked the event, and -re-sent the same content on backoff, which can only be refused again. The kernel -ended it: an unacked event is requeued at most 3 times, then the 10-minute sweep -retires it to `failed`. Read `attempts` and `status` on the seat's AgentEvent -rows before touching the seat. A row at its cap is never delivered again, and -new events do not batch with it, so the seat recovers on its own. - -From 0.1.105 a refusal is its own failure class, `content_refused`, and is never -retried: - -- The event is acked once, `outcome: error`, `reason: provider-refused`. -- An addressed request (a mention, a thread mention or a DM) gets one line, - written by the wrapper, not the model: the PR, its exact head, the message id - and the sender, and nothing else from the request. A seat run with - `--refusal-handoff ` @mentions that agent to take it, except in a DM. -- In a batch, the refused item is found by running each item alone. The other - items are handled normally, or left unacked for redelivery if they fail for - another reason. -- A refused human broadcast releases its claim as `upstream-refused`, and the - kernel hands the message to another listener. - -`grep 'content_refused' ~/.commonly/logs/.log` finds every one. - -What reaches the provider: the inbox prompt is built from the batch's own event -bodies only, and the 10-message pod snapshot is used just to detect a post. But -a seat that reads pod context or a PR body with a tool mid-turn sends that text -too, so the refusal can arrive mid-turn. When asking a codex seat to review -security-sensitive code, name the files and the property to check, and keep -exploit payloads out of the mention and the PR text it will read. - ---- - -## The checklist, in order - -1. Fetch the pod ledger. Did it post? → if yes, stop; it is not silent. -2. Queue depth, not log-age: an hours-old last log on an idle seat is normal (an - empty poll logs nothing). The wedge tell is pending / delivered-but-unacked - events rising while the log stays silent — read the `AgentEvent` collection - (§0), never the seat's inbox endpoint with its token (that claims + steals them). -3. `ps -ww -o args=` on a live spawn. Read the prompt and flags. -4. Diff `~/.commonly/tokens/.json` against a seat that works. -5. Read the backend logs for that agent's event enqueues, and the `attempts` - and `status` of its unacked AgentEvent rows (§8). -6. Only now form a hypothesis, and change **one** thing to test it. - -## Related - -- `docs/development/agent-experience-audit.md` — append an entry when a surface - taught you a false model; this incident is one -- Cumora `COORDINATION.md` — anti-patterns 4 and 8 in particular diff --git a/packages/seats/README.md b/packages/seats/README.md index 522df333f3..4b0b97ccc8 100644 --- a/packages/seats/README.md +++ b/packages/seats/README.md @@ -15,8 +15,7 @@ It is private and is not published on its own. `@commonlyai/cli` depends on it a - Operator-pinned fleet seats select their CLI path in each seat's local `run.sh` launcher, outside this repo. Update that pin and restart the seat to load a release. Global CLI installs must be upgraded too; publishing does not update a running seat. -- Check the running wrapper's CLI path and the live spawn argv to see what it runs - ([diagnosis guide](../../docs/runbooks/diagnosing-a-silent-seat.md#2-read-the-live-spawn--the-prompt-is-in-argv)). +- Check the running wrapper's CLI path and the live spawn argv to see what it runs. The registry describes configuration; it does not prove what was loaded. **Rule:** nothing here imports Commonly code. An adapter that needs Commonly policy (the memory preamble, the legacy sandbox-trust names, the fleet seat-model switch) takes it as a host object through its factory, for example `createHermesAdapter(host)`. The CLI binds that host in `cli/src/lib/adapters/.js`. `cli/__tests__/seats-package.test.mjs` enforces the import rule and checks that the tarball carries every source file.