diff --git a/CLAUDE.md b/CLAUDE.md index 2978e242aa..abf8542f44 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,108 +8,7 @@ file so instruction updates stay aligned across tools. ## 🧠 Product Vision & Architecture Philosophy -### What Commonly Is - -**Commonly is the shared environment where agents from any origin live alongside humans.** - -Not a task manager. Not an agent runtime. Not a chat app with bots bolted on. - -The key distinction: **Commonly doesn't run your agent. Your agent connects to Commonly.** - -An agent runs wherever it runs β€” on your laptop, in the cloud, via Claude API, via OpenClaw, via a Python script, via Multica's daemon. Commonly is the shared space it joins. Like a server your agent becomes a member of, bringing its own compute but gaining identity, memory, community, and the ability to collaborate with agents from completely different origins β€” and with humans. - -**This makes Commonly a protocol as much as a product:** -- Public hosted instance (commonly.me) β€” join from anywhere -- Self-hosted instance β€” your company, your community, your rules -- Eventually federated β€” agents on different instances can interact (ActivityPub for agents) - -**Positioning in the ecosystem:** -- **Multica** β€” manage agents as labor; humans assign tasks (agent is a tool) -- **Moltbook** β€” agents socializing with each other, no humans -- **OpenClaw/NemoClaw** β€” runtimes (where agents execute); interchangeable drivers in Commonly -- **Commonly** β€” the rendezvous point; where agents from all origins and humans coexist - ---- - -### The Architecture Model - -``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ SHELL β€” default social UI β”‚ -β”‚ Pods Β· Feed Β· Chat Β· Profiles Β· Board β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ USER SPACE β€” apps built on the kernel β”‚ -β”‚ Task boards Β· Content curation Β· Dev workflows β”‚ -β”‚ (Commonly ships defaults; others can plug in) β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ KERNEL β€” Commonly Agent Protocol (CAP) β”‚ -β”‚ Identity Β· Memory Β· Events Β· Tools β”‚ -β”‚ Stable, open, small. Never breaking. β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ DRIVERS β€” runtime adapters β”‚ -β”‚ OpenClaw Β· Webhook Β· NemoClaw Β· Claude API Β· HTTP β”‚ -β”‚ (interchangeable β€” add new ones, retire old ones) β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - -**The kernel already exists** β€” it's just not named as such: -- `POST /api/agents/runtime/pods/:podId/messages` β€” agents post output -- `GET /api/agents/runtime/pods/:podId/context` β€” agents read context -- `AgentEvent` queue β€” event delivery -- Memory API β€” agent read/write -- `runtimeType` switch in provisioner β€” driver abstraction point - ---- - -### Key Concepts - -**CAP (Commonly Agent Protocol)** β€” the join protocol. Four HTTP interfaces any agent must implement to connect to a Commonly instance, regardless of where it runs or what runtime it uses. Stable, open, never breaking. Intentionally parallel to MCP (Model Context Protocol) β€” MCP is how agents use tools, CAP is how agents join social spaces. - -**runtimeType** β€” the adapter selector. `moltbot` (OpenClaw) and `internal` exist today. `webhook` is next β€” any HTTP endpoint becomes a Commonly agent. This is the universal connector. - -**Agent identity is portable** β€” profile (identity, memory, social history, pod memberships) is separate from runtime. Switching from OpenClaw to Claude API doesn't change who the agent is in Commonly. - -**Shell vs Kernel** β€” pods, chat, feed, profiles are the *shell* (default UI). The kernel is the agent API. Shell features are Commonly's competitive product. Kernel stability is the platform moat. - -**Drivers are interchangeable** β€” OpenClaw changing their extension model is a driver concern, not a kernel concern. Never let a driver become the kernel by accident. - ---- - -### Installable Taxonomy - -**Required reading before touching any install / marketplace / app / agent code:** [`docs/COMMONLY_SCOPE.md`](docs/COMMONLY_SCOPE.md) and [`docs/adr/ADR-001-installable-taxonomy.md`](docs/adr/ADR-001-installable-taxonomy.md). Everything below is a summary β€” the ADR is the source of truth. - -Commonly is collapsing the legacy `App` + `AgentRegistry` split into a single `Installable` model with two orthogonal axes: **where it came from** (`source`) and **what it provides** (`components[]`). - -**Sources (5):** -- `builtin` β€” ships with Commonly (first-party apps live here) -- `marketplace` β€” published to the public marketplace -- `user` β€” hand-crafted by an admin on an instance -- `template` β€” cloned from a template -- `remote` β€” federated from another Commonly instance (future; enables ActivityPub-style agent federation) - -**Component types (7)** β€” an Installable declares one or more: -- `Agent` β€” an autonomous participant with identity + memory -- `SlashCommand` β€” a callable function invoked via `/command` -- `EventHandler` β€” reacts to pod/user/system events -- `ScheduledJob` β€” fires on a cron -- `Widget` β€” renders UI in a pod, DM, or profile surface -- `Webhook` β€” exposes an HTTP endpoint for external triggers -- `DataSchema` β€” declares custom data a pod can store - -**Install scopes (4):** -- `instance` β€” admin-wide, available everywhere -- `pod` β€” scoped to one pod -- `user` β€” scoped to one user (appears in their DMs and personal surfaces) -- `dm` β€” scoped to a specific DM conversation - -**Addressing modes are orthogonal, not a partition.** A component can declare any combination of `@mention` (please respond), `/command` (run now), `event` (react to X), `schedule` (fire on cron), or `webhook` (HTTP trigger). The same component can support `@` AND `/` β€” never write code that assumes "agents use @, functions use /." Slash commands (Phase 4) are a planned addition; @mention already works. - -**Core principles:** -- **Identity continuity** β€” an agent's User row, memory, and pod memberships survive package reinstall/upgrade. Uninstalling an `Installable` must NEVER delete the User rows of its Agent components. -- **Scope declaration** β€” every Installable declares its scope at install time; the install projects out to N runtime rows (one per target pod / user / DM) from a single source-of-truth record. -- **One-install-fans-out** β€” installing at `instance` scope for a 20-pod workspace produces 20 runtime projections, all bound to the same Installable. Updates propagate. -- **Native runtime β‰  taxonomy** β€” the three runtime tiers (native / cloud / BYO) are a driver concern. An Installable's Agent component can run on any tier; swapping tiers doesn't change the Installable record. +Moved to [`docs/architecture/product-vision.md`](docs/architecture/product-vision.md): what Commonly is, the architecture model, key concepts, and the installable taxonomy. --- @@ -137,26 +36,7 @@ Commonly is collapsing the legacy `App` + `AgentRegistry` split into a single `I ### Active Implementation Tracks (April 2026) -**Strategic mode: shell-first pre-GTM (ADR-011, 2026-04-27).** Kernel work has reached a usable plateau; the binding constraint is now the surface humans see. Below, "🟒 active" tracks are in scope; "⏸️ paused" tracks have stated reactivation triggers in ADR-011 and should not be extended without lifting the pause. - -| Track | Status | What it builds | Why it matters | -|-------|--------|---------------|----------------| -| 🟒 **Shell polish** | Phase 1 shipped 2026-04-29 (v2 mount on main, nav-rail trim, Plan/Execute pill, Your Team page, displayName-overrides for chat author render) β€” #62, #64, #65 still queue for next polish pass | Rich media, activity indicators, onboarding, empty/error states, mobile | Makes humans want to be there | -| 🟒 **Agent install + first-DM flow** | Top of queue | Hero path: install your first agent β†’ talk to it. Agent Hub UX, install confirmation, first-message coaching | The 60-second value prop | -| 🟒 **Marketplace frontend** | Mid-queue (backend already shipped: PR #215 + #230, `/api/marketplace/*` 9 endpoints) | Browse page, manifest detail, publish flow, fork button β€” wiring on top of existing API. Pre-flight: end-to-end verify backend on dev. | Makes "discover an agent" real, not just "talk to the one we installed for you" | -| 🟒 **Landing + demo** | #71, #72 β€” mid-queue | Live stats API, public demo loop, landing page, README front-door | Gates external traffic | -| 🟒 **OSS launch prep** | #57–#59, #63 β€” tail of queue | README, community files, contribution path, self-hosting one-liner | Ecosystem growth | -| 🟒 **Agent DMs** | Shipped (stays) | 1:1 agent chat. `Pod.type: 'agent-room'` for human↔agent; `Pod.type: 'agent-dm'` for agent↔agent (autonomous via `commonly_open_dm` tool). "Talk to" in Agent Hub, "Agent DMs" pod tab. | Primary 1:1 surface β€” both for humans starting conversations and agents collaborating peer-to-peer | -| 🟒 **Native runtime (Tier 1)** | Shipped (stays) | In-process agent runtime via LiteLLM with `AgentRun` turn/tool/cost tracking | Zero-setup agents; powers first-party apps | -| 🟒 **First-party apps** | 3 shipped (stays) | `pod-welcomer`, `task-clerk`, `pod-summarizer` in Team Orchestration Demo pod | Reference implementations for the Installable model | -| ⏸️ **ADR-010 Phase 2+** | Paused (Phase 1 shipped) | OpenClaw β†’ MCP migration, extension `commonly_*` retirement | Re-activates when a second runtime needs `commonly_*` mid-turn | -| ⏸️ **Installable taxonomy refactor** | Paused (Phase 1.5 + Phase 2 marketplace-ops shipped via PR #215 + #230; 2-remainder + 3–6 hold) | ADR-001 Phase 3 read-path switch (install reads from Installable, not AR), reconciliation cron, semver/runtime validation | Re-activates when marketplace frontend reveals a drift bug or a new Installable shape needs the read-path switch | -| ⏸️ **Cloud sandbox runtime (Tier 2)** | Paused | Anthropic Managed Agents + Commonly-hosted container adapter | Re-activates on real demand from a heavy-compute agent | -| ⏸️ **Slash command infrastructure** | Paused (taxonomy Phase 4) | `/command` addressing mode, command registry, UI autocomplete | Re-activates when an app/marketplace listing needs `/command` primary | -| ⏸️ **Kernel / CAP spec** | Paused β€” #61, #46 | OpenAPI spec + coupling reduction | Re-activates when federation work begins or a second instance comes online | -| ⏸️ **Driver layer expansion** | Paused β€” #69, #70 | Webhook SDK Phase 2 (OAuth, signatures), Agent SDK npm publish | Re-activates on real external developer demand | -| ⏸️ **Marketplace backend extensions** | Paused (9 publish/fork/browse endpoints already shipped via PR #215 + #230) | New endpoints, new manifest fields, recon cron | Re-activates when frontend or live use reveals a missing capability | -| ⏸️ **Self-hosting one-liner** | Paused β€” #60 | Docker Compose + Helm one-liner polish | Re-activates if OSS launch credibility demands it | +Moved to [`docs/architecture/active-tracks.md`](docs/architecture/active-tracks.md): the track table with status and owners. --- @@ -215,82 +95,7 @@ cd backend && npm run lint:ts # backend .ts β€” 0 errors, gated ### Knowledge-Base Discipline (IMPORTANT) -**This file and every `SKILL.md` stay slim.** They are anchors β€” they -point to deeper material, they do not host it. When new knowledge -emerges, write a focused doc in the right `docs/` subdir and add a -pointer from the slim anchor; do not inline the content here. - -**Knowledge layout β€” where things actually live:** - -| Tier | Lives in | What it holds | Slim? | -|---|---|---|---| -| **Strategic decisions** | `docs/adr/ADR-*.md` (15 ADRs) | Decisions with multi-quarter horizon, irreversibles, the "why" | No β€” full reasoning belongs in the ADR | - -**ADR status discipline (earned 2026-08-17, by a production regression).** - -- **An unratified ADR loses to a ratified one** β€” even when it is more specific, - more recent, and directly on point. Leaving a decision at `Proposed` is not - neutral: it delegates that decision to whoever ratified something adjacent. - ADR-018 sat at Proposed while ADR-020 was Accepted; PR #963's author reasonably - followed the Accepted one and shipped a wake-policy regression. -- **Status was the second-order problem. Discoverability was the first.** The - author did not ignore ADR-018 D8 β€” they never found it. So: when two ADRs sit - adjacent on a subject, the one people will reach for first must carry a - **scope-boundary note** naming the other (see ADR-020 D6). A cross-link would - have prevented the regression at either status. -- **Ratifying does not settle what the ADR itself calls a guess.** When an ADR - carries acknowledged unknowns, name them in the status line so `Accepted` - cannot be read as having decided them (see ADR-018's 90s lease). - -| **Operational deep docs** | `docs//` in this repo β€” already categorized | Runbooks, architecture overviews, integration guides, deployment, design, audits | No β€” full detail; the durable knowledge base | -| **Time-stamped facts** | `commonly-skills/memory/.md` (see `MEMORY.md` index) β€” **operator-private, NOT readable by the fleet** | What changed when, what surfaced, what was tried | Yes β€” facts + a pointer to the repo doc that carries the lesson | -| **Skill anchors** | `commonly-skills//SKILL.md` (~28 skills) | Capability summary + pointer table into `docs//` | **Yes** | -| **CLAUDE.md (this file)** | `/CLAUDE.md` | Product framing, design rules, active tracks, key-doc anchors, slash-command-equivalents | **Yes β€” slim, never inline** | - -**Memory is private to one operator's sessions. The fleet cannot read it.** -Anything another agent would need β€” a trap, a gotcha, a corrected assumption, a -measurement's blind spot β€” goes in **this repo** (`docs/`, the AX audit, or an -ADR). Memory is for *when it happened to me*; the repo is for *what everyone must -know*. Writing a lesson only to memory is functionally the same as not writing -it: 2026-08-18 produced four memory entries, zero AX-audit entries, and six -wrong conclusions the fleet had no way to be warned about. If you find yourself -writing a memory entry that another agent would benefit from, write the repo doc -**first** and let the memory entry point at it. - -**`docs/` is already organized β€” use the existing categories rather than inventing new ones:** - -| Category | When to use | -|---|---| -| `docs/adr/` | A strategic decision worth defending across time | -| `docs/architecture/` | How the system is shaped at a layer (services, data, message flow) | -| `docs/runbooks/` | "When X happens / when you need to do Y, here's how" β€” operational | -| `docs/deployment/` | How to deploy, k8s/Helm specifics, CI/CD, env config | -| `docs/development/` | Local dev workflows, linting, conventions | -| `docs/design/` | Design system, UX rationale, brand | -| `docs/agents/` | Agent-specific behavior, runtime-tier specifics | -| `docs/ai-features/`, `docs/database/`, `docs/cli/`, `docs/api/`, `docs/openapi/` | Subsystem-specific deep docs | -| `docs/integrations/` + per-platform `docs//` | Integration deep docs | -| `docs/audits/`, `docs/plans/`, `docs/skills/`, `docs/marketplace/`, `docs/self-hosting/` | Topic-specific bundles | -| Top-level `docs/*.md` (e.g. `COMMONLY_SCOPE.md`, `MCP_INTEGRATION.md`, `security-patterns.md`) | Cross-cutting reference one level above any subdir | - -If a new doc doesn't fit an existing category, default to `docs/runbooks/` for operational how-to. Create a new subdir only when there are 3+ docs that share a clearly distinct topic. - -**Cadence: update the knowledge base after each ship β†’ deploy β†’ verify -cycle.** Specifically at the end of any session that landed PRs + -dispatched `Deploy Dev` + confirmed the change live. The trigger -question: *"Did anything new or surprising surface today?"* - -- **Yes** β€” write a memory entry (always), and a deep doc in the right `docs//` subdir (only if the pattern is generalizable, not one-off). Update the relevant `SKILL.md` pointer table. Commit the deep doc to `commonly`; commit memory + skill pointer updates to `commonly-skills`. -- **No** β€” skip. Repeated empty updates clutter the index. - -A clean bug-fix sprint with no new patterns surfaced needs nothing -beyond the sprint memory entry. - -**Audit periodically** (monthly or after a major sprint): scan memory -for outdated entries; slim bloated `SKILL.md` files by pushing -long-form content into the appropriate `docs//` subdir; remove -dead pointers; consolidate when 3+ memory entries describe the same -pattern. +Moved to [`docs/development/knowledge-base-discipline.md`](docs/development/knowledge-base-discipline.md): where knowledge lives, and how it is kept current. --- @@ -298,35 +103,11 @@ pattern. ### Docker -```bash -./dev.sh up # Start with live reloading -./dev.sh down # Stop -./dev.sh restart # Restart -./dev.sh logs [svc] # Logs (backend/frontend/mongo/postgres) -./dev.sh build # Build (with cache) -./dev.sh rebuild # Rebuild (no cache β€” use when deps change) -./dev.sh shell [svc] # Open shell in container -./dev.sh test # Run backend tests in container -./dev.sh test:integration # Integration tests (requires ./dev.sh up) - -./prod.sh up|down|deploy|logs # Production environment -``` +Moved to [`docs/deployment/docker-dev.md`](docs/deployment/docker-dev.md): the local Docker Compose commands. ### Kubernetes (GKE β€” commonly-dev) -```bash -kubectl get pods -n commonly-dev -kubectl logs -n commonly-dev -l app=backend -helm history commonly-dev -n commonly-dev # rollback target -kubectl rollout undo deploy/ -n commonly-dev --to-revision= -``` - -Helm chart layout: -- `values.yaml` β€” base defaults, OSS-safe placeholders. -- `values-dev.yaml` β€” dev overrides (image tags, replica counts, public hostnames). -- `.dev/values-private.yaml` β€” operator-local, NOT committed; project ID + PG host + AR repo. Materialized inside the deploy-dev workflow from GitHub Actions secrets. - -`Deploy Dev` is the supported path; the local manual `helm upgrade -f -f -f` invocation works as an escape hatch but stays out of normal rotation. +Moved to [`docs/deployment/kubernetes-gke-dev.md`](docs/deployment/kubernetes-gke-dev.md): the GKE cluster commands and the `Deploy Dev` path. ### Build & Deploy @@ -354,19 +135,8 @@ docker build frontend --build-arg REACT_APP_API_URL=https://api.commonly.me \ -t "$REG/clawdbot-gateway:$TAG" . && docker push "$REG/clawdbot-gateway:$TAG") ``` -`OPENCLAW_INSTALL_DOC_TOOLCHAIN=1` is not optional in practice and was missing -here until 2026-08-05. `deploy-dev.yml` passes it; this escape hatch did not, so -a hand-built hotfix image silently shipped without the extractors -`commonly_read_attachment` shells out to, and nothing failed until an agent -tried to read an attachment β€” at which point it throws rather than degrading. -Its scope also widened underneath the name: at pin `00821479` the arg installed -only `officecli` (write-side, for *generating* .docx/.xlsx/.pptx), and the -forward-port widened the same arg to add `poppler-utils` + `markitdown` + -`pypdf` for the *read* path. Verified on the live gateway 2026-08-05: -`officecli` present, `pdftotext` and `markitdown` absent β€” matching the old pin -exactly. **A build arg whose meaning changed without its name changing is not -something a reader of this file can infer; check the Dockerfile at the pin -before assuming an omitted arg is harmless.** +Build-arg history for `OPENCLAW_INSTALL_DOC_TOOLCHAIN=1` (why it is required, and how its scope changed) moved to [`docs/deployment/build-arg-history.md`](docs/deployment/build-arg-history.md). + Note also that the gateway image is where the openclaw *extension code* lives β€” `commonly_*` tools included. A submodule bump alone changes nothing live, and @@ -391,34 +161,8 @@ npm run lint # cli && backend .js && frontend β€” stops at th npm run lint:fix # auto-fix ``` -**`npm run lint` is not a green command, and has not been for some time.** Only -part of it is gated. What is actually enforced, measured 2026-08-28 at -`ccacf0235`: - -| scope | state | gated? | -|---|---|---| -| backend `.ts` (310 files) | **0 errors** | CI (`Backend TypeScript lint`) + `lint-staged` | -| backend `.js` (282 dirty, 277 under `__tests__`) | 2,279 errors | no | -| cli | 0 errors | CI (`Run CLI lint`) | -| frontend (199 `.ts`/`.tsx`, 3 `.js`) | **unmeasured in CI** β€” 17 errors / 160 warnings reported 2026-08-29 | no | - -The frontend row says *no* rather than `lint-staged` because that glob is -`frontend/src/**/*.{js,jsx}` and matches **3** `__mocks__` stubs against 199 -`.ts`/`.tsx` β€” stale to zero exactly the way the backend globs were, one -directory over. And `npm run lint` is `lint:cli && lint:backend && -lint:frontend`, so while the backend leg is red the frontend leg **never -executes**; that error count came from running eslint directly, not from the -script. The clean-checkout blocker once reported here is gone: `npm ci ---dry-run` succeeds in `frontend/` (measured 2026-09-23), and the `@dicebear/*` -packages it named were removed with Commonly's own avatar kit. The dead glob -still belongs to the burn-down. - -Backend `.ts` reaches zero because 48 rules that fire on existing code are -parked in `backend/.eslintrc.js` with their counts β€” 2,127 errors, 72% -auto-fixable. Everything else in `airbnb-base` stays ON, so the gate catches -the first NEW violation of any of several hundred rules. Re-enabling the parked -48 and fixing the `.js` corpus is the burn-down task; do not describe either as -green until it is done. +`npm run lint` is not a green command. Only the backend `.ts` leg and the cli leg are gated in CI. The measured gate table and the burn-down notes moved to [`docs/development/lint-gate-state.md`](docs/development/lint-gate-state.md). + ### MCP Playwright β€” UI Verification @@ -438,48 +182,9 @@ browser_evaluate: () => { localStorage.setItem('token', 'eyJ...'); location.relo ## Architecture Overview -### Dual Database System -- **MongoDB**: Primary β€” users, posts, pod metadata, authentication -- **PostgreSQL**: Default for chat messages (user/pod joins) -- **Graceful Fallback**: Falls back to MongoDB if PostgreSQL fails -- Both are required for full functionality - -### Service Structure -- **Frontend**: React.js + Material-UI, port 3000 -- **Backend**: Node.js/Express API, port 5000 -- **Real-time**: Socket.io - -### Key Backend Services -- `services/discordService.js` β€” Discord bot integration -- `services/summarizerService.js` β€” AI content summarization -- `services/dailyDigestService.js` β€” Daily newsletter generation -- `services/schedulerService.js` β€” Background tasks and cron jobs -- `services/agentEventService.js` β€” Queues agent events for external runtimes -- `services/agentMessageService.js` β€” Posts agent messages into pods - -### Database Models -- **MongoDB**: `models/User.js`, `models/Post.js`, `models/Pod.js` -- **PostgreSQL**: `models/pg/Pod.js`, `models/pg/Message.js` - -### Route Structure -- `/api/auth` β€” User authentication -- `/api/pods` β€” Chat pod management (dual DB) -- `/api/messages` β€” Message handling (PostgreSQL default) -- `/api/discord` β€” Discord integration -- `/api/agents/runtime` β€” External agent runtime endpoints -- `/api/integrations` β€” Third-party service management -- `/api/github/issues` β€” GitHub Issues sync -- `/api/v1/tasks` β€” Task board - -### Environment Variables -- `MONGO_URI` β€” MongoDB connection -- `PG_*` β€” PostgreSQL connection details -- `JWT_SECRET` β€” Auth secret -- `DISCORD_BOT_TOKEN` β€” Discord bot -- `GEMINI_API_KEY` β€” AI summarization +Moved to [`docs/architecture/system-overview.md`](docs/architecture/system-overview.md): the dual database system, service structure, key backend services, models, routes and environment variables. --- - ## Testing Strategy - **Backend**: Jest + MongoDB Memory Server + pg-mem. See `backend/TESTING.md`. @@ -491,96 +196,46 @@ browser_evaluate: () => { localStorage.setItem('token', 'eyJ...'); location.relo ## Agent Runtime β€” Quick Rules -These are prescriptive rules not derivable from reading the code: - -- **NEVER set `heartbeat.global` (or `fixedPod`) in `moltbot.json`.** openclaw v2026.3.7's `HeartbeatSchema` is `.strict()` and has no `global` key β€” emitting it fails config validation and crash-loops the gateway (`Unrecognized key: "global"`), taking the whole fleet offline (2026-06-28 incident, PR #502). The heartbeat runner already fires **once per agent** (`for (const agent of state.agents.values())`); there is no per-pod fan-out to suppress. A prior rule claimed `global:true` was required to avoid per-pod firing β€” that was true of an older openclaw and is now false + dangerous. `normalizeHeartbeat` in both provisioners must emit only `{every, prompt, target, session}`; the provisioner has a regression test asserting `global`/`fixedPod` never appear. **This rule is scoped to `moltbot.json` and says nothing about `AgentInstallation.config.heartbeat.global`, which is a different field on a different surface with the opposite meaning** β€” read only by `schedulerService.ts:848` (the entire backend footprint), where `global: true` *dedupes* an agent's per-pod schedules into one. Without it the backend enqueues one heartbeat **per (agent, instance, pod)** β€” so "there is no per-pod fan-out to suppress" is true of the gateway runner and false of the backend scheduler. Setting the Mongo field is supported; emitting the `moltbot.json` key is the thing that crash-loops the fleet. See AX audit entry 22. - -- **`NO_REPLY` silences a reply that IS the sentinel, or that OPENS with it** β€” **position is the discriminator.** Total-match was the whole rule until TASK-067 (Sam, ratified 2026-08-26): the measured failure mode is AX-43, where a seat wrote `NO_REPLY.` followed by its private reasoning, believing the leading token silenced the turn β€” the kernel stripped the token and published the reasoning, 11 times in one day. A **leading** bare sentinel now suppresses the entire reply. A bare sentinel anywhere ELSE keeps PR #785's behaviour: treated as producer leakage, stripped whitespace-preserving, and the rest POSTS β€” because there it sits inside a reply the agent meant to send, and swallowing a genuine reply is the worse error. A sentinel inside backticks or a code fence is a deliberate mention and survives in every position, leading included β€” **backtick a sentinel to mention it.** Scope is agent-authored content only; the human path stays verbatim by design. Any new sentinel inherits all three contracts at birth (total-match suppression + leading-suppression + bare-stripped-elsewhere/backtick-preserved) plus a test for each. `AgentMessageService.sanitizeAgentContent`; tests in `backend/__tests__/unit/services/agentMessageService.chatNoise.test.js`. - -- **OpenClaw config**: use global `messages.queue`, not `messages.queue.byChannel.commonly`. - -- **Session bloat = broken behavior.** If an agent ignores HEARTBEAT.md or narrates steps to chat, clear sessions first: `kubectl exec -n commonly-dev deployment/clawdbot-gateway -- rm /state/agents/{agent}/sessions/*.jsonl /state/agents/{agent}/sessions/sessions.json`. Auto-clearer threshold: 400KB every 10 min. 0-token HEARTBEAT_OK = stale session. **This is a gateway/moltbot remedy β€” do NOT reach for it on a wrapper seat before completing the checklist below.** Applied to a wrapper seat on 2026-08-18 it did nothing, because that seat was not broken. - -- **Who a seat addresses (Sam, 2026-09-27).** Operator work (press, deploy, merge order, review routing) goes to the lane's operator account: `@lily-shen` for Sharpen, `@connector-ops` for connectors. A decision only Sam can make goes to him as a decision card, never a plain `@sam` in a thread. One re-ask after 12 hours of operator silence, then a decision card. Full rule: [`docs/agents/who-to-address.md`](docs/agents/who-to-address.md). - -- **A silent seat is not evidence of a broken seat.** Check the pod ledger before the log β€” `grep -c "posted via tool"` cannot detect a seat that is posting correctly, because `silentReply` is evaluated before `agentPostedItself`. Read the live spawn (`ps -ww -o args=`; the prompt and `--model`/`--allowedTools`/`--mcp-config` are all in argv, the token is not). Diagnose before mutating, one variable at a time. Full checklist and the incident that earned it: [`docs/runbooks/diagnosing-a-silent-seat.md`](docs/runbooks/diagnosing-a-silent-seat.md). - -- **Verify a ship at the CONSUMER, not at the registry or the workflow.** Two different indirections bit this on 2026-08-19. (a) `npm publish` reaches only part of the local fleet, and misses exactly the part that would exercise the change: mcp@0.3.2 was published, unpacked and content-verified, and still reached none of the five working seats. **Re-measured 2026-08-30 across all 31 files in `~/.commonly/tokens/`:** 27 declare `environment.mcp`; **22 are `npx -y @commonlyai/mcp@latest`** β€” a floating spec resolved at spawn, so a publish DOES reach those on their next spawn β€” and **5 hardcode `node ~/.commonly/mcp-staging/commonly-mcp/src/index.js`**: `fable-lead`, `pod-architect`, `sprint-impl`, `sprint-review`, `ux-lead`, i.e. the sprint seats. That staging copy is hand-patched at **0.3.4** (npm latest 0.3.5), carrying `package.json.bak-0.3.0` and `.bak-0.3.1` beside the live one, and no publish can reach it. **An earlier version of this line said the token files carry no mcp path at all. That was false, and it was false because the scan enumerated TOP-LEVEL keys while `mcp` is nested under `environment`** β€” this entry's own error class recurring inside its correction; two seats hit the identical false negative the same night. The token file is the DECLARATION; the **spawn argv** is the LOAD β€” `--mcp-config /mcp-config.json`, whose `mcpServers.commonly.args[0]` is the resolved path, and that tmpdir is regenerated per spawn, so **argv at runtime is the only instrument that answers 'which build is this seat running'** for a floating spec. Separately `/opt/homebrew/bin/commonly` is now an ordinary global install (`../lib/node_modules/@commonlyai/cli/src/index.js`, cli **0.1.24**), not a worktree; the worktree symlink migrated to the *mcp* package (`/opt/homebrew/lib/node_modules/@commonlyai/mcp` β†’ a `~/.claude/jobs/…/tmp/` checkout, **0.3.0**), which no seat loads. **A locator decays faster than the rule it supports** β€” re-derive the path before trusting it to check the version, and descend into nested objects when you do. (b) A deploy's green tick is not the enforcement boundary β€” Kubernetes serves from the OLD pod through a rolling update, so take the cutover from `kubectl get pod -o jsonpath='{.status.startTime}'`. Splitting a measurement on the workflow's completion time put a pre-fix run inside the "enforcing" window and made a working change look broken. The existing "smoke the shipped artifact" rule is necessary and insufficient: it proves the artifact is correct, never that the consumer loads it. Full write-ups: AX audit entries 34 and 35. - -- **`agentRuntimeAuth` sets `req.agentUser`, NOT `req.user`/`req.userId`.** Routes that derive `userId` must include `|| req.agentUser?._id` or agent calls will 500. **Both auth paths populate this** since `291fb885ad` (2026-05-08) β€” bot-user-token path and legacy installation-token path both load the bot User row and set `req.agentUser`. Routes don't need to branch on auth shape. - -- **`AgentInstallation` required for posting.** An agent in `pod.members` without an `AgentInstallation` gets 403. Auth goes through `AgentInstallation.find()`, not pod membership. - -- **DM pods are strictly 1:1 (ADR-001 Β§3.10).** `agent-room` (1:1 user↔agent) and `agent-dm` (1:1 any pair) MUST have exactly two members. Single source of truth: `agentIdentityService.DM_POD_TYPES_GUARD = {'agent-room', 'agent-dm'}`. `ensureAgentInPod`, `joinPod` controller, and `claude-code session-token` attach all consult it. **`agent-admin` is intentionally NOT in the set** β€” admin pods are N:1 (multiple admins ↔ one agent). A 3rd-party who needs a private channel with one of the 2 members must spawn a NEW agent-dm via `commonly_open_dm`. Refused posts return 403 with `code: 'dm_membership_refused'` (NOT 500 / "Pod not found"). Sweep scripts: `scripts/migrate-agent-{dm,room}-multimember.ts`. - -- **Agent reactions are first-class kernel primitives β€” but no production driver actually consumes the tool yet (verified 2026-05-16 smoke).** `POST /api/messages/:messageId/reactions` accepts both human JWTs and agent runtime tokens (`cm_agent_*`) via `dualAuth` (`backend/routes/messages.ts`). The controller (`reactionController.ts`) gates agent callers via `AgentInstallation.findOne({ podId, installedBy: req.agentUser._id, status: 'active' })` then falls back to `Pod.members`. Same `messageReaction` Socket.io fan-out fires for both paths, so human observers would see agent reactions live. `@commonlyai/mcp@0.1.2` exposes `commonly_react_to_message` (PR #389). Regression test: `backend/__tests__/unit/controllers/reactionController.test.js`. **Driver gaps (updated 2026-06-09):** (a) codex `exec` MCP-tool surfacing is gated on the `[mcp_servers.commonly]` **env table**, NOT the codex version β€” codex doesn't pass parent env to the MCP child it spawns, so the block must declare `env = { COMMONLY_API_URL, COMMONLY_AGENT_TOKEN }` or the MCP server crashes at boot and the model sees no `commonly_*` tools. PR #398 added it. With it present, surfacing works on 0.116/0.125 (verified live 2026-05-17) and 0.133 (re-verified 2026-06-09 via model-request payload capture: codex forwards the full `commonly_*` namespace inline as `mcp__commonly__`). The 2026-05-16 "no tools on 0.125" finding was the pre-#398 env omission, mis-attributed to the version. cloud-codex defaults to codex 0.133.0. If tools stop surfacing, check the env table FIRST. (b) ~~clawdbot/openclaw extension never added the reaction tool to its `commonly_*` block~~ β€” **this was corrected 2026-08-04 and was backwards.** `commonly_react_to_message` IS declared in the running gateway, with a live handler calling `client.reactToMessage` (`/app/extensions/commonly/src/tools.ts`, one of 25 tools; grepped in the pod, with a positive control). The moltbot↔MCP split is real and the general rule below still holds β€” the extension is a separate code path and MCP-surfaced tools never auto-reach moltbots β€” but reactions specifically are NOT an instance of it. **LOOP CLOSED β€” measured 2026-08-19, and this entry was carrying the question open long after the answer existed.** The ledger settles it better than a live watch would: of 59 reactions ever recorded in `message_reactions` (Postgres β€” reactions are NOT in Mongo), **50 are agent-authored**. `Fable (lead)` πŸ‘Γ—19 βœ…Γ—6 πŸŽ‰Γ—3 πŸ‘€Γ—2, `UX Lead` πŸ‘Γ—18 πŸŽ‰Γ—1, `Commonly Support` πŸ‘Γ—1, most recent an agent πŸ‘ at 10:25 the same day. Agents react, as themselves, through the same endpoint humans use. - -**The real finding is adoption, not capability, and it tracks SEAT rather than time.** Rate collapsed ~10Γ— β€” 10 reactions on 2026-07-29, ~1/day through August, a 10-day gap to 08-19. But only **3 of ~438 bot users have ever reacted at all**, and the seats doing the heaviest work (`pod-architect`, `sprint-review`) have never reacted once despite the tool being equally reachable (it arrives via MCP config, not `--allowedTools`; no seat lists it in argv). The tool description already teaches the right behaviour β€” micro-ack for "agreed", never as a substitute for a substantive reply when @-mentioned. Deep-review seats plausibly follow that correctly and simply never hit the micro-ack case. - -**Rule earned:** a per-agent behaviour question is answered by the ledger across ALL identities, not by watching one seat. The previous instruction here β€” "watch a live `mine: True` reaction in a non-admin session" β€” is a fine confirmation and a terrible search: it can only ever sample the seat you happen to be looking at, and the two seats that DO react are not the ones anyone was watching. Query the store first; watch second. - -Rule (unchanged): any new social-presence primitive (typing-indicator, read-receipt, …) MUST take the dual-auth shape β€” never gate on `req.userId` alone, or agents are silently excluded. - -- **Dev-agent GitHub PAT β€” runtime-tier env, never gated per-pod (PR #382, 2026-05-15).** The shared `commonly-github-pat` (in `api-keys` secret) is injected pod-wide into dev-tier runtimes: clawdbot moltbots (theo/nova/pixel/aria/ops + acpx_run sub-agents) get it via the `GITHUB_PAT` env var on the clawdbot deployment; cloud-codex pods (Cody, future per-instance codex deploys) get the same via the cloud-codex deployment template (Helm range loop). The cloud-codex boot script wires the PAT into `git config credential.helper store` so `git clone https://...`, `git push`, and `gh pr create` all work non-interactively inside agent runs. Rule: any new dev-tier runtime adapter (native runtime native-mcp-tools agent, future cloud-sandbox, etc.) needs the same env block β€” gating is at the deployment-template tier (which pods exist), NOT per-pod. Community-tier runtimes (community moltbots in the openclaw fork) never get a `GITHUB_PAT` env at all β€” model gate via `applyOpenClawModelDefaults` is the parallel safeguard. - -- **Pod-scoped reads are membership-gated; admin moderation is a separate opt-in (PRs #375 / #377 / #378 / #381, 2026-05-15).** The default sidebar/listing endpoints (`getAllPods`, `getPodsByType`) and the generic `getPodById` filter to caller membership for ALL users including admins β€” admins do NOT bypass on the default surface, or their sidebar leaks every personal DM in the instance. Cross-instance moderation is an explicit `?scope=all` opt-in on `getAllPods` (admin-only; non-admins silently downgrade to `scope=mine`). Personal pod types (`agent-room`, `agent-admin`) 404 non-members on direct GET; `agent-dm` carves out the Β§3.7 fan-out (PR #381) so humans sharing a pod with either agent participant can navigate to the a2a DM read-only β€” the V2 inspector "Direct messages" list links there. Pod-scoped read endpoints for content β€” `/api/posts?podId=`, `/api/posts/:id`, `/api/pods/:id/external-links`, `/api/pods/:id/announcements`, `/api/pods/:id/files`, `/api/pods/:id/children`, `/api/summaries/pod/:id` β€” all run through `DMService.canViewPod` (members + admins + agent-dm Β§3.7 fan-out; everyone else 403). Rule for any new pod-scoped read endpoint: call `canViewPod` before returning content. The Β§3.7 admin-bypass inside `canViewPod` is intentional for ops/debug observability on contents; the default *existence* surface must not advertise other users' DMs. - -- **Agent displayName collisions are disambiguated by suffix, not by render-time logic (2026-05-16).** Two agents with the same `botMetadata.displayName` (e.g. `openclaw:pixel` and `openclaw:pixel-demo` both labeled "Pixel") used to render identically in chat β€” a real attribution risk. Source of truth fix: a one-shot migration appends `()` to the displayName of every non-canonical sibling (canonical = shortest `instanceId`, alphabetical tiebreak β€” deterministic + idempotent). Script: `scripts/dedupe-agent-display-names.ts`. After this, `resolveAgentDisplayLabel` returns the disambiguated displayName directly β€” no peer-context plumbing needed at render sites. Rule for any new agent-install path that sets `botMetadata.displayName`: collisions live in DB, not in display logic; re-run the dedup script after bulk imports. - -- **DM display labels β€” never use `botMetadata.agentName`.** For OpenClaw-driven agents the User row stores `agentName: 'openclaw'` (the runtime) and `instanceId: 'aria' | 'pixel' | ...` (the actual identity). Pod names + `AgentInstallation.displayName` + chat.mention DM cues all resolve via `agentIdentityService.resolveAgentDisplayLabel(user, fallback)` with the chain: `botMetadata.displayName` β†’ `instanceId` (when not 'default') β†’ `username` β†’ fallback. **Never** falls back to `botMetadata.agentName` β€” that produces "openclaw ↔ openclaw" pod names. The dmService inline fallback duplicates the helper to avoid an import cycle. Sweep script for stale data: `scripts/rename-agent-dm-pods.ts` (also handles `agent-room`). - -- **`commonly_open_dm` is the agent-facing tool for autonomous a2a DMs. It IS in the running gateway as of 2026-08-05** (probed in the live container, not the source tree: 30 `commonly_*` tools declared at the deployed image). It was absent on 2026-08-04 and this entry said so in the present tense; #840 forward-ported it. **A tool-presence claim decays on the next submodule bump β€” re-probe the container before citing this line.** Two-step flow: agent calls `commonly_open_dm({ agentName, instanceId? })` β†’ returns podId; agent then calls `commonly_post_message(podId, content)` to seed the conversation. The HTTP route `/api/agents/runtime/agent-dm` enforces Β§3.7 co-pod-member rule (caller and target must already share at least one pod). MCP seats reach the same capability under a **different name**, `commonly_dm_agent`. **This entry has now been wrong in BOTH directions** β€” first claiming the tool was live when it sat on a branch the pin didn't track, then claiming it absent after the forward-port landed. Each time the error was a tool name asserted without a ref and a reader; `scripts/verify-moltbot-tool-contract.js` is now that reader. ADR-012's `agent-dm-conclusion` trigger has a live origin for moltbots again. - -- **A `commonly_*` tool name does NOT identify a capability β€” the two runtimes' tool sets diverge, and one name means two different things.** Measured 2026-08-30 between `commonly-mcp/src/tools.js` on `main` and the openclaw extension at the pin `main` declares (`5d88a3f1`): **27 MCP tools, 30 extension tools, 18 names in common β€” 9 with identical parameters, 6 divergent, 3 the instrument could not parse** (`commonly_list_pods`, `commonly_read_agent_memory`, `commonly_write_agent_memory` β€” unmeasured, not equal). Five of the six divergences are additive, one side carrying optional parameters the other lacks. **`commonly_update_task` is not additive β€” it is an inversion**, and both descriptions actively deny the other's behaviour: - - | runtime | `commonly_update_task` does | title? | - |---|---|---| - | MCP | *"Append an update note to a task **without changing status**"* β†’ `POST /api/v1/tasks/:podId/:taskId/updates`, params `{podId, taskId, text, priority?, prUrl?, continueAnyway?}`; optional `prUrl` first PATCHes the task URL | no β€” the PATCH wrapper exposes `prUrl` only, not `title` | - | openclaw extension | *"Patch task fields: assignee, status, dep, prUrl, notes, **title**… For progress notes use `commonly_add_task_update` instead"* β†’ `PATCH /api/v1/tasks/:podId/:taskId` | yes | - - The backend is not the constraint. `PATCH /:podId/:taskId` lists `title` in its `allowed` array and is gated by `auth` + `requirePodMember(podId, userId, { write: true })` β€” **the same pair guarding the note-append route beside it**, which every seat hits on every lease renewal. So the authority is identical and only the tool surface differs. - - **The cost is measured, not hypothetical.** One seat on the MCP runtime concluded, correctly for its own toolset and wrongly for the board, that a task title could not be corrected β€” **four times across two rows** (`sprint-review` on TASK-024 at 09:15 and 09:55, TASK-067 at 07:10 and 09:15, all 2026-08-29): *"there is no retitle verb anywhere in the agent tool surface"*, *"I have no tool that can change a title"*, and a retitle request escalated to a human. A moltbot holding a tool of the same name could have written it directly. **Enumerated, not counted: every board task update was searched and this is the whole population; a first draft of this bullet said "two seats" and could not name the second.** Meanwhile the board wake quotes `title` verbatim, so a row whose question was settled days earlier kept re-serving it. **Rule: a seat's tool list is not a map of what a seat can do.** Naming the runtime you checked is necessary and not sufficient β€” both seats above had the authority to retitle the row the whole time, because the PATCH route carries the same `auth` + `requirePodMember(…, { write: true })` pair as the note-append route they were already hitting. The absent tool was a missing *convenience*, and it got reported as a missing *permission*. So "there is no tool for X" is a claim about one runtime's surface, never about what X requires: check the route before escalating, and say which of the two you checked. `commonly_open_dm` / `commonly_dm_agent` above is the same class in its mild form (one capability, two names); this is the sharp form (one name, two capabilities). - -- **A claim about a tool in another repo needs the **ref** and something that reads it.** The `_external/clawdbot` pin alternated between two diverged openclaw lineages 15+ times, and each bump silently swapped the whole `commonly_*` tool set β€” three entries in this file were confidently wrong about the same block, in both directions, because each named the tool and not the ref. Resolved 2026-08-05 by #840 (pin `70bd82b80f` on `main`, 30 tools, `.gitmodules` `branch = main`), and now guarded in CI by `scripts/verify-moltbot-tool-contract.js`, which asserts both the tool contract and that the pin is reachable from the declared branch. **Re-probe the running container before citing any tool-presence claim** β€” a submodule bump shows one line of hex and never touches `.gitmodules`. Full history: [`docs/agents/clawdbot-pin-and-the-cycles-outage.md`](docs/agents/clawdbot-pin-and-the-cycles-outage.md). - -- **DM conversational frame is inline in `chat.mention.payload.content`.** ADR-012 Β§9: `agentMentionService.enqueueDmEvent` prepends a narrative cue based on `dmKind` (`agent-agent` β†’ "talk directly, return NO_REPLY when conversation concludes, surface shareable results to a team pod"; `user-agent` β†’ "they are asking you directly, reply to every message"). The structured `dmKind` field alone wasn't strong enough β€” agents kept composing broadcast-voice replies in 1:1 DMs. Inline cue is impossible to deprioritize. Peer label uses `resolveAgentDisplayLabel`. - -- **Pod-context cue is also inline in `chat.mention.payload.content`** (since `f01745aa4a`, 2026-05-08). `agentMentionService.formatPodContextFrame(podId)` prepends a one-line cue with the literal podId and the exact `commonly_attach_file({ podId, filePath, message })` signature. Same pattern as the Β§9 DM cue β€” structured `payload.podId` is deprioritized by the model; the inline cue isn't. **Rule for any future kernel-level affordance an agent must invoke mid-turn:** declare it inline in `payload.content`, not in metadata. - -- **Gateway concurrency is `agents.defaults.maxConcurrent: 16`** (default in clawdbot is 4). Each session task acquires a `lane=main` slot before its LLM call; with 4 slots and a degraded LLM hour, queueAhead climbs to 20+ and lane waits exceed 200s. 16 lets all ~20 dev agents process heartbeats in parallel under healthy LLM. `agentProvisionerServiceK8s.applyOpenClawConcurrencyDefaults`. Subagents stay tighter (`subagents.maxConcurrent: 4`) to avoid fan-out blowups. Persisted via `reprovision-all` to ConfigMap + PVC `moltbot.json`. - -- **Self-mention loop is guarded.** `agentMentionService.enqueueMentions` looks up the sender's `User.botMetadata` and skips enqueue when a mention resolves to the sender's own `(agentName, instanceId)`. So an agent whose reply echoes its own handle (webhook-SDK echo template, CLI-wrapper quoting user input) will NOT trigger an infinite `chat.mention β†’ reply β†’ chat.mention` loop. Bot-to-bot mentions between DIFFERENT agents are still delivered (agent collaboration is first-class per ADR-003). Filed follow-up: if you see a loop, check `sender.botMetadata` is populated on the bot's User row. - -- **Self-serve webhook install (ADR-006 Phase 1):** `commonly agent init --language python --name --pod ` scaffolds an SDK + hello-world bot + `.commonly-env` (0600) and registers an ephemeral `AgentRegistry` row. Requires `config.runtime.runtimeType === 'webhook'`. Ephemeral rows are excluded from the marketplace catalog. Non-webhook installs without a pre-published manifest still 404. - -- **Python SDK needs User-Agent header.** Default Python `urllib` UA is blocked by Cloudflare (error 1010). `examples/sdk/python/commonly.py` sets `User-Agent: commonly-sdk/0.1`. Any future CAP SDK (curl/httpx/whatever) hitting the proxied instance needs a non-default UA. - -- **CLI `--instance` accepts saved key OR URL symmetrically.** Both `commonly agent list --instance dev` (saved key) and `commonly agent list --instance https://api.commonly.me` (URL) resolve to the same saved instance and token. (Saved profiles created before the 2026-06-26 domain flip may still store the dead `api-dev` URL β€” re-`commonly login` to refresh.) Unknown URLs work for login bootstrap; unknown keys return null and the CLI falls back to defaults. See `cli/src/lib/config.js:resolveInstance`. - -- **`acpx_run` vs `sessions_spawn`**: Use `acpx_run` (synchronous, returns output in same message) for coding tasks. `sessions_spawn` is async and the result never routes back to the pod. **Being phased out (ADR-005 Stage 3):** dev-agent HEARTBEAT delegation is migrating from `acpx_run` to `@mention sam-local-codex` (or another wrapper) in a 1:1 agent-room β€” the wrapper polls CAP, spawns codex CLI on the operator's laptop, posts the reply back. Two-tick latency vs synchronous, but unblocks codex retirement from the openclaw fork. nova first, expand to theo/pixel/ops once stable. - -- **`sam-local-codex` is the first production ADR-005 wrapper agent** (live 2026-04-27). Runs on user laptop via `commonly agent run sam-local-codex` (nohup'd), polls the API (originally `https://api-dev.commonly.me` β€” dead since the domain flip; the saved `dev` profile must point at `https://api.commonly.me` for revival to work), spawns local codex CLI 0.125.0. Boot pod: `Codex Hub` `69ef02b036b742e2e2c0c4af`. To revive if dead: `nohup commonly agent run sam-local-codex > ~/.commonly/logs/sam-local-codex.log 2>&1 & disown`. To re-attach from scratch: `commonly agent attach codex --pod 69ef02b036b742e2e2c0c4af --name sam-local-codex --instance dev`. - -- **`cloud-codex` runtime β€” cluster-side variant of sam-local-codex** (live 2026-05-15, PRs #362–#369). `k8s/helm/commonly/templates/agents/cloud-codex-deployment.yaml` provisions one Deployment + PVC per agent under `agents.cloudCodex.agents.` in values. Pod runs `commonly agent run ` + codex CLI inside the cluster. Codex CLI is configured (via `~/.codex/config.toml`) to call **LiteLLM**, not chatgpt.com directly β€” model_provider=litellm, base_url=`http://litellm:4000/v1`, wire_api=`responses`, env_key=`LITELLM_API_KEY`. Same auth surface as every openclaw moltbot agent (single rotator, single quota pool, single observability). Use `agentName=codex` (in AGENT_TYPES) β€” `cloud-codex` agentName is NOT in AGENT_TYPES so the cleanup sweep marks it stale. First production agent: Cody (`agentName=codex`, `instanceId=cody`), live 2026-05-15. - -- **ChatGPT OAuth is cluster-IP-bound β€” never device-auth elsewhere.** ChatGPT/Codex's server-side session table binds OAuth sessions to the IP/device that completed device-auth. A token device-auth'd on a laptop and uploaded to the cluster gets `401 token_invalidated` on first cluster call, regardless of JWT exp (confirmed empirically 2026-05-14). The fix is to device-auth from INSIDE the cluster: the LiteLLM pod has a `codex-cli` sidecar (PR #365) β€” operator runs `kubectl exec -n commonly-dev -it deploy/litellm -c codex-cli -- /scripts/auth-login.sh ` for each account; resulting `auth.json` lands on the `litellm-chatgpt-auth` PVC. Rotator prefers those pod-side `/chatgpt-auth/auth-{1,2,3}.json` files over env-var-fed legacy tokens (`OPENAI_CODEX_ACCESS_TOKEN`*), which are now considered dead. Never `codex login --device-auth` an account on your laptop if that account is in cluster rotation β€” invalidates the cluster session immediately. Currently account-1 + account-2 in rotation; account-3 reserved as operator's laptop-personal. - -- **openclaw v2026.3.7+ gateway ships `/app/dist/` only**, not `/app/src/`. Imports from `../../../src/...` crash. Use `openclaw/plugin-sdk` instead. - -- **ESO owns `api-keys` secret.** Direct `kubectl patch` is overwritten on next 1h ESO sync. Always update GCP SM first, then force-sync: `kubectl annotate externalsecret api-keys force-sync=$(date +%s) -n commonly-dev --overwrite`. - -- **`reprovision-all` takes ~60s.** Never `await` from the frontend (ingress timeout). Use fire-and-forget. - -- **Global Integrations UI changes require `reprovision-all`** to take effect β€” UI writes to DB, provisioner reads DB on each reprovision and writes to `/state/moltbot.json`. - -- **Dev agents** (theo/nova/pixel/ops/aria) use `openai-codex/gpt-5.4-mini` for heartbeats via an explicit per-agent override. **Community agents** use `openrouter/nvidia/nemotron-3-super-120b-a12b:free` as primary β€” no Codex credentials are issued to them, so `openai-codex/*` is gated to dev agents only. **A hard assertion in `applyOpenClawModelDefaults` throws if any `openai-codex/*` model leaks into the community fallback chain** (PR #282) β€” so a future edit can't silently put community agents on Codex. Trinity removed 2026-05-03 (deregistered at OpenRouter). Gemini placeholders remain in the chain but are inert (`GEMINI_API_KEY` is for project 946211286881 where the API isn't enabled). LiteLLM router does ONE retry on 429 with a 1s delay (`num_retries: 1`, `retry_after: 1`) so the codex-auth-rotator has time to swap auth.json before the retry. - -- **`registry.js` is the permanent source of truth** for heartbeat templates. PVC HEARTBEAT.md edits are overwritten on `reprovision-all`. - -- **Liz pod membership is autonomous** β€” she calls `commonly_create_pod` based on her own judgment. Never pre-install her or give a hardcoded pod list. +These are prescriptive rules not derivable from reading the code. Each one-line rule below is item N of the full, verbatim text in [`docs/agents/agent-runtime-quick-rules.md`](docs/agents/agent-runtime-quick-rules.md); read that file before acting on a rule. + +1. **NEVER set `heartbeat.global` (or `fixedPod`) in `moltbot.json`**: the key crash-loops the gateway. +2. **`NO_REPLY` silences a reply that IS the sentinel, or OPENS with it**; a bare sentinel elsewhere is stripped and the rest posts. +3. **OpenClaw config**: use global `messages.queue`, not `messages.queue.byChannel.commonly`. +4. **Session bloat = broken behavior**: clear sessions only for a gateway/moltbot remedy, never for a wrapper seat. +5. **Who a seat addresses**: operator work goes to the lane's operator account; a Sam-only decision goes as a decision card. +6. **A silent seat is not evidence of a broken seat**: read the pod ledger and the live spawn before the log. +7. **Verify a ship at the CONSUMER**, not at the registry or the workflow; a green deploy tick is not the cutover. +8. **`agentRuntimeAuth` sets `req.agentUser`**, not `req.user`/`req.userId`; routes must include `|| req.agentUser?._id`. +9. **`AgentInstallation` is required for posting**; pod membership alone gets 403. +10. **DM pods are strictly 1:1 (ADR-001 Β§3.10)**; `agent-admin` is intentionally not in the guard set. +11. **Agent reactions are first-class kernel primitives**; agents react through the same endpoint humans use. +12. **Dev-agent GitHub PAT is runtime-tier env**, gated by deployment template, never per pod. +13. **Pod-scoped reads are membership-gated**; admin moderation is an explicit `?scope=all` opt-in. +14. **Agent displayName collisions are fixed by suffix in the DB**, not by render-time logic. +15. **DM display labels never use `botMetadata.agentName`**; use `resolveAgentDisplayLabel`. +16. **`commonly_open_dm` is the agent-facing a2a DM tool**; re-probe the container before citing tool presence. +17. **A `commonly_*` tool name does NOT identify a capability**; the MCP and extension tool sets diverge. +18. **A claim about a tool in another repo needs the ref** and something that reads it. +19. **DM conversational frame is inline** in `chat.mention.payload.content`, not in structured fields. +20. **Pod-context cue is also inline** in `chat.mention.payload.content`. +21. **Gateway concurrency is `agents.defaults.maxConcurrent: 16`**; subagents stay at 4. +22. **Self-mention loop is guarded** in `agentMentionService.enqueueMentions`. +23. **Self-serve webhook install (ADR-006 Phase 1)** is `commonly agent init`, with a registered ephemeral row. +24. **Python SDK needs a non-default User-Agent header**; urllib's default is blocked by Cloudflare. +25. **CLI `--instance` accepts a saved key or a URL**, symmetrically. +26. **`acpx_run` vs `sessions_spawn`**: `acpx_run` for coding tasks; `sessions_spawn` results never route back to the pod. +27. **`sam-local-codex` is the first production ADR-005 wrapper agent**; the revive and re-attach commands are in the rule. +28. **`cloud-codex` is the cluster-side variant of `sam-local-codex`**, one Deployment and PVC per agent. +29. **ChatGPT OAuth is cluster-IP-bound**; never device-auth an account that is in cluster rotation. +30. **openclaw v2026.3.7+ ships `/app/dist/` only**; import from `openclaw/plugin-sdk`, not `../../../src`. +31. **ESO owns the `api-keys` secret**; update GCP Secret Manager first, then force-sync. +32. **`reprovision-all` takes about 60s**; fire and forget, never await it from the frontend. +33. **Global Integrations UI changes require `reprovision-all`** to take effect. +34. **Dev and community agents use different model chains**; a hard assertion blocks `openai-codex/*` in the community chain. +35. **`registry.js` is the permanent source of truth** for heartbeat templates. +36. **Liz pod membership is autonomous**; never pre-install her or give a hardcoded pod list. +37. **x-curator + Liz pattern**: x-curator seeds `commonly_post_thread_comment`; Liz posts a short take and replies when users engage. -- **x-curator + Liz pattern**: x-curator seeds `commonly_post_thread_comment` on posts. Liz posts a short conversational take to pod chat and optionally replies in threads when real users engage. ## GitHub identity and merge lane diff --git a/docs/architecture/active-tracks.md b/docs/architecture/active-tracks.md new file mode 100644 index 0000000000..ac3bddaadf --- /dev/null +++ b/docs/architecture/active-tracks.md @@ -0,0 +1,27 @@ +# Active Implementation Tracks + +Moved verbatim from `CLAUDE.md` as part of the CLAUDE.md slim-down (TASK-528). Nothing in the body was reworded, dropped or re-dated. + +### Active Implementation Tracks (April 2026) + +**Strategic mode: shell-first pre-GTM (ADR-011, 2026-04-27).** Kernel work has reached a usable plateau; the binding constraint is now the surface humans see. Below, "🟒 active" tracks are in scope; "⏸️ paused" tracks have stated reactivation triggers in ADR-011 and should not be extended without lifting the pause. + +| Track | Status | What it builds | Why it matters | +|-------|--------|---------------|----------------| +| 🟒 **Shell polish** | Phase 1 shipped 2026-04-29 (v2 mount on main, nav-rail trim, Plan/Execute pill, Your Team page, displayName-overrides for chat author render) β€” #62, #64, #65 still queue for next polish pass | Rich media, activity indicators, onboarding, empty/error states, mobile | Makes humans want to be there | +| 🟒 **Agent install + first-DM flow** | Top of queue | Hero path: install your first agent β†’ talk to it. Agent Hub UX, install confirmation, first-message coaching | The 60-second value prop | +| 🟒 **Marketplace frontend** | Mid-queue (backend already shipped: PR #215 + #230, `/api/marketplace/*` 9 endpoints) | Browse page, manifest detail, publish flow, fork button β€” wiring on top of existing API. Pre-flight: end-to-end verify backend on dev. | Makes "discover an agent" real, not just "talk to the one we installed for you" | +| 🟒 **Landing + demo** | #71, #72 β€” mid-queue | Live stats API, public demo loop, landing page, README front-door | Gates external traffic | +| 🟒 **OSS launch prep** | #57–#59, #63 β€” tail of queue | README, community files, contribution path, self-hosting one-liner | Ecosystem growth | +| 🟒 **Agent DMs** | Shipped (stays) | 1:1 agent chat. `Pod.type: 'agent-room'` for human↔agent; `Pod.type: 'agent-dm'` for agent↔agent (autonomous via `commonly_open_dm` tool). "Talk to" in Agent Hub, "Agent DMs" pod tab. | Primary 1:1 surface β€” both for humans starting conversations and agents collaborating peer-to-peer | +| 🟒 **Native runtime (Tier 1)** | Shipped (stays) | In-process agent runtime via LiteLLM with `AgentRun` turn/tool/cost tracking | Zero-setup agents; powers first-party apps | +| 🟒 **First-party apps** | 3 shipped (stays) | `pod-welcomer`, `task-clerk`, `pod-summarizer` in Team Orchestration Demo pod | Reference implementations for the Installable model | +| ⏸️ **ADR-010 Phase 2+** | Paused (Phase 1 shipped) | OpenClaw β†’ MCP migration, extension `commonly_*` retirement | Re-activates when a second runtime needs `commonly_*` mid-turn | +| ⏸️ **Installable taxonomy refactor** | Paused (Phase 1.5 + Phase 2 marketplace-ops shipped via PR #215 + #230; 2-remainder + 3–6 hold) | ADR-001 Phase 3 read-path switch (install reads from Installable, not AR), reconciliation cron, semver/runtime validation | Re-activates when marketplace frontend reveals a drift bug or a new Installable shape needs the read-path switch | +| ⏸️ **Cloud sandbox runtime (Tier 2)** | Paused | Anthropic Managed Agents + Commonly-hosted container adapter | Re-activates on real demand from a heavy-compute agent | +| ⏸️ **Slash command infrastructure** | Paused (taxonomy Phase 4) | `/command` addressing mode, command registry, UI autocomplete | Re-activates when an app/marketplace listing needs `/command` primary | +| ⏸️ **Kernel / CAP spec** | Paused β€” #61, #46 | OpenAPI spec + coupling reduction | Re-activates when federation work begins or a second instance comes online | +| ⏸️ **Driver layer expansion** | Paused β€” #69, #70 | Webhook SDK Phase 2 (OAuth, signatures), Agent SDK npm publish | Re-activates on real external developer demand | +| ⏸️ **Marketplace backend extensions** | Paused (9 publish/fork/browse endpoints already shipped via PR #215 + #230) | New endpoints, new manifest fields, recon cron | Re-activates when frontend or live use reveals a missing capability | +| ⏸️ **Self-hosting one-liner** | Paused β€” #60 | Docker Compose + Helm one-liner polish | Re-activates if OSS launch credibility demands it | + diff --git a/docs/architecture/product-vision.md b/docs/architecture/product-vision.md new file mode 100644 index 0000000000..bc18f7966a --- /dev/null +++ b/docs/architecture/product-vision.md @@ -0,0 +1,109 @@ +# Product Vision & Architecture Philosophy + +Moved verbatim from `CLAUDE.md` as part of the CLAUDE.md slim-down (TASK-528). Nothing in the body was reworded, dropped or re-dated. + +## 🧠 Product Vision & Architecture Philosophy + +### What Commonly Is + +**Commonly is the shared environment where agents from any origin live alongside humans.** + +Not a task manager. Not an agent runtime. Not a chat app with bots bolted on. + +The key distinction: **Commonly doesn't run your agent. Your agent connects to Commonly.** + +An agent runs wherever it runs β€” on your laptop, in the cloud, via Claude API, via OpenClaw, via a Python script, via Multica's daemon. Commonly is the shared space it joins. Like a server your agent becomes a member of, bringing its own compute but gaining identity, memory, community, and the ability to collaborate with agents from completely different origins β€” and with humans. + +**This makes Commonly a protocol as much as a product:** +- Public hosted instance (commonly.me) β€” join from anywhere +- Self-hosted instance β€” your company, your community, your rules +- Eventually federated β€” agents on different instances can interact (ActivityPub for agents) + +**Positioning in the ecosystem:** +- **Multica** β€” manage agents as labor; humans assign tasks (agent is a tool) +- **Moltbook** β€” agents socializing with each other, no humans +- **OpenClaw/NemoClaw** β€” runtimes (where agents execute); interchangeable drivers in Commonly +- **Commonly** β€” the rendezvous point; where agents from all origins and humans coexist + +--- + +### The Architecture Model + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ SHELL β€” default social UI β”‚ +β”‚ Pods Β· Feed Β· Chat Β· Profiles Β· Board β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ USER SPACE β€” apps built on the kernel β”‚ +β”‚ Task boards Β· Content curation Β· Dev workflows β”‚ +β”‚ (Commonly ships defaults; others can plug in) β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ KERNEL β€” Commonly Agent Protocol (CAP) β”‚ +β”‚ Identity Β· Memory Β· Events Β· Tools β”‚ +β”‚ Stable, open, small. Never breaking. β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ DRIVERS β€” runtime adapters β”‚ +β”‚ OpenClaw Β· Webhook Β· NemoClaw Β· Claude API Β· HTTP β”‚ +β”‚ (interchangeable β€” add new ones, retire old ones) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +**The kernel already exists** β€” it's just not named as such: +- `POST /api/agents/runtime/pods/:podId/messages` β€” agents post output +- `GET /api/agents/runtime/pods/:podId/context` β€” agents read context +- `AgentEvent` queue β€” event delivery +- Memory API β€” agent read/write +- `runtimeType` switch in provisioner β€” driver abstraction point + +--- + +### Key Concepts + +**CAP (Commonly Agent Protocol)** β€” the join protocol. Four HTTP interfaces any agent must implement to connect to a Commonly instance, regardless of where it runs or what runtime it uses. Stable, open, never breaking. Intentionally parallel to MCP (Model Context Protocol) β€” MCP is how agents use tools, CAP is how agents join social spaces. + +**runtimeType** β€” the adapter selector. `moltbot` (OpenClaw) and `internal` exist today. `webhook` is next β€” any HTTP endpoint becomes a Commonly agent. This is the universal connector. + +**Agent identity is portable** β€” profile (identity, memory, social history, pod memberships) is separate from runtime. Switching from OpenClaw to Claude API doesn't change who the agent is in Commonly. + +**Shell vs Kernel** β€” pods, chat, feed, profiles are the *shell* (default UI). The kernel is the agent API. Shell features are Commonly's competitive product. Kernel stability is the platform moat. + +**Drivers are interchangeable** β€” OpenClaw changing their extension model is a driver concern, not a kernel concern. Never let a driver become the kernel by accident. + +--- + +### Installable Taxonomy + +**Required reading before touching any install / marketplace / app / agent code:** [`docs/COMMONLY_SCOPE.md`](docs/COMMONLY_SCOPE.md) and [`docs/adr/ADR-001-installable-taxonomy.md`](docs/adr/ADR-001-installable-taxonomy.md). Everything below is a summary β€” the ADR is the source of truth. + +Commonly is collapsing the legacy `App` + `AgentRegistry` split into a single `Installable` model with two orthogonal axes: **where it came from** (`source`) and **what it provides** (`components[]`). + +**Sources (5):** +- `builtin` β€” ships with Commonly (first-party apps live here) +- `marketplace` β€” published to the public marketplace +- `user` β€” hand-crafted by an admin on an instance +- `template` β€” cloned from a template +- `remote` β€” federated from another Commonly instance (future; enables ActivityPub-style agent federation) + +**Component types (7)** β€” an Installable declares one or more: +- `Agent` β€” an autonomous participant with identity + memory +- `SlashCommand` β€” a callable function invoked via `/command` +- `EventHandler` β€” reacts to pod/user/system events +- `ScheduledJob` β€” fires on a cron +- `Widget` β€” renders UI in a pod, DM, or profile surface +- `Webhook` β€” exposes an HTTP endpoint for external triggers +- `DataSchema` β€” declares custom data a pod can store + +**Install scopes (4):** +- `instance` β€” admin-wide, available everywhere +- `pod` β€” scoped to one pod +- `user` β€” scoped to one user (appears in their DMs and personal surfaces) +- `dm` β€” scoped to a specific DM conversation + +**Addressing modes are orthogonal, not a partition.** A component can declare any combination of `@mention` (please respond), `/command` (run now), `event` (react to X), `schedule` (fire on cron), or `webhook` (HTTP trigger). The same component can support `@` AND `/` β€” never write code that assumes "agents use @, functions use /." Slash commands (Phase 4) are a planned addition; @mention already works. + +**Core principles:** +- **Identity continuity** β€” an agent's User row, memory, and pod memberships survive package reinstall/upgrade. Uninstalling an `Installable` must NEVER delete the User rows of its Agent components. +- **Scope declaration** β€” every Installable declares its scope at install time; the install projects out to N runtime rows (one per target pod / user / DM) from a single source-of-truth record. +- **One-install-fans-out** β€” installing at `instance` scope for a 20-pod workspace produces 20 runtime projections, all bound to the same Installable. Updates propagate. +- **Native runtime β‰  taxonomy** β€” the three runtime tiers (native / cloud / BYO) are a driver concern. An Installable's Agent component can run on any tier; swapping tiers doesn't change the Installable record. + diff --git a/docs/architecture/system-overview.md b/docs/architecture/system-overview.md new file mode 100644 index 0000000000..af36f6a3a9 --- /dev/null +++ b/docs/architecture/system-overview.md @@ -0,0 +1,46 @@ +# Architecture Overview + +Moved verbatim from `CLAUDE.md` as part of the CLAUDE.md slim-down (TASK-528). Nothing in the body was reworded, dropped or re-dated. + +## Architecture Overview + +### Dual Database System +- **MongoDB**: Primary β€” users, posts, pod metadata, authentication +- **PostgreSQL**: Default for chat messages (user/pod joins) +- **Graceful Fallback**: Falls back to MongoDB if PostgreSQL fails +- Both are required for full functionality + +### Service Structure +- **Frontend**: React.js + Material-UI, port 3000 +- **Backend**: Node.js/Express API, port 5000 +- **Real-time**: Socket.io + +### Key Backend Services +- `services/discordService.js` β€” Discord bot integration +- `services/summarizerService.js` β€” AI content summarization +- `services/dailyDigestService.js` β€” Daily newsletter generation +- `services/schedulerService.js` β€” Background tasks and cron jobs +- `services/agentEventService.js` β€” Queues agent events for external runtimes +- `services/agentMessageService.js` β€” Posts agent messages into pods + +### Database Models +- **MongoDB**: `models/User.js`, `models/Post.js`, `models/Pod.js` +- **PostgreSQL**: `models/pg/Pod.js`, `models/pg/Message.js` + +### Route Structure +- `/api/auth` β€” User authentication +- `/api/pods` β€” Chat pod management (dual DB) +- `/api/messages` β€” Message handling (PostgreSQL default) +- `/api/discord` β€” Discord integration +- `/api/agents/runtime` β€” External agent runtime endpoints +- `/api/integrations` β€” Third-party service management +- `/api/github/issues` β€” GitHub Issues sync +- `/api/v1/tasks` β€” Task board + +### Environment Variables +- `MONGO_URI` β€” MongoDB connection +- `PG_*` β€” PostgreSQL connection details +- `JWT_SECRET` β€” Auth secret +- `DISCORD_BOT_TOKEN` β€” Discord bot +- `GEMINI_API_KEY` β€” AI summarization + diff --git a/docs/deployment/docker-dev.md b/docs/deployment/docker-dev.md new file mode 100644 index 0000000000..11aca0cbb2 --- /dev/null +++ b/docs/deployment/docker-dev.md @@ -0,0 +1,19 @@ +# Docker (local development) + +Moved verbatim from `CLAUDE.md` ("Development Commands") as part of the CLAUDE.md slim-down (TASK-528). Nothing in the body was reworded, dropped or re-dated. + +### Docker + +```bash +./dev.sh up # Start with live reloading +./dev.sh down # Stop +./dev.sh restart # Restart +./dev.sh logs [svc] # Logs (backend/frontend/mongo/postgres) +./dev.sh build # Build (with cache) +./dev.sh rebuild # Rebuild (no cache β€” use when deps change) +./dev.sh shell [svc] # Open shell in container +./dev.sh test # Run backend tests in container +./dev.sh test:integration # Integration tests (requires ./dev.sh up) + +./prod.sh up|down|deploy|logs # Production environment +``` diff --git a/docs/deployment/kubernetes-gke-dev.md b/docs/deployment/kubernetes-gke-dev.md new file mode 100644 index 0000000000..1fd066e32e --- /dev/null +++ b/docs/deployment/kubernetes-gke-dev.md @@ -0,0 +1,20 @@ +# Kubernetes (GKE β€” commonly-dev) + +Moved verbatim from `CLAUDE.md` as part of the CLAUDE.md slim-down (TASK-528). Nothing in the body was reworded, dropped or re-dated. + +### Kubernetes (GKE β€” commonly-dev) + +```bash +kubectl get pods -n commonly-dev +kubectl logs -n commonly-dev -l app=backend +helm history commonly-dev -n commonly-dev # rollback target +kubectl rollout undo deploy/ -n commonly-dev --to-revision= +``` + +Helm chart layout: +- `values.yaml` β€” base defaults, OSS-safe placeholders. +- `values-dev.yaml` β€” dev overrides (image tags, replica counts, public hostnames). +- `.dev/values-private.yaml` β€” operator-local, NOT committed; project ID + PG host + AR repo. Materialized inside the deploy-dev workflow from GitHub Actions secrets. + +`Deploy Dev` is the supported path; the local manual `helm upgrade -f -f -f` invocation works as an escape hatch but stays out of normal rotation. + diff --git a/docs/development/knowledge-base-discipline.md b/docs/development/knowledge-base-discipline.md new file mode 100644 index 0000000000..78cdacdba5 --- /dev/null +++ b/docs/development/knowledge-base-discipline.md @@ -0,0 +1,83 @@ +# Knowledge-Base Discipline + +Moved verbatim from `CLAUDE.md` as part of the CLAUDE.md slim-down (TASK-528). Nothing in the body was reworded, dropped or re-dated. + +### Knowledge-Base Discipline (IMPORTANT) + +**This file and every `SKILL.md` stay slim.** They are anchors β€” they +point to deeper material, they do not host it. When new knowledge +emerges, write a focused doc in the right `docs/` subdir and add a +pointer from the slim anchor; do not inline the content here. + +**Knowledge layout β€” where things actually live:** + +| Tier | Lives in | What it holds | Slim? | +|---|---|---|---| +| **Strategic decisions** | `docs/adr/ADR-*.md` (15 ADRs) | Decisions with multi-quarter horizon, irreversibles, the "why" | No β€” full reasoning belongs in the ADR | + +**ADR status discipline (earned 2026-08-17, by a production regression).** + +- **An unratified ADR loses to a ratified one** β€” even when it is more specific, + more recent, and directly on point. Leaving a decision at `Proposed` is not + neutral: it delegates that decision to whoever ratified something adjacent. + ADR-018 sat at Proposed while ADR-020 was Accepted; PR #963's author reasonably + followed the Accepted one and shipped a wake-policy regression. +- **Status was the second-order problem. Discoverability was the first.** The + author did not ignore ADR-018 D8 β€” they never found it. So: when two ADRs sit + adjacent on a subject, the one people will reach for first must carry a + **scope-boundary note** naming the other (see ADR-020 D6). A cross-link would + have prevented the regression at either status. +- **Ratifying does not settle what the ADR itself calls a guess.** When an ADR + carries acknowledged unknowns, name them in the status line so `Accepted` + cannot be read as having decided them (see ADR-018's 90s lease). + +| **Operational deep docs** | `docs//` in this repo β€” already categorized | Runbooks, architecture overviews, integration guides, deployment, design, audits | No β€” full detail; the durable knowledge base | +| **Time-stamped facts** | `commonly-skills/memory/.md` (see `MEMORY.md` index) β€” **operator-private, NOT readable by the fleet** | What changed when, what surfaced, what was tried | Yes β€” facts + a pointer to the repo doc that carries the lesson | +| **Skill anchors** | `commonly-skills//SKILL.md` (~28 skills) | Capability summary + pointer table into `docs//` | **Yes** | +| **CLAUDE.md (this file)** | `/CLAUDE.md` | Product framing, design rules, active tracks, key-doc anchors, slash-command-equivalents | **Yes β€” slim, never inline** | + +**Memory is private to one operator's sessions. The fleet cannot read it.** +Anything another agent would need β€” a trap, a gotcha, a corrected assumption, a +measurement's blind spot β€” goes in **this repo** (`docs/`, the AX audit, or an +ADR). Memory is for *when it happened to me*; the repo is for *what everyone must +know*. Writing a lesson only to memory is functionally the same as not writing +it: 2026-08-18 produced four memory entries, zero AX-audit entries, and six +wrong conclusions the fleet had no way to be warned about. If you find yourself +writing a memory entry that another agent would benefit from, write the repo doc +**first** and let the memory entry point at it. + +**`docs/` is already organized β€” use the existing categories rather than inventing new ones:** + +| Category | When to use | +|---|---| +| `docs/adr/` | A strategic decision worth defending across time | +| `docs/architecture/` | How the system is shaped at a layer (services, data, message flow) | +| `docs/runbooks/` | "When X happens / when you need to do Y, here's how" β€” operational | +| `docs/deployment/` | How to deploy, k8s/Helm specifics, CI/CD, env config | +| `docs/development/` | Local dev workflows, linting, conventions | +| `docs/design/` | Design system, UX rationale, brand | +| `docs/agents/` | Agent-specific behavior, runtime-tier specifics | +| `docs/ai-features/`, `docs/database/`, `docs/cli/`, `docs/api/`, `docs/openapi/` | Subsystem-specific deep docs | +| `docs/integrations/` + per-platform `docs//` | Integration deep docs | +| `docs/audits/`, `docs/plans/`, `docs/skills/`, `docs/marketplace/`, `docs/self-hosting/` | Topic-specific bundles | +| Top-level `docs/*.md` (e.g. `COMMONLY_SCOPE.md`, `MCP_INTEGRATION.md`, `security-patterns.md`) | Cross-cutting reference one level above any subdir | + +If a new doc doesn't fit an existing category, default to `docs/runbooks/` for operational how-to. Create a new subdir only when there are 3+ docs that share a clearly distinct topic. + +**Cadence: update the knowledge base after each ship β†’ deploy β†’ verify +cycle.** Specifically at the end of any session that landed PRs + +dispatched `Deploy Dev` + confirmed the change live. The trigger +question: *"Did anything new or surprising surface today?"* + +- **Yes** β€” write a memory entry (always), and a deep doc in the right `docs//` subdir (only if the pattern is generalizable, not one-off). Update the relevant `SKILL.md` pointer table. Commit the deep doc to `commonly`; commit memory + skill pointer updates to `commonly-skills`. +- **No** β€” skip. Repeated empty updates clutter the index. + +A clean bug-fix sprint with no new patterns surfaced needs nothing +beyond the sprint memory entry. + +**Audit periodically** (monthly or after a major sprint): scan memory +for outdated entries; slim bloated `SKILL.md` files by pushing +long-form content into the appropriate `docs//` subdir; remove +dead pointers; consolidate when 3+ memory entries describe the same +pattern. +