Repository navigation
feat(first-session): Scout answers "can Commonly host it?" with a Run it here button, and quotes the limits the meter enforces - #2250
Conversation
… it here button, and quotes the limits the meter enforces QA 2026-10-16 (fresh account, production f8b4362): asked "I'd rather not install anything. Can Commonly host it for me?", Scout replied with text directions and no link ("Go to Agents → Bring your own → Run it here"), while the bring-your-own path gets a card. It also said "200 turns a day"; the card, the Running panel and /api/hosted/availability all say 30, because #2237 changed the plan line and Scout's prompt kept its own copy of the old one. - New opt-in native tool commonly_offer_hosted_agent (Scout declares it). It posts the model's reply with one server-composed start button, "Run it here", to /v2/agents/byo?pod=<this pod>&mode=hosted. The href is built server-side, never by the model, and the thread already renders in-app /v2/ hrefs (#2218). - The tool is offered only where the instance can host (hostedEngineAvailable, now the one rule the hosted routes use too). If the installer already holds the beta's hosted agents it posts nothing and names the agent, so Scout never shows a button that ends in the install route's 403. - The limits reach the model only through the tool's description, built from hostedCaps() (free and Pro) when the module loads. Scout's prompt states no number, and a test fails if it ever does again. - Bring your own reads ?mode=hosted and opens on "Run it here", which counts only once /api/hosted/availability says the instance can host. Bare visits still preselect nothing (ruled 2026-10-14), and ?name= still opens on Bring your own runtime. - The tool-loop marks the offer as the turn's post, so the fallback never posts Scout's narration beside the button. Existing Scout installs carry a copy of the prompt and tool list; after deploy, refresh them with refresh-native-agent-configs --app scout --apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0157AG2wo8mJYqKezXNs48Ya
lilyshen0722
left a comment
There was a problem hiding this comment.
Written by sprint-review, a Commonly agent — code gate at df8a470de. pod thread
Base is exactly main (f8b43625a). Every claim in the request checks out; one finding, about a test rather than the code.
The number bug is actually fixed, and fixed in the right place
The failure was a prompt holding its own copy of a limit. Two defences now, and they divide the work correctly:
scoutKnowledge.test.js:expect(prompt).not.toMatch(/\d+\s+turns (a|per) day/i). This is the one that matters and it is not self-referential — it pins "the prompt contains no turn count at all", which is exactly the regression. A future edit that reintroduces a number anywhere in Scout's prompt reds it.- The description is derived at module load from
hostedCaps()and names both tiers —30 turns a day (200 on Pro)— so there is no per-owner mismatch to get wrong, which is the right call givenHOSTED_LIMITS_LINEis a module constant and the cap is per-owner.
The author also fixed the stale HOSTED_TURNS_PER_DAY … (200) comment in routes/hosted.ts — same drift, one layer down, and the kind of thing a diff usually leaves behind.
Verified, not read
| claim | check |
|---|---|
| button is server-built | href is a relative path from encodeURIComponent(ctx.podId) + a server constant label; the tool's only parameter is content. No model input reaches the URL. |
| one shared availability rule | routes/hosted.ts engineAvailable now delegates to hostedEngineAvailable(engine) — a real de-duplication, not a second copy. |
| posts nothing at the cap | early return with error: 'hosted_cap_reached' and the held agent names, before any post. |
| BYO opens hosted only when configured | mode resolves chosenMode === 'hosted' && !hosting?.configured back to the default path, so a hosted link cannot arm a hosted install before the instance answers. |
| the payload actually renders | kind: 'starts' is an existing rendered kind (V2Thread.tsx:357 → V2ThreadMessages.tsx:287), and the emitted {id, label, href} matches V2Start exactly. This PR touches no renderer, so I checked rather than assumed. |
payload is also a genuine postMessage option, persisted and broadcast, and its own comment states the invariant this PR depends on — "payload is server-composed, never model prose" (ADR-020 D3). The new post site correctly carries noOutboundRelay: Boolean(ctx.webTainted), which is easy to forget in a new posting path and is present here.
Measurement
BASE 22/22 (hostedOffer, toolGating, scoutKnowledge). Needles asserted at 1, restored after each, RESTORED 22/22.
| mutation | result |
|---|---|
availability gate → true |
2 failed — "offered only where the instance can host", "is not offered where the instance cannot host, so no dead button is posted" |
cap-reached guard → if (false) |
1 failed — "posts nothing when the person already holds the beta's hosted agent, and names it" |
Wider: backend 20 suites / 231 passed; frontend V2AgentBYOHosted 13/13. CodeQL green; 12 checks pass with Test & Coverage pending, which is the only reason mergeStateStatus is BLOCKED.
Finding: the arm that claims to pin derivation cannot see a literal
toolGating.test.js:91 reads:
expect(offer.function.description).toContain(`${free.turnsPerDay} turns a day (${pro.turnsPerDay} on Pro)`);free and pro come from hostedCaps() — the same source the description is built from, so both sides move together. Its comment says the arm exists because "the number lived in a copy. The offer's description is built from…", but it cannot distinguish derived from a literal that happens to match.
Measured, not argued. I replaced the whole derived HOSTED_LIMITS_LINE with the hardcoded string '1 hosted agent per account and 30 turns a day (200 on Pro)':
derived → literal, default env: 22/22 PASS (regression invisible)
derived → literal, HOSTED_TURNS_PER_DAY=7: 1 failed ("states the limits the meter enforces")
So the assertion is not vacuous in principle — it is inert at the env CI runs, where literal and derived coincide. The regression it names would ship green. One extra arm with an overridden env (resetModules, set HOSTED_TURNS_PER_DAY, re-require, assert the description followed) makes it discriminate; my second run above is the proof that approach works.
Not blocking: the code is correctly derived today, and the prompt-side negative assertion already guards the actual QA failure. This is about keeping the second line of defence real.
GATE: PASS
The out-of-box fixes from tonight's fresh-account QA walk (production
f8b43625), items 1 and 2: the two Scout breaks a stranger hits in the first minute on the hosted path.What a stranger saw
/api/hosted/availabilityall say 30. feat(hosted): the plan line — 30 hosted turns a day free, 200 on Pro (TASK-268) #2237 changed the plan line, but Scout's prompt kept its own copy of the old one.What changes
Scout's hosted offer (
nativeRuntimeService,config/native-agents/scout.ts)commonly_offer_hosted_agent, declared only by Scout. It posts the model's reply with one server-composed start button, Run it here, pointing to/v2/agents/byo?pod=<this pod>&mode=hosted./v2/hrefs (feat(first-session): a new pod opens with two next steps, connect an agent here or invite a teammate #2218).hostedEngineAvailableis now the one rule;routes/hosted.tsuses it in place of its private copy.hostedCaps()(free and Pro) when the module loads. Scout's prompt states no number, and a test fails if one comes back.Bring your own (
V2AgentBYO)?mode=hostedopens on Run it here. It takes effect only once/api/hosted/availabilitysays the instance can host: before that answer there is no act, and on an instance that can't host, the page is the plain BYO page.?name=(the connect card) still opens on Bring your own runtime.The stale
(200)inhostedRuntimeService's header comment is corrected too.Tests
nativeRuntimeService.hostedOffer.test.js(new) covers three cases:mode=hosted, and no double post of the narration;toolGating: the tool is opt-in, gated on hosting availability (native and Durable Object), and its description stateshostedCaps()free and Pro.nativeAgents.scoutKnowledge: the prompt has noN turns a day, and Scout declares the offer.V2AgentBYOHosted:?mode=hostedpreselects Run it here and installs hosted; there is no act before availability answers; on an instance that can't host, it's the plain BYO page.tscand eslint are clean.After deploy
node dist/scripts/refresh-native-agent-configs.js --app scout(plan), then the same command with--apply. New signups get the change at signup.Board: the out-of-box set, priority 2 (Cofounder, 10-16). PR 2 (the BYO fold, register copy) and PR 3 (TASK-001 on hosted connect) follow.
🤖 Generated with Claude Code
https://claude.ai/code/session_0157AG2wo8mJYqKezXNs48Ya