Skip to content

feat(first-session): Scout answers "can Commonly host it?" with a Run it here button, and quotes the limits the meter enforces - #2250

Merged
lilyshen0722 merged 1 commit into
mainfrom
growth/hosted-first-minute
Oct 16, 2026
Merged

lilyshen0722 merged 1 commit into
mainfrom
growth/hosted-first-minute

Conversation

@lilyshen0722

Copy link
Copy Markdown
Contributor

The out-of-box fixes from tonight's fresh-account QA walk (production f8b43625), items 1 and 2: the two Scout breaks a stranger hits in the first minute on the hosted path.

What a stranger saw

  • Asked "I'd rather not install anything. Can Commonly host it for me?", Scout answered with text directions only: "Go to the Agents page, click Bring your own, then Run it here". There was no card and no link, while the bring-your-own path gets a connect card.
  • Scout also said 200 turns a day. The card, the Running panel and /api/hosted/availability all say 30. feat(hosted): the plan line — 30 hosted turns a day free, 200 on Pro (TASK-268) #2237 changed the plan line, but Scout's prompt kept its own copy of the old one.

What changes

Scout's hosted offer (nativeRuntimeService, config/native-agents/scout.ts)

  • New opt-in native tool commonly_offer_hosted_agent, declared only by Scout. It posts the model's reply with one server-composed start button, Run it here, pointing to /v2/agents/byo?pod=<this pod>&mode=hosted.
  • It is offered only where the instance can host. hostedEngineAvailable is now the one rule; routes/hosted.ts uses it in place of its private copy.
  • If the installer already holds the beta's hosted agent, it posts nothing and names that agent. Scout then says so, rather than showing a button that ends in the install route's 403.
  • The limits reach the model only through the tool's description, built from hostedCaps() (free and Pro) when the module loads. Scout's prompt states no number, and a test fails if one comes back.
  • The tool loop treats the offer as the turn's post, so Scout's narration is never posted beside the button.

Bring your own (V2AgentBYO)

  • ?mode=hosted opens on Run it here. It takes effect only once /api/hosted/availability says the instance can host: before that answer there is no act, and on an instance that can't host, the page is the plain BYO page.
  • Bare visits still preselect nothing (ruled 2026-10-14), and ?name= (the connect card) still opens on Bring your own runtime.
  • Hosted needs no invitation code. It is capped per user, not entitlement-gated, and the QA account (no code) went live this way.

The stale (200) in hostedRuntimeService's header comment is corrected too.

Tests

  • nativeRuntimeService.hostedOffer.test.js (new) covers three cases:
    • one post, carrying exactly one button aimed at this pod with mode=hosted, and no double post of the narration;
    • nothing posted when the installer already holds a hosted agent, with that agent named back to the model;
    • the tool isn't offered where the instance can't host.
  • toolGating: the tool is opt-in, gated on hosting availability (native and Durable Object), and its description states hostedCaps() free and Pro.
  • nativeAgents.scoutKnowledge: the prompt has no N turns a day, and Scout declares the offer.
  • V2AgentBYOHosted: ?mode=hosted preselects Run it here and installs hosted; there is no act before availability answers; on an instance that can't host, it's the plain BYO page.
  • Mutation-checked: removing the post-once guard, re-adding "200 turns a day", dropping the preselect, and dropping the availability guard each fail a test.
  • Ran locally on Node 22: the touched suites plus 11 neighbours (seed and refresh scripts, moltbot tool contract, hostedEngine, webSearch, runCode, prompts). 220 + 51 backend and 13 frontend tests, all green. tsc and eslint are clean.

After deploy

  • Existing Scout installs carry a copy of the prompt and tool list. Run node dist/scripts/refresh-native-agent-configs.js --app scout (plan), then the same command with --apply. New signups get the change at signup.
  • Re-walk on a fresh account: ask Scout to host, click the button, and the page should land on Run it here; click once more to go live.

Board: the out-of-box set, priority 2 (Cofounder, 10-16). PR 2 (the BYO fold, register copy) and PR 3 (TASK-001 on hosted connect) follow.

🤖 Generated with Claude Code

https://claude.ai/code/session_0157AG2wo8mJYqKezXNs48Ya

… it here button, and quotes the limits the meter enforces

QA 2026-10-16 (fresh account, production f8b4362): asked "I'd rather not
install anything. Can Commonly host it for me?", Scout replied with text
directions and no link ("Go to Agents → Bring your own → Run it here"), while
the bring-your-own path gets a card. It also said "200 turns a day"; the card,
the Running panel and /api/hosted/availability all say 30, because #2237
changed the plan line and Scout's prompt kept its own copy of the old one.

- New opt-in native tool commonly_offer_hosted_agent (Scout declares it). It
  posts the model's reply with one server-composed start button, "Run it
  here", to /v2/agents/byo?pod=<this pod>&mode=hosted. The href is built
  server-side, never by the model, and the thread already renders in-app /v2/
  hrefs (#2218).
- The tool is offered only where the instance can host (hostedEngineAvailable,
  now the one rule the hosted routes use too). If the installer already holds
  the beta's hosted agents it posts nothing and names the agent, so Scout
  never shows a button that ends in the install route's 403.
- The limits reach the model only through the tool's description, built from
  hostedCaps() (free and Pro) when the module loads. Scout's prompt states no
  number, and a test fails if it ever does again.
- Bring your own reads ?mode=hosted and opens on "Run it here", which counts
  only once /api/hosted/availability says the instance can host. Bare visits
  still preselect nothing (ruled 2026-10-14), and ?name= still opens on
  Bring your own runtime.
- The tool-loop marks the offer as the turn's post, so the fallback never
  posts Scout's narration beside the button.

Existing Scout installs carry a copy of the prompt and tool list; after
deploy, refresh them with refresh-native-agent-configs --app scout --apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0157AG2wo8mJYqKezXNs48Ya

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Written by sprint-review, a Commonly agent — code gate at df8a470de. pod thread

Base is exactly main (f8b43625a). Every claim in the request checks out; one finding, about a test rather than the code.

The number bug is actually fixed, and fixed in the right place

The failure was a prompt holding its own copy of a limit. Two defences now, and they divide the work correctly:

  • scoutKnowledge.test.js: expect(prompt).not.toMatch(/\d+\s+turns (a|per) day/i). This is the one that matters and it is not self-referential — it pins "the prompt contains no turn count at all", which is exactly the regression. A future edit that reintroduces a number anywhere in Scout's prompt reds it.
  • The description is derived at module load from hostedCaps() and names both tiers — 30 turns a day (200 on Pro) — so there is no per-owner mismatch to get wrong, which is the right call given HOSTED_LIMITS_LINE is a module constant and the cap is per-owner.

The author also fixed the stale HOSTED_TURNS_PER_DAY … (200) comment in routes/hosted.ts — same drift, one layer down, and the kind of thing a diff usually leaves behind.

Verified, not read

claim check
button is server-built href is a relative path from encodeURIComponent(ctx.podId) + a server constant label; the tool's only parameter is content. No model input reaches the URL.
one shared availability rule routes/hosted.ts engineAvailable now delegates to hostedEngineAvailable(engine) — a real de-duplication, not a second copy.
posts nothing at the cap early return with error: 'hosted_cap_reached' and the held agent names, before any post.
BYO opens hosted only when configured mode resolves chosenMode === 'hosted' && !hosting?.configured back to the default path, so a hosted link cannot arm a hosted install before the instance answers.
the payload actually renders kind: 'starts' is an existing rendered kind (V2Thread.tsx:357 → V2ThreadMessages.tsx:287), and the emitted {id, label, href} matches V2Start exactly. This PR touches no renderer, so I checked rather than assumed.

payload is also a genuine postMessage option, persisted and broadcast, and its own comment states the invariant this PR depends on — "payload is server-composed, never model prose" (ADR-020 D3). The new post site correctly carries noOutboundRelay: Boolean(ctx.webTainted), which is easy to forget in a new posting path and is present here.

Measurement

BASE 22/22 (hostedOffer, toolGating, scoutKnowledge). Needles asserted at 1, restored after each, RESTORED 22/22.

mutation result
availability gate → true 2 failed — "offered only where the instance can host", "is not offered where the instance cannot host, so no dead button is posted"
cap-reached guard → if (false) 1 failed — "posts nothing when the person already holds the beta's hosted agent, and names it"

Wider: backend 20 suites / 231 passed; frontend V2AgentBYOHosted 13/13. CodeQL green; 12 checks pass with Test & Coverage pending, which is the only reason mergeStateStatus is BLOCKED.

Finding: the arm that claims to pin derivation cannot see a literal

toolGating.test.js:91 reads:

expect(offer.function.description).toContain(`${free.turnsPerDay} turns a day (${pro.turnsPerDay} on Pro)`);

free and pro come from hostedCaps() — the same source the description is built from, so both sides move together. Its comment says the arm exists because "the number lived in a copy. The offer's description is built from…", but it cannot distinguish derived from a literal that happens to match.

Measured, not argued. I replaced the whole derived HOSTED_LIMITS_LINE with the hardcoded string '1 hosted agent per account and 30 turns a day (200 on Pro)':

derived → literal, default env:        22/22 PASS   (regression invisible)
derived → literal, HOSTED_TURNS_PER_DAY=7:  1 failed  ("states the limits the meter enforces")

So the assertion is not vacuous in principle — it is inert at the env CI runs, where literal and derived coincide. The regression it names would ship green. One extra arm with an overridden env (resetModules, set HOSTED_TURNS_PER_DAY, re-require, assert the description followed) makes it discriminate; my second run above is the proof that approach works.

Not blocking: the code is correctly derived today, and the prompt-side negative assertion already guards the actual QA failure. This is about keeping the second line of defence real.

GATE: PASS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant