Give a Bot its computer when nobody is watching - #298
jerelvelarde wants to merge 1 commit into
Conversation
The computer tools were registered in the browser with `useFrontendTool`, and every handler was a `fetch` back to `/api/computers/:botId/...` — a round trip into this same process. That made an open tab load-bearing. A Bot whose person had closed the window had no browser, no workspace and no shell, because the only thing that could carry out a tool call had gone away, and an unattended run was out of reach entirely. This repository has already made that move once, for MCP, and `plugins/tools.ts` still says why: it "made a browser a hard requirement for a Bot to do anything, which rules out an embedded widget, a run nobody is watching, and any surface that is not our own app." So twelve of the fourteen move to `computer/tools.ts` and execute here, offered through the same per-run capability seam that already hands a Bot `message_bot` and `ask_person`. Nothing about governance moves with them: every acting tool still goes through ComputerGateway, which resolves the ref against the snapshot this server took, evaluates the policy, writes the audit row, and only then acts. Two deliberately stay in the browser. `computer_request_secret` and `computer_request_help` both end with a person typing into a masked box or taking the wheel, so both need somebody present by definition; moving them would produce a tool a headless run can call and can never have answered. Such a run is not left mute — it still holds `ask_person`. Rendering stays in the browser too, through `useRenderTool`, which draws a call without claiming to execute it. The renderers are unchanged: they already parsed a JSON result, which is what these return. The activity pane had to change with it. It was written from the handlers, "which run exactly once per call", and `render` runs on every paint. So recordActivity now takes the tool call's id and is idempotent on it, and its listeners are notified in a microtask rather than synchronously inside another component's render.
|
Autonomous/headless completion requirement: finish the server-side computer capability so unattended runs can execute the 12 non-human computer tools end-to-end without an open browser tab. Add a deterministic recording/integration test proving a real headless run invokes the capability seam and ComputerGateway (including policy + audit), and ensure the resulting capability is actually consumed by an unattended surface (routine/queue/worker). Keep request_secret/request_help browser-only and preserve ask_person as the human escalation path. Do not mark the work complete based only on unit tests; require build, typecheck, lint, and the headless integration/recording proof to pass. |
A routine's turn goes through exactly the path a person's chat turn does, as
the routine's owner: their grants, their connections, their thread. That is the
right design, and it is why `actor.id` cannot tell a scheduled run at three in
the morning from the same person typing.
The trail already draws that distinction. `AuditInitiator` is signed into the
run assertion and written onto the row, with the docstring "what caused a row,
where `actorUserId` is only whose authority it borrowed". The boundary could not
ask the same question, so this was unwritable:
deny: initiator.kind == "routine" && intent == "run_command"
`PolicyContext` now carries `initiator`, the kind and id the trail already
records. The id is there too, so a deployment can name one routine rather than
deciding about scheduled runs as a class, and `handoff` is its own kind.
Required on the type rather than optional, and flattened to two always-present
strings. cel-js throws on an unbound identifier and a throw fails closed, so a
field that were sometimes absent would turn one rule about routines into a
deployment that refused every ordinary click — the failure #115 exists to
prevent. `id` is "" for the kinds that carry none, the neutral `mcp.effect`
already uses.
Set at all three places a context is built. `plugins/store.ts` is where it is
real: a routine's tools run through there carrying the initiator its assertion
was signed with. The computer gateway passes a person, and that is truthful
rather than convenient — a Bot's computer is driven by frontend tools in the
browser, so every action reaching that gateway came from somebody's session.
#298 is the change that makes that untrue, and the comment says so. The
history replay reads the initiator off the row when it is there.
Nothing is refused that was not refused before.
Ten tests: a routine refused by a rule naming it while the identical action with
a person is allowed, one routine named without catching another, a rule naming
the initiator leaving an ordinary click alone, and a replayed row with no
initiator or an unrecognised one reading as a person. 52 pass in
computer-policy, 13 in policy-dry-run, and computer-gateway's 60 unchanged.
Answers the half of #484 that survived — the rest of that issue was my own
misreading, corrected there.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: David McKay <david@copilotkit.ai>
|
Reviewing this because #526 landed an hour ago and the two now interact in a way that is invisible The move itself reads right. The browser was the only thing that could drive a computer, which made The one thing: an unattended run will be judged and recorded as a person#526 added // gateway.ts, from #526
initiator: policyInitiator(),with a comment saying, in as many words, that this is truthful only because a Bot's computer is This PR is that change, and it passes computerTools({
gateway: computerGateway,
botId,
actor: { id: actorId, ...(isRealUser ? { userId: actorId } : {}) },
...
})So after this lands, a routine firing at three in the morning reaches Not your bug — the ordering is just unlucky. But it is load-bearing for the feature this PR is for: Smallest fix I can see
export type ActionActor = {
id: string;
userId?: string;
/** What caused this run. Absent reads as a person, which is what a chat turn is. */
initiator?: AuditInitiator;
};Then The last step is the only one with any depth to it. If you would rather keep this PR to the move, I am happy to send the threading as a follow-up once Two smaller notes
Happy to be wrong on any of this; the first one is the only one I would hold the PR for. |
3 participants
The problem
A Bot's computer is the thing this product is about: its own browser, its own logins, its own files,
and every action decided before it happens. Today none of it exists unless somebody has a tab open.
The fourteen computer tools were registered in the browser with
useFrontendTool, and each handlerwas a
fetchback to/api/computers/:botId/...— a round trip out of this process and into itagain. So the browser was not a client of the feature, it was a required component of it. Close the
window mid-task and the Bot loses its shell, its workspace and its browser, because the only thing
that could carry out a tool call has gone away. A run nobody is watching cannot use a computer at
all, which is the wall every unattended surface meets: a routine, a scheduled sweep, a delivery to a
Bot that a person is not sitting in front of.
This repository has already made this move once, for a different tool family, and
plugins/tools.tsstill says why in its opening paragraph:
Every word of that applies to the computer tools, which were left behind.
The approach
Twelve of the fourteen move to
server/src/computer/tools.tsand execute here. They are offeredthrough the per-run capability seam that already hands a Bot
message_botandask_person, sothere is no new plumbing: a
GrantedToolis already this codebase's word for "a tool the model maycall, executed here", and
escalation.tsandhandoff-tool.tsalready build them without goingnear a plugin grant.
Nothing about governance moves with them. Every acting tool still goes through
ComputerGateway,which resolves the ref against the snapshot this server took, evaluates the policy, writes the audit
row, and only then acts. The tools module hands the model a description of what it may call; the
gateway remains what decides whether a call happens.
Two deliberately stay in the browser.
computer_request_secretandcomputer_request_helpbothend with a person typing into a masked box or taking the wheel, so both need somebody present by
definition. Moving them would produce a tool a headless run can call and can never have answered — a
worse failure than not offering it, because the run stalls instead of adapting. Such a run is not
left mute: it still holds
ask_person, which is the honest exit for a Bot that needs a human.Rendering stays in the browser, through
useRenderTool, which draws a call without claiming toexecute it. The renderers themselves are unchanged, because they already parsed a JSON result and
that is what these return. The transcript still names the element the gateway resolved rather than
the ref the model sent.
The activity pane is the part that cost something. It was written from the handlers, and its own
docblock explained why: they "run exactly once per call", where a tool's
renderruns on everypaint. With the handlers gone,
renderis the only place the browser still sees a call, sorecordActivitynow takes the tool call's id and is idempotent on it. Its listeners are notified ina microtask rather than synchronously, because notifying from inside one component's render sets
state in another that is subscribed through
useSyncExternalStore. The cost is a required argumentat every call site and a
seenset that has to be cleared when a computer is wiped; both arecovered by tests, and the second is the one that would have rotted quietly.
What is not covered
own by design — the renderers are deliberately unchanged — so the only honest recording is a Bot
running a command with no tab attached, which needs a model credential this environment does not
have. It follows before this leaves draft.
and a workspace, but cannot ask for a password or hand over the wheel.
that walks through it.
report_refusalwrites its audit row through the capability seam rather than the existingPOST /api/agents/:id/declinedroute. Both now exist and write the same event type; the routeis still what the browser-side path would use, and collapsing them is left alone here rather than
changed in a commit about something else.
Verification
Run against a local Postgres with migrations applied.
mainatfb0c797: 2064 pass, 23 skip, 0 fail, 170 files.difference; nothing existing changed its result.
tsc --noEmitclean for bothserverandapp.biome lint --error-on-warnings .clean across 492 files.New and changed test files:
server/tests/computer-tools.test.ts— holds down that the Bot and the actor reach the gateway(the audit row is written from them), that a refusal comes back as an answer carrying the rule
rather than a throw that ends the run, that the failure modes stay distinguishable so a model can
tell a retry from a dead end, that bad arguments are refused before the gateway is touched, that
optional arguments are omitted rather than sent as
undefined, and that the two person-presenttools are not offered.
app/tests/computer-activity.test.ts— extended with the property the keyed signature exists for:one call is one entry however many times a render records it, and a wiped computer can record the
same call id again without it being swallowed as a duplicate.