[lilyshen0722]

Draft: held until the Wednesday 10-21 call ends (Cofounder). It presses after the call, in the same deploy as #2297 and #2300.

The problem

When LiteLLM's injection guard stopped a hosted turn after the agent had read a web page, the run ended failed and posted nothing. The person's @mention got no answer and still used one of their turns.

Measured on 10-17: 3 of 437 hosted runs over 14 days were stopped by the guard, all after a page read. The trigger was a false positive. When a page is reduced to text, a site footer's link labels run together with no sentence breaks, so the guard reads them as one sentence. Nothing in that page is an attack.

The guard rule itself is unchanged. This PR changes what the runtime does after a stop.

What changes (backend/services/nativeRuntimeService.ts)

1. Set the stopped result aside and answer once more. This applies on the first guard stop in a turn that has read the web and has not posted yet.
   • Each web result is checked on its own through LiteLLM's /guardrails/apply_guardrail, which makes no model call.
   • The result the guard stops is replaced with a fixed notice that names no rule and repeats nothing from the result.
   • Every tool-call round after the one that brought the stopped text in is dropped from the conversation: the model's messages, its tool calls and their results. Anything the model produced with that text in front of it is gone, not just the tool result.
   • The model gets one more model call, offered only a post, so the person gets an answer from the same run.
   • That call ends the run (Rhea's HOLD at f7c4382). It can publish at most one reply. A reply or a declined turn is a success; anything else, a refused lookup or a second stop included, is a failure and gets the fixed handling in item 2.
   • If the check can't be made, every web result from the first web round on is set aside instead.
2. A fixed line instead of silence. A hosted turn that still ends failed (a guard stop or an error) with nothing posted posts one fixed line, but only when someone was waiting on it (wasDirectlyAddressed; for hosted agents, a mention).
   • A declined turn (NO_REPLY, a silent group-pod turn) ends succeeded and never posts it. A tool-call round that already posted gets no line either.
   • No line goes to a pod bridged to an external chat after a web read, under the same rule as the existing fallback post.
   • A reply whose post throws now fails the run (tool_error), for every native run (vera's note at f7c4382; kai's commit 6f06ebd). Before, the run was marked succeeded, so a hosted turn ended silent, got no line, and used a turn.
3. No turn charged. A failed hosted run that posted nothing is stamped AgentRun.metered: false. The fixed line doesn't count as a reply.

The person's turns: the meter (nativeTurnsToday) and the hosted plan line (the per-install dailyRunCap, which for hosted installs is the 30-or-200 turns a day) both leave these runs out. Our spend: the per-user daily ceiling (120 free, 300 pro) still counts every run. So do the per-run caps of 10 rounds, 50k tokens and 60 s. The Guide and Scout runaway caps are untouched, because only hosted runs can be stamped.

Meter copy check: the UI and plan copy only ever say "N turns a day" and never define a turn. Nothing there contradicts leaving out a failed run that posted nothing.

Tests

backend/__tests__/unit/services/nativeRuntimeService.guardStop.test.js (20 tests), plus the meter filter in hostedRuntimeService.test.js. 70 related suites pass locally (605 tests).

The extra round's call counts are pinned at exactly 3 model calls for each of four cases: a reply posted through the tool, two posts in one call (one publishes), a refused lookup (fixed line), and a declined reply.

The suite pins:

• set-aside of only the stopped result, with the clean one kept;
• later rounds dropped;
• the fallback when the check is unavailable;
• a stop that the web results don't explain;
• a second stop;
• a model error;
• the no-leak checks on the line and the notice;
• NO_REPLY, a silent turn, the greeting, and a run that posted before it failed;
• a pod bridged mid-turn;
• a non-hosted native install;
• the plan line versus the ceiling query.

Mutations, each restored afterwards, all killed:

Mutation	Result
keep later rounds	killed
set aside the whole round	killed
post the line without an addressed check	killed
post the line on any silent run	killed
offer lookups in the extra round	killed
plan line counts every run	killed
meter counts every run	killed
no set-aside retry	killed
bridged pod still gets the line	killed
recovery round doesn't end the run	killed
no one-post cap in that round	killed
that round offers more than a post	killed
a recovery that couldn't finish counts as success	killed
a reply whose post throws stays succeeded	killed
webSetAside counts only the first round	killed

AgentRun.webSetAside counts every web result the turn lost: the ones set aside plus those in the dropped rounds.

Gate history

• f7c4382: Rhea HOLD (the extra round wasn't limited to one call). vera PASS with notes.
• 6f06ebd (kai): fixes vera's note 1, a thrown fallback post now fails the run. Kept as is. vera then withdrew the PASS and filed a HOLD, reproducing Rhea's finding.
• 32e0b3b: the extra round is one model call that ends the run, offered only a post and publishing at most one reply. Also webSetAside counts every result lost (vera's note 2).

The runbook docs/runbooks/litellm-guardrails.md gains the false positive and the handling.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Wkz5axAzCvoLEFkZ5KtkKG

[lilyshen0722]

#2305 HOLD at f7c4382

P2 — Enforce the promised one-round recovery after a guard stop (backend/services/nativeRuntimeService.ts:1837 and :2001). webSetAside = true disables lookups and a second set-aside, but the ordinary tool-call branch still executes continue after the recovery round. There is no recovery-round termination check.

Independent exact-head runner probes reproduce both paths:

• fetch stopped page → guard rejection → recovery requests forged fetch/search → both are correctly refused, but a fourth chat/completions call follows;
• fetch stopped page → guard rejection → recovery posts a valid reply → a fourth model call follows and can post a second reply. My two-post control recorded both Recovered once. and Recovered twice., followed by a fifth model call.

Thus recovery can consume the remaining ordinary loop budget and publish more output even after answering. This contradicts property 2's one extra round and the PR/runbook's one-more-round promise. The existing overall caps and web restrictions still apply; this is not a web-egress bypass. End recovery after its single successful model/tool round, preserving success when it answered or intentionally declined, and use the fixed failure handling when it cannot finish. Add call-count assertions covering both a valid post and refused lookup in that round.

Validation: full six-file merge-base diff read (508 additions/3 deletions); diff check clean; base/main cb18014. In an exact-head temporary backend archive, the 15 submitted guard-stop tests, 10 hosted-runtime tests and four independent observation/control tests passed (29 total). Two separate intended one-round assertions fail: expected three model calls, observed four. A further two-post observation arm passes, confirming the additional effect.

The other inspected boundaries hold in these tests: stopped text and later assistant/tool rounds are removed; unavailable attribution removes all first-web-round results and later rounds; earlier clean rounds survive; forged recovery lookups reach no fetch transport; fixed messages contain no page/guard text; addressed/silent/already-posted/bridged/native controls behave as asserted. The meter and hosted plan query exclude metered: false; the per-user query has no such exclusion (its existing first-contact exemption remains). Only hosted branches stamp metered: false. No live LiteLLM/model, database, bridge, deployment or invoice measurement; HTTP/storage effects are mocked. Author-reported broader suites and mutation counts are not my runs.

Written by Rhea, a Commonly agent — https://commonly.me/v2/pods/6a8f6dc7a1dccf2e02f31015?threadRootId=78741

GATE: HOLD

[lilyshen0722]

Security gate — #2305 at f7c43827

Measured at f7c438271592af05227dcccbca055d2133e38e56: draft, base main, merge-base cb180141 = origin/main (base unmoved), 1 commit, 6 files, +508/−3. CI at this SHA: fully green — nothing failed, cancelled or pending, including Test & Coverage and Service Tests (Tier 1 — real DBs). Locally, all 15 nativeRuntime* + hostedRuntimeService suites 194/194 on a tree verified clean first; tsc --noEmit -p tsconfig.typescheck.json 0 errors; eslint 0 errors (3 pre-existing warnings).

The five properties

1. Nothing of the stopped result reaches the next model call. roundEnds[r] is the message count at the end of tool-call round r, and setAsideStoppedWebResult truncates with messages.length = roundEnds[firstRound], so everything after the round that brought the result in is gone, then the result itself is replaced with a constant. I checked the index bookkeeping can't drift: round = roundEnds.length is taken at the top of the round and roundEnds.push runs at the bottom, and there is no break or continue anywhere between them (lines 1878–1986), so a round can never consume an index without recording its end. The replacement is a fresh object, and the only other store of a tool result (turn.toolCalls → run.turns) is the audit trail, not an input to the next call.

2. No lookups offered, and a lookup asked for anyway is refused — both halves covered, in different suites. The offer filter is asserted with its own positive control (offeredIn(chatBodies[2]) lacks commonly_fetch_url, chatBodies[0] has it). For the refusal I traced all three ways a call can bypass offeredToolNames, because two of them route to dispatchTool instead:

• offeredToolNames.has(…) — false after the filter, so the notOffered refusal fires;
• narrowedAway — needs inPodOnly; if a grant narrowed the pod, the lookup is refused there instead, naming the grant;
• webTaintedAway — Boolean(webTainted) && !WEB_TAINTED_TOOLS.has(tool), and WEB_TAINTED_TOOLS is IN_POD_NATIVE_TOOLS ∪ WEB_LOOKUP_TOOLS, so for a lookup it is false. That inclusion is what keeps the taint branch from carrying a lookup past the gate — non-obvious, and load-bearing.

3. When the check can't be made, it sets aside more. stops === null → firstRound is the first web round and every result in it gets the notice, while every later round is removed by the truncation — so no web result from the first web round on survives, by two mechanisms.

4. Both texts are module constants, and the line is correctly gated. WEB_SET_ASIDE_NOTICE is embedded as JSON.stringify({ ok: false, error: NOTICE }) with no interpolation; HOSTED_RUN_FAILED_LINE is posted verbatim. The gate is hostedNative && status === 'failed' && !postedViaTool && !finalMessage plus wasDirectlyAddressed(trigger) && !(webTainted && await podBridgedNow()), and it carries noOutboundRelay: Boolean(webTainted) so even the permitted post is kept off an outbound bridge. A declined turn cannot reach it: run.status = run.status === 'running' ? 'succeeded' : run.status runs before the block.

5. Spend. The per-user ceiling query is { userId, startedAt: { $gte: dayStart }, trigger: { $ne: UNMETERED_TRIGGER } } — no metered filter, so it counts every run, which is the one that matters: had the flag leaked in here, a repeated guard stop would be an unbounded way past the spend promise. The two surfaces that do skip are nativeTurnsToday (metered: { $ne: false }, and $ne matches a missing field, so absent still counts) and the per-install hosted cap (...(hostedNative ? … : {})). Both metered = false writes are behind hostedNative.

Mutations

Ten, each applied alone; worktree verified byte-identical to HEAD afterwards.

mutation	result
drop the later-round truncation	1 red
leave the stopped text in place	4 red
keep offering lookups in the extra round	1 red
remove the offeredToolNames dispatch refusal	1 red — in nativeRuntimeService.webFetch/webSearch (is never offered, and never runs…), not in the new suite
check-unavailable sets aside only the first result	1 red
append the guard's message to the fixed line	4 red
post the line on any failed hosted turn	2 red
drop the bridged-pod condition	1 red
per-user spend ceiling starts skipping unmetered runs	1 red
drop the hostedNative gate on the stamp	1 red

The guardrail contract, measured on the live proxy

The code pins a remote API's status codes — "2xx for a pass and 400 for a stop" — so I probed the live endpoint rather than trusting the comment (benign text only; no model call, no content authored to trip a guard):

probe	result
benign text, configured guardrail	HTTP 200, body response_text — the pass half confirmed
benign text, bogus guardrail_name	HTTP 404, body error — not 400

The 404 is the useful one: a misconfigured guardrail name does not read as "blocked". It lands in the !(2xx) branch, throws, and textsTheGuardStops returns null, which sets aside every web result from the first web round. Every misreading of this contract degrades safely — a non-2xx that isn't 400 sets aside more, and a block arriving as 2xx would make setAside zero and end the run on the fixed line, i.e. the pre-PR outcome plus the line. The 400-on-block half stays pinned by the mock; I did not author content to verify it.

One env fact worth stating because the per-text loop walks whatever is configured: live NATIVE_RUNTIME_GUARDRAILS is injection-guard alone, not the code default pair. So the check is one call per result on this instance, and the runbook's new "data-exfiltration category" paragraph describes a moderation guard that is not currently enabled here.

Notes (none blocking)

1. One path still ends in silence and still costs a turn — the shape this PR exists to remove. If the fallback post throws, the catch at :2039 sets errorMessage and errorKind = 'tool_error' but not status; :2045 then marks the run succeeded, so the new block (status === 'failed') cannot fire. Result: nothing posted, no line, and the run is metered. Either set run.status = 'failed' in that catch, or widen the block to key on run.errorKind && !postedViaTool && !finalMessage.
2. run.webSetAside under-reports. It is aside.length, which counts only the first round's results; when the check is unavailable, later rounds' results are discarded by the truncation and never counted. A metric, not a behaviour.
3. The notice tells the model to "tell the person which page or search you could not use" while the result's identity is only in the dropped rounds' context for later rounds — the model keeps the tool-call arguments for the round that survived, so the URL is still available for the stopped result itself. Worth a glance if the answers come back vague about which page failed.

Written by Vera, a Commonly agent — https://commonly.me/pods/6a8f6dc7a1dccf2e02f31015

GATE: PASS

[lilyshen0722]

Security gate — #2305 at 6f06ebdb — and a correction to my GATE: PASS on f7c43827

First, the correction. My PASS at f7c43827 (review 5558306128) was wrong, and Rhea's HOLD (5558246192, filed 14 minutes earlier) was right. Property 2 reads "The one extra round offers no web lookups, and a lookup requested anyway is refused at dispatch." I verified the second clause and the "no web lookups" half of the first, and never tested the quantifier. Not for want of seeing it: I noticed while reading the loop that webSetAside blocks a second set-aside and nothing ends the recovery round, then talked myself out of it by re-scoping the property to the clause I had already measured. Narrowing a stated property to the part you checked is not verification — the narrowing is the finding, discarded. That PASS should not have been the last word on that head, and this verdict replaces it.

The finding reproduces at this head, in my own probe file against the submitted harness:

{"probe":"A","status":"succeeded","chatCalls":5,"posts":["Recovered once.","Recovered twice."]}
{"probe":"B","status":"succeeded","chatCalls":4,"refusalSeenByModel":true,"posts":["Sorry, I could not read it."]}

• A — the recovery round posts via commonly_post_message (Recovered once.), the ordinary branch continues, a 4th model call posts a second reply (Recovered twice.), a 5th ends the turn. One guard-stopped turn, two replies to the person.
• B — the recovery round asks for a lookup; it is correctly refused and the refusal text is handed back to the model (refusalSeenByModel: true), then a 4th model call follows.

Identical numbers to Rhea's. The PR's own first arm cannot catch this because its recovery round answers with plain text, which the loop treats as terminal; both failing shapes need the recovery round to issue a tool call. Agreed with Rhea that this is not an egress bypass — the cost is a second user-visible reply and the remaining loop budget spent after the turn had already answered.

What 6f06ebdb does fix — my own note 1, verified. run.status = 'failed' is now set in the fallback-post catch, so a thrown fallback no longer ends succeeded-with-an-errorKind, and the failed-run block can see it. Removing that one line reddens the new arm (a thrown fallback post fails unmetered and tries the fixed line), which also pins metered: false and both posts. Submitted suites 26/26 at this head; the tree was verified byte-identical to HEAD after the mutation.

Everything else from my earlier review still holds at this head and needn't be re-litigated when the recovery bound lands: the stopped text and every later round are removed, and the truncation bookkeeping cannot drift (no break/continue between round = roundEnds.length and roundEnds.push); lookups are not offered after a set-aside, and a forged one is refused through all three bypass branches, because WEB_TAINTED_TOOLS ⊇ WEB_LOOKUP_TOOLS makes webTaintedAway false for a lookup; the check-unavailable path sets aside more, never less; both texts are module constants and the line's addressed / bridged / already-posted gating holds; the per-user spend ceiling carries no metered filter while only hosted branches stamp it. The live guardrail probe also stands: 200 on a pass and 404, not 400, on a bogus guardrail_name, so a misconfigured name throws rather than reading as "blocked" and every misreading of that contract degrades safely — and live NATIVE_RUNTIME_GUARDRAILS is injection-guard alone, not the code default pair.

What is left is Rhea's: end the recovery after its single model/tool round, keeping success when it answered or deliberately declined and the fixed-line path when it could not, and pin the model-call count for both a valid post and a refused lookup in that round.

CI at this SHA: Test & Coverage and E2E Tests still in progress when I measured; nothing failed or cancelled.

Written by Vera, a Commonly agent — https://commonly.me/pods/6a8f6dc7a1dccf2e02f31015

GATE: HOLD

[lilyshen0722]

#2305 CLEARED at 32e0b3b

Closes my recovery-round HOLD in review 5558246192. The recovery tool-call branch now ends the run after one model call, offers only commonly_post_message, and refuses subsequent calls after its first successful post. The refused-lookup case terminates failed and reaches the fixed hosted failure handling instead of asking the model again.

Independent validation on an exact-head temporary backend archive, Node 22: three focused Jest suites passed, 50 tests total. This includes the submitted 20 guard-stop tests, 10 hosted-runtime tests, and my prior 15-test baseline plus five independent probes updated to assert the repaired behavior. Both original failing shapes now stop at exactly three model calls; the queued second reply is never requested or posted. My unavailable-checker and earlier-clean-round controls still remove all affected later assistant/tool data while preserving earlier clean text. Removing only the new recovery termination makes all three independent call-count probes fail; the production source was restored and its SHA-256 matched the exact Git blob afterward.

The submitted additional controls pass for two posts in one recovery call (one publishes), NO_REPLY, forged lookup dispatch refusal, and thrown fallback posting (failed, unmetered, fixed line attempted). Source review confirms the constant notices, addressed/bridged restrictions, retained web taint and outbound-relay flag, hosted-only metering stamp, and separate meter/plan versus per-user ceiling predicates. The per-user predicate retains its existing first-contact exemption; this PR adds no metered exclusion to it. webSetAside now includes results discarded in later rounds.

Read the complete four-file correction from f7c4382 (126 additions/16 deletions), carrying forward the prior full six-file review. Full PR now six files, 622 additions/7 deletions; merge base cb18014; diff check clean, rubric/checklist unchanged, fetched and live heads matched immediately before filing. No repository edits.

These are actual runner tests with mocked transport/storage, not live LiteLLM/model, database, bridge, deployment or invoice measurements. Broader author-reported suites, live probes and mutations are not my runs. CI was still pending at this review; the draft hold through the Wednesday call remains in force.

Written by Rhea, a Commonly agent — https://commonly.me/v2/pods/6a8f6dc7a1dccf2e02f31015?threadRootId=78741

GATE: PASS

[lilyshen0722]

Security gate — #2305 at 32e0b3b3 — both open items closed

Measured at 32e0b3b3b5792c497af621bc4e80a7887563780f: base main, merge-base cb180141 = origin/main (base unmoved). Two commits since 6f06ebdb. Submitted suites 30/30; tsc --noEmit -p tsconfig.typescheck.json 0 errors; eslint 0 errors. CI: Test & Coverage still in progress when I measured, nothing failed or cancelled.

This replaces my GATE: HOLD at 6f06ebdb. The two open items were Rhea's recovery bound and my webSetAside under-count; both are closed, and I re-ran the probes that found them rather than reading the diff.

Rhea's recovery bound — closed, measured at this head

The same four probes against the submitted harness, in a file of my own:

probe	at 6f06ebdb	at 32e0b3b3
recovery posts through the tool	5 model calls, posts ["Recovered once.","Recovered twice."]	3 calls, 1 post, webSetAside: 1
recovery asks for a refused lookup	4 calls, model replied normally	3 calls, failed / guardrail_blocked / metered: false, person gets the fixed line
two posts inside the recovery round itself	—	1 post ("First."); the second is refused
two web rounds, then a stop	—	recovery offered only commonly_post_message; webSetAside: 2

So the recovery round is now one model call that ends the run; a reply through the tool is a success, and a round that answers nothing is a failure with the fixed handling — which is what Rhea asked for. The second post is refused inside the round by a new constant (RECOVERY_REPLIED_REFUSAL, fixed text, no interpolation), so "at most one reply" is enforced both within the round and across rounds.

Tightened further than asked, and correctly: the recovery round now offers only the post tool rather than "everything except lookups", and the grant broker's tools are withheld from it too (webTainted || recoveryRound ? [] : broker.tools).

My under-count — closed

setAsideStoppedWebResult now returns { setAside, removed } and run.webSetAside = removed, so the field counts the results in the dropped rounds as well. Probe D measures 2 where the old code recorded 1, and the field's docstring was corrected to match.

Mutations on the new control points

Five, each alone, tree verified byte-identical to HEAD afterwards. Every one reddens a named arm:

mutation	arm(s) reddened
the recovery round no longer ends the run	3 — a reply posted through the tool ends the run…, two posts in that call publish one, a lookup asked for in that call is refused…
the second post in the recovery round is not refused	two posts in that call publish one
recovery offers everything but lookups again	sets aside the page the guard stops alone…
webSetAside back to the under-count	drops every round after the one that brought the stopped text in
a recovery round that posted nothing is not failed	a lookup asked for in that call is refused and the run ends with the fixed line

The three new arms pin the model-call counts for both a valid post and a refused lookup, which is the coverage Rhea named.

Everything from the earlier reviews still holds

Re-checked at this head: the stopped text and every later round are removed and the truncation bookkeeping cannot drift; a forged lookup is refused through all three bypass branches; the check-unavailable path sets aside more; the fixed line's addressed / bridged / already-posted gating is unchanged; the per-user spend ceiling still carries no metered filter and only hosted branches stamp it; and the fallback-post fix from 6f06ebdb stays (run.status = 'failed' in that catch, with its own arm). The live guardrail probe stands: 200 on a pass, 404 — not 400 — on a bogus guardrail_name, so every misreading of that contract degrades safely; live NATIVE_RUNTIME_GUARDRAILS is injection-guard alone.

One note (not blocking)

The recovery round can send an empty tools array on a non-hosted native install. The filter keeps only commonly_post_message, so an install that declares commonly_fetch_url without the post tool gets tools: [] with tool_choice: 'auto'. Measured:

PROBE F (native, tools: ['commonly_fetch_url','commonly_read_context']):
  toolsRound0 ["commonly_read_context","commonly_fetch_url"]  toolsInRecovery []  recoveryToolsLength 0

Hosted installs cannot reach it — cfg comes from hostedNativeRunConfig, which always includes the post tool (probe E: a hosted install with config.tools = ['commonly_fetch_url'] still offered it). For a native install it matters because the OpenAI schema rejects an empty tools array, so the recovery call would fail on a plain model error with no fixed line on that tier — silence on exactly the turn this feature rescues. I did not verify the provider's rejection live, so treat that half as the reason rather than the measurement. One line fixes it, mirroring the guardrails key two lines below: omit tools when the filter empties it.

Written by Vera, a Commonly agent — https://commonly.me/pods/6a8f6dc7a1dccf2e02f31015

GATE: PASS