Repository navigation
feat(context-scout): daemon-owned intent suggestions over a decision engine - #4121
ScriptedAlchemy wants to merge 28 commits into
Conversation
"memory curator" and "transcript files" fired the session-recall hint on "document how the memory curator works in the README" and "delete transcript files older than 30 days"; both are now silent regression cases. The eval cases that used those phrases still match through "automation run" and "raw codex jsonl transcript". The held-out prompt set from the recall-model experiment is kept under evals/intent_engine/ for the daemon-owned intent pass that replaces hook-local hints. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB4wWE2tsLyyxJNkBCqRpD
Work-in-progress snapshot taken after a container restart, so the partial files survive: the pluggable decision-engine client (/v1/systemone), the session-window reader over the LCM raw-message store, and the intent settings block in the domain configuration. Not yet compiled or wired; later commits finish and verify them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB4wWE2tsLyyxJNkBCqRpD
The daemon intent pass needs two inputs: a pluggable typed-question client speaking the shared /v1/systemone shape (noul, choice, score; laya-serve, Kev, Ollama, llama.cpp), with disabled, unreachable, timed-out, refused, malformed and backpressured as distinct outcomes; and a bounded reader of the last N turns of a session from the LCM raw-message store, with tool calls extracted per provider and a rendered transcript truncated from the front so the newest text survives. Neither is wired yet; 33 unit tests cover both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB4wWE2tsLyyxJNkBCqRpD
Work-in-progress snapshot while the contracts, intent settings, and the move of the hint catalog and action-fact predicates into tracedecay-policy are still being finished and verified. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB4wWE2tsLyyxJNkBCqRpD
Plan 41 records the accepted design for one daemon-owned suggestion path (Context Scout) fed by a pluggable decision engine over the last few turns of a session plus deterministic action facts, and the phases that retire the hook-local keyword hints. The eval directory holds the frozen 121-case multi-turn dataset and the head-to-head: Intern-Decision-4B macro-F1 0.80 at a fixed 0.6 threshold, Kev-4B 0.69, Laya 0.42. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB4wWE2tsLyyxJNkBCqRpD
|
Work-in-progress snapshot: the intent selection policy under test and the regenerated TypeScript operations after the contract additions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB4wWE2tsLyyxJNkBCqRpD
Scripts, per-model results and the full report from the 4B head-to-head on the frozen 121-case set. Intern-Decision-4B scores higher at its best per-category thresholds but needs them at 0.8-0.9, raises a false hint on 35% of no-intent cases at a fixed 0.6, and cannot separate session recall; Kev-4B works at mid-range thresholds with 11% false hints, the best session-recall F1 (0.85), native /v1/systemone, and a cheap fine-tuning path, so it becomes the recommended default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB4wWE2tsLyyxJNkBCqRpD
Phase 1 of plan 41. Prompt and tool boundaries now admit an intent orchestration in its own lane (never joining or cancelling feedback work). The pass binds a session-scoped intent address, reads the last turns of the session from the LCM store (waiting up to the configured freshen budget for the boundary's turn to land), asks the configured decision engine one batched request, combines the answers with action facts, and enqueues at most one Session-evidence Scout suggestion that the next boundary claims. Delivery writes a hint_emitted analytics row so the existing settlement records acted or ignored; duplicate and budget suppressions are recorded too. Absent or disabled intent settings never contact the engine. Deviations from the plan text: freshen is a bounded wait for ingest rather than a daemon-driven ingest (the bounded ingest entry point is test-only); analytics rows use the LCM provider label so settlement can find the session; the intent window is Immediate, with deliveries inside min_delivery_gap_secs withheld as suppressed_budget. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB4wWE2tsLyyxJNkBCqRpD
…nt-suggestions Ports two master hint fixes into tracedecay-policy, where the PR moved the classifier: tool-action rules now precede free-text rules (62765ff), and in-repo /projects/ paths no longer imply project context (5ccea9b). The delivery-receipt call passes the global-db lease through AsRef after master changed its type.
results/ and summary.json are 59k lines of model-run output that run.py, run_http.py and score.py regenerate against the model servers. The dataset, its hash, the scripts, the README tables and REPORT.md stay. run_http.py now creates results/ itself.
The pass delivers through the Immediate window and freshens by a bounded wait for ingest rather than driving ingest itself.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 5 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| window.newest_timestamp_micros.is_some_and(|newest| { | ||
| newest.saturating_add(BOUNDARY_TIMESTAMP_SLACK_MICROS) >= boundary_at | ||
| }) |
There was a problem hiding this comment.
🔴 Previous turn judged as current intent
When transcript ingestion lags a boundary by less than one second, read_boundary_window accepts the previous turn. The current prompt or tool result then goes unjudged until another boundary.
Learn more
The intent pass polls the transcript store to include the hook boundary's turn before judging it. It compares the newest transcript timestamp with the hook observation time, allowing a full second of slack. A preceding row timestamped within that second passes the comparison even when the new row has not been ingested. The loop exits immediately, and ask_engine judges the previous window. A later boundary may judge the missed turn, but a final boundary has no later opportunity.
Example: A user prompt is recorded at 12:00:00.400, followed by another prompt at 12:00:00.800. If the second transcript row arrives late, the first row already satisfies the one-second timestamp slack, and the pass judges the first prompt again.
Recommended fix: Determine catch-up from a boundary-specific transcript identity or ingest cursor rather than a timestamp with a one-second allowance. If the provider only supplies coarse timestamps, wait through the configured freshen budget when the newest row could precede the boundary.
Was this helpful? React with 👍 or 👎 to provide feedback.
| let newest = messages.last_mut()?; | ||
| newest.tool_calls.clear(); | ||
| newest.text = head_on_char_boundary(&newest.text, limits.max_bytes).to_owned(); | ||
| if newest.text.is_empty() { | ||
| return None; | ||
| } | ||
| keep_from = messages.len() - 1; |
There was a problem hiding this comment.
🟡 Large tool calls produce empty windows
When a newest tool-only row exceeds max_bytes, select drops its tool calls and returns an empty window. The intent pass skips that tool boundary instead of judging its action.
Learn more
A typed-fact tool invocation is converted into a window message whose argument payload moves from text into tool_calls by window_message. The byte-budget fallback for an oversized newest message erases tool calls and truncates only text. For such a tool-only row, text is empty, so selection returns Empty despite the tool name being present. The pass consequently has neither an engine window nor the deterministic tool action facts.
Example: A Read invocation contains a 160-character argument excerpt and the operator configures max_bytes = 256. With a sufficiently long tool name or several recorded calls, the row exceeds 256 bytes; its empty text leaves no window, although a tool invocation was ingested.
Recommended fix: Preserve a bounded tool name or a truncated call when the newest row alone exceeds the byte budget. Apply the same byte accounting to the retained representation and add a test with a tool-only row over the budget.
Was this helpful? React with 👍 or 👎 to provide feedback.
| .recent_for_protected_session( | ||
| intent_address.protected_session_id, | ||
| control.configuration_revision, | ||
| now, | ||
| STARTUP_RECOVERY_LIMIT, | ||
| ) |
There was a problem hiding this comment.
🟡 Older session intents become suggestible again
After 32 newer deliveries, intent_session_state forgets an earlier delivered intent. decide_intents can then suggest it again in the same session.
Learn more
The intent pass excludes previously delivered intents using the state returned here. recent_matching truncates deliveries to the requested limit, and trim_receipts also retains only the latest 32 deliveries across the project. The older intent therefore disappears from both the transient view and the durable delivery ledger. Subsequent passes treat it as undelivered.
Example: One call_graph intent is delivered, followed by 32 code-evidence suggestions in the same session. A later call-graph question can produce another call_graph suggestion, rather than a duplicate suppression.
Recommended fix: Retain session-scoped delivered-intent markers independently of the bounded recent-delivery receipt list, and consult those markers in intent_session_state. Preserve the markers until the session's intended dedupe lifetime ends.
Was this helpful? React with 👍 or 👎 to provide feedback.
| let has_scheme_and_host = url | ||
| .strip_prefix("https://") | ||
| .or_else(|| url.strip_prefix("http://")) | ||
| .is_some_and(|rest| !rest.is_empty() && !rest.starts_with('/')); | ||
| if !has_scheme_and_host | ||
| || url.len() > DECISION_ENGINE_MAX_BASE_URL_BYTES | ||
| || url | ||
| .chars() | ||
| .any(|character| character.is_control() || character.is_whitespace()) | ||
| || !DECISION_ENGINE_TIMEOUT_MS_RANGE.contains(&self.timeout_ms) |
There was a problem hiding this comment.
🟥 Session transcripts sent to remote engine
When base_url names a remote host, ask_blocking sends the session transcript there. The URL validator accepts nonlocal HTTP endpoints, exposing user messages and tool arguments.
Was this helpful? React with 👍 or 👎 to provide feedback.
| let tail = match db | ||
| .lcm_session_replay_slice(&LcmSessionReplayRequest { | ||
| provider: request.provider.to_owned(), | ||
| session_id: request.session_id.to_owned(), | ||
| head_limit: 0, | ||
| tail_limit: fetch, | ||
| max_snippet_chars: 1, | ||
| summary_limit: 0, | ||
| max_summary_chars: 0, | ||
| }) |
There was a problem hiding this comment.
🟥 Cross-project transcript mixed into intent window
When a session ID spans project scopes, read_session_window selects both projects' messages by provider and session ID. The active project's engine receives the other project's transcript.
Was this helpful? React with 👍 or 👎 to provide feedback.
This branch has not been deployed
2 participants
Summary
Phase 1 of
docs/plans/tracedecay-v2/41-daemon-intent-pass-and-unified-suggestions.md: the daemon reads the last few turns of a session from its own transcript store, asks a pluggable local decision engine per-intent yes/no questions, combines the answers with deterministic action facts, and delivers at most one Context Scout suggestion through the channel every host already uses.It is opt-in. With
ContextScoutSettingsV1.intentabsent or disabled (the default), the engine is never contacted, no intent suggestion is produced, and a Claude tool call does no new work. Hosts and the prompt hot path (Phase 2) and deleting the legacy keyword path (Phase 3) are not in this PR.What lands
intent_pass.rs,decision_engine.rs(blocking client for the typed-question API) andsession_window.rs(last-N-turns reader overlcm_raw_messages) intracedecay-agent-hosts. Typed outcomes cover disabled, window empty or stale, suppressed, cancelled and enqueued.HookOrchestrationTriggerV1::IntentBoundaryfor prompt and completed or failed tool boundaries; Stop keeps its trigger and gains an intent companion. Intent work never joins, cancels or takes the permit of feedback work.run_production_intent_cycleruns it in the daemon.ContextScoutSettingsV1.intent(engine, window, question thresholds, delivery gap); session evidence and the intent receipt are additive contract variants. The dashboard settings patch now acceptscontext_scout_intent.tracedecay-policy; existingtool_hintstests pass unchanged.hint_emittedrow and suppression rows, so existing settlement writeshint_outcome.evals/intent_engine/holds the frozen dataset, runners, and the write-up that picked Kev-4B as the recommended default. Raw per-run outputs are not committed (.gitignored).Defects found by the end-to-end test and fixed here
The new journey test
claude_stop_delivers_the_intent_suggestion_the_daemon_judged_at_the_tool_boundaryspawns a daemon, runs real Claude hook commands against a loopback engine, and checks delivery at Stop. It exposed three defects the unit tests did not:lifecycle_absent. Claude tool calls never minted a Scout lifecycle, so the pass had no address to bind.NativeContextScoutLifecycleV1::host_mounts_lifecyclenow includes Claude.budget().exhausted. Engine failures no longer touch the model status slot or the budget.The gate
Fix 1 as first written made every Claude tool call, in every project, do a synchronous lifecycle write inside the 25 ms hook budget, with Context Scout disabled. It is now gated in
hook_runtime/admission.rs: a Claude tool call mints its lifecycle only when the project's intent pass is enabled (same check the pass makes: intent enabled and Context Scout not disabled). Kimi Code and OpenCode keep minting theirs unconditionally. The cost on the default Claude path is one configuration read in the daemon instead of a write. Test:claude_tool_calls_mint_a_lifecycle_only_while_the_intent_pass_is_enabled(fails with the gate removed).Also in this PR
project_composition.rs:Box::pinonrelease_capacity_retired_stores, which--config=clippyrejects aslarge_futureson current master (introduced by the merged capacity-release change, unrelated to this feature).Verification (merged head 55ff49a)
unit_testfor policy, domain, contracts, agent-hosts, daemon-service, mcp, application, dashboard-api;policy_suite,context_scout_evidence;//crates/tracedecay:unit_test,:mcp_suite,:daemon_suite,:hooks_lsp_suite: all pass.bazel build --config=clippyover the touched crates,rustfmt_check_2024on them,//:rustfmt_check: clean.contracts:check,typecheck,exec vitest run(215 files, 2094 tests);sdks/typescriptbuild andplugin/chatgpt-extensionbuild.mjs --check: pass.hooks_lsp_suiteflake:hook_spool_drain_test::an_edit_in_a_never_opened_linked_worktree_binds_it_and_reaches_the_daemonfails intermittently withhook_v2_catchup_requiredat about 10 to 13 percent of runs under parallel load on both this head (7 of 60 runs) and origin/master (4 of 30). Pre-existing, not introduced here. The new journey test passes 30 of 30.Not done
run_production_intent_cycleis covered by the journey test with one project and one loopback engine; multi-project and engine-failure-under-load runs are not.