Repository navigation
feat(hooks): ask an optional Laya model about session recall - #4089
ScriptedAlchemy wants to merge 1 commit into
Conversation
The session-recall keyword list caught 3 of 30 natural requests for earlier context in a held-out set. When TRACEDECAY_HINT_RECALL_MODEL_URL names a running laya-serve, the Codex prompt hook asks it one calibrated yes/no question for prompts no keyword rule classified; at p >= 0.6 it gives the session-recall hint. Unset, unreachable, slow, or malformed answers keep the keyword result, and delivery budget and dedupe are unchanged. Held-out result: 22/30 caught, 0/48 false hints. Drop the bare "memory curator" and "transcript files" recall phrases, which hinted on ordinary docs and cleanup tasks; the eval cases that used them still match through more specific phrases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HB4wWE2tsLyyxJNkBCqRpD
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 2 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| let timeout = timeout_ms | ||
| .and_then(|raw| raw.trim().parse::<u64>().ok()) | ||
| .filter(|ms| *ms > 0) | ||
| .map_or(DEFAULT_TIMEOUT, Duration::from_millis); |
There was a problem hiding this comment.
🟡 Model timeout exceeds prompt hook deadline
If TRACEDECAY_HINT_RECALL_MODEL_TIMEOUT_MS exceeds five seconds, a stalled model blocks the prompt hook past Codex's deadline. Codex terminates the hook before it writes workspace guidance or other hints.
Learn more
The installed Codex UserPromptSubmit command has a five-second deadline in CODEX_MANAGED_HOOKS. The configured model timeout is not bounded by that deadline, and codex_user_prompt_submit_context_with_root waits for the model before the handler writes its response. If the request stalls beyond five seconds, Codex kills the hook and loses the entire prompt context, not just the optional model hint.
Example: With TRACEDECAY_HINT_RECALL_MODEL_TIMEOUT_MS=10000, a stalled model keeps a prompt hook occupied for ten seconds. Codex's five-second deadline expires before the unindexed-project status line is emitted.
Recommended fix: Bound the model call by the remaining prompt-hook budget, reserving time to emit existing context; reject or clamp configured timeouts that exceed that budget.
Was this helpful? React with 👍 or 👎 to provide feedback.
| Some(Self { | ||
| endpoint: format!("{}/v1/systemone", base.trim_end_matches('/')), | ||
| agent, | ||
| }) |
There was a problem hiding this comment.
🟨 Remote HTTP endpoint exposes user prompts
When TRACEDECAY_HINT_RECALL_MODEL_URL names a remote HTTP server, asks_for_recall sends the full prompt without encryption. Network observers can read confidential prompt text.
Was this helpful? React with 👍 or 👎 to provide feedback.
Pull request was converted to draft
|
Superseded by #4121. This PR put a Laya client inside the hook-local classifier, which plan 22 says to remove; the replacement moves intent judgement into the daemon's single Context Scout channel for every host. The parts worth keeping (the two over-broad session-recall phrases and their regression cases, and the held-out prompt set) are carried over in #4121's first commit. Generated by Claude Code |
2 participants
Summary
TRACEDECAY_HINT_RECALL_MODEL_URLpoints at a running Laya server (laya-serve), the Codex prompt hook asks it one yes/no question, but only for prompts that no keyword rule matched. A probability of 0.6 or more yields the session-recall hint. With the variable unset, behaviour is unchanged.memory curatorandtranscript filesfrom the session-recall keywords. They fired the hint on ordinary tasks.evals/hint_recall_model/: a 78-prompt held-out set and a runner that scores the keyword list, the model, and the combined policy against a live server.Motivation
The session-recall keyword list (
asks_for_session_recall) catches every phrasing in its own eval matrix and almost nothing else. The held-out prompts were written separately from the keyword list: 30 natural requests for earlier context and 48 prompts that are not recall requests. 25 of those 48 mention memory, sessions, history or transcripts. Results overlaya-serve(laya 0.4.8, English checkpoint, CPU):On the 60 prompt-only cases of the existing Rust eval matrix, the combined policy keeps 8/8 recall cases. It adds one false hint: "change the button text to Save and run the narrow test" (p = 0.73).
Changes
tool_hints/recall_model.rs(new): a blockingureqclient forPOST /v1/systemone, using the sharedhttp_agent, no proxy and a whole-request timeout (TRACEDECAY_HINT_RECALL_MODEL_TIMEOUT_MS, default 250 ms). An unreachable server, a timeout, a non-200 status, malformed JSON or an out-of-range probability all count as "no opinion", and the keyword result stands.tool_hints.rs:decide_hint_with_recall_model. The model is consulted only for a bare prompt (no tool name, no command) whenclassify_hintreturned nothing, so it never overrides a keyword category.decide_hintis unchanged, and Cursor's tool-event hints still use it.codex.rs:codex_prompt_hintuses the model-aware decision. Delivery budget, dedupe and escalation (tracedecay-policy::hint_delivery) are untouched.classifiers.rs: drops the two broad phrases. The existing cases that used them still match throughautomation runandraw codex jsonl transcript. Two silent regression cases are added incases_real_world.rs.Test plan
github.com/bats-core/bats-core/archive/v1.10.0.tar.gz, so CI is the first Bazel run.bazel build //...passes when the change crosses package boundaries (single crate plusevals/).cargo test -p tracedecay-agent-hosts --lib -- hooks::tool_hints hooks::codex: 35 passed, 0 failed. This covers all hint eval matrices, 9 new tests (threshold, request shape, error and timeout fallback, model not consulted for keyword hits, tool events or disabled hints) and the Codex prompt dedupe test.cargo clippy -p tracedecay-agent-hosts --lib --tests: no warnings in this crate.python3 evals/hint_recall_model/run.py --url http://127.0.0.1:8765against a locallaya-serve, which produced the table above.Checklist
CHANGELOG.md: left to release-please, which generates it from the conventional commit..envfiles included.🤖 Generated with Claude Code
https://claude.ai/code/session_01HB4wWE2tsLyyxJNkBCqRpD
Generated by Claude Code