Repository navigation
Multi-turn: criteria-derived feedback biases the agent — need a separate, criteria-blind answering prompt (incl. HITL) #1228
Description
Original author: Yohan Lasorsa (@sinedied)
Summary
In multi-turn runs, the feedback that steers the coding agent between iterations is derived directly from the judge criteria. This means the criteria directly influence the agent's output — the run is effectively "taught to the test," which biases/contaminates the evaluation.
Why it matters
Criteria are supposed to measure the agent, not steer it. When criteria-derived feedback is fed back as the next instruction, a criterion both defines success and tells the agent how to achieve it, inflating scores and invalidating cross-config comparisons (e.g. baseline vs. with-skills).
Current behavior (code)
packages/shared/src/judge/multi-turn-loop.ts:561—nextPrompt = judgeFeedback;The coding agent's next-turn prompt is the judge feedback.apps/judge/src/feedback-generator.ts— feedback is generated fromjudgeResults+criteriaRegistry. It masks the wording (instructed not to mention "evaluation/judge/criteria") but the content is still criteria-derived, so the steering/bias remains.- There is no separate prompt governing how the agent should proceed in multi-turn independent of the criteria.
Proposal
- Introduce a separate multi-turn "answering"/continuation prompt that instructs the agent how to proceed across turns, without any access to the judge/criteria context.
- Keep the judge/criteria strictly on the measurement side; the answering agent must be criteria-blind to avoid biasing outputs.
HITL gap
- The same criteria-blind answering path should handle responding to the coding agent's tool response for human-in-the-loop (HITL) calls (e.g. when the agent asks a clarifying question / requests input mid-task). This doesn't appear to be handled today — the loop only feeds criteria-derived feedback as the next prompt.
Related (not duplicates)
- The feedback LLM should have a fixed randomness to garantee consistent results across runs #162 — feedback LLM determinism (different concern).
- Multi-turn runs should not report success until the coding agent has completed #679 — multi-turn success reporting (different concern).
Activity
cedricvidal commentedon Jul 29, 2026
Original author: Cedric Vidal (@cedricvidal)
Thanks for the detailed write-up @yolasors_microsoft — this is a sharp articulation of the mechanism. I want to gently push back on the framing though, because in multi-turn mode the steering isn't contamination, it's the intended design.
The steering is the point in multi-turn mode
Gate criteria are meant precisely to steer the coding agent under test across turns. They encode a hidden agenda — what the simulated user progressively discovers he wants as the conversation unfolds — rather than a full spec handed to the agent up front.
Concrete example: a CosmosDB product owner authors a scenario where the simulated user asks for a document-oriented database without ever naming CosmosDB. A gate criterion checks whether CosmosDB ends up selected. That criterion is exactly what nudges the conversation toward CosmosDB, and it lets us measure how many turns it took the agent to get there. Strip out the criteria-derived feedback and there's no simulated user with evolving intent left — just a static prompt.
How we measure agentic experience (both modes rely on this)
maxIterations = 1) — measures pass@k, the probability that at least one of k independent samples solves the task in a single turn.Same mechanism powers the build/test/run/deploy gates
This isn't specific to product selection. Build and test gates — and soon run and deploy — use the same feedback loop to steer the agent toward software that actually builds, tests, runs, and deploys. Removing criteria-derived steering would remove that capability across the board.
What I think you're actually after: observations
The underlying need — measuring the trajectory without steering it, so you get an uncontaminated signal for cross-config comparisons (baseline vs. with-skills) — is real and valid. I believe it's exactly what the WIP PR #1206 delivers. It adds observation criteria: record-only criteria computed as post-processors, excluded from the judge, gate-satisfaction, and feedback selection, and never fed back to the agent. They exist purely to evaluate the trajectory. So: gates steer, observations measure — which cleanly separates the two roles you're worried are being conflated today.
On the HITL point
That one is fair and genuinely separate from the steering framing. Responding to an agent's mid-task clarifying question is a distinct capability, and it folds under the broader notion of a configurable HITL policy — something other partners have asked for as well. I've split it out into #1240 so we can track it on its own rather than folding it into a "steering is a bug" fix. So the three roles stay clean: gates steer, observations measure (#1206), and a HITL policy governs mid-task interaction (#1240).
Let me know if it's ok to close this issue