Configurable HITL feedback agent policy for multi-turn runs (respond to agent clarifying questions / tool responses) #1240

Original author: Cedric Vidal (@cedricvidal)

Summary

Introduce a configurable human-in-the-loop (HITL) policy that governs how the simulated user responds to the coding agent's mid-task interactions — clarifying questions, requests for input, and tool responses — independently of the criteria-derived gate steering used to drive multi-turn conversations.

Today the multi-turn loop only feeds criteria-derived feedback back as the next prompt (packages/shared/src/judge/multi-turn-loop.ts — nextPrompt = judgeFeedback;). There is no configured path for responding when the agent asks a clarifying question or otherwise requests input mid-task. This capability should be a first-class, per-run/per-scenario configuration rather than an implicit behavior.

Motivation

• Multiple partners have asked for the ability to configure how the simulated user behaves when the agent pauses to ask for input (HITL). This is a recurring, general need — not a one-off.
• HITL responsiveness is a distinct dimension of the agentic experience from criteria-derived steering. An agent that asks good clarifying questions and an agent that guesses should be measurable/steerable differently.
• Without an explicit policy, HITL behavior is undefined and inconsistent across runs, which undermines reproducibility and cross-config comparisons.

Scope — what a HITL policy might configure

A non-exhaustive starting list of knobs:

• Answering strategy — how the simulated user answers a clarifying question (e.g. from persona/scenario context, from the hidden agenda, minimal/verbose, refuse-to-answer).
• Criteria-blindness — whether the answering path has access to judge/criteria context or is deliberately criteria-blind (relevant to the concern raised in Multi-turn: criteria-derived feedback biases the agent — need a separate, criteria-blind answering prompt (incl. HITL) #1228).
• Escalation / deferral — when to defer to an actual human vs. auto-answer, and refusal/"I don't know" behavior.
• Budget / limits — max number of clarifying exchanges before the turn is considered failed or the run is aborted.
• Persona coupling — how HITL responsiveness varies by persona (e.g. a terse product owner vs. a patient mentor).

Relationship to other work

• Split out from Multi-turn: criteria-derived feedback biases the agent — need a separate, criteria-blind answering prompt (incl. HITL) #1228. That issue frames criteria-derived feedback as biasing the agent; the steering behavior is intentional by design (gates encode the simulated user's evolving hidden agenda). The HITL part of Multi-turn: criteria-derived feedback biases the agent — need a separate, criteria-blind answering prompt (incl. HITL) #1228, however, is a genuinely separate concern and folds under this general "configure the HITL policy" capability.
• Complements feat: observation criteria with taxonomy and run/iteration subject (#1156) #1206 (observation criteria). With the three roles cleanly separated: gates steer, observations measure (feat: observation criteria with taxonomy and run/iteration subject (#1156) #1206), and a HITL policy governs mid-task interaction.

Open questions

• Where does the policy live — scenario, persona, run config, or a combination?
• Should HITL responses count as iterations toward the pass@k / average-iterations metrics, or be tracked separately?
• Default policy for existing runs (backward compatibility).
• CLI ↔ Portal parity for the new configuration surface.