fix(skills): stop headless runs halting at phase boundaries in Opus 5 - #1250
Draft
kieranklaassen wants to merge 14 commits into
Draft
fix(skills): stop headless runs halting at phase boundaries in Opus 5#1250kieranklaassen wants to merge 14 commits into
kieranklaassen wants to merge 14 commits into
Conversation
…tion ceremony Under headless runs the pipeline halted partway: skills asked blocking questions with no documented fallback, and hand-off prose addressed a "caller" that, inline, is the same model. Measured over 367 archived benchmark runs plus 12 controlled runs on unchanged code: /lfg pipeline adherence sits at 58% while the underlying task completes 75%, and the models that fail do so at one seam 47-58% of runs where succeeding models fail there 0-8%. Adds 55 inline unattended defaults across 22 skills so an unattended run takes a documented default, records the assumption, and continues instead of stopping. Removes self-verification passes over generated output, dispatch floors that mandated subagent fan-out regardless of warrant, no-skip-on-cost clauses, stale inventory, and per-skill boilerplate. Two supporting facts drove the removals. Current frontier models self-verify, so mandated verification passes cost tokens without improving results. And of 332 archived runs dispatching 5+ helpers, 331 never dispatched more than one concurrently, which falsifies the wall-clock justification this corpus wrote for its own mandated fan-out. 386,034 -> 304,764 words (-21.1%), 221 -> 190 markdown files. Every proposed cut was adversarially defended against docs/solutions learnings and tests first; 81 of 616 were skipped as load-bearing. Machine-readable contracts other skills grep, platform gotchas where wrong behavior looks like success, and genuinely interactive product behavior were held intact. Deliberately excluded: the hand-off/turn-boundary rewrite, which needs one complete pass with a test-retargeting budget. This is not yet a measured performance result. Mechanical gates green means nothing broke, not that adherence improved; against a 58% floor with 7.1x token variance that claim needs ~39 runs per arm. Co-Authored-By: Claude <noreply@anthropic.com>
…repo Pre-release model ids are under NDA until launch, and the realistic leak path is transcription: measuring a change against an unreleased model, then pasting the resulting table, benchmark row, or config snippet into docs/. Matches the shape of an early-access identifier rather than a list of codenames, because a test enumerating them would publish the very thing it protects. The forbidden suffix is assembled from fragments so the file does not contain it, and failures report file:line only, never the matched text, since CI logs are public too. Released model ids are deliberately not matched. Verified both directions: catches three-part ids, env assignments, pasted benchmark rows and the bare acronym, with no false positives on "cheapest", "reap", "heap", "leap", "release-please", or any public model id. Co-Authored-By: Claude <noreply@anthropic.com>
…e regression A new model performed worse driving this plugin than with no plugin at all. The obvious response, reading the skills and rewriting the bad sentences, produces a plausible fix list with no way to tell whether any item matters. Records the measurement path: mine the existing run archive before spending new runs, exclude broken runs as a first-class outcome (20% of ours scored as model failures and falsified our first headline), keep "followed the process" and "did the job" as separate metrics, localize the defect with a model split rather than by reading prose, and measure the A/A noise floor before claiming any effect. Includes the hypotheses this killed, which matter more than the confirmations: mandated fan-out does not cause the halts (the correlation runs the other way and is confounded by progress), the monotone effort effect was an artifact of broken runs, and roughly half of an outside analyst's paste-ready fix list did not apply to the current tree. Model identities are omitted; what transfers is the shape of the result. Co-Authored-By: Claude <noreply@anthropic.com>
…sh state Three artifacts had been sitting untracked: a nightly-plugin requirements brainstorm, an explainer-topics ideation page, and an evidence-based reduction plan derived from transcript analysis. All three are durable analysis in the conventional docs/ paths, so they belong in the repo rather than in a working tree indefinitely. Also ignores .herenow/, which holds published-site URLs written by the here-now skill. That is local tooling state, not repo content. Scanned all three for pre-release model identifiers and credentials before committing, since committing makes them public. Clean. Co-Authored-By: Claude <noreply@anthropic.com>
…tion The guide says to test whether a redundant-looking line is insurance for a more literal model before removing it, and to update the guide rather than multiply local carve-outs when the same justified exception recurs. A corpus-wide reduction took that exception 535 times across 31 skills, so it belongs here. Two gaps this closes. The existing instruction assumes the test exists; sometimes it cannot be run at all, because the benchmark harness refuses to load a modified skill on the weaker runtime, which makes the insurance hypothesis unfalsifiable with the available lever. And the evaluation priority list leads with the weakest realistic layer, which is the wrong order when the measured harm is on the strongest runtime available. Adds the resolution order for an untestable insurance hypothesis, the three conditions a capable-model-first program must meet, the priority inversion with its precondition, and a requirement to establish a noise floor from two identical copies before crediting any change. Also names the reversal explicitly: this repo has a prior learning concluding a defensive fix should ship even when a capable model cannot reproduce the failure, because weaker harnesses are targets too. A capable-model-first program reverses that reasoning and should say so rather than let a large diff imply it. Net addition to a guide used in a reduction program, deliberately: this is a named exception with a measurement precondition, which is protocol, not ceremony. Co-Authored-By: Claude <noreply@anthropic.com>
A matched pair on a full brief isolated this. The pre-cut arm died at ce-plan waiting on helpers the harness had terminated. The post-cut arm cleared that phase, wrote a 2,034-line plan, ran document review, then ended its turn with "Returning control to LFG" after 105 minutes and 617k output tokens. Phase transitions do not spawn anything. The Skill tool loads instructions into the same conversation: one session id, one model, one turn. The failing run shows two Skill calls and ten Agent calls in a single session file, so helper dispatch is a real boundary and the phase seam is not. Every "return control to the caller" is written about the seam that is not a call, which makes it a message the model addresses to itself, and a message with no tool call ends the session. Rewrites 53 sites in ten phrase families: return-control-to-a-party, ownership disclaimers across a same-model seam, envelope-as-return (verb swap only, every field name and status enum untouched), model-subject end-the-turn where work remains, output-format-as-stop, the filler-tool-call guard, terminal-sentinel justifications, and three families inside subagent prompt assets. Those last three matter more than their count suggests. Assets saying "report and stop" or "return only a gist" become live session terminators the moment a documented no-subagent fallback inlines them into the orchestrator, which several skills do. Fixed only in skills that document such a path. 88 sites verified as legitimate survivors across five screens: the user is the subject of the stop, the stop is a terminal no-op with no work left, the roles split across a real runtime boundary, the site is already an unattended default from the previous pass, or "stop" bounds an activity rather than the turn. Genuinely interactive stops are intact and one gained a missing unattended degradation path. Greppable markers keep their literal strings and lose the justification that taught the model a separate consumer was waiting. review.json already carries the machine-readable payload, so the reply is no longer declared the only channel. Two test retargets, both net-adding an assertion; no test deleted or weakened. Co-Authored-By: Claude <noreply@anthropic.com>
The pipeline had no instruction to advance. Steps 2 to 3, 3 to 4, 4 to 5, 5 to 6,
6 to 7, 7 to 8 and 8 to 9 each simply ended: the model finished a step and nothing
told it to begin the next. Every "proceed" in the file was negative ("Do NOT
proceed to step 2"), conditional, or jumped straight to step 10. Two phase gates
were also headed "GATE: STOP.", which reads as an instruction to halt rather than
to verify.
That combination is enough on its own to end a headless run, independent of the
hand-off framing removed in the previous commit: reaching a boundary with no
stated next move leaves ending the turn as the only reading.
Adds an explicit "continue to step N" at each transition, replaces both gate
headers with checkpoint wording, and states once up front that a gate means
verify and continue rather than stop and wait. No test pinned either gate string.
Found by prediction rather than by another benchmark run: the two prior halts both
occurred at a phase boundary, so the boundaries themselves were audited instead of
waiting three hours to observe the next one.
Co-Authored-By: Claude <noreply@anthropic.com>
…arve-out inline Re-reading the failing trace corrected the diagnosis. It recorded `ce-plan > ce-doc-review`, which looked like two pipeline phases completing — but `ce-doc-review` is invoked by `ce-plan` as its own sub-step 5.3.8, so the run never reached the pipeline's second stage. It stopped on the way out of the first. Three defects, all the same shape: a stop instruction separated from the condition that exempts it. The non-interactive contract defined completion and stopped there: "`ce-plan` is done once the plan file is written, the confidence check has run, and `ce-doc-review` has run headless." Being done was never connected to what happens next, and "I am done" reads as the end of the turn. The pre-response check told the agent to render the handoff menu and wait, two paragraphs after the sentence that exempts pipeline runs from it. A more literal reader loses a distant qualifier, so the exemption is now stated at the action it governs rather than earlier in the file. Source-document disambiguation asked the user and waited with no unattended path. It now takes the most recently modified match and records that as an explicit assumption in the plan. Co-Authored-By: Claude <noreply@anthropic.com>
The Return-to-Caller envelope was introduced by the bare label "Return:", the last instance of that verb left in the skill that historically absorbs most pipeline failures. Its prose had already been corrected to "records ... the remaining gates run later in the same session", so the heading contradicted the paragraph above it. Now reads "Record these fields, then continue with the run's next stage", which supplies the forward motion the label never had. Field names and the status enum are untouched; the surrounding contract test slices on section boundaries and does not pin the label, so no test change was needed. Co-Authored-By: Claude <noreply@anthropic.com>
…skipped step
Two sites stopped the whole run over a step that merely could not execute, and both
were invisible to every test run so far: the probe used for verification has no
browser surface, so this skill never fires in it.
The driver bootstrap told the user to install `agent-browser` and stop. The skill
already declares itself unattended in `mode:pipeline` ("never block on a question"),
and its dev-server check already carves out pipeline mode — this path did not. A
missing optional CLI now records every planned route as Skip, reports PARTIAL, and
continues.
Same fix in the pipeline orchestration reference, where a dev server that never
bound to its port also stopped the run.
Both use vocabulary the skill already had: unreachable flows are logged as Skip
with a reason and the run keeps going, with an overall PARTIAL result.
This matters more than the two lines suggest. Most benchmark briefs are web apps,
so any machine without `agent-browser` — or a dev server slow to bind — would have
halted at the browser step while the browserless probe stayed green.
Left alone: four nearby "report and stop" sites that are genuine terminal no-ops
(no commits to describe, no open PR, no feature-branch work). Nothing remains for
anyone to do there, so continuing would be the defect.
Co-Authored-By: Claude <noreply@anthropic.com>
…resolved The pre-create PR re-check told the agent to resolve `gh auth status` or connectivity before creating the PR. In an unattended run there is nobody to fix either, and no fallback was stated, so the instruction had no reachable outcome. Now reports the failure and skips PR creation — the commits and push already stand — then continues with the run's next stage. The sibling rule two lines down already says never to strand the flow between commit and PR; this applies the same principle to the case where gh itself is the thing that failed. Co-Authored-By: Claude <noreply@anthropic.com>
A remote-backed run completed the task, pushed a branch, handled an impossible PR creation gracefully, emitted DONE, and produced no halt signature. It also never invoked ce-plan or ce-work. Its own summary: "I ran planning and implementation myself rather than delegating ... the delegation overhead bought nothing. That's a deviation from the LFG script." So the task shipped while the plan artifact, the review, and the verification evidence the later steps read did not exist, and the run reported success anyway. This is a regression from removing the dispatch floors. That removal was right for what it targeted: prose mandating helper fan-out inside a skill regardless of warrant, contradicted by 331 of 332 archived runs never dispatching more than one helper concurrently. But it left nothing saying that the pipeline's own phase invocations are mandatory, and the model reasonably filled that silence with its own judgment about overhead. Now states once, up front, that each step's invocation is mandatory unless that step documents a skip condition (3, 6, 7 and 9 do), that task size is not a skip condition anywhere else, and that a skill decides its own internal delegation while whether the step runs at all is not its call. A genuinely uninvokable skill must be named and the run continued, never silently replaced by an inline substitute. Found by the two-metric split: task completion and pipeline adherence are tracked separately, so a run that did the work without the process reads as a defect rather than a pass. Co-Authored-By: Claude <noreply@anthropic.com>
Four commits landed on main during this work and collided with it in 70 hunks across 31 files. The overlap was almost entirely one feature: configurable docs_root replaced hardcoded `docs/` paths with a `<root>/` placeholder, in exactly the prose this branch had compressed. Resolved by keeping both intents rather than choosing a side: the branch's compressed, de-halted wording carrying main's `<root>/` placeholder. Verified after resolution that the branch's purpose survived: zero conflict markers, the halt-prose sweep back to its pre-merge count of one (a pre-existing mode:agent contract table cell, outside every conflict region), lfg's nine `continue to step N` connectives intact, six `<root>/` references ported in, and zero stale `docs/residual-review-findings` paths left behind. Also drops three analysis artifacts committed earlier in the branch that did not belong in the repo, leaving that commit's `.gitignore` entry for `.herenow/` as its only surviving content. Keeps deleted the three persona files this branch replaced with ce-pov's grounding checklist, which main had only touched to apply the docs_root rename. Co-Authored-By: Claude <noreply@anthropic.com>
Collaborator
Author
|
Testing a few hours locally |
kieranklaassen
changed the title
Five commits landed on main since the last merge, including the ce-retune skill and a fix that probes for a working Python interpreter instead of hardcoding python3. All four conflicts were the same shape: the interpreter-probe fix landed inside command blocks this branch had also edited. Resolved by taking main's side in every hunk — it is a genuine cross-platform fix, and none of it is turn-ending prose (its exit-1 guards bound a command, not the run). The halt-prose sweep now returns three hits, all inside ce-retune/references/halt-taxonomy.md, which documents those exact strings as grep patterns for finding the regression class — detection instructions, not live ones. The pre-merge baseline of one (a mode:agent contract table cell in ce-code-review) is unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Autonomous
/lfgruns stopped partway through. The model would finish a phase, correctly name the remaining steps, and end its turn without doing them. Its closing lines were things like "Returning control to LFG" and "waiting on those before composing" - addressed to a caller that does not exist, because phase transitions load instructions into the same conversation rather than spawning anything.Eight halts are fixed here. All eight reduce to one error: prose written as if a second party were waiting. A caller to hand back to, helpers to wait for, a next step someone else begins, a consumer who picks up after "done", a recipient for a
Return:, a user who installs a missing tool, someone who fixes broken credentials - and, once the mandates were cut too far, nobody to say the phases are required at all.The corpus also drops from 386,034 to 305,115 words (-21%), with 55 unattended defaults added where skills previously asked a question with no fallback.
Why the diff is mostly deletions
The failing mechanism was documentation of a runtime that had stopped existing. A
Skilltool call does not spawn a subprocess: it loads instructions inline and the same turn continues. A failing run's own log shows twoSkillcalls and ten subagent calls under a single session id, so helper dispatch is a real boundary and the phase seam is not. Underclaude -p, an assistant message with no tool call ends the session, which leaves "return a summary to the caller" exactly one expressible form.Return:Design decisions worth reviewing
Field names are data; prose is not. Every envelope field, status enum, and greppable terminal marker survives untouched, because three test suites and a live gate depend on them. What went is the justification clause around them.
Documentation completestays as a literal string; "so callers can detect completion" does not.88 stop sites were deliberately kept. Sorted by one question: who is actually on the other side. Kept when the user is the subject, when nothing remains for anyone to do, when the roles split across a real runtime boundary, when the site is already an unattended default, or when "stop" bounds an activity rather than the turn.
ce-explain's check-in pause andce-brainstorm's one-question interview are the product, not ceremony; they gained an unattended degradation path instead of a deletion.Removing the dispatch floors caused a regression, and the last commit fixes it. Cutting "you must delegate" was well-supported: vendor guidance says so, and 331 of 332 archived runs never dispatched more than one helper concurrently, which falsifies the wall-clock justification the corpus wrote for its own machinery. But it left nothing saying the pipeline's own phase invocations are mandatory, and a run then shipped a task inline while skipping
ce-planandce-workentirely, reporting success with no plan artifact.4cf86bfedraws the line: a skill decides its own internal delegation, but whether a step runs at all is not its call.Two shipped documents changed, not just skills.
docs/solutions/skill-design/portable-agent-skill-authoring.mdgains capable-model-first as a conditional deviation, because the guide's own rule says a recurring exception belongs in the guide rather than in 535 local carve-outs - and because this repo has a prior learning concluding the opposite, which deserves to be contradicted explicitly rather than by implication.Validation
Measured on a probe brief that traverses every phase boundary, same model and effort throughout.
The before column is an A/A control: two identical copies of the pre-change plugin, which completed 7 of 12 with output tokens ranging 21,872 to 155,682. That spread is why the sample size was fixed in advance at 8 runs rather than chosen after seeing results, and why earlier small-sample claims about this pipeline were unreliable.
Separately, a run against a repository with a configured remote exercised the shipping branch that no prior run had taken: the branch pushed, an impossible
gh pr createwas handled with a reasoned skip rather than a halt, and both the plan artifact and the residual-findings record were written. A live pull request with real CI is still unexercised.bun run test2582 pass / 0 fail,release:validateandplugin:validate --strictclean. Two test retargets, each net-adding an assertion; no test deleted or weakened.Test plan
bun run test,bun run release:validate,bun run plugin:validatetests/no-unreleased-model-names.test.tsfails if a pre-release model identifier reaches this public repo. It matches the shape of such an identifier rather than a list of names, since a test enumerating them would publish what it protects, and reports file:line only because CI logs are public too.