Skip to content

docs(rfc): record the worker-debate design and propose claim checks for critics - #33408

Merged
iamwhatever merged 1 commit into
mainfrom
docs/rfc-worker-debate
Nov 9, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
docs/rfc-worker-debate

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The team can read the whole worker-debate design, its history and the improvement the owner adopted in a single RFC, so the skill change in #33783 traces to an accepted decision.

The worker-debate protocol (a Conductor has several workers argue one question in files) is skill text only. It grew through four docs PRs (#26316, #26932, #27969, #29475) and never had an RFC. The one RFC that describes it, rfc-worker-messaging.md, predates three of them and still names max_rounds. The owner also observed that a debate's result is weaker than a "grill me" session's, and that the move from mixed models to one model is recorded only in a PR body.

Why it matters

The owner wants to share the design with the team. Without one record, a reader has to rebuild it from skill text and four PR bodies, and the reasons for the model change and the stop rules are easy to lose. Shallow critic objections cost debate rounds and let untested assumptions through to later work.

Not a goal

What changed (motivation → approach → change)

New docs/request-for-change/rfc-worker-debate.md (status accepted), indexed in docs/request-for-change/README.md. The status records the repository owner's decision of 2026-11-09 in the conductor chat, verbatim: "直接改 skill 实现这个方案" (adopt Option B as amended by the §2.7 debate). It takes effect when this document merges, the same way rfc-drop-goal-item-cap.md was recorded. implementation-prs: [33783].

Section Content
§1 Current design History of the four PRs; the protocol part by part with file:line on main b94f8cb3ac; a table of where other docs disagree with the skill text
§2 Critics question harder What grill sessions do (built-in grill skill, Research Lab grill tree, an external grill-me interviewer described in our own words); what the critic role card, Round 0 list and objection table lack; evidence from a real 2026-11-02 debate note (50 objections, 40 closed by adding text, plan grew 2.4x, an unchecked proposer claim about pod up --seed, an assumption about the fake backend that failed in use); four options compared; recommends Option B, claim checks: question-form objections on a quoted claim, answered with evidence, a proof column, a named non-author checker, fixed_verified only after a critic confirms the evidence holds
§3 Model choice Mixed families in #26316 → one model by default in #27969 (#27956), the reasons and what evidence exists; what one model loses; when a mixed panel is still worth it (owner asks; judgment-heavy question; one llm-council pass on the final)
§2.7 Debate on critic roles Result of a worker debate run 2026-11-09 (1 proposer; critics focus, overlap, cost; 46 rows; open blocking 2 → 2 → 0; 2 rounds + wrap up; about 80 min; 4 work items; 0 resends; OUTCOME: stable). Keeps roles set up front: no roles and roles after a roleless Round 0 are rejected; the one real miss (2026-11-02 fake backend) was a depth miss inside a role. Adds a Round 0 biggest-risk entry with a row, one Conductor side-by-side read in the existing round-1 relay, and a same-as rule. Amends Option B (§2.4): a named non-author checker for every proof cell and adopted recommendation, asking "what if it is wrong on its path", in the existing critique step; Q: marks question-form rows. Skill change about 34 lines, 0 new relays, rounds or work items. Known limits listed
§4 Open questions Seven, each with a recommendation; Q1 and Q2 marked answered by the debate; Q7 (untracked Round 0 majors) added

rfc-worker-messaging.md: two stale lines fixed in place, since the code wins and the doc is the bug: trigger T-b now names hard_rounds, and open question 4 (round check-ins) is marked answered by #26932 with a pointer to the new RFC.

Backwards compatibility

Compatible: docs only; no tool, schema, skill text or stored record changes.

Removes nothing: the two edited lines in rfc-worker-messaging.md correct names to match the current skill text; no path, key or capability is taken away.

Tests

Proof: not provable -- docs or config only

Ran locally on head 5f0fed3deb (docs-lint; the 238 tests ran on 2552279648, same RFC files): scripts/docs-lint.sh (pass), test/test_pr_quality_gates.py and test/test_pipeline_conductor_skill_contract.py (238 passed).

Manual verification

N/A — docs only. Every file:line in the RFC was re-read on main b94f8cb3ac (the goal-conductor/SKILL.md and crew-mode.md line numbers moved and were updated).

Related Issues

Closes #33405

Companion skill PR (Phase 1, implements §2.7 in goal-conductor/references/worker-debate.md): #33783. Its First Principles lane traces to this document, so merge this PR first, then rebase #33783.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 8, 2026 20:41
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 5f0fed3deb53711900e92e0b2313a055008adc07 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 5f0fed3deb53711900e92e0b2313a055008adc07 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 5f0fed3deb53711900e92e0b2313a055008adc07 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Docs-only RFC with no take-away; the corrected hard_rounds/check_every names match goal-conductor/references/worker-debate.md.

[DESIGN-REVIEWED] 5f0fed3

Description read

[DESCRIPTION-READ] 44a9f5f021648b61364efbef5c470add8f4a5ad1b8998b3d0587535d5d401d1d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of cd0c1239dccbd91bcc383ed760e62901fa833e04 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] cd0c123

False positive or not applicable? A repository writer can comment:
/ai-review override gpt cd0c1239dccbd91bcc383ed760e62901fa833e04 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 5f0fed3deb53711900e92e0b2313a055008adc07 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5f0fed3

Verdict parsed from the review's SHA-scoped output markers for commit 5f0fed3deb53711900e92e0b2313a055008adc07.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 5f0fed3deb53711900e92e0b2313a055008adc07 false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever force-pushed the docs/rfc-worker-debate branch from 1ebc6bd to c3ab8d0 Compare November 8, 2026 21:12
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 8, 2026
@iamwhatever
iamwhatever force-pushed the docs/rfc-worker-debate branch from c3ab8d0 to 589eb2d Compare November 8, 2026 23:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 8, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

RFC to record what already exists

@bolichen97

Copy link
Copy Markdown
Contributor

This head (589eb2d) conflicts with main. GitHub reports it as not mergeable. All 97 checks are green. Could you rebase onto main? The conflict is likely in the long docs/request-for-change/README.md audit paragraph, which other RFC PRs keep editing. I'll review the RFC on the rebased head.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 9, 2026
@neilieva

neilieva commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Option B checks the proposer's claims. I'd like it to check the critic's own work too.

I run mixed-model review panels often, the kind §3.3 points to llm-council for. Each critic is a spawn_run task pinned to a different model. A critic finishing successfully has not been good evidence that it reviewed anything. These are failures I've hit, and each one finished as a normal completion:

  • A critic made zero tool calls and said the file "was not provided", when the brief told it to read a file. This happened three times with the same model.
  • A critic cited line numbers in files it never opened.
  • A critic started in its own empty working directory, missed the repo path in its brief, and searched that empty folder. It spent all 120 turns without finding the code, and the turn limit stopped it.
  • A critic paged through a long file chunk by chunk and looped for 38 minutes, because a read past the end of the file returns an empty success.

I ended up checking every critic before reading its answer:

  1. The run ended as delivered, not on a turn limit or a cancel.
  2. The model it resolved to is the one it was assigned.
  3. It made at least one tool call when its brief lists files to read.
  4. Its reply isn't trivially short and doesn't say the material was missing.

Anything that fails gets rerun before synthesis. If it still fails, the synthesis says how many critics were valid, for example "7 of 8 valid".

For the debate protocol, a cheap version of this would fit next to Option B. A critic's objection counts only if it quotes text it read during the debate. And the Conductor checks a critic's run before ruling on its objections, the same way the raising critic checks evidence before fixed_verified. That could also give §2.6 one more number to count: how many objections came from critics that never read the material.

@iamwhatever
iamwhatever force-pushed the docs/rfc-worker-debate branch from 589eb2d to 430d853 Compare November 9, 2026 07:27
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 9, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 9, 2026
@iamwhatever
iamwhatever force-pushed the docs/rfc-worker-debate branch from 430d853 to 2552279 Compare November 9, 2026 07:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 9, 2026
@iamwhatever
iamwhatever force-pushed the docs/rfc-worker-debate branch from 2552279 to 3966d8c Compare November 9, 2026 09:22
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 9, 2026
@iamwhatever
iamwhatever force-pushed the docs/rfc-worker-debate branch from 3966d8c to cd0c123 Compare November 9, 2026 11:28
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 9, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Thanks for the rebase. cd0c123 merges cleanly with main and all 97 checks are green. This is a comment with a design brief, not an approval, as with every RFC.

Design brief:

  • What it does: it records the worker-debate protocol with file:line (§1), and adopts Option B, "claim checks", as amended by the 2026-11-09 critic-roles debate (§2.4, §2.7). A critic may ask a Q: question about one quoted claim. The proposer answers with evidence, not added text. The mechanism table gains a proof column. Every claim gets a named checker who did not write it and asks "what if it is wrong on its path". Critic roles stay set at dispatch, plus a biggest-risk entry in Round 0 and one Conductor pass before round 1. It also records why the panel moved to one model by default (§3), and fixes two stale lines in rfc-worker-messaging.md (max_rounds and open question 4). The front matter says accepted on the repository owner's decision of 2026-11-09, with the skill change in docs(goal-conductor): worker debate Round 0 biggest-risk rows and owned claim checks #33783.
  • Pros: it targets the three failures the 2026-11-02 debate record shows: objections that only add text, a proposer's code claim checked by two of three critics, and an assumption (the fake backend) that passed and then failed in use. It reuses the existing ids, rulings and relays, adding 0 rounds, 0 relays and 0 work items for about 34 lines of skill text. Phase 2 measures the result against the 2026-11-02 baseline.
  • Cons and limits: the case rests on one debate's record, not on a side-by-side comparison with a grill session, as §2.2 says. A Round 0 major that is not a critic's biggest risk gets no row (Q7, left open). The check is only as sharp as the checker. One model keeps shared blind spots on pure judgment calls; §3.3 points to a one-shot llm-council pass for those.
  • Long-term debt: low. It is prose a Conductor reads, with no tool, schema or stored record change, and Phase 2 has a revert exit.

Doc only; it changes no code.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 9, 2026 13:18
KiroCrewCaptain
KiroCrewCaptain previously approved these changes Nov 9, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. This is a docs-only RFC with no code, skill text, config or test changes, and PR Readiness is green on this head. A maintainer signed off the design.

The review checked the RFC's file and line references into worker-debate.md, goal-conductor/SKILL.md, the grill skill, crew-mode.md and pod/README.md at the audited commit, and they hold. Every relative link resolves at this head. The two edits to rfc-worker-messaging.md now match the skill text: hard_rounds replaces max_rounds, and open question 4 is marked answered by #26932. The RFC files are byte-identical to the commit the tests ran on. Done well: the RFC says plainly where its evidence stops (no saved grill transcript, no saved mid-turn model failure) and does not overclaim.

Optional:

  • rfc-worker-debate.md:105 gives the anchor budget as "half the goal's item cap", but #33601 removed that cap. A §1.3 drift row, plus a concrete default in #33783 or a note that it is deferred, would keep §1.2 accurate.
  • rfc-worker-debate.md:256 says "Three small rules", but Option B lists four (:259, :266, :270, :281).
  • Q5 (:532-535) puts a wording change into the Phase 1 PR, but the Migration plan limits Phase 1 to the §2.7 table. Marking Q5 as deferred, or adding it to the Phase 1 scope, would remove the ambiguity.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 9, 2026
@iamwhatever
iamwhatever force-pushed the docs/rfc-worker-debate branch from cd0c123 to 5f0fed3 Compare November 9, 2026 17:50
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 9, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

rfc change, merge first and code will follow

@iamwhatever
iamwhatever disabled auto-merge November 9, 2026 18:22
@iamwhatever
iamwhatever merged commit 76855bb into main Nov 9, 2026
93 of 98 checks passed
@iamwhatever
iamwhatever deleted the docs/rfc-worker-debate branch November 9, 2026 18:22
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs(rfc): record the worker-debate design and propose sharper critic questions

4 participants