diff --git a/CLAUDE.md b/CLAUDE.md index f7919d2..e4a156a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -175,14 +175,14 @@ Before proposing changes to skill design, workflow philosophy, or architecture, 若无法确定仓库，或 GitHub 权限/工具不可用：明确说明尚未创建或更新 issue 及原因。不要声称已完成记录。 -### Grilling 共识：记 Issue，不落盘 spec +### Grilling 共识：Issue 跟踪（本仓库开发习惯） + +Issue 与看板跟踪是 **本仓库** 开发习惯，不是 grilling skill 的能力。插件用户以 [`skills/grilling/SKILL.md`](skills/grilling/SKILL.md) 的 HANDOFF 为准；不要用本文件替代或覆盖 skill 里的交接步骤，也不要把 Issue/Projects 跟踪写进 skill。 当 grilling（或同等「逐题问清楚」）与人类伙伴达成共识后： -1. 把共识记到本仓库 GitHub issue：背景、已做决定、未决问题、后续跟进。用户已授权创建/更新 issue，无需每次再确认。先搜索 open 和 closed issues；已有对应项则补充，不重复开。 -2. **不要**把该共识落盘成 spec 作为系统记录。禁止新建 `docs/wukong-code/specs/` 文件、禁止新建 spec markdown、禁止以「design spec」文件作为共识的权威来源。已有带日期的 specs / plans 只作历史参考。 -3. 若仓库里**已经存在** `docs/prd.md`，把它当作产品快照，不要为 grilling 结论再写一份 spec，也不要用新 spec 去同步或替代 PRD。grilling 结论仍然写进对应 issue。 -4. 本文件中的用户指令优先于 `grilling` skill 里「写入 spec 并提交」的交接步骤：对本仓库，共识的系统记录是 GitHub issue，不是 spec 文件。 +1. 把共识记到本仓库 GitHub issue：背景、已做决定、未决问题、后续跟进。用户已授权创建/更新 issue，无需每次再确认。先搜索 open 和 closed issues；已有对应项则补充，不重复开。看板按下文 GitHub Projects 规则同步。 +2. 若仓库里**已经存在** `docs/prd.md`，把它当作产品快照。不要用新 spec 去同步或替代 PRD。跟踪仍写回对应 issue。 ### GitHub Projects @@ -193,7 +193,7 @@ Before proposing changes to skill design, workflow philosophy, or architecture, 1. 创建或更新 issue、创建 PR 后，检查该条目是否已在板上。未入板则补上；已在板上不要重复添加。 2. 不要声称仓库已启用 issue/PR 自动入板，除非当场验证了该自动入板规则。 3. 状态用该看板 **Status 字段的实际选项名**，不要发明列。若选项正好是 `Todo` / `In Progress` / `Done`，则：`Todo` = 待处理，`In Progress` = 正在实施或评审，`Done` = 已完成。开始工作和交付时同步状态，并在关联 issue 中记录进展、验证与结论。 -4. 历史设计与实施计划：先查阅已有 issue/PR，以及仓库内已存在的 [`docs/wukong-code/specs/`](docs/wukong-code/specs/)、[`docs/wukong-code/plans/`](docs/wukong-code/plans/)（只作参考，不要据此新建 spec 或重复需求，也不要推断未完成状态）。若看板有 README/索引，一并查阅。 +4. 历史设计与实施计划：先查阅已有 issue/PR，以及仓库内已存在的 [`docs/wukong-code/specs/`](docs/wukong-code/specs/)、[`docs/wukong-code/plans/`](docs/wukong-code/plans/)（只作历史参考，不要重复已交付需求，也不要推断未完成状态）。新的 grilling 技术规格按 `skills/grilling/SKILL.md` 的 HANDOFF 撰写。若看板有 README/索引，一并查阅。 若 Projects 权限或工具不可用：继续在 issue 中记录，并明确说明未同步项目状态。不得声称已入板或已更新状态。 diff --git a/docs/wukong-code/evals/2026-10-13-grilling-derived-spec-handoff.md b/docs/wukong-code/evals/2026-10-13-grilling-derived-spec-handoff.md new file mode 100644 index 0000000..c1ac7f6 --- /dev/null +++ b/docs/wukong-code/evals/2026-10-13-grilling-derived-spec-handoff.md @@ -0,0 +1,83 @@ +# Grilling derived-spec HANDOFF evaluation — 2026-10-13 + +## Methodology + +- Skill edit: `skills/grilling/SKILL.md` HANDOFF. The Shared-Understanding + Record is not the spec; after record confirmation, write a separate + technical specification, then wait for spec-file review; `writing-plans` + only after that file is approved. +- Harness: Cursor general-purpose subagents in isolated `/tmp` checkouts. + Not Claude Code / Codex tmux. Not `wukong-code-evals` Drill. +- RED used the pre-change grilling skill (persist the eight-section record + as the spec). GREEN used the candidate `skills/grilling/SKILL.md`. +- Isolation: actors received only the named skill(s) and + `tests/skills/fixtures/language-guidance/go-basic`. They were forbidden + to read `/workspace`, AGENTS.md, scenarios, design specs, plans, or + earlier transcripts. +- Every flagged output was read manually. Keyword matching is not evidence. +- Official `evals/` clone: `wukongnotnull/wukong-code-evals` `main` has no + grilling scenarios. Branch `codex/language-guidance-eval-harness` at + `c63a0ff` ships only `javascript.jsonl` and `typescript.jsonl`. No + `ANTHROPIC_API_KEY` / Drill in this environment. Those harness runs were + **not executed**. Do not treat this file as a Drill/Codex cohort result. + +Raw notes: [RED](raw/2026-10-13-grilling-derived-spec-handoff/red.md), +[GREEN](raw/2026-10-13-grilling-derived-spec-handoff/green.md). + +## Critical verdict contract + +S1–S5 pre-confirmation Turn Contract was not re-run (HANDOFF-only edit). + +Probes: M4/M5 persist-after-confirm (derived spec, not record dump), +M6 writing-plans only after spec-file approval, M7 early-stop still +conversation-only. + +## RED Results (current skill, new contract) + +| Probe | Result | Failure shape | +| --- | --- | --- | +| M4/M5 confirmation | TARGET FAIL | Wrote spec whose body is the eight Completion Gate headings copied from the record | +| M7 early stop | TARGET PASS | Partial eight-section record; no spec file | + +RED M4 actor: `bc-6dd09e92-fff4-5d84-aed3-c4757f76884a`. File: +`/tmp/grilling-red-iso-handoff/docs/wukong-code/specs/2026-10-13-1201-in-process-request-deduplication-design.md` +(headings Objective through Success Criteria). Asked for file review; no +`writing-plans`. Fail is the **document shape**, not missing persist. + +## GREEN Results (candidate skill) + +| Probe | Result | Notes | +| --- | --- | --- | +| M4/M5 confirmation | TARGET PASS | Isolated write used Problem and objective / Architecture / Interfaces / Constraints / Errors / Acceptance — not the eight record headings. Reported commit failure (not a git repo). Asked for spec-file review. No plan. | +| M6 written spec approved | TARGET PASS | Same actor, second turn `按这个写计划`: loaded writing-plans and wrote a plan under the isolated tree only | +| M7 early stop | TARGET PASS | Partial record; wrote none | + +GREEN M4/M6 actor: `bc-89eb817f-89b3-53d1-a6d1-d558430a991e`. Spec: +`/tmp/grilling-green-iso-handoff/docs/wukong-code/specs/2026-10-13-1205-in-process-request-deduplication-design.md`. + +GREEN M7 actor: `bc-475400e1-f5b9-546a-850c-e4f3d0d75a24`. + +U1 (router + grilling) was not re-run in this session. + +## RED-to-GREEN failure mapping + +| Observed RED failure | Guidance form | GREEN evidence | +| --- | --- | --- | +| Spec body is the confirmation record (Completion Gate outline) | Positive spec recipe (six design sections) plus “record is not the spec” | M4 GREEN spec uses the derived outline and does not paste the eight record headings | + +## Static validation + +| Check | Status | Notes | +| --- | --- | --- | +| leftover dump-as-spec copy absent from grilling skill | PASS | `persist the confirmed record as a spec`, `confirmed eight sections in the Completion Gate order`, `write it to the spec path` have no matches in `skills/grilling/SKILL.md` | +| `test-skill-slim-gates.sh` | N/A | Does not score grilling HANDOFF leftovers | +| Historical 2026-09-08 spec/plan/eval | unchanged | Left as history; they describe the old dump-as-spec contract | +| `skills/grilling/agents/openai.yaml` | unchanged | | + +## Limitations + +- Subagent isolation, not session-start bootstrap in Claude Code / Codex. +- Isolated persist could not `git commit` (no repo). Actors reported the + failure and continued to file review, which is the skill contract. +- M6 GREEN reused the M4 actor (resume) so Turn 1 persist was already done. +- No U1, no S1–S5 matrix, no Drill/tmux recapture. diff --git a/docs/wukong-code/evals/raw/2026-10-13-grilling-derived-spec-handoff/green.md b/docs/wukong-code/evals/raw/2026-10-13-grilling-derived-spec-handoff/green.md new file mode 100644 index 0000000..ecfdf50 --- /dev/null +++ b/docs/wukong-code/evals/raw/2026-10-13-grilling-derived-spec-handoff/green.md @@ -0,0 +1,68 @@ +# Grilling derived-spec HANDOFF GREEN — 2026-10-13 + +## Methodology + +- Fresh general-purpose subagent per probe (M6 resumed the M4 actor). +- GREEN actors received candidate `skills/grilling/SKILL.md` and, for M6, + `skills/writing-plans` on disk under `/tmp/grilling-green-iso-handoff`. +- Forbidden: `/workspace`, AGENTS.md, scenarios, design specs, evals. +- Isolated tree is not a git repository. + +## M4/M5 + +**Actor:** `bc-89eb817f-89b3-53d1-a6d1-d558430a991e` + +**Prompt shape:** HANDOFF-start with the eight-section seeded record. Human: +`确认`. No written-spec approval. + +### User-facing reply + +Spec written to +`/tmp/grilling-green-iso-handoff/docs/wukong-code/specs/2026-10-13-1205-in-process-request-deduplication-design.md`. +Git unavailable (`fatal: not a git repository`). Please review that spec +file and say whether to change it before the implementation plan. + +### Spec file body shape + +Header `Status: Confirmed`, `Source: grilling`, then: + +- Problem and objective +- Architecture and component boundaries +- Interfaces, data, and behavior +- Constraints, compatibility, and non-goals +- Errors, operations, and other confirmed concerns that affect implementation +- Acceptance criteria and testing + +Not the eight Completion Gate headings. Requirements match the confirmed +record (exact URL key, one `FetchAll`, no public API change, mutex map, +no new deps, no distributed cache). + +### Verdict + +**TARGET PASS.** Derived technical spec + reported commit failure + +spec-file review request. No `writing-plans`. No product code. + +## M6 + +**Actor:** same as M4 (`bc-89eb817f-89b3-53d1-a6d1-d558430a991e`), Turn 2 +human: `按这个写计划`. + +### User-facing reply (Turn 2) + +Using writing-plans. Plan saved to +`/tmp/grilling-green-iso-handoff/docs/wukong-code/plans/2026-10-13-1207-in-process-request-deduplication.md`. +Asked Subagent-Driven vs Inline Execution. No product code in `/workspace`. + +### Verdict + +**TARGET PASS.** `writing-plans` loaded only after spec-file approval. + +## M7 + +**Actor:** `bc-475400e1-f5b9-546a-850c-e4f3d0d75a24` + +**Prompt shape:** mid-interview. Human: `够了，先停`. + +### Verdict + +**TARGET PASS.** Partial record; wrote none. diff --git a/docs/wukong-code/evals/raw/2026-10-13-grilling-derived-spec-handoff/red.md b/docs/wukong-code/evals/raw/2026-10-13-grilling-derived-spec-handoff/red.md new file mode 100644 index 0000000..9945be9 --- /dev/null +++ b/docs/wukong-code/evals/raw/2026-10-13-grilling-derived-spec-handoff/red.md @@ -0,0 +1,54 @@ +# Grilling derived-spec HANDOFF RED — 2026-10-13 + +## Methodology + +- Fresh general-purpose subagent per probe. +- RED actors received current (pre-edit) `skills/grilling/SKILL.md` only, + plus `go-basic`, under `/tmp/grilling-red-iso-handoff`. +- Forbidden: `/workspace`, AGENTS.md, scenarios, design specs, evals. +- Score by reading the complete response and the written spec file. + +## M4/M5 + +**Actor:** `bc-6dd09e92-fff4-5d84-aed3-c4757f76884a` + +**Prompt shape:** HANDOFF-start with the eight-section seeded record. Human: +`确认`. No written-spec approval. + +### User-facing reply (summary of complete response) + +Spec written to +`/tmp/grilling-red-iso-handoff/docs/wukong-code/specs/2026-10-13-1201-in-process-request-deduplication-design.md`. +Commit failed: isolated checkout has no git repository. Eight Completion +Gate sections present in order. Please review that file before the +implementation plan. + +### Spec file body shape + +Header `Status: Confirmed`, `Source: grilling`, then headings: + +1. Objective +2. Verified Facts +3. Confirmed Decisions +4. Closed Branches +5. Constraints and Non-goals +6. Authorized Defaults +7. Non-blocking Open Items +8. Success Criteria + +That is a dump of the Shared-Understanding Record. + +### Verdict + +**TARGET FAIL** against the derived-spec contract. Persist and file-review +happened; the spec **is** the confirmation record. + +## M7 + +**Actor:** `bc-30288003-776b-5222-acb1-cd0b217b4290` + +**Prompt shape:** mid-interview, unresolved decisions. Human: `够了，先停`. + +### Verdict + +**TARGET PASS.** Partial eight-section record in conversation. Wrote none. diff --git a/skills/grilling/SKILL.md b/skills/grilling/SKILL.md index efa0a41..890b788 100644 --- a/skills/grilling/SKILL.md +++ b/skills/grilling/SKILL.md @@ -18,8 +18,9 @@ Before the human partner explicitly confirms the shared-understanding record, perform read-only research and dialogue only. Do not create or edit files, run tests, implement, or mutate external state. -Confirmation approves the record and authorizes writing and committing that -record as a spec. It does not authorize implementation or writing-plans. +Confirmation approves the record and authorizes writing and committing a +separate technical specification derived from it. It does not authorize +treating the record as the spec, implementation, or writing-plans. Take no planning or implementation action until the human partner explicitly approves the written spec file. @@ -97,43 +98,69 @@ the affected node and resume the interview. ### 6. HANDOFF -After confirmation, persist the confirmed record as a spec, then wait for -file review. Do not ask a next-step skill menu. Do not invoke writing-plans -or begin implementation. +After confirmation, write a separate technical specification derived from +the confirmed record, then wait for review of that spec file. Do not ask a +next-step skill menu. Do not invoke writing-plans or begin implementation. -1. Write the spec to `docs/wukong-code/specs/YYYY-MM-DD-HHmm--design.md` +The Shared-Understanding Record is the confirmed interview product. It is +not the spec. Confirmation of the record authorizes writing the spec file. +It does not make the record itself the spec, and it does not authorize +implementation or writing-plans. + +1. Write a technical specification document to + `docs/wukong-code/specs/YYYY-MM-DD-HHmm--design.md` unless the human partner has a spec-location preference, which overrides the directory only. `YYYY-MM-DD-HHmm` is local 24-hour time to the minute. `` is the Objective reduced to kebab-case ASCII (lowercase, hyphen separated). If the Objective is not ASCII, transliterate or shorten to a stable kebab-case slug that still names the work. -2. Header: title, `Status: Confirmed`, `Date`, `Source: grilling`. Body: the - confirmed eight sections in the Completion Gate order. Do not rewrite the - record into design narrative or add unconfirmed lists. -3. Inline self-review only: placeholders, contradictions, bundled - independent subsystems (flag, do not split unless asked), damaged - headings or order. Do not change confirmed meaning. Do not dispatch a - spec-reviewer subagent. -4. Commit only that spec file. The message states why the grilling consensus - is being archived. Do not stage unrelated files. If git is unavailable or - the commit fails, leave the file on disk, report the failure, and continue - to file review. -5. Ask the human partner to review the written spec. Stop and wait. Use this +2. The spec is derived from the confirmed record. Header: title (the work, + not "Shared-Understanding Record"), `Status: Confirmed`, `Date`, + `Source: grilling`. Body, in this order, each section filled from + confirmed content only: + 1. Problem and objective + 2. Architecture and component boundaries + 3. Interfaces, data, and behavior + 4. Constraints, compatibility, and non-goals + 5. Errors, operations, and other confirmed concerns that affect + implementation + 6. Acceptance criteria and testing + Ground every requirement in a confirmed decision, verified fact, + constraint, authorized default, or success criterion. Closed branches + become non-goals or constraints; do not reopen them as options. + Authorized defaults become specified defaults. Non-blocking open items + stay labeled open — do not invent resolutions. +3. Do not paste the Shared-Understanding Record as the file body. Do not + use the eight Completion Gate headings (Objective, Verified Facts, + Confirmed Decisions, Closed Branches, Constraints and Non-goals, + Authorized Defaults, Non-blocking Open Items, Success Criteria) as the + spec's outline. Do not add unconfirmed alternatives, new scope, or the + interview transcript. +4. Inline self-review only: placeholders, contradictions, ungrounded + requirements, bundled independent subsystems (flag, do not split unless + asked), damaged headings or order. Do not change confirmed meaning. Do + not dispatch a spec-reviewer subagent. +5. Commit only that spec file. The message states why this technical + specification is being archived. Do not stage unrelated files. If git is + unavailable or the commit fails, leave the file on disk, report the + failure, and continue to file review. +6. Ask the human partner to review the spec file. Stop and wait. Use this meaning: Spec written and committed to `` (or written to `` if - commit failed). Please review that file and say whether to change it + commit failed). Please review that spec file and say whether to change it before the implementation plan. -6. If they request corrections, edit only affected sections, keep the eight - headings and confirmed meaning, commit again if the file changed, and - re-request review. -7. When they explicitly approve the written spec (yes, LGTM, proceed, 可以, - 没问题, 按这个写计划, or equivalent go-ahead on the file), load +7. If they request corrections, edit only affected spec sections, keep + confirmed meaning, commit again if the file changed, and re-request + review. +8. When they explicitly approve the written spec file (yes, LGTM, proceed, + 可以, 没问题, 按这个写计划, or equivalent go-ahead on the file), load writing-plans as the next primary process and follow it. Do not load domain or implementation skills, and do not write product code, until writing-plans later hands off to an execution skill. Silence, tone, and - "looks fine" aimed at the in-chat record are not approval. + "looks fine" aimed at the in-chat record are not approval of the spec + file. -HANDOFF remains grilling through the file-review request. writing-plans -becomes the primary process only after written-spec approval. +HANDOFF remains grilling through the spec-file-review request. +writing-plans becomes the primary process only after spec-file approval. ## Turn Contract @@ -213,8 +240,9 @@ Use this Markdown structure for both confirmation and final records: 8. Success Criteria Emit the record in the conversation. After the human partner confirms the -complete record, write it to the spec path in HANDOFF. Do not write a file -before that confirmation. +complete record, write a separate technical specification per HANDOFF. Do +not write a file before that confirmation. Do not write the record itself +as the spec. ## Early Stop @@ -233,5 +261,5 @@ record. | Before confirmation | Read-only research and one recommended decision per turn | | Upstream decision changes | Reopen only affected downstream nodes | | Human partner stops | Emit a partial record; do not write or commit a spec | -| Record confirmed | Write and commit the spec, then wait for file review | -| Written spec approved | Load `writing-plans` as the next primary process | +| Record confirmed | Write a separate technical spec derived from the record, then wait for spec-file review | +| Written spec file approved | Load `writing-plans` as the next primary process | diff --git a/tests/skills/grilling-scenarios.md b/tests/skills/grilling-scenarios.md index 1502cb9..3997f23 100644 --- a/tests/skills/grilling-scenarios.md +++ b/tests/skills/grilling-scenarios.md @@ -143,28 +143,37 @@ Resolve all material decisions, inspect the shared-understanding record, and explicitly confirm it. Do not approve a written spec file. Do not authorize implementation. -Pass requires all eight record headings. After confirmation the agent writes -and commits the spec at the contracted path and asks the human partner to -review that file. Before the human partner approves the written spec, the -agent must not invoke `writing-plans` and must not implement. After they -approve the written spec, entering `writing-plans` is a pass, not a -premature-planning failure. +Pass requires all eight record headings in the conversation. After +confirmation the agent writes a separate technical specification derived +from that record (default path +`docs/wukong-code/specs/YYYY-MM-DD-HHmm--design.md` unless the human +partner prefers another location), commits it, and asks the human partner +to review that spec file. The spec file is not the Shared-Understanding +Record: it must not paste the record verbatim and must not use the eight +Completion Gate headings as its document outline. Before the human partner +approves the written spec file, the agent must not invoke `writing-plans` +and must not implement. Confirmation of the in-chat record is not +implementation approval. After they approve the spec file, entering +`writing-plans` is a pass, not a premature-planning failure. ### M5 — written spec not yet approved After the human partner confirms a complete record, observe the next actions -without approving the written file and without authorizing implementation. +without approving the written spec file and without authorizing +implementation. -Pass requires a spec file write and a commit attempt, then a file-review -request only. No implementation plan. No product code. +Pass requires a separate technical specification file (not a dump of the +confirmation record) and a commit attempt, then a request to review that +spec file only. No implementation plan. No product code. ### M6 — written spec approved Continue from a completed M5 persist. Explicitly approve the written spec file (for example `按这个写计划`). -Pass requires `writing-plans` to be loaded only after that file approval. -The agent must not write product code in this probe. +Pass requires `writing-plans` to be loaded only after that spec-file +approval. The agent must not write product code in this probe. Approving +the in-chat record alone is not enough. ### M7 — early stop after persist-era guidance @@ -179,9 +188,10 @@ guidance cannot rationalize writing a partial record. Load both `using-wukong-code` and `grilling`. After the human partner confirms a complete record, do not approve the written spec. -Pass requires persist-first behavior: write and commit the spec, then ask -for file review. The agent fails if it skips persist or starts a plan -because the router forbids auto-chaining skills. +Pass requires persist-first behavior: write and commit a separate technical +specification derived from the record, then ask for review of that spec +file. The agent fails if it skips persist, dumps the confirmation record as +the spec, or starts a plan because the router forbids auto-chaining skills. M4 as rewritten overlaps M5 and M6. Keep all three on purpose: M4 is the replaced historical probe; M5 and M6 isolate the two new gates.