From 50364058580f0f4cf71a166c82678111bdf9e64c Mon Sep 17 00:00:00 2001 From: Param Harrison Date: Sun, 1 Nov 2026 16:58:01 +0200 Subject: [PATCH 01/12] Assignments become markdown briefs with a walkthrough, and cover issues 152 and 153 Co-Authored-By: Claude Sonnet 5.5 --- .../ISSUE_TEMPLATE/assignment-approvers.md | 17 --- .github/ISSUE_TEMPLATE/assignment-canary.md | 17 --- .../assignment-clock-activity.md | 17 --- .../assignment-fuzz-tool-input.md | 17 --- .../ISSUE_TEMPLATE/assignment-judge-hook.md | 17 --- .../assignment-label-machine.md | 17 --- .github/ISSUE_TEMPLATE/assignment-my-mod.md | 19 --- .../ISSUE_TEMPLATE/assignment-own-workflow.md | 19 --- .../ISSUE_TEMPLATE/assignment-review-lens.md | 17 --- .../ISSUE_TEMPLATE/assignment-stage-names.md | 16 --- .../ISSUE_TEMPLATE/assignment-template-db.md | 18 --- .../assignment-worksheet-video.md | 17 --- assignments/README.md | 24 ++++ assignments/approvers.md | 54 ++++++++ assignments/canary.md | 54 ++++++++ assignments/clock-activity.md | 54 ++++++++ assignments/fuzz-tool-input.md | 54 ++++++++ assignments/judge-hook.md | 54 ++++++++ assignments/label-machine.md | 54 ++++++++ assignments/my-mod.md | 56 ++++++++ assignments/own-workflow.md | 56 ++++++++ assignments/recording-from-run-log.md | 57 ++++++++ assignments/review-lens.md | 54 ++++++++ assignments/stage-names.md | 53 ++++++++ assignments/template-db.md | 55 ++++++++ assignments/three-sigma-incident.md | 57 ++++++++ assignments/worksheet-video.md | 54 ++++++++ docs/factory/f10-assignments.md | 44 ++++-- levels.json | 1 + .../recording-from-run-log.test.ts | 34 +++++ .../assignments/three-sigma-incident.test.ts | 39 ++++++ tests/structure/assignments.test.ts | 128 +++++++++++++----- 32 files changed, 989 insertions(+), 255 deletions(-) delete mode 100644 .github/ISSUE_TEMPLATE/assignment-approvers.md delete mode 100644 .github/ISSUE_TEMPLATE/assignment-canary.md delete mode 100644 .github/ISSUE_TEMPLATE/assignment-clock-activity.md delete mode 100644 .github/ISSUE_TEMPLATE/assignment-fuzz-tool-input.md delete mode 100644 .github/ISSUE_TEMPLATE/assignment-judge-hook.md delete mode 100644 .github/ISSUE_TEMPLATE/assignment-label-machine.md delete mode 100644 .github/ISSUE_TEMPLATE/assignment-my-mod.md delete mode 100644 .github/ISSUE_TEMPLATE/assignment-own-workflow.md delete mode 100644 .github/ISSUE_TEMPLATE/assignment-review-lens.md delete mode 100644 .github/ISSUE_TEMPLATE/assignment-stage-names.md delete mode 100644 .github/ISSUE_TEMPLATE/assignment-template-db.md delete mode 100644 .github/ISSUE_TEMPLATE/assignment-worksheet-video.md create mode 100644 assignments/README.md create mode 100644 assignments/approvers.md create mode 100644 assignments/canary.md create mode 100644 assignments/clock-activity.md create mode 100644 assignments/fuzz-tool-input.md create mode 100644 assignments/judge-hook.md create mode 100644 assignments/label-machine.md create mode 100644 assignments/my-mod.md create mode 100644 assignments/own-workflow.md create mode 100644 assignments/recording-from-run-log.md create mode 100644 assignments/review-lens.md create mode 100644 assignments/stage-names.md create mode 100644 assignments/template-db.md create mode 100644 assignments/three-sigma-incident.md create mode 100644 assignments/worksheet-video.md create mode 100644 tests/assignments/recording-from-run-log.test.ts create mode 100644 tests/assignments/three-sigma-incident.test.ts diff --git a/.github/ISSUE_TEMPLATE/assignment-approvers.md b/.github/ISSUE_TEMPLATE/assignment-approvers.md deleted file mode 100644 index cae8c07..0000000 --- a/.github/ISSUE_TEMPLATE/assignment-approvers.md +++ /dev/null @@ -1,17 +0,0 @@ ---- - -name: "Assignment: name the approvers for a high-risk gate" -about: "Workshop assignment. Run it with the feature-atdd workflow." -title: "Assignment: name the approvers for a high-risk gate" -labels: assignment ---- - -Read the approver logins from `.factory/config.yaml` (an `approvers` list in `portableSchema`) and pass them to `realHost` from `runIssue`. - -Acceptance: -- A high-risk gate opens with two listed people, a medium one with one. -- One person commenting twice counts once. -- The logged-in user is no longer assumed. - -Workflow: feature-atdd -Red test: `tests/assignments/approvers.test.ts`. Run it with `ASSIGNMENT=1 npx vitest run tests/assignments/approvers.test.ts`; it must fail for the reason above before any work. diff --git a/.github/ISSUE_TEMPLATE/assignment-canary.md b/.github/ISSUE_TEMPLATE/assignment-canary.md deleted file mode 100644 index 1d3c7ea..0000000 --- a/.github/ISSUE_TEMPLATE/assignment-canary.md +++ /dev/null @@ -1,17 +0,0 @@ ---- - -name: "Assignment: canary prompt with rollback" -about: "Workshop assignment. Run it with the llm-eval workflow." -title: "Assignment: canary prompt with rollback" -labels: assignment ---- - -Add `api/src/agent/canary.ts` for the prompt in `api/src/agent/prompt.ts`. - -Acceptance: -- `routePrompt(requestId, share)` returns `"candidate"` or `"current"`, the same way for the same id, and near `share` over many ids. -- `decideCanary({ current, candidate, margin })` returns `"rollback"` when the candidate's pass rate falls below the current one by more than `margin`, otherwise `"keep"`. -- After a rollback the share is 0 and the event is logged. - -Workflow: llm-eval -Red test: `tests/assignments/canary.test.ts`. Run it with `ASSIGNMENT=1 npx vitest run tests/assignments/canary.test.ts`; it must fail for the reason above before any work. diff --git a/.github/ISSUE_TEMPLATE/assignment-clock-activity.md b/.github/ISSUE_TEMPLATE/assignment-clock-activity.md deleted file mode 100644 index 22c7e43..0000000 --- a/.github/ISSUE_TEMPLATE/assignment-clock-activity.md +++ /dev/null @@ -1,17 +0,0 @@ ---- - -name: "Assignment: add the clock activity kind" -about: "Workshop assignment. Run it with the feature-bdd workflow." -title: "Assignment: add the clock activity kind" -labels: assignment ---- - -Add the `clock` activity kind: the model supplies a time (`hour`, `minute`), the code draws the face. - -Acceptance: -- `clock` is in `ACTIVITY_KINDS` and `activitySchema` accepts it. -- `renderActivity` draws an SVG face. -- It is listed in `kinds` for the bands that allow it in `shared/src/bands.ts`, and `checkBand` rejects it in the others. - -Workflow: feature-bdd -Red test: `tests/assignments/clock-activity.test.ts`. Run it with `ASSIGNMENT=1 npx vitest run tests/assignments/clock-activity.test.ts`; it must fail for the reason above before any work. diff --git a/.github/ISSUE_TEMPLATE/assignment-fuzz-tool-input.md b/.github/ISSUE_TEMPLATE/assignment-fuzz-tool-input.md deleted file mode 100644 index a367db6..0000000 --- a/.github/ISSUE_TEMPLATE/assignment-fuzz-tool-input.md +++ /dev/null @@ -1,17 +0,0 @@ ---- - -name: "Assignment: fuzz the worksheet tool input" -about: "Workshop assignment. Run it with the security workflow." -title: "Assignment: fuzz the worksheet tool input" -labels: assignment ---- - -Property-test the worksheet tool input. Write the generator `arbWorksheet` in `shared/src/arb-worksheet.ts`: worksheet-shaped objects with hostile text, extreme sizes and every activity kind. - -Acceptance: -- Every worksheet the schema accepts renders, and markup in a text field never reaches the page raw. -- `checkBand` reports problems instead of throwing. -- Plant a mutant (an escaping step removed) and show the property fails on it. - -Workflow: security -Red test: `tests/assignments/fuzz-tool-input.test.ts`. Run it with `ASSIGNMENT=1 npx vitest run tests/assignments/fuzz-tool-input.test.ts`; it must fail for the reason above before any work. diff --git a/.github/ISSUE_TEMPLATE/assignment-judge-hook.md b/.github/ISSUE_TEMPLATE/assignment-judge-hook.md deleted file mode 100644 index 4ce4dfd..0000000 --- a/.github/ISSUE_TEMPLATE/assignment-judge-hook.md +++ /dev/null @@ -1,17 +0,0 @@ ---- - -name: "Assignment: an opt-in LLM-judge hook" -about: "Workshop assignment. Run it with the feature-atdd workflow." -title: "Assignment: an opt-in LLM-judge hook" -labels: assignment ---- - -Add `.claude/hooks/pretooluse-judge.mjs` exporting `shouldJudge(env)`, `decide(verdict)` and `diffToSend(diff)`. - -Acceptance: -- Off by default; on only when `JUDGE=1`. -- A `blocking` verdict blocks the commit, a `clean` one passes. -- A diff the secrets hook would block is never sent. - -Workflow: feature-atdd -Red test: `tests/assignments/judge-hook.test.ts`. Run it with `ASSIGNMENT=1 npx vitest run tests/assignments/judge-hook.test.ts`; it must fail for the reason above before any work. diff --git a/.github/ISSUE_TEMPLATE/assignment-label-machine.md b/.github/ISSUE_TEMPLATE/assignment-label-machine.md deleted file mode 100644 index 603e7c0..0000000 --- a/.github/ISSUE_TEMPLATE/assignment-label-machine.md +++ /dev/null @@ -1,17 +0,0 @@ ---- - -name: "Assignment: specify the label state machine" -about: "Workshop assignment. Run it with the feature-atdd workflow." -title: "Assignment: specify the label state machine" -labels: assignment ---- - -Add `factory/label-machine.ts` with `applyEvent(labels, event)` and `canBuild(labels)`, for the labels in `.factory/config.json`. Events: `plan-written`, `human-approved`, `build-requested`, `stuck`, `retry`. - -Invariants: -- A build never starts without `plan:approved`. -- A new plan always clears `plan:approved`. -- `needs-human` is never cleared by the factory, only by `human-approved`. - -Workflow: feature-atdd -Red test: `tests/assignments/label-machine.test.ts`. Run it with `ASSIGNMENT=1 npx vitest run tests/assignments/label-machine.test.ts`; it must fail for the reason above before any work. diff --git a/.github/ISSUE_TEMPLATE/assignment-my-mod.md b/.github/ISSUE_TEMPLATE/assignment-my-mod.md deleted file mode 100644 index 2402d03..0000000 --- a/.github/ISSUE_TEMPLATE/assignment-my-mod.md +++ /dev/null @@ -1,19 +0,0 @@ ---- - -name: "Assignment: a mod of your own" -about: "Workshop assignment. Run it with the feature-atdd workflow." -title: "Assignment: a mod of your own" -labels: assignment ---- - -Add a mod under `plugins/` that shows above the prompt. - -Acceptance: -- `.claude-plugin/plugin.json`, `hooks/hooks.json` and `hooks/register.tsx` exist. -- `claude plugin validate` passes. -- `hooks/register.test.tsx` feeds it a fixed state and checks what it draws. - -Run the test with `MOD=`. - -Workflow: feature-atdd -Red test: `tests/assignments/my-mod.test.ts`. Run it with `ASSIGNMENT=1 npx vitest run tests/assignments/my-mod.test.ts`; it must fail for the reason above before any work. diff --git a/.github/ISSUE_TEMPLATE/assignment-own-workflow.md b/.github/ISSUE_TEMPLATE/assignment-own-workflow.md deleted file mode 100644 index 982bf2a..0000000 --- a/.github/ISSUE_TEMPLATE/assignment-own-workflow.md +++ /dev/null @@ -1,19 +0,0 @@ ---- - -name: "Assignment: a workflow from your own company" -about: "Workshop assignment. Run it with the the workflow you write workflow." -title: "Assignment: a workflow from your own company" -labels: assignment ---- - -Add `.factory/workflows/.yaml` for . - -Acceptance: -- The first line is `# Method:`, and the file names its source. -- It uses only checks from `CHECK_NAMES` and loads. -- At least one step carries a check, with a failing input shown first. - -Run the test with `WORKFLOW=`. - -Workflow: the workflow you write -Red test: `tests/assignments/own-workflow.test.ts`. Run it with `ASSIGNMENT=1 npx vitest run tests/assignments/own-workflow.test.ts`; it must fail for the reason above before any work. diff --git a/.github/ISSUE_TEMPLATE/assignment-review-lens.md b/.github/ISSUE_TEMPLATE/assignment-review-lens.md deleted file mode 100644 index 8733865..0000000 --- a/.github/ISSUE_TEMPLATE/assignment-review-lens.md +++ /dev/null @@ -1,17 +0,0 @@ ---- - -name: "Assignment: add a performance review lens" -about: "Workshop assignment. Run it with the llm-eval workflow." -title: "Assignment: add a performance review lens" -labels: assignment ---- - -Add a `performance-reviewer` agent in `.claude/agents/performance-reviewer.md`. It reads the diff and replies with a JSON array of findings with file, line, severity, claim and evidence. - -Acceptance: -- Read-only tools (`Read, Grep, Glob`) and a `VERDICT:` line, like `security-reviewer`. -- A finding only one voter confirms is dropped. -- A workflow's `review: { lenses: [...] }` names it. - -Workflow: llm-eval -Red test: `tests/assignments/review-lens.test.ts`. Run it with `ASSIGNMENT=1 npx vitest run tests/assignments/review-lens.test.ts`; it must fail for the reason above before any work. diff --git a/.github/ISSUE_TEMPLATE/assignment-stage-names.md b/.github/ISSUE_TEMPLATE/assignment-stage-names.md deleted file mode 100644 index bd79e06..0000000 --- a/.github/ISSUE_TEMPLATE/assignment-stage-names.md +++ /dev/null @@ -1,16 +0,0 @@ ---- - -name: "Assignment: a model for every stage name" -about: "Workshop assignment. Run it with the llm-eval workflow." -title: "Assignment: a model for every stage name" -labels: assignment ---- - -Every stage name in every workflow resolves to a config stage, or the workflow test fails. - -Acceptance: -- Each name used in `.factory/workflows/` is a key of `stepStages` or one of `STAGE_NAMES`. -- Each step resolves to a model in `allowedModels`. - -Workflow: llm-eval -Red test: `tests/assignments/stage-names.test.ts`. Run it with `ASSIGNMENT=1 npx vitest run tests/assignments/stage-names.test.ts`; it must fail for the reason above before any work. diff --git a/.github/ISSUE_TEMPLATE/assignment-template-db.md b/.github/ISSUE_TEMPLATE/assignment-template-db.md deleted file mode 100644 index 91f0dfd..0000000 --- a/.github/ISSUE_TEMPLATE/assignment-template-db.md +++ /dev/null @@ -1,18 +0,0 @@ ---- - -name: "Assignment: move worksheet templates into a database" -about: "Workshop assignment. Run it with the spike, then feature-sdd workflow." -title: "Assignment: move worksheet templates into a database" -labels: assignment ---- - -Spike first: which store for worksheet templates, and what it rules out. File the spike alone and read its decision record before filing the build. - -Then add `createDbTemplateStore(file)` in `api/src/templates/db-template-store.ts` with the same `add`, `size` and `find` as `template-store.ts`. - -Acceptance: -- A second store opened on the same file finds a template the first saved. -- Adding the same template twice keeps one. - -Workflow: spike, then feature-sdd -Red test: `tests/assignments/template-db.test.ts`. Run it with `ASSIGNMENT=1 npx vitest run tests/assignments/template-db.test.ts`; it must fail for the reason above before any work. diff --git a/.github/ISSUE_TEMPLATE/assignment-worksheet-video.md b/.github/ISSUE_TEMPLATE/assignment-worksheet-video.md deleted file mode 100644 index a4bdbec..0000000 --- a/.github/ISSUE_TEMPLATE/assignment-worksheet-video.md +++ /dev/null @@ -1,17 +0,0 @@ ---- - -name: "Assignment: render a worksheet as a HyperFrames video" -about: "Workshop assignment. Run it with the ui-visual workflow." -title: "Assignment: render a worksheet as a HyperFrames video" -labels: assignment ---- - -Render a worksheet as a HyperFrames video with `renderWorksheetVideo` in `shared/src/render-video.ts`. - -Acceptance: -- Every item appears once, in page order, as one `data-scene` element. -- The answer key is the final scene, `data-scene="answer-key"`. -- `hyperframes lint` passes on the output. - -Workflow: ui-visual -Red test: `tests/assignments/worksheet-video.test.ts`. Run it with `ASSIGNMENT=1 npx vitest run tests/assignments/worksheet-video.test.ts`; it must fail for the reason above before any work. diff --git a/assignments/README.md b/assignments/README.md new file mode 100644 index 0000000..4f32479 --- /dev/null +++ b/assignments/README.md @@ -0,0 +1,24 @@ +# Assignments + +Pick one, read the whole brief, then file it as an issue with the command in its `## File it` section. The briefs are markdown files, not issue templates, so you can read the walkthrough before you commit to the task. + +Every assignment has a test that is skipped unless `ASSIGNMENT=1`. Run it red first, then make your factory turn it green. [F10](../docs/factory/f10-assignments.md) explains the steps shared by all of them. + +| Assignment | Module | Level | Workflow | Red test | +| --- | --- | --- | --- | --- | +| [name the approvers for a high-risk gate](approvers.md) | F6 | principal | `feature-atdd` | `tests/assignments/approvers.test.ts` | +| [canary prompt with rollback](canary.md) | F7 | senior | `llm-eval` | `tests/assignments/canary.test.ts` | +| [add the clock activity kind](clock-activity.md) | F2 | noob | `feature-bdd` | `tests/assignments/clock-activity.test.ts` | +| [fuzz the worksheet tool input](fuzz-tool-input.md) | F4 | senior | `security` | `tests/assignments/fuzz-tool-input.test.ts` | +| [an opt-in LLM-judge hook](judge-hook.md) | F13 | senior | `feature-atdd` | `tests/assignments/judge-hook.test.ts` | +| [specify the label state machine](label-machine.md) | F1 | principal | `feature-atdd` | `tests/assignments/label-machine.test.ts` | +| [a mod of your own](my-mod.md) | F3 | senior | `feature-atdd` | `tests/assignments/my-mod.test.ts` | +| [a workflow from your own company](own-workflow.md) | F8 | senior | `feature-atdd` | `tests/assignments/own-workflow.test.ts` | +| [record each factory module from a real run log](recording-from-run-log.md) | F0 | senior | `docs` | `tests/assignments/recording-from-run-log.test.ts` | +| [add a performance review lens](review-lens.md) | F5 | senior | `llm-eval` | `tests/assignments/review-lens.test.ts` | +| [a model for every stage name](stage-names.md) | F12 | noob | `llm-eval` | `tests/assignments/stage-names.test.ts` | +| [move worksheet templates into a database](template-db.md) | F8 | senior | `spike` | `tests/assignments/template-db.test.ts` | +| [seed a 3-sigma breach and run the incident workflow](three-sigma-incident.md) | F7 | principal | `incident` | `tests/assignments/three-sigma-incident.test.ts` | +| [render a worksheet as a HyperFrames video](worksheet-video.md) | F8 | senior | `ui-visual` | `tests/assignments/worksheet-video.test.ts` | + +Levels: `noob` needs only the start and daily lanes, `senior` the team and factory lanes, `principal` a design decision that has a cost. diff --git a/assignments/approvers.md b/assignments/approvers.md new file mode 100644 index 0000000..aa4197a --- /dev/null +++ b/assignments/approvers.md @@ -0,0 +1,54 @@ +# Assignment: name the approvers for a high-risk gate + +## Brief + +Read the approver logins from `.factory/config.yaml` (an `approvers` list in `portableSchema`) and pass them to `realHost` from `runIssue`. + +## Level + +principal + +## Acceptance + +- A high-risk gate opens with two listed people, a medium one with one. +- One person commenting twice counts once. +- The logged-in user is no longer assumed. + +## Workflow + +`feature-atdd`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/approvers.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `ASSIGNMENT=1 npx vitest run tests/assignments/approvers.test.ts`. + Expect: It fails because the config schema has no `approvers` key and the CLI still assumes the logged-in user. +2. File the issue: `gh issue create --title "Assignment: name the approvers for a high-risk gate" --body-file assignments/approvers.md --label assignment`. + Expect: It prints the new issue URL. +3. Write `factory/engine/portable.ts`, `factory/engine/cli.ts`. + Expect: The red test now fails on behaviour, not on a missing file. +4. Run the factory: `make factory ISSUE= FLAGS=--workflow=feature-atdd`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `ASSIGNMENT=1 npx vitest run tests/assignments/approvers.test.ts`. + Expect: It passes. + +## Discussion + +Why does one person commenting twice count once? + +- **Principal:** Who should be on the approver list for a high-risk change, and how do you keep that list from becoming a rubber stamp or a bottleneck? + +## File it + +```sh +gh issue create --title "Assignment: name the approvers for a high-risk gate" --body-file assignments/approvers.md --label assignment +``` diff --git a/assignments/canary.md b/assignments/canary.md new file mode 100644 index 0000000..fc98832 --- /dev/null +++ b/assignments/canary.md @@ -0,0 +1,54 @@ +# Assignment: canary prompt with rollback + +## Brief + +Add `api/src/agent/canary.ts` for the prompt in `api/src/agent/prompt.ts`. + +## Level + +senior + +## Acceptance + +- `routePrompt(requestId, share)` returns `"candidate"` or `"current"`, the same way for the same id, and near `share` over many ids. +- `decideCanary({ current, candidate, margin })` returns `"rollback"` when the candidate's pass rate falls below the current one by more than `margin`, otherwise `"keep"`. +- After a rollback the share is 0 and the event is logged. + +## Workflow + +`llm-eval`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/canary.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `ASSIGNMENT=1 npx vitest run tests/assignments/canary.test.ts`. + Expect: It fails because `api/src/agent/canary.ts` does not exist. +2. File the issue: `gh issue create --title "Assignment: canary prompt with rollback" --body-file assignments/canary.md --label assignment`. + Expect: It prints the new issue URL. +3. Write `api/src/agent/canary.ts`. + Expect: The red test now fails on behaviour, not on a missing file. +4. Run the factory: `make factory ISSUE= FLAGS=--workflow=llm-eval`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `ASSIGNMENT=1 npx vitest run tests/assignments/canary.test.ts`. + Expect: It passes. + +## Discussion + +Why must `routePrompt` give the same answer for the same request id? + +- **Principal:** Eval scores are noisy and small. How many requests does the canary need before a rollback is a decision and not a coin flip, and who owns that number? + +## File it + +```sh +gh issue create --title "Assignment: canary prompt with rollback" --body-file assignments/canary.md --label assignment +``` diff --git a/assignments/clock-activity.md b/assignments/clock-activity.md new file mode 100644 index 0000000..2353348 --- /dev/null +++ b/assignments/clock-activity.md @@ -0,0 +1,54 @@ +# Assignment: add the clock activity kind + +## Brief + +Add the `clock` activity kind: the model supplies a time (`hour`, `minute`), the code draws the face. + +## Level + +noob + +## Acceptance + +- `clock` is in `ACTIVITY_KINDS` and `activitySchema` accepts it. +- `renderActivity` draws an SVG face. +- It is listed in `kinds` for the bands that allow it in `shared/src/bands.ts`, and `checkBand` rejects it in the others. + +## Workflow + +`feature-bdd`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/clock-activity.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `ASSIGNMENT=1 npx vitest run tests/assignments/clock-activity.test.ts`. + Expect: It fails because `clock` is not yet in `ACTIVITY_KINDS`. +2. File the issue: `gh issue create --title "Assignment: add the clock activity kind" --body-file assignments/clock-activity.md --label assignment`. + Expect: It prints the new issue URL. +3. Write `shared/src/activity-kinds.ts`, `shared/src/render-activity.ts`, `shared/src/bands.ts`. + Expect: The red test now fails on behaviour, not on a missing file. +4. Run the factory: `make factory ISSUE= FLAGS=--workflow=feature-bdd`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `ASSIGNMENT=1 npx vitest run tests/assignments/clock-activity.test.ts`. + Expect: It passes. + +## Discussion + +Why does the model supply only `hour` and `minute` while the code draws the face? + +- **Principal:** A new activity kind touches the schema, the renderer and the band table. How would you make forgetting one of them a failing test? + +## File it + +```sh +gh issue create --title "Assignment: add the clock activity kind" --body-file assignments/clock-activity.md --label assignment +``` diff --git a/assignments/fuzz-tool-input.md b/assignments/fuzz-tool-input.md new file mode 100644 index 0000000..621f900 --- /dev/null +++ b/assignments/fuzz-tool-input.md @@ -0,0 +1,54 @@ +# Assignment: fuzz the worksheet tool input + +## Brief + +Property-test the worksheet tool input. Write the generator `arbWorksheet` in `shared/src/arb-worksheet.ts`: worksheet-shaped objects with hostile text, extreme sizes and every activity kind. + +## Level + +senior + +## Acceptance + +- Every worksheet the schema accepts renders, and markup in a text field never reaches the page raw. +- `checkBand` reports problems instead of throwing. +- Plant a mutant (an escaping step removed) and show the property fails on it. + +## Workflow + +`security`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/fuzz-tool-input.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `ASSIGNMENT=1 npx vitest run tests/assignments/fuzz-tool-input.test.ts`. + Expect: It fails because `shared/src/arb-worksheet.ts` does not exist. +2. File the issue: `gh issue create --title "Assignment: fuzz the worksheet tool input" --body-file assignments/fuzz-tool-input.md --label assignment`. + Expect: It prints the new issue URL. +3. Write `shared/src/arb-worksheet.ts`. + Expect: The red test now fails on behaviour, not on a missing file. +4. Run the factory: `make factory ISSUE= FLAGS=--workflow=security`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `ASSIGNMENT=1 npx vitest run tests/assignments/fuzz-tool-input.test.ts`. + Expect: It passes. + +## Discussion + +What did the planted mutant show that a single example test would not? + +- **Principal:** The generator is itself code that can be wrong. How would you know it reaches the inputs an attacker would send? + +## File it + +```sh +gh issue create --title "Assignment: fuzz the worksheet tool input" --body-file assignments/fuzz-tool-input.md --label assignment +``` diff --git a/assignments/judge-hook.md b/assignments/judge-hook.md new file mode 100644 index 0000000..04b9cef --- /dev/null +++ b/assignments/judge-hook.md @@ -0,0 +1,54 @@ +# Assignment: an opt-in LLM-judge hook + +## Brief + +Add `.claude/hooks/pretooluse-judge.mjs` exporting `shouldJudge(env)`, `decide(verdict)` and `diffToSend(diff)`. + +## Level + +senior + +## Acceptance + +- Off by default; on only when `JUDGE=1`. +- A `blocking` verdict blocks the commit, a `clean` one passes. +- A diff the secrets hook would block is never sent. + +## Workflow + +`feature-atdd`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/judge-hook.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `ASSIGNMENT=1 npx vitest run tests/assignments/judge-hook.test.ts`. + Expect: It fails because `.claude/hooks/pretooluse-judge.mjs` does not exist. +2. File the issue: `gh issue create --title "Assignment: an opt-in LLM-judge hook" --body-file assignments/judge-hook.md --label assignment`. + Expect: It prints the new issue URL. +3. Write `.claude/hooks/pretooluse-judge.mjs`. + Expect: The red test now fails on behaviour, not on a missing file. +4. Run the factory: `make factory ISSUE= FLAGS=--workflow=feature-atdd`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `ASSIGNMENT=1 npx vitest run tests/assignments/judge-hook.test.ts`. + Expect: It passes. + +## Discussion + +Why is the judge off unless `JUDGE=1`, and why is the diff filtered first? + +- **Principal:** A model that can block commits is a new single point of failure. What is its failure mode, and what happens to the team when it is down or wrong? + +## File it + +```sh +gh issue create --title "Assignment: an opt-in LLM-judge hook" --body-file assignments/judge-hook.md --label assignment +``` diff --git a/assignments/label-machine.md b/assignments/label-machine.md new file mode 100644 index 0000000..a4f8a31 --- /dev/null +++ b/assignments/label-machine.md @@ -0,0 +1,54 @@ +# Assignment: specify the label state machine + +## Brief + +Add `factory/label-machine.ts` with `applyEvent(labels, event)` and `canBuild(labels)`, for the labels in `.factory/config.json`. Events: `plan-written`, `human-approved`, `build-requested`, `stuck`, `retry`. + +## Level + +principal + +## Acceptance + +- A build never starts without `plan:approved`. +- A new plan always clears `plan:approved`. +- `needs-human` is never cleared by the factory, only by `human-approved`. + +## Workflow + +`feature-atdd`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/label-machine.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `ASSIGNMENT=1 npx vitest run tests/assignments/label-machine.test.ts`. + Expect: It fails because `factory/label-machine.ts` does not exist. +2. File the issue: `gh issue create --title "Assignment: specify the label state machine" --body-file assignments/label-machine.md --label assignment`. + Expect: It prints the new issue URL. +3. Write `factory/label-machine.ts`. + Expect: The red test now fails on behaviour, not on a missing file. +4. Run the factory: `make factory ISSUE= FLAGS=--workflow=feature-atdd`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `ASSIGNMENT=1 npx vitest run tests/assignments/label-machine.test.ts`. + Expect: It passes. + +## Discussion + +Which invariant is easiest to break with a sequence a human would never try? + +- **Principal:** Labels are the factory's only shared state. Where would you rather keep that state, and what does moving it cost? + +## File it + +```sh +gh issue create --title "Assignment: specify the label state machine" --body-file assignments/label-machine.md --label assignment +``` diff --git a/assignments/my-mod.md b/assignments/my-mod.md new file mode 100644 index 0000000..550885b --- /dev/null +++ b/assignments/my-mod.md @@ -0,0 +1,56 @@ +# Assignment: a mod of your own + +## Brief + +Add a mod under `plugins/` that shows above the prompt. + +Run the test with `MOD=`. + +## Level + +senior + +## Acceptance + +- `.claude-plugin/plugin.json`, `hooks/hooks.json` and `hooks/register.tsx` exist. +- `claude plugin validate` passes. +- `hooks/register.test.tsx` feeds it a fixed state and checks what it draws. + +## Workflow + +`feature-atdd`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/my-mod.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `MOD= ASSIGNMENT=1 npx vitest run tests/assignments/my-mod.test.ts`. + Expect: It fails because `MOD` is not set, so there is no mod folder to check. +2. File the issue: `gh issue create --title "Assignment: a mod of your own" --body-file assignments/my-mod.md --label assignment`. + Expect: It prints the new issue URL. +3. Write `plugins//.claude-plugin/plugin.json`, `plugins//hooks/hooks.json`, `plugins//hooks/register.tsx`, `plugins//hooks/register.test.tsx`. + Expect: The red test now fails on behaviour, not on a missing file. +4. Run the factory: `make factory ISSUE= FLAGS=--workflow=feature-atdd`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `MOD= ASSIGNMENT=1 npx vitest run tests/assignments/my-mod.test.ts`. + Expect: It passes. + +## Discussion + +What fixed state do you feed the mod, and what must it draw? + +- **Principal:** A mod runs inside every engineer's session. What is your review and rollout policy for one, and how do you pull a bad one back? + +## File it + +```sh +gh issue create --title "Assignment: a mod of your own" --body-file assignments/my-mod.md --label assignment +``` diff --git a/assignments/own-workflow.md b/assignments/own-workflow.md new file mode 100644 index 0000000..6eda86a --- /dev/null +++ b/assignments/own-workflow.md @@ -0,0 +1,56 @@ +# Assignment: a workflow from your own company + +## Brief + +Add `.factory/workflows/.yaml` for . + +Run the test with `WORKFLOW=`. + +## Level + +senior + +## Acceptance + +- The first line is `# Method:`, and the file names its source. +- It uses only checks from `CHECK_NAMES` and loads. +- At least one step carries a check, with a failing input shown first. + +## Workflow + +`feature-atdd`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/own-workflow.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `WORKFLOW= ASSIGNMENT=1 npx vitest run tests/assignments/own-workflow.test.ts`. + Expect: It fails because `WORKFLOW` is not set, so there is no workflow file to check. +2. File the issue: `gh issue create --title "Assignment: a workflow from your own company" --body-file assignments/own-workflow.md --label assignment`. + Expect: It prints the new issue URL. +3. Write `.factory/workflows/.yaml`. + Expect: The red test now fails on behaviour, not on a missing file. +4. Run the factory: `make factory ISSUE= FLAGS=--workflow=feature-atdd`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `WORKFLOW= ASSIGNMENT=1 npx vitest run tests/assignments/own-workflow.test.ts`. + Expect: It passes. + +## Discussion + +Which step of your workflow carries the check that matters most, and what input fails it? + +- **Principal:** A workflow encodes how your team ships. Who is allowed to change it, and how is a change reviewed so it cannot quietly weaken a check? + +## File it + +```sh +gh issue create --title "Assignment: a workflow from your own company" --body-file assignments/own-workflow.md --label assignment +``` diff --git a/assignments/recording-from-run-log.md b/assignments/recording-from-run-log.md new file mode 100644 index 0000000..d65c9e8 --- /dev/null +++ b/assignments/recording-from-run-log.md @@ -0,0 +1,57 @@ +# Assignment: record each factory module from a real run log + +## Brief + +For each module F0 to F13, record a real run and link its `run.log`. No mock output. + +How to run it: `make factory ISSUE= FLAGS=--workflow=docs --auth=api-key`. Read `docs/factory/f8-workflows.md`. + +## Level + +senior + +## Acceptance + +- `specs//run.log` exists for the run you record. +- `docs/recordings/run-.md` has the headings `## What happened` and `## What it proves`. +- Every line inside its fenced `log` block is a verbatim line of that `run.log`. +- `make check` passes with zero warnings and the new check is wired in. + +## Workflow + +`docs`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/recording-from-run-log.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `RUN= ASSIGNMENT=1 npx vitest run tests/assignments/recording-from-run-log.test.ts`. + Expect: It fails because `docs/recordings/run-.md` does not exist. +2. File the issue: `gh issue create --title "Assignment: record each factory module from a real run log" --body-file assignments/recording-from-run-log.md --label assignment`. + Expect: It prints the new issue URL. +3. Run the factory: `make factory ISSUE= FLAGS=--workflow=docs`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +4. Write `docs/recordings/run-.md`. + Expect: The red test now fails on behaviour, not on a missing file. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `RUN= ASSIGNMENT=1 npx vitest run tests/assignments/recording-from-run-log.test.ts`. + Expect: It passes. + +## Discussion + +Why must every line in the log block be a line of the real `run.log`? + +- **Principal:** A recording that is edited after the fact is a story, not evidence. How would you make a doctored log fail a check? + +## File it + +```sh +gh issue create --title "Assignment: record each factory module from a real run log" --body-file assignments/recording-from-run-log.md --label assignment +``` diff --git a/assignments/review-lens.md b/assignments/review-lens.md new file mode 100644 index 0000000..5f97972 --- /dev/null +++ b/assignments/review-lens.md @@ -0,0 +1,54 @@ +# Assignment: add a performance review lens + +## Brief + +Add a `performance-reviewer` agent in `.claude/agents/performance-reviewer.md`. It reads the diff and replies with a JSON array of findings with file, line, severity, claim and evidence. + +## Level + +senior + +## Acceptance + +- Read-only tools (`Read, Grep, Glob`) and a `VERDICT:` line, like `security-reviewer`. +- A finding only one voter confirms is dropped. +- A workflow's `review: { lenses: [...] }` names it. + +## Workflow + +`llm-eval`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/review-lens.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `ASSIGNMENT=1 npx vitest run tests/assignments/review-lens.test.ts`. + Expect: It fails because `.claude/agents/performance-reviewer.md` does not exist. +2. File the issue: `gh issue create --title "Assignment: add a performance review lens" --body-file assignments/review-lens.md --label assignment`. + Expect: It prints the new issue URL. +3. Write `.claude/agents/performance-reviewer.md`. + Expect: The red test now fails on behaviour, not on a missing file. +4. Run the factory: `make factory ISSUE= FLAGS=--workflow=llm-eval`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `ASSIGNMENT=1 npx vitest run tests/assignments/review-lens.test.ts`. + Expect: It passes. + +## Discussion + +Why is a finding that only one voter confirms dropped? + +- **Principal:** Each lens adds cost and noise. How would you decide a lens has earned its place, and when would you remove one? + +## File it + +```sh +gh issue create --title "Assignment: add a performance review lens" --body-file assignments/review-lens.md --label assignment +``` diff --git a/assignments/stage-names.md b/assignments/stage-names.md new file mode 100644 index 0000000..c3db119 --- /dev/null +++ b/assignments/stage-names.md @@ -0,0 +1,53 @@ +# Assignment: a model for every stage name + +## Brief + +Every stage name in every workflow resolves to a config stage, or the workflow test fails. + +## Level + +noob + +## Acceptance + +- Each name used in `.factory/workflows/` is a key of `stepStages` or one of `STAGE_NAMES`. +- Each step resolves to a model in `allowedModels`. + +## Workflow + +`llm-eval`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/stage-names.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `ASSIGNMENT=1 npx vitest run tests/assignments/stage-names.test.ts`. + Expect: It fails because some workflow stage names, for example `queue` in `bug-tdd.yaml`, are in neither `stepStages` nor `STAGE_NAMES`. +2. File the issue: `gh issue create --title "Assignment: a model for every stage name" --body-file assignments/stage-names.md --label assignment`. + Expect: It prints the new issue URL. +3. Write `.factory/config.json`. + Expect: The red test now fails on behaviour, not on a missing file. +4. Run the factory: `make factory ISSUE= FLAGS=--workflow=llm-eval`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `ASSIGNMENT=1 npx vitest run tests/assignments/stage-names.test.ts`. + Expect: It passes. + +## Discussion + +What happens to a step whose stage name is unmapped today? + +- **Principal:** A silent fallback to the agent's own model is a cost decision nobody made. Where else does your system fall back without saying so? + +## File it + +```sh +gh issue create --title "Assignment: a model for every stage name" --body-file assignments/stage-names.md --label assignment +``` diff --git a/assignments/template-db.md b/assignments/template-db.md new file mode 100644 index 0000000..5a1f9d2 --- /dev/null +++ b/assignments/template-db.md @@ -0,0 +1,55 @@ +# Assignment: move worksheet templates into a database + +## Brief + +Spike first: which store for worksheet templates, and what it rules out. File the spike alone and read its decision record before filing the build. + +Then add `createDbTemplateStore(file)` in `api/src/templates/db-template-store.ts` with the same `add`, `size` and `find` as `template-store.ts`. + +## Level + +senior + +## Acceptance + +- A second store opened on the same file finds a template the first saved. +- Adding the same template twice keeps one. + +## Workflow + +`spike`. Then file the build with `feature-sdd`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/template-db.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `ASSIGNMENT=1 npx vitest run tests/assignments/template-db.test.ts`. + Expect: It fails because `api/src/templates/db-template-store.ts` does not exist. +2. File the issue: `gh issue create --title "Assignment: move worksheet templates into a database" --body-file assignments/template-db.md --label assignment`. + Expect: It prints the new issue URL. +3. Write `api/src/templates/db-template-store.ts`. + Expect: The red test now fails on behaviour, not on a missing file. +4. Run the factory: `make factory ISSUE= FLAGS=--workflow=spike`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `ASSIGNMENT=1 npx vitest run tests/assignments/template-db.test.ts`. + Expect: It passes. + +## Discussion + +What did the spike's decision record rule out, and why? + +- **Principal:** The in-memory store becomes durable. What breaks for a deploy that runs two API instances, and would you still pick the same store? + +## File it + +```sh +gh issue create --title "Assignment: move worksheet templates into a database" --body-file assignments/template-db.md --label assignment +``` diff --git a/assignments/three-sigma-incident.md b/assignments/three-sigma-incident.md new file mode 100644 index 0000000..049c962 --- /dev/null +++ b/assignments/three-sigma-incident.md @@ -0,0 +1,57 @@ +# Assignment: seed a 3-sigma breach and run the incident workflow + +## Brief + +Seed a CI-failure spike in `.factory/bands.yaml` metrics, let `--maintain` write `intent.md`, then run `incident` to a PR. + +How to run it: `make factory ISSUE= FLAGS=--workflow=incident --auth=api-key`. Read `docs/factory/f8-workflows.md`. + +## Level + +principal + +## Acceptance + +- `fixtures/incidents/breach.json` holds a `metric`, a `direction`, a `history` and a `latest` value. +- `detect` reports at least 3 sigma for it and `actionFor` answers `intent`. +- The `intent.md` your run wrote under `specs//` names the metric. +- `make check` passes with zero warnings and the new check is wired in. + +## Workflow + +`incident`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/three-sigma-incident.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `ISSUE= ASSIGNMENT=1 npx vitest run tests/assignments/three-sigma-incident.test.ts`. + Expect: It fails because `fixtures/incidents/breach.json` does not exist. +2. File the issue: `gh issue create --title "Assignment: seed a 3-sigma breach and run the incident workflow" --body-file assignments/three-sigma-incident.md --label assignment`. + Expect: It prints the new issue URL. +3. Write `fixtures/incidents/breach.json`. + Expect: The red test now fails on behaviour, not on a missing file. +4. Run the factory: `make factory ISSUE= FLAGS=--workflow=incident`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `ISSUE= ASSIGNMENT=1 npx vitest run tests/assignments/three-sigma-incident.test.ts`. + Expect: It passes. + +## Discussion + +Which number in your history decides whether the latest value is a breach? + +- **Principal:** A three-sigma rule fires on noise at a known rate. What would you change so a team trusts the alerts, and what would you give up? + +## File it + +```sh +gh issue create --title "Assignment: seed a 3-sigma breach and run the incident workflow" --body-file assignments/three-sigma-incident.md --label assignment +``` diff --git a/assignments/worksheet-video.md b/assignments/worksheet-video.md new file mode 100644 index 0000000..bcd702e --- /dev/null +++ b/assignments/worksheet-video.md @@ -0,0 +1,54 @@ +# Assignment: render a worksheet as a HyperFrames video + +## Brief + +Render a worksheet as a HyperFrames video with `renderWorksheetVideo` in `shared/src/render-video.ts`. + +## Level + +senior + +## Acceptance + +- Every item appears once, in page order, as one `data-scene` element. +- The answer key is the final scene, `data-scene="answer-key"`. +- `hyperframes lint` passes on the output. + +## Workflow + +`ui-visual`. Workflows are listed in [F8](../docs/factory/f8-workflows.md). + +## Red test + +```sh +ASSIGNMENT=1 npx vitest run tests/assignments/worksheet-video.test.ts +``` + +It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. + +## Walkthrough + +1. Run the red test: `ASSIGNMENT=1 npx vitest run tests/assignments/worksheet-video.test.ts`. + Expect: It fails because `shared/src/render-video.ts` does not exist. +2. File the issue: `gh issue create --title "Assignment: render a worksheet as a HyperFrames video" --body-file assignments/worksheet-video.md --label assignment`. + Expect: It prints the new issue URL. +3. Write `shared/src/render-video.ts`. + Expect: The red test now fails on behaviour, not on a missing file. +4. Run the factory: `make factory ISSUE= FLAGS=--workflow=ui-visual`. + Expect: The run ends `awaiting-approval` or `needs-human`, and `specs//run.log` says which. +5. Run the gate: `make check`. + Expect: Zero errors and zero warnings. +6. Run the red test again: `ASSIGNMENT=1 npx vitest run tests/assignments/worksheet-video.test.ts`. + Expect: It passes. + +## Discussion + +Why does the video reuse the page's item order instead of rendering its own? + +- **Principal:** A video renderer and a page renderer now both walk the worksheet. What stops the two from drifting, and which of them should own the order? + +## File it + +```sh +gh issue create --title "Assignment: render a worksheet as a HyperFrames video" --body-file assignments/worksheet-video.md --label assignment +``` diff --git a/docs/factory/f10-assignments.md b/docs/factory/f10-assignments.md index e624190..3ac2fee 100644 --- a/docs/factory/f10-assignments.md +++ b/docs/factory/f10-assignments.md @@ -4,7 +4,7 @@ ## What exists -Each assignment below has an issue template in `.github/ISSUE_TEMPLATE/` and a red test in `tests/assignments/`. Every red test is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. `tests/structure/assignments.test.ts` checks that every section on this page names a template and a test that exist, that the template carries the `assignment` label, and that the test is gated on `ASSIGNMENT`. +Each assignment below has a brief in `assignments/` and a red test in `tests/assignments/`. The brief is a markdown file with a walkthrough, so you can read the whole task before you file anything. See [assignments/README.md](../../assignments/README.md) for the table. Every red test is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. `tests/structure/assignments.test.ts` checks that every section on this page names a brief and a test that exist, that every brief has its nine headings in order, and that the test is gated on `ASSIGNMENT`. Each red test imports the module you will write by a dynamic path, so it fails on a missing module until you create it, then on behaviour until you get it right. Because they are skipped, CI has never run them red. Run yours red first, as step 3 says. @@ -15,7 +15,7 @@ Choose one assignment, file it as an issue, write its red test first, and make y ## Steps (the same for every assignment) 1. Check you can run the factory: `make doctor`, then `npx vitest run factory/engine`. -2. File the issue from the template: `gh issue create --template assignment-.md`, or pick it in the web form. Add the `factory:ready` label (the label in `.factory/config.json`) when you want the factory to take it. `npx tsx factory/engine/main.ts --watch` lists what carries it. +2. File the issue from the brief: `gh issue create --title "Assignment: " --body-file assignments/<name>.md --label assignment`. Add the `factory:ready` label (the label in `.factory/config.json`) when you want the factory to take it. `npx tsx factory/engine/main.ts --watch` lists what carries it. 3. Run the red test with `ASSIGNMENT=1` and read the failure. It must fail for the reason the issue names. A test that passes before any work proves nothing, which is what the `red` check in `factory/engine/checks/red.ts` refuses. 4. Run the workflow: `npx tsx factory/engine/main.ts <issue> --workflow=<name>`. The script reads the variable named in `keySource` of `.factory/config.yaml` and loads the key inside its own process, so keep the key off the command line. With a Pro or Max plan, export `CLAUDE_CODE_OAUTH_TOKEN` from `claude setup-token` and add `--auth=subscription` instead ([Auth modes](auth-modes.md)). A key run stops as `needs-human` with the reason `spend cap` once it has spent `spend.stopAt` of `spend.capUsd` in `.factory/config.json`; a subscription run refuses further agent calls at `subscriptionCap`. 5. Read `specs/<issue>/run.log`, `specs/<issue>/cost.json` and `.factory/runs/<issue>/metrics.json`. If the run ended `needs-human`, the log says which check capped out or that the spend cap did. If it ended `awaiting-approval`, the log says how many people the risk tier asks for. @@ -25,7 +25,7 @@ Each red test is skipped by default, as `make prove-homework` is for the take-ho ## 1. A worksheet video with the HyperFrames timeline -- **Issue template:** `.github/ISSUE_TEMPLATE/assignment-worksheet-video.md` +- **Brief:** `assignments/worksheet-video.md` - **Red test:** `tests/assignments/worksheet-video.test.ts` - **Workflow:** `ui-visual`. - **Goal:** `shared/src/render.ts` turns a worksheet into a HyperFrames still composition, and `hyperframes` is a dependency in `package.json`. Add `renderWorksheetVideo` in `shared/src/render-video.ts` so a worksheet plays as a short video: items appear in order, the answer key last. @@ -33,7 +33,7 @@ Each red test is skipped by default, as `make prove-homework` is for the take-ho ## 2. Templates in a database -- **Issue template:** `.github/ISSUE_TEMPLATE/assignment-template-db.md` +- **Brief:** `assignments/template-db.md` - **Red test:** `tests/assignments/template-db.test.ts` - **Workflow:** `spike` first, then `feature-sdd`. - **Goal:** `api/src/templates/template-store.ts` keeps templates in a `Map`. Replace its storage with a database after a spike on which one. File the spike alone first and read its decision record before filing the build. @@ -41,7 +41,7 @@ Each red test is skipped by default, as `make prove-homework` is for the take-ho ## 3. A canary prompt with rollback when the eval drops -- **Issue template:** `.github/ISSUE_TEMPLATE/assignment-canary.md` +- **Brief:** `assignments/canary.md` - **Red test:** `tests/assignments/canary.test.ts` - **Workflow:** `llm-eval`. - **Goal:** serve a changed prompt from `api/src/agent/prompt.ts` to a small share of requests, compare its eval score to the current one, and roll back by itself when it drops. @@ -49,7 +49,7 @@ Each red test is skipped by default, as `make prove-homework` is for the take-ho ## 4. Fuzzing the tool input -- **Issue template:** `.github/ISSUE_TEMPLATE/assignment-fuzz-tool-input.md` +- **Brief:** `assignments/fuzz-tool-input.md` - **Red test:** `tests/assignments/fuzz-tool-input.test.ts` - **Workflow:** `security`. - **Goal:** the model's tool input is validated with zod (`worksheetSchema` in `shared/src/schema.ts`) before it renders. Fuzz that boundary and find an input that gets through and breaks a page. @@ -57,7 +57,7 @@ Each red test is skipped by default, as `make prove-homework` is for the take-ho ## 5. The label state machine as a property -- **Issue template:** `.github/ISSUE_TEMPLATE/assignment-label-machine.md` +- **Brief:** `assignments/label-machine.md` - **Red test:** `tests/assignments/label-machine.test.ts` - **Workflow:** `feature-atdd`. - **Goal:** the labels in `.factory/config.json` (`factory:ready`, `factory:planned`, `plan:approved`, `needs-human`, `factory:protected-ok`) form a state machine that `factory/run.ts` drives. State its rules in `factory/label-machine.ts` as `applyEvent(labels, event)` and `canBuild(labels)`, and check them with generated event sequences. TLA+ is the heavier option; a `fast-check` model-based test is enough. @@ -65,7 +65,7 @@ Each red test is skipped by default, as `make prove-homework` is for the take-ho ## 6. A new activity kind -- **Issue template:** `.github/ISSUE_TEMPLATE/assignment-clock-activity.md` +- **Brief:** `assignments/clock-activity.md` - **Red test:** `tests/assignments/clock-activity.test.ts` - **Workflow:** `feature-bdd`. - **Goal:** add a new kind of activity to a worksheet, for example a clock face. `ACTIVITY_KINDS` and `activitySchema` live in `shared/src/activity-kinds.ts`, `renderActivity` in `shared/src/render-activity.ts`, and each band lists the kinds it allows in `shared/src/bands.ts`. @@ -73,7 +73,7 @@ Each red test is skipped by default, as `make prove-homework` is for the take-ho ## 7. A workflow from your own company -- **Issue template:** `.github/ISSUE_TEMPLATE/assignment-own-workflow.md` +- **Brief:** `assignments/own-workflow.md` - **Red test:** `tests/assignments/own-workflow.test.ts` - **Workflow:** the one you write (see F8). - **Goal:** encode how your team really ships one kind of change. @@ -81,7 +81,7 @@ Each red test is skipped by default, as `make prove-homework` is for the take-ho ## 8. A new review lens -- **Issue template:** `.github/ISSUE_TEMPLATE/assignment-review-lens.md` +- **Brief:** `assignments/review-lens.md` - **Red test:** `tests/assignments/review-lens.test.ts` - **Workflow:** `llm-eval`. - **Goal:** add a reviewer that hunts one kind of defect. `reviewRound` in `factory/engine/review.ts` takes a list of lens agents, runs them in parallel, and keeps a finding only when two of three verifiers vote yes. A lens runs only if a workflow's `review: { lenses: [...] }` names it, so the assignment ends with that edit. @@ -89,7 +89,7 @@ Each red test is skipped by default, as `make prove-homework` is for the take-ho ## 9. An LLM-judge pre-commit hook -- **Issue template:** `.github/ISSUE_TEMPLATE/assignment-judge-hook.md` +- **Brief:** `assignments/judge-hook.md` - **Red test:** `tests/assignments/judge-hook.test.ts` - **Workflow:** `feature-atdd`. - **Goal:** the opt-in paid pattern. A hook asks a model to judge a staged change, and only runs when the person turns it on. @@ -97,7 +97,7 @@ Each red test is skipped by default, as `make prove-homework` is for the take-ho ## 10. A mod of your own -- **Issue template:** `.github/ISSUE_TEMPLATE/assignment-my-mod.md` +- **Brief:** `assignments/my-mod.md` - **Red test:** `tests/assignments/my-mod.test.ts` - **Workflow:** `feature-atdd`. - **Goal:** build a small change to the Claude interface. `plugins/factory-status` is the working example in this repo. @@ -105,7 +105,7 @@ Each red test is skipped by default, as `make prove-homework` is for the take-ho ## 11. Two people at a high-risk gate -- **Issue template:** `.github/ISSUE_TEMPLATE/assignment-approvers.md` +- **Brief:** `assignments/approvers.md` - **Red test:** `tests/assignments/approvers.test.ts` - **Workflow:** `feature-atdd`. - **Goal:** `gateMet` in `factory/engine/interpret.ts` asks for as many approvals as the risk tier needs, and `approvalIn` in `factory/engine/real-host.ts` already counts distinct listed logins. But `runIssue` in `factory/engine/cli.ts` passes one approver, the logged-in `gh` user, so a high-risk gate can never open in a real run. Let a repo name its approvers. @@ -113,12 +113,28 @@ Each red test is skipped by default, as `make prove-homework` is for the take-ho ## 12. A model for every stage name -- **Issue template:** `.github/ISSUE_TEMPLATE/assignment-stage-names.md` +- **Brief:** `assignments/stage-names.md` - **Red test:** `tests/assignments/stage-names.test.ts` - **Workflow:** `llm-eval`. - **Goal:** a step whose stage name is in neither `stepStages` nor `STAGE_NAMES` falls back to the agent's frontmatter model. The `docs` workflow's `write` step is already mapped (`factory/engine/docs-routing.test.ts`), but names such as `analyze` and `characterize` are not. Make an unmapped stage name fail a test instead of silently using the fallback. - **What the test asserts:** it loads every file in `.factory/workflows/` and lists each stage name that is neither a key of `stepStages` in `.factory/config.json` nor in `STAGE_NAMES`. The list must be empty, so it is red for the names still unmapped. A second check resolves each step through `resolveModel` in `factory/engine/models.ts` to a model in `allowedModels`. +## 13. Record each factory module from a real run log + +- **Brief:** `assignments/recording-from-run-log.md` +- **Red test:** `tests/assignments/recording-from-run-log.test.ts` +- **Workflow:** `docs`. +- **Goal:** a recording is evidence only when it quotes a run that happened. Record one module from its own `run.log`, with nothing typed by hand. +- **What the test asserts:** run it with `RUN=<n>`. `specs/<n>/run.log` and `docs/recordings/run-<n>.md` exist, every line in the recording's fenced `log` block is a verbatim line of that `run.log`, and the recording has the headings `## What happened` and `## What it proves`. + +## 14. Seed a 3-sigma breach and run the incident workflow + +- **Brief:** `assignments/three-sigma-incident.md` +- **Red test:** `tests/assignments/three-sigma-incident.test.ts` +- **Workflow:** `incident`. +- **Goal:** the factory turns a metric that leaves its band into an `intent.md`. Seed a breach on purpose and follow it through the `incident` workflow. +- **What the test asserts:** run it with `ISSUE=<n>`. `fixtures/incidents/breach.json` holds a metric, a direction, a history and a latest value. `detect` in `factory/engine/bands.ts` reports at least 3 sigma for it and `actionFor` answers `intent`. The `specs/<n>/intent.md` your run wrote names the metric. + ## Worked example: review calls in the spend cap `runReview` in `factory/engine/interpret.ts` counts every lens hunt, ballot and fix toward the run total and gives each the remaining-budget ceiling. `factory/engine/review-spend.test.ts` pins it. Read it before you cap another kind of call. diff --git a/levels.json b/levels.json index 2f37b71..bcbf937 100644 --- a/levels.json +++ b/levels.json @@ -210,6 +210,7 @@ "tests/structure/f5-review-doc.test.ts", "tests/structure/assignments.test.ts", "tests/assignments/**", + "assignments/**", ".factory/bands.yaml", "scripts/eval-score.ts", "scripts/eval-score.test.ts", diff --git a/tests/assignments/recording-from-run-log.test.ts b/tests/assignments/recording-from-run-log.test.ts new file mode 100644 index 0000000..fb81d30 --- /dev/null +++ b/tests/assignments/recording-from-run-log.test.ts @@ -0,0 +1,34 @@ +import { existsSync, readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; + +// Assignment: a recording made from a real run log. Skipped until you start: +// ASSIGNMENT=1 RUN=<n> npx vitest run tests/assignments/recording-from-run-log.test.ts +const on = process.env.ASSIGNMENT === "1"; +const run = process.env.RUN ?? ""; + +const logLines = (doc: string): string[] => { + const block = doc.match(/```log\n([\s\S]*?)```/); + return block?.[1]?.split("\n").filter((l) => l.trim() !== "") ?? []; +}; + +describe.skipIf(!on)("assignment: record a module from a real run log", () => { + it("has a run log and a recording for the run you name", () => { + expect(run, "set RUN=<n>, the spec number of your run").not.toBe(""); + expect(existsSync(`specs/${run}/run.log`)).toBe(true); + expect(existsSync(`docs/recordings/run-${run}.md`)).toBe(true); + }); + + it("quotes only lines the run really wrote", () => { + const doc = readFileSync(`docs/recordings/run-${run}.md`, "utf8"); + const real = new Set(readFileSync(`specs/${run}/run.log`, "utf8").split("\n")); + const quoted = logLines(doc); + expect(quoted.length).toBeGreaterThan(0); + expect(quoted.filter((l) => !real.has(l))).toEqual([]); + }); + + it("says what happened and what it proves", () => { + const doc = readFileSync(`docs/recordings/run-${run}.md`, "utf8"); + expect(doc).toMatch(/^## What happened$/m); + expect(doc).toMatch(/^## What it proves$/m); + }); +}); diff --git a/tests/assignments/three-sigma-incident.test.ts b/tests/assignments/three-sigma-incident.test.ts new file mode 100644 index 0000000..75bd858 --- /dev/null +++ b/tests/assignments/three-sigma-incident.test.ts @@ -0,0 +1,39 @@ +import { existsSync, readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; + +// Assignment: seed a 3-sigma breach and run the incident workflow. Skipped until you start: +// ASSIGNMENT=1 ISSUE=<n> npx vitest run tests/assignments/three-sigma-incident.test.ts +const on = process.env.ASSIGNMENT === "1"; +const FIXTURE = "fixtures/incidents/breach.json"; + +type Fixture = { + metric: string; + direction: "up-is-bad" | "down-is-bad"; + history: number[]; + latest: number; +}; + +describe.skipIf(!on)("assignment: a 3-sigma breach becomes an intent", () => { + it("has a fixture with a metric, a history and a latest value", () => { + expect(existsSync(FIXTURE)).toBe(true); + const f = JSON.parse(readFileSync(FIXTURE, "utf8")) as Fixture; + expect(f.metric).not.toBe(""); + expect(f.history.length).toBeGreaterThanOrEqual(3); + expect(typeof f.latest).toBe("number"); + }); + + it("trips the band at 3 sigma, which the factory answers with an intent", async () => { + const { detect, actionFor } = await import("../../factory/engine/bands.ts"); + const f = JSON.parse(readFileSync(FIXTURE, "utf8")) as Fixture; + const breach = detect(f.metric, f.history, f.latest, f.direction); + expect(breach.sigma).toBeGreaterThanOrEqual(3); + expect(actionFor(breach)).toBe("intent"); + }); + + it("has an intent.md from the incident run that names the metric", () => { + const issue = process.env.ISSUE ?? ""; + expect(issue, "set ISSUE=<n>, the issue your incident run used").not.toBe(""); + const f = JSON.parse(readFileSync(FIXTURE, "utf8")) as Fixture; + expect(readFileSync(`specs/${issue}/intent.md`, "utf8")).toContain(f.metric); + }); +}); diff --git a/tests/structure/assignments.test.ts b/tests/structure/assignments.test.ts index 1697d22..d549388 100644 --- a/tests/structure/assignments.test.ts +++ b/tests/structure/assignments.test.ts @@ -1,57 +1,119 @@ import { existsSync, readdirSync, readFileSync } from "node:fs"; import { describe, expect, it } from "vitest"; -// docs/factory/f10-assignments.md names an issue template and a red test for every assignment. -// This test holds the page to the files: a section without both, or a test that is not gated, fails. +// Assignments are markdown briefs in assignments/, listed in docs/factory/f10-assignments.md. +// This test holds the briefs, the page, the README table and the red tests to one another. const doc = readFileSync("docs/factory/f10-assignments.md", "utf8"); +const names = readdirSync("assignments") + .filter((f) => f.endsWith(".md") && f !== "README.md") + .map((f) => f.replace(/\.md$/, "")) + .sort(); +const brief = (name: string) => readFileSync(`assignments/${name}.md`, "utf8"); + +const HEADINGS = [ + "Brief", + "Level", + "Acceptance", + "Workflow", + "Red test", + "Walkthrough", + "Discussion", + "File it", +]; +const LEVELS = ["noob", "senior", "principal"]; + const sections = doc .split(/^## /m) .filter((s) => /^\d+\. /.test(s)) .map((s) => ({ title: s.split("\n")[0] ?? "", - template: s.match(/Issue template:\*\* `([^`]+)`/)?.[1], + brief: s.match(/Brief:\*\* `([^`]+)`/)?.[1], test: s.match(/Red test:\*\* `([^`]+)`/)?.[1], })); -const listOrEmpty = (dir: string) => (existsSync(dir) ? readdirSync(dir) : []); -describe("f10 assignments", () => { - it("lists assignments", () => { - expect(sections.length).toBeGreaterThan(0); - }); - - it("gives every section an issue template that exists and carries the assignment label", () => { - const bad = sections.flatMap(({ title, template }) => { - if (!template || !existsSync(template)) return [`${title}: template missing`]; - return /^labels: assignment$/m.test(readFileSync(template, "utf8")) - ? [] - : [`${title}: no assignment label`]; - }); - expect(bad).toEqual([]); - }); +const body = (text: string, heading: string) => + text + .split(new RegExp(`^## ${heading}$`, "m"))[1] + ?.split(/^## /m)[0] + ?.trim() ?? ""; - it("gives every section a red test that exists and is skipped unless ASSIGNMENT=1", () => { - const bad = sections.flatMap(({ title, test }) => { - if (!test || !existsSync(test)) return [`${title}: test missing`]; - const text = readFileSync(test, "utf8"); - return /describe\.skipIf\(!on\)/.test(text) && /ASSIGNMENT === "1"/.test(text) - ? [] - : [`${title}: not gated on ASSIGNMENT`]; - }); +describe("f10 assignments page", () => { + it("names a brief and a red test that exist, for every section", () => { + const bad = sections.flatMap(({ title, brief: b, test }) => [ + ...(b && existsSync(b) ? [] : [`${title}: brief missing`]), + ...(test && existsSync(test) ? [] : [`${title}: test missing`]), + ]); + expect(sections.length).toBeGreaterThan(0); expect(bad).toEqual([]); }); - it("has no template or test the page does not name", () => { - const named = new Set(sections.flatMap((s) => [s.template, s.test])); + it("names every brief and every red test, and nothing else", () => { + const named = new Set(sections.flatMap((s) => [s.brief, s.test])); const files = [ - ...listOrEmpty(".github/ISSUE_TEMPLATE") - .filter((f) => f.startsWith("assignment-")) - .map((f) => `.github/ISSUE_TEMPLATE/${f}`), - ...listOrEmpty("tests/assignments").map((f) => `tests/assignments/${f}`), + ...names.map((n) => `assignments/${n}.md`), + ...readdirSync("tests/assignments").map((f) => `tests/assignments/${f}`), ]; expect(files.filter((f) => !named.has(f))).toEqual([]); }); - it("does not call any test proposed", () => { + it("does not call a test proposed, and no longer talks of issue templates", () => { expect(doc).not.toMatch(/proposed/i); + expect(doc).not.toMatch(/issue template/i); + }); +}); + +describe("assignment briefs", () => { + it.each(names)("%s has its headings in order", (name) => { + const text = brief(name); + expect(text).toMatch(/^# Assignment: .+/); + const found = [...text.matchAll(/^## (.+)$/gm)].map((m) => m[1]); + expect(found).toEqual(HEADINGS); + }); + + it.each(names)("%s names a level, a workflow that exists and a gated red test", (name) => { + const text = brief(name); + expect(LEVELS).toContain(body(text, "Level")); + const workflow = body(text, "Workflow").match(/`([^`]+)`/)?.[1] ?? ""; + expect(existsSync(`.factory/workflows/${workflow}.yaml`)).toBe(true); + const command = body(text, "Red test").match(/```sh\n(.+)\n```/)?.[1]; + expect(command).toBe(`ASSIGNMENT=1 npx vitest run tests/assignments/${name}.test.ts`); + const test = readFileSync(`tests/assignments/${name}.test.ts`, "utf8"); + expect(test).toMatch(/describe\.skipIf\(!on\)/); + expect(test).toMatch(/ASSIGNMENT === "1"/); + }); + + it.each(names)("%s walks red to green, each step with an Expect line", (name) => { + const steps = body(brief(name), "Walkthrough") + .split(/^\d+\. /m) + .slice(1); + expect(steps.length).toBeGreaterThanOrEqual(3); + for (const step of steps) expect(step).toMatch(/\n\s+Expect: .+/); + expect(steps[0]).toContain(`tests/assignments/${name}.test.ts`); + expect(steps.at(-1)).toContain(`tests/assignments/${name}.test.ts`); + }); + + it.each(names)("%s asks a question and a Principal one", (name) => { + const discussion = body(brief(name), "Discussion"); + expect(discussion).toMatch(/^[^-].*\?$/m); + expect(discussion).toMatch(/^- \*\*Principal:\*\* .+\?$/m); + }); + + it.each(names)("%s files itself from this file", (name) => { + expect(body(brief(name), "File it")).toContain(`--body-file assignments/${name}.md`); + }); +}); + +describe("assignments/README.md", () => { + it("has one row per brief, linked, in the same set", () => { + const readme = readFileSync("assignments/README.md", "utf8"); + const linked = [...readme.matchAll(/^\| \[[^\]]+\]\(([a-z-]+)\.md\) \|/gm)].map((m) => m[1]); + expect([...linked].sort()).toEqual(names); + }); +}); + +describe("issue templates", () => { + it("hold no assignment, because assignments are briefs", () => { + const left = readdirSync(".github/ISSUE_TEMPLATE").filter((f) => f.startsWith("assignment-")); + expect(left).toEqual([]); }); }); From 52b495f0e18069446a0f702084081f6f182c2d81 Mon Sep 17 00:00:00 2001 From: Param Harrison <paramanantham@live.com> Date: Sun, 1 Nov 2026 17:03:35 +0200 Subject: [PATCH 02/12] Docs are held to the code: workflow table, paths, symbols and links each have a test The F8 table must match .factory/workflows, every path and make target a doc names must exist, every camelCase symbol must be in the code, and every relative link and anchor must resolve. Fixes found on the way: f6 cited a run.log that is not there, f7 wrote engine/retro.ts for factory/engine/retro.ts, context.md named a function Claude is only expected to invent. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --- docs/factory/f6-deploy.md | 2 +- docs/factory/f7-maintain.md | 2 +- docs/levels/context.md | 4 +- levels.json | 8 ++- tests/structure/doc-links.test.ts | 61 +++++++++++++++++++++++ tests/structure/doc-paths.allow.ts | 34 +++++++++++++ tests/structure/doc-paths.test.ts | 68 ++++++++++++++++++++++++++ tests/structure/doc-symbols.test.ts | 50 +++++++++++++++++++ tests/structure/lab-references.test.ts | 37 +------------- tests/structure/repo-roots.ts | 37 ++++++++++++++ tests/structure/workflows-doc.test.ts | 18 +++++++ 11 files changed, 281 insertions(+), 40 deletions(-) create mode 100644 tests/structure/doc-links.test.ts create mode 100644 tests/structure/doc-paths.allow.ts create mode 100644 tests/structure/doc-paths.test.ts create mode 100644 tests/structure/doc-symbols.test.ts create mode 100644 tests/structure/repo-roots.ts create mode 100644 tests/structure/workflows-doc.test.ts diff --git a/docs/factory/f6-deploy.md b/docs/factory/f6-deploy.md index 67258ad..709d486 100644 --- a/docs/factory/f6-deploy.md +++ b/docs/factory/f6-deploy.md @@ -61,7 +61,7 @@ Make them fail: in `risk.ts`, change the filter `a.kind === "human"` to accept e ## See it on a real run -`specs/35/ci.md` is the record a real `ci` stage wrote on run 35. It states that no CI job log was reachable because `gh` was not authenticated in that sandbox, quotes the only recorded failure from `specs/35/run.log` (`check plan-drift FAIL: files outside plan.md: api/src/agent/template-eval.ts`), says the fix was listing the file in `specs/35/plan.md`, and lists two local test failures it attributes to the sandbox. Read it as an honest example: the stage reported what it could not see instead of claiming green. +`specs/35/ci.md` is the record a real `ci` stage wrote on run 35. It states that no CI job log was reachable because `gh` was not authenticated in that sandbox, quotes the only recorded failure (`check plan-drift FAIL: files outside plan.md: api/src/agent/template-eval.ts`), says the fix was listing the file in `specs/35/plan.md`, and lists two local test failures it attributes to the sandbox. Read it as an honest example: the stage reported what it could not see instead of claiming green. No production release has run from this repo, since there is no deploy target. diff --git a/docs/factory/f7-maintain.md b/docs/factory/f7-maintain.md index a0185f9..355f5f9 100644 --- a/docs/factory/f7-maintain.md +++ b/docs/factory/f7-maintain.md @@ -53,7 +53,7 @@ Four measures and where they could come from. Deployment frequency and lead time Tests in `factory/engine/review.test.ts`: `a 3 sigma breach writes an intent` and `a value inside the band does nothing`. In `bands.ts`, change `z >= 3 ? 3` to `z >= 4 ? 3` and the first test fails, since a breach of 30 against a history near 10 no longer reaches the intent action. -Retro: `reverts a lesson that lowered the score` in `factory/engine/retro.test.ts` feeds scores 0.9 then 0.7 and expects `reverted`. Change `delta < 0` to `delta < -1` in `engine/retro.ts` and it fails. `treats a missing SCORE line as a broken eval` fails if a silent eval counts as zero. +Retro: `reverts a lesson that lowered the score` in `factory/engine/retro.test.ts` feeds scores 0.9 then 0.7 and expects `reverted`. Change `delta < 0` to `delta < -1` in `factory/engine/retro.ts` and it fails. `treats a missing SCORE line as a broken eval` fails if a silent eval counts as zero. Metrics: `counts a run first-pass only when it finished with no repeat round` in `metrics.test.ts`. diff --git a/docs/levels/context.md b/docs/levels/context.md index 1c7521a..0281894 100644 --- a/docs/levels/context.md +++ b/docs/levels/context.md @@ -23,11 +23,11 @@ Delete `AGENTS.md` and `.claude/rules/` on a scratch branch and run the prompt. ## With -Run the same prompt. Claude adds `formatBand` to `shared/src/bands.ts` with a test beside it. Run `/context` while editing in `web/` and see the web rule loaded, the api rule not. +Run the same prompt. Claude adds the helper under `shared/src/` with a test beside it. Run `/context` while editing in `web/` and see the web rule loaded, the api rule not. ## Run it -1. Type `claude`. Say "add a helper that formats grade names". Expect `formatBand` in `shared`. +1. Type `claude`. Say "add a helper that formats grade names". Expect the new helper and its test in `shared`, not in `api` or `web`. 2. Type `/context`. Expect `AGENTS.md` and one rule listed. 3. Type `/compact keep file paths and decisions`. Expect a short summary that keeps both. diff --git a/levels.json b/levels.json index bcbf937..d587d12 100644 --- a/levels.json +++ b/levels.json @@ -94,7 +94,12 @@ "tests/structure/labs.test.ts", "tests/structure/lab-artifacts.ts", "tests/structure/lab-solutions.test.ts", - "tests/structure/lab-references.test.ts" + "tests/structure/lab-references.test.ts", + "tests/structure/repo-roots.ts", + "tests/structure/doc-paths.test.ts", + "tests/structure/doc-paths.allow.ts", + "tests/structure/doc-symbols.test.ts", + "tests/structure/doc-links.test.ts" ], "execution": [ ".claude/agents/**", @@ -209,6 +214,7 @@ ".factory/workflows/**", "tests/structure/f5-review-doc.test.ts", "tests/structure/assignments.test.ts", + "tests/structure/workflows-doc.test.ts", "tests/assignments/**", "assignments/**", ".factory/bands.yaml", diff --git a/tests/structure/doc-links.test.ts b/tests/structure/doc-links.test.ts new file mode 100644 index 0000000..31d1440 --- /dev/null +++ b/tests/structure/doc-links.test.ts @@ -0,0 +1,61 @@ +import { existsSync, readdirSync, readFileSync, statSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { describe, expect, it } from "vitest"; + +// Every relative markdown link must reach a file, and every #anchor a heading in that file. +const walk = (dir: string): string[] => + readdirSync(dir).flatMap((name) => { + const p = join(dir, name); + return statSync(p).isDirectory() ? walk(p) : p.endsWith(".md") ? [p] : []; + }); +const docs = ["README.md", ...walk("docs"), ...walk("assignments")]; + +const withoutFences = (text: string) => text.replace(/^```[\s\S]*?^```/gm, ""); + +// GitHub's slug: lowercase, spaces to dashes, everything but letters, digits, dashes and underscores dropped. +const slug = (heading: string) => + heading + .toLowerCase() + .replace(/[`*_~]/g, (c) => (c === "_" ? "_" : "")) + .replace(/[^\w\- ]/g, "") + .replace(/ /g, "-"); + +const anchorsOf = (file: string): Set<string> => + new Set( + [...withoutFences(readFileSync(file, "utf8")).matchAll(/^#{1,6} +(.+?) *#*$/gm)].map((m) => + slug(m[1] ?? ""), + ), + ); + +function brokenLinks(file: string, text: string): string[] { + const bad: string[] = []; + for (const [, target = ""] of withoutFences(text).matchAll(/\]\(([^)\s]+)(?:\s+"[^"]*")?\)/g)) { + if (/^(https?:|mailto:)/.test(target)) continue; + const [path = "", anchor] = target.split("#"); + const resolved = path === "" ? file : join(dirname(file), path); + if (!existsSync(resolved)) bad.push(`${file}: ${target}`); + else if (anchor && resolved.endsWith(".md") && !anchorsOf(resolved).has(anchor)) { + bad.push(`${file}: ${target}`); + } + } + return bad; +} + +describe("doc links", () => { + it("reach a file and, when they name one, a heading in it", () => { + expect(docs.flatMap((f) => brokenLinks(f, readFileSync(f, "utf8")))).toEqual([]); + }); + + it("catches a missing file and a missing anchor", () => { + const text = "[a](f99.md) and [b](f6-deploy.md#nope) and [c](f6-deploy.md)."; + expect(brokenLinks("docs/factory/x.md", text)).toEqual([ + "docs/factory/x.md: f99.md", + "docs/factory/x.md: f6-deploy.md#nope", + ]); + }); + + it("slugs a heading the way GitHub does", () => { + expect(slug("Where to go after this lane")).toBe("where-to-go-after-this-lane"); + expect(slug("F6: Deploy (the `release` stage)")).toBe("f6-deploy-the-release-stage"); + }); +}); diff --git a/tests/structure/doc-paths.allow.ts b/tests/structure/doc-paths.allow.ts new file mode 100644 index 0000000..1be2462 --- /dev/null +++ b/tests/structure/doc-paths.allow.ts @@ -0,0 +1,34 @@ +// Paths the docs name that are not in the repo, each with the reason. A path that now exists is stale. +// A `make <target>` entry is stale once the Makefile defines the target. +export type Reason = + | "learner-made" + | "assignment-made" + | "gitignored" + | "example" + | "generated" + | "external"; + +export const ALLOWED: { path: string; reason: Reason }[] = [ + { path: "api/src/agent/canary.ts", reason: "assignment-made" }, + { path: "shared/src/arb-worksheet.ts", reason: "assignment-made" }, + { path: ".claude/hooks/pretooluse-judge.mjs", reason: "assignment-made" }, + { path: "factory/label-machine.ts", reason: "assignment-made" }, + { path: ".claude/agents/performance-reviewer.md", reason: "assignment-made" }, + { path: "api/src/templates/db-template-store.ts", reason: "assignment-made" }, + { path: "fixtures/incidents/breach.json", reason: "assignment-made" }, + { path: "shared/src/render-video.ts", reason: "assignment-made" }, + { path: ".claude/settings.local.json", reason: "gitignored" }, + { path: ".claude/logs/teams.log", reason: "gitignored" }, + { path: ".factory/runs/", reason: "gitignored" }, + { path: ".factory/series.json", reason: "generated" }, + { path: ".factory/holdout/", reason: "generated" }, + { path: "api/src/a.test.ts", reason: "example" }, + { path: "api/src/x.ts", reason: "example" }, + { path: "docs/a.md", reason: "example" }, + { path: ".factory/workflows/typo.yaml", reason: "example" }, + { path: "specs/1/", reason: "example" }, + { path: "specs/25/golden/__snapshots__/adult-learners.html", reason: "example" }, + { path: "make deploy", reason: "example" }, + { path: "fixtures/take-home/eval.json", reason: "learner-made" }, + { path: "api/src/scratch-greet.ts", reason: "learner-made" }, +]; diff --git a/tests/structure/doc-paths.test.ts b/tests/structure/doc-paths.test.ts new file mode 100644 index 0000000..cf03a11 --- /dev/null +++ b/tests/structure/doc-paths.test.ts @@ -0,0 +1,68 @@ +import { execFileSync } from "node:child_process"; +import { existsSync, readdirSync, readFileSync, statSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { ALLOWED } from "./doc-paths.allow.ts"; +import { present, ROOTS, targetsOf } from "./repo-roots.ts"; + +// A doc that names a file or a make target that is not there is a claim the code does not keep. +const walk = (dir: string): string[] => + readdirSync(dir).flatMap((name) => { + const p = join(dir, name); + return statSync(p).isDirectory() ? walk(p) : p.endsWith(".md") ? [p] : []; + }); +const docs = ["README.md", ...walk("docs"), ...walk("assignments")]; +const targets = targetsOf(readFileSync("Makefile", "utf8")); +// `factory/engine/x.ts` written as `engine/x.ts` is the slip this has caught before. +const BARE = /^(engine|src)\/[\w./-]+\.\w+$/; +const allowed = new Set(ALLOWED.map((a) => a.path)); + +function problems(file: string, text: string): string[] { + const bad: string[] = []; + for (const [, token = ""] of text.matchAll(/`([^`\n]+)`/g)) { + for (const [, target = ""] of token.matchAll(/\bmake\s+([a-z][\w-]*)/g)) { + if (!targets.has(target) && !allowed.has(`make ${target}`)) + bad.push(`${file}: make ${target}`); + } + const words = token.split(/\s+/); + for (const [i, word] of words.entries()) { + if (words[i - 1] === "-b") continue; // a new branch name, not a path + const path = word.replace(/^["']|["'.,;:)]+$/g, "").replace(/(:\d+)+$/, ""); + if (BARE.test(path)) bad.push(`${file}: ${path} (missing its root)`); + if (!ROOTS.some((r) => path.startsWith(r)) || /[*<>{}$]/.test(path)) continue; + if (path.startsWith(".labs/work/") || allowed.has(path) || allowed.has(`${path}/`)) continue; + if (!present(path) && !present(join(dirname(file), path))) bad.push(`${file}: ${path}`); + } + } + return bad; +} + +describe("doc paths", () => { + it("names only make targets and paths that exist", () => { + const bad = docs.flatMap((f) => problems(f, readFileSync(f, "utf8"))); + expect(bad).toEqual([]); + }); + + it("allows nothing that now exists, and every reason is honest", () => { + const stale = ALLOWED.filter((a) => + a.path.startsWith("make ") ? targets.has(a.path.slice(5)) : existsSync(a.path), + ).map((a) => `${a.path}: exists now`); + const ignored = ALLOWED.filter((a) => a.reason === "gitignored").filter((a) => { + try { + execFileSync("git", ["check-ignore", "--no-index", "-q", a.path]); + return false; + } catch { + return true; + } + }); + expect([...stale, ...ignored.map((a) => `${a.path}: not in .gitignore`)]).toEqual([]); + }); + + it("catches a missing path and a missing make target", () => { + const text = "Open `factory/engine/nope.ts` then run `make prove-all`."; + expect(problems("docs/x.md", text)).toEqual([ + "docs/x.md: factory/engine/nope.ts", + "docs/x.md: make prove-all", + ]); + }); +}); diff --git a/tests/structure/doc-symbols.test.ts b/tests/structure/doc-symbols.test.ts new file mode 100644 index 0000000..ab90f10 --- /dev/null +++ b/tests/structure/doc-symbols.test.ts @@ -0,0 +1,50 @@ +import { execFileSync } from "node:child_process"; +import { readdirSync, readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; + +// A doc that names a function must name one the code has. Only camelCase words are checked, +// since those are the ones a rename can strand. +const DIRS = ["api", "shared", "web", "factory", "scripts", "tests", ".claude", "plugins"]; +const NOT_OURS = new Set(["apiKeyHelper"]); // a Claude Code setting, not a symbol in this repo + +const docs = ["docs/factory", "docs/levels"].flatMap((dir) => + readdirSync(dir) + .filter((f) => f.endsWith(".md")) + .map((f) => `${dir}/${f}`), +); + +const symbolsIn = (text: string): string[] => + [...text.matchAll(/`([a-z]+[A-Z]\w*)(\(\))?`/g)].map((m) => m[1] ?? ""); + +const known = (names: string[]): Set<string> => { + const hits = new Set<string>(); + for (const name of names) { + try { + execFileSync("git", ["grep", "-qw", "-e", name, "--", ...DIRS, ...docs.map((d) => `:!${d}`)]); + hits.add(name); + } catch { + // git grep exits 1 when nothing matches + } + } + return hits; +}; + +describe("doc symbols", () => { + it("names only camelCase symbols the code has", () => { + const named = new Map(docs.map((d) => [d, [...new Set(symbolsIn(readFileSync(d, "utf8")))]])); + const all = [...new Set([...named.values()].flat())].filter((s) => !NOT_OURS.has(s)); + const found = known(all); + const missing = [...named].flatMap(([doc, names]) => + names.filter((n) => !NOT_OURS.has(n) && !found.has(n)).map((n) => `${doc}: ${n}`), + ); + expect(missing).toEqual([]); + }); + + it("reads a call with brackets as the same symbol", () => { + expect(symbolsIn("Call `classify()` and `plain` and `x-Y`.")).toEqual([]); + expect(symbolsIn("Call `classifyTier()` and `approvalsMet`.")).toEqual([ + "classifyTier", + "approvalsMet", + ]); + }); +}); diff --git a/tests/structure/lab-references.test.ts b/tests/structure/lab-references.test.ts index 3f8658f..9310a8c 100644 --- a/tests/structure/lab-references.test.ts +++ b/tests/structure/lab-references.test.ts @@ -1,33 +1,10 @@ -import { existsSync, readFileSync } from "node:fs"; +import { readFileSync } from "node:fs"; import { describe, expect, it } from "vitest"; -import levels from "../../levels.json" with { type: "json" }; import { type Lab, loadLabs } from "../../scripts/labs/catalog.ts"; +import { present, ROOTS, targetsOf } from "./repo-roots.ts"; // The gate swaps Claude steps for their output, so it never reads step text. This reads it: // every `make <target>` and every repo path a learner is told to use must exist. -// Paths under .labs/work/ are made by the learner, so they are not in ROOTS and never checked. -const ROOTS = [ - "api/", - "docs/", - "factory/", - "features/", - "fixtures/", - "lab/", - "labs/", - "plugins/", - "scripts/", - "shared/", - "specs/", - "tests/", - "web/", - ".claude/", - ".factory/", - ".github/", -]; - -const targetsOf = (makefile: string): Set<string> => - new Set([...makefile.matchAll(/^([a-z][\w-]*):/gm)].map((m) => m[1] ?? "")); - function badReferences( lab: Lab, targets: Set<string>, @@ -51,16 +28,6 @@ function badReferences( return bad; } -// A level snapshot holds only its own files, so a path counts when some level owns it. -const owned = (Object.values(levels) as string[][]).flat(); -const isOwned = (f: string) => { - const dir = `${f.replace(/\/$/, "")}/`; - return owned.some( - (g) => g === f || (g.endsWith("/**") && f.startsWith(g.slice(0, -2))) || g.startsWith(dir), - ); -}; -const present = (f: string) => existsSync(f) || isOwned(f); - const targets = targetsOf(readFileSync("Makefile", "utf8")); const labs = loadLabs("labs"); diff --git a/tests/structure/repo-roots.ts b/tests/structure/repo-roots.ts new file mode 100644 index 0000000..6aaf546 --- /dev/null +++ b/tests/structure/repo-roots.ts @@ -0,0 +1,37 @@ +import { existsSync } from "node:fs"; +import levels from "../../levels.json" with { type: "json" }; + +// Shared by the tests that read prose for repo paths and `make` targets. +// Paths under .labs/work/ are made by the learner, so they are not in ROOTS and never checked. +export const ROOTS = [ + "api/", + "docs/", + "factory/", + "features/", + "fixtures/", + "lab/", + "labs/", + "plugins/", + "scripts/", + "shared/", + "specs/", + "tests/", + "web/", + ".claude/", + ".factory/", + ".github/", +]; + +export const targetsOf = (makefile: string): Set<string> => + new Set([...makefile.matchAll(/^([a-z][\w-]*):/gm)].map((m) => m[1] ?? "")); + +// A level snapshot holds only its own files, so there a path counts when some level owns it. +// The full repo has .git and is checked strictly: an owned glob does not make a missing file real. +const owned = (Object.values(levels) as string[][]).flat(); +const isOwned = (f: string) => { + const dir = `${f.replace(/\/$/, "")}/`; + return owned.some( + (g) => g === f || (g.endsWith("/**") && f.startsWith(g.slice(0, -2))) || g.startsWith(dir), + ); +}; +export const present = (f: string) => existsSync(f) || (!existsSync(".git") && isOwned(f)); diff --git a/tests/structure/workflows-doc.test.ts b/tests/structure/workflows-doc.test.ts new file mode 100644 index 0000000..cd6939e --- /dev/null +++ b/tests/structure/workflows-doc.test.ts @@ -0,0 +1,18 @@ +import { readdirSync, readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; + +// The F8 table maps a kind of task to a workflow. This holds it to the yaml files that exist. +const doc = readFileSync("docs/factory/f8-workflows.md", "utf8"); + +const listed = [...doc.matchAll(/^\| [^|]+ \| `([a-z-]+)` +\|/gm)].map((m) => m[1]).sort(); +const defined = readdirSync(".factory/workflows") + .filter((f) => f.endsWith(".yaml")) + .map((f) => f.replace(/\.yaml$/, "")) + .sort(); + +describe("f8 workflow table", () => { + it("lists every workflow file, and nothing that has no file", () => { + expect(defined.length).toBeGreaterThan(0); + expect(listed).toEqual(defined); + }); +}); From 1515898cde7c60cfcfdba5a0d8fbd6ba9b631666 Mon Sep 17 00:00:00 2001 From: Param Harrison <paramanantham@live.com> Date: Sun, 1 Nov 2026 17:04:57 +0200 Subject: [PATCH 03/12] Lab proofs run one lane per test file Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --- levels.json | 6 ++ tests/structure/lab-harness.ts | 98 +++++++++++++++++++ .../structure/lab-solutions-advanced.test.ts | 8 ++ tests/structure/lab-solutions-daily.test.ts | 3 + tests/structure/lab-solutions-factory.test.ts | 3 + tests/structure/lab-solutions-start.test.ts | 3 + tests/structure/lab-solutions-team.test.ts | 3 + tests/structure/lab-solutions.test.ts | 96 ++---------------- 8 files changed, 133 insertions(+), 87 deletions(-) create mode 100644 tests/structure/lab-harness.ts create mode 100644 tests/structure/lab-solutions-advanced.test.ts create mode 100644 tests/structure/lab-solutions-daily.test.ts create mode 100644 tests/structure/lab-solutions-factory.test.ts create mode 100644 tests/structure/lab-solutions-start.test.ts create mode 100644 tests/structure/lab-solutions-team.test.ts diff --git a/levels.json b/levels.json index d587d12..baa9119 100644 --- a/levels.json +++ b/levels.json @@ -94,6 +94,12 @@ "tests/structure/labs.test.ts", "tests/structure/lab-artifacts.ts", "tests/structure/lab-solutions.test.ts", + "tests/structure/lab-harness.ts", + "tests/structure/lab-solutions-start.test.ts", + "tests/structure/lab-solutions-daily.test.ts", + "tests/structure/lab-solutions-team.test.ts", + "tests/structure/lab-solutions-factory.test.ts", + "tests/structure/lab-solutions-advanced.test.ts", "tests/structure/lab-references.test.ts", "tests/structure/repo-roots.ts", "tests/structure/doc-paths.test.ts", diff --git a/tests/structure/lab-harness.ts b/tests/structure/lab-harness.ts new file mode 100644 index 0000000..e5d8443 --- /dev/null +++ b/tests/structure/lab-harness.ts @@ -0,0 +1,98 @@ +import { spawnSync } from "node:child_process"; +import { + copyFileSync, + existsSync, + mkdirSync, + mkdtempSync, + readdirSync, + symlinkSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { describe, expect, it } from "vitest"; +import { type Lane, loadLabs } from "../../scripts/labs/catalog.ts"; +import { LAZY, NEEDS, SOLUTIONS, sh } from "./lab-artifacts.ts"; + +// Shared by the five lab-solutions-<lane> tests, so a lane can run, fail and be read alone. +const REPO = resolve("."); +// Proofs read this repo (its folders, shared/ files, make targets), so each run gets a scratch +// folder that links to it. The learner's work lands in the scratch folder, never in the repo. +const SKIP = new Set([".git", ".labs", ".gitignore", "CLAUDE.local.md"]); + +export function scratchRepo(): string { + const dir = mkdtempSync(join(tmpdir(), "lab-proof-")); + for (const name of readdirSync(REPO)) { + if (!SKIP.has(name)) symlinkSync(join(REPO, name), join(dir, name)); + } + copyFileSync(join(REPO, ".gitignore"), join(dir, ".gitignore")); + // A real repo with one commit, so labs that run git have something true to point at. + const git = (...args: string[]) => spawnSync("git", args, { cwd: dir }); + git("init", "-q"); + git( + "-c", + "user.name=lab", + "-c", + "user.email=lab@example.com", + "commit", + "-q", + "--allow-empty", + "-m", + "init", + ); + return dir; +} + +export function inScratchRepo(setup: string | undefined, proof: string) { + const dir = scratchRepo(); + if (setup) { + writeFileSync(join(dir, "..", `${dir.split("/").pop()}-setup.sh`), setup); + mkdirSync(join(dir, ".labs"), { recursive: true }); + spawnSync("sh", [join(dir, "..", `${dir.split("/").pop()}-setup.sh`)], { cwd: dir }); + } + return spawnSync("sh", ["-c", proof], { cwd: dir, encoding: "utf8" }).status; +} + +// Every proof item of a lane is run three ways: red alone with no work, red on a lazy artifact, +// and (when the files it needs are here) green once the learner's steps are done. +export function proveLane(lane: Lane) { + const labs = loadLabs("labs").filter((l) => l.lane === lane); + const reachable = (id: string) => (NEEDS[id] ?? []).every((f) => existsSync(join(REPO, f))); + const items = labs.flatMap((l) => l.proof.map((item, n) => ({ lab: l, item, n: n + 1 }))); + + describe(`${lane} lab proofs`, () => { + // A red item names one missing part, so every item must be red alone in an empty folder. + it.each(items)( + "$lab.id item $n is red with no work: $item.text", + { timeout: 240_000 }, + ({ item }) => { + expect(inScratchRepo(undefined, item.command)).not.toBe(0); + }, + ); + + it.each(labs)( + "$id is red on a lazy artifact with the right shape", + { timeout: 240_000 }, + (lab) => { + const codes = lab.proof.map((item) => inScratchRepo(LAZY[lab.id], item.command)); + expect(codes.some((code) => code !== 0)).toBe(true); + }, + ); + + it.each(items.filter(({ lab }) => reachable(lab.id)))( + "$lab.id item $n is green once the steps are done: $item.text", + { timeout: 240_000 }, + ({ lab, item }) => { + expect(inScratchRepo(SOLUTIONS[lab.id], item.command)).toBe(0); + }, + ); + }); +} + +// The staleness case: a saved file goes out of date when the deny list changes. +export function staleCloudGuards() { + const lab = loadLabs("labs").find((l) => l.id === "advanced-05"); + const changed = sh(SOLUTIONS["advanced-05"] ?? "", "echo stale > .labs/work/cloud-guards.txt"); + expect(lab, "advanced-05 is missing").toBeDefined(); + return inScratchRepo(changed, lab?.proof[0]?.command ?? "false"); +} diff --git a/tests/structure/lab-solutions-advanced.test.ts b/tests/structure/lab-solutions-advanced.test.ts new file mode 100644 index 0000000..4a1126e --- /dev/null +++ b/tests/structure/lab-solutions-advanced.test.ts @@ -0,0 +1,8 @@ +import { expect, it } from "vitest"; +import { proveLane, staleCloudGuards } from "./lab-harness.ts"; + +proveLane("advanced"); + +it("advanced-05 goes red when the deny list changes after cloud-guards.txt was saved", () => { + expect(staleCloudGuards()).not.toBe(0); +}); diff --git a/tests/structure/lab-solutions-daily.test.ts b/tests/structure/lab-solutions-daily.test.ts new file mode 100644 index 0000000..de75641 --- /dev/null +++ b/tests/structure/lab-solutions-daily.test.ts @@ -0,0 +1,3 @@ +import { proveLane } from "./lab-harness.ts"; + +proveLane("daily"); diff --git a/tests/structure/lab-solutions-factory.test.ts b/tests/structure/lab-solutions-factory.test.ts new file mode 100644 index 0000000..1c9d4cf --- /dev/null +++ b/tests/structure/lab-solutions-factory.test.ts @@ -0,0 +1,3 @@ +import { proveLane } from "./lab-harness.ts"; + +proveLane("factory"); diff --git a/tests/structure/lab-solutions-start.test.ts b/tests/structure/lab-solutions-start.test.ts new file mode 100644 index 0000000..0e48f53 --- /dev/null +++ b/tests/structure/lab-solutions-start.test.ts @@ -0,0 +1,3 @@ +import { proveLane } from "./lab-harness.ts"; + +proveLane("start"); diff --git a/tests/structure/lab-solutions-team.test.ts b/tests/structure/lab-solutions-team.test.ts new file mode 100644 index 0000000..2e53ad3 --- /dev/null +++ b/tests/structure/lab-solutions-team.test.ts @@ -0,0 +1,3 @@ +import { proveLane } from "./lab-harness.ts"; + +proveLane("team"); diff --git a/tests/structure/lab-solutions.test.ts b/tests/structure/lab-solutions.test.ts index 42f844c..bf841d5 100644 --- a/tests/structure/lab-solutions.test.ts +++ b/tests/structure/lab-solutions.test.ts @@ -1,98 +1,20 @@ -import { spawnSync } from "node:child_process"; -import { - copyFileSync, - existsSync, - mkdirSync, - mkdtempSync, - readdirSync, - symlinkSync, - writeFileSync, -} from "node:fs"; -import { tmpdir } from "node:os"; -import { join, resolve } from "node:path"; import { describe, expect, it } from "vitest"; import { loadLabs } from "../../scripts/labs/catalog.ts"; -import { LAZY, NEEDS, SOLUTIONS, sh } from "./lab-artifacts.ts"; +import { LAZY, NEEDS, SOLUTIONS } from "./lab-artifacts.ts"; -const REPO = resolve("."); -// Proofs read this repo (its folders, shared/ files, make targets), so each run gets a scratch -// folder that links to it. The learner's work lands in the scratch folder, never in the repo. -const SKIP = new Set([".git", ".labs", ".gitignore", "CLAUDE.local.md"]); - -function scratchRepo(): string { - const dir = mkdtempSync(join(tmpdir(), "lab-proof-")); - for (const name of readdirSync(REPO)) { - if (!SKIP.has(name)) symlinkSync(join(REPO, name), join(dir, name)); - } - copyFileSync(join(REPO, ".gitignore"), join(dir, ".gitignore")); - // A real repo with one commit, so labs that run git have something true to point at. - const git = (...args: string[]) => spawnSync("git", args, { cwd: dir }); - git("init", "-q"); - git( - "-c", - "user.name=lab", - "-c", - "user.email=lab@example.com", - "commit", - "-q", - "--allow-empty", - "-m", - "init", - ); - return dir; -} - -function inScratchRepo(setup: string | undefined, proof: string) { - const dir = scratchRepo(); - if (setup) { - writeFileSync(join(dir, "..", `${dir.split("/").pop()}-setup.sh`), setup); - mkdirSync(join(dir, ".labs"), { recursive: true }); - spawnSync("sh", [join(dir, "..", `${dir.split("/").pop()}-setup.sh`)], { cwd: dir }); - } - return spawnSync("sh", ["-c", proof], { cwd: dir, encoding: "utf8" }).status; -} - -describe("lab proofs", () => { - const labs = loadLabs("labs"); - const ids = labs.map((l) => l.id).sort(); - const reachable = (id: string) => (NEEDS[id] ?? []).every((f) => existsSync(join(REPO, f))); - const items = labs.flatMap((lab) => lab.proof.map((item, n) => ({ lab, item, n: n + 1 }))); +// The proofs themselves run per lane in lab-solutions-<lane>.test.ts. This holds the artifact +// tables to the lab catalog, so a lab cannot lose its solution and a deleted lab cannot leave one. +describe("lab proof artifacts", () => { + const ids = loadLabs("labs") + .map((l) => l.id) + .sort(); it("has a solution and a lazy artifact for every lab and none extra", () => { expect(Object.keys(SOLUTIONS).sort()).toEqual(ids); expect(Object.keys(LAZY).sort()).toEqual(ids); }); - // A red item names one missing part, so every item must be red alone in an empty folder. - it.each(items)( - "$lab.id item $n is red with no work: $item.text", - { timeout: 240_000 }, - ({ item }) => { - expect(inScratchRepo(undefined, item.command)).not.toBe(0); - }, - ); - - it.each(labs)( - "$id is red on a lazy artifact with the right shape", - { timeout: 240_000 }, - (lab) => { - const codes = lab.proof.map((item) => inScratchRepo(LAZY[lab.id], item.command)); - expect(codes.some((code) => code !== 0)).toBe(true); - }, - ); - - it.each(items.filter(({ lab }) => reachable(lab.id)))( - "$lab.id item $n is green once the steps are done: $item.text", - { timeout: 240_000 }, - ({ lab, item }) => { - expect(inScratchRepo(SOLUTIONS[lab.id], item.command)).toBe(0); - }, - ); - - it("advanced-05 goes red when the deny list changes after cloud-guards.txt was saved", () => { - const lab = labs.find((l) => l.id === "advanced-05"); - const changed = sh(SOLUTIONS["advanced-05"] ?? "", "echo stale > .labs/work/cloud-guards.txt"); - expect(lab, "advanced-05 is missing").toBeDefined(); - expect(inScratchRepo(changed, lab?.proof[0]?.command ?? "false")).not.toBe(0); + it("needs files only for labs that exist", () => { + expect(Object.keys(NEEDS).filter((id) => !ids.includes(id))).toEqual([]); }); }); From 4a496612b98799d8f0cb0ad9af6e3cd5494d4241 Mon Sep 17 00:00:00 2001 From: Param Harrison <paramanantham@live.com> Date: Sun, 1 Nov 2026 17:09:41 +0200 Subject: [PATCH 04/12] Tricks for every start and daily lab, with a generated cheat sheet Adds a Trick to start-03 and daily-01 to daily-04, each checked against claude --help or the lab itself, and scripts/labs/cheat-sheet.ts that renders them to docs/labs/cheat-sheet.md. A test holds the file to the render and requires a trick on every start and daily lab. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --- docs/labs/README.md | 2 ++ docs/labs/cheat-sheet.md | 17 +++++++++++++++++ labs/daily/01-resume-a-session.md | 4 ++++ labs/daily/02-path-scoped-rule.md | 4 ++++ labs/daily/03-first-skill.md | 4 ++++ labs/daily/04-first-hook.md | 4 ++++ labs/start/03-plan-first.md | 4 ++++ scripts/labs/cheat-sheet.test.ts | 21 +++++++++++++++++++++ scripts/labs/cheat-sheet.ts | 24 ++++++++++++++++++++++++ 9 files changed, 84 insertions(+) create mode 100644 docs/labs/cheat-sheet.md create mode 100644 scripts/labs/cheat-sheet.test.ts create mode 100644 scripts/labs/cheat-sheet.ts diff --git a/docs/labs/README.md b/docs/labs/README.md index fcc3adc..b9239ca 100644 --- a/docs/labs/README.md +++ b/docs/labs/README.md @@ -9,6 +9,8 @@ Two workshops share this folder. Pick yours once with `make lab PATH_ID=pro` or Each lane has a walkthrough that follows every step with the command, what you should see and why: [start](walkthrough-start.md), [daily](walkthrough-daily.md), [team](walkthrough-team.md), [advanced](walkthrough-advanced.md), [factory](walkthrough-factory.md). +The one-line tricks from every lab sit on one page: [cheat sheet](cheat-sheet.md). + ## Help fades as you go Every lab declares `**Help:** guided`, `hinted` or `goal`, and the path gets harder on purpose. diff --git a/docs/labs/cheat-sheet.md b/docs/labs/cheat-sheet.md new file mode 100644 index 0000000..564de6b --- /dev/null +++ b/docs/labs/cheat-sheet.md @@ -0,0 +1,17 @@ +# Cheat sheet + +Generated from the `## Trick` of each lab. Regenerate with `npx tsx scripts/labs/cheat-sheet.ts > docs/labs/cheat-sheet.md`. + +## start + +- **start-01** Ask Claude about the repo: Add `--output-format json` to `claude -p` and the answer comes back as one JSON result that a script can parse. +- **start-02** Run the gate yourself: Type `/gate` in Claude Code. The project command in `.claude/commands/gate.md` runs `make check` and reports only what failed. +- **start-03** Plan before you edit: Add `--permission-mode plan` to `claude -p` for any change you want to read before it happens. Claude plans and edits nothing. +- **start-04** Teach Claude a personal habit: Type `/memory` in Claude Code to see which instruction files loaded, so you can confirm your `CLAUDE.local.md` came in. + +## daily + +- **daily-01** Continue where you left off: Run `claude --continue` (or `-c`) to reopen the most recent conversation in this folder, or `claude --resume` to pick an older one. +- **daily-02** Write a rule that loads only where it applies: A rule file with no `paths:` list loads every session. Add the list and it costs context only when Claude works in matching files. +- **daily-03** Package a habit as a skill: Type `/` plus your skill name, such as `/commit-message`, to run it on purpose instead of waiting for Claude to match its description. +- **daily-04** Block a mistake with a hook: Start Claude Code with `claude --debug=hooks` to see each hook fire, what it received and what it returned. diff --git a/labs/daily/01-resume-a-session.md b/labs/daily/01-resume-a-session.md index f8c2794..f5e6f8a 100644 --- a/labs/daily/01-resume-a-session.md +++ b/labs/daily/01-resume-a-session.md @@ -57,6 +57,10 @@ Answer: No. Each call starts with an empty context, so --continue is what reopen 3. In the interactive app the same move is `claude --continue`, or `/resume` to choose from a list. 4. Commands, in order: `mkdir -p .labs/work`, `claude -p "Remember the word tangerine." --output-format json > .labs/work/session.json`, `claude -p --continue "What word did I ask you to remember?" > .labs/work/recall.md`, `cat .labs/work/recall.md`. +## Trick + +Run `claude --continue` (or `-c`) to reopen the most recent conversation in this folder, or `claude --resume` to pick an older one. + ## What you just learned A session is a saved conversation. Resume it instead of re-explaining the task. diff --git a/labs/daily/02-path-scoped-rule.md b/labs/daily/02-path-scoped-rule.md index e0af1b3..9dbe6be 100644 --- a/labs/daily/02-path-scoped-rule.md +++ b/labs/daily/02-path-scoped-rule.md @@ -65,6 +65,10 @@ Answer: Only when Claude reads a file whose path matches. Unmatched rules cost n 3. Write rules as checks you could fail, such as "an error to a customer never names the model". 4. Commands, in order: `ls .claude/rules`, then `cat` any file in it, `mkdir -p .labs/work/rules`. +## Trick + +A rule file with no `paths:` list loads every session. Add the list and it costs context only when Claude works in matching files. + ## What you just learned Rules cost context only when their paths match. Scope every rule to the files it governs. diff --git a/labs/daily/03-first-skill.md b/labs/daily/03-first-skill.md index 9263f3b..f173b81 100644 --- a/labs/daily/03-first-skill.md +++ b/labs/daily/03-first-skill.md @@ -64,6 +64,10 @@ Answer: Its description. Claude reads the descriptions first and opens the body 3. Good first steps: read the staged diff, write a subject under 60 characters, say why in the body. 4. Commands, in order: `ls .claude/skills`, then `cat` one SKILL.md, `mkdir -p .labs/work/skills/commit-message`. +## Trick + +Type `/` plus your skill name, such as `/commit-message`, to run it on purpose instead of waiting for Claude to match its description. + ## What you just learned A skill is loaded on demand. The description is the trigger, so write it as a situation. diff --git a/labs/daily/04-first-hook.md b/labs/daily/04-first-hook.md index 7046ee7..85f9cd3 100644 --- a/labs/daily/04-first-hook.md +++ b/labs/daily/04-first-hook.md @@ -63,6 +63,10 @@ Answer: Exit code 2 blocks the action. Any other path, including a clean exit 0, 3. Import `readFileSync` from `node:fs` and end with `process.exit(0)` for the allowed case. 4. Commands, in order: `cat .claude/hooks/pretooluse-file-guard.mjs`, `mkdir -p .labs/work/hooks`, `echo '{"tool_input":{"file_path":".env"}}' | node .labs/work/hooks/block-env.mjs; echo $?`. +## Trick + +Start Claude Code with `claude --debug=hooks` to see each hook fire, what it received and what it returned. + ## What you just learned A hook is a program, not a prompt. Exit code 2 blocks the action every time, even when the model would have forgotten the rule. diff --git a/labs/start/03-plan-first.md b/labs/start/03-plan-first.md index 0036bd0..c42a3dd 100644 --- a/labs/start/03-plan-first.md +++ b/labs/start/03-plan-first.md @@ -57,6 +57,10 @@ Answer: No. Plan mode is read-only, so Claude can read and search but the only t 2. Plan mode is read-only. Claude can look but not write. 3. Your prompt must ask for verification, or the plan may leave it out. +## Trick + +Add `--permission-mode plan` to `claude -p` for any change you want to read before it happens. Claude plans and edits nothing. + ## What you just learned Plan mode turns "do it" into "show me first". You review a plan for free instead of reviewing a diff. diff --git a/scripts/labs/cheat-sheet.test.ts b/scripts/labs/cheat-sheet.test.ts new file mode 100644 index 0000000..ddbade0 --- /dev/null +++ b/scripts/labs/cheat-sheet.test.ts @@ -0,0 +1,21 @@ +import { readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; +import { loadLabs } from "./catalog.ts"; +import { CHEAT_SHEET_COMMAND, renderCheatSheet } from "./cheat-sheet.ts"; + +const labs = loadLabs("labs"); + +describe("cheat sheet", () => { + it("is the render of the lab tricks, so no trick was edited without regenerating", () => { + expect(readFileSync("docs/labs/cheat-sheet.md", "utf8")).toBe(renderCheatSheet(labs)); + }); + + it("has a trick for every start and daily lab", () => { + const missing = labs.filter((l) => ["start", "daily"].includes(l.lane) && !l.trick); + expect(missing.map((l) => l.id)).toEqual([]); + }); + + it("names the command that regenerates it", () => { + expect(renderCheatSheet(labs)).toContain(CHEAT_SHEET_COMMAND); + }); +}); diff --git a/scripts/labs/cheat-sheet.ts b/scripts/labs/cheat-sheet.ts new file mode 100644 index 0000000..f20c7c0 --- /dev/null +++ b/scripts/labs/cheat-sheet.ts @@ -0,0 +1,24 @@ +import { pathToFileURL } from "node:url"; +import { LANES, type Lab, loadLabs } from "./catalog.ts"; + +export const CHEAT_SHEET_COMMAND = "npx tsx scripts/labs/cheat-sheet.ts > docs/labs/cheat-sheet.md"; + +// One line per lab that teaches a trick, grouped by lane. Generated, so a trick is edited in its lab. +export function renderCheatSheet(labs: Lab[]): string { + const sections = LANES.flatMap((lane) => { + const tricks = labs.filter((l) => l.lane === lane && l.trick); + if (tricks.length === 0) return []; + return [`## ${lane}`, "", ...tricks.map((l) => `- **${l.id}** ${l.title}: ${l.trick}`), ""]; + }); + return [ + "# Cheat sheet", + "", + `Generated from the \`## Trick\` of each lab. Regenerate with \`${CHEAT_SHEET_COMMAND}\`.`, + "", + ...sections, + ].join("\n"); +} + +if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) { + process.stdout.write(renderCheatSheet(loadLabs("labs"))); +} From d9d066c08ab24575475f374157361f0f492d01f3 Mon Sep 17 00:00:00 2001 From: Param Harrison <paramanantham@live.com> Date: Sat, 31 Oct 2026 07:10:53 +0200 Subject: [PATCH 05/12] docs(factory): diagrams and stories for every module Eight more hand-written SVGs: the artifact chain, risk ladder, workflow selector, sigma loop, model routing, test ladder, spend cap and cloud push. Every remaining module opens with a Story from a real ticket or retro item, then a picture and a Notice caption. f6 now says interpret.ts imports classify and APPROVALS from risk.ts, and that nothing imports deploy.ts. The visuals test now also fails any module with no picture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- docs/factory/f0-1-anatomy.md | 12 +++ docs/factory/f1-plan.md | 12 +++ docs/factory/f10-assignments.md | 12 +++ docs/factory/f11-cloud-sessions.md | 12 +++ docs/factory/f12-model-routing.md | 12 +++ docs/factory/f13-spend-and-safety.md | 12 +++ docs/factory/f2-design.md | 12 +++ docs/factory/f4-test.md | 12 +++ docs/factory/f6-deploy.md | 13 ++++ docs/factory/f7-maintain.md | 12 +++ docs/factory/f8-workflows.md | 12 +++ docs/factory/img/artifact-chain.svg | 91 +++++++++++++++++++++++ docs/factory/img/cloud-push.svg | 67 +++++++++++++++++ docs/factory/img/model-routing.svg | 96 ++++++++++++++++++++++++ docs/factory/img/risk-ladder.svg | 62 ++++++++++++++++ docs/factory/img/sigma-loop.svg | 68 +++++++++++++++++ docs/factory/img/spend-cap.svg | 59 +++++++++++++++ docs/factory/img/test-ladder.svg | 78 +++++++++++++++++++ docs/factory/img/workflow-selector.svg | 99 +++++++++++++++++++++++++ tests/structure/factory-visuals.test.ts | 5 ++ 20 files changed, 758 insertions(+) create mode 100644 docs/factory/img/artifact-chain.svg create mode 100644 docs/factory/img/cloud-push.svg create mode 100644 docs/factory/img/model-routing.svg create mode 100644 docs/factory/img/risk-ladder.svg create mode 100644 docs/factory/img/sigma-loop.svg create mode 100644 docs/factory/img/spend-cap.svg create mode 100644 docs/factory/img/test-ladder.svg create mode 100644 docs/factory/img/workflow-selector.svg diff --git a/docs/factory/f0-1-anatomy.md b/docs/factory/f0-1-anatomy.md index c39a69b..a822483 100644 --- a/docs/factory/f0-1-anatomy.md +++ b/docs/factory/f0-1-anatomy.md @@ -2,6 +2,18 @@ **Stage:** setup. **Layers:** context, boundary, skills, execution, sharing. **Workflows:** all of them use these files. +## Story + +**The ticket.** An item from the runs that built this repo, recorded in `docs/factory/harness-retro.md` (item 7). + +**Without the layer.** "The engine read `.factory/config.yaml` lazily, the first time a check asked. By then an agent had run in the same tree. A build that edited the file could change the gate commands the engine then ran." + +**What the layer did.** `e193fc7` (#89) added `pinPortable()`, which reads the file once before the first agent. The test "keeps the commands it read at the start when the file changes later" in `factory/engine/models.test.ts` holds it. Knowing which file is read, and when, is what this module is for. + +![The files a run writes in specs/<n>/, with what writes and what reads each one](img/artifact-chain.svg) + +_Notice:_ the work moves between stages as files on disk, so you can open any stage's output in an editor. + ## Goal Know what each file in `.claude/`, `.factory/` and the repo root does, which one wins when two disagree, and which hook pattern guards what. By the end you can open an unfamiliar repo and say which file would change a given behaviour. diff --git a/docs/factory/f1-plan.md b/docs/factory/f1-plan.md index a291a11..af40e57 100644 --- a/docs/factory/f1-plan.md +++ b/docs/factory/f1-plan.md @@ -2,6 +2,18 @@ **Stage:** Plan. **Layers:** skills, boundary. **Workflows:** `feature-atdd`, `feature-sdd`, `deps`, `incident`, `security` and `ui-visual` open with an `intent` step; every workflow with a `plan` step ends the stage with `plan.md`. +## Story + +**The ticket.** A `gate: human` step on a docs-only change, recorded in `docs/factory/harness-retro.md` (item 5). + +**Without the layer.** "A step marked `gate: human` on a docs-only change is tier low, so it needed 0 approvals and passed with nobody reading it." + +**What the layer did.** `3d6cac4` (#87), "A human gate always needs a person". `gateMet` in `factory/engine/interpret.ts` takes `Math.max(1, APPROVALS[tier])`, and `factory/engine/wiring.test.ts` pins it with "still needs one person for a docs-only change". + +![Three risk tiers set by the changed paths, with the approvals each needs](img/risk-ladder.svg) + +_Notice:_ the tier comes from paths alone, and the bottom rung still needs one person at a human gate. + ## Goal Turn a GitHub issue into decisions before any code exists: what outcome counts (`intent.md`), what kind of work it is (triage), and how much human approval it needs (the risk tier). By the end you can say which of the three is decided by a model and which by code. diff --git a/docs/factory/f10-assignments.md b/docs/factory/f10-assignments.md index 3ac2fee..6b79441 100644 --- a/docs/factory/f10-assignments.md +++ b/docs/factory/f10-assignments.md @@ -2,6 +2,18 @@ **Stage:** all of them. **Workflows shown:** whichever you choose below. +## Story + +**The ticket.** An open item in `docs/factory/harness-retro.md` (item 8). + +**Without the layer.** "`factory/engine/cli.ts` builds `approvers` as a list of one login, the person running the CLI." So "a high-tier gate could never reach two". + +**What the layer did.** Nothing yet. That gap is the assignment "Two people at a high-risk gate" below. Every assignment here starts from a gap like it. + +![The six stages from a ready ticket to a merged pull request, with the artifact and the deciding check under each](img/pipeline.svg) + +_Notice:_ each assignment changes one box or one arrow. Name it before you file the issue. + ## What exists Each assignment below has a brief in `assignments/` and a red test in `tests/assignments/`. The brief is a markdown file with a walkthrough, so you can read the whole task before you file anything. See [assignments/README.md](../../assignments/README.md) for the table. Every red test is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start. `tests/structure/assignments.test.ts` checks that every section on this page names a brief and a test that exist, that every brief has its nine headings in order, and that the test is gated on `ASSIGNMENT`. diff --git a/docs/factory/f11-cloud-sessions.md b/docs/factory/f11-cloud-sessions.md index 7e14dc9..130732e 100644 --- a/docs/factory/f11-cloud-sessions.md +++ b/docs/factory/f11-cloud-sessions.md @@ -2,6 +2,18 @@ **Stage:** all of them. **Layer:** execution, moved off your machine. **Prerequisite:** F3 (the builder works in its own worktree) and F7 (the factory runs on a schedule you do not watch). +## Story + +**The ticket.** Issue 35, whose CI stage ran in a sandbox. + +**Without the layer.** `specs/35/ci.md`: "`gh` is not authenticated in this sandbox". That is by design. Agents run with an allowlisted environment and an isolated home, so they never hold a login (`docs/factory/harness-retro.md`, item 3). + +**What the layer did.** The deny on `git push` in `.claude/settings.json` makes the same choice for cloud sessions. The clone carries it, the agent cannot push, and the push stays a person's step. + +![A cloud session's git push is refused by the cloned deny rule, with the three ways a team ships instead](img/cloud-push.svg) + +_Notice:_ every way out has a person in it: shipping the PR, pushing from a checkout, or writing the hook. + ## Goal Start several Claude Code cloud sessions on one repo, keep keys out of every prompt, understand what a cloud session can push here (its own branch) and what it cannot (`main`, force), hand the merge to a person or a gated step, and read what the sessions cost. diff --git a/docs/factory/f12-model-routing.md b/docs/factory/f12-model-routing.md index 2afec39..f7ff9ad 100644 --- a/docs/factory/f12-model-routing.md +++ b/docs/factory/f12-model-routing.md @@ -2,6 +2,18 @@ **Stage:** plan, build, review, verify. **Layer:** execution, priced. **Prerequisite:** F8 (workflows) and the cost ledger described in the FAQ. +## Story + +**The ticket.** The `docs` workflow. No run shows this one; the config and the workflow file do. + +**Without the layer.** `routes.docs.build` in `.factory/config.json` sends a docs build to Haiku, but `.factory/workflows/docs.yaml` names its step `write`, which is not in `stepStages`. The route never applies and the step runs on the middle model. + +**What the layer did.** `run.log` records the model on every round, so the miss is visible in a file, not hidden in a bill. The assignment "A model for every stage name" in F10 closes it. + +![Step names map through stepStages to a config stage and its model, with route overrides beside them](img/model-routing.svg) + +_Notice:_ the red box is a route that never fires. + ## Goal Send mechanical stages to the cheap model, building to the middle model and epic planning to the strongest one, see why an agent file alone cannot do that, and measure the mix from `cost.json`. diff --git a/docs/factory/f13-spend-and-safety.md b/docs/factory/f13-spend-and-safety.md index 9556c11..4fd66c5 100644 --- a/docs/factory/f13-spend-and-safety.md +++ b/docs/factory/f13-spend-and-safety.md @@ -2,6 +2,18 @@ **Stage:** all of them. **Layer:** boundary, with a price on it. **Prerequisite:** F11 (cloud sessions) and F12 (model routing). +## Story + +**The ticket.** Issue 35, the live spend limit for the worksheet api. + +**Without the layer.** `specs/35/build.md`: "`/chat` never read `LIVE_SPENT_USD`". The limit existed in the environment, and no code checked it. + +**What the layer did.** `server.ts` now refuses when `LIVE_SPENT_USD + spent >= 60`, before the model is called. The factory's own cap works the same way: `runStep` checks the run total before every agent call. This module shows where that check is and where it is not. + +![A bar from zero to the run cap, with the stop point and the spend the cap does not count](img/spend-cap.svg) + +_Notice:_ the dashed box is spend the cap does not see today. + ## Goal Know which limits actually stop a run, say plainly what the tool cannot control about credit, keep the API key out of every place a model or a log can read it, and use hooks as guardrails that fail closed. diff --git a/docs/factory/f2-design.md b/docs/factory/f2-design.md index 52337af..6284339 100644 --- a/docs/factory/f2-design.md +++ b/docs/factory/f2-design.md @@ -2,6 +2,18 @@ **Stage:** design. **Layers:** context and verification. **Workflows:** `feature-atdd`, `feature-bdd`, `feature-sdd`. +## Story + +**The ticket.** Issue 34, the workbook, recorded in `docs/factory/harness-retro.md` (item 2). + +**Without the layer.** "The build changed code, the spec hashed differently, and the resume regenerated the spec." The person had approved a spec that was then rewritten under them. + +**What the layer did.** `2f5d84f` (#73) made the spec and plan gates hash their own documents only. "keeps a spec or plan approval valid after later stages change other files" in `factory/engine/engine.test.ts` holds it, and "binds an approval to the artifact digest" keeps an edited spec from passing on an old approval. + +![The files a run writes in specs/<n>/, with what writes and what reads each one](img/artifact-chain.svg) + +_Notice:_ `spec.md` is read by the human gate by digest, so an approval covers that document and nothing else. + ## Goal Turn an `intent.md` into a `spec.md` that a check can grade, and stop for a human before any code exists. You write the same behaviour as examples, as Gherkin and as EARS requirements, and learn which one fits which ticket. diff --git a/docs/factory/f4-test.md b/docs/factory/f4-test.md index 3c959d4..1775682 100644 --- a/docs/factory/f4-test.md +++ b/docs/factory/f4-test.md @@ -2,6 +2,18 @@ **Stage:** test. **Layer:** verification. **Workflows:** `bug-tdd`, `feature-atdd`, `feature-bdd`, `feature-sdd`, `refactor`, `contract`, `ui-visual`, `llm-eval`. +## Story + +**The ticket.** Issue 35, the red stage, in `specs/35/red.md`. + +**Without the layer.** On the branch, "all 33 pass because the build is already here". A red check run on the branch would have seen green tests and proved nothing. + +**What the layer did.** Checked on `main`: "31 of 33 fail, 2 pass", and the two that pass are guards that hold on both sides. So "the `red` check must run with `ref: base`", which is what `factory/engine/checks/red.ts` does. + +![The checks grouped by size: small, medium and large](img/test-ladder.svg) + +_Notice:_ `red` and `lock` sit on the cheapest rung, and they decide whether the expensive checks above mean anything. + ## Goal Know which check proves which claim, run it at the right size, and see it fail on purpose. An agent optimises for whatever grades it, so each check below also names the trick it stops. diff --git a/docs/factory/f6-deploy.md b/docs/factory/f6-deploy.md index 709d486..283149c 100644 --- a/docs/factory/f6-deploy.md +++ b/docs/factory/f6-deploy.md @@ -2,6 +2,18 @@ **Stage:** Deploy. **Layer:** delivery. **Workflows:** the `ci` step in `bug-tdd` and `feature-atdd` (and the others that end with it). +## Story + +**The ticket.** Issue 35, the CI stage, in `specs/35/ci.md`. + +**Without the layer.** With no `gh` login in the sandbox there was no CI result to read. A stage that summarised from memory would have had nothing to stand on. + +**What the layer did.** The `ci` check runs `gh pr checks --watch`, so it cannot pass without a result. The stage wrote "No CI job log was reachable: `gh` is not authenticated in this sandbox", listed the local failures it blamed on the sandbox, and asked a person to "confirm both pass on a real CI runner". + +![Three risk tiers set by the changed paths, with the approvals each needs](img/risk-ladder.svg) + +_Notice:_ this ladder sets the count at a human gate today. `release` in `deploy.ts` asks it again before production, once something calls it. + ## Goal Decide how much human approval a change needs from the files it touches, never from who wrote it. Check the change against the latest `main`, watch CI, and release only with the approvals the tier demands on this exact commit. @@ -9,6 +21,7 @@ Decide how much human approval a change needs from the files it touches, never f ## What is built and what is design - Built and tested in `factory/engine/review.test.ts`: `classify` and `approvalsMet` (`risk.ts`), and `release`, `rollback` and `mergeQueue` (`deploy.ts`). +- Called by the runner: `interpret.ts` imports `classify` and `APPROVALS` from `risk.ts` and applies them at every `gate: human` step, with a floor of one person. - Reachable from a workflow: a step with `deploy: merge-queue`, `deploy: preview` or `deploy: production` runs no agent and calls `mergeQueue` or `release` (`deploy-step.ts`). Production needs at least one writer's approval of that stage, more for a high tier. `bug-tdd` ends with a `deploy: merge-queue` step. The approvals `release` checks are the logins `Host.writers` returns, the same people `approved` counted. - Missing target: `deploy.ts` runs `make deploy ENV=<env>`, and this repo's `Makefile` has no `deploy` target. The release and rollback commands therefore have nothing to call here. A learner adds a deploy target for their own host. This is a design, not a working deploy. - Built and used: the `ci` check in `factory/engine/checks/index.ts` runs `gh pr checks --watch`, and the workflow loop feeds a failure back to the builder. diff --git a/docs/factory/f7-maintain.md b/docs/factory/f7-maintain.md index 355f5f9..257a76b 100644 --- a/docs/factory/f7-maintain.md +++ b/docs/factory/f7-maintain.md @@ -2,6 +2,18 @@ **Stage:** Maintain. **Layer:** sharing and feedback. **Workflows:** `incident` (diagnose, plan, eval, fix, review). +## Story + +**The ticket.** None yet. Nothing feeds `detect` its metrics, so no run has breached a band. The story is the test. + +**Without the layer.** A drifting metric needs someone to notice it on a chart. + +**What the layer did.** `a 3 sigma breach writes an intent` in `factory/engine/review.test.ts` feeds a value of 30 against a history near 10 and expects a new `intent.md`. Arithmetic turns the drift into a ticket, with no model deciding. + +![The maintain loop: band metrics feed detect, the sigma level picks an action, and a breach starts the incident workflow](img/sigma-loop.svg) + +_Notice:_ the dashed parts are the ones only tests reach today. + ## Goal After a change ships, the factory watches itself. A metric that leaves its band becomes a ticket. An incident becomes a failing test before it becomes a fix. A lesson from a run is kept only if the eval score did not fall. diff --git a/docs/factory/f8-workflows.md b/docs/factory/f8-workflows.md index 9f0a239..c45258e 100644 --- a/docs/factory/f8-workflows.md +++ b/docs/factory/f8-workflows.md @@ -2,6 +2,18 @@ **Stage:** all of them. **Layer:** delivery, made configurable. **Workflows shown:** every file in `.factory/workflows/`. +## Story + +**The ticket.** Runs 29 and 31, recorded in the plan brief tests in `factory/engine/workflows.test.ts`. + +**Without the layer.** Run 29's plan held a note, and run 31 listed its own golden test. Both plans were refused. + +**What the layer did.** The shared plan brief in `.factory/config.yaml` now carries one sentence per failure, each pinned by a test that names the run. A workflow file is where a rule like that lives once, instead of in every prompt. + +![A decision tree for choosing a workflow, with the workflows a ticket names directly beside it](img/workflow-selector.svg) + +_Notice:_ the tree asks what evidence the task can give, never how big the ticket is. + ## Goal Write a workflow file from scratch, make the repo's own tests accept it, run it on an issue, and pick the right one for each kind of ticket. diff --git a/docs/factory/img/artifact-chain.svg b/docs/factory/img/artifact-chain.svg new file mode 100644 index 0000000..494b650 --- /dev/null +++ b/docs/factory/img/artifact-chain.svg @@ -0,0 +1,91 @@ +<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 960 310" width="960" height="310" role="img" aria-labelledby="t d"> + <title id="t">The artifact chain in specs/<n>/ + Seven files in the order a run writes them: intent.md, spec.md, plan.md, the golden tests, lock.json, review.json and run.log. Above each file is the agent or check that writes it; below is what reads it next. The planner writes intent, spec and plan; the human gate reads the spec; plan-drift reads the plan; red and lock read the golden tests; the fixer reads review.json. + + + + + + + + The artifact chain + Each agent call starts fresh, so the files in specs/<n>/ are what carry the work forward. + written by + read by + planner reply, + saved by the runner + + intent.md + later agents, + read from disk + + planner + + spec.md + the human gate, + approved by digest + + planner + file list only + + plan.md + plan-drift, on + each step it runs + + builder, red phase + tests only + + golden/ + red on the base + commit, then lock + + lock check + sha256 per test + + lock.json + lock, every + later round + + reviewRound + votes per finding + + review.json + the fixer + and a person + + the engine + every check + + run.log + you, after + the run ends + + classify() skips these run files when it sets the risk tier, except golden/. run.log is gitignored, so it stays where it ran. + diff --git a/docs/factory/img/cloud-push.svg b/docs/factory/img/cloud-push.svg new file mode 100644 index 0000000..89b4504 --- /dev/null +++ b/docs/factory/img/cloud-push.svg @@ -0,0 +1,67 @@ + + Why a cloud session cannot push, and the ways out + The cloned settings.json denies git push, so when the agent in a cloud session runs git push the permission rule refuses it. Three ways out follow: keep the deny and a person opens the pull request from the Changes card; keep the deny and teleport the session to a local checkout, run make check and push; or replace the deny with a tested hook that allows only factory branches, a change a person makes because the hook folder is protected. + + + + + + + + The push belongs to a person + A cloud session clones the repo, so .claude/settings.json comes with it, deny list included. + + .claude/settings.json + in the clone + + + the agent runs + git push + + + refused by the deny + Bash(git push *) + A session that cannot push + is the system working. + + + Keep the deny, a person ships + read the Changes card, open the PR from claude.ai/code + + + Keep the deny, teleport + claude --teleport, make check, push from your terminal + + + Narrow the deny with a hook + allow factory/* branches only; a person edits the hook + Pick one and write it in AGENTS.md. + The default here is the first. + diff --git a/docs/factory/img/model-routing.svg b/docs/factory/img/model-routing.svg new file mode 100644 index 0000000..bb634d2 --- /dev/null +++ b/docs/factory/img/model-routing.svg @@ -0,0 +1,96 @@ + + Model routing by stage + Each workflow step name maps through stepStages to a config stage, and the stage sets the model. Intent maps to triage on Haiku. Spec, plan and decide map to plan on Sonnet. Tests, build, fix and the other build steps map to build on Sonnet. Review maps to review on Sonnet. Ci, eval and reproduce map to verify on Haiku. Routes override this for an epic plan, which runs on Opus, and a docs build, which runs on Haiku. The docs workflow names its step write, which is unmapped, so it falls back to the middle model. Review lenses and verifiers take their model from frontmatter. + + + + + + + + Which model runs each step + resolveModel() in models.ts: step name, then stepStages, then a route for the ticket type, then the stage model. + step name in the workflow + stepStages + config.json model + + intent + + + triage + + + Haiku + + spec, plan, decide + + + plan + + + Sonnet + + tests, build, fix, harden, refactor, bump, release + + + build + + + Sonnet + + review + + + review + + + Sonnet + + ci, eval, reproduce + + + verify + + + Haiku + route overrides + + epic plan + Opus + + docs build + Haiku + + docs names its step write, which is in neither list, + so it falls back to the middle model and misses the route. + + Lens and verifier calls in a review pass no model: + frontmatter decides, Sonnet lenses and a Haiku verifier. + diff --git a/docs/factory/img/risk-ladder.svg b/docs/factory/img/risk-ladder.svg new file mode 100644 index 0000000..0b350e4 --- /dev/null +++ b/docs/factory/img/risk-ladder.svg @@ -0,0 +1,62 @@ + + Risk tiers and the approvals each needs + Three rungs. High covers paths that steer an agent or the pipeline, such as .claude, .factory, factory, scripts, CLAUDE.md, AGENTS.md, package.json and migrations, and needs two approvals. Medium covers everything else, including an empty diff, and needs one. Low covers only docs and fixtures and needs none from the table, but a human gate step still needs one person. + + + + + + + + Risk tier and approvals + classify() in risk.ts reads only the changed paths. The worst path sets the tier, and high is checked first. + + high + .github/ .claude/ .factory/ factory/ scripts/ + CLAUDE.md AGENTS.md REVIEW.md Makefile package.json migrations/ ../ + + 2 approvals + distinct people + + medium + everything else: api/, web/, shared/, tests/ + an empty diff is medium, never low + + 1 approval + distinct people + + low + only docs/ and fixtures/ + a gate: human step still needs 1, from Math.max(1, APPROVALS[tier]) + + 0 approvals + distinct people + An agent's approval never counts. An approval names one commit, so a push after it voids it. + diff --git a/docs/factory/img/sigma-loop.svg b/docs/factory/img/sigma-loop.svg new file mode 100644 index 0000000..888d395 --- /dev/null +++ b/docs/factory/img/sigma-loop.svg @@ -0,0 +1,68 @@ + + The sigma bands maintain loop + Three metrics in bands.yaml feed detect, which computes how many standard deviations the latest value is from the mean of the last 14. One sigma logs, two sigma starts a read-only diagnosis, three sigma writes a new intent.md through intentFor, which starts the incident workflow, and the next values feed the band again. The feed and the loop are dashed because only tests call detect today. + + + + + + + + Sigma bands: a metric that leaves its band becomes a ticket + bands.ts is plain arithmetic: mean and standard deviation over the last 14 values. No model decides a breach. + + .factory/bands.yaml + ci-failure-rate, up is bad + pr-cycle-time-hours, up is bad + eval-pass-rate, down is bad + + feed + + detect() + z from the window + + + 1 sigma + log it + + 2 sigma + diagnose, read-only agent + + 3 sigma + intentFor writes intent.md + + + incident + workflow + + the incident becomes a failing test, the fix ships, the next values feed the band again + + Tests only today: nothing calls detect() on a schedule, and nothing feeds it these metrics. + diff --git a/docs/factory/img/spend-cap.svg b/docs/factory/img/spend-cap.svg new file mode 100644 index 0000000..a24b0e9 --- /dev/null +++ b/docs/factory/img/spend-cap.svg @@ -0,0 +1,59 @@ + + The run spend cap + A bar from zero to the ten dollar cap. Agent calls run while the total is under seven dollars fifty, and each call gets a ceiling of the cap minus what is spent. At seven fifty the next call is not run, run.log records the stop and the outcome is needs-human with reason spend cap. Review lens hunts and verifier ballots are drawn dashed because they are not added to the total today. + + + + + + + + The run spend cap + runStep in interpret.ts checks the run total before every agent call. config.json: capUsd 10, stopAt 0.75. + + + + agent calls run, each with maxUsd = cap minus spent + next call not run + + $0 + + $7.50 stop + + $10 cap + + When it stops + run.log: <stage> not run: $X of $Y spent + reason "spend cap", outcome needs-human + + Not counted today + review lens hunts and verifier ballots + only the review fixer is added to the total + diff --git a/docs/factory/img/test-ladder.svg b/docs/factory/img/test-ladder.svg new file mode 100644 index 0000000..d713212 --- /dev/null +++ b/docs/factory/img/test-ladder.svg @@ -0,0 +1,78 @@ + + The checks, by size + Three rows of checks. Small checks are fast and hermetic: red, lock, plan-drift and green. Medium checks run a test folder: acceptance, holdout, properties, behaviour-unchanged and contract. Large checks run the app, the model or the whole repo: mutation, visual, eval, gate and ci. + + + + + + + + Checks by size + Every check in checks/index.ts declares a size. A workflow step lists the checks it must pass. + + small + fast and hermetic + + red + + lock + + plan-drift + + green + + medium + runs a test folder + + acceptance + + holdout + + properties + + behaviour-unchanged + + contract + + large + app, model or whole repo + + mutation + + visual + + eval + + gate + + ci + A bigger check costs more time. Pick the smallest one that proves the claim. + diff --git a/docs/factory/img/workflow-selector.svg b/docs/factory/img/workflow-selector.svg new file mode 100644 index 0000000..6f63172 --- /dev/null +++ b/docs/factory/img/workflow-selector.svg @@ -0,0 +1,99 @@ + + Choosing a workflow for a ticket + A decision tree. If there is no code to write yet, use spike. If the thing under test answers differently each time, use llm-eval. If you can write a test that fails today, use refactor when behaviour should stay the same and bug-tdd otherwise. If a person must judge done, use feature-bdd or feature-sdd, which put a human gate before the build. A side panel lists the workflows that the kind of ticket names directly: deps, security, contract, ui-visual, docs, incident and feature-atdd. + + + + + + + + Which workflow fits this ticket + main.ts does not pick for you: you name the workflow with --workflow=. Ask what evidence the task can give. + + Is there code to write yet? + + no + + spike + explore, then decide + + yes + + Does it answer differently each time? + + yes + + llm-eval + grade a pass rate + + no + + Can you write a test that fails today? + + yes + + Should behaviour stay? + + yes + + refactor + pin, then change + + no + + bug-tdd + red, lock, mutation + + no + + Must a person judge done? + + yes + + feature-bdd, feature-sdd + gate: human before the build + + Named by the ticket itself + deps + a dependency bump + security + a security fix + contract + the api and web boundary + ui-visual + a UI change + docs + documentation + incident + an incident or breach + feature-atdd + a small feature with examples + diff --git a/tests/structure/factory-visuals.test.ts b/tests/structure/factory-visuals.test.ts index d7ebd21..cba313d 100644 --- a/tests/structure/factory-visuals.test.ts +++ b/tests/structure/factory-visuals.test.ts @@ -6,6 +6,7 @@ import { describe, expect, it } from "vitest"; const dir = "docs/factory"; const MAX_SVG_BYTES = 40_000; const docs = readdirSync(dir).filter((f) => f.endsWith(".md")); +const modules = docs.filter((f) => /^f\d/.test(f)); const svgs = existsSync(`${dir}/img`) ? readdirSync(`${dir}/img`).filter((f) => f.endsWith(".svg")) : []; @@ -15,6 +16,10 @@ const images = (file: string) => ); describe("factory docs visuals", () => { + it("embeds at least one picture in every module", () => { + expect(modules.filter((f) => images(f).length === 0)).toEqual([]); + }); + it("points every picture at a file that exists, with alt text", () => { const bad = docs.flatMap((f) => images(f) From 4faa485081630cfcbd8d556c4295f5dedfdc7043 Mon Sep 17 00:00:00 2001 From: Param Harrison Date: Sun, 1 Nov 2026 17:15:09 +0200 Subject: [PATCH 06/12] docs(factory): true Stories for f5, f6, f10, f12 and a test that holds every quote story-quotes.test.ts checks each double-quoted phrase in a Story against a file the Story cites, a factory test, the retro or an assignment brief. f12 and the model-routing picture no longer say the docs write step misses its route. Co-Authored-By: Claude Sonnet 5.5 --- docs/factory/f10-assignments.md | 2 +- docs/factory/f12-model-routing.md | 8 ++--- docs/factory/f5-review.md | 2 +- docs/factory/f6-deploy.md | 2 +- docs/factory/img/model-routing.svg | 8 ++--- levels.json | 3 +- tests/structure/story-quotes.test.ts | 48 ++++++++++++++++++++++++++++ 7 files changed, 61 insertions(+), 12 deletions(-) create mode 100644 tests/structure/story-quotes.test.ts diff --git a/docs/factory/f10-assignments.md b/docs/factory/f10-assignments.md index 6b79441..cae058b 100644 --- a/docs/factory/f10-assignments.md +++ b/docs/factory/f10-assignments.md @@ -8,7 +8,7 @@ **Without the layer.** "`factory/engine/cli.ts` builds `approvers` as a list of one login, the person running the CLI." So "a high-tier gate could never reach two". -**What the layer did.** Nothing yet. That gap is the assignment "Two people at a high-risk gate" below. Every assignment here starts from a gap like it. +**What the layer did.** Nothing yet. That gap is the assignment on naming approvers for a high-risk gate, below. Every assignment here starts from a gap like it. ![The six stages from a ready ticket to a merged pull request, with the artifact and the deciding check under each](img/pipeline.svg) diff --git a/docs/factory/f12-model-routing.md b/docs/factory/f12-model-routing.md index f7ff9ad..51de681 100644 --- a/docs/factory/f12-model-routing.md +++ b/docs/factory/f12-model-routing.md @@ -4,15 +4,15 @@ ## Story -**The ticket.** The `docs` workflow. No run shows this one; the config and the workflow file do. +**The ticket.** The `docs` workflow. No run shows this one; the config, the workflow file and a test do. -**Without the layer.** `routes.docs.build` in `.factory/config.json` sends a docs build to Haiku, but `.factory/workflows/docs.yaml` names its step `write`, which is not in `stepStages`. The route never applies and the step runs on the middle model. +**Without the layer.** `routes.docs.build` in `.factory/config.json` sends a docs build to Haiku. `.factory/workflows/docs.yaml` names its step `write`, and a step name with no entry in `stepStages` would skip the route and run on the middle model. -**What the layer did.** `run.log` records the model on every round, so the miss is visible in a file, not hidden in a bill. The assignment "A model for every stage name" in F10 closes it. +**What the layer did.** `stepStages` maps `write` to `build`, so the `write` step takes the `build` model and the docs route replaces it with Haiku. `factory/engine/docs-routing.test.ts` asserts both: a docs ticket gets `claude-haiku-5-5`, a bug ticket gets the `build` default. `run.log` records the model on every call, so a miss shows in a file, not in a bill. ![Step names map through stepStages to a config stage and its model, with route overrides beside them](img/model-routing.svg) -_Notice:_ the red box is a route that never fires. +_Notice:_ the red box is the case a mapping prevents: a step name with no stage. ## Goal diff --git a/docs/factory/f5-review.md b/docs/factory/f5-review.md index 1491515..8131f83 100644 --- a/docs/factory/f5-review.md +++ b/docs/factory/f5-review.md @@ -6,7 +6,7 @@ **The ticket.** Issue 35, the build notes in `specs/35/build.md`. -**Without the layer.** The first note says "Blocked: two locked tests cannot pass as written" and proposes changing one. The next update says the blockers "no longer reproduce". A claim with no input that breaks it would have sent a person to chase it. +**Without the layer.** The first note in `specs/35/build.md` says "Blocked: two locked tests cannot pass as written" and proposes changing one. The next update says the blockers "no longer reproduce". A claim with no input that breaks it would have sent a person to chase it. **What the layer did.** A build note is not held to the review's rule, which is why the rule exists. A review finding must carry `evidence`, "the input that breaks it. No evidence, no finding" (`factory/engine/review.ts`), and two of three verifiers must agree before it reaches a person. diff --git a/docs/factory/f6-deploy.md b/docs/factory/f6-deploy.md index 283149c..cb0ee5c 100644 --- a/docs/factory/f6-deploy.md +++ b/docs/factory/f6-deploy.md @@ -12,7 +12,7 @@ ![Three risk tiers set by the changed paths, with the approvals each needs](img/risk-ladder.svg) -_Notice:_ this ladder sets the count at a human gate today. `release` in `deploy.ts` asks it again before production, once something calls it. +_Notice:_ this ladder sets the count at every human gate. `release` in `deploy.ts` asks `approvalsMet` again before production, and `interpret.ts` reaches `release` through `deploy-step.ts` for a `deploy: production` step. No shipped workflow has one yet: `bug-tdd` ends at the merge queue. ## Goal diff --git a/docs/factory/img/model-routing.svg b/docs/factory/img/model-routing.svg index bb634d2..38c706a 100644 --- a/docs/factory/img/model-routing.svg +++ b/docs/factory/img/model-routing.svg @@ -1,6 +1,6 @@ Model routing by stage - Each workflow step name maps through stepStages to a config stage, and the stage sets the model. Intent maps to triage on Haiku. Spec, plan and decide map to plan on Sonnet. Tests, build, fix and the other build steps map to build on Sonnet. Review maps to review on Sonnet. Ci, eval and reproduce map to verify on Haiku. Routes override this for an epic plan, which runs on Opus, and a docs build, which runs on Haiku. The docs workflow names its step write, which is unmapped, so it falls back to the middle model. Review lenses and verifiers take their model from frontmatter. + Each workflow step name maps through stepStages to a config stage, and the stage sets the model. Intent maps to triage on Haiku. Spec, plan and decide map to plan on Sonnet. Tests, build, write, fix and the other build steps map to build on Sonnet. Review maps to review on Sonnet. Ci, eval and reproduce map to verify on Haiku. Routes override this for an epic plan, which runs on Opus, and a docs build, which runs on Haiku. The docs workflow names its step write, which maps to build, so the docs route sends it to Haiku. Review lenses and verifiers take their model from frontmatter.