Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 0 additions & 17 deletions .github/ISSUE_TEMPLATE/assignment-approvers.md

This file was deleted.

17 changes: 0 additions & 17 deletions .github/ISSUE_TEMPLATE/assignment-canary.md

This file was deleted.

17 changes: 0 additions & 17 deletions .github/ISSUE_TEMPLATE/assignment-clock-activity.md

This file was deleted.

17 changes: 0 additions & 17 deletions .github/ISSUE_TEMPLATE/assignment-fuzz-tool-input.md

This file was deleted.

17 changes: 0 additions & 17 deletions .github/ISSUE_TEMPLATE/assignment-judge-hook.md

This file was deleted.

17 changes: 0 additions & 17 deletions .github/ISSUE_TEMPLATE/assignment-label-machine.md

This file was deleted.

19 changes: 0 additions & 19 deletions .github/ISSUE_TEMPLATE/assignment-my-mod.md

This file was deleted.

19 changes: 0 additions & 19 deletions .github/ISSUE_TEMPLATE/assignment-own-workflow.md

This file was deleted.

17 changes: 0 additions & 17 deletions .github/ISSUE_TEMPLATE/assignment-review-lens.md

This file was deleted.

16 changes: 0 additions & 16 deletions .github/ISSUE_TEMPLATE/assignment-stage-names.md

This file was deleted.

18 changes: 0 additions & 18 deletions .github/ISSUE_TEMPLATE/assignment-template-db.md

This file was deleted.

17 changes: 0 additions & 17 deletions .github/ISSUE_TEMPLATE/assignment-worksheet-video.md

This file was deleted.

2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ This repo is a GitHub template. Click "Use this template", then set these in you
- Labels `factory:triage`, `factory:planned`, `factory:ready`, `plan:approved` and `assignment`. The factory moves a ticket between them.
- `levels.json` owns every file a level ships. A new source file needs an owner there, or `make check` fails.

Teaching it? Use the [run of show](docs/run-of-show.md).
Learning on your own or with a team? Follow [learn in two days](docs/learn-in-two-days.md). The ideas worth keeping are on one page: [patterns](docs/patterns.md).

## Install the factory in your own repo

Expand Down
24 changes: 24 additions & 0 deletions assignments/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Assignments

Pick one, read the whole brief, then file it as an issue with the command in its `## File it` section. The briefs are markdown files, not issue templates, so you can read the walkthrough before you commit to the task.

Every assignment has a test that is skipped unless `ASSIGNMENT=1`. Run it red first, then make your factory turn it green. [F10](../docs/factory/f10-assignments.md) explains the steps shared by all of them.

| Assignment | Module | Level | Workflow | Red test |
| ------------------------------------------------------------------------------ | ------ | --------- | -------------- | -------------------------------------------------- |
| [name the approvers for a high-risk gate](approvers.md) | F6 | principal | `feature-atdd` | `tests/assignments/approvers.test.ts` |
| [canary prompt with rollback](canary.md) | F7 | senior | `llm-eval` | `tests/assignments/canary.test.ts` |
| [add the clock activity kind](clock-activity.md) | F2 | noob | `feature-bdd` | `tests/assignments/clock-activity.test.ts` |
| [fuzz the worksheet tool input](fuzz-tool-input.md) | F4 | senior | `security` | `tests/assignments/fuzz-tool-input.test.ts` |
| [an opt-in LLM-judge hook](judge-hook.md) | F13 | senior | `feature-atdd` | `tests/assignments/judge-hook.test.ts` |
| [specify the label state machine](label-machine.md) | F1 | principal | `feature-atdd` | `tests/assignments/label-machine.test.ts` |
| [a mod of your own](my-mod.md) | F3 | senior | `feature-atdd` | `tests/assignments/my-mod.test.ts` |
| [a workflow from your own company](own-workflow.md) | F8 | senior | `feature-atdd` | `tests/assignments/own-workflow.test.ts` |
| [record each factory module from a real run log](recording-from-run-log.md) | F0 | senior | `docs` | `tests/assignments/recording-from-run-log.test.ts` |
| [add a performance review lens](review-lens.md) | F5 | senior | `llm-eval` | `tests/assignments/review-lens.test.ts` |
| [a model for every stage name](stage-names.md) | F12 | noob | `llm-eval` | `tests/assignments/stage-names.test.ts` |
| [move worksheet templates into a database](template-db.md) | F8 | senior | `spike` | `tests/assignments/template-db.test.ts` |
| [seed a 3-sigma breach and run the incident workflow](three-sigma-incident.md) | F7 | principal | `incident` | `tests/assignments/three-sigma-incident.test.ts` |
| [render a worksheet as a HyperFrames video](worksheet-video.md) | F8 | senior | `ui-visual` | `tests/assignments/worksheet-video.test.ts` |

Levels: `noob` needs only the start and daily lanes, `senior` the team and factory lanes, `principal` a design decision that has a cost.
54 changes: 54 additions & 0 deletions assignments/approvers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Assignment: name the approvers for a high-risk gate

## Brief

Read the approver logins from `.factory/config.yaml` (an `approvers` list in `portableSchema`) and pass them to `realHost` from `runIssue`.

## Level

principal

## Acceptance

- A high-risk gate opens with two listed people, a medium one with one.
- One person commenting twice counts once.
- The logged-in user is no longer assumed.

## Workflow

`feature-atdd`. Workflows are listed in [F8](../docs/factory/f8-workflows.md).

## Red test

```sh
ASSIGNMENT=1 npx vitest run tests/assignments/approvers.test.ts
```

It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start.

## Walkthrough

1. Run the red test: `ASSIGNMENT=1 npx vitest run tests/assignments/approvers.test.ts`.
Expect: It fails because the config schema has no `approvers` key and the CLI still assumes the logged-in user.
2. File the issue: `gh issue create --title "Assignment: name the approvers for a high-risk gate" --body-file assignments/approvers.md --label assignment`.
Expect: It prints the new issue URL.
3. Write `factory/engine/portable.ts`, `factory/engine/cli.ts`.
Expect: The red test now fails on behaviour, not on a missing file.
4. Run the factory: `make factory ISSUE=<issue> FLAGS=--workflow=feature-atdd`.
Expect: The run ends `awaiting-approval` or `needs-human`, and `specs/<issue>/run.log` says which.
5. Run the gate: `make check`.
Expect: Zero errors and zero warnings.
6. Run the red test again: `ASSIGNMENT=1 npx vitest run tests/assignments/approvers.test.ts`.
Expect: It passes.

## Discussion

Why does one person commenting twice count once?

- **Principal:** Who should be on the approver list for a high-risk change, and how do you keep that list from becoming a rubber stamp or a bottleneck?

## File it

```sh
gh issue create --title "Assignment: name the approvers for a high-risk gate" --body-file assignments/approvers.md --label assignment
```
54 changes: 54 additions & 0 deletions assignments/canary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Assignment: canary prompt with rollback

## Brief

Add `api/src/agent/canary.ts` for the prompt in `api/src/agent/prompt.ts`.

## Level

senior

## Acceptance

- `routePrompt(requestId, share)` returns `"candidate"` or `"current"`, the same way for the same id, and near `share` over many ids.
- `decideCanary({ current, candidate, margin })` returns `"rollback"` when the candidate's pass rate falls below the current one by more than `margin`, otherwise `"keep"`.
- After a rollback the share is 0 and the event is logged.

## Workflow

`llm-eval`. Workflows are listed in [F8](../docs/factory/f8-workflows.md).

## Red test

```sh
ASSIGNMENT=1 npx vitest run tests/assignments/canary.test.ts
```

It is skipped unless `ASSIGNMENT=1`, so `make check` stays green until you start.

## Walkthrough

1. Run the red test: `ASSIGNMENT=1 npx vitest run tests/assignments/canary.test.ts`.
Expect: It fails because `api/src/agent/canary.ts` does not exist.
2. File the issue: `gh issue create --title "Assignment: canary prompt with rollback" --body-file assignments/canary.md --label assignment`.
Expect: It prints the new issue URL.
3. Write `api/src/agent/canary.ts`.
Expect: The red test now fails on behaviour, not on a missing file.
4. Run the factory: `make factory ISSUE=<issue> FLAGS=--workflow=llm-eval`.
Expect: The run ends `awaiting-approval` or `needs-human`, and `specs/<issue>/run.log` says which.
5. Run the gate: `make check`.
Expect: Zero errors and zero warnings.
6. Run the red test again: `ASSIGNMENT=1 npx vitest run tests/assignments/canary.test.ts`.
Expect: It passes.

## Discussion

Why must `routePrompt` give the same answer for the same request id?

- **Principal:** Eval scores are noisy and small. How many requests does the canary need before a rollback is a decision and not a coin flip, and who owns that number?

## File it

```sh
gh issue create --title "Assignment: canary prompt with rollback" --body-file assignments/canary.md --label assignment
```
Loading
Loading