diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index fe2adf0..8753bc7 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -10,30 +10,35 @@ permissions: jobs: test: - name: ${{ matrix.os }} / Node ${{ matrix.node }} + name: ${{ matrix.os }} / Node ${{ matrix.node }}${{ matrix.python && format(' / Python {0}', matrix.python) || '' }} runs-on: ${{ matrix.os }} timeout-minutes: 15 strategy: fail-fast: false matrix: + # Without python, a job uses the runner's own python3 (macOS's moves + # with the runner image). The 3.15 job tests the newest Python on + # purpose rather than waiting for an image to move. include: - os: ubuntu-latest node: 24 - os: ubuntu-latest - node: 22 + node: 24 + python: '3.15' - os: macos-latest node: 24 - os: windows-latest node: 24 + python: '3.12' steps: - uses: actions/checkout@v7 - uses: actions/setup-node@v7 with: node-version: ${{ matrix.node }} - uses: actions/setup-python@v6 - if: matrix.os == 'windows-latest' + if: matrix.python with: - python-version: '3.12' + python-version: ${{ matrix.python }} - name: Tool versions shell: bash run: | diff --git a/README.md b/README.md index b01a33a..e53a180 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,7 @@ Fedora: sudo dnf install -y git jq python3 nodejs ``` -The test suite needs Node 22 or later (`node --version`). If your distribution's `nodejs` +The test suite needs Node 24 or later (`node --version`). If your distribution's `nodejs` package is older, install a current release from https://nodejs.org instead; the skill itself does not use Node. @@ -98,7 +98,7 @@ too: install it there and follow the Linux steps. winget install OpenJS.NodeJS.LTS ``` - Node is optional (tests only; Node 22 or later). Any Python 3.8 or later works: the skill looks for + Node is optional (tests only; Node 24 or later). Any Python 3.8 or later works: the skill looks for `python3`, then `python`, then `py -3`, and skips the Microsoft Store `python3` stub. - **Git in another place:** the skill finds Git Bash on PATH (also through the `Git\cmd` folder the Git installer puts there) or under `C:\Program Files\Git`. @@ -281,7 +281,8 @@ session lists it, and you can resume with one word. Finished tasks are skipped. `scripts/setup` also locks the project checkout in git mode (two runs cannot switch one checkout to their feature branches), creates the feature branch and lane worktrees, saving any changes it has to discard under a git ref first. The phases run: pre-flight conflict check, prelude, - lanes in parallel, integration, join, E2E, a final review with three lenses and one fix + lanes in parallel, integration, join (its tasks in waves: those whose `depends_on` allow it + run side by side on their own branches and are merged after each wave), E2E, a final review with three lenses and one fix round, and a verify step that reruns the project checks (every lane's checks included) and any stale E2E or post-integration check on the exact revision delivered. The full pre-flight stops the run, naming the move that fixes it, when a task needs code from a task scheduled after it. diff --git a/SKILL.md b/SKILL.md index 4a84ada..fe12e30 100644 --- a/SKILL.md +++ b/SKILL.md @@ -77,8 +77,10 @@ manual. - Worth-it check, only when the Offer will be shown (not when the user already chose Parallel lanes by name, never on Resume): with the layout just built, run ` /scripts/worth-it --max-lanes --prelude --lane - [--lane ...] --join --project ` (comma-separated task ids, one - `--lane` per lane in lane order; an empty prelude or join is `''`). It prints JSON: + [--lane ...] --join [--join-deps : ...] --project ` + (comma-separated task ids, one `--lane` per lane in lane order; an empty prelude or join is + `''`; one `--join-deps` per join task, its earlier join ids joined by `+`, so the join counts in + waves). It prints JSON: `recommend` is `direct` when running in parallel saves 1 task-time or less (always for a single-lane plan), else `parallel`; a non-zero exit or any `status` other than `ok` leaves the Offer as it was. The rule, its evidence and the output: reference.md "Worth it". @@ -133,7 +135,9 @@ Fields: reference.md "Manifest fields". In order: - Git identity: `git -C var GIT_COMMITTER_IDENT` must print one, or every task commit fails partway through the run. If it fails: reference.md "Git identity". 3. Lanes, prelude, join, hooks, `depends_on`, `overlaps`, `excluded`: reference.md "Building - lanes" (`prelude` and `join` are not lane ids). + lanes" (`prelude` and `join` are not lane ids; no lane id starts with `u-`). Give every join + task `depends_on` (`[]` when it needs no earlier join task): the join then runs in waves, and + a join task without it waits for every earlier one (reference.md "Join waves"). 4. `profile` and tiers: reference.md "Profiles", "Tiers", "Batching". `profile: "lite"` only for one lane, at most 8 tasks, no security task, and no `hooks.post_integrate`; else `full`. Tiers `standard`, `sonnet`, `light` (security tasks always `standard`); `batch` @@ -163,7 +167,8 @@ and `usage_limit_pause` gives its usage-limit line (reference.md "Usage limit"). Each check and what it means: reference.md "Doctor" and "Launch conditions". 1. Dry run: call the Workflow tool as in Launch step 1, with `args` = the manifest with - `dry_run: true`. It returns `{dry_run, errors, agents, lanes_effective, notices, checks}` and + `dry_run: true`. It returns `{dry_run, errors, agents, lanes_effective, notices, checks, + join_waves}` and spawns nothing. If `errors` is non-empty, fix the manifest and repeat; never launch a manifest with errors. Then set `limits.max_agents` = 2 x `agents.length` and `limits.max_rulings` = 25 (reference.md "Budgets") so the table shows them. @@ -251,7 +256,7 @@ earlier snapshot belongs to the earlier launch. Relaunch once with the relaunch notice: carry budgets over (reference.md "Transient stops"). Set `repairs_spent` to the repairs in the stopped run's `final.repairs` plus the earlier value. First recompute `done`, `reviewed`, `deferred`, the `carry` notes, `carried_minors`, -`inputs_changed` and `backfill` as in Resume +`inputs_changed`, the join wave fields and `backfill` as in Resume steps 2-3, then rerun `scripts/setup --owner ` and take `start_points`; if validation rejects its `setup_skipped`, rerun setup. Launch again. No second table. A second transient stop is a real stop, unless it is a usage-limit stop (reference.md "Usage limit"). @@ -318,12 +323,15 @@ first, show its plan, and add `--yes` only on the user's yes. Keep the manifest file> [--spec ]`. Set `done`, `reviewed`, and `deferred` from it (taking a deferred task up: reference.md "Backfill"), and `repairs_spent` from earlier reports' `final.repairs`; put each `carry` entry into `notes`; copy `minors` into `carried_minors`, and `inputs` into - `inputs_changed` as `{plan: plan_changed, spec: spec_changed}` (reference.md "Carry-over"). Tasks in `stale` + `inputs_changed` as `{plan: plan_changed, spec: spec_changed}` (reference.md "Carry-over"). + Copy `join_started`, `unit_branches`, `wave_merges` and `start_bases` as they are + (reference.md "Join waves"). Tasks in `stale` (with their dependents) or `unbound` are left out of `reviewed`, so they are reviewed again: tell the user which and why, and say so when `inputs` reports a changed plan or spec. ` /scripts/coverage ` must exit 0 again (else fix the manifest's tasks or `excluded`). -3. `backfill`: ` /scripts/ledger backfill `. +3. `backfill`: ` /scripts/ledger backfill `, + and its `join_tip` when not null. Exit 3 lists records git does not confirm: show them and stop (reference.md "Backfill"). 4. Blocked tasks: show each reason; get the user's answer or plan fix before relaunching. Record an answer in `notes` as `{"": ""}`; a plan fix needs diff --git a/VERSION b/VERSION index 9edc58b..9f05f9f 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.6.4 +1.6.5 diff --git a/manifest.schema.json b/manifest.schema.json index cc0e28c..5ccf7ec 100644 --- a/manifest.schema.json +++ b/manifest.schema.json @@ -338,6 +338,45 @@ }, "additionalProperties": false }, + "join_started": { + "description": "1.6.5: ledger status join_started, a join task started in an earlier launch (the join then runs serially). Required once start_points.join is set.", + "type": "boolean" + }, + "unit_branches": { + "description": "1.6.5: ledger status unit_branches, the join units that ran on their own branch pl--u- in a lane worktree.", + "type": "object", + "additionalProperties": { + "type": "object", + "required": ["branch", "dir"], + "properties": { + "branch": { + "type": "string", + "pattern": "^pl-[a-z0-9-]+-u-[A-Za-z0-9_]" + }, + "dir": { + "$ref": "#/$defs/text" + } + } + } + }, + "wave_merges": { + "description": "1.6.5: ledger status wave_merges, the wave_merged events of earlier launches ({wave or recovery, units, head, overlaps, conflict_files}).", + "type": "array", + "items": { + "type": "object" + } + }, + "join_tip": { + "description": "1.6.5: ledger backfill join_tip, the join's last commit; integration's history audit allows final-fix commits after it.", + "$ref": "#/$defs/text" + }, + "start_bases": { + "description": "1.6.5: ledger status start_bases, the base each task not done started its latest attempt from; a join task resumed on the feature checkout after waves starts there.", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/text" + } + }, "done": { "$ref": "#/$defs/textList" }, diff --git a/reference.md b/reference.md index dc28a70..f66cf1c 100644 --- a/reference.md +++ b/reference.md @@ -14,6 +14,7 @@ attempt, with a worked example, is in adopt.md. `` below is - Flow details: notices, git identity, the confirmation header, transient stops - Building lanes: turning derive-lanes facts into prelude, lanes and join - Worth it: whether a layout saves enough to pay for a run's fixed phases +- Join waves: join tasks side by side on their own branches, merged after each wave - Profiles: full and lite - Tiers: standard, sonnet and light tasks, and escalation - Batching: light tasks run as one unit @@ -99,6 +100,11 @@ PATH, else `C:/Program Files/Git`), never the WSL `bash.exe` in System32. | `dry_run` | `true` only in the confirmation call. | | `done`, `reviewed` | `[]` for a new run; on resume, from `ledger status --plan`. Never by hand. | | `backfill` | Resume only: `{: {base, head}}` for done tasks, from `ledger backfill` (see Backfill below). Required for every done task; each `head` is the next task's review base. | +| `join_started` | Resume and transient relaunch: `ledger status`'s `join_started` (a join task started in an earlier launch). Required once `start_points.join` is set. A started join never runs waves again (see Join waves). | +| `unit_branches` | Resume and relaunch: `ledger status`'s `unit_branches`, `{: {branch, dir}}`: the join units that ran on their own branch. | +| `wave_merges` | Resume and relaunch: `ledger status`'s `wave_merges`, the earlier launches' wave merges (overlaps and conflicts for the final lenses). | +| `join_tip` | Resume and relaunch: `ledger backfill`'s `join_tip` when not null: the join's last commit, which integration's history audit allows final-fix commits after. | +| `start_bases` | Resume and relaunch: `ledger status`'s `start_bases`, `{: base}` for tasks not done, from their latest `started` event: a join task resumed on the feature checkout after waves starts there, so its earlier partial commits stay in its range. | | `notes` | Optional, resume: `{: ""}` for blocked questions; passed to that task's agents as "The user's answer for this task". | | `sp_dir` | Output of `find-superpowers`, or `null`. | | `agent_type` | Optional. The output of `bash /scripts/find-agent-type` (exit 0), else `null`. Recompute it at every launch, relaunch, and resume. When set, every agent except `e2e`, `post-integrate`, and `post-integrate fix` runs as that custom agent type (a lean toolset; hook instructions may need any tool); a spawn that fails with it (throws or returns no result) is retried once on the default type, and both attempts count toward `max_agents`. After the first such throw, or the second typed agent that returns no result while its retry succeeds, every later agent of the run uses the default type: the run log says so and the run result carries `agent_type_fallback: true`. | @@ -152,7 +158,9 @@ The header shown with the dry-run table lists: mode, base and feature branch, wo too. One line comes from the dry run's `checks` summary, for example "Checks: targeted entries for 9 of 12 units; delivery runs 3 checks (1 added by lane_commands); repairs up to 2", and its `warnings` follow it (today one: "changes to shared files will not widen task checks", when -`task_checks` has entries and `widen_paths` is empty). +`task_checks` has entries and `widen_paths` is empty). One line comes from the dry run's +`join_waves`: "Join: waves 1-2-4 (DJ1 | DJ2, DJ4 | DJ3, DJ5, DJ6, DJ7)", or "Join: serial +()" (see Join waves). One line says whether the run pauses at the usage limit (see "Usage limit"), from doctor's `usage_limit_pause` (`pause`, `reasons`) and what the session knows of itself: @@ -252,7 +260,8 @@ Otherwise: ## Worth it ` /scripts/worth-it --max-lanes [--prelude ] --lane -[--lane ...] [--join ] [--project ] [--history-dir ]` runs at the +[--lane ...] [--join ] [--join-deps : ...] [--project ] +[--history-dir ]` runs at the Assessment, after the layout and only when the Offer will be shown. It reads the plan only to check the layout: every id must be a task heading of the plan and appear once. @@ -287,6 +296,76 @@ Runs whose autopsy or manifest cannot be read are skipped, with a note. unreadable plan or a layout that names a task twice, names a task the plan lacks, or names none. Exit 0 in both cases (the Offer goes on as before); 2 on a usage error. Read-only. +## Join waves + +From 1.6.5 the launch that starts the join runs it in waves (design: the 1.6.5 join waves +design). The rule (`src/waves.js`, repeated in `scripts/_waves.py`): the join's units in plan +order (a light batch is one unit); a unit depends on every earlier unit when it (or a task of its +batch) has no `depends_on` field, else on the earlier units its `depends_on` names, and on the +earlier producer of each pre-flight finding against it. Its wave is one more than the highest +wave it depends on; a unit declaring a file (case-insensitive) an earlier unit of the wave +declares moves to the next wave. A wave runs at most `min(limits.max_parallel_lanes, usable lane +worktrees)` units at once; a lane with a `lane_commands` setup of its own is not usable. The join +runs serially (as before 1.6.5) under `profile: "lite"`, with fewer than two usable lane +worktrees, when no wave has two units, and when the join started in an earlier launch. + +- A single-unit wave runs on the feature checkout. A unit of a multi-unit wave runs on its own + branch `pl--u-` from the wave's base, in a lane worktree: + `start-task --unit-branch NAME SHA --run-id RUN --setup CMD ...` saves the worktree's + uncommitted changes under `refs/parallel-lanes//abandoned/` (unless it is already on + NAME: a retry), checks NAME out and runs the project's setup there. Its ledger lane, checks and + phase stay `join`. Once a unit of a wave stops, no further unit of it starts. +- Integration keeps the lane worktrees. After each multi-unit wave a wave merge agent (Sonnet) + runs `scripts/merge-wave DIR LEDGER_DIR --wave N --unit ID BASE HEAD ...`: it merges each + finished unit's recorded head with `join wave N: merge ID` (skipping one already merged), + aborts and reports a conflict, lists `overlaps` (paths two units changed), and records + `wave_merged`. A conflict goes to a wave resolver (Opus, one per conflict; on failure it aborts + the merge) in an autonomous run; a supervised run stops with the sha to merge by hand. A unit + that stopped is not merged, and the join stops after the merge. +- After the last multi-unit wave, its units' targeted checks (`task_checks`) run once on the + merged tip, with `--phase join`, and `wave_checked` is recorded. A failure gets one Opus fix + (`fix: make the wave checks pass`), its re-review and a recheck; still failing, an autonomous + run goes on (the final review and verify take it; `final.cannot_verify` lists it), a + supervised one stops. Checks that did not finish get no fix. +- A relaunch with `join_started` and `unit_branches`: setup puts each lane worktree left on one of + the run's unit branches back on its lane branch (output `returned`), saving its changes; a + recovery merge (`merge-wave --recovery`, `join recovery: merge ID`) brings in every unit that + ran on its own branch and is done and reviewed or deferred, and reports (`--known`, + `in_history`) which of the other such units an earlier launch already merged; the rest of the + join runs serially on the feature checkout from that tip. A merged unit whose approval went + stale is reviewed again over its recorded range without moving the tip. A unit that ran on its + own branch, is not done and reviewed, and is not in the history starts again from the tip + (`start-task --restart`: the ledger drops its earlier attempt), told its branch and abandoned + refs to reuse. Any other unit starts from its latest `started` base (`start_bases`), else the + tip. A unit branch an earlier launch left that is not built on the wave's base is saved under + the abandoned refs and made again (`start-task` prints `recreated`). Integration's history audit allows the + merge, resolver and wave checks fix commits, and final-fix commits after `join_tip`. +- `run-checks --phase NAME` (and `task-checks --phase`) sets `PL_PHASE=NAME` (prelude, lane, + join, integrate, post-integrate, final-fix, verify) for every check; without it `PL_PHASE` is + removed. A project's check script may read it: a join unit runs in a lane worktree. Task prompts + tell agents to set it for commands they run themselves. +- Ledger: appends hold each lane file's OS lock (up to 5 s); a reader skips a last line without a + newline that is not JSON. New events `wave_merged` and `wave_checked` (task `_run`); `started` + may carry `dir`, `unit_branch` and `restart`. `ledger status` adds `join_started`, + `unit_branches`, `wave_merges`, `start_bases`; `ledger backfill` skips unfinished join tasks + once the join started and adds `join_tip`. +- The dry run returns `join_waves` `{mode, reason, waves}` (before pre-flight's moves), counts the + wave merges (and the wave checks fix, re-review and recheck as the upper bound) or a relaunch's + recovery merge, and the launch notice ends with the wave sizes. Pre-flight is shown the waves; + a finding whose producer is earlier moves the task to a later wave (`preflight.wave_moves`). + The run result gains `join_waves` (per wave: units, head, conflict_files, resolved_files, + overlaps, checks), and the final lenses are told each wave's overlaps and resolved conflicts. +- `owners.json` gives join tasks a `wave` (two lanes or more, not lite); two join tasks of one + wave are `alongside`. Lane ids may not start with `u-`. `cleanup` claims lane worktrees on the + run's unit branches and removes merged unit branches; an unmerged one stays. autopsy reads the + wave labels (`wave N merge`, `wave N resolve ID`, `wave checks fix`, `wave checks re-review`, + `wave checks`, `recovery merge`, `recovery resolve ID`) as Join phase agents. A supervised + conflict stop gives the exact `git merge --no-ff -m 'join wave N: merge ID' ` to run by + hand, so integration's history audit accepts it on the relaunch. +- `worth-it --join-deps ID:DEPS` (one per join task) counts the join as the sum over its waves of + ceil(size / lanes at once), declared files not checked; without it, one slot per join task. + `tools/worth-it-backtest.py --waves` counts the archived joins in waves. + ## Profiles - `full` (default): prelude, lane worktrees, an integrate agent, a pre-flight agent, and three diff --git a/run.workflow.js b/run.workflow.js index 59b7029..cdc6be2 100644 --- a/run.workflow.js +++ b/run.workflow.js @@ -1,6 +1,6 @@ export const meta = { name: 'parallel-lanes', - description: 'parallel-lanes v1.6.4: Execute an implementation plan as parallel lanes of tasks with per-task review, then integrate.', + description: 'parallel-lanes v1.6.5: Execute an implementation plan as parallel lanes of tasks with per-task review, then integrate.', phases: [ { title: 'Pre-flight', detail: 'plan and spec conflicts' }, { title: 'Prelude', detail: 'shared tasks on the feature branch' }, @@ -400,6 +400,13 @@ function validateManifest(m) { else if (!new RegExp(laneIdPattern()).test(lane.id)) { err(`${where}.id: lane id ${JSON.stringify(lane.id)} must match ${laneIdPattern()}`); } else if (['prelude', 'join'].includes(lane.id.toLowerCase())) err(`lane ${lane.id}: id is reserved`); + else if (lane.id.toLowerCase().startsWith('u-')) { + // A lane's branch pl-- would then have a join unit + // branch's name, pl--u- (1.6.5; refs are + // case-insensitive on Windows). + err(`${where}.id: lane id ${JSON.stringify(lane.id)} must not start with u- ` + + '(join unit branches use pl--u-)'); + } else if (laneIds.has(lane.id)) err(`lane ${lane.id}: id appears more than once`); else { laneIds.add(lane.id); @@ -746,6 +753,43 @@ function validateManifest(m) { } } + // Join waves (1.6.5): what ledger status and backfill say about the join. + // Once integration recorded the join's start point, the session must say + // whether the join started (a started join never runs waves again). + if ('join_started' in m && typeof m.join_started !== 'boolean') err('join_started: must be true or false'); + if (isObject(m.start_points) && 'join' in m.start_points && !('join_started' in m)) { + err('join_started: missing; copy it from ledger status (start_points.join is set)'); + } + if ('join_tip' in m && !isText(m.join_tip)) err('join_tip: must be a non-empty string'); + if ('unit_branches' in m) { + if (!isObject(m.unit_branches)) { + err('unit_branches: must be an object'); + } else { + const joinTasks = new Set(Array.isArray(m.join) ? m.join.filter(isObject).map((t) => t.id) : []); + for (const [id, v] of Object.entries(m.unit_branches)) { + if (!joinTasks.has(id)) err(`unit_branches.${id}: not a join task`); + else if (!isObject(v) || typeof v.branch !== 'string' || !/^pl-[a-z0-9-]+-u-[A-Za-z0-9_]/.test(v.branch)) { + err(`unit_branches.${id}: branch must be a unit branch pl--u-`); + } else if (!isText(v.dir)) err(`unit_branches.${id}: dir must be a non-empty string`); + } + } + } + if ('start_bases' in m) { + if (!isObject(m.start_bases)) { + err('start_bases: must be an object'); + } else { + const all = new Set([m.prelude, ...(Array.isArray(m.lanes) ? m.lanes.map((l) => (isObject(l) ? l.tasks : [])) : []), m.join] + .flatMap((list) => (Array.isArray(list) ? list.filter(isObject).map((t) => t.id) : []))); + for (const [id, base] of Object.entries(m.start_bases)) { + if (!all.has(id)) err(`start_bases.${id}: not a task of the run`); + else if (!isText(base)) err(`start_bases.${id}: must be a non-empty string`); + } + } + } + if ('wave_merges' in m && !(Array.isArray(m.wave_merges) && m.wave_merges.every(isObject))) { + err('wave_merges: must be a list of objects'); + } + return errors; } @@ -879,7 +923,25 @@ function planAgents(m) { add('Integrate', null, null, 'integrate', sonnetHigh); if (m.hooks.post_integrate) add('Integrate', null, null, 'post_integrate', standard); } + // Join waves (1.6.5): a relaunch after waves starts with the recovery + // merge; a launch that runs waves merges each multi-unit wave, and when the + // last one has checks, lists their fix, its re-review and the recheck as + // the upper bound. + if (!lite && joinStarted(m) && Object.keys(m.unit_branches || {}).length > 0) { + add('Join', null, null, 'recovery_merge', sonnetHigh); + } addTasks('Join', null, m.join); + const joinPlan = lite ? null : wavePlan(m); + if (joinPlan && joinPlan.mode === 'waves') { + const multi = joinPlan.waves.filter((w) => w.length > 1); + for (let i = 0; i < multi.length; i += 1) add('Join', null, null, 'wave_merge', sonnetHigh); + const last = multi[multi.length - 1]; + if (waveChecksOf(m, joinUnits(m).filter((u) => last.includes(u.id))).length > 0) { + add('Join', null, null, 'wave_checks_fix', standard); + add('Join', null, null, 'wave_checks_re_review', standard); + add('Join', null, null, 'wave_checks', sonnetHigh); + } + } if (m.hooks.e2e) add('E2E', null, null, 'e2e', sonnetHigh); const lenses = lite ? ['final_review_combined'] @@ -898,8 +960,10 @@ function planAgents(m) { // agents; resume counts m.done. Expects a valid manifest. function launchNotices(m, agents) { const lanes = new Set(agents.filter((a) => a.lane !== null).map((a) => a.lane)).size; + const joinPlan = wavePlan(m); + const waves = joinPlan.mode === 'waves' ? `, join waves ${joinPlan.waves.map((w) => w.length).join('-')}` : ''; return { - launch: `parallel-lanes: launching run ${m.run_id}: ${lanes} lanes, ${agents.length} agents`, + launch: `parallel-lanes: launching run ${m.run_id}: ${lanes} lanes, ${agents.length} agents${waves}`, resume: `parallel-lanes: resuming run ${m.run_id}: ${m.done.length} tasks already committed`, }; } @@ -944,6 +1008,14 @@ function featureWhere(m, ledgerLane) { return { dir: featureDir(m), branch: m.repo.branch, lane: ledgerLane }; } +// Where a unit of a multi-unit join wave runs (1.6.5): its own branch in the +// lane worktree dir, created from the wave's base from; its ledger lane, its +// checks and its phase stay the join's, and there is no lane sync. +function unitWhere(m, unit, dir, from) { + const branch = unitBranch(m, unit); + return { dir, branch, lane: 'join', unit_branch: branch, from }; +} + // Scratch files for a task, kept in the ledger dir (outside every repo). function taskFiles(m, task) { const dir = m.repo.ledger_dir; @@ -1162,18 +1234,25 @@ function finalSetup(m) { return out; } +// The phase a ledger lane's checks run in (run-checks --phase, PL_PHASE): +// prelude and join tasks have their own; a lane's tasks run in lane. +function checksPhase(laneId) { + return laneId === 'prelude' || laneId === 'join' ? laneId : 'lane'; +} + // The scripts/run-checks call that runs each check of a [{group, command}] // list in dir, separately, and exits non-zero when any one fails. out // (optional) is the evidence file it writes under the ledger dir: // resultPath() goes in as given (already quoted), for the path the agent's -// reserve prints. null for an empty list. -function runChecksCommand(m, dir, checks, out = null) { +// reserve prints. phase (1.6.5) is run-checks' --phase: the commands run +// with PL_PHASE set to it. null for an empty list. +function runChecksCommand(m, dir, checks, out = null, phase = null) { if (checks.length === 0) return null; const parts = checks.flatMap((c) => ['--cmd', shellQuote(c.group), shellQuote(c.command)]); const target = out === resultPath() ? out : shellQuote(out); const evidence = out === null ? [] : ['--out', target, '--root', shellQuote(m.repo.ledger_dir)]; return [`cd ${shellQuote(dir)} && ${pythonCommand(m)} ${shellQuote(`${m.skill_dir}/scripts/run-checks`)}`, shellQuote(dir), - ...evidence, ...parts].join(' '); + ...evidence, ...(phase === null ? [] : ['--phase', shellQuote(phase)]), ...parts].join(' '); } // Every test, lint and build command of a ledger lane, as [{group, command}] @@ -1210,9 +1289,9 @@ function wideChecks(m, laneId) { } // The scripts/run-checks call for every test, lint and build command of a -// ledger lane in dir. null when there is no check command. -function checksCommand(m, laneId, dir, out = null) { - return runChecksCommand(m, dir, laneChecks(m, laneId), out); +// ledger lane in dir, in phase. null when there is no check command. +function checksCommand(m, laneId, dir, out = null, phase = null) { + return runChecksCommand(m, dir, laneChecks(m, laneId), out, phase); } // What an agent handed a run-checks call is told about its output and logs: @@ -1308,10 +1387,12 @@ function changeNothing() { // The general rule for any other long command (spec 4.4): it runs through // run-checks with the same procedure, under the reserve name long. Task -// prompts (taskContext) and phase prompts (phaseRules) carry it. -function longCheckRuleLines(m) { +// prompts (taskContext, with the task's phase) and phase prompts +// (phaseRules) carry it. +function longCheckRuleLines(m, phase = null) { const start = `cd 'DIR' && ${pythonCommand(m)} ${shellQuote(`${m.skill_dir}/scripts/run-checks`)} 'DIR' ` - + `--out ${resultPath()} --root ${shellQuote(m.repo.ledger_dir)} --cmd 'GROUP' 'COMMAND'`; + + `--out ${resultPath()} --root ${shellQuote(m.repo.ledger_dir)} ` + + `${phase === null ? '' : `--phase ${shellQuote(phase)} `}--cmd 'GROUP' 'COMMAND'`; return [ 'Long commands: any command you expect to run longer than 9 minutes that this prompt does not already hand you', 'as a check (a wide test filter, a long hook step) runs through run-checks, with DIR the checkout it runs in,', @@ -1322,11 +1403,11 @@ function longCheckRuleLines(m) { } // Project commands for a ledger lane; with dir, also the run-checks call that -// runs every check there at once, with the procedure under the reserve name -// name. -function commandsText(m, laneId, dir = null, name = 'checks') { +// runs every check there at once, in phase, with the procedure under the +// reserve name name. +function commandsText(m, laneId, dir = null, name = 'checks', phase = null) { const lines = ['setup', 'test', 'lint', 'build'].map((group) => `- ${group}: ${commandList(m, laneId, group)}`); - const all = dir === null ? null : checksCommand(m, laneId, dir, resultPath()); + const all = dir === null ? null : checksCommand(m, laneId, dir, resultPath(), phase); if (all !== null) { lines.push("- every check at once (each command's exit status is kept; it exits non-zero when any fails).", ...procedureLines(m, { name, start: all }).map((l) => ` ${l}`), ...checksLogLines()); @@ -1347,7 +1428,7 @@ function taskChecksCommand(m, task, where, base) { return [ `cd ${q(where.dir)} && ${pythonCommand(m)} ${q(`${m.skill_dir}/scripts/task-checks`)}`, q(where.dir), '--base', q(base), '--ledger', q(m.repo.ledger_dir), '--unit', q(task.id), '--evidence', resultPath(), - '--lane', q(where.lane), + '--lane', q(where.lane), '--phase', q(checksPhase(where.lane)), ...unitTasks(task).flatMap((t) => ['--task', q(t.id)]), ...declaredFiles(task).flatMap((f) => ['--declared', q(f)]), ...(Array.isArray(m.widen_paths) ? m.widen_paths : []).flatMap((p) => ['--widen', q(p)]), @@ -1556,12 +1637,15 @@ function taskContext(m, task, where, guidance = null, by = 'implementer') { '', 'Project commands (run from the worktree; the full set runs at integration and delivery):', commandsText(m, where.lane), + `Run project commands you start yourself with PL_PHASE=${checksPhase(where.lane)} set (for example`, + `\`PL_PHASE=${checksPhase(where.lane)} \`): the project's scripts may read it; the check calls`, + 'this prompt hands you set it themselves.', '', `Commit rules (follow exactly): ${m.commit_rules}`, 'Never commit anything under .superpowers/.', 'Never push, open pull requests, merge into the base branch, or copy work back to the project.', agentRules(), - ...longCheckRuleLines(m), + ...longCheckRuleLines(m, checksPhase(where.lane)), '', 'Contracts: report blocked for any change to a contract another lane consumes; ' + 'record smaller decisions as `Ruling: decision - why - cost if wrong` with:', @@ -1664,7 +1748,10 @@ function declaredFiles(task) { // superpowers present the review package for base..head. With opts.pkg and a // Files list it also lists the files base..head changes outside that list // (--scope); with opts.start (the task base, implement only) it records that -// base in the ledger once the briefs are written (--record-start). +// base in the ledger once the briefs are written (--record-start). For a +// join wave unit (where.unit_branch, 1.6.5) the implement start (opts.start) +// also puts the worktree on the unit's branch from where.from and runs the +// feature checkout's setup there (--unit-branch, --setup). function startCommand(m, task, where, opts = {}) { const sync = opts.sync || null; const pkg = opts.pkg || null; @@ -1673,6 +1760,10 @@ function startCommand(m, task, where, opts = {}) { shellQuote(where.dir), shellQuote(m.plan), '--artifacts', shellQuote(m.repo.ledger_dir), ]; if (present(sync)) parts.push('--sync', shellQuote(sync)); + if (present(where.unit_branch) && present(opts.start)) { + parts.push('--unit-branch', shellQuote(where.unit_branch), shellQuote(where.from), '--run-id', shellQuote(m.run_id), + ...finalSetup(m).flatMap((c) => ['--setup', shellQuote(c)])); + } if (pkg && m.sp_dir !== null) { const script = `${m.sp_dir}/subagent-driven-development/scripts/review-package`; const out = `${taskFiles(m, task).reviews}/${task.id}-${pkg.base}..${pkg.head}.diff`; @@ -1696,6 +1787,9 @@ function startCommand(m, task, where, opts = {}) { parts.push('--rulings', shellQuote(m.repo.ledger_dir), shellQuote(where.lane === 'join' ? '-' : where.lane)); if (present(opts.start)) { parts.push('--record-start', shellQuote(m.repo.ledger_dir), shellQuote(where.lane), shellQuote(opts.start)); + // A join unit begun again from the tip (1.6.5): the ledger drops its + // earlier attempt. + if (where.restart) parts.push('--restart'); } return parts.join(' '); } @@ -1705,12 +1799,16 @@ function startCommand(m, task, where, opts = {}) { function startBlock(m, task, where, opts, failure) { const what = [ ...(present(opts.sync) ? ['fast-forwards this worktree to the feature branch (it holds the prelude commits)'] : []), + ...(present(where.unit_branch) && present(opts.start) + ? [`puts this worktree on the unit's own branch ${where.unit_branch} (saving any uncommitted changes it finds ` + + 'under the run\'s abandoned refs) and runs the project\'s setup commands there'] : []), isBatch(task) ? 'regenerates every task brief from the current plan and prints it' : 'regenerates the task brief from the current plan and prints it', ...(opts.pkg && m.sp_dir !== null ? ['builds the review package'] : []), ...(opts.pkg && declaredFiles(task).length > 0 ? ["lists the files the range changes outside the Files list"] : []), - ...(present(opts.start) ? ['records the task base in the ledger'] : []), + ...(present(opts.start) ? [where.restart ? 'records the task base in the ledger as a restart' + : 'records the task base in the ledger'] : []), ]; const files = isBatch(task) ? 'brief files' : 'brief file'; return [ @@ -1832,6 +1930,26 @@ function reopenLines(m, task, where, reopen) { ]; } +// What a join unit begun again on the feature checkout is told (1.6.5 design +// 4.5): restart = {branch, dir, complete}, where its earlier attempt ran in a +// join wave of an earlier launch. That work never reached the feature branch. +function restartLines(m, task, restart, base) { + const q = shellQuote; + const dir = q(featureDir(m)); + const noun = unitNoun(task); + const folder = String(restart.dir).replace(/[\\/]+$/, '').split(/[\\/]/).pop().replace(/[^A-Za-z0-9._-]/g, '-'); + return [ + `This ${noun} ran before, in a join wave of an earlier launch, on its own branch ${restart.branch} in`, + `${restart.dir}, and that work never reached the feature branch. It was ${restart.complete + ? 'complete: committed, but its review did not finish' : 'partial: it stopped before it was done'}.`, + `Start again from the task base ${base} on the feature branch, and reuse what is right of the earlier work:`, + ` its commits: git -C ${dir} log HEAD..${q(restart.branch)} (and git -C ${dir} diff HEAD...${q(restart.branch)})`, + ` uncommitted changes saved from that worktree: git -C ${dir} for-each-ref ${q(`refs/parallel-lanes/${m.run_id}/abandoned/${folder}-`)}`, + ' (each such ref is a commit; read it with git show).', + 'Never merge, rebase or reset onto that branch as it is: bring over what is right as your own commits here.', + ]; +} + // Prompt for an implementer. base is the task base the script owns (the // previous task's head, or the feature tip); retry (optional) is {reason, // findings} when a previous attempt in this run blocked or failed review; @@ -1880,6 +1998,7 @@ function implementPrompt(m, task, where, base, retry = null, guidance = null, re ? ['Open review findings:', findingsText(retry.findings), ...startFindingNote(retry.findings)] : []), ].join('\n')); } + if (where.restart) parts.push('', restartLines(m, task, where.restart, base).join('\n')); const checks = taskChecksLines(m, task, where, base); if (checks.length > 0) parts.push('', checks.join('\n')); parts.push('', [ @@ -2314,7 +2433,10 @@ function relaunchPreflightLines(m) { ]; } -function preflightPrompt(m) { +// waves (1.6.5): the join's waves ([[unit id, ...], ...]) when the join runs in +// waves; pre-flight then also looks for a unit needing another unit of its own +// wave (design 4.7). +function preflightPrompt(m, waves = null) { return [ `You are the pre-flight reviewer for parallel-lanes run ${m.run_id}. You are read-only: do not modify any`, 'file, branch, or worktree.', @@ -2336,6 +2458,14 @@ function preflightPrompt(m) { ' lane or in the prelude produces (a function, a file format, markup, an API answer)', ' without naming that task in its Consumes. Return each in undeclared as', ' {task, producer, what}, what in one sentence.', + ...(waves === null ? [] : [ + '5. Join waves: the join runs in these waves. The units of a wave run side by side, each from the feature', + ' tip after the previous wave, without the work of the other units of their own wave:', + ...waves.map((w, i) => ` wave ${i + 1}: ${w.join(', ')}`), + ' For every task that needs work of another unit in its own wave (code, a contract, a test helper, a', + ' generated file), report it in code_deps (code) or undeclared (anything else) as {task, producer,', + ' what}: the run moves the task to a later wave.', + ]), 'Report serious problems (implementers would build the wrong thing, or check 1 or 2 fails) as conflicts,', 'one sentence each naming the tasks and plan or spec sections. Settle minor ambiguities yourself and report', 'each as a ruling in the form "Ruling: decision - why - cost if wrong".', @@ -2397,7 +2527,7 @@ function integratePrompt(m, preludeTip, opts = {}) { // and the lane_commands overrides, with the long-check procedure (1.6.1) // under the reserve name integrate, its logs beside the result. const setup = finalSetup(m); - const checks = runChecksCommand(m, featureDir(m), finalChecks(m), resultPath()); + const checks = runChecksCommand(m, featureDir(m), finalChecks(m), resultPath(), 'integrate'); // Heal mode reports checks that did not finish (1.6.2 spec 4.3): the // engine then starts no fix and stops. The other modes keep the general // exit-code rule. @@ -2416,6 +2546,7 @@ function integratePrompt(m, preludeTip, opts = {}) { ...checksLogLines().map((l) => ` ${l}`), ]), ]; + const keepWorktrees = opts.keepWorktrees || false; const cleanup = m.lanes.map((lane) => { const w = laneWhere(m, lane); return ` - ${q(w.dir)}: if git -C ${q(w.dir)} status --porcelain --untracked-files=all prints nothing, ` + @@ -2425,10 +2556,13 @@ function integratePrompt(m, preludeTip, opts = {}) { // Every join task committed in an earlier attempt: a final fix of that // attempt may have committed after the last join commit. const lastJoin = m.join.length > 0 ? (m.backfill || {})[m.join[m.join.length - 1].id] : null; - const joinDone = Boolean(lastJoin) && m.join.every((t) => m.done.includes(t.id)); + // With join waves (1.6.5) the join's last commit is the tip ledger + // backfill reports (join_tip), not the last plan-order task's head. + const joinTip = present(m.join_tip) ? m.join_tip : (lastJoin ? lastJoin.head : null); + const joinDone = present(joinTip) && m.join.every((t) => m.done.includes(t.id)); const finalFixes = joinDone ? [ ' Every join task was committed earlier in this run, so final-fix commits from an earlier attempt of', - ` this run may follow the last join commit ${lastJoin.head}: allow any commits after it. That includes`, + ` this run may follow the last join commit ${joinTip}: allow any commits after it. That includes`, ` repair commits after a failing verify (\`${repairFixMessage()}\`).`, ] : []; const tipLines = m.lanes.map((lane) => { @@ -2496,21 +2630,29 @@ function integratePrompt(m, preludeTip, opts = {}) { ' may list only merges (two parents) of the lane branches above, integration or post-integration fix', ' commits (yours, or from an earlier attempt in this run), and commits an earlier attempt at a join task', ` made (join tasks: ${joinIds}; their recorded commits are in ${m.repo.ledger_dir}/join.jsonl).`, + " A join that ran in waves also merged its units' heads ('join wave : merge ' and", + " 'join recovery: merge ', a resolver's included) and may have a wave checks fix", + ' (`fix: make the wave checks pass`).', ...finalFixes, " Any other commit landed on the feature branch outside the run's steps: fail listing it (do not", ' rewrite history).', `5. Committed scratch: git -C ${dir} diff --name-only ${q(`${m.repo.base_ref}...${m.repo.branch}`)}`, ' must list no path under .superpowers/; if it does, fail listing them (do not rewrite history).', - ...(testFailure === 'heal' ? [ - '6. Only when steps 1-5 passed and you return tests_failed false (every project command passes), clean', - ' up each lane. When you return tests_failed true, skip this step entirely: a fix agent and a rerun of', - ' these steps follow and the rerun cleans up (with checks_unfinished true the run stops instead and the', - ' lanes stay for the user). Cleanup:', - ] : ['6. Only when steps 1-5 passed, clean up each lane:']), - ...cleanup, - ' Leave a worktree with uncommitted files (and its branch) in place and list it in notes; never', - ' force a removal or a branch deletion. Cleanup never fails the integration: list anything step 6', - ' could not remove in notes and still return status done.', + ...(keepWorktrees ? [ + '6. Remove no lane worktree and delete no lane branch: the join runs in waves in the lane worktrees,', + " and the run's hand-back cleanup removes them.", + ] : [ + ...(testFailure === 'heal' ? [ + '6. Only when steps 1-5 passed and you return tests_failed false (every project command passes), clean', + ' up each lane. When you return tests_failed true, skip this step entirely: a fix agent and a rerun of', + ' these steps follow and the rerun cleans up (with checks_unfinished true the run stops instead and the', + ' lanes stay for the user). Cleanup:', + ] : ['6. Only when steps 1-5 passed, clean up each lane:']), + ...cleanup, + ' Leave a worktree with uncommitted files (and its branch) in place and list it in notes; never', + ' force a removal or a branch deletion. Cleanup never fails the integration: list anything step 6', + ' could not remove in notes and still return status done.', + ]), '', ...review, `Plan: ${m.plan}`, @@ -2556,6 +2698,227 @@ function resolveConflictsPrompt(m, preludeTip, conflictFiles) { ].join('\n'); } +// ---- Join waves (1.6.5 design 4.3, 4.4) ---- + +// The wave merge agent (Sonnet): scripts/merge-wave for the wave's finished +// units (units: [{id, base, head}], plan order) in the feature checkout; +// with checks (the last multi-unit wave's, waveChecksOf) and every unit +// merged, it runs them on the merged tip and records wave_checked. +// wave null: a relaunch's recovery merge (design 4.5), merge-wave --recovery. +// known ([{id, head}], recovery only): units that ran on their own branch and +// are not to be merged; merge-wave reports which are in the history already. +function waveMergePrompt(m, wave, units, checks = [], known = []) { + const q = shellQuote; + const dir = featureDir(m); + const recovery = wave === null; + const merge = [`cd ${q(dir)} && ${pythonCommand(m)} ${q(`${m.skill_dir}/scripts/merge-wave`)} ${q(dir)}`, + q(m.repo.ledger_dir), ...(recovery ? ['--recovery'] : ['--wave', q(String(wave))]), + ...units.flatMap((u) => ['--unit', q(u.id), q(u.base), q(u.head)]), + ...known.flatMap((u) => ['--known', q(u.id), q(u.head)])].join(' '); + const run = checks.length > 0 ? runChecksCommand(m, dir, checks, resultPath(), 'join') : null; + const record = (extra) => ledgerCommand(m, '_run', + { task: '_run', event: 'wave_checked', wave, head: '', ...extra }, dir); + return [ + recovery ? `You are the recovery merge agent for parallel-lanes run ${m.run_id}.` + : `You are the wave merge agent for parallel-lanes run ${m.run_id}, join wave ${wave}.`, + `Work in ${dir} on the feature branch ${m.repo.branch}; do not switch branches.`, + recovery ? (units.length > 0 + ? 'An earlier launch ran join units in waves; these finished and were reviewed but not merged. Merge their recorded heads.' + : 'An earlier launch ran join units in waves; none is left to merge. The command below only reports the head.') + : `The units of wave ${wave} ran side by side, each on its own branch; merge their recorded heads.`, + '', + '1. Run, as one call:', + ` ${merge}`, + " It merges each unit's head in this order (skipping one already merged), aborts a merge that", + ' conflicts and stops there, records the wave in the ledger, and prints one JSON line. Do not merge', + ' anything yourself, resolve a conflict, or change any file.', + '2. If it printed status conflict, stop: return status conflict with its head, merged, conflict_unit,', + ' conflict_files and overlaps. If it exited 2 or 3, stop: return status failed with its message.', + ...(run === null ? [] : [ + `3. It merged every unit: run the wave's checks on the merged tip (the last wave with more than one unit,`, + " whose merge no later task's checks cover). One run-checks call:", + ...procedureLines(m, { name: `wave-${wave}`, start: run }).map((l) => ` ${l}`), + ...checksLogLines().map((l) => ` ${l}`), + ' Never change code to make a check pass: a failure goes back to the run.', + `4. Record the result, with the full sha of git -C ${q(dir)} rev-parse HEAD:`, + ` every check passed: ${record({ ok: true })}`, + ` a check failed: ${record({ ok: false })}`, + ' every failing check exited 75, 124, 128 or above, or the checks stopped twice without a result:', + ` ${record({ ok: false, unfinished: true })}`, + ]), + '', + `Plan: ${m.plan}`, + keepFilesRule(), + phaseRules(m), + '', + 'Return status done (every unit merged), conflict or failed; head = the head merge-wave printed; merged,', + 'conflict_unit (empty when none), conflict_files, overlaps and in_history copied from its JSON;', + ...(run === null ? [] : ['checks = {ok, unfinished, notes} for the checks of step 3 (notes: the failing commands);']), + 'notes.', + ].join('\n'); +} + +function waveMergeSchema() { + return { + type: 'object', + properties: { + status: { type: 'string', enum: ['done', 'conflict', 'failed'] }, + head: { type: 'string' }, + merged: { type: 'array', items: { type: 'string' } }, + conflict_unit: { type: 'string' }, + conflict_files: { type: 'array', items: { type: 'string' } }, + in_history: { type: 'array', items: { type: 'string' } }, + overlaps: { + type: 'array', + items: { + type: 'object', + properties: { path: { type: 'string' }, units: { type: 'array', items: { type: 'string' } } }, + required: ['path', 'units'], + }, + }, + checks: { + type: 'object', + properties: { ok: { type: 'boolean' }, unfinished: { type: 'boolean' }, notes: { type: 'string' } }, + required: ['ok', 'unfinished', 'notes'], + }, + notes: { type: 'string' }, + }, + required: ['status', 'head', 'notes'], + }; +} + +// The wave resolver (Opus, autonomous): merges one unit's head that +// conflicted, keeping both sides' intent, and commits; on any failure it +// aborts the merge so the checkout is clean for a relaunch. +// wave null: a conflict of the recovery merge. +function waveResolvePrompt(m, wave, unit, conflictFiles) { + const q = shellQuote; + const dir = q(featureDir(m)); + const files = conflictFiles && conflictFiles.length > 0 ? conflictFiles.join(', ') : '(the files the merge reports)'; + const message = wave === null ? `join recovery: merge ${unit.id}` : `join wave ${wave}: merge ${unit.id}`; + return [ + `You are the wave resolver for parallel-lanes run ${m.run_id}, ${wave === null ? 'the recovery merge' : `join wave ${wave}`}.`, + `Work in ${featureDir(m)} on the feature branch ${m.repo.branch}; do not switch branches.`, + `Merging unit ${unit.id} (its recorded head ${unit.head}) conflicted in: ${files}.`, + '', + `1. git -C ${dir} status --porcelain --untracked-files=all must print nothing; otherwise return status`, + ' failed (never make it so by deleting, cleaning, or stashing files).', + `2. git -C ${dir} merge --no-ff -m ${q(message)} ${q(unit.head)}`, + " Resolve each conflict keeping the intent of both sides: read the plan tasks of the units that touched the", + ' file and keep what each meant to do, then commit the merge. Resolve the conflicts only; make no other', + ' change, run no project command, rewrite no history, and remove no worktree or branch.', + `3. If you cannot resolve it, run git -C ${dir} merge --abort so the checkout is clean, and return status`, + ' failed saying why.', + '', + `Plan: ${m.plan}`, + keepFilesRule(), + phaseRules(m), + '', + `Return status done when the merge is committed, else failed; head (the full sha printed by git -C ${dir}`, + 'rev-parse HEAD when you finish); notes (how you resolved each file).', + ].join('\n'); +} + +// The message the wave checks fix commits with. +function waveChecksFixMessage() { + return 'fix: make the wave checks pass'; +} + +// The wave checks fix (Opus, autonomous): one fix for the last multi-unit +// wave's failing checks, in the feature checkout; a re-review and a check-only +// rerun (waveChecksPrompt) follow. +function waveChecksFixPrompt(m, failure) { + const dir = shellQuote(featureDir(m)); + return [ + `You are fixing the failing wave checks of parallel-lanes run ${m.run_id}.`, + `Work in ${featureDir(m)} on ${m.repo.branch}; do not switch branches.`, + 'The units of the last join wave with more than one unit were merged; their targeted checks then failed:', + failure, + '', + 'Find the cause, fix it with the smallest change that is correct, and commit per the commit rules.', + commitWithLine(waveChecksFixMessage()), + 'You may rerun the failing commands named above to reproduce the failure and to confirm your fix; the run', + 'reruns the wave checks after you.', + '', + `Plan: ${m.plan}`, + `Spec: ${m.spec === null ? '(none)' : m.spec}`, + keepFilesRule(), + phaseRules(m), + '', + 'Return status done when your fix is committed and the failing checks pass, otherwise failed; head = the', + `full sha printed by git -C ${dir} rev-parse HEAD when you finish; notes = what you changed and the results.`, + ].join('\n'); +} + +// The re-review of a wave checks fix (Opus): the fix range only. +function waveChecksReReviewPrompt(m, base, head) { + return postIntegrateReReviewPrompt(m, base, head).replace('re-reviewing a post-integration fix', + 're-reviewing a fix for the failing join wave checks'); +} + +// The check-only rerun of the wave's checks after a fix (Sonnet). +function waveChecksPrompt(m, wave, checks) { + const q = shellQuote; + const dir = featureDir(m); + const run = runChecksCommand(m, dir, checks, resultPath(), 'join'); + const record = (extra) => ledgerCommand(m, '_run', + { task: '_run', event: 'wave_checked', wave, head: '', ...extra }, dir); + return [ + `You are rechecking join wave ${wave} of parallel-lanes run ${m.run_id} after a fix. Verify only: change no file,`, + `make no commit. Work in ${dir} on ${m.repo.branch}.`, + 'Run the wave checks, as one run-checks call:', + ...procedureLines(m, { name: `wave-${wave}`, start: run }), + ...checksLogLines(), + `Then record the result, with the full sha of git -C ${q(dir)} rev-parse HEAD:`, + ` every check passed: ${record({ ok: true })}`, + ` a check failed: ${record({ ok: false })}`, + ' every failing check exited 75, 124, 128 or above, or the checks stopped twice without a result:', + ` ${record({ ok: false, unfinished: true })}`, + '', + keepFilesRule(), + phaseRules(m), + '', + 'Return status done; head = that sha; ok, unfinished and notes (the failing commands) for the checks.', + ].join('\n'); +} + +function waveChecksSchema() { + return { + type: 'object', + properties: { + status: { type: 'string', enum: ['done', 'failed'] }, + head: { type: 'string' }, + ok: { type: 'boolean' }, + unfinished: { type: 'boolean' }, + notes: { type: 'string' }, + }, + required: ['status', 'head', 'ok', 'unfinished', 'notes'], + }; +} + +// What the final lenses are told about the join waves (design 4.9): each +// wave's paths two units changed, and the conflicts the resolver settled, +// from earlier launches (m.wave_merges, ledger status) and this one +// (m.join_waves_now, the run result's join_waves). +function waveLines(m) { + const lines = []; + const all = [...(Array.isArray(m.wave_merges) ? m.wave_merges : []), + ...(Array.isArray(m.join_waves_now) ? m.join_waves_now : [])]; + for (const w of all) { + const name = w.recovery ? 'the recovery merge' : `wave ${w.wave}`; + for (const o of Array.isArray(w.overlaps) ? w.overlaps : []) lines.push(`- ${name}: ${o.path} (${o.units.join(', ')})`); + const resolved = Array.isArray(w.resolved_files) && w.resolved_files.length > 0 ? w.resolved_files + : (Array.isArray(w.conflict_files) ? w.conflict_files : []); + if (resolved.length > 0) lines.push(`- ${name}: merge conflicts the wave resolver settled in ${resolved.join(', ')}`); + } + if (lines.length === 0) return []; + return [ + 'Join waves: units of a wave ran side by side on their own branches and were merged. Review these paths', + 'with every unit named in mind (both sides of a merge must still hold):', + ...lines, + ]; +} + // The message the final fix commits with. A function, not a constant: the // test harness's loadHelpers returns before top-level bindings initialize. function finalFixMessage() { @@ -2591,7 +2954,7 @@ function postIntegrateFixPrompt(m, failure, path = 'hook') { 'command.', ] : [ 'Rerun every project command afterwards and confirm they pass:', - commandsText(m, null, featureDir(m), 'post-integrate-fix'), + commandsText(m, null, featureDir(m), 'post-integrate-fix', 'post-integrate'), ]), m.hooks.post_integrate ? `Post-integration check to keep passing:\n${m.hooks.post_integrate}` : '', '', @@ -2640,7 +3003,7 @@ function postIntegratePrompt(m, checkOnly = false, checks = null) { ] : [ 'Change files only if the instructions call for it; commit any change per the commit rules and rerun', 'the project commands afterwards:', - commandsText(m, null, featureDir(m), 'post-integrate'), + commandsText(m, null, featureDir(m), 'post-integrate', 'post-integrate'), ]; return [ `You are the post-integration agent for parallel-lanes run ${m.run_id}.`, @@ -2769,6 +3132,7 @@ function finalReviewFrame(m, intro, focus, minors = [], other = false) { ...(other ? otherReviewerMinorsLines(m, minors) : taskMinorsLines(m, minors)), ...runRulingsLines(m), ...rulingsReadLines(m, featureDir(m)), + ...waveLines(m), "A finding that a ruling in force settles is not raised unless it is critical or important; then raise it and cite the ruling's id.", 'Files each task changed outside its Files list, with their owners (a read-only command):', ` ${ledgerReadCommand(m, featureDir(m), 'undeclared')}`, @@ -2829,7 +3193,7 @@ function finalFixPrompt(m, findings, base) { 'check at the delivered revision right after you, and a failure there goes to a repair agent.', ] : [ 'decline. Rerun every project command afterwards:', - commandsText(m, null, featureDir(m), 'final-fix'), + commandsText(m, null, featureDir(m), 'final-fix', 'final-fix'), ]), 'Commit your fixes per the commit rules.', commitWithLine(finalFixMessage()), @@ -2926,7 +3290,7 @@ function finalReReviewPrompt(m, base, head, findings, verify = null, repairs = [ function verifyPrompt(m, sha, from = null) { const dir = featureDir(m); const q = shellQuote; - const base = runChecksCommand(m, dir, finalChecks(m), resultPath()); + const base = runChecksCommand(m, dir, finalChecks(m), resultPath(), 'verify'); const cmd = from === null ? base : `${base} --descends-from ${q(from)}`; const setup = finalSetup(m); const stop = [ @@ -3484,6 +3848,167 @@ function agentTypeFor(m, label) { return m.agent_type; } +// ---- Join waves (1.6.5 design 4.1) ---- +// +// In the launch that starts the join, join units whose dependencies allow it +// run side by side, each on its own branch in a lane worktree, and each wave +// is merged back before the next starts. scripts/_waves.py repeats the rule +// for setup's owner map and worth-it; tests/waves.test.mjs holds both to the +// fixtures in tests/fixtures/waves/cases.json. + +// The branch a unit of a multi-unit wave runs on: pl--u-. +// Lane ids may not start with u-, so no lane branch has this form. +function unitBranch(m, unit) { + return `pl-${m.run_id}-u-${unit.id}`; +} + +// The ids a unit depends on: null when the unit (or any task of a batch) has +// no depends_on field (it then waits for every earlier unit); else the ids of +// every task's depends_on entries (objects with an id, or plain ids), less the +// unit's own task ids, without duplicates. A batch whose tasks are given as +// plain ids carries depends_on on the unit itself. +function waveDeps(unit) { + const tasks = Array.isArray(unit.tasks) ? unit.tasks : [unit]; + const objects = tasks.filter((t) => t !== null && typeof t === 'object'); + const holders = objects.length > 0 ? objects : [unit]; + if (holders.some((t) => !Array.isArray(t.depends_on))) return null; + const own = new Set(tasks.map((t) => (typeof t === 'string' ? t : t.id))); + const out = []; + for (const t of holders) { + for (const d of t.depends_on) { + const id = d !== null && typeof d === 'object' ? d.id : d; + if (!own.has(id) && !out.includes(id)) out.push(id); + } + } + return out; +} + +// A declared file as the wave rule compares it: '\' as '/', leading './' +// and repeated '/' dropped, lower-cased. +function waveFileKey(path) { + return String(path).replace(/\\/g, '/').replace(/\/+/g, '/').replace(/^(\.\/)+/, '').toLowerCase(); +} + +// The waves of the join's units, in plan order: [[unit id, ...], ...]. +// units: [{id, tasks?, depends_on?, files}] (a batch unit lists its tasks; +// its tasks' ids, plain or as objects, are what other units name). edges: +// [{unit, producer}] unit ids, pre-flight's findings. A unit depends on every +// earlier unit when waveDeps is null, else on the earlier units holding an id +// it names, and on each edge's producer that is earlier (later producers and +// later ids are ignored: validation refuses a code dependency on a later join +// task). Its wave is 1 + the highest wave it depends on; then, while an +// earlier unit of that wave declares one of its files (waveFileKey), the next. +function joinWaves(units, edges = []) { + const ids = (u) => (Array.isArray(u.tasks) ? u.tasks.map((t) => (typeof t === 'string' ? t : t.id)) : [u.id]); + const wave = []; + units.forEach((u, i) => { + const earlier = units.slice(0, i); + const deps = waveDeps(u); + const on = new Set(); + earlier.forEach((e, j) => { + if (deps === null || ids(e).some((id) => deps.includes(id))) on.add(j); + }); + for (const edge of edges) { + if (edge.unit !== u.id) continue; + const j = earlier.findIndex((e) => e.id === edge.producer); + if (j >= 0) on.add(j); + } + let w = 1 + Math.max(0, ...[...on].map((j) => wave[j])); + const mine = new Set((u.files || []).map(waveFileKey)); + const clash = (n) => earlier.some((e, j) => wave[j] === n && (e.files || []).some((f) => mine.has(waveFileKey(f)))); + while (clash(w)) w += 1; + wave.push(w); + }); + const out = []; + units.forEach((u, i) => { + while (out.length < wave[i]) out.push([]); + out[wave[i] - 1].push(u.id); + }); + return out.filter((w) => w.length > 0); +} + +// The join's units as the run forms them (batchGroup, batchUnit). +function joinUnits(m) { + const units = []; + for (let i = 0; i < m.join.length;) { + const group = batchGroup(m, m.join, i); + i += group.length; + units.push(group.length > 1 ? batchUnit(group) : group[0]); + } + return units; +} + +// Units of a wave that run at once: limits.max_parallel_lanes, capped by the +// lane worktrees a unit can use (a lane with a setup of its own in +// lane_commands is not one: its environment is not the project's). +function waveWidth(m) { + const own = m.lane_commands || {}; + const usable = m.lanes.filter((l) => !(Object.prototype.hasOwnProperty.call(own, l.id) + && own[l.id] && Object.prototype.hasOwnProperty.call(own[l.id], 'setup'))).length; + return Math.min(m.limits.max_parallel_lanes, usable); +} + +// The lane worktrees a wave unit can run in: those of the lanes waveWidth +// counts as usable, in lane order. +function waveLaneDirs(m) { + const own = m.lane_commands || {}; + return m.lanes.filter((l) => !(Object.prototype.hasOwnProperty.call(own, l.id) + && own[l.id] && Object.prototype.hasOwnProperty.call(own[l.id], 'setup'))).map((l) => laneWhere(m, l).dir); +} + +// The checks after the last multi-unit wave (design 4.4): the union of its +// units' targeted checks (their task_checks entries; a unit without one adds +// nothing), as [{group, command}] without repeats. +function waveChecksOf(m, units) { + const out = []; + for (const unit of units) { + const entry = taskChecksEntry(m, unit, 'join'); + for (const c of entry === null ? [] : entry.checks) { + if (!out.some((o) => o.group === c.group && o.command === c.command)) out.push(c); + } + } + return out; +} + +// The units a relaunch's recovery merge brings in (design 4.5): join units +// in plan order that ran on a unit branch (m.unit_branches) and whose every +// task is done and reviewed or deferred, as [{id, base, head}] from their +// backfill range; merge-wave skips one already in the feature history. +function recoveryUnits(m) { + const branches = m.unit_branches || {}; + const done = new Set(m.done || []); + const settled = new Set([...(m.reviewed || []), ...(m.deferred || [])]); + const out = []; + for (const unit of joinUnits(m)) { + const tasks = unitTasks(unit); + if (!tasks.some((t) => Object.prototype.hasOwnProperty.call(branches, t.id))) continue; + if (!tasks.every((t) => done.has(t.id) && settled.has(t.id))) continue; + const range = (m.backfill || {})[tasks[tasks.length - 1].id]; + if (range) out.push({ id: unit.id, base: range.base, head: range.head }); + } + return out; +} + +// The join started in an earlier launch: the ledger's join_started (copied +// into the manifest), or any join task done or deferred (fail-safe). +function joinStarted(m) { + if (m.join_started === true) return true; + const done = new Set([...(m.done || []), ...(m.deferred || [])]); + return m.join.some((t) => done.has(t.id)); +} + +// How the join runs: {mode: 'waves'|'serial', reason, waves}. waves is the +// rule's result either way; reason says why a join runs serially. +function wavePlan(m, edges = []) { + const waves = joinWaves(joinUnits(m), edges); + const serial = (reason) => ({ mode: 'serial', reason, waves }); + if (m.profile === 'lite') return serial('lite'); + if (joinStarted(m)) return serial('join started'); + if (waveWidth(m) < 2) return serial('fewer than two usable lane worktrees'); + if (!waves.some((w) => w.length > 1)) return serial('no wave has two units'); + return { mode: 'waves', reason: null, waves }; +} + // ---- Execution engine: per-task loop and lanes ---- // // Every function below is pure over its parameters (no top-level const/let), @@ -3953,6 +4478,177 @@ async function runTaskList(m, tasks, where, base, io, name, baseIsPhaseTip = fal return { results, stopped: null, head: prev }; } +// Run the join in waves (1.6.5 design 4.2, 4.3) from base, the feature tip. +// waves: [[unit id, ...], ...] from wavePlan. A single-unit wave runs its +// unit on the feature checkout from the tip, as a serial join does, and its +// head is the new tip. A multi-unit wave runs its units side by side, at +// most min(waveWidth(m), laneDirs.length) at once, each in a free lane +// worktree of laneDirs on its own branch from the tip (unitWhere); a dir is +// free again once its unit finished. Once a unit of the wave is not done or +// deferred, no further unit of it starts. Then merge(wave number, +// [{unit, result}] of the units that ran, in plan order, tip) merges the +// finished ones: {head} is the new tip, {stopped} ends the join there. A +// wave with a stopped unit ends the join after its merge, as a serial list +// stops at its first unfinished task. An unblocked unit's note goes to the +// later units that depend on it, else to every unit of the next wave. +// Returns {results (runTaskList's shape, plan order), stopped, head: the +// tip, waves: [{wave, units, head, merge}]}. +async function runJoinWaves(m, waves, base, laneDirs, io, merge) { + const units = new Map(joinUnits(m).map((u) => [u.id, u])); + const order = m.join.map((t) => t.id); + const results = []; + const waveLog = []; + const carried = new Map(); + const finished = (r) => r.status === 'done' || r.status === 'deferred'; + const noteOf = (unit) => unitTasks(unit).flatMap((t) => carried.get(t.id) || []).join('\n') || null; + const carry = (unit, w, text) => { + const ids = unitTasks(unit).map((t) => t.id); + const later = waves.slice(w + 1).flat().map((id) => units.get(id)); + const dependents = later.filter((u) => (waveDeps(u) || []).some((d) => ids.includes(d))); + for (const u of dependents.length > 0 ? dependents : (waves[w + 1] || []).map((id) => units.get(id))) { + for (const t of unitTasks(u).slice(0, 1)) { + carried.set(t.id, [...(carried.get(t.id) || []), `from ${unit.id}, unblocked by the adjudicator: ${text}`]); + } + } + }; + const record = (unit, r) => { + if (unitTasks(unit).length === 1) results.push(r); + else if (finished(r)) for (const t of unitTasks(unit)) results.push({ ...r, task: t.id, batch: unit.id }); + else results.push({ ...r, task: unitTasks(unit)[0].id, batch: unit.id }); + }; + const sorted = () => results.slice().sort((a, b) => order.indexOf(a.task) - order.indexOf(b.task)); + const run = async (unit, where, from) => { + try { + return await runTask(m, unit, where, from, io, null, noteOf(unit)); + } catch (e) { + return { task: unit.id, status: 'blocked', base: from, head: null, rounds: 0, tier_used: unit.tier, + notes: `error: ${e && e.message}`, rulings: [] }; + } + }; + let tip = base; + for (let w = 0; w < waves.length; w += 1) { + const wave = waves[w].map((id) => units.get(id)); + if (wave.length === 1) { + const r = await run(wave[0], featureWhere(m, 'join'), tip); + record(wave[0], r); + if (r.next_note) carry(wave[0], w, r.next_note); + if (!finished(r)) { + io.log(`Join: wave ${w + 1}: stopped at ${wave[0].id} (${r.notes})`); + return { results: sorted(), stopped: r.notes, head: tip, waves: waveLog }; + } + tip = r.head; + waveLog.push({ wave: w + 1, units: waves[w], head: tip, merge: null }); + continue; + } + const from = tip; + const ran = []; + let next = 0; + let stopped = null; + const worker = async (dir) => { + while (next < wave.length && stopped === null) { + const unit = wave[next]; + next += 1; + const r = await run(unit, unitWhere(m, unit, dir, from), from); + ran.push({ unit, result: r }); + if (!finished(r) && stopped === null) stopped = r.notes; + } + }; + const width = Math.min(waveWidth(m), laneDirs.length, wave.length); + io.log(`Join: wave ${w + 1}: ${waves[w].join(', ')} side by side (${width} at once)`); + await Promise.all(laneDirs.slice(0, width).map(worker)); + ran.sort((a, b) => wave.indexOf(a.unit) - wave.indexOf(b.unit)); + for (const { unit, result } of ran) { + record(unit, result); + if (result.next_note) carry(unit, w, result.next_note); + } + let merged = { head: tip }; + if (ran.some((x) => finished(x.result))) merged = await merge(w + 1, ran, from); + waveLog.push({ wave: w + 1, units: waves[w], head: merged.head || tip, merge: merged }); + if (present(merged.stopped)) { + io.log(`Join: wave ${w + 1}: merge stopped (${merged.stopped})`); + return { results: sorted(), stopped: merged.stopped, head: tip, waves: waveLog }; + } + tip = merged.head; + if (stopped !== null) { + io.log(`Join: wave ${w + 1}: stopped (${stopped})`); + return { results: sorted(), stopped, head: tip, waves: waveLog }; + } + } + return { results: sorted(), stopped: null, head: tip, waves: waveLog }; +} + +// Finish a join that ran in waves in an earlier launch (design 4.5), serially +// on the feature checkout from tip (the recovery merge's head), instead of +// runTaskList's plan-order chain: a task done and reviewed is skipped and +// does not move the tip; a task that ran on a unit branch (m.unit_branches) +// and is not done and reviewed starts again from the tip as a restart (its +// prompt names its earlier work); a task done but not reviewed that ran on +// the feature checkout, or whose unit's head is in the feature history +// (inHistory: unit ids the recovery merge found merged, its approval gone +// stale), is reviewed over its backfill range and moves the tip only when a +// fix moved it; any other task runs from its latest started base +// (m.start_bases, so an earlier attempt's partial commits stay in its range), +// else from the tip. Returns runTaskList's {results, stopped, head}, head +// the tip. +async function runJoinContinuation(m, tasks, where, tip, io, name, inHistory = new Set()) { + const results = []; + const branches = m.unit_branches || {}; + const own = (id) => Object.prototype.hasOwnProperty.call(branches, id); + const carried = new Map(); + const carry = (unit, i, text) => { + const ids = unitTasks(unit).map((t) => t.id); + const later = tasks.slice(i); + const dependents = later.filter((t) => (t.depends_on || []).some((d) => ids.includes(d.id))); + for (const t of dependents.length > 0 ? dependents : later.slice(0, 1)) { + carried.set(t.id, [...(carried.get(t.id) || []), `from ${unit.id}, unblocked by the adjudicator: ${text}`]); + } + }; + for (let i = 0; i < tasks.length;) { + const group = batchGroup(m, tasks, i); + i += group.length; + const task = group[0]; + const state = taskState(m, task.id); + const range = (m.backfill || {})[task.id]; + if (state === 'skip') { + results.push({ + task: task.id, status: 'skipped', base: range ? range.base : null, head: range ? range.head : null, + rounds: null, tier_used: null, notes: '', rulings: [], + }); + continue; + } + const unit = group.length > 1 ? batchUnit(group) : task; + const note = unitTasks(unit).flatMap((t) => carried.get(t.id) || []).join('\n') || null; + const ranOn = inHistory.has(unit.id) ? null : group.find((t) => own(t.id)); + let r; + if (ranOn) { + const b = branches[ranOn.id]; + r = await runTask(m, unit, { ...where, restart: { branch: b.branch, dir: b.dir, complete: state === 'review' } }, + tip, io, null, note); + } else if (state === 'review') { + r = range ? await runTask(m, unit, where, range.base, io, range, note) : { + task: task.id, status: 'blocked', base: tip, head: null, rounds: 0, tier_used: task.tier, + notes: 'done but not reviewed, and no backfill commits', rulings: [], + }; + } else { + const started = (m.start_bases || {})[task.id]; + r = await runTask(m, unit, where, present(started) ? started : tip, io, null, note); + } + if (r.next_note) carry(unit, i, r.next_note); + const finished = r.status === 'done' || r.status === 'deferred'; + if (unit === task) results.push(r); + else if (finished) for (const t of group) results.push({ ...r, task: t.id, batch: unit.id }); + else results.push({ ...r, task: task.id, batch: unit.id }); + if (!finished) { + io.log(`${name}: stopped at ${task.id} (${r.notes})`); + return { results, stopped: r.notes, head: tip }; + } + // A task reviewed on its recorded range is already in the history: it + // moves the tip only when a fix added a commit there. + if (!(state === 'review' && !ranOn && range && r.head === range.head)) tip = r.head; + } + return { results, stopped: null, head: tip }; +} + // Run a lane's tasks in order in its worktree, the first from base (the // feature tip the lane fast-forwards to); stop at the first task that is not // done. Returns {lane, results, stopped:reason|null, head}. @@ -4727,6 +5423,8 @@ async function runAll(manifest, io) { stoppedLanes.push({ lane, task: last && last.status !== 'done' ? last.task : null, reason: list.stopped }); }; let preflight = null; + // Pre-flight's findings between join units, as wave edges (design 4.7). + const waveEdges = []; let integrate = null; let e2e = null; let final = null; @@ -4738,6 +5436,13 @@ async function runAll(manifest, io) { let acceptance = null; // Set when a post-integrate fix's re-review returned no result (C2). let fixUnreviewed = false; + // Join waves (1.6.5): this launch's waves for the report (null for a + // serial join), the wave checks fix's re-review findings, whether that + // re-review returned no result, and the waves whose checks still fail. + let joinWaves = null; + const waveFixReview = []; + let waveFixUnreviewed = false; + const waveChecksOpen = []; // The post-integrate check's result and the revision it covered. let postCheck = null; const report = (status, reason = null) => ({ @@ -4747,6 +5452,7 @@ async function runAll(manifest, io) { stopped_lanes: stoppedLanes, preflight, integrate, + join_waves: joinWaves, e2e, final, verify, @@ -4832,7 +5538,9 @@ async function runAll(manifest, io) { io.log('parallel-lanes: pre-flight skipped: every task is done and reviewed'); } else { io.phase('Pre-flight'); - const pre = await call('pre-flight', 'Pre-flight', preflightPrompt(m), preflightSchema()); + const shown = wavePlan(m); + const pre = await call('pre-flight', 'Pre-flight', preflightPrompt(m, shown.mode === 'waves' ? shown.waves : null), + preflightSchema()); if (state.refused.length > 0) return budgetReport(); if (!pre) return report('stopped', 'no result from pre-flight'); // Undeclared dependencies only add context: they reach the briefs of @@ -4858,6 +5566,27 @@ async function runAll(manifest, io) { // carry-launch skips it (1.6.3); null when there is none. preflight = { conflicts: pre.conflicts, rulings: [...pre.rulings], undeclared: kept, schedule, adjudicated: null }; if (schedule.length > 0) return report('preflight_conflicts'); + // Findings between join units whose producer is earlier move the task's + // unit to a later wave (design 4.7); preflight.wave_moves reports them. + if (shown.mode === 'waves') { + const units = joinUnits(m); + const unitOf = (id) => units.find((u) => unitTasks(u).some((t) => t.id === id)); + for (const e of kept) { + const u = unitOf(e.task); + const p = unitOf(e.producer); + if (u && p && u !== p && units.indexOf(p) < units.indexOf(u)) waveEdges.push({ unit: u.id, producer: p.id }); + } + const waveOf = (waves, id) => waves.findIndex((w) => w.includes(id)) + 1; + const after = wavePlan(m, waveEdges).waves; + preflight.wave_moves = units.filter((u) => waveOf(shown.waves, u.id) !== waveOf(after, u.id)).map((u) => { + const edge = waveEdges.find((e) => e.unit === u.id); + return { unit: u.id, from: waveOf(shown.waves, u.id), to: waveOf(after, u.id), producer: edge ? edge.producer : null }; + }); + for (const mv of preflight.wave_moves) { + io.log(`parallel-lanes: pre-flight: ${mv.unit} moves from join wave ${mv.from} to ${mv.to}` + + `${mv.producer ? ` (it needs ${mv.producer})` : ''}`); + } + } m = withConsumesExtra(m, kept); // Pre-flight's rulings bind every task and final reviewer of this run // (taskContext and the final review prompts show m.run_rulings). @@ -4882,6 +5611,11 @@ async function runAll(manifest, io) { } } + // How the join runs (design 4.1): waves only in the launch that starts it, + // with pre-flight's findings as edges (design 4.7). + const joinPlan = wavePlan(m, waveEdges); + const wavesOn = joinPlan.mode === 'waves'; + io.phase('Prelude'); const prelude = await runTaskList(m, m.prelude, featureWhere(m, 'prelude'), starts.prelude || tip, counted, 'Prelude', true); @@ -4965,7 +5699,8 @@ async function runAll(manifest, io) { // cleanup deleted as merged when that commit is already in HEAD. const tips = laneTips(m, laneResults); let integ = await callM('integrate', 'Integrate', - integratePrompt(m, preludeTip, { conflictMode: 'abort', joinStartPoint: joinNoHook, laneTips: tips }), + integratePrompt(m, preludeTip, { conflictMode: 'abort', joinStartPoint: joinNoHook, laneTips: tips, + keepWorktrees: wavesOn }), statusSchema(), sonnetHigh); if (state.refused.length > 0) return budgetReport(); // Any Sonnet result that is not a finished integration (a done without a @@ -4992,6 +5727,7 @@ async function runAll(manifest, io) { resolverNotes: resolved ? resolved.notes : null, joinStartPoint: joinNoHook, laneTips: tips, + keepWorktrees: wavesOn, }), statusSchema(), standard); if (state.refused.length > 0) return budgetReport(); } @@ -5012,7 +5748,8 @@ async function runAll(manifest, io) { if (from === BUDGET) return budgetReport(); tip = from; integ = await callM('integrate', 'Integrate', - integratePrompt(m, preludeTip, { testFailure: 'heal', joinStartPoint: joinNoHook, laneTips: tips }), + integratePrompt(m, preludeTip, { testFailure: 'heal', joinStartPoint: joinNoHook, laneTips: tips, + keepWorktrees: wavesOn }), statusSchema(), standard); if (state.refused.length > 0) return budgetReport(); integrate = integrateReport(integ); @@ -5051,8 +5788,122 @@ async function runAll(manifest, io) { } io.phase('Join'); - const join = await runTaskList(m, m.join, featureWhere(m, 'join'), joinBase, - counted, 'Join', joinFromPhase); + let join; + const finished = (r) => r.status === 'done' || r.status === 'deferred'; + // Merge units ([{id, base, head}], plan order) into the feature checkout + // through merge-wave (design 4.3), wave null for a relaunch's recovery merge + // (design 4.5): each conflict goes to a resolver (autonomous) or stops the + // run with the sha to merge by hand (supervised). With checks (the last + // multi-unit wave's, design 4.4) the merge agent runs them; a failure gets + // one fix, its re-review and a recheck (autonomous) or stops (supervised). + // entry is the wave's report entry, filled in. Returns {head} or {stopped}. + const mergeUnits = async (wave, units, checks, entry, known = []) => { + const name = wave === null ? 'recovery' : `wave ${wave}`; + let r = null; + for (let attempt = 0; attempt <= units.length; attempt += 1) { + r = await callM(`${name} merge`, 'Join', waveMergePrompt(m, wave, units, checks, known), waveMergeSchema(), + sonnetHigh); + if (state.refused.length > 0) return { stopped: 'an agent was refused' }; + if (!r || r.status !== 'conflict') break; + const unit = units.find((u) => u.id === r.conflict_unit) + || units.find((u) => !(Array.isArray(r.merged) ? r.merged : []).includes(u.id)) || units[0]; + const files = Array.isArray(r.conflict_files) ? r.conflict_files : []; + for (const f of files) if (!entry.conflict_files.includes(f)) entry.conflict_files.push(f); + if (!autonomous) { + const message = wave === null ? `join recovery: merge ${unit.id}` : `join wave ${wave}: merge ${unit.id}`; + return { stopped: `join ${name}: ${unit.id} conflicts in ${files.join(', ') || 'files git reported'}; ` + + `merge it by hand with git -C ${shellQuote(featureDir(m))} merge --no-ff -m ${shellQuote(message)} ` + + `${unit.head}, resolve and commit, then relaunch` }; + } + const res = await callM(`${name} resolve ${unit.id}`, 'Join', waveResolvePrompt(m, wave, unit, files), + statusSchema(), standard); + if (state.refused.length > 0) return { stopped: 'an agent was refused' }; + if (!res || res.status !== 'done' || !isSha(res.head)) { + return { stopped: `join ${name}: the resolver could not merge ${unit.id}: ` + + `${res ? res.notes : `no result from ${name} resolve ${unit.id}`}` }; + } + for (const f of files) if (!entry.resolved_files.includes(f)) entry.resolved_files.push(f); + } + if (!r || r.status !== 'done' || !isSha(r.head)) { + return { stopped: `join ${name}: the merge failed: ${r ? r.notes : `no result from ${name} merge`}` }; + } + let head = r.head; + entry.head = head; + entry.overlaps = Array.isArray(r.overlaps) ? r.overlaps : []; + entry.in_history = Array.isArray(r.in_history) ? r.in_history : []; + if (checks.length > 0) { + const c = r.checks && typeof r.checks === 'object' ? r.checks + : { ok: false, unfinished: true, notes: `${name} merge returned no checks result` }; + entry.checks = { ok: c.ok === true, unfinished: c.ok !== true && c.unfinished === true, notes: c.notes || '', + fixed: false }; + if (!entry.checks.ok && !entry.checks.unfinished) { + if (!autonomous) return { stopped: `wave checks failed: ${entry.checks.notes}` }; + const fix = await callM('wave checks fix', 'Join', waveChecksFixPrompt(m, entry.checks.notes), + statusSchema(), standard); + if (state.refused.length > 0) return { stopped: 'an agent was refused' }; + if (fix && fix.status === 'done' && isSha(fix.head) && fix.head !== head) { + const rr = await callM('wave checks re-review', 'Join', waveChecksReReviewPrompt(m, head, fix.head), + postIntegrateReReviewSchema(), standard); + if (state.refused.length > 0) return { stopped: 'an agent was refused' }; + if (rr && Array.isArray(rr.findings)) waveFixReview.push(...rr.findings); + else waveFixUnreviewed = true; + head = fix.head; + entry.checks.fixed = true; + } + const again = await callM('wave checks', 'Join', waveChecksPrompt(m, wave, checks), waveChecksSchema(), + sonnetHigh); + if (state.refused.length > 0) return { stopped: 'an agent was refused' }; + entry.checks.ok = Boolean(again && again.ok === true); + entry.checks.unfinished = Boolean(again && !entry.checks.ok && again.unfinished === true); + entry.checks.notes = again ? again.notes || '' : 'no result from wave checks'; + if (!entry.checks.ok && !entry.checks.unfinished) waveChecksOpen.push({ wave, notes: entry.checks.notes }); + } + entry.head = head; + } + return { head }; + }; + const newEntry = (wave, units, head) => ({ wave, units, head, conflict_files: [], resolved_files: [], overlaps: [], + checks: null }); + if (wavesOn) { + const multi = joinPlan.waves.map((w, i) => (w.length > 1 ? i + 1 : 0)).filter((n) => n > 0); + const lastMulti = multi[multi.length - 1]; + const entries = []; + const mergeWave = async (wave, ran, base) => { + const units = ran.filter(({ result: r }) => r.status === 'done' + || (r.status === 'deferred' && isSha(r.head) && r.head !== r.base)) + .map(({ unit, result: r }) => ({ id: unit.id, base: r.base, head: r.head })); + const entry = newEntry(wave, joinPlan.waves[wave - 1], base); + entries.push(entry); + if (units.length === 0) return { head: base }; + const whole = ran.length === joinPlan.waves[wave - 1].length && ran.every((x) => finished(x.result)); + const checks = wave === lastMulti && whole ? waveChecksOf(m, ran.map((x) => x.unit)) : []; + return mergeUnits(wave, units, checks, entry); + }; + join = await runJoinWaves(m, joinPlan.waves, joinBase, waveLaneDirs(m), counted, mergeWave); + joinWaves = join.waves.map((w) => entries.find((e) => e.wave === w.wave) || newEntry(w.wave, w.units, w.head)); + m = { ...m, join_waves_now: joinWaves }; + } else if (!lite && joinStarted(m) && Object.keys(m.unit_branches || {}).length > 0) { + // A relaunch after waves (design 4.5): merge what the waves finished and + // had reviewed, then the rest serially from that tip. + const units = recoveryUnits(m); + // The other units that ran on their own branch: the merge reports which + // of them an earlier launch merged (their approval may have gone stale). + const known = joinUnits(m).filter((u) => !units.some((x) => x.id === u.id) + && unitTasks(u).some((t) => Object.prototype.hasOwnProperty.call(m.unit_branches, t.id))) + .map((u) => ({ id: u.id, range: (m.backfill || {})[unitTasks(u)[unitTasks(u).length - 1].id] })) + .filter((u) => u.range).map((u) => ({ id: u.id, head: u.range.head })); + const entry = { ...newEntry(null, units.map((u) => u.id), joinBase), recovery: true }; + const rec = await mergeUnits(null, units, [], entry, known); + if (units.length > 0) { + joinWaves = [entry]; + m = { ...m, join_waves_now: joinWaves }; + } + join = rec.stopped !== undefined ? { results: [], stopped: rec.stopped, head: joinBase } + : await runJoinContinuation(m, m.join, featureWhere(m, 'join'), rec.head, counted, 'Join', + new Set(entry.in_history || [])); + } else { + join = await runTaskList(m, m.join, featureWhere(m, 'join'), joinBase, counted, 'Join', joinFromPhase); + } record(join.results); if (join.stopped !== null) stopAt('join', join); if (state.refused.length > 0) return budgetReport(); @@ -5122,7 +5973,7 @@ async function runAll(manifest, io) { } io.phase('Final review'); - final = await runFinalReview(m, e2e, tip, counted, integrate ? integrate.fix_review : [], + final = await runFinalReview(m, e2e, tip, counted, [...(integrate ? integrate.fix_review : []), ...waveFixReview], hasChecks ? verifyAt : null, taskMinors, repair); // Checks that ran inside the final review (a fix agent ran) are the run's // verify: the report carries them once, at the top level. @@ -5141,6 +5992,14 @@ async function runAll(manifest, io) { final.cannot_verify.unshift({ requirement: 'the post-integrate re-review returned no result', source: 'run', why: 'the post-integration fix was delivered without a re-review', check_by: 're-review the post-integration fix' }); } + if (waveFixUnreviewed) { + final.cannot_verify.unshift({ requirement: 'the wave checks re-review returned no result', source: 'run', + why: 'the wave checks fix was delivered without a re-review', check_by: 're-review the wave checks fix' }); + } + for (const w of waveChecksOpen) { + final.cannot_verify.unshift({ requirement: `the join's wave ${w.wave} checks pass`, source: 'run', + why: `they still failed after the wave checks fix: ${w.notes}`, check_by: 'rerun them at the delivered revision' }); + } // Task minors live only in this launch's task results: the tasks an // earlier launch committed and reviewed are not re-run, so their approved // reviews' minor findings never reached the final lenses. Every resume @@ -5235,7 +6094,7 @@ function noHead(r, label) { // The skill version this script was built from (scripts/build stamps VERSION // here and in the meta description): the version actually running. Logged // first and returned with every result, dry runs included. -const VERSION = '1.6.4'; +const VERSION = '1.6.5'; log(`parallel-lanes v${VERSION} loaded`); if (args !== null && typeof args === 'object' && args.dry_run === true) { @@ -5252,6 +6111,9 @@ if (args !== null && typeof args === 'object' && args.dry_run === true) { notices: errors.length === 0 ? launchNotices(dry, agents) : null, // The check policy at a glance (spec 4.9); null when invalid. checks: errors.length === 0 ? checksSummary(dry) : null, + // How the join runs (1.6.5): {mode, reason, waves}, before pre-flight's + // moves; null when invalid. + join_waves: errors.length === 0 ? wavePlan(dry) : null, }; } diff --git a/scripts/_owners.py b/scripts/_owners.py index 41bd20e..36b1b9c 100644 --- a/scripts/_owners.py +++ b/scripts/_owners.py @@ -6,15 +6,25 @@ start-task --owners read it. {"repo": {"mode", "root", "git_dir"}, - "tasks": {id: {"where": "prelude" | | "join", "pos": n}}, + "tasks": {id: {"where": "prelude" | | "join", "pos": n, + "wave": n (join tasks, when the join can run in waves)}}, "files": {: [task ids, in run order]}} A task changing another task's file is judged by when that task runs: "earlier" (an earlier task of the same list, the prelude seen from a lane or -the join, a lane seen from the join), "alongside" (another lane), or "later". +the join, a lane seen from the join), "alongside" (another lane, or another +join task of the same wave, 1.6.5), or "later". A join task's wave is the +static rule's (_waves, without pre-flight's findings): stricter than a run +that moves a task to a later wave. """ import json +import os import posixpath +import sys + +sys.dont_write_bytecode = True +sys.path.insert(0, os.path.dirname(os.path.realpath(__file__))) +import _waves # noqa: E402 PHRASES = { "earlier": "earlier in the run", @@ -53,6 +63,13 @@ def build(manifest): key = path_key(f) if key is not None and t["id"] not in files.setdefault(key, []): files[key].append(t["id"]) + # A join that can run in waves (two lanes or more, not lite) records each + # join task's wave; one that cannot runs serially, so it records none. + join = manifest.get("join") or [] + if manifest.get("profile") != "lite" and len(manifest.get("lanes") or []) >= 2 and join: + for tid, wave in _waves.task_waves(join).items(): + if tid in tasks: + tasks[tid]["wave"] = wave return {"repo": {k: repo.get(k) for k in ("mode", "root", "git_dir")}, "tasks": tasks, "files": files} @@ -72,6 +89,10 @@ def relation(owners, unit_ids, other): if me is None or them is None: return None mine, theirs = me["where"], them["where"] + if mine == theirs == "join" and "wave" in me and "wave" in them: + if me["wave"] == them["wave"]: + return "alongside" + return "earlier" if them["wave"] < me["wave"] else "later" if mine == theirs: first = min(tasks[u]["pos"] for u in unit_ids if u in tasks) return "earlier" if them["pos"] < first else "later" diff --git a/scripts/_snapshot.py b/scripts/_snapshot.py new file mode 100644 index 0000000..3ac430e --- /dev/null +++ b/scripts/_snapshot.py @@ -0,0 +1,105 @@ +"""Save a worktree's uncommitted changes under the run's abandoned refs +before they are discarded. Shared by setup (a reused worktree) and +start-task (a lane worktree switched to a join unit's branch, 1.6.5). + +Import with: + sys.dont_write_bytecode = True + sys.path.insert(0, os.path.dirname(os.path.realpath(__file__))) + import _snapshot +""" + +import datetime +import os +import re +import subprocess +import sys +import tempfile + +sys.dont_write_bytecode = True + +# Variables that would point git at another repo than the one named. +GIT_REDIRECTS = ("GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE", + "GIT_OBJECT_DIRECTORY", "GIT_COMMON_DIR") +ENV = {k: v for k, v in os.environ.items() if k not in GIT_REDIRECTS} + + +class Failure(Exception): + """A git step failed: code is the exit code the caller should use.""" + + def __init__(self, code, message): + super().__init__(message) + self.code = code + + +def run_git(args, check=True): + res = subprocess.run(["git", *args], capture_output=True, text=True, encoding="utf-8", + errors="surrogateescape", env=ENV) + if check and res.returncode != 0: + raise Failure(1, f"git {' '.join(args)} failed: {res.stderr.strip()}") + return res + + +def snapshot(path, run_id, by="setup"): + """Save the changes of the worktree at path in commits and point a run + ref at the last one; the files in the worktree are untouched. + + The staged version first: git write-tree on the worktree's own index. + When that tree differs from HEAD's, it is committed on HEAD (the index + commit). Then every tracked change and untracked, non-ignored file goes, + through a temporary index, into a commit on top of the index commit, or + of HEAD when nothing was staged. write-tree can fail (an unmerged index, + a held index.lock): index_saved is then false, index_error holds git's + message, and the working snapshot is still made on HEAD. by names the + script in the commit messages. + Returns {"worktree", "ref", "commit", "index_commit", "index_saved"} and, + when index_saved is false, "index_error".""" + stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%S%fZ") + # The folder name as a ref component: run worktrees are lane- or + # feature; any other character a ref cannot hold (a space) becomes '-'. + name = re.sub(r"[^A-Za-z0-9._-]", "-", os.path.basename(os.path.normpath(path))).lstrip(".") or "worktree" + ref = f"refs/parallel-lanes/{run_id}/abandoned/{name}-{stamp}" + # Outside the raising helper: a failed write-tree is recorded, never fatal. + # It may refresh the index file, which the reset --hard after the snapshot + # replaces anyway. + staged = subprocess.run(["git", "-C", path, "write-tree"], capture_output=True, text=True, + encoding="utf-8", errors="surrogateescape", env=ENV) + index_saved = staged.returncode == 0 + index_commit = None + with tempfile.TemporaryDirectory() as tmp: + env = dict(ENV, GIT_INDEX_FILE=os.path.join(tmp, "index")) + + def git(*args): + res = subprocess.run(["git", "-C", path, *args], capture_output=True, text=True, + encoding="utf-8", errors="surrogateescape", env=env) + if res.returncode != 0: + raise Failure(1, f"saving the changes in {path} failed: git {args[0]}: {res.stderr.strip()}") + return res.stdout.strip() + + identity = ("-c", "user.name=parallel-lanes", "-c", "user.email=parallel-lanes@localhost") + if index_saved and staged.stdout.strip() != git("rev-parse", "--verify", "HEAD^{tree}"): + index_commit = git(*identity, "commit-tree", staged.stdout.strip(), "-p", "HEAD", "-m", + f"parallel-lanes {run_id}: staged changes in {path} saved before {by} discarded them") + git("read-tree", "HEAD") + git("add", "-A") + tree = git("write-tree") + commit = git(*identity, "commit-tree", tree, "-p", index_commit or "HEAD", "-m", + f"parallel-lanes {run_id}: uncommitted changes in {path} saved before {by} discarded them") + git("update-ref", ref, commit) + entry = {"worktree": path, "ref": ref, "commit": commit, "index_commit": index_commit, + "index_saved": index_saved} + if not index_saved: + entry["index_error"] = staged.stderr.strip() or f"git write-tree exited {staged.returncode}" + return entry + + +def discard_changes(path, run_id, discarded, preserved, by="setup"): + """Snapshot the worktree's uncommitted changes (when any) into preserved, + list each status line in discarded, then reset and clean the worktree.""" + status = run_git(["-C", path, "status", "--porcelain"]).stdout + for line in status.splitlines(): + if line: + discarded.append(f"{path}: {line}") + if status: + preserved.append(snapshot(path, run_id, by)) + run_git(["-C", path, "reset", "-q", "--hard", "HEAD"]) + run_git(["-C", path, "clean", "-q", "-fd"]) diff --git a/scripts/_waves.py b/scripts/_waves.py new file mode 100644 index 0000000..097f909 --- /dev/null +++ b/scripts/_waves.py @@ -0,0 +1,124 @@ +"""The join wave rule (1.6.5 design 4.1), as src/waves.js has it. + +The engine's JS rule is authoritative; this copy serves setup's owner map +(_owners.build) and worth-it. tests/waves.test.mjs holds both to the shared +fixtures in tests/fixtures/waves/cases.json. + +A unit is {"id", "files", optional "depends_on", optional "tasks"}: a batch +unit lists its tasks (ids, or task objects with an "id"). depends_on entries +are ids or objects with an "id". +""" +import re +import sys + +sys.dont_write_bytecode = True + + +def _id(item): + return item.get("id") if isinstance(item, dict) else item + + +def _task_ids(unit): + tasks = unit.get("tasks") + return [_id(t) for t in tasks] if isinstance(tasks, list) else [unit["id"]] + + +def deps(unit): + """The ids a unit depends on, less its own; None when it (or any task + of a batch given as task objects) has no depends_on field.""" + tasks = unit.get("tasks") + holders = [t for t in tasks if isinstance(t, dict)] if isinstance(tasks, list) else [] + if not holders: + holders = [unit] + if any(not isinstance(t.get("depends_on"), list) for t in holders): + return None + own = set(_task_ids(unit)) + out = [] + for t in holders: + for d in t["depends_on"]: + i = _id(d) + if i not in own and i not in out: + out.append(i) + return out + + +def file_key(path): + """A declared file as the rule compares it: '\\' as '/', repeated '/' + and leading './' dropped, lower-cased.""" + key = re.sub(r"/+", "/", str(path).replace("\\", "/")) + while key.startswith("./"): + key = key[2:] + return key.lower() + + +def waves(units, edges=()): + """The waves of units in plan order, as lists of unit ids. edges: + (unit_id, producer_id) pairs; a producer that is not earlier is + ignored, as is a depends_on id of a later unit.""" + edges = list(edges) + wave = [] + for i, unit in enumerate(units): + earlier = units[:i] + mine = deps(unit) + on = set() + for j, e in enumerate(earlier): + if mine is None or any(t in mine for t in _task_ids(e)): + on.add(j) + for u, producer in edges: + if u != unit["id"]: + continue + for j, e in enumerate(earlier): + if e["id"] == producer: + on.add(j) + w = 1 + max([wave[j] for j in on] or [0]) + files = set(file_key(f) for f in unit.get("files") or []) + + def clash(n): + return any(wave[j] == n and any(file_key(f) in files for f in e.get("files") or []) + for j, e in enumerate(earlier)) + + while clash(w): + w += 1 + wave.append(w) + out = [[] for _ in range(max(wave or [0]))] + for unit, w in zip(units, wave): + out[w - 1].append(unit["id"]) + return [w for w in out if w] + + +def join_units(join_tasks): + """The join's units, statically: consecutive tasks with the same + non-empty batch key form one unit {"id": "-", "tasks": + [ids], "files", "depends_on"} (depends_on None when a task has no + field); any other task is a unit of its own with the same keys.""" + units = [] + i = 0 + while i < len(join_tasks): + j = i + 1 + key = join_tasks[i].get("batch") + if key: + while j < len(join_tasks) and join_tasks[j].get("batch") == key: + j += 1 + group = join_tasks[i:j] + unit = {"id": group[0]["id"] if len(group) == 1 else "%s-%s" % (group[0]["id"], group[-1]["id"]), + "tasks": [t["id"] for t in group], + "files": [f for t in group for f in t.get("files") or []]} + unit["depends_on"] = deps({"id": unit["id"], "tasks": group}) + if unit["depends_on"] is None: + del unit["depends_on"] + units.append(unit) + i = j + return units + + +def task_waves(join_tasks): + """{task id: 1-based wave} for every join task (a batch's tasks share + their unit's wave).""" + units = join_units(join_tasks) + out = {} + for n, ids in enumerate(waves(units), 1): + for uid in ids: + unit = next(u for u in units if u["id"] == uid) + for t in unit["tasks"]: + out[t] = n + return out diff --git a/scripts/build b/scripts/build index 3b2ba24..2da6015 100755 --- a/scripts/build +++ b/scripts/build @@ -5,7 +5,7 @@ # # Concatenates, in this order: the meta literal of src/main.js (through its # first line that is exactly "};"), a generated-file comment, rules, -# validate, plan, prompts-task, prompts-phase, adjudicator, budget, engine, +# validate, plan, prompts-task, prompts-phase, adjudicator, budget, waves, engine, # phases, and the rest of src/main.js (the script body). Every @VERSION@ in # the output becomes the version in VERSION beside src/ (one x.y.z line; the # build fails and writes nothing otherwise), so the meta description and the @@ -35,7 +35,7 @@ while [ $# -gt 0 ]; do esac done -modules=(rules validate plan prompts-task prompts-phase adjudicator budget engine acceptance phases) +modules=(rules validate plan prompts-task prompts-phase adjudicator budget waves engine acceptance phases) for name in main "${modules[@]}"; do if [ ! -f "$src/$name.js" ]; then diff --git a/scripts/cleanup b/scripts/cleanup index 9329acd..33a4667 100755 --- a/scripts/cleanup +++ b/scripts/cleanup @@ -482,9 +482,17 @@ def make_plan(repo, m, run_folder): if repo.mode == "shadow": owned.append((repo.worktree_root + "/feature", repo_cfg["branch"])) + # A lane worktree a join wave left on one of the run's unit branches + # (pl--u-, 1.6.5) is the run's own too. + unit_prefix = "pl-" + run_id + "-u-" worktrees, owned_paths = [], [] for path, branch in owned: entry = owned_worktree(repo, listing, path, branch) + if entry is None and path.startswith(repo.worktree_root + "/lane-"): + here = next((e for e in listing if same_place(e["path"], path)), None) + if here is not None and (here["branch"] or "").startswith(unit_prefix): + branch = here["branch"] + entry = owned_worktree(repo, listing, path, branch) if entry is not None: worktrees.append(worktree_item(repo, entry, slash(path), branch)) owned_paths.append(path) @@ -508,6 +516,12 @@ def make_plan(repo, m, run_folder): sha = repo.sha("refs/heads/" + branch) if sha is not None: branches.append(lane_branch_item(repo, branch, sha, kept_at, head)) + # Join unit branches: removed like a lane branch, only once merged. + units = repo.out("for-each-ref", "--format=%(refname:short) %(objectname)", "refs/heads/" + unit_prefix + "*") + for line in units.splitlines(): + branch, _, sha = line.partition(" ") + if branch.startswith(unit_prefix): + branches.append(lane_branch_item(repo, branch, sha, kept_at, head)) feature_steps = [] sha = repo.sha("refs/heads/" + repo_cfg["branch"]) if sha is not None: diff --git a/scripts/ledger b/scripts/ledger index 8880e0e..982ddd0 100755 --- a/scripts/ledger +++ b/scripts/ledger @@ -33,7 +33,10 @@ and an 'event' of: reopened with 'reason': string (an approval that no longer holds: the task counts as not reviewed again) started with 'base': non-empty string (the task base, recorded by - start-task --record-start before the implementer writes) + start-task --record-start before the implementer writes); + a join wave unit (1.6.5) also has 'dir' (its worktree) and + 'unit_branch' (pl--u-), and a unit begun + again from the feature tip 'restart': true run_ended with 'status': non-empty string, and 'agents' and 'rulings': integers >= 0 (what one launch spent) accepted with 'text': non-empty string and 'head': non-empty string (the @@ -45,6 +48,17 @@ and an 'event' of: carried task _run only, with 'wf', 'pf' ({"PF-n": ruling id}) and 'bound' ([{file, line}]); carry-launch writes it last, once per launch + wave_merged task _run only, with 'units': non-empty list of unit ids, + 'head', and exactly one of 'wave' (integer >= 1) and + 'recovery': true; optional 'overlaps' ([{path, units}]) and + 'conflict_files' (paths); merge-wave writes it + wave_checked task _run only, with 'wave' (integer >= 1), 'head' and 'ok' + (true or false); 'unfinished': true when the checks stopped + without a result (ok false) +Appends to a lane file hold the file's own OS lock (_oslock), waiting up to +5 s for another appender (LEDGER_LOCK_WAIT_S overrides it for tests), else +exit 3 with nothing written. A reader skips a last line that has no newline +yet and is not JSON: an append in progress. A ruling may carry 'by' (implementer, reviewer, adjudicator, preflight; none: a task agent before 1.6.3, read as an implementer), 'scope' (lane, run), 'overrides' and 'files' (lists of strings), 'pf' (PF-n) and 'wf'. The values @@ -101,7 +115,16 @@ status reads every DIR/*.jsonl and prints "start_points": {...}, "carry": {...}, "ranges": {...}, "legacy": [...], "inconsistent": [...], "stale": [...], "unbound": [...], "inputs": {...}, "spent": {"agents", "rulings"}, "accepted": [{head, text}], - "undeclared": {...}} + "undeclared": {...}, "join_started": bool, "unit_branches": {...}, + "wave_merges": [...]} +join_started: join.jsonl holds a 'started' event (the join began in an +earlier launch: it then runs serially, 1.6.5). unit_branches: {task: {branch, +dir}} from each join task's latest 'started' event with a unit_branch, since +its latest restart. wave_merges: the _run wave_merged events in order, each +{wave or recovery, units, head, overlaps, conflict_files}. start_bases: {task: +base} of each task not done, from its latest 'started' event. A 'started' event +with restart true begins its task afresh: its earlier events (committed, +reviewed, blocked, settled) no longer count. done: tasks with a 'committed' or 'settled' event. reviewed: tasks with a 'reviewed' or 'settled' event after their last 'committed'. blocked: tasks with a 'blocked' event and no 'committed' or 'settled' after it. deferred: @@ -194,7 +217,14 @@ start_points.join), and head = the last commit of its last event; it is listed in derived. Every range is checked against git: base must be an ancestor of head, and a committed event's commits must be exactly git rev-list base..head of that event. Any failure is listed in errors and the exit is 3: a resume -must not narrow or guess a review range. +must not narrow or guess a review range. A list ends at its first task that +is not done, except the join once it started (join_started): a join that ran +in waves can have a later unit done beside an earlier one that stopped, so +its unfinished tasks are skipped. It also prints "join_tip": among the heads +of the wave_merged and wave_checked events and the backfill heads of join +tasks that did not run on a unit branch, the one every other is an ancestor +of (null when there is none); two heads neither of which descends from the +other are an error. Exit 0 on success; 2 on usage error or a malformed entry; 3 on an unsafe lane or task name, an unreadable ledger, plan or manifest, or a backfill error. @@ -213,6 +243,7 @@ import time sys.dont_write_bytecode = True sys.path.insert(0, os.path.dirname(os.path.realpath(__file__))) import _brief # noqa: E402 +import _oslock # noqa: E402 import _owners # noqa: E402 import _shell # noqa: E402 @@ -231,6 +262,8 @@ REQUIRED = { "accepted": "text", "minors": "open", "carried": "wf", + "wave_merged": "units", + "wave_checked": "head", } RULING_ROLES = ("implementer", "reviewer", "adjudicator", "preflight") RULING_SCOPES = ("lane", "run") @@ -239,6 +272,12 @@ PF_ID = re.compile(r"^PF-[1-9][0-9]*$") RUN_ONLY = ("minors", "carried") SETTLED_OUTCOMES = ("park", "unblock") START_POINTS = {"setup": "prelude", "join": "join"} +# Join waves (1.6.5): the _run events of a wave's merge and checks, and the +# branch form of a join unit's own branch (pl--u-). +WAVE_EVENTS = ("wave_merged", "wave_checked") +UNIT_BRANCH = re.compile(r"^pl-[a-z0-9-]+-u-[A-Za-z0-9_][A-Za-z0-9._-]*$") +# How long an append waits for another appender's lock on the same lane file. +LOCK_WAIT_S = 5.0 GIT_REDIRECTS = ("GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE", "GIT_OBJECT_DIRECTORY", "GIT_COMMON_DIR") ENV = {k: v for k, v in os.environ.items() if k not in GIT_REDIRECTS} @@ -306,6 +345,8 @@ def entry_error(entry): elif event in ("started", "run_ended"): if not is_text(value): return f"'{key}' must be a non-empty string" + if event == "started": + return started_fields_error(entry) if event == "run_ended": for field in ("agents", "rulings"): v = entry.get(field) @@ -320,6 +361,8 @@ def entry_error(entry): return ruling_fields_error(entry) elif event in RUN_ONLY: return run_only_error(entry) + elif event in WAVE_EVENTS: + return wave_error(entry) elif not isinstance(value, str): return f"'{key}' must be a string" return None @@ -346,6 +389,56 @@ def ruling_fields_error(entry): return None +def is_count(v, least): + return isinstance(v, int) and not isinstance(v, bool) and v >= least + + +def started_fields_error(entry): + """Why a started event's join wave fields (1.6.5) are malformed, or + None: dir, the worktree a unit ran in; unit_branch, its own branch; + restart (true), a unit begun again from the feature tip.""" + if "dir" in entry and not is_text(entry["dir"]): + return "'dir' must be a non-empty string" + if "unit_branch" in entry and not (isinstance(entry["unit_branch"], str) + and UNIT_BRANCH.fullmatch(entry["unit_branch"])): + return "'unit_branch' must be a branch of the form pl--u-" + if "restart" in entry and entry["restart"] is not True: + return "'restart' must be true when present" + return None + + +def wave_error(entry): + """Why a wave_merged or wave_checked event is malformed, or None.""" + if entry["task"] != "_run": + return f"a {entry['event']} event belongs to task _run" + if not is_text(entry.get("head")): + return "'head' must be a non-empty string" + if entry["event"] == "wave_merged": + if not (isinstance(entry.get("units"), list) and entry["units"] and is_text_list(entry["units"])): + return "'units' must be a non-empty list of unit ids" + has_wave, has_recovery = "wave" in entry, "recovery" in entry + if has_wave == has_recovery: + return "a wave_merged event has exactly one of 'wave' and 'recovery'" + if has_wave and not is_count(entry["wave"], 1): + return "'wave' must be an integer >= 1" + if has_recovery and entry["recovery"] is not True: + return "'recovery' must be true" + overlaps = entry.get("overlaps", []) + if not (isinstance(overlaps, list) and all(isinstance(o, dict) and is_text(o.get("path")) + and is_text_list(o.get("units")) for o in overlaps)): + return "'overlaps' must be a list of {path, units}" + if not is_text_list(entry.get("conflict_files", [])): + return "'conflict_files' must be a list of paths" + else: + if not is_count(entry.get("wave"), 1): + return "'wave' must be an integer >= 1" + if not isinstance(entry.get("ok"), bool): + return "'ok' must be true or false" + if "unfinished" in entry and not (entry["unfinished"] is True and entry["ok"] is False): + return "'unfinished' must be true, and only with ok false" + return None + + def run_only_error(entry): """Why a minors or carried event is malformed, or None.""" if entry["task"] != "_run": @@ -391,13 +484,37 @@ def append_entries(directory, lane, entries): os.chmod(directory, 0o700) line = "".join(json.dumps(e, ensure_ascii=True, separators=(",", ":")) + "\n" for e in entries) path = os.path.join(directory, f"{lane}.jsonl") - fd = os.open(path, os.O_WRONLY | os.O_APPEND | os.O_CREAT, 0o600) + # Join wave units append to one lane file at once (1.6.5), and an + # O_APPEND write is not atomic across processes on Windows: appends to a + # lane file take the file's own OS lock (_oslock: on Windows one byte far + # past the text, so readers and the append itself are not blocked), + # waiting a while for another appender. + fd = os.open(path, os.O_RDWR | os.O_APPEND | os.O_CREAT, 0o600) try: - os.write(fd, line.encode("ascii")) + wait = lock_wait_s() + deadline = time.monotonic() + wait + while not _oslock.try_lock(fd): + if time.monotonic() >= deadline: + fail(3, f"{lane}.jsonl stayed locked for {wait:g} s; nothing written") + time.sleep(0.05) + try: + # O_APPEND writes at the end wherever the lock left the offset. + os.write(fd, line.encode("ascii")) + finally: + _oslock.unlock(fd) finally: os.close(fd) +def lock_wait_s(): + """LOCK_WAIT_S, or LEDGER_LOCK_WAIT_S when set to a number >= 0 (tests).""" + try: + v = float(os.environ.get("LEDGER_LOCK_WAIT_S", LOCK_WAIT_S)) + except ValueError: + return LOCK_WAIT_S + return v if v >= 0 else LOCK_WAIT_S + + def append(directory, lane, raw): try: entry = json.loads(raw) @@ -468,15 +585,22 @@ def read_events_at(directory): path = os.path.join(directory, name) try: with open(path, encoding="ascii") as f: - lines = f.read().splitlines() + text = f.read() except (OSError, UnicodeDecodeError) as e: fail(3, f"cannot read {path}: {e}") + lines = text.splitlines() + # An append in progress: a last line with no newline yet that is not + # JSON is skipped, not an error (appenders write JSON and newline in + # one write, so a complete event always ends with a newline). + partial = len(lines) if text and not text.endswith("\n") else None for number, line in enumerate(lines, 1): if not line.strip(): continue try: entry = json.loads(line) except json.JSONDecodeError: + if number == partial: + continue entry = None error = "invalid JSON" if entry is None else entry_error(entry) if error: @@ -504,7 +628,14 @@ def summarize(events, plan_text=None, plan_given=False, spec_sha=None, plan_sha= if task not in history and want is not None and entry["base"] != want: start_mismatch[task] = (f"its first recorded range starts at {entry['base']}, not at the base " f"{want} it started at") - if event == "started": + if event == "started" and entry.get("restart") is True: + # A join unit begun again from the feature tip (1.6.5): its + # earlier attempt on its unit branch no longer counts. + for state in (done, reviewed_by, blocked, deferred, unblocked, last_ruling, review_event, + history, start_mismatch): + state.pop(task, None) + started[task] = entry["base"] + elif event == "started": evs = history.get(task) if evs and "base" in evs[0] and entry["base"] not in (evs[0]["base"], evs[-1].get("head")) \ and task not in start_mismatch: @@ -655,9 +786,63 @@ def summarize(events, plan_text=None, plan_given=False, spec_sha=None, plan_sha= "spent": spent, "accepted": accepted, "undeclared": undeclared, + # The base each task not done started its latest attempt from (1.6.5): + # a join task resumed on the feature checkout keeps its partial + # commits in its range. + "start_bases": {t: b for t, b in started.items() if not done.get(t)}, }, history +def join_facts(events_at): + """What the ledger says about the join's waves (1.6.5), from (file name, + line, event) triples: join_started (a started event in join.jsonl), + unit_branches ({task: {branch, dir}} from each join task's latest + started event with a unit_branch since its latest restart), and + wave_merges (the wave_merged events of _run.jsonl in order, as {wave or + recovery, units, head, overlaps, conflict_files}).""" + started, branches, merges = False, {}, [] + for name, _, entry in events_at: + event = entry["event"] + if name == "join.jsonl" and event == "started": + started = True + if entry.get("restart") is True: + branches.pop(entry["task"], None) + if "unit_branch" in entry: + branches[entry["task"]] = {"branch": entry["unit_branch"], "dir": entry.get("dir")} + elif name == "_run.jsonl" and event == "wave_merged": + item = {"wave": entry["wave"]} if "wave" in entry else {"recovery": True} + item.update({"units": entry["units"], "head": entry["head"], + "overlaps": entry.get("overlaps", []), "conflict_files": entry.get("conflict_files", [])}) + merges.append(item) + return {"join_started": started, "unit_branches": branches, "wave_merges": merges} + + +def join_tip(events, ranges, unit_branches, git): + """The join's tip: among the heads of wave_merged and wave_checked + events and the recorded heads of join tasks that did not run on a unit + branch, the one every other is an ancestor of. Returns (tip or None, + error or None).""" + heads = [] + for entry in events: + if entry["task"] == "_run" and entry["event"] in ("wave_merged", "wave_checked"): + heads.append(entry["head"]) + for task, rng in ranges.items(): + if task not in unit_branches: + heads.append(rng["head"]) + tip = None + for head in heads: + if tip is None: + tip = head + continue + if head == tip: + continue + if git("merge-base", "--is-ancestor", tip, head).returncode == 0: + tip = head + elif git("merge-base", "--is-ancestor", head, tip).returncode != 0: + return None, f"join_tip: heads {tip} and {head} do not descend from one another" + return tip, None + + def ruling_records(directory): """Every ruling as a dict: id (-R, n counting that file's rulings in append order), stem, file, line, role, scope and the event. @@ -999,10 +1184,12 @@ def undeclared(directory): def status(directory, plan=None, spec=None, manifest_path=None): plan_text = read_text(plan, "plan") if plan is not None else None manifest = load_manifest(manifest_path) if manifest_path is not None else None - events = read_events(directory) + events_at = read_events_at(directory) + events = [entry for _, _, entry in events_at] result, _ = summarize(events, plan_text, plan is not None, file_sha256(spec) if spec is not None else None, file_sha256(plan) if plan is not None else None, manifest) + result.update(join_facts(events_at)) snap = latest_minors(events) result["minors"] = {"open": [m.get("id") for m in snap["open"]] if snap else [], "covered": list(snap.get("covered") or []) if snap else []} @@ -1202,7 +1389,10 @@ def backfill(directory, manifest_path): lists = task_lists(m) git = repo_git(m) - summary, history = summarize(read_events(directory)) + events_at = read_events_at(directory) + events = [entry for _, _, entry in events_at] + summary, history = summarize(events) + facts = join_facts(events_at) done = set(summary["done"]) starts = m.get("start_points") or summary["start_points"] out, derived, errors = {}, [], [] @@ -1222,6 +1412,10 @@ def backfill(directory, manifest_path): for t in tasks: tid = t["id"] if tid not in done: + # A join that ran in waves (1.6.5) can have a later unit done + # beside an earlier one that stopped: each has its own range. + if name == "join" and facts["join_started"]: + continue break evs = history.get(tid, []) if tid in summary["ranges"]: @@ -1251,7 +1445,12 @@ def backfill(directory, manifest_path): prev = rng["head"] if name == "prelude": prelude_head = prev - print(json.dumps({"backfill": out, "derived": derived, "errors": errors})) + join_ids = {t["id"] for name, tasks in lists if name == "join" for t in tasks} + tip, tip_error = join_tip(events, {t: r for t, r in out.items() if t in join_ids}, + facts["unit_branches"], git) + if tip_error: + errors.append(tip_error) + print(json.dumps({"backfill": out, "derived": derived, "errors": errors, "join_tip": tip})) if errors: sys.exit(3) diff --git a/scripts/merge-wave b/scripts/merge-wave new file mode 100755 index 0000000..872d11f --- /dev/null +++ b/scripts/merge-wave @@ -0,0 +1,190 @@ +#!/usr/bin/env python3 +"""Merge a join wave's units into the feature checkout (1.6.5 join waves). + +usage: merge-wave DIR LEDGER_DIR (--wave N | --recovery) + [--unit ID BASE HEAD ...] [--known ID HEAD ...] + +DIR is the feature checkout, on the feature branch, and must be clean +(git status --porcelain --untracked-files=all prints nothing), else exit 3 +with nothing changed. Each --unit, in the order given (the engine passes +plan order), is a unit of the wave: its id, and the recorded base and head +of its range on its own branch (by sha, never by branch name). A unit whose +HEAD is already in DIR's history is skipped; any other is merged with +git merge --no-ff -m "join wave N: merge ID" (with --recovery, "join +recovery: merge ID": a relaunch bringing in units an earlier launch finished +but did not merge). A conflict aborts that merge (git merge --abort, so DIR +is clean again) and ends the loop: the units after it are not tried. + +Each --known ID HEAD (a relaunch's recovery merge) is a unit that ran on its +own branch and is not to be merged: in_history lists, after the merges, every +--unit then every --known whose HEAD is in DIR's history (a unit an earlier +launch merged), so the run does not redo merged work. + +overlaps: the paths that two or more units of this call changed (each +git diff --name-only BASE HEAD), among those now in DIR's history, as +[{path, units}] sorted by path, reported even when the merge was clean. + +When some unit is in DIR's history and this call merged one or hit a +conflict, it appends to LEDGER_DIR/_run.jsonl {"task": "_run", "event": +"wave_merged", "wave": N (or "recovery": true), "units": [the units in DIR's +history, in the given order], "head", "overlaps", "conflict_files"}. + +Prints one JSON object: + {"status": "merged"|"conflict", "head": , + "merged": [ids merged now], "skipped": [ids already merged], + "conflict_unit": id|null, "conflict_files": [paths], "overlaps": [...], + "in_history": [ids], "notes": [str]} +Exit 0 when every unit is merged (or there was none: --recovery may have +nothing to merge); 1 on a conflict; 2 on a usage error (no --wave or +--recovery, both, a --wave that is not an integer >= 1, --wave with no +--unit, an --unit without three values or with an unsafe id); 3 when DIR is +not a clean git checkout, a BASE or HEAD is not a commit, or a git step +fails. +""" + +import json +import os +import re +import subprocess +import sys + +sys.dont_write_bytecode = True +sys.path.insert(0, os.path.dirname(os.path.realpath(__file__))) +import _shell # noqa: E402 + +HERE = os.path.dirname(os.path.realpath(__file__)) +GIT_REDIRECTS = ("GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE", + "GIT_OBJECT_DIRECTORY", "GIT_COMMON_DIR") +ENV = {k: v for k, v in os.environ.items() if k not in GIT_REDIRECTS} +SAFE_ID = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9._-]*$") +USAGE = "usage: merge-wave DIR LEDGER_DIR (--wave N | --recovery) [--unit ID BASE HEAD ...] [--known ID HEAD ...]" + + +def fail(code, message): + print(f"merge-wave: {message}", file=sys.stderr) + sys.exit(code) + + +def git(directory, *args): + return subprocess.run(["git", "-C", directory, *args], capture_output=True, text=True, encoding="utf-8", + errors="surrogateescape", env=ENV) + + +def parse(argv): + if len(argv) < 2 or not argv[0] or not argv[1] or argv[0].startswith("-") or argv[1].startswith("-"): + fail(2, USAGE) + directory, ledger = _shell.native_path(argv[0]), _shell.native_path(argv[1]) + wave, recovery, units, known = None, False, [], [] + rest = argv[2:] + while rest: + if rest[0] == "--wave" and len(rest) >= 2 and wave is None: + if not re.fullmatch(r"[1-9][0-9]*", rest[1]): + fail(2, f"--wave {rest[1]!r} is not an integer >= 1") + wave, rest = int(rest[1]), rest[2:] + elif rest[0] == "--recovery" and not recovery: + recovery, rest = True, rest[1:] + elif rest[0] == "--unit" and len(rest) >= 4 and all(not v.startswith("--") for v in rest[1:4]): + if not SAFE_ID.fullmatch(rest[1]): + fail(2, f"unsafe unit id {rest[1]!r}") + units.append((rest[1], rest[2], rest[3])) + rest = rest[4:] + elif rest[0] == "--known" and len(rest) >= 3 and all(not v.startswith("--") for v in rest[1:3]): + if not SAFE_ID.fullmatch(rest[1]): + fail(2, f"unsafe unit id {rest[1]!r}") + known.append((rest[1], rest[2])) + rest = rest[3:] + else: + fail(2, USAGE) + if (wave is None) == (not recovery): + fail(2, "give exactly one of --wave N and --recovery") + if wave is not None and not units: + fail(2, "--wave needs at least one --unit") + return directory, ledger, wave, units, known + + +def commit_of(directory, ref): + res = git(directory, "rev-parse", "--verify", "--quiet", "--end-of-options", f"{ref}^{{commit}}") + if res.returncode != 0 or not res.stdout.strip(): + fail(3, f"{ref!r} is not a commit in {directory!r}; nothing changed") + return res.stdout.strip() + + +def head_of(directory): + res = git(directory, "rev-parse", "--verify", "HEAD^{commit}") + if res.returncode != 0: + fail(3, f"cannot read HEAD in {directory!r}: {res.stderr.strip()}") + return res.stdout.strip() + + +def changed(directory, base, head): + res = git(directory, "-c", "core.quotePath=false", "diff", "--name-only", "--end-of-options", base, head) + if res.returncode != 0: + fail(3, f"git diff {base} {head} failed: {res.stderr.strip()}") + return [p for p in res.stdout.splitlines() if p] + + +def main(argv): + _shell.setup_io() + directory, ledger, wave, units, known = parse(argv) + top = git(directory, "rev-parse", "--show-toplevel") + if top.returncode != 0: + fail(3, f"{directory!r} is not a git checkout") + dirty = git(directory, "status", "--porcelain", "--untracked-files=all") + if dirty.returncode != 0 or dirty.stdout.strip(): + fail(3, f"{directory!r} has uncommitted changes; nothing merged:\n{dirty.stdout.rstrip()}") + # Every sha resolves before the first merge: a bad one changes nothing. + units = [(uid, commit_of(directory, base), commit_of(directory, head)) for uid, base, head in units] + known = [(uid, commit_of(directory, head)) for uid, head in known] + + merged, skipped, notes = [], [], [] + conflict_unit, conflict_files = None, [] + for uid, _, head in units: + if git(directory, "merge-base", "--is-ancestor", head, "HEAD").returncode == 0: + skipped.append(uid) + continue + message = f"join recovery: merge {uid}" if wave is None else f"join wave {wave}: merge {uid}" + res = git(directory, "merge", "--no-ff", "--no-edit", "-m", message, "--end-of-options", head) + if res.returncode == 0: + merged.append(uid) + continue + files = git(directory, "-c", "core.quotePath=false", "diff", "--name-only", "--diff-filter=U") + conflict_files = sorted(p for p in files.stdout.splitlines() if p) + aborted = git(directory, "merge", "--abort") + if not conflict_files: + # Not a content conflict: git refused the merge for another reason. + notes.append(f"git merge of {uid} failed: {(res.stderr or res.stdout).strip()}") + if aborted.returncode != 0 and git(directory, "status", "--porcelain").stdout.strip(): + fail(3, f"git merge --abort failed in {directory!r}: {aborted.stderr.strip()}") + conflict_unit = uid + break + + head = head_of(directory) + inside = [u for u in units if u[0] in merged or u[0] in skipped] + in_history = [u[0] for u in inside] + [uid for uid, sha in known + if git(directory, "merge-base", "--is-ancestor", sha, "HEAD").returncode == 0] + by_path = {} + for uid, base, unit_head in inside: + for path in changed(directory, base, unit_head): + by_path.setdefault(path, []).append(uid) + overlaps = [{"path": p, "units": ids} for p, ids in sorted(by_path.items()) if len(ids) > 1] + + if inside and (merged or conflict_unit is not None): + event = {"task": "_run", "event": "wave_merged"} + event.update({"wave": wave} if wave is not None else {"recovery": True}) + event.update({"units": [u[0] for u in inside], "head": head, "overlaps": overlaps, + "conflict_files": conflict_files}) + res = subprocess.run([sys.executable, os.path.join(HERE, "ledger"), "append", ledger, "_run", + json.dumps(event)], capture_output=True, text=True, encoding="utf-8", + errors="replace", env=ENV) + if res.returncode != 0: + notes.append(f"recording wave_merged failed: {res.stderr.strip()}") + + print(json.dumps({"status": "conflict" if conflict_unit is not None else "merged", "head": head, + "merged": merged, "skipped": skipped, "conflict_unit": conflict_unit, + "conflict_files": conflict_files, "overlaps": overlaps, "in_history": in_history, + "notes": notes})) + sys.exit(1 if conflict_unit is not None else 0) + + +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/scripts/run-checks b/scripts/run-checks index a301851..825f5be 100755 --- a/scripts/run-checks +++ b/scripts/run-checks @@ -2,7 +2,13 @@ """Run a checkout's project commands one by one and keep every exit status. usage: run-checks DIR [--out FILE] [--root ROOT] [--descends-from SHA] - [--lock-held] --cmd GROUP COMMAND [--cmd ...] + [--lock-held] [--phase NAME] --cmd GROUP COMMAND [--cmd ...] + +--phase NAME (prelude, lane, join, integrate, post-integrate, final-fix or +verify, else a usage error) sets PL_PHASE=NAME for every command, the busy +retry included, so a project's check script can tell the run's phases apart +(a join unit runs in a lane worktree from 1.6.5); without --phase an inherited +PL_PHASE is removed. The JSON gains "phase": NAME, or null without it. Each COMMAND runs with bash -c in DIR (the bash _shell.find_bash resolves: Git Bash on Windows, never the WSL launcher), in the given order, and every @@ -145,7 +151,9 @@ BUSY_EXIT = 75 # sysexits EX_TEMPFAIL: busy or temporary, retried once BUSY_WAIT_S = 60 SHA = re.compile(r"^[0-9a-f]{7,40}$") USAGE = ("usage: run-checks DIR [--out FILE] [--root ROOT] [--descends-from SHA] [--lock-held] " - "--cmd GROUP COMMAND [--cmd GROUP COMMAND ...]") + "[--phase NAME] --cmd GROUP COMMAND [--cmd GROUP COMMAND ...]") +# The run phases a check can belong to (--phase, PL_PHASE). +PHASES = ("prelude", "lane", "join", "integrate", "post-integrate", "final-fix", "verify") # Where fail writes its error record: FILE once it is validated, unless # --lock-held says the caller writes FILE's records; None before that. @@ -177,7 +185,7 @@ def parse(argv): if not argv or not argv[0] or argv[0].startswith("-"): fail(2, USAGE) directory, rest = _shell.native_path(argv[0]), argv[1:] - out = root = descends = None + out = root = descends = phase = None lock_held = False cmds = [] while rest: @@ -187,6 +195,9 @@ def parse(argv): elif rest[0] == "--lock-held": lock_held = True rest = rest[1:] + elif rest[0] == "--phase" and len(rest) >= 2 and rest[1] in PHASES: + phase = rest[1] + rest = rest[2:] elif rest[0] in ("--out", "--root") and len(rest) >= 2 and rest[1]: if rest[0] == "--out": out = _shell.native_path(rest[1]) @@ -198,7 +209,7 @@ def parse(argv): rest = rest[2:] else: fail(2, USAGE) - return directory, out, root, cmds, descends, lock_held + return directory, out, root, cmds, descends, lock_held, phase def head_of(directory): @@ -376,10 +387,19 @@ def busy_wait_s(): return int(value) if value.isascii() and value.isdigit() else BUSY_WAIT_S -def run_command(command, directory, f): +def command_env(phase): + """The commands' environment: ENV with PL_PHASE=phase, or without + PL_PHASE when phase is None.""" + env = {k: v for k, v in ENV.items() if k != "PL_PHASE"} + if phase is not None: + env["PL_PHASE"] = phase + return env + + +def run_command(command, directory, f, env=None): """Run command with bash -c in directory, its stdout and stderr to the open log f; return its exit code.""" - res = subprocess.run(_shell.bash_argv("-c", command), cwd=directory, env=ENV, + res = subprocess.run(_shell.bash_argv("-c", command), cwd=directory, env=ENV if env is None else env, stdout=f, stderr=subprocess.STDOUT) return res.returncode @@ -387,7 +407,7 @@ def run_command(command, directory, f): def main(argv): global _record _shell.setup_io() - directory, out, root, cmds, descends, lock_held = parse(argv) + directory, out, root, cmds, descends, lock_held, phase = parse(argv) if out is not None and root is not None and (os.path.islink(out) or not _shell.contained(out, root)): fail(3, f"output path {out!r} is outside {root!r}") with contextlib.ExitStack() as stack: @@ -406,7 +426,7 @@ def main(argv): except OSError as e: fail(3, f"cannot lock {out!r}: {e}") try: - run(directory, out, root, cmds, descends) + run(directory, out, root, cmds, descends, phase) except _shell.BashNotFound as e: fail(3, str(e)) except Exception as e: @@ -414,8 +434,9 @@ def main(argv): raise -def run(directory, out, root, cmds, descends=None): +def run(directory, out, root, cmds, descends=None, phase=None): """The checks themselves; main has validated out against root.""" + env = command_env(phase) if cmds: # Found before anything runs: a missing bash fails no check halfway. _shell.find_bash() @@ -438,7 +459,7 @@ def run(directory, out, root, cmds, descends=None): "clean": not before, "tracked_before": tracked_before, "tracked_after": tracked_before, "untracked_before": untracked_lines(before), "untracked_after": untracked_lines(before), - "tree_changed": False, "descends_from": lineage} + "tree_changed": False, "descends_from": lineage, "phase": phase} emit(report, out) fingerprint_before = fingerprint(directory) results = [] @@ -446,7 +467,7 @@ def run(directory, out, root, cmds, descends=None): retry = None with open(log, "wb") as f: started, t0 = utc_stamp(), time.monotonic() - code = run_command(command, directory, f) + code = run_command(command, directory, f, env) if code == BUSY_EXIT: # Busy or temporary: wait, then run it once more into the same # log; the second exit is the result. @@ -457,7 +478,7 @@ def run(directory, out, root, cmds, descends=None): f.flush() time.sleep(waited) retry = {"first_exit": code, "waited_s": waited} - code = run_command(command, directory, f) + code = run_command(command, directory, f, env) duration, ended = time.monotonic() - t0, utc_stamp() result = {"group": group, "command": command, "exit": code, "started": started, "ended": ended, "duration_s": round(duration, 3), @@ -473,7 +494,7 @@ def run(directory, out, root, cmds, descends=None): "ok": all(r["exit"] == 0 for r in results) and after == head, "clean": not finish, "tracked_before": tracked_before, "tracked_after": tracked_after, "untracked_before": untracked_lines(before), "untracked_after": untracked_lines(finish), - "tree_changed": tree_changed(fingerprint_before, fingerprint_after)} + "tree_changed": tree_changed(fingerprint_before, fingerprint_after), "phase": phase} if after != head: report["head_after"] = after if lineage is not None: diff --git a/scripts/setup b/scripts/setup index 894519b..1b7bd30 100755 --- a/scripts/setup +++ b/scripts/setup @@ -111,20 +111,20 @@ run the setup commands with: the message names CLAUDE_CODE_GIT_BASH_PATH); unreadable (nothing changed). """ -import datetime import hashlib import json import os import re import subprocess import sys -import tempfile sys.dont_write_bytecode = True sys.path.insert(0, os.path.dirname(os.path.realpath(__file__))) import _locks # noqa: E402 import _owners # noqa: E402 import _shell # noqa: E402 +import _snapshot # noqa: E402 +from _snapshot import discard_changes, snapshot # noqa: E402,F401 (snapshot: tests load it from setup) SCRIPTS = os.path.dirname(os.path.abspath(__file__)) # Variables that would point git at another repo than the one named. @@ -136,10 +136,8 @@ RUN_ID = re.compile(r"^[a-z0-9-]+$") LANE_ID = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9._-]*$") -class Failure(Exception): - def __init__(self, code, message): - super().__init__(message) - self.code = code +# One exception class with _snapshot, whose git steps raise it too. +Failure = _snapshot.Failure def run_git(args, check=True): @@ -291,8 +289,10 @@ class Repo: self.git("worktree", "remove", norm) -def worktree_state(repo, path, branch): - """'missing' (absent or empty dir) or 'reuse'; raises on anything else. +def worktree_state(repo, path, branch, unit_prefix=None): + """'missing' (absent or empty dir), 'reuse', or 'unit' (a lane worktree + left on one of the run's join unit branches, whose name starts with + unit_prefix, 1.6.5); raises on anything else. A symlink is always refused, even one to an empty directory: git would create the worktree at the link's target, outside worktree_root. @@ -314,71 +314,13 @@ def worktree_state(repo, path, branch): or os.path.realpath(common) != repo.common_dir): raise refuse head = run_git(["-C", path, "symbolic-ref", "--quiet", "--short", "HEAD"], check=False) + if head.returncode == 0 and unit_prefix is not None and head.stdout.strip().startswith(unit_prefix): + return "unit" if head.returncode != 0 or head.stdout.strip() != branch: raise refuse return "reuse" -def snapshot(path, run_id): - """Save the changes of the worktree at path in commits and point a run - ref at the last one; the files in the worktree are untouched. - - The staged version first: git write-tree on the worktree's own index. - When that tree differs from HEAD's, it is committed on HEAD (the index - commit). Then every tracked change and untracked, non-ignored file goes, - through a temporary index, into a commit on top of the index commit, or - of HEAD when nothing was staged. write-tree can fail (an unmerged index, - a held index.lock): index_saved is then false, index_error holds git's - message, and the working snapshot is still made on HEAD. - Returns {"worktree", "ref", "commit", "index_commit", "index_saved"} and, - when index_saved is false, "index_error".""" - stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%S%fZ") - ref = f"refs/parallel-lanes/{run_id}/abandoned/{os.path.basename(path)}-{stamp}" - # Outside the raising helper: a failed write-tree is recorded, never fatal. - # It may refresh the index file, which the reset --hard after the snapshot - # replaces anyway. - staged = subprocess.run(["git", "-C", path, "write-tree"], capture_output=True, text=True, - encoding="utf-8", errors="surrogateescape", env=ENV) - index_saved = staged.returncode == 0 - index_commit = None - with tempfile.TemporaryDirectory() as tmp: - env = dict(ENV, GIT_INDEX_FILE=os.path.join(tmp, "index")) - - def git(*args): - res = subprocess.run(["git", "-C", path, *args], capture_output=True, text=True, - encoding="utf-8", errors="surrogateescape", env=env) - if res.returncode != 0: - raise Failure(1, f"saving the changes in {path} failed: git {args[0]}: {res.stderr.strip()}") - return res.stdout.strip() - - identity = ("-c", "user.name=parallel-lanes", "-c", "user.email=parallel-lanes@localhost") - if index_saved and staged.stdout.strip() != git("rev-parse", "--verify", "HEAD^{tree}"): - index_commit = git(*identity, "commit-tree", staged.stdout.strip(), "-p", "HEAD", "-m", - f"parallel-lanes {run_id}: staged changes in {path} saved before setup discarded them") - git("read-tree", "HEAD") - git("add", "-A") - tree = git("write-tree") - commit = git(*identity, "commit-tree", tree, "-p", index_commit or "HEAD", "-m", - f"parallel-lanes {run_id}: uncommitted changes in {path} saved before setup discarded them") - git("update-ref", ref, commit) - entry = {"worktree": path, "ref": ref, "commit": commit, "index_commit": index_commit, - "index_saved": index_saved} - if not index_saved: - entry["index_error"] = staged.stderr.strip() or f"git write-tree exited {staged.returncode}" - return entry - - -def discard_changes(path, run_id, discarded, preserved): - status = run_git(["-C", path, "status", "--porcelain"]).stdout - for line in status.splitlines(): - if line: - discarded.append(f"{path}: {line}") - if status: - preserved.append(snapshot(path, run_id)) - run_git(["-C", path, "reset", "-q", "--hard", "HEAD"]) - run_git(["-C", path, "clean", "-q", "-fd"]) - - def check_lock(run_id, owner): """Refuse unless the run's launch lock exists and owner matches it.""" lock = os.path.join(_locks.active_dir(), f"{run_id}.lock") @@ -470,7 +412,10 @@ def setup(m, owner=None): feature_state = (worktree_state(repo, feature, branch) if repo_cfg["mode"] == "shadow" else None) lanes = [] if lite else [(lane, lane_dir(m, lane), lane_branch(m, lane)) for lane in m["lanes"]] - lane_states = [worktree_state(repo, path, lb) for _, path, lb in lanes] + # A lane worktree a join wave left on a unit branch (1.6.5) goes back to + # its lane branch below; a lane with new work then runs as usual. + unit_prefix = "pl-" + m["run_id"] + "-u-" + lane_states = [worktree_state(repo, path, lb, unit_prefix) for _, path, lb in lanes] if any(setup_commands(m, lane_id) for lane_id in [None] + [lane["id"] for lane in m["lanes"]]): _shell.find_bash() @@ -478,6 +423,7 @@ def setup(m, owner=None): discarded = [] preserved = [] + returned = [] # Drops only the records of the run's own worktrees whose directory is gone. repo.prune_run_worktrees(repo_cfg["worktree_root"]) if not repo.has_branch(branch): @@ -488,7 +434,13 @@ def setup(m, owner=None): ensure_worktree(repo, feature, branch, base, feature_state, m["run_id"], discarded, preserved) check_branch(feature, branch) for (_, path, lb), state in zip(lanes, lane_states): - ensure_worktree(repo, path, lb, branch, state, m["run_id"], discarded, preserved) + if state == "unit": + unit = run_git(["-C", path, "symbolic-ref", "--short", "HEAD"]).stdout.strip() + discard_changes(path, m["run_id"], discarded, preserved) + run_git(["-C", path, "checkout", "-q", lb]) + returned.append({"worktree": path, "from": unit}) + else: + ensure_worktree(repo, path, lb, branch, state, m["run_id"], discarded, preserved) check_branch(path, lb) feature_cmds = setup_commands(m, None) @@ -527,7 +479,7 @@ def setup(m, owner=None): raise Failure(1, f"cannot write the owner map {owners_file!r}: {e}") worktrees = {lane["id"]: feature if lite else lane_dir(m, lane) for lane in m["lanes"]} return {"feature_head": head, "worktrees": worktrees, "discarded": discarded, "preserved": preserved, - "setup_skipped": skipped} + "setup_skipped": skipped, "returned": returned} def main(argv): diff --git a/scripts/start-task b/scripts/start-task index 598afd2..815d88d 100755 --- a/scripts/start-task +++ b/scripts/start-task @@ -5,10 +5,27 @@ usage: start-task DIR PLAN [--sync BRANCH] [--package SCRIPT BASE HEAD OUT] [--artifacts ROOT] [--record-start LEDGER_DIR LANE BASE] [--scope BASE HEAD --declared PATH [--declared PATH ...] [--owners FILE]] [--rulings LEDGER_DIR LANE] + [--unit-branch NAME SHA --run-id RUN [--setup COMMAND ...]] + [--restart] --brief TASK_ID OUT [--brief TASK_ID OUT ...] [--also TASK_ID PRODUCER_ID ...] In order: + --unit-branch a join wave unit (1.6.5) runs on its own branch NAME + (pl--u-) in a lane worktree DIR. When DIR is + already on NAME (a retry of the same unit) nothing changes + ("kept"). Otherwise DIR's uncommitted changes, if any, are + saved under refs/parallel-lanes/RUN/abandoned/ (the snapshot + setup makes of a reused worktree, _snapshot) and discarded, + and NAME is checked out ("switched"), or created at SHA when + it does not exist ("created"; a SHA that is not a commit exits + 3 before anything changes). A NAME an earlier launch left that + is not built on SHA has its head saved under the abandoned refs + (refs/parallel-lanes/RUN/abandoned/NAME-) and is made + again at SHA ("recreated"). Then each --setup COMMAND runs + with bash -c in DIR, in order (the feature checkout's setup: + the worktree's environment predates the merged lanes); one + that fails exits 1. Needs --run-id; not with --sync. --sync git -C DIR merge --ff-only BRANCH (a failure exits 3 and writes no brief). --brief scripts/task-brief PLAN TASK_ID OUT for each pair, in the given @@ -27,7 +44,11 @@ In order: "event": "started", "base": } for every --brief TASK_ID to LEDGER_DIR/LANE.jsonl (scripts/ledger append): the task base, recorded before the implementer writes. BASE must be - a commit in DIR, else exit 3 before anything is written. + a commit in DIR, else exit 3 before anything is written. With + --unit-branch the event also has "dir" (DIR, in the C:/ form on + Windows) and "unit_branch"; with --restart (which needs + --record-start), "restart": true: a join unit begun again from + the feature tip, whose earlier attempt the ledger then drops. --scope prints the paths git diff --name-only --no-renames BASE HEAD (in DIR) changes that are not a --declared PATH (normalized and case-insensitive; a rename changes both paths), one per line @@ -54,6 +75,8 @@ package script, so DIR alone names the repository. stdout, only on success: branch: head: + unit branch: NAME (created|switched|recreated|kept) (--unit-branch) + saved: () per snapshot --unit-branch made ===== brief : ===== then the brief text, per brief ===== rulings in force ===== then the rulings (--rulings) ===== review package ===== then the script's stdout (--package) @@ -68,9 +91,11 @@ task-brief's code if a brief fails, the ledger's if a started append fails. """ import argparse +import datetime import json import os import posixpath +import re import subprocess import sys @@ -80,10 +105,14 @@ sys.path.insert(0, HERE) import _brief # noqa: E402 import _owners # noqa: E402 import _shell # noqa: E402 +import _snapshot # noqa: E402 # Variables that would point git at another repo than the one named. GIT_REDIRECTS = ("GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE", "GIT_OBJECT_DIRECTORY", "GIT_COMMON_DIR") ENV = {k: v for k, v in os.environ.items() if k not in GIT_REDIRECTS} +# The same forms as the ledger's UNIT_BRANCH and validate.js' runIdPattern. +UNIT_BRANCH = re.compile(r"^pl-[a-z0-9-]+-u-[A-Za-z0-9_][A-Za-z0-9._-]*$") +RUN_ID = re.compile(r"^[a-z0-9-]+$") def fail(code, message): @@ -125,7 +154,22 @@ def parse(argv): parser.add_argument("--declared", action="append", default=[], metavar="PATH") parser.add_argument("--owners", metavar="FILE") parser.add_argument("--rulings", nargs=2, metavar=("LEDGER_DIR", "LANE")) + parser.add_argument("--unit-branch", nargs=2, metavar=("NAME", "SHA")) + parser.add_argument("--run-id", metavar="RUN") + parser.add_argument("--setup", action="append", default=[], metavar="COMMAND") + parser.add_argument("--restart", action="store_true") args = parser.parse_args(argv) + if args.unit_branch is not None: + if args.run_id is None or not RUN_ID.fullmatch(args.run_id): + parser.error("--unit-branch needs a --run-id of lowercase letters, digits and dashes") + if not UNIT_BRANCH.fullmatch(args.unit_branch[0]): + parser.error(f"--unit-branch {args.unit_branch[0]!r} is not of the form pl--u-") + if args.sync is not None: + parser.error("--unit-branch and --sync do not go together") + elif args.setup: + parser.error("--setup needs --unit-branch") + if args.restart and args.record_start is None: + parser.error("--restart needs --record-start") if args.owners is not None and args.scope is None: parser.error("--owners needs --scope") if args.scope is not None and not args.declared: @@ -171,6 +215,49 @@ def path_key(path): return posixpath.normpath(path).lower() +def switch_unit_branch(directory, args): + """Put DIR on the unit branch (--unit-branch NAME SHA) and run the + --setup commands there. Returns (NAME, the output lines).""" + name, sha = args.unit_branch + current = rev_parse(3, directory, "--abbrev-ref", "HEAD") + lines, preserved = [], [] + if current == name: + state = "kept" + else: + res = git(directory, "rev-parse", "--verify", "--quiet", "--end-of-options", f"{sha}^{{commit}}") + if res.returncode != 0 or not res.stdout.strip(): + fail(3, f"--unit-branch start {sha!r} is not a commit in {directory!r}; nothing changed") + start = res.stdout.strip() + old = git(directory, "rev-parse", "--verify", "--quiet", f"refs/heads/{name}") + exists = old.returncode == 0 + # A branch an earlier launch left that is not built on SHA (the tip + # moved since): its head is kept under the abandoned refs and the + # branch is made again at SHA, so the unit never builds on a stale base. + stale = exists and git(directory, "merge-base", "--is-ancestor", start, old.stdout.strip()).returncode != 0 + try: + _snapshot.discard_changes(directory, args.run_id, [], preserved, by="start-task") + except _snapshot.Failure as e: + fail(e.code, str(e)) + if stale: + stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%S%fZ") + ref = f"refs/parallel-lanes/{args.run_id}/abandoned/{name}-{stamp}" + if git(directory, "update-ref", ref, old.stdout.strip()).returncode != 0: + fail(1, f"saving the head of {name!r} under {ref} failed") + preserved.append({"ref": ref, "commit": old.stdout.strip()}) + res = git(directory, "checkout", "-q", name) if exists and not stale else \ + git(directory, "checkout", "-q", "-B" if stale else "-b", name, start) + if res.returncode != 0: + fail(1, f"checking out {name!r} in {directory!r} failed: {res.stderr.strip()}") + state = "recreated" if stale else ("switched" if exists else "created") + lines.append(f"unit branch: {name} ({state})\n") + lines += [f"saved: {p['ref']} ({p['commit']})\n" for p in preserved] + for command in args.setup: + res = subprocess.run(_shell.bash_argv("-c", command), cwd=directory, env=ENV) + if res.returncode != 0: + fail(1, f"setup command {command!r} exited {res.returncode} in {directory!r}") + return name, lines + + def main(argv): _shell.setup_io() try: @@ -198,6 +285,10 @@ def run(args): fail(3, f"--record-start base {args.record_start[2]!r} is not a commit in {directory!r}; nothing written") start_base = res.stdout.strip() + unit_lines = [] + if args.unit_branch is not None: + branch, unit_lines = switch_unit_branch(directory, args) + if args.sync is not None: res = git(directory, "merge", "--ff-only", "--end-of-options", args.sync) if res.returncode != 0: @@ -205,7 +296,7 @@ def run(args): fail(3, f"fast-forward of {directory!r} to {args.sync!r} failed: {detail}") head = rev_parse(1, directory, "HEAD") - out = [f"branch: {branch}\n", f"head: {head}\n"] + out = [f"branch: {branch}\n", f"head: {head}\n", *unit_lines] plan = _shell.native_path(os.path.abspath(args.plan)) for task_id, path in args.brief: extra = [a for t, producer in args.also if t == task_id for a in ("--also", producer)] @@ -277,7 +368,12 @@ def run(args): if start_base is not None: ledger_dir, lane, _ = args.record_start for task_id, _ in args.brief: - entry = json.dumps({"task": task_id, "event": "started", "base": start_base}) + event = {"task": task_id, "event": "started", "base": start_base} + if args.unit_branch is not None: + event.update({"dir": directory, "unit_branch": args.unit_branch[0]}) + if args.restart: + event["restart"] = True + entry = json.dumps(event) res = subprocess.run([sys.executable, os.path.join(HERE, "ledger"), "append", ledger_dir, lane, entry], env=ENV, capture_output=True, text=True, encoding="utf-8", errors="replace") if res.returncode != 0: diff --git a/scripts/task-checks b/scripts/task-checks index f897763..947dc52 100755 --- a/scripts/task-checks +++ b/scripts/task-checks @@ -6,10 +6,12 @@ usage: task-checks DIR --base SHA --ledger LEDGER_DIR --unit UNIT_ID --lane LANE [--project-test CMD ...] [--entry TASK_ID --targeted GROUP CMD [--targeted GROUP CMD ...]] --lane-cmd GROUP CMD [...] --wide-cmd GROUP CMD [...] - [--force-full REASON] [--evidence PATH] + [--force-full REASON] [--evidence PATH] [--phase NAME] Options come in any order. The workflow builds the call (spec 4.2 of the 1.6.0 check policy); the task's implementer runs it once before committing. +--phase NAME goes to run-checks, which sets PL_PHASE=NAME for the checks +(without it PL_PHASE is removed); NAME is one of run-checks' phases. Selection (policy.selected): lane no --entry: the unit has no task_checks entry, so the --lane-cmd @@ -133,9 +135,11 @@ PRINT_DETAIL = 2000 USAGE = ("usage: task-checks DIR --base SHA --ledger LEDGER_DIR --unit UNIT_ID --lane LANE " "--task ID [...] [--declared PATH ...] [--widen PATH ...] [--project-test CMD ...] " "[--entry TASK_ID --targeted GROUP CMD ...] --lane-cmd GROUP CMD [...] " - "--wide-cmd GROUP CMD [...] [--force-full REASON] [--evidence PATH]") + "--wide-cmd GROUP CMD [...] [--force-full REASON] [--evidence PATH] [--phase NAME]") +# run-checks' phases (--phase sets PL_PHASE for the checks). +PHASES = ("prelude", "lane", "join", "integrate", "post-integrate", "final-fix", "verify") SINGLE = {"--base": "base", "--ledger": "ledger", "--unit": "unit", "--lane": "lane", - "--entry": "entry", "--force-full": "forced", "--evidence": "evidence"} + "--entry": "entry", "--force-full": "forced", "--evidence": "evidence", "--phase": "phase"} REPEATED = {"--task": "tasks", "--declared": "declared", "--widen": "widen", "--project-test": "project_tests"} PAIRS = {"--targeted": "targeted", "--lane-cmd": "lane_cmds", "--wide-cmd": "wide_cmds"} @@ -181,7 +185,7 @@ def parse(argv): first usage error the reading met (checked by validate), or None. A token that is not an option is skipped.""" args = {"dir": None, "base": None, "ledger": None, "unit": None, "lane": None, "entry": None, - "forced": None, "evidence": None, "tasks": [], "declared": [], "widen": [], "project_tests": [], + "forced": None, "evidence": None, "phase": None, "tasks": [], "declared": [], "widen": [], "project_tests": [], "targeted": [], "lane_cmds": [], "wide_cmds": []} errors = [] rest = argv @@ -244,6 +248,8 @@ def validate(args): fail(2, f"--{key} is required") if not args["tasks"]: fail(2, "at least one --task is required") + if args["phase"] is not None and args["phase"] not in PHASES: + fail(2, f"--phase {args['phase']!r} is not one of {', '.join(PHASES)}") if not args["lane_cmds"] or not args["wide_cmds"]: fail(2, "--lane-cmd and --wide-cmd are required") if (args["entry"] is None) != (not args["targeted"]): @@ -470,6 +476,8 @@ def run_checks(args, chosen, policy, out): process holds), add policy, save, print, and exit.""" cmd = [sys.executable, os.path.join(HERE, "run-checks"), args["dir"], "--out", out, "--root", args["ledger"], "--lock-held"] + if args["phase"] is not None: + cmd += ["--phase", args["phase"]] for group, command in chosen: cmd += ["--cmd", group, command] res = subprocess.run(cmd, capture_output=True, text=True, encoding="utf-8", errors="replace", env=ENV) diff --git a/scripts/worth-it b/scripts/worth-it index f08a2cf..09b883a 100755 --- a/scripts/worth-it +++ b/scripts/worth-it @@ -4,7 +4,8 @@ for a run's fixed phases. usage: python3 worth-it PLAN --max-lanes N [--prelude IDS] --lane IDS [--lane IDS ...] - [--join IDS] [--project DIR] [--history-dir DIR] + [--join IDS] [--join-deps ID:DEPS ...] [--project DIR] + [--history-dir DIR] IDS are comma-separated task ids; --lane repeats, one per lane, in lane order and task order; --prelude, --join and a --lane may be empty. N is the lane @@ -13,7 +14,12 @@ cap the Assessment computed (an integer, at least 1). The layout: N tasks, L non-empty lanes, A = min(L, --max-lanes) lanes at once (at least 1). Task-slots S = prelude tasks + the makespan of the lanes' task counts assigned in order, each to the worker free earliest, across A -workers (the engine's runLanes) + join tasks. A direct build takes N +workers (the engine's runLanes) + the join. With --join-deps (1.6.5, one +per join task: ID, a colon, and the join ids it depends on joined by '+', +nothing for none; a join task without one depends on every earlier one), +the join counts the waves the engine runs it in (_waves, declared files not +checked): the sum over its waves of ceil(size / A); without, one slot per +join task (a serial join). A direct build takes N task-times; the layout saves N - S. recommend is "direct" when the saving is THRESHOLD or less, else "parallel". @@ -47,10 +53,11 @@ sys.dont_write_bytecode = True sys.path.insert(0, os.path.dirname(os.path.realpath(__file__))) import _history # noqa: E402 import _plan # noqa: E402 +import _waves # noqa: E402 import _shell # noqa: E402 USAGE = ("usage: worth-it PLAN --max-lanes N [--prelude IDS] --lane IDS [--lane IDS ...] [--join IDS] " - "[--project DIR] [--history-dir DIR]") + "[--join-deps ID:DEPS ...] [--project DIR] [--history-dir DIR]") # The back-test of 31 archived runs (2026-10-15, tools/worth-it-backtest.py; # the 1.6.4 design, section 4.3): the fixed phases a run adds beyond the @@ -61,7 +68,7 @@ USAGE = ("usage: worth-it PLAN --max-lanes N [--prelude IDS] --lane IDS [--lane THRESHOLD = 1 _SINGLE = ("--max-lanes", "--prelude", "--join", "--project", "--history-dir") -_OPTIONS = _SINGLE + ("--lane",) +_OPTIONS = _SINGLE + ("--lane", "--join-deps") class UsageError(Exception): @@ -70,15 +77,15 @@ class UsageError(Exception): def parse_args(args): """{plan, max_lanes, prelude, lanes, join, project, history_dir}.""" - opts = {"plan": None, "--lane": []} + opts = {"plan": None, "--lane": [], "--join-deps": []} i = 0 while i < len(args): a = args[i] if a in _OPTIONS: if i + 1 >= len(args) or args[i + 1] in _OPTIONS: raise UsageError(a + " needs a value") - if a == "--lane": - opts["--lane"].append(args[i + 1]) + if a in ("--lane", "--join-deps"): + opts[a].append(args[i + 1]) elif a in opts: raise UsageError(a + " given twice") else: @@ -107,6 +114,7 @@ def parse_args(args): "prelude": split_ids(opts.get("--prelude", "")), "lanes": [split_ids(v) for v in opts["--lane"]], "join": split_ids(opts.get("--join", "")), + "join_deps": parse_join_deps(opts["--join-deps"]), "project": opts.get("--project") or None, "history_dir": opts.get("--history-dir") or None, } @@ -116,13 +124,33 @@ def split_ids(value): return [x.strip() for x in value.split(",") if x.strip()] -def slots(prelude, lanes, join, at_once): +def parse_join_deps(values): + """{join id: [ids it depends on]} from ID:DEPS values (DEPS '+'-joined, + empty for none); a usage error for a value without ':' or an id given twice.""" + deps = {} + for value in values: + if ":" not in value: + raise UsageError("--join-deps needs ID:DEPS, not " + value) + task, _, rest = value.partition(":") + task = task.strip() + if not task or task in deps: + raise UsageError("--join-deps names " + (task or "no task") + (" twice" if task else "")) + deps[task] = [x.strip() for x in rest.split("+") if x.strip()] + return deps + + +def slots(prelude, lanes, join, at_once, join_waves=None): """prelude + the makespan of the lane sizes assigned in order, each to - the worker free earliest, across at_once workers + join.""" + the worker free earliest, across at_once workers + the join: join tasks, + or with join_waves (the join's wave sizes) the sum of ceil(size / + at_once).""" workers = [0] * max(1, min(at_once, len(lanes))) for size in lanes: i = workers.index(min(workers)) workers[i] += size + if join_waves is not None: + width = max(1, at_once) + join = sum((size + width - 1) // width for size in join_waves) return prelude + (max(workers) if lanes else 0) + join @@ -205,7 +233,17 @@ def assess(args): if error: return {"status": "unavailable", "reason": error, "notes": notes} at_once = max(1, min(len(lanes), args["max_lanes"])) - used = slots(len(args["prelude"]), [len(lane) for lane in lanes], len(args["join"]), at_once) + sizes = None + if args["join_deps"]: + unknown = [t for t in args["join_deps"] if t not in args["join"]] + if unknown: + return {"status": "unavailable", "reason": "--join-deps names " + ", ".join(unknown) + + ", not a task of --join", "notes": notes} + units = [dict({"id": t, "files": []}, **({"depends_on": args["join_deps"][t]} if t in args["join_deps"] else {})) + for t in args["join"]] + sizes = [len(w) for w in _waves.waves(units)] + notes.append("join waves " + "-".join(str(n) for n in sizes) + " (declared files not checked)") + used = slots(len(args["prelude"]), [len(lane) for lane in lanes], len(args["join"]), at_once, sizes) saved = len(layout) - used found = None if args["project"]: diff --git a/src/engine.js b/src/engine.js index a7e3742..e0ee579 100644 --- a/src/engine.js +++ b/src/engine.js @@ -467,6 +467,177 @@ async function runTaskList(m, tasks, where, base, io, name, baseIsPhaseTip = fal return { results, stopped: null, head: prev }; } +// Run the join in waves (1.6.5 design 4.2, 4.3) from base, the feature tip. +// waves: [[unit id, ...], ...] from wavePlan. A single-unit wave runs its +// unit on the feature checkout from the tip, as a serial join does, and its +// head is the new tip. A multi-unit wave runs its units side by side, at +// most min(waveWidth(m), laneDirs.length) at once, each in a free lane +// worktree of laneDirs on its own branch from the tip (unitWhere); a dir is +// free again once its unit finished. Once a unit of the wave is not done or +// deferred, no further unit of it starts. Then merge(wave number, +// [{unit, result}] of the units that ran, in plan order, tip) merges the +// finished ones: {head} is the new tip, {stopped} ends the join there. A +// wave with a stopped unit ends the join after its merge, as a serial list +// stops at its first unfinished task. An unblocked unit's note goes to the +// later units that depend on it, else to every unit of the next wave. +// Returns {results (runTaskList's shape, plan order), stopped, head: the +// tip, waves: [{wave, units, head, merge}]}. +async function runJoinWaves(m, waves, base, laneDirs, io, merge) { + const units = new Map(joinUnits(m).map((u) => [u.id, u])); + const order = m.join.map((t) => t.id); + const results = []; + const waveLog = []; + const carried = new Map(); + const finished = (r) => r.status === 'done' || r.status === 'deferred'; + const noteOf = (unit) => unitTasks(unit).flatMap((t) => carried.get(t.id) || []).join('\n') || null; + const carry = (unit, w, text) => { + const ids = unitTasks(unit).map((t) => t.id); + const later = waves.slice(w + 1).flat().map((id) => units.get(id)); + const dependents = later.filter((u) => (waveDeps(u) || []).some((d) => ids.includes(d))); + for (const u of dependents.length > 0 ? dependents : (waves[w + 1] || []).map((id) => units.get(id))) { + for (const t of unitTasks(u).slice(0, 1)) { + carried.set(t.id, [...(carried.get(t.id) || []), `from ${unit.id}, unblocked by the adjudicator: ${text}`]); + } + } + }; + const record = (unit, r) => { + if (unitTasks(unit).length === 1) results.push(r); + else if (finished(r)) for (const t of unitTasks(unit)) results.push({ ...r, task: t.id, batch: unit.id }); + else results.push({ ...r, task: unitTasks(unit)[0].id, batch: unit.id }); + }; + const sorted = () => results.slice().sort((a, b) => order.indexOf(a.task) - order.indexOf(b.task)); + const run = async (unit, where, from) => { + try { + return await runTask(m, unit, where, from, io, null, noteOf(unit)); + } catch (e) { + return { task: unit.id, status: 'blocked', base: from, head: null, rounds: 0, tier_used: unit.tier, + notes: `error: ${e && e.message}`, rulings: [] }; + } + }; + let tip = base; + for (let w = 0; w < waves.length; w += 1) { + const wave = waves[w].map((id) => units.get(id)); + if (wave.length === 1) { + const r = await run(wave[0], featureWhere(m, 'join'), tip); + record(wave[0], r); + if (r.next_note) carry(wave[0], w, r.next_note); + if (!finished(r)) { + io.log(`Join: wave ${w + 1}: stopped at ${wave[0].id} (${r.notes})`); + return { results: sorted(), stopped: r.notes, head: tip, waves: waveLog }; + } + tip = r.head; + waveLog.push({ wave: w + 1, units: waves[w], head: tip, merge: null }); + continue; + } + const from = tip; + const ran = []; + let next = 0; + let stopped = null; + const worker = async (dir) => { + while (next < wave.length && stopped === null) { + const unit = wave[next]; + next += 1; + const r = await run(unit, unitWhere(m, unit, dir, from), from); + ran.push({ unit, result: r }); + if (!finished(r) && stopped === null) stopped = r.notes; + } + }; + const width = Math.min(waveWidth(m), laneDirs.length, wave.length); + io.log(`Join: wave ${w + 1}: ${waves[w].join(', ')} side by side (${width} at once)`); + await Promise.all(laneDirs.slice(0, width).map(worker)); + ran.sort((a, b) => wave.indexOf(a.unit) - wave.indexOf(b.unit)); + for (const { unit, result } of ran) { + record(unit, result); + if (result.next_note) carry(unit, w, result.next_note); + } + let merged = { head: tip }; + if (ran.some((x) => finished(x.result))) merged = await merge(w + 1, ran, from); + waveLog.push({ wave: w + 1, units: waves[w], head: merged.head || tip, merge: merged }); + if (present(merged.stopped)) { + io.log(`Join: wave ${w + 1}: merge stopped (${merged.stopped})`); + return { results: sorted(), stopped: merged.stopped, head: tip, waves: waveLog }; + } + tip = merged.head; + if (stopped !== null) { + io.log(`Join: wave ${w + 1}: stopped (${stopped})`); + return { results: sorted(), stopped, head: tip, waves: waveLog }; + } + } + return { results: sorted(), stopped: null, head: tip, waves: waveLog }; +} + +// Finish a join that ran in waves in an earlier launch (design 4.5), serially +// on the feature checkout from tip (the recovery merge's head), instead of +// runTaskList's plan-order chain: a task done and reviewed is skipped and +// does not move the tip; a task that ran on a unit branch (m.unit_branches) +// and is not done and reviewed starts again from the tip as a restart (its +// prompt names its earlier work); a task done but not reviewed that ran on +// the feature checkout, or whose unit's head is in the feature history +// (inHistory: unit ids the recovery merge found merged, its approval gone +// stale), is reviewed over its backfill range and moves the tip only when a +// fix moved it; any other task runs from its latest started base +// (m.start_bases, so an earlier attempt's partial commits stay in its range), +// else from the tip. Returns runTaskList's {results, stopped, head}, head +// the tip. +async function runJoinContinuation(m, tasks, where, tip, io, name, inHistory = new Set()) { + const results = []; + const branches = m.unit_branches || {}; + const own = (id) => Object.prototype.hasOwnProperty.call(branches, id); + const carried = new Map(); + const carry = (unit, i, text) => { + const ids = unitTasks(unit).map((t) => t.id); + const later = tasks.slice(i); + const dependents = later.filter((t) => (t.depends_on || []).some((d) => ids.includes(d.id))); + for (const t of dependents.length > 0 ? dependents : later.slice(0, 1)) { + carried.set(t.id, [...(carried.get(t.id) || []), `from ${unit.id}, unblocked by the adjudicator: ${text}`]); + } + }; + for (let i = 0; i < tasks.length;) { + const group = batchGroup(m, tasks, i); + i += group.length; + const task = group[0]; + const state = taskState(m, task.id); + const range = (m.backfill || {})[task.id]; + if (state === 'skip') { + results.push({ + task: task.id, status: 'skipped', base: range ? range.base : null, head: range ? range.head : null, + rounds: null, tier_used: null, notes: '', rulings: [], + }); + continue; + } + const unit = group.length > 1 ? batchUnit(group) : task; + const note = unitTasks(unit).flatMap((t) => carried.get(t.id) || []).join('\n') || null; + const ranOn = inHistory.has(unit.id) ? null : group.find((t) => own(t.id)); + let r; + if (ranOn) { + const b = branches[ranOn.id]; + r = await runTask(m, unit, { ...where, restart: { branch: b.branch, dir: b.dir, complete: state === 'review' } }, + tip, io, null, note); + } else if (state === 'review') { + r = range ? await runTask(m, unit, where, range.base, io, range, note) : { + task: task.id, status: 'blocked', base: tip, head: null, rounds: 0, tier_used: task.tier, + notes: 'done but not reviewed, and no backfill commits', rulings: [], + }; + } else { + const started = (m.start_bases || {})[task.id]; + r = await runTask(m, unit, where, present(started) ? started : tip, io, null, note); + } + if (r.next_note) carry(unit, i, r.next_note); + const finished = r.status === 'done' || r.status === 'deferred'; + if (unit === task) results.push(r); + else if (finished) for (const t of group) results.push({ ...r, task: t.id, batch: unit.id }); + else results.push({ ...r, task: task.id, batch: unit.id }); + if (!finished) { + io.log(`${name}: stopped at ${task.id} (${r.notes})`); + return { results, stopped: r.notes, head: tip }; + } + // A task reviewed on its recorded range is already in the history: it + // moves the tip only when a fix added a commit there. + if (!(state === 'review' && !ranOn && range && r.head === range.head)) tip = r.head; + } + return { results, stopped: null, head: tip }; +} + // Run a lane's tasks in order in its worktree, the first from base (the // feature tip the lane fast-forwards to); stop at the first task that is not // done. Returns {lane, results, stopped:reason|null, head}. diff --git a/src/main.js b/src/main.js index 28603d6..2ec506b 100644 --- a/src/main.js +++ b/src/main.js @@ -34,6 +34,9 @@ if (args !== null && typeof args === 'object' && args.dry_run === true) { notices: errors.length === 0 ? launchNotices(dry, agents) : null, // The check policy at a glance (spec 4.9); null when invalid. checks: errors.length === 0 ? checksSummary(dry) : null, + // How the join runs (1.6.5): {mode, reason, waves}, before pre-flight's + // moves; null when invalid. + join_waves: errors.length === 0 ? wavePlan(dry) : null, }; } diff --git a/src/phases.js b/src/phases.js index 64fdea8..189ecc4 100644 --- a/src/phases.js +++ b/src/phases.js @@ -525,6 +525,8 @@ async function runAll(manifest, io) { stoppedLanes.push({ lane, task: last && last.status !== 'done' ? last.task : null, reason: list.stopped }); }; let preflight = null; + // Pre-flight's findings between join units, as wave edges (design 4.7). + const waveEdges = []; let integrate = null; let e2e = null; let final = null; @@ -536,6 +538,13 @@ async function runAll(manifest, io) { let acceptance = null; // Set when a post-integrate fix's re-review returned no result (C2). let fixUnreviewed = false; + // Join waves (1.6.5): this launch's waves for the report (null for a + // serial join), the wave checks fix's re-review findings, whether that + // re-review returned no result, and the waves whose checks still fail. + let joinWaves = null; + const waveFixReview = []; + let waveFixUnreviewed = false; + const waveChecksOpen = []; // The post-integrate check's result and the revision it covered. let postCheck = null; const report = (status, reason = null) => ({ @@ -545,6 +554,7 @@ async function runAll(manifest, io) { stopped_lanes: stoppedLanes, preflight, integrate, + join_waves: joinWaves, e2e, final, verify, @@ -630,7 +640,9 @@ async function runAll(manifest, io) { io.log('parallel-lanes: pre-flight skipped: every task is done and reviewed'); } else { io.phase('Pre-flight'); - const pre = await call('pre-flight', 'Pre-flight', preflightPrompt(m), preflightSchema()); + const shown = wavePlan(m); + const pre = await call('pre-flight', 'Pre-flight', preflightPrompt(m, shown.mode === 'waves' ? shown.waves : null), + preflightSchema()); if (state.refused.length > 0) return budgetReport(); if (!pre) return report('stopped', 'no result from pre-flight'); // Undeclared dependencies only add context: they reach the briefs of @@ -656,6 +668,27 @@ async function runAll(manifest, io) { // carry-launch skips it (1.6.3); null when there is none. preflight = { conflicts: pre.conflicts, rulings: [...pre.rulings], undeclared: kept, schedule, adjudicated: null }; if (schedule.length > 0) return report('preflight_conflicts'); + // Findings between join units whose producer is earlier move the task's + // unit to a later wave (design 4.7); preflight.wave_moves reports them. + if (shown.mode === 'waves') { + const units = joinUnits(m); + const unitOf = (id) => units.find((u) => unitTasks(u).some((t) => t.id === id)); + for (const e of kept) { + const u = unitOf(e.task); + const p = unitOf(e.producer); + if (u && p && u !== p && units.indexOf(p) < units.indexOf(u)) waveEdges.push({ unit: u.id, producer: p.id }); + } + const waveOf = (waves, id) => waves.findIndex((w) => w.includes(id)) + 1; + const after = wavePlan(m, waveEdges).waves; + preflight.wave_moves = units.filter((u) => waveOf(shown.waves, u.id) !== waveOf(after, u.id)).map((u) => { + const edge = waveEdges.find((e) => e.unit === u.id); + return { unit: u.id, from: waveOf(shown.waves, u.id), to: waveOf(after, u.id), producer: edge ? edge.producer : null }; + }); + for (const mv of preflight.wave_moves) { + io.log(`parallel-lanes: pre-flight: ${mv.unit} moves from join wave ${mv.from} to ${mv.to}` + + `${mv.producer ? ` (it needs ${mv.producer})` : ''}`); + } + } m = withConsumesExtra(m, kept); // Pre-flight's rulings bind every task and final reviewer of this run // (taskContext and the final review prompts show m.run_rulings). @@ -680,6 +713,11 @@ async function runAll(manifest, io) { } } + // How the join runs (design 4.1): waves only in the launch that starts it, + // with pre-flight's findings as edges (design 4.7). + const joinPlan = wavePlan(m, waveEdges); + const wavesOn = joinPlan.mode === 'waves'; + io.phase('Prelude'); const prelude = await runTaskList(m, m.prelude, featureWhere(m, 'prelude'), starts.prelude || tip, counted, 'Prelude', true); @@ -763,7 +801,8 @@ async function runAll(manifest, io) { // cleanup deleted as merged when that commit is already in HEAD. const tips = laneTips(m, laneResults); let integ = await callM('integrate', 'Integrate', - integratePrompt(m, preludeTip, { conflictMode: 'abort', joinStartPoint: joinNoHook, laneTips: tips }), + integratePrompt(m, preludeTip, { conflictMode: 'abort', joinStartPoint: joinNoHook, laneTips: tips, + keepWorktrees: wavesOn }), statusSchema(), sonnetHigh); if (state.refused.length > 0) return budgetReport(); // Any Sonnet result that is not a finished integration (a done without a @@ -790,6 +829,7 @@ async function runAll(manifest, io) { resolverNotes: resolved ? resolved.notes : null, joinStartPoint: joinNoHook, laneTips: tips, + keepWorktrees: wavesOn, }), statusSchema(), standard); if (state.refused.length > 0) return budgetReport(); } @@ -810,7 +850,8 @@ async function runAll(manifest, io) { if (from === BUDGET) return budgetReport(); tip = from; integ = await callM('integrate', 'Integrate', - integratePrompt(m, preludeTip, { testFailure: 'heal', joinStartPoint: joinNoHook, laneTips: tips }), + integratePrompt(m, preludeTip, { testFailure: 'heal', joinStartPoint: joinNoHook, laneTips: tips, + keepWorktrees: wavesOn }), statusSchema(), standard); if (state.refused.length > 0) return budgetReport(); integrate = integrateReport(integ); @@ -849,8 +890,122 @@ async function runAll(manifest, io) { } io.phase('Join'); - const join = await runTaskList(m, m.join, featureWhere(m, 'join'), joinBase, - counted, 'Join', joinFromPhase); + let join; + const finished = (r) => r.status === 'done' || r.status === 'deferred'; + // Merge units ([{id, base, head}], plan order) into the feature checkout + // through merge-wave (design 4.3), wave null for a relaunch's recovery merge + // (design 4.5): each conflict goes to a resolver (autonomous) or stops the + // run with the sha to merge by hand (supervised). With checks (the last + // multi-unit wave's, design 4.4) the merge agent runs them; a failure gets + // one fix, its re-review and a recheck (autonomous) or stops (supervised). + // entry is the wave's report entry, filled in. Returns {head} or {stopped}. + const mergeUnits = async (wave, units, checks, entry, known = []) => { + const name = wave === null ? 'recovery' : `wave ${wave}`; + let r = null; + for (let attempt = 0; attempt <= units.length; attempt += 1) { + r = await callM(`${name} merge`, 'Join', waveMergePrompt(m, wave, units, checks, known), waveMergeSchema(), + sonnetHigh); + if (state.refused.length > 0) return { stopped: 'an agent was refused' }; + if (!r || r.status !== 'conflict') break; + const unit = units.find((u) => u.id === r.conflict_unit) + || units.find((u) => !(Array.isArray(r.merged) ? r.merged : []).includes(u.id)) || units[0]; + const files = Array.isArray(r.conflict_files) ? r.conflict_files : []; + for (const f of files) if (!entry.conflict_files.includes(f)) entry.conflict_files.push(f); + if (!autonomous) { + const message = wave === null ? `join recovery: merge ${unit.id}` : `join wave ${wave}: merge ${unit.id}`; + return { stopped: `join ${name}: ${unit.id} conflicts in ${files.join(', ') || 'files git reported'}; ` + + `merge it by hand with git -C ${shellQuote(featureDir(m))} merge --no-ff -m ${shellQuote(message)} ` + + `${unit.head}, resolve and commit, then relaunch` }; + } + const res = await callM(`${name} resolve ${unit.id}`, 'Join', waveResolvePrompt(m, wave, unit, files), + statusSchema(), standard); + if (state.refused.length > 0) return { stopped: 'an agent was refused' }; + if (!res || res.status !== 'done' || !isSha(res.head)) { + return { stopped: `join ${name}: the resolver could not merge ${unit.id}: ` + + `${res ? res.notes : `no result from ${name} resolve ${unit.id}`}` }; + } + for (const f of files) if (!entry.resolved_files.includes(f)) entry.resolved_files.push(f); + } + if (!r || r.status !== 'done' || !isSha(r.head)) { + return { stopped: `join ${name}: the merge failed: ${r ? r.notes : `no result from ${name} merge`}` }; + } + let head = r.head; + entry.head = head; + entry.overlaps = Array.isArray(r.overlaps) ? r.overlaps : []; + entry.in_history = Array.isArray(r.in_history) ? r.in_history : []; + if (checks.length > 0) { + const c = r.checks && typeof r.checks === 'object' ? r.checks + : { ok: false, unfinished: true, notes: `${name} merge returned no checks result` }; + entry.checks = { ok: c.ok === true, unfinished: c.ok !== true && c.unfinished === true, notes: c.notes || '', + fixed: false }; + if (!entry.checks.ok && !entry.checks.unfinished) { + if (!autonomous) return { stopped: `wave checks failed: ${entry.checks.notes}` }; + const fix = await callM('wave checks fix', 'Join', waveChecksFixPrompt(m, entry.checks.notes), + statusSchema(), standard); + if (state.refused.length > 0) return { stopped: 'an agent was refused' }; + if (fix && fix.status === 'done' && isSha(fix.head) && fix.head !== head) { + const rr = await callM('wave checks re-review', 'Join', waveChecksReReviewPrompt(m, head, fix.head), + postIntegrateReReviewSchema(), standard); + if (state.refused.length > 0) return { stopped: 'an agent was refused' }; + if (rr && Array.isArray(rr.findings)) waveFixReview.push(...rr.findings); + else waveFixUnreviewed = true; + head = fix.head; + entry.checks.fixed = true; + } + const again = await callM('wave checks', 'Join', waveChecksPrompt(m, wave, checks), waveChecksSchema(), + sonnetHigh); + if (state.refused.length > 0) return { stopped: 'an agent was refused' }; + entry.checks.ok = Boolean(again && again.ok === true); + entry.checks.unfinished = Boolean(again && !entry.checks.ok && again.unfinished === true); + entry.checks.notes = again ? again.notes || '' : 'no result from wave checks'; + if (!entry.checks.ok && !entry.checks.unfinished) waveChecksOpen.push({ wave, notes: entry.checks.notes }); + } + entry.head = head; + } + return { head }; + }; + const newEntry = (wave, units, head) => ({ wave, units, head, conflict_files: [], resolved_files: [], overlaps: [], + checks: null }); + if (wavesOn) { + const multi = joinPlan.waves.map((w, i) => (w.length > 1 ? i + 1 : 0)).filter((n) => n > 0); + const lastMulti = multi[multi.length - 1]; + const entries = []; + const mergeWave = async (wave, ran, base) => { + const units = ran.filter(({ result: r }) => r.status === 'done' + || (r.status === 'deferred' && isSha(r.head) && r.head !== r.base)) + .map(({ unit, result: r }) => ({ id: unit.id, base: r.base, head: r.head })); + const entry = newEntry(wave, joinPlan.waves[wave - 1], base); + entries.push(entry); + if (units.length === 0) return { head: base }; + const whole = ran.length === joinPlan.waves[wave - 1].length && ran.every((x) => finished(x.result)); + const checks = wave === lastMulti && whole ? waveChecksOf(m, ran.map((x) => x.unit)) : []; + return mergeUnits(wave, units, checks, entry); + }; + join = await runJoinWaves(m, joinPlan.waves, joinBase, waveLaneDirs(m), counted, mergeWave); + joinWaves = join.waves.map((w) => entries.find((e) => e.wave === w.wave) || newEntry(w.wave, w.units, w.head)); + m = { ...m, join_waves_now: joinWaves }; + } else if (!lite && joinStarted(m) && Object.keys(m.unit_branches || {}).length > 0) { + // A relaunch after waves (design 4.5): merge what the waves finished and + // had reviewed, then the rest serially from that tip. + const units = recoveryUnits(m); + // The other units that ran on their own branch: the merge reports which + // of them an earlier launch merged (their approval may have gone stale). + const known = joinUnits(m).filter((u) => !units.some((x) => x.id === u.id) + && unitTasks(u).some((t) => Object.prototype.hasOwnProperty.call(m.unit_branches, t.id))) + .map((u) => ({ id: u.id, range: (m.backfill || {})[unitTasks(u)[unitTasks(u).length - 1].id] })) + .filter((u) => u.range).map((u) => ({ id: u.id, head: u.range.head })); + const entry = { ...newEntry(null, units.map((u) => u.id), joinBase), recovery: true }; + const rec = await mergeUnits(null, units, [], entry, known); + if (units.length > 0) { + joinWaves = [entry]; + m = { ...m, join_waves_now: joinWaves }; + } + join = rec.stopped !== undefined ? { results: [], stopped: rec.stopped, head: joinBase } + : await runJoinContinuation(m, m.join, featureWhere(m, 'join'), rec.head, counted, 'Join', + new Set(entry.in_history || [])); + } else { + join = await runTaskList(m, m.join, featureWhere(m, 'join'), joinBase, counted, 'Join', joinFromPhase); + } record(join.results); if (join.stopped !== null) stopAt('join', join); if (state.refused.length > 0) return budgetReport(); @@ -920,7 +1075,7 @@ async function runAll(manifest, io) { } io.phase('Final review'); - final = await runFinalReview(m, e2e, tip, counted, integrate ? integrate.fix_review : [], + final = await runFinalReview(m, e2e, tip, counted, [...(integrate ? integrate.fix_review : []), ...waveFixReview], hasChecks ? verifyAt : null, taskMinors, repair); // Checks that ran inside the final review (a fix agent ran) are the run's // verify: the report carries them once, at the top level. @@ -939,6 +1094,14 @@ async function runAll(manifest, io) { final.cannot_verify.unshift({ requirement: 'the post-integrate re-review returned no result', source: 'run', why: 'the post-integration fix was delivered without a re-review', check_by: 're-review the post-integration fix' }); } + if (waveFixUnreviewed) { + final.cannot_verify.unshift({ requirement: 'the wave checks re-review returned no result', source: 'run', + why: 'the wave checks fix was delivered without a re-review', check_by: 're-review the wave checks fix' }); + } + for (const w of waveChecksOpen) { + final.cannot_verify.unshift({ requirement: `the join's wave ${w.wave} checks pass`, source: 'run', + why: `they still failed after the wave checks fix: ${w.notes}`, check_by: 'rerun them at the delivered revision' }); + } // Task minors live only in this launch's task results: the tasks an // earlier launch committed and reviewed are not re-run, so their approved // reviews' minor findings never reached the final lenses. Every resume diff --git a/src/plan.js b/src/plan.js index 8ffc272..26eb3aa 100644 --- a/src/plan.js +++ b/src/plan.js @@ -112,7 +112,25 @@ function planAgents(m) { add('Integrate', null, null, 'integrate', sonnetHigh); if (m.hooks.post_integrate) add('Integrate', null, null, 'post_integrate', standard); } + // Join waves (1.6.5): a relaunch after waves starts with the recovery + // merge; a launch that runs waves merges each multi-unit wave, and when the + // last one has checks, lists their fix, its re-review and the recheck as + // the upper bound. + if (!lite && joinStarted(m) && Object.keys(m.unit_branches || {}).length > 0) { + add('Join', null, null, 'recovery_merge', sonnetHigh); + } addTasks('Join', null, m.join); + const joinPlan = lite ? null : wavePlan(m); + if (joinPlan && joinPlan.mode === 'waves') { + const multi = joinPlan.waves.filter((w) => w.length > 1); + for (let i = 0; i < multi.length; i += 1) add('Join', null, null, 'wave_merge', sonnetHigh); + const last = multi[multi.length - 1]; + if (waveChecksOf(m, joinUnits(m).filter((u) => last.includes(u.id))).length > 0) { + add('Join', null, null, 'wave_checks_fix', standard); + add('Join', null, null, 'wave_checks_re_review', standard); + add('Join', null, null, 'wave_checks', sonnetHigh); + } + } if (m.hooks.e2e) add('E2E', null, null, 'e2e', sonnetHigh); const lenses = lite ? ['final_review_combined'] @@ -131,8 +149,10 @@ function planAgents(m) { // agents; resume counts m.done. Expects a valid manifest. function launchNotices(m, agents) { const lanes = new Set(agents.filter((a) => a.lane !== null).map((a) => a.lane)).size; + const joinPlan = wavePlan(m); + const waves = joinPlan.mode === 'waves' ? `, join waves ${joinPlan.waves.map((w) => w.length).join('-')}` : ''; return { - launch: `parallel-lanes: launching run ${m.run_id}: ${lanes} lanes, ${agents.length} agents`, + launch: `parallel-lanes: launching run ${m.run_id}: ${lanes} lanes, ${agents.length} agents${waves}`, resume: `parallel-lanes: resuming run ${m.run_id}: ${m.done.length} tasks already committed`, }; } @@ -177,6 +197,14 @@ function featureWhere(m, ledgerLane) { return { dir: featureDir(m), branch: m.repo.branch, lane: ledgerLane }; } +// Where a unit of a multi-unit join wave runs (1.6.5): its own branch in the +// lane worktree dir, created from the wave's base from; its ledger lane, its +// checks and its phase stay the join's, and there is no lane sync. +function unitWhere(m, unit, dir, from) { + const branch = unitBranch(m, unit); + return { dir, branch, lane: 'join', unit_branch: branch, from }; +} + // Scratch files for a task, kept in the ledger dir (outside every repo). function taskFiles(m, task) { const dir = m.repo.ledger_dir; diff --git a/src/prompts-phase.js b/src/prompts-phase.js index 59b2ba8..71a9c56 100644 --- a/src/prompts-phase.js +++ b/src/prompts-phase.js @@ -253,7 +253,10 @@ function relaunchPreflightLines(m) { ]; } -function preflightPrompt(m) { +// waves (1.6.5): the join's waves ([[unit id, ...], ...]) when the join runs in +// waves; pre-flight then also looks for a unit needing another unit of its own +// wave (design 4.7). +function preflightPrompt(m, waves = null) { return [ `You are the pre-flight reviewer for parallel-lanes run ${m.run_id}. You are read-only: do not modify any`, 'file, branch, or worktree.', @@ -275,6 +278,14 @@ function preflightPrompt(m) { ' lane or in the prelude produces (a function, a file format, markup, an API answer)', ' without naming that task in its Consumes. Return each in undeclared as', ' {task, producer, what}, what in one sentence.', + ...(waves === null ? [] : [ + '5. Join waves: the join runs in these waves. The units of a wave run side by side, each from the feature', + ' tip after the previous wave, without the work of the other units of their own wave:', + ...waves.map((w, i) => ` wave ${i + 1}: ${w.join(', ')}`), + ' For every task that needs work of another unit in its own wave (code, a contract, a test helper, a', + ' generated file), report it in code_deps (code) or undeclared (anything else) as {task, producer,', + ' what}: the run moves the task to a later wave.', + ]), 'Report serious problems (implementers would build the wrong thing, or check 1 or 2 fails) as conflicts,', 'one sentence each naming the tasks and plan or spec sections. Settle minor ambiguities yourself and report', 'each as a ruling in the form "Ruling: decision - why - cost if wrong".', @@ -336,7 +347,7 @@ function integratePrompt(m, preludeTip, opts = {}) { // and the lane_commands overrides, with the long-check procedure (1.6.1) // under the reserve name integrate, its logs beside the result. const setup = finalSetup(m); - const checks = runChecksCommand(m, featureDir(m), finalChecks(m), resultPath()); + const checks = runChecksCommand(m, featureDir(m), finalChecks(m), resultPath(), 'integrate'); // Heal mode reports checks that did not finish (1.6.2 spec 4.3): the // engine then starts no fix and stops. The other modes keep the general // exit-code rule. @@ -355,6 +366,7 @@ function integratePrompt(m, preludeTip, opts = {}) { ...checksLogLines().map((l) => ` ${l}`), ]), ]; + const keepWorktrees = opts.keepWorktrees || false; const cleanup = m.lanes.map((lane) => { const w = laneWhere(m, lane); return ` - ${q(w.dir)}: if git -C ${q(w.dir)} status --porcelain --untracked-files=all prints nothing, ` + @@ -364,10 +376,13 @@ function integratePrompt(m, preludeTip, opts = {}) { // Every join task committed in an earlier attempt: a final fix of that // attempt may have committed after the last join commit. const lastJoin = m.join.length > 0 ? (m.backfill || {})[m.join[m.join.length - 1].id] : null; - const joinDone = Boolean(lastJoin) && m.join.every((t) => m.done.includes(t.id)); + // With join waves (1.6.5) the join's last commit is the tip ledger + // backfill reports (join_tip), not the last plan-order task's head. + const joinTip = present(m.join_tip) ? m.join_tip : (lastJoin ? lastJoin.head : null); + const joinDone = present(joinTip) && m.join.every((t) => m.done.includes(t.id)); const finalFixes = joinDone ? [ ' Every join task was committed earlier in this run, so final-fix commits from an earlier attempt of', - ` this run may follow the last join commit ${lastJoin.head}: allow any commits after it. That includes`, + ` this run may follow the last join commit ${joinTip}: allow any commits after it. That includes`, ` repair commits after a failing verify (\`${repairFixMessage()}\`).`, ] : []; const tipLines = m.lanes.map((lane) => { @@ -435,21 +450,29 @@ function integratePrompt(m, preludeTip, opts = {}) { ' may list only merges (two parents) of the lane branches above, integration or post-integration fix', ' commits (yours, or from an earlier attempt in this run), and commits an earlier attempt at a join task', ` made (join tasks: ${joinIds}; their recorded commits are in ${m.repo.ledger_dir}/join.jsonl).`, + " A join that ran in waves also merged its units' heads ('join wave : merge ' and", + " 'join recovery: merge ', a resolver's included) and may have a wave checks fix", + ' (`fix: make the wave checks pass`).', ...finalFixes, " Any other commit landed on the feature branch outside the run's steps: fail listing it (do not", ' rewrite history).', `5. Committed scratch: git -C ${dir} diff --name-only ${q(`${m.repo.base_ref}...${m.repo.branch}`)}`, ' must list no path under .superpowers/; if it does, fail listing them (do not rewrite history).', - ...(testFailure === 'heal' ? [ - '6. Only when steps 1-5 passed and you return tests_failed false (every project command passes), clean', - ' up each lane. When you return tests_failed true, skip this step entirely: a fix agent and a rerun of', - ' these steps follow and the rerun cleans up (with checks_unfinished true the run stops instead and the', - ' lanes stay for the user). Cleanup:', - ] : ['6. Only when steps 1-5 passed, clean up each lane:']), - ...cleanup, - ' Leave a worktree with uncommitted files (and its branch) in place and list it in notes; never', - ' force a removal or a branch deletion. Cleanup never fails the integration: list anything step 6', - ' could not remove in notes and still return status done.', + ...(keepWorktrees ? [ + '6. Remove no lane worktree and delete no lane branch: the join runs in waves in the lane worktrees,', + " and the run's hand-back cleanup removes them.", + ] : [ + ...(testFailure === 'heal' ? [ + '6. Only when steps 1-5 passed and you return tests_failed false (every project command passes), clean', + ' up each lane. When you return tests_failed true, skip this step entirely: a fix agent and a rerun of', + ' these steps follow and the rerun cleans up (with checks_unfinished true the run stops instead and the', + ' lanes stay for the user). Cleanup:', + ] : ['6. Only when steps 1-5 passed, clean up each lane:']), + ...cleanup, + ' Leave a worktree with uncommitted files (and its branch) in place and list it in notes; never', + ' force a removal or a branch deletion. Cleanup never fails the integration: list anything step 6', + ' could not remove in notes and still return status done.', + ]), '', ...review, `Plan: ${m.plan}`, @@ -495,6 +518,227 @@ function resolveConflictsPrompt(m, preludeTip, conflictFiles) { ].join('\n'); } +// ---- Join waves (1.6.5 design 4.3, 4.4) ---- + +// The wave merge agent (Sonnet): scripts/merge-wave for the wave's finished +// units (units: [{id, base, head}], plan order) in the feature checkout; +// with checks (the last multi-unit wave's, waveChecksOf) and every unit +// merged, it runs them on the merged tip and records wave_checked. +// wave null: a relaunch's recovery merge (design 4.5), merge-wave --recovery. +// known ([{id, head}], recovery only): units that ran on their own branch and +// are not to be merged; merge-wave reports which are in the history already. +function waveMergePrompt(m, wave, units, checks = [], known = []) { + const q = shellQuote; + const dir = featureDir(m); + const recovery = wave === null; + const merge = [`cd ${q(dir)} && ${pythonCommand(m)} ${q(`${m.skill_dir}/scripts/merge-wave`)} ${q(dir)}`, + q(m.repo.ledger_dir), ...(recovery ? ['--recovery'] : ['--wave', q(String(wave))]), + ...units.flatMap((u) => ['--unit', q(u.id), q(u.base), q(u.head)]), + ...known.flatMap((u) => ['--known', q(u.id), q(u.head)])].join(' '); + const run = checks.length > 0 ? runChecksCommand(m, dir, checks, resultPath(), 'join') : null; + const record = (extra) => ledgerCommand(m, '_run', + { task: '_run', event: 'wave_checked', wave, head: '', ...extra }, dir); + return [ + recovery ? `You are the recovery merge agent for parallel-lanes run ${m.run_id}.` + : `You are the wave merge agent for parallel-lanes run ${m.run_id}, join wave ${wave}.`, + `Work in ${dir} on the feature branch ${m.repo.branch}; do not switch branches.`, + recovery ? (units.length > 0 + ? 'An earlier launch ran join units in waves; these finished and were reviewed but not merged. Merge their recorded heads.' + : 'An earlier launch ran join units in waves; none is left to merge. The command below only reports the head.') + : `The units of wave ${wave} ran side by side, each on its own branch; merge their recorded heads.`, + '', + '1. Run, as one call:', + ` ${merge}`, + " It merges each unit's head in this order (skipping one already merged), aborts a merge that", + ' conflicts and stops there, records the wave in the ledger, and prints one JSON line. Do not merge', + ' anything yourself, resolve a conflict, or change any file.', + '2. If it printed status conflict, stop: return status conflict with its head, merged, conflict_unit,', + ' conflict_files and overlaps. If it exited 2 or 3, stop: return status failed with its message.', + ...(run === null ? [] : [ + `3. It merged every unit: run the wave's checks on the merged tip (the last wave with more than one unit,`, + " whose merge no later task's checks cover). One run-checks call:", + ...procedureLines(m, { name: `wave-${wave}`, start: run }).map((l) => ` ${l}`), + ...checksLogLines().map((l) => ` ${l}`), + ' Never change code to make a check pass: a failure goes back to the run.', + `4. Record the result, with the full sha of git -C ${q(dir)} rev-parse HEAD:`, + ` every check passed: ${record({ ok: true })}`, + ` a check failed: ${record({ ok: false })}`, + ' every failing check exited 75, 124, 128 or above, or the checks stopped twice without a result:', + ` ${record({ ok: false, unfinished: true })}`, + ]), + '', + `Plan: ${m.plan}`, + keepFilesRule(), + phaseRules(m), + '', + 'Return status done (every unit merged), conflict or failed; head = the head merge-wave printed; merged,', + 'conflict_unit (empty when none), conflict_files, overlaps and in_history copied from its JSON;', + ...(run === null ? [] : ['checks = {ok, unfinished, notes} for the checks of step 3 (notes: the failing commands);']), + 'notes.', + ].join('\n'); +} + +function waveMergeSchema() { + return { + type: 'object', + properties: { + status: { type: 'string', enum: ['done', 'conflict', 'failed'] }, + head: { type: 'string' }, + merged: { type: 'array', items: { type: 'string' } }, + conflict_unit: { type: 'string' }, + conflict_files: { type: 'array', items: { type: 'string' } }, + in_history: { type: 'array', items: { type: 'string' } }, + overlaps: { + type: 'array', + items: { + type: 'object', + properties: { path: { type: 'string' }, units: { type: 'array', items: { type: 'string' } } }, + required: ['path', 'units'], + }, + }, + checks: { + type: 'object', + properties: { ok: { type: 'boolean' }, unfinished: { type: 'boolean' }, notes: { type: 'string' } }, + required: ['ok', 'unfinished', 'notes'], + }, + notes: { type: 'string' }, + }, + required: ['status', 'head', 'notes'], + }; +} + +// The wave resolver (Opus, autonomous): merges one unit's head that +// conflicted, keeping both sides' intent, and commits; on any failure it +// aborts the merge so the checkout is clean for a relaunch. +// wave null: a conflict of the recovery merge. +function waveResolvePrompt(m, wave, unit, conflictFiles) { + const q = shellQuote; + const dir = q(featureDir(m)); + const files = conflictFiles && conflictFiles.length > 0 ? conflictFiles.join(', ') : '(the files the merge reports)'; + const message = wave === null ? `join recovery: merge ${unit.id}` : `join wave ${wave}: merge ${unit.id}`; + return [ + `You are the wave resolver for parallel-lanes run ${m.run_id}, ${wave === null ? 'the recovery merge' : `join wave ${wave}`}.`, + `Work in ${featureDir(m)} on the feature branch ${m.repo.branch}; do not switch branches.`, + `Merging unit ${unit.id} (its recorded head ${unit.head}) conflicted in: ${files}.`, + '', + `1. git -C ${dir} status --porcelain --untracked-files=all must print nothing; otherwise return status`, + ' failed (never make it so by deleting, cleaning, or stashing files).', + `2. git -C ${dir} merge --no-ff -m ${q(message)} ${q(unit.head)}`, + " Resolve each conflict keeping the intent of both sides: read the plan tasks of the units that touched the", + ' file and keep what each meant to do, then commit the merge. Resolve the conflicts only; make no other', + ' change, run no project command, rewrite no history, and remove no worktree or branch.', + `3. If you cannot resolve it, run git -C ${dir} merge --abort so the checkout is clean, and return status`, + ' failed saying why.', + '', + `Plan: ${m.plan}`, + keepFilesRule(), + phaseRules(m), + '', + `Return status done when the merge is committed, else failed; head (the full sha printed by git -C ${dir}`, + 'rev-parse HEAD when you finish); notes (how you resolved each file).', + ].join('\n'); +} + +// The message the wave checks fix commits with. +function waveChecksFixMessage() { + return 'fix: make the wave checks pass'; +} + +// The wave checks fix (Opus, autonomous): one fix for the last multi-unit +// wave's failing checks, in the feature checkout; a re-review and a check-only +// rerun (waveChecksPrompt) follow. +function waveChecksFixPrompt(m, failure) { + const dir = shellQuote(featureDir(m)); + return [ + `You are fixing the failing wave checks of parallel-lanes run ${m.run_id}.`, + `Work in ${featureDir(m)} on ${m.repo.branch}; do not switch branches.`, + 'The units of the last join wave with more than one unit were merged; their targeted checks then failed:', + failure, + '', + 'Find the cause, fix it with the smallest change that is correct, and commit per the commit rules.', + commitWithLine(waveChecksFixMessage()), + 'You may rerun the failing commands named above to reproduce the failure and to confirm your fix; the run', + 'reruns the wave checks after you.', + '', + `Plan: ${m.plan}`, + `Spec: ${m.spec === null ? '(none)' : m.spec}`, + keepFilesRule(), + phaseRules(m), + '', + 'Return status done when your fix is committed and the failing checks pass, otherwise failed; head = the', + `full sha printed by git -C ${dir} rev-parse HEAD when you finish; notes = what you changed and the results.`, + ].join('\n'); +} + +// The re-review of a wave checks fix (Opus): the fix range only. +function waveChecksReReviewPrompt(m, base, head) { + return postIntegrateReReviewPrompt(m, base, head).replace('re-reviewing a post-integration fix', + 're-reviewing a fix for the failing join wave checks'); +} + +// The check-only rerun of the wave's checks after a fix (Sonnet). +function waveChecksPrompt(m, wave, checks) { + const q = shellQuote; + const dir = featureDir(m); + const run = runChecksCommand(m, dir, checks, resultPath(), 'join'); + const record = (extra) => ledgerCommand(m, '_run', + { task: '_run', event: 'wave_checked', wave, head: '', ...extra }, dir); + return [ + `You are rechecking join wave ${wave} of parallel-lanes run ${m.run_id} after a fix. Verify only: change no file,`, + `make no commit. Work in ${dir} on ${m.repo.branch}.`, + 'Run the wave checks, as one run-checks call:', + ...procedureLines(m, { name: `wave-${wave}`, start: run }), + ...checksLogLines(), + `Then record the result, with the full sha of git -C ${q(dir)} rev-parse HEAD:`, + ` every check passed: ${record({ ok: true })}`, + ` a check failed: ${record({ ok: false })}`, + ' every failing check exited 75, 124, 128 or above, or the checks stopped twice without a result:', + ` ${record({ ok: false, unfinished: true })}`, + '', + keepFilesRule(), + phaseRules(m), + '', + 'Return status done; head = that sha; ok, unfinished and notes (the failing commands) for the checks.', + ].join('\n'); +} + +function waveChecksSchema() { + return { + type: 'object', + properties: { + status: { type: 'string', enum: ['done', 'failed'] }, + head: { type: 'string' }, + ok: { type: 'boolean' }, + unfinished: { type: 'boolean' }, + notes: { type: 'string' }, + }, + required: ['status', 'head', 'ok', 'unfinished', 'notes'], + }; +} + +// What the final lenses are told about the join waves (design 4.9): each +// wave's paths two units changed, and the conflicts the resolver settled, +// from earlier launches (m.wave_merges, ledger status) and this one +// (m.join_waves_now, the run result's join_waves). +function waveLines(m) { + const lines = []; + const all = [...(Array.isArray(m.wave_merges) ? m.wave_merges : []), + ...(Array.isArray(m.join_waves_now) ? m.join_waves_now : [])]; + for (const w of all) { + const name = w.recovery ? 'the recovery merge' : `wave ${w.wave}`; + for (const o of Array.isArray(w.overlaps) ? w.overlaps : []) lines.push(`- ${name}: ${o.path} (${o.units.join(', ')})`); + const resolved = Array.isArray(w.resolved_files) && w.resolved_files.length > 0 ? w.resolved_files + : (Array.isArray(w.conflict_files) ? w.conflict_files : []); + if (resolved.length > 0) lines.push(`- ${name}: merge conflicts the wave resolver settled in ${resolved.join(', ')}`); + } + if (lines.length === 0) return []; + return [ + 'Join waves: units of a wave ran side by side on their own branches and were merged. Review these paths', + 'with every unit named in mind (both sides of a merge must still hold):', + ...lines, + ]; +} + // The message the final fix commits with. A function, not a constant: the // test harness's loadHelpers returns before top-level bindings initialize. function finalFixMessage() { @@ -530,7 +774,7 @@ function postIntegrateFixPrompt(m, failure, path = 'hook') { 'command.', ] : [ 'Rerun every project command afterwards and confirm they pass:', - commandsText(m, null, featureDir(m), 'post-integrate-fix'), + commandsText(m, null, featureDir(m), 'post-integrate-fix', 'post-integrate'), ]), m.hooks.post_integrate ? `Post-integration check to keep passing:\n${m.hooks.post_integrate}` : '', '', @@ -579,7 +823,7 @@ function postIntegratePrompt(m, checkOnly = false, checks = null) { ] : [ 'Change files only if the instructions call for it; commit any change per the commit rules and rerun', 'the project commands afterwards:', - commandsText(m, null, featureDir(m), 'post-integrate'), + commandsText(m, null, featureDir(m), 'post-integrate', 'post-integrate'), ]; return [ `You are the post-integration agent for parallel-lanes run ${m.run_id}.`, @@ -708,6 +952,7 @@ function finalReviewFrame(m, intro, focus, minors = [], other = false) { ...(other ? otherReviewerMinorsLines(m, minors) : taskMinorsLines(m, minors)), ...runRulingsLines(m), ...rulingsReadLines(m, featureDir(m)), + ...waveLines(m), "A finding that a ruling in force settles is not raised unless it is critical or important; then raise it and cite the ruling's id.", 'Files each task changed outside its Files list, with their owners (a read-only command):', ` ${ledgerReadCommand(m, featureDir(m), 'undeclared')}`, @@ -768,7 +1013,7 @@ function finalFixPrompt(m, findings, base) { 'check at the delivered revision right after you, and a failure there goes to a repair agent.', ] : [ 'decline. Rerun every project command afterwards:', - commandsText(m, null, featureDir(m), 'final-fix'), + commandsText(m, null, featureDir(m), 'final-fix', 'final-fix'), ]), 'Commit your fixes per the commit rules.', commitWithLine(finalFixMessage()), @@ -865,7 +1110,7 @@ function finalReReviewPrompt(m, base, head, findings, verify = null, repairs = [ function verifyPrompt(m, sha, from = null) { const dir = featureDir(m); const q = shellQuote; - const base = runChecksCommand(m, dir, finalChecks(m), resultPath()); + const base = runChecksCommand(m, dir, finalChecks(m), resultPath(), 'verify'); const cmd = from === null ? base : `${base} --descends-from ${q(from)}`; const setup = finalSetup(m); const stop = [ diff --git a/src/prompts-task.js b/src/prompts-task.js index 6a30794..5ce6762 100644 --- a/src/prompts-task.js +++ b/src/prompts-task.js @@ -113,18 +113,25 @@ function finalSetup(m) { return out; } +// The phase a ledger lane's checks run in (run-checks --phase, PL_PHASE): +// prelude and join tasks have their own; a lane's tasks run in lane. +function checksPhase(laneId) { + return laneId === 'prelude' || laneId === 'join' ? laneId : 'lane'; +} + // The scripts/run-checks call that runs each check of a [{group, command}] // list in dir, separately, and exits non-zero when any one fails. out // (optional) is the evidence file it writes under the ledger dir: // resultPath() goes in as given (already quoted), for the path the agent's -// reserve prints. null for an empty list. -function runChecksCommand(m, dir, checks, out = null) { +// reserve prints. phase (1.6.5) is run-checks' --phase: the commands run +// with PL_PHASE set to it. null for an empty list. +function runChecksCommand(m, dir, checks, out = null, phase = null) { if (checks.length === 0) return null; const parts = checks.flatMap((c) => ['--cmd', shellQuote(c.group), shellQuote(c.command)]); const target = out === resultPath() ? out : shellQuote(out); const evidence = out === null ? [] : ['--out', target, '--root', shellQuote(m.repo.ledger_dir)]; return [`cd ${shellQuote(dir)} && ${pythonCommand(m)} ${shellQuote(`${m.skill_dir}/scripts/run-checks`)}`, shellQuote(dir), - ...evidence, ...parts].join(' '); + ...evidence, ...(phase === null ? [] : ['--phase', shellQuote(phase)]), ...parts].join(' '); } // Every test, lint and build command of a ledger lane, as [{group, command}] @@ -161,9 +168,9 @@ function wideChecks(m, laneId) { } // The scripts/run-checks call for every test, lint and build command of a -// ledger lane in dir. null when there is no check command. -function checksCommand(m, laneId, dir, out = null) { - return runChecksCommand(m, dir, laneChecks(m, laneId), out); +// ledger lane in dir, in phase. null when there is no check command. +function checksCommand(m, laneId, dir, out = null, phase = null) { + return runChecksCommand(m, dir, laneChecks(m, laneId), out, phase); } // What an agent handed a run-checks call is told about its output and logs: @@ -259,10 +266,12 @@ function changeNothing() { // The general rule for any other long command (spec 4.4): it runs through // run-checks with the same procedure, under the reserve name long. Task -// prompts (taskContext) and phase prompts (phaseRules) carry it. -function longCheckRuleLines(m) { +// prompts (taskContext, with the task's phase) and phase prompts +// (phaseRules) carry it. +function longCheckRuleLines(m, phase = null) { const start = `cd 'DIR' && ${pythonCommand(m)} ${shellQuote(`${m.skill_dir}/scripts/run-checks`)} 'DIR' ` - + `--out ${resultPath()} --root ${shellQuote(m.repo.ledger_dir)} --cmd 'GROUP' 'COMMAND'`; + + `--out ${resultPath()} --root ${shellQuote(m.repo.ledger_dir)} ` + + `${phase === null ? '' : `--phase ${shellQuote(phase)} `}--cmd 'GROUP' 'COMMAND'`; return [ 'Long commands: any command you expect to run longer than 9 minutes that this prompt does not already hand you', 'as a check (a wide test filter, a long hook step) runs through run-checks, with DIR the checkout it runs in,', @@ -273,11 +282,11 @@ function longCheckRuleLines(m) { } // Project commands for a ledger lane; with dir, also the run-checks call that -// runs every check there at once, with the procedure under the reserve name -// name. -function commandsText(m, laneId, dir = null, name = 'checks') { +// runs every check there at once, in phase, with the procedure under the +// reserve name name. +function commandsText(m, laneId, dir = null, name = 'checks', phase = null) { const lines = ['setup', 'test', 'lint', 'build'].map((group) => `- ${group}: ${commandList(m, laneId, group)}`); - const all = dir === null ? null : checksCommand(m, laneId, dir, resultPath()); + const all = dir === null ? null : checksCommand(m, laneId, dir, resultPath(), phase); if (all !== null) { lines.push("- every check at once (each command's exit status is kept; it exits non-zero when any fails).", ...procedureLines(m, { name, start: all }).map((l) => ` ${l}`), ...checksLogLines()); @@ -298,7 +307,7 @@ function taskChecksCommand(m, task, where, base) { return [ `cd ${q(where.dir)} && ${pythonCommand(m)} ${q(`${m.skill_dir}/scripts/task-checks`)}`, q(where.dir), '--base', q(base), '--ledger', q(m.repo.ledger_dir), '--unit', q(task.id), '--evidence', resultPath(), - '--lane', q(where.lane), + '--lane', q(where.lane), '--phase', q(checksPhase(where.lane)), ...unitTasks(task).flatMap((t) => ['--task', q(t.id)]), ...declaredFiles(task).flatMap((f) => ['--declared', q(f)]), ...(Array.isArray(m.widen_paths) ? m.widen_paths : []).flatMap((p) => ['--widen', q(p)]), @@ -507,12 +516,15 @@ function taskContext(m, task, where, guidance = null, by = 'implementer') { '', 'Project commands (run from the worktree; the full set runs at integration and delivery):', commandsText(m, where.lane), + `Run project commands you start yourself with PL_PHASE=${checksPhase(where.lane)} set (for example`, + `\`PL_PHASE=${checksPhase(where.lane)} \`): the project's scripts may read it; the check calls`, + 'this prompt hands you set it themselves.', '', `Commit rules (follow exactly): ${m.commit_rules}`, 'Never commit anything under .superpowers/.', 'Never push, open pull requests, merge into the base branch, or copy work back to the project.', agentRules(), - ...longCheckRuleLines(m), + ...longCheckRuleLines(m, checksPhase(where.lane)), '', 'Contracts: report blocked for any change to a contract another lane consumes; ' + 'record smaller decisions as `Ruling: decision - why - cost if wrong` with:', @@ -615,7 +627,10 @@ function declaredFiles(task) { // superpowers present the review package for base..head. With opts.pkg and a // Files list it also lists the files base..head changes outside that list // (--scope); with opts.start (the task base, implement only) it records that -// base in the ledger once the briefs are written (--record-start). +// base in the ledger once the briefs are written (--record-start). For a +// join wave unit (where.unit_branch, 1.6.5) the implement start (opts.start) +// also puts the worktree on the unit's branch from where.from and runs the +// feature checkout's setup there (--unit-branch, --setup). function startCommand(m, task, where, opts = {}) { const sync = opts.sync || null; const pkg = opts.pkg || null; @@ -624,6 +639,10 @@ function startCommand(m, task, where, opts = {}) { shellQuote(where.dir), shellQuote(m.plan), '--artifacts', shellQuote(m.repo.ledger_dir), ]; if (present(sync)) parts.push('--sync', shellQuote(sync)); + if (present(where.unit_branch) && present(opts.start)) { + parts.push('--unit-branch', shellQuote(where.unit_branch), shellQuote(where.from), '--run-id', shellQuote(m.run_id), + ...finalSetup(m).flatMap((c) => ['--setup', shellQuote(c)])); + } if (pkg && m.sp_dir !== null) { const script = `${m.sp_dir}/subagent-driven-development/scripts/review-package`; const out = `${taskFiles(m, task).reviews}/${task.id}-${pkg.base}..${pkg.head}.diff`; @@ -647,6 +666,9 @@ function startCommand(m, task, where, opts = {}) { parts.push('--rulings', shellQuote(m.repo.ledger_dir), shellQuote(where.lane === 'join' ? '-' : where.lane)); if (present(opts.start)) { parts.push('--record-start', shellQuote(m.repo.ledger_dir), shellQuote(where.lane), shellQuote(opts.start)); + // A join unit begun again from the tip (1.6.5): the ledger drops its + // earlier attempt. + if (where.restart) parts.push('--restart'); } return parts.join(' '); } @@ -656,12 +678,16 @@ function startCommand(m, task, where, opts = {}) { function startBlock(m, task, where, opts, failure) { const what = [ ...(present(opts.sync) ? ['fast-forwards this worktree to the feature branch (it holds the prelude commits)'] : []), + ...(present(where.unit_branch) && present(opts.start) + ? [`puts this worktree on the unit's own branch ${where.unit_branch} (saving any uncommitted changes it finds ` + + 'under the run\'s abandoned refs) and runs the project\'s setup commands there'] : []), isBatch(task) ? 'regenerates every task brief from the current plan and prints it' : 'regenerates the task brief from the current plan and prints it', ...(opts.pkg && m.sp_dir !== null ? ['builds the review package'] : []), ...(opts.pkg && declaredFiles(task).length > 0 ? ["lists the files the range changes outside the Files list"] : []), - ...(present(opts.start) ? ['records the task base in the ledger'] : []), + ...(present(opts.start) ? [where.restart ? 'records the task base in the ledger as a restart' + : 'records the task base in the ledger'] : []), ]; const files = isBatch(task) ? 'brief files' : 'brief file'; return [ @@ -783,6 +809,26 @@ function reopenLines(m, task, where, reopen) { ]; } +// What a join unit begun again on the feature checkout is told (1.6.5 design +// 4.5): restart = {branch, dir, complete}, where its earlier attempt ran in a +// join wave of an earlier launch. That work never reached the feature branch. +function restartLines(m, task, restart, base) { + const q = shellQuote; + const dir = q(featureDir(m)); + const noun = unitNoun(task); + const folder = String(restart.dir).replace(/[\\/]+$/, '').split(/[\\/]/).pop().replace(/[^A-Za-z0-9._-]/g, '-'); + return [ + `This ${noun} ran before, in a join wave of an earlier launch, on its own branch ${restart.branch} in`, + `${restart.dir}, and that work never reached the feature branch. It was ${restart.complete + ? 'complete: committed, but its review did not finish' : 'partial: it stopped before it was done'}.`, + `Start again from the task base ${base} on the feature branch, and reuse what is right of the earlier work:`, + ` its commits: git -C ${dir} log HEAD..${q(restart.branch)} (and git -C ${dir} diff HEAD...${q(restart.branch)})`, + ` uncommitted changes saved from that worktree: git -C ${dir} for-each-ref ${q(`refs/parallel-lanes/${m.run_id}/abandoned/${folder}-`)}`, + ' (each such ref is a commit; read it with git show).', + 'Never merge, rebase or reset onto that branch as it is: bring over what is right as your own commits here.', + ]; +} + // Prompt for an implementer. base is the task base the script owns (the // previous task's head, or the feature tip); retry (optional) is {reason, // findings} when a previous attempt in this run blocked or failed review; @@ -831,6 +877,7 @@ function implementPrompt(m, task, where, base, retry = null, guidance = null, re ? ['Open review findings:', findingsText(retry.findings), ...startFindingNote(retry.findings)] : []), ].join('\n')); } + if (where.restart) parts.push('', restartLines(m, task, where.restart, base).join('\n')); const checks = taskChecksLines(m, task, where, base); if (checks.length > 0) parts.push('', checks.join('\n')); parts.push('', [ diff --git a/src/validate.js b/src/validate.js index 366fd4c..0c924eb 100644 --- a/src/validate.js +++ b/src/validate.js @@ -307,6 +307,13 @@ function validateManifest(m) { else if (!new RegExp(laneIdPattern()).test(lane.id)) { err(`${where}.id: lane id ${JSON.stringify(lane.id)} must match ${laneIdPattern()}`); } else if (['prelude', 'join'].includes(lane.id.toLowerCase())) err(`lane ${lane.id}: id is reserved`); + else if (lane.id.toLowerCase().startsWith('u-')) { + // A lane's branch pl-- would then have a join unit + // branch's name, pl--u- (1.6.5; refs are + // case-insensitive on Windows). + err(`${where}.id: lane id ${JSON.stringify(lane.id)} must not start with u- ` + + '(join unit branches use pl--u-)'); + } else if (laneIds.has(lane.id)) err(`lane ${lane.id}: id appears more than once`); else { laneIds.add(lane.id); @@ -653,6 +660,43 @@ function validateManifest(m) { } } + // Join waves (1.6.5): what ledger status and backfill say about the join. + // Once integration recorded the join's start point, the session must say + // whether the join started (a started join never runs waves again). + if ('join_started' in m && typeof m.join_started !== 'boolean') err('join_started: must be true or false'); + if (isObject(m.start_points) && 'join' in m.start_points && !('join_started' in m)) { + err('join_started: missing; copy it from ledger status (start_points.join is set)'); + } + if ('join_tip' in m && !isText(m.join_tip)) err('join_tip: must be a non-empty string'); + if ('unit_branches' in m) { + if (!isObject(m.unit_branches)) { + err('unit_branches: must be an object'); + } else { + const joinTasks = new Set(Array.isArray(m.join) ? m.join.filter(isObject).map((t) => t.id) : []); + for (const [id, v] of Object.entries(m.unit_branches)) { + if (!joinTasks.has(id)) err(`unit_branches.${id}: not a join task`); + else if (!isObject(v) || typeof v.branch !== 'string' || !/^pl-[a-z0-9-]+-u-[A-Za-z0-9_]/.test(v.branch)) { + err(`unit_branches.${id}: branch must be a unit branch pl--u-`); + } else if (!isText(v.dir)) err(`unit_branches.${id}: dir must be a non-empty string`); + } + } + } + if ('start_bases' in m) { + if (!isObject(m.start_bases)) { + err('start_bases: must be an object'); + } else { + const all = new Set([m.prelude, ...(Array.isArray(m.lanes) ? m.lanes.map((l) => (isObject(l) ? l.tasks : [])) : []), m.join] + .flatMap((list) => (Array.isArray(list) ? list.filter(isObject).map((t) => t.id) : []))); + for (const [id, base] of Object.entries(m.start_bases)) { + if (!all.has(id)) err(`start_bases.${id}: not a task of the run`); + else if (!isText(base)) err(`start_bases.${id}: must be a non-empty string`); + } + } + } + if ('wave_merges' in m && !(Array.isArray(m.wave_merges) && m.wave_merges.every(isObject))) { + err('wave_merges: must be a list of objects'); + } + return errors; } diff --git a/src/waves.js b/src/waves.js new file mode 100644 index 0000000..2b0ae4a --- /dev/null +++ b/src/waves.js @@ -0,0 +1,160 @@ +// ---- Join waves (1.6.5 design 4.1) ---- +// +// In the launch that starts the join, join units whose dependencies allow it +// run side by side, each on its own branch in a lane worktree, and each wave +// is merged back before the next starts. scripts/_waves.py repeats the rule +// for setup's owner map and worth-it; tests/waves.test.mjs holds both to the +// fixtures in tests/fixtures/waves/cases.json. + +// The branch a unit of a multi-unit wave runs on: pl--u-. +// Lane ids may not start with u-, so no lane branch has this form. +function unitBranch(m, unit) { + return `pl-${m.run_id}-u-${unit.id}`; +} + +// The ids a unit depends on: null when the unit (or any task of a batch) has +// no depends_on field (it then waits for every earlier unit); else the ids of +// every task's depends_on entries (objects with an id, or plain ids), less the +// unit's own task ids, without duplicates. A batch whose tasks are given as +// plain ids carries depends_on on the unit itself. +function waveDeps(unit) { + const tasks = Array.isArray(unit.tasks) ? unit.tasks : [unit]; + const objects = tasks.filter((t) => t !== null && typeof t === 'object'); + const holders = objects.length > 0 ? objects : [unit]; + if (holders.some((t) => !Array.isArray(t.depends_on))) return null; + const own = new Set(tasks.map((t) => (typeof t === 'string' ? t : t.id))); + const out = []; + for (const t of holders) { + for (const d of t.depends_on) { + const id = d !== null && typeof d === 'object' ? d.id : d; + if (!own.has(id) && !out.includes(id)) out.push(id); + } + } + return out; +} + +// A declared file as the wave rule compares it: '\' as '/', leading './' +// and repeated '/' dropped, lower-cased. +function waveFileKey(path) { + return String(path).replace(/\\/g, '/').replace(/\/+/g, '/').replace(/^(\.\/)+/, '').toLowerCase(); +} + +// The waves of the join's units, in plan order: [[unit id, ...], ...]. +// units: [{id, tasks?, depends_on?, files}] (a batch unit lists its tasks; +// its tasks' ids, plain or as objects, are what other units name). edges: +// [{unit, producer}] unit ids, pre-flight's findings. A unit depends on every +// earlier unit when waveDeps is null, else on the earlier units holding an id +// it names, and on each edge's producer that is earlier (later producers and +// later ids are ignored: validation refuses a code dependency on a later join +// task). Its wave is 1 + the highest wave it depends on; then, while an +// earlier unit of that wave declares one of its files (waveFileKey), the next. +function joinWaves(units, edges = []) { + const ids = (u) => (Array.isArray(u.tasks) ? u.tasks.map((t) => (typeof t === 'string' ? t : t.id)) : [u.id]); + const wave = []; + units.forEach((u, i) => { + const earlier = units.slice(0, i); + const deps = waveDeps(u); + const on = new Set(); + earlier.forEach((e, j) => { + if (deps === null || ids(e).some((id) => deps.includes(id))) on.add(j); + }); + for (const edge of edges) { + if (edge.unit !== u.id) continue; + const j = earlier.findIndex((e) => e.id === edge.producer); + if (j >= 0) on.add(j); + } + let w = 1 + Math.max(0, ...[...on].map((j) => wave[j])); + const mine = new Set((u.files || []).map(waveFileKey)); + const clash = (n) => earlier.some((e, j) => wave[j] === n && (e.files || []).some((f) => mine.has(waveFileKey(f)))); + while (clash(w)) w += 1; + wave.push(w); + }); + const out = []; + units.forEach((u, i) => { + while (out.length < wave[i]) out.push([]); + out[wave[i] - 1].push(u.id); + }); + return out.filter((w) => w.length > 0); +} + +// The join's units as the run forms them (batchGroup, batchUnit). +function joinUnits(m) { + const units = []; + for (let i = 0; i < m.join.length;) { + const group = batchGroup(m, m.join, i); + i += group.length; + units.push(group.length > 1 ? batchUnit(group) : group[0]); + } + return units; +} + +// Units of a wave that run at once: limits.max_parallel_lanes, capped by the +// lane worktrees a unit can use (a lane with a setup of its own in +// lane_commands is not one: its environment is not the project's). +function waveWidth(m) { + const own = m.lane_commands || {}; + const usable = m.lanes.filter((l) => !(Object.prototype.hasOwnProperty.call(own, l.id) + && own[l.id] && Object.prototype.hasOwnProperty.call(own[l.id], 'setup'))).length; + return Math.min(m.limits.max_parallel_lanes, usable); +} + +// The lane worktrees a wave unit can run in: those of the lanes waveWidth +// counts as usable, in lane order. +function waveLaneDirs(m) { + const own = m.lane_commands || {}; + return m.lanes.filter((l) => !(Object.prototype.hasOwnProperty.call(own, l.id) + && own[l.id] && Object.prototype.hasOwnProperty.call(own[l.id], 'setup'))).map((l) => laneWhere(m, l).dir); +} + +// The checks after the last multi-unit wave (design 4.4): the union of its +// units' targeted checks (their task_checks entries; a unit without one adds +// nothing), as [{group, command}] without repeats. +function waveChecksOf(m, units) { + const out = []; + for (const unit of units) { + const entry = taskChecksEntry(m, unit, 'join'); + for (const c of entry === null ? [] : entry.checks) { + if (!out.some((o) => o.group === c.group && o.command === c.command)) out.push(c); + } + } + return out; +} + +// The units a relaunch's recovery merge brings in (design 4.5): join units +// in plan order that ran on a unit branch (m.unit_branches) and whose every +// task is done and reviewed or deferred, as [{id, base, head}] from their +// backfill range; merge-wave skips one already in the feature history. +function recoveryUnits(m) { + const branches = m.unit_branches || {}; + const done = new Set(m.done || []); + const settled = new Set([...(m.reviewed || []), ...(m.deferred || [])]); + const out = []; + for (const unit of joinUnits(m)) { + const tasks = unitTasks(unit); + if (!tasks.some((t) => Object.prototype.hasOwnProperty.call(branches, t.id))) continue; + if (!tasks.every((t) => done.has(t.id) && settled.has(t.id))) continue; + const range = (m.backfill || {})[tasks[tasks.length - 1].id]; + if (range) out.push({ id: unit.id, base: range.base, head: range.head }); + } + return out; +} + +// The join started in an earlier launch: the ledger's join_started (copied +// into the manifest), or any join task done or deferred (fail-safe). +function joinStarted(m) { + if (m.join_started === true) return true; + const done = new Set([...(m.done || []), ...(m.deferred || [])]); + return m.join.some((t) => done.has(t.id)); +} + +// How the join runs: {mode: 'waves'|'serial', reason, waves}. waves is the +// rule's result either way; reason says why a join runs serially. +function wavePlan(m, edges = []) { + const waves = joinWaves(joinUnits(m), edges); + const serial = (reason) => ({ mode: 'serial', reason, waves }); + if (m.profile === 'lite') return serial('lite'); + if (joinStarted(m)) return serial('join started'); + if (waveWidth(m) < 2) return serial('fewer than two usable lane worktrees'); + if (!waves.some((w) => w.length > 1)) return serial('no wave has two units'); + return { mode: 'waves', reason: null, waves }; +} diff --git a/tests/check-policy-prompts.test.mjs b/tests/check-policy-prompts.test.mjs index 91a5709..7499efc 100644 --- a/tests/check-policy-prompts.test.mjs +++ b/tests/check-policy-prompts.test.mjs @@ -439,11 +439,19 @@ test('the general long-command rule reaches implementers and phase agents but no if (name === 'adjudicator') { assert.ok(!p.includes(rule) && !p.includes('longer than 9 minutes'), name); } else { - assert.ok(p.includes(rule), name); + // Task prompts carry it with their ledger lane's phase (1.6.5: run-checks + // --phase); phase prompts without one. + const phased = [null, 'prelude', 'lane', 'join'].map((ph) => longCheckRuleLines(m, ph).join('\n')); + assert.ok(phased.some((r) => p.includes(r)), name); } } }); +test('a task prompt\'s long-command rule names the task\'s phase', () => { + const rule = flat(longCheckRuleLines(manifest(), 'join').join('\n')); + assert.ok(rule.includes("--root '/work/ledger' --phase 'join' --cmd 'GROUP' 'COMMAND'"), rule); +}); + test('the result placeholder is single-quoted in every start command', () => { for (const [name, p] of Object.entries(everyPrompt())) { const flags = p.match(/--(out|evidence) \S+/g) || []; diff --git a/tests/cleanup.test.mjs b/tests/cleanup.test.mjs index a5fa903..e542d72 100644 --- a/tests/cleanup.test.mjs +++ b/tests/cleanup.test.mjs @@ -866,3 +866,45 @@ test('cleanup: the plan alone takes no lock', () => { cleanupOk(c); assert.ok(!existsSync(c.active) || readdirSync(c.active).length === 0, 'the active dir holds no file'); }); + +// ---- 1.6.5: join unit branches (pl--u-) ---- + +test('merged unit branches and their worktrees are removed', () => { + const c = newCase(); + git(c.lanes.a, 'checkout', '-q', '-b', 'pl-r1-u-J1', 'feature/x'); + commitFile(c.lanes.a, 'j1.txt', 'unit J1\n'); + git(c.project, 'merge', '-q', '--no-ff', '-m', 'join wave 1: merge J1', 'pl-r1-u-J1'); + const plan = cleanupOk(c); + const wt = plan.worktrees.find((w) => samePath(w.path, c.lanes.a)); + assert.ok(wt, JSON.stringify(plan.worktrees)); + assert.equal(wt.branch, 'pl-r1-u-J1'); + assert.equal(wt.action, 'remove', JSON.stringify(wt)); + const unit = plan.branches.find((b) => b.branch === 'pl-r1-u-J1'); + assert.ok(unit, JSON.stringify(plan.branches)); + assert.equal(unit.action, 'remove', JSON.stringify(unit)); + assert.equal(plan.foreign.length, 0, JSON.stringify(plan.foreign)); + cleanupOk(c, ['--yes']); + assert.equal(sh('git', ['-C', c.project, 'rev-parse', '--verify', '--quiet', 'refs/heads/pl-r1-u-J1']).code, 1); + assert.ok(!existsSync(c.lanes.a)); +}); + +test('an unmerged unit branch is kept', () => { + const c = newCase(); + git(c.lanes.b, 'checkout', '-q', '-b', 'pl-r1-u-J2', 'feature/x'); + commitFile(c.lanes.b, 'j2.txt', 'unit J2, blocked\n'); + git(c.lanes.b, 'checkout', '-q', 'pl-r1-b'); + const plan = cleanupOk(c); + const unit = plan.branches.find((b) => b.branch === 'pl-r1-u-J2'); + assert.ok(unit, JSON.stringify(plan.branches)); + assert.equal(unit.action, 'keep'); + assert.match(unit.reason, /not merged/); + cleanupOk(c, ['--yes']); + assert.equal(sh('git', ['-C', c.project, 'rev-parse', '--verify', '--quiet', 'refs/heads/pl-r1-u-J2']).code, 0); +}); + +test('another run\'s unit branch is not claimed', () => { + const c = newCase(); + git(c.project, 'branch', 'pl-r2-u-J1', 'feature/x'); + const plan = cleanupOk(c); + assert.ok(!plan.branches.some((b) => b.branch === 'pl-r2-u-J1'), JSON.stringify(plan.branches)); +}); diff --git a/tests/dryrun.test.mjs b/tests/dryrun.test.mjs index 18b3a34..8b6e7eb 100644 --- a/tests/dryrun.test.mjs +++ b/tests/dryrun.test.mjs @@ -480,3 +480,75 @@ test('task_checks leaves finalChecks, finalSetup and expectedChecks unchanged', assert.deepEqual(finalSetup(withChecks), finalSetup(base)); assert.deepEqual(expectedChecks(withChecks), expectedChecks(base)); }); + +// ---- 1.6.5: join waves in the dry run ---- + +const dep = (...ids) => ids.map((id) => ({ id, kind: 'code' })); +// d1a's join shape: waves [DJ1], [DJ2, DJ4], [DJ3, DJ5, DJ6, DJ7]. +function wavesManifest(overrides = {}) { + return manifest({ + lanes: [ + { id: 'alpha', name: 'Lane alpha', tasks: [task('T2', ['src/a.js'])] }, + { id: 'beta', name: 'Lane beta', tasks: [task('T3', ['src/b.js'])] }, + { id: 'gamma', name: 'Lane gamma', tasks: [task('T4', ['src/c.js'])] }, + ], + join: [ + task('DJ1', ['j1.cs'], { depends_on: [] }), + task('DJ2', ['j2.cs'], { depends_on: dep('DJ1') }), + task('DJ3', ['j3.cs'], { depends_on: dep('DJ1', 'DJ2') }), + task('DJ4', ['j4.cs'], { depends_on: dep('DJ1') }), + task('DJ5', ['j5.cs'], { depends_on: dep('DJ1', 'DJ4') }), + task('DJ6', ['j6.cs'], { depends_on: dep('DJ4') }), + task('DJ7', ['j7.cs'], { depends_on: dep('DJ4') }), + ], + ...overrides, + }); +} + +test('d1a-shaped manifest gives waves 1-2-4', async () => { + const dry = await loadScript({ args: wavesManifest() }); + assert.deepEqual(dry.errors, []); + assert.deepEqual(dry.join_waves, { + mode: 'waves', reason: null, waves: [['DJ1'], ['DJ2', 'DJ4'], ['DJ3', 'DJ5', 'DJ6', 'DJ7']], + }); +}); + +test('a serial join says why', async () => { + const dry = await loadScript({ args: manifest() }); + assert.deepEqual(dry.join_waves, { mode: 'serial', reason: 'no wave has two units', waves: [['T5']] }); + assert.ok(!dry.notices.launch.includes('join waves')); + assert.ok(!dry.agents.some((a) => /^(wave_|recovery_)/.test(a.role))); +}); + +test('the wave merge agents are counted', () => { + // The last multi-unit wave's checks (DJ3's task_checks entry) add the fix + // and its re-review and recheck, as the upper bound. + const checked = wavesManifest({ task_checks: { DJ3: { test: ['npm test -- dj3'] } } }); + const roles = planAgents(checked).filter((a) => a.phase === 'Join' && !a.task).map((a) => a.role); + assert.deepEqual(roles, ['wave_merge', 'wave_merge', 'wave_checks_fix', 'wave_checks_re_review', 'wave_checks']); + const merge = planAgents(wavesManifest()).find((a) => a.role === 'wave_merge'); + assert.deepEqual([merge.model, merge.effort], ['sonnet', 'high']); + const fix = planAgents(checked).find((a) => a.role === 'wave_checks_fix'); + assert.deepEqual([fix.model, fix.effort], ['opus', 'high']); + const unchecked = planAgents(wavesManifest()).filter((a) => a.phase === 'Join' && !a.task).map((a) => a.role); + assert.deepEqual(unchecked, ['wave_merge', 'wave_merge'], 'no checks after the last wave: no fix to reserve'); +}); + +test('a relaunch lists the recovery merge', () => { + const m = wavesManifest({ + dry_run: false, + done: ['T2', 'T3', 'T4', 'DJ1'], reviewed: ['T2', 'T3', 'T4', 'DJ1'], + backfill: Object.fromEntries(['T2', 'T3', 'T4', 'DJ1'].map((id) => [id, { base: H(`${id}b`), head: H(id) }])), + join_started: true, + unit_branches: { DJ2: { branch: 'pl-run-1-u-DJ2', dir: '/work/wt/lane-alpha' } }, + }); + const agents = planAgents(m); + const join = agents.filter((a) => a.phase === 'Join'); + assert.equal(join[0].role, 'recovery_merge'); + assert.ok(!agents.some((a) => a.role === 'wave_merge'), 'a relaunch runs the rest serially'); +}); + +test('the launch notice names the waves', async () => { + const dry = await loadScript({ args: wavesManifest() }); + assert.match(dry.notices.launch, /, join waves 1-2-4$/); +}); diff --git a/tests/fixtures/waves/cases.json b/tests/fixtures/waves/cases.json new file mode 100644 index 0000000..a524c41 --- /dev/null +++ b/tests/fixtures/waves/cases.json @@ -0,0 +1,891 @@ +[ + { + "name": "m1a", + "units": [ + { + "id": "J1", + "files": [ + "f1", + "f2", + "f3", + "f4", + "f5", + "f6" + ], + "depends_on": [ + "I4", + "I5", + "I6", + "I7", + "I8", + "PL2", + "PL3" + ] + }, + { + "id": "J2", + "files": [ + "f7", + "f8", + "f9" + ], + "depends_on": [ + "J1", + "N1" + ] + }, + { + "id": "J3", + "files": [ + "f10", + "f11" + ], + "depends_on": [ + "I7", + "PL3", + "P3" + ] + } + ], + "edges": [], + "expected": [ + [ + "J1", + "J3" + ], + [ + "J2" + ] + ] + }, + { + "name": "sb1", + "units": [ + { + "id": "JB1", + "files": [ + "f12", + "f13", + "f14", + "f15" + ], + "depends_on": [ + "CR3", + "CR4", + "AU2", + "AU3", + "WD1" + ] + }, + { + "id": "JB2", + "files": [ + "f16", + "f17", + "f8", + "f18", + "f19" + ], + "depends_on": [ + "AU2", + "NW1", + "NW2", + "NW3", + "NW4", + "WD2", + "JB1" + ] + }, + { + "id": "JB3", + "files": [ + "f20", + "f21", + "f22", + "f23", + "f10" + ], + "depends_on": [ + "CR4", + "NW1", + "PL1", + "PL2" + ] + }, + { + "id": "JB4", + "files": [ + "f24", + "f25" + ], + "depends_on": [ + "BP1", + "JB3" + ] + } + ], + "edges": [], + "expected": [ + [ + "JB1", + "JB3" + ], + [ + "JB2", + "JB4" + ] + ] + }, + { + "name": "sc1", + "units": [ + { + "id": "CJ1", + "files": [ + "f26", + "f27", + "f28", + "f29", + "f30", + "f31", + "f32", + "f33", + "f34", + "f35", + "f12", + "f14" + ], + "depends_on": [ + "CR2", + "CR3", + "CR4", + "CK1", + "CK2", + "CA1", + "CA4", + "CA5", + "CV2", + "CH3", + "CH9", + "CH5", + "CH6" + ] + }, + { + "id": "CJ2", + "files": [ + "f3", + "f36", + "f37", + "f7", + "f16", + "f1", + "f8", + "f38", + "f6", + "f9", + "f18", + "f39", + "f40", + "f4" + ], + "depends_on": [ + "CR5", + "CA2", + "CA3", + "CI1", + "CS3", + "CS4", + "CV3", + "CV4", + "CV5", + "CH5", + "CH6", + "CH7", + "CJ1" + ] + }, + { + "id": "CJ3", + "files": [ + "f41", + "f42", + "f8", + "f38", + "f16", + "f43", + "f44" + ], + "depends_on": [ + "CR5", + "CJ1", + "CJ2" + ] + }, + { + "id": "CJ4", + "files": [ + "f45", + "f46", + "f47", + "f48", + "f49", + "f50", + "f51", + "f52" + ], + "depends_on": [ + "CA3", + "CI1", + "CV3", + "CV2", + "CV4", + "CV5", + "CH5", + "CH6" + ] + }, + { + "id": "CJ5", + "files": [ + "f53", + "f54", + "f55", + "f56", + "f57", + "f58" + ], + "depends_on": [ + "CR5", + "CS3", + "CJ3" + ] + } + ], + "edges": [], + "expected": [ + [ + "CJ1", + "CJ4" + ], + [ + "CJ2" + ], + [ + "CJ3" + ], + [ + "CJ5" + ] + ] + }, + { + "name": "d1a", + "units": [ + { + "id": "DJ1", + "files": [ + "f59", + "f60", + "f3", + "f12", + "f1", + "f61", + "f62", + "f6", + "f14", + "f4", + "f18", + "f43", + "f9" + ], + "depends_on": [ + "DS2", + "DU2", + "DU4", + "DPL2", + "DI2", + "DI3", + "DI4", + "DI5", + "DI6", + "DI7", + "DW1", + "DW2", + "DW4", + "DW5", + "DC1", + "DA1", + "DT2" + ] + }, + { + "id": "DJ2", + "files": [ + "f63", + "f64", + "f8", + "f16", + "f65", + "f66", + "f67", + "f36", + "f7", + "f41", + "f68", + "f69", + "f70", + "f71", + "f72", + "f39", + "f18", + "f43", + "f9" + ], + "depends_on": [ + "DS3", + "DN1", + "DN2", + "DN3", + "DU1", + "DU2", + "DU3", + "DU5", + "DI2", + "DI7", + "DW3", + "DT1", + "DJ1" + ] + }, + { + "id": "DJ3", + "files": [ + "f73", + "f74", + "f75" + ], + "depends_on": [ + "DP2", + "DU1", + "DU2", + "DT1", + "DJ1", + "DJ2" + ] + }, + { + "id": "DJ4", + "files": [ + "f76", + "f77", + "f78", + "f79", + "f80", + "f81" + ], + "depends_on": [ + "DS3", + "DN1", + "DN2", + "DN3", + "DPL2", + "DI2", + "DI3", + "DI4", + "DI7", + "DW3", + "DW4", + "DJ1" + ] + }, + { + "id": "DJ5", + "files": [ + "f82", + "f83" + ], + "depends_on": [ + "DW3", + "DW5", + "DC1", + "DJ1", + "DJ4" + ] + }, + { + "id": "DJ6", + "files": [ + "f84", + "f85" + ], + "depends_on": [ + "DW1", + "DW2", + "DA1", + "DJ4" + ] + }, + { + "id": "DJ7", + "files": [ + "f86", + "f87" + ], + "depends_on": [ + "DPL2", + "DI5", + "DI6", + "DW4", + "DC1", + "DJ4" + ] + } + ], + "edges": [], + "expected": [ + [ + "DJ1" + ], + [ + "DJ2", + "DJ4" + ], + [ + "DJ3", + "DJ5", + "DJ6", + "DJ7" + ] + ] + }, + { + "name": "no-field", + "units": [ + { + "id": "a", + "files": [ + "src/a.cs" + ] + }, + { + "id": "b", + "files": [ + "src/b.cs" + ] + }, + { + "id": "c", + "files": [ + "src/c.cs" + ] + } + ], + "edges": [], + "expected": [ + [ + "a" + ], + [ + "b" + ], + [ + "c" + ] + ] + }, + { + "name": "empty-deps", + "units": [ + { + "id": "a", + "files": [ + "src/a.cs" + ], + "depends_on": [] + }, + { + "id": "b", + "files": [ + "src/b.cs" + ], + "depends_on": [] + }, + { + "id": "c", + "files": [ + "src/c.cs" + ], + "depends_on": [] + } + ], + "edges": [], + "expected": [ + [ + "a", + "b", + "c" + ] + ] + }, + { + "name": "outside-ids-ignored", + "units": [ + { + "id": "a", + "files": [ + "src/a.cs" + ], + "depends_on": [ + "L1" + ] + }, + { + "id": "b", + "files": [ + "src/b.cs" + ], + "depends_on": [ + "L2", + "P1" + ] + } + ], + "edges": [], + "expected": [ + [ + "a", + "b" + ] + ] + }, + { + "name": "file-overlap", + "units": [ + { + "id": "a", + "files": [ + "src/X.cs" + ], + "depends_on": [] + }, + { + "id": "b", + "files": [ + "./SRC//x.cs" + ], + "depends_on": [] + } + ], + "edges": [], + "expected": [ + [ + "a" + ], + [ + "b" + ] + ] + }, + { + "name": "file-overlap-backslash", + "units": [ + { + "id": "a", + "files": [ + "src\\y.cs" + ], + "depends_on": [] + }, + { + "id": "b", + "files": [ + "src/Y.cs" + ], + "depends_on": [] + } + ], + "edges": [], + "expected": [ + [ + "a" + ], + [ + "b" + ] + ] + }, + { + "name": "overlap-cascade", + "units": [ + { + "id": "a", + "files": [ + "s.cs" + ], + "depends_on": [] + }, + { + "id": "b", + "files": [ + "s.cs" + ], + "depends_on": [] + }, + { + "id": "c", + "files": [ + "src/c.cs" + ], + "depends_on": [ + "b" + ] + } + ], + "edges": [], + "expected": [ + [ + "a" + ], + [ + "b" + ], + [ + "c" + ] + ] + }, + { + "name": "overlap-moves-past-taken-wave", + "units": [ + { + "id": "a", + "files": [ + "s.cs" + ], + "depends_on": [] + }, + { + "id": "b", + "files": [ + "t.cs" + ], + "depends_on": [ + "a" + ] + }, + { + "id": "c", + "files": [ + "s.cs", + "t.cs" + ], + "depends_on": [] + } + ], + "edges": [], + "expected": [ + [ + "a" + ], + [ + "b" + ], + [ + "c" + ] + ] + }, + { + "name": "edge", + "units": [ + { + "id": "a", + "files": [ + "src/a.cs" + ], + "depends_on": [] + }, + { + "id": "b", + "files": [ + "src/b.cs" + ], + "depends_on": [] + } + ], + "edges": [ + { + "unit": "b", + "producer": "a" + } + ], + "expected": [ + [ + "a" + ], + [ + "b" + ] + ] + }, + { + "name": "forward-edge-ignored", + "units": [ + { + "id": "a", + "files": [ + "src/a.cs" + ], + "depends_on": [] + }, + { + "id": "b", + "files": [ + "src/b.cs" + ], + "depends_on": [] + } + ], + "edges": [ + { + "unit": "a", + "producer": "b" + } + ], + "expected": [ + [ + "a", + "b" + ] + ] + }, + { + "name": "forward-dep-ignored", + "units": [ + { + "id": "a", + "files": [ + "src/a.cs" + ], + "depends_on": [ + "b" + ] + }, + { + "id": "b", + "files": [ + "src/b.cs" + ], + "depends_on": [] + } + ], + "edges": [], + "expected": [ + [ + "a", + "b" + ] + ] + }, + { + "name": "batch", + "units": [ + { + "id": "a", + "files": [ + "src/a.cs" + ], + "depends_on": [] + }, + { + "id": "b1-b2", + "tasks": [ + "b1", + "b2" + ], + "files": [ + "x.cs", + "y.cs" + ], + "depends_on": [ + "a" + ] + }, + { + "id": "c", + "files": [ + "src/c.cs" + ], + "depends_on": [ + "b2" + ] + } + ], + "edges": [], + "expected": [ + [ + "a" + ], + [ + "b1-b2" + ], + [ + "c" + ] + ] + }, + { + "name": "single", + "units": [ + { + "id": "a", + "files": [ + "src/a.cs" + ] + } + ], + "edges": [], + "expected": [ + [ + "a" + ] + ] + }, + { + "name": "empty", + "units": [], + "edges": [], + "expected": [] + }, + { + "name": "batch-ids-independent", + "units": [ + { + "id": "a", + "files": [ + "a.cs" + ], + "depends_on": [] + }, + { + "id": "b1-b2", + "tasks": [ + "b1", + "b2" + ], + "files": [ + "x.cs" + ], + "depends_on": [] + } + ], + "edges": [], + "expected": [ + [ + "a", + "b1-b2" + ] + ] + } +] diff --git a/tests/io.test.mjs b/tests/io.test.mjs index a04a4a6..de5e881 100644 --- a/tests/io.test.mjs +++ b/tests/io.test.mjs @@ -174,11 +174,13 @@ test('a path that cannot be converted fails with the helper message, not a trace writeFileSync(bash, ''); const env = { ...process.env, CLAUDE_CODE_GIT_BASH_PATH: bash }; // Standard modules load before the platform changes (some pick their - // implementation by platform at import); _shell reads it at call time. + // implementation by platform at import), and so does _oslock (fcntl or + // msvcrt); _shell reads it at call time. const code = [ 'import argparse, glob, hashlib, json, os, posixpath, re, runpy, shutil, subprocess, sys', ...helperImports(SCRIPTS), 'import _shell', + 'import _oslock', 'sys.platform = "win32"', 'sys.argv = sys.argv[1:]', 'runpy.run_path(sys.argv[0], run_name="__main__")', diff --git a/tests/join-waves-e2e.test.mjs b/tests/join-waves-e2e.test.mjs new file mode 100644 index 0000000..9bcc991 --- /dev/null +++ b/tests/join-waves-e2e.test.mjs @@ -0,0 +1,230 @@ +// Join waves end to end on real git (1.6.5 design section 6): the real setup, +// start-task, finish-task, ledger and merge-wave scripts, driven by scripted +// agents that run the commands their prompts hand out. A 2-lane plan whose +// join is one wave of two units; then a run whose second unit stops after a +// commit, and a relaunch through setup, the ledger, the recovery merge and the +// serial continuation. +import { test, after } from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { existsSync, mkdirSync, readFileSync, realpathSync, rmSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { loadScript, SKILL_DIR } from './harness.mjs'; +import { BASH, mergeEnv, tempDir } from './platform.mjs'; + +const SCRIPTS = join(SKILL_DIR, 'scripts'); +const TMP = realpathSync(tempDir('pl-join-waves-e2e-')); +after(() => rmSync(TMP, { recursive: true, force: true })); +const GIT_ENV = { + GIT_CONFIG_GLOBAL: '/dev/null', GIT_CONFIG_NOSYSTEM: '1', + GIT_AUTHOR_NAME: 'T', GIT_AUTHOR_EMAIL: 't@example.invalid', + GIT_COMMITTER_NAME: 'T', GIT_COMMITTER_EMAIL: 't@example.invalid', +}; +const fwd = (p) => p.replace(/\\/g, '/'); + +function sh(command, env = {}) { + const r = spawnSync(BASH, ['-c', command], { encoding: 'utf8', env: mergeEnv(process.env, GIT_ENV, env) }); + return { code: r.status, stdout: r.stdout.replace(/\r\n/g, '\n'), stderr: r.stderr }; +} +function git(dir, ...args) { + const r = spawnSync('git', ['-C', dir, ...args], { encoding: 'utf8', env: mergeEnv(process.env, GIT_ENV) }); + assert.equal(r.status, 0, `git ${args.join(' ')}: ${r.stderr}`); + return r.stdout.replace(/\r\n/g, '\n').trim(); +} +function py(script, args, env = {}) { + const r = spawnSync('python3', ['-B', join(SCRIPTS, script), ...args], + { encoding: 'utf8', env: mergeEnv(process.env, GIT_ENV, env) }); + return { code: r.status, stdout: r.stdout, stderr: r.stderr }; +} + +const PLAN = ['# Plan', '', ...['A1', 'B1', 'J1', 'J2'].flatMap((id) => [ + `### Task ${id}: write ${id}`, '', `**Files:** \`${id}.txt\``, '', `Write ${id}.txt.`, '']), +].join('\n'); + +let counter = 0; +function newCase() { + counter += 1; + const root = fwd(join(TMP, `case${counter}`)); + const repo = `${root}/repo`; + mkdirSync(repo, { recursive: true }); + git(repo, 'init', '-q', '-b', 'main'); + writeFileSync(join(repo, 'README.md'), 'hello\n'); + git(repo, 'add', '-A'); + git(repo, 'commit', '-q', '-m', 'init'); + writeFileSync(join(root, 'plan.md'), PLAN); + const task = (id, extra = {}) => ({ id, title: `write ${id}`, files: [`${id}.txt`], tier: 'standard', security: false, ...extra }); + const m = { + version: 1, run_id: 'w1', plan: `${root}/plan.md`, spec: null, commit_rules: 'plain ASCII', + repo: { mode: 'git', root: repo, git_dir: null, base_ref: 'main', branch: 'feature/w', worktree_root: `${root}/wt`, + ledger_dir: `${root}/ledger` }, + commands: { setup: [], test: [], lint: [], build: [] }, + prelude: [], + lanes: [{ id: 'a', name: 'Lane a', tasks: [task('A1')] }, { id: 'b', name: 'Lane b', tasks: [task('B1')] }], + join: [task('J1', { depends_on: [] }), task('J2', { depends_on: [] })], + hooks: {}, limits: { review_rounds: 2, max_parallel_lanes: 2 }, dry_run: false, done: [], reviewed: [], + sp_dir: null, skill_dir: fwd(SKILL_DIR), autonomy: 'supervised', + }; + return { root, repo, m, active: `${root}/active` }; +} + +// scripts/setup as the holder of the run's launch lock; its output. +function setup(c, m) { + const path = `${c.root}/manifest.json`; + writeFileSync(path, JSON.stringify(m)); + if (!c.token) { + const got = spawnSync(BASH, [join(SCRIPTS, 'active-run'), 'acquire', m.run_id, path], + { encoding: 'utf8', env: mergeEnv(process.env, { PL_ACTIVE_DIR: c.active }) }); + assert.equal(got.status, 0, got.stderr); + c.token = got.stdout.trim(); + } + const r = py('setup', [path, '--owner', c.token], { PL_ACTIVE_DIR: c.active }); + assert.equal(r.code, 0, r.stderr); + return JSON.parse(r.stdout); +} + +// The line after the first line of prompt that matches re, trimmed. +function lineAfter(prompt, re) { + const lines = prompt.split('\n'); + const i = lines.findIndex((l) => re.test(l)); + assert.ok(i >= 0 && i + 1 < lines.length, `no line after ${re} in:\n${prompt}`); + return lines[i + 1].trim(); +} +const lineWith = (prompt, needle) => { + const line = prompt.split('\n').find((l) => l.includes(needle)); + assert.ok(line, `no line with ${needle}`); + return line.trim(); +}; +const dirOf = (command) => command.match(/^cd '([^']*)'/)[1]; +function run1(command) { + const r = sh(command); + assert.equal(r.code, 0, `${command}\n${r.stderr}${r.stdout}`); + return r.stdout; +} + +// Scripted agents. stopAfterCommit: unit ids whose implementer commits, +// records it, and then reports blocked (with an uncommitted change left). +function agents(c, { stopAfterCommit = [] } = {}) { + const calls = []; + const featureHead = () => git(c.repo, 'rev-parse', 'HEAD'); + const agent = async (prompt, opts) => { + calls.push({ label: opts.label, prompt }); + const label = opts.label; + const [id, role] = label.split(' '); + if (label === 'pre-flight') return { conflicts: [], rulings: [], undeclared: [], code_deps: [] }; + if (label === 'integrate') { + for (const lane of ['a', 'b']) { + const merged = sh(`git -C '${c.repo}' merge-base --is-ancestor 'pl-w1-${lane}' HEAD`).code === 0; + if (!merged) git(c.repo, 'merge', '-q', '--no-ff', '-m', `merge lane ${lane}`, `pl-w1-${lane}`); + } + return { status: 'done', head: featureHead(), notes: 'merged' }; + } + if (/^(wave \d+|recovery) merge$/.test(label)) { + const r = sh(lineAfter(prompt, /^1\. Run, as one call:/)); + assert.ok(r.code === 0 || r.code === 1, r.stderr); + const out = JSON.parse(r.stdout); + return { ...out, status: out.status === 'merged' ? 'done' : 'conflict', conflict_unit: out.conflict_unit || '' }; + } + if (label.startsWith('final review')) return { findings: [], cannot_verify: [], head: featureHead() }; + if (role === 'implement') { + const start = lineAfter(prompt, /^Run this first, as one call/); + run1(start); + const dir = dirOf(start); + writeFileSync(join(dir, `${id}.txt`), `${id}\n`); + git(dir, 'add', '-A'); + git(dir, 'commit', '-q', '-m', `feat: write ${id}`); + run1(lineWith(prompt, "/scripts/finish-task' ")); + const head = git(dir, 'rev-parse', 'HEAD'); + if (stopAfterCommit.includes(id)) { + writeFileSync(join(dir, `${id}-wip.txt`), 'unfinished\n'); + return { status: 'blocked', head, tests: '', notes: `${id} could not finish` }; + } + return { status: 'done', head, tests: '(none)', notes: '', changed_lines: 1 }; + } + if (role === 'review') { + const cmd = lineWith(prompt, "/scripts/ledger' reviewed").replace('', '0'); + run1(cmd); + return { verdict: 'approve', findings: [], cannot_verify: [] }; + } + throw new Error(`unscripted agent: ${label}`); + }; + return { agent, calls }; +} + +async function runAll(m, scripted) { + const parallel = (thunks) => Promise.all(thunks.map((t) => t().catch(() => null))); + return loadScript({ args: m, agent: scripted.agent, parallel, phase: () => {}, log: () => {} }); +} +const subjects = (c, range) => git(c.repo, 'log', '--first-parent', '--format=%s', range).split('\n').filter(Boolean); +const ledgerStatus = (c) => { + const r = py('ledger', ['status', c.m.repo.ledger_dir, '--plan', c.m.plan]); + assert.equal(r.code, 0, r.stderr); + return JSON.parse(r.stdout); +}; + +test('a join wave runs on unit branches in lane worktrees and is merged', async () => { + const c = newCase(); + const setupResult = setup(c, c.m); + const s = agents(c); + const result = await runAll({ ...c.m, setup_result: setupResult }, s); + assert.equal(result.status, 'complete', JSON.stringify(result.reason || result.errors)); + for (const id of ['J1', 'J2']) { + const p = s.calls.find((x) => x.label === `${id} implement`).prompt; + assert.ok(p.includes(`--unit-branch 'pl-w1-u-${id}'`), id); + assert.ok(/cd '[^']*\/wt\/lane-[ab]'/.test(lineAfter(p, /^Run this first/)), `${id} runs in a lane worktree`); + } + const subs = subjects(c, 'main..feature/w'); + assert.ok(subs.includes('join wave 1: merge J1') && subs.includes('join wave 1: merge J2'), subs.join('\n')); + assert.ok(existsSync(join(c.repo, 'J1.txt')) && existsSync(join(c.repo, 'J2.txt'))); + const events = readFileSync(`${c.m.repo.ledger_dir}/_run.jsonl`, 'utf8').trim().split('\n').map((l) => JSON.parse(l)); + assert.ok(events.some((e) => e.event === 'wave_merged' && e.wave === 1 && e.units.join() === 'J1,J2')); + assert.deepEqual(result.join_waves.map((w) => w.units), [['J1', 'J2']]); +}); + +test('a stop mid-wave relaunches through setup, recovery and the serial continuation', async () => { + const c = newCase(); + const first = await runAll({ ...c.m, setup_result: setup(c, c.m) }, agents(c, { stopAfterCommit: ['J2'] })); + assert.equal(first.status, 'stopped'); + assert.equal(first.tasks.J1.status, 'done'); + assert.equal(first.tasks.J2.status, 'blocked'); + assert.ok(subjects(c, 'main..feature/w').includes('join wave 1: merge J1'), 'the finished unit was merged'); + assert.ok(!subjects(c, 'main..feature/w').includes('join wave 1: merge J2'), 'the stopped unit was not'); + const j2Head = git(c.repo, 'rev-parse', 'pl-w1-u-J2'); + + // The relaunch, as SKILL.md's resume builds it: setup returns both lane + // worktrees from their unit branches, saving J2's unfinished change. + const again = setup(c, c.m); + assert.deepEqual(again.returned.map((r) => r.from).sort(), ['pl-w1-u-J1', 'pl-w1-u-J2'], JSON.stringify(again.returned)); + const j2Tree = fwd(again.returned.find((r) => r.from === 'pl-w1-u-J2').worktree); + const saved = again.preserved.find((p) => fwd(p.worktree) === j2Tree); + assert.ok(saved, 'the unfinished change was saved'); + assert.equal(git(c.repo, 'show', `${saved.ref}:J2-wip.txt`), 'unfinished'); + const status = ledgerStatus(c); + assert.equal(status.join_started, true); + assert.deepEqual(Object.keys(status.unit_branches).sort(), ['J1', 'J2']); + const manifestPath = `${c.root}/manifest.json`; + writeFileSync(manifestPath, JSON.stringify(c.m)); + const bf = py('ledger', ['backfill', c.m.repo.ledger_dir, manifestPath]); + assert.equal(bf.code, 0, bf.stdout + bf.stderr); + const back = JSON.parse(bf.stdout); + const m2 = { + ...c.m, setup_result: again, done: status.done, reviewed: status.reviewed, deferred: status.deferred, + backfill: back.backfill, join_started: status.join_started, unit_branches: status.unit_branches, + wave_merges: status.wave_merges, ...(back.join_tip ? { join_tip: back.join_tip } : {}), + start_points: status.start_points, + }; + const s = agents(c); + const second = await runAll(m2, s); + assert.equal(second.status, 'complete', JSON.stringify(second.reason || second.errors)); + const labels = s.calls.map((x) => x.label); + assert.ok(labels.includes('recovery merge')); + assert.ok(!labels.some((l) => /^J1 /.test(l)), 'J1 is merged and reviewed: not run again'); + const j2 = s.calls.find((x) => x.label === 'J2 implement').prompt; + assert.ok(j2.includes('--restart') && j2.includes('pl-w1-u-J2'), 'J2 restarts, told where its work was'); + assert.ok(!j2.includes('--unit-branch'), 'on the feature checkout'); + // Integration merged no unit work, and J2's earlier head never reached the feature branch. + assert.equal(sh(`git -C '${c.repo}' merge-base --is-ancestor '${j2Head}' feature/w`).code, 1); + assert.ok(existsSync(join(c.repo, 'J2.txt')), 'J2 was redone on the feature branch'); + const after2 = ledgerStatus(c); + assert.ok(after2.done.includes('J2') && after2.reviewed.includes('J2')); + assert.deepEqual(after2.inconsistent, []); +}); diff --git a/tests/join-waves-run.test.mjs b/tests/join-waves-run.test.mjs new file mode 100644 index 0000000..4ada85b --- /dev/null +++ b/tests/join-waves-run.test.mjs @@ -0,0 +1,501 @@ +// Join waves through the whole run (1.6.5 design 4.3-4.9): the wave merge +// agent, the wave resolver, the checks after the last multi-unit wave and +// their fix, integration keeping the lane worktrees, and the final lenses +// told the overlaps. Stub agents through loadScript. +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { loadHelpers, loadScript, H } from './harness.mjs'; + +const { waveResolvePrompt, waveChecksFixPrompt, waveChecksPrompt } = await loadHelpers([ + 'waveResolvePrompt', 'waveChecksFixPrompt', 'waveChecksPrompt', +]); + +function task(id, deps, extra = {}) { + return { + id, title: `Task ${id}`, files: [`src/${id}.js`], tier: 'standard', security: false, + ...(deps === undefined ? {} : { depends_on: deps.map((d) => ({ id: d, kind: 'code' })) }), ...extra, + }; +} + +function manifest(overrides = {}) { + return { + version: 1, + run_id: 'run-1', + plan: '/work/plan.md', + spec: '/work/spec.md', + commit_rules: 'plain ASCII', + repo: { + mode: 'git', root: '/work/repo', git_dir: null, base_ref: 'main', branch: 'pl/run-1', + worktree_root: '/work/wt', ledger_dir: '/work/ledger', + }, + commands: { setup: ['npm ci'], test: ['npm test'], lint: [], build: [] }, + prelude: [], + lanes: [ + { id: 'alpha', name: 'Lane alpha', tasks: [task('A1')] }, + { id: 'beta', name: 'Lane beta', tasks: [task('B1')] }, + ], + // Waves: [J1, J2], then [J3]. + join: [task('J1', []), task('J2', []), task('J3', ['J1', 'J2'])], + hooks: {}, + limits: { review_rounds: 5, max_parallel_lanes: 3 }, + dry_run: false, + done: [], + reviewed: [], + sp_dir: null, + skill_dir: '/skills/parallel-lanes', + setup_result: { + feature_head: H('F0'), discarded: [], worktrees: { alpha: '/work/wt/lane-alpha', beta: '/work/wt/lane-beta' }, + }, + ...overrides, + }; +} + +const done = (base, head) => ({ status: 'done', base, head, tests: 'npm test: pass', notes: '' }); +const blocked = (base, notes = 'stuck') => ({ status: 'blocked', base, head: base, tests: '', notes }); +const approve = () => ({ verdict: 'approve', findings: [], cannot_verify: [] }); +const verified = (sha) => ({ + checkout: '/work/repo', branch: 'pl/run-1', head: sha, results: [{ group: 'test', command: 'npm test', exit: 0 }], + ok: true, clean: true, tracked_before: [], tracked_after: [], untracked_before: [], untracked_after: [], +}); +const merged = (head, extra = {}) => ({ + status: 'done', head, merged: ['J1', 'J2'], conflict_unit: null, conflict_files: [], overlaps: [], checks: null, + notes: '', ...extra, +}); +const lens = (head) => ({ findings: [], cannot_verify: [], head }); + +// Every agent of a clean run whose join ends at tip. +function script(tip, extra = {}) { + return { + 'pre-flight': [{ conflicts: [], rulings: [], undeclared: [], code_deps: [] }], + 'A1 implement': [done('a', H('A1'))], 'A1 review': [approve()], + 'B1 implement': [done('b', H('B1'))], 'B1 review': [approve()], + integrate: [{ status: 'done', head: H('I1'), notes: 'merged' }], + 'J1 implement': [done(H('I1'), H('J1'))], 'J1 review': [approve()], + 'J2 implement': [done(H('I1'), H('J2'))], 'J2 review': [approve()], + 'wave 1 merge': [merged(H('M1'))], + 'J3 implement': [done(H('M1'), H('J3'))], 'J3 review': [approve()], + 'final review sp': [lens(tip)], 'final review security': [lens(tip)], 'final review correctness': [lens(tip)], + verify: [verified(tip)], + ...extra, + }; +} + +async function run(m, scripted) { + const calls = []; + const logs = []; + const agent = async (prompt, opts) => { + calls.push({ prompt, ...opts }); + const queue = scripted[opts.label]; + if (opts.label === 'availability probe' && !queue) return { ok: true }; + if (!queue || queue.length === 0) throw new Error(`unscripted agent call: ${opts.label}`); + return queue.shift(); + }; + const parallel = (thunks) => Promise.all(thunks.map((t) => t().catch(() => null))); + const result = await loadScript({ args: m, agent, parallel, phase: () => {}, log: (msg) => logs.push(msg) }); + return { result, calls, logs }; +} +const labels = (calls) => calls.map((c) => c.label); +const callOf = (calls, label) => calls.find((c) => c.label === label); +const at = (calls, label) => labels(calls).indexOf(label); + +test('each multi-unit wave is merged before the next starts', async () => { + const { result, calls } = await run(manifest(), script(H('J3'))); + assert.equal(result.status, 'complete', JSON.stringify(result.reason || result.errors)); + assert.ok(at(calls, 'J1 review') < at(calls, 'wave 1 merge') && at(calls, 'J2 review') < at(calls, 'wave 1 merge')); + assert.ok(at(calls, 'wave 1 merge') < at(calls, 'J3 implement')); + const mergePrompt = callOf(calls, 'wave 1 merge').prompt; + assert.ok(mergePrompt.includes("/scripts/merge-wave' '/work/repo' '/work/ledger' --wave '1'"), mergePrompt); + assert.ok(mergePrompt.includes(`--unit 'J1' '${H('I1')}' '${H('J1')}' --unit 'J2' '${H('I1')}' '${H('J2')}'`), mergePrompt); + assert.equal(callOf(calls, 'wave 1 merge').model, 'sonnet'); + assert.ok(callOf(calls, 'J3 implement').prompt.includes(`Task base: ${H('M1')}`)); + assert.deepEqual(result.join_waves.map((w) => [w.wave, w.units, w.head]), + [[1, ['J1', 'J2'], H('M1')], [2, ['J3'], H('J3')]]); + assert.equal(result.tasks.J3.status, 'done'); +}); + +test('a conflict goes to one resolver per conflict, then merges again', async () => { + const conflict = merged(H('I1'), { status: 'conflict', merged: ['J1'], conflict_unit: 'J2', conflict_files: ['src/x.js'] }); + const { result, calls } = await run(manifest(), script(H('J3'), { + 'wave 1 merge': [conflict, merged(H('M1'))], + 'wave 1 resolve J2': [{ status: 'done', head: H('R1'), notes: 'kept both' }], + })); + assert.equal(result.status, 'complete', JSON.stringify(result.reason)); + assert.deepEqual(labels(calls).filter((l) => l.startsWith('wave')), ['wave 1 merge', 'wave 1 resolve J2', 'wave 1 merge']); + const resolver = callOf(calls, 'wave 1 resolve J2'); + assert.ok(resolver.prompt.includes(H('J2')) && resolver.prompt.includes('src/x.js')); + assert.equal(resolver.model, 'opus'); + assert.deepEqual(result.join_waves[0].resolved_files, ['src/x.js']); +}); + +test('a failed resolver stops the run', async () => { + const conflict = merged(H('I1'), { status: 'conflict', merged: ['J1'], conflict_unit: 'J2', conflict_files: ['src/x.js'] }); + const { result, calls } = await run(manifest(), script(H('J3'), { + 'wave 1 merge': [conflict], + 'wave 1 resolve J2': [{ status: 'failed', head: H('I1'), notes: 'cannot reconcile' }], + })); + assert.equal(result.status, 'stopped'); + assert.match(result.reason, /join stopped/); + assert.ok(!labels(calls).includes('J3 implement')); + assert.match(result.stopped_lanes.find((s) => s.lane === 'join').reason, /resolver could not merge J2.*cannot reconcile/); +}); + +test('a supervised conflict stops with the sha to merge by hand', async () => { + const conflict = merged(H('I1'), { status: 'conflict', merged: ['J1'], conflict_unit: 'J2', conflict_files: ['src/x.js'] }); + const { result, calls } = await run(manifest({ autonomy: 'supervised' }), script(H('J3'), { 'wave 1 merge': [conflict] })); + assert.equal(result.status, 'stopped'); + assert.ok(!labels(calls).some((l) => l.includes('resolve'))); + const why = result.stopped_lanes.find((s) => s.lane === 'join').reason; + assert.ok(why.includes(H('J2')) && /by hand/.test(why) && why.includes('src/x.js'), why); + assert.ok(why.includes(`merge --no-ff -m 'join wave 1: merge J2' ${H('J2')}`), `the exact command: ${why}`); +}); + +const withChecks = (extra = {}) => manifest({ + task_checks: { J1: { test: ['npm test -- j1'] }, J2: { test: ['npm test -- j2'] } }, ...extra, +}); + +test('the last multi-unit wave\'s checks run in its merge', async () => { + const { result, calls } = await run(withChecks(), script(H('J3'), { + 'wave 1 merge': [merged(H('M1'), { checks: { ok: true, unfinished: false, notes: 'pass' } })], + })); + assert.equal(result.status, 'complete', JSON.stringify(result.reason)); + const p = callOf(calls, 'wave 1 merge').prompt; + assert.ok(p.includes("--phase 'join' --cmd 'test' 'npm test -- j1' --cmd 'test' 'npm test -- j2'"), p); + assert.ok(p.includes('"event":"wave_checked"'), p); + assert.deepEqual(result.join_waves[0].checks, { ok: true, unfinished: false, notes: 'pass', fixed: false }); +}); + +test('failing wave checks get a fix, a re-review and a recheck', async () => { + const { result, calls } = await run(withChecks(), script(H('J3'), { + 'wave 1 merge': [merged(H('M1'), { checks: { ok: false, unfinished: false, notes: 'npm test -- j2 failed' } })], + 'wave checks fix': [{ status: 'done', head: H('W1'), notes: 'fixed' }], + 'wave checks re-review': [{ findings: [] }], + 'wave checks': [{ status: 'done', head: H('W1'), ok: true, unfinished: false, notes: 'pass' }], + 'J3 implement': [done(H('W1'), H('J3'))], + })); + assert.equal(result.status, 'complete', JSON.stringify(result.reason)); + assert.ok(at(calls, 'wave checks fix') < at(calls, 'wave checks re-review')); + assert.ok(at(calls, 'wave checks re-review') < at(calls, 'wave checks')); + assert.ok(at(calls, 'wave checks') < at(calls, 'J3 implement')); + assert.ok(callOf(calls, 'wave checks fix').prompt.includes('npm test -- j2 failed')); + assert.equal(callOf(calls, 'wave checks fix').model, 'opus'); + assert.ok(callOf(calls, 'J3 implement').prompt.includes(`Task base: ${H('W1')}`)); + assert.equal(result.join_waves[0].head, H('W1')); + assert.equal(result.join_waves[0].checks.ok, true); + assert.equal(result.join_waves[0].checks.fixed, true); +}); + +test('still failing wave checks do not stop an autonomous run', async () => { + const { result } = await run(withChecks(), script(H('J3'), { + 'wave 1 merge': [merged(H('M1'), { checks: { ok: false, unfinished: false, notes: 'red' } })], + 'wave checks fix': [{ status: 'done', head: H('W1'), notes: 'tried' }], + 'wave checks re-review': [{ findings: [] }], + 'wave checks': [{ status: 'done', head: H('W1'), ok: false, unfinished: false, notes: 'still red' }], + 'J3 implement': [done(H('W1'), H('J3'))], + })); + assert.equal(result.status, 'complete', JSON.stringify(result.reason)); + assert.equal(result.join_waves[0].checks.ok, false); + assert.match(result.join_waves[0].checks.notes, /still red/); + assert.ok(result.final.cannot_verify.some((c) => typeof c === 'object' && /wave 1 checks/.test(c.requirement)), + JSON.stringify(result.final.cannot_verify)); +}); + +test('supervised failing wave checks stop', async () => { + const { result, calls } = await run(withChecks({ autonomy: 'supervised' }), script(H('J3'), { + 'wave 1 merge': [merged(H('M1'), { checks: { ok: false, unfinished: false, notes: 'red' } })], + })); + assert.equal(result.status, 'stopped'); + assert.ok(!labels(calls).includes('wave checks fix')); + assert.match(result.stopped_lanes.find((s) => s.lane === 'join').reason, /wave checks failed: red/); +}); + +test('unfinished wave checks get no fix', async () => { + const { result, calls } = await run(withChecks(), script(H('J3'), { + 'wave 1 merge': [merged(H('M1'), { checks: { ok: false, unfinished: true, notes: 'timed out' } })], + })); + assert.equal(result.status, 'complete', JSON.stringify(result.reason)); + assert.ok(!labels(calls).includes('wave checks fix')); + assert.equal(result.join_waves[0].checks.unfinished, true); +}); + +test('an unreviewed wave checks fix is reported', async () => { + const { result } = await run(withChecks(), script(H('J3'), { + 'wave 1 merge': [merged(H('M1'), { checks: { ok: false, unfinished: false, notes: 'red' } })], + 'wave checks fix': [{ status: 'done', head: H('W1'), notes: 'fixed' }], + 'wave checks re-review': [null], 'wave checks re-review retry': [null], + 'wave checks': [{ status: 'done', head: H('W1'), ok: true, unfinished: false, notes: 'pass' }], + 'J3 implement': [done(H('W1'), H('J3'))], + })); + assert.equal(result.status, 'complete', JSON.stringify(result.reason)); + assert.ok(result.final.cannot_verify.some((c) => typeof c === 'object' && /wave checks re-review returned no result/.test(c.requirement)), + JSON.stringify(result.final.cannot_verify)); +}); + +test('integration keeps the lane worktrees when waves run', async () => { + const { calls } = await run(manifest(), script(H('J3'))); + assert.ok(!callOf(calls, 'integrate').prompt.includes('worktree remove')); + const serial = manifest({ join: [task('J1'), task('J2')] }); + const plain = await run(serial, script(H('J2'), { + 'J2 implement': [done(H('J1'), H('J2'))], 'wave 1 merge': [], 'J3 implement': [], 'J3 review': [], + })); + assert.equal(plain.result.status, 'complete', JSON.stringify(plain.result.reason)); + assert.ok(callOf(plain.calls, 'integrate').prompt.includes('worktree remove'), 'a serial join cleans up as before'); + assert.equal(plain.result.join_waves, null); +}); + +test('lenses are told overlaps and resolved conflicts', async () => { + const conflict = merged(H('I1'), { status: 'conflict', merged: ['J1'], conflict_unit: 'J2', conflict_files: ['src/c.js'] }); + const { calls } = await run(manifest(), script(H('J3'), { + 'wave 1 merge': [conflict, merged(H('M1'), { overlaps: [{ path: 'src/shared.js', units: ['J1', 'J2'] }] })], + 'wave 1 resolve J2': [{ status: 'done', head: H('R1'), notes: 'kept both' }], + })); + for (const name of ['final review sp', 'final review security', 'final review correctness']) { + const p = callOf(calls, name).prompt; + assert.ok(p.includes('src/shared.js (J1, J2)'), `${name}: overlaps`); + assert.ok(p.includes('src/c.js'), `${name}: resolved conflicts`); + } +}); + +test('waveResolvePrompt aborts on failure', () => { + const p = waveResolvePrompt(manifest(), 1, { id: 'J2', head: H('J2') }, ['src/x.js']); + assert.ok(p.includes("git -C '/work/repo' merge --abort"), p); + assert.ok(p.includes("'join wave 1: merge J2'"), p); + assert.ok(p.includes(H('J2')) && p.includes('src/x.js')); +}); + +test('waveChecksFixPrompt uses the fix message', () => { + const p = waveChecksFixPrompt(manifest(), 'npm test failed'); + assert.ok(p.includes('fix: make the wave checks pass'), p); + assert.ok(p.includes('npm test failed')); + const c = waveChecksPrompt(manifest(), 2, [{ group: 'test', command: 'npm test -- j1' }]); + assert.ok(c.includes("--phase 'join' --cmd 'test' 'npm test -- j1'"), c); + assert.ok(c.includes('"wave":2'), c); +}); + +// ---- Task 10: a relaunch after waves ran: recovery merge, restarts, serial from the tip ---- + +const { integratePrompt, recoveryUnits, validateManifest } = await loadHelpers(['integratePrompt', 'recoveryUnits', 'validateManifest']); + +// A relaunch of manifest(): the lanes and J1 (wave 1, on the feature checkout) +// done and reviewed; wave 2's J2 and J3 ran on unit branches; join_started. +// over: the manifest fields to change (done, reviewed, backfill, ...). +function relaunch(over = {}) { + const base = manifest(); + const range = (id, b) => ({ base: b, head: H(id) }); + return { + ...base, + done: ['A1', 'B1', 'J1', 'J2'], + reviewed: ['A1', 'B1', 'J1', 'J2'], + backfill: { A1: range('A1', 'a'), B1: range('B1', 'b'), J1: range('J1', H('I1')), J2: range('J2', H('J1')) }, + start_points: { prelude: H('F0'), join: H('I1') }, + join_started: true, + unit_branches: { + J2: { branch: 'pl-run-1-u-J2', dir: '/work/wt/lane-alpha' }, + J3: { branch: 'pl-run-1-u-J3', dir: '/work/wt/lane-beta' }, + }, + ...over, + }; +} +function relaunchScript(tip, extra = {}) { + return { + 'pre-flight': [{ conflicts: [], rulings: [], undeclared: [], code_deps: [] }], + integrate: [{ status: 'done', head: H('I1'), notes: 'already merged' }], + 'recovery merge': [{ status: 'done', head: H('R1'), merged: ['J2'], conflict_unit: '', conflict_files: [], + overlaps: [], notes: '' }], + 'J3 implement': [done(H('R1'), H('J3'))], 'J3 review': [approve()], + 'final review sp': [lens(tip)], 'final review security': [lens(tip)], 'final review correctness': [lens(tip)], + verify: [verified(tip)], + ...extra, + }; +} + +test('relaunch merges reviewed wave units then continues from the tip', async () => { + const { result, calls } = await run(relaunch(), relaunchScript(H('J3'))); + assert.equal(result.status, 'complete', JSON.stringify(result.reason || result.errors)); + const rec = callOf(calls, 'recovery merge'); + assert.ok(rec.prompt.includes("--recovery --unit 'J2'"), rec.prompt); + assert.ok(rec.prompt.includes(`--unit 'J2' '${H('J1')}' '${H('J2')}'`), 'by recorded range'); + assert.ok(!rec.prompt.includes("--unit 'J3'")); + assert.ok(at(calls, 'integrate') < at(calls, 'recovery merge') && at(calls, 'recovery merge') < at(calls, 'J3 implement')); + const j3 = callOf(calls, 'J3 implement').prompt; + assert.ok(j3.includes(`Task base: ${H('R1')}`), 'the restart starts at the recovery head'); + assert.ok(j3.includes("cd '/work/repo'"), 'on the feature checkout'); + assert.ok(j3.includes('--restart'), 'its started event is a restart'); + assert.ok(!j3.includes('--unit-branch'), 'no unit branch on the continuation'); + assert.ok(j3.includes('pl-run-1-u-J3') && j3.includes('/work/wt/lane-beta'), 'it is told where its earlier work is'); + assert.ok(j3.includes('refs/parallel-lanes/run-1/abandoned/lane-beta-'), 'and the abandoned refs'); + assert.match(j3, /partial/); + const review = callOf(calls, 'J3 review').prompt; + assert.ok(review.includes(`${H('R1')}..${H('J3')}`), 'its review covers only its own range'); + assert.equal(result.tasks.J3.status, 'done'); + assert.equal(result.final.head, H('J3')); + assert.ok(!labels(calls).some((l) => /^J[12] /.test(l)), 'merged and reviewed units are not run again'); +}); + +test('a done but unreviewed wave unit restarts and is not recovered', async () => { + const m = relaunch({ done: ['A1', 'B1', 'J1', 'J2', 'J3'], reviewed: ['A1', 'B1', 'J1', 'J2'], + backfill: { ...relaunch().backfill, J3: { base: H('J1'), head: H('J3old') } } }); + const { result, calls } = await run(m, relaunchScript(H('J3'))); + assert.equal(result.status, 'complete', JSON.stringify(result.reason || result.errors)); + assert.ok(!callOf(calls, 'recovery merge').prompt.includes("--unit 'J3'")); + const j3 = callOf(calls, 'J3 implement').prompt; + assert.ok(j3.includes('--restart')); + assert.match(j3, /complete/); + assert.ok(!labels(calls).includes('J3 review') || at(calls, 'J3 implement') < at(calls, 'J3 review'), + 'implemented again, not only reviewed'); +}); + +test('the recovery merge runs with nothing to merge', async () => { + const m = relaunch({ done: ['A1', 'B1', 'J1'], reviewed: ['A1', 'B1', 'J1'], + backfill: { A1: relaunch().backfill.A1, B1: relaunch().backfill.B1, J1: relaunch().backfill.J1 } }); + const { result, calls } = await run(m, relaunchScript(H('J3'), { + 'recovery merge': [{ status: 'done', head: H('J1'), merged: [], conflict_unit: '', conflict_files: [], overlaps: [], notes: '' }], + 'J2 implement': [done(H('J1'), H('J2'))], 'J2 review': [approve()], + 'J3 implement': [done(H('J2'), H('J3'))], + })); + assert.equal(result.status, 'complete', JSON.stringify(result.reason || result.errors)); + assert.ok(callOf(calls, 'recovery merge').prompt.includes("--recovery")); + assert.ok(callOf(calls, 'J2 implement').prompt.includes(`Task base: ${H('J1')}`)); + assert.ok(callOf(calls, 'J3 implement').prompt.includes(`Task base: ${H('J2')}`), 'serial from the tip'); +}); + +test('a supervised recovery conflict stops', async () => { + const { result, calls } = await run(relaunch({ autonomy: 'supervised' }), relaunchScript(H('J3'), { + 'recovery merge': [{ status: 'conflict', head: H('I1'), merged: [], conflict_unit: 'J2', conflict_files: ['src/x.js'], + overlaps: [], notes: '' }], + })); + assert.equal(result.status, 'stopped'); + assert.ok(!labels(calls).includes('J3 implement')); + const why = result.stopped_lanes.find((s) => s.lane === 'join').reason; + assert.ok(why.includes(H('J2')) && /by hand/.test(why), why); +}); + +test('an autonomous recovery conflict goes to a resolver', async () => { + const { result, calls } = await run(relaunch(), relaunchScript(H('J3'), { + 'recovery merge': [ + { status: 'conflict', head: H('I1'), merged: [], conflict_unit: 'J2', conflict_files: ['src/x.js'], overlaps: [], notes: '' }, + { status: 'done', head: H('R1'), merged: [], conflict_unit: '', conflict_files: [], overlaps: [], notes: '' }, + ], + 'recovery resolve J2': [{ status: 'done', head: H('R1'), notes: 'kept both' }], + })); + assert.equal(result.status, 'complete', JSON.stringify(result.reason)); + assert.ok(callOf(calls, 'recovery resolve J2').prompt.includes("'join recovery: merge J2'")); +}); + +test('a relaunch of a serial join keeps the serial path', async () => { + const m = relaunch({ unit_branches: {}, done: ['A1', 'B1', 'J1'], reviewed: ['A1', 'B1', 'J1'], + backfill: { A1: relaunch().backfill.A1, B1: relaunch().backfill.B1, J1: relaunch().backfill.J1 } }); + const { result, calls } = await run(m, relaunchScript(H('J3'), { + 'J2 implement': [done(H('J1'), H('J2'))], 'J2 review': [approve()], + 'J3 implement': [done(H('J2'), H('J3'))], + })); + assert.equal(result.status, 'complete', JSON.stringify(result.reason || result.errors)); + assert.ok(!labels(calls).includes('recovery merge')); + assert.ok(!callOf(calls, 'J2 implement').prompt.includes('--restart')); +}); + +test('recoveryUnits takes reviewed or deferred units with a unit branch', () => { + const m = relaunch({ done: ['A1', 'B1', 'J1', 'J2', 'J3'], reviewed: ['A1', 'B1', 'J1', 'J2'], deferred: ['J3'], + backfill: { ...relaunch().backfill, J3: { base: H('J1'), head: H('J3') } } }); + assert.deepEqual(recoveryUnits(m), [ + { id: 'J2', base: H('J1'), head: H('J2') }, + { id: 'J3', base: H('J1'), head: H('J3') }, + ]); + assert.deepEqual(recoveryUnits(relaunch({ unit_branches: {} })), []); +}); + +test('the audit allowance uses join_tip', () => { + const m = relaunch({ done: ['A1', 'B1', 'J1', 'J2', 'J3'], reviewed: ['A1', 'B1', 'J1', 'J2', 'J3'], + backfill: { ...relaunch().backfill, J3: { base: H('R1'), head: H('J3') } }, join_tip: H('W9') }); + const p = integratePrompt(m, H('pre'), {}); + assert.ok(p.includes(`may follow the last join commit ${H('W9')}`), p); + assert.ok(p.includes("'join wave : merge '") && p.includes("'join recovery: merge '"), p); + assert.ok(p.includes('fix: make the wave checks pass'), p); +}); + +// ---- Task 11: pre-flight sees the waves ---- + +test('pre-flight is shown the waves', async () => { + const { calls } = await run(manifest(), script(H('J3'))); + const p = callOf(calls, 'pre-flight').prompt; + assert.ok(p.includes('wave 1: J1, J2') && p.includes('wave 2: J3'), p); + assert.ok(p.includes('the run moves the task to a later wave'), p); +}); + +test('an earlier-producer finding moves the unit to a later wave and is reported', async () => { + const m = manifest({ join: [task('J1', []), task('J2', []), task('J3', [])] }); + const { result, calls } = await run(m, script(H('J3'), { + 'pre-flight': [{ conflicts: [], rulings: [], undeclared: [{ task: 'J3', producer: 'J2', what: 'a test helper' }], + code_deps: [] }], + 'J3 implement': [done(H('M1'), H('J3'))], + })); + assert.equal(result.status, 'complete', JSON.stringify(result.reason || result.errors)); + assert.deepEqual(result.preflight.wave_moves, [{ unit: 'J3', from: 1, to: 2, producer: 'J2' }]); + assert.ok(callOf(calls, 'wave 1 merge').prompt.includes("--unit 'J2'") && !callOf(calls, 'wave 1 merge').prompt.includes("--unit 'J3'")); + assert.ok(at(calls, 'wave 1 merge') < at(calls, 'J3 implement')); + assert.ok(callOf(calls, 'J3 implement').prompt.includes("cd '/work/repo'"), 'J3 runs alone, on the feature checkout'); +}); + +test('a later-producer code dependency is still a schedule conflict', async () => { + const { result } = await run(manifest(), script(H('J3'), { + 'pre-flight': [{ conflicts: [], rulings: [], undeclared: [], code_deps: [{ task: 'J1', producer: 'J2', what: 'calls it' }] }], + })); + assert.equal(result.status, 'preflight_conflicts'); + assert.match(result.preflight.schedule[0].fix, /move J1 to join after J2/); +}); + +test('a serial join\'s pre-flight prompt is unchanged', async () => { + const m = manifest({ join: [task('J1'), task('J2')] }); + const { calls } = await run(m, script(H('J2'), { + 'J2 implement': [done(H('J1'), H('J2'))], 'wave 1 merge': [], 'J3 implement': [], 'J3 review': [], + })); + assert.ok(!callOf(calls, 'pre-flight').prompt.includes('Join waves')); +}); + +// ---- Final review fixes ---- + +test('a stale merged wave unit is reviewed over its range, not redone', async () => { + const m = relaunch({ reviewed: ['A1', 'B1', 'J1'] }); + const { result, calls } = await run(m, relaunchScript(H('J3'), { + 'recovery merge': [{ status: 'done', head: H('R1'), merged: [], in_history: ['J2'], conflict_unit: '', + conflict_files: [], overlaps: [], notes: '' }], + 'J2 review': [approve()], + })); + assert.equal(result.status, 'complete', JSON.stringify(result.reason || result.errors)); + const rec = callOf(calls, 'recovery merge').prompt; + assert.ok(rec.includes(`--known 'J2' '${H('J2')}'`), rec); + assert.ok(!labels(calls).includes('J2 implement'), 'merged work is not redone'); + assert.ok(callOf(calls, 'J2 review').prompt.includes(`${H('J1')}..${H('J2')}`), 'reviewed over its recorded range'); + assert.ok(callOf(calls, 'J3 implement').prompt.includes(`Task base: ${H('R1')}`), 'the tip does not move back'); + assert.equal(result.final.head, H('J3')); +}); + +test('a stale feature-checkout unit does not move the join head back', async () => { + const m = relaunch({ reviewed: ['A1', 'B1', 'J2'] }); + const { result, calls } = await run(m, relaunchScript(H('J3'), { + 'recovery merge': [{ status: 'done', head: H('R1'), merged: [], in_history: ['J2'], conflict_unit: '', + conflict_files: [], overlaps: [], notes: '' }], + 'J1 review': [approve()], + })); + assert.equal(result.status, 'complete', JSON.stringify(result.reason || result.errors)); + assert.ok(callOf(calls, 'J1 review').prompt.includes(`${H('I1')}..${H('J1')}`)); + assert.ok(callOf(calls, 'J3 implement').prompt.includes(`Task base: ${H('R1')}`), 'J3 starts at the tip, not at J1'); + assert.equal(result.final.head, H('J3')); +}); + +test('a serial unit stopped part-way restarts its range at its started base', async () => { + const m = relaunch({ unit_branches: { J2: { branch: 'pl-run-1-u-J2', dir: '/work/wt/lane-alpha' } }, + start_bases: { J3: H('S3') } }); + const { result, calls } = await run(m, relaunchScript(H('J3'), { + 'J3 implement': [done(H('S3'), H('J3'))], + })); + assert.equal(result.status, 'complete', JSON.stringify(result.reason || result.errors)); + const p = callOf(calls, 'J3 implement').prompt; + assert.ok(p.includes(`Task base: ${H('S3')}`), 'its earlier partial commits stay in its range'); + assert.ok(!p.includes('--restart')); +}); + +test('start_bases must name tasks of the run', () => { + assert.ok(validateManifest(relaunch({ start_bases: { J3: H('S3') } })).length === 0); + assert.ok(validateManifest(relaunch({ start_bases: { NOPE: H('S3') } })).some((e) => e.includes('start_bases'))); + assert.ok(validateManifest(relaunch({ start_bases: [] })).some((e) => e.includes('start_bases'))); +}); diff --git a/tests/join-waves.test.mjs b/tests/join-waves.test.mjs new file mode 100644 index 0000000..1ca38eb --- /dev/null +++ b/tests/join-waves.test.mjs @@ -0,0 +1,254 @@ +// Join waves in the engine (1.6.5 design 4.2-4.5): runJoinWaves runs each +// multi-unit wave's units side by side on unit branches in lane worktrees and +// hands them to a merge; a single-unit wave runs on the feature checkout. +// Stubbed agents and a stubbed merge. +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { loadHelpers, H } from './harness.mjs'; + +const { + runJoinWaves, unitWhere, wavePlan, startCommand, +} = await loadHelpers(['runJoinWaves', 'unitWhere', 'wavePlan', 'startCommand']); + +function task(id, deps, extra = {}) { + return { + id, title: `Task ${id}`, files: [`src/${id}.js`], tier: 'standard', security: false, + ...(deps === undefined ? {} : { depends_on: deps.map((d) => ({ id: d, kind: 'code' })) }), ...extra, + }; +} + +function manifest(overrides = {}) { + return { + version: 1, + run_id: 'run-1', + plan: '/work/plan.md', + spec: '/work/spec.md', + commit_rules: 'plain ASCII', + repo: { + mode: 'git', root: '/work/repo', git_dir: null, base_ref: 'main', branch: 'pl/run-1', + worktree_root: '/work/wt', ledger_dir: '/work/ledger', + }, + commands: { setup: ['npm ci'], test: ['npm test'], lint: [], build: [] }, + prelude: [], + lanes: [ + { id: 'alpha', name: 'Lane alpha', tasks: [task('A1')] }, + { id: 'beta', name: 'Lane beta', tasks: [task('B1')] }, + { id: 'gamma', name: 'Lane gamma', tasks: [task('C1')] }, + ], + join: [task('J1', []), task('J2', ['J1']), task('J3', ['J1'])], + hooks: {}, + limits: { review_rounds: 5, max_parallel_lanes: 3 }, + dry_run: false, + done: [], + reviewed: [], + sp_dir: null, + skill_dir: '/skills/parallel-lanes', + autonomy: 'supervised', + ...overrides, + }; +} +const DIRS = ['/work/wt/lane-alpha', '/work/wt/lane-beta', '/work/wt/lane-gamma']; + +const done = (base, head) => ({ status: 'done', base, head, tests: 'npm test: pass', notes: '' }); +const blocked = (base, notes = 'stuck') => ({ status: 'blocked', base, head: base, tests: '', notes }); +const approve = () => ({ verdict: 'approve', findings: [], cannot_verify: [] }); + +// Stub agent: per-label result queues, or functions of the call. Records +// every call with its start and end order. +function stub(script) { + const calls = []; + const logs = []; + let clock = 0; + const agent = async (prompt, opts) => { + const call = { prompt, ...opts, started: clock += 1 }; + calls.push(call); + await new Promise((resolve) => setTimeout(resolve, 5)); + const queue = script[opts.label]; + if (!queue || queue.length === 0) throw new Error(`unscripted agent call: ${opts.label}`); + const next = queue.shift(); + const out = typeof next === 'function' ? await next(call) : next; + call.ended = clock += 1; + return out; + }; + return { io: { agent, log: (msg) => logs.push(msg) }, calls, logs }; +} +// A merge stub: records its calls and returns a new head per wave. +function merger(result = null) { + const calls = []; + const merge = async (wave, unitResults, base) => { + calls.push({ wave, units: unitResults.map((u) => u.unit.id), statuses: unitResults.map((u) => u.result.status), base }); + return result ? result(wave) : { head: H(`merge${wave}`) }; + }; + return { merge, calls }; +} +const plan = (m) => wavePlan(m).waves; +const callOf = (s, label) => s.calls.find((c) => c.label === label); + +test('units of a wave run concurrently in lane worktrees on unit branches', async () => { + const m = manifest(); + const s = stub({ + 'J1 implement': [done(H('t0'), H('j1'))], 'J1 review': [approve()], + 'J2 implement': [done(H('j1'), H('j2'))], 'J2 review': [approve()], + 'J3 implement': [done(H('j1'), H('j3'))], 'J3 review': [approve()], + }); + const mg = merger(); + const r = await runJoinWaves(m, plan(m), H('t0'), DIRS, s.io, mg.merge); + assert.equal(r.stopped, null); + const j2 = callOf(s, 'J2 implement'); + const j3 = callOf(s, 'J3 implement'); + assert.ok(j2.started < j3.ended && j3.started < j2.ended, 'J2 and J3 overlap'); + assert.ok(j2.prompt.includes("--unit-branch 'pl-run-1-u-J2'"), j2.prompt); + assert.ok(j2.prompt.includes(`cd '${DIRS[0]}'`), 'J2 takes the first lane worktree'); + assert.ok(j3.prompt.includes("--unit-branch 'pl-run-1-u-J3'") && j3.prompt.includes(`cd '${DIRS[1]}'`)); + assert.ok(j2.prompt.includes(`--unit-branch 'pl-run-1-u-J2' '${H('j1')}' --run-id 'run-1'`), 'from the wave base'); + assert.deepEqual(mg.calls, [{ wave: 2, units: ['J2', 'J3'], statuses: ['done', 'done'], base: H('j1') }]); + assert.equal(r.head, H('merge2')); + assert.deepEqual(r.results.map((x) => [x.task, x.status]), [['J1', 'done'], ['J2', 'done'], ['J3', 'done']]); + assert.deepEqual(r.waves.map((w) => [w.wave, w.units, w.head]), + [[1, ['J1'], H('j1')], [2, ['J2', 'J3'], H('merge2')]]); +}); + +test('unit events go to join', async () => { + const m = manifest(); + const s = stub({ + 'J1 implement': [done(H('t0'), H('j1'))], 'J1 review': [approve()], + 'J2 implement': [done(H('j1'), H('j2'))], 'J2 review': [approve()], + 'J3 implement': [done(H('j1'), H('j3'))], 'J3 review': [approve()], + }); + await runJoinWaves(m, plan(m), H('t0'), DIRS, s.io, merger().merge); + const p = callOf(s, 'J2 implement').prompt; + assert.ok(p.includes(`--record-start '/work/ledger' 'join' '${H('j1')}'`), p); + assert.ok(p.includes("--rulings '/work/ledger' '-'"), 'a join unit sees every lane\'s rulings'); + assert.ok(!p.includes('--sync'), 'no lane sync'); + assert.ok(p.includes("--phase 'join'"), 'its checks run in the join phase'); + for (const c of s.calls) assert.equal(c.phase, 'Join'); +}); + +test('a single-unit wave runs on the feature checkout', async () => { + const m = manifest(); + const s = stub({ + 'J1 implement': [done(H('t0'), H('j1'))], 'J1 review': [approve()], + 'J2 implement': [done(H('j1'), H('j2'))], 'J2 review': [approve()], + 'J3 implement': [done(H('j1'), H('j3'))], 'J3 review': [approve()], + }); + await runJoinWaves(m, plan(m), H('t0'), DIRS, s.io, merger().merge); + const p = callOf(s, 'J1 implement').prompt; + assert.ok(p.includes("cd '/work/repo'"), p); + assert.ok(!p.includes('--unit-branch'), p); +}); + +test('the next wave starts from the merge head', async () => { + const m = manifest({ join: [task('J1', []), task('J2', []), task('J3', ['J1', 'J2'])] }); + const s = stub({ + 'J1 implement': [done(H('t0'), H('j1'))], 'J1 review': [approve()], + 'J2 implement': [done(H('t0'), H('j2'))], 'J2 review': [approve()], + 'J3 implement': [done(H('merge1'), H('j3'))], 'J3 review': [approve()], + }); + const mg = merger(); + const r = await runJoinWaves(m, plan(m), H('t0'), DIRS, s.io, mg.merge); + assert.ok(callOf(s, 'J3 implement').prompt.includes(`Task base: ${H('merge1')}`)); + assert.equal(r.head, H('j3'), 'the last single-unit wave moves the tip'); + assert.equal(mg.calls.length, 1); +}); + +test('a wave wider than the pool reuses a finished unit\'s worktree', async () => { + const m = manifest({ limits: { review_rounds: 5, max_parallel_lanes: 2 }, + join: [task('J1', []), task('J2', []), task('J3', [])] }); + const s = stub({ + 'J1 implement': [done(H('t0'), H('j1'))], 'J1 review': [approve()], + 'J2 implement': [done(H('t0'), H('j2'))], 'J2 review': [approve()], + 'J3 implement': [done(H('t0'), H('j3'))], 'J3 review': [approve()], + }); + const mg = merger(); + const r = await runJoinWaves(m, plan(m), H('t0'), DIRS, s.io, mg.merge); + assert.equal(r.stopped, null); + const dirOf = (id) => DIRS.find((d) => callOf(s, `${id} implement`).prompt.includes(`cd '${d}'`)); + assert.equal(dirOf('J1'), DIRS[0]); + assert.equal(dirOf('J2'), DIRS[1]); + assert.ok([DIRS[0], DIRS[1]].includes(dirOf('J3')), 'only the two usable worktrees'); + const j3 = callOf(s, 'J3 implement'); + const before = callOf(s, `${dirOf('J3') === DIRS[0] ? 'J1' : 'J2'} review`); + assert.ok(before.ended < j3.started, 'J3 starts after the unit before it in that worktree finished'); + assert.deepEqual(mg.calls[0].units, ['J1', 'J2', 'J3']); +}); + +test('no unit starts after a sibling stops', async () => { + const m = manifest({ limits: { review_rounds: 5, max_parallel_lanes: 2 }, + join: [task('J1', []), task('J2', []), task('J3', [])] }); + const s = stub({ + 'J1 implement': [blocked(H('t0'), 'cannot build')], + 'J2 implement': [done(H('t0'), H('j2'))], 'J2 review': [approve()], + }); + const mg = merger(); + const r = await runJoinWaves(m, plan(m), H('t0'), DIRS, s.io, mg.merge); + assert.ok(!s.calls.some((c) => c.label.startsWith('J3')), 'J3 never starts'); + assert.deepEqual(mg.calls, [{ wave: 1, units: ['J1', 'J2'], statuses: ['blocked', 'done'], base: H('t0') }]); + assert.match(r.stopped, /cannot build/); + assert.equal(r.head, H('merge1'), 'the finished unit is merged before the join stops'); + assert.deepEqual(r.results.map((x) => [x.task, x.status]), [['J1', 'blocked'], ['J2', 'done']]); +}); + +test('a merge that stops ends the join at the wave base', async () => { + const m = manifest({ join: [task('J1', []), task('J2', [])] }); + const s = stub({ + 'J1 implement': [done(H('t0'), H('j1'))], 'J1 review': [approve()], + 'J2 implement': [done(H('t0'), H('j2'))], 'J2 review': [approve()], + }); + const r = await runJoinWaves(m, plan(m), H('t0'), DIRS, s.io, merger(() => ({ stopped: 'conflict in a.js' })).merge); + assert.equal(r.stopped, 'conflict in a.js'); + assert.equal(r.head, H('t0')); +}); + +test('a batch runs as one wave unit', async () => { + const light = (id, deps) => task(id, deps, { tier: 'light', batch: 'k' }); + const m = manifest({ join: [task('J1', []), light('J2', []), light('J3', [])] }); + assert.deepEqual(plan(m), [['J1', 'J2-J3']]); + const s = stub({ + 'J1 implement': [done(H('t0'), H('j1'))], 'J1 review': [approve()], + 'J2-J3 implement': [done(H('t0'), H('j23'))], 'J2-J3 review': [approve()], + }); + const r = await runJoinWaves(m, plan(m), H('t0'), DIRS, s.io, merger().merge); + assert.ok(callOf(s, 'J2-J3 implement').prompt.includes("--unit-branch 'pl-run-1-u-J2-J3'")); + assert.deepEqual(r.results.map((x) => [x.task, x.batch || null, x.head]), + [['J1', null, H('j1')], ['J2', 'J2-J3', H('j23')], ['J3', 'J2-J3', H('j23')]]); +}); + +test('an unblock note goes to the dependent unit, else the next wave', async () => { + const m = manifest({ autonomy: 'autonomous', + join: [task('J1', []), task('J2', []), task('J3', ['J2']), task('J4', ['J1'])] }); + // waves: [J1, J2], [J3, J4] + assert.deepEqual(plan(m), [['J1', 'J2'], ['J3', 'J4']]); + const unblock = { outcome: 'unblock', text: 'use the stub', head: H('j2') }; + const s = stub({ + 'J1 implement': [done(H('t0'), H('j1'))], 'J1 review': [approve()], + 'J2 implement': [{ status: 'question', base: H('t0'), head: H('t0'), tests: '', notes: '', question: 'which?' }], + 'J2 adjudicate': [unblock], + 'J3 implement': [done(H('merge1'), H('j3'))], 'J3 review': [approve()], + 'J4 implement': [done(H('merge1'), H('j4'))], 'J4 review': [approve()], + }); + const r = await runJoinWaves(m, plan(m), H('t0'), DIRS, s.io, merger().merge); + assert.equal(r.results.find((x) => x.task === 'J2').status, 'deferred'); + assert.ok(callOf(s, 'J3 implement').prompt.includes('from J2, unblocked by the adjudicator: use the stub')); + assert.ok(!callOf(s, 'J4 implement').prompt.includes('unblocked by the adjudicator')); +}); + +test('setup commands go to the unit\'s start command', () => { + const m = manifest({ commands: { setup: ['npm ci', 'make seed'], test: ['npm test'], lint: [], build: [] } }); + const where = unitWhere(m, m.join[1], DIRS[0], H('w')); + assert.deepEqual(where, { dir: DIRS[0], branch: 'pl-run-1-u-J2', lane: 'join', unit_branch: 'pl-run-1-u-J2', from: H('w') }); + const cmd = startCommand(m, m.join[1], where, { start: H('w') }); + assert.ok(cmd.includes(`--unit-branch 'pl-run-1-u-J2' '${H('w')}' --run-id 'run-1' --setup 'npm ci' --setup 'make seed'`), cmd); + const review = startCommand(m, m.join[1], where, {}); + assert.ok(!review.includes('--unit-branch'), 'only the implement start switches branches'); +}); + +test('the join head is the last tip', async () => { + const m = manifest({ join: [task('J1', []), task('J2', [])] }); + const s = stub({ + 'J1 implement': [done(H('t0'), H('j1'))], 'J1 review': [approve()], + 'J2 implement': [done(H('t0'), H('j2'))], 'J2 review': [approve()], + }); + const r = await runJoinWaves(m, plan(m), H('t0'), DIRS, s.io, merger().merge); + assert.equal(r.head, H('merge1')); + assert.notEqual(r.head, H('j2')); +}); diff --git a/tests/ledger-waves.test.mjs b/tests/ledger-waves.test.mjs new file mode 100644 index 0000000..53bdb34 --- /dev/null +++ b/tests/ledger-waves.test.mjs @@ -0,0 +1,399 @@ +// 1.6.5 join waves in the ledger: concurrent appends take a lock, a reader +// skips a half-written last line, the wave events and the started fields +// (design 4.5); restarts, join_started, unit branches, wave merges, the +// join's backfill and join_tip. +import { test, after } from 'node:test'; +import assert from 'node:assert/strict'; +import { spawn, spawnSync } from 'node:child_process'; +import { appendFileSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { SKILL_DIR } from './harness.mjs'; +import { tempDir, mergeEnv } from './platform.mjs'; + +const SCRIPTS = join(SKILL_DIR, 'scripts'); +const TMP = tempDir('pl-ledger-waves-'); +after(() => rmSync(TMP, { recursive: true, force: true })); + +let counter = 0; +function freshDir(name = 'ledger') { + counter += 1; + const dir = join(TMP, `case ${counter}`, name); + mkdirSync(dir, { recursive: true }); + return dir; +} + +function run(script, args, env = {}, cwd = undefined) { + const res = spawnSync('python3', ['-B', join(SCRIPTS, script), ...args], { + encoding: 'utf8', env: mergeEnv(process.env, env), cwd, + }); + return { code: res.status, stdout: res.stdout, stderr: res.stderr }; +} +const ledger = (...args) => run('ledger', args); + +function appendOk(dir, lane, entry) { + const res = ledger('append', dir, lane, JSON.stringify(entry)); + assert.equal(res.code, 0, res.stderr); +} + +function statusOf(dir, ...extra) { + const res = ledger('status', dir, ...extra); + assert.equal(res.code, 0, res.stderr); + return JSON.parse(res.stdout); +} + +const lines = (dir, lane) => readFileSync(join(dir, `${lane}.jsonl`), 'ascii').split('\n').filter((l) => l !== ''); + +// A Python child that loads scripts/ledger as a module (spec_from_loader and +// exec_module: load_module is gone in Python 3.15). +const LOADER = [ + 'import sys, os', + 'sys.dont_write_bytecode = True', + `sys.path.insert(0, ${JSON.stringify(SCRIPTS)})`, + 'import importlib.machinery, importlib.util', + `ldr = importlib.machinery.SourceFileLoader("ledger_mod", ${JSON.stringify(join(SCRIPTS, 'ledger'))})`, + 'L = importlib.util.module_from_spec(importlib.util.spec_from_loader("ledger_mod", ldr))', + 'ldr.exec_module(L)', +]; + +function child(code) { + return spawn('python3', ['-B', '-c', [...LOADER, ...code].join('\n')], { env: mergeEnv(process.env) }); +} +const exited = (p) => new Promise((resolve) => p.on('exit', (code) => resolve(code))); + +// Holds the lane's append lock until killed; resolves once it is held. +function holdLock(dir, lane) { + const p = spawn('python3', ['-B', '-c', [ + 'import sys, os, time', + 'sys.dont_write_bytecode = True', + `sys.path.insert(0, ${JSON.stringify(SCRIPTS)})`, + 'import _oslock', + `fd = os.open(${JSON.stringify(join(dir, `${lane}.jsonl`))}, os.O_RDWR | os.O_CREAT, 0o600)`, + 'assert _oslock.try_lock(fd)', + 'print("held", flush=True)', + 'time.sleep(60)', + ].join('\n')], { env: mergeEnv(process.env) }); + return new Promise((resolve, reject) => { + p.stdout.on('data', (d) => { if (String(d).includes('held')) resolve(p); }); + p.on('exit', (code) => reject(new Error(`lock holder exited ${code}`))); + }); +} + +function appendAsync(dir, lane, entry, env = {}) { + const start = Date.now(); + const p = spawn('python3', ['-B', join(SCRIPTS, 'ledger'), 'append', dir, lane, JSON.stringify(entry)], + { env: mergeEnv(process.env, env) }); + let stderr = ''; + p.stderr.on('data', (d) => { stderr += d; }); + return new Promise((resolve) => p.on('exit', (code) => resolve({ code, stderr, ms: Date.now() - start }))); +} + +// ---- Task 2: writes ---- + +test('concurrent appenders all land', async () => { + const dir = freshDir(); + const kids = []; + for (let k = 0; k < 8; k += 1) { + kids.push(child([ + 'for i in range(50):', + ` L.append_entries(${JSON.stringify(dir)}, "alpha", [{"task": "T${k}", "event": "ruling", "text": "r%d" % i}])`, + ])); + } + const codes = await Promise.all(kids.map(exited)); + assert.deepEqual(codes, new Array(8).fill(0)); + const got = lines(dir, 'alpha'); + assert.equal(got.length, 400); + for (const l of got) JSON.parse(l); +}); + +test('a held lock makes an append wait', async () => { + const dir = freshDir(); + const holder = await holdLock(dir, 'alpha'); + const pending = appendAsync(dir, 'alpha', { task: 'T1', event: 'ruling', text: 'r' }); + setTimeout(() => holder.kill(), 1000); + const r = await pending; + assert.equal(r.code, 0, r.stderr); + assert.ok(r.ms >= 900, `appended after ${r.ms} ms`); + assert.equal(lines(dir, 'alpha').length, 1); +}); + +test('a lock held past the wait refuses and writes nothing', async () => { + const dir = freshDir(); + appendOk(dir, 'alpha', { task: 'T1', event: 'ruling', text: 'first' }); + const holder = await holdLock(dir, 'alpha'); + try { + const r = await appendAsync(dir, 'alpha', { task: 'T1', event: 'ruling', text: 'second' }, + { LEDGER_LOCK_WAIT_S: '1' }); + assert.equal(r.code, 3, r.stderr); + assert.match(r.stderr, /alpha\.jsonl stayed locked for 1 s/); + } finally { + holder.kill(); + } + assert.equal(lines(dir, 'alpha').length, 1); +}); + +test('a partial last line is ignored', () => { + const dir = freshDir(); + appendOk(dir, 'alpha', { task: 'T1', event: 'started', base: 'abc' }); + appendFileSync(join(dir, 'alpha.jsonl'), '{"task":"T1","ev'); + const s = statusOf(dir); + assert.deepEqual(s.done, []); + assert.equal(ledger('append', dir, 'beta', JSON.stringify({ task: 'T2', event: 'ruling', text: 'r' })).code, 0); +}); + +test('a bad middle line still fails', () => { + const dir = freshDir(); + appendOk(dir, 'alpha', { task: 'T1', event: 'started', base: 'abc' }); + appendFileSync(join(dir, 'alpha.jsonl'), '{"task":"T1","ev\n'); + appendOk(dir, 'alpha', { task: 'T1', event: 'started', base: 'abc' }); + assert.equal(ledger('status', dir).code, 3); + const dir2 = freshDir(); + writeFileSync(join(dir2, 'alpha.jsonl'), '{"task":"T1","event":"nope"}'); + assert.equal(ledger('status', dir2).code, 3, 'a complete-looking but invalid last line without a newline'); +}); + +test('wave_merged and wave_checked are accepted on _run only', () => { + const dir = freshDir(); + appendOk(dir, '_run', { task: '_run', event: 'wave_merged', wave: 2, units: ['DJ2', 'DJ4'], head: 'abc', + overlaps: [{ path: 'a.cs', units: ['DJ2', 'DJ4'] }], conflict_files: [] }); + appendOk(dir, '_run', { task: '_run', event: 'wave_merged', recovery: true, units: ['DJ2'], head: 'abd' }); + appendOk(dir, '_run', { task: '_run', event: 'wave_checked', wave: 3, head: 'abe', ok: true }); + appendOk(dir, '_run', { task: '_run', event: 'wave_checked', wave: 3, head: 'abe', ok: false, unfinished: true }); + for (const bad of [ + { task: 'DJ2', event: 'wave_merged', wave: 2, units: ['DJ2'], head: 'abc' }, + { task: 'DJ2', event: 'wave_checked', wave: 2, head: 'abc', ok: true }, + { task: '_run', event: 'wave_merged', wave: 2, units: [], head: 'abc' }, + { task: '_run', event: 'wave_merged', wave: 0, units: ['a'], head: 'abc' }, + { task: '_run', event: 'wave_merged', wave: 2, units: ['a'] }, + { task: '_run', event: 'wave_merged', wave: 2, units: ['a'], head: 'abc', overlaps: 'x' }, + { task: '_run', event: 'wave_checked', wave: 2, head: 'abc', ok: 'yes' }, + { task: '_run', event: 'wave_checked', wave: 2, head: 'abc', ok: true, unfinished: true }, + { task: '_run', event: 'wave_checked', head: 'abc', ok: true }, + ]) { + assert.equal(ledger('append', dir, '_run', JSON.stringify(bad)).code, 2, JSON.stringify(bad)); + } +}); + +test('wave_merged needs exactly one of wave and recovery', () => { + const dir = freshDir(); + for (const bad of [ + { task: '_run', event: 'wave_merged', units: ['a'], head: 'abc' }, + { task: '_run', event: 'wave_merged', wave: 1, recovery: true, units: ['a'], head: 'abc' }, + { task: '_run', event: 'wave_merged', recovery: false, units: ['a'], head: 'abc' }, + ]) { + assert.equal(ledger('append', dir, '_run', JSON.stringify(bad)).code, 2, JSON.stringify(bad)); + } +}); + +test('started accepts dir, unit_branch and restart', () => { + const dir = freshDir(); + appendOk(dir, 'join', { task: 'DJ2', event: 'started', base: 'abc', dir: 'C:/wt/lane-a', + unit_branch: 'pl-d1a-u-DJ2' }); + appendOk(dir, 'join', { task: 'DJ2', event: 'started', base: 'abd', restart: true }); +}); + +test('started refuses a unit_branch of another form', () => { + const dir = freshDir(); + for (const bad of [ + { task: 'DJ2', event: 'started', base: 'abc', unit_branch: 'pl-d1a-a' }, + { task: 'DJ2', event: 'started', base: 'abc', unit_branch: '' }, + { task: 'DJ2', event: 'started', base: 'abc', dir: '' }, + { task: 'DJ2', event: 'started', base: 'abc', restart: false }, + ]) { + assert.equal(ledger('append', dir, 'join', JSON.stringify(bad)).code, 2, JSON.stringify(bad)); + } +}); + +// ---- Task 3: reads ---- + +const GIT_ENV = { + GIT_CONFIG_GLOBAL: '/dev/null', GIT_CONFIG_NOSYSTEM: '1', + GIT_AUTHOR_NAME: 'T', GIT_AUTHOR_EMAIL: 't@example.invalid', + GIT_COMMITTER_NAME: 'T', GIT_COMMITTER_EMAIL: 't@example.invalid', +}; +function git(dir, ...args) { + const res = spawnSync('git', ['-C', dir, ...args], { encoding: 'utf8', env: mergeEnv(process.env, GIT_ENV) }); + assert.equal(res.status, 0, res.stderr); + return res.stdout.replace(/\r\n/g, '\n').trim(); +} +const jtask = (id) => ({ id, title: id, files: [], tier: 'standard', security: false }); + +// A repo on branch feat (initial commit W), a ledger dir, and a manifest whose +// join is ids with start point W. +function joinCase(ids) { + counter += 1; + const root = join(TMP, `join ${counter}`); + const repo = join(root, 'repo'); + mkdirSync(repo, { recursive: true }); + git(repo, 'init', '-q', '-b', 'feat'); + git(repo, 'commit', '-q', '--allow-empty', '-m', 'init'); + const w = git(repo, 'rev-parse', 'HEAD'); + const ledgerDir = join(root, 'ledger'); + const manifest = join(root, 'manifest.json'); + writeFileSync(manifest, JSON.stringify({ + repo: { mode: 'git', root: repo, git_dir: null }, prelude: [], lanes: [], join: ids.map(jtask), + start_points: { join: w }, + })); + return { root, repo, w, ledger: ledgerDir, manifest }; +} +// Commit a file on branch (created from `from` when it does not exist) and +// record the task's range from..HEAD with finish-task; returns the head. +function unitCommit(c, branch, from, taskId, name) { + const exists = spawnSync('git', ['-C', c.repo, 'rev-parse', '--verify', '--quiet', `refs/heads/${branch}`], + { env: mergeEnv(process.env, GIT_ENV) }).status === 0; + git(c.repo, 'checkout', '-q', ...(exists ? [branch] : ['-b', branch, from])); + writeFileSync(join(c.repo, name), `${name}\n`); + git(c.repo, 'add', name); + git(c.repo, 'commit', '-q', '-m', name); + const res = run('finish-task', [c.repo, branch, from, c.ledger, 'join', '--task', taskId]); + assert.equal(res.code, 0, res.stderr); + return git(c.repo, 'rev-parse', 'HEAD'); +} +const backfillOf = (c) => { + const res = ledger('backfill', c.ledger, c.manifest); + return { code: res.code, ...JSON.parse(res.stdout) }; +}; + +test('1.6.4 ledgers give join_started false', () => { + const dir = freshDir(); + appendOk(dir, 'prelude', { task: 'T1', event: 'started', base: 'abc' }); + appendOk(dir, 'join', { task: 'J1', event: 'committed', commits: ['abd'], base: 'abc', head: 'abd' }); + const s = statusOf(dir); + assert.equal(s.join_started, false); + assert.deepEqual(s.unit_branches, {}); + assert.deepEqual(s.wave_merges, []); +}); + +test('a started event in join.jsonl gives join_started', () => { + const dir = freshDir(); + appendOk(dir, 'join', { task: 'J1', event: 'started', base: 'abc' }); + assert.equal(statusOf(dir).join_started, true); +}); + +test('unit_branches lists the latest unit branch and dir', () => { + const dir = freshDir(); + appendOk(dir, 'join', { task: 'J2', event: 'started', base: 'abc', dir: 'C:/wt/lane-a', unit_branch: 'pl-r-u-J2' }); + appendOk(dir, 'join', { task: 'J3', event: 'started', base: 'abc', dir: 'C:/wt/lane-a', unit_branch: 'pl-r-u-J3' }); + appendOk(dir, 'join', { task: 'J3', event: 'started', base: 'abc', dir: 'C:/wt/lane-b', unit_branch: 'pl-r-u-J3' }); + appendOk(dir, 'join', { task: 'J2', event: 'started', base: 'abe', restart: true }); + assert.deepEqual(statusOf(dir).unit_branches, { J3: { branch: 'pl-r-u-J3', dir: 'C:/wt/lane-b' } }); +}); + +test('a restart resets a done-then-blocked task', () => { + const dir = freshDir(); + appendOk(dir, 'join', { task: 'J2', event: 'started', base: 'w0', unit_branch: 'pl-r-u-J2', dir: 'C:/wt/a' }); + appendOk(dir, 'join', { task: 'J2', event: 'committed', commits: ['h1'], base: 'w0', head: 'h1' }); + appendOk(dir, 'join', { task: 'J2', event: 'blocked', reason: 'review_rounds' }); + let s = statusOf(dir); + assert.ok(s.done.includes('J2') && s.blocked.includes('J2')); + appendOk(dir, 'join', { task: 'J2', event: 'started', base: 'tip', restart: true }); + s = statusOf(dir); + assert.ok(!s.done.includes('J2')); + assert.ok(!s.blocked.includes('J2')); + assert.deepEqual(s.inconsistent, []); + assert.equal(s.ranges.J2, undefined); + appendOk(dir, 'join', { task: 'J2', event: 'committed', commits: ['h2'], base: 'tip', head: 'h2' }); + s = statusOf(dir); + assert.deepEqual(s.ranges.J2, { base: 'tip', head: 'h2' }); + assert.deepEqual(s.inconsistent, []); +}); + +test('a restart at a new base is not a start mismatch', () => { + const dir = freshDir(); + appendOk(dir, 'join', { task: 'J2', event: 'started', base: 'w0' }); + appendOk(dir, 'join', { task: 'J2', event: 'committed', commits: ['h1'], base: 'w0', head: 'h1' }); + appendOk(dir, 'join', { task: 'J2', event: 'started', base: 'other', restart: true }); + assert.deepEqual(statusOf(dir).inconsistent, []); + const plain = freshDir(); + appendOk(plain, 'join', { task: 'J2', event: 'started', base: 'w0' }); + appendOk(plain, 'join', { task: 'J2', event: 'committed', commits: ['h1'], base: 'w0', head: 'h1' }); + appendOk(plain, 'join', { task: 'J2', event: 'started', base: 'other' }); + assert.equal(statusOf(plain).inconsistent.length, 1, 'without restart it is still a mismatch'); +}); + +test('wave_merges lists merge events in order', () => { + const dir = freshDir(); + appendOk(dir, '_run', { task: '_run', event: 'wave_merged', wave: 2, units: ['J2', 'J3'], head: 'm2', + overlaps: [{ path: 'a', units: ['J2', 'J3'] }], conflict_files: [] }); + appendOk(dir, '_run', { task: '_run', event: 'wave_checked', wave: 2, head: 'm2', ok: true }); + appendOk(dir, '_run', { task: '_run', event: 'wave_merged', recovery: true, units: ['J4'], head: 'm3' }); + assert.deepEqual(statusOf(dir).wave_merges, [ + { wave: 2, units: ['J2', 'J3'], head: 'm2', overlaps: [{ path: 'a', units: ['J2', 'J3'] }], conflict_files: [] }, + { recovery: true, units: ['J4'], head: 'm3', overlaps: [], conflict_files: [] }, + ]); +}); + +test('backfill continues past an unfinished wave unit', () => { + const c = joinCase(['A', 'B', 'C']); + for (const [id, branch] of [['A', 'pl-r-u-A'], ['B', 'pl-r-u-B'], ['C', 'pl-r-u-C']]) { + appendOk(c.ledger, 'join', { task: id, event: 'started', base: c.w, dir: 'C:/wt/x', unit_branch: branch }); + } + const a = unitCommit(c, 'pl-r-u-A', c.w, 'A', 'a.txt'); + const cc = unitCommit(c, 'pl-r-u-C', c.w, 'C', 'c.txt'); + const out = backfillOf(c); + assert.equal(out.code, 0, JSON.stringify(out.errors)); + assert.deepEqual(out.backfill, { A: { base: c.w, head: a }, C: { base: c.w, head: cc } }); +}); + +test('backfill keeps chaining a serial join', () => { + const c = joinCase(['A', 'B', 'C']); + const a = unitCommit(c, 'feat', c.w, 'A', 'a.txt'); + unitCommit(c, 'side', a, 'C', 'c.txt'); + const out = backfillOf(c); + assert.equal(out.code, 0, JSON.stringify(out.errors)); + assert.deepEqual(out.backfill, { A: { base: c.w, head: a } }); + assert.equal(out.join_tip, a); +}); + +test('join_tip is the descendant of wave and serial heads', () => { + const c = joinCase(['A', 'B', 'D']); + for (const [id, branch] of [['A', 'pl-r-u-A'], ['B', 'pl-r-u-B']]) { + appendOk(c.ledger, 'join', { task: id, event: 'started', base: c.w, dir: 'C:/wt/x', unit_branch: branch }); + } + const a = unitCommit(c, 'pl-r-u-A', c.w, 'A', 'a.txt'); + const b = unitCommit(c, 'pl-r-u-B', c.w, 'B', 'b.txt'); + git(c.repo, 'checkout', '-q', 'feat'); + git(c.repo, 'merge', '-q', '--no-ff', '-m', 'join wave 1: merge A', a); + git(c.repo, 'merge', '-q', '--no-ff', '-m', 'join wave 1: merge B', b); + const m = git(c.repo, 'rev-parse', 'HEAD'); + appendOk(c.ledger, '_run', { task: '_run', event: 'wave_merged', wave: 1, units: ['A', 'B'], head: m }); + appendOk(c.ledger, 'join', { task: 'D', event: 'started', base: m }); + const d = unitCommit(c, 'feat', m, 'D', 'd.txt'); + const out = backfillOf(c); + assert.equal(out.code, 0, JSON.stringify(out.errors)); + assert.equal(out.join_tip, d); + assert.deepEqual(out.backfill.D, { base: m, head: d }); +}); + +test('join_tip refuses unordered heads', () => { + const c = joinCase(['A', 'D']); + appendOk(c.ledger, 'join', { task: 'A', event: 'started', base: c.w, dir: 'C:/wt/x', unit_branch: 'pl-r-u-A' }); + const a = unitCommit(c, 'pl-r-u-A', c.w, 'A', 'a.txt'); + git(c.repo, 'checkout', '-q', 'feat'); + git(c.repo, 'merge', '-q', '--no-ff', '-m', 'join wave 1: merge A', a); + const m = git(c.repo, 'rev-parse', 'HEAD'); + appendOk(c.ledger, '_run', { task: '_run', event: 'wave_merged', wave: 1, units: ['A'], head: m }); + appendOk(c.ledger, 'join', { task: 'D', event: 'started', base: c.w }); + unitCommit(c, 'other', c.w, 'D', 'd.txt'); + const out = backfillOf(c); + assert.equal(out.code, 3); + assert.ok(out.errors.some((e) => /^join_tip: heads \S+ and \S+ do not descend from one another$/.test(e)), + JSON.stringify(out.errors)); + assert.equal(out.join_tip, null); +}); + +test('no join work gives join_tip null', () => { + const c = joinCase(['A']); + const out = backfillOf(c); + assert.equal(out.code, 0); + assert.equal(out.join_tip, null); +}); + +test('start_bases lists the latest started base of each task not done', () => { + const dir = freshDir(); + appendOk(dir, 'join', { task: 'J1', event: 'started', base: 'a1' }); + appendOk(dir, 'join', { task: 'J2', event: 'started', base: 'b1' }); + appendOk(dir, 'join', { task: 'J2', event: 'committed', commits: ['b2'], base: 'b1', head: 'b2' }); + appendOk(dir, 'join', { task: 'J3', event: 'started', base: 'c1', unit_branch: 'pl-r-u-J3', dir: 'C:/wt/a' }); + appendOk(dir, 'join', { task: 'J3', event: 'started', base: 'c2', restart: true }); + assert.deepEqual(statusOf(dir).start_bases, { J1: 'a1', J3: 'c2' }); +}); diff --git a/tests/merge-wave.test.mjs b/tests/merge-wave.test.mjs new file mode 100644 index 0000000..723d8b1 --- /dev/null +++ b/tests/merge-wave.test.mjs @@ -0,0 +1,219 @@ +// scripts/merge-wave (1.6.5 design 4.3): merges a join wave's units into the +// feature checkout by recorded sha, in the order given, idempotently; a +// conflict is aborted and reported; overlaps are the paths two units both +// changed; a wave_merged event goes to the ledger. Real git, real ledger. + +import { test, after } from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { existsSync, mkdirSync, readFileSync, realpathSync, rmSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { SKILL_DIR } from './harness.mjs'; +import { tempDir, mergeEnv } from './platform.mjs'; + +const SCRIPTS = join(SKILL_DIR, 'scripts'); +const TMP = realpathSync(tempDir('pl-merge-wave-')); +after(() => rmSync(TMP, { recursive: true, force: true })); +const GIT_ENV = { + GIT_CONFIG_GLOBAL: '/dev/null', GIT_CONFIG_NOSYSTEM: '1', + GIT_AUTHOR_NAME: 'T', GIT_AUTHOR_EMAIL: 't@example.invalid', + GIT_COMMITTER_NAME: 'T', GIT_COMMITTER_EMAIL: 't@example.invalid', +}; + +function git(dir, ...args) { + const res = spawnSync('git', ['-C', dir, ...args], { encoding: 'utf8', env: mergeEnv(process.env, GIT_ENV) }); + assert.equal(res.status, 0, `git ${args.join(' ')}: ${res.stderr}`); + return res.stdout.replace(/\r\n/g, '\n').trim(); +} + +function mergeWave(c, ...args) { + const res = spawnSync('python3', ['-B', join(SCRIPTS, 'merge-wave'), c.repo, c.ledger, ...args], + { encoding: 'utf8', env: mergeEnv(process.env, GIT_ENV) }); + let out = null; + try { + out = JSON.parse(res.stdout); + } catch { + out = null; + } + return { code: res.status, out, stderr: res.stderr }; +} + +const events = (c) => (existsSync(join(c.ledger, '_run.jsonl')) + ? readFileSync(join(c.ledger, '_run.jsonl'), 'utf8').trim().split('\n').filter(Boolean).map((l) => JSON.parse(l)) + : []); + +let counter = 0; +// A repo on feat at W with shared.txt (five lines); unit branches made by +// unit(c, id, edits) from W, each a commit writing files. +function newCase() { + counter += 1; + const root = join(TMP, `case ${counter}`); + const repo = join(root, 'repo'); + mkdirSync(repo, { recursive: true }); + git(repo, 'init', '-q', '-b', 'feat'); + writeFileSync(join(repo, 'shared.txt'), 'one\ntwo\nthree\nfour\nfive\n'); + git(repo, 'add', '-A'); + git(repo, 'commit', '-q', '-m', 'init'); + return { root, repo, ledger: join(root, 'ledger'), w: git(repo, 'rev-parse', 'HEAD') }; +} +// A unit branch from W with one commit setting each file of files to its +// text; returns ['--unit', id, W, head]. +function unit(c, id, files, from = c.w) { + git(c.repo, 'checkout', '-q', '-b', `pl-r-u-${id}`, from); + for (const [name, text] of Object.entries(files)) writeFileSync(join(c.repo, name), text); + git(c.repo, 'add', '-A'); + git(c.repo, 'commit', '-q', '-m', `unit ${id}`); + const head = git(c.repo, 'rev-parse', 'HEAD'); + git(c.repo, 'checkout', '-q', 'feat'); + return ['--unit', id, from, head]; +} +const firstParents = (c) => git(c.repo, 'log', '--first-parent', '--format=%s', `${c.w}..HEAD`).split('\n').filter(Boolean); + +test('merges in the given order', () => { + const c = newCase(); + const a = unit(c, 'A', { 'a.txt': 'a\n' }); + const b = unit(c, 'B', { 'b.txt': 'b\n' }); + const r = mergeWave(c, '--wave', '2', ...b, ...a); + assert.equal(r.code, 0, r.stderr); + assert.equal(r.out.status, 'merged'); + assert.deepEqual(r.out.merged, ['B', 'A']); + assert.deepEqual(r.out.skipped, []); + assert.deepEqual(firstParents(c), ['join wave 2: merge A', 'join wave 2: merge B']); + assert.equal(r.out.head, git(c.repo, 'rev-parse', 'HEAD')); + const [ev] = events(c); + assert.deepEqual(ev, { task: '_run', event: 'wave_merged', wave: 2, units: ['B', 'A'], head: r.out.head, + overlaps: [], conflict_files: [] }); +}); + +test('a rerun skips merged units and changes nothing', () => { + const c = newCase(); + const a = unit(c, 'A', { 'a.txt': 'a\n' }); + const b = unit(c, 'B', { 'b.txt': 'b\n' }); + const first = mergeWave(c, '--wave', '1', ...a, ...b); + assert.equal(first.code, 0, first.stderr); + const again = mergeWave(c, '--wave', '1', ...a, ...b); + assert.equal(again.code, 0, again.stderr); + assert.deepEqual(again.out.merged, []); + assert.deepEqual(again.out.skipped, ['A', 'B']); + assert.equal(again.out.head, first.out.head); + assert.equal(git(c.repo, 'rev-parse', 'HEAD'), first.out.head); + assert.equal(events(c).length, 1, 'nothing merged, nothing recorded'); +}); + +test('a conflict aborts and reports the unit and files', () => { + const c = newCase(); + const a = unit(c, 'A', { 'shared.txt': 'one\nTWO-A\nthree\nfour\nfive\n' }); + const b = unit(c, 'B', { 'shared.txt': 'one\nTWO-B\nthree\nfour\nfive\n', 'b.txt': 'b\n' }); + const r = mergeWave(c, '--wave', '1', ...a, ...b); + assert.equal(r.code, 1, r.stderr); + assert.equal(r.out.status, 'conflict'); + assert.deepEqual(r.out.merged, ['A']); + assert.equal(r.out.conflict_unit, 'B'); + assert.deepEqual(r.out.conflict_files, ['shared.txt']); + assert.equal(git(c.repo, 'status', '--porcelain'), '', 'the checkout is clean after the abort'); + assert.equal(r.out.head, git(c.repo, 'rev-parse', 'HEAD')); + const [ev] = events(c); + assert.deepEqual(ev.units, ['A']); + assert.deepEqual(ev.conflict_files, ['shared.txt']); +}); + +test('after a manual resolution a rerun merges the rest', () => { + const c = newCase(); + const a = unit(c, 'A', { 'shared.txt': 'one\nTWO-A\nthree\nfour\nfive\n' }); + const b = unit(c, 'B', { 'shared.txt': 'one\nTWO-B\nthree\nfour\nfive\n' }); + const d = unit(c, 'D', { 'd.txt': 'd\n' }); + assert.equal(mergeWave(c, '--wave', '1', ...a, ...b, ...d).code, 1); + // The resolver merges B by hand. + spawnSync('git', ['-C', c.repo, 'merge', '--no-ff', '-m', 'join wave 1: merge B', b[3]], + { env: mergeEnv(process.env, GIT_ENV) }); + writeFileSync(join(c.repo, 'shared.txt'), 'one\nTWO-A-B\nthree\nfour\nfive\n'); + git(c.repo, 'add', '-A'); + git(c.repo, 'commit', '-q', '--no-edit'); + const r = mergeWave(c, '--wave', '1', ...a, ...b, ...d); + assert.equal(r.code, 0, r.stderr); + assert.deepEqual(r.out.skipped, ['A', 'B']); + assert.deepEqual(r.out.merged, ['D']); +}); + +test('overlaps lists paths two units changed', () => { + const c = newCase(); + const a = unit(c, 'A', { 'shared.txt': 'ONE\ntwo\nthree\nfour\nfive\n', 'a.txt': 'a\n' }); + const b = unit(c, 'B', { 'shared.txt': 'one\ntwo\nthree\nfour\nFIVE\n' }); + const d = unit(c, 'D', { 'd.txt': 'd\n' }); + const r = mergeWave(c, '--wave', '3', ...a, ...b, ...d); + assert.equal(r.code, 0, r.stderr); + assert.deepEqual(r.out.overlaps, [{ path: 'shared.txt', units: ['A', 'B'] }]); + assert.equal(readFileSync(join(c.repo, 'shared.txt'), 'utf8'), 'ONE\ntwo\nthree\nfour\nFIVE\n'); + assert.deepEqual(events(c)[0].overlaps, [{ path: 'shared.txt', units: ['A', 'B'] }]); +}); + +test('a batch unit is merged once', () => { + const c = newCase(); + const batch = unit(c, 'J2-J3', { 'j2.txt': '2\n', 'j3.txt': '3\n' }); + const r = mergeWave(c, '--wave', '2', ...batch); + assert.equal(r.code, 0, r.stderr); + assert.deepEqual(r.out.merged, ['J2-J3']); + assert.deepEqual(firstParents(c), ['join wave 2: merge J2-J3']); +}); + +test('a dirty checkout is refused', () => { + const c = newCase(); + const a = unit(c, 'A', { 'a.txt': 'a\n' }); + writeFileSync(join(c.repo, 'stray.txt'), 'x\n'); + const r = mergeWave(c, '--wave', '1', ...a); + assert.equal(r.code, 3); + assert.match(r.stderr, /uncommitted changes/); + assert.equal(git(c.repo, 'rev-parse', 'HEAD'), c.w); +}); + +test('--recovery records recovery in the event', () => { + const c = newCase(); + const a = unit(c, 'A', { 'a.txt': 'a\n' }); + const r = mergeWave(c, '--recovery', ...a); + assert.equal(r.code, 0, r.stderr); + assert.deepEqual(firstParents(c), ['join recovery: merge A']); + const [ev] = events(c); + assert.equal(ev.recovery, true); + assert.equal('wave' in ev, false); +}); + +test('--recovery with no unit reports HEAD and records nothing', () => { + const c = newCase(); + const r = mergeWave(c, '--recovery'); + assert.equal(r.code, 0, r.stderr); + assert.deepEqual(r.out, { status: 'merged', head: c.w, merged: [], skipped: [], conflict_unit: null, + conflict_files: [], overlaps: [], in_history: [], notes: [] }); + assert.deepEqual(events(c), []); +}); + +test('an unknown sha is refused', () => { + const c = newCase(); + const r = mergeWave(c, '--wave', '1', '--unit', 'A', c.w, '0'.repeat(40)); + assert.equal(r.code, 3); + assert.equal(git(c.repo, 'rev-parse', 'HEAD'), c.w); +}); + +test('usage errors exit 2', () => { + const c = newCase(); + const a = unit(c, 'A', { 'a.txt': 'a\n' }); + for (const args of [[...a], ['--wave', '0', ...a], ['--wave', '1', '--recovery', ...a], ['--wave', '1'], + ['--wave', '1', '--unit', 'A', c.w], ['--wave', '1', '--unit', '../x', a[2], a[3]]]) { + assert.equal(mergeWave(c, ...args).code, 2, args.join(' ')); + } +}); + +test('--known units are reported in_history without being merged', () => { + const c = newCase(); + const a = unit(c, 'A', { 'a.txt': 'a\n' }); + const b = unit(c, 'B', { 'b.txt': 'b\n' }); + assert.equal(mergeWave(c, '--wave', '1', ...a).code, 0); + const r = mergeWave(c, '--recovery', '--known', 'A', a[3], '--known', 'B', b[3]); + assert.equal(r.code, 0, r.stderr); + assert.deepEqual(r.out.merged, []); + assert.deepEqual(r.out.in_history, ['A']); + assert.deepEqual(events(c).length, 1, 'nothing merged, nothing recorded'); + const both = mergeWave(c, '--recovery', '--unit', ...b.slice(1), '--known', 'A', a[3]); + assert.equal(both.code, 0, both.stderr); + assert.deepEqual(both.out.in_history, ['B', 'A']); + assert.equal(mergeWave(c, '--recovery', '--known', 'A').code, 2, 'a --known needs ID and HEAD'); +}); diff --git a/tests/phase-prompts.test.mjs b/tests/phase-prompts.test.mjs index 936d4bc..1422202 100644 --- a/tests/phase-prompts.test.mjs +++ b/tests/phase-prompts.test.mjs @@ -348,3 +348,41 @@ test('integration with no check commands says so in step 3', () => { assert.equal(runChecksLines(p).length, 0); assert.ok(p.includes('(no test, lint or build command)'), p); }); + +// ---- 1.6.5: every check call names its phase (run-checks --phase, PL_PHASE) ---- + +const { implementPrompt, featureWhere, laneWhere } = await loadHelpers(['implementPrompt', 'featureWhere', 'laneWhere']); + +// Every run-checks or task-checks call line of a prompt (the long-command +// template of a phase prompt aside: it has no phase). +const checkCalls = (p) => p.split('\n').filter((l) => /\/scripts\/(run|task)-checks' /.test(l)); + +test('every run-checks and task-checks call in a prompt names its phase', () => { + const m = manifest(); + const supervised = manifest({ autonomy: 'supervised' }); + const f = [{ id: 'F1', severity: 'important', file: 'src/a.js', line: 1, issue: 'x', fix: 'y' }]; + const cases = [ + ['integrate', integratePrompt(m, H('pre'), {}), true], + ['verify', verifyPrompt(m, H('d')), true], + ['verify', verifyPrompt(m, H('d'), H('from')), true], + ['post-integrate', postIntegrateFixPrompt(m, 'the hook failed', 'hook'), true], + ['post-integrate', postIntegratePrompt(m, false, null), true], + ['final-fix', finalFixPrompt(supervised, f, H('tip')), true], + ['prelude', implementPrompt(m, m.prelude[0], featureWhere(m, 'prelude'), H('b')), false], + ['lane', implementPrompt(m, m.lanes[0].tasks[0], laneWhere(m, m.lanes[0]), H('b')), false], + ['join', implementPrompt(m, m.join[0], featureWhere(m, 'join'), H('b')), false], + ]; + for (const [phase, prompt, phasePrompt] of cases) { + const calls = checkCalls(prompt).filter((l) => !(phasePrompt && l.includes("--cmd 'GROUP' 'COMMAND'"))); + assert.ok(calls.length > 0, `${phase}: no check call in the prompt`); + for (const line of calls) assert.ok(line.includes(`--phase '${phase}'`), `${phase}: ${line}`); + } +}); + +test('task prompts tell agents to set PL_PHASE for commands they run themselves', () => { + const m = manifest(); + const join = flat(implementPrompt(m, m.join[0], featureWhere(m, 'join'), H('b'))); + assert.ok(join.includes('Run project commands you start yourself with PL_PHASE=join set'), join); + const lane = flat(implementPrompt(m, m.lanes[0].tasks[0], laneWhere(m, m.lanes[0]), H('b'))); + assert.ok(lane.includes('PL_PHASE=lane'), lane); +}); diff --git a/tests/profiles.test.mjs b/tests/profiles.test.mjs index 40912be..8a12d08 100644 --- a/tests/profiles.test.mjs +++ b/tests/profiles.test.mjs @@ -154,7 +154,7 @@ test('lite: the lane works in the feature checkout on the feature branch with le }); test('lite: an empty prelude starts the lane at the saved setup start point', async () => { - const m = liteManifest({ prelude: [], start_points: { prelude: H('SP0'), join: H('SJ0') } }); + const m = liteManifest({ prelude: [], start_points: { prelude: H('SP0'), join: H('SJ0') }, join_started: false }); const script = { ...taskScript(['T2', 'T3', 'T4']), 'final review': [{ findings: [], cannot_verify: [], head: H('T4-h') }], diff --git a/tests/run-checks-phase.test.mjs b/tests/run-checks-phase.test.mjs new file mode 100644 index 0000000..9937808 --- /dev/null +++ b/tests/run-checks-phase.test.mjs @@ -0,0 +1,131 @@ +// scripts/run-checks --phase (1.6.5 design 4.6): every command, first attempt +// and busy retry alike, runs with PL_PHASE set to the phase; without --phase +// an inherited PL_PHASE is removed. task-checks passes its --phase through. +// Real scripts, real git repos. + +import { test, after } from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { existsSync, mkdirSync, readFileSync, realpathSync, rmSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { SKILL_DIR } from './harness.mjs'; +import { tempDir } from './platform.mjs'; + +const SCRIPTS = join(SKILL_DIR, 'scripts'); +const TMP = realpathSync(tempDir('pl-run-checks-phase-')); +after(() => rmSync(TMP, { recursive: true, force: true })); +const GIT_ENV = { + GIT_CONFIG_GLOBAL: '/dev/null', GIT_CONFIG_NOSYSTEM: '1', + GIT_AUTHOR_NAME: 'T', GIT_AUTHOR_EMAIL: 't@example.invalid', + GIT_COMMITTER_NAME: 'T', GIT_COMMITTER_EMAIL: 't@example.invalid', +}; + +function git(dir, ...args) { + const res = spawnSync('git', ['-C', dir, ...args], { encoding: 'utf8', env: { ...process.env, ...GIT_ENV } }); + assert.equal(res.status, 0, res.stderr); + return res.stdout.trim(); +} + +let serial = 0; +function fresh() { + serial += 1; + const dir = join(TMP, `repo${serial}`); + const record = join(TMP, `phase${serial}.txt`).replace(/\\/g, '/'); + mkdirSync(dir, { recursive: true }); + git(dir, 'init', '-q', '-b', 'main'); + writeFileSync(join(dir, 'a.txt'), 'a\n'); + git(dir, 'add', '-A'); + git(dir, 'commit', '-q', '-m', 'init'); + return { dir, record }; +} + +// A command that appends "" to record and exits +// codes[n - 1] on its nth run. +function recorder(record, codes = [0]) { + const js = `const fs = require("fs"); const p = ${JSON.stringify(record)};` + + ' const prev = fs.existsSync(p) ? fs.readFileSync(p, "utf8") : "";' + + ' const n = prev.split("\\n").filter(Boolean).length + 1;' + + ' fs.appendFileSync(p, (process.env.PL_PHASE === undefined ? "-unset-" : process.env.PL_PHASE) + "\\n");' + + ` const codes = ${JSON.stringify(codes)}; process.exit(codes[Math.min(n, codes.length) - 1]);`; + return `node -e '${js}'`; +} +const recorded = (record) => readFileSync(record, 'utf8').split('\n').filter(Boolean); + +function runChecks(dir, extra, cmds, env = {}) { + const argv = [join(SCRIPTS, 'run-checks'), dir, ...extra]; + for (const [group, command] of cmds) argv.push('--cmd', group, command); + const res = spawnSync('python3', ['-B', ...argv], { + encoding: 'utf8', env: { ...process.env, ...GIT_ENV, TMPDIR: TMP, RUN_CHECKS_BUSY_WAIT_S: '0', ...env }, + }); + return { status: res.status, stdout: res.stdout, stderr: res.stderr }; +} + +test('PL_PHASE reaches every command', () => { + const { dir, record } = fresh(); + const r = runChecks(dir, ['--phase', 'join'], [['test', recorder(record)], ['lint', recorder(record)]]); + assert.equal(r.status, 0, r.stderr); + assert.deepEqual(recorded(record), ['join', 'join']); + assert.equal(JSON.parse(r.stdout).phase, 'join'); +}); + +test('PL_PHASE survives the busy retry', () => { + const { dir, record } = fresh(); + const r = runChecks(dir, ['--phase', 'prelude'], [['test', recorder(record, [75, 0])]]); + assert.equal(r.status, 0, r.stderr); + assert.deepEqual(recorded(record), ['prelude', 'prelude']); +}); + +test('an inherited PL_PHASE is removed without --phase', () => { + const { dir, record } = fresh(); + const r = runChecks(dir, [], [['test', recorder(record)]], { PL_PHASE: 'lane' }); + assert.equal(r.status, 0, r.stderr); + assert.deepEqual(recorded(record), ['-unset-']); + assert.equal(JSON.parse(r.stdout).phase, null); +}); + +test('an unknown phase is a usage error', () => { + const { dir, record } = fresh(); + for (const extra of [['--phase', 'wave'], ['--phase'], ['--phase', '']]) { + const r = runChecks(dir, extra, [['test', recorder(record)]]); + assert.equal(r.status, 2, `${extra.join(' ')}: ${r.stderr}`); + } + assert.equal(existsSync(record), false); +}); + +test('every phase name is accepted', () => { + for (const phase of ['prelude', 'lane', 'join', 'integrate', 'post-integrate', 'final-fix', 'verify']) { + const { dir, record } = fresh(); + const r = runChecks(dir, ['--phase', phase], [['test', recorder(record)]]); + assert.equal(r.status, 0, `${phase}: ${r.stderr}`); + assert.deepEqual(recorded(record), [phase]); + } +}); + +test('task-checks passes --phase through', () => { + const { dir, record } = fresh(); + const ledger = join(TMP, `ledger${serial}`); + mkdirSync(ledger, { recursive: true }); + const base = git(dir, 'rev-parse', 'HEAD'); + const cmd = recorder(record); + const res = spawnSync('python3', ['-B', join(SCRIPTS, 'task-checks'), dir, '--base', base, '--ledger', ledger, + '--unit', 'J2', '--lane', 'join', '--task', 'J2', '--phase', 'join', + '--lane-cmd', 'test', cmd, '--wide-cmd', 'test', cmd], { + encoding: 'utf8', env: { ...process.env, ...GIT_ENV, TMPDIR: TMP, PL_PHASE: 'lane' }, + }); + assert.equal(res.status, 0, res.stderr); + assert.deepEqual(recorded(record), ['join']); +}); + +test('task-checks without --phase removes PL_PHASE too', () => { + const { dir, record } = fresh(); + const ledger = join(TMP, `ledger${serial}`); + mkdirSync(ledger, { recursive: true }); + const base = git(dir, 'rev-parse', 'HEAD'); + const cmd = recorder(record); + const res = spawnSync('python3', ['-B', join(SCRIPTS, 'task-checks'), dir, '--base', base, '--ledger', ledger, + '--unit', 'A1', '--lane', 'a', '--task', 'A1', '--lane-cmd', 'test', cmd, '--wide-cmd', 'test', cmd], { + encoding: 'utf8', env: { ...process.env, ...GIT_ENV, TMPDIR: TMP, PL_PHASE: 'join' }, + }); + assert.equal(res.status, 0, res.stderr); + assert.deepEqual(recorded(record), ['-unset-']); +}); diff --git a/tests/run.test.mjs b/tests/run.test.mjs index 6b91fba..975cf78 100644 --- a/tests/run.test.mjs +++ b/tests/run.test.mjs @@ -1421,7 +1421,7 @@ test('setup_result under profile lite maps the lane to the feature checkout', as }); test('start_points: the saved prelude and join heads are the bases of the first tasks', async () => { - const m = manifest({ start_points: { prelude: H('SP0'), join: H('SJ0') } }); + const m = manifest({ start_points: { prelude: H('SP0'), join: H('SJ0') }, join_started: false }); const { result, calls } = await run(m, { ...phaseScript(), ...taskScript(ALL) }); assert.equal(result.status, 'complete'); assert.deepEqual(result.tasks.T1.commits, [H('SP0'), H('T1-h')]); diff --git a/tests/setup-script.test.mjs b/tests/setup-script.test.mjs index 0d86a45..91595e2 100644 --- a/tests/setup-script.test.mjs +++ b/tests/setup-script.test.mjs @@ -170,7 +170,7 @@ test('setup: a fresh git run creates the branch and lane worktrees and prints se const base = git(c.project, 'rev-parse', 'main'); const { worktrees, ...result } = setupOk(c, manifest(c)); - assert.deepEqual(result, { feature_head: base, discarded: [], preserved: [], setup_skipped: [] }); + assert.deepEqual(result, { feature_head: base, discarded: [], preserved: [], setup_skipped: [], returned: [] }); assertSamePaths(worktrees, { a: laneDir(c, 'a'), b: laneDir(c, 'b') }); assert.equal(git(c.project, 'rev-parse', '--abbrev-ref', 'HEAD'), 'feature/x'); assert.equal(git(laneDir(c, 'a'), 'rev-parse', '--abbrev-ref', 'HEAD'), 'pl-r1-a'); @@ -401,7 +401,7 @@ test('setup: shadow mode creates the feature worktree and never touches the proj }); const feature = `${c.worktreeRoot}/feature`; const { worktrees, ...result } = setupOk(c, m); - assert.deepEqual(result, { feature_head: baseline, discarded: [], preserved: [], setup_skipped: [] }); + assert.deepEqual(result, { feature_head: baseline, discarded: [], preserved: [], setup_skipped: [], returned: [] }); assertSamePaths(worktrees, { a: laneDir(c, 'a'), b: laneDir(c, 'b') }); assert.equal(git(feature, 'rev-parse', '--abbrev-ref', 'HEAD'), 'pl-r1'); assert.equal(git(laneDir(c, 'a'), 'rev-parse', '--abbrev-ref', 'HEAD'), 'pl-r1-a'); @@ -1047,3 +1047,79 @@ test('setup: setup and the engine agree on pending', async () => { const result = setupOk(c, m); assert.deepEqual(result.setup_skipped, expected); }); + +// ---- 1.6.5: a lane worktree left on a join unit branch ---- + +// Leaves lane a's worktree on unit branch pl-r1-u-J1 with one commit and an +// uncommitted change; returns the unit commit. +function leaveOnUnitBranch(c) { + const wt = laneDir(c, 'a'); + git(wt, 'checkout', '-q', '-b', 'pl-r1-u-J1'); + write(wt, 'unit.txt', 'unit work\n'); + git(wt, 'add', '-A'); + git(wt, 'commit', '-q', '-m', 'unit work'); + const head = git(wt, 'rev-parse', 'HEAD'); + write(wt, 'wip.txt', 'in progress\n'); + return head; +} + +test('a worktree on a unit branch is returned to its lane branch', () => { + const c = newCase(); + const m = manifest(c); + setupOk(c, m); + const unitHead = leaveOnUnitBranch(c); + const res = setup(c, m); + assert.equal(res.code, 0, res.stderr); + const out = JSON.parse(res.stdout); + const wt = laneDir(c, 'a'); + assert.equal(git(wt, 'rev-parse', '--abbrev-ref', 'HEAD'), 'pl-r1-a'); + assert.equal(git(c.project, 'rev-parse', 'pl-r1-u-J1'), unitHead, 'the unit branch keeps its commit'); + assert.equal(out.returned.length, 1); + assert.ok(samePath(out.returned[0].worktree, wt), out.returned[0].worktree); + assert.equal(out.returned[0].from, 'pl-r1-u-J1'); + const saved = out.preserved.find((p) => samePath(p.worktree, wt)); + assert.ok(saved, JSON.stringify(out.preserved)); + assert.equal(git(c.project, 'show', `${saved.ref}:wip.txt`), 'in progress'); + assert.equal(git(wt, 'status', '--porcelain'), ''); +}); + +test('unit-branch worktree found in any path form', () => { + const c = newCase(); + const m = manifest(c); + setupOk(c, m); + leaveOnUnitBranch(c); + const other = manifest(c, { repo: { ...m.repo, worktree_root: c.worktreeRoot.replace(/\\/g, '/') } }); + const res = setup(c, other); + assert.equal(res.code, 0, res.stderr); + assert.equal(JSON.parse(res.stdout).returned.length, 1); + assert.equal(git(laneDir(c, 'a'), 'rev-parse', '--abbrev-ref', 'HEAD'), 'pl-r1-a'); +}); + +test('a lane worktree with nothing on a unit branch reports no return', () => { + const c = newCase(); + const m = manifest(c); + setupOk(c, m); + const res = setup(c, m); + assert.equal(res.code, 0, res.stderr); + assert.deepEqual(JSON.parse(res.stdout).returned, []); +}); + +test('a detached or foreign-branch lane worktree is still refused', () => { + for (const move of [(wt) => git(wt, 'checkout', '-q', '--detach'), (wt) => git(wt, 'checkout', '-q', '-b', 'other')]) { + const c = newCase(); + const m = manifest(c); + setupOk(c, m); + move(laneDir(c, 'a')); + const res = setup(c, m); + assert.equal(res.code, 3, res.stderr); + assert.match(res.stderr, /is not a worktree of this repo on pl-r1-a/); + } +}); + +test('another run\'s unit branch is refused', () => { + const c = newCase(); + const m = manifest(c); + setupOk(c, m); + git(laneDir(c, 'a'), 'checkout', '-q', '-b', 'pl-r2-u-J1'); + assert.equal(setup(c, m).code, 3); +}); diff --git a/tests/skill.test.mjs b/tests/skill.test.mjs index d5f2b2c..1dc2cbd 100644 --- a/tests/skill.test.mjs +++ b/tests/skill.test.mjs @@ -304,15 +304,17 @@ test('reference.md documents the Workflow concurrency cap', () => { assert.ok(text.includes('CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS'), 'reference.md lacks the concurrency cap'); }); -test('README asks for Node 22 and no longer calls Windows preliminary', () => { +test('README asks for Node 24 and no longer calls Windows preliminary', () => { const text = read(README_MD).toString('utf8'); - assert.ok(text.includes('Node 22') || text.includes('**node** 22'), 'README does not ask for Node 22'); + assert.ok(text.includes('Node 24'), 'README does not ask for Node 24'); + assert.ok(!text.includes('Node 22'), 'README still asks for Node 22'); assert.ok(!/preliminary/i.test(text), 'README still calls Windows support preliminary'); }); -test('CI tests Node 22, not Node 18', () => { +test('CI tests Node 24, not Node 22 or 18', () => { const text = read(CI_YML).toString('utf8'); - assert.ok(text.includes('node: 22'), 'CI has no Node 22 job'); + assert.ok(text.includes('node: 24'), 'CI has no Node 24 job'); + assert.ok(!text.includes('node: 22'), 'CI still has a Node 22 job'); assert.ok(!text.includes('node: 18'), 'CI still has a Node 18 job'); }); @@ -677,7 +679,7 @@ test('reference.md documents the check policy fields', () => { test('SKILL.md lists checks among the dry-run fields and points to the check policy fields', () => { const skill = flat(SKILL_MD); - assert.ok(skill.includes('`{dry_run, errors, agents, lanes_effective, notices, checks}`'), 'dry-run fields'); + assert.ok(skill.includes('`{dry_run, errors, agents, lanes_effective, notices, checks, join_waves}`'), 'dry-run fields'); assert.ok(skill.includes('`task_checks`') && skill.includes('`widen_paths`'), 'manifest step'); }); diff --git a/tests/task-helpers.test.mjs b/tests/task-helpers.test.mjs index 07cabce..6dcc926 100644 --- a/tests/task-helpers.test.mjs +++ b/tests/task-helpers.test.mjs @@ -595,3 +595,145 @@ test('finish-task ignores GIT_DIR and friends pointing at another repo', () => { }); assert.deepEqual(done(c), ['T1']); }); + +// ---- 1.6.5: start-task --unit-branch (a join wave unit in a lane worktree) ---- + +const briefOut = (c, id = 'T1') => join(c.root, 'briefs', `${id}.md`); +const abandoned = (c) => git(c.project, 'for-each-ref', '--format=%(refname)', 'refs/parallel-lanes/r1/abandoned') + .split('\n').filter(Boolean); +const unitStart = (c, ...extra) => startTask(c.lane, c.plan, '--unit-branch', 'pl-r1-u-T1', c.initial, + '--run-id', 'r1', ...extra, '--brief', 'T1', briefOut(c)); + +test('a new unit branch is created at SHA', () => { + const c = newCase(); + commit(c.lane, 'lane.txt', 'lane work\n', 'lane work'); + const res = unitStart(c); + assert.equal(res.code, 0, res.stderr); + assert.equal(git(c.lane, 'rev-parse', '--abbrev-ref', 'HEAD'), 'pl-r1-u-T1'); + assert.equal(git(c.lane, 'rev-parse', 'HEAD'), c.initial); + assert.match(res.stdout, /^branch: pl-r1-u-T1\n/); + assert.match(res.stdout, /\nunit branch: pl-r1-u-T1 \(created\)\n/); + assert.equal(git(c.lane, 'rev-parse', 'lane-a') === c.initial, false, 'the lane branch keeps its commit'); +}); + +test('switching saves uncommitted work under abandoned refs', () => { + const c = newCase(); + write(join(c.lane, 'a.txt'), 'edited\n'); + write(join(c.lane, 'new.txt'), 'untracked\n'); + const res = unitStart(c); + assert.equal(res.code, 0, res.stderr); + const refs = abandoned(c); + assert.equal(refs.length, 1); + assert.equal(git(c.project, 'show', `${refs[0]}:new.txt`), 'untracked'); + assert.equal(git(c.project, 'show', `${refs[0]}:a.txt`), 'edited'); + assert.equal(git(c.lane, 'status', '--porcelain'), ''); + assert.match(res.stdout, new RegExp(`\\nsaved: ${refs[0]} \\([0-9a-f]{40}\\)\\n`)); + assert.match(res.stdout, /\nunit branch: pl-r1-u-T1 \(created\)\n/); +}); + +test('a retry on the same unit branch keeps uncommitted work', () => { + const c = newCase(); + assert.equal(unitStart(c).code, 0); + const head = commit(c.lane, 'unit.txt', 'unit work\n', 'unit work'); + write(join(c.lane, 'wip.txt'), 'in progress\n'); + const res = unitStart(c); + assert.equal(res.code, 0, res.stderr); + assert.match(res.stdout, /\nunit branch: pl-r1-u-T1 \(kept\)\n/); + assert.equal(git(c.lane, 'rev-parse', 'HEAD'), head); + assert.equal(readFileSync(join(c.lane, 'wip.txt'), 'utf8'), 'in progress\n'); + assert.deepEqual(abandoned(c), []); +}); + +test('switching back to an existing unit branch keeps its commits', () => { + const c = newCase(); + assert.equal(unitStart(c).code, 0); + const head = commit(c.lane, 'unit.txt', 'unit work\n', 'unit work'); + git(c.lane, 'checkout', '-q', 'lane-a'); + const res = unitStart(c); + assert.equal(res.code, 0, res.stderr); + assert.match(res.stdout, /\nunit branch: pl-r1-u-T1 \(switched\)\n/); + assert.equal(git(c.lane, 'rev-parse', 'HEAD'), head); +}); + +test('the setup commands run in the worktree after the checkout', () => { + const c = newCase(); + const res = unitStart(c, '--setup', 'git rev-parse --abbrev-ref HEAD > setup.txt', '--setup', 'echo two >> setup.txt'); + assert.equal(res.code, 0, res.stderr); + assert.equal(readFileSync(join(c.lane, 'setup.txt'), 'utf8').replace(/\r\n/g, '\n'), 'pl-r1-u-T1\ntwo\n'); +}); + +test('a failing setup command exits 1', () => { + const c = newCase(); + const res = unitStart(c, '--setup', 'exit 7', '--record-start', c.ledgerDir, 'join', c.initial); + assert.equal(res.code, 1); + assert.match(res.stderr, /setup command 'exit 7' exited 7/); + assert.ok(!existsSync(join(c.ledgerDir, 'join.jsonl')), 'no started event'); +}); + +test('started records dir in forward-slash form and the unit branch', () => { + const c = newCase(); + const res = unitStart(c, '--record-start', c.ledgerDir, 'join', c.initial); + assert.equal(res.code, 0, res.stderr); + const [ev] = readFileSync(join(c.ledgerDir, 'join.jsonl'), 'utf8').trim().split('\n').map((l) => JSON.parse(l)); + assert.equal(ev.unit_branch, 'pl-r1-u-T1'); + assert.equal(ev.base, c.initial); + assert.ok(!ev.dir.includes('\\'), ev.dir); + assert.ok(samePath(ev.dir, c.lane), `${ev.dir} is not ${c.lane}`); + assert.equal('restart' in ev, false); +}); + +test('--restart adds restart to the started event', () => { + const c = newCase(); + const res = unitStart(c, '--record-start', c.ledgerDir, 'join', c.initial, '--restart'); + assert.equal(res.code, 0, res.stderr); + const [ev] = readFileSync(join(c.ledgerDir, 'join.jsonl'), 'utf8').trim().split('\n').map((l) => JSON.parse(l)); + assert.equal(ev.restart, true); +}); + +test('--restart without --unit-branch records a restart on the current branch', () => { + const c = newCase(); + const res = startTask(c.lane, c.plan, '--record-start', c.ledgerDir, 'join', 'HEAD', '--restart', + '--brief', 'T1', briefOut(c)); + assert.equal(res.code, 0, res.stderr); + const [ev] = readFileSync(join(c.ledgerDir, 'join.jsonl'), 'utf8').trim().split('\n').map((l) => JSON.parse(l)); + assert.deepEqual(ev, { task: 'T1', event: 'started', base: c.initial, restart: true }); +}); + +test('unit-branch usage errors', () => { + const c = newCase(); + const usage = (...args) => startTask(c.lane, c.plan, ...args, '--brief', 'T1', briefOut(c)).code; + assert.equal(usage('--unit-branch', 'pl-r1-u-T1', c.initial), 2, 'no --run-id'); + assert.equal(usage('--unit-branch', 'pl-r1-u-T1', c.initial, '--run-id', 'r1', '--sync', 'main'), 2); + assert.equal(usage('--setup', 'true'), 2, '--setup needs --unit-branch'); + assert.equal(usage('--restart'), 2, '--restart needs --record-start'); + assert.equal(usage('--unit-branch', 'lane-a', c.initial, '--run-id', 'r1'), 2, 'not a unit branch name'); + assert.equal(usage('--unit-branch', 'pl-r1-u-T1', c.initial, '--run-id', 'R!'), 2, 'unsafe run id'); +}); + +test('an unknown SHA for a new unit branch exits 3 and changes nothing', () => { + const c = newCase(); + write(join(c.lane, 'a.txt'), 'edited\n'); + const res = startTask(c.lane, c.plan, '--unit-branch', 'pl-r1-u-T1', '0'.repeat(40), '--run-id', 'r1', + '--brief', 'T1', briefOut(c)); + assert.equal(res.code, 3); + assert.equal(git(c.lane, 'rev-parse', '--abbrev-ref', 'HEAD'), 'lane-a'); + assert.equal(readFileSync(join(c.lane, 'a.txt'), 'utf8'), 'edited\n'); + assert.deepEqual(abandoned(c), []); +}); + +test('an existing unit branch not built on SHA is saved and made again at SHA', () => { + const c = newCase(); + assert.equal(unitStart(c).code, 0); + const old = commit(c.lane, 'unit.txt', 'built on the old base\n', 'unit work'); + git(c.lane, 'checkout', '-q', 'lane-a'); + const tip = commit(c.lane, 'lane2.txt', 'lane rework\n', 'lane rework'); + const res = startTask(c.lane, c.plan, '--unit-branch', 'pl-r1-u-T1', tip, '--run-id', 'r1', '--brief', 'T1', briefOut(c)); + assert.equal(res.code, 0, res.stderr); + assert.match(res.stdout, /\nunit branch: pl-r1-u-T1 \(recreated\)\n/); + assert.equal(git(c.lane, 'rev-parse', 'HEAD'), tip); + const saved = git(c.project, 'for-each-ref', '--format=%(refname) %(objectname)', 'refs/parallel-lanes/r1/abandoned') + .split('\n').filter((l) => l.includes('pl-r1-u-T1-')); + assert.equal(saved.length, 1, 'the old branch head is kept under the abandoned refs'); + assert.ok(saved[0].endsWith(old)); + assert.match(res.stdout, new RegExp(`\\nsaved: refs/parallel-lanes/r1/abandoned/pl-r1-u-T1-\\S+ \\(${old}\\)\\n`)); +}); diff --git a/tests/validate.test.mjs b/tests/validate.test.mjs index 06ad907..b201df5 100644 --- a/tests/validate.test.mjs +++ b/tests/validate.test.mjs @@ -569,6 +569,8 @@ test('start_points: optional prelude and join shas', () => { m.start_points = {}; assert.deepEqual(validateManifest(m), []); m.start_points = { prelude: 'abc', join: 'def' }; + // A join start point needs join_started (1.6.5): the session copies both. + m.join_started = false; assert.deepEqual(validateManifest(m), []); m.start_points = { prelude: '' }; assertError(validateManifest(m), 'start_points.prelude'); @@ -770,3 +772,54 @@ test('inputs_changed says whether the plan and spec changed since the last launc 'inputs_changed: must be {plan: true|false|null, spec: true|false|null}'); } }); + +test('lane ids may not start with u-, in any letter case (join unit branches use pl--u-)', () => { + for (const id of ['u-x', 'U-x', 'u-']) { + const m = validManifest(); + renameLane(m, 0, id); + assertError(validateManifest(m), `lane id ${JSON.stringify(id)}`, 'must not start with u-'); + } + for (const id of ['u', 'ux', 'au-x', 'u_x']) { + const m = validManifest(); + renameLane(m, 0, id); + assert.deepEqual(validateManifest(m), [], id); + } +}); + +test('start_points.join without join_started is invalid', () => { + const m = validManifest(); + m.start_points = { join: 'abc' }; + assertError(validateManifest(m), 'join_started', 'missing'); + m.join_started = false; + assert.deepEqual(validateManifest(m), []); + m.join_started = 'yes'; + assertError(validateManifest(m), 'join_started', 'must be true or false'); +}); + +test('unit_branches must name join tasks and unit branches', () => { + const joinId = validManifest().join[0].id; + const ok = validManifest(); + ok.unit_branches = { [joinId]: { branch: 'pl-x-u-J', dir: '/work/wt/lane-a' } }; + assert.deepEqual(validateManifest(ok), []); + for (const [bad, what] of [ + [{ NOPE: { branch: 'pl-x-u-J', dir: '/w' } }, 'not a join task'], + [{ [joinId]: { branch: 'pl-x-a', dir: '/w' } }, 'unit branch'], + [{ [joinId]: { branch: 'pl-x-u-J' } }, 'dir'], + [[], 'must be an object'], + ]) { + const m = validManifest(); + m.unit_branches = bad; + assertError(validateManifest(m), 'unit_branches', what); + } +}); + +test('join_tip and wave_merges are checked', () => { + const m = validManifest(); + m.join_tip = ''; + assertError(validateManifest(m), 'join_tip', 'non-empty string'); + const w = validManifest(); + w.wave_merges = [{ wave: 1, units: ['J'], head: 'abc', overlaps: [], conflict_files: [] }]; + assert.deepEqual(validateManifest(w), []); + w.wave_merges = ['x']; + assertError(validateManifest(w), 'wave_merges', 'list of objects'); +}); diff --git a/tests/waves.test.mjs b/tests/waves.test.mjs new file mode 100644 index 0000000..7e0f022 --- /dev/null +++ b/tests/waves.test.mjs @@ -0,0 +1,220 @@ +// The join wave rule (1.6.5 design 4.1): src/waves.js (the engine's, authoritative) +// and scripts/_waves.py (setup's owner map, worth-it) give the same waves on +// the shared fixtures in tests/fixtures/waves/cases.json. The real cases come +// from the archived d1a, sb1, sc1 and m1a manifests with every file path +// anonymised (f, equality kept). +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { readFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { loadHelpers, SKILL_DIR } from './harness.mjs'; + +const SCRIPTS = join(SKILL_DIR, 'scripts'); +const CASES = JSON.parse(readFileSync(join(SKILL_DIR, 'tests', 'fixtures', 'waves', 'cases.json'), 'utf8')); + +const { + joinWaves, waveDeps, wavePlan, unitBranch, joinUnits, waveWidth, joinStarted, +} = await loadHelpers([ + 'joinWaves', 'waveDeps', 'wavePlan', 'unitBranch', 'joinUnits', 'waveWidth', 'joinStarted', +]); + +// Runs a Python snippet with scripts/ importable (no bytecode); stdin is JSON, +// stdout is JSON. +function python(lines, input) { + const code = [ + 'import json, sys', + 'sys.dont_write_bytecode = True', + `sys.path.insert(0, ${JSON.stringify(SCRIPTS)})`, + 'import _waves', + ...lines, + ].join('\n'); + const r = spawnSync('python3', ['-B', '-c', code], { encoding: 'utf8', input: JSON.stringify(input) }); + assert.equal(r.status, 0, r.stderr); + return JSON.parse(r.stdout); +} + +const task = (id, deps, extra = {}) => ({ + id, title: `Task ${id}`, files: [`src/${id}.cs`], tier: 'standard', security: false, + ...(deps === undefined ? {} : { depends_on: deps.map((d) => ({ id: d, kind: 'code' })) }), ...extra, +}); + +function manifest(overrides = {}) { + return { + run_id: 'run-1', profile: 'standard', + lanes: [{ id: 'a', tasks: [task('A1')] }, { id: 'b', tasks: [task('B1')] }, { id: 'c', tasks: [task('C1')] }], + join: [task('J1', []), task('J2', ['J1']), task('J3', ['J1'])], + limits: { max_parallel_lanes: 3, review_rounds: 5 }, + done: [], reviewed: [], deferred: [], + ...overrides, + }; +} + +test('the JS rule gives every fixture case its expected waves', () => { + for (const c of CASES) assert.deepEqual(joinWaves(c.units, c.edges), c.expected, c.name); +}); + +test('the Python rule gives every fixture case its expected waves', () => { + const got = python([ + 'cases = json.load(sys.stdin)', + 'print(json.dumps([_waves.waves(c["units"], [(e["unit"], e["producer"]) for e in c["edges"]]) for c in cases]))', + ], CASES); + CASES.forEach((c, i) => assert.deepEqual(got[i], c.expected, c.name)); +}); + +test('the real archived joins give the waves the design names', () => { + const sizes = Object.fromEntries(CASES.filter((c) => ['d1a', 'sb1', 'sc1', 'm1a'].includes(c.name)) + .map((c) => [c.name, joinWaves(c.units, c.edges).map((w) => w.length)])); + assert.deepEqual(sizes, { d1a: [1, 2, 4], sb1: [2, 2], sc1: [2, 1, 1, 1], m1a: [2, 1] }); +}); + +test('waveDeps merges a batch\'s dependencies and drops its own ids', () => { + const unit = { id: 'J2-J3', tasks: [task('J2', ['J1', 'L1']), task('J3', ['J2', 'J1'])] }; + assert.deepEqual(waveDeps(unit), ['J1', 'L1']); + assert.deepEqual(waveDeps(task('J1', [])), []); + assert.equal(waveDeps(task('J1')), null); +}); + +test('waveDeps is null when one task of a batch has no depends_on field', () => { + assert.equal(waveDeps({ id: 'J2-J3', tasks: [task('J2', ['J1']), task('J3')] }), null); +}); + +test('joinUnits forms batches as the run does', () => { + const light = (id, deps) => task(id, deps, { tier: 'light', batch: 'b' }); + const m = manifest({ join: [task('J1', []), light('J2', ['J1']), light('J3', ['J1']), task('J4', ['J3'])] }); + assert.deepEqual(joinUnits(m).map((u) => u.id), ['J1', 'J2-J3', 'J4']); + assert.deepEqual(joinWaves(joinUnits(m)), [['J1'], ['J2-J3'], ['J4']]); +}); + +test('wavePlan runs waves when two units can run side by side', () => { + assert.deepEqual(wavePlan(manifest()), { mode: 'waves', reason: null, waves: [['J1'], ['J2', 'J3']] }); +}); + +test('wavePlan says why a join runs serially', () => { + const serial = (m) => wavePlan(m).reason; + assert.equal(serial(manifest({ profile: 'lite' })), 'lite'); + assert.equal(serial(manifest({ join_started: true })), 'join started'); + assert.equal(serial(manifest({ done: ['J1'], reviewed: ['J1'] })), 'join started'); + assert.equal(serial(manifest({ done: ['J1'], deferred: ['J1'] })), 'join started'); + assert.equal(serial(manifest({ lanes: [{ id: 'a', tasks: [task('A1')] }] })), 'fewer than two usable lane worktrees'); + assert.equal(serial(manifest({ + lanes: [{ id: 'a', tasks: [task('A1')] }, { id: 'b', tasks: [task('B1')] }], + lane_commands: { b: { setup: ['make b'] } }, + })), 'fewer than two usable lane worktrees'); + assert.equal(serial(manifest({ limits: { max_parallel_lanes: 1, review_rounds: 5 } })), + 'fewer than two usable lane worktrees'); + assert.equal(serial(manifest({ join: [task('J1'), task('J2'), task('J3')] })), 'no wave has two units'); + for (const m of [manifest({ profile: 'lite' }), manifest({ join_started: true })]) { + assert.equal(wavePlan(m).mode, 'serial'); + assert.deepEqual(wavePlan(m).waves, [['J1'], ['J2', 'J3']]); + } +}); + +test('waveWidth counts lane worktrees without a setup of their own', () => { + assert.equal(waveWidth(manifest()), 3); + assert.equal(waveWidth(manifest({ limits: { max_parallel_lanes: 2, review_rounds: 5 } })), 2); + assert.equal(waveWidth(manifest({ lane_commands: { a: { setup: ['x'] }, b: { test: ['y'] } } })), 2); +}); + +test('joinStarted reads the flag and the join tasks\' state', () => { + assert.equal(joinStarted(manifest()), false); + assert.equal(joinStarted(manifest({ join_started: false })), false); + assert.equal(joinStarted(manifest({ join_started: true })), true); + assert.equal(joinStarted(manifest({ done: ['A1'], reviewed: ['A1'] })), false); + assert.equal(joinStarted(manifest({ done: ['J2'] })), true); +}); + +test('wavePlan takes extra edges', () => { + const plan = wavePlan(manifest(), [{ unit: 'J3', producer: 'J2' }]); + assert.deepEqual(plan, { mode: 'serial', reason: 'no wave has two units', waves: [['J1'], ['J2'], ['J3']] }); +}); + +test('unitBranch names the unit branch', () => { + assert.equal(unitBranch(manifest(), { id: 'DJ3' }), 'pl-run-1-u-DJ3'); + assert.equal(unitBranch(manifest(), { id: 'J2-J3' }), 'pl-run-1-u-J2-J3'); +}); + +test('_waves.task_waves and join_units', () => { + const d1a = CASES.find((c) => c.name === 'd1a'); + const got = python(['tasks = json.load(sys.stdin)', 'print(json.dumps(_waves.task_waves(tasks)))'], d1a.units); + assert.equal(got.DJ5, 3); + assert.equal(got.DJ1, 1); + assert.equal(got.DJ4, 2); + const batch = [{ id: 'J1', files: ['a'], depends_on: [] }, + { id: 'J2', files: ['b'], batch: 'k', depends_on: [{ id: 'J1' }] }, + { id: 'J3', files: ['c'], batch: 'k', depends_on: [{ id: 'J1' }] }, + { id: 'J4', files: ['d'], depends_on: [{ id: 'J3' }] }]; + const units = python(['tasks = json.load(sys.stdin)', 'print(json.dumps(_waves.join_units(tasks)))'], batch); + assert.deepEqual(units.map((u) => [u.id, u.tasks]), [['J1', ['J1']], ['J2-J3', ['J2', 'J3']], ['J4', ['J4']]]); + assert.deepEqual(units[1].depends_on, ['J1']); + const waves = python(['tasks = json.load(sys.stdin)', 'print(json.dumps(_waves.task_waves(tasks)))'], batch); + assert.deepEqual(waves, { J1: 1, J2: 2, J3: 2, J4: 3 }); +}); + +// ---- owners.json: join waves and alongside (design 4.8) ---- + +function owners(lines, input) { + return python(['import _owners', ...lines], input); +} +const ownerManifest = (extra = {}) => ({ + repo: { mode: 'git', root: '/r', git_dir: null }, + prelude: [{ id: 'P1', files: ['p.cs'] }], + lanes: [{ id: 'a', tasks: [{ id: 'A1', files: ['a.cs'] }] }, { id: 'b', tasks: [{ id: 'B1', files: ['b.cs'] }] }], + join: [{ id: 'J1', files: ['j1.cs'], depends_on: [] }, { id: 'J2', files: ['j2.cs'], depends_on: [] }, + { id: 'J3', files: ['j3.cs'], depends_on: [{ id: 'J1' }] }], + ...extra, +}); +const relations = (m) => owners([ + 'm = json.load(sys.stdin)', + 'o = _owners.build(m)', + 'print(json.dumps({"waves": {t: v.get("wave") for t, v in o["tasks"].items() if v["where"] == "join"},', + ' "J1-J2": _owners.relation(o, ["J1"], "J2"), "J2-J1": _owners.relation(o, ["J2"], "J1"),', + ' "J3-J2": _owners.relation(o, ["J3"], "J2"), "J1-J3": _owners.relation(o, ["J1"], "J3"),', + ' "J1-A1": _owners.relation(o, ["J1"], "A1")}))', +], m); + +test('join tasks in one wave are alongside', () => { + const r = relations(ownerManifest()); + assert.deepEqual(r.waves, { J1: 1, J2: 1, J3: 2 }); + assert.equal(r['J1-J2'], 'alongside'); + assert.equal(r['J2-J1'], 'alongside'); + assert.equal(r['J1-A1'], 'earlier', 'a lane is still earlier than the join'); +}); + +test('join tasks in different waves are earlier and later', () => { + const r = relations(ownerManifest()); + assert.equal(r['J3-J2'], 'earlier'); + assert.equal(r['J1-J3'], 'later'); +}); + +test('owners.json without wave keeps the old relation', () => { + const r = owners([ + 'm = json.load(sys.stdin)', + 'o = _owners.build(m)', + 'for t in o["tasks"].values(): t.pop("wave", None)', + 'print(json.dumps([_owners.relation(o, ["J1"], "J2"), _owners.relation(o, ["J2"], "J1")]))', + ], ownerManifest()); + assert.deepEqual(r, ['later', 'earlier']); +}); + +test('a join that cannot run in waves gets no wave in owners.json', () => { + for (const extra of [{ profile: 'lite' }, { lanes: [{ id: 'a', tasks: [{ id: 'A1', files: ['a.cs'] }] }] }]) { + const r = relations(ownerManifest(extra)); + assert.deepEqual(r.waves, { J1: null, J2: null, J3: null }); + assert.equal(r['J1-J2'], 'later'); + } +}); + +test('wave labels are Join phase agents to autopsy, not task agents', () => { + const got = python([ + 'import _transcripts', + 'ids = json.load(sys.stdin)', + 'labels = ["wave 2 merge", "wave 2 resolve J2", "wave checks fix", "wave checks re-review", "wave checks",', + ' "recovery merge", "recovery resolve J2", "J2 implement"]', + 'print(json.dumps([list(_transcripts.parse_label(l, ids)[:2]) for l in labels]))', + ], ['J1', 'J2']); + assert.deepEqual(got, [ + [null, 'wave 2 merge'], [null, 'wave 2 resolve J2'], [null, 'wave checks fix'], [null, 'wave checks re-review'], + [null, 'wave checks'], [null, 'recovery merge'], [null, 'recovery resolve J2'], ['J2', 'implement'], + ]); +}); diff --git a/tests/worth-it-backtest.test.mjs b/tests/worth-it-backtest.test.mjs index 97a4dfe..160032e 100644 --- a/tests/worth-it-backtest.test.mjs +++ b/tests/worth-it-backtest.test.mjs @@ -32,11 +32,14 @@ function archive(runs) { mkdirSync(folder); const lanes = {}; const prelude = []; + const joinTasks = []; for (const t of r.tasks) { if (t.lane === null) prelude.push({ id: t.id }); + // lane 'join': an independent join task (depends_on [], its own file). + else if (t.lane === 'join') joinTasks.push({ id: t.id, depends_on: [], files: [`${t.id}.cs`] }); else (lanes[t.lane] = lanes[t.lane] || []).push({ id: t.id }); } - writeFileSync(join(folder, 'p.lanes.json'), JSON.stringify({ prelude, join: [], + writeFileSync(join(folder, 'p.lanes.json'), JSON.stringify({ prelude, join: joinTasks, lanes: Object.entries(lanes).map(([id, tasks]) => ({ id, tasks })), limits: { max_parallel_lanes: 5 } })); const agents = r.tasks.map((t) => ({ task: t.id, launch: 'wf_1', start: T(r.day, 600), end: T(r.day, 600 + t.s) })); const end = 600 + Math.max(...r.tasks.map((t) => t.s)); @@ -82,3 +85,14 @@ test('the back-test refuses the live history dir', () => { const usage = spawnSync('python3', ['-B', SCRIPT], { encoding: 'utf8', env: env() }); assert.equal(usage.status, 2); }); + +test('--waves counts an independent join as one wave', () => { + const dir = archive([{ id: 'c1', day: 4, project: '/z/r', tasks: [{ id: 'T1', lane: 'a', s: 600 }, + { id: 'T2', lane: 'b', s: 600 }, { id: 'J1', lane: 'join', s: 600 }, { id: 'J2', lane: 'join', s: 600 }] }]); + const serial = spawnSync('python3', ['-B', SCRIPT, dir], { encoding: 'utf8', env: env() }); + assert.equal(serial.status, 0, serial.stderr); + assert.match(serial.stdout, /^c1 .* N\s+4 .* slots\s+3 /m); + const waves = spawnSync('python3', ['-B', SCRIPT, dir, '--waves'], { encoding: 'utf8', env: env() }); + assert.equal(waves.status, 0, waves.stderr); + assert.match(waves.stdout, /^c1 .* N\s+4 .* slots\s+2 /m); +}); diff --git a/tests/worth-it.test.mjs b/tests/worth-it.test.mjs index 7ed0e3c..d1ace62 100644 --- a/tests/worth-it.test.mjs +++ b/tests/worth-it.test.mjs @@ -223,3 +223,38 @@ test('no history dir is fine', () => { const b = run([plan(3), '--max-lanes', '2', '--lane', 'T1', '--lane', 'T2,T3', '--project', '/x/proj', '--history-dir', bad]); assert.deepEqual([b.status, b.json.status, b.json.scale], [0, 'ok', null]); }); + +// ---- 1.6.5: the join counted in waves (--join-deps) ---- + +// T1-T3 one per lane, T4-T10 the join in d1a's shape: waves 1-2-4. +const D1A_DEPS = ['T4:', 'T5:T4', 'T6:T4+T5', 'T7:T4', 'T8:T4+T7', 'T9:T7', 'T10:T7']; +const d1aArgs = (p, deps = D1A_DEPS) => [p, '--max-lanes', '3', '--lane', 'T1', '--lane', 'T2', '--lane', 'T3', + '--join', 'T4,T5,T6,T7,T8,T9,T10', ...deps.flatMap((d) => ['--join-deps', d]), '--history-dir', NOWHERE]; + +test('join waves count ceil(size / lanes)', () => { + const r = run(d1aArgs(plan(10))); + assert.equal(r.status, 0, r.stderr); + assert.equal(r.json.status, 'ok', JSON.stringify(r.json)); + // 1 (the lanes) + waves 1, 2, 4 at 3 at once: 1 + 1 + 2. + assert.equal(r.json.slots, 5); + assert.equal(r.json.saved, 5); + assert.ok(r.json.notes.includes('join waves 1-2-4 (declared files not checked)'), JSON.stringify(r.json.notes)); +}); + +test('no --join-deps keeps a serial join', () => { + const r = run(d1aArgs(plan(10), [])); + assert.equal(r.json.slots, 8); + assert.ok(!r.json.notes.some((n) => n.startsWith('join waves'))); +}); + +test('--join-deps for an unknown id is unavailable', () => { + const r = run(d1aArgs(plan(10), ['T4:', 'T99:T4'])); + assert.equal(r.status, 0); + assert.equal(r.json.status, 'unavailable'); + assert.match(r.json.reason, /T99/); +}); + +test('slots takes join wave sizes', () => { + assert.deepEqual(module(['print(json.dumps([W.slots(0, [1, 1, 1], 7, 3, [1, 2, 4]), W.slots(0, [1, 1, 1], 7, 3)]))']), + [5, 8]); +}); diff --git a/tools/worth-it-backtest.py b/tools/worth-it-backtest.py index 6990f06..2b60195 100755 --- a/tools/worth-it-backtest.py +++ b/tools/worth-it-backtest.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 """The back-test behind scripts/worth-it's THRESHOLD (1.6.4). -usage: python3 -B tools/worth-it-backtest.py HISTORY_COPY_DIR [mean|median] +usage: python3 -B tools/worth-it-backtest.py HISTORY_COPY_DIR [mean|median] [--waves] Reads a COPY of a history dir (index.json and the run folders, each with its autopsy.json and *.lanes.json); it refuses the live history dir @@ -22,6 +22,11 @@ - for thresholds 1 and 2, how often the rule (direct when N - S is at most the threshold) agrees with each run's own comparison (direct = N x T + R against the actual wall, a model, not a measured direct build). +With --waves (1.6.5) the join of a manifest with two lanes or more counts +its waves (_waves, from depends_on and the declared files) as worth-it +--join-deps does: what 1.6.5 would schedule. Archived runs ran their join +serially, so their actual walls and the in-sample errors still describe the +serial join; --waves shows how the rule's verdicts move. Exit 0, or 2 on a usage error or the live history dir. """ import glob @@ -36,13 +41,14 @@ SCRIPTS = os.path.join(os.path.dirname(os.path.dirname(os.path.realpath(__file__))), "scripts") sys.path.insert(0, SCRIPTS) import _history # noqa: E402 +import _waves # noqa: E402 _loader = importlib.machinery.SourceFileLoader("_worth_it", os.path.join(SCRIPTS, "worth-it")) _spec = importlib.util.spec_from_loader("_worth_it", _loader) W = importlib.util.module_from_spec(_spec) _loader.exec_module(W) -USAGE = "usage: worth-it-backtest.py HISTORY_COPY_DIR [mean|median]" +USAGE = "usage: worth-it-backtest.py HISTORY_COPY_DIR [mean|median] [--waves]" def _read_json(path): @@ -50,7 +56,7 @@ def _read_json(path): return json.load(f) -def load_runs(hdir, aggregate): +def load_runs(hdir, aggregate, waves=False): runs = [] for entry in _history.read_index(hdir)["runs"]: folder = _history.run_folder(hdir, entry) @@ -73,11 +79,16 @@ def load_runs(hdir, aggregate): cap = (manifest.get("limits") or {}).get("max_parallel_lanes") or len(lanes) or 1 at_once = max(1, min(len(lanes), cap)) prelude, join = ids(manifest.get("prelude")), ids(manifest.get("join")) + sizes = None + if waves and len(lanes) >= 2 and join: + ran_join = [t for t in manifest.get("join") or [] if isinstance(t, dict) and t.get("id") in ran] + sizes = [len(w) for w in _waves.waves(_waves.join_units(ran_join))] project = (entry.get("project") or "?").replace("\\", "/").rstrip("/").rsplit("/", 1)[-1] runs.append({ "run": entry.get("run_id"), "project": project, "status": entry.get("status"), "date": entry.get("first_launch") or entry.get("folder") or "", - "N": len(ran), "lanes": len(lanes), "slots": W.slots(len(prelude), [len(x) for x in lanes], len(join), at_once), + "N": len(ran), "lanes": len(lanes), + "slots": W.slots(len(prelude), [len(x) for x in lanes], len(join), at_once, sizes), "wall": figures["wall_s"], "F": figures["fixed_s"], "R": figures["final_review_s"], "T": aggregate(ran.values()), }) @@ -136,6 +147,8 @@ def report(runs, label): def main(argv): + waves = "--waves" in argv + argv = [a for a in argv if a != "--waves"] if not argv or len(argv) > 2 or (len(argv) == 2 and argv[1] not in ("mean", "median")): print(USAGE, file=sys.stderr) return 2 @@ -146,7 +159,7 @@ def main(argv): return 2 label = argv[1] if len(argv) == 2 else "mean" try: - runs = load_runs(hdir, statistics.mean if label == "mean" else statistics.median) + runs = load_runs(hdir, statistics.mean if label == "mean" else statistics.median, waves) except _history.HistoryError as e: print("worth-it-backtest: " + str(e), file=sys.stderr) return 2