diff --git a/README.md b/README.md index 52bc225..b01a33a 100644 --- a/README.md +++ b/README.md @@ -269,7 +269,9 @@ session lists it, and you can resume with one word. Finished tasks are skipped. 1. **Assess.** `scripts/derive-lanes` groups the tasks into lanes (at most `min(5, CPU cores + 2)`), a *prelude* (shared groundwork that runs first), and a *join* (tasks that need the merged result). Plans with only 1-2 tasks are handed back to - Superpowers. + Superpowers. `scripts/worth-it` then counts how many task-times running the lanes in + parallel saves; when that is 1 or less (always for a single lane), a direct build is + recommended first, and you can still pick parallel lanes. 2. **Manifest.** Claude writes a JSON manifest covering lanes, model tier per task (`standard`, `sonnet`, or `light`; security-sensitive tasks are always `standard`), project commands, commit rules taken from your CLAUDE.md, and budgets. @@ -365,7 +367,7 @@ Other things to know: | `reference.md` | Manifest fields, lane building, tiers, budgets, recovery details | | `adopt.md` | Adopting earlier work from a hand-run attempt, with a worked example | | `run.workflow.js` | The orchestrator, built from `src/` by `scripts/build` | -| `scripts/` | derive-lanes, setup, ledger, shadow, run-report, active-run, task-checks, run-checks, await-checks, audit, tool-outcomes, stop-cause, and other helpers | +| `scripts/` | derive-lanes, worth-it, setup, ledger, shadow, run-report, active-run, task-checks, run-checks, await-checks, audit, tool-outcomes, stop-cause, and other helpers | | `hooks/` | The SessionStart and notice hooks | | `agents/` | The `parallel-lanes-worker` agent definition | | `tests/` | `node tests/run-tests.mjs` runs every `tests/*.test.mjs`, four files at a time (`--concurrency N` or `PL_TEST_CONCURRENCY` to change) | diff --git a/SKILL.md b/SKILL.md index 71fc7d4..4a84ada 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,6 +1,6 @@ --- name: parallel-lanes -description: Use when an approved implementation plan is about to be executed (at the execution-method handoff), or when resuming a stopped parallel-lanes run. +description: Runs an approved implementation plan as parallel lanes of tasks in separate git worktrees, each task implemented and reviewed by its own agents, then integrated, checked and reviewed as a whole. Use when an approved implementation plan is about to be executed (at the execution-method handoff), or when resuming a stopped parallel-lanes run. --- # Parallel lanes @@ -36,6 +36,7 @@ after the marker is fixed). |---|---| | The skill fires (first output) | `parallel-lanes invoked: evaluating for parallel execution` | | Stepping aside | `parallel-lanes: not a fit (); recommending ` | +| The worth-it check favours a direct build | `parallel-lanes: running in parallel saves about task-times, less than a run's fixed phases; recommending ` | | Launching a new run | `parallel-lanes: launching run : lanes, agents` | | Launching with tasks already done | `parallel-lanes: resuming run : tasks already committed` | | Relaunch after a transient stop | `parallel-lanes: relaunching run after a transient stop` | @@ -67,17 +68,26 @@ lanes, prelude, and join per reference.md "Building lanes" (exit 3: no parseable headings; step aside). Runnable tasks exclude ones the plan marks after-merge, operator, or manual. -- Parallel lanes is the default executor: run as Parallel for any approved plan of 3+ - runnable tasks, a single lane included. +- Parallel lanes is the default executor for any approved plan of 3+ runnable tasks, unless + the worth-it check below recommends a direct build. - Step aside only for a 1-2 task plan or an explicit request for Subagent-driven or Native: print `not a fit ()` (e.g. `2 tasks`), recommending Native when the tasks edit the same files in sequence, else Subagent-driven. Then present the plan's normal execution choices with that recommendation and stop. Run nothing. +- Worth-it check, only when the Offer will be shown (not when the user already chose Parallel + lanes by name, never on Resume): with the layout just built, run + ` /scripts/worth-it --max-lanes --prelude --lane + [--lane ...] --join --project ` (comma-separated task ids, one + `--lane` per lane in lane order; an empty prelude or join is `''`). It prints JSON: + `recommend` is `direct` when running in parallel saves 1 task-time or less (always for a + single-lane plan), else `parallel`; a non-zero exit or any `status` other than `ok` leaves the Offer as it was. + The rule, its evidence and the output: reference.md "Worth it". ## Offer Skip this only if the user already chose parallel lanes by name. Otherwise present these -choices in order, recommend the first, and wait for the pick: +choices, recommend one, and wait for the pick. With `recommend: parallel`, or the worth-it +check unavailable, keep this order and recommend the first: 1. Parallel lanes (recommended) - lanes in separate worktrees, superpowers implementer + reviewer per task, then integration, project checks, and final reviews. ~ agents. @@ -87,6 +97,21 @@ choices in order, recommend the first, and wait for the pick: X is about 2 x runnable tasks + 10; the table gives the exact number. "Go ahead" or "execute the plan" is not a choice of option 1; ask. +Under the choices print `Worth it: these lanes run tasks in about task-slots, saving +about task-times; a run's fixed phases cost about 1-2.` and, when the result has a +`scale`, `For scale: this project's past runs took about min per task (-).` +(minutes, rounded). When the check is unavailable print `Worth it: unavailable ().` +instead. + +With `recommend: direct`: print the worth-it notice first, then put the direct method first and +recommend it (Native when the tasks edit the same files in sequence, else Subagent-driven, as +for a step-aside), Parallel lanes second and the other method third. Under them print the +Worth it line, the scale line if any, and `A direct build still ends with a whole-branch +review; it gives up the run's worktree isolation, integration and delivery checks, E2E and the +multi-lens final review with its verify gate.` This is not a step-aside: if the user picks +Parallel lanes, continue as usual; if they pick the direct method, stop and leave the plan to +that method, as after a step-aside. + ## Building the manifest Fields: reference.md "Manifest fields". In order: diff --git a/VERSION b/VERSION index 266146b..9edc58b 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.6.3 +1.6.4 diff --git a/hooks/bootstrap.md b/hooks/bootstrap.md index c607082..a515772 100644 --- a/hooks/bootstrap.md +++ b/hooks/bootstrap.md @@ -1 +1 @@ -This applies to the main session at the execution-method handoff, never an agent executing a single task. There, parallel-lanes is the default plan executor: when an approved implementation plan is about to be executed, or a stopped parallel-lanes run is to be resumed, invoke the parallel-lanes skill FIRST. It takes precedence over the superpowers Subagent-driven and Native execution options. Superpowers still governs brainstorming, specs, plans, TDD, and debugging, and supplies the per-task review prompts. Use a superpowers execution mode only when parallel-lanes steps aside or the user explicitly asks for one. A multi-agent run still needs the user's explicit yes to the dry-run table. +This applies to the main session at the execution-method handoff, never an agent executing a single task. There, parallel-lanes is the default plan executor: when an approved implementation plan is about to be executed, or a stopped parallel-lanes run is to be resumed, invoke the parallel-lanes skill FIRST. It takes precedence over the superpowers Subagent-driven and Native execution options. Superpowers still governs brainstorming, specs, plans, TDD, and debugging, and supplies the per-task review prompts. Use a superpowers execution mode only when parallel-lanes steps aside, recommends a direct build and the user picks it, or the user explicitly asks for one. A multi-agent run still needs the user's explicit yes to the dry-run table. diff --git a/reference.md b/reference.md index b4950ac..dc28a70 100644 --- a/reference.md +++ b/reference.md @@ -7,6 +7,37 @@ attempt, with a worked example, is in adopt.md. `` below is `${CLAUDE_CONFIG_DIR:-~/.claude}`; every helper uses the same one. `` below is the manifest's `python` (see Paths and Python). +## Contents + +- Paths and Python: finding Python, how every helper command starts, path forms +- Manifest fields: every manifest field, what it holds and its limits +- Flow details: notices, git identity, the confirmation header, transient stops +- Building lanes: turning derive-lanes facts into prelude, lanes and join +- Worth it: whether a layout saves enough to pay for a run's fixed phases +- Profiles: full and lite +- Tiers: standard, sonnet and light tasks, and escalation +- Batching: light tasks run as one unit +- Adjudicator: blocked tasks, rulings and their limits +- Carry-over: rulings, task minors and owners passed between agents +- Fix commits: the message of every commit after a task's first +- Budgets: agent and ruling limits, the repair reserve +- Check ownership: which agent runs which checks, task-checks selection +- Long checks: reserving, starting and waiting on a check +- Exit codes: the busy, timeout and signal codes a check may return +- Active-run markers and stops: launch locks, setup exit 4, stopping a run +- Usage limit: the platform pause and the agents_unavailable stop +- Report: run-report output, the report contents, the run tools line +- Verify evidence: check-verify and the acceptance gates +- Backfill: rebuilding commit ranges in an older ledger +- Shadow repos: runs on a folder that is not a git repo +- Run history: archive-run, the history folder and its index +- Autopsy: where a run's time went, slowdowns and suggestions +- Audit: what the archive proves about each task +- Doctor: the machine checks before a dry run +- Launch conditions: doctor's result bound to each launch +- Cleanup: removing a finished run's branches and worktrees +- Uninstall: removing the skill + ## Paths and Python `python` is the output of `bash /scripts/find-python`: the absolute path of the @@ -47,7 +78,7 @@ PATH, else `C:/Program Files/Git`), never the WSL `bash.exe` in System32. | `commands` | `setup`, `test`, `lint`, `build`: lists of shell commands run from a checkout. Take them from the plan's conventions, then the project (package.json scripts, pyproject, Makefile). Use `[]` for a group the project lacks. | | `lane_commands` | Optional `{: {setup?, test?, lint?, build?}}` overrides for lanes that need only a subset (e.g. backend lanes skip the frontend suite). They are the lane's own checks: integration and verify run every distinct one, so a narrower run of a project check belongs in `task_checks`, not here. | | `task_checks` | Optional `{: {test, lint?, build?}}`: the targeted checks a task runs once before committing (see Check ownership). `test` is required and names at least one command; an absent `lint` or `build` inherits the lane's. A batch uses its last task's entry. Every command must be a narrower run of a check the delivery inventory already runs, in the same checkout and environment; must run correctly at the task that uses it (list the lane's test files created or changed by that task and the lane's earlier tasks, cumulatively, so no entry names a file a later task creates); must fail when it selects no tests (name existing test files; a filter that passes on zero matches needs a wrapper that fails); and keeps an explicit worker limit like the project's test command (for example `node --test --test-concurrency=4 ...`). Derive the entries from the briefs' test files and commands; a task you cannot map gets no entry and runs its lane's checks. Never part of the delivery inventory. | -| `widen_paths` | Optional list of project-relative paths (a trailing `/` marks a directory) of shared infrastructure: test harness and helpers, package and lock files, build and CI config, modules many lanes import. A task whose change touches one runs project-wide checks instead of its targeted ones. Take them from the plan's shared files; no absolute path or `..`. | +| `widen_paths` | Optional list of project-relative paths (a trailing `/` marks a directory; an existing directory named without it counts as one too) of shared infrastructure: test harness and helpers, package and lock files, build and CI config, modules many lanes import. A task whose change touches one runs project-wide checks instead of its targeted ones. Take them from the plan's shared files; no absolute path or `..`. | | `repairs_spent` | Optional integer, default 0: repairs after a failing verify that earlier launches of this run made (the sum of `final.repairs` lengths in their reports; a report without `final` made none). Set it on a transient relaunch or a resume, like the budget carry-over; a run allows 2 in all (see Check ownership). | | `prelude` | Tasks run first on the feature branch, before lanes. | | `lanes` | `[{id, name, setup_note?, tasks}]`. `id` matches `^[A-Za-z0-9_][A-Za-z0-9._-]*$` (it names the ledger file); `name` is the progress phase label. Ids `prelude` and `join` are reserved. No file may appear in two lanes unless an `overlaps` entry records it; files are compared normalized (`./a`, `a//b`, `a/../b`) and case-insensitively. | @@ -96,8 +127,9 @@ Details SKILL.md points to. `` = absolute plan path. `` = `Subagent-driven` or `Native` (superpowers executing-plans). `` = distinct lanes in the dry-run `agents` list, `` = its length, -`` = length of `done`. The dry run returns the launch and resume lines filled in -(`notices`); the script logs the same line; you still print it. +`` = length of `done`. `` = the worth-it result's `saved` (see "Worth it"), and in +that notice `` is the direct method recommended. The dry run returns the launch and +resume lines filled in (`notices`); the script logs the same line; you still print it. ### Git identity @@ -107,6 +139,10 @@ one (`git config --global user.name` and `user.email`) or to name one that `comm passes with `-c` (`git -c user.name=... -c user.email=... commit`). `scripts/setup` does not check this, since a `-c` identity is legitimate. +A repository with SHA-256 object names (`git rev-parse --show-object-format` prints `sha256`) is +not supported: the run names commits by 40-hex SHA-1 ids. `scripts/setup` refuses it (exit 3, +nothing changed); tell the user. + ### Confirmation header The header shown with the dry-run table lists: mode, base and feature branch, worktree_root, @@ -213,6 +249,44 @@ Otherwise: 8. Every plan task the run does not run goes in `excluded` with its reason; `scripts/coverage ` must exit 0 (no plan task missing, no unknown or repeated id). +## Worth it + +` /scripts/worth-it --max-lanes [--prelude ] --lane +[--lane ...] [--join ] [--project ] [--history-dir ]` runs at the +Assessment, after the layout and only when the Offer will be shown. It reads the plan only to +check the layout: every id must be a task heading of the plan and appear once. + +**The rule.** N tasks; A lanes at once = min(lanes, `--max-lanes`). The task-slots S are the prelude +tasks + the lanes' task counts queued whole, in lane order, each to the worker free earliest, +across A workers (as the engine's `runLanes` runs them) + the join tasks. A direct build takes N +task-times, so the layout saves N - S. `recommend` is `direct` when that is at most `THRESHOLD` +(1), else `parallel`. A single-lane layout saves 0. + +**Why 1.** A back-test of 31 archived runs (2026-10-15; `tools/worth-it-backtest.py`, the 1.6.4 +design section 4.3): the fixed phases a run adds beyond the whole-branch review a direct build +also ends with (pre-flight, integrate, E2E, verify) cost a median 1.23 task-times (quartiles 0.92 +to 1.67). With 1 the rule agreed with each run's own comparison on 28 of 31 runs; with 2 on 24. +Minutes were tried and dropped: estimated from a project's earlier runs they were off by -74% to ++1788%, since task time depends on the plan. + +**Biases.** The comparison leans towards a run: no direct build is archived, so "direct" is +modelled; a run's task time includes a per-task review (Native has none) and the contention of +parallel lanes; the run side leaves out the session's own steps (dry run, setup, confirmation, +hand-back). A `direct` recommendation is the safer of the two to trust. The rule assumes the +full profile; a `lite` run's fixed phases are smaller. + +**The scale line.** With `--project`, `scale` holds the mean, lowest and highest task time of +this project's complete archived runs (`--project` matched as `history --project` matches), each +run's from its autopsy.json and manifest (the latest launch that ran task agents; a task's time +is the summed wall time of its agents). Minutes for scale only; it never changes `recommend`. +Runs whose autopsy or manifest cannot be read are skipped, with a note. + +**Output.** One JSON line: `{"status": "ok", "tasks", "lanes", "lanes_at_once", "slots", +"saved", "threshold", "recommend", "scale": {"runs", "task_s_mean", "task_s_min", +"task_s_max"} or null, "notes"}`, or `{"status": "unavailable", "reason", "notes"}` for an +unreadable plan or a layout that names a task twice, names a task the plan lacks, or names none. +Exit 0 in both cases (the Offer goes on as before); 2 on a usage error. Read-only. + ## Profiles - `full` (default): prelude, lane worktrees, an integrate agent, a pre-flight agent, and three @@ -314,7 +388,8 @@ From 1.6.3 what earlier agents decided or found reaches the agents after them, o **The ledger holds every ruling.** A `ruling` event may carry `by` (`implementer`, `reviewer`, `adjudicator`, `preflight`), `scope` (`lane` or `run`), `overrides` (the ids it departs from), -`files` (paths an adjudicator ruling permits outside a task's Files list), `pf` and `wf`. The +`files` (paths an adjudicator ruling permits outside a task's Files list; an implementer lists +the files its ruling explains, which permits nothing), `pf` and `wf`. The ledger checks the values, not whether an agent told the truth. A ruling's id is `-R`, n counting that ledger file's rulings in append order (`rig-R4`, `_run-R2`); this launch's pre-flight rulings are numbered `PF-1`... in the prompts until `carry-launch` records @@ -408,7 +483,7 @@ raised it). need no second table are `setup_result`, `start_points`, and the relaunch carry-over (budgets and `repairs_spent`). - Repairs (see Check ownership) count toward `max_agents`. Each is admitted only while - `repairReserve` agents are free (7, plus 4 with an e2e hook and 2 with a post-integrate hook; + `repairReserve` agents are free (7, plus 4 with an e2e hook and 2 with a post-integrate hook in the full profile; the 7 include one availability probe, see "Usage limit"), so neither the repair nor the checks it makes necessary are ever refused: with less room there is no repair, and the run ends as it would without one. @@ -487,7 +562,12 @@ tool call never ends the agent's turn. worktree never ran the project's setup, and integration still runs every check. Reasons, all recorded in order: `no_entry`, `no_files` (no Files list), `outside_files` (a -changed path the Files list does not cover), `widen_path` (a changed path under +changed path the Files list does not cover, nor an adjudicator ruling in force permits: one filed +under the unit's task, or under `_run` for a path no task's Files list names (neither the path +nor a folder above it), in the folder of a file the unit declares or in a directory it declares +(folders compared by case); a permit names files, and an +entry naming a directory permits nothing; such a path is +listed in `policy.permitted` with the ruling's id), `widen_path` (a changed path under `widen_paths`), `forced` (the agent's `--force-full ''`, for a shared contract, configuration, generated or ignored inputs, a dependency that is not a direct import, or a targeted list that turned out invalid), `git_failed`. Changed paths are the working tree @@ -497,7 +577,8 @@ agent's `--force-full`. Matching is case-sensitive; a `\` in a changed path is n Safety: ids must be safe file names and the base a sha (else exit 2, nothing written); the evidence file `/checks/task-..json` is created exclusively inside the ledger (else exit 3); exit 1 always means a check failed, and any other failure exits 3 with an -error record. The evidence is a record and a count: nothing reads it to accept, skip or reuse +error record. The printed line lists at most 50 changed paths and cuts long reasons, with a `printed_cut` +note; the evidence file keeps everything. The evidence is a record and a count: nothing reads it to accept, skip or reuse checks. **Repair after a failing verify** (autonomous runs only; supervised runs reject as before): @@ -509,7 +590,7 @@ checks. run (`repairs_spent` carries earlier launches'). The reserve is 7 (the repair, its verify and the re-review that must follow, each with one retry, and the availability probe a dead pair among them may start; see "Usage limit"), plus 4 with an e2e hook and 2 with a - post-integrate hook (the rechecks a moved head needs), so nothing the repair makes necessary + post-integrate hook in the full profile (the rechecks a moved head needs), so nothing the repair makes necessary is ever refused. - One Opus agent, `verify repair `: the failing commands with log and tail, the smallest correct fix, focused checks only, `fix: make the delivery checks pass`. It never weakens, @@ -799,7 +880,9 @@ its first log line, `parallel-lanes v loaded`), `agents` (per agent: la `resolved_models` with message counts, effort, input, output, cache read, and cache creation tokens), `tiers` (totals per tier), `totals`, `models` (agents per resolved model), and `unavailable`, `output_incomplete`, `escalations`, `fix_rounds`, `retries` counts. A field -that cannot be read is the string `unavailable`, never a guess. Effort comes from the +that cannot be read is the string `unavailable`, never a guess. With `--out` the full report +goes to FILE and the script prints a summary for the hand-back: the same object without +`agents`, plus `report` (FILE) and `agent_count`. Effort comes from the workflow result file, which the Workflow tool writes to `/workflows/wf_.json` for a transcript directory `/subagents/workflows/wf_/` (`.json` beside the directory is the fallback): its `agent_settings` @@ -1270,6 +1353,15 @@ copies a run that ended after a launch into the run history (never a dry run). R folders already there (imported runs, hand-archived runs). Every folder holding a `*.lanes.json` is one run; the report is the first of `*.report.json`, `*.run-report.json` and `report.json`. Folders keep their names; it prints `{"runs": N, "skipped": [...]}`. +- **Comparing two runs:** `history --runs A,B [--json]` (two run ids or folder names) prints, + for each run, its plan's sha256, task ids, lanes, a hash of its `commands`, versions, launches, + status, acceptance, wall time, fixed phases, task time, full-suite time and tokens, and whether + the pair is comparable: only when the archived plans hash the same, the task ids, lanes and + commands match and both autopsies are current. Otherwise `not comparable` with every reason. + Partial token counts (the usual case: transcripts keep mid-stream output counts) are a note, + not a reason. An out-of-date or missing autopsy (every archived run after an `AUTOPSY_VERSION` + bump) gives a reason ending "(run history first)": a plain `history` run refreshes them. It writes nothing, never reruns autopsy and never prints a + percentage. An id the index holds twice, or the same run twice, exits 2. - **Imported folders are read-only:** do not run their manifests. `--reindex`, `autopsy` and `history` only add their own files (`index.json`, `autopsy.json`, `autopsy.md`, `history.json`, `history.md`) and never change or remove an imported file. @@ -1298,7 +1390,7 @@ agents with `tool_s` and `other_s`, the critical path, idle gaps), `tokens`, `ch `orphan_snapshots` lists the snapshot files that name no archived launch. `autopsy.md` gets a "Launch conditions" section after "Where time went", and the summary one line after the slowdowns when a recorded launch has a warned, failed or unread check. See "Launch conditions". - `AUTOPSY_VERSION` is 6, so `history` runs autopsy again on an older `autopsy.json`. + `AUTOPSY_VERSION` is 7, so `history` runs autopsy again on an older `autopsy.json`. - `tool_s` is the union of an agent's tool-call spans; the rest is `other_s` (model time, transport and scheduling), never called model time. The critical path is a timing-based estimate. A lane is matched by the manifest lane `name`, else `Lane `. @@ -1323,7 +1415,24 @@ agents with `tool_s` and `other_s`, the critical path, idle gaps), `tokens`, `ch - `waits` (`checks_via_procedure`, `wait_calls`, `stopped`, `improvised`) counts the long-check procedure ("Long checks"): executions that reserved, started and waited, the `await-checks wait` calls, those that exited 5, and the improvised waits (`monitor_errors`, `sleep_blocked`, - `harness_output_reads`); `unavailable` for a run without tool events. + `harness_output_reads`); `unavailable` for a run without tool events. `unwaited` lists the + background check starts no `await-checks wait` on their file followed, and `left_running` those + whose last wait said still running (exit 4), each `{agent, file}`: no await-checks wait on its + file followed to the end (the agent may still have read the result another way, after a + completion notice). Claude Code 2.1.300 discourages waiting on background commands, so a + run on it is worth checking here. + - `commands` (1.6.4, `checks.commands`): check time per configured command, from every file in the ledger's + `checks/` with a results list (task checks, integrate, verify, final fix, accept): one entry + `{check, group, runs, total_s, median_s, project_suite}` per configured command that ran, in + check id order; `project_suite` is true for a `commands.` entry (no lane or task + prefix). `commands_unconfigured` (`runs`, `total_s`) counts results whose command the + manifest does not name (a focused rerun), without their text; `commands_skipped` counts + error records, unreadable files and results without a duration. These come from the + ledger, so they are reported for a run without tool events too. + - `widened` (1.6.4): `{targeted_median_s, calls: [{unit, codes, s}]}`: each task-checks + evidence selected `widened` with its reason codes and duration, and the median duration of + the targeted ones. `autopsy.md` shows them as the "Widened task checks" table, each against + the run's median targeted call (measured, not a cost claim). - `repairs` (`attempts`, `eligible`, `by_status`, from `final.repairs`) and `setup` (`skipped`, `no_setup_commands`, from `setup_result.setup_skipped`) are `unavailable` for runs before 1.6.0. @@ -1528,7 +1637,11 @@ Warnings: `memory` (under 4 GB available, or the commit charge near its limit), WindowsApps alias), `heavy_processes` (Windows: AppXSvc over 1500 MB; elsewhere a load average above the CPU count), `active_runs` (another run holds a launch lock on this machine), `defender` (Windows), `concurrency` -(`CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS` unset or lower than the lanes at once) and +(`CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS` unset or lower than the lanes at once), +`host_budget` (the lanes at once, lowered to that variable when it is smaller, exceed the +logical CPU count: "4 lanes at once, 2 CPUs"; expected at the lane cap on a machine with 4 or +fewer CPUs, since the cap is min(5, CPUs + 2); a project whose checks run several workers should +keep lanes x workers within the CPUs) and `usage_limit_pause` (Claude Code may not pause the run at the usage limit; below). A read that fails is a warning with detail `unknown`. @@ -1559,7 +1672,7 @@ Exit 0: no hard failure. Exit 3: at least one hard failure. Exit 2: doctor itsel `--json` prints `{"status": "ok" | "blocked", "failures": [...], "warnings": [...], "skipped": [...], "checks": [...], "observed_at": ..., "doctor_version": ..., "run_id": ..., "usage_limit_pause": {...}}`. -- **`checks`:** one entry `{"check", "outcome"}` for each of the thirteen checks, in run order, +- **`checks`:** one entry `{"check", "outcome"}` for each of the fourteen checks, in run order, with outcome `passed`, `warning`, `failed` or `skipped`. A check that recorded nothing is `skipped` and also appears in `skipped` with reason `not_recorded`; a check recorded more than once keeps its worst outcome (failed, warning, skipped, passed). diff --git a/run.workflow.js b/run.workflow.js index 89dce8e..59b7029 100644 --- a/run.workflow.js +++ b/run.workflow.js @@ -1,6 +1,6 @@ export const meta = { name: 'parallel-lanes', - description: 'parallel-lanes v1.6.3: Execute an implementation plan as parallel lanes of tasks with per-task review, then integrate.', + description: 'parallel-lanes v1.6.4: Execute an implementation plan as parallel lanes of tasks with per-task review, then integrate.', phases: [ { title: 'Pre-flight', detail: 'plan and spec conflicts' }, { title: 'Prelude', detail: 'shared tasks on the feature branch' }, @@ -63,6 +63,8 @@ function agentRules() { 'sed, or inline scripts can be corrupted by shell quoting (apostrophes, backslashes, $ and backticks). ' + "Running the project's own tools that change files is fine: generators (for example scripts/build), " + 'formatters, codemods, and other maintenance commands.', + 'Read a file right before you edit or overwrite it (and again after anything else may have changed it): ' + + 'an Edit or Write on a file not read first, or changed since, fails and costs a retry.', ].join('\n'); } @@ -1391,10 +1393,13 @@ function taskChecksLines(m, task, where, base) { // The line that points a reviewer at the unit's task-check evidence (spec // 4.3). Files are task-..json: the '.' after the id keeps a -// batch id (T2-T3) apart from its first task's (T2). +// batch id (T2-T3) apart from its first task's (T2). A dotted id still +// collides (unit 1's pattern matches unit 1.2's files), so the line names +// policy.unit as the filter. function taskEvidenceLine(m, task) { return [ `Task-check evidence for this unit: ${m.repo.ledger_dir}/checks/task-${task.id}..json (n = 1, 2, ...);`, + `another unit's files can match that pattern, so read only the files whose policy.unit is ${task.id};`, 'each records its selection (policy.selected) and reasons. Do not rerun the checks. Report a wrong selection', 'as a finding (for example targeted checks on a change that clearly reaches another lane).', ].join('\n'); @@ -1585,6 +1590,8 @@ function carryOverLines(m, task, where, by) { 'A file owned by a task in another lane running alongside, or by a later task (or a later task\'s planned part of', 'a file you share), needs a pre-flight or adjudicator ruling in force that lists it; without one, report status', 'question instead of changing it. A file of a task earlier in the run is fine when your task needs the change.', + 'When you change a file outside your Files list, record a ruling saying why with "files": [""] added to its', + 'JSON, listing those files: it permits nothing, it shows reviewers and later tasks what changed outside the list.', 'A check failure you judge outside your task (it passed on a rerun, or the failing test covers code your task did', 'not touch): record it as a ruling whose JSON adds "scope": "run", with the text', 'Observed: failed - - .', @@ -2765,7 +2772,9 @@ function finalReviewFrame(m, intro, focus, minors = [], other = false) { "A finding that a ruling in force settles is not raised unless it is critical or important; then raise it and cite the ruling's id.", 'Files each task changed outside its Files list, with their owners (a read-only command):', ` ${ledgerReadCommand(m, featureDir(m), 'undeclared')}`, - "Check each change was needed and, where it is another lane's or a later task's file, covered by a ruling.", + 'Check each change was needed. A file of a task earlier in the run needs no ruling. A file owned by a task in', + 'another lane running alongside, or a later task, needs a pre-flight or adjudicator ruling in force that covers it', + "(cite its id); an implementer's own ruling never does.", 'Also scan every commit message in the range and the whole diff for anything the commit rules forbid;', 'report each as a finding (for a commit message use file "commit " and line 0).', ...fixMessagesLines(), @@ -5226,7 +5235,7 @@ function noHead(r, label) { // The skill version this script was built from (scripts/build stamps VERSION // here and in the meta description): the version actually running. Logged // first and returned with every result, dry runs included. -const VERSION = '1.6.3'; +const VERSION = '1.6.4'; log(`parallel-lanes v${VERSION} loaded`); if (args !== null && typeof args === 'object' && args.dry_run === true) { diff --git a/scripts/_autopsy.py b/scripts/_autopsy.py index 12858fd..ba28302 100644 --- a/scripts/_autopsy.py +++ b/scripts/_autopsy.py @@ -99,7 +99,7 @@ sys.path.insert(0, os.path.dirname(os.path.realpath(__file__))) import _transcripts # noqa: E402 -AUTOPSY_VERSION = 6 +AUTOPSY_VERSION = 7 UNAVAILABLE = _transcripts.UNAVAILABLE TOKEN_FIELDS = _transcripts.TOKEN_FIELDS IDLE_GAP_S = 60 @@ -1472,6 +1472,8 @@ def _task_checks_args(words): # await-checks wait's exit code for a check that stopped without a result. STOPPED_EXIT = 5 +# await-checks wait's exit code while the check is still running. +STILL_RUNNING_EXIT = 4 # A harness output file an agent may poll: tasks/.output, either slash. _HARNESS_OUTPUT = re.compile(r"(?.output file, with either slash, blocked calls included.""" + tasks/.output file, with either slash, blocked calls included. + + unwaited lists the background check starts (procedure_file) no + 'await-checks wait' on their file followed, and left_running those whose + last wait said the check was still running (exit 4): the agent stopped + waiting before the check ended (1.6.4: Claude Code 2.1.300 discourages + waiting on background commands). Each is {agent, file}.""" if not _has_tool_events(run): return UNAVAILABLE via = waits = stopped = 0 seen = False improvised = {"monitor_errors": 0, "sleep_blocked": 0, "harness_output_reads": 0} + unwaited, left_running = [], [] for a in _all_agents(run): calls = _calls(a) procedures = _procedures(calls) + label = a["entry"]["label"] + for i, c in enumerate(calls): + path = procedure_file(c) + if path is None: + continue + after = _wait_indexes(calls, i, _basename(path)) + if not after: + unwaited.append({"agent": label, "file": path}) + elif calls[after[-1]]["end"] is not None and exit_code(calls[after[-1]]["text"]) == STILL_RUNNING_EXIT: + left_running.append({"agent": label, "file": path}) reserved = False for i, c in enumerate(calls): found = _await_call(c) @@ -1709,7 +1728,8 @@ def waits_section(run): improvised["harness_output_reads"] += 1 if not seen and not _at_least(_engine_version(run), PROCEDURE_VERSION): via = UNAVAILABLE - return {"checks_via_procedure": via, "wait_calls": waits, "stopped": stopped, "improvised": improvised} + return {"checks_via_procedure": via, "wait_calls": waits, "stopped": stopped, "improvised": improvised, + "unwaited": unwaited, "left_running": left_running} # A backslash before a path character (a Windows path separator); shlex @@ -2042,6 +2062,84 @@ def task_check_evidence(run): return entries, malformed +def _median(values): + ordered = sorted(values) + n = len(ordered) + if not n: + return None + return ordered[n // 2] if n % 2 else (ordered[n // 2 - 1] + ordered[n // 2]) / 2.0 + + +def command_costs(run): + """(commands, unconfigured, skipped, widened) from every file in the + ledger's checks/ whose top level holds a results list (task checks, + integrate, verify, final fix, accept): + - commands: [{check, group, runs, total_s, median_s, project_suite}] in + configured_checks order, for each configured command a timed result + ran; a result counts when its command equals the configured one and + its duration_s is a number. project_suite: a commands. entry + (no lane or task prefix). Never command text: check ids only. + - unconfigured: {runs, total_s} of timed results whose command the + manifest does not name (a focused rerun), without their text. + - skipped: files that are error records (an error key and no results) + or unreadable, plus results without a numeric duration_s. + - widened: {targeted_median_s, calls: [{unit, codes, s}]}: the + task-checks evidence (task-*.json, name order) selected widened, each + with its reason codes and the sum of its results' duration_s, and the + median of the targeted ones' sums (UNAVAILABLE without one).""" + configured = configured_checks(run["manifest"]) + by_command = dict((command, check_id) for check_id, command, _full in configured) + times = {} + unconfigured = {"runs": 0, "total_s": 0.0} + skipped = 0 + targeted, widened = [], [] + ledger = find_ledger(run["folder"]) + for path in _files(os.path.join(ledger, "checks"), "*.json") if ledger else []: + try: + with open(path, encoding="utf-8") as f: + data = json.load(f) + except (OSError, ValueError): + skipped += 1 + continue + if not isinstance(data, dict) or not isinstance(data.get("results"), list): + continue + if not data["results"] and "error" in data: + skipped += 1 + continue + total = 0.0 + for r in data["results"]: + s = r.get("duration_s") if isinstance(r, dict) else None + if not _is_num(s): + skipped += 1 + continue + total += s + check_id = by_command.get(r.get("command")) + if check_id is None: + unconfigured["runs"] += 1 + unconfigured["total_s"] += s + else: + times.setdefault(check_id, []).append(s) + policy = data.get("policy") + if _basename(path).startswith("task-") and isinstance(policy, dict) and isinstance(policy.get("unit"), str): + if policy.get("selected") == "targeted": + targeted.append(total) + elif policy.get("selected") == "widened": + codes = [x.get("code") for x in policy.get("reasons") or [] + if isinstance(x, dict) and isinstance(x.get("code"), str)] + widened.append({"unit": policy["unit"], "codes": codes, "s": seconds(total)}) + commands = [] + for check_id, _command, _full in configured: + if check_id in times: + ts = times[check_id] + commands.append({"check": check_id, "group": check_id.split("#")[0].rsplit(".", 1)[-1], + "runs": len(ts), "total_s": seconds(sum(ts)), "median_s": seconds(_median(ts)), + "project_suite": "." not in check_id}) + unconfigured["total_s"] = seconds(unconfigured["total_s"]) + median = _median(targeted) + return commands, unconfigured, skipped, {"targeted_median_s": UNAVAILABLE if median is None else seconds(median), + "calls": widened} + + def _match_evidence(found, evidence): """Give each task-checks execution of found (sorted by start) its evidence. A call with --evidence F (meta evidence_file) gets the entry @@ -2143,11 +2241,14 @@ def checks_section(run): the test group).""" repairs = repairs_section(run) setup = setup_section(run["manifest"]) + commands, unconfigured, skipped, widened = command_costs(run) + costs = {"commands": commands, "commands_unconfigured": unconfigured, "commands_skipped": skipped, + "widened": widened} if not _has_tool_events(run): - return {"executions": UNAVAILABLE, "by_check": UNAVAILABLE, - "full_suite_runs": {"total": UNAVAILABLE, "inferred": UNAVAILABLE, "by_role": UNAVAILABLE}, - "task_check_calls": UNAVAILABLE, "project_test_runs": UNAVAILABLE, - "repairs": repairs, "setup": setup, "waits": UNAVAILABLE} + return dict({"executions": UNAVAILABLE, "by_check": UNAVAILABLE, + "full_suite_runs": {"total": UNAVAILABLE, "inferred": UNAVAILABLE, "by_role": UNAVAILABLE}, + "task_check_calls": UNAVAILABLE, "project_test_runs": UNAVAILABLE, + "repairs": repairs, "setup": setup, "waits": UNAVAILABLE}, **costs) checks = configured_checks(run["manifest"]) on_path = set(id(a) for a in critical_agents(run)) ledger = find_ledger(run["folder"]) @@ -2218,7 +2319,7 @@ def checks_section(run): "inferred": sum(1 for x in executions if x["full_suite"] == INFERRED), "by_role": by_role}, "task_check_calls": calls, "project_test_runs": project, "repairs": repairs, "setup": setup, - "waits": waits_section(run)} + "waits": waits_section(run), **costs} # ---- suggestions ---- @@ -2313,15 +2414,28 @@ def _uncovered(start, end, covers): return pieces +def _procedure_spans(agent): + """[(start, end)] of the agent's procedure start and wait calls that + returned.""" + calls = _calls(agent) + indexes = set() + for start, waits in _procedures(calls).items(): + indexes.add(start) + indexes.update(waits) + return [(calls[i]["start"], calls[i]["end"]) for i in sorted(indexes) if calls[i]["end"] is not None] + + def unexplained_section(run, slowdowns): """The largest TOP_UNEXPLAINED spans over IDLE_GAP_S that no slowdown covers. Each agent's time is cut at its events (a model turn, a tool call); a slowdown covers its own agent's time, and one without an agent - (relaunch, idle gap) covers any agent's.""" + (relaunch, idle gap) covers any agent's. A procedure execution's start + and wait calls (_procedures) cover their own time: it is check time, + counted in the checks section.""" found = [] for a in _timed(_all_agents(run)): covers = merged([(s, e) for sd in slowdowns for owner, s, e in sd["_covers"] - if owner is None or owner is a]) + if owner is None or owner is a] + _procedure_spans(a)) times = sorted(set([a["start"]] + [e["ts"] for e in a["events"]])) for start, end in zip(times, times[1:]): for s, e in _uncovered(start, end, covers): @@ -2918,6 +3032,16 @@ def _md_waits(waits): "await-checks wait; the wait calls, and those that said stopped (exit 5). Unavailable for an " "engine before 1.6.1 that made no await-checks call.") lines.append("") + if isinstance(waits, dict): + for key, title in (("unwaited", "Background checks never waited on"), + ("left_running", "Checks left running when the waits stopped")): + items = waits.get(key) or [] + agents = sorted({i["agent"] for i in items if i.get("agent")}) + lines.append("%s: %d%s" % (title, len(items), " (%s)" % ", ".join(agents) if agents else "")) + lines.append(" Checks started in the background where no await-checks wait on its file followed, and those " + "whose last wait said still running (exit 4): the agent did not wait for the result the " + "procedure way (it may still have read it another way).") + lines.append("") improvised = waits.get("improvised") if isinstance(waits, dict) else None if isinstance(improvised, dict): lines.append("Improvised waits: Monitor errors %d, blocked sleeps %d, harness output reads %d" % ( @@ -2933,7 +3057,8 @@ def _md_waits(waits): def _md_checks(a): checks = a["checks"] if checks["executions"] == UNAVAILABLE: - return ["Check executions: unavailable (the run has no timed tool events).", ""] + _md_counters(checks) + return (["Check executions: unavailable (the run has no timed tool events)."] + + _md_widened(checks.get("widened")) + [""] + _md_counters(checks)) executions = checks["executions"] lines = ["%s; %d on the critical path; %d in the background." % ( _count_word(len(executions), "execution"), sum(1 for x in executions if x["on_critical_path"]), @@ -2946,7 +3071,21 @@ def _md_checks(a): lines += ["", "Check ids name manifest commands: test#1 is commands.test[0], " ".test#1 is lane_commands..test[0]. .task.test#1 is " "task_checks..test[0]."] - return lines + [""] + _md_counters(checks) + return lines + _md_widened(checks.get("widened")) + [""] + _md_counters(checks) + + +def _md_widened(widened): + """The widened task-checks table, or nothing without a widened call.""" + if not isinstance(widened, dict) or not widened.get("calls"): + return [] + median = widened.get("targeted_median_s") + rows = [] + for c in widened["calls"]: + diff = c["s"] - median if _is_num(c["s"]) and _is_num(median) else None + rows.append([c["unit"], ", ".join(c["codes"]) or "-", _secs(c["s"]), _secs(median), + UNAVAILABLE if diff is None else ("+" if diff >= 0 else "-") + _secs(abs(diff))]) + return ["", "Widened task checks, compared with the run's median targeted call (measured):", ""] + _table( + ["Unit", "Reasons", "Duration", "Median targeted", "Difference"], rows) def _md_slowdowns(a): diff --git a/scripts/_doctor.py b/scripts/_doctor.py index f746bc6..dc210ac 100644 --- a/scripts/_doctor.py +++ b/scripts/_doctor.py @@ -70,6 +70,13 @@ compared here and never printed or returned. - concurrency: CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS is unset, or lower than min(limits.max_parallel_lanes, lane count). +- host_budget: the lanes at once (as concurrency counts them, lowered to + CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS when that is a smaller integer) + exceed the logical CPU count (detail " lanes at once, CPUs"); a + count that cannot be read is "unknown". Expected at the skill's lane cap + on a machine with 4 or fewer CPUs (the cap is min(5, CPUs + 2)). A project + whose checks run several workers should keep lanes x workers within the + CPUs. - usage_limit_pause: Claude Code may not pause the run at the usage limit: claude --version (on Windows also claude.exe and claude.cmd) is older than 2.1.271 or cannot be read, or autoContinueAtUsageLimit is false in the @@ -100,7 +107,7 @@ # Every check, in the order run_checks runs them. CHECKS = ("git_identity", "base_not_current", "python", "dirty_checkout", "shadow_repo", "memory", "disk", "python_alias", "heavy_processes", "active_runs", "defender", - "concurrency", "usage_limit_pause") + "concurrency", "host_budget", "usage_limit_pause") # A check recorded more than once keeps the outcome ranked highest. _OUTCOME_RANK = {"passed": 0, "skipped": 1, "warning": 2, "failed": 3} @@ -186,6 +193,7 @@ class Probes: Windows "commit_limit" and "commit_available" in bytes}, or None. disk_free(path) -> the free bytes on path's drive, or None. load_average() -> (1-minute load average, CPU count), or None. + cpu_count() -> the logical CPU count, or None. appx_working_set() -> AppXSvc's working set in bytes (0 when it has no process), or None. Windows only. defender_exclusions() -> Defender's exclusion paths, or None when they @@ -202,7 +210,7 @@ class Probes: def __init__(self, env=None, platform=None, git=None, ls_remote=None, run=None, memory=None, disk_free=None, load_average=None, appx_working_set=None, - defender_exclusions=None, active_runs=None, which=None, read_json=None): + defender_exclusions=None, active_runs=None, which=None, read_json=None, cpu_count=None): self.env = dict(os.environ) if env is None else env self.platform = sys.platform if platform is None else platform self.git = git or self._git @@ -211,6 +219,7 @@ def __init__(self, env=None, platform=None, git=None, ls_remote=None, run=None, self.memory = memory or self._memory self.disk_free = disk_free or self._disk_free self.load_average = load_average or self._load_average + self.cpu_count = cpu_count or os.cpu_count self.appx_working_set = appx_working_set or self._appx_working_set self.defender_exclusions = defender_exclusions or self._defender_exclusions self.active_runs = active_runs or self._active_runs @@ -912,13 +921,19 @@ def check_defender(manifest, probes, report): report.passed(check) -def check_concurrency(manifest, probes, report): - check = "concurrency" +def _lanes_at_once(manifest): + """min(limits.max_parallel_lanes, lane count), the lane count when the + cap is not an integer.""" lanes = manifest.get("lanes") count = len(lanes) if isinstance(lanes, list) else 0 limits = manifest.get("limits") if isinstance(manifest.get("limits"), dict) else {} cap = limits.get("max_parallel_lanes") - at_once = min(cap, count) if isinstance(cap, int) and not isinstance(cap, bool) else count + return min(cap, count) if isinstance(cap, int) and not isinstance(cap, bool) else count + + +def check_concurrency(manifest, probes, report): + check = "concurrency" + at_once = _lanes_at_once(manifest) value = _shell._getenv(probes.env, CONCURRENCY_VARIABLE) if value is None or not value.strip(): report.warn(check, CONCURRENCY_VARIABLE + " is unset, so the Workflow tool's own cap " @@ -936,6 +951,26 @@ def check_concurrency(manifest, probes, report): report.passed(check) +def check_host_budget(manifest, probes, report): + """The lanes running at once (as check_concurrency counts them, lowered + to the concurrency variable when it is a smaller integer) against the + logical CPU count.""" + check = "host_budget" + at_once = _lanes_at_once(manifest) + value = _shell._getenv(probes.env, CONCURRENCY_VARIABLE) + try: + at_once = min(at_once, int(value.strip())) if value and value.strip() else at_once + except ValueError: + pass + cpus = _amount(_read(probes.cpu_count)) + if not cpus: + report.warn(check, "unknown") + elif at_once > cpus: + report.warn(check, "%d lanes at once, %d CPUs" % (at_once, cpus)) + else: + report.passed(check) + + def _claude_version(probes, reasons): """The x.y.z of claude --version, or None (with a reason) when unknown.""" names = ["claude"] @@ -1058,6 +1093,7 @@ def run_checks(manifest, probes): check_active_runs(manifest, probes, report) check_defender(manifest, probes, report) check_concurrency(manifest, probes, report) + check_host_budget(manifest, probes, report) pause = check_usage_limit_pause(probes, report) result = report.result() result["usage_limit_pause"] = pause diff --git a/scripts/_history.py b/scripts/_history.py index afb03da..cf69628 100644 --- a/scripts/_history.py +++ b/scripts/_history.py @@ -41,6 +41,7 @@ """ import contextlib +import datetime import json import os import socket @@ -163,6 +164,143 @@ def resolved(p): return p.startswith(f.rstrip(os.sep) + os.sep) +# ---- selecting runs (shared by scripts/history and scripts/worth-it) ---- + +UNAVAILABLE = "unavailable" + + +def _windows_form(p): + return p[1:3] == ":/" or p.startswith("//") + + +def matches_project(entry, name): + """Whether the entry's project path is name, or ends in the component + name.""" + project = entry.get("project") + if not isinstance(project, str) or not project or project == UNAVAILABLE: + return False + key = _key_form(project) + if key == _key_form(name): + return True + last = key.rstrip("/").rsplit("/", 1)[-1] + return last == (name.lower() if _windows_form(key) else name) + + +def run_folder(hdir, entry): + """The entry's folder as an absolute path inside hdir; None when it has + no usable folder name or the name leads outside hdir.""" + folder = entry.get("folder") + if not isinstance(folder, str) or not folder or os.path.isabs(folder) or _shell._drive_form(folder): + return None + path = os.path.normpath(os.path.join(hdir, folder)) + if os.path.normcase(path) == os.path.normcase(os.path.normpath(hdir)) or not inside(path, hdir): + return None + return path + + +# ---- one run's figures (1.6.4) ---- + +# The engine's phase labels, matched exactly: a lane may be named "Lane verify". +FIXED_PHASES = ("Pre-flight", "Integrate", "E2E", "Final review", "Verify") +FINAL_PHASE = "Final review" + + +def _ts(value): + """Epoch seconds of an ISO 8601 UTC stamp ('...Z'), or None.""" + if not isinstance(value, str): + return None + try: + return datetime.datetime.fromisoformat(value.replace("Z", "+00:00")).timestamp() + except ValueError: + return None + + +def _union(spans): + total, start, end = 0.0, None, None + for s, e in sorted(spans): + if start is None: + start, end = s, e + elif s > end: + total += end - start + start, end = s, e + else: + end = max(end, e) + return total + (end - start if start is not None else 0.0) + + +def _task_ids(manifest): + ids = set() + groups = [manifest.get("prelude"), manifest.get("join")] + lanes = manifest.get("lanes") + groups += [lane.get("tasks") for lane in lanes if isinstance(lane, dict)] if isinstance(lanes, list) else [] + for group in groups: + for task in group if isinstance(group, list) else []: + if isinstance(task, dict) and isinstance(task.get("id"), str): + ids.add(task["id"]) + return ids + + +def latest_task_launch(analysis, manifest): + """(launch, {task id: summed agent wall seconds}) for the latest launch + (by its earliest task-agent start) whose agents include agents of a + manifest task; None when there is none or the data is missing.""" + time_ = analysis.get("time") if isinstance(analysis, dict) else None + agents = time_.get("agents") if isinstance(time_, dict) else None + if not isinstance(agents, list) or not isinstance(manifest, dict): + return None + ids = _task_ids(manifest) + launches = {} + for a in agents: + if not isinstance(a, dict) or a.get("task") not in ids: + continue + if not (a.get("launch") is None or isinstance(a.get("launch"), str)): + continue + start, end = _ts(a.get("start")), _ts(a.get("end")) + if start is None or end is None or end < start: + continue + launches.setdefault(a.get("launch"), []).append((a["task"], start, end)) + if not launches: + return None + launch = max(launches, key=lambda wf: min(s for _, s, _ in launches[wf])) + per_task = {} + for task, start, end in launches[launch]: + per_task[task] = per_task.get(task, 0.0) + (end - start) + return launch, per_task + + +def run_figures(analysis, manifest): + """One run's figures from its autopsy analysis, for the latest launch (by + its earliest task-agent start) whose agents include agents of a manifest + task: {launch, tasks, task_s, fixed_s, final_review_s, wall_s}. task_s is + the mean over that launch's tasks of each task's summed agent wall time + (every agent with that task: implement, review, fix, re-review, + adjudicate); fixed_s the union of its FIXED_PHASES, final_review_s of its + FINAL_PHASE, wall_s the span of its phases. None when no launch ran task + agents or the data is missing.""" + time_ = analysis.get("time") if isinstance(analysis, dict) else None + phases = time_.get("phases") if isinstance(time_, dict) else None + found = latest_task_launch(analysis, manifest) + if found is None or not isinstance(phases, list): + return None + launch, per_task = found + spans = [] + for p in phases: + if isinstance(p, dict) and p.get("launch") == launch: + start, end = _ts(p.get("start")), _ts(p.get("end")) + if start is not None and end is not None and end >= start: + spans.append((p.get("phase"), start, end)) + if not spans: + return None + return { + "launch": launch, + "tasks": len(per_task), + "task_s": round(sum(per_task.values()) / len(per_task), 1), + "fixed_s": round(_union([(s, e) for name, s, e in spans if name in FIXED_PHASES]), 1), + "final_review_s": round(_union([(s, e) for name, s, e in spans if name == FINAL_PHASE]), 1), + "wall_s": round(max(e for _, _, e in spans) - min(s for _, s, _ in spans), 1), + } + + def _empty(): return {"version": VERSION, "runs": []} diff --git a/scripts/doctor b/scripts/doctor index cc45909..2c9e317 100755 --- a/scripts/doctor +++ b/scripts/doctor @@ -14,7 +14,7 @@ the skipped checks. With --json it prints run_checks' result: "run_id": ""|null, "usage_limit_pause": {"pause": "yes"|"no"|"unknown", "reasons": [...], "version": "x.y.z"|null, "setting": true|false|null}} -checks lists all thirteen checks in the order they run, each with the outcome +checks lists all fourteen checks in the order they run, each with the outcome it recorded (see _doctor.py). usage_limit_pause says whether Claude Code will pause the run at the usage limit, as far as doctor can see; it only warns. diff --git a/scripts/history b/scripts/history index 9c06828..f4162e7 100755 --- a/scripts/history +++ b/scripts/history @@ -4,6 +4,7 @@ trends by engine version, and suggestions ranked across runs. usage: python3 history [--history-dir DIR] [--since YYYY-MM-DD] [--project NAME] + python3 history --runs A,B [--json] [--history-dir DIR] The history dir is _history.history_dir(DIR): --history-dir, else PL_HISTORY_DIR, else /parallel-lanes/history. @@ -74,9 +75,37 @@ history.md is plain ASCII: the per-run table, recurring slowdowns, token trends, suggestions and the insufficient-data list. Neither file holds command, output or prompt text. +history --runs A,B compares two runs (each a run id or a folder name) and +writes nothing: it never runs autopsy, so a run whose autopsy is missing or +not current (as above) is reported as such. It prints text, or with --json: + + {"runs": [, ], "comparable": bool, "reasons": [str], "notes": [str]} + +where each run has run_id, folder, plan, plan_sha256 (of the archived plan: +the file in the run folder named as the index entry's plan; unavailable when +absent), task_ids (sorted), lanes (lane ids in order), commands_sha256 (of +the manifest's commands, keys sorted), engine_version, claude_code_version +(the most common version in the transcripts' first lines), launches, +status, acceptance, wall_s, fixed_s, task_s (_history.run_figures), +full_suite_s (the sum of median_s of the project_suite entries of +checks.commands), tokens_total, tokens_partial and autopsy (current, stale +or missing); a figure that cannot be read is "unavailable". The pair is +comparable only when none of these reasons holds, listed in this order: +"plan unavailable for ", "plan differs", "tasks differ", "lanes differ", +"commands differ", "autopsy for (run history first)": +a plain history run brings every autopsy up to date. Partial token counts +are no reason: notes ("tokens partial for : its token totals are lower +bounds") say so. The text starts with "comparable" or "not comparable: ", +then a "note: " line per note and the values of each run side by side; +never a percentage. --runs with +--since or --project, the same run twice, an id the index does not hold or +holds more than once (all its folders are named) exit 2. + Exit 0 on success; 2 on a usage error, a history dir or index.json that cannot be read, or history.json and history.md that cannot be written. """ +import glob +import hashlib import json import os import re @@ -90,11 +119,13 @@ import _history # noqa: E402 import _shell # noqa: E402 UNAVAILABLE = _autopsy.UNAVAILABLE -USAGE = "usage: history [--history-dir DIR] [--since YYYY-MM-DD] [--project NAME]" +USAGE = ("usage: history [--history-dir DIR] [--since YYYY-MM-DD] [--project NAME]\n" + " history --runs A,B [--json] [--history-dir DIR]") SUMMARY_MAX_LINES = 15 TOP_RECURRING = 3 _DAY = re.compile(r"^(\d{4}-\d\d-\d\d)") -_OPTIONS = ("--history-dir", "--since", "--project") +_OPTIONS = ("--history-dir", "--since", "--project", "--runs") +_FLAGS = ("--json",) UNCLEAN_TEXT = ("The checks likely ran on uncommitted changes or with untracked files in the checkout. " + _autopsy.UNCLEAN_TEXT) @@ -107,12 +138,18 @@ class UsageError(Exception): def parse_args(args): - """{option: value} for --history-dir, --since and --project (None when - absent). --since must be a real calendar day.""" - opts = dict.fromkeys(_OPTIONS) + """{option: value} for --history-dir, --since, --project and --runs (None + when absent; --runs becomes its two ids) and --json (True when given). + --since must be a real calendar day; --runs takes no --since or + --project and two different ids; --json goes with --runs.""" + opts = dict.fromkeys(_OPTIONS + _FLAGS) i = 0 while i < len(args): a = args[i] + if a in _FLAGS: + opts[a] = True + i += 1 + continue if a not in _OPTIONS: raise UsageError("unknown argument " + a) if i + 1 >= len(args) or not args[i + 1] or args[i + 1] in _OPTIONS: @@ -127,6 +164,17 @@ def parse_args(args): ok = False if not ok: raise UsageError("--since needs a date as YYYY-MM-DD, not " + since) + if opts["--runs"] is not None: + if opts["--since"] is not None or opts["--project"] is not None: + raise UsageError("--runs compares two runs; it takes no --since or --project") + pair = [x.strip() for x in opts["--runs"].split(",")] + if len(pair) != 2 or not all(pair): + raise UsageError("--runs needs two runs, as A,B") + if pair[0] == pair[1]: + raise UsageError("--runs names the same run twice") + opts["--runs"] = pair + elif opts["--json"]: + raise UsageError("--json goes with --runs") return opts @@ -149,21 +197,7 @@ def run_date(entry): return UNAVAILABLE -def _windows_form(p): - return p[1:3] == ":/" or p.startswith("//") - - -def matches_project(entry, name): - """Whether the entry's project path is name, or ends in the component - name.""" - project = entry.get("project") - if not isinstance(project, str) or not project or project == UNAVAILABLE: - return False - key = _history._key_form(project) - if key == _history._key_form(name): - return True - last = key.rstrip("/").rsplit("/", 1)[-1] - return last == (name.lower() if _windows_form(key) else name) +matches_project = _history.matches_project def select(entries, since, project): @@ -179,16 +213,7 @@ def select(entries, since, project): return out -def run_folder(hdir, entry): - """The entry's folder as an absolute path inside hdir; None when it has - no usable folder name or the name leads outside hdir.""" - folder = entry.get("folder") - if not isinstance(folder, str) or not folder or os.path.isabs(folder) or _shell._drive_form(folder): - return None - path = os.path.normpath(os.path.join(hdir, folder)) - if os.path.normcase(path) == os.path.normcase(os.path.normpath(hdir)) or not _history.inside(path, hdir): - return None - return path +run_folder = _history.run_folder # ---- autopsies ---- @@ -568,6 +593,129 @@ def render_summary(history, hdir, redone, md_path, opts): return "\n".join(lines) + "\n" +# ---- history --runs A,B ---- + +def resolve_run(index, token): + """The one index entry whose run_id or folder is token; UsageError when + there is none or more than one (naming their folders).""" + token = token.rstrip("/\\") + hits = [e for e in index["runs"] + if e.get("run_id") == token or (isinstance(e.get("folder"), str) and e["folder"].rstrip("/\\") == token)] + if not hits: + raise UsageError("no run " + token + " in the index") + if len(hits) > 1: + raise UsageError("run " + token + " is in the index more than once: " + + ", ".join(str(e.get("folder")) for e in hits)) + return hits[0] + + +def _sha256_file(path): + try: + with open(path, "rb") as f: + return hashlib.sha256(f.read()).hexdigest() + except OSError: + return UNAVAILABLE + + +def _claude_code_version(folder, lines_per_file=50): + """The most common 'version' in the first lines of the run's agent + transcripts, else UNAVAILABLE.""" + counts = {} + for wf in _autopsy.wf_dirs(folder): + for path in sorted(glob.glob(os.path.join(glob.escape(wf), "agent-*.jsonl"))): + try: + with open(path, encoding="utf-8", errors="replace") as f: + for n, line in enumerate(f): + if n >= lines_per_file: + break + try: + version = json.loads(line).get("version") + except (ValueError, AttributeError): + continue + if isinstance(version, str) and version: + counts[version] = counts.get(version, 0) + 1 + except OSError: + continue + return max(sorted(counts), key=lambda v: counts[v]) if counts else UNAVAILABLE + + +def run_card(hdir, entry): + """One run's values for --runs (see the usage text).""" + folder = run_folder(hdir, entry) + data = _read_json(os.path.join(folder, "autopsy.json")) if folder else None + if not isinstance(data, dict): + data, state = {}, "missing" + else: + state = "current" if current_autopsy(folder) is not None else "stale" + manifest_path = _autopsy.find_manifest(folder) if folder and os.path.isdir(folder) else None + manifest = _read_json(manifest_path) if manifest_path else None + manifest = manifest if isinstance(manifest, dict) else {} + plan = entry.get("plan") if _known(entry.get("plan")) else UNAVAILABLE + plan_name = plan.replace("\\", "/").rsplit("/", 1)[-1] if plan != UNAVAILABLE else "" + plan_sha = _sha256_file(os.path.join(folder, plan_name)) if folder and plan_name else UNAVAILABLE + lanes = manifest.get("lanes") if isinstance(manifest.get("lanes"), list) else None + ids = sorted(_history._task_ids(manifest)) if manifest else UNAVAILABLE + commands = manifest.get("commands") + row = build_row(entry, data) + figures = _history.run_figures(data, manifest) or {} + costs = _dig(data, "checks", "commands") + suite = [c.get("median_s") for c in costs if isinstance(c, dict) and c.get("project_suite")] \ + if isinstance(costs, list) else [] + return { + "run_id": row["run_id"], "folder": row["folder"], "plan": plan, "plan_sha256": plan_sha, + "task_ids": ids, + "lanes": [lane.get("id") for lane in lanes if isinstance(lane, dict)] if lanes is not None else UNAVAILABLE, + "commands_sha256": hashlib.sha256(json.dumps(commands, sort_keys=True).encode("utf-8")).hexdigest() + if commands is not None else UNAVAILABLE, + "engine_version": row["engine_version"], + "claude_code_version": _claude_code_version(folder) if folder and os.path.isdir(folder) else UNAVAILABLE, + "launches": entry.get("launches", UNAVAILABLE), "status": entry.get("status", UNAVAILABLE), + "acceptance": row["acceptance"], + "wall_s": figures.get("wall_s", UNAVAILABLE), "fixed_s": figures.get("fixed_s", UNAVAILABLE), + "task_s": figures.get("task_s", UNAVAILABLE), + "full_suite_s": round(sum(suite), 1) if suite and all(_num(x) for x in suite) else UNAVAILABLE, + "tokens_total": row["tokens_total"], "tokens_partial": row["tokens_partial"], "autopsy": state, + } + + +def compare_runs(cards): + """{runs, comparable, reasons} for two run cards.""" + a, b = cards + reasons = ["plan unavailable for " + c["run_id"] for c in cards if c["plan_sha256"] == UNAVAILABLE] + if not reasons and a["plan_sha256"] != b["plan_sha256"]: + reasons.append("plan differs") + for key, reason in (("task_ids", "tasks differ"), ("lanes", "lanes differ"), ("commands_sha256", "commands differ")): + if a[key] == UNAVAILABLE or a[key] != b[key]: + reasons.append(reason) + reasons += ["autopsy %s for %s (run history first)" % (c["autopsy"], c["run_id"]) + for c in cards if c["autopsy"] != "current"] + # Partial token counts are the norm (transcripts keep mid-stream output counts): a note, + # never a reason, or no real pair would be comparable. + notes = ["tokens partial for %s: its token totals are lower bounds" % c["run_id"] + for c in cards if c["tokens_partial"] is True] + return {"runs": cards, "comparable": not reasons, "reasons": reasons, "notes": notes} + + +def render_runs(result): + """The text form: the verdict, then each value of both runs.""" + lines = ["comparable" if result["comparable"] else "not comparable: " + "; ".join(result["reasons"])] + lines += ["note: " + n for n in result.get("notes", [])] + a, b = result["runs"] + for key in a: + show = lambda v: ", ".join(str(x) for x in v) if isinstance(v, list) else str(v) # noqa: E731 + lines.append("%s: %s | %s" % (key, show(a[key]), show(b[key]))) + return "\n".join(_ascii(line) for line in lines) + "\n" + + +def runs_main(hdir, index, pair, as_json): + entries = [resolve_run(index, token) for token in pair] + if entries[0] is entries[1]: + raise UsageError("--runs names the same run twice") + result = compare_runs([run_card(hdir, e) for e in entries]) + sys.stdout.write(json.dumps(result, indent=2) + "\n" if as_json else render_runs(result)) + return 0 + + def main(argv): _shell.setup_io() try: @@ -585,6 +733,12 @@ def main(argv): except _history.HistoryError as e: print("history: " + str(e), file=sys.stderr) return 2 + if opts["--runs"] is not None: + try: + return runs_main(hdir, index, opts["--runs"], bool(opts["--json"])) + except UsageError as e: + print("history: " + str(e), file=sys.stderr) + return 2 selected = select(index["runs"], opts["--since"], opts["--project"]) analyses = [] redone = 0 diff --git a/scripts/ledger b/scripts/ledger index c6f23b0..8880e0e 100755 --- a/scripts/ledger +++ b/scripts/ledger @@ -75,7 +75,8 @@ entries of tasks done in this launch and the carried ids this launch's final review raised, plus each task done in this launch's minor_findings less the ids the final review raised (none without one). Prints {"carried", "rulings", "open_minors"}, or {"carried", "already": true}; exit 3 and nothing written -when no readable result is found. +when no readable result is found or DIR does not exist. Its events go to +_run.jsonl in one write. minors prints the latest snapshot: '[] : - (fix: )' per minor, or '(none)'. write-owners writes DIR/owners.json @@ -146,9 +147,10 @@ head) that is not in its files, nor in the files of a task with the same range unquoted. owner is the first other task whose files list the path (null when none does) and relation when that task runs: earlier, alongside or later. minors: {"open": [ids], "covered": [task ids]} from the latest minors event -(empty lists without one). overrides: [{id, by, task, overrides, unknown}] for -each ruling that names others it departs from (resolved; unknown lists the -ones that name no ruling). +(empty lists without one). overrides: [{id, by, task, overrides, unknown, +targets}] for each ruling that names others it departs from (resolved; unknown +lists the ones that name no ruling; targets gives each known one's id, role +and lane, the ledger file it is in). A missing DIR has empty lists, no start points, and no carry. accept appends {"task": "_run", "event": "accepted", "head": , @@ -364,11 +366,19 @@ def run_only_error(entry): def write_entry(directory, lane, entry): + append_entries(directory, lane, [entry]) + + +def append_entries(directory, lane, entries): + """Append entries to DIR/.jsonl in one write: every entry is + validated first, so an invalid one writes none, and a reader never sees + half of them (carry-launch's rulings, snapshot and marker).""" if not LANE.fullmatch(lane): fail(3, f"unsafe lane name {lane!r}") - error = entry_error(entry) - if error: - fail(2, error) + for entry in entries: + error = entry_error(entry) + if error: + fail(2, error) # Parallel lanes may make their first append at the same moment, so the # directory may appear between any check and create. Only the process that # creates it sets the mode (makedirs honors umask); the rest just append. @@ -379,7 +389,7 @@ def write_entry(directory, lane, entry): fail(3, f"ledger path {directory!r} exists and is not a directory") else: os.chmod(directory, 0o700) - line = json.dumps(entry, ensure_ascii=True, separators=(",", ":")) + "\n" + line = "".join(json.dumps(e, ensure_ascii=True, separators=(",", ":")) + "\n" for e in entries) path = os.path.join(directory, f"{lane}.jsonl") fd = os.open(path, os.O_WRONLY | os.O_APPEND | os.O_CREAT, 0o600) try: @@ -723,9 +733,12 @@ def ruling_line(r, targets): return line -def rulings(directory, lane=None): - """Print the rulings in force, then the overridden ones. With lane: the - rulings of _run, the prelude and that lane, and every run-scope one.""" +def ruling_view(directory, lane=None): + """(shown records, {id: [(target, state)]}, {overridden id: by id}): the + rulings shown (with lane: those of _run, the prelude and that lane, and + every run-scope one), each one's override targets, and the overrides that + take effect: within the rules, and only when the ruling that makes it is + shown too.""" records = ruling_records(directory) by_id = {r["id"]: r for r in records} overrides = resolve_overrides(records, carried_events(directory)) @@ -733,8 +746,6 @@ def rulings(directory, lane=None): def shown(r): return lane is None or r["stem"] in ("_run", "prelude", lane) or r["scope"] == "run" - # An override takes effect only within the rules, and in this view only - # when the ruling that makes it is shown here too. marked = {} overridden_by = {} for r in records: @@ -745,9 +756,66 @@ def rulings(directory, lane=None): marked[r["id"]].append((target, state)) if state == "known" and shown(r): overridden_by.setdefault(target, r["id"]) + return [r for r in records if shown(r)], marked, overridden_by + + +def adjudicator_permits(directory, task_ids, declared=()): + """[(path, ruling id)]: the files the adjudicator rulings in force permit + for a unit of task_ids that declares the files declared: a ruling filed + under one of task_ids, or under _run for a path that neither it nor a + folder above it is named by a task's Files list in /owners.json + (none of _run's without a readable owner map) and that sits in the folder + of a declared file, or in a declared directory itself (_folder; folders + compare by case). A permit names files only. Implementer, reviewer + and pre-flight rulings permit nothing. task-checks counts these paths as + declared.""" + shown, _marked, overridden_by = ruling_view(directory) + try: + owned = set(_owners.load(owners_path(directory))["files"]) + except (OSError, ValueError): + owned = None + folders = {_folder(d) for d in declared if _owners.path_key(d) is not None} + out = [] + for r in shown: + e = r["event"] + if r["role"] != "adjudicator" or r["id"] in overridden_by: + continue + for path in e.get("files") or []: + key = _owners.path_key(path) + # A permit names files: a directory ("src/", "./") would cover every path under it. + if not isinstance(path, str) or key in (None, ".") or path.endswith(("/", "\\")): + continue + if e.get("task") in task_ids or (e.get("task") == "_run" and owned is not None + and not _owned_or_under(key, owned) + and _folder(path, as_file=True) in folders): + out.append((path, r["id"])) + return out + - in_force = [ruling_line(r, marked[r["id"]]) for r in records if shown(r) and r["id"] not in overridden_by] - gone = [f"{r['id']} by {overridden_by[r['id']]}" for r in records if shown(r) and r["id"] in overridden_by] +def _folder(path, as_file=False): + """The folder a declared path names, compared by case: a directory + entry ('src/') is its own folder, a file's is its parent; '' is the + project root. With as_file, path is always read as a file.""" + p = path.replace("\\", "/") + folder = posixpath.normpath(p) if p.endswith("/") and not as_file else posixpath.dirname(posixpath.normpath(p)) + return "" if folder == "." else folder + + +def _owned_or_under(key, owned): + """Whether the path key, or a folder above it, is in a Files list.""" + while key: + if key in owned: + return True + key = posixpath.dirname(key) + return False + + +def rulings(directory, lane=None): + """Print the rulings in force, then the overridden ones. With lane: the + rulings of _run, the prelude and that lane, and every run-scope one.""" + records, marked, overridden_by = ruling_view(directory, lane) + in_force = [ruling_line(r, marked[r["id"]]) for r in records if r["id"] not in overridden_by] + gone = [f"{r['id']} by {overridden_by[r['id']]}" for r in records if r["id"] in overridden_by] lines = in_force or ["(none)"] if gone: lines += ["Overridden:"] + gone @@ -831,10 +899,13 @@ def overrides_list(directory): overrides, unknown}], overrides resolved (PF-n through carry-launch), unknown the ids that resolve to no ruling.""" records = ruling_records(directory) + by_id = {r["id"]: r for r in records} resolved = resolve_overrides(records, carried_events(directory)) return [{"id": r["id"], "by": r["role"], "task": r["event"]["task"], "overrides": [t for t, _ in resolved[r["id"]]], - "unknown": [t for t, state in resolved[r["id"]] if state == "unknown"]} + "unknown": [t for t, state in resolved[r["id"]] if state == "unknown"], + "targets": [{"id": t, "role": by_id[t]["role"], "lane": by_id[t]["stem"]} + for t, state in resolved[r["id"]] if state == "known"]} for r in records if resolved[r["id"]]] @@ -889,7 +960,7 @@ def write_owners(directory, manifest_path): os.replace(tmp, owners_path(directory)) except OSError as e: fail(3, f"cannot write {owners_path(directory)!r}: {e}") - print(json.dumps({"owners": owners_path(directory), "files": len(data["files"])})) + print(json.dumps({"owners": owners_path(directory).replace("\\", "/"), "files": len(data["files"])})) def owners(directory, unit_ids, paths): @@ -1033,6 +1104,8 @@ def carry_launch(directory, source): """Record what a launch knew that the ledger does not: pre-flight's own rulings (by preflight, PF-n), the open task minors, and which PF-n overrides recorded during the launch name. Once per wf id.""" + if not os.path.isdir(directory): + fail(3, f"no ledger dir {directory!r}") path = find_launch_file(source) try: data = json.loads(read_text(path, "launch result")) if path else None @@ -1067,26 +1140,27 @@ def carry_launch(directory, source): # adjudicator ruling in _run (a supervised stop) there was none. adjudicated = len(texts) - 1 pf = {} - written = 0 + entries = [] + run_rulings = len([r for r in ruling_records(directory) if r["stem"] == "_run"]) for i, text in enumerate(texts): key = f"PF-{i + 1}" if isinstance(adjudicated, int) and not isinstance(adjudicated, bool) and i == adjudicated: if adj_ids: pf[key] = adj_ids[-1] continue - write_entry(directory, "_run", {"task": "_run", "event": "ruling", "text": text, "by": "preflight", - "scope": "run", "pf": key, "wf": wf}) - written += 1 - pf[key] = [r["id"] for r in ruling_records(directory) if r["stem"] == "_run"][-1] + entries.append({"task": "_run", "event": "ruling", "text": text, "by": "preflight", + "scope": "run", "pf": key, "wf": wf}) + pf[key] = f"_run-R{run_rulings + len(entries)}" + written = len(entries) bound_before = {(b.get("file"), b.get("line")) for c in carried_events(directory) for b in c.get("bound") or [] if isinstance(b, dict)} bound = [{"file": r["file"], "line": r["line"]} for r in ruling_records(directory) if any(PF_ID.fullmatch(o) for o in r["event"].get("overrides") or []) and (r["file"], r["line"]) not in bound_before] open_now, covered = open_minors(latest_minors(events), result, data.get("args")) - write_entry(directory, "_run", {"task": "_run", "event": "minors", "wf": wf, "open": open_now, - "covered": covered}) - write_entry(directory, "_run", {"task": "_run", "event": "carried", "wf": wf, "pf": pf, "bound": bound}) + entries.append({"task": "_run", "event": "minors", "wf": wf, "open": open_now, "covered": covered}) + entries.append({"task": "_run", "event": "carried", "wf": wf, "pf": pf, "bound": bound}) + append_entries(directory, "_run", entries) print(json.dumps({"carried": wf, "rulings": written, "open_minors": len(open_now)})) diff --git a/scripts/run-report b/scripts/run-report index 529ee90..afd4b57 100755 --- a/scripts/run-report +++ b/scripts/run-report @@ -6,8 +6,9 @@ usage: TRANSCRIPT_DIR holds agent-*.meta.json (description, workflowPhase, optional model) and agent-*.jsonl (resolved message.model and usage per line). -MANIFEST is the run manifest, used to recognise task ids. Prints JSON (or -writes it to FILE): +MANIFEST is the run manifest, used to recognise task ids. Prints JSON, or +with --out writes it to FILE and prints a summary for the hand-back: the +same object without agents, as {report: FILE as given, agent_count: N, ...}: {engine_version: "x.y.z", agents: [...], tiers: {: {...}}, totals: {...}, models: {...}, unavailable: N, output_incomplete: N, @@ -77,7 +78,7 @@ def main(argv): if i + 1 >= len(args): print("run-report: --out needs a file", file=sys.stderr) return 2 - out = args[i + 1] + out = out_given = args[i + 1] del args[i:i + 2] if len(args) != 2: print("usage: run-report TRANSCRIPT_DIR MANIFEST [--out FILE]", file=sys.stderr) @@ -98,9 +99,13 @@ def main(argv): except (OSError, ValueError) as e: print(f"run-report: cannot read manifest: {e}", file=sys.stderr) return 2 - text = json.dumps(_transcripts.build_report(transcript_dir, manifest if isinstance(manifest, dict) else {}), indent=2) + "\n" + report = _transcripts.build_report(transcript_dir, manifest if isinstance(manifest, dict) else {}) + text = json.dumps(report, indent=2) + "\n" if out: _shell.write_text(out, text) + rest = {k: v for k, v in report.items() if k != "agents"} + summary = {"report": out_given, "agent_count": len(report["agents"]), **rest} + sys.stdout.write(json.dumps(summary, indent=2) + "\n") else: sys.stdout.write(text) return 0 diff --git a/scripts/setup b/scripts/setup index fb99d90..894519b 100755 --- a/scripts/setup +++ b/scripts/setup @@ -7,6 +7,9 @@ usage: MANIFEST is the run manifest (JSON). setup does what the Setup agent of run.workflow.js does, without an agent: +A SHA-256 repository (git rev-parse --show-object-format) is refused: exit 3, +nothing changed; the run names commits by 40-hex SHA-1 ids. + git mode (the main checkout repo.root is the feature checkout): refuses a main checkout with uncommitted changes (exit 3, nothing created; its changes are never discarded), creates repo.branch from repo.base_ref @@ -452,6 +455,13 @@ def setup(m, owner=None): feature = feature_dir(m) # Every refusal comes before the first change. + # The run names commits by 40-hex SHA-1 ids (isSha): a SHA-256 repository + # would fail partway through. Only an explicit sha256 answer refuses, so + # an old git that does not know the option is not refused by mistake. + object_format = repo.git("rev-parse", "--show-object-format", check=False).stdout.strip() + if object_format == "sha256": + raise Failure(3, "the repository uses SHA-256 object names; parallel-lanes supports only SHA-1 " + "repositories (nothing was changed)") if repo_cfg["mode"] == "git": dirty = run_git(["-C", feature, "status", "--porcelain"]).stdout if dirty: diff --git a/scripts/task-checks b/scripts/task-checks index d79e4d4..f897763 100755 --- a/scripts/task-checks +++ b/scripts/task-checks @@ -19,7 +19,7 @@ Selection (policy.selected): targeted an entry and no widening reason: the --targeted list runs. policy.reasons lists every reason that applies, as {code, detail}, in this order: no_entry, no_files (no --declared), outside_files (a changed path the -Files list does not cover), widen_path (a changed path a --widen entry +Files list does not cover, nor a ruling permits), widen_path (a changed path a --widen entry covers), forced (--force-full, its REASON as detail), git_failed (the changed paths could not be listed). @@ -30,12 +30,25 @@ ignored), NUL-delimited, unique and sorted. Changes to ignored files are not seen: widening for them is the agent's --force-full. Coverage: '\\' becomes '/'; an entry ending in '/' is a directory and covers -every path under it; any other entry covers the equal path. Paths are +every path under it; any other entry covers the equal path, except a --widen +entry that names an existing directory in DIR, which covers it as if it ended +in '/' (widening is the safe side; --declared keeps the strict rule). Paths are normalized (posixpath.normpath) and compared case-sensitively, so on a case-insensitive filesystem a case mismatch only widens. A --declared or --widen value that is absolute or leaves the project after normalizing is a usage error. +Permitted paths: a changed path outside the Files list that an adjudicator +ruling in force permits this unit (its 'files', file paths only: an entry +naming a directory permits nothing; filed under one of the --task ids, or +under _run for a path that neither it nor a folder above it is named by a +task's Files list in LEDGER_DIR/owners.json, in the folder of a --declared +file or in a declared directory itself, folders compared by case) counts as declared and is listed in policy.permitted as +{path, ruling}. Implementer, reviewer and pre-flight rulings permit nothing. +The ledger script's adjudicator_permits finds them. When the ledger cannot +be read none are permitted; when the owner map cannot be read, none of +_run's are. + The chosen list runs through run-checks (DIR --out FILE --root LEDGER_DIR --lock-held). FILE, the result file, is: - with --evidence PATH: PATH, reserved beforehand by await-checks reserve @@ -56,11 +69,14 @@ never takes run-checks' report, written before policy is added, for the result. run-checks writes its JSON to FILE and its logs beside it; this helper adds policy: {unit, tasks, lane, base, started, entry, selected, reasons, changed, - offered: {targeted, lane, wide}, project_test_executed} + permitted, offered: {targeted, lane, wide}, project_test_executed} started is the UTC time this helper began (ISO 8601, milliseconds, Z); project_test_executed is true when an executed command equals a --project-test value. The file is rewritten in one step -(_shell.write_text_atomic) and printed as one JSON line. When run-checks +(_shell.write_text_atomic) and printed as one JSON line. The printed copy +lists at most 50 of policy.changed and cuts each reason's detail at 2000 +characters; when it cut anything it adds printed_cut, a note naming FILE, +which keeps everything. When run-checks prints no JSON (a crash, or its exit 3), the file holds {"results": [], "ok": false, "error": , "policy": ...}, printed too. @@ -88,6 +104,9 @@ error record once FILE is known). Exit 1 always means a check failed. """ import contextlib +import importlib.machinery +import importlib.util +import io import json import os import posixpath @@ -109,6 +128,8 @@ ENV = {k: v for k, v in os.environ.items() if k not in GIT_REDIRECTS} SAFE_ID = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9._-]*$") SHA = re.compile(r"^[0-9a-f]{7,40}$") ERROR_TAIL = 2000 +PRINT_PATHS = 50 +PRINT_DETAIL = 2000 USAGE = ("usage: task-checks DIR --base SHA --ledger LEDGER_DIR --unit UNIT_ID --lane LANE " "--task ID [...] [--declared PATH ...] [--widen PATH ...] [--project-test CMD ...] " "[--entry TASK_ID --targeted GROUP CMD ...] --lane-cmd GROUP CMD [...] " @@ -232,6 +253,7 @@ def validate(args): for value in [args["unit"], args["lane"], *args["tasks"]] + ([args["entry"]] if args["entry"] else []): if not SAFE_ID.match(value): fail(2, f"unsafe id {value!r}") + args["declared_raw"] = list(args["declared"]) for flag, key in (("--declared", "declared"), ("--widen", "widen")): keys = [] for value in args[key]: @@ -242,6 +264,9 @@ def validate(args): args[key] = keys args["dir"] = _shell.native_path(args["dir"]) args["ledger"] = _shell.native_path(args["ledger"]) + # A --widen entry naming an existing directory covers it without the '/': + # widening is the safe side. --declared keeps the strict rule. + args["widen"] = [(e, d or os.path.isdir(os.path.join(args["dir"], e))) for e, d in args["widen"]] def git_paths(directory, *git_args): @@ -274,9 +299,32 @@ def covered(path, entries): return False +def permits(args): + """[((path, is_dir), ruling id)]: the paths the adjudicator rulings in + force permit this unit (ledger's adjudicator_permits, one helper for both + scripts); [] when the ledger cannot be read, so the strict rule holds.""" + try: + loader = importlib.machinery.SourceFileLoader("_pl_ledger", os.path.join(HERE, "ledger")) + spec = importlib.util.spec_from_loader("_pl_ledger", loader) + ledger = importlib.util.module_from_spec(spec) + with contextlib.redirect_stderr(io.StringIO()): + loader.exec_module(ledger) + found = ledger.adjudicator_permits(args["ledger"], args["tasks"], args["declared_raw"]) + except (Exception, SystemExit): # noqa: BLE001 - any failure keeps the strict rule + return [] + out = [] + for path, ruling in found: + key = entry_key(path) if isinstance(path, str) else None + # A permit covers its own path only, never a directory's contents. + if key is not None and not key[1] and key[0] != ".": + out.append((key, ruling)) + return out + + def select(args, changed): - """(selected, reasons) per the docstring's rules.""" + """(selected, reasons, permitted) per the docstring's rules.""" reasons = [] + permitted = [] if args["entry"] is None: reasons.append({"code": "no_entry", "detail": "no task_checks entry for this unit"}) if not args["declared"]: @@ -284,6 +332,12 @@ def select(args, changed): if changed is not None: if args["declared"]: outside = [p for p in changed if not covered(p, args["declared"])] + allowed = permits(args) if outside else [] + for p in list(outside): + ruling = next((rid for key, rid in allowed if covered(p, [key])), None) + if ruling is not None: + permitted.append({"path": p, "ruling": ruling}) + outside.remove(p) if outside: reasons.append({"code": "outside_files", "detail": "outside the Files list: " + ", ".join(outside)}) shared = [p for p in changed if covered(p, args["widen"])] @@ -294,8 +348,8 @@ def select(args, changed): if changed is None: reasons.append({"code": "git_failed", "detail": "could not list the changed files"}) if args["entry"] is None: - return "lane", reasons - return ("widened" if reasons else "targeted"), reasons + return "lane", reasons, permitted + return ("widened" if reasons else "targeted"), reasons, permitted def reserve(ledger, unit): @@ -316,6 +370,23 @@ def reserve(ledger, unit): n += 1 +def printed(report, out): + """report as printed: long path lists cut, the full report left in out.""" + policy = report.get("policy") + if not isinstance(policy, dict): + return report + changed = policy.get("changed") or [] + long_details = [r for r in policy.get("reasons") or [] if len(r.get("detail") or "") > PRINT_DETAIL] + if len(changed) <= PRINT_PATHS and not long_details: + return report + cut = dict(policy, changed=changed[:PRINT_PATHS], + reasons=[dict(r, detail=r["detail"][:PRINT_DETAIL] + " ... (cut)") if r in long_details else r + for r in policy["reasons"]]) + note = (f"this printed copy lists {min(len(changed), PRINT_PATHS)} of {len(changed)} changed paths " + f"and cuts long reasons; {out} holds the full report") + return dict(report, policy=cut, printed_cut=note) + + def save(path, data): """Write data to path as one JSON line, in one step.""" _shell.write_text_atomic(path, json.dumps(data) + "\n") @@ -378,13 +449,14 @@ def run(argv): def plan(args, started): """(the command list to run, the policy) for validated args.""" changed = changed_paths(args["dir"], args["base"]) - selected, reasons = select(args, changed) + selected, reasons, permitted = select(args, changed) chosen = {"targeted": args["targeted"], "lane": args["lane_cmds"], "widened": args["wide_cmds"]}[selected] as_list = lambda pairs: [{"group": g, "command": c} for g, c in pairs] # noqa: E731 policy = { "unit": args["unit"], "tasks": args["tasks"], "lane": args["lane"], "base": args["base"], "started": started, "entry": args["entry"], "selected": selected, "reasons": reasons, "changed": changed if changed is not None else [], + "permitted": permitted, "offered": {"targeted": as_list(args["targeted"]) if args["entry"] is not None else None, "lane": as_list(args["lane_cmds"]), "wide": as_list(args["wide_cmds"])}, "project_test_executed": False, @@ -410,13 +482,13 @@ def run_checks(args, chosen, policy, out): "error": (res.stderr.strip() or f"run-checks exited {res.returncode} with no JSON")[-ERROR_TAIL:]} report["policy"] = policy save(out, report) - print(json.dumps(report)) + print(json.dumps(printed(report, out))) sys.exit(3) executed = {r.get("command") for r in report.get("results", []) if isinstance(r, dict)} policy["project_test_executed"] = any(c in executed for c in args["project_tests"]) report["policy"] = policy save(out, report) - print(json.dumps(report)) + print(json.dumps(printed(report, out))) sys.exit(res.returncode if res.returncode in (0, 1) else 3) diff --git a/scripts/worth-it b/scripts/worth-it new file mode 100755 index 0000000..f08a2cf --- /dev/null +++ b/scripts/worth-it @@ -0,0 +1,237 @@ +#!/usr/bin/env python3 +"""Say whether running a plan's lane layout in parallel saves enough to pay +for a run's fixed phases. + +usage: + python3 worth-it PLAN --max-lanes N [--prelude IDS] --lane IDS [--lane IDS ...] + [--join IDS] [--project DIR] [--history-dir DIR] + +IDS are comma-separated task ids; --lane repeats, one per lane, in lane order +and task order; --prelude, --join and a --lane may be empty. N is the lane +cap the Assessment computed (an integer, at least 1). + +The layout: N tasks, L non-empty lanes, A = min(L, --max-lanes) lanes at +once (at least 1). Task-slots S = prelude tasks + the makespan of the lanes' +task counts assigned in order, each to the worker free earliest, across A +workers (the engine's runLanes) + join tasks. A direct build takes N +task-times; the layout saves N - S. recommend is "direct" when the saving is +THRESHOLD or less, else "parallel". + +The plan is read only to check the layout: every id must be a task heading +of the plan (_plan.task_sections) and appear once. + +With --project, a scale line: the mean, lowest and highest task time of this +project's complete archived runs (history index entries whose status is +complete and whose project matches, as history --project matches), each +run's task time from its autopsy.json and manifest (_history.run_figures). +For scale only; it never changes the recommendation. + +Prints one JSON object: + {"status": "ok", "tasks": N, "lanes": L, "lanes_at_once": A, "slots": S, + "saved": N - S, "threshold": THRESHOLD, "recommend": "parallel"|"direct", + "scale": {"runs", "task_s_mean", "task_s_min", "task_s_max"} | null, + "notes": [str]} +or {"status": "unavailable", "reason": str, "notes": [str]} when the plan +cannot be read or the layout is not valid. Read-only. + +Exit 0, unavailable results included (the session's Offer goes on as +before); 2 on a usage error (no PLAN, no --max-lanes, a --max-lanes that is +not an integer of at least 1, an unknown option or a missing value). +""" +import glob +import json +import os +import sys + +sys.dont_write_bytecode = True +sys.path.insert(0, os.path.dirname(os.path.realpath(__file__))) +import _history # noqa: E402 +import _plan # noqa: E402 +import _shell # noqa: E402 + +USAGE = ("usage: worth-it PLAN --max-lanes N [--prelude IDS] --lane IDS [--lane IDS ...] [--join IDS] " + "[--project DIR] [--history-dir DIR]") + +# The back-test of 31 archived runs (2026-10-15, tools/worth-it-backtest.py; +# the 1.6.4 design, section 4.3): the fixed phases a run adds beyond the +# whole-branch review a direct build also ends with cost a median 1.23 +# task-times (quartiles 0.92 to 1.67). A layout saving 1 task-time or less +# does not come out ahead: at 1 the rule agreed with each run's own +# comparison on 28 of 31 runs, at 2 on 24. +THRESHOLD = 1 + +_SINGLE = ("--max-lanes", "--prelude", "--join", "--project", "--history-dir") +_OPTIONS = _SINGLE + ("--lane",) + + +class UsageError(Exception): + """Bad arguments; the message says why (exit 2).""" + + +def parse_args(args): + """{plan, max_lanes, prelude, lanes, join, project, history_dir}.""" + opts = {"plan": None, "--lane": []} + i = 0 + while i < len(args): + a = args[i] + if a in _OPTIONS: + if i + 1 >= len(args) or args[i + 1] in _OPTIONS: + raise UsageError(a + " needs a value") + if a == "--lane": + opts["--lane"].append(args[i + 1]) + elif a in opts: + raise UsageError(a + " given twice") + else: + opts[a] = args[i + 1] + i += 2 + elif a.startswith("--"): + raise UsageError("unknown option " + a) + elif opts["plan"] is None: + opts["plan"] = a + i += 1 + else: + raise UsageError("one PLAN only, not also " + a) + if not opts["plan"]: + raise UsageError("PLAN is required") + if "--max-lanes" not in opts: + raise UsageError("--max-lanes is required") + try: + max_lanes = int(opts["--max-lanes"]) + except ValueError: + max_lanes = 0 + if max_lanes < 1: + raise UsageError("--max-lanes needs an integer of at least 1, not " + opts["--max-lanes"]) + return { + "plan": opts["plan"], + "max_lanes": max_lanes, + "prelude": split_ids(opts.get("--prelude", "")), + "lanes": [split_ids(v) for v in opts["--lane"]], + "join": split_ids(opts.get("--join", "")), + "project": opts.get("--project") or None, + "history_dir": opts.get("--history-dir") or None, + } + + +def split_ids(value): + return [x.strip() for x in value.split(",") if x.strip()] + + +def slots(prelude, lanes, join, at_once): + """prelude + the makespan of the lane sizes assigned in order, each to + the worker free earliest, across at_once workers + join.""" + workers = [0] * max(1, min(at_once, len(lanes))) + for size in lanes: + i = workers.index(min(workers)) + workers[i] += size + return prelude + (max(workers) if lanes else 0) + join + + +def decide(saved): + return "direct" if saved <= THRESHOLD else "parallel" + + +def plan_ids(path): + with open(path, encoding="utf-8") as f: + lines = f.read().splitlines() + return [s.id for s in _plan.task_sections(lines, _plan.unfenced_mask(lines))] + + +def layout_error(layout, known): + """Why the layout is not valid, or None.""" + if not layout: + return "the layout names no task" + seen = set() + for task in layout: + if task in seen: + return "task " + task + " appears twice in the layout" + seen.add(task) + if task not in known: + return "task " + task + " is not a task heading of the plan" + return None + + +def _read_json(path): + with open(path, encoding="utf-8") as f: + return json.load(f) + + +def scale(hdir, project, notes=None): + """{runs, task_s_mean, task_s_min, task_s_max} over this project's + complete archived runs, or None (with a note when notes is a list).""" + notes = [] if notes is None else notes + try: + index = _history.read_index(hdir) + except _history.HistoryError as e: + notes.append("scale: " + str(e)) + return None + found, skipped = [], 0 + for entry in index["runs"]: + if entry.get("status") != "complete" or not _history.matches_project(entry, project): + continue + folder = _history.run_folder(hdir, entry) + figures = None + try: + manifests = sorted(glob.glob(os.path.join(glob.escape(folder), "*.lanes.json"))) if folder else [] + if manifests: + figures = _history.run_figures(_read_json(os.path.join(folder, "autopsy.json")), + _read_json(manifests[0])) + except Exception: # noqa: BLE001 - one damaged run must never reach the rule + figures = None + if figures is None: + skipped += 1 + else: + found.append((entry.get("run_id"), figures["task_s"])) + if not found: + notes.append("scale: no complete run of this project with readable figures" + + (" (%d skipped)" % skipped if skipped else "")) + return None + if skipped: + notes.append("scale: %d run(s) of this project skipped (no readable autopsy or manifest, or no task agents)" + % skipped) + times = [t for _, t in found] + return {"runs": [r for r, _ in found], "task_s_mean": round(sum(times) / len(times), 1), + "task_s_min": min(times), "task_s_max": max(times)} + + +def assess(args): + notes = [] + try: + known = set(plan_ids(_shell.native_path(args["plan"]))) + except (OSError, UnicodeDecodeError, _shell.BashNotFound) as e: + return {"status": "unavailable", "reason": "cannot read the plan: " + str(e), "notes": notes} + lanes = [lane for lane in args["lanes"] if lane] + layout = args["prelude"] + [t for lane in lanes for t in lane] + args["join"] + error = layout_error(layout, known) + if error: + return {"status": "unavailable", "reason": error, "notes": notes} + at_once = max(1, min(len(lanes), args["max_lanes"])) + used = slots(len(args["prelude"]), [len(lane) for lane in lanes], len(args["join"]), at_once) + saved = len(layout) - used + found = None + if args["project"]: + try: + found = scale(_history.history_dir(args["history_dir"]), args["project"], notes) + except Exception as e: # noqa: BLE001 - the scale line never changes the rule + notes.append("scale: %s: %s" % (type(e).__name__, e)) + return {"status": "ok", "tasks": len(layout), "lanes": len(lanes), "lanes_at_once": at_once, "slots": used, + "saved": saved, "threshold": THRESHOLD, "recommend": decide(saved), "scale": found, "notes": notes} + + +def main(argv): + _shell.setup_io() + try: + args = parse_args(argv) + except UsageError as e: + print("worth-it: " + str(e), file=sys.stderr) + print(USAGE, file=sys.stderr) + return 2 + try: + result = assess(args) + except Exception as e: # noqa: BLE001 - the Offer must never wait on this helper + result = {"status": "unavailable", "reason": "%s: %s" % (type(e).__name__, e), "notes": []} + print(json.dumps(result)) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/src/prompts-phase.js b/src/prompts-phase.js index 1317143..59b2ba8 100644 --- a/src/prompts-phase.js +++ b/src/prompts-phase.js @@ -711,7 +711,9 @@ function finalReviewFrame(m, intro, focus, minors = [], other = false) { "A finding that a ruling in force settles is not raised unless it is critical or important; then raise it and cite the ruling's id.", 'Files each task changed outside its Files list, with their owners (a read-only command):', ` ${ledgerReadCommand(m, featureDir(m), 'undeclared')}`, - "Check each change was needed and, where it is another lane's or a later task's file, covered by a ruling.", + 'Check each change was needed. A file of a task earlier in the run needs no ruling. A file owned by a task in', + 'another lane running alongside, or a later task, needs a pre-flight or adjudicator ruling in force that covers it', + "(cite its id); an implementer's own ruling never does.", 'Also scan every commit message in the range and the whole diff for anything the commit rules forbid;', 'report each as a finding (for a commit message use file "commit " and line 0).', ...fixMessagesLines(), diff --git a/src/prompts-task.js b/src/prompts-task.js index 63bf98c..6a30794 100644 --- a/src/prompts-task.js +++ b/src/prompts-task.js @@ -344,10 +344,13 @@ function taskChecksLines(m, task, where, base) { // The line that points a reviewer at the unit's task-check evidence (spec // 4.3). Files are task-..json: the '.' after the id keeps a -// batch id (T2-T3) apart from its first task's (T2). +// batch id (T2-T3) apart from its first task's (T2). A dotted id still +// collides (unit 1's pattern matches unit 1.2's files), so the line names +// policy.unit as the filter. function taskEvidenceLine(m, task) { return [ `Task-check evidence for this unit: ${m.repo.ledger_dir}/checks/task-${task.id}..json (n = 1, 2, ...);`, + `another unit's files can match that pattern, so read only the files whose policy.unit is ${task.id};`, 'each records its selection (policy.selected) and reasons. Do not rerun the checks. Report a wrong selection', 'as a finding (for example targeted checks on a change that clearly reaches another lane).', ].join('\n'); @@ -538,6 +541,8 @@ function carryOverLines(m, task, where, by) { 'A file owned by a task in another lane running alongside, or by a later task (or a later task\'s planned part of', 'a file you share), needs a pre-flight or adjudicator ruling in force that lists it; without one, report status', 'question instead of changing it. A file of a task earlier in the run is fine when your task needs the change.', + 'When you change a file outside your Files list, record a ruling saying why with "files": [""] added to its', + 'JSON, listing those files: it permits nothing, it shows reviewers and later tasks what changed outside the list.', 'A check failure you judge outside your task (it passed on a rerun, or the failing test covers code your task did', 'not touch): record it as a ruling whose JSON adds "scope": "run", with the text', 'Observed: failed - - .', diff --git a/src/rules.js b/src/rules.js index 8603c83..ba16d3a 100644 --- a/src/rules.js +++ b/src/rules.js @@ -48,6 +48,8 @@ function agentRules() { 'sed, or inline scripts can be corrupted by shell quoting (apostrophes, backslashes, $ and backticks). ' + "Running the project's own tools that change files is fine: generators (for example scripts/build), " + 'formatters, codemods, and other maintenance commands.', + 'Read a file right before you edit or overwrite it (and again after anything else may have changed it): ' + + 'an Edit or Write on a file not read first, or changed since, fails and costs a retry.', ].join('\n'); } diff --git a/tests/autopsy-await.test.mjs b/tests/autopsy-await.test.mjs index dfcd11d..b86cdc0 100644 --- a/tests/autopsy-await.test.mjs +++ b/tests/autopsy-await.test.mjs @@ -190,7 +190,11 @@ test('a procedure check is one measured execution timed from its report', () => // timeout, no long call (the first wait blocked for 540 s), and the waits // that exit 4 are no repeated failure. assert.deepEqual(kinds(json), []); - assert.deepEqual(json.checks.waits, { checks_via_procedure: 1, wait_calls: 3, stopped: 0, improvised: NOTHING }); + assert.deepEqual(json.checks.waits, { checks_via_procedure: 1, wait_calls: 3, stopped: 0, improvised: NOTHING, + unwaited: [], left_running: [] }); + // The waits are the check execution's time (checks section), not open + // questions: d1a listed five 541.6 s waits as unexplained. + assert.deepEqual(json.unexplained, []); // The same calls without a wait are the 1.6.0 background run. const bare = buildRun({ agents: [{ label: 'integrate', phase: 'Integrate', start: '10:00:00', end: '10:13:00', calls: [ @@ -295,7 +299,8 @@ test('waits counts procedure checks, wait calls and stops', () => { }, }); const { json, md } = autopsy(dir); - assert.deepEqual(json.checks.waits, { checks_via_procedure: 2, wait_calls: 4, stopped: 1, improvised: NOTHING }); + assert.deepEqual(json.checks.waits, { checks_via_procedure: 2, wait_calls: 4, stopped: 1, improvised: NOTHING, + unwaited: [{ agent: 'final fix', file: lone }], left_running: [] }); assert.deepEqual(json.checks.executions.map((x) => [x.agent, x.timing, x.s]), [ // --exact reuses its path: the stopped start has no report of its own. ['verify', 'unavailable', 'unavailable'], @@ -371,7 +376,7 @@ test('an older run says checks_via_procedure unavailable', () => { for (const version of ['1.6.0', '1.5.1', 'unavailable']) { const { json, md } = autopsy(buildRun({ agents, files, version })); assert.deepEqual(json.checks.waits, { checks_via_procedure: 'unavailable', wait_calls: 0, stopped: 0, - improvised: { ...NOTHING, monitor_errors: 1 } }, version); + improvised: { ...NOTHING, monitor_errors: 1 }, unwaited: [], left_running: [] }, version); assert.match(md, /^Checks via the procedure: unavailable; 0 wait calls; 0 stopped$/m); } // From 1.6.1 on, no procedure call is a count of 0. @@ -393,8 +398,35 @@ test('an older run says checks_via_procedure unavailable', () => { assert.match(autopsy(untimed).md, /^Checks via the procedure: unavailable$/m); }); -test('the version is 6', () => { - assert.equal(python(['print(json.dumps(_autopsy.AUTOPSY_VERSION))']), 6); +test('the version is 7', () => { + assert.equal(python(['print(json.dumps(_autopsy.AUTOPSY_VERSION))']), 7); const dir = buildRun({ agents: [{ label: 'A1 implement', start: '10:00:05', end: '10:01:00' }] }); - assert.equal(autopsy(dir).json.autopsy_version, 6); + assert.equal(autopsy(dir).json.autopsy_version, 7); +}); + +test('checks.waits lists background checks never waited on, and checks left running when the waits stopped', () => { + const out = checksFile('integrate.1.json'); + const late = checksFile('task-A1.1.json'); + const dir = buildRun({ + agents: [ + { label: 'integrate', phase: 'Integrate', start: '10:00:00', end: '10:13:00', calls: [ + bash(reserve('integrate'), '10:00:05', '10:00:06', { text: out }), + bash(runChecks(out), '10:00:10', '10:00:11', { background: true, text: started('bx1') }), + ] }, + { label: 'A1 implement', start: '10:00:00', end: '10:20:00', calls: [ + bash(reserve('task-A1'), '10:00:05', '10:00:06', { text: late }), + bash(taskChecks('A1', late), '10:00:10', '10:00:11', { background: true, text: started('bx2') }), + bash(wait(late), '10:00:12', '10:09:12', { exit: 4, text: RUNNING(late) }), + ] }, + ], + }); + const { json, md } = autopsy(dir); + assert.deepEqual(json.checks.waits.unwaited, [{ agent: 'integrate', file: out }]); + assert.deepEqual(json.checks.waits.left_running, [{ agent: 'A1 implement', file: late }]); + assert.ok(md.includes('Background checks never waited on: 1 (integrate)'), md); + assert.ok(md.includes('Checks left running when the waits stopped: 1 (A1 implement)'), md); + // The evidence shows no await-checks wait followed, not that the agent never got the result + // (a completion notice and a Read of the file would also deliver it). + assert.ok(!md.includes('ended without the check'), md); + assert.ok(md.replace(/\s+/g, ' ').includes('no await-checks wait on its file followed'), md); }); diff --git a/tests/autopsy-check-policy.test.mjs b/tests/autopsy-check-policy.test.mjs index 619ae35..ecb64a3 100644 --- a/tests/autopsy-check-policy.test.mjs +++ b/tests/autopsy-check-policy.test.mjs @@ -430,13 +430,13 @@ test('legacy full_suite_runs is unchanged on the existing fixtures', () => { assert.ok(existsSync(join(dir, 'autopsy.md'))); }); -test('AUTOPSY_VERSION is 6', () => { +test('AUTOPSY_VERSION is 7', () => { assert.deepEqual(python(['print(json.dumps([_autopsy.AUTOPSY_VERSION, _autopsy.SUGGESTIONS["long_tool_call"]]))']), [ - 6, 'A single tool call likely ran for over 2 minutes. Try a narrower command while iterating, and the task ' + 7, 'A single tool call likely ran for over 2 minutes. Try a narrower command while iterating, and the task ' + 'checks once before committing.', ]); const dir = buildRun({ launches: [{ agents: [{ label: 'A1 implement', start: '10:00:05', end: '10:01:00' }] }] }); - assert.equal(autopsy(dir).json.autopsy_version, 6); + assert.equal(autopsy(dir).json.autopsy_version, 7); }); test('autopsy counts an empty task evidence file as malformed and unmatched', () => { @@ -462,3 +462,75 @@ test('autopsy counts an empty task evidence file as malformed and unmatched', () ], [dir]); assert.deepEqual(found, [['task-A1.2.json'], 1]); }); + +// 1.6.4: check time per configured command (by check id, never command text) +// from every evidence file with a results list, and the widened-checks table. +const result = (command, s, group = 'test') => ({ group, command, exit: 0, started: T('10:00:00'), + ended: T('10:00:00'), duration_s: s, log: '/l/checks/x.1.log', tail: '' }); +const ONE_AGENT = [{ agents: [{ label: 'A1 implement', start: '10:00:05', end: '10:01:00' }] }]; + +test('checks.commands sums each configured command by check id', () => { + const m = manifest({ task_checks: { A1: { test: [TARGETED] } } }); + const dir = buildRun({ m, launches: ONE_AGENT, evidenceFiles: { + 'task-A1.1.json': evidence({ unit: 'A1', selected: 'widened', started: T('10:00:10'), results: [result(TEST_CMD, 300)] }), + 'task-A2.1.json': evidence({ unit: 'A2', selected: 'widened', started: T('10:00:20'), + results: [result(TEST_CMD, 500), result('eslint .', 4, 'lint')] }), + 'task-A3.1.json': evidence({ unit: 'A3', started: T('10:00:30'), results: [result(TARGETED, 60)] }), + 'integrate.1.json': { checkout: '/w', results: [result(TEST_CMD, 700)], ok: true }, + } }); + const c = autopsy(dir).json.checks; + assert.deepEqual(c.commands, [ + { check: 'test#1', group: 'test', runs: 3, total_s: 1500, median_s: 500, project_suite: true }, + { check: 'lint#1', group: 'lint', runs: 1, total_s: 4, median_s: 4, project_suite: true }, + { check: 'A1.task.test#1', group: 'test', runs: 1, total_s: 60, median_s: 60, project_suite: false }, + ]); + assert.deepEqual(c.commands_unconfigured, { runs: 0, total_s: 0 }); + assert.equal(c.commands_skipped, 0); +}); + +test('commands the manifest does not name are counted without their text', () => { + const dir = buildRun({ launches: ONE_AGENT, evidenceFiles: { + 'long.1.json': { checkout: '/w', results: [result('node focused-SECRET-123.mjs', 90)], ok: true }, + } }); + const { json, md } = autopsy(dir); + assert.deepEqual(json.checks.commands_unconfigured, { runs: 1, total_s: 90 }); + assert.deepEqual(json.checks.commands, []); + assert.ok(!JSON.stringify(json).includes('SECRET-123')); + assert.ok(!md.includes('SECRET-123')); +}); + +test('error records and untimed results are skipped and counted', () => { + const dir = buildRun({ launches: ONE_AGENT, evidenceFiles: { + 'task-A1.1.json': evidence({ unit: 'A1', started: T('10:00:10'), results: [], ok: false, error: 'boom' }), + 'integrate.1.json': '{not json', + 'verify-x.json': { checkout: '/w', results: [{ group: 'test', command: TEST_CMD, exit: 0 }], ok: true }, + 'notes.json': { something: 'else' }, + } }); + const c = autopsy(dir).json.checks; + assert.deepEqual(c.commands, []); + assert.equal(c.commands_skipped, 3); +}); + +test('the widened table lists each widened call against the median targeted call', () => { + const dir = buildRun({ launches: ONE_AGENT, evidenceFiles: { + 'task-A1.1.json': evidence({ unit: 'A1', selected: 'widened', started: T('10:00:10'), + reasons: [{ code: 'outside_files', detail: 'outside the Files list: x' }, { code: 'widen_path', detail: 'y' }], + results: [result(TEST_CMD, 800), result('eslint .', 100, 'lint')] }), + 'task-A2.1.json': evidence({ unit: 'A2', started: T('10:00:20'), results: [result(TARGETED, 100)] }), + 'task-A3.1.json': evidence({ unit: 'A3', started: T('10:00:30'), results: [result(TARGETED, 300)] }), + } }); + const { json, md } = autopsy(dir); + assert.deepEqual(json.checks.widened, { targeted_median_s: 200, + calls: [{ unit: 'A1', codes: ['outside_files', 'widen_path'], s: 900 }] }); + assert.match(md, /^Widened task checks, compared with the run's median targeted call \(measured\):$/m); + assert.ok(md.includes('| A1 | outside_files, widen_path | 900.0 s | 200.0 s | +700.0 s |'), md); +}); + +test('no widened table without a widened call', () => { + const dir = buildRun({ launches: ONE_AGENT, evidenceFiles: { + 'task-A2.1.json': evidence({ unit: 'A2', started: T('10:00:20'), results: [result(TARGETED, 100)] }), + } }); + const { json, md } = autopsy(dir); + assert.deepEqual(json.checks.widened, { targeted_median_s: 100, calls: [] }); + assert.ok(!md.includes('Widened task checks')); +}); diff --git a/tests/autopsy.test.mjs b/tests/autopsy.test.mjs index f9dfd07..6b1bf4f 100644 --- a/tests/autopsy.test.mjs +++ b/tests/autopsy.test.mjs @@ -102,7 +102,7 @@ test('done lists each task and the acceptance', () => { 'tokens', 'checks', 'slowdowns', 'detectors', 'suggestions', 'unexplained', 'launch_conditions', 'orphan_snapshots', ]); - assert.equal(A.autopsy_version, 6); + assert.equal(A.autopsy_version, 7); assert.equal(A.run_id, 'f1'); assert.equal(A.engine_version, '1.5.0'); assert.deepEqual(A.done.tasks, { @@ -1092,6 +1092,10 @@ test('checks without tool events are unavailable, not zero', () => { full_suite_runs: { total: 'unavailable', inferred: 'unavailable', by_role: 'unavailable' }, task_check_calls: 'unavailable', project_test_runs: 'unavailable', repairs: 'unavailable', setup: 'unavailable', waits: 'unavailable', + // 1.6.4: check time per command comes from the ledger's evidence files, not + // the transcripts, so it is reported without tool events (run3 has none). + commands: [], commands_unconfigured: { runs: 0, total_s: 0 }, commands_skipped: 0, + widened: { targeted_median_s: 'unavailable', calls: [] }, }); }); diff --git a/tests/carry-prompts.test.mjs b/tests/carry-prompts.test.mjs index 8471a4e..992d510 100644 --- a/tests/carry-prompts.test.mjs +++ b/tests/carry-prompts.test.mjs @@ -71,6 +71,11 @@ test('implementers check owners before going outside their Files list, record ob assert.ok(p.includes('"scope": "run"'), p); assert.ok(p.includes('"overrides": ['), p); assert.ok(p.includes('may override only an implementer ruling of its own lane'), p); + // P7 (d1a): DJ4's rulings explained its edits to earlier lanes' files in text only; with + // "files" the ruling names them. It permits nothing (task-checks counts an adjudicator's only). + const flat = p.replace(/\s+/g, ' '); + assert.ok(flat.includes('When you change a file outside your Files list, record a ruling saying why with "files": ['), p); + assert.ok(flat.includes('it permits nothing'), p); }); test('the adjudicator reads every ruling and may scope, override and permit files', () => { @@ -95,6 +100,12 @@ test('final reviewers, the final fix, the re-review and the verify repair read e assert.ok(p.includes('A finding that a ruling in force settles is not raised unless it is critical or important;' + ' then raise it and cite the ruling\'s id.'), p); assert.ok(p.includes(`${LEDGER} undeclared '/work/ledger'`), p); + // d1a F22: "another lane's file" made a lens demand a ruling for an earlier lane's file, + // which the task rules allow; the lens rule must match them. + const flat = p.replace(/\s+/g, ' '); + assert.ok(flat.includes('A file of a task earlier in the run needs no ruling'), p); + assert.ok(flat.includes('a task in another lane running alongside, or a later task'), p); + assert.ok(!flat.includes("another lane's or a later task's file"), p); } }); diff --git a/tests/check-policy-prompts.test.mjs b/tests/check-policy-prompts.test.mjs index d568cd3..91a5709 100644 --- a/tests/check-policy-prompts.test.mjs +++ b/tests/check-policy-prompts.test.mjs @@ -179,6 +179,8 @@ test('review prompts name the unit\'s evidence files and invite a finding on a w assert.ok(reviewPrompt(m, t(m, 'T2'), where, H('a'), H('b')).includes(line)); assert.ok(reReviewPrompt(m, t(m, 'T2'), where, H('b'), H('c'), []).includes(line)); assert.ok(taskEvidenceLine(m, batchUnit([t(m, 'T2'), t(m, 'T3')])).includes('task-T2-T3..json')); + // 1.6.0 deferred minor: task-1..json also matches unit 1.2's task-1.2..json files. + assert.ok(flat(line).includes('read only the files whose policy.unit is T2'), line); }); test('the fix fallback is not the implement fallback', () => { diff --git a/tests/doctor.test.mjs b/tests/doctor.test.mjs index 0dec1dd..c177e28 100644 --- a/tests/doctor.test.mjs +++ b/tests/doctor.test.mjs @@ -103,7 +103,7 @@ function fixture() { const CHECK_NAMES = [ 'git_identity', 'base_not_current', 'python', 'dirty_checkout', 'shadow_repo', 'memory', 'disk', 'python_alias', 'heavy_processes', 'active_runs', 'defender', 'concurrency', - 'usage_limit_pause', + 'host_budget', 'usage_limit_pause', ]; // No scenario may leave a check unrecorded: every result the helpers return @@ -441,13 +441,15 @@ test('doctor changes nothing', () => { // that calls run_checks. The hard checks still run, against the fixture repo. const GB = 1024 ** 3; -const WARNING_CHECKS = ['memory', 'disk', 'python_alias', 'heavy_processes', 'active_runs', 'defender', 'concurrency']; +const WARNING_CHECKS = ['memory', 'disk', 'python_alias', 'heavy_processes', 'active_runs', 'defender', 'concurrency', + 'host_budget']; // A fake that raises stands for a read that fails. const RAISE = 'raise'; const QUIET = { memory: { available: 16 * GB }, disk_free: 100 * GB, load_average: [0.5, 8], + cpu_count: 8, appx_working_set: 0, defender_exclusions: [], active_runs: [], @@ -537,7 +539,7 @@ function snippet(lines, ...args) { test('every check records exactly one outcome', () => { const f = fixture(); const r = fakeDoctor(f.manifest()); - assert.equal(r.checks.length, 13, JSON.stringify(r.checks)); + assert.equal(r.checks.length, 14, JSON.stringify(r.checks)); assert.deepEqual(r.checks.map((c) => c.check), CHECK_NAMES); const expected = {}; for (const x of r.failures) expected[x.check] = 'failed'; @@ -621,7 +623,7 @@ test('an unrecorded check is never passed', () => { assert.deepEqual(r.skipped.filter((s) => s.check === check), [{ check, reason: 'not_recorded' }]); } } - assert.equal(r.skipped.length, 12); + assert.equal(r.skipped.length, 13); assert.equal(r.status, 'ok'); }); @@ -741,6 +743,29 @@ test('the concurrency cap warns when below lanes at once', () => { assert.equal(named(capped.warnings, 'concurrency'), undefined, JSON.stringify(capped)); }); +test('host_budget warns when lanes at once exceed the CPUs', () => { + const f = fixture(); + const lanes = [{ id: 'a' }, { id: 'b' }, { id: 'c' }, { id: 'd' }]; + const four = f.manifest({}, { lanes, limits: { max_parallel_lanes: 5 } }); + const r = fakeDoctor(four, { fakes: { cpu_count: 2 } }); + assert.equal(named(r.warnings, 'host_budget').detail, '4 lanes at once, 2 CPUs'); + assert.equal(r.status, 'ok', JSON.stringify(r)); + // At the boundary it passes. + const two = fakeDoctor(f.manifest({}, { lanes: lanes.slice(0, 2), limits: { max_parallel_lanes: 5 } }), + { fakes: { cpu_count: 2 } }); + assert.equal(outcome(two, 'host_budget'), 'passed'); + // max_parallel_lanes and a lower concurrency variable cap the lanes at once. + const capped = fakeDoctor(f.manifest({}, { lanes, limits: { max_parallel_lanes: 2 } }), { fakes: { cpu_count: 2 } }); + assert.equal(outcome(capped, 'host_budget'), 'passed'); + const variable = fakeDoctor(four, { fakes: { cpu_count: 2 }, env: { CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS: '2' } }); + assert.equal(outcome(variable, 'host_budget'), 'passed'); + // A CPU count that cannot be read is a warning saying unknown, as for the other reads. + for (const value of [null, RAISE, 0]) { + const unknown = fakeDoctor(four, { fakes: { cpu_count: value } }); + assert.equal(named(unknown.warnings, 'host_budget').detail, 'unknown', String(value)); + } +}); + test('real probes run on this platform without error', () => { const f = fixture(); const active = join(f.dir, 'active'); @@ -941,5 +966,5 @@ test('usage_limit_pause with a real claude on PATH and malformed settings never assert.match(r.out.usage_limit_pause.reasons.join(' '), /user settings file .* could not be read/); assert.equal(outcome(r.out, 'usage_limit_pause'), 'warning'); assert.equal(named(r.out.failures, 'usage_limit_pause'), undefined); - assert.equal(r.out.checks.length, 13); + assert.equal(r.out.checks.length, 14); }); diff --git a/tests/dryrun.test.mjs b/tests/dryrun.test.mjs index 9cdf42e..18b3a34 100644 --- a/tests/dryrun.test.mjs +++ b/tests/dryrun.test.mjs @@ -445,6 +445,8 @@ test('repair_limit is 0 for a supervised manifest and 2 for an autonomous one', assert.equal(repairLimit(manifest({ autonomy: 'supervised' })), 0); assert.equal(repairReserve(manifest()), 7); assert.equal(repairReserve(manifest({ hooks: { e2e: 'x', post_integrate: 'y' } })), 13); + // The post-integrate 2 apply in the full profile only (reference.md says so since 1.6.4). + assert.equal(repairReserve(manifest({ profile: 'lite', hooks: { post_integrate: 'y' } })), 7); assert.equal(repairFixMessage(), 'fix: make the delivery checks pass'); const { result } = await dryRun(policyManifest({ autonomy: 'supervised', repairs_spent: 1 })); assert.equal(result.checks.repair_limit, 0); diff --git a/tests/engine.test.mjs b/tests/engine.test.mjs index dc6cb10..11ab8f4 100644 --- a/tests/engine.test.mjs +++ b/tests/engine.test.mjs @@ -1300,6 +1300,8 @@ test('every agent gets the file-editing rule', () => { 'Change file contents with your file-editing tools (Edit, Write).', 'can be corrupted by shell quoting', 'generators (for example scripts/build), formatters, codemods', + // f1a 17, hx1 10, d1a 12 edit_retry slowdowns: an Edit or Write on a file not read first fails. + 'Read a file right before you edit or overwrite it', ]) { assert.ok(r.includes(phrase), phrase); } diff --git a/tests/history-figures.test.mjs b/tests/history-figures.test.mjs new file mode 100644 index 0000000..bd48851 --- /dev/null +++ b/tests/history-figures.test.mjs @@ -0,0 +1,99 @@ +// _history.run_figures (1.6.4): one run's task time, fixed phases and final +// review from its autopsy, shared by scripts/history and scripts/worth-it; and +// the project matching and run folder helpers moved there from scripts/history. +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { join } from 'node:path'; +import { SKILL_DIR } from './harness.mjs'; + +const SCRIPTS = join(SKILL_DIR, 'scripts'); + +// Runs a Python snippet with scripts/ importable (no bytecode); returns its stdout as JSON. +function python(lines, input = null) { + const code = [ + 'import json, sys', + 'sys.dont_write_bytecode = True', + `sys.path.insert(0, ${JSON.stringify(SCRIPTS)})`, + 'import _history', + ...lines, + ].join('\n'); + const r = spawnSync('python3', ['-B', '-c', code], { encoding: 'utf8', input }); + assert.equal(r.status, 0, r.stderr); + return JSON.parse(r.stdout); +} + +const T = (hms) => `2026-10-15T${hms}.000Z`; +const agent = (task, role, launch, start, end) => ({ label: `${task} ${role}`, task, role, launch, start: T(start), end: T(end) }); +const phase = (name, launch, start, end) => ({ phase: name, launch, start: T(start), end: T(end) }); +const MANIFEST = { prelude: [], join: [], lanes: [{ id: 'a', tasks: [{ id: 'A1' }, { id: 'B1' }] }] }; +const figures = (analysis, manifest = MANIFEST) => + python(['a, m = json.load(sys.stdin)', 'print(json.dumps(_history.run_figures(a, m)))'], JSON.stringify([analysis, manifest])); + +// Launch wf_b started first (10:00) and ran no task; wf_a (later, 12:00, though its id +// sorts first) ran A1 and B1; wf_c (14:00) ran only phases. +const ANALYSIS = { + time: { + agents: [ + agent('A1', 'implement', 'wf_b', '10:01:00', '10:05:00'), + agent('A1', 'implement', 'wf_a', '12:10:00', '12:20:00'), + agent('A1', 'review', 'wf_a', '12:20:00', '12:22:00'), + agent('B1', 'implement', 'wf_a', '12:10:00', '12:15:00'), + { label: 'pre-flight', task: null, role: 'pre-flight', launch: 'wf_a', start: T('12:00:00'), end: T('12:10:00') }, + ], + phases: [ + phase('Pre-flight', 'wf_b', '10:00:00', '10:01:00'), + phase('Pre-flight', 'wf_a', '12:00:00', '12:10:00'), + phase('Lane a', 'wf_a', '12:10:00', '12:22:00'), + phase('Integrate', 'wf_a', '12:22:00', '12:30:00'), + phase('Final review', 'wf_a', '12:30:00', '12:50:00'), + phase('Verify', 'wf_a', '12:40:00', '12:45:00'), + phase('Pre-flight', 'wf_c', '14:00:00', '14:30:00'), + ], + }, +}; + +test('run_figures uses the latest launch that ran tasks', () => { + // wf_b's A1 ran earlier than wf_a; wf_c ran no task agent. + const f = figures(ANALYSIS); + assert.deepEqual(f, { launch: 'wf_a', tasks: 2, task_s: 510, fixed_s: 2280, final_review_s: 1200, wall_s: 3000 }); +}); + +test('run_figures matches phase labels exactly', () => { + const a = JSON.parse(JSON.stringify(ANALYSIS)); + a.time.phases.push(phase('Lane verify', 'wf_a', '12:50:00', '13:20:00')); + const f = figures(a); + assert.equal(f.fixed_s, 2280); + assert.equal(f.wall_s, 4800); +}); + +test('run_figures is None without task agents', () => { + const a = { time: { agents: [{ label: 'pre-flight', task: null, launch: 'wf_a', start: T('12:00:00'), end: T('12:10:00') }], + phases: ANALYSIS.time.phases } }; + assert.equal(figures(a), null); + assert.equal(figures({ time: {} }), null); + assert.equal(figures({}), null); + // An agent of a task the manifest does not name does not count. + assert.equal(figures(ANALYSIS, { prelude: [], join: [], lanes: [{ id: 'a', tasks: [{ id: 'Z9' }] }] }), null); +}); + +test('matches_project still matches path forms', () => { + const r = python([ + 'e = {"project": "C:/x/proj"}', + 'print(json.dumps([_history.matches_project(e, n) for n in ["C:/x/proj", "c:/X/PROJ", "proj", "PROJ", "other"]]' + + ' + [_history.matches_project({"project": "/home/u/proj"}, n) for n in ["/home/u/proj", "proj", "PROJ"]]' + + ' + [_history.matches_project({"project": "unavailable"}, "unavailable")]))', + ]); + assert.deepEqual(r, [true, true, true, true, false, true, true, false, false]); +}); + +test('run_folder stays inside the history dir', () => { + const r = python([ + 'import os', + 'h = os.path.abspath("hist")', + 'print(json.dumps([_history.run_folder(h, {"folder": "2026-10-15-d1a"}) == os.path.join(h, "2026-10-15-d1a"),' + + ' _history.run_folder(h, {"folder": "../x"}), _history.run_folder(h, {"folder": ""}),' + + ' _history.run_folder(h, {})]))', + ]); + assert.deepEqual(r, [true, null, null, null]); +}); diff --git a/tests/history.test.mjs b/tests/history.test.mjs index 9d55d1e..8e497a1 100644 --- a/tests/history.test.mjs +++ b/tests/history.test.mjs @@ -316,3 +316,144 @@ test('--since and --project filter runs', () => { assert.equal(res.code, 2); assert.match(res.stderr, /index\.json/); }); + +// ---- history --runs A,B (1.6.4) ---- + +// A history of fa and fb with the same archived plan and current autopsies +// (made by a plain history run first). +function comparablePair() { + const h = makeHistory(); + for (const dir of [h.a, h.b]) writeFileSync(join(dir, 'p.md'), '# Plan\n\n### Task A1: x\n'); + historyOk(h.hist); + return h; +} +function runsOf(hist, ...extra) { + const r = spawnSync('python3', [HISTORY, '--history-dir', hist, '--runs', 'fa,fb', ...extra], + { encoding: 'utf8', env: cleanEnv() }); + let json = null; + try { json = JSON.parse(r.stdout); } catch { /* text output */ } + return { code: r.status, stdout: r.stdout, stderr: r.stderr, json }; +} + +test('--runs compares two runs of the same plan as comparable', () => { + const { hist } = comparablePair(); + const r = runsOf(hist, '--json'); + assert.equal(r.code, 0, r.stderr); + assert.equal(r.json.comparable, true, JSON.stringify(r.json.reasons)); + assert.deepEqual(r.json.reasons, []); + const [fa, fb] = r.json.runs; + assert.deepEqual([fa.run_id, fb.run_id], ['fa', 'fb']); + assert.equal(fa.plan_sha256, fb.plan_sha256); + assert.match(fa.plan_sha256, /^[0-9a-f]{64}$/); + assert.deepEqual(fa.lanes, ['a', 'b']); + assert.deepEqual(fa.task_ids, fb.task_ids); + assert.equal(fa.commands_sha256, fb.commands_sha256); + assert.deepEqual([fa.autopsy, fb.autopsy], ['current', 'current']); + for (const key of ['wall_s', 'fixed_s', 'task_s', 'full_suite_s', 'tokens_total', 'tokens_partial', 'engine_version', + 'claude_code_version', 'launches', 'status', 'acceptance', 'folder', 'plan']) { + assert.ok(key in fa, key); + } + const text = runsOf(hist); + assert.equal(text.code, 0, text.stderr); + assert.equal(text.stdout.split('\n')[0], 'comparable'); +}); + +test('--runs lists every reason two runs are not comparable', () => { + const { hist, a, b } = comparablePair(); + writeFileSync(join(b, 'p.md'), '# Plan\n\n### Task A1: changed\n'); + editJson(join(b, 'p.lanes.json'), (m) => { + m.lanes[1].id = 'c'; + m.lanes[1].tasks.push({ id: 'Z9', title: 'Z9', tier: 'standard' }); + m.commands.test = ['npm test']; + }); + rmSync(join(a, 'autopsy.json')); + editLines(join(b, 'wf_1', 'agent-a2.jsonl'), (lines) => { + lines[2] = lines[2].slice(0, 60); + return lines; + }); + historyOk(hist); + rmSync(join(a, 'autopsy.json')); + const r = runsOf(hist, '--json'); + assert.equal(r.code, 0, r.stderr); + assert.equal(r.json.comparable, false); + assert.deepEqual(r.json.reasons, ['plan differs', 'tasks differ', 'lanes differ', 'commands differ', + 'autopsy missing for fa (run history first)']); + const text = runsOf(hist); + assert.equal(text.stdout.split('\n')[0], `not comparable: ${r.json.reasons.join('; ')}`); +}); + +test('--runs notes partial tokens without calling the pair not comparable', () => { + // Final review (1.6.4): every real archived run has partial token counts (transcripts keep + // mid-stream output counts), so "tokens partial" as a reason made no real pair comparable. + const { hist, b } = comparablePair(); + editLines(join(b, 'wf_1', 'agent-a2.jsonl'), (lines) => { + lines[2] = lines[2].slice(0, 60); + return lines; + }); + historyOk(hist); + const r = runsOf(hist, '--json'); + assert.equal(r.code, 0, r.stderr); + assert.equal(r.json.comparable, true, JSON.stringify(r.json.reasons)); + assert.equal(r.json.runs[1].tokens_partial, true); + assert.deepEqual(r.json.notes, ['tokens partial for fb: its token totals are lower bounds']); + const text = runsOf(hist).stdout.split('\n'); + assert.equal(text[0], 'comparable'); + assert.equal(text[1], 'note: tokens partial for fb: its token totals are lower bounds'); +}); + +test('--runs marks a stale autopsy and a missing plan', () => { + const { hist, a, b } = comparablePair(); + editJson(join(a, 'autopsy.json'), (x) => { x.autopsy_version = 1; }); + rmSync(join(b, 'p.md')); + const r = runsOf(hist, '--json'); + assert.deepEqual([r.json.runs[0].autopsy, r.json.runs[1].plan_sha256], ['stale', 'unavailable']); + assert.deepEqual(r.json.reasons, ['plan unavailable for fb', 'autopsy stale for fa (run history first)']); +}); + +test('--runs prints no percentage and writes nothing', () => { + const { hist, a } = comparablePair(); + const before = ['history.json', 'history.md'].map((n) => readFileSync(join(hist, n), 'utf8')); + const autopsyBefore = readFileSync(join(a, 'autopsy.json'), 'utf8'); + rmSync(join(a, 'autopsy.md')); + const r = runsOf(hist); + assert.equal(r.code, 0, r.stderr); + assert.ok(!r.stdout.includes('%'), r.stdout); + assert.deepEqual(['history.json', 'history.md'].map((n) => readFileSync(join(hist, n), 'utf8')), before); + // It never reruns autopsy: the stale run stays as it was. + assert.equal(readFileSync(join(a, 'autopsy.json'), 'utf8'), autopsyBefore); + assert.ok(!existsSync(join(a, 'autopsy.md'))); +}); + +test('--runs refuses ambiguous ids', () => { + const { hist } = comparablePair(); + const same = spawnSync('python3', [HISTORY, '--history-dir', hist, '--runs', 'fa,fa'], { encoding: 'utf8', env: cleanEnv() }); + assert.equal(same.status, 2); + assert.match(same.stderr, /twice|same run/); + // An id the index holds twice: both folders are named. + editJson(join(hist, 'index.json'), (idx) => { idx.runs.push({ ...idx.runs[0], folder: '2026-10-11-fa' }); }); + const dup = spawnSync('python3', [HISTORY, '--history-dir', hist, '--runs', 'fa,fb'], { encoding: 'utf8', env: cleanEnv() }); + assert.equal(dup.status, 2); + assert.match(dup.stderr, /2026-10-10-fa/); + assert.match(dup.stderr, /2026-10-11-fa/); + const missing = spawnSync('python3', [HISTORY, '--history-dir', hist, '--runs', 'fb,nope'], { encoding: 'utf8', env: cleanEnv() }); + assert.equal(missing.status, 2); + assert.match(missing.stderr, /nope/); +}); + +test('--runs with --since or --project is a usage error', () => { + const { hist } = comparablePair(); + for (const extra of [['--since', '2026-10-01'], ['--project', 'proj'], []]) { + const args = extra.length ? ['--runs', 'fa,fb', ...extra] : ['--runs', 'fa']; + const r = spawnSync('python3', [HISTORY, '--history-dir', hist, ...args], { encoding: 'utf8', env: cleanEnv() }); + assert.equal(r.status, 2, JSON.stringify(args)); + assert.match(r.stderr, /usage/); + } +}); + +test('--runs takes a folder name with a trailing slash', () => { + const { hist } = comparablePair(); + const r = spawnSync('python3', [HISTORY, '--history-dir', hist, '--runs', '2026-10-10-fa/,fb', '--json'], + { encoding: 'utf8', env: cleanEnv() }); + assert.equal(r.status, 0, r.stderr); + assert.equal(JSON.parse(r.stdout).runs[0].run_id, 'fa'); +}); diff --git a/tests/ledger-carry.test.mjs b/tests/ledger-carry.test.mjs index a370b89..2d13a72 100644 --- a/tests/ledger-carry.test.mjs +++ b/tests/ledger-carry.test.mjs @@ -221,7 +221,8 @@ test("a ruling that overrides a launch's PF-n resolves to that launch's carried assert.ok(out.includes('alpha-R1 [adjudicator, T2, lane] NEW (overrides _run-R2)'), out); assert.ok(out.split('Overridden:')[1].includes('_run-R2 by alpha-R1'), out); assert.deepEqual(statusOf(dir).overrides, - [{ id: 'alpha-R1', by: 'adjudicator', task: 'T2', overrides: ['_run-R2'], unknown: [] }]); + [{ id: 'alpha-R1', by: 'adjudicator', task: 'T2', overrides: ['_run-R2'], unknown: [], + targets: [{ id: '_run-R2', role: 'preflight', lane: '_run' }] }]); }); test('a 1.6.2 result with pre-flight conflicts: the last pre-flight ruling is the adjudication, carried once', () => { @@ -394,3 +395,68 @@ test('a final review result without carried_minors_open raised none of the carri })); assert.deepEqual(statusOf(dir).minors.open, ['T1-1']); }); + +// ---- 1.6.4: the 1.6.3 deferred minors ---- + +// Loads scripts/ledger as a module in a python3 subprocess and runs code with +// it as L; returns the process result. +function withLedgerModule(code, args = []) { + // spec_from_loader and exec_module, as task-checks loads ledger (load_module is deprecated). + const loader = 'import importlib.machinery, importlib.util, sys\n' + + "ldr = importlib.machinery.SourceFileLoader('ledger', sys.argv[1])\n" + + "L = importlib.util.module_from_spec(importlib.util.spec_from_loader('ledger', ldr))\n" + + `ldr.exec_module(L)\n${code}`; + // -B: no __pycache__ beside the scripts (the suite leaves the checkout unchanged). + return spawnSync('python3', ['-B', '-c', loader, join(SCRIPTS, 'ledger'), ...args], { encoding: 'utf8' }); +} + +test('append_entries writes every entry or none: an invalid one stops all of them', () => { + const dir = freshDir(); + const good = { task: '_run', event: 'ruling', by: 'preflight', text: 'R' }; + const bad = { task: '_run', event: 'minors', wf: 'wf_x', open: 'not a list' }; + const res = withLedgerModule( + 'import json\ntry:\n L.append_entries(sys.argv[2], "_run", json.loads(sys.argv[3]))\nexcept SystemExit as e:\n print("exit", e.code)\n', + [dir, JSON.stringify([good, bad])]); + assert.match(res.stdout, /exit 2/, res.stderr); + assert.equal(rulingsOut(dir).trim(), '(none)'); + const ok = withLedgerModule('import json\nL.append_entries(sys.argv[2], "_run", json.loads(sys.argv[3]))\n', + [dir, JSON.stringify([good, { ...good, text: 'S' }])]); + assert.equal(ok.status, 0, ok.stderr); + assert.deepEqual(readFileSync(join(dir, '_run.jsonl'), 'utf8').trim().split('\n').map((l) => JSON.parse(l).text), ['R', 'S']); +}); + +test('carry-launch refuses a ledger dir that does not exist and creates nothing', () => { + const missing = join(freshDir(), 'no such ledger'); + const config = freshDir('config'); + const res = carry(missing, launchFile(config, 'wf_nodira-001', { rulings: ['R'] })); + assert.equal(res.code, 3, res.stdout); + assert.equal(spawnSync('python3', ['-c', 'import os,sys; print(os.path.exists(sys.argv[1]))', missing], + { encoding: 'utf8' }).stdout.trim(), 'False'); +}); + +test("status overrides names each overridden ruling's role and lane", () => { + const dir = freshDir(); + appendOk(dir, '_run', { task: '_run', event: 'ruling', by: 'adjudicator', text: 'RUN' }); + appendOk(dir, 'alpha', { task: 'T2', event: 'ruling', by: 'implementer', text: 'X', overrides: ['_run-R1', 'zeta-R9'] }); + assert.deepEqual(statusOf(dir).overrides, [{ + id: 'alpha-R1', by: 'implementer', task: 'T2', overrides: ['_run-R1', 'zeta-R9'], unknown: ['zeta-R9'], + targets: [{ id: '_run-R1', role: 'adjudicator', lane: '_run' }], + }]); +}); + +test('write-owners prints the owner map path with forward slashes', () => { + const root = freshDir('proj'); + const res = ledger('write-owners', join(root, 'ledger'), ownersManifest(root)); + assert.equal(res.code, 0, res.stderr); + assert.ok(!JSON.parse(res.stdout).owners.includes('\\'), res.stdout); +}); + +test("a batch's minors are carried once, under the batch's first task", () => { + const dir = freshDir(); + const config = freshDir('config'); + // A batch result repeats the unit's minors for each of its tasks. + const unit = [minor('T2-T3-1', 'T2'), minor('T2-T3-2', 'T2')]; + carry(dir, launchFile(config, 'wf_batch1-001', { tasks: { T2: { ...done(unit), batch: 'T2-T3' }, T3: { ...done(unit), batch: 'T2-T3' } } })); + assert.deepEqual(statusOf(dir).minors, { open: ['T2-T3-1', 'T2-T3-2'], covered: ['T2', 'T3'] }); + assert.ok(ledger('minors', dir).stdout.includes('[T2-T3-1] T2 '), 'listed under the first task'); +}); diff --git a/tests/recovery.test.mjs b/tests/recovery.test.mjs index dac5d8f..a4694a5 100644 --- a/tests/recovery.test.mjs +++ b/tests/recovery.test.mjs @@ -290,6 +290,21 @@ test('ledger status --manifest lists the files a task changed outside its Files assert.deepEqual(status.undeclared, { T1: [{ path: 'b.txt', owner: null, relation: null }] }); }); +test("a batch's shared range counts every batch task's files, in status and in ledger undeclared", () => { + const c = newCase(); + withManifest(c, [ptask('T1', { files: ['a.txt'] }), ptask('T2', { files: ['b.txt'] }), ptask('T3', { files: ['c.txt'] })]); + for (const name of ['a.txt', 'b.txt', 'd.txt']) write(join(c.repo, name), `${name}\n`); + git(c.repo, 'add', '.'); + git(c.repo, 'commit', '-q', '-m', 'batch T1-T2'); + // One range for the batch T1-T2: a.txt and b.txt are declared, d.txt is not; c.txt is T3's. + finish(c, c.start, '--task', 'T1', '--task', 'T2'); + const expected = { path: 'd.txt', owner: null, relation: null }; + const status = ok(py('ledger', 'status', c.ledger, '--manifest', c.manifest)); + assert.deepEqual(status.undeclared, { T1: [expected], T2: [expected] }); + ok(py('ledger', 'write-owners', c.ledger, c.manifest)); + assert.deepEqual(ok(py('ledger', 'undeclared', c.ledger)), status.undeclared); +}); + test('run_ended events add up to the budget spent across launches', () => { const c = newCase(); append(c, { task: '_run', event: 'run_ended', status: 'stopped', agents: 5, rulings: 2 }); diff --git a/tests/run-report.test.mjs b/tests/run-report.test.mjs index ad8bf07..988fd69 100644 --- a/tests/run-report.test.mjs +++ b/tests/run-report.test.mjs @@ -100,12 +100,15 @@ test('effort is unavailable for every agent without the workflow result, never d for (const a of report.agents) assert.equal(a.effort, 'unavailable', a.label); }); -test('--out writes the report to a file', () => { +test('--out writes the report to a file and prints its summary', () => { + // SKILL.md's Report step passes --out and appends the output: it used to be empty (m1a review). const out = join(TMP, 'report.json'); const r = run([FIX, MANIFEST, '--out', out]); assert.equal(r.code, 0, r.stderr); - assert.equal(r.stdout, ''); assert.deepEqual(JSON.parse(readFileSync(out, 'utf8')), report); + const summary = JSON.parse(r.stdout); + const { agents, ...rest } = report; + assert.deepEqual(summary, { report: out, agent_count: agents.length, ...rest }); }); test('bad usage exits 2', () => { diff --git a/tests/setup-script.test.mjs b/tests/setup-script.test.mjs index 20a78b7..0d86a45 100644 --- a/tests/setup-script.test.mjs +++ b/tests/setup-script.test.mjs @@ -250,6 +250,21 @@ test('setup: a dirty main checkout exits 3 and creates nothing', () => { assert.equal(readFileSync(join(c.project, 'README.md'), 'utf8'), 'uncommitted\n'); }); +test('setup: a SHA-256 repository exits 3 with a clear message and creates nothing', () => { + const c = newCase(); + rmSync(join(c.project, '.git'), { recursive: true, force: true }); + git(c.project, 'init', '-q', '-b', 'main', '--object-format=sha256'); + git(c.project, 'add', '-A'); + git(c.project, 'commit', '-q', '-m', 'init'); + const res = setup(c, manifest(c)); + assert.equal(res.code, 3, res.stderr); + assert.equal(res.stdout, ''); + assert.match(res.stderr, /SHA-256/); + assert.equal(sh('git', ['-C', c.project, 'show-ref', '--verify', '--quiet', 'refs/heads/feature/x']).code, 1); + assert.equal(existsSync(c.worktreeRoot), false); + assert.equal(existsSync(c.ledgerDir), false); +}); + test('setup: ignored files in the main checkout survive', () => { const c = newCase(); write(c.project, 'scratch/notes.txt', 'mine\n'); diff --git a/tests/skill.test.mjs b/tests/skill.test.mjs index 6ca9028..d5f2b2c 100644 --- a/tests/skill.test.mjs +++ b/tests/skill.test.mjs @@ -10,7 +10,10 @@ const ADOPT_MD = join(SKILL_DIR, 'adopt.md'); const README_MD = join(SKILL_DIR, 'README.md'); const CI_YML = join(SKILL_DIR, '.github', 'workflows', 'tests.yml'); -const DESCRIPTION = 'Use when an approved implementation plan is about to be executed (at the ' + +// What the skill does, then when to use it (the Agent Skills best-practices guide asks for both). +const DESCRIPTION = 'Runs an approved implementation plan as parallel lanes of tasks in separate git ' + + 'worktrees, each task implemented and reviewed by its own agents, then integrated, checked and ' + + 'reviewed as a whole. Use when an approved implementation plan is about to be executed (at the ' + 'execution-method handoff), or when resuming a stopped parallel-lanes run.'; const NOTICES = { @@ -18,6 +21,8 @@ const NOTICES = { stepAside: 'parallel-lanes: not a fit (); recommending ', launch: 'parallel-lanes: launching run : lanes, agents', resume: 'parallel-lanes: resuming run : tasks already committed', + worthIt: "parallel-lanes: running in parallel saves about task-times, less than a run's fixed phases; " + + 'recommending ', }; const TABLE_NOTE = 'each task can add up to 2x review_rounds more agents (fix and re-review rounds)'; @@ -220,9 +225,23 @@ test('SKILL.md resume rechecks coverage, passes the manifest to ledger status, a // notices, the consent gate, the hand-back). Material needed only sometimes // moves whole to reference.md, with SKILL.md pointing to it. Never shorten // sentences or drop the reasons behind rules to stay under the limit. -test('SKILL.md stays under 3000 words', () => { +test('SKILL.md stays under 4000 words', () => { const words = read(SKILL_MD).toString('utf8').split(/\s+/).filter(Boolean).length; - assert.ok(words < 3000, `SKILL.md has ${words} words`); + assert.ok(words < 4000, `SKILL.md has ${words} words`); +}); + +test('reference.md opens with a contents list naming every section in order', () => { + // The best-practices guide: a reference file over 100 lines needs a table of contents at the + // top, so a partial read (head -100) still shows everything the file holds. + const text = read(REFERENCE_MD).toString('utf8'); + const lines = text.split('\n'); + const headings = lines.filter((l) => l.startsWith('## ')).map((l) => l.slice(3)); + assert.equal(headings[0], 'Contents'); + const start = lines.indexOf('## Contents'); + assert.ok(start >= 0 && start < 20, 'the contents list is near the top'); + const end = lines.findIndex((l, i) => i > start && l.startsWith('## ')); + const entries = lines.slice(start + 1, end).filter((l) => l.startsWith('- ')).map((l) => l.slice(2).split(':')[0]); + assert.deepEqual(entries, headings.slice(1)); }); test('reference.md has sections for profiles, tiers, batching, adjudicator, budgets, and the report', () => { @@ -750,7 +769,9 @@ test('reference.md documents the 1.6.2 report fields, doctor check and repair re assert.ok(report.includes(word), `Report lacks ${word}`); } const doctor = section(ref, 'Doctor'); - assert.ok(doctor.includes('`usage_limit_pause`') && doctor.includes('thirteen checks'), doctor); + assert.ok(doctor.includes('`usage_limit_pause`') && doctor.includes('fourteen checks'), doctor); + // host_budget is expected at the skill's own lane cap on a small machine (1.6.4). + assert.ok(doctor.replace(/\s+/g, ' ').includes('expected at the lane cap'), 'host_budget at the cap'); assert.ok(section(ref, 'Launch conditions').includes('usage_limit_pause'), 'Launch conditions lacks usage_limit_pause'); assert.ok(section(ref, 'Budgets').includes('(7, plus 4 with an e2e hook'), 'Budgets reserve'); assert.ok(section(ref, 'Check ownership').includes('The reserve is 7'), 'Check ownership reserve'); @@ -759,3 +780,36 @@ test('reference.md documents the 1.6.2 report fields, doctor check and repair re test('README lists stop-cause', () => { assert.match(flat(README_MD), /\| `scripts\/` \|[^|]*\bstop-cause\b/); }); + +// 1.6.4: the worth-it check at the Assessment and the Offer. +const OFFER_LINES = [ + "Worth it: these lanes run tasks in about task-slots, saving about task-times; a run's fixed " + + 'phases cost about 1-2.', + "For scale: this project's past runs took about min per task (-).", + "A direct build still ends with a whole-branch review; it gives up the run's worktree isolation, integration and " + + 'delivery checks, E2E and the multi-lens final review with its verify gate.', + 'Worth it: unavailable ().', +]; + +test('SKILL.md runs worth-it after the layout and shows its lines at the Offer', () => { + const text = flat(SKILL_MD); + assert.ok(text.includes('/scripts/worth-it'), 'SKILL.md names scripts/worth-it'); + for (const line of OFFER_LINES) assert.ok(text.includes(line), `SKILL.md lacks: ${line}`); + assert.ok(!text.includes('a single lane included'), 'the single-lane rule changed in 1.6.4'); + assert.ok(text.includes('single-lane'), 'SKILL.md says what happens to a single-lane plan'); + assert.ok(text.includes('only when the Offer'), 'the check runs only when the Offer is shown'); + // Final review (1.6.4): a usage error exits 2 with no JSON; it must not stop the Offer either. + assert.ok(text.includes('a non-zero exit or any `status` other than `ok` leaves the Offer as it was'), 'fail open'); +}); + +test('reference.md documents worth-it, and bootstrap.md lets a direct build follow its recommendation', () => { + const ref = read(REFERENCE_MD).toString('utf8'); + assert.ok(ref.split('\n').includes('## Worth it'), 'reference.md has a Worth it section'); + for (const word of ['THRESHOLD', 'task-slots', 'back-test', 'lite', 'checks.commands', '--runs', 'host_budget']) { + assert.ok(ref.includes(word), `reference.md lacks ${word}`); + } + const boot = flat(join(SKILL_DIR, 'hooks', 'bootstrap.md')); + assert.ok(boot.includes('Use a superpowers execution mode only when parallel-lanes steps aside, recommends a direct ' + + 'build and the user picks it, or the user explicitly asks for one.'), boot); +}); + diff --git a/tests/task-checks.test.mjs b/tests/task-checks.test.mjs index f555a63..b0787a3 100644 --- a/tests/task-checks.test.mjs +++ b/tests/task-checks.test.mjs @@ -122,6 +122,168 @@ test('a widen path widens even inside the Files list', () => { assert.deepEqual(codes(json), ['widen_path']); }); +test('a widen entry naming an existing directory without a trailing / covers files under it', () => { + // 1.6.0 deferred minor: 'test/helpers' (no '/') matched only the equal + // path, so a change under the directory never widened. + const r = repo({ 'test/helpers/h.js': '1\n', 'src/a.js': '1\n' }); + writeFileSync(join(r.dir, 'test/helpers/h.js'), '2\n'); + const { json } = run(r, { args: ['--declared', 'test/helpers/h.js', '--widen', 'test/helpers'] }); + assert.equal(json.policy.selected, 'widened'); + assert.deepEqual(codes(json), ['widen_path']); + // A --declared entry keeps the strict rule: without '/' it covers the equal path only. + const declared = run(r, { args: ['--declared', 'test/helpers'] }); + assert.deepEqual(codes(declared.json), ['outside_files']); + // A widen entry that is a file, or names nothing, still covers the equal path only. + const file = run(r, { args: ['--declared', 'test/helpers/h.js', '--widen', 'src/a.js', '--widen', 'gone'] }); + assert.equal(file.json.policy.selected, 'targeted'); +}); + +// The ledger's ruling events and owner map, as the run writes them. +function ledgerFile(r, stem, events) { + writeFileSync(join(r.ledger, `${stem}.jsonl`), events.map((e) => JSON.stringify(e)).join('\n') + '\n'); +} +function ownerMap(r, files) { + writeFileSync(join(r.ledger, 'owners.json'), JSON.stringify({ + tasks: { U1: { where: 'alpha', pos: 0 }, U2: { where: 'alpha', pos: 1 }, U9: { where: 'beta', pos: 0 } }, files })); +} +const permit = (task, files, extra = {}) => ({ event: 'ruling', task, by: 'adjudicator', text: 'Ruling: x - y - z', files, ...extra }); + +test('a file an adjudicator ruling in force permits counts as declared, and the policy names the ruling', () => { + // d1a DI3: the pre-flight adjudicator permitted ToolRackSlot.cs (no task listed it), filed under _run; + // task-checks widened anyway (outside_files), 1089 s instead of about 600 s. + const r = repo({ 'src/a.js': '1\n' }); + writeFileSync(join(r.dir, 'src/a.js'), '2\n'); + writeFileSync(join(r.dir, 'src/slot.js'), 'new\n'); + ledgerFile(r, '_run', [permit('_run', ['src/slot.js'])]); + ownerMap(r, { 'src/a.js': ['U1'] }); + const { status, json } = run(r, { args: ['--declared', 'src/a.js'] }); + assert.equal(status, 0); + assert.equal(json.policy.selected, 'targeted'); + assert.deepEqual(json.policy.permitted, [{ path: 'src/slot.js', ruling: '_run-R1' }]); + // Filed under the unit's own task (in its lane file), the files count whatever the owner map says. + ownerMap(r, { 'src/a.js': ['U1'], 'src/slot.js': ['U9'] }); + rmSync(join(r.ledger, '_run.jsonl')); + ledgerFile(r, 'alpha', [permit('U1', ['src/slot.js'])]); + assert.deepEqual(run(r, { args: ['--declared', 'src/a.js'] }).json.policy.permitted, + [{ path: 'src/slot.js', ruling: 'alpha-R1' }]); +}); + +test('only an adjudicator ruling in force, for this unit or an unowned path, permits a file', () => { + const r = repo({ 'src/a.js': '1\n' }); + writeFileSync(join(r.dir, 'src/a.js'), '2\n'); + writeFileSync(join(r.dir, 'src/slot.js'), 'new\n'); + const widens = (why) => { + const { status, json } = run(r, { args: ['--declared', 'src/a.js'] }); + assert.equal(status, 0, why); + assert.deepEqual(codes(json), ['outside_files'], why); + assert.deepEqual(json.policy.permitted, [], why); + }; + const reset = () => { for (const n of readdirSync(r.ledger)) if (n.endsWith('.jsonl') || n === 'owners.json') rmSync(join(r.ledger, n)); }; + ownerMap(r, { 'src/a.js': ['U1'] }); + // An implementer's own ruling never permits (it could switch off its own widening). + ledgerFile(r, 'alpha', [permit('U1', ['src/slot.js'], { by: 'implementer' })]); + widens('implementer ruling'); + ledgerFile(r, 'alpha', [permit('U1', ['src/slot.js'], { by: 'reviewer' })]); + widens('reviewer ruling'); + // Another task's ruling. + ledgerFile(r, 'alpha', [permit('U2', ['src/slot.js'])]); + widens('another task'); + reset(); + // A _run ruling for a path another task owns, or with no owner map to tell. + ledgerFile(r, '_run', [permit('_run', ['src/slot.js'])]); + ownerMap(r, { 'src/a.js': ['U1'], 'src/slot.js': ['U9'] }); + widens('owned elsewhere'); + rmSync(join(r.ledger, 'owners.json')); + widens('no owner map'); + // A _run ruling's unowned path counts only for a unit declaring a file in the same folder + // (d1a: ToolRackSlot.cs beside DI3's ToolRack.cs), so it does not exempt every other task. + ownerMap(r, { 'src/a.js': ['U1'] }); + mkdirSync(join(r.dir, 'other')); + writeFileSync(join(r.dir, 'other/far.js'), 'new\n'); + rmSync(join(r.dir, 'src/slot.js')); + ledgerFile(r, '_run', [permit('_run', ['other/far.js'])]); + widens('another folder'); + rmSync(join(r.dir, 'other'), { recursive: true }); + writeFileSync(join(r.dir, 'src/slot.js'), 'new\n'); + // An overridden ruling is no longer in force. + ownerMap(r, { 'src/a.js': ['U1'] }); + ledgerFile(r, '_run', [permit('_run', ['src/slot.js']), permit('_run', [], { overrides: ['_run-R1'] })]); + widens('overridden'); + // An unreadable ledger keeps the strict rule. + writeFileSync(join(r.ledger, '_run.jsonl'), '{not json\n'); + widens('unreadable ledger'); +}); + +test('a permit names files only: a directory, or an owned folder, never covers another lane\'s file', () => { + // Final review (1.6.4): a _run adjudicator ruling naming "src/sub/" was not in the owner map + // (it lists files) and became a directory entry covering U9's src/sub/owned.js. + const r = repo({ 'src/a.js': '1\n', 'src/sub/owned.js': '1\n', 'b.js': '1\n' }); + writeFileSync(join(r.dir, 'src/a.js'), '2\n'); + writeFileSync(join(r.dir, 'src/sub/owned.js'), '2\n'); + const widens = (why, declared = 'src/a.js') => { + const { status, json } = run(r, { args: ['--declared', declared] }); + assert.equal(status, 0, why); + assert.deepEqual(codes(json), ['outside_files'], why); + assert.deepEqual(json.policy.permitted, [], why); + }; + ownerMap(r, { 'src/a.js': ['U1'], 'src/sub/owned.js': ['U9'] }); + for (const dir of ['src/sub/', 'src/sub\\', 'src', './', '.']) { + ledgerFile(r, '_run', [permit('_run', [dir])]); + widens(`_run permit ${dir}`); + // A ruling filed under the unit's own task names files only, too. + rmSync(join(r.ledger, '_run.jsonl')); + ledgerFile(r, 'alpha', [permit('U1', [dir])]); + widens(`task permit ${dir}`); + rmSync(join(r.ledger, 'alpha.jsonl')); + } + // A folder another task's Files list names owns every path under it. + ownerMap(r, { 'src/a.js': ['U1'], 'src/sub': ['U9'] }); + ledgerFile(r, '_run', [permit('_run', ['src/sub/owned.js'])]); + widens('owned folder'); +}); + +test('the permit folder guard: a declared directory is its own folder, and folders compare by case', () => { + // Deferred minors of the 1.6.4 final review, included in 1.6.4. + const r = repo({ 'src/a.js': '1\n', 'Src2/a.js': '1\n' }); + writeFileSync(join(r.dir, 'src/a.js'), '2\n'); + writeFileSync(join(r.dir, 'build.cfg'), 'new\n'); + ownerMap(r, { 'src/a.js': ['U1'] }); + ledgerFile(r, '_run', [permit('_run', ['build.cfg'])]); + // A unit declaring only the directory src/ is not in build.cfg's (root) folder. + const dir = run(r, { args: ['--declared', 'src/'] }); + assert.deepEqual(codes(dir.json), ['outside_files']); + assert.deepEqual(dir.json.policy.permitted, []); + // A declared root file is. + writeFileSync(join(r.dir, 'root.txt'), 'x\n'); + rmSync(join(r.dir, 'src/a.js')); + git(r.dir, 'checkout', '--', 'src/a.js'); + const root = run(r, { args: ['--declared', 'root.txt'] }); + assert.deepEqual(root.json.policy.permitted, [{ path: 'build.cfg', ruling: '_run-R1' }]); + rmSync(join(r.dir, 'root.txt')); + rmSync(join(r.dir, 'build.cfg')); + // Src2/new.js is not in the folder of a declared src2/a.js on a case-sensitive comparison. + writeFileSync(join(r.dir, 'Src2/new.js'), 'x\n'); + ledgerFile(r, '_run', [permit('_run', ['Src2/new.js'])]); + const cased = run(r, { args: ['--declared', 'src2/a.js'] }); + assert.deepEqual(cased.json.policy.permitted, []); +}); + +test('a malformed owner map, and a backslash or mixed-case permit of an owned file, permit nothing', () => { + const r = repo({ 'src/a.js': '1\n', 'src/sub/owned.js': '1\n' }); + writeFileSync(join(r.dir, 'src/a.js'), '2\n'); + writeFileSync(join(r.dir, 'src/sub/owned.js'), '2\n'); + ledgerFile(r, '_run', [permit('_run', ['src/sub/owned.js'])]); + writeFileSync(join(r.ledger, 'owners.json'), '{"tasks": [], "files": 3}'); + assert.deepEqual(run(r, { args: ['--declared', 'src/sub/x.js', '--declared', 'src/a.js'] }).json.policy.permitted, []); + ownerMap(r, { 'src/a.js': ['U1'], 'src/sub/owned.js': ['U9'] }); + for (const path of ['src\\sub\\owned.js', 'SRC/Sub/owned.js']) { + ledgerFile(r, '_run', [permit('_run', [path])]); + const res = run(r, { args: ['--declared', 'src/sub/x.js', '--declared', 'src/a.js'] }); + assert.deepEqual(res.json.policy.permitted, [], path); + assert.deepEqual(codes(res.json), ['outside_files'], path); + } +}); + test('an untracked file outside the Files list widens', () => { const r = repo({ 'src/a.js': '1\n' }); writeFileSync(join(r.dir, 'new.js'), 'x\n'); @@ -130,6 +292,27 @@ test('an untracked file outside the Files list widens', () => { assert.deepEqual(json.policy.changed, ['new.js']); }); +test('the printed report caps long path lists; the evidence file keeps them whole', () => { + // 1.6.0 deferred minor: the printed line carried every changed and outside path. + const r = repo({ 'src/a.js': '1\n' }); + const names = Array.from({ length: 60 }, (_, i) => `gen/${'x'.repeat(40)}${String(i).padStart(2, '0')}.js`); + mkdirSync(join(r.dir, 'gen')); + for (const n of names) writeFileSync(join(r.dir, n), 'x\n'); + const { status, json } = run(r, { args: ['--declared', 'src/a.js'] }); + assert.equal(status, 0); + const saved = JSON.parse(readFileSync(join(r.ledger, 'checks', 'task-U1.1.json'), 'utf8')); + assert.deepEqual(saved.policy.changed, names); + assert.ok(saved.policy.reasons[0].detail.includes(names[59])); + assert.equal(saved.printed_cut, undefined); + assert.deepEqual(json.policy.changed, names.slice(0, 50)); + assert.ok(json.policy.reasons[0].detail.length <= 2100); + assert.ok(!json.policy.reasons[0].detail.includes(names[59])); + assert.match(json.printed_cut, /50 of 60 changed paths/); + assert.ok(json.printed_cut.includes('task-U1.1.json')); + // Everything else is the file's report. + assert.deepEqual({ ...json, policy: null, printed_cut: undefined }, { ...saved, policy: null, printed_cut: undefined }); +}); + test('a rename counts both paths', () => { const r = repo({ 'src/a.js': '1\n' }); git(r.dir, 'mv', 'src/a.js', 'src/z.js'); diff --git a/tests/worth-it-backtest.test.mjs b/tests/worth-it-backtest.test.mjs new file mode 100644 index 0000000..97a4dfe --- /dev/null +++ b/tests/worth-it-backtest.test.mjs @@ -0,0 +1,84 @@ +// tools/worth-it-backtest.py (1.6.4): the back-test behind worth-it's threshold, +// run on a copy of a history dir. It prints tables, not a verdict; the test +// checks it runs on a small fixture archive and refuses the live history dir. +import { test, after } from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { mkdirSync, rmSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { SKILL_DIR } from './harness.mjs'; +import { mergeEnv, tempDir } from './platform.mjs'; + +const SCRIPT = join(SKILL_DIR, 'tools', 'worth-it-backtest.py'); +const TMP = tempDir('pl-backtest-'); +after(() => rmSync(TMP, { recursive: true, force: true })); + +function env(extra = {}) { + const e = mergeEnv(process.env); + for (const key of Object.keys(e)) { + if (['PL_HISTORY_DIR', 'CLAUDE_CONFIG_DIR'].includes(key.toUpperCase())) delete e[key]; + } + return { ...e, ...extra }; +} + +const T = (day, s) => new Date(Date.UTC(2026, 9, day, 10) + s * 1000).toISOString(); +// A run: tasks [{id, lane, s}] (lane null: prelude), its day, project. +function archive(runs) { + const dir = join(TMP, `hist-${runs.length}-${Date.now()}`); + mkdirSync(dir, { recursive: true }); + const entries = []; + for (const r of runs) { + const folder = join(dir, r.id); + mkdirSync(folder); + const lanes = {}; + const prelude = []; + for (const t of r.tasks) { + if (t.lane === null) prelude.push({ id: t.id }); + else (lanes[t.lane] = lanes[t.lane] || []).push({ id: t.id }); + } + writeFileSync(join(folder, 'p.lanes.json'), JSON.stringify({ prelude, join: [], + lanes: Object.entries(lanes).map(([id, tasks]) => ({ id, tasks })), limits: { max_parallel_lanes: 5 } })); + const agents = r.tasks.map((t) => ({ task: t.id, launch: 'wf_1', start: T(r.day, 600), end: T(r.day, 600 + t.s) })); + const end = 600 + Math.max(...r.tasks.map((t) => t.s)); + writeFileSync(join(folder, 'autopsy.json'), JSON.stringify({ time: { agents, phases: [ + { phase: 'Pre-flight', launch: 'wf_1', start: T(r.day, 0), end: T(r.day, 600) }, + { phase: 'Lane a', launch: 'wf_1', start: T(r.day, 600), end: T(r.day, end) }, + { phase: 'Final review', launch: 'wf_1', start: T(r.day, end), end: T(r.day, end + 900) }, + ] } })); + entries.push({ run_id: r.id, folder: r.id, project: r.project, status: 'complete', first_launch: T(r.day, 0) }); + } + writeFileSync(join(dir, 'index.json'), JSON.stringify({ version: 1, runs: entries })); + return dir; +} + +const RUNS = [ + { id: 'a1', day: 1, project: '/x/p', tasks: [{ id: 'T1', lane: 'a', s: 600 }, { id: 'T2', lane: 'b', s: 900 }] }, + { id: 'a2', day: 2, project: '/x/p', tasks: [{ id: 'T1', lane: null, s: 300 }, { id: 'T2', lane: 'a', s: 600 }, + { id: 'T3', lane: 'b', s: 600 }, { id: 'T4', lane: 'c', s: 600 }] }, + { id: 'b1', day: 3, project: '/y/q', tasks: [{ id: 'T1', lane: 'a', s: 1200 }, { id: 'T2', lane: 'a', s: 1200 }] }, +]; + +test('the back-test prints its tables on a fixture archive', () => { + const dir = archive(RUNS); + const r = spawnSync('python3', ['-B', SCRIPT, dir], { encoding: 'utf8', env: env() }); + assert.equal(r.status, 0, r.stderr); + for (const marker of ['in-sample:', 'out-of-sample:', '(F - R) / T quartiles', 'agrees']) { + assert.ok(r.stdout.includes(marker), `${marker} missing:\n${r.stdout}`); + } + // a2: prelude T1, lanes [1, 1, 1] across 3 workers: 2 task-slots of 4 tasks, saved 2. + assert.match(r.stdout, /^a2 .* N\s+4 .* slots\s+2 /m); + // b1: one lane, saved 0. + assert.match(r.stdout, /^b1 .* N\s+2 .* slots\s+2 /m); + const median = spawnSync('python3', ['-B', SCRIPT, dir, 'median'], { encoding: 'utf8', env: env() }); + assert.equal(median.status, 0, median.stderr); + assert.match(median.stdout, /aggregate: median/); +}); + +test('the back-test refuses the live history dir', () => { + const dir = archive(RUNS); + const r = spawnSync('python3', ['-B', SCRIPT, dir], { encoding: 'utf8', env: env({ PL_HISTORY_DIR: dir }) }); + assert.equal(r.status, 2); + assert.match(r.stderr, /copy the history first/); + const usage = spawnSync('python3', ['-B', SCRIPT], { encoding: 'utf8', env: env() }); + assert.equal(usage.status, 2); +}); diff --git a/tests/worth-it.test.mjs b/tests/worth-it.test.mjs new file mode 100644 index 0000000..7ed0e3c --- /dev/null +++ b/tests/worth-it.test.mjs @@ -0,0 +1,225 @@ +// scripts/worth-it (1.6.4): from the session's lane layout, the task-slots a +// run needs (prelude, lanes queued whole in order across the lanes at once, +// join), the task-times it saves, and a recommendation (direct when it saves 1 +// or fewer); plus a scale line from the project's archived runs. +import { test, after } from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { SKILL_DIR } from './harness.mjs'; +import { mergeEnv, tempDir } from './platform.mjs'; + +const SCRIPT = join(SKILL_DIR, 'scripts', 'worth-it'); +const TMP = tempDir('pl-worth-it-'); +after(() => rmSync(TMP, { recursive: true, force: true })); + +function env() { + const e = mergeEnv(process.env); + for (const key of Object.keys(e)) { + if (['PL_HISTORY_DIR', 'CLAUDE_CONFIG_DIR'].includes(key.toUpperCase())) delete e[key]; + } + return e; +} + +// The script's module functions, loaded as task-checks loads ledger (no bytecode). +function module(lines) { + const code = [ + 'import json, sys, importlib.machinery, importlib.util', + 'sys.dont_write_bytecode = True', + `loader = importlib.machinery.SourceFileLoader("_wi", ${JSON.stringify(SCRIPT)})`, + 'spec = importlib.util.spec_from_loader("_wi", loader)', + 'W = importlib.util.module_from_spec(spec)', + 'loader.exec_module(W)', + ...lines, + ].join('\n'); + const r = spawnSync('python3', ['-B', '-c', code], { encoding: 'utf8' }); + assert.equal(r.status, 0, r.stderr); + return JSON.parse(r.stdout); +} + +let plans = 0; +function plan(n) { + plans += 1; + const path = join(TMP, `plan${plans}.md`); + const body = Array.from({ length: n }, (_, i) => `### Task T${i + 1}: Step ${i + 1}\n\nDo it.\n`).join('\n'); + writeFileSync(path, `# Plan\n\n${body}`); + return path; +} + +function run(args) { + const r = spawnSync('python3', ['-B', SCRIPT, ...args], { encoding: 'utf8', env: env() }); + let json = null; + try { json = JSON.parse(r.stdout); } catch { /* left null */ } + return { status: r.status, json, stderr: r.stderr }; +} + +const NOWHERE = join(TMP, 'no-history'); + +test('slots queues whole lanes in order', () => { + assert.deepEqual(module(['print(json.dumps([W.slots(0, [3, 3, 3, 3, 3, 3], 0, 5), W.slots(2, [4, 1, 1], 3, 2),' + + ' W.slots(0, [], 5, 1), W.slots(1, [2, 2], 1, 2)]))']), [6, 9, 5, 4]); +}); + +test('the threshold is 1', () => { + assert.deepEqual(module(['print(json.dumps([W.THRESHOLD, W.decide(1), W.decide(2), W.decide(0), W.decide(-1)]))']), + [1, 'direct', 'parallel', 'direct', 'direct']); +}); + +test('a single lane is direct', () => { + const r = run([plan(4), '--max-lanes', '3', '--lane', 'T1,T2,T3,T4', '--history-dir', NOWHERE]); + assert.equal(r.status, 0, r.stderr); + assert.equal(r.json.status, 'ok'); + assert.deepEqual([r.json.tasks, r.json.lanes, r.json.lanes_at_once, r.json.slots, r.json.saved, r.json.threshold, + r.json.recommend], [4, 1, 1, 4, 0, 1, 'direct']); +}); + +test('a two-lane layout that saves 2 is parallel', () => { + const r = run([plan(5), '--max-lanes', '2', '--lane', 'T1,T2,T3', '--lane', 'T4,T5', '--history-dir', NOWHERE]); + assert.equal(r.status, 0, r.stderr); + assert.deepEqual([r.json.slots, r.json.saved, r.json.recommend], [3, 2, 'parallel']); + // The prelude and join run in sequence around the lanes; empty values are allowed. + const p = run([plan(6), '--max-lanes', '5', '--prelude', 'T1', '--lane', 'T2,T3', '--lane', 'T4', '--join', 'T5,T6', + '--history-dir', NOWHERE]); + assert.deepEqual([p.json.slots, p.json.saved, p.json.recommend], [5, 1, 'direct']); + const e = run([plan(3), '--max-lanes', '2', '--prelude', '', '--lane', 'T1', '--lane', 'T2,T3', '--join', '', + '--history-dir', NOWHERE]); + assert.deepEqual([e.json.status, e.json.slots], ['ok', 2]); +}); + +test('bad layouts are unavailable with exit 0', () => { + const p = plan(4); + for (const [why, layout] of [ + ['twice in two lanes', ['--lane', 'T1,T2', '--lane', 'T2,T3']], + ['in prelude and a lane', ['--prelude', 'T1', '--lane', 'T1,T2']], + ['not in the plan', ['--lane', 'T1,T9']], + ['no task at all', ['--lane', '']], + ]) { + const r = run([p, '--max-lanes', '3', ...layout, '--history-dir', NOWHERE]); + assert.equal(r.status, 0, `${why}: ${r.stderr}`); + assert.equal(r.json.status, 'unavailable', why); + assert.ok(typeof r.json.reason === 'string' && r.json.reason.length > 0, why); + } + // An unreadable plan is unavailable too. + const r = run([join(TMP, 'missing.md'), '--max-lanes', '3', '--lane', 'T1', '--history-dir', NOWHERE]); + assert.deepEqual([r.status, r.json.status], [0, 'unavailable']); +}); + +test('usage errors exit 2', () => { + const p = plan(3); + for (const args of [ + [p, '--lane', 'T1,T2,T3'], + [p, '--max-lanes', '0', '--lane', 'T1'], + [p, '--max-lanes', 'x', '--lane', 'T1'], + [p, '--max-lanes', '2', '--lane', 'T1', '--bogus', '1'], + ['--max-lanes', '2', '--lane', 'T1'], + ]) { + const r = run(args); + assert.equal(r.status, 2, JSON.stringify(args)); + assert.match(r.stderr, /usage/, JSON.stringify(args)); + } +}); + +// A history dir with index.json and run folders, each holding a manifest and an +// autopsy.json whose one launch ran tasks A1 and B1 for the given seconds each. +const T = (s) => new Date(Date.UTC(2026, 9, 15, 10) + s * 1000).toISOString(); +function history(runs) { + plans += 1; + const dir = join(TMP, `hist${plans}`); + mkdirSync(dir, { recursive: true }); + const entries = []; + for (const r of runs) { + entries.push({ run_id: r.id, folder: r.id, project: r.project, status: r.status || 'complete' }); + const folder = join(dir, r.id); + mkdirSync(folder); + writeFileSync(join(folder, 'p.lanes.json'), JSON.stringify({ prelude: [], join: [], + lanes: [{ id: 'a', tasks: [{ id: 'A1' }, { id: 'B1' }] }] })); + if (r.autopsy === 'missing') continue; + if (r.autopsy === 'garbage') { writeFileSync(join(folder, 'autopsy.json'), '{not json'); continue; } + const end = r.autopsy === 'no-agents' ? 60 : Math.max(r.a1, r.b1); + const agents = r.autopsy === 'no-agents' ? undefined : [ + { task: 'A1', launch: 'wf_1', start: T(0), end: T(r.a1) }, + { task: 'B1', launch: 'wf_1', start: T(0), end: T(r.b1) }, + ]; + writeFileSync(join(folder, 'autopsy.json'), JSON.stringify({ time: { agents, + phases: [{ phase: 'Lane a', launch: 'wf_1', start: T(0), end: T(end) }] } })); + } + writeFileSync(join(dir, 'index.json'), JSON.stringify({ version: 1, runs: entries })); + return dir; +} + +test('scale line from fixture history', () => { + const h = history([ + { id: 'r1', project: '/x/proj', a1: 600, b1: 1200 }, + { id: 'r2', project: '/x/proj', a1: 300, b1: 300 }, + { id: 'r3', project: '/x/proj', a1: 9000, b1: 9000, status: 'stopped' }, + { id: 'r4', project: '/x/other', a1: 50, b1: 50 }, + ]); + const r = run([plan(4), '--max-lanes', '2', '--lane', 'T1,T2', '--lane', 'T3,T4', '--project', '/x/proj', + '--history-dir', h]); + assert.equal(r.status, 0, r.stderr); + assert.deepEqual(r.json.scale, { runs: ['r1', 'r2'], task_s_mean: 600, task_s_min: 300, task_s_max: 900 }); + // Without --project there is no scale line. + const n = run([plan(4), '--max-lanes', '2', '--lane', 'T1,T2', '--lane', 'T3,T4', '--history-dir', h]); + assert.equal(n.json.scale, null); +}); + +test('scale line matches the project in any path form', () => { + const h = history([{ id: 'w1', project: 'C:/Studio/Proj', a1: 100, b1: 100 }]); + // The Git Bash form /c/x is a drive path only on Windows. + const forms = ['C:/Studio/Proj', 'c:\\studio\\proj', 'C:/Studio/Proj/', 'Proj'] + .concat(process.platform === 'win32' ? ['/c/Studio/Proj', '/c/studio/proj/'] : []); + for (const name of forms) { + const r = run([plan(3), '--max-lanes', '2', '--lane', 'T1', '--lane', 'T2,T3', '--project', name, '--history-dir', h]); + assert.equal(r.status, 0, r.stderr); + assert.deepEqual(r.json.scale && r.json.scale.runs, ['w1'], name); + } +}); + +test('scale line skips unreadable or incomplete autopsies', () => { + const h = history([ + { id: 'm1', project: '/x/proj', autopsy: 'missing' }, + { id: 'g1', project: '/x/proj', autopsy: 'garbage' }, + { id: 'n1', project: '/x/proj', autopsy: 'no-agents' }, + ]); + const r = run([plan(3), '--max-lanes', '2', '--lane', 'T1', '--lane', 'T2,T3', '--project', '/x/proj', '--history-dir', h]); + assert.equal(r.status, 0, r.stderr); + assert.equal(r.json.scale, null); + assert.ok(r.json.notes.some((n) => /scale/.test(n)), JSON.stringify(r.json.notes)); + assert.deepEqual([r.json.saved, r.json.recommend], [1, 'direct']); + const ok = history([{ id: 'k1', project: '/x/proj', a1: 60, b1: 60 }, { id: 'g2', project: '/x/proj', autopsy: 'garbage' }]); + const s = run([plan(3), '--max-lanes', '2', '--lane', 'T1', '--lane', 'T2,T3', '--project', '/x/proj', '--history-dir', ok]); + assert.deepEqual(s.json.scale.runs, ['k1']); + // The note names every cause a run is skipped for, a run with no task agents included. + assert.ok(s.json.notes.some((n) => n.includes('or no task agents')), JSON.stringify(s.json.notes)); +}); + +test('a malformed archived run never changes the recommendation', () => { + // Final review (1.6.4): an agent whose launch is a list, or a manifest whose lanes is a + // number, raised TypeError through the scale line and made the whole result unavailable. + const h = history([{ id: 'ok1', project: '/x/proj', a1: 60, b1: 60 }, + { id: 'bad1', project: '/x/proj', a1: 60, b1: 60 }, { id: 'bad2', project: '/x/proj', a1: 60, b1: 60 }]); + const auto = join(h, 'bad1', 'autopsy.json'); + const a = JSON.parse(readFileSync(auto, 'utf8')); + a.time.agents[0].launch = ['wf_1']; + writeFileSync(auto, JSON.stringify(a)); + writeFileSync(join(h, 'bad2', 'p.lanes.json'), JSON.stringify({ prelude: [], join: [], lanes: 5 })); + const r = run([plan(4), '--max-lanes', '2', '--lane', 'T1,T2', '--lane', 'T3,T4', '--project', '/x/proj', '--history-dir', h]); + assert.equal(r.status, 0, r.stderr); + assert.deepEqual([r.json.status, r.json.saved, r.json.recommend], ['ok', 2, 'parallel']); + assert.ok(r.json.scale.runs.includes('ok1'), JSON.stringify(r.json)); + assert.ok(!r.json.scale.runs.includes('bad2'), JSON.stringify(r.json)); +}); + +test('no history dir is fine', () => { + const r = run([plan(3), '--max-lanes', '2', '--lane', 'T1', '--lane', 'T2,T3', '--project', '/x/proj', '--history-dir', NOWHERE]); + assert.equal(r.status, 0, r.stderr); + assert.deepEqual([r.json.status, r.json.scale], ['ok', null]); + // An index that is not a run index leaves the scale line out, never the result. + plans += 1; + const bad = join(TMP, `hist${plans}`); + mkdirSync(bad); + writeFileSync(join(bad, 'index.json'), '[]'); + const b = run([plan(3), '--max-lanes', '2', '--lane', 'T1', '--lane', 'T2,T3', '--project', '/x/proj', '--history-dir', bad]); + assert.deepEqual([b.status, b.json.status, b.json.scale], [0, 'ok', null]); +}); diff --git a/tools/worth-it-backtest.py b/tools/worth-it-backtest.py new file mode 100755 index 0000000..6990f06 --- /dev/null +++ b/tools/worth-it-backtest.py @@ -0,0 +1,158 @@ +#!/usr/bin/env python3 +"""The back-test behind scripts/worth-it's THRESHOLD (1.6.4). + +usage: python3 -B tools/worth-it-backtest.py HISTORY_COPY_DIR [mean|median] + +Reads a COPY of a history dir (index.json and the run folders, each with its +autopsy.json and *.lanes.json); it refuses the live history dir +(_history.history_dir(): PL_HISTORY_DIR, else the config dir's), exit 2. It +writes nothing and prints tables, not a verdict: + +- per run, for the latest launch that ran task agents + (_history.latest_task_launch): N tasks run, L lanes, the task-slots S of + the manifest's layout limited to those tasks (worth-it's slots, with the + manifest's max_parallel_lanes), the actual wall time, the in-sample error + of F + S x T (T = the mean, or median, of the tasks' summed agent time; F + and R = _history.run_figures' fixed_s and final_review_s), and the + out-of-sample error from the means of earlier complete runs (the same + project first, else all); +- the in-sample and out-of-sample error summaries; +- the quartiles of (F - R) / T, the fixed phases a run adds beyond the final + review a direct build also ends with, in task-times; +- for thresholds 1 and 2, how often the rule (direct when N - S is at most + the threshold) agrees with each run's own comparison (direct = N x T + R + against the actual wall, a model, not a measured direct build). +Exit 0, or 2 on a usage error or the live history dir. +""" +import glob +import importlib.machinery +import importlib.util +import json +import os +import statistics +import sys + +sys.dont_write_bytecode = True +SCRIPTS = os.path.join(os.path.dirname(os.path.dirname(os.path.realpath(__file__))), "scripts") +sys.path.insert(0, SCRIPTS) +import _history # noqa: E402 + +_loader = importlib.machinery.SourceFileLoader("_worth_it", os.path.join(SCRIPTS, "worth-it")) +_spec = importlib.util.spec_from_loader("_worth_it", _loader) +W = importlib.util.module_from_spec(_spec) +_loader.exec_module(W) + +USAGE = "usage: worth-it-backtest.py HISTORY_COPY_DIR [mean|median]" + + +def _read_json(path): + with open(path, encoding="utf-8") as f: + return json.load(f) + + +def load_runs(hdir, aggregate): + runs = [] + for entry in _history.read_index(hdir)["runs"]: + folder = _history.run_folder(hdir, entry) + manifests = sorted(glob.glob(os.path.join(glob.escape(folder), "*.lanes.json"))) if folder else [] + if not manifests: + continue + try: + analysis = _read_json(os.path.join(folder, "autopsy.json")) + manifest = _read_json(manifests[0]) + except (OSError, ValueError): + continue + found = _history.latest_task_launch(analysis, manifest) + figures = _history.run_figures(analysis, manifest) + if found is None or figures is None: + continue + ran = found[1] + ids = lambda tasks: [t.get("id") for t in tasks or [] if isinstance(t, dict) and t.get("id") in ran] # noqa: E731 + lanes = [ids(lane.get("tasks")) for lane in manifest.get("lanes") or [] if isinstance(lane, dict)] + lanes = [lane for lane in lanes if lane] + cap = (manifest.get("limits") or {}).get("max_parallel_lanes") or len(lanes) or 1 + at_once = max(1, min(len(lanes), cap)) + prelude, join = ids(manifest.get("prelude")), ids(manifest.get("join")) + project = (entry.get("project") or "?").replace("\\", "/").rstrip("/").rsplit("/", 1)[-1] + runs.append({ + "run": entry.get("run_id"), "project": project, "status": entry.get("status"), + "date": entry.get("first_launch") or entry.get("folder") or "", + "N": len(ran), "lanes": len(lanes), "slots": W.slots(len(prelude), [len(x) for x in lanes], len(join), at_once), + "wall": figures["wall_s"], "F": figures["fixed_s"], "R": figures["final_review_s"], + "T": aggregate(ran.values()), + }) + runs.sort(key=lambda r: r["date"]) + return runs + + +def summary(errors): + if not errors: + return "n=0" + return "n=%d median %+.0f%% range %+.0f%%..%+.0f%%" % ( + len(errors), 100 * statistics.median(errors), 100 * min(errors), 100 * max(errors)) + + +def report(runs, label): + print("aggregate: %s; runs with data: %d" % (label, len(runs))) + print("%-10s %-20s %3s %2s %5s %8s | %9s %5s | %8s %5s %8s %s" % ( + "run", "project", "N", "ln", "slots", "actual", "in-sample", "err", "oos", "err", "direct", "source")) + in_sample, out_sample = [], [] + for i, r in enumerate(runs): + estimate = r["F"] + r["slots"] * r["T"] + in_sample.append(estimate / r["wall"] - 1) + line = "%-10s %-20s N %2d %2d slots %2d %8.1f | %9.1f %+4.0f%% |" % ( + r["run"], r["project"][:20], r["N"], r["lanes"], r["slots"], r["wall"] / 60, estimate / 60, + 100 * in_sample[-1]) + prior = [p for p in runs[:i] if p["status"] == "complete"] + same = [p for p in prior if p["project"] == r["project"]] + base, source = (same, "project") if same else (prior, "history") + if base: + t = statistics.mean(p["T"] for p in base) + f = statistics.mean(p["F"] for p in base) + fr = statistics.mean(p["R"] for p in base) + oos = f + r["slots"] * t + out_sample.append(oos / r["wall"] - 1) + line += " %8.1f %+4.0f%% %8.1f %s(%d)" % (oos / 60, 100 * out_sample[-1], (r["N"] * t + fr) / 60, source, + len(base)) + else: + line += " (no earlier run)" + print(line) + print("in-sample:", summary(in_sample)) + print("out-of-sample:", summary(out_sample)) + ratios = sorted((r["F"] - r["R"]) / r["T"] for r in runs if r["T"] > 0) + if ratios: + q = (ratios[len(ratios) // 4], statistics.median(ratios), ratios[3 * len(ratios) // 4]) + print("(F - R) / T quartiles %.2f %.2f %.2f range %.2f %.2f" % (q + (ratios[0], ratios[-1]))) + for threshold in (1, 2): + agree, direct = 0, [] + for r in runs: + rule = "direct" if r["N"] - r["slots"] <= threshold else "parallel" + own = "direct" if r["N"] * r["T"] + r["R"] <= r["wall"] else "parallel" + agree += rule == own + if rule == "direct": + direct.append(r["run"]) + print("threshold %d: the rule agrees with each run's own comparison on %d of %d runs; direct for %s" % ( + threshold, agree, len(runs), ", ".join(direct) or "none")) + + +def main(argv): + if not argv or len(argv) > 2 or (len(argv) == 2 and argv[1] not in ("mean", "median")): + print(USAGE, file=sys.stderr) + return 2 + hdir = os.path.abspath(argv[0]) + live = _history.history_dir() + if os.path.normcase(os.path.realpath(hdir)) == os.path.normcase(os.path.realpath(live)): + print("worth-it-backtest: %s is the live history dir; copy the history first" % live, file=sys.stderr) + return 2 + label = argv[1] if len(argv) == 2 else "mean" + try: + runs = load_runs(hdir, statistics.mean if label == "mean" else statistics.median) + except _history.HistoryError as e: + print("worth-it-backtest: " + str(e), file=sys.stderr) + return 2 + report(runs, label) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:]))