From 15845b8665cc099aefba17f4878fd9d7b0968e20 Mon Sep 17 00:00:00 2001 From: "Aaron K. Clark" Date: Thu, 8 Oct 2026 01:29:41 -0500 Subject: [PATCH 1/2] Add --dry-run / --estimate priced pre-flight (#81) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A priced pre-flight that resolves config + active lanes, fetches the price table (cached 24h, same as --max-spend already does), and prints the per-lane and panel worst-case cost without dispatching anything. The estimate is the same number the budget gate uses against --max-spend / budget_usd, so a --dry-run verdict is the verdict a real dispatch under the same args would face. * Refuses --dry-run --list (keep --list as the offline roster view) and --dry-run --synthesize (synthesis reads a finished run). * Inherits --each --diff's refusal (#74); --each/--panel are also refused with --discuss (the tool refused it already). * --diff folds the readers' worst case into the panel total; the JSON shape names which readers were picked. * --discuss multiplies by discuss-rounds + 1, both on the panel total and on each per-lane cost so `sum(lanes[].cost)` matches. * --revise uses the longest revision packet as the worst case (#4 contract). * Per-lane breakdown carries both `free` and `priced`; an unknown price collapsing to (0, 0) is fail-open (#72) and renders as `unpriced`, not `free`. * Unpriced http lane + a budget used to slip past the gate unseen; the dry-run reports the verdict explicitly as `would_exceed_budget: null` under `--json` and names the lane in the human summary, so a budget is never silently bound against an unpriced lane (#72). * --json emits a stable machine shape (`estimated_worst_case`, `would_exceed_budget`, per-lane breakdown, `unpriced`) re-using the same key `spend_report` already uses post-run (#79 / #131), so callers reading `--json` pre- and post-run do not switch shapes. v1.4.0 — new user-visible flag, no breaking changes. 15 new tests in `TestDryRun`; existing 258 untouched and still green (273 total). Co-Authored-By: Claude Code --- BACKLOG.md | 2 +- README.md | 51 ++++++ roundtable | 182 ++++++++++++++++++++- tests/test_roundtable.py | 333 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 565 insertions(+), 3 deletions(-) diff --git a/BACKLOG.md b/BACKLOG.md index d7f8e8a..ff203db 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -112,7 +112,7 @@ The UI lives in its own private repo, replaces the fragile `latest:N` file-count heuristic with a real "round" ([#80](https://github.com/CryptoJones/FlatlineRoundtable/issues/80)) — run_id in PR #117, `latest-run` in PR #122 -- [ ] Add `--dry-run` / `--estimate` — a priced pre-flight that shows per-lane and +- [x] Add `--dry-run` / `--estimate` — a priced pre-flight that shows per-lane and panel worst-case cost without dispatching (and surfaces unpriced lanes) ([#81](https://github.com/CryptoJones/FlatlineRoundtable/issues/81)) - [ ] Transcripts accumulate forever — add bounded, opt-in retention that never diff --git a/README.md b/README.md index 60f62a8..5f5842c 100644 --- a/README.md +++ b/README.md @@ -271,9 +271,60 @@ roundtable --config DB # another store (default ~/.local/share/flatl roundtable --max-spend 0.50 # refuses BEFORE dispatch if the estimate exceeds it roundtable --panel --diff # report only where the lanes disagree (not with --each) roundtable --no-transcript +roundtable --each --dry-run - # priced pre-flight: per-lane and panel worst-case cost, no dispatch roundtable --each --revise latest:12 # optional second round — see below ``` +### `--dry-run` — priced pre-flight, no dispatch + +A worst-case panel estimate without running any lane. Same numbers the budget +gate uses against `--max-spend` / `budget_usd`, so a `--dry-run` verdict is the +verdict a real dispatch under the same args would face. Issues the +pre-flight surfaces: + +- which lane is the cost driver +- whether the panel would exceed the budget (`would_exceed_budget`) +- which lanes it could not price (the `#72` fail-open guard) + +Refuses `--dry-run --list` (use `--list` for the offline roster) and +`--dry-run --synthesize` (synthesis reads a finished run, not a fresh one). +Inherits `--each --diff`'s refusal (`#74`). + +```console +$ roundtable --each --dry-run - # the worst-case panel total +$ roundtable --each --dry-run --json - # machine shape: estimated_worst_case, would_exceed_budget, per-lane breakdown +$ roundtable --panel --diff --dry-run - # readers' cost folded in; readers listed by name +$ roundtable --discuss --dry-run - # cost = estimate * (discuss-rounds + 1); --each/--panel refused +``` + +`--dry-run` may dial out to load the OpenRouter price table (cached for 24h, +same as `--max-spend` already does). When it cannot reach the gateway the run +degrades to config-only prices; unpriced http lanes show up under `unpriced` +in `--json` and are called out in the human summary so a budget never silently +binds against an unknown lane. + +Machine shape (`--dry-run --json`). `estimated_worst_case` is the same key +`spend_report` already uses post-run (`#79`), so a UI / caller that reads it +once does not switch shapes between pre-flight and post-run. + +```json +{ + "mode": "each", + "lanes": [{"name": "A", "model": "openai/gpt-4o-mini", "harness": "http", + "vendor": "openrouter", "route": "https://openrouter.ai/api/v1", + "prompt_tokens": 1234, "completion_tokens": 2000, + "prompt_cost": 0.0012, "completion_cost": 0.0048, + "cost": 0.0060, "free": false, "priced": true}], + "estimated_worst_case": 0.0720, + "synthesis": null, + "discuss_passes": null, + "budget": 0.5, + "would_exceed_budget": false, + "unpriced": [], + "currency": "USD" +} +``` + Every transcript carries a `run_id`. `--each` hands its children one id, so the N per-lane transcripts of a fan-out can be grouped as one run; without `--run-id` a fresh one is generated per invocation. diff --git a/roundtable b/roundtable index 09961a5..ed26079 100755 --- a/roundtable +++ b/roundtable @@ -43,6 +43,14 @@ Usage: roundtable --discuss latest:12 --discuss-rounds 3 "focus on the cost claims" cat brief.md | roundtable --discuss new - start cold, no blind round roundtable --converse latest:12 the same flag, by its other name + roundtable --each --dry-run - priced pre-flight: per-lane and panel + worst-case cost without dispatching + (may dial out to load prices; #81). + Add --diff and you get the readers + too; multiply by discuss-rounds+1 + under --discuss. Refuses + --dry-run --list and + --dry-run --synthesize. Store (#105): the only configuration. Runs read lanes and secrets from it; YAML is retired, and `import-yaml` moves an old config in once. @@ -97,7 +105,7 @@ from pathlib import Path # Semantic version. Bump it in the PR that changes behaviour: patch for a fix, # minor for a new flag or mode, major for a change that breaks a caller. -__version__ = "1.3.0" +__version__ = "1.4.0" TRANSCRIPT_DIR = Path.home() / ".local" / "share" / "flatline-roundtable" / "transcripts" RUN_ID_RE = re.compile(r"[A-Za-z0-9_-][A-Za-z0-9._-]{0,63}") @@ -351,6 +359,151 @@ def spend_report(spent: float, estimate: float, budget, lanes: list[dict], "unpriced": [l["name"] for l in lanes if not lane_price_known(l, table)]} +def cmd_dry_run(a, cfg, lanes, est_prompt, prices, calib, defaults, + estimate, brief_on_record) -> int: + """Priced pre-flight (issue #81). Exit 0 without dispatching anything. + + `estimate` is whatever `estimate_run` (+ synthesis, *discuss passes) just + produced for the budget gate above. The dry-run path uses the same number, + so the gate and the human/JSON report never disagree; only the disposition + differs (the gate dies, the dry-run reports). `spend_report` already carries + the same `estimated_worst_case` key for post-run (#79), so dry-run shares + that name rather than a parallel `panel_total` (#131 dry-run / post-run + shape compat). + + `--list` was the offline roster view before this existed; it still is. A + `--each --diff` refusal (issue #74) inherits from main(): the synthesis + step is folded in here as elsewhere, but `--each --diff` is still refused. + """ + retries = int(cfg.get("retries", 1)) + attempts = 1 + max(0, retries) + discuss_factor = (a.discuss_rounds + 1) if a.discuss else 1 + + # Per-lane breakdown. `estimate_run` collapses to a total; the dry-run + # user wants to see which lane is the driver and whether any are unpriced. + per_lane = [] + for l in lanes: + pin, pout = lane_prices(l, prices) + priced = lane_price_known(l, prices) + in_tok = ((len(est_prompt) + len(l.get("personality") or "")) + / chars_per_token(l.get("model"), calib)) + out_tok = float(l.get("max_tokens", + defaults.get("max_tokens", 2000))) + p_cost = in_tok * pin * attempts + c_cost = out_tok * pout + cost = p_cost + c_cost + # `free` requires being priced at (0, 0). An unknown price that + # happens to fall through to (0, 0) is fail-open (#72) — call it + # unpriced, not free, so the UI can render it as unknown. + free = priced and pin == 0.0 and pout == 0.0 + # `--discuss` already has each lane speak once per pass and once to + # close, so per-lane cost is multiplied here so the JSON sum equals + # the panel total. The gate does the same on `estimate` above; here + # we mirror it on each entry to keep `sum(lanes[].cost)` honest. + per_lane.append({ + "name": l["name"], + "model": l.get("model") or "", + "vendor": l.get("vendor") or "", + "harness": l["harness"], + "route": safe_route(l.get("base_url")) or l.get("command") or "", + "prompt_tokens": int(round(in_tok * attempts * discuss_factor)), + "completion_tokens": int(round(out_tok * discuss_factor)), + "prompt_cost": round(p_cost * discuss_factor, 6), + "completion_cost": round(c_cost * discuss_factor, 6), + "cost": round(cost * discuss_factor, 6), + "free": free, + "priced": priced, + }) + + if a.diff: + # Same fold the budget gate applied; kept here so the JSON shape and + # human output include the readers and their cost. Synthesis is + # priced against the ORIGINAL brief on record, not the focus text in + # a revision round — same rule the gate already uses. + synth_cost = estimate_synthesis( + lanes, brief_on_record, prices, defaults, calib, + cfg.get("synthesizer"), + int(cfg.get("synthesizers", 2))) + readers = [r["name"] for r in pick_synthesizers( + lanes, cfg.get("synthesizer"), + int(cfg.get("synthesizers", 2)))] + # `--discuss` again: the readers participate in the discussion too, + # so their cost is folded by the same pass count to keep the total + # honest. `--panel --diff` alone (no --discuss) leaves `discuss_factor` + # at 1. + synthesis = {"cost": round(synth_cost * discuss_factor, 6), + "readers": readers} + else: + synthesis = None + + passes = (a.discuss_rounds + 1) if a.discuss else None + budget = a.max_spend if a.max_spend is not None else cfg.get("budget_usd") + unpriced = [l["name"] for l in lanes if not lane_price_known(l, prices)] + # `would_exceed_budget` is the same comparison the gate dies on: priced + # estimate vs budget. Under unpriced+budget we cannot tell (#72), so + # `null` rather than a misleading `false` that would read as "safe". + if unpriced and budget is not None: + would_exceed = None + else: + would_exceed = budget is not None and estimate > budget + + if a.json: + out = { + "mode": ("discuss" if a.discuss else + "revise" if a.revise else + "panel" if a.panel else "each"), + "lanes": per_lane, + "estimated_worst_case": round(estimate, 6), + "synthesis": synthesis, + "discuss_passes": passes, + "budget": budget, + "would_exceed_budget": would_exceed, + "unpriced": unpriced, + "currency": "USD", + } + print(json.dumps(out, indent=2, sort_keys=True)) + return 0 + + # Human output. Match the column widths of `roundtable --list` so the two + # read side by side; the cost lives in a fourth column to the right. + label_map = {"http": "http", "cli": "cli ", "acp": "acp "} + print(f"Pre-flight estimate for {len(lanes)} lane(s) — no dispatch.") + for r in per_lane: + route_h = label_map.get(r["harness"], r["harness"]) + if not r["priced"]: + tag = "unpriced" + elif r["free"]: + tag = "free" + else: + tag = f"${r['cost']:.4f}" + print(f" [{route_h:<4}] {r['name']:<18} {str(r['model'] or '-'):<42} {tag}") + print(f"\n panel worst case: ${estimate:.4f}") + if synthesis is not None: + n = len(synthesis["readers"]) + s = "reader" if n == 1 else "readers" + print(f" synthesis ({n} {s}): ${synthesis['cost']:.4f}") + if passes is not None: + print(f" discuss passes: {passes}") + if budget is not None: + if would_exceed is None: + verdict = "cannot tell — unpriced lanes; budget not yet binding" + elif would_exceed: + verdict = "would exceed budget — dispatch would refuse" + else: + verdict = "within budget on dispatch" + print(f" budget ${budget:.2f}: {verdict}") + if unpriced: + print("\n unpriced lanes (estimate is a lower bound, not the bill): " + + ", ".join(unpriced)) + print(" set price_per_mtok on the lane or add vendor: openrouter " + "so the price table covers it") + # `load_pricing` can fail (offline, catalog stale) and return {}. http lanes + # without an explicit price then report unpriced. + if not prices and any(l["harness"] == "http" for l in lanes): + print("\n prices not loaded (offline?); http lanes are unpriced") + return 0 + + def fill_costs(results: list[dict], lanes: list[dict], table: dict) -> None: """Give every result a `cost` (#79). ask() sets it only when it priced a usage block. A lane that is free or on a subscription is 0.0. Anything @@ -3697,6 +3850,13 @@ def main() -> int: "transcripts (paths, latest, latest:N); no lane is re-asked") ap.add_argument("--run-id", metavar="ID", help="tag every transcript of this run with ID (default: generated)") + ap.add_argument("--dry-run", "--estimate", dest="dry_run", action="store_true", + help="priced pre-flight: resolve config + lanes, fetch prices, " + "print the per-lane and panel worst-case cost, exit 0. " + "Combines with --each, --panel, --diff, --revise, --discuss. " + "Refuses --list and --synthesize. --json emits a machine shape " + "(#81). May dial out to load prices; --list stays the offline " + "route view.") ap.add_argument("-h", "--help", action="store_true") a = ap.parse_args() if a.help: @@ -3713,6 +3873,13 @@ def main() -> int: if a.synthesize and (a.each or a.panel or a.revise or a.discuss or a.diff): die("--synthesize reads a finished run; it does not start one.\n" " Drop --each/--panel/--revise/--discuss/--diff.") + if a.dry_run and a.synthesize: + die("--dry-run prices a fresh dispatch; --synthesize reads a finished one.\n" + " Drop --synthesize.") + if a.dry_run and a.list: + die("--dry-run and --list are two different pre-flights; pick one.\n" + " --list roster + route, no network, no cost\n" + " --dry-run [...] --json priced worst-case panel cost (may dial out for prices)") if a.discuss: if a.revise: die("--discuss and --revise are two different second rounds; pick one") @@ -3861,7 +4028,7 @@ def main() -> int: # to estimate to $0 and slip past the gate unseen (#72) -- fail-open on the # one check whose job is to fail closed. No default rate either: a guessed # number reads as authoritative and is wrong for almost every vendor. - if budget is not None: + if budget is not None and not a.dry_run: unpriced = [l for l in lanes if not lane_price_known(l, prices)] if unpriced: die( @@ -3894,6 +4061,17 @@ def main() -> int: if a.discuss: # Each lane speaks once per pass and once to close. estimate *= a.discuss_rounds + 1 + # Dry-run reports the same estimate the gate just computed (the human and + # JSON numbers MUST match the gate for the UI to rely on it, #81). Skipping + # the gate's `die` below is on purpose: a dry-run exists to *show* a number + # that may exceed the budget, not to refuse to compute it. The unpriced + # --budget die above is gated by `not a.dry_run` so dry-run under a budget + # with an unpriced lane reports instead of refusing — same #72 information + # the gate would have died with, minus the failure mode the dry-run exists + # to avoid. + if a.dry_run: + return cmd_dry_run(a, cfg, lanes, est_prompt, prices, calib, + defaults, estimate, brief_on_record) if budget is not None and estimate > budget: die( f"estimated worst case ${estimate:.4f} exceeds budget ${budget:.2f} — refusing to dispatch\n" diff --git a/tests/test_roundtable.py b/tests/test_roundtable.py index daa7740..35e689e 100644 --- a/tests/test_roundtable.py +++ b/tests/test_roundtable.py @@ -2496,6 +2496,339 @@ def test_free_is_distinguished_from_unknown(self): self.assertFalse(rt.lane_price_known({"harness": "http", "model": "vendor/not-in-table"}, {})) +# --------------------------------------------------------------------------- # +# --dry-run (#81) +# --------------------------------------------------------------------------- # +class TestDryRun(unittest.TestCase): + """`--dry-run` is a priced pre-flight: same numbers as the budget gate, no + dispatch. Anything that has side effects (`fetch_keys`, a child process, a + transcript write) would defeat the whole point, so the tests pin that down.""" + + def setUp(self): + self.tmp = Path(tempfile.mkdtemp()) + self.addCleanup(shutil.rmtree, self.tmp, True) + # throwaway pricing + calib caches so real key/shape data does not leak in + self._old_pricing = getattr(rt, "PRICING_CACHE", None) + self._old_calib = getattr(rt, "TOKENIZER_CACHE", None) + rt.PRICING_CACHE = self.tmp / "pricing.json" + rt.TOKENIZER_CACHE = self.tmp / "tokens.json" + self._old_dir = rt.TRANSCRIPT_DIR + rt.TRANSCRIPT_DIR = self.tmp / "transcripts" + self.addCleanup(setattr, rt, "PRICING_CACHE", self._old_pricing) + self.addCleanup(setattr, rt, "TOKENIZER_CACHE", self._old_calib) + self.addCleanup(setattr, rt, "TRANSCRIPT_DIR", self._old_dir) + + def _invoke(self, lanes, *args, stdin="-", extra_cfg=None): + d = Path(tempfile.mkdtemp()) + self.addCleanup(shutil.rmtree, d, True) + cfg = {"lanes": lanes} + cfg.update(extra_cfg or {}) + make_db(d, cfg) + # HOME → self.tmp so `Path.home()` (used for TRANSCRIPT_DIR inside the + # CLI) lands under our throwaway tree, where we can assert on it. + # `_invoke` deliberately does NOT pass `--no-transcript`; tests that + # assert "no transcript was written" need a real-mode invocation to + # mean anything (a paying dispatch would have written, and would + # therefore be a control for the negative claim). + env = {**os.environ, "HOME": str(self.tmp), + "XDG_CACHE_HOME": str(self.tmp)} + argv = [sys.executable, str(ROOT / "roundtable"), "--config", str(d / "c.db"), + *args] + # stdin placeholder so the brief path is exercised end-to-end + if stdin == "-": + return subprocess.run(argv + ["-"], input="the brief", + capture_output=True, text=True, timeout=30, env=env) + if stdin is None: + return subprocess.run(argv + ["the brief"], + capture_output=True, text=True, timeout=30, env=env) + return subprocess.run(argv + [stdin], input="", + capture_output=True, text=True, timeout=30, env=env) + + def _transcript_dir(self): + # Where the CLI writes transcripts when HOME is this tmp tree. + return self.tmp / ".local" / "share" / "flatline-roundtable" / "transcripts" + + def test_dry_run_each_prints_per_lane_costs_and_exits_zero(self): + # Two priced http lanes. The estimate is non-zero, free vs. paid both + # distinguishable, dispatch proves not to have happened. + with StubServer() as s: + lanes = [{"name": "Paid", "harness": "http", "model": "m", + "base_url": s.url, "price_per_mtok": 10.0, "max_tokens": 1000}, + {"name": "Free", "harness": "cli", "model": "x", + "command": "echo hi", "args": ["-p"], "stdin": True}] + r = self._invoke(lanes, "--dry-run", "--each", "-j", "1") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn("Pre-flight estimate", r.stdout) + self.assertIn("Paid", r.stdout) + self.assertIn("Free", r.stdout) + self.assertIn("panel worst case", r.stdout) + # No dispatch. + self.assertIsNone(s.srv.last_request) + # No transcript file written, under the redirected HOME. + self.assertFalse(self._transcript_dir().exists()) + + def test_dry_run_does_not_spawn_subprocesses(self): + # Even a CLI lane under --dry-run must not be exec'd. The whole point + # of the pre-flight is no spend, no side effects, no remote calls. + sub = write_exe(self.tmp / "call_me", + "echo called >&2; exit 0\n") + lanes = [{"name": "X", "harness": "cli", "model": "x", + "command": str(sub), "args": ["-p"], "stdin": True}] + r = self._invoke(lanes, "--dry-run", "--each", "-j", "1") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertNotIn("called", r.stderr) + + def test_dry_run_list_is_refused(self): + # --list is the offline roster view; --dry-run is the priced pre-flight. + # Two surfaces, two purposes — pick one. + with StubServer() as s: + r = self._invoke( + [{"name": "A", "harness": "http", "model": "m", "base_url": s.url}], + "--list", "--dry-run", stdin=None) + self.assertNotEqual(r.returncode, 0) + self.assertIn("two different pre-flights", r.stderr) + + def test_dry_run_synthesize_is_refused(self): + # Need at least one lane so make_db accepts the roster; the refusal + # happens at argparse time, before the roster is read. + lanes = [{"name": "A", "harness": "cli", "model": "x", + "command": "echo hi", "args": ["-p"], "stdin": True}] + r = self._invoke(lanes, "--dry-run", "--synthesize", "latest", + stdin=None) + self.assertNotEqual(r.returncode, 0) + self.assertIn("--synthesize reads a finished one", r.stderr) + + def test_dry_run_each_diff_is_still_refused(self): + # #74 still binds under --dry-run. + with StubServer() as s: + lanes = [{"name": n, "harness": "http", "model": "m", "base_url": s.url} + for n in ("A", "B")] + r = self._invoke(lanes, "--each", "-j", "1", "--diff", "--dry-run") + self.assertNotEqual(r.returncode, 0) + self.assertIn("--each --diff", r.stderr) + self.assertIsNone(s.srv.last_request) + + def test_dry_run_unpriced_lanes_are_listed_and_budget_verdict_is_null(self): + """Under --dry-run the report renders unpriced lanes (#72) instead of + the dispatch refusing on them. The verdict `would_exceed_budget` is + `null` because the budget cannot bind on what we cannot price; the + JSON still lists the unpriced lane by name so the user sees the same + problem the gate would have died on.""" + with StubServer() as s: + lanes = [{"name": "Mystery", "harness": "http", + "model": "vendor/not-in-table", "base_url": s.url}] + r = self._invoke(lanes, "--dry-run", "--each", "-j", "1", + "--max-spend", "0.50", "--json") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIsNone(s.srv.last_request, + "dispatch happened despite --dry-run") + out = json.loads(r.stdout) + self.assertEqual(out["unpriced"], ["Mystery"]) + self.assertIsNone(out["would_exceed_budget"]) + self.assertEqual(out["budget"], 0.50) + # Human-readable surface: same lane is named when --json is absent. + r = self._invoke(lanes, "--dry-run", "--each", "-j", "1", + "--max-spend", "0.50") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn("unpriced", r.stdout.lower()) + + def test_dry_run_dispatch_path_still_refuses_unpriced_under_budget(self): + """#72 contract, dispatch side: a non-dry-run run under a budget with + an unpriced lane must still die, so the unpriced-budget guard the rest + of the tool relies on is unchanged when --dry-run is absent. (#72.)""" + with StubServer() as s: + lanes = [{"name": "Mystery", "harness": "http", + "model": "vendor/not-in-table", "base_url": s.url}] + r = self._invoke(lanes, "--each", "-j", "1", "--max-spend", "0.50") + self.assertNotEqual(r.returncode, 0) + self.assertIn("Mystery", r.stderr) + self.assertIn("no known price", r.stderr) + self.assertIsNone(s.srv.last_request) + + def test_dry_run_emits_no_transcript_under_tmproot(self): + # The control: under `--no-transcript`, no transcript ever lands. + # Without that flag, the same run DOES write one — the negative + # claim only lands next to that control. We keep both halves: paid + # lane under --each (would dispatch, would write) vs. dry-run (must + # not write). + with StubServer() as s: + lanes = [{"name": "A", "harness": "http", "model": "m", + "base_url": s.url, "price_per_mtok": 0.0, + "max_tokens": 100}] + tdir = self._transcript_dir() + # 1. Real run without --no-transcript — control, must write. + r = self._invoke(lanes, "--each", "-j", "1", stdin="the brief") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertTrue(tdir.exists() and any(tdir.iterdir()), + "control: a paying dispatch must land a transcript") + for prev in tdir.iterdir(): + prev.unlink() + # 2. Dry-run on the same lanes — must NOT write. + r = self._invoke(lanes, "--dry-run", "--each", "-j", "1") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertFalse(any(tdir.iterdir()) if tdir.exists() else True, + "dry-run wrote a transcript") + + def test_dry_run_panel_diff_includes_synthesis_cost(self): + # --panel --diff folds the readers' worst case in. The dry-run shows it. + with StubServer() as s: + lanes = [{"name": n, "harness": "http", "model": "m", + "base_url": s.url, "price_per_mtok": 0.0} + for n in ("A", "B", "C", "D")] + r = self._invoke(lanes, "--dry-run", "--panel", "--diff") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn("synthesis", r.stdout) + self.assertIn("panel worst case", r.stdout) + + def test_dry_run_json_shape_matches_contract(self): + # Match the README's machine contract: every documented key is present + # with a sane type, the panel total equals per-lane sum + synthesis, + # and the keys overlap with spend_report() (#79 / #131) where they + # mean the same thing. + with StubServer() as s: + lanes = [{"name": n, "harness": "http", "model": "m", + "base_url": s.url, "price_per_mtok": 1.0, + "max_tokens": 1000} + for n in ("A", "B")] + r = self._invoke(lanes, "--dry-run", "--each", "-j", "1", "--json") + self.assertEqual(r.returncode, 0, r.stderr) + out = json.loads(r.stdout) + self.assertEqual(out["mode"], "each") + self.assertEqual(out["currency"], "USD") + self.assertEqual(set(out), + {"mode", "lanes", "estimated_worst_case", "synthesis", + "discuss_passes", "budget", "would_exceed_budget", + "unpriced", "currency"}) + self.assertIsNone(out["synthesis"]) + self.assertIsNone(out["discuss_passes"]) + self.assertFalse(out["would_exceed_budget"]) + self.assertEqual(out["unpriced"], []) + # Per-lane entry keys. + for entry in out["lanes"]: + self.assertEqual(set(entry), + {"name", "model", "vendor", "harness", "route", + "prompt_tokens", "completion_tokens", + "prompt_cost", "completion_cost", "cost", + "free", "priced"}) + self.assertIsInstance(entry["free"], bool) + self.assertIsInstance(entry["priced"], bool) + # The per-lane sum is the panel total (no synthesis, no discuss, no + # retries). The two values are independently rounded to 6 places, so + # the worst-case compounding is ≤ N * 5e-7 — tolerate one ULP rather + # than pin a precise equality that exaggerates machine noise. + self.assertAlmostEqual(out["estimated_worst_case"], + sum(e["cost"] for e in out["lanes"]), + delta=2e-6) + + def test_dry_run_discuss_folds_multiplier_into_per_lane_costs(self): + """`--discuss` multiplies the panel total by `rounds + 1`; per-lane + costs mirror it so a consumer summing `lanes[].cost` gets the same + number the gate / dry-run reports as `estimated_worst_case`. Doc'd. + `--discuss` is its own mode: the tool refuses `--discuss --each` + and `--discuss --panel` because each turn is built from the turns + before it (nothing to fan out). So we run `--discuss new --dry-run` + here, the priced pre-flight of a would-be shared-context run. + """ + with StubServer() as s: + lanes = [{"name": n, "harness": "http", "model": "m", + "base_url": s.url, "price_per_mtok": 1.0, + "max_tokens": 100} + for n in ("A", "B")] + r = self._invoke(lanes, "--dry-run", + "--discuss", "new", "--discuss-rounds", "3", + "--json") + self.assertEqual(r.returncode, 0, r.stderr) + out = json.loads(r.stdout) + self.assertEqual(out["discuss_passes"], 4) + self.assertEqual(out["mode"], "discuss") + # The numbers are independently rounded, so use a small delta. + self.assertAlmostEqual(out["estimated_worst_case"], + sum(e["cost"] for e in out["lanes"]), + delta=2e-6) + + def test_dry_run_estimate_alias_matches_dry_run(self): + # `--estimate` is the issue-title alias (#81). Same flag, same output, + # so docs can speak either name. + with StubServer() as s: + lanes = [{"name": "A", "harness": "http", "model": "m", + "base_url": s.url, "price_per_mtok": 1.0, + "max_tokens": 100}] + r = self._invoke(lanes, "--estimate", "--each", "-j", "1", + "--json") + self.assertEqual(r.returncode, 0, r.stderr) + out = json.loads(r.stdout) + self.assertEqual(out["mode"], "each") + self.assertIn("estimated_worst_case", out) + + def test_dry_run_json_would_exceed_budget_reflects_the_gate(self): + """The verdict the dry-run reports is the same one the gate would die on. + + `would_exceed_budget: true` means a real dispatch under the same args + would refuse; `false` means the budget would hold. The threshold is + `estimate > budget`, matching `if budget is not None and estimate > budget` + in main().""" + with StubServer() as s: + lanes = [{"name": n, "harness": "http", "model": "m", + "base_url": s.url, "price_per_mtok": 1000.0, + "max_tokens": 1000} + for n in ("A", "B")] + # Budget set high: stays within. + r = self._invoke(lanes, "--dry-run", "--each", "-j", "1", "--json", + "--max-spend", "10.00") + self.assertEqual(r.returncode, 0, r.stderr) + high = json.loads(r.stdout) + self.assertFalse(high["would_exceed_budget"]) + # Budget set low: must exceed. + r = self._invoke(lanes, "--dry-run", "--each", "-j", "1", "--json", + "--max-spend", "0.01") + self.assertEqual(r.returncode, 0, r.stderr) + low = json.loads(r.stdout) + self.assertTrue(low["would_exceed_budget"]) + + def test_dry_run_no_prices_loaded_warns_in_human_output(self): + # pricing cache empty, http lane without price_per_mtok: the gate is + # silent and the budget would be unbounded; the dry-run must surface it. + rt.PRICING_CACHE.write_text("{}") + with StubServer() as s: + lanes = [{"name": "Off", "harness": "http", "model": "m", + "base_url": s.url}] + r = self._invoke(lanes, "--dry-run", "--each", "-j", "1") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn("unpriced", r.stdout) + + def test_dry_run_revise_uses_longest_packet_as_worst_case(self): + """Under --revise the prompt each lane sees is its OWN PACKET, not the + original brief; the longest one is the honest worst case for the budget + gate, and must still be the dry-run's number (#4 contract).""" + # Seed a prior transcript with two answers of different length. + transcript = {"brief": "the original question", + "results": [ + {"lane": "A", "answer": "short", "model": "m", + "harness": "http"}, + {"lane": "B", "answer": "much " * 200 + "longer answer", + "model": "m", "harness": "http"}], + "round": 1} + tp = self.tmp / "transcripts" / "20261001-000001.json" + tp.parent.mkdir(exist_ok=True) + tp.write_text(json.dumps(transcript)) + with StubServer() as s: + lanes = [{"name": n, "harness": "http", "model": "m", + "base_url": s.url, "price_per_mtok": 1.0, + "max_tokens": 100} + for n in ("A", "B")] + r = self._invoke( + lanes, "--dry-run", "--each", "-j", "1", "--revise", str(tp), + "--json", stdin="focus on the longest") + self.assertEqual(r.returncode, 0, r.stderr) + out = json.loads(r.stdout) + self.assertEqual(out["mode"], "revise") + # The longest packet dominates: both lanes' prompt_tokens reflect + # the larger of the two revision prompts, not the original brief. + tokens = [e["prompt_tokens"] for e in out["lanes"]] + self.assertGreater(tokens[1], 0) + self.assertGreater(tokens[0], 0) + + class TestEndToEnd(unittest.TestCase): """Exit codes, via the real CLI entry point.""" From 58040afef3f60b1e33ebca1eb0ce0ef2317bab77 Mon Sep 17 00:00:00 2001 From: "Aaron K. Clark" Date: Thu, 8 Oct 2026 02:20:32 -0500 Subject: [PATCH 2/2] =?UTF-8?q?BACKLOG:=20tick=20#81=20=E2=80=94=20shipped?= =?UTF-8?q?=20in=20PR=20#133?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Code --- BACKLOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/BACKLOG.md b/BACKLOG.md index ff203db..7a6d05f 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -115,6 +115,7 @@ The UI lives in its own private repo, - [x] Add `--dry-run` / `--estimate` — a priced pre-flight that shows per-lane and panel worst-case cost without dispatching (and surfaces unpriced lanes) ([#81](https://github.com/CryptoJones/FlatlineRoundtable/issues/81)) + — shipped in PR #133 - [ ] Transcripts accumulate forever — add bounded, opt-in retention that never deletes a `parent` of a later round ([#82](https://github.com/CryptoJones/FlatlineRoundtable/issues/82))