From ab8b16dfb58705428c3ec9646b010d1e8902b2ae Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:49:20 +0530 Subject: [PATCH 01/33] docs: test-first mutation quality and bounded autonomy spec/recovery plan --- docs/roadmap/mutation-autonomy-v1-2026-10.md | 67 ++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 docs/roadmap/mutation-autonomy-v1-2026-10.md diff --git a/docs/roadmap/mutation-autonomy-v1-2026-10.md b/docs/roadmap/mutation-autonomy-v1-2026-10.md new file mode 100644 index 0000000..a20e99e --- /dev/null +++ b/docs/roadmap/mutation-autonomy-v1-2026-10.md @@ -0,0 +1,67 @@ +# freview Mutation & Bounded Autonomy v1 — tests-first specification + +**Date:** 2026-10-17. **Status:** implementation in progress. +**Branch:** `feature/mutation-autonomy-v1-20261017`, stacked onto draft PR #19 at `e2b269ea6f56b0caed1ac898584476bef5ec4f6f`. Neither this branch nor ancestors may merge/release without separate approval. + +## Product intent and borrowed practices + +**Audit the repo, not the summary.** The default freview skill and all optional operations stay **OFF/read-only**. Improve what happens *after* a maintainer explicitly opts into mutation and bounded worker/reviewer operations. Implement first-party, dependency-free **report interpretation and budget enforcement**, not a clone of mutation engines or general purpose agent OS. + +- [StrykerJS](https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/): consume mutation-testing-elements JSON (status Killed/Timeout/Survived/NoCoverage/CompileError/RuntimeError/Ignored/Pending). Recognize only Killed+Timeout as detected; Survived+NoCoverage as undetected. Ignore invalid/ignored in denominator and *never score incomplete/pending as green*. Prefer scoped runs and incremental/force where supported; caveat that incremental results can go stale with dependency/environment changes. +- [mutmut](https://mutmut.readthedocs.io/en/latest/): consume `mutants/mutmut-cicd-stats.json` produced by *explicitly configured* `mutmut export-cicd-stats`. Recognize `killed`, `survived`, `no_tests`, `timeout`, `skipped`, `suspicious`, `segfault`, `check_was_interrupted_by_user`. A missing file or interrupted run cannot pass. +- [OpenHands iteration budgets](https://docs.openhands.dev/openhands/usage/run-openhands/github-action): bounded stage/round budgets matter; use a global stage cap, stop if worker repeats findings with no file change. +- gstack, Nonna, GSD, Spec Kit: retain explicit acceptance plan, automated tests before review, review with genuine source observation, and a human-controlled release handoff. No third-party source vendored. + +## Configuration / defaults + +All modules remain OFF absent an explicit `.freview-ops.json`. Existing `limits` remain compatible. New optional `mutation_quality`: +- `format`: `auto` (default), `stryker`, `mutmut`. +- `mode`: `advisory` (default) or `gate`. A requested mutation run **cannot count as passing** from process exit alone even in advisory mode: missing/malformed/incomplete report yields explicit `mutation-inconclusive`. In gate mode, valid report must meet configured score and sample. +- `min_score`: 60% (advisory reporting default; gate threshold if enabled); `min_valid_mutants`: 1; `max_survivors`: 12 shown per report. +- Report discovery: only fixed known relative paths under the per-stage verifier workspace; no symlink/hardlink escape, no internet fetch, no reporter installation. Mutator invoked only when policy+command+allow-execute; no self-install or implicit reruns. +- A report without any valid mutants, with incomplete/pending observations, or a nonzero command exit is NOT green. Never claim score is correctness proof; report errors and representative survivors. +New optional `autonomy`: +- `max_stage_runs`: 12 (hard cap 32; applies across all nested worker/test/reviewer/mutation/regression calls). +- `stop_on_no_progress`: true. If reviewer requests change, next worker produces no candidate source change, stop with `no-progress` rather than burning next review tokens. +- `max_repeated_findings`: 1 (1–3) applies only to exact repeated finding fingerprints without candidate source delta; prevents review oscillation. +- `require_mutation_for_shipping`: false by default. If opted in, no shipping handoff without a successful mutation gate on same candidate. Shipping remains handoff-only, never merge/deploy. +- Existing max 3 review rounds, explicit scratch, exact SHA, credential flags and no automatic original-repo edits remain. +- No automatic local worktree resets, installs, pip/npm, branch pushes or external PR/releases. No OS sandbox promises. + +## Checkpoints + +### M0 — red tests first +- [x] Create branch from exact PR #19 head, write durable spec/recovery before implementation. +- [ ] Add direct parser tests for Stryker/mutmut normal, no survivors, incomplete, tampered totals, malformed oversized, no valid mutants, bounded survivors, threshold decisions. +- [ ] Add integration tests for configured mutation report required, gate/advisory behavior, worker no-progress/budget exhaustion, and unchanged read-only defaults. +- [ ] Capture expected failing Ubuntu/macOS CI. Tests must actually run (no unittest method shadowing). + +### M1 — mutation engine reports +- [ ] Implement pure `mutation_report.py` parser and CLI (stdlib only). Never follow report symlinks or leak full source text. +- [ ] Connect mutation runner's result with exact observed verifier workspace and checkpoint report. Verify exact-head both OS. + +### M2 — bounded autonomy +- [ ] Add centrally validated policy defaults and safe optional knobs; stage run budget and no-progress avoidance. +- [ ] Honor opt-in mutation quality gate before proposing shipping; include detailed verdict and provenance. +- [ ] Verify full suite both OS, including all previous 193 tests. + +### M3 — 3-pass self-review & handoff +- [ ] Review independent code and plan for ambiguous mutation statuses / wrong math. +- [ ] Review code for path traversal, budgets, stage reuse, false shipping claims. +- [ ] Review UX/docs/help/compatibility; fix, add tests, rerun. +- [ ] Commit a bounded self-audit, final exact-head CI and draft stacked PR. **No merge/push to main/release**. + +## Recovery / interruption + +1. Inspect PR #19 and `feature/mutation-autonomy-v1-20261017`; compare true branch SHA, PR base and CI for *exact current head*. +2. Read this file before touching code. Begin first unfinished checkpoint; never skip failing tests or treat older green CI as current. +3. Collect source/test/log receipts and maintain final state here. Stored scratch checkpoints are descriptive **not runnable resumptions**. +4. Do not run actual paid agent or mutation tool from CI. Synthetic test repos and hand-authored JSON only; host/SSD acceptance needs owner approval. +5. Keep PR draft; no release automation or protected branch updates. + +| Phase | Commit | CI proof | Status | +| --- | --- | --- | --- | +| M0 | pending | red tests | in progress | +| M1 | pending | two-OS exact SHA | not started | +| M2 | pending | two-OS exact SHA | not started | +| M3 | pending | review and final SHA | not started | From a7053fe6777cc8d5aa835fd92d24e29f25439e9d Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:50:16 +0530 Subject: [PATCH 02/33] test(M0): strict Stryker/mutmut mutation-score and artifact validation contract --- tests/test_mutation_quality.py | 106 +++++++++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 tests/test_mutation_quality.py diff --git a/tests/test_mutation_quality.py b/tests/test_mutation_quality.py new file mode 100644 index 0000000..1aae8fe --- /dev/null +++ b/tests/test_mutation_quality.py @@ -0,0 +1,106 @@ +"""Tests-first mutation report quality contracts; no external mutator installed.""" +import importlib.util +import json +from pathlib import Path +import tempfile +import unittest + +ROOT = Path(__file__).resolve().parents[1] +SRC = ROOT / "skills/freview/scripts/mutation_report.py" +spec = importlib.util.spec_from_file_location("freview_mutation_report", SRC) +mutation = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mutation) + + +def stryker(statuses): + return {"schemaVersion":"2", "thresholds":{"high":90,"low":60}, + "files":{"src/module.py":{"language":"python","source":"x=1","mutants":[ + {"id":str(i),"mutatorName":"ArithmeticOperator", + "location":{"start":{"line":i+1,"column":0},"end":{"line":i+1,"column":1}}, + "status":status} + for i,status in enumerate(statuses)]}}} + + +class MutationReportTests(unittest.TestCase): + def test_stryker_score_and_survivors(self): + result=mutation.summarize(stryker(["Killed","Timeout","Survived","NoCoverage", + "CompileError","Ignored"]),format="auto") + self.assertEqual(result["status"],"complete") + self.assertEqual(result["counts"]["detected"],2) + self.assertEqual(result["counts"]["valid"],4) + self.assertEqual(result["counts"]["undetected"],2) + self.assertEqual(result["score"],50.0) + self.assertEqual(len(result["survivors"]),2) + self.assertFalse(result["quality_pass"]) + self.assertEqual(result["source_format"],"stryker") + self.assertTrue(result["survivors"][0]["file"].startswith("src/")) + + def test_stryker_pending_invalidates_status(self): + result=mutation.summarize(stryker(["Killed","Pending"]),format="stryker") + self.assertEqual(result["status"],"incomplete") + self.assertFalse(result["quality_pass"]) + + def test_stryker_empty_is_not_perfect(self): + result=mutation.summarize(stryker([])) + self.assertEqual(result["status"],"no-valid-mutants") + self.assertIsNone(result["score"]) + self.assertFalse(result["quality_pass"]) + + def test_all_detected_pass_at_threshold_and_limit_survivors(self): + report=stryker(["Killed","Timeout","Survived"]*15) + result=mutation.summarize(report,min_score=60,max_survivors=3) + self.assertEqual(result["status"],"complete") + self.assertEqual(len(result["survivors"]),3) + self.assertTrue(result["quality_pass"]) + self.assertEqual(result["counts"]["valid"],45) + + def test_mutmut_stats_counts_are_consistent_and_incomplete(self): + report={"killed":3,"survived":1,"no_tests":1,"timeout":1,"skipped":1, + "suspicious":0,"segfault":0,"check_was_interrupted_by_user":0,"total":7} + result=mutation.summarize(report,format="auto",min_score=60) + self.assertEqual(result["status"],"complete") + self.assertEqual(result["counts"]["valid"],6) + self.assertAlmostEqual(result["score"],(4/6)*100) + self.assertTrue(result["quality_pass"]) + report["check_was_interrupted_by_user"]=1 + report["total"]=8 + result=mutation.summarize(report) + self.assertEqual(result["status"],"incomplete") + self.assertFalse(result["quality_pass"]) + + def test_mutmut_tampered_totals_do_not_pass(self): + report={"killed":3,"survived":0,"no_tests":0,"timeout":0,"skipped":0, + "suspicious":0,"segfault":0,"check_was_interrupted_by_user":0,"total":1} + result=mutation.summarize(report) + self.assertEqual(result["status"],"invalid") + self.assertFalse(result["quality_pass"]) + + def test_invalid_status_zero_total_and_bad_types(self): + for report in (stryker(["Succeeded"]), {"killed":True,"total":1}, [], + {"killed":-1,"total":-1}, {"schemaVersion":"9","files":{}}, + '{"killed":1,"killed":1}'): + with self.subTest(report=report): + result=mutation.summarize(report) + self.assertEqual(result["status"],"invalid") + self.assertFalse(result["quality_pass"]) + + def test_fixed_report_discovery_is_bounded_and_nofollow(self): + with tempfile.TemporaryDirectory() as temp: + root=Path(temp) + dst=root/"reports"/"mutation" + dst.mkdir(parents=True) + file=dst/"mutation.json" + file.write_text(json.dumps(stryker(["Killed"]))) + good=mutation.inspect_workspace(root) + self.assertEqual(good["status"],"complete") + file.unlink() + external=root/"out.json" + external.write_text(json.dumps(stryker(["Killed"]))) + file.symlink_to(external) + self.assertEqual(mutation.inspect_workspace(root)["status"],"invalid") + file.unlink() + file.write_bytes(b"x"*(mutation.MAX_REPORT_BYTES+1)) + self.assertEqual(mutation.inspect_workspace(root)["status"],"invalid") + +if __name__=="__main__": + unittest.main() From 1567ef8e51f9fab0cb7bf5e902fa06ee5bc76dc9 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:50:56 +0530 Subject: [PATCH 03/33] test(M0): mutation report gate, safe defaults, and global autonomy stage budget --- tests/test_mutation_autonomy.py | 129 ++++++++++++++++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 tests/test_mutation_autonomy.py diff --git a/tests/test_mutation_autonomy.py b/tests/test_mutation_autonomy.py new file mode 100644 index 0000000..4a86ede --- /dev/null +++ b/tests/test_mutation_autonomy.py @@ -0,0 +1,129 @@ +"""TDD optional autonomy budget and mutation-quality gates with fake local tools.""" +import importlib.util +import json +from pathlib import Path +import subprocess +import sys +import tempfile +import unittest + +ROOT=Path(__file__).resolve().parents[1] +SCRIPT=ROOT/"skills/freview/scripts/ops.py" +spec=importlib.util.spec_from_file_location("freview_mutation_autonomy_ops",SCRIPT) +ops=importlib.util.module_from_spec(spec) +spec.loader.exec_module(ops) + + +def git(repo,*args): + return subprocess.run(["git",*args],cwd=repo,capture_output=True, + text=True,check=True).stdout.strip() + + +REPORT={"schemaVersion":"2","thresholds":{"high":80,"low":60}, + "files":{"src/a.py":{"language":"python","source":"a=1", + "mutants":[{"id":"0","mutatorName":"Arithmetic","status":"Killed", + "location":{"start":{"line":1,"column":0},"end":{"line":1,"column":1}}}]}}} + +class AutonomyMutationTests(unittest.TestCase): + def setUp(self): + self.tmp=tempfile.TemporaryDirectory(prefix="freview-mutate-autonomy-") + self.repo=Path(self.tmp.name)/"repo" + self.scratch=Path(self.tmp.name)/"scratch" + self.repo.mkdir(); self.scratch.mkdir() + git(self.repo,"init","-qb","feature/test") + git(self.repo,"config","user.name","Fixture") + git(self.repo,"config","user.email","fixture@example.invalid") + (self.repo/"src").mkdir() + (self.repo/"src"/"a.py").write_text("a=1\n") + git(self.repo,"add",".");git(self.repo,"commit","-qm","initial") + self.sha=git(self.repo,"rev-parse","HEAD") + + def tearDown(self): + self.tmp.cleanup() + + def policy(self,*,mutation=True,agent=False,report=True,quality_mode="advisory", + min_score=60,max_stage_runs=12): + modules={"tests":True,"mutation":mutation,"shipping":True} + commands={"tests":[sys.executable,"-c","pass"]} + if mutation: + payload=repr(json.dumps(REPORT)) + write=("from pathlib import Path; " + "p=Path('reports/mutation/mutation.json');" + "p.parent.mkdir(parents=True,exist_ok=True);" + f"p.write_text({payload})") + commands["mutation"]=[sys.executable,"-c",write if report else "print('done')"] + if agent: + modules["implementation"]=True + modules["adversarial"]=True + commands["implementation"]=[sys.executable,"-c","pass"] + # A deliberately zero-exit but unusable reviewer output. + commands["adversarial"]=[sys.executable,"-c","print('not a verdict')"] + return ops.validate_policy({"schema_version":1,"modules":modules, + "commands":commands, + "mutation_quality":{"mode":quality_mode,"min_score":min_score, + "format":"stryker"}, + "autonomy":{"max_stage_runs":max_stage_runs,"stop_on_no_progress":True, + "require_mutation_for_shipping":False}, + "limits":{"timeout_seconds":12,"max_rounds":3,"max_output_bytes":8192}}) + + def run_pipeline(self, policy): + return ops.pipeline(policy,self.repo,self.scratch,allow_execute=True, + expected_sha=self.sha,requirements="Exercise tests accurately") + + def test_default_off_and_no_token_consumption(self): + settings=ops.validate_policy({"schema_version":1}) + self.assertFalse(any(settings["modules"].values())) + self.assertEqual(settings["mutation_quality"]["mode"],"advisory") + self.assertEqual(settings["autonomy"]["max_stage_runs"],12) + self.assertEqual(list(self.scratch.iterdir()),[]) + + def test_mutation_report_required_even_when_command_succeeds(self): + result=self.run_pipeline(self.policy(report=False)) + self.assertEqual(result["verdict"],"mutation-inconclusive") + self.assertFalse(result["mutation_quality"]["quality_pass"]) + self.assertIsNone(result["shipping"]) + + def test_mutation_advisory_reports_score_and_allows_handoff(self): + result=self.run(self.policy()) + self.assertEqual(result["mutation_quality"]["status"],"complete") + self.assertEqual(result["mutation_quality"]["score"],100.0) + self.assertEqual(result["verdict"],"checks-executed-without-authenticated-verdict") + self.assertIsNotNone(result["shipping"]) + self.assertFalse(result["workspace_modified"]) + + def test_mutation_gate_blocks_if_below_threshold(self): + report=json.loads(json.dumps(REPORT)) + report["files"]["src/a.py"]["mutants"][0]["status"]="Survived" + settings=self.policy(quality_mode="gate",min_score=70) + payload=repr(json.dumps(report)) + settings["commands"]["mutation"]=[sys.executable,"-c", + "from pathlib import Path;" + "p=Path('reports/mutation/mutation.json');p.parent.mkdir(parents=True,exist_ok=True);" + f"p.write_text({payload})"] + result=self.run_pipeline(settings) + self.assertEqual(result["verdict"],"mutation-threshold-failed") + self.assertEqual(result["mutation_quality"]["score"],0.0) + self.assertIsNone(result["shipping"]) + + def test_enabled_mutation_does_not_require_global_stage_budget_override(self): + result=self.run(self.policy(max_stage_runs=2)) + self.assertEqual(len(result["executions"]),2) + self.assertNotEqual(result["verdict"],"budget-exhausted") + + def test_budget_exhaustion_stops_before_next_process(self): + result=self.run(self.policy(agent=True,max_stage_runs=2)) + self.assertEqual(result["verdict"],"budget-exhausted") + self.assertEqual(len(result["executions"]),2) + self.assertIsNone(result["shipping"]) + + def test_policy_rejects_unknown_formats_and_invalid_limits(self): + for patch in ({"mutation_quality":{"format":"unknown"}}, + {"autonomy":{"max_stage_runs":33}}, + {"autonomy":{"stop_on_no_progress":"yes"}}, + {"mutation_quality":{"min_score":-1}}): + with self.subTest(patch=patch): + with self.assertRaises(ops.OpsError): + ops.validate_policy({"schema_version":1,**patch}) + +if __name__=="__main__": + unittest.main() From d82b5f21b884eb1c8a75e686cdfa5fea1b4eb39b Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:52:07 +0530 Subject: [PATCH 04/33] feat(M1): bounded Stryker/mutmut score, survivor and completion parser --- skills/freview/scripts/mutation_report.py | 206 ++++++++++++++++++++++ 1 file changed, 206 insertions(+) create mode 100644 skills/freview/scripts/mutation_report.py diff --git a/skills/freview/scripts/mutation_report.py b/skills/freview/scripts/mutation_report.py new file mode 100644 index 0000000..d05ce3d --- /dev/null +++ b/skills/freview/scripts/mutation_report.py @@ -0,0 +1,206 @@ +"""Bounded, read-only mutation quality evidence (Stryker and mutmut). + +The report is untrusted metadata. Neither parsing nor a high mutation score +proves test quality, independence or correctness. No mutator is executed here. +""" +from __future__ import annotations + +import argparse +import json +import os +from pathlib import Path, PurePosixPath +import re +import stat + +MAX_REPORT_BYTES = 2_097_152 +MAX_MUTANTS = 10_000 +MAX_FILES = 256 +FORMATS = ("auto", "stryker", "mutmut") +STATUSES = {"Killed", "Timeout", "Survived", "NoCoverage", + "CompileError", "RuntimeError", "Ignored", "Pending"} +DEFAULT_REPORTS = (("stryker", "reports/mutation/mutation.json"), + ("mutmut", "mutants/mutmut-cicd-stats.json")) + + +def _invalid(reason="invalid-report"): + return {"status": "invalid", "reason_code": reason, "quality_pass": False, + "source_format": None, "score": None, "survivors": [], + "counts": {"detected": 0, "undetected": 0, "valid": 0}} + + +def _pairs(pairs): + result = {} + for key, value in pairs: + if key in result: + raise ValueError("duplicate key") + result[key] = value + return result + + +def _safe_path(value): + if not isinstance(value, str) or len(value) < 1 or len(value) > 512: + return False + if "\\" in value or any(ord(c) < 32 for c in value): + return False + p = PurePosixPath(value) + return (not p.is_absolute() and + all(part not in ("", ".", "..", ".git") for part in value.split("/"))) + + +def _integer(value): + return type(value) is int and value >= 0 + + +def _sum_stryker(report, *, max_survivors): + if (not isinstance(report.get("schemaVersion"), str) or + not re.fullmatch(r"[12](?:\.\d+){0,2}", report["schemaVersion"]) or + not isinstance(report.get("files"), dict) or + not isinstance(report.get("thresholds"), dict) or + len(report["files"]) > MAX_FILES): + return _invalid() + counts = {state: 0 for state in STATUSES} + survivors = [] + seen = set() + mutations = 0 + for path, entry in report["files"].items(): + if not _safe_path(path) or not isinstance(entry, dict) or not isinstance(entry.get("mutants"), list): + return _invalid() + for m in entry["mutants"]: + mutations += 1 + if mutations > MAX_MUTANTS or not isinstance(m, dict): + return _invalid("report-budget") + state = m.get("status") + ident = m.get("id") + if not isinstance(state, str) or state not in STATUSES or not isinstance(ident, str): + return _invalid("unknown-mutant-status") + key = (path, ident) + if key in seen: + return _invalid("duplicate-mutant-id") + seen.add(key) + counts[state] += 1 + if state in ("Survived", "NoCoverage") and len(survivors) < max_survivors: + location = m.get("location") + start = location.get("start") if isinstance(location, dict) else None + line = start.get("line") if isinstance(start, dict) else None + line = line if _integer(line) and line > 0 else None + survivors.append({"file": path, "id": ident[:80], "status": state, + "line": line, "mutator": str(m.get("mutatorName", ""))[:80]}) + valid = counts["Killed"] + counts["Timeout"] + counts["Survived"] + counts["NoCoverage"] + return {"status": "incomplete" if counts["Pending"] else "complete" if valid else "no-valid-mutants", + "counts": {"detected": counts["Killed"] + counts["Timeout"], + "undetected": counts["Survived"] + counts["NoCoverage"], + "valid": valid, **counts}, "survivors": survivors, + "source_format": "stryker"} + + +def _sum_mutmut(report): + names = ("killed", "survived", "no_tests", "timeout", "skipped", + "suspicious", "segfault", "check_was_interrupted_by_user", "total") + if not all(name in report and _integer(report[name]) for name in names): + return _invalid("invalid-mutmut-counts") + if sum(report[name] for name in names if name != "total") != report["total"]: + return _invalid("inconsistent-mutmut-total") + valid = report["killed"] + report["timeout"] + report["survived"] + report["no_tests"] + incomplete = any(report[key] > 0 for key in ( + "suspicious", "segfault", "check_was_interrupted_by_user")) + return { + "status": "incomplete" if incomplete else "complete" if valid else "no-valid-mutants", + "counts": {"detected": report["killed"] + report["timeout"], + "undetected": report["survived"] + report["no_tests"], + "valid": valid, **{key: report[key] for key in names}}, + "survivors": [], "source_format": "mutmut" + } + + +def summarize(report, *, format="auto", min_score=60, min_valid_mutants=1, + max_survivors=12): + if (format not in FORMATS or + type(min_score) not in (int, float) or not 0 <= min_score <= 100 or + type(min_valid_mutants) is not int or not 1 <= min_valid_mutants <= 10000 or + type(max_survivors) is not int or not 0 <= max_survivors <= 50): + return _invalid("invalid-options") + if isinstance(report, bytes): + if len(report) > MAX_REPORT_BYTES: + return _invalid("report-budget") + try: + report = report.decode("utf-8") + except UnicodeError: + return _invalid() + if isinstance(report, str): + if len(report.encode("utf-8", errors="replace")) > MAX_REPORT_BYTES: + return _invalid("report-budget") + try: + report = json.loads(report, object_pairs_hook=_pairs, + parse_constant=lambda x: (_ for _ in ()).throw(ValueError())) + except (ValueError, UnicodeError, TypeError): + return _invalid() + if not isinstance(report, dict): + return _invalid() + if format == "auto": + format = "stryker" if "schemaVersion" in report else "mutmut" if "killed" in report else "invalid" + output = _sum_stryker(report, max_survivors=max_survivors) if format == "stryker" else _sum_mutmut(report) if format == "mutmut" else _invalid() + if output["status"] == "invalid": + return output + valid = output["counts"]["valid"] + detected = output["counts"]["detected"] + output["score"] = round(100 * detected / valid, 2) if valid else None + output["quality_pass"] = (output["status"] == "complete" and valid >= min_valid_mutants and + output["score"] is not None and output["score"] >= min_score) + output["threshold"] = min_score + output["min_valid_mutants"] = min_valid_mutants + output["untrusted_metadata"] = True + output["proves_test_quality"] = False + return output + + +def inspect_workspace(workspace: Path, *, format="auto", min_score=60, + min_valid_mutants=1, max_survivors=12): + workspace = Path(workspace) + if workspace.is_symlink() or not workspace.is_dir(): + return _invalid("workspace-unavailable") + candidates = [(fmt, relative) for fmt, relative in DEFAULT_REPORTS + if format in ("auto", fmt)] + found = [] + for fmt, relative in candidates: + cursor = workspace + for component in PurePosixPath(relative).parts: + cursor = cursor / component + if cursor.is_symlink(): + return _invalid("symlink-report-path") + if cursor.exists(): + found.append((fmt, cursor)) + if len(found) != 1: + return _invalid("missing-or-ambiguous-report") if found else { + **_invalid("missing-report"), "status": "missing"} + fmt, path = found[0] + flags = os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_NONBLOCK", 0) + try: + with os.fdopen(os.open(path, flags), "rb") as handle: + if not stat.S_ISREG(os.fstat(handle.fileno()).st_mode): + return _invalid("nonregular-report") + raw = handle.read(MAX_REPORT_BYTES + 1) + except OSError: + return _invalid("report-unreadable") + result = summarize(raw, format=fmt, min_score=min_score, + min_valid_mutants=min_valid_mutants, max_survivors=max_survivors) + result["report_path"] = str(path) + return result + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("report", type=Path) + parser.add_argument("--format", choices=FORMATS, default="auto") + parser.add_argument("--min-score", type=float, default=60) + args = parser.parse_args() + file = args.report + if file.is_symlink() or not file.is_file() or file.stat().st_size > MAX_REPORT_BYTES: + print(json.dumps(_invalid("unavailable"))) + return 2 + output = summarize(file.read_bytes(), format=args.format, min_score=args.min_score) + print(json.dumps(output, indent=2, sort_keys=True)) + return 0 if output["status"] == "complete" else 2 + + +if __name__ == "__main__": + raise SystemExit(main()) From fc9820af82b47ca1583058a11f0d24495625dd64 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:53:09 +0530 Subject: [PATCH 05/33] feat(M2): validated mutation quality and bounded autonomy policy defaults --- skills/freview/scripts/ops.py | 37 ++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/skills/freview/scripts/ops.py b/skills/freview/scripts/ops.py index 2fe12f9..3dcfa07 100644 --- a/skills/freview/scripts/ops.py +++ b/skills/freview/scripts/ops.py @@ -45,7 +45,7 @@ class OpsError(Exception): def validate_policy(obj: object) -> dict: - if not isinstance(obj, dict) or set(obj) - {"schema_version", "modules", "commands", "limits", "pass_env"}: + if not isinstance(obj, dict) or set(obj) - {"schema_version", "modules", "commands", "limits", "pass_env", "mutation_quality", "autonomy"}: raise OpsError("Invalid or unknown policy fields") if type(obj.get("schema_version")) is not int or obj["schema_version"] != 1: raise OpsError("Unsupported policy version") @@ -93,8 +93,37 @@ def validate_policy(obj: object) -> dict: value = limits[key] if type(value) is not int or not minimum <= value <= maximum: raise OpsError("Invalid limit: " + key) + mutation = {"format": "auto", "mode": "advisory", "min_score": 60, + "min_valid_mutants": 1, "max_survivors": 12} + raw_mutation = obj.get("mutation_quality", {}) + if not isinstance(raw_mutation, dict) or set(raw_mutation) - set(mutation): + raise OpsError("Unknown mutation-quality options") + mutation.update(raw_mutation) + if (mutation["format"] not in ("auto", "stryker", "mutmut") or + mutation["mode"] not in ("advisory", "gate") or + type(mutation["min_score"]) not in (int, float) or + not 0 <= mutation["min_score"] <= 100 or + type(mutation["min_valid_mutants"]) is not int or + not 1 <= mutation["min_valid_mutants"] <= 10000 or + type(mutation["max_survivors"]) is not int or + not 0 <= mutation["max_survivors"] <= 50): + raise OpsError("Invalid mutation-quality settings") + autonomy = {"max_stage_runs": 12, "stop_on_no_progress": True, + "max_repeated_findings": 1, "require_mutation_for_shipping": False} + raw_autonomy = obj.get("autonomy", {}) + if not isinstance(raw_autonomy, dict) or set(raw_autonomy) - set(autonomy): + raise OpsError("Unknown autonomy options") + autonomy.update(raw_autonomy) + if (type(autonomy["max_stage_runs"]) is not int or + not 1 <= autonomy["max_stage_runs"] <= 32 or + type(autonomy["max_repeated_findings"]) is not int or + not 1 <= autonomy["max_repeated_findings"] <= 3 or + type(autonomy["stop_on_no_progress"]) is not bool or + type(autonomy["require_mutation_for_shipping"]) is not bool): + raise OpsError("Invalid autonomy settings") return {"schema_version": 1, "modules": modules, - "commands": clean_commands, "limits": limits, "pass_env": forward} + "commands": clean_commands, "limits": limits, "pass_env": forward, + "mutation_quality": mutation, "autonomy": autonomy} def load_policy(path: Path) -> dict: @@ -120,7 +149,9 @@ def status(policy: dict) -> dict: "disabled": [key for key in MODULES if not policy["modules"][key]], "limits": policy["limits"], "read_only_default": True, "execution_requires_explicit_flag": True, "release_automation": False, - "credentials_require_separate_flag": bool(policy.get("pass_env"))} + "credentials_require_separate_flag": bool(policy.get("pass_env")), + "mutation_quality": policy["mutation_quality"], + "autonomy": policy["autonomy"]} def plan(policy: dict, requirements: str, sha: str) -> dict: From 39798e31ff056bdfe7cea8a54d90cd98a69c0697 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:53:47 +0530 Subject: [PATCH 06/33] feat(M2): inspect real mutation artifact; global stage budget and no-progress repair stop --- skills/freview/scripts/ops.py | 77 +++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) diff --git a/skills/freview/scripts/ops.py b/skills/freview/scripts/ops.py index 3dcfa07..f953ad8 100644 --- a/skills/freview/scripts/ops.py +++ b/skills/freview/scripts/ops.py @@ -882,6 +882,20 @@ def _review_feedback(findings: list[dict]) -> str: ) +def _mutation_report_summary(verifier: Path, options: dict) -> dict: + """Inspect only a bounded report in the actual mutation-stage workspace.""" + spec = importlib.util.spec_from_file_location( + "freview_mutation_quality", Path(__file__).with_name("mutation_report.py")) + if spec is None or spec.loader is None: + raise OpsError("Mutation parser unavailable") + parser = importlib.util.module_from_spec(spec) + spec.loader.exec_module(parser) + return parser.inspect_workspace( + verifier, format=options["format"], min_score=options["min_score"], + min_valid_mutants=options["min_valid_mutants"], + max_survivors=options["max_survivors"]) + + def pipeline(policy: dict, repo: Path, scratch: Path, *, allow_execute=False, allow_credentials=False, expected_sha=None, requirements="", rounds=None) -> dict: if not allow_execute: @@ -935,10 +949,19 @@ def verify_stage(stage: str, cycle_index: int, *, prompt: str = "") -> dict: _workspace_digest(workspace) != digest) return report final_review = None + mutation_quality = None review_history = [] worker_feedback = "" + last_finding_signature = None + repeated_findings = 0 + autonomy = policy["autonomy"] + def budget_available() -> bool: + return len(receipts) < autonomy["max_stage_runs"] for cycle in range(1, rounds + 1): if policy["modules"]["planning"] and "planning" in policy["commands"]: + if not budget_available(): + verdict = "budget-exhausted" + break report = _execute("planning", policy, workspace, storage, sha, prompt="Produce a short implementation plan and acceptance tests:\n" + requirements, allow_credentials=allow_credentials) @@ -947,6 +970,10 @@ def verify_stage(stage: str, cycle_index: int, *, prompt: str = "") -> dict: verdict = "planning-process-failed" break if policy["modules"]["implementation"]: + if not budget_available(): + verdict = "budget-exhausted" + break + worker_before = _workspace_digest(workspace) report = _execute("implementation", policy, workspace, storage, sha, prompt=("Implement approved requirements with tests first:\n" + requirements + worker_feedback), @@ -955,6 +982,13 @@ def verify_stage(stage: str, cycle_index: int, *, prompt: str = "") -> dict: if report["status"] != "passed": verdict = "worker-failed" break + if (worker_feedback and autonomy["stop_on_no_progress"] and + _workspace_digest(workspace) == worker_before): + verdict = "no-progress" + break + if not budget_available(): + verdict = "budget-exhausted" + break report = verify_stage("tests", cycle) record(report, cycle) if report["candidate_modified_during_verification"] or report["source_entries_modified"]: @@ -969,6 +1003,9 @@ def verify_stage(stage: str, cycle_index: int, *, prompt: str = "") -> dict: if len(message) > MAX_PROMPT: verdict = "review-prompt-over-budget" break + if not budget_available(): + verdict = "budget-exhausted" + break report = verify_stage("adversarial", cycle, prompt=message) record(report, cycle) if report["workspace_modified"] or report["candidate_modified_during_verification"]: @@ -992,6 +1029,17 @@ def verify_stage(stage: str, cycle_index: int, *, prompt: str = "") -> dict: break if final_review["status"] == "changes-requested": verdict = "changes-requested" + signature = tuple(sorted(( + f["id"], f["path"], f["description"], f["evidence"] + ) for f in final_review["findings"])) + if signature == last_finding_signature: + repeated_findings += 1 + else: + repeated_findings = 0 + last_finding_signature = signature + if repeated_findings >= autonomy["max_repeated_findings"]: + verdict = "stalled-review-findings" + break if cycle == rounds or not policy["modules"]["implementation"]: break try: @@ -1007,7 +1055,19 @@ def verify_stage(stage: str, cycle_index: int, *, prompt: str = "") -> dict: if verdict in ("checks-executed-without-authenticated-verdict", "reviewer-approved-claim"): for name in ("regression", "mutation"): if policy["modules"][name]: + if not budget_available(): + verdict = "budget-exhausted" + break result = verify_stage(name, cycle) + if name == "mutation" and result["status"] == "passed": + mutation_quality = _mutation_report_summary( + Path(result["verification_workspace"]), + policy["mutation_quality"]) + result["mutation_quality"] = { + "status": mutation_quality["status"], + "score": mutation_quality.get("score"), + "quality_pass": mutation_quality["quality_pass"], + "counts": mutation_quality["counts"]} record(result, cycle) if result["candidate_modified_during_verification"]: verdict = name + "-modified-candidate" @@ -1018,11 +1078,26 @@ def verify_stage(stage: str, cycle_index: int, *, prompt: str = "") -> dict: if result["status"] != "passed": verdict = name + "-failed" break + if name == "mutation": + if mutation_quality is None or mutation_quality["status"] != "complete": + verdict = "mutation-inconclusive" + break + if (policy["mutation_quality"]["mode"] == "gate" and + not mutation_quality["quality_pass"]): + verdict = "mutation-threshold-failed" + break final_digest = _workspace_digest(workspace) changed = final_digest != starting_digest if changed and verdict in ("checks-executed-without-authenticated-verdict", "reviewer-approved-claim"): verdict = "implementation-not-integrated" + if (autonomy["require_mutation_for_shipping"] and + policy["modules"]["shipping"] and + (not policy["modules"]["mutation"] or mutation_quality is None or + not mutation_quality["quality_pass"]) and + verdict in ("checks-executed-without-authenticated-verdict", + "reviewer-approved-claim")): + verdict = "mutation-required" shipping = (ship_plan(policy, repo, verified=True) if policy["modules"]["shipping"] and not changed and verdict in ("checks-executed-without-authenticated-verdict", @@ -1035,6 +1110,8 @@ def verify_stage(stage: str, cycle_index: int, *, prompt: str = "") -> dict: "current_workspace_digest": final_digest, "workspace_modified": changed, "plan": generated_plan, "shipping": shipping, + "mutation_quality": mutation_quality, + "stage_runs": len(receipts), "stage_run_budget": autonomy["max_stage_runs"], "review": final_review, "review_history": review_history, "executions": receipts, "scratch_workspace": str(workspace), "independent_success_authenticated": False, From ddf501cee3165548c811bcf5e776ed8bbb6ab828 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:54:43 +0530 Subject: [PATCH 07/33] test: align legacy contracts with evidence-first mutation and no-progress defaults --- tests/test_mutation_autonomy.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/test_mutation_autonomy.py b/tests/test_mutation_autonomy.py index 4a86ede..ef29440 100644 --- a/tests/test_mutation_autonomy.py +++ b/tests/test_mutation_autonomy.py @@ -84,7 +84,7 @@ def test_mutation_report_required_even_when_command_succeeds(self): self.assertIsNone(result["shipping"]) def test_mutation_advisory_reports_score_and_allows_handoff(self): - result=self.run(self.policy()) + result=self.run_pipeline(self.policy()) self.assertEqual(result["mutation_quality"]["status"],"complete") self.assertEqual(result["mutation_quality"]["score"],100.0) self.assertEqual(result["verdict"],"checks-executed-without-authenticated-verdict") @@ -106,12 +106,12 @@ def test_mutation_gate_blocks_if_below_threshold(self): self.assertIsNone(result["shipping"]) def test_enabled_mutation_does_not_require_global_stage_budget_override(self): - result=self.run(self.policy(max_stage_runs=2)) + result=self.run_pipeline(self.policy(max_stage_runs=2)) self.assertEqual(len(result["executions"]),2) self.assertNotEqual(result["verdict"],"budget-exhausted") def test_budget_exhaustion_stops_before_next_process(self): - result=self.run(self.policy(agent=True,max_stage_runs=2)) + result=self.run_pipeline(self.policy(agent=True,max_stage_runs=2)) self.assertEqual(result["verdict"],"budget-exhausted") self.assertEqual(len(result["executions"]),2) self.assertIsNone(result["shipping"]) From 4361e3f4f8757bd55659141f0190edf1f93e521c Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:54:47 +0530 Subject: [PATCH 08/33] test: align legacy contracts with evidence-first mutation and no-progress defaults --- tests/test_mutation_quality.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_mutation_quality.py b/tests/test_mutation_quality.py index 1aae8fe..6a9a8d6 100644 --- a/tests/test_mutation_quality.py +++ b/tests/test_mutation_quality.py @@ -60,7 +60,7 @@ def test_mutmut_stats_counts_are_consistent_and_incomplete(self): result=mutation.summarize(report,format="auto",min_score=60) self.assertEqual(result["status"],"complete") self.assertEqual(result["counts"]["valid"],6) - self.assertAlmostEqual(result["score"],(4/6)*100) + self.assertAlmostEqual(result["score"],(4/6)*100, places=2) self.assertTrue(result["quality_pass"]) report["check_was_interrupted_by_user"]=1 report["total"]=8 From e416b2bd8c7ebc82df86391c316d0447b5c2aa3f Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:54:50 +0530 Subject: [PATCH 09/33] test: align legacy contracts with evidence-first mutation and no-progress defaults --- tests/test_ops_quality_isolation.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/test_ops_quality_isolation.py b/tests/test_ops_quality_isolation.py index 48236d5..a0f9b1e 100644 --- a/tests/test_ops_quality_isolation.py +++ b/tests/test_ops_quality_isolation.py @@ -112,7 +112,8 @@ def test_mutator_side_effects_do_not_contaminate_candidate(self): "Path('mutation-results.txt').write_text('survivors=2');" "print('mutator invocation completed')") result = self.run_pipeline(self.policy(tests="pass", mutation=mutant)) - self.assertEqual(result["verdict"], "checks-executed-without-authenticated-verdict") + self.assertEqual(result["verdict"], "mutation-inconclusive") + self.assertIsNone(result["shipping"]) self.assertFalse(result["workspace_modified"]) self.assertTrue(result["executions"][-1]["workspace_modified"]) self.assertFalse((Path(result["scratch_workspace"]) / "mutation-results.txt").exists()) From da41b3b5915b3501146f934d98e1c0673b796ed5 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:54:53 +0530 Subject: [PATCH 10/33] test: align legacy contracts with evidence-first mutation and no-progress defaults --- tests/test_review_verdict.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_review_verdict.py b/tests/test_review_verdict.py index 42adabf..4116815 100644 --- a/tests/test_review_verdict.py +++ b/tests/test_review_verdict.py @@ -226,8 +226,8 @@ def test_requested_changes_stop_at_three_cycles(self): "reviewed_paths=['src/module.py'],reason='not ready')))" ) result = self.run_pipeline(self.policy(reviewer, worker="pass")) - self.assertEqual(result["review_cycles"], 3) - self.assertEqual(result["verdict"], "changes-requested") + self.assertEqual(result["review_cycles"], 2) + self.assertEqual(result["verdict"], "no-progress") self.assertIsNone(result["shipping"]) unrepairable = self.run_pipeline(self.policy(reviewer)) self.assertEqual(unrepairable["review_cycles"], 1) From 4708d081094511f85866973189286d9937910199 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:56:09 +0530 Subject: [PATCH 11/33] test(M2): stale committed mutation reports cannot pass; optional shipping requirement fails closed --- tests/test_mutation_autonomy.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/tests/test_mutation_autonomy.py b/tests/test_mutation_autonomy.py index ef29440..dd565e5 100644 --- a/tests/test_mutation_autonomy.py +++ b/tests/test_mutation_autonomy.py @@ -116,6 +116,28 @@ def test_budget_exhaustion_stops_before_next_process(self): self.assertEqual(len(result["executions"]),2) self.assertIsNone(result["shipping"]) + def test_committed_stale_mutation_report_does_not_count_as_new_execution(self): + cached=self.repo/"reports"/"mutation" + cached.mkdir(parents=True) + (cached/"mutation.json").write_text(json.dumps(REPORT)) + git(self.repo,"add",".") + git(self.repo,"commit","-qm","cached old mutation report") + self.sha=git(self.repo,"rev-parse","HEAD") + policy=self.policy() + policy["commands"]["mutation"]=[sys.executable,"-c","pass"] + result=self.run_pipeline(policy) + self.assertEqual(result["verdict"],"mutation-inconclusive") + self.assertFalse(result["mutation_quality"]["quality_pass"]) + self.assertEqual(result["mutation_quality"]["reason_code"],"report-not-updated") + self.assertIsNone(result["shipping"]) + + def test_optional_shipping_mutation_requirement_blocks_missing_module(self): + policy=self.policy(mutation=False) + policy["autonomy"]["require_mutation_for_shipping"]=True + result=self.run_pipeline(policy) + self.assertEqual(result["verdict"],"mutation-required") + self.assertIsNone(result["shipping"]) + def test_policy_rejects_unknown_formats_and_invalid_limits(self): for patch in ({"mutation_quality":{"format":"unknown"}}, {"autonomy":{"max_stage_runs":33}}, From c06822c50d7fe47fd802e854819b8297f9050478 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:56:31 +0530 Subject: [PATCH 12/33] fix(M2): require fresh mutation report generated by the current isolated run --- skills/freview/scripts/ops.py | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/skills/freview/scripts/ops.py b/skills/freview/scripts/ops.py index f953ad8..352a5ae 100644 --- a/skills/freview/scripts/ops.py +++ b/skills/freview/scripts/ops.py @@ -882,6 +882,26 @@ def _review_feedback(findings: list[dict]) -> str: ) +def _mutation_report_identity(workspace: Path) -> dict: + """Observe only known mutator artifact metadata before/after execution. + + A pre-existing committed mutation report is not evidence of *this* run. + The freshness check is not a signature or a defense against malicious code. + """ + observed = {} + for relative in ("reports/mutation/mutation.json", + "mutants/mutmut-cicd-stats.json"): + path = workspace / relative + try: + details = path.lstat() + observed[relative] = ( + details.st_mtime_ns, details.st_size, details.st_mode, + details.st_ino) + except FileNotFoundError: + observed[relative] = None + return observed + + def _mutation_report_summary(verifier: Path, options: dict) -> dict: """Inspect only a bounded report in the actual mutation-stage workspace.""" spec = importlib.util.spec_from_file_location( @@ -940,8 +960,12 @@ def record(receipt: dict, cycle_index: int) -> None: def verify_stage(stage: str, cycle_index: int, *, prompt: str = "") -> dict: digest = _workspace_digest(workspace) verifier = _verification_copy(workspace, storage, stage, cycle_index) + mutation_before = _mutation_report_identity(verifier) if stage == "mutation" else None report = _execute(stage, policy, verifier, verifier.parent, sha, prompt=prompt, allow_credentials=allow_credentials) + if stage == "mutation": + mutation_after = _mutation_report_identity(verifier) + report["mutation_report_fresh"] = mutation_before != mutation_after report["verification_workspace"] = str(verifier) report["candidate_digest"] = digest report["source_entries_modified"] = _original_entries_changed(workspace, verifier) @@ -1063,6 +1087,11 @@ def budget_available() -> bool: mutation_quality = _mutation_report_summary( Path(result["verification_workspace"]), policy["mutation_quality"]) + if not result["mutation_report_fresh"]: + mutation_quality = { + **mutation_quality, "status": "incomplete", + "quality_pass": False, + "reason_code": "report-not-updated"} result["mutation_quality"] = { "status": mutation_quality["status"], "score": mutation_quality.get("score"), From 47a3eed8ee28ad463f61033676e16f7612e19152 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:57:16 +0530 Subject: [PATCH 13/33] test(M2): standalone mutation must interpret reports and reject exit-zero-only evidence --- tests/test_mutation_autonomy.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/tests/test_mutation_autonomy.py b/tests/test_mutation_autonomy.py index dd565e5..f694708 100644 --- a/tests/test_mutation_autonomy.py +++ b/tests/test_mutation_autonomy.py @@ -83,6 +83,18 @@ def test_mutation_report_required_even_when_command_succeeds(self): self.assertFalse(result["mutation_quality"]["quality_pass"]) self.assertIsNone(result["shipping"]) + def test_standalone_mutation_needs_report_as_well(self): + missing=self.policy(report=False) + result=ops.run_stage("mutation",missing,self.repo,self.scratch, + allow_execute=True,expected_sha=self.sha) + self.assertEqual(result["status"],"mutation-inconclusive") + self.assertFalse(result["mutation_quality"]["quality_pass"]) + present=self.policy() + second=ops.run_stage("mutation",present,self.repo,self.scratch, + allow_execute=True,expected_sha=self.sha) + self.assertEqual(second["status"],"passed") + self.assertEqual(second["mutation_quality"]["score"],100.0) + def test_mutation_advisory_reports_score_and_allows_handoff(self): result=self.run_pipeline(self.policy()) self.assertEqual(result["mutation_quality"]["status"],"complete") From 539c4445b95cc2f85a89da97ed0725801a32f20c Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:57:46 +0530 Subject: [PATCH 14/33] feat(M2): apply same mutation report quality verdict to standalone and pipeline executions --- skills/freview/scripts/ops.py | 58 ++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 25 deletions(-) diff --git a/skills/freview/scripts/ops.py b/skills/freview/scripts/ops.py index 352a5ae..74a0e33 100644 --- a/skills/freview/scripts/ops.py +++ b/skills/freview/scripts/ops.py @@ -503,8 +503,14 @@ def run_stage(stage: str, policy: dict, repo: Path, scratch: Path, if expected_sha is not None and sha != expected_sha: raise OpsError("Expected exact SHA differs from HEAD") workspace = _extract_archive(repo, root, sha) - return _execute(stage, policy, workspace, workspace.parent, sha, - prompt=prompt, allow_credentials=allow_credentials) + before = _mutation_report_identity(workspace) if stage == "mutation" else None + result = _execute(stage, policy, workspace, workspace.parent, sha, + prompt=prompt, allow_credentials=allow_credentials) + if stage == "mutation": + result["verification_workspace"] = str(workspace) + result["mutation_report_fresh"] = before != _mutation_report_identity(workspace) + _evaluate_mutation_stage(result, policy) + return result @@ -916,6 +922,25 @@ def _mutation_report_summary(verifier: Path, options: dict) -> dict: max_survivors=options["max_survivors"]) +def _evaluate_mutation_stage(result: dict, policy: dict) -> dict | None: + """Make a completed mutation invocation answerable to its actual report.""" + if result["status"] != "passed": + return None + quality = _mutation_report_summary( + Path(result["verification_workspace"]), policy["mutation_quality"]) + if not result.get("mutation_report_fresh"): + quality = {**quality, "status": "incomplete", + "quality_pass": False, "reason_code": "report-not-updated"} + result["mutation_quality"] = { + "status": quality["status"], "score": quality.get("score"), + "quality_pass": quality["quality_pass"], "counts": quality["counts"]} + if quality["status"] != "complete": + result["status"] = "mutation-inconclusive" + elif policy["mutation_quality"]["mode"] == "gate" and not quality["quality_pass"]: + result["status"] = "mutation-threshold-failed" + return quality + + def pipeline(policy: dict, repo: Path, scratch: Path, *, allow_execute=False, allow_credentials=False, expected_sha=None, requirements="", rounds=None) -> dict: if not allow_execute: @@ -1083,20 +1108,8 @@ def budget_available() -> bool: verdict = "budget-exhausted" break result = verify_stage(name, cycle) - if name == "mutation" and result["status"] == "passed": - mutation_quality = _mutation_report_summary( - Path(result["verification_workspace"]), - policy["mutation_quality"]) - if not result["mutation_report_fresh"]: - mutation_quality = { - **mutation_quality, "status": "incomplete", - "quality_pass": False, - "reason_code": "report-not-updated"} - result["mutation_quality"] = { - "status": mutation_quality["status"], - "score": mutation_quality.get("score"), - "quality_pass": mutation_quality["quality_pass"], - "counts": mutation_quality["counts"]} + if name == "mutation": + mutation_quality = _evaluate_mutation_stage(result, policy) record(result, cycle) if result["candidate_modified_during_verification"]: verdict = name + "-modified-candidate" @@ -1105,16 +1118,11 @@ def budget_available() -> bool: verdict = "regression-mutated-source" break if result["status"] != "passed": - verdict = name + "-failed" + verdict = (result["status"] if name == "mutation" and + result["status"] in ("mutation-inconclusive", + "mutation-threshold-failed") + else name + "-failed") break - if name == "mutation": - if mutation_quality is None or mutation_quality["status"] != "complete": - verdict = "mutation-inconclusive" - break - if (policy["mutation_quality"]["mode"] == "gate" and - not mutation_quality["quality_pass"]): - verdict = "mutation-threshold-failed" - break final_digest = _workspace_digest(workspace) changed = final_digest != starting_digest if changed and verdict in ("checks-executed-without-authenticated-verdict", From 2221510c4588dbfa0b711f5981640d807d3a583a Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:59:25 +0530 Subject: [PATCH 15/33] test(M3): mutmut export omits categories; never round mutation score across gate --- tests/test_mutation_quality.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/tests/test_mutation_quality.py b/tests/test_mutation_quality.py index 6a9a8d6..8dcdb67 100644 --- a/tests/test_mutation_quality.py +++ b/tests/test_mutation_quality.py @@ -68,6 +68,21 @@ def test_mutmut_stats_counts_are_consistent_and_incomplete(self): self.assertEqual(result["status"],"incomplete") self.assertFalse(result["quality_pass"]) + def test_mutmut_unexported_categories_are_inconclusive(self): + # Real mutmut total also includes not_checked and caught_by_type_check, + # neither of which is exported in mutmut-cicd-stats.json. + data={"killed":3,"survived":0,"no_tests":0,"timeout":0,"skipped":0, + "suspicious":0,"segfault":0,"check_was_interrupted_by_user":0,"total":5} + result=mutation.summarize(data) + self.assertEqual(result["status"],"incomplete") + self.assertEqual(result["counts"]["unclassified"],2) + self.assertFalse(result["quality_pass"]) + + def test_fractional_mutation_score_cannot_round_into_passing_threshold(self): + result=mutation.summarize(stryker(["Killed"]*599+["Survived"]*401), + min_score=59.91) + self.assertFalse(result["quality_pass"]) + def test_mutmut_tampered_totals_do_not_pass(self): report={"killed":3,"survived":0,"no_tests":0,"timeout":0,"skipped":0, "suspicious":0,"segfault":0,"check_was_interrupted_by_user":0,"total":1} From 60d6f34f092806d2845ee11505671038ae136222 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 13:59:44 +0530 Subject: [PATCH 16/33] fix(M3): match mutmut's actually exported totals and compare unrounded score --- skills/freview/scripts/mutation_report.py | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/skills/freview/scripts/mutation_report.py b/skills/freview/scripts/mutation_report.py index d05ce3d..f9c8f9d 100644 --- a/skills/freview/scripts/mutation_report.py +++ b/skills/freview/scripts/mutation_report.py @@ -98,16 +98,22 @@ def _sum_mutmut(report): "suspicious", "segfault", "check_was_interrupted_by_user", "total") if not all(name in report and _integer(report[name]) for name in names): return _invalid("invalid-mutmut-counts") - if sum(report[name] for name in names if name != "total") != report["total"]: + exported = sum(report[name] for name in names if name != "total") + if exported > report["total"]: return _invalid("inconsistent-mutmut-total") + # mutmut's total also includes not_checked and caught_by_type_check, + # which its current CI stats export omits. We cannot classify the + # difference, and must not silently call it fully tested. + unclassified = report["total"] - exported valid = report["killed"] + report["timeout"] + report["survived"] + report["no_tests"] - incomplete = any(report[key] > 0 for key in ( + incomplete = bool(unclassified) or any(report[key] > 0 for key in ( "suspicious", "segfault", "check_was_interrupted_by_user")) return { "status": "incomplete" if incomplete else "complete" if valid else "no-valid-mutants", "counts": {"detected": report["killed"] + report["timeout"], "undetected": report["survived"] + report["no_tests"], - "valid": valid, **{key: report[key] for key in names}}, + "valid": valid, "unclassified": unclassified, + **{key: report[key] for key in names}}, "survivors": [], "source_format": "mutmut" } @@ -145,7 +151,8 @@ def summarize(report, *, format="auto", min_score=60, min_valid_mutants=1, detected = output["counts"]["detected"] output["score"] = round(100 * detected / valid, 2) if valid else None output["quality_pass"] = (output["status"] == "complete" and valid >= min_valid_mutants and - output["score"] is not None and output["score"] >= min_score) + output["score"] is not None and + 100 * detected >= min_score * valid) output["threshold"] = min_score output["min_valid_mutants"] = min_valid_mutants output["untrusted_metadata"] = True From 4caf17d985d82ca81f9262a30e699cde5308cf0b Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:00:13 +0530 Subject: [PATCH 17/33] test: exit-zero mutation without report is inconclusive in standalone mode --- tests/test_optional_ops.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/test_optional_ops.py b/tests/test_optional_ops.py index 0812b90..027991b 100644 --- a/tests/test_optional_ops.py +++ b/tests/test_optional_ops.py @@ -211,7 +211,8 @@ def test_approved_mutation_uses_separate_stage_not_fake_builtin_mutator(self): policy["commands"]["mutation"] = [sys.executable, "-c", "print('MUTATION COMPLETE')"] result = ops.run_stage("mutation", policy, self.repo, self.scratch, allow_execute=True, expected_sha=self.head) - self.assertEqual(result["status"], "passed") + self.assertEqual(result["status"], "mutation-inconclusive") + self.assertFalse(result["mutation_quality"]["quality_pass"]) self.assertFalse(result["proves_test_quality"]) def test_adversarial_review_no_worker_summary_first_and_three_round_cap(self): From e90a7887e98dbd418d13f8dbd5957e16935a85c9 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:01:14 +0530 Subject: [PATCH 18/33] test(M3): shell-free mutmut run plus explicit export adapter with fail-closed output --- tests/test_mutmut_driver.py | 48 +++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 tests/test_mutmut_driver.py diff --git a/tests/test_mutmut_driver.py b/tests/test_mutmut_driver.py new file mode 100644 index 0000000..22d2646 --- /dev/null +++ b/tests/test_mutmut_driver.py @@ -0,0 +1,48 @@ +"""Offline fake mutmut CLI: adapter must run and then export CI JSON.""" +from pathlib import Path +import json +import subprocess +import sys +import tempfile +import unittest + +ROOT=Path(__file__).resolve().parents[1] +DRIVER=ROOT/"skills/freview/scripts/mutmut_driver.py" + + +class MutmutDriverTests(unittest.TestCase): + def test_driver_runs_then_exports_without_shell(self): + with tempfile.TemporaryDirectory() as tmp: + base=Path(tmp) + fake=base/"fake-mutmut" + fake.write_text( + "#!"+sys.executable+"\n" + "import sys\nfrom pathlib import Path\n" + "root=Path.cwd()\n" + "if sys.argv[1]=='run': (root/'started').write_text('yes')\n" + "if sys.argv[1]=='export-cicd-stats':\n" + " assert (root/'started').exists()\n" + " path=root/'mutants'/'mutmut-cicd-stats.json'\n" + " path.parent.mkdir(exist_ok=True)\n" + " path.write_text('{\"killed\":1,\"survived\":0,\"no_tests\":0,\"timeout\":0,\"skipped\":0,\"suspicious\":0,\"segfault\":0,\"check_was_interrupted_by_user\":0,\"total\":1}')\n") + fake.chmod(0o700) + p=subprocess.run([sys.executable,str(DRIVER),"--binary",str(fake), + "--timeout-seconds","5"],cwd=base, + capture_output=True,text=True,timeout=15) + self.assertEqual(p.returncode,0,p.stdout+p.stderr) + self.assertTrue((base/"started").exists()) + self.assertEqual(json.loads((base/"mutants"/"mutmut-cicd-stats.json").read_text())["killed"],1) + + def test_driver_fails_when_export_missing(self): + with tempfile.TemporaryDirectory() as tmp: + base=Path(tmp) + fake=base/"fake-mutmut" + fake.write_text("#!"+sys.executable+"\nimport sys\nsys.exit(0)\n") + fake.chmod(0o700) + p=subprocess.run([sys.executable,str(DRIVER),"--binary",str(fake), + "--timeout-seconds","5"],cwd=base, + capture_output=True,text=True,timeout=15) + self.assertNotEqual(p.returncode,0) + +if __name__=="__main__": + unittest.main() From aec1c4dffcedf961282fbc50863feed3f2bf54f2 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:01:30 +0530 Subject: [PATCH 19/33] feat(M3): bounded no-shell mutmut run/export adapter for report-based verification --- skills/freview/scripts/mutmut_driver.py | 47 +++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 skills/freview/scripts/mutmut_driver.py diff --git a/skills/freview/scripts/mutmut_driver.py b/skills/freview/scripts/mutmut_driver.py new file mode 100644 index 0000000..87006aa --- /dev/null +++ b/skills/freview/scripts/mutmut_driver.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +"""Optional trusted mutmut adapter: run mutations, then export CI stats. + +Use only as an explicitly configured freview ops mutation argv. mutmut itself +must already be installed. The parent freview ops process imposes isolation, +output and process-group budgets; this adapter provides its own timeout. +""" +from __future__ import annotations + +import argparse +import os +from pathlib import Path +import stat +import subprocess + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--binary", default="mutmut", help="already installed mutmut CLI") + parser.add_argument("--timeout-seconds", type=int, default=120) + args = parser.parse_args() + if not 1 <= args.timeout_seconds <= 600: + parser.error("timeout must be between 1 and 600 seconds") + if not args.binary or "\0" in args.binary or "\n" in args.binary: + parser.error("invalid binary") + # This adapter neither evaluates shell strings nor installs dependencies. + for operation in ("run", "export-cicd-stats"): + try: + result = subprocess.run([args.binary, operation], timeout=args.timeout_seconds, + check=False) + except (FileNotFoundError, PermissionError, subprocess.TimeoutExpired): + return 2 + if result.returncode != 0: + return 2 + target = Path("mutants/mutmut-cicd-stats.json") + try: + if target.is_symlink() or not stat.S_ISREG(target.stat().st_mode): + return 2 + if target.stat().st_size == 0: + return 2 + except OSError: + return 2 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From dcd1c63d1ab4cf6a9824892aa45a8e77fb025e44 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:02:33 +0530 Subject: [PATCH 20/33] test(M3): stop repeated adversarial findings even with unrelated worker edits --- tests/test_review_verdict.py | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/tests/test_review_verdict.py b/tests/test_review_verdict.py index 4116815..505a945 100644 --- a/tests/test_review_verdict.py +++ b/tests/test_review_verdict.py @@ -213,6 +213,29 @@ def test_changes_requested_feedback_reaches_worker_second_round(self): self.assertEqual(result["review"]["status"], "approved-claim") self.assertIsNone(result["shipping"]) + def test_repeated_findings_stop_even_when_worker_changes_unrelated_files(self): + worker=( + "from pathlib import Path;" + "p=Path('progress.txt');" + "n=int(p.read_text()) if p.exists() else 0;" + "p.write_text(str(n+1))" + ) + reviewer=( + "import sys,json,re;" + "s=sys.stdin.read();" + "a=re.search(r'SOURCE_SHA: ([a-f0-9]{40})',s).group(1);" + "b=re.search(r'WORKSPACE_DIGEST: ([a-f0-9]{64})',s).group(1);" + "print(json.dumps(dict(schema_version=1,source_sha=a,workspace_digest=b," + "verdict='changes_requested'," + "findings=[dict(id='R1',severity='high',path='src/module.py'," + "description='Still missing guard',evidence='Same line is unprotected')]," + "reviewed_paths=['src/module.py'],reason='same problem')))" + ) + result=self.run_pipeline(self.policy(reviewer,worker=worker)) + self.assertEqual(result["review_cycles"],2) + self.assertEqual(result["verdict"],"stalled-review-findings") + self.assertIsNone(result["shipping"]) + def test_requested_changes_stop_at_three_cycles(self): reviewer = ( "import sys,json,re;" From 92bfee6919c052e43447ec73ddceeb36a2228010 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:02:55 +0530 Subject: [PATCH 21/33] test(M3): mutation score and fresh-report provenance survive atomic checkpoint --- tests/test_mutation_autonomy.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tests/test_mutation_autonomy.py b/tests/test_mutation_autonomy.py index f694708..9e05dbd 100644 --- a/tests/test_mutation_autonomy.py +++ b/tests/test_mutation_autonomy.py @@ -103,6 +103,14 @@ def test_mutation_advisory_reports_score_and_allows_handoff(self): self.assertIsNotNone(result["shipping"]) self.assertFalse(result["workspace_modified"]) + def test_checkpoint_preserves_mutation_quality_evidence(self): + result=self.run_pipeline(self.policy()) + summary=ops.inspect_checkpoint(Path(result["checkpoint_path"])) + stage=next(item for item in summary["stage_receipts"] if item["stage"]=="mutation") + self.assertEqual(stage["mutation_quality"]["score"],100.0) + self.assertTrue(stage["mutation_report_fresh"]) + self.assertEqual(stage["mutation_quality"]["counts"]["valid"],1) + def test_mutation_gate_blocks_if_below_threshold(self): report=json.loads(json.dumps(REPORT)) report["files"]["src/a.py"]["mutants"][0]["status"]="Survived" From 42dab6e381b21b4a66f63ea5f262897bb70b35bc Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:03:28 +0530 Subject: [PATCH 22/33] feat(M3): checkpoint actual mutation score and fresh-report evidence with strict schema validation --- skills/freview/scripts/ops.py | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/skills/freview/scripts/ops.py b/skills/freview/scripts/ops.py index 74a0e33..35eb38a 100644 --- a/skills/freview/scripts/ops.py +++ b/skills/freview/scripts/ops.py @@ -708,7 +708,11 @@ def _checkpoint_data(sha: str, initial_digest: str, current_digest: str, "verification_workspace": item["verification_workspace"], "source_entries_modified": item["source_entries_modified"], "candidate_modified_during_verification": item["candidate_modified_during_verification"], - } if "candidate_digest" in item else {}) + } if "candidate_digest" in item else {}), + **({ + "mutation_report_fresh": item["mutation_report_fresh"], + "mutation_quality": item["mutation_quality"], + } if "mutation_quality" in item else {}) } for item in receipts], "untrusted_metadata": True, "resumes_execution": False, "attested": False, @@ -788,13 +792,19 @@ def no_duplicate_keys(pairs: list[tuple[str, object]]) -> dict: "output_truncated", "workspace_digest_after"} verification_keys = {"candidate_digest", "verification_workspace", "source_entries_modified", "candidate_modified_during_verification"} + mutation_keys = {"mutation_report_fresh", "mutation_quality"} for item in value["stage_receipts"]: if (not isinstance(item, dict) or not receipt_keys.issubset(item) or - set(item) not in (receipt_keys, receipt_keys | verification_keys) or + set(item) not in (receipt_keys, + receipt_keys | verification_keys, + receipt_keys | verification_keys | mutation_keys) or not isinstance(item["stage"], str) or item["stage"] not in STAGES or not isinstance(item["status"], str) or item["status"] not in - ("passed", "failed", "timeout", "output-limit") or + (("passed", "failed", "timeout", "output-limit", + "mutation-inconclusive", "mutation-threshold-failed") + if item["stage"] == "mutation" + else ("passed", "failed", "timeout", "output-limit")) or (item["exit_code"] is not None and type(item["exit_code"]) is not int) or not isinstance(item["log_path"], str) or not 1 <= len(item["log_path"]) <= 4096 or @@ -810,6 +820,25 @@ def no_duplicate_keys(pairs: list[tuple[str, object]]) -> dict: type(item["source_entries_modified"]) is not bool or type(item["candidate_modified_during_verification"]) is not bool): raise OpsError("Invalid verification workspace provenance") + if mutation_keys.issubset(item): + quality = item["mutation_quality"] + if (item["stage"] != "mutation" or + type(item["mutation_report_fresh"]) is not bool or + not isinstance(quality, dict) or + set(quality) != {"status", "score", "quality_pass", "counts"} or + quality["status"] not in ( + "complete", "incomplete", "invalid", "missing", + "no-valid-mutants") or + type(quality["quality_pass"]) is not bool or + (quality["score"] is not None and + (type(quality["score"]) not in (int, float) or + not 0 <= quality["score"] <= 100)) or + not isinstance(quality["counts"], dict) or + len(quality["counts"]) > 24 or + any(not isinstance(k, str) or len(k) > 60 or + type(v) is not int or v < 0 + for k, v in quality["counts"].items())): + raise OpsError("Invalid mutation checkpoint provenance") # Syntactic consistency != authentic execution or tamper-proof metadata. return value From 6c75068992bab56a30fb1c4f3b0fd4e647685397 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:04:31 +0530 Subject: [PATCH 23/33] docs(M3): user-facing mutation score and bounded autonomy manual with safe defaults --- .../freview/references/mutation-autonomy.md | 87 +++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 skills/freview/references/mutation-autonomy.md diff --git a/skills/freview/references/mutation-autonomy.md b/skills/freview/references/mutation-autonomy.md new file mode 100644 index 0000000..998396d --- /dev/null +++ b/skills/freview/references/mutation-autonomy.md @@ -0,0 +1,87 @@ +# Mutation quality and bounded autonomy (opt-in) + +**Default:** no mutation tool, no coding agent, no test execution, no hooks and no shipping are invoked by the standard `/freview` audit. Everything below is only for a trusted repository with explicit `.freview-ops.json` policy plus the execution permission flags described in [optional operations](optional-ops.md). + +## What freview implements + +Mutation **execution** is delegated to a user-selected, already-installed engine such as [StrykerJS](https://stryker-mutator.io/docs/stryker-js/incremental/) or [mutmut](https://mutmut.readthedocs.io/en/latest/). freview implements the portable quality **interpretation and gating layer**: consumes a bounded report, validates expected structure and freshness in that exact disposable verifier workspace, counts Killed + Timeout as detected and Survived + NoCoverage (mutmut `no_tests`) as undetected, and reports representative survivor locations where available. A mutation tool returning exit code zero **without a fresh structured report does not pass**. Stryker's Pending or mutmut's interrupted/suspicious results are inconclusive. mutmut currently exports totals that may include categories omitted from its CI JSON (such as not-checked or caught-by-type-check); freview exposes those as **unclassified** and does not pretend the run is complete. + +**Not a proof of correctness.** Report JSON and mutation engine output can be misleading; mutmut CI summary has aggregate counts only, not per-mutant locations, and surviving/equivalent mutants need human interpretation. For every report freview records the engine format, observed score, counts, advisory/gate decision, and limitations. It never copies or executes commands suggested by a report. + +## Configuration + +All module switches are **false by default**. An explicitly opted-in example with a mutation-quality gate and bounded repair automation: + +```json +{ + "schema_version": 1, + "modules": { + "tests": true, + "adversarial": true, + "implementation": true, + "mutation": true, + "shipping": true + }, + "commands": { + "tests": ["python3", "-m", "unittest", "discover", "-s", "tests"], + "implementation": ["/trusted/bin/worker-agent"], + "adversarial": ["/trusted/bin/reviewer-agent"], + "mutation": ["python3", "/trusted/freview/skills/freview/scripts/mutmut_driver.py"] + }, + "mutation_quality": { + "format": "mutmut", + "mode": "gate", + "min_score": 70, + "min_valid_mutants": 5, + "max_survivors": 12 + }, + "autonomy": { + "max_stage_runs": 12, + "stop_on_no_progress": true, + "max_repeated_findings": 1, + "require_mutation_for_shipping": true + }, + "limits": { + "timeout_seconds": 120, + "max_rounds": 2, + "max_output_bytes": 16384 + } +} +``` + +Replace every sample path and command with **an already-installed, trusted executable**. This is a policy example, not something freview creates or enables automatically. The worker and reviewer need compatible noninteractive adapters, and *must* have authorization. `mutmut_driver.py` runs `mutmut run` followed by `mutmut export-cicd-stats` without using a shell; it neither installs mutmut nor guarantees mutations can run on every codebase. + +Alternatively, for StrykerJS, configure an already-installed `./node_modules/.bin/stryker` executable with `["./node_modules/.bin/stryker", "run", "--reporters", "json"]` and `mutation_quality.format="stryker"`; it must emit a standard `reports/mutation/mutation.json` file. Scope costly mutation runs to changed, high-risk code when the engine supports it. Stryker incremental mode can save work, but its change detection is not infallible when dependencies/environment change; `--force` is useful for a fresh scoped verification. Do not run `npm install`, `pip install` or download packages during a default audit. + +## Default behavior and tuning + +| Setting | Default | What it does | +| --- | --- | --- | +| Mutation module | **off** | No mutator runs | +| `mutation_quality.format` | `auto` | Recognize the two fixed report paths; ambiguous/missing is invalid or inconclusive | +| `mutation_quality.mode` | `advisory` | Show score without a threshold-enforced gate; **missing or incomplete evidence still blocks** | +| `mutation_quality.min_score` | 60 | Default advisory target; enforced only in `gate` mode | +| `mutation_quality.min_valid_mutants` | 1 | A zero-valid-mutant report is never treated as perfect | +| `mutation_quality.max_survivors` | 12 | Bounded survivor hints, not complete raw source | +| `autonomy.max_stage_runs` | 12 (maximum 32) | Whole-pipeline cap across planning, worker, tests, reviewer, regression, mutation | +| `autonomy.stop_on_no_progress` | `true` | Stop if worker does not change source after receiving actionable review feedback | +| `autonomy.max_repeated_findings` | 1 | Stop unchanged repeated finding signatures to avoid expensive oscillations | +| `autonomy.require_mutation_for_shipping` | `false` | Optional stricter *handoff-only* gate requiring a scored passing mutation run | +| `limits.max_rounds` | 3 (maximum 3) | Hard ceiling on review/fix iterations | + +**Safety note:** Even if a mutation score clears the configured gate, this only clears an optional checklist item. Production source is never auto-merged, committed, pushed or deployed by this layer. `shipping` returns a human handoff rather than an actual release. + +## Evidence, status and recovery + +- An enabled mutation program with exit code 0 but missing, stale, malformed or interrupted report produces `mutation-inconclusive`. +- With `mode="gate"`, a complete report below score or minimum valid-mutant threshold produces `mutation-threshold-failed`. In advisory mode, a *complete* low-scoring report is shown with `quality_pass=false`, but does not block the **human handoff** unless `require_mutation_for_shipping` is enabled. +- Any stage exceeding the configurable global cap yields `budget-exhausted` before launching another subprocess. +- A worker that receives findings and leaves the candidate unchanged yields `no-progress`. Repeated identical findings despite edits yield `stalled-review-findings`. +- Tests are run in a separate verifier copy of the current candidate; mutation programs can edit only their own verifier copy (unless they break OS-level boundaries). The original source SHA and source changes are kept distinct from test/mutant artifacts. +- Scratch `checkpoint.json` includes mutation score/counts and the fresh-report observation. A checkpoint is **not a signed attestation or a runnable resume**. Use `ops.py inspect-checkpoint --checkpoint FILE` to inspect only, and re-run exact-SHA tests after integrating any generated changes. + +Execution still requires `--allow-execute`, a clean immutable source commit, explicit `--scratch`, and `--allow-credentials` if the policy explicitly requests any environment secrets. **A cloned file tree is not an OS/container sandbox.** Do not execute untrusted repositories without additional isolation. For external-SSD-only project policies, select a verified mounted scratch path and never silently fall back to internal storage. + +## Honest acceptance boundary + +CI fixtures cover parser edge cases, fake mutator CLI commands and local-Git stage budgets on Ubuntu/macOS. They do **not** demonstrate paid model-provider invocation, real survivor-killing tests, signed provenance, prompt-injection resistance of arbitrary agents, real mutation workloads, or production autonomous deployment. Those are separate host-level and human release gates. From ec5c9d299c8bed2167282c1565032feace1ab4e8 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:05:03 +0530 Subject: [PATCH 24/33] ci/docs: include portable mutation helpers and explicit safe configuration defaults --- .github/workflows/checks.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 382713c..5c27b28 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -24,7 +24,7 @@ jobs: with: python-version: '3.11' - name: Check optional helper syntax - run: python3 -m py_compile skills/freview/scripts/repo_snapshot.py skills/freview/scripts/review.py skills/freview/scripts/report_tools.py skills/freview/scripts/duplicates.py skills/freview/scripts/agent_doctor.py skills/freview/scripts/spec_inventory.py skills/freview/scripts/receipt_check.py skills/freview/scripts/ops.py skills/freview/scripts/review_verdict.py + run: python3 -m py_compile skills/freview/scripts/repo_snapshot.py skills/freview/scripts/review.py skills/freview/scripts/report_tools.py skills/freview/scripts/duplicates.py skills/freview/scripts/agent_doctor.py skills/freview/scripts/spec_inventory.py skills/freview/scripts/receipt_check.py skills/freview/scripts/ops.py skills/freview/scripts/review_verdict.py skills/freview/scripts/mutation_report.py skills/freview/scripts/mutmut_driver.py - name: Run no-dependency fixtures run: python3 -m unittest discover -s tests -v - name: Exercise read-only local snapshot From d0ed8147af62aa011943e7d92301b4d353432b00 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:05:06 +0530 Subject: [PATCH 25/33] ci/docs: include portable mutation helpers and explicit safe configuration defaults --- tests/test_skill_layout.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_skill_layout.py b/tests/test_skill_layout.py index 7ce8cfc..580be54 100644 --- a/tests/test_skill_layout.py +++ b/tests/test_skill_layout.py @@ -26,7 +26,7 @@ def test_references_and_helper_are_bundled(self): for ref in references: with self.subTest(ref=ref): self.assertTrue((SKILL_DIR / ref).is_file()) - for tool in ("repo_snapshot.py", "review.py", "report_tools.py", "duplicates.py", "agent_doctor.py", "spec_inventory.py", "receipt_check.py", "ops.py", "review_verdict.py"): + for tool in ("repo_snapshot.py", "review.py", "report_tools.py", "duplicates.py", "agent_doctor.py", "spec_inventory.py", "receipt_check.py", "ops.py", "review_verdict.py", "mutation_report.py", "mutmut_driver.py"): self.assertTrue((SKILL_DIR / "scripts" / tool).is_file()) for asset in ("assets/schemas/freview-config.schema.json", "assets/schemas/freview-report.schema.json", From a0aee43818c43e5f96b370dd02d28ac936b8ddc7 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:05:10 +0530 Subject: [PATCH 26/33] ci/docs: include portable mutation helpers and explicit safe configuration defaults --- examples/freview-ops.example.json | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/examples/freview-ops.example.json b/examples/freview-ops.example.json index c51a228..1073c6f 100644 --- a/examples/freview-ops.example.json +++ b/examples/freview-ops.example.json @@ -26,5 +26,18 @@ "max_rounds": 2, "max_output_bytes": 16384 }, - "pass_env": [] + "pass_env": [], + "mutation_quality": { + "format": "auto", + "mode": "advisory", + "min_score": 60, + "min_valid_mutants": 1, + "max_survivors": 12 + }, + "autonomy": { + "max_stage_runs": 12, + "stop_on_no_progress": true, + "max_repeated_findings": 1, + "require_mutation_for_shipping": false + } } From 59a2025ec60fa59f96586ca7d00ddf6d81884546 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:05:31 +0530 Subject: [PATCH 27/33] docs(M3): add mutation/autonomy entry points without changing default read-only freview --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index 09b7243..338b386 100644 --- a/README.md +++ b/README.md @@ -97,6 +97,12 @@ python3 -I skills/freview/scripts/ops.py plan --repo . --requirements "Desired b A completed reviewer process cannot approve itself with plain text: freview validates exact source/workspace identity and retains [non-resuming atomic scratch checkpoints](skills/freview/references/review-verdict-protocol.md). The helper runs the worker in a **disposable committed-tree candidate**, then tests, reviewers and mutation tools in their own **fresh verification copies** (all under explicitly approved scratch), never in the original checkout. Test caches do not count as worker changes; a test modifying pre-existing source files or a reviewer modifying its copy blocks its respective gate. This is isolation of files, **not a secure OS sandbox**. Real Claude/Codex tokens/accounts, native agent activation, mutation scores, correctness, real release authorization and cross-agent independence remain separately verifiable and are **not** certified by synthetic CI fixtures. When a worker modifies the disposable source snapshot, freview records its workspace fingerprint and **blocks the shipping handoff until those edits are integrated and independently checked**. Shipping is a handoff, not an automated push, merge or deployment. [Design and recovery journal](docs/roadmap/optional-ops-2026-10.md). +### Mutation-quality gates and bounded autonomy (opt-in) + +The ops companion now understands Stryker JSON and mutmut CI stats **as actual mutation evidence**, rather than trusting a mutator's exit code alone. It reports detected, surviving/no-coverage, pending and unknown mutants, plus a bounded sample of survivor locations. Missing, stale, interrupted or malformed reports are inconclusive; a separately enabled score gate can block a handoff. The portable mutmut runner invokes an **already-installed** mutmut CLI and exports its stats without a shell. + +Worker/reviewer loops additionally enforce a **global subprocess budget** and stop when a worker makes no source progress or repeated findings remain unchanged. All behavior is explicitly configurable; all modules remain OFF by default. Scratch checkpoints retain evidence without automatically resuming or signing an audit. See the [mutation and autonomy guide](skills/freview/references/mutation-autonomy.md) and [configuration example](examples/freview-ops.example.json). + ## Review modes | Capability | Quick | Standard (daily) | Full | From 7689ad54a0179ada17cfe51b2c41f5daf6855f8a Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:05:35 +0530 Subject: [PATCH 28/33] docs(M3): add mutation/autonomy entry points without changing default read-only freview --- skills/freview/SKILL.md | 1 + 1 file changed, 1 insertion(+) diff --git a/skills/freview/SKILL.md b/skills/freview/SKILL.md index eae405c..9b40104 100644 --- a/skills/freview/SKILL.md +++ b/skills/freview/SKILL.md @@ -42,6 +42,7 @@ The local CLI explicitly imports its bundled sibling helper to support `python3 The companion local helper can produce a **local-only machine-readable JSON snapshot/report** with `python3 -I /scripts/review.py --mode standard --format json`, or write to an explicit `--output` path on approved scratch storage. The agent's comprehensive report is a separate artifact following [JSON report schema](assets/schemas/freview-report.schema.json). Never mislabel the helper's metadata-only output as a completed AI/remote audit. Read only relevant modules: +- [Mutation evidence and bounded autonomy (OFF by default)](references/mutation-autonomy.md) — strict Stryker/mutmut report quality, budget-limited repair loops and human release hold; never invoke these on audit discovery alone - [Adversarial verdict contract and interruption recovery](references/review-verdict-protocol.md) — strict reviewer JSON, exact scratch identity, durable read-only checkpoints; no automatic resume - [Optional operations (OFF by default)](references/optional-ops.md) — opt-in planning, isolated testing, optional reviewer/mutation/baseline checks, Git hooks and shipping handoff; **never run these merely because config exists** - [Specification inventory, evidence reconciliation and opt-in independent verification](references/evidence-reconciliation.md) — requirements sources, structured receipts, and limits on conclusions; no automatic execution From be0175a0d6f87056bf6a1255bc14ae92b1373885 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:06:46 +0530 Subject: [PATCH 29/33] docs: reconcile optional ops manual with real mutation reports and bounded autonomy defaults --- skills/freview/references/optional-ops.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/skills/freview/references/optional-ops.md b/skills/freview/references/optional-ops.md index 68bcd03..332527c 100644 --- a/skills/freview/references/optional-ops.md +++ b/skills/freview/references/optional-ops.md @@ -12,7 +12,7 @@ All nine modules are **disabled** unless a project explicitly enables them: `spe | implementation | Execute configured worker CLI on a **disposable committed snapshot** with spec prompt | No edits to original; review diff and apply separately | | tests | Execute an exact user-selected argv test command in scratch | No test discovery scripts automatically installed/executed | | adversarial | Launch independently configured reviewer CLI, fresh process; no worker summary in initial prompt | Reviewer verdict remains untrusted text; process exit 0 is **not** approval | -| mutation | Launch explicitly installed third-party mutation command in disposable snapshot | No internal mutator or interpreted score; review raw reports | +| mutation | Launch an already-installed Stryker or mutmut command in isolated verifier, validate fresh report and interpret score/survivors | Advisory by default; opt-in score gate; missing/incomplete evidence blocks; no built-in mutation engine | | regression | Run configured command, or dedicated `sensitivity` against exact baseline and candidate | Old-code failure can be unrelated; not correctness proof | | hooks | Explicitly install owned pre-commit/pre-push branch guard; refuse conflicts | Git hook is bypassable, not a server-side GitHub rule | | shipping | Generate exact-ref handoff checklist only, no publish | Separate human approval to merge/deploy/release elsewhere | @@ -63,7 +63,7 @@ No ambient scratch fallback. Resolve external SSD mount **before** choosing scra ## Config and controls -`schema_version:1`, `modules` booleans, `commands` explicit argv arrays (not shell command strings), `limits` caps, and optional `pass_env`. Defaults: 120 seconds per command; at most three review rounds; 16 KiB captured output; no forwarded credentials. Caps: 600 seconds, 3 rounds, 64 KiB output. Unexpected keys/invalid types block. +`schema_version:1`, `modules` booleans, `commands` explicit argv arrays (not shell command strings), `limits` caps, optional `pass_env`, `mutation_quality` and `autonomy`. Defaults: 120 seconds per command, at most three review rounds, a **total of 12 executed stages**, 16 KiB captured output, no forwarded credentials. Hard caps: 600 seconds/stage, 3 review rounds, 32 stages, 64 KiB output. Unexpected keys/invalid types block. See [mutation/autonomy defaults and tuning](mutation-autonomy.md). If a CLI needs existing account auth, set an **allowlisted** `pass_env` in the *trusted* policy and supply **both** `--allow-execute` and `--allow-credentials`. No keys are inherited implicitly. This is still not a sandbox, and forwarding secrets exposes them to that CLI and any child process it starts. Never install an agent tool from an untrusted repo as a side effect of review. @@ -85,6 +85,6 @@ Hooks never overwrite existing hooks or `core.hooksPath`. The pre-push guard ref ## Tokens, security and evaluation -Use Standard freview read-only for daily checks. Use an opt-in pipeline only for genuinely significant changes. Short prompts and <=3 review rounds avoid unbounded agent loops; the default enabled module set is empty, so default agent token expenditure is zero. Only return summarized receipts with private logs left under explicitly selected scratch. +Use Standard freview read-only for daily checks. Use an opt-in pipeline only for genuinely significant changes. Short prompts, <=3 review rounds, a global subprocess cap, no-progress detection and repeated-finding protection limit token consumption; default enabled modules remain empty, so no agent execution occurs without opt-in. Only return summarized receipts with private logs left under explicitly selected scratch. The tests validate **deterministic integration behavior** with fake local programs and Git repos. CI must not invoke real Claude/Codex providers, paid APIs, mutation services, remote publishing or physical hardware. A real full self-review needs actual eligible agents and sandbox authorization, and must report any absent evidence. Do not call a schema-valid receipt an execution attestation. From ebfafccfe3308fd7b22bd953a707f88596f6bcb3 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:08:03 +0530 Subject: [PATCH 30/33] docs(M3): three bounded mutation/autonomy review passes and observed limits --- ...-10-17-mutation-autonomy-bounded-review.md | 46 +++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 docs/audits/2026-10-17-mutation-autonomy-bounded-review.md diff --git a/docs/audits/2026-10-17-mutation-autonomy-bounded-review.md b/docs/audits/2026-10-17-mutation-autonomy-bounded-review.md new file mode 100644 index 0000000..6ff3e07 --- /dev/null +++ b/docs/audits/2026-10-17-mutation-autonomy-bounded-review.md @@ -0,0 +1,46 @@ +# freview-on-freview: mutation and autonomy v1 bounded review + +**Date:** 2026-10-17. **Scope:** stacked draft PR #20 over #19. +**Method:** one bounded GPT source/test-contract review in three passes, GitHub Actions Ubuntu/macOS deterministic fixtures. **Not** three independent models or a real mutmut/Stryker mutation campaign. + +## Product question + +Does freview make *opt-in mutation and autonomous repair safer, measurable, configurable and not misleading* while remaining a read-only repository auditor by default? + +**Answer from bounded evidence:** yes for the deterministic report/runner semantics and synthetic fixtures. Real provider/mutator reliability, survivor usefulness, OS sandbox behavior and protected release mechanics remain unobserved. + +## Pass 1 — mutation score and real tool compatibility + +- Consulted [Stryker official mutant states](https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/) and the [mutation-testing-elements report schema](https://github.com/stryker-mutator/mutation-testing-elements/blob/master/packages/report-schema/src/mutation-testing-report-schema.json): Killed/Timeout = detected; Survived/NoCoverage = undetected; Pending makes report incomplete; ignored/error mutants are not counted as valid. +- Implemented a stdlib-only, size-bounded, duplicate-key-safe parser and fixed report discovery; includes bounded survivor path/line hints without copying report source. Zero valid mutants or missing/ambiguous/oversized/symlinked reports cannot pass. Score thresholds compare the **unrounded** numerator, avoiding an accidental pass at rounding boundaries. +- Independently inspected [mutmut implementation](https://github.com/boxed/mutmut/blob/main/src/mutmut/__main__.py): its `export-cicd-stats` includes a `total` whose components may include **unexported** `not_checked` and `caught_by_type_check`. Instead of inventing categories or rejecting every real report, we mark the difference `unclassified` and **incomplete**. Tests cover interrupted, suspicious, unknown and inconsistent totals. +- Added `mutmut_driver.py` to run already-installed `mutmut run` and export the supported CI JSON without invoking a shell. Fake executable tests prove command sequencing and fail-closed missing export; no real engine installed or executed here. +- **Limit:** a successful scored report is not proof tests cover intended behaviors, and native mutation generation was not exercised. + +## Pass 2 — autonomy loops and evidence gates + +- Configurable, bounded `autonomy.max_stage_runs` (default 12, hard max 32) limits total invoked stages, with timeout/output limits per stage and max 3 review rounds. +- On actionable review findings, stop if the worker makes no candidate source change; also stop if the exact finding signature repeats despite unrelated edits. Both stop conditions are regression-tested. +- Mutation command exit code zero alone **cannot pass** even in advisory mode. Both `run mutation` and `pipeline` now require a newly written report in the disposable mutation verifier workspace. A committed stale report cannot be credited to the current run; incomplete, missing or malformed report blocks. The optional `gate` mode requires configurable minimum valid mutants and score. +- `require_mutation_for_shipping` can be enabled to require a quality passing mutation check before producing even a **handoff-only** shipping checklist. +- Mutation score/counts and fresh-report observation are persisted in atomic, read-only scratch checkpoints. Checkpoints are **not** signed execution attestations and do not automatically resume commands. +- **Limit:** stage and report identifiers are still local metadata, not cryptographically trusted, and the verification copy is not an OS sandbox. + +## Pass 3 — integration, UX, regressions and release gates + +- Old tests that expected success from a mutator's zero exit were updated to correctly expect `mutation-inconclusive`; earlier three-loop fixtures were updated for no-progress stop. Initial red CI tested missing modules, then later CI caught integration and schema assumptions. No green state was inferred from older test runs. +- `examples/freview-ops.example.json` explicitly documents the new controls with every module still disabled, and the README/SKILL/optional ops manual link to a detailed [mutation/autonomy guide](../../skills/freview/references/mutation-autonomy.md). +- CI includes the new scripts in compilation checks and static bundled-file tests. No third-party source code was copied; all shipped scripts use the Python standard library. +- **Release hold:** draft PRs #16–#20 are not merged; `main` remains untouched. No real paid reviewer or mutmut/Stryker run, external SSD smoke, remote shipping, branch-protection configuration or installation performed in this cycle. + +## Evidence + +- Deliberate test-first red run: https://github.com/raiseCatError/freview/actions/runs/39017038480. +- Corrected legacy semantics green: https://github.com/raiseCatError/freview/actions/runs/39017271191. +- Stale-report provenance fix green: https://github.com/raiseCatError/freview/actions/runs/39017378141. +- Standalone mutation and mutmut fake runner integration green: https://github.com/raiseCatError/freview/actions/runs/39017691880. +- Mutation checkpoint summary schema green: https://github.com/raiseCatError/freview/actions/runs/39017817033. +- Packaged skill and policy default example green: https://github.com/raiseCatError/freview/actions/runs/39017925557. +- **Final exact-branch-head CI still needs to be fetched and verified after this document and checkpoint journal commits.** Older greens do not certify later heads. + +**Verdict:** scoped feature and code-quality work complete pending final exact-head CI. No basis to claim authenticated multi-agent autonomous shipping or actual mutation engine score on a production repository. User confirmation is required before merging stacked PRs or running connected tools on a real checkout. From 703b359fbd317fb718a7cef0cba630cbf2cc940d Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:08:41 +0530 Subject: [PATCH 31/33] docs(M3): close staged checkpoints and record exact source/test CI and release gates --- docs/roadmap/mutation-autonomy-v1-2026-10.md | 44 ++++++++++++-------- 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/docs/roadmap/mutation-autonomy-v1-2026-10.md b/docs/roadmap/mutation-autonomy-v1-2026-10.md index a20e99e..1c1aaf2 100644 --- a/docs/roadmap/mutation-autonomy-v1-2026-10.md +++ b/docs/roadmap/mutation-autonomy-v1-2026-10.md @@ -1,6 +1,6 @@ # freview Mutation & Bounded Autonomy v1 — tests-first specification -**Date:** 2026-10-17. **Status:** implementation in progress. +**Date:** 2026-10-17. **Status:** source and tests complete, final documentation-head CI pending. **Branch:** `feature/mutation-autonomy-v1-20261017`, stacked onto draft PR #19 at `e2b269ea6f56b0caed1ac898584476bef5ec4f6f`. Neither this branch nor ancestors may merge/release without separate approval. ## Product intent and borrowed practices @@ -23,7 +23,7 @@ All modules remain OFF absent an explicit `.freview-ops.json`. Existing `limits` New optional `autonomy`: - `max_stage_runs`: 12 (hard cap 32; applies across all nested worker/test/reviewer/mutation/regression calls). - `stop_on_no_progress`: true. If reviewer requests change, next worker produces no candidate source change, stop with `no-progress` rather than burning next review tokens. -- `max_repeated_findings`: 1 (1–3) applies only to exact repeated finding fingerprints without candidate source delta; prevents review oscillation. +- `max_repeated_findings`: 1 (1–3) stops identical repeated finding fingerprints, even if the worker makes unrelated changes; prevents review oscillation. - `require_mutation_for_shipping`: false by default. If opted in, no shipping handoff without a successful mutation gate on same candidate. Shipping remains handoff-only, never merge/deploy. - Existing max 3 review rounds, explicit scratch, exact SHA, credential flags and no automatic original-repo edits remain. - No automatic local worktree resets, installs, pip/npm, branch pushes or external PR/releases. No OS sandbox promises. @@ -32,24 +32,24 @@ New optional `autonomy`: ### M0 — red tests first - [x] Create branch from exact PR #19 head, write durable spec/recovery before implementation. -- [ ] Add direct parser tests for Stryker/mutmut normal, no survivors, incomplete, tampered totals, malformed oversized, no valid mutants, bounded survivors, threshold decisions. -- [ ] Add integration tests for configured mutation report required, gate/advisory behavior, worker no-progress/budget exhaustion, and unchanged read-only defaults. -- [ ] Capture expected failing Ubuntu/macOS CI. Tests must actually run (no unittest method shadowing). +- [x] Add direct parser tests for Stryker/mutmut normal, no survivors, incomplete, tampered totals, malformed oversized, no valid mutants, bounded survivors, threshold decisions. +- [x] Add integration tests for configured mutation report required, gate/advisory behavior, worker no-progress/budget exhaustion, and unchanged read-only defaults. +- [x] Capture expected failing Ubuntu/macOS CI. Tests must actually run (no unittest method shadowing). ### M1 — mutation engine reports -- [ ] Implement pure `mutation_report.py` parser and CLI (stdlib only). Never follow report symlinks or leak full source text. -- [ ] Connect mutation runner's result with exact observed verifier workspace and checkpoint report. Verify exact-head both OS. +- [x] Implement pure `mutation_report.py` parser and CLI (stdlib only). Never follow report symlinks or leak full source text. +- [x] Connect mutation runner's result with exact observed verifier workspace and checkpoint report. Verify exact-head both OS. ### M2 — bounded autonomy -- [ ] Add centrally validated policy defaults and safe optional knobs; stage run budget and no-progress avoidance. -- [ ] Honor opt-in mutation quality gate before proposing shipping; include detailed verdict and provenance. -- [ ] Verify full suite both OS, including all previous 193 tests. +- [x] Add centrally validated policy defaults and safe optional knobs; stage run budget and no-progress avoidance. +- [x] Honor opt-in mutation quality gate before proposing shipping; include detailed verdict and provenance. +- [x] Verify full suite both OS, including all previous 193 tests. ### M3 — 3-pass self-review & handoff -- [ ] Review independent code and plan for ambiguous mutation statuses / wrong math. -- [ ] Review code for path traversal, budgets, stage reuse, false shipping claims. -- [ ] Review UX/docs/help/compatibility; fix, add tests, rerun. -- [ ] Commit a bounded self-audit, final exact-head CI and draft stacked PR. **No merge/push to main/release**. +- [x] Review independent code and plan for ambiguous mutation statuses / wrong math. +- [x] Review code for path traversal, budgets, stage reuse, false shipping claims. +- [x] Review UX/docs/help/compatibility; fix, add tests, rerun. +- [x] Commit a bounded self-audit, final exact-head CI and draft stacked PR. **No merge/push to main/release**. ## Recovery / interruption @@ -61,7 +61,15 @@ New optional `autonomy`: | Phase | Commit | CI proof | Status | | --- | --- | --- | --- | -| M0 | pending | red tests | in progress | -| M1 | pending | two-OS exact SHA | not started | -| M2 | pending | two-OS exact SHA | not started | -| M3 | pending | review and final SHA | not started | +| M0 | 1567ef8e | [Intentional red tests before implementation](https://github.com/raiseCatError/freview/actions/runs/39017038480) | Complete | +| M1 | c06822c5 | [Report parser + stale-report regression Ubuntu/macOS](https://github.com/raiseCatError/freview/actions/runs/39017378141) | Complete | +| M2 | 4caf17d9 / 42dab6e3 | [Standalone consistency](https://github.com/raiseCatError/freview/actions/runs/39017607854), [checkpoint evidence both OS](https://github.com/raiseCatError/freview/actions/runs/39017817033) | Complete | +| M3 | latest branch | [Three bounded passes](../audits/2026-10-17-mutation-autonomy-bounded-review.md); final docs-head CI must be verified | Final gate pending | + +## Completion and recovery checkpoint (2026-10-17) + +- Test-first M0 red run: https://github.com/raiseCatError/freview/actions/runs/39017038480. The new parser did not exist when tests were committed. Subsequent backward-compatibility failures were investigated and old exit-code-only tests were updated to assert the stronger mutation contract. +- Green code/fixture receipts: M1 https://github.com/raiseCatError/freview/actions/runs/39017378141; M2 https://github.com/raiseCatError/freview/actions/runs/39017607854 and https://github.com/raiseCatError/freview/actions/runs/39017817033. The fake mutmut driver test passed at https://github.com/raiseCatError/freview/actions/runs/39017691880. +- M3 bounded self-review: [three-pass source and evidence review](../audits/2026-10-17-mutation-autonomy-bounded-review.md). Source uses official Stryker report format and verified mutmut exporter details, but never ran a real external mutation engine. `mutmut` CI exporter omits classifications included in its `total`; unclassified counts yield incomplete rather than a guessed score. Unrounded threshold comparisons block false green from rounding. +- All user-controlled programs remain explicit opt-in. Mutation and autonomy are off without policy; shipping is a **handoff only**. Real agent/mutation physical-host and external SSD sandbox acceptance remain unperformed. +- **Final gate:** verify exact current branch SHA and Ubuntu/macOS CI after every documentation commit, update PR #20 acceptance checks; leave PRs #16–#20 in draft and `main` unchanged. No merge, publish, release, install or background task. From 9fed609bb2f844b994d9e9ae38a45d4401bec85b Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:08:56 +0530 Subject: [PATCH 32/33] docs(M3): index mutation/autonomy audit with honest host-acceptance limits --- docs/audits/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/audits/README.md b/docs/audits/README.md index 05ce1e1..f40e587 100644 --- a/docs/audits/README.md +++ b/docs/audits/README.md @@ -12,6 +12,7 @@ An evidence log, **not a list of software features or an endorsement of any agen | [Optional ops three-pass bounded review](2026-10-17-optional-ops-three-pass-self-review.md) | Stacked draft PR #17, test-first development, three bounded review–fix–CI cycles | Host-side source inspection and synthetic Ubuntu/macOS test evidence. **No actual independent second model, mutation run, native installed-agent QA or autonomous publication.** | | [Structured reviewer verdicts and scratch recovery self-review](2026-10-17-structured-review-and-checkpoints.md) | Stacked draft PR #18, checked against PR #17 at `4ad489ec` | Test-first source review, verified synthetic reviewer JSON protocol, recovery manifest and three bounded verification gates. **No real independent model activation or automatic release.** | | [Verifier isolation and code hygiene](2026-10-17-ops-quality-isolation.md) | Stacked draft PR #19 above #18, Ubuntu/macOS synthetic Git fixture evidence | Fixed test-cache contamination, source-write isolation, mode-aware workspace identities and verifier-copy provenance. **No independent external model invocation or OS sandbox certification.** | +| [Mutation evidence and autonomy bounded review](2026-10-17-mutation-autonomy-bounded-review.md) | Stacked draft PR #20, test-first implementation atop #19 | Stryker/mutmut parser, stale-report safety, scored mutation gate, stage budget, repeated-finding stop, complete fake CI evidence; **no real mutation engine, provider identity or production shipping verified**. | The NMSh field-test generated four **private physical-host documents** (Markdown/JSON audit, freview product field critique and exact-command manifest). Those documents can include local worktree paths and unreleased project state. They are **not published verbatim** to the public repository; the cross-review provides a bounded, privacy-conscious summary instead. From d7e991f616ccd3231f667c6a230ac4c349813e68 Mon Sep 17 00:00:00 2001 From: J Date: Sat, 17 Oct 2026 14:11:21 +0530 Subject: [PATCH 33/33] =?UTF-8?q?docs:=20record=20completed=20M0=E2=80=93M?= =?UTF-8?q?3=20with=20exact=20Ubuntu/macOS=20verification=20and=20release?= =?UTF-8?q?=20hold?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/roadmap/mutation-autonomy-v1-2026-10.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/roadmap/mutation-autonomy-v1-2026-10.md b/docs/roadmap/mutation-autonomy-v1-2026-10.md index 1c1aaf2..6a2a200 100644 --- a/docs/roadmap/mutation-autonomy-v1-2026-10.md +++ b/docs/roadmap/mutation-autonomy-v1-2026-10.md @@ -1,6 +1,6 @@ # freview Mutation & Bounded Autonomy v1 — tests-first specification -**Date:** 2026-10-17. **Status:** source and tests complete, final documentation-head CI pending. +**Date:** 2026-10-17. **Status:** implementation, review and initial final-head CI complete; owner merge/release held. **Branch:** `feature/mutation-autonomy-v1-20261017`, stacked onto draft PR #19 at `e2b269ea6f56b0caed1ac898584476bef5ec4f6f`. Neither this branch nor ancestors may merge/release without separate approval. ## Product intent and borrowed practices @@ -64,7 +64,7 @@ New optional `autonomy`: | M0 | 1567ef8e | [Intentional red tests before implementation](https://github.com/raiseCatError/freview/actions/runs/39017038480) | Complete | | M1 | c06822c5 | [Report parser + stale-report regression Ubuntu/macOS](https://github.com/raiseCatError/freview/actions/runs/39017378141) | Complete | | M2 | 4caf17d9 / 42dab6e3 | [Standalone consistency](https://github.com/raiseCatError/freview/actions/runs/39017607854), [checkpoint evidence both OS](https://github.com/raiseCatError/freview/actions/runs/39017817033) | Complete | -| M3 | latest branch | [Three bounded passes](../audits/2026-10-17-mutation-autonomy-bounded-review.md); final docs-head CI must be verified | Final gate pending | +| M3 | 9fed609b | [Three bounded passes](../audits/2026-10-17-mutation-autonomy-bounded-review.md) + [217 tests, Ubuntu/macOS exact-head success](https://github.com/raiseCatError/freview/actions/runs/39018155210) | Complete as of 9fed609b; this journal update requires its own CI | ## Completion and recovery checkpoint (2026-10-17) @@ -73,3 +73,9 @@ New optional `autonomy`: - M3 bounded self-review: [three-pass source and evidence review](../audits/2026-10-17-mutation-autonomy-bounded-review.md). Source uses official Stryker report format and verified mutmut exporter details, but never ran a real external mutation engine. `mutmut` CI exporter omits classifications included in its `total`; unclassified counts yield incomplete rather than a guessed score. Unrounded threshold comparisons block false green from rounding. - All user-controlled programs remain explicit opt-in. Mutation and autonomy are off without policy; shipping is a **handoff only**. Real agent/mutation physical-host and external SSD sandbox acceptance remain unperformed. - **Final gate:** verify exact current branch SHA and Ubuntu/macOS CI after every documentation commit, update PR #20 acceptance checks; leave PRs #16–#20 in draft and `main` unchanged. No merge, publish, release, install or background task. + +## Final confirmation and handoff + +- Verified `9fed609bb2f844b994d9e9ae38a45d4401bec85b` at https://github.com/raiseCatError/freview/actions/runs/39018155210: **Ubuntu and macOS jobs both successful**; Ubuntu recorded **217 passing tests**. The two-OS CI jobs provide source/test evidence, **not** proof of real mutation engine performance or independent agent review. +- Stacked PRs #16, #17, #18, #19, #20 verified **draft and unmerged**; `main` unchanged at `2646e4c35e2383b108a752893d846cd48989f2a8` at time of inspection. +- This final journal amendment itself changes the branch HEAD: check the workflow for the newer exact SHA and update PR #20 once it turns green. No further code edits intended. The **release hold remains active** pending separate human approval and real provider/host mutation tests.