Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
ab8b16d
docs: test-first mutation quality and bounded autonomy spec/recovery …
raiseCatError Oct 17, 2026
a7053fe
test(M0): strict Stryker/mutmut mutation-score and artifact validatio…
raiseCatError Oct 17, 2026
1567ef8
test(M0): mutation report gate, safe defaults, and global autonomy st…
raiseCatError Oct 17, 2026
d82b5f2
feat(M1): bounded Stryker/mutmut score, survivor and completion parser
raiseCatError Oct 17, 2026
fc9820a
feat(M2): validated mutation quality and bounded autonomy policy defa…
raiseCatError Oct 17, 2026
39798e3
feat(M2): inspect real mutation artifact; global stage budget and no-…
raiseCatError Oct 17, 2026
ddf501c
test: align legacy contracts with evidence-first mutation and no-prog…
raiseCatError Oct 17, 2026
4361e3f
test: align legacy contracts with evidence-first mutation and no-prog…
raiseCatError Oct 17, 2026
e416b2b
test: align legacy contracts with evidence-first mutation and no-prog…
raiseCatError Oct 17, 2026
da41b3b
test: align legacy contracts with evidence-first mutation and no-prog…
raiseCatError Oct 17, 2026
4708d08
test(M2): stale committed mutation reports cannot pass; optional ship…
raiseCatError Oct 17, 2026
c06822c
fix(M2): require fresh mutation report generated by the current isola…
raiseCatError Oct 17, 2026
47a3eed
test(M2): standalone mutation must interpret reports and reject exit-…
raiseCatError Oct 17, 2026
539c444
feat(M2): apply same mutation report quality verdict to standalone an…
raiseCatError Oct 17, 2026
2221510
test(M3): mutmut export omits categories; never round mutation score …
raiseCatError Oct 17, 2026
60d6f34
fix(M3): match mutmut's actually exported totals and compare unrounde…
raiseCatError Oct 17, 2026
4caf17d
test: exit-zero mutation without report is inconclusive in standalone…
raiseCatError Oct 17, 2026
e90a788
test(M3): shell-free mutmut run plus explicit export adapter with fai…
raiseCatError Oct 17, 2026
aec1c4d
feat(M3): bounded no-shell mutmut run/export adapter for report-based…
raiseCatError Oct 17, 2026
dcd1c63
test(M3): stop repeated adversarial findings even with unrelated work…
raiseCatError Oct 17, 2026
92bfee6
test(M3): mutation score and fresh-report provenance survive atomic c…
raiseCatError Oct 17, 2026
42dab6e
feat(M3): checkpoint actual mutation score and fresh-report evidence …
raiseCatError Oct 17, 2026
6c75068
docs(M3): user-facing mutation score and bounded autonomy manual with…
raiseCatError Oct 17, 2026
ec5c9d2
ci/docs: include portable mutation helpers and explicit safe configur…
raiseCatError Oct 17, 2026
d0ed814
ci/docs: include portable mutation helpers and explicit safe configur…
raiseCatError Oct 17, 2026
a0aee43
ci/docs: include portable mutation helpers and explicit safe configur…
raiseCatError Oct 17, 2026
59a2025
docs(M3): add mutation/autonomy entry points without changing default…
raiseCatError Oct 17, 2026
7689ad5
docs(M3): add mutation/autonomy entry points without changing default…
raiseCatError Oct 17, 2026
be0175a
docs: reconcile optional ops manual with real mutation reports and bo…
raiseCatError Oct 17, 2026
ebfafcc
docs(M3): three bounded mutation/autonomy review passes and observed …
raiseCatError Oct 17, 2026
703b359
docs(M3): close staged checkpoints and record exact source/test CI an…
raiseCatError Oct 17, 2026
9fed609
docs(M3): index mutation/autonomy audit with honest host-acceptance l…
raiseCatError Oct 17, 2026
d7e991f
docs: record completed M0–M3 with exact Ubuntu/macOS verification and…
raiseCatError Oct 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
with:
python-version: '3.11'
- name: Check optional helper syntax
run: python3 -m py_compile skills/freview/scripts/repo_snapshot.py skills/freview/scripts/review.py skills/freview/scripts/report_tools.py skills/freview/scripts/duplicates.py skills/freview/scripts/agent_doctor.py skills/freview/scripts/spec_inventory.py skills/freview/scripts/receipt_check.py skills/freview/scripts/ops.py skills/freview/scripts/review_verdict.py
run: python3 -m py_compile skills/freview/scripts/repo_snapshot.py skills/freview/scripts/review.py skills/freview/scripts/report_tools.py skills/freview/scripts/duplicates.py skills/freview/scripts/agent_doctor.py skills/freview/scripts/spec_inventory.py skills/freview/scripts/receipt_check.py skills/freview/scripts/ops.py skills/freview/scripts/review_verdict.py skills/freview/scripts/mutation_report.py skills/freview/scripts/mutmut_driver.py
- name: Run no-dependency fixtures
run: python3 -m unittest discover -s tests -v
- name: Exercise read-only local snapshot
Expand Down
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,12 @@ python3 -I skills/freview/scripts/ops.py plan --repo . --requirements "Desired b

A completed reviewer process cannot approve itself with plain text: freview validates exact source/workspace identity and retains [non-resuming atomic scratch checkpoints](skills/freview/references/review-verdict-protocol.md). The helper runs the worker in a **disposable committed-tree candidate**, then tests, reviewers and mutation tools in their own **fresh verification copies** (all under explicitly approved scratch), never in the original checkout. Test caches do not count as worker changes; a test modifying pre-existing source files or a reviewer modifying its copy blocks its respective gate. This is isolation of files, **not a secure OS sandbox**. Real Claude/Codex tokens/accounts, native agent activation, mutation scores, correctness, real release authorization and cross-agent independence remain separately verifiable and are **not** certified by synthetic CI fixtures. When a worker modifies the disposable source snapshot, freview records its workspace fingerprint and **blocks the shipping handoff until those edits are integrated and independently checked**. Shipping is a handoff, not an automated push, merge or deployment. [Design and recovery journal](docs/roadmap/optional-ops-2026-10.md).

### Mutation-quality gates and bounded autonomy (opt-in)

The ops companion now understands Stryker JSON and mutmut CI stats **as actual mutation evidence**, rather than trusting a mutator's exit code alone. It reports detected, surviving/no-coverage, pending and unknown mutants, plus a bounded sample of survivor locations. Missing, stale, interrupted or malformed reports are inconclusive; a separately enabled score gate can block a handoff. The portable mutmut runner invokes an **already-installed** mutmut CLI and exports its stats without a shell.

Worker/reviewer loops additionally enforce a **global subprocess budget** and stop when a worker makes no source progress or repeated findings remain unchanged. All behavior is explicitly configurable; all modules remain OFF by default. Scratch checkpoints retain evidence without automatically resuming or signing an audit. See the [mutation and autonomy guide](skills/freview/references/mutation-autonomy.md) and [configuration example](examples/freview-ops.example.json).

## Review modes

| Capability | Quick | Standard (daily) | Full |
Expand Down
46 changes: 46 additions & 0 deletions docs/audits/2026-10-17-mutation-autonomy-bounded-review.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# freview-on-freview: mutation and autonomy v1 bounded review

**Date:** 2026-10-17. **Scope:** stacked draft PR #20 over #19.
**Method:** one bounded GPT source/test-contract review in three passes, GitHub Actions Ubuntu/macOS deterministic fixtures. **Not** three independent models or a real mutmut/Stryker mutation campaign.

## Product question

Does freview make *opt-in mutation and autonomous repair safer, measurable, configurable and not misleading* while remaining a read-only repository auditor by default?

**Answer from bounded evidence:** yes for the deterministic report/runner semantics and synthetic fixtures. Real provider/mutator reliability, survivor usefulness, OS sandbox behavior and protected release mechanics remain unobserved.

## Pass 1 — mutation score and real tool compatibility

- Consulted [Stryker official mutant states](https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/) and the [mutation-testing-elements report schema](https://github.com/stryker-mutator/mutation-testing-elements/blob/master/packages/report-schema/src/mutation-testing-report-schema.json): Killed/Timeout = detected; Survived/NoCoverage = undetected; Pending makes report incomplete; ignored/error mutants are not counted as valid.
- Implemented a stdlib-only, size-bounded, duplicate-key-safe parser and fixed report discovery; includes bounded survivor path/line hints without copying report source. Zero valid mutants or missing/ambiguous/oversized/symlinked reports cannot pass. Score thresholds compare the **unrounded** numerator, avoiding an accidental pass at rounding boundaries.
- Independently inspected [mutmut implementation](https://github.com/boxed/mutmut/blob/main/src/mutmut/__main__.py): its `export-cicd-stats` includes a `total` whose components may include **unexported** `not_checked` and `caught_by_type_check`. Instead of inventing categories or rejecting every real report, we mark the difference `unclassified` and **incomplete**. Tests cover interrupted, suspicious, unknown and inconsistent totals.
- Added `mutmut_driver.py` to run already-installed `mutmut run` and export the supported CI JSON without invoking a shell. Fake executable tests prove command sequencing and fail-closed missing export; no real engine installed or executed here.
- **Limit:** a successful scored report is not proof tests cover intended behaviors, and native mutation generation was not exercised.

## Pass 2 — autonomy loops and evidence gates

- Configurable, bounded `autonomy.max_stage_runs` (default 12, hard max 32) limits total invoked stages, with timeout/output limits per stage and max 3 review rounds.
- On actionable review findings, stop if the worker makes no candidate source change; also stop if the exact finding signature repeats despite unrelated edits. Both stop conditions are regression-tested.
- Mutation command exit code zero alone **cannot pass** even in advisory mode. Both `run mutation` and `pipeline` now require a newly written report in the disposable mutation verifier workspace. A committed stale report cannot be credited to the current run; incomplete, missing or malformed report blocks. The optional `gate` mode requires configurable minimum valid mutants and score.
- `require_mutation_for_shipping` can be enabled to require a quality passing mutation check before producing even a **handoff-only** shipping checklist.
- Mutation score/counts and fresh-report observation are persisted in atomic, read-only scratch checkpoints. Checkpoints are **not** signed execution attestations and do not automatically resume commands.
- **Limit:** stage and report identifiers are still local metadata, not cryptographically trusted, and the verification copy is not an OS sandbox.

## Pass 3 — integration, UX, regressions and release gates

- Old tests that expected success from a mutator's zero exit were updated to correctly expect `mutation-inconclusive`; earlier three-loop fixtures were updated for no-progress stop. Initial red CI tested missing modules, then later CI caught integration and schema assumptions. No green state was inferred from older test runs.
- `examples/freview-ops.example.json` explicitly documents the new controls with every module still disabled, and the README/SKILL/optional ops manual link to a detailed [mutation/autonomy guide](../../skills/freview/references/mutation-autonomy.md).
- CI includes the new scripts in compilation checks and static bundled-file tests. No third-party source code was copied; all shipped scripts use the Python standard library.
- **Release hold:** draft PRs #16–#20 are not merged; `main` remains untouched. No real paid reviewer or mutmut/Stryker run, external SSD smoke, remote shipping, branch-protection configuration or installation performed in this cycle.

## Evidence

- Deliberate test-first red run: https://github.com/raiseCatError/freview/actions/runs/39017038480.
- Corrected legacy semantics green: https://github.com/raiseCatError/freview/actions/runs/39017271191.
- Stale-report provenance fix green: https://github.com/raiseCatError/freview/actions/runs/39017378141.
- Standalone mutation and mutmut fake runner integration green: https://github.com/raiseCatError/freview/actions/runs/39017691880.
- Mutation checkpoint summary schema green: https://github.com/raiseCatError/freview/actions/runs/39017817033.
- Packaged skill and policy default example green: https://github.com/raiseCatError/freview/actions/runs/39017925557.
- **Final exact-branch-head CI still needs to be fetched and verified after this document and checkpoint journal commits.** Older greens do not certify later heads.

**Verdict:** scoped feature and code-quality work complete pending final exact-head CI. No basis to claim authenticated multi-agent autonomous shipping or actual mutation engine score on a production repository. User confirmation is required before merging stacked PRs or running connected tools on a real checkout.
1 change: 1 addition & 0 deletions docs/audits/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ An evidence log, **not a list of software features or an endorsement of any agen
| [Optional ops three-pass bounded review](2026-10-17-optional-ops-three-pass-self-review.md) | Stacked draft PR #17, test-first development, three bounded review–fix–CI cycles | Host-side source inspection and synthetic Ubuntu/macOS test evidence. **No actual independent second model, mutation run, native installed-agent QA or autonomous publication.** |
| [Structured reviewer verdicts and scratch recovery self-review](2026-10-17-structured-review-and-checkpoints.md) | Stacked draft PR #18, checked against PR #17 at `4ad489ec` | Test-first source review, verified synthetic reviewer JSON protocol, recovery manifest and three bounded verification gates. **No real independent model activation or automatic release.** |
| [Verifier isolation and code hygiene](2026-10-17-ops-quality-isolation.md) | Stacked draft PR #19 above #18, Ubuntu/macOS synthetic Git fixture evidence | Fixed test-cache contamination, source-write isolation, mode-aware workspace identities and verifier-copy provenance. **No independent external model invocation or OS sandbox certification.** |
| [Mutation evidence and autonomy bounded review](2026-10-17-mutation-autonomy-bounded-review.md) | Stacked draft PR #20, test-first implementation atop #19 | Stryker/mutmut parser, stale-report safety, scored mutation gate, stage budget, repeated-finding stop, complete fake CI evidence; **no real mutation engine, provider identity or production shipping verified**. |

The NMSh field-test generated four **private physical-host documents** (Markdown/JSON audit, freview product field critique and exact-command manifest). Those documents can include local worktree paths and unreleased project state. They are **not published verbatim** to the public repository; the cross-review provides a bounded, privacy-conscious summary instead.

Expand Down
Loading
Loading