Repository navigation
Add governed self-evolution for Copilot workflows - #296
Merged
Merged
Conversation
…orkflow fingerprints Meta-improvement engine P0, first slice (design: meta-improvement-engine-design.md §3, §7.1). Every piece is additive; production behaviour is unchanged unless TRACE_STORE_ROOT is set. trace_store: `tool_call` joins the record kinds; the index gains workflow, unit_id and fingerprint columns behind `PRAGMA user_version` (schema 2). Opening a store never migrates: a v0 index runs in compatibility mode (named column inserts omit the new columns; querying by them raises IndexNotMigrated). `migrate_index` is the only upgrade path: cross-process O_EXCL lock with stale takeover, a writer-pause gate, a backup, one BEGIN IMMEDIATE transaction for ALTER + backfill of every existing row from its stored JSON + indexes + user_version, then verification (no JSON value left in a NULL column; each workflow's indexed ids equal the JSONL scan). `rebuild_index` builds into a temp file and swaps atomically under the same gate, with `to_schema=0` as the pin-downgrade path; `compare_index` and `verify_index` are read-only. `bind_store`/`current_store` let the choke points capture without importing anything. Capture at the two choke points: `tools.dispatch` records every call (allowed, failed, refused) as a tool_call with args/result blobs; `LLM.create` records every provider call as a model_call with system, messages and reply blobs, and a failed call as a record with error set and no outputs. Both swallow their own failures. improve/: workflow declarations (`enroll`: kind, unit, item_key, fingerprint.covers, capture, shadow tool/executable lists; malformed files raise) with two builtin declarations, and the declared configuration fingerprint (`fingerprint.compute`: playbook sha, settings, prompt file hashes, routing, tools, knowledge snapshot, copilot sha; incomplete covers yield no fingerprint and name what was missing; `diff` names the changed cover). The executor binds the store for the run when settings.trace_store_root is set and wraps every step in the unit context (run_id, playbook, step, unit_id, and for enrolled steps workflow, item and fingerprint). `infermatrix-copilot improve migrate-index|rebuild-index|rollback-index| compare-index|verify-index` operate the index; the pause gate checks the knowledge ledger (every repository row paused) and the review bot's marker file (RB_REVIEW_PAUSE_MARKER). Review round 1 (gpt-6.1-sol-high) fixes: verification runs inside the migration transaction, after a JSONL catch-up, so a failed check rolls the schema change back too; the operator gate requires quiescence, not pause flags (no live ledger lease, a non-empty drain-ack marker from the bot), and rebuild catches up records that raced the swap; the playbook hash comes from settings.playbooks_dir when that override is set. Review rounds 2-3 fixes: the migration lock is an advisory flock on index.migrate.lock — released by the kernel when the holder exits, so there is no stale state and no takeover path for a delayed contender to race (an unlink or rename-aside recovery let it remove a fresh holder's lock); the file is never unlinked; the regression test delays the contender until the holder is inside the critical section and counts concurrent holders. The fingerprint covers `resolved_models` (Settings.tier_target for both tiers) so a fallback change such as agent_model with ECO_MODEL empty changes it; harness paths are captured too — HarnessLLM.create records the one-shot, run_harness_step records the whole session as one model_call (final text, session usage, iterations, tools used), and the bridge spec carries the parent's store root and unit context so the MCP bridge subprocess binds both per call; model_call records keep the offered tool definitions (inputs.tools) and the structured conversation (inputs.messages, result.format=messages/1) instead of a single prompt string, and kb replay/export recognise that shape explicitly. Tests: test_improve_p0.py (schema/compat mode, gated migration with rollback, lock exclusivity and stale takeover, backfill + query equivalence, rebuild to schema 0 with a legacy positional insert, compare read-only, capture at both choke points, declarations, fingerprint determinism and diff, executor context binding, the CLI gate). Full suite green except test_mcp.py, which fails on main too (mcp 2.x installed on this host). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… clones with a pinned base
Meta-improvement engine P0, second slice (design §8.2). Everything is
off by default; a shadow (experiment) run turns it on through
IMPROVE_SHADOW=1 and PR_CONTEXT_SOURCE=snapshot, both set only by the
engine's shadow environment.
Layer 1, tool boundary. ToolScope gains read_roots (every read/exec path is
realpath-resolved and must lie under a root, so `..` and symlinks cannot
escape), deny_prefixes (the clone's .git/, whose alternates file names the
source object store), strict_extras and executables. `shadow_scope()` builds
the shadow scope. tools.dispatch enforces the read fence on builtin reads and
on extras that declare read_path_arg, refuses extras a strict scope does not
allow and any extra marked internal_write (skill_update_candidate is), and
tool_definitions_for filters those before the model sees them. Extras keep
their historical bypass under an ordinary scope.
Layer 2, step boundary. Under settings.improve_shadow the executor refuses
any step whose risk is not read/report (step_refused, run blocked, the
experiment invalid). The agent runtime hardens whatever scope a step asked
for through improve.shadow.harden_scope, using the workflow declaration's
shadow_tools; repo_map caches in the run dir instead of the knowledge tree.
Layer 3, credential boundary. improve.shadow.shadow_env builds an allowlist
environment (model credentials for the tiers, repo path pointing at the
shadow checkout, trace root; no GH_*/GITHUB_*/*_TOKEN/*_PAT) with PATH set to
a one-off directory of symlinks to the whitelisted executables (python3,
git, grep, never gh); assert_boundaries checks an environment before
dispatch; smoke exercises the read tools inside the checkout.
Layer 4, repository boundary. make_shadow_clone creates an independent
`git clone --shared --no-checkout` (a linked worktree would share the
source's remotes), removes its remote, publishes the pinned base tip as
refs/improve/base and refs/{remotes/origin,heads}/<base>, and checks out the
head. The review repo tools take an explicit base tip and compute
merge-base(base_tip, HEAD), the same semantics as pr.fetch_diff's pinned
diff, instead of guessing origin/main, which was also wrong for PRs against
a non-default branch; pr.fetch_diff now publishes pr_base_sha/pr_base_ref.
Input path. pr_context_source=snapshot makes pr.fetch_diff, pr.gate_check
and pr.fetch_ci_failures read the staged snapshot from state (never gh or
git fetch; a missing or mismatched snapshot blocks, and a pinned head that
differs is stale). improve.stage_items runs before isolation: resolve, fetch
pinned refs, pinned diff, context bundle with discussion excluded, gate
report and CI checks (gate_report_for / ci_checks_for are shared with the
live gate step), the shadow clone, a snapshot hash over the review inputs,
and an item_staged decision record with the snapshot as a blob.
Review round 1 (gpt-6.1-sol-high) fixes: harden_scope strips the ensemble
lens suffix (`agent.review_diff#contracts`) before the declaration lookup,
so ensemble lenses keep their declared tools; ci_checks_for treats parseable
list output as the data whatever gh's exit code (gh exits non-zero for
pending or failing checks while printing the JSON) and returns None only
when the checks are unavailable; staging refuses an item whose gate report
or CI checks cannot be captured instead of writing an empty snapshot.
Review round 2 fixes: RepoMap.build_index takes an `accept` fence evaluated
on the resolved path, the shadow repo_map passes the scope's read check so
a symlink inside the checkout cannot leak a host file's symbols (fenced maps
use their own cache key); a shadow run never touches skill metadata or the
usage journal (the post-step touch is skipped under improve_shadow).
Review round 3 fixes: the bridge spec carries read_roots, deny_prefixes,
strict_extras and executables both ways, so a harness's MCP bridge enforces
the same fences as the in-process loop (round-trip test dispatches through
the restored scope); the snapshot check validates the repository as well as
the PR number, since PR numbers are repository-local.
Tests: test_improve_p0b.py covers the read-fence escapes (absolute, `..`,
symlink, alternates) refused and the same calls unchanged under a plain
scope; strict extras filtered and refused without executing; the allowlist
environment and PATH (gh unreachable); the executor refusal; the clone
(no remote, base refs, source remotes/worktrees byte-identical, never
overwritten) and the base-bound tools on a PR whose target branch advanced
after the fork; smoke plus the knowledge tree hash unchanged after a
repo_map call; snapshot-mode steps with gh monkeypatched to fail; staging
end to end on a local repository. Full suite green except test_mcp.py
(mcp 2.x on this host, fails on main too).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…le (dry-run) Meta-improvement engine P1 (design §4, §5, §9). The cycle reads trace/1, never a model; it writes only its own ledger directory and its own trace records. Proposals stay in the ledger: nothing reaches GitHub until P4. improve/reader.py groups records into units of work by context.unit_id (records without one, the knowledge service's and the review bot's, fall back to run_id:step so nothing in the store is invisible) and reads a window by JSONL day file. improve/lints.py is the Tier 1 catalogue: 17 deterministic lints, each with a version and the campaign finding it comes from (L01 truncation, L02 budget exhaustion, L03 empty final after tools, L04 contract reject, L05 parse-failure rate vs baseline, L06 duplicate rendering, L07 silent fallback / served-model mismatch, L08 zero-yield pass, L09 cap drops, L10 unresolved anchors, L11 cost/latency robust-z outliers, L12 refusals and out-of-scope edits, L13 provider errors which quarantine the unit, L14 trace/1-native completeness, L15 repeated identical tool calls, L16 evidence-cap drops, L17 epistemic claims without a backing tool call). A lint whose evidence shape a producer does not emit yet finds nothing. The robust z floors the MAD at a tenth of the median so a uniform history does not flag every change as infinite. improve/ledger.py keeps one JSON ledger per workflow: per-cycle stats and lint rates, rolling baseline samples, the proposal state machine (open -> experiment-registered -> supported|neutral|refuted|underpowered -> landed|closed, 30 days untouched -> stale), one live proposal per lint or stage, holds, channel liveness; every change is also a decision record. improve/cycle.py runs a cycle: preflight (improve_enabled is the kill switch; holds are reported), lints and baselines per workflow, Tier 1 proposals when a lint's unit rate worsened 1.5x with at least 3 units, the stale sweep, a Markdown + JSON report and a decision(type=cycle) record, then the cursor advances. The engine's own records are units of the next cycle (self-enrolment). is_due/maybe_run_weekly implement the weekly slot (Monday 05:00 UTC by default); the kb serve scheduler calls it every tick under its lease, isolated like its other tick items. workflow_improve is a new read-only L2 task kind with the vetted workflow-improve playbook (improve.preflight -> improve.lint -> improve.ledger -> report.final_summary); the intent and chat enums list it. Settings: improve_enabled, improve_cycle_weekday/hour, improve_ledger_dir, improve_budget_usd_week and improve_budget_judge_calls_week (enforced in P3). CLI: improve cycle, improve ledger, improve lints. Review round 1 (gpt-6.1-sol-high) fixes: units are attributed whole to the cycle in which they END (records read from a two-day lookback before the window; a unit still in flight at the cutoff is deferred untouched), so a lint that needs both halves of a unit still fires; every ledger workflow is swept and reported each cycle, including ones with no unit in the window, so a workflow that stopped producing records still goes stale and reports a dead channel; a cross-process file lock (trace_store.file_lock, the same primitive as the migration lock) serializes whole cycles per ledger directory including the due check, and every ledger read-modify-write takes the workflow's lock with a per-process temp name. Review round 2 fixes: a unit counts only with completion evidence (a terminal decision after its last call, or an hour of silence before the cutoff) and only once — the cursor keeps the counted ids for the lookback span, so a unit still running at a cutoff is deferred and never counted twice; the scheduler, the CLI and the playbook share one default ledger directory (cycle.DEFAULT_LEDGER_DIR / ledger_dir_for); L11 prices a unit with the same settings the baseline used; the unresolved-anchor pattern matches `file.py:?` at a whitespace or text boundary; the CLI entry block sits after the helpers it calls; the reader defers an unterminated trailing JSONL line and skips a torn one instead of aborting the cycle. Review round 3 fixes: a unit that finished before the window counts only if it was still deferred (incomplete) at the window's start, so the lookback never pulls in earlier completed work (the first cycle reports seven days, an explicit --since window means what it says); L07 compares served and requested models through canonical_model with the configured aliases, exactly as the served-model guard does. Review round 4 fix: the cursor persists the ACTUAL deferred set and the engine's origin instead of re-deriving completion at the window start from timestamps (a record can become visible after the cutoff: a torn trailing line); a unit that finished before the window counts when it was deferred last time or when it became visible inside the engine's contiguous coverage (never counted before), and is historical otherwise; regression test with a torn decision line across two cycles. The coverage origin resets to the window start whenever continuity breaks (an explicit window elsewhere), so work skipped between windows never leaks into a later contiguous cycle. A cursor written before the deferred set existed recovers the old cycle's pending units from their own records at the old cutoff (a unit complete then was counted then and is never replayed), once, on upgrade. Tests: test_improve_p1.py — reader grouping and fallbacks, a clean unit has no findings, every lint fires on its fixture, baseline-dependent lints, completeness, quarantine, the ledger state machine and stale sweep, two cycles end to end (cursor, proposal opened on worsening, quarantined unit out of the baseline, cycle records), kill switch/holds/slot, the scheduler hook, kind/playbook/planner resolution, the playbook through the executor, the CLI. Full suite green except test_mcp.py (host mcp 2.x, fails on main). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…er 2 forensics Meta-improvement engine P2 (design §6, §11.2). Tier 2 is outcome-anchored: a workflow that declares an outcome adapter gets a deterministic coverage matrix and stage-of-loss attribution; a workflow without one keeps Tier 1 only, and nothing here makes a quality claim without an outcome record. improve/adapters: the contract keeps two data models apart — the gold matrix (match(): hit/miss/disputed/unlabeled per curated gold entry, the carrier of recall and of stage-of-loss) and finding-level validity (findings(): valid/invalid/unlabeled per produced finding, the carrier of precision) — and keeps a paired judge's review-level scores as they are (review_scores()). Every method reads outcome records already in the store, so matrices and scores recompute identically; scores_from derives the aggregates and stamps each with its source. improve/gold.py: the curated gold set the raw inline comments lack. gold_id = sha256(item + path + normalized concern)[:12]; `improve gold draft` writes a draft from the raw comments for a human to curate, only a file marked curated is ever used, a reworded concern is a new entry and the file's sha is the version an experiment freezes. improve/judges.py: run_judge — API judges through LLM.create, CLI judges (cursor/claude/codex) through governed_subprocess in a fresh empty workspace, tool-less by construction (a tool call in the stream discards the verdict), both recorded as model_call records, both gated by the P3 governor when one is given. improve/adapters/review_eval.py: the eval PR-review adapter — imports judge_val verdicts (roles + blinding resolved) as judge_verdict outcomes, runs the gold_match judge per gold entry (three votes, majority, a hit needs a verbatim quote found in the review) into gold_match outcomes, match() reads only those, findings are unlabeled (the judge scores whole reviews). improve/adapters/rb_review.py: the bot's production adapter — proxy labels from PR review threads (resolved/outdated/affirmative reply → accepted, push-back → disputed, else silent), descriptive-only, no gold, an unavailable thread is an "unavailable" record never a fabricated label. improve/forensics.py: coverage_matrix from match(); the fixed S0–S10 taxonomy; forensics agents that see one unit's trace through read-only tools (trace_get/trace_blob/trace_query, every result fenced as untrusted data, no builtin tools), must cite record ids of that unit (an uncited or unparseable answer is S0), run per miss cell by two model families with a disagreement recorded as disputed; punch_list ranks loss per (workflow, stage) with S10 routed to measurement health. improve/stats.py: paired item-clustered t intervals (ported from paired_analysis.py), items_required, the five labels, Cohen's kappa. improve/meta.py + eval/dataset/meta/: the frozen meta-benchmark (cases with human stage labels, lint samples), export/load, `improve meta lint-check`. improve.forensics step joins the playbook; settings improve_gt_dir/improve_judgments_dir/improve_eval_arm. Review round 1 (gpt-6.1-sol-high) fixes: the forensics step collects outcomes before the matrix (imports the paired judge's verdicts, runs gold_match with the judge from settings.improve_judge, reports what it collected and how many cells stay unlabeled); a lower-is-better metric is judged on the mirrored interval; CLI judge blindness inspects message content blocks and codex item types, not only event types; the codex judge uses --skip-git-repo-check, the workspace, stdin for the prompt and the nested item.text; trace_blob serves only blobs the unit's own records reference. Review round 2 fixes: the executor writes every unit's terminal `step_result` decision with the rendered artifact (review_text / answer_draft as blobs) and the published findings — the real review step publishes them through step outputs and never wrote a decision, so the adapter judged an empty body; gold_match now judges nothing when a unit has no rendered review (cells stay unlabeled); the forensics step fetches outcomes for every unit and derives finding-level scores independently of gold coverage, so the bot adapter's proxy labels are collected; every consumer of the declarations (executor, shadow hardening, cycle, forensics) goes through enroll.declarations_for(settings), which honours improve_workflows_dirs. Review round 3 fixes: a curated gold file applies only to its own item (repository and PR, and the head when it names one — PR numbers are repository-local); a bot finding is labelled on ITS OWN thread (bot-authored, carrying the finding's id or title, near its line) so a human's resolved thread at the same line never labels it; gold_match records a verdict only on a real majority of all three votes and leaves inconclusive cells unrecorded so the next run retries them instead of caching a coin flip. Round 4: id markers match whole (f-1 is not f-10), a lone nearby thread with a different identity is not the finding's, and head-specific gold requires that exact head; the title fallback matches the thread's own title exactly (its first line minus the id marker), never a thread whose title extends the requested one or carries another finding's marker, and rejects an ambiguous match; a finding without a title matches by its full comment against the thread body in the bot's publication format: the finding's own rendering (comment plus its evidence field, as published) must equal the thread's text minus marker and severity, so a comment that itself ends in an Evidence line is distinct from one whose evidence was appended; findings may say `file` or `path`. Tests: test_improve_p2.py — gold draft/load/id discipline, judge paths (governor reserve/settle, tool-using judge discarded, API path), paired stats/labels/power/kappa, the eval adapter end to end with a fake judge (idempotent imports, quote check, derived scores with sources), the bot adapter (classification, proxy labels, cached outcomes, unavailable threads), forensics (matrix, fenced tools, dispute, S0 on uncited claims, punch list, measurement health), meta export/load/lint samples, the forensics step's skips. Full suite green except test_mcp.py (host mcp 2.x). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Meta-improvement engine P3 (design §8, §10). improve/budget.py: two hard envelopes per ISO week — dollars and judge CLI calls — persisted under the ledger directory and serialized across processes. Every model call is RESERVED before it leaves the process and SETTLED after, so settled + reserved never exceeds the envelope, concurrent workers included; a reservation that does not fit raises BudgetRefused and the call is never sent. The reservation is a true upper bound: max_tokens for the output and the request's UTF-8 byte count for the input (every BPE-family token is at least one byte); a model with no price is refused, never priced at zero; a settlement above its reservation is recorded as a budget_breach incident, marks the governor breached and aborts (fail closed). LLM.create takes the reservation when a governor is bound (governed()/current_governor()), releases it on a failed request and settles on success; judges.run_judge already reserves judge calls. improve/experiments.py: registration writes the decision rule down before any arm runs (an experiment_registered record with the fingerprint diff and both manifests as blobs) and refuses what the protocol cannot adjudicate — a workflow without an adapter or with a descriptive-only one (the bot's), an arm that resolves to a harness backend (harness-not-isolated), identical fingerprints, a diff or override that touches the frozen meta-benchmark, an item without curated gold, a duplicate item set, a planning estimate above the remaining envelope. The item count required comes from the metric's historical item-level sd (the wave-4 prior when there is none); registering underpowered is allowed and labelled. Running stages every item before isolation, runs arm and incumbent per item and replicate in the shadow environment (the copilot as a subprocess with the snapshot handed over through PR_SNAPSHOT_FILE, which pr.fetch_diff now reads in snapshot mode), checks every produced unit's fingerprint against the registered one, quarantines provider errors, budget cuts, crashes and mismatches, scores each unit through the adapter, and adjudicates paired within item with power recomputed on the retained items: supported / refuted / neutral / underpowered / invalid, recorded as experiment_verdict and pushed to the proposal's state. improve.experiments step (registered experiments run before forensics, on their reserved budget) joins the playbook; the cycle and the forensics step run under the week's governor; forensics reports budget exhaustion instead of stopping the cycle. CLI: improve budget, improve experiment register|run|list. Review round 1 (gpt-6.1-sol-high) fixes: a shadow child binds its own governor from the environment (IMPROVE_GOVERNED=1 + IMPROVE_LEDGER_DIR, set by shadow_env and required by assert_boundaries) against the same weekly files, so its calls are metered; experiments configure the judge from settings.improve_judge (invalid, and said so, without one) and score the shadow reviews themselves — a blind paired judge call per (item, replicate) writing judge_verdict outcomes for both units, gold_match for the gold-level metric; registration takes a persisted NAMED hold that the run converts into per-call reservations, so two registrations cannot claim the same funds; settlement is cache-aware (reads at 0.1x, writes at 1.25x, as metrics prices them) and the reservation bounds prompt bytes at the cache-write rate; every reservation is pinned to its week (its id carries it) and settled or released there; a recorded breach stops every governor instance and process until an operator clears it; a failed subprocess (rc != 0) is excluded and every unit is identified by a unique per-run tag the executor stamps into the trace context, never by a time window. Review round 2 fixes: the shadow child's budget configuration is the supplied governor's (its ledger directory and envelopes, pinned through assert_boundaries' expected values) and no arm override may touch the engine's own IMPROVE_* configuration; LLM.create releases a reservation only when the failure happened before the request was sent — a failure after the provider answered settles the real usage, a failure after sending with unknown usage forfeits the whole reservation; the paired judge writes outcomes only for a whole, sane verdict (both candidates, finite scores in [0, 1], a valid winner), so an unusable reply excludes the pair instead of scoring it zero. Review round 3 fixes: arm overrides are parsed exactly as the shadow child parses them (as environment variables through the Settings class, validators and JSON included), never through model_copy, so the registered fingerprint is the child's; IMPROVE_* overrides are refused at registration; every terminal failure of a run (missing judge, boundary violation, an exception) releases the experiment's planning hold; the experiment's state is read inside its lock so a waiting caller never reruns a finished experiment; judge reservations are week-pinned tokens settled where they were taken. Tests: test_improve_p3.py — reservation/settlement/refusal (bytes bound, unpriced model, hard envelope, judge envelope, weekly persistence), breach handling, six concurrent reservations, the LLM.create hook (refused call never sent, release on failure, unmetered without a governor), every registration refusal and a successful registration's record, a supported experiment over eight items with a scripted unit runner, exclusions (staging failure, fingerprint mismatch, L13) recomputing power to underpowered, the snapshot-file hand-over, the experiments step, the CLI. Full suite green except test_mcp.py (host mcp 2.x). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tion and the meta-benchmark adapter
Meta-improvement engine P4 (design §9.2, §11.1, §11.2).
improve/publish.py: the engine holds no GitHub token, so a proposal leaves
it as an action file (open / update / close) in an outbox directory that
the maintainer routine's `propose` command picks up and answers with an
ack; the routine also observes each proposal issue (human comments,
`maintainer: hold`, referencing pull requests, closure) into an inbox the
engine reads back. Rule 4: evidence is cited only by record id and blob
hash; every excerpt is drawn verbatim from the redacted blob (<= 20 lines,
long lines cut to a prefix) so a reader can recover it from the hash, and
lint_proposal refuses anything else (unresolvable ids, a Tier 1 proposal
with fewer than three records, a Tier 2 proposal without a stage S1-S9 or
a loss, an excerpt that is not a line of its blob, a credential pattern
left in the body); a refused proposal is recorded and never published.
The issue body follows the fixed template (claim, stage or lint, loss,
evidence, suggested pre-registration with the register command, ledger
link, marker) and carries improve:<workflow> / improve:tier<n> /
improve:<state> labels. plan/publish write each action once (a pending
action waits for its ack for seven days), skip workflows on hold (rule 5:
lints and experiments continue) and record every write. sync applies
acks (issue url, published state; a dry-run ack from a routine still in
shadow publishes nothing) and observations: a human comment is a channel
touch, a hold comment sets a channel hold the next observation without it
releases, a merged referencing pull request lands the proposal, a closure
on GitHub closes it. A landed Tier 1 proposal's issue closes only once a
later cycle shows the lint's defect rate below the rate at opening.
Steps: improve.mode publishes the run mode and item key; improve.sync
(read) reads the outbox back; improve.publish is the cycle's only outward
write, declared risk="push" and gated like pr.post_review — without the
task's post intent it only previews, with it but ALLOW_POST=0 it is a dry
run, with both it writes the outbox (blocked loudly when no outbox is
configured); the executor's shadow refusal stops it before it runs. The
forensics step now opens one Tier 2 proposal per punch-list stage with a
loss (never S0/S10), each with a suggested pre-registration (metric, the
items with gold, the item count the historical sd requires, what the
arm's fingerprint should cover for that stage); descriptive-only
workflows get proxy proposals without a suggestion. Tier 1 proposals
record the rate at opening. Settings: improve_outbox_dir,
improve_proposal_repo, improve_meta_dir.
Self-enrolment (§11.2): improve/workflows/workflow-improve.yaml declares
the engine's own forensics role (`workflow-improve.improve.forensics`,
item `{improve_item}`) with a fingerprint over the forensics, lint,
statistics and adapter sources and the resolved models. With
`--task-param meta_case=<case>` the playbook runs only that step over one
frozen meta case (every cycle step is gated off) and writes a `meta_eval`
outcome: the engine's stage labels beside the human ones, agreement,
Cohen's kappa and the Tier 1 catalogue's recall on the injected lint
samples. adapters/meta_bench.py scores it: hit when the label agrees,
disputed when the families disagreed, unlabeled for S0; no judge is
involved (human_labelled), so the cross-family rule holds by
construction. experiments.run stages a self-experiment item by copying
the case and the lint samples into the shadow directory (nothing
narrative: no doc/, no judgments), runs the copilot child on it through
IMPROVE_META_DIR, needs no judge for a human-labelled adapter and picks
the unit enrolled in the experiment's workflow rather than the child's
last unit. CLI: improve publish [--dry-run], improve sync, improve meta
bench.
Records that forensics writes while attributing keep the caller's unit
item when one is bound (the investigated unit and item are named in the
result), so the engine's own unit over a meta case is keyed by the case.
Review round 1 (gpt-6.1-sol-high) fixes: the self-experiment child names
its repo (`--task-param repo=meta`, the one alias its REPO_PATHS exposes)
so run_playbook accepts it instead of falling back to default_repo, and
shadow_env now writes REPO_PATHS in the JSON form Settings parses (the
`alias=path` form failed every real shadow child at Settings load); a
`maintainer: hold` is remembered per proposal and the workflow's channel
hold is aggregated over its proposals, so observing an unheld proposal
never releases a hold another proposal's issue still carries (an
operator's own hold is never touched by the channel); the Tier 1 rate
check skips cycles whose units were all quarantined, so a cycle that
measured nothing cannot verify a landed proposal.
Review round 2 fixes: a private-key block is redacted whole (header to
END, or to the end of the text when cut), never just its header; a
publication (plan, action files, pending notes) runs under one lock per
outbox, so concurrent publishers cannot emit two actions for one proposal;
an ack is applied only when its action id is the proposal's pending action
— a delayed answer to a superseded action is consumed as stale and may
only add the issue url it created when none is known, never clear the
newer pending action or rewrite the published state.
Review round 3 fix: sync applies acks under the publisher's lock, so the
pending-action check and its clearing cannot interleave with a publisher
retrying that action (a stale ack could otherwise clear the newer pending
id and let a further publish duplicate it).
Tests: test_improve_p4.py — citation excerpts (verbatim, redacted,
recoverable, refused when tampered), the linter's refusals and the
template, the outbox round trip (once per action, acks, dry-run acks,
failures, touches, holds and their release, landing, the Tier 1 rate
check, refuted -> closed), the double-gated publish step and the shadow
refusal, Tier 2 proposals from the punch list, the meta case run and
adapter, a self-experiment adjudicated without a judge, the meta-mode
playbook run as the engine's own unit, the real CLI entry for a shadow
child (repo alias accepted, plan printed), a hold that survives another
proposal's silence, a fully quarantined cycle, a whole-key redaction, four
concurrent publishers, a stale ack after a retry, sync blocked behind a
publisher's lock, the CLI. Full suite green except
test_mcp.py (host mcp 2.x).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…an offline paired benchmark
The drafting step of knowledge intake (one merged upstream PR in, typed
knowledge operations out) becomes a declared workflow, `kb-intake.draft`:
static, one step call per event, item `{repo}#{pr}`, fingerprint over the
drafting code, the generator, the drafting strategy and zcode's reasoning
level (all as routing knobs read from the environment, then the same-named
setting where the tree has it, so the cover never goes missing). Its
outcome adapter scores a unit with the knowledge quality gate itself:
`gate_block` / `gate_summary` outcome records (the L1 result and the pinned
L2 judge's verdict per block, as the production gate computes them) give
finding-level validity (pass = valid, fail = invalid, unsure = unlabeled)
and the review-level numbers the engine pairs (`net_pass`, `gate_score`,
`precision`, `yield_pass`, `fail`, `human`, `empty`); the gold matrix is the
incumbent's gate-passing rules for the item — their COMPLETE sections as the
contracts, versioned by content — filled by a 3-vote `gold_match` judge whose
hits must quote the proposal verbatim; an empty draft misses
deterministically.
`eval/kb_distill/` is the repo-invariant benchmark that produces such units
from any knowledge-service state directory: `items.py` rebuilds each
recorded production draft's prompt from the base tree byte for byte (so the
recorded reply is a genuine sample of the current drafting code); `harness.py`
replays the incumbent, runs live arms (generator x strategy, N replicates;
a strategy the tree's drafting code does not implement is refused, never
mislabelled), judges every unit through the PRODUCTION gate
(`kb_service.gate.run_gate`: L1, upstream fact attestation against a bare
mirror with the service's read-only PR lookup, L2 per block with the
per-rule evidence expansion, the owner-directory consistency check,
protected rules and circuit breakers; a change set the gate fails lands
nothing, so its blocks count as failed), reports paired deltas with
`improve.stats`, and runs the engine's Tier 1 lints per arm;
`experiment.py` drives the engine's protocol end to end (gold, pre-registered
experiment, run with the benchmark as unit runner, reuse of judged units of
the same fingerprint, adjudication on the retained items).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
# Conflicts: # doc/architecture/SPEC/trace_store.md # src/infermatrix_copilot/trace_store.py
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrade the meta-improvement engine into a governed self-evolution loop for Copilot workflows and the engine itself. Includes the unmerged P0–P4 foundation and knowledge-intake adapter required to run the loop, followed by workflow readiness, scoped candidate generation, immutable source experiments, trusted parent evaluation, resumable scheduling, and deployment observation.
PR review, knowledge intake, and engine forensics have independent drivers; other workflows receive mechanical diagnostics and reproducer-based fixes without unverified quality claims. Code candidates require a filesystem/network sandbox and remain human-merged. Evolution is disabled by default, preserves the weekly $20 shared budget, and uses a versioned outbox consumed by the companion omni-maintainer change. Chinese usage documentation covers enrollment, snapshots, manual annotations, candidate inspection, and publication.
Validation: pre-integration suite 2,074 passed / 18 skipped; final scoped checks 113 passed / 1 skipped. Integrated current main with both trace schema migration and native call journals preserved. Documentation links, citations, strict SPEC coverage/freshness, both knowledge validators, and knowledge validator tests pass; both GitHub CI runs on head 757daf2 passed, including the complete integrated test suite, installed wheel/runtime-data smoke, CLI/doctor smoke, and repository documentation/knowledge checks. Live candidate execution and real-data end-to-end acceptance are deferred because this host has max_user_namespaces=0; simulated three-driver flows pass and there is no unsandboxed fallback.