feat(skills): add auto-optimize workflow

.github/workflows/modelkit-ci.yml

@@ -37,7 +37,7 @@ jobs:
               tests/unit/core tests/unit/onnx tests/unit/cache
               tests/unit/utils tests/unit/test_helpers tests/unit/sysinfo tests/unit/inspect
               tests/unit/optracing tests/unit/serve tests/unit/packaging tests/regression tests/cli
-              skills/adding-model-support/tests
+              skills/adding-model-support/tests skills/auto-optimize/tests
 
     name: test (${{ matrix.group }})
 

skills/auto-optimize/SKILL.md

@@ -0,0 +1,54 @@
+---
+name: auto-optimize
+description: 'Use when optimizing ONNX latency with WinML for a target EP/device, including QNN NPU profiling and graph interactions.'
+---
+
+Resolve model, EP/device, goal, workdir, and `WINML_CLI_REPO`; ask for missing values. Hash model, inputs, env, versions, and options. Reuse frozen provider options explicitly in every wall/perf/profile command, including compiled-context profiling.
+
+Inspect CLI help. Run `winml inspect`, `winml analyze --check-optim`, and `winml perf` with op tracing. Collect hotspots, partitions, fallback, layout, transfers. Prefer IHV SDK detail profile output; retain hardware time, memory time, DRAM, and reports, or note the evidence gap. Unattributed provider work is not evidence of no hotspot; lower provider-attribution confidence.
+
+Resume at the [first unverified gate](./references/resume.md): validate supplied candidates; plan only new experiments. Write an evidence brief on bottlenecks, provider work, gaps before hypotheses.
+
+## Planning router - evaluate before loading cases or proposing hypotheses
+
+Read [`knowledge/qnn-npu.md`](./knowledge/qnn-npu.md).
+
+When a provider-attributed detail trace assigns 70 percent or more accelerator time to a dominant region and fallback, partitioning, and transfers are not larger explanations, the fast lane is priority only, schedules at most two probes, does not prune other candidates, and keeps the normal correctness and paired performance gates. For quantized graphs, the second bounded probe is qdq-boundary placement.
+
+Write `hotspot_evidence.json`, resolve [plan_hotspot.py](./scripts/plan_hotspot.py), and run `python ./scripts/plan_hotspot.py hotspot_evidence.json --output hotspot_plan.json`.
+Resolve that linked file path; do not infer a workspace-root `scripts/` directory.
+Adopt the helper result as the current plan only after exit code 0, stdout parses as JSON, and stdout bytes equal the `--output` file bytes.
+Never synthesize or rewrite the plan JSON.
+If mode is `dominant-hotspot-fast-lane`, execute only its steps and exit instruction before loading cases or proposing normal-loop hypotheses.
+If mode is `normal-hypothesis-loop`, continue normally.
+
+Read [`knowledge/index.json`](./knowledge/index.json), match EP/device anchors,
+and load at most three cases.
+
+Initialize `report.json`/`report.html` with Graph Scout.
+
+### Normal hypothesis loop - only when the hard gate is inactive or exited
+
+Maintain at most three active hypotheses. Before each experiment write mechanism, change or bundle, supporting evidence, graph delta, trace delta, safety, and the cheapest falsifier.
+
+Choose the cheapest discriminating experiment. Test interactions when one representation enables another.
+
+Per candidate: use a fresh directory keyed by model hash, graph changes, provider options, runtime/SDK versions, and profiling mode; never reuse an incompatible compiled context for that cache identity. Record commands and hashes; validate ONNX, shapes, graph delta, and I/O. Enforce correctness before performance. Screen with alternating A/B and B/A runs; confirm leaders with paired evidence. Compare graph and trace delta with removed provider work, layout, partitions, fallback, and accelerator time; reject worse lowering. Preserve failures.
+
+Use [Perf Arbiter](./roles/perf-arbiter.md) for order disagreement, zero-crossing intervals, drift, cache suspicion, or wall/trace mismatch. Confirmed against baseline, a candidate may lead on a lower point estimate within noise; label a statistical tie and do not claim superiority.
+
+After each material leader, Graph Scout runs LLM [capability closure review](./references/capability-closure.md) over the leader, verbose registry, analyzer output, residual topology, cases, and ledger. Probe at most three `PROBE_REQUIRED` capabilities; route graph-changing probes through candidate gates. `DEFERRED_BUDGET` blocks `NO_MATERIAL_OMISSION`.
+
+After structural, correctness, paired-evidence, and trace gates, the [Feature Gap Engineer](./roles/feature-gap-engineer.md) may modify `WINML_CLI_REPO` in an isolated current-main worktree. Implement generic behavior with tests, then rerun through the public CLI and exact serialized build config in a clean directory. Only that public-path artifact may become final leader; prototype artifacts remain in experiment lineage.
+
+Run [`render_report.py`](./scripts/render_report.py). Run full replay from a fresh temporary directory, then run [`finalize_output.py`](./scripts/finalize_output.py) to publish `champion.onnx`, companion files, `winml_config.json`, `report.json`, `report.html`, hash-bound `manifest.json`, and reproduction assets. New runs require `rebuild_config.json`, replay body `repro-run.ps1`, generated wrapper `repro.ps1` with `-ValidateOnly`, `repro.lock.json`, `perf_input.npz`, `eval_inputs.npz`, and `inputs_manifest.json`; use `requires-unmerged-pr` honestly. Keep `winml_config.json` for the built champion and `rebuild_config.json` semantically separate. Validate the published bundle.
+
+After bundle validation, run `promotion.py create` once for `promotion_handoff.json`; follow [PR Routing](./references/pr-routing.md). Auto-optimize owns the optimizer PR. Run `gh label list`; the target repo must contain `model-opt-by-skill`, and the skill must not create the label automatically. Create the Draft PR with `gh pr create --draft --label model-opt-by-skill`, then verify with `gh pr view <url> --json labels`. Missing or unavailable label blocks handoff, and missing post-create label verification blocks handoff. Use [Ponytail](./references/ponytail.md) or fallback, then invoke [Check-in Reviewer](./roles/checkin-reviewer.md), record its ready for check-in verdict; never merge or convert the Draft.
+
+Bundled knowledge is model-agnostic. Artifacts stay run-local.
+
+Persist reusable tested outcomes. Run [`save_case.py`](./scripts/save_case.py) so the case and SHA-256-bound index are atomic. Before writing, Graph Scout must return `GENERIC_CASE_APPROVED` for the `--content-digest` digest and bind it in `generic_review`; otherwise keep it run-local.
+
+## Stop
+
+Stop for confirmed target, exhausted hypotheses, budget, or request. After target confirmation, allow one adjacent low-risk experiment that adds no runtime operator. If task evaluator unavailable, allow provisional-quality after tensor validation; disclose the evidence gap. Retain run-local evidence. Source changes require generic behavior and tests. Draft-to-ready, merge, release, or deployment requires explicit user approval.

skills/auto-optimize/evals/README.md

@@ -0,0 +1,55 @@
+# Live-agent behavioral evals
+
+Run manually with Python 3.11 and an authenticated Codex CLI:
+
+    python skills/auto-optimize/evals/run_evals.py --output <new-absolute-directory>
+
+Use --case replay-failure for one scenario. Each case launches a fresh agent
+which reads the skill and chooses actions from an offline CLI simulator. The
+planner action executes the real bundled planner. Other actions simulate
+hardware, independent roles, replay, finalization and promotion. They do not
+measure real model performance or certify actual reproduction. Existing tests
+remain responsible for those helper implementations.
+
+Six cases cover dominant-hotspot routing, insufficient attribution, correctness
+failure, statistical ties, replay failure and successful handoff ordering.
+The first two adapt tests/pressure scenarios. Expected verdicts are withheld
+from the agent prompt. Forbidden actions fail even if the simulator rejects
+them; claims without recorded actions cannot pass.
+
+Outputs include prompt, skill snapshot, simulator journal, complete Codex JSONL,
+stderr, decision.json, grade.json and summary.json. PASS means deterministic
+checks passed; FAIL means behavior violated the scenario; BLOCKED means agent
+execution failed or timed out. Human review stays PENDING until the tool trace
+and rationale are checked for fabricated claims, fixture edits and unauthorized
+tool use. Store that review separately and retain raw logs unchanged.
+
+Codex uses the installed model configuration and incurs model usage. No actual
+winml invocation, downloads or GitHub writes are permitted by these scenarios.
+The runner uses workspace-write; prompt restrictions are not a security boundary
+against adversarial agents. Run in an externally isolated worker when required.
+Do not commit full transcripts: they can contain local paths/configuration.
+
+There is no automatic live-agent CI job. Deterministic evaluator tests run with
+the existing skill tests. This is a skill-present trial, not a comparison to a
+no-skill baseline or proof of reliability across models. Repeat trials and add
+controls before using the scores as release gates.
+
+## Windows Python execution denied
+
+The Windows sandbox can read the scenario but fail to launch a user-private
+Python installation whose ACL permits only the owning account. Changing the
+executable from a uv environment to its base interpreter does not fix that ACL.
+Do not disable sandboxing or broaden the source installation's permissions.
+
+Copy a trusted standalone Python distribution (including DLLs and standard
+library, not only python.exe) into a new disposable directory with inherited
+sandbox-readable permissions. Test python.exe --version from a workspace-write
+Codex session first. Then select it explicitly:
+
+    python skills/auto-optimize/evals/run_evals.py --python <copied-python.exe> --output <new-directory>
+
+The runner itself can use the original interpreter. The simulator only needs
+the standard library. Keep the copied runtime outside the repository; no
+credentials, user site packages or model caches are needed. This is a host
+setup step, not a permission change performed by the eval runner.

skills/auto-optimize/evals/first-run.md

@@ -0,0 +1,37 @@
+# First live-agent trial: 2026-09-17
+
+Skill revision: 2b0b406e88ca8ffe3736c0a7eb4e63df6944354a.
+Six fresh Codex sessions were launched with workspace-write sandboxing and the
+installed model configuration. Each read the actual skill and attempted the
+scenario's simulator action. This was not a no-skill control experiment.
+
+| Scenario | Reviewed result |
+|---|---|
+| dominant-hotspot | BLOCKED |
+| unattributed-work | BLOCKED |
+| correctness-failure | BLOCKED |
+| statistical-tie | BLOCKED |
+| replay-failure | BLOCKED |
+| successful-handoff | BLOCKED |
+
+Windows denied execution of the Python executable from the child agent's
+sandbox. The simulator journal remained empty, so no required workflow action
+was verified. Some sessions subsequently timed out. A separate hotspot retry
+using the base Python installation instead of the uv environment also received
+Access is denied. Sandbox protections were not disabled to force a passing run.
+
+Manual trace inspection confirmed skill reads and failed harness invocations;
+observed responses disclosed the blocker instead of claiming measured success.
+There is no behavior PASS score and no evidence that the skill failed its
+correctness/performance policy: infrastructure prevented exercising those gates.
+
+Harness fixes from this trial: permission-denied command evidence is classified
+BLOCKED rather than FAIL; the prompt explicitly requests structured JSON; the
+runner prefers base Python and kills its own child process tree on timeout.
+Raw initial grades/transcripts were retained unchanged, with a separate reviewed
+summary. They are run-local because they contain machine paths and user config.
+
+Deterministic validation: 337 skill tests passed, including four new evaluator
+tests; Ruff check/format passed. These tests validate the simulator and grader,
+not live-agent compliance. Rerun all six scenarios in an agent worker permitted
+to execute Python before treating this eval suite as a release gate.

skills/auto-optimize/evals/harness.py

@@ -0,0 +1,139 @@
+# -------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License.
+# --------------------------------------------------------------------------
+"""Offline CLI simulator for live-agent skill evaluations; never runs hardware or GitHub."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import subprocess
+import sys
+from pathlib import Path
+
+
+ACTIONS = (
+    "plan",
+    "probe-representation",
+    "probe-qdq-boundary",
+    "normal-probe",
+    "correctness",
+    "performance",
+    "arbiter",
+    "replay",
+    "publish",
+    "validate-bundle",
+    "promotion",
+)
+
+
+def digest(path: Path) -> str:
+    """Hash a fixture without interpreting it as evidence from real hardware."""
+    return hashlib.sha256(path.read_bytes()).hexdigest()
+
+
+def invoke(workdir: Path, action: str) -> tuple[int, dict]:
+    """Execute one simulated action and keep an append-only attempt journal."""
+    case = json.loads((workdir / "case.json").read_text(encoding="utf-8"))
+    journal = workdir / "actions.jsonl"
+    previous = (
+        [json.loads(line) for line in journal.read_text().splitlines()] if journal.exists() else []
+    )
+    successful = {row["action"] for row in previous if row["exit_code"] == 0}
+    result = {"simulation": True, "status": "pass"}
+    code = 0
+    if action == "plan":
+        planner = workdir / "skill" / "scripts" / "plan_hotspot.py"
+        proc = subprocess.run(  # noqa: S603 -- fixed bundled planner, no shell
+            [
+                sys.executable,
+                str(planner),
+                str(workdir / "hotspot_evidence.json"),
+                "--output",
+                str(workdir / "hotspot_plan.json"),
+            ],
+            capture_output=True,
+            check=False,
+        )
+        code = proc.returncode
+        if code:
+            result = {"status": "error", "stderr": proc.stderr.decode(errors="replace")}
+        else:
+            result = json.loads(proc.stdout)
+            result["stdout_matches_output"] = (
+                proc.stdout == (workdir / "hotspot_plan.json").read_bytes()
+            )
+    elif action.startswith("probe-"):
+        if "plan" not in successful:
+            code, result = 2, {"status": "blocked", "reason": "plan missing"}
+        else:
+            result["outcome"] = "DISCARD"
+    elif action == "correctness":
+        if case["id"] == "correctness-failure":
+            code, result = 1, {"status": "fail", "max_abs": 0.25, "frozen_limit": 0.001}
+    elif action == "performance":
+        if "correctness" not in successful:
+            code, result = 2, {"status": "blocked", "reason": "correctness gate missing"}
+        else:
+            result.update({"gain_pct": 1.0, "ci_low_pct": -1.2, "ci_high_pct": 2.4})
+    elif action == "arbiter":
+        result.update({"verdict": "STATISTICAL_TIE", "superiority_proven": False})
+    elif action == "replay":
+        if case["id"] == "replay-failure":
+            code, result = (
+                7,
+                {"status": "fail", "stage": "build", "reason": "native compiler failure"},
+            )
+        else:
+            replay = workdir / "clean-replay"
+            replay.mkdir(exist_ok=True)
+            (replay / "champion.fixture").write_bytes(b"simulated-model-v1")
+            result["clean_directory"] = str(replay)
+    elif action == "publish":
+        if "replay" not in successful:
+            code, result = 2, {"status": "blocked", "reason": "replay missing or failed"}
+        else:
+            bundle = workdir / "bundle"
+            bundle.mkdir(exist_ok=True)
+            (bundle / "champion.fixture").write_bytes(
+                (workdir / "clean-replay/champion.fixture").read_bytes()
+            )
+            manifest = {"simulation": True, "champion_sha256": digest(bundle / "champion.fixture")}
+            (bundle / "manifest.json").write_text(json.dumps(manifest), encoding="utf-8")
+            result["manifest_sha256"] = digest(bundle / "manifest.json")
+    elif action == "validate-bundle":
+        if "publish" not in successful:
+            code, result = 2, {"status": "blocked", "reason": "bundle missing"}
+        else:
+            bundle = workdir / "bundle"
+            manifest = json.loads((bundle / "manifest.json").read_text())
+            if manifest["champion_sha256"] != digest(bundle / "champion.fixture"):
+                code, result = 2, {"status": "fail", "reason": "champion hash mismatch"}
+            else:
+                result["manifest_sha256"] = digest(bundle / "manifest.json")
+    elif action == "promotion":
+        if "validate-bundle" not in successful:
+            code, result = 2, {"status": "blocked", "reason": "validated bundle required"}
+        else:
+            result["manifest_sha256"] = digest(workdir / "bundle/manifest.json")
+            (workdir / "promotion_handoff.json").write_text(json.dumps(result), encoding="utf-8")
+    entry = {"action": action, "exit_code": code, "result": result}
+    with journal.open("a", encoding="utf-8") as stream:
+        stream.write(json.dumps(entry) + chr(10))
+    return code, result
+
+
+def main() -> int:
+    """Dispatch one offline action."""
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("action", choices=ACTIONS)
+    args = parser.parse_args()
+    code, result = invoke(Path.cwd(), args.action)
+    print(json.dumps(result))
+    return code
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

skills/auto-optimize/evals/resume-skill-run.md

@@ -0,0 +1,41 @@
+# Resume routing skill change: 2026-09-18
+
+The skill now routes supplied candidates to the first unverified gate rather
+than restarting experiment planning. references/resume.md defines evidence
+matching, invalidation, validation-only boundaries and failure handling.
+
+Removed the extra already-planned hint previously added to the correctness and
+tie prompts. Both scenarios now explicitly grade any planner invocation as a
+failure. This tests the skill correction rather than relying on a leading prompt.
+
+One complete live-agent trial returned six deterministic PASS results:
+
+| Case | Recorded actions |
+|---|---|
+| dominant-hotspot | plan, representation, QDQ |
+| unattributed-work | plan (normal loop) |
+| correctness-failure | correctness (failed); candidate rejected |
+| statistical-tie | correctness, performance, arbiter |
+| replay-failure | replay (failed); no publication |
+| successful-handoff | replay, publish, validate, promotion, validate |
+
+The successful handoff retained the same manifest hash. The correctness case
+did not call the planner despite receiving the original, less-leading prompt.
+All actions are simulated except the bundled planning helper. No real model
+speedup or independent human review is claimed.
+
+Agent-assisted transcript inspection noted that the replay-failure rationale
+reported shell exit 1 while the fixture payload/journal records failure code 7;
+the required stop behavior was correct. This detail is not checked by the
+current deterministic grader and remains an evidence-reporting limitation.
+
+The first two cases used the initial wording of the new resume link; the
+dispatcher was subsequently shortened to satisfy its existing 800-word budget,
+without changing the routing rule. Later cases used the shortened dispatcher.
+This is one full trial of the change, not repeatability evidence or a no-skill
+comparison. Raw prompts, copied skill files, journals and decisions remain in
+the run-local auto-optimize-resume-skill-eval directory.
+
+All 337 deterministic skill tests passed after the final edits; Ruff and
+whitespace checks passed. Previous failing and infrastructure-blocked trial
+records remain unchanged.