AlgoCoach-Flywheel

Verifier-backed 算法教练评测与后训练系统：练习运行时、provenance 数据飞轮、PromptGym-Code 模拟器、SFT/DPO 评测连成一条闭环。

A verifier-backed evaluation and post-training system for algorithmic coaching.

AlgoCoach 把本地练习运行时、可执行的 Python 判题、带 provenance 的数据治理、 PromptGym-Code 模拟和 SFT/DPO 评测接成一条链路。本仓库是公开的策展投影： 克隆后即可跑 CPU smoke、打开 Web demo、查看评测证据。

AlgoCoach connects a local practice runtime, executable Python verification, provenance-aware data governance, PromptGym-Code simulation, and SFT/DPO evaluation.

3 分钟跑通验证 · 5–8 分钟演示路径 · 查看 440 例 protocol split · 查看 provenance / leakage audit

_{真实 Web practice runtime：执行 buggy Python 提交，返回逐用例验证、错误类型与结构化反馈；无 API key 时自动使用 rules fallback。}

Evidence at a glance / 证据速览

Evidence class	Public receipt	Engineering takeaway
Runnable corpus	42 runner-ready problems	local practice + executable feedback
440-case freeze	15 problems / 440 served cases	frozen protocol-split comparison surface
Provenance / leakage	SFT 68/68 · DPO 50/50 provenance rows · served leakage 0/2160	training and evaluation lineage remains auditable
SFT model-selection decision	base 0.382 → SFT 0.568 (+18.6 pp)	keep the SFT baseline; route DPO gain/regression slices into regression mining

Sources: 440-case protocol split and system integrity metrics.

What the system demonstrates

practice runtime
  -> runner / verifier
  -> provenance data flywheel
  -> simulator / protocol-split eval
  -> SFT / DPO evidence
  -> user-facing coaching

• Execution-grounded coaching: feedback can be checked against runnable code.
• Data and split governance: examples retain provenance instead of becoming an opaque training pile.
• Evaluation discipline: protocol splits and confidence intervals constrain post-training claims.
• Reproducible public surface: the CPU smoke and Web demo work without a private model checkpoint.

3-minute proof path

python -m pip install -r requirements.txt
python scripts/run_public_smoke.py
python -m unittest discover -s tests -p 'test_public_smoke.py' -q

The smoke compares an oracle upper-control with generic rule controls on a 20-case mini packet and writes an inspectable report. For the interactive path:

python scripts/server.py doctor
python -m uvicorn app.main:app --host 127.0.0.1 --port 8000

Then follow docs/DEMO_PATH.md for two problems. An API key is optional; without one, the coach falls back to rules and the runnable verification path remains available.

Public projection contents

• app/ — practice runtime, runner, verifier, and simulator packages
• data/problems_public/ — public problem bank for the Web UI
• data/public_smoke/ + reports/public_smoke/ — CPU smoke packet and receipts
• data/simulator/eval_cases.jsonl — official 440-case freeze
• reports/eval/*.md — curated evaluation summaries
• reports/verifier/ — executable verifier artifacts used by the demo path
• docs/DEMO_PATH.md — interview-friendly live walkthrough

Excluded by design: private job materials, paper drafts, training corpora, exports/checkpoints, local secrets, scratch artifacts, and bulk eval sidecars.

Additional evaluation evidence

Evidence class	Public receipt	Engineering takeaway
DPO − SFT	−0.005, 95% CI [−0.027, +0.018]	fixed-rate uplift is not stable; stop DPO promotion and keep the SFT baseline
Construct validity	N=30 · agreement 0.700 · τ 0.520	the proxy complements human judgment
Executed repair	14/20 fixed	execution-grounded repair audit; small-N only

Sources: SFT/DPO uncertainty, construct validity, and executed repair pilot.

Claim boundary

• rule_targeted is an oracle upper-control, not a deployable coach.
• DPO has weak atom/reward signals here; it does not support fixed-rate uplift.
• modeled_heldout_proxy is a parser/audit proxy; executed repair is a separate small-N harness (14/20 on the 2026-07-04 clean rerun).
• Construct-validity supports complementarity with human judgment, not replacement.
• Real-user traces are calibration/sanity only; no population learning claim.
• Paper status: under review at a COLM 2026 workshop.

Reproduce checklist

python scripts/run_public_smoke.py
python -m unittest discover -s tests -p 'test_public_smoke.py' -q
python scripts/server.py doctor