Repository navigation
Add zcode GLM-5.3 as a kb judge backend with an evolvable judge tuning surface - #298
Conversation
…g surface - KB_JUDGE_FAMILY_WAIVER: explicit operator opt-in letting the judge share a model family with a generator (same-backend draft+grade deployments); logged at startup, other invariants unchanged - kb_service/judge_tuning.py: the judge-side evolution surface (both system prompts, verdict aggregation, neighbour budget); gate.py (protected) now delegates to it; initial values byte-identical to the pre-extraction baseline so evolution starts from production behaviour - kb-intake.draft declaration: KB_JUDGE/KB_JUDGE_FAMILY_WAIVER become fingerprint routing covers (units judged by different judges are no longer fingerprint-identical), judge_tuning.py joins the prompt hash and the evolution paths, ZCODE_REASONING_LEVEL becomes a declared setting - tests: waiver semantics, byte-identity, aggregation ordering, delegation hook, zcode judge round-trip through the gateway
IMPROVE_JUDGE=cli:zcode:GLM-5.3 runs judge calls through ZCodeTransport.complete — the same tool-less one-shot the kb gate's ModelGateway uses (empty scratch cwd, every native tool removed, containment audit, served-model assertion, ambient ZCODE_REASONING_LEVEL) — so the experiment-internal gate judge is the production instrument itself, not an API proxy of it. Subscription billing keeps usd=0; calls reserve/settle against the judge-call envelope like any CLI judge and record as model_call with the served model. The trusted kb-intake driver's Gateway picks this up unchanged (judge_spec_from now accepts the provider; unknown providers still refuse). Judge-tuning baseline test now pins recorded digests instead of git HEAD (which no longer carries the prompts).
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7426d79d72
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| paths: [src/infermatrix_copilot/kb_service/intake.py, | ||
| src/infermatrix_copilot/kb_service/judge_tuning.py] |
There was a problem hiding this comment.
Allowing candidates to modify judge_tuning.py changes the prompts, context budget, or verdict aggregation used by the production gate, but calibration_current() still identifies a calibration only by judge label and case-set digest (runtime.py:468-469). After such a candidate is deployed, the previous calibration therefore remains current and auto_merge can immediately use an uncalibrated gate; include a digest/version of this tuning surface in calibration records and currentness checks.
Useful? React with 👍 / 👎.
| if waived: | ||
| logger.warning( | ||
| "KB_JUDGE_FAMILY_WAIVER is set: judge %s grades drafts from the same " | ||
| "model family as %s; keep the calibration set current", | ||
| judge.label(), candidate.label()) | ||
| break |
There was a problem hiding this comment.
When a zcode judge was previously calibrated with a non-zcode generator, enabling a zcode fallback plus this waiver leaves rt.judge.label() and the case-set digest unchanged, so calibration_current() accepts the old record and auto-merge proceeds without the required waiver-era recalibration. Record the waiver/generator-family configuration in the calibration identity or explicitly invalidate existing records when this path is enabled.
Useful? React with 👍 / 👎.
| usage=usage, seconds=round(time.monotonic() - started, 3), | ||
| result={"format": "prompt/1", "usd": 0.0, "subscription": True}, error=error) |
There was a problem hiding this comment.
When ZCODE_PROVIDER_ID selects a custom API provider, ZCodeTransport.subscription_billing is false and calls can incur metered charges, yet this path only reserves the judge-call count and unconditionally records usd: 0.0 and subscription: true. Those calls bypass the engine's weekly dollar envelope and can spend without limit; inspect the transport's billing mode and either reject non-subscription providers or reserve and settle their actual USD cost.
Useful? React with 👍 / 👎.
| settings: [KB_DRAFT_MAX_OPERATIONS] | ||
| paths: [src/infermatrix_copilot/kb_service/intake.py, | ||
| src/infermatrix_copilot/kb_service/judge_tuning.py] | ||
| settings: [KB_DRAFT_MAX_OPERATIONS, ZCODE_REASONING_LEVEL] |
There was a problem hiding this comment.
For eligible kb-intake evolution runs the generator targets are required to be API-backed, while public_settings() does not pass zcode_reasoning_level to the worker and the shared zcode evaluation judge is constructed from the host Settings(), not the per-arm override. Consequently a ZCODE_REASONING_LEVEL candidate changes its fingerprint but neither arm's execution, so random paired differences can promote an untested no-op setting; either wire the override into the arm-specific model call or remove it from the mutation policy.
Useful? React with 👍 / 👎.
Summary
Two commits making
zcode:GLM-5.3a first-class judge backend beside GPT-6 Sol, and giving the self-evolve engine a judge-side tuning surface. All defaults unchanged — nothing activates without explicit env.96e06c3— judge backend + evolvable surfaceKB_JUDGE_FAMILY_WAIVER: explicit operator opt-in letting the judge share a model family with a generator (same-backend draft+grade, e.g. GLM drafting + GLM grading); logged at startup; every other invariant unchanged. Without the env the cross-family refusal is exactly as before.kb_service/judge_tuning.py: the judge-side evolution surface (both system prompts incl. dimension rubric wording, answer→verdict aggregation, neighbour context budget).gate.py(PROTECTED) only delegates; initial values are byte-identical to the pre-extraction baseline (pinned by sha256 in tests) so evolution starts from production behaviour.kb-intake.draftdeclaration:KB_JUDGE/KB_JUDGE_FAMILY_WAIVERjoin the fingerprint routing covers (units judged by different judges are no longer fingerprint-identical);judge_tuning.pyjoins the prompt hash +evolution.paths;ZCODE_REASONING_LEVELbecomes a declared evolution setting (KB_JUDGEitself stays inFORBIDDEN_SETTINGS— candidates can never swap the judge).:effortsuffix, reasoning level viaZCODE_REASONING_LEVEL, no spend-cap enforcement, timeout→humanfail-closed).7426d79—cli:zcodeas an engine judge route (gold mode)IMPROVE_JUDGE=cli:zcode:GLM-5.3runs judge calls through the productionZCodeTransport.completeone-shot — the same tool-less instrument the kb gate uses (empty scratch cwd, all native tools removed, containment audit, served-model assertion). The trusted kb-intake driver picks it up unchanged. Subscription billing keepsusd=0; calls reserve/settle against the judge-call envelope. Unknown CLI judge providers still refuse.goldevaluation mode and calibration work.Evidence (full report:
.kb-judge-glm53/REPORT.mdon the ops box)humanescalation as rejection, so it is currently unpassable by any judge. Not a zcode-specific defect.Testing
test_kb_intake_gate.py,test_kb_generator_fallback.py,test_improve_kb_intake_adapter.py,test_improve_p2.py(+ new waiver/round-trip/delegation/baseline-digest tests),test_evolution.py,test_autonomous_evolution.py, kb-init suites — all green (162 passed, 1 sandbox skip on hosts without user namespaces).