diff --git a/doc/architecture/SPEC/kb_service.md b/doc/architecture/SPEC/kb_service.md index 7f3893b3..8afa8c16 100644 --- a/doc/architecture/SPEC/kb_service.md +++ b/doc/architecture/SPEC/kb_service.md @@ -817,12 +817,35 @@ infermatrix-copilot kb init REPO --stage feature-discovery --from-existing \ ``` `--dry-run` 保存目录和 PR 正文供审查；省略它时沿用 init 的发布权限与 PR 流程。 +已有知识库没有本地 skeleton 记录时，目录 PR 合并后可运行 +`kb init REPO --stage modules --from-existing --pin FULL_SHA`，从已合并的仓库 +入口、owner 路由及页面复核结构，再补充导航。随后使用 +`--stage knowledge --from-existing`；已有 modules 记录仍须完成并合并， +已有 skeleton 记录的状态与合并门禁同样保留。该模式不生成历史阶段记录， +不绕过发现目录的源码及哈希绑定；省略 `--from-existing` 时保留原阶段顺序。 +重复初始化若遇到已合并旧批次仍保留的发布分支，可在新批次启动前设置 +`KB_INIT_BRANCH_SUFFIX=`，使用 `kb/init-REPO-STAGE-SUFFIX`；未设置时 +沿用旧分支名及检查点身份。后缀仅允许 1–40 位小写字母、数字和连字符， +首尾不能为连字符；非法值在模型调用前拒绝。配置的后缀绑定输入身份及 +已准备发布的分支，恢复发布时不能更换；不会删除或覆盖旧批次分支。 订阅模式明确使用 Zcode GLM‑5.3 提取、Codex 独立评审，不使用 Claude Code。 默认并发上限为 13，两轮按顺序运行，轮内共享线程池；Zcode 请求沿用共享排期。 `KB_DISCOVERY_GENERATOR`、`KB_DISCOVERY_JUDGE`、`KB_DISCOVERY_CONCURRENCY` 覆盖发现阶段配置；提取与评审必须属于不同模型家族。`--budget-usd` 与 `--unlimited-subscription` 互斥。未报告实际费用时保持未知，记账值不当作发票费用。 +`KB_DISCOVERY_PACKET_CHARS` 设置每次提取组合的正文字符预算（整数 24000–192000， +默认 24000）；仅组合已有完整分片，不删除或截断输入。`KB_DISCOVERY_START_INTERVAL_S` +设置 Zcode 初始启动间隔（有限数值 1–60 秒，默认 15 秒）；原生 429/1302 仍触发共享 +冷却和自适应限速。较大的包可能仍有候选遗漏，不能据此声称所有功能已发现。 +`ZCODE_REASONING_LEVEL=low|high|max` 设置实际写入 Zcode 会话配置的推理强度，默认 max； +Zcode 模型角色中的 `effort` 被 transport 忽略，不能代替该设置。网关追踪和原生 attempt +均记录 `native_reasoning_level`；发现的输入身份、检查点及紧凑报告 `run_config` 同时绑定 +包大小、启动间隔和实际配置的推理强度。改变任何一项需新批次；旧检查点未记录这些 +配置时不能安全恢复，须保留旧档案并使用新的状态目录。 +返回恰好 24 个候选的分片记录 `candidate_limit_reached`，报告列出数量及任务 ID， +作为候选可能遗漏的线索；本轮不自动拆分重采样，也不把饱和结果称为穷尽。 + 发现使用完整文件库存和分片，不把前 N 个文件、前 N 行当作全仓库。范围、排除规则、 文档入口与功能分母按仓库配置；未知语言使用文本和入口线索，并保留未解析关系。 没有文档不阻断源码轮，正式功能允许 `docs: []`。目录以外的疑似第一方代码列为 diff --git a/doc/architecture/SPEC/playbooks/PLAYBOOKS.md b/doc/architecture/SPEC/playbooks/PLAYBOOKS.md index ff8ec5d5..32ed305a 100644 --- a/doc/architecture/SPEC/playbooks/PLAYBOOKS.md +++ b/doc/architecture/SPEC/playbooks/PLAYBOOKS.md @@ -63,6 +63,8 @@ success, steps[]`。 pr-history → harvest-calibration），每阶段一个由人合并的 PR。只由 `kb init REPO --stage S` 运行，不经 `kb serve`，不写服务账本。 `feature-discovery` 先提取文档功能基线，再遍历固定源码与测试库存；独立评审后的目录 PR 合并后冻结，下游核对目录哈希与源码版本。新初始化模板启用发现；旧仓库保持兼容。 +`from_existing` 可用于发现、modules 与解释性知识：缺少本地 skeleton 记录时 +核验已合并入口与 owner 页面；已有阶段记录及发现目录仍保留完成和合并门禁。 默认 GLM‑5.3 提取、Codex 独立评审，共享并发 13；支持 `from_existing`、 `retry_unfinished`、`budget_usd` 或 `unlimited_subscription`。未知项不增加正式功能计数， 原生追踪归档在 Git 外。详细契约见 [`../kb_service.md`](../kb_service.md)。 diff --git a/doc/architecture/SPEC/trace_store.md b/doc/architecture/SPEC/trace_store.md index 0767b14c..363385d2 100644 --- a/doc/architecture/SPEC/trace_store.md +++ b/doc/architecture/SPEC/trace_store.md @@ -1,6 +1,6 @@ # trace_store.py —— 规范 - + `LOC ~500 · trace/1：Copilot 与 RB 共用的模型调用/工具调用/判定/结果记录 · refactor-status: stable` @@ -18,6 +18,8 @@ `native-events.jsonl`，保留原生会话、输出、错误与调度等待事件。结束时绑定唯一的 `model_call` 回执及输出、用量、耗时与状态；journal 不是另一笔调用，不能重复计费。 中断未完成的 journal 保留输入和已收到的事件，未报告用量及费用保持未知。 + 对 Zcode，网关调用的 model 与 attempt.model 额外记录 `native_reasoning_level`， + 表示实际写入会话配置的 low/high/max；role.effort 不是该 transport 的有效推理设置。 - `bind_store(store)` / `current_store()`：把一个 store 绑定为当前进程/任务的采集汇（`tools.dispatch` 与 `LLM.create` 两个 choke point 据此写 `tool_call` / `model_call`，失败调用记 `error` 且无 `outputs`）； diff --git a/eval/feature-discovery/README.md b/eval/feature-discovery/README.md index 27d9dad2..0be5490a 100644 --- a/eval/feature-discovery/README.md +++ b/eval/feature-discovery/README.md @@ -24,6 +24,14 @@ infermatrix-copilot kb init REPO --stage feature-discovery --from-existing \ `KB_DISCOVERY_GENERATOR`、`KB_DISCOVERY_JUDGE`、`KB_DISCOVERY_CONCURRENCY` 可覆盖。 `--budget-usd` 与不限订阅模式互斥。未报告实际费用时保持未知。 +批量提取可设置 `KB_DISCOVERY_PACKET_CHARS`（24000–192000，默认 24000），完整索引 +分片和读取范围保持不变。`KB_DISCOVERY_START_INTERVAL_S`（1–60 秒，默认 15）只配置 +启动间隔，仍保留 429/1302 冷却。`ZCODE_REASONING_LEVEL=low|high|max` 控制实际原生 +推理配置；模型角色的 effort 后缀不会替代它。三项设置记录在检查点和报告的 +`run_config`；原生追踪另存 `native_reasoning_level`。配置变化或旧检查点缺少这些身份 +字段时使用新状态目录，保留旧轨迹。扩大包不会证明已发现所有功能，也不取消独立评审。 +恰好达到每包 24 个候选上限的任务另列数量和 ID，保留为潜在遗漏线索。 + 检查点绑定源码、范围、种子目录、索引版本、提示词与模型配置。成功分片、评审和待审修正稿可复用；预算中断不标为完成。竞争启动同一批次不会覆盖检查点。单项最多三次修正；通道故障保持未知，不重做已成功的提取。仅有文档的未知声明不通过重复改写变成实现证明。 紧凑摘要写入 `eval/feature-discovery/`。完整输入、原生输出、工具事件、配置、用量和修正过程留在 Git 外。恢复时核验每个完成分片的原生归档，包括返回空候选的分片。Zcode 保存原生流式事件；Codex 保存 CLI 结束后按原顺序转发的事件。 diff --git a/playbooks/kb-init.yaml b/playbooks/kb-init.yaml index 4ab6c77f..5579e268 100644 --- a/playbooks/kb-init.yaml +++ b/playbooks/kb-init.yaml @@ -18,7 +18,7 @@ params: pin: "upstream commit to pin (default: the default branch head)" pr_count: "pr-history only: merged upstream PRs to replay (default: adapter pr_history_count or 1000)" budget_usd: "discovery/pr-history/depth: cumulative spend ceiling; increasing it resumes the same pinned batch" - from_existing: "discovery/explanatory knowledge: enrich a merged index and owner routes without local earlier-stage records" + from_existing: "discovery/modules/explanatory knowledge: enrich a merged index and owner routes without local skeleton records; existing stage merge gates remain required" retry_unfinished: "feature-discovery/knowledge-deepen: retry unfinished work while retaining accepted work and cumulative spend" subscription_generator: "explicit subscription backend; generator USD unreported, fees outside stage USD accounting" unlimited_subscription: "feature-discovery/knowledge-deepen: authenticated Zcode GLM-5.3 generator and independent Codex subscription judge without fixed accounting ceiling; incompatible with budget_usd" diff --git a/plugins/infermatrix-copilot/skills/imkbinit/SKILL.md b/plugins/infermatrix-copilot/skills/imkbinit/SKILL.md index 0c482946..1f2a586e 100644 --- a/plugins/infermatrix-copilot/skills/imkbinit/SKILL.md +++ b/plugins/infermatrix-copilot/skills/imkbinit/SKILL.md @@ -87,10 +87,15 @@ The next stage is the first one that isn't done. If an earlier stage's PR is open, stop and say it's waiting for the owner's review. If a record says `blocked`, report its `problems` and stop. -For an explicit knowledge rerun of an already merged KB with no local -skeleton/modules records, use `--stage knowledge --from-existing`. The CLI +For an explicit breadth rerun of an already merged KB with no local skeleton +record, use `--stage modules --from-existing`. Merge its PR before running +`--stage knowledge --from-existing`; no skeleton record is manufactured. +For a knowledge-only rerun with no local skeleton/modules records, use +`--stage knowledge --from-existing`. The CLI checks the merged index and every owner route. Existing stage records still keep their review and merge gates; never manufacture records to skip them. +An enabled or started feature-discovery batch must still finish and its +catalog PR must merge before publishing modules or knowledge. For implementation knowledge in an existing KB, use the independent stage `knowledge-deepen` (`kb deepen ` is its `--from-existing` alias). @@ -98,6 +103,12 @@ It is distinct from the ordinary `deepen` stage that generates hot-module rules. `kb widen ` similarly selects `knowledge --from-existing`. Use a new state directory when the knowledge baseline or source pin changes; retain prior native records and accepted blocks as audit evidence. +If a previous merged batch's publication branch still exists, set +`KB_INIT_BRANCH_SUFFIX=` before starting the new batch. It appends +the slug to the default stage branch without replacing old branches. Use +1–40 lowercase letters, digits or hyphens, with no leading or trailing hyphen. +The suffix is part of the checkpoint identity; retain it when resuming or +publishing that batch, including a publication waiting for PR confirmation. ## 3. Run it @@ -143,6 +154,15 @@ What each stage opens: Zcode GLM-5.3 extracts and independent Codex reviews with shared concurrency 13. `KB_DISCOVERY_GENERATOR`, `KB_DISCOVERY_JUDGE` and `KB_DISCOVERY_CONCURRENCY` override the stage configuration; discovery rejects same-family review. + `KB_DISCOVERY_PACKET_CHARS` groups complete indexed chunks (24000–192000, + default 24000); `KB_DISCOVERY_START_INTERVAL_S` sets the initial Zcode spacing + (1–60 seconds, default 15) while native rate-limit cooldown stays enabled. + `ZCODE_REASONING_LEVEL=low|high|max` controls the effective native reasoning; + a Zcode role's effort suffix does not. These settings are frozen in the batch + and report, with reasoning also recorded in both trace and attempt archives. + Changed settings or old checkpoints without them require a fresh state + directory; retain the previous archive. Larger packets do not prove all + features were found. `--from-existing` checks merged skeleton routes, while `--retry-unfinished` resumes supported work. `--budget-usd` controls cumulative accounting; `--unlimited-subscription` authenticates both roles and removes that ceiling. diff --git a/src/infermatrix_copilot/kb_service/cli.py b/src/infermatrix_copilot/kb_service/cli.py index 6334c0d7..7edaa200 100644 --- a/src/infermatrix_copilot/kb_service/cli.py +++ b/src/infermatrix_copilot/kb_service/cli.py @@ -146,8 +146,8 @@ def _init_command(args, state_dir: Path) -> int: return 2 params["acceptance_mode"] = args.acceptance_mode if args.from_existing: - if args.stage not in ("feature-discovery", "knowledge", "knowledge-deepen"): - print("--from-existing is for feature-discovery, knowledge or knowledge-deepen only", file=sys.stderr) + if args.stage not in ("feature-discovery", "modules", "knowledge", "knowledge-deepen"): + print("--from-existing is for feature-discovery, modules, knowledge or knowledge-deepen only", file=sys.stderr) return 2 params["from_existing"] = "true" if args.subscription_generator: @@ -227,7 +227,7 @@ def main(argv: list[str] | None = None) -> int: init.add_argument("--pin", help="upstream commit to pin (default: the default branch head)") init.add_argument("--suggest-seeds", action="store_true") init.add_argument("--from-existing", action="store_true", - help="enrich a merged KB without local skeleton/modules records (discovery and knowledge stages)") + help="enrich a merged KB without local skeleton records (discovery, modules and knowledge stages)") init.add_argument("--subscription-generator", action="store_true", help="explicit subscription generator; unreported fees outside stage USD accounting") init.add_argument("--unlimited-subscription", action="store_true", diff --git a/src/infermatrix_copilot/kb_service/init_feature_discovery.py b/src/infermatrix_copilot/kb_service/init_feature_discovery.py index 10e1afe5..74fa5ac9 100644 --- a/src/infermatrix_copilot/kb_service/init_feature_discovery.py +++ b/src/infermatrix_copilot/kb_service/init_feature_discovery.py @@ -12,6 +12,8 @@ from dataclasses import dataclass, replace import hashlib import json +import math +import os from pathlib import Path import re import threading @@ -26,12 +28,44 @@ VERSION = "feature-discovery-v1" MAX_PACKET_CHARS = 24000 +MAX_CONFIGURED_PACKET_CHARS = 192000 +DEFAULT_START_INTERVAL_S = 15.0 MAX_CANDIDATES = 24 MAX_ATTEMPTS = 4 MAX_BASELINE_CHARS = 16000 RELATIONS = {"new", "implementation_supplement", "alias", "subcapability", "shared_component", "outdated", "unknown"} _SLUG = re.compile(r"[a-z0-9][a-z0-9-]{0,39}\Z") + + +def discovery_run_config(rt, environ=None): + """Freeze bounded scheduling and effective native reasoning before dispatch. + + Increasing packets groups the same complete index chunks; it never narrows + scope. Old checkpoints without this configuration need a new batch because + their effective native reasoning setting was not recorded. + """ + env = os.environ if environ is None else environ + raw_packet = env.get("KB_DISCOVERY_PACKET_CHARS", str(MAX_PACKET_CHARS)) + try: + if not re.fullmatch(r"[0-9]+", str(raw_packet)): + raise ValueError + packet_chars = int(raw_packet) + if not MAX_PACKET_CHARS <= packet_chars <= MAX_CONFIGURED_PACKET_CHARS: + raise ValueError + except (ValueError, TypeError): + raise InitError(f"KB_DISCOVERY_PACKET_CHARS must be an integer from {MAX_PACKET_CHARS} to {MAX_CONFIGURED_PACKET_CHARS}") from None + try: + start_interval = float(env.get("KB_DISCOVERY_START_INTERVAL_S", DEFAULT_START_INTERVAL_S)) + if not math.isfinite(start_interval) or not 1.0 <= start_interval <= 60.0: + raise ValueError + except (ValueError, TypeError): + raise InitError("KB_DISCOVERY_START_INTERVAL_S must be finite and between 1 and 60 seconds") from None + level = getattr(rt.gateway, "zcode_reasoning_level", "max") if rt.generator.provider == "zcode" else None + if level is not None and level not in ("low", "high", "max"): + raise InitError("effective Zcode reasoning level must be low, high or max") + return {"packet_chars": packet_chars, "zcode_start_interval_s": start_interval, + "zcode_reasoning_level": level} SYSTEM_DISCOVER = """Discover distinct software capabilities from the supplied pinned evidence. First documentation establishes a baseline; source/test packets expand it. Read actual bodies, not just headings, filenames or scan lists. A capability @@ -195,10 +229,16 @@ class DiscoveryEngine: Only the coordinator mutates checkpoint state. Workers return immutable results and use a shared generator/judge concurrency ceiling. """ - def __init__(self, index, *, seeds, owners, state, call, save, concurrency=13, repository="", pin=""): + def __init__(self, index, *, seeds, owners, state, call, save, concurrency=13, repository="", pin="", + packet_chars=None): + packet_chars = MAX_PACKET_CHARS if packet_chars is None else packet_chars + if isinstance(packet_chars, bool) or not isinstance(packet_chars, int) \ + or not MAX_PACKET_CHARS <= packet_chars <= MAX_CONFIGURED_PACKET_CHARS: + raise ValueError("discovery packet size outside supported bounds") self.index, self.seeds, self.owners, self.state = index, seeds, owners, state self.call, self.save, self.concurrency = call, save, concurrency self.repository, self.pin = repository, pin + self.packet_chars = packet_chars self.state.setdefault("tasks", {}) self.state.setdefault("candidates", {}) self.state.setdefault("reviews", {}) @@ -218,7 +258,7 @@ def _packets(self, kind): packet, used = [], 0 for chunk in chunks: size = len(chunk.get("text", "")) + 150 - if packet and used + size > MAX_PACKET_CHARS: + if packet and used + size > self.packet_chars: yield packet packet, used = [], 0 packet.append(chunk) @@ -253,7 +293,8 @@ def _extract(self, key, packet, baseline): row["generator_receipts"] = [proof] row["origin_rounds"] = [payload["round"]] checked.append(row) - return {"status": "complete", "attempts": attempt + 1, "candidates": checked, "invalid_candidates": invalid, "receipt": proof} + return {"status": "complete", "attempts": attempt + 1, "candidates": checked, "invalid_candidates": invalid, "receipt": proof, + "candidate_limit_reached": len(reply.data["candidates"]) == MAX_CANDIDATES} except BudgetExhausted: raise except ModelUnavailable as exc: @@ -534,6 +575,11 @@ def _cache_reusable(self, previous): def _mode_identity(self): return False + def _resume_input_problems(self, previous, digest): + return [] if previous.inputs_digest == digest else [ + "discovery inputs changed; restore the original configuration to resume this prepared publication, " + "or preserve its checkpoint and use a new state directory"] + def _init_identity(self): return repr(replace(self.lifecycle.init, budget_usd=0.0)) + ( ":subscription-generator" if self.rt.subscription_generator else "") + ( @@ -553,8 +599,10 @@ def _base_for_run(self, latest): def _input_options(self): policy = self.rt.knowledge.show(self._base_sha, self._coverage_policy_path()) or "" + self.discovery_run_config = discovery_run_config(self.rt, self.rt.environ) return {"discovery_version": VERSION, "prompt_sha256": _hash([SYSTEM_DISCOVER, SYSTEM_REVIEW]), "catalog_seed_sha256": hashlib.sha256(policy.encode()).hexdigest(), - "discovery_concurrency": getattr(self.rt, "discovery_concurrency", 13)} + "discovery_concurrency": getattr(self.rt, "discovery_concurrency", 13), + "discovery_run_config": self.discovery_run_config} def _restore_progress(self, previous): if previous and previous.discovery: @@ -602,6 +650,11 @@ def _verify_archived_receipt(self, proof, expected_role, text=None): if model.get("model") != expected_role.model or model.get("provider") != expected_role.provider \ or model.get("effort", "") != expected_role.effort: raise ValueError("archive model differs from requested discovery role") + if expected_role.provider == "zcode": + configured_level = getattr(self.rt.gateway, "zcode_reasoning_level", "max") + if model.get("native_reasoning_level") != configured_level \ + or attempt.get("model", {}).get("native_reasoning_level") != configured_level: + raise ValueError("archive effective Zcode reasoning level differs or is unrecorded") served = model.get("served_model") if expected_role.provider == "zcode" and expected_role.model.casefold() == "glm-5.3" and served \ and re.sub(r"[^a-z0-9]", "", served.casefold()) != "glm53": @@ -690,10 +743,12 @@ def _build_catalog(self, tree): seeds = raw["features"] state = self.record.discovery identity = inputs_digest(version=VERSION, prompts=_hash([SYSTEM_DISCOVER, SYSTEM_REVIEW]), index=index.sha256, seeds=seeds, - generator=self.rt.generator.label(), judge=self.rt.judge.label()) + generator=self.rt.generator.label(), judge=self.rt.judge.label(), + run_config=self.discovery_run_config) if state.get("identity") and state["identity"] != identity: return self._blocked(["discovery index or catalog identity changed; create a new batch"]) state["identity"] = identity + state["run_config"] = self.discovery_run_config if self.retry_unfinished: for task in state.get("tasks", {}).values(): if task.get("status") == "unknown": @@ -709,7 +764,8 @@ def _build_catalog(self, tree): budget = _ConcurrentBudget(self.budget, Path(self.rt.state_dir) / "init" / self.lifecycle.repo / "discovery-budget.json", identity) if hasattr(self.rt.gateway, "configure_zcode_pacing") and self.rt.generator.provider == "zcode": from .depth_pacing import SharedZcodePacer - pacer = SharedZcodePacer(Path(self.rt.state_dir) / "init" / "discovery-zcode-pacing.json") + pacer = SharedZcodePacer(Path(self.rt.state_dir) / "init" / "discovery-zcode-pacing.json", + start_interval=self.discovery_run_config["zcode_start_interval_s"]) pacer.prepare() self.rt.gateway.configure_zcode_pacing(pacer) @@ -732,7 +788,8 @@ def save(): engine = DiscoveryEngine(index, seeds=seeds, owners=[o.owner for o in self.owners], state=state, call=call, save=save, concurrency=getattr(self.rt, "discovery_concurrency", 13), - repository=self.lifecycle.full_name, pin=self.record.pin) + repository=self.lifecycle.full_name, pin=self.record.pin, + packet_chars=self.discovery_run_config["packet_chars"]) scanned = engine.scan() self._validate_saved_archives(state) save() @@ -785,8 +842,10 @@ def save(): "implementation_supplements": counts.get("implementation_supplement", 0), "aliases": counts.get("alias", 0), "outdated": counts.get("outdated", 0), "unknown": counts.get("unknown", 0), + "candidate_limit_tasks": sum(t.get("candidate_limit_reached", False) for t in state["tasks"].values()), "invalid_candidates": sum(len(t.get("invalid_candidates", [])) for t in state["tasks"].values())} report = {"schema_version": 1, "repo": self.lifecycle.repo, "pin": self.record.pin, + "run_config": self.discovery_run_config, "complete": True, "done": True, "catalog_sha256": catalog_sha, "feature_ids": [f["id"] for f in features], "features": [{"id": f["id"], "owner": f["owner"], "title": f["title"]} for f in features], @@ -798,6 +857,7 @@ def save(): "tests": len(index.tests), "tasks": len(state["tasks"]), "unknown_tasks": sum(t["status"] == "unknown" for t in state["tasks"].values())}, "failures": index.failures, "scope_suggestions": index.scope_suggestions, + "candidate_limit_task_ids": [key for key, task in sorted(state["tasks"].items()) if task.get("candidate_limit_reached")], "invalid_candidates": [{"task": key, **item} for key, task in sorted(state["tasks"].items()) for item in task.get("invalid_candidates", [])], "index_sha256": index.sha256, diff --git a/src/infermatrix_copilot/kb_service/init_history.py b/src/infermatrix_copilot/kb_service/init_history.py index 3508421e..7e145702 100644 --- a/src/infermatrix_copilot/kb_service/init_history.py +++ b/src/infermatrix_copilot/kb_service/init_history.py @@ -423,7 +423,7 @@ def _publish(self, changed: dict[str, str]) -> InitRecord: return record prepared = save_prepared( InitRecord.path(rt.state_dir, record.repo, self.STAGE).with_name("pr-history-publish.json"), - base_sha=record.kb_base_sha, branch=f"kb/init-{record.repo}-pr-history", files=changed, + base_sha=record.kb_base_sha, branch=self._publication_branch(), files=changed, commits=commits, draft=True, title=title, body=self._body(), author=author, when=record.started_at) record.pr = {"prepared": str(prepared)} record.status = "publishing" diff --git a/src/infermatrix_copilot/kb_service/init_stages.py b/src/infermatrix_copilot/kb_service/init_stages.py index d482777b..f8e14e49 100644 --- a/src/infermatrix_copilot/kb_service/init_stages.py +++ b/src/infermatrix_copilot/kb_service/init_stages.py @@ -59,6 +59,8 @@ MAX_SEED_PAGES = 8 MAX_EXCERPT_BYTES = 8 * 1024 MAX_PROMPT_DOC_BYTES = 120_000 +BRANCH_SUFFIX_ENV = "KB_INIT_BRANCH_SUFFIX" +_BRANCH_SUFFIX = re.compile(r"[a-z0-9](?:[a-z0-9-]{0,38}[a-z0-9])?") _SLUG = re.compile(r"[a-z0-9][a-z0-9-]{0,40}") _ID_PREFIX = re.compile(r"^([A-Z][A-Z0-9]{0,7})-") _TOP_HEADING = re.compile(r"(?m)^#{1,6}\s") @@ -133,6 +135,15 @@ def validate_change(base: Mapping[str, str], head: Mapping[str, str], *, observe # -- stage entry ------------------------------------------------------------------- +def _init_branch_suffix(rt: InitRuntime) -> str: + """Validate a new batch's optional publication name before any model call.""" + suffix = rt.environ.get(BRANCH_SUFFIX_ENV, "") + if not isinstance(suffix, str) or (suffix and not _BRANCH_SUFFIX.fullmatch(suffix)): + raise InitError(f"{BRANCH_SUFFIX_ENV} must be an optional lowercase slug of 1–40 characters " + "using letters, digits and hyphens, with no leading or trailing hyphen") + return suffix + + def run_stage(rt: InitRuntime, lifecycle, stage: str, *, dry_run: bool, pin: str | None = None, pr_count: int | None = None, budget_usd: float | None = None, from_existing: bool = False, subscription_generator: bool = False, @@ -143,8 +154,9 @@ def run_stage(rt: InitRuntime, lifecycle, stage: str, *, dry_run: bool, pin: str record (also saved under ``/init//.json``).""" if stage not in STAGES + INDEPENDENT_STAGES: raise InitError(f"unknown stage {stage!r}; one of {STAGES + INDEPENDENT_STAGES}") - if from_existing and stage not in ("feature-discovery", "knowledge", "knowledge-deepen"): - raise InitError("--from-existing is for feature-discovery or explanatory knowledge stages only") + _init_branch_suffix(rt) # publication configuration is refused before any model call + if from_existing and stage not in ("feature-discovery", "modules", "knowledge", "knowledge-deepen"): + raise InitError("--from-existing is for feature-discovery, modules or explanatory knowledge stages only") if retry_unfinished and stage not in ("feature-discovery", "knowledge-deepen"): raise InitError("--retry-unfinished is for feature-discovery or knowledge-deepen only") if acceptance_mode not in ("strict", "lightweight") or ( @@ -461,6 +473,7 @@ class _Stage: def run(self) -> InitRecord: rt, lc, stage = self.rt, self.lifecycle, self.STAGE init = lc.init + self._branch_suffix = _init_branch_suffix(rt) self.repo_dir = lc.knowledge_dir base_sha = self._base_for_run(rt.knowledge.fetch()) self._base_sha = base_sha @@ -476,9 +489,14 @@ def run(self) -> InitRecord: pin = upstream.resolve(self.pin or chain.pin or "HEAD") self._discovery_gate(chain, pin) self.overlay = dict(chain.repo_files) + input_options = dict(self._input_options()) + if self._branch_suffix: + # Leave legacy checkpoint identities unchanged when the option is + # unset, but bind an explicit batch name before extracting anything. + input_options["publication_branch_suffix"] = self._branch_suffix digest = inputs_digest(stage=stage, repo=lc.repo, pin=pin, kb=base_sha, init=self._init_identity(), generator=rt.generator.label(), judge=rt.judge.label(), dry_run=self._mode_identity(), - chain=chain.key, **self._input_options()) + chain=chain.key, **input_options) record_path = InitRecord.path(rt.state_dir, lc.repo, stage) previous = InitRecord.load(rt.state_dir, lc.repo, stage) if previous is not None and previous.pr.get("prepared") and previous.status in ("publishing", "blocked"): @@ -490,6 +508,9 @@ def run(self) -> InitRecord: self.record = previous return self._blocked(chain.problems) problems = self._resume_input_problems(previous, digest) + if load_prepared(previous.pr["prepared"]).get("branch") != self._publication_branch(): + problems.append("prepared publication belongs to a different branch suffix; " + f"restore {BRANCH_SUFFIX_ENV} to its original value to resume") if self._frozen_discovery and previous.discovery.get("catalog_binding") != self._discovery_binding(): problems.append("prepared publication belongs to a different discovery catalog; preserve it and start a new batch") if problems: @@ -546,8 +567,18 @@ def _chain(self) -> _Chain: chain = _Chain() parts: list[str] = [] publisher = None + existing_skeleton = self.from_existing and self.STAGE in ("modules", "knowledge", "knowledge-deepen") \ + and InitRecord.load(self.rt.state_dir, self.lifecycle.repo, "skeleton") is None + if existing_skeleton: + # An explicit rerun may start from a merged owner map. Only the + # absent skeleton record is replaced by this verification; any + # actual modules record keeps its own completion and merge gates. + chain = self._existing_skeleton_chain() + parts.append(chain.key) for stage in STAGES[:STAGES.index(self._chain_boundary())]: record = InitRecord.load(self.rt.state_dir, self.lifecycle.repo, stage) + if stage == "skeleton" and existing_skeleton: + continue if stage == "feature-discovery": # Centralized after pin resolution so --from-existing subclasses # cannot bypass this gate, and the explicit pin must match. @@ -717,7 +748,7 @@ def _discovery_binding(self) -> dict: "report_sha256": self._frozen_discovery_report_sha256} def _existing_skeleton_chain(self) -> _Chain: - """Discovery can enrich a merged skeleton without inventing records.""" + """Explicit enrichment can use a merged skeleton without inventing records.""" if InitRecord.load(self.rt.state_dir, self.lifecycle.repo, "skeleton") is not None: return _Stage._chain(self) base = self.rt.knowledge.knowledge_files(self._base_sha) @@ -730,10 +761,10 @@ def _existing_skeleton_chain(self) -> _Chain: return _Chain(problems=[f"--from-existing invalid skeleton routes: {exc}"]) problems = [] if not base.get(f"{self.repo_dir}/{INDEX_NAME}") or route_source == "none" or not owners: - problems.append("--from-existing discovery needs a merged repository index and owner routes") + problems.append("--from-existing needs a merged repository index and owner routes") for owner in owners: if not owner.path.startswith(self.repo_dir + "/") or owner.path not in base: - problems.append(f"--from-existing discovery owner page missing or outside repository: {owner.path}") + problems.append(f"--from-existing owner page missing or outside repository: {owner.path}") return _Chain(key=f"existing-discovery:{self._base_sha}", problems=problems) def _discovery_gate(self, chain: _Chain, pin: str) -> None: @@ -1297,6 +1328,10 @@ def _d5_prose(self, text: str) -> str: return neutral_headings("\n".join(kept).strip()) # -- output ---------------------------------------------------------------------- + def _publication_branch(self) -> str: + suffix = getattr(self, "_branch_suffix", "") + return f"kb/init-{self.lifecycle.repo}-{self.STAGE}" + (f"-{suffix}" if suffix else "") + def _publish(self, changed: dict[str, str]) -> InitRecord: rt, lc, record = self.rt, self.lifecycle, self.record stage = self.STAGE @@ -1314,7 +1349,7 @@ def _publish(self, changed: dict[str, str]) -> InitRecord: else: prepared = save_prepared( InitRecord.path(rt.state_dir, lc.repo, stage).with_name(f"{stage}-publish.json"), - base_sha=record.kb_base_sha, branch=f"kb/init-{lc.repo}-{stage}", files=changed, + base_sha=record.kb_base_sha, branch=self._publication_branch(), files=changed, title=title, body=body, author=self.author, when=record.started_at) record.status = "publishing" record.pr = {"prepared": str(prepared)} diff --git a/src/infermatrix_copilot/kb_service/models.py b/src/infermatrix_copilot/kb_service/models.py index 806fd4e2..0d52e94a 100644 --- a/src/infermatrix_copilot/kb_service/models.py +++ b/src/infermatrix_copilot/kb_service/models.py @@ -130,6 +130,11 @@ def configure_zcode_pacing(self, pacer): """Only the explicit native campaign configures shared dispatch pacing.""" self._zcode_pacer = pacer + @property + def zcode_reasoning_level(self): + """Effective native setting; the Zcode role's effort is not this value.""" + return str(getattr(self._settings, "zcode_reasoning_level", "") or "max") + def _transport(self, provider: str): if self._factory is not None: return self._factory(provider) @@ -184,6 +189,8 @@ def _call_json(self, role: ModelRole, *, system: str, prompt: str, started = time.time() identity = {"role": role.name, "requested": role.label(), "provider": role.provider, "model": role.model, "effort": role.effort, "fallback_from": fallback_from} + if role.provider == "zcode": + identity["native_reasoning_level"] = self.zcode_reasoning_level payload = {"system": system if record_payload else "", "prompt": prompt if record_payload else ""} begin = getattr(self._recorder, "begin_call", None) archive = begin({**identity, **payload}) if callable(begin) else None diff --git a/src/infermatrix_copilot/kb_service/runtime.py b/src/infermatrix_copilot/kb_service/runtime.py index 90b14f44..31b73eac 100644 --- a/src/infermatrix_copilot/kb_service/runtime.py +++ b/src/infermatrix_copilot/kb_service/runtime.py @@ -127,7 +127,8 @@ def record(entry: dict) -> dict | None: inputs={"system": entry["system"], "prompt": entry["prompt"]}, outputs={"reply": entry["reply"], **({"native_events": entry["native_events"]} if "native_events" in entry else {})}, - model={k: entry.get(k, "") for k in ("role", "provider", "model", "effort", "served_model")}, + model={**{k: entry.get(k, "") for k in ("role", "provider", "model", "effort", "served_model")}, + **({"native_reasoning_level": entry["native_reasoning_level"]} if "native_reasoning_level" in entry else {})}, usage=entry.get("usage") or {}, seconds=entry.get("seconds"), # spend: the requested stop threshold and the reported cost # (None when unset/unknown), on failures too diff --git a/src/infermatrix_copilot/trace_store.py b/src/infermatrix_copilot/trace_store.py index 50337264..bfcbf953 100644 --- a/src/infermatrix_copilot/trace_store.py +++ b/src/infermatrix_copilot/trace_store.py @@ -712,7 +712,8 @@ def __init__(self, store: TraceStore, entry: Mapping[str, Any]): self.session_ids: set[str] = set() self.meta = {"schema": "native-attempt/1", "id": self.id, "status": "inflight", "at": store._clock(), "context": current_context(), - "model": {k: entry.get(k, "") for k in ("role", "provider", "model", "effort")}, + "model": {**{k: entry.get(k, "") for k in ("role", "provider", "model", "effort")}, + **({"native_reasoning_level": entry["native_reasoning_level"]} if "native_reasoning_level" in entry else {})}, "inputs": {k: store.put_blob(str(entry.get(k, ""))) for k in ("system", "prompt")}, "usage": {}, "cost_usd": None, "served_model": "", "seconds": None, "native_session_ids": [], "env": environment_fingerprint()} diff --git a/test/test_kb_feature_discovery.py b/test/test_kb_feature_discovery.py index 1cf68a9b..96309b60 100644 --- a/test/test_kb_feature_discovery.py +++ b/test/test_kb_feature_discovery.py @@ -13,6 +13,42 @@ PIN = 'a' * 40 +@pytest.mark.parametrize('packet_chars', [24000, 96000, 160000, 192000]) +def test_configured_packets_offer_every_complete_chunk_once(tmp_path, packet_chars): + idx = index(tmp_path) + # Multiple files and chunks exercise boundaries without reducing the index. + for n in range(9): + (tmp_path / f'src/library_{n}.py').write_text('value = "observable contract"\n' * 1500) + idx = build_discovery_index(tmp_path, pin=PIN, scope={'roots': ['src/'], 'exclude': []}, doc_globs=['docs/*.md']) + engine = DiscoveryEngine(idx, seeds=[], owners=[], state={}, call=None, save=lambda: None, + packet_chars=packet_chars) + offered = [c for kind in ('doc', 'source') for packet in engine._packets(kind) for c in packet] + assert sorted(map(_hash, offered)) == sorted(map(_hash, idx.chunks)) + assert len(offered) == len(idx.chunks) + assert all(sum(len(c.get('text', '')) + 150 for c in packet) <= packet_chars + for kind in ('doc', 'source') for packet in engine._packets(kind)) + + +@pytest.mark.parametrize('packet_chars', [True, 0, 23999, 192001, 96000.5]) +def test_invalid_engine_packet_size_is_rejected(tmp_path, packet_chars): + with pytest.raises(ValueError, match='packet size'): + DiscoveryEngine(index(tmp_path), seeds=[], owners=[], state={}, call=None, + save=lambda: None, packet_chars=packet_chars) + + +def test_packet_candidate_cap_is_an_explicit_omission_lead(tmp_path): + idx = index(tmp_path, docs=False) + def call(role, system, prompt, validate): + return reply(role, {'candidates': [candidate(key=f'capability-{n}') for n in range(24)]}) + engine = DiscoveryEngine(idx, seeds=[], owners=[], state={}, call=call, save=lambda: None, + packet_chars=96000) + assert engine.scan() + assert len(engine.state['tasks']) == 1 + task = next(iter(engine.state['tasks'].values())) + assert task['candidate_limit_reached'] + assert len(task['candidates']) == 24 + + def reply(role, data, archived=True): text = json.dumps(data) return ModelReply(ModelRole(role, 'zcode' if role == 'generator' else 'codex', 'GLM-5.3' if role == 'generator' else 'gpt-6-sol'), diff --git a/test/test_kb_feature_discovery_stage.py b/test/test_kb_feature_discovery_stage.py index 54e12347..be0cefdf 100644 --- a/test/test_kb_feature_discovery_stage.py +++ b/test/test_kb_feature_discovery_stage.py @@ -3,18 +3,20 @@ import hashlib import json from pathlib import Path +from types import SimpleNamespace import pytest import yaml -from infermatrix_copilot.kb_service.init_feature_discovery import SYSTEM_DISCOVER, SYSTEM_REVIEW +from infermatrix_copilot.kb_service.init_feature_discovery import SYSTEM_DISCOVER, SYSTEM_REVIEW, discovery_run_config +from infermatrix_copilot.kb_service.init_support import InitError from infermatrix_copilot.kb_service.init_stages import run_stage from infermatrix_copilot.kb_service.models import ModelGateway from infermatrix_copilot.kb_service.runtime import trace_recorder from infermatrix_copilot.llm import Block, Reply from infermatrix_copilot.trace_store import TraceStore -from test_kb_init_skeleton import _commit, _index, _lifecycle, _runtime, _tree, world # noqa: F401 +from test_kb_init_skeleton import FakeGh, _commit, _index, _lifecycle, _runtime, _tree, world # noqa: F401 class NativeScript: @@ -103,6 +105,146 @@ def _policy(): "page": "repos/toy/components/core/feature-existing-step.md"}]} +def test_discovery_runtime_configuration_and_effective_reasoning(): + rt = SimpleNamespace(generator=SimpleNamespace(provider='zcode'), + gateway=ModelGateway(SimpleNamespace(zcode_reasoning_level='low'))) + assert discovery_run_config(rt, {}) == {'packet_chars': 24000, + 'zcode_start_interval_s': 15.0, 'zcode_reasoning_level': 'low'} + assert discovery_run_config(rt, {'KB_DISCOVERY_PACKET_CHARS': '96000', + 'KB_DISCOVERY_START_INTERVAL_S': '2.5'}) == {'packet_chars': 96000, + 'zcode_start_interval_s': 2.5, 'zcode_reasoning_level': 'low'} + + +@pytest.mark.parametrize('key,value', [('KB_DISCOVERY_PACKET_CHARS', '23999'), + ('KB_DISCOVERY_PACKET_CHARS', '192001'), ('KB_DISCOVERY_PACKET_CHARS', '96000.5'), + ('KB_DISCOVERY_PACKET_CHARS', 'garbage'), ('KB_DISCOVERY_START_INTERVAL_S', '0'), + ('KB_DISCOVERY_START_INTERVAL_S', 'nan'), ('KB_DISCOVERY_START_INTERVAL_S', 'inf'), + ('KB_DISCOVERY_START_INTERVAL_S', '61')]) +def test_invalid_discovery_configuration_refuses_before_model_dispatch(key, value): + rt = SimpleNamespace(generator=SimpleNamespace(provider='zcode'), gateway=ModelGateway(None)) + with pytest.raises(InitError, match=key): + discovery_run_config(rt, {key: value}) + + +@pytest.mark.parametrize('changed', ['packet', 'pacing', 'reasoning']) +def test_completed_discovery_cannot_resume_under_changed_configuration(world, monkeypatch, changed): + rt, lifecycle, script, store = _setup(world) + first = _run(rt, lifecycle) + assert first.status == 'dry_run', first.problems + calls = len(script.calls) + if changed == 'packet': + rt.environ['KB_DISCOVERY_PACKET_CHARS'] = '96000' + elif changed == 'pacing': + rt.environ['KB_DISCOVERY_START_INTERVAL_S'] = '10' + else: + rt.gateway._settings = SimpleNamespace(zcode_reasoning_level='low') + second = _run(rt, lifecycle) + assert second.status == 'blocked' + assert any('inputs changed' in p for p in second.problems) + assert len(script.calls) == calls + + +@pytest.mark.parametrize('changed', ['packet', 'pacing', 'reasoning']) +def test_prepared_discovery_rejects_changed_config_and_restores_exact_publication(world, changed): + from infermatrix_copilot.kb_service.init_support import load_prepared + rt, lifecycle, script, _ = _setup(world) + rt.environ.update(ALLOW_PUSH='1', ALLOW_POST='1', KB_INIT_GIT_AUTHOR='t ') + gh = FakeGh(fail_create=1) + rt.gh_run = gh + + def publish(): + return run_stage(rt, lifecycle, 'feature-discovery', dry_run=False, from_existing=True, + unlimited_subscription=True) + + pending = publish() + assert pending.status == 'blocked' and pending.pr.get('prepared'), pending.problems + prepared = load_prepared(pending.pr['prepared']) + prepared_bytes = Path(pending.pr['prepared']).read_bytes() + calls, pushes = len(script.calls), list(gh.pushed) + if changed == 'packet': + rt.environ['KB_DISCOVERY_PACKET_CHARS'] = '96000' + elif changed == 'pacing': + rt.environ['KB_DISCOVERY_START_INTERVAL_S'] = '10' + else: + rt.gateway._settings = SimpleNamespace(zcode_reasoning_level='low') + rejected = publish() + assert rejected.status == 'blocked' + assert any('discovery inputs changed' in p for p in rejected.problems) + assert len(script.calls) == calls and gh.pushed == pushes + assert Path(pending.pr['prepared']).read_bytes() == prepared_bytes + rt.environ.pop('KB_DISCOVERY_PACKET_CHARS', None) + rt.environ.pop('KB_DISCOVERY_START_INTERVAL_S', None) + rt.gateway._settings = None + restored = publish() + assert restored.status == 'published', restored.problems + assert len(script.calls) == calls and gh.pushed == pushes + assert load_prepared(restored.pr['prepared']) == prepared + + +def test_effective_native_reasoning_is_bound_to_report_and_both_archives(world, monkeypatch): + rt, lifecycle, script, store = _setup(world) + rt.environ.update(KB_DISCOVERY_PACKET_CHARS='96000', KB_DISCOVERY_START_INTERVAL_S='10') + rt.gateway._settings = SimpleNamespace(zcode_reasoning_level='low') + record = _run(rt, lifecycle) + assert record.status == 'dry_run', record.problems + _, report, _ = _report(record) + assert report['run_config'] == record.discovery['run_config'] == { + 'packet_chars': 96000, 'zcode_start_interval_s': 10.0, 'zcode_reasoning_level': 'low'} + for candidate in report['candidates']: + for proof in candidate['generator_receipts']: + archived = store.get(proof['trace_id']) + assert archived['model']['effort'] == '' + assert archived['model']['native_reasoning_level'] == 'low' + attempt = json.loads((store.root / 'attempts' / archived['result']['native_attempt_id'] / 'attempt.json').read_text()) + assert attempt['model']['native_reasoning_level'] == 'low' + + +def test_discovery_uses_runtime_environment_instead_of_ambient_environment(world, monkeypatch): + monkeypatch.setenv('KB_DISCOVERY_PACKET_CHARS', 'invalid-ambient-value') + rt, lifecycle, _, _ = _setup(world) + record = _run(rt, lifecycle) + assert record.status == 'dry_run', record.problems + assert record.discovery['run_config']['packet_chars'] == 24000 + + +def test_zcode_role_effort_does_not_mislabel_effective_native_configuration(world): + rt, lifecycle, _, store = _setup(world) + rt.environ['KB_DISCOVERY_GENERATOR'] = 'zcode:GLM-5.3:low' + rt.gateway._settings = SimpleNamespace(zcode_reasoning_level='max') + record = _run(rt, lifecycle) + assert record.status == 'dry_run', record.problems + _, report, _ = _report(record) + assert report['run_config']['zcode_reasoning_level'] == 'max' + for candidate in report['candidates']: + for proof in candidate['generator_receipts']: + model = store.get(proof['trace_id'])['model'] + assert model['effort'] == 'low' + assert model['native_reasoning_level'] == 'max' + + +def test_configured_reasoning_environment_is_loaded_by_settings(monkeypatch): + from infermatrix_copilot.config import Settings + monkeypatch.setenv('ZCODE_REASONING_LEVEL', 'low') + assert ModelGateway(Settings(_env_file=None)).zcode_reasoning_level == 'low' + + +def test_native_archive_reasoning_metadata_mismatch_is_rejected(world): + from infermatrix_copilot.kb_service.init_feature_discovery import _FeatureDiscovery, _prompt, validate_candidates + from infermatrix_copilot.kb_service.models import ModelRole, ModelUnavailable + rt, lifecycle, _, store = _setup(world) + role = ModelRole('generator', 'zcode', 'GLM-5.3') + reply = rt.gateway.call_json(role, system=SYSTEM_DISCOVER, + prompt=_prompt({'round': 'source', 'files': []}), validate=validate_candidates) + archived = store.get(reply.trace_id) + path = store.root / 'attempts' / archived['result']['native_attempt_id'] / 'attempt.json' + attempt = json.loads(path.read_text()) + attempt['model']['native_reasoning_level'] = 'low' + path.write_text(json.dumps(attempt)) + stage = _FeatureDiscovery(rt, lifecycle, dry_run=True, pin=None) + with pytest.raises(ModelUnavailable, match='reasoning level'): + stage._verify_native_receipt(reply) + + def test_first_source_only_catalog_has_native_review_and_no_knowledge_rewrite(world): rt, lifecycle, script, store = _setup(world) record = _run(rt, lifecycle) diff --git a/test/test_kb_init_branch_suffix.py b/test/test_kb_init_branch_suffix.py new file mode 100644 index 00000000..50f380af --- /dev/null +++ b/test/test_kb_init_branch_suffix.py @@ -0,0 +1,103 @@ +"""Repeated init publications preserve old branches and frozen batch names.""" + +import json +import subprocess + +import pytest + +from infermatrix_copilot.kb_service.init_stages import BRANCH_SUFFIX_ENV, run_stage +from infermatrix_copilot.kb_service.init_support import InitError, load_prepared + +from test_kb_init_existing_modules import ENV, _merged_baseline +from test_kb_init_modules import CardGateway, _modules_lifecycle +from test_kb_init_skeleton import FakeGateway, FakeGh, _lifecycle, _runtime, world # noqa: F401 + + +class BranchGh: + """Real commit plumbing, scripted independent remote refs and PRs.""" + + def __init__(self, heads): + self.heads = dict(heads) + self.prs = {} + self.pushed_branches = [] + + def __call__(self, cmd, **kwargs): + if cmd[:3] == ["gh", "pr", "list"]: + branch = cmd[cmd.index("--head") + 1] + return subprocess.CompletedProcess(cmd, 0, json.dumps(self.prs.get(branch, [])).encode(), b"") + if cmd[:3] == ["gh", "pr", "create"]: + branch = cmd[cmd.index("--head") + 1] + self.prs[branch] = [{"number": 42, "headRefOid": self.heads["refs/heads/" + branch]}] + return subprocess.CompletedProcess(cmd, 0, b"https://example/pull/42", b"") + if "ls-remote" in cmd: + ref = cmd[-1] + text = self.heads.get(ref, "") + return subprocess.CompletedProcess(cmd, 0, (f"{text}\t{ref}\n" if text else "").encode(), b"") + if "push" in cmd: + value = next(arg for arg in cmd if ":refs/heads/" in arg and not arg.startswith("--")) + commit, ref = value.split(":", 1) + assert ref not in self.heads and f"--force-with-lease={ref}:" in cmd + self.heads[ref] = commit + self.pushed_branches.append(ref) + return subprocess.CompletedProcess(cmd, 0, b"", b"") + return subprocess.run(cmd, **kwargs) + + +def test_second_modules_batch_uses_a_distinct_branch_and_keeps_old_ref(world): + pin, _, _, _ = _merged_baseline(world) + old_ref = "refs/heads/kb/init-toy-modules" + old_commit = world["kb_sha"] + gh = BranchGh({old_ref: old_commit}) + environment = {**ENV, BRANCH_SUFFIX_ENV: "new-init-20261005"} + record = run_stage(_runtime(world, CardGateway(), state_dir=world["tmp"] / "new-batch", + environ=environment, gh_run=gh), + _modules_lifecycle(feature_discovery_required=True), "modules", + dry_run=False, pin=pin, from_existing=True) + assert record.status == "published", record.problems + assert record.pr["branch"] == "kb/init-toy-modules-new-init-20261005" + assert gh.heads[old_ref] == old_commit + assert gh.pushed_branches == ["refs/heads/" + record.pr["branch"]] + assert load_prepared(record.pr["prepared"])["branch"] == record.pr["branch"] + + +@pytest.mark.parametrize("suffix", ["Upper", "bad/name", "bad..name", "-start", "end-", "a" * 41, + "bad name", "$(command)", "é", True]) +def test_invalid_branch_suffix_is_refused_before_models_or_publication(world, suffix): + gateway, gh = FakeGateway(), FakeGh() + with pytest.raises(InitError, match=BRANCH_SUFFIX_ENV): + run_stage(_runtime(world, gateway, environ={**ENV, BRANCH_SUFFIX_ENV: suffix}, gh_run=gh), + _lifecycle(), "skeleton", dry_run=False) + assert not gateway.calls and not gh.pushed + + +def test_unset_and_empty_suffix_preserve_default_checkpoint_identity(world): + gateway = FakeGateway() + rt = _runtime(world, gateway) + original = run_stage(rt, _lifecycle(), "skeleton", dry_run=True) + calls = len(gateway.calls) + rt.environ[BRANCH_SUFFIX_ENV] = "" + cached = run_stage(rt, _lifecycle(), "skeleton", dry_run=True) + assert cached.inputs_digest == original.inputs_digest + assert len(gateway.calls) == calls + + +def test_prepared_publication_rejects_changed_suffix_and_resumes_original_without_models(world): + environment = {**ENV, BRANCH_SUFFIX_ENV: "batch-one"} + gateway, gh = FakeGateway(), FakeGh(fail_create=1) + rt = _runtime(world, gateway, environ=environment, gh_run=gh) + pending = run_stage(rt, _lifecycle(), "skeleton", dry_run=False) + assert pending.status == "blocked" and pending.pr.get("prepared") + prepared = load_prepared(pending.pr["prepared"]) + assert prepared["branch"] == "kb/init-toy-skeleton-batch-one" + original_calls = len(gateway.calls) + original_pushes = list(gh.pushed) + rt.environ[BRANCH_SUFFIX_ENV] = "batch-two" + rejected = run_stage(rt, _lifecycle(), "skeleton", dry_run=False) + assert rejected.status == "blocked" and any("different branch suffix" in p for p in rejected.problems) + assert len(gateway.calls) == original_calls and gh.pushed == original_pushes + assert load_prepared(pending.pr["prepared"]) == prepared + rt.environ[BRANCH_SUFFIX_ENV] = "batch-one" + published = run_stage(rt, _lifecycle(), "skeleton", dry_run=False) + assert published.status == "published", published.problems + assert published.pr["branch"] == prepared["branch"] + assert len(gateway.calls) == original_calls and gh.pushed == original_pushes diff --git a/test/test_kb_init_existing_modules.py b/test/test_kb_init_existing_modules.py new file mode 100644 index 00000000..52fad915 --- /dev/null +++ b/test/test_kb_init_existing_modules.py @@ -0,0 +1,135 @@ +"""Merged owner maps can seed breadth reruns without synthetic stage records.""" + +import hashlib +import json +import subprocess +from types import SimpleNamespace + +import pytest +import yaml + +from infermatrix_copilot.kb_service.init_stages import run_stage +from infermatrix_copilot.kb_service.init_support import InitRecord + +from test_kb_feature_discovery_integration import _artifacts, _record, _stage +from test_kb_init_knowledge import KnowledgeGateway +from test_kb_init_modules import CardGateway, _modules_lifecycle, _skeleton, _tree_at, _world_with_tools +from test_kb_init_skeleton import FakeGh, _commit, _runtime, _tree, world # noqa: F401 + + +ENV = {"ALLOW_PUSH": "1", "ALLOW_POST": "1", "KB_INIT_GIT_AUTHOR": "t "} + + +def _merged_baseline(world): + _world_with_tools(world) + skeleton = _skeleton(world) + files, report, path = _artifacts(skeleton.pin) + policy = yaml.safe_load(files["adapters/toy/knowledge-coverage.yaml"]) + policy["core"]["roots"] = ["pkg/", "tools/"] + files["adapters/toy/knowledge-coverage.yaml"] = yaml.safe_dump(policy) + report["catalog_sha256"] = hashlib.sha256(files["adapters/toy/knowledge-coverage.yaml"].encode()).hexdigest() + files[path] = json.dumps(report) + _commit(world["origin"], {**_tree(skeleton), **files}, "merged skeleton and discovered catalog") + return skeleton.pin, files, report, path + + +def test_explicit_existing_modules_uses_merged_catalog_without_local_skeleton(world): + pin, _, report, _ = _merged_baseline(world) + state = world["tmp"] / "existing" + rt = _runtime(world, CardGateway(), state_dir=state) + lifecycle = _modules_lifecycle(feature_discovery_required=True) + legacy = run_stage(rt, lifecycle, "modules", dry_run=True, pin=pin) + assert legacy.status == "blocked" and "run the skeleton stage first" in legacy.problems + record = run_stage(rt, lifecycle, "modules", dry_run=True, pin=pin, from_existing=True) + assert record.status == "dry_run", record.problems + assert record.coverage["modules"]["after"] == 1 + assert record.discovery["catalog_binding"]["catalog_sha256"] == report["catalog_sha256"] + assert not InitRecord.path(state, "toy", "skeleton").exists() + + +def test_existing_modules_cannot_replace_missing_merged_owner_map(world): + gateway = CardGateway() + record = run_stage(_runtime(world, gateway), _modules_lifecycle(), "modules", + dry_run=True, from_existing=True) + assert record.status == "blocked" and any("merged repository index" in p for p in record.problems) + assert not gateway.calls + assert not InitRecord.path(world["tmp"] / "state", "toy", "skeleton").exists() + + +@pytest.mark.parametrize("status", ["blocked", "dry_run"]) +def test_existing_modules_preserves_local_skeleton_status_gate(world, status): + pin, _, _, _ = _merged_baseline(world) + state = world["tmp"] / "existing" + InitRecord(stage="skeleton", repo="toy", pin=pin, status=status).save(state) + gateway = CardGateway() + record = run_stage(_runtime(world, gateway, state_dir=state, environ=ENV), + _modules_lifecycle(feature_discovery_required=True), "modules", + dry_run=False, pin=pin, from_existing=True) + assert record.status == "blocked" + assert any("skeleton" in p and ("blocked" in p or "dry run" in p) for p in record.problems) + assert not gateway.calls + + +def test_existing_modules_preserves_unmerged_discovery_gate(world): + pin, files, report, path = _merged_baseline(world) + state = world["tmp"] / "existing" + stage = _stage(world) + stage.rt.state_dir = state + discovery = _record(stage, files, report, path, status="published") + discovery.pr = {"number": 42} + discovery.save(state) + gateway = CardGateway() + rt = _runtime(world, gateway, state_dir=state, + gh_run=lambda *args, **kwargs: subprocess.CompletedProcess(args, 0, b'{"state":"OPEN"}', b"")) + record = run_stage(rt, _modules_lifecycle(feature_discovery_required=True), "modules", + dry_run=True, pin=pin, from_existing=True) + assert record.status == "blocked" and "merge the feature-discovery PR before continuing" in record.problems + assert not gateway.calls + assert not InitRecord.path(state, "toy", "skeleton").exists() + + +def test_knowledge_after_existing_modules_keeps_module_merge_gate_without_skeleton(world): + pin, _, _, _ = _merged_baseline(world) + state = world["tmp"] / "existing" + lifecycle = _modules_lifecycle(feature_discovery_required=True) + modules = run_stage(_runtime(world, CardGateway(), state_dir=state, environ=ENV, gh_run=FakeGh()), + lifecycle, "modules", dry_run=False, pin=pin, from_existing=True) + assert modules.status == "published", modules.problems + + def pr_state(value): + return lambda *args, **kwargs: subprocess.CompletedProcess(args, 0, ('{"state":"' + value + '"}').encode(), b"") + + gateway = KnowledgeGateway() + rt = _runtime(world, gateway, state_dir=state, gh_run=pr_state("OPEN")) + modules.status = "blocked" + modules.save(state) + unfinished = run_stage(rt, lifecycle, "knowledge", dry_run=True, pin=pin, from_existing=True) + assert unfinished.status == "blocked" and "the modules stage is blocked; finish it first" in unfinished.problems + modules.status = "published" + modules.save(state) + waiting = run_stage(rt, lifecycle, "knowledge", dry_run=True, pin=pin, from_existing=True) + assert waiting.status == "blocked" and any("merge the modules PR" in p for p in waiting.problems) + assert not gateway.calls + _commit(world["origin"], _tree_at(world, modules.pr["head_sha"]), "merge breadth rerun") + rt.gh_run = pr_state("MERGED") + record = run_stage(rt, lifecycle, "knowledge", dry_run=True, pin=pin, from_existing=True) + assert record.status == "dry_run", record.problems + assert record.coverage["knowledge"]["total_owners"] == 2 + assert record.discovery["catalog_binding"] == modules.discovery["catalog_binding"] + assert not InitRecord.path(state, "toy", "skeleton").exists() + + +def test_modules_cli_forwards_existing_baseline_flag(monkeypatch, tmp_path): + from infermatrix_copilot.kb_service.cli import main + from infermatrix_copilot.kb_service import runner + + calls = [] + + def run(*args, **kwargs): + calls.append(kwargs["params"]) + return SimpleNamespace(status="done"), tmp_path + + monkeypatch.setattr(runner, "run_playbook", run) + assert main(["--state-dir", str(tmp_path), "init", "toy", "--stage", "modules", + "--from-existing", "--pin", "a" * 40, "--dry-run"]) == 0 + assert calls == [{"stage": "modules", "dry_run": "true", "pin": "a" * 40, "from_existing": "true"}]