diff --git a/doc/architecture/SPEC/kb_service.md b/doc/architecture/SPEC/kb_service.md index e75fe9250..944823866 100644 --- a/doc/architecture/SPEC/kb_service.md +++ b/doc/architecture/SPEC/kb_service.md @@ -318,6 +318,26 @@ issue（同一报告只开一个），标签不存在时不带标签重试，绝 有待异步接受、撤回或原始 intake 候选的提交拒绝进入此路径。`kb status` 显示已核验的 `supervised_commits`。 - 测试：`test_kb_reconcile_reviewed.py`。 +## 2026-10-06 已审阅源事件的监督完成 + +- `kb reconcile-reviewed --event-coverage FILE --plan PLAN --target SHA --allow-merger LOGIN --require-check NAME --reason TEXT` + 接收明确的 `commits` 和 `events` JSON；每项事件给出 `id`、`outcome`（`covered`、`already_covered`、`no_rule`）、 + 操作员审阅理由 `reason`，覆盖事件还须给出 `rules: [{path, rule_id}]`。路径以 `repos/` 开始，不含 `knowledge/`。 + `no_rule` 不声称任何规则。覆盖规则必须在精确目标快照中有效，且引用该事件的原始上游 PR。 +- 只有原始 `merged_pr` payload 缺少 `merge_commit_sha` 时，可显式给该项完整 `source_merge_sha`； + 从注册 adapter 的 `RepoLifecycle.full_name` 取得上游，实时核验 GitHub PR 的 repository/number/URL、 + merged 状态、精确 merge/head SHA 和 merge time（原始 timestamp 存在时须相符）。此补充证明进入签名计划与回执， + 不改写原始 payload 字节/哈希，不能覆盖既有 merge SHA；apply 与重复 apply 都从签名行保留 pin 并重新取证，漂移即拒绝。 +- 目标须已监督接收并实际激活；认证操作员、真实 owner merge、精确 head 上的当前 CI、原始事件身份和 payload 哈希、 + 规则块、两个校验器及完整快照 manifest 均进入签名计划。选择不能包含不存在的事件或拆开已有源事件 batch。 +- `--apply PLAN` 持有独占 ledger lease 和激活锁，重新验证全部证据；源事件、规则、main/active 或 CI 漂移即拒绝。 + 不同签名用途 `kb-reviewed-event-plan` / `kb-reviewed-event-settlement`，不可作为知识历史接收回执使用。 + `event-settlements/<内容哈希>.json` 经 fsync、硬链接原子创建且禁止替换；崩溃后可用原计划完成相同的 ledger 提交。 +- 仅所选 pending/rejected/drafted 事件，或原始 detail 精确为 `no rules` 的 done 事件，可记录诚实的 `manual_reviewed_merged` 和回执引用。后者允许纠正过早丢弃，签名回执保留原始 done/no rules 决定；其他终态不得覆盖。 + 其完整源 batch 和 refinement 后代也记录 `manual_reviewed_merged`，停止重复生成；原来的失败判决和 verdict 不变。 + 只解决被完整覆盖 changeset 的人工队列，不影响其他事件、PR、发布模式或知识信任。 +- 测试：`test_kb_reviewed_events.py`。 + ## 2026-09-29 一致性冲突只算本次改动的 一致性评审看整个 owner 目录，但只有**涉及本次改动的规则**（新增、编辑、退役、被替代的规则 ID）的冲突才使变更失败； 两条都未被改动的规则之间的冲突已经在 main 上，记入该目录的 `preexisting`，不计入变更（精炼也无法修复它，否则该目录 diff --git a/doc/architecture/SPEC/knowledge_service/curation.md b/doc/architecture/SPEC/knowledge_service/curation.md index b5e3be514..6e3ad3b24 100644 --- a/doc/architecture/SPEC/knowledge_service/curation.md +++ b/doc/architecture/SPEC/knowledge_service/curation.md @@ -1,6 +1,13 @@ # knowledge_service/curation.py — provider curation contract - + + +`KnowledgeCurator.reviewed_rule_evidence(page_text, rule_id=..., source_reference=...)` +uses the canonical parser to verify one active, source-citing rule and returns +its exact UTF-8 section SHA256. `installed_knowledge_file(path)` reads canonical +`knowledge/repos/*.md` package resources from the installed provider, independent +of workspaces and environment overrides. Consumers can prove reviewed coverage +without copying domain parsing or importing provider-private modules. `KnowledgeCurator` composes provider-owned catalog, bounded evidence prompt, proposal validation, append-only apply, fixed validator execution, and byte-exact diff --git a/knowledge/repos/jiuwenswarm/components/agent-server-runtime/runtime-agent-adapter/_index.md b/knowledge/repos/jiuwenswarm/components/agent-server-runtime/runtime-agent-adapter/_index.md index 8343d72c2..bdbc30e6b 100644 --- a/knowledge/repos/jiuwenswarm/components/agent-server-runtime/runtime-agent-adapter/_index.md +++ b/knowledge/repos/jiuwenswarm/components/agent-server-runtime/runtime-agent-adapter/_index.md @@ -1,7 +1,7 @@ --- title: "runtime-agent-adapter" created: 2026-10-01 -updated: 2026-10-01 +updated: 2026-10-06 type: index tags: [jiuwenswarm] sources: [] @@ -12,3 +12,5 @@ sources: [] 理解本 owner 的源码职责、接口和集成边界时查这里。源码接口记录是静态契约；功能语义、配置和取舍沿功能页查证。通用审查方法不放在这里。 - [runtime-agent-adapter 源码接口与集成边界 01](source-contracts-01.md) - [runtime-agent-adapter 源码接口与集成边界 02](source-contracts-02.md) + +- [Agent adapter 运行时规则](rules.md) — Code profile 的模型路由 rail 挂载与 session 重建。 diff --git a/knowledge/repos/jiuwenswarm/components/agent-server-runtime/runtime-agent-adapter/rules.md b/knowledge/repos/jiuwenswarm/components/agent-server-runtime/runtime-agent-adapter/rules.md new file mode 100644 index 000000000..aeabf1b4d --- /dev/null +++ b/knowledge/repos/jiuwenswarm/components/agent-server-runtime/runtime-agent-adapter/rules.md @@ -0,0 +1,18 @@ +--- +title: "Agent adapter 运行时规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [jiuwenswarm, components] +sources: ["PR #7689"] +confidence: high +--- + +# Agent adapter 运行时规则 + +## JW-RUNTIME-ROUTING-1a — Code 的模型路由 rail 由规范化 sub-mode 决定 + +- 触发：修改 `CodeAdapter._build_agent_rails`、Code session 重建或模型档位路由的挂载。 +- 强制：以规范化后的 `sub_mode` 区分 profile；`normal`/`plan` 分支追加 `_model_routing_rail`，builder 为 `_build_model_routing`，参数使用同一 `config_base`；team 分支将该属性设为 `None`。重建时顶层 `mode="agent"` 的兼容默认不能替代 sub-mode 判定。 +- 禁止：只凭顶层 mode 决定挂载；给已由 relay 解析具体模型 ID 的 team profile 再挂同一档位 rail；把挂载 diff 或注释当作四档实际转换已验证的证据。 +- 验收：分别检查默认/规范化后的 `normal`、`plan` 与 team 分支，断言 builder、config 和属性清除一致；需要宣称档位转换有效时另核对 rail 的转换实现及实际路由测试。^[PR #7689] diff --git a/knowledge/repos/vllm-omni/benchmark/_index.md b/knowledge/repos/vllm-omni/benchmark/_index.md index 314f10e28..5361f062e 100644 --- a/knowledge/repos/vllm-omni/benchmark/_index.md +++ b/knowledge/repos/vllm-omni/benchmark/_index.md @@ -1,7 +1,7 @@ --- title: "vLLM-Omni 性能与 Profiling" created: 2026-07-10 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, benchmark] sources: [benchmarks/tts/bench_tts.py, benchmarks/tts/model_configs.yaml, "PR #6522", "PR #6634", "PR #6818", vllm_omni/benchmarks/data_modules/omniinteract_dataset.py, vllm_omni/benchmarks/omniinteract.py, vllm_omni/benchmarks/patch/patch.py, tests/benchmarks/patch/test_patch.py, vllm_omni/benchmarks/duplex/, vllm_omni/entrypoints/cli/benchmark/omni_duplex_eval.py, vllm_omni/experimental/fullduplex/client.py] @@ -21,7 +21,7 @@ sources: [benchmarks/tts/bench_tts.py, benchmarks/tts/model_configs.yaml, "PR #6 | 遇到什么 | 查看哪里 | |---|---| -| 新增/审查 perf JSON、warmup 覆盖、realtime artifact、Hub dataset fallback、per-request timeout 或 `/v1/videos` 轮询预算、最终性能复测与控制漂移 | [benchmark rules](rules.md) 新增核对：BENCH-1f2、BENCH-1g。 | +| 新增/审查 perf JSON、warmup 覆盖、realtime artifact、Hub dataset fallback、per-request timeout 或 `/v1/videos` 轮询预算、最终性能复测与控制漂移、startup/raw-load测量边界或archive baseline采样窗口 | [benchmark rules](rules.md) 新增核对：BENCH-1f2、BENCH-1g。 | | 查看旧 benchmark 总览 | [overview](overview.md) | | 运行 AR graph、Hunyuan 或 MiniCPM-o Omni-DuplexEval benchmark | [benchmark guides](guides/_index.md) | | 调查 profiling、模型加载和性能验证错误 | [benchmark incidents](incidents/_index.md) | diff --git a/knowledge/repos/vllm-omni/benchmark/rules.md b/knowledge/repos/vllm-omni/benchmark/rules.md index 6a7a7fe84..f47d70936 100644 --- a/knowledge/repos/vllm-omni/benchmark/rules.md +++ b/knowledge/repos/vllm-omni/benchmark/rules.md @@ -1,10 +1,10 @@ --- title: "vLLM-Omni Benchmark 规则" created: 2026-09-05 -updated: 2026-09-26 +updated: 2026-10-06 type: rule tags: [vllm-omni, benchmark] -sources: ["PR #7648", "PR #6817", tests/dfx/perf/scripts/run_benchmark.py, tests/benchmarks/test_omniinteract.py, "PR #7130", "PR #7259", "PR #7624", "PR #7504", "PR #8107"] +sources: ["PR #7648", "PR #6817", tests/dfx/perf/scripts/run_benchmark.py, tests/benchmarks/test_omniinteract.py, "PR #7130", "PR #7259", "PR #7624", "PR #7504", "PR #8107", "PR #7312", "PR #6538"] confidence: high --- @@ -93,3 +93,33 @@ confidence: high - 强制：`run_diffusion_benchmark.py` 只跑 `is_diffusion_perf_config` 为真的 case（`benchmark_params[].dataset`）；`run_benchmark.py` 只跑 omni-bench case（`dataset_name`）。schema 过滤看字段，不看 `mark`。迁移 JSON 后，所有引用该文件的 CUDA/NPU/AMD 步骤必须一起换 runner，并改用对应的 `BENCHMARK_DIR` 与 artifact glob。 - 禁止：一边已迁 omni-bench、一边仍调 diffusion runner（会 skip 全部 case，pytest 0 selected / exit 5）；把某一平台的 runner 修复外推为其他 pipeline 已对齐。 - 验收：扫描全部 `.buildkite` 调用，断言 runner 与 JSON schema 一致；HunyuanVideo-1.5 t2v 等已迁 JSON 不得再回到 `run_diffusion_benchmark.py`。^[PR #8107] + +## BENCH-1i — startup 与 raw-load 数字必须保持测量边界并拒绝残缺样本 + +- 触发：修改 MammothModa2 startup/storage benchmark、stage log parser 或 raw safetensors loading诊断。 +- 强制：分别报告import、Omni构建、first/subsequent generate与process→first image；保存实际 + script hash、source/config/依赖、线程/cgroup、cache条件与每次fresh-process样本。parallel stage + intervals可重叠，不能相加；weight-load与model-load差值只是setup估计，缺字段保持未测量。 +- 强制：无 `BENCH_JSON` 为incomplete，`--require-complete`使其失败；storage wrapper传播engine + exit，删除旧summary/当前label旧输出，仅汇总本次成功logs。shard selection必须全匹配且非空。 + raw-loader materialization含page faults/CPU allocation/copy，H2D含GPU allocation；CUDA setup + 和operation warmup在timer外，不能称为disk bandwidth或engine-level read/cast/copy分解。 +- 禁止：把requested eviction当保证cold machine/NFS server cache；用一次fixed-seed image equality + 宣称跨prompt/mode质量等价。1Hz device-global GPU采样含其他users且可能漏transient peak， + process-tree RSS也只是采样；parser first-milestone的stage timeline不证明multi-replica全体时间。 +- 验收：覆盖day/year rollover、无completion、engine非零退出、旧log/summary、缺shard/partial + selection；报告rotated fresh-process重复样本及spread，memory polling另跑。性能结果只适用 + 实际hardware/config，CUDA graph memory/headroom和质量另做matched验证。^[PR #7312] + +## BENCH-1j — archived baseline 刷新必须绑定明确采样窗口与 workload + +- 触发：据nightly archive改DFX perf JSON硬件bucket或调高latency/降低throughput reference。 +- 强制：逐metric/hardware/concurrency给出immutable raw-run links、成功数、排除原因、软件/ + workload差异与重算方法；同window nightly等权算术平均，最后一次round，不暗加margin。 + temporary realtime backend与ordinary HTTP Seed-TTS、duplex timing边界不可混用；不同版本/ + 性能period分开，并披露选旧window或只有two-night coverage的敏感性限制。 +- 禁止：选择性删除慢night或把未知revision的archived均值当fixed-version因果对照；把更新 + reference说成优化目标已达成，或在无consumer时说成active performance regression gate。 +- 验收：独立由原始数据重算全部更改值，保持benchmark/deployment/tolerances未改且bucket/ + sweep位置一致；说明完整metric counts/失败数。PR #6538的69值属于历史reference，不能 + 推断H100 reference-audio变更解释全部shift或C8 RTF<1目标已实现。^[PR #6538] diff --git a/knowledge/repos/vllm-omni/ci/_index.md b/knowledge/repos/vllm-omni/ci/_index.md index 0a694940a..fc1855187 100644 --- a/knowledge/repos/vllm-omni/ci/_index.md +++ b/knowledge/repos/vllm-omni/ci/_index.md @@ -1,7 +1,7 @@ --- title: "vLLM-Omni CI" created: 2026-07-10 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, ci] sources: [.buildkite/cuda/pipeline.yml, docs/contributing/ci/test_system_overview.md, tests/diffusion/quantization/test_svdquant_config.py, tests/diffusion/quantization/test_svdquant_linear.py, tests/diffusion/quantization/test_svdquant_tp_loading.py, tests/diffusion/quantization/test_wan_autoround_mxfp4.py, tests/e2e/offline_inference/test_wan21_autoround_mxfp4.py, "PR #5544", "PR #6162", "PR #6170", "PR #6303", "PR #6390", "PR #6613", .buildkite/cuda/test-nightly.yml, tests/e2e/online_serving/run_minicpmo_realtime_duplex_server_vad.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py, tests/e2e/online_serving/test_qwen_image_expansion.py, tests/dfx/perf/tests/test_qwen_image_vllm_omni.json, tests/platforms/npu/test_diffusion_attn_backend_selector.py, "PR #6054", .buildkite/cuda/test-merge.yml, .buildkite/cuda/test-ready.yml, tests/e2e/online_serving/test_hunyuan_video_15_expansion.py, tests/dfx/perf/tests/test_hunyuanvideo15_t2v_vllm_omni.json, tests/dfx/perf/tests/test_hunyuanvideo15_i2v_vllm_omni.json, "PR #6349", .buildkite/amd/scripts/bootstrap-amd-omni.sh, .buildkite/amd/test-amd-merge.yml, .buildkite/amd/test-amd-ready.yml, tests/diffusion/distributed/test_tensor_parallel.py, tests/diffusion/offloader/test_diffusion_layerwise_offload.py, tests/helpers/clean.py, "PR #6704", tests/model_executor/models/minicpmo_4_5/test_pipeline.py, tests/model_executor/stage_input_processors/test_minicpmo_4_5_async_chunk.py, "PR #5464", "PR #6730", "PR #6745", "PR #6727", tests/diffusion/quantization/test_quantization_quality.py, "PR #5831", tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json, tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json, "PR #6743", tests/diffusion/models/minimax_h3/test_minimax_h3_quantization_quality.py, "PR #6742", "PR #6650", pyproject.toml, tests/helpers/mark.py, tests/helpers/tests/test_mark.py, tools/pre_commit/check_test_marks.py, "PR #6174", .buildkite/cuda/test-weekly.yml, tests/e2e/offline_inference/test_dots_tts_expansion.py, "PR #6556", tests/e2e/online_serving/minimax_h3/] @@ -174,11 +174,11 @@ sources: [.buildkite/cuda/pipeline.yml, docs/contributing/ci/test_system_overvie ## XPU base-image resolution and support-table boundary -- The Intel Buildkite XPU lane targets vLLM `v0.28.0`. Its Omni image layers on +- The Intel Buildkite XPU lane and Docker default must target the same checkout release. Its Omni image layers on `vllm/vllm-openai-xpu:${VLLM_VERSION}` and clears that image's serving entrypoint so the result remains a shell/test container. The base carries the matching XPU runtime and vLLM; Omni installs only its own layers. Keep the Dockerfile default and lane environment version - aligned. + aligned; the source PR #8528 aligns both and the install command to v0.31.0. ^[PR #8528] - Before building, CI pulls the published upstream base freshly. If that pull fails it may use an existing local copy; only if neither exists does it build the same vLLM tag from upstream `docker/Dockerfile.xpu`, optionally under `VLLM_BASE`. Thus a fallback is provenance-aligned, @@ -198,19 +198,22 @@ sources: [.buildkite/cuda/pipeline.yml, docs/contributing/ci/test_system_overvie Qwen3-TTS CustomVoice `async_chunk` Whisper-validation case is an 80-minute, `NonBlocking` quarantined step. It must return to the blocking sharded lane only after the AMD failure is resolved; a passing neighboring shard does not establish that result. -- ROCm memory assertions sample steady-state inference and report peak usage relative to - each run's initial device use, after cache cleanup. They therefore do not count retained - compiler/workspace allocations from a preceding topology as model memory, and the ROCm - layerwise-offload floor remains a conservative 512 MiB signal rather than a portability - or performance claim. Cleanup diagnostics use bounded `rocm-smi` calls because an - optional `amd-smi` CPER ioctl can remain uninterruptibly blocked. ^[PR #6704] +- ROCm offload memory assertion现在由实际diffusion worker内同步reset/read的process-local + allocator peak支持，不能用parent device-global total-free减initial归因model saving；RPC返回 + rank0不证明all-rank max。机制/quality证据边界见[DIFF-4x](../components/diffusion/rules-worker-observability.md)。 + cleanup diagnostics仍有独立timeout，保守offload floor不构成portability或performance保证。 + ^[PR #6704] ^[PR #8189] + ## 目录内容 +- [Graph 测试 buffer](rules-graph-buffer-fixtures.md) — fake estimator 未写 tail 的确定性初始化。 + | 遇到什么 | 查看哪里 | |---|---| +| 构建wheel runtime data、无VCS archive parity或声明Python版本的安装/导入支持 | [构建与版本合同](rules-build-compatibility.md) | | 审查硬件 lane（含 MiniMax-H3 DLO DP2 ready smoke 的证据边界）、回归 fence、CI 工具供应链、ASR 文本比较或 xdist/shared fixture、Buildkite 失败日志与运行时差异 | [CI rules](rules.md) 新增核对：OMNI-CI-1g、OMNI-CI-1h。 | -| AMD/ROCm timeout、quarantine、memory signal、single-card diffusion job 的 multi-card marker 排除、Qwen3-TTS argv 或 Qwen3-Omni control-plane fixture | [AMD/ROCm CI rules](rules-amd.md) 新增核对：OMNI-CI-2i、OMNI-CI-2i2、OMNI-CI-2i3、OMNI-CI-2i4。 | +| AMD/ROCm timeout、quarantine、memory signal、single-card diffusion job 的 multi-card marker 排除、Qwen3-TTS argv 或 Qwen3-Omni control-plane fixture | [AMD/ROCm CI rules](rules-amd.md) 新增核对：OMNI-CI-2i、OMNI-CI-2i2、OMNI-CI-2i3、OMNI-CI-2i4、OMNI-CI-2i5、OMNI-CI-2i6。 | | CUDA L4 Kubernetes preset、GPU-count shard、resource/retry policy | [L4 Kubernetes CI rules](rules-l4-k8s.md) | | Whisper 转写 helper 的 GPU 首选、16 GiB 门槛、CPU fallback 或 CUDA ready/merge source dependency | [Whisper 转写 CI 规则](rules-whisper-transcription.md) | | 查看仓库特有 CI 陷阱 | [CI guides](guides/_index.md) | diff --git a/knowledge/repos/vllm-omni/ci/rules-amd.md b/knowledge/repos/vllm-omni/ci/rules-amd.md index 96e5e0c7d..0c53b295d 100644 --- a/knowledge/repos/vllm-omni/ci/rules-amd.md +++ b/knowledge/repos/vllm-omni/ci/rules-amd.md @@ -1,10 +1,10 @@ --- title: "AMD/ROCm CI 规则" created: 2026-09-05 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, ci] -sources: ["PR #6704", "PR #6830", "PR #6884", .buildkite/amd/, tests/helpers/clean.py, tests/helpers/stage_config.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py, "PR #7234", "PR #6966", "PR #7395", "PR #6978", "PR #7398"] +sources: ["PR #6704", "PR #6830", "PR #6884", .buildkite/amd/, tests/helpers/clean.py, tests/helpers/stage_config.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py, "PR #7234", "PR #6966", "PR #7395", "PR #6978", "PR #7398", "PR #8342", "PR #7935", "PR #7933", "PR #8527", "PR #8189", "PR #8520"] confidence: high --- @@ -16,7 +16,8 @@ confidence: high cleanup diagnostics、Qwen3-TTS argv，或 Qwen3-Omni sleep/abort control-plane E2E 变更。 - 强制：debug-only READY+MERGE composition 保留 normal branch selection、一个 shared build dependency 和 named groups；MI300 long steps 有 explicit bounds，已知 failing case 保持独立、time-bounded - `NonBlocking`。memory comparisons 在 model/offload 后采样、减每 run initial device use 并清 cache； + `NonBlocking`。memory comparisons 在 model/offload 后采样并清 cache；原 device-global + incremental sampling 已由 worker-local allocator peak 合同替代，见 DIFF-4x； optional cleanup diagnostic 也必须 bounded。ROCm offload threshold 只是一项机制 signal，不是 portability/capacity/performance claim。^[PR #6704] - 强制：MI300 blocking layerwise-offload step 只 deselect exact Stable Audio parameter，并以 dependent @@ -43,6 +44,14 @@ confidence: high - 禁止：让 `cards_2+` 测试在单卡 worker 上 spawn rank1→GPU1 导致 `invalid device ordinal`；用邻近 green shard 宣称 multi-GPU routing 已修好。 - 验收：pipeline argv/collection 断言单卡 job 排除 multi-card markers、双卡 job 收集目标文件；硬件 marker helper 覆盖 ROCm 声明。^[PR #7234] +- Simple CPU diffusion shard 即使 module 同时带 cpu，也必须排除全部 multi-card case；Pi0.5 + 的昂贵 module-scoped CPU fixture 可整体迁到独立 blocking lane，避免拆到不同 shard 重复启动。 + AuK 四个 graph case 用独立 GPU selector；Ulysses mask-layout 在真实 two-GPU lane执行。 +- PR #8342 的 `not cards_2…cards_8` 仅修 multi-card misrouting，不保证排除所有 cards_1 GPU + case。MERGE 的 Ulysses command 又位于 Z-Image TP 后；前一 command hang/fail 时目标可能未执行。 + 审查须检查目标 node 的实际终态，不从 YAML presence/邻近 pass 推断全树 CPU/GPU routing 闭环。 + 专用 CPU/GPU per-test marker 防空转合同见 OMNI-CI-1a。^[PR #8342] + ## OMNI-CI-2i — AMD bootstrap 必须按 ready/merge-test 标签选择 L2/L3 suite - 触发：修改 `.buildkite/amd` bootstrap、`select_test_suites.py`、AMD PR label 路由，或 skip-ci 对 AMD suite 的过滤。 @@ -64,9 +73,42 @@ confidence: high - 禁止：把 `nightly-test` 当成无 suite 的噪声 label；用 L2/L3 diff gate 静默丢掉已选 nightly；把实验 nightly 阈值外推为 CUDA H100 基线。 - 验收：渲染 `DEBUG_TEST_YAML=nightly`、`NIGHTLY=1`、组合 label 与 docs-only+nightly，断言 suite spec 与子 pipeline 叶子集合。^[PR #6978] +- 新 AMD nightly 使用 shared native runner、精确 node/selector 的 fail-closed collect-only， + 再执行同一 selection，JUnit 从 /tmp staged 到 checkout-relative artifact root。rendered template + 已注入 amd-build 依赖时不重复 source field；通用 template/schema tests 与 rendered inspection + 验证所有权，避免每新增 job 都复制只复述配置的 UT。 +- LingBot ROCm 单卡仅给 CFG-off case MI325 eligibility，batch-CFG sibling 仍 CUDA-only；Cosmos3 + 保留 CUDA marks并增加 exact ROCm T2I node。NonBlocking、collection 与 artifact 不是模型质量 + 或目标 MI300 pass。两个 source 的最终 step budget 都为 90min。^[PR #7935] ^[PR #7933] +- Cosmos3 prewarm 与 server 使用同一 job-local AITER/Torch cache。仅在 ROCm、真实 FA可用且 + `AITER_JIT_DIR` 明示时，以 masked/unmasked BF16 MHA 分别检查输出 shape/finite 和两项预期 + `.so` 存在；prewarm 55min、test 25min 均以 TERM/kill bounds 留出 90min step信封。^[PR #7933] + ## OMNI-CI-2i4 — AMD entrypoints GPU ready job 必须单次 pytest、有界超时并检测进程泄漏 - 触发：修改 `.buildkite/amd/test-amd-ready.yml` 的 entrypoints/R2-02 GPU coverage、artifact 根路径，或 teardown 进程快照比较。 - 强制：`mi300_1` NonBlocking job 设 `VLLM_WORKER_MULTIPROC_METHOD=spawn`，marker 排除 `cards_2`–`cards_8`，只跑一次 verbose pytest（禁止第二趟 `--collect-only`）。内层 `timeout` 短于 Buildkite step，为 teardown/artifact 留窗口。artifact 根在 Buildkite checkout；前后 `ps` 快照用 PID+start-time 身份比较，泄漏则 fail closed。零 collection 不得被允许通过。 - 禁止：依赖 PID-only 比较掩盖 reuse；把 CUDA pipeline 定义当作 ROCm ready 覆盖；用邻近 green shard 宣称本 job 合同已满足。 - 验收：结构断言单次 pytest、超时信封、marker、六类 artifact 与 `process-cleanup` PASS；真实 MI300 跑通 selected node。^[PR #7398] + +## OMNI-CI-2i5 — AMD exit-status retry 只能表达有界恢复策略 + +- 触发：为 AMD READY/MERGE Simple Diffusion shards 增加 automatic retry。 +- 强制：两 suite 保持相同 `exit_status: 134, limit: 1`，Kubernetes retry 从 fresh pod 开始。 + assertion exit1、timeout124、OOM137 与其他 exit不在该 retry selector 中。 +- 禁止：将任意134诊断为GPU hang，或让源码确定性失败/冷编译timeout进入无界重跑；成功retry + 不抹去原失败证据。job-specific配置UT不应替代 generic retry/template保留与真实失败日志。 +- 验收：rendered READY/MERGE 保留唯一134 retry及既有timeout；从original/retry job日志核对 + source SHA、pod身份、exit与terminal结果，分别报告status选择和实际根因。^[PR #8520] + +## OMNI-CI-2i6 — AMD AITER cache 复用必须保留完整文件与 blocking backend 覆盖 + +- 触发：将cold-compiling diffusion attention tests迁入共享AITER lane或修改test backend override。 +- Mammoth READY 保留整个 attention file：blocking model lane 在 production `OmniDiffusionConfig` + context 下显式 TORCH_SDPA；default AITER 的完整 file 位于已有 nonblocking attention suite 后， + 共用已暖的 job cache。直接构造 DiT 时也必须进入相同 config context，并断言 env override到达 + `TransformerBlock`。不能只挪一个 node、只设 env 或只延长 timeout 来宣称避免 cold compilation； + 两条 backend/lane 分开资格化，warm default run 不代替 blocking SDPA或MERGE coverage。^[PR #8527] +- 验收：rendered argv覆盖完整file与两个backend；production config context下直接构造layer + 证明显式backend到达consumer，default backend仍命中平台默认；exact target jobs分别记录 + collection、cache复用和terminal结果。静态/缺依赖的local run不能宣称MI300 runtime pass。^[PR #8527] diff --git a/knowledge/repos/vllm-omni/ci/rules-build-compatibility.md b/knowledge/repos/vllm-omni/ci/rules-build-compatibility.md new file mode 100644 index 000000000..ff017ff59 --- /dev/null +++ b/knowledge/repos/vllm-omni/ci/rules-build-compatibility.md @@ -0,0 +1,76 @@ +--- +title: "CI 构建产物与 Python 支持合同" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, ci] +sources: ["PR #8459", "PR #7956", "PR #7985", pyproject.toml, requirements/common.txt, tools/check_wheel_package_data.py, .github/workflows/build_wheel.yml, "PR #8528", "PR #5976", "PR #6606", "PR #6674"] +confidence: high +--- + +# CI 构建产物与 Python 支持合同 + +入口与其他规则见 [CI 共享规则](rules.md)。 + + +## OMNI-CI-2e — release rebase 的 image、容器资源与实际辅助负载必须成套对齐 + +- 触发：升级 vLLM/torch base image，或新版本在 startup 增加资源检查、编译缓存与 warmup 行为。 +- 强制：正式 release 直接使用同版本 base image 的自洽依赖集；只有目标是未发布 SHA 时才恢复 + wheel 重装及其 torch/CUDA/NumPy ABI 修复。容器显式满足新 startup check（共享内存等），硬件数 + 同时计入测试辅助模型：主模型占卡时，Whisper 等 grader 需要 spare accelerator 才能避免 CPU 超时。 +- 禁止：在 release image 上保留补偿旧 image 的依赖手术；用 Docker 默认 `/dev/shm`；只按被测模型 + 卡数配置 lane 而忽略 grader；把部分 nightly green 当全部兼容问题已关闭。 +- v0.28 对齐的 release-boundary example：CUDA/CI/ROCm base image 与 installed vLLM 都指向 + `v0.28.0`；CI Dockerfile 在官方同版本 image 出现后移除临时 wheel reinstall，而不是把旧 + 0.27 image 上的 ABI repair 带入 release。文档安装 pin 也必须同步；非 release SHA 才恢复 + 明确的 wheel/ABI block。^[PR #6606] +- NPU v0.28 对齐要求 `Dockerfile.npu`、`.npu.a3`、`.npu.ci`、`.npu.ci.a3` 四个既有 image + 一起继承 matching `vllm-ascend:v0.28.0` 并显式设置 `VLLM_TARGET_DEVICE=npu`，同时保留 spawn + multiprocessing。该四-image 合同不能从 CI image 成功外推为 production、全模型或性能兼容。 + ^[PR #6674] +- v0.31 对齐时 CUDA/CI/ROCm 使用官方同版 release image；CI dependency resolution 后 + 检查 installed vLLM exact version，GPU job 开始时再验证 usable CUDA、FlashInfer 与 + `vllm._C_stable_libtorch` 导入。Omni 版本 override 使用 `VLLM_OMNI_VERSION_OVERRIDE`， + 避免通用 setuptools-scm override 污染 dependency source build 的真实版本；CPU image + builder 的 version check 不能替代 GPU runtime check。保留原 accuracy thresholds， + failed ASR 音频 artifact 只用于诊断，不能降低请求/ASR 断言。^[PR #8459] +- 验收：image tag 与目标 release 一致；startup 通过共享内存 preflight；目标 lane 证明实际 grader + device；每个残余失败独立记录，尤其数值/质量回归不能由 API 兼容测试替代。^[PR #5976] + +- XPU Dockerfile 的 `VLLM_VERSION` 默认、Intel CI build env 与安装文档显式 build arg 必须 + 与同一 checkout 目标 release 对齐；PR #8528 把三者对齐到 v0.31.0，是该 source 的 release + example，不证明所有模型/XPU hardware 兼容。审计 default 和 explicit override，不能依赖 + 文档命令碰巧修正旧 Docker 默认；published base/fallback 的来源合同仍保留。^[PR #8528] + +## OMNI-CI-2j — 声明的 Python 范围必须同时支持安装、导入与 fixture 生命周期 + +- 触发：修改 `requires-python` 所覆盖版本的依赖、typing/dataclass 表达式或 test teardown。 +- 强制：在最低声明版本验证 `.[dev]` 解析和真实受影响模块导入。3.10 的 field-less + `DuplexCommand` mixin 用显式 `__slots__ = ()`，避免 `dataclass(slots=True)` 重复继承字段， + 与另一 slotted wire base 合成时产生 layout conflict；实例仍不得获得 `__dict__`。 + `Self` 与 `Concatenate[int, ...]` 使用 backport，后者须声明 `typing_extensions>=4.13`， + 因 4.12.x 在 3.10 仍重新导出不接受 trailing ellipsis 的 `typing.Concatenate`。 +- 强制：dev PyAV 按 Python 分支解析（本 source 的 3.10 为 17.1.0，3.11+ 为 18.0.0）， + `tomllib` fallback 的 `tomli` 在 `<3.11` 显式声明；缺 TOML parser 不得静默改变 marker。 + TestClient 已关闭所属 loop 后，清空 final-output task 引用，只 cancel/await 仍在运行的 task。 +- 禁止：仅在 3.12 静态通过就声称最低版本支持；依赖传递安装碰巧给出足够新的 backport； + 对已完成、所属 loop 已关闭的 future 再从 pytest loop `gather`；把 scoped 修复说成新增版本 CI lane。 +- 验收：最低版本重现旧安装/导入失败后修复通过，并与新版本对照 command construction、payload、 + pickle 与无 `__dict__`；验证 dependency markers、typing import 和 finished/running task cleanup。 + 原 source 未新增 3.10/3.11 CI job，仍须分别记录真实版本的测试结果与无关硬件失败。^[PR #7985] + +## OMNI-CI-2k — runtime package data 必须在无 VCS 构建与安装后完整 + +- 触发：增加 runtime-owned YAML、audio/model asset、JIT source、chat template，或调整 wheel data policy。 +- 强制：`pyproject.toml` 显式枚举 runtime data families，并关闭隐式 `include-package-data`， + 使 Git file finder 不决定运行时 payload。清除旧 build/egg-info 后构建 VCS wheel；另从 + `git archive` 构建无 `.git`、无缓存 `SOURCES.txt` 的 wheel，并安装到独立 target tree。 +- 强制：比较两种 wheel 的完整 `vllm_omni/**` 文件集合；源端 deploy、distributed csrc、 + Covo speaker prompt、Step-Audio assets 与 chat templates 的非 Python 文件须同时存在于 + archive ZIP 和 installed tree 且可读取。新增 runtime data family 时同步 inventory/checker。 +- 禁止：从源码文件存在推断已安装包可用；仅验证 VCS wheel；让旧 egg-info 掩盖遗漏。 + 该 checker 比较集合和存在/可读性，不提供所有 member 的字节一致或模型运行证明。 +- 验收：归档构建能重现旧缺文件，修复后 source inventory 无遗漏、VCS/archive package + member sets 相同，installed files 可读；删一项 package-data 或引入 VCS-only 文件必须失败。 + packaging-only 证据不需要模型运行，但不能提升为 inference、JIT 编译或跨平台 runtime pass。^[PR #7956] diff --git a/knowledge/repos/vllm-omni/ci/rules-graph-buffer-fixtures.md b/knowledge/repos/vllm-omni/ci/rules-graph-buffer-fixtures.md new file mode 100644 index 000000000..4a1429cf6 --- /dev/null +++ b/knowledge/repos/vllm-omni/ci/rules-graph-buffer-fixtures.md @@ -0,0 +1,23 @@ +--- +title: "Graph 测试 buffer 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, ci] +sources: ["PR #8330"] +confidence: high +--- + +# Graph 测试 buffer 规则 + +## OMNI-CI-GRAPH-1a — fake estimator 未写的输出区域必须初始化 + +- 触发：修改 CFM graph eager/replay oracle、fake estimator 或 cache-tail fixture。 +- 强制：synthetic estimator 只写 chunk 前缀时，其他会参与 parity 比较的 cnn/attention + output 区域用确定值初始化；eager 与 graph 比较相同有效区域及定义好的 tail。 +- 禁止：用 torch.empty 的未写尾部作为 oracle，让偶发 NaN/垃圾值形成 graph 数值回归； + 把 fixture 初始化等同于真实 model cache-tail 已验证。 +- 验收：未写 tail 不含非有限值，模拟 estimator 遗漏写入时 oracle 稳定；独立真实 + estimator 测试仍要检查历史 cache 的读写 ownership。^[PR #8330] + +更一般的 fixture 生命周期查 [CI fixture 规则](rules-test-fixtures.md)。 diff --git a/knowledge/repos/vllm-omni/ci/rules-test-fixtures.md b/knowledge/repos/vllm-omni/ci/rules-test-fixtures.md index f54e75a42..06476fd28 100644 --- a/knowledge/repos/vllm-omni/ci/rules-test-fixtures.md +++ b/knowledge/repos/vllm-omni/ci/rules-test-fixtures.md @@ -1,10 +1,10 @@ --- title: "CI 并行测试与 engine fixture 合同" created: 2026-08-23 -updated: 2026-09-10 +updated: 2026-10-06 type: rule tags: [vllm-omni, ci] -sources: ["PR #3422", "PR #5074", "PR #5255", "PR #5310", "PR #5402", "PR #5524", "PR #5543", "PR #5670", "PR #5713", "PR #5780", "PR #5823", "PR #5836", "PR #5957", "PR #5976", docker/Dockerfile.ci, docker/Dockerfile.xpu, .buildkite/intel/scripts/run-xpu-test.sh, .buildkite/cuda/test-merge.yml, .buildkite/cuda/test-ready.yml, "PR #5845", "PR #5872", "PR #6008", "PR #6048", "PR #6056", "PR #6096", "PR #6102", "PR #6202", "PR #6208", "PR #6273", "PR #6293", "PR #6311", "PR #6339", "PR #6343", "PR #6468", "PR #6523", "PR #6613", "PR #6555", "PR #6650", .buildkite/common/scripts/run_cov_split.sh, pyproject.toml, tests/helpers/tests/test_mark.py, tools/pre_commit/check_test_marks.py, .buildkite/common/scripts/upload_pipeline.py, .buildkite/cuda/test-nightly.yml, .buildkite/cuda/test-weekly.yml, .buildkite/npu/test-npu-nightly.yml, .pre-commit-config.yaml, tests/helpers/clean.py, tests/helpers/client.py, tests/helpers/mark.py, tests/helpers/runtime.py, tests/helpers/stage_config.py, tests/buildkite/test_upload_pipeline.py, tests/dfx/perf/scripts/run_benchmark.py, tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/dfx/perf/tests/test_qwen_image_vllm_omni.json, tests/dfx/stability/, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_flux_kontext_expansion.py, tests/e2e/online_serving/test_minicpmo_4_5.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_expansion.py, tests/e2e/online_serving/test_qwen_image_expansion.py, tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py, tests/model_tests/diffusion/diff_model_builders.py, tests/model_tests/diffusion/model_settings.py, tests/model_tests/diffusion/test_alignment.py, tools/nightly/run_nightly_jobs.sh, tools/pre_commit/check_tts_adapter.py, tests/tools/test_check_tts_adapter.py, .buildkite/amd/scripts/bootstrap-amd-omni.sh, .buildkite/amd/test-amd-merge.yml, .buildkite/amd/test-amd-ready.yml, tests/diffusion/distributed/test_tensor_parallel.py, tests/diffusion/offloader/test_diffusion_layerwise_offload.py, "PR #6704", tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json, tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json, "PR #6743", "PR #6696", vllm_omni/benchmarks/metrics/metrics.py, vllm_omni/benchmarks/patch/patch.py, tests/benchmarks/metrics/test_metrics.py, tests/benchmarks/patch/test_patch.py, "PR #6674", docker/Dockerfile.npu, docker/Dockerfile.npu.a3, docker/Dockerfile.npu.ci, docker/Dockerfile.npu.ci.a3, "PR #6818", "PR #6830", "PR #6884", tests/diffusion/conftest.py, tests/diffusion/attention/test_flash_attn.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py, "PR #6947", "PR #7225"] +sources: ['PR #8447', 'PR #3422', 'PR #5074', 'PR #5255', 'PR #5310', 'PR #5402', 'PR #5524', 'PR #5543', 'PR #5670', 'PR #5713', 'PR #5780', 'PR #5823', 'PR #5836', 'PR #5957', 'PR #5976', docker/Dockerfile.ci, docker/Dockerfile.xpu, .buildkite/intel/scripts/run-xpu-test.sh, .buildkite/cuda/test-merge.yml, .buildkite/cuda/test-ready.yml, 'PR #5845', 'PR #5872', 'PR #6008', 'PR #6048', 'PR #6056', 'PR #6096', 'PR #6102', 'PR #6202', 'PR #6208', 'PR #6273', 'PR #6293', 'PR #6311', 'PR #6339', 'PR #6343', 'PR #6468', 'PR #6523', 'PR #6613', 'PR #6555', 'PR #6650', .buildkite/common/scripts/run_cov_split.sh, pyproject.toml, tests/helpers/tests/test_mark.py, tools/pre_commit/check_test_marks.py, .buildkite/common/scripts/upload_pipeline.py, .buildkite/cuda/test-nightly.yml, .buildkite/cuda/test-weekly.yml, .buildkite/npu/test-npu-nightly.yml, .pre-commit-config.yaml, tests/helpers/clean.py, tests/helpers/client.py, tests/helpers/mark.py, tests/helpers/runtime.py, tests/helpers/stage_config.py, tests/buildkite/test_upload_pipeline.py, tests/dfx/perf/scripts/run_benchmark.py, tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/dfx/perf/tests/test_qwen_image_vllm_omni.json, tests/dfx/stability/, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_flux_kontext_expansion.py, tests/e2e/online_serving/test_minicpmo_4_5.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_expansion.py, tests/e2e/online_serving/test_qwen_image_expansion.py, tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py, tests/model_tests/diffusion/diff_model_builders.py, tests/model_tests/diffusion/model_settings.py, tests/model_tests/diffusion/test_alignment.py, tools/nightly/run_nightly_jobs.sh, tools/pre_commit/check_tts_adapter.py, tests/tools/test_check_tts_adapter.py, .buildkite/amd/scripts/bootstrap-amd-omni.sh, .buildkite/amd/test-amd-merge.yml, .buildkite/amd/test-amd-ready.yml, tests/diffusion/distributed/test_tensor_parallel.py, tests/diffusion/offloader/test_diffusion_layerwise_offload.py, 'PR #6704', tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json, tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json, 'PR #6743', 'PR #6696', vllm_omni/benchmarks/metrics/metrics.py, vllm_omni/benchmarks/patch/patch.py, tests/benchmarks/metrics/test_metrics.py, tests/benchmarks/patch/test_patch.py, 'PR #6674', docker/Dockerfile.npu, docker/Dockerfile.npu.a3, docker/Dockerfile.npu.ci, docker/Dockerfile.npu.ci.a3, 'PR #6818', 'PR #6830', 'PR #6884', tests/diffusion/conftest.py, tests/diffusion/attention/test_flash_attn.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py, 'PR #6947', 'PR #7225', 'PR #7528', 'PR #8347'] confidence: high --- @@ -12,6 +12,13 @@ confidence: high 入口与其他规则见 [共享规则](rules.md)。 +## OMNI-CI-PLATFORM-1a — 平台参数化测试只由公开 merge 入口解析一次 + +- 触发：测试 deploy profile 在 CUDA/ROCm/NPU 等平台的 scheduler 构造，或 mock `merge_pipeline_deploy` 的平台选择。 +- 强制：mock 实际 consumer 读取的 `current_omni_platform.device_name`，加载原始 deploy 后调用 `merge_pipeline_deploy`，让该入口完成它拥有的 platform resolution；保留各 profile/platform 的能力断言。 +- 禁止：先手动 `_apply_platform_overrides`，再让 merge 按 worker 的真实平台覆盖一次；用 host 平台替代当前参数，或删掉非 host 的参数化 case 消除失败。 +- 验收：在 ROCm worker 上仍执行 CUDA 等所有参数化 profile case，检查结果对应被 mock 的平台；同一测试在另一 host 平台也满足原断言。该修复只改变 test setup，不证明新的 production 平台能力。^[PR #8447] + ## OMNI-CI-2b — 并行测试基础设施隔离任务失败和共享状态 - 触发：xdist、共享下载 fixture、长寿命 judge/transcriber worker、进程池 retry、spawned @@ -35,6 +42,17 @@ confidence: high exception cleanup。该 fixture 只覆盖 `tests/diffusion/`；Ovis 的原始 leak 若在树外运行，仍须在 source 修复。^[PR #6208] ^[PR #6339] ^[PR #6468] ^[PR #6523] ^[PR #6830] +- 当前 distributed rendezvous 必须直接复用 `vllm.utils.network_utils.get_file_store_init_method()`， + 每个 baseline/SP/group 单独调用，不再复制本地 temporary-file helper；AMD/XPU image 也须验证 + 该真实 import 可用。替换 helper 不改变 real TCP service 的 `get_open_port` 合同。^[PR #7528] +- CPU reference thread bound 是双重 opt-in：module 请求 `single_threaded_cpu` 且 item 带 `cpu`。 + fixture 放 `tests/helpers/fixtures`，由根 `pytest_plugins` 注册，过 marker guard 后才 lazy-import + Torch；snapshot 前值，把 `set_num_threads(1)` 本身放入 restoring `try`，在正常、异常及 setup + 失败时都从 `finally` 恢复。非 CPU item 不加载 Torch、不改 GPU/thread settings。 +- 验收必须覆盖真实 plugin resolution、cap/restore、non-CPU no-op 与 setup failure；重新启用 + HiFT small-chunk ROCm reference matrix 后记录其真实执行与每项 restoration。shard wall time + 只是该 run 的方向性观察，不是控制变量性能 benchmark。^[PR #8347] + ## OMNI-CI-2c — 昂贵 engine fixture 复用必须恢复状态并隔离拓扑 - 触发：为减少 checkpoint cold start，把 function-scoped engine/server 提升到 class/module diff --git a/knowledge/repos/vllm-omni/ci/rules.md b/knowledge/repos/vllm-omni/ci/rules.md index 0da9345c8..a221c3bab 100644 --- a/knowledge/repos/vllm-omni/ci/rules.md +++ b/knowledge/repos/vllm-omni/ci/rules.md @@ -1,10 +1,10 @@ --- title: "vLLM-Omni CI 规则" created: 2026-08-23 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, ci] -sources: ["PR #7648", "PR #3422", "PR #5074", "PR #5255", "PR #5310", "PR #5402", "PR #5524", "PR #5543", "PR #5670", "PR #5713", "PR #5780", "PR #5823", "PR #5836", "PR #5957", "PR #5976", docker/Dockerfile.ci, docker/Dockerfile.xpu, .buildkite/intel/scripts/run-xpu-test.sh, .buildkite/cuda/test-merge.yml, .buildkite/cuda/test-ready.yml, "PR #5845", "PR #5872", "PR #6008", "PR #6048", "PR #6056", "PR #6096", "PR #6102", "PR #6202", "PR #6208", "PR #6273", "PR #6293", "PR #6311", "PR #6339", "PR #6343", "PR #6468", "PR #6523", "PR #6613", "PR #6555", "PR #6650", .buildkite/common/scripts/run_cov_split.sh, pyproject.toml, tests/helpers/tests/test_mark.py, tools/pre_commit/check_test_marks.py, .buildkite/common/scripts/upload_pipeline.py, .buildkite/cuda/test-nightly.yml, .buildkite/cuda/test-weekly.yml, .buildkite/npu/test-npu-nightly.yml, .pre-commit-config.yaml, tests/helpers/clean.py, tests/helpers/client.py, tests/helpers/mark.py, tests/helpers/runtime.py, tests/helpers/stage_config.py, tests/buildkite/test_upload_pipeline.py, tests/dfx/perf/scripts/run_benchmark.py, tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/dfx/perf/tests/test_qwen_image_vllm_omni.json, tests/dfx/stability/, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_flux_kontext_expansion.py, tests/e2e/online_serving/test_minicpmo_4_5.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_expansion.py, tests/e2e/online_serving/test_qwen_image_expansion.py, tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py, tests/model_tests/diffusion/diff_model_builders.py, tests/model_tests/diffusion/model_settings.py, tests/model_tests/diffusion/test_alignment.py, tools/nightly/run_nightly_jobs.sh, tools/pre_commit/check_tts_adapter.py, tests/tools/test_check_tts_adapter.py, .buildkite/amd/scripts/bootstrap-amd-omni.sh, .buildkite/amd/test-amd-merge.yml, .buildkite/amd/test-amd-ready.yml, tests/diffusion/distributed/test_tensor_parallel.py, tests/diffusion/offloader/test_diffusion_layerwise_offload.py, "PR #6704", tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json, tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json, "PR #6743", "PR #6696", vllm_omni/benchmarks/metrics/metrics.py, vllm_omni/benchmarks/patch/patch.py, tests/benchmarks/metrics/test_metrics.py, tests/benchmarks/patch/test_patch.py, "PR #6674", docker/Dockerfile.npu, docker/Dockerfile.npu.a3, docker/Dockerfile.npu.ci, docker/Dockerfile.npu.ci.a3, "PR #6818", "PR #6830", "PR #6884", tests/diffusion/conftest.py, tests/diffusion/attention/test_flash_attn.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py, "PR #6947", "PR #7118", "PR #6597", "PR #7371", "PR #7534"] +sources: ['PR #8459', 'PR #7648', 'PR #3422', 'PR #5074', 'PR #5255', 'PR #5310', 'PR #5402', 'PR #5524', 'PR #5543', 'PR #5670', 'PR #5713', 'PR #5780', 'PR #5823', 'PR #5836', 'PR #5957', 'PR #5976', docker/Dockerfile.ci, docker/Dockerfile.xpu, .buildkite/intel/scripts/run-xpu-test.sh, .buildkite/cuda/test-merge.yml, .buildkite/cuda/test-ready.yml, 'PR #5845', 'PR #5872', 'PR #6008', 'PR #6048', 'PR #6056', 'PR #6096', 'PR #6102', 'PR #6202', 'PR #6208', 'PR #6273', 'PR #6293', 'PR #6311', 'PR #6339', 'PR #6343', 'PR #6468', 'PR #6523', 'PR #6613', 'PR #6555', 'PR #6650', .buildkite/common/scripts/run_cov_split.sh, pyproject.toml, tests/helpers/tests/test_mark.py, tools/pre_commit/check_test_marks.py, .buildkite/common/scripts/upload_pipeline.py, .buildkite/cuda/test-nightly.yml, .buildkite/cuda/test-weekly.yml, .buildkite/npu/test-npu-nightly.yml, .pre-commit-config.yaml, tests/helpers/clean.py, tests/helpers/client.py, tests/helpers/mark.py, tests/helpers/runtime.py, tests/helpers/stage_config.py, tests/buildkite/test_upload_pipeline.py, tests/dfx/perf/scripts/run_benchmark.py, tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/dfx/perf/tests/test_qwen_image_vllm_omni.json, tests/dfx/stability/, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_flux_kontext_expansion.py, tests/e2e/online_serving/test_minicpmo_4_5.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_expansion.py, tests/e2e/online_serving/test_qwen_image_expansion.py, tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py, tests/model_tests/diffusion/diff_model_builders.py, tests/model_tests/diffusion/model_settings.py, tests/model_tests/diffusion/test_alignment.py, tools/nightly/run_nightly_jobs.sh, tools/pre_commit/check_tts_adapter.py, tests/tools/test_check_tts_adapter.py, .buildkite/amd/scripts/bootstrap-amd-omni.sh, .buildkite/amd/test-amd-merge.yml, .buildkite/amd/test-amd-ready.yml, tests/diffusion/distributed/test_tensor_parallel.py, tests/diffusion/offloader/test_diffusion_layerwise_offload.py, 'PR #6704', tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json, tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json, 'PR #6743', 'PR #6696', vllm_omni/benchmarks/metrics/metrics.py, vllm_omni/benchmarks/patch/patch.py, tests/benchmarks/metrics/test_metrics.py, tests/benchmarks/patch/test_patch.py, 'PR #6674', docker/Dockerfile.npu, docker/Dockerfile.npu.a3, docker/Dockerfile.npu.ci, docker/Dockerfile.npu.ci.a3, 'PR #6818', 'PR #6830', 'PR #6884', tests/diffusion/conftest.py, tests/diffusion/attention/test_flash_attn.py, tests/buildkite/test_amd_pipeline.py, tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py, 'PR #6947', 'PR #7118', 'PR #6597', 'PR #7371', 'PR #7534', 'PR #8346', 'PR #8239', 'PR #8528'] confidence: high --- @@ -70,6 +70,15 @@ selector、90-minute timeout 与 explicit dependencies 只证明该 job 的配 PR 报告的 collection=1 和 local two-card B300 184.97 s 是该精确 workload 的作者报告，不能 提升为 Buildkite H100 pass、通用并发可靠性或性能 gate。^[PR #6555] +- 一个文件同时含 CPU unit 与真实 GPU kernel 时，tier/modality 可以 module-level，但 `cpu` + 只给 CPU case；GPU case 用硬件 helper，避免 module `cpu` 把它们送进 Simple shards。专用 + AR paged-attention GPU lane 设置 `VLLM_OMNI_AR_FA_REQUIRED=1`：缺可用 FlashAttention 必须 + fail，optional local run 才可 skip。READY/MERGE ROCm lane 保持相同完整数值矩阵，CUDA merge + 保留对应 lane；job-local AITER/Torch extension cache 和内层 timeout 留出 step cleanup 窗口。 +- 验收对 whole-file 按实际 CPU/GPU selectors 分别 collection，CPU 不收 GPU case、GPU 不收 + CPU case；required backend 不可用必须红，不能以所有 GPU case skip 换绿。PR #8346 的精确 + matrix 为 57 CPU/16 GPU，数字只资格化该 head，不是未来固定 census。^[PR #8346] + ## OMNI-CI-1b — 回归 fence 必须锚定可观察合同且非空转 - 触发：新增 bug regression、API guard、route census 或 middleware fence。 @@ -100,13 +109,21 @@ PR 报告的 collection=1 和 local two-card B300 184.97 s 是该精确 workload example 时，仍须保持原 marker 与测试语义。 - 禁止：把 `nightly-test` label 当 weekly CPU coverage gate；让 `NIGHTLY=1` 顺带占用 L2/L3；用 pipeline 已上传推断嵌套 group 一定执行；通过 `pytest --collect-only | grep ...` - 枚举文件却不传播 collection 失败或不检查结果非空。PR #6311 合并时 weekly TTS 循环仍有 - 这个 fail-green 缺口，不能把该 lane 的绿色结果当作完整 TTS collection 证据。 + 枚举文件却不传播 collection 失败或不检查结果非空。PR #6311 的旧 weekly TTS 动态循环曾有 + 这个 fail-green 缺口；当前显式 family inventory 与失败传播见下述 PR #8239 合同。 - 验收：对 docs-only、CI-level-only、普通 main、三种 schedule env、CUDA/NPU 与每个 PR label 逐项渲染 bootstrap，并断言 child upload、`--e2e` 和 group-level `if`；coverage helper 分别 验证 weekly-main 拆分上传与其他环境 combined/no-cov。任何动态 collection 都要让 import error、零 selection 和单文件失败使 job 红。^[PR #6311] +- 长 TTS weekly sweep 可按模型 family 拆分为有界 step，保持一文件一 pytest process、原 + env/selector/run-level；显式 file list 循环累积任何失败并最终非零退出。新增文件要审计全部 + family inventory，不能保留旧 collect-only pipeline 的 fail-green 行为。 +- tier/cadence 迁移按 case 而非整个 module：MiniCPM continuous-window 为 full_model，在 CUDA/ + NPU duplex nightly 跑并从 broad sweep 排除，另两 window case 仍 advanced_model；π0.5 标 slow + 后属于 weekly YAML 的 `NON_CRITICAL=1` group，不能仅凭文件名声称 `WEEKLY=1` 会执行它。 + 验收同时渲染 group `if`、marker 与去重路径；静态 wiring 不提供 full-checkpoint pass。^[PR #8239] + ## OMNI-CI-2a — CI 工具、schema 和 hook 是可复现供应链 - 触发：修改 pre-commit/Buildkite schema、下载 lint 工具、SPDX/import 检查或 stability marker。 @@ -130,24 +147,10 @@ PR 报告的 collection=1 和 local two-card B300 184.97 s 是该精确 workload 现有 unit test 另外直接断言 count 等于 budget，所以下降但未改 constant 时 hook 会通过， 而该 test 若被收集仍会失败；两者应统一为同一非递增语义。^[PR #6008] -## OMNI-CI-2e — release rebase 的 image、容器资源与实际辅助负载必须成套对齐 - -- 触发：升级 vLLM/torch base image，或新版本在 startup 增加资源检查、编译缓存与 warmup 行为。 -- 强制：正式 release 直接使用同版本 base image 的自洽依赖集；只有目标是未发布 SHA 时才恢复 - wheel 重装及其 torch/CUDA/NumPy ABI 修复。容器显式满足新 startup check（共享内存等），硬件数 - 同时计入测试辅助模型：主模型占卡时，Whisper 等 grader 需要 spare accelerator 才能避免 CPU 超时。 -- 禁止：在 release image 上保留补偿旧 image 的依赖手术；用 Docker 默认 `/dev/shm`；只按被测模型 - 卡数配置 lane 而忽略 grader；把部分 nightly green 当全部兼容问题已关闭。 -- v0.28 对齐的 release-boundary example：CUDA/CI/ROCm base image 与 installed vLLM 都指向 - `v0.28.0`；CI Dockerfile 在官方同版本 image 出现后移除临时 wheel reinstall，而不是把旧 - 0.27 image 上的 ABI repair 带入 release。文档安装 pin 也必须同步；非 release SHA 才恢复 - 明确的 wheel/ABI block。^[PR #6606] -- NPU v0.28 对齐要求 `Dockerfile.npu`、`.npu.a3`、`.npu.ci`、`.npu.ci.a3` 四个既有 image - 一起继承 matching `vllm-ascend:v0.28.0` 并显式设置 `VLLM_TARGET_DEVICE=npu`，同时保留 spawn - multiprocessing。该四-image 合同不能从 CI image 成功外推为 production、全模型或性能兼容。 - ^[PR #6674] -- 验收：image tag 与目标 release 一致；startup 通过共享内存 preflight；目标 lane 证明实际 grader - device；每个残余失败独立记录，尤其数值/质量回归不能由 API 兼容测试替代。^[PR #5976] +## 构建产物与 release rebase + +镜像与checkout版本、容器startup资源、辅助grader负载、Python支持范围及无VCS wheel合同， +见[构建与版本规则](rules-build-compatibility.md)（`OMNI-CI-2e`, `OMNI-CI-2j`, `OMNI-CI-2k`）。 ## OMNI-CI-3a — DFX baseline artifact 与性能回归 gate 是两个合同 diff --git a/knowledge/repos/vllm-omni/components/configuration/_index.md b/knowledge/repos/vllm-omni/components/configuration/_index.md index 5e9d78845..a704a889b 100644 --- a/knowledge/repos/vllm-omni/components/configuration/_index.md +++ b/knowledge/repos/vllm-omni/components/configuration/_index.md @@ -1,7 +1,7 @@ --- title: "vLLM-Omni Configuration" created: 2026-07-16 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, components, config] sources: ["claude-workflow-starter-private@296ea45", vllm_omni/config/] @@ -34,11 +34,12 @@ sources: ["claude-workflow-starter-private@296ea45", vllm_omni/config/] | 遇到什么 | 查看哪里 | |---|---| +| API speech_cache budgets、继承与 process-local singleton | [speech cache](rules-speech-cache.md) | | 理解配置从 deploy、CLI、默认 factory 到 structured/legacy config 的稳定边界 | [配置构造架构](architecture.md) | | 根据 PR 描述直达 strict schema、pipeline sampling constraints、deploy/topology、composable strategy、global engine args 的 owner 校验、CLI-only 负向 alias 或显存配置的规则组与第一批源码、环境变量 inventory | [配置开发门禁与代码地图](rules.md) 新增核对：CONF-3e。 | | 对象存储 URI、HF cache snapshot 或空 `config.json` 的模型名称回退与 pipeline 路由 | [模型引用路由规则](rules-model-reference-routing.md) | -| stage full-payload transport capability、topology-owned projection 或 deploy/CLI override rejection | [stage transport capability](rules-stage-transport.md) | +| stage full-payload transport capability、topology-owned projection、async-chunk 自动模式或 deploy/CLI override rejection | [stage transport capability](rules-stage-transport.md) | | diffusion Ulysses SymmMem transport flag 的 deploy/CLI/default-stage projection | [diffusion parallel transport rules](rules-diffusion-parallel-transport.md) | | diffusion attention shorthand、structured config 与 global override 的优先级和最终规范表示 | [diffusion attention precedence](rules-diffusion-attention-precedence.md) | | legacy stage engine args 的 copy boundary、connector/default injection、repeated build mutation、global engine args 的 owner 校验或 CLI-only 负向 alias | [legacy engine-args isolation](rules-legacy-engine-args.md) 新增核对：VOMNI-CFG-1u。 | @@ -50,6 +51,6 @@ sources: ["claude-workflow-starter-private@296ea45", vllm_omni/config/] | 审查或新增 process 环境变量、stage `env` 和安全诊断输出 | [environment-variable contract](environment-variable-contract.md) | | 计算多 stage 显存预算 | [deployment memory budget](deployment-memory-budget.md) | | 添加新模型和注册点 | [adding a model](adding-a-model.md) | -| composable strategy 的 axis、并行拓扑、load balancing owner 与 resolve_omni_config 统一解析 | [并行拓扑合同](rules-parallel-topology.md) | +| composable strategy、并行拓扑、VAE batch mode 的 WORLD→DP 门禁与 load balancing owner | [并行拓扑合同](rules-parallel-topology.md) | -| 冻结 topology、辅助 stage 注入、模型专用 deploy profile 与 LLM `additional_config` 投影 | [topology 与部署 profile](rules-topology-profiles.md) | +| 冻结 topology、辅助 stage、mixed V1/MRv2 runner、deploy profile 与 `additional_config` 投影 | [topology 与部署 profile](rules-topology-profiles.md) | diff --git a/knowledge/repos/vllm-omni/components/configuration/rules-parallel-topology.md b/knowledge/repos/vllm-omni/components/configuration/rules-parallel-topology.md index a98d68317..d9f81292e 100644 --- a/knowledge/repos/vllm-omni/components/configuration/rules-parallel-topology.md +++ b/knowledge/repos/vllm-omni/components/configuration/rules-parallel-topology.md @@ -1,10 +1,10 @@ --- title: "并行拓扑合同" created: 2026-09-04 -updated: 2026-09-09 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, config] -sources: [vllm_omni/config/composable_parallel/, vllm_omni/config/config_factory.py, vllm_omni/config/stage_config.py, "PR #5531", "PR #5140"] +sources: [vllm_omni/config/composable_parallel/, vllm_omni/config/config_factory.py, vllm_omni/config/stage_config.py, "PR #5531", "PR #5140", "PR #8162"] confidence: high --- @@ -71,3 +71,11 @@ confidence: high - 强制：标准与 headless 生产路径只调用 `resolve_omni_config()` 获得 `OmniConfigResolution`（structured config、runtime compatibility stages、strategy-derived LB、effective deploy path）；factory 的 create/legacy 方法是 resolver 与测试的下层 primitive，engine/entrypoint 不得绕过 resolver 直接选源。通用单 stage diffusion 仅在 registry 解析失败后作为 fallback；legacy `stage_configs_path`/`stage_configs` 必须显式报错。 - 禁止：engine 侧二次合成 default diffusion stage 或重复 merge LoRA/attention/quant/cache/profiler；把 OmegaConf-compatible stages 宣传为稳定公共 ABI；静默恢复已删除的 stage-configs-path 行为。 - 验收：同一 model+deploy+override 在标准与 headless 路径得到等价 topology 与 LB policy；覆盖 registry 命中、generic diffusion fallback、以及拒绝已删除 legacy 参数。^[PR #5140] + + +## CONF-4g — VAE batch mode 必须在最终 WORLD 拓扑上保持 DP/PP/CFG=1 + +- 触发：修改`vae_parallel_mode=batch`、diffusion/Omni并行配置校验或WORLD→DP推导。 +- 强制：两类配置都接受batch mode，但仅允许DP=PP=CFG=1；DP尚未指定时，必须在WORLD-derived DP完成后再次检查，不能把`None`当成已证明的1。runtime decoder合同见[batch VAE rules](../diffusion/rules-vae-batch.md)。 +- 禁止：batch mode与独立请求的DP或PP/CFG group组合；只检查显式DP而遗漏推导值；把TP rank数当成请求合批许可。 +- 验收：覆盖各axis>1的拒绝、有效TP topology、omittedDP推导出>1的拒绝，以及两类配置的相同mode枚举。^[PR #8162] diff --git a/knowledge/repos/vllm-omni/components/configuration/rules-speech-cache.md b/knowledge/repos/vllm-omni/components/configuration/rules-speech-cache.md new file mode 100644 index 000000000..7a065302c --- /dev/null +++ b/knowledge/repos/vllm-omni/components/configuration/rules-speech-cache.md @@ -0,0 +1,17 @@ +--- +title: "API speech cache 配置规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components] +sources: ["PR #7883"] +--- + +# API speech cache 配置规则 + +## CONF-SPEECH-1a — Speech cache budget 必须严格解析并保持 API process 的作用域 + +- 触发：修改顶层 speech_cache deploy inheritance、API 初始化或 speaker cache singleton。 +- 强制：仅允许严格非负 integer 的 resolve_max_bytes、resolve_max_entries、speaker_max_bytes；未知字段与非 Mapping 拒绝，config frozen。默认分别 4 GiB、2048、512 MiB；zero 禁用对应 cache。薄 overlay 对 speech_cache 递归合并，显式 zero 保留，其他顶层 scalar 继续替换。API 使用最终选中的 deploy（含模型默认）并记录最终 budget；显式 speaker singleton budget 冲突必须拒绝，首次初始化在 lock 内。 +- 禁止：把 bool/string 当 integer budget；替换薄 overlay 时丢掉继承项；恢复旧环境变量作为第二配置接口；把 API waveform/artifact LRU 误认为 stage engine cache、预分配内存或总 RSS/显存上限。 +- 验收：negative/bool/float/unknown/non-Mapping、inheritance+zero、默认 deploy、diffusion/LLM API 初始化、zero storage 与并发 singleton 冲突分别覆盖；检查 entry/bytes eviction，不以 RSS 必须等于 budget 为断言。 ^[PR #7883] diff --git a/knowledge/repos/vllm-omni/components/configuration/rules-stage-transport.md b/knowledge/repos/vllm-omni/components/configuration/rules-stage-transport.md index c412697ff..06368e422 100644 --- a/knowledge/repos/vllm-omni/components/configuration/rules-stage-transport.md +++ b/knowledge/repos/vllm-omni/components/configuration/rules-stage-transport.md @@ -1,10 +1,10 @@ --- title: "Stage transport capability" created: 2026-09-04 -updated: 2026-09-04 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, config, scheduler, connectors] -sources: [vllm_omni/config/stage_config.py, vllm_omni/config/omni_config.py, vllm_omni/engine/arg_utils.py, tests/config/test_omni_config.py, tests/config/test_config_factory.py, tests/engine/test_arg_utils.py, "PR #6149"] +sources: [vllm_omni/config/stage_config.py, vllm_omni/config/omni_config.py, vllm_omni/engine/arg_utils.py, tests/config/test_omni_config.py, tests/config/test_config_factory.py, tests/engine/test_arg_utils.py, "PR #6149", "PR #5099"] --- # Stage transport capability @@ -18,3 +18,10 @@ sources: [vllm_omni/config/stage_config.py, vllm_omni/config/omni_config.py, vll - 强制：完整 payload 的消费能力在 `StagePipelineConfig` 中声明，并在 legacy 与 structured 配置构造时无条件覆盖进最终 `OmniStageModelConfig`、`OmniEngineArgs` 与 `OmniModelConfig`；必须保留 `False`，且 deploy 的 flat key、`engine_args`、`engine_extras` 和 CLI 都不能伪造或覆盖该 topology 结论。 - 禁止：用 architecture/stage 名 allowlist 重新推断能力；让 free-form deploy/CLI 在 producer 校验之后改写 transport；把 async producer 的 `async_chunk` 发送职责误当作下游 full-payload wait capability。 - 验收：覆盖旧 allowlist 对应的 pipeline declarations 和一个 token-only control；三个 legacy YAML 拼写及 CLI override 均因没有 structured owner 而拒绝；从 pipeline 到最终 model config 断言 true/false 均完整保留。^[PR #6149] + +## VOMNI-CFG-1v — async-chunk 的自动模式必须保留 unset 并按 pipeline 能力解析 + +- 触发：修改 `DeployConfig.async_chunk`、CLI overlay、async-chunk processor 选择，或 legacy/structured 两路的 stage 配置构造。 +- 强制：缺省及 CLI `None` 保留自动模式；只有多 stage、存在 stage 间输入边且声明 next-stage async processor、并且没有 native AR→DiT `kv_transfer_config` 时才默认启用。显式 `False` 必须保留；显式请求单 stage、不支持的 pipeline 或 KV-transfer 冲突时必须告警并关闭。两条构造路径复用同一解析及 per-stage edge 校验。 +- 禁止：把 CLI `None` 经 `bool()` 转为禁用；对所有多 stage 默认开启；把“不支持时关闭”扩大为吞掉 producer/consumer 的不一致配置错误；让两条构造路径选择不同 processor。 +- 验收：分别覆盖 omitted、`None`、true、false、单 stage、缺 processor、KV-transfer，以及相邻 stages 模式不一致；断言最终 stage flags、sync/async processor 和应有告警，而不是只检查中间 deploy 对象。^[PR #5099] diff --git a/knowledge/repos/vllm-omni/components/configuration/rules-topology-profiles.md b/knowledge/repos/vllm-omni/components/configuration/rules-topology-profiles.md index c94da905c..c269dc004 100644 --- a/knowledge/repos/vllm-omni/components/configuration/rules-topology-profiles.md +++ b/knowledge/repos/vllm-omni/components/configuration/rules-topology-profiles.md @@ -1,10 +1,10 @@ --- title: "topology 与部署 profile 合同" created: 2026-09-04 -updated: 2026-09-10 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, config] -sources: ["PR #4222", "PR #4795", "PR #5604", "PR #5842", "PR #5885", "PR #6082", vllm_omni/config/stage_config.py, vllm_omni/config/config_factory.py, vllm_omni/engine/stage_init_utils.py, vllm_omni/engine/stage_runtime.py, tests/engine/test_async_omni_engine_stage_init.py, vllm_omni/deploy/, "PR #6186", "PR #6813", "PR #6291", "PR #6829", vllm_omni/config/pipeline_registry.py, tests/config/test_config_factory.py, tests/config/test_omni_config.py, tests/config/test_pipeline_registry.py, tests/utils/test_tracking_parser.py, "PR #6230", vllm_omni/entrypoints/utils.py, tests/entrypoints/test_serve.py, tests/entrypoints/test_utils.py, "PR #7272"] +sources: ["PR #4222", "PR #4795", "PR #5604", "PR #5842", "PR #5885", "PR #6082", vllm_omni/config/stage_config.py, vllm_omni/config/config_factory.py, vllm_omni/engine/stage_init_utils.py, vllm_omni/engine/stage_runtime.py, tests/engine/test_async_omni_engine_stage_init.py, vllm_omni/deploy/, "PR #6186", "PR #6813", "PR #6291", "PR #6829", vllm_omni/config/pipeline_registry.py, tests/config/test_config_factory.py, tests/config/test_omni_config.py, tests/config/test_pipeline_registry.py, tests/utils/test_tracking_parser.py, "PR #6230", vllm_omni/entrypoints/utils.py, tests/entrypoints/test_serve.py, tests/entrypoints/test_utils.py, "PR #7272", "PR #8184"] confidence: high --- @@ -124,3 +124,11 @@ confidence: high - 强制：LLM/AR stage 把 `additional_config` 纳入 runtime typed owner，深拷贝投影到最终 engine args；diffusion stage 继续只由 diffusion config projection 拥有该字段，runtime 侧保持 `None`。未知字段校验与跨 stage 隔离不变。 - 禁止：因 diffusion 已有同名字段就拒绝 LLM stage；让 LLM 与 diffusion 共享同一 mutable dict；在 projection 静默丢弃 NPU Code2Wav 等已声明的 LLM `additional_config`。 - 验收：LLM stage round-trip 非默认值到 engine args 且 mutation 不回写 source/邻 stage；MiniCPM-o NPU profile 读回 graph keys；diffusion stage 断言值只在 `diffusion_config.additional_config`。^[PR #7272] + + +## CONF-5m — stage runner override 必须与 native session 和平台门禁一起解析 + +- 触发：修改 `StageDeployConfig.model_runner`、runner resolution、平台overlay或 mixed V1/MRv2 pipeline。 +- 强制：stage的`v1|v2`覆盖deploy默认，legacy/structured两路把同一结果投影到`use_v2_model_runner`；native MRv2、存在upstream input的stage只允许`session_mode=turn`。平台V1 fallback清除既有stage V2 opt-in，完成platform stage overlay后再校验最终值；NPU/XPU最终含任一V2仍须拒绝。 +- 禁止：只按pipeline默认选择runner；通过free-form engine args绕过structured owner；让native下游MRv2接受streaming/duplex；在平台overlay前校验一次后漏掉stage override。 +- 验收：stage/deploy precedence、invalid runner、native turn/control与非turn拒绝、V1平台fallback及overlay后NPU/XPU拒绝都覆盖legacy/structured结果。^[PR #8184] diff --git a/knowledge/repos/vllm-omni/components/diffusion/_index.md b/knowledge/repos/vllm-omni/components/diffusion/_index.md index a65f4fa69..2b3836755 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/_index.md +++ b/knowledge/repos/vllm-omni/components/diffusion/_index.md @@ -1,7 +1,7 @@ --- title: "Diffusion" created: 2026-07-10 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, components, diffusion] sources: ["PR #5543", vllm_omni/platforms/cuda/platform.py, vllm_omni/diffusion/attention/backends/flashinfer_attn.py] @@ -9,6 +9,8 @@ sources: ["PR #5543", vllm_omni/platforms/cuda/platform.py, vllm_omni/diffusion/ # Diffusion +- [IPC 与客户端错误](rules-ipc-errors.md) — dead-peer send、abort no-op 与单 GPU status 保留。 + - 源码入口：`vllm_omni/diffusion/` 全树，含 17 个子模块：attention、cache、diffusion_kv、distributed、executor、hooks、layers、lora、model_loader、models、offloader、postprocess、profiler、quantization、sched、utils、worker - 源码校验：以上子模块均已在 `main @ 44d3ae10`（upstream `main` 快照）验证存在；MiniMax-H3 的 VAE eager ops 仍由模型 owner 维护，其他 shared/模型机制按各自规则审查 @@ -28,15 +30,21 @@ sources: ["PR #5543", vllm_omni/platforms/cuda/platform.py, vllm_omni/diffusion/ | 遇到什么 | 查看哪里 | |---|---| +| Wan exact/fused BF16预算、channels_last首帧Conv2d与cache/fallback | [Wan fastpath规则](rules-wan-fastpath.md) | +| FlashInfer FP32 per-tensor scale、TRTLLM SAGE ragged/Smooth-K 与量化环境变量 precedence | [FlashInfer quantization rules](rules-flashinfer-quantization.md) | +| TRTLLM opaque custom-op workspace/fake dtype 与实际 verified execution scope | [TRTLLM execution rules](rules-trtllm-execution.md) | +| AR-Diffusion kernel legal page、ragged sink/recent KV、scratch offset 与固定 block-table width | [AR paging geometry rules](rules-ar-paging-geometry.md) | +| Ray cluster placement、env/RPC transport、bounded teardown 与 idle failure | [Ray executor rules](rules-ray-executor.md) | | 理解共享职责和数据流 | [architecture](architecture.md) | | 根据 PR 描述直达 execution parity、checkpoint/artifact identity、quality evidence 或 system-runtime 异常清理规则组与第一批源码、custom-op mutable schema | [rules 与代码地图](rules.md) | | image/diffusion timing、step normalization、missing-versus-zero 与 benchmark 解释边界 | [metrics evidence rules](rules-metrics.md) | | vLLM/torch rebase、MoE/quant helper 漂移、kernel backend capability 与 matched accuracy | [upstream 兼容规则](rules-upstream-compat.md) | -| worker process title、拓扑局部 rank 与日志前缀 | [worker observability rules](rules-worker-observability.md) | +| worker process title、拓扑局部 rank、日志前缀与offload worker-local allocator峰值 | [worker observability rules](rules-worker-observability.md) | | 平台 IR-op priority、Inductor/eager 默认顺序、pipeline compile 准入与模型 hook 合并 | [platform runtime rules](rules-platform-runtime.md) | | CustomOp XPU platform dispatch、PyTorch-native fallback、CUDA contract 与非同义 override 边界 | [CustomOp dispatch rules](rules-custom-op-dispatch.md) | | runtime temporary 与 loader-scoped parameter dtype、shared RMSNorm accuracy | [tensor dtype rules](rules-tensor-dtype.md) | | Wan VAE height/width spatial reshard、empty tail、attention extent | [Wan spatial-shard rules](rules-wan-spatial-shard.md) | +| 完整图像 batch VAE decode、native tiling、uneven/idle rank 与 gather 顺序 | [Batch VAE decode rules](rules-vae-batch.md) | | multi-DiT、dotted `_dit_modules`、loader-to-offloader handoff 与跨 Cache-DiT/compile/LoRA/offload lifecycle、batch sampling-key 的 provided 标志 | [component lifecycle rules](rules-component-lifecycle.md) 新增核对：DIFF-2ag。 | | PEFT 与 distilled LoRA、startup fusion、delta/key/alpha、Qwen/Wan transformer mapping、PEFT suspend/resume | [LoRA rules](rules-lora.md) 新增核对：DIFF-2ag2、DIFF-2ag3。 | @@ -60,3 +68,13 @@ sources: ["PR #5543", vllm_omni/platforms/cuda/platform.py, vllm_omni/diffusion/ - [Diffusion 平台 kernel 与设备合同](rules-platform-kernels.md) 新增核对：DIFF-1ai5。 + +- [测试替身与类型契约](rules-test-contracts.md) — streaming executor、payload 收窄与实际 CPU/type 门禁。 + +- [HSDP pre-sharded 权重加载](rules-hsdp-weight-loading.md) — meta-first binding、rank-local I/O 与真实双卡 parity。 + +- [Attention execution-path 能力](rules-attention-contracts.md) — 有效 metadata、FA4 编译边界与未迁移平台。 + +shared RMSNorm 的empty-input与fused精度边界沿 tensor dtype rules 的DIFF-NORM-1a核对。 + +- [registered video SHM 与 borrowed mapping ownership](rules-video-transport.md)。 diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-ar-paging-geometry.md b/knowledge/repos/vllm-omni/components/diffusion/rules-ar-paging-geometry.md new file mode 100644 index 000000000..c045b8d8d --- /dev/null +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-ar-paging-geometry.md @@ -0,0 +1,37 @@ +--- +title: "AR-Diffusion 分页几何与可见 KV 合同" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, diffusion] +sources: ["PR #6481", vllm_omni/experimental/ar_diffusion/runner.py, vllm_omni/experimental/ar_diffusion/kv_cache/manager.py, vllm_omni/experimental/ar_diffusion/kv_cache/paged.py, vllm_omni/experimental/ar_diffusion/kv_cache/paged_attention.py, tests/diffusion/ar_diffusion/test_capability_runner.py, tests/diffusion/ar_diffusion/test_kv_cache.py, tests/diffusion/ar_diffusion/test_paged_attention.py] +confidence: high +--- + +# AR-Diffusion 分页几何与可见 KV 合同 + +本页细化 [AR resident capacity DIFF-4o](rules-system-runtime.md#diff-4o-ar-diffusion-容量必须覆盖持久状态分页图输入必须显式)；K/V独立存储和 stepwise session 生命周期仍按 [system runtime](rules-system-runtime.md) 审查。 + +## DIFF-15a — frame eviction 与 kernel page size 必须分离且保留 token 预算 + +- 触发：修改 AR-Diffusion `paging_block_size`、resolution geometry、reachable attention kernel 或 managed/scratch capacity。 +- 强制：从实际 dispatch module 查询 legal block sizes，消费 exact integer 或 `MultipleOf(base)`；frame tokens 已合法则保留原 page，否则用最细合法 page。当前可达 CUDA FA/ROCm AITER/upstream FA 回答 `MultipleOf(16)`，832×480的 1560 tokens/frame 因而分页16；不能把16写死为所有 backend 能力。空 legal set 必须拒绝。 +- 强制：以 `ceil(chunk_size / block_size)` 把 resident、in-flight、scratch 和 headroom 的 frame 项转换为 block 项，仍覆盖原 token 预算及独立 persistent reservations。kernel改变时 legal-size query 与 dispatch同时更新；FA4 head256 的128page只有实际 caller 开启 `supports_fa4_hd256=True` 才可成为约束，当前AR caller未开。 +- 禁止：用 frame size 直接当 kernel page、从GPU型号推 legal blocks、把不可达 FA4 分支施加到实际 FA2，或用细页减少 token 容量承诺。不能把“默认分辨率可构造并运行”解释成 realtime。 +- 验收：覆盖 exact/MultipleOf、多种合法集、1560→16、已对齐frame保持原page、空集与容量/预算；实际可达CUDA/ROCm分别做 numerical检查。832×480是否在旧实现可跑也依赖 kernel：最终 review 的 Hopper/FA3 baseline已能运行，不能声称所有旧路径都失败。^[PR #6481] + +## DIFF-15b — 可见 block table 必须保留每个真实 token 且绝不读 unwritten slot + +- 触发：修改 ragged AR chunk 写入、sink/recent window选择、eviction、`video_block_table` 或 action KV 拼接。 +- 强制：按实际 storage token positions选择 sink与recent的相交 blocks，按位置排序且 shared tail block去重；已evicted位置的null entries不当作KV。`kv_len`统计实际written slots，仅最后一个block可partial；前面的partial block后仍有block时 fail fast。action tokens位于video后时，video长度必须整page，否则明确拒绝。 +- 强制：保留straddling sink/recent的边界block；whole-block kernel可读边界上至多 `block_size-1` 个已写但已出window的真实旧token，不能把它们与未写slot混为一谈。24-token chunks/16page、sink1/window2的第五chunk必须保留absolute blocks0、1、4、5、6、7，不能丢block4。 +- 禁止：以可见block数×page size冒充written length、从resident list两端数block而忽略真实位置、让shape正确的action路径读partial video的空洞，或把shared tail列两次。 +- 验收：在每个chunk的scratch与commit forward，poison全部unwritten slots并与dense reference对照；覆盖fill后eviction、两端straddling、reset边界、partial中间block和partial video+action拒绝。只测第一chunk或shape不能证明滑窗正确。^[PR #6481] + +## DIFF-15c — ragged scratch、eviction 与固定宽度必须保持同一 storage 坐标 + +- 触发：修改 noncommit scratch、`visible_window_blocks`、block-table compaction、history staging 或 graph固定shape。 +- 强制：chunk从history尾的实际offset开始；noncommit可写session已拥有但尚未写入的tail slots，再接scratch pages，后续commit覆盖同slots且noncommit不得推进committed state。action scratch offset只数真正scratch pages。eviction的sink end向上取整、free end向下取整保护straddling pages；compaction只移除同时为whole chunk与whole page的gap，并保持model/storage位置关系。 +- 强制：固定video capacity为 `ceil(sink_tokens/block_size) + ceil((window_tokens + block_size - gcd(chunk_size, block_size))/block_size)`，再加action capacity；table width与`max_seq_len`来自同一block capacity，不能floor合并token数。unaligned chunk或start offset禁止复用旧history staging，必须完整restage window。 +- 禁止：scratch从slot0起导致history tail空洞、compact任意page gap改变chunk offset，或用当前table长度掩盖容量低估导致graph shape churn。不能将optional gather+reuse的既有性能承诺带到每次full restage的ragged配置。 +- 验收：覆盖30-token chunk从pos30起跨3个16page、sink/recent各自rounding、eviction后坐标与长度、scratch不提交、compaction和steady fixedwidth；1560 geometry的video capacity1756blocks，加一action reservation为1757。最终head Hopper/FA3 SP2/SP4 A/B仅证明所报standard paged workload未出现明显性能变化；optional staged served性能没有被该table测量。^[PR #6481] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-attention-contracts.md b/knowledge/repos/vllm-omni/components/diffusion/rules-attention-contracts.md new file mode 100644 index 000000000..cb83eec59 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-attention-contracts.md @@ -0,0 +1,18 @@ +--- +title: "Attention execution-path 能力规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, diffusion] +sources: ["PR #7379"] +confidence: high +--- + +# Attention execution-path 能力规则 + +## DIFF-ATTN-CONTRACT-1a — capability 必须描述实际执行路径和已证明的编译边界 + +- 触发：修改 `resolve_execution_path`、attention metadata normalization、FA4 custom op 或支持/编译模式声明。 +- 强制：resolver 使用执行时同一有效 metadata，包括 layer 加入的 KV-cache dtype、并行策略与 HSDP 外边界。FA4 的迁移声明限于 CUDA/BF16、非因果、无 mask/packing/paged-KV/piecewise/量化/并行或外边界的 dense 路径；确认 Q/K/V dtype 和 device 一致、Q/K head dim 相同并通过 FA4 自身维度 validator。fake 输出最后一维取 V，保持输出 contiguous。 +- 禁止：只看 Q 就声明兼容 Q/K/V；把 CPU vendor-kernel doubles 当作 NPU/ROCm 数值或 compiler 支持；validator 缺失仍标 `SUPPORTED`。未知 mask 语义仍是 `UNMIGRATED`，实际 UNKNOWN dispatch 可以读取 mask 值，不能宣称该路径没有 device→host 同步。 +- 验收：覆盖混合 dtype/device、非法 head dim、validator 缺失、有效 KV dtype 与各外边界；FA4 以真实 kernel/eager/FP32 SDPA 对照 fullgraph 输出并核对 fake shape；NPU/ROCm wrapper 测试通过仍保持未迁移，直到目标硬件验证完成。^[PR #7379] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-component-lifecycle.md b/knowledge/repos/vllm-omni/components/diffusion/rules-component-lifecycle.md index a83d56a34..f6dba1be5 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-component-lifecycle.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-component-lifecycle.md @@ -1,10 +1,10 @@ --- title: "Diffusion component lifecycle 规则" created: 2026-09-02 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #5720", "PR #5853", "PR #5882", "PR #5884", "PR #6486", "PR #6591", vllm_omni/diffusion/cache/base.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/cache/cachedit/runtime.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/models/interface.py, vllm_omni/diffusion/offloader/module_collector.py, vllm_omni/diffusion/offloader/startup.py, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/sched/interface.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/cache/test_cache_dit_request_runtime.py, tests/diffusion/models/sana_video/test_cache_offload.py, tests/diffusion/test_diffusion_model_runner.py, tests/diffusion/test_diffusion_scheduler.py, "PR #6070", "vllm_omni/diffusion/models/ltx2/ltx2_recipes.py", "PR #6072", "PR #7078", "PR #7047"] +sources: ["PR #5720", "PR #5853", "PR #5882", "PR #5884", "PR #6486", "PR #6591", vllm_omni/diffusion/cache/base.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/cache/cachedit/runtime.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/models/interface.py, vllm_omni/diffusion/offloader/module_collector.py, vllm_omni/diffusion/offloader/startup.py, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/sched/interface.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/cache/test_cache_dit_request_runtime.py, tests/diffusion/models/sana_video/test_cache_offload.py, tests/diffusion/test_diffusion_model_runner.py, tests/diffusion/test_diffusion_scheduler.py, "PR #6070", "vllm_omni/diffusion/models/ltx2/ltx2_recipes.py", "PR #6072", "PR #7078", "PR #7047", "PR #8453"] confidence: high --- @@ -109,7 +109,9 @@ residency 留在模型 owner。规则入口与其他共享机制仍见 [Diffusio enable boundary；该 boundary 必须 take-and-remove state exactly once。backend enable 从 hook/ staging/lease acquisition 到 initial prefetch 失败时先 quiesce and disable，再关闭仍属 loader 的 state；已提交 final-layout restore 在 preferred mode 只能以 fresh canonical model retry，required - mode 必须传播失败。worker shutdown 必须在 destroy distributed state 前 disable offloader；registered + mode 必须传播失败。worker shutdown 必须在 destroy distributed state 前关闭 offloader； + layerwise shutdown 与可再次 enable 的 disable 分开：前者清 hook/state 时不恢复整份 DiT + 到 GPU，后者仍为后续 reuse 恢复 weights。registered HWR mmap teardown 先 drain/release source references、unregister all ranges，再 close lease。若 unregister 失败，保留 registration 与 open lease（跨 fresh retry/backend GC 亦然）供 retry 或 process exit，不能 让 finalizer unmap CUDA 仍持有的页面。 @@ -118,7 +120,8 @@ residency 留在模型 owner。规则入口与其他共享机制仍见 [Diffusio - 验收：覆盖 absent/taken startup state、backend construction/enable/prefetch failure、single fresh retry 与 required no-retry；断言 partial backend disable、carrier/lease close、fresh path bypasses HWR and checkpoint mmap；另注入 unregistration failure，断言 retryable retention、backend drop 和 - distributed teardown ordering。^[PR #6486] ^[PR #6591] + distributed teardown ordering；layerwise shutdown 不调用 restore-next-block 且清全部 hooks， + 不把 mock cleanup 成功外推为每个模型退出都满足 executor grace。^[PR #6486] ^[PR #6591] ^[PR #8453] ## DIFF-2ad — batch-first sampling gate 的 provided 标志必须进入 RequestBatchSamplingParamsKey diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-flashinfer-quantization.md b/knowledge/repos/vllm-omni/components/diffusion/rules-flashinfer-quantization.md new file mode 100644 index 000000000..d9d4c0e91 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-flashinfer-quantization.md @@ -0,0 +1,36 @@ +--- +title: "FlashInfer attention quantization 合同" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, diffusion] +sources: ["PR #7431", vllm_omni/diffusion/attention/backends/flashinfer_attn.py, vllm_omni/diffusion/attention/backends/trtllm_attn.py, vllm_omni/diffusion/data.py, tests/config/test_environment_variables.py, tests/diffusion/attention/test_trtllm_attn.py] +confidence: high +--- + +# FlashInfer attention quantization 合同 + +本页细化 [FlashInfer plan 合同 DIFF-4h](rules-system-runtime.md#diff-4h-flashinfer-mixed-dtype-plan-必须绑定完整-runtime-shape-与-mask-内容)；TRTLLM packed/SP admission 仍由 [DIFF-1m](rules-attention.md#diff-1m-trtllm-packed-paddingskip-softmax-与-sage-必须保留-producersp-和-schedule-合同) 约束。 + +## DIFF-13a — FP8 per-tensor scale 必须在 FP32 中计算并被实际 kernel 消费 + +- 触发：修改 `FLASHINFER_ATTN` 的 dtype override、`_per_tensor_quantize` 或 wrapper scale 参数。 +- 强制：8-bit 浮点转换用 `scale = max(amax(abs(x)).float(), 1e-6) / finfo(dtype).max`，在 FP32 中乘 reciprocal 后才 cast；非 8-bit cast 返回 scale 1。Q/K/V 分别计算 scale，wrapper 必须消费对应 scale，输出恢复原 query dtype。现有 public override allowlist 仍为 Q/K fp16/bf16、V 另含 fp8_e4m3，不能从通用 helper 推断 Q/K FP8 override 已开放。 +- 强制：仅在 wrapper 明确抛 `NotImplementedError` 时走无 scale 重试；非单位 Q 或 K scale 必须报 unsupported，只有单位 Q/K 的 V scale 可在输出后乘回。不能吞掉其他执行错误。 +- 禁止：直接 cast FP8 冒充有 scale 的量化、在 FP16 中计算 reciprocal 导致 NaN，或无视 Q/K scale 后得到错误 softmax。当前 FlashInfer API 仍要 host scalar，`_extract_scalar().item()` 位于 compiler-disabled 边界；不得声称 device scalar 或每次调用无同步已实现。 +- 验收：覆盖零/小值、FP16 大 reciprocal、独立 Q/K/V scale、无 scale fallback 的 V-only 正例与 Q/K 拒绝、输出 dtype。硬件数值须固定实际 wrapper/backend；review 接受 host scalar 限制是当前 API 边界，后续 device-scalar 改动须另测。^[PR #7431] + +## DIFF-13b — TRTLLM SAGE 必须消费 ragged metadata、Smooth-K 与实际版本能力 + +- 触发：修改 `TRTLLM_ATTN` 的量化初始化、SAGE quantizer、ragged tail 或 causal dispatch。 +- 强制：noncausal quantized 路径解析实际 `flashinfer.trtllm_sage_attention_quantize`；可解析版本低于 `0.6.18rc10` 时 fail fast。quantizer 必须收到 Q/KV cumulative lengths 和 `smooth_k=True`，并按实际返回合同消费 scale/layout；短或非整 tile 的 ragged document 不再以 device `.item()` 检查静默禁用量化。causal 路径显式清除量化 dtype/block 配置，保持既有 dense causal 行为。 +- 强制：TRTLLM kernel dispatch 传 `skip_all_rows_active_check=True`，保留 producer 的真实 ragged 长度、packed-padding 与 SP admission；不能用去掉 host sync 的动机删除正确性 metadata。 +- 禁止：把早期 try-import symbol 当版本证明，或将未解析/缺失版本描述成当前实现已 fail closed；merged code 在该情况继续解析实际 symbol，故版本字符串不提供已验证能力证明。不得把 causal attention 当 SAGE 支持范围。 +- 验收：覆盖 known-old 版本拒绝、实际 quantizer 缺失、ragged `S % 64 != 0`、Smooth-K 参数及 causal 禁用；真实 Blackwell 对 dense reference 检验量化误差。作者所报 B200 测试属于其注明的 vLLM/build/head，不替代最终 target 的重新验证。^[PR #7431] + +## DIFF-13c — attention quant 环境变量必须服从显式 default 与 backend fallback + +- 触发：修改 `build_attention_config`、`DIFFUSION_ATTENTION_BACKEND`、`DIFFUSION_ATTENTION_QUANT` 或环境变量 inventory。 +- 强制：先 normalize config；已有 explicit default 时立即保留它。只有缺 default、backend env 存在且不为 `auto` 时才构造 env default，并按 backend-independent `AttnQuantSpec` 解析 `:[::]`。只允许 2 或 4 个非空字段；block pair `(0,0)` 表示不传 block override，其余整数 pair 交给 spec 校验。已有 per-role config 保留。 +- 禁止：将 quant env 绑定少数 backend 的特殊 allowlist、以 quant env 单独激活 backend，或让 `auto`/显式 default 被 env quant 覆盖。generic spec 能解析不等于任意 backend 都有实际量化能力。 +- 验收：覆盖 explicit default/per-role precedence、无 backend env、`auto`、两字段/四字段、零 pair、空字段与非法整数；inventory 与文档使用同一公共变量名，backend 执行另按能力合同验证。^[PR #7431] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-hsdp-weight-loading.md b/knowledge/repos/vllm-omni/components/diffusion/rules-hsdp-weight-loading.md new file mode 100644 index 000000000..11178ba47 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-hsdp-weight-loading.md @@ -0,0 +1,18 @@ +--- +title: "HSDP pre-sharded 权重加载规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, diffusion] +sources: ["PR #7948"] +confidence: high +--- + +# HSDP pre-sharded 权重加载规则 + +## DIFF-HSDP-LOAD-1a — pre-sharded 路径须先验证 binding，再分片并读取本 rank 权重 + +- 触发：修改 `hsdp_weight_load_strategy`、meta-first HSDP loader、checkpoint binding 或模型的增量 meta 初始化。 +- 强制：`full` 保持默认，`pre_sharded` 只进入支持该契约的 default diffusion loader；在权重 I/O 前检查完整的专用 transformer sources、key/shape、runtime layout 与 dtype 一致性。先对 meta 参数执行真实 HSDP 分片，再 materialize FSDP-owned local storage 和读取 safetensors 的 rank-local slices；保留 nonpersistent buffer、非 HSDP encoder/VAE 加载及 post-load hooks，并拒绝遗留 meta tensor。 +- 禁止：接受 quantization、LoRA、需要 transform 的 binding 或隐式 dtype 转换；在未适配模型上先加载整份参数再声称 pre-sharded；把一个 Cosmos checkpoint 的 host-memory 结果推广到所有模型/拓扑。 +- 验收：覆盖不完整 sources、layout/dtype mismatch、unsupported combination 和剩余 meta 的拒绝；真实两 rank Cosmos3/Edge 与 full-load 结果对齐，测试 marker 必须命中对应 CUDA 两卡 lane，不能以 CPU binding 测试替代该 parity。^[PR #7948] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-ipc-errors.md b/knowledge/repos/vllm-omni/components/diffusion/rules-ipc-errors.md new file mode 100644 index 000000000..77b90be99 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-ipc-errors.md @@ -0,0 +1,40 @@ +--- +title: "Diffusion IPC 与客户端错误规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, diffusion] +sources: ["PR #7812", "PR #8474"] +confidence: high +--- + +# Diffusion IPC 与客户端错误规则 + +## DIFF-IPC-1a — 无 peer 的 PUSH 发送不得阻塞 orchestrator，abort 必须保持 no-op 语义 + +- 触发：修改 StageDiffusionClient 的 shutdown/abort/add-request/collective-RPC 发送与死副本清理。 +- 强制：shutdown 与 abort 使用 nonblocking send；无 peer/已 engine_dead 的 abort 返回， + 让 poll 路径观察死亡并驱逐副本，不能因 cleanup raise 而杀死整个 orchestrator。 + add-request/RPC 在 NOBLOCK+POLLOUT 重试中检查实际 subprocess liveness，进程死亡时 + raise EngineDeadError；只有成功发送后登记 pending RPC。 +- 禁止：只读 monitor 的 `_engine_dead` flag 后继续 blocking send（进程可在两者之间死亡）； + 将 abort no-op 改成 raise；把 live-peer backpressure 说成无等待的成功；无 peer 测试只 + 预置 dead flag，或忘了 close socket/context 让 pytest exit 挂住。 +- 验收：dead flag 尚 false 但 subprocess 已退出时，shutdown/abort 有界返回且无异常， + add-request/RPC 明确失败；live peer 保持发送；测试实际 return/error 并完整关闭 ZMQ + context，避免只断言线程退出而漏掉错误。^[PR #7812] + +## DIFF-ERROR-1a — single-GPU 与 non-step fallback 必须保留 pipeline 的 client status + +- 触发：修改 UniProcDiffusionExecutor.execute_request、non-step runner fallback 或异常输出。 +- 强制：pipeline 异常通过 DiffusionOutput.from_exception 构造，保留 OmniClientError 的 + 4xx metadata；普通 RuntimeError 保持无 client status 并由上层作为 server error。 + 与 multiprocess direct / stepwise / engine error path 使用同一分类。 +- 禁止：只保存 str(exc) 使单 GPU 的 400/422 变成 500；因为请求失败就把 executor 标为 + 永久死亡；只检查 error message 而不检查传至 API 的 status。 +- 验收：single-GPU direct 的 422、non-step fallback 的 400 与 RuntimeError control， + 断言 status 到达输出/API，executor 仍可接下一请求；CPU/mock 不外推全部模型 GPU + admission 路径的正确性。^[PR #8474] + +共享 teardown 见 [输出生命周期](rules-output-lifecycle.md)； +公共 HTTP 错误与请求合同见 [Serving 规则](../serving/rules.md)。 diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-platform-kernels.md b/knowledge/repos/vllm-omni/components/diffusion/rules-platform-kernels.md index d1dc9a120..ea226354e 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-platform-kernels.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-platform-kernels.md @@ -1,10 +1,10 @@ --- title: "Diffusion 平台 kernel 与设备合同" created: 2026-07-20 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #4341", "PR #5001", "PR #5087", "PR #5088", "PR #5136", "PR #5255", "PR #5344", "PR #5543", "PR #5720", "PR #5737", "PR #5764", "PR #5801", "PR #5802", "PR #5838", "PR #5839", "PR #5848", "PR #5872", "PR #5881", "PR #5896", "PR #5981", "PR #6094", "PR #6102", "PR #6279", "PR #6385", "PR #6445", "PR #6651", "PR #6722", "PR #5831", "PR #6989", vllm_omni/diffusion/attention/backends/flashinfer_attn.py, vllm_omni/diffusion/attention/backends/ring/ring_kernels.py, vllm_omni/diffusion/attention/parallel/ulysses.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/data.py, vllm_omni/diffusion/distributed/hsdp.py, vllm_omni/diffusion/executor/multiproc_executor.py, vllm_omni/diffusion/layers/norm.py, vllm_omni/diffusion/layers/rope.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/model_loader/diffusers_loader.py, vllm_omni/diffusion/model_metadata.py, vllm_omni/diffusion/offloader/, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/engine/async_omni_engine.py, vllm_omni/entrypoints/openai/api_server.py, vllm_omni/quantization/component_config.py, vllm_omni/quantization/factory.py, tests/diffusion/attention/test_attention_sp.py, tests/diffusion/attention/test_ulysses_uaa.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/layers/test_norm.py, tests/diffusion/layers/test_rope_broadcast.py, tests/diffusion/offloader/test_distributed_layerwise_backend.py, tests/diffusion/test_diffusion_config_propagation.py, tests/diffusion/test_multiproc_engine_concurrency.py, "PR #4755", "PR #5990", "vllm_omni/diffusion/layers/fused_qk_norm_rope.py", "vllm_omni/diffusion/cache/teacache/extractors.py", "vllm_omni/diffusion/models/minimax_h3/minimax_h3_transformer.py", "tests/diffusion/layers/test_fused_qk_norm_rope.py", "PR #6165", "PR #5677", "vllm_omni/diffusion/quantization/hsdp_fp8.py", "tests/diffusion/quantization/test_hsdp_fp8.py", "PR #4845", "PR #6173", "PR #6070", "vllm_omni/diffusion/models/ltx2/ltx2_components.py", "vllm_omni/diffusion/model_loader/hub_prefetch.py", "PR #5910", "PR #5676", "PR #6125", "PR #6320", "PR #5877", "vllm_omni/diffusion/vllm_config.py", "PR #6283", "vllm_omni/diffusion/layers/activation.py", "PR #6281", "vllm_omni/diffusion/attention/ops/minimax_h3_modulation.py", "PR #6130", "PR #6364", "PR #6073", "PR #4820", vllm_omni/diffusion/attention/backends/fastvideo_vsa.py, "PR #6150", "PR #6410", tests/diffusion/layers/test_activation.py, tests/diffusion/layers/test_fused_qk_norm_rope_npu.py, "PR #6982"] +sources: ["PR #4341", "PR #5001", "PR #5087", "PR #5088", "PR #5136", "PR #5255", "PR #5344", "PR #5543", "PR #5720", "PR #5737", "PR #5764", "PR #5801", "PR #5802", "PR #5838", "PR #5839", "PR #5848", "PR #5872", "PR #5881", "PR #5896", "PR #5981", "PR #6094", "PR #6102", "PR #6279", "PR #6385", "PR #6445", "PR #6651", "PR #6722", "PR #5831", "PR #6989", vllm_omni/diffusion/attention/backends/flashinfer_attn.py, vllm_omni/diffusion/attention/backends/ring/ring_kernels.py, vllm_omni/diffusion/attention/parallel/ulysses.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/data.py, vllm_omni/diffusion/distributed/hsdp.py, vllm_omni/diffusion/executor/multiproc_executor.py, vllm_omni/diffusion/layers/norm.py, vllm_omni/diffusion/layers/rope.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/model_loader/diffusers_loader.py, vllm_omni/diffusion/model_metadata.py, vllm_omni/diffusion/offloader/, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/engine/async_omni_engine.py, vllm_omni/entrypoints/openai/api_server.py, vllm_omni/quantization/component_config.py, vllm_omni/quantization/factory.py, tests/diffusion/attention/test_attention_sp.py, tests/diffusion/attention/test_ulysses_uaa.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/layers/test_norm.py, tests/diffusion/layers/test_rope_broadcast.py, tests/diffusion/offloader/test_distributed_layerwise_backend.py, tests/diffusion/test_diffusion_config_propagation.py, tests/diffusion/test_multiproc_engine_concurrency.py, "PR #4755", "PR #5990", "vllm_omni/diffusion/layers/fused_qk_norm_rope.py", "vllm_omni/diffusion/cache/teacache/extractors.py", "vllm_omni/diffusion/models/minimax_h3/minimax_h3_transformer.py", "tests/diffusion/layers/test_fused_qk_norm_rope.py", "PR #6165", "PR #5677", "vllm_omni/diffusion/quantization/hsdp_fp8.py", "tests/diffusion/quantization/test_hsdp_fp8.py", "PR #4845", "PR #6173", "PR #6070", "vllm_omni/diffusion/models/ltx2/ltx2_components.py", "vllm_omni/diffusion/model_loader/hub_prefetch.py", "PR #5910", "PR #5676", "PR #6125", "PR #6320", "PR #5877", "vllm_omni/diffusion/vllm_config.py", "PR #6283", "vllm_omni/diffusion/layers/activation.py", "PR #6281", "vllm_omni/diffusion/attention/ops/minimax_h3_modulation.py", "PR #6130", "PR #6364", "PR #6073", "PR #4820", vllm_omni/diffusion/attention/backends/fastvideo_vsa.py, "PR #6150", "PR #6410", tests/diffusion/layers/test_activation.py, tests/diffusion/layers/test_fused_qk_norm_rope_npu.py, "PR #6982", "PR #8342"] confidence: high --- @@ -46,3 +46,15 @@ confidence: high - 强制：`interleaved=False` 保持 half-split `(d, d+rotary_dim/2)` 与既有 MiniMax-H3 fused geometry（`head_dim==128`、`rotary_dim==96`）；`interleaved=True` 使用相邻对 `(2i, 2i+1)`，CUDA Triton 仅在 even `rotary_dim <= head_dim <= 256` 时融合，否则 eager。consumer 的 packed `[cos|sin]` table 必须匹配所选 pairing，不得把 half-split 与 adjacent-pair frequency 混用。融合路径的 token 门槛由 op 级 env 覆盖；短于门槛时整条 eager 链必须 bit-exact。新增 env 必须进入 public inventory 与 docs。 - 禁止：把 interleaved 默认塞进 MiniMax half-split path；无门槛强制融合；只改一个 consumer 却不更新共享 op 合同或 inventory。 - 验收：分别覆盖 half-split 与 interleaved 的 fused/eager 数值、geometry 拒绝、门槛上下切换，以及 env 缺省/覆盖；Boogu 集成断言 table 去重与 `[B*S, D]` 复用。^[PR #6982] + +## DIFF-1ai6 — AuK plain graph parity 必须保持平台精度与输出所有权边界 + +- 触发：修改AuK VAE eager/graph replay test、ROCm tolerance或graph pool retirement。 +- 强制：shape/dtype相同、双方finite；NVIDIA plain graph继续exact equality，ROCm仅FP32 plain + graph使用max absolute≤1e-6、RMS≤1e-7、rtol=0。bucket/compiled fusion原有不同容差保持独立。 + 返回waveform拥有自己的storage，后续replay和整generation/pool retirement不得改旧输出。 +- 禁止：将ROCm budget扩展到NVIDIA、compiled/bucket path或任意audio quality；用宽relative + tolerance隐藏near-zero误差；CPU tile/thread调整删掉halo/stitching/输出所有权检查。 +- 验收：四GPU graph case在独立hardware lane执行；跨length、重复replay、cache满retirement后 + 逐项核对已保存输出，另测CPU tile coverage/stitching与线程finally恢复。目标job真实pass只证明 + 精确AuK matrix，不等同整AMD aggregate或模型质量/性能资格。^[PR #8342] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-ray-executor.md b/knowledge/repos/vllm-omni/components/diffusion/rules-ray-executor.md new file mode 100644 index 000000000..23a43fabf --- /dev/null +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-ray-executor.md @@ -0,0 +1,45 @@ +--- +title: "Ray diffusion executor placement 与生命周期合同" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, diffusion] +sources: ["PR #7938", vllm_omni/diffusion/executor/ray_executor.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/diffusion/stage_diffusion_proc.py, vllm_omni/diffusion/stage_diffusion_client.py, vllm_omni/engine/stage_init_utils.py, vllm_omni/engine/stage_engine_startup.py, tests/diffusion/test_ray_executor.py, docs/configuration/stage_configs.md, docs/design/module/diffusion/diffusion_runtime.md] +confidence: high +--- + +# Ray diffusion executor placement 与生命周期合同 + +通用 request/RPC/output ownership 见 [output lifecycle](rules-output-lifecycle.md)，并行拓扑见 [parallelism](parallelism.md)。 + +## DIFF-16a — Ray placement 必须按 cluster full-GPU capacity 分配并区分 global/local rank + +- 触发：修改 Ray diffusion executor startup、placement group、actor GPU、rank或rendezvous。 +- 强制：每actor请求1个完整GPU；借用current placement group时按每bundle的整数GPU容量求和，fractional bundles不能凑成one-GPU worker。自建group用每worker一GPU的`PACK` bundles，等待ready失败时立即回收；actor创建即登记，后续startup失败释放已创建actors。借用group的ownership必须保留。 +- 强制：按host连续排列global ranks，优先有GPU的driver host作rank0；每actor在自己的Ray visibility下使用local device0，global rank单独传入。单host rendezvous可用loopback，多host必须用rank0 actor IP和可达TCP port；worker只在local/env rendezvous路径设置localhost。PyTorch拥有distributed data plane，Ray拥有placement/process lifecycle。 +- 禁止：用driver local GPU数拒绝合法cluster请求，或宣称orchestrator的local device locks协调了其他host；以global rank当local device index；让半成品actors或failed-ready group泄漏。 +- 验收：覆盖multi-host ranks/rendezvous、fractional与multi-GPU bundles、startup各phase失败、ready timeout和borrowed group；非Ray保留local count与lock校验。真实多node还需验证network、GPU assignment和collective，不可由mock placement推断。^[PR #7938] + +## DIFF-16b — Ray env 与 RPC 必须保留 actor identity、reply ownership 和 CPU transport + +- 触发：修改 `ray_worker_env`、stage runtime.env、actor RPC envelopes、DP wave或结果serialization。 +- 强制：转发driver inference环境的既有prefix和显式names，stage `runtime.env`优先且可携带plugin变量；随后过滤device visibility、rank/rendezvous、worker host/side-channel地址和`RAY_*`，保持每actor自己的identity。`ray_worker_env`是内部`init=False`传输状态，不进入user config projection。 +- 强制：`exec_all_ranks`确保所有collective参与者执行；unique reply仅指定rank返回，DP wave仅每replica primary返回且按rank/DP rank恢复request顺序。返回前递归detach并移至CPU，覆盖typed `DiffusionMediaOutput`、DiffusionOutput、RunnerOutput/BatchRunnerOutput和nested containers；不能复制CUDA tensors到driver作为隐式同机路径。 +- 强制：DLO AllGather DP wave必须有一致shape/CFG/denoise/output/LoRA compatibility、相同`extra_args`及非空prompt；text-encoder AllGather还须一致positive/negative precomputed-embedding字段存在性。single request AllGather也使用bounded wave timeout。 +- 禁止：转发driver `CUDA_VISIBLE_DEVICES`/`MASTER_*`覆盖actor分配，重复返回nonprimary结果，或让不兼容请求进入可能deadlock的collective。 +- 验收：typed/untyped stage env传播和precedence、identity剥离、global/local rank、all-rank执行+one-rank reply、typed media CPU转换；覆盖DP compatibility拒绝和single-request AllGather timeout。Ray backend不使用mp的async output pumps，不能继承其IPC性能承诺。^[PR #7938] + +## DIFF-16c — Ray normal shutdown 必须先 bounded cleanup，idle failure 也必须关闭整组并通知 stage + +- 触发：修改Ray actor shutdown、monitor、health probes、RPC errors、engine close或stage teardown。 +- 强制：normal shutdown先对各actor调用`WorkerWrapperBase.shutdown()`，等待总计至多5秒，再kill actors并移除executor-owned placement group；borrowed group保留。finalizer/partial startup/failure路径可立即kill，shutdown幂等。engine close在join可能等待Ray的worker thread前先shutdown executor；stage client给subprocess至多10秒完成remote cleanup，再signal fallback。 +- 强制：独立monitor concurrency group监视actor death，health probes用独立health group；idle actor failure、RPC timeout/actor/task error或health failure使整组fail closed，释放peers与owned group并拒绝后续工作。failure callback通过`call_soon_threadsafe`到stage event loop，即使无request也触发fatal teardown；closed期间正常actor death不报新failure。被close中断的busy loop从tail退出，使pending RPC失败而非永远等待。 +- 禁止：normal shutdown直接kill跳过distributed group/KV connector/offload cleanup；仅标记failed却保留green stage health或peer资源；让首个rank失败后仍给其他可能卡在collective的rank提交新RPC。monitor thread不得强引用executor阻止finalizer。 +- 验收：覆盖drain成功/超时/异常、borrowed group、shutdown幂等、startup部分失败、idle actor death、RPC三类failure、health timeout与callback在loop建立前后、close打断inflight和signal fallback。mock tests证明控制面；实际Ray/GPU测试另证worker-owned资源释放。^[PR #7938] + +## DIFF-16d — Ray rollout 必须显式选择 backend 并限定 cluster 与性能证据 + +- 触发：修改diffusion backend默认、cluster部署文档、dependencies或multi-node性能/支持声明。 +- 强制：Ray仅在`distributed_executor_backend=ray`时选择；省略仍单GPU `uni`、多GPU `mp`，显式single-GPU mp保留，`external_launcher`仍未支持。复用已初始化Ray或调用`ray.init()`并由Ray处理`RAY_ADDRESS`，没有diffusion独有address参数；每node安装依赖并在Omni前启动cluster，executor不替用户在其他host启动Ray。 +- 禁止：从一个Cosmos3/Ulysses示例或mock tests声称任意pipeline、topology、quality或compute-bound workload普遍加速；把作者报34→20秒的4→8GB200 scale-up当作隔离软件改动的same-topology A/B。旧head static review和CI资源失败归因也不是最终runtime验证。 +- 验收：文档和实际config默认/显式路径一致，缺Ray报可操作错误；真实multinode记录model、head/base、node/GPU、parallel config、request、warmup/repeats、latency spread与quality等价，E2E claim另给stage attribution。作者的bounded test suite和example保留其原scope，不扩展为完整support矩阵。^[PR #7938] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-tensor-dtype.md b/knowledge/repos/vllm-omni/components/diffusion/rules-tensor-dtype.md index 983e88e7f..7e35993af 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-tensor-dtype.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-tensor-dtype.md @@ -1,10 +1,10 @@ --- title: "Diffusion tensor dtype 规则" created: 2026-09-03 -updated: 2026-09-04 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #5067", "PR #5068", "PR #5174", "PR #5981", "Issue #5880", vllm_omni/diffusion/layers/norm.py, tests/diffusion/layers/test_norm.py, "PR #6070", "vllm_omni/diffusion/models/ltx2/ltx2_latents.py"] +sources: ["PR #5067", "PR #5068", "PR #5174", "PR #5981", "Issue #5880", vllm_omni/diffusion/layers/norm.py, tests/diffusion/layers/test_norm.py, "PR #6070", "vllm_omni/diffusion/models/ltx2/ltx2_latents.py", "PR #7489"] confidence: high --- @@ -36,3 +36,11 @@ confidence: high - 禁止：只对 LTX-2.5 应用 token-major layout；在 BF16 mask 上提前乘 `noise_scale`；直接用 CUDA scalar tensor 做 sigma 除法；让 padding 或共享全局 RNG 改变请求的随机状态；仅凭 shape 和可运行性宣称数值 parity。 - 验收：对 2、2.3、2.5 参数化 seeded generated/provided latent 测试，逐值核对 layout、stride、后续 RNG 状态、FP32 noising 和 sigma arithmetic；ancestral step 再与固定官方 reference 对照，硬件 parity 必须固定 attention backend 与 checkpoint revision。^[PR #6070] + + +## DIFF-NORM-1a — shared RMSNorm 必须对 empty stream 保持 native fallback 与明确精度边界 + +- 触发：修改sharedRMSNorm的CUDA/HIPdispatch或MammothModa2caption/refiner/QKnorm替换。 +- 强制：compile或x.numel()==0走native，保留residual处理及fused异常fallback；替换Qwen2RMSNorm保持checkpoint参数名/shape/eps，包括captionembedder重建。共享FP32accumulation仍由DIFF-1c拥有。 +- 禁止：对CFG无条件branch的zero-tokenstreamlaunch fusedkernel；将一次rounding的shared/fused结果宣称与先cast后weightmultiply的Qwen2路径bit-identical；fallback结果冒充真实fused精度证据。 +- 验收：CPU/acceleratorempty、residual、compilecontrol及checkpointload覆盖；真实Previewhidden/QKshape验证fusedpath被调用、相对旧路径一BF16step界及独立float64referenceerror；模型parity/性能不能从单kernel测试外推。 ^[PR #7489] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-test-contracts.md b/knowledge/repos/vllm-omni/components/diffusion/rules-test-contracts.md new file mode 100644 index 000000000..8e95f2977 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-test-contracts.md @@ -0,0 +1,18 @@ +--- +title: "Diffusion 测试替身与类型契约" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, diffusion] +sources: ["PR #8418"] +confidence: high +--- + +# Diffusion 测试替身与类型契约 + +## DIFF-FIXTURE-1a — streaming executor 替身须保留实际消费的状态与类型 + +- 触发：修改 `_PipelineBackedEngine`、executor liveness watcher 或 streaming output 测试的 payload/queue helper。 +- 强制：替身提供被实际路径读取的 `is_dead` 初值、failure callback 与 health-check 接口，保留已有 streaming 断言和 CPU collection。修复类型收窄时先绑定 `dict.get()` 结果再 `isinstance`，非 dict payload/metadata 仍返回空 dict；queue 构造的类型参数与其真实输出一致。 +- 禁止：遗漏 watcher 读取的属性；以删除 streaming 断言、改 CI 路由或跳过变更文件 mypy 解决 fixture 失败；由这次类型改写推断其他合法写法一概错误。固定 `is_dead=False` 只覆盖存活路径，不能证明 fatal teardown 已被测试。 +- 验收：目标 streaming module 的 `core_model and cpu` pytest 与变更文件 mypy 门禁通过；若同时宣称死亡处理 coverage，另使替身进入死亡分支并断言 `DIFFUSION_PROC_DEAD` 与 teardown。^[PR #8418] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-trtllm-execution.md b/knowledge/repos/vllm-omni/components/diffusion/rules-trtllm-execution.md new file mode 100644 index 000000000..41c0ab46e --- /dev/null +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-trtllm-execution.md @@ -0,0 +1,29 @@ +--- +title: "TRTLLM custom op 与执行能力合同" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, diffusion] +sources: ["PR #7214", vllm_omni/diffusion/attention/backends/trtllm_attn.py, tests/diffusion/attention/test_trtllm_attn.py, tests/diffusion/attention/test_trtllm_contract.py, docs/design/feature/attention_execution_contract_poc.md] +confidence: high +--- + +# TRTLLM custom op 与执行能力合同 + +通用 attention admission 见 [attention rules](rules-attention.md)；量化数值见 [FlashInfer quantization](rules-flashinfer-quantization.md)。 + +## DIFF-14a — TRTLLM custom op 必须封装 JIT I/O 并准确声明 workspace 与 fake output + +- 触发：修改 `vllm_omni::trtllm_ragged_attention` 注册、FlashInfer JIT/cubin 调用、fullgraph compile 或 fake implementation。 +- 强制：真实 FlashInfer dispatcher 的 cached JIT、cubin/file I/O 位于 opaque custom-op 内；schema 仅声明 `workspace_buffer` mutation，Q/K/V、sequence metadata 和 scale tensors 保持输入合同。fake output shape 为 `(*query.shape[:-1], value.shape[-1])`，在 query device；FP8 e4m3/e5m2 与 INT8 query 返回 BF16，其余保留 query dtype。 +- 强制：实际 dispatcher 与 fake shape/dtype 一致；SAGE scale/block metadata、skip-softmax sentinel、causal 与 ragged lengths 通过显式参数传入，不能把 Python I/O重新暴露给编译图。 +- 禁止：fake 无条件用 query dtype、把 workspace mutation 藏在纯函数 schema 中，或因 trace 成功就假定不同 dtype/VO head dimension 输出正确。 +- 验收：编译测试中 FlashInfer I/O trap 不逃入 fullgraph，fake/real 输出 shape、device、BF16低精度输出一致；覆盖 workspace mutation、非等 QK/VO dimension 与 scale 参数。真实数值另用对应硬件/kernel验证，CPU fake tests 只证明 schema。^[PR #7214] + +## DIFF-14b — TRTLLM verified execution 必须来自已初始化 backend 的真实输入 + +- 触发：修改 TRTLLM `resolve_execution`、capability metadata、backend construction、packed metadata 或 compile admission。 +- 强制：pre-construction query 清除 caller 自填的 kernel variant，返回 `UNMIGRATED`。初始化后由实际 quant/skip/packed 路径和真实 query dtype、causal、piecewise、metadata 构造合同。verified `CUSTOM_OP` 只覆盖 dense、noncausal、BF16、head size 128、equal Q/K/V heads、CUDA SM100/SM103；parallel、paged KV、piecewise 和 outer HSDP/compile boundary 没有该验证承诺。 +- 强制：Q/K/V rank4、dtype/device、K/V shape、batch/head dimension 与非零尺寸必须一致；`attn_mask` 拒绝；四个 packed key 要么齐全要么皆无，`PackedPaddingMetadata` 另要求完整 packed metadata。missing FlashInfer 或非法真实输入为 unsupported；SAGE、skip-softmax、packed、其他架构或未验证 geometry 保持 `UNMIGRATED`。 +- 禁止：把 caller 声明、成功 import 或 uninitialized backend 标成 verified，或把 advisory `UNMIGRATED` 当作关闭普通执行/compile 的新硬门禁。不能把早期 Blackwell测试、Cosmos parallel timing 或 GH200 上 Blackwell skips外推成最终 parallel/Blackwell合同证明。 +- 验收：覆盖 construction 前后、SM100/SM103 与 SM90/SM120、actual config 覆盖 caller hints、dtype/shape/mask/partial metadata 拒绝和 unverified path 矩阵。最终 supported scope 须在对应 Blackwell上执行 custom op 与 dense reference；静态架构 guards 不提供硬件数值证据。^[PR #7214] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-upstream-compat.md b/knowledge/repos/vllm-omni/components/diffusion/rules-upstream-compat.md index 5eeedb7d0..b8c5dac7f 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-upstream-compat.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-upstream-compat.md @@ -1,15 +1,31 @@ --- title: "Diffusion upstream 兼容规则" created: 2026-09-02 -updated: 2026-09-05 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #5976", vllm_omni/diffusion/compile.py, vllm_omni/diffusion/layers/fused_moe.py, vllm_omni/diffusion/models/diffusers_adapter/pipeline_diffusers_adapter.py, vllm_omni/quantization/_copy_missing_attrs.py, tests/diffusion/test_compile.py, tests/e2e/accuracy/test_qwen_image.py, "PR #6287", "PR #6273"] +sources: ["PR #8459", "PR #5976", vllm_omni/diffusion/compile.py, vllm_omni/diffusion/layers/fused_moe.py, vllm_omni/diffusion/models/diffusers_adapter/pipeline_diffusers_adapter.py, vllm_omni/quantization/_copy_missing_attrs.py, tests/diffusion/test_compile.py, tests/e2e/accuracy/test_qwen_image.py, "PR #6287", "PR #6273"] confidence: high --- # Diffusion upstream 兼容规则 +## DIFF-RELEASE-FP8-1a — online FP8 与 serialized checkpoint 必须由不同 config 接纳 + +- 触发：升级 upstream quantization API、diffusion `fp8` factory 或 checkpoint config reconciliation。 +- 强制：未序列化 BF16/FP16 权重的既有 `fp8` 入口使用 `DiffusionFp8Config` 对接 + native `OnlineQuantizationConfig`，保留 linear/MoE 的 per-tensor static weight spec 与 + ignored layers；checkpoint 已 serialized 时使用 upstream `Fp8Config`，磁盘 metadata + 可经现有 reconciliation 替换 online config。method 名仍为 `fp8`，不能因此混用 config。 +- 强制：online compatibility config 只接纳 dynamic activation；拒绝 serialized flag、 + block-wise weight size 与 store dtype。upstream online shorthand 经 canonical resolver + 生成 `QuantizationConfigArgs`；显式 native args 保持原对象，无可用 args 时报错。 +- 禁止：向新版 serialized-only `Fp8Config` 传 online 用法；为了构造成功静默忽略 + activation/block/store 选项；把 native kernel smoke 当作量化音视频质量已合格。 +- 验收：online/serialized 两路、ignored layers、显式 args identity、无 args 与非法 + options；GPU 用真实 BF16 weights 执行 native online kernel，模型质量与不同 FP8 + layout/HSDP 另按对应 owner 合同验证。^[PR #8459] + ## DIFF-4k — upstream API shim 与 accuracy backend 必须按能力而非名字对齐 - 触发：vLLM/torch bump 删除或重命名 MoE/quant/platform API，或 kernel hub 没有当前 image build。 diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-vae-batch.md b/knowledge/repos/vllm-omni/components/diffusion/rules-vae-batch.md new file mode 100644 index 000000000..d3abe7fbd --- /dev/null +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-vae-batch.md @@ -0,0 +1,17 @@ +--- +title: "Batch VAE decode 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, diffusion, distributed] +sources: ["PR #8162", vllm_omni/diffusion/distributed/autoencoders/autoencoder_kl.py, vllm_omni/diffusion/distributed/autoencoders/autoencoder_kl_flux2.py, vllm_omni/diffusion/distributed/autoencoders/distributed_vae_executor.py] +--- + +# Batch VAE decode 规则 + +## DIFF-VAE-BATCH-1a — 完整图像 chunk 必须保持 native decode 与原 batch 顺序 + +- 触发：修改 AutoencoderKL/Flux2 的 batch parallel decode、split/merge 或 VAE factory routing。 +- 强制：配置通过 [CONF-4g](../configuration/rules-parallel-topology.md) 的 WORLD 拓扑门禁后，只沿 batch 轴分配完整图像的连续 chunk。每个 active rank 调用 native decoder，按原 index gather 并 broadcast 完整 batch；保留本地 Diffusers slicing/tiling。B1、degree≤1 或 distributed 未初始化时走 native decode；unsupported VAE class 在初始化拒绝。 +- 禁止：在 callback 中调用已包装的 `self.decode` 造成递归；把独立请求拼成 batch；将 batch mode 禁用空间分布式执行误写为禁用本地 tiling；让 idle rank 解码伪造图像或改变输出顺序。 +- 验收：KL/Flux2 覆盖 B1、uneven batch、idle rank、degree1、未初始化以及 tiling/slicing on/off；所有 rank 输出与 native reference 对齐，dtype、return_dict/tuple、图像顺序和 factory 支持边界不变。^[PR #8162] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-video-transport.md b/knowledge/repos/vllm-omni/components/diffusion/rules-video-transport.md new file mode 100644 index 000000000..d0fa95b00 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-video-transport.md @@ -0,0 +1,24 @@ +--- +title: "Generated-video IPC ownership 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components] +sources: ["PR #7355"] +--- + +# Generated-video IPC ownership 规则 + +## DIFF-VIDEO-IPC-1a — Registered SHM 必须限定 video 并保留完整 cleanup 所有权 + +- 触发：修改 enable_registered_shm、diffusion pack/unpack、video_output_index 或 CUDA注册失败处理。 +- 强制：开关默认 false 且必须 bool；仅 async NVIDIA CUDA D2H stream 的 typed video/明确标记 legacy video tuple 项可注册，其他媒体、latents、audio、CPU/HIP与sync保持原路。等待 producer event，再将原 bits copy 到注册 SHM，BF16用 uint16 view运输并view恢复。完成copy才unregister/发布handle；pack按完整成功才提交字段，失败unlink本次ownedsegments。CUDAcleanup不确定时保留mapping/source/stream引用到worker退出并拒绝新注册，不能unmap仍在用的地址。 +- 禁止：把去掉hostcopies说成没有D2H；widen BF16在GPU额外分配FP32；提前清registered flag让失败重试遗漏；把一项video标记推广整个(video,audio)tuple；1MB routing门槛不是普遍性能阈值。 +- 验收：video/nonvideo、marked/unmarked、BF16bits、CPU/HIP/sync、register/copy/unregister失败和多字段pack失败覆盖；copy完成前mapping仍live，失败不发布半包。大小payload分开评估，不能从历史大video样本宣称当前主线普遍加速。 ^[PR #7355] + +## DIFF-VIDEO-IPC-1b — Borrowed receiver view 必须持有 mapping 到最后一个消费者 + +- 触发：修改 borrow_on_unpack、NumPy/Torch view 或 SHM name unlink。 +- 强制：借用view以base owner携带SHM lease，unlink名称后mapping仍活；np.asarray、slice和torch.from_numpy继续持有owner直到最后消费者释放。普通路径仍copy后close/unlink；BF16view保留bitwidth/dtype。 +- 禁止：只在ndarray subclass属性挂owner而被np.asarray丢掉；unlink立即close仍在消费的mapping；跨进程假定Python引用自动转移生命周期。 +- 验收：删除原array后保留slice/torchview并消费、最后引用释放、fallbackcopy与dtypeview回归；同RGB和完整decodeMP4内容对照，不能仅校验descriptor。 ^[PR #7355] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-wan-fastpath.md b/knowledge/repos/vllm-omni/components/diffusion/rules-wan-fastpath.md new file mode 100644 index 000000000..6ef2640af --- /dev/null +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-wan-fastpath.md @@ -0,0 +1,41 @@ +--- +title: "Wan VAE fastpath 精度与首帧合同" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, diffusion] +sources: ["PR #8188", "PR #8179", tests/diffusion/distributed/test_wan_decoder_fast_path.py, vllm_omni/diffusion/distributed/autoencoders/wan_vae_fastpath/forwards.py, tests/diffusion/distributed/test_wan_vae_fastpath_kernels.py, benchmarks/diffusion/bench_wan_vae_decode.py] +confidence: high +--- + +# Wan VAE fastpath 精度与首帧合同 + +共享入口见 [Diffusion 规则](rules.md)；分片边界见 [Wan spatial-shard](rules-wan-spatial-shard.md)。 + +## DIFF-3d — Wan decoder exact 与 fused 的数值预算必须按精度边界分别验证 + +- 触发：修改 decoder fastpath 的 exact/fused level、RMSNorm+SiLU cast 边界或 parity assertions。 +- 强制：exact level 保持 `torch.equal`；fused FP32 保持 `atol=1e-5, rtol=1e-4`。 + fused BF16 在 norm-affine/SiLU 后提前写回 BF16，使用绝对 max error ≤ 四个 BF16 eps + （0.03125）和整体 RMS error ≤ 一个 eps（0.0078125），两项必须同时满足。 +- 禁止：把 fused FP32 称为 bit-identical；只用 near-zero relative tolerance 或只检查平均误差， + 使极端像素或广泛漂移漏过；将该 tanh-bounded decoder 合同扩展到任意模型/量化质量。 +- 验收：相同 reference/candidate latents 驱动同 session 的两个 chunk 与 fresh session，分别 + 覆盖 local 和 two-rank sharded decode、FP32/BF16、exact/fused，并检查 cache 从 zero→history→zero。 + CPU/plumbing 与 exact AMD target case pass 需分开记录，邻近 CI 红/绿不改变本数值合同。^[PR #8188] + +## DIFF-3e — 首帧 Conv2d 只能替换无历史的受支持 channels_last 因果卷积 + +- 触发：优化 `wan_vae_fastpath/forwards.py` 的 first-frame causal Conv3d、cache 或 deferred bias。 +- 强制：只在既有 CUDA inference path、`channels_last` opt-in、T=1、无历史 payload、受支持 + dtype、kernel `(3,3,3)`/`(3,1,1)`、unit stride/dilation、groups=1、zero native padding 与 + 精确 causal `_padding` 时用 Conv2d。有限输入 `[0,0,x]` 只读取最后 temporal weight slice； + 每次取当前 weight 并在该次调用 packing，不能缓存 reload/offload 后陈旧的 GPU copy。 +- 强制：cache 更新继续保存未被 autocast 舍入的输入与前导零，后续 T=1 有历史时恢复 Conv3d； + channels-last output、deferred bias 的 activation dtype、fallback 与 cache ownership 保持原合同。 +- 禁止：把该实数等价说成 floating-point bit identity；在 lossless、multi-frame、非支持 padding/ + groups/stride/dilation 或有历史时应用优化；只测试第一帧而漏掉下一帧 cache。 +- 验收：GPU 覆盖首帧和 cached 后续帧、None/Rep 起始、dtype/autocast/bias、unsupported fallback + 与 weight reload。性能用同 head、同 VAE 的 `--first-frame-ablation` 仅切此 helper，保留实际 + 命中计数、ABBA samples、warmup、packing 成本、峰值和输出差异；单卡首帧/整段 VAE 结果 + 分别报告，不外推完整生成 latency 或其它硬件。^[PR #8179] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules-worker-observability.md b/knowledge/repos/vllm-omni/components/diffusion/rules-worker-observability.md index 1bc4814a1..e8829e5df 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules-worker-observability.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules-worker-observability.md @@ -1,10 +1,10 @@ --- title: "Diffusion worker observability rules" created: 2026-09-05 -updated: 2026-09-05 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, diffusion] -sources: ["PR #6722", vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/diffusion/distributed/parallel_state.py, tests/diffusion/test_diffusion_worker_process_title.py, tests/diffusion/distributed/test_expert_parallel_layout.py] +sources: ["PR #6722", vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/diffusion/distributed/parallel_state.py, tests/diffusion/test_diffusion_worker_process_title.py, tests/diffusion/distributed/test_expert_parallel_layout.py, "PR #8189"] confidence: high --- @@ -16,3 +16,16 @@ confidence: high - 强制：early startup 保留通用 `DiffusionWorker`；`initialize_model_parallel()` 完成后、模型加载前，按 DP、PP、SP、CFG、TP、FS、RP、EP 顺序从 active group 的 `rank_in_group` 追加非 singleton 维度，并把同一名字同时交给 `set_process_title(..., prefix="vLLM-Omni")` 和 `decorate_logs()`。FS/RP 只在 HSDP 查询，RP 只在多 replica layout 查询；EP 只在 expert parallel enabled 时查询。 - 禁止：在 parallel state 可用前读 group；用 global rank 代替 topology local rank；让非 HSDP/非 EP 路径访问 FS/RP/EP；只更新 OS title 或日志之一；因 optional `setproctitle` 不可用让 worker 启动失败。 - 验收：覆盖初始化前 generic 名称且不读 group、singleton、省略规则、DP/PP/SP/CFG/TP、FS、multi-replica RP、conditional EP、缺少 `setproctitle` 仍装饰日志，以及 Linux `ps` 可见 title；本提交的验证边界为 unit/CPU Gloo，未提供本地 GPU 证据。^[PR #6722] + +## DIFF-17a — offload memory A/B 必须在实际 worker 内测 process-local allocator peak + +- 触发：修改layerwise-offload memory assertion、collective RPC probe或共享GPU上的memory归因。 +- 强制：model构造后在diffusion worker内synchronize并reset allocator peak，完成generate后再次 + synchronize并读取max_memory_allocated。用probe+operation过滤嵌套RPC返回，缺reset ack或 + snapshot必须失败；比较各次reported worker peak，仍保留offload状态和audio等价checks。 +- 禁止：用parent的device `total-free`减initial来归因model saving，混入同卡sibling jobs；将 + device-global轮询或reset其他进程的allocator当有效替代。RPC当前只返回rank0 payload，取返回 + 结果max不能提升为所有ranks的max，更不能称为total device capacity或reserved-memory峰值。 +- 验收：CPU覆盖nested/wrong probe/wrong operation/缺结果；独立inference process以相同 + workload测baseline/offload，记录worker-local MB与真实启用状态，必要时反序复测filesystem + cache影响。多rank全体峰值须先增加明确all-rank返回/归约，不能从单卡source外推。^[PR #8189] diff --git a/knowledge/repos/vllm-omni/components/diffusion/rules.md b/knowledge/repos/vllm-omni/components/diffusion/rules.md index 8d16627f1..b9d9c8986 100644 --- a/knowledge/repos/vllm-omni/components/diffusion/rules.md +++ b/knowledge/repos/vllm-omni/components/diffusion/rules.md @@ -1,7 +1,7 @@ --- title: "Diffusion 共享规则" created: 2026-07-20 -updated: 2026-09-21 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, diffusion] sources: ["PR #7648", "PR #4341", "PR #5001", "PR #5087", "PR #5088", "PR #5136", "PR #5255", "PR #5344", "PR #5543", "PR #5720", "PR #5737", "PR #5764", "PR #5801", "PR #5802", "PR #5838", "PR #5839", "PR #5848", "PR #5872", "PR #5881", "PR #5896", "PR #5981", "PR #6094", "PR #6102", "PR #6279", "PR #6385", "PR #6445", "PR #6651", "PR #6722", "PR #5831", "PR #6989", vllm_omni/diffusion/attention/backends/flashinfer_attn.py, vllm_omni/diffusion/attention/backends/ring/ring_kernels.py, vllm_omni/diffusion/attention/parallel/ulysses.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/data.py, vllm_omni/diffusion/distributed/hsdp.py, vllm_omni/diffusion/executor/multiproc_executor.py, vllm_omni/diffusion/layers/norm.py, vllm_omni/diffusion/layers/rope.py, vllm_omni/diffusion/lora/manager.py, vllm_omni/diffusion/model_loader/diffusers_loader.py, vllm_omni/diffusion/model_metadata.py, vllm_omni/diffusion/offloader/, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/engine/async_omni_engine.py, vllm_omni/entrypoints/openai/api_server.py, vllm_omni/quantization/component_config.py, vllm_omni/quantization/factory.py, tests/diffusion/attention/test_attention_sp.py, tests/diffusion/attention/test_ulysses_uaa.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/layers/test_norm.py, tests/diffusion/layers/test_rope_broadcast.py, tests/diffusion/offloader/test_distributed_layerwise_backend.py, tests/diffusion/test_diffusion_config_propagation.py, tests/diffusion/test_multiproc_engine_concurrency.py, "PR #4755", "PR #5990", "vllm_omni/diffusion/layers/fused_qk_norm_rope.py", "vllm_omni/diffusion/cache/teacache/extractors.py", "vllm_omni/diffusion/models/minimax_h3/minimax_h3_transformer.py", "tests/diffusion/layers/test_fused_qk_norm_rope.py", "PR #6165", "PR #5677", "vllm_omni/diffusion/quantization/hsdp_fp8.py", "tests/diffusion/quantization/test_hsdp_fp8.py", "PR #4845", "PR #6173", "PR #6070", "vllm_omni/diffusion/models/ltx2/ltx2_components.py", "vllm_omni/diffusion/model_loader/hub_prefetch.py", "PR #5910", "PR #5676", "PR #6125", "PR #6320", "PR #5877", "vllm_omni/diffusion/vllm_config.py", "PR #6283", "vllm_omni/diffusion/layers/activation.py", "PR #6281", "vllm_omni/diffusion/attention/ops/minimax_h3_modulation.py", "PR #6130", "PR #6364", "PR #6073", "PR #4820", vllm_omni/diffusion/attention/backends/fastvideo_vsa.py, "PR #6150", "PR #6410", tests/diffusion/layers/test_activation.py, tests/diffusion/layers/test_fused_qk_norm_rope_npu.py] @@ -51,6 +51,7 @@ confidence: high | loader-owned offload plan、backend enable/prefetch failure、fresh canonical retry | `checkpoint-distributed`：`DIFF-12a` | `offloader/startup.py` → `offloader/__init__.py::enable_offload_backend` → runner | | LPIPS/PSNR/相似度阈值、CPU offload、量化质量证据 | `quality-evidence`：`DIFF-3a` | changed exact case → runner `execute_model` → model pipeline；A/B 同路径 | | Wan VAE spatial shard、gather/trim/reshard、empty tail、attention extent | `distributed-vae`：`DIFF-3b` | `distributed/autoencoders/wan_spatial_shard.py` → patched decoder attention/conv → gather final frame | +| 完整图像 batch VAE decode、native tiling 与 gather 顺序 | [DIFF-VAE-BATCH-1a](rules-vae-batch.md) | `distributed/autoencoders/autoencoder_kl.py` batch split → native decode → ordered gather | | paged KV/cache、backend/platform、GQA/layout、Ring/Ulysses、SymmMem A2A、FlashInfer quant、masked cross-attention、预算与 admission | `system-runtime`：`DIFF-4a`–`4j`, `DIFF-4v`, `DIFF-4x` | engine init → metadata/config → attention parallel/backend → platform hook → scheduler/serving | | worker/RPC 异常、rank-status、traceback/device cache 清理 | `system-runtime`：`DIFF-4d`；reply ownership 见 [output/runtime](rules-output-lifecycle.md) `DIFF-4z` | `diffusion_worker.py::{_execute_rpc,_worker_busy_loop}` 的 execute/reply/status 路径 | | worker title、log prefix、DP/PP/SP/CFG/TP/FS/RP/EP local rank | [worker observability](rules-worker-observability.md)：`DIFF-4w` | `diffusion_worker.py::_setup_diffusion_worker_proc_title_and_log_prefix` → `distributed/parallel_state.py` group getters | diff --git a/knowledge/repos/vllm-omni/components/distributed/_index.md b/knowledge/repos/vllm-omni/components/distributed/_index.md index 34df5642b..aaecad74a 100644 --- a/knowledge/repos/vllm-omni/components/distributed/_index.md +++ b/knowledge/repos/vllm-omni/components/distributed/_index.md @@ -55,3 +55,6 @@ sources: [vllm_omni/distributed/omni_connectors/, vllm_omni/distributed/omni_coo | 跨 stage `async_chunk` 流式语义 | [async chunk](async-chunk.md) | | TP KV receive consensus、chunk boundary、active-window 合同与 Mooncake TCP write completion | [distributed rules](rules.md) 新增核对：DIST-1l、DIST-1l2、DIST-1l3、DIST-1l4。 | +- [SHM arrival hints、独立 work event 与 polling fallback](rules-shm-readiness.md)。 + +- [stage-0-final、CFG KV/chunk 分区与全部stage后回收](rules-shm-final-cleanup.md)。 diff --git a/knowledge/repos/vllm-omni/components/distributed/rules-shm-final-cleanup.md b/knowledge/repos/vllm-omni/components/distributed/rules-shm-final-cleanup.md new file mode 100644 index 000000000..91161ccbe --- /dev/null +++ b/knowledge/repos/vllm-omni/components/distributed/rules-shm-final-cleanup.md @@ -0,0 +1,24 @@ +--- +title: "Stage final SHM cleanup 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components] +sources: ["PR #7245"] +--- + +# Stage final SHM cleanup 规则 + +## DIST-FINAL-1a — Stage-0-final 的 chunk omission 必须与 KV override 分开 + +- 触发：修改 AR save_async 条件、omni_final_stage_id、CFG companion 或 metadata cache。 +- 强制：stage-0-final 没有 downstream chunk consumer，完成时不发送空 finished SHM；CFG force-KV override 只影响 KV，不重新启用 chunk。payload identity 变化使 cached flags 失效，streaming replacement 主动清 cache；必须在 free_request 重写 metadata 前读取 omission。异常携带 inter-stage output 的 stage-0-final 记录 warning 并仍不 put。 +- 禁止：复用仅按 request ID 缓存的旧 final/KV 决定；把 force-KV 当 chunk consumer；普通 downstream request 也跳过 terminal，从泄漏修复变成接收端挂起。 +- 验收：text-only、CFG force-KV、普通 downstream、metadata 替换与 in-place lifecycle reset 分别检查 put/KV 行为；用实际 SHM+lockfile 扫描验证 final 请求不留下无人接收 segment。 ^[PR #7245] + +## DIST-FINAL-1b — Undrained segment 只能在全部 stage 完成后按 owner prefix 回收 + +- 触发：修改 orchestrator cleanup、release RPC、adapter save-thread reclaim 或 SHM prefix cleanup。 +- 强制：只有知道所有 stage 已结束的 orchestrator 发 best-effort background release；强引用 task，stage/replica 并发 RPC 每个至多等 5 s，失败 warning，非 async_chunk 跳过。广播 live replicas 而不依赖已释放 binding；adapter 映射 external ID 后排队到 save thread，仅回收自己 tracked 的 `__`，未知 ID 幂等。内部请求 ID 继续带唯一 suffix/UUID，延迟 release 不能命中后续请求。 +- 禁止：scheduler thread 执行 unlink/I/O；producer/consumer 未结束就删除 terminal；按模糊 request prefix 扫全目录；因 RPC 成功就声称强事务/网络或资源释放必达。 +- 验收：early consumer stop、正常 terminal drain、unknown/repeated release、相似 ID、late put、dead replica timeout 与不同 internal ID 并发回归；保留原 producer fencing。 ^[PR #7245] diff --git a/knowledge/repos/vllm-omni/components/distributed/rules-shm-readiness.md b/knowledge/repos/vllm-omni/components/distributed/rules-shm-readiness.md new file mode 100644 index 000000000..3caa128d4 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/distributed/rules-shm-readiness.md @@ -0,0 +1,17 @@ +--- +title: "SHM arrival readiness 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components] +sources: ["PR #8184"] +--- + +# SHM arrival readiness 规则 + +## DIST-WAKE-1a — FIFO wakeup 只能作为提示，正确性必须保留有界 polling + +- 触发：修改 SharedMemoryConnector arrival wakeup、native receiver loop、send/receive work event 或 connector shutdown。 +- 强制：launcher 为同一 deployment 的 SHM connectors 生成并序列化同一 wakeup_scope，接收者各持 UUID FIFO；put 后非阻塞广播目标 stage 的所有当前 FIFO，校验 FIFO 类型且不跟随 symlink。receive 在 poll 前取 generation、无 progress 后等新 generation，不能以已存在未来 chunk 的 key 作 readiness predicate。发送和接收分别用独立 event，shutdown 唤醒两者；graceful close 只删自己 FIFO，空目录才移除。wakeup 不可达/禁用/满 pipe 时保留 timed polling；poll ms 必须 finite/positive，否则 warning 后 5 ms，两个等待路径都用最终间隔。 +- 禁止：按 EngineCore parent 分裂已声明的 deployment scope；缓存跨 receiver restart 的 writer；清理 peer FIFO；让 receive 清掉 send wakeup；把本机 FIFO 提示声称为 multi-node transport 或 hard-crash 自动回收保证。 +- 验收：不同 parent、多个 replica/restart、hint-before-poll、blocked future chunk、满/缺失 FIFO、无效 poll interval 与 shutdown 并发分别覆盖；禁用 wakeup 仍交付同一数据且不忙等。 ^[PR #8184] diff --git a/knowledge/repos/vllm-omni/components/model-executor/_index.md b/knowledge/repos/vllm-omni/components/model-executor/_index.md index bfb910340..6bb36bef5 100644 --- a/knowledge/repos/vllm-omni/components/model-executor/_index.md +++ b/knowledge/repos/vllm-omni/components/model-executor/_index.md @@ -1,7 +1,7 @@ --- title: "Model Executor" created: 2026-07-10 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, components, model-executor] sources: [] @@ -9,6 +9,8 @@ sources: [] # Model Executor +- [MRv2 optional codec](rules-mrv2-optional-codec.md) — missing attribute 与 eager-MTP replay eligibility。 + - 源码入口：`vllm_omni/model_executor/`（layers、model_loader、models、stage_input_processors）、 `vllm_omni/model_extras/`（模型专有请求/prompt 转换）、`vllm_omni/worker/`（gpu_*_worker、 gpu_*_model_runner、mixins）、`vllm_omni/inputs/`（runner 输入预处理）和设备平台层 @@ -34,12 +36,24 @@ sources: [] | 遇到什么 | 查看哪里 | |---|---| +| Breeze ROCm RNG fallback、CUDA generator registration或depth-layer Triton autotune | [Breeze平台合同](rules-breeze-rocm.md) | | 理解共享职责和阶段边界 | [architecture](architecture.md) | | 根据 PR 描述直达 stage config、runner preprocess、stage runtime、bridge/batch 或 loader 的规则组与第一批源码 | [rules 与代码地图](rules.md) | | runtime info、跨 stage payload、batch 与 request RNG 合同 | [跨 stage bridge 与 batch 合同](rules-bridge-batch.md) | | loader 的 dtype/config 获取、fused shard 与多模块 checkpoint 载入 | [loader 合同](rules-loader-contract.md) | | shared image example envelope 与 model_extras 参数声明 | [image task envelope 合同](rules-image-task-envelope.md) | -| Omni 输出类型与字段/复制合同 | [输出类型合同](rules-output-contract.md) 新增核对：EXEC-7c、EXEC-7d。 | +| Omni 输出类型、字段/复制合同与V2 native routed-expert auxiliary output | [输出类型合同](rules-output-contract.md) 新增核对：EXEC-7c、EXEC-7d。 | | 采样循环不变量、热路径缓存、AR 音频侧路、preprocess phase/one-token prefill | [运行时热路径合同](rules-runtime-hot-paths.md) | | NPU runner、ROCm 分页注意力、NPU 模型补丁 | [平台后端合同](rules-platform-backends.md) 新增核对：EXEC-10f、EXEC-10f2。 | + +- [MRv2 attention capture 边界](rules-attention-capture.md) — mixed FULL、separate decode 与当前请求 boundary。 + +- [release processor/dummy、profiling、state selection 与 auxiliary output：EXEC-RELEASE-1a–1d](rules-release-contracts.md)。 + +- [显式seed、whole-MTPgraph与NVIDIA/ROCm抽样边界](rules-request-rng.md)。 +- [异步output slab、producer事件与storage所有权](rules-output-snapshots.md)。 + +- [MRv2 aux tree、narrow sampler、sampled embedding 与 generation输出](rules-mrv2-output-contracts.md)。 + +- [settled row predicate与preemption replay fallback](rules-settled-preprocess.md)。 diff --git a/knowledge/repos/vllm-omni/components/model-executor/rules-attention-capture.md b/knowledge/repos/vllm-omni/components/model-executor/rules-attention-capture.md new file mode 100644 index 000000000..cad875a9b --- /dev/null +++ b/knowledge/repos/vllm-omni/components/model-executor/rules-attention-capture.md @@ -0,0 +1,18 @@ +--- +title: "MRv2 attention capture 边界规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, model-executor] +sources: ["PR #8422"] +confidence: high +--- + +# MRv2 attention capture 边界规则 + +## EXEC-ATTN-BOUND-1a — capture query bound 必须区分 mixed FULL 与 separate decode + +- 触发：修改 MRv2 `OmniModelState.prepare_attn` 的 capture metadata、CUDA graph mode 或 multimodal embedding 的 request boundaries。 +- 强制：仅在 capture、没有显式 `max_query_len` 且 graph mode 不使用 separate routine 时，把 mixed FULL 的 bound 提升到 token bucket 的最坏情况；separate decode 保持其 uniform query bound，已有显式 bound 和 runtime metadata 保持原语义。传给 multimodal embedding 的 `query_start_loc_np` 只取当前 `num_reqs + 1` 个 boundary。 +- 禁止：用 capture dummy 的平均每行长度约束 mixed replay；把 separate decode 的 FA3 GQA capture 扩成 bucket query；把预分配 boundary array 的旧尾部作为当前 request。 +- 验收：真实 FA3 覆盖 `FULL_AND_PIECEWISE`/`FULL_DECODE_ONLY` 的 capture 与 replay，并分别断言 mixed worst-case、uniform decode 和显式 bound；mixed multimodal batch 断言只读取当前请求边界。CPU shape/stub 检查不能替代 kernel replay。^[PR #8422] diff --git a/knowledge/repos/vllm-omni/components/model-executor/rules-breeze-rocm.md b/knowledge/repos/vllm-omni/components/model-executor/rules-breeze-rocm.md new file mode 100644 index 000000000..528f12ebd --- /dev/null +++ b/knowledge/repos/vllm-omni/components/model-executor/rules-breeze-rocm.md @@ -0,0 +1,27 @@ +--- +title: "Breeze ROCm graph 与 autotune 合同" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, model-executor] +sources: ["PR #8342", vllm_omni/model_executor/models/breeze_tts_2/depth_decoder.py, tests/model_executor/models/test_breeze_tts_2_first_code_sampler.py, tests/model_executor/models/test_breeze_tts_2_graphs.py] +confidence: high +--- + +# Breeze ROCm graph 与 autotune 合同 + +入口见 [Model Executor 规则](rules.md)。 + +## EXEC-17a — Breeze graph RNG 与深度层 autotune 必须按真实平台能力分支 + +- 触发：修改 Breeze first-code sampler、depth decoder graph 测试或 depth-layer compile options。 +- 强制：captured-generator-state 路径仅在 NVIDIA CUDA 且 `CUDAGraph.register_generator_state` + 可用时成立。ROCm 保留既有 eager sampling fallback，测试应断言无 sampler graph，同时继续 + 验证 token、每 request generator state、全局 RNG 与返回 tensor 的独立存储。 +- 强制：深度层 compile 固定 `epilogue_fusion=False`；`max_autotune` 在 ROCm 为 false， + 其他平台保持 true。greedy/不支持 generator registration 时不得要求随机 replay 命中 capture。 +- 禁止：因 `torch.cuda` 命名就把 ROCm 当 NVIDIA；为 graph-count 断言关闭 token/RNG/owned-output + 验证；把关闭一个不稳定 Triton GEMM autotune 路径外推为无 compile 或全模型性能保证。 +- 验收：CPU options test 分别断言 ROCm/non-ROCm；GPU matrix 分别验证 capture 与 eager fallback + 的 token/state/output，且 fallback 不污染全局 RNG。已报告的精确 AMD R2-01 结果只资格化该 + source/config，不代替其它 hardware 或全模型质量与性能证据。^[PR #8342] diff --git a/knowledge/repos/vllm-omni/components/model-executor/rules-mrv2-optional-codec.md b/knowledge/repos/vllm-omni/components/model-executor/rules-mrv2-optional-codec.md new file mode 100644 index 000000000..3076a41fc --- /dev/null +++ b/knowledge/repos/vllm-omni/components/model-executor/rules-mrv2-optional-codec.md @@ -0,0 +1,24 @@ +--- +title: "MRv2 optional codec 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, model-executor] +sources: ["PR #8484"] +confidence: high +--- + +# MRv2 optional codec 规则 + +## EXEC-MRV2-CODEC-1a — shared eager-MTP preprocess 不得假定 model 自带 stream decoder + +- 触发：修改 OmniModelState.run_preprocess、eager-MTP replay 或 optional codec access。 +- 强制：先以 getattr(model, stream_decoder, None) 验证 codec 存在，只有具有 codec 的 + Talker 才走相应 replay；无 attribute 与显式 None 均保持正常 preprocess/sample rows。 +- 禁止：把 Qwen3-TTS 的 in-Talker codec 属性强加给 Qwen3-Omni 等 model；MagicMock + 自动生成 attribute 或测试 helper 总设 None 掩盖真实缺失情况。 +- 验收：parameterize attribute 存在/None/删除三条路径，断言样本保留行为与 replay + eligibility；真实目标 MRv2 deploy 另做 warmup/serving，作者单一硬件吞吐不外推。^[PR #8484] + +attention metadata 见 [MRv2 capture](rules-attention-capture.md)； +model-specific Talker/codec 拓扑见 [Qwen-Omni](../../models/qwen-omni/_index.md)。 diff --git a/knowledge/repos/vllm-omni/components/model-executor/rules-mrv2-output-contracts.md b/knowledge/repos/vllm-omni/components/model-executor/rules-mrv2-output-contracts.md new file mode 100644 index 000000000..efd5f5414 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/model-executor/rules-mrv2-output-contracts.md @@ -0,0 +1,31 @@ +--- +title: "MRv2 shared output 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components] +sources: ["PR #8184"] +--- + +# MRv2 shared output 规则 + +## EXEC-MRV2-1a — FULL graph auxiliary output 必须显式声明稳定 tensor tree + +- 触发：修改 supports_mrv2_full_graph_aux_outputs、tuple output capture/replay、make_omni_output_mrv2 或模型 stage hooks。 +- 强制：只有声明且返回 tuple 的模型进入 FULL aux path；每个非空 pytree leaf 都是 tensor，leading dimension 与 hidden token axis 一致，所有 capture 的 tree spec 稳定，否则明确拒绝。capture 临时 wrapper/aux flag 在 finally 恢复；replay unflatten 后用本步 live batch/request state 生成输出，保留 real-token 与 padded-token 的切片边界。未声明 tuple/捕获 side-state 继续排除 FULL；外层 stage wrapper 必须暴露子模块有效 hooks，遗漏有 warning。 +- 禁止：把任意 Python/request metadata 捕入 graph；重放过期 prefill payload；容忍变化的 aux tree 或把 padding 发布给真实请求；模型没有 opt-in 就默认改用新 hook。 +- 验收：稳定嵌套树、非tensor/空/错 token-axis、跨 capture spec 变化、eager/FULL/padded batch 与 live metadata 改变都覆盖，异常后原 forward/flag 恢复。 ^[PR #8184] + +## EXEC-MRV2-1b — Narrow sampler 与 sampled embedding 发布必须按实际模型和有效行投影 + +- 触发：修改 custom sampler、logits_vocab_size、identity decode preprocessing 或 embed.sampled handoff。 +- 强制：模型已声明的 narrow head 在 sampler 构造前同步 runner/request-state/warmup vocab；custom sampler 或 static staged-write shortcut 仅按显式 hook/capability。identity decode 跳过仅限非 eager-MTP 的 decode 行。sampled embedding 仅给 async non-text multimodal handoff；一步必须每请求一枚 sample，未完成 prefill 的行留空，最后 prefill/有效 decode 行才取 model embedding；copy stream 等 producer event 后嵌套合并，不能覆盖其他 payload。speculative 多 sample 不走此发布。 +- 禁止：用 HF 大词表覆盖 codec head；给 partial prefill 发布丢弃 sample；无条件复用 upstream/custom sampler staged-write 生命周期；把 embed.sampled 优化说成所有模型性能保证。 +- 验收：narrow/no-declared vocab、custom/default sampler、partial/final prefill、mixed rows、multi-sample 与异步 producer stream分别回归，token/embedding 请求顺序和其他 payload 不变。 ^[PR #8184] + +## EXEC-MRV2-1c — Generation 输出必须保持真实请求 ID、精确 token 数和完成的 host 数据 + +- 触发：修改 generation runner request-owned cache、requires_exact_input_shape、V1 batched D2H 或 sparse conditioning。 +- 强制：requires_request_ids 的模型接收本步 scheduler ID、按 batch 顺序；声明 exact input 的 MRv2 generation eager 路径检查实际 input token 数等于 scheduled 数，错配拒绝，dummy run 不套用。V1 pinned CUDA 输出逐项 nonblocking copy 后全步一次同步，未 pinned/非CUDA保留 blocking；返回前 host 完整。per-row tensor/list cardinality 不可改变，None payload 保留区别，稀疏 conditioning 的 None 表示无更新并跳过。 +- 禁止：给每行合成固定/占位 ID；把 padded input 给按每请求 seq count 切分的模型；发布尚未完成 D2H 的 host tensor；把 None 归一为 {} 或丢失请求位置。 +- 验收：异长、batch>1、真实 ID reorder、exact-shape mismatch、dummy/control、pinned/unpinned/CPU 与 delayed D2H 覆盖，对旧逐项 copy 检查字段和数值一致。 ^[PR #8184] diff --git a/knowledge/repos/vllm-omni/components/model-executor/rules-output-contract.md b/knowledge/repos/vllm-omni/components/model-executor/rules-output-contract.md index 395202c60..8cc27b293 100644 --- a/knowledge/repos/vllm-omni/components/model-executor/rules-output-contract.md +++ b/knowledge/repos/vllm-omni/components/model-executor/rules-output-contract.md @@ -1,15 +1,28 @@ --- title: "Omni 输出类型合同" created: 2026-09-04 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, model-executor] -sources: ["PR #5146", "PR #6152", vllm_omni/outputs/, "PR #7448", "PR #7608"] +sources: ['PR #5146', 'PR #6152', vllm_omni/outputs/, 'PR #7448', 'PR #7608', 'PR #7843', 'PR #8500'] confidence: high --- # Omni 输出类型合同 +## EXEC-OUTPUT-1a — output type 只接受 canonical 名称，字段命名不证明表示正确 + +- 触发：修改 OutputModality parsing、engine_output_type config、payload naming 或 DELTA retention。 +- 强制：只接受 text/image/audio/latent/token_ids 的 lowercase 无空白名称与非空 `+`/`,` + 组合；None 保持默认，显式空值穿过 config 并在 output-processor 初始化失败，清理已 + 启动 replica。TOKEN_IDS 是独立类型且不 drain；generic model_outputs/hidden 沿共享 + default-key 命名，显式 auxiliary keys 保留，已有 per-key accumulation strategy 优先。 +- 禁止：把 token_ids 当 text alias；lower/strip 吞掉错误配置；因为 generic hidden 被 + 改名就当其 tensor values 已转成离散 IDs；改变已有 latent 表示或丢弃非流式中间全量上下文。 +- 验收：canonical/alias/case/whitespace/empty 解析、explicit-empty startup cleanup、dim0 + accumulation、DELTA retention 与 key override；tensor normalization 前后值不变， + 独立 auxiliary outputs 有明确 keys。平台-specific auxiliary producer 另验。^[PR #7843] + `EXEC-7a`–`EXEC-7b`：Omni 输出必须是扁平的 `RequestOutput` 子类，并保持其字段与复制合同。触发条件与其余审查组见 [model-executor 共享规则](rules.md) 的 Direct 代码快速入口。 ## EXEC-7a — Omni 输出必须是扁平的 RequestOutput 子类 @@ -41,3 +54,15 @@ confidence: high - 强制：`consolidate_tensors` 接收 modality，并对每个 key 调用 `get_accumulation_strategy(modality, key)`。流水线通过 `register_key_accumulation_strategy` 注册覆盖；Qwen3-TTS 必须将 `codes.audio`→`CONCAT_DIM0`、`codes.ref`→`REPLACE`。`CONCAT_LAST` 失败仍可 flatten 后拼接；其他策略失败必须带 key 名 raise，不得静默 keep-last。 - 禁止：整 payload 共用单一 modality 默认；把 codec-frame 矩阵当 `CONCAT_LAST` 波形；把非 `audio` 键的 concat 失败吞成 keep-last。 - 验收：合成张量证明 `codes.audio` 沿 dim0 拼满、`codes.ref` 只保留一份，以及错误策略 raise；真实波形键仍走 `CONCAT_LAST`。^[PR #7608] + +## EXEC-7e — routed-expert 回归必须验证受支持的 native V2 auxiliary output + +- 触发：升级vLLM native auxiliary output或修改GPU routed-expert tests。 +- 强制：GPU auxiliary output的consumer是V2 runner `aux_output_connector_output`；测试经真实 + native copy/process边界，保留decode/prefill rows、token_start和sampling mask，分别验证不同 + sampled/rejected token counts都传入copy。legacy GPU辅助输出继续由既有配置gate拒绝。 +- 禁止：mock已删除的legacy `_omni_extract_routed_experts`，为保住旧测试恢复dead API；或只 + 检查sampled count，使相同/全零rejected count隐藏遗漏。 +- 验收：compact width1/2的native V2结果保持row/offset/mask并出现两项不同count arrays； + legacy config仍拒绝。test-only修复不改变production支持范围；旧vLLM0.30环境静态验证不证明 + vLLM0.31 runtime，需单独的exact-head目标lane结果。^[PR #8500] diff --git a/knowledge/repos/vllm-omni/components/model-executor/rules-output-snapshots.md b/knowledge/repos/vllm-omni/components/model-executor/rules-output-snapshots.md new file mode 100644 index 000000000..9fa039124 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/model-executor/rules-output-snapshots.md @@ -0,0 +1,25 @@ +--- +title: "Async output snapshot 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, model-executor] +sources: ["PR #8213"] +--- + +# Async output snapshot 规则 + + +## EXEC-SNAPSHOT-1a — 打包异步输出必须保留 producer 与 consumer 的 storage 所有权 + +- 触发：修改PackedOutputSnapshot、dtype/device slab grouping、reuse_existing_storage或跨stream D2H。 +- 强制：按dtype/device分组保持pytree/shape/requestorder；只有contiguous leaves共享同storage且offset连续匹配copyplan才可重用slab。slot复用需等上一consumer，copy stream需等producer_event；host slab每output自有。quantized/非strided走原per-tensorfallback。 +- 禁止：把mutablegraph/slotview当owned snapshot；省略producerevent或在D2H完成前复用device；为打包改变dtype、request归属或叶子数值；已有ownedpacked输出再做重复GPUcopy。 +- 验收：dtype/device混合、empty/非contiguous/quantizedfallback、错offset、reuseon/off、独立producerstream、延迟D2H与后续slotoverwrite都覆盖，retainedhostoutput仍与原snapshot逐值一致。 ^[PR #8213] + +## EXEC-SNAPSHOT-1b — Inline 与 background AR 输出必须分别遵守元数据寿命 + +- 触发：修改 AR output materialization、query_start_loc CPU metadata、pooler hidden payload 或异步后处理 snapshot。 +- 强制：先决定输出是否 background；background 路径独立复制下一步会改写的 scheduler、sample/logprob/prompt-logprob/nan 与 query-start metadata，inline 在下一步开始前完成可借用当前 view。只有 hidden 关闭、无有效 multimodal payload、无 prefix-cache 工作、无待执行 model postprocess 时才跳过 payload 工作；已由 bookkeeping detach 的 request IDs/index map 不重复复制。 +- 禁止：把 inline 可借用的 buffer带入 background；只看 include_hidden 就跳过 prefix cache 或 postprocess；以少一次 copy 换取请求/长度错配。 +- 验收：延迟 background consumer 同时推进下一步、inline/background 对照、token-only CosyVoice、prefix-cache/postprocess 路径和 retained output 各回归，结果与未优化路径一致。 ^[PR #8473] diff --git a/knowledge/repos/vllm-omni/components/model-executor/rules-platform-backends.md b/knowledge/repos/vllm-omni/components/model-executor/rules-platform-backends.md index e98949e3e..5fde6e648 100644 --- a/knowledge/repos/vllm-omni/components/model-executor/rules-platform-backends.md +++ b/knowledge/repos/vllm-omni/components/model-executor/rules-platform-backends.md @@ -1,15 +1,26 @@ --- title: "平台后端合同" created: 2026-09-04 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, model-executor] -sources: ["PR #5886", "PR #6061", "PR #6096", vllm_omni/platforms/, "PR #5604", "PR #6293", "PR #5571", "vllm_omni/platforms/xpu/platform.py", "PR #5569", "vllm_omni/platforms/xpu/utils.py", "PR #5048", "PR #6350", "PR #6102", "PR #6563", "PR #6054", vllm_omni/platforms/npu/platform.py, tests/platforms/npu/test_diffusion_platform.py, tests/platforms/npu/test_diffusion_attn_backend_selector.py, "PR #6674", vllm_omni/platforms/npu/worker/npu_ar_model_runner.py, vllm_omni/platforms/npu/worker/npu_generation_model_runner.py, vllm_omni/platforms/npu/worker/npu_model_runner.py, "PR #7301", "PR #7393"] +sources: ["PR #5886", "PR #6061", "PR #6096", vllm_omni/platforms/, "PR #5604", "PR #6293", "PR #5571", "vllm_omni/platforms/xpu/platform.py", "PR #5569", "vllm_omni/platforms/xpu/utils.py", "PR #5048", "PR #6350", "PR #6102", "PR #6563", "PR #6054", vllm_omni/platforms/npu/platform.py, tests/platforms/npu/test_diffusion_platform.py, tests/platforms/npu/test_diffusion_attn_backend_selector.py, "PR #6674", vllm_omni/platforms/npu/worker/npu_ar_model_runner.py, vllm_omni/platforms/npu/worker/npu_generation_model_runner.py, vllm_omni/platforms/npu/worker/npu_model_runner.py, "PR #7301", "PR #7393", "PR #8330"] confidence: high --- # 平台后端合同 +## EXEC-NPU-INIT-1a — 模型补丁必须在 platform 初始化完成后、worker 建模之前安装 + +- 触发：修改 NPUOmniPlatform constructor/set_device、VoxCPM2 patch import 或 worker class。 +- 强制：会经模型 transitive dependencies 再解析 current_omni_platform 的补丁不得在 + platform constructor 导入；在 super.set_device 之后、worker 构造模型之前幂等安装， + 保持既有 custom-op/device 设置与 worker class。 +- 禁止：因为直接 model import 看起来 lazy 就在 constructor 安装；只验证 normal serve + 而不检查先 import 模型的 collection；把 CPU/mock 的 import-order 测试当作 NPU graph parity。 +- 验收：platform 构造不安装 VoxCPM2 patch，set_device 顺序为底层设备→补丁→custom ops， + 重复安装不重复 wrap；pytest collection 与真实 NPU execution 分开记录。^[PR #8330] + `EXEC-10a`–`10e`、`EXEC-12a`–`EXEC-13h`：NPU 平台 runner 的 runtime mode 与 dummy-run 接口、ROCm 分页注意力的 packed KV varlen 路径，以及 NPU 模型补丁的注册与回归。触发条件与其余审查组见 [model-executor 共享规则](rules.md) 的 Direct 代码快速入口。 ## EXEC-10a — NPU 平台 runner 必须保持 runtime mode 与 dummy-run 接口合同 diff --git a/knowledge/repos/vllm-omni/components/model-executor/rules-release-contracts.md b/knowledge/repos/vllm-omni/components/model-executor/rules-release-contracts.md new file mode 100644 index 000000000..0babb3979 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/model-executor/rules-release-contracts.md @@ -0,0 +1,71 @@ +--- +title: "Model Executor release API 与输出所有权合同" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, model-executor] +sources: ["PR #8459", vllm_omni/inputs/mm_processor.py, vllm_omni/inputs/preprocess.py, vllm_omni/worker_v2/omni_model_runner.py, vllm_omni/platforms/npu/worker/aux_output.py] +confidence: high +--- + +# Model Executor release API 与输出所有权合同 + +## EXEC-RELEASE-1a — dummy budget 与 processor API 迁移必须保留模型 conditioning + +- 触发：升级 multimodal processor、dummy-input builder、cache 或 renderer truncation API。 +- 强制：Omni builder 的自定义 `get_dummy_processor_inputs` 仍到达模型 consumer，保留 + reference `prompt_text/ref_text` 等 kwargs；标准 builder 走 upstream fallback。已移除 + `_get_hf_mm_data` 时从实际 item 提取 processor/passthrough data，保持 Omni custom + processor 所需的 raw key（如 `audios`），不无意套 upstream 的 renamed `audio`/dummy-text。 +- 强制：cache 从当前 `ProcessorInputs.cache` 取得；multimodal result/prompt updates + 使用目标 release 的结果合同。sync/async singleton path 都传 `tok_params`，对展开后的 + prompt 执行 upstream truncation，不能切断 media placeholder；no-media kwargs 仍到达处理器。 +- 禁止：只修 import/signature 而丢模型 dummy conditioning；全局 processor cache 代替 + 每请求 cache；绕过标准 fallback 或通过粗截 token 消除 expanded prompt 超长。 +- 验收：真实 CosyVoice3/GLM-TTS/OmniVoice/MiMo builder 路径和标准 builder control； + raw processor keys、passthrough、空 modality、audio-in-video 成对 cache 与 prompt updates； + sync/async、有无 media 的 truncation 保留完整媒体边界。mock 结果不证明真实模型质量。 + ^[PR #8459] + +## EXEC-RELEASE-1b — profiling pool、capture 与 dummy codec 输入必须分别遵守 release 合同 + +- 触发：升级 runner 的 `profile_only`、`randomize_inputs`、dummy input 或 attention metadata API。 +- 强制：`capture_model(profile_only=True)` 透传 upstream profiling flag，但不留下 model-owned + MTP graphs；仅真实 capture 阶段录制它们，临时 forward/aux flag 在异常和正常退出都恢复。 + dummy `is_padding=not is_profile`、active LoRA 数及 model-specific attention metadata 传到 + 当前 upstream owner。语言模型可按 flag 随机输入，codec dummy 保留模型词表内有效 token。 +- 禁止：把临时 profiling pool 中的 capture 当成生产 replay graph；用语言模型随机词表 + 填 codec IDs；generation profiling 为测 KV 而执行 Code2Wav 并把 transient workspace 算入预算。 +- 验收：profile/real capture、tensor/tuple output、正常/异常恢复与 MTP capture 次数； + AR/generation dummy paths 分别验证 randomize、padding flag、合法 codec IDs、upstream + execution state 与 attention metadata 透传。GPU replay 和资源结论需另外真实验证。^[PR #8459] + +## EXEC-RELEASE-1c — auxiliary Omni stage 只能包装 upstream 默认 model state + +- 触发：stage MRv2 selection、input validator 或没有模型自有 lifecycle hook 的 auxiliary stage。 +- 强制：模型自有 factory/capability 先决定 Omni state；否则调用 upstream factory，只有 + exact `DefaultModelState` 且属于 `OmniModelConfig` stage 时补 Omni lifecycle。专用 upstream + state 与 plain vLLM config 原样保留。input/logits validator 使用本 stage 的实际 + `use_v2_model_runner`，不能让 process-wide preference 覆盖部署选择。 +- 禁止：把所有非 Omni architecture 都无条件包装；因 shared auxiliary stage 没有模型 hook + 而丢 request/output lifecycle；用全局 V2 env 改变 legacy stage 的 CFG 参数验证器。 +- 验收：auxiliary default、specialized upstream、plain vLLM 和 model-owned factory 四路， + 再将全局 preference 与 stage flag 故意设为相反，验证选择与实际 runner 一致。^[PR #8459] + +## EXEC-RELEASE-1d — native auxiliary output 必须冻结 request 行并在交付后释放 + +- 触发：升级 routed-expert/aux-output connector、async output 或 NPU legacy worker。 +- 强制：scheduler 打包 connector metadata，worker `begin_step` 包括无 forward 的控制步； + prepare output 先冻结 request IDs、query-start 与 computed-token frontier，CPU copy 完成后 + 才 materialize。AR 携带实际 sampled/rejected counts，generation 无 sampled tokens 时用零 + counts；仅有 pending auxiliary output 时也必须进入支持它的交付路径。 +- 强制：NPU legacy capture 获取 remap/shared-expert expansion 前的 logical expert IDs， + 每个 router 只包装一次；排除 partial-prefill 的 sampled count 和 speculative rejection + 未接受的 routing suffix。关闭/重建 connector 清理 capture hooks；disabled capture 保留 + 普通 async 路径。CUDA/ROCm auxiliary output 要求支持的 V2 stage，NPU 使用专门 legacy adapter。 +- 禁止：继续读旧 `routed_experts` 字段绕开 native connector；让下一 batch 改写未交付的 + 行映射；无 forward 时跳过 terminal/hash metadata；把 NPU CPU contract 测试说成 NPU + hardware 已验证，或把旧 CUDA legacy stage 的 unsupported 配置静默放行。 +- 验收：request snapshot 改写、partial prefill、speculative rejection、generation 零 counts、 + auxiliary-only async 交付、no-forward teardown、重复初始化与 disabled control；NPU + 实机与跨设备 parity 独立验收。^[PR #8459] diff --git a/knowledge/repos/vllm-omni/components/model-executor/rules-request-rng.md b/knowledge/repos/vllm-omni/components/model-executor/rules-request-rng.md new file mode 100644 index 000000000..c0dcdc913 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/model-executor/rules-request-rng.md @@ -0,0 +1,26 @@ +--- +title: "Request RNG 与 MTP graph 边界" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, model-executor] +sources: ["PR #8009", "PR #8341"] +--- + +# Request RNG 与 MTP graph 边界 + + +## EXEC-RNG-1a — 显式逐行 generator 必须先于 whole-MTP graph 决策 + +- 触发：修改 runner _talker_mtp_forward、graph wrapper 或 tts_local_seed 的 generator 创建。 +- 强制：先解析并按 request/device 缓存逐行 generator，再决定 graph mode；任何 row 有显式 generator 时，整个 MTP 调用必须 eager 且只取真实 decode batch 行。无 generator 且 wrapper 受支持时保留 graph replay；request 结束回收 generator。 +- 禁止：把 generator 参数传给只 replay startup dummy capture 的 wrapper 后声称 seed 被消费；按 model 关闭整条 stage 的 graph，或用 padded rows 消耗 seeded RNG。 +- 验收：CUDA/Ascend graph-wrapper control 覆盖 mixed seeded/unseeded、seed0、同请求多步与 request teardown；断言真实 Python MTP 消费 generator、各行状态独立，generator-free batch 仍 replay。 ^[PR #8009] + + +## EXEC-RNG-1b — seeded exponential 的 NVIDIA kernel 不能由 tensor.is_cuda 单独放行 + +- 触发：修改 batched_seeded_exponential_supported 或直接 fill_exponential_rows 调用。 +- 强制：NVIDIA Triton/Philox kernel 必须由 current_platform.is_cuda() 放行；ROCm/HIP及其他平台使用 Torch ordered row draws。无 rows 时每 generator 对应 flatten 后等大连续slice；有 rows 时严格沿所选行顺序，None 沿默认 generator 消耗。 +- 禁止：把 HIP 的 tensor.is_cuda 当作 NVIDIA capability；仅替换 fast_logf 就宣称跨平台 RNG 等价；让直接 Higgs/codebook caller 绕过平台门禁或重排共享 generator 的抽样。 +- 验收：平台dispatch、五次重复sample、seeded/default generator 完整state、FP32/FP64、grouped-codebook、selected-row、sharedgenerator及mixedseed controls都与 Torch逐行 reference 对照。 ^[PR #8341] diff --git a/knowledge/repos/vllm-omni/components/model-executor/rules-settled-preprocess.md b/knowledge/repos/vllm-omni/components/model-executor/rules-settled-preprocess.md new file mode 100644 index 000000000..5ed051cec --- /dev/null +++ b/knowledge/repos/vllm-omni/components/model-executor/rules-settled-preprocess.md @@ -0,0 +1,17 @@ +--- +title: "MRv2 settled row preprocess 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components] +sources: ["PR #8477"] +--- + +# MRv2 settled row preprocess 规则 + +## EXEC-SETTLED-1a — Vectorized settled rows 必须保留 scalar predicate 和 replay fallback + +- 触发：修改MRv2run_preprocess、_split_settled_rows、eagerMTPmetadata或preemptedstreamrestore。 +- 强制：settled仅当本步scheduledtokens=1且slotowner恰等当前requestID；返回settled与remaining两组保持roworder/queryoffset。当identitydecode另有skip或pendingpreemptionreplay时保留原rowloop，buffer/slot清理同步清settledowner。eagerMTPtuple一次transpose仍按原row/request/prefill/firstaudio/position字段布局，ready更新保留slot→ID；只有待restore状态非空才扫描，CPU保存history恢复到当前workerdevice后再执行。 +- 禁止：只靠slot命中旧request；把preemptionreplay当普通settledskip；转置时重排firstaudio/prefill flags；以CPU/GPU驻留判断扩展为任意device迁移保证。 +- 验收：随机slotpermutation、staleID、mixedtokencount、pendingreplay、queryoffset与scalarpredicate对照；暂停/恢复streamdecoder和非emptyrestore保留原PCM，热路径改写不改变requeststate。 ^[PR #8477] diff --git a/knowledge/repos/vllm-omni/components/scheduler/_index.md b/knowledge/repos/vllm-omni/components/scheduler/_index.md index 2ef1a8f31..61ce1e062 100644 --- a/knowledge/repos/vllm-omni/components/scheduler/_index.md +++ b/knowledge/repos/vllm-omni/components/scheduler/_index.md @@ -1,7 +1,7 @@ --- title: "Scheduler（AR/生成请求调度）" created: 2026-07-16 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, components, scheduler] sources: [vllm_omni/core/sched/omni_ar_scheduler.py, vllm_omni/core/sched/omni_generation_scheduler.py, vllm_omni/core/sched/omni_scheduler_mixin.py, vllm_omni/core/sched/output.py, vllm_omni/core/prefix_cache.py, docs/design/module/ar_runtime.md, docs/design/module/archive/ar_module.md] @@ -42,3 +42,11 @@ sources: [vllm_omni/core/sched/omni_ar_scheduler.py, vllm_omni/core/sched/omni_g | 按 PR 描述直达 prefix cache、token budget、upstream 接口或 side-stream 首批源码 | [rules / Direct 代码快速入口](rules.md#direct-代码快速入口) | - [rules-shared-lifecycle](rules-shared-lifecycle.md) 新增核对：SCHED-6a2。 + +- [Running prefix-cache reset 能力](rules-prefix-reset-capability.md) — active request 的拓扑能力检查与拒绝边界。 + +- [release 双 waiting queue、deferred set 与 resumable cleanup：SCHED-RELEASE-1a、SCHED-5g](rules-release-queues.md)。 + +- [empty terminal、staged-payloadreceivegate与tail守恒](rules-stream-terminals.md)。 + +- [native sender-only admission 与 opt-in first-chunk express](rules-first-chunk-express.md)。 diff --git a/knowledge/repos/vllm-omni/components/scheduler/rules-first-chunk-express.md b/knowledge/repos/vllm-omni/components/scheduler/rules-first-chunk-express.md new file mode 100644 index 000000000..73d4d01fd --- /dev/null +++ b/knowledge/repos/vllm-omni/components/scheduler/rules-first-chunk-express.md @@ -0,0 +1,24 @@ +--- +title: "Native first-chunk scheduling 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components] +sources: ["PR #8184"] +--- + +# Native first-chunk scheduling 规则 + +## SCHED-NATIVE-1a — Sender-only native stage 不能等待不存在的 input chunk + +- 触发：修改 shared scheduler 初始化、stage runner 选择、native input coordinator 或 resumable request admission。 +- 强制：scheduler 使用最终 stage model_config 的 V1/V2选择，与worker一致；native async coordinator 仅用于 stage0 或真正 receives_chunks 的 stage，sender-only downstream 保持可调度。resumable 请求进入 orchestrator 时核对有效 final stage 范围内 native MRv2 下游，发现即返回 request-scoped client error；模型 plugin 不能绕过 turn-only guard，adapter 缺席的 resume 路径安全返回。 +- 禁止：强制所有 scheduler V1；把 native sender capability 等同 receiver；把 mixed runner 支持外推为 duplex/streaming session protocol 支持。配置 precedence 与平台 veto 仍见 CONF-5m。 +- 验收：V1/V2 sender-only、receiver、stage0、turn/resumable、不同 final stage 与不存在 adapter 分别回归；错误只结束目标请求，无永久 WAITING_FOR_CHUNK。 ^[PR #8184] + +## SCHED-EXPRESS-1a — First-chunk express 必须默认关闭并保留 continuation 机会 + +- 触发：修改 VLLM_OMNI_CODEC_FIRST_CHUNK_EXPRESS、generation waiting/running 排序或 continuation playback slack。 +- 强制：显式 opt-in 且 native chunks 才可 express；仅就绪且无 in-flight 的首块触发，express 不能连续两步，延后已启动 continuation 后恢复 waiting 顺序。启用正 slack threshold 时所有已就绪 continuation 都必须有足够 audio credit；未知/多声道 layout 不赚 credit。credit 仅来自 mono tensor+正整数 sample rate，以首发 monotonic 时刻和发出长度估算，request free 清状态。 +- 禁止：把 emitted audio credit 说成实际客户端已播放量；在默认关闭时改调度；把最多隔步 continuation 调度描述为无饥饿/低延迟保证；为首块忽略已有 in-flight 所有权。 +- 验收：持续首块压力、mixed first/continuation、slack不足/未知/充足、stop/free/reuse与feature关闭分别覆盖；检查不连续 express、原顺序与尾块完整，并报告首次音频和continuation underrun。 ^[PR #8184] diff --git a/knowledge/repos/vllm-omni/components/scheduler/rules-prefix-reset-capability.md b/knowledge/repos/vllm-omni/components/scheduler/rules-prefix-reset-capability.md new file mode 100644 index 000000000..05d0cdd66 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/scheduler/rules-prefix-reset-capability.md @@ -0,0 +1,18 @@ +--- +title: "Running prefix-cache reset 能力规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, scheduler] +sources: ["PR #8259"] +confidence: high +--- + +# Running prefix-cache reset 能力规则 + +## SCHED-RESET-1a — active reset 在 preemption 前检查拓扑 capability + +- 触发：修改共享 scheduler 的 running prefix-cache reset，或接入持有已消费帧/排队 PCM 的 stateful decoder。 +- 强制：由最终 model config 的 `supports_running_prefix_cache_reset` 声明能力，scheduler 不按模型 architecture 特判。当请求要求 reset running requests、当前有 running request 且 capability 为 false 时，必须在调用父级 reset/preemption 前返回 `False`；完成或 abort 后才可重试。fused Qwen3-TTS 的 false pin 不能被 legacy runtime override 或未归属的 structured field 翻转。 +- 禁止：先丢弃 in-flight tokens 再检查不可回滚 codec 状态；仅改共享 scheduler 就宣称所有拓扑支持 active reset；把 idle reset 或既有两阶段默认能力一并禁用。 +- 验收：覆盖 active unsupported 的无 mutation 拒绝、idle reset、支持拓扑、legacy override 和 structured owner 拒绝；connector prompt replacement 的 stale fence 同时执行 [SCHED-6d](rules.md#sched-6d-显式-prompt-replacement-必须一次性释放旧状态并回到-admission)。^[PR #8259] diff --git a/knowledge/repos/vllm-omni/components/scheduler/rules-release-queues.md b/knowledge/repos/vllm-omni/components/scheduler/rules-release-queues.md new file mode 100644 index 000000000..87e0637d1 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/scheduler/rules-release-queues.md @@ -0,0 +1,37 @@ +--- +title: "Scheduler release waiting queue 合同" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, scheduler] +sources: ["PR #8459", "PR #6089", "PR #6360", "PR #6680", vllm_omni/core/sched/omni_scheduler_mixin.py, vllm_omni/core/sched/omni_generation_scheduler.py] +confidence: high +--- + +# Scheduler release waiting queue 合同 + +## SCHED-RELEASE-1a — vLLM 0.31 input gate 必须保留 KV holder 的队列优先级 + +- 触发：vLLM rebase 改变 waiting queues、Omni input gating 或 active admission limits。 +- 强制：一次 scheduler tick 的 input-ready pass 统一访问 `kv_holding_waiting` 与 + `waiting`，优先访问 KV holder，connector state 只消费一次。park/restore/requeue + 依据 upstream `_holds_kv_blocks` 回到对应物理队列；移除依据实际 membership，不能 + 依赖可能已被 input metadata 改写的 computed frontier。grammar、remote-KV 与 + streaming-update 等 upstream blocked status 不得被 Omni readiness 覆写。 +- 强制：`deferred_waiting` 是独立的 blocked/deferred request set；恢复接纳和终态清理 + 要同步它与两条物理队列。临时关闭 waiting admission 时两队一起保存/替换，在 `finally` + 恢复并保留顺序；reserved slots 对齐 `max_num_active_reqs`。generation path 同样先选 + KV-holding queue，再选 fresh queue；CFG parent/companion 跨两队仍需配对。 +- 禁止：仅处理旧 `waiting/skipped_waiting` 模型；metadata 修改后按新状态去错误队列 + remove；KV holder 恢复为 fresh 请求；因延迟输入覆盖 upstream blocked 状态。 +- 验收：AR/generation、full-payload/async-chunk、不同 queue policy 与双物理队列并存 + 时覆盖 park→restore；检查 connector 单次消费、KV holder 优先、blocked waits 保持、 + abort sweep、streaming counter 与跨队 CFG 配对。API import 成功不能代替此状态机验收。 + ^[PR #8459] + +## SCHED-5g — resumable async-chunk 终态清理必须以 live queue 所有权为准 + +- 触发：resumable async-chunk 请求在一个流式 segment 结束时进入 `FINISHED_STOPPED`，但仍由 `running`、`waiting`、`kv_holding_waiting` 或 connector 隐藏队列持有，随后 session close、取消或其他路径调用 `finish_requests`。 +- 强制：先物化可能被多层消费的 request-id iterator，并在 adapter 清理前快照 `deferred_waiting` 中承担流式等待计数的请求；只对仍有活队列所有权的目标 resumable 终态请求恢复状态，deferred streaming wait 恢复为 `WAITING_FOR_STREAMING_REQ`，`running`/`waiting` 按实际队列对齐；下游 async-chunk 的 segment stop 必须先清除该 segment 的 finished 标记。只有 connector `receives_chunks`、最终 stage `model_config.session_mode == "duplex"` 且 request 仍为 `WAITING_FOR_STREAMING_REQ` 时，才从 `kv_holding_waiting` 转为 ordinary `WAITING` 并恢复 connector polling；sender-only 或 turn-mode stage 保持 parked，等待显式 streaming update。同一 update 尾部按 stale stopped 集合清理时不得移除已重新入队的 request；running purge 必须限定本次 finish 集合，并确保 `_free_request`、coordinator 与 connector 清理恰好执行一次。 +- 禁止：把任意已完成请求重新打开；对脱离所有 live queue、可能等待 deferred block free 的终态请求调用释放；因 request 在本轮进入时是 `WAITING_FOR_CHUNK` 就撤销其同轮 requeue；全局清空 running 中无关的 resumable segment；重复消费单遍 request-id iterator，或因非流式 skipped 请求错误减少 streaming counter。 +- 验收：AR 与 generation scheduler 均覆盖 hidden、`running`、`waiting`、`kv_holding_waiting`、脱离队列及无关 resumable 请求；另覆盖 connector-fed duplex receiver 从 segment stop 同轮转回 `WAITING`，断言它保留在 waiting queue、离开 KV-holding/deferred wait、流式等待计数递减且 segment-finished 标记清除；sender-only duplex 与 connector-fed turn-mode controls 必须仍 parked，同时也清除标记。首次 finish 恰好释放、第二次无操作，并验证 generator request-id 输入。PR 作者报告的 H100 E2E 与 574-pass CPU/config suite 是提交时证据，不是当前环境或跨硬件保证。^[PR #6089] ^[PR #6360] ^[PR #6680] ^[PR #8459] diff --git a/knowledge/repos/vllm-omni/components/scheduler/rules-stream-terminals.md b/knowledge/repos/vllm-omni/components/scheduler/rules-stream-terminals.md new file mode 100644 index 000000000..26d9270fc --- /dev/null +++ b/knowledge/repos/vllm-omni/components/scheduler/rules-stream-terminals.md @@ -0,0 +1,18 @@ +--- +title: "Streaming terminal 与 staged payload 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, scheduler] +sources: ["PR #8213"] +--- + +# Streaming terminal 与 staged payload 规则 + + +## SCHED-STREAM-1a — empty terminal 必须在上一 consumable chunk 被调度后才能替换 prompt + +- 触发：修改connector receivegate、control-message drain、generation prompt replacement或emptyEOF。 +- 强制：missingcodes不改prompt；presentempty snapshot清旧chunk/placeholder。该区分依赖receivegate：consumable payload仍staged时不能获取下一chunk；按requestack/schedule顺序推进，保护真实tail不被后续emptymetadata覆写。MOSS、Qwen3、Higgs、Fish、Cosy的terminal格式分别核对。 +- 禁止：用truthiness混同missing与empty；last-writer-wins drain先收真实tail再收EOF后只调度empty；重复decode上一prompt，或为一个模型clear逻辑绕过全局receivegate。 +- 验收：上一chunk未schedule时emptyEOF不能被fetch；正常tail→ack/schedule→emptyfinish、same-drainattempt、cancel与多request交错都验证frame守恒与no-duplicate。模型控制组必须穿过真实scheduler/connector，不只独立调用metadata更新。 ^[PR #8213] diff --git a/knowledge/repos/vllm-omni/components/scheduler/rules.md b/knowledge/repos/vllm-omni/components/scheduler/rules.md index dfe230cfb..76cd46217 100644 --- a/knowledge/repos/vllm-omni/components/scheduler/rules.md +++ b/knowledge/repos/vllm-omni/components/scheduler/rules.md @@ -1,10 +1,10 @@ --- title: "Scheduler 规则" created: 2026-07-16 -updated: 2026-09-05 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, scheduler] -sources: ["PR #5957", "PR #5976", tests/core/sched/test_omni_ar_scheduler_stale_drain.py, tests/core/sched/test_omni_ar_scheduler_streaming.py, "vllm-omni-rebase-agent@122a9468:agent/skills/fix-talker-truncated-prefill-prefix-cache-key-cap/SKILL.md", "vllm-omni-rebase-agent@122a9468:agent/skills/gpu-hang-low-max-num-batched-tokens/SKILL.md", vllm_omni/worker/gpu_ar_model_runner.py, vllm_omni/core/prefix_cache.py, vllm_omni/utils/mm_outputs.py, vllm_omni/core/sched/omni_ar_scheduler.py, vllm_omni/core/sched/omni_generation_scheduler.py, vllm_omni/core/sched/omni_scheduler_mixin.py, vllm_omni/core/sched/omni_scheduling_coordinator.py, vllm_omni/core/sched/output.py, tests/core/test_prefix_cache.py, tests/core/test_prefix_cache_async_write.py, tests/core/sched/test_omni_scheduler_mixin_shared.py, tests/core/sched/test_omni_scheduler_mixin_timeouts.py, tests/distributed/omni_connectors/test_chunk_transfer_adapter.py, tests/utils/test_mm_outputs.py, tests/entrypoints/test_omni_new_request_data.py, "PR #4106", "PR #5310", "PR #5461", "PR #4795", "PR #5842", "PR #6021", "PR #6033", "PR #6089", "PR #6149", "PR #6360", "PR #6406", "PR #6150", "PR #6619", "PR #6680", "PR #6626", "PR #6529", tests/core/sched/test_omni_ar_scheduler_aborted_queue_sweep.py, tests/core/sched/test_omni_scheduler_streaming_input_counter.py, tests/core/sched/test_omni_sched_deferred_free_fence.py, "PR #6831", tests/e2e/online_serving/test_nemotron_voicechat_duplex.py] +sources: ["PR #8459", "PR #8259", "PR #5957", "PR #5976", tests/core/sched/test_omni_ar_scheduler_stale_drain.py, tests/core/sched/test_omni_ar_scheduler_streaming.py, "vllm-omni-rebase-agent@122a9468:agent/skills/fix-talker-truncated-prefill-prefix-cache-key-cap/SKILL.md", "vllm-omni-rebase-agent@122a9468:agent/skills/gpu-hang-low-max-num-batched-tokens/SKILL.md", vllm_omni/worker/gpu_ar_model_runner.py, vllm_omni/core/prefix_cache.py, vllm_omni/utils/mm_outputs.py, vllm_omni/core/sched/omni_ar_scheduler.py, vllm_omni/core/sched/omni_generation_scheduler.py, vllm_omni/core/sched/omni_scheduler_mixin.py, vllm_omni/core/sched/omni_scheduling_coordinator.py, vllm_omni/core/sched/output.py, tests/core/test_prefix_cache.py, tests/core/test_prefix_cache_async_write.py, tests/core/sched/test_omni_scheduler_mixin_shared.py, tests/core/sched/test_omni_scheduler_mixin_timeouts.py, tests/distributed/omni_connectors/test_chunk_transfer_adapter.py, tests/utils/test_mm_outputs.py, tests/entrypoints/test_omni_new_request_data.py, "PR #4106", "PR #5310", "PR #5461", "PR #4795", "PR #5842", "PR #6021", "PR #6033", "PR #6089", "PR #6149", "PR #6360", "PR #6406", "PR #6150", "PR #6619", "PR #6680", "PR #6626", "PR #6529", tests/core/sched/test_omni_ar_scheduler_aborted_queue_sweep.py, tests/core/sched/test_omni_scheduler_streaming_input_counter.py, tests/core/sched/test_omni_sched_deferred_free_fence.py, "PR #6831", tests/e2e/online_serving/test_nemotron_voicechat_duplex.py] --- # Scheduler 规则 @@ -267,12 +267,6 @@ modules=[online_serving, worker_runner]，status=active，run_count=38，2026-06 - 验收：覆盖解析、stall/reset、完成/abort 与健康同批请求；超时为 `FINISHED_ERROR`。Nemotron 固定 `240 < 300`，server request-ID/error 先于 client timeout，健康 control 两者皆不触达。^[PR #6033] ^[PR #6831] -## SCHED-5g — resumable async-chunk 终态清理必须以 live queue 所有权为准 - -- 触发：resumable async-chunk 请求在一个流式 segment 结束时进入 `FINISHED_STOPPED`，但仍由 `running`、`waiting`、`skipped_waiting` 或 connector 隐藏队列持有，随后 session close、取消或其他路径调用 `finish_requests`。 -- 强制：先物化可能被多层消费的 request-id iterator，并在 adapter 清理前快照 `skipped_waiting` 中承担流式等待计数的请求；只对仍有活队列所有权的目标 resumable 终态请求恢复状态，`skipped_waiting` 恢复为 `WAITING_FOR_STREAMING_REQ`，`running`/`waiting` 按实际队列对齐；下游 async-chunk 的 segment stop 必须先清除该 segment 的 finished 标记。只有 connector `receives_chunks`、最终 stage `model_config.session_mode == "duplex"` 且 request 仍为 `WAITING_FOR_STREAMING_REQ` 时，才从 `skipped_waiting` 转为 ordinary `WAITING` 并恢复 connector polling；sender-only 或 turn-mode stage 保持 parked，等待显式 streaming update。同一 update 尾部按 stale stopped 集合清理时不得移除已重新入队的 request；running purge 必须限定本次 finish 集合，并确保 `_free_request`、coordinator 与 connector 清理恰好执行一次。 -- 禁止：把任意已完成请求重新打开；对脱离所有 live queue、可能等待 deferred block free 的终态请求调用释放；因 request 在本轮进入时是 `WAITING_FOR_CHUNK` 就撤销其同轮 requeue；全局清空 running 中无关的 resumable segment；重复消费单遍 request-id iterator，或因非流式 skipped 请求错误减少 streaming counter。 -- 验收：AR 与 generation scheduler 均覆盖 hidden、`running`、`waiting`、`skipped_waiting`、脱离队列及无关 resumable 请求；另覆盖 connector-fed duplex receiver 从 segment stop 同轮转回 `WAITING`，断言它保留在 waiting queue、离开 skipped queue、流式等待计数递减且 segment-finished 标记清除；sender-only duplex 与 connector-fed turn-mode controls 必须仍 parked，同时也清除标记。首次 finish 恰好释放、第二次无操作，并验证 generator request-id 输入。PR 作者报告的 H100 E2E 与 574-pass CPU/config suite 是提交时证据，不是当前环境或跨硬件保证。^[PR #6089] ^[PR #6360] ^[PR #6680] ## SCHED-5i — consumed connector contract failure 必须在 scheduler thread 终止 live request @@ -323,9 +317,9 @@ modules=[online_serving, worker_runner]，status=active，run_count=38，2026-06 ## SCHED-6d — 显式 prompt replacement 必须一次性释放旧状态并回到 admission - 触发：running 或 computed 的 async prompt 显式携带 `replace_streaming_prompt`，或 ready 的 replacement 进入 scheduler。 -- 强制：替换先释放旧 KV/encoder state 一次，清除 in-flight prefill 所有权和 connector watermark；stale fence 以当前 `num_in_flight_tokens` 赋值，再将请求回到 `WAITING` 并走正常 admission。 +- 强制：替换先释放旧 KV/encoder state 一次，清除 in-flight prefill 所有权和 connector watermark；stale fence 先设 `drop_stale_output=True`，以当前 `num_in_flight_tokens` 赋值，再将请求回到 `WAITING` 并走正常 admission。 - 禁止：只更新 prompt 而继承旧 cache/watermark；把同一 in-flight frame 在更新和 replacement 路径重复计入 stale；绕过 scheduler admission 直接恢复运行。 -- 验收：覆盖 running replacement、ready async replacement 的重复 reset 和正常重新调度；断言 KV/encoder 释放各一次、watermark 清零、stale counter 恰好可排空且首个新 segment frame 不被丢弃。 ^[PR #6406] +- 验收：覆盖 running replacement、ready async replacement 的重复 reset 和正常重新调度；断言 KV/encoder 释放各一次、watermark 清零、stale counter 恰好可排空且首个新 segment frame 不被丢弃。 ^[PR #6406] ^[PR #8259] ## SCHED-6e — 流式 segment 边界必须在 request mutation 前冻结发送 watermark @@ -341,9 +335,9 @@ modules=[online_serving, worker_runner]，status=active，run_count=38，2026-06 - 强制：同一迟到 frame 必须联合结算 scheduled-token stale window 与 `async_tokens_to_discard`；前者按 scheduled-token accounting，后者按该 frame 的 generated-token 数递减，任一 fence 命中就不得 append/emit。处理 pending input 前后都从 - `waiting`、`skipped_waiting` 与 `running` 清除 `FINISHED_ABORTED`，随后按 live queue ownership - 重算 `num_waiting_for_streaming_input`。 -- 禁止：两个 stale domain 只消费一个；只清普通 waiting/running 而遗留 hidden skipped queue； + `waiting`、`kv_holding_waiting` 与 `running` 清除 `FINISHED_ABORTED`，随后按 live queue ownership + 清理相应 `deferred_waiting` 集合成员，重算 `num_waiting_for_streaming_input`。 +- 禁止：两个 stale domain 只消费一个；只清普通 waiting/running 而遗留 KV-holding queue 或 deferred set； 用增减猜 streaming counter；把 unacknowledged response snapshot 当作有界 lifetime，或把 sliding-recompute/后续 playback ACK 修复归因于本提交。 - 验收：覆盖一个 frame 同时命中两种 fence、仅命中任一 fence、三类 queue 的 abort sweep、 diff --git a/knowledge/repos/vllm-omni/components/serving/_index.md b/knowledge/repos/vllm-omni/components/serving/_index.md index cc5c17f6f..4b53d8573 100644 --- a/knowledge/repos/vllm-omni/components/serving/_index.md +++ b/knowledge/repos/vllm-omni/components/serving/_index.md @@ -1,7 +1,7 @@ --- title: "Serving" created: 2026-07-10 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, components, serving] sources: [] @@ -31,14 +31,27 @@ sources: [] | 理解入口到 engine 的边界，选择 unified HTTP/WS endpoint、区分 wire protocol，或确认 PersonaPlex/JoyVL 是否是独立 server | [architecture 与 API route owner](architecture.md) | | 根据 PR 描述直达 request contract、stage sampling constraints、streaming format、artifact readiness、factory lifecycle 或 metrics 的规则组与第一批源码 | [rules 与代码地图](rules.md) | | Prometheus family、pipeline waiting/running gauge、stage/replica snapshot 或 collector lifecycle | [metrics 生命周期规则](rules-metrics.md) 新增核对：SERV-2g。 | -| upstream launcher shutdown、renderer warmup 与多 replica compile-cache 隔离 | [upstream 兼容规则](rules-upstream-compat.md) 新增核对：SERV-7f、SERV-7e、SERV-7g、SERV-7h。 | +| upstream launcher shutdown、renderer warmup、多 replica compile-cache 隔离与parsed CLI logging初始化 | [upstream 兼容规则](rules-upstream-compat.md) 新增核对：SERV-7f、SERV-7e、SERV-7g、SERV-7h。 | | stage/replica 死亡、request-local failure、readiness/liveness 或驱逐后 cleanup | [fault-isolation rules](rules-fault-isolation.md) 新增核对：SERV-5u。 | | batched chat frontend fan-out、choice cardinality、whole-batch error/cancellation | [batch chat rules](rules-batch-chat.md) | | engine startup/shutdown、stage lifecycle、acknowledged abort 的 shutdown race、duplex stage prewarm/CFG companion，或 opt-in event-driven orchestration / reader-poller reconcile / final-output drain | [engine 生命周期规则](rules-engine-lifecycle.md) | -| duplex stage-keyed segment state、跨 stage boundary/metadata 隔离 | [duplex 编排规则](rules-duplex-orchestration.md) | +| duplex stage-keyed segment state、跨 stage boundary/metadata 隔离与model harness共享有界output buffer | [duplex 编排规则](rules-duplex-orchestration.md) | | adapter-owned full-duplex session、deferred cognition/gate、barge-in cancellation、response ownership、`to_thread` worker drain、close/flush/session cleanup，或 startup warmup/realtime admission | [session lifecycle 规则](rules-session-lifecycle.md) 新增核对：SERV-6j。 | | acknowledged abort 的 RPC ACK、queue/router shutdown race、cancellation cleanup 或 backpressure/timeout 保持 | [acknowledged abort 生命周期规则](rules-abort-lifecycle.md) | | assembled FastAPI route 覆盖、method/path ownership 或 request-body forwarding | [app assembly 规则](rules-app-assembly.md) | | stage config→EngineArgs projection、explicit `devices`、TP/local-DP/PP、replica layout 或 worker 创建前的 layout guard | [stage 启动与设备布局规则](rules-stage-startup.md) 新增核对：SERV-12c、SERV-8b。 | | 公开请求字段的校验、来源冲突、alias/extras 归一与 consumer view | [请求输入合同](rules-request-input.md) 新增核对：SERV-4r、SERV-4r2、SERV-4r3、SERV-4r4、SERV-4s、SERV-4r5、SERV-4i2。 | | speech 输出采样率 capability、resample、streaming header/flush | [speech 输出规则](rules-speech-output.md) 新增核对：SERV-9c。 | +| OpenAI Realtime routing、PCM16 admission、部分 session 更新、response 终态与有界 history/truncate | [Realtime 请求与会话规则](rules-realtime-openai.md) | + +- [Realtime video 增量 prefill 规则](rules-video-stream-prefill.md) — warmup/abort、窗口身份与已解码帧 snapshot。 + +- [Sleep/wake ACK 与 admission](rules-sleep-control.md) — worker/RPC failure、partial-stage recovery 与 stage-ID 预校验。 + +- [effective video config 与 borrowed RGB eligibility](rules-video-encoding.md)。 + +- [duplex bounded delivery、cancel guard 与creation payload大小](rules-duplex-output-delivery.md)。 + +- [voice删除403/404与typedexceptions](rules-voice-management.md)。 + +- [sleep/wakeup非空stage_ids与nonnegativelevel](rules-control-request-schema.md)。 diff --git a/knowledge/repos/vllm-omni/components/serving/rules-control-request-schema.md b/knowledge/repos/vllm-omni/components/serving/rules-control-request-schema.md new file mode 100644 index 000000000..adc963f29 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/serving/rules-control-request-schema.md @@ -0,0 +1,17 @@ +--- +title: "Omni control request schema 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components] +sources: ["PR #4740"] +--- + +# Omni control request schema 规则 + +## SERV-CONTROL-1a — Sleep/wakeup HTTP schema 必须在 RPC 前拒绝空目标和负 level + +- 触发：修改 OmniSleepRequest/OmniWakeupRequest 或 /v1/omni/sleep、/v1/omni/wakeup 请求字段。 +- 强制：两类stage_ids仍required list[int]且min_length=1；sleep level默认2、ge=0。无效schema由HTTPvalidation拒绝，保留有效level0/非空target通路；stage是否存在、backendlevel能力和部分成功由后续owner处理。 +- 禁止：把空[]解释为所有stage；负level转默认；额外从此PR推断list元素非负/unique、只允许level1/2或已完成真实模型sleep测试。 +- 验收：emptytargets与negativelevel返回422/对应HTTPvalidationdetail；缺字段、level0/default及单/多stage合法结构通过，然后独立执行已有stage/admission/ACK状态验收。 ^[PR #4740] diff --git a/knowledge/repos/vllm-omni/components/serving/rules-duplex-orchestration.md b/knowledge/repos/vllm-omni/components/serving/rules-duplex-orchestration.md index c3ba34800..08d70a2ff 100644 --- a/knowledge/repos/vllm-omni/components/serving/rules-duplex-orchestration.md +++ b/knowledge/repos/vllm-omni/components/serving/rules-duplex-orchestration.md @@ -1,10 +1,10 @@ --- title: "Serving duplex 编排规则" created: 2026-09-05 -updated: 2026-09-05 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, serving] -sources: ["PR #6529", vllm_omni/engine/orchestrator.py, tests/engine/test_orchestrator_segment_stage_keying.py, tests/engine/test_orchestrator_stage_input_bridge.py] +sources: ["PR #6529", vllm_omni/engine/orchestrator.py, tests/engine/test_orchestrator_segment_stage_keying.py, tests/engine/test_orchestrator_stage_input_bridge.py, "PR #8489"] confidence: high --- @@ -23,3 +23,15 @@ confidence: high - 验收：多 stage 交错上报 boundary，逐 stage 验证 metrics、final-output、bridge 与 duplex context 只见自身 token/metadata；覆盖 missing-stage fresh state 与不会经过 raw-output loop 的 diffusion branch。^[PR #6529] + +## SERV-6k — model-specific duplex harness 必须共享同一有界输出 buffer + +- 触发：修改 `OpenDuplexSessionMessage` 合同、generic Harness字段或PersonaPlex/Nemotron harness。 +- 强制：从manager使用的同一 `DuplexSessionRuntimeConfig` 派生max_pending_output_bytes/events， + 创建单个 `DuplexOutputBuffer`，将同一object传给open message和Harness供event drain；构造 + 扩展后的Harness用显式keywords，保持production `output_buffer` 必填。 +- 禁止：将required参数改成optional/default来掩盖stale tests；open和drain使用不同buffer， + 或复制一套与manager不一致的limits；用旧positional field顺序让queue/buffer错位。 +- 验收：generic与两model-specific runner suites使用相同公开open/drain合同，旧代码重现fixture + error/失败后修复通过；核对正常events与teardown，test-only修复不证明新模型或全transport能力。 + 原source在同allocation A/B得到64通过；目标CI证据仍须按真实source/hardware单独资格化。^[PR #8489] diff --git a/knowledge/repos/vllm-omni/components/serving/rules-duplex-output-delivery.md b/knowledge/repos/vllm-omni/components/serving/rules-duplex-output-delivery.md new file mode 100644 index 000000000..45e0c4b61 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/serving/rules-duplex-output-delivery.md @@ -0,0 +1,31 @@ +--- +title: "Duplex output delivery 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components] +sources: ["PR #7644"] +--- + +# Duplex output delivery 规则 + +## SERV-DUPLEX-OUT-1a — Per-session output 必须有界且不能阻塞其他 session + +- 触发：修改 DuplexOutputBuffer、manager/handle shared outbox、overflow、open rollback 或 closure。 +- 强制：同进程engine/caller共享每session一个buffer，producer不阻塞；默认compactJSON含base64计2MiB/512events，error+ResponseDone独立reserve64KiB/8events，另有一个held event与独立finalclosure slot。普通事件overflow fail该response并只close该session；保留待发ending身份改failed，不重复已接受ending，移除失败partialhistory且保留enginecleanup binding。reserve满可省略error/ending，closure仍有位；oversizedclose details压缩。open rollback在首次await前去掉bufferowner，lateordinaryoutput不能回unboundedsharedqueue或重开handle。 +- 禁止：把预算说成全部Python/RSS、rawstage mailbox或replaystore上限；阻塞全局pump等慢consumer；已close后接收lateaudio；承诺closure必达网络或资源cleanup必成功。 +- 验收：ordinary/reserve/closure边界、overflowpendingending/已acceptedending、failedopen再cancel、latesend和两个独立session同时覆盖；slow一个session期间另一个继续有序输出。 ^[PR #7644] + +## SERV-DUPLEX-OUT-1b — Cancelled audio 必须在 sequencing 前再次验证 + +- 触发：修改barge-in/silentclose、queued/heldaudio、WebSocketattachmentlock、replayjournal或localiteration。 +- 强制：取消按response ID与through_epoch先invalidate queued/heldaudio，active及cancelleddraining都在await stageabort前处理；其他response和nonaudio原顺序保留。WebSocket在connectionlock内以bufferguard再次验证，再分配sequence、journal与on_accepted；guard不能跨networkawait。localhandle共享同validity，单consumer完成当前event后再get。 +- 禁止：只在dequeue时校验而让heldaudio等待lock后误发；清空其他response；回撤已经sequenced/journaled/transmitted audio，或把模型emission的sent_ms当物理播放/传输证明。 +- 验收：锁前hold再cancel、queued+held、draining、silentclose、localiterator和另外session自然完成分别覆盖；无效audio既不sequenced也不journaled/callback，已交付音频需要客户端停止播放。 ^[PR #7644] + +## SERV-DUPLEX-OUT-1c — Creation event payload 必须保持入队时计量大小 + +- 触发：修改response creation projector、conversation item增长或bufferbyteaccounting。 +- 强制：growing state item与三类creation event使用不同dict/content，后续text/transcript delta只增长state，queuedcreationpayload保持原shape/size；有效audio前的completion FIFO顺序保持。 +- 禁止：共享growingcontent使已计量事件无界长大；宣称此creation fix覆盖后续truncation的所有completion/retrievalmutableitem问题；以CPUfixture说明所有模型真实E2E已验证。 +- 验收：真实projector产生creation→入队计量→追加text/transcript，检查state增长而queuedpayload/bytes不变；不同modelharness必须传manager与handle共享的output_buffer，新ownerfixture不能遗漏新增openmessage字段。 ^[PR #7644] diff --git a/knowledge/repos/vllm-omni/components/serving/rules-engine-lifecycle.md b/knowledge/repos/vllm-omni/components/serving/rules-engine-lifecycle.md index 3221baaa8..ae68e82a6 100644 --- a/knowledge/repos/vllm-omni/components/serving/rules-engine-lifecycle.md +++ b/knowledge/repos/vllm-omni/components/serving/rules-engine-lifecycle.md @@ -39,18 +39,6 @@ confidence: high - 验收：sleep 两个 stage、只 wake 一个时 generation 仍拒绝，随后 wake 另一个才放行。 ^[PR #4834] -### SERV-5b — 只有成功 ACK 和真实 backend capability 才能转为 warm - -- 触发：worker ACK 可返回 error，或不同 backend 对 level-2 restore 能力不同。 -- 强制：逐目标确认成功 ACK 后再清状态；level-2 能力按 backend/stage 表达。 -- 禁止：错误 ACK 也清 tag；用 engine 全局禁令误伤已经支持 restore 的 diffusion worker。 -- 验收：失败 ACK 保留 sleeping 状态；支持 level-2 的 diffusion 路径仍能 - sleep → wake → generate，不支持的 stage 在调用 worker 前明确拒绝。 - assembled app 对不支持的 level-2 wake 保留 sleeping state，并把既有 - `NotImplementedError` 映射为 OpenAI-style structured HTTP 501；bare route mock 只证明 - exception propagation，不能代替这条 live contract。^[PR #4834] ^[PR #4905] - ^[PR #4912] ^[PR #5713] - ### SERV-5c — 共享服务类重构必须闭合全部构造状态 - 触发：移动 model-specific capability 到 adapter、增加 factory/`__new__` 快路径，或让同一 @@ -148,34 +136,6 @@ confidence: high - 禁止：让 server 保存或修改静态能力；把上传 voice 合并进 built-in speaker 集合后在删除时 `discard`，或把 frozen dataclass 误解为内部 profile mapping 的深层不可变；在共享 serving 中恢复 model-type capability 分支。 - 验收：构造相关 adapter，断言能力在初始化后进入 snapshot、集合类型正确且内置/预计算/上传 voice 的并集完整；删除与 built-in 同名的上传 voice 后内置 voice 仍存在，并验证无 adapter 的 diffusion 路径不会触发 adapter 属性错误或专属维度校验。^[PR #6138] -### SERV-5m — AR lifecycle 控制必须先封锁 admission，并以 backend 完成信号收尾 - -- 触发：修改 `AsyncOmni` 的 pause/resume、sleep/wake 或 abort，或改变 - `StagePool.collective_rpc` 到 AR EngineCore / diffusion worker 的控制面路由。 -- 强制：在 sleep drain/offload 前设置 frontend admission gate；AR stage 经 orchestrator - 路由到 EngineCore 的 `pause_scheduler`、`resume_scheduler`、`sleep`、`wake_up`，diffusion - stage 继续走 worker sleep/wake RPC。`wake_up()` 不得替 AR/mixed engine 解除 admission，只有 - `resume_generation()` 可以解除；已 paused 时的定向 pause 和 cache clear 仍须执行。stage ID - 必须先校验范围。只允许这四个已知 EngineCore helper 使用 `*_async` fast path，并保持 caller - timeout；其余 RPC 仍走 collective path。需要 frontend cleanup 的 abort 必须关联 result，等待 - orchestrator 完成 stage abort、binding release 和 request cleanup 后才移除 frontend state，失败或 - timeout 必须保留 state 并传播。 -- 强制：sleep 先关闭 admission，再等待所有 in-flight `_admitting` submissions drained 后才 offload；P0 - 先 reset MM cache，stage tag 保留 per-stage scope。wake 不重新开启 AR admission，仍只由 - `resume_generation()` 解除。AR abort 必须把 cumulative terminal-prefix token 由 output processor - 传至 stage pool/orchestrator 并 ACK 回 async queue；先物理 abort，再 commit output/request state，且 - state 延迟到 generator 消费后清理。只移除最后一个 child，只有最后 terminal output 收敛；control RPC - exceptions 必须传播。diffusion abort 路径未因本 PR 改写。 -- 禁止：以单一 frontend flag 代替 AR scheduler 控制；在 sleep RPC 后才阻止 `generate()`；让 - wake 隐式 resume；因已 paused 跳过不同 stage scope 或 cache reset；将任意同名 `*_async` helper - 绕过 collective timeout；在 abort ack 前 pop request state，或将 abort failure 当成功。 -- 验收：覆盖 AR-only、diffusion-only 和 mixed stage 路由；sleep 进行时 generation 等待，AR - sleep → wake 后仍需显式 resume，跨 AR+diffusion 的 sleep/wake E2E 必须在 post-wake - `generate()` 前显式调用 `resume_generation()`，而 diffusion-only sleep → wake 可恢复 admission；重复/定向 - pause 仍调用 scheduler 与 cache clear，非法 stage ID 产生明确错误，fast path 仅命中四个方法且 - timeout 生效；abort success 后才清理 frontend state，orchestrator error 与 timeout 时 state 保留。 - 本 PR 的测试/PR body 没有独立 GPU performance 或全量 diffusion-abort evidence。^[PR #6084] ^[PR #6581] ^[PR #6367] - ### SERV-5n — remote replica attach/detach 必须按本次 membership generation 判定 - 触发：修改 `MembershipController` 的 register、watcher snapshot、unregister、shutdown 或 remote client factory。 diff --git a/knowledge/repos/vllm-omni/components/serving/rules-realtime-openai.md b/knowledge/repos/vllm-omni/components/serving/rules-realtime-openai.md new file mode 100644 index 000000000..620561565 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/serving/rules-realtime-openai.md @@ -0,0 +1,102 @@ +--- +title: "OpenAI Realtime 请求与会话规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, serving] +sources: ["PR #7285", "PR #8279", "PR #8287", "PR #8294"] +confidence: high +--- + +# OpenAI Realtime 请求与会话规则 + +## Direct 代码快速入口 + +| PR 描述信号 | 规则组 | 第一批源码 | +|---|---|---| +| `/v1/realtime`、duplex query、Qwen stages | SERV-RT-1a | `entrypoints/openai/api_server.py::realtime_endpoint` → `entrypoints/duplex/openai.py::dispatch_realtime_websocket` | +| PCM16、session.update、turn detection | SERV-RT-1b | `entrypoints/openai/realtime/connection.py` 的 audio-format validation、session update 与 input buffer | +| response.cancel、max_output_tokens、terminal event | SERV-RT-1c | `OpenAIFullDuplexConnection._run_response`、`ActiveResponse` | +| history memory、deferred truncate、tools | SERV-RT-1d | `realtime/session.py` 的 history insert/replace → `connection.py` 的 truncate/tool admission | + +## SERV-RT-1a — Realtime route 必须保留已配置 native duplex 的优先级 + +- 触发：修改 `/v1/realtime` 的 query routing、模型 stage 判定或 connection factory。 +- 强制：已配置 native duplex handler 时，无 duplex query 或显式 `1/true/on` 继续进入该 + engine-owned handler；显式启用 duplex 却未配置 handler 时返回 unsupported error。 + 其余请求才进入 OpenAI dispatch：Qwen3-Omni 必须具有 thinker/talker/code2wav 三个 stage， + 运行期 typed stage 的 architecture 从 `stage.model_config.model_arch` 读取，角色从 + `stage.model_stage` 读取；不能沿用早期 `stage.engine_args` mapping/object lookup。 + 三个角色必须各一个，非空 model query 必须匹配 served name。 + OpenAI connection 的 buffered input/`response.create` 生命周期与 native continuous duplex + 各自维持自己的输入与 response ownership。 +- 禁止：按模型名把所有 Qwen 请求强行改走 OpenAI connection；给非 Qwen fallback 塞入 + Qwen token timing；把 client-driven buffer 宣称为同一 generation 的持续音频输入。 +- 验收：路由矩阵覆盖 configured/unconfigured handler、缺失/true/false query、完整/缺失 + Qwen stage、typed pipeline factory config、重复 stage/其他 architecture 与错误 model； + 通过 assembled WebSocket 检查正常 turn server 进入 OpenAI connection，分别检查 native + 与 OpenAI protocol 的生命周期。不要求 OpenAI 标准客户端发送自定义 `playback.ack` + 才能完成会话。^[PR #7285] ^[PR #8279] + +## SERV-RT-1b — 输入格式、部分 session 更新与 VAD 能力必须显式验证 + +- 触发：修改 Realtime PCM decode、`session.update`、turn detection 或 buffer admission。 +- 强制：只接收当前实现支持的 24 kHz PCM16；base64、偶数字节及单次 append/总 buffer + 上限在 decode/写入前验证。部分更新使用 `exclude_unset` 后递归合并；显式 null 是更新， + 不能当作缺省值忽略。OpenAI connection 当前将 server/semantic VAD 归一为 disabled， + native handler 的 VAD 行为由自己的配置决定。 +- 禁止：把任何输入音频格式静默按 PCM16 解码；因字段未出现就清空旧配置；把保留的 + VAD 字段或 session.update acknowledgement 当作 server VAD 已实现。 +- 验收：覆盖不支持的格式/采样率、坏 base64、奇数字节、append/total overflow、嵌套部分 + 更新及显式 null；已有 configuration 不因无关字段更新而丢失。^[PR #7285] + +## SERV-RT-1c — 每个 response 必须有一次且只有一次真实终态 + +- 触发：修改 response task 的初次 send、取消、异常路径、token limit 或 final events。 +- 强制：用户取消即使发生在初始 output item 发送之前，也以 cancelled/client_cancelled + 收尾；terminal-event fence 防止重复终态。`max_tokens` finish 转成 incomplete、原因 + `max_output_tokens`；只有未完成且非正常取消的路径才发送 server error，并释放 active response。 +- 禁止：finally 将已取消任务改成 failed；重复发送 response.done；把达到 token limit 当作 + completed 成功或丢弃未完成原因。 +- 验收：阻塞首次 send 后取消、生成中取消、disconnect、正常完成、token-limit 和异常 + 分别断言终态类型、次数与 active state 清理；测试取消时序而不依赖固定 sleep。^[PR #7285] + +## SERV-RT-1d — history 写入与延迟 truncate 必须在模型之外保有界和 identity + +- 触发：修改 conversation item insert/replace/delete、音频 truncate、history memory 或 tool admission。 +- 强制：history insert/replace 候选在写入前检查 64 MiB 上限，不依赖下一次 response.create。 + 当前 response 未产生完整 token IDs 时保存整个 truncate event，含 item/content_index； + 完成后按同一 item 应用，已删除 item 保持删除。invalid content_index 或超出音频长度的 + 请求先拒绝再修改 history；ack 保持请求的 content_index。 + 不支持的 MCP tools 在 session 配置过滤，在 response 入口明确拒绝，不执行隐藏 fallback。 +- 禁止：history 无限制增长直到生成时才检查；延迟 truncate 丢失 content_index 或把已删除 + item 重新插入；把 shared history storage 绑定某模型的 token/ms 算法。 +- 验收：仅持续插入而不生成也命中 memory cap；覆盖 truncate-before-final、删除后延迟 + 回调、invalid index/end、连续 truncate 最新值及不支持 tools。Qwen prefix 对齐细节归 + [模型规则](../../models/qwen-omni/rules-realtime.md)。^[PR #7285] + +共用 worker drain 与 session close 合同见 [session lifecycle](rules-session-lifecycle.md)； +公共路由装配见 [app assembly](rules-app-assembly.md)。 + +## SERV-RT-1e — dispatcher import 不得提前初始化回指它的 OpenAI package + +- 触发：移动 duplex dispatcher、OpenAI connection imports 或 API-server package exports。 +- 强制：会触发 `openai.__init__`→api_server→duplex.openai 回环的 connection imports + 留在对应 WebSocket handler 内；tests patch handler 真正导入的 definition module。 +- 禁止：只验证正常 serve 先导入 api_server 的顺序；把 lazy symbol patch 在 dispatcher + 顶层造成测试绑定旧 alias。 +- 验收：独立 process 分别先 import duplex.openai 与 api_server，均完成；再验证两类 + WebSocket dispatch，不能以 guard suite collect 成功代替实际路由。^[PR #8287] + +## SERV-RT-1f — Realtime E2E 客户端必须按当前 wire protocol 等待完整 response + +- 触发：修改 Qwen3-Omni/OpenAI Realtime replay client、PCM chunking 或终态断言。 +- 强制：OpenAI buffered path 按 24 kHz PCM16 算 bytes/ms，session.update 使用 nested + session 对象；append 后 commit，再 response.create。消费 audio 的 `delta`、transcript + delta/done 的相应字段，记录 audio.done 后继续读到 response.done，再判定整次 response。 + native duplex 的输入 fixture 按自身协商/resample 路径处理，不能混用 buffered helper。 +- 禁止：发送旧 final=true commit、用旧 `audio` 字段读 delta、在 audio.done 提前退出； + 以一次模型自我介绍的固定措辞当作协议验收或将 native VAD 证明转给 buffered OpenAI path。 +- 验收：至少一个非空且偶数字节 audio delta、audio.done 与 response.done 均观测到， + sample-rate/chunk duration 与实际输入一致；对 conversation 语义使用合适的 model/task + 断言，避免偶发同义词误报协议失败。^[PR #8294] diff --git a/knowledge/repos/vllm-omni/components/serving/rules-sleep-control.md b/knowledge/repos/vllm-omni/components/serving/rules-sleep-control.md new file mode 100644 index 000000000..cc0d3207a --- /dev/null +++ b/knowledge/repos/vllm-omni/components/serving/rules-sleep-control.md @@ -0,0 +1,64 @@ +--- +title: "Sleep / wake admission 与 ACK 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, serving] +sources: ["PR #4834", "PR #4905", "PR #4912", "PR #5713", "PR #6084", "PR #6581", "PR #6367", "PR #7811", "PR #8501"] +confidence: high +--- + +# Sleep / wake admission 与 ACK 规则 + +### SERV-5b — 只有成功 ACK 和真实 backend capability 才能转为 warm + +- 触发：worker ACK 可返回 error，或不同 backend 对 level-2 restore 能力不同。 +- 强制：逐目标确认成功 ACK 后再清状态；level-2 能力按 backend/stage 表达。diffusion + ERROR ACK（object/dict）与 RPC error envelope（含 boolean error+reason、unsupported） + 必须转成带原因的 RuntimeError；无 task_id 的失败 envelope 不交给 ACK resolver。失败 sleep + 仍记录可能部分 offload 的 diffusion stage，HTTP sleeping set 只镜像 engine 实际记录的 stage。 + wake 只清成功 stage 的 tags，失败仍保持 admission；level-2 pre-offload 失败可重试，只有 + 已有 replica 真正丢弃 weights 才保留不可恢复标志。HTTP runtime failure 返回结构化 500。 +- 禁止：错误 ACK 也清 tag；用 engine 全局禁令误伤已经支持 restore 的 diffusion worker。 +- 验收：失败 ACK 保留 sleeping 状态；支持 level-2 的 diffusion 路径仍能 + sleep → wake → generate，不支持的 stage 在调用 worker 前明确拒绝。 + assembled app 对不支持的 level-2 wake 保留 sleeping state，并把既有 + `NotImplementedError` 映射为 OpenAI-style structured HTTP 501；bare route mock 只证明 + exception propagation，不能代替这条 live contract。^[PR #4834] ^[PR #4905] + ^[PR #4912] ^[PR #5713] ^[PR #7811] +- 验收补充：覆盖 object/dict/RPC-error、混合 stage 的部分成功、HTTP sleep failure→wake + 可达、invalid IDs 不污染 sleeping set，以及 level-2 全失败与部分成功的不同恢复能力。 + 显式 stage 查询返回该范围内是否仍有 sleeping tag。^[PR #7811] + +### SERV-5m — AR lifecycle 控制必须先封锁 admission，并以 backend 完成信号收尾 + +- 触发：修改 `AsyncOmni` 的 pause/resume、sleep/wake 或 abort，或改变 + `StagePool.collective_rpc` 到 AR EngineCore / diffusion worker 的控制面路由。 +- 强制：在 sleep drain/offload 前设置 frontend admission gate；AR stage 经 orchestrator + 路由到 EngineCore 的 `pause_scheduler`、`resume_scheduler`、`sleep`、`wake_up`，diffusion + stage 继续走 worker sleep/wake RPC。`wake_up()` 不得替 AR/mixed engine 解除 admission，只有 + `resume_generation()` 可以解除；已 paused 时的定向 pause 和 cache clear 仍须执行。stage ID + 必须先校验范围，且在设置 `_paused` / admission gate 之前完成，非法 ID 不得改变 + admission 或等待 in-flight drain；HTTP ValueError 映射为 400。只允许这四个已知 EngineCore helper 使用 `*_async` fast path，并保持 caller + timeout；其余 RPC 仍走 collective path。需要 frontend cleanup 的 abort 必须关联 result，等待 + orchestrator 完成 stage abort、binding release 和 request cleanup 后才移除 frontend state，失败或 + timeout 必须保留 state 并传播。 +- 强制：sleep 先关闭 admission，再等待所有 in-flight `_admitting` submissions drained 后才 offload；P0 + 先 reset MM cache，stage tag 保留 per-stage scope。wake 不重新开启 AR admission，仍只由 + `resume_generation()` 解除。AR abort 必须把 cumulative terminal-prefix token 由 output processor + 传至 stage pool/orchestrator 并 ACK 回 async queue；先物理 abort，再 commit output/request state，且 + state 延迟到 generator 消费后清理。只移除最后一个 child，只有最后 terminal output 收敛；control RPC + exceptions 必须传播。diffusion abort 路径未因本 PR 改写。 +- 禁止：以单一 frontend flag 代替 AR scheduler 控制；在 sleep RPC 后才阻止 `generate()`；让 + wake 隐式 resume；因已 paused 跳过不同 stage scope 或 cache reset；将任意同名 `*_async` helper + 绕过 collective timeout；在 abort ack 前 pop request state，或将 abort failure 当成功。 +- 验收：覆盖 AR-only、diffusion-only 和 mixed stage 路由；sleep 进行时 generation 等待，AR + sleep → wake 后仍需显式 resume，跨 AR+diffusion 的 sleep/wake E2E 必须在 post-wake + `generate()` 前显式调用 `resume_generation()`，而 diffusion-only sleep → wake 可恢复 admission；重复/定向 + pause 仍调用 scheduler 与 cache clear，非法 stage ID 产生明确错误，fast path 仅命中四个方法且 + timeout 生效；abort success 后才清理 frontend state，orchestrator error 与 timeout 时 state 保留。 + 本 PR 的测试/PR body 没有独立 GPU performance 或全量 diffusion-abort evidence。^[PR #6084] ^[PR #6581] ^[PR #6367] ^[PR #8501] +- 验收补充：invalid sleep stage 后立即 generate 仍可进入；验证在非法输入下没有 sleep RPC、 + cache reset、paused-state mutation，HTTP sleep/wake 的 client errors 保持 400。^[PR #8501] + +其他 stage 与 session 生命周期见 [engine lifecycle](rules-engine-lifecycle.md)。 diff --git a/knowledge/repos/vllm-omni/components/serving/rules-upstream-compat.md b/knowledge/repos/vllm-omni/components/serving/rules-upstream-compat.md index e5127c74b..cdd511ff2 100644 --- a/knowledge/repos/vllm-omni/components/serving/rules-upstream-compat.md +++ b/knowledge/repos/vllm-omni/components/serving/rules-upstream-compat.md @@ -1,15 +1,41 @@ --- title: "Serving upstream 兼容规则" created: 2026-09-02 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, components, serving] -sources: ["PR #5976", "PR #5957", vllm_omni/engine/stage_engine_startup.py, vllm_omni/entrypoints/openai/api_server.py, vllm_omni/entrypoints/utils.py, vllm_omni/request.py, tests/engine/test_stage_engine_startup_cache_env.py, tests/config/test_endpoint_policy.py, "PR #5036", "PR #6642", "PR #6773", "PR #6707", vllm_omni/config/endpoint_policy.py, "PR #6051", "PR #7426", "PR #5647"] +sources: ['PR #8459', 'PR #5976', 'PR #5957', vllm_omni/engine/stage_engine_startup.py, vllm_omni/entrypoints/openai/api_server.py, vllm_omni/entrypoints/utils.py, vllm_omni/request.py, tests/engine/test_stage_engine_startup_cache_env.py, tests/config/test_endpoint_policy.py, 'PR #5036', 'PR #6642', 'PR #6773', 'PR #6707', vllm_omni/config/endpoint_policy.py, 'PR #6051', 'PR #7426', 'PR #5647', 'PR #8518'] confidence: high --- # Serving upstream 兼容规则 +## SERV-RELEASE-KV-1a — 新 release KV hints 必须无损穿过 request 升级与 wire + +- 触发：新增 upstream request 字段或修改 Omni `generate`、stage submission、wire conversion。 +- 强制：nonstream 与 streaming 首次 submission 都将 `kv_hints` 传入真实 input processor； + `EngineCoreRequest` 升级为 Omni request、wire transport 和 scheduler conversion 保留 + `kv_hints` 与 session identity。diffusion stage 不具有该 AR/LLM KV cache，显式拒绝 hints。 + fixed prompt-token scores 从 worker CPU snapshot 经 scheduler 到 engine output 保持 + request identity，partial prefill 不伪造完整 score 输出。 +- 禁止：仅在入口接纳字段却在重新包装 dataclass 时丢失；忽略 diffusion 的 hints； + 在 async scores copy 完成前读取，或让另一 request 的 scores 混入当前输出。 +- 验收：有/无 hints、stream/nonstream、payload upgrade→wire→scheduler，及 diffusion + refusal；fixed-token scores 分别覆盖 host copy 与真实 CUDA copy 路径。接口通过不等于 + remote KV 性能或完整模型 logprob 质量已经验收。^[PR #8459] + +## SERV-RELEASE-KV-1b — 释放 KV memory 必须在完成 pause 后经 EngineCore 执行 + +- 触发：修改 `release_kv_cache_memory`、stage RPC 路由、pause/wake lifecycle。 +- 强制：仅选择 AR stages；要求 frontend pause 已完成且 `_admitting==0`，Stage 0 被选择 + 时先清 frontend multimodal cache，再用 EngineCore helper 释放 KV。成功后记录 KV sleep， + 恢复前保持 admission hold；以 `wake_up(tags=["kv_cache"])` 恢复再 resume。 +- 禁止：仅调用 worker collective RPC 绕过 EngineCore paused/resident 检查；缺失 helper + 时报成功；释放 diffusion stage；RPC 或 pause failure 后伪造成功的 frontend 状态。 +- 验收：未 pause 拒绝、admission drain、AR/diffusion 混合选择、真实 helper 路由、 + unsupported helper/error propagation、KV wake tag 与 resume admission。mock 测试 + 只证明路由及状态合同，真实显存释放量需另外测量。^[PR #8459] + ## SERV-7a — upstream launcher 生命周期兼容必须保持模式检测语义 - 触发：upstream launcher、renderer warmup 或 shutdown 读取的 engine-client 属性变化。 @@ -92,3 +118,15 @@ confidence: high 已删除就宣称所有内部兼容路径已移除。 - 验收：公开 rejection 与 canonical forwarding 测试通过；每个暂留内部/direct 路径有独立 compatibility test，直到后续迁移显式删除。 ^[PR #5647] + +## SERV-7i — parsed logging settings 必须先于 Omni command validation 与 dispatch 生效 + +- 触发：升级vLLM logger初始化合同或修改Omni CLI/module API-server入口。 +- 强制：Omni CLI在parse_args后、recognized command validate前调用upstream + `configure_logging_from_args(args)`，然后dispatch；直接module API server也在parse后、 + model同步/后续validation与startup前调用同一入口。不得只依赖import-time logger defaults。 +- 禁止：只修 `vllm serve --omni` 而遗漏 `python -m ...api_server`，或在validate/dispatch后 + 才应用用户settings；logo可见不能替代level/config实际生效或模型startup证明。 +- 验收：两个真实entrypoint分别核对parsed settings→configure→validate/startup顺序，并用 + 显式logging参数检查效果。source的logo观察只支持初始化修复，未解决module入口仍显示 + vLLM logo的既有问题，也不提供其它功能/模型运行资格。^[PR #8518] diff --git a/knowledge/repos/vllm-omni/components/serving/rules-video-encoding.md b/knowledge/repos/vllm-omni/components/serving/rules-video-encoding.md new file mode 100644 index 000000000..057bbebe7 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/serving/rules-video-encoding.md @@ -0,0 +1,17 @@ +--- +title: "Borrowed RGB video encoding 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components] +sources: ["PR #7355"] +--- + +# Borrowed RGB video encoding 规则 + +## SERV-VIDEO-1a — Borrowed PyAV frame 必须有精确 eligibility 与最终配置来源 + +- 触发：修改 Videos API encoding options、remote diffusion metadata fallback 或 borrowed RGB frame生成。 +- 强制：enable_borrowed_frames默认false；API先读effective diffusion config，remote metadata缺失时回读resolved diffusion stage的diffusion_config/engine_args。共享validation准备所有frames后，仅uint8、3D、末轴3且每帧pixel/channel strides=(3,1)进入from_numpy_buffer，允许row padding；其他输入走原planar/legacyfallback。音频mux/sample-rate/codec options保持，预编码bytes直接返回。 +- 禁止：从metadata-onlyremoteclient丢掉明确stageoverride；对float/灰度/错stride借用；保留AVFrame却释放其backingstorage；把RGB→MP4的局部copy结果声称包括VAE/HTTP或所有shape性能提升。 +- 验收：inline/remote配置、关闭开关、row-paddedRGB、非RGB/uint8、audio mux、preencodedbytes与持有frame跨引用释放回归；比较hash与完整MP4decode，小payload和coldregistration单独计时。 ^[PR #7355] diff --git a/knowledge/repos/vllm-omni/components/serving/rules-video-stream-prefill.md b/knowledge/repos/vllm-omni/components/serving/rules-video-stream-prefill.md new file mode 100644 index 000000000..36afc5802 --- /dev/null +++ b/knowledge/repos/vllm-omni/components/serving/rules-video-stream-prefill.md @@ -0,0 +1,32 @@ +--- +title: "Realtime video 增量 prefill 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components, serving] +sources: ["PR #5894"] +confidence: high +--- + +# Realtime video 增量 prefill 规则 + +## VSP-PREFILL-1a — 增量 vision warmup 由 stage-0 APC 与 text-only 输出共同决定 + +- 触发：修改 video-stream session 的增量 prefill 选择、warmup prompt、frame window 或 query processor kwargs。 +- 强制：只有 stage-0 prefix caching 开启且本 session 不输出 audio 时使用增量路径；audio-output 保留 legacy query prompt。增量 warmup 只预填当前已解码 frame prefix，不混入本次问题/音频；query 使用同一帧序列，不能再按 `num_frames` 子采样。`use_audio_in_video=True` 时，warmup 与增量 query 都传相同 processor kwarg；legacy query 仍仅在确有 input audio 时传递。 +- 禁止：添加与上述实际能力不一致的 session 开关；把 text-only cache 命中推广为 audio-output 或 duplex conversation-item 的支持；将 cap 处丢弃旧半窗描述为对整条流 resample，或据有界窗口测量声称无限 history 的 TTFT 不增长。 +- 验收：覆盖 APC/text-only 与 audio-output 路径、warmup/query kwargs 一致性、cap 前相同窗口及 cap 后 half-window eviction；增量等待完整窗口 decode，legacy 仅等待所选 frames。^[PR #5894] + +## VSP-FRAME-1a — warmup 身份、cache lifetime 与 consumed metadata 绑定同一帧 snapshot + +- 触发：修改 video frame FIFO、重复帧 cache、延迟/失败 decode、warmup 去重或 `video.frames.consumed`。 +- 强制：context signature 包含完整有序 frame identity，不能仅用长度和末帧。重复 buffered frame 共享 cache entry，任一副本仍在 buffer 或被 in-flight query/warmup pin 住时不得回收；failed-decode marker/readiness 也遵守该 lifetime。一次选择成功解码的 frame indices，同时用于 prompt image parts 与 consumed metadata，释放最后 pin 后补做 deferred eviction。 +- 禁止：FIFO 移走一个副本就删除共享 entry；在 query 等待期间撤销 BAD marker；从一份 selection 生成 metadata、另一份 selection 构造图片；以相同末帧误判不同窗口已经 warmed。 +- 验收：覆盖 `[A,B]`→`[C,B]` 的 delayed warmup、重复帧 eviction、query snapshot 后发生 decode failure 与全部 pin 释放；consumed frame IDs/latest timestamp 只反映实际进入 prompt 的图片。^[PR #5894] + +## VSP-ABORT-1a — 取消 warmup/query 必须先跨过 stage-0 提交屏障 + +- 触发：在 multimodal preprocessing、warmup generate 或 query submission 尚未完成时取消/中断 video request。 +- 强制：新 query 先结束旧 warmup；preprocess 已更新 sender MM cache 时用 shield 等待它完成，将对应 request 提交 stage 0 后再传播取消并 abort。session-owned JPEG decode readiness 与 request task 分离，取消模型请求不得级联取消共享 decode。 +- 禁止：preprocess 记下 hash 后在 stage-0 接收前 abort，留下 receiver 未见过的 cache identity；以取消 warmup 破坏被 query pin 住的 decode；task 退出后遗留模型 request。 +- 验收：在 preprocess 与 submit 的确定性屏障分别注入取消，断言 stage-0 submit 先于 abort、后续 query 无孤立 hash/cache miss，共享 decode 仍可完成且任务/请求最终释放。^[PR #5894] diff --git a/knowledge/repos/vllm-omni/components/serving/rules-voice-management.md b/knowledge/repos/vllm-omni/components/serving/rules-voice-management.md new file mode 100644 index 000000000..357ae26be --- /dev/null +++ b/knowledge/repos/vllm-omni/components/serving/rules-voice-management.md @@ -0,0 +1,17 @@ +--- +title: "Speech voice management 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, components] +sources: ["PR #7795"] +--- + +# Speech voice management 规则 + +## SERV-VOICE-1a — Voice deletion 必须按异常类型区分 builtin 与不存在 + +- 触发：修改 /v1/audio/voices delete、uploaded speaker storage 或错误映射。 +- 强制：在 upload lock 内以规范化name判断，builtin集合排除uploaded项；builtin/default删除抛InvalidPresetVoiceReferenceError，未知name抛InvalidVoiceReferenceError，API先捕preset子类映射403/ForbiddenError，再捕父类映射404/NotFoundError，成功返回明确成功响应。uploaded删除继续清metadata/data-URI/speakerartifactcache，unlink失败保持带voice name的warning。 +- 禁止：以message substring决定403/404；将builtin当不存在返回404；把async delete恢复为无法表达原因的bool；只在testclient中模拟抛异常而不检查实际HTTPstatus/body。 +- 验收：builtin、default、unknown（含带not found文字的合法name）、uploaded成功与unlink失败分别覆盖route和handler；错误类继承/捕获顺序和case-normalizedname仍正确。 ^[PR #7795] diff --git a/knowledge/repos/vllm-omni/components/serving/rules.md b/knowledge/repos/vllm-omni/components/serving/rules.md index b0256cfa2..d25787d90 100644 --- a/knowledge/repos/vllm-omni/components/serving/rules.md +++ b/knowledge/repos/vllm-omni/components/serving/rules.md @@ -23,7 +23,7 @@ confidence: high | batched chat、fan-out/fan-in、sub-request ID、choice collapse、whole-batch error | `batch-chat-contract`：`SERV-4i`–`4k` | `api_server.py::create_batch_chat_completion` → `batch_serving.py::OmniOpenAIServingChatBatch` → ordinary chat completion children | | `chat_template_kwargs`、raw HTTP/SDK `extra_body`、text/audio modalities、choices、空音频 | `chat-multimodal-contract`：`SERV-4c` + 命中模型规则 | upstream `ChatCompletionRequest` → `serving_chat.py::{OmniOpenAIServingChat._preprocess_chat,OmniOpenAIServingChat.chat_completion_full_generator,OmniOpenAIServingChat._create_text_choice,OmniOpenAIServingChat._create_audio_choice}` | | endpoint restriction、route/app-state guard、capability、公开 400 | `endpoint-capability`：`SERV-4c`, `SERV-4d`, `SERV-5d`, [SERV-5s](rules-app-assembly.md#serv-5s-覆盖-upstream-http-route-前必须先移除同-methodpath-的旧-route) | endpoint policy → `api_server.py::build_app` assembled app → public handler | -| pause/resume、sleep/wake、partial stage/tag、ACK、generation admission、abort cleanup acknowledgment 或 shutdown-time abort race | `engine-lifecycle`：`SERV-5a`, `SERV-5b`, `SERV-5m`；[SERV-5t](rules-abort-lifecycle.md#serv-5t-acknowledged-abort-在-shutdown-边界只忽略已识别的-transport-closure) | `entrypoints/async_omni.py::{AsyncOmni.pause_generation,AsyncOmni.resume_generation,AsyncOmni.sleep,AsyncOmni.wake_up,AsyncOmni._abort}` → `engine/{async_engine_utils,stage_pool,orchestrator,async_omni_engine}.py` → AR EngineCore helper / diffusion worker RPC | +| pause/resume、sleep/wake、partial stage/tag、ACK、generation admission、abort cleanup acknowledgment 或 shutdown-time abort race | `engine-lifecycle`：`SERV-5a`；[sleep/wake ACK 与 admission](rules-sleep-control.md)：`SERV-5b`, `SERV-5m`；[SERV-5t](rules-abort-lifecycle.md#serv-5t-acknowledged-abort-在-shutdown-边界只忽略已识别的-transport-closure) | `entrypoints/async_omni.py::{AsyncOmni.pause_generation,AsyncOmni.resume_generation,AsyncOmni.sleep,AsyncOmni.wake_up,AsyncOmni._abort}` → `engine/{async_engine_utils,stage_pool,orchestrator,async_omni_engine}.py` → AR EngineCore helper / diffusion worker RPC | | stage/replica death、remote membership register/watch/detach、`EngineDeadError`、`StageUnavailableError`、health/readiness、eviction | `engine-lifecycle`：[SERV-5f](rules-fault-isolation.md#serv-5f-stage-death-按-replica-隔离request-fatal-不等于-process-fatal)、[SERV-5n](rules-engine-lifecycle.md#serv-5n-remote-replica-attachdetach-必须按本次-membership-generation-判定) | `engine/membership_controller.py` register/watcher generation → `engine/stage_pool.py` live set → `engine/orchestrator.py` poll/dispatch/cleanup → `entrypoints/{omni_base,async_omni}.py` error transport | | serving class/factory 重构、optional adapter、diffusion/no-TTS 实例、warmup | `engine-lifecycle`：`SERV-5c` | `entrypoints/openai/serving_speech.py` 的所有 factory/`__new__` 路径 → `warmup`、voice upload/list、speech request caller | | vLLM rebase、chat parser/renderer API moved or removed | `engine-lifecycle` | target-release `vllm.parser.*` / renderer API → `serving_chat.py` / `api_server.py`; remove a compatibility fallback only after the pinned base image is the target release ^[PR #6606] | @@ -47,7 +47,7 @@ confidence: high | `artifact-readiness` | artifact/content cache、capability、ready/mark/discard | `SERV-3a`, `SERV-3b`, `SERV-3c` | | `chat-multimodal-contract` | chat template kwargs、SDK flatten、text/audio response shape | `SERV-4c`（见 [请求输入合同](rules-request-input.md)）+ 命中模型规则 | | `endpoint-capability` | endpoint restriction、route/app-state guard、公开 400 | `SERV-4c`, `SERV-4d` 见 [请求输入合同](rules-request-input.md)；`SERV-5d` 见 engine lifecycle；`SERV-5s` 见 [app assembly](rules-app-assembly.md) | -| `engine-lifecycle` | pause/resume、sleep/wake、partial stage/tag、ACK、generation admission、abort cleanup/shutdown race、streaming raw terminal、event-driven orchestration、factory 状态矩阵、TTS adapter detection、replica membership/fault isolation | `SERV-5a`–`SERV-5e`、`SERV-5g`–`SERV-5r`（见 [engine 生命周期规则](rules-engine-lifecycle.md)）、[SERV-5t](rules-abort-lifecycle.md#serv-5t-acknowledged-abort-在-shutdown-边界只忽略已识别的-transport-closure)，`SERV-5f` | +| `engine-lifecycle` | pause/resume、sleep/wake、partial stage/tag、ACK、generation admission、abort cleanup/shutdown race、streaming raw terminal、event-driven orchestration、factory 状态矩阵、TTS adapter detection、replica membership/fault isolation | `SERV-5a`、`SERV-5c`–`SERV-5e`、`SERV-5g`–`SERV-5r`（`SERV-5m` 见 [sleep control](rules-sleep-control.md)，其余见 [engine 生命周期规则](rules-engine-lifecycle.md)）、[SERV-5t](rules-abort-lifecycle.md#serv-5t-acknowledged-abort-在-shutdown-边界只忽略已识别的-transport-closure)，`SERV-5f` | | `full-duplex` | duplex opt-in、stage prewarm/fence、async-chunk、CFG companion、stage-keyed segment lifecycle，adapter-owned deferred cognition/session cleanup，或 startup warmup/realtime admission | `SERV-6a`–`SERV-6f` 见 [engine 生命周期规则](rules-engine-lifecycle.md)；`SERV-6g` 见 [duplex 编排规则](rules-duplex-orchestration.md)；`SERV-6h`、`SERV-6i` 见 [session lifecycle](rules-session-lifecycle.md) | | `request-contract` | 请求字段、来源、冲突、dispatcher、consumer view、stage sampling constraints、serialization bounds、transform artifact ownership | `SERV-4a`–`4h`, `SERV-4l`–`4q`，全部见 [请求输入合同](rules-request-input.md) | | `batch-chat-contract` | frontend fan-out、identity、choice cardinality、error/cancellation | `SERV-4i`–`4k`，见 [batch chat rules](rules-batch-chat.md) | diff --git a/knowledge/repos/vllm-omni/models/_index.md b/knowledge/repos/vllm-omni/models/_index.md index 682c64f4c..566303450 100644 --- a/knowledge/repos/vllm-omni/models/_index.md +++ b/knowledge/repos/vllm-omni/models/_index.md @@ -32,3 +32,5 @@ PersonaPlex 的 staged 与 unified full-duplex 双入口、Mimi 状态和 lockst 选择一个模型 owner 后停止枚举其他模型。跨模型共用的不变量应进入 [components](../components/_index.md)，不要复制到多个模型目录。 + +- [MAGI-2 Preview本地BF16routedMoE](magi2/_index.md)。 diff --git a/knowledge/repos/vllm-omni/models/auk/_index.md b/knowledge/repos/vllm-omni/models/auk/_index.md new file mode 100644 index 000000000..d7f5bfb56 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/auk/_index.md @@ -0,0 +1,21 @@ +--- +title: "Tencent AuK / AuK-Flash" +created: 2026-10-06 +updated: 2026-10-06 +type: index +tags: [vllm-omni, models, diffusion] +sources: ["PR #7469", "PR #7881", "PR #8300", "PR #8328", "PR #8305"] +confidence: high +--- + +# Tencent AuK / AuK-Flash + +本 owner 对应 `diffusion/models/auk/` 的 DiT/codec 与 +`entrypoints/openai/tts_adapters/auk.py` 的 instruction-driven Speech API adapter。 + +| 遇到什么 | 查看哪里 | +|---|---| +| task shortcut、DiT graph/request context、codec tiling、reference latent cache 或 fused activation | [AuK 规则](rules.md) | + +共享执行查 [Diffusion owner](../../components/diffusion/_index.md)； +Speech API 的公共输入与输出查 [Serving owner](../../components/serving/_index.md)。 diff --git a/knowledge/repos/vllm-omni/models/auk/rules.md b/knowledge/repos/vllm-omni/models/auk/rules.md new file mode 100644 index 000000000..8bebb2871 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/auk/rules.md @@ -0,0 +1,126 @@ +--- +title: "AuK / AuK-Flash 请求与加速规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, diffusion] +sources: ["PR #7469", "PR #7881", "PR #8300", "PR #8328", "PR #8305"] +confidence: high +--- + +# AuK / AuK-Flash 请求与加速规则 + +## Direct 代码快速入口 + +| PR 描述信号 | 规则组 | 第一批源码 | +|---|---|---| +| Speech API、task_type、instruction、duration | AUK-1a | `tts_adapters/auk.py::AuKAdapter` → `model_extras/auk.py` | +| DiT graph、CFG、conditioning、time grid | AUK-2a / AUK-2b / AUK-2c | `diffusion/models/auk/cudagraph_wrapper.py` → `auk_transformer.py::prepare/step` | +| codec graph、tiling、halo、compiled bucket | AUK-3a | `vae_cudagraph.py::plan_tiles`、`AuKVAEDecodeGraph` → `auk_vae.py::decode_context_frames` | +| reference cache、encoder FULL graph、Triton Snake | AUK-3b / AUK-3c / AUK-3d | `pipeline_auk.py::_encode_source`、`deploy/auk.yaml`、`auk_vae.py::AliasFreeActivation` | + +## AUK-1a — task shortcut 必须在 adapter 中一次性转换并保留 benchmark 任务口径 + +- 触发：修改 AuK Speech API、`task_type`、instruction normalization 或 TTS benchmark mapping。 +- 强制：完整 `instructions` 优先；CustomVoice/Base shortcut 用 JSON quoting 包住 target + text，转换后清空 input 并消费 task_type，batch validate/build 的第二次 normalize 不再 + 套模板。entry stage 按 `AuKForConditionalGeneration` architecture 识别，不能只靠 + generic encoder stage 名。无 reference 必须有正有限 duration；自定义 t_grid 至少两项、 + 有限严格递增，在 encoder dispatch 前拒绝错误；seed/steps/CFG override 进入 diffusion stage。 + benchmark 由 model config 映射任务并在导出记录 task/task_type，通用 harness 不改模型 prompt。 +- 禁止：用单引号直接拼可能含引号的文本；把 input 当作任意 TTS transcript 自动推断 + instruction；按 served checkpoint path 选择任务；遗漏 reference/no-duration 边界。 +- 验收：单条与 batch 请求反复 normalize 结果相同；覆盖引号/反斜杠/Unicode、字段冲突、 + 完整 instruction、invalid duration/grid 和 stage override；benchmark 结果保留任务元数据。 + 特定语料 WER 不外推到任意 instruction/reference。^[PR #7469] + +## AUK-2a — 共享 pool 的 DiT graphs 必须整代退役，capture 失败不能重试损坏 generation + +- 触发：修改 AuK DiT graph cache capacity、capture exception、pool 或 replay buffer。 +- 强制：wrapper 使用自己的 graph pool；容量满且需新 shape 时整代清空，不能只销毁 + 一条仍与存活 graph 共享地址的 entry。capture 失败清空、禁用 wrapper 并传播当前 + 错误，后续请求才走 eager；replay 返回裁去 padding 后的 clone，避免被下一步覆盖。 +- 禁止：把 wrapper-private pool 单独当作逐 entry eviction 安全证明；`graph.reset()` + 后仍留共享地址的旧 peers；捕获失败后继续 replay 或每个新 shape 再试 capture。 +- 验收：填满 cache 再引入新 key，旧 generation 全部退役；注入 capture failure 后 + 当前失败且后续 eager；连续 replay 不改先前持有输出，实测 CUDA allocator lifetime。 + CPU mock 的 cache 检查不等于 GPU address reuse 已验收。^[PR #7469] + +## AUK-2b — graph 几何与 per-request conditioning 必须分离 + +- 触发：修改 DiT bucketing、prepare/step split、CFG、padding bias 或 compilation hook。 +- 强制：key 包含 bucketed target/text/reference geometry 与 CFG 模式；timestep 和同一 + 模式内的 CFG strength 更新静态 scalar buffer。每个新 request 重新 prepare text + projection、两支 reference embedding、padding bias/RoPE，再复制到当前 graph context； + graph 只捕获 step+CFG combine。full compile 编译 sampler 实际调用的 step；regional + compile 使用 double/single repeated blocks。startup warmup 按当前 variant 的 CFG 与 + 默认 time grid，覆盖有/无 reference；显式空 warmup 列表可禁用。 +- 禁止：相同 geometry 即复用上一 prompt 的 context；按旧 64-frame alignment 宣称最新 + bucket 仍不变（后续为 32/32/50）；编译未被 sampler 调用的 forward；把 cuDNN SDPA + preference 当作所有 shape/backend 的恒定收益。 +- 验收：同 bucket 换 prompt/reference/CFG strength、跨 bucket 后返回、Base/Flash 与 + warmup→首请求均对照 eager 的 condition/velocity；compile 和 capture 的 startup 成本 + 与稳态 latency 分开测量。^[PR #7469] ^[PR #8300] + +## AUK-2c — 预计算 adaLN 必须随请求的 time grid 刷新，step index 是图内可变输入 + +- 触发：修改 timestep modulation、time-grid 长度、图 key 或 conv position embedding。 +- 强制：prepare 按整个当前 time grid 计算各层 shift/scale/gate；key 包含 grid length， + 相同长度但不同 grid 的新请求也必须刷新 context。每一步更新静态 step-index buffer + 选择对应 modulation 行。unfold+batched GEMM 的 grouped conv 保持原 Conv1d 几何与边界。 +- 禁止：将首步 modulation 固定进 graph；只因 grid length 相同就沿用旧 grid；把 + 特定 H20/H200 测量外推成 Flash 四步或任意设备的相同 latency 收益。 +- 验收：同长度不同时间值、不同长度、所有 step 的 eager/graph velocity 对比，连同 + grouped Conv1d 的独立数值 oracle；warmup 用 variant 默认 grid 匹配真实 key。^[PR #8328] + +## AUK-3a — codec tiling 必须保留 decoder halo、buffer lifetime 与实际 fallback + +- 触发：修改 VAE compile buckets、decode_tiles、tile size 或 graph failure handling。 +- 强制：halo 从 decoder receptive-field 计算，tile 大于左右 context 总和；每窗仅贡献 + 有完整 context 的 interior，真片边界例外。选最小成功 capture 的 compiled bucket， + 右侧 pad 后 crop，尾窗可选更小 bucket。plain graphs 共享 private pool 且整代退役； + compiled warmup failure 可选其他 bucket/plain path，运行期 plain capture/replay 错误 + 仍传播。保留跨 iteration 的 graph-backed chunk 必须复制；latent output 跳过 codec。 +- 禁止：把 decode_tiles helper 当作 HTTP streaming 已启用；声称 tiling 令总输出内存 + 与长度无关（pipeline 仍组装完整 waveform）；把 compiled fusion/right padding 的 + tolerance 当作 plain exact-length graph 的 bit equality；凭空承诺 runtime eager fallback。 +- 验收：独立 whole eager oracle 检查 halo 内部/真边界、短尾窗、禁用 tile、batched/CPU/ + outer-capture eager 路径与持有 chunk；compiled/tiling 采用数值 tolerance 并检查尾部。 + 配置 bucket 是参数而非固定能力，后续 deployment 可改为 160/320/640。^[PR #7881] + +## AUK-3b — reference latent cache 只缓存 prepared waveform 的 posterior mean + +- 触发：修改 AuK reference audio 预处理、LRU key、capacity 或 VAE sample。 +- 强制：content digest 与 prepared waveform 长度共同作 key；只缓存 deterministic + posterior mean，随机 posterior draw 每次重新计算且保留 generator。capacity 是非负 + integer，bool/float/字符串非法，显式 0 禁用；LRU 有界。返回 cached tensor 的 caller 不得写入。 +- 禁止：按 path、tensor address 或原始压缩音频 bytes 缓存；缓存随机 posterior；用 int() + 静默截断 capacity 或负值变禁用；后处理原地污染共享 latents。 +- 验收：相同内容复用、内容/长度改变 miss、LRU eviction、sample 绕过、关闭 cache 与 + invalid capacity；同 seed 的随机路径与无 cache oracle 一致。^[PR #8305] + +## AUK-3c — encoder FULL graph 验收必须观测跨请求的真实 stage handoff + +- 触发：修改 AuK stage 0 capture sizes、FULL graph 或 prompt condition transfer。 +- 强制：交替不同 prompt 与 token length，覆盖同 bucket、跨 bucket 后返回，逐请求 + 比较 stage 0 交给 DiT 的 text condition 与 eager。只变 encoder 的实验将 stage 1 设为 + eager，保留真实 stage handoff，避免无关 DiT/codec compile 成本盖住边界。 +- 禁止：只检查 audio valid 就证明 prompt condition 没被静态 buffer 覆盖；用同一 + prompt 重复请求代替 bucket 变化；为测试再引入未使用的生产控制 API。 +- 验收：assert 实际进入预期 capture buckets、condition cosine/误差和重复 prompt 一致性； + 单次报告阈值与硬件范围明确，不能外推所有 tokenizer/model variant。^[PR #8305] + +## AUK-3d — fused alias-free activation 必须按真实 eligibility 保留 eager + +- 触发：修改 SnakeBeta upsample/activation/FIR fusion、Triton import 或 kernel indexing。 +- 强制：仅 CUDA fp32、3-D 输入、SnakeBeta 与匹配 filter/crop geometry 使用 fused path； + 共享 `vllm.triton_utils` 的 availability 控制 import/registration，其他输入保持 eager。 + parity oracle 直接执行 downsample(act(upsample(x)))，不增加运行时 fusion toggle。 +- 禁止：不匹配 geometry 或非 fp32 强行调用 kernel；将特定 SNR/activation error 扩大成 + 任意 waveform 逐 bit 相同；忽略 stride、partial block 或修改输入 buffer。 +- 验收：长度 511/512/513 与 1023/1024/1025、短输入、contiguous/transposed/sliced、 + causal 与非 causal 对照独立 eager，并断言 input 未修改；缺 Triton/CPU fallback 可导入。 + 性能收益与质量通过真实模型另核验。^[PR #8305] + +公共 request/result ownership 见 [Serving 规则](../../components/serving/rules.md)； +共享 cache 与编译生命周期见 [Diffusion 规则](../../components/diffusion/rules.md)。 diff --git a/knowledge/repos/vllm-omni/models/bagel/_index.md b/knowledge/repos/vllm-omni/models/bagel/_index.md index 62b6cb2a2..ccf1fcd06 100644 --- a/knowledge/repos/vllm-omni/models/bagel/_index.md +++ b/knowledge/repos/vllm-omni/models/bagel/_index.md @@ -1,7 +1,7 @@ --- title: "BAGEL（统一模型多形态部署参照）" created: 2026-07-21 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, models, diffusion] sources: ["PR #6359", "PR #7049", vllm_omni/model_executor/models/bagel/, vllm_omni/diffusion/models/bagel/, vllm_omni/deploy/bagel.yaml] @@ -44,7 +44,7 @@ sources: ["PR #6359", "PR #7049", vllm_omni/model_executor/models/bagel/, vllm_o | 遇到什么 | 查看哪里 | 说明 | |---|---|---| | KV 桥接、3 路 CFG、MoT、变体拓扑 | [architecture](architecture.md) | AR→DiT 数据流与 reviewer 陷阱 | -| CFG 分支 position ID 打包、BAGEL-3 step wave | [rules](rules.md) | 1-D RoPE 与 Lance mRoPE 的序列轴合同；BAGEL 专有 step/packing、有效几何与 fallback 清理 新增核对：BAGEL-4。 | +| CFG position ID、step wave、输出画布或复制相位的 SP dispatch | [rules](rules.md) | BAGEL-1–BAGEL-5：序列轴、step/packing、有效几何、fallback 清理和 prefill/cache-update 的 local attention。 | ## 配置与 checkpoint 差异 diff --git a/knowledge/repos/vllm-omni/models/bagel/rules.md b/knowledge/repos/vllm-omni/models/bagel/rules.md index c504b5431..ef7e4d8aa 100644 --- a/knowledge/repos/vllm-omni/models/bagel/rules.md +++ b/knowledge/repos/vllm-omni/models/bagel/rules.md @@ -1,10 +1,10 @@ --- title: "BAGEL 实现规则" created: 2026-09-02 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, models, diffusion] -sources: ["PR #5775", "PR #5884", "PR #6359", "PR #7049", vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/diffusion_engine.py, vllm_omni/diffusion/models/bagel/bagel_transformer.py, vllm_omni/diffusion/models/bagel/pipeline_bagel.py, vllm_omni/diffusion/models/lance/lance_transformer.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/test_diffusion_engine_dummy_run.py, tests/diffusion/models/bagel/test_step_execution.py, "PR #7287"] +sources: ["PR #5775", "PR #5884", "PR #6359", "PR #7049", vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/diffusion_engine.py, vllm_omni/diffusion/models/bagel/bagel_transformer.py, vllm_omni/diffusion/models/bagel/pipeline_bagel.py, vllm_omni/diffusion/models/lance/lance_transformer.py, vllm_omni/diffusion/worker/diffusion_model_runner.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/test_diffusion_engine_dummy_run.py, tests/diffusion/models/bagel/test_step_execution.py, "PR #7287", "PR #8177"] confidence: high --- @@ -15,6 +15,7 @@ confidence: high | PR 描述信号 | 规则 | |---|---| | CFG position ID、nested DiT owner、step image wave/request state | `BAGEL-1`–`BAGEL-3` | +| prefill、ViT/VAE 或 CFG cache-update 在 SP 下重复进入 collective | `BAGEL-5` | ## BAGEL-1：CFG position ID 沿序列轴合并 @@ -101,3 +102,10 @@ regression 必须对 request 和 step 两种 mode 都断言两个 inference step **合同**：画布优先级为 **显式请求 > `kv_metadata["image_shape"]` > 输入图像推导**。`size=auto` 在 API 层解析为源尺寸后，必须用 `height_not_provided`/`width_not_provided` 标记非意图画布；preprocessor 对派生画布同样置位。显式边长按 latent stride 向下对齐并告警；超过 checkpoint 上限以 `OmniClientError`（HTTP 400）拒绝。源图仍可为 VAE/ViT prefill 做 stride 对齐，但不得覆盖已请求的输出画布。非流式 generation/edit 响应的 `size` 必须来自实际生成图像。 **验收**：显式 `size` 不被源图或 AR 发布的 KV shape 覆盖；`size=auto` 与纯离线 img2img 保持派生对齐；越界拒绝；响应 `size` 与解码图像一致。^[PR #7287] + +## BAGEL-5 — 复制的 prefill/cache-update 相位必须跳过 SP strategy + +- 触发：修改 `PackedAttentionMoT` 的 `_forward_und`、`_forward_gen`、attention 构造或 CFG cache-update。 +- 强制：每个 SP rank 持有完整序列的 causal prefill、ViT/VAE 和 CFG cache-update 使用 `skip_sequence_parallel=True` 的 causal/local noncausal attention；参数无关的 local noncausal 层与 strategy 层分开。`_forward_gen` 仅在 SP active 且 `not update_past_key_values` 时进入 joint denoise strategy。 +- 禁止：把复制的完整序列当 shard 送入 Ulysses all-to-all；让 `_forward_und` 进入 strategy；为修复 cache-update 而关闭原本分片的 denoise 路径。 +- 验收：断言构造期 causal/local opt-out 与 joint noncausal 的 opt-in；覆盖 SP cache-update、SP denoise、非 SP 和 causal/noncausal prefill 的实际 dispatch 与 KV shape。^[PR #8177] diff --git a/knowledge/repos/vllm-omni/models/cosmos3/_index.md b/knowledge/repos/vllm-omni/models/cosmos3/_index.md index 2a95ee543..4b7df8e1c 100644 --- a/knowledge/repos/vllm-omni/models/cosmos3/_index.md +++ b/knowledge/repos/vllm-omni/models/cosmos3/_index.md @@ -1,7 +1,7 @@ --- title: "Cosmos3" created: 2026-07-20 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, models, diffusion] sources: ["PR #4657", "PR #5001", "PR #5634", "PR #6920", docs/features/session_state_manager.md, recipes/cosmos3/Cosmos3-Nano.md, vllm_omni/diffusion/models/cosmos3/, vllm_omni/diffusion/models/cosmos3/pipeline_cosmos3.py, vllm_omni/experimental/world_models/adapters/state_cosmos3_adapter.py, vllm_omni/platforms/rocm/platform.py, tests/diffusion/models/cosmos3/test_cosmos3_pipeline.py] @@ -57,3 +57,5 @@ request 的 denoise 生命周期内使用，不提供跨请求记忆或并发安 - 审查 Cosmos3 Edge/Distilled scheduler、RNG、guidance、offload、distributed-VAE multi-chunk transfer 或 capability claim。 - 同一问题影响多个 diffusion 模型时返回 [Diffusion rules](../../components/diffusion/rules.md)。 + +sampling state FP32 与 transformer dtype boundary：COSMOS-8a；沿本目录 rules 审查入口核对。 diff --git a/knowledge/repos/vllm-omni/models/cosmos3/rules.md b/knowledge/repos/vllm-omni/models/cosmos3/rules.md index ba36581b3..4deed4190 100644 --- a/knowledge/repos/vllm-omni/models/cosmos3/rules.md +++ b/knowledge/repos/vllm-omni/models/cosmos3/rules.md @@ -1,10 +1,10 @@ --- title: "Cosmos3 规则" created: 2026-07-20 -updated: 2026-09-26 +updated: 2026-10-06 type: rule tags: [vllm-omni, models, diffusion] -sources: ["PR #4657", "PR #5001", "PR #5634", "PR #6049", docs/features/session_state_manager.md, recipes/cosmos3/Cosmos3-Nano.md, vllm_omni/diffusion/models/cosmos3/, vllm_omni/model_extras/cosmos3.py, vllm_omni/model_extras/registry.py, vllm_omni/experimental/world_models/adapters/state_cosmos3_adapter.py, vllm_omni/platforms/rocm/platform.py, tests/diffusion/models/cosmos3/test_session_memory_equivalence.py, tests/diffusion/models/cosmos3/test_cosmos3_pipeline.py, "PR #6107", "PR #5614", "PR #6325", "PR #6913", "PR #6920", "PR #7427", "PR #7971"] +sources: ["PR #4657", "PR #5001", "PR #5634", "PR #6049", docs/features/session_state_manager.md, recipes/cosmos3/Cosmos3-Nano.md, vllm_omni/diffusion/models/cosmos3/, vllm_omni/model_extras/cosmos3.py, vllm_omni/model_extras/registry.py, vllm_omni/experimental/world_models/adapters/state_cosmos3_adapter.py, vllm_omni/platforms/rocm/platform.py, tests/diffusion/models/cosmos3/test_session_memory_equivalence.py, tests/diffusion/models/cosmos3/test_cosmos3_pipeline.py, "PR #6107", "PR #5614", "PR #6325", "PR #6913", "PR #6920", "PR #7427", "PR #7971", "PR #7592"] confidence: high --- @@ -173,3 +173,11 @@ confidence: high - 强制：T2I、video 与 policy checkpoint 共享 `model_type=cosmos3_omni` 和同一 `model_index.json` `_class_name`。`OMNI_PIPELINES` 不得注册可被该 metadata 自动命中的 `cosmos3_omni`。policy 用 `cosmos3_policy`，omni overlay 用 `cosmos3_omni_deploy`，两者都不声明 `hf_architectures` / `diffusers_class_name`，只经 YAML `pipeline:` 选中。单 stage overlay 必须 `async_chunk: false`；`guardrails: false` 等 stage extras 只有选中该 key 后才会合并。无 `--deploy-config` 时 T2I/video 仍走 CLI 单 stage fallback；`--no-guardrails` 仍是 CLI-only 路径。 - 禁止：为了让 `--deploy-config` 生效而把 `cosmos3_omni` 做成 auto-detect（会吞掉 policy/video）；静默丢弃 deploy YAML 却宣称 guardrails/stage extras 已生效。 - 验收：断言 `cosmos3_omni` 不在 registry、opt-in key 无 auto-capture 字段；`cosmos3_omni.yaml` 经 `merge_pipeline_deploy` 得到 `guardrails is False` 与 `final_output_type="video"`。^[PR #7971] + + +## COSMOS-8a — 采样状态与 transformer forward 必须分开控制 dtype + +- 触发：修改 Cosmos3 noise、scheduler/CFG state、conditioning mask 或 transformer 输入/输出 cast。 +- 强制：默认 sampling_dtype 为 FP32；初始 latent、条件图像/控制 mask、action/sound 和 scheduler 更新沿该 dtype 保持。仅在 predict_noise 的指定模型输入边界递归转 model dtype，transformer tensor/tuple 输出再转 sampling dtype。 +- 禁止：把默认 sampling state 随 BF16 权重整体降精度；遗漏 nested list/tuple 的输入 cast，或让 transformer 输出 dtype 反向决定 scheduler dtype。代码允许显式 sampling_dtype override，不能把默认策略写成禁止任何 override。 +- 验收：断言默认 FP32 state、BF16 model boundary 和 FP32 prediction return；覆盖 tensor/tuple、nested media、action/sound、transfer 以及显式同 model dtype 的 control。改变默认精度还须固定输入与 seed 比较输出质量，shape smoke 不能替代。^[PR #7592] diff --git a/knowledge/repos/vllm-omni/models/cosyvoice3/_index.md b/knowledge/repos/vllm-omni/models/cosyvoice3/_index.md index fc987e266..593f8c78e 100644 --- a/knowledge/repos/vllm-omni/models/cosyvoice3/_index.md +++ b/knowledge/repos/vllm-omni/models/cosyvoice3/_index.md @@ -50,6 +50,9 @@ sources: ["PR #5673", "PR #5869", "PR #6424", "PR #6955", vllm_omni/data_entry_k | 遇到什么 | 查看哪里 | 说明 | |---|---|---| +| live conditioning、full payload 与 token-only streaming | [rules-conditioning-handoff](rules-conditioning-handoff.md) | 可执行约束与验收 | +| 有界 HiFT graph、precision signature 与 capability profile | [rules-streaming-graphs](rules-streaming-graphs.md) | 可执行约束与验收 | +| packed Flow/HiFT、ragged frames、reference cache、RAS/standard | [rules-packed-inference](rules-packed-inference.md) | 可执行约束与验收 | | RAS 停止机制、双交接注册、TRT 门 | [architecture](architecture.md) | 数据流与 reviewer 陷阱 | | CosyVoice3 TensorRT CFM 的 stream handoff、allocator lifetime、context pool、plan 并发发布、tmp 所有权或 cleanup failure | [rules](rules.md) | `COSYVOICE3-1a/1b` 的顺序、发布边界与验收 新增核对：COSYVOICE3-2a、COSYVOICE3-1c。 | @@ -71,3 +74,5 @@ sources: ["PR #5673", "PR #5869", "PR #6424", "PR #6955", vllm_omni/data_entry_k - 审查 cosyvoice3 的停止逻辑、流式 chunk 数学或 TRT 引擎构建;跨 GPU 部署 性能问题先读 YAML 头注的连续性数据。 - 语义验收见 [model-validation](../../review/guides/model-validation.md)。 + +- [标准FP32 F0、TF32恢复与nativevocoder](rules-f0-device.md)。 diff --git a/knowledge/repos/vllm-omni/models/cosyvoice3/rules-conditioning-handoff.md b/knowledge/repos/vllm-omni/models/cosyvoice3/rules-conditioning-handoff.md new file mode 100644 index 000000000..90e0b168b --- /dev/null +++ b/knowledge/repos/vllm-omni/models/cosyvoice3/rules-conditioning-handoff.md @@ -0,0 +1,24 @@ +--- +title: "CosyVoice3 conditioning handoff 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models] +sources: ["PR #8343", "PR #8473"] +--- + +# CosyVoice3 conditioning handoff 规则 + +## COSY-PAYLOAD-1a — Live conditioning 必须在 runner replay 后按当前请求重建 + +- 触发：修改 talker output、code2wav full-payload adapter、conditioning unwrap 或输出 accumulation policy。 +- 强制：talker forward 统一返回 hidden tensor，两个 runner 在 replay 后附加当前请求 conditioning，不能捕获某次 prefill 的 payload。接受 Mapping（含结构化 payload）；单请求列表/tuple conditioning 必须恰有一项，未拆 batch 明确拒绝。`speech_token`、`speech_feat`、`embedding`、`speech_token_len` 保留非空 tensor 与真实长度；full replacement/REPLACE accumulation 默认在 output module 定义，不能依赖 worker processor import 才生效。 +- 禁止：只对普通 dict 保留字段；在 graph 中持有可过期的 prefill conditioning；拼接不同 chunk 的变长 reference；遗漏真实 speech_token_len 或误把多请求列表当单请求。 +- 验收：V1/MRv2、eager/replay、混合 prefill/decode、不同宽度 reference、Mapping payload 和单项/多项列表分别覆盖；同请求 conditioning 更新可见且不同请求不能交叉。 ^[PR #8343] + +## COSY-TOKEN-1a — Token-only chunk 必须继续推进 streaming transfer + +- 触发：修改 async CosyVoice talker hidden payload、`requires_token_updates` 或 AR transfer 条件。 +- 强制：async talker 可关闭 pooler hidden payload，code2wav 仍消费 codec IDs 与 conditioning；processor 明确声明 requires_token_updates。即使没有 inter-stage tensor payload，只要有新 token 且 processor 需要更新，或请求停止/结束，就执行 chunk transfer；stage-0-final exclusion 在 free_request 修改 metadata 前计算。 +- 禁止：以空 hidden payload 作为跳过所有传输的条件；把 token-only 优化用于仍需要 hidden 的模型；释放 request 后再判断 final-stage 身份而丢失 terminal/update。 +- 验收：有 token 无 tensor、无 token terminal、普通 tensor 更新、stage-0-final 与另一 hidden-dependent model 分别回归；code2wav 接收的 token 和尾块与优化前一致。 ^[PR #8473] diff --git a/knowledge/repos/vllm-omni/models/cosyvoice3/rules-f0-device.md b/knowledge/repos/vllm-omni/models/cosyvoice3/rules-f0-device.md new file mode 100644 index 000000000..9eb2702ef --- /dev/null +++ b/knowledge/repos/vllm-omni/models/cosyvoice3/rules-f0-device.md @@ -0,0 +1,17 @@ +--- +title: "CosyVoice3 F0 device 与 native vocoder 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models] +sources: ["PR #7518"] +--- + +# CosyVoice3 F0 device 与 native vocoder 规则 + +## COSY-F0-1a — F0 device 优化必须保留标准 FP32 路径与 TF32 恢复 + +- 触发：修改 causal HiFT F0 inference、COSYVOICE3_F0_ON_CPU、causal convolution cache 或 STFT window。 +- 强制：标准loader的predictor保持FP32；CUDA且无CPUescape时在speech_feat设备执行，迁移时明确FP32，input.float后结果回输入dtype。临时禁matmul/cuDNN TF32，finally恢复两个旧flag；CPUescape/非CUDA用CPU float输入并返回原device/dtype。使用x.new_zeros/f0.new_zeros继承device/dtype，CausalConv1d默认cache=None；periodic FP32 torch.hann_window为nonpersistentbuffer，整数math.prod/cumulative scale保持原upsample/crop。 +- 禁止：把标准FP32加载的正确性外推为任意.half/混合dtype模块都自动修复；保留默认CPUtensor再每块to(device)；使TF32异常后泄漏；用duration一致/高cosine声称waveformbitidentity或通用CUDAgraph可用。原有有界mel/phase合同保持COSYVOICE3-1c，后续packed load→move→fold合同保持COSY-PACKED-1b。 +- 验收：stream/finalize voicedF0、CPUescape、默认GPU、predictorplacement、TF32开启/异常后恢复、causalcache与periodicwindow数值分别覆盖；用显式容差检查CPU/GPUwaveform与总samples，性能和graph证据独立归属实际revision。 ^[PR #7518] diff --git a/knowledge/repos/vllm-omni/models/cosyvoice3/rules-packed-inference.md b/knowledge/repos/vllm-omni/models/cosyvoice3/rules-packed-inference.md new file mode 100644 index 000000000..92274afcd --- /dev/null +++ b/knowledge/repos/vllm-omni/models/cosyvoice3/rules-packed-inference.md @@ -0,0 +1,38 @@ +--- +title: "CosyVoice3 packed inference 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models] +sources: ["PR #8224"] +--- + +# CosyVoice3 packed inference 规则 + +## COSY-PACKED-1a — Packed Flow 必须保持有效帧、请求顺序与数值边界 + +- 触发：修改 opt-in packed full-response/streaming Flow、ragged prompt packing、batched HiFT 或混合 prefill/decode conditioning。 +- 强制：packed 模式仅在对应开关启用且 CUDA Hopper capability major 为 9 时生效；保留 10-step Euler，Flow 可用 BF16，F0 与 HiFT 保持 FP32。每行先拼 prompt 和 continuation 再右侧 padding；长度分组、finalize 分组后恢复原请求顺序，输出按每行有效长度裁剪。conditioning 按完整 batch 的 request slot 对齐，非 prefill 行保留空位。packed Flow 绕过 Flow TRT，不能据此禁用独立的 speaker TRT。 +- 禁止：把 reference padding 当有效帧；独立 padding prompt/continuation 留中间空洞；把 grouped 输出按排序顺序发布；沿用旧函数注释声称主线 F0 用 FP64；把 opt-in 变成所有设备默认或宣称 RAS/standard 音质等价。 +- 验收：对未修改的 Flow 参考路径覆盖异长 prompt、batch 大于 8、混合 prefill/decode、full-response/streaming 与 chunk-causal attention；核对有效帧、输出顺序与数值容差，真实 compile/cold-start 另测，不能只靠同实现自比较或 monkeypatch。 ^[PR #8224] + +## COSY-PACKED-1b — Streaming 缓存必须按请求保存自有状态并在终止时释放 + +- 触发：修改 packed streaming mel/phase cache、首块直接 finalize、请求完成/取消或 weight-norm folding。 +- 强制：packed streaming 缓存 mel/phase 用 detached GPU clone，普通路径保留 CPU contiguous 状态；首块 full-response shortcut 仅限 packed 模式、所有项 finalize、无既有 cache 且 token offset 为零。完成、取消与错误通过同一锁按 request ID 幂等清理。checkpoint 严格加载后把 HiFT/F0 移到最终设备，再折叠 weight norm 为普通 Parameter；兼容 legacy/modern API 且重复折叠不改变结果。 +- 禁止：对保留历史的后续块再次使用首块捷径；共享可改写 cache view；移设备前折叠造成 CPU/GPU 漂移；把原始 checkpoint 再加载进已折叠实例。原有有界 mel 窗与 phase 合同仍见 `COSYVOICE3-1c`。 +- 验收：同 request 的连续块、mixed finalize、重复 cancel/end、空状态与已存在状态分别回归；保留旧状态后推进下一块不得改变旧值，load→move→fold 数值和 state-dict 边界分开验证。 ^[PR #8224] + +## COSY-CACHE-1a — Reference conditioning 必须按真实音频身份缓存并返回独立张量 + +- 触发：修改匿名/注册 reference audio 的预处理、speaker artifact cache 或异步预取。 +- 强制：匿名键包含连续 waveform bytes、sample rate、shape、dtype、model directory 与 speaker backend；注册键包含 voice name 和 upload generation。24 kHz conditioning 按 token/feature 最短有效长度对齐。cache 中存 CPU clone，命中也逐字段 clone 返回；reference/target text 单独 tokenize，不混入音频 artifact。匿名预取先复制输入并复用相同 resampler 与 target sample rate，后续 processor 不重复 resample；注册 voice 不走匿名预取。 +- 禁止：用 URL/文件名代替 waveform 内容身份；复用可被下游修改的 cache tensor；漏掉 generation/backend 或跨模型共享结果；把 process-local 有界 artifact cache 说成预分配显存或全局 RSS 上限。 +- 验收：同内容复用、同名重上传、采样率/dtype/shape/backend 改变、24 kHz 截齐、命中后原地修改以及预取开关 on/off 均覆盖；conditioning 与未缓存预处理逐字段一致。 ^[PR #8224] + +## COSY-SAMPLE-1a — RAS 与 standard 模式必须保留生成历史和控制 token 语义 + +- 触发：修改 `cosyvoice3_sampling_mode`、RAS nucleus/repetition replacement、per-row RNG 或 standard-mode 停止条件。 +- 强制：仅接受 `ras`/`standard`，RAS 是基础默认；RAS nucleus 基于完整分布的 top-p 前缀并受 top-k 上限约束，stable tie 按较小 token ID 排序；重复判定只看已生成历史，不惩罚 prompt。seed/position 对应请求，replacement 使用独立随机流。standard 模式显式选择，并保留整段 200 个 speech-control token 的停止范围。 +- 禁止：把 RAS fused 路径扩展到不支持的 top-k 范围；把 resumed generated token 当 prompt 而丢失惩罚状态；用 batch row 代替 request RNG 身份；以吞吐数据证明两种采样分布/质量相同。 +- 验收：temperature=0、stable ties、top-p/top-k、generated-only repetition、resume、不同 seed 和混合请求顺序回归；standard/RAS 分别验停止、音频长度与质量，不能只看隐藏状态或短历史 benchmark。 ^[PR #8224] diff --git a/knowledge/repos/vllm-omni/models/cosyvoice3/rules-streaming-graphs.md b/knowledge/repos/vllm-omni/models/cosyvoice3/rules-streaming-graphs.md new file mode 100644 index 000000000..63a56adb8 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/cosyvoice3/rules-streaming-graphs.md @@ -0,0 +1,24 @@ +--- +title: "CosyVoice3 bounded HiFT graph 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models] +sources: ["PR #8420", "PR #8433"] +--- + +# CosyVoice3 bounded HiFT graph 规则 + +## COSY-GRAPH-1a — HiFT graph 缓存必须有界并返回自有结果 + +- 触发：修改 `HiFTDecodeGraphs`、packed DiT attention 的 query span 或 graph/cached-iSTFT 开关。 +- 强制：CPU、nested capture 和未满足缓存条件的调用走 eager；signature 包含 mel/source shape、device、两者 dtype、finalize、TF32 与 autocast precision 状态。seen LRU 上限 128，graph 上限 32，同 signature 第三次出现才 warm/capture；warm stream 依赖调用者 stream，replay 后返回 clone。F0、noise、phase 与请求状态留在 deterministic decode graph 外。streaming DiT query bound 用 static chunk size，full response 用 page-table span；不能以完整 KV prefix 代替 query span。启用 decode graphs 会同时启用 cached iSTFT。 +- 禁止：把 capture output view 当持久请求输出；忽略 precision/finalize 导致错图；无限增长缓存；把 eligibility/capacity 的 eager 回退描述为 capture 异常恢复——此实现的 capture 异常会传播，不能宣称已自动 disable/retry。 +- 验收：首两次 eager、第三次 capture、复用、32/128 边界、CPU/nested capture、precision 与 finalize 变更分别覆盖；旧返回值在下一次 replay 后保持不变，graph/eager 数值及首次编译失败语义单独验证。 ^[PR #8420] + +## COSY-PROFILE-1a — 自动 profile 只能在完整设备能力检查通过后选择 + +- 触发：修改 CosyVoice3 pipeline resolver、packed streaming deploy 或 offline example sampling budget。 +- 强制：resolver 仅识别 CosyVoice3 HF config；通过 platform/NVML 查询、不初始化父进程 CUDA。只有 NVIDIA CUDA、capability major 恰为 9、显存至少 140 GiB 且 MPS control 可执行存在，才选 optimized standard profile；否则 generic，保留显式 deploy/CLI 优先级。optimized profile 自带 stage-0 standard HF override；stop tokens 包含全部 200 个 speech controls。example 根据最终 stage-0 config 取模式并复制默认 sampling，把 min_tokens 限制在有效 max_tokens 内。 +- 禁止：只按 GPU 名称或 `major>=9` 推断 eligibility；选择 profile 后忽略显式 override；把标准采样标成 RAS 等价；把 graph pool 算作 gpu_memory_utilization 内的全部内存，或把第三次 lazy capture 的耗时隐去。 +- 验收：mock 不同 capability/memory/MPS availability、wrong model type、显式 override 与严格 token budget；验证父进程未 init CUDA、低预算不会因 min_tokens 超预算停滞，并分别报告 cold/warm latency 与音频长度/质量。 ^[PR #8433] diff --git a/knowledge/repos/vllm-omni/models/fish-qwen3-omni/_index.md b/knowledge/repos/vllm-omni/models/fish-qwen3-omni/_index.md index a613eb27e..d73ed28d1 100644 --- a/knowledge/repos/vllm-omni/models/fish-qwen3-omni/_index.md +++ b/knowledge/repos/vllm-omni/models/fish-qwen3-omni/_index.md @@ -1,7 +1,7 @@ --- title: "Fish Speech S2 Pro（fish_qwen3_omni,双 AR 单 stage）" created: 2026-07-21 -updated: 2026-07-21 +updated: 2026-10-06 type: index tags: [vllm-omni, models] sources: [vllm_omni/model_executor/models/fish_speech/, vllm_omni/attention/fish_kvcache_attn.py, vllm_omni/deploy/fish_qwen3_omni.yaml] @@ -63,3 +63,5 @@ sources: [vllm_omni/model_executor/models/fish_speech/, vllm_omni/attention/fish - 审查 fish 的 KV 注意力、背压流式或 chunked-prefill 约束改动;上调并发前 先读 high_concurrency YAML 的 knob 语义。 - 语义验收见 [model-validation](../../review/guides/model-validation.md)。 + +- [Fish Speech streaming 规则](rules.md) — 自适应 chunk 的绝对帧 frontier 与 terminal。 diff --git a/knowledge/repos/vllm-omni/models/fish-qwen3-omni/rules.md b/knowledge/repos/vllm-omni/models/fish-qwen3-omni/rules.md new file mode 100644 index 000000000..2a4980859 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/fish-qwen3-omni/rules.md @@ -0,0 +1,24 @@ +--- +title: "Fish Speech S2 Pro streaming 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, model-executor] +sources: ["PR #8422"] +confidence: high +--- + +# Fish Speech S2 Pro streaming 规则 + +## Direct 代码快速入口 + +| PR 描述信号 | 规则组 | 第一批源码 | +|---|---|---| +| initial/steady chunk、backlog、left context、finish/terminal | `FISH-STREAM-1a` | `vllm_omni/model_executor/stage_input_processors/fish_speech.py::slow_ar_to_dac_decoder_async_chunk`；`tests/model_executor/stage_input_processors/test_fish_speech_async_chunk.py` | + +## FISH-STREAM-1a — 自适应 chunk 使用绝对 emitted-frame frontier + +- 触发：修改 Fish Speech async chunk processor、initial/steady chunk policy 或 backlog 驱动的 chunk-size 切换。 +- 强制：以 request-owned `emitted_frames` 记录已发送帧的绝对边界，chunk size 只决定何时发出；每次覆盖全部 pending span，并带有界 left context。完成时 flush 整个尾部，只发送一次 terminal；状态放在 `request_payload` 的私有、不与输入 payload 字段冲突的 key 下，沿 connector completion/cancellation 生命周期回收。 +- 禁止：用 chunk 次数乘当前 chunk size 推导已发送边界；切换策略后重发或跳过帧；完成后重复 terminal；仅缩小并发掩盖 frontier 错误。 +- 验收：参数化 steady、finish 时降档、midstream 升档与振荡策略，包含/不包含 initial chunk、list/tensor payload，剔除 left context 后每个原始 frame 恰好出现一次，尾部和 terminal 完整。^[PR #8422] diff --git a/knowledge/repos/vllm-omni/models/glm-image/_index.md b/knowledge/repos/vllm-omni/models/glm-image/_index.md index a2e46c253..b44fa9cec 100644 --- a/knowledge/repos/vllm-omni/models/glm-image/_index.md +++ b/knowledge/repos/vllm-omni/models/glm-image/_index.md @@ -9,6 +9,8 @@ sources: [vllm_omni/model_executor/models/glm_image/, vllm_omni/diffusion/models # GLM-Image +- [Token-ID handoff 规则](rules.md) — generated/source IDs 与辅助 hidden 的表示边界。 + 以下事实在 `main @ 5d44868e` 复核。**树内定位**（知识树自身的路由约定,非源码 事实）：AR→DiT 图像编辑（i2i/IT2I）行为对齐时的参照家族——hunyuan 的 [it2i-gap](../hunyuan-image3/it2i-gap.md) 以它为对齐基准,定位依据见 diff --git a/knowledge/repos/vllm-omni/models/glm-image/rules.md b/knowledge/repos/vllm-omni/models/glm-image/rules.md new file mode 100644 index 000000000..ff563dd81 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/glm-image/rules.md @@ -0,0 +1,31 @@ +--- +title: "GLM-Image token-ID handoff 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, diffusion] +sources: ["PR #7843"] +confidence: high +--- + +# GLM-Image token-ID handoff 规则 + +## Direct 代码快速入口 + +| PR 描述信号 | 规则组 | 第一批源码 | +|---|---|---| +| TOKEN_IDS、generated/source IDs、AR→DiT、auxiliary hidden | `GLMIMAGE-IDS-1a`；共享 naming/accumulation 另查 [输出合同](../../components/model-executor/rules-output-contract.md) | `vllm_omni/model_executor/stage_input_processors/glm_image.py::ar2diffusion`；`vllm_omni/outputs/output_modality.py`、`vllm_omni/outputs/output_processor.py`；`tests/model_executor/stage_input_processors/test_glm_image.py` | + +## GLMIMAGE-IDS-1a — generated IDs 与 source-image IDs 必须保留各自 handoff + +- 触发：修改 GLM-Image engine_output_type、AR auxiliary output 或 ar2diffusion bridge。 +- 强制：generated IDs 仍走 cumulative_token_ids，source-image IDs 保持 ids.prior_image； + TOKEN_IDS 类型只改变共享 naming/retention，不为 GLM 另加 generic hidden 的特殊重命名。 + 辅助 hidden 即使命名为 token_ids，值也仍是原 tensor，不能替代 bridge 的真实离散 IDs。 +- 禁止：为 output-type 改名重写已正确的 ar2diffusion；把仅 eager 单 NVIDIA T2I/I2I 的 + matched PNG parity 视为 NPU/optional auxiliary producer 或全模式表示验证。 +- 验收：T2I 与 I2I 分别断言 generated/source IDs 到达正确输入、EOS 处理与已有语义保持； + 若新增 auxiliary consumer，另核实其 dtype/shape/value 和命名含义。^[PR #7843] + +共享 parsing 与 accumulation 见 [输出合同](../../components/model-executor/rules-output-contract.md)； +AR→DiT 数据流见 [模型架构](architecture.md)。 diff --git a/knowledge/repos/vllm-omni/models/helios/_index.md b/knowledge/repos/vllm-omni/models/helios/_index.md index 6eb6c9b5e..50f9edc25 100644 --- a/knowledge/repos/vllm-omni/models/helios/_index.md +++ b/knowledge/repos/vllm-omni/models/helios/_index.md @@ -9,7 +9,9 @@ sources: [vllm_omni/diffusion/models/helios/, vllm_omni/diffusion/registry.py] # Helios -以下事实在 `main @ 5d44868e` 复核（源码派生页,尚无本模型的运行经验沉淀）。 +- [Cross-attention cache 规则](rules.md) — source weakref、地址重用与 intra-request KV reuse。 + +以下事实在 `main @ 5d44868e` 复核（历史结构快照；当前 cache 约束见模型规则）。 ## 名称与范围 diff --git a/knowledge/repos/vllm-omni/models/helios/rules.md b/knowledge/repos/vllm-omni/models/helios/rules.md new file mode 100644 index 000000000..0dcfe355a --- /dev/null +++ b/knowledge/repos/vllm-omni/models/helios/rules.md @@ -0,0 +1,32 @@ +--- +title: "Helios cross-attention cache 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, diffusion] +sources: ["PR #8064"] +confidence: high +--- + +# Helios cross-attention cache 规则 + +## Direct 代码快速入口 + +| PR 描述信号 | 规则组 | 第一批源码 | +|---|---|---| +| cross-attention KV、text projection、source weakref、LRU、clear | `HELIOS-CACHE-1a` | `vllm_omni/diffusion/models/helios/cross_attn_cache.py::SourceTensorLRUCache` → `vllm_omni/diffusion/models/helios/helios_transformer.py::HeliosTransformer3DModel` | + +## HELIOS-CACHE-1a — source tensor 指纹必须有 liveness，不能以地址等同跨请求内容 + +- 触发：修改 Helios text projection / cross-attention KV cache 的 key、LRU 或生命周期。 +- 强制：data_ptr/shape/stride/dtype/device/version 指纹旁保存 source tensor weakref； + lookup 检查原 source 仍存活，dead ref 命中也必须 miss 并清除，insert 主动清 dead + entries，LRU 有界。request 持有 source 时允许 intra-request denoise reuse；显式清缓存 + 同时清 projection 与 KV 两套。inference tensor 没 version counter 时保持安全路径。 +- 禁止：地址重用时让新 prompt 命中旧 KV；缓存强引用把 source 永久钉住；把没有 version + counter 当成可检测任意原地修改（inference tensor mutation 仍需请求级边界）。 +- 验收：持有 source 时 hit、GC 后相同伪指纹 miss、version/shape/stride 改变 miss、LRU + eviction 与两套 clear；连续不同 prompt GPU 请求另验证条件不会交叉污染。^[PR #8064] + +共用请求 cache 边界查 [Diffusion cache](../../components/diffusion/cache-acceleration.md)； +模型结构查 [Helios index](_index.md)。 diff --git a/knowledge/repos/vllm-omni/models/higgs-audio/_index.md b/knowledge/repos/vllm-omni/models/higgs-audio/_index.md index 175214992..a9503499c 100644 --- a/knowledge/repos/vllm-omni/models/higgs-audio/_index.md +++ b/knowledge/repos/vllm-omni/models/higgs-audio/_index.md @@ -1,7 +1,7 @@ --- title: "Higgs-Audio（V2/V3 双谱系 TTS;higgs_multimodal_qwen3 即 V3）" created: 2026-07-21 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, models] sources: ["PR #6422", "PR #7065", vllm_omni/entrypoints/openai/serving_speech.py, vllm_omni/model_executor/models/higgs_audio_v2/, vllm_omni/model_executor/models/higgs_audio_v3/, vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_tokenizer.py, vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py, vllm_omni/transformers_utils/configs/higgs_audio_v3.py, tests/entrypoints/openai_api/test_serving_speech.py, tests/model_executor/models/higgs_audio_v3/test_higgs_audio_v3.py, tests/e2e/online_serving/test_higgs_audio_v3.py] @@ -77,3 +77,5 @@ sources: ["PR #6422", "PR #7065", vllm_omni/entrypoints/openai/serving_speech.py codec 常量（8×1026,25 fps × hop 960;v2 称 BOS/EOS 1024/1025,v3 称 BOC/EOC 1024/1025）。 - 语义验收见 [model-validation](../../review/guides/model-validation.md)。 + +- [V3 MRv2 state、sampling 与 audio snapshots](rules-mrv2-audio.md) diff --git a/knowledge/repos/vllm-omni/models/higgs-audio/rules-mrv2-audio.md b/knowledge/repos/vllm-omni/models/higgs-audio/rules-mrv2-audio.md new file mode 100644 index 000000000..ceec7b628 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/higgs-audio/rules-mrv2-audio.md @@ -0,0 +1,32 @@ +--- +title: "Higgs V3 MRv2 audio state 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, model-executor] +sources: ["PR #8226"] +confidence: high +--- + +# Higgs V3 MRv2 audio state 规则 + +## HIGGS-MRV2-1a — model-owned state 的 admission、RNG 与退休必须按 request 闭合 + +- 触发：修改 Higgs `create_omni_model_state`、request slot、sampler registration、MRv2 profile 或 audio feedback。 +- 强制：在 graph capture 前固定 decode-state storage，由模型 factory 创建 Higgs state。MRv2 要求 `audio_async_payload`；该模式同时要求 async scheduling 和 prompt-mode classification。真实 request 支持 temperature/top-k/top-p，明确拒绝 text penalties、logprobs/prompt_logprobs、非零 min_tokens/min_p、allowed/bad tokens 和 logit bias；sampling 阶段拒绝 grammar 与 draft tokens，upstream synthetic warmup 保留标准 sampler 路径。request seed 建立独立 generator，eager/graph 都用对应 rows 的 exponential noise；退休时清理 model decode state、request/slot/generator 和 cached metadata。graph padding rows 不拥有 request，feedback embedding 保持零。 +- 禁止：把 Talker 内 graph sampling 状态或 RNG 作为 batch-global历史；为接受 unsupported request 而默默忽略采样约束；退休 slot 后继续消费其 audio feedback；将 H200 opt-in profile 外推为默认部署或跨平台能力。 +- 验收：request add/remove/reorder/reuse、seeded eager/replay、mixed prefill/decode 和 padding rows 对照；每个 unsupported option 显式失败，warmup 对照仍走 upstream path，退休后相同 slot 的新 request 无旧帧或 RNG 污染。^[PR #8226] + +## HIGGS-MRV2-1b — audio snapshot 必须在采样后拥有 storage，并在 D2H 后纯化转换 + +- 触发：修改 `post_sample_multimodal_outputs`、Higgs async snapshot/finalizer、batch chunk adapter 或 whole-utterance code conversion。 +- 强制：仅在采样后按当前 batch row 导出 audio codes；async GPU staging 由每步 snapshot拥有，D2H 完成后 finalizer 只读取 CPU snapshot与该步 invalid rows，不能读 mutable decode/request state。保留二维 empty `[0,Q]` row，避免 terminal/invalid step 覆盖此前累计 full-response rows。chunk batch adapter 保持单请求 accumulation、de-delay、right holdback、left context 和 emitted-frame counters；whole-utterance native/legacy paths 共用同一 de-delay conversion，之后将 special/padding codes 替换为零并保留原 residual ramp-down removal。 +- 禁止：提前导出 forward 的上一帧 state；重用共享 host staging 给未完成 snapshot；把 terminal empty 改成一维值；clamp EOC 到合法 codec id；将 scheduling placeholder 当作实际 audio payload（包括 empty terminal payload）。 +- 验收：异步 copy 与后续 sampling/reorder 并发时，旧 snapshot 仍对应原 request；覆盖 invalid/terminal empty、整段与 chunk adapter 等价、mixed batch completion，以及 payload为空时不解码 placeholder。^[PR #8226] + +## HIGGS-MRV2-1c — sampler 与 codec graphs 必须在启动捕获并保持真实 shape 与输出所有权 + +- 触发：修改 Higgs full-sample graph startup、codec graph shapes、graph pool/storage 或 mixed FULL attention metadata。 +- 强制：sampler graph 在启动以相同 noise 核对 eager/captured state transition，保留 captured state/cache 的 allocations；serving 只 replay 已有 shape，unsupported metadata/shape 走 regular sampler。Code2Wav graph 以 exact `(batch,frames)` 为 key，time 和 batch 均不 padding，未捕获 shape eager；返回前 clone audio，避免下一 group/replay 覆盖同 pool storage。mixed FULL capture 的 query bound 不能使用 dummy 平均长度，后续 separate-decode/explicit-bound 约束遵循 [attention capture rule](../../components/model-executor/rules-attention-capture.md)。 +- 禁止：serving 临时 capture；以一条 batch shape 替代所有 convolution boundaries；返回 graph-owned audio view；由 CPU mock、一次 FA3 replay或历史 benchmark 推断所有 shape、checkpoint、硬件的音频质量和性能。 +- 验收：分别验证 sampler state/noise parity、exact codec singleton/batch/frame shapes、bucket miss eager、连续 group output不互相覆盖，以及真实 mixed-prefill attention capture/replay；每项优化须有实际 gate/replay 证据。^[PR #8226] diff --git a/knowledge/repos/vllm-omni/models/higgs-audio/rules.md b/knowledge/repos/vllm-omni/models/higgs-audio/rules.md index 9b03dc24e..d83d9a09d 100644 --- a/knowledge/repos/vllm-omni/models/higgs-audio/rules.md +++ b/knowledge/repos/vllm-omni/models/higgs-audio/rules.md @@ -1,10 +1,10 @@ --- title: "Higgs-Audio 规则" created: 2026-09-05 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, models, model-executor] -sources: ["PR #6422", "PR #7065", vllm_omni/entrypoints/openai/serving_speech.py, vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py, vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_tokenizer.py, vllm_omni/worker/gpu_model_runner.py, tests/entrypoints/openai_api/test_serving_speech.py, tests/model_executor/models/higgs_audio_v3/test_higgs_audio_v3.py, tests/e2e/online_serving/test_higgs_audio_v3.py, "PR #7076"] +sources: ["PR #8426", "PR #6422", "PR #7065", vllm_omni/entrypoints/openai/serving_speech.py, vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py, vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_tokenizer.py, vllm_omni/worker/gpu_model_runner.py, tests/entrypoints/openai_api/test_serving_speech.py, tests/model_executor/models/higgs_audio_v3/test_higgs_audio_v3.py, tests/e2e/online_serving/test_higgs_audio_v3.py, "PR #7076"] confidence: high --- @@ -55,6 +55,13 @@ confidence: high 路径重写时 cache salt 仍变化。H100 voice-clone E2E 未在该 PR 的 lanes 运行，不能把 unit coverage 宣称为 end-to-end 证据。^[PR #7065] +## HIGGS-FIXTURE-1a — 绕过构造器的 state fixture 须显式建立被测路径依赖 + +- 触发：修改 Higgs V3 `make_state`、以 `object.__new__` 构造的 model state，或被该替身调用的 `add_request`/eager-MTP 路径。 +- 强制：根据实际被测调用初始化 fixture 的全部依赖；绕过构造器时显式创建 `_stream_pos` 和与该 state 绑定的 `EagerMTPState(state)`，不能假设正常 `__init__` 已执行。 +- 禁止：只设置模型/runner mock 就直接访问未建立的 eager state；用作者旧树的 CI 通过排除后续组合树上的 fixture 失配；把 PR 的待运行测试计划说成已通过。 +- 验收：运行 `tests/model_executor/models/higgs_audio_v3/test_mrv2_state.py` 的 CPU case，并让真实被测方法访问这些依赖；初始化缺失时回归应失败。^[PR #8426] + ## HIGGS-3a — 共享 reference-code encode 必须屏蔽请求取消并以 task 身份退休 - 触发：修改 Higgs-Audio V3 `_resolve_higgs_audio_v3_ref_codes`、artifact-keyed inflight map。 diff --git a/knowledge/repos/vllm-omni/models/indextts2/_index.md b/knowledge/repos/vllm-omni/models/indextts2/_index.md index b97823636..c2df194d2 100644 --- a/knowledge/repos/vllm-omni/models/indextts2/_index.md +++ b/knowledge/repos/vllm-omni/models/indextts2/_index.md @@ -1,7 +1,7 @@ --- title: "IndexTTS2（非流式两 stage,GPT talker + S2Mel/BigVGAN）" created: 2026-07-21 -updated: 2026-07-21 +updated: 2026-10-06 type: index tags: [vllm-omni, models] sources: [vllm_omni/model_executor/models/indextts2/, vllm_omni/deploy/indextts2.yaml, vllm_omni/model_executor/stage_input_processors/indextts2.py] @@ -58,3 +58,5 @@ sources: [vllm_omni/model_executor/models/indextts2/, vllm_omni/deploy/indextts2 - 审查 indextts2 的全载荷合同、tokenizer 旁路或 stage-1 hf_overrides 性能 开关;想给它加流式先看 architecture 里的非流式根因。 - 语义验收见 [model-validation](../../review/guides/model-validation.md)。 + +- [request-endlatent、generichidden与整段S2Mel交付](rules.md)。 diff --git a/knowledge/repos/vllm-omni/models/indextts2/rules.md b/knowledge/repos/vllm-omni/models/indextts2/rules.md new file mode 100644 index 000000000..3292b982c --- /dev/null +++ b/knowledge/repos/vllm-omni/models/indextts2/rules.md @@ -0,0 +1,23 @@ +--- +title: "IndexTTS2 request-end payload 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models] +sources: ["PR #8478"] +--- + +# IndexTTS2 request-end payload 规则 + +## Direct 代码快速入口 + +| PR 描述信号 | 规则组 | 第一批源码 | +|---|---|---| +| generic pooler hidden、request end、latent/code-only、S2Mel payload | `ITTS2-1a` | `vllm_omni/model_executor/models/indextts2/indextts2_talker.py::IndexTTS2TalkerForConditionalGeneration` → `vllm_omni/model_executor/stage_input_processors/indextts2.py::talker2s2mel_full_payload` | + +## ITTS2-1a — S2Mel 的整段 latent 必须在 request end 交付 + +- 触发：修改IndexTTS2Talker的genericpoolerhidden、request-endbuffer或talker2s2mel_full_payload。 +- 强制：2.0 latent与2.5code-only均设置omni_pooler_payload_include_hidden=False、omni_payload_at_request_end=True。latent从hidden_states.latent、codes从codes.mel及meta交付；request-endGPU累积沿共享owned-device-snapshot合同，并保持code/latent同长。 +- 禁止：每step复制S2Mel不读的pooler hidden；把去掉generic hidden解释为去掉alignedlatent；在request end前交付截断整段或混合request；把旧v0.28benchmark当currentmain复测。 +- 验收：两代×use_gpt_latent控制断言flags与实际consumerfields；snapshot/accumulate/cancel/reorder、同长非空latent、no-latent及prefix-cachematerialization保持。性能/ASR结论需同commit同deployment独立复测。 ^[PR #8478] diff --git a/knowledge/repos/vllm-omni/models/kandinsky6/_index.md b/knowledge/repos/vllm-omni/models/kandinsky6/_index.md new file mode 100644 index 000000000..9a529eef7 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/kandinsky6/_index.md @@ -0,0 +1,21 @@ +--- +title: "Kandinsky 6" +created: 2026-10-06 +updated: 2026-10-06 +type: index +tags: [vllm-omni, models, diffusion] +sources: ["PR #8537"] +confidence: high +--- + +# Kandinsky 6 + +本 owner 对应 `vllm_omni/diffusion/models/kandinsky6/` 与 +`Kandinsky6TI2VAPipeline`；不能继承其他视频模型的请求、checkpoint 或缓存假设。 + +| 遇到什么 | 查看哪里 | +|---|---| +| Hub 分目录权重、音视频输出、参考图、请求缓存与 VAE patch parallel | [模型规则](rules.md) | + +共享执行与输出协议查 [Diffusion owner](../../components/diffusion/_index.md)； +公开 HTTP 请求与帧数校验查 [Serving owner](../../components/serving/_index.md)。 diff --git a/knowledge/repos/vllm-omni/models/kandinsky6/rules.md b/knowledge/repos/vllm-omni/models/kandinsky6/rules.md new file mode 100644 index 000000000..07013aab3 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/kandinsky6/rules.md @@ -0,0 +1,102 @@ +--- +title: "Kandinsky 6 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, diffusion] +sources: ["PR #8537"] +confidence: high +--- + +# Kandinsky 6 规则 + +## Direct 代码快速入口 + +| PR 描述信号 | 规则组 | 第一批源码 | +|---|---|---| +| Kandinsky 6、TI2VA、checkpoint、Hub folder | K6-1a | `diffusion/models/kandinsky6/pipeline_kandinsky6.py::_WEIGHT_SUBFOLDERS`、`_adapt_k6_weight_name` | +| 音视频输出、sample rate、video-only | K6-1b | `get_kandinsky6_post_process_func`、`postprocess_audio` → `output_formatter.py::normalize_diffusion_postprocess_output` | +| 参考图、tail conditioning、视频 defaults | K6-1c | `prepare_encode`、`forward`、`append_i2va_tail_condition`；`model_extras/kandinsky6.py` → `entrypoints/openai/serving_video.py` | +| text projection cache、MagCache、TeaCache | K6-2a | `Kandinsky6Transformer3DModel.clear_text_proj_cache`、`K6StepCache`、`Kandinsky6TI2VAPipeline.forward` | +| patch parallel、VAE tiling | K6-2b | `modeling_kandinsky6_vae.py` 的 tile split/exec/merge → `tests/diffusion/models/kandinsky6/test_vae_patch_parallel.py` | +| HSDP、SP、TP 权重 shard | K6-2c | `kandinsky6_transformer.py` 的 `_sp_plan`、`_is_transformer_block`、`_shard_loaded_weight` | + +## K6-1a — Hub 组件目录与权重名称映射必须分别闭环 + +- 触发：修改 Kandinsky 6 checkpoint discovery、空模块构造或权重 loader。 +- 强制：保持 `transformer/`、`vae/`、`text_encoder/`、`text_encoder_2/`、 + `audio_vae/` 到各模块的独立加载；`videoT` / `audioT` 映射到 + `video_dec_block` / `audio_dec_block`，Qwen language/visual 与 audio VAE/vocoder + 按 `_adapt_k6_weight_name` 的各自前缀映射。tokenizer 从独立 `tokenizer/` 目录取。 + 模块初始 device 必须服从共享 resolved offload policy。 +- 禁止：把 Hub bundle 当作单一 transformer checkpoint；对所有组件套同一个前缀替换； + 读取已退役 offload flags 来覆盖 resolved policy。 +- 验收：`test_hub_component_keys_map_onto_pipeline_modules` 覆盖视频、音频、 + Qwen language/visual 与无需改名的权重；共享 legacy-flag-reader 检查保持通过， + 真正的 checkpoint 加载另核对 missing/unexpected keys。^[PR #8537] + +## K6-1b — 音频采样率必须穿过 post-process 与 formatter + +- 触发：修改 Kandinsky 6 `DiffusionOutput`、post-process、video/audio formatter 或 mux。 +- 强制：post-process 返回平铺的 `video`、`audio`、`audio_sample_rate`、`fps`； + int16 PCM 转为 mux 使用的 float32 波形，44.1 kHz 采样率进入 formatter 的 audio metadata。 + 无音频时同时省略 audio 与 sample-rate；解包单请求产生的 batched audio list。 +- 禁止：额外包一层 envelope 导致 formatter 丢失 sample-rate，或把 video-only 请求补成 + 静音音轨；不得把纯函数格式测试当作真实 AAC/MP4 音视频对齐已验收。 +- 验收：覆盖 PCM 范围转换、flat payload 的 metadata、无音频与单元素 list；真实 mux + 另核对容器的音频采样率、视频 fps 和解码结果。^[PR #8537] + +## K6-1c — 参考图是被剔除的条件尾帧，公开 defaults 不固定视频长度 + +- 触发：修改 Kandinsky 6 T2VA/I2VA 请求解析、reference image、latent packing 或视频 defaults。 +- 强制：非空 prompt、每请求一条 prompt、最多一张 reference image；图片从 + `multi_modal_data.image` 进入，保持 `tail_cond_first_frame` 的追加、条件 mask 和 + `generated_visual_mask` 在 decode 前排除条件尾帧。公开 recipe 的 `4k+1` 帧数与 + 16 整除分辨率必须与 latent 时空布局一致；`duration_seconds=None` 的 serving defaults + 允许显式帧数，固定时长模型继续执行自己的帧数约束。 +- 禁止：把参考图当成额外生成帧或无 mask 的首帧；把 Kandinsky 的默认 125 帧误判为 + 所有请求的固定长度，或据此解除固定时长模型的约束。 +- 验收：分别检查无图/单图/多图拒绝与条件尾帧剔除；video-server 测试区分 + defaults-only 与 fixed-duration 模型；真实 I2VA 另验证帧数和参考图条件效果。 + 不把配置默认值或 tensor shape 测试当作生成质量已验收。^[PR #8537] + +## K6-2a — 文本投影缓存只属于当前 generation，跳步缓存必须有真实首步 + +- 触发：修改 Kandinsky 6 text projection、请求复用或 denoise step cache。 +- 强制：每次新 generation 清除 transformer 的 text projection cache；step execution 的 + prepare 路径与 monolithic `forward` 都要保持此边界。Mag/Tea step cache 先完成并保存 + 真实 denoise 输出，之后才能在阈值与 skip budget 内复用。 +- 禁止：把 tensor `data_ptr()` 当作跨请求内容 identity，因 CUDA 地址复用而继承上一 + prompt 的 embedding；跳过尚无缓存的首步；把已注册 TeaCache 或配置为 uncalibrated + 当作已完成真实模型质量校准。 +- 验收：连续两个不同 prompt 的请求与 warmup→首请求检查不会命中旧文本投影； + `test_step_cache_skips_only_after_a_real_step` 验证首步不跳、store 后才允许跳； + 缓存质量与分布式收益分别以真实模型结果核验。^[PR #8537] + +## K6-2b — VAE patch parallel 必须保持单卡 tile 网格及合并语义 + +- 触发：修改 Kandinsky 6 VAE decode 的 tile split、rank 分发或 merge。 +- 强制：保持单卡 optimal tiling 的 shape/stride 与 tile grid，分布式 split/exec/merge + 按 grid coordinates 重组同一 decode；每个 rank 执行 tile，而不是重复 decode 全图。 +- 禁止：打开 patch parallel 后绕过 optimal tiling，或用真实 transformer 的单卡 smoke + 证明 VAE 多 rank tile 重组正确。 +- 验收：`test_distributed_decode_tiles_match_local_tiled_decode`、optimal-tiling + 回归和两 rank CPU/gloo tile 分发测试各自通过；stub decode 的一致性不外推为真实 + Hunyuan VAE 的 GPU 数值 parity 或性能收益。^[PR #8537] + +## K6-2c — 模型并行计划必须保持视频、音频与文本的各自边界 + +- 触发：修改 Kandinsky 6 transformer 的 HSDP block selection、SP plan 或 TP 权重加载。 +- 强制：HSDP predicate 保持 indexed `visual_transformer_blocks`、`text_transformer_blocks`、 + `video_text_transformer_blocks` 与 `audio_text_transformer_blocks` 的明确匹配，不能把 + embeddings 误当 transformer block。 + SP 对 visual tokens 与其 RoPE 使用对应 shard，并在 visual head 前沿 token 维 gather； + TP loader 依各参数的 loader contract 取 shard，形状已经匹配的 replicated bias 保持完整。 +- 禁止：按模块名字含 `visual` 就扩大 HSDP 范围；漏掉 gather 而把局部视频作为完整结果； + 对已匹配 bias 二次切片；将 plan/predicate 单测外推为全部多卡音视频数值已验证。 +- 验收：`test_hsdp_predicate_matches_visual_blocks_only_when_indexed`、SP plan、 + column-parallel weight 与 replicated row-bias 单测分别覆盖边界；真实多 rank 运行 + 另比较输出 shape、video/audio 条件与单卡结果。^[PR #8537] + +共享输出与进程协议见 [Diffusion 输出规则](../../components/diffusion/rules-output-lifecycle.md)； +真实模型能力的证据边界见 [模型验证](../../review/guides/model-validation.md)。 diff --git a/knowledge/repos/vllm-omni/models/lingbot-video/_index.md b/knowledge/repos/vllm-omni/models/lingbot-video/_index.md index 95147e594..1ce476180 100644 --- a/knowledge/repos/vllm-omni/models/lingbot-video/_index.md +++ b/knowledge/repos/vllm-omni/models/lingbot-video/_index.md @@ -1,7 +1,7 @@ --- title: "LingBot-Video" created: 2026-08-10 -updated: 2026-08-10 +updated: 2026-10-06 type: index tags: [vllm-omni, models, diffusion] sources: [vllm_omni/diffusion/models/lingbot_video/, vllm_omni/model_extras/lingbot_video.py, vllm_omni/diffusion/registry.py, "PR #5311"] @@ -26,3 +26,5 @@ sources: [vllm_omni/diffusion/models/lingbot_video/, vllm_omni/model_extras/ling - PR 涉及 `lingbot_video`、`LingBotVideoPipeline`、T2I/T2V/TI2V 选择、LingBot 分辨率预设、 causal-VAE 帧网格或首帧条件。 - 性能数字只能回到可复现 benchmark；本页不把 PR 的单机耗时/显存结果升级为 gate。 + +decoded keyed media 的 zero-to-one range：LBV-2b；沿本目录 rules 审查入口核对。 diff --git a/knowledge/repos/vllm-omni/models/lingbot-video/rules.md b/knowledge/repos/vllm-omni/models/lingbot-video/rules.md index f07d68b8d..808e4f848 100644 --- a/knowledge/repos/vllm-omni/models/lingbot-video/rules.md +++ b/knowledge/repos/vllm-omni/models/lingbot-video/rules.md @@ -1,10 +1,10 @@ --- title: "LingBot-Video 规则" created: 2026-08-10 -updated: 2026-09-04 +updated: 2026-10-06 type: rule tags: [vllm-omni, models, diffusion] -sources: [vllm_omni/diffusion/models/lingbot_video/request_utils.py, vllm_omni/diffusion/models/lingbot_video/image_condition.py, vllm_omni/diffusion/models/lingbot_video/pipeline_lingbot_video.py, vllm_omni/entrypoints/openai/serving_video.py, vllm_omni/model_extras/lingbot_video.py, vllm_omni/model_extras/registry.py, examples/offline_inference/text_to_image/text_to_image.py, examples/offline_inference/image_to_video/image_to_video.py, tests/diffusion/models/lingbot_video/test_request_utils.py, tests/diffusion/models/lingbot_video/test_image_condition.py, tests/diffusion/models/lingbot_video/test_pipeline_lingbot_video.py, tests/entrypoints/openai_api/test_image_server.py, tests/entrypoints/openai_api/test_video_server.py, "PR #5311", "PR #5976", "PR #6049", "Issue #5883"] +sources: [vllm_omni/diffusion/models/lingbot_video/request_utils.py, vllm_omni/diffusion/models/lingbot_video/image_condition.py, vllm_omni/diffusion/models/lingbot_video/pipeline_lingbot_video.py, vllm_omni/entrypoints/openai/serving_video.py, vllm_omni/model_extras/lingbot_video.py, vllm_omni/model_extras/registry.py, examples/offline_inference/text_to_image/text_to_image.py, examples/offline_inference/image_to_video/image_to_video.py, tests/diffusion/models/lingbot_video/test_request_utils.py, tests/diffusion/models/lingbot_video/test_image_condition.py, tests/diffusion/models/lingbot_video/test_pipeline_lingbot_video.py, tests/entrypoints/openai_api/test_image_server.py, tests/entrypoints/openai_api/test_video_server.py, "PR #5311", "PR #5976", "PR #6049", "Issue #5883", "PR #7037"] --- # LingBot-Video 规则 @@ -87,7 +87,7 @@ sources: [vllm_omni/diffusion/models/lingbot_video/request_utils.py, vllm_omni/d - 触发：修改 pipeline return、postprocess、image/video formatter、batch/output count 或异常映射。 - 强制：pipeline 返回 `DiffusionOutput(output={"image": value})` 或 `{ "video": value }`；postprocess - 必须要求二者恰好一个、按 output type 转换而不丢 key。T2I decode 必须恰好一帧并移除 frame + 必须要求二者恰好一个、按 output type 转换而不丢 key。decoded keyed image/video 的 `[0,1]` tensor 必须转为 NumPy FP32 路径，避免 video API 将 floating tensor 当作 `[-1,1]` 再归一化；即使 keyed `output_type=pt` 也如此。bare pt tensor 与 image/video latent 保持既有 tensor 行为。T2I decode 必须恰好一帧并移除 frame 轴；latent output 保持 tensor。 `num_outputs_per_prompt != 1` 与 normalization 的 TypeError/ValueError 在 pipeline `forward` 统一转为 `OmniClientError`，使 image/video 路径都返回 HTTP 400；不要只在 image API 加不对称 @@ -95,5 +95,5 @@ sources: [vllm_omni/diffusion/models/lingbot_video/request_utils.py, vllm_omni/d - 禁止：postprocess 丢掉 modality key；让 n>1 成为 500；以内部防御性 ValueError 代替公开 request normalization。 - 验收：同时覆盖 formatter 的 image/video key、T2I 单帧 shape、TI2V input cardinality、最终 - image size limit、frame-count precedence 和 n>1 的 400。PR 所报 H200 单次耗时、峰值显存与 + image size limit、frame-count precedence 和 n>1 的 400；FP32/BF16 与 omitted/pt/np 的黑、中灰、白须穿过真实 API frame conversion 后保持原 `[0,1]` 值，不能只测全白；bare pt 与 latent identity control 保持。^[PR #7037] PR 所报 H200 单次耗时、峰值显存与 bitwise parity 缺少本知识树可复跑的命令/产物，不能作为性能或精度 gate。 diff --git a/knowledge/repos/vllm-omni/models/ltx2/rules.md b/knowledge/repos/vllm-omni/models/ltx2/rules.md index 819e00d6d..70686ce5d 100644 --- a/knowledge/repos/vllm-omni/models/ltx2/rules.md +++ b/knowledge/repos/vllm-omni/models/ltx2/rules.md @@ -1,15 +1,26 @@ --- title: "LTX-2.5 DiffVAE 规则" created: 2026-09-04 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, models, ltx2, diffusion] -sources: ["PR #6189", "PR #7000", "PR #7020", recipes/LTX/LTX-2.5.md, requirements/common.txt, docs/user_guide/diffusion/attention_backends/huggingface_hub.md, vllm_omni/diffusion/models/ltx2/ltx2_components.py, vllm_omni/diffusion/models/ltx2/ltx2_conditioning.py, vllm_omni/diffusion/models/ltx2/ltx2_diffusion_decoder.py, vllm_omni/diffusion/models/ltx2/ltx2_diffusion_decoder_distributed.py, vllm_omni/diffusion/models/ltx2/ltx2_latents.py, vllm_omni/diffusion/models/ltx2/ltx2_request.py, vllm_omni/diffusion/models/ltx2/ltx2_runtime.py, tests/diffusion/models/ltx2/test_ltx25_pipeline.py, tests/diffusion/models/ltx2/test_ltx2_output_cuda.py, tests/diffusion/models/ltx2/test_ltx2_pipeline.py, tests/diffusion/models/ltx2/test_ltx2_vae.py, "PR #7231", "PR #7350"] +sources: ["PR #6189", "PR #7000", "PR #7020", recipes/LTX/LTX-2.5.md, requirements/common.txt, docs/user_guide/diffusion/attention_backends/huggingface_hub.md, vllm_omni/diffusion/models/ltx2/ltx2_components.py, vllm_omni/diffusion/models/ltx2/ltx2_conditioning.py, vllm_omni/diffusion/models/ltx2/ltx2_diffusion_decoder.py, vllm_omni/diffusion/models/ltx2/ltx2_diffusion_decoder_distributed.py, vllm_omni/diffusion/models/ltx2/ltx2_latents.py, vllm_omni/diffusion/models/ltx2/ltx2_request.py, vllm_omni/diffusion/models/ltx2/ltx2_runtime.py, tests/diffusion/models/ltx2/test_ltx25_pipeline.py, tests/diffusion/models/ltx2/test_ltx2_output_cuda.py, tests/diffusion/models/ltx2/test_ltx2_pipeline.py, tests/diffusion/models/ltx2/test_ltx2_vae.py, "PR #7231", "PR #7350", "PR #8453"] confidence: high --- # LTX-2.5 DiffVAE 规则 +## LTX-ACCURACY-1a — 单 H100 的 LTX-2 distilled T2V reference 必须保留 matched offload profile + +- 触发：修改 LTX official-similarity matrix、ltx2_distilled_t2v fixture 或 offload 选择。 +- 强制：该全分辨率单 H100 case 启用 layerwise offload，reference 与 Omni 保持 bf16 + arithmetic、相同 checkpoint/seed/recipe 与独立 similarity thresholds；shutdown 用共享 + 无权重恢复的 teardown 合同，不能为 OOM 改 accuracy 数值口径。 +- 禁止：一个 case 通过就扩大到 LTX 全 matrix、其他卡或普遍性能收益；把 official 与 + Omni 的不同 offload 执行时间当作相同性能条件。 +- 验收：实际跑该 full_model hardware selector，核对视频/音频指标及 clean exit； + 单次作者 H100 结果仅支持该 case 的范围。^[PR #8453] + ## Direct 代码快速入口 | PR 描述信号 | 规则 | diff --git a/knowledge/repos/vllm-omni/models/magi2/_index.md b/knowledge/repos/vllm-omni/models/magi2/_index.md new file mode 100644 index 000000000..1c11cd804 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/magi2/_index.md @@ -0,0 +1,15 @@ +--- +title: "MAGI-2 Preview model owner" +created: 2026-10-06 +updated: 2026-10-06 +type: index +tags: [vllm-omni, models, diffusion] +sources: ["PR #7206", vllm_omni/diffusion/models/magi2/] +--- + +# MAGI-2 Preview + +- 触发：MAGI-2本地routedMoE、derivedweightcache、BF16kernel/launch或modelcheckpoint加载。 +- 源码：`diffusion/models/magi2/mh_moe.py`、`fused_moe_kernels.py`、`modeling_magi2.py`。 +- [BF16 routed MoE规则](rules.md)：eligibility、cache失效、head/routing/numerical/launch验收。 +- EP/framework通信归shared distributed owner，attention/其他modelfamily不得从此fusedcontract外推。 diff --git a/knowledge/repos/vllm-omni/models/magi2/rules.md b/knowledge/repos/vllm-omni/models/magi2/rules.md new file mode 100644 index 000000000..4f92c063c --- /dev/null +++ b/knowledge/repos/vllm-omni/models/magi2/rules.md @@ -0,0 +1,31 @@ +--- +title: "MAGI-2 BF16 routed MoE 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models] +sources: ["PR #7206"] +--- + +# MAGI-2 BF16 routed MoE 规则 + +## Direct 代码快速入口 + +| PR 描述信号 | 规则组 | 第一批源码 | +|---|---|---| +| BF16 dispatch、packing/cache、route buffer、reload/device move | `MAGI2-MOE-1a` | `vllm_omni/diffusion/models/magi2/mh_moe.py::Magi2MultiHeadMoE`；`vllm_omni/diffusion/models/magi2/modeling_magi2.py` 的 weight-loading caller | +| grouped GEMM、SwiGLU7、head/expert routing、launch config | `MAGI2-MOE-1b` | `vllm_omni/diffusion/models/magi2/fused_moe_kernels.py::invoke_fused_moe_bf16`；`tests/diffusion/models/magi2/test_moe_kernel_contract.py` | + +## MAGI2-MOE-1a — Fused BF16 dispatch 必须保留 eligibility 和派生 cache 失效 + +- 触发：修改Magi2MultiHeadMoE本地forward、gate/uppacking、routebuffers或checkpointreload。 +- 强制：仅eager CUDA/MUSA+BF16+非MAGI2_DETERMINISTIC且不在torch.compile内走fused；CUDA还要求tokens>=4096，MUSA无该CUDA门槛，其余native。gate/up相邻行pack在load后准备，nonpersistentcache按parameterID、storagepointer、可用_version、device/dtype失效；inference tensor无version时显式reload仍清cache，.to/_apply清派生weight和routebuffer。routebuffer按upperbound容量重用/resize，livepaddedcount留device，block超过liveprefix提前返回。 +- 禁止：恢复已移除opt-inenv；每forward重新复制完整expertbank；用.item()读取paddedcount；把derivedweights加入checkpoint；跨reload/devicechange使用旧storage；把operator历史诊断说成视频E2E加速或确定性保证。 +- 验收：CUDA4095/4096、MUSA、CPU/nonBF16/deterministic/compile、empty、inplace/mmap/inference reload、device移动、bufferreuse/resize分别回归；真实head/expert数下独立量cache/packing成本与native对照。 ^[PR #7206] + +## MAGI2-MOE-1b — Routed GEMM 必须保持 head 归属、数值与 launch 合同 + +- 触发：修改BF16groupedgate/up+downkernel、routinglayout、SwiGLU7或launchconfig。 +- 强制：按localhead加expertbankoffset，gate/up以相邻行pair，FP32dotaccumulation通过clampedSwiGLU7(gate<=7、up[-7,7]、alpha1.702、up+1)，写BF16intermediate；down另以FP32累加并只乘一次routeweight再BF16cast，按topk恢复原token/head。launcher检查BF16A/B/C、contiguous输出/routeweights、整数1Droutebuffers、共同device、非空匹配维度、even gate/up、positive topk、blockaligned容量和白名单launchvalues。非法配置拒绝；missingexpert down置零，overflow/paddingmask在int64offset计算前建立。 +- 禁止：把分段BF16rounding说成全程FP32/逐位nativeparity；重复routeweight、混headexpertindex；只测toyexperts而不测productionlocalshape；用BF16atomicadd缺失代表此fusedpath不能运行，它不依赖该atomic。 +- 验收：wrapper负例、unevenK/N、padding/empty/missingexpert、多head/topk、routeweight一次和int64offset合同；对独立native/CPUreference采用明确容差并检查真正fuseddispatch，性能报告保留平台、warmup/cache和operator边界。 ^[PR #7206] diff --git a/knowledge/repos/vllm-omni/models/mammoth-moda2/_index.md b/knowledge/repos/vllm-omni/models/mammoth-moda2/_index.md index c87ac8ae3..67376a996 100644 --- a/knowledge/repos/vllm-omni/models/mammoth-moda2/_index.md +++ b/knowledge/repos/vllm-omni/models/mammoth-moda2/_index.md @@ -1,7 +1,7 @@ --- title: "MammothModa2（Preview/Dev 的 AR→DiT 与 AR-only 拓扑）" created: 2026-09-04 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, models, diffusion] sources: ["PR #6694", vllm_omni/model_executor/models/mammoth_moda2/, vllm_omni/diffusion/models/mammoth_moda2/, vllm_omni/model_executor/stage_input_processors/mammoth_moda2.py, vllm_omni/deploy/mammoth_moda2.yaml, vllm_omni/deploy/mammoth_moda2_ar.yaml] @@ -49,3 +49,8 @@ sources: ["PR #6694", vllm_omni/model_executor/models/mammoth_moda2/, vllm_omni/ - Preview/Dev 的支持范围以合并后的 upstream docs/registry 为准：Preview 有文生图与 image understanding；Dev 在 supported-model 表只声明 AR-only image understanding。不要从 PR 作者的单张 GPU 示例外推硬件、吞吐、质量或全部任务支持。^[PR #6694] + +T2I None/default 与16-pixel dimension validation：MAMMO-1d；沿本目录 rules 审查入口核对。 +- [step execution、request scheduler 与 CFG](rules-step-runtime.md) +- [TeaCache joint context 与 fused QK RoPE](rules-cache-kernels.md) +- [image route、AR grid 与 legacy image stage](rules-image-serving.md) diff --git a/knowledge/repos/vllm-omni/models/mammoth-moda2/rules-cache-kernels.md b/knowledge/repos/vllm-omni/models/mammoth-moda2/rules-cache-kernels.md new file mode 100644 index 000000000..46bed5921 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/mammoth-moda2/rules-cache-kernels.md @@ -0,0 +1,25 @@ +--- +title: "MammothModa2 TeaCache 与 QK RoPE 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, diffusion] +sources: ["PR #5357", "PR #7969"] +confidence: high +--- + +# MammothModa2 TeaCache 与 QK RoPE 规则 + +## MAMMO-TEACACHE-1a — cache residual 必须对应完整 joint transformer 输入和真实 CFG branch + +- 触发：修改 Mammoth TeaCache backend/extractor、DiT refiners、postprocess 或 conditional/unconditional 调用。 +- 强制：通过共享 hooks backend 将 TeaCache 安装到 `gen_transformer`，并保持 runtime 的 `transformer` alias 指向该 DiT。Extractor 复用模型 input validation、embedding preparation 与 refiners，再以各 row 的有效 text/image spans 构建 joint hidden states 和 mask；第一 transformer layer 的 norm1 modulation input 驱动 cache decision。复用 residual 后仍执行原 norm_out 和 image-span unpatchify。正/负 CFG 调用显式传 `teacache_branch=positive/negative`，hook 验证 branch 值并优先使用该 hint，不能依赖固定交替次数推断间歇性 uncond。 +- 禁止：把 text/image refiner 或 VAE 输出作为 transformer residual缓存；让正负 branch 共用 residual；将 1024×1024、50-step full-compute traces 拟合的 coefficients 和默认 threshold 0.075 当成其他尺寸、schedule、task 的质量保证。 +- 验收：对照无 hook forward 核对 extractor 的 joint sequence、modulated input 和 postprocess；交错正负与跳过 uncond 的序列证明 branch 隔离，错误 branch 显式失败；backend 注册和 request/cache reset 必须走生产 runtime，质量仅按固定输入/seed/steps/threshold复核。^[PR #5357] + +## MAMMO-QKROPE-1a — fused QK norm/RoPE 只消费模型保证的 adjacent-pair tables + +- 触发：修改 Mammoth `AttnProcessor`、`_apply_qk_norm_rope`、rotary table packing 或 shared fused operator接入。 +- 强制：只有模型的 `RotaryPosEmbedReal` producer 保证相邻 lanes 重复时才 opt in fusion；检查 Q/K norm epsilon、batch/sequence/head width、CUDA kernel support 和 cos/sin shape/device/dtype。每 forward 为 context/noise/joint 三种 tables 各准备一次 packed FP32 table，由对应 layers 复用；显式 `None` 表示 support/token gate 已选择 eager，不能在每层再次选择。保留 Q/K 不同 head counts 的 GQA layout；embedding-only `_prepare_embeddings` 返回值保持原语义。 +- 禁止：将通用独立 even/odd lanes 的四通道 RoPE 当成 adjacent-pair table；取消平台、dtype、geometry 或 token gate；将一次 RTX 4090 crossover 或 fused-kernel parity 外推为所有模型/平台速度和端到端质量保证。 +- 验收：模型-produced repeated-pair 与普通 independent-lane tables 分别核对 fused/eager 和 fallback；覆盖 GQA、prepared-None、shape/device/dtype 错误与 token threshold，并断言每种 table 只准备一次。^[PR #7969] diff --git a/knowledge/repos/vllm-omni/models/mammoth-moda2/rules-image-serving.md b/knowledge/repos/vllm-omni/models/mammoth-moda2/rules-image-serving.md new file mode 100644 index 000000000..b623df88b --- /dev/null +++ b/knowledge/repos/vllm-omni/models/mammoth-moda2/rules-image-serving.md @@ -0,0 +1,18 @@ +--- +title: "MammothModa2 image serving 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, serving] +sources: ["PR #7293"] +confidence: high +--- + +# MammothModa2 image serving 规则 + +## MAMMO-SERV-1a — image 能力、AR envelope 与 decoded output 不得绑定唯一 stage type + +- 触发：修改 image-route admission、Mammoth multistage model resolution、T2I prompt/sampling 或 image output extraction。 +- 强制：image-generation admission 通过共享 `is_image_generation_stage` 识别 classical diffusion，或声明 `final_output=true` 且 `final_output_type=image/images` 的 final stage；优先读取 diffusion od_config model class，缺失时从 image-output stage 的 wrapper arch 解析。T2I builder 按 resolved model class dispatch，在没有 reference images/bot_task/system-prompt override 的普通 T2I 路径生成 AR envelope；grid budget 为 `ar_height*(ar_width+1)+1`，与 offline 共用 helper。声明的 extra-body args、seed 和 dimensions 进入正确 stage；legacy LLM image stage 一请求一图、拒绝 `n>1`。结果优先取 images，否则使用 shared extractor 从 completion multimodal payload 取图。 +- 禁止：仅因不是 diffusion stage 就返回 503，或迁移为 diffusion stage 后跳过仍必需的 AR prompt builder；丢弃允许的 request model extras；把 image-generation capability check 扩大成 img2img 支持（image edit 仍按 classical diffusion gate）。 +- 验收：legacy generation-LLM 与 diffusion topology 均覆盖 online image generation、wrapper model-class fallback、AR grid预算、extra args 和 completion image extraction；保留不支持 edit/multiple-image 的明确错误。^[PR #7293] diff --git a/knowledge/repos/vllm-omni/models/mammoth-moda2/rules-step-runtime.md b/knowledge/repos/vllm-omni/models/mammoth-moda2/rules-step-runtime.md new file mode 100644 index 000000000..911f8a5e8 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/mammoth-moda2/rules-step-runtime.md @@ -0,0 +1,18 @@ +--- +title: "MammothModa2 step runtime 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, diffusion, scheduler] +sources: ["PR #7954"] +confidence: high +--- + +# MammothModa2 step runtime 规则 + +## MAMMO-STEP-1a — continuous batch 必须按 request 保存 scheduler、conditioning 与 CFG 进度 + +- 触发：修改 `prepare_encode/denoise_step/step_scheduler/post_decode`、Mammoth admission compatibility key 或 step deploy。 +- 强制：admission 将模型实际解析的 step count 写回 sampling；step 模式按 height/width 分组，request 模式仍将 total steps 纳入 compatibility key。每个 `StepRequestState` 独占 seeded latents、timesteps、FlowMatch scheduler、step index、text/image conditioning、guidance scale 和 CFG range；encoding 从 step 0 重启，不宣称支持 mid-denoise resume。标准 text/negative tensors 放 InputBatch，AR image embeddings/masks 放 request extra，并在当前 batch 重新 pad；只有全部无 image layout 或全部完整 image layout 才可同批。每 request 只有一个 latent row；按其 `step_index/total_steps` 决定 CFG，混合 active/inactive 时仅替换 active rows。 +- 禁止：因为 batch 中某 request 需要 CFG 就对全部 rows 强制 blend；共享 mutable scheduler/RNG 或用 batch-global进度；丢弃 resolved step count；将不同 spatial layout 或缺一半的 AR image condition 静默合批。 +- 验收：交错 arrival、不同 total steps/CFG windows、不同 seeds 的请求对照独立执行；覆盖 mixed active CFG、negative fields、AR image layout 错误与一行一请求断言；每 request scheduler step 后递增自己的 index，final decode 使用其 latents 和 VAE scaling/shift。^[PR #7954] diff --git a/knowledge/repos/vllm-omni/models/mammoth-moda2/rules.md b/knowledge/repos/vllm-omni/models/mammoth-moda2/rules.md index f26c872ff..a4caa1ca9 100644 --- a/knowledge/repos/vllm-omni/models/mammoth-moda2/rules.md +++ b/knowledge/repos/vllm-omni/models/mammoth-moda2/rules.md @@ -1,10 +1,10 @@ --- title: "MammothModa2 规则" created: 2026-09-04 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, models, model-executor, diffusion] -sources: ["PR #6694", vllm_omni/model_extras/mammothmodal2_preview.py, vllm_omni/model_executor/models/mammoth_moda2/mammoth_moda2.py, vllm_omni/model_executor/stage_input_processors/mammoth_moda2.py, vllm_omni/diffusion/models/mammoth_moda2/pipeline_mammothmoda2_dit.py, tests/model_extras/test_model_extras.py, tests/e2e/offline_inference/test_mammoth_moda2_expansion.py, "PR #7102"] +sources: ["PR #6694", vllm_omni/model_extras/mammothmodal2_preview.py, vllm_omni/model_executor/models/mammoth_moda2/mammoth_moda2.py, vllm_omni/model_executor/stage_input_processors/mammoth_moda2.py, vllm_omni/diffusion/models/mammoth_moda2/pipeline_mammothmoda2_dit.py, tests/model_extras/test_model_extras.py, tests/e2e/offline_inference/test_mammoth_moda2_expansion.py, "PR #7102", "PR #7482"] confidence: high --- @@ -57,3 +57,11 @@ confidence: high - 强制：跨 stage 传输保持 AR 源 dtype（FP16/BF16）的 contiguous `full_hidden_states`；EngineCore/OmniSerializer 以 raw bytes 保留 bfloat16，不得为“过 numpy 边界”无条件 `.float()` 扩成 FP32。DiT 在选完 text/image 条件行之后，才在既有最终转换点落到 model device/dtype。 - 禁止：在 bridge 或 condition split 处提前 host-side float32 加倍 staging 与 H2D；把精度优化扩成 request-end streaming / Layer-2 lifecycle 设计；或假定旧 FP32 wire 注释仍有效。 - 验收：FP16/BF16 经 serialize/deserialize 往返 dtype 与数值不变；condition split 输出保持 transfer dtype 直至最终 cast；connector 级 BF16 round-trip 覆盖。^[PR #7102] + + +## MAMMO-1d — T2I 尺寸必须在 AR grid 构造前拒绝非法值 + +- 触发：修改 MammothModa2 prompt builder 的 height/width 默认、AR grid 或 image metadata。 +- 强制：每一轴仅 None 使用1024；显式值必须为正且是16的倍数，再以16-pixel patch 构造 grid。三种 registry/model class alias 共享校验；一个轴省略不能掩盖另一轴非法值。 +- 禁止：用 value or 1024 把0变成默认；floor division 静默截断513等尺寸；让 prompt grid 与 metadata/request image size 不一致。 +- 验收：三个 class alias 覆盖双 None、单轴省略、16边界与普通矩形；每轴负数、0、1/15/17/513及另一轴省略的 control 都须 ValueError，合法尺寸的 prompt grid 与 metadata 同时正确。^[PR #7482] diff --git a/knowledge/repos/vllm-omni/models/ming-omni-tts/_index.md b/knowledge/repos/vllm-omni/models/ming-omni-tts/_index.md index 5898b39cd..3502255e2 100644 --- a/knowledge/repos/vllm-omni/models/ming-omni-tts/_index.md +++ b/knowledge/repos/vllm-omni/models/ming-omni-tts/_index.md @@ -1,7 +1,7 @@ --- title: "Ming-Omni-TTS" created: 2026-07-20 -updated: 2026-07-31 +updated: 2026-10-06 type: index tags: [vllm-omni, models, model-executor] sources: ["PR #4341", vllm_omni/model_executor/models/ming_tts/] @@ -33,3 +33,5 @@ LLM/conditioning 产生 latent condition，CFM/DiT solver 运行 ODE/SDE 采样 - 审查 Ming-TTS dense/MoE、CFM CUDA Graph、solver dtype、CFG 或 Audio VAE 输出。 - 通用公开 speech 参数与 streaming 错误转到 [Serving rules](../../components/serving/rules.md)。 + +ISTFT FP32 operands → complex64：MING-4a；沿本目录 rules 审查入口核对。 diff --git a/knowledge/repos/vllm-omni/models/ming-omni-tts/rules.md b/knowledge/repos/vllm-omni/models/ming-omni-tts/rules.md index 8ffe09182..24942272a 100644 --- a/knowledge/repos/vllm-omni/models/ming-omni-tts/rules.md +++ b/knowledge/repos/vllm-omni/models/ming-omni-tts/rules.md @@ -1,10 +1,10 @@ --- title: "Ming-Omni-TTS 规则" created: 2026-07-20 -updated: 2026-09-04 +updated: 2026-10-06 type: rule tags: [vllm-omni, models, model-executor] -sources: ["PR #4341", "PR #6119"] +sources: ["PR #4341", "PR #6119", "PR #7538"] confidence: high --- @@ -60,3 +60,11 @@ confidence: high - 强制：引擎内的 `SpeakerEmbeddingExtractor` 默认使用 `allow_download=False`，依赖已预取且包含 `campplus.onnx` 的模型目录；引擎构造前的离线 `end2end` 提取必须显式传 `allow_download=True`，并限制解析到 `campplus.onnx`。 - 禁止：在线请求路径在缓存未命中时隐式联网，离线引擎前处理使用 cache-only 却假定文件已经预取，或在 `VLLM_USE_MODELSCOPE` 下假定 HF 的 `local_files_only` 分支能够完成 ModelScope 首次解析。 - 验收：覆盖本地已解析目录、缓存命中/未命中和离线显式下载三条路径；确认缺失 `campplus.onnx` 明确失败，ModelScope 预取后的目录走短路，并验证 serving 与离线提取都能产生 speaker embedding。 ^[PR #6119] + + +## MING-4a — ISTFT complex spectrum 必须由 FP32 operands 显式构造 + +- 触发：修改 common/ming/audio_dsp.py 的 ISTFTHead.forward 或 magnitude/phase 到 complex spectrum 的转换。 +- 强制：先将 mag、x、y 显式转 float32，再构造 mag * (x + 1j*y)，得到 complex64；后续 streaming buffers、window 与 fused/native ISTFT 选择合同保持。 +- 禁止：依赖 BF16/FP16 complex promotion，使新 torch dtype 落到 FFT 不支持的 bcomplex32/complex32；把局部 spectrum precision 修复写成全模型 FP32 或通用 XPU 性能提升。 +- 验收：覆盖 BF16、FP16、FP32 spectrum dtype 与 native FFT；FP32 control 要保持逐值一致；fused gate 与 streaming last-chunk/buffer 路径分别验证。当前 PR 无新增 regression test，模型 smoke 不能替代这些 dtype 边界检查。^[PR #7538] diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/_index.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/_index.md index 9712b8554..d1632134e 100644 --- a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/_index.md +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/_index.md @@ -1,10 +1,10 @@ --- title: "MiniCPM-o 4.5" created: 2026-07-20 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, models, model-executor] -sources: ["PR #3642", "PR #5382", "PR #5524", "PR #5638", "PR #5792", "PR #5869", "PR #6056", "PR #6154", "PR #6170", "PR #6318", "PR #6404", "PR #6678", "PR #6628", tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/engine/duplex/test_duplex_deploy_config.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py, vllm_omni/experimental/fullduplex/client.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/stage0.py, vllm_omni/entrypoints/duplex/realtime_input.py, vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py, vllm_omni/model_executor/models/minicpmo_4_5/, tests/model_executor/models/minicpmo_4_5/test_cuda_graph_wrapper.py, tests/model_executor/models/minicpmo_4_5/test_cfm_graph_capture_gating.py, tests/model_executor/models/minicpmo_4_5/test_talker_batching.py, vllm_omni/platforms/npu/models/minicpmo_4_5_code2wav.py, "PR #6082", "PR #6587"] +sources: ["PR #3642", "PR #5382", "PR #5524", "PR #5638", "PR #5792", "PR #5869", "PR #6056", "PR #6154", "PR #6170", "PR #6318", "PR #6404", "PR #6678", "PR #6628", tests/dfx/perf/tests/test_minicpmo_4_5.json, tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json, tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py, tests/engine/duplex/test_duplex_deploy_config.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py, vllm_omni/experimental/fullduplex/client.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/stage0.py, vllm_omni/entrypoints/duplex/realtime_input.py, vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py, vllm_omni/model_executor/models/minicpmo_4_5/, tests/model_executor/models/minicpmo_4_5/test_cuda_graph_wrapper.py, tests/model_executor/models/minicpmo_4_5/test_cfm_graph_capture_gating.py, tests/model_executor/models/minicpmo_4_5/test_talker_batching.py, vllm_omni/platforms/npu/models/minicpmo_4_5_code2wav.py, "PR #6082", "PR #6587", "PR #8007", "PR #8332", "PR #8443", "PR #8462", "PR #8222"] confidence: high --- @@ -34,7 +34,7 @@ Step-Audio2 的 token2wav。engine cache/profile 与 fallback 门禁见 MCPMO-1c 四份 bundled deploy 在 CUDA 上另默认开启 HiFT graph：只 capture pre-iSTFT 子图，按 connector chunk/cache shape 预捕并限量 lazy capture，且不服从 stage `enforce_eager`；非 CUDA 回 eager。 HiFT/CFM 的 shape、显存、并发、整代退休与部分-graph 性能证据边界见 -[Code2Wav CUDA graph 规则](rules-cuda-graphs.md)。 新增核对：MCPMO-1k。 +[Code2Wav CUDA graph 规则](rules-cuda-graphs.md)。 新增核对：MCPMO-1k；legacy step-level 与 Whole-Euler 必须按实际路径区分。 Thinker 的 Whisper/APM audio encoder 仍构造 dense `[B,1,T,T]` mask；chunk mask 已用 broadcasted query/key index 代替逐 row Python fill，但不改变 chunk/left-context/lookahead @@ -71,3 +71,10 @@ Realtime video 不以「每 200 ms append」定义 frame cadence：首个约 103 - [MiniCPM-o 4.5 Code2Wav 运行时合同](rules-code2wav-runtime.md)：`MCPMO-1f`, `MCPMO-1g`, `MCPMO-1h`, `MCPMO-1i`。 - [rules-loader](rules-loader.md) 新增核对：MCPMO-5a2。 + +- [Whole-Euler 与 resident graph 合同](rules-resident-graphs.md)：MCPMO-GRAPH-1a–1e，涵盖显式 backend、共享 arena、总预算、ragged fallback、ordinary TF32 与 exact continuation 预捕。 +- [输入 encoder graph 合同](rules-encoder-graphs.md)：MCPMO-ENCODER-1a/1b，保持 exact packed batch、资格、输出与 stream 所有权、admission 与 terminal capture failure。 +- camera tile 与 first-append reservation 查本页已链接的 native duplex 规则 MCPMO-DUPLEX-TILE-1a。 + +- [MRv2 turn 与输出状态](rules-mrv2-state.md)：MCPMO-MRV2-1a–1d，区分 native V1 duplex、live row ledger、codec sampling、共享 V1 hooks 与显式 cached ISTFT。 +- [native LISTEN speech unit的turn end](rules-duplex-turn-closure.md)。 diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-cuda-graphs.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-cuda-graphs.md index e2b3e1cdd..9569444d9 100644 --- a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-cuda-graphs.md +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-cuda-graphs.md @@ -1,10 +1,10 @@ --- title: "MiniCPM-o 4.5 Code2Wav CUDA graph 规则" created: 2026-09-04 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, models, model-executor] -sources: [vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py, vllm_omni/model_executor/models/minicpmo_4_5/batched_token2wav.py, vllm_omni/model_executor/models/minicpmo_4_5/cuda_graph_wrapper.py, tests/model_executor/models/minicpmo_4_5/test_cfm_graph_capture_gating.py, tests/model_executor/models/minicpmo_4_5/test_cuda_graph_wrapper.py, "PR #5869", "PR #6082", "PR #6587", "PR #7416"] +sources: [vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py, vllm_omni/model_executor/models/minicpmo_4_5/batched_token2wav.py, vllm_omni/model_executor/models/minicpmo_4_5/cuda_graph_wrapper.py, tests/model_executor/models/minicpmo_4_5/test_cfm_graph_capture_gating.py, tests/model_executor/models/minicpmo_4_5/test_cuda_graph_wrapper.py, "PR #5869", "PR #6082", "PR #6587", "PR #7416", "PR #8007", "PR #8443", "PR #8222"] confidence: high --- @@ -13,24 +13,28 @@ confidence: high ## MCPMO-1d — HiFT graph 只捕获稳定 pre-iSTFT 子图并限界 shape cache - 触发：`enable_hift_graph`、capture batch/chunk 配置、HiFT inference 分段或 streaming cache shape。 -- 强制：只在 parameter device 为 CUDA 时启用；非 CUDA 回 eager。capture bucket 由 codec chunk、left context、Flow lookahead、token→mel ratio 与 mel/source cache 推导；启动预捕 uncached/cached shape，未知 final shape 最多 lazy capture 8 个，超限或无可容纳 batch 时回 eager。active stream capture 中禁止嵌套 capture/replay并回 eager；静态输入先清零再复制真实 batch，输出 slice 后 clone。 -- 强制：graph 只覆盖 `_inference_pre_istft`；`torch.istft` 的 NOLA CPU-GPU sync 与 waveform clamp 留在 eager `_finalize_decode`。HiFT window/harmonic IDs 必须是随 module device 移动的 non-persistent buffers。开关独立于 stage `enforce_eager`；四份 bundled MiniCPM-o deploy 默认开启。 -- 禁止：无 connector 配置、非正 chunk、负 left context、source/mel cache 不整除时静默 capture；将未来 `initial_codec_chunk_frames` 当已预捕（当前会 lazy capture）；把 lazy graph limit 说成总显存上界，因为启动 graphs、global pool、static tensors 与其他 graph owners 仍驻留。active-capture 分支虽绕过 nested replay，却调用包含 NOLA sync 的完整 eager decode；未做真实 outer-graph 测试前，不能把日志中的 “fallback” 当成可捕获保证。custom `capture_batch_sizes` 也尚未校验正数/去重。 -- 验收：CUDA test 比较 cached/uncached graph 与 eager，CPU/mock 覆盖 lazy capture/limit fallback，deploy test 固定默认开关；另测非 CUDA disable、invalid config、unsupported batch、nested capture 和 variable final chunk。static input/output 与 lazy graph map 为无锁可变状态；并发 replay/capture 需先证明被上层串行化或增加互斥与重叠调用测试。PR 的单 A800 profile 约 30→5 ms/chunk、250→43 ms/request，只绑定部分 graph、单 prompt/commit `8fa28d88`，无 repeats/端到端质量，不能泛化为稳定 speedup。^[PR #5869] +- 强制：只在 parameter device 为 CUDA 时启用；非 CUDA 回 eager。capture bucket 由 codec chunk、left context、Flow lookahead、token→mel ratio 与 mel/source cache 推导；新版在首个 forward 的预捕阶段准备 uncached/cached、initial 和配置的 exact shape；只有合法 capture buckets 的缺失 graph 可 lazy capture（默认上限 8），未知 final shape、超限或无法容纳 batch 时回 eager。active stream capture 中禁止嵌套 capture/replay并回 eager；静态输入先清零再复制真实 batch，输出 slice 后 clone。 +- 强制：graph 只覆盖 `_inference_pre_istft`；ISTFT 与 waveform clamp 留在 graph 外的 `_finalize_decode`；MiniCPM 显式 cached ISTFT 的首 shape 仍检查 NOLA，后续复用合同见 MCPMO-MRV2-1d。HiFT window/harmonic IDs 必须是随 module device 移动的 non-persistent buffers。开关独立于 stage `enforce_eager`；四份 bundled MiniCPM-o deploy 默认开启。 +- 禁止：无 connector 配置、非正 chunk、负 left context、source/mel cache 不整除时静默 capture；不核对新版 exact initial/continuation 配置就将 `initial_codec_chunk_frames` 当已预捕（旧版仅 lazy capture，新版预捕合同见 MCPMO-GRAPH-1e）；把 lazy graph limit 说成总显存上界，因为启动 graphs、global pool、static tensors 与其他 graph owners 仍驻留。active-capture 分支虽绕过 nested replay，却调用完整 eager decode（native ISTFT 或 cached envelope 首次 miss 仍可能 NOLA sync）；未做真实 outer-graph 测试前，不能把日志中的 “fallback” 当成可捕获保证。custom `capture_batch_sizes` 也尚未校验正数/去重。 +- 验收：CUDA test 比较 cached/uncached graph 与 eager，CPU/mock 覆盖 lazy capture/limit fallback，deploy test 固定默认开关；另测非 CUDA disable、invalid config、unsupported batch、nested capture 和 variable final chunk。static input/output 与 lazy graph map 为无锁可变状态；并发 replay/capture 需先证明被上层串行化或增加互斥与重叠调用测试。PR 的单 A800 profile 约 30→5 ms/chunk、250→43 ms/request，只绑定部分 graph、单 prompt/commit `8fa28d88`，无 repeats/端到端质量，不能泛化为稳定 speedup。^[PR #5869] ^[PR #8443] ^[PR #8222] -## MCPMO-1e — CFM DiT CUDA Graph 必须按 shape 整代退休并保持 eager parity +## MCPMO-1e — legacy step-level CFM DiT Graph 必须按 shape 整代退休并保持 eager parity -- 触发：修改 MiniCPM-o 4.5 Code2Wav CFM DiT estimator 的 CUDA Graph 开关、`blocks_forward_chunk` graph target、按 shape 捕获/重放、cache retirement 或 eager fallback。 +- 触发：修改 MiniCPM-o 4.5 Code2Wav legacy step-level CFM DiT estimator 的 `blocks_forward_chunk` graph target、按 shape 捕获/重放、cache retirement 或 eager fallback；Whole-Euler 的完整 solver capture 另见 MCPMO-GRAPH-1a/1b。 - 强制：仅在 CUDA 且显式启用时捕获 `estimator.blocks_forward_chunk`；`t_embedder`、`torch.cat`、`expand`、buffer 准备和 10-step Euler loop 保持 eager。graph key 必须包含所有输入的 shape、dtype 和 device；`cnn_cache=None` 与 `att_cache=None` 转为 graph 可接受的零值时，必须保持与 eager `[None] * depth` 的首 chunk 数值语义，重放结果和 cache 输出需 clone。 - 强制：`cfm_max_graphs` 是一代的有界容量。满载遇到新 shape 时，先同步，再只对本 wrapper 拥有的全部 `CUDAGraph` 调用 `reset()` 并清空 cache，随后捕获新一代的首图；绝不可逐 entry/LRU eviction。不得进行 process-wide cuBLAS workspace 清理或 `empty_cache()`：HiFT 等共享 global graph pool 的存活图不属于 CFM 的退休范围。`max_graphs <= 0` 必须禁用捕获并 eager。 - 强制：warmup 或 capture 发生异常时，在该 wrapper 的剩余生命周期内保持禁用并退休其自有 generation；不能在可能脏的 capture stream/allocator 状态继续捕获。无法由 key 重建的 dtype 仅标记该 shape eager，不得禁用其他可捕获 shape；capture admission 不得依赖 device free-memory。维护固定字段的 calls/hits/captures/flushes/eager 累计 telemetry。 - 禁止：捕获整个 `_estimator_step` 或手写替代 graph target；使用无界 shape cache、逐图 eviction、跨 owner 的 process-wide cleanup，或在 nested capture 中重放；捕获失败后伪装成功；把 CFM graph 的配置、性能或 parity 结论外推到 HiFT、encoder、TRT 或其他模型。 - 验收：CUDA 测试覆盖 uncached/cached shape 的 graph/eager 数值 parity、`None` cache parity、lazy capture、整代 flush 后 HiFT replay parity、capture/warmup failure 后不再 capture、unsupported dtype shape-local eager 与 `max_graphs <= 0` eager；CPU/mock 覆盖 active capture/non-CUDA、free-memory 非 gating、telemetry 和 inert non-CUDA memory reporting，并确认 deploy 配置值实际到达 estimator。^[PR #6082] ^[PR #6587] -## MCPMO-1k — CFM graph 的变长 chunk 必须先对齐帧桶，padding 不得进入 cache +## MCPMO-1k — 显式启用 CFM 帧桶时，padding 不得进入有效输出与 cache - 触发：修改 MiniCPM-o 4.5 Code2Wav 的 CFM decode chunk、`cfm_graph_bucket_frames`、reference-audio 长度，或 attention/CNN cache 边界。 -- 强制：捕获前把 mel 帧 pad 到 `bucket_frames` 的倍数，输出再裁回有效帧，使 chunk 长度落在同一 grid。每份开启 CFM graph 的 shipping profile 必须显式写 `cfm_graph_bucket_frames`（当前 16），不能靠代码默认。`bucket_frames <= 1`、graphs 已禁用、ragged valid-lengths，或 pad 会越过 decoder noise capacity 时不 pad。 +- 强制：显式开启 `cfm_graph_bucket_frames > 1` 且 graph 路径适用时，把 mel 帧 pad 到桶的倍数，输出裁回有效帧；profile 的 bucket 设置要与实际 graph 路径核对。Whole-Euler 部署允许默认 0、不做旧版 frame padding；不得继续要求所有 CFM profile 固定为 16。`bucket_frames <= 1`、graphs 已禁用、ragged valid-lengths，或 pad 会越过 decoder noise capacity 时不 pad。 - 强制：padded 列必须被真实 `attn_mask` 排除；零 key 仍会计入 softmax 分母。成为下一 chunk 的 key 或左上下文之前，清零 attention cache 的 padded 列，以及各 block CNN cache 中来自 padding 的尾部。speaker 向量会铺满每一帧，清零必须作用在 clone 上。conditioning pad 复制最后一帧有效值。reference audio 先转 mono、抗混叠重采样到 24 kHz，再按 `ref_audio_max_seconds`（默认 6s）截断或补零，使各请求共享一个 L0；默认 prompt 走同一路径，`soundfile` 的 `(samples, channels)` 先转置。 -- 禁止：让首尾 chunk 或 `plan_token2wav_encode_slices` 的任意长度各自成为 graph key；只清零 `x`/`mu`/`cond` 而留下非零 speaker 列；把桶化 parity 外推到 HiFT、TRT 或其他模型。 -- 验收：覆盖 pad/trim、noise overflow 不 pad、attn_mask 与 cache 清零、reference 归一化后的单一 L0，以及四份 CFM-graph profile 都带上 `cfm_graph_bucket_frames`。^[PR #7416] +- 禁止：在已选择 frame bucketing 的路径中让任意有效长度另成 graph key；只清零 `x`/`mu`/`cond` 而留下非零 speaker 列；把桶化 parity 外推到 HiFT、TRT 或其他模型。 +- 验收：覆盖 pad/trim、noise overflow 不 pad、attn_mask 与 cache 清零、reference 归一化后的单一 L0，以及 shipping profile 的真实 bucket 设置；Whole-Euler 另检查同 query bucket 中不同有效长度的 recurrent parity。^[PR #7416] ^[PR #8007] + +Whole-Euler、resident slot 的总预算及新版 exact HiFT/flow-encoder 预捕见 +[resident graph 合同](rules-resident-graphs.md)；输入 encoder 的资格与 terminal failure +见 [encoder graph 合同](rules-encoder-graphs.md)。 diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-duplex-turn-closure.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-duplex-turn-closure.md new file mode 100644 index 000000000..70f7054d8 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-duplex-turn-closure.md @@ -0,0 +1,17 @@ +--- +title: "MiniCPM-o duplex turn closure 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models] +sources: ["PR #8227"] +--- + +# MiniCPM-o duplex turn closure 规则 + +## MCPMO-TURN-1a — turn_eos 后关闭 speech unit 的 LISTEN 不能吞掉最终 Talker handoff + +- 触发：修改 MiniCPM-o DuplexPolicy、plugin decide_output、native data-plane decision 或 llm2tts delta 边界。 +- 强制：最后一枚为 LISTEN 时，从当前 unit 反向扫描，只在遇到上一 LISTEN/chunk_eos/chunk_tts_eos 前已出现 turn_eos 才判为 speech-closure；plugin 和 native data-plane 都保留该 unit 给 Talker。completion、cumulative 与 segment token 视图选择最长可用当前历史，不能只看最后 delta。llm2tts 在 turn_eos 结束 handoff，尾 LISTEN 不进入 TTS；plain LISTEN 和 mid-speech forced LISTEN 继续 direct-response。 +- 禁止：把所有 ending-LISTEN unit 直接短路；跨前一 unit 找旧 turn_eos；去掉 LISTEN-terminated 最终speech而丢掉 turn end/own-token hidden rows；把本 native duplex 修复当 MRv2 turn/duplex eligibility 变更。 +- 验收：policy truth table、previous terminator、delta+cumulative、plugin/native 两条decision、最终speech+turn_end与plain LISTEN分别回归；soft interrupt/response.done 和下一unit hidden-state归属另做真实模型验收。 ^[PR #8227] diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-duplex.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-duplex.md index 80024a380..220ca7e28 100644 --- a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-duplex.md +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-duplex.md @@ -1,15 +1,32 @@ --- title: "MiniCPM-o 4.5 native duplex 规则" created: 2026-09-04 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, models, model-executor] -sources: ["PR #6318", "PR #6346", "PR #6404", "PR #6458", "PR #6619", "PR #6630", "PR #6678", "PR #6626", "PR #6767", "PR #6821", vllm_omni/deploy/minicpmo_4_5.yaml, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_tts.py, vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py, tests/model_executor/models/minicpmo_4_5/test_talker_batching.py, vllm_omni/deploy/minicpmo_4_5_2gpu.yaml, vllm_omni/deploy/minicpmo_4_5_3gpu.yaml, vllm_omni/deploy/minicpmo_4_5_8x4090.yaml, examples/online_serving/minicpmo/realtime_duplex_demo.py, vllm_omni/experimental/fullduplex/client.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/adapter.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/session.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/stage0.py, vllm_omni/entrypoints/duplex/realtime_input.py, vllm_omni/entrypoints/duplex/runtime_adapter.py, vllm_omni/entrypoints/duplex/runtime_bridge.py, vllm_omni/entrypoints/duplex/serving.py, vllm_omni/entrypoints/duplex/session_runner.py, vllm_omni/experimental/fullduplex/video_stacking.py, tests/config/test_config_factory.py, tests/engine/duplex/test_duplex_deploy_config.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py, tests/entrypoints/openai_api/test_duplex_handler.py, tests/examples/test_minicpmo_realtime_duplex_simple_demo.py, "PR #6529", vllm_omni/entrypoints/duplex/protocol.py, vllm_omni/entrypoints/duplex/realtime_state.py, tests/entrypoints/openai/test_duplex_protocol.py, "PR #6799", "PR #7416"] +sources: ["PR #6318", "PR #6346", "PR #6404", "PR #6458", "PR #6619", "PR #6630", "PR #6678", "PR #6626", "PR #6767", "PR #6821", vllm_omni/deploy/minicpmo_4_5.yaml, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_tts.py, vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py, tests/model_executor/models/minicpmo_4_5/test_talker_batching.py, vllm_omni/deploy/minicpmo_4_5_2gpu.yaml, vllm_omni/deploy/minicpmo_4_5_3gpu.yaml, vllm_omni/deploy/minicpmo_4_5_8x4090.yaml, examples/online_serving/minicpmo/realtime_duplex_demo.py, vllm_omni/experimental/fullduplex/client.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/adapter.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/session.py, vllm_omni/model_executor/models/minicpmo_4_5/duplex/stage0.py, vllm_omni/entrypoints/duplex/realtime_input.py, vllm_omni/entrypoints/duplex/runtime_adapter.py, vllm_omni/entrypoints/duplex/runtime_bridge.py, vllm_omni/entrypoints/duplex/serving.py, vllm_omni/entrypoints/duplex/session_runner.py, vllm_omni/experimental/fullduplex/video_stacking.py, tests/config/test_config_factory.py, tests/engine/duplex/test_duplex_deploy_config.py, tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py, tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py, tests/entrypoints/openai_api/test_duplex_handler.py, tests/examples/test_minicpmo_realtime_duplex_simple_demo.py, "PR #6529", vllm_omni/entrypoints/duplex/protocol.py, vllm_omni/entrypoints/duplex/realtime_state.py, tests/entrypoints/openai/test_duplex_protocol.py, "PR #6799", "PR #7416", "PR #8462", "PR #8007"] confidence: high --- # MiniCPM-o 4.5 native duplex 规则 +## MCPMO-DUPLEX-TILE-1a — camera token 预算必须与真正 image processor 的 tile 一致 + +- 触发：修改 native duplex vision tile、first-append prompt reservation 或 processor lookup。 +- 强制：从 Stage 0 实际 checkpoint 的 image-processor config 读取正整数 + `scale_resolution`，以平方作为 tile pixels；使用 + `ImageProcessingMixin.get_image_processor_dict(..., local_files_only=True)` 的查找顺序， + nested `processor_config.json.image_processor` 优先，否则读取 `preprocessor_config.json`。 + 不从受 `hf_overrides` 影响的 `hf_config.slice_config/image_size` 推断 processor tile。 +- 强制：无可读 processor config、非法 side 或 bool 时保留 `None`，继续 sliced-count + reservation；不访问网络补取。一个 append 的 vision token count 只计算一次，first-unit + 分支复用它；独立 audio budget 与公开 `duplex_scheduler_token_budget` 合同保持一致。 +- 禁止：用 config.json 的较大 tile 缩减真实 processor 的 sliced token 预算；把 missing + config 视为零 vision tokens；为相同 append 重复 decode base frame。 +- 验收：processor/config tile 故意不一致、nested config 优先、非法/缺失本地配置、 + 首 append 仅一次 base-frame decode 和 audio budget 不变；500×500 的已切片 pair 在 + processor side448/config side560 时仍保留264 tokens，而不是132。^[PR #8462] + ## MCPMO-4b — native duplex session update 必须如实拒绝不可重建的上下文 - 触发：运行中更新 MiniCPM-o native duplex 的 `instructions`/native-mode。 @@ -47,14 +64,16 @@ confidence: high ## MCPMO-4e — shipping profile 必须共服 chat 与 native duplex - 触发：修改任一 `minicpmo_4_5*.yaml`、duplex session placement 或 Stage 1 Talker 默认/运行时采样参数。 -- 强制：所有 shipping profile 设置 `session_mode=duplex`，且 `active_stream_window` 只限制 - first-packet admission，不是 session lifetime cap。default/2gpu/3gpu profile 的 window、 - `duplex_session.max_sessions` 与 Stage 0/1 `max_num_seqs` 均为 4；8×4090 profile 因 24GB - 资源边界保持 window/stage capacity 1，并沿用 default session capacity 1。replica overlay 从 base - 继承这些字段。YAML 保留 chat 的 Stage 1 `min_tokens=50` 与既有 codec 参数，native duplex adapter - 仅在该 session 的 runtime config 覆盖 `min_tokens=0`；AR stage 保留默认 async scheduler。^[PR #6767] +- 强制：所有 shipping profile 设置 `session_mode=duplex`；shipping profile 省略 + `active_stream_window`，默认 0 的 unlimited path 见 MCPMO-4j。PR #8007 合并的 default + profile 将 `duplex_session.max_sessions` 与 Stage 0/1/2 `max_num_seqs` 提升至 16， + 2gpu/3gpu profile 仍为 4；8×4090 的 stage capacity 与 default session capacity 保持 1。 + replica overlay 从 base 继承这些字段。YAML 保留 chat 的 Stage 1 `min_tokens=50`，native + duplex adapter 只在 session runtime config 覆盖为 0；AR stage 保留默认 async scheduler。 + default Talker 的 KV budget 为 4 GiB，因为 physical KV preemption 后的增量 full-attention + replay 尚不支持，不能仅增加 admission 而忽略实际 KV 容量。^[PR #6767] ^[PR #7416] ^[PR #8007] - 禁止：恢复独立 `minicpmo_4_5_duplex.yaml`；在 replica overlay 重复并漂移 session 配置；为 duplex 全局关闭 async scheduling；或把 runtime-only floor 写回 YAML 而改变 chat TTS。 -- 验收：展开四份 base profile 及三个 replica overlay，断言全部 duplex-enabled、窗口与 session capacity 正确；同一 server 分别验证 `/v1/realtime?duplex=1` 和 chat，前者 Stage 1 看到 `min_tokens=0`、后者仍看到 `50`。删除旧 overlay 是部署文件名迁移，外部显式引用必须同步更新。PR #6767 的 L20X concurrency=4 观察只证明该精确 profile/workload 的 first-packet admission，不构成通用吞吐、延迟或硬件保证。^[PR #6458] ^[PR #6619] ^[PR #6767] +- 验收：展开四份 base profile 及三个 replica overlay，断言全部 duplex-enabled、窗口与各 profile 的 session/stage capacity 正确；同一 server 分别验证 `/v1/realtime?duplex=1` 和 chat，前者 Stage 1 看到 `min_tokens=0`、后者仍看到 `50`。删除旧 overlay 是部署文件名迁移，外部显式引用必须同步更新。PR #6767 的 L20X concurrency=4 观察只证明该精确 profile/workload 的 first-packet admission，不构成通用吞吐、延迟或硬件保证。^[PR #6458] ^[PR #6619] ^[PR #6767] - admission/expiry E2E probe 必须从实际启动的 deploy YAML 经 `load_deploy_config(...).duplex_session.max_sessions` 取得 admission limit，不能保留 diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-encoder-graphs.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-encoder-graphs.md new file mode 100644 index 000000000..d401e3650 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-encoder-graphs.md @@ -0,0 +1,49 @@ +--- +title: "MiniCPM-o 4.5 输入 encoder CUDA graph 合同" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, model-executor] +sources: ["PR #8332", vllm_omni/model_executor/models/minicpmo_4_5/encoder_cuda_graph.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_llm.py] +confidence: high +--- + +# MiniCPM-o 4.5 输入 encoder CUDA graph 合同 + +## MCPMO-ENCODER-1a — exact-shape 输入图必须保留 packed batch、mask 与输出所有权 + +- 触发：修改 MiniCPM-o SigLIP 或 stateless Whisper/APM encoder 的 CUDA graph adapter。 +- 强制：通过 upstream `SupportsEncoderCudaGraph` / `EncoderCudaGraphManager` 执行 + 已完成 packing 的完整本地 batch，把它作为一个不可拆的 manager item；只 copy + runner configuration，不改原配置或再做 manager-level DP。key 包含 CUDA stream + 与所有输入的 shape/dtype/device；每次 replay 更新静态输入与 mask，返回输出 clone。 +- 强制：vision 只捕获 transformer stack；position/mask 的 host decisions 留在外面。 + audio 只捕获 stateless encoder、projection 与 pooling。CPU、training、grad/autocast、 + nested capture、padded FlashAttention vision、FP16 Whisper overflow guard、请求 + intermediate audio layer 与 stateful streaming audio 保持 eager；`enforce_eager` 优先禁用。 +- 禁止：用 padding 扩大 exact-shape 接纳范围；把下一次 replay 可覆盖的输出 buffer + 直接保留为历史 embedding；将 Code2Wav continuation cache 当作 stateless input。 +- 验收：变更同 shape 的内容/mask并保留前次 embedding，检查当前结果与历史值；比较 + vision/audio eager parity，覆盖各 eager 资格与多 stream buffer 隔离。CUDA 图的实际 + capture/replay 由 CUDA 测试验证，CPU eager 测试只证明自身分支。^[PR #8332] + +## MCPMO-ENCODER-1b — 输入图接纳预算与 capture 失败必须分别处理 + +- 触发：修改 input encoder graph 的 admission、pool sharing、显存门限或失败恢复。 +- 强制：默认同 shape/stream 第二次调用才 capture，每 encoder 最多 4 图，admission + history 也有界；`max_graphs=0` 禁用，不 eviction/自动重捕。默认 1 GiB free-memory + floor 只阻止新 capture，低于 floor 时已存在图继续 replay。门限不是显存预留或 fit 保证。 +- 强制：同 encoder/device/replay stream 的 graphs 可共享 pool/capture stream； + 新 capture 等待旧 replay 及输出 clone 完成，随后 replay stream 等待 capture stream。 + 不同 encoder 与 replay stream 隔离 mutable buffers；共享 pool 上的 graphs 不得并发。 +- 强制：capture 异常原样传播并把该实例置为 terminal failed；后续调用拒绝继续 CUDA + eager 或 retry capture，需重启 worker，可显式禁用 encoder graphs 后恢复。 +- 禁止：把 warmup/capacity/memory miss 与可能污染 CUDA context 的 capture 异常合并 + 为普通 eager fallback；把共享 pool 当成允许并行 replay 的保证。 +- 验收：一次性 shape 不耗尽 capture slots；容量满、配置零与显存 floor 各自计数； + 低显存仍重放旧图；注入 capture failure 后第二次调用不再 capture/eager；晚 capture + 后任意顺序重放仍保留前次输出与 stream/encoder 隔离。性能证据需绑定最终 admission + policy、exact head 与 workload，不能直接沿用较早 head 的数值。^[PR #8332] + +Code2Wav 的 resident attention 与 stateful continuation 图见 +[Whole-Euler 合同](rules-resident-graphs.md)。 diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-mrv2-state.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-mrv2-state.md new file mode 100644 index 000000000..bf40e51b7 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-mrv2-state.md @@ -0,0 +1,80 @@ +--- +title: "MiniCPM-o 4.5 MRv2 turn 与输出状态合同" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, model-executor] +sources: ["PR #8222", vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni.py, vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_tts.py, vllm_omni/worker_v2/omni_sampler.py] +confidence: high +--- + +# MiniCPM-o 4.5 MRv2 turn 与输出状态合同 + +## MCPMO-MRV2-1a — MRv2 turn Thinker 的 row ledger 必须来自 replay 后的 live batch + +- 触发：MiniCPM MRv2 Thinker、FULL graph output、stage handoff 或 runner profile selection。 +- 强制：MRv2 Thinker 仅支持 turn，`async_chunk=false` 保留完整 llm2tts payload；native + duplex Thinker/Talker 使用 V1 session runtime，显式拒绝 MRv2 duplex。MRv2 Thinker 保留 + native multimodal encoder/cache，不声明会绕过它的 custom preprocess；仍保留真实 token IDs。 +- 强制：FULL graph forward 只返回 tensor；在 replay 后的 output adapter 从当前 + `input_batch.input_ids/positions` 重建 latent row ledger，不能复用 warmup 的 Python + metadata。profile 选择仍显式：generic turn MRv2 的 Thinker/Talker/codec 容量16/8/8， + H200 turn profile16/16/8且 Talker KV4GiB；V1 default 不自动迁移成 V2。 +- 禁止：把 MRv2 turn 的功能 probe 外推为 duplex MRv2、interruption 或 speech-quality + 验收；将已移除的 experimental Flow 路径的历史性能数字作为当前 mainline codec 收益。 +- 验收：两个不同 live batch 重放同图，核对 latent token/position identities；验证完整 + handoff、native multimodal embeddings、profile capacities、duplex refusal 与 V1 selection。 + 真实 turn、V1 duplex 和质量/性能分别绑定各自 exact head 与执行路径。^[PR #8222] + +## MCPMO-MRV2-1b — Talker device sampling 必须保留 codec window、EOS 与请求 slot + +- 触发：MiniCPM MRv2 sampler、codec history、EOS、length cap 或 `codes.audio` 输出。 +- 强制：经 `ModelState.custom_sampler` 注册 `OmniSampler` adapter，委托原 sampler 的 + request state 与 staged writes，不修改 sampler 的 `__class__` 或另起 sampling hook。 + 16-code frequency window 只取当前 request slot 在 prompt 之后的有效输出；保留 upstream + min_tokens、temperature/top-k/top-p、seeded sampling 与频率/存在惩罚。 +- 强制：`codes.audio` 对应本步 decode 的输入 codec ID，而非本步刚采样的下一 ID； + `meta.codec_frame_valid` 排除 prefill、EOS 与 empty/finished rows，bridge 在 CPU snapshot + 后只取有效 rows。empty condition、已读 EOS 或有效 token limit 的 forced-EOS mask + 只消费一次，作用在 sampling 结果，sampled/rejected counts 保留 upstream 合同。 +- 禁止：用全 prompt repetition penalty 替换 codec window；request compaction 后读错 + slot history；输出 EOS/prefill 作为 audio code；在热路径逐行同步 sampled ID 到 host。 +- 验收：窗口频率、prompt exclusion、非默认 sampling controls、slot reordering、empty、 + EOS、length cap、validity mask 与 counts；标准 sampling 不被重复执行。component parity + 不等同完整波形或 speech-quality 等价。^[PR #8222] + +## MCPMO-MRV2-1c — 共享 async/batched hooks 必须同时保护 V1 请求与输出所有权 + +- 触发：MiniCPM `use_async_omni_output`、batched Talker preprocess、pinned codec upload + 或 model-owned CPU snapshot finalizer。 +- 强制：这些共享 hooks 也会影响默认 V1 turn/native duplex；V1 live one-token rows 先 + 批量 embedding，codec ID 的 pinned host copy 在 Talker forward 前排队，按 request ID + 与 row 保存；等对应 event 后才消费值。finished rows 保留 shape-correct zero/empty delta， + request release 清理 pending IDs 与 audio state。connector codec 输入上传也保留 pinned ownership。 +- 强制：graph output 在 copy stream 上获得 CPU-owned snapshot，D2H 完成后才执行 + multimodal finalizer。已经按 request 分区的 `RequestOutputSnapshot` 不重复 generic + partition，仍验证 inter-stage/client list 长度等于 batch；`OmniSamplingOutput` 的 + payload、hidden-state 与 ownership flags 区分 `None`（保留）和空 dict（替换）。 +- 禁止：只跑 MRv2 YAML 就宣布默认 V1 未改变；返回可被下次 replay 覆写的 output views； + 在 finalizer 重采样、错位分区或等本步 forward 时偷偷增加同步；保留已结束请求的 codec delta。 +- 验收：默认 V1 scalar/batched 路径比较 embedding、logits、sampled IDs、delta、finished + flags 和 request state；pinned async snapshot 与同步 CPU 数据比较并检查独立 storage。 + native V1 duplex 的 interruption/resume 与 whole-model音频质量另行验收。^[PR #8222] + +## MCPMO-MRV2-1d — cached ISTFT 必须是显式 shared API 的 shape 限定优化 + +- 触发：MiniCPM Code2Wav 启用 shared HiFT cached ISTFT 或修改 overlap-envelope cache。 +- 强制：消费 HiFT 的 `enable_cached_istft()` API，不从外部写 private flag；shared HiFT + 未 opt-in 时保持原 `torch.istft`。仅对三维 one-sided spectrum、一维 n_fft 长度 window + 使用等价的 center/unnormalized/no-explicit-length 算法；其余走 native path。 +- 强制：overlap envelope 按 frame count、n_fft/hop、window device/dtype/storage identity + 缓存；第一次 shape 仍检查 NOLA，后续同一不变 window 可复用。window 内容发生 mutation + 时必须让旧 envelope 失效，`data_ptr()` 本身不是内容未变的保证。 +- 禁止：把该有限 ISTFT 合同推广到其他 window/length/normalization；让模型 consumer + 暗改共享 vocoder 默认；将单个 CUDA component 的 bitwise parity 外推为端到端质量。 +- 验收：未 opt-in 的 shared native control、目标 shape 的 cached/native waveform parity、 + 首 shape NOLA 与复用后的 CUDA同步行为；TF32 正常/异常恢复另见 MCPMO-GRAPH-1d。 + ^[PR #8222] + +Code2Wav 的后续 Whole-Euler/slots 演进见 [resident graph 合同](rules-resident-graphs.md)； +输入 encoder 的独立 stateless graph 合同见 [encoder graphs](rules-encoder-graphs.md)。 diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-resident-graphs.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-resident-graphs.md new file mode 100644 index 000000000..7bacb0d6d --- /dev/null +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules-resident-graphs.md @@ -0,0 +1,93 @@ +--- +title: "MiniCPM-o 4.5 Whole-Euler 与 resident graph 合同" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, model-executor] +sources: ["PR #8007", "PR #8443", vllm_omni/model_executor/models/minicpmo_4_5/cuda_graph_wrapper.py, vllm_omni/model_executor/models/minicpmo_4_5/batched_token2wav.py] +confidence: high +--- + +# MiniCPM-o 4.5 Whole-Euler 与 resident graph 合同 + +## MCPMO-GRAPH-1a — Whole-Euler 必须保持 backend 选择与共享 arena 的生命周期 + +- 触发：修改 Code2Wav Whole-Euler capture/replay、micro-batch 分组或 attention arena。 +- 强制：CUDA Whole-Euler 捕获完整 Euler solve；显式 TRT stepper 存在时不构造或重放 + Whole-Euler，`enable_whole_euler=false` 保留 step-level graph 路径。NPU 使用自己的 + platform graph runner；新增 estimator 参数必须同步平台 patch 的签名。 +- 强制：所有 micro-batch 的 graph entry 在任何 replay 前一次性取得；后续 capture 若 + 退休早先 graph，必须重新取得整组，不能一部分已更新 request cache 后再整体 eager。 + arena 的 input/output attention views 共用 storage；扩大该 storage 前先同步并退休 + 引用它的自有 graphs。首个 capture 失败、尚无 cache entry 时也必须释放 arena。 +- 禁止：用 batch 大小改变显式 TRT 选择；留下多份 stacked attention history；把图的 + shape/count 测试当作 padding 后 recurrent cache 已满足数值 parity。fallback Euler + 热循环不得每步从 device scalar 取 Python `float`；timeline 的 host step sizes 在循环外准备。 +- 验收：同 bucket 的不同有效长度经过真实 masked attention 与 causal convolution 后， + 比较有效 mel、CNN/attention cache 并继续消费下一 chunk；检查 TRT 双开关、NPU + estimator 签名、arena 首捕失败清理、storage growth 与整组重取。新版 Whole-Euler + 总预算见 MCPMO-GRAPH-1b，不沿用旧版本满容量时不断整代重捕的策略。^[PR #8007] ^[PR #8443] + +## MCPMO-GRAPH-1b — resident slots 与 arena 必须共同服从 max_graphs + +- 触发：修改 Whole-Euler resident attention slots、fused DiT body、precapture 或 graph 上限。 +- 强制：slot pool 只在受支持的 fused body、FP32 attention cache 与 NVIDIA SM80+ + tiled-attention 路径启用；不满足条件时保留 arena 路径。slot 保留 request 自己的 + attention history，query 容量至少覆盖配置宽度与 200 帧，不能把不同请求的 cache 混为一条。 +- 强制：`len(_cache)+len(_slot_graphs)` 共同受 `max_graphs` 限制；arena、slot 的 lazy + capture 与 precapture 都使用同一剩余预算。未命中且预算耗尽时返回未接纳，让调用方 + 走有效 fallback；不得扩容预算或仅检查 arena cache。显式 `micro_batch_size` 和 + `max_graph_batch` 仍优先于 scheduler 推导的默认值。 +- 禁止：把固定 query/storage 预留或有限 graph 数量等同总显存上界；将 resident 路径的 + 性能外推到未启用 fused body/slots、其他精度、硬件或 workload。 +- 验收：分别用 arena 与 resident slot 填满预算，再请求第三种 shape；确认预算未增长， + precapture 不越界，未接纳路径可完成 eager solve；另核对显式 batch override 与 request + history 的逐行 parity。真实性能数据需绑定 exact head、配置、硬件和 workload。^[PR #8443] + +## MCPMO-GRAPH-1c — onset 与 continuation 合并必须保留逐行 cache offset + +- 触发：修改 `cfm_onset_merge`、`cfm_row_offset_merge` 或 ragged Code2Wav batch。 +- 强制：混合 onset/continuation 时，CFM 路径不 stack 长度不同的 Conformer cache； + `_stack_flow_cache(..., include_conformer=False)` 只跳过此路径不用的字段。每行按自身 + attention offset 选择 noise slice、有效 key mask 和输出 cache；CFG 的 cond/uncond + 行排列、CNN cache 与请求索引保持一致，HiFT 按可兼容的 cache shape 分组。 +- 强制：Whole-Euler 拒绝 mixed-offset shape 时按 offset 分组求解；同 offset 子组必须 + 最终到达 step/eager 路径，不能因 `row_offset_merge` 开关再次递归进入同一 fallback。 +- 禁止：为使 tensor 可 stack 而伪造相同 cache 长度；预算耗尽或不支持 shape 时无限递归； + 把保序 shape smoke 当成真实流式音频质量已验收。 +- 验收：混合新 onset 与已推进的 continuation；强制 graph replay 返回 `None`，验证 + 有限次 fallback 完成并保持原行顺序；对每行 cache、有效 mel 与后续 chunk 做独立 parity。 + ^[PR #8443] + +## MCPMO-GRAPH-1d — CFM ordinary TF32 必须与 HiFT 精度分开并保留旧配置键 + +- 触发：修改 MiniCPM-o Code2Wav TF32 开关、fused kernels 或 graph capture 精度。 +- 强制：`MINICPMO_CODE2WAV_TF32` 优先于 shipping 键 `token2wav_allow_tf32`，再兼容 + `code2wav_allow_tf32`；当前默认 ordinary `tf32`，显式 false/off 关闭。dense CFM + QKV/MLP 的 `allow_tf32` 与 tiled attention 的 `input_precision="tf32"` 属同一 + ordinary TF32 策略；CFM 调用恢复原全局 matmul policy，HiFT convolution 保持 IEEE FP32。 +- 禁止：把历史字符串 `tf32x3` / `3xtf32` 的兼容 alias 宣称为 compensated TF32x3； + 忽略旧 shipping YAML 键；在 capture 与 replay 间只改开关却宣称 graph 内核精度已变。 +- 验收：覆盖默认、环境优先级、两个 YAML 键、off 与旧 alias；检查 policy 恢复以及 + capture 所用精度下的 DiT/cache parity。性能与音频质量结论分别绑定实际 precision， + 不从普通 TF32 kernel 或单元测试推出端到端质量等价。^[PR #8443] + +## MCPMO-GRAPH-1e — Code2Wav exact encoder 与 HiFT 预捕必须覆盖真实 continuation shape + +- 触发：修改 `cfm_encoder_cuda_graph`、Code2Wav precapture 或 HiFT exact chunk buckets。 +- 强制：Code2Wav flow encoder graph 是默认关闭的 CUDA continuation 优化；仅支持 + 具备固定 position-encoding tables 的 CosyVoice2 upsample-Conformer，保留 eager + onset、final/flush、其他 row/token/cache shape。graph key 覆盖实际 row 数、token + width 与可达 cache 长度；无额外 padding。输出 views 会被下次 replay 改写，需保留时 clone。 +- 强制：权重加载后首个 `forward` 的 precapture 路径按配置预捕 HiFT，再用真实默认 + prompt 特征准备 Whole-Euler 与 flow-encoder shapes；HiFT 仍只捕获 pre-iSTFT， + iSTFT/finalize 保持原边界。不得把 initial/continuation width 默认视为已覆盖，检查 exact + bucket 配置与实际预捕 key。Whole-Euler 预捕受 MCPMO-GRAPH-1b 的总预算约束。 +- 禁止：在输入 encoder 的 stateless graphs 与 Code2Wav 的 stateful continuation graphs + 之间复用资格或 failure policy；把 fake-capture CPU 测试说成真实 CUDA capture 已通过。 +- 验收：枚举 streaming trim 后可达 cache shapes，逐 shape 比较 encoder 与 eager； + 对未支持的 layouts/widths 检查 eager，CUDA 测试另验证真实 capture 的 bit-exact replay； + HiFT 检查 initial、steady、final chunk 与 retained output 的边界。^[PR #8443] + +输入 SigLIP/Whisper graph 见 [encoder graphs](rules-encoder-graphs.md)；legacy step-level +CFM graph 与 padding 合同见 [CUDA graph 规则](rules-cuda-graphs.md)。 diff --git a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules.md b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules.md index 65134626d..8308eabbe 100644 --- a/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules.md +++ b/knowledge/repos/vllm-omni/models/minicpm-o-4-5/rules.md @@ -38,8 +38,8 @@ confidence: high | tokenizer/processor、`trust_remote_code` | MCPMO-1a | pipeline/config factory → `model_executor/models/minicpmo_4_5/` loader | | TTS extra、backend 初始化、空音频 | MCPMO-1b | `minicpmo_4_5_omni_tts.py::MiniCPMO45OmniTTSForConditionalGeneration`、`minicpmo_4_5_token2wav.py::MiniCPMO45Token2wav` | | Code2Wav TensorRT、DiT/Campplus、engine cache/profile | MCPMO-1c | `minicpmo_4_5_code2wav.py::MiniCPMO45Code2Wav` → `batched_token2wav.py::BatchedToken2Wav._estimator_step`；共享实现 `step_audio2/step_audio2_dit_trt.py` | -| HiFT CUDA Graph、chunk bucket、lazy capture、iSTFT | [`MCPMO-1d`](rules-cuda-graphs.md#mcpmo-1d-hift-graph-只捕获稳定-pre-istft-子图并限界-shape-cache) | `cuda_graph_wrapper.py::HiFTGraphWrapper` → `BatchedToken2Wav._hift_inference` → shared `HiFTGenerator` | -| CFM DiT CUDA Graph、shape cache、generation retirement、eager fallback | [`MCPMO-1e`](rules-cuda-graphs.md#mcpmo-1e-cfm-dit-cuda-graph-必须按-shape-整代退休并保持-eager-parity) | `cuda_graph_wrapper.py::CFMGraphWrapper` → `BatchedToken2Wav._estimator_step` | +| HiFT CUDA Graph、chunk bucket、lazy capture、iSTFT | [`MCPMO-1d`](rules-cuda-graphs.md#mcpmo-1e-legacy-step-level-cfm-dit-graph-必须按-shape-整代退休并保持-eager-parity) | `cuda_graph_wrapper.py::HiFTGraphWrapper` → `BatchedToken2Wav._hift_inference` → shared `HiFTGenerator` | +| CFM DiT CUDA Graph、shape cache、generation retirement、eager fallback | [`MCPMO-1e`](rules-cuda-graphs.md#mcpmo-1e-legacy-step-level-cfm-dit-graph-必须按-shape-整代退休并保持-eager-parity) | `cuda_graph_wrapper.py::CFMGraphWrapper` → `BatchedToken2Wav._estimator_step` | | batch、`runtime_info`、stage handoff | MCPMO-3a/3b | `stage_input_processors/minicpmo_4_5_omni.py` → `minicpmo_4_5_omni.py::MiniCPMO45OmniForConditionalGeneration` → TTS/code2wav | | Seed-TTS、chat `ref_audio`、runtime-ref、Code2Wav handoff | [MCPMO-3e](rules-code2wav-batching.md#mcpmo-3e-minicpm-o-45-reference-audio-必须经过私有-prompt-到-code2wav-handoff) | serving chat MediaConnector → original prompt private key → `llm2tts` buffer → Code2Wav | | Talker codec sampling、repetition penalty、request RNG/compaction | MCPMO-3c | `minicpmo_4_5_omni_tts.py::{make_omni_output,_sample_audio_codes,_apply_batched_repetition_penalty}` → `test_talker_batching.py` | @@ -195,8 +195,8 @@ confidence: high - 触发：MiniCPM-o native duplex 的连续音频、LISTEN/SPEAK handoff 或 server VAD interruption。 - 强制：模型说话时仍把音频单元追加到同一可恢复 Stage0，保留 KV/runtime state 和上一 terminator；仅确认的 empty/control-only handoff 可在没有 speech conditioning 时跳过，其他 - 缺 latent/hidden-state 的 speech handoff 仍报错。LISTEN 是成功且不需要 Talker hidden state， - SPEAK 才转交 Talker。默认模型拥有的模式固定为 `listen_only`，不会创建/调用 Silero；只有 + 缺 latent/hidden-state 的 speech handoff 仍报错。plain LISTEN 是成功且不需要 Talker hidden state；含 turn_eos 的 speech-closure + LISTEN 仍须转交 Talker，边界见 [turn closure](rules-duplex-turn-closure.md)。默认模型拥有的模式固定为 `listen_only`，不会创建/调用 Silero；只有 `turn_detection.type=server_vad` 且 `interrupt_response=true` 才选择 `barge_in_on_speech`， 并拒绝 `interrupt_response=false` 的第三模式。server VAD hard cancellation 必须显式 opt-in， 阈值不得落到使 silence branch 不可达的边界。 diff --git a/knowledge/repos/vllm-omni/models/minimax-h3/_index.md b/knowledge/repos/vllm-omni/models/minimax-h3/_index.md index be59e1239..2051a945f 100644 --- a/knowledge/repos/vllm-omni/models/minimax-h3/_index.md +++ b/knowledge/repos/vllm-omni/models/minimax-h3/_index.md @@ -1,7 +1,7 @@ --- title: "MiniMax H3" created: 2026-08-05 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, models, diffusion] sources: ["PR #5703", "PR #5706", "PR #5709", "PR #5720", "PR #5723", "PR #5737", "PR #5740", "PR #5752", "PR #5756", "PR #5764", "PR #5779", "PR #5785", "PR #5801", "PR #5824", "PR #5829", "PR #5837", "PR #5840", "PR #5850", "PR #5857", "PR #5881", "PR #5885", "PR #5891", "PR #5896", "PR #5914", "PR #5946", "PR #5972", "PR #5978", "PR #5991", "PR #5863", "PR #6476", "PR #6555", "PR #6556", .buildkite/cuda/test-nightly.yml, .buildkite/cuda/test-ready.yml, .buildkite/cuda/test-merge.yml, apps/ComfyUI-vLLM-Omni/comfyui_vllm_omni/, docs/design/architecture_overview.md, docs/models/supported_models.md, docs/user_guide/quantization/fp8.md, vllm_omni/config/omni_config.py, vllm_omni/diffusion/attention/backends/flash_attn.py, vllm_omni/diffusion/attention/backends/rainfusion_attn.py, vllm_omni/diffusion/attention/backends/trtllm_attn.py, vllm_omni/diffusion/cache/cachedit/backend.py, vllm_omni/diffusion/cache/teacache/, vllm_omni/diffusion/forward_context.py, vllm_omni/diffusion/layers/norm.py, vllm_omni/diffusion/layers/rope.py, vllm_omni/diffusion/model_metadata.py, vllm_omni/diffusion/models/minimax_h3/, vllm_omni/model_executor/models/minimax_h3/, vllm_omni/model_executor/stage_input_processors/minimax_h3.py, vllm_omni/diffusion/registry.py, vllm_omni/diffusion/sched/sigma_schedule.py, vllm_omni/diffusion/utils/hf_utils.py, vllm_omni/entrypoints/omni_base.py, vllm_omni/platforms/npu/platform.py, vllm_omni/platforms/rocm/platform.py, vllm_omni/quantization/int8_config.py, recipes/MiniMaxAI/MiniMax-H3.md, recipes/MiniMaxAI/MiniMax-H3-4090.md, recipes/MiniMaxAI/MiniMax-H3-5090.md, recipes/MiniMaxAI/MiniMax-H3-MUSA.md, recipes/MiniMaxAI/MiniMax-H3-NPU.md, recipes/MiniMaxAI/MiniMax-H3-Spark-GB10.md, recipes/MiniMaxAI/MiniMax-H3-RTX-PRO-5000.md, recipes/MiniMaxAI/MiniMax-H3-RTX-PRO-6000.md, tests/diffusion/attention/test_rainfusion_plan.py, tests/diffusion/attention/test_trtllm_attn.py, tests/diffusion/cache/test_cache_backends.py, tests/diffusion/cache/test_teacache_extractors.py, tests/diffusion/layers/test_norm.py, tests/diffusion/layers/test_rope_broadcast.py, tests/diffusion/models/minimax_h3/, tests/diffusion/quantization/test_int8_config.py, tests/e2e/accuracy/minimax_h3/, tests/e2e/features/comfyui/test_comfyui_integration.py, tests/e2e/online_serving/minimax_h3/, tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py, vllm_omni/entrypoints/openai/video_api_utils.py, "PR #6213", "PR #6742", "PR #6666", "PR #6824", "PR #6714"] @@ -154,10 +154,10 @@ execution，不能代替 retained accuracy gate、Buildkite H100 result 或其 ## 审查入口 H3 input matrix/media ingress，以及 text-encoder completeness、online FP8 的 component namespace、loader 顺序、joint quality 与 offload 边界见 -[media rules](rules-media.md)（含 `text_conditioning/v1` schema）与 [MiniMax H3 rules](rules.md#direct-代码快速入口) 新增核对：MMH3-2r、MMH3-2s、MMH3-2t。 +[media rules](rules-media.md)（含 `text_conditioning/v1` schema 与 MMH3-2u 的 reference 尺寸门禁）与 [MiniMax H3 rules](rules.md#direct-代码快速入口) 新增核对：MMH3-2r、MMH3-2s、MMH3-2t。 ；checkpoint transform、quantized loader 与 text-encoder fused-source 完整性正文见 [loading rules](rules-loading.md)；DLO、consumer/H100/ROCm -部署和硬件证据正文见 [deployment rules](rules-deployment.md)；conditioned VAE 确定性、modular task 选择与 request 级 Cache-DiT/TeaCache/sigma schedule/Turbo LoRA 生命周期、alias metadata 同构与 Turbo artifact 合同见 [缓存与任务生命周期规则](rules-cache-task.md)。 新增核对：MMH3-2r2。 +部署和硬件证据正文见 [deployment rules](rules-deployment.md)；conditioned VAE 确定性、modular task 选择与 request 级 Cache-DiT/TeaCache/sigma schedule/Turbo LoRA 生命周期、alias metadata 同构、Turbo artifact 与 MMH3-2v 的 sampler 状态隔离见 [缓存与任务生命周期规则](rules-cache-task.md)。 新增核对：MMH3-2r2。 H3 VAE decoder 的 model-local eager dispatch、remote-code eligibility、exactness guards、selective @@ -167,3 +167,5 @@ Qwen3-VL text encoder 的 cuDNN SDPA process-global-state restore、encoder-rank CPU-only validation limit 见 [encoder state rules](rules-encoder-state.md)。 - [rules-quality](rules-quality.md) 新增核对：MMH3-4a2。 + +- [latent upscaler、second-pass refinement 与容量门禁](rules-latent-refinement.md) diff --git a/knowledge/repos/vllm-omni/models/minimax-h3/rules-cache-task.md b/knowledge/repos/vllm-omni/models/minimax-h3/rules-cache-task.md index b69e459eb..85d6ee174 100644 --- a/knowledge/repos/vllm-omni/models/minimax-h3/rules-cache-task.md +++ b/knowledge/repos/vllm-omni/models/minimax-h3/rules-cache-task.md @@ -1,10 +1,10 @@ --- title: "MiniMax H3 缓存与任务生命周期规则" created: 2026-09-04 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, models, diffusion] -sources: ["PR #5703", "PR #5720", "PR #5810", "PR #5837", "PR #5840", "PR #5853", "PR #5991", "PR #6476", "PR #6550", "PR #6666", "PR #6714", "PR #6909", vllm_omni/diffusion/models/minimax_h3/batched_packing.py, vllm_omni/diffusion/models/minimax_h3/fasth3.py, vllm_omni/diffusion/models/minimax_h3/lora.py, vllm_omni/diffusion/models/minimax_h3/minimax_h3_transformer.py, vllm_omni/diffusion/models/minimax_h3/npu/lora.py, vllm_omni/diffusion/models/minimax_h3/pipeline_minimax_h3.py, vllm_omni/diffusion/models/minimax_h3/quality_policy.py, vllm_omni/diffusion/models/minimax_h3/vae.py, vllm_omni/diffusion/cache/cachedit/runtime.py, vllm_omni/diffusion/sched/sigma_schedule.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/entrypoints/openai/video_api_utils.py, tests/diffusion/models/minimax_h3/test_minimax_h3_contract.py, tests/diffusion/models/minimax_h3/test_minimax_h3_fasth3.py, tests/diffusion/models/minimax_h3/test_minimax_h3_lora.py, tests/diffusion/models/minimax_h3/test_minimax_h3_native_lora.py, tests/diffusion/models/minimax_h3/test_minimax_h3_parallel.py, tests/diffusion/models/minimax_h3/test_minimax_h3_step_execution.py, tests/entrypoints/openai_api/test_video_api_utils.py, "PR #7162", "PR #7062", "PR #7838"] +sources: ["PR #5703", "PR #5720", "PR #5810", "PR #5837", "PR #5840", "PR #5853", "PR #5991", "PR #6476", "PR #6550", "PR #6666", "PR #6714", "PR #6909", vllm_omni/diffusion/models/minimax_h3/batched_packing.py, vllm_omni/diffusion/models/minimax_h3/fasth3.py, vllm_omni/diffusion/models/minimax_h3/lora.py, vllm_omni/diffusion/models/minimax_h3/minimax_h3_transformer.py, vllm_omni/diffusion/models/minimax_h3/npu/lora.py, vllm_omni/diffusion/models/minimax_h3/pipeline_minimax_h3.py, vllm_omni/diffusion/models/minimax_h3/quality_policy.py, vllm_omni/diffusion/models/minimax_h3/vae.py, vllm_omni/diffusion/cache/cachedit/runtime.py, vllm_omni/diffusion/sched/sigma_schedule.py, vllm_omni/diffusion/worker/diffusion_worker.py, vllm_omni/entrypoints/openai/video_api_utils.py, tests/diffusion/models/minimax_h3/test_minimax_h3_contract.py, tests/diffusion/models/minimax_h3/test_minimax_h3_fasth3.py, tests/diffusion/models/minimax_h3/test_minimax_h3_lora.py, tests/diffusion/models/minimax_h3/test_minimax_h3_native_lora.py, tests/diffusion/models/minimax_h3/test_minimax_h3_parallel.py, tests/diffusion/models/minimax_h3/test_minimax_h3_step_execution.py, tests/entrypoints/openai_api/test_video_api_utils.py, "PR #7162", "PR #7062", "PR #7838", "PR #8378"] confidence: high --- @@ -238,3 +238,11 @@ Cache-DiT、TeaCache、distilled sigma schedule 与 Turbo LoRA 的生命周期 - 强制：默认 latent-tail continuation 只在 Ref2VA request execution（非 step）下启用；window/overlap 必须落在 `17n+5` 网格且 window∈[107,345]、overlapsteady、ramp-first与initial优先级、graphfallback均断言。省略steady的default差异须有control，不能凭统一parser假定所有default一致。 ^[PR #8124] ^[PR #8213] + + +## MOSS-CODEC-1b — fast first-chunk handoff 必须有界且只能推进一次 slot 状态 + +- 触发：修改first-chunk receive hook、专用decode线程/stream、delivery sink或cancel cleanup。 +- 强制：只claim形状匹配、route有效且slot尚未占用的首chunk，clone codes并在admission lock内排队。复用generic prepared first-audio sink；main/cancel按slot handoff event排序；wait timeout必须正有限(默认30s)。关闭/workererror拒绝新job并fail已接收route；sink失败仅重送已decode PCM，finalcodecPCM不带upstream marker。 +- 禁止：无界等engine线程；在partial replay/error/timeout后release或重新decode同slot；cancel后重建receive readiness；首chunk由fast/main两条路重复交付或被丢弃。 +- 验收：fast与main交错、sink拒绝、workererror、close/submit竞态、timeout、超过capacity的cancel/reuse和首次已decode标记覆盖；PCM/frame仅一次、slot不提前重用，所有waiter可结束。 ^[PR #8213] + + +## MOSS-REF-1c — 共享 reference worker 必须结束所有等待者并保持 result cardinality + +- 触发：修改shared reference encoder、worker graph resources、batch-window队列或Unix-socketclient生命周期。 +- 强制：reference graphs保持显式VLLM_OMNI_MOSS_REF_GRAPHS opt-in，encoding在MRv2外；每worker拥有自己的stream/graphresources。outstanding集合覆盖已取走的job，close使全部job收到error。encode_fn返回数量必须等于输入clips；不匹配整个batch失败，已由close完成的job不能被late结果覆写。 +- 禁止：zip短结果留下后续caller永久wait；关闭只清pending而遗漏in-flight；在不同worker间复用mutablegraphbuffer；把shared目录或新graphs默认为全族能力。 +- 验收：关闭pending/in-flight、startupfail、结果过少/过多、两个并发client独立连接、batching与single-flight、late完成均覆盖；无异常hang且每clip一份结果/错误，graphoff原路径保持。 ^[PR #8213] diff --git a/knowledge/repos/vllm-omni/models/moss-tts/rules-registered-reference.md b/knowledge/repos/vllm-omni/models/moss-tts/rules-registered-reference.md new file mode 100644 index 000000000..cd78982c6 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/moss-tts/rules-registered-reference.md @@ -0,0 +1,17 @@ +--- +title: "MOSS 注册 reference 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models] +sources: ["PR #7883"] +--- + +# MOSS 注册 reference 规则 + +## MOSS-REF-1d — 注册 voice 必须绑定不可变 generation 并保留 conditioning 身份 + +- 触发：修改注册 MOSS voice 的 reference binding、cold resolver 或 prefix-cache salt。 +- 强制：仅在服务端验证已上传 voice、generation>0 且没有 inline audio override 时，绑定 name/created_at/file_path/ref_text snapshot；blank ref_text 才填默认。cold resolver 在线程中校验路径仍在 uploaded directory、文件 generation 匹配，保留原 WAV PCM16→float32/channel-mixing 处理。缺失/删除 generation 失败，不能换用同名新上传。内部 registered key 不交给普通 URI resolver，解析后移除临时 primary key；salt 以已验证的 name/generation 替代主 ref_audio，继续包含第二 reference 与其余 conditioning。 +- 禁止：信任客户端给出的 registered tuple/path；热命中/冷 miss 产生不同 salt；复用旧 generation cache；改变 Nano 的既有 waveform 路径或为省序列化改变音频量化。 +- 验收：hot/cold、re-upload、delete、generation/file mismatch、inline override、ref_text 和第二 reference 变化分别覆盖；cold 音频与旧 resolver 一致，上传数据无需驻留 data URI，旧 MOSS 内容/任务 salt 合同仍保持。 ^[PR #7883] diff --git a/knowledge/repos/vllm-omni/models/moss-tts/rules.md b/knowledge/repos/vllm-omni/models/moss-tts/rules.md index cf6108e72..c4b609b47 100644 --- a/knowledge/repos/vllm-omni/models/moss-tts/rules.md +++ b/knowledge/repos/vllm-omni/models/moss-tts/rules.md @@ -1,10 +1,10 @@ --- title: "MOSS-TTS 规则" created: 2026-09-02 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, models, model-executor] -sources: ["PR #5635", "PR #6908", vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_codec.py, vllm_omni/model_executor/models/moss_tts/audio_tokenizer.py, "PR #6241", "vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_local.py", "vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_local_depth.py", "vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_talker.py", "PR #6543", "PR #4982", vllm_omni/entrypoints/openai/tts_adapters/moss_tts.py, vllm_omni/entrypoints/openai/serving_speech.py, vllm_omni/model_executor/models/moss_tts/reference_encoder.py, vllm_omni/model_executor/models/moss_tts_nano/modeling_moss_tts_nano.py, tests/entrypoints/openai_api/test_tts_adapter.py, tests/entrypoints/openai_api/test_serving_speech.py, tests/model_executor/models/moss_tts/test_reference_encoder.py, "PR #7885"] +sources: ["PR #5635", "PR #6908", vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_codec.py, vllm_omni/model_executor/models/moss_tts/audio_tokenizer.py, "PR #6241", "vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_local.py", "vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_local_depth.py", "vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_talker.py", "PR #6543", "PR #4982", vllm_omni/entrypoints/openai/tts_adapters/moss_tts.py, vllm_omni/entrypoints/openai/serving_speech.py, vllm_omni/model_executor/models/moss_tts/reference_encoder.py, vllm_omni/model_executor/models/moss_tts_nano/modeling_moss_tts_nano.py, tests/entrypoints/openai_api/test_tts_adapter.py, tests/entrypoints/openai_api/test_serving_speech.py, tests/model_executor/models/moss_tts/test_reference_encoder.py, "PR #7885", "PR #8463", "PR #8447"] confidence: high --- @@ -118,3 +118,19 @@ confidence: high - 强制：共享 resolve LRU 只存 owned、C-contiguous `float32` 数组，容量按 `nbytes` 记账；finalize 必须 `copy=True` 切断对更大 decode 缓冲的 view。list 接口仅在调用边界临时 `.tolist()`，不得保留第二份 list 缓存。server-side encoder 走 array resolver；tensor 构造仍 copy，隔离缓存数组。drainer 在 `finally` 中丢弃已完成的 `first`/`jobs` 引用后再等待下一项。 - 禁止：把 `list[float]` 作为长期缓存存储；让完成 batch 在等下一队列项时继续持有；调用方原地 mutate 缓存数组。 - 验收：覆盖数组所有权、list 兼容路径、cache hit 不保留第二 list，以及 completed-batch 生命周期释放。^[PR #7885] + + +## MOSS-REF-BACKEND-1a — reference attention 必须按平台保持 kernel keyword 与返回合同 + +- 触发：修改 MOSS reference encoder 的 local-window varlen attention 或 ROCm backend dispatch。 +- 强制：NVIDIA 使用 vLLM FlashAttention 并传 fa_version；ROCm 优先 AITER、仅 ImportError 时回退 upstream flash_attn，均不传 NVIDIA 专有 fa_version。AITER 且 grad enabled 且 Q/K/V 任一 requires_grad 时请求 return_lse=True；统一解包 tuple 的 attention tensor。 +- 禁止：在 ROCm 无条件导入 NVIDIA extension；把 AITER 的 LSE keyword 传给 upstream fallback；在 inference/graph path 额外分配 LSE，或改变 causal window 与 padded-query mask。 +- 验收：覆盖 NVIDIA、AITER grad/inference 和 upstream fallback 的调用参数/返回类型；两种 FA version 的窗口 SDPA parity、padded-row zeroing 与 valid causal prefix 不受未来 padding 污染都必须保持。^[PR #8463] + + +## MOSS-REF-TEST-1a — Unix socket 测试必须使用短路径并按 fixture 生命周期释放 + +- 触发：MOSS shared reference encoder 测试创建 ref-encoder.sock，或修改共享目录 fixture。 +- 强制：为 socket 使用独立短 TemporaryDirectory(prefix="moss-ref-")，与 pytest 嵌套 tmp_path_factory 路径脱钩；fixture 在 context 内 yield，使目录覆盖 client/host 使用期；保留跨 encoder single-flight 与相同 artifact/key/音频断言。 +- 禁止：把 socket 放到可能超过 AF_UNIX 限制的深层 pytest 目录；仅缩短路径就削弱共享/cleanup 断言，或把测试修复宣称为 production 行为变化。 +- 验收：在深层 pytest base dir 的 CPU/ROCm lane 跑 test_shared_reference_encoder.py 与 test_inline_reference_shared_across_encoders，确认无 AF_UNIX path too long 且原断言成立。^[PR #8447] diff --git a/knowledge/repos/vllm-omni/models/nemotron-voicechat/rules.md b/knowledge/repos/vllm-omni/models/nemotron-voicechat/rules.md index 3c807e2a6..587bbdde8 100644 --- a/knowledge/repos/vllm-omni/models/nemotron-voicechat/rules.md +++ b/knowledge/repos/vllm-omni/models/nemotron-voicechat/rules.md @@ -21,7 +21,7 @@ confidence: high ## NVC-1a — 原生 duplex 是 frame-locked、可恢复的三 stage 数据面 - 触发：修改 duplex deploy、native runtime extension、Stage-0 append，或 Thinker→Talker→Code2Wav handoff。 -- 强制：只接受每 80 ms 一帧的 1280-sample/16 kHz PCM 输入；Stage 0 跨 wake 保留可恢复 request，text/function side channel 只消费 scheduler 在本次 wake 冻结的 `new_token_ids`。Talker 的累计 code history 在送入有状态 Code2Wav 前只传未见 suffix；native `codec_streaming` 可在当前累计 timeline 耗尽时结束一个 segment，offline async-stream 则必须同时等 `upstream_finished`，包括零进度 wake。共享的下游 receiver requeue/cleanup 遵循 [SCHED-5g](../../components/scheduler/rules.md#sched-5g-resumable-async-chunk-终态清理必须以-live-queue-所有权为准)。 +- 强制：只接受每 80 ms 一帧的 1280-sample/16 kHz PCM 输入；Stage 0 跨 wake 保留可恢复 request，text/function side channel 只消费 scheduler 在本次 wake 冻结的 `new_token_ids`。Talker 的累计 code history 在送入有状态 Code2Wav 前只传未见 suffix；native `codec_streaming` 可在当前累计 timeline 耗尽时结束一个 segment，offline async-stream 则必须同时等 `upstream_finished`，包括零进度 wake。共享的下游 receiver requeue/cleanup 遵循 [SCHED-5g](../../components/scheduler/rules-release-queues.md#sched-5g-resumable-async-chunk-终态清理必须以-live-queue-所有权为准)。 - 禁止：把 scheduler segment finish 当作 codec stream lifetime finish；重放累计 code prefix 到 causal codec cache；从 `supports_core_resumable_request` 推出公开 reusable KV lease（该 adapter 明确 `supports_core_kv_lease=false`）。 - 验收：固定 PCM contract、append/runtime contract、scheduler resume 和 data-plane tests 均须通过；offline async-stream completion 也须保持既有终端语义。^[PR #6089] diff --git a/knowledge/repos/vllm-omni/models/personaplex/_index.md b/knowledge/repos/vllm-omni/models/personaplex/_index.md index 0f1b2e9c2..7eaf2daf1 100644 --- a/knowledge/repos/vllm-omni/models/personaplex/_index.md +++ b/knowledge/repos/vllm-omni/models/personaplex/_index.md @@ -1,7 +1,7 @@ --- title: "PersonaPlex" created: 2026-09-02 -updated: 2026-09-06 +updated: 2026-10-06 type: index tags: [vllm-omni, models, model-executor] sources: ["PR #4771", vllm_omni/model_executor/models/personaplex/, vllm_omni/model_executor/models/personaplex/duplex/, vllm_omni/deploy/personaplex.yaml] @@ -28,7 +28,7 @@ confidence: high | 遇到什么 | 查看哪里 | |---|---| | stage、frame、codec/session 状态和 serving 边界 | [architecture](architecture.md) | -| frame accounting、de-delay、lease、greedy/config 门禁 | [rules](rules.md) | +| frame accounting、de-delay、lease、greedy/config 门禁、Mimi codebook prefix与vectorized prefill | [rules](rules.md) | ## 什么时候查这里 diff --git a/knowledge/repos/vllm-omni/models/personaplex/rules.md b/knowledge/repos/vllm-omni/models/personaplex/rules.md index 76c161935..83e2f98c1 100644 --- a/knowledge/repos/vllm-omni/models/personaplex/rules.md +++ b/knowledge/repos/vllm-omni/models/personaplex/rules.md @@ -1,10 +1,10 @@ --- title: "PersonaPlex 规则" created: 2026-09-02 -updated: 2026-09-06 +updated: 2026-10-06 type: rule tags: [vllm-omni, models, model-executor] -sources: ["PR #4771", "PR #6318", docs/design/fullduplex-personaplex.md, vllm_omni/model_executor/models/personaplex/duplex/serving_adapter.py, vllm_omni/model_executor/stage_input_processors/personaplex.py, tests/e2e/features/fullduplex/, tests/entrypoints/openai_api/test_duplex_handler.py] +sources: ["PR #4771", "PR #6318", docs/design/fullduplex-personaplex.md, vllm_omni/model_executor/models/personaplex/duplex/serving_adapter.py, vllm_omni/model_executor/stage_input_processors/personaplex.py, tests/e2e/features/fullduplex/, tests/entrypoints/openai_api/test_duplex_handler.py, "PR #7399", "PR #7481"] confidence: high --- @@ -87,3 +87,29 @@ confidence: high speech 宣称成 `supports_barge_in=true`。 - 验收：public capability/config 不出现 dead sampling controls，非 greedy 请求明确拒绝或 归一化；capability 保持 `supports_barge_in=false`。^[PR #4771] + +## PPLEX-1d — Mimi encode 只计算被消费的 codebook prefix + +- 触发：修改 `PersonaPlexMimiCodec.encode_frame()` 的HF quantizer调用或返回codec layout。 +- 强制：向真实 `quantizer.encode` 传 `num_quantizers=CODEBOOKS`（当前8），只计算semantic+所需 + acoustic prefix；返回仍是contiguous long `[batch,CODEBOOKS]`，不得修改checkpoint加载、decoder + 的32-codebook capacity或streaming front-end state。前8的residual选择不依赖后24层。 +- 禁止：先算32再slice并声称避免suffix工作；重排prefix浮点运算；以synthetic quantizer-only + speedup推断full codec/LM、audio quality、GPU memory或并发session收益。 +- 验收：非零real HF quantizer fixture与旧full32路径逐token相等，spy断言prefix层执行且suffix + 零调用；batch1/2/9下跨ring-wrap、slot reset与full reset仍prefix相等，decoder容量不变。 + stateless quantizer之后重复断言所有front-end内部state不是prefix优化必要证据；GPU微测需 + 单独记录版本、warmup、交替order、source hash与synthetic范围。^[PR #7399] + +## PPLEX-1e — prefill vectorization 必须保留逐 token embedding 语义 + +- 触发：优化talker `_build_prefill_embed` 或duplex first-append的voice/persona prefill。 +- 强制：非负offset/span用一次 `[1,1+n_q,span]` stack和一次现有embedding调用；text超过末尾 + 用zero_text=3，agent行用有效silence prefix，user行用有效user_sine或silence fallback，过短 + code tensor按原分支处理。embedding仍按既有固定table-add顺序，不作沿span的浮点reduction。 +- 禁止：每token `.item()` 将device scalar读回Python；改padding、audio fallback、dtype/device/ + contiguous layout或输入tensor；重复first-append prefill。helper速度不能外推serving整体速度。 +- 验收：faithful旧loop和手算layout在CPU/GPU FP32/BF16/FP16下逐byte相等，覆盖nonzero offset、 + partial/at-end/past-end/empty text、strided/short/absent audio与chunk拼接；拒绝所有scalar read + 时仍只有一次embedding调用。Stage0真实embedding integration证明first append只一次prefill， + second append只含live frame；组合stack的E2E结果不能单独归因此PR。^[PR #7481] diff --git a/knowledge/repos/vllm-omni/models/qwen-omni/_index.md b/knowledge/repos/vllm-omni/models/qwen-omni/_index.md index dbbcdd311..26630ed72 100644 --- a/knowledge/repos/vllm-omni/models/qwen-omni/_index.md +++ b/knowledge/repos/vllm-omni/models/qwen-omni/_index.md @@ -1,7 +1,7 @@ --- title: "Qwen-Omni 家族（Qwen2.5-Omni / Qwen3-Omni / Qwen3-TTS）" created: 2026-07-16 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, models, qwen-omni] sources: ["PR #5073", "PR #5671", "PR #5687", "PR #5976", "PR #6284", "PR #4322", "PR #6886", "PR #7019", vllm_omni/model_executor/models/qwen2_5_omni/qwen2_5_omni.py, vllm_omni/model_executor/models/qwen2_5_omni/qwen2_5_omni_thinker.py, vllm_omni/model_executor/models/qwen3_omni/quantization.py, vllm_omni/model_executor/models/qwen3_omni/qwen3_omni_moe_thinker.py, vllm_omni/model_executor/models/registry.py, vllm_omni/model_executor/models/qwen2_5_omni/pipeline.py, vllm_omni/model_executor/models/qwen3_omni/pipeline.py, vllm_omni/model_executor/models/qwen3_omni/qwen3_omni.py, vllm_omni/config/pipeline_registry.py, vllm_omni/deploy/qwen3_omni_moe.yaml, vllm_omni/deploy/qwen3_omni_moe_thinking.yaml, vllm_omni/engine/stage_init_utils.py, vllm_omni/quantization/component_config.py, tests/engine/test_stage_engine_args.py, docs/design/qwen3_omni_tts_performance_optimization.md] @@ -60,5 +60,9 @@ sources: ["PR #5073", "PR #5671", "PR #5687", "PR #5976", "PR #6284", "PR #4322" | 遇到什么 | 查看哪里 | |---|---| | stage 拓扑、代际差异与官方性能优化结论 | [architecture](architecture.md) | +| Realtime audio history 与 Qwen token/ms 近似 truncate 对齐 | [Realtime 音频 history 规则](rules-realtime.md) | | Qwen3-Omni Thinker MRoPE、CUDA compilation custom-op boundary、AWQ/compressed-tensors 名称映射、MoE backend default/override、固定种子音频回归、audio-encoder head/TP divisibility、Qwen2.5 code2wav soft-fail/旧 speech helper，或 code predictor 的 RoPE theta/rope_parameters 读取 | [Qwen-Omni rules](rules.md) 新增核对：QOMNI-1h、QOMNI-1h2、QOMNI-1h3、QOMNI-1h4。 | + +| Thinker audio resampler 或 deepstack meta-device buffer | [Thinker 输入与 buffer 规则](rules-thinker-input.md) | +- [MRv2 sampled handoff、first audio 与 decoder opt-in](rules-mrv2-audio.md) diff --git a/knowledge/repos/vllm-omni/models/qwen-omni/rules-mrv2-audio.md b/knowledge/repos/vllm-omni/models/qwen-omni/rules-mrv2-audio.md new file mode 100644 index 000000000..c8cc55e7e --- /dev/null +++ b/knowledge/repos/vllm-omni/models/qwen-omni/rules-mrv2-audio.md @@ -0,0 +1,32 @@ +--- +title: "Qwen3-Omni MRv2 audio handoff 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, model-executor] +sources: ["PR #8223"] +confidence: high +--- + +# Qwen3-Omni MRv2 audio handoff 规则 + +## QOMNI-MRV2-1a — sampled embedding 和 codec frame 必须对应当前 request 的有效 sample + +- 触发：修改 Thinker→Talker early handoff、sampled embeddings、MTP eager frames 或 Talker→Code2Wav payload。 +- 强制：只有 staged Thinker 且无 speculative config 时发布 early sampled embedding；speculative 路径保留原 capture-stream handoff。完成 prefill 的 sample 可补齐 Talker prompt 的首 generated-token embedding；随后按 sampled-text-stream 状态发送每步新 sample。partial/resumed prefill 显式发布 empty sample marker，保留 mode 并跳过已经发过的 capture text；非终态 decode 缺 sample 应失败。MRv2 eager MTP 以 token-major codes 最后一 row 和显式 frame-valid 决定是否 append，该 row 必须 clone 拥有 storage；V1/deferred frame 仍保留 zero-placeholder/stop-token 分支。codec vocabulary 和 quantizer 数从模型 config 读取。 +- 禁止：用 speculative 变宽 sample 套用单 token early handoff；把 empty marker 当成丢失 output 或重新发送 prefill text；将 graph-owned batch output 的 view 留给下一步；把 EOS/special CB0 作为有效 codec frame。 +- 验收：覆盖首 prefill sample、全 batch partial prefill、preemption resume、缺 sample、EOS 和 V1/deferred 对照；batch reorder 后每 request codes/embeddings 保持对应，下一次 replay 不覆盖已提交 payload。^[PR #8223] + +## QOMNI-MRV2-1b — Talker first audio 只在一帧无上下文的显式配置下启用 + +- 触发：修改 `talker_first_audio_enabled`、first-frame decoder、chunk ramp 或 first-audio trimming。 +- 强制：要求 connector `talker_first_audio=true`、initial chunk 恰好一 frame、环境开关允许、CUDA MRv2 async-chunk、本地 TP=PP=1、local executor 且 prefix cache 关闭。Talker copy 与 Code2Wav 使用同 class/config/dtype/device；side-stream first-frame graph 每 bucket 独立 pool，输出必须 copy 为新 FP32 tensor。只在第一 codec chunk 携带 first-audio marker，并由最终音频路径裁掉 Code2Wav 对该 prefix 的重复样本；其他 chunks 继续按 ramp、steady size 和左上下文发送。 +- 禁止：以通用 capability guard 代替模型 opt-in；将多帧、有历史上下文、跨 rank 或 prefix replay 视为同一首帧路径；让两个 process 的可选 cuDNN autotune 推导出 bitwise PCM 保证；第一音频和 Code2Wav copy 重复输出。 +- 验收：逐项关闭 eligibility 后回到 regular codec path；核对 singleton/batched first-frame prefix 和 owned output，marker 只到 chunk 0；首帧音频加 trimmed 后续 chunks 与固定 decoder reference 核对时序/长度/容差。^[PR #8223] + +## QOMNI-MRV2-1c — predictor 与 length-grouped decoder 优化必须保留 opt-in 和逐 row trim + +- 触发：修改 residual-codebook predictor sampling、incremental/short-KV、Code2Wav length grouping、fused Snake 或 cuDNN warmup。 +- 强制：deploy 的 `subtalker_sampling_params` 写入 predictor 的 stored top-k/top-p/do_sample 设置；不可假设 per-call参数会覆盖 stored sampler。incremental KV 使用显式 connector opt-in 和平台编译能力；fp16 或不支持 Inductor 时保留 re-prefill，short-KV 还须显式请求、CUDA BF16。decoder grouping 根据实际 lengths、graph buckets 和 captured row counts 规划，恢复原 row 顺序；crop 仍以本 group decoded window 的真实 tail 与每 row 的 left context/有效 frames 计算，遵循 [mixed-length trim rule](rules.md)。fused Snake 仅在显式 CUDA opt-in 下替换；cuDNN benchmark 仅在 eager warmup/非 capture 生效并通过 context 恢复进程 settings。 +- 禁止：把 group padding 当作短 row 有效未来 context；取消平台/dtype fallback；在 capture 内做算法搜索；用 historical concurrent benchmark 或单个 kernel A/B 宣称 speech-quality parity、普遍吞吐或 latency 保证。 +- 验收：覆盖 stored sampling/greedy、fp16与平台 fallback、short/steady mixed lengths、bucket miss 与 row reorder；分别核对裁剪样本、settings在异常后恢复，以及固定 input/seed 下优化路径的真实数值与 replay证据。^[PR #8223] diff --git a/knowledge/repos/vllm-omni/models/qwen-omni/rules-realtime.md b/knowledge/repos/vllm-omni/models/qwen-omni/rules-realtime.md new file mode 100644 index 000000000..d94ad0abd --- /dev/null +++ b/knowledge/repos/vllm-omni/models/qwen-omni/rules-realtime.md @@ -0,0 +1,25 @@ +--- +title: "Qwen3-Omni Realtime 音频 history 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, qwen-omni] +sources: ["PR #7285"] +confidence: high +--- + +# Qwen3-Omni Realtime 音频 history 规则 + +## QOMNI-2a — Realtime truncate 的 Qwen token 前缀只能作为近似对齐 + +- 触发：修改 Qwen3-Omni audio history、token IDs 或 `audio_end_ms`→text prefix。 +- 强制：当前约 383 ms/token 的换算仅用于 Qwen path；token count 四舍五入并限制在 + 已生成 IDs 的区间，缺 token IDs 返回空前缀。只有音频长度与 content_index 合法时才 + 改 history；共享会话存储、deferred event 与 terminal response 归 serving owner。 +- 禁止：把固定 token/ms 当作逐 token 强制对齐或精确词级时间戳；将 Qwen ratio 放进 + generic session，使其他模型或 native duplex 继承；把播放确认扩展当作 OpenAI 标准必需输入。 +- 验收：检查 0 ms、半 token rounding、超出 IDs 的 clamp、空 IDs 与合法 truncate； + 精确字幕/词级对齐需要独立真实音频证据，不能由这套近似推导。^[PR #7285] + +完整 route、PCM 与 history 生命周期见 [OpenAI Realtime 规则](../../components/serving/rules-realtime-openai.md)； +Thinker/Talker 与 stage transfer 见 [Qwen-Omni 规则](rules.md)。 diff --git a/knowledge/repos/vllm-omni/models/qwen-omni/rules-thinker-input.md b/knowledge/repos/vllm-omni/models/qwen-omni/rules-thinker-input.md new file mode 100644 index 000000000..68df4e1cc --- /dev/null +++ b/knowledge/repos/vllm-omni/models/qwen-omni/rules-thinker-input.md @@ -0,0 +1,26 @@ +--- +title: "Qwen3-Omni Thinker 输入与 buffer 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, qwen-omni] +sources: ["PR #8306", "PR #8461"] +--- + +# Qwen3-Omni Thinker 输入与 buffer 规则 + + +## QOMNI-INPUT-1a — audio parser 必须显式选中模型验证过的 resampler + +- 触发：修改 Qwen3-Omni Thinker MultiModalDataParser 的 audio resampling。 +- 强制：模型 parser 显式设置 audio_resample_method="pyav"，避免 vLLM 默认 backend 变化改写长音频路径；采样率、长度与其他 parser 参数仍按原模型配置。 +- 禁止：为修复一个 audio-input regression 改全局 resampler、image/video pipeline 或 CI baseline；只依据 import 成功宣称质量/TTFP/RTF 改善。 +- 验收：断言 parser 实际选择 PyAV，使用同长音频、原采样率与重采样目标复测 audio-input mean_e2el；短音频与非 audio control 保持。PR 本身只加 parser 参数，没有新增 regression test。^[PR #8306] + + +## QOMNI-INPUT-1b — deepstack buffer 不能继承 ambient meta device + +- 触发：修改 Qwen3-Omni Thinker deepstack_input_embeds、禁用 vision tower 的构造或 buffer resize。 +- 强制：初始 torch.zeros 明确使用 vllm_config.device_config.device，保持期望 dtype 与 [levels,max_tokens,hidden] shape；后续 resize 沿已物化 buffer device，不受 surrounding no-init-weights/meta context 影响。 +- 禁止：在 vision limit=0 时把未被权重 loader 替换的辅助 buffer 留在 meta；依赖环境默认 device，或只测试 vision-enabled 普通构造。 +- 验收：真实 meta initialization context 下覆盖 vision limit0/1与多个 token capacity；断言非meta、目标device/dtype/shape和零值，并在 resize后再次核对。^[PR #8461] diff --git a/knowledge/repos/vllm-omni/models/qwen3-tts/_index.md b/knowledge/repos/vllm-omni/models/qwen3-tts/_index.md index ad1fe6f70..5df407adb 100644 --- a/knowledge/repos/vllm-omni/models/qwen3-tts/_index.md +++ b/knowledge/repos/vllm-omni/models/qwen3-tts/_index.md @@ -35,3 +35,5 @@ confidence: high built-in `supported_speakers` 不属于 stored voice。 - 其他源码入口与具体不变量见 Qwen3-TTS rules；模型家族结构见 [Qwen-Omni](../qwen-omni/_index.md)。 + +- [当前step prompt projection、requestID cache 与speaker embedding](rules-prompt-preprocess.md)。 diff --git a/knowledge/repos/vllm-omni/models/qwen3-tts/rules-prompt-preprocess.md b/knowledge/repos/vllm-omni/models/qwen3-tts/rules-prompt-preprocess.md new file mode 100644 index 000000000..cf46247d9 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/qwen3-tts/rules-prompt-preprocess.md @@ -0,0 +1,17 @@ +--- +title: "Qwen3-TTS prompt batching 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models] +sources: ["PR #8477"] +--- + +# Qwen3-TTS prompt batching 规则 + +## Q3TTS-PROMPT-1a — Prompt projection batching 必须只消费当前请求的切片 + +- 触发：修改preprocess_infos_batch、_project_texts_batch、speakerembeddingcache或build_prompt_embeds。 +- 强制：仅initialprefill的nonstreamingCustomVoice/VoiceDesign批处理；保留Base/显式streaming和unsupportedinputserialfallback。texttemplate3leading/5trailing剥离后拼所有text一次projection，同一pinned H2D准备full/textids；按requestID保存当前step切片，下一step重置并消费时pop。speakerID/embedcache按device+speaker，未知speaker仍拒绝；保留EOS/pad/tail和语言条件。 +- 禁止：用row/slot代requestID或重用上一步同名请求embedding；把Base默认nonstreaming改成true；为batching改变token/mode/模板；把上游历史H200numbers说成本次验证或忽略NUMA/cold/filteredround边界。 +- 验收：CustomVoice/VoiceDesign serialvsbatched、不同textlength、reusedID、unconsumedcache、streaming/Base/invalidinput、device/speaker和EOS/tail均覆盖；projection与完整prompt逐元素对照，再独立检验seededPCM和servingcancel/recovery。 ^[PR #8477] diff --git a/knowledge/repos/vllm-omni/models/seedvr2/_index.md b/knowledge/repos/vllm-omni/models/seedvr2/_index.md new file mode 100644 index 000000000..0e6266dde --- /dev/null +++ b/knowledge/repos/vllm-omni/models/seedvr2/_index.md @@ -0,0 +1,21 @@ +--- +title: "SeedVR2" +created: 2026-10-06 +updated: 2026-10-06 +type: index +tags: [vllm-omni, models, diffusion] +sources: ["PR #8102"] +confidence: high +--- + +# SeedVR2 + +本 owner 对应 `diffusion/models/seedvr2/` 的 restoration pipeline、admission 与 +model-specific long-video route hook。 + +| 遇到什么 | 查看哪里 | +|---|---| +| restoration clip budget、长视频动态窗口、时序拼接、作业取消与文件生命周期 | [SeedVR2 规则](rules.md) | + +共享模型加载与并行执行查 [Diffusion owner](../../components/diffusion/_index.md)； +公共 HTTP 装配查 [Serving owner](../../components/serving/_index.md)。 diff --git a/knowledge/repos/vllm-omni/models/seedvr2/rules.md b/knowledge/repos/vllm-omni/models/seedvr2/rules.md new file mode 100644 index 000000000..c2fc102ad --- /dev/null +++ b/knowledge/repos/vllm-omni/models/seedvr2/rules.md @@ -0,0 +1,98 @@ +--- +title: "SeedVR2 restoration 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, diffusion] +sources: ["PR #8102"] +confidence: high +--- + +# SeedVR2 restoration 规则 + +## Direct 代码快速入口 + +| PR 描述信号 | 规则组 | 第一批源码 | +|---|---|---| +| OOM、clip pixels、VAE tiling、Ulysses | SEED-1a | `diffusion/models/seedvr2/config.py`、`pipeline_seedvr2.py::validate_clip_size` | +| 长视频、overlap、loop、PTS | SEED-1b | `long_video.py::_window`、`_segment`、`_run` | +| CRF、lossless、loopback、color | SEED-1c / SEED-1f | `long_video.py::_restore`、`_encode` → `pipeline_seedvr2.py` 的 correction helpers | +| job restart、cancel、upload、TTL | SEED-1d / SEED-1e | `long_video.py::register_routes`、`_status`、`_cleanup` | + +## SEED-1a — clip admission 必须按实际 padded 帧数和显存 profile 计算 + +- 触发：修改 SeedVR2 frame/pixel budget、VAE sharding 或环境变量 override。 +- 强制：先把输入帧数补到 `4n+1`，用 padded 数检查 max frames、单帧 pixels 与总 clip + pixels。sharded profile 同时要求 VAE tiling、height sharding、Ulysses degree≥4；自动 + budget 使用可见设备中最小显存并限制在校准边界内，无法读显存时取保守校准下界。 + 显式正整数 `VLLM_OMNI_*` override 优先，承认其可超出经校准的 OOM 保护。 +- 禁止：按未 padded 数放行；仅 degree 达标就启用放宽 budget；忽略异构设备中的最小 + 显存；把某台卡的 clip calibration 当成所有 degree/硬件都不会 OOM。 +- 验收：覆盖 padded 边界、各个 sharded 条件缺失、degree 4 与更高 degree、显存未知/ + 异构、合法 override 与非法非正值；实际 GPU memory/质量另用目标 profile 核验。^[PR #8102] + +## SEED-1b — 长视频窗口必须服从 admission，拼接必须保持目标帧数与连续时间轴 + +- 触发：修改 long-video window size、source loop、overlap blending 或 encoder PTS。 +- 强制：窗口取 clip-pixel/max-frame/long-window 上限内最大的 `4n+1`，且大于四帧 overlap； + 不能按历史 PR 描述固定为 12 帧。窗口 stride 是实际 batch 长度减四，overlap 用 + `(offset+1)/5` 渐变；source 只在显式 loop 时回绕。输出由一个 encoder 连续写入， + 从 0 按 24 fps 编号，并断言最终写出目标帧数；每个 restored window 校验尺寸、fps、 + 帧数与连续 PTS，最多两个待处理 future 保持内存有界。 +- 禁止：按固定窗口 stride 跳/重复 source 帧；重置每段输出时间戳；以每段解码成功 + 替代整片长度、边界 blend 和 source wrap 的验收。 +- 验收：独立 oracle 检查多窗口、短尾段、显式 loop 与不 loop、动态预算变化及四帧 + overlap 的值/索引；输出恰好 N 帧且 PTS 单调连续。7200 帧上限不等于该长度的 + GPU E2E 已完成，长片真实验收另记录目标硬件与结果。^[PR #8102] + +## SEED-1c — 内部 restoration hop 的 lossless 与最终视频压缩必须分别核验 + +- 触发：修改 long-video segment encode、内部 `/v1/videos/sync` 调用或最终 mux。 +- 强制：当前 worker 经 loopback HTTP 调用同一 sync route，逐窗保持 lossless codec + 选项、one-step/CFG 1 和 color 参数；shared entrypoint 只通过模型的 `register_routes` + hook 注册 model-specific 逻辑。source 含音频时保留音轨并转 AAC，音频是否回绕与显式 + `loop_input` 策略一致；无音轨时不补静音。 +- 禁止：把已改为 qp=0 的中间 codec 说成 in-process 执行；因为中间 lossless 就保证 + 最终 CRF18/yuv420p MP4 像素完全不变；把 loopback 的可达性推广成 TLS/UDS 部署支持。 +- 验收：检查每窗调用参数和 restored 输出合同，再独立检查最终 codec、音频 loop、 + 时长和色度变化；若改变 transport，覆盖真实 server binding/auth lifecycle。^[PR #8102] + +## SEED-1d — 长视频作业必须有单一 owner，并按窗口边界完成取消 + +- 触发：修改 long-video submission、worker restart、GET/DELETE 或并发作业控制。 +- 强制：提交前要求空 prompt、32-bit unsigned seed、1–7200 个目标帧与至少 16 且可被 + 16 整除的尺寸，并验证动态窗口大于 overlap。只接受一个 API server process，且同时 + 一个 active job；status 原子写入包含 + owner PID。GET 对 queued/running 但 PID 不属于当前 owner 的记录报告 failed。 + DELETE 创建取消标记并先返回 cancelling，worker 在下一窗口边界观察后停止。 +- 禁止：重启后让旧 running job 永久显示仍执行；把取消 acknowledgement 当作当前 + GPU 工作已立即停止；取消时释放 owner 让第二个 job 进入尚未排空的资源。 +- 验收：覆盖第二次 submission、多个 API server、stale PID、每个 window boundary + 前后取消与结束状态；必须区分 acknowledgement 与实际 worker 停止。^[PR #8102] + +## SEED-1e — 上传、job 路径与文件回收必须各自有明确边界 + +- 触发：修改 long-video upload、job ID、result/status 文件、TTL cleanup 或错误响应。 +- 强制：写盘时按 chunk 检查 8 GiB upload cap，空上传拒绝，失败上传清理已建目录； + 在任何 path 操作前验证 32 位 lowercase hex job ID。status 与最终 MP4 经临时文件 + rename 发布，完整 MP4 发布后才写 result marker，未有 result 不开放 download；不能 + 把当前直接写出的 result JSON 误称原子提交。TTL sweep 由新 submission 触发，只清理到期 settled/stale-owner + 目录，不清理 active owner；内部异常只在日志保留细节，客户端接收安全分类错误。 +- 禁止：用事后 size check 代替写盘限额；未验 job ID 就拼路径；把 submission-triggered + TTL 写成后台定时回收；将原始 exception/auth/path 暴露到客户端。 +- 验收：覆盖超限/空/中断上传、非法 ID、半成品结果、到期与 active 目录、内部错误 + 分类；没有新 submission 时文件保留的行为必须符合实际 lifecycle。^[PR #8102] + +## SEED-1f — color correction 参数必须从 admission 穿过每个窗口 + +- 触发：修改长视频 extra_params、color method 或 restoration 后处理。 +- 强制：extra_params 是 JSON object；color method 只接受 lab/wavelet/adain/none，null + 使用默认 lab；admission 后对每个 window 保持同一选择。处理 frames 使用有界批次， + RGB 值的 correction 与最终视频的 chroma subsampling 分别评价。 +- 禁止：worker 默默换回默认方法；允许任意 method 在后台才失败；把颜色统计修正 + 宣称为逐像素不变或由压缩视频直接证明原始 RGB 值保真。 +- 验收：invalid method 在 submission 前拒绝，四种合法 method 分别核对 window forwarding； + source/target reference 与 bounded batch 输出另用独立数值 oracle 比较。^[PR #8102] + +公开 route 与 artifact readiness 见 [Serving 请求规则](../../components/serving/rules.md)； +分布式执行与 VAE ownership 见 [Diffusion 规则](../../components/diffusion/rules.md)。 diff --git a/knowledge/repos/vllm-omni/models/sensenova-u1/rules.md b/knowledge/repos/vllm-omni/models/sensenova-u1/rules.md index 0dc1ed0ad..069c083f4 100644 --- a/knowledge/repos/vllm-omni/models/sensenova-u1/rules.md +++ b/knowledge/repos/vllm-omni/models/sensenova-u1/rules.md @@ -14,6 +14,7 @@ confidence: high | 信号 | 规则 | 第一批源码 | |---|---|---| +| component online FP8与distill LoRA startup guard | SENSENOVA-2a | pipeline → language_model quant config | | U1.5 distilled 8-step LoRA、kohya keys、one-way fusion | SENSENOVA-1a | `pipeline_sensenova_u1.py::load_lora_weights` → loader → parameter `weight_loader` | | `VLLM_OMNI_SENSENOVA_PAGED_DECODE`、think/text decode、capture/reuse/sleep | SENSENOVA-1b | `pipeline_sensenova_u1.py::_decode_context` → `paged_decode.py` | | GQA K/V、three-axis RoPE、single-token mask | SENSENOVA-1c | `sensenova_u1_transformer.py` → pipeline forward | @@ -38,3 +39,10 @@ confidence: high - 强制：native GQA 不预 expand K/V；每 forward 一次构建/共享 three-axis RoPE embeddings；single-token decode 不注入 all-zero mask。 - 禁止：把此模型 exception 推广给 shared attention，或用单测外推 quality/performance。 - 验收：GQA/reference、RoPE sharing 和 maskless decode 各有 model test；真实 output parity/accelerator coverage 独立取得。^[PR #6516] + +## SENSENOVA-2a — Online FP8 必须路由 language_model 并明确拒绝 distilled LoRA 冲突 + +- 触发：修改 SenseNova pipeline quantization config、language model构造、onlineFP8或distill LoRA启动。 +- 强制：resolve_component_quant_config使用精确language_model键，再把结果传SenseNovaU1ForCausalLM；两侧understanding/generationparallel linears接受它，vision/embedding/lm_head不因此量化。仅非checkpoint-serialized fp8且lora_backend=distill并有lora_path的组合在model构造前抛明确ValueError；BF16、无LoRA及已serializedFP8不能被此guard误拒。componentmatch/unmatched/default/explicitdisable各保持。 +- 禁止：用全局scalarconfig测试声称验证了component名；漏掉guard直到weight_loader裸assert；把降低显存说成所有硬件更快/qualitybitparity，或把A800/vLLM0.29的Cutlass禁用要求推广所有平台。 +- 验收：真实ComponentQuantizationConfig四种路由、三类guardcontrol和onlineFP8+distill冲突覆盖；按实际kernel/hardware分开测加载、text/image、质量和step/E2Elatency，不能只看启动成功。 ^[PR #7955] diff --git a/knowledge/repos/vllm-omni/models/voxcpm2/_index.md b/knowledge/repos/vllm-omni/models/voxcpm2/_index.md index 112e3da91..7820c8ace 100644 --- a/knowledge/repos/vllm-omni/models/voxcpm2/_index.md +++ b/knowledge/repos/vllm-omni/models/voxcpm2/_index.md @@ -1,7 +1,7 @@ --- title: "VoxCPM2（单 stage AR + 模型内扩散侧路,48 kHz）" created: 2026-07-21 -updated: 2026-09-22 +updated: 2026-10-06 type: index tags: [vllm-omni, models] sources: [vllm_omni/model_executor/models/voxcpm2/, vllm_omni/deploy/voxcpm2.yaml, vllm_omni/worker/gpu_ar_model_runner.py] @@ -66,4 +66,6 @@ sources: [vllm_omni/model_executor/models/voxcpm2/, vllm_omni/deploy/voxcpm2.yam 评审"统一改用 gpu_memory_utilization"类清理时本家族是刻意例外。 - 语义验收见 [model-validation](../../review/guides/model-validation.md)。 -- [rules](rules.md) 新增核对：VOXCPM2-2a。 +- [rules](rules.md)：VOXCPM2-2a 的请求噪声隔离与 VOXCPM2-3a 的 LocDiT 参数/solver 边界。 + +- [NPU estimator exact graphs 与延迟平台注册](rules-npu-graphs.md) diff --git a/knowledge/repos/vllm-omni/models/voxcpm2/rules-npu-graphs.md b/knowledge/repos/vllm-omni/models/voxcpm2/rules-npu-graphs.md new file mode 100644 index 000000000..ee59dd864 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/voxcpm2/rules-npu-graphs.md @@ -0,0 +1,18 @@ +--- +title: "VoxCPM2 NPU estimator graph 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, model-executor] +sources: ["PR #7275"] +confidence: high +--- + +# VoxCPM2 NPU estimator graph 规则 + +## VOXCPM2-NPU-1a — NPU graph 只包装已加载 LocDiT estimator 的 exact tensor 调用 + +- 触发：修改 NPU VoxCPM2 platform adapter、constructor patch、estimator forward 或 platform resolution。 +- 强制：模型构造/权重加载完成后，NPU adapter 幂等包装 `feat_decoder.estimator.forward(x,mu,t,cond,dt)`，使用 `NPUExactGraphRunner` 的 exact-input gate 和有限 graph budget（本提交为 8）；不支持必要 NPU graph APIs 时保留 eager。CFM solver、request-local噪声、feature encode 与 VAE decode 保持各自原路径，不能把 estimator graph 等同完整 decode graph。`graph_tools` 只在真正 NPU worker setup 延迟导入；访问 platform 时使用模块当前值，避免 inspection import 初始化 NPU或保存未解析 sentinel。CUDA unified decode graph 的 scheduler deferral 仍由 CUDA capability 选择。 +- 禁止：在模型 inspection 子进程提前初始化 NPU；重复 constructor patch 或重复包装 estimator；把 NPU fast path塞进硬件 YAML 或修改共享 runtime knobs 来假冒统一能力；把 CPU mock 或单次 Atlas A2观察当成真实 NPU parity/通用性能证明。 +- 验收：CPU contract 验证延迟 import、幂等 patch、正确五 tensor wiring、unsupported API eager 和 NPU 不触发 CUDA waiting deferral；真实 NPU 另核对 estimator eager/replay 数值、exact signature miss、graph budget 与多请求 seed/生命周期。^[PR #7275] diff --git a/knowledge/repos/vllm-omni/models/voxcpm2/rules.md b/knowledge/repos/vllm-omni/models/voxcpm2/rules.md index 6d27ada25..e40009498 100644 --- a/knowledge/repos/vllm-omni/models/voxcpm2/rules.md +++ b/knowledge/repos/vllm-omni/models/voxcpm2/rules.md @@ -1,10 +1,10 @@ --- title: "VoxCPM2 请求噪声规则" created: 2026-09-22 -updated: 2026-09-22 +updated: 2026-10-06 type: rule tags: [vllm-omni, models] -sources: ["PR #7866"] +sources: ["PR #7866", "PR #7010"] --- # VoxCPM2 请求噪声规则 @@ -14,6 +14,7 @@ sources: ["PR #7866"] | PR 描述信号 | 规则组 | 第一批 live 源码 | |---|---|---| | request seed、CFM noise、batched prefill/decode、graph replay | `request-noise`：`VOXCPM2-2a` | `worker/gpu_ar_model_runner.py` request seed → `model_executor/models/voxcpm2/` request generator 与各 noise fill 站点 | +| LocDiT timesteps、CFG 或 runtime overrides | `VOXCPM2-3a` | `model_executor/models/voxcpm2/runtime_config.py` coercion → talker 与 solver | ## VOXCPM2-2a — 请求 seed 必须驱动模型内 CFM/噪声流，且与 batch 位置无关 @@ -21,3 +22,10 @@ sources: ["PR #7866"] - 强制：`SamplingParams.seed` 到达 vLLM sampler 不等于到达模型内噪声点。有 seed 的请求必须在首次 prefill 从 runner 的 `_omni_seed` 构造请求私有 `Generator`，并在每个噪声 fill 站点（eager、cuda-graph、batched prefill/decode、unified replay）对 shape `(1, …)` 的行切片使用该 generator。同 text+seed 的噪声流必须与 batch 组成、request id 无关；无 seed 保持全局 RNG；`deterministic_cfm_noise` 的 request-id 哈希只服务 replay，且请求 seed 优先。 - 禁止：仅依赖 `SamplingParams.seed` 或全局 `normal_()` 宣称 speech seed 合同；用 request-id 哈希冒充用户 seed；让 seeded 行的噪声依赖同 batch 其他请求。 - 验收：跨 request 同 seed 字节一致、异 seed 敏感、batch 位次无关、seed 优先于 replay 哈希；覆盖所有噪声站点。^[PR #7866] + +## VOXCPM2-3a — LocDiT 设置必须统一校验并保留至少一次 estimator 调用 + +- 触发：修改 `voxcpm2_runtime_config`、LocDiT `inference_timesteps`/`cfg_value`/`cfg_cutoff_ratio`，或 eager、batched、graph solver 参数。 +- 强制：三字段只从 `_VoxCPM2RuntimeConfig` 消费，默认 `10`、`2.0`、`1.0`；`inference_timesteps` 钳制到至少 `2`，`cfg_value` 必须有限，否则初始化失败，`cfg_cutoff_ratio` 夹至 `[0,1]`。所有 solver 路径使用同一已解析值。 +- 禁止：让 CFG-Zero* 的一阶输入跳过全部 estimator 并返回原始噪声；让 NaN/inf CFG 流入 solver；在 talker 或 graph path 保留另一套硬编码默认。有限负 CFG 当前仍可解析，不能把未实现的非负门禁写成已有合同。 +- 验收：覆盖负/0/1/2/4/10 timesteps、非有限 CFG、cutoff 两端以及不同 batch size；断言 estimator 调用数、有限且非原始噪声的输出、输入 noise 未被修改，以及省略配置的默认保持。^[PR #7010] diff --git a/knowledge/repos/vllm-omni/models/yue2/_index.md b/knowledge/repos/vllm-omni/models/yue2/_index.md new file mode 100644 index 000000000..97b2573e0 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/yue2/_index.md @@ -0,0 +1,17 @@ +--- +title: "YuE2 模型规则入口" +created: 2026-10-06 +updated: 2026-10-06 +type: index +tags: [vllm-omni, models, model-executor] +sources: ["PR #8407"] +--- + +# YuE2 模型规则入口 + +YuE2 的 AR semantic frames、NAR/VAE synthesis queue 与请求完成边界属于本模型 owner；共享 runner 的输入 hook 只负责按模型 opt-in 转发 metadata。 + +- [YuE2 synthesis 与完成边界规则](rules.md) + +- [请求/native tokenizer 与数值边界](rules-interface.md) +- [SheetSage2 score 与 speech request handoff](rules-score-handoff.md) diff --git a/knowledge/repos/vllm-omni/models/yue2/rules-interface.md b/knowledge/repos/vllm-omni/models/yue2/rules-interface.md new file mode 100644 index 000000000..cad45c061 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/yue2/rules-interface.md @@ -0,0 +1,25 @@ +--- +title: "YuE2 请求与数值合同" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, serving, model-executor] +sources: ["PR #7886"] +confidence: high +--- + +# YuE2 请求与数值合同 + +## YUE2-INPUT-1a — speech endpoint 必须保留 text-to-music 的真实请求语义 + +- 触发：修改 `Yue2Adapter.validate/build/apply_sampling_overrides`、native tokenizer、semantic prompt 或 speech 参数透传。 +- 强制：`input` 是非空 lyrics，`instructions` 是非空音乐 style；`cot=off` 不接受 ABC，`melody/full` 必须带非空 `abc`，online v1 不自动生成 ABC。使用 checkpoint-native `qwen.tiktoken` 生成 token-id prompt，并在启动 warmup 解析 tokenizer。`max_new_tokens` 是 25 fps 的 audio-frame budget，须满足 preset 上下限和 prefix+frames 的 context bound；engine 的 `max_tokens` 多留一个 finishing step。完整 prefix ids 随 request extras 传入 NAR conditioning，不能从 prefix-cache hit 后的 uncached tail 重建。显式 seed 保留，缺 seed 时每个请求独立抽取；phase sampling preset 和 stop ids 由模型固定。 +- 禁止：接受 streaming、reference audio/text、language/task_type、非 default voice、非 1.0 speed 或用户覆盖 fixed sampling，却默默忽略；将 `max_tokens` 恰好卡在最后一帧；在 driver 路径使用普通 HF 文本 tokenizer 替代 native BPE。 +- 验收：覆盖缺 lyrics/style、ABC 三态、unsupported speech fields、frame/context 两层 budget、prefix-cache hit 和有/无 seed；最后一帧后仍能交付 finishing 音频，输出为 48 kHz stereo。^[PR #7886] + +## YUE2-NUMERIC-1a — MoT 权重、whole-song noise 和 FP32 VAE 必须保持各自边界 + +- 触发：修改 `partition_checkpoint_weights/load_weights`、`song_chunks`、NAR solve 或 tiled VAE decode。 +- 强制：单一 checkpoint 中 AR tensors 交给 backbone loader，NAR/projection tensors 按模型命名映射并检查 missing/unexpected，同时将手工加载 keys 纳入 loader 的 loaded set；deterministic latent position table 重建。按 request seed 在 CPU 一次生成全曲 FP32 noise，再沿 chunk boundaries 切片；每个 NAR chunk 在 `finally` 关闭 cached engine。VAE 在启动加载、纳入实际内存 profiling，decoder 保持 FP32 并关闭 autocast；terminal decode 使用 1024-frame core、16-frame halo，检查 finite 输出并保持 channels-first stereo。 +- 禁止：每个 chunk 重新 seed/抽 noise；将 VAE 注册进 AR checkpoint module tree而被全模型 dtype 转换或权重覆盖；把 AR BF16 policy 应用于 VAE；将 tiled decode 的边界正确性或一次模型观测推广为通用质量/性能保证。 +- 验收：验证 AR/side namespace、缺权重失败和 loaded-key accounting；相同 seed 的全曲 noise 与 chunk 拼接一致；FP32 full/tiled decode 核对边界、长度、finite 值和 stereo layout。异步 synthesis 的 queue/HOLD/完成确认另遵循 [synthesis rules](rules.md)。^[PR #7886] diff --git a/knowledge/repos/vllm-omni/models/yue2/rules-score-handoff.md b/knowledge/repos/vllm-omni/models/yue2/rules-score-handoff.md new file mode 100644 index 000000000..4ffc8623d --- /dev/null +++ b/knowledge/repos/vllm-omni/models/yue2/rules-score-handoff.md @@ -0,0 +1,18 @@ +--- +title: "YuE2 SheetSage2 score handoff 规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, serving] +sources: ["PR #8371"] +confidence: high +--- + +# YuE2 SheetSage2 score handoff 规则 + +## YUE2-SCORE-1a — transcription 成功与可发送 YuE request 必须是两个显式门禁 + +- 触发：修改 `tools/sheetsage2_transcribe.py` 的参数、remote-code/local-snapshot loading、ABC/MIDI 输出或 YuE speech request export。 +- 强制：加载权重前检查显式 `--trust-remote-code`、已有 input audio、finite positive duration、空/新 output directory、local model directory，以及 request export 成对的非空 lyrics/style；seed 仅用于请求导出并保持合法整数范围。默认远端 model/code 使用工具声明的固定 revisions，offline/local snapshot 保留相对 Python imports 和 `local_files_only`；模型与 MERT 权重先按 FP32 合并，执行 dtype 由 transcription 路径选择。生成 YuE request 前须同时得到非空 ABC、无 ABC error 和 `MThd` MIDI bytes；完整 score 选择 `cot=full`，melody-only 选择 `cot=melody`，保留 lyrics、style、model 和可选 seed。 +- 禁止：未完成参数检查就加载大模型；local offline 模式补下载依赖或改写 upstream snapshot；在 ABC/MIDI 任一失败时仍产出可发送 request；将 transcription helper 当成 online YuE 自带音频转谱能力。 +- 验收：无效参数和不可信 remote code 在模型加载前失败；local snapshot 含 transitive relative imports 的 offline case 可运行；ABC/MIDI 失败返回非零且不写 request，成功 export 的 cot 与 score 模式、lyrics/style/seed 一致。^[PR #8371] diff --git a/knowledge/repos/vllm-omni/models/yue2/rules.md b/knowledge/repos/vllm-omni/models/yue2/rules.md new file mode 100644 index 000000000..73e214f57 --- /dev/null +++ b/knowledge/repos/vllm-omni/models/yue2/rules.md @@ -0,0 +1,34 @@ +--- +title: "YuE2 synthesis 与完成边界规则" +created: 2026-10-06 +updated: 2026-10-06 +type: rule +tags: [vllm-omni, models, model-executor] +sources: ["PR #8407"] +confidence: high +--- + +# YuE2 synthesis 与完成边界规则 + +## Direct 代码快速入口 + +| PR 描述信号 | 规则组 | 第一批源码 | +|---|---|---| +| HOLD、side-stream synthesis、cancel、preemption、完成音频重交付 | `YUE2-SYNTH-1a`、`YUE2-SYNTH-1b` | `vllm_omni/model_executor/models/yue2/yue2.py` 的 `_SynthesisJob`、`_SynthesisQueue`、`_ship_audio`；`tests/worker/test_yue2_request_state.py` | +| speech/native tokenizer、lyrics/style/ABC、frame budget | [请求合同](rules-interface.md)：`YUE2-INPUT-1a` | `vllm_omni/entrypoints/openai/tts_adapters/yue2.py::Yue2Adapter`；`vllm_omni/tokenizers/yue2_prompt.py`、`vllm_omni/tokenizers/yue2_tokenizer.py` | +| MoT weight mapping、whole-song noise、NAR、FP32 tiled VAE | [数值合同](rules-interface.md)：`YUE2-NUMERIC-1a` | `vllm_omni/model_executor/models/yue2/yue2.py` 的 `partition_checkpoint_weights`、`song_chunks` 与 VAE decode | +| SheetSage2、ABC/MIDI、offline snapshot、speech request export | [Score handoff](rules-score-handoff.md)：`YUE2-SCORE-1a` | `tools/sheetsage2_transcribe.py`；`tests/tools/test_sheetsage2_transcribe.py` | + +## YUE2-SYNTH-1a — side-stream synthesis 的 HOLD、队列和 buffer lifetime 必须闭合 + +- 触发：修改 `_SynthesisJob`/`_SynthesisQueue`、HOLD budget、request admission 或 preemption history reconciliation。 +- 强制：通过模型 opt-in hook 接收每个 request 的 sampling extra args/discard 状态；synthesis queue 串行运行 song job，并限制已提交但未完成的 work units。HOLD 只在 CUDA、预算和上下文 headroom 都允许时发出，HOLD 不进入 codec history；无 HOLD headroom 或预算用尽时允许在 finishing step 等待同一 synthesis queue。取消/错误后停止新提交，并在本 job 已提交 kernel 完成后才释放其 graphs、buffers 和 generator。 +- 禁止：宣称 finishing step 永不阻塞；把整个 ODE/VAE pass 一次塞入 launch queue；用另一 song 的 stream-wide 完成代替本 job event；缺失 admission metadata 却把 forwarding stub 当作模型请求已正确建立的证明。 +- 验收：覆盖 HOLD/zero-budget、排队取消、active 取消、history rollback 和资源回收；用实际模型 admission 测试与 eager/AR-graph、chunked/unchunked lifecycle case 分别证明路径，graph 声明须断言对应 replay counter。^[PR #8407] + +## YUE2-SYNTH-1b — 完成音频必须跨过 scheduler 的终态确认边界 + +- 触发：修改 YuE2 `_ship_audio`、finished row、`end_len` 或完成 step 的 preemption/replay。 +- 强制：保留已完成音频和对应终止 token，直到 scheduler 接受该 end boundary。若完成 step 的输出被丢弃、resumed row 的 accepted count 尚未到 `end_len`，重新交付保留的音频与 end token，复用已完成 synthesis；非丢弃的完整 accepted sequence 才能作为 history reconciliation 边界。 +- 禁止：finished row 无条件跳过而连续输出零 token；在重新交付时再次运行 synthesis；把未确认的部分 recompute 当作完整 accepted history；finish/abort 后遗留 job 或 timing state。 +- 验收：确定性回归让 scheduler 丢弃交付 step，恢复后仍得到同一音频/end token且只完成一次 synthesis；再覆盖真实 eager/graph preemption、取消与后续请求恢复。^[PR #8407] diff --git a/src/infermatrix_copilot/kb_service/cli.py b/src/infermatrix_copilot/kb_service/cli.py index 784befba2..8efd9d8bb 100644 --- a/src/infermatrix_copilot/kb_service/cli.py +++ b/src/infermatrix_copilot/kb_service/cli.py @@ -224,6 +224,8 @@ def main(argv: list[str] | None = None) -> int: reconcile.add_argument("--allow-merger", action="append", default=[]) reconcile.add_argument("--require-check", action="append", default=[]) reconcile.add_argument("--reason", default="") + reconcile.add_argument("--event-coverage", type=Path, + help="settle selected source events against already admitted, active owner-merged rules") run = sub.add_parser("run") run.add_argument("--playbook", required=True) run.add_argument("--repo", required=True) @@ -353,13 +355,25 @@ def main(argv: list[str] | None = None) -> int: try: key = _signing_key() if args.plan: - envelope = make_plan(rt, key, target=args.target or "", allow_mergers=args.allow_merger, - required_checks=args.require_check, reason=args.reason) + options = {"target": args.target or "", "allow_mergers": args.allow_merger, + "required_checks": args.require_check, "reason": args.reason} + if args.event_coverage: + from .event_settlement import make_plan as make_event_plan + + envelope = make_event_plan(rt, key, coverage=json.loads(args.event_coverage.read_text()), **options) + else: + envelope = make_plan(rt, key, **options) with args.plan.open("xb") as handle: handle.write(canonical_json(envelope)) print(f"verified reconciliation plan: {args.plan}; no commit admitted") else: - receipt = apply_plan(rt, json.loads(args.apply.read_text()), key) + envelope = json.loads(args.apply.read_text()) + if envelope.get("purpose") == "kb-reviewed-event-plan": + from .event_settlement import apply_plan as apply_event_plan + + receipt = apply_event_plan(rt, envelope, key) + else: + receipt = apply_plan(rt, envelope, key) print(f"supervised reconciliation receipt: {receipt}; pause and publication modes unchanged") return 0 except (ValueError, OSError, KeyError, RuntimeError, subprocess.SubprocessError) as exc: diff --git a/src/infermatrix_copilot/kb_service/event_settlement.py b/src/infermatrix_copilot/kb_service/event_settlement.py new file mode 100644 index 000000000..8981b68e5 --- /dev/null +++ b/src/infermatrix_copilot/kb_service/event_settlement.py @@ -0,0 +1,254 @@ +"""Complete source intake after explicit owner review, merge and activation. + +These receipts confer no knowledge-history trust and invent no model verdict. +They record the operator's dispositions of immutable, selected source events. +""" + +from __future__ import annotations + +import json +import os +import re +import tempfile +from datetime import datetime + +from ..knowledge_service.lifecycle import CITATION, Page, safe_source_path +from ..knowledge_service.signing import canonical_json, sign, verify +from .activate import activation_lock, verify_snapshot +from .reconcile import ( + ReconciliationError, _digest, _merge_evidence, _repository, _validate_target, trusted_commits, +) + +PLAN_PURPOSE = "kb-reviewed-event-plan" +RECEIPT_PURPOSE = "kb-reviewed-event-settlement" + + +def _source(event: dict) -> dict: + return {k: event[k] for k in ("id", "repo", "source", "external_id", "created_at", "status", "detail")} | { + "payload_sha256": _digest(event["payload"])} + + +def _merged_time(value: object) -> datetime: + if not isinstance(value, str): + raise ReconciliationError("upstream merge time must be an explicit timestamp") + try: + parsed = datetime.fromisoformat(value.replace("Z", "+00:00")) + except ValueError as exc: + raise ReconciliationError("upstream merge time is invalid") from exc + if parsed.tzinfo is None: + raise ReconciliationError("upstream merge time must have a timezone") + return parsed + + +def _historical_merge(rt, event: dict, item: dict) -> dict: + """Attest missing historical metadata without rewriting the source payload.""" + payload = event["payload"] + expected = item.get("source_merge_sha") + if "merge_commit_sha" in payload: + sha = payload["merge_commit_sha"] + if not isinstance(sha, str) or not re.fullmatch(r"[0-9a-f]{40}", sha): + raise ReconciliationError("original source merge SHA is invalid") + if "source_merge_sha" in item: + if expected != sha: + raise ReconciliationError("supplied source merge SHA conflicts with the immutable payload") + raise ReconciliationError("source_merge_sha is only for an original payload missing its merge SHA") + return {} + if not isinstance(expected, str) or not re.fullmatch(r"[0-9a-f]{40}", expected): + raise ReconciliationError("historical source requires an explicit full source_merge_sha") + lifecycle = rt.registry.get(event["repo"]) + upstream = getattr(lifecycle, "full_name", "") + if not isinstance(upstream, str) or not re.fullmatch(r"[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+", upstream): + raise ReconciliationError("historical source requires the configured upstream repository") + number = int(event["external_id"]) + pr = rt.github.get(f"/repos/{upstream}/pulls/{number}") + if not isinstance(pr, dict): + raise ReconciliationError("upstream PR evidence must be an object") + base, head_data = pr.get("base"), pr.get("head") + if not isinstance(base, dict) or not isinstance(base.get("repo"), dict) or not isinstance(head_data, dict): + raise ReconciliationError("upstream PR repository and head evidence are malformed") + base, head = base["repo"], head_data.get("sha") + if type(pr.get("number")) is not int or pr["number"] != number \ + or not isinstance(base.get("full_name"), str) or base["full_name"].lower() != upstream.lower() \ + or pr.get("html_url") != f"https://github.com/{upstream}/pull/{number}" \ + or pr.get("merged") is not True or pr.get("merge_commit_sha") != expected \ + or not isinstance(head, str) or not re.fullmatch(r"[0-9a-f]{40}", head): + raise ReconciliationError("historical source has unmatched upstream PR merge evidence") + merged_at = pr.get("merged_at") + merged_time = _merged_time(merged_at) + if "merged_at" in payload and _merged_time(payload["merged_at"]) != merged_time: + raise ReconciliationError("upstream merge time conflicts with the immutable payload") + return {"source_merge_sha": expected, "upstream_merge": { + "repository": upstream, "number": number, "url": pr["html_url"], + "merge_sha": expected, "head_sha": head, "merged_at": merged_at}} + + +def _active(rt, target: str, manifest: str) -> None: + snapshot = rt.state_dir / "snapshots" / target + link = rt.state_dir / "active" + if rt.ledger.active_snapshot() != target or not link.is_symlink() or link.resolve() != snapshot.resolve(): + raise ReconciliationError("event settlement requires the exact target to be active") + verify_snapshot(snapshot) + if _digest(json.loads((snapshot / "MANIFEST.json").read_text())) != manifest: + raise ReconciliationError("active snapshot differs from the verified target tree") + + +def prepare(rt, *, target: str, allow_mergers: list[str], required_checks: list[str], reason: str, + coverage: dict, settled: dict | None = None, receipt: str = "") -> dict: + if not re.fullmatch(r"[0-9a-f]{40}", target) or not reason.strip() or not allow_mergers or not required_checks: + raise ReconciliationError("target SHA, owner allowlist, required checks and reason are mandatory") + if not isinstance(coverage, dict) or set(coverage) != {"commits", "events"} \ + or not isinstance(coverage["commits"], list) or not isinstance(coverage["events"], list) \ + or not coverage["commits"] or not coverage["events"]: + raise ReconciliationError("coverage requires explicit merged knowledge commits and selected events") + if any(not isinstance(sha, str) for sha in coverage["commits"]): + raise ReconciliationError("coverage commits must be immutable merge SHA strings") + if rt.knowledge.fetch() != target: + raise ReconciliationError("main differs from the explicitly pinned target") + repository, allowed = _repository(rt.knowledge), {a.lower() for a in allow_mergers} + actor = rt.github.get("/user").get("login", "") + if actor.lower() not in allowed: + raise ReconciliationError("authenticated operator is not an allowlisted owner") + trusted = trusted_commits(rt) + first_parent = set(rt.knowledge._git("rev-list", "--first-parent", target).decode().split()) + commits = [] + for sha in sorted(set(coverage["commits"])): + if not isinstance(sha, str) or not re.fullmatch(r"[0-9a-f]{40}", sha) or sha not in trusted or sha not in first_parent: + raise ReconciliationError("coverage merge must already have supervised admission on the target history") + parents = rt.knowledge._git("show", "-s", "--format=%P", sha).decode().split() + manifest = rt.knowledge.raw_manifest(parents[0], sha) + if not any(row["path"].startswith("knowledge/") for row in manifest): + raise ReconciliationError("coverage commit must merge actual knowledge content") + commit = {"sha": sha, "parents": parents, "manifest_sha256": _digest(manifest)} + commit["pr"] = _merge_evidence(rt.github, repository, commit, allowed, set(required_checks)) + commits.append(commit) + validation = _validate_target(rt.knowledge, target) + _active(rt, target, validation["snapshot_manifest_sha256"]) + files, rows, seen = rt.knowledge.knowledge_files(target), [], set() + previous = {r["source_event"]["id"]: r for r in (settled or {}).get("events", [])} + for item in coverage["events"]: + if not isinstance(item, dict) or set(item) - {"id", "outcome", "rules", "reason", "source_merge_sha"}: + raise ReconciliationError("invalid selected event coverage") + event_id, outcome, reason_text = item.get("id"), item.get("outcome"), item.get("reason", "") + if type(event_id) is not int or event_id in seen or outcome not in {"covered", "already_covered", "no_rule"}: + raise ReconciliationError("event IDs must be unique integers with explicit dispositions") + if not isinstance(reason_text, str) or not reason_text.strip(): + raise ReconciliationError("every event disposition needs the operator's review reason") + seen.add(event_id) + event = rt.ledger.event(event_id) + if event["source"] != "merged_pr" or not re.fullmatch(r"[1-9][0-9]*", event["external_id"]) \ + or not isinstance(event["payload"], dict): + raise ReconciliationError("settlement requires an immutable merged upstream PR event") + source = _source(event) + prior = previous.get(event_id) + if prior and event["status"] == "done": + detail = json.loads(event["detail"]) + if detail.get("receipt") == receipt and detail.get("outcome") == "manual_reviewed_merged": + source["status"], source["detail"] = prior["source_event"]["status"], prior["source_event"]["detail"] + premature_discard = source["status"] == "done" and source["detail"] == "no rules" + if source["status"] not in {"pending", "rejected", "drafted"} and not premature_discard: + raise ReconciliationError("only pending, rejected, drafted or original done/no rules events can be settled") + historical = _historical_merge(rt, event, item) + rules = item.get("rules", []) + if not isinstance(rules, list) or (outcome == "no_rule" and rules) or (outcome != "no_rule" and not rules): + raise ReconciliationError("covered events need exact rule references; no-rule events cannot claim rules") + proved = [] + for reference in rules: + if not isinstance(reference, dict) or set(reference) != {"path", "rule_id"}: + raise ReconciliationError("rules require exact knowledge-relative path and rule_id") + path, rule_id = reference["path"], reference["rule_id"] + if not safe_source_path(path) or not path.startswith(f"repos/{event['repo']}/") or path not in files: + raise ReconciliationError("covered rule must be on a canonical page in the event's repository") + matches = [section for section in Page.parse(files[path]).rules() if section.rule_id == rule_id] + if len(matches) != 1 or matches[0].footer.status != "active": + raise ReconciliationError("covered rule is missing, duplicated or retired") + if f"PR #{event['external_id']}" not in {m.group("ref") for m in CITATION.finditer(matches[0].text)}: + raise ReconciliationError("covered rule must cite the original upstream PR") + proved.append({**reference, "section_sha256": _digest(matches[0].text)}) + rows.append({"source_event": source, "outcome": outcome, "reason": reason_text.strip(), + "rules": sorted(proved, key=lambda r: (r["path"], r["rule_id"])), **historical}) + changesets = rt.ledger.changesets_for_events(seen) + previous_changesets = {row["id"]: row for row in (settled or {}).get("changesets", [])} + for change in changesets: + if change["pr_number"] is not None or change["pending_item"]: + raise ReconciliationError("source changeset has a PR or pending publication; settle it first") + if change["status"] == "manual_reviewed_merged" and change["detail"].get("manual_reviewed_event_settlement") == receipt: + prior = previous_changesets.get(change["id"]) + if not prior: + raise ReconciliationError("settled changeset is not covered by this receipt") + change["status"] = prior["status"] + change["detail"].pop("manual_reviewed_event_settlement") + if rt.knowledge.fetch() != target: + raise ReconciliationError("main changed during event review") + _active(rt, target, validation["snapshot_manifest_sha256"]) + return {"schema_version": 1, "repository": repository, "target": target, + "target_tree": rt.knowledge._git("rev-parse", f"{target}^{{tree}}").decode().strip(), + "actor": actor, "reason": reason.strip(), "allow_mergers": sorted(allowed), + "required_checks": sorted(set(required_checks)), "commits": commits, + "events": sorted(rows, key=lambda r: r["source_event"]["id"]), + "changesets": [{k: row[k] for k in ("id", "repo", "status", "detail")} for row in changesets], + **validation, "prepared_at": rt.clock()} + + +def make_plan(rt, key, **options) -> dict: + return sign(PLAN_PURPOSE, prepare(rt, **options), key) + + +def _receipts(rt, public) -> list[tuple]: + directory = rt.state_dir / "event-settlements" + if directory.is_symlink(): + raise ReconciliationError("event settlement directory must not be a symlink") + result = [] + for path in sorted(directory.glob("*.json")): + if path.is_symlink() or not path.is_file(): + raise ReconciliationError("event settlement receipts must be regular immutable files") + envelope = json.loads(path.read_text()) + if path.stem != _digest(envelope): + raise ReconciliationError("event settlement receipt was modified") + payload = verify(RECEIPT_PURPOSE, envelope, public) + if payload.get("schema_version") != 1 or payload.get("repository") != _repository(rt.knowledge): + raise ReconciliationError("event settlement receipt identifies another repository") + result.append((path, payload)) + return result + + +def apply_plan(rt, envelope: dict, key): + planned = verify(PLAN_PURPOSE, envelope, key.public_key()) + with rt.ledger.lease() as owner, activation_lock(rt.state_dir): + existing = next(((path, data) for path, data in _receipts(rt, key.public_key()) + if data.get("plan_sha256") == _digest(envelope)), None) + coverage = {"commits": [row["sha"] for row in planned["commits"]], + "events": [{"id": row["source_event"]["id"], "outcome": row["outcome"], "reason": row["reason"], + "rules": [{k: ref[k] for k in ("path", "rule_id")} for ref in row["rules"]], + **({"source_merge_sha": row["source_merge_sha"]} if "source_merge_sha" in row else {})} + for row in planned["events"]]} + fresh = prepare(rt, target=planned["target"], allow_mergers=planned["allow_mergers"], + required_checks=planned["required_checks"], reason=planned["reason"], coverage=coverage, + settled=existing[1] if existing else None, receipt=existing[0].stem if existing else "") + if {k: v for k, v in fresh.items() if k != "prepared_at"} != {k: v for k, v in planned.items() if k != "prepared_at"}: + raise ReconciliationError("source events or reviewed evidence changed; prepare a new plan") + if existing: + path = existing[0] + else: + receipt = sign(RECEIPT_PURPOSE, {**fresh, "settled_at": rt.clock(), "plan_sha256": _digest(envelope)}, key) + directory = rt.state_dir / "event-settlements" + directory.mkdir(parents=True, exist_ok=True) + path = directory / f"{_digest(receipt)}.json" + fd, temporary = tempfile.mkstemp(prefix=".receipt-", dir=directory) + try: + with os.fdopen(fd, "wb") as handle: + handle.write(canonical_json(receipt)) + handle.flush() + os.fsync(handle.fileno()) + os.link(temporary, path) + directory_fd = os.open(directory, os.O_RDONLY) + try: + os.fsync(directory_fd) + finally: + os.close(directory_fd) + finally: + os.unlink(temporary) + rt.ledger.settle_reviewed_events(owner, fresh["events"], fresh["changesets"], path.stem) + rt.trace("outcome", context={"playbook": "kb-reconcile-reviewed", "repo": fresh["repository"]}, + result={"outcome": "manual_reviewed_merged", "target": fresh["target"], "receipt": path.name, + "actor": fresh["actor"], "events": len(fresh["events"])}) + return path diff --git a/src/infermatrix_copilot/kb_service/ledger.py b/src/infermatrix_copilot/kb_service/ledger.py index ea585e040..686977bfd 100644 --- a/src/infermatrix_copilot/kb_service/ledger.py +++ b/src/infermatrix_copilot/kb_service/ledger.py @@ -334,6 +334,69 @@ def events(self, repo: str, status: str, *, limit: int = 100) -> list[dict[str, (repo, status, limit)).fetchall() return [{**dict(r), "payload": json.loads(r["payload"])} for r in rows] + def event(self, event_id: int) -> dict[str, Any]: + row = self._conn.execute("SELECT * FROM events WHERE id=?", (event_id,)).fetchone() + if row is None: + raise ValueError(f"source event {event_id} does not exist") + return {**dict(row), "payload": json.loads(row["payload"])} + + def changesets_for_events(self, event_ids: set[int]) -> list[dict]: + """Whole source batches and their refinement descendants, never a partial batch.""" + rows = [_changeset_row(r) for r in self._conn.execute("SELECT * FROM changesets")] + selected = set() + for row in rows: + ids = set(row["detail"].get("event_ids") or []) + if ids & event_ids: + if not ids <= event_ids: + raise ValueError("event settlement must cover the whole existing source batch") + selected.add(row["id"]) + while True: + descendants = {row["id"] for row in rows if any( + item.get("changeset") in selected for item in row["detail"].get("refine_history") or [])} + if descendants <= selected: + break + selected |= descendants + return [row for row in rows if row["id"] in selected] + + def settle_reviewed_events(self, owner: str, rows: list[dict], changesets: list[dict], receipt: str) -> None: + """Apply verified manual dispositions atomically; preserve model verdicts.""" + import hashlib + + from ..knowledge_service.signing import canonical_json + + now = self._clock() + with self.fenced(owner) as cur: + for item in rows: + source = item["source_event"] + row = cur.execute("SELECT * FROM events WHERE id=?", (source["id"],)).fetchone() + if row is None or any(row[k] != source[k] for k in ("repo", "source", "external_id", "created_at")) \ + or hashlib.sha256(canonical_json(json.loads(row["payload"]))).hexdigest() != source["payload_sha256"]: + raise ValueError("source event identity or payload changed before settlement") + completed = json.dumps({"outcome": "manual_reviewed_merged", "receipt": receipt, + "disposition": item["outcome"], "reason": item["reason"]}, sort_keys=True) + if row["status"] == "done" and row["detail"] == completed: + continue + if row["status"] != source["status"] or row["detail"] != source["detail"]: + raise ValueError("source event changed before settlement") + cur.execute("UPDATE events SET status='done', detail=?, updated_at=? WHERE id=?", + (completed, now, source["id"])) + completed_changesets = set() + for change in changesets: + row = cur.execute("SELECT status, detail FROM changesets WHERE id=?", (change["id"],)).fetchone() + detail = json.loads(row["detail"]) + if detail.get("manual_reviewed_event_settlement") == receipt and row["status"] == "manual_reviewed_merged": + completed_changesets.add(change["id"]) + continue + if row["status"] != change["status"] or detail != change["detail"]: + raise ValueError("source changeset changed before settlement") + detail["manual_reviewed_event_settlement"] = receipt + cur.execute("UPDATE changesets SET status='manual_reviewed_merged', detail=?, updated_at=? WHERE id=?", + (json.dumps(detail, sort_keys=True), now, change["id"])) + completed_changesets.add(change["id"]) + for queue in cur.execute("SELECT id, changeset_id FROM human_queue WHERE resolved_at IS NULL").fetchall(): + if queue["changeset_id"] in completed_changesets: + cur.execute("UPDATE human_queue SET resolved_at=? WHERE id=?", (now, queue["id"])) + def set_event_status(self, event_id: int, status: str, detail: str = "") -> None: with self.tx() as cur: cur.execute( diff --git a/src/infermatrix_copilot/knowledge_service/curation.py b/src/infermatrix_copilot/knowledge_service/curation.py index 58dcc3867..1da6b00a7 100644 --- a/src/infermatrix_copilot/knowledge_service/curation.py +++ b/src/infermatrix_copilot/knowledge_service/curation.py @@ -20,6 +20,32 @@ class KnowledgeCurator(ApplyMixin, ProposalMixin, PromptMixin, CatalogMixin): + @staticmethod + def reviewed_rule_evidence(page_text: str, *, rule_id: str, source_reference: str) -> dict[str, str]: + """Verify one active, source-citing rule and return its exact block digest.""" + from .lifecycle import Page + + sections = [section for section in Page.parse(page_text).rules() if section.rule_id == rule_id] + if len(sections) != 1 or sections[0].footer.status != "active" or source_reference not in sections[0].citations: + raise KnowledgeCurationError("reviewed rule must exist once, be active, and cite its source") + return {"rule_id": rule_id, "source_reference": source_reference, + "section_sha256": hashlib.sha256(sections[0].text.encode("utf-8")).hexdigest()} + + @staticmethod + def installed_knowledge_file(path: str) -> bytes: + """Read a canonical governed page from this installed provider package.""" + from importlib.resources import files + + from .lifecycle import safe_source_path + + if not safe_source_path(path) or not path.startswith("knowledge/repos/") or not path.endswith(".md"): + raise InvalidRequestError("installed knowledge path must be a canonical repository Markdown page") + target = files("infermatrix_copilot").joinpath(*path.split("/")) + try: + return target.read_bytes() + except (OSError, ValueError) as exc: + raise KnowledgeCurationError("installed provider knowledge page is unavailable") from exc + def __init__( self, workspace_root: str | Path, diff --git a/src/infermatrix_copilot/knowledge_service/signing.py b/src/infermatrix_copilot/knowledge_service/signing.py index 0430da7ab..13a71b42c 100644 --- a/src/infermatrix_copilot/knowledge_service/signing.py +++ b/src/infermatrix_copilot/knowledge_service/signing.py @@ -24,7 +24,8 @@ ) PURPOSES = ("kb-gate-verdict", "kb-outbox-item", "kb-control", "kb-ack", - "kb-reconciliation-plan", "kb-reviewed-reconciliation") + "kb-reconciliation-plan", "kb-reviewed-reconciliation", + "kb-reviewed-event-plan", "kb-reviewed-event-settlement") class SignatureError(ValueError): diff --git a/test/fixtures/direct_routing/golden_routes.jsonl b/test/fixtures/direct_routing/golden_routes.jsonl index 91e68b962..a552d86ad 100644 --- a/test/fixtures/direct_routing/golden_routes.jsonl +++ b/test/fixtures/direct_routing/golden_routes.jsonl @@ -54,7 +54,7 @@ {"pr":7963,"title":"[CI/Build][MiniCPM-o] Gate Seed-TTS WER over chat completions instead of the native duplex path","body":"## Purpose\r\n\r\nFix the two failing MiniCPM-o 4.5 nightly accuracy tests (Buildkite build 15686, commit 9f552135):\r\n\r\n```\r\nFAILED test_minicpmo_4_5_duplex_seed_tts_wer_bench[three-stage-single-gpu] WER 0.0797 > 0.05\r\nFAILED test_minicpmo_4_5_duplex_seed_tts_wer_bench[three-stage-single-gpu-bf16-attention-cache] WER 0.0795 > 0.05\r\n```\r\n\r\n**Why the backing API changes to `/v1/chat/completions`:** the turn-based path is gone from the new full-duplex session, but the original test was calibrated for turn-based mode.\r\n\r\n- Before #7413 (Unified Full-duplex Framework) the test's client drove `/v1/realtime` with `conversation.item.create`. The server ran that through the turn-based whole-utterance Talker, the same path `/v1/chat/completions` uses, and the 0.05 gate was set on that path (WER 0.023).\r\n- The unified duplex session has no turn-based path over `/v1/realtime` any more. #7413 rewrote the client to seed `duplex_initial_user_text` and stream silence, so since then the test has exercised the native per-unit path for the first time. The gate was never calibrated for that path and it cannot meet it:\r\n\r\n| Path, same 200 turns (50 sessions x 4) | Mean WER |\r\n|---|---|","changed_files":["tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py"],"expected":{"status":"ready","selected_by":"title_body","changed_files_role":"scope_validation_only","routes":[{"owner":"model:minicpm-o-4-5","path":"repos/vllm-omni/models/minicpm-o-4-5/rules.md","reason":"title/body model: minicpm-o-4-5","quick_map_status":"ok","read_required":false},{"owner":"serving","path":"repos/vllm-omni/components/serving/rules.md","reason":"title/body: server","quick_map_status":"truncated","read_required":true}],"scope_validation":[]}} {"pr":7961,"title":"[Bugfix][XPU] Enable LTX-2 family on XPU","body":"\r\n\r\nTwo defects stop every LTX-2 class checkpoint from running on a 32 GiB Intel\r\nXPU card.\r\n\r\n## 1. The text encoder is never declared offloadable\r\n\r\n`LTX2Pipeline` declares `_layerwise_offload_blocks_attrs` for its DiT but no\r\n`_offload_plan`, so the layerwise backends have no block path for the Gemma\r\ntext encoder and it stays resident next to the DiT. On `Lightricks/LTX-2` that\r\nencoder is 48.07 GiB in bf16, larger than the 35.16 GiB transformer it feeds.\r\n\r\nFix: declare the encoder's block path. It is duck-typed on the encoder exposing\r\n`model.language_model.layers`, so Gemma3 and Gemma4 are both covered.\r\n\r\nPeak memory on `Lightricks/LTX-2` drops from 30,069 MiB (OOM) to 18,979 MiB\r\nagainst a 32,650 MiB card.\r\n\r\n## 2. `_run_ltx_vocoder` relies on fp32 autocast, which only CUDA supports\r\n\r\nThe BWE path casts its input with `.float()` and expects\r\n`torch.autocast(dtype=torch.float32)` to lift the bf16 weights to match. XPU and\r\nCPU autocast accept only fp16/bf16 and degrade silently, so fp32 activations\r\nmeet bf16 weights.\r\n\r\nFix: extend the existing MPS branch, which already materializes fp32 weights for\r\nthis case, to the other backends whose autocast lacks fp32.\r\n\r\nOnly chec","changed_files":["vllm_omni/diffusion/models/ltx2/ltx2_components.py","vllm_omni/diffusion/models/ltx2/ltx2_runtime.py"],"expected":{"status":"ready","selected_by":"title_body","changed_files_role":"scope_validation_only","routes":[{"owner":"model-executor","path":"repos/vllm-omni/components/model-executor/rules.md","reason":"title/body: checkpoint","quick_map_status":"ok","read_required":false},{"owner":"diffusion","path":"repos/vllm-omni/components/diffusion/rules.md","reason":"title/body: dit","quick_map_status":"ok","read_required":false}],"scope_validation":[{"owner":"diffusion","changed_files":["vllm_omni/diffusion/models/ltx2/ltx2_components.py","vllm_omni/diffusion/models/ltx2/ltx2_runtime.py"],"selected_from_description":true}]}} {"pr":7959,"title":"[Bugfix] Reject stray <|AUDIO|> placeholder with use_audio_in_video in Qwen2.5-Omni","body":"## Purpose\n\nRunning the Qwen2.5-Omni offline example with `--query-type use_audio_in_video` kills the thinker engine on current `main`:\n\n```text\nFile \".../qwen2_5_omni/qwen2_5_omni.py\", line 489, in get_mrope_input_positions\n audio_seqlen = audio_seqlens[audio_idx]\nIndexError: list index out of range\n...\nvllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue.\n```\n\n**Repro (before this PR):**\n\n```bash\ncd examples/offline_inference/qwen2_5_omni\npython end2end.py --model Qwen/Qwen2.5-Omni-3B --query-type use_audio_in_video\n```\n\n**Root cause.** The example prompt contains `<|vision_bos|><|VIDEO|><|vision_eos|>` *plus* a separate `<|audio_bos|><|AUDIO|><|audio_eos|>`. With `use_audio_in_video=True`, `_maybe_apply_prompt_updates` drops the `audio` prompt updates and derives the audio placeholder from the video placeholder (the audio is interleaved into the video span). The standalone `<|AUDIO|>` token is therefore left unexpanded with no audio item behind it. It passes input processing, reaches the engine, and `get_mrope_input_positions` indexes past the end of `audio_feature_lengths`, which takes down the whole stage instead of failing the one request.\n\n**Fix.**\n\n","changed_files":["examples/offline_inference/qwen2_5_omni/end2end.py","tests/model_executor/models/qwen2_5_omni/test_per_video_audio_mask.py","vllm_omni/model_executor/models/qwen2_5_omni/qwen2_5_omni_thinker.py"],"expected":{"status":"scope_fallback","selected_by":"title_body+changed_files","changed_files_role":"selected_fallback_routes","routes":[{"owner":"serving","path":"repos/vllm-omni/components/serving/rules.md","reason":"title/body: request","quick_map_status":"truncated","read_required":true},{"owner":"model-executor","path":"repos/vllm-omni/components/model-executor/rules.md","reason":"changed files: vllm_omni/model_executor/models/qwen2_5_omni/qwen2_5_omni_thinker.py","quick_map_status":"ok","read_required":false}],"scope_validation":[{"owner":"model-executor","changed_files":["vllm_omni/model_executor/models/qwen2_5_omni/qwen2_5_omni_thinker.py"],"selected_from_description":false}]}} -{"pr":7953,"title":"[Bugfix] Restore GLM-Image unshifted timesteps after set_timesteps","body":"## Summary\n\nFixes #7951.\n\n`diffusers` 0.40 `FlowMatchEulerDiscreteScheduler.set_timesteps` overwrites caller-supplied timesteps with `sigmas * num_train_timesteps`. GLM-Image integrates on resolution-shifted sigmas but conditions the DiT on the unshifted timesteps it was trained with, so the overwrite produced magenta/grainy images after the #6459 pin bump.\n\nRestore the provided unshifted timesteps after `set_timesteps` in GLM-Image's `retrieve_timesteps` (both-args path). Shifted sigmas still drive the Euler update.\n\n## Test plan\n\n- [x] CPU unit test `tests/diffusion/models/glm_image/test_glm_image_retrieve_timesteps.py` (1 passed)\n- [x] `ruff check` / `ruff format --check` on changed files\n- [ ] No GPU image render in this environment","changed_files":["tests/diffusion/models/glm_image/test_glm_image_retrieve_timesteps.py","vllm_omni/diffusion/models/glm_image/pipeline_glm_image.py"],"expected":{"status":"ready","selected_by":"title_body","changed_files_role":"scope_validation_only","routes":[{"owner":"diffusion","path":"repos/vllm-omni/components/diffusion/rules.md","reason":"title/body: diffusion, dit","quick_map_status":"ok","read_required":false}],"scope_validation":[{"owner":"diffusion","changed_files":["vllm_omni/diffusion/models/glm_image/pipeline_glm_image.py"],"selected_from_description":true}]}} +{"pr":7953,"title":"[Bugfix] Restore GLM-Image unshifted timesteps after set_timesteps","body":"## Summary\n\nFixes #7951.\n\n`diffusers` 0.40 `FlowMatchEulerDiscreteScheduler.set_timesteps` overwrites caller-supplied timesteps with `sigmas * num_train_timesteps`. GLM-Image integrates on resolution-shifted sigmas but conditions the DiT on the unshifted timesteps it was trained with, so the overwrite produced magenta/grainy images after the #6459 pin bump.\n\nRestore the provided unshifted timesteps after `set_timesteps` in GLM-Image's `retrieve_timesteps` (both-args path). Shifted sigmas still drive the Euler update.\n\n## Test plan\n\n- [x] CPU unit test `tests/diffusion/models/glm_image/test_glm_image_retrieve_timesteps.py` (1 passed)\n- [x] `ruff check` / `ruff format --check` on changed files\n- [ ] No GPU image render in this environment","changed_files":["tests/diffusion/models/glm_image/test_glm_image_retrieve_timesteps.py","vllm_omni/diffusion/models/glm_image/pipeline_glm_image.py"],"expected":{"status":"ready","selected_by":"title_body","changed_files_role":"scope_validation_only","routes":[{"owner":"model:glm-image","path":"repos/vllm-omni/models/glm-image/rules.md","reason":"title/body model: glm-image","quick_map_status":"ok","read_required":false},{"owner":"diffusion","path":"repos/vllm-omni/components/diffusion/rules.md","reason":"title/body: diffusion, dit","quick_map_status":"ok","read_required":false}],"scope_validation":[{"owner":"diffusion","changed_files":["vllm_omni/diffusion/models/glm_image/pipeline_glm_image.py"],"selected_from_description":true}]}} {"pr":7930,"title":"[Model] Default Qwen3-TTS to experimental Model Runner V2","body":"## Summary\n\nMake the bundled Qwen3-TTS deploy profile select Model Runner V2 on CUDA, and document MRV2 as experimental with an explicit opt-out. Follow-up to #7781, which enabled MRV2 as opt-in only.\n\n- `vllm_omni/deploy/qwen3_tts.yaml`: `model_runner: v2`, stage-0 talker prefill bound to 512 tokens (the validated basic MRV2 profile), and `platforms:` sections that pin V1 on NPU, XPU, ROCm and MUSA.\n- `qwen3_tts_mrv2.yaml`: kept as an explicit MRV2 profile; header updated because the base file now selects V2 as well. Removing the now-redundant profile can be a separate cleanup.\n- Config tests: the \"default profiles are explicit opt-in\" test is split — `qwen3_tts_high_concurrency.yaml` stays V1 with its opt-in MRV2 sibling, and a new test pins the Qwen3-TTS default plus per-platform fallbacks.\n- Docs: `docs/configuration/stage_configs.md` and `recipes/Qwen/Qwen3-TTS.md` note the experimental default and how to opt out.\n\n## Scope\n\nQwen3-TTS only. Qwen3-Omni and MOSS keep V1; their opt-in MRV2 profiles are untouched, and no other model family is affected.\n\n## Testing\n\nStatic checks passed locally: YAML validation, a standalone simulation of config resolution (CUDA → v2; NPU/XPU/ROCm/","changed_files":["docs/configuration/stage_configs.md","recipes/Qwen/Qwen3-TTS.md","tests/config/test_omni_config.py","vllm_omni/deploy/qwen3_tts.yaml","vllm_omni/deploy/qwen3_tts_mrv2.yaml"],"expected":{"status":"ready","selected_by":"title_body","changed_files_role":"scope_validation_only","routes":[{"owner":"model:qwen3-tts","path":"repos/vllm-omni/models/qwen3-tts/rules.md","reason":"title/body model: qwen3-tts","quick_map_status":"ok","read_required":false},{"owner":"configuration","path":"repos/vllm-omni/components/configuration/rules.md","reason":"title/body: config, configuration, yaml","quick_map_status":"ok","read_required":false}],"scope_validation":[{"owner":"configuration","changed_files":["vllm_omni/deploy/qwen3_tts.yaml","vllm_omni/deploy/qwen3_tts_mrv2.yaml"],"selected_from_description":true}]}} {"pr":7928,"title":"[BugFix] Download Anima assets for README CI test","body":"## Issue\n\nFixes #7910\n\nThe Anima README test executes placeholder model paths, causing `HFValidationError` before model loading.\n\n## Fix\n\nDownload the Anima checkpoint and required components before running the test, then replace the placeholders with their local cache paths. Link the asset preparation recipe in the README.\n\n## Validation\n\n- Two CPU regression tests and all pre-commit checks passed.\n- Real downloads passed the previously failing path validation.\n\n","changed_files":["examples/offline_inference/text_to_image/README.md","tests/diffusion/test_diffusion_engine.py","tests/examples/offline_inference/test_text_to_image.py","vllm_omni/diffusion/diffusion_engine.py"],"expected":{"status":"scope_fallback","selected_by":"title_body+changed_files","changed_files_role":"selected_fallback_routes","routes":[{"owner":"model-executor","path":"repos/vllm-omni/components/model-executor/rules.md","reason":"title/body: checkpoint","quick_map_status":"ok","read_required":false},{"owner":"diffusion","path":"repos/vllm-omni/components/diffusion/rules.md","reason":"changed files: vllm_omni/diffusion/diffusion_engine.py","quick_map_status":"ok","read_required":false}],"scope_validation":[{"owner":"diffusion","changed_files":["vllm_omni/diffusion/diffusion_engine.py"],"selected_from_description":false}]}} {"pr":7926,"title":"[Bugfix][Core] Fix stage initialization spawn-lock/device-lock inversion","body":"## Purpose\n\nFixes #7290.\n\n**Failure:** Shared-GPU stage initialization can stall until the device-lock timeout, logging `Timeout waiting for device ... initialization lock, proceeding anyway`. One stage holds a GPU flock and needs the spawn-environment lock; another holds the spawn-environment lock while waiting for that GPU flock.\n\n**Root cause:** Device-flock acquisition runs inside the process-wide spawn-environment lock. #7328 mitigates the original scheduling scenario by grouping overlapping devices, but does not remove the inverted lock ordering.\n\n**Fix:** Pass resolved physical devices explicitly to `acquire_device_locks` and wait outside the spawn-environment lock in the current common LLM launch path. Apply the same ordering to distributed-local diffusion, retaining scoped device visibility for configuration and process creation, lock ownership, and failure cleanup. Colocated diffusion initialization still runs under its scoped device environment. Remove the now-unused runtime scope wrappers.\n\nThis applies the approach discussed in #4895 to current main and also covers the diffusion follow-up discussed in #7372. The diffusion hazard is latent under current scheduling: inli","changed_files":["tests/engine/test_async_omni_engine_stage_init.py","vllm_omni/engine/stage_engine_startup.py","vllm_omni/engine/stage_init_utils.py","vllm_omni/engine/stage_runtime.py"],"expected":{"status":"ready","selected_by":"title_body","changed_files_role":"scope_validation_only","routes":[{"owner":"configuration","path":"repos/vllm-omni/components/configuration/rules.md","reason":"title/body: configuration","quick_map_status":"ok","read_required":false},{"owner":"diffusion","path":"repos/vllm-omni/components/diffusion/rules.md","reason":"title/body: diffusion","quick_map_status":"ok","read_required":false},{"owner":"distributed","path":"repos/vllm-omni/components/distributed/_index.md","reason":"title/body: distributed","quick_map_status":"ok","read_required":false}],"scope_validation":[]}} diff --git a/test/test_kb_reviewed_events.py b/test/test_kb_reviewed_events.py new file mode 100644 index 000000000..aa75c63f3 --- /dev/null +++ b/test/test_kb_reviewed_events.py @@ -0,0 +1,354 @@ +"""Manual source completion needs real merged admission and the exact active rules.""" + +from __future__ import annotations + +import json +from dataclasses import replace + +import pytest + +from infermatrix_copilot.kb_service import event_settlement as settlement, reconcile +from infermatrix_copilot.kb_service.activate import activate, switch_active +from infermatrix_copilot.kb_service.ledger import LeaseError +from infermatrix_copilot.knowledge_service.signing import SignatureError, verify +from test_kb_intake_gate import PAGE +from test_kb_reconcile_reviewed import world # noqa: F401 + + +@pytest.fixture(autouse=True) +def settlement_repository(world, monkeypatch): + monkeypatch.setattr(settlement, "_repository", reconcile._repository) + + +def _ready(w, *, status="pending", no_rule=False): + reconcile.apply_plan(w.rt, reconcile.make_plan(w.rt, w.key, **w.options), w.key) + activate(w.rt, w.sha) + event = w.rt.ledger.record_event("demo", "merged_pr", "11" if no_rule else "10", + {"source_reference": "PR #11" if no_rule else "PR #10", + "merge_commit_sha": "a" * 40, "body": "immutable upstream evidence"}) + if status != "pending": + w.rt.ledger.set_event_status(event, status, "original model rejection" if status == "rejected" else "old draft") + coverage = {"commits": [w.sha], "events": [{"id": event, "outcome": "no_rule" if no_rule else "already_covered", + "reason": "owner reviewed source; no durable rule" if no_rule else "owner reviewed the cited existing rule", + "rules": [] if no_rule else [{"path": PAGE, "rule_id": "DEMO-1a"}]}]} + w.rt.ledger.release_lease(w.rt.lease_owner) + return event, coverage + + +@pytest.mark.parametrize("status", ["pending", "rejected", "drafted"]) +def test_selected_event_completes_honestly_and_future_events_stay_pending(world, status): + w = world + event, coverage = _ready(w, status=status) + plan = settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + assert w.rt.ledger.event(event)["status"] == status + future = w.rt.ledger.record_event("demo", "merged_pr", "12", {"merge_commit_sha": "b" * 40}) + receipt = settlement.apply_plan(w.rt, plan, w.key) + assert settlement.apply_plan(w.rt, plan, w.key) == receipt + completed = w.rt.ledger.event(event) + assert completed["status"] == "done" + assert json.loads(completed["detail"])["outcome"] == "manual_reviewed_merged" + assert w.rt.ledger.event(future)["status"] == "pending" + assert reconcile.trusted_commits(w.rt) == {w.sha} # settlement adds no history trust + verified = verify(settlement.RECEIPT_PURPOSE, json.loads(receipt.read_text()), w.key.public_key()) + assert verified["events"][0]["source_event"]["status"] == status + + +def test_no_rule_requires_signed_reason_without_inventing_a_rule(world): + w = world + event, coverage = _ready(w, no_rule=True) + plan = settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + receipt = settlement.apply_plan(w.rt, plan, w.key) + assert json.loads(w.rt.ledger.event(event)["detail"])["disposition"] == "no_rule" + assert verify(settlement.RECEIPT_PURPOSE, json.loads(receipt.read_text()), w.key.public_key())["events"][0]["rules"] == [] + + +@pytest.mark.parametrize("no_rule", [False, True]) +def test_original_no_rules_disposition_can_be_reviewed_without_erasing_history(world, no_rule): + w = world + event, coverage = _ready(w, status="done", no_rule=no_rule) + w.rt.ledger.set_event_status(event, "done", "no rules") + plan = settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + receipt = settlement.apply_plan(w.rt, plan, w.key) + assert settlement.apply_plan(w.rt, plan, w.key) == receipt + evidence = verify(settlement.RECEIPT_PURPOSE, json.loads(receipt.read_text()), w.key.public_key())["events"][0] + assert evidence["source_event"]["status"] == "done" + assert evidence["source_event"]["detail"] == "no rules" + assert evidence["outcome"] == ("no_rule" if no_rule else "already_covered") + assert json.loads(w.rt.ledger.event(event)["detail"])["outcome"] == "manual_reviewed_merged" + + +@pytest.mark.parametrize("detail", ["old draft", "no rules ", '{"outcome":"manual_reviewed_merged","receipt":"other"}']) +def test_other_completed_events_cannot_have_their_history_replaced(world, detail): + w = world + event, coverage = _ready(w, status="done") + w.rt.ledger.set_event_status(event, "done", detail) + with pytest.raises(reconcile.ReconciliationError, match="original done/no rules"): + settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + assert w.rt.ledger.event(event)["detail"] == detail + + +def test_failed_batches_and_descendants_stop_but_verdicts_and_unrelated_queue_remain(world): + w = world + event, coverage = _ready(w) + with w.rt.ledger.lease() as owner: + original = w.rt.ledger.new_changeset_id("demo", "intake") + w.rt.ledger.stage_intake(owner, "demo", original, status="failed", drafted_events=[event], + detail={"event_ids": [event], "decision": {"status": "fail"}}, + verdicts=[{"layer": "gate", "verdict": "fail"}], human_reason="model failed") + child = w.rt.ledger.new_changeset_id("demo", "refine") + w.rt.ledger.stage_intake(owner, "demo", child, kind="refine", status="failed", drafted_events=[], + detail={"refine_history": [{"changeset": original}], "decision": {"status": "fail"}}, + verdicts=[{"layer": "gate", "verdict": "fail"}], human_reason="refinement failed") + w.rt.ledger.enqueue_human("demo", "unrelated task") + before = w.rt.ledger._conn.execute("SELECT * FROM verdicts").fetchall() + plan = settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + settlement.apply_plan(w.rt, plan, w.key) + assert w.rt.ledger.changeset(original)["status"] == "manual_reviewed_merged" + assert w.rt.ledger.changeset(child)["status"] == "manual_reviewed_merged" + assert w.rt.ledger.changeset(original)["detail"]["decision"]["status"] == "fail" + assert w.rt.ledger._conn.execute("SELECT * FROM verdicts").fetchall() == before + remaining = w.rt.ledger.human_queue() + assert "unrelated task" in [row["reason"] for row in remaining] + assert not any(row["changeset_id"] in {original, child} for row in remaining) + + +@pytest.mark.parametrize("drift", ["payload", "identity", "status", "created_at"]) +def test_source_replacement_or_state_drift_refuses_before_receipt(world, drift): + w = world + event, coverage = _ready(w) + plan = settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + if drift == "payload": + w.rt.ledger._conn.execute("UPDATE events SET payload=? WHERE id=?", (json.dumps({"merge_commit_sha": "b" * 40}), event)) + elif drift == "identity": + w.rt.ledger._conn.execute("UPDATE events SET external_id='15' WHERE id=?", (event,)) + elif drift == "created_at": + w.rt.ledger._conn.execute("UPDATE events SET created_at=created_at+1 WHERE id=?", (event,)) + else: + w.rt.ledger.set_event_status(event, "rejected", "a newer result") + with pytest.raises(reconcile.ReconciliationError): + settlement.apply_plan(w.rt, plan, w.key) + assert not (w.rt.state_dir / "event-settlements").exists() + + +@pytest.mark.parametrize("failure", ["missing_rule", "wrong_scope", "no_citation", "no_reason", "no_merge", "failed_check"]) +def test_unproven_coverage_cannot_complete_source_events(world, failure): + w = world + event, coverage = _ready(w) + if failure == "missing_rule": + coverage["events"][0]["rules"][0]["rule_id"] = "MISSING-1" + elif failure == "wrong_scope": + coverage["events"][0]["rules"][0]["path"] = "repos/other/rules.md" + elif failure == "no_citation": + w.rt.ledger._conn.execute("UPDATE events SET external_id='15' WHERE id=?", (event,)) + elif failure == "no_reason": + coverage["events"][0]["reason"] = "" + elif failure == "no_merge": + coverage["commits"] = [w.base] + else: + w.rt.github.checks[0]["conclusion"] = "failure" + with pytest.raises(reconcile.ReconciliationError): + settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + assert w.rt.ledger.event(event)["status"] == "pending" + + +def test_wrong_active_target_and_corrupt_receipts_refuse(world): + w = world + event, coverage = _ready(w) + plan = settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + switch_active(w.rt.state_dir, w.rt.state_dir / "snapshots" / w.base) + with pytest.raises(reconcile.ReconciliationError, match="exact target"): + settlement.apply_plan(w.rt, plan, w.key) + activate(w.rt, w.sha) + receipt = settlement.apply_plan(w.rt, plan, w.key) + receipt.write_text("{}") + with pytest.raises(reconcile.ReconciliationError, match="modified"): + settlement.apply_plan(w.rt, plan, w.key) + + +def test_plan_is_not_a_receipt_and_running_scheduler_blocks_apply(world): + w = world + _event, coverage = _ready(w) + plan = settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + directory = w.rt.state_dir / "event-settlements" + directory.mkdir() + (directory / f"{reconcile._digest(plan)}.json").write_text(json.dumps(plan)) + with pytest.raises(SignatureError): + settlement.apply_plan(w.rt, plan, w.key) + for path in directory.glob("*.json"): + path.unlink() + with w.rt.ledger.lease(): + with pytest.raises(LeaseError): + settlement.apply_plan(w.rt, plan, w.key) + + +def test_partial_failed_batch_is_refused(world): + w = world + event, coverage = _ready(w) + other = w.rt.ledger.record_event("demo", "merged_pr", "11", {"merge_commit_sha": "b" * 40}) + with w.rt.ledger.lease() as owner: + w.rt.ledger.stage_intake(owner, "demo", "whole-batch", status="failed", verdicts=[], human_reason="", + detail={"event_ids": [event, other]}, drafted_events=[event, other]) + with pytest.raises(ValueError, match="whole existing source batch"): + settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + + +def test_crash_after_receipt_before_commit_retries_without_another_receipt(world, monkeypatch): + w = world + event, coverage = _ready(w) + plan = settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + real_settle = w.rt.ledger.settle_reviewed_events + + def crash(*_args): + raise RuntimeError("simulated process exit before ledger commit") + + monkeypatch.setattr(w.rt.ledger, "settle_reviewed_events", crash) + with pytest.raises(RuntimeError, match="simulated"): + settlement.apply_plan(w.rt, plan, w.key) + assert w.rt.ledger.event(event)["status"] == "pending" + (receipt,) = (w.rt.state_dir / "event-settlements").glob("*.json") + monkeypatch.setattr(w.rt.ledger, "settle_reviewed_events", real_settle) + assert settlement.apply_plan(w.rt, plan, w.key) == receipt + assert w.rt.ledger.event(event)["status"] == "done" + + +def test_active_snapshot_corruption_and_nonexistent_source_event_refuse(world): + w = world + event, coverage = _ready(w) + coverage["events"][0]["id"] = event + 100 + with pytest.raises(ValueError, match="does not exist"): + settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + coverage["events"][0]["id"] = event + plan = settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + page = w.rt.state_dir / "active" / "knowledge" / PAGE + page.write_text(page.read_text() + "\nunauthorized mutation\n") + with pytest.raises(RuntimeError, match="snapshot"): + settlement.apply_plan(w.rt, plan, w.key) + assert w.rt.ledger.event(event)["status"] == "pending" + + +def _historical_ready(w, monkeypatch, *, status="pending"): + event, coverage = _ready(w, status=status) + original = w.rt.ledger.event(event)["payload"] + original.pop("merge_commit_sha") + original["merged_at"] = "2026-10-01T01:02:03Z" + raw = json.dumps(original, indent=2) # retain the exact legacy bytes, not just their canonical digest + w.rt.ledger._conn.execute("UPDATE events SET payload=? WHERE id=?", (raw, event)) + item = coverage["events"][0] + item["source_merge_sha"] = "a" * 40 + repository = w.rt.registry["demo"].full_name + upstream = {"number": 10, "html_url": f"https://github.com/{repository}/pull/10", + "merged": True, "merged_at": original["merged_at"], "merge_commit_sha": "a" * 40, + "head": {"sha": "b" * 40}, "base": {"repo": {"full_name": repository}}} + calls, existing_get = [], w.rt.github.get + + def get(path, **params): + if path == f"/repos/{repository}/pulls/10": + calls.append(path) + return upstream + return existing_get(path, **params) + + monkeypatch.setattr(w.rt.github, "get", get) + return event, coverage, upstream, raw, calls + + +@pytest.mark.parametrize("status", ["pending", "drafted", "rejected"]) +def test_historical_merge_attestation_preserves_payload_and_revalidates_retry(world, monkeypatch, status): + w = world + event, coverage, upstream, raw, calls = _historical_ready(w, monkeypatch, status=status) + plan = settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + planned = verify(settlement.PLAN_PURPOSE, plan, w.key.public_key())["events"][0] + assert planned["source_merge_sha"] == "a" * 40 + assert planned["source_event"]["payload_sha256"] == reconcile._digest(json.loads(raw)) + assert planned["upstream_merge"] == {"repository": w.rt.registry["demo"].full_name, "number": 10, + "url": upstream["html_url"], "merge_sha": "a" * 40, + "head_sha": "b" * 40, "merged_at": upstream["merged_at"]} + receipt = settlement.apply_plan(w.rt, plan, w.key) + assert settlement.apply_plan(w.rt, plan, w.key) == receipt + assert len(calls) == 3 # prepare, apply and retry each obtain fresh upstream evidence + assert w.rt.ledger._conn.execute("SELECT payload FROM events WHERE id=?", (event,)).fetchone()[0] == raw + assert "merge_commit_sha" not in w.rt.ledger.event(event)["payload"] + evidence = verify(settlement.RECEIPT_PURPOSE, json.loads(receipt.read_text()), w.key.public_key())["events"][0] + assert evidence["upstream_merge"] == planned["upstream_merge"] + assert evidence["source_event"]["status"] == status + + +@pytest.mark.parametrize("failure", ["missing_sha", "short_sha", "wrong_sha", "wrong_number", "wrong_repo", + "wrong_url", "unmerged", "missing_head", "bad_time", "wrong_time", + "missing_registry", "invalid_upstream", "malformed_base", "malformed_head"]) +def test_historical_source_needs_exact_configured_live_merge_evidence(world, monkeypatch, failure): + w = world + event, coverage, upstream, raw, _calls = _historical_ready(w, monkeypatch) + if failure == "missing_sha": + coverage["events"][0].pop("source_merge_sha") + elif failure == "short_sha": + coverage["events"][0]["source_merge_sha"] = "a" * 12 + elif failure == "wrong_sha": + upstream["merge_commit_sha"] = "c" * 40 + elif failure == "wrong_number": + upstream["number"] = 11 + elif failure == "wrong_repo": + upstream["base"]["repo"]["full_name"] = "other/demo" + elif failure == "wrong_url": + upstream["html_url"] = "https://github.com/other/demo/pull/10" + elif failure == "unmerged": + upstream["merged"] = False + elif failure == "missing_head": + upstream["head"]["sha"] = "" + elif failure == "bad_time": + upstream["merged_at"] = "2026-10-01" + elif failure == "wrong_time": + upstream["merged_at"] = "2026-10-02T01:02:03Z" + elif failure == "missing_registry": + w.rt.registry.pop("demo") + elif failure == "malformed_base": + upstream["base"] = "other/demo" + elif failure == "malformed_head": + upstream["head"] = "b" * 40 + else: + w.rt.registry["demo"] = replace(w.rt.registry["demo"], full_name="../wrong/demo") + with pytest.raises(reconcile.ReconciliationError): + settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + assert not (w.rt.state_dir / "event-settlements").exists() + assert w.rt.ledger.event(event)["status"] == "pending" + assert w.rt.ledger._conn.execute("SELECT payload FROM events WHERE id=?", (event,)).fetchone()[0] == raw + + +@pytest.mark.parametrize("drift", ["head", "merge", "time", "repository"]) +def test_historical_upstream_drift_refuses_apply_before_receipt(world, monkeypatch, drift): + w = world + event, coverage, upstream, _raw, _calls = _historical_ready(w, monkeypatch) + plan = settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + if drift == "head": + upstream["head"]["sha"] = "c" * 40 + elif drift == "merge": + upstream["merge_commit_sha"] = "c" * 40 + elif drift == "time": + upstream["merged_at"] = "2026-10-02T01:02:03Z" + else: + upstream["base"]["repo"]["full_name"] = "other/demo" + with pytest.raises(reconcile.ReconciliationError): + settlement.apply_plan(w.rt, plan, w.key) + assert not (w.rt.state_dir / "event-settlements").exists() + assert w.rt.ledger.event(event)["status"] == "pending" + + +@pytest.mark.parametrize("expected", ["a" * 40, "c" * 40]) +def test_historical_override_cannot_replace_existing_immutable_merge(world, expected): + w = world + event, coverage = _ready(w) + coverage["events"][0]["source_merge_sha"] = expected + with pytest.raises(reconcile.ReconciliationError): + settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + assert w.rt.ledger.event(event)["payload"]["merge_commit_sha"] == "a" * 40 + + +def test_historical_proof_does_not_relax_whole_batch_selection(world, monkeypatch): + w = world + event, coverage, _upstream, _raw, _calls = _historical_ready(w, monkeypatch) + other = w.rt.ledger.record_event("demo", "merged_pr", "11", {"merge_commit_sha": "b" * 40}) + with w.rt.ledger.lease() as owner: + w.rt.ledger.stage_intake(owner, "demo", "whole-historical-batch", status="failed", verdicts=[], human_reason="", + detail={"event_ids": [event, other]}, drafted_events=[event, other]) + with pytest.raises(ValueError, match="whole existing source batch"): + settlement.make_plan(w.rt, w.key, coverage=coverage, **w.options) + assert w.rt.ledger.event(other)["status"] == "drafted" diff --git a/test/test_knowledge_ops_l1.py b/test/test_knowledge_ops_l1.py index 17c0914c1..ba84babd6 100644 --- a/test/test_knowledge_ops_l1.py +++ b/test/test_knowledge_ops_l1.py @@ -234,7 +234,9 @@ def test_real_tree_operations_pass_both_validators(tmp_path): Op("replace", page, first, _rule("SCHEDKB-1b", cite="PR #8203"), new_rule_id="SCHEDKB-1b", new_page=new_page, evidence="PR #8203"), ] - result = apply_operations(files, ops, release="v0.31.0", today=TODAY) + # The shipped tree advances independently of the synthetic fixtures. + today = max(TODAY, Page.parse(files[page]).frontmatter_field("updated")) + result = apply_operations(files, ops, release="v0.31.0", today=today) head = {**files, **result.files} # active pages still citing the retired IDs are flagged for the change set # to fix; the point here is that both validators accept the written tree diff --git a/test/test_sdk_knowledge_v1.py b/test/test_sdk_knowledge_v1.py index 27a0aa92e..9ed00fb56 100644 --- a/test/test_sdk_knowledge_v1.py +++ b/test/test_sdk_knowledge_v1.py @@ -139,6 +139,34 @@ def test_catalog_is_repo_scoped_sorted_and_path_free(workspace): assert all(not Path(item).is_absolute() for item in all_pages) +def test_reviewed_rule_evidence_is_active_source_bound_and_byte_exact(): + import hashlib + + proof = KnowledgeCurator.reviewed_rule_evidence(PAGE_TEXT, rule_id="X-1", source_reference="PR #1") + section = PAGE_TEXT[PAGE_TEXT.index("## X-1"):] + assert proof["section_sha256"] == hashlib.sha256(section.encode()).hexdigest() + for text, rule, source in [(PAGE_TEXT, "X-0", "PR #1"), (PAGE_TEXT, "X-1", "PR #2"), + (PAGE_TEXT + "\n", "X-1", "PR #1")]: + with pytest.raises(KnowledgeCurationError): + KnowledgeCurator.reviewed_rule_evidence(text, rule_id=rule, source_reference=source) + + +@pytest.mark.parametrize("path", ["/knowledge/repos/x/rules.md", "knowledge/repos/../x/rules.md", "knowledge/tools/check.py"]) +def test_installed_knowledge_file_rejects_noncanonical_paths(path): + with pytest.raises(InvalidRequestError): + KnowledgeCurator.installed_knowledge_file(path) + + +def test_installed_knowledge_file_uses_package_resources_without_workspace_overrides(monkeypatch, tmp_path): + import importlib.resources + + page = tmp_path / PAGE + page.parent.mkdir(parents=True) + page.write_text(PAGE_TEXT) + monkeypatch.setattr(importlib.resources, "files", lambda package: tmp_path) + assert KnowledgeCurator.installed_knowledge_file(PAGE) == PAGE_TEXT.encode() + + def _non_empty(text: str) -> int: return sum(1 for line in text.splitlines() if line.strip())