diff --git a/docs/drama3-ascend-910b1.md b/docs/drama3-ascend-910b1.md new file mode 100644 index 000000000..b5531a2a0 --- /dev/null +++ b/docs/drama3-ascend-910b1.md @@ -0,0 +1,97 @@ +# Drama-3 (fish_omni) on two Ascend 910B1 + +Status: **INCONCLUSIVE** for the requested RTF targets. The decode FULL-graph +change is author-reported, including a second measurement by the same author. +The RTF < 0.1 (C1) and RTF < 1 (C64) bars are not met on this hardware. +These figures are not independently reproducible validation until exact +revision and evidence references are retained. + +## Provenance + +Measurements below are author-reported by amy-why-3459 on a shared 8x910B1 +host (cards 3/4), 2026-10-23/24. They are not independently reproducible +validation until exact revision and evidence references are retained. +`plan/plan.json` stores `` and +`` as placeholders, not pinned SHAs. +The upstream change is described as vLLM-Omni work "Replay Drama 3 decode +steps as FULL graphs" on the mutable branch +`perf/ascend-drama3-full-decode-graphs` (described as based on `c829af4`). +That branch name is not an immutable candidate revision. The Drama-3 NPU +runner path it builds on is +[vllm-omni#9106](https://github.com/vllm-project/vllm-omni/pull/9106) (merged). +Raw logs and the sweep JSON are external. They were retained under +`/evidence/` and `/runs/` and are not in +this repository, so they are unavailable for review from this checkout. +This document stores the summary, not those artifacts. A field marking the +summary locally verified would not make those external artifacts reviewable. + +## RunSpec + +| Field | Value | +| --- | --- | +| repo | vllm-project/vllm-omni, author-described as `c829af4` plus an unpinned FULL-graph change on mutable branch `perf/ascend-drama3-full-decode-graphs` | +| backend | vLLM-Omni 0.32.0 nightly image `quay.io/ascend/vllm-omni:nightly-20261022-6b35bca`, vLLM-Ascend 0.19.1rc2, CANN 9.1.0, torch_npu 2.10.0.post4 | +| device | 2 x Ascend 910B1 (A2), stage 0 TP2 + expert parallel (devices 0,1), stage 1 DAC on device 1 | +| model | `fishaudio/drama-3` (Qwen3.5 MoE 40 layers, 256 experts / 8 active, block-FP8; 13-layer codebook tower; 44.1 kHz DAC) | +| scenario | streaming `/v1/audio/speech`, Seed-TTS-Eval `en`, one fixed pre-encoded reference (85 frames x 10 codebooks), `max_tokens=500`, 64 prompts, 1 warmup | + +Metrics: audio seconds counted from SSE PCM14 payloads at 44.1 kHz mono; +`frames/s = audio_s x 21.5 / wall`; per-request `RTF = (end - start) / audio_s` +so queueing and prefill are inside the number, as the target definition asks. + +## Kept + +| Change | C1 frames/s | C1 RTF mean | C1 RTF p95 | C1 TTFA p50 | Source | +| --- | ---: | ---: | ---: | ---: | --- | +| baseline: stage-0 PIECEWISE capture | 17.31 | 1.237 | 1.301 | 0.91 s | upstream `fish_omni.yaml` | +| decode `FULL_AND_PIECEWISE`, sizes to `max_num_seqs` | 22.4 | 0.962 | 1.141 | 0.80 s | this work, run 1 | +| same, repeat | 22.91 | 0.947 | 1.036 | 0.78 s | this work, run 2 | +| same, final sweep (host loaded) | 18.77 | 1.147 | 1.353 | 0.89 s | `runs/final/final-v2.json` | + +The host is shared (load average 15–20 during runs); the two clean repeats +are the attribution, the loaded sweep is the conservative bound. The gain is +one graph replay per decode step instead of ~53 layers of eager launches. + +Batching on the same build, best of repeats: +16 streams 133 frames/s, 32 streams 137, 64 streams 305 (best of three: +137 / 197 / 188, one later run 305); final sweep under load: 65 / 137 / 269. +Spreading inside the sweep is large enough that single readings of C8–C32 +should not be compared across tables. + +## Rejected + +| Change | Result | Why it stays out | +| --- | --- | --- | +| bind `fla_npu.ops.ascendc.causal_conv1d_update` into vLLM's GDN update | C1 22.4–22.6 frames/s, RTF 0.95 — identical to the unbound build; FULL capture succeeds with the portable `causal_conv1d_update` bound | the vLLM-Ascend warning claims the fallback stalls decode-FULL capture, but on this stack it does not; no measurable effect | +| keep the checkpoint's block-FP8 experts as FP8 | not attempted as a change: vLLM-Ascend builds `AscendW8A8MXFP8Dynamic*` only on 950 (`is_950()`), so A2 expands every expert to bf16 at load | would need a native A2 block-FP8 grouped GEMM; out of scope here | +| `mamba_ssm_cache_dtype: bfloat16` | already the default in this deploy config | not a lever | + +## Where the time goes + +Phase timers in `fish_omni_model.forward` (eager, one stream, one request): +backbone ~195 ms, tower + sampling ~72 ms, writeback ~0.1 ms per frame. With +FULL graphs the same step is ~44 ms, so the graph removes most of the launch +overhead, not the arithmetic. + +The C1 RTF < 0.1 target was not reached in this measured configuration. +The kept rows above are the recorded outcome. Resident expert-weight +capacity does not establish that the target's 4.65 ms/frame wall budget +(0.1 s / 21.5 frames) is physically impossible: 8 of 256 experts are +active, so that resident footprint is not the bytes read per generated +frame. No bandwidth lower bound is claimed. Retaining one would require +active-byte traffic, the applicable bandwidth, and an explicit calculation, +which this record does not contain. + +## Operating notes learned the hard way + +- CANN 9.1.0 does not ship `aclnnAddRmsNormBias`; vLLM-Ascend's custom-op + vendor package must be registered (`${ASCEND_OPP_PATH}/vendors/config.ini` + `load_priority`, plus readable directory modes). A `750 root:root` vendor + directory makes `dlopen` fail with "Permission denied" and surfaces as a + missing-operator error. +- `fla_npu`'s `.pth` rewrites `ASCEND_CUSTOM_OPP_PATH` at interpreter start; + set `FLA_NPU_DISABLE_PTH=1` when a specific vendor list is required. +- `/health` answers before stage 0 finishes warming. A readiness probe must + issue one real synthesis request, or a sweep starts against a dead port. +- Stage 1 (DAC) initialises after stage 0; the first request pays its engine + startup unless the probe absorbs it. diff --git a/evidence/drama3-ascend-20261023/README.md b/evidence/drama3-ascend-20261023/README.md new file mode 100644 index 000000000..41144aed7 --- /dev/null +++ b/evidence/drama3-ascend-20261023/README.md @@ -0,0 +1,16 @@ +# Drama-3 Ascend campaign (2026-10-23/24) + +Verdict: **INCONCLUSIVE**. One optimization is author-reported and kept (decode +FULL graphs), two candidate changes were rejected. RTF targets are recorded +as unmet. The numbers are author-reported, not independently reproducible +validation: plan revisions are placeholders, the patch names a mutable +branch, and raw logs and benchmark outputs are external and unavailable in +this checkout. Registered in `tools/evidence_inventory.json` as a campaign +bundle with no freeze manifest; unavailable external artifacts are listed +there, and no freeze hashes were added. Details, RunSpec and operating notes: +[docs/drama3-ascend-910b1.md](../../docs/drama3-ascend-910b1.md). + +- [plan/plan.json](plan/plan.json) — recorded protocol and success/stop rules; revision fields are placeholders +- [results/summary.json](results/summary.json) — author-reported kept/rejected numbers +- [patch/README.md](patch/README.md) — mutable branch pointer; the patch is not vendored +- [validator/README.md](validator/README.md) — re-run notes; raw benchmark outputs are external diff --git a/evidence/drama3-ascend-20261023/patch/README.md b/evidence/drama3-ascend-20261023/patch/README.md new file mode 100644 index 000000000..8af2ca26b --- /dev/null +++ b/evidence/drama3-ascend-20261023/patch/README.md @@ -0,0 +1,10 @@ +# Patch + +Candidate pointer: mutable branch `perf/ascend-drama3-full-decode-graphs` +(`[Perf][Ascend] Replay Drama 3 decode steps as FULL graphs`), described as +based on `c829af4`. Two files are named: `vllm_omni/deploy/fish_omni.yaml` +and a CPU test in +`tests/model_executor/models/fish_omni/test_fish_omni_config.py`. + +The branch name is mutable and is not an immutable revision. The patch is +not vendored here, and no exact candidate SHA is retained in this bundle. diff --git a/evidence/drama3-ascend-20261023/plan/plan.json b/evidence/drama3-ascend-20261023/plan/plan.json new file mode 100644 index 000000000..c1232d86b --- /dev/null +++ b/evidence/drama3-ascend-20261023/plan/plan.json @@ -0,0 +1,16 @@ +{ + "purpose": "Decode-graph optimization campaign for Drama-3 on Ascend 910B1; RTF targets are recorded, not claimed", + "hypothesis": "Drama-3 stage 0 samples one whole frame per stream per step, so decode is launch-bound; replaying decode as a single FULL ACL graph should cut single-stream RTF and raise frames/s without changing outputs", + "isolated_variable": "stage-0 NPU compilation_config.cudagraph_mode: PIECEWISE vs FULL_AND_PIECEWISE", + "control_revision": "", + "candidate_revision": "", + "revision_status": "control_revision and candidate_revision are placeholders. Exact baseline and candidate SHAs are not retained in this bundle.", + "engine_revision": "vLLM-Omni 0.32.0 nightly 2026-10-22 image; vLLM-Ascend 0.19.1rc2; CANN 9.1.0", + "devices": "2x Ascend 910B1 (A2), stage 0 TP2 + EP on devices 0,1; DAC stage 1 on device 1", + "protocol": "fishaudio/drama-3, streaming /v1/audio/speech, Seed-TTS-Eval en, one fixed pre-encoded reference (85 frames x 10 codebooks), max_tokens=500, 64 prompts, 1 warmup, concurrency 1..64", + "metrics": "frames/s = audio_s x 21.5 / wall; RTF = (end - start) / audio_s including queueing and prefill; TTFA from first SSE audio event", + "targets": {"rtf_mean_c1_lt": 0.1, "rtf_mean_c64_lt": 1.0}, + "success": "C1 RTF mean improves by at least 20% with no request failures and no change to sampling semantics", + "stop": "Stop on stage-0 readiness failure, request failures above 1 percent, or host load above twice the quiet baseline", + "notes": "Host is shared (load average 15-20); single readings of C8-C32 are not comparable across tables. Only the FULL-graph change is in the candidate; the causal_conv1d_update binding was tested and rejected." +} diff --git a/evidence/drama3-ascend-20261023/results/summary.json b/evidence/drama3-ascend-20261023/results/summary.json new file mode 100644 index 000000000..7c88b61e6 --- /dev/null +++ b/evidence/drama3-ascend-20261023/results/summary.json @@ -0,0 +1,19 @@ +{ + "campaign": "drama3-ascend-20261023", + "verdict": "INCONCLUSIVE", + "verdict_reason": "Targets unmet (C1 RTF 0.95-1.15 vs <0.1; C64 RTF 2-5 vs <1); the kept change is measured but the acceptance bars for this campaign are not.", + "kept_change": { + "name": "decode FULL graphs for Drama-3 stage 0 on Ascend", + "c1_frames_per_s": {"control": 17.31, "candidate": 22.4, "repeat": 22.91, "loaded": 18.77}, + "c1_rtf_mean": {"control": 1.237, "candidate": 0.962, "repeat": 0.947, "loaded": 1.147}, + "c1_ttfa_p50_s": {"control": 0.91, "candidate": 0.80, "repeat": 0.78, "loaded": 0.89}, + "batching": {"c16_best": 133, "c32_best": 137, "c64_best": 305, "c64_repeats": [137, 197, 188]} + }, + "rejected": [ + {"change": "bind fla_npu causal_conv1d_update into the GDN decode path", "result": "no measurable effect; FULL capture succeeds with the portable update", "evidence": "serve-full-without-conv1d-control.log"}, + {"change": "native block-FP8 experts on A2", "result": "not available: MXFP8 method is 950-only; A2 expands experts to bf16", "evidence": "vllm-ascend fp8_block.py is_950() branch"} + ], + "measurement_status": "author-reported; not independently reproducible validation until exact revision and evidence references are retained", + "artifacts_retained_at": "/evidence and /runs", + "artifacts_availability": "external; unavailable in this checkout" +} diff --git a/evidence/drama3-ascend-20261023/validator/README.md b/evidence/drama3-ascend-20261023/validator/README.md new file mode 100644 index 000000000..d04ceed93 --- /dev/null +++ b/evidence/drama3-ascend-20261023/validator/README.md @@ -0,0 +1,13 @@ +# Validator + +Re-run: start the service with the candidate deploy config on two 910B1 cards, +then run the streaming benchmark (`bench/bench_stream.py` in the campaign +workspace) over concurrency 1, 2, 4, 8, 16, 32, 64 with 64 prompts, 1 warmup. + +Pass requires: no request failures, C1 RTF mean at least 20% below the control +build measured the same day, and identical sampling parameters. Measure the +control and candidate back to back; this host is shared and single readings +move by tens of percent. + +The benchmark script and raw outputs are external and are not in this +checkout, so these notes are not an independently replayable validator. diff --git a/tests/test_evidence_inventory.py b/tests/test_evidence_inventory.py index 9edd30863..4efa342b2 100644 --- a/tests/test_evidence_inventory.py +++ b/tests/test_evidence_inventory.py @@ -145,6 +145,6 @@ def test_checked_in_inventory(): repo = Path(__file__).resolve().parents[1] result = validate_inventory(repo) assert result["errors"] == [] - assert len(result["bundles"]) == 21 + assert len(result["bundles"]) == 22 assert result["checked_hashes"] == 430 assert result["unavailable"] # Historical raw artifacts are not in this checkout. diff --git a/tools/evidence_inventory.json b/tools/evidence_inventory.json index 6ca2cb388..6fc71a38f 100644 --- a/tools/evidence_inventory.json +++ b/tools/evidence_inventory.json @@ -105,6 +105,13 @@ ], "external": ["Owner-approved 8K search protocol activated prospectively in Issue67; campaign remains INCONCLUSIVE. Private source wrappers, operational logs, model outputs and runtime identities remain in controlled task storage."] }, + "drama3-ascend-20261023": { + "kind": "campaign", + "manifests": [], + "external": [ + "Author-reported summary only. Raw logs, benchmark/sweep outputs, and the candidate patch are external and unavailable in this checkout. No freeze manifest is recorded, and none is invented." + ] + }, "drama3-b300-scale150": { "kind": "campaign", "manifests": [{"path": "plan/freeze.json", "schema": "bundle-sha256"}],