recipe(vitpose): add unvalidated QNN NPU candidates - #1419
Qiong Wu (qiowu) (DingmaomaoBJTU) wants to merge 1 commit into
Conversation
|
Independent agent limited review, relayed by the publishing agent. LIMITED_SCOPE_HONESTY_CHECK: NO_FINDINGSIndependent post-publication review under charter revision 2 and the explicit “先发待验证 Draft” exception. This is not model-support approval, Goal acceptance, ordinary workflow approval, a GitHub Review, or permission to merge or mark ready. Reviewed the actual published body and complete diff at head 1fd43e7, base/parent da5dbcd.
At 2026-09-15 17:09 UTC, all 9 visible GitHub checks were successful; the earlier publication snapshot was pending. Neither state validates these model candidates. Before this comment, there were 0 conversation comments, 0 review threads, and 0 GitHub reviews. Link-check limitation: seven linked GitHub files and #1401 resolved through live metadata. The two pinned Hugging Face link patterns and existing source-file hashes match the frozen identity; a live Hugging Face metadata request failed at the network connection, so remote reachability is not claimed. Review verification used read-only Git/GitHub and small text/JSON/XML/hash inspection. No Python/project imports, tests, CLI help, model commands, dependency/provider operations, payload downloads, cleanup, source/recipe edits, or auth changes were performed. No fixes are requested within this limited review; intentional missing model validation is not a finding against the authorized unvalidated Draft. This opinion may be delivered only as a normal comment, never a GitHub Review. |
Summary
This proposes two QNN/NPU recipes for
usyd-community/vitpose-plus-base(keypoint-detection): an optimization-enabled FP32 fallback candidate and a W8A8 candidate with retained historical quality failures. The scope is recipe-only, with no CLI, source, test, or production-index changes. The user-selected revision-2 “先发待验证 Draft” exception permits publication before model acceptance, not acceptance itself. A trustworthy canonical no-optimize baseline remains pending separate #1401; that is not a proven runtime dependency of these optimization-enabled recipe builds.Model metadata
Frozen source-only profile for checkpoint
usyd-community/vitpose-plus-base, historical revision92be54d7a29e42fad47b6e2ca01dd9e685a61e0d, model typevitpose, classVitPoseForPoseEstimation. The four fields below retain their stored order, claims, and confidence labels. These labels concern checkpoint/Transformers source facts, not WinML target support; no ONNX graph was loaded for the profile.What the model does
A top-down RGB image and person-box model that processes normalized person crops through a ViT-style backbone and pose decoder to produce 17-keypoint heatmaps, which the processor maps to image coordinates. The person detector is external and is not part of this model.
Primary user stories
Supported tasks
keypoint-detectioncheckpoint,transformersEvidence: pinned model card, pinned checkpoint configuration, and Transformers v4.57.6 VitPose. These support surfaces are not QNN/NPU validation results.
Model architecture
The model uses a patch-embedded ViT-style encoder with twelve MoE-augmented encoder layers, followed by a final backbone normalization/stage output and a classic deconvolutional pose decoder that produces 17 heatmaps.
Frozen-profile provenance caveat: the outline includes external image/box preprocessing and coordinate postprocessing. Within each encoder layer,
dataset_indexselects the 192-wide expert contribution, combined with 576 shared features to make 768; the six-expert line does not mean concatenating all six expert outputs. MoE is inside the encoder, not the decoder; final backbone normalization follows the encoder stack. This is the original profile's clarification, not new architecture or runtime analysis.Validation and support evidence
Baseline
da5dbcd5812daaabb64b89b47c24db2da2d4e1c7. The tester verified imported WinML source against that local tree with the two candidate additions present. Current committed candidate:1fd43e7abd3914620f090255c0a7546694d9b205, whose parent is the tested source base; the committed recipe bytes match the tester's frozen candidates. Shipment verified local HEAD and the fork branch at that exact candidate SHA. This is Git/byte provenance, not rerun tests, model validation, or validation of newer main. Earlier provenance separately recorded source project version 0.3.1 and installed distribution metadata 0.3.0; those are distinct identifiers, not proof of environment equivalence.optimize=true, fusion settings retained, no authoredskip_optimize); neither FP32 nor W8A8 is the canonical raw/no-optimize baseline.dcad2f4a29cb34af71d7b138bcafc4276a70a4ed, not regenerated or validated on the recorded test base. It omitted/auto(default true), had/optim={}, static uint8-weight/uint16-activation quantization, and/compile=null. That earlier unqualified config probe also triggered QNN catalog acquisition; it was not repeated or represented as side-effect-free evidence.VENDOR+OVERRIDE, exit 0: vendor and post-WinML task lists both containedkeypoint-detection, withadded_by_winml=[]. The vendor exporter wasoptimum.exporters.onnx.model_configs.VitPoseOnnxConfig; the existing WinML override waswinml.modelkit.models.hf.vitpose.VitPoseIOConfig. This is not a new exporter/task implementation or a current model test.cf437687d1c629299df3260a81f02395dc4743cfreported completion in 71.0 s, including 52.7 s export and 17.1 s optimize. Its exact original argv is unrecovered: an Optimize stage is not proof that a no-optimize request was honored. The separate historical CPU smoke is reported below. Current build time, graph/node counts, latency, and task metrics are not established.Goal
Revision 2 explicitly supersedes the provisional revision-1 L3 aspiration for Draft delivery only. Effort is limited to copying two historical per-checkpoint/target configurations with warnings and the portable W8A8 processor identifier: L0-shaped recipe scope, not an accepted support tier. The amended delivery goal is two candidates plus candid dependent-Draft publication and an independent limited scope/honesty review. This PR publishes the committed candidates under that exception; the completed limited review is recorded in Outcome and is not model-support approval or a normal-workflow
APPROVE.validation_goal_status=PENDING_NOT_ACHIEVED;highest_achieved_goal=null. L0, L1, L2, and L3 remain NOT_RUN for all four required QNN/NPU precisions. Schema checks do not establish model build acceptance. The reported free-space figure, 18.4 GiB versus the 25 GiB model-stage minimum, was not re-probed; no model work or cleanup was authorized. Missing model evidence intentionally does not block this Draft-only handoff. Future merge, ready status, or validated-support claims still require their own acceptance evidence; no such gate is waived.Outcome
This Draft proposes commit
1fd43e7abd3914620f090255c0a7546694d9b205from fork/branchDingmaomaoBJTU/winml-cli:dingmaomaobjtu/add-usyd-community-vitpose-plus-base-recipe, targetingmicrosoft/winml-climain. The commit adds exactly two candidate files (+57 / +77, no deletions):Both remain CANDIDATE / UNVALIDATED; coverage is UNVALIDATED, not full or partial accepted support. Deferred tuples are
qnn/npu/fp32,qnn/npu/fp16,qnn/npu/w8a8, andqnn/npu/w8a16. No QNN/NPU FP16 or W8A16 recipe is added. No source, CLI, tracked test, API, generic recipe, or production-index change is included, and nothing is deleted. The generic FP16 recipe, generic W8A8 recipe, and recipes README/index remain unchanged.Learner findings are an internal scratch-only handoff, not appended model knowledge or a completed methodology update. Under the explicit exception, no new Lane A/source/skill/profile changes are included or required for this Draft-only step. Ordinary support approval is not claimed. The separate limited scope/honesty review is recorded below; no GitHub approval, ready-for-review transition, or merge is authorized. GitHub check status is separate from the tester's bounded local checks and does not establish model acceptance.
Publication closure — 2026-09-15 17:27 UTC: head
1fd43e7abd3914620f090255c0a7546694d9b205; independent limited review, transparently relayed: LIMITED_SCOPE_HONESTY_CHECK: NO_FINDINGS. Not model approval: candidates remain UNVALIDATED, current L0–L3 NOT_RUN. Latest API snapshot (2026-09-15 17:25 UTC): 9/9 visible CI checks SUCCESS, not model-support validation.Independent bounded offline checks, as recorded by the tester (not rerun by the explainer):
Actual local environment: CPython 3.11.16, AMD64, 64-bit, existing Transformers 4.57.6 / Hugging Face Hub 0.36.2 compatibility overlay, pytest 9.1.1, pytest-cov 7.1.0, and pytest-timeout 2.4.0. The editor-selected Python 3.14.3 was not used for these imports/tests. No packages were installed and no fresh public environment was certified.
Earlier harness mistakes were corrected scratch-only: recognize Windows NUL before path normalization; compare tuple-backed serializer shapes after actual JSON encoding while retaining strict historical JSON comparisons; isolate only the explicit alias-policy test with a pure-intent resolver mock. No production/candidate/tracked-test fix or skipped failing test was used. Actual candidate loaders, serialization, completeness validation, and discovery were not mocked. The final guarded run recorded zero forbidden attempts; saved preservation checks covered 62 protected text hashes and three repository states, not a recursive model-payload inventory.
Retained native ORT import warning:
Init provider bridge failed.fromCreateOrtEnv. It is not a target-runtime verdict or a pytest warning record; the run is not described as warning-free. No model payload I/O, model/config generation, export, inference, explicit provider discovery/acquisition/registration, network/download, installation, or cleanup work was performed in the bounded tester phase. Its sole CLI invocation was help-only to verify option spelling. The tester did not run full suites or repository-wide static checks and did not verify CI; CI is reported separately in the API snapshot above.Per-EP/device/precision results and Functional smoke Eval
Current results on the recorded local base — no historical acceptance carried forward:
Current per-tuple datasets/revisions, task metrics, runtime artifacts, and memory values are not established. FP16 has no exact QNN/NPU historical attempt; a generic filename proves neither support nor unsupported status. W8A16 retains its historical failure below, with no new recipe, repair/search, or performance ranking.
Historical numeric evidence only
Historical QNN source:
169a6f0122d1e1dd7d60c9605b5f19add26a209b. Historical checkpoint revision:92be54d7a29e42fad47b6e2ca01dd9e685a61e0d. Historical environment metadata records ONNX Runtime Windows ML 1.24.5.202604171637 and QNN EP package 2.2480.53.0. These identify old evidence, not execution of the new portable configurations; model payloads were not reopened or rehashed for these checks.The recorded threshold is 0.99. Full-set W8A8 sample cosines are 0.9957561025512272, 0.9937934557606259, 0.982657340445321; the posthoc core retains only the first two. Neither its higher minimum nor the full-set overall cosine overturns the original full-set/wallpaper failures. These comparisons are not representative W8A8 task accuracy.
Historical paired latency only — no new performance measurement
The recorded protocol was 8 alternating A/B and B/A pairs, 16 sessions, 20 warmups and 100 measured iterations per session, QNN/NPU with burst mode and graph-finalization mode 3. Tester checked the 16 stored child results, each with 100 raw samples; no benchmark was rerun. Here “baseline” means the historical optimization-enabled FP32 comparator, not the pending canonical raw baseline.
Recorded mean paired speedup: 2.5840062576234804x, 95% CI [2.539734340098991, 2.620416130903574]. Recorded mean gain fraction: 0.6127990435719093, 95% CI [0.6059867221376429, 0.6183396084021368]. Recorded p50 paired speedup: 2.631350525801282x, 95% CI [2.6149376770791664, 2.648977705483742]; p50 gain fraction 0.6199336171460872, 95% CI [0.6175667739774706, 0.6224590182454179]. These are stored paired statistics, not a newly computed ratio of rounded means; averages of session p50s are not pooled percentiles. No aggregate throughput is invented from latency.
Historical no-memory scope is retained: no child result has a memory section, and CPU RSS cannot substitute for QNN memory. Latency does not cancel failed W8A8 fidelity or establish correctness-gated support for either portable candidate.
Functional smoke Eval — current FP32 CPU NOT_RUN
There is no current final-artifact FP32 CPU evaluation, current sample/crop-cap verification, or current task metric. The separate historical one-image CPU result is end-to-end operability evidence only, not representative accuracy or W8A8 accuracy. Its source attribution,
cf437687d1c629299df3260a81f02395dc4743cf, comes from the original charter, not an embedded source SHA in the result.The saved COCO17 functional subset used the validation label, no shuffle, seed 42, and one image, one prediction, one ground truth; the result does not record a dataset revision. Raw metrics:
map=0.9999999999999998,map_large=0.9999999999999998,map_50=0.9999999999999999,map_75=0.9999999999999999,map_medium=-1.0,mar=1.0,mar_50=1.0,mar_75=1.0. The medium-size metric has no denominator; it is not negative task accuracy. AP approximately 1 on this one image does not establish general quality.The separate old CPU FP32 performance smoke used 1 warmup, 3 iterations, with mean 180.186 ms, p50 179.868 ms, 5.55 samples/s, and total RSS delta 30.6 MB. None of these values closes current model acceptance or measures QNN memory.
Delta
The tester independently verified complete recursive JSON comparisons: object order/whitespace ignored, array order/scalar types preserved, and absent distinct from null. Only
/_noteis excluded from the semantic counts; its full new values are disclosed below. FP32/W8A8 have 0/1 semantic changes against their respective historical configurations, and 5/7 against the historical auto-config anchor. These are not byte-equality or runtime-equivalence claims.Public-display redaction: the sole historical machine-local processor string is represented below as
"<historical-local-processor>", not printed as an inaccessible absolute path. This is a documentation token, not a literal configuration value or a pinned revision. The tester compared the exact original string; all other displayed values are literal, with every object field retained./_note/quant/model_id"<historical-local-processor>"(redacted string)"usyd-community/vitpose-plus-base"; 1 changeThe following table gives the full anchor delta. “Both” rows apply separately to FP32 and W8A8, yielding exactly five and seven changed pointers respectively. Replacing an object with null is one enclosing-pointer change, not a count of its children.
/autofalse/optim/gelu_fusiontrue/optim/matmul_add_fusiontrue/compilenull{"execution_provider":"qnn","provider_options":{"device_type":"NPU","htp_performance_mode":"burst","htp_graph_finalization_optimization_mode":"3"},"provider_option_file_keys":[],"enable_ep_context":true,"embed_context":false,"compiler":"ort","qnn_sdk_root":null,"device":"npu","ep_device":null,"validate":true}/quant{"mode":"static","samples":10,"calibration_method":"minmax","weight_type":"uint8","activation_type":"uint16","per_channel":false,"symmetric":false,"weight_symmetric":null,"activation_symmetric":null,"save_calibration":false,"distribution":"uniform","seed":null,"calibration_load_path":null,"calibration_save_path":null,"op_types_to_quantize":null,"nodes_to_exclude":null,"task":"keypoint-detection","model_id":"<historical-local-processor>","model_type":"vitpose"}null/quant/activation_type"uint16""uint8"/quant/seednull42/quant/model_id"<historical-local-processor>"(redacted string)"usyd-community/vitpose-plus-base"For W8A8, the full old
/quantobject above is retained except for the three listed scalar replacements; no null/default/exclusion fields are dropped. It remains static minmax, 10 samples, uint8 weights/activations,per_channel=false, andsymmetric=false. Both candidates also retain the following complete historical blocks:/loader{"task":"keypoint-detection","model_class":"VitPoseForPoseEstimation","model_type":"vitpose"}/export{"opset_version":17,"batch_size":1,"export_params":true,"do_constant_folding":true,"verbose":false,"dynamo":false,"enable_hierarchy_tags":true,"clean_onnx":false,"hierarchy_tag_format":"full","input_tensors":[{"name":"pixel_values","dtype":"float32","shape":[1,3,256,192],"value_range":[0,1]}],"output_tensors":[{"name":"heatmaps"}],"compatibility":{"transformers_attention":"eager"}}Additional first-field warnings, excluded from counts but not hidden:
/_notewas absent in both historical configurations and the anchor./_note"UNVALIDATED CANDIDATE - DRAFT_ONLY_UNVALIDATED. Current-main model verification NOT_RUN; historical evidence only, not a model-support claim. #1401 is needed for a trustworthy canonical raw/no-optimize baseline. This optimization-enabled FP32 recipe is a fallback candidate, not that canonical raw baseline. The companion W8A8 candidate failed historical full-three-person and wallpaper parity; its posthoc two-person pass is not general quality evidence. Portable calibration processor/revision binding for W8A8 is not yet verified."/_note"UNVALIDATED CANDIDATE - DRAFT_ONLY_UNVALIDATED. Current-main model verification NOT_RUN; historical evidence only, not a model-support claim. #1401 is needed for a trustworthy canonical raw/no-optimize baseline. Historical W8A8 full-three-person and wallpaper parity FAILED; posthoc two-person PASS does not establish general quality or overturn those failures. Portable calibration processor/revision binding and exact portable-config equivalence are not yet verified."WinMLBuildConfig.from_dict()ignores/_note, serialization omits it, and the alternate loader retains the raw dictionary. The warning is advisory, not an execution barrier. Likewise,compile.validate=truerequests future validation; it is not evidence that compilation/validation ran. No eval block orskip_optimizefield was authored.Intentional discovery change:
qnn/npuandQNNExecutionProvider/NPUlookup selects only the two target candidates. The target directory replaces generic-directory lookup; it does not union in generic FP16. Untargeted and missing QNN/GPU lookup still uses the unchanged generic FP16/W8A8 recipes. Automated consumers can select these unvalidated candidates despite the warning.Calibration and reducibility: historical calibration records real images, not uniform noise inferred from
distribution=uniform. The export dummy range[0,1]is not the actual processor-normalized input range.seed=42proves neither effective seeded row selection nor pinned calibration rows. The portable processor ID is not revision-pinned; immutable processor/revision binding, calibration-row provenance, and exact portable-config runtime equivalence remain UNVALIDATED. No revision field was invented. These are frozen per-checkpoint/target graph, precision, power/latency, and external-context packaging choices—not a family-wide optimum, a default recommendation, or a metadata-derived CLI repair. No code bug fix or recipe-free acceptance is claimed; the no-recipe gate is deferred only for Draft publication. The production recipes README remains unchanged.Analyze summary — component level and op level
Current analysis: NOT_RUN. No current built graph exists for this validation scope; the source-only architecture profile is not component mapping, static operator coverage, or runtime execution evidence.
The unresolved gap is the absence of a current built graph and its analysis. Missing counts are not zero counts; none are generated from the source outline or carried forward from older artifacts.
Reproduce commands
Command provenance: exact historical build/perf/Eval argv is unrecovered; stored command templates and the future commands below are not historical reproductions. Shipment is limited to publication, Git/GitHub metadata, and preservation checks; no tests or model commands were rerun. The following public commands are transcribed without modification from the tester's handoff.
Portable spelling of the completed 60-test repository subset: run from the winml-cli repository root in an already prepared compatible CPython 3.11 environment with repository dependencies, pytest, pytest-cov, and pytest-timeout. The actual tester replay used the existing Transformers 4.57.6 / Hub 0.36.2 overlay; this does not certify a newly provisioned environment. This selection covers only the same 34 config + 26 discovery tests—not the additional 37 guarded checks, eight integrity checks, or any model run. It needs no unpublished harness. This replay was performed by the tester, not repeated for shipment.
Separate future recipe-authoritative builds — NOT_EXECUTED. These are not part of the offline replay or authorization to start model work. Use a separately authorized, resource-ready validation session after reviewing the canonical-baseline dependency. Only option spelling was checked through CLI help; there is no build result. The HF ID in these proposals does not pin the historical checkpoint/processor revision.
Recipe authority is preserved:
-c,-m, and-oonly, with no precision, EP/device, shape, optimization, or calibration rescue overrides and no quantization/compile bypass. Both builds remain optimization-enabled; the FP32 proposal is not the pending canonical no-optimize baseline. No current support, performance, memory, fidelity, or task-accuracy result is implied.