test(eval): cover functional VitPose keypoint path

[Qiong Wu (qiowu) (DingmaomaoBJTU)]

Summary

Add test-only functional Eval coverage for usyd-community/vitpose-base. The regression uses a deterministic self-generated 24 x 17 RGB fixture and generated COCO-style annotations, and generates a small local ONNX boundary model under the test temporary directory. It exercises the complete offline path:

saved datasets.Image row / PIL RGB decode -> nested COCO person boxes -> local VitPoseImageProcessor -> WinML ORT CPU inference -> pose post-processing -> real COCO OKS metric

This is coverage, not a production bug fix. It changes no evaluator, model adapter, public API, recipe, or README behavior; uses no network at test time; and checks in no model or other large artifact. The 148-byte decoded PNG and its annotations contain no external media or official COCO annotation content. The generated ONNX model validates the runtime boundary only and is not evidence of numerical parity with VitPose. Numerical parity and functional model support are reported separately below from real usyd-community/vitpose-base candidate artifacts.

Historical context:

• feat(keypoint-detection): enable ViTPose config/build/perf #905 merged the ViTPose config/build/perf and class-mapping foundation.
• feat(keypoint-detection): add COCO OKS-AP evaluation #949 merged the keypoint evaluator, COCO dataset builder, OKS metric, and mocked unit coverage.
• recipe(vitpose-base): add CPU keypoint detection recipes #1189 merged the exact CPU fp32/fp16 recipes used here. Those recipes remain unchanged and duplicate recipes are intentionally prohibited.
• fix: add VitPose config-driven dummy inputs #1299 merged config-driven ViTPose dummy inputs.
• fix(eval): resolve VitPose image processor metadata #1310 is an open draft with sibling ViTPose context only; it is not merged evidence for this change.

Model metadata

What the model does

ViTPose Base is a top-down human pose estimator. It consumes processor-normalized RGB person crops at 256 x 192 and produces 17 COCO body-keypoint heatmaps at 64 x 48 for conversion to image-space keypoint coordinates and confidence scores.

Primary user stories

• Supply an RGB image and one or more person bounding boxes to obtain 17 body-keypoint coordinates and confidence scores.
• Feed estimated body joints into downstream pose analysis such as action, fitness-form, animation, or surveillance workflows.

Supported tasks

• keypoint-detection through the checkpoint, Transformers, Optimum ONNX export, and WinML.
• The exact checkpoint uses the ordered COCO 17-keypoint layout expected by the evaluator and metric.

Model architecture

The frozen architecture is a plain ViT backbone with 16 x 16 patch projection, 12 pre-normalized transformer encoder blocks at width 768, and a classic two-stage transposed-convolution decoder:

VitPoseForPoseEstimation
├── VitPoseBackbone
│   ├── Patch projection (RGB 256 x 192, patch 16 x 16 -> width 768 tokens)
│   ├── Learned position embeddings
│   └── Encoder layer x 12
│       ├── LayerNorm -> self-attention (12 heads) -> residual
│       └── LayerNorm -> MLP (768 -> 3072 -> 768, GELU) -> residual
└── VitPoseClassicDecoder
    ├── ConvTranspose 768 -> 256 -> ConvTranspose 256 -> 256
    └── 1 x 1 Conv -> 17 heatmaps [1, 17, 64, 48]

Validation and support evidence

Baseline

Baseline/current origin/main is e28b128f5c2f69ecb2d73b63d2aea0a5ee8bddd0 with WinML 0.3.0. Recipe-free current-main config, build, perf, and a bounded real-data Eval passed before the candidate was produced. The baseline one-row FP32 CPU Eval processed one official COCO val2017 image/person and reported map=0.90, map_50=1.0, map_75=1.0, and mar=0.9; this is a functional smoke, not representative accuracy.

The support gap was automated regression coverage: existing evaluator tests mocked the processor and model rather than materializing a saved image row and traversing the complete processor, ORT, post-processing, and metric path.

Goal

Add focused functional Eval regression coverage for the complete saved COCO row -> RGB PIL decode -> correctly nested person boxes -> local VitPose processor -> ONNX heatmaps -> pose post-processing -> COCO OKS metric path while preserving production behavior and the existing #1189 recipes.

The committed ceiling was L3. The bounded test must prove operability and finite output/count semantics without treating a one-sample result as benchmark quality.

Outcome

Final sealed verdict: PASS with full required tuple coverage. The recorded shipped tier is L2, and the highest goal verdict is L3 PASS.

• L0 PASS: CPU fp32 and fp16 config/build/ONNX validation passed. FP32 has 411 nodes and 359,982,080 bytes of external data. FP16 has 413 nodes, 214 FLOAT16 initializers, FLOAT boundary casts, and 179,986,944 bytes of external data (fp16/fp32 storage ratio 0.4999886216558335).
• L1 PASS: both CPU tuples completed 100 measured iterations after 10 warmups.
• L2 PASS: one identical official COCO processor crop was passed by named pixel_values input to PyTorch and both real ONNX candidate artifacts. FP32: cosine 0.9999999999997321, max abs 6.854534149169922e-07, mean abs 1.3132776034408424e-08. FP16: cosine 0.9999991237385805, max abs 0.0005743205547332764, mean abs 3.4362907409603e-05.
• L3 PASS: the bounded real FP32 CPU Eval described below completed on final candidate ac5c5c107afc112cd56d2ab364227d166bf8acfe.
• Quality gates passed: license insertion, Ruff, mypy across 439 source files, the commands CI partition (3642 passed, 9 skipped, 1 warning), and the focused functional/evaluator/metric set (17 passed).

L0 config/build/structure, L1 perf, Analyze, and the underlying L2 parity run were reused from the pre-repair candidate because the recipes, model/export paths, dependencies, and sealed artifacts were unchanged; they were attested on the final candidate, with L2 receiving a refreshed input attestation. The official one-row FP32 CPU Eval and fixture/tests/static gates were rerun on the final candidate. The local generated ONNX boundary model is not used for L0-L3 parity or support claims.

Model-knowledge findings vitpose-017 through vitpose-021 and a refinement to vitpose-016 were curated locally. Their Lane A publication is pending external repository access and is not a dependency of this PR. The run found no methodology friction, so no _meta finding or skill-contract edit is required.

Per-EP/device/precision results

EP	Device	Precision	Verdict	Mean	p50	p90	p95	p99	Throughput	RAM load	RAM inference	RAM total	VRAM
CPUExecutionProvider	cpu	fp32	PASS	91.65 ms	90.96 ms	98.13 ms	100.13 ms	103.98 ms	10.91 samples/s	+0.7 MB	+26.2 MB	+26.9 MB	N/A
CPUExecutionProvider	cpu	fp16	PASS	144.92 ms	143.96 ms	152.3 ms	154.43 ms	156.77 ms	6.90 samples/s	+0.6 MB	+31.1 MB	+31.6 MB	N/A

Bounded functional smoke Eval:

• Tuple: CPUExecutionProvider / cpu / fp32
• Dataset: one saved official MS COCO val2017 image, one person, one crop, no shuffle, selection seed 42
• Caps: one sample, one ONNX call, 180-second wall-time bound
• Metric: real COCO OKS keypoint AP/AR
• Result: map=0.8999999999999999, map_50=0.9999999999999999, map_75=0.9999999999999999, mar=0.9, num_predictions=1, num_ground_truths=1, num_images=1
• Claim: functional smoke only; not representative accuracy or benchmark quality

Delta

• Add tests/unit/eval/test_keypoint_detection_functional.py (191 lines).
• Add tests/fixtures/generated_pose_24x17.png.b64 (2 lines; 148 decoded bytes), a deterministic self-generated RGB PNG paired with generated COCO-style bbox, area, and keypoints.
• Total: 2 files, 193 insertions.
• The test saves a typed one-row dataset, decodes its image as RGB PIL, preserves [image][person][xywh] box nesting, loads a local VitPoseImageProcessor, generates a three-op ONNX boundary graph under tmp_path, runs WinML ORT on CPU, post-processes 17 finite keypoints/scores, and invokes the real COCO OKS metric. The checked-in boundary regression is separate from the official one-row COCO Eval used for final-SHA model support evidence.
• Review repair: replace the COCO-derived fixture and transformed annotations with fully generated test data, and normalize ONNX usage to the module-qualified onnx.helper and onnx.TensorProto import style.
• No src/, recipe, README, documentation, script, network, checked-in ONNX, or public API change. The existing recipe(vitpose-base): add CPU keypoint detection recipes #1189 fp32/fp16 recipes are unchanged and are not duplicated.

Analyze summary

Both fp32 and fp16 Analyze runs emitted complete JSON but returned exit 1 because accelerator rule matrices retain unknown classifications. This is ANALYZE-PARTIAL-SUCCESS, not a model-validation failure. ConvTranspose remains unknown for NvTensorRTRTXExecutionProvider, QNNExecutionProvider, and OpenVINOExecutionProvider; providers without shipped rule data remain all-unknown.

Component-level summary

Artifact	Architecture regions	Component node counts	Mapped	Unmapped	Explicit gaps
fp32	patch/position embeddings; 12x transformer encoder attention/MLP; classic transposed-convolution decoder	embeddings 5; encoder aggregate 396; attention child 240; MLP child 108; decoder 7	408	3	final backbone LayerNorm, transpose, reshape
fp16	same frozen regions plus graph boundary casts	embeddings 5; encoder aggregate 396; attention child 240; MLP child 108; decoder 7	408	5	input cast, final backbone LayerNorm, transpose, reshape, output cast

Attention and MLP child rows intentionally overlap the encoder aggregate. Decoder mapping uses graph-tail topology and the [1,17,64,48] tensor boundary.

Op-level summary

Artifact	Operators	Types	Dominant counts	Actionable rule gap
fp32	411	13	Add 110; MatMul 96; Reshape 50; Transpose 50; Mul 36	ConvTranspose unknown in the accelerator rule rows above
fp16	413	14	Add 110; MatMul 96; Reshape 50; Transpose 50; Mul 36; Cast 2	ConvTranspose unknown in the accelerator rule rows above

Reproduce commands

$OUT='temp/vitpose-base-repro'
$COCO_ONE='<path-to-saved-one-row-coco-dataset>'
winml build -c examples/recipes/usyd-community_vitpose-base/cpu/cpu/keypoint-detection_fp32_config.json -m usyd-community/vitpose-base -o $OUT/fp32 --ep cpu --device cpu --precision fp32 --no-analyze
winml build -c examples/recipes/usyd-community_vitpose-base/cpu/cpu/keypoint-detection_fp16_config.json -m usyd-community/vitpose-base -o $OUT/fp16 --ep cpu --device cpu --precision fp16 --no-analyze
winml analyze --model $OUT/fp32/model.onnx --ep all --output $OUT/analyze_fp32_all.json
winml analyze --model $OUT/fp16/model.onnx --ep all --output $OUT/analyze_fp16_all.json
winml perf -m $OUT/fp32/model.onnx --device cpu --ep cpu
winml perf -m $OUT/fp16/model.onnx --device cpu --ep cpu
winml eval -m $OUT/fp32/model.onnx --model-id usyd-community/vitpose-base --task keypoint-detection --ep cpu --device cpu --dataset $COCO_ONE --split validation --samples 1 --no-shuffle -o $OUT/eval_fp32_one.json --overwrite --format json

[Qiong Wu (qiowu) (DingmaomaoBJTU)]

REQUEST_CHANGES

PR: #1360
Reviewed SHA: e3824792aa864a2d2db446c6b1c2c48cacce8bb4
Review scope: Independent reviewer verdict for this exact head
Server state: Draft; required label present; 9/9 checks COMPLETED/SUCCESS

Verdict

REQUEST_CHANGES

The functional and quality evidence passes at the reviewed SHA, but two blockers remain. Both must be fixed before this can be approved.

Review Threads

• Total: 1
• Open: 1
• PRRT_kwDORaJc5s6cJpXI — tests/unit/eval/test_keypoint_detection_functional.py:18 — unresolved, non-outdated CodeQL finding for mixed import onnx and from onnx import ... imports.

Successful Evidence

• GitHub checks: 9/9 COMPLETED/SUCCESS.
• Commands CI partition: 3642 passed, 9 skipped, 1 warning.
• Focused functional/evaluator/metric set: 17 passed.
• Static/quality gates:
  • pre_commit run insert-license --all-files: PASS.
  • ruff check src/ tests/: PASS.
  • mypy -p winml.modelkit: PASS in 439 source files.
• Coverage status: full tuple coverage; L0, L1, L2, and L3 all PASS for the committed CPU fp32/fp16 scope. The saved-row functional smoke reached the complete COCO row -> RGB decode -> VitPose processor -> ONNX heatmaps -> pose post-processing -> COCO OKS metric boundary.

Required Producer Fixes

1. Provide image-specific redistribution/license attribution for the checked-in derived COCO image payload. A source URL and annotation ID are insufficient: they identify provenance but do not establish redistribution rights or satisfy image-specific license/attribution obligations. Add durable evidence in the checked-in fixture/test metadata that identifies the image's applicable license and required attribution, or use a self-owned or permissively licensed replacement preserving the functional boundary through RGB decode, preprocessing, ONNX inference, post-processing, and metric evaluation.

2. Resolve the CodeQL import finding. Normalize ONNX imports in tests/unit/eval/test_keypoint_detection_functional.py so the file does not mix import onnx with from onnx import ..., and make the resulting CodeQL check pass.

Follow-up Required

• The explainer must reply and resolve thread PRRT_kwDORaJc5s6cJpXI after the import fix is pushed.
• The tester must refresh evidence for any test and quality results invalidated by the producer changes.
• All checks must rerun on the new SHA and pass; evidence for e3824792aa864a2d2db446c6b1c2c48cacce8bb4 must not be treated as coverage of the replacement head.

[Qiong Wu (qiowu) (DingmaomaoBJTU)]

Addressed both requested changes in ac5c5c1. The checked-in regression now uses a deterministic self-generated 24x17 RGB fixture (148 decoded bytes) and generated bbox, area, and keypoints; no external media or official COCO annotation content remains. ONNX imports are normalized to module-qualified onnx.helper and onnx.TensorProto usage. The replacement tester verdict is PASS with full tuple coverage: the focused keypoint set is 17 passed, the full commands partition is 3642 passed / 9 skipped / 1 warning, license insertion, Ruff, and mypy passed, and the separate official one-row COCO FP32 CPU Eval was rerun on the final SHA with map=0.8999999999999999, map_50=0.9999999999999999, map_75=0.9999999999999999, mar=0.9, and counts 1/1/1.