Skip to content

fix(examples): center the C++ letterbox and undo its padding offset - #320

Merged
isLinXu merged 1 commit into
mainfrom
fix/cpp-centered-letterbox
Oct 24, 2026
Merged

isLinXu merged 1 commit into
mainfrom
fix/cpp-centered-letterbox

Conversation

@isLinXu

@isLinXu isLinXu commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator

What this fixes

Preprocess() in examples/cpp/common/yolo_postprocess.hpp built its canvas as a top-left letterbox with a black fill, while Python uses ultralytics.data.augment.LetterBox(center=True), i.e. a centered placement filled with 114 grey. The C++ examples were therefore feeding the model a different picture than model.predict() does, and none of the Postprocess* functions subtracted the padding before dividing by the resize ratio, so boxes came back displaced by pad / scale.

For the usual 1080x810 test frame at imgsz=640 that is an 80 row shift in model space, i.e. 135 px of vertical offset in original-image coordinates. Boxes still looked plausible because both sides applied their own inverse consistently, but the tensor the model saw was wrong — which is exactly how a "boxes are right, confidences drift" report looks (see #318).

Changes

File Change
common/yolo_postprocess.hpp Preprocess() now reproduces LetterBox(center=True): min-side ratio, centered padding split with the same round(pad / 2 - 0.1) / round(pad / 2 + 0.1) rule, 114 grey fill. It reports the transform in a new yolo::Letterbox {scale, pad, unpad}.
common/yolo_postprocess.hpp All Postprocess* take that Letterbox instead of a bare float scale and map back with orig = (model - pad) / scale — boxes, keypoints (pose, incl. per-keypoint x/y) and OBB centers. Segment/semantic crop the padding band before resampling to the original frame.
six backends ONNXRuntime, OpenCV-DNN, OpenVINO, MNN, LibTorch, Triton thread the struct through.
common/README.md Documents the contract.

The pad offset is carried as a struct rather than an extra default argument on purpose: a caller that forgets an optional pad = 0 would silently reproduce this bug, whereas the signature change is caught at compile time.

Verification

No OpenCV in this environment, so I checked the two things that can actually be wrong — geometry and the inverse mapping — against the Python implementation:

  • Geometry parity. For 14 (image size, imgsz in {320, 640}) combinations, including portrait frames (333x777, 500x700) where the padding is on the x axis, scale, unpad and pad match a literal transcription of LetterBox.get_params exactly:

        w     h imgsz |     scale      unpad        pad | reference
     1080   810   640 |  0.592593 (640, 480) (0.0, 80.0) | (0.59259..., (640, 480), (0, 80)) ok
      333   777   640 |  0.823681 (274, 640) (183.0, 0.0) | (0.82368..., (274, 640), (183, 0)) ok
    
  • Round trip. An original box 100,200 -> 300,400 projected into model space and pushed back through the new PostprocessDetect returns 100,199,200,200. The old mapping gives y = 335 for the same input.

  • Padding. Corners of the padded bands are 114,114,114 after the BGR->RGB conversion, and the first content row lands at y = pad.y.

clang++ -std=c++17 -Wall -Wextra is clean on the header. I could not build the backends themselves here (no ORT/OpenVINO/MNN/libtorch installed), so CI or a reviewer with the toolchains should confirm; the changes to them are mechanical (one local variable and the argument in the call sites).

Not in this PR

Two smaller mismatches in the same file, so they are visible rather than silently lumped in:

  • cv::dnn::NMSBoxes is class-agnostic, while Ultralytics validates with per-class NMS (agnostic=False). On frames with several classes this suppresses boxes that Python keeps.
  • Boxes are truncated to cv::Rect (int), and are not clipped to the image bounds the way scale_boxes clips them.

Happy to do either as a follow-up if you want them.

Preprocess() in examples/cpp placed the resized image in the top-left corner of
the canvas and padded with black zeros, while
ultralytics.data.augment.LetterBox centers the image and pads with 114 grey. For
a 1080x810 frame the C++ examples therefore fed the model a picture shifted by
80 rows against what model.predict() sends, and every Postprocess* divided by
the resize ratio without removing that offset first, so the boxes came back
displaced by the same amount.

- Preprocess() now reproduces LetterBox(center=True): min-side ratio, centered
  padding split with the same round(pad / 2 - 0.1) rule, 114 grey fill, and it
  reports the transform in a new yolo::Letterbox {scale, pad, unpad}.
- Every Postprocess* takes that Letterbox and maps model-space coordinates back
  with orig = (model - pad) / scale. Masks and semantic maps crop the padding
  band before they are resampled to the original frame.
- The six backends (ONNXRuntime, OpenCV-DNN, OpenVINO, MNN, LibTorch, Triton)
  thread the struct through instead of a bare `float scale`.
- The contract is documented in examples/cpp/common/README.md.

Verified against Python: for 14 (image size, imgsz in {320, 640}) combinations
the scale, unpadded size and left/top padding match LetterBox.get_params
exactly, and a box round-tripped through the new post-processor returns
100,199,200,200 from an original 100,200,300,400 (1 px of int truncation),
where the old mapping produced y=335.
@isLinXu
isLinXu merged commit 744cbc5 into main Oct 24, 2026
10 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant