fix(session): stop RFX geometry from wrapping u16

[Ki Hyun Park (kihyun1998)]

The RemoteFX decoder computed tile rectangles (xIdx * 64) and clip rectangles (x + width - 1) in u16 from values the server sends. A large tile index overflowed, panicking debug builds and wrapping in release builds, where the tile could land at the wrong position. A zero width or height could underflow the same way and widen the clip to the whole channel, or produce an inverted rectangle.

Tile positions are now checked in u16 and clip rectangles computed in u32. A tile that starts past the u16 range is skipped, which draws the same as a tile outside the clip region, and a tile crossing the end of the range is cut there. Clip rectangles use exclusive bounds limited to the channel, so a zero-sized region rectangle or channel clips nothing, and a rectangle entirely outside the channel no longer leaves a one-pixel strip at its edge.

#2095 suggested failing the frame for such a tile; skipping it keeps tiles past the u16 range consistent with other off-screen tiles.

Fixes #2095

[Ki Hyun Park (kihyun1998)]

Validation:

• decode_draws_nothing_for_unreachable_geometry re-encodes the MS-RDPRFX 4.2 example sequence with one field changed: a tile xIdx or yIdx of 1024, a region rectangle width or height of 0, or a channel width or height of 0. Each case asserts that the image stays untouched.
• On master every case fails. Debug builds panic, each at the line computing its own axis (rfx.rs:247, 248, 249, 250, 262, 263); release builds draw, and the image assertion fails.
• Each assertion was seen failing under a targeted mutation: wrapping the tile x or y instead of skipping it, skipping every tile (caught by decode_decodes_valid_sequence_of_messages), dropping either half of the empty-rectangle check, and ignoring the channel height. An off-by-one in the 65536 limit is not reachable through the 64x64 example image, so no test pins that boundary.
• On master, a region rectangle at x = 100 in the 64-pixel channel draws a 64-pixel column at x = 63; with this change it draws nothing.
• The existing inline clipping_rectangles_track_surface_framebuffer_offset still passes when run by name. It does not run in CI because ironrdp-session has [lib] test = false; moving it is left for a separate change.
• This branch merges cleanly with fix(session): reject out-of-range RFX quant indexes #2094, which touches the same two files.
• cargo xtask ci passes locally up to the web lint, which fails here only because this Windows checkout has CRLF line endings; no web files change.

Note

Human-tuned, LLM-assisted content.

[github-actions]

The PR reworks RemoteFX decode geometry in ironrdp-session to compute tile and clip rectangles in u32, drop empty or out-of-channel clipping rectangles, and skip tiles whose rectangle does not fit in u16, eliminating server-controlled u16 wrapping and debug panics. Independent inspection of pr-head confirms the arithmetic, the empty-rectangle filter, the exclusive 65536 clamp, and the new rstest coverage in the CI-run testsuite are correct and well-targeted. All four specialist candidates are valid low-severity issues: a fully visible edge tile near the u16 boundary is dropped instead of clipped, the 65536 clamp constant has no CI-executable test, a frame whose tiles are all skipped still reports the clip extents as the changed rectangle, and the left/top tile arithmetic could be simplified to checked u16 ops. None blocks the change.

Push a commit after addressing these findings. If no code change is needed, you may resolve inline threads and comment @github-actions review-ready to request human review.

[github-actions]

This pull request may overlap with #2094.

Both PRs harden the RemoteFX decoder in ironrdp-session against out-of-range server-supplied values in the same file, though #2094 checks quantization table indexes while this PR fixes tile/clipping rectangle geometry overflow and empty-rectangle handling; shared scope is the same decode path, not the same defect.

This notice is advisory only. Automated review continues as usual, and how these pull requests relate is for maintainers and authors to decide.

Note

LLM-assisted content (no human feedback).

[Ki Hyun Park (kihyun1998)]

On the overlap with #2094: the two fix different defects in the same decoder and are independent. #2094 checks the tile quantization indexes in map_tiles_data; this PR changes tiles_to_rectangles and clipping_rectangles. Their heads (40f2c4db and ded9f7d3) merge without conflicts in either order, so they can land separately.

Note

Human-tuned, LLM-assisted content.