Skip to content

Fix rtl string qa NULL-deref crashes (test bug + source hardening)#486

Open
leginee wants to merge 2 commits into
trunkfrom
Fix-rtl-string-qa-NULL-deref-crashes-(test-bug-+-source-hardening)
Open

Fix rtl string qa NULL-deref crashes (test bug + source hardening)#486
leginee wants to merge 2 commits into
trunkfrom
Fix-rtl-string-qa-NULL-deref-crashes-(test-bug-+-source-hardening)

Conversation

@leginee

@leginee leginee commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

This PR is the first out of 3 fixes from test migration. The original test did not make sense to me. I added the test for NULL pointers in the code itself as guards. And turned the tests into boundry tests.
The code is ai generated. best is to have more eyes to check it.

commit message for conveniance

Error case 1 of the test migration: ~40 *_000 cases in the rtl string qa suites passed NULL into C string functions that dereference it (e.g. rtl_str_compare(NULL, NULL)), causing 0xC0000005 AVs. These were dormant under the old dmake build and only surface now that the tests actually run. NULL violates the functions' documented contract ("must be null-terminated"), so the defect was in the tests, not the (correct, unchanged) source. Fixed both sides for defense in depth.

Tests (qa/rtl/ostring/rtl_str.cxx, rtl_string.cxx, qa/rtl/oustring/rtl_ustr.cxx):

  • Rewrote the UB NULL-deref cases as contract-respecting boundary tests (empty string, prefix/ordering-sign < 0 / > 0), which also closes a previously-untested coverage gap (result sign was never asserted).
  • Added real assertions to the safe NULL-with-length-0 cases, which document the length-bounded functions' tolerance as a regression guard.

Source (rtl/source/strtmpl.c, ustring.c):

  • Added entry-point NULL guards: OSL_PRECOND (diagnoses misuse loudly in non-product builds, compiles out in product builds) plus a defined empty-string fallback so the library never dereferences NULL. Guards sit at function entry, outside the per-character loops, so string-processing throughput is unchanged.
  • strtmpl.c: one edit covers both the sal_Char and sal_Unicode instantiations. getLength is the choke point (guarding it transitively protects hashCode, lastIndexOf*, indexOfStr, trim); compare/compareIgnoreAsciiCase/indexOfChar/replaceChar/ toAscii{Lower,Upper}Case/valueOfChar guarded directly.
  • ustring.c: guarded the 6 mixed UTF-16/ASCII comparison helpers; length-bounded args clamp the length to 0 to avoid NULL+0 pointer arithmetic.

BUILD.bazel: rtl_str/rtl_ustr/rtl_string removed from the "known upstream failures" notes; they now pass.

Verified: sal3.dll rebuilds (both template instantiations) and rtl_str / rtl_ustr / rtl_string all pass.


/* ======================================================================= */
/* NULL-pointer guards */
/* */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove these NULL-pointer guards in release mode for better performance, and only use them in debug builds.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you think their performance impact is that visible?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DamjanJovanovic : switches added.

leginee and others added 2 commits June 21, 2026 07:53
Error case 1 of the test migration: ~40 *_000 cases in the rtl string
qa suites passed NULL into C string functions that dereference it
(e.g. rtl_str_compare(NULL, NULL)), causing 0xC0000005 AVs. These were
dormant under the old dmake build and only surface now that the tests
actually run. NULL violates the functions' documented contract
("must be null-terminated"), so the defect was in the tests, not the
(correct, unchanged) source. Fixed both sides for defense in depth.

Tests (qa/rtl/ostring/rtl_str.cxx, rtl_string.cxx,
qa/rtl/oustring/rtl_ustr.cxx):
- Rewrote the UB NULL-deref cases as contract-respecting boundary
  tests (empty string, prefix/ordering-sign < 0 / > 0), which also
  closes a previously-untested coverage gap (result sign was never
  asserted).
- Added real assertions to the safe NULL-with-length-0 cases, which
  document the length-bounded functions' tolerance as a regression
  guard.

Source (rtl/source/strtmpl.c, ustring.c):
- Added entry-point NULL guards: OSL_PRECOND (diagnoses misuse loudly
  in non-product builds, compiles out in product builds) plus a
  defined empty-string fallback so the library never dereferences
  NULL. Guards sit at function entry, outside the per-character loops,
  so string-processing throughput is unchanged.
- strtmpl.c: one edit covers both the sal_Char and sal_Unicode
  instantiations. getLength is the choke point (guarding it
  transitively protects hashCode, lastIndexOf*, indexOfStr, trim);
  compare/compareIgnoreAsciiCase/indexOfChar/replaceChar/
  toAscii{Lower,Upper}Case/valueOfChar guarded directly.
- ustring.c: guarded the 6 mixed UTF-16/ASCII comparison helpers;
  length-bounded args clamp the length to 0 to avoid NULL+0 pointer
  arithmetic.

BUILD.bazel: rtl_str/rtl_ustr/rtl_string removed from the
"known upstream failures" notes; they now pass.

Verified: sal3.dll rebuilds (both template instantiations) and
rtl_str / rtl_ustr / rtl_string all pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@leginee leginee force-pushed the Fix-rtl-string-qa-NULL-deref-crashes-(test-bug-+-source-hardening) branch from 2c29486 to 8f1ce3d Compare June 21, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants