Skip to content

Fix take() spinning forever on a fragmented change removed by a GAP - #6571

Open
mfinean wants to merge 5 commits into
eProsima:masterfrom
mfinean:fix/gap-orphan-fragmented-change
Open

mfinean wants to merge 5 commits into
eProsima:masterfrom
mfinean:fix/gap-orphan-fragmented-change

Conversation

@mfinean

@mfinean mfinean commented Oct 13, 2026 •

Copy link
Copy Markdown

Description

A non-keyed DataReaderHistory puts a fragmented change into its instance as soon as the first fragment arrives. If a GAP then removes the half-assembled change, it gets released but stays in the instance, and the next take() loops on it forever while holding the reader's mutex. In our ROS 2 Jazzy stack (Fast DDS 2.14.6) this froze processes during bring-up (#6567).

Keyed topics don't hit this because, since #4261, a keyed change only joins its instance in completed_change. The last commit does the same for non-keyed topics, and the first reverts the original change to process_gap_msg, which is no longer needed. That exposed two bugs keyed topics already have, fixed in the two commits in between:

  1. A change created complete (one DATA_FRAG carrying every fragment) is added to its instance twice, and take() never returns.
  2. With KEEP_LAST, a fragmented change that completes after a newer one stays in the history as unread, but take() can't reach it. It's now discarded.

Behaviour change for non-keyed readers, matching keyed: on KEEP_LAST, a partial sample no longer evicts the oldest sample until it's complete.

Tests: DDSDataReader.positioning_gap_removes_partially_received_sample reproduces the hang (take() never returns without the fix). The single_data_frag_with_all_fragments_* and keep_last_discards_sample_completed_after_newer_one_* tests cover the two keyed bugs and fail on master. DataReaderHistory.non_keyed_fragmented_change_added_to_instance_when_completed covers the instance timing.

@Mergifyio backport 3.2.x 2.14.x 2.6.x

2.6.x will need a manual port.

Fixes #6567

Contributor Checklist

  • Commit messages follow the project guidelines.
  • The code follows the style guidelines of this project.
  • Tests that thoroughly check the new feature have been added/Regression tests checking the bug and its fix have been added; the added tests pass locally
  • N/A Any new/modified methods have been properly documented using Doxygen.
  • N/A Any new configuration API has an equivalent XML API (with the corresponding XSD extension)
  • Changes are backport compatible: they do NOT break ABI nor change library core behavior. (No ABI change. The non-keyed KEEP_LAST eviction timing changes, as described above.)
  • Changes are API compatible.
  • N/A New feature has been added to the versions.md file (if applicable).
  • N/A New feature has been documented/Current behavior is correctly described in the documentation.
  • Applicable backports have been included in the description.

Reviewer Checklist

  • The PR has a milestone assigned.
  • The title and description correctly express the PR's purpose.
  • Check contributor checklist is correct.
  • If this is a critical bug fix, backports to the critical-only supported branches have been requested.
  • Check CI results: changes do not issue any warning.
  • Check CI results: failing tests are unrelated with the changes.

@MiguelCompany
MiguelCompany self-requested a review October 13, 2026 12:42
@MiguelCompany MiguelCompany added this to the v3.6.3.0 milestone Oct 14, 2026
@MiguelCompany

Copy link
Copy Markdown
Member

Thank you @mfinean for taking the time to open a PR.

Would you mind rebasing and resolving conflicts?

process_gap_msg removed a partly assembled change with History::remove_change_nts,
which released it but left it in its DataReaderInstance. The next take() then
looped on the released change while holding the reader's mutex.

Remove it through the virtual remove_change_nts, as the HEARTBEAT path does.

Signed-off-by: Mark Finean <33692727+mfinean@users.noreply.github.com>
@mfinean
mfinean force-pushed the fix/gap-orphan-fragmented-change branch from c991ba3 to a435457 Compare October 14, 2026 09:03
@mfinean

mfinean commented Oct 14, 2026

Copy link
Copy Markdown
Author

Rebased onto master. The conflict was with 1008253, which rewrote the GAP handling in process_gap_msg as a single pass over the history. That pass still removes the partly assembled change with History::remove_change_nts, so I applied the same one-line change there. On the rebased branch the regression test fails without it (take() never returns) and passes with it.

@MiguelCompany
MiguelCompany requested review from MiguelCompany and removed request for MiguelCompany October 14, 2026 10:39
MiguelCompany
MiguelCompany previously approved these changes Oct 14, 2026

@MiguelCompany MiguelCompany left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with green CI

@mfinean

mfinean commented Oct 14, 2026

Copy link
Copy Markdown
Author

Thanks for re-running. The four red jobs all fail the same way on master, so I don't think they come from this change:

  • ubuntu-ci fastdds_test: example_tests.benchmark hits the test's 10 s limit on one case per attempt, a different case each time (DEFAULT/NONE, UDPv4/NONE, DEFAULT/MEDIUM). The NONE cases send a single int, so nothing is fragmented. It needed retries on #6564 too (job).
  • windows-ci examples (v142, v143): example_tests.delivery_mechanisms times out, as it does in master's nightly Windows CI (job). The first attempt failed earlier: building the Docker image, Chocolatey returned 406 for OpenSSL 3.1.1.
  • windows-ci unittest-II (v142): SHMTransportTests.memory_bounds times out, also as in the nightly (job).

The new test, DDSDataReader.positioning_gap_removes_partially_received_sample, passes on Ubuntu, both Windows toolsets and under ASan, TSan and UBSan.

@MiguelCompany

Copy link
Copy Markdown
Member

@mfinean Yes, those tests are known to fail from time to time, and are unrelated to the changes in this PR.

There is one thing that bothers me here: the comment we are removing (We call the History overload to avoid callbacks).
It was introduced in #1501, so it scares me that we could introduce some regression in other tools (namely discovery server).

I also think that the behavior should be consistent for keyed and non-keyed topics, so we could go with the other route you proposed in #6567: do not put partially assembled changes into an unkeyed instance until completed_change.

What do you think?

@MiguelCompany

Copy link
Copy Markdown
Member

@mfinean Friendly ping on my previous comment

… when completed

This reverts commit a435457, which removed the partly assembled change through the
virtual remove_change_nts in process_gap_msg. The following commits fix the hang the
other way, by adding a non-keyed fragmented change to its instance only in
completed_change, as keyed changes already are, so process_gap_msg keeps its
History::remove_change_nts call.

Signed-off-by: Mark Finean <33692727+mfinean@users.noreply.github.com>
A DATA_FRAG submessage may carry every fragment of a sample. Fast DDS writers
send one fragment per DATA_FRAG, but other implementations may not. The reader
then creates the change already complete, change_received adds it to its
instance, and completed_change added it again. take() found the change again
after releasing it, and looped on it while holding the reader's mutex.

Call completed_change only for a change completed by a later DATA_FRAG.

Signed-off-by: Mark Finean <33692727+mfinean@users.noreply.github.com>
…AST instance

When a fragmented change completed older than every sample its full KEEP_LAST
instance keeps, completed_change_keep_last returned true without adding it to
the instance. The change stayed in the history, was notified and counted as
unread, but take() could not reach it. It stayed there until the reader was
deleted.

Return false without a rejection reason instead, and have the reader remove
the change and mark it irrelevant, as it already does for a change rejected
by the instances limit.

Signed-off-by: Mark Finean <33692727+mfinean@users.noreply.github.com>
…pleted

A non-keyed DataReaderHistory added a change to its instance with its first
fragment, while a keyed one waits for completed_change. When a GAP covered a
partly assembled change, process_gap_msg released it with
History::remove_change_nts, which left it in the instance. The next take()
looped on the released change while holding the reader's mutex.

Wait for completed_change on non-keyed topics too, as on keyed ones. A partly
received sample then also stops replacing the oldest sample of a KEEP_LAST
instance before it is complete.

Signed-off-by: Mark Finean <33692727+mfinean@users.noreply.github.com>
@mfinean

mfinean commented Nov 8, 2026

Copy link
Copy Markdown
Author

Sorry for the slow reply, and thanks for the pointer. I've looked into both.

On the comment: I don't think removing it would have broken discovery server. All the builtin readers, discovery server's included, use a plain ReaderHistory, and they already remove partly assembled changes through the virtual remove_change_nts on the HEARTBEAT path (remove_fragmented_changes_until). As far as I can tell the comment was added in #1501 to keep the old non-virtual behaviour. It's moot now anyway, since your route doesn't touch process_gap_msg.

On keyed vs non-keyed: agreed, that's the cleaner fix. I've pushed it on top: the first new commit reverts my original change, so process_gap_msg is back as it was, and the last one makes non-keyed changes only go into their instance in completed_change, the same as keyed changes since #4261. The regression test still fails without it and passes with it.

Doing that moved non-keyed topics onto two bugs that keyed topics already have on master, so the two commits in between fix those:

  1. If a single DATA_FRAG carries every fragment, the change is added to its instance twice (in change_received and again in completed_change), and take() never returns.
  2. With KEEP_LAST, a fragmented sample that completes after a newer one has arrived stays in the history as unread, but take() can't reach it.

Both have tests that fail on master for keyed topics. I'm happy to split those two into their own PR if that's easier to review.

One behaviour change for non-keyed KEEP_LAST readers: a partial sample no longer evicts the oldest sample until it's complete. Keyed readers already work this way. The unit tests and the DDS blackbox suite (916 tests, discovery server included) pass locally.

@MiguelCompany
MiguelCompany self-requested a review November 16, 2026 11:57

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GAP for a partially reassembled sample leaves a released change in an unkeyed DataReader's instance; take() then loops forever

2 participants