Repository navigation
Conversation
|
Thank you @mfinean for taking the time to open a PR. Would you mind rebasing and resolving conflicts? |
process_gap_msg removed a partly assembled change with History::remove_change_nts, which released it but left it in its DataReaderInstance. The next take() then looped on the released change while holding the reader's mutex. Remove it through the virtual remove_change_nts, as the HEARTBEAT path does. Signed-off-by: Mark Finean <33692727+mfinean@users.noreply.github.com>
c991ba3 to
a435457
Compare
|
Rebased onto master. The conflict was with 1008253, which rewrote the GAP handling in |
|
Thanks for re-running. The four red jobs all fail the same way on master, so I don't think they come from this change:
The new test, |
|
@mfinean Yes, those tests are known to fail from time to time, and are unrelated to the changes in this PR. There is one thing that bothers me here: the comment we are removing ( I also think that the behavior should be consistent for keyed and non-keyed topics, so we could go with the other route you proposed in #6567: What do you think? |
… when completed This reverts commit a435457, which removed the partly assembled change through the virtual remove_change_nts in process_gap_msg. The following commits fix the hang the other way, by adding a non-keyed fragmented change to its instance only in completed_change, as keyed changes already are, so process_gap_msg keeps its History::remove_change_nts call. Signed-off-by: Mark Finean <33692727+mfinean@users.noreply.github.com>
A DATA_FRAG submessage may carry every fragment of a sample. Fast DDS writers send one fragment per DATA_FRAG, but other implementations may not. The reader then creates the change already complete, change_received adds it to its instance, and completed_change added it again. take() found the change again after releasing it, and looped on it while holding the reader's mutex. Call completed_change only for a change completed by a later DATA_FRAG. Signed-off-by: Mark Finean <33692727+mfinean@users.noreply.github.com>
…AST instance When a fragmented change completed older than every sample its full KEEP_LAST instance keeps, completed_change_keep_last returned true without adding it to the instance. The change stayed in the history, was notified and counted as unread, but take() could not reach it. It stayed there until the reader was deleted. Return false without a rejection reason instead, and have the reader remove the change and mark it irrelevant, as it already does for a change rejected by the instances limit. Signed-off-by: Mark Finean <33692727+mfinean@users.noreply.github.com>
…pleted A non-keyed DataReaderHistory added a change to its instance with its first fragment, while a keyed one waits for completed_change. When a GAP covered a partly assembled change, process_gap_msg released it with History::remove_change_nts, which left it in the instance. The next take() looped on the released change while holding the reader's mutex. Wait for completed_change on non-keyed topics too, as on keyed ones. A partly received sample then also stops replacing the oldest sample of a KEEP_LAST instance before it is complete. Signed-off-by: Mark Finean <33692727+mfinean@users.noreply.github.com>
|
Sorry for the slow reply, and thanks for the pointer. I've looked into both. On the comment: I don't think removing it would have broken discovery server. All the builtin readers, discovery server's included, use a plain On keyed vs non-keyed: agreed, that's the cleaner fix. I've pushed it on top: the first new commit reverts my original change, so Doing that moved non-keyed topics onto two bugs that keyed topics already have on master, so the two commits in between fix those:
Both have tests that fail on master for keyed topics. I'm happy to split those two into their own PR if that's easier to review. One behaviour change for non-keyed KEEP_LAST readers: a partial sample no longer evicts the oldest sample until it's complete. Keyed readers already work this way. The unit tests and the DDS blackbox suite (916 tests, discovery server included) pass locally. |
Description
A non-keyed
DataReaderHistoryputs a fragmented change into its instance as soon as the first fragment arrives. If a GAP then removes the half-assembled change, it gets released but stays in the instance, and the nexttake()loops on it forever while holding the reader's mutex. In our ROS 2 Jazzy stack (Fast DDS 2.14.6) this froze processes during bring-up (#6567).Keyed topics don't hit this because, since #4261, a keyed change only joins its instance in
completed_change. The last commit does the same for non-keyed topics, and the first reverts the original change toprocess_gap_msg, which is no longer needed. That exposed two bugs keyed topics already have, fixed in the two commits in between:take()never returns.take()can't reach it. It's now discarded.Behaviour change for non-keyed readers, matching keyed: on KEEP_LAST, a partial sample no longer evicts the oldest sample until it's complete.
Tests:
DDSDataReader.positioning_gap_removes_partially_received_samplereproduces the hang (take()never returns without the fix). Thesingle_data_frag_with_all_fragments_*andkeep_last_discards_sample_completed_after_newer_one_*tests cover the two keyed bugs and fail on master.DataReaderHistory.non_keyed_fragmented_change_added_to_instance_when_completedcovers the instance timing.@Mergifyio backport 3.2.x 2.14.x 2.6.x
2.6.x will need a manual port.
Fixes #6567
Contributor Checklist
versions.mdfile (if applicable).Reviewer Checklist