Large number of consecutive messages queued/delayed when using RMW Fast-RTPS 6.2.2 (with Fast DDS 2.6.3)

[nvcyc]

Bug report

Required Info:

• Operating System:
  • Ubuntu 20.04
• Installation type:
  • from source using rosinstall_generator
• Version or commit hash:
  • (no issue) Fast-RTPS 6.2.1: https://github.com/ros2/rmw_fastrtps/tree/6.2.1
  • (has an issue) Fast-RTPS 6.2.2: https://github.com/ros2/rmw_fastrtps/tree/6.2.2
• DDS implementation:
  • RMW Fast-RTPS (rmw_fastrtps_cpp) 6.2.2
  • Fast DDS (fastrtps) 2.6.3

Steps to reproduce issue

With the RMW Fast-RTPS 6.2.2 (which is the version currently associated with Humble when installed using rosinstall_generator), when continuously publishing messages from a publisher to a subscriber, we observed that intermittently a large number of consecutive messages may be queued and not delivered to the subscriber immediately for unknown reasons at random times

When such a symptom happened, we also observed that some of those queued messages can be successfully delivered to the subscriber (i.e., the subscriber’s callback is called) after stopping publishing messages and starting to publish new messages.

This symptom seems to happen more often when the message publisher rate is high, yet it can still be observed occasionally when the rate is low.

This issue is not observed once we downgrade RMW Fast-RTPS to 6.2.1.

Expected behavior

With RMW Fast-RTPS 6.2.1, messages are delivered as expected.

Actual behavior

With RMW Fast-RTPS 6.2.2, a large number of consecutive messages may be queued and not delivered to their subscriber at random times.

[clalancette]

@MiguelCompany @EduPonz Can you take a look?

[fujitatomoya]

@nvcyc do we have the same problem with rolling?

[hemalshahNV]

We have not tried on Rolling. Is there a version of the fastrtps-rmw this may already be resolved in and would it be compatible with Humble?

[fujitatomoya]

@hemalshahNV

Just thought there could be difference, if it does not happen in rolling, that would be also information track the problem.

(no issue) Fast-RTPS 6.2.1: https://github.com/ros2/rmw_fastrtps/tree/6.2.1
(has an issue) Fast-RTPS 6.2.2: https://github.com/ros2/rmw_fastrtps/tree/6.2.2

just curious, why you guys use these specific branches?

Ubuntu 20.04

that means you use Foxy as Tier I, right? Why not using foxy branch for rmw_fastrtps?

https://github.com/ros2/ros2/blob/3bc9b4cd9a6661f09ae5f271d4f5f4a3b11cccd9/ros2.repos#L286-L289

Fast-RTPS 6.2.2 (which is the version currently associated with Humble

This is incorrect.

• https://github.com/eProsima/Fast-DDS there is not such branch i can find though, isnt that 2.6.2?
• Using Humble with Ubuntu 20.04 is Tier 3 platform, this is only community effort support as described in https://docs.ros.org/en/humble/Installation/Alternatives/Ubuntu-Development-Setup.html#system-requirements

[EduPonz]

This is incorrect.

I guess the OP is referring to rmw_fastrtps 6.2.2; the corresponding Fast DDS version should be v2.6.2. Is that correct @nvcyc? In that case, it'd be great if you could update the ticket description

[Mario-DL]

Yes, I think that @nvcyc was referring to the v6.2.2 version of rmw_fastrtps as @EduPonz was pointing. Fast DDS should be 2.6.2.

Please, check if the issue persists using PR 3195 in Fast DDS

[nvcyc]

Thanks for helping me clarify the versions in discussion. Yes, I was initially referring to the RMW FastRTPS rmw_fastrtps_cpp version 6.2.2 in this repository. The related Fast DDS fastrtps version was 2.6.3 when we observed the issue.

I just tested Fast DDS 2.6.4 that contains the PR @Mario-DL mentioned with RMW FastRTPS 6.2.2 and I no longer observe the frame drop/delay issue.

I'll rebuild our environment with rosinstall_generator (that now pulls fastrtps version 2.6.4-1) and verify again before closing this ticket.

[clalancette]

I'm going to assume that this was fixed by 2.6.4, and close this out. If that is not the case, please feel free to reopen.