Python action server failing in Bitbucket CI when receiving request from C++ client

[alireza-moayyedi]

Bug report

Required Info:

• Operating System:
  • Custom Docker image using Ubuntu 22.04 and Humble run in Bitbucket Pipelines
• Installation type:
  • Binaries
• Version or commit hash:
  • ros-humble-rmw-fastrtps-cpp Version: 6.2.6-1jammy.20240217.063351
  • ros-humble-fastrtps Version: 2.6.7-1jammy.20240217.034748
  • Basically everything latest as I have recurring docker builds.
• DDS implementation:
  • FastDDS
• Client library (if applicable):
  • rclpy

Steps to reproduce issue

I get random CI faillures when I run my ros2 launch tests in Bitbucket pipelines. But the problem always occurs in the same manner i.e., when my C++ node's action client sends a request and my python node's action server crashes with the following error:

File "/opt/ros/humble/local/lib/python3.10/dist-packages/rclpy/action/server.py", line 312, in _execute_goal_request
     self._handle.send_goal_response(request_header, response_msg)
 rclpy._rclpy_pybind11.RCLError: Failed to send goal response: client will not receive response, at ./src/rmw_response.cpp:154, at ./src/rcl/service.c:314

Running the same docker image on a pc works completely fine. Tried it on many different computers. Only CI for some reason throws me this error unexpectedly. The occurance is completely random as rerunning the same pipeline may or may not succeed. I also noticed that it always fails on the first action request. In other words, it either finishes the pipeline successfully, or it fails right in the beginning throwing the abovementioned error. The behavior is a bit similar to what is described here ros2/ros2#1253 but it seems like that issue has been already addressed and closed.

I also want to add that this problem has surfaced since some weeks ago. I did not have it in the past.

Thanks in advance for your help.

[fujitatomoya]

@alireza-moayyedi i think you hit ros2/ros2#1462

The behavior is a bit similar to what is described here ros2/ros2#1253 but it seems like that issue has been already addressed and closed.

that one is for the service, now you are having the similar problem with action. i will try to fix, but if you are willing to make contribution, i would love to do the review.

[alireza-moayyedi]

@fujitatomoya Hello, thanks for your reply. I made this PR ros2/rclcpp#2463 and started a conversation there. I would be thankful if you could take a look at it.

[fujitatomoya]

@alireza-moayyedi @alsora

can you take a look at,

• add RCL_RET_TIMEOUT to action service response. rcl#1138
• Do not generate the exception when action service response timeout. rclcpp#2464
• update RCL_RET_TIMEOUT error handling with action service response. rclpy#1258

i think these work to fix this issue, and ros2/ros2#1462

[alireza-moayyedi]

@fujitatomoya Great! Thanks a lot!

[fujitatomoya]

closing in favor of ros2/ros2#1462