test_record, test_record_regex, test_record_all are flaky

[ivanpauno]

Description

** clear and concise description of the bug is **

The three mentioned tests sometimes fail in CI jobs:

• https://ci.ros2.org/view/nightly/job/nightly_osx_repeated/2312/#showFailuresLink

Expected Behavior

** clear and concise description of what you expected to happen **

Shouldn't fail.

Actual Behavior

** clear and concise description of what actually happened. include copied logs whenever possible **

The sometimes fail, particularly in the "repeated" jobs.

System (please complete the following information)

• OS: macOS, might affect other OSs as well
• ROS 2 Distro: Rolling
• Version: 453281c

Additional context

** Add any other context about the problem here **

[emersonknapp]

Yeah - these tests have been a real problem. I'm hoping that some of the refactoring work @Karsten1987 is doing will make it a lot easier to construct these tests in a reliable way. We've got a few similar (but not identical) tickets in the backlog - I'd like to try and make a "code/test quality burndown board" to focus on when we feature-freeze rosbag2 on May 1.

[tfoote]

Also linux with cyclonedds_cpp is now failing more often than passing on repeated: https://ci.ros2.org/job/nightly_linux-aarch64_repeated/1593/testReport/(root)/projectroot/test_record__rmw_cyclonedds_cpp/history/

Also related #766

[tfoote]

More instances here:

• http://build.ros2.org/job/Fci__nightly-release_ubuntu_focal_amd64/295/testReport/

Also potentially test_play

• http://build.ros2.org/job/Rci__nightly-debug_ubuntu_focal_amd64/395/testReport/
• https://ci.ros2.org/job/nightly_linux_repeated/2363/testReport/junit/(root)/projectroot/test_play__rmw_fastrtps_cpp/

[sloretz]

Also potentially test_play

Another example of test_play being flaky on Ubuntu Focal with FastDDS

The test timed out while running RosBag2PlayTestFixture.recorded_messages_are_played_for_filtered_topics

projectroot.test_play__rmw_fastrtps_cpp
rosbag2_transport.test_play__rmw_fastrtps_cpp.gtest.missing_result

Console output

      Start 29: test_play__rmw_fastrtps_cpp

29: Test command: /home/jenkins-agent/workspace/nightly_linux_repeated/venv/bin/python3.8 "-u" "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/rosbag2_transport/test_results/rosbag2_transport/test_play__rmw_fastrtps_cpp.gtest.xml" "--package-name" "rosbag2_transport" "--output-file" "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/rosbag2_transport/ament_cmake_gmock/test_play__rmw_fastrtps_cpp.txt" "--env" "RMW_IMPLEMENTATION=rmw_fastrtps_cpp" "--command" "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/rosbag2_transport/test_play__rmw_fastrtps_cpp" "--gtest_output=xml:/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/rosbag2_transport/test_results/rosbag2_transport/test_play__rmw_fastrtps_cpp.gtest.xml"
29: Test timeout computed to be: 60
29: -- run_test.py: extra environment variables:
29:  - RMW_IMPLEMENTATION=rmw_fastrtps_cpp
29: -- run_test.py: invoking following command in '/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/rosbag2_transport':
29:  - /home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/rosbag2_transport/test_play__rmw_fastrtps_cpp --gtest_output=xml:/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/rosbag2_transport/test_results/rosbag2_transport/test_play__rmw_fastrtps_cpp.gtest.xml
29: Running main() from gmock_main.cc
29: [==========] Running 9 tests from 2 test suites.
29: [----------] Global test environment set-up.
29: [----------] 4 tests from RosBag2PlayTestFixture
29: [ RUN      ] RosBag2PlayTestFixture.recorded_messages_are_played_for_all_topics
29: [       OK ] RosBag2PlayTestFixture.recorded_messages_are_played_for_all_topics (982 ms)
29: [ RUN      ] RosBag2PlayTestFixture.recorded_messages_are_played_for_all_topics_with_unknown_type
29: [WARN] [1632150895.972629901] [rosbag2_player]: Ignoring a topic 'topic3', reason: package 'unknown_msgs' not found, searching: [/home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosbag2_transport, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosbag2_compression, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosbag2_cpp, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosbag2_storage_default_plugins, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosbag2_storage, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosbag2_test_common, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rclcpp, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/libstatistics_collector, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rcl, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rmw_implementation, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rmw_fastrtps_cpp, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/test_msgs, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rmw_connextddsmicro, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rmw_connextdds, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/std_msgs, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/statistics_msgs, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosgraph_msgs, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosbag2_interfaces, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rmw_fastrtps_shared_cpp, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rmw_cyclonedds_cpp, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rmw_connextdds_common, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rcl_interfaces, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/action_msgs, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/unique_identifier_msgs, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rmw_dds_common, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/builtin_interfaces, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_default_runtime, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_typesupport_cpp, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_generator_py, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_typesupport_c, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_typesupport_fastrtps_c, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_typesupport_introspection_cpp, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_typesupport_fastrtps_cpp, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rcl_yaml_param_parser, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rcl_logging_spdlog, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rcl_logging_noop, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/pluginlib, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_typesupport_introspection_c, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_generator_c, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rmw, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rcl_logging_interface, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/libyaml_vendor, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/class_loader, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_runtime_c, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rcpputils, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_cmake, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rcutils, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/tracetools, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_runtime_cpp, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_parser, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/console_bridge_vendor, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/spdlog_vendor, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rti_connext_dds_cmake_module, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_typesupport_interface, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_adapter, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rmw_implementation_cmake, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/python_cmake_module, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/fastrtps_cmake_module, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_index_cpp, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_ros, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_lint_common, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_xmllint, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_uncrustify, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_pep257, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_flake8, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_cpplint, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_cppcheck, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_uncrustify, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_copyright, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/yaml_cpp_vendor, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/uncrustify_vendor, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/tinyxml2_vendor, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/sqlite3_vendor, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/shared_queues_vendor, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rpyutils, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rosidl_cli, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_lint_cmake, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_gmock, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/domain_coordinator, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_xmllint, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_lint_cmake, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_lint_auto, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_index_python, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cpplint, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_pytest, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_gtest, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_copyright, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_test, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_target_dependencies, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_export_targets, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_export_interfaces, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_export_dependencies, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_pep257, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_version, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_python, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_libraries, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_include_directories, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_export_link_flags, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_export_libraries, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_export_include_directories, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_export_definitions, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_flake8, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_core, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_package, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_lint, /home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cppcheck].
29: [       OK ] RosBag2PlayTestFixture.recorded_messages_are_played_for_all_topics_with_unknown_type (972 ms)
29: [ RUN      ] RosBag2PlayTestFixture.recorded_messages_are_played_for_filtered_topics
    Test #29: test_play__rmw_fastrtps_cpp ........................***Timeout  60.01 sec

[Blast545]

Another instance here: https://build.ros2.org/view/Gci/job/Gci__nightly-release_ubuntu_focal_amd64/54/testReport/(root)/projectroot/test_record__rmw_cyclonedds_cpp/

[clalancette]

#988 should improve the situation, I believe.

[MichaelOrlov]

• Closing as stale issue.