test_services_cpp__rmw_cyclonedds_cpp tests are flakey

[tfoote]

Reviewing the failing tests on CI test_services_cpp__rmw_cyclonedds_cpp is failing regularly on multiple platforms of CI

linux_repeated: https://ci.ros2.org/job/nightly_linux_repeated/1869/testReport/projectroot/test/test_services__rmw_cyclonedds_cpp/history/

linux_release: https://ci.ros2.org/job/nightly_linux_release/1540/testReport/(root)/projectroot/test_services_cpp__rmw_cyclonedds_cpp/history/

Linux aarch64 repeated: https://ci.ros2.org/job/nightly_linux-aarch64_repeated/1190/testReport/projectroot/test/test_services__rmw_cyclonedds_cpp/history/

osx_repeated: https://ci.ros2.org/job/nightly_osx_repeated/1951/testReport/projectroot/test/test_services__rmw_cyclonedds_cpp/history/

windows repeated: https://ci.ros2.org/job/nightly_win_rep/1846/testReport/(root)/projectroot/test_services_cpp__rmw_cyclonedds_cpp/history/

Example stack trace: https://ci.ros2.org/job/nightly_osx_repeated/1951/testReport/projectroot/test/test_services__rmw_cyclonedds_cpp/

-- run_test.py: extra environment variables:
 - RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
-- run_test.py: extra environment variables to append:
 - DYLD_LIBRARY_PATH+=/Users/osrf/jenkins-agent/workspace/nightly_osx_repeated/ws/build/rcl
-- run_test.py: invoking following command in '/Users/osrf/jenkins-agent/workspace/nightly_osx_repeated/ws/build/rcl/test':
 - /Users/osrf/jenkins-agent/workspace/nightly_osx_repeated/venv/bin/python3 -m launch_testing.launch_test /Users/osrf/jenkins-agent/workspace/nightly_osx_repeated/ws/build/rcl/test/test/test_services__rmw_cyclonedds_cpp_.py --junit-xml=/Users/osrf/jenkins-agent/workspace/nightly_osx_repeated/ws/build/rcl/test_results/rcl/test_services__rmw_cyclonedds_cpp.xunit.xml --package-name=rcl
[INFO] [launch]: All log files can be found below /Users/osrf/.ros/log/2020-05-12-03-39-21-129212-mini1-86109
[INFO] [launch]: Default logging verbosity is set to INFO
test_services (rcl.TestTwoExecutables)
Test that the executable under test terminates after a finite amount of time. ... [INFO] [service_fixture-1]: process started with pid [86111]
[INFO] [client_fixture-2]: process started with pid [86112]
[INFO] [client_fixture-2]: sending signal 'SIGINT' to process[client_fixture-2]
[INFO] [service_fixture-1]: sending signal 'SIGINT' to process[service_fixture-1]
[ERROR] [client_fixture-2]: process has died [pid 86112, exit code -2, cmd '/Users/osrf/jenkins-agent/workspace/nightly_osx_repeated/ws/build/rcl/test/client_fixture__rmw_cyclonedds_cpp service_fixture'].
[ERROR] [service_fixture-1]: process has died [pid 86111, exit code -2, cmd '/Users/osrf/jenkins-agent/workspace/nightly_osx_repeated/ws/build/rcl/test/service_fixture__rmw_cyclonedds_cpp'].
FAIL

======================================================================
FAIL: test_services (rcl.TestTwoExecutables)
Test that the executable under test terminates after a finite amount of time.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/osrf/jenkins-agent/workspace/nightly_osx_repeated/ws/build/rcl/test/test/test_services__rmw_cyclonedds_cpp_.py", line 42, in test_services
    proc_info.assertWaitForShutdown(process=executable_under_test, timeout=10)
  File "/Users/osrf/jenkins-agent/workspace/nightly_osx_repeated/ws/install/launch_testing/lib/python3.8/site-packages/launch_testing/proc_info_handler.py", line 144, in assertWaitForShutdown
    assert success, "Timed out waiting for process '{}' to finish".format(process)
AssertionError: Timed out waiting for process '<launch.actions.execute_process.ExecuteProcess object at 0x1039075b0>' to finish

----------------------------------------------------------------------
Ran 1 test in 10.003s

FAILED (failures=1)
test_services (rcl.TestTwoExecutablesAfterShutdown)
Test that the executable under test finished cleanly. ... FAIL

======================================================================
FAIL: test_services (rcl.TestTwoExecutablesAfterShutdown)
Test that the executable under test finished cleanly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/osrf/jenkins-agent/workspace/nightly_osx_repeated/ws/build/rcl/test/test/test_services__rmw_cyclonedds_cpp_.py", line 50, in test_services
    launch_testing.asserts.assertExitCodes(
  File "/Users/osrf/jenkins-agent/workspace/nightly_osx_repeated/ws/install/launch_testing/lib/python3.8/site-packages/launch_testing/asserts/assert_exit_codes.py", line 62, in assertExitCodes
    assert info.returncode in allowable_exit_codes, 'Proc {} exited with code {}'.format(
AssertionError: Proc client_fixture-2 exited with code -2

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)
-- run_test.py: return code 1
-- run_test.py: verify result file '/Users/osrf/jenkins-agent/workspace/nightly_osx_repeated/ws/build/rcl/test_results/rcl/test_services__rmw_cyclonedds_cpp.xunit.xml'

[sloretz]

It looks this issue has been solved. The failing test is no longer marked as xfail: ros2/rcl#904

I'll close this for now.