Newly flaky tests 19 april 2017

[dhood]

The following tests failed during the "flakiness finder" repeat-until-fail nightly jobs last night, but passed in the "flakiness ignorer" repeat-until-pass nightly jobs.

It's good that they are passing when repeated, but I don't remember them being flaky before.

http://ci.ros2.org/job/nightly_linux_repeated/624/testReport/junit/(root)/TestStateMachineInfo/available_transitions/ http://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/15/testReport/junit/(root)/TestStateMachineInfo/available_transitions/

Stacktrace

[TestStateMachineInfo.available_transitions](/home/rosbuild/ci_scripts/ws/src/ros2/rclcpp/rclcpp_lifecycle/test/test_state_machine_info.cpp:66)
Value of: transition.start_state().label().empty()
  Actual: true
Expected: false

ros2/rclcpp#324

---

projectroot.test_showimage_cam2image__rmw_fastrtps_cpp
timeout (30s)

---

test.test_timer.test_timer_zero_callbacks1000hertz

Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/usr/lib/python3.5/unittest/case.py", line 600, in run
    testMethod()
  File "/home/rosbuild/ci_scripts/venv/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/rosbuild/ci_scripts/ws/src/ros2/rclpy/rclpy/test/test_timer.py", line 150, in test_timer_zero_callbacks1000hertz
    func_launch(func_zero_callback, ['0.001'], 'received callbacks when not expected')
  File "/home/rosbuild/ci_scripts/ws/src/ros2/rclpy/rclpy/test/test_timer.py", line 127, in func_launch
    args=(function, args)
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 253, in apply
    return self.apply_async(func, args, kwds).get()
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
    raise self._value
AssertionError: shouldn't have received any callback, received 1

waits for 0.0005s for a timer with period 0.001s, checking it doesn't receive anything, but it does. Underneath it must have taken longer than expected to wake up.

---

test_pendulum__rmw_connext_cpp.test_executable

Stacktrace

  File "/usr/lib/python3.5/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/usr/lib/python3.5/unittest/case.py", line 600, in run
    testMethod()
  File "/home/rosbuild/ci_scripts/venv/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/rosbuild/ci_scripts/ws/build/pendulum_control/test_pendulum__rmw_connext_cpp.py", line 60, in test_executable
    handler.check()
  File "/home/rosbuild/ci_scripts/ws/install/launch_testing/lib/python3.5/site-packages/launch_testing/__init__.py", line 106, in check
    (output_lines, self.expected_lines))
Example output ([b'Initial major pagefaults: 0', b'Initial minor pagefaults: 8625', b'PRESWriterHistoryDriver_completeBeAsynchPub:!make_sample_reclaimable', b'rttest statistics:', b'  - Minor pagefaults: 36', b'  - Major pagefaults: 0', b'  Latency (time after deadline was missed):', b'    - Min: 17403 ns', b'    - Max: 16213992 ns', b'    - Mean: 284679 ns', b'    - Standard deviation: 685.729', b'', b'', b'PendulumMotor received 242 messages', b'PendulumController received 245 messages']) does not match expected output ([b'Initial major pagefaults: \\d+', b'Initial minor pagefaults: \\d+', b'rttest statistics:', b'  - Minor pagefaults: \\d+', b'  - Major pagefaults: \\d+', b'  Latency \\(time after deadline was missed\\):', b'    - Min: \\d+ ns', b'    - Max: \\d+ ns', b'    - Mean: \\d+.\\d+ ns', b'    - Standard deviation: \\d+(\\.\\d+)?(e[\\+\\-]\\d+)?', b'', b'', b'PendulumMotor received \\d+ messages', b'PendulumController received \\d+ messages'])

ros2/demos#126

---

projectroot.test_showimage_cam2image__rmw_fastrtps_cpp

----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rosbuild/ci_scripts/venv/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/rosbuild/ci_scripts/ws/build/image_tools/test_showimage_cam2image__rmw_fastrtps_cpp_.py", line 32, in test_reliable_qos
    _test_executables(pub_executable_args, sub_executable_args)
  File "/home/rosbuild/ci_scripts/ws/build/image_tools/test_showimage_cam2image__rmw_fastrtps_cpp_.py", line 95, in _test_executables
    handler.check()
  File "/home/rosbuild/ci_scripts/ws/install/launch_testing/lib/python3.5/site-packages/launch_testing/__init__.py", line 106, in check
    (output_lines, self.expected_lines))
launch_testing.UnmatchedOutputError: Example output ([b"Publishing data on topic 'image'", b'\x1b[31;1m[\x1b[37;1mRTPS_ERROR\x1b[31;1m Error] \x1b[37mWriter 1.f.0.2.8e.d2.0.0.0.0.0.0|0.0.2.3 cannot match reader 1.f.0.2.8d.d2.0.0.0.0.0.0|0.0.2.4\x1b[34;1m -> Function \x1b[36mpairingReaderProxy\x1b[m', b'Publishing image #1', b'Publishing image #2', b'signal SIGINT', b'signal_handler(2)']) does not match expected output ([b"Publishing data on topic 'image'", b'Publishing image #1'])
-------------------- >> begin captured stdout << ---------------------
(test_showimage) pid 53901: ['/home/rosbuild/ci_scripts/ws/build/image_tools/showimage', '-r', '1', '-s', '0', '-b'] (InMemoryHandler: test_showimage, all > console)
(test_cam2image) pid 53902: ['/home/rosbuild/ci_scripts/ws/build/image_tools/cam2image', '-r', '1', '-s', '0', '-b', '-f', '5'] (InMemoryHandler: test_cam2image, all > console)
[test_showimage] Subscribing to topic 'image'
[test_cam2image] Publishing data on topic 'image'
[test_cam2image] [NON-XML-CHAR-0x1B][31;1m[[NON-XML-CHAR-0x1B][37;1mRTPS_ERROR[NON-XML-CHAR-0x1B][31;1m Error] [NON-XML-CHAR-0x1B][37mWriter 1.f.0.2.8e.d2.0.0.0.0.0.0|0.0.2.3 cannot match reader 1.f.0.2.8d.d2.0.0.0.0.0.0|0.0.2.4[NON-XML-CHAR-0x1B][34;1m -> Function [NON-XML-CHAR-0x1B][36mpairingReaderProxy[NON-XML-CHAR-0x1B][m
[test_cam2image] Publishing image #1
[test_cam2image] Publishing image #2
[test_showimage] Received image #1
[test_showimage] signal_handler(2)

Update [15 May 2017]: This did not fail when tested 40 times so might have fixed itself http://ci.ros2.org/job/ci_linux-aarch64/180/

Update [5 June 2017]: More details in this ticket: #21

---

projectroot.test_demo_cyclic_pipeline__rmw_fastrtps_cpp
timeout 30s

[dirk-thomas]

@dhood Is there any further information on these flaky tests in other repos? Or anything on investigating them?

[dhood]

apologies for the delay, this slipped past me. links to relevant tickets are in the original post, where appropriate.