forked from eclipse-iceoryx/iceoryx
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
iox-eclipse-iceoryx#378 initial iceoryx systemtest
Signed-off-by: Dietrich Krönke <[email protected]>
- Loading branch information
Showing
9 changed files
with
410 additions
and
55 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| # Iceoryx Systemtests | ||
|
|
||
| ## Introduction | ||
| To ensure quality standards in iceoryx, we are using automated testing to verify functionality on unittest and integrationtest level. | ||
| Additionally we need to make sure that the customer-facing API is functional and system-level requirements are fulfilled. | ||
|
|
||
| For that purpose we bring in tests which simulate customer behavior to have automatic testing of (Mis)Use-cases. | ||
| As testing framework we use the [launch_testing](https://github.com/ros2/launch/tree/master/launch_testing) from ROS 2. | ||
| The tests are in a ROS 2 CMake package where the test executables are build and tested by python scripts. | ||
| In these scripts the testing is currently done by evaluating stdout output of the processes and the exit codes. | ||
|
|
||
| advantages: | ||
| - automatic test if processes have exited unexpectedly | ||
| - test against custom or error return codes to evaluate error-cases | ||
| - test the order of printed messages | ||
|
|
||
| limitations: | ||
| - testing against stdout is error prone | ||
| - limited functionality for performance testing because the stdout is buffered (messages could be reordered) | ||
|
|
||
| ## Setup | ||
| For building and executing the tests you need to have ROS2 installed. Please follow the instructions on https://index.ros.org/doc/ros2/Installation. | ||
| The systemtests are currently tested on ROS 2 "Foxy Fitzroy" in Ubuntu 20.04 LTS. | ||
|
|
||
| For a basic setup you need to install the following packages: | ||
| ```bash | ||
| sudo apt install ros-foxy-ros-base ros-foxy-ros-testing ros-foxy-launch-testing ros-foxy-ament-cmake python3-colcon-common-extensions | ||
| ``` | ||
|
|
||
| After installing you need to source ROS 2 to make the environment available in your terminal: | ||
| ```bash | ||
| source /opt/ros/foxy/setup.bash | ||
| ``` | ||
|
|
||
| **_NOTE:_** You can add the source command to your `~/.bashrc` for automatic loading the ROS2 workspace at boot time. | ||
|
|
||
| Required for the colcon build of iceoryx is that the repository is located within a ROS workspace like this: | ||
| ``` | ||
| iceoryx_workspace | ||
| └── src | ||
| └── iceoryx | ||
| ├── cmake | ||
| ├── cpptoml_vendor | ||
| ├── doc | ||
| ├── iceoryx_binding_c | ||
| ├── iceoryx_dds | ||
| ├── iceoryx_examples | ||
| ├── iceoryx_meta | ||
| ├── iceoryx_posh | ||
| ├── iceoryx_systemtest | ||
| ├── iceoryx_utils | ||
| └── tools | ||
| ``` | ||
|
|
||
| ## Test Build and Execution | ||
|
|
||
| For you go into your iceoryx_workspace folder and do the colcon build: | ||
| ```bash | ||
| colcon build | ||
| ``` | ||
| Expected output should be like this: `Summary: 13 packages finished [24.1s]` | ||
| Colcon automatically creates the folders `build`, `install` and `log`. | ||
|
|
||
| For executing tests you can use colcon too: | ||
| ```bash | ||
| colcon test | ||
| ``` | ||
| For the case that a test fails the output look like this | ||
| ```bash | ||
| --- stderr: iceoryx_systemtest | ||
| Errors while running CTest | ||
| --- | ||
| Finished <<< iceoryx_systemtest [7.49s] [ with test failures ] | ||
|
|
||
| Summary: 13 packages finished [7.80s] | ||
| 1 package had stderr output: iceoryx_systemtest | ||
| 1 package had test failures: iceoryx_systemtest | ||
| ``` | ||
| In the `log/` folder you can find then the logfiles for the test execution in the `stdout_stderr.log` | ||
| ## Open points | ||
| - use an alternative way of tracing test information of the test processes without involving iceoryx (e.g. DDS or some tracing lib) | ||
| - add gtest for detailed testing in the test processes. |
99 changes: 99 additions & 0 deletions
99
iceoryx_systemtest/iceoryx_systemtest/test_data_exchange.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| # Copyright (c) 2021 by Apex.AI Inc. All rights reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| import os | ||
|
|
||
| import unittest | ||
|
|
||
| import launch | ||
| from launch_ros.substitutions import ExecutableInPackage | ||
| import launch_testing | ||
| import launch_testing.actions | ||
| from launch_testing.asserts import assertSequentialStdout | ||
|
|
||
| import pytest | ||
|
|
||
|
|
||
| @pytest.mark.launch_test | ||
| def generate_test_description(): | ||
|
|
||
| proc_env = os.environ.copy() | ||
|
|
||
| roudi_executable = ExecutableInPackage( | ||
| package='iceoryx_systemtest', executable='iox-roudi') | ||
| roudi_process = launch.actions.ExecuteProcess( | ||
| cmd=[roudi_executable, '-l', 'debug'], | ||
| env=proc_env, output='screen', | ||
| sigterm_timeout='20' | ||
| ) | ||
|
|
||
| publisher_executable = ExecutableInPackage( | ||
| package='iceoryx_systemtest', executable='iox-publisher-systemtest') | ||
| publisher_process = launch.actions.ExecuteProcess( | ||
| cmd=[publisher_executable], | ||
| env=proc_env, output='screen') | ||
|
|
||
| subscriber_executable = ExecutableInPackage( | ||
| package='iceoryx_systemtest', executable='iox-subscriber-systemtest') | ||
| subscriber_process = launch.actions.ExecuteProcess( | ||
| cmd=[subscriber_executable], | ||
| env=proc_env, output='screen') | ||
|
|
||
| return launch.LaunchDescription([ | ||
| roudi_process, | ||
| publisher_process, | ||
| subscriber_process, | ||
| launch_testing.actions.ReadyToTest() | ||
| ]), {'roudi_process': roudi_process, 'publisher_process': publisher_process, 'subscriber_process': subscriber_process} | ||
|
|
||
| #These tests will run concurrently with the dut process. After this test is done, | ||
| #the launch system will shut down RouDi | ||
| class TestProcess(unittest.TestCase): | ||
| def test_roudi_ready(self, proc_output): | ||
| proc_output.assertWaitFor( | ||
| 'RouDi is ready for clients', timeout=45, stream='stdout') | ||
|
|
||
| def test_apps_ready(self, proc_output): | ||
| proc_output.assertWaitFor( | ||
| 'Application iox_publisher_systemtest started', timeout=45, stream='stdout') | ||
| proc_output.assertWaitFor( | ||
| 'Application iox_subscriber_systemtest started', timeout=45, stream='stdout') | ||
|
|
||
| def test_simple_data_exchange(self, proc_output): | ||
| proc_output.assertWaitFor( | ||
| 'Sent two times value: 5', timeout=45, stream='stdout') | ||
| proc_output.assertWaitFor( | ||
| 'Got value: 5', timeout=45, stream='stdout') | ||
|
|
||
| # These tests run after shutdown and examine the stdout log | ||
| @launch_testing.post_shutdown_test() | ||
| class TestProcessOutput(unittest.TestCase): | ||
| def test_exit_code(self, proc_info): | ||
| launch_testing.asserts.assertExitCodes(proc_info) | ||
|
|
||
| def test_publisher_sequence_output(self, proc_output, publisher_process): | ||
| with assertSequentialStdout(proc_output, publisher_process) as cm: | ||
| cm.assertInStdout('Sent two times value: 1') | ||
| cm.assertInStdout('Sent two times value: 2') | ||
| cm.assertInStdout('Sent two times value: 3') | ||
| cm.assertInStdout('Sent two times value: 4') | ||
| cm.assertInStdout('Sent two times value: 5') | ||
|
|
||
| def test_subscriber_sequence_output(self, proc_output, subscriber_process): | ||
| with assertSequentialStdout(proc_output, subscriber_process) as cm: | ||
| #cm.assertInStdout('Got value: 1') | ||
| cm.assertInStdout('Got value: 2') | ||
| cm.assertInStdout('Got value: 3') | ||
| cm.assertInStdout('Got value: 4') | ||
| cm.assertInStdout('Got value: 5') |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.