tests: use test_interactive_test_util when possible #12461

fjmolinas · 2019-10-15T13:14:57Z

Contribution description

In order to be able to run tests on as many boards as possible use test_interactive_test_util in all possible cases.

Many tests depend on resetting the board after the terminal has been opened to make sure the captured output is the start of the tests.

When BOARD's can't provide this feature using test_interactive_test_util allows the application to wait for a sync from the test script to actually start.

Changes introduced:

0e1c79e: modify test runner to accept a sync option
28948d6: add shell commands for interactive sync
faf8fa1: used interactive sync when possible. The module is added in auto-init. I did't split the commit since introducing it before the test changes would make most tests failed.

Can be splited from the PR:

e5b1cd7: improve tests/isr_yield_higher

TODO in follow-up:

reset before term
not reset if sync=True:

    if sync is True:
        try:
            subprocess.check_output(('make', 'reset'), env=env,
                                    stderr=subprocess.PIPE)
        except subprocess.CalledProcessError:
            # make reset yields error on some boards even if successful
            pass

Testing procedure

Run all tests on multiple boards that support resetting, nothing should have changed:

native

python3 dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native  --with-test-only
ERROR:native:Tests failed: 7
Failures during test:
- [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
- [tests/gnrc_ipv6_ext_frag](tests/gnrc_ipv6_ext_frag/test.failed)
- [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
- [tests/gnrc_tcp](tests/gnrc_tcp/test.failed)
- [tests/lwip](tests/lwip/test.failed)
- [tests/netstats_l2](tests/netstats_l2/test.failed)

samr21-xpro

python3 dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . samr21-xpro  --with-test-only
Failures during test:
- [examples/suit_update](examples/suit_update/test.failed)
- [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
- [tests/gnrc_ipv6_ext_frag](tests/gnrc_ipv6_ext_frag/test.failed)
- [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
- [tests/gnrc_tcp](tests/gnrc_tcp/test.failed)
- [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)

iotlab-m3

ERROR:iotlab-m3:Tests failed: 7
Failures during test:
- [examples/suit_update](examples/suit_update/test.failed)
- [tests/gnrc_ipv6_ext](tests/gnrc_ipv6_ext/test.failed)
- [tests/gnrc_ipv6_ext_frag](tests/gnrc_ipv6_ext_frag/test.failed)
- [tests/gnrc_rpl_srh](tests/gnrc_rpl_srh/test.failed)
- [tests/gnrc_sock_dns](tests/gnrc_sock_dns/test.failed)
- [tests/gnrc_tcp](tests/gnrc_tcp/test.failed)
- [tests/pkg_fatfs_vfs](tests/pkg_fatfs_vfs/test.failed)

Green murdock to see I haven't missed any dependency

Issues/PRs references

Part of #12448

fjmolinas · 2019-10-16T07:51:35Z

With this PR test failed goes down from to 16 for ``z1`

PR:ERROR:z1:Tests failed: 76

Failures during test:
- [tests/bench_runtime_coreapis](tests/bench_runtime_coreapis/test.failed)
- [tests/bench_sizeof_coretypes](tests/bench_sizeof_coretypes/test.failed)
- [tests/bitarithm_timings](tests/bitarithm_timings/test.failed)
- [tests/build_system_cflags_spaces](tests/build_system_cflags_spaces/test.failed)
- [tests/buttons](tests/buttons/test.failed)
- [tests/cb_mux](tests/cb_mux/test.failed)
- [tests/cb_mux_bench](tests/cb_mux_bench/test.failed)
- [tests/driver_grove_ledbar](tests/driver_grove_ledbar/test.failed)
- [tests/driver_my9221](tests/driver_my9221/test.failed)
- [tests/embunit](tests/embunit/test.failed)
- [tests/event_wait_timeout](tests/event_wait_timeout/test.failed)
- [tests/evtimer_msg](tests/evtimer_msg/test.failed)
- [tests/float](tests/float/test.failed)
- [tests/fmt_print](tests/fmt_print/test.failed)
- [tests/gnrc_ipv6_nib](tests/gnrc_ipv6_nib/test.failed)
- [tests/gnrc_ipv6_nib_6ln](tests/gnrc_ipv6_nib_6ln/test.failed)
- [tests/gnrc_ndp](tests/gnrc_ndp/test.failed)
- [tests/gnrc_sock_ip](tests/gnrc_sock_ip/test.failed)
- [tests/gnrc_sock_udp](tests/gnrc_sock_udp/test.failed)
- [tests/heap_cmd](tests/heap_cmd/test.failed)
- [tests/irq](tests/irq/test.failed)
- [tests/isr_yield_higher](tests/isr_yield_higher/test.failed)
- [tests/l2util](tests/l2util/test.failed)
- [tests/libc_newlib](tests/libc_newlib/test.failed)
- [tests/log_color](tests/log_color/test.failed)
- [tests/log_printfnoformat](tests/log_printfnoformat/test.failed)
- [tests/msg_avail](tests/msg_avail/test.failed)
- [tests/msg_send_receive](tests/msg_send_receive/test.failed)
- [tests/msg_try_receive](tests/msg_try_receive/test.failed)
- [tests/mutex_order](tests/mutex_order/test.failed)
- [tests/netdev_test](tests/netdev_test/test.failed)
- [tests/od](tests/od/test.failed)
- [tests/periph_gpio](tests/periph_gpio/test.failed)
- [tests/periph_timer](tests/periph_timer/test.failed)
- [tests/pipe](tests/pipe/test.failed)
- [tests/pkg_c25519](tests/pkg_c25519/test.failed)
- [tests/pkg_cayenne-lpp](tests/pkg_cayenne-lpp/test.failed)
- [tests/pkg_heatshrink](tests/pkg_heatshrink/test.failed)
- [tests/pkg_jsmn](tests/pkg_jsmn/test.failed)
- [tests/pkg_lora-serialization](tests/pkg_lora-serialization/test.failed)
- [tests/pkg_nanocbor](tests/pkg_nanocbor/test.failed)
- [tests/pkg_tiny-asn1](tests/pkg_tiny-asn1/test.failed)
- [tests/pkg_tweetnacl](tests/pkg_tweetnacl/test.failed)
- [tests/pkg_ucglib](tests/pkg_ucglib/test.failed)
- [tests/pkg_umorse](tests/pkg_umorse/test.failed)
- [tests/posix_semaphore](tests/posix_semaphore/test.failed)
- [tests/pthread](tests/pthread/test.failed)
- [tests/pthread_barrier](tests/pthread_barrier/test.failed)
- [tests/pthread_cleanup](tests/pthread_cleanup/test.failed)
- [tests/pthread_condition_variable](tests/pthread_condition_variable/test.failed)
- [tests/pthread_cooperation](tests/pthread_cooperation/test.failed)
- [tests/pthread_flood](tests/pthread_flood/test.failed)
- [tests/pthread_rwlock](tests/pthread_rwlock/test.failed)
- [tests/pthread_tls](tests/pthread_tls/test.failed)
- [tests/riotboot_hdr](tests/riotboot_hdr/test.failed)
- [tests/rmutex](tests/rmutex/test.failed)
- [tests/sched_testing](tests/sched_testing/test.failed)
- [tests/shell](tests/shell/test.failed)
- [tests/test_tools](tests/test_tools/test.failed)
- [tests/thread_basic](tests/thread_basic/test.failed)
- [tests/thread_cooperation](tests/thread_cooperation/test.failed)
- [tests/thread_exit](tests/thread_exit/test.failed)
- [tests/thread_flags](tests/thread_flags/test.failed)
- [tests/thread_flags_xtimer](tests/thread_flags_xtimer/test.failed)
- [tests/thread_flood](tests/thread_flood/test.failed)
- [tests/thread_msg](tests/thread_msg/test.failed)
- [tests/thread_msg_block_w_queue](tests/thread_msg_block_w_queue/test.failed)
- [tests/thread_msg_block_wo_queue](tests/thread_msg_block_wo_queue/test.failed)
- [tests/thread_msg_seq](tests/thread_msg_seq/test.failed)
- [tests/thread_race](tests/thread_race/test.failed)
- [tests/trickle](tests/trickle/test.failed)
- [tests/xtimer_hang](tests/xtimer_hang/test.failed)
- [tests/xtimer_msg_receive_timeout](tests/xtimer_msg_receive_timeout/test.failed)
- [tests/xtimer_now64_continuity](tests/xtimer_now64_continuity/test.failed)
- [tests/xtimer_periodic_wakeup](tests/xtimer_periodic_wakeup/test.failed)
- [tests/xtimer_remove](tests/xtimer_remove/test.failed)
- [tests/xtimer_reset](tests/xtimer_reset/test.failed)
- [tests/xtimer_usleep](tests/xtimer_usleep/test.failed)

PR:ERROR:z1:Tests failed: 16

Failures during compilation:
- [tests/gnrc_ndp](tests/gnrc_ndp/compilation.failed)

Failures during test:
- [tests/bench_runtime_coreapis](tests/bench_runtime_coreapis/test.failed)
- [tests/bitarithm_timings](tests/bitarithm_timings/test.failed)
- [tests/driver_grove_ledbar](tests/driver_grove_ledbar/test.failed)
- [tests/driver_my9221](tests/driver_my9221/test.failed)
- [tests/event_wait_timeout](tests/event_wait_timeout/test.failed)
- [tests/evtimer_msg](tests/evtimer_msg/test.failed)
- [tests/periph_gpio](tests/periph_gpio/test.failed)
- [tests/pkg_c25519](tests/pkg_c25519/test.failed)
- [tests/pkg_heatshrink](tests/pkg_heatshrink/test.failed)
- [tests/pkg_lora-serialization](tests/pkg_lora-serialization/test.failed)
- [tests/pkg_tiny-asn1](tests/pkg_tiny-asn1/test.failed)
- [tests/pkg_tweetnacl](tests/pkg_tweetnacl/test.failed)
- [tests/pthread_tls](tests/pthread_tls/test.failed)
- [tests/shell](tests/shell/test.failed)
- [tests/thread_flags_xtimer](tests/thread_flags_xtimer/test.failed)

Te compilation issue in tests/gnrc_ndp is because of fw size.

fjmolinas · 2019-10-16T07:51:51Z

I'm going to trigger murdock to see which applications don't compile because of size anymore.

keestux · 2019-10-21T18:08:19Z

dist/pythonlibs/testrunner/__init__.py

@@ -22,10 +22,12 @@
 TIMEOUT = int(os.environ.get('RIOT_TEST_TIMEOUT') or 10)


-def run(testfunc, timeout=TIMEOUT, echo=True, traceback=False):
+def run(testfunc, timeout=TIMEOUT, echo=True, traceback=False, sync=False):


Have you considered to make sync default to True? Wouldn't that reduce the number of files you have to change for this PR?
There are 168 line with "run(testfunc" and in the PR 154 with sync=True.

Sure, I didn't initially because I wasn't sure if the PR would need to be split and introduce changes slowly, in which case a default false was better.

I can remove them at the end, for now I rather keep it help me see easily which tests I have changed and which not :)

I can remove them at the end, for now I rather keep it help me see easily which tests I have changed and which not :)

By the end I mean after squashing :)

Maybe it is time now, if all tests are not regressing, please squash and update the with default sync=True

I'll test the squash version then push.

tests/isr_yield_higher/main.c

tests/README.md

dist/pythonlibs/testrunner/__init__.py

sys/shell/commands/shell_commands.c

fjmolinas · 2019-10-22T09:14:11Z

rebased

fjmolinas · 2019-10-22T14:55:58Z

rebased

MrKevinWeiss · 2019-11-05T13:05:06Z

I can make this a priority if you have time to work on it (I think we need to move fast due to moving target rebasing).

My only concern with this is that it changes the behaviour of the tests when you just want to term in and read values. if there is a way to include everything only when make test is called and not when make term is.

Also I think it would be good to evaluate this on the arduino-mega2560. I have had some problems with getting stuck in the bootloader mode when trying to poll if the terminal is ready (I believe you need a 2.6 second timeout after opening a serial connection otherwise it could take up to 10 seconds or so to go out of bootloader mode, this 10 seconds resets whenever you try to talk to it). I don't see this problem on a cheap offbrand china version but on the standard ones this is how the bootloader seems to behave.

fjmolinas · 2019-11-12T10:07:49Z

I can make this a priority if you have time to work on it (I think we need to move fast due to moving target rebasing).

I was on holiday but I'm back and can get back to working on this.

Also I think it would be good to evaluate this on the arduino-mega2560. I have had some problems with getting stuck in the bootloader mode when trying to poll if the terminal is ready (I believe you need a 2.6 second timeout after opening a serial connection otherwise it could take up to 10 seconds or so to go out of bootloader mode, this 10 seconds resets whenever you try to talk to it). I don't see this problem on a cheap offbrand china version but on the standard ones this is how the bootloader seems to behave.

I will test and see what happens.

My only concern with this is that it changes the behaviour of the tests when you just want to term in and read values. if there is a way to include everything only when make test is called and not when make term is.

I'll look into this

fjmolinas · 2019-11-12T10:10:19Z

rebased

dist/pythonlibs/testrunner/spawn.py

sys/shell/commands/sc_interactive_sync.c

fjmolinas · 2019-11-26T11:47:48Z

Failing tests:

    tests/driver_nrfmin/nrf52dk:gnu
    tests/driver_nrfmin/nrf52dk:llvm
    tests/malloc/nrf52dk:llvm
    tests/nordic_softdevice/nrf52dk:gnu

The first two and the 4th are fixed by removing the child.expect_exact("All up, running the shell now"). I can't reproduce the third failure locally.

aabadie · 2019-11-26T17:45:20Z

@fjmolinas, please add arduino-uno, arduino-duemilanove, arduino-nano and atmega328p boards to BOARD_INSUFFICIENT_MEMORY and we are good to go with this one!

fjmolinas · 2019-11-27T09:26:40Z

I think tests/periph_wdt migh have failed because of #12812 (comment).

aabadie

ACK

Once the CI is green, this can be merged.

Automatically call the test_utils_interactive_sync synchronization if it is used.

- Define test_utils_interactive_sync as DEFAULT_MODULE in Makefile.tests_common - For tests disabling autoinit, add test_utils_interactive_sync to main - Add DISABLE_MODULE += test_utils_interactive_sync for tests requiring sudo, `tests/shell`, `tests/minimal` and `tests/stdin` - Add shell_commands to tests/periph_wdt and tests/struct_tm_utility to pull `r` and `s` commands - Remove includes and usage in `tests/main.c` for tests that where already using test_utils_interactive_sync

- When using test_interactive_sync_utils, stdin and many more prints/puts are included. These all go into .bss/.data which quickly fills up RAM.

aabadie · 2019-11-27T16:56:49Z

All green!

fjmolinas · 2019-11-27T18:28:46Z

@keestux @aabadie Thanks for the review! @MrKevinWeiss Thanks for testing!

With RIOT-OS#12941 and RIOT-OS#13613 some of the blacklisting introduced in RIOT-OS#12461 are no longer needed, since `test_interactive_test_util` is lighter or adds no extra code.

fjmolinas added Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation Area: tests Area: tests and testing framework CI: run tests If set, CI server will run tests on hardware for the labeled PR labels Oct 15, 2019

fjmolinas requested review from aabadie and MrKevinWeiss October 15, 2019 13:14

fjmolinas mentioned this pull request Oct 15, 2019

tests: run tests on all boards. #12448

Open

9 tasks

fjmolinas added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Oct 16, 2019

aabadie mentioned this pull request Oct 20, 2019

boards/arduino-mkr: feather-m0: sodaq-*: provide stdio over USB and setup automatic flash with bossa #12304

Merged

fjmolinas mentioned this pull request Oct 21, 2019

dist/pythonlibs/testrunner: add reset option, reset before cleanterm #12520

Closed

keestux reviewed Oct 21, 2019

View reviewed changes

tests/isr_yield_higher/main.c Outdated Show resolved Hide resolved

keestux reviewed Oct 21, 2019

View reviewed changes

tests/README.md Outdated Show resolved Hide resolved

keestux reviewed Oct 21, 2019

View reviewed changes

dist/pythonlibs/testrunner/__init__.py Outdated Show resolved Hide resolved

keestux reviewed Oct 21, 2019

View reviewed changes

sys/shell/commands/shell_commands.c Outdated Show resolved Hide resolved

fjmolinas mentioned this pull request Oct 22, 2019

tests/isr_yield_higher: adapt TEST_TIME to start-up time #12539

Merged

fjmolinas force-pushed the pr_apply_interactive_sync_tests branch from fee1d9e to 3245eaa Compare October 22, 2019 09:14

fjmolinas force-pushed the pr_apply_interactive_sync_tests branch from 3245eaa to ddce923 Compare October 22, 2019 14:55

fjmolinas removed the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Oct 22, 2019

aabadie mentioned this pull request Nov 6, 2019

Failing tests on AVR (tested with atmega256rfr2-xpro) #12651

Closed

13 tasks

MrKevinWeiss mentioned this pull request Nov 12, 2019

cpu/esp8266: complete reimplementation based on ESP8266 RTOS SDK #11108

Merged

20 tasks

kaspar030 mentioned this pull request Nov 26, 2019

murdock: enable esp32-wroom-32 for CI testing #11449

Merged

kaspar030 reviewed Nov 26, 2019

View reviewed changes

dist/pythonlibs/testrunner/spawn.py Outdated Show resolved Hide resolved

kaspar030 reviewed Nov 26, 2019

View reviewed changes

sys/shell/commands/sc_interactive_sync.c Outdated Show resolved Hide resolved

aabadie mentioned this pull request Nov 26, 2019

tests/stdin: fix automatic test on slow platforms #12816

Merged

fjmolinas removed the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Nov 26, 2019

fjmolinas force-pushed the pr_apply_interactive_sync_tests branch from 9979fb5 to cf95904 Compare November 26, 2019 16:08

fjmolinas added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Nov 26, 2019

aabadie mentioned this pull request Nov 27, 2019

tests/malloc: reduce default chunk size #12813

Merged

fjmolinas force-pushed the pr_apply_interactive_sync_tests branch from c4127ed to 513d4c5 Compare November 27, 2019 09:25

fjmolinas mentioned this pull request Nov 27, 2019

tests: fix tests matching varied sized group regex #12822

Merged

aabadie approved these changes Nov 27, 2019

View reviewed changes

fjmolinas and others added 6 commits November 27, 2019 15:07

sys/shell/commands: add interactive_sync commands

838a88f

testrunner: auto sync with test_utils_interactive_sync if present

eb9cb19

Automatically call the test_utils_interactive_sync synchronization if it is used.

tests/README.md: add automated test guideline

34c6063

tests: insufficient mem. atmega328p based boards

0996fb8

- When using test_interactive_sync_utils, stdin and many more prints/puts are included. These all go into .bss/.data which quickly fills up RAM.

tests/gnrc_ndp: add z1 to insufficient memory

175c48f

fjmolinas force-pushed the pr_apply_interactive_sync_tests branch from 513d4c5 to 175c48f Compare November 27, 2019 14:08

fjmolinas added a commit to fjmolinas/RIOT that referenced this pull request Nov 27, 2019

MERGECOMMIT RIOT-OS#12461

d72706b

aabadie merged commit 6e2513a into RIOT-OS:master Nov 27, 2019

fjmolinas deleted the pr_apply_interactive_sync_tests branch November 27, 2019 18:28

fjmolinas mentioned this pull request Dec 13, 2019

[POC] stdio: avr pgmspace #12940

Open

fjmolinas added this to the Release 2020.01 milestone Dec 13, 2019

fjmolinas mentioned this pull request Mar 24, 2020

tests: remove uneeded blacklist #13695

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: use test_interactive_test_util when possible #12461

tests: use test_interactive_test_util when possible #12461

fjmolinas commented Oct 15, 2019

fjmolinas commented Oct 16, 2019 •

edited

Loading

fjmolinas commented Oct 16, 2019

keestux Oct 21, 2019 •

edited

Loading

fjmolinas Oct 22, 2019

fjmolinas Oct 22, 2019

MrKevinWeiss Nov 13, 2019

fjmolinas Nov 13, 2019

fjmolinas commented Oct 22, 2019

fjmolinas commented Oct 22, 2019

MrKevinWeiss commented Nov 5, 2019

fjmolinas commented Nov 12, 2019

fjmolinas commented Nov 12, 2019

fjmolinas commented Nov 26, 2019

aabadie commented Nov 26, 2019

fjmolinas commented Nov 27, 2019

aabadie left a comment

aabadie commented Nov 27, 2019

fjmolinas commented Nov 27, 2019

tests: use test_interactive_test_util when possible #12461

tests: use test_interactive_test_util when possible #12461

Conversation

fjmolinas commented Oct 15, 2019

Contribution description

Testing procedure

Issues/PRs references

fjmolinas commented Oct 16, 2019 • edited Loading

fjmolinas commented Oct 16, 2019

keestux Oct 21, 2019 • edited Loading

Choose a reason for hiding this comment

fjmolinas Oct 22, 2019

Choose a reason for hiding this comment

fjmolinas Oct 22, 2019

Choose a reason for hiding this comment

MrKevinWeiss Nov 13, 2019

Choose a reason for hiding this comment

fjmolinas Nov 13, 2019

Choose a reason for hiding this comment

fjmolinas commented Oct 22, 2019

fjmolinas commented Oct 22, 2019

MrKevinWeiss commented Nov 5, 2019

fjmolinas commented Nov 12, 2019

fjmolinas commented Nov 12, 2019

fjmolinas commented Nov 26, 2019

aabadie commented Nov 26, 2019

fjmolinas commented Nov 27, 2019

aabadie left a comment

Choose a reason for hiding this comment

aabadie commented Nov 27, 2019

fjmolinas commented Nov 27, 2019

fjmolinas commented Oct 16, 2019 •

edited

Loading

keestux Oct 21, 2019 •

edited

Loading