[TRACKING] The RIOT shell as a testing interface

[jcarrano]

Description

The RIOT testing system uses the shell quite extensively. While this is in itself a good idea, the current embodiment has a few issues:

• A few bug in the shell.
• Lack of a programmatic way of interacting with it (this was pointed out before in Programmatic shell access #4967,) It would be desirable to have a library for interacting with the shell that can also be used from the testing scripts
• The build system / OS integration, via tools like pyterm which modify the input and output.
• Parsing is done by each test scripts via pexpect. Without strict restrictions on what to expect (and in particular which output to reject) a common failure mode of tests is to time out waiting for text that is never going to show up.

The plan is to:

• Fix the bugs.
• Remove the requirement for pyterm, in favour of a dumber terminal (if someone needs command history, there is rlwrap for that.)
• Write a module that abstracts shell interaction.
• Define and implement an easily parseable format for command output.
• Integrate all of that into the testing infrastructure.

Breakdown

• Fix shell bugs
  • sys/stdio_uart: do not cast function pointer. sys/stdio_uart: do not cast function pointer. #10312
  • sys/shell: ensure character is flushed when echoing. sys/shell: ensure character is flushed when echoing. #10630
  • sys/shell: correctly detect and handle long lines sys/shell: correctly detect and handle long lines #10635 sys/shell: correctly detect and handle long lines #13195
• Easily parseable commands:
  • doc/rdm: RDM on a common api for testing doc/rdm: RDM on a common api for testing #10624
• Self describing interfaces
  • sys/app_metadata: Add a way to print application metadata sys/app_metadata: Add a way to print application metadata #10961
  • sys/optparse: Add command line parser. sys/optparse: Add command line parser. #9538

[kaspar030]

* Remove the requirement for pyterm, in favour of a dumber terminal (if someone needs command history, there is [rlwrap](https://linux.die.net/man/1/rlwrap) for that.)

This has already been done. pyterm was just too damn slow to startup on the RasPis used by the CI. Just provide e.g., "RIOT_TERMINAL=picocom". Currently needs support in makefiles/tools/serial.inc.mk.

[jcarrano]

pyterm was just too damn slow to startup on the RasPis used by the CI

Then why have it on the CI only? If it is possible to remove it from the CI it should be possible to remove it from the official distribution.

In my view even picocom is unnecessary if the test script could directly read and write to the TTY. Once the tty device is configured, the terminal program is just copying bytes from /dev/ttyX to a pipe and back.