sys/test_utils/result_output: Initial implementations and API of turo #15950

MrKevinWeiss · 2021-02-08T16:40:43Z

Contribution description

The latest in an effort to add an abstraction layer for RIOT tests allowing a more stable testing API.
This focuses more on how results are printed or output.
Using the results_output API will allow a guaranteed way output data, meaning no more needing to adapt regex everytime a printf is changed.
The exact output can be changed depending on the module selected, this means if you don't like it you can just implement a variant that suits your needs.
The eventual goal will be to coordinate this with riotctrl so data can be used immediately when writing python based automated tests.

There are currently 4 variant implemented:

json: used for CI and computer interaction
txt: used for developer interaction
check: Only ensuring the output follows a correct structure
led: An example of using the turo_t *ctx to flash LEDs

More information is in the sys/test_utils/results_output.h file.

Note: The test_utils_result_output_json is ~~not~~ fully json compliant ~~as it contains trailing commas. This is allows simple flushing of information and no need for states. Many parses accept trailing commas such as rapidjson python parser~~.

Testing procedure

OUTPUT_FORMAT={txt, check, json, led} make all -C tests/turo

note: native has "leds" but they don't fit the LED0_PIN define so an assertion error will be thrown because native uses GPIO_UNDEF

Issues/PRs references

supersedes #15039
also supersedes #11224
Implementation of a cleaner idea of #10624

MrKevinWeiss · 2021-02-09T09:09:41Z

TODO:

Add module to uncrustify?
Add kconfig options
Add unittest to get full coverage (I think I don't want to complicate the test/example too much)

MrKevinWeiss · 2021-02-09T17:07:11Z

All that is done, I probably should split this PR up. @leandrolanzieri thanks for the kconfig help

aabadie · 2021-02-09T17:15:13Z

Do you really expect that people who write test applications will use turo_ calls instead of printf ? Do you think this makes the code of "tests used as examples" helpful ?

MrKevinWeiss · 2021-02-09T18:54:24Z

Do you really expect that people who write test applications will use turo_ calls instead of printf ? Do you think this makes the code of "tests used as examples" helpful ?

I have been asking about this for a while... For simple things maybe not (hence optional) but if I am writing a more complex test that I uses a matching python module for might be useful.

Does it make sense to try to enforce a structure when writing tests?

For the tests or modules that use, say json strings, would it not be better to use some common functions instead of always printf("{\"my_escape_char\": \"\\\"}\n")

aabadie · 2021-02-09T20:07:32Z

Does it make sense to try to enforce a structure when writing tests?

I agree that in certain situations such a module could be useful: for benchmark tests or very complex tests, as you say. For tests that are just there as usage examples, it's important to keep them as self-explanatory as possible and to limit the usage of test specific modules, eventually they could be moved to the examples directory instead (but in this case, we need a hierarchy of directories instead of the actual flat organization 🤔 ).

MrKevinWeiss · 2021-02-10T08:31:49Z

we need a hierarchy of directories instead of the actual flat organization

I think that has also been discussed quite a bit :)

@aabadie Is that a blocker for you or can we proceed with the concept (again, optional only if it makes sense for that test)?

miri64 · 2021-02-17T11:26:10Z

tests/turo/main.c

+ * Copyright (C) 2014 Freie Universität Berlin
+ *
+ * This file is subject to the terms and conditions of the GNU Lesser
+ * General Public License v2.1. See the file LICENSE in the top level
+ * directory for more details.
+ */
+
+/**
+ * @ingroup tests
+ * @{
+ *
+ * @file
+ * @brief GET_CPU_ID() test application
+ *
+ * @author Martine Lenders <[email protected]>
+ * @author Hauke Petersen <[email protected]>


This does not seem to be correct ;-)

Could be... could be

MrKevinWeiss · 2021-03-11T09:53:50Z

Now the LED module is dripped this is a bit simpler PR. Any remaining objections?

miri64 · 2021-03-11T11:01:03Z

#15950 (comment) remains uncommented.

miri64

If this one was addressed, I think we can merge.

miri64 · 2021-03-11T12:39:04Z

sys/include/test_utils/result_output.h

+ *
+ * Some of the design decisions include:
+ * - ability to flush immediately to prevent the need for large buffers
+ * - selectable output format based on USEMODULE


I think this was not addressed yet too.

miri64

This is ready to go in my opinion. I think your changes were addressed too, @kaspar030, right?

miri64 · 2021-03-12T14:00:19Z

Pleases squash, @MrKevinWeiss.

@kaspar030

@kaspar030 gave a thumbs up to my ACK, which I interpret that we can dismiss his review. Please re-request changes, if that is not the case.

miri64 · 2021-03-12T23:05:16Z

Sadly, Murdock's not happy :(

miri64 · 2021-03-12T23:07:14Z

Took me a while to interpret the output: apparently there are now different binaries generated with and without Kconfig.

MrKevinWeiss · 2021-03-13T10:16:51Z

Hmmm, I will look into it.

MrKevinWeiss · 2021-03-15T09:38:14Z

It seems I struggled with the rebasing and removing of the module. I make the turo module optional in kconfig and default select json.

MrKevinWeiss · 2021-03-15T10:32:26Z

@miri64 I think things are looking good now!

miri64 · 2021-03-15T10:34:54Z

🎉 then let's go!

MrKevinWeiss · 2021-03-15T11:32:35Z

Thanks for the review!

MrKevinWeiss added Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation Area: tests Area: tests and testing framework Area: sys Area: System labels Feb 8, 2021

MrKevinWeiss added this to the Release 2021.04 milestone Feb 8, 2021

MrKevinWeiss requested review from miri64, jia200x, kaspar030, leandrolanzieri and fjmolinas February 8, 2021 16:40

MrKevinWeiss self-assigned this Feb 8, 2021

MrKevinWeiss requested review from aabadie, MichelRottleuthner and smlng as code owners February 8, 2021 16:40

This was referenced Feb 8, 2021

congure: initial import of a congestion control framework #15951

Merged

congure_quic: initial import of QUIC congestion control #15952

Merged

congure_reno: initial import of TCP Reno congestion control #15953

Merged

MrKevinWeiss force-pushed the pr/turo/initial branch from 0290023 to 84f95c3 Compare February 9, 2021 08:56

MrKevinWeiss requested a review from cgundogan as a code owner February 9, 2021 17:02

miri64 mentioned this pull request Feb 9, 2021

congure_abe: initial import of TCP Alternative Backoff with ECN for CongURE #15968

Merged

MrKevinWeiss mentioned this pull request Feb 10, 2021

sys/fmt: Add print_s64_dec to header #15979

Merged

MrKevinWeiss added the State: waiting for other PR State: The PR requires another PR to be merged first label Feb 10, 2021

MrKevinWeiss mentioned this pull request Feb 10, 2021

pkg/umorse: Add kconfig option #15980

Closed

MrKevinWeiss force-pushed the pr/turo/initial branch from eb43f7f to 887e4e2 Compare February 11, 2021 13:28

miri64 reviewed Feb 22, 2021

View reviewed changes

miri64 requested changes Mar 11, 2021

View reviewed changes

miri64 approved these changes Mar 12, 2021

View reviewed changes

miri64 added CI: run tests If set, CI server will run tests on hardware for the labeled PR CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Mar 12, 2021

MrKevinWeiss force-pushed the pr/turo/initial branch from 4696894 to 01f7173 Compare March 12, 2021 15:05

MrKevinWeiss added 4 commits March 12, 2021 16:15

sys/result_output: Add initial json implementation

3cbd8a3

sys/result_output: Add initial text implementation

d178293

sys/result_output: Add initial structure check

2c2be26

sys/result_output: Expose result output

6a3eb46

MrKevinWeiss force-pushed the pr/turo/initial branch from 01f7173 to b1be11c Compare March 12, 2021 15:23

MrKevinWeiss added 4 commits March 15, 2021 10:36

sys/result_output: Add kconfig

38884d2

tests/unittests/turo: Add turo unittest

e9b6530

tests/turo: Add automated turo test for json

e7cc489

dist/tools/uncrustify: Add turo module

2e5ff00

MrKevinWeiss force-pushed the pr/turo/initial branch from b1be11c to 2e5ff00 Compare March 15, 2021 09:37

miri64 merged commit fd36c62 into RIOT-OS:master Mar 15, 2021

MrKevinWeiss deleted the pr/turo/initial branch March 15, 2021 11:32

This was referenced Mar 18, 2021

sys/result_output: Initial result output abstraction #15039

Closed

sys/test_helper: Add test_helper module #11224

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sys/test_utils/result_output: Initial implementations and API of turo #15950

sys/test_utils/result_output: Initial implementations and API of turo #15950

MrKevinWeiss commented Feb 8, 2021 •

edited

Loading

MrKevinWeiss commented Feb 9, 2021 •

edited

Loading

MrKevinWeiss commented Feb 9, 2021

aabadie commented Feb 9, 2021 •

edited

Loading

MrKevinWeiss commented Feb 9, 2021

aabadie commented Feb 9, 2021

MrKevinWeiss commented Feb 10, 2021

miri64 Feb 17, 2021

MrKevinWeiss Feb 22, 2021

MrKevinWeiss Feb 22, 2021

MrKevinWeiss commented Mar 11, 2021

miri64 commented Mar 11, 2021

miri64 left a comment

miri64 Mar 11, 2021

miri64 left a comment

miri64 commented Mar 12, 2021

miri64 commented Mar 12, 2021

miri64 commented Mar 12, 2021

MrKevinWeiss commented Mar 13, 2021

MrKevinWeiss commented Mar 15, 2021

MrKevinWeiss commented Mar 15, 2021

miri64 commented Mar 15, 2021

MrKevinWeiss commented Mar 15, 2021

sys/test_utils/result_output: Initial implementations and API of turo #15950

sys/test_utils/result_output: Initial implementations and API of turo #15950

Conversation

MrKevinWeiss commented Feb 8, 2021 • edited Loading

Contribution description

Testing procedure

Issues/PRs references

MrKevinWeiss commented Feb 9, 2021 • edited Loading

MrKevinWeiss commented Feb 9, 2021

aabadie commented Feb 9, 2021 • edited Loading

MrKevinWeiss commented Feb 9, 2021

aabadie commented Feb 9, 2021

MrKevinWeiss commented Feb 10, 2021

miri64 Feb 17, 2021

Choose a reason for hiding this comment

MrKevinWeiss Feb 22, 2021

Choose a reason for hiding this comment

MrKevinWeiss Feb 22, 2021

Choose a reason for hiding this comment

MrKevinWeiss commented Mar 11, 2021

miri64 commented Mar 11, 2021

miri64 left a comment

Choose a reason for hiding this comment

miri64 Mar 11, 2021

Choose a reason for hiding this comment

miri64 left a comment

Choose a reason for hiding this comment

miri64 commented Mar 12, 2021

miri64 commented Mar 12, 2021

miri64 commented Mar 12, 2021

MrKevinWeiss commented Mar 13, 2021

MrKevinWeiss commented Mar 15, 2021

MrKevinWeiss commented Mar 15, 2021

miri64 commented Mar 15, 2021

MrKevinWeiss commented Mar 15, 2021

MrKevinWeiss commented Feb 8, 2021 •

edited

Loading

MrKevinWeiss commented Feb 9, 2021 •

edited

Loading

aabadie commented Feb 9, 2021 •

edited

Loading