Release 2018.07 - RC2

[cladmi]

This issue lists the status of all tests for the Release Candidate 2 of the 2018.07 release.

Specs tested:

• 01-ci (Related PRs: 01-ci: update scripts #67) (Can someone else than me (cladmi) do it ? I will run them too but would love to have another tester than RC1)
  • Task #01 @jia200x @miri64 @cladmi @cladmi with docker using (docker: Use system GIT_CACHE_DIR if available RIOT#9651 and makefiles/docker.inc.mk: Use directories in RIOT when possible RIOT#9646)
  • Task #02 @miri64 @cladmi
  • Task #03 @miri64 @cladmi
  • Task #04 @miri64 @cladmi
• 02-tests (Related PRs: 09-compile-and-test-one-board: add script to test board #68) @cladmi and others (see task 9) Only automated tests
  • Task #01
  • Task #02
  • Task #03
• 03-single-hop-ipv6-icmp @PeterKietzmann
  • Task #01
  • Task #02
  • Task #03
• 04-single-hop-6lowpan-icmp @PeterKietzmann
  • Task #01
  • Task #02
  • Task #03
  • Task #04
  • (Task #05 ) @PeterKietzmann
  • (Task #06) @PeterKietzmann
  • (Task #07) @miri64
  • (Task #08) @miri64
• 05-single-hop-route @jia200x
  • Task #01
  • Task #02
  • Task #03
  • Task #04
• 06-single-hop-udp (Related Issues: Task 6: Questionable test parameters #53) @jia200x
  • Task #01
  • Task #02
• 07-multi-hop (Related Issues: Task 7: arbitrary error margin? #54 ) @jia200x
  • Task #01
  • Task #02
  • Task #03
  • Task #04
• 08-interop
  • Task #01 @PeterKietzmann
  • Task #02 @miri64
  • Task #03 Was successful with RC1 tested by @kYc0o so I validate it here too
  • Task #04 @miri64 (failed, but fix exists; see Release 2018.07 - RC2 #69 (comment))
  • Task #05 @miri64
  • Task #06 @miri64
  • Task #07 @miri64
  • Task #08
  • Task #09 @PeterKietzmann
  • Task #10
• 09-compile-and-test-one-board - (PR 68)
  • Proposal to run compilation and tests on other boards than the default ones
    The output is not blocking but can help getting more feedback on our board support.
  • iotlab-m3 on IoT-LAB frontend @cladmi
  • iotlab-a8-m3 on IoT-LAB frontend @cladmi
  • arduino-zero on IoT-LAB frontend @cladmi
  • samr21-xpro on IoT-LAB frontend @cladmi
  • iotlab-m3 on laptop @cladmi
  • samr21-xpro on laptop @cladmi
  • native on laptop @cladmi
  • wsn430-v1_3b compilation on IoT-LAB frontend @cladmi
  • wsn430-v1_4 compilation on IoT-LAB frontend @cladmi
  • b-l072z-lrwan1 @jia200x
  • stm32f4discovery @jcarrano
  • arduino-zero @aabadie
  • nrf52840dk @aabadie
  • nucleo-f070rb @aabadie
  • nucleo-l152re @aabadie
  • arduino-mega2560 @maribu
  • bluepill @MrKevinWeiss
  • nrf52dk @MrKevinWeiss
  • nucleo-l073rz @MrKevinWeiss
  • arduino-duemilanove @ZetaR60 in docker
  • arduino-uno @ZetaR60 in docker
  • mega-xplained @ZetaR60 in docker

[miri64]

I've tested Task 8.8 and task 8.10, the issue seems to be resolved on the IEEE 802.15.4 level, however, the lwIP node seems to ignore any packages send to it by the GNRC node (neither UDP nor neighbor advertisements are considered from what I can gather from a sniff). I could investigate.

[miri64]

Well the bug was marked fixed in Feb 2018. The release RIOT's lwIP support is based on is from Sep 2017, so this is no surprise 😌

[miri64]

I've done 1.1 on my machine over the weekend. I had failed jobs for

• examples/ccn-lite-relay (for all boards except native)
• tests/mcuboot (only board tested nrf52dk)
• tests/unittests (for acd52832, b-l475e-iot01a, nucleo-l452re, ruuvitag, stm32f429i-disc1, stm32l476g-disco, stm32mindev, teensy31, thingy52, and ublox-c030-u201)

The problem with ccn-lite-relay I was able to fix by updating cmake (which is weird, because I was able to compile it on the same machine in the past), same goes for the unitttests. For mcuboot there were just the python dependencies missing that are required for the build.

[cladmi]

I had the same issues when I tested RC1. For ccn-lite-relay maybe having a version test in the package build could be nice instead of just printing random errors.

I also re-run task 1 and everything worked on my computer. I will try running it also in docker.

[jia200x]

I re run the test script in 05-single-hop-route for Task 4, and I'm getting 1 out of 10 packets loss randomly :/. The script executes everything without delay between commands. I'm pretty sure it will work if the script is ran manually (so give time between commands)

The weird thing is this doesn't happen when running in actual nodes (iotlab-m3). Any ideas on what might be going wrong?

[miri64]

@jia200x why are you using timing between commands? Aren't you using tests/gnrc_udp for this (which was designed for these kind of tests)?

[jia200x]

@miri64 for single hop route I'm using ping from gnrc_networking.

What I mean, if I execute all commands manually (so there's around 1 sec between each command) it seems to work. When using the script, the first ICMP packet gets timeout randomly :/

[jia200x]

tried with the gnrc_udp test and get similar results in Task 4

[miri64]

Is it always the first?

[jia200x]

Yes. E.g:

Running test task4 from <class '__main__.TestNative'>
        Get first interface...ok
        Disabling Router Advertisement...ok
        Disabling Router Advertisement...ok
        Get IP address...ok
        Get IP address...ok
        Add IP address...ok
        Add NIB route...ok
        Add NIB route...ok
        Pinging node...x.........ok
  File "/home/jialamos/Development/gnrc_networking/children.py", line 58, in run
    t(nodes)
  File "new.py", line 80, in task4
    assert(packet_loss < 1)
FAILED

or

Running test task4 from <class '__main__.TestNative'>
        Get first interface...ok
        Disabling Router Advertisement...ok
        Disabling Router Advertisement...ok
        Get IP address...ok
        Get IP address...ok
        Add IP address...ok
        Add NIB route...ok
        Add NIB route...ok
        Pinging node...xx........ok
  File "/home/jialamos/Development/gnrc_networking/children.py", line 58, in run
    t(nodes)
  File "new.py", line 80, in task4
    assert(packet_loss < 1)
FAILED

Always the first packets.

[miri64]

I don't know what your script looks like, so I don't 100% understand the output ;-) but I can't reproduce manually:

• Pinged node

main(): This is RIOT! (Version: 2018.10-devel-86-g48f04-sarajevo-HEAD)
RIOT network stack example application
All up, running the shell now
>  ifconfig 6 -rtr_adv     
 ifconfig 6 -rtr_adv
success: unset option
> ifconfig 6 add beef::1/64
ifconfig 6 add beef::1/64
success: added beef::1/64 to interface 6

• Pinging node

main(): This is RIOT! (Version: 2018.10-devel-86-g48f04-sarajevo-HEAD)
RIOT network stack example application
All up, running the shell now
>  ifconfig 6 -rtr_adv         
 ifconfig 6 -rtr_adv
success: unset option
> nib route add 6 :: fe80::f03f:e7ff:feb6:c150
nib route add 6 :: fe80::f03f:e7ff:feb6:c150
> ping6 10 beef::1 1024 10
ping6 10 beef::1 1024 10
1032 bytes from beef::1: id=83 seq=1 hop limit=64 time = 0.188 ms
1032 bytes from beef::1: id=83 seq=2 hop limit=64 time = 0.211 ms
1032 bytes from beef::1: id=83 seq=3 hop limit=64 time = 0.180 ms
1032 bytes from beef::1: id=83 seq=4 hop limit=64 time = 0.149 ms
1032 bytes from beef::1: id=83 seq=5 hop limit=64 time = 0.179 ms
1032 bytes from beef::1: id=83 seq=6 hop limit=64 time = 0.149 ms
1032 bytes from beef::1: id=83 seq=7 hop limit=64 time = 0.194 ms
1032 bytes from beef::1: id=83 seq=8 hop limit=64 time = 0.184 ms
1032 bytes from beef::1: id=83 seq=9 hop limit=64 time = 0.137 ms
1032 bytes from beef::1: id=83 seq=10 hop limit=64 time = 0.187 ms
--- beef::1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 0.0692301 s
rtt min/avg/max = 0.137/0.175/0.211 ms
> 

[cladmi]

For the release, I would like to have tests results for different boards/cpu types:
Could you please run task 09 for your boards ? #68

• ikea-tradfri (efm32) @basilfx
• hifive1 (fe310/riscV) @kenrabold
• your favorite arduino board(s) @maribu @ZetaR60
• native mac @smlng
• your favorite kinetis board(s) @gebart
• one/some of the nucleo boards @aabadie
• mips board @neiljay

It would give a hint on the current board support and the automated tests suites state.

Thank you in advance.

[miri64]

Task 8.4 fails, but is fixed by RIOT-OS/RIOT#9522. I mark that PR for the release and open a PR against the release specs for doc on the required CFLAGS.

[miri64]

[…] and open a PR against the release specs for doc on the required CFLAGS.

See #71.

[aabadie]

@cladmi, I'll run the tests on nucleo-f070rb, nucleo-l152re, arduino-zero and nrf52840dk

[maribu]

Arduino Mega2560

[cladmi]

@maribu thank you :)

[aabadie]

Arduino Zero, Nrf52840dk and nucleo-f070rb

[ZetaR60]

@cladmi I'll do mega-xplained, arduino-uno, and arduino-duemilanove.

[ZetaR60]

mega-xplained, arduino-uno, and arduino-duemilanove

[cladmi]

Task 02-01:

• driver_my9221 is executed even if the hardware is of course not available
• Some tests are executed with echo=False so no output by default.

[cladmi]

Summary of 02-tests task 02/03 including also other boards than m3 and samr21-xpro.

samr21-xpro and iotlab-m3 are now only subject to false negatives with tests that are executed even if the required hardware is not present or would have required a specific setup which is not done.

---

Detailed test results of all tested boards can be found in here:

https://github.com/cladmi/RIOT_2018_07_tests_results

Thanks to all the persons that already ran tests on their board.

Compilation

Real bug

• sdcard_spi.c: 1012: cast to uint64_t should be on SD_HC_BLOCK_SIZE
  found with avr-gcc 6.4.0
  unrecognized command line option '-Wno-implicit-fallthrough' [-Werror]

Toolchain issues

• mcuboot: missing PyCrypto
• ccn-lite-relay: wrong cmake
• drivers/at/at.c: incompatible cast with gcc 8 [-Werror=cast-function-type] (pr drivers/at: fix invalid function pointer cast. RIOT#9740)
• pkg/lwip: incompatible cast with gcc [-Werror=cast-function-type] (fixed in master lwip: fix lwip thread_create() wrapper RIOT#9588)

Flash

Unreliable flash

• bluepill: cannot 'flash' for many of the tests @MrKevinWeiss is it a board or a problem with your setup ?
  could flash a few ones (cbor/driver_my9221/driver_grove_ledbar)
• nucleo-l073rz: cannot 'flash' for many of the tests
  could flash a few ones (cbor/driver_my9221/driver_grove_ledbar)
• arduino-zero
  • few test failed due to iot-lab not handling a request
• arduino-duemilanove:
  • avrdude: for many tests it fails after:
    stk500_recv(): programmer is not responding

Flash detecting rom overflow

• arduino-uno:
  • flash fails sometime with "address out of range" could be link overflow
    Better Info on flash & RAM requirements RIOT#9628
• arduino-duemilanove:
  • flash fails sometime with "address out of range" could be link overflow
    Better Info on flash & RAM requirements RIOT#9628

Term

• mega-xplained: Unreliable term output as mentioned in results for mega-xplained, arduino-uno, arduino-duemilanove cladmi/RIOT_2018_07_tests_results#13

  • In some tests I get the output of the previous test
    So I will not do a detailed analysis of the results.
    They are similar to arduino-mega2560

• nrf52dk:

• cannot open /dev/ttyACM0

• Unexpected end of file in expect script at "child.expect..."

Test

False negative that are present for every boards.
Should find a way to "disable" them or not fail if the hardware is of course
not there.
For tests that need special hardware/prepare setup, maybe another test command ?
'manual-test' 'test-with-specific-setup' or define a variable to say
'BOARD_HAS_BEEN_PREPARED' to enable tests

• pkg_fatfs_vfs: needs special setup using an sdcard
• driver_my9221: needs special hardware
• driver_ds1307: needs special hardware
• driver_grove_ledbar: needs special hardware
• driver_hd44780: needs special hardware
• cbor: broken

I think issues related to testing 2018.07-RC2 and not the last version
including the fix

• gnrc_netif: failed on some boards (Should be fixed with the last version)
  • arduino-zero aabadie
  • nrf52840dk
  • nucleo-l152re
  • stm32discovery

Test script reliability

• libfixmath: think an issue with timeout ? should maybe check
  • Timeout in expect script at "child.expect_exact('Unary.')" (tests/libfixmath/tests/01-run.py:34)
  • arduino-uno: timeouts
• libfixmath_unittests:
  • b-l072z-lrwan1: timeout before the end, increase global timeout I think
• od:
  • arduino-zero on iotlab only (not aabadie)
• pkg_tiny-asn1:
  • arduino-mega2560: could not allocate the memory for the ASN.1 object => OK
    test could handle it as still success (or 'skip' but not handled now)
• pthread_condition_variable
  • b-l072z-lrwan1
    • Timeout before test finished, test could match smaller steps as there is
      more output than just the end (would reduce main timeout too)
• xtimer_hang:
  • arduino-mega2560: test timeout before being at 100% could do smaller step as
    there is intermediate outputs

No output on arduino

• arduino-uno/mega2560/duemilanove/xplained: Test fails before seeing output
  I think some are linked to a ram/rom overflow not detected: Better Info on flash & RAM requirements RIOT#9628
  • bloom_bytes
  • button
  • cb_mux_bench
  • evtimer_msg
  • gnrc_ipv6_ext
  • gnrc_ipv6_nib
  • gnrc_ipv6_nib_6ln
  • gnrc_ndp
  • gnrc_sixlowpan
  • isr_yield_higher
  • mutex-order
  • netdev_test
  • pkg_tiny-asn1
  • pkg_u8g2
  • pkg_ucglib
  • posix_semaphore
  • ps_schedstatistics
  • rmutex
  • thread_cooperation
  • thread_msg
  • thread_msg_seq
  • xtimer_periodic_wakeup
  • xtimer_periodic_wakeup

Note

• arduino-duemilanove:
  • errors seems similar to arduino-uno, so I do not repeat them

General issues

• micro-ecc-with-hwrng:
  • nrf52840dk maybe just a timeout? or not hwrng ?
• pthread_cooperation:
  • nucleo-f070rb
• warn_conflict:
  • stm32discovery: test target fails... 'attempt to rename spec'
    • also maybe it should not be part of 'test' but a 'compile_test'
• trickle:
  • arduino-mega2560: stuck after TRICKLE_RESET (known issue ?)
  • arduino-uno: stuck after TRICKLE_RESET (known issue ?)
  • stm32discovery: test failure detected by firmware
• micro-ecc:
  • arduino-mega2560: is not compatible with 8 bit I think? or too slow
  • arduino-uno: is not compatible with 8 bit I think? or too slow
• gnrc_sock_udp:
  • arduino-mega2560: node reboots in the middle of sock_udp_recv
• gnrc_sock_ip:
  • mega-explained: timeout on a line that has been printed...
• pipe:
  • arduino-mega2560: output is strange...
  • arduino-uno: output is strange...
• pkg_jsmn:
  • arduino-mega2560: output is strange...
  • arduino-uno: output is strange...
• bitarithm_msb:
  • arduino-mega2560: test timeouts?
  • arduino-uno: test timeouts?
  • mega-explained: timeout before 'Start'
• rng:
  • arduino-mega2560: nodes reboots in the middle of the test or start before
    being ready
• shell:
  • arduino-duemilanove: error after 'reboot'
• ps_schedstatistics:
  • arduino-mega2560: prints 'creating thread #XX' but fails at showing '>'
• periph_timer:
  • arduino-mega2560: TIMER_0/1: ERROR on initialization
  • arduino-uno: TIMER_0/1: ERROR on initialization
• posix_semaphore:
  • arduino-mega2560: 'waited too long 3999744 usec' for a 1 second sleep

[miri64]

pkg/lwip: incompatible cast with gcc [-Werror=cast-function-type]

That should have been fixed by RIOT-OS/RIOT#9588, but maybe there is another one I did not know about?

[cladmi]

pkg/lwip: incompatible cast with gcc [-Werror=cast-function-type]

hat should have been fixed by RIOT-OS/RIOT#9588, but maybe there is another one I did not know about?

In master yes, not on the release branch :) I should have said it in the output.

[miri64]

Mh... I believed it to be on the release branch as well. Should I provide a backport?

[cladmi]

No its ok, no worry :)

[miri64]

We are on 2018.10-RC2 already, so this one can be closed ;-).