murdock: enable esp32-wroom-32 for CI testing #11449

kaspar030 · 2019-04-25T15:02:32Z

Contribution description

I've added three esp32-wroom-32 boards to the CI. This PR enables them for testing.

Testing procedure

Let CI run with "run tests" set.

Issues/PRs references

none

kaspar030 · 2019-04-25T15:40:35Z

hrmpf, the reset circuitry isn't working properly. the boards needs capacitors between EN and GND. :(

gschorcht · 2019-04-25T16:21:27Z

tests/shell/tests/01-run.py

-    '\t  1 | pending  Q |  15',
-    '\t  2 | running  Q |   7'
+    r'\s+pid | state    Q | pri',
+    r'\s+1 | pending  Q |\s+\d+',


Is this related to ESP32? SCHED_PRIO_LEVELS seems to be not architecture specific but a common variable that might change from application to application as well.

This is not ESP32 related. I'll cut out the commit into another PR.

the esp Makefile states # if any WiFi interface is used, the number of priority levels has to be 32.

Do you remember the reason why?

A couple of tests fail because the priorities change with SCHED_PRIO_LEVELS=32.

BTW, I have seen that there is still an inconsistency in cpu/esp32/Makefile.include. SCHED_PRIO_LEVELS is always set to 32.

CFLAGS += -DSCHED_PRIO_LEVELS=32 ... # if any WiFi interface is used, the number of priority levels has to be 32 ifneq (,$(filter esp_wifi_any,$(USEMODULE))) CFLAGS += -DSCHED_PRIO_LEVELS=32 endif

I took a look to the tests that failed. All of them are tests that don't require module esp_wifi. So, I would prefer to reactivate the conditional setting of SCHED_PRIO_LEVELS.

CFLAGS += -DSCHED_PRIO_LEVELS=32 -CFLAGS += -DSCHED_PRIO_LEVELS=32 ... # if any WiFi interface is used, the number of priority levels has to be 32 ifneq (,$(filter esp_wifi_any,$(USEMODULE))) CFLAGS += -DSCHED_PRIO_LEVELS=32 endif This should help the tests which insist on certain PIDs and will work for networking applications as before.

So, I would prefer to reactivate the conditional setting of SCHED_PRIO_LEVELS.

ok!

This did the trick for most tests.

The ps regex is currently wrong, as | is a regex character. It should be escaped to have a proper test.

These changes are not necessary any longer. After changing the thread priority levels in PR #12753 everything is working without changes.

gschorcht · 2019-04-25T16:23:32Z

cpu/esp32/Makefile.include

@@ -140,6 +140,8 @@ LINKFLAGS += -Wl,--warn-unresolved-symbols
 FLASH_MODE ?= dout  # FIX configuration, DO NOT CHANGE
 FLASH_FREQ  = 40m   # FIX configuration, DO NOT CHANGE
 FLASH_SIZE ?= 2MB
+FLASHFILE ?= $(ELFFILE)


For which purpose is this define required? Is it something used by murdock?

Yes, the "test-murdock" target needs to know which file to send to the RasPi worker nodes.

When setting FLASHFILE, the flash/preflash targets should actually be using FLASHFILE instead of ELFFILE. I have two commits in #8838 that I can PR.

This now has its PR #11648

boards/common/esp32/Makefile.include

gschorcht · 2019-04-25T16:26:22Z

boards/common/esp32/Makefile.include

+  # used by Murdock
+  export RESET = dtr $(PORT) 0 && dtr $(PORT) 1
+else
+  export RESET ?= esptool.py --before default_reset run


Doesn't it do the job you need?

Unfortunately not, it doesn't work with the "make test" target. In that use-case, the test terminal program has the tty open, then "make reset" is called. esptool tries to connect to the bootloader instead of just resetting via RTS/DTR, and fails because the test terminal is hijacking the communication.

Also, on the RPi workers, spawning python takes seconds. ;)

Is there not a python alternative for RIOT that could be included in the repo ?

I mean, flashing is done with python, the tests are run with python, so using python for this would be easier than having to install external programs.

And the RPI could use its own C implementation anyway by overwriting RESET and RESET_FLAGS.

Ideally local testing is not much different from what murdock does. And for Murdock, spawning python is slow. We should maybe just integrate the (5 line) c tool into the RIOT repo.

gschorcht · 2019-04-25T16:35:27Z

hrmpf, the reset circuitry isn't working properly. the boards needs capacitors between EN and GND. :(

I guess it depends on the board you really have. The original ESP32 DevKitC V4 which is the master for all clones with ESP32-WROOM-32 modules has a capacitor between EN and GND.

kaspar030 · 2019-04-26T10:50:06Z

I guess it depends on the board you really have.

They look like this, but I got them cheaply from China, so they might be clones. The printing on the backside is a little off.

The original ESP32 DevKitC V4 which is the master for all clones with ESP32-WROOM-32 modules has a capacitor between EN and GND.

I tried adding one, but only had 100uF caps lying around. Weirdly, esptool.py can now properly change to boot mode, reset or flash the device and get it's chip id etc. but the user code doesn't run anymore (board outputs only the garbage as seen in this PR's failed tests).

gschorcht · 2019-04-26T11:42:55Z

but the user code doesn't run anymore (board outputs only the garbage as seen in this PR's failed tests).

How exactly looks the garbage? Maybe, there is a timing problem when changing the system clock and the UART hasn't the right baudrate anymore.

kaspar030 · 2019-04-29T08:26:50Z

How exactly looks the garbage? Maybe, there is a timing problem when changing the system clock and the UART hasn't the right baudrate anymore.

That's how it looked (wrong baudrate). Anyhow, I changed the caps to 2.2uF (from 100uF) and now it seems to work fine! Let's see how many tests run through...

gschorcht · 2019-04-29T14:41:04Z

That's how it looked (wrong baudrate). Anyhow, I changed the caps to 2.2uF (from 100uF) and now it seems to work fine! Let's see how many tests run through...

So was just the timing.

kaspar030 · 2019-04-30T09:47:27Z

I've pulled out a general fix for tests/posix_semaphore in #11467.

kaspar030 · 2019-05-06T11:28:17Z

@gschorcht could you maybe take over this PR? I think all the CI specific parts should be working. Some tests are still failing for reasons you might be better suited to fix...

gschorcht · 2019-05-24T00:04:44Z

@kaspar030 I took a look to the remaining tests that fail. For most of them it is not really clear to me why they fail. Even though ESP32 produce the right output, the python script runs into timeout. For example, tests/thread_msg_block_w_queue gives the following results:

Starting RIOT kernel on PRO cpu
I (296) [main_trampoline]: main(): This is RIOT! (Version: buildtest)
sender_thread start
main thread alive

Timeout in expect script at "child.expect('sender_thread start\r\n')" (tests/thread_msg_block_w_queue/tests/01-run.py:14)

Even though the test application prints out sender_thread start, the python script gets a timeout while waiting for it. This is the same for tests/thread_msg_block_wo_queue, tests/thread_basic and even tests/xtimer_usleep_short:

Starting RIOT kernel on PRO cpu
I (295) [main_trampoline]: main(): This is RIOT! (Version: buildtest)
This test will call xtimer_usleep for values from 500 down to 0
going to sleep 500 usecs...
going to sleep 499 usecs...
...
going to sleep 0 usecs...
[SUCCESS]

Timeout in expect script at "child.expect(u"This test will call xtimer_usleep for values from \\d+ down to \\d+\r\n")" (tests/xtimer_usleep_short/tests/01-run.py:15)

tests/posix_semaphore failed since the timer takes 1000102 us instead of 1000000 us due the 100 us margin. This problem is solved by your PR #11467.

tests/ssp fails because the compiler optimizes out the whole test_funcfunction 😟 Maybe the best way is to blacklist ESP32 boards for this test.

kaspar030 · 2019-07-04T12:59:51Z

Even though the test application prints out sender_thread start, the python script gets a timeout while waiting for it. This is the same for tests/thread_msg_block_wo_queue, tests/thread_basic and even tests/xtimer_usleep_short:

might be the line endings (\r\n). I'll try using $ instead.

kaspar030 · 2019-08-15T14:44:52Z

might be the line endings (\r\n).

This really does seem line ending related. Somehow sometimes there's a "0x0a" missing.

gschorcht · 2019-11-28T15:55:48Z

@kaspar030 CI tests succeed. The PR still needs squashing.

kaspar030 · 2019-11-29T09:44:46Z

The PR still needs squashing.

#12816 is just temporarily merged here, but not merged yet. need to wait...

gschorcht · 2019-11-29T13:57:18Z

The PR still needs squashing.

#12816 is just temporarily merged here, but not merged yet. need to wait...

Yes, of course. For any reason, I thought PR #12816 has been merged already, my fault.

kaspar030 · 2019-12-05T13:18:55Z

#12816 is just temporarily merged here, but not merged yet. need to wait...

#12816 is merged, I rebased this one.

kaspar030 · 2019-12-05T14:00:31Z

tests/pkg_spiffs failed with timeout...

gschorcht · 2019-12-05T14:10:36Z

tests/pkg_spiffs failed with timeout...

It doesn't seem to be related to the test. The log only shows

dwqw: command timed out

whatever it means.

gschorcht · 2019-12-05T14:12:13Z

It failed again on pi-6356c652.

kaspar030 · 2019-12-05T14:12:30Z

dwqw: command timed out
whatever it means.

That means the test took longer than 5 minutes to complete and has been cancelled. Currently Murdock cannot capture the test output in that case.

gschorcht · 2019-12-05T14:16:21Z

dwqw: command timed out
whatever it means.
That means the test took longer than 5 minutes to complete and has been cancelled. Currently Murdock cannot capture the test output in that case.

I see. Might it be a problem of the ESP32 node on pi-6356c652. This was the node which also had the problems the last time. The test itself should work after we increased the timeout with commit 47a4ff0.

kaspar030 · 2019-12-05T14:18:50Z

I see. Might it be a problem of the ESP32 node on pi-6356c652. This was the node which also had the problem the last time.

Well, the ESP is a new one. Lets hope that Pi isn't eating esp boards...

Here's the local uart output of the test application:

Help: Press s to start test, r to print it is ready
START
main(): This is RIOT! (Version: 2020.01-devel-1274-g2b934d)
EXCEPTION!! exccause=0 (IllegalInstructionCause) @800df20d excvaddr=00000000


heap: 184120 (free 183292) byte

register set
pc      : 401238e8      ps      : 00060130      exccause: 00000000      excvaddr: 00000000
epc1    : 401238e8      epc2    : 00000000      epc3    : 00000000      epc4    : 00000000
a0      : 800df20d      a1      : 3ffb2800      a2      : 3ffb2894      a3      : 3ffb2870
a4      : 00000001      a5      : 3ffb28a8      a6      : 00000000      a7      : 00000001
a8      : 800da710      a9      : 3ffb2800      a10     : 3ffb289c      a11     : 3ffb2800
a12     : 3ffb289c      a13     : 3ffb107c      a14     : 00000000      a15     : 00000001
lbeg    : 40084bc8      lend    : 40084bd3      lcount  : 00000000
ets Jun  8 2016 00:22:57

rst:0x7 (TG0WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DOUT, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:4004
load:0x40078000,len:7304
load:0x40080000,len:7016
entry 0x40080344
W (29) boot: PRO CPU has been reset by WDT.
W (29) boot: WDT reset info: PRO CPU PC=0x40082e04
W (29) boot: WDT reset info: APP CPU PC=0x3ccc4d07

Help: Press s to start test, r to print it is ready

gschorcht · 2019-12-05T14:19:49Z

Ups, that's a crash. Let me check.

gschorcht · 2019-12-05T14:29:34Z

I can reproduce the crash which is new. SPIFFS was definitely working with the last tests.

gschorcht · 2019-12-05T14:56:18Z

It seems that the problem came in with the merge of PR #12810. Even though ESP32 flash driver was touched by this PR, the changes are only changes in log output during intialization. The problem doesn't occur when compiled with debug option. I have to investigate it further.

gschorcht · 2019-12-06T08:40:34Z

It seems that the problem came in with the merge of PR #12810. Even though ESP32 flash driver was touched by this PR, the changes are only changes in log output during intialization. The problem doesn't occur when compiled with debug option. I have to investigate it further.

This is one of the problems I really hate. The test works with debug compile option, it works in QEMU, it works when board configuration is printed at startup and it works if any printf is added to the test or the SPI flash driver. But it can't be used with OCD. How to figure out the problem under these conditions? Right now, I'm completely at a loss.

Sure, we could enable the output of the startup information as it was before as a workaround for this test. However, this would probably not solve the cause of the problem.

My guess is that the problem is caused by an asynchronous UART interrupt while the cache for the program code in SPI flash is disabled due to a write operation to the SPI flash drive.

gschorcht · 2019-12-06T10:45:17Z

@kaspar030 Problem solved. I had to place the code of VFS module in IRAM because the cache for program code in SPI flash memory is disabled during SPI flash write operations. PR #12890 fixes the problem.

I have also reduced the test timeouts for tests/pkg_spiffs and pkg_littlefs.

kaspar030 · 2019-12-06T11:45:07Z

@kaspar030 Problem solved.

Nice catch! I've rebased with #12890 merged.

gschorcht · 2019-12-06T12:13:43Z

Wow, all light are green. Do we wanna to merge it now?

kaspar030 · 2019-12-06T12:29:27Z

Wow, all light are green. Do we wanna to merge it now?

Yeah! I'll let maintainers@ know to watch out for esp32 CI results for a while.

kaspar030 · 2019-12-06T12:52:02Z

Good job @gschorcht!

kaspar030 requested review from smlng and gschorcht April 25, 2019 15:02

kaspar030 force-pushed the murdock_add_esp32-wroom-32 branch from a2c392e to 7e95b97 Compare April 25, 2019 15:05

kaspar030 added State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet and removed CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Apr 25, 2019

gschorcht reviewed Apr 25, 2019

View reviewed changes

boards/common/esp32/Makefile.include Outdated Show resolved Hide resolved

gschorcht reviewed Apr 25, 2019

View reviewed changes

kaspar030 added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Apr 29, 2019

kaspar030 mentioned this pull request Apr 30, 2019

tests/posix_semaphore: unify and increase allowed test4 margin #11467

Merged

gschorcht self-assigned this May 6, 2019

cladmi mentioned this pull request Jun 6, 2019

cpu/esp*: use FLASHFILE for esp32 and esp8266 boards #11648

Merged

kaspar030 force-pushed the murdock_add_esp32-wroom-32 branch from 4a11c1e to cf786ee Compare July 4, 2019 12:59

kaspar030 force-pushed the murdock_add_esp32-wroom-32 branch from 67ac0a9 to b5a4841 Compare August 15, 2019 13:44

kaspar030 force-pushed the murdock_add_esp32-wroom-32 branch from aa9fdf7 to 2bbb504 Compare December 5, 2019 13:12

kaspar030 removed the State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet label Dec 5, 2019

gschorcht mentioned this pull request Dec 6, 2019

cpu/esp*: fixes for tests/pkg_spiffs and tests/pkg_littlefs #12890

Merged

kaspar030 added 2 commits December 6, 2019 12:42

murdock: enable on-hardware tests on esp32-wroom-32

a7c9079

tests/driver_hd44780: disable CI for esp32-wroom-32

f88d55c

kaspar030 force-pushed the murdock_add_esp32-wroom-32 branch from 2bbb504 to f88d55c Compare December 6, 2019 11:43

gschorcht approved these changes Dec 6, 2019

View reviewed changes

kaspar030 merged commit 220d642 into RIOT-OS:master Dec 6, 2019

kaspar030 deleted the murdock_add_esp32-wroom-32 branch December 6, 2019 12:29

fjmolinas added this to the Release 2020.01 milestone Dec 13, 2019

remiphilippe mentioned this pull request Jul 14, 2020

mdt_erase success, but vfs_format resets board (esp32-heltec-lora32-v2) #14506

Open

murdock: enable esp32-wroom-32 for CI testing #11449

murdock: enable esp32-wroom-32 for CI testing #11449

Conversation

kaspar030 commented Apr 25, 2019

Contribution description

Testing procedure

Issues/PRs references

kaspar030 commented Apr 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gschorcht commented Apr 25, 2019

kaspar030 commented Apr 26, 2019

gschorcht commented Apr 26, 2019

kaspar030 commented Apr 29, 2019

gschorcht commented Apr 29, 2019

kaspar030 commented Apr 30, 2019

kaspar030 commented May 6, 2019

gschorcht commented May 24, 2019 • edited Loading

kaspar030 commented Jul 4, 2019

kaspar030 commented Aug 15, 2019

gschorcht commented Nov 28, 2019 • edited Loading

kaspar030 commented Nov 29, 2019

gschorcht commented Nov 29, 2019

kaspar030 commented Dec 5, 2019

kaspar030 commented Dec 5, 2019

gschorcht commented Dec 5, 2019

gschorcht commented Dec 5, 2019

kaspar030 commented Dec 5, 2019

gschorcht commented Dec 5, 2019 • edited Loading

kaspar030 commented Dec 5, 2019

gschorcht commented Dec 5, 2019

gschorcht commented Dec 5, 2019

gschorcht commented Dec 5, 2019 • edited Loading

gschorcht commented Dec 6, 2019

gschorcht commented Dec 6, 2019 • edited Loading

kaspar030 commented Dec 6, 2019

gschorcht commented Dec 6, 2019

kaspar030 commented Dec 6, 2019

kaspar030 commented Dec 6, 2019

gschorcht commented May 24, 2019 •

edited

Loading

gschorcht commented Nov 28, 2019 •

edited

Loading

gschorcht commented Dec 5, 2019 •

edited

Loading

gschorcht commented Dec 5, 2019 •

edited

Loading

gschorcht commented Dec 6, 2019 •

edited

Loading