Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

_NVIC_SystemReset stuck in infinite loop when calling pm_reboot through shell after flashing with J-Link #13044

Open
tinstructor opened this issue Jan 8, 2020 · 19 comments
Open

_NVIC_SystemReset stuck in infinite loop when calling pm_reboot through shell after flashing with J-Link #13044

tinstructor opened this issue Jan 8, 2020 · 19 comments
Labels
Area: boards Area: Board ports Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors)

Comments

@tinstructor
Copy link

tinstructor commented Jan 8, 2020 •
edited
Loading

For some curious reason, after flashing several openmote-b nodes with a J-Link programmer, trying to reboot said nodes via the shell "reboot" command causes those nodes to hang forever. I verified that the infinite for loop in cpu/cortexm_common/include/vendor/core_cm3.h is reached as it's supposed to while awaiting a reset. However, this reset actually never occurs, causing the node to hang forever. I also tested this with openmote-b nodes that where only ever flashed via USB and they work just fine. This issue persists even after disconnecting the J-Link programmer and, worse, after reflashing the node via USB. As such I suspect that flashing over J-Link writes to certain registers that are otherwise untouched when flashing over USB.

__NO_RETURN __STATIC_INLINE void __NVIC_SystemReset(void)
{
  __DSB();                                                          /* Ensure all outstanding memory accesses included
                                                                       buffered write are completed before reset */
  SCB->AIRCR  = (uint32_t)((0x5FAUL << SCB_AIRCR_VECTKEY_Pos)    |
                           (SCB->AIRCR & SCB_AIRCR_PRIGROUP_Msk) |
                            SCB_AIRCR_SYSRESETREQ_Msk    );         /* Keep priority group unchanged */
  __DSB();                                                          /* Ensure completion of memory access */

  for(;;)                                                           /* wait until reset */
  {
    /* this loop is called infinitely until reset, which never 
       happens and thus causes the node to hang forever */
    __NOP();
  }
}
@fjmolinas fjmolinas added Area: boards Area: Board ports Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) labels Jan 8, 2020
@tinstructor
Copy link
Author

This post might be worth looking into. Is the nRST pin somehow being pulled high (permanently)?

@tinstructor
Copy link
Author

@benpicco I know you've got extensive knowledge on this platform. Any idea as to what might cause this issue?

@tinstructor
Copy link
Author

tinstructor commented Jan 13, 2020 •
edited
Loading

Final try: @haukepetersen and @MrKevinWeiss do you have any idea what may cause this issue?

@fjmolinas
Copy link
Contributor

@tinstructor can you give more details on how this issue surfaced to try to reproduce. Do I need to flash multiple times for it to show up? I have flashed a couple without experiencing your issue. My Jlink version is

SEGGER J-Link Commander V6.56d (Compiled Dec 12 2019 13:05:00)
DLL version V6.56d, compiled Dec 12 2019 13:04:51

@tinstructor
Copy link
Author

My Jlink version is

SEGGER J-Link Commander V6.60c (Compiled Dec 23 2019 16:18:17)
DLL version V6.60c, compiled Dec 23 2019 16:17:59

Ok so what I did is, I flashed the openmote-b "successfully" via J-link with the following command:

PROGRAMMER=jlink make flash BOARD=openmote-b

and when I subsequently login to the shell (after disconnecting the Jlink flash tool) via make term PORT=/dev/ttyUSB1 BOARD=openmote-b, this happens:

Twisted not available, please install it if you want to use pyterm's JSON capabilities
2020-01-13 13:28:09,106 # Connect to serial port /dev/ttyUSB1
Welcome to pyterm!
Type '/exit' to exit.
help
2020-01-13 13:28:12,776 # help
2020-01-13 13:28:12,777 # Command              Description
2020-01-13 13:28:12,777 # ---------------------------------------
2020-01-13 13:28:12,778 # reboot               Reboot the node
2020-01-13 13:28:12,792 # ps                   Prints information about running threads.
2020-01-13 13:28:12,793 # random_init          initializes the PRNG
2020-01-13 13:28:12,793 # random_get           returns 32 bit of pseudo randomness
2020-01-13 13:28:12,794 # ifconfig             Configure network interfaces
> reboot
2020-01-13 13:28:19,992 #  reboot

The node then hangs forever until I manually perform a reset with the hardware reset button.

@fjmolinas
Copy link
Contributor

What application are you uploading? I still can't reproduce, I will try your JLkin version as well:

PROGRAMMER=jlink BOARD=openmote-b make -C examples/default/ flash-only 
/home/francisco/workspace/RIOT/dist/tools/jlink/jlink.sh flash /home/francisco/workspace/RIOT/examples/default/bin/openmote-b/default.bin
### Flashing Target ###
### Flashing at base address 0x200000 with offset 0 ###
SEGGER J-Link Commander V6.56d (Compiled Dec 12 2019 13:05:00)
DLL version V6.56d, compiled Dec 12 2019 13:04:51

J-Link Commander will now exit on Error

J-Link Command File read successfully.
Processing script file...

J-Link connection not established yet but required for command.
Connecting to J-Link via USB...O.K.
Firmware: J-Link EDU Mini V1 compiled Dec 13 2019 11:16:57
Hardware version: V1.00
S/N: 801014916
License(s): FlashBP, GDB
VTref=2.472V
Target connection not established yet but required for command.
Device "CC2538SF53" selected.


Connecting to target via JTAG
InitTarget: Found ICE-Pick with ID: 0x8B96402F
InitTarget: Found CPU TAP 0x4BA00477
Scanning AP map to find all available APs
AP[1]: Stopped AP scan as end of AP map has been reached
AP[0]: AHB-AP (IDR: 0x24770011)
Iterating through AP map to find AHB-AP to use
AP[0]: Core found
AP[0]: AHB-AP ROM base: 0xE00FF000
CPUID register: 0x412FC230. Implementer code: 0x41 (ARM)
Found Cortex-M3 r2p0, Little endian.
FPUnit: 6 code (BP) slots and 2 literal slots
CoreSight components:
ROMTbl[0] @ E00FF000
ROMTbl[0][0]: E000E000, CID: B105E00D, PID: 002BB000 SCS
ROMTbl[0][1]: E0001000, CID: B105E00D, PID: 002BB002 DWT
ROMTbl[0][2]: E0002000, CID: B105E00D, PID: 002BB003 FPB
ROMTbl[0][3]: E0000000, CID: B105E00D, PID: 002BB001 ITM
ROMTbl[0][4]: E0040000, CID: B105900D, PID: 002BB923 TPIU-Lite
Cortex-M3 identified.
Halting CPU for downloading file.
Downloading file [/home/francisco/workspace/RIOT/examples/default/bin/openmote-b/default.bin]...
Comparing flash   [100%] Done.
J-Link: Flash download: Bank 0 @ 0x00200000: Skipped. Contents already match
O.K.


Sleep(100)



Script processing completed.
BOARD=openmote-b make -C examples/default/ term 
make: Entering directory '/home/francisco/workspace/RIOT/examples/default'
/home/francisco/workspace/RIOT/dist/tools/pyterm/pyterm -p "/dev/riot/tty-openmote-b" -b "115200" 
2020-01-13 13:46:51,404 # Connect to serial port /dev/riot/tty-openmote-b
helpWelcome to pyterm!
Type '/exit' to exit.
help
2020-01-13 13:46:52,619 # help
2020-01-13 13:46:52,621 # Command              Description
2020-01-13 13:46:52,621 # ---------------------------------------
2020-01-13 13:46:52,622 # reboot               Reboot the node
2020-01-13 13:46:52,636 # ps                   Prints information about running threads.
2020-01-13 13:46:52,638 # random_init          initializes the PRNG
2020-01-13 13:46:52,639 # random_get           returns 32 bit of pseudo randomness
2020-01-13 13:46:52,641 # ifconfig             Configure network interfaces
2020-01-13 13:46:52,653 # txtsnd               Sends a custom string as is over the link layer
2020-01-13 13:46:52,655 # saul                 interact with sensors and actuators using SAUL
> reboot
2020-01-13 13:46:53,707 #  reboot
2020-01-13 13:46:53,723 # main(): This is RIOT! (Version: 2020.01-devel-1726-g19774-HEAD)
2020-01-13 13:46:53,724 # Welcome to RIOT!
> reboot
2020-01-13 13:46:54,731 #  reboot
2020-01-13 13:46:54,732 # main(): This is RIOT! (Version: 2020.01-devel-1726-g19774-HEAD)
2020-01-13 13:46:54,747 # Welcome to RIOT!
>

@tinstructor
Copy link
Author

@fjmolinas This is the output of the flash command:

### Flashing Target ###
### Flashing at base address 0x200000 with offset 0 ###
SEGGER J-Link Commander V6.60c (Compiled Dec 23 2019 16:18:17)
DLL version V6.60c, compiled Dec 23 2019 16:17:59

J-Link Commander will now exit on Error

J-Link Command File read successfully.
Processing script file...

J-Link connection not established yet but required for command.
Connecting to J-Link via USB...O.K.
Firmware: J-Link EDU Mini V1 compiled Dec 13 2019 11:16:57
Hardware version: V1.00
S/N: 801009716
License(s): FlashBP, GDB
VTref=2.475V
Target connection not established yet but required for command.
Device "CC2538SF53" selected.


Connecting to target via JTAG
InitTarget: Found ICE-Pick with ID: 0x8B96402F
InitTarget: Found CPU TAP 0x4BA00477
Scanning AP map to find all available APs
AP[1]: Stopped AP scan as end of AP map has been reached
AP[0]: AHB-AP (IDR: 0x24770011)
Iterating through AP map to find AHB-AP to use
AP[0]: Core found
AP[0]: AHB-AP ROM base: 0xE00FF000
CPUID register: 0x412FC230. Implementer code: 0x41 (ARM)
Found Cortex-M3 r2p0, Little endian.
FPUnit: 6 code (BP) slots and 2 literal slots
CoreSight components:
ROMTbl[0] @ E00FF000
ROMTbl[0][0]: E000E000, CID: B105E00D, PID: 002BB000 SCS
ROMTbl[0][1]: E0001000, CID: B105E00D, PID: 002BB002 DWT
ROMTbl[0][2]: E0002000, CID: B105E00D, PID: 002BB003 FPB
ROMTbl[0][3]: E0000000, CID: B105E00D, PID: 002BB001 ITM
ROMTbl[0][4]: E0040000, CID: B105900D, PID: 002BB923 TPIU-Lite
Cortex-M3 identified.
Halting CPU for downloading file.
Downloading file [/home/relsas/RIOT-benpicco/examples/interference/bin/openmote-b/interference.bin]...
Comparing flash   [100%] Done.
Erasing flash     [100%] Done.
Programming flash [100%] Done.
Verifying flash   [100%] Done.
J-Link: Flash download: Bank 0 @ 0x00200000: 2 ranges affected (481280 bytes)
J-Link: Flash download: Total time needed: 5.891s (Prepare: 0.061s, Compare: 0.281s, Erase: 0.085s, Program: 5.370s, Verify: 0.075s, Restore: 0.017s)
O.K.


Sleep(100)



Script processing completed.

The example I'm flashing is something I made myself: https://github.com/tinstructor/RIOT/tree/interfere/examples/interference

@tinstructor
Copy link
Author

You'll need to modify the flash command a little bit though:

PROGRAMMER=jlink make flash BOARD=openmote-b GNRC_NETIF_NUMOF=1

@fjmolinas
Copy link
Contributor

@tinstructor is your issue reproducible with master?

@tinstructor
Copy link
Author

@fjmolinas which application do you suggest I try?

@fjmolinas
Copy link
Contributor

@fjmolinas which application do you suggest I try?

examples/default or tests/shell since they have reboot

@fjmolinas
Copy link
Contributor

@tinstructor I check-outed your branch, still cant reproduce.

Could you put the output of ./dist/tools/ci/print_toolchain_versions.sh. Also can you use BUILD_IN_DOCKER=1?

@tinstructor
Copy link
Author

@fjmolinas I tried examples/default and I still encounter the same issue:

relsas@relsas-VirtualBox:~/RIOT-benpicco/examples/default$ PROGRAMMER=jlink make flash BOARD=openmote-b
Building application "default" for "openmote-b" with MCU "cc2538".

"make" -C /home/relsas/RIOT-benpicco/boards/openmote-b
"make" -C /home/relsas/RIOT-benpicco/core
"make" -C /home/relsas/RIOT-benpicco/cpu/cc2538
"make" -C /home/relsas/RIOT-benpicco/cpu/cc2538/periph
"make" -C /home/relsas/RIOT-benpicco/cpu/cortexm_common
"make" -C /home/relsas/RIOT-benpicco/cpu/cortexm_common/periph
"make" -C /home/relsas/RIOT-benpicco/drivers
"make" -C /home/relsas/RIOT-benpicco/drivers/periph_common
"make" -C /home/relsas/RIOT-benpicco/drivers/saul
"make" -C /home/relsas/RIOT-benpicco/sys
"make" -C /home/relsas/RIOT-benpicco/sys/auto_init
"make" -C /home/relsas/RIOT-benpicco/sys/auto_init/saul
"make" -C /home/relsas/RIOT-benpicco/sys/fmt
"make" -C /home/relsas/RIOT-benpicco/sys/isrpipe
"make" -C /home/relsas/RIOT-benpicco/sys/newlib_syscalls_default
"make" -C /home/relsas/RIOT-benpicco/sys/phydat
"make" -C /home/relsas/RIOT-benpicco/sys/ps
"make" -C /home/relsas/RIOT-benpicco/sys/saul_reg
"make" -C /home/relsas/RIOT-benpicco/sys/shell
"make" -C /home/relsas/RIOT-benpicco/sys/shell/commands
"make" -C /home/relsas/RIOT-benpicco/sys/stdio_uart
"make" -C /home/relsas/RIOT-benpicco/sys/tsrb
   text    data     bss     dec     hex filename
  14936     136    2620   17692    451c /home/relsas/RIOT-benpicco/examples/default/bin/openmote-b/default.elf
/home/relsas/RIOT-benpicco/dist/tools/jlink/jlink.sh flash /home/relsas/RIOT-benpicco/examples/default/bin/openmote-b/default.bin
### Flashing Target ###
### Flashing at base address 0x200000 with offset 0 ###
SEGGER J-Link Commander V6.60c (Compiled Dec 23 2019 16:18:17)
DLL version V6.60c, compiled Dec 23 2019 16:17:59

J-Link Commander will now exit on Error

J-Link Command File read successfully.
Processing script file...

J-Link connection not established yet but required for command.
Connecting to J-Link via USB...O.K.
Firmware: J-Link EDU Mini V1 compiled Dec 13 2019 11:16:57
Hardware version: V1.00
S/N: 801009716
License(s): FlashBP, GDB
VTref=2.476V
Target connection not established yet but required for command.
Device "CC2538SF53" selected.


Connecting to target via JTAG
InitTarget: Found ICE-Pick with ID: 0x8B96402F
InitTarget: Found CPU TAP 0x4BA00477
Scanning AP map to find all available APs
AP[1]: Stopped AP scan as end of AP map has been reached
AP[0]: AHB-AP (IDR: 0x24770011)
Iterating through AP map to find AHB-AP to use
AP[0]: Core found
AP[0]: AHB-AP ROM base: 0xE00FF000
CPUID register: 0x412FC230. Implementer code: 0x41 (ARM)
Found Cortex-M3 r2p0, Little endian.
FPUnit: 6 code (BP) slots and 2 literal slots
CoreSight components:
ROMTbl[0] @ E00FF000
ROMTbl[0][0]: E000E000, CID: B105E00D, PID: 002BB000 SCS
ROMTbl[0][1]: E0001000, CID: B105E00D, PID: 002BB002 DWT
ROMTbl[0][2]: E0002000, CID: B105E00D, PID: 002BB003 FPB
ROMTbl[0][3]: E0000000, CID: B105E00D, PID: 002BB001 ITM
ROMTbl[0][4]: E0040000, CID: B105900D, PID: 002BB923 TPIU-Lite
Cortex-M3 identified.
Halting CPU for downloading file.
Downloading file [/home/relsas/RIOT-benpicco/examples/default/bin/openmote-b/default.bin]...
Comparing flash   [100%] Done.
Erasing flash     [100%] Done.
Programming flash [100%] Done.
Verifying flash   [100%] Done.
J-Link: Flash download: Bank 0 @ 0x00200000: 1 range affected (47104 bytes)
J-Link: Flash download: Total time needed: 2.507s (Prepare: 0.060s, Compare: 1.367s, Erase: 0.513s, Program: 0.541s, Verify: 0.005s, Restore: 0.017s)
O.K.


Sleep(100)



Script processing completed.

relsas@relsas-VirtualBox:~/RIOT-benpicco/examples/default$ make term PORT=/dev/ttyUSB1 BOARD=openmote-b
/home/relsas/RIOT-benpicco/dist/tools/pyterm/pyterm -p "/dev/ttyUSB1" -b "115200" 
Twisted not available, please install it if you want to use pyterm's JSON capabilities
2020-01-13 13:57:48,533 # Connect to serial port /dev/ttyUSB1
Welcome to pyterm!
Type '/exit' to exit.
reboot
2020-01-13 13:57:51,581 # reboot

@tinstructor
Copy link
Author

The output of ./dist/tools/ci/print_toolchain_versions.sh is:

relsas@relsas-VirtualBox:~/RIOT-benpicco$ ./dist/tools/ci/print_toolchain_versions.sh

Operating System Environment
-----------------------------
       Operating System: "Ubuntu" "18.04.3 LTS (Bionic Beaver)"
                 Kernel: Linux 5.0.0-37-generic x86_64 x86_64

Installed compiler toolchains
-----------------------------
             native gcc: gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
      arm-none-eabi-gcc: arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 8-2018-q4-major) 8.2.1 20181213 (release) [gcc-8-branch revision 267074]
                avr-gcc: missing
       mips-mti-elf-gcc: missing
             msp430-gcc: missing
   riscv-none-embed-gcc: missing
   xtensa-esp32-elf-gcc: missing
   xtensa-lx106-elf-gcc: missing
                  clang: missing

Installed compiler libs
-----------------------
   arm-none-eabi-newlib: "3.0.0"
    mips-mti-elf-newlib: missing
riscv-none-embed-newlib: missing
xtensa-esp32-elf-newlib: missing
xtensa-lx106-elf-newlib: missing
               avr-libc: missing (missing)

Installed development tools
---------------------------
                  cmake: cmake version 3.10.2
               cppcheck: missing
                doxygen: 1.8.13
                    git: git version 2.17.1
                   make: GNU Make 4.1
                openocd: Open On-Chip Debugger 0.10.0+dev-00932-g85a460d5 (2019-09-23-11:58)
                 python: Python 2.7.17
                python2: Python 2.7.17
                python3: Python 3.6.8
                 flake8: error: /usr/bin/python3: No module named flake8
             coccinelle: missing

@fjmolinas
Copy link
Contributor

@aabadie can you give this a try and reproduce?

@tinstructor
Copy link
Author

tinstructor commented Jan 13, 2020 •
edited
Loading

@fjmolinas I remember that I also had to put CCA_BACKDOOR_ENABLE to 0 (in boards/openmote-b/include/board.h) in order to unbrick the device after a previous person had used it. However, ever since I've put it back to 1. It might be that I've mistakenly left it unchanged for the other nodes (they are all reflashed with the parameter set to 1 as of today though) and so that might've affected something?

/**
 * @name    Flash Customer Configuration Area (CCA) parameters
 * @{
 */
#ifndef UPDATE_CCA
#define UPDATE_CCA                (1)
#endif

#define CCA_BACKDOOR_ENABLE       (1)
#define CCA_BACKDOOR_PORT_A_PIN   (6) /**< BSL_BOOT Pin */
#define CCA_BACKDOOR_ACTIVE_LEVEL (0) /**< Active low */

@tinstructor
Copy link
Author

@fjmolinas The issue seems to be gone when I put CCA_BACKDOOR_ENABLE to 0 again. However that's not supposed to be how that works I think.

@fjmolinas
Copy link
Contributor

@fjmolinas The issue seems to be gone when I put CCA_BACKDOOR_ENABLE to 0 again. However that's not supposed to be how that works I think.

I'll try to look into this more.. I don't have many ideas right now.

@MrKevinWeiss
Copy link
Contributor

I have struggled with the boot and nrst pins for a while now. I am pretty busy but after Wednesday I can dive deeper into it.

I will note that lockup does occur on some platforms when the nrst pin is held high as the program tries to pull the nrst down when rebooting and waits until it reads low (forever).

The nrst pin is controlled by an additional msp430 chip that is intended to standardize the signals from the ftdi RTS and DTR pins used to set the boot0 and nrst for bootloading over usb.

It may be something that in RIOT that has it initial state a push-pull instead of an open drain output with a pullup (as it should be for nrst pins).

If you guys don't solve it by the time I am free I will jump on it.

@miri64 miri64 added this to the Release 2020.07 milestone Jul 6, 2020
@MrKevinWeiss MrKevinWeiss removed this from the Release 2021.07 milestone Jul 15, 2021
@tinstructor tinstructor mentioned this issue Nov 30, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Area: boards Area: Board ports Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@miri64 @tinstructor @MrKevinWeiss @fjmolinas