Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v4.3.1 long time running crashed (IDFGH-6048) #7731

Closed
achao1104 opened this issue Oct 19, 2021 · 10 comments
Closed

v4.3.1 long time running crashed (IDFGH-6048) #7731

achao1104 opened this issue Oct 19, 2021 · 10 comments
Assignees
@o-marshmallow
Labels
Awaiting Response awaiting a response from the author Resolution: Done Issue is done internally Status: Done Issue is done internally

Comments

@achao1104
Copy link

Hello,

idf version is v4.3.1 and we used esp32s2 modules.
we have made up 3 devices, and keep them running for a long time, after around 30 hours running, one of them crashed.
we have checked the dump content, it looks it is not due to our applicaiton codes, please help.

D:\project\esp32\project\solardevice>python D:\project\esp32\v4.3.1\components\espcoredump\espcoredump.py info_corefile -t raw -c D:\project\esp32\temp\a.dump D:\project\esp32\project\solardevice\build\solardevice.elf
espcoredump.py v0.4-dev

==================== ESP32 CORE DUMP START ====================

Crashed task handle: 0x3ff9f36c, name: 'main', GDB name: 'process 1073345388'

================== CURRENT THREAD REGISTERS ===================
exccause 0x1d (StoreProhibitedCause)
excvaddr 0x0
epc1 0x0
epc2 0x0
epc3 0x0
epc4 0x0
epc5 0x0
epc6 0x0
epc7 0x0
eps2 0x0
eps3 0x0
eps4 0x0
eps5 0x0
eps6 0x0
eps7 0x40089320
pc 0x4002edb9 0x4002edb9 <panic_abort+21>
lbeg 0x0 0
lend 0x0 0
lcount 0x0 0
sar 0x8 8
ps 0x60c20 396320
threadptr
br
scompare1
acclo
acchi
m0
m1
m2
m3
expstate
f64r_lo
f64r_hi
f64s
fcr
fsr
a0 0x8002f58c -2147289716
a1 0x3ffdacb0 1073589424
a2 0x3ffdacf0 1073589488
a3 0x3ffdad3d 1073589565
a4 0xa 10
a5 0x3f00a435 1057006645
a6 0x40000aa0 1073744544
a7 0x0 0
a8 0x0 0
a9 0x1 1
a10 0x3ffdad3d 1073589565
a11 0x3ffdad3d 1073589565
a12 0xa 10
a13 0xa 10
a14 0x3ffdac99 1073589401
a15 0xff000000 -16777216

==================== CURRENT THREAD STACK =====================
#0 0x4002edb9 in panic_abort (details=0x3ffdacf0 "abort() was called at PC 0x40019fbe on core 0") at D:/project/esp32/idf-v4.3.1/components/esp_system/panic.c:379
#1 0x4002f58c in esp_system_abort (details=0x3ffdacf0 "abort() was called at PC 0x40019fbe on core 0") at D:/project/esp32/idf-v4.3.1/components/esp_system/system_api.c:112
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

======================== THREADS INFO =========================
Id Target Id Frame

  • 1 process 1073345388 0x4002edb9 in panic_abort (details=0x3ffdacf0 "abort() was called at PC 0x40019fbe on core 0") at D:/project/esp32/idf-v4.3.1/components/esp_system/panic.c:379
    2 process 1073347272 0x401038fa in esp_pm_impl_waiti () at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:202
    3 process 1073620612 0x40030042 in xQueueReceive (xQueue=0x3ffed26c, pvBuffer=0x3ffe2640, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
    4 process 1073657212 0x40030042 in xQueueReceive (xQueue=0x3ffee02c, pvBuffer=0x3fff42c0, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
    5 process 1073616168 vTaskDelay (xTicksToDelay=100) at D:/project/esp32/idf-v4.3.1/components/freertos/tasks.c:1530
    6 process 1073599144 0x40030042 in xQueueReceive (xQueue=0x3ffec4e8, pvBuffer=0x3ffdd270, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
    7 process 1073347616 0x40031d80 in prvProcessTimerOrBlockTask (xListWasEmpty=, xNextExpireTime=0) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
    8 process 1073611432 0x40030042 in xQueueReceive (xQueue=0x3ffecd48, pvBuffer=0x3ffe0278, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
    9 process 1073606988 0x40030042 in xQueueReceive (xQueue=0x3ffec93c, pvBuffer=0x3ffdf114, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
    10 process 1073602308 0x40030042 in xQueueReceive (xQueue=0x3ffec68c, pvBuffer=0x3ffdde90, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
    11 process 1073344660 0x4001a8d4 in ?? ()
    12 process 1073626328 0x40030042 in xQueueReceive (xQueue=0x3ffe283c, pvBuffer=0x3ffe3ca0, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33

==================== THREAD 1 (TCB: 0x3ff9f36c, name: 'main') =====================
#0 0x4002edb9 in panic_abort (details=0x3ffdacf0 "abort() was called at PC 0x40019fbe on core 0") at D:/project/esp32/idf-v4.3.1/components/esp_system/panic.c:379
#1 0x4002f58c in esp_system_abort (details=0x3ffdacf0 "abort() was called at PC 0x40019fbe on core 0") at D:/project/esp32/idf-v4.3.1/components/esp_system/system_api.c:112
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

==================== THREAD 2 (TCB: 0x3ff9fac8, name: 'IDLE') =====================
#0 0x401038fa in esp_pm_impl_waiti () at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:202
#1 0x400c310e in esp_vApplicationIdleHook () at D:/project/esp32/idf-v4.3.1/components/esp_common/src/freertos_hooks.c:63
#2 0x40030577 in prvIdleTask (pvParameters=) at D:/project/esp32/idf-v4.3.1/components/freertos/tasks.c:3839

==================== THREAD 3 (TCB: 0x3ffe2684, name: 'uart_event') =====================
#0 0x40030042 in xQueueReceive (xQueue=0x3ffed26c, pvBuffer=0x3ffe2640, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
#1 0x4009f588 in uart_event_task_entry (param=0x3ffeddf8) at ../components/modem/src/esp_modem.c:240

==================== THREAD 4 (TCB: 0x3ffeb57c, name: 'uart_event') =====================
#0 0x40030042 in xQueueReceive (xQueue=0x3ffee02c, pvBuffer=0x3fff42c0, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
#1 0x4009f588 in uart_event_task_entry (param=0x3fff2ae0) at ../components/modem/src/esp_modem.c:240

==================== THREAD 5 (TCB: 0x3ffe1528, name: 'cloud_action_ta') =====================
#0 vTaskDelay (xTicksToDelay=100) at D:/project/esp32/idf-v4.3.1/components/freertos/tasks.c:1530
#1 0x4008e5b1 in delay_ms (msnum=) at ../main/util.c:431
#2 0x4009438d in cloud_action_task (pvParameters=) at ../main/bizutil.c:67

==================== THREAD 6 (TCB: 0x3ffdd2a8, name: 'tiT') =====================
#0 0x40030042 in xQueueReceive (xQueue=0x3ffec4e8, pvBuffer=0x3ffdd270, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
#1 0x40036e08 in sys_arch_mbox_fetch (mbox=, msg=0x3ffdd270, timeout=500) at D:/project/esp32/idf-v4.3.1/components/lwip/port/esp32/freertos/sys_arch.c:325
#2 0x40037092 in tcpip_timeouts_mbox_fetch (mbox=, msg=) at D:/project/esp32/idf-v4.3.1/components/lwip/lwip/src/api/tcpip.c:110
#3 tcpip_thread (arg=) at D:/project/esp32/idf-v4.3.1/components/lwip/lwip/src/api/tcpip.c:148

==================== THREAD 7 (TCB: 0x3ff9fc20, name: 'Tmr Svc') =====================
#0 0x40031d80 in prvProcessTimerOrBlockTask (xListWasEmpty=, xNextExpireTime=0) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
#1 prvTimerTask (pvParameters=) at D:/project/esp32/idf-v4.3.1/components/freertos/timers.c:555

==================== THREAD 8 (TCB: 0x3ffe02a8, name: 'pir_task') =====================
#0 0x40030042 in xQueueReceive (xQueue=0x3ffecd48, pvBuffer=0x3ffe0278, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
#1 0x4008e489 in pir_task (pvParameters=) at ../main/util.c:1165

==================== THREAD 9 (TCB: 0x3ffdf14c, name: 'lora_task') =====================
#0 0x40030042 in xQueueReceive (xQueue=0x3ffec93c, pvBuffer=0x3ffdf114, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
#1 0x4008fe73 in lora_task (pvParameters=) at ../main/lorautil.c:232

==================== THREAD 10 (TCB: 0x3ffddf04, name: 'sys_evt') =====================
#0 0x40030042 in xQueueReceive (xQueue=0x3ffec68c, pvBuffer=0x3ffdde90, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
#1 0x40104800 in esp_event_loop_run (event_loop=0x3ffec670, ticks_to_run=4294967295) at D:/project/esp32/idf-v4.3.1/components/esp_event/esp_event.c:624
#2 0x40104973 in esp_event_loop_run_task (args=0x3ffec670) at D:/project/esp32/idf-v4.3.1/components/esp_event/esp_event.c:115

==================== THREAD 11 (TCB: 0x3ff9f094, name: 'esp_timer') =====================
#0 0x4001a8d4 in ?? ()
#1 0x40032018 in vPortExitCritical (mux=0x3ffce2f8 ) at D:/project/esp32/idf-v4.3.1/components/freertos/port/xtensa/port.c:473
#2 0x40031590 in ulTaskNotifyTake (xClearCountOnExit=1, xTicksToWait=4294967295) at D:/project/esp32/idf-v4.3.1/components/freertos/tasks.c:5203
#3 0x4008aade in timer_task (arg=) at D:/project/esp32/idf-v4.3.1/components/esp_timer/src/esp_timer.c:347

==================== THREAD 12 (TCB: 0x3ffe3cd8, name: 'wifi') =====================
#0 0x40030042 in xQueueReceive (xQueue=0x3ffe283c, pvBuffer=0x3ffe3ca0, xTicksToWait=) at D:/project/esp32/idf-v4.3.1/components/hal/esp32s2/include/hal/cpu_ll.h:33
#1 0x400abcf4 in queue_recv_wrapper (queue=0x3ffe283c, item=0x3ffe3ca0, block_time_tick=4294967295) at D:/project/esp32/idf-v4.3.1/components/esp_wifi/esp32s2/esp_adapter.c:352
#2 0x4003a3fc in ppTask () at D:/project/esp32/idf-v4.3.1/components/esp_timer/src/esp_timer.c:525

======================= ALL MEMORY REGIONS ========================
Name Address Size Attrs
.rtc.text 0x40070000 0x2c R XA
.rtc.force_fast 0x3ff9e02c 0x0 RW
.rtc.data 0x50000000 0x1c RW A
.rtc_noinit 0x5000001c 0x0 RW
.rtc.force_slow 0x5000001c 0x0 RW
.iram0.vectors 0x40024000 0x403 R XA
.iram0.text 0x40024404 0x1948c R XA
.dram0.data 0x3ffcd890 0x5738 RW A
.noinit 0x3ffd2fc8 0x0 RW
.flash.appdesc 0x3f000020 0x100 R A
.flash.rodata 0x3f000120 0x22ab0 RW A
.flash.text 0x40080020 0x8962b R XA
.iram0.text_end 0x4003d890 0x0 RW
.dram0.heap_start 0x3ffd8570 0x0 RW
.coredump.tasks.data 0x3ff9f36c 0x154 RW
.coredump.tasks.data 0x3ffdac20 0x2c4 RW
.coredump.tasks.data 0x3ff9fac8 0x154 RW
.coredump.tasks.data 0x3ff9f9d0 0xf0 RW
.coredump.tasks.data 0x3ffe2684 0x154 RW
.coredump.tasks.data 0x3ffe2570 0x10c RW
.coredump.tasks.data 0x3ffeb57c 0x154 RW
.coredump.tasks.data 0x3fff41f0 0x100 RW
.coredump.tasks.data 0x3ffe1528 0x154 RW
.coredump.tasks.data 0x3ffe1410 0x110 RW
.coredump.tasks.data 0x3ffdd2a8 0x154 RW
.coredump.tasks.data 0x3ffdd170 0x130 RW
.coredump.tasks.data 0x3ff9fc20 0x154 RW
.coredump.tasks.data 0x3ffdb620 0xc8 RW
.coredump.tasks.data 0x3ffe02a8 0x154 RW
.coredump.tasks.data 0x3ffe0180 0x120 RW
.coredump.tasks.data 0x3ffdf14c 0x154 RW
.coredump.tasks.data 0x3ffdf040 0x104 RW
.coredump.tasks.data 0x3ffddf04 0x154 RW
.coredump.tasks.data 0x3ffdddb0 0x14c RW
.coredump.tasks.data 0x3ff9f094 0x154 RW
.coredump.tasks.data 0x3ffd9dd0 0x110 RW
.coredump.tasks.data 0x3ffe3cd8 0x154 RW
.coredump.tasks.data 0x3ffe3bb0 0x120 RW

===================== ESP32 CORE DUMP END =====================

Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1238, in _readerthread
buffer.append(fh.read())
OSError: [Errno 22] Invalid argument

Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\admin\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1238, in _readerthread
buffer.append(fh.read())
OSError: [Errno 22] Invalid argument

WARNING: Attempt to terminate the GDB process failed, because it is already terminated. Skip
Done!
@espressif-bot espressif-bot added the Status: Opened Issue is new label Oct 19, 2021
@github-actions github-actions bot changed the title v4.3.1 long time running crashed v4.3.1 long time running crashed (IDFGH-6048) Oct 19, 2021
@o-marshmallow
Copy link
Collaborator

Hi @achao1104 ,

Is it possible to post the code or an example to reproduce the example? I am aware that it happens only after 30 hours but without any code and/or ELF file, it is difficult to make a guess.

@achao1104
Copy link
Author

Hello,

I have sent the elf to your colleage.

thanks.

@zztiger123
Copy link
Collaborator

Hi @o-marshmallow ，

Here is the log and elf.
20211019_coredump.zip
f.

@filzek
Copy link

filzek commented Oct 27, 2021

I have open a similar ticket to this problem, it really happend exact at SENTINEL - HAS PASSED: [28] hours [57] minutes [102913] seconds after it sync with cloud/NTP.
Guru Meditation Error: Core 1 panic'ed (LoadStoreAlignment). Exception was unhandled.

This doenst happen in SDK 4.2 at all.

@AxelLin
Copy link
Contributor

AxelLin commented Nov 14, 2021

@o-marshmallow
Any finding?

@AxelLin
Copy link
Contributor

AxelLin commented Jan 16, 2022

Hi @achao1104
Does this issue still happen on v4.3.2?

@achao1104
Copy link
Author

achao1104 commented Jan 16, 2022 via email

@o-marshmallow
Copy link
Collaborator

Hi @achao1104 @AxelLin ,
To me this sounds like a memory/heap corruption, can you try cherry-picking the commit d300a9cfe3bf09f83448b9c9075c2adccebac7d2 and try again?

@AxelLin
Copy link
Contributor

AxelLin commented Feb 11, 2022

@achao1104 Any update regarding this issue? (Did you try with #7731 (comment) ?)

@o-marshmallow o-marshmallow added the Awaiting Response awaiting a response from the author label Mar 9, 2022
@o-marshmallow o-marshmallow removed their assignment Mar 14, 2022
@o-marshmallow
Copy link
Collaborator

I am closing this issue, if you still have this issue, feel free to re-open

@espressif-bot espressif-bot added Resolution: Done Issue is done internally Status: Done Issue is done internally and removed Status: Opened Issue is new labels Mar 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@o-marshmallow o-marshmallow

Labels
Awaiting Response awaiting a response from the author Resolution: Done Issue is done internally Status: Done Issue is done internally
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@AxelLin @o-marshmallow @filzek @achao1104 @zztiger123 @espressif-bot