backtrace for abort / panic not available in coredump (IDFGH-4276) #6124
Labels
Comments
github-actions
bot
changed the title
|
@mmadela Thanks for raising this issue. The issue is likely related to the handling of |
|
i have a same problem. waiting for progress |
|
in the meanwhile I realized, that a backtrace is present when coredump is downloaded from flash and parsed by the espcoredump.py |
|
` } ` in APP code, add "assert(0)" for testing coredump to flash. coredump info below:
=============================================================== ================== CURRENT THREAD REGISTERS =================== ==================== CURRENT THREAD STACK ===================== ======================== THREADS INFO ========================= [New process 1] [New process 2] [New process 3] [New process 4] [New process 5] [New process 6] [New process 7] [New process 8] [New process 9] [New process 10] [New process 11] [New process 12] [New process 13] [New process 14] [New process 15] [New process 16] [New process 17] [New process 18] [Current thread is 1 ()] ======================= ALL MEMORY REGIONS ======================== ===================== ESP32 CORE DUMP END =====================Done!
|
|
Backtrace is saved to flash only if this is configured so, see sdkconfig |
|
Core dumps store the backtraces of It may help at this point to provide another concrete and clear example of what the issue is: during core dump analysis, stack traces on threads that have called I ran into this this week: my ESP32 project (lc-esp32) has some long-term stability issues, so I have enabled some of the IDF and FreeRTOS heap debugging features--notably in this case the "panic on memory allocation failure" feature: I also configured it to save core dumps: It looks like I can't attach my binary or core dump here, but here's my sdkconfig. When I invoke I get this full dump analysis is attached. Note that several threads have full backtraces to The faulting thread, however, is truncated: I'm lucky that the Hopefully @igrr will be able to chime in with news sometime soon. |
|
It is the configuration info in sdkconfig file below. I have configured the coredump to flash option. and the coredump info is that has been read from esp32 by “python espcoredump.py -p COM3 info_corefile -o 4128768 .\build\ec600s-esp32.elf” I am using ESP-IDF v3.3.4. |
|
please verify, if your partitions.csv contains line |
|
It is the partition table info below. |
|
try to read the crash data then parse the file |
|
debug code below: According to your @mmadela reply
it is still “Backtrace stopped” yet. |
|
may be it lays on IDF version, you are using 3.3.4, I am 4.1.1 |
3 tasks
espressif-bot
added
Status: In Progress
|
@mmadela @maqian-cn backtrace issue has been fixed in the new toolchain release, esp-2021r2. This toolchain version will be included into the upcoming v4.4 release of ESP-IDF. If you are using older versions of ESP-IDF, you won't be able to use esp-2021r2 toolchain. However it is sufficient to use just the |
noahp
added a commit
to memfault/conda-recipes
that referenced
this issue
Oct 26, 2021
Pull in the patch from here: > espressif/esp-idf#6124 (comment) Which was applied to the espressif binutils-gdb fork here: > espressif/binutils-gdb@9a47478
noahp
added a commit
to memfault/conda-recipes
that referenced
this issue
Oct 27, 2021
Pull in the patch from here: > espressif/esp-idf#6124 (comment) Which was applied to the espressif binutils-gdb fork here: > espressif/binutils-gdb@9a47478
noahp
added a commit
to memfault/conda-recipes
that referenced
this issue
Oct 27, 2021
Pull in the patch from here: > espressif/esp-idf#6124 (comment) Which was applied to the espressif binutils-gdb fork here: > espressif/binutils-gdb@9a47478 Verified after the patch, loading a core file with a previously-truncated backtrace now shows the full backtrace.
noahp
added a commit
to memfault/conda-recipes
that referenced
this issue
Oct 27, 2021
Pull in the patch from here: > espressif/esp-idf#6124 (comment) Which was applied to the espressif binutils-gdb fork here: > espressif/binutils-gdb@9a47478 Verified after the patch, loading a core file with a previously-truncated backtrace now shows the full backtrace.
espressif-bot
pushed a commit
that referenced
this issue
Oct 28, 2021
Updated GDB to 9.2 version for xtensa chips
Fixed coredump work for xtensa chips
Fixed backtrace for xtensa chips
Fixed multilib for riscv32 chips
Fixed running GDB on some RaspberryPi configuration for riscv32 chip
Fixed support of fnmatch(), iconv() and some other posix functions in stdlib
Closes #6124
Closes #2484
Closes #3264
Closes espressif/crosstool-NG#13
Closes espressif/crosstool-NG#16
espressif-bot
pushed a commit
that referenced
this issue
Oct 29, 2021
Updated GDB to 9.2 version for xtensa chips
Fixed coredump work for xtensa chips
Fixed backtrace for xtensa chips
Fixed multilib for riscv32 chips
Fixed running GDB on some RaspberryPi configuration for riscv32 chip
Fixed support of fnmatch(), iconv() and some other posix functions in stdlib
Closes #6124
Closes #2484
Closes #3264
Closes espressif/crosstool-NG#13
Closes espressif/crosstool-NG#16
espressif-bot
added
Status: In Progress
|
Unfortunately the fix for the backtraces on |
|
@igrr Do you have a recommendation for a way to break the app like |
|
@igrr: We are also experiencing this issue on v4.4.1 and would really appreciate a fix. Is there any type of work-around that we can do to get meaningful core-dumps in case of panic? |
|
@igrr, @KaeLL : I can confirm that this issue is related to the |
|
Hi @ztefanjo, i'm sorry I have missed your previous comment, I was on leave. Here is the patch you can apply to release/v4.4 to prevent the issue for noreturn functions. We had this change in review a while ago, but it wasn't accepted because the issue may still occur for "noreturn" functions in application code. Given that we had to revert the fix in GDB, I'll put this patch for review once again. @KaeLL for your question, you can trigger some exception (like |
|
@igrr When using the patch I get this error (this is on a newly checked out version of v4.4 after applying the patch). Changing 'buf' to 'msg' does fix the problem, but you may want to fix the patch. |
espressif-bot
added
Resolution: NA
|
We have updated GDB used in the |
|
Will there be backports? Will it make the cut for 5.0? |
espressif-bot
pushed a commit
that referenced
this issue
Sep 2, 2022
How to get backtrace from
|
espressif-bot
pushed a commit
that referenced
this issue
Sep 15, 2022
3 tasks
|
Patching esp-idf with this change fixed the backtrace for me in 4.4.2. Looking forward to an official fix in the stable release though! diff --git a/components/esp_common/include/esp_err.h b/components/esp_common/include/esp_err.h
index f317f8a09f..8a574d905d 100644
--- a/components/esp_common/include/esp_err.h
+++ b/components/esp_common/include/esp_err.h
@@ -75,7 +75,7 @@ const char *esp_err_to_name(esp_err_t code);
const char *esp_err_to_name_r(esp_err_t code, char *buf, size_t buflen);
/** @cond */
-void _esp_error_check_failed(esp_err_t rc, const char *file, int line, const char *function, const char *expression) __attribute__((noreturn));
+void _esp_error_check_failed(esp_err_t rc, const char *file, int line, const char *function, const char *expression);
/** @cond */
void _esp_error_check_failed_without_abort(esp_err_t rc, const char *file, int line, const char *function, const char *expression);
diff --git a/components/esp_rom/include/esp32c3/rom/libc_stubs.h b/components/esp_rom/include/esp32c3/rom/libc_stubs.h
index 233355e9c2..14fffb3bae 100644
--- a/components/esp_rom/include/esp32c3/rom/libc_stubs.h
+++ b/components/esp_rom/include/esp32c3/rom/libc_stubs.h
@@ -77,7 +77,7 @@ struct syscall_stub_table
void (*_retarget_lock_release_recursive)(_LOCK_T lock);
int (*_printf_float)(struct _reent *data, void *pdata, FILE * fp, int (*pfunc) (struct _reent *, FILE *, const char *, size_t len), va_list * ap);
int (*_scanf_float) (struct _reent *rptr, void *pdata, FILE *fp, va_list *ap);
- void (*__assert_func) (const char *file, int line, const char * func, const char *failedexpr) __attribute__((noreturn));
+ void (*__assert_func) (const char *file, int line, const char * func, const char *failedexpr);
void (*__sinit) (struct _reent *r);
void (*_cleanup_r) (struct _reent* r);
};
diff --git a/components/esp_rom/include/esp32h2/rom/libc_stubs.h b/components/esp_rom/include/esp32h2/rom/libc_stubs.h
index df78851c1b..3b653718d0 100644
--- a/components/esp_rom/include/esp32h2/rom/libc_stubs.h
+++ b/components/esp_rom/include/esp32h2/rom/libc_stubs.h
@@ -90,7 +90,7 @@ struct syscall_stub_table
#endif
int (*_printf_float)(struct _reent *data, void *pdata, FILE * fp, int (*pfunc) (struct _reent *, FILE *, const char *, size_t len), va_list * ap);
int (*_scanf_float) (struct _reent *rptr, void *pdata, FILE *fp, va_list *ap);
- void (*__assert_func) (const char *file, int line, const char * func, const char *failedexpr) __attribute__((noreturn));
+ void (*__assert_func) (const char *file, int line, const char * func, const char *failedexpr);
void (*__sinit) (struct _reent *r);
void (*_cleanup_r) (struct _reent* r);
};
diff --git a/components/esp_rom/include/esp32s3/rom/libc_stubs.h b/components/esp_rom/include/esp32s3/rom/libc_stubs.h
index 1298eeda55..80589169f4 100644
--- a/components/esp_rom/include/esp32s3/rom/libc_stubs.h
+++ b/components/esp_rom/include/esp32s3/rom/libc_stubs.h
@@ -78,7 +78,7 @@ struct syscall_stub_table
void (*_retarget_lock_release_recursive)(_LOCK_T lock);
int (*_printf_float)(struct _reent *data, void *pdata, FILE * fp, int (*pfunc) (struct _reent *, FILE *, const char *, size_t len), va_list * ap);
int (*_scanf_float) (struct _reent *rptr, void *pdata, FILE *fp, va_list *ap);
- void (*__assert_func) (const char *file, int line, const char * func, const char *failedexpr) __attribute__((noreturn));
+ void (*__assert_func) (const char *file, int line, const char * func, const char *failedexpr);
void (*__sinit) (struct _reent *r);
void (*_cleanup_r) (struct _reent* r);
};
diff --git a/components/esp_system/esp_system.c b/components/esp_system/esp_system.c
index 8d2457b2c0..35e762ffc7 100644
--- a/components/esp_system/esp_system.c
+++ b/components/esp_system/esp_system.c
@@ -123,7 +123,7 @@ const char *esp_get_idf_version(void)
return IDF_VER;
}
-void __attribute__((noreturn)) esp_system_abort(const char *details)
+void esp_system_abort(const char *details)
{
panic_abort(details);
}
diff --git a/components/esp_system/include/esp_private/panic_internal.h b/components/esp_system/include/esp_private/panic_internal.h
index d43a705aba..c42941c427 100644
--- a/components/esp_system/include/esp_private/panic_internal.h
+++ b/components/esp_system/include/esp_private/panic_internal.h
@@ -76,7 +76,7 @@ void panic_print_hex(int h);
#define panic_print_hex(h)
#endif
-void __attribute__((noreturn)) panic_abort(const char *details);
+void panic_abort(const char *details);
void panic_arch_fill_info(void *frame, panic_info_t *info);
diff --git a/components/esp_system/include/esp_system.h b/components/esp_system/include/esp_system.h
index b98b719490..373d85dce9 100644
--- a/components/esp_system/include/esp_system.h
+++ b/components/esp_system/include/esp_system.h
@@ -129,7 +129,7 @@ uint32_t esp_get_minimum_free_heap_size( void );
*
* @param details Details that will be displayed during panic handling.
*/
-void __attribute__((noreturn)) esp_system_abort(const char* details);
+void esp_system_abort(const char* details);
#ifdef __cplusplus
}
diff --git a/components/esp_system/panic.c b/components/esp_system/panic.c
index 722975b7c7..a45c31ec01 100644
--- a/components/esp_system/panic.c
+++ b/components/esp_system/panic.c
@@ -385,7 +385,7 @@ void esp_panic_handler(panic_info_t *info)
}
-void IRAM_ATTR __attribute__((noreturn, no_sanitize_undefined)) panic_abort(const char *details)
+void IRAM_ATTR __attribute__((no_sanitize_undefined)) panic_abort(const char *details)
{
g_panic_abort = true;
s_panic_abort_details = (char *) details;
diff --git a/components/esp_system/ubsan.c b/components/esp_system/ubsan.c
index 8f5e123cc7..fa4e0f35f0 100644
--- a/components/esp_system/ubsan.c
+++ b/components/esp_system/ubsan.c
@@ -111,7 +111,7 @@ struct invalid_builtin_data {
};
-static void __ubsan_default_handler(struct source_location *loc, const char *func) __attribute__((noreturn));
+static void __ubsan_default_handler(struct source_location *loc, const char *func);
/*
* When compiling with -fsanitize=undefined the compiler expects functions
diff --git a/components/newlib/abort.c b/components/newlib/abort.c
index 38ae5b73c4..daf8d601d6 100644
--- a/components/newlib/abort.c
+++ b/components/newlib/abort.c
@@ -19,7 +19,7 @@
#include "soc/soc_caps.h"
#include "hal/cpu_hal.h"
-void __attribute__((noreturn)) abort(void)
+void abort(void)
{
#define ERR_STR1 "abort() was called at PC 0x"
#define ERR_STR2 " on core "
diff --git a/components/newlib/assert.c b/components/newlib/assert.c
index 29173d56f6..4ab5ee3c35 100644
--- a/components/newlib/assert.c
+++ b/components/newlib/assert.c
@@ -37,7 +37,7 @@ static inline void ra_to_str(char *addr)
/* Overriding assert function so that whenever assert is called from critical section,
* it does not lead to a crash of its own.
*/
-void __attribute__((noreturn)) __assert_func(const char *file, int line, const char *func, const char *expr)
+void __assert_func(const char *file, int line, const char *func, const char *expr)
{
#if CONFIG_COMPILER_OPTIMIZATION_ASSERTIONS_SILENT
char buff[sizeof(ASSERT_STR) + 11 + 1] = ASSERT_STR;
@@ -86,7 +86,7 @@ void __attribute__((noreturn)) __assert_func(const char *file, int line, const c
#endif /* CONFIG_COMPILER_OPTIMIZATION_ASSERTIONS_SILENT */
}
-void __attribute__((noreturn)) __assert(const char *file, int line, const char *failedexpr)
+void __assert(const char *file, int line, const char *failedexpr)
{
__assert_func(file, line, NULL, failedexpr);
} |
3 tasks
Have you found a solution? |
|
I did this void Sys::saveCoreDump(){
size_t size = 0;
size_t address = 0;
char* coreDump;
if( !alloc(coreDump, 640) ){ return; } // Check if allocation was successfull
if (esp_core_dump_image_get(&address, &size) != ESP_OK) { free(coreDump); return; } // Try to get the bin image pointers
const esp_partition_t* pt = NULL;
pt = esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_COREDUMP, "coredump");
if (pt == NULL) { free(coreDump); return; } // Check if we found the partition or not.
uint8_t bf[256];
int16_t toRead;
db.remove(CORE_DUMP_PATH); // Remove the previous file if any
for (int16_t i = 0; i < (size / 256) + 1; i++) {
strcpy(coreDump, "");
toRead = (size - i * 256) > 256 ? 256 : (size - i * 256);
esp_err_t er = esp_partition_read(pt, i * 256, bf, toRead);
if (er != ESP_OK) {
break;
}
for (int16_t j = 0; j < 256; j++) {
char str_tmp[3];
sprintf(str_tmp, "%02x", bf[j]);
strcat(coreDump, str_tmp);
}
db.append(CORE_DUMP_PATH, coreDump); // Append the bin content
}
esp_core_dump_image_erase(); // Erase partition
free(coreDump);
}esp_err_t Sys::esp_core_dump_image_erase() {
/* Find the partition that could potentially contain a (previous) core dump. */
const esp_partition_t* core_part = esp_partition_find_first(ESP_PARTITION_TYPE_DATA,
ESP_PARTITION_SUBTYPE_DATA_COREDUMP,
"coredump");
if (!core_part) {
logger.write("No core dump partition found!|n");
return ESP_ERR_NOT_FOUND;
}
if (core_part->size < sizeof(uint32_t)) {
logger.write("Too small core dump partition!\n");
return ESP_ERR_INVALID_SIZE;
}
esp_err_t err = ESP_OK;
err = esp_partition_erase_range(core_part, 0, core_part->size);
if (err != ESP_OK) {
Serial.printf("Failed to erase core dump partition (%d)!\n", err);
return err;
}
// on encrypted flash esp_partition_erase_range will leave encrypted
// garbage instead of 0xFFFFFFFF so overwriting again to safely signalize
// deleted coredumps
const uint32_t invalid_size = 0xFFFFFFFF;
err = esp_partition_write(core_part, 0, &invalid_size, sizeof(invalid_size));
if (err != ESP_OK) {
logger.write("Failed to write core dump partition size (%d)!\n", err);
}
return err;
}Where db.append is just a wrapper for LittleFs append. I don't know how to decode this. I'm trying the espcoredump.py script for now to get a meaningfull output. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Environment
git describe --tagsto find it): v4.1xtensa-esp32-elf-gcc --versionto find it): (crosstool-NG esp-2020r2) 8.2.0Problem Description
the CoreDump component is intended to locate problems in the sources, but it cannot provide a backtrace in case of panic / assert / abort
Expected Behavior
Complete backtrace is available
Actual Behavior
Just 2 frames are available
Steps to reproduce
assert(0);somewhere, for example in the loop, line 45 of hello_world_main.c, see below too\esp-idf-v4.1\components\espcoredump\espcoredump.py -p COM15 info_corefile D:\eclipse_cpp\workspace\hello_world\build\hello-world.elfCode to reproduce this issue
Other items if possible
sdkconfig.txt
The text was updated successfully, but these errors were encountered: