Capture core files from GHA

[jiridanek]

There is a feeble attempt to upload files with core in their name, but that does not actually ever get any core files.

https://github.com/skupperproject/skupper-router/runs/5428604821?check_suite_focus=true#step:9:2719

67: c-benchmarks: ../src/posix/threading.c:58: sys_mutex_lock: Assertion `result == 0' failed.
22/34 Test #67: c-benchmarks ............................***Failed   82.45 sec

It is likely that coredumpctl has to be used. That might interact with Docker in some way... Also, ASAN and TSAN binaries reserve massive amounts of virtual memory (20 TB or so) which may end up disabling core file dumping.

Would be best to create a branch, on the branch intentionally create a crashing binary that produces coredump quickly and then investigate methods of getting the coredump uploaded.

[jiridanek]

The Ubuntu default is not systemd-coredump, it is apport.

$ cat /proc/sys/kernel/core_pattern
|/usr/share/apport/apport %p %s %c %d %P %E

There might be some value in working with it. It probably can automatically generate stacktrace without having to fetch the file... On the other hand, we can run gdb ourselves as a step in pipeline, so it is no great deal.

[jiridanek]

Useful testing command

/usr/bin/python3 -c 'import ctypes; ctypes.cdll["libc.so.6"].abort()'

[jiridanek]

Easiest solution to get traces would be to run all executables under gdb, the way @ssorj once recommended

gdb -ex run -ex 'thread apply all bt' -ex 'set confirm off' -ex quit --args <your prog>

The difficulty is in Leak Sanitizer being unable to work when there is another program ptracing the process. That can be workarounded by a gdb script that breaks when lsan is starting and detaches gdb. See https://stackoverflow.com/questions/71366407/can-addresssanitizer-be-induced-to-dump-a-stracktrace-upon-abort-or-general/71369893#71369893 for the idea of this solution.

It is possible to get Address Sanitizer itself to handle SIGABRT and produce a stacktrace, but it is apparently not reliable. In the one instance where I hoped it would help (crash in c-benchmarks in #2) the sanitizer reported internal error and produced no stacktrace.

One advantage of coredumping is that it automatically gets stacktrace of a child process; with gdb, only the parent (or only the forked child, depending on gdb settings) will be traced). This is not really relevant for dispatch tests as a whole, though.

The linked PR takes the coredumping route. For completeness, it should probably also locate and archive all shared libraries the binary uses... I am not going to do that, because hopefully the stacktrace will be enough, and if not, the libraries can be recovered by recreating the environment in docker. See openthread/openthread#5861 for a project which implemented the libraries archival.