[7.0][mono][arm64] futex-internal.h: No such file or directory - futex_wait_cancelable native crash in Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests #87444

carlossanlop · 2023-06-13T00:16:52Z

Error Blob

Unsure if this is a known issue or not. I didn't find an open with this method failing in the callstack. Feel free to close as dup if you find one.

{
  "ErrorMessage": "futex-internal.h: No such file or directory",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

Affected 7.0 PR: [release/7.0-staging] Reset OOB packages from 7.0.7 #86300
Affected test: Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests
Job: Libraries Test Run release mono Linux arm64 Debug
Job link: https://dev.azure.com/dnceng-public/public/_build/results?buildId=304452&view=logs&j=3f95ca23-483b-5dc9-4b05-6bd24fb925c2&t=0c1dd9d1-4ef7-54d5-2d62-e4ab330605ca&l=56
Queue: (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8
Full callstack:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0xffaf543edff0 (LWP 25) "dotnet" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0xffaf533ff1c0 (LWP 26) "SGen worker" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf53e477c0 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0xffaf516921c0 (LWP 27) "dotnet" 0x0000ffaf54088ef8 in __GI___poll (fds=0xffaf4c003ae0, nfds=281128488714676, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  4    Thread 0xffaf514911c0 (LWP 28) "Finalizer" 0x0000ffaf543b0a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0xffaf53e387d8 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0xffaf491de1c0 (LWP 29) "dotnet" 0x0000ffaf543b1ac0 in __libc_read (fd=<optimized out>, buf=0xffaf491dd9a7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
  6    Thread 0xffaf490c71c0 (LWP 32) ".NET Long Runni" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf3800d10c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  7    Thread 0xffaf48ec61c0 (LWP 33) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0xffaf00000000, expected=0, futex_word=0xffaf3c00ce48) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  8    Thread 0xffaf48b3b1c0 (LWP 34) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf30020f28) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  9    Thread 0xffaf48ada1c0 (LWP 35) ".NET ThreadPool" mono_arch_flush_icache (code=0xffaf269b60c8 "\375{\275\251\375\003", size=172) at /__w/1/s/src/mono/mono/mini/mini-arm64.c:2079
  10   Thread 0xffaf488861c0 (LWP 36) ".NET Long Runni" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf2800e9bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  11   Thread 0xffaf486851c0 (LWP 37) ".NET Long Runni" 0x0000ffaf543b2d5c in __waitpid (pid=<optimized out>, stat_loc=0xffaf4867e070, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
  12   Thread 0xffaf276891c0 (LWP 39) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf276883b8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142

Thread 12 (Thread 0xffaf276891c0 (LWP 39)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf276883b8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf27688340, mutex=0xffaf380c8520, cond=0xffaf27688390) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf27688390, mutex=0xffaf380c8520, abstime=0xffaf27688340) at pthread_cond_wait.c:667
#3  0x0000ffaf53c45e98 in mono_os_cond_timedwait (cond=0xffaf27688390, mutex=0xffaf380c8520, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#4  0x0000ffaf53c4b494 in mono_coop_cond_timedwait (cond=0xffaf27688390, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
#5  mono_lifo_semaphore_timed_wait (semaphore=0xffaf380c8520, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
#6  0x0000ffaf48aea098 in ?? ()
#7  0x0000ffaf53ba6f28 in ves_icall_System_Threading_Thread_YieldInternal () at /__w/1/s/src/mono/mono/metadata/icall.c:7185
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 11 (Thread 0xffaf486851c0 (LWP 37)):
#0  0x0000ffaf543b2d5c in __waitpid (pid=<optimized out>, stat_loc=0xffaf4867e070, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x0000ffaf53d8706c in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xffaf4867ead0, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x0000ffaf53d455ac in mono_handle_native_crash (signal=0xffaf53b47770 "SIGSEGV", mctx=0xffaf4867ead0, info=0xffaf4867ee30) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
#4  0x0000ffaf53cafa80 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xffaf4867ee30, context=0xffaf4867eeb0, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3774
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x0000ffaf2789be98 in ?? ()
#8  0x0000ffaf5347b3a8 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 10 (Thread 0xffaf488861c0 (LWP 36)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf2800e9bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf2800e960, cond=0xffaf2800e990) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf2800e990, mutex=0xffaf2800e960) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xffaf2800e960) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000000000000002 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 9 (Thread 0xffaf48ada1c0 (LWP 35)):
#0  mono_arch_flush_icache (code=0xffaf269b60c8 "\375{\275\251\375\003", size=172) at /__w/1/s/src/mono/mono/mini/mini-arm64.c:2079
#1  0x0000ffaf53ca673c in mono_codegen (cfg=0xffaf357a7990) at /__w/1/s/src/mono/mono/mini/mini.c:2212
#2  0x0000ffaf53ca8da0 in mini_method_compile (method=<optimized out>, opts=374417919, flags=JIT_FLAG_RUN_CCTORS, parts=0, aot_method_index=-1) at /__w/1/s/src/mono/mono/mini/mini.c:3935
#3  0x0000ffaf53caa0b8 in mono_jit_compile_method_inner (method=0xffaf34f93480, opt=172, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini.c:4129
#4  0x0000ffaf53caeba0 in mono_jit_compile_method_with_opt (method=0xffaf34f93480, opt=374417919, jit_only=0, error=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2708
#5  jit_compile_method_with_opt_cb (arg=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2763
#6  jit_compile_method_with_opt (params=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2779
#7  0x0000ffaf53cae018 in mono_jit_compile_method (method=<optimized out>, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2798
#8  0x0000ffaf53d47a18 in common_call_trampoline (regs=0xffaf48ad59e0, code=0xffaf269b4500 "", m=0xffaf34f93480, vt=0xffaf3540ef10, vtable_slot=<optimized out>, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:618
#9  0x0000ffaf53d48788 in mono_vcall_trampoline (regs=0xffaf48ad59e0, code=0xffaf269b4500 "", slot=1, tramp=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:840
#10 0x0000ffaf53920504 in ?? ()
#11 0x0000ffaf5368b8d0 in ?? ()
#12 0x0000ffaf5368cec8 in ?? ()
Backtrace stopped: not enough registers or memory available to unwind further

Thread 8 (Thread 0xffaf48b3b1c0 (LWP 34)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf30020f28) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf48b3a2a8, mutex=0xffaf30020ed0, cond=0xffaf30020f00) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf30020f00, mutex=0xffaf30020ed0, abstime=0xffaf48b3a2a8) at pthread_cond_wait.c:667
#3  0x0000ffaf50fa945c in SystemNative_LowLevelMonitor_TimedWait (monitor=0xffaf30020ed0, timeoutMilliseconds=500) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ffaf48ae0894 in ?? ()
#5  0x0000ffaf490d71d0 in ?? ()
#6  0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 7 (Thread 0xffaf48ec61c0 (LWP 33)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0xffaf00000000, expected=0, futex_word=0xffaf3c00ce48) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf48ec50f8, mutex=0xffaf3c00cdf0, cond=0xffaf3c00ce20) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf3c00ce20, mutex=0xffaf3c00cdf0, abstime=0xffaf48ec50f8) at pthread_cond_wait.c:667
#3  0x0000ffaf50fa945c in SystemNative_LowLevelMonitor_TimedWait (monitor=0xffaf3c00cdf0, timeoutMilliseconds=12000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ffaf48ae0894 in ?? ()
#5  0x0000ffaf490d71d0 in ?? ()
#6  0x0000ffaf48ec5910 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 6 (Thread 0xffaf490c71c0 (LWP 32)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf3800d10c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf3800d0b0, cond=0xffaf3800d0e0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf3800d0e0, mutex=0xffaf3800d0b0) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xffaf3800d0b0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000ffaf5342f150 in ?? ()
#6  0x0000ffaf534302b0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 5 (Thread 0xffaf491de1c0 (LWP 29)):
#0  0x0000ffaf543b1ac0 in __libc_read (fd=<optimized out>, buf=0xffaf491dd9a7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x0000ffaf50fa86ac in SignalHandlerLoop (arg=0xab11907743d0) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:323
#2  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3dd9f) at pthread_create.c:463
#3  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 4 (Thread 0xffaf514911c0 (LWP 28)):
#0  0x0000ffaf543b0a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0xffaf53e387d8 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0xffaf53e387d8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#2  0x0000ffaf543b0b60 in __new_sem_wait_slow (sem=0xffaf53e387d8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#3  0x0000ffaf53c1c130 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../utils/mono-os-semaphore.h:204
#4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-semaphore.h:41
#5  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:891
#6  0x0000ffaf53bf6e8c in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1202
#7  0x0000ffaf53bf6d38 in start_wrapper (data=0xab118feeff20) at /__w/1/s/src/mono/mono/metadata/threads.c:1264
#8  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e01f) at pthread_create.c:463
#9  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 3 (Thread 0xffaf516921c0 (LWP 27)):
#0  0x0000ffaf54088ef8 in __GI___poll (fds=0xffaf4c003ae0, nfds=281128488714676, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x0000ffaf53e125a0 in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470
#2  ds_ipc_poll (poll_handles_data=0xffaf4c0032d0, poll_handles_data_len=1, timeout_ms=4294967295, callback=0xffaf53e11950 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1096
#3  0x0000ffaf53e0fb4c in ds_ipc_stream_factory_get_next_available_stream (callback=0xffaf53e11950 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:395
#4  0x0000ffaf53e0e3c4 in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:127
#5  0x0000ffaf53e11930 in ep_rt_thread_mono_start_func (data=0xab118fed3d40) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1356
#6  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e16f) at pthread_create.c:463
#7  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 2 (Thread 0xffaf533ff1c0 (LWP 26)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf53e477c0 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf53e47768 <lock>, cond=0xffaf53e47798 <work_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf53e47798 <work_cond>, mutex=0xffaf53e47768 <lock>) at pthread_cond_wait.c:655
#3  0x0000ffaf53c97b5c in mono_os_cond_wait (cond=0xffaf53e477c0 <work_cond+40>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
#4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
#5  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
#6  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e10f) at pthread_create.c:463
#7  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 1 (Thread 0xffaf543edff0 (LWP 25)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xab119041d6c0, cond=0xab119041d6f0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xab119041d6f0, mutex=0xab119041d6c0) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xab119041d6c0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000ffaf5342b0e0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Report

Build	Definition	Test	Pull Request
442524	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#93642
442020	dotnet/runtime	ComInterfaceGenerator.Unit.Tests.WorkItemExecution	#93373
2293756	dotnet-runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution	#34502
441666	dotnet/runtime	ComInterfaceGenerator.Unit.Tests.WorkItemExecution	#93373
441292	dotnet/runtime	LibraryImportGenerator.Unit.Tests.WorkItemExecution	#91800
440443	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#93582
440204	dotnet/runtime	System.IO.Pipelines.Tests.WorkItemExecution	#77359
439954	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#92881
439461	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93545
438315	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#92881
436469	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#93408
436470	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#93390
435859	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93280
435814	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#93380
435488	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#93182
435309	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#93182
435247	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93369
434561	dotnet/runtime	JSImportGenerator.Unit.Tests.WorkItemExecution	#93282
433593	dotnet/runtime	System.Numerics.Tensors.Tests.WorkItemExecution	#93296
433363	dotnet/runtime	System.Numerics.Tensors.Tests.WorkItemExecution	#93296
433289	dotnet/runtime	LibraryImportGenerator.Unit.Tests.WorkItemExecution	#93292
433207	dotnet/runtime	LibraryImportGenerator.Unit.Tests.WorkItemExecution	#93289
432794	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
432457	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93259
432361	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution
432035	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#93182
431388	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#93191
431038	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#93182
431005	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#93180
430413	dotnet/runtime	Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.WorkItemExecution
429831	dotnet/runtime	System.ComponentModel.EventBasedAsync.Tests.WorkItemExecution	#93112
428844	dotnet/runtime	System.Text.RegularExpressions.Tests.RegexMatchTests.Match_ExcessPrefix	#93078
428746	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#92980
428612	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution	#93006
428315	dotnet/runtime	JSImportGenerator.Unit.Tests.WorkItemExecution	#93025
428288	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93053
428216	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93049
427970	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92974
427728	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#93027
427388	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93010
2283304	dotnet-runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#33850
426948	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#79432
426756	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#92980
426746	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution
426427	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#92967
2282611	dotnet-runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#33850
425523	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92864
424117	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92863
423438	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution
423068	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#92829
422545	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92057
422012	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92579
420717	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#92714
420222	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#91633
420178	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92579
420129	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#89587
419941	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92674
419275	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#86787
2276547	dotnet-runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#33850
418883	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#89587
418864	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92578
418525	dotnet/runtime	PayloadGroup0.WorkItemExecution	#92503
418328	dotnet/runtime	IntrinsicsInSystemPrivateCoreLib.Tests.WorkItemExecution	#92503
417623	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#92337
417573	dotnet/runtime	Microsoft.Extensions.Hosting.Tests.ConsoleLifetimeExitTests.EnsureEnvironmentExitCode	#92465
417063	dotnet/runtime	LibraryImportGenerator.Unit.Tests.WorkItemExecution	#92503
416958	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92534
415444	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92410
415389	dotnet/runtime	JSImportGenerator.Unit.Tests.WorkItemExecution
415330	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92457
415290	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#92454
415255	dotnet/runtime	LibraryImportGenerator.Unit.Tests.WorkItemExecution	#92451
414574	dotnet/runtime	JSImportGenerator.Unit.Tests.WorkItemExecution	#90074
414177	dotnet/runtime	PayloadGroup0.WorkItemExecution
413933	dotnet/runtime	LibraryImportGenerator.Unit.Tests.WorkItemExecution	#92287
413702	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92385
413187	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92354
412471	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#92287
412440	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution	#90436
412090	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution	#90436
411570	dotnet/runtime	PayloadGroup0.WorkItemExecution
410140	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#90436

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
5	17	82

Known issue validation

Build: 🔎
Result validation: ⚠️ Provided build not found. Provide a valid build in the "Build: 🔎" line.
Validation performed at: 9/1/2023 6:28:52 PM UTC

The text was updated successfully, but these errors were encountered:

ghost · 2023-06-13T00:17:07Z

Tagging subscribers to this area: @dotnet/area-extensions-logging
See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

Unsure if this is a known issue or not. I didn't find an open with this method failing in the callstack. Feel free to close as dup if you find one.

{
  "ErrorMessage": "in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": true
}

Reproduction Steps

Affected 7.0 PR: [release/7.0-staging] Reset OOB packages from 7.0.7 #86300
Affected test: Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests
Job: Libraries Test Run release mono Linux arm64 Debug
Job link: https://dev.azure.com/dnceng-public/public/_build/results?buildId=304452&view=logs&j=3f95ca23-483b-5dc9-4b05-6bd24fb925c2&t=0c1dd9d1-4ef7-54d5-2d62-e4ab330605ca&l=56
Queue: (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8
Full callstack:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0xffaf543edff0 (LWP 25) "dotnet" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0xffaf533ff1c0 (LWP 26) "SGen worker" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf53e477c0 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0xffaf516921c0 (LWP 27) "dotnet" 0x0000ffaf54088ef8 in __GI___poll (fds=0xffaf4c003ae0, nfds=281128488714676, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  4    Thread 0xffaf514911c0 (LWP 28) "Finalizer" 0x0000ffaf543b0a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0xffaf53e387d8 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0xffaf491de1c0 (LWP 29) "dotnet" 0x0000ffaf543b1ac0 in __libc_read (fd=<optimized out>, buf=0xffaf491dd9a7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
  6    Thread 0xffaf490c71c0 (LWP 32) ".NET Long Runni" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf3800d10c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  7    Thread 0xffaf48ec61c0 (LWP 33) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0xffaf00000000, expected=0, futex_word=0xffaf3c00ce48) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  8    Thread 0xffaf48b3b1c0 (LWP 34) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf30020f28) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  9    Thread 0xffaf48ada1c0 (LWP 35) ".NET ThreadPool" mono_arch_flush_icache (code=0xffaf269b60c8 "\375{\275\251\375\003", size=172) at /__w/1/s/src/mono/mono/mini/mini-arm64.c:2079
  10   Thread 0xffaf488861c0 (LWP 36) ".NET Long Runni" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf2800e9bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  11   Thread 0xffaf486851c0 (LWP 37) ".NET Long Runni" 0x0000ffaf543b2d5c in __waitpid (pid=<optimized out>, stat_loc=0xffaf4867e070, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
  12   Thread 0xffaf276891c0 (LWP 39) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf276883b8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142

Thread 12 (Thread 0xffaf276891c0 (LWP 39)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf276883b8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf27688340, mutex=0xffaf380c8520, cond=0xffaf27688390) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf27688390, mutex=0xffaf380c8520, abstime=0xffaf27688340) at pthread_cond_wait.c:667
#3  0x0000ffaf53c45e98 in mono_os_cond_timedwait (cond=0xffaf27688390, mutex=0xffaf380c8520, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#4  0x0000ffaf53c4b494 in mono_coop_cond_timedwait (cond=0xffaf27688390, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
#5  mono_lifo_semaphore_timed_wait (semaphore=0xffaf380c8520, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
#6  0x0000ffaf48aea098 in ?? ()
#7  0x0000ffaf53ba6f28 in ves_icall_System_Threading_Thread_YieldInternal () at /__w/1/s/src/mono/mono/metadata/icall.c:7185
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 11 (Thread 0xffaf486851c0 (LWP 37)):
#0  0x0000ffaf543b2d5c in __waitpid (pid=<optimized out>, stat_loc=0xffaf4867e070, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x0000ffaf53d8706c in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xffaf4867ead0, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x0000ffaf53d455ac in mono_handle_native_crash (signal=0xffaf53b47770 "SIGSEGV", mctx=0xffaf4867ead0, info=0xffaf4867ee30) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
#4  0x0000ffaf53cafa80 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xffaf4867ee30, context=0xffaf4867eeb0, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3774
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x0000ffaf2789be98 in ?? ()
#8  0x0000ffaf5347b3a8 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 10 (Thread 0xffaf488861c0 (LWP 36)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf2800e9bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf2800e960, cond=0xffaf2800e990) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf2800e990, mutex=0xffaf2800e960) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xffaf2800e960) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000000000000002 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 9 (Thread 0xffaf48ada1c0 (LWP 35)):
#0  mono_arch_flush_icache (code=0xffaf269b60c8 "\375{\275\251\375\003", size=172) at /__w/1/s/src/mono/mono/mini/mini-arm64.c:2079
#1  0x0000ffaf53ca673c in mono_codegen (cfg=0xffaf357a7990) at /__w/1/s/src/mono/mono/mini/mini.c:2212
#2  0x0000ffaf53ca8da0 in mini_method_compile (method=<optimized out>, opts=374417919, flags=JIT_FLAG_RUN_CCTORS, parts=0, aot_method_index=-1) at /__w/1/s/src/mono/mono/mini/mini.c:3935
#3  0x0000ffaf53caa0b8 in mono_jit_compile_method_inner (method=0xffaf34f93480, opt=172, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini.c:4129
#4  0x0000ffaf53caeba0 in mono_jit_compile_method_with_opt (method=0xffaf34f93480, opt=374417919, jit_only=0, error=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2708
#5  jit_compile_method_with_opt_cb (arg=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2763
#6  jit_compile_method_with_opt (params=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2779
#7  0x0000ffaf53cae018 in mono_jit_compile_method (method=<optimized out>, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2798
#8  0x0000ffaf53d47a18 in common_call_trampoline (regs=0xffaf48ad59e0, code=0xffaf269b4500 "", m=0xffaf34f93480, vt=0xffaf3540ef10, vtable_slot=<optimized out>, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:618
#9  0x0000ffaf53d48788 in mono_vcall_trampoline (regs=0xffaf48ad59e0, code=0xffaf269b4500 "", slot=1, tramp=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:840
#10 0x0000ffaf53920504 in ?? ()
#11 0x0000ffaf5368b8d0 in ?? ()
#12 0x0000ffaf5368cec8 in ?? ()
Backtrace stopped: not enough registers or memory available to unwind further

Thread 8 (Thread 0xffaf48b3b1c0 (LWP 34)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf30020f28) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf48b3a2a8, mutex=0xffaf30020ed0, cond=0xffaf30020f00) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf30020f00, mutex=0xffaf30020ed0, abstime=0xffaf48b3a2a8) at pthread_cond_wait.c:667
#3  0x0000ffaf50fa945c in SystemNative_LowLevelMonitor_TimedWait (monitor=0xffaf30020ed0, timeoutMilliseconds=500) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ffaf48ae0894 in ?? ()
#5  0x0000ffaf490d71d0 in ?? ()
#6  0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 7 (Thread 0xffaf48ec61c0 (LWP 33)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0xffaf00000000, expected=0, futex_word=0xffaf3c00ce48) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf48ec50f8, mutex=0xffaf3c00cdf0, cond=0xffaf3c00ce20) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf3c00ce20, mutex=0xffaf3c00cdf0, abstime=0xffaf48ec50f8) at pthread_cond_wait.c:667
#3  0x0000ffaf50fa945c in SystemNative_LowLevelMonitor_TimedWait (monitor=0xffaf3c00cdf0, timeoutMilliseconds=12000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ffaf48ae0894 in ?? ()
#5  0x0000ffaf490d71d0 in ?? ()
#6  0x0000ffaf48ec5910 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 6 (Thread 0xffaf490c71c0 (LWP 32)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf3800d10c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf3800d0b0, cond=0xffaf3800d0e0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf3800d0e0, mutex=0xffaf3800d0b0) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xffaf3800d0b0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000ffaf5342f150 in ?? ()
#6  0x0000ffaf534302b0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 5 (Thread 0xffaf491de1c0 (LWP 29)):
#0  0x0000ffaf543b1ac0 in __libc_read (fd=<optimized out>, buf=0xffaf491dd9a7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x0000ffaf50fa86ac in SignalHandlerLoop (arg=0xab11907743d0) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:323
#2  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3dd9f) at pthread_create.c:463
#3  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 4 (Thread 0xffaf514911c0 (LWP 28)):
#0  0x0000ffaf543b0a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0xffaf53e387d8 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0xffaf53e387d8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#2  0x0000ffaf543b0b60 in __new_sem_wait_slow (sem=0xffaf53e387d8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#3  0x0000ffaf53c1c130 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../utils/mono-os-semaphore.h:204
#4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-semaphore.h:41
#5  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:891
#6  0x0000ffaf53bf6e8c in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1202
#7  0x0000ffaf53bf6d38 in start_wrapper (data=0xab118feeff20) at /__w/1/s/src/mono/mono/metadata/threads.c:1264
#8  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e01f) at pthread_create.c:463
#9  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 3 (Thread 0xffaf516921c0 (LWP 27)):
#0  0x0000ffaf54088ef8 in __GI___poll (fds=0xffaf4c003ae0, nfds=281128488714676, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x0000ffaf53e125a0 in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470
#2  ds_ipc_poll (poll_handles_data=0xffaf4c0032d0, poll_handles_data_len=1, timeout_ms=4294967295, callback=0xffaf53e11950 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1096
#3  0x0000ffaf53e0fb4c in ds_ipc_stream_factory_get_next_available_stream (callback=0xffaf53e11950 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:395
#4  0x0000ffaf53e0e3c4 in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:127
#5  0x0000ffaf53e11930 in ep_rt_thread_mono_start_func (data=0xab118fed3d40) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1356
#6  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e16f) at pthread_create.c:463
#7  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 2 (Thread 0xffaf533ff1c0 (LWP 26)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf53e477c0 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf53e47768 <lock>, cond=0xffaf53e47798 <work_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf53e47798 <work_cond>, mutex=0xffaf53e47768 <lock>) at pthread_cond_wait.c:655
#3  0x0000ffaf53c97b5c in mono_os_cond_wait (cond=0xffaf53e477c0 <work_cond+40>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
#4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
#5  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
#6  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e10f) at pthread_create.c:463
#7  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 1 (Thread 0xffaf543edff0 (LWP 25)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xab119041d6c0, cond=0xab119041d6f0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xab119041d6f0, mutex=0xab119041d6c0) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xab119041d6c0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000ffaf5342b0e0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Author:	carlossanlop
Assignees:	-
Labels:	`arch-arm64`, `blocking-clean-ci`, `area-VM-meta-mono`, `runtime-mono`, `area-Extensions-Logging`, `Known Build Error`
Milestone:	-

carlossanlop · 2023-08-28T16:14:59Z

There are variants that KnownBuild is not collecting and tagging, so I think this issue is much more impactful that we are seeing. Here is an example:

cc @lewing @steveisok

PR: [release/8.0] Fix diagnostic creation to ensure including the URI #91145
Affected test: System.Linq
Queue: Libraries Test Run release mono_interpreter linux x64 Debug
Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=386567&view=logs&j=9310185d-988f-5930-4a09-6918c1bb36fb&t=70410075-93d6-55d0-0186-00493ab8b56c&l=55
Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-91145-merge-0704fab9cc0c4c5580/System.Linq.Queryable.Tests/1/console.e9ef3c29.log?helixlogtype=result

Output (expand)

===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Linq.Queryable.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Linq.Queryable.Tests (found 785 test cases)
  Starting:    System.Linq.Queryable.Tests (parallel test collections = on, max threads = 2)

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7f17cef12771 - Unknown
	0x7f17ceeb9abe - Unknown
	0x7f17cee21b31 - Unknown
	0x7f17cf74d140 - Unknown
	0x7f17cef15cc9 - Unknown
	0x7f17cef14296 - Unknown
	0x7f17cf041b97 - Unknown
	0x7f17cf04a74b - Unknown
	0x7f17cf001d53 - Unknown
	0x7f17cf00c36b - Unknown
	0x7f17cef2695f - Unknown
	0x7f17cef25115 - Unknown
	0x7f17cef1699a - Unknown
	0x7f17cef14296 - Unknown
	0x7f17cf041b97 - Unknown
	0x7f17cf059db4 - Unknown
	0x7f17cf741ea7 - Unknown
	0x7f17cf330a2f - Unknown

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 23]
[New LWP 24]
[New LWP 25]
[New LWP 26]
[New LWP 29]
[New LWP 30]
[New LWP 31]
[New LWP 32]
[New LWP 33]
[New LWP 34]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
DW_FORM_rnglistx index pointing outside of .debug_rnglists offset array [in module /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so.dbg]
DW_FORM_rnglistx index pointing outside of .debug_rnglists offset array [in module /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libSystem.Native.so.dbg]
futex_wait_cancelable (private=0, expected=0, futex_word=0x558565f264d4) at ../sysdeps/nptl/futex-internal.h:186
186	../sysdeps/nptl/futex-internal.h: No such file or directory.
  Id   Target Id                                        Frame 
* 1    Thread 0x7f17cf230740 (LWP 22) "dotnet"          futex_wait_cancelable (private=0, expected=0, futex_word=0x558565f264d4) at ../sysdeps/nptl/futex-internal.h:186
  2    Thread 0x7f17ce3ff700 (LWP 23) "SGen worker"     futex_wait_cancelable (private=0, expected=0, futex_word=0x7f17cf113d68 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:186
  3    Thread 0x7f17cc601700 (LWP 24) ".NET EventPipe"  0x00007f17cf32496f in __GI___poll (fds=0x7f17c8002890, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x7f17cc400700 (LWP 25) "Finalizer"       futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17cf104f50 <finalizer_sem>) at ../sysdeps/nptl/futex-internal.h:323
  5    Thread 0x7f17c558d700 (LWP 26) ".NET SigHandler" __libc_read (nbytes=1, buf=0x7f17c558cee7, fd=7) at ../sysdeps/unix/sysv/linux/read.c:26
  6    Thread 0x7f17c54f0700 (LWP 29) ".NET Long Runni" futex_wait_cancelable (private=0, expected=0, futex_word=0x7f17b4248df0) at ../sysdeps/nptl/futex-internal.h:186
  7    Thread 0x7f17c51ee700 (LWP 30) ".NET TP Worker"  futex_abstimed_wait_cancelable (private=0, abstime=0x7f17c51ed668, clockid=-987834928, expected=0, futex_word=0x7f17b80a7250) at ../sysdeps/nptl/futex-internal.h:323
  8    Thread 0x7f17c5268700 (LWP 31) ".NET TP Gate"    futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17ac000bc0) at ../sysdeps/nptl/futex-internal.h:323
  9    Thread 0x7f17c490b700 (LWP 32) ".NET TP Worker"  futex_abstimed_wait_cancelable (private=0, abstime=0x7f17c490a6e8, clockid=-997153184, expected=0, futex_word=0x7f17c490a768) at ../sysdeps/nptl/futex-internal.h:323
  10   Thread 0x7f17c4609700 (LWP 33) ".NET Long Runni" 0x00007f17cf2fc747 in __GI___wait4 (pid=35, stat_loc=0x7f17c4605080, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
  11   Thread 0x7f17c4307700 (LWP 34) ".NET Long Runni" futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17cf0fef10 <suspend_semaphore>) at ../sysdeps/nptl/futex-internal.h:323

Thread 11 (Thread 0x7f17c4307700 (LWP 34) ".NET Long Runni"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17cf0fef10 <suspend_semaphore>) at ../sysdeps/nptl/futex-internal.h:323
#1  do_futex_wait (sem=sem@entry=0x7f17cf0fef10 <suspend_semaphore>, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x00007f17cf74b278 in __new_sem_wait_slow (sem=0x7f17cf0fef10 <suspend_semaphore>, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x00007f17cefcacc8 in mono_os_sem_timedwait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cefca965 in mono_threads_wait_pending_operations () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cf08d546 in unified_suspend_stop_world () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cf08d24d in sgen_client_stop_world () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cf0ae2cb in sgen_stop_world () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf0ab08e in sgen_perform_collection () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf0aaff3 in sgen_ensure_free_space () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf0a647d in sgen_alloc_obj_nolock () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007f17cf0a6999 in sgen_alloc_obj () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#12 0x00007f17cf08f0d6 in mono_gc_alloc_obj () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#13 0x00007f17cef1c01a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#14 0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#15 0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#16 0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#17 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f17c4609700 (LWP 33) ".NET Long Runni"):
#0  0x00007f17cf2fc747 in __GI___wait4 (pid=35, stat_loc=0x7f17c4605080, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
#1  0x00007f17cef128a2 in mono_dump_native_crash_info () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#2  0x00007f17ceeb9abe in mono_handle_native_crash () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#3  0x00007f17cee21b31 in mono_sigsegv_signal_handler_debug () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  <signal handler called>
#5  0x00007f17cef15cc9 in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf04a74b in mono_runtime_try_invoke_byrefs () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf001d53 in ves_icall_InternalInvoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf00c36b in ves_icall_InternalInvoke_raw () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007f17cef2695f in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#12 0x00007f17cef25115 in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#13 0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#14 0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#15 0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#16 0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#17 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f17c490b700 (LWP 32) ".NET TP Worker"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f17c490a6e8, clockid=-997153184, expected=0, futex_word=0x7f17c490a768) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0x7f17c490a6e8, clockid=-997153184, mutex=0x7f17b40970b0, cond=0x7f17c490a740) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x7f17c490a740, mutex=0x7f17b40970b0, abstime=0x7f17c490a6e8) at pthread_cond_wait.c:656
#3  0x00007f17cefc39f3 in mono_os_cond_timedwait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cefc8899 in mono_lifo_semaphore_timed_wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef268d7 in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef25115 in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#12 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f17c5268700 (LWP 31) ".NET TP Gate"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17ac000bc0) at ../sysdeps/nptl/futex-internal.h:323
#1  do_futex_wait (sem=sem@entry=0x7f17ac000bc0, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x00007f17cf74b278 in __new_sem_wait_slow (sem=0x7f17ac000bc0, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x00007f17cefca7c8 in mono_thread_info_wait_for_resume () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cefd02c1 in mono_threads_exit_gc_safe_region_unbalanced_internal () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef250de in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f17c51ee700 (LWP 30) ".NET TP Worker"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f17c51ed668, clockid=-987834928, expected=0, futex_word=0x7f17b80a7250) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0x7f17c51ed668, clockid=-987834928, mutex=0x7f17b80a7200, cond=0x7f17b80a7228) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x7f17b80a7228, mutex=0x7f17b80a7200, abstime=0x7f17c51ed668) at pthread_cond_wait.c:656
#3  0x00007f17ce82ca97 in SystemNative_LowLevelMonitor_TimedWait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007f17cef268d7 in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef250cf in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f17c54f0700 (LWP 29) ".NET Long Runni"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f17b4248df0) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f17b4248da0, cond=0x7f17b4248dc8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x7f17b4248dc8, mutex=0x7f17b4248da0) at pthread_cond_wait.c:638
#3  0x00007f17ce82c937 in SystemNative_LowLevelMonitor_Wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007f17cef2688e in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef250cf in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f17c558d700 (LWP 26) ".NET SigHandler"):
#0  __libc_read (nbytes=1, buf=0x7f17c558cee7, fd=7) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __libc_read (fd=7, buf=0x7f17c558cee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x00007f17ce82c086 in SignalHandlerLoop () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#3  0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#4  0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f17cc400700 (LWP 25) "Finalizer"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17cf104f50 <finalizer_sem>) at ../sysdeps/nptl/futex-internal.h:323
#1  do_futex_wait (sem=sem@entry=0x7f17cf104f50 <finalizer_sem>, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x00007f17cf74b278 in __new_sem_wait_slow (sem=0x7f17cf104f50 <finalizer_sem>, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x00007f17cf081008 in finalizer_thread () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cf059d09 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f17cc601700 (LWP 24) ".NET EventPipe"):
#0  0x00007f17cf32496f in __GI___poll (fds=0x7f17c8002890, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f17cefa455e in ds_ipc_poll () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#2  0x00007f17cefa1288 in ds_ipc_stream_factory_get_next_available_stream () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#3  0x00007f17cef9f7af in server_thread () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cefa39c1 in ep_rt_thread_mono_start_func () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f17ce3ff700 (LWP 23) "SGen worker"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f17cf113d68 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f17cf113d18 <lock>, cond=0x7f17cf113d40 <work_cond>) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x7f17cf113d40 <work_cond>, mutex=0x7f17cf113d18 <lock>) at pthread_cond_wait.c:638
#3  0x00007f17cf0e8ac3 in thread_func () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5  0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f17cf230740 (LWP 22) "dotnet"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x558565f264d4) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x558565f26480, cond=0x558565f264a8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x558565f264a8, mutex=0x558565f26480) at pthread_cond_wait.c:638
#3  0x00007f17ce82c937 in SystemNative_LowLevelMonitor_Wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007f17cef2688e in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef250cf in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf048808 in mono_runtime_exec_main_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cee7a51f in mono_jit_exec () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007f17cee7cd8b in mono_main () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#12 0x00007f17ceef0ad7 in monovm_execute_assembly () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#13 0x00007f17cf14c778 in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#14 0x00007f17cf17adea in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#15 0x00007f17cf17b0df in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#16 0x00007f17cf17c082 in corehost_main () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#17 0x00007f17cf1f6afc in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#18 0x00007f17cf1f1c73 in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#19 0x00007f17cf1ee0ec in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#20 0x00007f17cf1ed1cb in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#21 0x00007f17cf1e6115 in hostfxr_main_startupinfo () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#22 0x0000558562e2e312 in ?? ()
#23 0x0000558562e2e835 in ?? ()
#24 0x00007f17cf258d0a in __libc_start_main (main=0x558562e2e750, argc=18, argv=0x7ffcc15b9eb8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcc15b9ea8) at ../csu/libc-start.c:308
#25 0x0000558562e21979 in ?? ()
[Inferior 1 (process 22) detached]

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7f17cef15cc9):0x7f17cef15cb9  1d 00 49 89 db ff 24 c1 49 8b 44 24 08 48 8b 00  ..I...$.I.D$.H..
0x7f17cef15cc9  48 8b 70 18 48 8d 3d de 48 e6 ff 31 c0 e8 05 9f  H.p.H.=.H..1....
0x7f17cef15cd9  09 00 48 8d 3d 48 90 e4 ff be 17 0f 00 00 e8 c4  ..H.=H..........
0x7f17cef15ce9  a3 09 00 0f 1f 40 00 4c 89 e6 4c 89 db 48 83 c3  [email protected]..

=================================================================
	Managed Stacktrace:
=================================================================
	  at <unknown> <0xffffffff>
	  at System.Reflection.RuntimeMethodInfo:InternalInvoke <0x00096>
	  at System.Reflection.MethodBaseInvoker:InterpretedInvoke_Method <0x0003c>
	  at System.Reflection.MethodBaseInvoker:InvokeWithNoArgs <0x0006e>
	  at System.Reflection.RuntimeMethodInfo:Invoke <0x000fe>
	  at System.Reflection.MethodBase:Invoke <0x0002e>
	  at Xunit.Sdk.TestInvoker`1:CallTestMethod <0x00030>
	  at <<InvokeTestMethodAsync>b__1>d:MoveNext <0x00242>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__1 <0x00060>
	  at <AggregateAsync>d__4:MoveNext <0x0007a>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at Xunit.Sdk.ExecutionTimer:AggregateAsync <0x00074>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__0 <0x00082>
	  at <RunAsync>d__9:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at <InvokeTestMethodAsync>d__48:MoveNext <0x00122>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestInvoker`1:InvokeTestMethodAsync <0x00074>
	  at Xunit.Sdk.XunitTestInvoker:InvokeTestMethodAsync <0x000b8>
	  at <<RunAsync>b__47_0>d:MoveNext <0x003ba>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestInvoker`1:<RunAsync>b__47_0 <0x00060>
	  at <RunAsync>d__10`1:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at Xunit.Sdk.TestInvoker`1:RunAsync <0x0003a>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestMethodAsync <0x000a8>
	  at <InvokeTestAsync>d__4:MoveNext <0x00178>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestAsync <0x00074>
	  at <>c__DisplayClass43_0:<RunAsync>b__0 <0x0002c>
	  at <RunAsync>d__10`1:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at <RunAsync>d__43:MoveNext <0x002aa>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCaseRunner:RunTestAsync <0x00104>
	  at <RunAsync>d__19:MoveNext <0x001ee>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCaseRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCase:RunAsync <0x0006a>
	  at Xunit.Sdk.XunitTestMethodRunner:RunTestCaseAsync <0x00076>
	  at <RunTestCasesAsync>d__32:MoveNext <0x000fe>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestMethodRunner`1:RunTestCasesAsync <0x00060>
	  at <RunAsync>d__31:MoveNext <0x00114>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestMethodRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestClassRunner:RunTestMethodAsync <0x000a2>
	  at <RunTestMethodsAsync>d__38:MoveNext <0x003a4>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestClassRunner`1:RunTestMethodsAsync <0x00060>
	  at <RunAsync>d__37:MoveNext <0x0020c>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestClassRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCollectionRunner:RunTestClassAsync <0x000aa>
	  at <RunTestClassesAsync>d__28:MoveNext <0x001a4>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCollectionRunner`1:RunTestClassesAsync <0x00060>
	  at <RunAsync>d__27:MoveNext <0x0020c>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCollectionRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestAssemblyRunner:RunTestCollectionAsync <0x00076>
	  at <>c__DisplayClass14_2:<RunTestCollectionsAsync>b__2 <0x00092>
	  at System.Threading.Tasks.Task`1:InnerInvoke <0x00052>
	  at <>c:<.cctor>b__281_0 <0x0001c>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x001f0>
	  at System.Threading.Tasks.Task:ExecuteEntry <0x000b6>
	  at <>c:<.cctor>b__8_0 <0x0001c>
	  at Xunit.Sdk.MaxConcurrencySyncContext:RunOnSyncContext <0x00042>
	  at <>c__DisplayClass11_0:<WorkerThreadProc>b__0 <0x00056>
	  at System.Action`1:invoke_void_T <0x00080>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Object:lambda_method2 <0x0004a>
	  at Xunit.Sdk.ExecutionContextHelper:Run <0x00034>
	  at Xunit.Sdk.MaxConcurrencySyncContext:WorkerThreadProc <0x0016c>
	  at <>c:<QueueUserWorkItem>b__5_0 <0x00038>
	  at System.Threading.Tasks.Task:InnerInvoke <0x00092>
	  at <>c:<.cctor>b__281_0 <0x0001c>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x001f0>
	  at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x00086>
	  at <>c:<.cctor>b__10_0 <0x00020>
	  at System.Object:runtime_invoke_direct_void__this__ <0x0022c>
	  at <unknown> <0x00000>
=================================================================
./RunTests.sh: line 168:    22 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Linq.Queryable.Tests.runtimeconfig.json --depsfile System.Linq.Queryable.Tests.deps.json xunit.console.dll System.Linq.Queryable.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Fri 25 Aug 2023 11:45:42 PM UTC ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.

carlossanlop · 2023-09-01T18:29:56Z

Here's another recent hit:

8.0 PR: [release/8.0] Options Source Gen Fixes #91432
Queue: Libraries Test Run release mono_interpreter linux x64 Debug
Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=392484&view=logs&j=9310185d-988f-5930-4a09-6918c1bb36fb&t=70410075-93d6-55d0-0186-00493ab8b56c&l=55
Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-91432-merge-2d8b0c654bb74f798e/System.Linq.Queryable.Tests/1/console.ee04ed98.log?helixlogtype=result
Output:

===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Linq.Queryable.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Linq.Queryable.Tests (found 785 test cases)
  Starting:    System.Linq.Queryable.Tests (parallel test collections = on, max threads = 2)

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7fa43adec811 - Unknown
	0x7fa43ad93b2e - Unknown
	0x7fa43acfbb31 - Unknown
	0x7fa43b627140 - Unknown
	0x7fa43adefd69 - Unknown
	0x7fa43adee336 - Unknown
	0x7fa43af1bc27 - Unknown
	0x7fa43af247db - Unknown
	0x7fa43aedbdf3 - Unknown
	0x7fa43aee640b - Unknown
	0x7fa43ae009ff - Unknown
	0x7fa43adff1b5 - Unknown
	0x7fa43adf0a3a - Unknown
	0x7fa43adee336 - Unknown
	0x7fa43af1bc27 - Unknown
	0x7fa43af33e44 - Unknown
	0x7fa43b61bea7 - Unknown
	0x7fa43b20aa2f - Unknown

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 23]
[New LWP 24]
[New LWP 25]
[New LWP 26]
[New LWP 29]
[New LWP 30]
[New LWP 31]
[New LWP 32]
[New LWP 33]
[New LWP 34]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
DW_FORM_rnglistx index pointing outside of .debug_rnglists offset array [in module /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so.dbg]
DW_FORM_rnglistx index pointing outside of .debug_rnglists offset array [in module /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libSystem.Native.so.dbg]
futex_wait_cancelable (private=0, expected=0, futex_word=0x558319c09954) at ../sysdeps/nptl/futex-internal.h:186
186	../sysdeps/nptl/futex-internal.h: No such file or directory.
  Id   Target Id                                        Frame 
* 1    Thread 0x7fa43b10a740 (LWP 22) "dotnet"          futex_wait_cancelable (private=0, expected=0, futex_word=0x558319c09954) at ../sysdeps/nptl/futex-internal.h:186
  2    Thread 0x7fa43a3ff700 (LWP 23) "SGen worker"     futex_wait_cancelable (private=0, expected=0, futex_word=0x7fa43afeddf8 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:186
  3    Thread 0x7fa438601700 (LWP 24) ".NET EventPipe"  0x00007fa43b1fe96f in __GI___poll (fds=0x7fa434002890, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x7fa4382ff700 (LWP 25) "Finalizer"       futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fa43afdefe0 <finalizer_sem>) at ../sysdeps/nptl/futex-internal.h:323
  5    Thread 0x7fa431467700 (LWP 26) ".NET SigHandler" __libc_read (nbytes=1, buf=0x7fa431466ee7, fd=7) at ../sysdeps/unix/sysv/linux/read.c:26
  6    Thread 0x7fa4310c8700 (LWP 29) ".NET Long Runni" futex_wait_cancelable (private=0, expected=0, futex_word=0x7fa42402e050) at ../sysdeps/nptl/futex-internal.h:186
  7    Thread 0x7fa4313ca700 (LWP 30) ".NET TP Worker"  futex_abstimed_wait_cancelable (private=0, abstime=0x7fa4313c9668, clockid=826054096, expected=0, futex_word=0x7fa420028730) at ../sysdeps/nptl/futex-internal.h:323
  8    Thread 0x7fa431142700 (LWP 31) ".NET TP Gate"    futex_abstimed_wait_cancelable (private=0, abstime=0x7fa4311417b8, clockid=823400224, expected=0, futex_word=0x7fa41800ade0) at ../sysdeps/nptl/futex-internal.h:323
  9    Thread 0x7fa4307e7700 (LWP 32) ".NET TP Worker"  futex_abstimed_wait_cancelable (private=0, abstime=0x7fa4307e66e8, clockid=813590112, expected=0, futex_word=0x7fa4307e6768) at ../sysdeps/nptl/futex-internal.h:323
  10   Thread 0x7fa4304e5700 (LWP 33) ".NET Long Runni" 0x00007fa43ae02a20 in interp_cprop () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
  11   Thread 0x7fa417fff700 (LWP 34) ".NET Long Runni" 0x00007fa43b1d6747 in __GI___wait4 (pid=35, stat_loc=0x7fa417ffb080, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27

Thread 11 (Thread 0x7fa417fff700 (LWP 34) ".NET Long Runni"):
#0  0x00007fa43b1d6747 in __GI___wait4 (pid=35, stat_loc=0x7fa417ffb080, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
#1  0x00007fa43adec942 in mono_dump_native_crash_info () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#2  0x00007fa43ad93b2e in mono_handle_native_crash () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#3  0x00007fa43acfbb31 in mono_sigsegv_signal_handler_debug () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  <signal handler called>
#5  0x00007fa43adefd69 in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43af247db in mono_runtime_try_invoke_byrefs () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43aedbdf3 in ves_icall_InternalInvoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007fa43aee640b in ves_icall_InternalInvoke_raw () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007fa43ae009ff in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#12 0x00007fa43adff1b5 in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#13 0x00007fa43adf0a3a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#14 0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#15 0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#16 0x00007fa43af33e44 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#17 0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7fa4304e5700 (LWP 33) ".NET Long Runni"):
#0  0x00007fa43ae02a20 in interp_cprop () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#1  0x00007fa43ae16f5c in generate () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#2  0x00007fa43ae166df in mono_interp_transform_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#3  0x00007fa43ae2b483 in tier_up_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007fa43ae2b1a2 in mono_interp_tier_up_frame_enter () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43adfc8c3 in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43af33e44 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#10 0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7fa4307e7700 (LWP 32) ".NET TP Worker"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7fa4307e66e8, clockid=813590112, expected=0, futex_word=0x7fa4307e6768) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0x7fa4307e66e8, clockid=813590112, mutex=0x7fa42407f440, cond=0x7fa4307e6740) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x7fa4307e6740, mutex=0x7fa42407f440, abstime=0x7fa4307e66e8) at pthread_cond_wait.c:656
#3  0x00007fa43ae9da93 in mono_os_cond_timedwait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007fa43aea2939 in mono_lifo_semaphore_timed_wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43ae00977 in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adff1b5 in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43adf0a3a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007fa43af33e44 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#12 0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7fa431142700 (LWP 31) ".NET TP Gate"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7fa4311417b8, clockid=823400224, expected=0, futex_word=0x7fa41800ade0) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0x7fa4311417b8, clockid=823400224, mutex=0x7fa41800ad90, cond=0x7fa41800adb8) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x7fa41800adb8, mutex=0x7fa41800ad90, abstime=0x7fa4311417b8) at pthread_cond_wait.c:656
#3  0x00007fa4380f8a97 in SystemNative_LowLevelMonitor_TimedWait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007fa43ae00977 in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43adff16f in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adf0a3a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43af33e44 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7fa4313ca700 (LWP 30) ".NET TP Worker"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7fa4313c9668, clockid=826054096, expected=0, futex_word=0x7fa420028730) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0x7fa4313c9668, clockid=826054096, mutex=0x7fa4200286e0, cond=0x7fa420028708) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x7fa420028708, mutex=0x7fa4200286e0, abstime=0x7fa4313c9668) at pthread_cond_wait.c:656
#3  0x00007fa4380f8a97 in SystemNative_LowLevelMonitor_TimedWait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007fa43ae00977 in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43adff16f in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adf0a3a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43af33e44 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7fa4310c8700 (LWP 29) ".NET Long Runni"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7fa42402e050) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fa42402e000, cond=0x7fa42402e028) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x7fa42402e028, mutex=0x7fa42402e000) at pthread_cond_wait.c:638
#3  0x00007fa4380f8937 in SystemNative_LowLevelMonitor_Wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007fa43ae0092e in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43adff16f in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adf0a3a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43af33e44 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7fa431467700 (LWP 26) ".NET SigHandler"):
#0  __libc_read (nbytes=1, buf=0x7fa431466ee7, fd=7) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __libc_read (fd=7, buf=0x7fa431466ee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x00007fa4380f8086 in SignalHandlerLoop () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#3  0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#4  0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fa4382ff700 (LWP 25) "Finalizer"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7fa43afdefe0 <finalizer_sem>) at ../sysdeps/nptl/futex-internal.h:323
#1  do_futex_wait (sem=sem@entry=0x7fa43afdefe0 <finalizer_sem>, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x00007fa43b625278 in __new_sem_wait_slow (sem=0x7fa43afdefe0 <finalizer_sem>, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x00007fa43af5b098 in finalizer_thread () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007fa43af33d99 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fa438601700 (LWP 24) ".NET EventPipe"):
#0  0x00007fa43b1fe96f in __GI___poll (fds=0x7fa434002890, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fa43ae7e5fe in ds_ipc_poll () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#2  0x00007fa43ae7b328 in ds_ipc_stream_factory_get_next_available_stream () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#3  0x00007fa43ae7984f in server_thread () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007fa43ae7da61 in ep_rt_thread_mono_start_func () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fa43a3ff700 (LWP 23) "SGen worker"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7fa43afeddf8 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fa43afedda8 <lock>, cond=0x7fa43afeddd0 <work_cond>) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x7fa43afeddd0 <work_cond>, mutex=0x7fa43afedda8 <lock>) at pthread_cond_wait.c:638
#3  0x00007fa43afc2b53 in thread_func () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007fa43b61bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5  0x00007fa43b20aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fa43b10a740 (LWP 22) "dotnet"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x558319c09954) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x558319c09900, cond=0x558319c09928) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x558319c09928, mutex=0x558319c09900) at pthread_cond_wait.c:638
#3  0x00007fa4380f8937 in SystemNative_LowLevelMonitor_Wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007fa43ae0092e in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007fa43adff16f in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007fa43adf0a3a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007fa43adee336 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007fa43af1bc27 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007fa43af22898 in mono_runtime_exec_main_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007fa43ad5457f in mono_jit_exec () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007fa43ad56deb in mono_main () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#12 0x00007fa43adcab77 in monovm_execute_assembly () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#13 0x00007fa43b026778 in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#14 0x00007fa43b054dea in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#15 0x00007fa43b0550df in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#16 0x00007fa43b056082 in corehost_main () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#17 0x00007fa43b0d0afc in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#18 0x00007fa43b0cbc73 in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#19 0x00007fa43b0c80ec in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#20 0x00007fa43b0c71cb in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#21 0x00007fa43b0c0115 in hostfxr_main_startupinfo () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#22 0x0000558317ede312 in ?? ()
#23 0x0000558317ede835 in ?? ()
#24 0x00007fa43b132d0a in __libc_start_main (main=0x558317ede750, argc=18, argv=0x7ffc8c56dc78, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc8c56dc68) at ../csu/libc-start.c:308
#25 0x0000558317ed1979 in ?? ()
[Inferior 1 (process 22) detached]

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7fa43adefd69):0x7fa43adefd59  1d 00 49 89 db ff 24 c1 49 8b 44 24 08 48 8b 00  ..I...$.I.D$.H..
0x7fa43adefd69  48 8b 70 18 48 8d 3d 3e 48 e6 ff 31 c0 e8 05 9f  H.p.H.=>H..1....
0x7fa43adefd79  09 00 48 8d 3d 67 8f e4 ff be 17 0f 00 00 e8 c4  ..H.=g..........
0x7fa43adefd89  a3 09 00 0f 1f 40 00 4c 89 e6 4c 89 db 48 83 c3  [email protected]..

=================================================================
	Managed Stacktrace:
=================================================================
	  at <unknown> <0xffffffff>
	  at System.Reflection.RuntimeMethodInfo:InternalInvoke <0x00096>
	  at System.Reflection.MethodBaseInvoker:InterpretedInvoke_Method <0x0003c>
	  at System.Reflection.MethodBaseInvoker:InvokeWithNoArgs <0x0006e>
	  at System.Reflection.RuntimeMethodInfo:Invoke <0x000fe>
	  at System.Reflection.MethodBase:Invoke <0x0002e>
	  at Xunit.Sdk.TestInvoker`1:CallTestMethod <0x00030>
	  at <<InvokeTestMethodAsync>b__1>d:MoveNext <0x00242>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__1 <0x00060>
	  at <AggregateAsync>d__4:MoveNext <0x0007a>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at Xunit.Sdk.ExecutionTimer:AggregateAsync <0x00074>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__0 <0x00082>
	  at <RunAsync>d__9:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at <InvokeTestMethodAsync>d__48:MoveNext <0x00122>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestInvoker`1:InvokeTestMethodAsync <0x00074>
	  at Xunit.Sdk.XunitTestInvoker:InvokeTestMethodAsync <0x000b8>
	  at <<RunAsync>b__47_0>d:MoveNext <0x003ba>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestInvoker`1:<RunAsync>b__47_0 <0x00060>
	  at <RunAsync>d__10`1:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at Xunit.Sdk.TestInvoker`1:RunAsync <0x0003a>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestMethodAsync <0x000a8>
	  at <InvokeTestAsync>d__4:MoveNext <0x00178>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestAsync <0x00074>
	  at <>c__DisplayClass43_0:<RunAsync>b__0 <0x0002c>
	  at <RunAsync>d__10`1:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at <RunAsync>d__43:MoveNext <0x002aa>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCaseRunner:RunTestAsync <0x00104>
	  at <RunAsync>d__19:MoveNext <0x001ee>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCaseRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCase:RunAsync <0x0006a>
	  at Xunit.Sdk.XunitTestMethodRunner:RunTestCaseAsync <0x00076>
	  at <RunTestCasesAsync>d__32:MoveNext <0x000fe>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestMethodRunner`1:RunTestCasesAsync <0x00060>
	  at <RunAsync>d__31:MoveNext <0x00114>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestMethodRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestClassRunner:RunTestMethodAsync <0x000a2>
	  at <RunTestMethodsAsync>d__38:MoveNext <0x003a4>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestClassRunner`1:RunTestMethodsAsync <0x00060>
	  at <RunAsync>d__37:MoveNext <0x0020c>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestClassRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCollectionRunner:RunTestClassAsync <0x000aa>
	  at <RunTestClassesAsync>d__28:MoveNext <0x001a4>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCollectionRunner`1:RunTestClassesAsync <0x00060>
	  at <RunAsync>d__27:MoveNext <0x0020c>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCollectionRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestAssemblyRunner:RunTestCollectionAsync <0x00076>
	  at <>c__DisplayClass14_2:<RunTestCollectionsAsync>b__2 <0x00092>
	  at System.Threading.Tasks.Task`1:InnerInvoke <0x00052>
	  at <>c:<.cctor>b__281_0 <0x0001c>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x001f0>
	  at System.Threading.Tasks.Task:ExecuteEntry <0x000b6>
	  at <>c:<.cctor>b__8_0 <0x0001c>
	  at Xunit.Sdk.MaxConcurrencySyncContext:RunOnSyncContext <0x00042>
	  at <>c__DisplayClass11_0:<WorkerThreadProc>b__0 <0x00056>
	  at System.Action`1:invoke_void_T <0x00080>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Object:lambda_method2 <0x0004a>
	  at Xunit.Sdk.ExecutionContextHelper:Run <0x00034>
	  at Xunit.Sdk.MaxConcurrencySyncContext:WorkerThreadProc <0x0016c>
	  at <>c:<QueueUserWorkItem>b__5_0 <0x00038>
	  at System.Threading.Tasks.Task:InnerInvoke <0x00092>
	  at <>c:<.cctor>b__281_0 <0x0001c>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x001f0>
	  at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x00086>
	  at <>c:<.cctor>b__10_0 <0x00020>
	  at System.Object:runtime_invoke_direct_void__this__ <0x0022c>
	  at <unknown> <0x00000>
=================================================================
/root/helix/work/workitem/e
./RunTests.sh: line 168:    22 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Linq.Queryable.Tests.runtimeconfig.json --depsfile System.Linq.Queryable.Tests.deps.json xunit.console.dll System.Linq.Queryable.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
----- end Fri 01 Sep 2023 12:29:36 AM UTC ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
ulimit -c value: unlimited
[ 2375.998335]  entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[ 2375.998337] RIP: 0033:0x7f6cadb3311b
[ 2375.998338] Code: 0f 05 48 3d 00 f0 ff ff 77 45 c3 0f 1f 40 00 48 83 ec 18 89 7c 24 0c e8 f3 fb ff ff 8b 7c 24 0c 41 89 c0 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 89 44 24 0c e8 31 fc ff ff 8b 44
[ 2375.998339] RSP: 002b:00007f6ca2bcfe30 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
[ 2375.998340] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f6cadb3311b
[ 2375.998341] RDX: 0000000000000000 RSI: 0000000000000007 RDI: 0000000000000012
[ 2375.998342] RBP: 00007f6ca2bcfe60 R08: 0000000000000000 R09: 0000000000000006
[ 2375.998342] R10: 00007f6ca2bcfd90 R11: 0000000000000293 R12: 0000000000000000
[ 2375.998343] R13: 0000000000000000 R14: 00007f6c80001c60 R15: 0000000000000012
[ 2375.998345] ---[ end trace f9ac0af0f2808493 ]---
[ 2378.475234] can: request_module (can-proto-0) failed.
[ 2383.576928] docker0: port 1(vethe5ca674) entered disabled state
[ 2383.577594] veth17fbed9: renamed from eth0
[ 2383.678445] docker0: port 1(vethe5ca674) entered disabled state
[ 2383.684547] device vethe5ca674 left promiscuous mode
[ 2383.684562] docker0: port 1(vethe5ca674) entered disabled state
[ 2396.822816] docker0: port 1(veth13b9403) entered blocking state
[ 2396.822818] docker0: port 1(veth13b9403) entered disabled state
[ 2396.822974] device veth13b9403 entered promiscuous mode
[ 2397.131010] eth0: renamed from vethaaad365
[ 2397.187877] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 2397.187902] IPv6: ADDRCONF(NETDEV_CHANGE): veth13b9403: link becomes ready
[ 2397.187935] docker0: port 1(veth13b9403) entered blocking state
[ 2397.187937] docker0: port 1(veth13b9403) entered forwarding state
[ 2405.259833] docker0: port 1(veth13b9403) entered disabled state
[ 2405.259898] vethaaad365: renamed from eth0
[ 2405.368180] docker0: port 1(veth13b9403) entered disabled state
[ 2405.372802] device veth13b9403 left promiscuous mode
[ 2405.372815] docker0: port 1(veth13b9403) entered disabled state
[ 2416.166844] docker0: port 1(vethb84acd7) entered blocking state
[ 2416.166846] docker0: port 1(vethb84acd7) entered disabled state
[ 2416.166958] device vethb84acd7 entered promiscuous mode
[ 2416.483555] eth0: renamed from vethaf30f8c
[ 2416.541940] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 2416.541964] IPv6: ADDRCONF(NETDEV_CHANGE): vethb84acd7: link becomes ready
[ 2416.541997] docker0: port 1(vethb84acd7) entered blocking state
[ 2416.541998] docker0: port 1(vethb84acd7) entered forwarding state
[ 2418.189035] docker0: port 1(vethb84acd7) entered disabled state
[ 2418.189097] vethaf30f8c: renamed from eth0
[ 2418.295760] docker0: port 1(vethb84acd7) entered disabled state
[ 2418.302104] device vethb84acd7 left promiscuous mode
[ 2418.302115] docker0: port 1(vethb84acd7) entered disabled state
[ 2426.257601] docker0: port 1(veth3cb3d94) entered blocking state
[ 2426.257603] docker0: port 1(veth3cb3d94) entered disabled state
[ 2426.257677] device veth3cb3d94 entered promiscuous mode
[ 2426.552209] eth0: renamed from veth4fbbe8b
[ 2426.613649] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 2426.613672] IPv6: ADDRCONF(NETDEV_CHANGE): veth3cb3d94: link becomes ready
[ 2426.613703] docker0: port 1(veth3cb3d94) entered blocking state
[ 2426.613705] docker0: port 1(veth3cb3d94) entered forwarding state
Waiting a few seconds for any dump to be written..
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 0
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dump..
... found no dump in /root/helix/work/workitem/e
+ export _commandExitCode=134
+ python /root/helix/work/correlation/reporter/run.py https://dev.azure.com/dnceng-public/ public 8436822 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiI3N2Y3M2NkOC00YzQxLTRhNTEtYTMxOS1hMjA4MzFlYjllNGIiLCJzaWQiOiIwZjRlZmJjYy0wYmM5LTQ4NDItODJkZS1hNDQ1MGI2NzJhZDkiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzM5MjQ4NCIsImpvYnJlZiI6IjI0NzhhZDEyLTk2Y2ItNDQ2OS1iNTFlLTQxNmViNjQyN2FiMDo5MzEwMTg1ZC05ODhmLTU5MzAtNGEwOS02OTE4YzFiYjM2ZmIiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvMzkyNDg0Iiwib3JjaGlkIjoiMjQ3OGFkMTItOTZjYi00NDY5LWI1MWUtNDE2ZWI2NDI3YWIwLmJ1aWxkLmxpYnJhcmllc190ZXN0X3J1bl9yZWxlYXNlX21vbm9faW50ZXJwcmV0ZXJfbGludXhfeDY0X2RlYnVnLl9fZGVmYXVsdCIsInJlcG9JZHMiOiIiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286NmZjYzkyZTUtNzNhNy00Zjg4LThkMTMtZDkwNDViNDVmYjI3IiwibmJmIjoxNjkzNTI2NjczLCJleHAiOjE2OTM1MzY4NzN9.sjxF_cyOrQMYjxUnK59W_yzEa1-RQcHRClnJxborVUk4CqfdJVqT8Goio5w8ZXPuOuCAqgaGdfuHrW1sWPk37OSr9a164QAhbHFdVwQ2DhQXxwHUDTUkV5xOBkmzB38nHtD1b36Zdr4PG1e_DjjNi1wScWuUi2gSyUNqKIx2lWgWmeSDgXcs4VazGHOnIFOiNiqiTmNq86jul-1ZnpXsHLObbJHl-0eH0spPVxwKAIiyvKZhxdRjxQxx1bBuPLTklJ9BigepHNFgEP9Wz0pGuSvJizLVKLesBiZ9c39sLDGgKF1g_aNr7dnd7xcMlp1GQ7VL7G7KzCacRyjzkuCprA
2023-09-01T00:29:46.993Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2023-09-01T00:29:46.993Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/e' for test results files
2023-09-01T00:29:46.994Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/uploads' for test results files
2023-09-01T00:29:46.994Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2023-09-01T00:29:46.994Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/root/helix/work/workitem/e/__test_report.json'
2023-09-01T00:29:46.994Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1565 bytes
+ exit 134
+ export _commandExitCode=134
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 134

[END EXECUTION]
Exit Code:134

carlossanlop · 2023-10-18T15:49:09Z

This is very impactful and affecting both 7.0 and 8.0. @lewing do you know who can take a look? This is mono.

lewing · 2023-10-18T16:05:05Z

@lambdageek can you triage this?

lambdageek · 2023-10-18T16:47:46Z

@carlossanlop Please don't make "futex-internal.h: No such file or directory" tracking issues. That is a generic error message when gdb tries to look up source code for libc after a crash on Linux (mono invokes gdb from its crash handler if it is installed on the system). It will catch literally every single native Mono crash - any two completely unrelated issues.

carlossanlop · 2023-10-18T16:50:06Z

Ok, in that case, let's close this and start over.

carlossanlop added arch-arm64 blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-VM-meta-mono runtime-mono specific to the Mono runtime area-Extensions-Logging Known Build Error Use this to report build issues in the .NET Helix tab labels Jun 13, 2023

ghost added the untriaged New issue has not been triaged by the area owner label Jun 13, 2023

carlossanlop added the os-linux Linux OS (any supported distro) label Jun 13, 2023

tarekgh removed the area-Extensions-Logging label Jun 13, 2023

marek-safar removed the untriaged New issue has not been triaged by the area owner label Jul 26, 2023

marek-safar added this to the 7.0.x milestone Jul 26, 2023

This was referenced Aug 24, 2023

Crash in Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution #90019

Open

[release/7.0-staging] Update dependencies from dotnet/emsdk #91083

Merged

carlossanlop mentioned this issue Oct 18, 2023

[release/8.0] Update dependencies from dotnet/roslyn #93642

Merged

carlossanlop closed this as completed Oct 18, 2023

ghost locked as resolved and limited conversation to collaborators Nov 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[7.0][mono][arm64] futex-internal.h: No such file or directory - futex_wait_cancelable native crash in Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests #87444

[7.0][mono][arm64] futex-internal.h: No such file or directory - futex_wait_cancelable native crash in Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests #87444

carlossanlop commented Jun 13, 2023 •

edited by build-analysis bot

Loading

ghost commented Jun 13, 2023

Error Blob

Reproduction Steps

carlossanlop commented Aug 28, 2023

carlossanlop commented Sep 1, 2023

carlossanlop commented Oct 18, 2023

lewing commented Oct 18, 2023

lambdageek commented Oct 18, 2023

carlossanlop commented Oct 18, 2023

[7.0][mono][arm64] futex-internal.h: No such file or directory - futex_wait_cancelable native crash in Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests #87444

[7.0][mono][arm64] futex-internal.h: No such file or directory - futex_wait_cancelable native crash in Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests #87444

Comments

carlossanlop commented Jun 13, 2023 • edited by build-analysis bot Loading

Error Blob

Reproduction Steps

Report

Summary

Known issue validation

ghost commented Jun 13, 2023

Error Blob

Reproduction Steps

carlossanlop commented Aug 28, 2023

carlossanlop commented Sep 1, 2023

carlossanlop commented Oct 18, 2023

lewing commented Oct 18, 2023

lambdageek commented Oct 18, 2023

carlossanlop commented Oct 18, 2023

carlossanlop commented Jun 13, 2023 •

edited by build-analysis bot

Loading