Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Random crashes with no errors logged, related to memory access violations #68602

Closed
gelvinp opened this issue Nov 13, 2022 · 9 comments
Closed

Comments

@gelvinp
Copy link
Contributor

gelvinp commented Nov 13, 2022

Godot version

4.0 beta 4 official

System information

Windows 10

Issue description

I've been experiencing a lot of random crashes today, more so than usual, and the behavior is completely unpredictable.
I've seen crashes from running my game from the editor, running with the official export templates, and running with my own compiled export templates. Sometimes it crashes after being on the main menu for half a second, sometimes it crashes a few minutes into the game. Sometimes it crashes immediately five or six times in a row, and other times I can play through the entire game five or six times in a row without crash. Annoyingly, since I've been poking around with this, I haven't been changing the actual project, so there isn't a node or a script that I'm adding/removing that's correlating to it crashing or not.

There's no logs output to the console or to file, even with the --verbose flag, the entire log is just

Godot Engine v4.0.beta4.official.e6751549c - https://godotengine.org
Vulkan API 1.2.0 - Using Vulkan Device #0: NVIDIA - NVIDIA GeForce RTX 2060 SUPER

Additionally, I've built and ran in visual studio, trying to catch whatever might be causing it, but I haven't even been able to get breakpoints working so that hasn't been fruitful.

The windows Event Viewer logs show the following:

Faulting application name: Godot_v4.0-beta4_win64.exe, version: 4.0.0.0, time stamp: 0x00000000
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x000000000000471a
Faulting process id: 0x1200
Faulting application start time: 0x01d8f74569c0ad11
Faulting application path: C:\Users\thePG\Godot\Godot 4.0-beta4\Godot_v4.0-beta4_win64.exe
Faulting module path: unknown
Report Id: ff82a632-4992-424a-bc9f-b42a1b2a2baa
Faulting package full name: 
Faulting package-relative application ID: 

Unlike #68443, I'm not using C#, or mono builds of the engine, and I don't have a pop up dialog with an error code. The game freezes, the audio stutters, and then it quits to desktop.

Finally, loading up AppCrashView shows 21 crashes since I started working this afternoon, most of them with an access violation. Given that I've only had 7 other crashes since the 4th of november, I'm pretty sure it's something I added to my project today. Interestingly, the faulting module seems to alternate between the editor exe and "StackHash_{digits}", as seen below:
image

If there's anything I can collect or send to help track down where this is crashing and why, please let me know.

Steps to reproduce

I'm not sure I'm able to replicate this in a second minimal example project, I don't know what specifically is causing the issue. The huge uptick in crashes started around 3PM. I've looked at my git commit history around that time, and besides renaming a scene, and adding some AudioStreamPlayer3Ds, I haven't done anything I didn't do yesterday or the days or weeks before, yet the crashes really only started at 3 PM. Additionally, none of the work I did at that time affected the main menu scene, so I'm not sure how that would have caused the game to crash within seconds of launching. None of the changed files would even have been loaded yet.

This has been happening regardless of how I start my game. I've ran from the editor, from the project manager, from exports with the official templates, custom compiled exports. The only step is running my game, and then there's about a 50% chance that either it will crash immediately several times in a row, or it will be fine.

Given the lack of any logs, and visual studio not actually setting breakpoints, I'm at a loss as to where to continue trying to figure out where this is breaking.

Minimal reproduction project

No response

@Chaosus
Copy link
Member

Chaosus commented Nov 13, 2022

Such problems often happens if Godot running on the dying HDD, systems with corrupted USB-ports, memory, drivers, installed malware programs (or even good one like MSI Afterburner). I suggest checking your system up.

@gelvinp
Copy link
Contributor Author

gelvinp commented Nov 13, 2022

I'm reasonably confident this isn't a hardware problem, I upgraded by machine a few months ago and as part of that, put it through some stress tests to make sure it was all working. Additionally, this windows install is only 4 months old, in line with that hardware upgrade, so I don't have any conflicting or stale drivers. I've sent the files off to some other people as well to see if they find crashes.

I've compiled godot with asan, and it looks like there's a few things happening.

  1. There is reliably a heap buffer overflow when setting the windows native icon
=================================================================
==20768==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x1214ecaa8426 at pc 0x7ff70e694a9c bp 0x0050fdbfc1d0 sp 0x0050fdbfb950
WRITE of size 144 at 0x1214ecaa8426 thread T0
    #0 0x7ff70e694ac9 in __asan_wrap_memmove D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\sanitizer_common\sanitizer_common_interceptors.inc:794
    #1 0x7ff71756bc95 in _fread_nolock_s minkernel\crts\ucrt\src\appcrt\stdio\fread.cpp:131
    #2 0x7ff71756be88 in fread_s minkernel\crts\ucrt\src\appcrt\stdio\fread.cpp:56
    #3 0x7ff71756bdeb in fread minkernel\crts\ucrt\src\appcrt\stdio\fread.cpp:239
    #4 0x7ff70ffe97eb in FileAccessWindows::get_buffer(unsigned char *, unsigned __int64) const C:\Users\thePG\Documents\Godot\Entombed\build\godot\drivers\windows\file_access_windows.cpp:268
    #5 0x7ff70e5f7396 in DisplayServerWindows::set_native_icon(class String const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\display_server_windows.cpp:1885
    #6 0x7ff70e6d3026 in Main::start(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\main\main.cpp:2990
    #7 0x7ff70e58bc3c in widechar_main(int, wchar_t **) C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:178
    #8 0x7ff70e58bf26 in _main(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:203
    #9 0x7ff70e58bfa1 in main C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:217
    #10 0x7ff70e58ad9c in WinMain C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:230
    #11 0x7ff717531d09 in invoke_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:102
    #12 0x7ff717531d09 in __scrt_common_main_seh D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #13 0x7ffa700c74b3  (C:\Windows\System32\KERNEL32.DLL+0x1800174b3)
    #14 0x7ffa709226a0  (C:\Windows\SYSTEM32\ntdll.dll+0x1800526a0)

0x1214ecaa8426 is located 0 bytes to the right of 166-byte region [0x1214ecaa8380,0x1214ecaa8426)
allocated by thread T0 here:
    #0 0x7ff70e690a47 in __sanitizer::BufferedStackTrace::Unwind D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\sanitizer_common\sanitizer_stacktrace.h:115
    #1 0x7ff70e690a47 in realloc D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_malloc_win.cpp:144
    #2 0x7ff7155a4c3c in Memory::realloc_static(void *, unsigned __int64, bool) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\os\memory.cpp:129
    #3 0x7ff70e5f72ce in DisplayServerWindows::set_native_icon(class String const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\display_server_windows.cpp:1884
    #4 0x7ff70e6d3026 in Main::start(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\main\main.cpp:2990
    #5 0x7ff70e58bc3c in widechar_main(int, wchar_t **) C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:178
    #6 0x7ff70e58bf26 in _main(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:203
    #7 0x7ff70e58bfa1 in main C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:217
    #8 0x7ff70e58ad9c in WinMain C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:230
    #9 0x7ff717531d09 in invoke_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:102
    #10 0x7ff717531d09 in __scrt_common_main_seh D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #11 0x7ffa700c74b3  (C:\Windows\System32\KERNEL32.DLL+0x1800174b3)
    #12 0x7ffa709226a0  (C:\Windows\SYSTEM32\ntdll.dll+0x1800526a0)

SUMMARY: AddressSanitizer: heap-buffer-overflow D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\sanitizer_common\sanitizer_common_interceptors.inc:794 in __asan_wrap_memmove
Shadow bytes around the buggy address:
  0x04398a3d5030: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x04398a3d5040: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
  0x04398a3d5050: fa fa fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x04398a3d5060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
  0x04398a3d5070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x04398a3d5080: 00 00 00 00[06]fa fa fa fa fa fa fa fa fa 00 00
  0x04398a3d5090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x04398a3d50a0: 00 00 00 fa fa fa fa fa fa fa fa fa 00 00 00 00
  0x04398a3d50b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x04398a3d50c0: 00 00 fa fa fa fa fa fa fa fa fd fd fd fd fd fd
  0x04398a3d50d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==20768==ABORTING

I don't think this is what was causing the crash, but it is worth noting. Disabling the native icon lets me move on and get to the main menu screen, which (with asan) crashes every time after a few seconds.

  1. The crashes always involve GodotStep3D and are either read access violations:
=================================================================
==27912==ERROR: AddressSanitizer: access-violation on unknown address 0x000000000010 (pc 0x7ff70e5c655b bp 0x000000000000 sp 0x0021e35feb50 T32)
==27912==The signal is caused by a READ memory access.
==27912==Hint: address points to the zero page.
    #0 0x7ff70e5c655a in HashMapComparatorDefault<__int64>::compare(__int64 const &, __int64 const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\hashfuncs.h:391
    #1 0x7ff7164bd93c in HashMap<__int64, struct WorkerThreadPool::Group *, struct HashMapHasherDefault, struct HashMapComparatorDefault<__int64>, class DefaultTypedAllocator<struct HashMapElement<__int64, struct WorkerThreadPool::Group *>>>::_lookup_pos(__int64 const &, unsigned int &) const C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\hash_map.h:119
    #2 0x7ff7164c50e8 in HashMap<__int64, struct WorkerThreadPool::Group *, struct HashMapHasherDefault, struct HashMapComparatorDefault<__int64>, class DefaultTypedAllocator<struct HashMapElement<__int64, struct WorkerThreadPool::Group *>>>::getptr(__int64 const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\hash_map.h:300
    #3 0x7ff7164a97ed in WorkerThreadPool::wait_for_group_task_completion(__int64) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\object\worker_thread_pool.cpp:372
    #4 0x7ff7152be959 in GodotStep3D::step(class GodotSpace3D *, float) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_3d\godot_step_3d.cpp:367
    #5 0x7ff714b941a6 in GodotPhysicsServer3D::step(float) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_3d\godot_physics_server_3d.cpp:1644
    #6 0x7ff714bee8bf in PhysicsServer3DWrapMT::thread_step(float) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:40
    #7 0x7ff714c1cf53 in CommandQueueMT::Command1<class PhysicsServer3DWrapMT, void (__cdecl PhysicsServer3DWrapMT::*)(float), float>::call(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\command_queue_mt.h:322
    #8 0x7ff7142c115d in CommandQueueMT::_flush(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\command_queue_mt.h:373
    #9 0x7ff714321345 in CommandQueueMT::wait_and_flush(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\command_queue_mt.h:414
    #10 0x7ff714bee5b7 in PhysicsServer3DWrapMT::thread_loop(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:59
    #11 0x7ff714bee29c in PhysicsServer3DWrapMT::_thread_callback(void *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:47
    #12 0x7ff715901df7 in Thread::callback(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\os\thread.cpp:65
    #13 0x7ff715904227 in std::invoke<void (__cdecl *)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *, struct Thread::Settings, void (__cdecl *)(void *), void *>(void (__cdecl *&&)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *&&, struct Thread::Settings &&, void (__cdecl *&&)(void *), void *&&) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include\type_traits:1558
    #14 0x7ff715903ac4 in std::thread::_Invoke<class std::tuple<void (__cdecl *)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *, struct Thread::Settings, void (__cdecl *)(void *), void *>, 0, 1, 2, 3, 4>(void *) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include\thread:55
    #15 0x7ff7175764cb in thread_start<unsigned int (__cdecl*)(void *),1> minkernel\crts\ucrt\src\appcrt\startup\thread.cpp:97
    #16 0x7ff70e680589 in __asan::AsanThread::ThreadStart(unsigned __int64, struct __sanitizer::atomic_uintptr_t *) D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_thread.cpp:262
    #17 0x7ffa700c74b3  (C:\Windows\System32\KERNEL32.DLL+0x1800174b3)
    #18 0x7ffa709226a0  (C:\Windows\SYSTEM32\ntdll.dll+0x1800526a0)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: access-violation C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\hashfuncs.h:391 in HashMapComparatorDefault<__int64>::compare(__int64 const &, __int64 const &)
Thread T32 created by T0 here:
    #0 0x7ff70e67ef7f in __sanitizer::BufferedStackTrace::Unwind D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\sanitizer_common\sanitizer_stacktrace.h:115
    #1 0x7ff70e67ef7f in __asan_wrap_CreateThread D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_win.cpp:149
    #2 0x7ff71757688a in _beginthreadex minkernel\crts\ucrt\src\appcrt\startup\thread.cpp:209
    #3 0x7ff715903cd8 in std::thread::_Start<void (__cdecl *)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *, struct Thread::Settings const &, void (__cdecl *&)(void *), void *&>(void (__cdecl *&&)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *&&, struct Thread::Settings const &, void (__cdecl *&)(void *), void *&) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include\thread:75
    #4 0x7ff71590348b in std::thread::thread<void (__cdecl *)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *, struct Thread::Settings const &, void (__cdecl *&)(void *), void *&, 0>(void (__cdecl *&&)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *&&, struct Thread::Settings const &, void (__cdecl *&)(void *), void *&) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include\thread:90
    #5 0x7ff7159022e5 in Thread::start(void (__cdecl *)(void *), void *, struct Thread::Settings const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\os\thread.cpp:82
    #6 0x7ff714beeabc in PhysicsServer3DWrapMT::init(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:100
    #7 0x7ff70e6d67a8 in initialize_physics(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\main\main.cpp:242
    #8 0x7ff70e6c95d6 in Main::setup2(unsigned __int64) C:\Users\thePG\Documents\Godot\Entombed\build\godot\main\main.cpp:2357
    #9 0x7ff70e6c0e9c in Main::setup(char const *, int, char **const, bool) C:\Users\thePG\Documents\Godot\Entombed\build\godot\main\main.cpp:1868
    #10 0x7ff70e58ba61 in widechar_main(int, wchar_t **) C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:164
    #11 0x7ff70e58bf26 in _main(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:203
    #12 0x7ff70e58bfa1 in main C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:217
    #13 0x7ff70e58ad9c in WinMain C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:230
    #14 0x7ff717531d09 in invoke_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:102
    #15 0x7ff717531d09 in __scrt_common_main_seh D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #16 0x7ffa700c74b3  (C:\Windows\System32\KERNEL32.DLL+0x1800174b3)
    #17 0x7ffa709226a0  (C:\Windows\SYSTEM32\ntdll.dll+0x1800526a0)

==27912==ABORTING

or, heap use-after-frees:

=================================================================
==21084==ERROR: AddressSanitizer: heap-use-after-free on address 0x11de2df63960 at pc 0x7ff7164bac5a bp 0x00c6605fe5b0 sp 0x00c6605fe5b0
WRITE of size 8 at 0x11de2df63960 thread T32
    #0 0x7ff7164bac59 in HashMap<__int64, struct WorkerThreadPool::Group *, struct HashMapHasherDefault, struct HashMapComparatorDefault<__int64>, class DefaultTypedAllocator<struct HashMapElement<__int64, struct WorkerThreadPool::Group *>>>::_insert(__int64 const &, struct WorkerThreadPool::Group *const &, bool) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\hash_map.h:233
    #1 0x7ff7164b8307 in HashMap<__int64, struct WorkerThreadPool::Group *, struct HashMapHasherDefault, struct HashMapComparatorDefault<__int64>, class DefaultTypedAllocator<struct HashMapElement<__int64, struct WorkerThreadPool::Group *>>>::operator[](__int64 const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\hash_map.h:512
    #2 0x7ff7164a77d9 in WorkerThreadPool::_add_group_task(class Callable const &, void (__cdecl *)(void *, unsigned int), void *, struct WorkerThreadPool::BaseTemplateUserdata *, int, int, bool, class String const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\object\worker_thread_pool.cpp:322
    #3 0x7ff7152bf669 in WorkerThreadPool::add_template_group_task<class GodotStep3D, void (__cdecl GodotStep3D::*)(unsigned int, void *), std::nullptr_t>(class GodotStep3D *, void (__cdecl GodotStep3D::*)(unsigned int, void *), std::nullptr_t, int, int, bool, class String const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\object\worker_thread_pool.h:181
    #4 0x7ff7152be8f2 in GodotStep3D::step(class GodotSpace3D *, float) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_3d\godot_step_3d.cpp:366
    #5 0x7ff714b941a6 in GodotPhysicsServer3D::step(float) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_3d\godot_physics_server_3d.cpp:1644
    #6 0x7ff714bee8bf in PhysicsServer3DWrapMT::thread_step(float) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:40
    #7 0x7ff714c1cf53 in CommandQueueMT::Command1<class PhysicsServer3DWrapMT, void (__cdecl PhysicsServer3DWrapMT::*)(float), float>::call(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\command_queue_mt.h:322
    #8 0x7ff7142c115d in CommandQueueMT::_flush(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\command_queue_mt.h:373
    #9 0x7ff714321345 in CommandQueueMT::wait_and_flush(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\command_queue_mt.h:414
    #10 0x7ff714bee5b7 in PhysicsServer3DWrapMT::thread_loop(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:59
    #11 0x7ff714bee29c in PhysicsServer3DWrapMT::_thread_callback(void *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:47
    #12 0x7ff715901df7 in Thread::callback(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\os\thread.cpp:65
    #13 0x7ff715904227 in std::invoke<void (__cdecl *)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *, struct Thread::Settings, void (__cdecl *)(void *), void *>(void (__cdecl *&&)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *&&, struct Thread::Settings &&, void (__cdecl *&&)(void *), void *&&) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include\type_traits:1558
    #14 0x7ff715903ac4 in std::thread::_Invoke<class std::tuple<void (__cdecl *)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *, struct Thread::Settings, void (__cdecl *)(void *), void *>, 0, 1, 2, 3, 4>(void *) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include\thread:55
    #15 0x7ff7175764cb in thread_start<unsigned int (__cdecl*)(void *),1> minkernel\crts\ucrt\src\appcrt\startup\thread.cpp:97
    #16 0x7ff70e680589 in __asan::AsanThread::ThreadStart(unsigned __int64, struct __sanitizer::atomic_uintptr_t *) D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_thread.cpp:262
    #17 0x7ffa700c74b3  (C:\Windows\System32\KERNEL32.DLL+0x1800174b3)
    #18 0x7ffa709226a0  (C:\Windows\SYSTEM32\ntdll.dll+0x1800526a0)

0x11de2df63960 is located 16 bytes inside of 48-byte region [0x11de2df63950,0x11de2df63980)
freed by thread T32 here:
    #0 0x7ff70e6907a4 in __sanitizer::BufferedStackTrace::Unwind D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\sanitizer_common\sanitizer_stacktrace.h:115
    #1 0x7ff70e6907a4 in free D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_malloc_win.cpp:112
    #2 0x7ff7155a4e6a in Memory::free_static(void *, bool) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\os\memory.cpp:168
    #3 0x7ff7164b2c59 in memdelete<struct HashMapElement<__int64, struct WorkerThreadPool::Group *>>(struct HashMapElement<__int64, struct WorkerThreadPool::Group *> *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\os\memory.h:112
    #4 0x7ff7164c2667 in DefaultTypedAllocator<struct HashMapElement<__int64, struct WorkerThreadPool::Group *>>::delete_allocation(struct HashMapElement<__int64, struct WorkerThreadPool::Group *> *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\os\memory.h:206
    #5 0x7ff7164c3304 in HashMap<__int64, struct WorkerThreadPool::Group *, struct HashMapHasherDefault, struct HashMapComparatorDefault<__int64>, class DefaultTypedAllocator<struct HashMapElement<__int64, struct WorkerThreadPool::Group *>>>::erase(__int64 const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\hash_map.h:349
    #6 0x7ff7164a9d4c in WorkerThreadPool::wait_for_group_task_completion(__int64) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\object\worker_thread_pool.cpp:405
    #7 0x7ff7152be700 in GodotStep3D::step(class GodotSpace3D *, float) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_3d\godot_step_3d.cpp:347
    #8 0x7ff714b941a6 in GodotPhysicsServer3D::step(float) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_3d\godot_physics_server_3d.cpp:1644
    #9 0x7ff714bee8bf in PhysicsServer3DWrapMT::thread_step(float) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:40
    #10 0x7ff714c1cf53 in CommandQueueMT::Command1<class PhysicsServer3DWrapMT, void (__cdecl PhysicsServer3DWrapMT::*)(float), float>::call(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\command_queue_mt.h:322
    #11 0x7ff7142c115d in CommandQueueMT::_flush(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\command_queue_mt.h:373
    #12 0x7ff714321345 in CommandQueueMT::wait_and_flush(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\command_queue_mt.h:414
    #13 0x7ff714bee5b7 in PhysicsServer3DWrapMT::thread_loop(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:59
    #14 0x7ff714bee29c in PhysicsServer3DWrapMT::_thread_callback(void *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:47
    #15 0x7ff715901df7 in Thread::callback(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\os\thread.cpp:65
    #16 0x7ff715904227 in std::invoke<void (__cdecl *)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *, struct Thread::Settings, void (__cdecl *)(void *), void *>(void (__cdecl *&&)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *&&, struct Thread::Settings &&, void (__cdecl *&&)(void *), void *&&) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include\type_traits:1558
    #17 0x7ff715903ac4 in std::thread::_Invoke<class std::tuple<void (__cdecl *)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *, struct Thread::Settings, void (__cdecl *)(void *), void *>, 0, 1, 2, 3, 4>(void *) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include\thread:55
    #18 0x7ff7175764cb in thread_start<unsigned int (__cdecl*)(void *),1> minkernel\crts\ucrt\src\appcrt\startup\thread.cpp:97
    #19 0x7ff70e680589 in __asan::AsanThread::ThreadStart(unsigned __int64, struct __sanitizer::atomic_uintptr_t *) D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_thread.cpp:262
    #20 0x7ffa700c74b3  (C:\Windows\System32\KERNEL32.DLL+0x1800174b3)
    #21 0x7ffa709226a0  (C:\Windows\SYSTEM32\ntdll.dll+0x1800526a0)

previously allocated by thread T32 here:
    #0 0x7ff70e6908f4 in __sanitizer::BufferedStackTrace::Unwind D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\sanitizer_common\sanitizer_stacktrace.h:115
    #1 0x7ff70e6908f4 in malloc D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_malloc_win.cpp:121
    #2 0x7ff7155a48a3 in Memory::alloc_static(unsigned __int64, bool) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\os\memory.cpp:75
    #3 0x7ff7155a4ef9 in operator new(unsigned __int64, char const *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\os\memory.cpp:40
    #4 0x7ff7164b2dde in DefaultTypedAllocator<struct HashMapElement<__int64, struct WorkerThreadPool::Group *>>::new_allocation<struct HashMapElement<__int64, struct WorkerThreadPool::Group *>>(struct HashMapElement<__int64, struct WorkerThreadPool::Group *> const &&) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\os\memory.h:205
    #5 0x7ff7164baa52 in HashMap<__int64, struct WorkerThreadPool::Group *, struct HashMapHasherDefault, struct HashMapComparatorDefault<__int64>, class DefaultTypedAllocator<struct HashMapElement<__int64, struct WorkerThreadPool::Group *>>>::_insert(__int64 const &, struct WorkerThreadPool::Group *const &, bool) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\hash_map.h:223
    #6 0x7ff7164b8307 in HashMap<__int64, struct WorkerThreadPool::Group *, struct HashMapHasherDefault, struct HashMapComparatorDefault<__int64>, class DefaultTypedAllocator<struct HashMapElement<__int64, struct WorkerThreadPool::Group *>>>::operator[](__int64 const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\hash_map.h:512
    #7 0x7ff7164a77d9 in WorkerThreadPool::_add_group_task(class Callable const &, void (__cdecl *)(void *, unsigned int), void *, struct WorkerThreadPool::BaseTemplateUserdata *, int, int, bool, class String const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\object\worker_thread_pool.cpp:322
    #8 0x7ff7152bf669 in WorkerThreadPool::add_template_group_task<class GodotStep3D, void (__cdecl GodotStep3D::*)(unsigned int, void *), std::nullptr_t>(class GodotStep3D *, void (__cdecl GodotStep3D::*)(unsigned int, void *), std::nullptr_t, int, int, bool, class String const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\object\worker_thread_pool.h:181
    #9 0x7ff7152be699 in GodotStep3D::step(class GodotSpace3D *, float) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_3d\godot_step_3d.cpp:346
    #10 0x7ff714b941a6 in GodotPhysicsServer3D::step(float) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_3d\godot_physics_server_3d.cpp:1644
    #11 0x7ff714bee8bf in PhysicsServer3DWrapMT::thread_step(float) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:40
    #12 0x7ff714c1cf53 in CommandQueueMT::Command1<class PhysicsServer3DWrapMT, void (__cdecl PhysicsServer3DWrapMT::*)(float), float>::call(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\command_queue_mt.h:322
    #13 0x7ff7142c115d in CommandQueueMT::_flush(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\command_queue_mt.h:373
    #14 0x7ff714321345 in CommandQueueMT::wait_and_flush(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\command_queue_mt.h:414
    #15 0x7ff714bee5b7 in PhysicsServer3DWrapMT::thread_loop(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:59
    #16 0x7ff714bee29c in PhysicsServer3DWrapMT::_thread_callback(void *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:47
    #17 0x7ff715901df7 in Thread::callback(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\os\thread.cpp:65
    #18 0x7ff715904227 in std::invoke<void (__cdecl *)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *, struct Thread::Settings, void (__cdecl *)(void *), void *>(void (__cdecl *&&)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *&&, struct Thread::Settings &&, void (__cdecl *&&)(void *), void *&&) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include\type_traits:1558
    #19 0x7ff715903ac4 in std::thread::_Invoke<class std::tuple<void (__cdecl *)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *, struct Thread::Settings, void (__cdecl *)(void *), void *>, 0, 1, 2, 3, 4>(void *) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include\thread:55
    #20 0x7ff7175764cb in thread_start<unsigned int (__cdecl*)(void *),1> minkernel\crts\ucrt\src\appcrt\startup\thread.cpp:97
    #21 0x7ff70e680589 in __asan::AsanThread::ThreadStart(unsigned __int64, struct __sanitizer::atomic_uintptr_t *) D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_thread.cpp:262
    #22 0x7ffa700c74b3  (C:\Windows\System32\KERNEL32.DLL+0x1800174b3)
    #23 0x7ffa709226a0  (C:\Windows\SYSTEM32\ntdll.dll+0x1800526a0)

Thread T32 created by T0 here:
    #0 0x7ff70e67ef7f in __sanitizer::BufferedStackTrace::Unwind D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\sanitizer_common\sanitizer_stacktrace.h:115
    #1 0x7ff70e67ef7f in __asan_wrap_CreateThread D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_win.cpp:149
    #2 0x7ff71757688a in _beginthreadex minkernel\crts\ucrt\src\appcrt\startup\thread.cpp:209
    #3 0x7ff715903cd8 in std::thread::_Start<void (__cdecl *)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *, struct Thread::Settings const &, void (__cdecl *&)(void *), void *&>(void (__cdecl *&&)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *&&, struct Thread::Settings const &, void (__cdecl *&)(void *), void *&) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include\thread:75
    #4 0x7ff71590348b in std::thread::thread<void (__cdecl *)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *, struct Thread::Settings const &, void (__cdecl *&)(void *), void *&, 0>(void (__cdecl *&&)(class Thread *, struct Thread::Settings const &, void (__cdecl *)(void *), void *), class Thread *&&, struct Thread::Settings const &, void (__cdecl *&)(void *), void *&) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include\thread:90
    #5 0x7ff7159022e5 in Thread::start(void (__cdecl *)(void *), void *, struct Thread::Settings const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\os\thread.cpp:82
    #6 0x7ff714beeabc in PhysicsServer3DWrapMT::init(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\physics_server_3d_wrap_mt.cpp:100
    #7 0x7ff70e6d67a8 in initialize_physics(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\main\main.cpp:242
    #8 0x7ff70e6c95d6 in Main::setup2(unsigned __int64) C:\Users\thePG\Documents\Godot\Entombed\build\godot\main\main.cpp:2357
    #9 0x7ff70e6c0e9c in Main::setup(char const *, int, char **const, bool) C:\Users\thePG\Documents\Godot\Entombed\build\godot\main\main.cpp:1868
    #10 0x7ff70e58ba61 in widechar_main(int, wchar_t **) C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:164
    #11 0x7ff70e58bf26 in _main(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:203
    #12 0x7ff70e58bfa1 in main C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:217
    #13 0x7ff70e58ad9c in WinMain C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:230
    #14 0x7ff717531d09 in invoke_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:102
    #15 0x7ff717531d09 in __scrt_common_main_seh D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #16 0x7ffa700c74b3  (C:\Windows\System32\KERNEL32.DLL+0x1800174b3)
    #17 0x7ffa709226a0  (C:\Windows\SYSTEM32\ntdll.dll+0x1800526a0)

SUMMARY: AddressSanitizer: heap-use-after-free C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\templates\hash_map.h:233 in HashMap<__int64, struct WorkerThreadPool::Group *, struct HashMapHasherDefault, struct HashMapComparatorDefault<__int64>, class DefaultTypedAllocator<struct HashMapElement<__int64, struct WorkerThreadPool::Group *>>>::_insert(__int64 const &, struct WorkerThreadPool::Group *const &, bool)
Shadow bytes around the buggy address:
  0x0411f26ec6d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0411f26ec6e0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0411f26ec6f0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0411f26ec700: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0411f26ec710: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
=>0x0411f26ec720: fa fa fd fd fd fd fd fd fa fa fd fd[fd]fd fd fd
  0x0411f26ec730: fa fa fd fd fd fd fd fd fa fa 00 00 00 00 00 00
  0x0411f26ec740: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0411f26ec750: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0411f26ec760: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0411f26ec770: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==21084==ABORTING

Both of these types of errors mention group tasks, and occasionally I also get printed to the console the following error:

ERROR: Invalid Group ID
   at: (core\object\worker_thread_pool.cpp:375)

Not sure why this only prints on the ASAN build but I'm sure there's a compile flag somewhere responsible.

I had enabled "Run on separate thread" for 3D physics, and disabling this stops ASAN from immediately killing the game on the main menu, but ASAN immediately kills the game as soon as it tries to compile fog material shaders:

=================================================================
==23600==ERROR: AddressSanitizer: global-buffer-overflow on address 0x7ff71b4a2d68 at pc 0x7ff71459402c bp 0x00a1403c5380 sp 0x00a1403c5388
READ of size 4 at 0x7ff71b4a2d68 thread T0
    #0 0x7ff71459402b in ShaderLanguage::get_scalar_type(enum ShaderLanguage::DataType) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\rendering\shader_language.cpp:4314
    #1 0x7ff7145b0afe in ShaderLanguage::_validate_function_call(struct ShaderLanguage::BlockNode *, struct ShaderLanguage::FunctionInfo const &, struct ShaderLanguage::OperatorNode *, enum ShaderLanguage::DataType *, class StringName *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\rendering\shader_language.cpp:3062
    #2 0x7ff7145bd672 in ShaderLanguage::_parse_expression(struct ShaderLanguage::BlockNode *, struct ShaderLanguage::FunctionInfo const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\rendering\shader_language.cpp:5154
    #3 0x7ff7145d0bf6 in ShaderLanguage::_parse_and_reduce_expression(struct ShaderLanguage::BlockNode *, struct ShaderLanguage::FunctionInfo const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\rendering\shader_language.cpp:6679
    #4 0x7ff7145df679 in ShaderLanguage::_parse_block(struct ShaderLanguage::BlockNode *, struct ShaderLanguage::FunctionInfo const &, bool, bool, bool) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\rendering\shader_language.cpp:7714
    #5 0x7ff7145fc0b0 in ShaderLanguage::_parse_shader(class HashMap<class StringName, struct ShaderLanguage::FunctionInfo, struct HashMapHasherDefault, struct HashMapComparatorDefault<class StringName>, class DefaultTypedAllocator<struct HashMapElement<class StringName, struct ShaderLanguage::FunctionInfo>>> const &, class Vector<struct ShaderLanguage::ModeInfo> const &, class HashSet<class String, struct HashMapHasherDefault, struct HashMapComparatorDefault<class String>> const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\rendering\shader_language.cpp:9547
    #6 0x7ff7145fe49c in ShaderLanguage::compile(class String const &, struct ShaderLanguage::ShaderCompileInfo const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\rendering\shader_language.cpp:9764
    #7 0x7ff7146790a9 in ShaderCompiler::compile(enum RenderingServer::ShaderMode, class String const &, struct ShaderCompiler::IdentifierActions *, class String const &, struct ShaderCompiler::GeneratedCode &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\rendering\shader_compiler.cpp:1363
    #8 0x7ff7147d3971 in RendererRD::Fog::FogShaderData::set_code(class String const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\rendering\renderer_rd\environment\fog.cpp:350
    #9 0x7ff7147b2342 in RendererRD::MaterialStorage::shader_set_code(class RID, class String const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\rendering\renderer_rd\storage_rd\material_storage.cpp:2425
    #10 0x7ff714310e2b in RenderingServerDefault::shader_set_code(class RID, class String const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\rendering\rendering_server_default.h:226
    #11 0x7ff71390ef75 in FogMaterial::_update_shader(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\resources\fog_material.cpp:145
    #12 0x7ff71391354f in FogMaterial::get_rid(void) const C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\resources\fog_material.cpp:103
    #13 0x7ff7133a682b in FogVolume::set_material(class Ref<class Material> const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\3d\fog_volume.cpp:85
    #14 0x7ff7133ab89d in call_with_variant_args_helper<class FogVolume, class Ref<class Material> const &, 0>(class FogVolume *, void (__cdecl FogVolume::*)(class Ref<class Material> const &), class Variant const **, struct Callable::CallError &, struct IndexSequence<0>) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\variant\binder_common.h:262
    #15 0x7ff7133aaf4a in call_with_variant_args_dv<class FogVolume, class Ref<class Material> const &>(class FogVolume *, void (__cdecl FogVolume::*)(class Ref<class Material> const &), class Variant const **, int, struct Callable::CallError &, class Vector<class Variant> const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\variant\binder_common.h:409
    #16 0x7ff7133b01ae in MethodBindT<class FogVolume, class Ref<class Material> const &>::call(class Object *, class Variant const **, int, struct Callable::CallError &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\object\method_bind.h:318
    #17 0x7ff71587825f in ClassDB::set_property(class Object *, class StringName const &, class Variant const &, bool *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\object\class_db.cpp:1071
    #18 0x7ff71565d2c5 in Object::set(class StringName const &, class Variant const &, bool *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\object\object.cpp:249
    #19 0x7ff7109668f6 in SceneState::instantiate(enum SceneState::GenEditState) const C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\resources\packed_scene.cpp:331
    #20 0x7ff710970593 in PackedScene::instantiate(enum PackedScene::GenEditState) const C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\resources\packed_scene.cpp:1749
    #21 0x7ff710964785 in SceneState::instantiate(enum SceneState::GenEditState) const C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\resources\packed_scene.cpp:168
    #22 0x7ff710970593 in PackedScene::instantiate(enum PackedScene::GenEditState) const C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\resources\packed_scene.cpp:1749
    #23 0x7ff710964785 in SceneState::instantiate(enum SceneState::GenEditState) const C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\resources\packed_scene.cpp:168
    #24 0x7ff710970593 in PackedScene::instantiate(enum PackedScene::GenEditState) const C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\resources\packed_scene.cpp:1749
    #25 0x7ff710964785 in SceneState::instantiate(enum SceneState::GenEditState) const C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\resources\packed_scene.cpp:168
    #26 0x7ff710970593 in PackedScene::instantiate(enum PackedScene::GenEditState) const C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\resources\packed_scene.cpp:1749
    #27 0x7ff710964785 in SceneState::instantiate(enum SceneState::GenEditState) const C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\resources\packed_scene.cpp:168
    #28 0x7ff710970593 in PackedScene::instantiate(enum PackedScene::GenEditState) const C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\resources\packed_scene.cpp:1749
    #29 0x7ff710980e7f in call_with_variant_args_retc_helper<class PackedScene, class Node *, enum PackedScene::GenEditState, 0>(class PackedScene *, class Node * (__cdecl PackedScene::*)(enum PackedScene::GenEditState) const, class Variant const **, class Variant &, struct Callable::CallError &, struct IndexSequence<0>) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\variant\binder_common.h:731
    #30 0x7ff71097b427 in call_with_variant_args_retc_dv<class PackedScene, class Node *, enum PackedScene::GenEditState>(class PackedScene *, class Node * (__cdecl PackedScene::*)(enum PackedScene::GenEditState) const, class Variant const **, int, class Variant &, struct Callable::CallError &, class Vector<class Variant> const &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\variant\binder_common.h:526
    #31 0x7ff71099f44d in MethodBindTRC<class PackedScene, class Node *, enum PackedScene::GenEditState>::call(class Object *, class Variant const **, int, struct Callable::CallError &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\object\method_bind.h:542
    #32 0x7ff70eafc75c in GDScriptFunction::call(class GDScriptInstance *, class Variant const **, int, struct Callable::CallError &, struct GDScriptFunction::CallState *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\modules\gdscript\gdscript_vm.cpp:1644
    #33 0x7ff70e9828ce in GDScriptInstance::callp(class StringName const &, class Variant const **, int, struct Callable::CallError &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\modules\gdscript\gdscript.cpp:1634
    #34 0x7ff715662a31 in Object::callp(class StringName const &, class Variant const **, int, struct Callable::CallError &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\object\object.cpp:711
    #35 0x7ff715f43a2b in Variant::callp(class StringName const &, class Variant const **, int, class Variant &, struct Callable::CallError &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\variant\variant_call.cpp:1048
    #36 0x7ff70eafb847 in GDScriptFunction::call(class GDScriptInstance *, class Variant const **, int, struct Callable::CallError &, struct GDScriptFunction::CallState *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\modules\gdscript\gdscript_vm.cpp:1555
    #37 0x7ff70e9828ce in GDScriptInstance::callp(class StringName const &, class Variant const **, int, struct Callable::CallError &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\modules\gdscript\gdscript.cpp:1634
    #38 0x7ff715662a31 in Object::callp(class StringName const &, class Variant const **, int, struct Callable::CallError &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\object\object.cpp:711
    #39 0x7ff715f43a2b in Variant::callp(class StringName const &, class Variant const **, int, class Variant &, struct Callable::CallError &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\variant\variant_call.cpp:1048
    #40 0x7ff70eafb847 in GDScriptFunction::call(class GDScriptInstance *, class Variant const **, int, struct Callable::CallError &, struct GDScriptFunction::CallState *) C:\Users\thePG\Documents\Godot\Entombed\build\godot\modules\gdscript\gdscript_vm.cpp:1555
    #41 0x7ff70e9828ce in GDScriptInstance::callp(class StringName const &, class Variant const **, int, struct Callable::CallError &) C:\Users\thePG\Documents\Godot\Entombed\build\godot\modules\gdscript\gdscript.cpp:1634
    #42 0x7ff71060d08a in Node::_gdvirtual__process_call<0>(double) C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\main\node.h:238
    #43 0x7ff7105db457 in Node::_notification(int) C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\main\node.cpp:56
    #44 0x7ff70e86582f in Node::_notificationv(int, bool) C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\main\node.h:45
    #45 0x7ff715663cdf in Object::notification(int, bool) C:\Users\thePG\Documents\Godot\Entombed\build\godot\core\object\object.cpp:790
    #46 0x7ff71055c26b in SceneTree::_notify_group_pause(class StringName const &, int) C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\main\scene_tree.cpp:868
    #47 0x7ff710569890 in SceneTree::process(double) C:\Users\thePG\Documents\Godot\Entombed\build\godot\scene\main\scene_tree.cpp:466
    #48 0x7ff70e6d466e in Main::iteration(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\main\main.cpp:3170
    #49 0x7ff70e5b0a47 in OS_Windows::run(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\os_windows.cpp:1001
    #50 0x7ff70e58bc4d in widechar_main(int, wchar_t **) C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:179
    #51 0x7ff70e58bf26 in _main(void) C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:203
    #52 0x7ff70e58bfa1 in main C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:217
    #53 0x7ff70e58ad9c in WinMain C:\Users\thePG\Documents\Godot\Entombed\build\godot\platform\windows\godot_windows.cpp:230
    #54 0x7ff717531d09 in invoke_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:102
    #55 0x7ff717531d09 in __scrt_common_main_seh D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #56 0x7ffa700c74b3  (C:\Windows\System32\KERNEL32.DLL+0x1800174b3)
    #57 0x7ffa709226a0  (C:\Windows\SYSTEM32\ntdll.dll+0x1800526a0)

0x7ff71b4a2d68 is located 8 bytes to the right of global variable '`ShaderLanguage::get_scalar_type'::`2'::scalar_types' defined in 'shader_language.cpp:4287:23' (0x7ff71b4a2d00) of size 96
SUMMARY: AddressSanitizer: global-buffer-overflow C:\Users\thePG\Documents\Godot\Entombed\build\godot\servers\rendering\shader_language.cpp:4314 in ShaderLanguage::get_scalar_type(enum ShaderLanguage::DataType)
Shadow bytes around the buggy address:
  0x11fbe0714550: 00 00 01 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x11fbe0714560: 00 00 00 01 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x11fbe0714570: 00 00 00 04 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x11fbe0714580: 00 00 06 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x11fbe0714590: 00 04 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
=>0x11fbe07145a0: 00 00 00 00 00 00 00 00 00 00 00 00 f9[f9]f9 f9
  0x11fbe07145b0: f9 f9 f9 f9 f9 f9 f9 f9 00 00 00 00 00 00 00 00
  0x11fbe07145c0: 00 00 00 00 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x11fbe07145d0: 00 00 00 00 00 00 00 00 01 f9 f9 f9 f9 f9 f9 f9
  0x11fbe07145e0: f9 f9 f9 f9 f9 f9 f9 f9 00 00 00 00 00 00 02 f9
  0x11fbe07145f0: f9 f9 f9 f9 f9 f9 f9 f9 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==23600==ABORTING

Removing the fog volumes from my game lets me play through the whole thing without being killed by ASAN, so I'm reasonably confident the problem lies in that "Run on separate thread" option.

I'll leave it disabled for a while, and see if I notice fewer crashes.

@Chaosus Chaosus added this to the 4.0 milestone Nov 13, 2022
@gelvinp
Copy link
Contributor Author

gelvinp commented Nov 13, 2022

I'll also open separate issues for the native icon and fog volumes, since those only crash the game when using ASAN and are unrelated.

@gelvinp
Copy link
Contributor Author

gelvinp commented Nov 13, 2022

I've just heard back from a friend who had a crash on their machine as well, so even more sure it isn't a hardware issue. I've managed to reproduce (sorta) the physics thread asan crash in a minimal project with only a Camera3D and the "Run on separate thread" issue checked. It reliably crashes within ~10 seconds, except this time the heap use-after-free has alternated between GodotStep2D and GodotStep3D, which I find weird because it's a 3D scene, and 2D physics isn't set to run on a separate thread.

Haven't seen a crash outside of asan in the minimal project either way, so I'm gonna start git bisecting my main project and see what happens.

EDIT: Just as I posted this comment, the instance running from the editor (i.e. no asan) crashed as well. Same thing, no error message, just suddenly died, with an access violation logged in EventViewer

If it's useful here's the minimal project:
PhysicsThread.zip

@dotnetdummy
Copy link

I have similar problems with a 2D game using beta 10. The crashes are seemingly random without any output anywhere (except in the Windows Event Viewer saying the app has crashed). I see the crashes on both my desktop PC with Win11, and on my Macbook Air M2.

The game is build using the C# version of Godot 4, and the crashes occur both in the Godot editor and when in Jetbrains Rider.

I'm gonna setup a different project to see if the crashes also occur there, but just as a heads up that it most likely isn't hardware related.

@the-sink
Copy link
Contributor

the-sink commented Feb 15, 2023

Just started experiencing this issue on my project after switching to beta17. Can't figure out what part of the project is causing this, but it works fine in beta16. It'll crash immediately after clicking on the project in the project list, editor window doesn't even have time to appear!

I might try and bisect this since I have a case where it's consistently happening in one version but not another.

Edit:
I bisected the crash and traced it back to #71989, but this commit was not actually the issue. I just apparently forgot to update one of my plugins, oops... I still wonder if it's feasible to try and avoid these hard crashes as much as possible as they give virtually no information.

@YuriSizov YuriSizov modified the milestones: 4.0, 4.1 Feb 27, 2023
@akien-mga akien-mga changed the title [Godot 4 Beta 4] Random crashes with no errors logged, related to memory access violations Random crashes with no errors logged, related to memory access violations Jun 23, 2023
@akien-mga
Copy link
Member

Can you still reproduce this in 4.0.3 and 4.1-beta3 or later?

@akien-mga akien-mga modified the milestones: 4.1, 4.2 Jun 23, 2023
@gelvinp
Copy link
Contributor Author

gelvinp commented Jun 23, 2023

I left both the MRP running on 4.0.3 (two instances, one ASAN one normal), and an instance of the project I was initially working on when I noticed this, with the 3d run in separate thread enabled, for an hour, and did not experience any crashes.

@akien-mga
Copy link
Member

Awesome, thanks! Closing as fixed.

@akien-mga akien-mga modified the milestones: 4.2, 4.1 Jun 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants