Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Infinite loop at shutdown #47748

Closed
hhugo opened this issue Apr 27, 2023 · 11 comments · Fixed by #51290
Closed

Infinite loop at shutdown #47748

hhugo opened this issue Apr 27, 2023 · 11 comments · Fixed by #51290

Comments

@hhugo
Copy link

hhugo commented Apr 27, 2023

Version

v16.18.1, v18.16.0 and v20.0.0

Platform

Linux 5.15.0-41-generic #44-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

I don't have an easy way to reproduce the issue.

How often does it reproduce? Is there a required condition?

It triggers once every 10 runs maybe, it triggers much more often if I set break breakpoints (e.g. node::PerIsolatePlatformData::FlushForegroundTasksInternal)

What is the expected behavior? Why is that the expected behavior?

I expect the process to terminate successfully.

What do you see instead?

The process hangs at 100% cpu after having done its normal job.

Attaching gdb gives me traces like

#0  v8::internal::Object::IsJSReceiver (cage_base=..., this=<optimized out>) at ../deps/v8/src/objects/tagged-impl.h:142
#1  v8::internal::LookupIterator::GetRoot (index=18446744073709551615, lookup_start_object=..., isolate=0x5571659076c0) at ../deps/v8/src/objects/lookup-inl.h:285
#2  v8::internal::LookupIterator::Start<false> (this=0x7ffedf9a85c0) at ../deps/v8/src/objects/lookup.cc:65
#3  0x000055715f0d5c83 in v8::internal::LookupIterator::LookupIterator (configuration=v8::internal::LookupIterator::PROTOTYPE_CHAIN, lookup_start_object=..., key=..., receiver=..., isolate=0x5571659076c0, this=0x7ffedf9a85c0) at ../deps/v8/src/objects/lookup.h:34
#4  v8::internal::Runtime::GetObjectProperty (isolate=isolate@entry=0x5571659076c0, lookup_start_object=lookup_start_object@entry=..., key=key@entry=..., receiver=..., receiver@entry=..., is_found=is_found@entry=0x0) at ../deps/v8/src/runtime/runtime-object.cc:40
#5  0x000055715e54420d in v8::Object::Get (this=<optimized out>, context=..., key=...) at ../deps/v8/src/handles/handles.h:137
#6  0x000055715e1e87bd in node::errors::TriggerUncaughtException (isolate=0x5571659076c0, error=..., message=..., from_promise=false) at ../src/node_errors.cc:1138
#7  0x000055715e1e69b6 in node::errors::PerIsolateMessageListener (message=..., error=...) at ../src/node_errors.cc:962
#8  0x000055715e89ef4d in v8::internal::MessageHandler::ReportMessageNoExceptions (isolate=0x5571659076c0, loc=<optimized out>, message=..., api_exception_obj=...) at ../deps/v8/src/execution/messages.cc:192
#9  0x000055715e89f67b in v8::internal::MessageHandler::ReportMessage (isolate=0x5571659076c0, loc=0x7ffedf9a9060, message=...) at ../deps/v8/src/handles/handles.h:137
#10 0x000055715e88287b in v8::internal::Isolate::ReportPendingMessages (this=this@entry=0x5571659076c0) at ../deps/v8/src/execution/isolate.cc:2695
#11 0x000055715e84df88 in v8::internal::(anonymous namespace)::Invoke (isolate=isolate@entry=0x5571659076c0, params=...) at ../deps/v8/src/execution/execution.cc:372
#12 0x000055715e84f0e1 in v8::internal::Execution::CallBuiltin (isolate=isolate@entry=0x5571659076c0, builtin=..., receiver=receiver@entry=..., argc=argc@entry=1, argv=argv@entry=0x7ffedf9a9270) at ../deps/v8/src/execution/execution.cc:551
#13 0x000055715e54eba9 in v8::internal::InvokeFinalizationRegistryCleanupFromTask (context=..., context@entry=..., finalization_registry=finalization_registry@entry=..., callback=...) at ../deps/v8/src/api/api.cc:11369
#14 0x000055715e994cd0 in v8::internal::FinalizationRegistryCleanupTask::RunInternal (this=0x557167724dc0) at ../deps/v8/src/heap/finalization-registry-cleanup-task.cc:88
#15 0x000055715e2bcb4c in node::PerIsolatePlatformData::RunForegroundTask (this=0x5571658f7200, task=std::unique_ptr<v8::Task> = {...}) at ../src/node_platform.cc:430
#16 0x000055715e2bd179 in node::PerIsolatePlatformData::FlushForegroundTasksInternal (this=0x5571658f7200) at ../src/node_platform.cc:494
#17 0x000055715e2bcde7 in node::NodePlatform::DrainTasks (this=0x5571659ae6f0, isolate=0x5571659076c0) at ../src/node_platform.cc:457
#18 0x000055715e03f75f in node::FreeEnvironment (env=0x55716599dcd0) at ../src/api/environment.cc:514
#19 0x000055715e039e84 in node::FunctionDeleter<node::Environment, &node::FreeEnvironment>::operator() (this=0x7ffedf9a9650, pointer=0x55716599dcd0) at ../src/util.h:682
#20 0x000055715e038d36 in std::unique_ptr<node::Environment, node::FunctionDeleter<node::Environment, &node::FreeEnvironment> >::~unique_ptr (this=0x7ffedf9a9650, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/unique_ptr.h:361
#21 0x000055715e2668c6 in node::NodeMainInstance::Run (this=0x7ffedf9a96e0) at ../src/node_main_instance.cc:86
#22 0x000055715e166f2a in node::LoadSnapshotDataAndRun (snapshot_data_ptr=0x7ffedf9a97c0, result=0x55716584a220) at ../src/node.cc:1208
#23 0x000055715e167312 in node::StartInternal (argc=1, argv=0x5571659e83d0) at ../src/node.cc:1259
#24 0x000055715e1673ca in node::Start (argc=1, argv=0x7ffedf9a99c8) at ../src/node.cc:1266
#25 0x000055715fc01b64 in main (argc=1, argv=0x7ffedf9a99c8) at ../src/node_main.cc:97

and

#0  v8::internal::ReadOnlyRoots::GetLocation (this=this@entry=0x7ffedf9a9260, root_index=root_index@entry=v8::internal::RootIndex::kUndefinedValue) at ../deps/v8/src/roots/roots-inl.h:102
#1  0x000055715f0702e2 in v8::internal::ReadOnlyRoots::address_at (root_index=v8::internal::RootIndex::kUndefinedValue, this=0x7ffedf9a9260) at ../deps/v8/src/roots/roots-inl.h:144
#2  v8::internal::ReadOnlyRoots::object_at (root_index=v8::internal::RootIndex::kUndefinedValue, this=0x7ffedf9a9260) at ../deps/v8/src/roots/roots-inl.h:135
#3  v8::internal::ReadOnlyRoots::unchecked_undefined_value (this=0x7ffedf9a9260) at ../deps/v8/src/roots/roots-inl.h:96
#4  v8::internal::ReadOnlyRoots::CheckType_undefined_value (this=this@entry=0x7ffedf9a9260) at ../deps/v8/src/roots/roots.cc:57
#5  0x000055715e9b5963 in v8::internal::ReadOnlyRoots::undefined_value (this=0x7ffedf9a9260) at ../deps/v8/src/roots/roots-inl.h:96
#6  v8::internal::Object::IsUndefined (roots=..., this=<optimized out>) at ../deps/v8/src/objects/objects-inl.h:140
#7  v8::internal::Object::IsUndefined (isolate=0x5571659076c0, this=<optimized out>) at ../deps/v8/src/objects/objects-inl.h:124
#8  v8::internal::Heap::HasDirtyJSFinalizationRegistries (this=0x557165914978) at ../deps/v8/src/heap/heap-inl.h:522
#9  v8::internal::Heap::PostFinalizationRegistryCleanupTaskIfNeeded (this=0x557165914978) at ../deps/v8/src/heap/heap.cc:6552
#10 0x000055715e994d59 in v8::internal::FinalizationRegistryCleanupTask::RunInternal (this=0x5571676ed5e0) at ../deps/v8/src/heap/finalization-registry-cleanup-task.cc:98
#11 0x000055715e2bcb4c in node::PerIsolatePlatformData::RunForegroundTask (this=0x5571658f7200, task=std::unique_ptr<v8::Task> = {...}) at ../src/node_platform.cc:430
#12 0x000055715e2bd179 in node::PerIsolatePlatformData::FlushForegroundTasksInternal (this=0x5571658f7200) at ../src/node_platform.cc:494
#13 0x000055715e2bcde7 in node::NodePlatform::DrainTasks (this=0x5571659ae6f0, isolate=0x5571659076c0) at ../src/node_platform.cc:457
#14 0x000055715e03f75f in node::FreeEnvironment (env=0x55716599dcd0) at ../src/api/environment.cc:514
#15 0x000055715e039e84 in node::FunctionDeleter<node::Environment, &node::FreeEnvironment>::operator() (this=0x7ffedf9a9650, pointer=0x55716599dcd0) at ../src/util.h:682
#16 0x000055715e038d36 in std::unique_ptr<node::Environment, node::FunctionDeleter<node::Environment, &node::FreeEnvironment> >::~unique_ptr (this=0x7ffedf9a9650, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/unique_ptr.h:361
#17 0x000055715e2668c6 in node::NodeMainInstance::Run (this=0x7ffedf9a96e0) at ../src/node_main_instance.cc:86
#18 0x000055715e166f2a in node::LoadSnapshotDataAndRun (snapshot_data_ptr=0x7ffedf9a97c0, result=0x55716584a220) at ../src/node.cc:1208
#19 0x000055715e167312 in node::StartInternal (argc=1, argv=0x5571659e83d0) at ../src/node.cc:1259
#20 0x000055715e1673ca in node::Start (argc=1, argv=0x7ffedf9a99c8) at ../src/node.cc:1266
#21 0x000055715fc01b64 in main (argc=1, argv=0x7ffedf9a99c8) at ../src/node_main.cc:97

The process never return from node::NodePlatform::DrainTasks

Additional information

Based on some name seen in the backtrace, the following information might be useful.

My process use FinalizationRegistry with a finalization function calling into wasm. Concretely, I've a mapping between some js value and some wasm allocations, and have finalisers on some js value that free the memory on the wasm side (calling free).

I'm currently unable to reproduce the issue when removing the call to (wasm) free in the finaliser.

@hhugo
Copy link
Author

hhugo commented Apr 27, 2023

I'm currently unable to reproduce the issue when removing the call to (wasm) free in the finaliser.

I've just seen the issue without the call to wasm.

Here is another backtrace.

#0  v8::internal::Isolate::ThrowIllegalOperation (this=this@entry=0x5618e49e46c0) at ../deps/v8/src/execution/isolate.cc:2389
#1  0x00005618de44d8b4 in v8::internal::(anonymous namespace)::Invoke (isolate=isolate@entry=0x5618e49e46c0, params=...) at ../deps/v8/src/execution/execution.cc:370
#2  0x00005618de44f0e1 in v8::internal::Execution::CallBuiltin (isolate=isolate@entry=0x5618e49e46c0, builtin=..., receiver=receiver@entry=..., argc=argc@entry=1, argv=argv@entry=0x7ffd4e2e9c70) at ../deps/v8/src/execution/execution.cc:551
#3  0x00005618de14eba9 in v8::internal::InvokeFinalizationRegistryCleanupFromTask (context=..., context@entry=..., finalization_registry=finalization_registry@entry=..., callback=...) at ../deps/v8/src/api/api.cc:11369
#4  0x00005618de594cd0 in v8::internal::FinalizationRegistryCleanupTask::RunInternal (this=0x5618e518dda0) at ../deps/v8/src/heap/finalization-registry-cleanup-task.cc:88
#5  0x00005618ddebcb4c in node::PerIsolatePlatformData::RunForegroundTask (this=0x5618e49d4200, task=std::unique_ptr<v8::Task> = {...}) at ../src/node_platform.cc:430
#6  0x00005618ddebd179 in node::PerIsolatePlatformData::FlushForegroundTasksInternal (this=0x5618e49d4200) at ../src/node_platform.cc:494
#7  0x00005618ddebcde7 in node::NodePlatform::DrainTasks (this=0x5618e4a8b6f0, isolate=0x5618e49e46c0) at ../src/node_platform.cc:457
#8  0x00005618ddc3f75f in node::FreeEnvironment (env=0x5618e4a7acd0) at ../src/api/environment.cc:514
#9  0x00005618ddc39e84 in node::FunctionDeleter<node::Environment, &node::FreeEnvironment>::operator() (this=0x7ffd4e2ea050, pointer=0x5618e4a7acd0) at ../src/util.h:682
#10 0x00005618ddc38d36 in std::unique_ptr<node::Environment, node::FunctionDeleter<node::Environment, &node::FreeEnvironment> >::~unique_ptr (this=0x7ffd4e2ea050, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/unique_ptr.h:361
#11 0x00005618dde668c6 in node::NodeMainInstance::Run (this=0x7ffd4e2ea0e0) at ../src/node_main_instance.cc:86
#12 0x00005618ddd66f2a in node::LoadSnapshotDataAndRun (snapshot_data_ptr=0x7ffd4e2ea1c0, result=0x5618e4927220) at ../src/node.cc:1208
#13 0x00005618ddd67312 in node::StartInternal (argc=1, argv=0x5618e4ac53d0) at ../src/node.cc:1259
#14 0x00005618ddd673ca in node::Start (argc=1, argv=0x7ffd4e2ea3c8) at ../src/node.cc:1266
#15 0x00005618df801b64 in main (argc=1, argv=0x7ffd4e2ea3c8) at ../src/node_main.cc:97

I don't know if ThrowIllegalOperation is something benign

@hhugo
Copy link
Author

hhugo commented Apr 27, 2023

And another trace mentioning wasm but I don't know if it is relevant

#1  0x00005641a54dd93f in v8::base::MemoryProtectionKey::GetKeyPermission (key=1) at ../deps/v8/src/base/platform/memory-protection-key.cc:221
#2  0x00005641a3976a6c in v8::internal::RwxMemoryWriteScope::IsPKUWritable () at ../deps/v8/src/common/code-memory-access.cc:40
#3  0x00005641a469fc49 in v8::internal::wasm::WasmCodeManager::MemoryProtectionKeyWritable () at ../deps/v8/src/wasm/wasm-code-manager.cc:2079
#4  0x00005641a3a4e242 in v8::internal::(anonymous namespace)::Invoke (isolate=isolate@entry=0x5641aaa6e6c0, params=...) at ../deps/v8/src/execution/execution.cc:287
#5  0x00005641a3a4f0e1 in v8::internal::Execution::CallBuiltin (isolate=isolate@entry=0x5641aaa6e6c0, builtin=..., receiver=receiver@entry=..., argc=argc@entry=1, argv=argv@entry=0x7ffdbb7ba9a0) at ../deps/v8/src/execution/execution.cc:551
#6  0x00005641a374eba9 in v8::internal::InvokeFinalizationRegistryCleanupFromTask (context=..., context@entry=..., finalization_registry=finalization_registry@entry=..., callback=...) at ../deps/v8/src/api/api.cc:11369
#7  0x00005641a3b94cd0 in v8::internal::FinalizationRegistryCleanupTask::RunInternal (this=0x5641aca16170) at ../deps/v8/src/heap/finalization-registry-cleanup-task.cc:88
#8  0x00005641a34bcb4c in node::PerIsolatePlatformData::RunForegroundTask (this=0x5641aaa5e280, task=std::unique_ptr<v8::Task> = {...}) at ../src/node_platform.cc:430
#9  0x00005641a34bd179 in node::PerIsolatePlatformData::FlushForegroundTasksInternal (this=0x5641aaa5e280) at ../src/node_platform.cc:494
#10 0x00005641a34bcde7 in node::NodePlatform::DrainTasks (this=0x5641aaa506d0, isolate=0x5641aaa6e6c0) at ../src/node_platform.cc:457
#11 0x00005641a323f75f in node::FreeEnvironment (env=0x5641aaa6a960) at ../src/api/environment.cc:514
#12 0x00005641a3239e84 in node::FunctionDeleter<node::Environment, &node::FreeEnvironment>::operator() (this=0x7ffdbb7bad80, pointer=0x5641aaa6a960) at ../src/util.h:682
#13 0x00005641a3238d36 in std::unique_ptr<node::Environment, node::FunctionDeleter<node::Environment, &node::FreeEnvironment> >::~unique_ptr (this=0x7ffdbb7bad80, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/unique_ptr.h:361
#14 0x00005641a34668c6 in node::NodeMainInstance::Run (this=0x7ffdbb7bae10) at ../src/node_main_instance.cc:86
#15 0x00005641a3366f2a in node::LoadSnapshotDataAndRun (snapshot_data_ptr=0x7ffdbb7baef0, result=0x5641aa9b1220) at ../src/node.cc:1208
#16 0x00005641a3367312 in node::StartInternal (argc=2, argv=0x5641aaa52a90) at ../src/node.cc:1259
#17 0x00005641a33673ca in node::Start (argc=2, argv=0x7ffdbb7bb0f8) at ../src/node.cc:1266
#18 0x00005641a4e01b64 in main (argc=2, argv=0x7ffdbb7bb0f8) at ../src/node_main.cc:97

@hhugo
Copy link
Author

hhugo commented Apr 28, 2023

@targos
Copy link
Member

targos commented Apr 28, 2023

Possibly related to #47452

@kenyhenry
Copy link

can you push the code to try reproduce this issue ?

@hhugo
Copy link
Author

hhugo commented Nov 21, 2023

I don't have a minimal repro.
Here is some data obtained using perf, no sure it's useful.

  14,74%  node     node                 [.] v8::internal::DebuggableStackFrameIterator::DebuggableStackFrameIterator                                                                                       ▒
   7,54%  node     libc.so.6            [.] pthread_mutex_lock@@GLIBC_2.2.5                                                                                                                                ▒
   7,40%  node     node                 [.] v8::internal::StackFrameIterator::StackFrameIterator                                                                                                           ▒
   6,66%  node     libc.so.6            [.] pthread_mutex_unlock@@GLIBC_2.2.5                                                                                                                              ▒
   4,05%  node     node                 [.] node::NodePlatform::GetForegroundTaskRunner                                                                                                                    ▒
   2,49%  node     node                 [.] v8::internal::InvokeFinalizationRegistryCleanupFromTask                                                                                                        ▒
   2,35%  node     node                 [.] node::errors::TriggerUncaughtException                                                                                                                         ▒
   2,29%  node     libc.so.6            [.] malloc                                                                                                                                                         ▒
   2,16%  node     node                 [.] v8::internal::MessageHandler::ReportMessageNoExceptions                                                                                                        ▒
   2,09%  node     libc.so.6            [.] _int_free                                                                                                                                                      ▒
   1,68%  node     node                 [.] v8::internal::Heap::PostFinalizationRegistryCleanupTaskIfNeeded                                                                                                ▒
   1,66%  node     node                 [.] v8::internal::FinalizationRegistryCleanupTask::RunInternal                                                                                                     ▒
   1,56%  node     node                 [.] v8::internal::Factory::NewJSMessageObject                                                                                                                      ▒
   1,51%  node     node                 [.] v8::internal::Isolate::ReportPendingMessages                                                                                                                   ▒
   1,49%  node     node                 [.] node::PerIsolatePlatformData::FlushForegroundTasksInternal                                                                                                     ▒
   1,41%  node     node                 [.] v8::internal::Factory::New                                                                                                                                     ▒
   1,40%  node     node                 [.] v8::internal::(anonymous namespace)::Invoke                                                                                                                    ◆
   1,31%  node     node                 [.] v8::internal::Isolate::ThrowIllegalOperation                                                                                                                   ▒
   1,27%  node     node                 [.] v8::internal::MessageHandler::ReportMessage                                                                                                                    ▒
   1,20%  node     node                 [.] v8::internal::Isolate::ComputeLocation                                                                                                                         ▒
   1,18%  node     node                 [.] v8::Object::Get                                                                                                                                                ▒
   1,17%  node     node                 [.] v8::internal::Runtime::GetObjectProperty                                                                                                                       ▒
   1,17%  node     libc.so.6            [.] cfree@GLIBC_2.2.5                                                                                                                                              ▒
   1,15%  node     node                 [.] v8::internal::CancelableTask::CancelableTask                                                                                                                   ▒
   1,07%  node     node                 [.] v8::internal::Cancelable::~Cancelable                                                                                                                          ▒
   1,04%  node     node                 [.] v8::internal::Isolate::CreateMessageOrAbort                                                                                                                    ▒
   0,86%  node     node                 [.] v8::internal::Isolate::OptionalRescheduleException                                                                                                             ▒
   0,84%  node     node                 [.] std::_Deque_base<std::unique_ptr<v8::Task, std::default_delete<v8::Task> >, std::allocator<std::unique_ptr<v8::Task, std::default_delete<v8::Task> > > >::_M_in▒
   0,83%  node     node                 [.] v8::internal::LookupIterator::LookupInRegularHolder<false>                                                                                                     ▒
   0,76%  node     node                 [.] v8::CallDepthScope<true>::CallDepthScope                                                                                                                       ▒
   0,75%  node     node                 [.] non-virtual thunk to v8::internal::CancelableTask::Run()                                                                                                       ▒
   0,74%  node     node                 [.] v8::internal::LookupIterator::FetchValue                                                                                                                       ▒
   0,71%  node     node                 [.] v8::CallDepthScope<false>::CallDepthScope                                                                                                                      ▒
   0,71%  node     node                 [.] v8::internal::LookupIterator::Start<false>                                                                                                                     ▒
   0,68%  node     node                 [.] v8::EscapableHandleScope::EscapableHandleScope                                                                                                                 ▒
   0,63%  node     node                 [.] v8::TryCatch::TryCatch                                                                                                                                         ▒
   0,61%  node     libstdc++.so.6.0.30  [.] operator new                                                                                                                                                   ▒
   0,61%  node     node                 [.] uv_mutex_lock                                                                                                                                                  ▒
   0,59%  node     node                 [.] v8::internal::Isolate::ComputeLocationFromSimpleStackTrace                                                                                                     ▒
   0,58%  node     node                 [.] v8::TryCatch::~TryCatch                                                                                                                                        ▒
   0,58%  node     node                 [.] v8::internal::PropertyKey::PropertyKey                                                                                                                         ▒
   0,56%  node     node                 [.] uv_mutex_unlock                                                                                                                                                ▒
   0,56%  node     libc.so.6            [.] pthread_cond_signal@@GLIBC_2.3.2                                                                                                                               ▒
   0,56%  node     node                 [.] std::__detail::_Map_base<v8::Isolate*, std::pair<v8::Isolate* const, std::pair<node::IsolatePlatformDelegate*, std::shared_ptr<node::PerIsolatePlatformData> > ▒
   0,56%  node     node                 [.] v8::internal::Isolate::FireCallCompletedCallbackInternal                                                                                                       ▒
   0,54%  node     node                 [.] v8::internal::Execution::CallBuiltin

@hhugo
Copy link
Author

hhugo commented Nov 26, 2023

possibly related to https://bugs.chromium.org/p/v8/issues/detail?id=14281

@paulrutter
Copy link

paulrutter commented Dec 7, 2023

A colleague of mine analyzed one of these hanging processes with gdb. His analysis might be useful.
From debugging with gdb he gathered the following information:

When modifying the return values of ThrowOnJavascriptExecution::IsAllowed(isolate) and DumpOnJavascriptExecution::IsAllowed(isolate) using gdb to return true, the process exits properly.

We haven't been able to pinpoint exactly which piece of code is the culprit, as it happens in different node contexts. Hope this helps.
For us, this behavior started since we moved to node18 recently.

@bcoe
Copy link
Contributor

bcoe commented Dec 18, 2023

@legendecas I saw your work in #45907, which adds env->set_can_call_into_js(false);, prior to env->RunCleanup();. I'm wondering if this might be the culprit, since there are a few places in the codebase where we have a callback registered with the FinalizationRegistry, and it seems like this potentially triggers in some cases after we've starting throwing on JavaScript execution?

@mcollina
Copy link
Member

I think this is something we want to fix. Essentially if there is anything left in the FinalizationRegistry to clean up, we should be able to call into JS to do so.

The alternative is that we drain all FGs before shutting down.

@legendecas
Copy link
Member

env->set_can_call_into_js(false); doesn't affect V8 internals, rather, it is the Isolate::DisallowJavascriptExecutionScope that actually caused the problem.

#47748 (comment) is an excellent summary of the problem. When node is going to free an Environment and disallows JavaScript execution, V8 still tries to re-enqueue a FinalizationRegistryCleanupTask in a FinalizationRegistryCleanupTask because the FinalizationRegistry was not marked as not needed for cleanup when scoped in a Isolate::DisallowJavascriptExecutionScope.

A script to reliably reproduce the problem:

// Flags: --expose-gc
const reg = new FinalizationRegistry(() => {
  console.log('This FR callback should never be called');
});

function register() {
  // Create a temporary object in a new function scope to allow it to be GC-ed.
  reg.register({});
}

process.on('exit', () => {
  // This is the final chance to execute JavaScript.
  console.log('exit');
  register();
  // Queue a FinalizationRegistryCleanupTask by a testing gc request.
  gc();
});

I don't think this is a problem related to D8's https://bugs.chromium.org/p/v8/issues/detail?id=14281 because D8 didn't disallow JavaScript execution at its quit call. D8's issue is a dead lock issue on disposing an Isolate inside a platform task because it has to ensure that concurrent tasks of that Isolate are finished at disposal.

I think this is something we want to fix. Essentially if there is anything left in the FinalizationRegistry to clean up, we should be able to call into JS to do so.

I believe the callback of a FinalizationRegistry is not guaranteed to be called, and https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/FinalizationRegistry#notes_on_cleanup_callbacks explains it well. Particularly, when a worker_thread is requested to shut down from other threads, it should try to avoid calling into JavaScript again.

I think a solution could be to avoid enqueueing a FinalizationRegistryCleanupTask again when it fails to drain the pending weak cells in V8. I'll sum up a patch to fix the infinite loop.

nodejs-github-bot pushed a commit that referenced this issue Jan 8, 2024
At the point of `FreeEnvironment` and onwards, no JavaScript execution
associated with the Environment should be triggered.

Avoid draining platform tasks that can trigger JavaScript execution in
`FreeEnvironment`. The holder of `node::Environment` should immediately
call `node::MultiIsolatePlatform::UnregisterIsolate` and
`v8::Isolate::Dispose` to cancel pending foreground tasks and join
concurrent tasks after the environment was freed.

`NodePlatform` can properly handle the case in `RunForegroundTask` when
an Isolate out-lives its associated `node::Environment`.

PR-URL: #51290
Fixes: #47748
Fixes: #49344
Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
RafaelGSS pushed a commit that referenced this issue Jan 11, 2024
At the point of `FreeEnvironment` and onwards, no JavaScript execution
associated with the Environment should be triggered.

Avoid draining platform tasks that can trigger JavaScript execution in
`FreeEnvironment`. The holder of `node::Environment` should immediately
call `node::MultiIsolatePlatform::UnregisterIsolate` and
`v8::Isolate::Dispose` to cancel pending foreground tasks and join
concurrent tasks after the environment was freed.

`NodePlatform` can properly handle the case in `RunForegroundTask` when
an Isolate out-lives its associated `node::Environment`.

PR-URL: #51290
Fixes: #47748
Fixes: #49344
Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
marco-ippolito pushed a commit to marco-ippolito/node that referenced this issue Jan 12, 2024
At the point of `FreeEnvironment` and onwards, no JavaScript execution
associated with the Environment should be triggered.

Avoid draining platform tasks that can trigger JavaScript execution in
`FreeEnvironment`. The holder of `node::Environment` should immediately
call `node::MultiIsolatePlatform::UnregisterIsolate` and
`v8::Isolate::Dispose` to cancel pending foreground tasks and join
concurrent tasks after the environment was freed.

`NodePlatform` can properly handle the case in `RunForegroundTask` when
an Isolate out-lives its associated `node::Environment`.

PR-URL: nodejs#51290
Fixes: nodejs#47748
Fixes: nodejs#49344
Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Medhansh404 pushed a commit to Medhansh404/node that referenced this issue Jan 19, 2024
At the point of `FreeEnvironment` and onwards, no JavaScript execution
associated with the Environment should be triggered.

Avoid draining platform tasks that can trigger JavaScript execution in
`FreeEnvironment`. The holder of `node::Environment` should immediately
call `node::MultiIsolatePlatform::UnregisterIsolate` and
`v8::Isolate::Dispose` to cancel pending foreground tasks and join
concurrent tasks after the environment was freed.

`NodePlatform` can properly handle the case in `RunForegroundTask` when
an Isolate out-lives its associated `node::Environment`.

PR-URL: nodejs#51290
Fixes: nodejs#47748
Fixes: nodejs#49344
Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
richardlau pushed a commit that referenced this issue Mar 25, 2024
At the point of `FreeEnvironment` and onwards, no JavaScript execution
associated with the Environment should be triggered.

Avoid draining platform tasks that can trigger JavaScript execution in
`FreeEnvironment`. The holder of `node::Environment` should immediately
call `node::MultiIsolatePlatform::UnregisterIsolate` and
`v8::Isolate::Dispose` to cancel pending foreground tasks and join
concurrent tasks after the environment was freed.

`NodePlatform` can properly handle the case in `RunForegroundTask` when
an Isolate out-lives its associated `node::Environment`.

PR-URL: #51290
Fixes: #47748
Fixes: #49344
Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
richardlau pushed a commit that referenced this issue Mar 25, 2024
At the point of `FreeEnvironment` and onwards, no JavaScript execution
associated with the Environment should be triggered.

Avoid draining platform tasks that can trigger JavaScript execution in
`FreeEnvironment`. The holder of `node::Environment` should immediately
call `node::MultiIsolatePlatform::UnregisterIsolate` and
`v8::Isolate::Dispose` to cancel pending foreground tasks and join
concurrent tasks after the environment was freed.

`NodePlatform` can properly handle the case in `RunForegroundTask` when
an Isolate out-lives its associated `node::Environment`.

PR-URL: #51290
Fixes: #47748
Fixes: #49344
Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
richardlau pushed a commit that referenced this issue May 16, 2024
At the point of `FreeEnvironment` and onwards, no JavaScript execution
associated with the Environment should be triggered.

Avoid draining platform tasks that can trigger JavaScript execution in
`FreeEnvironment`. The holder of `node::Environment` should immediately
call `node::MultiIsolatePlatform::UnregisterIsolate` and
`v8::Isolate::Dispose` to cancel pending foreground tasks and join
concurrent tasks after the environment was freed.

`NodePlatform` can properly handle the case in `RunForegroundTask` when
an Isolate out-lives its associated `node::Environment`.

PR-URL: #51290
Fixes: #47748
Fixes: #49344
Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants