Can the CoreCLR library be unloaded from the memory process?

[raffaeler]

I host the CoreCLR by myself.
Since I wrapped the "LoadLibrary / FreeLibrary" "dlopen / dlclose" in a C++ class using the RAII paradigm, at the very end my app tries to unload the CLR and crashes.
This happens with both Windows specific API (mscoree) and with platform-neutral API (coreclr_initialize & Co).

If I comment out the coreclr unloading, everything works fine and the process closes normally.
What is the rationale behind the CoreCLR library? Can it be unloaded from the process or should I avoid unloading it?

[jkotas]

CoreCLR.so/dll does not support unloading.

[raffaeler]

This makes sense, but coreruncommon.cpp tries to unload it:
dlclose@coreruncommon

Is that a bug?

[jkotas]

Yes, the dlclose call should be removed.

[raffaeler]

Is at least unloading the (one and only) AppDomain legal?

[jkotas]

The "unload" of the one and only AppDomain will just deliver the callbacks like AppDomain.ProcessExit. It won't actually kill the appdomain.

[raffaeler]

So I understand there is no memory gain in calling UnloadAppDomain or coreclr_shutdown, right?

By the way this is the behavior I see:

• the process crashing when calling UnloadAppDomain
• the call coreclr_shutdown working as expected
• the entry point for coreclr_shutdown_2 not even being found

Thank you again

[jkotas]

no memory gain in calling UnloadAppDomain or coreclr_shutdown, right?

Right.

the call coreclr_shutdown working as expected

coreclr_shutdown calls UnloadAppDomain. It is surprising that coreclr_shutdown works for you, but UnloadAppDomain does not.

the entry point for coreclr_shutdown_2 not even being found

This entry point was added for .NET Core 2.0. Are you using recent .NET Core 2.0 builds?

[raffaeler]

coreclr_shutdown calls UnloadAppDomain. It is surprising that coreclr_shutdown works for you, but UnloadAppDomain does not.

It surprised me too, but substantially I have created two projects calling either the Windows specific code or the portable one. The layout of the calls are substantially the same and I am not mixing the APIs of course.
The project running coreclr_shutdown works on windows, ubuntu x64 and arm.

This entry point was added for .NET Core 2.0. Are you using recent .NET Core 2.0 builds?

I am using the official 2.0 preview, therefore it is definitely possible the coreclr_shutdown_2 was added later.

[mazong1123]

@raffaeler Are you able to find coreclr_shutdown_2 in latest build? It should be shipped with .NET Core 2.0 preview.

[DemiMarie]

@jkotas May I ask why unloading is not supported? Is it because there is no way to shutdown background threads cleanly?

[jkotas]

Yes, cleaning up everything reliably is hard: shutdown all thread, free all memory, unregister all callbacks, ... .

[RussKeldorph]

Not sure which area this belongs to. Feel free to change.

[TheLastRar]

I'm writing a plugin which uses coreclr to call into a managed assembly.

I'm having an issue when the program that uses my plugin tries to exit it hangs, even when I call coreclr_shutdown in my plugin and I believe this issue is related.

Is there a workaround I could use?
I suspect (but can't confirm) there is a stray thread left running which stops the program from closing fully.

Edit: This is on a self-built x86 Linux debug build.

[jkotas]

I'm having an issue when the program that uses my plugin tries to exit it hangs

Could you please attach debugger and find our where does it hang?

[TheLastRar]

	__kernel_vsyscall		No symbols loaded.
 	libpthread.so.0!__lll_lock_wait		No symbols loaded.
 	libpthread.so.0!pthread_mutex_lock		No symbols loaded.
 	Threading::Mutex::Acquire(Threading::Mutex * const this, Threading::Mutex * const this@entry) Line 149	C++	Symbols loaded.
 	Threading::Mutex::Wait(Threading::Mutex * const this) Line 207	C++	Symbols loaded.
 	LinuxPipeThread::~LinuxPipeThread(LinuxPipeThread * const this) Line 85	C++	Symbols loaded.
(call stack continues until main())

the effected line is;
pthread_mutex_lock( &m_mutex );

m_mutex is is a pthread_mutex_t

[jkotas]

Is LinuxPipeThread code that you have written? I do not think we have anything like that in CoreCLR.

[TheLastRar]

It's code contained by the program that is using my plugin (it's not my code nor is it from CoreCLR).

The code in question is here https://github.com/PCSX2/pcsx2/blob/1.4.x/pcsx2/Linux/LnxConsolePipe.cpp

Edit: I've also not posted the full call stack, I didn't think anything beyond what I included was usefull

[jkotas]

It does not look like a problem with CoreCLR or related to this issue from what you have shared so far. You need to debug the LinuxPipeThread code and figure our what it is waiting for.

[TheLastRar]

You have to excuse me, I'm not particularly well versed in debugging multithreaded programs.

The thread that is blocking is a stderr redirect thread (PCSX2 replaces stdout/stderr with pipes)
These threads should exit when the std redirect pipes are closed.
The hang is presumable caused by a leftover handle on stderr/stdout

Figuring this out forced me to learn how to better debug multi-threaded applications better.
Because of that, I discovered that there are 4 other threads, which belong to CoreCLR.

Call stacks below at the time of the hang;

	__kernel_vsyscall		No symbols loaded.
 	libpthread.so.0!open		No symbols loaded.
 	libcoreclr.so!TwoWayPipe::WaitForConnection(TwoWayPipe * this) Line 90	C++	Symbols loaded.
 	libcoreclr.so!DbgTransportSession::TransportWorker(DbgTransportSession * this) Line 1320	C++	Symbols loaded.
 	libcoreclr.so!DbgTransportSession::TransportWorkerStatic(LPVOID pvContext) Line 1236	C++	Symbols loaded.
 	libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void * pvParam) Line 1684	C++	Symbols loaded.
 	libpthread.so.0!start_thread		No symbols loaded.
 	libc.so.6!clone		No symbols loaded.

	__kernel_vsyscall		No symbols loaded.
 	libpthread.so.0!pthread_cond_wait@@GLIBC_2.3.2		No symbols loaded.
 	libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::ThreadNativeWaitData * ptnwdNativeWaitData, DWORD dwTimeout, CorUnix::ThreadWakeupReason * ptwrWakeupReason, DWORD * pdwSignaledObject) Line 479	C++	Symbols loaded.
 	libcoreclr.so!CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalSynchronizationManager * this, CorUnix::CPalThread * pthrCurrent, DWORD dwTimeout, bool fAlertable, bool fIsSleep, CorUnix::ThreadWakeupReason * ptwrWakeupReason, DWORD * pdwSignaledObject) Line 302	C++	Symbols loaded.
 	libcoreclr.so!CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread * pThread, DWORD nCount, const HANDLE * lpHandles, BOOL bWaitAll, DWORD dwMilliseconds, BOOL bAlertable, BOOL bPrioritize) Line 592	C++	Symbols loaded.
 	libcoreclr.so!WaitForMultipleObjectsEx(DWORD nCount, const HANDLE * lpHandles, BOOL bWaitAll, DWORD dwMilliseconds, BOOL bAlertable) Line 188	C++	Symbols loaded.
 	libcoreclr.so!DebuggerRCThread::MainLoop(DebuggerRCThread * this) Line 1234	C++	Symbols loaded.
 	libcoreclr.so!DebuggerRCThread::ThreadProc(DebuggerRCThread * this) Line 1037	C++	Symbols loaded.
 	libcoreclr.so!DebuggerRCThread::ThreadProcStatic() Line 1633	C++	Symbols loaded.
 	libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void * pvParam) Line 1684	C++	Symbols loaded.
 	libpthread.so.0!start_thread		No symbols loaded.
 	libc.so.6!clone		No symbols loaded.

	__kernel_vsyscall		No symbols loaded.
 	libpthread.so.0!pthread_cond_wait@@GLIBC_2.3.2		No symbols loaded.
 	libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::ThreadNativeWaitData * ptnwdNativeWaitData, DWORD dwTimeout, CorUnix::ThreadWakeupReason * ptwrWakeupReason, DWORD * pdwSignaledObject) Line 479	C++	Symbols loaded.
 	libcoreclr.so!CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalSynchronizationManager * this, CorUnix::CPalThread * pthrCurrent, DWORD dwTimeout, bool fAlertable, bool fIsSleep, CorUnix::ThreadWakeupReason * ptwrWakeupReason, DWORD * pdwSignaledObject) Line 302	C++	Symbols loaded.
 	libcoreclr.so!CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread * pThread, DWORD nCount, const HANDLE * lpHandles, BOOL bWaitAll, DWORD dwMilliseconds, BOOL bAlertable, BOOL bPrioritize) Line 592	C++	Symbols loaded.
 	libcoreclr.so!WaitForSingleObjectEx(HANDLE hHandle, DWORD dwMilliseconds, BOOL bAlertable) Line 125	C++	Symbols loaded.
 	libcoreclr.so!CLREventWaitHelper2(HANDLE handle, DWORD dwMilliseconds, BOOL alertable) Line 385	C++	Symbols loaded.
 	libcoreclr.so!CLREventWaitHelper(void*, unsigned int, int)::$_1::operator()(CLREventWaitHelper(void*, unsigned int, int)::Param*) const(const class {...} * this, Param * pParam) Line 411	C++	Symbols loaded.
 	libcoreclr.so!CLREventWaitHelper(HANDLE handle, DWORD dwMilliseconds, BOOL alertable) Line 413	C++	Symbols loaded.
 	libcoreclr.so!CLREventBase::WaitEx(CLREventBase * this, DWORD dwMilliseconds, WaitMode mode, PendingSync * syncState) Line 483	C++	Symbols loaded.
 	libcoreclr.so!CLREventBase::Wait(CLREventBase * this, DWORD dwMilliseconds, BOOL alertable, PendingSync * syncState) Line 426	C++	Symbols loaded.
 	libcoreclr.so!FinalizerThread::FinalizerThreadStart(void*)::$_0::operator()(FinalizerThread::FinalizerThreadStart(void*)::__EEParam*) const(const class {...} * this, FinalizerThread::__EEParam * __pEEParam) Line 827	C++	Symbols loaded.
 	libcoreclr.so!FinalizerThread::FinalizerThreadStart(void * args) Line 854	C++	Symbols loaded.
 	libcoreclr.so!Thread::intermediateThreadProc(PVOID arg) Line 2245	C++	Symbols loaded.
 	libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void * pvParam) Line 1684	C++	Symbols loaded.
 	libpthread.so.0!start_thread		No symbols loaded.
 	libc.so.6!clone		No symbols loaded.

	__kernel_vsyscall		No symbols loaded.
 	libc.so.6!poll		No symbols loaded.
 	libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadPrepareForShutdown() Line 3220	C++	Symbols loaded.
 	libcoreclr.so!CorUnix::CPalSynchronizationManager::WorkerThread(LPVOID pArg) Line 1960	C++	Symbols loaded.
 	libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void * pvParam) Line 1684	C++	Symbols loaded.
 	libpthread.so.0!start_thread		No symbols loaded.
 	libc.so.6!clone		No symbols loaded.

[TheLastRar]

Update:

As I mentioned earlier the code in LinuxPipeThread replaces stdout/stderr with pipes
It then starts up threads that continuously reads from that pipe until it reads eof.

During destruction, the read file descriptor is closed and waits for the read threads to exit (which they quickly do as it reaches eof of the pipe)

However, CoreCLR create an extra set of file descriptors for the stdout/stderr pipes and keeps them open even after coreclr_shutdown has been called, causing the read threads in LinuxPipeThread to continue running, hanging the process.

If I forcible close the extra descriptors (by searching though the data returned from getdtablesize()), the hang dosn't occur and the program closes normally, confirming that it's the extra file descriptors opened by CoreCLR that is causing the hang.

[jkotas]

CoreCLR create an extra set of file descriptors for the stdout/stderr pipes and keeps them open

This issue is tracked by: #25394