PlasmaMax & WSL crash on open file dialog

[dpogue]

Reported by Semjay in the OU Discord: https://discord.com/channels/282045216221822978/282046743338811392/1179074885029482548

This is probably happening in Max 8, but we should see if it affects newer versions too.

Max crash when trying to select any sound file.
This is due to an interaction between Max, common dialogs and the Windows Subsystem for Linux.

We came up with two possible fixes:
a.) By disabling WSL
or
b.) Adding a line to plAudioComponents.cpp line 1027 ofn.flagsEx = OFN_Ex_NOPLACESBAR;

This prevents the left hand panel on the dialog box showing, which stops WSL starting up and crashing Max.
We chose to do this.

[jfmherokiller]

the question tho is if we should just disable that bar by default or have some way of detecting it/adding some kind of setting for the user to disable it if crashing does occur.

[dpogue]

the question tho is if we should just disable that bar by default or have some way of detecting it/adding some kind of setting for the user to disable it if crashing does occur.

Disabling it entirely is a good quick fix. It's also worth looking for other spots that try to show an open file dialog and see if they are affected.

Of potential note, this is using the older GetOpenFileName API and there's a newer thing that was introduced in Vista: https://learn.microsoft.com/en-us/windows/win32/shell/common-file-dialog

[jfmherokiller]

also can confirm that 2023 and 2024 use the same cmake hints

[jfmherokiller]

for the sake of documentation this is the places bar

and while disabling would be a quick and easy fix. it would also affect all other versions of the plugin unless we possibly constrain it with a MAX_VERSION_MAJOR or other check.

[jfmherokiller]

for documentation sake it seems 3ds max 8 has these as the version numbers

[Deledrius]

Generally (although not always) the versions being checked are commented in MaxCompat:
https://github.com/H-uru/Plasma/blob/master/Sources/Tools/MaxMain/MaxCompat.h

[jfmherokiller]

in combination with the max ifdef I also made this function to check if wsl is installed

bool CheckIfWslInstalled() {
    auto verinfo = hsGetWindowsVersion();
    auto Is10OrAbove = verinfo.dwMajorVersion >= 10;
    DWORD fileAttributes = GetFileAttributesA("wsl.exe");
    bool DoesWslExist = (fileAttributes != INVALID_FILE_ATTRIBUTES && !(fileAttributes & FILE_ATTRIBUTE_DIRECTORY));
    return Is10OrAbove && DoesWslExist;
}

[Hoikas]

I'm not sure if this is the same issue, but any time I try to expand the "Save in" in any open or save file dialog in 3ds Max 7 and 8 on Windows 11, I get a crash in a worker thread with the following backtrace:

 	KernelBase.dll!_RaiseException@16()	Unknown
 	ucrtbase.dll!__CxxThrowException@8()	Unknown
>	p9np.dll!wil::details::ThrowResultExceptionInternal(struct wil::FailureInfo const &)	Unknown
 	p9np.dll!wil::ThrowResultException(struct wil::FailureInfo const &)	Unknown
 	p9np.dll!wil::details::ReportFailure_NoReturn<0>(void *,unsigned int,char const *,char const *,char const *,void *,struct wil::details::ResultStatus const &,unsigned short const *,enum wil::details::ReportFailureOptions)	Unknown
 	p9np.dll!wil::details::ReportFailure_Base<0,0>(void *,unsigned int,char const *,char const *,char const *,void *,struct wil::details::ResultStatus const &,unsigned short const *,enum wil::details::ReportFailureOptions)	Unknown
 	p9np.dll!wil::details::ReportFailure_Hr<0>(void *,unsigned int,char const *,char const *,char const *,void *,long)	Unknown
 	p9np.dll!wil::details::in1diag3::_Throw_Hr(void *,unsigned int,char const *,long)	Unknown
 	p9np.dll!<lambda>(void)()	Unknown
 	kernel32.dll!@BaseThreadInitThunk@12()	Unknown
 	ntdll.dll!___RtlUserThreadStart@8()	Unknown
 	ntdll.dll!__RtlUserThreadStart@8()	Unknown

I can manually move up directory levels until I get to what should be "My Computer," which also crashes.

[dpogue]

hmm, p9np.dll is definitely related to WSL stuff

[dgelessus]

Since it seems to be throwing an exception based on a HRESULT - can you see either the exception or the HRESULT code in the crash dump/debugger?

[Hoikas]

I tried to get the HRESULT from the debugger, but I was unable to see any details about the exception. This is probably more that I am not well versed in WinDbg than it not being possible.

[Hoikas]

I stumbled on microsoft/WSL#4027 (comment) tonight when googling for errors in p9np.dll. The odd thing to me is that this crash is happening both on my host computer but also in a Windows 11 Hyper-V VM that I'm running. I went into regedit in the Windows 11 VM and removed P9NP from HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\NetworkProvider\Order, and the crash no longer occurs on the virtual machine. I wonder if it would be worth opening an issue on microsoft/WSL about this?