Microsoft.Win32.OpenFileDialog.ShowDialog does not work

[jbe2277]

• .NET Core SDK: 3.0.100-preview5-011256 [e6f157e5a4]
• Microsoft.WindowsDesktop.App 3.0.0-preview5-27610-56
• VS 2019: 16.1.0 Preview 1.0
• Windows version: Win10 1809 (Azure VM, connected via Remote Desktop)
• Does the bug reproduce also in WPF for .NET Framework 4.8?: No

Calling Microsoft.Win32.OpenFileDialog.ShowDialog(ownerWindow) does not work as expected. It does not show the dialog but returns immediately with true. The property FileName is set to empty string "".

Reproduce:

• Clone https://github.com/jbe2277/waf.git
• Checkout the netcoreapp3.0 branch
• Open System.Waf.sln in VS 2019.
• Set Samples/Writer/Writer.Presentation as start-up project
• Start the application and click on Open.

Expected: The OpenFileDialog should be shown. You could compare this with the master branch (targets .NET 4.6.1)

Actual: A error message comes from the application that a file with empty file name cannot be opened.

[ryalanms]

Thanks, @jbe2277. There is an internal email thread discussing the same problem, but the repro wasn't consistent. I'll look at this immediately.

[jbe2277]

Info: I'm using the Azure VM with VS 2019. So I'm connected via RDP (Remote Desktop).

[ryalanms]

The window handle parameter is correct in RunVistaDialog but corrupted before it reaches the COM component.

comdlg32.dll!CFileOpenSave::Show(HWND__ * hwndOwner) Line 9050	C++
[Managed to Native Transition]	

PresentationFramework.dll!Microsoft.Win32.FileDialog.RunVistaDialog(System.IntPtr hwndOwner) Line 2114 C#
PresentationFramework.dll!Microsoft.Win32.FileDialog.RunDialog(System.IntPtr hwndOwner) Line 959 C#
PresentationFramework.dll!Microsoft.Win32.CommonDialog.ShowDialog() Line 138 C#

[ryalanms]

+@vatsan-madhavan

[ryalanms]

WPF is returning a struct for the HRESULT instead of Int32, which is no longer supported as of recent (strict) coreclr changes.

[weltkante]

@ryalanms thats unfortunate, we also have a HRESULT struct in our own interop code, made the interop so much more readable.

Do you have a link to the coreclr changes?

[weltkante]

Never mind, found dotnet/coreclr#23955 with ongoing discussion about restoring that particular behavior

[ryalanms]

Fixed by dotnet/coreclr#23974.