AssemblyLoadContext.Unload does not unload a Wpf Library

[netcorefan1]

• .NET Core Version: 3.1.9 and .Net 5.0.100
• Windows version: 10.0.18363
• Does the bug reproduce also in WPF for .NET Framework 4.8?: Not applicable since AssemblyLoadContext is not supported

I am trying to load and unload on demand a Wpf App library (and all the related dependencies). Everything works, but none of the assemblies get unloaded when calling the Unload method.
If I replace the Wpf library with a .Net Core library containing a few sample methods, I can see the library removed from VS Modules window after a couple of GC iterations.
I am not sure if the problem I am reporting is related to this and/or this.
I attached a sample project which is structured like this:

Project 1 (MainApp, a console project with added System.Windows.Forms reference to enable message pump)

class Program
{
        class WpfAppAssemblyLoadContext : AssemblyLoadContext
        {
            public WpfAppAssemblyLoadContext() : base(true) { }
            protected override Assembly Load(AssemblyName assemblyName) => null;
        }

        [MethodImpl(MethodImplOptions.NoInlining),]
        public static void TestRun()
        {
            var context  = new WpfAppAssemblyLoadContext();
            var assembly = context.LoadFromAssemblyPath($"{Environment.CurrentDirectory}\\WpfLibrary.dll");
            var inst     = (IProxy) assembly.CreateInstance("WpfLibrary.MainWindow");
            inst.ShowWindow();
            inst.CloseWindow();
            context.Unload();
            assembly = null;
            context  = null;
            inst     = null;
        }

        [STAThread,]
        static void Main(string[] args)
        {
            TestRun();
            for (var i = 0; i < 100; i++) {
                GC.Collect();
                GC.WaitForPendingFinalizers();
            }

            Application.Run();
        }
}

Project 2 (ProxyInterface)

namespace ProxyInterface
{
    public interface IProxy
    {
        void ShowWindow();
        void CloseWindow();
    }
}

Project 3 (a regular wpf library with implementation of interface in Project 2 )

namespace WpfLibrary
{
    public partial class MainWindow : Window, IProxy
    {
        public MainWindow()
        {
            InitializeComponent();
        }

       public void ShowWindow() { Show();}
       public void CloseWindow() { Close();}
    }
}

UnloadWpfLibrary.zip
(Solution file inside "MainApp" folder)

[netcorefan1]

I just want to update this issue with a new sample project where I applied the workaround suggested from @janvorli .
UnloadWpfLibraryWithWorkaround.zip
This time one of the two assemblies is unloaded, but all of the others are still loaded included WpfLibrary.
I think that for me it's time to give up and recur to IPC (named pipes) although I am not sure if this could be a valid replacement.
May be I missed something and someone more expert can do further progress and attach here the project with the correct modifications, it would be of great benefit for all the users that want to use ALC to load and unload WPF.
It would be a total of 4 projects just to load and unload a wpf assembly on demand and this is not exactly clean, but if the final result is the same it would be acceptable.