Your Implementation of ShellBrowser and ShellView

[tajbender]

Hi, David,

I hope you and your family are healthful. I'm still working on Shell stuff, perhaps you remember, and I was looking into your implementation of ShellView and ShellBrowser.

I understand that these are still work in progress, however, I was just wondering why you did derive the ShellView object from Control, whereas the ShellBrowser is not.

Shouldn't it be the other way round, or did I not get the point?!?

IShellBrowser interface (shobjidl_core.h)
12/05/2018 2 minutes to read
Implemented by hosts of Shell views (objects that implement IShellView). Exposes methods that provide services for
the view it is hosting and other objects that run in the context of the Explorer window.

I always thought the concept is to have a ShellBrowser as the container (control) who is hosting the ShellViews, which are different for every place (i.e. folder) you navigate to. Or did I miss the point?

Thank you very much for your efforts, regards,
tajbender

[dahall]

We are all well, thank you. I hope the same for you and yours.

To be honest, I was hacking and trying to get anything to work. So far, no success. If you have some time to play with all four of those shell based controls (ShellBrowser, ShellView, ExplorerBrowser and ShellNamespaceTreeControl) I'd appreciate another set of eyes and brains on the problem.

[tajbender]

Hi, David, thanks for your quick reply.

Well, currently I'm on vacation and would be glad to spend some time on these. Nevertheless it will take a while for me to get them into stable shape, but I'll do my best.

I'll bring in Pull Requests for you to do the final code review, cause my C# skills are far from yours. I just wanted to make sure you currently don't work on them to avoid double work.

[dahall]

Wonderful! Thank you. I'll avoid working on that assembly for a while.

[tajbender]

I don't think that's even necessary, I'll concentrate on the Controls themselves.

[tajbender]

Hello David,

I just wanted to let you know about an intermediate result, as I think it may be important for you: After integrating your version of ShellBrowser into electrifier source tree, and stripping out it's ShellView dependencies, I still had the same runtime error as before when using your packaged version. No real surprise so far.

However, I just was wondering cause my own Quick'n'Dirty implementation I had before ran well and was very similar at that point. Then I removed the IOleCommandTarget interface from its declaration, and voilà it works as expected now.

You can browse the branch via here: electrifier/v1.23.Graph-electrifier@4e2bed2

I'm just wondering since as you know, IOleCommandTarget is a very simple and straightforward Interface. But in the MSDN-Docs I found a small hint, take a look:

https://docs.microsoft.com/en-us/windows/win32/api/docobj/nf-docobj-iolecommandtarget-exec

Notes to Implementers
A command target must implement this function; E_NOTIMPL is not a valid return value.

I'll dig further into it soon, but wanted to hear your thoughts on this.

Regards,
tajbender

[dahall]

I just revisited IOleCommandTarget and it is one of my worst implementations. I've updated it and moved it to Ole32 where it belongs. The last param on QueryStatus would have caused stack problems how it was written.

New version:

[ComImport, Guid("B722BCCB-4E68-101B-A2BC-00AA00404770"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IOleCommandTarget
{
	[PreserveSig]
	HRESULT QueryStatus([In, Optional] GuidPtr pguidCmdGroup, uint cCmds, [In, Out, MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 1)] OLECMD[] prgCmds, [In, Optional] /* OLECMDTEXT* */ IntPtr pCmdText);

	[PreserveSig]
	HRESULT Exec([In, Optional] GuidPtr pguidCmdGroup, uint nCmdID, uint nCmdexecopt, [In, Optional] /* VARIANT* */ IntPtr pvaIn, [Out, Optional] /* VARIANT* */ IntPtr pvaOut);
}

[dahall]

Oh, and I apologize, but after that change I had to modify ShellBrowser.cs. I just committed both those changes.

[tajbender]

No problem, I'll give it a try and will let you know about my findings. Would be nice if that was the bug that caused the problems with Shell Controls we had the last weeks.

Thank you very much so far!

[tajbender]

This is so weird... I'm still investigating the issue, and after a while just running the application, without doing anything with the control, I get an exception:

Managed Debugging Assistant 'DisconnectedContext' : 'Transition into COM context 0xa1f11df0 for this RuntimeCallableWrapper failed with the following error: The object invoked has disconnected from its clients. (Exception from HRESULT: 0x80010108 (RPC_E_DISCONNECTED)). This is typically because the COM context 0xa1f11df0 where this RuntimeCallableWrapper was created has been disconnected or it is busy doing something else. Releasing the interfaces from the current COM context (COM context 0xa1f10920). This may cause corruption or data loss. To avoid this problem, please ensure that all COM contexts/apartments/threads stay alive and are available for context transition, until the application is completely done with the RuntimeCallableWrappers that represents COM components that live inside them.'

As said, I'm not using the control, not even the application. It's just running in the background within the VS-debugger.
On the other hand, when using the Release-Build everything seems to be fine 😭

[dahall]

Odd. Which app are you running and with which code base?

…

On Sun, Dec 27, 2020, 6:57 AM Thorsten Jung ***@***.***> wrote: This is so weird... I'm still investigating the issue, and after a while just running the application, without doing anything with the control, I get an exception: Managed Debugging Assistant 'DisconnectedContext' : 'Transition into COM context 0xa1f11df0 for this RuntimeCallableWrapper failed with the following error: The object invoked has disconnected from its clients. (Exception from HRESULT: 0x80010108 (RPC_E_DISCONNECTED)). This is typically because the COM context 0xa1f11df0 where this RuntimeCallableWrapper was created has been disconnected or it is busy doing something else. Releasing the interfaces from the current COM context (COM context 0xa1f10920). This may cause corruption or data loss. To avoid this problem, please ensure that all COM contexts/apartments/threads stay alive and are available for context transition, until the application is completely done with the RuntimeCallableWrappers that represents COM components that live inside them.' As said, I'm not using the control, not even the application. It's just running in the background within the VS-debugger. On the other hand, when using the Release-Build everything seems to be fine 😭 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#188 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADNPM5ZM5V6QCISWXYR4CU3SW44KZANCNFSM4U5RVWFA> .

[dahall]

This doesn’t explain Debug vs Release, but it kind of sounds like an error tied to a handle or COM interface getting released via garbage collection that shouldn’t be due to dependencies. In the library, I made a lot of those thing “safe” so that they didn’t have to be manually cleaned up. I wonder if I overdid it. From: Thorsten Jung <[email protected]> Sent: Sunday, December 27, 2020 6:57 AM To: dahall/Vanara <[email protected]> Cc: David Hall <[email protected]>; Comment <[email protected]> Subject: Re: [dahall/Vanara] Your Implementation of ShellBrowser and ShellView (#188) This is so weird... I'm still investigating the issue, and after a while just running the application, without doing anything with the control, I get an exception: Managed Debugging Assistant 'DisconnectedContext' : 'Transition into COM context 0xa1f11df0 for this RuntimeCallableWrapper failed with the following error: The object invoked has disconnected from its clients. (Exception from HRESULT: 0x80010108 (RPC_E_DISCONNECTED)). This is typically because the COM context 0xa1f11df0 where this RuntimeCallableWrapper was created has been disconnected or it is busy doing something else. Releasing the interfaces from the current COM context (COM context 0xa1f10920). This may cause corruption or data loss. To avoid this problem, please ensure that all COM contexts/apartments/threads stay alive and are available for context transition, until the application is completely done with the RuntimeCallableWrappers that represents COM components that live inside them.' As said, I'm not using the control, not even the application. It's just running in the background within the VS-debugger. On the other hand, when using the Release-Build everything seems to be fine 😭 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#188 (reply in thread)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADNPM5ZM5V6QCISWXYR4CU3SW44KZANCNFSM4U5RVWFA> . <https://github.com/notifications/beacon/ADNPM55DZNG5UXAZD65ZNHLSW44KZA5CNFSM4U5RVWFKYY3PNVWWK3TUL52HS4DFWFCGS43DOVZXG2LPNZBW63LNMVXHJKTDN5WW2ZLOORPWSZGOAAB3V2Y.gif>

[tajbender]

Hello David, yes, in my understanding such a problem should be tied to the Garbage Collection process, which is kind of strange cause it even happens when I create one single Instance of the control, and then leave the Application idle.

So even the most eagerly GC should see the stuff is still in use - This makes me crazy 😒, but it's good to see you don't have an instant idea why this could take place.

I'm testing this within a real life application environment to get access to log-files, and in a simple "control-only-app". Both the same, latest Visual Studio, .net Framework v4.7.2.

And I yes, I know you spent hundreds of hours into Vanara to make things safe and clean, that's why I'm so excited to use it. But this one particular thing is a really strange one, and yes, that's why I want to get an understanding what is happening there.

However, I hope you had a nice X-Mas and will have a happy new year!

Finally, to sum it up, nothing to do for you currently, I'm still having some options to track this little, nasty bug down. Just wanted to let you know that I'm still working on it, and I really hope you have some happy holidays!

We will read us soon. 👍

[tajbender]

Hello David,

first of all, a happy new year to you with all my best wishes 🎉. Then, I hope it's okay to be on first-name terms, my name is Thorsten, aka taj.

Just wanted to let you know that I'm making progress, at least the ShellBrowser-Control is kind of functional so far. There are still minor issues, but as you might know some aspects of this shell stuff is poorly documented so it's just try & error.

Certainly, the code has to be cleaned up and refactored, but I've been playing with the control, both with and without VS-Debugger, and in most terms it seems to be stable as far as I can see.

If you accept the pull request, I'd be glad to prepare the Control and add some Test-Cases for Vanara's audience some day soon. Perhaps you even find the time to do a code review, I'm sure you'll find rough edges that could be improved.

Thanks in advance for your reply,
regards,
tajbender

[dahall]

That's wonderful news. Thank you for all your work to get here. I'd be glad to code review and accept the PRs.

[tajbender]

I still have problems getting the ShellBrowser to accept Keyboard input. Although it has focus, keyboard input is processed by ShellNamepaceTree. Any hints on this?

[dahall]

Check out the code at

Vanara/Windows.Forms/Controls/ShellNamespaceTreeControl.cs

Line 602 in 3093056

bool IMessageFilter.PreFilterMessage(ref Message m)

and see if it is to blame.

[tajbender]

Hello, @dahall,

sorry for the delay, but I had to get back to my occupation and didn't have much time the last days to get things done on the ShellBrowser.

However, the most time I was just testing the Component and it seems to work fine so far. In fact I had hundreds of test runs which didn't cause problems, although for one single time I had a failure with a lost RPC-Context again - maybe this is the same Issue that's present with ExplorerBrowser.

As you move on rapidly I'd give it a try and prepare the pull request this weekend, for the ShellBrowser at least. Then I'd move on to the ExplorerBrowser to see if I can isolate this rare Issue while testing this one.

To be honest, I hope the more eyes look on the code the more chances we have to get this Issues out of the way, cause it occurs so rarely that it's hard to track it down. What do you think, would this be okay to you?

Thanks in advance,
tajbender

[dahall]

Yes. Please prepare the PR and then I'll review it to see if I find something obvious. I'll then merge it and release it so others can use and test.

[tajbender]

It's an ugly planet, a bug planet!

Please be patient with me, still got those nasty lost RPC-Contexts. Now I reverted back to an older version, which seems to work fine.

But I can't understand what the differences should be between this working one and the other one, which got polished for release.

[dahall]

No problem. I really appreciate you spending so much time to make this work.

[tajbender]

Just a reminder for @tajbender: Found this one:

https://www.mirabulus.com/it/blog/2020/03/01/how-to-find-source-of-disconnected-context-with-managed-debugging-assistant

I'll give it a try the next days...

[tajbender]

Hello @dahall,

One Question, which may sound dump, but I'm really not aware what the approriate way handling these COM-References with multiple Interfaces should be:

Let's take a look where my ShellBrowser crashes:

And now, let's take a look at the creation of these references for iShellFolder and iShellFolder2 in class Vanara\Windows.Shell\ShellObjects\ShellItem.cs:

/// <summary>Initializes this instance with the specified IShellItem.</summary>
/// <param name="si">The IShellItem object.</param>
protected void Init(IShellItem si)
{
iShellItem = si ?? throw new ArgumentNullException(nameof(si));
iShellItem2 = si as IShellItem2;
}

So you can see the creation of the ShellItem2-Reference is done by a simple casting (without using the Marshaller). I wonder if the reference count for this object gets increased while doing so.

I know this is done if you use Marshal-class, e.g. https://docs.microsoft.com/en-gb/dotnet/api/system.runtime.interopservices.marshal.getcominterfaceforobject?view=net-5.0

This method returns an interface pointer that represents the requested interface on the specified object. It is particularly useful if you have an unmanaged method that expects to be passed an interface pointer. Calling an object with this method causes the reference count to increment on the interface pointer before the pointer is returned. Always use Marshal.Release to decrement the reference count once you have finished with the pointer. You must adhere to the rules defined by COM when using raw COM interface pointers.

But, on the other hand, are you sure this is the case for a simple Casting as done here?

Thanks in Advance,

tajbender

[dahall]

A while ago I read that with COM objects created using CoClass or Marshal.GetCom... methods, you must call Marshal.ReleaseComObject. However, casted interfaces are treated like managed objects but attached to the reference count of the parent and are released at the time ReleaseComObject is called.

[tajbender]

@dahall Sorry, I'm not sure you did get a proper notification of my last message, cause I've been editing it.

So, what's your opinion on this, do you think this is a serious problem in your ShellItem.Dispose()-method? Could you take a deeper look at it?

[dahall]

Thanks @tajbender. I dig into it tomorrow and confirm what I wrote last week. If I find a leak, I'll commit the fix.

[dahall]

After a lot of reading today, I have determined the following:

1. There is a lot of bad information regarding Marshal.ReleaseComObject on the internet, including on microsoft.com and stackoverflow.com.
2. My conclusion is that it and FinalReleaseComObject should never be called by this library (see here and here).
3. In almost all cases where it has been used, the interface variable (field or property) holding the reference should be set to null. This is for all interfaces, regardless of how they were created (direct or by casting).
4. The only exception to the above is in controlled instances where you are 100% confident that the interface reference will never be used again. An example of this is when you create an interface, call a method or property and then return without returning any reference to the created interface. This should always be done in a try..finally block.
5. In instances where a COM object must be completely released before reuse or to prevent an out-of-proc server (like Excel) to shutdown, the pattern of calling GC.Collect() followed by GC.WaitForPendingFinalizers() should be followed after ensuring that the runtime has no more references to critical objects.

I am starting the process now of going through the entire Vanara project to look for uses of Marshal.ReleaseComObject and replacing with nulling the variable. This will go out with 3.3.3 in the next day or two.

[tajbender]

Wow, this is great... If the guys at Microsoft know about this and advice to let the Garbage Collector do the work, then we should be fine.

Sounds like many work for you, though, but I'm not in a hurry. Keep up the good work!

[dahall]

My work is done. I checked it in yesterday. It is ready for your edits and testing.

[dahall]

I pushed a new formal release (3.3.3) today. Let me know your progress against the updates, especially those I mentioned in this thread yesterday. Also, please let me know if there is more I can do to help.

[tajbender]

Hello, @dahall, thank you very, very much.

These are my results: I did build Vanara yesterday using your latest GitHub-Repository. Today, I used this build within my ShellBrowser-Implementation on a long term test for about eight hours, navigating from folder to folder every now and then.

What should I say: No problems 🎉 🙌

It works just pretty well within the debugger without raising any exception any more. As far as I can see, on my side, the problems we talked about the last couple of days are gone.

Nice work, thank you very, very much.

Now I can go on and will pull the ShellBrowser the next days! Cya then, regards, tajbender 👍 ⚡

[tajbender]

Hallo @dahall,

sorry for delay, I've been sick for some days. I'll do some remaining enhancements to the Control, test them und push a CR the next days.

Thanks for your support!

[tajbender]

Hello, David, I've created the pull request #221.

I've left my comments regarding open and resolved issues in the file header, just for maintenance. It works pretty well, including ViewStateStreams to remember the last View Mode of individual folders, but it has some downgrades, too, especially when browsing the Quick access virtual folder introduced in Windows 10:

As you can see, the expand buttons are in Vista Style and when using groups the icons won't get displayed properly for group number 2+. I've done some research on this but didn't find a solution yet. Maybe, to be honest, that is because that 'old' ShellBrowser just can't view this folder properly?!?

However, beside that I've tested it extensively, and didn't notice any more problems regarding DisconnectedContext.

[tajbender]

Hello, @dahall,

I've played with the ShellBrowser a little bit, again, and found out that when using IFolderViewSettings-Interface, in some cases at least I get the Icons in the Quick Access-folder to work properly, more or less.

What do you think, would it be worth to investigate further? To be honest, I never worked with this interface, as the last time I worked with those Shell stuff I used Windows XP 😁

Perhaps I can find out more, although documentation on these issues is not existent, so it's more or less try & error, again 😞

Thanks for your thoughts on this, and Happy Easter 🥚 to all of you!

[dahall]

Happy Easter to you too. If this is interesting or valuable to you, please continue to explore. However, IMHO the IFolderViewSettings interface is so poorly documented and only rarely used by Shell Namespace Extensions, I don't believe it will get used much by consumers of this library.

[tajbender]

Hello, David,

thanks for accepting the Pull Request, and for cleaning up and revising the code I've developed so far.

In the meantime I've done some more research and came across an IListView-Interface, that is apparently used in Windows Vista and above, although it has slightly been altered for Windows7+.

I guess you won't be too happy to have undocumented stuff in Vanara, but it has been documented by Software Analyst Geoff Chappell, and seems to be used in the wild: csharp.hotexamples.com - IListView

Currently I'm playing around with that beast, but before wasting too much time, I'd like to ask if you're willing to include such stuff at all?
Although, the ViewHandler-stuff in my ShellBrowser-Implementation is undocumented, too, but seems to work for ages now 😁

Thanks in advance for sharing you thoughts on this,
and have a nice weekend, regards,

tajbender

[dahall]

I am open to officially undocumented but widely documented inclusions. Sounds like a good addition.

[tajbender]

Thanks, David, for your quick reply. Well, are you able to extract the P/Invoke wrapper from C-/C++-Sources?

Beside the site mentioned above, i found an CodeProject article that uses these techniques: Undocumented List View Features, which is released as public domain: listviewundoc_src.zip

I thought, some time ago, you mentioned you have your own Tool for converting these, or do you use the (in)official tool that can be found somewhere in the depth of Windows SDK? I just can't remember how it is called, though...

Thank you very much, regards,

tajbender

[dahall]

I don't have a tool, but I have a regex script that will take C++ and IDL code and kind of convert it to C#. However, I usually do a lot of manual work to ensure it is the best transformation. I'll take a peek at the link and see what I can come up with.

[tajbender]

For your convenience, the only difference between the Vista- and Win7-Version is a new method:

• EnableAlphaShadow(bool).

For any reason, it has not been appended to the end of the function table of the IListView-Interface, but has been inserted before other methods.

This is what the function does:

The following is the version of the Interface I hacked quick&dirty tonight after too much coffee, but perhaps it may be helpful to you 😃

IListView.zip

Thanks in advance,
sincerely yours,
tajbender

edit: And this is how i got the handle to the interface:

                this.ViewWindow = this.ShellView.CreateViewWindow(null, folderSettings, owner, owner.ClientRectangle);
                var sysListView32Handle = User32.GetChildWindow(ViewWindow);

                var iid = typeof(IListView).GUID;
                var result = SendMessage(sysListView32Handle, LVM_QUERYINTERFACE, ref iid, out IntPtr listviewhandle);
                var ilistview = (IListView)Marshal.GetTypedObjectForIUnknown(listviewhandle, typeof(IListView));

                ilistview.SetBackgroundColor(0x00fafafa);
                ilistview.SetTextBackgroundColor(0x00fafafa);
                ilistview.SetTextColor(0x00ff0000);

                //ilistview.EnableAlphaShadow(true);