Intermittent crashing when incorrect receiver format selected

[Dom2364]

Recently my VRS started crashing randomly every few hours (about 1-4 times per day, seems to be more frequent in the daytime with more aircraft airborne). After finding an error in the log file related to one of our feeders, I checked the stats for that feeder and it was sending data but no decoded messages, and after changing the format to Compressed VRS it started working again. It was originally set to Beast/AVR and should have been sending Beast. Turns out they were pushing 2 feeds, one being Beast and the other Compressed VRS, and whenever the receiver would reconnect VRS would pick a different feed. After disabling the receiver VRS hasn't crashed since.

Version: 3.0.0 Beta
OS: Windows Server 2012 R2

Screenshot of error from log file
I only saw one instance of that error in the log file when I checked it so it seems to be very intermittent and the timing is separate to the crashes.

Error from event viewer after crashes:

Application: VirtualRadar.exe
Framework Version: v4.0.30319
Description: The application requested process termination through System.Environment.FailFast(string message).
Message: An exception was not handled in an AsyncLocal<T> notification callback.
Stack:
   at System.Environment.FailFast(System.String, System.Exception)
   at System.Threading.ExecutionContext.OnAsyncLocalContextChanged(System.Threading.ExecutionContext, System.Threading.ExecutionContext)
   at System.Threading.ExecutionContext.SetExecutionContext(System.Threading.ExecutionContext, Boolean)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart()

[vradarserver]

I can see where the exception shown in the log is getting thrown, it's during the creation of an IApplicationInformation object at the start of BackgroundThreadMethod in BackgroundThreadQueue. I suspect this was because the connection that was using that background thread queue was closed before it had a chance to start up, which would tie in with the whole thing of being sent the wrong feed format, but it isn't obvious how that would lead to the exception you're seeing in the event viewer.

The .NET framework will halt programs that allow an unhandled exception to bubble up out of a thread handler function (unless the exception is a ThreadAbortException). I think the message you're seeing in the event viewer is because something in VRS let an exception bubble up back to the framework. An unhandled exception could be logged before it's allowed to bubble out of the handler but the handler that logged the exception you saw in your log would not let an exception bubble out of it, so this one is somewhere else. Which is unfortunate, because by their nature there's unlikely to be a log for it and VRS is rammed full of background thread handlers :)

I've tried setting up a Beast receiver and connecting it to a Compressed VRS feed but so far no joy... it's not disconnecting and reconnecting, it's just sitting there not decoding anything.

It might be an idea for you to configure your Windows to create a crash dump file the next time that VRS halts unexpectedly. I should be able to tell from that what it was doing at the time it crashed. However, that does involve editing the registry (and presumably re-enabling the dodgy connection so that it triggers the crash).

The instructions to configure crash dumps are here: https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps. The short version is:

1. Under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps create a key called virtualradar.exe.
2. Go into the virtualradar.exe key and create a DWORD value called DumpType.
3. Set DumpType to 1.

If / when VRS crashes again Windows should create a file in %LOCALAPPDATA%\CrashDumps for it. If you could upload that to a file sharing site and email me the URL for it then I can download it and take a look, see if I can figure out where it's going wrong.

That said, I'm going to leave this instance with the Beast receiver running for a while, see if I can't reproduce it here.

[Dom2364]

Thanks for the quick response, I have asked that feeder to return to the previous setup if possible, and enabled crash dumps on the server. I have also set up VRS on my PC with crash dumps enabled to receive 2 push feeds on the same port, one Compressed VRS and one Beast with as many messages as I can get pushing from AussieADSB to hopefully increase the chances of a crash. Also if you want full dumps instead of mini dumps I can send those instead.

[vradarserver]

Just to keep this current, @Dom2364 was kind enough to send a crash dump but unfortunately the stack frame for the thread that threw the exception doesn't have the thread handler in it.

The current plan is to try:

1. Sending random bytes to a Beast \ AVR receiver and see if that crashes it.
2. Send a valid Beast feed for a few seconds and then start sending random gibberish. My recollection of the Beast decoder is that it listens for magic numbers to establish the feed type (Beast / text AVR / binary AVR I think?) and then once its established the feed type it behaves differently. It might be that the listener only behaves itself with compressed VRS feeds if it's still waiting for a magic sequence to appear on the feed.