FATAL errors on websocket hiccups

[JustArchi]

SteamRE/SteamKit#426

2017-08-12 23:21:23|ArchiSteamFarm-9707|FATAL|ASF|OnUnobservedTaskException() System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The remote party closed the WebSocket connection without completing the close handshake.) ---> System.Net.WebSockets.WebSocketException: The remote party closed the WebSocket connection without completing the close handshake. ---> System.IO.IOException: Unable to transfer data on the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Security.SslStreamInternal.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStreamInternal.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, LazyAsyncResult asyncResult)
   at System.Net.Security.SslStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncCallback asyncCallback, Object asyncState)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrim[TInstance,TArgs](TInstance thisRef, TArgs args, Func`5 beginMethod, Func`3 endMethod)
   at System.IO.Stream.BeginEndWriteAsync(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.Net.WebSockets.ManagedWebSocket.SendFrameLockAcquiredNonCancelableAsync(MessageOpcode opcode, Boolean endOfMessage, ArraySegment`1 payloadBuffer)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake. ---> System.IO.IOException: Unable to transfer data on the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Security.SslStreamInternal.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStreamInternal.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, LazyAsyncResult asyncResult)
   at System.Net.Security.SslStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncCallback asyncCallback, Object asyncState)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrim[TInstance,TArgs](TInstance thisRef, TArgs args, Func`5 beginMethod, Func`3 endMethod)
   at System.IO.Stream.BeginEndWriteAsync(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.Net.WebSockets.ManagedWebSocket.SendFrameLockAcquiredNonCancelableAsync(MessageOpcode opcode, Boolean endOfMessage, ArraySegment`1 payloadBuffer)<---

[JustArchi]

This issue should be fixed since ASF V3.0.2.0 onwards (which implementes alpha7 version of SK2 2.0). If you want to help, please ensure that you're using ASF V3.0.2.0 and your SteamProtocols setting in ASF.json is set to 4 (websocket). If you spot any unhandled exceptions related to networking or websockets, especially similar to above, please let me know. Thanks!

[guihkx]

I've just got a bunch of those. I know today is maintenance day, but at the time I got those errors, Steam seemed to be operating fine, I made sure to keep an eye on https://steamstat.us/. Anyway, here's the log.txt:

https://gist.github.com/GUiHKX/6c82462161895a699e0a7744b822a76e/raw/592687b3c1dc7c65c208b3c4a93e982cfe885e6a/log.txt

My timezone is UTC-3, btw. And here's my ASF.json:

{
  "AutoRestart": false,
  "AutoUpdates": false,
  "Blacklist": [],
  "ConnectionTimeout": 60,
  "CurrentCulture": null,
  "Debug": false,
  "FarmingDelay": 15,
  "GiftsLimiterDelay": 1,
  "Headless": false,
  "IdleFarmingPeriod": 3,
  "InventoryLimiterDelay": 3,
  "IPCHost": "127.0.0.1",
  "IPCPort": 1242,
  "LoginLimiterDelay": 10,
  "MaxFarmingTime": 4,
  "MaxTradeHoldDuration": 15,
  "OptimizationMode": 0,
  "Statistics": false,
  "SteamOwnerID": <hidden>,
  "SteamProtocols": 4,
  "UpdateChannel": 2
}

[guihkx]

I have one more log file to share, thanks to my shitty internet. This one was generated on Windows, so that's why isn't in english... Maybe I've got a new error as well? (At least I couldn't find this message in the previous log I sent):

2017-08-30 22:59:06|dotnet-7144|FATAL|ASF|OnUnobservedTaskException() System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (There is already one outstanding 'ReceiveAsync' call for this WebSocket instance. ReceiveAsync and SendAsync can be called simultaneously, but at most one outstanding operation for each of them is allowed at the same time.) ---> System.InvalidOperationException: There is already one outstanding 'ReceiveAsync' call for this WebSocket instance. ReceiveAsync and SendAsync can be called simultaneously, but at most one outstanding operation for each of them is allowed at the same time.
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.WebSockets.WinHttpWebSocket.<ReceiveAsync>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at SteamKit2.WebSocketConnection.WebSocketContext.<ReadMessageAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at SteamKit2.WebSocketConnection.WebSocketContext.<RunCore>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at SteamKit2.WebSocketConnection.WebSocketContext.Dispose()
   at SteamKit2.WebSocketConnection.DisconnectCore(Boolean userInitiated, WebSocketContext specificContext)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.InvalidOperationException: There is already one outstanding 'ReceiveAsync' call for this WebSocket instance. ReceiveAsync and SendAsync can be called simultaneously, but at most one outstanding operation for each of them is allowed at the same time.
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.WebSockets.WinHttpWebSocket.<ReceiveAsync>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at SteamKit2.WebSocketConnection.WebSocketContext.<ReadMessageAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at SteamKit2.WebSocketConnection.WebSocketContext.<RunCore>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at SteamKit2.WebSocketConnection.WebSocketContext.Dispose()
   at SteamKit2.WebSocketConnection.DisconnectCore(Boolean userInitiated, WebSocketContext specificContext)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)<---

Here's the complete log.txt:

https://gist.githubusercontent.com/GUiHKX/1f600e14f6c4826af38e34efe58b618d/raw/49e69d41357ca8ea28cec3dcf4c62beae7b351c2/log.txt

Hope it helps.

[JustArchi]

Bug hunting continues with 3.0.2.3.

If you want to help, please ensure that you're using ASF V3.0.2.3 and your SteamProtocols setting in ASF.json is set to 4 (websocket). If you spot any unhandled exceptions related to networking or websockets, especially similar to above, please let me know. Thanks!

[guihkx]

It has happened again. I'm sorry for the huge 2MB file, but it took me too long to notice those errors. Anyway, here it is:

https://gist.githubusercontent.com/GUiHKX/822bf93ea5bec76eb3cea72b45a50880/raw/c9458854fd0a060d9e7ea610daa66904c075b40f/log.txt

[K4r0qtuYNE5G4qgZ]

I get the exact same error logs posted here so it is the same, i only did not post tons of logs that's all.
I also code apps myself and even test them for companies, so i assure you it's not non-sense and unrelated. But if you prefer me to post the same logs as the people above instead i will do that.

I also assure you that my Internet is flawless and decent, i have an 1Gbit fiber optics here and usually only have an hour overall per year or so where it isn't available, so it's certainly not an problem on my side.

And like i said, if you believe or not, if ASF has this hiccups it seems to affect the Steam Client too. That's why i will test that now with ASF V2.3.2.0 by let it run for a week or two to see if it still happens or if it only happens with the V3.x.x.x versions.

But if you aren't interested in the results and out of prejudice suppose i only waste your time and also only confuse then i will also not waste my time anymore too, it's not like i don't have more important things to do you know.

[JustArchi]

Read what the issue is about.

FATAL errors on websocket hiccups

WebSocket protocol is enabled only when you set SteamProtocols to 4. Latest stable ASF V3, V3.0.1.9 has SteamProtocol set to 1, which uses only TCP protocol.

Can you explain to me how are you getting websocket crashes if you're not using websocket protocol?

Reporting issues without backing them up with mandatory log is automatically ignored, you're probably not realizing what you're talking about despite of "having a knowledge", by taking ASF log lines out of the context and incorrectly matching bug with technical issue. What you're doing right now is spamming without any constructive development feedback. Like I told you above, if you want to report an issue, then back it up with log, or don't waste your time and continue doing "more important things" if you can't follow this simple tip.

[K4r0qtuYNE5G4qgZ]

It was set to 6 for V3.0.2.2 and now 4 for V3.0.2.4 and i got the same errors.

And no, i can't explain why it crashes the sockets, i only know it does if there are short steam disconnects like those maintenance periods or unplanned ones.

There also was an huge .net update that came yesterday, so i will also test if this one fix anything.

EDIT: Ok, i will post huge Logs in future if this makes you happy, but they are in german currently, so i will set it to english only for you then. Or well simply ignore it and not any longer waste my time here.

[JustArchi]

It crashes the sockets because you're using experimental unsupported configuration, which is hitting exactly the issue we're resolving here. If you're expecting stable experience then don't use pre-releases and pre-release settings. If on the other hand you want to help the development, then don't complain and provide constructive feedback, like @guihkx is doing all the time, helping me greatly to solve all remaining ASF issues.

Pre-release is not "faster stable". Either you want to help the development, or you should not use it. And if you want to help, then you shouldn't make it harder for me. A simple "I've encountered the same issue on ASF V3.0.2.4" would be enough.

[Zuko]

Edited.

Gist below.

[JustArchi]

If you could post full exceptions instead of half-cut then that would be great 🙂.

[Zuko]

Ups: https://gist.github.com/Zuko/1a6d5f0855ff77836c822eeef65b64a6

[JustArchi]

Yep, looks like same already reported issue. What OS?

[Zuko]

CentOS 7 x64

[JustArchi]

Thanks, added that info to already opened ticket.

[guihkx]

Is this a new one? Dunno if it's related at all, and I wasn't sure if I should open a new issue...

2017-10-31 20:57:25|dotnet-2332|FATAL|ASF|OnUnobservedTaskException() System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The operation was canceled.) ---> System.OperationCanceledException: The operation was canceled.
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.OperationCanceledException: The operation was canceled.<---
 

FULL LOG.TXT

I just got that on Windows, a few minutes ago. I'm running the generic variant. Here's my ASF.json:

{
  "AutoRestart": false,
  "AutoUpdates": false,
  "BackgroundGCPeriod": 0,
  "Blacklist": [],
  "ConnectionTimeout": 60,
  "CurrentCulture": null,
  "Debug": false,
  "FarmingDelay": 15,
  "GiftsLimiterDelay": 1,
  "Headless": false,
  "IdleFarmingPeriod": 8,
  "InventoryLimiterDelay": 3,
  "IPCHost": "127.0.0.1",
  "IPCPort": 1242,
  "LoginLimiterDelay": 10,
  "MaxFarmingTime": 5,
  "MaxTradeHoldDuration": 15,
  "OptimizationMode": 0,
  "Statistics": false,
  "SteamOwnerID": <hidden>,
  "SteamProtocols": 4,
  "UpdateChannel": 0
}

[JustArchi]

There is not even a stacktrace, I'm not sure from where that one happened but if there is no extra info related to that then I can't guess anyway. Ignore.

[JustArchi]

This issue should be completely fixed with next .NET Core release, which is 2.1. ASF will follow once it goes stable. Ref: https://github.com/dotnet/corefx/issues/24002

The issue remains open until it's fixed in ASF.

[HeartOfSun]

"SteamProtocols": 1, but:

2018-02-05 08:40:16|ArchiSteamFarm-7725|FATAL|ASF|OnUnobservedTaskException() System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The remote party closed the WebSocket connection without completing the close handshake.) ---> System.Net.WebSockets.WebSocketException: The remote party closed the WebSocket connection without completing the close handshake. ---> System.IO.IOException: Unable to write data to the transport connection: The lazily-initialized type does not have a public, parameterless constructor.. ---> System.MissingMemberException: The lazily-initialized type does not have a public, parameterless constructor.
   at System.Threading.LazyHelpers`1.ActivatorFactorySelector()
   at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Func`1 valueFactory)
   at System.Net.Sockets.Socket.RentSocketAsyncEventArgs(Boolean isReceive)
   at System.Net.Sockets.Socket.SendAsync(ArraySegment`1 buffer, SocketFlags socketFlags, Boolean fromNetworkStream)
   at System.Net.Sockets.NetworkStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 size, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 size, CancellationToken cancellationToken)
   at System.Net.WebSockets.ManagedWebSocket.SendFrameLockAcquiredNonCancelableAsync(MessageOpcode opcode, Boolean endOfMessage, ArraySegment`1 payloadBuffer)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake. ---> System.IO.IOException: Unable to write data to the transport connection: The lazily-initialized type does not have a public, parameterless constructor.. ---> System.MissingMemberException: The lazily-initialized type does not have a public, parameterless constructor.
   at System.Threading.LazyHelpers`1.ActivatorFactorySelector()
   at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Func`1 valueFactory)
   at System.Net.Sockets.Socket.RentSocketAsyncEventArgs(Boolean isReceive)
   at System.Net.Sockets.Socket.SendAsync(ArraySegment`1 buffer, SocketFlags socketFlags, Boolean fromNetworkStream)
   at System.Net.Sockets.NetworkStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 size, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 size, CancellationToken cancellationToken)
   at System.Net.WebSockets.ManagedWebSocket.SendFrameLockAcquiredNonCancelableAsync(MessageOpcode opcode, Boolean endOfMessage, ArraySegment`1 payloadBuffer)<---

ASF Version 3.1.0.0, under Debian 8

[JustArchi]

Websockets are also being used in IPC GUI now, and your issue was already been resolved by .NET Core team.

[JadynWong]

the same problem @HeartOfSun centos7 dotnet-sdk2.1.4

[JustArchi]

Websockets are also being used in IPC GUI now, and your issue was already been resolved by .NET Core team.

Which part of the above message you didn't understand @JadynWong?

[JadynWong]

sorry.i understand. you mean is that I need to wait for the dotnet core team to update the runtime to fix this problem

[JustArchi]

@w-victorique Stop spamming with stacktraces in entirely unrelated issues. You don't have even a single instance of websocket namespace. Ask in appropriate place for that (and GitHub is not appropriate place for technical issues).

[JustArchi]

Fixed by #799