ERROR: The operation was canceled. -> Unable to read data from the transport connection: Operation canceled.

[maxd]

Hello,

When I run dotnet symbol to download debug symbols for .NET Core SDK 2.1.10 it throw the following error:

...
Writing: /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.10/System.Diagnostics.FileVersionInfo.pdb
Cached: /root/.dotnet/symbolcache/system.componentmodel.annotations.pdb/5b25c24677e342c39b52d7d0bb347779FFFFFFFF/system.componentmodel.annotations.pdb
Writing: /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.10/System.ComponentModel.Annotations.pdb
Cached: /root/.dotnet/symbolcache/system.private.corelib.pdb/d86a24c58e91402d8f6d21028185ef75FFFFFFFF/system.private.corelib.pdb
Writing: /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.10/System.Private.CoreLib.pdb
ERROR: The operation was canceled. -> Unable to read data from the transport connection: Operation canceled.

Looks like dotnet symbol is trying to download large file and can't do it because exceed read timeout. Could you please check it?

Steps to reproduce:

1. Create Dockerfile:

   FROM mcr.microsoft.com/dotnet/core/sdk:2.1.603@sha256:8eda326c4c0bd038f3cc5aeb5c8af8374b88f1b90eb444ed91bab42f2e17840f
   
   RUN dotnet tool install -g dotnet-symbol
   
   ENV PATH=$PATH:/root/.dotnet/tools
   
   RUN dotnet symbol --diagnostics --symbols --output /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.10 '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.10/*'
   

2. Run docker build to build this docker file:

   docker build . -f Dockerfile
   

It should show error described above.

Thank you.

[mikem8361]

I just got back from vacation. I'll look into this as soon as I can.

[mikem8361]

Of course it doesn't repro for me on my machine/network.

I have a version of the dotnet-symbol tool with the request timeout doubled to 4 minutes. I have attached the new nuget package that you can try before I push it but I'm not sure how to install it in your above docker repro. The install command would need to be modified to point to a local "source" or directory like:

dotnet tool install -g dotnet-symbol --version 1.0.2 --add-source /tmp/dotnet_symbol_nuget_package_directory

dotnet-symbol.1.0.2.zip

You will have to download this zip and rename it it to dotnet-symbol.1.0.2.nupkg and put it in a directory that can be accessed by your docker container.

[maxd]

Yes, this fix works! Thank you :)

Here is updated Dockerfile for any case:

FROM mcr.microsoft.com/dotnet/core/sdk:2.1.603@sha256:8eda326c4c0bd038f3cc5aeb5c8af8374b88f1b90eb444ed91bab42f2e17840f

COPY dotnet-symbol.1.0.2.nupkg /packages/

RUN dotnet tool install -g dotnet-symbol --version 1.0.2 --add-source /packages

ENV PATH=$PATH:/root/.dotnet/tools

RUN dotnet symbol --diagnostics --symbols --output /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.10 '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.10/*'

How do you think is it possible to pass timeout through command line argument or environment variable and doesn't hardcode it?

[mikem8361]

I'm not sure if I want to add the complexity of either if hard coding this request timeout increase doesn't have any ill effects.

[mikem8361]

Version 1.0.2 will have the fix for the issue. It will be available as soon as nuget is finished its validation.

dotnet tool update -g dotnet-symbol

[kevingosse]

I've run into the same issue. I'm trying to download the symbols for libcoreclr.so, and it fails with the Unable to read data from the transport connection: Operation canceled message.

I extracted the URI (http://msdl.microsoft.com/download/symbols/_.debug%2Felf-buildid-sym-07da889c2290c495ba1c723452ed6f5b98cd8fc2%2F_.debug) and ran the download on my web browser, the file is 86MB and it's downloading at ~270 KB/s, so it's going to take more than 5 minutes. Of course the question would be "why is the symbol server so slow right now?", but I guess overriding the timeout is still needed in some cases.

[am11]

It is not a coincidence that multiple people are reporting these timeout issues as an unexpected behavior (dotnet/diagnostics#602, dotnet/diagnostics#412). This internet connection downloads 10 GB in less than 200 seconds while streaming Netflix videos, so downloading 70 MB file (for singlefilehost symbols) should take about 1.1 second, while it waits for four minutes and timesout. It looks like the symbol servers are not setup with CDN, so folks outside the US will hit the slow connection (but I don't know why it is THIS slow?)

I think --timeout command-line argument in various tools that interact with symbol server would be an OK short-term pragmatic solution (please add it or just a --no-timeout?; timeout option is currently missing in all tools in the latest versions); but the real solution would be to setup CDN for symbol server, or move the blobs under a better storage plan such as; geo-zone redundant storage.

@mikem8361, should we report it as a new issue or could this be reopened for tracking purposes? Diagnostic tools are unusable for some of us.

[mikem8361]

@am11 could you submit a new issue to add a --timeout option to dotnet-symbol? Thanks.

[am11]

could you submit a new issue to add a --timeout option to dotnet-symbol? Thanks.

#246.