SDK build sporadically failing due to multiple pipelines trying to concurrently upload 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'

[danmoseley]

This may possibly just corrupt the blob instead. @joeloff is investigating. We probably should prioritize fixing...

2021-08-11T22:57:06.4524916Z Attempt '1' for uploading chunk '20' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'.
2021-08-11T22:57:06.4525531Z Generate new HttpRequest for uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip', chunk '20' of '21'.
2021-08-11T22:57:06.4526168Z Start uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip' to server, chunk '20'.
2021-08-11T22:57:06.4526978Z Finished upload chunk '20' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip', elapsed 511 (ms), response code 'Accepted'.
2021-08-11T22:57:06.4527796Z Attempt '1' for uploading chunk '21' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'.
2021-08-11T22:57:06.4528585Z Generate new HttpRequest for uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip', chunk '21' of '21'.
2021-08-11T22:57:06.4529202Z Start uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip' to server, chunk '21'.
2021-08-11T22:57:06.4532145Z Chunk '21' attempt '1' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip' fail to send request to server. Error: Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Blob is incomplete (missing block). Blob: af03054df7faeb11b563501ac5b0ff56, Expected Offset: 0, Actual Offset: 83886080
 ---> System.InvalidOperationException: Blob is incomplete (missing block). Blob: af03054df7faeb11b563501ac5b0ff56, Expected Offset: 0, Actual Offset: 83886080
   --- End of inner exception stack trace ---
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.UploadFileAsync(Int64 containerId, String itemPath, Stream fileStream, Byte[] contentId, Int64 fileLength, Boolean isGzipped, Guid scopeIdentifier, CancellationToken cancellationToken, Int32 chunkSize, Int32 chunkRetryTimes, Boolean uploadFirstChunk, Object userState).
2021-08-11T22:57:06.4535515Z Backoff 6.48 seconds before attempt '2' chunk '21' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'.
2021-08-11T22:57:06.4536162Z Attempt '2' for uploading chunk '21' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'.
2021-08-11T22:57:06.4536822Z Generate new HttpRequest for uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip', chunk '21' of '21'.
2021-08-11T22:57:06.4537527Z Start uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip' to server, chunk '21'.
2021-08-11T22:57:06.4539684Z Chunk '21' attempt '2' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip' fail to send request to server. Error: Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Bad Request
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.UploadFileAsync(Int64 containerId, String itemPath, Stream fileStream, Byte[] contentId, Int64 fileLength, Boolean isGzipped, Guid scopeIdentifier, CancellationToken cancellationToken, Int32 chunkSize, Int32 chunkRetryTimes, Boolean uploadFirstChunk, Object userState).
2021-08-11T22:57:06.4542219Z Backoff 5.466 seconds before attempt '3' chunk '21' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'.
2021-08-11T22:57:06.4542842Z Attempt '3' for uploading chunk '21' of file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip'.
2021-08-11T22:57:06.4543477Z Generate new HttpRequest for uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip', chunk '21' of '21'.
2021-08-11T22:57:06.4544114Z Start uploading file 'BlobArtifacts/dotnet-toolset-internal-6.0.100-ci.zip' to server, chunk '21'.
2021-08-11T22:57:06.4544643Z ##[section]Async Command End: Upload Artifact

Same thing happened here:

#19582 (comment)

See also dotnet/core-eng#14015

@dotnet/domestic-cat

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[danmoseley]

This is happening regularly, eg., in the maestro ingestion today for runtime

https://dev.azure.com/dnceng/public/_build/results?buildId=1289894&view=logs&jobId=60adcdc4-a9f0-5668-20f7-b72181c68fb2&j=e98fba6a-08a4-5b7c-1ef9-6d6c234aaf57&t=1735c990-7aa9-5be6-fc9c-fceb366f4f34&s=ff05ad62-bb9a-53b6-ce9f-72f329a63e7c

https://dev.azure.com/dnceng/public/_build/results?buildId=1289894&view=logs&jobId=60adcdc4-a9f0-5668-20f7-b72181c68fb2&j=adc369b2-ee17-52c3-72b2-7129c9e8cda1&t=78acff75-e3a6-582d-ecc2-9f0bf523e0bc&s=ff05ad62-bb9a-53b6-ce9f-72f329a63e7c

both are the full framework ones -- probably because they take about the same time. Both are trying to upload the same blob 'BlobArtifacts/dotnet-toolset-langpack-6.0.100-ci.zip'

@marcpopMSFT this is probably important to look into because we don't know what is getting uploaded, whether it's corrupted etc. It's also causing retries which consume resources.

[joeloff]

I'll take a look today. We probably only need one or need to add the configuration property as a discriminator.

[joeloff]

So this is a zip that contains the full SDK. Looks like it's being used as a transport for dotnet/installer that we then unzip and harvest into the .NET SDK installer. Not sure why we have the langpack, because the normal zip contains the resource assemblies already.

@dsplaisted any concerns if we only publish this for the release build?

[joeloff]

I searched dotnet for dotnet-toolset-internal references. It only shows the SDK, but it's used in the installer repo, so now I'm not sure if I should trust the search. I'm trying to determine whether anyone else besides dotnet/installer depends on this artifact

[joeloff]

Chatted with a few folks. We can just turn off publishing on the debug leg for CI.

[lewing]

Saw this today https://dev.azure.com/dnceng/public/_build/results?buildId=1298276&view=logs&j=ffcccca8-8d4a-5f1e-019c-64de63d1f0d8&t=b45037e0-dc32-5dd3-9457-1b637e753ec3&l=881

[marcpopMSFT]

Jacques made a fix that we think is sufficient. Closing and can reactivate if we see more instances that we're sure include the fix (not sure about Larry's pipeline or not)

[radical]

I'm seeing this in https://github.com/dotnet/sdk/pull/21453/checks?check_run_id=3726720363

https://dev.azure.com/dnceng/public/_build/results?buildId=1389026&view=logs&jobId=34e9329e-985c-5329-c909-963bb6b2b6e6&j=fa59fe4e-195c-56fa-189b-58fd241f10dd&t=afa1ce5e-d2bc-56f3-ad78-7883f25a3e3a

and https://dev.azure.com/dnceng/public/_build/results?buildId=1389026&view=logs&jobId=34e9329e-985c-5329-c909-963bb6b2b6e6&j=34e9329e-985c-5329-c909-963bb6b2b6e6&t=3ca52205-8bcc-5172-569b-1b163f5a51f2

Start uploading file 'BlobArtifacts/dotnet-toolset-langpack-6.0.100-ci.zip' to server, chunk '7'.
Chunk '7' attempt '1' of file 'BlobArtifacts/dotnet-toolset-langpack-6.0.100-ci.zip' fail to send request to server. Error: Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Blob is incomplete (missing block). Blob: df99f97fe41fec11981f501ac512cb76, Expected Offset: 0, Actual Offset: 8388608
 ---> System.InvalidOperationException: Blob is incomplete (missing block). Blob: df99f97fe41fec11981f501ac512cb76, Expected Offset: 0, Actual Offset: 8388608
   --- End of inner exception stack trace ---
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.UploadFileAsync(Int64 containerId, String itemPath, Stream fileStream, Byte[] contentId, Int64 fileLength, Boolean isGzipped, Guid scopeIdentifier, CancellationToken cancellationToken, Int32 chunkSize, Int32 chunkRetryTimes, Boolean uploadFirstChunk, Object userState).
Backoff 5.11 seconds before attempt '2' chunk '7' of file 'BlobArtifacts/dotnet-toolset-langpack-6.0.100-ci.zip'.
Attempt '2' for uploading chunk '7' of file 'BlobArtifacts/dotnet-toolset-langpack-6.0.100-ci.zip'.
Generate new HttpRequest for uploading file 'BlobArtifacts/dotnet-toolset-langpack-6.0.100-ci.zip', chunk '7' of '7'.
Start uploading file 'BlobArtifacts/dotnet-toolset-langpack-6.0.100-ci.zip' to server, chunk '7'.
Chunk '7' attempt '2' of file 'BlobArtifacts/dotnet-toolset-langpack-6.0.100-ci.zip' fail to send request to server. Error: Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Blob is incomplete (missing block). Blob: df99f97fe41fec11981f501ac512cb76, Expected Offset: 0, Actual Offset: 8388608
 ---> System.InvalidOperationException: Blob is incomplete (missing block). Blob: df99f97fe41fec11981f501ac512cb76, Expected Offset: 0, Actual Offset: 8388608
   --- End of inner exception stack trace ---
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.UploadFileAsync(Int64 containerId, String itemPath, Stream fileStream, Byte[] contentId, Int64 fileLength, Boolean isGzipped, Guid scopeIdentifier, CancellationToken cancellationToken, Int32 chunkSize, Int32 chunkRetryTimes, Boolean uploadFirstChunk, Object userState).
Backoff 7.118 seconds before attempt '3' chunk '7' of file 'BlobArtifacts/dotnet-toolset-langpack-6.0.100-ci.zip'.
Attempt '3' for uploading chunk '7' of file 'BlobArtifacts/dotnet-toolset-langpack-6.0.100-ci.zip'.
Generate new HttpRequest for uploading file 'BlobArtifacts/dotnet-toolset-langpack-6.0.100-ci.zip', chunk '7' of '7'.
Start uploading file 'BlobArtifacts/dotnet-toolset-langpack-6.0.100-ci.zip' to server, chunk '7'.
Async Command End: Upload Artifact
##[error]File upload failed even after retry.

[Tarrasse]

This issue is being discussed in arcade
dotnet/arcade#9026

[marcpopMSFT]

Latest fix for this potentially here: https://github.com/dotnet/sdk/pull/27899/files

I think briefly templating made this more common in the past by accidentally publishing and that gave us enough impetus to figure out what was likely going on (-publish was automatically set by one of the scripts we were using)