Confusing error when pulling records failed

[heaths]

Recently, for a particular platform, session records weren't pulled but there was little diagnostic information that made that obvious in the CI. Instead, a bunch of tests failed with something like,

{"Message":"There is no active playback session under recording id 222ce98c-2dfe-4d87-8435-62b5bca362f2.","Status":"BadRequest"}

Or even,

{"Message":"Unable to parse the line 786","Status":"BadRequest"}

The latter comes from https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Store/GitStoreBreadcrumb.cs#L32. This could use more information about why the failure happened, perhaps, such that we are able to self-diagnose what occurred.

See https://dev.azure.com/azure-sdk/public/_build/results?buildId=2932431&view=logs&j=0b29554c-b98e-53a9-91b0-6a3a0eec25d0&t=f0fc8583-d8ae-545a-b5e1-1fe75e05b074&s=ff05ad62-bb9a-53b6-ce9f-72f329a63e7c for where this issue occured.

[JoshLove-msft]

@scbedd this is happening in more services. Any updates on the investigation?

[archerzz]

In my case, the error message is like below. (See here for more details.)

Error Message:
   Azure.RequestFailedException : Service request failed.
Status: 400 (Bad Request)

Content:
{"Message":"Unable to parse the line taLake_369e19e7ec","Status":"BadRequest"}

---

Is it possible that Github is throttling us so that pulling records fails intermittently? In the error message above, taLake_369e19e7ec looks like a git commit id.
It's known that sometimes (very rare), the whole pipeline could fail due to git checkout failure which is because of rate limits. The pipeline which will create many jobs running in parallel, it is possible that Github throttles us as well.

[scbedd]

This is resolved. See the #37868 for a description as to what was breaking. See #6604 for actual resolution. we changed the design so that breadcrumbs are not shared anymore. With that being the case, parallel access of the assets folder via multiple processes is no longer a breaking issue.