500 errors when attempting to upload artifacts

[agrobbin]

We have started getting semi-frequent 500 errors when attempting to upload artifacts after a failed test run:

##### Begin Diagnostic HTTP information #####
Status Code: 500
Status Message: Internal Server Error

I'm unable to determine the cause, but it is pretty frequent (though I wouldn't say constant). I wish I had more debug information to provide, but there's not much else in the debug output.

Version

• V1
• V2

Environment

• self-hosted
• Linux
• Windows
• Mac

Pulling this out of #71 into its own issue, as @konradpabjan requested!

[agrobbin]

@konradpabjan I believe this was due to some special characters being included in the itemPath query param when making the chunk upload request. Sanitizing the filenames seems to have resolved the issue!

[konradpabjan]

What were the characters that you had to sanitize @agrobbin?

Any invalid characters should be caught beforehand 🤔https://github.com/actions/toolkit/blob/9ba7c679ad74f6a80017feeb94b7a7c19b590796/packages/artifact/src/internal/utils.ts#L271

I've managed to repo this 500 randomly (caused by some timeouts behind the scenes). We currently don't retry 500s during upload so I was planning on doing some back-end updates so that 503s (which do get retried) get returned so the upload doesn't fail.

[agrobbin]

Here is the full path (not scrubbed at all as nothing is private):

failures%2Fscreenshots%2Ffailures_r_spec_example_groups_admin_companies_page_visiting_logged_in_as_an_admin_updating_an_activity_item_______then_do%0A________expect%28company_page_activity_items_first%29_to_have_text__%23%7Bmy_item_content_body_574.png

[Geod24]

Also seeing a lot of those 500. We have 3 compilers x 3 OS jobs (so 9 total) and every other push we see at least one of them fail, which forces us to re-trigger the whole build. Re-triggers can be even more flakey, and I've found that removing the artifacts before re-triggering (our artifacts are pretty big) seems to help.
Those failures are not new, but since 2.0.1 they actually fail the job which made them more obvious.

See e.g. https://github.com/bpfkorea/agora/runs/712332960?check_suite_focus=true (but ignore the last failure on CI / Run (macOS-10.15, ldc-1.21.0), that one's on us).

[Arusekk]

+1 here: https://github.com/Gallopsled/pwntools/pull/1570/checks?check_run_id=754787700
The both runs upload to the same artifact simultaneously, so it looks like a race condition (at least in this case).

[konradpabjan]

We've done some work behind the scenes to address users hitting 500s during artifact upload. In most cases they were caused by timeouts. If there was a timeout in the backend, a 503 should now be returned and the action will retry the upload.

If there are any more instances of 500s being hit, please ping me or create a new issue

[fwitzke]

@konradpabjan +1

Run actions/upload-artifact@v2
With the provided path, there will be 1 file(s) uploaded
##[error]Unexpected response. Unable to upload chunk to https://pipelines.actions.githubusercontent.com/iTOPOiur04xYCFvYHEbjY7NM0fzhmdkPMjXzfrsHUmSkZvTBa7/_apis/resources/Containers/10873357?itemPath=test-results.xml
##### Begin Diagnostic HTTP information #####
Status Code: 500
Status Message: Internal Server Error
Header Information: {
  "cache-control": "no-store,no-cache",
  "pragma": "no-cache",
  "content-length": "276",
  "content-type": "application/json; charset=utf-8",
  "strict-transport-security": "max-age=2592000",
  "x-tfs-processid": "98f88b46-bb0d-4ece-af51-fbf9accea451",
  "activityid": "2883cdb4-59a4-40b1-9d45-b38ad4030dda",
  "x-tfs-session": "2883cdb4-59a4-40b1-9d45-b38ad4030dda",
  "x-vss-e2eid": "2883cdb4-59a4-40b1-9d45-b38ad4030dda",
  "x-vss-senderdeploymentid": "193695a0-0dcd-ade4-f810-b10ad24a9829",
  "x-frame-options": "SAMEORIGIN",
  "x-msedge-ref": "Ref A: A42EED5E374A4A93903F020EBFD9BDCF Ref B: BN3EDGE1007 Ref C: 2020-08-20T15:44:30Z",
  "date": "Thu, 20 Aug 2020 15:44:29 GMT"
}
###### End Diagnostic HTTP information ######
##[error]aborting artifact upload

[sytolk]

Run actions/upload-artifact@v2
With the provided path, there will be 1 file(s) uploaded
##[error]Unexpected response. Unable to upload chunk to https://pipelines.actions.githubusercontent.com/KaT39FJT37kFRdwZoYLUjNT4Lv31nE7NmIWS4TSUpcsxLq0EmI/_apis/resources/Containers/6139310?itemPath=test-reports%2FTSN49---E2E-Testing-suite-v3%0A_macos.html
##### Begin Diagnostic HTTP information #####
Status Code: 500
Status Message: Internal Server Error
Header Information: {
  "cache-control": "no-store,no-cache",
  "pragma": "no-cache",
  "content-length": "322",
  "content-type": "application/json; charset=utf-8",
  "strict-transport-security": "max-age=2592000",
  "x-tfs-processid": "a4d24082-b83a-4b73-8375-e7a1e43332e0",
  "activityid": "d0c2340f-2b1d-4adb-96b2-e4e3b8468fab",
  "x-tfs-session": "d0c2340f-2b1d-4adb-96b2-e4e3b8468fab",
  "x-vss-e2eid": "d0c2340f-2b1d-4adb-96b2-e4e3b8468fab",
  "x-vss-senderdeploymentid": "193695a0-0dcd-ade4-f810-b10ad24a9829",
  "x-frame-options": "SAMEORIGIN",
  "x-msedge-ref": "Ref A: 165CDA6884484EF0935727AEFC3D85BD Ref B: ATAEDGE1608 Ref C: 2020-09-08T12:37:57Z",
  "date": "Tue, 08 Sep 2020 12:37:57 GMT"
}
###### End Diagnostic HTTP information ######
##[warning]Aborting upload for /Users/runner/work/tagspaces/tagspaces/tests/test-reports/TSN49---E2E-Testing-suite-v3
_macos.html due to failure
##[error]aborting artifact upload
Total size of all the files uploaded is 0 bytes
Finished uploading artifact test-reports. Reported size is 0 bytes. There were 1 items that failed to upload
##[error]An error was encountered when uploading test-reports. There were 1 items that failed to upload.

[shinny-yangyang]

Run actions/upload-artifact@v2
Multiple search paths detected. Calculating the least common ancestor of all paths
The least common ancestor is /home/runner/work/fclib/fclib. This will be the root directory of the artifact
With the provided path, there will be 7 file(s) uploaded
Total file count: 7 ---- Processed file #6 (85.7%)
Error: Unexpected response. Unable to upload chunk to https://pipelines.actions.githubusercontent.com/HDmNiYkKaBDWtRbp7Gf5PkvQrTyiC1LOXsYziIh8oy0tc3hH8W/_apis/resources/Containers/1878826?itemPath=fclib%2Fdist%2Flib%2Flibfclib.a
##### Begin Diagnostic HTTP information #####
Status Code: 500
Status Message: Internal Server Error
Header Information: {
  "cache-control": "no-store,no-cache",
  "pragma": "no-cache",
  "content-length": "328",
  "content-type": "application/json; charset=utf-8",
  "strict-transport-security": "max-age=2592000",
  "x-tfs-processid": "9bafbdbf-0717-4532-82e7-ee84f4d689ca",
  "activityid": "bb45378b-bad4-4b42-8852-39583db8d3b3",
  "x-tfs-session": "bb45378b-bad4-4b42-8852-39583db8d3b3",
  "x-vss-e2eid": "bb45378b-bad4-4b42-8852-39583db8d3b3",
  "x-vss-senderdeploymentid": "a07ab14e-025a-39c3-8d53-788cd7ce207f",
  "x-frame-options": "SAMEORIGIN",
  "x-msedge-ref": "Ref A: B16D4A25054A44AD888D6917AD936999 Ref B: BN3EDGE0321 Ref C: 2020-09-30T05:45:37Z",
  "date": "Wed, 30 Sep 2020 05:45:38 GMT"
}
###### End Diagnostic HTTP information ######
Warning: Aborting upload for /home/runner/work/fclib/fclib/dist/lib/libfclib.a due to failure
Error: aborting artifact upload
Total size of all the files uploaded is 33568561 bytes
Finished uploading artifact fclib. Reported size is 33568561 bytes. There were 1 items that failed to upload
Error: An error was encountered when uploading fclib. There were 1 items that failed to upload.

[jasonkuhrt]

Error: Unexpected response. Unable to upload chunk to https://pipelines.actions.githubusercontent.com/KGBrfZf1VSWW6kPIg3rQCl1xskiggBV0eRPwnx8VzG6xlnFjI9/_apis/resources/Containers/11866587?itemPath=Prisma+Client+JS+Benchmarks+Results%2F2.16.1%2Fcomplex--database%3Dpostgres--nodeVersion%3D14.x.md
10
##### Begin Diagnostic HTTP information #####
11
Status Code: 500
12
Status Message: Internal Server Error
13
Header Information: {
14
  "cache-control": "no-store,no-cache",
15
  "pragma": "no-cache",
16
  "content-length": "276",
17
  "content-type": "application/json; charset=utf-8",
18
  "strict-transport-security": "max-age=2592000",
19
  "x-tfs-processid": "02a6efc7-191f-4df8-be35-b633ed6f7631",
20
  "activityid": "c79d0959-af14-4aca-8123-f7d571d19ecc",
21
  "x-tfs-session": "c79d0959-af14-4aca-8123-f7d571d19ecc",
22
  "x-vss-e2eid": "c79d0959-af14-4aca-8123-f7d571d19ecc",
23
  "x-vss-senderdeploymentid": "13a19993-c6bc-326c-afb4-32c5519f46f0",
24
  "x-frame-options": "SAMEORIGIN",
25
  "x-ratelimit-resource": "Pipelines/Short",
26
  "x-ratelimit-limit": "90",
27
  "x-ratelimit-remaining": "27",
28
  "x-ratelimit-reset": "1612924643",
29
  "x-msedge-ref": "Ref A: B43B49E121724AAEB76E2A74F71777FA Ref B: BN3EDGE0514 Ref C: 2021-02-10T02:32:38Z",
30
  "date": "Wed, 10 Feb 2021 02:32:38 GMT"
31
}
32
###### End Diagnostic HTTP information ######
33
Warning: Aborting upload for /home/runner/work/prisma-client-js-benchmarks/prisma-client-js-benchmarks/results/2.16.1/complex--database=postgres--nodeVersion=14.x.md due to failure
34
Error: aborting artifact upload
35
Total size of all the files uploaded is 47382 bytes
36
Finished uploading artifact Prisma Client JS Benchmarks Results. Reported size is 47382 bytes. There were 54 items that failed to upload
37
Error: An error was encountered when uploading Prisma Client JS Benchmarks Results. There were 54 items that failed to upload.

[joshuapinter]

I'm seeing a similar thing on the latest version (I'm assuming, since we're using actions/checkout@v2:

With the provided path, there will be 10 files uploaded
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 5548.578148597127 milliseconds before continuing the upload at offset 0
An error has been caught http-client index 1, retrying the upload
Error: Client has already been disposed.
    at HttpClient.request (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:5694:19)
    at HttpClient.sendStream (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:5655:21)
    at UploadHttpClient.<anonymous> (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:7104:37)
    at Generator.next (<anonymous>)
    at .../_work/_actions/actions/upload-artifact/v2/dist/index.js:6834:71
    at new Promise (<anonymous>)
    at module.exports.608.__awaiter (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:6830:12)
    at uploadChunkRequest (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:7102:46)
    at UploadHttpClient.<anonymous> (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:7139:38)
    at Generator.next (<anonymous>)
    Exponential backoff for retry #1. Waiting for 5773.927291417539 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #2. Waiting for 9362.643214894475 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
Total file count: 10 ---- Processed file #9 (90.0%)
Finished backoff for retry #2, continuing with upload
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #3. Waiting for 14276.414696905516 milliseconds before continuing the upload at offset 0
Total file count: 10 ---- Processed file #9 (90.0%)
Total file count: 10 ---- Processed file #9 (90.0%)
Finished backoff for retry #3, continuing with upload
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #4. Waiting for 20561.212693027403 milliseconds before continuing the upload at offset 0
Total file count: 10 ---- Processed file #9 (90.0%)
Total file count: 10 ---- Processed file #9 (90.0%)
Finished backoff for retry #4, continuing with upload
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #5. Waiting for 31215.480894221528 milliseconds before continuing the upload at offset 0
Total file count: 10 ---- Processed file #9 (90.0%)
Total file count: 10 ---- Processed file #9 (90.0%)
Total file count: 10 ---- Processed file #9 (90.0%)
Finished backoff for retry #5, continuing with upload
A 500 status code has been received, will attempt to retry the upload
##### Begin Diagnostic HTTP information #####
Status Code: 500
Status Message: Internal Server Error
Header Information: {
  "cache-control": "no-store,no-cache",
  "pragma": "no-cache",
  "content-length": "328",
  "content-type": "application/json; charset=utf-8",
  "strict-transport-security": "max-age=2592000",
  "x-tfs-processid": "...",
  "activityid": "...",
  "x-tfs-session": "...",
  "x-vss-e2eid": "...",
  "x-vss-senderdeploymentid": "...",
  "x-frame-options": "SAMEORIGIN",
  "x-cache": "CONFIG_NOCACHE",
  "x-msedge-ref": "Ref A: ... Ref B: ... Ref C: 2021-08-05T01:33:28Z",
  "date": "Thu, 05 Aug 2021 01:33:28 GMT"
}
###### End Diagnostic HTTP information ######
Retry limit has been reached for chunk at offset 0 to https://pipelines.actions.githubusercontent.com/.../_apis/resources/Containers/...?itemPath=...
Warning: Aborting upload for ... due to failure
Error: aborting artifact upload
Total size of all the files uploaded is 329038 bytes
Finished uploading artifact .... Reported size is 329038 bytes. There were 1 items that failed to upload
Error: An error was encountered when uploading .... There were 1 items that failed to upload.

(Redacted for privacy.)