-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add checksum validation on artifact upload #1063
Conversation
eee8fbe
to
d5c547c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
270: chore(deps): update actions/upload-artifact action to v3.1.1 r=renovate[bot] a=renovate[bot] [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [actions/upload-artifact](https://togithub.com/actions/upload-artifact) | action | minor | `v3.0.0` -> `v3.1.1` | --- ### Release Notes <details> <summary>actions/upload-artifact</summary> ### [`v3.1.1`](https://togithub.com/actions/upload-artifact/releases/tag/v3.1.1) [Compare Source](https://togithub.com/actions/upload-artifact/compare/v3.1.0...v3.1.1) - Update actions/core package to latest version to remove `set-output` deprecation warning [#​351](https://togithub.com/actions/upload-artifact/issues/351) ### [`v3.1.0`](https://togithub.com/actions/upload-artifact/releases/tag/v3.1.0) [Compare Source](https://togithub.com/actions/upload-artifact/compare/v3.0.0...v3.1.0) ##### What's Changed - Bump [`@​actions/artifact](https://togithub.com/actions/artifact)` to v1.1.0 ([https://github.com/actions/upload-artifact/pull/327](https://togithub.com/actions/upload-artifact/pull/327)) - Adds checksum headers on artifact upload ([https://github.com/actions/toolkit/pull/1095](https://togithub.com/actions/toolkit/pull/1095)) ([https://github.com/actions/toolkit/pull/1063](https://togithub.com/actions/toolkit/pull/1063)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/OpenPoolProject/stratum). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC4xOS4wIiwidXBkYXRlZEluVmVyIjoiMzQuMTkuMCJ9--> 271: chore(deps): update github/codeql-action action to v1.1.31 r=renovate[bot] a=renovate[bot] [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github/codeql-action](https://togithub.com/github/codeql-action) | action | minor | `v1.0.26` -> `v1.1.31` | --- ### Release Notes <details> <summary>github/codeql-action</summary> ### [`v1.1.31`](https://togithub.com/github/codeql-action/compare/v1.1.30...v1.1.31) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.30...v1.1.31) ### [`v1.1.30`](https://togithub.com/github/codeql-action/compare/v1.1.29...v1.1.30) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.29...v1.1.30) ### [`v1.1.29`](https://togithub.com/github/codeql-action/compare/v1.1.28...v1.1.29) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.28...v1.1.29) ### [`v1.1.28`](https://togithub.com/github/codeql-action/compare/v1.1.27...v1.1.28) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.27...v1.1.28) ### [`v1.1.27`](https://togithub.com/github/codeql-action/compare/v1.1.26...v1.1.27) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.26...v1.1.27) ### [`v1.1.26`](https://togithub.com/github/codeql-action/compare/v1.1.25...v1.1.26) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.25...v1.1.26) ### [`v1.1.25`](https://togithub.com/github/codeql-action/compare/v1.1.24...v1.1.25) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.24...v1.1.25) ### [`v1.1.24`](https://togithub.com/github/codeql-action/compare/v1.1.23...v1.1.24) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.23...v1.1.24) ### [`v1.1.23`](https://togithub.com/github/codeql-action/compare/v1.1.22...v1.1.23) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.22...v1.1.23) ### [`v1.1.22`](https://togithub.com/github/codeql-action/compare/v1.1.21...v1.1.22) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.21...v1.1.22) ### [`v1.1.21`](https://togithub.com/github/codeql-action/compare/v1.1.20...v1.1.21) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.20...v1.1.21) ### [`v1.1.20`](https://togithub.com/github/codeql-action/compare/v1.1.19...v1.1.20) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.19...v1.1.20) ### [`v1.1.19`](https://togithub.com/github/codeql-action/compare/v1.1.18...v1.1.19) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.18...v1.1.19) ### [`v1.1.18`](https://togithub.com/github/codeql-action/compare/v1.1.17...v1.1.18) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.17...v1.1.18) ### [`v1.1.17`](https://togithub.com/github/codeql-action/compare/v1.1.16...v1.1.17) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.16...v1.1.17) ### [`v1.1.16`](https://togithub.com/github/codeql-action/compare/v1.1.15...v1.1.16) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.15...v1.1.16) ### [`v1.1.15`](https://togithub.com/github/codeql-action/compare/v1.1.14...v1.1.15) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.14...v1.1.15) ### [`v1.1.14`](https://togithub.com/github/codeql-action/compare/v1.1.13...v1.1.14) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.13...v1.1.14) ### [`v1.1.13`](https://togithub.com/github/codeql-action/compare/v1.1.12...v1.1.13) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.12...v1.1.13) ### [`v1.1.12`](https://togithub.com/github/codeql-action/compare/v1.1.11...v1.1.12) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.11...v1.1.12) ### [`v1.1.11`](https://togithub.com/github/codeql-action/compare/v1.1.10...v1.1.11) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.10...v1.1.11) ### [`v1.1.10`](https://togithub.com/github/codeql-action/compare/v1.1.9...v1.1.10) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.9...v1.1.10) ### [`v1.1.9`](https://togithub.com/github/codeql-action/compare/v1.1.8...v1.1.9) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.8...v1.1.9) ### [`v1.1.8`](https://togithub.com/github/codeql-action/compare/v1.1.7...v1.1.8) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.7...v1.1.8) ### [`v1.1.7`](https://togithub.com/github/codeql-action/compare/v1.1.6...v1.1.7) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.6...v1.1.7) ### [`v1.1.6`](https://togithub.com/github/codeql-action/compare/v1.1.5...v1.1.6) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.5...v1.1.6) ### [`v1.1.5`](https://togithub.com/github/codeql-action/compare/v1.1.4...v1.1.5) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.4...v1.1.5) ### [`v1.1.4`](https://togithub.com/github/codeql-action/compare/v1.1.3...v1.1.4) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.3...v1.1.4) ### [`v1.1.3`](https://togithub.com/github/codeql-action/compare/v1.1.2...v1.1.3) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.2...v1.1.3) ### [`v1.1.2`](https://togithub.com/github/codeql-action/compare/v1.1.1...v1.1.2) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.1...v1.1.2) ### [`v1.1.1`](https://togithub.com/github/codeql-action/compare/v1.1.0...v1.1.1) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.0...v1.1.1) ### [`v1.1.0`](https://togithub.com/github/codeql-action/compare/v1.0.32...v1.1.0) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.32...v1.1.0) ### [`v1.0.32`](https://togithub.com/github/codeql-action/compare/v1.0.31...v1.0.32) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.31...v1.0.32) ### [`v1.0.31`](https://togithub.com/github/codeql-action/compare/v1.0.30...v1.0.31) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.30...v1.0.31) ### [`v1.0.30`](https://togithub.com/github/codeql-action/compare/v1.0.29...v1.0.30) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.29...v1.0.30) ### [`v1.0.29`](https://togithub.com/github/codeql-action/compare/v1.0.28...v1.0.29) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.28...v1.0.29) ### [`v1.0.28`](https://togithub.com/github/codeql-action/compare/v1.0.27...v1.0.28) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.27...v1.0.28) ### [`v1.0.27`](https://togithub.com/github/codeql-action/compare/v1.0.26...v1.0.27) [Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.26...v1.0.27) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/OpenPoolProject/stratum). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC4xOS4wIiwidXBkYXRlZEluVmVyIjoiMzQuMTkuMCJ9--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
I ran into this problem with a service that handled large file uploads and the root cause was that our continuous deployment was killing our containers after a shorter than expected timeout. In our case, during deployment, hosts are taken out of the pool and in-flight requests are allowed to finish but it was not known to us at the time that a timeout was even in effect. Also if you're wondering how I even stumbled upon this: https://youtu.be/9qljpi5jiMQ?t=722 |
## Version 7.8.0 ### ✨ New & Improved * compat parser ([#1004](#1004)) ([ead267e](ead267e)) ### 🛠 Fixes & Updates * bold in tables ([#1006](#1006)) ([bc481d9](bc481d9)) * **deps:** bump actions/setup-node from 3 to 4 ([#821](#821)) ([f96ea89](f96ea89)), closes [actions/setup-node#866](actions/setup-node#866) [actions/setup-node#868](actions/setup-node#868) [actions/setup-node#876](actions/setup-node#876) [actions/setup-node#868](actions/setup-node#868) [actions/setup-node#861](actions/setup-node#861) [actions/setup-node#859](actions/setup-node#859) [actions/setup-node#870](actions/setup-node#870) [actions/setup-node#872](actions/setup-node#872) [actions/setup-node#875](actions/setup-node#875) [actions/setup-node#831](actions/setup-node#831) [actions/setup-node#803](actions/setup-node#803) [actions/setup-node#809](actions/setup-node#809) [actions/setup-node#816](actions/setup-node#816) [actions/setup-node#794](actions/setup-node#794) [actions/setup-node#710](actions/setup-node#710) [actions/setup-node#812](actions/setup-node#812) [actions/setup-node#808](actions/setup-node#808) [actions/setup-node#804](actions/setup-node#804) [actions/setup-node#802](actions/setup-node#802) [actions/setup-node#807](actions/setup-node#807) [#876](#876) [#868](#868) [#866](#866) * **deps:** bump actions/upload-artifact from 3 to 4 ([#846](#846)) ([5a72802](5a72802)), closes [actions/upload-artifact#464](actions/upload-artifact#464) [actions/upload-artifact#313](actions/upload-artifact#313) [actions/upload-artifact#436](actions/upload-artifact#436) [#374](#374) [#375](#375) [#351](#351) [actions/upload-artifact#327](actions/upload-artifact#327) [actions/toolkit#1095](actions/toolkit#1095) [actions/toolkit#1063](actions/toolkit#1063) [#515](#515) [#511](#511) [#509](#509) [#505](#505) * **deps:** bump braces from 3.0.2 to 3.0.3 ([#907](#907)) ([6b017a7](6b017a7)), closes [#40](#40) [#37](#37) [#27](#27) * **deps:** bump core-js from 3.36.1 to 3.38.1 ([#1013](#1013)) ([58ceef7](58ceef7)), closes [#1357](https://github.com/readmeio/markdown/issues/1357) [#1361](https://github.com/readmeio/markdown/issues/1361) [#1352](https://github.com/readmeio/markdown/issues/1352) [#1345](https://github.com/readmeio/markdown/issues/1345) [#1344](https://github.com/readmeio/markdown/issues/1344) [#1361](https://github.com/readmeio/markdown/issues/1361) * **deps-dev:** bump css-loader from 6.11.0 to 7.1.2 ([#1015](#1015)) ([6be77cf](6be77cf)), closes [#1600](https://github.com/readmeio/markdown/issues/1600) [#1590](https://github.com/readmeio/markdown/issues/1590) [#1577](https://github.com/readmeio/markdown/issues/1577) [#1600](https://github.com/readmeio/markdown/issues/1600) [#1590](https://github.com/readmeio/markdown/issues/1590) [#1577](https://github.com/readmeio/markdown/issues/1577) [#1600](https://github.com/readmeio/markdown/issues/1600) [#1596](https://github.com/readmeio/markdown/issues/1596) [#1595](https://github.com/readmeio/markdown/issues/1595) [#1591](https://github.com/readmeio/markdown/issues/1591) [#1587](https://github.com/readmeio/markdown/issues/1587) [#1586](https://github.com/readmeio/markdown/issues/1586) * **deps:** bump dset from 3.1.3 to 3.1.4 ([#1018](#1018)) ([be480d7](be480d7)) * **deps:** bump github/codeql-action from 2 to 3 ([#847](#847)) ([4b97fe6](4b97fe6)), closes [#2203](https://github.com/readmeio/markdown/issues/2203) [#2195](https://github.com/readmeio/markdown/issues/2195) [#2185](https://github.com/readmeio/markdown/issues/2185) [#2156](https://github.com/readmeio/markdown/issues/2156) [#2151](https://github.com/readmeio/markdown/issues/2151) [#2147](https://github.com/readmeio/markdown/issues/2147) [#2141](https://github.com/readmeio/markdown/issues/2141) [#2124](https://github.com/readmeio/markdown/issues/2124) [#2121](https://github.com/readmeio/markdown/issues/2121) [/github.com/github/codeql-action/blob/main/#3230---08-jan-2024](https://github.com/readmeio//github.com/github/codeql-action/blob/main//issues/3230---08-jan-2024) [#2106](https://github.com/readmeio/markdown/issues/2106) [#2083](https://github.com/readmeio/markdown/issues/2083) [#2096](https://github.com/readmeio/markdown/issues/2096) [#2073](https://github.com/readmeio/markdown/issues/2073) [#2079](https://github.com/readmeio/markdown/issues/2079) [#2200](https://github.com/readmeio/markdown/issues/2200) [#2198](https://github.com/readmeio/markdown/issues/2198) [#2197](https://github.com/readmeio/markdown/issues/2197) * **deps-dev:** bump jest-puppeteer from 8.0.6 to 10.1.4 ([#1016](#1016)) ([33aa434](33aa434)), closes [#604](#604) [#602](#602) [#599](#599) [#586](#586) [#578](#578) [#576](#576) [#605](#605) * **deps:** bump micromatch from 4.0.5 to 4.0.8 ([#1019](#1019)) ([7be267e](7be267e)), closes [#266](#266) * **deps-dev:** bump mini-css-extract-plugin from 2.8.1 to 2.9.1 ([#1017](#1017)) ([9fb25da](9fb25da)) * **deps-dev:** bump semantic-release from 22.0.12 to 24.2.0 ([#1011](#1011)) ([443c843](443c843)), closes [#3462](https://github.com/readmeio/markdown/issues/3462) [#3460](https://github.com/readmeio/markdown/issues/3460) [#3436](https://github.com/readmeio/markdown/issues/3436) [#3423](https://github.com/readmeio/markdown/issues/3423) [#3462](https://github.com/readmeio/markdown/issues/3462) [#3489](https://github.com/readmeio/markdown/issues/3489) [#3488](https://github.com/readmeio/markdown/issues/3488) [#3486](https://github.com/readmeio/markdown/issues/3486) [#3487](https://github.com/readmeio/markdown/issues/3487) [#3485](https://github.com/readmeio/markdown/issues/3485) [#3483](https://github.com/readmeio/markdown/issues/3483) [#3450](https://github.com/readmeio/markdown/issues/3450) [#3481](https://github.com/readmeio/markdown/issues/3481) [#3424](https://github.com/readmeio/markdown/issues/3424) * **deps-dev:** bump webpack from 5.91.0 to 5.95.0 ([#1014](#1014)) ([b9c8dd2](b9c8dd2)) * update conventional commits ([50a4be9](50a4be9)) * upgrade conventionalcommits ([3646513](3646513)) <!--SKIP CI-->
Relevant Issues:
tl;dr adds base64 encodings of the CRC64 and MD5 to the headers of a chunk upload:
Recently we've been running into issues when artifacts are getting corrupted during the upload process. This is extremely rare (seen < 1% during my tests) but happens nonetheless. There still needs to be more investigation as to why these are being corrupted, but we've narrowed the problem areas to be between the upload and file container service handler after isolated testing with md5 checksum headers.
This PR adds a CRC64 and MD5 checksum to the header. Since NodeJS's crypto library (openssl bindings) doesn't have CRC64, I added a simple implementation based on Go's
hash/crc64
pkg. Also this is tailored to the CRC polynomial used by azure storage. We're already using this same polynomial places deeper in the stack.I ran some benchmarks and this CRC64 implementation is a bit faster than sha256/md5: