Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix upload over 2 gb #524

Merged
merged 20 commits into from
Feb 23, 2024

Conversation

cosa65
Copy link
Member

@cosa65 cosa65 commented Feb 20, 2024

Description

Details

URL to issue

Link to staging deployment URL (or set N/A)

N/A

Links to any PRs or resources related to this PR

Integration test branch

master

Merge checklist

Your changes will be ready for merging after all of the steps below have been completed.

Code updates

Have best practices and ongoing refactors being observed in this PR

  • Migrated any selector / reducer used to the new format.
  • All new dependency licenses have been checked for compatibility

Manual/unit testing

  • Tested changes using InfraMock locally or no tests required for change, e.g. Kubernetes chart updates.
  • Validated that current unit tests for code work as expected and are sufficient for code coverage or no unit tests required for change, e.g. documentation update.
  • Unit tests written or no unit tests required for change, e.g. documentation update.

Integration testing

You must check the box below to run integration tests on the latest commit on your PR branch.
Integration tests have to pass before the PR can be merged. Without checking the box, your PR
will not pass the required status checks for merging.

  • Started end-to-end tests on the latest commit.

Documentation updates

  • Relevant Github READMEs updated or no GitHub README updates required.
  • Relevant Wiki pages created/updated or no Wiki updates required.

Optional

  • Staging environment is unstaged before merging.
  • Photo of a cute animal attached to this PR.

Signed-off-by: cosa65 <[email protected]>
… don't require it anymore as a result of the removal

Signed-off-by: cosa65 <[email protected]>
Signed-off-by: cosa65 <[email protected]>
Signed-off-by: cosa65 <[email protected]>
Comment on lines 46 to 56
const promises = [];
const parts = Math.ceil(size / FILE_CHUNK_SIZE);

for (let i = 0; i < parts; i += 1) {
promises.push(
s3.getSignedUrlPromise('uploadPart', {
...baseParams,
PartNumber: i + 1,
}),
);
}
Copy link
Member Author

@cosa65 cosa65 Feb 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now we don't try to predict how many parts we'll send, we just send them whenever the read/compress stream makes them available and request a signed url for each of them. This request is managed with the new endpoint in uploadController, which calls the new function getPartUploadSignedUrl

@cosa65 cosa65 added the safe to run Marks the PR as safe to run checks on. IMPORTANT only add if from a trusted source. label Feb 20, 2024
@cosa65 cosa65 changed the title [DRAFT] Fix upload over 2 gb Fix upload over 2 gb Feb 21, 2024
Copy link

codecov bot commented Feb 22, 2024

Codecov Report

Attention: Patch coverage is 78.78788% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 89.90%. Comparing base (31d3016) to head (4e30768).

Files Patch % Lines
src/api.v2/helpers/s3/signedUrl.js 46.15% 4 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #524      +/-   ##
==========================================
- Coverage   90.11%   89.90%   -0.22%     
==========================================
  Files         169      171       +2     
  Lines        3754     3765      +11     
  Branches      429      429              
==========================================
+ Hits         3383     3385       +2     
- Misses        348      354       +6     
- Partials       23       26       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@cosa65 cosa65 merged commit abe83eb into hms-dbmi-cellenics:master Feb 23, 2024
@cosa65 cosa65 deleted the fix-upload-over-2-gb branch February 23, 2024 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
safe to run Marks the PR as safe to run checks on. IMPORTANT only add if from a trusted source.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants