Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use multipart uploads for creating S3 objects #14382

Closed
wants to merge 1 commit into from
Closed

Use multipart uploads for creating S3 objects #14382

wants to merge 1 commit into from

Conversation

pierrebeaucamp
Copy link

This patch replaces the previous s3.PutObject call with an Uploader instance from the s3manager package.

This allows uploading S3 objects larger than 5GB (see bug report that this is closing).

I tested this locally, but did not run the acceptance test as I don't want to cover the cost for those.

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #788

Release note for CHANGELOG:

NONE

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccXXX'

...

This patch replaces the previous `s3.PutObject` call with an `Uploader`
instance from the `s3manager` package.

Fixes #788.
@pierrebeaucamp pierrebeaucamp requested a review from a team July 29, 2020 18:21
@ghost ghost added size/S Managed by automation to categorize the size of a PR. needs-triage Waiting for first response or review from a maintainer. service/s3 Issues and PRs that pertain to the s3 service. labels Jul 29, 2020
@leonfs
Copy link

leonfs commented Aug 16, 2021

Wow - this one would be gold! Is there an update on what's blocking it?

@breathingdust breathingdust added enhancement Requests to existing resources that expand the functionality or scope. and removed needs-triage Waiting for first response or review from a maintainer. labels Sep 20, 2021
@zhelding
Copy link
Contributor

Pull request #21306 has significantly refactored the AWS Provider codebase. As a result, most PRs opened prior to the refactor now have merge conflicts that must be resolved before proceeding.

Specifically, PR #21306 relocated the code for all AWS resources and data sources from a single aws directory to a large number of separate directories in internal/service, each corresponding to a particular AWS service. This separation of code has also allowed for us to simplify the names of underlying functions -- while still avoiding namespace collisions.

We recognize that many pull requests have been open for some time without yet being addressed by our maintainers. Therefore, we want to make it clear that resolving these conflicts in no way affects the prioritization of a particular pull request. Once a pull request has been prioritized for review, the necessary changes will be made by a maintainer -- either directly or in collaboration with the pull request author.

For a more complete description of this refactor, including examples of how old filepaths and function names correspond to their new counterparts: please refer to issue #20000.

For a quick guide on how to amend your pull request to resolve the merge conflicts resulting from this refactor and bring it in line with our new code patterns: please refer to our Service Package Refactor Pull Request Guide.

@github-actions
Copy link

github-actions bot commented Jun 9, 2022

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 9, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement Requests to existing resources that expand the functionality or scope. service/s3 Issues and PRs that pertain to the s3 service. size/S Managed by automation to categorize the size of a PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

EntityTooLarge: Your proposed upload exceeds the maximum allowed size
4 participants