-
Notifications
You must be signed in to change notification settings - Fork 239
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
command: retry upload on s3.NoSuchUpload #470
Merged
Merged
Commits on Jul 21, 2022
-
command: retry upload on NoSuchUploadError
With this change, doUpload method will retry the multipart upload on NoSuchUploadError if the "retry-on-no-such-upload" flag is used and the remote file has not been modified after the beginning of the execution (or does not exists at all) . Otherwise (if it has been modified) it will assume that upload was successfull, and ignore the error. Beware that, this workaround approach has some drawbacks to consider: 1) If upload really failed but a third person succesfuly modified the remote file (dsturl), then this code will incorrectly return "success". 2) It would be better to use checksum rather than the last modification date but, unfortunately it is not supported in aws-sdk-go v1.*, and it is a bit tricky to use ETag's as a replacement of checksum for the multipart uploads. Refer https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums see also the following as an example of using ETag as checksum https://github.com/peak/s3hash Updates #450
Configuration menu - View commit details
-
Copy full SHA for 1fca6bb - Browse repository at this point
Copy the full SHA 1fca6bbView commit details
Commits on Jul 22, 2022
-
move retry (for NoSuchUpload) logic to (s3).Put method
Write simple test. The tests does not check if the logic works as expected for when the varios last modification dates of remote. All the test assumes that there is no (such) object in remote (since the response is cleared & replaced with NoSuchUpload error).
Configuration menu - View commit details
-
Copy full SHA for 1a058fc - Browse repository at this point
Copy the full SHA 1a058fcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 64d7ce7 - Browse repository at this point
Copy the full SHA 64d7ce7View commit details
Commits on Jul 25, 2022
-
Configuration menu - View commit details
-
Copy full SHA for d4352f1 - Browse repository at this point
Copy the full SHA d4352f1View commit details
Commits on Jul 27, 2022
-
Remove unused "StartTime" variable. Change flag name to "no-such-upload-retry-count" from "retry-on-no-such-upload". Also change corresponding fields' name to [N]oSuchUploadRetryCount. Move code generation to an external method.
Configuration menu - View commit details
-
Copy full SHA for 5a9f226 - Browse repository at this point
Copy the full SHA 5a9f226View commit details -
Configuration menu - View commit details
-
Copy full SHA for eea85f0 - Browse repository at this point
Copy the full SHA eea85f0View commit details -
use "Debug" level to log retry attemp
It will use debug level to log retry attemp. Debug level is also the level. that s3.customRetryer write logs, see also https://github.com/peak/s5cmd/blob/822043839c0e581f9b481c9ef30dd88a04463a09/storage/s3.go#L932-L934
Configuration menu - View commit details
-
Copy full SHA for c9364ee - Browse repository at this point
Copy the full SHA c9364eeView commit details -
Configuration menu - View commit details
-
Copy full SHA for ce25daf - Browse repository at this point
Copy the full SHA ce25dafView commit details
Commits on Jul 28, 2022
-
Configuration menu - View commit details
-
Copy full SHA for db73dbf - Browse repository at this point
Copy the full SHA db73dbfView commit details -
Configuration menu - View commit details
-
Copy full SHA for a74a470 - Browse repository at this point
Copy the full SHA a74a470View commit details -
Configuration menu - View commit details
-
Copy full SHA for 824ca84 - Browse repository at this point
Copy the full SHA 824ca84View commit details -
Configuration menu - View commit details
-
Copy full SHA for 58618f3 - Browse repository at this point
Copy the full SHA 58618f3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 569ce13 - Browse repository at this point
Copy the full SHA 569ce13View commit details
Commits on Aug 2, 2022
-
Configuration menu - View commit details
-
Copy full SHA for aa0d9e1 - Browse repository at this point
Copy the full SHA aa0d9e1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9770a80 - Browse repository at this point
Copy the full SHA 9770a80View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1d68c6f - Browse repository at this point
Copy the full SHA 1d68c6fView commit details -
Configuration menu - View commit details
-
Copy full SHA for f356166 - Browse repository at this point
Copy the full SHA f356166View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8fef88e - Browse repository at this point
Copy the full SHA 8fef88eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9936674 - Browse repository at this point
Copy the full SHA 9936674View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9f12292 - Browse repository at this point
Copy the full SHA 9f12292View commit details -
Configuration menu - View commit details
-
Copy full SHA for e62b475 - Browse repository at this point
Copy the full SHA e62b475View commit details -
Configuration menu - View commit details
-
Copy full SHA for 41a516e - Browse repository at this point
Copy the full SHA 41a516eView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.