Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

command: retry upload on s3.NoSuchUpload #470

Merged
merged 22 commits into from
Aug 2, 2022
Merged

command: retry upload on s3.NoSuchUpload #470

merged 22 commits into from
Aug 2, 2022

Commits on Jul 21, 2022

  1. command: retry upload on NoSuchUploadError

    With this change, doUpload method will retry the multipart upload on NoSuchUploadError if the "retry-on-no-such-upload" flag is used and the remote file has not been modified after the beginning of the execution (or does not exists at all) . Otherwise (if it has been modified) it will assume that upload was successfull, and ignore the error.
    
    Beware that, this workaround approach has some drawbacks to consider:
    1) If upload really failed but a third person succesfuly modified the remote file (dsturl), then this code will incorrectly return "success".
    	2) It would be better to use checksum rather than the last modification date but, unfortunately it is not supported in aws-sdk-go v1.*, and it is a bit tricky to use ETag's as a replacement of checksum for the multipart uploads. Refer https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums
    	 see also the following as an example of using ETag as checksum https://github.com/peak/s3hash
    
    Updates #450
    kucukaslan committed Jul 21, 2022
    Configuration menu
    Copy the full SHA
    1fca6bb View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2022

  1. move retry (for NoSuchUpload) logic to (s3).Put method

    Write simple test.
    
    The tests does not check if the logic works as expected for when the varios last modification dates of remote. All the test assumes that there is no (such) object in remote (since the response is cleared & replaced with NoSuchUpload error).
    kucukaslan committed Jul 22, 2022
    Configuration menu
    Copy the full SHA
    1a058fc View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    64d7ce7 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2022

  1. Configuration menu
    Copy the full SHA
    d4352f1 View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2022

  1. update according to review

    Remove unused "StartTime" variable.
    Change flag name to  "no-such-upload-retry-count" from "retry-on-no-such-upload". Also change corresponding fields' name to [N]oSuchUploadRetryCount.
    Move code generation to an external method.
    kucukaslan committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    5a9f226 View commit details
    Browse the repository at this point in the history
  2. update usage information

    ... even though it is a hidden flag.
    kucukaslan committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    eea85f0 View commit details
    Browse the repository at this point in the history
  3. use "Debug" level to log retry attemp

    It will use debug level to log retry attemp.  Debug level is also the level. that s3.customRetryer write logs, see also
    https://github.com/peak/s5cmd/blob/822043839c0e581f9b481c9ef30dd88a04463a09/storage/s3.go#L932-L934
    kucukaslan committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    c9364ee View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    ce25daf View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2022

  1. update according to review

    kucukaslan committed Jul 28, 2022
    Configuration menu
    Copy the full SHA
    db73dbf View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a74a470 View commit details
    Browse the repository at this point in the history
  3. add e2e test

    kucukaslan committed Jul 28, 2022
    Configuration menu
    Copy the full SHA
    824ca84 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    58618f3 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    569ce13 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2022

  1. Configuration menu
    Copy the full SHA
    aa0d9e1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9770a80 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1d68c6f View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    f356166 View commit details
    Browse the repository at this point in the history
  5. rename code to id

    kucukaslan committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    8fef88e View commit details
    Browse the repository at this point in the history
  6. rename to *ID from *Id

    kucukaslan committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    9936674 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    9f12292 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    e62b475 View commit details
    Browse the repository at this point in the history
  9. rename id to ID

    kucukaslan committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    41a516e View commit details
    Browse the repository at this point in the history