Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Checksum at object stores #388

Merged
merged 21 commits into from
Jun 14, 2022
Merged

Checksum at object stores #388

merged 21 commits into from
Jun 14, 2022

Commits on Jun 8, 2022

  1. Add checksum generation on object store download

    @parasj
    parasj committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    b3564f1 View commit details
    Browse the repository at this point in the history

  2. Check MD5 on S3 upload

    @parasj
    parasj committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    7beddfc View commit details
    Browse the repository at this point in the history

  3. Add upload checksum check for Azure

    @parasj
    parasj committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    2515094 View commit details
    Browse the repository at this point in the history

  4. Update chunk requests with checksums, then check when uploading

    @parasj
    parasj committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    81c854d View commit details
    Browse the repository at this point in the history

  5. Check whole object upload MD5 hashes

    @parasj
    parasj committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    f77ee5c View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2022

  1. Works with checksum off

    @parasj
    parasj committed Jun 9, 2022
    Configuration menu
    Copy the full SHA
    8713f9c View commit details
    Browse the repository at this point in the history

  2. Enable upload md5 check

    @parasj
    parasj committed Jun 9, 2022
    Configuration menu
    Copy the full SHA
    e17807a View commit details
    Browse the repository at this point in the history

  3. Cleanup

    @parasj
    parasj committed Jun 9, 2022
    Configuration menu
    Copy the full SHA
    0d795da View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2022

  1. Merge branch 'main' into dev/checksum

    @parasj
    parasj committed Jun 10, 2022
    Configuration menu
    Copy the full SHA
    83ae1a5 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2022

  1. Update badge

    @parasj
    parasj committed Jun 11, 2022
    Configuration menu
    Copy the full SHA
    ae68241 View commit details
    Browse the repository at this point in the history

  2. Multipart uploads / downloads for GCP + service accounts for GCP (#360) 

    * adding service acounts for gcp

* gcp xml api for multipart uploads

Co-authored-by: Paras Jain <[email protected]>
    @sarahwooders @parasj
    sarahwooders and parasj committed Jun 11, 2022
    Configuration menu
    Copy the full SHA
    6088471 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into dev/checksum

    @parasj
    parasj authored Jun 11, 2022
    Configuration menu
    Copy the full SHA
    c4038a1 View commit details
    Browse the repository at this point in the history

  4. Add multipart checksums

    @parasj
    parasj committed Jun 11, 2022
    Configuration menu
    Copy the full SHA
    4c68d07 View commit details
    Browse the repository at this point in the history

  5. pytype fix

    @parasj
    parasj committed Jun 11, 2022
    Configuration menu
    Copy the full SHA
    9984a55 View commit details
    Browse the repository at this point in the history

  6. Working checksums for multipart GCP transfers

    @parasj
    parasj committed Jun 11, 2022
    Configuration menu
    Copy the full SHA
    2d03f84 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2022

  1. Cleanup

    @parasj
    parasj committed Jun 12, 2022
    Configuration menu
    Copy the full SHA
    ebb1b8a View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2022

  1. No indent

    @parasj
    parasj committed Jun 14, 2022
    Configuration menu
    Copy the full SHA
    ac93b82 View commit details
    Browse the repository at this point in the history

  2. Delete existing service account keys (#411)

    @parasj
    parasj committed Jun 14, 2022
    Configuration menu
    Copy the full SHA
    c1ed80a View commit details
    Browse the repository at this point in the history

  3. Fix issue 368 (#410) 

    * Refactored src dest querying code into helper method

* Sync design implemented

* Sync bug fixed + redesign implemented

* Style and typing fixes

* typing issue:

* Style formatting

* Added TransferObjectList dataclass

* Minor further refactor of TransferObjectList, add new replicate_helper_random function

* Further clean up

* Move TransferObjectList

* Consolidate replicate_helper_cp and replicate_helper_random

* Inline replicate helper

Co-authored-by: Paras Jain <[email protected]>
    @antonzabreyko @parasj
    antonzabreyko and parasj committed Jun 14, 2022
    Configuration menu
    Copy the full SHA
    7ccc25e View commit details
    Browse the repository at this point in the history

  4. Migrate internal CLI to new API (#415)

    @parasj
    parasj committed Jun 14, 2022
    Configuration menu
    Copy the full SHA
    8b164ca View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into dev/checksum

    @parasj
    parasj authored Jun 14, 2022
    Configuration menu
    Copy the full SHA
    eb27488 View commit details
    Browse the repository at this point in the history