Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Implement sliced downloads in GSClient #389

Merged

Conversation

joconnor-ecaa
Copy link
Contributor

@joconnor-ecaa joconnor-ecaa commented Dec 21, 2023

Implements sliced object downloads in GSClient using google.cloud.storage.transfer_manager as documented here.

Closes #387


Contributor checklist:

  • I have read and understood CONTRIBUTING.md
  • Confirmed an issue exists for the PR, and the text Closes #issue appears in the PR summary (e.g., Closes #123).
  • Confirmed PR is rebased onto the latest base
  • Confirmed failure before change and success after change
  • Any generic new functionality is replicated across cloud providers if necessary
  • Tested manually against live server backend for at least one provider
  • Added tests for any new functionality
  • Linting passes locally
  • Tests pass locally
  • Updated HISTORY.md with the issue that is addressed and the PR you are submitting. If the top section is not `## UNRELEASED``, then you need to add a new section to the top of the document for your change.

@joconnor-ecaa joconnor-ecaa changed the title feat: Implement sliced downloads in GSClient WIP: Implement sliced downloads in GSClient Dec 21, 2023
@joconnor-ecaa joconnor-ecaa marked this pull request as ready for review December 29, 2023 09:02
Copy link

codecov bot commented Dec 29, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (41aa53d) 93.8% compared to head (f467b56) 93.5%.
Report is 3 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff            @@
##           master    #389     +/-   ##
========================================
- Coverage    93.8%   93.5%   -0.4%     
========================================
  Files          22      22             
  Lines        1566    1570      +4     
========================================
- Hits         1470    1468      -2     
- Misses         96     102      +6     
Files Coverage Δ
cloudpathlib/gs/gsclient.py 93.1% <100.0%> (-1.4%) ⬇️

... and 2 files with indirect coverage changes

@pjbull pjbull changed the base branch from master to 389-live-tests December 29, 2023 17:34
@pjbull pjbull merged commit f65de44 into drivendataorg:389-live-tests Dec 29, 2023
25 of 27 checks passed
@pjbull
Copy link
Member

pjbull commented Dec 29, 2023

Merged into 389-live-tests to open PR #391 that runs the live tests.

pjbull added a commit that referenced this pull request Dec 29, 2023
* WIP: Implement sliced downloads in GSClient (#389)

* feat: Implement sliced downloads in GSClient

* fix: remove unintended import changes

* Mock transfer_manager. Test both worker types.

* Update HISTORY.md

---------

Co-authored-by: Joe O'Connor <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GSPath: Implement sliced object downloads
2 participants