Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(worker): Prefer the s3-PUBLIC remote when found #3146

Merged
merged 2 commits into from
Sep 10, 2024
Merged

fix(worker): Prefer the s3-PUBLIC remote when found #3146

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3f134c1
fix(worker): Prefer the s3-PUBLIC remote when found
nellh Sep 9, 2024
5622b6f
tests(worker): Update test_parse_remote_line with new format includin…
nellh Sep 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion services/datalad/datalad_service/common/annex.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@
if remoteConfig['type'] == 'S3' and remoteConfig['bucket'] in S3_BUCKETS_WHITELIST:
remoteUuid = remoteLine[0:36]
remoteUrl = remoteConfig['publicurl'] if 'publicurl' in remoteConfig else None
return {'uuid': remoteUuid, 'url': remoteUrl}
return {'uuid': remoteUuid, 'url': remoteUrl, 'name': remoteConfig['name']}


def parse_rmet_line(remote, rmetLine):
Expand Down Expand Up @@ -174,6 +174,10 @@
matched_remote = parse_remote_line(line)
if matched_remote:
remote = matched_remote
# Prefer the s3-PUBLIC remote if we find one
# TODO - Identify the best remote dynamically
if matched_remote['name'] == 's3-PUBLIC':
break

Check warning on line 180 in services/datalad/datalad_service/common/annex.py

View check run for this annotation

Codecov / codecov/patch

services/datalad/datalad_service/common/annex.py#L179-L180 

Added lines #L179 - L180 were not covered by tests
# Check if we found a useful external remote
if remote:
# Read the rest of the files.
Expand Down
3 changes: 2 additions & 1 deletion services/datalad/tests/test_annex.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,8 @@ def test_compute_rmet_sha256_annex():
def test_parse_remote_line():
remote = parse_remote_line("""57894849-d0c8-4c62-8418-3627be18a196 autoenable=true bucket=openneuro.org datacenter=US encryption=none exporttree=yes fileprefix=ds002778/ host=s3.amazonaws.com name=s3-PUBLIC partsize=1GiB port=80 public=yes publicurl=http://openneuro.org.s3.amazonaws.com/ storageclass=STANDARD type=S3 versioning=yes timestamp=1588743361.538097946s""")
assert remote == {'url': 'http://openneuro.org.s3.amazonaws.com/',
'uuid': '57894849-d0c8-4c62-8418-3627be18a196'}
'uuid': '57894849-d0c8-4c62-8418-3627be18a196',
'name': 's3-PUBLIC'}


def test_parse_rmet_line():
Expand Down
Loading