Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a way to pass remote settings to commands like add/run #3441

Closed
pared opened this issue Mar 5, 2020 · 6 comments
Closed

Implement a way to pass remote settings to commands like add/run #3441

pared opened this issue Mar 5, 2020 · 6 comments
Labels
discussion requires active participation to reach a conclusion question I have a question?

Comments

@pared
Copy link
Contributor

pared commented Mar 5, 2020

Context:
https://discordapp.com/channels/485586884165107732/563406153334128681/684696237810647051

When initializing the output here, (during dvc add for example) we cannot specify remote. It is fine as long as we work with aws buckets.

In this case, user has MinIO set up locally.

  1. There is s3 bucket s3://cache in endpoint-url: localhost:9000.
    Cache set up can be achieved with dvc remote modify myremote endpointurl

  2. User has another bucket that contains data, lets say s3://data/file also on this MinIO.

  3. User cannot dvc add s3://data/file because there is no way to specify that we want dvc to handle bucket under our own endpoint-url.

I tried tinkering with aws cli tool, hoping there will be some way to set up default endpoint and stumbled upon this issue:
aws/aws-cli#1270

there is some workaround, but I wonder whether it would not be better for us to provide a way to handle it.

@triage-new-issues triage-new-issues bot added the triage Needs to be triaged label Mar 5, 2020
@pared pared added the discussion requires active participation to reach a conclusion label Mar 5, 2020
@triage-new-issues triage-new-issues bot removed the triage Needs to be triaged label Mar 5, 2020
@pared pared added question I have a question? triage Needs to be triaged labels Mar 5, 2020
@triage-new-issues triage-new-issues bot removed the triage Needs to be triaged label Mar 5, 2020
@efiop
Copy link
Contributor

efiop commented Mar 5, 2020

@pared How about dvc remote add mys3 ... + dvc remote modify + dvc add remote://path?

@efiop
Copy link
Contributor

efiop commented Mar 5, 2020

Closing in favor of iterative/dvc.org#108

@efiop efiop closed this as completed Mar 5, 2020
@efiop
Copy link
Contributor

efiop commented Mar 5, 2020

For the record:
if user has

[cache]
    s3 = s3cache
[core]
    remote = logs
['remote "s3cache"']
    url = s3://dvc-storage
    endpointurl = http://localhost:9000/
    use_ssl = False
['remote "logs"']
    url = s3://logs
    endpointurl = http://localhost:9000/
    use_ssl = False

then the command should be dvc add remote://logs/shapes_128_128_20180914T1515

@pared
Copy link
Contributor Author

pared commented Mar 5, 2020

Fr the record, setting up remote seems to solve the problem, though we have another one:
When I set up MinIO locally with 2 buckets : dvc-cache and data, where data contains file and try to run:

#!/bin/bash

rm -rf repo
mkdir repo

pushd repo
git init --quiet
dvc init -q

export AWS_ACCESS_KEY_ID="minioadmin"
export AWS_SECRET_ACCESS_KEY="minioadmin"  

dvc remote add s3cache s3://dvc-cache/cache
dvc config cache.s3 s3cache
dvc remote modify s3cache endpointurl http://localhost:9000
dvc remote modify s3cache use_ssl False

dvc remote add miniodata s3://data
dvc remote modify miniodata endpointurl http://localhost:9000
dvc remote modify miniodata use_ssl False

dvc add remote://miniodata/file

adding fails with:

ERROR: ETag mismatch detected when copying file to cache! (expected:
 '4e102ec8d6ab714aae04d9e3c7b4c190-1', actual: 'ca9e5ed43f3fbee6edec
bb5ac6fba77e-1')

@efiop
Copy link
Contributor

efiop commented Mar 5, 2020

@pared Should we create a separate issue for that?

@pared
Copy link
Contributor Author

pared commented Mar 6, 2020

@efiop I believe this problem has been already noticed by @MrOutis:
#2629 (comment)
But since #2629 is about default MinIO ETag, i will create issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion requires active participation to reach a conclusion question I have a question?
Projects
None yet
Development

No branches or pull requests

2 participants