push/pull/fetch: raise an error if download/upload failed

[mhham]

Currently, a simple error message is showed (using gs):

ERROR: failed to download '....' to '....' - ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

This is problematic when pulling some data while building a docker image that relies on this data.
Instead a failing download should raise an error.

[efiop]

Hi @mhham !

Thanks for reporting this issue! The original intension was to just not make it fatal if some download fails, but that approach is clearly faulty. We need to download everything we can and then error-out, so dvc pull doesn't report "success".

[efiop]

• refactor download/upload methods to unify them
• raise an error if download/upload failed, but download/upload what we can

[peper0]

Is it really fixed? I've got version 0.59.0 and for such a command:

dvc pull -r nonexisting_remote && echo "everything is ok"

I get:

ERROR: failed to pull data from the cloud - config file error: unable to find remote section 'nonexisting_remote'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
everything is ok

I expected not to see everything is ok. Am I missing something?

[efiop]

@peper0 I am unable to reproduce:

$ dvc pull -r nonexisting_remote && echo "everything is ok"

ERROR: failed to pull data from the cloud - config file error: unable to find remote section 'nonexisting_remote'
                                                                                                                 
Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!                         

[peper0]

More detailed info:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.2 LTS
Release:	18.04
Codename:	bionic

$ dvc --version
0.59.0

$ md5sum $( which dvc )
6b9c0c77ece2dbbb3c4890c827e00078  /usr/bin/dvc

$ dvc pull -r nonexisting_remote
ERROR: failed to pull data from the cloud - config file error: unable to find remote section 'nonexisting_remote'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!

$ echo $?
0

On my friend's machine with similar config the problem is reproducible.

[peper0]

It can be easily reproduced in docker. Dockerfile:

FROM ubuntu:18.04

RUN apt update
RUN apt install -y wget
RUN wget https://dvc.org/deb/dvc.list -O /etc/apt/sources.list.d/dvc.list
RUN apt update 
RUN apt install -y dvc
WORKDIR /work
RUN dvc init --no-scm
CMD bash -c 'dvc pull -r bleble && echo "command succesfull, but should not be"'

$ docker build . -t dvc-bug

$ docker run -it dvc-bug 
ERROR: failed to pull data from the cloud - config file error: unable to find remote section 'bleble'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
command succesfull, but should not be