DVC New Cloud-supported Google Drive Remote

[nickdelgrosso]

This pull request is about adding monitoring and usage controls to the dvc remote, so we don't hit the same "usage limit" error we encountered yesterday. Following the directions on the DVC documentation here, the dvc database has been migrated to a new remote. Extra benefits:

• A special read-only "service account" has been created for travis, that doesn't collide with the normal developer login (we had read-only before, but through a different mechanism).
• No fiddling with Travis encryption.
• Pull requests across repos should work (we can run tests on community submissions now)
• Usage monitoring and raising if needed. Shouldn't be a problem, but if we start hitting limits we'll understand why.

To use it, I recommend everyone update dvc (pip install --upgrade dvc) to the latest version (they had a 1.0 release recently) and delete the json file so google asks to re-authenticate on the next dvc pull. From there it should work as before.

[chriski777]

This is awesome! Will community developers still be able to call dvc pull without authenticating?

[nickdelgrosso]

Yes, if they pull from the travis repo: dvc pull -r gdrive-travis

[nickdelgrosso]

Two more commits to fix some quality-of-life issues:

• I just did a little more digging through the monitor and found the source of the "usage exceeded" error: DVC was making way too many calls at once for google drive. The DVC devs reccomend just tracking a whole folder when big data directories are involved, in order to have fewer calls and to more-efficiently do downloads and uploads.
• Adding new files for new regression tests involves git-adding a bunch of dvc files at once.
• Also, every time we changed branches, pull/pushed, or commited, DVC would add a "/tmp" line to its .gitignore file. Very annoying to stash it every time we want to make a change! This is a well-known issue for certain repos that the devs weren't able to track down the source of, but have somehow fixed for new repos. They recommend just re-initializing the repo with the latest DVC version.

To fix this, I re-initialized the DVC repo. You'll notice that there's now only one .dvc file (which tracks the data/testdata folder as a whole), and hopefully that that .gitignore problem is gone!

[nickdelgrosso]

Okay now the .gitignore:/tmp issue is fixed. Seems to be because a credentials file was committed to the .dvc/tmp directory. Moving it to .dvc/cred stopped the .gitignore:/tmp behavior

[carsen-stringer]

lol probably my file :D I was being lazy with the stashing of the dvc folder -- thanks nick this is merged! I will make the dvc pull -r gdrive-travis the default instructions in the readme