External dependencies are shared/global state

[somewacko]

Another conversation from Discord: This page describes how to set up external dependencies, where stages can write to and read from remote objects.

Could this cause problems for projects with multiple users in the same repo? For example, say two teammates have different versions of the codebase checked out, and both run the same dvc repro command at the same time that produces and reads from an external dependency -- wouldn't this cause a race condition where one user might end up accessing the other's object? (or worse, the wrong object gets cached?) AFAIK there's not locking mechanism for external objects.

If so, this should probably be noted on the docs (happy to volunteer!) But this also poses a hairy problem for projects with external dependencies since these objects are always volatile. How should this sort of thing be handled?

[efiop]

@zo7 Correct, in those cases where dependencies might change, we recommend users having external workspaces defined. For example, you could do something like

dvc remote add myworkspace s3://bucket/$USER --local # notice the --local, which writes to .dvc/config.local which is not tracked by git
dvc run -d remote://myworkspace/data ...

this way your workspace is dynamic and depends on the user, so there are no race conditions between different users(unless they are using more than 1 repo instance at the same time, of course).

If so, this should probably be noted on the docs (happy to volunteer!)

We would really appreciate that :) The docs repo is https://github.com/iterative/dvc.org, let us know if you need any help.

[ghost]

Just to clarify, it would be the same cache but different remotes ^

[somewacko]

Oh cool! I didn't realize you could set a local workspace like that.

Would that make sense to suggest setting up local workspaces in the docs? Would this have to be a step that users have to do manually when setting up the repo before they can dvc repro? (i.e. git clone -> dvc pull -> dvc remote)

unless they are using more than 1 repo instance at the same time

How do you think this should work for a server or CI system that calls dvc repro on multiple branches all at one? Maybe set up a local workspace based on the git SHA?

[efiop]

@zo7

Would that make sense to suggest setting up local workspaces in the docs? Would this have to be a step that users have to do manually when setting up the repo before they can dvc repro? (i.e. git clone -> dvc pull -> dvc remote)

Yes, it would make total sense. I guess it would be suitable to add somewhere around https://dvc.org/doc/user-guide/external-outputs . It seems like it would suit as a part of iterative/dvc.org#108 Maybe you would even consider contributing a doc based on your experience? 😉

How do you think this should work for a server or CI system that calls dvc repro on multiple branches all at one? Maybe set up a local workspace based on the git SHA?

Yes, that would totally work 🙂

[efiop]

Closing. Docs ticket is iterative/dvc.org#108