cases: DVC in Production

[dmpetrov]

Inspired by iterative/dvc#2212 and https://www.reddit.com//r/MachineLearning/comments/ea0825/d_experiment_management_in_kubernetes/

We don't have any explanation on how to work in "production":

1. How to bring an artifact when CI/CD is done? GitLab build materials? Temporary bucket/storage and custom script?
2. Best practices of sharing cache in k8s.

[jorgeorpinel]

@dmpetrov I read both context links. Here are some answers, please confirm if this makes sense:

How to bring an artifact when CI/CD is done? GitLab build materials?

The recommended way to sync pipeline status updates back from production to the master DVC repo would be to git commit/push (in a release branch) from prod. It can be part of the GH/GL CI job.

Question: what happens with multiple prod envs, as git pull/merge --ours could be needed, which might result in a disconnected/meaningless commit history. Does each env get a branch?

Temporary bucket/storage and custom script?

Why not remote storage? That's DVC's feature for syncing caches.

One trick would be to setup the dev env's remote storage = the prod env cache, this way there's no 3rd data store (if no backups are needed and to reduce transfers).

Best practices of sharing cache in k8s.

Unfortunately I'm not super familiar but can't all the containers be setup to mount a shared drive? Then just https://dvc.org/doc/use-cases/shared-development-server#configure-the-external-shared-cache

p.s. TBH these specific Q & As would be better for a Best Practice page IMO (rel #72). We can have both but we need to be a bit higher-level for the use case (currently discussing in #2490 (comment)).

[jorgeorpinel]

Closing in favor of #2544.