how-to: spot env changes in pipelines

[aguschin]

The results of dvc repro could be different if we update libraries in our environment, but still dvc repro won't spot that. It would be great to have some suggested solutions for this in the docs, for example, in "User Guide > How to". For typical python setup the workaround would to add stage with pip freeze, something like this:

stages:
  check-requirements:
    cmd: pip freeze > requirements.txt
    outs:
    - requirements.txt:
        cache: false
  first-stage-in-pipeline:
    deps:
    - requirements.txt
    ...

It there are other good solutions, would be great to add them too.

[jorgeorpinel]

Related core issues: #3439, #2378.

The results of dvc repro could be different if we update libraries in our environment, but still dvc repro won't spot that...
the workaround would to add stage with pip freeze

No need for the check-requirements stage probably? Making requirements.txt a dependency should be enough typically (assuming its maintained manually).

Something else we've recommended in the past is to keep a version number in a code comment (and update it when imports change, which can be automated). That way repro will pick up that the code file has changed.

[jorgeorpinel]

p.s. are you planning to contribute a draft of this short how-to @aguschin? Feel free to.

I would only like to clarify when/which imports can be expected to cause results to change. Ideally only internal project imports, but not really external libraries, right? i.e. I'm not sure about recommending to depend on package config files (like requirements.txt).

[aguschin]

No need for the check-requirements stage probably? Making requirements.txt a dependency should be enough typically (assuming its maintained manually).

It's a good question, @jorgeorpinel ! For the minimal example, we could easily add requirements.txt dependency only, avoiding check-requirements stage. But as a practitioner, in most cases I would prefer adding the additional stage which will re-check if I remembered to run pip freeze > requirements.txt after updating the environment. The logic here would be the following: if I forgot to update requirements.txt and run git push, dvc push after some experiments, other people which will git pull, dvc pull my changes may fail trying reproduce my exact results (I mean, the dvc repro -f results would be different). It's similar to pre-commit hooks in a sense that I would prefer it to run every time before dvc repro. So it would kind of pre-repro hook :)

p.s. are you planning to contribute a draft of this short how-to @aguschin? Feel free to.

If it seems goods for the "how to", I can create a draft covering both cases with the explanation why we need the more advanced one.

I would only like to clarify when/which imports can be expected to cause results to change

You are correct, some libraries can't change the result usually (one obvious example would be pytest or pylint). But it's not usually very clear which updates can change the results of stages. It's not happens often overall, but I would say it could take place when you update some data-processing library or ml-models library. The event can be rare, but still can waste your time when you will try to figure out why results aren't reproducible. One way is to suggest the user to select subset of libraries, but that could be just an overhead in the sense you'll need to review it, keep updated and such. Sometimes we can limit the list of libraries which we depend on (for example, excluding --dev dependencies in Pipenv, Poetry) and we can provide a hint on that, but for simplicity I would still use requirements.txt as the main example.

I hope it is more clear now. Please provide me your thoughts on this :)

[jorgeorpinel]

I would prefer adding the additional stage which will re-check if I remembered to run pip freeze

We can def mention this as a tip, but I wouldn't have it as an "official" part of the how-to steps. Also DVC is lang-agnostic and that it's very Python-centric. Prob not every platform has a command to print a list of req paks.

similar to pre-commit hooks

So why not use a pre-commit hook? 🙂 Sorry, I'm just still not convinced that it's the intended use for DVC stages. It can def. be a nice trick though.

If it seems goods for the "how to", I can create a draft covering both cases

The 2 cases I see are:

a) Make the pipeline depend on a file containing the list of required packages and versions (with requirements.txt Python example). We can add the "pre-commit" stage trick in a hidden/expandable section too but it's the came solution.
b) Keep a version number or some other mark to signal DVC reproduction in code comments (code files should already be stage dependencies) which gets updated as needed (e.g. by a subset of libs like you mentioned). We can even link to some automated semver tools as an additional rec.

some libraries can't change the result usually... it could take place when you update some data-processing library or ml-models library. The event can be rare

TBH if this is very rare, more of an error or edge case, I wouldn't recommend that route (a) in our docs. WDYT @dberenbaum ?

(And in that case, having only (b), would the doc be too short to be worth it?)

Final though: this may be more of a Best Practice than a how-to. How tos are more like a FAQ — something users ask about regularly. I'd check with the core team first to get a better impression of how recurrent this Q is. They handle most support cases.

[jorgeorpinel]

p.s. Best Practices is a hypothetical new docs section we've been planning for years 😆 So if this is a BP, we can just add a checkbox to iterative/dvc.org#72 for now.

[dberenbaum]

I think it's a really useful suggestion that could help in some circumstances, but I also understand the hesitancy to introduce it as a definitive how-to.

A few questions that come to mind:

1. There are other languages to support, and even within Python, this might not be the best practice. For example, lots of people might prefer poetry, conda, etc. to manage environments. What suggested languages, package managers, etc. make sense to include?
2. Why not have the first stage do something like python3 -m venv .venv; source .venv/bin/activate; python -m pip install -r requirements.txt? Otherwise, it seems like users with slightly different environments (which can happen easily with pip freeze since the details can include things like local paths) are stuck having to reproduce the work of others all the time.
3. This suggests that DVC should handle the reproducibility of not only the project code, but also the environment in which it is run. I'm honestly not sure what position we should take on that. It makes sense that the environment should be tracked and reproducible, but how far does this go? What about the infrastructure/hardware used? That's probably a question that we will consider more as we spend more time integrating DVC and CML and considering remote execution of pipelines.

Despite those concerns, this suggestion doesn't need to be universally applicable to be useful. It's more about what methods make sense to mention, and what is the right way to frame them? I like the best practices suggestion, or maybe a blog post, but interested to hear what others think.

[aguschin]

@dberenbaum, thank you for good points on this!

1. Indeed, we can't cover every language and every environment manager in Python. But I would say, that within main tools in Python I know the concept still works - you just need to use different file (Pipenv.lock for Pipenv, poetry.lock for Poetry, environment.yml for Conda). That said, I think we can introduce this as a concept for python presumably extendable to some other languages, if this makes sense.
2. Good point! Do you mean installation with "pip install -e ." as something that introduces these local paths? If so, then local paths in requirements.txt already make code non-reproducible. I could go and change any code script in the package installed by "pip install -e" and no one will spot that. I would say, if someone uses "-e" flag in pip for any library except for one located in the same repo, then there is no sense in adding requirements.txt in the dependencies in dvc.yaml, and even more adding a stage like "check-requirements". On the other hand, if the package is located inside the repo then you are going to stage changes and commit them. The same happens in "dvc exp run" for example, which uses unstaged changes on deps to produce experiment and someone may forget to commit them later (we have a good reminder after "dvc exp run" to commit them though).
   And, if you meant another example which introduces local paths to requirements.txt, please describe it.
3. Agreed, the infrastructure and hardware also could change the results and this is a not a simple question what to do with it. Also I would say that (in my humble ds practice) that library versions have more frequent impact on reproducibility than a hardware. With that said, considering scenarios on how to fix the environment may be a little step towards the overall goal, covering large part of this topic.

@jorgeorpinel, thank you for the concerns! I think few of them are answered above, few more:

So why not use a pre-commit hook?

The pre-commit hook is used just before commit and thus don't solve the problem. Let's suppose I've updated environment, run several experiments and then stage a commit. I could write a hook that will at this points will update my requirements.txt (or whatever file we use), but my experiment which I ran locally, already associated with the previous commit which has outdated environment to this moment.

How tos are more like a FAQ — something users ask about regularly

In this case I don't think it is how-to, more like a practice which is useful if you want to prevent this issue (usually after few cases when you stumble upon this). For the blog post this also could be a little short - but I don't know what length a blog post is supposed to have.

Keep a version number or some other mark to signal DVC reproduction in code comments

Do we have any example on this? I couldn't find one, unfortunately. As of myself, I never used this kind of practice, so it's a bit hard to imagine the exact reasons why this could be preferable. It would be great to see one to understand how this practice can be applied.

[jorgeorpinel]

Do we have any example on this?

"""My beautiful script

Version 0.1.7  # Bump patch no. if dependencies were updated.

...
"""

import ...

[dberenbaum]

2\. Why not have the first stage do something like `python3 -m venv .venv; source .venv/bin/activate; python -m pip install -r requirements.txt`? Otherwise, it seems like users with slightly different environments (which can happen easily with `pip freeze` since the details can include things like local paths) are stuck having to reproduce the work of others all the time.

What I meant by this is that if you want a stage to handle dependencies, why not have that stage create the desired environment instead of only report the environment? That way, the environment becomes not only a versioned dependency but also reproducible.

3\. Agreed, the infrastructure and hardware also could change the results and this is a not a simple question what to do with it. Also I would say that (in my humble ds practice) that library versions have more frequent impact on reproducibility than a hardware. With that said, considering scenarios on how to fix the environment may be a little step towards the overall goal, covering large part of this topic.

Yes, sounds good. My question (directed at anyone who's interested) is to what extent should reproducibility of the environment and infrastructure be a goal of DVC? Maybe this belongs in a separate discussion.

[shcheklein]

My 2cs on this. I see two different problems:

1. Making env reproducible
2. Detect changes in the env in the pipeline

I would consider them separately (even if they might overlap). To my mind, if we have a requirements.txt that is saved into Git it already makes env reproducible to some extent? We don't need pipelines or DVC for that (or we need it only if'd like have a safety check).

On the other hand, making a pipeline that can detect changes in the env can be beneficial for the development phase so that dvc repro can spot it (to the @aguschin point) as I update libraries. It's a different use case to me.

[dberenbaum]

Good point. I took the conversation towards reproducibility because in other parts of DVC, we generally have stages reproduce the versioned outputs, so a stage might:

• have a dependency requirements.txt (or environment.yaml, etc.)
• create the environment
• output pip freeze to some other file

However, it makes sense to start with pip freeze to detect changes as @aguschin suggested.

I still prefer it as a blog post or best practice for a specific use case rather than a definitive how-to in part because it feels a bit like a hack (not a bad thing, it's probably the best workaround given the constraints of dvc). If we make it too "official," we may signal that dvc supports tracking package dependencies, which I don't think is accurate right now.

For example, if the output of the next stage in the pipeline doesn't change, I think dvc repro will assume every other stage has not changed, but in fact the environment is a dependency to every stage. What if my preprocessing stage isn't impacted but my training stage would have changed had it been re-run?

[daavoo]

Interesting debate, I'm coming from #6115

Just a quick comment about the pip freeze approach. We actually tried a similar "hack" in our team as the one being proposed to be documented. We found an issue with pip freeze not reflecting changes in dependencies not installed with pip (i.e. system dependencies like CUDA or cuDNN) .

This might look like an edge case but switching versions of CUDA/cuDNN could result in significant impacts regarding model performance, training/inference speed and could even break the dvc,yaml pipeline (i.e. the train stage can't run because the installed pytorch/tensorflow version is incompatible with the installed CUDA version). We have actually experienced all of those edge cases 😅

In the end we opt for combining DVC and MLFlow Projects. Maybe the documentation/blog bost could propose this synergy as an alternative solution?

At least until an alternative solution is implemented in the DVC side or a fully agreement is achieved regarding what "best practice" should be recommended.

[jorgeorpinel]

Should this be moved to the dvc repo discussions for now?