Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update data-pipelines.md #1645

Merged
merged 9 commits into from
Jul 31, 2020
25 changes: 14 additions & 11 deletions content/docs/start/data-pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -292,17 +292,20 @@ prepare:
DVC pipelines (`dvc.yaml` file, `dvc run`, and `dvc repro` commands) solve a few
important problems:

- _Automation_ - run sequence of steps in a "smart" way that makes iterating on
the project faster. It automatically determines which parts of a project need
to be run, it caches "runs" and results β€” all to avoid running the same stage
again.
- _Reproducibility_ - it can describe and capture what data should be used and
what commands to run to produce an ML model, for example. It's described and
captured in way that is easy to put into Git. It means that it's easy to
version and share.
- _Continuous Delivery and Continuous Integration (CI/CD) for ML_ - describing
project in way that it can be reproduced (built) is the fist necessary step
before introducing CI/CD systems.
- _Automation_ - run a sequence of steps in a "smart" way to make iterating on
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
your project faster. DVC caches "runs" and results in stages and automatically
determines which parts of a project need to be run to avoid unnecessary
re-runs.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
- _Reproducibility_ - `dvc.yaml` and `dvc.lock` files describe and capture what
data to use and what commands to run to produce an ML model. Storing these
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
files in Git makes it easy to version and share.
- _Continuous Delivery and Continuous Integration (CI/CD) for ML_ - reproducible
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
ML pipelines allow CI/CD systems to retrain
([repro](https://dvc.org/doc/command-reference/repro)) models on fresh
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
datasets with identical preprocessing and training stages, version upstream
models and datasets, and easily
[compare](https://dvc.org/doc/start/experiments#comparing-experiments) metrics
with currently deployed models.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

## Visualize

Expand Down