Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

glossary: create "stage" entry #2198

Closed
wants to merge 47 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
dac8b8b
guide: intro (empty) EM guide + basic links
jorgeorpinel Feb 3, 2021
352aaac
Merge branch 'master' into guide/experiments
jorgeorpinel Feb 3, 2021
b1e4c58
guide: intro, structure, and run-cache section in Exp Mgmt
jorgeorpinel Feb 3, 2021
92c3b1f
guide: finish Exp Mgmt intro, shorten run-cache section
jorgeorpinel Feb 4, 2021
70e4f91
guide: shorten Exp Mgmt intro and its run-cache section more, and
jorgeorpinel Feb 5, 2021
f73ae78
guide: finish explaining ephimeral exps
jorgeorpinel Feb 9, 2021
6b9b509
guide: improves to ephimeral exps, intro persisten exps in Exp Mgmt
jorgeorpinel Feb 9, 2021
203fa3c
guide: more info on ephimeral exps and wrap up persistent exps section
jorgeorpinel Feb 9, 2021
fd344d6
guide: begin Orging exps in Exp Mgmt
jorgeorpinel Feb 9, 2021
c7cc388
guide: copy edits Ephemeral experiments
jorgeorpinel Feb 9, 2021
2fabec3
Merge branch 'master' into guide/experiments
jorgeorpinel Feb 10, 2021
ea260f6
guide: remove 0. Tests and add add Checkpoints section
jorgeorpinel Feb 10, 2021
3099d1a
guide: 3 main exp forms in Exp Mgmt
jorgeorpinel Feb 10, 2021
d012334
guide: simplify Exp Mgmt tech details
jorgeorpinel Feb 10, 2021
006068c
guide: finish Checkpoints section in Exp Mgmt
jorgeorpinel Feb 11, 2021
520563b
guide: mention params, metrics, plots, and other copy edits to Exp Mgmt
jorgeorpinel Feb 11, 2021
b571f50
guide: wrap up comprehensive Ecx Mgmt doc, add checkpoint to dvc.yaml…
jorgeorpinel Feb 16, 2021
b540d4e
Merge branch 'master' into guide/experiments
jorgeorpinel Feb 16, 2021
143a2b2
glossary: create "stage" entry
iesahin Feb 16, 2021
f1bfda3
ref: move checkpoint field into dvc.yaml output spec
jorgeorpinel Feb 16, 2021
51fa327
glossary: migrate exp tooltip to new format (fontmatter)
jorgeorpinel Feb 16, 2021
1ea2da3
modified to have a more informal language and removed BC text from th…
iesahin Feb 17, 2021
3d7aed3
cmd: add dvc update --to-remote docs (#2175)
isidentical Feb 18, 2021
ba8178e
Draft of Feb Heartbeat (#2190)
jendefig Feb 18, 2021
bdf1396
dvc 2.0 pre-release
dmpetrov Feb 18, 2021
be79555
Update content/blog/2021-02-18-dvc-2-0-pre-release.md
jorgeorpinel Feb 18, 2021
f84dac4
Update content/blog/2021-02-18-dvc-2-0-pre-release.md
jorgeorpinel Feb 18, 2021
25b13fe
Update content/blog/2021-02-18-dvc-2-0-pre-release.md
jorgeorpinel Feb 18, 2021
ed3e96f
Update content/blog/2021-02-18-dvc-2-0-pre-release.md
jorgeorpinel Feb 18, 2021
3788662
Update content/blog/2021-02-18-dvc-2-0-pre-release.md
jorgeorpinel Feb 18, 2021
8db8665
Update content/blog/2021-02-18-dvc-2-0-pre-release.md
jorgeorpinel Feb 18, 2021
9503aa0
Update content/blog/2021-02-18-dvc-2-0-pre-release.md
jorgeorpinel Feb 18, 2021
cd27812
Update content/blog/2021-02-18-dvc-2-0-pre-release.md
jorgeorpinel Feb 18, 2021
47b85cb
Update content/blog/2021-02-18-dvc-2-0-pre-release.md
jorgeorpinel Feb 18, 2021
26fda1d
feedback & experiments
dmpetrov Feb 18, 2021
a2310da
live metrics
dmpetrov Feb 18, 2021
5209585
Update content/blog/2021-02-18-dvc-2-0-pre-release.md
jorgeorpinel Feb 18, 2021
308c825
show live metrics
dmpetrov Feb 18, 2021
3a2ffeb
add tables in checkpoint section
dmpetrov Feb 18, 2021
7c6e34c
Merge pull request #2211 from iterative/dvc-20-pre-release
dmpetrov Feb 19, 2021
943e55f
Merge pull request #2146 from iterative/guide/experiments
jorgeorpinel Feb 19, 2021
4398f7c
pre-release blog fixes
dmpetrov Feb 19, 2021
d651fc0
exp run - fix resume table
dmpetrov Feb 19, 2021
e3e3653
fix duplication issue in table
dmpetrov Feb 19, 2021
1908fa1
Merge pull request #2218 from iterative/dvc-20-pre-release-fixes
dmpetrov Feb 19, 2021
87942d0
document stage:add (#2208)
skshetry Feb 19, 2021
8293e53
Merge branch 'iesahin/issue1579' of github.com:iterative/dvc.org into…
iesahin Feb 20, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
guide: 3 main exp forms in Exp Mgmt
jorgeorpinel committed Feb 10, 2021
commit 3099d1acd35a5f3a2b96996767e037d2d06a85d5
73 changes: 42 additions & 31 deletions content/docs/user-guide/experiment-management.md
Original file line number Diff line number Diff line change
@@ -3,35 +3,23 @@
Data science and ML are iterative processes that tend to require a large number
of attempts during their course, for example to develop data features,
hyperspace exploration, deep learning optimization, etc. DVC helps you codify
and manage all of your <abbr>experiments</abbr>, considering the following
levels at which they may exist:

1. DVC enters the scene with an automatic log of every stage `dvc repro` runs.
2. Create [ephemeral experiments](#ephemeral-experiments) and
[in-code checkpoints](#checkpoints-in-python-code) that virtually branch off
your current workspace. DVC lets you quickly visualize and compare them from
terminal. The best ones can be promoted to the next level, and the rest
archived.
and manage all of your <abbr>experiments</abbr>, supporting these main
experimentation approaches:

1. Create [ephemeral experiments](#ephemeral-experiments) that derive from your
latest project version, without having to keep track manually. DVC does that
for you, letting you list and compare them. The best ones can be promoted,
and the rest archived.
2. Place [in-code checkpoints](#checkpoints-in-python-code) that form series of
(ephemeral) experiments. DVC helps you capture them at runtime, and manage
them as batches.
3. [Persistent experiments](#persistent-experiments) have their results
**committed** to Git. They can be selected from previous levels, or created
from scratch. This is where you may want to consider the different
[ways to organize](#organizing-experimentats) them in your project (as
branches, folders, etc.).
**committed** to Git. They can be selected from existing attempts, or created
from scratch.

> Note that DVC assumes that all experiments are deterministic (see **Avoiding
> unexpected behavior** in `dvc run`).
## Automatic log of stage runs (run-cache)

Every time you `dvc repro` pipelines, DVC logs the unique signature of each
stage run (to `.dvc/cache/runs` by default). If it never happened before, the
stage command(s) are executed normally. Every subsequent time a
[stage](/doc/command-reference/run) runs under the same conditions, the previous
results can be restored instantly, without wasting time or computing resources.

✅ This built-in feature is called <abbr>run-cache</abbr> and it can
dramatically improve performance. It's enabled out-of-the-box (but can be
disabled with the `--no-run-cache` command option).
At this point you may also want to consider the different
[ways to organize](#organizing-experimentats) experiments in your project (as
Git branches, as folders, etc.).

## Ephemeral experiments

@@ -70,16 +58,26 @@ branch off there. Each reference has a unique signature similar to the

</details>

Note that `dvc exp run` also logs and reuses
[stage runs](#automatic-log-of-stage-runs-run-cache) in the
<abbr>run-cache</abbr> by default.

> See `dvc exp` for more details and other commands.
## Checkpoints in Python code

⚠️ This feature is only available in DVC 2.0, and for Git-enabled
<abbr>repositories</abbr> ⚠️

...

<details>

### How are checkpoints captured by DVC?

When DVC runs checkpoint-enabled stages, a new transient commit is generated in
the experiment's virtual branch each time the code calls
`dvc.api.make_checkpoint()` or writes a `.dvc/tmp/DVC_CHECKPOINT` signal file.
See `dvc exp run` for more details

</details>

## Persistent experiments

When your experiments are good enough to save or share, you may want to store
@@ -93,6 +91,19 @@ have the right `dvc.yaml` and `dvc.lock` file pair, as well as the corresponding
See [Get Started: Experiments](/doc/start/experiments) for a hands-on intro
guide on regular experiments.

## Automatic log of stage runs (run-cache)

Every time you `dvc repro` pipelines or `dvc exp run` experiments, DVC logs the
unique signature of each stage run (to `.dvc/cache/runs` by default). If it
never happened before, the stage command(s) are executed normally. Every
subsequent time a [stage](/doc/command-reference/run) runs under the same
conditions, the previous results can be restored instantly, without wasting time
or computing resources.

✅ This built-in feature is called <abbr>run-cache</abbr> and it can
dramatically improve performance. It's enabled out-of-the-box (but can be
disabled with the `--no-run-cache` command option).

## Organizing experiments

Automatic [stage run logs](#automatic-log-of-stage-runs-run-cache) are dumped