Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: regular update (late June) #456

Merged
merged 10 commits into from
Jun 27, 2019
11 changes: 6 additions & 5 deletions static/docs/commands-reference/add.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ usage: dvc add [-h] [-q | -v] [-R] [--no-commit] [-f FILE]
targets [targets ...]

positional arguments:
targets Input files/directories.
targets Input files/directories to add.
```

## Description
Expand Down Expand Up @@ -78,10 +78,11 @@ This way you bring data provenance and make your project reproducible.

## Options

- `-R`, `--recursive` - `targets` is expected to contain directory path(s).
Determines the files to add by searching each target directory and its
subdirectories for data files. For each file found, a new DVC-file is created
using the process described in this command's description.
- `-R`, `--recursive` - `targets` is expected to contain at least one directory
path for this option to have effect. Determines the files to add by searching
each target directory and its subdirectories for data files. For each file
found, a new DVC-file is created using the process described in this command's
description.

- `--no-commit` - do not put files/directories into cache. A DVC-file is
created, and an entry is added to `.dvc/state`, while nothing is added to the
Expand Down
10 changes: 7 additions & 3 deletions static/docs/commands-reference/checkout.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ usage: dvc checkout [-h] [-q | -v] [-d] [-f] [-R]
[targets [targets ...]]

positional arguments:
targets DVC files.
targets DVC-files to checkout. Optional. (Finds all
DVC-files in the workspace by default.)
```

## Description
Expand Down Expand Up @@ -80,10 +81,13 @@ be pulled from a remote cache using `dvc pull`.
backward from the target stage(s) in the corresponding pipeline(s). This means
DVC will not checkout files referenced in later stage(s) than `targets`.

- `-R`, `--recursive` - performs recursive checkout for target directory.
- `-R`, `--recursive` - `targets` is expected to contain at least one directory
path for this option to have effect. Determines the files to checout by
searching each target directory and its subdirectories for DVC-files to
inspect.

- `-f`, `--force` - does not prompt when removing workspace files. Changing the
current set of DVC files with SCM commands like `git checkout` can result in
current set of DVC-files with SCM commands like `git checkout` can result in
the need for DVC to remove files which should not exist in the current state
and are missing in the local cache (they are not committed in DVC terms). This
option controls whether the user will be asked to confirm these files removal.
Expand Down
16 changes: 8 additions & 8 deletions static/docs/commands-reference/commit.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# commit

Record changes to the repository by updating DVC-files and saving outputs to
cache.
Record changes to the repository by updating
[DVC-files](/doc/user-guide/dvc-file-format) and saving outputs to cache.

## Synopsis

Expand All @@ -10,7 +10,8 @@ usage: dvc commit [-h] [-q | -v] [-f] [-d] [-R]
[targets [targets ...]]

positional arguments:
targets DVC files.
targets DVC-files to commit. Optional. (Finds all
DVC-files in the workspace by default.)
```

## Description
Expand Down Expand Up @@ -69,11 +70,10 @@ It handles that last step of adding the file to the DVC cache.
backward from the target stage(s) in the corresponding pipeline(s). This means
DVC will not commit files referenced in later stage(s) than `targets`.

- `-R`, `--recursive` - `targets` is expected to contain directory path(s).
Determines the files to commit by searching each target directory and its
subdirectories for DVC-files to inspect. Along with providing `targets`, or
`targets` and `--with-deps`, this is another way to limit the scope of
DVC-files to commit.
- `-R`, `--recursive` - `targets` is expected to contain at least one directory
path for this option to have effect. Determines the files to commit by
searching each target directory and its subdirectories for DVC-files to
inspect.

- `-f`, `--force` - commit data even if checksums for dependencies or outputs
did not change.
Expand Down
12 changes: 7 additions & 5 deletions static/docs/commands-reference/fetch.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ usage: dvc fetch [-h] [-q | -v] [-j JOBS] [--show-checksums]
[targets [targets ...]]

positional arguments:
targets DVC files.
targets Limit command scope to these DVC-files. Using -R,
directories to search DVC-files in can also be given.
```

## Description
Expand All @@ -22,7 +23,7 @@ the local cache, but not directly into the workspace. This makes the data files
available for linking (or copying) into the workspace. (Refer to
[dvc config cache.type](/doc/commands-reference/config#cache).) Along with
`dvc checkout`, it's performed automatically by `dvc pull` when the target
DVC-files are not already in the local cache:
[DVC-files](/doc/user-guide/dvc-file-format) are not already in the local cache:

```
Controlled files Commands
Expand Down Expand Up @@ -83,9 +84,10 @@ specified in DVC-files currently in the workspace are considered by `dvc fetch`
backward from the target stage(s) in the corresponding pipeline(s). This means
DVC will not fetch files referenced in later stage(s) than `targets`.

- `-R`, `--recursive` - this option tells DVC that `targets` are directories
(not DVC-files), and to traverse them recursively. All DVC-files found will be
read in order to determine the set of data files to fetch.
- `-R`, `--recursive` - `targets` is expected to contain at least one directory
path for this option to have effect. Determines the files to fetch by
searching each target directory and its subdirectories for DVC-files to
inspect.

- `-j JOBS`, `--jobs JOBS` - number of threads to run simultaneously to handle
the downloading of files from the remote. Using more jobs may improve the
Expand Down
2 changes: 1 addition & 1 deletion static/docs/commands-reference/lock.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ if `--force` is provided.
usage: dvc lock [-h] [-q | -v] targets [targets ...]

positional arguments:
targets DVC-files.
targets DVC-files to lock.
```

## Description
Expand Down
23 changes: 11 additions & 12 deletions static/docs/commands-reference/metrics_show.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,15 @@ It will find and print all metric files (default) or a specified metric file in
the current branch (if `path` is provided) or across all branches/tags (if `-a`
or`-T` specified respectively).

Optional `path` should be path to a directory (if `-R` options specified) or a
regular metric file. If `path` is a directory, recursively search and process
all metric files in path.
The optional `path` argument can represent a DVC metric file or a directory. If
`path` is a directory, recursively search and process all metric files in it
with the `-R` option.

If metric type is specified (via `-t`) it overrides the type and xpath defined
(if any) in the corresponding DVC-file (usually, using `dvc metrics modify`
command). If `-t` is not specified `-x` is ignored. The type can be detected by
the file extension automatically if the type is supported.
If metric type is specified (via `-t`) it overrides the type and `--xpath`
defined (if any) in the corresponding DVC-file (usually, using
`dvc metrics modify` command). If `-t` is not specified `-x` is ignored. The
type can be detected by the file extension automatically if the type is
supported.

## Options

Expand Down Expand Up @@ -71,11 +72,9 @@ the file extension automatically if the type is supported.
can be used to compare different variants of an experiment if tags are used
for checkpoints.

- `-R`, `--recursive` - if `path` is a directory, recursively search and process
all metric files in path. If `path` is a file it is processed as a regular
metric file. This is done to tolerate the difference across branches or tags
when `-a` or `-T` options are used. For example, if `path` is a directory in
one branch and a regular file in another.
- `-R`, `--recursive` - `path` is expected to be a directory for this option to
have effect. Determines the metric files to show by searching each target
directory and its subdirectories for DVC-files to inspect.

## Examples

Expand Down
16 changes: 14 additions & 2 deletions static/docs/commands-reference/pipeline.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# pipeline

A set of commands to manage pipelines:
A set of commands to manage [pipelines](/doc/get-started/pipeline):
[show](/doc/commands-reference/pipeline-show) - visualize or
[list](/doc/commands-reference/pipeline-list) - list existing pipelines.

Expand All @@ -17,7 +17,19 @@ positional arguments:

## Description

Manage pipeline.
A data pipeline, in general, is a chain of commands that process data files. It
produces intermediate data and a final result. For example, Machine Learning
(ML) pipelines typically start a with large raw datasets, include featurization
and training intermediate stages, and produce a final model, as well as certain
metrics.

In DVC, pipeline stage files and commands, their data I/O, interdependencies,
and results (intermediate or final) are defined with `dvc add` and `dvc run`,
among other commands. This allows us to form one or more pipelines of stages
connected by their dependencies and outputs.

`dvc pipeline` commands help users display the existing project pipelines in
different ways.

## Options

Expand Down
22 changes: 19 additions & 3 deletions static/docs/commands-reference/pipeline_list.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,34 @@ are independent of each other.
usage: dvc pipeline list [-h] [-q | -v]
```

## Description

`dvc list` displays a list of all existing stages in the project, grouped in
their corresponding pipeline(s) when connected. (See `dvc pipeline`.)

> Note that the stages in these lists are in ascending order, that is, from last
> to first.

## Options

- `-h`, `--help` - prints the usage/help message, and exit.

- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
problems arise, otherwise 1.

- `-v`, `--verbose` - displays detailed tracing information.

## Examples

- List available pipelines:

```dvc
$ dvc pipeline list

Dvcfile
===============
======================================================================
raw.dvc
data.dvc
output.dvc

======================================================================
2 pipeline(s) total
```
25 changes: 19 additions & 6 deletions static/docs/commands-reference/pipeline_show.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,6 @@ Show [stages](/doc/commands-reference/run) in a pipeline that lead to the
specified stage. By default it lists
[DVC-files](/doc/user-guide/dvc-file-format).

The `-c` and `-o` options allow to list or visualize a pipeline commands or data
files flow instead.

## Synopsis

```usage
Expand All @@ -15,9 +12,20 @@ usage: dvc pipeline show [-h] [-q | -v] [-c | -o] [-l] [--ascii]
[targets [targets ...]]

positional arguments:
targets DVC-files. 'Dvcfile' by default.
targets DVC-files to show pipeline for. Optional.
(Finds all DVC-files in the workspace by default.)
```

## Description

`dvc show` displays the stages of a pipeline up to one or more target DVC-files
(stage files). If specific `targets` are omitted, `Dvcfile` will be assumed. The
`-c` and `-o` options allow to list the corresponding commands or data file flow
instead of stages.

> Note that the stages in these lists are in descending order, that is, from
> first to last.

## Options

- `-c`, `--commands` - show pipeline as a list (graph, if `--ascii` or `--dot`
Expand All @@ -36,13 +44,19 @@ positional arguments:

- `-l`, `--locked` - print locked stages only. See `dvc lock`.

- `-h`, `--help` - prints the usage/help message, and exit.

- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
problems arise, otherwise 1.

- `-v`, `--verbose` - displays detailed tracing information.

## Examples

- Default mode: show stage files that `output.dvc` recursively depends on:

```dvc
$ dvc pipeline show output.dvc

raw.dvc
data.dvc
output.dvc
Expand All @@ -52,7 +66,6 @@ positional arguments:

```dvc
$ dvc pipeline show output.dvc --commands

download.py s3://mybucket/myrawdata raw
cleanup.py raw data
process.py data output
Expand Down
14 changes: 7 additions & 7 deletions static/docs/commands-reference/pull.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

Downloads missing files and directories from
[remote storage](/doc/commands-reference/remote) to the local cache based on
DVC-files in the workspace, then links the downloaded files into the workspace.
[DVC-files](/doc/user-guide/dvc-file-format) in the workspace, then links the
downloaded files into the workspace.

## Synopsis

Expand All @@ -12,7 +13,8 @@ usage: dvc pull [-h] [-q | -v] [-j JOBS] [--show-checksums]
[targets [targets ...]]

positional arguments:
targets DVC files.
targets Limit command scope to these DVC-files. Using -R,
directories to search DVC-files in can also be given.
```

## Description
Expand Down Expand Up @@ -79,11 +81,9 @@ reflinks or hardlinks to put it in the workspace without copying. See
backward from the target stage(s) in the corresponding pipeline(s). This means
DVC will not pull files referenced in later stage(s) than `targets`.

- `-R`, `--recursive` - `targets` values is expected to be a directory path.
Determines the files to download by searching the named directory and its
subdirectories for DVC-files to download data for. Along with providing a
`target`, or `target` along with `--with-deps` it is yet another way to cut
the scope of DVC-files to download.
- `-R`, `--recursive` - `targets` is expected to contain at least one directory
path for this option to have effect. Determines the files to pull by searching
each target directory and its subdirectories for DVC-files to inspect.

- `-f`, `--force` - does not prompt when removing working directory files, which
occurs during the process of updating the workspace. This option surfaces
Expand Down
11 changes: 5 additions & 6 deletions static/docs/commands-reference/push.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ usage: dvc push [-h] [-q | -v] [-j JOBS] [--show-checksums]
[targets [targets ...]]

positional arguments:
targets DVC files.
targets Limit command scope to these DVC-files. Using -R,
directories to search DVC-files in can also be given.
```

## Description
Expand Down Expand Up @@ -89,11 +90,9 @@ to push.
backward from the target stage(s) in the corresponding pipeline(s). This means
DVC will not push files referenced in later stage(s) than `targets`.

- `-R`, `--recursive` - `targets` is expected to contain directory path(s).
Determines the files to upload by searching each target directory and its
subdirectories for DVC-files to inspect. Along with providing a `target`, or
`target` and `--with-deps`, this is another way to limit the scope of
DVC-files to upload.
- `-R`, `--recursive` - `targets` is expected to contain at least one directory
path for this option to have effect. Determines the files to push by searching
each target directory and its subdirectories for DVC-files to inspect.

- `-j JOBS`, `--jobs JOBS` - specifies number of jobs to run simultaneously
while uploading files to the remote cache. The effect is to control the number
Expand Down
2 changes: 1 addition & 1 deletion static/docs/commands-reference/remote_add.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ usage: dvc remote add [-h] [--global] [--system] [--local] [-q | -v]
[-d] [-f] name url

positional arguments:
name Name.
name Name of the remote.
url URL. (See supported URLs below.)
```

Expand Down
3 changes: 2 additions & 1 deletion static/docs/commands-reference/remove.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ want to use or share in the future.
usage: dvc remove [-h] [-q | -v] [-o | -p] [-f] targets [targets ...]

positional arguments:
targets DVC-files.
targets DVC-files to remove. Optional. (Finds all
DVC-files in the workspace by default.)
```

Check also [Update Tracked Files](/doc/user-guide/update-tracked-file) to see
Expand Down
7 changes: 6 additions & 1 deletion static/docs/commands-reference/repro.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ usage: dvc repro [-h] [-q | -v] [-f] [-s] [-c CWD] [-m] [--dry] [-i]
[--downstream] [targets [targets ...]]

positional arguments:
targets DVC file to reproduce (default - 'Dvcfile').
targets DVC-file to reproduce. 'Dvcfile' by default.
```

## Description
Expand Down Expand Up @@ -57,6 +57,11 @@ specified), and updates stage files with the new checksum information.
reproduced as part of the pipeline in the parent directory, or as an
independent unit.

- `-R`, `--recursive` - `targets` is expected to contain at least one directory
path for this option to have effect. Determines the stages to reproduce by
searching each target directory and its subdirectories for DVC-files to
inspect.

- `--no-commit` - do not save outputs to cache. Useful when running different
experiments and you don't want to fill up your cache with temporary files. Use
`dvc commit` when you are ready to save your results to cache.
Expand Down
Loading