Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] install: add docker usage documentation #811

Closed
wants to merge 6 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions src/Documentation/Markdown/Markdown.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,16 @@ import Tooltip from '../../Tooltip'
import Collapsible from 'react-collapsible'
// syntax highlighter
import { Light as SyntaxHighlighter } from 'react-syntax-highlighter'
import { docco } from 'react-syntax-highlighter/dist/cjs/styles/hljs'
import python from 'react-syntax-highlighter/dist/cjs/languages/hljs/python'
import yaml from 'react-syntax-highlighter/dist/cjs/languages/hljs/yaml'
import ini from 'react-syntax-highlighter/dist/cjs/languages/hljs/ini'
import bash from 'react-syntax-highlighter/dist/cjs/languages/hljs/bash'
import diff from 'react-syntax-highlighter/dist/cjs/languages/hljs/diff'
import vim from 'react-syntax-highlighter/dist/cjs/languages/hljs/vim'
import {
docco,
python,
yaml,
ini,
bash,
diff,
vim,
dockerfile
} from 'react-syntax-highlighter/dist/cjs/styles/hljs'
import usage from './lang/usage'
import dvc from './lang/dvc'
import linker from './utils/remark-linker'
Expand All @@ -31,6 +34,7 @@ SyntaxHighlighter.registerLanguage('ini', ini)
SyntaxHighlighter.registerLanguage('bash', bash)
SyntaxHighlighter.registerLanguage('vim', vim)
SyntaxHighlighter.registerLanguage('diff', diff)
SyntaxHighlighter.registerLanguage('dockerfile', dockerfile)

function flatten(text, child) {
return typeof child === 'string'
Expand Down
4 changes: 4 additions & 0 deletions src/Documentation/sidebar.json
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,10 @@
"label": "Running DVC on Windows",
"slug": "running-dvc-on-windows"
},
{
"label": "Using Docker",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's make it part of the Install section

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All right)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see both static/docs/install/docker.md and static/docs/user-guide/using-docker.md present in this PR. Should only review docs/install then?

"slug": "using-docker"
},
{
"label": "Anonymized Usage Analytics",
"slug": "analytics"
Expand Down
114 changes: 114 additions & 0 deletions static/docs/install/docker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Using Docker

Currently we don't provide any official Docker image, however you can easily
create it yourself.

## Writing the Dockerfile

We support several ways to install DVC, the two recommended ones are through
`pip` and `conda`.

You can base your image on the official Python one:

```dockerfile
FROM python

# Remember to include the cloud providers that you plan to use,
# for example: "dvc[s3]"
RUN pip install dvc
```

Or use a Conda based image:

```dockerfile
FROM continuumio/miniconda3

RUN conda install dvc -y --quiet
```

For a recommended reading about writing Dockerfiles for Python applications, you
can take a look at https://pythonspeed.com/docker/

## Using Docker for your development environment

Since `dvc` is a command line application, we have several recommendations to
enhance the experience:

- Make sure your _locale_ is set to UTF-8
- Install the Zsh shell and its respective autocompletion script.
- Install `less` so you can pipe the output of `dvc pipeline show`.
- Enable movements with `Ctrl + Arrow` by editing the _inputrc_ file.
- Use `pip install dvc` at the end of the file for better caching.
- Set your `WORKDIR` to the path where you are going to mount your project.

If you are using Linux, remember to map your user to your container. This way,
you make sure your files are still writable. For more information:
https://docs.docker.com/engine/security/userns-remap/

### Example Dockerfile

The following is a **suggestion** for a Dockerfile to use during development:

```dockerfile
FROM python:3.7
Comment on lines +52 to +53
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe explain why 3.7, since the first example is just python.


# Use UTF-8 as the default locale
ENV LANG=C.UTF-8

# Better movement with [Ctrl + →] and [Ctrl + ←]
ENV INPUTRC=/etc/inputrc

RUN set -ex \
&& echo '"\e[1;5C": forward-word' >> /etc/inputrc \
&& echo '"\e[1;5D": backward-word' >> /etc/inputrc
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

# Set working directory.
# Mount your project under the same path.
WORKDIR /usr/src

# Install useful programs
RUN apt-get update -y && apt-get install -y \
zsh \
less \
git

# Set Zsh as the default shell
SHELL ["/usr/bin/zsh", "-c"]
Comment on lines +75 to +76
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why change the default? (Bash)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume to accommodate the custom zsh syntax config... Speaking of which, it would be nice to provide equivalent bash config if possible

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which goes to my point. Probably an unnecessary complication for a Dockerfile sample.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@casperdcl , bash is not as customizable as zsh, completion is really basic 😬

Copy link
Contributor

@jorgeorpinel jorgeorpinel Nov 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would agree given the argument about providing a better experience, but I think most people are used to Bash, so not sure. It's not like we recommend Zsh over Bash in our docs. This one may be unnecessary.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


# Enable autocompletion scripts for DVC
# and rice the completion style with `zstyle`:
# http://zsh.sourceforge.net/Doc/Release/Zsh-Modules.html#The-zsh_002fzutil-Module
RUN wget \
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
-O /usr/local/share/zsh/site-functions/_dvc \
https://raw.githubusercontent.com/iterative/dvc/master/scripts/completion/dvc.zsh \
&& echo "autoload -U compinit && compinit" >> /root/.zshrc \
&& echo "zstyle ':completion:*:*:*:*:*' menu select" >> /root/.zshrc \
&& echo "zstyle ':completion:*:matches' group 'yes'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:options' description 'yes'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:options' auto-description '%d'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:corrections' format ' %F{green}-- %d (errors: %e) --%f'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:descriptions' format ' %F{yellow}-- %d --%f'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:messages' format ' %F{purple} -- %d --%f'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:warnings' format ' %F{red}-- no matches found --%f'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:default' list-prompt '%S%M matches%s'" >> /root/.zshrc \
&& echo "zstyle ':completion:*' format ' %F{yellow}-- %d --%f'" >> /root/.zshrc \
&& echo "zstyle ':completion:*' group-name ''" >> /root/.zshrc \
&& echo "zstyle ':completion:*' verbose yes" >> /root/.zshrc

# Install DVC supporting all the remotes
RUN pip install "dvc[all]"

# vim: ft=dockerfile
Comment on lines +100 to +101
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not familiar with vim. What does this do? Just curious.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forces Dockerfile syntax highlighting for the whole file when editing in vim

Copy link
Contributor

@jorgeorpinel jorgeorpinel Nov 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, thanks!

In that case I would remove this. Why provide something editor-specific?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jorgeorpinel , it is nice to have it)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it doesn't hurt but still, this is not related to the shell experience, or DVC, or Docker, so I would remove it. Should we vote @shcheklein @casparjespersen? So far it seems like it's Ramon for, myself against.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

```

Then, build the image and run it:

```console
$ TAG="dvc:development"
Comment on lines +106 to +107
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May need to use ```dvc here given our docs engine automatic highlighting styles. Please check how each looks running the docs locally.

$ docker build -t $TAG .
Comment on lines +107 to +108
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need for tag?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 yep, building a container with a tag is a good practice

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK! But how about just using development or dev? dvc will probably be in the image name. (I read https://cloud.google.com/solutions/best-practices-for-building-containers#properly_tag_your_images.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

p.s. Maybe use an example image name somewhere in this doc?

$ docker run --interactive \
--tty \
--rm \
--volume $PWD:/usr/src \
$TAG
```
4 changes: 4 additions & 0 deletions static/docs/user-guide/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ guides related to contributing to this
- [Contributing](/doc/user-guide/contributing) is related with getting involved
directly in developing DVC. Contributors are very welcomed in our
[community](/support)!
- [Running DVC on Windows](/doc/user-guide/running-dvc-on-windows) contains
useful information for our Windows users.
- [Using Docker](/doc/user-guide/using-docker) explains how to set up a Docker
image to develop with DVC and which considerations to have.

The remaining guides are about optional productivity enhancements and other tips
and tools related to advanced usage of DVC.
114 changes: 114 additions & 0 deletions static/docs/user-guide/using-docker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Using Docker

Currently we don't provide any official Docker image, however you can easily
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have the Docker file ready, why don't we create an Image?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shcheklein , being honest, I think that having an official image would bring more harm than benefits (maintenance burden, releases, etc.).

This is only an example for the ones who are completely lost at the topic. It is not intended to serve as a reference for best practices on writing Docker images using dvc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's much point documenting anything other than a reference for best practices... 😕

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

publishing a complete Dockerfile like this is also a burden - we'll need to keep it updated, right?

if we see some value in Docker I would create that image (it's quite straightforward to make it part of CI) or put a Dockerfile under Git and make a doc on how to start using it. It's quite strange to publish a complete Dockerfile this way.

@casperdcl not sure I got what way your argument is going :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes trying to explain my ethos here: #811 (comment)

create it yourself.

## Writing the Dockerfile

We support several ways to install DVC, the two recommended ones are through
`pip` and `conda`.

You can base your image on the official Python one:

```dockerfile
FROM python

# Remember to include the cloud providers that you plan to use,
# for example: "dvc[s3]"
RUN pip install dvc
```

Or use a Conda based image:

```dockerfile
FROM continuumio/miniconda3

RUN conda install dvc -y --quiet
```

For a recommended reading about writing Dockerfiles for Python applications, you
can take a look at https://pythonspeed.com/docker/

## Using Docker for your development environment

Since `dvc` is a command line application, we have several recommendations to
enhance the experience:

- Make sure your _locale_ is set to UTF-8
- Install the Zsh shell and its respective autocompletion script.
- Install `less` so you can pipe the output of `dvc pipeline show`.
- Enable movements with `Ctrl + Arrow` by editing the _inputrc_ file.
- Use `pip install dvc` at the end of the file for better caching.
- Set your `WORKDIR` to the path where you are going to mount your project.

If you are using Linux, remember to map your user to your container. This way,
you make sure your files are still writable. For more information:
https://docs.docker.com/engine/security/userns-remap/

### Example Dockerfile

The following is a **suggestion** for a Dockerfile to use during development:

```dockerfile
FROM python

# Use UTF-8 as the default locale
ENV LANG=C.UTF-8

# Better movement with [Ctrl + →] and [Ctrl + ←]
ENV INPUTRC=/etc/inputrc

RUN set -ex \
&& echo '"\e[1;5C": forward-word' >> /etc/inputrc \
&& echo '"\e[1;5D": backward-word' >> /etc/inputrc

# Set working directory.
# Mount your project under the same path.
WORKDIR /usr/src

# Install useful programs
RUN apt-get update -y && apt-get install -y \
zsh \
less \
git

# Set Zsh as the default shell
SHELL ["/usr/bin/zsh", "-c"]

# Enable autocompletion scripts for DVC
This conversation was marked as resolved.
Show resolved Hide resolved
# and rice the completion style with `zstyle`:
# http://zsh.sourceforge.net/Doc/Release/Zsh-Modules.html#The-zsh_002fzutil-Module
RUN wget \
-O /usr/local/share/zsh/site-functions/_dvc \
https://raw.githubusercontent.com/iterative/dvc/master/scripts/completion/dvc.zsh \
&& echo "autoload -U compinit && compinit" >> /root/.zshrc \
&& echo "zstyle ':completion:*:*:*:*:*' menu select" >> /root/.zshrc \
&& echo "zstyle ':completion:*:matches' group 'yes'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:options' description 'yes'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:options' auto-description '%d'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:corrections' format ' %F{green}-- %d (errors: %e) --%f'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:descriptions' format ' %F{yellow}-- %d --%f'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:messages' format ' %F{purple} -- %d --%f'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:warnings' format ' %F{red}-- no matches found --%f'" >> /root/.zshrc \
&& echo "zstyle ':completion:*:default' list-prompt '%S%M matches%s'" >> /root/.zshrc \
&& echo "zstyle ':completion:*' format ' %F{yellow}-- %d --%f'" >> /root/.zshrc \
&& echo "zstyle ':completion:*' group-name ''" >> /root/.zshrc \
&& echo "zstyle ':completion:*' verbose yes" >> /root/.zshrc

# Install DVC supporting all the remotes
RUN pip install "dvc[all]"

# vim: ft=dockerfile
```

Then, build the image and run it:

```console
$ TAG="dvc:development"
$ docker build -t $TAG .
$ docker run --interactive \
--tty \
--rm \
--volume $PWD:/usr/src \
$TAG
```