-
Notifications
You must be signed in to change notification settings - Fork 394
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] install: add docker usage documentation #811
Changes from all commits
0ffd189
eab9504
58367ae
f216f40
ef7ce3e
46d4b63
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,114 @@ | ||
# Using Docker | ||
|
||
Currently we don't provide any official Docker image, however you can easily | ||
create it yourself. | ||
|
||
## Writing the Dockerfile | ||
|
||
We support several ways to install DVC, the two recommended ones are through | ||
`pip` and `conda`. | ||
|
||
You can base your image on the official Python one: | ||
|
||
```dockerfile | ||
FROM python | ||
|
||
# Remember to include the cloud providers that you plan to use, | ||
# for example: "dvc[s3]" | ||
RUN pip install dvc | ||
``` | ||
|
||
Or use a Conda based image: | ||
|
||
```dockerfile | ||
FROM continuumio/miniconda3 | ||
|
||
RUN conda install dvc -y --quiet | ||
``` | ||
|
||
For a recommended reading about writing Dockerfiles for Python applications, you | ||
can take a look at https://pythonspeed.com/docker/ | ||
|
||
## Using Docker for your development environment | ||
|
||
Since `dvc` is a command line application, we have several recommendations to | ||
enhance the experience: | ||
|
||
- Make sure your _locale_ is set to UTF-8 | ||
- Install the Zsh shell and its respective autocompletion script. | ||
- Install `less` so you can pipe the output of `dvc pipeline show`. | ||
- Enable movements with `Ctrl + Arrow` by editing the _inputrc_ file. | ||
- Use `pip install dvc` at the end of the file for better caching. | ||
- Set your `WORKDIR` to the path where you are going to mount your project. | ||
|
||
If you are using Linux, remember to map your user to your container. This way, | ||
you make sure your files are still writable. For more information: | ||
https://docs.docker.com/engine/security/userns-remap/ | ||
|
||
### Example Dockerfile | ||
|
||
The following is a **suggestion** for a Dockerfile to use during development: | ||
|
||
```dockerfile | ||
FROM python:3.7 | ||
Comment on lines
+52
to
+53
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe explain why 3.7, since the first example is just |
||
|
||
# Use UTF-8 as the default locale | ||
ENV LANG=C.UTF-8 | ||
|
||
# Better movement with [Ctrl + →] and [Ctrl + ←] | ||
ENV INPUTRC=/etc/inputrc | ||
|
||
RUN set -ex \ | ||
&& echo '"\e[1;5C": forward-word' >> /etc/inputrc \ | ||
&& echo '"\e[1;5D": backward-word' >> /etc/inputrc | ||
jorgeorpinel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
# Set working directory. | ||
# Mount your project under the same path. | ||
WORKDIR /usr/src | ||
|
||
# Install useful programs | ||
RUN apt-get update -y && apt-get install -y \ | ||
zsh \ | ||
less \ | ||
git | ||
|
||
# Set Zsh as the default shell | ||
SHELL ["/usr/bin/zsh", "-c"] | ||
Comment on lines
+75
to
+76
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why change the default? (Bash) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I assume to accommodate the custom There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Which goes to my point. Probably an unnecessary complication for a Dockerfile sample. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @casperdcl , There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would agree given the argument about providing a better experience, but I think most people are used to Bash, so not sure. It's not like we recommend Zsh over Bash in our docs. This one may be unnecessary. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Moved to iterative/dvc#2844 (review) |
||
|
||
# Enable autocompletion scripts for DVC | ||
# and rice the completion style with `zstyle`: | ||
# http://zsh.sourceforge.net/Doc/Release/Zsh-Modules.html#The-zsh_002fzutil-Module | ||
RUN wget \ | ||
jorgeorpinel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
-O /usr/local/share/zsh/site-functions/_dvc \ | ||
https://raw.githubusercontent.com/iterative/dvc/master/scripts/completion/dvc.zsh \ | ||
&& echo "autoload -U compinit && compinit" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:*:*:*:*' menu select" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:matches' group 'yes'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:options' description 'yes'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:options' auto-description '%d'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:corrections' format ' %F{green}-- %d (errors: %e) --%f'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:descriptions' format ' %F{yellow}-- %d --%f'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:messages' format ' %F{purple} -- %d --%f'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:warnings' format ' %F{red}-- no matches found --%f'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:default' list-prompt '%S%M matches%s'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*' format ' %F{yellow}-- %d --%f'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*' group-name ''" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*' verbose yes" >> /root/.zshrc | ||
|
||
# Install DVC supporting all the remotes | ||
RUN pip install "dvc[all]" | ||
|
||
# vim: ft=dockerfile | ||
Comment on lines
+100
to
+101
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not familiar with There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. forces Dockerfile syntax highlighting for the whole file when editing in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see, thanks! In that case I would remove this. Why provide something editor-specific? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jorgeorpinel , it is nice to have it) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I guess it doesn't hurt but still, this is not related to the shell experience, or DVC, or Docker, so I would remove it. Should we vote @shcheklein @casparjespersen? So far it seems like it's Ramon for, myself against. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Moved to iterative/dvc#2844 (review) |
||
``` | ||
|
||
Then, build the image and run it: | ||
|
||
```console | ||
$ TAG="dvc:development" | ||
Comment on lines
+106
to
+107
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. May need to use ```dvc here given our docs engine automatic highlighting styles. Please check how each looks running the docs locally. |
||
$ docker build -t $TAG . | ||
Comment on lines
+107
to
+108
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Need for tag? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🤔 yep, building a container with a tag is a good practice There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. OK! But how about just using There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. p.s. Maybe use an example image name somewhere in this doc? |
||
$ docker run --interactive \ | ||
--tty \ | ||
--rm \ | ||
--volume $PWD:/usr/src \ | ||
$TAG | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,114 @@ | ||
# Using Docker | ||
|
||
Currently we don't provide any official Docker image, however you can easily | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If we have the Docker file ready, why don't we create an Image? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @shcheklein , being honest, I think that having an official image would bring more harm than benefits (maintenance burden, releases, etc.). This is only an example for the ones who are completely lost at the topic. It is not intended to serve as a reference for best practices on writing Docker images using There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think there's much point documenting anything other than a reference for best practices... 😕 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. publishing a complete Dockerfile like this is also a burden - we'll need to keep it updated, right? if we see some value in Docker I would create that image (it's quite straightforward to make it part of CI) or put a Dockerfile under Git and make a doc on how to start using it. It's quite strange to publish a complete Dockerfile this way. @casperdcl not sure I got what way your argument is going :) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes trying to explain my ethos here: #811 (comment) |
||
create it yourself. | ||
|
||
## Writing the Dockerfile | ||
|
||
We support several ways to install DVC, the two recommended ones are through | ||
`pip` and `conda`. | ||
|
||
You can base your image on the official Python one: | ||
|
||
```dockerfile | ||
FROM python | ||
|
||
# Remember to include the cloud providers that you plan to use, | ||
# for example: "dvc[s3]" | ||
RUN pip install dvc | ||
``` | ||
|
||
Or use a Conda based image: | ||
|
||
```dockerfile | ||
FROM continuumio/miniconda3 | ||
|
||
RUN conda install dvc -y --quiet | ||
``` | ||
|
||
For a recommended reading about writing Dockerfiles for Python applications, you | ||
can take a look at https://pythonspeed.com/docker/ | ||
|
||
## Using Docker for your development environment | ||
|
||
Since `dvc` is a command line application, we have several recommendations to | ||
enhance the experience: | ||
|
||
- Make sure your _locale_ is set to UTF-8 | ||
- Install the Zsh shell and its respective autocompletion script. | ||
- Install `less` so you can pipe the output of `dvc pipeline show`. | ||
- Enable movements with `Ctrl + Arrow` by editing the _inputrc_ file. | ||
- Use `pip install dvc` at the end of the file for better caching. | ||
- Set your `WORKDIR` to the path where you are going to mount your project. | ||
|
||
If you are using Linux, remember to map your user to your container. This way, | ||
you make sure your files are still writable. For more information: | ||
https://docs.docker.com/engine/security/userns-remap/ | ||
|
||
### Example Dockerfile | ||
|
||
The following is a **suggestion** for a Dockerfile to use during development: | ||
|
||
```dockerfile | ||
FROM python | ||
|
||
# Use UTF-8 as the default locale | ||
ENV LANG=C.UTF-8 | ||
|
||
# Better movement with [Ctrl + →] and [Ctrl + ←] | ||
ENV INPUTRC=/etc/inputrc | ||
|
||
RUN set -ex \ | ||
&& echo '"\e[1;5C": forward-word' >> /etc/inputrc \ | ||
&& echo '"\e[1;5D": backward-word' >> /etc/inputrc | ||
|
||
# Set working directory. | ||
# Mount your project under the same path. | ||
WORKDIR /usr/src | ||
|
||
# Install useful programs | ||
RUN apt-get update -y && apt-get install -y \ | ||
zsh \ | ||
less \ | ||
git | ||
|
||
# Set Zsh as the default shell | ||
SHELL ["/usr/bin/zsh", "-c"] | ||
|
||
# Enable autocompletion scripts for DVC | ||
This conversation was marked as resolved.
Show resolved
Hide resolved
|
||
# and rice the completion style with `zstyle`: | ||
# http://zsh.sourceforge.net/Doc/Release/Zsh-Modules.html#The-zsh_002fzutil-Module | ||
RUN wget \ | ||
-O /usr/local/share/zsh/site-functions/_dvc \ | ||
https://raw.githubusercontent.com/iterative/dvc/master/scripts/completion/dvc.zsh \ | ||
&& echo "autoload -U compinit && compinit" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:*:*:*:*' menu select" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:matches' group 'yes'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:options' description 'yes'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:options' auto-description '%d'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:corrections' format ' %F{green}-- %d (errors: %e) --%f'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:descriptions' format ' %F{yellow}-- %d --%f'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:messages' format ' %F{purple} -- %d --%f'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:warnings' format ' %F{red}-- no matches found --%f'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*:default' list-prompt '%S%M matches%s'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*' format ' %F{yellow}-- %d --%f'" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*' group-name ''" >> /root/.zshrc \ | ||
&& echo "zstyle ':completion:*' verbose yes" >> /root/.zshrc | ||
|
||
# Install DVC supporting all the remotes | ||
RUN pip install "dvc[all]" | ||
|
||
# vim: ft=dockerfile | ||
``` | ||
|
||
Then, build the image and run it: | ||
|
||
```console | ||
$ TAG="dvc:development" | ||
$ docker build -t $TAG . | ||
$ docker run --interactive \ | ||
--tty \ | ||
--rm \ | ||
--volume $PWD:/usr/src \ | ||
$TAG | ||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's make it part of the Install section
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All right)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see both static/docs/install/docker.md and static/docs/user-guide/using-docker.md present in this PR. Should only review docs/install then?