[WIP] install: add docker usage documentation #811

ghost · 2019-11-22T18:22:08Z

IMPORTANT NOTES (please read, then delete):

Have you followed the guidelines in
Contributing Documentation?
Please use the title to provide a clear one-line present-tense summary of the
changes introduced in the PR. For example: "Introduce the first version of the
collection editor.".
Please make sure to mention "Fix #bugnum" (if applicable) in the description
of the PR. This enables GitHub to link the PR to the corresponding bug.

Related to iterative/dvc#2774

shcheklein

Thanks 🎉 !

First comments:

please, check the contribution guide and install the pre-commit to pass style checks
most likely section should be under the install section
please do a branch on the main repo so that it can be auto-deployed to review it
don't forget to update sidebar.json to enable it

Will read it an leave more comments a bit later :)

shcheklein · 2019-11-22T18:41:09Z

static/docs/user-guide/using-docker.md

+      && echo "zstyle ':completion:*' verbose yes" >> /root/.zshrc
+
+# Install DVC with s3 enabled
+RUN pip install "dvc[s3]"


why s3 only?

probably not the best idea to install into the global space - we advertise a pretty bad practice with this

why don't we use conda, like @casperdcl suggested in one of the tickets?

Specifically $CONDA_PREFIX/etc/conda/activate.d/ and deactivate.d/

probably not the best idea to install into the global space - we advertise a pretty bad practice with this

@shcheklein , why do you consider it a bad idea?
I don't see any problem at all, to be honest; a docker container should only be dealing with one stuff, anyways.

why s3 only?

@shcheklein , I was about to use dvc[all] but pyarrow requires CPython and it is not supporten on Python 3.8 yet; I decided to leave it as an exercise to the reader, the idea is not to provide the best Dockerfile, but just an example to display the bigger picture.

why don't we use conda, like @casperdcl suggested in one of the tickets?

There's a note already about installing DVC through conda, an example with two Dvcfiles seems excessive. I'm trying to cover just the basics.

I would think the best Dockerfile is exactly what we need to advocate. Otherwise we're saying "oh yeah we know of the existence of this thing you call docker and you can probably use it with our stuff; we've made a page about it on our website but couldn't be bothered to make it fully correct. Here haz some vaguely not-quite-correct musing of how it might sort of look. k's thx"

python 3.7 instead of 3.8? Don't see any need to use bleeding edge...

@casperdcl , I see your point; but I don't think there's a correct image 🙁 . It may vary from user to user, this is sort of like a baseline.
Also, this Dockerfile is for development only, "production" is such a hard thing to get right.

@casperdcl , don't get me wrong, I'd be happy to have a more sound version of this image)

open to discussion and commits, as always)!

@MrOutis so why should it be Python 3.8 then, let's install 3.7.

why do you consider it a bad idea?

Probably not a big deal for Docker - since it's already an isolated environment. It's just publishing it this way in docs can be considered as an instruction to do the same outside Docker, so was thinking if we should use apt-get for example, or conda or some other sane way to deliver the tool. May be it's just my bias - "never ever do sudo pip install something" :)

@shcheklein , we are running with root 😅 , we would need to start by creating a specific user for running the application, installing sudo and then add the user to the sudo group. Also, allowing to sudo without password, so editing the sudoers list 😬

Not sure if this is the best practice, would need to research

I think Ivan meant to NOT ever use sudo before pip (on global env). And suggested installing DVC under a virtualenv, even if its a Docker image.

I don't think that would be useful though, because you kind of expect your entire container to have DVC available everywhere if you're installing it that way.

static/docs/user-guide/using-docker.md

shcheklein · 2019-11-22T18:45:58Z

static/docs/user-guide/using-docker.md

@@ -0,0 +1,108 @@
+# Using Docker
+
+Currently we don't provide any official Docker image, however you can easily


If we have the Docker file ready, why don't we create an Image?

@shcheklein , being honest, I think that having an official image would bring more harm than benefits (maintenance burden, releases, etc.).

This is only an example for the ones who are completely lost at the topic. It is not intended to serve as a reference for best practices on writing Docker images using dvc.

I don't think there's much point documenting anything other than a reference for best practices... 😕

publishing a complete Dockerfile like this is also a burden - we'll need to keep it updated, right?

if we see some value in Docker I would create that image (it's quite straightforward to make it part of CI) or put a Dockerfile under Git and make a doc on how to start using it. It's quite strange to publish a complete Dockerfile this way.

@casperdcl not sure I got what way your argument is going :)

Yes trying to explain my ethos here: #811 (comment)

shcheklein · 2019-11-22T18:48:17Z

static/docs/user-guide/using-docker.md

+you make sure your files are still writtable. For more information:
+https://docs.docker.com/engine/security/userns-remap/
+
+### Dockerfile


would be beneficial to just provide a link to the file and put it into Git so that we can iterate with a regular process?

the section should be more focused on philosophy (like above ^^) that explains certain decision, how to start using it, proving links, etc

would be beneficial to just provide a link to the file and put it into Git so that we can iterate with a regular process?

for the user I don't think it would bring significant value, and for us, well, I don't mind keeping the Dockerfile example on the docs)

the section should be more focused on philosophy (like above ^^) that explains certain decision, how to start using it, proving links, etc

for the user I don't think it would bring significant value

At least it's just easier to get that file, instead of copying it. It's the way code files are usually distributed, right? :)

@shcheklein , all right, let's do this)

where should I put the example Dockerfile?

DVC core? At least it's a better place than the website and docs :) Let's imagine we decide to package an image via CI - we will be doing it probably in DVC core, or in a separate repo. Probably not in dvc.org.

Still not sure that we want to do this, but let's try, anyways 😅

static/docs/user-guide/index.md

static/docs/user-guide/using-docker.md

ghost · 2019-11-22T22:58:36Z

@casperdcl, here's whats stopping me from submitting an official image:

Should we create a user specifically to run dvc ?
How to map the user's id and its group id to the user running the container?
How and who will manage the release process? Should it be automated?
What base image should we use and why? (python, conda, alpine, flavor-of-the-month, etc.)
Is it going to be used interactively (like an SSH session) or running a container for each invocation (ENTRYPOINT ["dvc"])?
Is this image going to be used on Development or Production? Does it matter?

🙂

casperdcl · 2019-11-22T23:12:07Z

I'm not fussed about an image atm, but would like to see some code for an image. Priorities:

install dvc includine deps
install dvc[all] including deps
install optional extras (zsh syntax highlighting, inputrc tweaks, etc)

lower priority:

mess around with UIDs
provide an image (can be automated on travis, including building a second image with dvc[tests] before deploy etc)

static/docs/user-guide/using-docker.md

shcheklein · 2019-11-23T01:15:49Z

@MrOutis

💯 percent. I would expect even that the value we provide with this Docker/image (they are the same for me more or less since we'll have to support them) be the first and most important part of this document and should determine exact place and the structure.

It seems also to me that Django is not a correct example. I know that people package command line tools with Docket to run them from the host machine w/o dealing with setup and stuff. In case of DVC it's not clear if it's needed.

jorgeorpinel · 2019-11-23T04:53:18Z

static/docs/install/docker.md

+```dockerfile
+FROM python:3.7


Maybe explain why 3.7, since the first example is just python.

static/docs/install/docker.md

jorgeorpinel · 2019-11-23T04:56:46Z

static/docs/install/docker.md

+#  Set Zsh as the default shell
+SHELL ["/usr/bin/zsh", "-c"]


Why change the default? (Bash)

I assume to accommodate the custom zsh syntax config... Speaking of which, it would be nice to provide equivalent bash config if possible

Which goes to my point. Probably an unnecessary complication for a Dockerfile sample.

@casperdcl , bash is not as customizable as zsh, completion is really basic 😬

I would agree given the argument about providing a better experience, but I think most people are used to Bash, so not sure. It's not like we recommend Zsh over Bash in our docs. This one may be unnecessary.

Moved to iterative/dvc#2844 (review)

static/docs/install/docker.md

jorgeorpinel · 2019-11-23T04:58:51Z

static/docs/install/docker.md

+
+# vim: ft=dockerfile


Not familiar with vim. What does this do? Just curious.

forces Dockerfile syntax highlighting for the whole file when editing in vim

I see, thanks!

In that case I would remove this. Why provide something editor-specific?

@jorgeorpinel , it is nice to have it)

I guess it doesn't hurt but still, this is not related to the shell experience, or DVC, or Docker, so I would remove it. Should we vote @shcheklein @casparjespersen? So far it seems like it's Ramon for, myself against.

Moved to iterative/dvc#2844 (review)

jorgeorpinel · 2019-11-23T05:00:30Z

static/docs/install/docker.md

+$ TAG="dvc:development"
+$ docker build -t $TAG .


Need for tag?

🤔 yep, building a container with a tag is a good practice

OK! But how about just using development or dev? dvc will probably be in the image name. (I read https://cloud.google.com/solutions/best-practices-for-building-containers#properly_tag_your_images.)

p.s. Maybe use an example image name somewhere in this doc?

casperdcl · 2019-11-23T17:56:19Z

I feel a little more clarification is required on my part.

Personally I do not envision actually running a dvc docker image in order to dvc init/add/status etc as an alternative to installing within a python env (e.g. iterative/dvc#2760 (comment)). Perhaps it may be useful in a larger image as a tool to manage automated pushing/pulling data in the same way as git can be used as a replacement for wget/curl, but that's not my main concern right now.

Personally I would find a docker image useful (and would like it maintained) to be confident of how to install things. At the moment the first thing I tend to look at to figure out how to install something properly (regardless of how good a project's install documentation thinks it is) is .travis.yml/.circleci/config.yml. If available I definitely look at Dockerfiles instead. @tensorflow/dockerfiles/gpu.Dockerfile for example is small and IMHO can't be beaten by any amount of installation documentation -- and oh my goodness there is a lot of it -- even if I never use that file. I even made a minor change to that file which I think makes a world of a difference to ease native system installation.

Even if in our case dvc would have a tiny Dockerfile, it's still useful to potential end users to demonstrate how easy it is to install. And it turns out it's not completely trivial -- there are runtime non-python dependencies on at least libc, libffi, and git. There are a bunch of build dependencies. There are python deps and dvc != dvc[all], both of which exclude both ssh_gssapi and hdfs. There are bash completion scripts and zsh syntax highlighting. inputrc tweaks. Probably also more things unlisted... At the very least a maintained Dockerfile is a good way for devs to concisely document major new features/dependencies in a more readable way than CI config files.

It's possible that a maintained snapcraft.yml (iterative/dvc#2778) satisfies what I'm looking for, but I still think a Dockerfile presents cleaner installation documentation.

casperdcl

it seems odd that the whole file seems to be duplicated (install/docker and user-guide/using-docker) - isn't there a template or redirection mechanism which can be used instead?

Sorry that some of my comments appear in the relevant threads above as well as duplicated below... Silly GitHub UI design where Ctrl+Enter on a PR comment "starts a review" rather than appends to the selected conversation thread.

casperdcl · 2019-11-23T18:01:54Z

static/docs/install/docker.md

+#  Set Zsh as the default shell
+SHELL ["/usr/bin/zsh", "-c"]


I assume to accommodate the custom zsh syntax config... Speaking of which, it would be nice to provide equivalent bash config if possible

casperdcl · 2019-11-23T18:03:26Z

static/docs/install/docker.md

+
+# vim: ft=dockerfile


forces Dockerfile syntax highlighting for the whole file when editing in vim

casperdcl · 2019-11-23T18:05:44Z

static/docs/user-guide/using-docker.md

@@ -0,0 +1,108 @@
+# Using Docker
+
+Currently we don't provide any official Docker image, however you can easily


Yes trying to explain my ethos here: #811 (comment)

ghost · 2019-11-25T17:23:40Z

Thank you all for your input and interest!

shcheklein · 2019-11-25T17:46:15Z

... but I still think a Dockerfile presents cleaner installation documentation.

@casperdcl we have installers on Windows, Mac OS (including brew) that take care of everything. Do they solve the problem? Linux is a bit more complicated (due to variety of systems as far as I understand), but apt and rpm packages should solve most of the problems as well, right?

I'm just trying understand better the potential value the Dockerfile/image would create. Also, @MrOutis I still have a concern that if we don't integrate it into CI it would cost a lot of confusion to users pretty soon since they will pretty soon see an outdated version of it.

ghost · 2019-11-25T17:59:15Z

Even if in our case dvc would have a tiny Dockerfile, it's still useful to potential end users to demonstrate how easy it is to install. And it turns out it's not completely trivial -- there are runtime non-python dependencies on at least libc, libffi, and git. There are a bunch of build dependencies. There are python deps and dvc != dvc[all], both of which exclude both ssh_gssapi and hdfs. There are bash completion scripts and zsh syntax highlighting. inputrc tweaks. Probably also more things unlisted... At the very least a maintained Dockerfile is a good way for devs to concisely document major new features/dependencies in a more readable way than CI config files.

@casperdcl:

inputrc is usually like that on a desktop environment (i.e. ubuntu), we need to tweak it because the Docker doesn't provide it
libc, libffi, git are dependencies that are super common (already provided in some operative systems), installation is easy.
We already provide instructions for autocompletion, both Bash and Zsh: https://dvc.org/doc/install/completion
AFAIK, you don't need hdfs or ssh installed to use dvc[all]

but I still think a Dockerfile presents cleaner installation documentation.

Maybe we can start by improving the current installation docs? 🙂

ghost · 2019-11-25T18:09:31Z

I'm just trying understand better the potential value the Dockerfile/image would create.

@shcheklein , it doesn't bring any value to me, and if it is only for documentation purposes, I'd say that is better to pinpoint what could be improved in our current installation docs.

jorgeorpinel · 2019-11-25T19:12:39Z

static/docs/install/docker.md

+```console
+$ TAG="dvc:development"


May need to use ```dvc here given our docs engine automatic highlighting styles. Please check how each looks running the docs locally.

shcheklein · 2019-11-25T19:43:38Z

it doesn't bring any value to me, and if it is only for documentation purposes, I'd say that is better to pinpoint what could be improved in our current installation docs.

so, why are we doing this PR then? :)

I can imagine that if we had a Dockerfile (with comments) in Git that we run as part of CI and just put a link to it in the relevant (install/linux) doc it would be better (in terms of support) and more useful than creating a separate document with an embedded file that we would have to support in a peculiar way.

ghost · 2019-11-25T20:09:00Z

so, why are we doing this PR then? :)

I thought I was the only one that who didn't see value on this one

casperdcl · 2019-11-25T22:33:30Z

@shcheklein yes, if we have a recommended way to install dvc for each OS that is a single installer file or command (such as apt get install dvc) which is guaranteed to install every single possible optional subfeature and dependencies then I won't care about installation instructions or Dockerfiles.

If any amount of user setup is required (pre-install deps, config, auto completion etc) then I only want to look at 1 code file. This could be a single shell script that sets up absolutely everything, or a Dockerfile. If you don't think either is a good idea I will continue to look at .travis.yml no matter how great anyone thinks the installation instructions on the website are now or at any point in future :)

shcheklein · 2019-11-25T22:38:48Z

@casperdcl thanks! kk, got your point. So it's a nice addition for the installation docs. And probably even apt-get does not cover everything. How do we decide what image do we provide it for? Ubuntu or something else? It also takes care of Linux only, right?

I still think that dvc.org is not the right place to host the Dockerfile. It's the right place to have a link to an image or a Git-controlled Dockerfile that is being regularly tested if we go this way.

casperdcl · 2019-11-25T22:40:24Z

libc, libffi, git are dependencies that are super common

That is in no way the same as saying they are guaranteed to exist. apt get install dvc would definitely have to include these as deps. This should be the same here.

I'll repeat my earlier sentiment that I'm not sure we need a Dockerfile at all (whether embedded in a webpage or otherwise). But if we do provide one it should be complete and well-written (which is not the same as saying correct).

casperdcl · 2019-11-25T22:44:10Z

@shcheklein I agree in the sense I'd recommend a Dockerfile in the main repo (possibly in a subfolder) and have it added to CI tests (both of which I can easily help with); but I think the meta-discussion of "do we really need docker anything" needs to be resolved first.

shcheklein · 2019-11-25T22:51:12Z

@casperdcl @MrOutis so, considering that users asked this, that apt-get does not cover everything (yet?), I'm in favor of adding this. Not in this way though, Ramon. Let's do this in the core repo. Otherwise it just breaks the point and becomes yet another not-ever-reproducible piece of Markdown. Also, we should clearly communicate what it's created for when we put a link to it - "end-to-end reproducible instruction to install everything you need to start with DVC" or may be @casperdcl can advise on a better message.

ghost · 2019-11-26T00:03:33Z

@shcheklein: iterative/dvc#2844

ghost · 2019-11-26T00:10:42Z

I'll remove myself from the assignment and continue to work on other stuff.
The idea was to ignite the discussion with a more hands-on approach.
I'll close this PR in favor if this one: iterative/dvc#2844
@casperdcl , do you want to continue the work on this? 🙂

user guide: add docker usage documentatino

0ffd189

ghost requested a review from jorgeorpinel November 22, 2019 18:22

shcheklein temporarily deployed to dvc-org-pr-811 November 22, 2019 18:22 Inactive

ghost requested a review from efiop November 22, 2019 18:22

ghost mentioned this pull request Nov 22, 2019

provide docker image(s) iterative/dvc#2774

Closed

shcheklein requested changes Nov 22, 2019

View reviewed changes

shcheklein reviewed Nov 22, 2019

View reviewed changes

static/docs/user-guide/using-docker.md Show resolved Hide resolved

shcheklein reviewed Nov 22, 2019

View reviewed changes

static/docs/user-guide/using-docker.md Outdated Show resolved Hide resolved

shcheklein reviewed Nov 22, 2019

View reviewed changes

jorgeorpinel reviewed Nov 22, 2019

View reviewed changes

static/docs/user-guide/index.md Outdated Show resolved Hide resolved

jorgeorpinel suggested changes Nov 22, 2019

View reviewed changes

static/docs/user-guide/index.md Outdated Show resolved Hide resolved

casperdcl changed the title ~~user guide: add docker usage documentatino~~ user guide: add docker usage documentation Nov 22, 2019

casperdcl reviewed Nov 22, 2019

View reviewed changes

static/docs/user-guide/using-docker.md Outdated Show resolved Hide resolved

casperdcl reviewed Nov 22, 2019

View reviewed changes

static/docs/user-guide/using-docker.md Outdated Show resolved Hide resolved

shcheklein temporarily deployed to dvc-org-pr-811 November 22, 2019 22:08 Inactive

This comment has been minimized.

Sign in to view

Mr. Outis added 3 commits November 22, 2019 18:31

user guide: add using docker to the sidebar and adjust index

eab9504

using docker: add reference for zstyle and install all deps

58367ae

markdown: register only the new languages

f216f40

ghost commented Nov 23, 2019

View reviewed changes

static/docs/user-guide/using-docker.md Outdated Show resolved Hide resolved

ghost force-pushed the docker-docs branch from 37d97c9 to ef7ce3e Compare November 23, 2019 00:37

shcheklein deployed to dvc-org-pr-811 November 23, 2019 00:37 View deployment

💅 spell checks

ef7ce3e

ghost changed the title ~~user guide: add docker usage documentation~~ [WIP] install: add docker usage documentation Nov 23, 2019

jorgeorpinel reviewed Nov 23, 2019

View reviewed changes

static/docs/install/docker.md Show resolved Hide resolved

jorgeorpinel reviewed Nov 23, 2019

View reviewed changes

static/docs/install/docker.md Show resolved Hide resolved

jorgeorpinel reviewed Nov 23, 2019

View reviewed changes

casperdcl suggested changes Nov 23, 2019

View reviewed changes

weekly-digest bot mentioned this pull request Nov 24, 2019

Weekly Digest (17 November, 2019 - 24 November, 2019) #815

Closed

jorgeorpinel reviewed Nov 25, 2019

View reviewed changes

ghost mentioned this pull request Nov 26, 2019

docker: add example Dockerfile iterative/dvc#2844

Closed

ghost closed this Nov 26, 2019

jorgeorpinel deleted the docker-docs branch January 23, 2020 17:01

jorgeorpinel mentioned this pull request Feb 9, 2020

installation troubleshooting #983

Closed

iesahin mentioned this pull request Mar 15, 2021

mnist tutorial failing iterative/katacoda-scenarios#15

Open

This pull request was closed.

		@@ -0,0 +1,108 @@
		# Using Docker

		Currently we don't provide any official Docker image, however you can easily

		```dockerfile
		FROM python:3.7

		```console
		$ TAG="dvc:development"

[WIP] install: add docker usage documentation #811

[WIP] install: add docker usage documentation #811

Conversation

ghost commented Nov 22, 2019

shcheklein left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost Nov 22, 2019 • edited by ghost Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Nov 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost Nov 23, 2019 • edited by ghost Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ghost commented Nov 22, 2019 • edited by ghost Loading

casperdcl commented Nov 22, 2019 • edited Loading

shcheklein commented Nov 23, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Nov 25, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Nov 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

casperdcl commented Nov 23, 2019 • edited Loading

casperdcl left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Nov 25, 2019

shcheklein commented Nov 25, 2019

ghost commented Nov 25, 2019

ghost commented Nov 25, 2019

Choose a reason for hiding this comment

shcheklein commented Nov 25, 2019

ghost commented Nov 25, 2019

casperdcl commented Nov 25, 2019

shcheklein commented Nov 25, 2019

casperdcl commented Nov 25, 2019

casperdcl commented Nov 25, 2019 • edited Loading

shcheklein commented Nov 25, 2019

ghost commented Nov 26, 2019

ghost commented Nov 26, 2019

ghost Nov 22, 2019 •

edited by ghost

Loading

jorgeorpinel Nov 23, 2019 •

edited

Loading

ghost Nov 23, 2019 •

edited by ghost

Loading

ghost commented Nov 22, 2019 •

edited by ghost

Loading

casperdcl commented Nov 22, 2019 •

edited

Loading

jorgeorpinel Nov 25, 2019 •

edited

Loading

jorgeorpinel Nov 24, 2019 •

edited

Loading

casperdcl commented Nov 23, 2019 •

edited

Loading

casperdcl left a comment •

edited

Loading

casperdcl commented Nov 25, 2019 •

edited

Loading