Hard-to-find documentation for a Docker image

[mik-laj]

Hello,

Currently, documentation for the Docker image is available in the middle of the "Production Deployment" page. This makes it difficult for the user to find this documentation. Also, this section is called "Production deployment", but this image is also used in the development environment. I think we should extract the image sections, but I'm not sure where. I am thinking of a few solutions:
a) create a new page/section in apache-airflow package. Then the documentation will available as a new menu item in the current documentation package. See "Content" section on http://airflow.apache.org/docs/apache-airflow/stable/index.html
b) create a new documentation package. Then the documentation will be available as a new item on docuemtation index

I prefer the second solution, but I am open to discussions.

It is worth considering that I am working on creating a new package for the Helm Chart. See:#14643

Probably related: #11740

CC: @potiuk @kaxil @ashb @vikramkoka @ryw

[ashb]

Docs for the helm chart feel right as a new "package" (I really like that idea!) but docker image feels more like it is about/for apache-airflow so I favour option a.

My thinking is that if you want to learn how to run Airflow, your going to be looking at the docs of apache-airflow for config reference etc, and to me docker sort of fits in there. I think the other reason why it feels wrong as separate doc package is because it's not a package we actually release.

[mik-laj]

Keycloak publishes documentation for an image separately from the documentation for a project.
Keycloak documentation index: https://www.keycloak.org/documentation.html
Keycloak server container documentation: https://github.com/keycloak/keycloak-containers/blob/12.0.4/server/README.md

Jupyter has very extensive documentation for the image and publishes it separately also.
Documentation index: https://jupyter.org/documentation
Docker image: https://jupyter-docker-stacks.readthedocs.io/en/latest/

Have you seen a project with non-trivial docker-images that describes building images in the main project documentation? I'd love to see it.

[ashb]

Good question, I'll have a look around

[ashb]

(Screenshots are of the section of the doc Table of Contents of the projects)

Elastic(search):

https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#install-elasticsearch

Prisma (v1, v2 has changed the architecture to become just a library so doesn't have a server component any longer:

https://v1.prisma.io/docs/1.34/prisma-server/deployment-environments/docker-rty1/

Keycloak is kind of a hybrid -- it's in the normal docs menu, it just links out to a README.md.

It's a fairly uncommon case that projects document much about their docker file anyway from what I can see, jupyter seems to be the exception in having it's docker docks on a totally different url to the project docs from my search just now.

[potiuk]

I think many people are asking for docker image and have hard time finding it. Separate section (option b) is fine). However ☺️ think there is one caveat with the current airflow/providers/split which it will only reinforce.

Once you are in airflow documentation, it is not easily to realize that there are more components (providers and image when it is separated out). Most people will not go to the docs of airflow via url - they will reach it via Google search or history in the browser. Then - when you are already in airflow docs, there is no way of knowing that you should click on the airflow logo to get to the airflow + providers + image directory. And it is not very accessibility friendly. So while separating it out is fine, we should - i think also add some way for people. To know that provider/image documentation is there - other than clicking at the Airflow logo. Maybe a separate section in the table of content should be added ? Something that will clearly say hat You can reach out to full set of airflow- related docs this way?

Not sure about name though? Additional documentation? Beyond Airflow ?

[mik-laj]

@potiuk Terraform has a drop down menu in the section under the link with the documentation.

They also have links to other documentation packages in the side menu.

[mik-laj]

I would like to add a few comments. I am working on more docker-composer example files so that users can test more easily.
See: #8605 (comment)
When we create a new package of documentation for all things related to Docker, it will be easier to put it together.

Another thing that I am still missing is the description of how to use DockerOperator together with Docker-compose. Many users try to use this together but have problems with it because /var/docker.socks cannot be read by the airflow user. The simplest solution is: to usetecnativa/docker-socket-proxy:0.1.1. See: #8605

So this documentation will increasingly describe how to deploy Airflow in a Docker/docker-compose environment much like the documentation for Helm Chart describes how to deploy Airflow in a Kubernetes environment.

Also, note that the documentation that describes how to build a Docker image describes building an image for multiple versions of Airflow. You can use the same instruction to build the image for Airflow 1.10 and for Airflow 2.0. The only difference is the additional flag.