[MRG] Define an interface for Container engines

[manics]

Summary

Closes #682

• Abstract the interface that builds a Dockerfile (defined in repo2docker/engine.py). Engines are LoggingConfigurable- in future this could allow advanced configuration though the repo2docker config file. This is the file that needs the most review since it defines the specification for others.
• Implement docker (via docker-py) as the default implementation (see repo2docker/docker.py)

This is based on my attempt at a podman backend in #806

In general most of the Docker Client methods could be easily implemented in a Podman wrapper.

Details

The main exceptions are build() and push_image():

• The Docker client can take a tar file-like-object as the Docker context. To implement it in Podman you have to untar it into a temporary directory. This is inconvenient, though not a blocker since Python makes it easy to untar.
• Build logs: The output format used by Docker is unclear, and the content of each line is specific to Docker. In #806 I worked around it by taking whatever podman spits out and wrapping it as {"stream": line} which is enough to convince repo2docker:
  • https://github.com/jupyter/repo2docker/blob/1be717b4320c4024b4ea0f118f45e6213eefff53/repo2docker/app.py#L759-L760.
  • In this PR I've decided it's easier to just return strings, and throw an exception to indicate an error. Engine classes have a flag string_output to indicate whether they follow the Docker API in returning objects or if they return plain strings.
• Push image logs: The docker client outputs logs and progress information as events in a way that isn't portable to podman:
  • https://github.com/jupyter/repo2docker/blob/1be717b4320c4024b4ea0f118f45e6213eefff53/repo2docker/app.py#L493-L498
  • as with build() I've switched to plain strings with the option of keeping the docker format.

Remaining todos:

• Fix unit tests
• Remove commented lines/in-progress notes
• Update podman backend WIP: Podman backend #806 to implement this interface as verification of the design, either in this repo or as an external plugin with a repo2docker.engine entrypoint

Other engines

I've switched to using an entrypoint for ContainerEngine implementations. The default docker engine remains in this project. I've moved #806 (WIP: Podman backend) into a new project: https://github.com/manics/repo2docker-podman. Instructions are included in that project's README.

[manics]

This is a new dependency, but it's already used in other Jupyter projects such as jupyterhub for managing entrypoints:
https://github.com/jupyterhub/jupyterhub/blob/bc7bb5076ff2deabc7702c75dce208166d14568e/requirements.txt#L4

[manics]

pre-commit/black formats the files as Python 3.6 but this is incompatible with Python 3.5 and the above config doesn't work.
See the # fmt: off comments in repo2docker/docker.py for the alternative workaround.

[manics]

Hopefully #849 will fix this

[manics]

custom_context and fileobj and very biased towards the Docker API. However it's fairly easy in Python to untar this tarfile-like-object into a temporary directory so it can be made to work.

[meeseeksmachine]

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/brainstorming-repo2docker-action-vm-on-gcp-aws-azure/5345/16

[meeseeksmachine]

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/binderhub-for-hpc/143/18

[yuvipanda]

@manics I'm happy to merge this now if you can do a rebase - so we can catch any test failures that might happen right after we merge. I tried doing it locally and ran into a couple conflicts...

[meeseeksmachine]

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/building-a-the-littlest-binderhub/9824/1

[manics]

@yuvipanda Rebased! Just the CircleCI test that's still failing (#1050)

[yuvipanda]

@manics awesome. That's fine I think.

[yuvipanda]

@manics exciting change!!!! Thank you for working on it and pushing it through. I think once the podman backend and repo2docker have a a release, what do you think of a blog post?

[manics]

I'm happy to contribute to a blog post! I need to figure out a more reliable way to install podman as the tests have broken again: manics/repo2podman#2