Add support for running docker containers by using Docker Engine API

[donbeave]

Hello, guys!

This is related to that issue:
spring-cloud/spring-cloud-dataflow#3966

I have checked that LocalAppDeployer is using docker command to start containers, but if use springcloud/spring-cloud-skipper-server docker image, the skipper will fail because there is no docker command inside that container. That's why I'm thinking about implementing a new AppDeployer, DockerAppDeployer, which will use Docker Engine API (https://docs.docker.com/engine/api/v1.40/) to start containers. Docker Engine API is just a REST protocol, which available in /var/run/docker.sock, it also available for Windows platform as well, not only Linux/macOS.

@jvalkeal please let me know, what do you think about this idea? I'll happy to come with PR if you find it useful too.

[jvalkeal]

I remember looking this around early 2017(docker support were added late 2016) as I'm not a big fan of using docker client commands. Don't remember why I abandoned it at a time but I think it was because direct use of rest api was just so super awkward and there were no java libs helping with an issue. I'd be happy to see support of using api directly.

Have you looked what it would take to do this? There seem to be docker-java lib which might help but I've never used it.

[donbeave]

Yeah, this library https://github.com/docker-java/docker-java looks good to me, I didn’t use it too, but looks like it’s just a wrapper around that socket API.

It doesn’t seems difficult to change to use that library directly, I’ll prepare a PR with some changes and we can discuss more detailed.

[jvalkeal]

lol, just realised that my colleague @bsideup is one of its maintainers so I feel even more relaxed to use that lib.

@tzolov did some experiments to mount socket to a dataflow/skipper containers to use a deployer and while it kinda worked it was super awkward as you also needed docker client.

Exposing docker engine as a tcp would also give super easy way to use it directly from a containers as mapping a port from a host is much easier than getting a socket there.

[bsideup]

@jvalkeal that's correct :) Thanks for your trust 😊 docker-java is Testcontainers' underlying library and is being used by many projects.

Tip: consider using either OkHttp transport (or to-be-released Apache HttpClient5 one) as the most feature rich. There is also one based on Netty but it requires native dependencies and does not work with npipe sockets.

Exposing docker engine as a tcp would also give super easy way to use it directly from a containers as mapping a port from a host is much easier than getting a socket there.

FYI in Docker world, the sockets are preferred over TCP because they have better security characteristics.

[tzolov]

Hi there,
The Docker Engine API and docker-java seems the right way to go forward, assuming it will support multiple platforms gracefully.

Until this happens though there is an easy patch that would allow running Docker Apps with Docker Compose as explained here: https://docs.google.com/document/d/1gZrqF6QCVqdYVJ8LAModPr_O0dJUSGW-KSFAI1tuYcE/edit?usp=sharing

I just submitted #186 as a workaround until the Docker Engine API and docker-java solution comes around.

[jvalkeal]

@bsideup ok thx, we were gonna ping you about usage before we do something stupid with that lib :)

[tzolov]

Hi @donbeave, just wanted to check if you are still interested to contribute the suggested solution?

[donbeave]

Hi @tzolov! Actually my colleague @efimmatytsin want to take it over. We just started to adopt Spring Cloud Data Flow in @scentbird, that’s why didn’t start to work on this issue yet. We want to back to this issue in July-August. If you ok with this timeframe we will be happy to contribute, but if you need it early we also ok if you take it over.