Move Docker build definitions to project repositories

[abesto]

Problem

Currently all Dockerfiles live in this repository for all OpenZipkin projects, and all Docker images are released together. See RELEASE.md@71b4c251 for the current release process. This enforced synchronization sets bad expectations for rarely maintained projects (what do you mean out of date? the image is two hours old), and also potentially delays releases of more active projects. The current setup is also surprising, Dockerfiles are expected to live in the same repository as the project itself.

Possible approaches

Approach 1: in-repo

• Move the Dockerfiles and files used by them (for example in ADD commands) to the respective repository of each project
• Configuration on Quay.io for each image needs to be updated to use the Dockerfile from the new location
  • At this point the release process is still monolithic, but we've reduced surprises a bit by moving Dockerfiles to separate repos and prepared for the next steps.
• Copy-paste relevant parts of release.sh@d1e38a5580 to each project
• Configure Travis job to run the Docker build or the normal build depending on how the build was triggered (special tags for Docker builds), as described in Add multiple jobs per repository travis-ci/travis-ci#2646.

Pro: Dockerfile in the project repository, as expected
Con: Extra complexity in Travis config; no way to tell at a glance whether a Travis build is a normal build or a Docker build. This goes away if we completely get rid of release.sh, letting us remove Travis from the Docker build pipeline.

Approach 2: mini repos

• For each project with a Docker image create a new repository $PROJECT-docker
• (Same as Approach 1) Configure on Quay.io for each image needs to be updated to use the Dockerfile from the new position
  • At this point the release process is still monolithic, but we've reduced surprises a bit by moving Dockerfiles to separate repos and prepared for the next steps.
• (Same as Approach 1) Copy-paste relevant parts of release.sh@d1e38a5580 to each project
• Create new Travis job

Pro: low complexity, clear separation of Travis jobs
Con: Dockerfile in separate repository (may be a pro, depending on where you sit); lots of repositories (may be irrelevant)

Follow-up

Process

Once this is done, the release of new Docker images very clearly becomes the responsibility of the maintainers of each project. This is a good thing, but we all need to be clear and on board with this change. Versioning of images becomes a question: how do we track what versions are compatible with what other versions, if we track it at all? Is the version of the image the version of Zipkin always, sometimes, never? How do we show a breaking change in an image that comes without a new release of Zipkin? Personally I think we can put off coming up with strict answers to these questions, but let's keep them in mind going forward, and keep open the option of sitting down to find definite answers if it becomes necessary.

Tech, long term: goodbye to release.sh

In the long term, I want to get rid of release.sh. It's custom complexity that will always mean extra maintenance. Here's the functionality we need to drop or re-implement using standard tools to make it go away, and my current best idea for doing so. (Will turn this into a new issue once we decide our course)

1. Build zipkin-base
   • Migrating to Docker Hub will remove the need for scripting here
2. Bump the version of zipkin-base in the rest of the images
   • This step is made manual and the responsibility is passed to the repository of each project
3. Build the rest of the images
   • Migrating to Docker Hub will remove the need for scripting here
4. Sync all images to Docker Hub
   • Migrating to Docker Hub makes this go away completely

Tech, short term: DRY

Exploding release.sh to multiple repositories introduces a lot of code repetition, which will lead to tiny differences in each project (I'm working on a half-serious theory of evolutionary release scripts), badness all around. Ways to mitigate this badness:

• Each new release script will have a smaller scope than the original monolithic one, this can be used to remove some abstractions and complexity. I'm planning to do some of this during the initial migration.
• Some of the complexity is due to orchestrating Quay.io and Dockerhub. Dockerhub has released features since the initial setup of the Docker Zipkin build process that should enable dropping Quay.io
• If there's still a lot of repeated complexity, we can move that to a shared script in a shared repository (something like common.sh in docker-zipkin that each build process downloads at run-time). This is a different kind of complexity, let's see if it's worth it when we get there.

I intend to put a limited amount of effort into this and work on getting rid of release.sh completely instead.

[abesto]

@adriancole I'd appreciate your thoughts on this. Also: who else should be involved?

[codefromthecrypt]

It seems we should trim the weeds around this problem space, so it is less conflated with known difficulty. For this reason, I'm +1 on migrating to dockerhub before doints this. We can then re-summarize the desc.

Some other options I thought about while reading (not well thought through)

• release of docker can still be monolithic if we do submodules for the participants.
• we can use a specific docker release function w/o copy/paste if we release it, or check it out as a part of travis setup for the modules. inceptiony, but yeah.

[abesto]

Thanks for the input!

• Docker Hub first: I'm kind of on the fence about this. Your argument totally makes sense, I'll think more on it.
• Monolithic release: yes, especially if go with mini repos; I'd avoid checking out all the Zipkin source code ever. Do you think the monolithic release is a Good Thing (tm)?
• Custom release function without copy/paste: indeed, that's what pretty much what I had in mind in the last bullet-point under DRY. Glad we're on the same page there.

[codefromthecrypt]

• Docker Hub first: I'm kind of on the fence about this. Your argument
  totally makes sense, I'll think more on it.

ack. I understand these days the best repo changes a lot!

• Monolithic release: yes, especially if go with mini repos; I'd avoid
  checking out all the Zipkin source code ever. Do you think the monolithic
  release is a Good Thing (tm)?

agreed maybe submodule was a bad idea :P too bad we can't just fetch a
subtree by tag.. most of the time, there are only 2 files relating to
docker!

wrt "monolithic release" I think this is just a challenge when we publish
things like docker-compose files. I suppose we could lower the maintenance
by scaling back the version precision. Very open to better practice here..
I don't have a great answer for a modular pipeline and also a compose file.

• Custom release function without copy/paste: indeed, that's what
  pretty much what I had in mind in the last bullet-point under DRY. Glad
  we're on the same page there.

me too

[abesto]

Alright, let's do Docker Hub first. I think it's straightforward, I'll start work on it and open up things for discussion if I come to non-trivial trade-offs.