Use Docker for isolated testing

[e2]

Summary

Allows locally running test in a fully-provisioned Docker container.

Details

Added script/run_tests_in_docker which should pretty much do everything out of the box, including:

1. Building the image from scratch (ideally, the Aruba team should provide a base image for this on http://hub.docker.io, so people can just quickly download what they need).
2. Docker Compose file for automating the setup
3. Example .gemrc file for caching gems
4. Optimized for reruns (gems are smartly cached before testing starts - by copying the Gemfile and Gemfile.lock to "pre-bundle" with needed gems)
5. Automatically works when Dockerfile is changed (so users can just add packages/steps there and then just rerun the tests).
6. RVM is included and ready (with MRI 2.3.0 as default), so developers can easily switch to e.g. JRuby before rerunning tests.

Also, I updated the bootstrap script:

1. Refactored.
2. Logging to temp file (explained in commit)
3. More dependencies (Python and JDK)

Motivation and Context

This mostly avoids any and every issue caused by custom environments on a typical desktop. (E.g. locale, shell configuration, app versions, missing requirements before tests start, failures on travis, multiple Ruby versions, etc).

And it's just as fast as on the host, so it makes little sense not to use it. (Except maybe disk space on laptops with tiny SSD drives).

This is why I marked using Docker for tests as "recommended" in CONTRIBUTING.md.

There's no downside, especially since now Windows users can quickly tests their development tree in a Docker container (via a VM, but still much faster and convenient that through a PR+Travis).

(It may even be possible to transparently run tests in Windows - e.g. run a silent install of Ruby inside a Wine docker image - or maybe even on Windows containers, but that's beyond my interests).

How Has This Been Tested?

Should work on any Linux system with Docker and Docker Compose installed.

Environment should more or less resemble that of Travis. If not, any future differences can be easily accounted for in the Dockerfile.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[maxmeyer]

Looks good to me, but I only scanned over your changes! Will have a deeper look into it the next couple of days - maybe even tomorrow. Thanks for the helpful explanations!

First thoughts: Maybe we could add some parameter to the (non)existing bin/test-script to make it either run the test suite in docker or on local system. WDYT? Both are variants of a test run.

[e2]

First thoughts: Maybe we could add some parameter to the (non)existing bin/test-script to make it either run the test suite in docker or on local system. WDYT? Both are variants of a test run.

I have no idea what you mean by bin/test-script.

The current script/test is for Travis (mostly) and it's too brittle for local testing.
The script/run_tests_in_docker is analogous, but it's for local-development only (since it references sources directly).

Ideally, I'd get Aruba using a real TMP directory for testing, like /tmp. It would be great if the source directory could be read-only, because then you could paralellize tests using multiple containers.

Of course this only makes sense locally.

There are lots of improvements possible here, but the above is worthwhile standalone as-is.

E.g. it should be easier to select JRuby, although it isn't a bad idea to just change that in the Dockerfile and rerun the tests. But this isn't a lot of overhead, so even now it's not worth doing.

Also, the "local docker" script also supports parameters (see the changes in Contributing.md file), so they're interchangable. I wouldn't add parameters, because it would make running commands more confusing and error problem. (I use args for shelling to a bash login session already).

[e2]

Mostly, this makes local tests green - with no other changes to the codebase.

So at the very least it will help contributors working on both 1.x and pre-1.x codebases - by making their tests green regardless of setup.

[ghost]

I put together another Dockerfile based on your work, but using Alpine Linux instead. I prefer this one because of the reasons found in here: https://www.brianchristner.io/docker-is-moving-to-alpine-linux/. We will get a much smaller image.

You may want to have a look at this commit. Work done here is not finished yet: Working with proxies and gnupg is quite a hassle. Maybe I find some time tonight to finish this. Feel free to take over whatever you want.
https://github.com/dg-ratiodata/aruba/commit/1f779b3f717abfe02e52084c314ab745cf41dcfe

[e2]

We will get a much smaller image.

Including all rubies will actually make the image huge, so it defeats the purpose.

Also the base image should allow further "development". Just like alpine is the base here, it's best to have an "aruba testing environment" as the base, and installing requirements (rubies, gems, pythong, jdk, extra tools) as part of "bootstrapping", which shouldn't happen in the actual Docker image.

Also, RVM doesn't support Alpine, so this is a deal-breaker.

The idea is to prioritize developer satisfaction over file size. Alpine is based on busybox, so installing anything else is pretty much like slitting your wrists. It means tens of hours (if not hundreds) just trying to get a tool to install there.

That's why I picked Ubuntu - either every tool you'd ever want (strace, gems with compiled extensions, editors, profiling, etc.) is installable, or Docker workarounds are well known.

Just to give you a sense of the workload: if you expect RVM to work flawlessly with Alpine before there's a possibility to get docker support in testing Aruba ... then my use_gemset PR will have to wait another year until RVM catches up to using Alpine.

Also, Ubuntu isn't huge at all: 187MB (Alpine: 5MB to start, but 131MB without RVM yet). rvm requirements takes up about 130 MB by itself - there's no jumping over that, unless you don't want to allow building specific Rubies version (which makes the container a lot less convenient for development). Installing just 2.3.0 with RVM already takes up 130 MB.

My aruba_tests image is currently 700MB, but ...

1. Gems for Aruba: 74MB
2. jdk (for javac alone for Aruba::ShellScript testing): 150MB
3. Python (same thing): 23 MB
4. Zsh (same thing): 23 MB

So the Aruba-specific part is already almost 300 MB (!).

Next:

1. Ruby 2.3.0: 130 MB
2. RVM: 8 MB
3. RVM requirements (for MRI alone): 171 MB

So that's almost 600 MB for a 800MB image! Which adds, up, because the Ubuntu image is 180MB.

Relatively "tiny" by comparison. And it's very likely that anyone with a Docker installation ALREADY has the Ubuntu images available locally.

The idea is also not to replicate Travis, but to allow the developer flexibility to do what Travis can't do, which include:

1. testing one thing quickly
2. running all tests immediately
3. running custom Ruby versions
4. custom tools installed which aren't present (or usable) on Travis, such as JRuby with --debug option

If someone really wants to save space or one-time bandwidth ... then just using Travis is a better alternative.

Also, the more flexibility, the better - because it doesn't make sense to add "Dockerfile maintenance" to the list of things to do in Aruba.

So my suggestion: unless there's something horribly wrong with this PR, it would be great if you can just merge it. (Other people can add pull requests later).

The biggest "optimization" for now is: having an official base image uploaded (so Aruba doesn't have to rely on my own - though, I wouldn't mind if it did).

The reasons:

1. A base image means: Docker will be less likely to accidentally "rebuild" the image almost form scratch. (E.g. can happen during rebasing, etc.)
2. A base image means: necessities are there, and people and tweak and build on that (by modifying the local Docker image - or providing a custom one that's gitignored).
3. With a "bare-bones" base image with Ruby 2.3.0, there isn't much downloading or maintaining. Meanwhile, if javac is removed from Aruba test dependencies, all that needs to change is the local Dockerfile (while the official image stays the same). If someone needs JRuby, the image has RVM to make adding this to the local image easy.

Also, docker-compose is simply more convenient for developing and tweaking. (Because it's "smart", unlike running docker commands directly). Especially since this is a new feature, it's detrimental to try and make it "perfect" on the first shot.

(Meanwhile, tests are still failing locally without this PR being merged - so "optimizing" the Dockerfile is absolutely not a priority at this point).

A 500MB base image based on Ubuntu means only 300MB more if Ubuntu image is already present (and likely will be). This is cheap to download - and still much faster than waiting for Travis. Even an Aruba git checkout can take a bit.

Here's what I took into account when creating this image:

1. Convenience (RVM to install anything you want, Ubuntu to install any extras, editors, gems you may need)
2. Less maintenance on Aruba team's side (a "generic enough" base image can last for years while still being useful).
3. Documentation (Almost every Docker/package issue has a solution for Ubuntu images)
4. I already had an RVM-based image worked out fine - which is tricky to do. Tweaking anything can be a time sink. Switching distro base images is asking for a real-life horror story. (Especially when switching to busybox).

Also, some tests can detect Docker and just clobber whatever they want without any regards for safety. E.g. Removing home directory? Not an issue. Files created in wrong directory? Not an issue. The worst case scenario is losing uncommited changes in sources.

[e2]

Out of curiosity I checked out: https://github.com/nicdoye/alpine-rvm-gcc

(It shows some of the wrist-slitting necessary to get things done).

Result of running it: 570MB image. (RVM install alone took a whopping 430MB - probably because everything has to compiled for Alpine, since nothing is binarily compatible with it.

Conclusion: Alpine is great for routers and microservices ... but ill-suited for any kind of development.

[ghost]

This doesn't make sense. I would like to be able to build the image locally and this file is required by `Dockerfile.

What I suggest:
Create a directory .docker/latest (latest -> docker tag) and place the file there. This is practise I normally use to have files for each tag available.

[ghost]

Thanks for your work and comments. They are quite helpful. I now think using ubuntu is a good choice. Plus: Have a look at the news there's some partnership between canonical and microsoft which will result in running ubuntu natively on windows.

Please address my comments and I'm going to merge your PR.

[ghost]

Do we really need docker compose for that? This add another dependency for new developers. This should be simple enough to just have a rake task for that. Feel free to modify the task I added to master recently.

[e2]

I've reworked things. This is still a work in progress.

1. The CONTRIBUTING.md is way too long right now.
2. I split the Dockerfile into two: "base" (Dockerfile.base) for Aruba team (for a "long-term" base image) and "tests" for people to add custom addons themselves
3. The "gemrc" thing is no inside the Dockerfile
4. The "split" is reflected in the new docker-compose.yml, so it's easy to build the base image, the test one, or run the test (I've kept the names you provided)
5. Slightly tweaks to the Rakefile, but mostly making the tasks a bit less intimidating.
6. Slightly updated example in CONTRIBUTING.md showing how things are still a bit of a mess when it comes to invocation
7. I think an environment variable is better to choose between docker and non-docker tests. It's more explicit and doesn't mess around with the arguments.

Issues:

1. Docker Compose works flawlessly for me.
2. The Rakefile - not so much (It works, just sometimes things aren't right - you can confirm or not).
3. I forgot to comment out the gemrc thing...

[e2]

Two more things I didn't address maybe:

1. Docker Compose is a single binary in Python you can copy to your ~/bin directory. (Releases on Github).
2. The reason for not using Rake solely for managing images is: you end up storing the configuration inside the Rakefile. It's just much cleaner to use a docker-compose.yml
3. If you're using JRuby, jruby should be a symlink target for ruby. RVM has lots of cool features for scripting and picking the right version. It does a really solid job, as long as everything outside it is configured correctly (login shells, no PATH overrides in login dotfiles, variables properly exposed to TMux, using RVM wrappers and environment files, etc.). Managing "Ruby variables" is complex in of itself. I just can't imagine not using RVM.

[maxmeyer]

FYI https://hub.docker.com/r/feduxorg/centos-geminabox/ :-)

[e2]

I think I'm using some derivative of geminabox right now. (as a Docker micro-service, of course).

[maxmeyer]

STDOUT is required? STDOUT.puts should be the same as puts.

[maxmeyer]

@e2 Can you rebase I changed quite a lot in CONTRIBUTING.md. I think for now let's use docker-compose.yml only and don't depend on docker-compose.

Regarding the Rakefile:
I use this kind of task in my docker-projects. Can you paste an error message or something similar?

Mmmh. Also I see the good things docker compose brings in, I'm not fully convinced since it's an extra dependency. For now I would like to leave it away.

Regarding the dockerfiles:
Would be a "big" deal to only have a single file? I find it confusing to have two of them. I see our discussion about size, but although it's java in there, I would prefer "simplicy" over size.

Rest of changes:
I like them. Let's fix the comments above and we're good to go.

[e2]

I'm not fully convinced since it's an extra dependency.

I'd rather convince you. "rake people" wont care about docker - unless they're getting failures. So they don't care. Meanwhile, "docker people" will only benefit from docker-compose.

Thirdly, docker-compose helps Windows users get started - docker-compose means a lot less cryptic errors for people who don't get docker much.

Also, editing a docker-compose.yml is much more intuitive than tampering with a Rakefile.

Ideally, I'd say the Rakefile should process the docker-compose.yml to get variables.

Also, the Rakefile isn't too helpful in providing custom options to docker. Meanwhile, the docker-compose.yml is very well documented.

The reason for splitting files in two is: the base one is intended to be an official, downloadable image to speed up preparing a checkout for testing - because the download should be a lot faster that building that base image from scratch.

Also, once there's a official base image, the whole Dockerfile.base can be removed. (and moved to a separate repo altogether).

Your argument about "simplicity" could be applied to docker-compose - way simpler than docker or tweaking the Rakefile. And it isn't a huge dependency in size, either. Installing is trivial too.

Docker-compose isn't a default/"forced" dependency here. To me it would be like having a .rubocop.yml file without Rubocop gem dependency in development.

In 2016 when human time is more precious than ever, installing docker compose and tweaking the docker-compose.yml will always take less time than tweaking the Rakefile and potentially making mistakes (which I've done myself).

Not to mention: passing arguments to a Rake take is the most shocking surprise ever...

[e2]

I'll rebase when I get back home.

[e2]

Changes:

1. I moved the docs on testing to TESTING.md (to it can be further reorganized/simplified/expanded without making CONTRIBUTING.md seeming too long)
2. I reorganized the Rakefile and made it use the config from docker-compose.yml directly. Basically, I "implemented" docker-compose inside the Rakefile. Which just shows that if the Rakefile used docker-compose directly, there'd be a lot less code there.

[e2]

Note: I haven't tested the Rakefile yet - previously it didn't work well because it didn't handle the second image.

Here's the "intended" workflow overview for clarity:

1. Official base image cucumber/aruba is fetched automatically by Docker. (Currently: it's built using Dockerfile.base since no official base image exists).
2. Developer image is built - mostly for adding "expensive" dependencies, custom gemrc, pre-caching gems, installing a custom version of Ruby (e.g. JRuby), installing custom user tools for debugging (e.g. strace), maybe a custom editor and adding/changing anything missing from the base image.
3. Running a container based off of the developer image - by default running all the tests.

[maxmeyer]

Thanks @e2 I'm going to merge this, but I will merge the Docker*.files into one. The "testing.md" is not part of this PR, correct?

[e2]

The "testing.md" is not part of this PR, correct?

I added the TESTING.md file (happened probably because it isn't needed for tests/specs to work).

Thanks @e2 I'm going to merge this, but I will merge the Docker*.files into one.

It may be seem "less complex", I get that.

But, if you make the base image public, you can just remove the Dockerfile.base altogether. (It will be available there anyway). Just like you don't need to build the docker/ubuntu image yourself.

So if you create a cucumber/aruba image and remove Docker.base, you'll still have one Docker file too.

The only reason I didn't do this myself is: I don't have access to create and publish an "official" docker image for the base.

That's the ONLY reason.

[ghost]

(Partly) incorporated by #382.