Proposal: Functional test POC for version and gitcommit

[chanwit]

As @aluzzardi suggested in #376 that Swarm requires functional tests, this PR serves as a POC of doing it automatically using a VirtualBox-based framework from http://github.com/chanwit/marine.

To enable functional testing, follow steps below:

$ marine prepare -i=files/ubuntu-14.10-server-amd64.ova -o=files/ubuntu-0.ova -f
INFO[0000] Removed "ubuntu"
INFO[0019] Imported "ubuntu"
INFO[0020] Modified nic2 for "ubuntu"
INFO[0021] Started "ubuntu"
INFO[0041] VM "ubuntu" ready to connect
INFO[0041] Sudo: "wget -qO- --no-check-certificate https://get.docker.com/ | bash"
INFO[0245] Sudo: "service docker start"
INFO[0246] Sudo: "docker pull golang:1.3"
INFO[0365] Stopping "ubuntu"
INFO[0380] VM "ubuntu" is now poweroff
INFO[0380] Snapshot "ubuntu/origin" taken
INFO[0500] Exported "ubuntu" to files/ubuntu-0.ova
INFO[0502] Removed "ubuntu"

$ export SWARM_FUNC_TEST_IMAGE=files/ubuntu-0.ova

$ SWARM_FUNC_TEST=true go test -v github.com/docker/swarm/functest
=== RUN TestVersionCommitId
INFO[0046] Imported "ubuntu"
INFO[0047] Modified nic2 for "ubuntu"
INFO[0047] Snapshot "ubuntu/origin" taken
INFO[0048] Modified nic2 for "box001"
INFO[0048] Clone: Machine has been successfully cloned as "box001"
INFO[0048] Started "box001"
INFO[0068] VM "box001" ready to connect
INFO[0068] Sudo: "git clone --depth 1 http://github.com/docker/swarm"
INFO[0084] Sudo: "cd swarm* && docker build -t swarm:build ."
INFO[0094] Sudo: "docker run --rm -t swarm:build --version"
INFO[0094] Stopping "box001"
INFO[0107] VM "box001" is now poweroff
INFO[0108] Removed "box001"
--- PASS: TestVersionCommitId (108.02 seconds)
PASS
ok      github.com/docker/swarm/functest        108.027s

Anyway, codes of functional tests do not affect Swarm itself directly, so it can totally be optional.
/cc @vieux

[chanwit]

It seems you guys don't buy this PoC. I shall close it then. Cheers!

[aluzzardi]

@chanwit: Hey, sorry, we've been busy with 0.1.0 and didn't get the time to review PRs except bugfixes.

I'll take a look soon.

That being said, I initially was considering writing integration tests in shell, using something like https://github.com/sstephenson/bats

I believe the integration tests will be mostly starting docker daemons, swarms, running commands, etc etc so shell scripting sounds like a good candidate for this.

Thoughts?

[chanwit]

Dont worry @aluzzardi. I am happy to spend my time doing this :)

I will check it out and get back to you.

[david-guenault]

@chanwit @aluzzardi did you consider taking a look at cucumber to get a more maintanable set of functional tests ? => https://cukes.info/. I used it for webservices/webapplications functional testing and worked like a charm.

[chanwit]

@david-guenault thank you for pointing out. I am finding a way to use docker machine to test swarm at the moment. But it needs some tweak IMO.