[Updated] Add quick and full integration test targets #355
Conversation
This commit adds integration test code in two phases: - quick-integration.hcl, for CI builds. Run with `make integration` - full-integration.sh, for nightly builds. This is triggered by setting CVG_FULL_INTEGRATION in the environment We are using docker for testing the same HCL file against multiple linux OSes. We need vagrant because we need an environment to test LVM modules, and it's a test of our plugin. Future Work: - Port blackbox tests to integration tests where appropriate - Set up nightly build machine, and add integration test to wercker - Right now, the docker target creates containers serially. I would like to add a target where we simulate a cluster with containers. Should we use docker compose for that? an HCL file? what?
There was a problem hiding this comment.
nice! the quick integration tests worked great for me. we can clean up some of the tasks as we introduce more features (such as #253) but this seems like a great start.
I left some other feedback around the full integration test which does not really work right now. What does everyone think about just moving forward with the quick tests so that we at least have some integration testing in place while we work on solving the remaining problems?
| @@ -0,0 +1,5 @@ | |||
| #!/bin/bash | |||
| cd $(dirname "$0") | |||
There was a problem hiding this comment.
untested but maybe try something like cd "$(dirname "$0")" || exit 1 to cleanup code climate issues
| @echo | ||
| @echo === quick integration test === | ||
| $(INTEGRATION_CMD) testing/quick-integration.hcl | ||
| @ if [ -n "$$CVG_FULL_INTEGRATION" ];\ |
There was a problem hiding this comment.
I think we should probably document the use of this env var somewhere. maybe in CONTRIBUTING.md?
There was a problem hiding this comment.
I'll add a note there, good catch
| ] | ||
| cvg.install = true | ||
| end | ||
| end |
There was a problem hiding this comment.
this is out of date with the most recent version of the Vagrant plugin.
There are errors in the configuration of this machine. Please fix
the following errors and try again:
vagrant-converge:
* The following settings shouldn't exist: bikeshed
* No value provided for hcl
There are errors in the configuration of this machine. Please fix
the following errors and try again:
vagrant-converge:
* The following settings shouldn't exist: bikeshed
* No value provided for hcl
Also, effective testing using vagrant is blocked by #356. I wonder if we should just leave this part out of the PR until we solve those problems.
There was a problem hiding this comment.
Alright, I'll remove it. I'll switch it back to echo NOT IMPLEMENTED YET, and that'll fix the code climate issue too
Since vagrant is currently blocked, we will need to add it back in at a later time. I also added in a new testing section to CONTRIBUTING that describes our various testing make targets
|
Thanks for this, Sam. We'll probably use this as a basis for integration testing in the future but I'm going to close it for now so we have a clearer view of our open PRs. |
This commit adds integration test code in two phases:
make integrationCVG_FULL_INTEGRATION in the environment
We are using docker for testing the same HCL file
against multiple linux OSes. We need vagrant because we need
an environment to test LVM modules, and it's a test of our plugin.
Future Work:
to add a target where we simulate a cluster with containers. Should we
use docker compose for that? an HCL file? what?