-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tooling for integration tests #321
Conversation
Codecov Report
@@ Coverage Diff @@
## master #321 +/- ##
==========================================
- Coverage 83.38% 82.57% -0.82%
==========================================
Files 167 170 +3
Lines 8943 9031 +88
==========================================
Hits 7457 7457
- Misses 1161 1249 +88
Partials 325 325
Continue to review full report at Codecov.
|
.PHONY: precommit | ||
precommit: | ||
$(MAKE) gotidy | ||
$(MAKE) ci |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I couldn't find a target called ci
so I removed this. However, it would be nice to have a pre-commit hook, maybe using pre-commit.
setup: | ||
steps: | ||
- checkout | ||
- restore_module_cache |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
save_module_cache
is not executed anywhere.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah that's getting addressed in #310.
- integration-tests: | ||
filters: | ||
tags: | ||
only: /.*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this mean it'll run on all branches and all tags?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just copied from unit tests. But yes looks like that's the case.
@@ -294,3 +317,24 @@ jobs: | |||
"body": "Link to failed job: '"${CIRCLE_BUILD_URL}"'." | |||
}' | |||
when: on_fail | |||
|
|||
integration-tests: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are using the machine executor because we want to be able to run docker? I thought enabling remote docker execution context on docker executor allowed to spawn docker containers from regular executors as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah you can spawn docker containers but you can't connect to them directly. See https://circleci.com/docs/2.0/building-docker-images/#accessing-services
It is not possible to start a service in remote docker and ping it directly from a primary container or to start a primary container that can ping a service in remote docker.
$(GOTEST) $(GOTEST_OPT_WITH_COVERAGE) ./... && \ | ||
go tool cover -html=coverage.txt -o coverage.html ); \ | ||
done | ||
@$(MAKE) for-all CMD="make do-unit-tests-with-cover" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Why do we want to run each module separately now?
- Does this mean we'll need to upload coverage reports from each module to coveralls now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Why do we want to run each module separately now?
It was already being done separately, it's just that the looping was done in bash inside the makefile vs. calling for-all. Thought it'd be better to use for-all and also have the make target available in common so you can run it on an individual module if desired.
- Does this mean we'll need to upload coverage reports from each module to coveralls now?
I think that was already the case. It already generated a coverage.txt inside each module which codecov collects at the end of circleci run.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. For some reason I thought we used to run a single go test ./...
command to test the entire repo in a single run.
.PHONY: integration-tests-with-cover | ||
integration-tests-with-cover: | ||
@echo $(INTEGRATION_TEST_MODULES) | ||
@$(MAKE) for-all CMD="make do-integration-tests-with-cover" ALL_MODULES="$(INTEGRATION_TEST_MODULES)" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see we've added build tag to the integration test below. Do we need to specify modules like this integration tests? Could we run only tests with integration build flag here and skip the same ones in unit tests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You mean have a way to automatically detect which modules contain a // +build integration
directive? It's possible with a regex but seems hacky. I don't know of a simple way to do it and didn't want to spend time to optimize too heavily given it's still early days for integrations tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't think we need to worry about skipping anything for unit tests since pretty much all modules should have unit tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Never mind, you are already doing what I suggested (using --tags). The bash/make code is hard to read for me as must be obvious by now to you :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This adds make and circleci config needed for running separate integration tests. These tests are intended to still be run from go test so that they can output coverage data but they may be longer running and talk to external services like docker.
@tigrannajaryan not sure, something with codecov was out of sync, rebasing against master seemed to fix it. There's 3 new files listed without coverage because we don't appear to have been running tests for the root module before but now are. |
This adds make and circleci config needed for running separate integration tests. These tests are intended to still be run from go test so that they can output coverage data but they may be longer running and talk to external services like docker.
This adds make and circleci config needed for running separate integration
tests. These tests are intended to still be run from go test so that they can
output coverage data but they may be longer running and talk to external
services like docker.
Note: this is pulling out the build changes from #306 to breakup the PRs.