-
Notifications
You must be signed in to change notification settings - Fork 801
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Catch unit test failures in make test
#5635
Conversation
Surprisingly, this has not worked since tests were introduced to the makefile _in 2017_. We've apparently been relying on CI's grepping for go test failure lines or something. Rather trivially, without `-o pipefail` this only returns error messages from `tee`: ```bash > false | tee -a file.log > echo $? 0 ``` Which means this also does not return `exit 1`: ```bash for ... do go test whatever | tee -a file.log; done ``` Which means you can fail a test and the tests keep going, and it exits 0: ```bash > make test ... FAIL github.com/uber/cadence/common/util 0.245s FAIL ok github.com/uber/cadence/common/locks 0.252s coverage: 93.3% of statements ... > echo $? 0 ``` Which kinda stinks for local development. Failing tests can be hard to spot in the massive output of `V=1`, or dozens of lines off-screen if a lot of successful packages ran afterward. Now this will continue running tests, but will clearly fail with a list of all failed packages: ```bash > make test ... FAIL github.com/uber/cadence/common/util 0.245s FAIL ok github.com/uber/cadence/common/locks 0.252s coverage: 93.3% of statements ... Failed packages: ./common/util make: *** [test] Error 1 > echo $? 1 ``` Which is roughly how `go test ./...` works: it fails but keeps going, and does an `exit 1` if there were failures along the way (though it does not show the list of packages).
Pull Request Test Coverage Report for Build 018d5b55-f370-4608-bd47-098425223d02
💛 - Coveralls |
@@ -597,10 +608,10 @@ cover_profile: | |||
$Q for dir in $(PKG_TEST_DIRS); do \ | |||
mkdir -p $(BUILD)/"$$dir"; \ | |||
go test "$$dir" $(TEST_ARG) -coverprofile=$(BUILD)/"$$dir"/coverage.out || exit 1; \ | |||
cat $(BUILD)/"$$dir"/coverage.out | grep -v "^mode: \w\+" >> $(UNIT_COVER_FILE); \ | |||
(cat $(BUILD)/"$$dir"/coverage.out | grep -v "^mode: \w\+" >> $(UNIT_COVER_FILE)) || true; \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the grep filter for? the fact that multiline comments are hard would make me lean to towards throwing into a bash script
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Historically, it was because you can just concat multiple coverage files to have go transparently compute the combined coverage... except for those mode lines, which would cause it to error.
Nowadays I know there are better options in a few ways, but I haven't dug into them in detail yet.
Re script file: yeah, these loops are borderline or slightly past I think. But it is hopefully temporary and will also-hopefully be simpler after a modernization pass, so I'm kinda reluctant to do more than the minimum / putting it in here keeps the badness visible until it's replaced / etc :/
I could be convinced to do otherwise, just didn't feel like it was worth it at the moment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm with nit about logs
Surprisingly, this has not worked since tests were introduced to the makefile in 2017.
We've apparently been relying on CI's grepping for go test failure lines or something.
Rather trivially, without
-o pipefail
this only returns error messages fromtee
:Which means this also does not return
exit 1
:Which means you can fail a test and the tests keep going, and it exits 0:
Which kinda stinks for local development. Failing tests can be hard to spot in the massive output of
V=1
, or dozens of lines off-screen if a lot of successful packages ran afterward.Now this will continue running tests, but will clearly fail with a list of all failed packages:
Which is roughly how
go test ./...
works: it fails but keeps going, and does anexit 1
if there were failures along the way (though it does not show the list of packages).This also removes
bins
as a prerequisite for tests, as:make build
which is MUCH more complete