Send Slack notification for UI/Unit test failures #9853
Conversation
You can test the changes from this Pull Request by:
|
jostnes
added
the
category: tooling
There was a problem hiding this comment.
I updated the 3 .yml files under .buildkite to point to the latest version. I'm not sure if that's the right way but please me know if this shouldn't be the case.
Yep that's the best way to go 👍 🙇
- Purely technically speaking, for this to work you only needed to update the
.ymlfile that would call commands that used the newannotate_test_failures, and sincepipeline.ymlis the only one callingrun-ui-tests.shandrun-unit-tests.sh, this was the only one that required to use the latest plugin version. - Practically speaking, it's way nicer and better practice to try to use the same version of the plugin on all our pipelines, for consistency (and to avoid confusing issues where CI commands would work in a pipeline but not another and it could take us quite a long time to realize that it was because of plugin versions being different). So thank you for doing that (as I see that the versions were not consistent before your PR already)!
- As a side note just FYI, we hope that at some point in the future we could define that value in a single place and have all
.ymlfiles refer to the same source of truth for the plugin version… which would solve the risk of inconsistent versions between the.ymlfiles. But we're not there yet 😛 so in the meantime, manualy updating the version consistently in all.ymlfiles is the way to go 🙂
| if [ "$BUILDKITE_BRANCH" = "trunk" ]; then | ||
| annotate_test_failures "fastlane/test_output/WooCommerce.xml" --slack "build-and-ship" |
There was a problem hiding this comment.
Open question: I wonder if you wouldn't want to get Slack notifications for any release/* branch too? 🤔
There was a problem hiding this comment.
That's a good question. Thinking about faster feedback, it makes sense.
There was a problem hiding this comment.
while the original scope was only to turn this on for trunk, yeah i agree it's probably worth extending this to include release/* branches too, especially since we saw at least one release build recently that started failing because of tests 😬
i updated the condition to include release/* in 06c13dd and tested that using this draft PR (used test/ instead of release/ to test the condition).
Description
This PR updates the version of
a8c-ci-toolkitplugin to the latest v2.17.0 to get the recent change where a Slack notification will be sent when there's a test failure. This change will be for both Unit and UI test failures.I updated the 3 .yml files under
.buildkiteto point to the latest version. I'm not sure if that's the right way but please me know if this shouldn't be the case.Testing instructions
This was previously tested using this draft PR, when there's a failing test, a notification is sent to the channel set in
annotate_test_failures. When there are no test failures, the notification is not sent.