Support JUnit XML test report

[Amulyam24]

What this PR does / why we need it:
This PR provides the ability to generate JUnit XML test reports and populate the prow dashboard accordingly for the unit tests.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #328

Specify your PR reviewers:

/cc @gyliu513 @jichenjc @xunpan @mkumatag

Special notes for your reviewer:

/area provider/ibmcloud

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Release note:

NONE

[k8s-ci-robot]

Hi @Amulyam24. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Amulyam24]

cc @Rajalakshmi-Girish

[mkumatag]

/ok-to-test

[mkumatag]

@Amulyam24: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Rerun command
pull-cluster-api-provider-ibmcloud-test ee1889e link /test pull-cluster-api-provider-ibmcloud-test
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

seems like this failure is happening here as well @Amulyam24 @Rajalakshmi-Girish

[mkumatag]

/retest

[Rajalakshmi-Girish]

@Amulyam24: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
Test name Commit Details Rerun command
pull-cluster-api-provider-ibmcloud-test ee1889e link /test pull-cluster-api-provider-ibmcloud-test
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

seems like this failure is happening here as well @Amulyam24 @Rajalakshmi-Girish

@Amulyam24 This failure seems to occur when make lint takes more than 1minute to execute.
You may try including --timeout option at https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/blob/main/Makefile#L116

[Rajalakshmi-Girish]

The default value is 1m as mentioned here https://golangci-lint.run/usage/configuration/#command-line-options

[Amulyam24]

After the initial test failure with timeout, the retests fail with /bin/sh: 1: go-junit-report: not found
I'll also make the changes to add a higher timeout to avoid the issue.

[Amulyam24]

@Amulyam24: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
Test name Commit Details Rerun command
pull-cluster-api-provider-ibmcloud-test ee1889e link /test pull-cluster-api-provider-ibmcloud-test
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

seems like this failure is happening here as well @Amulyam24 @Rajalakshmi-Girish

@Amulyam24 This failure seems to occur when make lint takes more than 1minute to execute.
You may try including --timeout option at https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/blob/main/Makefile#L116

Thanks @Rajalakshmi-Girish , will update it.

[mkumatag]

After the initial test failure with timeout, the retests fails with /bin/sh: 1: go-junit-report: not found
I'll make the changes to add a higher timeout as well to avoid the issue.

this may be because you just vendored but not really installed it. just see if you can install it like here

[Amulyam24]

@mkumatag, made the required changes, PTAL.

[mkumatag]

looking at the test results, it just shows only one entry, not sure why?!

[mkumatag]

@Amulyam24 see if this helps for the ginkgo - https://onsi.github.io/ginkgo/#generating-junit-xml-output

[Amulyam24]

looking at the test results, it just shows only one entry, not sure why?!

Yes, go-junit-report is only considering the test suite as a whole and not the individual test cases unlike using the ginkgo package. I'll look more on it.

I'm wondering if we can make optional, generate xml only when mentioned(especially via CI scripts)

@mkumatag, are you mentioning about making it optional via the scripts/ci-test.sh or through prow yaml. I'll have to explore the possibility of making it optional with the current PR changes and flow, since with prow, the presence of artifacts will always lead to generation of the XML reports.

[mkumatag]

@mkumatag, are you mentioning about making it optional via the scripts/ci-test.sh or through prow yaml. I'll have to explore the possibility of making it optional with the current PR flow, since with prow, the presence of artifacts will always lead to generation of the XML reports.

not with prow, e.g: if someone triggers these tests manually then it shouldn't generate the xml and generate the xml only if the option mentioned like some env like GENERATE_JUNIT, some of the logic can be derived from the kubernetes/kubernetes repo(something can be worked as followup PR).

[Amulyam24]

@mkumatag, updated with changes. PTAL.

[mkumatag]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Amulyam24, mkumatag

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mkumatag]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment