Handle flasky tests

[HoussemNasri]

Description

Here's a typical scenario: I open a pull request, the build kicks off, and about an hour and a half later, the pipeline fails. When I check the failed tests, it’s something completely unrelated to my changes—a classic flaky test problem.

Here are some suggestions:

• Run flaky tests in a separate pipeline
  If flaky tests have their own pipeline, we can simply rerun that part when it fails instead of rerunning everything.

• Disable flaky tests (not ideal)

• Try the Jenkins Flaky Test Handler plugin
  Here’s the plugin. It’s designed to identify and rerun flaky tests automatically, which could help in our situation.

Why This Matters:

• Save Resources: Rerunning the whole pipeline for a flaky test is a waste of hardware and cloud resources.
• Boost Productivity: Each time a flaky test fails, we’re looking at at least an hour’s delay to rerun the pipeline. Fixing this would speed up the pull request merge process, making life easier for everyone.

[chibenwa]

It’s designed to identify and rerun flaky tests automatically, which could help in our situation.

How about fixing the flacky test?

When we encoutered an unrelated failing test we just open a ticket about it investigate it and fix it.

rerun flaky tests automatically

I'm against it. That would be hiding issues that would only grow worse. Let's rather fix them.

[HoussemNasri]

How about fixing the flacky test?

Question is: is it always possible to fix a flasky test.

When we encoutered an unrelated failing test we just open a ticket about it investigate it and fix it

+1

Could be useful:

• https://www.codewithjason.com/how-i-fix-flaky-tests/
• https://www.codewithjason.com/why-tests-flake-more-on-ci-than-locally/

I'm against it. That would be hiding issues that would only grow worse. Let's rather fix them.

The plugin would also categorize a test as flasky if it fails and succeeds on rerun so it can be used to detect flasky tests too.

[chibenwa]

Question is: is it always possible to fix a flasky test.

Yes

[chibenwa]

The plugin would also categorize a test as flasky if it fails and succeeds on rerun so it can be used to detect flasky tests too.

A broken build is IMO a better incetive to fix stuff