@Test Method is Retried Endlessly When Parameters Are Modified Before Rerun

[Riabchykova]

TestNG Version

7.10.2

Expected behavior

Test method is retried according to IRetryAnalyzer configuration when parameters are modified before rerun

Actual behavior

Test method is retried endlessly when parameters are modified before rerun

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Summary:

In my current project, test data needs to be slightly modified during retries because the data is saved to a database during test execution and on reruns, errors occur due to the data already existing in the database.

To handle this, parameters are modified within the ITestListener.onTestSkipped method during reruns to generate new values for each retry. The object remains the same type, but some fields (e.g., description or UUID-number) are updated.

Issue:

After upgrading TestNG from 7.7.1 to 7.10.2, test methods with updated parameters are invoked endlessly.

While debugging, I traced the issue to the method org.testng.internal.BaseTestMethod.getRetryAnalyzerConsideringMethodParameters(ITestResult tr). In this method, a unique keyAsString is generated for each rerun, which results in the creation of a new IRetryAnalyzer every time.
Corresponding PR: #2935

Additional Info:

I attempted to use DataProvider with cacheDataForTestRetries = false, but it fails to supply new parameters to the test methods.
As a result, I'm currently blocked from upgrading TestNG due to this issue.
Related issue: #3157
Related pr: #3076

Test case sample

Test class

public class EndlessRetryTest {
    Logger logger = Logger.getLogger(getClass().getName());

    @Test(dataProvider = "retryTest")
    public void retryTest(SomeDataEntity someDataEntity) {

        logger.info("Param : " + someDataEntity.getName());
        assert 1 == 0;
    }

    @DataProvider(name = "retryTest")
    public Object[][] getData() {

        return new Object[][]{{new SomeDataEntity()}};
    }

    @Data
    class SomeDataEntity implements Randomizer {
        String name = "Default";

        @Override
        public void randomize() {
            this.name = UUID.randomUUID().toString();
        }
    }
}

RetryListener class

public class RetryListener implements IAnnotationTransformer, ITestListener {
    @Override
    public void transform(ITestAnnotation iTestAnnotation, Class aClass, Constructor constructor, Method method) {

        iTestAnnotation.setRetryAnalyzer(Retrier.class);
    }

    @Override
    public void onTestSkipped(ITestResult result) {

        Object[] parameters = result.getParameters();

        for (Object parameter : parameters) {

            if (parameter instanceof Randomizer) {

                ((Randomizer) parameter).randomize();
            }
        }
    }
}

Retrier class

public class Retrier implements IRetryAnalyzer {
    private final AtomicInteger counter = new AtomicInteger(1);

    @Override
    public boolean retry(ITestResult result) {
        return counter.getAndIncrement() != 3;
    }

}

Randomizer interface

public interface Randomizer {
    void randomize();
}

retry_testng.xml

 <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Retry_Suite" configfailurepolicy="continue" parallel="methods">
    <test verbose="1" name="Retry_Test" enabled="true" thread-count="5">
        <classes>
            <class name="testpackage.EndlessRetryTest">
            </class>
        </classes>
    </test>
    <listeners>
        <listener class-name="testlisteners.RetryListener"/>
    </listeners>
</suite>

The output during the test run is as follows:

Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : Default

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : 4615d8f0-c025-4e06-97d4-07701368d2e4

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : 2636d0e8-6053-47ed-9591-403a234002d7

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : aad2824a-73d5-4622-9c4b-410a72bd892d

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : a2162904-86d1-4fc5-9dba-da9381bcc1bf

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : 37c00a0a-c430-4362-9900-cf24ff0d0b21

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : ad3f9daa-dd6e-4215-9bac-220450d39e65

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : d51f2091-e779-45bf-b27f-a9bb3b3f669e

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : 4925c9be-aaa1-4249-9c74-722807f485d1

 (and further endless)

[krmahadevan]

@Riabchykova - The problem lies in your test code and not with TestNG

You are using the lombok annotation @Data. This is a short cut for generating the following

• @Getter
• @Setter
• @RequiredArgsConstructor
• @ToString
• @EqualsAndHashCode

Now the thing that is goofing up your code is the @EqualsAndHashCode which by default considers the data members in your SomeDataEntity to generate the hashCode.

Everytime you alter your name parameter via the randomize() function, you are effectively altering the uniqueness of the SomeDataEntity object and thus causing TestNG to work with different test data.

You can fix this by doing one of the following after you get rid of the @Data annotation.

• Use @Getter because that is the only thing that you are using to retrieve the value in your test class (or)
• Abandon lombok annotation and use the classic way of working with getters/setters which can be generated by IntelliJ IDE.

Am closing this issue. Please feel free to comment back (possibly with a simple standalone project instead of the sample code) if the issue persists.

In a nutshell your @Data is what is goofing up things and it's very easy to miss its havoc :)

[Riabchykova]

@krmahadevan
Thank you for your thorough investigation and detailed explanation! I’ve made the changes you suggested, and the solution works perfectly. I appreciate your help in identifying the root cause. Thanks again for your support!