testng-results xml reports config skips from base classes as ignored #2886

mgesreha · 2023-03-17T15:01:44Z

TestNG Version

7.7.1

Expected behavior

<testng-results ignored="2" total="5" passed="3" failed="0" skipped="0">
should contains only the results of @test annotated methods

Actual behavior

While I have only 3 test cases are all passed, and all configuration methods are passed also, I have 2 extra ignored with total 5 tests cases instead of 3 which is wrong

Is the issue reproducible on runner?

Test case sample

N/A

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

The text was updated successfully, but these errors were encountered:

krmahadevan · 2023-03-18T00:43:09Z

@mgesreha - Please attach a sample project that can be used by us to reproduce the problem.

mgesreha · 2023-03-19T08:22:37Z

@krmahadevan
Thank you for your response, kindly find the sample project in the below repo
https://github.com/mgesreha/testng_issue

krmahadevan · 2023-03-19T13:25:18Z

@mgesreha - I am not sure I quite understand what the issue is.

In total there are 3 @Test methods from 2 test classes.

Here's how the emailable reports look like

Here's how the html reports look like

Here's how the IntelliJ console output looks like

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Kpi1
Kpi2
mocktest1
PASSED: platform.kpi.KpiTest.testClinicalKpiAlphabeticalSorting
PASSED: platform.kpi.KpiTest.testOperationalKpiAlphabeticalSorting
PASSED: platform.kpi.mockTest.mocktest1

===============================================
    C:/CX/Candidates/CXP-Test-Automation/src/test/java/platform/kpi/KpiTest.java
    Tests run: 3, Failures: 0, Skips: 0
===============================================


===============================================
All Test Suite
Total tests run: 3, Passes: 3, Failures: 0, Skips: 0
===============================================


Process finished with exit code 0

Here's the console output when I run mvn clean test

[INFO] --- surefire:2.22.2:test (default-test) @ CXP-Test-Automation ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Kpi1
Kpi2
mocktest1
PASSED: platform.kpi.KpiTest.testOperationalKpiAlphabeticalSorting
PASSED: platform.kpi.mockTest.mocktest1
PASSED: platform.kpi.KpiTest.testClinicalKpiAlphabeticalSorting

===============================================
    C:/CX/Candidates/CXP-Test-Automation/src/test/java/platform/kpi/KpiTest.java
    Tests run: 3, Failures: 0, Skips: 0
===============================================

[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.857 s - in TestSuite
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.303 s
[INFO] Finished at: 2023-03-19T18:53:34+05:30
[INFO] ------------------------------------------------------------------------

The emailable reports and the html reports are the same as earlier even when run via maven command line.

So can you help me understand what is the problem and what is the expectation ? Also please update the sample project to remove off all 3rd party libraries (I see that you have playright).

mgesreha · 2023-03-19T14:44:29Z

@krmahadevan
Thank you for your comment.
The issue is related to testng-results XML report not other reports

<?xml version="1.0" encoding="UTF-8"?>
<testng-results ignored="2" total="5" passed="3" failed="0" skipped="0">
  <reporter-output>
  </reporter-output>
  <suite started-at="2023-03-19T09:05:45 CET" name="All Test Suite" finished-at="2023-03-19T09:05:47 CET" duration-ms="1179">
    <groups>
    </groups>
    <test started-at="2023-03-19T09:05:45 CET" name="C:/CX/Candidates/CXP-Test-Automation/src/test/java/platform/kpi/KpiTest.java" finished-at="2023-03-19T09:05:47 CET" duration-ms="1179">
      <class name="platform.kpi.mockTest">
        <test-method is-config="true" signature="createContextAndPage()[pri:0, instance:platform.kpi.mockTest@f202d6d]" started-at="2023-03-19T09:05:46 CET" name="createContextAndPage" finished-at="2023-03-19T09:05:47 CET" duration-ms="258" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- createContextAndPage -->
        <test-method signature="mocktest1()[pri:0, instance:platform.kpi.mockTest@f202d6d]" started-at="2023-03-19T09:05:47 CET" name="mocktest1" finished-at="2023-03-19T09:05:47 CET" duration-ms="3" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- mocktest1 -->
        <test-method is-config="true" signature="closeContext()[pri:0, instance:platform.kpi.mockTest@f202d6d]" started-at="2023-03-19T09:05:47 CET" name="closeContext" finished-at="2023-03-19T09:05:47 CET" duration-ms="11" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- closeContext -->
      </class> <!-- platform.kpi.mockTest -->
      <class name="platform.kpi.KpiTest">
        <test-method is-config="true" signature="launchBrowser()[pri:0, instance:platform.kpi.KpiTest@5b12012e]" started-at="2023-03-19T09:05:40 CET" name="launchBrowser" finished-at="2023-03-19T09:05:45 CET" duration-ms="5689" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- launchBrowser -->
        <test-method is-config="true" signature="createContextAndPage()[pri:0, instance:platform.kpi.KpiTest@5b12012e]" started-at="2023-03-19T09:05:46 CET" name="createContextAndPage" finished-at="2023-03-19T09:05:46 CET" duration-ms="279" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- createContextAndPage -->
        <test-method signature="testClinicalKpiAlphabeticalSorting()[pri:0, instance:platform.kpi.KpiTest@5b12012e]" started-at="2023-03-19T09:05:46 CET" name="testClinicalKpiAlphabeticalSorting" finished-at="2023-03-19T09:05:46 CET" duration-ms="77" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- testClinicalKpiAlphabeticalSorting -->
        <test-method is-config="true" signature="closeContext()[pri:0, instance:platform.kpi.KpiTest@5b12012e]" started-at="2023-03-19T09:05:46 CET" name="closeContext" finished-at="2023-03-19T09:05:46 CET" duration-ms="26" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- closeContext -->
        <test-method is-config="true" signature="createContextAndPage()[pri:0, instance:platform.kpi.KpiTest@5b12012e]" started-at="2023-03-19T09:05:46 CET" name="createContextAndPage" finished-at="2023-03-19T09:05:46 CET" duration-ms="224" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- createContextAndPage -->
        <test-method signature="testOperationalKpiAlphabeticalSorting()[pri:0, instance:platform.kpi.KpiTest@5b12012e]" started-at="2023-03-19T09:05:46 CET" name="testOperationalKpiAlphabeticalSorting" finished-at="2023-03-19T09:05:46 CET" duration-ms="3" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- testOperationalKpiAlphabeticalSorting -->
        <test-method is-config="true" signature="closeContext()[pri:0, instance:platform.kpi.KpiTest@5b12012e]" started-at="2023-03-19T09:05:46 CET" name="closeContext" finished-at="2023-03-19T09:05:46 CET" duration-ms="11" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- closeContext -->
        <test-method is-config="true" signature="closeBrowser()[pri:0, instance:platform.kpi.KpiTest@5b12012e]" started-at="2023-03-19T09:05:47 CET" name="closeBrowser" finished-at="2023-03-19T09:05:47 CET" duration-ms="68" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- closeBrowser -->
      </class> <!-- platform.kpi.KpiTest -->
    </test> <!-- C:/CX/Candidates/CXP-Test-Automation/src/test/java/platform/kpi/KpiTest.java -->
  </suite> <!-- All Test Suite -->
</testng-results>

krmahadevan · 2023-03-19T14:54:10Z

@mgesreha - I still DONOT understand what the issue is. The xml you shared vs the project you shared seem to be totally un-related. I only saw the below test methods in your project

<?xml version="1.0" encoding="UTF-8"?>
<testng-results ignored="0" total="3" passed="3" failed="0" skipped="0">
  <reporter-output>
  </reporter-output>
  <suite started-at="2023-03-19T18:53:33 IST" name="All Test Suite" finished-at="2023-03-19T18:53:33 IST" duration-ms="293">
    <groups>
    </groups>
    <test started-at="2023-03-19T18:53:33 IST" name="C:/CX/Candidates/CXP-Test-Automation/src/test/java/platform/kpi/KpiTest.java" finished-at="2023-03-19T18:53:33 IST" duration-ms="293">
      <class name="platform.kpi.mockTest">
        <test-method signature="mocktest1()[pri:0, instance:platform.kpi.mockTest@226b143b]" started-at="2023-03-19T18:53:33 IST" name="mocktest1" finished-at="2023-03-19T18:53:33 IST" duration-ms="2" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- mocktest1 -->
      </class> <!-- platform.kpi.mockTest -->
      <class name="platform.kpi.KpiTest">
        <test-method signature="testClinicalKpiAlphabeticalSorting()[pri:0, instance:platform.kpi.KpiTest@4af46df3]" started-at="2023-03-19T18:53:33 IST" name="testClinicalKpiAlphabeticalSorting" finished-at="2023-03-19T18:53:33 IST" duration-ms="66" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- testClinicalKpiAlphabeticalSorting -->
        <test-method signature="testOperationalKpiAlphabeticalSorting()[pri:0, instance:platform.kpi.KpiTest@4af46df3]" started-at="2023-03-19T18:53:33 IST" name="testOperationalKpiAlphabeticalSorting" finished-at="2023-03-19T18:53:33 IST" duration-ms="2" status="PASS">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- testOperationalKpiAlphabeticalSorting -->
      </class> <!-- platform.kpi.KpiTest -->
    </test> <!-- C:/CX/Candidates/CXP-Test-Automation/src/test/java/platform/kpi/KpiTest.java -->
  </suite> <!-- All Test Suite -->
</testng-results>

The issue is related to testng-results XML report not other reports

Please clarify what am I missing here.

krmahadevan · 2023-03-20T03:30:40Z

@mgesreha - I am guessing that you are referring to this line

<testng-results ignored="2" total="5" passed="3" failed="0" skipped="0">

and your concern is that I have only 3 @Test methods and so where did this count of 5 come from and what are these 2 that are being shown as ignored.

It would have been really good if you had started off with this line, so that we could have gotten to the bottom of the confusion a lot more easier. Anyways moving on.

Here's what is happening:

There is a @BeforeSuite and a @AfterSuite that resides in your base class.
There are two test classes that extend from this base class.
@BeforeSuite and @AfterSuite are designed to execute ONLY once per <suite> tag.
So when TestNG picks up your second test class, it ignores these two methods from the base class and that get added to the excluded result.
The bug happens in the testng-results.xml because, TestNG does not count the number of ignored @Test methods and instead just counts the number of ignored methods (which can also include configuration methods which is what is happening in this case)

Closes testng-team#2886 TestNG by design excludes suite level configs When there is inheritance involved and multiple Children of the same base class are included in a suite. But when reporting the results as xml TestNG is not excluding these configs from the reports. Fixed this discrepancy.

mgesreha · 2023-03-20T09:05:45Z

@krmahadevan
Sorry for the confusion, I already mentioned this in the issue description, but I missed the Code quotes, so It didn't appear and I didn't notice that.

When do I expect this fix to be released?
also why your report doesn't show this issue? and why your XML report doesn't include the configuration methods?
<testng-results ignored="0" total="3" passed="3" failed="0" skipped="0">

krmahadevan · 2023-03-20T09:26:59Z

@mgesreha

When do I expect this fix to be released?

Well it should be soon. I am hoping to get around to a few more bug fixes before releasing.

also why your report doesn't show this issue? why your XML report doesn't include the configuration methods?

Because your configuration methods includes playwright setup and I didnt want to have it done. So I disabled them. Which explains why it didn't show up first.

Closes #2886 TestNG by design excludes suite level configs When there is inheritance involved and multiple Children of the same base class are included in a suite. But when reporting the results as xml TestNG is not excluding these configs from the reports. Fixed this discrepancy.

krmahadevan added the Need: sample label Mar 18, 2023

krmahadevan changed the title ~~testng-results xml report shows wrong numbers~~ testng-results xml reports config skips from base classes as ignored Mar 20, 2023

krmahadevan added Feature: report xml and removed Need: sample labels Mar 20, 2023

krmahadevan added this to the 7.8.0 milestone Mar 20, 2023

krmahadevan mentioned this issue Mar 20, 2023

Ignore Config skips in xml reports #2887

Merged

3 tasks

krmahadevan closed this as completed in #2887 Mar 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

testng-results xml reports config skips from base classes as ignored #2886

testng-results xml reports config skips from base classes as ignored #2886

mgesreha commented Mar 17, 2023 •

edited

Loading

krmahadevan commented Mar 18, 2023

mgesreha commented Mar 19, 2023

krmahadevan commented Mar 19, 2023

mgesreha commented Mar 19, 2023 •

edited by krmahadevan

Loading

krmahadevan commented Mar 19, 2023

krmahadevan commented Mar 20, 2023

mgesreha commented Mar 20, 2023

krmahadevan commented Mar 20, 2023

testng-results xml reports config skips from base classes as ignored #2886

testng-results xml reports config skips from base classes as ignored #2886

Comments

mgesreha commented Mar 17, 2023 • edited Loading

TestNG Version

Expected behavior

Actual behavior

Is the issue reproducible on runner?

Test case sample

Contribution guidelines

krmahadevan commented Mar 18, 2023

mgesreha commented Mar 19, 2023

krmahadevan commented Mar 19, 2023

mgesreha commented Mar 19, 2023 • edited by krmahadevan Loading

krmahadevan commented Mar 19, 2023

krmahadevan commented Mar 20, 2023

mgesreha commented Mar 20, 2023

krmahadevan commented Mar 20, 2023

mgesreha commented Mar 17, 2023 •

edited

Loading

mgesreha commented Mar 19, 2023 •

edited by krmahadevan

Loading