Add system-err and system-out to testcase #47

mclayton7 · 2021-09-05T19:35:00Z

This should fix Include stdout for tests that pass #41
Switches to the Jenkins JUnit 5 xsd, which required the removal of the empty properties tag
Add stdout/stderr to system-out and system-err tags to individual test cases

mclayton7 · 2021-09-05T19:37:20Z

test/JUnit.Xml.TestLogger.AcceptanceTests/JUnitTestLoggerAcceptanceTests.cs

+                .Where(x => x.Descendants().Any(y => y.Name.LocalName == "system-err"))
+                .ToList();
+
+            Assert.AreEqual(1, testCasesWithErrorOut.Count());


@Siphonophora I'm having trouble getting system-err to work on a per-testcase basis. Here I would expect the stderr from FailTest11() to be found. Any ideas why it's not getting picked up?

@mclayton7 I suspect its due to how nunit, which we use for these tests, is behaving. Looking at their docs, they make the distinction between standard output whose messages go to the test level and errors which get displayed right away. https://docs.nunit.org/articles/nunit/writing-tests/TestContext.html

Based on what we see in the loggers, I think the immediate messages are coming across as test messages https://github.com/spekt/testlogger/blob/master/src/TestLogger/Core/TestRunMessageWorkflow.cs but are not attached to the results. While normal console.writeline is going to both the standard out and is being attached to the test result.

I can look more later in the week, but if i'm right then this behavior will depend on the test library and we won't be able to do much on the logging side.

@Siphonophora I think you're correct. I can remove the stderr logging here since it doesn't work, and just leave the stdout.

I think it would be good if we could do a quick test with mstest and xunit, so we can put a note in the docs about how each behaves.

mclayton7 · 2021-09-05T20:10:54Z

src/JUnit.Xml.TestLogger/JunitXmlSerializer.cs

+
+            if (!string.IsNullOrWhiteSpace(stdOut.ToString()))
+            {
+                testcaseElement.Add(new XElement("system-out", stdOut.ToString()));


I also didn't wrap this with XCData like the NUnit one because I wanted to match the existing system-out/system-err printing. I don't know enough about JUnit to know if parsers should expect CDATA or not.

Also something we could do a quick test on. But if the logged text has any xml characters that need to be escaped, I expect thats what will happen here. If we wrap it in CDATA, the characters won't be escaped and should therefore be more readable.

https://en.wikipedia.org/wiki/CDATA

@Siphonophora would you like me to wrap the per-testcase output in CDATA and the per-testsuite output? I'd consider this to be a breaking change because it's not currently wrapped on line 229.

I'm not sure I would call it a breaking change so long as we are still passing the xml validation. But, we can break this out to its own issue and decide on it later.

Siphonophora · 2021-09-19T17:01:21Z

The other thing I think we need here is to change the documentation in the main readme to reflect the change in the schema we are using.

It should also clarify that the system-out will show up at the test case level AND at the file level. I don't have any good ideas on how to prevent this duplication.

Siphonophora · 2021-11-07T16:37:48Z

@mclayton7 Do you want to circle back to finish this? If you don't have time I will see if I can move it forward.

mclayton7 · 2021-11-10T23:07:44Z

@Siphonophora sorry I lost track of this, I started looking into how other test runners worked in this repository: https://github.com/mclayton7/dotnet_test_output_reporting which led me down the rabbit hole of GitHub actions. I'll pick this back up and try to finish it this weekend!

Siphonophora · 2021-11-12T22:42:55Z

@mclayton7 No worries. I appreciate the help.

Siphonophora · 2021-11-22T21:54:29Z

README.md

@@ -47,6 +47,89 @@ test logs from over-writing eachother.

 [config-wiki]: https://github.com/spekt/testlogger/wiki/Logger-Configuration

+### Junit Output Notes


Siphonophora · 2021-11-22T22:03:26Z

README.md

@@ -47,6 +47,89 @@ test logs from over-writing eachother.

 [config-wiki]: https://github.com/spekt/testlogger/wiki/Logger-Configuration

+### Junit Output Notes
+
+The xml reports will have standard output (stdout) and standard error (stderr) from the test output. Stdout is logged on a per-testcase basis, but stderr


This section looks a bit incomplete. The beginning trails off and parts of mstest seems missing. I think the examples are helpful, but lets put them in another file for just the standard and error out.

Since you are pointing out that failed tests behave differently can we just show all the options. Just 2 test methods should be fine.
*Passing Test
* std out
* error out
*Failing Test
* std out
* error out

I think it would be fine to have the docs show more complete examples in a details tag if its getting too long.

In this file, lets have a short section after ### Customizing Junit XML Contents which points out that this stuff varies per testing framework, and links to all the details.

Siphonophora · 2021-11-22T22:05:35Z

README.md


 - <https://github.com/spekt/nunit.testlogger>
 - <https://github.com/spekt/xunit.testlogger>
 - <https://github.com/spekt/appveyor.testlogger>

 ## Usage

-The JUnit Test Logger generates xml reports in the [Ant Junit Format](https://github.com/windyroad/JUnit-Schema), which the JUnit 5 repository refers to as the de-facto standard. While the generated xml complies with that schema, it does not contain values in every case. For example, the logger currently does not log any `properties`. Please [refer to a sample file](docs/assets/TestResults.xml) to see an example. If you find that the format is missing data required by your CI/CD system, please open an issue or PR.
+The JUnit Test Logger generates xml reports in the [Jenkins JUnit Format](https://github.com/junit-team/junit5/blob/main/platform-tests/src/test/resources/jenkins-junit.xsd), which the JUnit 5 repository refers to as the de-facto standard. While the generated xml complies with that schema, it does not contain values in every case. For example, the logger currently does not log any `properties`. Please [refer to a sample file](docs/assets/TestResults.xml) to see an example. If you find that the format is missing data required by your CI/CD system, please open an issue or PR.


Lets make sure to update the sample file once all other changes are complete.

davetheninja · 2022-05-26T10:06:59Z

Is this MR still alive? last updates were from 2021.

I have an urgent requirement to provide a way to supress and redirect std-our and std-error from the test results XML to a separate file and wondered if I should pick up from here or start fresh?

Background:

We have multiple test suites generating 170-250 mb of std-out/err per suite. Gitlab completely chokes on such large results files.

We obviously need the output for debugging and diagnostics so suppression on its own isn't good enough, I want to redirect to a secondary log file instead and expose as a build artefact.

Thanks

Siphonophora · 2022-05-26T13:21:20Z

@davetheninja This one isn't dead per se, but I think what you are talking about is a different change altogether. It seems like what you really need is a feature flag to suppress logging of all std/error out in the junit report. That would be a simple change, and I could see it making sense to do.

For the other piece, I don't think this logger should do redirection. I would suggest taking a look at the trx or html loggers that are built into vstest. https://github.com/microsoft/vstest-docs/blob/main/docs/report.md

You can run more than one logger at a time dotnet test --logger:"junit" --logger:"trx", so you can still run the junit one for the benefit of gitlab. But then you have the trx file which would give you both the console output AND all the the available context and test results to go with it.

davetheninja · 2022-05-26T13:58:19Z

Yes, actually came to same conclusion after going through the code. Redirection doesn’t fit the contract for the serialiser. That being said, suppression is not an option for me, I need the data available. Have come up with a possible solution which I will try tomorrow where I have the test results cml being generated as usuals, but have a .post hook in gitlab ci stage and use power shell to extract the error and out elements to two log files - removing the out err body within the results xmlThat should hopefully kill both birds with one stone. Thanks David

Update to Jenkins JUnit 5 xsd

b655abe

mclayton7 force-pushed the add_stdout_to_test_cases branch from f89c1a5 to b2bddea Compare September 5, 2021 19:36

mclayton7 commented Sep 5, 2021

View reviewed changes

Add system-out and system-error to each testcase

507b9cb

mclayton7 force-pushed the add_stdout_to_test_cases branch from b2bddea to 507b9cb Compare November 15, 2021 01:10

Siphonophora requested changes Nov 22, 2021

View reviewed changes

codito mentioned this pull request Jun 24, 2024

Update JUnit XSD to support additional data spekt/testlogger#83

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add system-err and system-out to testcase #47

Add system-err and system-out to testcase #47

mclayton7 commented Sep 5, 2021

mclayton7 Sep 5, 2021

Siphonophora Sep 6, 2021

mclayton7 Sep 7, 2021

Siphonophora Sep 19, 2021

mclayton7 Sep 5, 2021

Siphonophora Sep 19, 2021

mclayton7 Nov 15, 2021

Siphonophora Nov 22, 2021

Siphonophora commented Sep 19, 2021

Siphonophora commented Nov 7, 2021

mclayton7 commented Nov 10, 2021

Siphonophora commented Nov 12, 2021

Siphonophora Nov 22, 2021

Siphonophora Nov 22, 2021

Siphonophora Nov 22, 2021

davetheninja commented May 26, 2022

Siphonophora commented May 26, 2022

davetheninja commented May 26, 2022 via email •

edited

Loading

		@@ -47,6 +47,89 @@ test logs from over-writing eachother.

		[config-wiki]: https://github.com/spekt/testlogger/wiki/Logger-Configuration

		### Junit Output Notes

Add system-err and system-out to testcase #47

Are you sure you want to change the base?

Add system-err and system-out to testcase #47

Conversation

mclayton7 commented Sep 5, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Siphonophora commented Sep 19, 2021

Siphonophora commented Nov 7, 2021

mclayton7 commented Nov 10, 2021

Siphonophora commented Nov 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davetheninja commented May 26, 2022

Siphonophora commented May 26, 2022

davetheninja commented May 26, 2022 via email • edited Loading

davetheninja commented May 26, 2022 via email •

edited

Loading