-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Output diff of expected and actual values in ConsoleLauncher
#4017
base: main
Are you sure you want to change the base?
Conversation
Tasks completed: 1. Locate ConsoleLauncher output 2. Check if actual and expected of AssertionFailedError are both type of CharSequence To do list: 1. Add diff function dependency 2. Using diff to generate the output desired Issue: junit-team#3139
Tasks completed: 1. Locate ConsoleLauncher output 2. Check if actual and expected of AssertionFailedError are both type of CharSequence 3. Add diff function dependency 4. Using diff to generate the output desired Issue: junit-team#3139
delete one line of extra code
Fixed different profile issue. Tasks completed: 1. Locate ConsoleLauncher output 2. Check if actual and expected of AssertionFailedError are both type of CharSequence 3. Add diff function dependency 4. Using diff to generate the output desired ToDo: 1.Automatic test case for diff output Issue: junit-team#3139
Add external diff module to module documentation For unknown reason, the test cases can not find the external module while the gradle can build normally. Issue: junit-team#3139
ConsoleLauncher
I closed it because the original author didn't respond anymore. But if you're interested you could create a new branch in your repo based on their work and make the changes that were requested in #3397. Please let me know whether you want to proceed that way. |
I prefer staying in this PR. The primary goal is to output The extra changes requested in #3397:
should come later as it would be easier once we have one successful example. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The extra changes requested in #3397:
Shadowing the original external function
Using theme to highlight thediff info
spreadingdiff info
to all the Junit API outputshould come later as it would be easier once we have one successful example.
We usually only merge complete PRs that leave main
ready to be released. But I can help with the shadowing.
@@ -180,11 +186,39 @@ private Optional<TestExecutionListener> createXmlWritingListener(PrintWriter out | |||
private void printSummary(TestExecutionSummary summary, PrintWriter out) { | |||
// Otherwise the failures have already been printed in detail | |||
if (EnumSet.of(Details.NONE, Details.SUMMARY, Details.TREE).contains(outputOptions.getDetails())) { | |||
//adding diff code here |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be done in place where we print the exceptions (as it is in #3397). Otherwise, it would be difficult to find the test that belongs to the diff in case of multiple failing tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Which part do you wish to change? The first print happens in DefaultLaucher
and the second happens in MutableTestExecutionSummary
. Both are in junit-platform-laucher
module. Changes will affect all Junit API output I believe?
I can try to work on it over the weekend if you wish.
At the same time, can you help to find why the package documentation test case fails (Test cases can not find diff package
once I add it to the module-info.java for documentation)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We also need to make sure that the diff is printed if a details mode other than Details.NONE
, Details.SUMMARY
, or Details.TREE
is used. We should be able to extract the diff-printing code to a separate class and call it from MutableTestExecutionSummary
, and the other listeners (FlatPrintingListener
etc.).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We also need to make sure that the diff is printed if a details mode other than
Details.NONE
,Details.SUMMARY
, orDetails.TREE
is used. We should be able to extract the diff-printing code to a separate class and call it fromMutableTestExecutionSummary
, and the other listeners (FlatPrintingListener
etc.).
The solutions I can think of:
- add diff output function to
junit-platform-commons...util.StringUtils
- Use the diff output function in
FlatPrintingListener
,TreePrintingListener
,VerboseTreePrintingListener
andMutableTestExecutionSummary
I will start to work on it if the solution looks right for you
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- add diff output function to
junit-platform-commons...util.StringUtils
I think it should stay in junit-platform-console. Please create a separate DiffPrinter
class or similar.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
DiffPrinter
class has been added.- Marked the location of adding diff to
FlatPrintingListener
andVerboseTreePrintingListener
- Will the diff info be generated after the stack trace of the exception?
1. Add diffPrinter class 2. replace diff in consoleTestExecutor with diffPrinter - will be deleted evetually 3. mark the location of adding diff in flatPrinter and verboseTreePrinter. Issue: junit-team#3139
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the diffs need to reference the failures, i.e. something like this (where (2)
and (3)
reference failures from the list above):
Failures (3):
(1) JUnit Jupiter:AssertionsDemo:groupedAssertions()
MethodSource [className = 'example.AssertionsDemo', methodName = 'groupedAssertions', methodParameterTypes = '']
=> org.opentest4j.AssertionFailedError: expected: <2> but was: <3>
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
example.AssertionsDemo.groupedAssertions(AssertionsDemo.java:52)
(2) JUnit Jupiter:AssertionsDemo:dependentAssertions()
MethodSource [className = 'example.AssertionsDemo', methodName = 'dependentAssertions', methodParameterTypes = '']
=> org.opentest4j.AssertionFailedError: expected: <a
b
c> but was: <b
c
d>
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1145)
example.AssertionsDemo.dependentAssertions(AssertionsDemo.java:63)
(3) JUnit Jupiter:AssertionsDemo:standardAssertions()
MethodSource [className = 'example.AssertionsDemo', methodName = 'standardAssertions', methodParameterTypes = '']
=> org.opentest4j.AssertionFailedError: expected: <fooooooo> but was: <booooooy>
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1145)
example.AssertionsDemo.standardAssertions(AssertionsDemo.java:41)
Diffs (Markdown):
(2) JUnit Jupiter:AssertionsDemo:groupedAssertions()
| expected | actual |
| -------- | ------ |
| ~~a~~ | b |
| b | c |
| c | **d** |
(3) JUnit Jupiter:AssertionsDemo:standardAssertions()
| expected | actual |
| -------- | ------ |
| ~~f~~oooooo~~o~~ | **b**oooooo**y** |
WDYT?
static void printDiff(PrintWriter out, String expected, String actual) { | ||
DiffRowGenerator generator = DiffRowGenerator.create().showInlineDiffs(true).inlineDiffByWord(true).oldTag( | ||
f -> "~").newTag(f -> "**").build(); | ||
List<DiffRow> rows = generator.generateDiffRows(Arrays.asList(expected), Arrays.asList(actual)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't we have to split the expected
and actual
Strings into lines here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What the difference is if we split them into lines?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wasn't sure. I haven't used this library before and I was just curious why it expects lists of Strings. Do you know the reason?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know the details but all their methods use List<String>
as input.
junit-platform-console/src/main/java/org/junit/platform/console/tasks/DiffPrinter.java
Outdated
Show resolved
Hide resolved
junit-platform-console/src/main/java/org/junit/platform/console/tasks/DiffPrinter.java
Outdated
Show resolved
Hide resolved
junit-platform-console/src/main/java/org/junit/platform/console/tasks/DiffPrinter.java
Outdated
Show resolved
Hide resolved
junit-platform-console/src/main/java/org/junit/platform/console/tasks/DiffPrinter.java
Outdated
Show resolved
Hide resolved
junit-platform-console/src/main/java/org/junit/platform/console/tasks/DiffPrinter.java
Show resolved
Hide resolved
…e/tasks/DiffPrinter.java Co-authored-by: Marc Philipp <[email protected]>
…e/tasks/DiffPrinter.java Co-authored-by: Marc Philipp <[email protected]>
…e/tasks/DiffPrinter.java Co-authored-by: Marc Philipp <[email protected]>
Only if one word has space, show inlineDiffByWord.
I can only get the test's display name (the actual test method name). In |
If we list them separately, we should definitely output the same info as done by |
1. Add a `testPlan` getter to `platform/launcher/listeners/SummaryGeneratingListener.java`. 2. DiffPrinter now needs testPlan to display the testMethod info. Issue: junit-team#3139
The Update |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
printFoundTestsSummary()
was not called /MutableTestExecutionSummary
was not created inConsoleTestExecutor
in summary mode. What's worse, thetestPlan
was never visible toConsoleTestExecutor
Therefore, I add atestPlan
getter toplatform/launcher/listeners/SummaryGeneratingListener.java
. Will that create any security issue?
Rather than adding it to SummaryGeneratingListener
, I'd register another (local) listener for this purpose.
Update
DiffPrinter
to containtestPlan
as an attribute and display the testMethod info when printing. Preview is below.
Looks like we're getting closer. We're still missing the (2)
etc. references, though.
For some reason, GitHub didn't let me push a commit to your PR so I opened a PR: XJ114514#1
We use Predictive Test Selection so the prior build that ignored them probably didn't select the ones that failed in the most recent build.
Sorry for not being clear. I think we should add those there as well. |
add testPlanCapturer listener
This reverts commit eef975b.
I manually combined the change you made to this PR, but the testPlan from testPlanCapturer is null when using. Can you check about it, sorry I can not offer help because I don't understand how the TestExecutionListener works.
I found the |
I tried to run the |
@XJ114514 Let's put this PR on hold for now. I'm currently working on a new HTML report format that would make it much easier to display a diff. |
Overview
Current PR can successfully build and generate
junit-platform-console-standalone-1.12.0-SNAPSHOT.jar
containing diff outputs for two different Strings to the console.Changes made in this draft PR:
junit-platform-console/.../tasks/ConsoleTestExecutor.java
(output of console)libs.versions.toml
andjunit-platform-console.gradle.kts
ofjunit-platform-console
release-notes-5.12.0-M1.adoc
Tasks that need to finish
junit-platform-console/.../module-info.java
(causing testing failure because the auto test cases can not find the diff package)ConsoleTestExecutorTests.java
I hereby agree to the terms of the JUnit Contributor License Agreement.
Definition of Done
@API
annotationscontinue from #3397
Issue: #3139