-
Notifications
You must be signed in to change notification settings - Fork 335
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Don't error on old test reports #348
Conversation
@timja I believe this change broke https://github.com/jenkinsci/maven-plugin/blob/0f600837dbea3d8a86984f34f9cc7eb2e24950da/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java#L175 which was created due to https://issues.jenkins.io/browse/JENKINS-31524 which I believe is still a valid concern. My knowledge on the internals here is zero so I believe you are the best one to evaluate if JENKINS-31524 is still a concern or is no more relevant |
Hi I've had a bit of a look and can't say I really understand the maven plugin change that was done. The safest way would be for the user to use separate directories anyway. |
Thanks @timja I am on the same situation, will dig deeper. |
But if nothing is rebuilt, and hence the tests are not run, (and the file is outdated) why would we want to report the tests as run / passing in the UI when they are not run? Because the tests are not run how do you know they pass. For example a test that fails if the date > 19-03-2022 would pass today. Iirc (unable to check now) but this issue reported was always fixable if the user wanted it by setting a system property? The issue we also have now IIUC is that we will also report tests that no longer exist. |
This wasn't fixable by a system property. This fundamentally did not work with gradle's test results previously without work arounds. This change means that the plugin reports test results that are there. If you don't want them to be reported then you need to not have them there.
In the case of gradle the tests will run again as soon as the source set cache is invalided, i.e. if you change one of the tests or the source code. It will not run tests over and over again by just clicking build though, unless you pass some command line args which you can do. |
does it work now with
And that is the issue - the plugin should report tests that ran which was the behaviour prior to this change. I doubt I can be convinced otherwise - likely as I will no doubt fail to convince the gradle users otherwise. Which then means that the only conclusion here is that this needs to be an option. |
This reverts commit e294b1b.
This appears to leave behind an unused parameter junit-plugin/src/main/java/hudson/tasks/junit/TestResult.java Lines 142 to 143 in 86b06fd
|
This is an outdated feature that doesn't reflect modern build tools that won't rebuild if all the inputs are the same (and they aren't forced to rebuild)
A multitude of hacks for gradle that are no longer needed after this:
https://stackoverflow.com/questions/13879667/how-to-fix-test-reports-were-found-but-none-of-them-are-new-did-tests-run-in
Seems like there was no test coverage for this and I don't think it makes sense to add any for a non feature
Fixes #201