[SPARK-44377][BUILD] Exclude Junit5 dependencies from jersey-test-framework-provider-simple
#41944
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Oh, thank you so much for catching it. @LuciferYang ❤️ |
panbingkun
approved these changes
Jul 11, 2023
HyukjinKwon
approved these changes
Jul 11, 2023
|
Merged to master. Thanks @HyukjinKwon @panbingkun |
ragnarok56
pushed a commit
to ragnarok56/spark
that referenced
this pull request
Mar 2, 2024
…amework-provider-simple` ### What changes were proposed in this pull request? The following situation has led to Java tests no longer be executed when performing maven tests on the core module: 1. SPARK-44316 upgrade Jersey from 2.36 to 2.40. 2. Jersey 2.38 start to use [Junit5 instead of Junit4](eclipse-ee4j/jersey#5123). 3. Spark core module uses `org.glassfish.jersey.test-framework.providers:jersey-test-framework-provider-simple:2.40:test`, which cascades and introduces the dependencies of Junit5. So this pr manual exclude `Junit5` dependencies from `org.glassfish.jersey.test-framework.providers:jersey-test-framework-provider-simple:2.40:test` to make Java tests of core module can be tested using Maven. Additionally, this pr leaves a TODO to remove this exclusion after Spark migrates to Junit5 for Java testing: - [SPARK-44170](https://issues.apache.org/jira/browse/SPARK-44170): Migrating Junit4 to Junit5 ### Why are the changes needed? Make Java tests of core module can be tested using Maven after SPARK-44316. This issue only affects Maven, not SBT. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass Git Hub Action - Manual test: run `mvn clean install -pl core -am -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest` **Before** The core module is not running any Java tests ``` [INFO] --- maven-surefire-plugin:3.1.2:test (default-test) spark-core_2.12 --- [INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] [INFO] Results: [INFO] [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] [INFO] --- maven-surefire-plugin:3.1.2:test (test) spark-core_2.12 --- [INFO] Skipping execution of surefire because it has already been run for this configuration ``` **After** The Java test run of the core module was successful ``` [INFO] --- maven-surefire-plugin:3.1.2:test (default-test) spark-core_2.12 --- [INFO] Using auto detected provider org.apache.maven.surefire.junitcore.JUnitCoreProvider [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running test.org.apache.spark.Java8RDDAPISuite [INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.845 s -- in test.org.apache.spark.Java8RDDAPISuite [INFO] Running test.org.apache.spark.JavaAPISuite [INFO] Tests run: 92, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.40 s -- in test.org.apache.spark.JavaAPISuite [INFO] Running test.org.apache.spark.JavaSparkContextSuite [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.587 s -- in test.org.apache.spark.JavaSparkContextSuite [INFO] Running org.apache.spark.launcher.SparkLauncherSuite [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.977 s -- in org.apache.spark.launcher.SparkLauncherSuite [INFO] Running org.apache.spark.JavaJdbcRDDSuite [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.491 s -- in org.apache.spark.JavaJdbcRDDSuite [INFO] Running org.apache.spark.shuffle.sort.PackedRecordPointerSuite [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s -- in org.apache.spark.shuffle.sort.PackedRecordPointerSuite [INFO] Running org.apache.spark.shuffle.sort.ShuffleInMemoryRadixSorterSuite [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 s -- in org.apache.spark.shuffle.sort.ShuffleInMemoryRadixSorterSuite [INFO] Running org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite [INFO] Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.829 s -- in org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite [INFO] Running org.apache.spark.shuffle.sort.ShuffleInMemorySorterSuite [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.052 s -- in org.apache.spark.shuffle.sort.ShuffleInMemorySorterSuite [INFO] Running org.apache.spark.memory.TaskMemoryManagerSuite [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s -- in org.apache.spark.memory.TaskMemoryManagerSuite [INFO] Running org.apache.spark.util.SerializableConfigurationSuite [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.058 s -- in org.apache.spark.util.SerializableConfigurationSuite [INFO] Running org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorterRadixSortSuite [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 s -- in org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorterRadixSortSuite [INFO] Running org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorterSuite [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s -- in org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorterSuite [INFO] Running org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorterSuite [INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.488 s -- in org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorterSuite [INFO] Running org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorterRadixSortSuite [INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.433 s -- in org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorterRadixSortSuite [INFO] Running org.apache.spark.io.ReadAheadInputStreamSuite [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.145 s -- in org.apache.spark.io.ReadAheadInputStreamSuite [INFO] Running org.apache.spark.io.NioBufferedInputStreamSuite [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.540 s -- in org.apache.spark.io.NioBufferedInputStreamSuite [INFO] Running org.apache.spark.unsafe.map.BytesToBytesMapOnHeapSuite [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.771 s -- in org.apache.spark.unsafe.map.BytesToBytesMapOnHeapSuite [INFO] Running org.apache.spark.unsafe.map.BytesToBytesMapOffHeapSuite [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.647 s -- in org.apache.spark.unsafe.map.BytesToBytesMapOffHeapSuite [INFO] Running org.apache.spark.api.java.OptionalSuite [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s -- in org.apache.spark.api.java.OptionalSuite [INFO] Running org.apache.spark.resource.JavaResourceProfileSuite [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s -- in org.apache.spark.resource.JavaResourceProfileSuite [INFO] [INFO] Results: [INFO] [INFO] Tests run: 264, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] [INFO] --- maven-surefire-plugin:3.1.2:test (test) spark-core_2.12 --- [INFO] Skipping execution of surefire because it has already been run for this configuration ... Run completed in 21 minutes, 45 seconds. Total number of tests run: 3196 Suites: completed 300, aborted 0 Tests: succeeded 3196, failed 0, canceled 5, ignored 10, pending 0 All tests passed. ``` Closes apache#41944 from LuciferYang/SPARK-44377. Authored-by: yangjie01 <[email protected]> Signed-off-by: yangjie01 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The following situation has led to Java tests no longer be executed when performing maven tests on the core module:
org.glassfish.jersey.test-framework.providers:jersey-test-framework-provider-simple:2.40:test, which cascades and introduces the dependencies of Junit5.So this pr manual exclude
Junit5dependencies fromorg.glassfish.jersey.test-framework.providers:jersey-test-framework-provider-simple:2.40:testto make Java tests of core module can be tested using Maven.Additionally, this pr leaves a TODO to remove this exclusion after Spark migrates to Junit5 for Java testing:
Why are the changes needed?
Make Java tests of core module can be tested using Maven after SPARK-44316. This issue only affects Maven, not SBT.
Does this PR introduce any user-facing change?
No
How was this patch tested?
run
mvn clean install -pl core -am -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTestBefore
The core module is not running any Java tests
After
The Java test run of the core module was successful