[SPARK-23971] [BACKPORT-2.3] Should not leak Spark sessions across test suites #21197
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## What changes were proposed in this pull request? Many suites currently leak Spark sessions (sometimes with stopped SparkContexts) via the thread-local active Spark session and default Spark session. We should attempt to clean these up and detect when this happens to improve the reproducibility of tests. ## How was this patch tested? Existing tests Author: Eric Liang <[email protected]> Closes apache#21058 from ericl/clear-session.
|
ok to test |
|
test this please |
HyukjinKwon
approved these changes
May 1, 2018
|
Retest this please. |
|
Test build #89983 has finished for PR 21197 at commit
|
|
Retest this please. |
|
ok to test |
|
test this please |
|
Ur, currently, SparkR test failed in many PR consistently. I can see the following Error message and SparkR test seems to run twice according to the log. Could you take a look please, @shivaram and @felixcheung ? |
|
Test build #90016 has finished for PR 21197 at commit
|
|
retest this please |
|
#21197 (comment) This looks pretty much similar with the past one. Probably we should ask some help to R dev again if it consistently fails. Looks fine now though. |
|
Test build #90025 has finished for PR 21197 at commit
|
|
retest this please |
|
It seems this patch doesn't fix the problem... the test still hang |
|
Is this the error? Seems like intermittent problem from CRAN. Let me know if you see this again.
Also its just the log text repeated, but the test run.
checking CRAN incoming feasibility ...Error in .check_package_CRAN_incoming(pkgdir) :
dims [product 24] do not match the length of object [0]
Execution halted
|
|
Thank you for confirming, @felixcheung ! |
|
@felixcheung . The same failure occurs again in #21210 , too. |
|
I filed this flakiness as SPARK-24152. |
|
Test build #90065 has finished for PR 21197 at commit
|
|
Thanks! Merged to 2.3 |
asfgit
pushed a commit
that referenced
this pull request
May 2, 2018
…t suites This PR is to backport the PR #21058 to Apache 2.3. This should be the cause why we saw the test regressions in Apache 2.3 branches: https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.3-test-sbt-hadoop-2.6/317/testReport/org.apache.spark.sql.execution.datasources.parquet/ParquetQuerySuite/SPARK_15678__not_use_cache_on_overwrite/history/ https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.3-test-sbt-hadoop-2.7/318/testReport/junit/org.apache.spark.sql/DataFrameSuite/inputFiles/history/ --- ## What changes were proposed in this pull request? Many suites currently leak Spark sessions (sometimes with stopped SparkContexts) via the thread-local active Spark session and default Spark session. We should attempt to clean these up and detect when this happens to improve the reproducibility of tests. ## How was this patch tested? Existing tests Author: Eric Liang <[email protected]> Closes #21197 from gatorsmile/backportSPARK-23971.
otterc
pushed a commit
to linkedin/spark
that referenced
this pull request
Mar 22, 2023
1) Use host/domain from dfsCluster 2) Use semaphores to remove timing based flakeyness 3) Ensure spark context is closed 4) Ignore ParquetAvroCompatibilitySuite temporarily 5) Adaptation of SPARK-28247 to 2.3 6) Disable codegen tests - these are fixed in 2.4 and 3.x, but require sql backports to 2.3 7) PR apache#21197 to handle test failures due to leaking spark session 8) PR#20926: Set default Spark session in test-only spark sessions 9) PR apache#20971: Active SparkSession should be set by getOrCreate 10) PR apache#21446: Random.nextString is not safe for directory namePrefix 11) Disabling QueryStageSuite#'adaptive skewed join' 12) SPARK-24318: Fix flakey SortShuffleSuite 13) Fix ForeachSinkSuite test failure 14) Diabling StreamingQuerySuite.'status, lastProgress, and recentProgress' 15) Disabling ResolvedDataSourceSuite.'avro: show deploy guide for loading the external avro module' 16) PR# 23405: Avoid to use Random.nextString in StreamingInnerJoinSuite 17) Adaptation of PR#25849, add LinkedIn repo as default to spark.sql.maven.additionalRemoteRepositories 18) Disabling HiveExternalCatalogVersionsSuite - only 2.4.6 and 3.0.0 are available from apache mirrors 19) Disabling INFER_AND_SAVE test in HiveSchemaInferenceSuite - Address Erik's review comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is to backport the PR #21058 to Apache 2.3. This should be the cause why we saw the test regressions in Apache 2.3 branches:
https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.3-test-sbt-hadoop-2.6/317/testReport/org.apache.spark.sql.execution.datasources.parquet/ParquetQuerySuite/SPARK_15678__not_use_cache_on_overwrite/history/
https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.3-test-sbt-hadoop-2.7/318/testReport/junit/org.apache.spark.sql/DataFrameSuite/inputFiles/history/
What changes were proposed in this pull request?
Many suites currently leak Spark sessions (sometimes with stopped SparkContexts) via the thread-local active Spark session and default Spark session. We should attempt to clean these up and detect when this happens to improve the reproducibility of tests.
How was this patch tested?
Existing tests