[SPARK-20301][FLAKY-TEST] Fix Hadoop Shell.runCommand flakiness in Structured Streaming tests #17613
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tdas
reviewed
Apr 12, 2017
| true | ||
| } | ||
| // Report trigger as finished and construct progress object. | ||
| finishTrigger(dataAvailable) |
There was a problem hiding this comment.
why did you move this out of the reportTimeTaken { ... }?
There was a problem hiding this comment.
I don't think I moved it out. Is the diff and whitespace confusing?
tdas
reviewed
Apr 12, 2017
| @@ -277,6 +277,11 @@ trait StreamTest extends QueryTest with SharedSQLContext with Timeouts { | |||
|
|
|||
| def threadState = | |||
| if (currentStream != null && currentStream.microBatchThread.isAlive) "alive" else "dead" | |||
| def threadStackTrace = if (currentStream != null && currentStream.microBatchThread.isAlive) { | |||
|
Test build #75720 has finished for PR 17613 at commit
|
|
Test build #75726 has started for PR 17613 at commit |
brkyvz
changed the title
|
retest this please |
|
Test build #75733 has finished for PR 17613 at commit
|
|
LGTM. Merging to master |
peter-toth
pushed a commit
to peter-toth/spark
that referenced
this pull request
Oct 6, 2018
…ructured Streaming tests ## What changes were proposed in this pull request? Some Structured Streaming tests show flakiness such as: ``` [info] - prune results by current_date, complete mode - 696 *** FAILED *** (10 seconds, 937 milliseconds) [info] Timed out while stopping and waiting for microbatchthread to terminate.: The code passed to failAfter did not complete within 10 seconds. ``` This happens when we wait for the stream to stop, but it doesn't. The reason it doesn't stop is that we interrupt the microBatchThread, but Hadoop's `Shell.runCommand` swallows the interrupt exception, and the exception is not propagated upstream to the microBatchThread. Then this thread continues to run, only to start blocking on the `streamManualClock`. ## How was this patch tested? Thousand retries locally and [Jenkins](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75720/testReport) of the flaky tests Author: Burak Yavuz <[email protected]> Closes apache#17613 from brkyvz/flaky-stream-agg.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Some Structured Streaming tests show flakiness such as:
This happens when we wait for the stream to stop, but it doesn't. The reason it doesn't stop is that we interrupt the microBatchThread, but Hadoop's
Shell.runCommandswallows the interrupt exception, and the exception is not propagated upstream to the microBatchThread. Then this thread continues to run, only to start blocking on thestreamManualClock.How was this patch tested?
Thousand retries locally and Jenkins of the flaky tests