[SPARK-19113][SS][Tests]Set UncaughtExceptionHandler in onQueryStarted to ensure catching fatal errors during query initialization #16492
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…al errors during query initialization
|
Test build #70995 has finished for PR 16492 at commit
|
|
Test build #70998 has finished for PR 16492 at commit
|
srowen
approved these changes
Jan 7, 2017
jaceklaskowski
suggested changes
Jan 7, 2017
| @@ -238,7 +238,7 @@ class StreamSuite extends StreamTest { | |||
| } | |||
| } | |||
|
|
|||
| testQuietly("fatal errors from a source should be sent to the user") { | |||
| testQuietly("handle fatal errors thrown from the stream thread correctly") { | |||
There was a problem hiding this comment.
correctly is superfluous and could be removed.
| @@ -235,7 +235,10 @@ trait StreamTest extends QueryTest with SharedSQLContext with Timeouts { | |||
| */ | |||
| def testStream( | |||
| _stream: Dataset[_], | |||
| outputMode: OutputMode = OutputMode.Append)(actions: StreamAction*): Unit = { | |||
| outputMode: OutputMode = OutputMode.Append)(actions: StreamAction*): Unit = synchronized { | |||
| // `synchronized` is added to prevent the user from calling `testStream` concurrently because | |||
There was a problem hiding this comment.
Can you elaborate why StreamingQueryListener can be used concurrently? I'm reading the comment: "it does not work because something else does not work". Why? What's wrong with StreamingQueryListener?
|
Test build #71086 has finished for PR 16492 at commit
|
|
Merged to master |
asfgit
pushed a commit
that referenced
this pull request
Jan 10, 2017
…ed to ensure catching fatal errors during query initialization ## What changes were proposed in this pull request? StreamTest sets `UncaughtExceptionHandler` after starting the query now. It may not be able to catch fatal errors during query initialization. This PR uses `onQueryStarted` callback to fix it. ## How was this patch tested? Jenkins Author: Shixiong Zhu <[email protected]> Closes #16492 from zsxwing/SPARK-19113.
|
I also cherry-picked to 2.1 since Jenkins also run 2.1 tests. |
asfgit
pushed a commit
that referenced
this pull request
Jan 18, 2017
…waitInitialization to avoid breaking tests ## What changes were proposed in this pull request? #16492 missed one race condition: `StreamExecution.awaitInitialization` may throw fatal errors and fail the test. This PR just ignores `StreamingQueryException` thrown from `awaitInitialization` so that we can verify the exception in the `ExpectFailure` action later. It's fine since `StopStream` or `ExpectFailure` will catch `StreamingQueryException` as well. ## How was this patch tested? Jenkins Author: Shixiong Zhu <[email protected]> Closes #16567 from zsxwing/SPARK-19113-2. (cherry picked from commit c050c12) Signed-off-by: Shixiong Zhu <[email protected]>
ghost
pushed a commit
to dbtsai/spark
that referenced
this pull request
Jan 18, 2017
…waitInitialization to avoid breaking tests ## What changes were proposed in this pull request? apache#16492 missed one race condition: `StreamExecution.awaitInitialization` may throw fatal errors and fail the test. This PR just ignores `StreamingQueryException` thrown from `awaitInitialization` so that we can verify the exception in the `ExpectFailure` action later. It's fine since `StopStream` or `ExpectFailure` will catch `StreamingQueryException` as well. ## How was this patch tested? Jenkins Author: Shixiong Zhu <[email protected]> Closes apache#16567 from zsxwing/SPARK-19113-2.
uzadude
pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
…ed to ensure catching fatal errors during query initialization ## What changes were proposed in this pull request? StreamTest sets `UncaughtExceptionHandler` after starting the query now. It may not be able to catch fatal errors during query initialization. This PR uses `onQueryStarted` callback to fix it. ## How was this patch tested? Jenkins Author: Shixiong Zhu <[email protected]> Closes apache#16492 from zsxwing/SPARK-19113.
uzadude
pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
…waitInitialization to avoid breaking tests ## What changes were proposed in this pull request? apache#16492 missed one race condition: `StreamExecution.awaitInitialization` may throw fatal errors and fail the test. This PR just ignores `StreamingQueryException` thrown from `awaitInitialization` so that we can verify the exception in the `ExpectFailure` action later. It's fine since `StopStream` or `ExpectFailure` will catch `StreamingQueryException` as well. ## How was this patch tested? Jenkins Author: Shixiong Zhu <[email protected]> Closes apache#16567 from zsxwing/SPARK-19113-2.
cmonkey
pushed a commit
to cmonkey/spark
that referenced
this pull request
Feb 15, 2017
…ed to ensure catching fatal errors during query initialization ## What changes were proposed in this pull request? StreamTest sets `UncaughtExceptionHandler` after starting the query now. It may not be able to catch fatal errors during query initialization. This PR uses `onQueryStarted` callback to fix it. ## How was this patch tested? Jenkins Author: Shixiong Zhu <[email protected]> Closes apache#16492 from zsxwing/SPARK-19113.
cmonkey
pushed a commit
to cmonkey/spark
that referenced
this pull request
Feb 15, 2017
…waitInitialization to avoid breaking tests ## What changes were proposed in this pull request? apache#16492 missed one race condition: `StreamExecution.awaitInitialization` may throw fatal errors and fail the test. This PR just ignores `StreamingQueryException` thrown from `awaitInitialization` so that we can verify the exception in the `ExpectFailure` action later. It's fine since `StopStream` or `ExpectFailure` will catch `StreamingQueryException` as well. ## How was this patch tested? Jenkins Author: Shixiong Zhu <[email protected]> Closes apache#16567 from zsxwing/SPARK-19113-2.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
StreamTest sets
UncaughtExceptionHandlerafter starting the query now. It may not be able to catch fatal errors during query initialization. This PR usesonQueryStartedcallback to fix it.How was this patch tested?
Jenkins