[SPARK-16251][SPARK-20200][Core][Test]Flaky test: org.apache.spark.rdd.LocalCheckpointSuite.missing checkpoint block fails with informative message #18314
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
srowen
reviewed
Jun 15, 2017
| @@ -168,6 +172,10 @@ class LocalCheckpointSuite extends SparkFunSuite with LocalSparkContext { | |||
| // Collecting the RDD should now fail with an informative exception | |||
| val blockId = RDDBlockId(rdd.id, numPartitions - 1) | |||
| bmm.removeBlock(blockId) | |||
| // Wait until the block has been removed successfully. | |||
| eventually(timeout(1 seconds), interval(100 milliseconds)) { | |||
| assert(bmm.getBlockStatus(blockId).size == 0) | |||
There was a problem hiding this comment.
@srowen, actually, do you maybe know why === is preferred over ==? Up to my knowledge, === is preferred as it gives a better error message and I saw several comments saying so by few committers before.
However, it seems some do not think so. I raised an issue about a year ago - databricks/scala-style-guide#36 but I am still confused.
There was a problem hiding this comment.
I was just curious. === might be still preferred per the documentation.
|
Test build #78095 has finished for PR 18314 at commit
|
|
Test build #78098 has finished for PR 18314 at commit
|
asfgit
pushed a commit
that referenced
this pull request
Jun 15, 2017
…dd.LocalCheckpointSuite.missing checkpoint block fails with informative message ## What changes were proposed in this pull request? Currently we don't wait to confirm the removal of the block from the slave's BlockManager, if the removal takes too much time, we will fail the assertion in this test case. The failure can be easily reproduced if we sleep for a while before we remove the block in BlockManagerSlaveEndpoint.receiveAndReply(). ## How was this patch tested? N/A Author: Xingbo Jiang <[email protected]> Closes #18314 from jiangxb1987/LocalCheckpointSuite. (cherry picked from commit 7dc3e69) Signed-off-by: Wenchen Fan <[email protected]>
asfgit
pushed a commit
that referenced
this pull request
Jun 15, 2017
…dd.LocalCheckpointSuite.missing checkpoint block fails with informative message ## What changes were proposed in this pull request? Currently we don't wait to confirm the removal of the block from the slave's BlockManager, if the removal takes too much time, we will fail the assertion in this test case. The failure can be easily reproduced if we sleep for a while before we remove the block in BlockManagerSlaveEndpoint.receiveAndReply(). ## How was this patch tested? N/A Author: Xingbo Jiang <[email protected]> Closes #18314 from jiangxb1987/LocalCheckpointSuite. (cherry picked from commit 7dc3e69) Signed-off-by: Wenchen Fan <[email protected]>
asfgit
pushed a commit
that referenced
this pull request
Jun 15, 2017
…dd.LocalCheckpointSuite.missing checkpoint block fails with informative message ## What changes were proposed in this pull request? Currently we don't wait to confirm the removal of the block from the slave's BlockManager, if the removal takes too much time, we will fail the assertion in this test case. The failure can be easily reproduced if we sleep for a while before we remove the block in BlockManagerSlaveEndpoint.receiveAndReply(). ## How was this patch tested? N/A Author: Xingbo Jiang <[email protected]> Closes #18314 from jiangxb1987/LocalCheckpointSuite. (cherry picked from commit 7dc3e69) Signed-off-by: Wenchen Fan <[email protected]>
|
LGTM, merging to master/2.2/2.1/2.0 |
dataknocker
pushed a commit
to dataknocker/spark
that referenced
this pull request
Jun 16, 2017
…dd.LocalCheckpointSuite.missing checkpoint block fails with informative message ## What changes were proposed in this pull request? Currently we don't wait to confirm the removal of the block from the slave's BlockManager, if the removal takes too much time, we will fail the assertion in this test case. The failure can be easily reproduced if we sleep for a while before we remove the block in BlockManagerSlaveEndpoint.receiveAndReply(). ## How was this patch tested? N/A Author: Xingbo Jiang <[email protected]> Closes apache#18314 from jiangxb1987/LocalCheckpointSuite.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently we don't wait to confirm the removal of the block from the slave's BlockManager, if the removal takes too much time, we will fail the assertion in this test case.
The failure can be easily reproduced if we sleep for a while before we remove the block in BlockManagerSlaveEndpoint.receiveAndReply().
How was this patch tested?
N/A