Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-23093][SS] Don't change run id when reconfiguring a continuous processing query. #20282

Closed
wants to merge 6 commits into from
Closed

[SPARK-23093][SS] Don't change run id when reconfiguring a continuous processing query. #20282

wants to merge 6 commits into from

Conversation

jose-torres
Copy link
Contributor

@jose-torres jose-torres commented Jan 16, 2018 •
edited
Loading

What changes were proposed in this pull request?

Keep the run ID static, using a different ID for the epoch coordinator to avoid cross-execution message contamination.

How was this patch tested?

new and existing unit tests

@SparkQA
Copy link

SparkQA commented Jan 16, 2018

Test build #86198 has finished for PR 20282 at commit 8092a4d.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

zsxwing
zsxwing requested changes Jan 16, 2018
View reviewed changes
Copy link
Member

@zsxwing zsxwing left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM except one minor comment about the id.

...src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala Outdated
ContinuousExecution.START_EPOCH_KEY, currentBatchId.toString)
sparkSession.sparkContext.setLocalProperty(
ContinuousExecution.RUN_ID_KEY, runId.toString)
val epochCoordinatorId = UUID.randomUUID.toString
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you add run_id + random_uuid so that it's easy to tell which query this epoch coordinator belongs to?

@zsxwing
Copy link
Member

zsxwing commented Jan 16, 2018

LGTM

@SparkQA
Copy link

SparkQA commented Jan 17, 2018

Test build #86203 has finished for PR 20282 at commit 6bc61b0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing
Copy link
Member

zsxwing commented Jan 17, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Jan 17, 2018

Test build #86236 has finished for PR 20282 at commit 70917a5.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class KafkaContinuousReader(
  • case class KafkaContinuousReadTask(
  • class KafkaContinuousDataReader(
  • class KafkaContinuousWriter(
  • case class KafkaContinuousWriterFactory(
  • class KafkaContinuousDataWriter(
  • case class KafkaSourceOffset(partitionToOffsets: Map[TopicPartition, Long]) extends OffsetV2
  • case class KafkaSourcePartitionOffset(topicPartition: TopicPartition, partitionOffset: Long)

@SparkQA
Copy link

SparkQA commented Jan 17, 2018

Test build #86250 has finished for PR 20282 at commit 70917a5.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class KafkaContinuousReader(
  • case class KafkaContinuousReadTask(
  • class KafkaContinuousDataReader(
  • class KafkaContinuousWriter(
  • case class KafkaContinuousWriterFactory(
  • class KafkaContinuousDataWriter(
  • case class KafkaSourceOffset(partitionToOffsets: Map[TopicPartition, Long]) extends OffsetV2
  • case class KafkaSourcePartitionOffset(topicPartition: TopicPartition, partitionOffset: Long)

@jose-torres
Copy link
Contributor Author

The auto-merge in 70917a5 somehow reverted part of this PR, causing the newly added test to fail.

@SparkQA
Copy link

SparkQA commented Jan 17, 2018

Test build #86285 has finished for PR 20282 at commit f9b2fd1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing
Copy link
Member

zsxwing commented Jan 17, 2018

LGTM. Merging to master and 2.3.

asfgit pushed a commit that referenced this pull request Jan 17, 2018
@jose-torres @zsxwing
[SPARK-23093][SS] Don't change run id when reconfiguring a continuous…
79ccd0c 
… processing query.

## What changes were proposed in this pull request?

Keep the run ID static, using a different ID for the epoch coordinator to avoid cross-execution message contamination.

## How was this patch tested?

new and existing unit tests

Author: Jose Torres <[email protected]>

Closes #20282 from jose-torres/fix-runid.

(cherry picked from commit e946c63)
Signed-off-by: Shixiong Zhu <[email protected]>
@asfgit asfgit closed this in e946c63 Jan 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zsxwing zsxwing zsxwing requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jose-torres @SparkQA @zsxwing