Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-19268][SS]Disallow adaptive query execution for streaming queries #16683

Closed
wants to merge 1 commit into from
Closed

[SPARK-19268][SS]Disallow adaptive query execution for streaming queries #16683

wants to merge 1 commit into from

Conversation

zsxwing
Copy link
Member

@zsxwing zsxwing commented Jan 23, 2017

What changes were proposed in this pull request?

As adaptive query execution may change the number of partitions in different batches, it may break streaming queries. Hence, we should disallow this feature in Structured Streaming.

How was this patch tested?

test("SPARK-19268: Adaptive query execution should be disallowed").

@tdas
Copy link
Contributor

tdas commented Jan 23, 2017

LGTM.

@SparkQA
Copy link

SparkQA commented Jan 23, 2017

Test build #71881 has finished for PR 16683 at commit 2651015.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@tdas
Copy link
Contributor

tdas commented Jan 24, 2017

jenkins test this

@zsxwing
Copy link
Member Author

zsxwing commented Jan 24, 2017

retest this please

@SparkQA
Copy link

SparkQA commented Jan 24, 2017

Test build #71897 has finished for PR 16683 at commit 2651015.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing
Copy link
Member Author

zsxwing commented Jan 24, 2017

Thanks. Merging to master and 2.1.

asfgit pushed a commit that referenced this pull request Jan 24, 2017
…ries

## What changes were proposed in this pull request?

As adaptive query execution may change the number of partitions in different batches, it may break streaming queries. Hence, we should disallow this feature in Structured Streaming.

## How was this patch tested?

`test("SPARK-19268: Adaptive query execution should be disallowed")`.

Author: Shixiong Zhu <[email protected]>

Closes #16683 from zsxwing/SPARK-19268.

(cherry picked from commit 60bd91a)
Signed-off-by: Shixiong Zhu <[email protected]>
@asfgit asfgit closed this in 60bd91a Jan 24, 2017
@zsxwing zsxwing deleted the SPARK-19268 branch January 24, 2017 06:36
uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
…ries

## What changes were proposed in this pull request?

As adaptive query execution may change the number of partitions in different batches, it may break streaming queries. Hence, we should disallow this feature in Structured Streaming.

## How was this patch tested?

`test("SPARK-19268: Adaptive query execution should be disallowed")`.

Author: Shixiong Zhu <[email protected]>

Closes apache#16683 from zsxwing/SPARK-19268.
cmonkey pushed a commit to cmonkey/spark that referenced this pull request Feb 15, 2017
…ries

## What changes were proposed in this pull request?

As adaptive query execution may change the number of partitions in different batches, it may break streaming queries. Hence, we should disallow this feature in Structured Streaming.

## How was this patch tested?

`test("SPARK-19268: Adaptive query execution should be disallowed")`.

Author: Shixiong Zhu <[email protected]>

Closes apache#16683 from zsxwing/SPARK-19268.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants