Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Python tests should support running with Adaptive Query Execution enabled #146

Closed
andygrove opened this issue Jun 10, 2020 · 0 comments · Fixed by #462
Closed

[FEA] Python tests should support running with Adaptive Query Execution enabled #146

andygrove opened this issue Jun 10, 2020 · 0 comments · Fixed by #462
Assignees
Labels
feature request New feature or request test Only impacts tests

Comments

@andygrove
Copy link
Contributor

andygrove commented Jun 10, 2020

Is your feature request related to a problem? Please describe.
Many of the current Python integration tests check that queries (or subsets of queries) are going to run on the GPU by inspecting the execution plan made available from the DataFrame API. This approach does not work when Adaptive Query Execution (AQE) is enabled, because the execution plan is determined during query execution. This means we cannot currently test the plugin with AQE to see what issues need to be resolved to fully support it.

Describe the solution you'd like
The Python tests should work with or without AQE enabled. We will need to research how best to do this with PySpark.

This SO post provides a possible way to use listeners from PySpark:
https://stackoverflow.com/questions/44082957/how-to-add-a-sparklistener-from-pyspark-in-python

Describe alternatives you've considered
N/A

Additional context
N/A

@andygrove andygrove added feature request New feature or request ? - Needs Triage Need team to review and classify test Only impacts tests labels Jun 10, 2020
@andygrove andygrove self-assigned this Jun 23, 2020
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Jun 29, 2020
@andygrove andygrove added this to the Aug 3 - Aug 14 milestone Aug 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request test Only impacts tests
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants