You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Many of the current Python integration tests check that queries (or subsets of queries) are going to run on the GPU by inspecting the execution plan made available from the DataFrame API. This approach does not work when Adaptive Query Execution (AQE) is enabled, because the execution plan is determined during query execution. This means we cannot currently test the plugin with AQE to see what issues need to be resolved to fully support it.
Describe the solution you'd like
The Python tests should work with or without AQE enabled. We will need to research how best to do this with PySpark.
Is your feature request related to a problem? Please describe.
Many of the current Python integration tests check that queries (or subsets of queries) are going to run on the GPU by inspecting the execution plan made available from the DataFrame API. This approach does not work when Adaptive Query Execution (AQE) is enabled, because the execution plan is determined during query execution. This means we cannot currently test the plugin with AQE to see what issues need to be resolved to fully support it.
Describe the solution you'd like
The Python tests should work with or without AQE enabled. We will need to research how best to do this with PySpark.
This SO post provides a possible way to use listeners from PySpark:
https://stackoverflow.com/questions/44082957/how-to-add-a-sparklistener-from-pyspark-in-python
Describe alternatives you've considered
N/A
Additional context
N/A
The text was updated successfully, but these errors were encountered: