PythonRunner Changes [databricks] #10274
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes NVIDIA#10224 Replace broken install using apt by downloading Maven from apache.org. Signed-off-by: Gera Shegalov <[email protected]>
fix NVIDIA#9493 fix NVIDIA#9844 The python runner uses two separate threads to write and read data with Python processes, however on DB13.3, it becomes single-threaded, which means reading and writing run on the same thread. Now the first reading is always ahead of the first writing. But the original BatchQueue will wait on the first reading until the first writing is done. Then it will wait forever. Change made: - Update the BatchQueue to support asking for a batch instead of waiting unitl one is inserted into the queue. This can eliminate the order requirement of reading and writing. - Introduce a new class named BatchProducer to work with the new BatchQueue to support rows number peek on demand for the reading. - Apply this new BatchQueue to relevant plans. - Update the Python runners to support writing one batch one time for the singled-threaded model. - Found an issue about PythonUDAF and RunningWindoFunctionExec, it may be a bug specific to DB 13.3, and add a test (test_window_aggregate_udf_on_cpu) for it. - Other small refactors --------- Signed-off-by: Firestarman <[email protected]>
This PR moves the BatchQueue into the DataProducer to share the same lock as the output iterator returned by asIterator, and make the batch movement from the input iterator to the batch queue be an atomic operation to eliminate the race when appending the batches to the queue.
…ode. (NVIDIA#9902) Signed-off-by: Firestarman <[email protected]>
…DIA#10232) This PR removes the old 330db shims in favor of the new Shims, similar to the one in 341db. **Tests:** Ran udf_test.py on Databricks 11.3 and they all passed. fixes NVIDIA#10228 --------- Signed-off-by: raza jafri <[email protected]>
razajafri
requested review from
jlowe,
revans2,
tgravescs,
GaryShen2008 and
NvTimLiu
as code owners
sameerz
added
the
task
|
build |
jlowe
reviewed
Jan 26, 2024
| {"spark": "331"} | ||
| {"spark": "332"} | ||
| {"spark": "332cdh"} | ||
| {"spark": "332db"} |
There was a problem hiding this comment.
Curious why this version of Databricks is here -- did it really not get the same changes but the version before and after did?
There was a problem hiding this comment.
Ah, Databricks 12.2 doesn't have this change yet. Sounds like we're about to get broken again when it eventually does. Guess we'll backport to that one in the future when that happens.
jlowe
approved these changes
Jan 26, 2024
NvTimLiu
added a commit
to NvTimLiu/spark-rapids
that referenced
this pull request
Jan 30, 2024
update download page to v23.12.2 for the Databricks hotfix: NVIDIA#10274 Signed-off-by: Tim Liu <[email protected]>
NvTimLiu
added a commit
that referenced
this pull request
Jan 31, 2024
update download page to v23.12.2 for the Databricks hotfix: #10274 Signed-off-by: Tim Liu <[email protected]>
razajafri
added a commit
to razajafri/spark-rapids
that referenced
this pull request
Feb 16, 2024
This reverts commit dacc6fe.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a backport of the changes from branch-24.02 which are required for the plugin to build on Databricks 11.3 after changes from apache/spark#42385 were ported over by the Databricks team.
This PR consists of the following (clean) cherry-picks that were needed. The order of cherry-pick is from bottom to top.
Some changes may not be needed but I left them there so the cherry-pick is a clean pick