Fix Multithreaded Readers working with Unity Catalog on Databricks [databricks] #8296
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Thomas Graves <[email protected]>
|
need to fix 3.4 |
Signed-off-by: Thomas Graves <[email protected]>
|
Spark 3.4 changed the PartitionedFile.filePath to be a SparkPath so had to shim it |
Note shims could have been avoided (with a miniscule perf hit on Spark 3.4+) by simply tacking a |
tgravescs
changed the title
|
build |
jlowe
previously approved these changes
May 16, 2023
abellina
reviewed
May 16, 2023
sql-plugin/src/main/spark330db/scala/com/nvidia/spark/rapids/shims/ReaderUtils.scala
Outdated
Show resolved
Hide resolved
|
failure fetching a jar, going to rekick tests |
|
build |
…hims/ReaderUtils.scala Co-authored-by: Alessandro Bellina <[email protected]>
abellina
approved these changes
May 16, 2023
razajafri
added a commit
to razajafri/spark-rapids
that referenced
this pull request
Apr 30, 2024
…ricks [databricks] (NVIDIA#8296)" This reverts commit 5f40711.
razajafri
added a commit
to razajafri/spark-rapids
that referenced
this pull request
Apr 30, 2024
…ricks [databricks] (NVIDIA#8296)" This reverts commit 5f40711.
razajafri
added a commit
to razajafri/spark-rapids
that referenced
this pull request
Apr 30, 2024
…ricks [databricks] (NVIDIA#8296)"
razajafri
added a commit
that referenced
this pull request
May 1, 2024
…tabricks] (#10756) * Use cached ThreadPoolExecutor * Revert "Fix Multithreaded Readers working with Unity Catalog on Databricks [databricks] (#8296)" * Signing off Signed-off-by: Raza Jafri <[email protected]> * Removed spark311 version of ReaderUtils.scala --------- Signed-off-by: Raza Jafri <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of #8210
The issue here is our multi-threaded and coalescing readers fail to read with - org.apache.spark.util.TaskCompletionListenerException: org.apache.spark.SparkException: Missing Credential Scope.
This happens when the Unity Catalog feature on databricks is enabled. Interestingly enough, its just on at the account level even those this particular job isn't using it directly.
This PR only handles Parquet files because that is all I'm currently able to test with. I am working on setting up our own environment to test Unity Catalog with but at the moment I tested this fix in customers Databricks 11.3 environment. I tested multithreaded, coalescing, and perfile. For coalescing I also tested with filtering in parallel on and off.
the only way I've been able to reproduce at this point is to do a delta write to one table and then a read from another one. I have limited options as its customer env. My guess might be that the write is setting up credentials scope to one thing and then when we go to do the read the scope is set wrong and we have to explicitly set it for the read to work.
For the fix, I found a way to get the hadoop conf that seems to have the necessary confs for credentials and you use that conf in the reader threads.