-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Databricks 11.3 exception - Multithreaded parquet reader - Missing Credential Scope #8210
Comments
Note the exception is related to unity catalog and the job that had this failure had that enabled. |
Note the stack trace changes for the location it gets thrown when using the coalescing reader without parallel footer filtering. So it seems it definitely just happens when we are doing the parsing in separate threads so likely something local or something we aren't passing on. |
the only way I've been able to reproduce at this point is to do a delta write to a one table and then a read from another one. I have limited options as its customer env. My guess might be that the write is setting up credentials scope to one thing and then when we go to do the read the scope is set wrong and we have to explicitly set it for the read to work. |
parquet was handled under PR 8296. Also filed #8242 to find a good way to integration test against Unity |
Also need to make sure to test Unity with Alluxio |
I believe the thing left here was just to try to find a reproduce case to add specific test for this fix. |
Describe the bug
Seeing an exception running on Databricks 11.3 when reading parquet files, I tried with both 23.04 released jar and 23.06 snapshot jar, here it is using the multithreaded reader. Fails with the coalescing reader as well, but the PERFILE worked.
The text was updated successfully, but these errors were encountered: