Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider falling back to Spark on unsupported data sources #174

Closed
nevi-me opened this issue Jun 12, 2022 · 2 comments
Closed

Consider falling back to Spark on unsupported data sources #174

nevi-me opened this issue Jun 12, 2022 · 2 comments
Labels
enhancement New feature or request

Comments

@nevi-me
Copy link
Contributor

nevi-me commented Jun 12, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

It appears that when Blaze encounters a data source that's not yet supported, it panics. An example is the Databricks file system (dbfs), which is interpreted as an object store. I haven't yet tested JDBC sources.

Describe the solution you'd like

If Blaze encounters an unsupported object store or other data source, it should defer back to Spark reading the data.
I appreaciate that this might not result in good performance gains on some workloads, but it at least prevents the entire query from failing.

Describe alternatives you've considered

Supporting custom data sources directly, which will take time. I have a DataFusion RDBMS WIP that I'd be able to extend Blaze with, but it's still far away from being generally usable.

Additional context

I was trying out some TPC-H queries with Blaze, so i'm opening issues for things that I observed.

@nevi-me nevi-me added the enhancement New feature or request label Jun 12, 2022
@richox
Copy link
Collaborator

richox commented Jun 17, 2022

maybe we can add some additional check of the file uri in convertFileSourceScanExec?

@richox
Copy link
Collaborator

richox commented Sep 26, 2023

it is supported now, all unsupported operators (including scan) we fallback to spark execution.

@richox richox closed this as completed Sep 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants