You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The more filtering that can be pushed to the parquet reading, the faster a query will run in general as less work is needed to decode and process data that will eventually be filtered from the plan
There are several ongoing workstreams that will eventually lead to pushing down substantial additional filtering into the parquet scan that should substantially increase performance for datafusion. I wanted to capture them here to provide more visibility
Describe the solution you'd like
Here are some of the tasks I have collected. There are likely more -- please add them (either directly or via comments)
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The more filtering that can be pushed to the parquet reading, the faster a query will run in general as less work is needed to decode and process data that will eventually be filtered from the plan
There are several ongoing workstreams that will eventually lead to pushing down substantial additional filtering into the parquet scan that should substantially increase performance for datafusion. I wanted to capture them here to provide more visibility
cc @Ted-Jiang @tustvold @thinkharderdev
Describe the solution you'd like
Here are some of the tasks I have collected. There are likely more -- please add them (either directly or via comments)
ParquetRecordBatchStream
to support row filtering in DataFusion arrow-rs#2270The text was updated successfully, but these errors were encountered: