Parquet related changes for remote read #707
Conversation
|
For those who are interested in this change, there are more discussions and relevant context in this design doc: https://docs.google.com/document/d/1ZEZqvdohrot0ewtTNeaBtqczOIJ1Q0OnX9PqMMxpOF8/edit# |
| @@ -36,6 +36,9 @@ pub trait TryClone: Sized { | |||
| pub trait ParquetReader: Read + Seek + Length + TryClone {} | |||
| impl<T: Read + Seek + Length + TryClone> ParquetReader for T {} | |||
|
|
|||
| pub trait ThreadSafeParquetReader: ParquetReader + Send + Sync + 'static {} | |||
There was a problem hiding this comment.
An alternative approach, that we could use in DataFusion, would be to implement something like ThreadSafeFileSource
I think the core change that is needed to work with remote s3-like object storages is to make / allow for an async api in the parquet reader.
e.g. #111
Here is one potential way of doing it from @jorgecarleitao 's arrow2 crate: jorgecarleitao/arrow2#260
There was a problem hiding this comment.
The more I think about what we are trying to do in datafusion, the more I am convinced what is needed is some async way to read parquet files - #111
I have some ideas about how to do this -- I will try to write them up / draft them in the next few days
|
Hi @alamb, have you got a chance to work on making parquet reader async? |
No description provided.