-
Notifications
You must be signed in to change notification settings - Fork 786
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support get offsets or blocks info from arrow file. #5252
Comments
See #5249 |
Sorry, I didn't find what I was looking for in this project. Could you give me more hints? |
The FileDecoder provides a mechanism to control how the various parts of a file are decoded and processed, as per the linked DF ticket. Getting the blocks and offsets from FileReader isn't very useful as there is no way to actually control the IO that it performs. |
I'm not sure if I understand it correctly. According to the doctest of FileDecoder, we can get footer firstly, then use FileDecoder to read different recordBatches. |
Correct, which gives you the ability to decode said RecordBatch in parallel in much the same way as we do for parquet row groups |
Get it! Thanks. |
|
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
It seems that there isn't a pub function to provide offsets or blocks info from arrow file.
Describe the solution you'd like
In arrow-ipc/src/reader.rs,
Describe alternatives you've considered
Additional context
related to apache/datafusion#8503
The text was updated successfully, but these errors were encountered: