Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Read "arrow" (IPC and streaming) files usning org.apache.arrow.dataset.jni.NativeDatasetFactory in Java API #13760

Closed
igor-suhorukov opened this issue Jul 31, 2022 · 2 comments · Fixed by #13811
Closed

Read "arrow" (IPC and streaming) files usning org.apache.arrow.dataset.jni.NativeDatasetFactory in Java API #13760

igor-suhorukov opened this issue Jul 31, 2022 · 2 comments · Fixed by #13811

Comments

@igor-suhorukov
Copy link
Contributor

How to fetch "arrow" (IPC and streaming) files usning org.apache.arrow.dataset.jni.NativeDatasetFactory in Java API. This chapter is missing in documentation. It is possible to do in CPP/Python API
@lwhite1
Copy link
Contributor

lwhite1 commented Aug 2, 2022

Hi @igor-suhorukov,

I would need to check, but I don't think this is implemented in Java Dataset yet.

Do you need to use Datasets for your application? (in other words, is your data spread over multiple files or too big to fit in memory?) If not, you should be able to use a FileReader load the data.

@igor-suhorukov
Copy link
Contributor Author

igor-suhorukov commented Aug 3, 2022 •
edited
Loading

This functionality required to implement Arrow file/Stream input format in my use case to process large amount of existing geospatial ARROW format data in Apache Spark data source. Optimized Analytics Package (OAP) for Spark also can leverage this feature of Dataset on JVM. They use FileSystemDatasetFactory in this Spark gazelle_plugin adapter .

Jira ticket for this improvement.

@igor-suhorukov igor-suhorukov mentioned this issue Aug 3, 2022
Open
@lidavidm lidavidm linked a pull request Aug 8, 2022 that will close this issue
ARROW-17303: [Java][Dataset] Read Arrow IPC files by NativeDatasetFactory (#13760) #13811
Merged
lidavidm pushed a commit that referenced this issue Aug 8, 2022
@igor-suhorukov
ARROW-17303: [Java][Dataset] Read Arrow IPC files by NativeDatasetFac…
78351ce 
…tory (#13760) (#13811)

This PR allow developers to create Dataset from ARROW IPC files in JVM code like:
`FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
            FileFormat.ARROW_IPC, arrowDatasetURL);`

It is foundation for Apache Spark arrow data source to process huge existing partitioned datasets in ARROW file format without additional data format conversion

Lead-authored-by: Igor Suhorukov <[email protected]>
Co-authored-by: igor.suhorukov <[email protected]>
Signed-off-by: David Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ARROW-17303: [Java][Dataset] Read Arrow IPC files by NativeDatasetFactory (#13760)
2 participants
@lwhite1 @igor-suhorukov