Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Java] Filter and project Datasets with Substrait expressions #34252

Closed
ianmcook opened this issue Feb 18, 2023 · 0 comments · Fixed by #35570
Closed

[Java] Filter and project Datasets with Substrait expressions #34252

ianmcook opened this issue Feb 18, 2023 · 0 comments · Fixed by #35570
Labels
Component: Java Type: enhancement
Milestone
14.0.0

Comments

@ianmcook
Copy link
Member

Describe the enhancement requested

The features requested in #33985 will allow Substrait expressions to be passed to the Dataset Filter and Project methods. After that is implemented, we should expose it to Java through the JNI Dataset bindings.

Component(s)

Java
@ianmcook ianmcook mentioned this issue Feb 18, 2023
Open
@davisusanibar davisusanibar mentioned this issue May 12, 2023
Merged
7 tasks
davisusanibar added a commit to davisusanibar/arrow that referenced this issue May 30, 2023
@davisusanibar
Merge branch 'apacheGH-35579-parquet-dataset-field-refs' into apacheG…
d74c116 
…H-34252
davisusanibar added a commit to davisusanibar/arrow that referenced this issue Jun 1, 2023
@davisusanibar
Merge branch 'main' into apacheGH-34252
b0fb9a5
davisusanibar added a commit to davisusanibar/arrow that referenced this issue Jun 6, 2023
@davisusanibar
Merge branch 'main' into apacheGH-34252
dbe2622
davisusanibar added a commit to davisusanibar/arrow that referenced this issue Sep 7, 2023
@davisusanibar
Merge branch 'main' into apacheGH-34252
b9e2818
davisusanibar added a commit to davisusanibar/arrow that referenced this issue Sep 12, 2023
@davisusanibar
Merge branch 'main' into apacheGH-34252
5a4f462
davisusanibar added a commit to davisusanibar/arrow that referenced this issue Sep 15, 2023
@davisusanibar
Merge branch 'main' into apacheGH-34252
a3ddae6
davisusanibar added a commit to davisusanibar/arrow that referenced this issue Sep 15, 2023
@davisusanibar
Merge branch 'apacheGH-34252' of github.com:davisusanibar/arrow into a…
dadc809 
…pacheGH-34252
lidavidm added a commit that referenced this issue Sep 20, 2023
@davisusanibar @westonpace
@benibus @lidavidm @danepitkin
GH-34252: [Java] Support ScannerBuilder::Project or ScannerBuilder::F…
00481a2 
…ilter as a Substrait proto extended expression (#35570)

### Rationale for this change

To close #34252

### What changes are included in this PR?

This is a proposal to try to solve:
1. Receive a list of Substrait scalar expressions and use them to Project a Dataset
- [x] Draft a Substrait Extended Expression to test (this will be generated by 3rd party project such as Isthmus)
- [x] Use C++ draft PR to Serialize/Deserialize Extended Expression proto messages
- [x] Create JNI Wrapper for ScannerBuilder::Project 
- [x] Create JNI API
- [x] Testing coverage
- [x] Documentation

Current problem is: `java.lang.RuntimeException: Inferring column projection from FieldRef FieldRef.FieldPath(0)`. Not able to infer by column position by able to infer by colum name. This problem is solved by #35798

This PR needs/use this PRs/Issues:
- #34834
- #34227
- #35579

2. Receive a Boolean-valued Substrait scalar expression and use it to filter a Dataset
- [x] Working to identify activities

### Are these changes tested?

Initial unit test added.

### Are there any user-facing changes?

No
* Closes: #34252

Lead-authored-by: david dali susanibar arce <[email protected]>
Co-authored-by: Weston Pace <[email protected]>
Co-authored-by: benibus <[email protected]>
Co-authored-by: David Li <[email protected]>
Co-authored-by: Dane Pitkin <[email protected]>
Signed-off-by: David Li <[email protected]>
@lidavidm lidavidm added this to the 14.0.0 milestone Sep 20, 2023
@LuciferYang LuciferYang mentioned this issue Nov 4, 2023
Closed
dongjoon-hyun pushed a commit to apache/spark that referenced this issue Nov 4, 2023
@LuciferYang @dongjoon-hyun
[SPARK-45781][BUILD] Upgrade Arrow to 14.0.0
749c79e 
### What changes were proposed in this pull request?
This pr upgrade Apache Arrow from 13.0.0 to 14.0.0.

### Why are the changes needed?
The Apache Arrow 14.0.0 release brings a number of enhancements and bug fixes.
‎
In terms of bug fixes, the release addresses several critical issues that were causing failures in integration jobs with Spark([GH-36332](apache/arrow#36332)) and problems with importing empty data arrays([GH-37056](apache/arrow#37056)). It also optimizes the process of appending variable length vectors([GH-37829](apache/arrow#37829)) and includes C++ libraries for MacOS AARCH 64 in Java-Jars([GH-38076](apache/arrow#38076)).
‎
The new features and improvements focus on enhancing the handling and manipulation of data. This includes the introduction of DefaultVectorComparators for large types([GH-25659](apache/arrow#25659)), support for extended expressions in ScannerBuilder([GH-34252](apache/arrow#34252)), and the exposure of the VectorAppender class([GH-37246](apache/arrow#37246)).
‎
The release also brings enhancements to the development and testing process, with the CI environment now using JDK 21([GH-36994](apache/arrow#36994)). In addition, the release introduces vector validation consistent with C++, ensuring consistency across different languages([GH-37702](apache/arrow#37702)).
‎
Furthermore, the usability of VarChar writers and binary writers has been improved with the addition of extra input methods([GH-37705](apache/arrow#37705)), and VarCharWriter now supports writing from `Text` and `String`([GH-37706](apache/arrow#37706)). The release also adds typed getters for StructVector, improving the ease of accessing data([GH-37863](apache/arrow#37863)).

The full release notes as follows:
- https://arrow.apache.org/release/14.0.0.html

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #43650 from LuciferYang/arrow-14.

Lead-authored-by: yangjie01 <[email protected]>
Co-authored-by: YangJie <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
loicalleyne pushed a commit to loicalleyne/arrow that referenced this issue Nov 13, 2023
@davisusanibar @westonpace
@benibus @lidavidm @danepitkin @loicalleyne
apacheGH-34252: [Java] Support ScannerBuilder::Project or ScannerBuil…
9c3cb63 
…der::Filter as a Substrait proto extended expression (apache#35570)

### Rationale for this change

To close apache#34252

### What changes are included in this PR?

This is a proposal to try to solve:
1. Receive a list of Substrait scalar expressions and use them to Project a Dataset
- [x] Draft a Substrait Extended Expression to test (this will be generated by 3rd party project such as Isthmus)
- [x] Use C++ draft PR to Serialize/Deserialize Extended Expression proto messages
- [x] Create JNI Wrapper for ScannerBuilder::Project 
- [x] Create JNI API
- [x] Testing coverage
- [x] Documentation

Current problem is: `java.lang.RuntimeException: Inferring column projection from FieldRef FieldRef.FieldPath(0)`. Not able to infer by column position by able to infer by colum name. This problem is solved by apache#35798

This PR needs/use this PRs/Issues:
- apache#34834
- apache#34227
- apache#35579

2. Receive a Boolean-valued Substrait scalar expression and use it to filter a Dataset
- [x] Working to identify activities

### Are these changes tested?

Initial unit test added.

### Are there any user-facing changes?

No
* Closes: apache#34252

Lead-authored-by: david dali susanibar arce <[email protected]>
Co-authored-by: Weston Pace <[email protected]>
Co-authored-by: benibus <[email protected]>
Co-authored-by: David Li <[email protected]>
Co-authored-by: Dane Pitkin <[email protected]>
Signed-off-by: David Li <[email protected]>
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
@davisusanibar @westonpace
@benibus @lidavidm @danepitkin @dgreiss
apacheGH-34252: [Java] Support ScannerBuilder::Project or ScannerBuil…
5becae0 
…der::Filter as a Substrait proto extended expression (apache#35570)

### Rationale for this change

To close apache#34252

### What changes are included in this PR?

This is a proposal to try to solve:
1. Receive a list of Substrait scalar expressions and use them to Project a Dataset
- [x] Draft a Substrait Extended Expression to test (this will be generated by 3rd party project such as Isthmus)
- [x] Use C++ draft PR to Serialize/Deserialize Extended Expression proto messages
- [x] Create JNI Wrapper for ScannerBuilder::Project 
- [x] Create JNI API
- [x] Testing coverage
- [x] Documentation

Current problem is: `java.lang.RuntimeException: Inferring column projection from FieldRef FieldRef.FieldPath(0)`. Not able to infer by column position by able to infer by colum name. This problem is solved by apache#35798

This PR needs/use this PRs/Issues:
- apache#34834
- apache#34227
- apache#35579

2. Receive a Boolean-valued Substrait scalar expression and use it to filter a Dataset
- [x] Working to identify activities

### Are these changes tested?

Initial unit test added.

### Are there any user-facing changes?

No
* Closes: apache#34252

Lead-authored-by: david dali susanibar arce <[email protected]>
Co-authored-by: Weston Pace <[email protected]>
Co-authored-by: benibus <[email protected]>
Co-authored-by: David Li <[email protected]>
Co-authored-by: Dane Pitkin <[email protected]>
Signed-off-by: David Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: Java Type: enhancement
Projects
None yet
Milestone
14.0.0
2 participants
@ianmcook @lidavidm