GH-41730: [Java] Adding variadicBufferCounts to RecordBatch #41732
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vibhatha
changed the title
|
@github-actions crossbow submit -g java |
|
Revision: 5e91dab Submitted crossbow builds: ursacomputing/crossbow @ actions-ffc3f5185b
|
lidavidm
requested changes
May 22, 2024
java/vector/src/main/java/org/apache/arrow/vector/ipc/message/ArrowRecordBatch.java
Show resolved
Hide resolved
java/vector/src/main/java/org/apache/arrow/vector/ipc/message/ArrowRecordBatch.java
Outdated
Show resolved
Hide resolved
java/vector/src/main/java/org/apache/arrow/vector/VectorLoader.java
Outdated
Show resolved
Hide resolved
java/vector/src/main/java/org/apache/arrow/vector/VectorUnloader.java
Outdated
Show resolved
Hide resolved
github-actions
bot
added
awaiting changes
|
@github-actions crossbow submit -g java |
|
Revision: 67fc7ce Submitted crossbow builds: ursacomputing/crossbow @ actions-312ec2ec22
|
vibhatha
commented
May 22, 2024
| } | ||
|
|
||
| private void appendNodes(FieldVector vector, List<ArrowFieldNode> nodes, List<ArrowBuf> buffers) { | ||
| private long getVariadicBufferCount(FieldVector vector) { |
There was a problem hiding this comment.
there is a slight issue in adding an interface for this. The VectorUnloader and StructVectorUnloader are in vector and c modules respectively. So we cannot have a common interface for both. So would it be fine if we add BaseVectorUnloader interface for both modules? I didn't add it yet.
|
@github-actions crossbow submit -g java |
|
Revision: eb2f473 Submitted crossbow builds: ursacomputing/crossbow @ actions-c98dc1f751
|
lidavidm
reviewed
May 23, 2024
| int length, List<ArrowFieldNode> nodes, List<ArrowBuf> buffers, | ||
| ArrowBodyCompression bodyCompression, List<Long> variadicBufferCounts, boolean alignBuffers, | ||
| boolean retainBuffers) { | ||
| super(); |
There was a problem hiding this comment.
why can't the other constructors delegate here?
There was a problem hiding this comment.
I updated, is it fine?
github-actions
bot
added
awaiting changes
If you mean C data interface things, not yet, still need some work. |
github-actions
bot
added
awaiting change review
lidavidm
approved these changes
May 24, 2024
github-actions
bot
added
awaiting merge
vibhatha
added a commit
to vibhatha/arrow
that referenced
this pull request
May 25, 2024
…ache#41732) ### Rationale for this change This PR adds the `variadicBufferCounts` attribute to `ArrowRecordBatch` in Java module. Furthermore, it also updates the `TypeLayout` functions `getTypeBufferCount` and `getTypeLayout` functions along with the corresponding test cases. Previously these changes were listed as issues apache#40934, apache#40935 and apache#40931. These two tickets will also be closed by this PR. ### What changes are included in this PR? The introduced two functions to `TypeLayout` is deprecating the old API and adds a new API. In this PR we are updating a few modules to use the new API. Corresponding tests for the changed functions have also been added. This also updates the usage of `ArrowRecordBatch` across other modules and `TypeLayout` usage across a few modules. Some modules were excluded as mentioned in the issues non-goals section to be completed in a follow up effort as the scope and required tasks remain at large. These modules will still use the deprecated API for TypeLayouts, but documented in the code for updating to the new API in a follow up effort. ### Closing Subtasks - [X] apache#40934 - [X] apache#40935 - [X] apache#40931 ### Are these changes tested? The changes are tested using existing tests and new tests ### Are there any user-facing changes? Yes **This PR includes breaking changes to public APIs.** * GitHub Issue: apache#41730 Lead-authored-by: Vibhatha Lakmal Abeykoon <[email protected]> Co-authored-by: Vibhatha Abeykoon <[email protected]> Signed-off-by: David Li <[email protected]>
|
After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 0c96be3. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them. |
This was referenced May 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
This PR adds the
variadicBufferCountsattribute toArrowRecordBatchin Java module. Furthermore, it also updates theTypeLayoutfunctionsgetTypeBufferCountandgetTypeLayoutfunctions along with the corresponding test cases. Previously these changes were listed as issues #40934, #40935 and #40931. These two tickets will also be closed by this PR.What changes are included in this PR?
The introduced two functions to
TypeLayoutis deprecating the old API and adds a new API. In this PR we are updating a few modules to use the new API. Corresponding tests for the changed functions have also been added.This also updates the usage of
ArrowRecordBatchacross other modules andTypeLayoutusage across a few modules. Some modules were excluded as mentioned in the issues non-goals section to be completed in a follow up effort as the scope and required tasks remain at large. These modules will still use the deprecated API for TypeLayouts, but documented in the code for updating to the new API in a follow up effort.Closing Subtasks
Are these changes tested?
The changes are tested using existing tests and new tests
Are there any user-facing changes?
Yes
This PR includes breaking changes to public APIs.
variadicBufferCountstoRecordBatch#41730