ARROW-9861: [Java] Support big-endian in DecimalVector #8056

kiszk · 2020-08-26T06:22:55Z

This PR fixes failures in DecimalVectorTest on a big-endian platform

github-actions · 2020-08-26T06:23:42Z

https://issues.apache.org/jira/browse/ARROW-9861

emkornfield · 2020-10-28T02:50:26Z

@kiszk you'll probably need to make some fixes for Decimal256 as well, I think.

@BryanCutler you mentioned you could devote some time to reviewing Big Endian changes? Would you mind taking a look through this one and @kiszk other Java changes?

BryanCutler · 2020-10-28T03:23:43Z

Sure, I can take a look. It might a day or two before I can though @kiszk .

kiszk · 2020-10-28T03:55:54Z

@emkornfield Sure, I will work for supporting Decimal256.

BryanCutler

LGTM, just a couple minor things

BryanCutler · 2020-10-30T22:29:49Z

java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java

-    PlatformDependent.putLong(addressOfValue + Long.BYTES, padValue);
+    if (LITTLE_ENDIAN) {
+      PlatformDependent.putLong(addressOfValue, value);
+      PlatformDependent.putLong(addressOfValue + Long.BYTES, padValue);


This seems a bit odd that it's writing long values at 2 different indices, but I guess that was here before. Do you know what it's trying to do here?

Hmm, I guess it's to write a long value to be used as BigDecimal? So it writes the long in 8-bytes and then pads the remaining 8-bytes. It would be nice if the doc was a little better, but no big deal..

This is used for DecialVector that is a fixed-width (16-byte) vector. This routine extends the signed bit to a new long. I will write a document in the comment.

In addition, this PR will support to write it to a 256-bit entry for the future.

BryanCutler · 2020-10-30T22:40:09Z

java/vector/src/main/java/org/apache/arrow/vector/DecimalVector.java

    } else {
-      throw new IllegalArgumentException(
-          "Invalid decimal value length. Valid length in [1 - 16], got " + length);
+      if (length <= TYPE_WIDTH) {


why not return if length == TYPE_WIDTH?

If we add if (length == TYPE_WIDTH) return before line 244, no data is copied from value to outAddress. I will add a comment copy data from value to outAddress after line 244.

ok, I see above it is already set during the swap. I guess there is no harm in calling PlatformDependent.setMemory(outAddress, DecimalVector.TYPE_WIDTH - length, pad) if length == TYPE_WIDTH

BryanCutler · 2020-10-30T22:42:17Z

java/vector/src/main/java/org/apache/arrow/vector/DecimalVector.java

    }
+    throw new IllegalArgumentException(
+        "Invalid decimal value length. Valid length in [1 - 16], got " + length);


This should probably be in a Preconditions.checkArgument, but we are not trying to change things in this PR so don't need to do that here

BryanCutler · 2020-11-02T05:00:51Z

java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java

+      }
+
+      // Write LE data
+      byte [] padByes = bytes[0] < 0 ? minus_one : zeroes;


you could move padBytes out of the if statements

BryanCutler · 2020-11-02T05:03:58Z

java/vector/src/test/java/org/apache/arrow/vector/util/DecimalUtilityTest.java

+        Assert.assertEquals(expected, actual);
+      }
+
+      long [] longValues = new long[] {Long.MIN_VALUE, 0 , Long.MAX_VALUE};


I don't think you need 2 loops here, just move the Integer.MAX_VALUE and MIN_VALUE here

Sure, I will drop lines 49-57.

kiszk · 2020-11-02T10:02:41Z

I will update the Decimal256Vector class late today or tomorrow.

BryanCutler

Looks good, just a few minor nits

BryanCutler · 2020-11-02T17:49:36Z

java/vector/src/main/java/org/apache/arrow/vector/DecimalVector.java

    } else {
-      throw new IllegalArgumentException(
-          "Invalid decimal value length. Valid length in [1 - 16], got " + length);
+      if (length <= TYPE_WIDTH) {


ok, I see above it is already set during the swap. I guess there is no harm in calling PlatformDependent.setMemory(outAddress, DecimalVector.TYPE_WIDTH - length, pad) if length == TYPE_WIDTH

BryanCutler · 2020-11-02T21:53:32Z

java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java

-    PlatformDependent.putLong(addressOfValue, value);
+  public static void writeLongToArrowBuf(long value, ArrowBuf bytebuf, int index, int byteWidth) {
+    if (byteWidth != 16 && byteWidth != 32) {
+      throw new UnsupportedOperationException("DeciimalUtility.writeLongToArrowBuf() currently supports " +


typo: DeciimalUtility -> DecimalUtility

BryanCutler · 2020-11-02T21:55:03Z

java/vector/src/test/java/org/apache/arrow/vector/complex/impl/TestComplexCopier.java

@@ -526,7 +526,7 @@ public void testCopyFixedSizedListOfDecimalsVector() {
      to.addOrGetVector(FieldType.nullable(new ArrowType.Decimal(3, 0, 128)));

      DecimalHolder holder = new DecimalHolder();
-      holder.buffer = allocator.buffer(DecimalUtility.DECIMAL_BYTE_LENGTH);
+      holder.buffer = allocator.buffer(16);


Could you use DecimalVector.TYPE_WIDTH here?

BryanCutler · 2020-11-02T21:55:17Z

java/vector/src/test/java/org/apache/arrow/vector/complex/writer/TestComplexWriter.java

@@ -310,7 +310,7 @@ public void listDecimalType() {
      listVector.allocateNew();
      UnionListWriter listWriter = new UnionListWriter(listVector);
      DecimalHolder holder = new DecimalHolder();
-      holder.buffer = allocator.buffer(DecimalUtility.DECIMAL_BYTE_LENGTH);
+      holder.buffer = allocator.buffer(16);


BryanCutler

LGTM. Looks like a flaky test, will try again.

BryanCutler · 2020-11-04T18:17:32Z

The test error with Java JNI looks unrelated and seems to be an env issue with ORC, I'll go ahead with merging this.

BryanCutler · 2020-11-04T18:25:10Z

merged to master, thanks @kiszk !

kiszk · 2020-11-05T05:14:15Z

@BryanCutler Thank you. One comment.
According to the document, benchmarking is necessary before merging features (I think that the test code does not affect the performance)

Benchmarks for performance critical parts of the code to demonstrate no regression.

I am working for benchmark bot for Java here. It would be good to merge new features after this bot will be available.

cc @emkornfield

BryanCutler · 2020-11-05T18:50:45Z

I did not see anything that looks like it would affect performance here, but I agree we should get some benchmarks going to be sure. I will look at you other PR next.

This PR fixes failures in DecimalVectorTest on a big-endian platform Closes apache#8056 from kiszk/ARROW-9861 Authored-by: Kazuaki Ishizaki <[email protected]> Signed-off-by: Bryan Cutler <[email protected]>

kiszk changed the title ~~ARROW-9861: [java] Support big-endian in DecimalVector~~ ARROW-9861: [Java] Support big-endian in DecimalVector Aug 26, 2020

kiszk force-pushed the ARROW-9861 branch from 2c5c7e6 to 1eab081 Compare October 6, 2020 18:06

kiszk force-pushed the ARROW-9861 branch from 1eab081 to 896f378 Compare October 30, 2020 06:38

BryanCutler requested changes Nov 2, 2020

View reviewed changes

kiszk force-pushed the ARROW-9861 branch from 896f378 to 89ee07a Compare November 2, 2020 10:01

BryanCutler reviewed Nov 2, 2020

View reviewed changes

kiszk added 5 commits November 3, 2020 02:08

support big-endian in DecimalVector

1db584c

address review comments

0d696fa

support multiple byteWidth in writeLongToArrowBuf()

aa14a87

support BigEndian in Decimal256Vector

d62d990

make code simple

ff33c3c

kiszk force-pushed the ARROW-9861 branch from b6d6562 to 8b530eb Compare November 3, 2020 02:09

BryanCutler approved these changes Nov 3, 2020

View reviewed changes

address review comments

d9e9ce9

kiszk force-pushed the ARROW-9861 branch from 8b530eb to d9e9ce9 Compare November 3, 2020 05:22

BryanCutler closed this in 5355f7c Nov 4, 2020

asfimport mentioned this pull request Nov 5, 2020

[Java] Failed Arrow Vector on big-endian platform #25898

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-9861: [Java] Support big-endian in DecimalVector #8056

ARROW-9861: [Java] Support big-endian in DecimalVector #8056

kiszk commented Aug 26, 2020

github-actions bot commented Aug 26, 2020

emkornfield commented Oct 28, 2020

BryanCutler commented Oct 28, 2020

kiszk commented Oct 28, 2020

BryanCutler left a comment

BryanCutler Oct 30, 2020

BryanCutler Nov 2, 2020

kiszk Nov 2, 2020

kiszk Nov 2, 2020

BryanCutler Oct 30, 2020

kiszk Nov 2, 2020

BryanCutler Nov 2, 2020

BryanCutler Oct 30, 2020

BryanCutler Nov 2, 2020

BryanCutler Nov 2, 2020

kiszk Nov 2, 2020

kiszk commented Nov 2, 2020

BryanCutler left a comment

BryanCutler Nov 2, 2020

BryanCutler Nov 2, 2020

BryanCutler Nov 2, 2020

BryanCutler Nov 2, 2020

BryanCutler left a comment

BryanCutler commented Nov 4, 2020

BryanCutler commented Nov 4, 2020

kiszk commented Nov 5, 2020

BryanCutler commented Nov 5, 2020

ARROW-9861: [Java] Support big-endian in DecimalVector #8056

ARROW-9861: [Java] Support big-endian in DecimalVector #8056

Conversation

kiszk commented Aug 26, 2020

github-actions bot commented Aug 26, 2020

emkornfield commented Oct 28, 2020

BryanCutler commented Oct 28, 2020

kiszk commented Oct 28, 2020

BryanCutler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kiszk commented Nov 2, 2020

BryanCutler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BryanCutler left a comment

Choose a reason for hiding this comment

BryanCutler commented Nov 4, 2020

BryanCutler commented Nov 4, 2020

kiszk commented Nov 5, 2020

BryanCutler commented Nov 5, 2020