RelationalGroupedDataset/FlatMapGroupsInPandas Does not Support Decimal

[dbeavon]

Today I was using the RecordBatch UDF variation of groupby/apply. I was very surprised that decimals aren't yet supported in Apache Arrow 2.0. Here is the error.

[2021-04-09T23:16:30.5294618Z] [DESKTOP] [Error] 


[TaskRunner] [0] ProcessStream() failed with exception: System.NotSupportedException: An ArrowArray cannot be built for type Decimal.

   at Apache.Arrow.ArrowArrayFactory.BuildArray(ArrayData data)
   at Apache.Arrow.Ipc.ArrowReaderImplementation.BuildArrays(Schema schema, ByteBuffer messageBuffer, RecordBatch recordBatchMessage)
   at Apache.Arrow.Ipc.ArrowReaderImplementation.CreateArrowObjectFromMessage(Message message, ByteBuffer bodyByteBuffer, IMemoryOwner`1 memoryOwner)
   at Apache.Arrow.Ipc.ArrowStreamReaderImplementation.<>c__DisplayClass10_0.<ReadRecordBatch>b__0(Memory`1 messageBuff)
   at Apache.Arrow.Ipc.ArrowStreamReaderImplementation.ReadRecordBatch()
   at Apache.Arrow.Ipc.ArrowStreamReaderImplementation.ReadNextRecordBatch()
   at Microsoft.Spark.Worker.Command.ArrowBasedCommandExecutor.GetInputIterator(Stream inputStream)+MoveNext() 

Is this a work-in-progress already? Or should I submit a feature request?

I found the error message here:
https://github.com/apache/arrow/blob/master/csharp/src/Apache.Arrow/Arrays/ArrowArrayFactory.cs

... and that seems to allow these two possible types, that are supposedly decimal arrays
Decimal128Array Decimal256Array

... so I'm not sure why I'm getting a failure.

I am going to try casting to floating point for now and see if that works. There are arrows arrays (FloatArray and DoubleArray) that may allow floating point columns in a RecordBatch.

I'd appreciate any information about whether it should be possible to use decimals in a UDF based on RecordBatch .

[dbeavon]

Based on some more investigation, I believe that the decimal type wasn't available at the time that the arrow nuget package (2.0) was released.

It appears that Arrow 2.0 was released in October 2020, but decimals weren't added until March 2021. Here are some links:

• https://issues.apache.org/jira/browse/ARROW-11422
• ARROW-11422: [C#] add decimal support apache/arrow#9356