Returning complex types from a vector UDF

[dbeavon]

Is there any technique to return a complex type from a vector UDF?

I've been reviewing the available "VectorUdf" functions defined in ArrowFunctions and DataFrameFunctions but I cannot see how a complex type would be returned.

I really like the way we can return a "Microsoft.Spark.Sql.GenericRow" from a standard Udf function, but that seems slow. I'd rather operate on larger batches of data if possible (ie. spark partitions).

Any help would be appreciated. I haven't been able to find anything on my own yet. I discovered an Apache Arrow data type called "Apache.Arrow.StructArray" and I'm wondering if that could be used by way of one of the implementations of VectorUdf in ArrowFunctions. However there are virtually no references to "Apache.Arrow.StructArray" in the project, and no examples.

Another approach I've used to return complex data types from a large batch of a dataframe is by relying on GroupBy/Apply in RelationalGroupedDataset. But I think I'm overusing/abusing that feature.

[dbeavon]

Here is the error I get when attempting to use ArrowFunctions to return a complex type (Apache.Arrow.StructArray):

System.ArgumentException
  HResult=0x80070057
  Message=Apache.Arrow.StructArray is not supported.
  Source=Microsoft.Spark
  StackTrace:
   at Microsoft.Spark.Utils.UdfUtils.GetReturnType(Type type)
   at Microsoft.Spark.Sql.Functions.CreateUdf[TResult](String name, Delegate execute, PythonEvalType evalType)
   at Microsoft.Spark.Sql.Functions.CreateVectorUdf[TResult](String name, Delegate execute)
   at Microsoft.Spark.Sql.ArrowFunctions.VectorUdf[T1,T2,TResult](Func`3 udf)
   at MyProject.Program.TestVectorUdf(DataFrame p_Df)  

[dbeavon]

I stumbled on a reference to the same thing in the issues list:
(issue #826 )

It looks like this is a work-in-progress...

I'm guessing there isn't any other straight-forward way to return complex data from a vector UDF. Will close this discussion for now.

[cosmincatalin]

I really like the way we can return a "Microsoft.Spark.Sql.GenericRow" from a standard Udf function, but that seems slow.

Can you offer an example of this?