Multiple output columns for UDF (in DataFrameFunctions)

[dbeavon]

I haven't found a pattern for returning more than one column back from an operation on a worker node.

I found Microsoft.Spark.Sql.DataFrameFunctions and related DataFrameUdfWrapper. But the only functionality that seems to be available is the ability to return a single column.

In some of my prior scala code, there are transformations that attach two or more additional columns onto an existing dataframe. You can see pseudocode of that here.

#857 (comment)

... you will see that in this example we only attach a single additional column DIM_ResultSurrogate. But you can see that the code can easily be extended to return other columns as well.

I'm not looking to chain two UDF's. I want to do all the work in a single UDF, and return multiple distinct columns of information if possible. Is there any way to do that? Maybe a way to return some sort of tuples class that will fit into a single arrow column output?

[imback82]

Do you need something like this?

spark/src/csharp/Microsoft.Spark.E2ETest/UdfTests/UdfComplexTypesTests.cs

Line 752 in d5e95b4

// Test UDF that returns a Row object with multiple columns.

Feel free to browse the file since it has all the UDF use cases we support.