Converting generic row result column back into a dataframe again

[dbeavon]

Assuming I return a row from a udf like so...

TestUdfWithReturnAsRowType():

spark/src/csharp/Microsoft.Spark.E2ETest/UdfTests/UdfComplexTypesTests.cs

Line 728 in 3fb684c

public void TestUdfWithReturnAsRowType()

... is there a way to cast the collection of row datatypes back into a dataframe ?

The only thing I've managed to do is this wierd thing where I collect the column and then send the GenericRow collection back thru spark.CreateDataFrame. See below.

            IEnumerable<GenericRow> StuffForFrame = SimplifiedDataFrame.Select(udf22(Col("FiscalYear"), Col("FiscalWeek"), Col("SimplifiedFiscalDate")).As("OutputRow")).Collect()
                            .Select(p => new GenericRow(p.GetAs<Row>(0).Values)).ToList();


            var Again = p_SparkSession.CreateDataFrame(StuffForFrame, SchemaForResults);

This seems very inefficient. I'd rather not collect any data back to the driver at all if I can avoid it. I just want the resulting GenericRow from my UDF to be treated like it is a distributed dataframe.

Hopefully this is clear. Any pointers would be appreciated.

[imback82]

For the example in the test you shared:

Row[] rows = _df.Select(udf(nameCol).As("col"), nameCol).Collect().ToArray();

_df.Select(udf(nameCol).As("col"), nameCol) returns a DataFrame, so I am confused with what you are trying to do: "to cast the collection of row datatypes back into a dataframe ?"

The column type of udf(nameCol).As("col") is the schema you specify in the UDF, so you can keep operating on the dataframe based on the schema. Try to run _df.Select(udf(nameCol).As("col"), nameCol).PrintSchema() to see the schema of the dataframe after applying the UDF.

[dbeavon]

@imback82 Thanks for the reply.

The goal is to unbox that GenericRow from the single column where it lives. The entire thing is living (nested) in a single column of the dataframe. I want a new dataframe that has each of the individual columns of the GenericRow.

+--------------------+
|           OutputRow|
+--------------------+
|[17116, 2020, 1, ...|
|[17117, 2020, 1, ...|
+--------------------+


root
 |-- OutputRow: struct (nullable = true)
 |    |-- DIM_time: integer (nullable = true)
 |    |-- FiscalYear: integer (nullable = true)
 |    |-- FiscalWeek: integer (nullable = true)
 |    |-- FiscalDate: string (nullable = true)

Instead of what you see above, I want a more basic data frame with simple columns named DIM_time, FiscalYear, FiscalWeek, FiscalDate.

The only way I could get that to happen is to collect everything to the driver and then create a fresh dataframe again.

Perhaps there is a Select(Col("OutputRow.DIM_time").As("DIM_time")) mechanism or something like that? Let me know if I'm missing something obvious. I would especially like to avoid collecting to the driver if possible. I suppose I should be able to break apart the columns again with another UDF but it seems a bit crazy/redundant.

The only reason I sent back a GenericRow was because my original UDF had more than one piece of information that it needed to send back in the column results.

[imback82]

Perhaps there is a Select(Col("OutputRow.DIM_time").As("DIM_time")) mechanism or something like that?

Yes, this should work. Or you can just do Select("OutputRow.*").

[dbeavon]

I feel a bit silly. I had not yet tested anything like that yet, because I didn't think it would work. I hadn't ever come across any examples like this and didn't realize that spark dataframes had so much native support for complex nested types.

Even the spark sql seems to allow the use of the nested members:
spark.Sql("SELECT OutputRow.* FROM TEMP_Test").Show();

This flexibility opens up lots of interesting possibilities. I'm guessing that it won't be long and I'll probably overuse/abuse this ... all my columns will have complex datatypes.

I'll mark this question answered. I do have a follow-up question about returning complex types from a vector UDF. The examples that demonstrated complex types were built using basic UDF's ... but it would be helpful to be able to return a complex type from a vector UDF as well. My experiences with the vector UDF's so far have been limited to DataFrameFunctions, so I need to play with ArrowFunctions as well ... maybe they support sending back a vector of GenericRow.