Preparing ArrowStringDataFrameColumn

[dbeavon]

I'm trying to write a UDF that returns an ArrowStringDataFrameColumn with arbitrary strings. It doesn't seem straightforward.

Can someone please point me to some sample code?
Here I am able to just clone another string column:

            Func<Column, Column, Column> udf2 = VectorUdf<Int32DataFrameColumn, ArrowStringDataFrameColumn, ArrowStringDataFrameColumn>(
                (ages, names) =>
                {

                  
                    if (ages.Length != names.Length)
                    {
                        throw new ArgumentException("Arrays need to be the same length");
                    }


                    var Results = (ArrowStringDataFrameColumn)names.Clone();
                    return Results;



                }
                );

but if I want to build my own results from scratch, I keep getting an exception that says the array is immutable.

The only thing I've been able to find in my googling is this code that initializes the string data in an obscure way (link below). It is hard to believe this would be the only way to prepare string data.

https://github.com/dotnet/corefxlab/blob/e0c8705741ad2b66a1bbc70ff5bf5e02886eca1e/tests/Microsoft.Data.Analysis.Tests/DataFrameTests.cs#L32

Hopefully I'm on the right track. I initially tried to use the "StringDataFrameColumn" but that doesn't seem to be the right class since my UDF's reject it in every way.

[imback82]

Maybe you can do something like

spark/src/csharp/Microsoft.Spark.UnitTest/WorkerFunctionTests.cs

Lines 94 to 96 in a6f4e91

	string[] input = { "arg1" };
	var column = (StringArray)ToArrowArray(input);
	ArrowStringDataFrameColumn ArrowStringDataFrameColumn = ToArrowStringDataFrameColumn(column);

?

@pgovind, do you know an easy way to construct ArrowStringDataFrameColumn?

[pgovind]

if I want to build my own results from scratch, I keep getting an exception that says the array is immutable.

Can you tell me how you are trying to construct an ArrowStringDataFrameColumn here? ArrowStringDataFrameColumn is an immutable column, so it doesn't allow modification after it is constructed. Also, there is no convenient way to construct an ArrowStringDataFrameColumn (ASDFC) right now because nobody has needed it. ASDFC is primarily being used to only shuttle data to/from spark today. However, if you have a good use case (which it sounds like you do here), I can quickly add some convenience APIs.

Current options:

1. If you have an existing ASDFC, you can call ASDFC.Clone() or ASDFC.Clone(indicesYouWantInTheResult) to clone the existing ASDFC or get a subset.
2. If you have an existing ASDFC, you can call ASDFC.Apply(Func<string, string>) to return a new ASDFC. This will call func for every value in your existing ASDFC, and whatever func returns will be added to the new returned ASDFC.
3. If you can't use 1 or 2, and you are working with strings, you can use Reflection. Create an empty ASDFC => Use Reflection to call the private Append(ReadOnlySpan<byte>) method to append a string to the column. I recognize this is inconvenient, so if you fall into this use case here, let me know and I'll consider adding some convenience APIs.

[dbeavon]

@pgovind Thanks for the reply. Sorry for the delay, I was out sick one day and the playing catch-up ever since.

After I posted, I did find ways to build ASDFC indirectly, as I recall it was via the help of a delegate as you pointed out in option 2. It wasn't that intuitive/discoverable. But perhaps it is because I'm still overcoming the learning curve.

Learning this stuff is a bit hard. Things are more complicated than just learning .Net for Spark. There are also some underlying libraries that we have to learn about (ie. more than we might initially want to) like Apache Arrow and Microsoft.Data.Analysis. Those appear to be fundamental building blocks and I suppose there is no way to avoid learning it all .

I did find that Apache Arrow is in need of some TLC in relation to C#. Is there Microsoft guidance somewhere that would direct us to favor Microsoft.Data.Analysis dataframe udfs rather than using Apache Arrow vector udfs? Or is Microsoft perhaps going to consider improving Apache Arrow for C#? You wouldn't think that implementation should be a second-class citizen to all the other languages. And the README for the C# implementation is a bit disconcerting. I wouldn't want my managers to ever find and read that, or they would probably send me back to using the scala or python languages in a hurry.

[dbeavon]

@pgovind
You asked...
Can you tell me how you are trying to construct an ArrowStringDataFrameColumn here?

Consider this example....
https://github.com/dotnet/spark/blob/main/examples/Microsoft.Spark.CSharp.Examples/Sql/Batch/VectorDataFrameUdfs.cs

The example performs a group/apply that results in a dataframe with two integer values. Notice how those two integer columns were constructed using the "Append" method. Given that pattern, suppose a developer decided they wanted to return one additional column, which was some arbitrary string (ASDFC). This is an surprisingly difficult thing to accomplish!

Without having a very deep understanding of how either works, I find myself wondering if RecordBatches are the more preferred class (rather than FxDataFrame's) for these group/apply vector UDF's. Today I initially started trying the FxDataFrames, but I should probably test the behavior of RecordBatches as well since they seem fairly interchangeable in the context of group/apply.

In short, I do agree with you that a discoverable API is needed (rather than using Reflection and calling hidden members). Otherwise these string columns don't feel very approachable for the casual developer. I suspect the need for option 3 will come up quite regularly. Today I unintentionally started writing some code that used the Append() method on an ASDFC and it wasn't working, nor throwing exceptions. It turns out I was unintentionally running the extension method from IEnumerable! ;-)

Maybe your proposed API could simply involve an additional builder class or something like that...