Change Schema of Existing Dataframe

[dbeavon]

In scala I occasionally use this syntax to fine-tune the nullability of a column:
spark.createDataFrame(df.rdd, schema=schema)

This allows me to keep the dataframe the same, but make assertions about the nulls. The nulls need to be fine-tuned prior to writing the data to SQL (eg. via com.microsoft.sqlserver.jdbc.spark). The database won't allow loading nullable data into a non-nullable SQL Server column.

Is there any approach for this in the .net libraries? I cannot find the "rdd" member, and I'm assuming that is by design. But it would be helpful to know how to adjust the approach I've used in the past.

Here is more information about the "rdd" approach
https://stackoverflow.com/a/48740115/4455524

The "withcolumn" approach doesn't seem to allow me to customize the nullability.

Any tips would be greatly appreciated.

[dbeavon]

I ended up using SelectExpr and passing it expressions like so:

"COALESCE(SourceModule,'') AS SourceModule",

[dbeavon]

I still think it would be helpful for this project to provide an equivalent for:

spark.createDataFrame(df.rdd, schema=schema)

That is an extremely common way of swapping the null criteria on columns, and it is helpful when using it in conjunction with the SQL Server connector.

[imback82]

Have you tried

spark/src/csharp/Microsoft.Spark/Sql/DataFrameNaFunctions.cs

Line 13 in 3fb684c

public sealed class DataFrameNaFunctions : IJvmObjectReferenceProvider

?

[dbeavon]

@imback82

Thanks for the tip.
No, I didn't know about them and never used them in scala either (presumably they exist there). I searched for a variety of things (eg. "schema") in the repo and it never came up in my searches.

I will take a look. I assume you are telling me that the "Fill" methods will change the definitions of those columns to indicate that they are not nullable. The ultimate goal is to make sure that the SQL Server connector (com.microsoft.sqlserver.jdbc.spark) doesn't complain at me. It complains based on the schema, rather than the contents of the data.

It would help if you can point me to a website or project with a lot more samples. I suspect there are lots of functionality that I'm unaware of. It takes a long time to discover various features in Spark. I've always thought the Spark API isn't very "discoverable" (both on the scala side and .Net). You can usually find what you need via google searches but you have to be pretty specific and already have a really good idea what you are looking for. That will turn up the scala results, and you cross your fingers and try to find the corresponding thing in this github project (which wasn't possible in the case of "createDataFrame" on an existing rdd).