[FEA] support ConcatWs sql function

[revans2]

Is your feature request related to a problem? Please describe.
it would be great to support the concat_ws SQL function

rapidsai/cudf#3726 was filed to get support from cudf.

[sameerz]

Follow up work on cudf, concatenating arrays of strings: rapidsai/cudf#7727

[tgravescs]

The Spark behavior of concatws:

Separator parameter Api differences:

• SQL can pass column of strings as separator
• python and scala api's, only string as separator

Null Behavior:

• if separator is NULL, columns are null
• if column value is null it leaves off separator and value
• if passing in array of all nulls, then its left out without separator
• if array contains a null, it skips it and leaves separator off -> *** this seems different then cudf (https://github.com/rapidsai/cudf/pull/7929/files) but need to see if can work around***
• if all the values of a row are null, it returns an empty string -> this seems different then cudf behavior

Behavior:

• 0 input columns with separate specified return column of empty strings, except if separator null, then value is null.
• 1 column just return column unless Null separator then need column of null match number of rows
• if column value is empty is puts on separator
• if passing in empty array, then its left off without separator
• if passing array with empty strings, separator included
• 2 or more columns add in separator between column values
• For SQL if column specified as separator, the separator in the corresponding row is used
• Empty Array would normally return empty string, but it appears it actually special if joining with something else, it leaves off the separator for that empty array, comment below shows example

[tgravescs]

Note that concat and concat_ws have different behavior for nulls when all rows are null:

+-------+
|nullcol|
+-------+
|   null|
|notnull|
+-------+

>>> spark.sql("select concat_ws('-', nullcol, nullcol) as res from df").show(truncate=False)
+---------------+
|res            |
+---------------+
|               |
|notnull-notnull|
+---------------+

>>> spark.sql("select concat(nullcol, nullcol) as res from df").show(truncate=False)
+--------------+
|res           |
+--------------+
|null          |
|notnullnotnull|
+--------------+

>>> spark.sql("select concat(null, d) as res from df").show(truncate=False)
+----+
|res |
+----+
|null|
|null|
+----+

[tgravescs]

similar cudf behavior for arrays with nulls doesn't match Spark, cudf will put null if any elements in array null, spark skips them. if all nulls, then get empty string

>>> res = spark.sql("select concat_ws('-', array(1, 2, null, 4)) as res from df")
>>> res.show()
+-----+
|  res|
+-----+
|1-2-4|
|1-2-4|
+-----+

>>> spark.sql("select concat_ws('-', array(null, null)) as res from df").show(truncate=False)
+---+
|res|
+---+
|   |
|   |
+---+

array handling for concat is different:

>>> spark.sql("select concat(array(null, 1), array(d)) as res from df").show(truncate=False)
+---------------+
|res            |
+---------------+
|[null, 1, 123] |
|[null, 1, 1234]|
+---------------+

[tgravescs]

I discovered another weird case with the CPU where is you are concatenating an empty array and then another value, it leaves off the separator:

+--------------+
|      arrnames|
+--------------+
|            []|
|[Alice2, Bob2]|
+--------------+

>>> spark.conf.set("spark.rapids.sql.enabled", "false")
>>> res = df.select(concat_ws('-', df.arrnames, lit('z')))
>>> res.show()
+-------------------------+
|concat_ws(-, arrnames, z)|
+-------------------------+
|                        z|
|            Alice2-Bob2-z|
+-------------------------+

[tgravescs]

another example of arrays with null in middle:

+--------------+
|      arrnames|
+--------------+
|[a, null, ccc]|
|[Alice2, Bob2]|
+--------------+

res = df.select(concat_ws('***',df.arrnames).alias('s'))

+-------------+
|            s|
+-------------+
|      a***ccc|
|Alice2***Bob2|
+-------------+

[tgravescs]

example of spark with array of nulls, doesn't matter how many nulls in array, as long as all nulls, spark skips it and leaves off separator.

+-------+
|   name|
+-------+
|beatles|
|  romeo|
+-------+

dfnew.select(concat_ws("-", array(lit(null)), col("name"), lit('a')).alias("s")).show()
+---------+
|        s|
+---------+
|beatles-a|
|  romeo-a|
+---------+

[tgravescs]

cudf Java layer PR: rapidsai/cudf#8289