[FEA] Support Window functions(lead & lag) for ArrayType

[Dooyoung-Hwang]

I request a feature to run lead & lag functions with ArrayType[AtomicType], ArrayType[ArrayType[AtomicType]] on GPU.

• Test code

from pyspark.sql import Window
data = [(1,0.1, 0.4, 1), (2, 0.2, 0.5,2), (3, -1.0, 2.2,3)]
schema = StructType([ \
    StructField("a",IntegerType(),True), \
    StructField("b",FloatType(),True), \
    StructField("c",FloatType(),True), \
    StructField("d",IntegerType(),True)
  ])
  
win_spec = Window().partitionBy(col("a")).orderBy(desc("d"))
df1 = spark.createDataFrame(data=data,schema=schema).withColumn("arr1", array("b", "c")) \
        .withColumn("arr2", lead(col("arr1"), 1).over(win_spec))
df2 = spark.createDataFrame(data=data,schema=schema).withColumn("arr1", array("b", "c")) \
        .withColumn("arr2", lag(col("arr1"), 1).over(win_spec)) \

print(df1._jdf.queryExecution())
print(df2._jdf.queryExecution())

• Output message in driver log

!Expression lead(arr1#6082, 1, null) cannot run on GPU because expression Lead lead(arr1#6082, 1, null) produces an unsupported type ArrayType(FloatType,true); expression AttributeReference arr1#6082 produces an unsupported type ArrayType(FloatType,true)
!Expression lag(arr1#6187, 1, null) cannot run on GPU because expression Lag lag(arr1#6187, 1, null) produces an unsupported type ArrayType(FloatType,true); expression AttributeReference arr1#6187 produces an unsupported type ArrayType(FloatType,true)

[wbo4958]

@mythrocks Do you have any idea if cudf has supported this ?

[GaryShen2008]

@mythrocks Do you have any idea if cudf has supported this ?

Mithun has merged the PR rapidsai/cudf#8062

[wbo4958]

close this issue, Since the PRs have been merged.