[FEA] elementAt, getArrayItem and getMapValue have different behaviors since Spark-3.1.1 when honoring AnsiMode #2272
Labels
task
Work required that improves the product but is not user facing
Milestone
Is your feature request related to a problem? Please describe.
apache/spark#30297 introduce different behaviors for element_at, getArrayItem and getMapValue when invalid index or key is given to lhs with ansi mode enabled and disabled.
For
select element_at(Array(1,2,3), 4)
:Spark 3.0.2 + spark.sql.ansi.enabled=true:
Spark 3.1.1 + spark.sql.ansi.enabled=true:
For
select element_at(map(1, 'a', 2, 'b'), 3)
:Spark 3.0.2 + spark.sql.ansi.enabled=true:
Spark 3.1.1 + spark.sql.ansi.enabled=true:
Initial support for elementAt, getArrayItem will always return null for invalid index or key.
Describe the solution you'd like
Put corresponding code into 3.1.1 shim layers.
Additional context
Initial PR: #2260
The text was updated successfully, but these errors were encountered: