[GLUTEN-5341][VL][TEST] Fix SPARK-42782: Hive compatibility check for get_json_object #5467
Conversation
|
Run Gluten Clickhouse CI |
1 similar comment
|
Run Gluten Clickhouse CI |
|
@zhli1142015 Could you please review? Thanks |
| @@ -126,7 +126,7 @@ class VeloxTestSettings extends BackendTestSettings { | |||
| enableSuite[GlutenHashExpressionsSuite] | |||
| enableSuite[GlutenIntervalExpressionsSuite] | |||
| enableSuite[GlutenJsonFunctionsSuite] | |||
| // Disable for Spark3.5. | |||
| // * in get_json_object expression not supported in velox | |||
There was a problem hiding this comment.
Is there any issue to track this, if not please open one, thanks.
There was a problem hiding this comment.
Sure, I will open one, this is the PR where get_json_object support was added facebookincubator/velox@20e4678
|
CC @PHILO-HE |
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenJsonFunctionsSuite.scala
Outdated
Show resolved
Hide resolved
| runTest(json, "$.store.bicycle", "{\"price\":19.95,\"color\":\"red\"}") | ||
| runTest(json, "$.store.book", book) | ||
| runTest(json, "$.store.book[0]", book0) | ||
| runTest(json, "$.store.book[*]", null) // not supported in velox |
There was a problem hiding this comment.
As null is not the expected result for Spark user, I think we should comment or just remove such test cases. And it would be nice to document this limitation here: https://github.com/apache/incubator-gluten/blob/main/docs/velox-backend-limitations.md#json-functions
There was a problem hiding this comment.
Added, thanks for the suggestion @PHILO-HE
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenJsonFunctionsSuite.scala
Outdated
Show resolved
Hide resolved
ayushi-agarwal
force-pushed
the
json-suite-fix
branch
from
bd8de58
to
7aa8827
Compare
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
I have addressed it, thanks. |
PHILO-HE
changed the title
|
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
|
ulysses-you
changed the title
What changes were proposed in this pull request?
Fixes get_json_object test introduced in spark 3.5 to check compatibility with hive. * character is not supported in velox in json queries.
(Fixes: #5341)
How was this patch tested?
Ran in local