Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Complex AST expressions can crash with non-matching operand type error #9733

Closed
jlowe opened this issue Nov 15, 2023 · 1 comment · Fixed by #9893
Closed

[BUG] Complex AST expressions can crash with non-matching operand type error #9733

jlowe opened this issue Nov 15, 2023 · 1 comment · Fixed by #9893
Assignees
Labels
bug Something isn't working cudf_dependency An issue or PR with this label depends on a new feature in cudf test Only impacts tests

Comments

@jlowe
Copy link
Member

jlowe commented Nov 15, 2023

Related to rapidsai/cudf#14409. Some expressions can cause the AST to produce a non-matching operand type failure, e.g.:

scala> spark.conf.set("spark.rapids.sql.projectAstEnabled", "true")
scala> spark.range(10).withColumn("x", col("id")).repartition(1).selectExpr("(id < x) == (id < (id + x))").collect
23/11/15 21:30:31 ERROR Executor: Exception in task 0.0 in stage 5.0 (TID 3)
ai.rapids.cudf.CudfException: CUDF failure at:/home/jenkins/agent/workspace/jenkins-spark-rapids-jni_nightly-pre_release-203-cuda11/thirdparty/cudf/cpp/src/ast/expression_parser.cpp:149: An AST expression was provided non-matching operand types.
	at ai.rapids.cudf.ast.CompiledExpression.computeColumn(Native Method)
	at ai.rapids.cudf.ast.CompiledExpression.computeColumn(CompiledExpression.java:88)
	at com.nvidia.spark.rapids.GpuProjectAstExec$$anon$1.$anonfun$next$15(basicPhysicalOperators.scala:452)
[...]

We may need to consider analyzing the Spark expression tree and falling back to the CPU for cases that will trigger the bug reported in rapidsai/cudf#14409 until that is fixed.

@jlowe jlowe added bug Something isn't working ? - Needs Triage Need team to review and classify cudf_dependency An issue or PR with this label depends on a new feature in cudf labels Nov 15, 2023
@sameerz sameerz added test Only impacts tests and removed ? - Needs Triage Need team to review and classify labels Nov 21, 2023
@sameerz
Copy link
Collaborator

sameerz commented Nov 21, 2023

Jason will add a test to see whether the cudf change fixed this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working cudf_dependency An issue or PR with this label depends on a new feature in cudf test Only impacts tests
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants