Numeric, String, Boolean comparisons with literal NULL

[WinkerDu]

Which issue does this PR close?

Closes #1179 and #2482 .

Rationale for this change

This pr aims to solve Numeric, String, Boolean comparisons (Binary Expressions) with literal NULL
For now, DF would fail when running the following SQLs:

> select * from (VALUES (1, 'foo' ,2.3), (2, 'bar', 5.4)) as t;
+---------+---------+---------+
| column1 | column2 | column3 |
+---------+---------+---------+
| 1       | foo     | 2.3     |
| 2       | bar     | 5.4     |
+---------+---------+---------+
2 rows in set. Query took 0.004 seconds.
> select column1 < null from (VALUES (1, 'foo' ,2.3), (2, 'bar', 5.4)) as t;
Plan("'Int64 < Utf8' can't be evaluated because there isn't a common type to coerce the types to")
> select column2 < null from (VALUES (1, 'foo' ,2.3), (2, 'bar', 5.4)) as t;
ArrowError(ExternalError(Internal("compute_utf8_op_scalar for 'lt' failed to cast literal value NULL")))
> select column3 < null from (VALUES (1, 'foo' ,2.3), (2, 'bar', 5.4)) as t;
Plan("'Float64 < Utf8' can't be evaluated because there isn't a common type to coerce the types to")

What changes are included in this PR?

• Introduces null_coercion to comparison_eq_coercion and comparison_order_coercion
• Enhances binary expression macros to support NULL scalar value comparing with array, like binary_array_op_dyn_scalar

Are there any user-facing changes?

No.

[WinkerDu]

cc @alamb @andygrove @yjshen @xudong963
Please have a review, thank you!

[yjshen]

1. I think we should respect SortOptions here for ScalaValue.
2. Question: how about the NULL != column case? Do we have an expression normalizer that asserts Nulls often coming as the right-hand operand for a binary comparison?

[alamb]

@yjshen what do you mean SortOptions? I don't think these expressions are used directly in ORDER BY clauses -- they are used instead for expressions like a < b

Adding some more tests in sql/expr.rs for NULL < column and NULL > column sounds like a good idea.

[yjshen]

After checking Postgres and MySQL, I found I'm wrong on the expected behavior while comparing columns with NULL using < > ..., they will return NULL in all cases. I was expecting to apply null first or null last while comparing column with NULL, so always true or always false at the first time.

[WinkerDu]

Thanks @alamb @yjshen
I agree this macro is just used in expression evaluate, maybe we can ignore SortOptions in this pr.
I'll look into this case NULL != column carefully and add more cases in ut.
Besides, I am super busy these days, I'll do it ASAP 🤟

[alamb]

I will spend some time writing the NULL != column

[alamb]

Follow on PR: #2510 turns out there was a bug!

[alamb]

This looks great to me -- thank you @WinkerDu

I think we should make sure @yjshen is satisfied with the answers about SortOptions and a few more tests would be good. But otherwise 👍

Very nice

[alamb]

@yjshen what do you mean SortOptions? I don't think these expressions are used directly in ORDER BY clauses -- they are used instead for expressions like a < b

Adding some more tests in sql/expr.rs for NULL < column and NULL > column sounds like a good idea.

[yjshen]

Thanks @WinkerDu. I think the current behavior that always yields null while comparing with null is consistent with other systems, so no more efforts are required on this.

It would be nice to guarantee expression normalization exists and always have scalar value as the right operand after normalization. I think this goes beyond the scope of this PR and we could do it as follow-ups.

Thanks for your perseverance and nice work!

[alamb]

Agree with @yjshen -- thank you @WinkerDu

[alamb]

I will spend some time writing the NULL != column