Basic support for `IN` and `NOT IN` Subqueries by rewriting them to `SEMI` / `ANTI` Join #2421

korowa · 2022-05-02T20:08:33Z

Which issue does this PR close?

Partially #488.

Rationale for this change

Naive implementation of optimizer rule for replacing InSubquery with join, which allows to execute queries with IN (subquery) in case of proper WHERE condition.

What changes are included in this PR?

SubqueryFilterToJoin rule is able to replace Filter input in logical plan with Sem/AntiJoin in case InSubquery is a part of logical conjunction - this precondition allows to pushdown IN predicate before other predicates in Filter.

Cases when IN (subquery) cannot be pushed to Filters input due to its result being required for predicate evaluation, are handled by returning NotImplemented error for now.

Are there any user-facing changes?

Queries with IN (subquery) predicates start executing for described above filter combinations

andygrove

Thanks @korowa this is looking good. This actually adds support for benchmark queries 16 and 18, which previously were not supported.

If you want to, you could enable these queries in the benchmark test at benchmarks/src/bin/tpch.rs by adding these tests. This could also be done as a separate PR.

#[tokio::test]
async fn run_q16() -> Result<()> {
    run_query(16).await
}

#[tokio::test]
async fn run_q18() -> Result<()> {
    run_query(18).await
}

andygrove · 2022-05-02T21:26:51Z

datafusion/core/src/optimizer/subquery_filter_to_join.rs

+                })?;
+
+                if !subqueries_in_regular.is_empty() {
+                    return Err(DataFusionError::NotImplemented(


Can we just revert to the original query here rather than fail?

All fixed. My idea was to give user some explanation about why query is "incorrect" by throwing errors.

Now I see that if DF is able to produce this kind of logical plan, then it's valid (at least for some purposes maybe), even if we don't have physical implementation for some of its parts yet.

andygrove · 2022-05-02T21:27:33Z

datafusion/core/src/optimizer/subquery_filter_to_join.rs

+                        let right_key = right_schema.field(0).qualified_column();
+                        let left_key = match *expr.clone() {
+                            Expr::Column(col) => col,
+                            _ => return Err(DataFusionError::NotImplemented(


Same comment here. Can we just abort the optimization attempt rather than fail.

andygrove · 2022-05-02T21:27:48Z

datafusion/core/src/optimizer/subquery_filter_to_join.rs

+                        Ok(())
+                    }
+                    _ => Err(DataFusionError::Plan(
+                        "Unknown expression while rewriting subquery to joins"


Same comment here.

Dandandan · 2022-05-03T12:00:52Z

datafusion/core/src/optimizer/subquery_filter_to_join.rs

+                            )),
+                        };
+
+                        let join_type = match negated {


can we use if/else here?

Dandandan · 2022-05-03T12:03:49Z

datafusion/core/src/optimizer/subquery_filter_to_join.rs

+                        };
+
+                        let schema = build_join_schema(
+                            new_input.schema(),


Better to use optimized_input.schema() and avoid creating a mut

Switched to try_fold, no mutable variables required now

Dandandan · 2022-05-03T12:04:11Z

datafusion/core/src/optimizer/subquery_filter_to_join.rs

+                            null_equals_null: false,
+                        });
+
+                        Ok(())


and return result here to use below

Now folding

alamb

Thank you @korowa. Really nice work. 🏅

This is very cool -- both good code as well as well tested.

alamb · 2022-05-04T01:23:22Z

benchmarks/src/bin/tpch.rs

@@ -1074,6 +1074,16 @@ mod tests {
        run_query(14).await
    }

+    #[tokio::test]


alamb · 2022-05-04T01:24:10Z

datafusion/core/src/optimizer/filter_push_down.rs

@@ -305,7 +267,7 @@ fn optimize(plan: &LogicalPlan, mut state: State) -> Result<LogicalPlan> {
        LogicalPlan::Analyze { .. } => push_down(&state, plan),
        LogicalPlan::Filter(Filter { input, predicate }) => {
            let mut predicates = vec![];
-            split_members(predicate, &mut predicates);
+            utils::split_conjunction(predicate, &mut predicates);


alamb · 2022-05-04T01:24:52Z

datafusion/core/src/optimizer/utils.rs

@@ -556,6 +556,41 @@ pub fn rewrite_expression(expr: &Expr, expressions: &[Expr]) -> Result<Expr> {
    }
 }

+/// converts "A AND B AND C" => [A, B, C]


👍 for moving these functions into utils

alamb · 2022-05-04T01:27:40Z

datafusion/core/src/optimizer/subquery_filter_to_join.rs

+                                )),
+                            };
+
+                            let join_type = if *negated {


alamb · 2022-05-04T01:33:25Z

datafusion/core/src/optimizer/subquery_filter_to_join.rs

+            .build()?;
+
+        let expected = "Projection: #test.b [b:UInt32]\
+        \n  Anti Join: #test.c = #test.c [a:UInt32, b:UInt32, c:UInt32]\


I can't tell from this plan if the predicate is correct (because #test is used as the relation name in both the inner and outer query.

It might make these tests more readable if the relation name in the subquery was something different (like test_sq) so that this join predicate appears as #test.c = #sq_ test.c

Done - now table names in tests are different, so it should by much easier to read

alamb · 2022-05-04T01:34:15Z

datafusion/core/src/optimizer/subquery_filter_to_join.rs

+            .build()?;
+
+        let expected = "Projection: #test.b [b:UInt32]\
+        \n  Semi Join: #test.b = #test.a [a:UInt32, b:UInt32, c:UInt32]\


alamb · 2022-05-04T01:35:48Z

datafusion/core/src/optimizer/subquery_filter_to_join.rs

+        assert_optimized_plan_eq(&plan, expected);
+        Ok(())
+    }
+}


I think it would also be helpful to coverage of the negative cases (aka cases that can't be rewritten like x IN (select ...) OR y = 5?

Added cases for unsupported filter expression and for filters input being rewritten while filter remains untouched (checks that falling back to original query doesn't affect its recursive call results)

alamb

This is really nice work @korowa -- thank you so much.

alamb · 2022-05-04T13:50:44Z

datafusion/core/src/optimizer/subquery_filter_to_join.rs

            .project(vec![col("test.b")])?
            .build()?;

        let expected = "Projection: #test.b [b:UInt32]\
-        \n  Semi Join: #test.c = #test.c [a:UInt32, b:UInt32, c:UInt32]\
+        \n  Semi Join: #test.c = #sq.c [a:UInt32, b:UInt32, c:UInt32]\


much easier to read -- thank you

alamb · 2022-05-04T13:51:47Z

datafusion/core/src/optimizer/subquery_filter_to_join.rs

+    }
+
+    /// Test for filter input modification in case filter not supported
+    /// Outer filter expression not modified while inner converted to join


korowa · 2022-05-04T14:02:35Z

Thanks for reviews! I hope I'll follow up with PR(s?) for currently unsupported cases soon.

naive in subquery implementation

2dcdc67

github-actions bot added the datafusion Changes in the datafusion crate label May 2, 2022

andygrove requested a review from Dandandan May 2, 2022 20:33

andygrove approved these changes May 2, 2022

View reviewed changes

andygrove requested a review from alamb May 2, 2022 21:13

andygrove reviewed May 2, 2022

View reviewed changes

korowa added 2 commits May 3, 2022 07:55

16 and 18 tpch queries enabled in benchmark

4a2fd2c

rollback rewriting instead of fail

4f15cd1

korowa force-pushed the naive_in_subquery branch from 29ad278 to 4f15cd1 Compare May 3, 2022 05:55

Dandandan reviewed May 3, 2022

View reviewed changes

try_fold used for input plan rewriting

4b13c2f

korowa force-pushed the naive_in_subquery branch from 70b833d to 4b13c2f Compare May 3, 2022 14:32

alamb changed the title ~~naive InSubquery implementation~~ Basic support for IN and NOT IN Subqueries by rewriting them to SEMI / ANTI May 4, 2022

alamb approved these changes May 4, 2022

View reviewed changes

korowa force-pushed the naive_in_subquery branch from f6cae49 to e6065b4 Compare May 4, 2022 06:23

test readability & negative test cases

271695a

korowa force-pushed the naive_in_subquery branch from e6065b4 to 271695a Compare May 4, 2022 06:26

alamb approved these changes May 4, 2022

View reviewed changes

alamb merged commit e8ba45c into apache:master May 4, 2022

This was referenced May 4, 2022

TPC-H Query 16 #167

Closed

TPC-H Query 18 #169

Closed

jon-chuang mentioned this pull request May 5, 2022

Add EXISTS and IN subquery rewriting for correlated filters at filter depth 1 #2451

Closed

korowa deleted the naive_in_subquery branch May 5, 2022 09:53

alamb mentioned this pull request May 9, 2022

Optimize queries that contain IN subqueries by rewriting as semi join #488

Closed

alamb changed the title ~~Basic support for IN and NOT IN Subqueries by rewriting them to SEMI / ANTI~~ Basic support for IN and NOT IN Subqueries by rewriting them to SEMI / ANTI Join May 9, 2022

avantgardnerio mentioned this pull request Jul 2, 2022

Optimize where exists sub-queries into aggregate and join #2813

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Basic support for `IN` and `NOT IN` Subqueries by rewriting them to `SEMI` / `ANTI` Join #2421

Basic support for `IN` and `NOT IN` Subqueries by rewriting them to `SEMI` / `ANTI` Join #2421

korowa commented May 2, 2022 •

edited

Loading

andygrove left a comment •

edited

Loading

andygrove May 2, 2022

korowa May 3, 2022

andygrove May 2, 2022

korowa May 3, 2022

andygrove May 2, 2022

korowa May 3, 2022

Dandandan May 3, 2022

korowa May 3, 2022

Dandandan May 3, 2022

korowa May 3, 2022

Dandandan May 3, 2022

korowa May 3, 2022

alamb left a comment

alamb May 4, 2022

alamb May 4, 2022

alamb May 4, 2022

alamb May 4, 2022

alamb May 4, 2022

korowa May 4, 2022

alamb May 4, 2022

alamb May 4, 2022

korowa May 4, 2022 •

edited

Loading

alamb left a comment

alamb May 4, 2022

alamb May 4, 2022

korowa commented May 4, 2022

Basic support for IN and NOT IN Subqueries by rewriting them to SEMI / ANTI Join #2421

Basic support for IN and NOT IN Subqueries by rewriting them to SEMI / ANTI Join #2421

Conversation

korowa commented May 2, 2022 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

andygrove left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

korowa May 4, 2022 • edited Loading

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

korowa commented May 4, 2022

Basic support for `IN` and `NOT IN` Subqueries by rewriting them to `SEMI` / `ANTI` Join #2421

Basic support for `IN` and `NOT IN` Subqueries by rewriting them to `SEMI` / `ANTI` Join #2421

korowa commented May 2, 2022 •

edited

Loading

andygrove left a comment •

edited

Loading

korowa May 4, 2022 •

edited

Loading