Exponential processing time and memory usage in PredicatePushDown #10455
Assignees
Labels
Comments
yingsu00
pushed a commit
to yingsu00/presto
that referenced
this issue
Jul 25, 2018
This is to fix github issue prestodb#10455. We used to inline all predicate when doing push down predicates. This could have problems when the projection columns contains complex expressions like the following: WITH t1 (v) AS (VALUES 1), t2 AS( select if(v = 0, v, v) v from t1 ), t3 AS( select if(v = 0, v, v) v from t2 ), t4 AS( select if(v = 0, v, v) v from t3 ), t5 AS( select if(v = 0, v, v) v from t4 ), t6 AS( select if(v = 0, v, v) v from t5 ), t7 AS( select if(v = 0, v, v) v from t6 ), t8 AS( select if(v = 0, v, v) v from t7 ), t9 AS( select if(v = 0, v, v) v from t8 ), t10 AS( select if(v = 0, v, v) v from t9 ), t11 AS( select if(v = 0, v, v) v from t10 ), t12 AS( select if(v = 0, v, v) v from t11 ), t13 AS( select if(v = 0, v, v) v from t12 ), t14 AS( select if(v = 0, v, v) v from t13 ), t15 AS( select if(v = 0, v, v) v from t14 ), t16 AS( select if(v = 0, v, v) v from t15 ) select * from t16 where v = 0 This short-term fix is to adjust the inlining heuristics to only do it if the expressions are trivial or appear only once (similar to how the InlineProjections rule works)
yingsu00
pushed a commit
that referenced
this issue
Aug 2, 2018
This is to fix github issue #10455. We used to inline all predicate when doing push down predicates. This could have problems when the projection columns contains complex expressions like the following: WITH t1 (v) AS (VALUES 1), t2 AS( select if(v = 0, v, v) v from t1 ), t3 AS( select if(v = 0, v, v) v from t2 ), t4 AS( select if(v = 0, v, v) v from t3 ), t5 AS( select if(v = 0, v, v) v from t4 ), t6 AS( select if(v = 0, v, v) v from t5 ), t7 AS( select if(v = 0, v, v) v from t6 ), t8 AS( select if(v = 0, v, v) v from t7 ), t9 AS( select if(v = 0, v, v) v from t8 ), t10 AS( select if(v = 0, v, v) v from t9 ), t11 AS( select if(v = 0, v, v) v from t10 ), t12 AS( select if(v = 0, v, v) v from t11 ), t13 AS( select if(v = 0, v, v) v from t12 ), t14 AS( select if(v = 0, v, v) v from t13 ), t15 AS( select if(v = 0, v, v) v from t14 ), t16 AS( select if(v = 0, v, v) v from t15 ) select * from t16 where v = 0 This short-term fix is to adjust the inlining heuristics to only do it if the expressions are trivial or appear only once (similar to how the InlineProjections rule works)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Queries of this shape consume exponential amounts of memory and CPU:
The issue appears to be caused by https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/sql/planner/optimizations/PredicatePushDown.java#L203.
One possible short-term fix is to adjust the inlining heuristics to only do it if the expressions are trivial or appear only once (similar to how the InlineProjections rule works)
The longer-term fix is to move PredicatePushdown to iterative optimizer rules that simply avoid unproductive pushdown actions like the one caused by this query.
The text was updated successfully, but these errors were encountered: