fix: implement consistent formatting for constraint expressions #1985
Conversation
| let expr = into_expr(expr, &schema, &state)?; | ||
| let expr_str = fmt_expr_to_sql(&expr)?; |
There was a problem hiding this comment.
Is this going from SQL string expr to Datafusion Expr and then back to sql string expression?
There was a problem hiding this comment.
Yes exactly.
If the provided expression is already a Datafusion expression then into_expr will simply return it otherwise it will parse the string expression.
Either way if a String is provided or a proper DF expression it needs to be normalized somewhere.
There was a problem hiding this comment.
Which SQL dialect does Datafusion actually use?
There was a problem hiding this comment.
postgres ... but the actual dialect expressions in the delta log should comply with is as of yet unspecified :).
There was a problem hiding this comment.
Should we put SQLglot in between on the python side then, so we can allow more flexibility there?
There was a problem hiding this comment.
We use the generic dialect which in my mind is similar to postgres. Given we are only parsing expressions and not DML or DDL statements we should be fine.
https://docs.rs/sqlparser/latest/sqlparser/dialect/struct.GenericDialect.html
There was a problem hiding this comment.
Should we put SQLglot in between on the python side then, so we can allow more flexibility there?
I'm indifferent to that feature. My only concern is if we start exposing operations to other languages (i.e delta-core) then we should have fairly consistent interfaces.
A decision / documentation needs to be made on which sql dialect must be used when exposing expressions to the log. If I had to guess they would pick hive sql.
Description
Implements consistent formatting for constraint expressions so something like
value < 1000is normalized tovalue < 1000Also includes drive by improvements.
Related Issue(s)