Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combine Expr::Wildcard and Wxpr::QualifiedWildcard, add wildcard() expr fn #8105

Merged
merged 1 commit into from
Nov 12, 2023

Conversation

alamb
Copy link
Contributor

@alamb alamb commented Nov 9, 2023

Which issue does this PR close?

Closes #8104

Rationale for this change

Per #8104

As @waynexia points out on #8008 (comment), having Expr::Wildcard and Expr::QualifiedWildcard is confusing as they are both representing the same basic pattern, *

Specifically, I think it could make subtle bugs more likely (e.g. using Expr::Wildcard instead of using Expr::QualifiedWildcard)

What changes are included in this PR?

  1. Change Expr::Wildcard to ExprWildcard { qualifier: Option<String> } and remove Expr::QualifiedWildcard
  2. Update code
  3. Add LogicalPlan serialization support

Are these changes tested?

Are there any user-facing changes?

Yes, users can't use Expr::Wildcard anymore -- they either have to use Expr::Wildcard { qualifier: None} or wildcard()

@github-actions github-actions bot added sql SQL Planner logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate labels Nov 9, 2023
@alamb alamb added the api change Changes the API exposed to users of the crate label Nov 9, 2023
@alamb alamb marked this pull request as ready for review November 9, 2023 13:37
.aggregate(vec![col("b")], vec![count(Wildcard)])?
.sort(vec![count(Wildcard).sort(true, false)])?
.aggregate(vec![col("b")], vec![count(wildcard())])?
.sort(vec![count(wildcard()).sort(true, false)])?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this shows the difference in API for users (calling wildcard()), which I think is more consistent with other functions

projected_expr.extend(expand_wildcard(input_schema, &plan, None)?)
}
Expr::QualifiedWildcard { ref qualifier } => projected_expr
.extend(expand_qualified_wildcard(qualifier, input_schema, None)?),
Expr::Wildcard {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here is the only place where Wildcard and QualifiedWildcard are treated substantially differently

@@ -1052,11 +1054,6 @@ impl TryFrom<&Expr> for protobuf::LogicalExprNode {
})),
}
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a bonus, qualified wildcards can now be serialized to/from proto -- I doubt this feature is widely used, but it is nice to avoid the rough edge I think

Copy link
Member

@waynexia waynexia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me. Thanks for taking this 👍

Comment on lines +111 to +113
pub fn wildcard() -> Expr {
Expr::Wildcard { qualifier: None }
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a fn wildcard_with_qualifier() or fn wildcard(qualifier: None)? Though it's not used in our examples.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking that it was such a niche usecase that if anyone needed the qualifier they could make it explicitly via Expr::Wildcard{ qualifier: Some(...)}, but that is indeed messier

I would be happy to add qualified_wildcard or similar if others think it would be valuable

@alamb
Copy link
Contributor Author

alamb commented Nov 9, 2023

I plan to leave this PR open for a few days to given anyone who might be interested time to comment. Thank you @waynexia for the review 🙏

@alamb alamb merged commit 96ef1af into apache:main Nov 12, 2023
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api change Changes the API exposed to users of the crate core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules sql SQL Planner
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Combine Expr::Wildcard and Expr::QualifiedWildcard, add wildcard() expr fn
2 participants