[SPARK-14112] [SQL] [WIP] Unique Constraints over a Set of AttributeReferences

[gatorsmile]

What changes were proposed in this pull request?

This PR is to introduce unique constraints over a set of AttributeReferences. Below are just two of use cases:

• Remove unnecessary Distinct from the plan, as shown in the PR: [SPARK-14032] [SQL] Eliminate Unnecessary Distinct/Aggregate #11854.
• Eliminate the no-op Join from the plan, as shown in the PR:
  [SPARK-11077] [SQL] Join elimination in Catalyst #9089

We can infer the output uniqueness of the current operator through the uniqueness of the child node's output. The bottom-up propagation rule of unique constraints is

• Distinct, Intersect, Except always return distinct values.
• Aggregate has three cases:
  1. When its aggregate expressions is identical to its grouping expressions, it is equivalent to Distinct. It can always return distinct values.
  2. When the child's outputSet is subset of its own outputSet, it keeps the unique constraints of the child.
  3. Otherwise, it does not enforce unique constraints
• UnaryNode Filter, BroadcastHint, Sort, Window, GlobalLimit, LocalLimit, Sample and SubqueryAlias still keep the unique constraints of the child.
• BinaryNode Left-semi Join keeps the unique constraints of the left child.
• Project keeps the unique constraints of the child if and only if the child's outputSet is subset of its own outputSet

How was this patch tested?

TODO: add a set of test cases for verifying the propagation rules

[SparkQA]

Test build #54017 has finished for PR 11930 at commit aea6146.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

@gatorsmile can you close this? looks like a WIP that hasn't been updated in a long time. You have 16 PRs open at the moment, actually.

[gatorsmile]

Sure, let me close it. I will slow it down to make sure my open PRs are less than 20. Thanks!