distsql: Set operations (UNION, INTERSECT, EXCEPT)

[irfansharif]

Implementation of a processor "core" type from #7587.
Set operations would include UNION, UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL.

This processor, like join processors, would have two input streams and one output stream. Some of existing processor core type implementations under pkg/sql/distsql and spec definitions in pkg/sql/distsql/processors.proto can be used for reference.

[a6802739]

@irfansharif , I could have a look on it.

[irfansharif]

@a6802739: go for it! let me know if you need any pointers.

[a6802739]

@irfansharif , okay, thanks a lot. At first, I'll have a see on it.

[a6802739]

@irfansharif, should I add a Message UnionSpec in pkg/sql/distsql/processors.proto, and generate a new struct Union under pkg/sql/distsql?

When I see the real implemention of aggregator, I couldn't quite understand what does buckets means?
For example in distsql/processors.pb.go

GroupCols []uint32 `protobuf:"varint,2,rep,name=group_cols,json=groupCols" json:"group_cols,omitempty"`

what does the GroupCols means?

And in distsql/aggregator.go, what does the buckets mean?

buckets   map[string]struct{} // The set of bucket keys.

And should I just translate sql/union.go to distsql/union.go?

cc @irfansharif .

[irfansharif]

hey @a6802739,

for the spec definition going with SetSpec would be best, this would internally specify which operation to apply (UNION, INTERSECT, EXCEPT). As for the struct itself going with a simple type set struct { ... } should suffice (under pkg/sql/distsql/set.go).

And should I just translate sql/union.go to distsql/union.go?

I would the implementations would function similarly but keep in mind under distsql we're operating under the abstraction that we simply operate on streams. There's also a lot of additional baggage in sql/union.go given our current structure constructing a planNode to evaluate queries which we don't need to satisfy here.

As far as your questions concerning aggregator.go, feel free to ping me on gitter @irfansharif and will be happy to walk you through it there just for the sake of not polluting this thread. for reference I would advise you to look at distinct.go as opposed to aggregator.go, it's more pertinent to Set as compared to aggregator responsible for grouping aggregators (SUM, AVG, etc).

[irfansharif]

cc @arjunravinarayan, given you expressed interest in taking this on as well.

[RaduBerinde]

This issue has two parts:

• physical processor implementation for set operations
• physical planning for set operations

The first would be better as a one-off project for someone who is not familiar with distsql. The latter is not very difficult but it requires understanding a lot of the existing planning code.

[rjnn]

Hey @a6802739, what's your status on this? I'm thinking of taking this one on since it's been sitting here for a while, unless you've already done some work on this.

[vivekmenezes]

abhishek has been assigned this issue to complete the work on INTERSECT and EXCEPT