changefeedccl: support watching partitions

[danhhz]

So a geo-partitioned table can have a changefeed on each partition directing it to a geo-local sink. To save cross-ocean bandwidth and eventually for GDPR compliance.

The semantics are fuzzy around what should happen when a watched partition is altered. We could start by erroring either the changefeed or the alter partition and then do something better once we figure out what user expectations are.

Jira issue: CRDB-4905

[danhhz]

Related customer feedback #25229 (comment)

[rolandcrosby]

For the first pass at this in 2.1.x we should support changefeeds on partitions and cancel the changefeed if the partition definition changes. We can try to prioritize graceful handling of repartitioning in a follow-on work item but supporting the "easy" case here will be a quick win and unblock customer use cases.

[danhhz]

Sat down to think about this in detail and I'm leaning strongly to punting all support for this to 2.2. This will require a change to the jobs definition, which will require a minimum version cluster setting bump (because of rolling upgrades). There's no way to make 2.1.0 at this point, so we'd have to cherry-pick this into a patch release, which will get a lot of pushback, even for changefeeds. This is absolutely a new feature, which even given the changefeed exceptions, should ideally be avoided in a patch release.

If forced we could probably do something clever to deal with the above, but we have a reasonable workaround: make one changefeed per kafka cluster. They'll each do a bunch of extra work that will have to be filtered client side. (Sounds like Kindred is planning on doing this anyway).

@rolandcrosby unless you object strongly, I'm going to move this to 2.2

[rolandcrosby]

That's fine with me.

[miretskiy]

Closing this issue; you can target partition with cdc expressions;
it's true that we don't have ability to target to geo specific sink; but
we'll tackle it as part of larger push to better work in multi region.

/cc @amruss -- in case you want to reopen.