Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

changefeedccl: detection of descriptor changes #28982

Closed
danhhz opened this issue Aug 22, 2018 · 1 comment
Closed

changefeedccl: detection of descriptor changes #28982

danhhz opened this issue Aug 22, 2018 · 1 comment
Assignees
Labels
A-cdc Change Data Capture C-cleanup Tech debt, refactors, loose ends, etc. Solution not expected to significantly change behavior.
Milestone

Comments

@danhhz
Copy link
Contributor

danhhz commented Aug 22, 2018

Changefeeds currently work by turning a user-provided list of tables into a set of kv spans to watch and setting up watches on them that emit changed data into a buffer (currently polling via ExportRequest and soon push via RangeFeed). Separately, the buffer is continually read and the changes in it are turned into rows and emitted.

One thing that hasn't yet been solved satisfactorily is how to detect when the set of watched spans change (so that we can handle it). Some situations when this could happen:

  1. Once we support watching a database and a table is added to the database
  2. Once we support watching a partition and the partition definition changes
  3. Once we support change changing the primary index on a table

A closely related issue is detecting schema changes:

  1. We decided in the rfc that attempting to truncate a table being watched by a changefeed will reject the truncate with an error (there's no performant and unsurprising way for us to deal with this in the changefeed) but it's not clear how this will work in the core version (which doesn't have a jobs entry marking the table as watched). Currently, the behavior of both enterprise and core is that truncates are detected and error the changefeed.
  2. The semantics of schema changes that require a backfill are tricky: changefeedccl: support (or workaround) schema changes with backfills #28635

All of these are derivable from table and database descriptor changes.

I'm going to use this issue as a log of my thoughts on how to solve this problem.

@danhhz danhhz added C-cleanup Tech debt, refactors, loose ends, etc. Solution not expected to significantly change behavior. A-cdc Change Data Capture labels Aug 22, 2018
@danhhz danhhz added this to the 2.2 milestone Aug 22, 2018
@danhhz danhhz self-assigned this Aug 22, 2018
@danhhz
Copy link
Contributor Author

danhhz commented Sep 6, 2018

Turns out this was more straightforward than I thought. #29367 does the hard bits and everything left is naturally part of #28633, so closing this

@danhhz danhhz closed this as completed Sep 6, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-cdc Change Data Capture C-cleanup Tech debt, refactors, loose ends, etc. Solution not expected to significantly change behavior.
Projects
None yet
Development

No branches or pull requests

1 participant