You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Changefeeds currently work by turning a user-provided list of tables into a set of kv spans to watch and setting up watches on them that emit changed data into a buffer (currently polling via ExportRequest and soon push via RangeFeed). Separately, the buffer is continually read and the changes in it are turned into rows and emitted.
One thing that hasn't yet been solved satisfactorily is how to detect when the set of watched spans change (so that we can handle it). Some situations when this could happen:
Once we support watching a database and a table is added to the database
Once we support watching a partition and the partition definition changes
Once we support change changing the primary index on a table
A closely related issue is detecting schema changes:
We decided in the rfc that attempting to truncate a table being watched by a changefeed will reject the truncate with an error (there's no performant and unsurprising way for us to deal with this in the changefeed) but it's not clear how this will work in the core version (which doesn't have a jobs entry marking the table as watched). Currently, the behavior of both enterprise and core is that truncates are detected and error the changefeed.
All of these are derivable from table and database descriptor changes.
I'm going to use this issue as a log of my thoughts on how to solve this problem.
The text was updated successfully, but these errors were encountered:
danhhz
added
C-cleanup
Tech debt, refactors, loose ends, etc. Solution not expected to significantly change behavior.
A-cdc
Change Data Capture
labels
Aug 22, 2018
Changefeeds currently work by turning a user-provided list of tables into a set of kv spans to watch and setting up watches on them that emit changed data into a buffer (currently polling via ExportRequest and soon push via RangeFeed). Separately, the buffer is continually read and the changes in it are turned into rows and emitted.
One thing that hasn't yet been solved satisfactorily is how to detect when the set of watched spans change (so that we can handle it). Some situations when this could happen:
A closely related issue is detecting schema changes:
All of these are derivable from table and database descriptor changes.
I'm going to use this issue as a log of my thoughts on how to solve this problem.
The text was updated successfully, but these errors were encountered: