Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
changefeedccl: add prototype of EXPERIMENTAL_CHANGEFEED
This implementation is a prototype that will be used to get early feedback from users about whether the way we structure kafka topics/partitions/encodings work for them. It works by polling at a configurable interval (default 1 second). Each poll picks a new highwater timestamp and sends a kv ExportRequest per covered span. These requests are configured to return (inline in the response) any kvs that were written between the previous highwater mark and the new one. These responses are fed to a RowFetcher for conversion to sql rows. There is a 1:1 mapping between tables and which kafka topic they're emitted to. The topic name is some prefix (an option in the sql command) plus the table name. Each row is emitted as an kafka entry with key unset and a json value mapping table field names to corresponding datums. This format will almost certainly be tweaked in followup PRs but this is sufficient to get everything working end-to-end with kafka feeding into elasticsearch. Changed rows are emitted "at least once" and may be duplicated if there are failures or restarts. Changefeeds are kept as jobs and can be paused, resumed, and cancelled. Current limitations: - All changes are fed through a single node, limiting scalability and creating a hotspot on that node - DELETEs don't work - Schema changes are ignored - Interleaved tables create duplicate entries - Job progress is always 0% (bad ux) Release note: None
- Loading branch information