Skip to content

Commit

Permalink
changefeedccl: add prototype of EXPERIMENTAL_CHANGEFEED
Browse files Browse the repository at this point in the history
This implementation is a prototype that will be used to get early
feedback from users about whether the way we structure kafka
topics/partitions/encodings work for them.

It works by polling at a configurable interval (default 1 second). Each
poll picks a new highwater timestamp and sends a kv ExportRequest per
covered span. These requests are configured to return (inline in the
response) any kvs that were written between the previous highwater mark
and the new one. These responses are fed to a RowFetcher for conversion
to sql rows.

There is a 1:1 mapping between tables and which kafka topic they're
emitted to. The topic name is some prefix (an option in the sql command)
plus the table name. Each row is emitted as an kafka entry with key
unset and a json value mapping table field names to corresponding
datums. This format will almost certainly be tweaked in followup PRs but
this is sufficient to get everything working end-to-end with kafka
feeding into elasticsearch. Changed rows are emitted "at least once" and
may be duplicated if there are failures or restarts.

Changefeeds are kept as jobs and can be paused, resumed, and cancelled.

Current limitations:
- All changes are fed through a single node, limiting scalability and
  creating a hotspot on that node
- DELETEs don't work
- Schema changes are ignored
- Interleaved tables create duplicate entries
- Job progress is always 0% (bad ux)

Release note: None
  • Loading branch information
danhhz committed Apr 6, 2018
1 parent f2daf40 commit b6dd0ab
Show file tree
Hide file tree
Showing 11 changed files with 1,232 additions and 114 deletions.
2 changes: 1 addition & 1 deletion Gopkg.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions pkg/ccl/ccl_init.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (
// ccl init hooks
_ "github.com/cockroachdb/cockroach/pkg/ccl/backupccl"
_ "github.com/cockroachdb/cockroach/pkg/ccl/buildccl"
_ "github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl"
_ "github.com/cockroachdb/cockroach/pkg/ccl/cliccl"
_ "github.com/cockroachdb/cockroach/pkg/ccl/importccl"
_ "github.com/cockroachdb/cockroach/pkg/ccl/partitionccl"
Expand Down
Loading

0 comments on commit b6dd0ab

Please sign in to comment.