changefeedccl: Prevent slow consumers from blocking rangefeed Processor #32945
Labels
A-cdc
Change Data Capture
C-performance
Perf of queries or internals. Solution not expected to change functional behavior.
Comments
mrtracy
added
C-performance
mrtracy
pushed a commit
to mrtracy/cockroach
that referenced
this issue
Jan 7, 2019
Modifies the rangefeed package so that each individual registration maintains an output buffer which is processed by a dedicated goroutine. This eliminates the possibility of a slow consumer blocking the processor from making progress. Previously, writing events to registrations was synchronous, meaning that a single slow consumer could cause the publishing process to block. If publishing was blocked for too long, the processor along with all registrations were immediately torn down. With this change, a slow consumer will no longer block its siblings; if a single consumer blocks for too long, it will enter an "overflow" state and will be torn down as soon as its existing buffer is fully processed. The registry/registration interface has been modified fairly significantly to accomodate this; the processor itself has seen modest modification (mostly moving functionality to the registry level and removing no-longer-needed support for the previous functionality). Some of the processor tests have been modified to account for the now asynchronous nature of publishing events to individual streams. Fixes cockroachdb#32945 Release note: None
mrtracy
pushed a commit
to mrtracy/cockroach
that referenced
this issue
Jan 14, 2019
Modifies the rangefeed package so that each individual registration maintains an output buffer which is processed by a dedicated goroutine. This eliminates the possibility of a slow consumer blocking the processor from making progress. Previously, writing events to registrations was synchronous, meaning that a single slow consumer could cause the publishing process to block. If publishing was blocked for too long, the processor along with all registrations were immediately torn down. With this change, a slow consumer will no longer block its siblings; if a single consumer blocks for too long, it will enter an "overflow" state and will be torn down as soon as its existing buffer is fully processed. The registry/registration interface has been modified fairly significantly to accomodate this; the processor itself has seen modest modification (mostly moving functionality to the registry level and removing no-longer-needed support for the previous functionality). Some of the processor tests have been modified to account for the now asynchronous nature of publishing events to individual streams. Fixes cockroachdb#32945 Release note: None
mrtracy
pushed a commit
to mrtracy/cockroach
that referenced
this issue
Jan 15, 2019
Modifies the rangefeed package so that each individual registration maintains an output buffer which is processed by a dedicated goroutine. This eliminates the possibility of a slow consumer blocking the processor from making progress. Previously, writing events to registrations was synchronous, meaning that a single slow consumer could cause the publishing process to block. If publishing was blocked for too long, the processor along with all registrations were immediately torn down. With this change, a slow consumer will no longer block its siblings; if a single consumer blocks for too long, it will enter an "overflow" state and will be torn down as soon as its existing buffer is fully processed. The registry/registration interface has been modified fairly significantly to accomodate this; the processor itself has seen modest modification (mostly moving functionality to the registry level and removing no-longer-needed support for the previous functionality). Some of the processor tests have been modified to account for the now asynchronous nature of publishing events to individual streams. Fixes cockroachdb#32945 Release note: None
craig bot
pushed a commit
that referenced
this issue
Jan 16, 2019
33557: storage/rangefeed: per registration buffer r=danhhz a=mrtracy Modifies the rangefeed package so that each individual registration maintains an output buffer which is processed by a dedicated goroutine. This eliminates the possibility of a slow consumer blocking the processor from making progress. Previously, writing events to registrations was synchronous, meaning that a single slow consumer could cause the publishing process to block. If publishing was blocked for too long, the processor along with all registrations were immediately torn down. With this change, a slow consumer will no longer block its siblings; if a single consumer blocks for too long, it will enter an "overflow" state and will be torn down as soon as its existing buffer is fully processed. The registry/registration interface has been modified fairly significantly to accomodate this; the processor itself has seen modest modification (mostly moving functionality to the registry level and removing no-longer-needed support for the previous functionality). Some of the processor tests have been modified to account for the now asynchronous nature of publishing events to individual streams. Fixes #32945 Release note: None Co-authored-by: Matt Tracy <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Work item to track fixing slow-consumer blocking in the rangefeed processor. This involves installing a per-registration event buffer which the processor writes to, along with a per-registration goroutine which reads from that buffer and outputs to the target stream.
Detailed problem and proposal outlined in #32944.
The text was updated successfully, but these errors were encountered: