bulkio: refactor bulk processors to use ProcessorBase #57293
Assignees
Labels
A-disaster-recovery
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-disaster-recovery
Comments
pbardea
added
C-enhancement
5 tasks
craig bot
pushed a commit
that referenced
this issue
Dec 14, 2020
57124: importccl: refactor import processor to use ProcessorBase r=pbardea a=pbardea This commit refactors the import processor to use ProcessorBase. Although this processor doesn't have any inputs and the main utility that ProcessorBase provides is management of draining inputs, there is still some utility for this processor to embed a ProcessorBase. As tracing and other infrastructural improvements are made to ProcessorBase, this allows the import processor to stay up to date with these changes. The way that this processor works is that it kicks off a goroutine on start that will begin the import and start sending progress updates over an internally maintained channel. Next() will continually read from that channel until it is fully consumed, and then read the returned summary generated by the goroutine. Import is build this way so that it can manage concurrency internally to the processor. Relates to #57293. Release note: None Co-authored-by: Paul Bardea <[email protected]>
|
This got closed prematurely -- reopening until the other processor changes are merged. |
craig bot
pushed a commit
that referenced
this issue
Feb 1, 2021
57128: backupccl: refactor backup/restore processors to use ProcessorBase r=adityamaru a=pbardea This PR refactors the remaining processors used in backup/restore to use ProcessorBase. Informs #57293. See individual commits. Co-authored-by: Paul Bardea <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Processors are expected to embed a ProcessorBase. BulkIO processors behave differently from typical SQL processors and have not in the past. It would be good to refactor these to use processor base as it is more actively maintained and supports mechanisms for draining errors through a flow if the bulk IO flows get more complex.
One thing of note is that BulkIO processors typically execute long-lasting requests and therefore benefit from running in parallel (they are frequently waiting on network or IO and are therefore not CPU bound).
The text was updated successfully, but these errors were encountered: