detect binlog purged and report to Drainer #1017
Conversation
|
No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md |
csuzhangxc
changed the title
csuzhangxc
changed the title
|
No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md |
|
/run-all-tests tidb=release-4.0 tikv=release-4.0 pd=release-4.0 |
|
/run-all-tests tidb=release-4.0 tikv=release-4.0 pd=release-4.0 |
|
/run-all-tests tidb=release-4.0 tikv=release-4.0 pd=release-4.0 |
| p.logger.Error("some binlogs have been purged in pump") | ||
| }) | ||
|
|
||
| time.Sleep(time.Second) |
There was a problem hiding this comment.
why not exit directly? since isBinlogPurged never set back to false.
There was a problem hiding this comment.
if exit directly, then systemd will restart it again (but that's meaningless). if not exist, log and metrics can still keep working.
for isBinlogPurged, the user should handle it manually and re-establish the replication.
| return | ||
| } | ||
|
|
||
| select { | ||
| case values <- value: | ||
| log.Debug("send value success") | ||
| case <-time.After(gcMaxBlockTime): | ||
| // do not update `last` anymore. | ||
| log.Warn("can not send the binlog for a long time, will try to read again", zap.Duration("duration", gcMaxBlockTime), zap.Int64("current TS", decodeTSKey(iter.Key()))) |
There was a problem hiding this comment.
should we send error to errs here?
There was a problem hiding this comment.
send the binlog for a long time often caused by a slow drainer. if we send errors, the replication will be broken and try to send again, that's should make it worse.
There was a problem hiding this comment.
so, if drainer encounter the purge error , all pump will reach this warning 🤔️, but both drainer and pump still running. until the user manually fix it
There was a problem hiding this comment.
? not all pump, but only this one. and pump should still be running because may other drainers can still work normally.
There was a problem hiding this comment.
oh, I know what you mean. any advice?
|
@csuzhangxc: PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
|
What problem does this PR solve?
detect binlog purged and report to Drainer.
fix TOOL-2236 (internal only)
What is changed and how it works?
Check List
Tests
Code changes
Side effects
Related changes
Release note