Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Prevent the MaxReplicationLagModule from flooding the vttablet log #14874

Closed
ejortegau opened this issue Jan 2, 2024 · 0 comments · Fixed by #14875
Closed

Comments

@ejortegau
Copy link
Contributor

Feature Description

If the MaxReplicationLagModuleis used (e.g., because you enable the Transaction Throttler), the vttablet logs are FULL with messages like these:

...
I0102 08:27:27.659354   39655 max_replication_lag_module.go:384] rate was: not changed from: 3215 to: 3215
alias: us_east_1b-0170349799 lag: 0s
last change: 0.1s rate: 0 good/bad?  skipped b/c:  good/bad: 485/0
state (old/tested/new): I/I/I
lag before: n/a (n/a ago) rates (primary/replica): 0/0 backlog (old/new): 0/0
reason: skipping this replica because we're waiting for the next lag record from the 'replica under test': us_east_1b-0170354798
I0102 08:27:27.684415   39655 max_replication_lag_module.go:384] rate was: not changed from: 3215 to: 3215
alias: us_east_1d-0170385729 lag: 0s
last change: 0.1s rate: 0 good/bad?  skipped b/c:  good/bad: 485/0
state (old/tested/new): I/I/I
lag before: n/a (n/a ago) rates (primary/replica): 0/0 backlog (old/new): 0/0
reason: skipping this replica because we're waiting for the next lag record from the 'replica under test': us_east_1b-0170354798
I0102 08:27:28.546290   39655 max_replication_lag_module.go:384] rate was: not changed from: 3215 to: 3215
alias: us_east_1b-0170354798 lag: 0s
last change: 1.0s rate: 0 good/bad?  skipped b/c:  good/bad: 485/0
state (old/tested/new): I/I/I
lag before: n/a (n/a ago) rates (primary/replica): 0/0 backlog (old/new): 0/0
reason: waiting 4.0 more seconds to see if the lag has changed
I0102 08:27:28.547133   39655 max_replication_lag_module.go:384] rate was: not changed from: 3215 to: 3215
alias: us_east_1d-0170385965 lag: 0s
last change: 1.0s rate: 0 good/bad?  skipped b/c:  good/bad: 485/0
state (old/tested/new): I/I/I
lag before: n/a (n/a ago) rates (primary/replica): 0/0 backlog (old/new): 0/0
reason: skipping this replica because we're waiting for the next lag record from the 'replica under test': us_east_1b-0170354798
...

A log record is generated every time that the max rate is recalculated - which happens quite often, hence the log is very noisy.

Request: do not pollute the log with MaxReplicationLagModule.recalculateRate() info.

Use Case(s)

Looking for anything in the log when the transaction throttler is enabled is difficult due to the amount of noise generated by this logging.

@ejortegau ejortegau added Needs Triage This issue needs to be correctly labelled and triaged Type: Feature labels Jan 2, 2024
@GuptaManan100 GuptaManan100 added Component: Cluster management and removed Needs Triage This issue needs to be correctly labelled and triaged labels Jan 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants