Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clock offset is not clear enough #8806

Open
rleungx opened this issue Nov 13, 2024 · 2 comments · May be fixed by #8826
Open

clock offset is not clear enough #8806

rleungx opened this issue Nov 13, 2024 · 2 comments · May be fixed by #8826
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@rleungx
Copy link
Member

rleungx commented Nov 13, 2024

Enhancement Task

func (t *timestampOracle) UpdateTimestamp() error {
	prevPhysical, prevLogical := t.getTSO()

	now := time.Now()
	t.metrics.saveEvent.Inc()

	jetLag := typeutil.SubRealTimeByWallClock(now, prevPhysical)
	if jetLag > 3*t.updatePhysicalInterval && jetLag > jetLagWarningThreshold {
		log.Warn("clock offset",
			logutil.CondUint32("keyspace-group-id", t.keyspaceGroupID, t.keyspaceGroupID > 0),
			zap.Duration("jet-lag", jetLag),
			zap.Time("prev-physical", prevPhysical),
			zap.Time("now", now),
			zap.Duration("update-physical-interval", t.updatePhysicalInterval))
		t.metrics.slowSaveEvent.Inc()
	}
        ...

From the above code, if the current system time is much later than the previous physical time or runtime issue, the log will be printed. But clock offset could be one of the reasons. So here, we'd better use a clearer log message that is less confusing.

@rleungx rleungx added the type/enhancement The issue or PR belongs to an enhancement. label Nov 13, 2024
@okJiang
Copy link
Member

okJiang commented Nov 20, 2024

clock offset could be one of the reasons

What other reasons need to be explained? PD restart? Leader transfer?

Or is it sufficient to just mention that there hasn't been a physical time update for a while, which may have caused a clock offset?

@JmPotato @rleungx

@JmPotato
Copy link
Member

clock offset could be one of the reasons

What other reasons need to be explained? PD restart? Leader transfer?

Or is it sufficient to just mention that there hasn't been a physical time update for a while, which may have caused a clock drift?

@JmPotato @rleungx

If the etcd suffers from a slow IO performance, the TSO updating may fail to advance the physical part, which will also cause the "clock offset".

@okJiang okJiang linked a pull request Nov 20, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants