Skip to content

Commit

Permalink
kv: disable timestamp cache + current clock assertion
Browse files Browse the repository at this point in the history
Closes #60580.
Closes #60736.
Closes #60779.
Closes #61060.

This was added in 218a5a3. The check was more of a sanity check that we have and
always have had an understand of the timestamps that can enter the timestamp
cache. The fact that it's failing is a clear indication that there were issues
in past releases, because a lease transfer used to only be safe if the outgoing
leaseholder's clock was above the time of any read in its timestamp cache. We
now ship a snapshot of the timestamp cache on lease transfers, so that invariant
is less important.

I'd still like to get to the bottom of this, but I'll do so on my own branch,
off of master where it's causing disruption.

Release justification: avoid assertion failures
  • Loading branch information
nvanbenschoten committed Feb 25, 2021
1 parent 0f0c6d8 commit 54f29fe
Showing 1 changed file with 8 additions and 1 deletion.
9 changes: 8 additions & 1 deletion pkg/kv/kvserver/replica_tscache.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,14 @@ func (r *Replica) addToTSCacheChecked(
// directly instead of using the local HLC clock as a proxy for its high
// water-mark, but it serves as a good proxy for proper handling of HLC
// clock updates and, by extension, observed timestamps.
if !ts.Synthetic && st.Now.ToTimestamp().Less(ts) {
//
// TODO(nvanbenschoten): this is currently disabled because we seem to
// regularly hit it on master. Now that we ship a snapshot of the timestamp
// cache on lease transfers instead of just the current clock time, the
// property this is asserting is no longer quite as important, so we can
// disable the check. However, it would still be nice to track down how we
// can hit this.
if !ts.Synthetic && st.Now.ToTimestamp().Less(ts) && false {
log.Fatalf(ctx, "Unsafe timestamp cache update! Cannot add timestamp %s to timestamp "+
"cache after evaluating %v (resp=%v; err=%v) with local hlc clock at timestamp %s. "+
"Non-synthetic timestamps should always lag the local hlc clock.", ts, ba, br, pErr, st.Now)
Expand Down

0 comments on commit 54f29fe

Please sign in to comment.