Reported validation error: start time after end time

[jmacd]

We have an unconfirmed report that data points send from the sidecar have start time after end time. As printed in the sidecar log sending to Lightstep:

level=warn ts=2021-01-28T21:02:58.422Z caller=queue_manager.go:506 component=queue_manager msg="unrecoverable write error" err="rpc error: code = InvalidArgument desc = [metric name]: must be positive for kind=delta|cumulative]"

This should be impossible, we should check this in the sidecar to be absolutely sure.

[jmacd]

I have evidence that this is caused by a race condition. First, the code responsible for determining resets does not actually check for a negative duration:

// getResetAdjusted takes a sample for a referenced series and returns
// its reset timestamp and adjusted value.
// If the last return argument is false, the sample should be dropped.
func (c *seriesCache) getResetAdjusted(ref walRef, t promTime, v float64) (promTime, float64, bool) {
	c.mtx.Lock()
	e, ok := c.entries[ref]
	c.mtx.Unlock()
	if !ok {
		return 0, 0, false
	}
	hasReset := e.hasReset
	e.hasReset = true
	if !hasReset {
		e.resetTimestamp = t
		e.resetValue = v
		// If we just initialized the reset timestamp, this sample should be skipped.
		// We don't know the window over which the current cumulative value was built up over.
		// The next sample for will be considered from this point onwards.
		return 0, 0, false
	}
	if v < e.resetValue {
		// If the series was reset, set the reset timestamp to be one millisecond
		// before the timestamp of the current sample.
		// We don't know the true reset time but this ensures the range is non-zero
		// while unlikely to conflict with any previous sample.
		e.resetValue = 0
		e.resetTimestamp = t - 1
	}
	return e.resetTimestamp, v - e.resetValue, true
}

(This is similar to the upstream Stackdriver code base.)

The cause of the incident simply requires that samples from a timeseries appear out-of-order to the seriesCache?

[jmacd]

On further investigation, this report is inconsistent. The log message quoted above does not mention time. #145 may fix a related problem, closing this until we see it again.