Add out-of-order sample support to the TSDB

[jesusvazquez]

Closes #8535

This implementation is based on this design doc:
https://docs.google.com/document/d/1Kppm7qL9C-BJB1j6yb6-9ObG3AbdZnFUBYPNNWwDBYM/edit?usp=sharing

This commit adds support to accept out-of-order ("OOO") sample into the TSDB
up to a configurable time allowance. If OOO is enabled, overlapping queries
are automatically enabled.

[jesusvazquez]

Having some trouble making windows tests pass, will focus on this soon. Need to address other work first.

[jesusvazquez]

This is now ready for review 🎉 cc @codesome @bwplotka (I'm quoting you since you are both the tsdb maintainers)

[wgliang]

Can anyone answer what is the target version of this new feature?

[codesome]

It is in 2.39.x

[juliusv]

@jesusvazquez one question: in

prometheus/scrape/scrape.go

Lines 1453 to 1457 in 92d6980

	// Call sl.append again with an empty scrape to trigger stale markers.
	// If the target has since been recreated and scraped, the
	// stale markers will be out of order and ignored.
	// sl.context would have been cancelled, hence using sl.appenderCtx.
	app := sl.appender(sl.appenderCtx)

we rely on the TSDB rejecting out-of-order samples (to not mark series from targets as stale when the target disappears briefly and then reappears a short time later). Does that behavior break when turning on out-of-order appends?

[roidelapluie]

@jesusvazquez one question: in

prometheus/scrape/scrape.go

Lines 1453 to 1457 in 92d6980

	// Call sl.append again with an empty scrape to trigger stale markers.
	// If the target has since been recreated and scraped, the
	// stale markers will be out of order and ignored.
	// sl.context would have been cancelled, hence using sl.appenderCtx.
	app := sl.appender(sl.appenderCtx)

we rely on the TSDB rejecting out-of-order samples (to not mark series from targets as stale when the target disappears briefly and then reappears a short time later). Does that behavior break when turning on out-of-order appends?

Yes #11730

[kushWithoutWax]

Hi . A couple of questions on this -

1. The document mentions a couple of problems - ingesting OOO samples and ingesting old samples.
   Is ingesting old samples also implemented as part of this PR ?
2. Is there a flag to enable this feature ?

[jesusvazquez]

The document mentions a couple of problems - ingesting OOO samples and ingesting old samples.
Is ingesting old samples also implemented as part of this PR ?

Yes its implemented, a sample is too old if its older than the window you configure. Else its ingested and its just out of order.

Is there a flag to enable this feature ?

You can enable it doing this in your config file:

storage:
  tsdb:
    out_of_order_time_window: 30m

[kushWithoutWax]

Is there a max value for out_of_order_time_window ?

[jesusvazquez]

No, there is no limit. I have only experience running this in production with a max window of 1 month. More should be feasible. No big impact on resources expected.

[kushWithoutWax]

Thank you for the info!

[syedishaq13129]

how does it work? we are also facing Out of order issue while scrapping eks metrics.. and after enabling the out of order we are getting this error..

ts=2024-03-14T09:32:59.364Z caller=dedupe.go:112 component=remote level=error remote_name=8843e5 url=http://10.0.36.190:9090/api/v1/write msg="non-recoverable error" count=2000 exemplarCount=0 err="server returned HTTP status 400 Bad Request: too old sample"

No, there is no limit. I have only experience running this in production with a max window of 1 month. More should be feasible. No big impact on resources expected.

How does the out of order paramater works , because we are also facing the same issue..

[jesusvazquez]

I'm helping @syedishaq13129 on the CNCF slack.

We think he might need to bump his out of order window to solve the error he is getting.