receive: Expose write chunk queue as flag

[philipgough]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

This change proposes to expose, via the hidden --tsdb.write-queue-size flag a chunk write queue for receive
which was upstreamed to TSDB via prometheus/prometheus#10051.

For clarification, at the time this feature was contributed, the queue was enabled by default but this was subsequently
made opt-in via prometheus/prometheus#10425 for the reasons discussed/reported in prometheus/prometheus#10377

There were some follow on improvements in:

• Reduce chunk write queue memory usage 1 prometheus/prometheus#10873
• Reduce chunk write queue memory usage 2 prometheus/prometheus#10874

Verification

I ran a three load test against Thanos Receive pushing two million active series oover a period of 4hrs ramping up and reaching peak within 1hr:

1. v0.27.0 release with 6 receivers
2. v0.27.0 release with 12 receivers
3. This branch with the queue enabled at 20m

They are displayed on the following graphs from left to right.

Shows a significant reduction in error rate on the last run:

Shows we generally see a correlation between higher error rates and increase in the head chunks metrics.
This appears to be correlated with the duration (latency spikes) in receive and replication.

The result of enabling the queue appears to be great reduction in p90 latency and slight improvement in the p99 taking most request within the default 5s timeout for replication.

Other things to note was that there was no measurable regression in our query path SLOs due to enabling this feature.
Memory remained stable throughout and we even avoided the spike we saw when running with 6 replicas on the latest release.

[fpetkovski]

This makes sense to me 👍

[fpetkovski]

Can we just add some hints in the docs (or flag description) on what a good value might be for the queue size?

[matej-g]

This is looking good to me! Even though it's experimental, as @fpetkovski said maybe adding a word or two on what it does and what are some recommended values to try would be nice.

[philipgough]

@fpetkovski, @matej-g:

I haven't documented this flag purposely and there are a few reasons for that, which is why it is hidden and should remain so for now IMO.

As you mentioned, it's an experimental feature flag here, and it is also experimental upstream. I'm still trying to run tests to tweak the value and I'd be happy to add proper and more through documentation when I discover the sweet spot and have a better understanding of the tradeoffs it brings (if any).

As of now, I've had good results so far with making the queue size equal to the number of active series that are being pushed, but I would not yet like to document that as optimal and as you can see in the description, the existing implementation is actively being iterated on so any provided values might well change.

Does that reasoning make sense for exclusion of additional docs right now?

[yeya24]

I like this change.
For the documentation part, I think it is fine for this pr as the flag is marked as hidden.

[fpetkovski]

This sounds good to me 👍