Increase sync-block-duration default

[jpds]

I think the store API's --sync-block-duration should be increased from its current 3m to 1h.

New blocks are normally set to be shipped every 2h by the sidecars, and the more recent metrics are handled by the queriers.

I self host my own S3 storage layer and this is what 3m looks like on the backend:

This is 15m:

And this two store APIs set to 45m (earlier today):

[fpetkovski]

I agree that 3m is too aggressive and can cause excessive load and costs on object storage. However, a default of 1h might be too high since some users can have a max head block duration that's also 1h. Maybe something like 15m could be a good middle ground, especially because this is a default and changing it affect everyone who does not override it (which is probably most users).

[matej-g]

15 minutes sounds like a reasonable value to me 👍

[jpds]

I've spent the past week experimenting with different sync values on my store APIs, and I think the ideal value for the default would be 1h3m.

The extra 3m is arbitrary, however the repeater at https://github.com/thanos-io/thanos/blob/v0.31.0/pkg/runutil/runutil.go#L69 has no way of taking into account the duration of the previous run. A process, started at :58 past the hour, will always run sync tasks at :58 past, which could be non-ideal when new blocks are uploaded at :00 by the shipper. Hence the extra value to nudge this forward as syncs happen.

I also found 15m to be too frequent, especially when more store APIs are spun up.

[yeya24]

Isn't it a configurable flag? So you can change it based on your own usecase. 15m here seems a sane default that works for most of the usecases.