Store: Make initial sync more robust

Added re-try mechanism for store inital sync, where if the initial sync fails, it tries to do the initial sync again every 5 seconds for 15 seconds duration (total 3 re-tries for initial sync of store).

Signed-off-by: Kartik-Garg <[email protected]>

Store: Make initial sync more robust

Added re-try mechanism for store inital sync, where if the initial sync fails, it tries to do the initial sync again every 5 seconds for 15 seconds duration (total 3 re-tries for initial sync of store).

Signed-off-by: Kartik-Garg <[email protected]>

Merge release 0.30 into main (thanos-io#6041)

* compact: remove cancel on SyncMetas errors (thanos-io#5923)

in a favour of 86b4039 SyncMetas will retry if it's retriable.
Also, the cleanPartialMarked calls are surrounded by runutil.Repeat() will be repeated,
the ones not and are not retriable will throw an interrupt to run.Group() by returning err
and Group will call cancel() as it's configured for its interrupt func.

Signed-off-by: Seena Fallah <[email protected]>

Signed-off-by: Seena Fallah <[email protected]>

* Cut v0.30.0-rc.0 (thanos-io#5992)

* Cut v0.30.0-rc.0

Signed-off-by: bwplotka <[email protected]>

* mdox fix.

Signed-off-by: bwplotka <[email protected]>

Signed-off-by: bwplotka <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>

* Cut 0.30.0 (thanos-io#6011)

Signed-off-by: bwplotka <[email protected]>

Signed-off-by: bwplotka <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>

* *: cut 0.30.1 (thanos-io#6017)

* fix duplicate metrics registration in redis client (thanos-io#6009)

* fix duplicate metrics registration in redis client

Signed-off-by: Kama Huang <[email protected]>

* fixed test

Signed-off-by: Kama Huang <[email protected]>

Signed-off-by: Kama Huang <[email protected]>

* *: cut 0.30.1

Add CHANGELOG entry.

Signed-off-by: Giedrius Statkevičius <[email protected]>

Signed-off-by: Kama Huang <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>
Co-authored-by: Kama Huang <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>

* Tracing: Fix sampler defaults (thanos-io#5887)

* Fix sampler defaults

Signed-off-by: Matej Gera <[email protected]>

* Add CHANGELOG

Signed-off-by: Matej Gera <[email protected]>

* Replace checkout with git-shallow-clone (thanos-io#5829)

Signed-off-by: Matej Gera <[email protected]>

Signed-off-by: Matej Gera <[email protected]>

Signed-off-by: Matej Gera <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>

* CHANGELOG: fix

Signed-off-by: Giedrius Statkevičius <[email protected]>

Signed-off-by: Seena Fallah <[email protected]>
Signed-off-by: bwplotka <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: Kama Huang <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Co-authored-by: Seena Fallah <[email protected]>
Co-authored-by: Kama Huang <[email protected]>

cmd/thanos/store.go

@@ -49,10 +49,10 @@ import (
 	"github.com/thanos-io/thanos/pkg/ui"
 )
 
-// const (
-// 	timeoutDuration  = 15
-// 	intervalDuration = 5
-// )
+const (
+	timeoutDuration  = 15
+	intervalDuration = 5
+)
 
 type storeConfig struct {
 	indexCacheConfigs           extflag.PathOrContent
@@ -386,17 +386,7 @@ func runStore(
 
 			level.Info(logger).Log("msg", "initializing bucket store")
 			begin := time.Now()
-
-			//This will stop retrying after 15 seconds.
-			initialSyncCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
-			defer cancel()
-
-			//If error occurs, it will re-try after every 5 seconds, but only for 15 seconds, (so total re-try is three).
-			err := runutil.Retry(5*time.Second, initialSyncCtx.Done(), func() error {
-				return bs.InitialSync(ctx)
-			})
-
-			if err != nil {
+			if err := bs.InitialSync(ctx); err != nil {
 				close(bucketStoreReady)
 				return errors.Wrap(err, "bucket store initial sync")
 			}