Changed -querier.query-ingesters-within and -blocks-storage.tsdb.close-idle-tsdb-timeout defaults from 0 to 13h

CHANGELOG.md

@@ -248,6 +248,8 @@
 * [CHANGE] Ingester: change default value of `-ingester.final-sleep` from `30s` to `0s`. #981
 * [CHANGE] Ruler: `-experimental.ruler.enable-api` flag has been renamed to `-ruler.enable-api` and is now stable. #913
 * [CHANGE] Alertmanager: `-experimental.alertmanager.enable-api` flag has been renamed to `-alertmanager.enable-api` and is now stable. #913
+* [CHANGE] Ingester: changed default value of `-blocks-storage.tsdb.retention-period` from `6h` to `24h`. #966
+* [CHANGE] Changed default value of `-querier.query-ingesters-within` and `-blocks-storage.tsdb.close-idle-tsdb-timeout` from `0` to `13h`. #966
 * [FEATURE] Query Frontend: Add `cortex_query_fetched_chunks_total` per-user counter to expose the number of chunks fetched as part of queries. This metric can be enabled with the `-frontend.query-stats-enabled` flag (or its respective YAML config option `query_stats_enabled`). #31
 * [FEATURE] Query Frontend: Add experimental querysharding for the blocks storage (instant and range queries). You can now enable querysharding for blocks storage (`-store.engine=blocks`) by setting `-frontend.parallelize-shardable-queries` to `true`. The following additional config and exported metrics have been added. #79 #80 #100 #124 #140 #148 #150 #151 #153 #154 #155 #156 #157 #158 #159 #160 #163 #169 #172 #196 #205 #225 #226 #227 #228 #230 #235 #240 #239 #246 #244 #319 #330 #371 #385 #400 #458 #586 #630 #660 #707
   * New config options:

cmd/mimir/help-all.txt.tmpl

@@ -460,7 +460,7 @@ Usage of ./cmd/mimir/mimir:
   -blocks-storage.tsdb.block-ranges-period value
     	TSDB blocks range period. (default 2h0m0s)
   -blocks-storage.tsdb.close-idle-tsdb-timeout duration
-    	If TSDB has not received any data for this duration, and all blocks from TSDB have been shipped, TSDB is closed and deleted from local disk. If set to positive value, this value should be equal or higher than -querier.query-ingesters-within flag to make sure that TSDB is not closed prematurely, which could cause partial query results. 0 or negative value disables closing of idle TSDB.
+    	If TSDB has not received any data for this duration, and all blocks from TSDB have been shipped, TSDB is closed and deleted from local disk. If set to positive value, this value should be equal or higher than -querier.query-ingesters-within flag to make sure that TSDB is not closed prematurely, which could cause partial query results. 0 or negative value disables closing of idle TSDB. (default 13h0m0s)
   -blocks-storage.tsdb.dir string
     	Local directory to store TSDBs in the ingesters. (default "tsdb")
   -blocks-storage.tsdb.flush-blocks-on-shutdown
@@ -484,7 +484,7 @@ Usage of ./cmd/mimir/mimir:
   -blocks-storage.tsdb.memory-snapshot-on-shutdown
     	True to enable snapshotting of in-memory TSDB data on disk when shutting down.
   -blocks-storage.tsdb.retention-period duration
-    	TSDB blocks retention in the ingester before a block is removed. This should be larger than the block_ranges_period and large enough to give store-gateways and queriers enough time to discover newly uploaded blocks. (default 6h0m0s)
+    	TSDB blocks retention in the ingester before a block is removed. This should be larger than the -blocks-storage.tsdb.block-ranges-period, -querier.query-store-after and large enough to give store-gateways and queriers enough time to discover newly uploaded blocks. (default 24h0m0s)
   -blocks-storage.tsdb.series-hash-cache-max-size-bytes uint
     	Max size - in bytes - of the in-memory series hash cache. The cache is shared across all tenants and it's used only when query sharding is enabled. (default 1073741824)
   -blocks-storage.tsdb.ship-concurrency int
@@ -1156,7 +1156,7 @@ Usage of ./cmd/mimir/mimir:
   -querier.max-samples int
     	Maximum number of samples a single query can load into memory. This config option should be set on query-frontend too when query sharding is enabled. (default 50000000)
   -querier.query-ingesters-within duration
-    	Maximum lookback beyond which queries are not sent to ingester. 0 means all queries are sent to ingester.
+    	Maximum lookback beyond which queries are not sent to ingester. 0 means all queries are sent to ingester. (default 13h0m0s)
   -querier.query-label-names-with-matchers-enabled
     	True to enable queriers to use an optimized implementation which passes down to ingesters the label matchers when running the label names API. Can be enabled once all ingesters run a version >= the one where this option has been introduced.
   -querier.query-store-after duration
@@ -1166,7 +1166,7 @@ Usage of ./cmd/mimir/mimir:
   -querier.scheduler-address string
     	Hostname (and port) of scheduler that querier will periodically resolve, connect to and receive queries from. Only one of -querier.frontend-address or -querier.scheduler-address can be set. If neither is set, queries are only received via HTTP endpoint.
   -querier.shuffle-sharding-ingesters-lookback-period duration
-    	When distributor's sharding strategy is shuffle-sharding and this setting is > 0, queriers fetch in-memory series from the minimum set of required ingesters, selecting only ingesters which may have received series since 'now - lookback period'. The lookback period should be greater or equal than the configured 'query store after' and 'query ingesters within'. If this setting is 0, queriers always query all ingesters (ingesters shuffle sharding on read path is disabled).
+    	When distributor's sharding strategy is shuffle-sharding and this setting is > 0, queriers fetch in-memory series from the minimum set of required ingesters, selecting only ingesters which may have received series since 'now - lookback period'. The lookback period should be greater or equal than the configured -querier.query-store-after and -querier.query-ingesters-within. If this setting is 0, queriers always query all ingesters (ingesters shuffle sharding on read path is disabled).
   -querier.store-gateway-client.tls-ca-path string
     	Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used.
   -querier.store-gateway-client.tls-cert-path string

cmd/mimir/help.txt.tmpl

@@ -396,7 +396,7 @@ Usage of ./cmd/mimir/mimir:
   -blocks-storage.tsdb.block-ranges-period value
     	TSDB blocks range period. (default 2h0m0s)
   -blocks-storage.tsdb.close-idle-tsdb-timeout duration
-    	If TSDB has not received any data for this duration, and all blocks from TSDB have been shipped, TSDB is closed and deleted from local disk. If set to positive value, this value should be equal or higher than -querier.query-ingesters-within flag to make sure that TSDB is not closed prematurely, which could cause partial query results. 0 or negative value disables closing of idle TSDB.
+    	If TSDB has not received any data for this duration, and all blocks from TSDB have been shipped, TSDB is closed and deleted from local disk. If set to positive value, this value should be equal or higher than -querier.query-ingesters-within flag to make sure that TSDB is not closed prematurely, which could cause partial query results. 0 or negative value disables closing of idle TSDB. (default 13h0m0s)
   -blocks-storage.tsdb.dir string
     	Local directory to store TSDBs in the ingesters. (default "tsdb")
   -blocks-storage.tsdb.flush-blocks-on-shutdown
@@ -418,7 +418,7 @@ Usage of ./cmd/mimir/mimir:
   -blocks-storage.tsdb.memory-snapshot-on-shutdown
     	True to enable snapshotting of in-memory TSDB data on disk when shutting down.
   -blocks-storage.tsdb.retention-period duration
-    	TSDB blocks retention in the ingester before a block is removed. This should be larger than the block_ranges_period and large enough to give store-gateways and queriers enough time to discover newly uploaded blocks. (default 6h0m0s)
+    	TSDB blocks retention in the ingester before a block is removed. This should be larger than the -blocks-storage.tsdb.block-ranges-period, -querier.query-store-after and large enough to give store-gateways and queriers enough time to discover newly uploaded blocks. (default 24h0m0s)
   -blocks-storage.tsdb.series-hash-cache-max-size-bytes uint
     	Max size - in bytes - of the in-memory series hash cache. The cache is shared across all tenants and it's used only when query sharding is enabled. (default 1073741824)
   -blocks-storage.tsdb.ship-concurrency int
@@ -800,7 +800,7 @@ Usage of ./cmd/mimir/mimir:
   -querier.max-samples int
     	Maximum number of samples a single query can load into memory. This config option should be set on query-frontend too when query sharding is enabled. (default 50000000)
   -querier.query-ingesters-within duration
-    	Maximum lookback beyond which queries are not sent to ingester. 0 means all queries are sent to ingester.
+    	Maximum lookback beyond which queries are not sent to ingester. 0 means all queries are sent to ingester. (default 13h0m0s)
   -querier.query-label-names-with-matchers-enabled
     	True to enable queriers to use an optimized implementation which passes down to ingesters the label matchers when running the label names API. Can be enabled once all ingesters run a version >= the one where this option has been introduced.
   -querier.query-store-after duration
@@ -810,7 +810,7 @@ Usage of ./cmd/mimir/mimir:
   -querier.scheduler-address string
     	Hostname (and port) of scheduler that querier will periodically resolve, connect to and receive queries from. Only one of -querier.frontend-address or -querier.scheduler-address can be set. If neither is set, queries are only received via HTTP endpoint.
   -querier.shuffle-sharding-ingesters-lookback-period duration
-    	When distributor's sharding strategy is shuffle-sharding and this setting is > 0, queriers fetch in-memory series from the minimum set of required ingesters, selecting only ingesters which may have received series since 'now - lookback period'. The lookback period should be greater or equal than the configured 'query store after' and 'query ingesters within'. If this setting is 0, queriers always query all ingesters (ingesters shuffle sharding on read path is disabled).
+    	When distributor's sharding strategy is shuffle-sharding and this setting is > 0, queriers fetch in-memory series from the minimum set of required ingesters, selecting only ingesters which may have received series since 'now - lookback period'. The lookback period should be greater or equal than the configured -querier.query-store-after and -querier.query-ingesters-within. If this setting is 0, queriers always query all ingesters (ingesters shuffle sharding on read path is disabled).
   -querier.timeout duration
     	The timeout for a query. This config option should be set on query-frontend too when query sharding is enabled. (default 2m0s)
   -query-scheduler.grpc-client-config.backoff-max-period duration

docs/sources/blocks-storage/querier.md

@@ -109,7 +109,7 @@ querier:
   # Maximum lookback beyond which queries are not sent to ingester. 0 means all
   # queries are sent to ingester.
   # CLI flag: -querier.query-ingesters-within
-  [query_ingesters_within: <duration> | default = 0s]
+  [query_ingesters_within: <duration> | default = 13h]
 
   # True to enable queriers to use an optimized implementation which passes down
   # to ingesters the label matchers when running the label names API. Can be
@@ -162,9 +162,9 @@ querier:
   # > 0, queriers fetch in-memory series from the minimum set of required
   # ingesters, selecting only ingesters which may have received series since
   # 'now - lookback period'. The lookback period should be greater or equal than
-  # the configured 'query store after' and 'query ingesters within'. If this
-  # setting is 0, queriers always query all ingesters (ingesters shuffle
-  # sharding on read path is disabled).
+  # the configured -querier.query-store-after and
+  # -querier.query-ingesters-within. If this setting is 0, queriers always query
+  # all ingesters (ingesters shuffle sharding on read path is disabled).
   # CLI flag: -querier.shuffle-sharding-ingesters-lookback-period
   [shuffle_sharding_ingesters_lookback_period: <duration> | default = 0s]
 
@@ -653,10 +653,11 @@ blocks_storage:
     [block_ranges_period: <list of duration> | default = 2h0m0s]
 
     # TSDB blocks retention in the ingester before a block is removed. This
-    # should be larger than the block_ranges_period and large enough to give
-    # store-gateways and queriers enough time to discover newly uploaded blocks.
+    # should be larger than the -blocks-storage.tsdb.block-ranges-period,
+    # -querier.query-store-after and large enough to give store-gateways and
+    # queriers enough time to discover newly uploaded blocks.
     # CLI flag: -blocks-storage.tsdb.retention-period
-    [retention_period: <duration> | default = 6h]
+    [retention_period: <duration> | default = 24h]
 
     # How frequently the TSDB blocks are scanned and new ones are shipped to the
     # storage. 0 means shipping is disabled.
@@ -722,7 +723,7 @@ blocks_storage:
     # prematurely, which could cause partial query results. 0 or negative value
     # disables closing of idle TSDB.
     # CLI flag: -blocks-storage.tsdb.close-idle-tsdb-timeout
-    [close_idle_tsdb_timeout: <duration> | default = 0s]
+    [close_idle_tsdb_timeout: <duration> | default = 13h]
 
     # True to enable snapshotting of in-memory TSDB data on disk when shutting
     # down.

docs/sources/blocks-storage/store-gateway.md

@@ -729,10 +729,11 @@ blocks_storage:
     [block_ranges_period: <list of duration> | default = 2h0m0s]
 
     # TSDB blocks retention in the ingester before a block is removed. This
-    # should be larger than the block_ranges_period and large enough to give
-    # store-gateways and queriers enough time to discover newly uploaded blocks.
+    # should be larger than the -blocks-storage.tsdb.block-ranges-period,
+    # -querier.query-store-after and large enough to give store-gateways and
+    # queriers enough time to discover newly uploaded blocks.
     # CLI flag: -blocks-storage.tsdb.retention-period
-    [retention_period: <duration> | default = 6h]
+    [retention_period: <duration> | default = 24h]
 
     # How frequently the TSDB blocks are scanned and new ones are shipped to the
     # storage. 0 means shipping is disabled.
@@ -798,7 +799,7 @@ blocks_storage:
     # prematurely, which could cause partial query results. 0 or negative value
     # disables closing of idle TSDB.
     # CLI flag: -blocks-storage.tsdb.close-idle-tsdb-timeout
-    [close_idle_tsdb_timeout: <duration> | default = 0s]
+    [close_idle_tsdb_timeout: <duration> | default = 13h]
 
     # True to enable snapshotting of in-memory TSDB data on disk when shutting
     # down.

docs/sources/configuration/config-file-reference.md

@@ -793,7 +793,7 @@ The `querier_config` configures the querier.
 # Maximum lookback beyond which queries are not sent to ingester. 0 means all
 # queries are sent to ingester.
 # CLI flag: -querier.query-ingesters-within
-[query_ingesters_within: <duration> | default = 0s]
+[query_ingesters_within: <duration> | default = 13h]
 
 # True to enable queriers to use an optimized implementation which passes down
 # to ingesters the label matchers when running the label names API. Can be
@@ -845,9 +845,9 @@ store_gateway_client:
 # 0, queriers fetch in-memory series from the minimum set of required ingesters,
 # selecting only ingesters which may have received series since 'now - lookback
 # period'. The lookback period should be greater or equal than the configured
-# 'query store after' and 'query ingesters within'. If this setting is 0,
-# queriers always query all ingesters (ingesters shuffle sharding on read path
-# is disabled).
+# -querier.query-store-after and -querier.query-ingesters-within. If this
+# setting is 0, queriers always query all ingesters (ingesters shuffle sharding
+# on read path is disabled).
 # CLI flag: -querier.shuffle-sharding-ingesters-lookback-period
 [shuffle_sharding_ingesters_lookback_period: <duration> | default = 0s]
 
@@ -3117,10 +3117,11 @@ tsdb:
   [block_ranges_period: <list of duration> | default = 2h0m0s]
 
   # TSDB blocks retention in the ingester before a block is removed. This should
-  # be larger than the block_ranges_period and large enough to give
-  # store-gateways and queriers enough time to discover newly uploaded blocks.
+  # be larger than the -blocks-storage.tsdb.block-ranges-period,
+  # -querier.query-store-after and large enough to give store-gateways and
+  # queriers enough time to discover newly uploaded blocks.
   # CLI flag: -blocks-storage.tsdb.retention-period
-  [retention_period: <duration> | default = 6h]
+  [retention_period: <duration> | default = 24h]
 
   # How frequently the TSDB blocks are scanned and new ones are shipped to the
   # storage. 0 means shipping is disabled.
@@ -3184,7 +3185,7 @@ tsdb:
   # prematurely, which could cause partial query results. 0 or negative value
   # disables closing of idle TSDB.
   # CLI flag: -blocks-storage.tsdb.close-idle-tsdb-timeout
-  [close_idle_tsdb_timeout: <duration> | default = 0s]
+  [close_idle_tsdb_timeout: <duration> | default = 13h]
 
   # True to enable snapshotting of in-memory TSDB data on disk when shutting
   # down.

docs/sources/proposals/blocks-storage-bucket-index.md

@@ -93,7 +93,7 @@ Queriers and store-gateways don't need a strongly consistent view over the bucke
 The maximum delay between a change into the bucket is picked up by queriers and store-gateways depends on the configuration and is the minimum time between:
 
 - New blocks uploaded by ingester:
-  `min(-querier.query-ingesters-within, -blocks-storage.tsdb.retention-period)` (default: 6h)
+  `min(-querier.query-ingesters-within, -blocks-storage.tsdb.retention-period)` (default: 13h)
 - New blocks uploaded/deleted by compactor:
   `-compactor.deletion-delay` (default: 12h)
 

operations/mimir-tests/test-defaults-generated.yaml

@@ -1006,7 +1006,6 @@ spec:
         - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.isolation-enabled=false
-        - -blocks-storage.tsdb.retention-period=24h
         - -blocks-storage.tsdb.ship-interval=1m
         - -consul.hostname=consul.default.svc.cluster.local:8500
         - -distributor.health-check-ingesters=true

operations/mimir-tests/test-disable-chunk-streaming-generated.yaml

@@ -1331,7 +1331,6 @@ spec:
         - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.isolation-enabled=false
-        - -blocks-storage.tsdb.retention-period=24h
         - -blocks-storage.tsdb.ship-interval=1m
         - -consul.hostname=consul.default.svc.cluster.local:8500
         - -distributor.health-check-ingesters=true

operations/mimir-tests/test-query-sharding-generated.yaml

@@ -1334,7 +1334,6 @@ spec:
         - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.isolation-enabled=false
-        - -blocks-storage.tsdb.retention-period=24h
         - -blocks-storage.tsdb.ship-interval=1m
         - -consul.hostname=consul.default.svc.cluster.local:8500
         - -distributor.health-check-ingesters=true

operations/mimir-tests/test-shuffle-sharding-generated.yaml

@@ -1340,7 +1340,6 @@ spec:
         - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.isolation-enabled=false
-        - -blocks-storage.tsdb.retention-period=24h
         - -blocks-storage.tsdb.ship-interval=1m
         - -consul.hostname=consul.default.svc.cluster.local:8500
         - -distributor.health-check-ingesters=true

operations/mimir-tests/test-storage-azure-generated.yaml

@@ -1342,7 +1342,6 @@ spec:
         - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.isolation-enabled=false
-        - -blocks-storage.tsdb.retention-period=24h
         - -blocks-storage.tsdb.ship-interval=1m
         - -consul.hostname=consul.default.svc.cluster.local:8500
         - -distributor.health-check-ingesters=true

operations/mimir-tests/test-storage-gcs-generated.yaml

@@ -1330,7 +1330,6 @@ spec:
         - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.isolation-enabled=false
-        - -blocks-storage.tsdb.retention-period=24h
         - -blocks-storage.tsdb.ship-interval=1m
         - -consul.hostname=consul.default.svc.cluster.local:8500
         - -distributor.health-check-ingesters=true