From 1427f53e9ee9fb2a7e972483bb5f54fd56ea19dd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Peter=20S=CC=8Ctibrany=CC=81?= Date: Fri, 15 May 2020 16:58:40 +0200 Subject: [PATCH 1/4] Documentation and CHANGELOG.md entry for #2579. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Peter Štibraný --- CHANGELOG.md | 1 + docs/components/store.md | 26 +++++++++++++++++++------- 2 files changed, 20 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 855c3518a0..8492b191fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,7 @@ We use *breaking* word for marking changes that are not backward compatible (rel - [#2521](https://github.com/thanos-io/thanos/pull/2521) Sidecar: add `thanos_sidecar_reloader_reloads_failed_total`, `thanos_sidecar_reloader_reloads_total`, `thanos_sidecar_reloader_watch_errors_total`, `thanos_sidecar_reloader_watch_events_total` and `thanos_sidecar_reloader_watches` metrics. - [#2412](https://github.com/thanos-io/thanos/pull/2412) ui: add React UI from Prometheus upstream. Currently only accessible from Query component as only `/graph` endpoint is migrated. - [#2532](https://github.com/thanos-io/thanos/pull/2532) Store: Added hidden option for experimental caching bucket, that can cache chunks into shared memcached. This can speed up querying and reduce number of requests to object storage. +- [#2579](https://github.com/thanos-io/thanos/pull/2579) Store: Experimental caching bucket can now cache metadata as well. Config has changed from #2532. ### Changed diff --git a/docs/components/store.md b/docs/components/store.md index 74a70f348d..180c47bca1 100644 --- a/docs/components/store.md +++ b/docs/components/store.md @@ -274,7 +274,7 @@ While the remaining settings are **optional**: ## Caching Bucket -Thanos Store Gateway supports a "caching bucket" with chunks caching to speed up loading of chunks from TSDB blocks. Currently only memcached "backend" is supported: +Thanos Store Gateway supports a "caching bucket" with chunks and metadata caching to speed up loading of chunks from TSDB blocks. Currently only memcached "backend" is supported: ```yaml backend: memcached @@ -282,22 +282,34 @@ backend_config: addresses: - localhost:11211 -caching_config: - chunk_subrange_size: 16000 - max_chunks_get_range_requests: 3 - chunk_object_size_ttl: 24h - chunk_subrange_ttl: 24h +chunk_subrange_size: 16000 +max_chunks_get_range_requests: 3 +chunk_object_size_ttl: 24h +chunk_subrange_ttl: 24h +blocks_iter_ttl: 5m +metafile_exists_ttl: 2h +metafile_doesnt_exist_ttl: 15m +metafile_content_ttl: 24h +metafile_max_size: 1MiB ``` `backend_config` field for memcached supports all the same configuration as memcached for [index cache](#memcached-index-cache). -`caching_config` is a configuration for chunks cache and supports the following optional settings: +`caching_config` is a configuration for cache and supports the following optional settings for chunks caching: - `chunk_subrange_size`: size of segment of chunks object that is stored to the cache. This is the smallest unit that chunks cache is working with. - `max_chunks_get_range_requests`: how many "get range" sub-requests may cache perform to fetch missing subranges. - `chunk_object_size_ttl`: how long to keep information about chunk file length in the cache. - `chunk_subrange_ttl`: how long to keep individual subranges in the cache. +Following options are used for metadata caching (meta.json files, deletion mark files, iteration result): + +- `blocks_iter_ttl`: how long to cache result of iterating blocks. +- `metafile_exists_ttl`: how long to cache information about whether meta.json or deletion mark file exists +- `metafile_doesnt_exist_ttl`: how long to cache information about whether meta.json or deletion mark file doesn't exist +- `metafile_content_ttl`: how long to cache content of meta.json and deletion mark files. +- `metafile_max_size`: maximum size of cached meta.json and deletion mark file. Larger files are not cached. + Note that chunks cache is an experimental feature, and these fields may be renamed or removed completely in the future. ## Index Header From 3535b69cfae54b6c011aafb01a027c54892800c9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Peter=20S=CC=8Ctibrany=CC=81?= Date: Fri, 15 May 2020 19:08:30 +0200 Subject: [PATCH 2/4] Document hidden option to configure caching bucket, to make it less hidden. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Peter Štibraný --- CHANGELOG.md | 2 +- docs/components/store.md | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8492b191fb..4b1c45ea1e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,7 @@ We use *breaking* word for marking changes that are not backward compatible (rel - [#2502](https://github.com/thanos-io/thanos/pull/2502) Added `hints` field to `SeriesResponse`. Hints in an opaque data structure that can be used to carry additional information from the store and its content is implementation specific. - [#2521](https://github.com/thanos-io/thanos/pull/2521) Sidecar: add `thanos_sidecar_reloader_reloads_failed_total`, `thanos_sidecar_reloader_reloads_total`, `thanos_sidecar_reloader_watch_errors_total`, `thanos_sidecar_reloader_watch_events_total` and `thanos_sidecar_reloader_watches` metrics. - [#2412](https://github.com/thanos-io/thanos/pull/2412) ui: add React UI from Prometheus upstream. Currently only accessible from Query component as only `/graph` endpoint is migrated. -- [#2532](https://github.com/thanos-io/thanos/pull/2532) Store: Added hidden option for experimental caching bucket, that can cache chunks into shared memcached. This can speed up querying and reduce number of requests to object storage. +- [#2532](https://github.com/thanos-io/thanos/pull/2532) Store: Added hidden option `--store.caching-bucket.config=` (or `--store.caching-bucket.config-file=`) for experimental caching bucket, that can cache chunks into shared memcached. This can speed up querying and reduce number of requests to object storage. - [#2579](https://github.com/thanos-io/thanos/pull/2579) Store: Experimental caching bucket can now cache metadata as well. Config has changed from #2532. ### Changed diff --git a/docs/components/store.md b/docs/components/store.md index 180c47bca1..fa3268060b 100644 --- a/docs/components/store.md +++ b/docs/components/store.md @@ -274,7 +274,9 @@ While the remaining settings are **optional**: ## Caching Bucket -Thanos Store Gateway supports a "caching bucket" with chunks and metadata caching to speed up loading of chunks from TSDB blocks. Currently only memcached "backend" is supported: +Thanos Store Gateway supports a "caching bucket" with chunks and metadata caching to speed up loading of chunks from TSDB blocks. To configure caching, one needs to use `--store.caching-bucket.config=` or `--store.caching-bucket.config-file=`. + +Currently only memcached "backend" is supported: ```yaml backend: memcached From 222a9ab9e5245e46fb10c02882ceef8ecc0abeb4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Peter=20S=CC=8Ctibrany=CC=81?= Date: Mon, 18 May 2020 07:28:15 +0200 Subject: [PATCH 3/4] Removed mention of caching_config. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Peter Štibraný --- docs/components/store.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/components/store.md b/docs/components/store.md index fa3268060b..0458b9f309 100644 --- a/docs/components/store.md +++ b/docs/components/store.md @@ -297,7 +297,7 @@ metafile_max_size: 1MiB `backend_config` field for memcached supports all the same configuration as memcached for [index cache](#memcached-index-cache). -`caching_config` is a configuration for cache and supports the following optional settings for chunks caching: +Additional options to configure various aspects of chunks cache are available: - `chunk_subrange_size`: size of segment of chunks object that is stored to the cache. This is the smallest unit that chunks cache is working with. - `max_chunks_get_range_requests`: how many "get range" sub-requests may cache perform to fetch missing subranges. @@ -312,7 +312,7 @@ Following options are used for metadata caching (meta.json files, deletion mark - `metafile_content_ttl`: how long to cache content of meta.json and deletion mark files. - `metafile_max_size`: maximum size of cached meta.json and deletion mark file. Larger files are not cached. -Note that chunks cache is an experimental feature, and these fields may be renamed or removed completely in the future. +Note that chunks and metadata cache is an experimental feature, and these fields may be renamed or removed completely in the future. ## Index Header From 78a9abe34cd58db26ad677760f7603062b696fa8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Peter=20S=CC=8Ctibrany=CC=81?= Date: Mon, 18 May 2020 08:06:48 +0200 Subject: [PATCH 4/4] Added periods. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Peter Štibraný --- docs/components/store.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/components/store.md b/docs/components/store.md index 0458b9f309..14066b531a 100644 --- a/docs/components/store.md +++ b/docs/components/store.md @@ -307,8 +307,8 @@ Additional options to configure various aspects of chunks cache are available: Following options are used for metadata caching (meta.json files, deletion mark files, iteration result): - `blocks_iter_ttl`: how long to cache result of iterating blocks. -- `metafile_exists_ttl`: how long to cache information about whether meta.json or deletion mark file exists -- `metafile_doesnt_exist_ttl`: how long to cache information about whether meta.json or deletion mark file doesn't exist +- `metafile_exists_ttl`: how long to cache information about whether meta.json or deletion mark file exists. +- `metafile_doesnt_exist_ttl`: how long to cache information about whether meta.json or deletion mark file doesn't exist. - `metafile_content_ttl`: how long to cache content of meta.json and deletion mark files. - `metafile_max_size`: maximum size of cached meta.json and deletion mark file. Larger files are not cached.