v0.34.0-rc.0

v0.34.0-rc.0 is out!

Thank you to all contributors who have contributed to this release. It wouldn't be possible without you.

Please take note that the default value of the flag --sync-block-duration has been updated from 3m to 15m!

You can find the changelog with all of the details below. Let's also celebrate all our new contributors!
Please try it out and let us know if you spot any problems! Full-release/next rc will be in 3 days!

Changelog

Fixed

• #7011 Query Frontend: queries with negative offset should check whether it is cacheable or not.
• #6874 Sidecar: fix labels returned by 'api/v1/series' in presence of conflicting external and inner labels.
• #7009 Rule: Fix spacing error in URL.

Added

• #6756 Query: Add query.enable-tenancy & query.tenant-label-name options to allow enforcement of tenancy on the query path, by injecting labels into queries (uses prom-label-proxy internally).
• #6944 Receive: Added a new flag for maximum retention bytes.
• #6891 Objstore: Bump objstore which adds support for Azure Workload Identity.
• #6453 Sidecar: Added --reloader.method to support configuration reloads via SIHUP signal.
• #6925 Store Gateway: Support float native histogram.
• #6954 Index Cache: Support tracing for fetch APIs.
• #6943 Ruler: Added keep_firing_for field in alerting rule.
• #6972 Store Gateway: Apply series limit when streaming series for series actually matched if lazy postings is enabled.
• #6984 Store Gateway: Added --store.index-header-lazy-download-strategy to specify how to lazily download index headers when lazy mmap is enabled.
• #6887 Query Frontend: breaking ⚠️ Add tenant label to relevant exported metrics. Note that this change may cause some pre-existing custom dashboard queries to be incorrect due to the added label.
• #7028 Query|Query Frontend: Add new --query-frontend.enable-x-functions flag to enable experimental extended functions.
• #6884 Tools: Add upload-block command to upload blocks to object storage.

Changed

• #6539 Store: breaking ⚠️ Changed --sync-block-duration default 3m to 15m.

Removed

New Contributors

• @lpreethvika made their first contribution in #6829
• @danielblando made their first contribution in #6850
• @rikhil-s made their first contribution in #6891
• @sinkingpoint made their first contribution in #6886
• @MeenuyD made their first contribution in #6907
• @mercxry made their first contribution in #6933
• @wenxu1024 made their first contribution in #6902
• @kartikaysaxena made their first contribution in #6927
• @sagnik3788 made their first contribution in #6952
• @JHeilCoveo made their first contribution in #6943
• @pawarpranav83 made their first contribution in #6998
• @tasrieit made their first contribution in #7023
• @Pratham1812 made their first contribution in #7026
• @alecrajeev made their first contribution in #7032
• @Player256 made their first contribution in #6539

Full Changelog: v0.33.0...v0.34.0-rc.0

v0.33.0

v0.33.0 is out!
Thank you to all contributors who have contributed to this release. It wouldn't be possible without you.

Do take note of some of the breaking metric changes in the querier and store around tenancy.

You can find the changelog with all of the details below. Let's also celebrate all our new contributors!
Please try it out and let us know if you spot any problems!

Changelog

Fixed

• #6817 Store Gateway: fix matchersToPostingGroups label values variable got shadowed bug.

Added

• #6891 Objstore: Bump objstore which adds support for Azure Workload Identity.
• #6605 Query Frontend: Support vertical sharding binary expression with metric name when no matching labels specified.
• #6308 Ruler: Support configuration flag that allows customizing template for alert message.
• #6760 Query Frontend: Added TLS support in --query-frontend.downstream-tripper-config and --query-frontend.downstream-tripper-config-file
• #6749 Store Gateway: Added thanos_store_index_cache_fetch_duration_seconds histogram for tracking latency of fetching data from index cache.
• #6690 Store: breaking ⚠️ Add tenant label to relevant exported metrics. Note that this change may cause some pre-existing dashboard queries to be incorrect due to the added label.
• #6530 / #6690 Query: Add command line arguments for configuring tenants and forward tenant information to Store Gateway.
• #6765 Index Cache: Add enabled_items to index cache config to selectively cache configured items. Available item types are Postings, Series and ExpandedPostings.
• #6773 Index Cache: Add ttl to control the ttl to store items in remote index caches like memcached and redis.
• #6794 Query: breaking ⚠️ Add tenant label to relevant exported metrics. Note that this change may cause some pre-existing custom dashboard queries to be incorrect due to the added label.

Changed

• #6698 Receive: Change write log level from warn to info.
• #6753 mixin(Rule): breaking ⚠️ Fixed the mixin rules with duplicate names and updated the promtool version from v0.37.0 to v0.47.0
• #6772 *: Bump prometheus to v0.47.2-0.20231006112807-a5a4eab679cc
• #6794 Receive: the exported HTTP metrics now uses the specified default tenant for requests where no tenants are found.

Removed

• #6686 Remove deprecated --log.request.decision flag. We now use --request.logging-config to set logging decisions.

New Contributors

• @Vanshikav123 made their first contribution in #6628
• @verejoel made their first contribution in #6640
• @harsh-ps-2003 made their first contribution in #6646
• @lmarques03 made their first contribution in #6662
• @zhuoyuan-liu made their first contribution in #6308
• @ritaCanavarro made their first contribution in #6544
• @SFernandoS made their first contribution in #6725
• @Preethivika made their first contribution in #6753
• @bazooka3000 made their first contribution in #6760
• @nishchay-veer made their first contribution in #6515
• @nelsonmarcos made their first contribution in #6832
• @donuts-are-good made their first contribution in #6834

Full Changelog: v0.32.5...v0.33.0

v0.33.0-rc.0

v0.33.0-rc.0 is out!
Thank you to all contributors who have contributed to this release. It wouldn't be possible without you.

Do take note of some of the breaking metric changes in the querier and store around tenancy.

You can find the changelog with all of the details below. Let's also celebrate all our new contributors!
Please try it out and let us know if you spot any problems! Full-release/next rc will be in 3 days!

Changelog

Fixed

• #6817 Store Gateway: fix matchersToPostingGroups label values variable got shadowed bug.

Added

• #6891 Objstore: Bump objstore which adds support for Azure Workload Identity.
• #6605 Query Frontend: Support vertical sharding binary expression with metric name when no matching labels specified.
• #6308 Ruler: Support configuration flag that allows customizing template for alert message.
• #6760 Query Frontend: Added TLS support in --query-frontend.downstream-tripper-config and --query-frontend.downstream-tripper-config-file
• #6749 Store Gateway: Added thanos_store_index_cache_fetch_duration_seconds histogram for tracking latency of fetching data from index cache.
• #6690 Store: breaking ⚠️ Add tenant label to relevant exported metrics. Note that this change may cause some pre-existing dashboard queries to be incorrect due to the added label.
• #6530 / #6690 Query: Add command line arguments for configuring tenants and forward tenant information to Store Gateway.
• #6765 Index Cache: Add enabled_items to index cache config to selectively cache configured items. Available item types are Postings, Series and ExpandedPostings.
• #6773 Index Cache: Add ttl to control the ttl to store items in remote index caches like memcached and redis.
• #6794 Query: breaking ⚠️ Add tenant label to relevant exported metrics. Note that this change may cause some pre-existing custom dashboard queries to be incorrect due to the added label.

Changed

• #6698 Receive: Change write log level from warn to info.
• #6753 mixin(Rule): breaking ⚠️ Fixed the mixin rules with duplicate names and updated the promtool version from v0.37.0 to v0.47.0
• #6772 *: Bump prometheus to v0.47.2-0.20231006112807-a5a4eab679cc
• #6794 Receive: the exported HTTP metrics now uses the specified default tenant for requests where no tenants are found.

Removed

• #6686 Remove deprecated --log.request.decision flag. We now use --request.logging-config to set logging decisions.

New Contributors

• @Vanshikav123 made their first contribution in #6628
• @verejoel made their first contribution in #6640
• @harsh-ps-2003 made their first contribution in #6646
• @lmarques03 made their first contribution in #6662
• @zhuoyuan-liu made their first contribution in #6308
• @ritaCanavarro made their first contribution in #6544
• @SFernandoS made their first contribution in #6725
• @Preethivika made their first contribution in #6753
• @bazooka3000 made their first contribution in #6760
• @nishchay-veer made their first contribution in #6515
• @nelsonmarcos made their first contribution in #6832
• @donuts-are-good made their first contribution in #6834

Full Changelog: v0.32.5...v0.33.0-rc.0

v0.32.5

This patch release brings a fix for Receive, for a bug that allowed the head series limiter to be run without explicitly setting it. It also brings a fix for Store GW, for a bug that caused /api/v1/labels to not filter external labels.

Alongside this, we also build with Go 1.21.3 and golang.org/x/net v0.17 in this release to address Go CVE.
Please try it out and let us know if you spot any problems! Enjoy! 🎉

Changes

Fixed

• #6615 #6805: Build with Go 1.21 and bump golang.org/x/net to v0.17 for addressing Go CVE.
• #6802 Receive: head series limiter should not run if no head series limit is set.
• #6816 Store: fix prometheus store label values matches for external labels.

v0.32.4

This patch release updates our object storage library to address a few issues that were introduced earlier.
Please try it out and let us know if you spot any problems! Enjoy! 🎉

Changes

Fixed

• #6746 Objstore: Upgrade with fixes objstore#77 and objstore#78.

v0.32.3

This patch release again brings a few fixes for the Store Gateway. In the previous release, certain users reported receiving a duplicate labelset error on some of their queries. This has now been addressed.
Also, includes an optimization for GCS objstore interactions and bumps in dependencies like prometheus, objstore and promql-engine.
Please try it out and let us know if you spot any problems! Enjoy! 🎉

Changes

Fixed

• #6692 Store: Fix matching bug when using empty alternative in regex matcher, for example (a||b).
• #6679 Store: Fix block deduplication
• #6706 Store: Series responses should always be sorted

Changed

• #6664 *: Update Prometheus to 2.46.1.
• #6722 *: Optimize iterations on GCS buckets by requesting only object names.
• #6544 Objstore: Update objstore to latest version which adds a new metric regarding uploaded TSDB bytes

v0.32.2

This patch release again brings a few fixes for the Store Gateway. In the previous release, certain users reported receiving a nil postings error on some of their queries, alongside certain race conditions, and empty query stats on debug messages. These have now been addressed.
Please try it out and let us know if you spot any problems! Enjoy! 🎉

Changes

Fixed

• #6675 Store: Fix race when iterating blocks
• #6679 Store: Record stats even on ExpandPostings error
• #6681 Store: Fix forgotten field in store stats merge
• #6684 Store: Fix postings reader short reads to address nil postings bug

v0.32.1

This patch release contains a few fixes for the Store Gateway. In the previous release, a memory issue was detected in the label names set implementation, alongside certain ignored or unchecked errors in store postings. These have now been addressed.
Please try it out and let us know if you spot any problems! Enjoy! 🎉

Changes

Fixed

• #6650 Store: fix error handling in decodePostings
• #6654 Store: fix ignored error in postings
• #6655 Store: fix bufio pool handling
• #6669 Store: Fix mutable stringset memory usage

v0.32.0

v0.32.0 is out after a long wait, as we were busy fixing a rather challenging issue!
Thank you to all contributors who have contributed to this release. It wouldn't be possible without you.
Some of the highlights include support for PromQL query explanations in the UI when using the thanos PromQL engine, AZ-aware replication for Receive and other new flags, tools bucket replicate improvements, and lots of optimizations and bug/race fixes!
Do take note of some of the breaking metric name changes and the change in container image user.
You can find the changelog with all of the details below. Let's also celebrate all our new contributors!
Please try it out and let us know if you spot any problems! Enjoy! 🎉

Changes

Added

• #6437 Receive: make tenant stats limit configurable
• #6369 Receive: add az-aware replication support for Ketama algorithm
• #6185 Tracing: tracing in OTLP support configuring service_name.
• #6192 Store: add flag bucket-web-label to select the label to use as timeline title in web UI
• #6195 Receive: add flag tsdb.too-far-in-future.time-window to prevent clock skewed samples to pollute TSDB head and block all valid incoming samples.
• #6273 Mixin: Allow specifying an instance name filter in dashboards
• #6163 Receiver: Add hidden flag --receive-forward-max-backoff to configure the max backoff for forwarding requests.
• #5777 Receive: Allow specifying tenant-specific external labels in Router Ingestor.
• #6352 Store: Expose store gateway query stats in series response hints.
• #6420 Index Cache: Cache expanded postings.
• #6441 Compact: Compactor will set index_stats in meta.json file with max series and chunk size information.
• #6466 Mixin (Receive): add limits alerting for configuration reload and meta-monitoring.
• #6467 Mixin (Receive): add alert for tenant reaching head series limit.
• #6528 Index Cache: Add histogram metric thanos_store_index_cache_stored_data_size_bytes for item size.
• #6560 Thanos ruler: add flag to optionally disable adding Thanos params when querying metrics
• #6574 Tools: Add min and max compactions range flags to bucket replicate command.
• #6593 Store: Add thanos_bucket_store_chunk_refetches_total metric to track number of chunk refetches.
• #6264 Query: Add Thanos logo in navbar
• #6234 Query: Add ability to switch between thanos and prometheus engines dynamically via UI and API.
• #6346 Query: Add ability to generate SQL-like query explanations when thanos engine is used.

Fixed

• #6503 *: Change the engine behind ContentPathReloader to be completely independent of any filesystem concept. This effectively fixes this configuration reload when used with Kubernetes ConfigMaps, Secrets, or other volume mounts.
• #6456 Store: fix crash when computing set matches from regex pattern
• #6427 Receive: increased log level for failed uploads to error
• #6172 query-frontend: return JSON formatted errors for invalid PromQL expression in the split by interval middleware.
• #6171 Store: fix error handling on limits.
• #6183 Receiver: fix off by one in multitsdb flush that will result in empty blocks if the head only contains one sample
• #6197 Exemplar OTel: Fix exemplar for otel to use traceId instead of spanId and sample only if trace is sampled
• #6207 Receive: Remove the shipper once a tenant has been pruned.
• #6216 Receiver: removed hard-coded value of EnableExemplarStorage flag and set it according to max-exemplar value.
• #6222 mixin(Receive): Fix tenant series received dashboard widget.
• #6218 mixin(Store): handle ResourceExhausted as a non-server error. As a consequence, this error won't contribute to Store's grpc errors alerts.
• #6271 Receive: Fix segfault in LabelValues during head compaction.
• #6306 Tracing: tracing in OTLP utilize the OTEL_TRACES_SAMPLER env variable
• #6330 Store: Fix inconsistent error for series limits.
• #6342 Cache/Redis: Upgrade rueidis to v1.0.2 to to improve error handling while shrinking a redis cluster.
• #6325 Store: return gRPC resource exhausted error for byte limiter.
• #6399 *: Fix double-counting bug in http_request_duration metric
• #6428 Report gRPC connnection errors in the logs.
• #6519 Reloader: Use timeout for initial apply.
• #6509 Store Gateway: Remove memWriter from fileWriter to reduce memory usage when sync index headers.
• #6556 Thanos compact: respect block-files-concurrency setting when downsampling
• #6592 Query Frontend: fix bugs in vertical sharding without and union function to allow more queries to be shardable.
• #6317 *: Fix internal label deduplication bug, by resorting store response set.
• #6189 Rule: Fix panic when calling API /api/v1/rules?type=alert.

Changed

• #6049 Compact: breaking ⚠️ Replace group with resolution in compact metrics to avoid cardinality explosion on compact metrics for large numbers of groups.
• #6168 Receiver: Make ketama hashring fail early when configured with number of nodes lower than the replication factor.
• #6201 Query-Frontend: Disable absent and absent_over_time for vertical sharding.
• #6212 Query-Frontend: Disable scalar for vertical sharding.
• #6107 breaking ⚠️ Change default user id in container image from 0(root) to 1001
• #6228 Conditionally generate debug messages in ProxyStore to avoid memory bloat.
• #6231 mixins: Add code/grpc-code dimension to error widgets.
• #6244 mixin(Rule): Add rule evaluation failures to the Rule dashboard.
• #6303 Store: added and start using streamed snappy encoding for postings list instead of block based one. This leads to constant memory usage during decompression. This approximately halves memory usage when decompressing a postings list in index cache.
• #6071 Query Frontend: breaking ⚠️ Add experimental native histogram support for which we updated and aligned with the Prometheus common model, which is used for caching so a cache reset required.
• #6163 Receiver: changed default max backoff from 30s to 5s for forwarding requests. Can be configured with --receive-forward-max-backoff.
• #6327 *: breaking ⚠️ Use histograms instead of summaries for instrumented handlers.
• #6322 Logging: Avoid expensive log.Valuer evaluation for disallowed levels.
• #6358 Query: Add +Inf bucket to query duration metrics
• #6363 Store: Check context error when expanding postings.
• #6405 Index Cache: Change postings cache key to include the encoding format used so that older Thanos versions would not try to decode it during the deployment of a new version.
• #6479 Store: breaking ⚠️ Rename thanos_bucket_store_cached_series_fetch_duration_seconds to thanos_bucket_store_series_fetch_duration_seconds and thanos_bucket_store_cached_postings_fetch_duration_seconds to thanos_bucket_store_postings_fetch_duration_seconds.
• #6474 Store/Compact: Reduce a ...

v0.32.0-rc.1

• #6612: Store: Fix missing flush when handling pushed down queries