Merge branch 'main' into fix-rules-manager-filenames

.bingo/prometheus.mod

@@ -9,6 +9,7 @@ replace (
 	github.com/cockroachdb/cmux => github.com/cockroachdb/cmux v0.0.0-20170110192607-30d10be49292
 	github.com/cockroachdb/cockroach => github.com/cockroachdb/cockroach v0.0.0-20170608034007-84bc9597164f
 	github.com/googleapis/gnostic => github.com/googleapis/gnostic v0.2.3-0.20180520015035-48a0ecefe2e4
+	github.com/gophercloud/gophercloud => github.com/gophercloud/gophercloud v0.18.0
 	github.com/miekg/dns => github.com/miekg/dns v1.0.4
 	github.com/prometheus/client_golang => github.com/prometheus/client_golang v0.9.0-pre1.0.20180607123607-faf4ec335fe0
 	github.com/prometheus/common => github.com/prometheus/common v0.0.0-20180518154759-7600349dcfe1

CHANGELOG.md

@@ -11,15 +11,18 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
 ## Unreleased
 
 ### Added
+- [#4453](https://github.com/thanos-io/thanos/pull/4453) Tools: Add flag `--selector.relabel-config-file` / `--selector.relabel-config` / `--max-time` / `--min-time` to filter served blocks.
+- [#4482](https://github.com/thanos-io/thanos/pull/4482) COS: Add http_config for cos object store client.
 
 ### Fixed
 
 - [#4442](https://github.com/thanos-io/thanos/pull/4442) rule: fix reload signal not working
-- [#4468] (https://github.com/thanos-io/thanos/pull/4468) Rule: Fix temporary rule filename composition issue
+- [#4468](https://github.com/thanos-io/thanos/pull/4468) Rule: Fix temporary rule filename composition issue
+- [#4476](https://github.com/thanos-io/thanos/pull/4476) UI: fix incorrect html escape sequence used for '>' symbol.
 
 ### Changed
 
-## [v0.22.0 - in progress](https://github.com/thanos-io/thanos/tree/release-0.22)
+## [v0.22.0](https://github.com/thanos-io/thanos/tree/release-0.22) - 2021.07.22
 
 ### Added
 
@@ -39,6 +42,7 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
 - [#4384](https://github.com/thanos-io/thanos/pull/4384) Fix the experimental PromQL editor when used on multiple line.
 - [#4342](https://github.com/thanos-io/thanos/pull/4342) ThanosSidecarUnhealthy doesn't fire if the sidecar is never healthy
 - [#4388](https://github.com/thanos-io/thanos/pull/4388) Receive: fix bug in forwarding remote-write requests within the hashring via gRPC when TLS is enabled on the HTTP server but not on the gRPC server.
+- [#4442](https://github.com/thanos-io/thanos/pull/4442) Ruler: fix SIGHUP reload signal not working.
 
 ### Changed
 

CONTRIBUTING.md

@@ -1,6 +1,6 @@
 # Contributing
 
-This document explain the process of contributing to the Thanos project.
+This document explains the process of contributing to the Thanos project.
 
 First of all please follow the [CODE_OF_CONDUCT](CODE_OF_CONDUCT.md) in all your interactions within the project.
 
@@ -9,9 +9,9 @@ First of all please follow the [CODE_OF_CONDUCT](CODE_OF_CONDUCT.md) in all your
 The philosophy of Thanos and our community borrows heavily from UNIX philosophy and the Golang programming language.
 
 * Each subcommand should do one thing and do it well.
-  * eg. thanos query proxies incoming calls to known store API endpoints merging the result
+  * eg. Thanos query proxies incoming calls to known store API endpoints merging the result
 * Write components that work together.
-  * e.g. blocks should be stored in native prometheus format
+  * e.g. blocks should be stored in native Prometheus format
 * Make it easy to read, write, and run components.
   * e.g. reduce complexity in system design and implementation
 
@@ -84,7 +84,7 @@ It's key to get familiarized with the style guide and mechanics of Thanos, espec
 
 * Find any directory in your system your want Thanos repo in. e.g `~/Repos` -
   * `cd ~/Repos`
-  * Make sure that the GOBIN, GOPATH and GOPROXY (useful) environment variables are set and that GOBIN is included in your PATH. For example -
+  * Make sure that the GOBIN, GOPATH and GOPROXY (useful) environment variables are set and that GOBIN is included in your PATH. You may use `GOPROXY=https://goproxy.cn` as an alternative if you cannot visit `https://proxy.golang.org`. For example -
 
     ```
     export GOBIN="~/Repos/thanos/.bin" # It's nice to have local tooling installed and stored locally.
@@ -210,7 +210,7 @@ You have to commit the changes to `go.mod` and `go.sum` before submitting the pu
 At some point during development it is useful, in addition to running unit or e2e tests, to run and play with Thanos components manually. While you can run any component manually by crafting specific flags for a test setup, there are already some nice tools and scripts available. Consider the following methods:
 
 * `make quickstart`: this command spins up a simple setup of all Thanos components.
-* `make test-e2e`: the e2e tests cover most of the setups and functionality Thanos offers. It's extremely easy to add `time.Sleep(10* time.Minutes)` at certain points in the tests (e.g for compactor [here](https://github.com/thanos-io/thanos/blob/8f492a9f073f819019dd9f044e346a1e1fa730bc/test/e2e/compact_test.go#L379)). This way when you run `make test-e2e`, the test will sleep for some time, allowing you to connect to the setup manually using the port printed in the logs. For example:
+* `make test-e2e`: the e2e tests cover most of the setups and functionality Thanos offers. It's extremely easy to add `time.Sleep(10* time.Minute)` at certain points in the tests (e.g for compactor [here](https://github.com/thanos-io/thanos/blob/8f492a9f073f819019dd9f044e346a1e1fa730bc/test/e2e/compact_test.go#L379)). This way when you run `make test-e2e`, the test will sleep for some time, allowing you to connect to the setup manually using the port printed in the logs. For example:
 
 ```bash
 querier-1: level=info name=querier-1 ts=2020-04-01T12:53:56.101029491Z caller=http.go:56 service=http/server component=query msg="listening for requests and metrics" address=:80

VERSION

@@ -1 +1 @@
-0.22.0-dev
+0.23.0-dev

cmd/thanos/tools_bucket.go

@@ -53,6 +53,7 @@ import (
 	"github.com/thanos-io/thanos/pkg/replicate"
 	"github.com/thanos-io/thanos/pkg/runutil"
 	httpserver "github.com/thanos-io/thanos/pkg/server/http"
+	"github.com/thanos-io/thanos/pkg/store"
 	"github.com/thanos-io/thanos/pkg/ui"
 	"github.com/thanos-io/thanos/pkg/verifier"
 	"golang.org/x/text/language"
@@ -334,6 +335,12 @@ func registerBucketWeb(app extkingpin.AppClause, objStoreConfig *extflag.PathOrC
 	interval := cmd.Flag("refresh", "Refresh interval to download metadata from remote storage").Default("30m").Duration()
 	timeout := cmd.Flag("timeout", "Timeout to download metadata from remote storage").Default("5m").Duration()
 	label := cmd.Flag("label", "Prometheus label to use as timeline title").String()
+	filterConf := &store.FilterConfig{}
+	cmd.Flag("min-time", "Start of time range limit to serve. Thanos tool bucket web will serve only blocks, which happened later than this value. Option can be a constant time in RFC3339 format or time duration relative to current time, such as -1d or 2h45m. Valid duration units are ms, s, m, h, d, w, y.").
+		Default("0000-01-01T00:00:00Z").SetValue(&filterConf.MinTime)
+	cmd.Flag("max-time", "End of time range limit to serve. Thanos tool bucket web will serve only blocks, which happened earlier than this value. Option can be a constant time in RFC3339 format or time duration relative to current time, such as -1d or 2h45m. Valid duration units are ms, s, m, h, d, w, y.").
+		Default("9999-12-31T23:59:59Z").SetValue(&filterConf.MaxTime)
+	selectorRelabelConf := *extkingpin.RegisterSelectorRelabelFlags(cmd)
 
 	cmd.Setup(func(g *run.Group, logger log.Logger, reg *prometheus.Registry, tracer opentracing.Tracer, _ <-chan struct{}, _ bool) error {
 		comp := component.Bucket
@@ -414,8 +421,22 @@ func registerBucketWeb(app extkingpin.AppClause, objStoreConfig *extflag.PathOrC
 			return errors.Wrap(err, "bucket client")
 		}
 
+		relabelContentYaml, err := selectorRelabelConf.Content()
+		if err != nil {
+			return errors.Wrap(err, "get content of relabel configuration")
+		}
+
+		relabelConfig, err := block.ParseRelabelConfig(relabelContentYaml, block.SelectorSupportedRelabelActions)
+		if err != nil {
+			return err
+		}
 		// TODO(bwplotka): Allow Bucket UI to visualize the state of block as well.
-		fetcher, err := block.NewMetaFetcher(logger, block.FetcherConcurrency, bkt, "", extprom.WrapRegistererWithPrefix(extpromPrefix, reg), nil, nil)
+		fetcher, err := block.NewMetaFetcher(logger, block.FetcherConcurrency, bkt, "", extprom.WrapRegistererWithPrefix(extpromPrefix, reg),
+			[]block.MetadataFilter{
+				block.NewTimePartitionMetaFilter(filterConf.MinTime, filterConf.MaxTime),
+				block.NewLabelShardedMetaFilter(relabelConfig),
+				block.NewDeduplicateFilter(),
+			}, nil)
 		if err != nil {
 			return err
 		}

docs/components/rule.md

@@ -197,14 +197,14 @@ On HTTP address Ruler exposes its UI that shows mainly Alerts and Rules page (si
 
 ## Ruler HA
 
-Ruler aims to use a similar approach to the one that Prometheus has. You can configure external labels, as well as simple relabelling.
+Ruler aims to use a similar approach to the one that Prometheus has. You can configure external labels, as well as relabelling.
 
 In case of Ruler in HA you need to make sure you have the following labelling setup:
 
 * Labels that identify the HA group ruler and replica label with different value for each ruler instance, e.g: `cluster="eu1", replica="A"` and `cluster=eu1, replica="B"` by using `--label` flag.
 * Labels that need to be dropped just before sending to alermanager in order for alertmanager to deduplicate alerts e.g `--alert.label-drop="replica"`.
 
-Full relabelling is planned to be done in future and is tracked here: https://github.com/thanos-io/thanos/issues/660
+Advanced relabelling configuration is possible with the `--alert.relabel-config` and `--alert.relabel-config-file` flags. The configuration format is identical to the [`alert_relabel_configs`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#alert_relabel_configs) field of Prometheus. Note that Thanos Ruler drops the labels listed in `--alert.label-drop` before alert relabelling.
 
 ## Flags
 

docs/components/tools.md

@@ -228,6 +228,21 @@ Flags:
       --log.format=logfmt       Log format to use. Possible options: logfmt or
                                 json.
       --log.level=info          Log filtering level.
+      --max-time=9999-12-31T23:59:59Z  
+                                End of time range limit to serve. Thanos tool
+                                bucket web will serve only blocks, which
+                                happened earlier than this value. Option can be
+                                a constant time in RFC3339 format or time
+                                duration relative to current time, such as -1d
+                                or 2h45m. Valid duration units are ms, s, m, h,
+                                d, w, y.
+      --min-time=0000-01-01T00:00:00Z  
+                                Start of time range limit to serve. Thanos tool
+                                bucket web will serve only blocks, which
+                                happened later than this value. Option can be a
+                                constant time in RFC3339 format or time duration
+                                relative to current time, such as -1d or 2h45m.
+                                Valid duration units are ms, s, m, h, d, w, y.
       --objstore.config=<content>  
                                 Alternative to 'objstore.config-file' flag
                                 (mutually exclusive). Content of YAML file that
@@ -240,6 +255,20 @@ Flags:
                                 https://thanos.io/tip/thanos/storage.md/#configuration
       --refresh=30m             Refresh interval to download metadata from
                                 remote storage
+      --selector.relabel-config=<content>  
+                                Alternative to 'selector.relabel-config-file'
+                                flag (mutually exclusive). Content of YAML file
+                                that contains relabeling configuration that
+                                allows selecting blocks. It follows native
+                                Prometheus relabel-config syntax. See format
+                                details:
+                                https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
+      --selector.relabel-config-file=<file-path>  
+                                Path to YAML file that contains relabeling
+                                configuration that allows selecting blocks. It
+                                follows native Prometheus relabel-config syntax.
+                                See format details:
+                                https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
       --timeout=5m              Timeout to download metadata from remote storage
       --tracing.config=<content>  
                                 Alternative to 'tracing.config-file' flag

docs/proposals-accepted/202107-protobuf-openapi-httpapi.md

@@ -0,0 +1,82 @@
+---
+type: proposal
+title: Expose Thanos APIs to OpenAPI/protobuf and expose on website and UIs
+status: accepted
+owner: Hangzhi
+menu: proposals-accepted
+---
+
+## **Expose Thanos APIs to OpenAPI/protobuf and expose on website and UIs**
+
+* Owners:
+  * Hangzhi
+* Related Tickets:
+  * Program Proposal: https://github.com/thanos-io/thanos/issues/4102
+  * Need for auto-documentation in Prometheus: https://github.com/prometheus/prometheus/issues/7192, https://github.com/prometheus/prometheus/issues/5567
+* Other docs:
+  * A proposal of LFX mentorship: https://docs.google.com/document/d/1GeyFCk4QkVSrx4pQ4OSmOA1KE_FvmJRi_aq9hr2GoZQ/edit?usp=sharing
+
+This design doc is proposing a design for Thanos HTTP API defined in protobuf and OpenAPI.
+
+## **Motivations**
+
+To improve Thanos usage for users, we would like to define our HTTP APIs in protobuf/OpenAPI and expose those in the repository. OpenAPI is a language for describing REST APIs with a widely-used tooling ecosystem. With OpenAPI, developers can generate live documentation, validate APIs and even generate client and server stubs from OpenAPI to use our APIs efficiently. Also, the auto-generated documentation problem prevent documentation errors (Prometheus#7192, Prometheus#5567). Protocol Buffers (a.k.a., protobuf) is well-known as a mechanism for serializing structured data, and it's usually used to define gRPC APIs. Also, protobuf specification could be used to define REST API. We hope to use protobuf to define our APIs for consistency, while we also want to leverage the tooling ecosystem of OpenAPI.
+
+So, we want to define REST APIs in protobuf, generate OpenAPI definition from protobuf with [gnostic](https://github.com/google/gnostic). This would allow users to use tools for documentation, validation, type checking, and even interface code generation to use our APIs efficiently.
+
+Similarly, we want to reuse this work in Prometheus.
+
+### **Pitfalls of the current solution**
+
+* Documentation, Server code, and client code boilerplate needs to be written manually.
+* Hand-written interface code is expensive to write and hard to maintain.
+* It's hard to discover the current API programmatically.
+* When modifying the API, it is very difficult to know if this breaks downstream users.
+
+## **Goals**
+
+* Define all APIs in protobuf.
+* Be able to generate OpenAPI3 from protobuf.
+* Auto-generate documentation with OpenAPI3 specification.
+* Generate server code from API specification (OpenAPI3 or protobuf).
+
+## **Non-Goals**
+
+* Don't mix gRPC with HTTP APIs in the same protobuf package
+
+## **How**
+
+* Define Thanos APIs in protobuf specification.
+* Generate OpenAPI from protobuf with gnostic extension.
+* Generate documentation from OpenAPI with [Swagger](https://github.com/swagger-api/swagger-codegen).
+* Generate server and client API stubs from OpenAPI with Swagger.
+
+## **Alternatives**
+
+## **Define HTTP RESTful API only in OpenAPI. Not in protobuf.**
+
+1. Pros:
+   1. There are might be some complexity, edge case and extra tooling to make the 3-step process (proto -> OpenAPI -> documentation to work.
+
+2. Cons:
+   1. We write API definitions in protobuf for consistency. OpenAPI is less consistent compared to protobuf in a project built on Golang.
+   2. Protobuf definition is more concise to write.
+
+## **Define APIs in gRPC and have RESTful APIs alongside with grpc - gateway**
+
+1. Pros:
+   1. We can have gRPC and RESTful APIs at the same time.
+   2. We have gRPC APIs like rules API.
+2. Cons:
+   1. There are might be some complexity, edge cases and extra tooling to make the process (define gRPC API and get RESTful APIs with grpc-gateway) work.
+   2. We need to redefine our API service in gPRC since most of Thanos APIs and Prometheus APIs are RESTful APIs.
+   3. We need to run another sidecar (complexity of running the system).
+   4. Semantics of gRPC and HTTP might be different and surprising for end user.
+   5. We want to reuse in Prometheus and Prometheus does not support gRPC (gRPC dependency was removed from codebase).
+   6. Same port library is not maintained. (cmux)
+
+## **Action Plan**
+
+* [ ] Experiments in rules API.
+* [ ] Define all HTTP APIs in Protobuf.
+* [ ] Generate documentation, client code, and server stubs from OpenAPI specification.

docs/storage.md

@@ -389,6 +389,14 @@ config:
   app_id: ""
   secret_key: ""
   secret_id: ""
+  http_config:
+    idle_conn_timeout: 1m30s
+    response_header_timeout: 2m
+    tls_handshake_timeout: 10s
+    expect_continue_timeout: 1s
+    max_idle_conns: 100
+    max_idle_conns_per_host: 100
+    max_conns_per_host: 0
 ```
 
 Set the flags `--objstore.config-file` to reference to the configuration file.

pkg/block/metadata/meta.go

@@ -114,6 +114,7 @@ func (m *Matchers) UnmarshalYAML(value *yaml.Node) (err error) {
 type DeletionRequest struct {
 	Matchers  Matchers             `json:"matchers" yaml:"matchers"`
 	Intervals tombstones.Intervals `json:"intervals,omitempty" yaml:"intervals,omitempty"`
+	RequestID string               `json:"request_id,omitempty" yaml:"request_id,omitempty"`
 }
 
 type File struct {