-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Querier/Store v0.8.0 regression: Thanos now considers identically externally labeled Store and Sidecar instances as having duplicate label sets. #1632
Comments
Thanks for report. It seems to be a bug. |
Bug reproduced on query v0.7.0 and store v0.8.0, and not reproduced on query v0.8.0 and store v0.7.0 |
@IKSIN Yes, because store v0.7.0 will not expose advertised labels. Will fix ASAP. |
Thanks for reporting! It looks like we missed quite a simple scenario. What we changed is v0.8.0 store is advertising labels: It takes all blocks it has and advertise them all. If you have just a simple case of one store and just one Prometheus sidecar - there is overlap in advertise labels logic and querier blocks one. We are working on a fix to Querier and we will release patch release We can fix Querier for |
I am happy to take it @jojohappy as it's 1am your time. |
…locked for older Queriers. Fixes #1632 Signed-off-by: Bartek Plotka <[email protected]>
… blocked for older Queriers. Changes: * Cleaned up the code a bit. * Ensured proper locks. * thanos_store_nodes_grpc_connections is not per store type and external labels. thanos_store_node_info is marked as deprecated. * Added (optional, but enabled by default) compatibility label to store GW Info method with flag to disable it. Store only adds it if there is any labelset to advertise. * Disabled any strict deduplication in external labels detection; just warning! * More tests; compatibility tests for query pre 0.8.0 and new store GW. * Remove compatibility label if spotted. Fixes #1632 Signed-off-by: Bartek Plotka <[email protected]>
Fix is in review: #1636 I built
|
… blocked for older Queriers. Changes: * Cleaned up the code a bit. * Ensured proper locks. * thanos_store_nodes_grpc_connections is not per store type and external labels. thanos_store_node_info is marked as deprecated. * Added (optional, but enabled by default) compatibility label to store GW Info method with flag to disable it. Store only adds it if there is any labelset to advertise. * Disabled any strict deduplication in external labels detection; just warning! * More tests; compatibility tests for query pre 0.8.0 and new store GW. * Remove compatibility label if spotted. Fixes #1632 Signed-off-by: Bartek Plotka <[email protected]>
… blocked for older Queriers. (#1636) Changes: * Cleaned up the code a bit. * Ensured proper locks. * thanos_store_nodes_grpc_connections is not per store type and external labels. thanos_store_node_info is marked as deprecated. * Added (optional, but enabled by default) compatibility label to store GW Info method with flag to disable it. Store only adds it if there is any labelset to advertise. * Disabled any strict deduplication in external labels detection; just warning! * More tests; compatibility tests for query pre 0.8.0 and new store GW. * Remove compatibility label if spotted. Fixes #1632 Signed-off-by: Bartek Plotka <[email protected]>
… blocked for older Queriers. Changes: * Cleaned up the code a bit. * Ensured proper locks. * thanos_store_nodes_grpc_connections is not per store type and external labels. thanos_store_node_info is marked as deprecated. * Added (optional, but enabled by default) compatibility label to store GW Info method with flag to disable it. Store only adds it if there is any labelset to advertise. * Disabled any strict deduplication in external labels detection; just warning! * More tests; compatibility tests for query pre 0.8.0 and new store GW. * Remove compatibility label if spotted. Fixes #1632 Signed-off-by: Bartek Plotka <[email protected]>
… blocked for older Queriers. Changes: * Cleaned up the code a bit. * Ensured proper locks. * thanos_store_nodes_grpc_connections is not per store type and external labels. thanos_store_node_info is marked as deprecated. * Added (optional, but enabled by default) compatibility label to store GW Info method with flag to disable it. Store only adds it if there is any labelset to advertise. * Disabled any strict deduplication in external labels detection; just warning! * More tests; compatibility tests for query pre 0.8.0 and new store GW. * Remove compatibility label if spotted. Fixes #1632 Signed-off-by: Bartek Plotka <[email protected]>
…ase 0.8.1 (#1640) * Pinned `x/sys` to certain version that is buildable for windows platform. (#1626) Signed-off-by: Bartek Plotka <[email protected]> * Post release update. (#1628) Signed-off-by: Bartek Plotka <[email protected]> * Removed Querier duplicate labels checks & made sure store will be not blocked for older Queriers. Changes: * Cleaned up the code a bit. * Ensured proper locks. * thanos_store_nodes_grpc_connections is not per store type and external labels. thanos_store_node_info is marked as deprecated. * Added (optional, but enabled by default) compatibility label to store GW Info method with flag to disable it. Store only adds it if there is any labelset to advertise. * Disabled any strict deduplication in external labels detection; just warning! * More tests; compatibility tests for query pre 0.8.0 and new store GW. * Remove compatibility label if spotted. Fixes #1632 Signed-off-by: Bartek Plotka <[email protected]> * Changelog fixes (#1634) Signed-off-by: Bartek Plotka <[email protected]>
Fixed version is released https://github.com/thanos-io/thanos/releases/tag/v0.8.1 🎉 Thanks all for help in this 🎖️ |
Working great tyvm! |
… blocked for older Queriers. (#1636) Changes: * Cleaned up the code a bit. * Ensured proper locks. * thanos_store_nodes_grpc_connections is not per store type and external labels. thanos_store_node_info is marked as deprecated. * Added (optional, but enabled by default) compatibility label to store GW Info method with flag to disable it. Store only adds it if there is any labelset to advertise. * Disabled any strict deduplication in external labels detection; just warning! * More tests; compatibility tests for query pre 0.8.0 and new store GW. * Remove compatibility label if spotted. Fixes #1632 Signed-off-by: Bartek Plotka <[email protected]> Signed-off-by: Giedrius Statkevičius <[email protected]>
Thanos, Prometheus and Golang version used: 0.8.0, 2.13.0, 1.13.1
Object Storage Provider: Azure
What happened: 0.7.0 I used the "replica" label with values "a" and b". I used --query.replica-label=replica in my Thanos Query nodes. Now, if I put the Store and Sidecar as stores in the same Thanos Query instance, one or the other gets dropped for labels as not unique.
What you expected to happen: Uniqueness of labels only be enforced for the same Thanos type (store versus store, sidecar versus sidecar).
How to reproduce it (as minimally and precisely as possible):
Full logs to relevant components:
{"address":"thanos-query-app-swarm-sidecars:10903","caller":"storeset.go:261","component":"storeset","duplicates":2,"extLset":"{monitor=\"apps-app-swarm\",region=\"useast2\",replica=\"a\"},{monitor=\"apps-app-swarm\",region=\"useast2\",replica=\"b\"},{monitor=\"infra-app-swarm\",region=\"useast2\",replica=\"a\"},{monitor=\"infra-app-swarm\",region=\"useast2\",replica=\"b\"},{monitor=\"services-app-swarm\",region=\"useast2\",replica=\"a\"},{monitor=\"services-app-swarm\",region=\"useast2\",replica=\"b\"}","level":"warn","msg":"dropping store, external labels are not unique","ts":"2019-10-11T06:33:29.563785804Z"}
Anything else we need to know: Again this worked in 0.7.0. I did notice in the UI that the label sets for my Stores used to be blank, and now they are not. If I had to guess, the "glitch" got "fixed" there and caused a collision, but I am not seeing the "new way" to ensure that my given Thanos Store for a given Thanos Sidecar "replica" remains uniquely labelled.
The text was updated successfully, but these errors were encountered: