store:/api/v1/label/<name>/values matchers not work

[R4scal]

Hello. I encountered the problem described in #5469

Today we upgraded grafana and queries label_values() moved from /api/v1/series to /api/v1/label/<label>/values.

Now we got all label values for all queries, matchers not wok. For example:

New api

curl "http://thanos-query/api/v1/label/job/values?match%5B%5D=up" | jq '.data | length'
295

curl "http://thanos-query/api/v1/label/job/values?match%5B%5D=test9" | jq '.data | length'
295

Old api

curl "http://thanos-query/api/v1/series?match%5B%5D=up" | jq '.data | length'
23172

curl "http://thanos-query/api/v1/series?match%5B%5D=test9" | jq '.data | length'
0

Thanos: v0.32.5
Prometheus: v.2.45.1

[douglascamata]

@R4scal can you share you whole datasource configuration? Also, please try setting the type of "Prometheus" behind it to Thanos.

[R4scal]

The problem not in grafana, direct queries to prometheus works fine.

curl "http://prometheus/api/v1/label/job/values?match%5B%5D=up" | jq '.data | length
8

curl "http://prometheus/api/v1/label/job/values?match%5B%5D=test9" | jq '.data | length
0

looking for func (qapi *QueryAPI) labelValues

[douglascamata]

How can I know what is curl "http://prometheus/api/v1/label/job/values?match%5B%5D=test9" | jq '.data | length supposed to return? I don't what data you have, so from my point of view 0 could be correct and not correct. 🤷

So, please, include more information. Tell us about the data that you have, what you expect, and what you get.

[douglascamata]

Note that the different APIs are doing VERY different things.

curl "http://thanos-query/api/v1/label/job/values?match%5B%5D=up" | jq '.data | length' is giving you the unique values of the label job for the up series.

curl "http://thanos-query/api/v1/series?match%5B%5D=up" | jq '.data | length' is giving you all the series with name up.

Do you see the difference?

[R4scal]

The main problem that query /api/v1/label/job/values on thanos ignores match. I tried to describe this in first message.
up valid metric
test9 - metric that doesn't exist

I should get 0 for a non-existent metric. But /api/v1/label/job/values return non-zero (all labels) for any match

[MichaHoffmann]

I can reproduce this for external labels and have prepared a fix. I suppose up is not an external label though; for internal labels my quick test setup worked properly; can you explain your topology somewhat please so we have better idea where to look? Additionally; are you able to share a block, that would greatly help!

[MichaHoffmann]

https://github.com/thanos-io/thanos/pull/6879/files here the fix for external labels; but probably not related to this issue

[MichaHoffmann]

At least for sidecar and internal labels, my query setup produces the expected results:

$ curl -sq "http://localhost:10904/api/v1/label/service/values?match%5B%5D=xxx" | jq '.data '
[]
 fedora  ~ 
$ curl -sq "http://localhost:10904/api/v1/label/service/values?match%5B%5D=foo" | jq '.data '
[
  "onair-cassandra-dc1-all-pods-service"
]

but it might be a very specific setup ( from debugging another issue, i just had it around still )

[R4scal]

I have complex topology:

1. query-frontend (central)
2. query (central)
3. query (per az)
4. store and sidecar

My issue not for external labels

[MichaHoffmann]

Can you test the curl for the other queries too? Is it maybe related to the query frontend?

[R4scal]

tested. All queries affected

[MichaHoffmann]

Ok, thank you for verifying. Sice it works for me with sidecar and query just joins all returned labels, it must be related to bucket store i think. ( Which is weird; we have acceptance tests for this in principle ).

[R4scal]

I tested without store (only sidecar) and it works fine.

[R4scal]

Looks like it store issue :(

[MichaHoffmann]

Good stuff, thank you!

[MichaHoffmann]

Can you share information about the store instance, which version, which config, etc?

[R4scal]

I found an old v0.19.0 store that produce labels. Trying to fix this

[R4scal]

I can confirm issue with v0.19.0. With v0.32.5 all works.

[MichaHoffmann]

Thanks for confirming! I'll close the issue then!

[R4scal]

Thanks for help!

[Jakob3xD]

Can we please reopen the issue.

This issue is still present for me in v0.35.5 and I noticed it due to grafana update. grafana/grafana#78043
We only run v0.35.5 and some rulers running v0.35.2. I did not found any other version in our setup.
Our setup is also multi cluster/ multi region with top level querier and bottom querier.

I expect to only get the label value "fsn1" but instead I get all regions.

curl api.com/api/v1/label/region/values?match[]=opensearch_cluster_status{monitor="opensearch", cluster="fsn1-dc14-opensearch1"}&start=1701065544&end=1701076344

{
  "status":"success",
  "data":["ash","fsn1","hel1","hil","nbg1"]
}

Query Range with the same query and time range. I reduced the json output to keep it simple. There are some more values returned and some more nodes that all have similar labels labels like in the example below. Only the instance and exported_instance changes.

curl api.com/api/v1/query_range?query=opensearch_cluster_status{monitor%3D"opensearch"%2C cluster%3D"fsn1-dc14-opensearch1"}&start=1701065544&end=1701076344&step=30s

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {
          "__name__": "opensearch_cluster_status",
          "cluster": "fsn1-dc14-opensearch1",
          "exported_instance": "172.27.9.1:9200",
          "exported_job": "opensearch",
          "instance": "fsn1-dc14-opensearch1-x1.<someurl>",
          "job": "internal",
          "monitor": "opensearch",
          "region": "fsn1"
        },
        "values": [
          [
            1701065520,
            "0"
          ]
        ]
      },
      {
        "metric": {
          "__name__": "opensearch_cluster_status",
          "cluster": "fsn1-dc14-opensearch1",
          "exported_instance": "172.27.9.2:9200",
          "exported_job": "opensearch",
          "instance": "fsn1-dc14-opensearch1-x1.<someurl>",
          "job": "internal",
          "monitor": "opensearch",
          "region": "fsn1"
        },
        "values": [
          [
            1701065520,
            "0"
          ]
        ]
      },
      {
        "metric": {
          "__name__": "opensearch_cluster_status",
          "cluster": "fsn1-dc14-opensearch1",
          "exported_instance": "172.27.9.3:9200",
          "exported_job": "opensearch",
          "instance": "fsn1-dc14-opensearch1-x2.<someurl>",
          "job": "internal",
          "monitor": "opensearch",
          "region": "fsn1"
        },
        "values": [
          [
            1701065520,
            "0"
          ]
        ]
      },
      {
        "metric": {
          "__name__": "opensearch_cluster_status",
          "cluster": "fsn1-dc14-opensearch1",
          "exported_instance": "172.27.9.4:9200",
          "exported_job": "opensearch",
          "instance": "fsn1-dc14-opensearch1-x2.<someurl>",
          "job": "internal",
          "monitor": "opensearch",
          "region": "fsn1"
        },
        "values": [
          [
            1701065520,
            "0"
          ]
        ]
      }
    ]
  }
}

[MichaHoffmann]

@Jakob3xD can you describe your setup please?

[Jakob3xD]

@MichaHoffmann Following is a graphical version of our setup.
We have several clusters. Each cluster has a Thanos Frontend for caching, with a to level Thanos Query. The top level query has all sub level queries of all clusters as target so we can query metrics from every cluster for every cluster. The sub level query has several sidecars and stores as target. This is all running withing k8s clusters using Prometheus Operator and some logic outside of it for scaling. region and monitor are external_labels.

[MichaHoffmann]

I think i fixed the issue for external labels here https://github.com/thanos-io/thanos/pull/6879/files; your rulers are not upgraded and the issue was in sidecar and tsdb store ( rulers use tsdb store i think ); can you upgrade rulers and retry please?

[MichaHoffmann]

Oh wait that was never merged; let me ping on the PR

[GiedriusS]

Closing as dup of #6959 which was closed now. It was fixed by https://github.com/thanos-io/thanos/pull/6879/files.