api/v1/series returns wrong series with certain queries

[arnaud-mey]

After setting up the cluster and ingesting data from prometheus we are facing a problem with the series API.
curl 'http://localhost:7201/api/v1/series?match[]=up\{\}'
is a never ending stream of json (we stopped at 8GB)
curl 'http://localhost:7201/api/v1/series?match[]=up\{job="kube-state-metrics"\}'
just returns a single data object (as expected). Grafana tries to utilize this API to get a list of labels (label_values()).
We are using m3dbnode v0.7.3 with m3operator v0.1.4.
Is this a bug or are we doing anything wrong? Our configuration is pretty much that fo the documentation. If you need any more infos from us, please let us know. Thanks in advance.

[richardartoul]

@arnikola @benraskin92 Can one of you chime in here? Is this related to the new API we're building or is this a separate issue

[evanhuang996]

I think it's related to #1453

[evanhuang996]

@arnaud-mey If you want a quick fix, just change the query path to grafana -> prometheus->m3query or m3coordinator->m3db

[arnaud-mey]

Thanks for your reply. We put a Prometheus in front of the m3coordinator using the remote read api. That seems to work quite well.

[arnikola]

Yeah, it's the same issue as #1453; although the root cause of the issue you're seeing is a bit different, but it should be fixed in #1481

[robskillington]

Now that 0.8.0 is released, can we verify this behavior is no longer an issue and close this?

[evanhuang996]

It's working well now.

[arnikola]

Great to hear, thanks 👍

[arnikola]

@arnaud-mey we would recommend not putting a prom in front of coordinator; you're likely to get better performance connecting graphana directly to query now that 0.8.0 is landed

[arnaud-mey]

yeah we tested 0.8.0 by now and we get the expected behavior now :)

thanks a lot!