[UI] Hide unhealthy stores after some time

[metalmatze]

Thanos, Prometheus and Golang version used
This was on Kubernetes running Thanos v0.2.1.

What happened
I was looking at the Stores UI and saw a lot of unhealthy stores even though some were gone for 192+ hours.

What you expected to happen
I don't really care about stores that are reported as unhealthy after, I would say, 24 hours.

How to reproduce it (as minimally and precisely as possible):
Not actually sure how to reproduce, probably deploy a new store every now and then, but keep at least one store for persisting the gossip state?!

Full logs to relevant components

Anything else we need to know
I would be happy to take a look at this and try to fix it, if you could give me a guideline on what type of semantics the filtering should happen on.

/cc @bwplotka

[adrien-f]

Mmm I'm not sure about this one. I tried to mimic Prometheus' Targets page when doing this. Does it removes unhealthy targets about a while ?

If those stores are supposed to be gone, shouldn't the Query config be updated with a fresher list of stores ?

[bwplotka]

It removes, (hopefully), but seems like they are still on page.

The main problem is that we indentify those StoreAPIs by IP and it's quite wrong. As pod restarts the IP is different but it's actually same storeAPI (: maybe that's a bug?

[ipstatic]

@bwplotka Did another PR resolve this?

[bwplotka]

Sorry I should link the fix (thought it was linked before): #910

[bwplotka]

It's fixed on master, yes, thanks to @adrien-f

[ipstatic]

Great!