index out of range [-1] #2645
Labels
Comments
|
Is there some kind of stack trace printed with the error, perhaps in the Thanos Query logs or somewhere else? Where is this printed in general? Because these don't seem similar to our log format. Do you have a way of reproducing it? 🤔 It's probably impossible to tell where is a (possible) error without a stack trace. |
|
Hey @GiedriusS , |
|
|
These are the logs from the cortex-query-frontend for those queries. |
|
And this is the log from the associated |
|
I couldnt find any error logs on the |
|
That's definitely very useful! Would it be possible for you to check this with |
|
|
I did the same as ipyker - I added a new Luckily when I remove |
|
That is super helpful feedback, we will take a closer look tomorrow,
thanks! (:
…On Sun, 31 May 2020 at 02:12, Sevag Hanssian ***@***.***> wrote:
I did the same as ipyker - removed a previous --query.replica-label,
restarted the Thanos Query, and this started occuring.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#2645 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABVA3O7PYSEQS77WJWZ633LRUGVHNANCNFSM4NH56CGQ>
.
|
|
Hey @bwplotka , Would setting up a query pod with none of the |
|
On it, now |
|
Ok, got it. The problem is that Querier just have problems with more than 2 replica labels. Fixing. Thanks for spotting this and quickly describing issue ❤️ |
|
That makes sense - I have 2 replica labels, and encountered this when adding a 3rd. Thanks for the quick turnaround. |
bwplotka
added a commit
that referenced
this issue
Jun 5, 2020
…d panic seen when using larger number of replicas and small series. Fixes #2645 Signed-off-by: Bartlomiej Plotka <[email protected]>
|
Silly missed edge case fixed here: #2728 |
bwplotka
added a commit
that referenced
this issue
Jun 5, 2020
…d panic seen when using larger number of replicas and small series. Fixes #2645 Signed-off-by: Bartlomiej Plotka <[email protected]>
bwplotka
added a commit
that referenced
this issue
Jun 10, 2020
…d panic seen when using larger number of replicas and small series. Fixes #2645 Signed-off-by: Bartlomiej Plotka <[email protected]>
bwplotka
added a commit
that referenced
this issue
Jun 10, 2020
…d panic seen when using larger number of replicas and small series. Fixes #2645 Signed-off-by: Bartlomiej Plotka <[email protected]>
bwplotka
added a commit
that referenced
this issue
Jun 10, 2020
…d panic seen when using larger number of replicas and small series. Fixes #2645 Signed-off-by: Bartlomiej Plotka <[email protected]>
bwplotka
added a commit
that referenced
this issue
Jun 10, 2020
…d panic seen when using larger number of replicas and small series. (#2728) Fixes #2645 Signed-off-by: Bartlomiej Plotka <[email protected]>
|
Will this make it into 0.13? |
|
Not really, but it is available on master right now (we build images for
master). We run the current master on production now, so I would recommend
upgrading at least querier if you are interested in this fix. Also next
release is pretty soon (3 weeks?)
TL;DR the bug was bit more complex. It was panicking for all requests that
return series with less or same number labels vs number of replica labels
(:
…On Wed, 10 Jun 2020 at 15:40, Sevag Hanssian ***@***.***> wrote:
Will this make it into 0.13?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#2645 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABVA3O6FN4N3P373E7VSTVTRV6LPTANCNFSM4NH56CGQ>
.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thanos Version used: v0.12.2
Object Storage Provider: Ceph S3
What happened: When I run queries against historic dates, for certain dates we get a response
unexpected error: runtime error: index out of range [-1]{'field': 'subscribed_cores', 'date': datetime.date(2019, 12, 27), 'runtime': 1.5100421905517578, 'text': '{"status":"error","errorType":"execution","error":"unexpected error: runtime error: index out of range [-1]"}\n', 'value': None, 'status': 'error', 'error': 'unexpected error: runtime error: index out of range [-1]'}
Other than that we also sometimes get an empty response if not the error but never the actual value.
{'field': 'subscribed_cores', 'date': datetime.date(2019, 12, 27), 'runtime': 1.227376699447632, 'text': '{"status":"success","data":{"resultType":"vector","result":[]}}\n', 'value': None, 'status': 'success', 'error': None}
What you expected to happen: Get the value for that date.
CC: @andmatt
The text was updated successfully, but these errors were encountered: