server panic from math query

[codyaray]

Here's the query executed in the CLI

> SELECT processed_rules / rules_alerted FROM "sentinel_run" WHERE time > now() - 30d

And the server panics with the following before the goroutine stack traces:

[query] 2015/10/29 12:31:24 SELECT processed_rules / rules_alerted FROM "sentinel"."default".sentinel_run WHERE time > now() - 30d
panic: runtime error: index out of range

This dataset is fairly reproducible (just parsed a log file into metrics). I can try to simplify and share the text file and parser if you need it, but will need permission from the company first.

[codyaray]

Possibly unrelated issue.

On some queries, I'm getting a strange error:

"ERR: invalid character 'j' looking for beginning of value"

One theory is that this is produced when you divide by zero. I have two series, both of which sum to zero during the last day.

> SELECT sum("catchall_alerted"), sum("processed_catchall") FROM sentinel_run WHERE time > now() - 1d
name: sentinel_run
------------------
time                sum sum
2015-10-28T18:37:12.268614503Z  0   0

When I divide them, I receive this error:

> SELECT sum("catchall_alerted") / sum("processed_catchall") FROM sentinel_run WHERE time > now() - 1d
ERR: invalid character 'j' looking for beginning of value

I can reproduce it by dividing by zero directly (scaling by zero, in Graphite terminology):

SELECT sum("catchall_alerted") / 0 FROM sentinel_run WHERE time > now() - 1d
ERR: invalid character 'j' looking for beginning of value

And this sum works fine dividing by one (sanity check):

SELECT sum("catchall_alerted") / 1 FROM sentinel_run WHERE time > now() - 1d
name: sentinel_run
------------------
time
2015-10-28T18:06:07.580551548Z    0

Unfortunately, sometimes dividing by zero doesn't reproduce it. But notice it also excludes the second column entirely in this case:

> SELECT sum("catchall_processed") / 0 FROM sentinel_run WHERE time > now() - 1d
name: sentinel_run
------------------
time
2015-10-28T18:39:14.081565231Z

Maybe helpful, maybe noise. :)

[beckettsean]

@codyaray the initial error is a duplicate of #4046. There's a known issue with math on multiple fields.

The invalid character 'j'... error looks like a known issue if your data source passes NaN values: #4104. The write path has been fixed in 0.9.5 to prevent NaN points from persisting, but I'm not aware of a fix for existing data that shows the issue.

Is there a possibility of NaN values in your data?

[pauldix]

Closing this for now since it seems fixed in current versions. Let us know if it's still an issue.