Vizualize a custom aggregation

[savva-khalaman]

Hello,

I may duplicate some previous requests (#1331). Still, is there a way to visualize a custom metric aggregation besides the predefined ones (e.g. count, sum, average, etc)?

Specifically, I would like define a scripted_metric aggregation that computes precision/recall statistics and is represented with a bar chart. It would be nice, if there would be an option to define a JSON of a custom aggregation and Kibana would use it as is.

Is it possible? Am I missing something?

Thank you,
Savva

[rashidkpc]

Currently not possible, filing as an enhancement request.

[savva-khalaman]

Rashid, could you please elaborate more on why it is not possible right now? Does this functionality require changes in the logic of the embedded web-server? Or update of the UI and the way request is build may be sufficient?
In the last case, I can try to implement it. Your advise where to start will be very valuable.

[ssoto]

Let me explain my UC.
I have some information related with an "primary key", suppose it is the hostname that sends the data. We define hostname like server_name field.
That server_name field is not indexed, but I can get it ussing related ip address that it is indexed.
I need agregate information ussing server_name custom field.
Is possible do that in Kibana4?
Regards

[lcs777]

+1

[baselsalam]

+1

In our use case, a P0 scenario is metrics that are "weighted".
Example: Success rate = "sum(success)" / "sum(attempts)".

We got this working via an ElasticSearch "scripted_metric".
However there is no way to express this in Kibana, even by hand editing the JS code under visState. (Edit visualization Object).

A nice UI friendly editor for scripted_metric can be quite complicated,
but is there at least a way to specify it in the JS code for the visualization?

** Example code for sum(a)/sum(b): **

1. With dynamic scripting enabled:

  "scripted_metric": {
    "init_script": "_agg[\"numerator\"] = []; _agg[\"denominator\"] = [];"
    "map_script": "_agg.numerator << (doc[\"a\"]); _agg.denominator = << (doc[\"b\"].value);"
    "combine_script": "num_sum=0; den_sum=0; for (t in _agg.numerator) { num_sum+= t}; for (t in _agg.denominator) { den_sum += t }; return [num_sum: num_sum, den_sum: den_sum];"
    "reduce_script": "num_sum = 0; den_sum=0; for (a in _aggs) { num_sum += a.num_sum; den_sum += a.den_sum; }; return (num_sum/den_sum);"

2. With dynamic scripting disabled:

• The 4 scripts referenced are saved in ES's config/scripts, they're basically the contents of the script fields above, scripts as above without the beginning/ending doublequotes, and the backslashes escaping the inline doublequotes.

  "scripted_metric": {
    "init_script_file": "dividesums_init"
    "map_script": "dividesums_map"
    "combine_script": "dividesums_combine"
    "reduce_script": "dividesums_reduce"

What I simply need is for Kibana to eventually generate this type of agg in the query,
which does succeed in ES:

      "aggs": {
        "my_dividesums_metric": {
          "scripted_metric": {
            "init_script_file": "reliability_init",
            "map_script_file": "reliability_map",
            "combine_script_file": "reliability_combine",
            "reduce_script_file": "reliability_reduce"
          }
        }
      }

I've tried putting this in the "scripted field" textbox, but there's an error indicating what's expected is purely the value of "script", example:
"script": "<< this goes in the textbox for scripted field>>"

Perhaps if there's a different type of scripted field where the value you're putting is the contents of "scripted_field" ? Example:
"scripted_metric": "<< this goes in the textbox for scripted metric>>"

[baselsalam]

Ah, digging through the code I can see why this isn't working even after editing the visState:
There is no "scripted_metric" under agg_types/metrics.

So, when I modify the JS to the following, I get "Unable to get property 'params' of undefined or null reference".

    {
      "id": "1",
      "type": "scripted_metric",
      "schema": "metric",
      "params": {
        "init_script_file": "reliability_init",
        "map_script_file": "reliability_map",
        "combine_script_file": "reliability_combine",
        "reduce_script_file": "reliability_reduce"
      }
    },

[keikha]

I'm wondering if this is possible through JSON input in Kibana 4. @rashidkpc do you have any recommendation on that? Can we use a scripted metric in the JSON input section of the metric?

[pmoust]

I am interested in this as well

[JPriya2]

+1

[mseneby]

+1

[prateek05]

+1

[justinfx]

👍

[AlexIoannides]

+1

[dubrowgn]

Any progress on this one? Currently, it seems to be impossible to do aggregations of pre-aggregated data because of this limitation. As an example, historical telemetry data can be represented as number of requests (reqs) and the average request duration (time). To properly aggregate this data in Kibana, we need to be able to perform something like the following: sum(reqs * time) / sum(reqs). I can aggregate either side of the division, but see no way to do simple calculations on those aggregations (division in this case).

Edit: related #4584

[ramanenka]

+1

[tbragin]

I think this is a duplicate of "scripted metric aggregation" #2646. Since that issue has way more comments, closing this one in favor of the other one. Please feel free to transfer your comment there!

To answer some questions in this thread, if you'd like to do time-series math, such as

sum(reqs * time) / sum(reqs)

you may consider trying out the Timelion Kibana plugin: https://github.com/elastic/timelion