Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Provides further details on aggregations in datafeeds #55462

Merged
merged 4 commits into from
Apr 22, 2020
Merged

[DOCS] Provides further details on aggregations in datafeeds #55462

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9a6590d
[DOCS] Provides further details on aggregations in datafeeds.
szabosteve Apr 20, 2020
5462456
[DOCS] Fixes section ID.
szabosteve Apr 20, 2020
363579d
Apply suggestions from code review
szabosteve Apr 21, 2020
ed2ed7b
[DOCS] Addresses feedback.
szabosteve Apr 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 51 additions & 16 deletions docs/reference/ml/anomaly-detection/aggregations.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ TIP: If you use a terms aggregation and the cardinality of a term is high, the
aggregation might not be effective and you might want to just use the default
search and scroll behavior.

[discrete]
[[aggs-limits-dfeeds]]
==== Requirements and limitations

There are some limitations to using aggregations in {dfeeds}. Your aggregation
must include a `date_histogram` aggregation, which in turn must contain a `max`
aggregation on the time field. This requirement ensures that the aggregated data
Expand All @@ -25,7 +29,14 @@ You must also consider the interval of the date histogram aggregation carefully.
The bucket span of your {anomaly-job} must be divisible by the value of the
`calendar_interval` or `fixed_interval` in your aggregation (with no remainder).
If you specify a `frequency` for your {dfeed}, it must also be divisible by this
interval.
interval. You cannot feed an {anomaly-job} with a date histogram with an
szabosteve marked this conversation as resolved.
Show resolved Hide resolved
interval measured in months because the length of the month is not fixed.
{dfeeds-cap} tolerate weeks or smaller units.

IMPORTANT: The name of the aggregation and the name of the field that the agg
szabosteve marked this conversation as resolved.
Show resolved Hide resolved
operates on need to mach, otherwise the aggregation doesn't work. For example,
szabosteve marked this conversation as resolved.
Show resolved Hide resolved
if you use a `max` aggregation on a time field called `responsetime`, the name
of the aggregation must be also `responsetime`.

TIP: As a rule of thumb, if your detectors use <<ml-metric-functions,metric>> or
<<ml-sum-functions,sum>> analytical functions, set the date histogram
Expand All @@ -34,6 +45,11 @@ finer, more granular time buckets, which are ideal for this type of analysis. If
your detectors use <<ml-count-functions,count>> or <<ml-rare-functions,rare>>
functions, set the interval to the same value as the bucket span.


[discrete]
[[aggs-include-jobs]]
==== Including aggregations in {anomaly-jobs}

When you create or update an {anomaly-job}, you can include the names of
aggregations, for example:

Expand Down Expand Up @@ -85,13 +101,13 @@ PUT _ml/datafeeds/datafeed-farequote
"time": { <1>
"max": {"field": "time"}
},
"airline": { <1>
"airline": { <2>
"terms": {
"field": "airline",
"size": 100
},
"aggregations": {
"responsetime": { <1>
"responsetime": { <3>
"avg": {
"field": "responsetime"
}
Expand All @@ -107,15 +123,25 @@ PUT _ml/datafeeds/datafeed-farequote

<1> In this example, the aggregations have names that match the fields that they
operate on. That is to say, the `max` aggregation is named `time` and its
field is also `time`. The same is true for the aggregations with the names
`airline` and `responsetime`.
field also needs to be `time`.
<2> In this example, the aggregations have names that match the fields that they
szabosteve marked this conversation as resolved.
Show resolved Hide resolved
operate on. That is to say, the `term` aggregation is named `airline` and its
field also needs to be `airline`.
<3> In this example, the aggregations have names that match the fields that they
operate on. That is to say, the `avg` aggregation is named `responsetime` and
its field also needs to be `responsetime`.

Your {dfeed} can contain multiple aggregations, but only the ones with names
that match values in the job configuration are fed to the job.

IMPORTANT: Your {dfeed} can contain multiple aggregations, but only the ones
with names that match values in the job configuration are fed to the job.

{dfeeds-cap} support complex nested aggregations, this example uses the `derivative`
pipeline aggregation to find the first order derivative of the counter
`system.network.out.bytes` for each value of the field `beat.name`.
[discrete]
[[aggs-dfeeds]]
==== Nested aggregations in {dfeeds}

{dfeeds-cap} support complex nested aggregations, this example uses the
szabosteve marked this conversation as resolved.
Show resolved Hide resolved
`derivative` pipeline aggregation to find the first order derivative of the
counter `system.network.out.bytes` for each value of the field `beat.name`.

[source,js]
----------------------------------
Expand Down Expand Up @@ -154,6 +180,11 @@ pipeline aggregation to find the first order derivative of the counter
----------------------------------
// NOTCONSOLE


[discrete]
[[aggs-single-dfeeds]]
==== Single bucket aggregations in {dfeeds}

{dfeeds-cap} not only supports multi-bucket aggregations, but also single bucket
aggregations. The following shows two `filter` aggregations, each gathering the
number of unique entries for the `error` field.
Expand Down Expand Up @@ -201,6 +232,11 @@ number of unique entries for the `error` field.
----------------------------------
// NOTCONSOLE


[discrete]
[[aggs-define-dfeeds]]
==== Defining aggregations in {dfeeds}

When you define an aggregation in a {dfeed}, it must have the following form:

[source,js]
Expand Down Expand Up @@ -239,7 +275,7 @@ When you define an aggregation in a {dfeed}, it must have the following form:
The top level aggregation must be either a
{ref}/search-aggregations-bucket.html[bucket aggregation] containing as single
sub-aggregation that is a `date_histogram` or the top level aggregation is the
required `date_histogram`. There must be exactly one `date_histogram`
required `date_histogram`. There must be exactly one `date_histogram`
aggregation. For more information, see
{ref}/search-aggregations-bucket-datehistogram-aggregation.html[Date histogram aggregation].

Expand All @@ -248,9 +284,9 @@ NOTE: The `time_zone` parameter in the date histogram aggregation must be set to

Each histogram bucket has a key, which is the bucket start time. This key cannot
be used for aggregations in {dfeeds}, however, because they need to know the
time of the latest record within a bucket. Otherwise, when you restart a {dfeed},
it continues from the start time of the histogram bucket and possibly fetches
the same data twice. The max aggregation for the time field is therefore
time of the latest record within a bucket. Otherwise, when you restart a
{dfeed}, it continues from the start time of the histogram bucket and possibly
fetches the same data twice. The max aggregation for the time field is therefore
necessary to provide the time of the latest record within a bucket.

You can optionally specify a terms aggregation, which creates buckets for
Expand Down Expand Up @@ -280,8 +316,7 @@ GET .../_search {
By default, {es} limits the maximum number of terms returned to 10000. For high
cardinality fields, the query might not run. It might return errors related to
circuit breaking exceptions that indicate that the data is too large. In such
cases, do not use aggregations in your {dfeed}. For more
information, see
cases, do not use aggregations in your {dfeed}. For more information, see
{ref}/search-aggregations-bucket-terms-aggregation.html[Terms aggregation].

You can also optionally specify multiple sub-aggregations. The sub-aggregations
Expand Down