From 9a6590d1a7ddf463de92cbcc05271e37f4965991 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Istv=C3=A1n=20Zolt=C3=A1n=20Szab=C3=B3?= Date: Mon, 20 Apr 2020 15:24:02 +0200 Subject: [PATCH 1/4] [DOCS] Provides further details on aggregations in datafeeds. --- .../anomaly-detection/aggregations.asciidoc | 67 ++++++++++++++----- 1 file changed, 51 insertions(+), 16 deletions(-) diff --git a/docs/reference/ml/anomaly-detection/aggregations.asciidoc b/docs/reference/ml/anomaly-detection/aggregations.asciidoc index 1243983de5670..8009c462bf544 100644 --- a/docs/reference/ml/anomaly-detection/aggregations.asciidoc +++ b/docs/reference/ml/anomaly-detection/aggregations.asciidoc @@ -15,6 +15,10 @@ TIP: If you use a terms aggregation and the cardinality of a term is high, the aggregation might not be effective and you might want to just use the default search and scroll behavior. +[discrete] +[[aggs-limits-dfeeds]] +==== Requirements and limitations + There are some limitations to using aggregations in {dfeeds}. Your aggregation must include a `date_histogram` aggregation, which in turn must contain a `max` aggregation on the time field. This requirement ensures that the aggregated data @@ -25,7 +29,14 @@ You must also consider the interval of the date histogram aggregation carefully. The bucket span of your {anomaly-job} must be divisible by the value of the `calendar_interval` or `fixed_interval` in your aggregation (with no remainder). If you specify a `frequency` for your {dfeed}, it must also be divisible by this -interval. +interval. You cannot feed an {anomaly-job} with a date histogram with an +interval measured in months because the length of the month is not fixed. +{dfeeds-cap} tolerate weeks or smaller units. + +IMPORTANT: The name of the aggregation and the name of the field that the agg +operates on need to mach, otherwise the aggregation doesn't work. For example, +if you use a `max` aggregation on a time field called `responsetime`, the name +of the aggregation must be also `responsetime`. TIP: As a rule of thumb, if your detectors use <> or <> analytical functions, set the date histogram @@ -34,6 +45,11 @@ finer, more granular time buckets, which are ideal for this type of analysis. If your detectors use <> or <> functions, set the interval to the same value as the bucket span. + +[discrete] +[[aggs-limits-dfeeds]] +==== Including aggregations in {anomaly-jobs} + When you create or update an {anomaly-job}, you can include the names of aggregations, for example: @@ -85,13 +101,13 @@ PUT _ml/datafeeds/datafeed-farequote "time": { <1> "max": {"field": "time"} }, - "airline": { <1> + "airline": { <2> "terms": { "field": "airline", "size": 100 }, "aggregations": { - "responsetime": { <1> + "responsetime": { <3> "avg": { "field": "responsetime" } @@ -107,15 +123,25 @@ PUT _ml/datafeeds/datafeed-farequote <1> In this example, the aggregations have names that match the fields that they operate on. That is to say, the `max` aggregation is named `time` and its -field is also `time`. The same is true for the aggregations with the names -`airline` and `responsetime`. +field also needs to be `time`. +<2> In this example, the aggregations have names that match the fields that they +operate on. That is to say, the `term` aggregation is named `airline` and its +field also needs to be `airline`. +<3> In this example, the aggregations have names that match the fields that they +operate on. That is to say, the `avg` aggregation is named `responsetime` and +its field also needs to be `responsetime`. + +Your {dfeed} can contain multiple aggregations, but only the ones with names +that match values in the job configuration are fed to the job. -IMPORTANT: Your {dfeed} can contain multiple aggregations, but only the ones -with names that match values in the job configuration are fed to the job. -{dfeeds-cap} support complex nested aggregations, this example uses the `derivative` -pipeline aggregation to find the first order derivative of the counter -`system.network.out.bytes` for each value of the field `beat.name`. +[discrete] +[[aggs-dfeeds]] +==== Nested aggregations in {dfeeds} + +{dfeeds-cap} support complex nested aggregations, this example uses the +`derivative` pipeline aggregation to find the first order derivative of the +counter `system.network.out.bytes` for each value of the field `beat.name`. [source,js] ---------------------------------- @@ -154,6 +180,11 @@ pipeline aggregation to find the first order derivative of the counter ---------------------------------- // NOTCONSOLE + +[discrete] +[[aggs-single-dfeeds]] +==== Single bucket aggregations in {dfeeds} + {dfeeds-cap} not only supports multi-bucket aggregations, but also single bucket aggregations. The following shows two `filter` aggregations, each gathering the number of unique entries for the `error` field. @@ -201,6 +232,11 @@ number of unique entries for the `error` field. ---------------------------------- // NOTCONSOLE + +[discrete] +[[aggs-define-dfeeds]] +==== Defining aggregations in {dfeeds} + When you define an aggregation in a {dfeed}, it must have the following form: [source,js] @@ -239,7 +275,7 @@ When you define an aggregation in a {dfeed}, it must have the following form: The top level aggregation must be either a {ref}/search-aggregations-bucket.html[bucket aggregation] containing as single sub-aggregation that is a `date_histogram` or the top level aggregation is the -required `date_histogram`. There must be exactly one `date_histogram` +required `date_histogram`. There must be exactly one `date_histogram` aggregation. For more information, see {ref}/search-aggregations-bucket-datehistogram-aggregation.html[Date histogram aggregation]. @@ -248,9 +284,9 @@ NOTE: The `time_zone` parameter in the date histogram aggregation must be set to Each histogram bucket has a key, which is the bucket start time. This key cannot be used for aggregations in {dfeeds}, however, because they need to know the -time of the latest record within a bucket. Otherwise, when you restart a {dfeed}, -it continues from the start time of the histogram bucket and possibly fetches -the same data twice. The max aggregation for the time field is therefore +time of the latest record within a bucket. Otherwise, when you restart a +{dfeed}, it continues from the start time of the histogram bucket and possibly +fetches the same data twice. The max aggregation for the time field is therefore necessary to provide the time of the latest record within a bucket. You can optionally specify a terms aggregation, which creates buckets for @@ -280,8 +316,7 @@ GET .../_search { By default, {es} limits the maximum number of terms returned to 10000. For high cardinality fields, the query might not run. It might return errors related to circuit breaking exceptions that indicate that the data is too large. In such -cases, do not use aggregations in your {dfeed}. For more -information, see +cases, do not use aggregations in your {dfeed}. For more information, see {ref}/search-aggregations-bucket-terms-aggregation.html[Terms aggregation]. You can also optionally specify multiple sub-aggregations. The sub-aggregations From 5462456d1c3f0cee7835c0cf6e399b50d2778b0d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Istv=C3=A1n=20Zolt=C3=A1n=20Szab=C3=B3?= Date: Mon, 20 Apr 2020 15:41:21 +0200 Subject: [PATCH 2/4] [DOCS] Fixes section ID. --- docs/reference/ml/anomaly-detection/aggregations.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/ml/anomaly-detection/aggregations.asciidoc b/docs/reference/ml/anomaly-detection/aggregations.asciidoc index 8009c462bf544..dc82dcf479878 100644 --- a/docs/reference/ml/anomaly-detection/aggregations.asciidoc +++ b/docs/reference/ml/anomaly-detection/aggregations.asciidoc @@ -47,7 +47,7 @@ functions, set the interval to the same value as the bucket span. [discrete] -[[aggs-limits-dfeeds]] +[[aggs-include-jobs]] ==== Including aggregations in {anomaly-jobs} When you create or update an {anomaly-job}, you can include the names of From 363579d1531acc7f3d4343d998ab052e8c141627 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Istv=C3=A1n=20Zolt=C3=A1n=20Szab=C3=B3?= Date: Tue, 21 Apr 2020 08:54:29 +0200 Subject: [PATCH 3/4] Apply suggestions from code review Co-Authored-By: Lisa Cawley --- docs/reference/ml/anomaly-detection/aggregations.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/ml/anomaly-detection/aggregations.asciidoc b/docs/reference/ml/anomaly-detection/aggregations.asciidoc index dc82dcf479878..c7c9469d063cd 100644 --- a/docs/reference/ml/anomaly-detection/aggregations.asciidoc +++ b/docs/reference/ml/anomaly-detection/aggregations.asciidoc @@ -34,7 +34,7 @@ interval measured in months because the length of the month is not fixed. {dfeeds-cap} tolerate weeks or smaller units. IMPORTANT: The name of the aggregation and the name of the field that the agg -operates on need to mach, otherwise the aggregation doesn't work. For example, +operates on need to match, otherwise the aggregation doesn't work. For example, if you use a `max` aggregation on a time field called `responsetime`, the name of the aggregation must be also `responsetime`. @@ -139,7 +139,7 @@ that match values in the job configuration are fed to the job. [[aggs-dfeeds]] ==== Nested aggregations in {dfeeds} -{dfeeds-cap} support complex nested aggregations, this example uses the +{dfeeds-cap} support complex nested aggregations. This example uses the `derivative` pipeline aggregation to find the first order derivative of the counter `system.network.out.bytes` for each value of the field `beat.name`. From ed2ed7b2c1b1420d8002eb59cdce1bf0ce89a8ef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Istv=C3=A1n=20Zolt=C3=A1n=20Szab=C3=B3?= Date: Tue, 21 Apr 2020 09:01:48 +0200 Subject: [PATCH 4/4] [DOCS] Addresses feedback. --- .../anomaly-detection/aggregations.asciidoc | 26 +++++++++---------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/docs/reference/ml/anomaly-detection/aggregations.asciidoc b/docs/reference/ml/anomaly-detection/aggregations.asciidoc index c7c9469d063cd..3c5cf5508b944 100644 --- a/docs/reference/ml/anomaly-detection/aggregations.asciidoc +++ b/docs/reference/ml/anomaly-detection/aggregations.asciidoc @@ -25,19 +25,19 @@ aggregation on the time field. This requirement ensures that the aggregated data is a time series and the timestamp of each bucket is the time of the last record in the bucket. -You must also consider the interval of the date histogram aggregation carefully. -The bucket span of your {anomaly-job} must be divisible by the value of the -`calendar_interval` or `fixed_interval` in your aggregation (with no remainder). -If you specify a `frequency` for your {dfeed}, it must also be divisible by this -interval. You cannot feed an {anomaly-job} with a date histogram with an -interval measured in months because the length of the month is not fixed. -{dfeeds-cap} tolerate weeks or smaller units. - IMPORTANT: The name of the aggregation and the name of the field that the agg operates on need to match, otherwise the aggregation doesn't work. For example, if you use a `max` aggregation on a time field called `responsetime`, the name of the aggregation must be also `responsetime`. +You must also consider the interval of the date histogram aggregation carefully. +The bucket span of your {anomaly-job} must be divisible by the value of the +`calendar_interval` or `fixed_interval` in your aggregation (with no remainder). +If you specify a `frequency` for your {dfeed}, it must also be divisible by this +interval. {anomaly-jobs-cap} cannot use date histograms with an interval +measured in months because the length of the month is not fixed. {dfeeds-cap} +tolerate weeks or smaller units. + TIP: As a rule of thumb, if your detectors use <> or <> analytical functions, set the date histogram aggregation interval to a tenth of the bucket span. This suggestion creates @@ -124,12 +124,10 @@ PUT _ml/datafeeds/datafeed-farequote <1> In this example, the aggregations have names that match the fields that they operate on. That is to say, the `max` aggregation is named `time` and its field also needs to be `time`. -<2> In this example, the aggregations have names that match the fields that they -operate on. That is to say, the `term` aggregation is named `airline` and its -field also needs to be `airline`. -<3> In this example, the aggregations have names that match the fields that they -operate on. That is to say, the `avg` aggregation is named `responsetime` and -its field also needs to be `responsetime`. +<2> Likewise, the `term` aggregation is named `airline` and its field is also +named `airline`. +<3> Likewise, the `avg` aggregation is named `responsetime` and its field is +also named `responsetime`. Your {dfeed} can contain multiple aggregations, but only the ones with names that match values in the job configuration are fed to the job.