Skip to content

Commit

Permalink
Add a MovingFunction pipeline aggregation, deprecate MovingAvg agg (e…
Browse files Browse the repository at this point in the history
…lastic#29594)

This pipeline aggregation gives the user the ability to script functions that "move" across a window
of data, instead of single data points.  It is the scripted version of MovingAvg pipeline agg.

Through custom script contexts, we expose a number of convenience methods:

 - MovingFunctions.max()
 - MovingFunctions.min()
 - MovingFunctions.sum()
 - MovingFunctions.unweightedAvg()
 - MovingFunctions.linearWeightedAvg()
 - MovingFunctions.ewma()
 - MovingFunctions.holt()
 - MovingFunctions.holtWinters()
 - MovingFunctions.stdDev()

The user can also define any arbitrary logic via their own scripting, or combine with the above methods.
  • Loading branch information
polyfractal authored and ywelsch committed May 23, 2018
1 parent b754092 commit b3c21a6
Show file tree
Hide file tree
Showing 31 changed files with 2,876 additions and 177 deletions.
3 changes: 3 additions & 0 deletions docs/reference/aggregations/pipeline.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ POST /_search
}
--------------------------------------------------
// CONSOLE
// TEST[warning:The moving_avg aggregation has been deprecated in favor of the moving_fn aggregation.]
<1> The metric is called `"the_sum"`
<2> The `buckets_path` refers to the metric via a relative path `"the_sum"`

Expand Down Expand Up @@ -136,6 +137,7 @@ POST /_search
}
--------------------------------------------------
// CONSOLE
// TEST[warning:The moving_avg aggregation has been deprecated in favor of the moving_fn aggregation.]
<1> By using `_count` instead of a metric name, we can calculate the moving average of document counts in the histogram

The `buckets_path` can also use `"_bucket_count"` and path to a multi-bucket aggregation to use the number of buckets
Expand Down Expand Up @@ -231,6 +233,7 @@ include::pipeline/stats-bucket-aggregation.asciidoc[]
include::pipeline/extended-stats-bucket-aggregation.asciidoc[]
include::pipeline/percentiles-bucket-aggregation.asciidoc[]
include::pipeline/movavg-aggregation.asciidoc[]
include::pipeline/movfn-aggregation.asciidoc[]
include::pipeline/cumulative-sum-aggregation.asciidoc[]
include::pipeline/bucket-script-aggregation.asciidoc[]
include::pipeline/bucket-selector-aggregation.asciidoc[]
Expand Down
15 changes: 13 additions & 2 deletions docs/reference/aggregations/pipeline/movavg-aggregation.asciidoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
[[search-aggregations-pipeline-movavg-aggregation]]
=== Moving Average Aggregation

deprecated[6.4.0, The Moving Average aggregation has been deprecated in favor of the more general
<<search-aggregations-pipeline-movfn-aggregation,Moving Function Aggregation>>. The new Moving Function aggregation provides
all the same functionality as the Moving Average aggregation, but also provides more flexibility.]

Given an ordered series of data, the Moving Average aggregation will slide a window across the data and emit the average
value of that window. For example, given the data `[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]`, we can calculate a simple moving
average with windows size of `5` as follows:
Expand Down Expand Up @@ -74,6 +78,7 @@ POST /_search
--------------------------------------------------
// CONSOLE
// TEST[setup:sales]
// TEST[warning:The moving_avg aggregation has been deprecated in favor of the moving_fn aggregation.]

<1> A `date_histogram` named "my_date_histo" is constructed on the "timestamp" field, with one-day intervals
<2> A `sum` metric is used to calculate the sum of a field. This could be any metric (sum, min, max, etc)
Expand Down Expand Up @@ -180,6 +185,7 @@ POST /_search
--------------------------------------------------
// CONSOLE
// TEST[setup:sales]
// TEST[warning:The moving_avg aggregation has been deprecated in favor of the moving_fn aggregation.]

A `simple` model has no special settings to configure

Expand Down Expand Up @@ -233,6 +239,7 @@ POST /_search
--------------------------------------------------
// CONSOLE
// TEST[setup:sales]
// TEST[warning:The moving_avg aggregation has been deprecated in favor of the moving_fn aggregation.]

A `linear` model has no special settings to configure

Expand Down Expand Up @@ -295,7 +302,7 @@ POST /_search
--------------------------------------------------
// CONSOLE
// TEST[setup:sales]

// TEST[warning:The moving_avg aggregation has been deprecated in favor of the moving_fn aggregation.]

[[single_0.2alpha]]
.EWMA with window of size 10, alpha = 0.2
Expand Down Expand Up @@ -355,6 +362,7 @@ POST /_search
--------------------------------------------------
// CONSOLE
// TEST[setup:sales]
// TEST[warning:The moving_avg aggregation has been deprecated in favor of the moving_fn aggregation.]

In practice, the `alpha` value behaves very similarly in `holt` as `ewma`: small values produce more smoothing
and more lag, while larger values produce closer tracking and less lag. The value of `beta` is often difficult
Expand Down Expand Up @@ -446,7 +454,7 @@ POST /_search
--------------------------------------------------
// CONSOLE
// TEST[setup:sales]

// TEST[warning:The moving_avg aggregation has been deprecated in favor of the moving_fn aggregation.]

[[holt_winters_add]]
.Holt-Winters moving average with window of size 120, alpha = 0.5, beta = 0.7, gamma = 0.3, period = 30
Expand Down Expand Up @@ -508,6 +516,7 @@ POST /_search
--------------------------------------------------
// CONSOLE
// TEST[setup:sales]
// TEST[warning:The moving_avg aggregation has been deprecated in favor of the moving_fn aggregation.]

==== Prediction

Expand Down Expand Up @@ -550,6 +559,7 @@ POST /_search
--------------------------------------------------
// CONSOLE
// TEST[setup:sales]
// TEST[warning:The moving_avg aggregation has been deprecated in favor of the moving_fn aggregation.]

The `simple`, `linear` and `ewma` models all produce "flat" predictions: they essentially converge on the mean
of the last value in the series, producing a flat:
Expand Down Expand Up @@ -631,6 +641,7 @@ POST /_search
--------------------------------------------------
// CONSOLE
// TEST[setup:sales]
// TEST[warning:The moving_avg aggregation has been deprecated in favor of the moving_fn aggregation.]

<1> Minimization is enabled with the `minimize` parameter

Expand Down
Loading

0 comments on commit b3c21a6

Please sign in to comment.