Cumilative Sum doesn't work with other pipeline aggregations

[simianhacker]

Cumulative sum aggregation returns an error when using it with other pipeline aggregations. This happens consistently with both bucket_scripts and derivatives.

Steps to Reproduce:

1. Index some Metricbeat data
2. Run the following query:

GET metricbeat-*/_search
{
  "size": 0,
  "query": {
    "range": {
      "@timestamp": {
        "gte": "now-15m/m",
        "lte": "now"
      }
    }
  },
  "aggs": {
    "timeseries": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "10s"
      },
      "aggs": {
        "maxTX": {
          "max": {
            "field": "system.network.out.bytes"
          }
        },
        "calculation": {
          "bucket_script": {
            "buckets_path": {
              "maxTX": "maxTX"
            }, 
            "script": { 
              "source": "params.maxTX" 
            }
          }
        },
        "cumsum": {
          "cumulative_sum": {
            "buckets_path": "calculation"
          }
        }
      }
    }
  }
}

3. Receive following response:

{
  "error": {
    "root_cause": [],
    "type": "search_phase_execution_exception",
    "reason": "",
    "phase": "fetch",
    "grouped": true,
    "failed_shards": [],
    "caused_by": {
      "type": "null_pointer_exception",
      "reason": null
    }
  },
  "status": 503
}

[jasontedor]

@simianhacker The server logs should contain a stack trace for that null pointer exception. Do you have it?

[simianhacker]

Woops... forgot to add that... here you go:

[2017-11-27T16:06:27,676][WARN ][r.suppressed             ] path: /infra-*/_search, params: {index=infra-*}
org.elasticsearch.action.search.SearchPhaseExecutionException:
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:272) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:92) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:622) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.run(TimedRunnable.java:41) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.lang.NullPointerException
        at org.elasticsearch.search.aggregations.pipeline.cumulativesum.CumulativeSumPipelineAggregator.reduce(CumulativeSumPipelineAggregator.java:82) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.search.aggregations.InternalAggregation.reduce(InternalAggregation.java:123) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:77) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:523) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:500) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:417) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.action.search.SearchPhaseController$1.reduce(SearchPhaseController.java:736) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:102) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:45) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:87) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        ... 4 more

[jasontedor]

Thanks. It looks like an NPE from unboxing a null, although that appears to only arise if an invalid aggregation path exception was swallowed previously.

Can you take a look @colings86?

[colings86]

I think this occurs because for an empty bucket (a bucket where the doc count is 0), the bucket_script is skipped and doesn't output anything for the bucket. This means that when the cumulative sum aggregator try to access the bucket script value for that bucket it can't find it (hence the invalid aggregation path) and null is returned. This issue relates to #27377 which is also caused because the value is not output in the bucket when the doc count is 0.

[polyfractal]

@elastic/es-search-aggs