Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Transform] Regression: continuous transform fails after start with [composite] failed to parse field [sources] #60125

Closed
hendrikmuhs opened this issue Jul 23, 2020 · 1 comment · Fixed by #60196 or #60276
Closed

[Transform] Regression: continuous transform fails after start with [composite] failed to parse field [sources] #60125

hendrikmuhs opened this issue Jul 23, 2020 · 1 comment · Fixed by #60196 or #60276
Assignees
@hendrikmuhs
Labels
blocker >bug :ml/Transform Transform v7.9.0

Comments

@hendrikmuhs
Copy link

Creating a continuous transform with 1 histogram or date histogram succeeds but fails after _start. This is a regression introduced by #55287.

Repro:

PUT _transform/histogram-pivot
{
  "source": {
    "index": "kibana_sample_data_logs"
  },
  "dest": {
    "index": "kibana_sample_data_logs-pivot"
  },
  "sync": {
    "time": {
      "field": "timestamp"
    }
  }, 
  "pivot": {
    "group_by": {
      "h": {
        "histogram": {
          "field": "bytes",
          "interval": 50
          }
      }
    },
    "aggregations": {
      "max": {
        "max": {
          "field": "bytes"
        }
      }
    }
  }
}

failure:

org.elasticsearch.common.xcontent.XContentParseException: [1:13] [composite] failed to parse field [sources]
        at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:531) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ObjectParser.parseArray(ObjectParser.java:523) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:555) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:324) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:171) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.xpack.transform.transforms.pivot.Pivot.createCompositeAggregationSources(Pivot.java:361) ~[?:?]
        at org.elasticsearch.xpack.transform.transforms.pivot.Pivot.buildChangeCollector(Pivot.java:219) ~[?:?]
        at org.elasticsearch.xpack.transform.transforms.TransformIndexer.initializeFunction(TransformIndexer.java:349) ~[?:?]
        at org.elasticsearch.xpack.transform.transforms.TransformIndexer.lambda$onStart$10(TransformIndexer.java:249) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.xpack.transform.transforms.TransformIndexer.lambda$onStart$11(TransformIndexer.java:300) [transform-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.xpack.transform.persistence.IndexBasedTransformConfigManager.parseTransformLenientlyFromSource(IndexBasedTransformConfigManager.java:666) [transform-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.xpack.transform.persistence.IndexBasedTransformConfigManager.lambda$getTransformConfiguration$7(IndexBasedTransformConfigManager.java:355) [transform-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:83) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:545) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:117) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:350) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:344) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:231) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:119) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:125) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:47) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:95) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:710) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: org.elasticsearch.common.xcontent.XContentParseException: Failed to build [composite] after last required field arrived
        at org.elasticsearch.common.xcontent.ConstructingObjectParser$Target.buildTarget(ConstructingObjectParser.java:534) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ConstructingObjectParser$Target.constructorArg(ConstructingObjectParser.java:460) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ConstructingObjectParser$Target.access$900(ConstructingObjectParser.java:408) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ConstructingObjectParser.lambda$declareField$3(ConstructingObjectParser.java:220) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareField$9(ObjectParser.java:386) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:529) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        ... 33 more
Caused by: java.lang.IllegalArgumentException: Composite [sources] cannot be null or empty
        at org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregationBuilder.validateSources(CompositeAggregationBuilder.java:177) ~[elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregationBuilder.<init>(CompositeAggregationBuilder.java:70) ~[elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregationBuilder.lambda$static$0(CompositeAggregationBuilder.java:55) ~[elasticsearch-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ConstructingObjectParser$Target.buildTarget(ConstructingObjectParser.java:521) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ConstructingObjectParser$Target.constructorArg(ConstructingObjectParser.java:460) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ConstructingObjectParser$Target.access$900(ConstructingObjectParser.java:408) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ConstructingObjectParser.lambda$declareField$3(ConstructingObjectParser.java:220) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareField$9(ObjectParser.java:386) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:529) ~[elasticsearch-x-content-7.9.0-SNAPSHOT.jar:7.9.0-SNAPSHOT]
        ... 33 more
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml/Transform)

@hendrikmuhs hendrikmuhs self-assigned this Jul 23, 2020
hendrikmuhs pushed a commit to hendrikmuhs/elasticsearch that referenced this issue Jul 27, 2020
do not create change collector if group_by configuration does not sup…
23aaf6b 
…port

change detection

fixes elastic#60125
@hendrikmuhs hendrikmuhs mentioned this issue Jul 27, 2020
Merged
hendrikmuhs pushed a commit that referenced this issue Jul 27, 2020
[Transform] Fix Regression: continuous transform can fail for (date) …
6398f73 
…histogram group_by(#60196)

do not create change collector if group_by configuration does not support change detection

fixes #60125
hendrikmuhs pushed a commit that referenced this issue Jul 27, 2020
[Transform] Fix Regression: continuous transform can fail for (date) …
95c99ca 
…histogram group_by(#60196)

do not create change collector if group_by configuration does not support change detection

fixes #60125
hendrikmuhs pushed a commit that referenced this issue Jul 27, 2020
[Transform] Fix Regression: continuous transform can fail for (date) …
e184759 
…histogram group_by(#60196)

do not create change collector if group_by configuration does not support change detection

fixes #60125
hendrikmuhs pushed a commit to hendrikmuhs/elasticsearch that referenced this issue Jul 28, 2020
re-work elastic#60196, to not skip building change collectors as othe…
115a11a 
…rwise

date histogram only pivots would run slow

relates elastic#60125
@hendrikmuhs hendrikmuhs mentioned this issue Jul 28, 2020
Merged
hendrikmuhs pushed a commit that referenced this issue Jul 29, 2020
[Transform] fix performance regression introduced in #60196 (#60276)
2b9b4a9 
re-work #60196, to not skip building change collectors as otherwise date histogram only
pivots would run slow

relates #60125
hendrikmuhs pushed a commit that referenced this issue Jul 29, 2020
[Transform] fix performance regression introduced in #60196 (#60276)
5eb04fb 
re-work #60196, to not skip building change collectors as otherwise date histogram only
pivots would run slow

relates #60125
hendrikmuhs pushed a commit that referenced this issue Jul 29, 2020
[Transform] fix performance regression introduced in #60196 (#60276)
579a186 
re-work #60196, to not skip building change collectors as otherwise date histogram only
pivots would run slow

relates #60125
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hendrikmuhs hendrikmuhs

Labels
blocker >bug :ml/Transform Transform v7.9.0
Projects
None yet
Milestone
No milestone
2 participants
@hendrikmuhs @elasticmachine