Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

StackOverflowError somewhere in the cluster applier thread #36360

Closed
DaveCTurner opened this issue Dec 7, 2018 · 3 comments
Closed

StackOverflowError somewhere in the cluster applier thread #36360

DaveCTurner opened this issue Dec 7, 2018 · 3 comments
Assignees
@dimitris-athanasiou
Labels
>bug :Core/Infra/Resiliency Keep running when everything is ok. Die quickly if things go horribly wrong. :ml Machine learning v6.4.2

Comments

@DaveCTurner
Copy link
Contributor

DaveCTurner commented Dec 7, 2018 •
edited by jasontedor
Loading

A user reported that their master node died with the following exception

[2018-11-16T09:59:05,744][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [REDACTED] fatal error in thread [elasticsearch[REDACTED][clusterApplierService#updateTask][T#1]], exiting
java.lang.StackOverflowError: null
        at java.util.Collections$UnmodifiableList$1.<init>(Collections.java:1339) ~[?:1.8.0_111]
        at java.util.Collections$UnmodifiableList.listIterator(Collections.java:1337) ~[?:1.8.0_111]
        at java.util.Collections$UnmodifiableList$1.<init>(Collections.java:1339) ~[?:1.8.0_111]
        at java.util.Collections$UnmodifiableList.listIterator(Collections.java:1337) ~[?:1.8.0_111]

This repeats until the end of the file. As far as I can tell, this can't be an infinite recursion, but it could be very deeply nested if an unmodifiable list is repeatedly rewrapped.

It is possible this was introduced by #31957 which looks like it might re-wraps some lists in unmodifiableList each time a new DatafeedConfig is constructed.
@DaveCTurner DaveCTurner added >bug :Core/Infra/Resiliency Keep running when everything is ok. Die quickly if things go horribly wrong. :ml Machine learning v6.4.2 labels Dec 7, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra

@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core

@dimitris-athanasiou dimitris-athanasiou self-assigned this Dec 7, 2018
@dimitris-athanasiou
Copy link
Contributor

Ouch. That is horrible. I'll work on a fix for this asap.

dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this issue Dec 7, 2018
@dimitris-athanasiou
[ML] Prevent stack overflow while copying ML jobs and datafeeds
bdb7d58 
ML jobs and datafeeds wrap collections into their unmodifiable
equivalents in their constructor. However, the copying builder
does not make a copy of some of those collections resulting
in wrapping those again and again. This can eventually result
to stack overflow.

This commit addressed this issue by copying the collections in
question in the copying builder constructor.

Closes elastic#36360
@dimitris-athanasiou dimitris-athanasiou mentioned this issue Dec 7, 2018
Merged
dimitris-athanasiou added a commit that referenced this issue Dec 7, 2018
@dimitris-athanasiou
[ML] Prevent stack overflow while copying ML jobs and datafeeds (#36370)
7870ae8 
ML jobs and datafeeds wrap collections into their unmodifiable
equivalents in their constructor. However, the copying builder
does not make a copy of some of those collections resulting
in wrapping those again and again. This can eventually result
to stack overflow.

This commit addressed this issue by copying the collections in
question in the copying builder constructor.

Closes #36360
dimitris-athanasiou added a commit that referenced this issue Dec 7, 2018
@dimitris-athanasiou
[ML] Prevent stack overflow while copying ML jobs and datafeeds (#36370)
915cc1b 
ML jobs and datafeeds wrap collections into their unmodifiable
equivalents in their constructor. However, the copying builder
does not make a copy of some of those collections resulting
in wrapping those again and again. This can eventually result
to stack overflow.

This commit addressed this issue by copying the collections in
question in the copying builder constructor.

Closes #36360
dimitris-athanasiou added a commit that referenced this issue Dec 7, 2018
@dimitris-athanasiou
[ML] Prevent stack overflow while copying ML jobs and datafeeds (#36370)
ea7c8d4 
ML jobs and datafeeds wrap collections into their unmodifiable
equivalents in their constructor. However, the copying builder
does not make a copy of some of those collections resulting
in wrapping those again and again. This can eventually result
to stack overflow.

This commit addressed this issue by copying the collections in
question in the copying builder constructor.

Closes #36360
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dimitris-athanasiou dimitris-athanasiou

Labels
>bug :Core/Infra/Resiliency Keep running when everything is ok. Die quickly if things go horribly wrong. :ml Machine learning v6.4.2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@DaveCTurner @dimitris-athanasiou @elasticmachine