Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate OOM during functional tests #97378

Closed
tylersmalley opened this issue Apr 16, 2021 · 5 comments · Fixed by #97876
Closed

Investigate OOM during functional tests #97378

tylersmalley opened this issue Apr 16, 2021 · 5 comments · Fixed by #97876
Labels
Team:Operations Team label for Operations Team

Comments

@tylersmalley
Copy link
Contributor

tylersmalley commented Apr 16, 2021 •
edited
Loading

https://kibana-ci.elastic.co/job/elastic+kibana+master/13377/execution/node/706/log/

09:05:17                 │ERROR fatal error in thread [elasticsearch[kibana-ci-immutable-ubuntu-16-tests-xxl-1618585333397510879][system_read][T#2]], exiting
09:05:17                 │      
09:05:17                 │ERROR java.lang.OutOfMemoryError: Java heap space
09:05:17                 │      	at java.base/java.util.Arrays.copyOfRange(Arrays.java:3821)
09:05:17                 │      	at java.base/java.lang.String.<init>(String.java:397)
09:05:17                 │      	at java.base/java.util.Base64$Encoder.encodeToString(Base64.java:346)
09:05:17                 │      	at org.elasticsearch.xpack.core.async.AsyncTaskIndexService.encodeResponse(AsyncTaskIndexService.java:462)
09:05:17                 │      	at org.elasticsearch.xpack.core.async.AsyncTaskIndexService.updateResponse(AsyncTaskIndexService.java:205)
09:05:17                 │      	at org.elasticsearch.xpack.search.TransportSubmitAsyncSearchAction.onFinalResponse(TransportSubmitAsyncSearchAction.java:178)
@botelastic botelastic bot added the needs-team Issues missing a team label label Apr 16, 2021
tylersmalley pushed a commit that referenced this issue Apr 16, 2021
Skip test to try and stabilize master
194355f 
#97378

Signed-off-by: Tyler Smalley <[email protected]>
tylersmalley pushed a commit that referenced this issue Apr 16, 2021
Skip test to try and stabilize master
948847c 
#97378

Signed-off-by: Tyler Smalley <[email protected]>
@tylersmalley
Copy link
Contributor Author

master: 194355f
7.x 948847c

@flash1293 flash1293 added Team:Operations Team label for Operations Team and removed needs-team Issues missing a team label labels Apr 19, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations (Team:Operations)

@flash1293
Copy link
Contributor

Not sure about the root cause, but these test failures are also caused by OOMs: #97387

@spalger
Copy link
Contributor

spalger commented Apr 19, 2021

We saw a wide range of tests fail with OOMs over the weekend, we'd skip a test and then another test would start failing. It doesn't seem like the tests themselves are related to the errors directly.

@spalger
Copy link
Contributor

spalger commented Apr 21, 2021

Should be fixed by #97813, I'll try unskipping these tests today.

spalger added a commit to spalger/kibana that referenced this issue Apr 21, 2021
@spalger
Revert "Skip test to try and stabilize master" (elastic#97378)
ecfce31 
This reverts commit 194355f.
@spalger spalger mentioned this issue Apr 21, 2021
Merged
spalger added a commit that referenced this issue Apr 22, 2021
@spalger @kibanamachine
Revert skips added while triaging ES OOMs (#97876)
6bb2893 
* Revert "skip flaky suite (#97382)"

This reverts commit e321f57.

* Revert "skip flaky suite (#97387)"

This reverts commit a89b756.

* Revert "Skip test to try and stabilize master" (#97378)

This reverts commit 194355f.

* upload heap dumps when they are created

Co-authored-by: spalger <[email protected]>
Co-authored-by: Kibana Machine <[email protected]>
kibanamachine added a commit to kibanamachine/kibana that referenced this issue Apr 22, 2021
@spalger @kibanamachine
Revert skips added while triaging ES OOMs (elastic#97876)
d5999e1 
* Revert "skip flaky suite (elastic#97382)"

This reverts commit e321f57.

* Revert "skip flaky suite (elastic#97387)"

This reverts commit a89b756.

* Revert "Skip test to try and stabilize master" (elastic#97378)

This reverts commit 194355f.

* upload heap dumps when they are created

Co-authored-by: spalger <[email protected]>
Co-authored-by: Kibana Machine <[email protected]>
kibanamachine added a commit to kibanamachine/kibana that referenced this issue Apr 22, 2021
@spalger @kibanamachine
Revert skips added while triaging ES OOMs (elastic#97876)
2fb0d8d 
* Revert "skip flaky suite (elastic#97382)"

This reverts commit e321f57.

* Revert "skip flaky suite (elastic#97387)"

This reverts commit a89b756.

* Revert "Skip test to try and stabilize master" (elastic#97378)

This reverts commit 194355f.

* upload heap dumps when they are created

Co-authored-by: spalger <[email protected]>
Co-authored-by: Kibana Machine <[email protected]>
kibanamachine added a commit that referenced this issue Apr 22, 2021
@kibanamachine @spalger
Revert skips added while triaging ES OOMs (#97876) (#98066)
84046ed 
* Revert "skip flaky suite (#97382)"

This reverts commit e321f57.

* Revert "skip flaky suite (#97387)"

This reverts commit a89b756.

* Revert "Skip test to try and stabilize master" (#97378)

This reverts commit 194355f.

* upload heap dumps when they are created

Co-authored-by: spalger <[email protected]>
Co-authored-by: Kibana Machine <[email protected]>

Co-authored-by: Spencer <[email protected]>
Co-authored-by: spalger <[email protected]>
kibanamachine added a commit that referenced this issue Apr 22, 2021
@kibanamachine @spalger
Revert skips added while triaging ES OOMs (#97876) (#98067)
08c9152 
* Revert "skip flaky suite (#97382)"

This reverts commit e321f57.

* Revert "skip flaky suite (#97387)"

This reverts commit a89b756.

* Revert "Skip test to try and stabilize master" (#97378)

This reverts commit 194355f.

* upload heap dumps when they are created

Co-authored-by: spalger <[email protected]>
Co-authored-by: Kibana Machine <[email protected]>

Co-authored-by: Spencer <[email protected]>
Co-authored-by: spalger <[email protected]>
@MadameSheema MadameSheema mentioned this issue Apr 27, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Team:Operations Team label for Operations Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Revert skips added while triaging ES OOMs spalger/kibana
4 participants
@tylersmalley @spalger @flash1293 @elasticmachine