Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] CircuitBreakingException in qa:google-cloud-storage "10_repository/Register a repository with a non existing client" test #32050

Closed
cbuescher opened this issue Jul 13, 2018 · 3 comments · Fixed by #32074
Closed

[CI] CircuitBreakingException in qa:google-cloud-storage "10_repository/Register a repository with a non existing client" test #32050

cbuescher opened this issue Jul 13, 2018 · 3 comments · Fixed by #32074
Labels
:Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload >test-failure Triaged test failures from CI

Comments

@cbuescher
Copy link
Member

Failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/6050/console

Error:

java.lang.AssertionError: Failure at [repository_gcs/10_repository:191]: the error message was expected to match the provided regex but didn't
Expected: repository_exception
     but: was "{root_cause=[{type=circuit_breaking_exception, reason=[parent] Data too large, data for [<http_request>] would be [510357400/486.7mb], which is larger than the limit of [510027366/486.3mb], bytes_wanted=510357400, bytes_limit=510027366, stack_trace=CircuitBreakingException[[parent] Data too large, data for [<http_request>] would be [510357400/486.7mb], which is larger than the limit of [510027366/486.3mb]]\n\tat org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:257)\n\tat org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128)\n\tat org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:231)\n\tat org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)\n\tat org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:173)\n\tat 
[...]

I couldn't reproduce locally so far:

REPRODUCE WITH: ./gradlew :plugins:repository-gcs:qa:google-cloud-storage:integTestRunner \
  -Dtests.seed=76E4624F4840A694 \
  -Dtests.class=org.elasticsearch.repositories.gcs.GoogleCloudStorageRepositoryClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=repository_gcs/10_repository/Register a repository with a non existing client}" \
  -Dtests.security.manager=true \
  -Dtests.jvm.argline="-XX:-UseConcMarkSweepGC -XX:+UseG1GC" \
  -Dtests.locale=he \
  -Dtests.timezone=Etc/GMT-5
@cbuescher cbuescher added >test-failure Triaged test failures from CI :Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload labels Jul 13, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra

@cbuescher
Copy link
Member Author

@danielmitterdorfer could this be somehow related to #31767? Just a wild guess because I saw this was just merged recently.

@danielmitterdorfer
Copy link
Member

It is quite likely caused by #31767. I also understand why this does not reproduce easily because the circuit breaker trips based on current memory usage which we cannot 100% control in tests.

@danielmitterdorfer danielmitterdorfer self-assigned this Jul 16, 2018
danielmitterdorfer added a commit to danielmitterdorfer/elasticsearch that referenced this issue Jul 16, 2018
@danielmitterdorfer
Turn off real-mem breaker in REST tests
656a326 
With this commit we disable the real-memory circuit breaker in REST
tests as this breaker is based on real memory usage over which we have
no (full) control in tests and the REST client is not yet ready to retry
on circuit breaker exceptions.

This is only meant as a temporary measure to avoid spurious test
failures while we ensure that the REST client can handle those
situations appropriately.

Closes elastic#32050
Relates elastic#31767
Relates elastic#31986
@danielmitterdorfer danielmitterdorfer mentioned this issue Jul 16, 2018
Merged
danielmitterdorfer added a commit that referenced this issue Jul 16, 2018
@danielmitterdorfer
Turn off real-mem breaker in REST tests
e0cfa16 
With this commit we disable the real-memory circuit breaker in REST
tests as this breaker is based on real memory usage over which we have
no (full) control in tests and the REST client is not yet ready to retry
on circuit breaker exceptions.

This is only meant as a temporary measure to avoid spurious test
failures while we ensure that the REST client can handle those
situations appropriately.

Closes #32050
Relates #31767
Relates #31986 
Relates #32074
@danielmitterdorfer danielmitterdorfer removed their assignment Jul 16, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Turn off real-mem breaker in REST tests danielmitterdorfer/elasticsearch
3 participants
@danielmitterdorfer @cbuescher @elasticmachine