SOLR-16968: The MemoryCircuitBreaker now uses average heap usage #1905
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
janhoy
commented
Sep 8, 2023
solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java
Outdated
Show resolved
Hide resolved
cpoerschke
reviewed
Sep 8, 2023
solr/core/src/test/org/apache/solr/util/BaseTestCircuitBreaker.java
Outdated
Show resolved
Hide resolved
solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java
Outdated
Show resolved
Hide resolved
solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java
Outdated
Show resolved
Hide resolved
solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerRegistry.java
Outdated
Show resolved
Hide resolved
solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerRegistry.java
Outdated
Show resolved
Hide resolved
solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java
Outdated
Show resolved
Hide resolved
solr/core/src/java/org/apache/solr/util/circuitbreaker/AveragingMetricProvider.java
Show resolved
Hide resolved
solr/core/src/java/org/apache/solr/util/circuitbreaker/AveragingMetricProvider.java
Outdated
Show resolved
Hide resolved
|
@atris feel free to weigh in here. The intention is to get fewer "false positive" request rejections due to short spikes. Now there will be some delay (0-30s) from heavy memory pressure starts until the breaker starts tripping, and likewise some time before we allow requests again after cooling down. So if the client implements some kind of exponential backoff logic on receipt of HTTP 429 errors, the system will stabilize right under the memory threshold. If they don't, it will be an oscillating pattern on and off, but hopefully give Solr time to recover in between. Do you, perhaps from experience, see any argument for a shorter reaction time? |
janhoy
force-pushed
the
SOLR-16968-Averaging-Memory-Circuit-Breaker
branch
from
9ed1662 to
8d67701
Compare
sigram
reviewed
Sep 10, 2023
solr/core/src/java/org/apache/solr/util/circuitbreaker/AveragingMetricProvider.java
Show resolved
Hide resolved
janhoy
force-pushed
the
SOLR-16968-Averaging-Memory-Circuit-Breaker
branch
from
7728cf0 to
18968f1
Compare
|
Planning to merge this on friday... |
janhoy
added a commit
that referenced
this pull request
Sep 22, 2023
(cherry picked from commit 8ca4a5d)
justinrsweeney
pushed a commit
to cowpaths/fullstory-solr
that referenced
this pull request
Nov 2, 2023
justinrsweeney
pushed a commit
to cowpaths/fullstory-solr
that referenced
this pull request
Apr 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/SOLR-16968
Introduced a new utility class
AveragingMetricProviderthat will cache the average for adoublevalue provided by the breaker. Uses a scheduled executor every 5 seconds, which is shared by all instances of this class. This same utility can be used by other CBs to achieve the same, but I started withMemoryCircuitBreaker.As this requires some cleanup of executor threads, CircuitBreaker now implements
Closeableand the registry will close all CBs on close, which is also called fromCoreContainer#doClose. I also added some synchonization on the map, which isn't really necessary in normal runtime, but I figured useful for tests.NOTE that this may conflict with other in-flight CB PRs...