Add field data memory circuit breaker #4261

dakrone · 2013-11-26T20:29:21Z

This adds the field data circuit breaker, which is used to estimate
the amount of memory required to load field data before loading it. It
then raises a CircuitBreakingException if the limit is exceeded.

It is configured with two parameters:

indices.fielddata.cache.breaker.limit - the maximum number of bytes
of field data to be loaded before circuit breaking. Defaults to
indices.fielddata.cache.size if set, unbounded otherwise.

indices.fielddata.cache.breaker.overhead - a contast for all field
data estimations to be multiplied with before aggregation. Defaults to
1.03.

Both settings can be configured dynamically using the cluster update
settings API.

s1monw · 2013-11-26T20:39:24Z

Cool stuff @dakrone

jpountz · 2013-12-06T22:14:13Z

docs/reference/cluster/nodes-stats.asciidoc

@@ -60,6 +60,9 @@ By default, `indices` stats are returned. With options for `indices`,
 	Transport statistics about sent and received bytes in
 	cluster communication

+`breaker`::


I think breaker is too generic a name?

How about circuit-breaker?

dakrone · 2013-12-06T22:30:42Z

I realized that the FieldDataEstimator class is no longer needed, as estimations have been moved into their respective field data loading classes, so I'll remove it.

jpountz · 2013-12-06T22:31:46Z

src/main/java/org/elasticsearch/index/fielddata/plain/PackedArrayIndexFieldData.java

+                double estimatedBytes = ((RamAccountingTermsEnum)termsEnum).getTotalBytes();
+                breaker.addWithoutBreaking(-(long)((estimatedBytes * breaker.getOverhead()) - actualUsed));
+            } else {
+                logger.warn("Trying to adjust circuit breaker, but TermsEnum has not been wrapped!");


Should we have an assertion here?

I think an assertion makes sense, I'll do that instead of the if statement.

dakrone · 2013-12-11T21:07:13Z

Updated code and force-pushed another squashed commit (because there were going to be merge conflicts regardless, and I'd rather rebase and deal with them now rather than after reviews).

Changes:

Move all the files into better/more-applicable packages
Breaker stats are now under the key fielddata_breaker and the class is called FieldDataBreakerStats
Use constants instead of strings for reused field data filter settings
TermsEnum is wrapped in a filter if BlockTreeStats can't be used (for people using custom postings formats)
Fix "unwinding" of breaker in the event a different exception occurs while loading field data
De-interface-ify MemoryAggregatingCircuitBreaker to become concrete MemoryCircuitBreaker
Logger passed through to MemoryCircuitBreaker to preserve which area is using the breaker
Remove "field data" from strings in MemoryCircuitBreaker to make it a bit more generic (reflecting the package move to common.breaker)

I may have forgotten other changes that went in, so more reviews welcome :)

imotov · 2013-12-12T23:24:48Z

src/main/java/org/elasticsearch/indices/fielddata/breaker/InternalCircuitBreakerService.java

+        this.maxBytes = settings.getAsBytesSize(CIRCUIT_BREAKER_MAX_BYTES_SETTING, new ByteSizeValue(fieldDataMax)).bytes();
+        this.overhead = settings.getAsDouble(CIRCUIT_BREAKER_OVERHEAD_SETTING, DEFAULT_OVERHEAD_CONSTANT);
+
+        this.breaker = new MemoryCircuitBreaker(new ByteSizeValue(maxBytes), overhead, 0, logger);


It looks like breaker is initialized twice. Here and then in doStart()

I'll remove the doStart() one.

dakrone · 2013-12-19T19:38:47Z

Pushed a new version of the circuit breaker that addresses @imotov's comments.

kimchy · 2013-12-24T11:19:24Z

src/main/java/org/elasticsearch/indices/fielddata/breaker/InternalCircuitBreakerService.java

+public class InternalCircuitBreakerService extends AbstractLifecycleComponent<InternalCircuitBreakerService> implements CircuitBreakerService {
+
+    public static final String CIRCUIT_BREAKER_MAX_BYTES_SETTING = "indices.fielddata.cache.breaker.limit";
+    public static final String CIRCUIT_BREAKER_OVERHEAD_SETTING = "indices.fielddata.cache.breaker.overhead";


the settings names do not match the package, it should be indices.fielddata.breaker.xxx

okay, I'll change those.

s1monw · 2014-01-02T20:35:41Z

src/test/java/org/elasticsearch/indices/fielddata/breaker/CircuitBreakerServiceTests.java

+                        .setSource(MapBuilder.<String, Object>newMapBuilder().put("test", "value" + id).map()).execute().actionGet();
+            }
+
+            // refresh


there is a refresh() shortcut in ElasticsearchIntegationTest

This adds the field data circuit breaker, which is used to estimate the amount of memory required to load field data before loading it. It then raises a CircuitBreakingException if the limit is exceeded. It is configured with two parameters: `indices.fielddata.cache.breaker.limit` - the maximum number of bytes of field data to be loaded before circuit breaking. Defaults to `indices.fielddata.cache.size` if set, unbounded otherwise. `indices.fielddata.cache.breaker.overhead` - a contast for all field data estimations to be multiplied with before aggregation. Defaults to 1.03. Both settings can be configured dynamically using the cluster update settings API.

s1monw · 2014-01-02T20:43:56Z

...est/java/org/elasticsearch/indices/fielddata/breaker/RandomExceptionCircuitBreakerTests.java

+                startObject("type").
+                startObject("properties").
+                startObject("test")
+                .field("type", "string")


can we have some more types like long / double etc. as well as use a random field_data impl? We could have something like

.field("type", "string") .startObject("fielddata") .field("format", randomStringFieldDataFormat())

and something like this for numeric as well:

private static String randomNumericFieldDataFormat() { return randomFrom(Arrays.asList("array", "compressed", "doc_values")); }

private static String randomBytesFieldDataFormat() { return randomFrom(Arrays.asList("paged_bytes", "fst", "doc_values")); }

I guess we can add those to ElasticsearchIntegrationTest

s1monw · 2014-01-02T21:13:52Z

LGTM please squash and push 👍

dakrone · 2014-01-02T22:08:12Z

Merged in a754224, closing #4592

dakrone mentioned this pull request Nov 26, 2013

Create a circuit breaker to prevent searches from bringing down a node #2929

Closed

jpountz reviewed Dec 6, 2013
View reviewed changes

imotov reviewed Dec 12, 2013
View reviewed changes

kimchy reviewed Dec 24, 2013
View reviewed changes

s1monw reviewed Jan 2, 2014
View reviewed changes

Add more randomness to RandomExceptionCircuitBreakerTests

6c70e1c

dakrone closed this Jan 2, 2014

dakrone deleted the circuit-breaker-squashed branch April 21, 2014 23:00

dakrone added the :Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload label Oct 25, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add field data memory circuit breaker #4261

Add field data memory circuit breaker #4261

dakrone commented Nov 26, 2013

s1monw commented Nov 26, 2013

jpountz Dec 6, 2013

dakrone Dec 6, 2013

dakrone commented Dec 6, 2013

jpountz Dec 6, 2013

dakrone Dec 6, 2013

dakrone commented Dec 11, 2013

imotov Dec 12, 2013

dakrone Dec 13, 2013

dakrone commented Dec 19, 2013

kimchy Dec 24, 2013

dakrone Dec 24, 2013

s1monw Jan 2, 2014

s1monw Jan 2, 2014

s1monw commented Jan 2, 2014

dakrone commented Jan 2, 2014

Add field data memory circuit breaker #4261

Add field data memory circuit breaker #4261

Conversation

dakrone commented Nov 26, 2013

s1monw commented Nov 26, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dakrone commented Dec 6, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dakrone commented Dec 11, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dakrone commented Dec 19, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s1monw commented Jan 2, 2014

dakrone commented Jan 2, 2014