advanced setting to control search request preference #17271

nreese · 2018-03-20T15:34:34Z

Provides advance setting courier:setRequestPreference. When true (default), perference is set to sessionId as it always has been. When false, preference is not set in the search header.

elasticmachine · 2018-03-20T16:22:37Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

nreese · 2018-03-20T18:16:41Z

@gchaps Can you please provide feedback on the wording for courier:setRequestPreference description?

elasticmachine · 2018-03-20T18:59:53Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

stacey-gammon

Cool!

spalger · 2018-03-20T21:13:49Z

src/core_plugins/kibana/ui_setting_defaults.js

+      value: true,
+      description: '<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-preference.html" target="_blank" rel="noopener noreferrer">Request Preference</a> ' +
+        ' controls the shard copies used for search execution.' +
+        ' Set to true to execute all search requests on the same shards. ' +


I think the description should mention the original reason we started using preference, which I believe was caching (Please correct me if I'm wrong @rashidkpc).

Like your description states, not sending preference sort of randomizes the node that handles the request, which means that the caches from a previous request might not be reusable on a second request and will need to be rebuilt.

What about this

"Set to true to execute all search requests on the same shards. This has the benefit of reusing Shard Request Cache across requests. Set to false to have search request execution randomized among all available shard copies. Setting to false may provide better performance since requests can be spread across all shard copies but may result in inconsistent results as different shards may be in different refresh states."

I wonder how many requests can actually even use the cache since... "Most queries that use now (see Date Mathedit) cannot be cached." from warning in shard-request-cache.html

There are other caches, like the caches from filters, which I'm almost positive are not impacted by using now.

spalger

LGTM

elasticmachine · 2018-03-20T22:50:11Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

LucaWintergerst · 2018-03-21T15:59:11Z

The way this is implemented right now might conflict with the zone awareness in Elasticsearch

When executing search or GET requests, with shard awareness enabled, Elasticsearch will prefer using local shards — shards in the same awareness group — to execute the request. This is usually faster than crossing between racks or across zone boundaries.

If I set this setting to false, the performance might be much worse as I will only hit half of the cluster!

A solution would be to set the preference to an empty string. This will then override the es default in case awareness is being used.
The reason why I'm proposing this is that this will preserve the behaviour that users experience right now

Ideally a user would be able to choose between sessionId none (which would be an empty string) local (which would mirror the behaviour when using zones)

nreese · 2018-03-21T16:33:00Z

@LucaWintergerst Thanks for brining this up before the changes were committed. There is no reason why the setting can't be a list instead of a boolean.

Just a couple of questions to make sure I understand everything.

Why would not setting the preference with zone awareness enabled only hit half of the cluster?
What happens if zone awareness is not enabled and preference is set to an empty string?

LucaWintergerst · 2018-03-21T17:24:30Z

Let me try to explain in a bit more detail.
Shard awareness is used to make elasticsearch nodes aware of the underlying hardware. In most cases a cluster will be tagged with two awareness values to ensure that the replica shards reside on a different set of hardware than the primaries. (a set could be a different server rack, a different physical host when running in VMs, a different region in AWS, etc.)
When awareness is used, the preference always defaults to hitting nodes tagged with the same attribute as the node which received the request from Kibana. If I have two zones, my query will only run on 50% of my nodes.
Since kibana is currently using preference: sessionId it overrides that default behaviour.
If we were to completely remove the preference we would suddenly default to the local shards behaviour which would cause queries to only run in a subset of the nodes for many users.
an empty string when zone awareness is not set will behave the same way as no preference. I will get someone from the ES team to confirm

LucaWintergerst · 2018-03-21T18:20:52Z

after talking to a developer from the ES team we agreed that since this is an advanced setting it would be fine the way it is now.
However, it would be even better if we could make it fully customizable so all possible preferences can be used (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-preference.html)

How much work would it be to make it customizable?
We would need the following functions:

setRequestPreference: $SESSION_ID <- default
setRequestPreference: custom (set it to a user defined string)
setRequestPreference: none (do not set it at all)

We need this distinction as none will not be the same as an empty string.
I would love to get additional feedback on this.

nreese · 2018-03-21T18:32:27Z

I don't think we could do it with a single parameter but I could do this with 2 parameters.

setRequestPreference: list of sessionId, custom, none
customRequestPreference: string input that allows custom preference (defaults to _local). customRequestPreference is only used when setRequestPreference is set to custom.

How does that sound?

LucaWintergerst · 2018-03-21T18:37:53Z

that sounds great to me

gchaps · 2018-03-22T16:22:41Z

I'm thinking of something like this:

Allows you to set which shards handle your search requests. $Session_ID restricts operations to shards with your session id. custom allows you to define a your own preference, such as a web session ID or username. none means do not set a preference. This differs from an empty string which means no preference. Learn more

I'm not sure how much you want to explain about customRequestPreference

…uestPreference

nreese · 2018-03-27T15:18:18Z

@spalger @stacey-gammon As discussed above, I have changed courier:setRequestPreference to be a select and added courier:customRequestPreference.

@gchaps I changed the wording of the session id to keep the bit about caching and added more details to none.

elasticmachine · 2018-03-27T16:01:46Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

gchaps · 2018-03-27T16:16:56Z

@nreese I recommend breaking the last sentence into two sentences:

This might provide better performance because requests can be spread across all shard copies. However, results might be inconsistent because different shards might be in different refresh states.

We typically use "might" instead of "may" for localization reasons.

elasticmachine · 2018-03-27T20:38:37Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

stacey-gammon · 2018-03-28T13:56:17Z

@LucaWintergerst - The Preference documents say:

By default, the operation is randomized among the available shard copies.

But if I am understanding correctly, that is only the default behavior when zone awareness is not set? When zone awareness is set, then the default behavior, when no preference is given, is _local? If I am understanding all this correctly, maybe the docs should be updated? (cc @gchaps)

@nreese - Looks great! Confirmed the requests look as expected and if I set it to something like _shards:8,9 I get no results (since I don't actually have those shards).

LucaWintergerst · 2018-03-28T14:16:07Z

@stacey-gammon it's not just _local, it is kind of like this:_prefer_nodes=<all nodes in that awareness zone> (I have to admit that I don't know how it is done exactly within Elasticsearch)
Updating the docs sounds like a good idea. Perhaps changing it to this?:
By default, the operation is randomized among the available shard copies, unless allocation awareness is used

* advanced setting to control search request preference * add header tests * add sentince about caching to description * change courier:setRequestPreference to list and add courier:customRequestPreference * update setting text

nreese requested review from spalger and stacey-gammon March 20, 2018 18:14

stacey-gammon approved these changes Mar 20, 2018

View reviewed changes

spalger reviewed Mar 20, 2018

View reviewed changes

spalger approved these changes Mar 20, 2018

View reviewed changes

nreese mentioned this pull request Mar 21, 2018

Allow customisation of "preferences" for _msearch #15573

Closed

nreese added 3 commits March 27, 2018 08:37

advanced setting to control search request preference

89782a4

add header tests

2303eed

add sentince about caching to description

08cba76

nreese force-pushed the request_preference branch from b301d65 to 08cba76 Compare March 27, 2018 14:37

change courier:setRequestPreference to list and add courier:customReq…

50f365a

…uestPreference

update setting text

892cfe3

stacey-gammon approved these changes Mar 28, 2018

View reviewed changes

nreese merged commit 3fecd2d into elastic:master Mar 28, 2018

nreese mentioned this pull request Mar 28, 2018

[6.x] advanced setting to control search request preference (#17271) #17447

Merged

gchaps mentioned this pull request Mar 29, 2018

[DOCS] Update intro to Search APIs > Request Body Search > Preference elastic/elasticsearch#29302

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

advanced setting to control search request preference #17271

advanced setting to control search request preference #17271

nreese commented Mar 20, 2018

elasticmachine commented Mar 20, 2018

nreese commented Mar 20, 2018

elasticmachine commented Mar 20, 2018

stacey-gammon left a comment

spalger Mar 20, 2018

nreese Mar 20, 2018

nreese Mar 20, 2018

spalger Mar 20, 2018

spalger Mar 20, 2018

spalger left a comment

elasticmachine commented Mar 20, 2018

LucaWintergerst commented Mar 21, 2018 •

edited

Loading

nreese commented Mar 21, 2018

LucaWintergerst commented Mar 21, 2018 •

edited

Loading

LucaWintergerst commented Mar 21, 2018 •

edited

Loading

nreese commented Mar 21, 2018

LucaWintergerst commented Mar 21, 2018

gchaps commented Mar 22, 2018

nreese commented Mar 27, 2018 •

edited

Loading

elasticmachine commented Mar 27, 2018

gchaps commented Mar 27, 2018

elasticmachine commented Mar 27, 2018

stacey-gammon commented Mar 28, 2018

LucaWintergerst commented Mar 28, 2018

advanced setting to control search request preference #17271

advanced setting to control search request preference #17271

Conversation

nreese commented Mar 20, 2018

elasticmachine commented Mar 20, 2018

💔 Build Failed

nreese commented Mar 20, 2018

elasticmachine commented Mar 20, 2018

💚 Build Succeeded

stacey-gammon left a comment

Choose a reason for hiding this comment

spalger Mar 20, 2018

Choose a reason for hiding this comment

nreese Mar 20, 2018

Choose a reason for hiding this comment

nreese Mar 20, 2018

Choose a reason for hiding this comment

spalger Mar 20, 2018

Choose a reason for hiding this comment

spalger Mar 20, 2018

Choose a reason for hiding this comment

spalger left a comment

Choose a reason for hiding this comment

elasticmachine commented Mar 20, 2018

💚 Build Succeeded

LucaWintergerst commented Mar 21, 2018 • edited Loading

nreese commented Mar 21, 2018

LucaWintergerst commented Mar 21, 2018 • edited Loading

LucaWintergerst commented Mar 21, 2018 • edited Loading

nreese commented Mar 21, 2018

LucaWintergerst commented Mar 21, 2018

gchaps commented Mar 22, 2018

nreese commented Mar 27, 2018 • edited Loading

elasticmachine commented Mar 27, 2018

💚 Build Succeeded

gchaps commented Mar 27, 2018

elasticmachine commented Mar 27, 2018

💚 Build Succeeded

stacey-gammon commented Mar 28, 2018

LucaWintergerst commented Mar 28, 2018

LucaWintergerst commented Mar 21, 2018 •

edited

Loading

LucaWintergerst commented Mar 21, 2018 •

edited

Loading

LucaWintergerst commented Mar 21, 2018 •

edited

Loading

nreese commented Mar 27, 2018 •

edited

Loading