Introduce ability to minimize round-trips in CCS #37828

javanna · 2019-01-24T15:59:33Z

With #37566 we have introduced the ability to merge multiple search responses into one. That makes it possible to expose a new way of executing cross-cluster search requests, that makes CCS much faster whenever there is network latency between the CCS coordinating node and the remote clusters. The coordinating node can now send a single search request to each remote cluster, which gets reduced by each one of them. from + size results are requested to each cluster, and the reduce phase in each cluster is non final (meaning that buckets are not pruned and pipeline aggs are not executed). The CCS coordinating node performs an additional, final reduction, which produces one search response out of the multiple responses received from the different clusters.

This new execution path will be activated by default for any CCS request unless a scroll is provided or inner hits are requested as part of field collapsing. The search API accepts now a new parameter called ccs_minimize_roundtrips that allows to opt-out of the default behaviour.

Relates to #32125

elasticmachine · 2019-01-24T15:59:36Z

Pinging @elastic/es-search

jimczi

LGTM, I left a minor comment regarding another pr that will conflict with this one.

server/src/main/java/org/elasticsearch/action/search/TransportSearchAction.java

… reduce mode

s1monw

I left a question

s1monw · 2019-01-28T10:43:31Z

docs/reference/modules/cross-cluster-search.asciidoc

+ involved, as it treats all shards the same, at the cost of sending many
+ requests to each remote cluster.
+
+- `auto`: `remote` is used whenever possible. In case a scroll is provided,


so what happens if I specify remote and I have inner hits?

I wonder if we have to have a mode at all. can't we simply go with local_reduce: true|false if it's true we force local if not we try remote if possible?

so what happens if I specify remote and I have inner hits?

If you specify remote and you provide a scroll or request inner hits as part of fields collapsing, you get back an error. I think it's better to let users know that what they have requested is not supported when they are explicit in their request.

I wonder if we have to have a mode at all. can't we simply go with local_reduce: true|false if it's true we force local if not we try remote if possible?

I think that having remote (or false) act only as a preference is confusing, as users are requesting something that may not be possible. Ideally, if somebody sets local_reduce: false (or reduce_mode: remote), we would never go and fan out to the shards. It sounds weird to me to have an option act as a proper on/off behaviour and the other one as a preference, unless they have different names.

If we want to have only two values and get rid of the auto option, we could have reduce_mode: always_local | prefer_remote which I find much more self-explanatory than a boolean. Yet I am not sure it is much different from what we have now in the PR.

or we could also rename auto to prefer_remote and keep remote.

we discussed this a bit with @jimczi and the team, we could go with a boolean but I would like to have some force part in the name of the option. We could go with ccs_force_local_reduce which defaults to false, or maybe even ccs_optimize_for_latency which defaults to true? I think I prefer the latter as it tells when such option should be used. Any other thoughts/preference?

clintongormley · 2019-01-28T16:39:55Z

We could go with ccs_force_local_reduce which defaults to false, or maybe even ccs_optimize_for_latency which defaults to true? I think I prefer the latter as it tells when such option should be used.

With ccs_optimize_for_latency it is not clear to me which option I'd be opting for - i'd have to really understand the feature. Also, given that we want to use remote reduce as the default because we think it'll be better, then i think ccs_force_local_reduce: true works as a good way to opt out of that behaviour.

s1monw · 2019-01-28T17:24:22Z

With ccs_optimize_for_latency it is not clear to me which option I'd be opting for - i'd have to really understand the feature. Also, given that we want to use remote reduce as the default because we think it'll be better, then i think ccs_force_local_reduce: true works as a good way to opt out of that behaviour.

I am not sure if ccs_force_local_reduce explains what's going on. the reduce is not relevant here IMO. This is more about the network calls we do. Something like css_minimize_roundtrips: true|false would be more descriptive IMO.

zuketo · 2019-01-28T22:48:32Z

Does the term WAN help here? E.g. ccs_wan_optimized: true|false

"We're operating across a large network, let's keep this set to true" vs "We're operating within a local network, let's investigate changing this setting"

javanna · 2019-01-29T09:55:48Z

I am not sure if ccs_force_local_reduce explains what's going on. the reduce is not relevant here IMO. This is more about the network calls we do. Something like css_minimize_roundtrips: true|false would be more descriptive IMO.

I agree with this, I like ccs_minimize_roundtrips, I think it explains the trade-off well, at the same time staying high-level and without mentioning reduction. I briefly chatted to @clintongormley and he also agrees. I will adapt the PR accordingly.

s1monw · 2019-01-29T12:51:54Z

thanks @javanna

Note that an enum is still returned in the _clusters section of the search response.

…dates

javanna · 2019-01-31T09:07:43Z

heya @s1monw can you have another look? it should be ready.

javanna · 2019-01-31T09:52:50Z

run elasticsearch-ci/2

s1monw

LGTM thanks @javanna

javanna · 2019-01-31T14:12:33Z

thanks a lot everybody involved! ;)

* master: (100 commits) Push primary term to replication tracker (elastic#38044) Introduce ability to minimize round-trips in CCS (elastic#37828) Don't Assert Ack on when Publish Timeout is 0 in Test (elastic#38077) Reduce object creation in Rounding class (elastic#38061) Treat put-mapping calls with `_doc` as a top-level key as typed calls. (elastic#38032) Fix test bug when testing the merging of mappings and templates. (elastic#38021) spelling: java script -- not JavaScript (elastic#37057) Enable SSL in reindex with security QA tests (elastic#37600) Disable BWC tests during backport (elastic#38074) SQL: Added SSL configuration options tests (elastic#37875) Minor fixes in the release notes script. (elastic#37967) Fix typo in docs. (elastic#38018) Update Lucene repo for 7.0.0-alpha2 (elastic#37985) Fix size of rolling-upgrade bootstrap config (elastic#38031) fix DateIndexNameProcessorTests offset pattern (elastic#38069) Speed up converting of temporal accessor to zoned date time (elastic#37915) Work around JDK8 timezone bug in tests (elastic#37968) Correct arg names when update mapping/settings from leader (elastic#38063) Introduce ssl settings to reindex from remote (elastic#37527) Mute testRetentionLeasesSyncOnExpiration ...

javanna added 14 commits January 24, 2019 16:43

Adapt SearchRequest constructor

337fd75

add default from and size public constants

f100b46

[TEST] add search handler to test transport

f3daf69

add comment on registering same cluster multiple times

e05725e

wip

eae918b

add search request ccs_execution_mode option

2703d74

done

c9b737b

import

a6404e3

progress

0b064a9

fix

650e876

rename execution mode to reduce mode

74d52cb

rename reduce modes

c5d9dc5

last renames and update docs

589d69e

update spec

56d845e

javanna added >feature release highlight WIP :Search/Search Search-related issues that do not fall into other categories v7.0.0 labels Jan 24, 2019

javanna requested a review from jimczi January 24, 2019 15:59

jimczi reviewed Jan 24, 2019

View reviewed changes

server/src/main/java/org/elasticsearch/action/search/TransportSearchAction.java Outdated Show resolved Hide resolved

javanna mentioned this pull request Jan 24, 2019

Cross-cluster search alternate execution mode #32125

Closed

11 tasks

add auto ccs_reduce_mode

01e9b27

javanna changed the title ~~Introduce ccs remote reduce mode~~ Introduce CCS remote reduce mode Jan 24, 2019

javanna added 5 commits January 24, 2019 22:11

fix failing xpack qa test and expand others

0e77e76

unify listeners and extract method that converts auto to an effective…

bf3fdb7

… reduce mode

Merge branch 'master' into feature/ccs_local_reduction

58e3327

adapt and expand unit tests

ee25840

extract ccsRemoteReduce method and add unit tests for it

e554e2b

jimczi approved these changes Jan 25, 2019

View reviewed changes

javanna requested a review from s1monw January 28, 2019 07:20

s1monw requested changes Jan 28, 2019

View reviewed changes

Merge branch 'master' into feature/ccs_local_reduction

da6c3cb

javanna added 11 commits January 29, 2019 16:50

Rename ccs_reduce_mode to ccs_minimize_roundtrips and make it a boolean.

8c67836

Note that an enum is still returned in the _clusters section of the search response.

Merge branch 'master' into feature/ccs_local_reduction

0f1d784

Merge branch 'master' into feature/ccs_local_reduction

2bb80c4

add assert that scrollId is never set in search response

9574278

Merge branch 'master' into feature/ccs_local_reduction

f735784

remove ccs_reduction from _clusters section

27ae7dd

fix selection between the two modes which got lost with the recent up…

4d8266f

…dates

revert minor change

86b91cf

fix failing test

2f1bbaf

fix parameter name

91d5df8

Merge branch 'master' into feature/ccs_local_reduction

0165ec1

s1monw approved these changes Jan 31, 2019

View reviewed changes

javanna changed the title ~~Introduce CCS remote reduce mode~~ Introduce ability to minimize round-trips in CCS requests Jan 31, 2019

javanna merged commit 622fb78 into elastic:master Jan 31, 2019

javanna changed the title ~~Introduce ability to minimize round-trips in CCS requests~~ Introduce ability to minimize round-trips in CCS Jan 31, 2019

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce ability to minimize round-trips in CCS #37828

Introduce ability to minimize round-trips in CCS #37828

javanna commented Jan 24, 2019 •

edited

Loading

elasticmachine commented Jan 24, 2019

jimczi left a comment

s1monw left a comment

s1monw Jan 28, 2019

s1monw Jan 28, 2019

javanna Jan 28, 2019

javanna Jan 28, 2019 •

edited

Loading

javanna Jan 28, 2019

clintongormley commented Jan 28, 2019

s1monw commented Jan 28, 2019

zuketo commented Jan 28, 2019

javanna commented Jan 29, 2019

s1monw commented Jan 29, 2019

javanna commented Jan 31, 2019

javanna commented Jan 31, 2019

s1monw left a comment

javanna commented Jan 31, 2019

Introduce ability to minimize round-trips in CCS #37828

Introduce ability to minimize round-trips in CCS #37828

Conversation

javanna commented Jan 24, 2019 • edited Loading

elasticmachine commented Jan 24, 2019

jimczi left a comment

Choose a reason for hiding this comment

s1monw left a comment

Choose a reason for hiding this comment

s1monw Jan 28, 2019

Choose a reason for hiding this comment

s1monw Jan 28, 2019

Choose a reason for hiding this comment

javanna Jan 28, 2019

Choose a reason for hiding this comment

javanna Jan 28, 2019 • edited Loading

Choose a reason for hiding this comment

javanna Jan 28, 2019

Choose a reason for hiding this comment

clintongormley commented Jan 28, 2019

s1monw commented Jan 28, 2019

zuketo commented Jan 28, 2019

javanna commented Jan 29, 2019

s1monw commented Jan 29, 2019

javanna commented Jan 31, 2019

javanna commented Jan 31, 2019

s1monw left a comment

Choose a reason for hiding this comment

javanna commented Jan 31, 2019

javanna commented Jan 24, 2019 •

edited

Loading

javanna Jan 28, 2019 •

edited

Loading