Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node level can match action #78765

Merged
merged 38 commits into from
Oct 18, 2021
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
76b467a
WIP
ywelsch Sep 30, 2021
07e9f51
happy path
ywelsch Sep 30, 2021
53d1071
more work
ywelsch Oct 4, 2021
2263bbd
more edits
ywelsch Oct 4, 2021
66d2095
more stuff
ywelsch Oct 5, 2021
4a8bf13
CCS multi-version test passing
ywelsch Oct 5, 2021
a69bcbc
unit tests
ywelsch Oct 6, 2021
fe56d5a
simpler diff?
ywelsch Oct 6, 2021
75e2bb5
more renaming
ywelsch Oct 6, 2021
3e1d6e1
Merge remote-tracking branch 'elastic/master' into node-level-can-match
ywelsch Oct 6, 2021
44b5783
single list
ywelsch Oct 6, 2021
a65bd63
reset prefiltter settings
ywelsch Oct 6, 2021
41b61bd
fix serialzation
ywelsch Oct 6, 2021
dd93c93
thread pool
ywelsch Oct 7, 2021
1a718cf
Merge remote-tracking branch 'elastic/master' into node-level-can-match
ywelsch Oct 7, 2021
36bb4a1
javadoc
ywelsch Oct 11, 2021
9bf0259
Merge remote-tracking branch 'elastic/master' into node-level-can-match
ywelsch Oct 11, 2021
da5707f
merge conflict
ywelsch Oct 11, 2021
ea79ae2
docs
ywelsch Oct 12, 2021
9c3196f
Merge remote-tracking branch 'elastic/master' into node-level-can-match
ywelsch Oct 12, 2021
5a81c07
remove dead code
ywelsch Oct 14, 2021
e26160f
avoid cast
ywelsch Oct 14, 2021
02c7193
simpler and rename
ywelsch Oct 14, 2021
ca49aa8
Merge remote-tracking branch 'elastic/master' into node-level-can-match
ywelsch Oct 14, 2021
9670258
checkstyle
ywelsch Oct 14, 2021
803068b
cosmetics
ywelsch Oct 14, 2021
a3457ef
original indices
ywelsch Oct 14, 2021
d032fef
Add can match qa test
dnhatn Oct 14, 2021
24baa8b
implement indicesrequest
ywelsch Oct 14, 2021
171e530
Merge branch 'node-level-can-match' of github.com:ywelsch/elasticsear…
ywelsch Oct 14, 2021
259f049
merge originalindices
ywelsch Oct 14, 2021
16f45ed
better
ywelsch Oct 14, 2021
c0e35ee
radnomize use of can-match phase
ywelsch Oct 15, 2021
bf6b194
Merge remote-tracking branch 'elastic/master' into node-level-can-match
ywelsch Oct 15, 2021
ade3b7b
Merge branch 'master' into node-level-can-match
elasticmachine Oct 15, 2021
cc83808
rename
ywelsch Oct 18, 2021
aada13a
Merge remote-tracking branch 'elastic/master' into node-level-can-match
ywelsch Oct 18, 2021
c1cc951
Merge branch 'node-level-can-match' of github.com:ywelsch/elasticsear…
ywelsch Oct 18, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/reference/modules/threadpool.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ There are several thread pools, but the important ones include:
For count/search/suggest/get operations on `search_throttled indices`.
Thread pool type is `fixed` with a size of `1`, and queue_size of `100`.

`search_coordination`::
For lightweight search-related coordination operations. Thread pool type is
`fixed` with a size of a max of `min(5, (`<<node.processors,
`# of allocated processors`>>`) / 2)`, and queue_size of `1000`.

`get`::
For get operations. Thread pool type is `fixed`
with a size of <<node.processors, `# of allocated processors`>>,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ static int indexDocs(RestHighLevelClient client, String index, int numDocs) thro
return numDocs;
}

void verifySearch(String localIndex, int localNumDocs, String remoteIndex, int remoteNumDocs) {
void verifySearch(String localIndex, int localNumDocs, String remoteIndex, int remoteNumDocs, Integer preFilterShardSize) {
try (RestHighLevelClient localClient = newLocalClient()) {
Request request = new Request("POST", "/_search");
final int expectedDocs;
Expand All @@ -212,6 +212,12 @@ void verifySearch(String localIndex, int localNumDocs, String remoteIndex, int r
if (UPGRADE_FROM_VERSION.onOrAfter(Version.V_7_0_0)) {
request.addParameter("ccs_minimize_roundtrips", Boolean.toString(randomBoolean()));
}
if (preFilterShardSize == null && randomBoolean()) {
preFilterShardSize = randomIntBetween(1, 100);
}
if (preFilterShardSize != null) {
request.addParameter("pre_filter_shard_size", Integer.toString(preFilterShardSize));
}
int size = between(1, 100);
request.setJsonEntity("{\"sort\": \"f\", \"size\": " + size + "}");
Response response = localClient.getLowLevelClient().performRequest(request);
Expand Down Expand Up @@ -245,7 +251,32 @@ public void testBWCSearchStates() throws Exception {
configureRemoteClusters(getNodes(remoteClient.getLowLevelClient()));
int iterations = between(1, 20);
for (int i = 0; i < iterations; i++) {
verifySearch(localIndex, localNumDocs, CLUSTER_ALIAS + ":" + remoteIndex, remoteNumDocs);
verifySearch(localIndex, localNumDocs, CLUSTER_ALIAS + ":" + remoteIndex, remoteNumDocs, null);
}
localClient.indices().delete(new DeleteIndexRequest(localIndex), RequestOptions.DEFAULT);
remoteClient.indices().delete(new DeleteIndexRequest(remoteIndex), RequestOptions.DEFAULT);
}
}

public void testCanMatch() throws Exception {
String localIndex = "test_can_match_local_index";
String remoteIndex = "test_can_match_remote_index";
try (RestHighLevelClient localClient = newLocalClient();
RestHighLevelClient remoteClient = newRemoteClient()) {
localClient.indices().create(new CreateIndexRequest(localIndex)
.settings(Settings.builder().put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, between(5, 20))),
RequestOptions.DEFAULT);
int localNumDocs = indexDocs(localClient, localIndex, between(10, 100));

remoteClient.indices().create(new CreateIndexRequest(remoteIndex)
.settings(Settings.builder().put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, between(5, 20))),
RequestOptions.DEFAULT);
int remoteNumDocs = indexDocs(remoteClient, remoteIndex, between(10, 100));

configureRemoteClusters(getNodes(remoteClient.getLowLevelClient()));
int iterations = between(1, 10);
for (int i = 0; i < iterations; i++) {
verifySearch(localIndex, localNumDocs, CLUSTER_ALIAS + ":" + remoteIndex, remoteNumDocs, between(1, 10));
dnhatn marked this conversation as resolved.
Show resolved Hide resolved
}
localClient.indices().delete(new DeleteIndexRequest(localIndex), RequestOptions.DEFAULT);
remoteClient.indices().delete(new DeleteIndexRequest(remoteIndex), RequestOptions.DEFAULT);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -724,7 +724,11 @@ private static void assumeMultiClusterSetup() {
private static SearchRequest initSearchRequest() {
List<String> indices = Arrays.asList(INDEX_NAME, "my_remote_cluster:" + INDEX_NAME);
Collections.shuffle(indices, random());
return new SearchRequest(indices.toArray(new String[0]));
final SearchRequest request = new SearchRequest(indices.toArray(new String[0]));
if (randomBoolean()) {
request.setPreFilterShardSize(between(1, 20));
}
return request;
}

private static void duelSearch(SearchRequest searchRequest, Consumer<SearchResponse> responseChecker) throws Exception {
Expand Down
16 changes: 16 additions & 0 deletions server/src/main/java/org/elasticsearch/action/OriginalIndices.java
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@

import java.io.IOException;
import java.util.Arrays;
import java.util.Objects;

/**
* Used to keep track of original indices within internal (e.g. shard level) requests
Expand Down Expand Up @@ -67,4 +68,19 @@ public String toString() {
", indicesOptions=" + indicesOptions +
'}';
}

@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
OriginalIndices that = (OriginalIndices) o;
return Arrays.equals(indices, that.indices) && Objects.equals(indicesOptions, that.indicesOptions);
}

@Override
public int hashCode() {
int result = Objects.hash(indicesOptions);
result = 31 * result + Arrays.hashCode(indices);
return result;
}
}
Loading