-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
EQL: Filter out null join keys in sequence queries #78195
Conversation
Pinging @elastic/es-ql (Team:QL) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
// do not join on null values | ||
if (keyNames.isEmpty() == false) { | ||
BoolQueryBuilder nullValuesFilter = boolQuery(); | ||
for (int keyIndex = 0; keyIndex < keyNames.size(); keyIndex++) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use foreach instead:
for (String keyName : keyNames) {
}
BoolQueryBuilder nullValuesFilter = boolQuery(); | ||
for (int keyIndex = 0; keyIndex < keyNames.size(); keyIndex++) { | ||
// add an "exists" query for each join key to filter out any non-existent values | ||
nullValuesFilter.must(existsQuery(keyNames.get(keyIndex))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need to score, just use filter. Which removes the need for wrapping it into a bool query and simply use RuntimeUtils.addFilter
for every existsQuery.
is enabled in this cluster).
@elasticmachine run elasticsearch-ci/bwc elasticsearch-ci/part-2 |
@elasticmachine run elasticsearch-ci/part-2 |
.setHttpClientConfigCallback(new HttpClientConfigCallback() { | ||
@Override | ||
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) { | ||
return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider); | ||
} | ||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it makes it generally easier with testing, so good to have it, but curious if this was added for a specific reason.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, there is a better reason: since #70114 security is enabled in this test cluster.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you.
(teaching me a lesson about reading PRs chronologically...)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lgtm.
Superseded by #79677. |
Joining on
null
keys in sequences can lead to a high amount of queries and matches in a cluster or CCS multi-cluster scenario where some indices have mappings with existent fields while others don't. This change removes support for joining onnull
values by filtering them out pro-actively with anexists
filter.