Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/main' into fix_alias
Browse files Browse the repository at this point in the history
  • Loading branch information
gaobinlong committed Jan 9, 2024
2 parents b6c3048 + 2de44a7 commit a93447f
Show file tree
Hide file tree
Showing 15 changed files with 313 additions and 77 deletions.
3 changes: 2 additions & 1 deletion .github/ISSUE_TEMPLATE/bug_template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ body:
description: Choose a specific OpenSearch component your bug belongs to. If you are unsure which to select or if the component is not present, select "Other".
multiple: false
options:
- Other
- # Empty first option to force selection
- Build
- Clients
- Cluster Manager
Expand All @@ -24,6 +24,7 @@ body:
- Indexing:Replication
- Indexing
- Libraries
- Other
- Plugins
- Search:Aggregations
- Search:Performance
Expand Down
3 changes: 2 additions & 1 deletion .github/ISSUE_TEMPLATE/feature_request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ body:
description: Choose a specific OpenSearch component your feature request belongs to. If you are unsure of which component to select or if the component is not present, select "Other".
multiple: false
options:
- Other
- # Empty first option to force selection
- Build
- Clients
- Cluster Manager
Expand All @@ -31,6 +31,7 @@ body:
- Indexing:Replication
- Indexing
- Libraries
- Other
- Plugins
- Search:Aggregations
- Search:Performance
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/links.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- uses: actions/checkout@v4
- name: lychee Link Checker
id: lychee
uses: lycheeverse/lychee-action@v1.8.0
uses: lycheeverse/lychee-action@v1.9.0
with:
args: --accept=200,403,429 --exclude-mail **/*.html **/*.md **/*.txt **/*.json --exclude-file .lychee.excludes
fail: true
Expand Down
5 changes: 4 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
### Dependencies
- Bumps jetty version to 9.4.52.v20230823 to fix GMS-2023-1857 ([#9822](https://github.com/opensearch-project/OpenSearch/pull/9822))
- Bump Lucene from 9.7.0 to 9.8.0 ([10276](https://github.com/opensearch-project/OpenSearch/pull/10276))
- Bump `commons-io:commons-io` from 2.13.0 to 2.15.1 ([#10294](https://github.com/opensearch-project/OpenSearch/pull/10294), [#11001](https://github.com/opensearch-project/OpenSearch/pull/11001), [#11002](https://github.com/opensearch-project/OpenSearch/pull/11002), [#11446](https://github.com/opensearch-project/OpenSearch/pull/11446), [#11554](https://github.com/opensearch-project/OpenSearch/pull/11554), [#11560](https://github.com/opensearch-project/OpenSearch/pull/11560))
- Bump `commons-io:commons-io` from 2.13.0 to 2.15.1 ([#10294](https://github.com/opensearch-project/OpenSearch/pull/10294), [#11001](https://github.com/opensearch-project/OpenSearch/pull/11001), [#11002](https://github.com/opensearch-project/OpenSearch/pull/11002), [#11446](https://github.com/opensearch-project/OpenSearch/pull/11446), [#11554](https://github.com/opensearch-project/OpenSearch/pull/11554), [#11560](https://github.com/opensearch-project/OpenSearch/pull/11560), [#11796](https://github.com/opensearch-project/OpenSearch/pull/11796))
- Bump `com.google.api.grpc:proto-google-common-protos` from 2.10.0 to 2.25.1 ([#10208](https://github.com/opensearch-project/OpenSearch/pull/10208), [#10298](https://github.com/opensearch-project/OpenSearch/pull/10298))
- Bump `com.netflix.nebula.ospackage-base` from 11.4.0 to 11.6.0 ([#10295](https://github.com/opensearch-project/OpenSearch/pull/10295), [#11630](https://github.com/opensearch-project/OpenSearch/pull/11630))
- Bump `org.apache.zookeeper:zookeeper` from 3.9.0 to 3.9.1 ([#10506](https://github.com/opensearch-project/OpenSearch/pull/10506))
Expand Down Expand Up @@ -160,6 +160,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Bump `com.maxmind.geoip2:geoip2` from 4.1.0 to 4.2.0 ([#11559](https://github.com/opensearch-project/OpenSearch/pull/11559))
- Bump `org.apache.commons:commons-lang3` from 3.13.0 to 3.14.0 ([#11691](https://github.com/opensearch-project/OpenSearch/pull/11691))
- Bump `com.maxmind.db:maxmind-db` from 3.0.0 to 3.1.0 ([#11693](https://github.com/opensearch-project/OpenSearch/pull/11693))
- Bump `net.java.dev.jna:jna` from 5.13.0 to 5.14.0 ([#11798](https://github.com/opensearch-project/OpenSearch/pull/11798))
- Bump `lycheeverse/lychee-action` from 1.8.0 to 1.9.0 ([#11795](https://github.com/opensearch-project/OpenSearch/pull/11795))

### Changed
- Mute the query profile IT with concurrent execution ([#9840](https://github.com/opensearch-project/OpenSearch/pull/9840))
Expand All @@ -185,6 +187,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Automatically add scheme to discovery.ec2.endpoint ([#11512](https://github.com/opensearch-project/OpenSearch/pull/11512))
- Restore support for Java 8 for RestClient ([#11562](https://github.com/opensearch-project/OpenSearch/pull/11562))
- Add deleted doc count in _cat/shards ([#11678](https://github.com/opensearch-project/OpenSearch/pull/11678))
- Capture information for additional query types and aggregation types ([#11582](https://github.com/opensearch-project/OpenSearch/pull/11582))

### Deprecated

Expand Down
2 changes: 1 addition & 1 deletion buildSrc/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ dependencies {
api 'com.netflix.nebula:gradle-info-plugin:12.1.6'
api 'org.apache.rat:apache-rat:0.15'
api 'commons-io:commons-io:2.15.1'
api "net.java.dev.jna:jna:5.13.0"
api "net.java.dev.jna:jna:5.14.0"
api 'com.github.johnrengelman:shadow:8.1.1'
api 'org.jdom:jdom2:2.0.6.1'
api "org.jetbrains.kotlin:kotlin-stdlib-jdk8:${props.getProperty('kotlin')}"
Expand Down
2 changes: 1 addition & 1 deletion plugins/repository-hdfs/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ dependencies {
api 'commons-collections:commons-collections:3.2.2'
api "org.apache.commons:commons-compress:${versions.commonscompress}"
api 'org.apache.commons:commons-configuration2:2.9.0'
api 'commons-io:commons-io:2.14.0'
api 'commons-io:commons-io:2.15.1'
api 'org.apache.commons:commons-lang3:3.14.0'
implementation 'com.google.re2j:re2j:1.7'
api 'javax.servlet:servlet-api:2.5'
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
f11560da189ab563a5c8e351941415430e9304ea
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.action.search;

import org.opensearch.search.aggregations.AggregationBuilder;
import org.opensearch.search.aggregations.PipelineAggregationBuilder;
import org.opensearch.telemetry.metrics.tags.Tags;

import java.util.Collection;

/**
* Increments the counters related to Aggregation Search Queries.
*/
public class SearchQueryAggregationCategorizer {

private static final String TYPE_TAG = "type";
private final SearchQueryCounters searchQueryCounters;

public SearchQueryAggregationCategorizer(SearchQueryCounters searchQueryCounters) {
this.searchQueryCounters = searchQueryCounters;
}

public void incrementSearchQueryAggregationCounters(Collection<AggregationBuilder> aggregatorFactories) {
for (AggregationBuilder aggregationBuilder : aggregatorFactories) {
incrementCountersRecursively(aggregationBuilder);
}
}

private void incrementCountersRecursively(AggregationBuilder aggregationBuilder) {
// Increment counters for the current aggregation
String aggregationType = aggregationBuilder.getType();
searchQueryCounters.aggCounter.add(1, Tags.create().addTag(TYPE_TAG, aggregationType));

// Recursively process sub-aggregations if any
Collection<AggregationBuilder> subAggregations = aggregationBuilder.getSubAggregations();
if (subAggregations != null && !subAggregations.isEmpty()) {
for (AggregationBuilder subAggregation : subAggregations) {
incrementCountersRecursively(subAggregation);
}

Check warning on line 45 in server/src/main/java/org/opensearch/action/search/SearchQueryAggregationCategorizer.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/action/search/SearchQueryAggregationCategorizer.java#L44-L45

Added lines #L44 - L45 were not covered by tests
}

// Process pipeline aggregations
Collection<PipelineAggregationBuilder> pipelineAggregations = aggregationBuilder.getPipelineAggregations();
for (PipelineAggregationBuilder pipelineAggregation : pipelineAggregations) {
String pipelineAggregationType = pipelineAggregation.getType();
searchQueryCounters.aggCounter.add(1, Tags.create().addTag(TYPE_TAG, pipelineAggregationType));
}

Check warning on line 53 in server/src/main/java/org/opensearch/action/search/SearchQueryAggregationCategorizer.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/org/opensearch/action/search/SearchQueryAggregationCategorizer.java#L51-L53

Added lines #L51 - L53 were not covered by tests
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,15 @@ final class SearchQueryCategorizer {

final SearchQueryCounters searchQueryCounters;

final SearchQueryAggregationCategorizer searchQueryAggregationCategorizer;

public SearchQueryCategorizer(MetricsRegistry metricsRegistry) {
searchQueryCounters = new SearchQueryCounters(metricsRegistry);
searchQueryAggregationCategorizer = new SearchQueryAggregationCategorizer(searchQueryCounters);
}

public void categorize(SearchSourceBuilder source) {
QueryBuilder topLevelQueryBuilder = source.query();

logQueryShape(topLevelQueryBuilder);
incrementQueryTypeCounters(topLevelQueryBuilder);
incrementQueryAggregationCounters(source.aggregations());
Expand All @@ -56,9 +58,11 @@ private void incrementQuerySortCounters(List<SortBuilder<?>> sorts) {
}

private void incrementQueryAggregationCounters(AggregatorFactories.Builder aggregations) {
if (aggregations != null) {
searchQueryCounters.aggCounter.add(1);
if (aggregations == null) {
return;
}

searchQueryAggregationCategorizer.incrementSearchQueryAggregationCounters(aggregations.getAggregatorFactories());
}

private void incrementQueryTypeCounters(QueryBuilder topLevelQueryBuilder) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,26 +9,14 @@
package org.opensearch.action.search;

import org.apache.lucene.search.BooleanClause;
import org.opensearch.index.query.BoolQueryBuilder;
import org.opensearch.index.query.MatchPhraseQueryBuilder;
import org.opensearch.index.query.MatchQueryBuilder;
import org.opensearch.index.query.MultiMatchQueryBuilder;
import org.opensearch.index.query.QueryBuilder;
import org.opensearch.index.query.QueryBuilderVisitor;
import org.opensearch.index.query.QueryStringQueryBuilder;
import org.opensearch.index.query.RangeQueryBuilder;
import org.opensearch.index.query.RegexpQueryBuilder;
import org.opensearch.index.query.TermQueryBuilder;
import org.opensearch.index.query.WildcardQueryBuilder;
import org.opensearch.index.query.functionscore.FunctionScoreQueryBuilder;
import org.opensearch.telemetry.metrics.tags.Tags;

/**
* Class to visit the querybuilder tree and also track the level information.
* Class to visit the query builder tree and also track the level information.
* Increments the counters related to Search Query type.
*/
final class SearchQueryCategorizingVisitor implements QueryBuilderVisitor {
private static final String LEVEL_TAG = "level";
private final int level;
private final SearchQueryCounters searchQueryCounters;

Expand All @@ -42,29 +30,7 @@ private SearchQueryCategorizingVisitor(SearchQueryCounters counters, int level)
}

public void accept(QueryBuilder qb) {
if (qb instanceof BoolQueryBuilder) {
searchQueryCounters.boolCounter.add(1, Tags.create().addTag(LEVEL_TAG, level));
} else if (qb instanceof FunctionScoreQueryBuilder) {
searchQueryCounters.functionScoreCounter.add(1, Tags.create().addTag(LEVEL_TAG, level));
} else if (qb instanceof MatchQueryBuilder) {
searchQueryCounters.matchCounter.add(1, Tags.create().addTag(LEVEL_TAG, level));
} else if (qb instanceof MatchPhraseQueryBuilder) {
searchQueryCounters.matchPhrasePrefixCounter.add(1, Tags.create().addTag(LEVEL_TAG, level));
} else if (qb instanceof MultiMatchQueryBuilder) {
searchQueryCounters.multiMatchCounter.add(1, Tags.create().addTag(LEVEL_TAG, level));
} else if (qb instanceof QueryStringQueryBuilder) {
searchQueryCounters.queryStringQueryCounter.add(1, Tags.create().addTag(LEVEL_TAG, level));
} else if (qb instanceof RangeQueryBuilder) {
searchQueryCounters.rangeCounter.add(1, Tags.create().addTag(LEVEL_TAG, level));
} else if (qb instanceof RegexpQueryBuilder) {
searchQueryCounters.regexCounter.add(1, Tags.create().addTag(LEVEL_TAG, level));
} else if (qb instanceof TermQueryBuilder) {
searchQueryCounters.termCounter.add(1, Tags.create().addTag(LEVEL_TAG, level));
} else if (qb instanceof WildcardQueryBuilder) {
searchQueryCounters.wildcardCounter.add(1, Tags.create().addTag(LEVEL_TAG, level));
} else {
searchQueryCounters.otherQueryCounter.add(1, Tags.create().addTag(LEVEL_TAG, level));
}
searchQueryCounters.incrementCounter(qb, level);
}

public QueryBuilderVisitor getChildVisitor(BooleanClause.Occur occur) {
Expand Down
Loading

0 comments on commit a93447f

Please sign in to comment.