POC Dynamic threadpool #158
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Andriy Redko <[email protected]>
* add `Strings#isDigits` API inspiration taken from [this SO answer][SO]. note that the stream is not parallelised to avoid the overhead of this as the method is intended to be called primarily with shorter strings where the time to set up would take longer than the actual check. [SO]: https://stackoverflow.com/a/35150400 Signed-off-by: Ralph Ursprung <[email protected]> * add `phone` & `phone-search` analyzer + tokenizer this is largely based on [elasticsearch-phone] and internally uses [libphonenumber]. this intentionally only ports a subset of the features: only `phone` and `phone-search` are supported right now, `phone-email` can be added if/when there's a clear need for it. using `libphonenumber` is required since parsing phone numbers is a non-trivial task (even though it might seem trivial at first glance!), as can be seen in the list [falsehoods programmers believe about phone numbers][falsehoods]. this allows defining the region to be used when analysing a phone number. so far only the generic "unkown" region (`ZZ`) had been used which worked as long as international numbers were prefixed with `+` but did not work when using local numbers (e.g. a number stored as `+4158...` was not matched against a number entered as `004158...` or `058...`). example configuration for an index: ```json { "index": { "analysis": { "analyzer": { "phone": { "type": "phone" }, "phone-search": { "type": "phone-search" }, "phone-ch": { "type": "phone", "phone-region": "CH" }, "phone-search-ch": { "type": "phone-search", "phone-region": "CH" } } } } } ``` this creates four analyzers: `phone` and `phone-search` which do not explicitly specify a region and thus fall back to `ZZ` (unknown region, regional version of international dialing prefix (e.g. `00` instead of `+` in most of europe) will not be recognised) and `phone-ch` and `phone-search-ch` which will try to parse the phone number as a swiss phone number (thus e.g. `00` as a prefix is recognised as the international dialing prefix). note that the analyzer is (currently) not meant to find phone numbers in large text documents - instead it should be used on fields which contain just the phone number (though extra text will be ignored) and it collects the whole content of the field into a `String` in memory, making it unsuitable for large field values. this has been implemented in a new plugin which is however part of the central opensearch repository as it was deemed too big an overhead to have it in a separate repository but not important enough to bundle it directly in `analysis-common` (see the discussion on the issue and the PR for further details). note that the new plugin has been added to the exclude list of the javadoc check as this check is overzealous and also complains in many cases where it shouldn't (e.g. on overridden methods - which it should theoretically not do - or constructors which don't even exist). the check first needs to be improved before this exclusion could be removed. closes opensearch-project#11326 [elasticsearch-phone]: https://github.com/purecloudlabs/elasticsearch-phone [libphonenumber]: https://github.com/google/libphonenumber [falsehoods]: https://github.com/google/libphonenumber/blob/master/FALSEHOODS.md Signed-off-by: Ralph Ursprung <[email protected]> --------- Signed-off-by: Ralph Ursprung <[email protected]>
…ges as non-breaking (opensearch-project#16181) Signed-off-by: Andriy Redko <[email protected]>
…anslog (opensearch-project#16151) Signed-off-by: Sachin Kale <[email protected]>
…ct#16179) Signed-off-by: Gaurav Bafna <[email protected]>
…pensearch-project#16194) * Fix warnings from SLF4J on startup when repository-s3 is installed Signed-off-by: Craig Perkins <[email protected]> * Add to CHANGELOG Signed-off-by: Craig Perkins <[email protected]> * Fix precommit Signed-off-by: Craig Perkins <[email protected]> --------- Signed-off-by: Craig Perkins <[email protected]>
…ch-project#16192) Signed-off-by: Sachin Kale <[email protected]>
Signed-off-by: Gaurav Bafna <[email protected]>
* Adding _list/shards API Signed-off-by: Harsh Garg <[email protected]>
* Update Apache Lucene to 9.12.0 Signed-off-by: Andriy Redko <[email protected]> * change to IOContext READONCE in locations where the file is not expected to be read multiple times. Signed-off-by: Marc Handalian <[email protected]> * Use READ IOContext for all non Segment* files when copying node-node Signed-off-by: Marc Handalian <[email protected]> * Fixing more test failures Signed-off-by: Andriy Redko <[email protected]> * Move Composite912Codec under org.opensearch.index.codec.composite.composite912 package Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Co-authored-by: Marc Handalian <[email protected]>
…4993) * Avoid deep copy and other allocation improvements * Refactoring based on PR Comments and added JavaDocs * Added more comments * Added character for Triggering Jenkins build * Changes to cover collectZeroDocEntries method * Updated comment based on change in method's functionality * Added test to cover branches in collectZeroDocEntriesIfRequired * Rebased and resolved changelog conflict --------- Signed-off-by: expani <[email protected]>
…ject#15925) * cancellation related Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * add better cancellation reason Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskCancellationTests.java Signed-off-by: Kiran Prakash <[email protected]> * refactor Signed-off-by: Kiran Prakash <[email protected]> * refactor Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskCancellation.java Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskCancellation.java Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskCancellation.java Signed-off-by: Kiran Prakash <[email protected]> * Update DefaultTaskSelectionStrategy.java Signed-off-by: Kiran Prakash <[email protected]> * refactor Signed-off-by: Kiran Prakash <[email protected]> * refactor node level threshold Signed-off-by: Kiran Prakash <[email protected]> * use query group task Signed-off-by: Kaushal Kumar <[email protected]> * code clean up and refactorings Signed-off-by: Kaushal Kumar <[email protected]> * add unit tests and fix existing ones Signed-off-by: Kaushal Kumar <[email protected]> * uncomment the test case Signed-off-by: Kaushal Kumar <[email protected]> * update CHANGELOG Signed-off-by: Kaushal Kumar <[email protected]> * fix imports Signed-off-by: Kaushal Kumar <[email protected]> * add queryGroupService Signed-off-by: Kaushal Kumar <[email protected]> * refactor and add UTs for new constructs Signed-off-by: Kaushal Kumar <[email protected]> * fix javadocs Signed-off-by: Kaushal Kumar <[email protected]> * remove code clutter Signed-off-by: Kaushal Kumar <[email protected]> * change annotation version and task selection strategy Signed-off-by: Kaushal Kumar <[email protected]> * rename a util class Signed-off-by: Kaushal Kumar <[email protected]> * remove wrappers from resource type Signed-off-by: Kaushal Kumar <[email protected]> * apply spotless Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * add rename changes Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * initial changes Signed-off-by: Kaushal Kumar <[email protected]> * refactor changes and logical bug fix Signed-off-by: Kaushal Kumar <[email protected]> * add chanegs Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * temp changes Signed-off-by: Kaushal Kumar <[email protected]> * add UTs Signed-off-by: Kaushal Kumar <[email protected]> * add changelog Signed-off-by: Kaushal Kumar <[email protected]> * add task completion listener hook Signed-off-by: Kaushal Kumar <[email protected]> * add remaining pieces to make the feature functional Signed-off-by: Kaushal Kumar <[email protected]> * extend stats and fix bugs Signed-off-by: Kaushal Kumar <[email protected]> * fix bugs and add logic to make SBP work with wlm Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * fix bugs and SBP ITs Signed-off-by: Kaushal Kumar <[email protected]> * add missed applyCluster state change Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * decouple queryGroupService and cancellationService Signed-off-by: Kaushal Kumar <[email protected]> * replace StateApplier with StateListener interface Signed-off-by: Kaushal Kumar <[email protected]> * fix precommit errors Signed-off-by: Kaushal Kumar <[email protected]> --------- Signed-off-by: Kiran Prakash <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]> Co-authored-by: Kiran Prakash <[email protected]>
* Segmented cache changes for TieredCache Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding change log Signed-off-by: Sagar Upadhyaya <[email protected]> * Allow segment number to be power of two Signed-off-by: Sagar Upadhyaya <[email protected]> * Moving common tiered cache IT methods to a common base class Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding disk took time IT test with multiple segment Signed-off-by: Sagar Upadhyaya <[email protected]> * Correcting changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing invalid segment count variable name Signed-off-by: Sagar Upadhyaya <[email protected]> * Introducing new settings for size for respective cache tier Signed-off-by: Sagar Upadhyaya <[email protected]> * Changing the default segmentCount logic Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing missing java doc issue Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]>
…project#15249) --------- Signed-off-by: Bharathwaj G <[email protected]>
* Refactoring builder tests Signed-off-by: Bharathwaj G <[email protected]> * adding date tests Signed-off-by: Bharathwaj G <[email protected]> --------- Signed-off-by: Bharathwaj G <[email protected]>
…#16110) * Change successfulSearchShardIndices to Set<Index> Signed-off-by: David Zane <[email protected]> * Update CHANGELOG.md Signed-off-by: Ankit Jain <[email protected]> --------- Signed-off-by: David Zane <[email protected]> Signed-off-by: Ankit Jain <[email protected]> Co-authored-by: Ankit Jain <[email protected]>
Signed-off-by: Gaurav Bafna <[email protected]>
…azure (opensearch-project#16217) * Bump com.azure:azure-json in /plugins/repository-azure Bumps [com.azure:azure-json](https://github.com/Azure/azure-sdk-for-java) from 1.1.0 to 1.3.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@v1.1.0...v1.3.0) --- updated-dependencies: - dependency-name: com.azure:azure-json dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…roject#16228) Signed-off-by: Sachin Kale <[email protected]>
gbbafna
force-pushed
the
dynamic-threadpool
branch
2 times, most recently
from
71df521
to
8d75efd
Compare
…es (opensearch-project#16237) * Remove force to TLSv1.2 in gradle.properties Signed-off-by: Craig Perkins <[email protected]> * Add TLSv1.3 Signed-off-by: Craig Perkins <[email protected]> --------- Signed-off-by: Craig Perkins <[email protected]>
…rch-project#16240) Signed-off-by: Sayali Gaikawad <[email protected]>
…16238) * Fix Cache breaking change Signed-off-by: Sagar Upadhyaya <[email protected]> * Empty commit to trigger build Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]>
…omposite99Codec (opensearch-project#16227) * Refactoring builder tests * composite912 codec refactor changes --------- Signed-off-by: Bharathwaj G <[email protected]>
…plugins/repository-gcs (opensearch-project#16216) * Bump com.google.api-client:google-api-client in /plugins/repository-gcs Bumps [com.google.api-client:google-api-client](https://github.com/googleapis/google-api-java-client) from 2.2.0 to 2.7.0. - [Release notes](https://github.com/googleapis/google-api-java-client/releases) - [Changelog](https://github.com/googleapis/google-api-java-client/blob/main/CHANGELOG.md) - [Commits](googleapis/google-api-java-client@v2.2.0...v2.7.0) --- updated-dependencies: - dependency-name: com.google.api-client:google-api-client dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: gaobinlong <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: gaobinlong <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
…s-fixture (opensearch-project#16212) * Bump com.squareup.okio:okio in /test/fixtures/hdfs-fixture Bumps [com.squareup.okio:okio](https://github.com/square/okio) from 3.9.0 to 3.9.1. - [Release notes](https://github.com/square/okio/releases) - [Changelog](https://github.com/square/okio/blob/master/CHANGELOG.md) - [Commits](square/okio@parent-3.9.0...3.9.1) --- updated-dependencies: - dependency-name: com.squareup.okio:okio dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…opensearch-project#16213) * Bump io.grpc:grpc-api from 1.57.2 to 1.68.0 in /plugins/discovery-gce Bumps [io.grpc:grpc-api](https://github.com/grpc/grpc-java) from 1.57.2 to 1.68.0. - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](grpc/grpc-java@v1.57.2...v1.68.0) --- updated-dependencies: - dependency-name: io.grpc:grpc-api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> * Centralize grpc version management Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Andriy Redko <[email protected]>
…6144) Signed-off-by: Finn Carroll <[email protected]>
…ch-project#16254) * The protobuf-java leaks through client library dependencies Signed-off-by: Andriy Redko <[email protected]> * Address code review comments Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: Andriy Redko <[email protected]>
…StoreFetchMultiNodeMultiIndexesUsingBatchAction (opensearch-project#16093) Signed-off-by: kkewwei <[email protected]>
…h-project#16250) Signed-off-by: Sooraj Sinha <[email protected]>
…search-project#16248) Signed-off-by: Sachin Kale <[email protected]>
…15777) * changelog Signed-off-by: Ruirui Zhang <[email protected]> * address comments Signed-off-by: Ruirui Zhang <[email protected]> * add tests Signed-off-by: Ruirui Zhang <[email protected]> * modify uri Signed-off-by: Ruirui Zhang <[email protected]> * address comments Signed-off-by: Ruirui Zhang <[email protected]> * modify based on comments Signed-off-by: Ruirui Zhang <[email protected]> * changelog Signed-off-by: Ruirui Zhang <[email protected]> * address comments Signed-off-by: Ruirui Zhang <[email protected]> * add tests Signed-off-by: Ruirui Zhang <[email protected]> * modify uri Signed-off-by: Ruirui Zhang <[email protected]> * modify based on comments Signed-off-by: Ruirui Zhang <[email protected]> * modify based on comments Signed-off-by: Ruirui Zhang <[email protected]> * revise Signed-off-by: Ruirui Zhang <[email protected]> * address comments Signed-off-by: Ruirui Zhang <[email protected]> * changelog Signed-off-by: Ruirui Zhang <[email protected]> * address comments Signed-off-by: Ruirui Zhang <[email protected]> * add tests Signed-off-by: Ruirui Zhang <[email protected]> * modify uri Signed-off-by: Ruirui Zhang <[email protected]> * modify based on comments Signed-off-by: Ruirui Zhang <[email protected]> * modify based on comments Signed-off-by: Ruirui Zhang <[email protected]> * git pull Signed-off-by: Ruirui Zhang <[email protected]> * rebase Signed-off-by: Ruirui Zhang <[email protected]> * encapsulate querygroupstats in wlmstats Signed-off-by: Ruirui Zhang <[email protected]> * fix UT Signed-off-by: Ruirui Zhang <[email protected]>
…roject#15955) * add IT for workload management CRUD APIs Signed-off-by: Ruirui Zhang <[email protected]> * modify based on comments Signed-off-by: Ruirui Zhang <[email protected]> * add IT for workload management CRUD APIs Signed-off-by: Ruirui Zhang <[email protected]>
* Reconfigure remote state thread pool count Signed-off-by: Sooraj Sinha <[email protected]>
…ch-project#16267) Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
* fix wlm stats output Signed-off-by: Kaushal Kumar <[email protected]> * rename wlm stats vars Signed-off-by: Kaushal Kumar <[email protected]> * fix ut failure Signed-off-by: Kaushal Kumar <[email protected]> --------- Signed-off-by: Kaushal Kumar <[email protected]>
…rch-project#16281) Signed-off-by: Sachin Kale <[email protected]>
gbbafna
force-pushed
the
dynamic-threadpool
branch
from
07c95a7
to
4bedb9c
Compare
Signed-off-by: Gaurav Bafna <[email protected]>
Signed-off-by: Gaurav Bafna <[email protected]>
Signed-off-by: Gaurav Bafna <[email protected]>
… listener on it Signed-off-by: Gaurav Bafna <[email protected]>
gbbafna
force-pushed
the
dynamic-threadpool
branch
from
b161100
to
cc11f7e
Compare
Signed-off-by: Gaurav Bafna <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
[Describe what this change achieves]
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.