From eaf698b8858a2d9c0b7931ba826099dc2d27db4d Mon Sep 17 00:00:00 2001 From: dblock Date: Thu, 8 Jul 2021 13:17:15 -0500 Subject: [PATCH 01/11] Removed docs, this folder no longer exists. Signed-off-by: dblock --- DEVELOPER_GUIDE.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 252080fce0ffa..efc3c276b8d18 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -11,7 +11,6 @@ - [Visual Studio Code](#visual-studio-code) - [Eclipse](#eclipse) - [Project Layout](#project-layout) - - [`docs`](#docs) - [`distribution`](#distribution) - [`libs`](#libs) - [`modules`](#modules) @@ -154,10 +153,6 @@ We would like to support Eclipse, but few of us use it and has fallen into disre This repository is split into many top level directories. The most important ones are: -### `docs` - -Documentation for the project. - ### `distribution` Builds our tar and zip archives and our rpm and deb packages. From 209fe0754d20d148b0d6d55f76c31aa25b6342ea Mon Sep 17 00:00:00 2001 From: dblock Date: Thu, 8 Jul 2021 13:20:38 -0500 Subject: [PATCH 02/11] Adjusted links to documentation-website. Signed-off-by: dblock --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 86597c6dc02af..46807b5665629 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -53,7 +53,7 @@ If you've thought of a way that OpenSearch could be better, we want to hear abou ### Documentation Changes -If you would like to contribute to the documentation, please see [OpenSearch Docs repo](https://github.com/opensearch-project/documentation-website/blob/main/README.md). To contribute javadocs, please see [this open issue on github](https://github.com/opensearch-project/OpenSearch/issues/221). +If you would like to contribute to the documentation, please do so in the [documentation-website](https://github.com/opensearch-project/documentation-website) repo. To contribute javadocs, please first check [OpenSearch#221](https://github.com/opensearch-project/OpenSearch/issues/221). ### Contributing Code From 3503b1ed556673125589beca82b56b02dcb03132 Mon Sep 17 00:00:00 2001 From: dblock Date: Thu, 8 Jul 2021 15:37:20 -0500 Subject: [PATCH 03/11] Removed references to x-pack. Signed-off-by: dblock --- TESTING.md | 8 ++++---- buildSrc/version.properties | 1 - qa/translog-policy/build.gradle | 2 +- .../cluster/metadata/IndexNameExpressionResolver.java | 2 +- 4 files changed, 6 insertions(+), 7 deletions(-) diff --git a/TESTING.md b/TESTING.md index 1fd4d5fc9ce68..abb56d61751c2 100644 --- a/TESTING.md +++ b/TESTING.md @@ -372,11 +372,11 @@ A specific version can be tested as well. For example, to test bwc with version ./gradlew v5.3.2#bwcTest -Use -Dtest.class and -Dtests.method to run a specific bwcTest test. For example to run a specific tests from the x-pack rolling upgrade from 7.7.0: +Use -Dtest.class and -Dtests.method to run a specific bwcTest test. For example to test a rolling upgrade from 7.7.0: - ./gradlew :x-pack:qa:rolling-upgrade:v7.7.0#bwcTest \ - -Dtests.class=org.opensearch.upgrades.UpgradeClusterClientYamlTestSuiteIT \ - -Dtests.method="test {p0=*/40_ml_datafeed_crud/*}" + ./gradlew :qa:rolling-upgrade:v7.7.0#bwcTest \ + -Dtests.class=org.opensearch.upgrades.RecoveryIT \ + -Dtests.method=testHistoryUUIDIsGenerated Tests are run for versions that are not yet released but with which the current version will be compatible with. These are automatically checked out and built from source. See [VersionCollection](./buildSrc/src/main/java/org/opensearch/gradle/VersionCollection.java) and [distribution/bwc/build.gradle](./distribution/bwc/build.gradle) for more information. diff --git a/buildSrc/version.properties b/buildSrc/version.properties index d9ab87ba5ecdc..07e40a22b6c03 100644 --- a/buildSrc/version.properties +++ b/buildSrc/version.properties @@ -26,7 +26,6 @@ joda = 2.10.4 # when updating this version, you need to ensure compatibility with: # - plugins/ingest-attachment (transitive dependency, check the upstream POM) # - distribution/tools/plugin-cli -# - x-pack/plugin/security bouncycastle=1.64 # test dependencies randomizedrunner = 2.7.1 diff --git a/qa/translog-policy/build.gradle b/qa/translog-policy/build.gradle index 6ceba19cebdd9..8e161ba3fe1a3 100644 --- a/qa/translog-policy/build.gradle +++ b/qa/translog-policy/build.gradle @@ -40,7 +40,7 @@ apply from : "$rootDir/gradle/bwc-test.gradle" for (Version bwcVersion : BuildParams.bwcVersions.indexCompatible) { if (bwcVersion.before('6.3.0')) { // explicitly running restart on the current node does not work in step 2 - // below when plugins are installed, wihch is the case for x-pack as a plugin + // below when plugins are installed, which is the case for some plugins // prior to 6.3.0 continue } diff --git a/server/src/main/java/org/opensearch/cluster/metadata/IndexNameExpressionResolver.java b/server/src/main/java/org/opensearch/cluster/metadata/IndexNameExpressionResolver.java index fb3d854a7de23..e130bf95feef5 100644 --- a/server/src/main/java/org/opensearch/cluster/metadata/IndexNameExpressionResolver.java +++ b/server/src/main/java/org/opensearch/cluster/metadata/IndexNameExpressionResolver.java @@ -362,7 +362,7 @@ private static boolean addIndex(IndexMetadata metadata, Context context) { // trappy to hide throttled indices by default. In order to avoid breaking backward compatibility, // we changed it to look at the `index.frozen` setting instead, since frozen indices were the only // type of index to use the `search_throttled` threadpool at that time. - // NOTE: We can't reference the Setting object, which is only defined and registered in x-pack. + // NOTE: The Setting object was defined in an external plugin prior to OpenSearch fork. return (context.options.ignoreThrottled() && metadata.getSettings().getAsBoolean("index.frozen", false)) == false; } From 783ed88e78807455c527ee4f5dd72488ad19eeaa Mon Sep 17 00:00:00 2001 From: dblock Date: Thu, 8 Jul 2021 15:49:50 -0500 Subject: [PATCH 04/11] Fixed broken link, VersionCollection was renamed to BwcVersions. Signed-off-by: dblock --- TESTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TESTING.md b/TESTING.md index abb56d61751c2..b9fedf26da21b 100644 --- a/TESTING.md +++ b/TESTING.md @@ -378,7 +378,7 @@ Use -Dtest.class and -Dtests.method to run a specific bwcTest test. For example -Dtests.class=org.opensearch.upgrades.RecoveryIT \ -Dtests.method=testHistoryUUIDIsGenerated -Tests are run for versions that are not yet released but with which the current version will be compatible with. These are automatically checked out and built from source. See [VersionCollection](./buildSrc/src/main/java/org/opensearch/gradle/VersionCollection.java) and [distribution/bwc/build.gradle](./distribution/bwc/build.gradle) for more information. +Tests are run for versions that are not yet released but with which the current version will be compatible with. These are automatically checked out and built from source. See [BwcVersions](./buildSrc/src/main/java/org/opensearch/gradle/BwcVersions.java) and [distribution/bwc/build.gradle](./distribution/bwc/build.gradle) for more information. When running `./gradlew check`, minimal bwc checks are also run against compatible versions that are not yet released. From 92e7fe4ada905c08d0776c67366696f9714d31c5 Mon Sep 17 00:00:00 2001 From: dblock Date: Thu, 8 Jul 2021 15:51:13 -0500 Subject: [PATCH 05/11] Removed link to Elasticsearch issue. Signed-off-by: dblock --- TESTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TESTING.md b/TESTING.md index b9fedf26da21b..4e3663a2da8b1 100644 --- a/TESTING.md +++ b/TESTING.md @@ -424,7 +424,7 @@ In short, most new functionality should come with unit tests, and optionally RES ### Refactor code to make it easier to test -Unfortunately, a large part of our code base is still hard to unit test. Sometimes because some classes have lots of dependencies that make them hard to instantiate. Sometimes because API contracts make tests hard to write. Code refactors that make functionality easier to unit test are encouraged. If this sounds very abstract to you, you can have a look at [this pull request](https://github.com/elastic/elasticsearch/pull/16610) for instance, which is a good example. It refactors `IndicesRequestCache` in such a way that: - it no longer depends on objects that are hard to instantiate such as `IndexShard` or `SearchContext`, - time-based eviction is applied on top of the cache rather than internally, which makes it easier to assert on what the cache is expected to contain at a given time. +Unfortunately, a large part of our code base is still hard to unit test. Sometimes because some classes have lots of dependencies that make them hard to instantiate. Sometimes because API contracts make tests hard to write. Code refactors that make functionality easier to unit test are encouraged. ## Bad practices From 14e102d78ec1d700041f8ed00843575cebbb03b6 Mon Sep 17 00:00:00 2001 From: dblock Date: Thu, 8 Jul 2021 15:53:37 -0500 Subject: [PATCH 06/11] Removed reference to past JaCoCo coverage. Signed-off-by: dblock --- TESTING.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/TESTING.md b/TESTING.md index 4e3663a2da8b1..05ac1b45db2d4 100644 --- a/TESTING.md +++ b/TESTING.md @@ -442,9 +442,7 @@ Multi-threaded tests are often not reproducible due to the fact that there is no Generating test coverage reports for OpenSearch is currently not possible through Gradle. However, it *is* possible to gain insight in code coverage using IntelliJ’s built-in coverage analysis tool that can measure coverage upon executing specific tests. Eclipse may also be able to do the same using the EclEmma plugin. -Test coverage reporting used to be possible with JaCoCo when OpenSearch was using Maven as its build system. Since the switch to Gradle though, this is no longer possible, seeing as the code currently used to build OpenSearch does not allow JaCoCo to recognize its tests. For more information on this, see the discussion in [issue #28867](https://github.com/elastic/elasticsearch/issues/28867). - -Read your IDE documentation for how to attach a debugger to a JVM process. +Please read your IDE documentation for how to attach a debugger to a JVM process. # Building with extra plugins From d207e937c71b9919283eeec4d3109814920f0623 Mon Sep 17 00:00:00 2001 From: dblock Date: Thu, 8 Jul 2021 15:54:43 -0500 Subject: [PATCH 07/11] Use JDK 11 for running tests at a minimum. Signed-off-by: dblock --- TESTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TESTING.md b/TESTING.md index 05ac1b45db2d4..33989ee349fe5 100644 --- a/TESTING.md +++ b/TESTING.md @@ -42,7 +42,7 @@ You will need the following pieces of software to run these tests: - Docker & Docker Compose - Vagrant -- JDK 14 +- JDK 11 - Gradle # Creating packages From 57b0169a973f33a5a0c04779cfcbe1237916a7e3 Mon Sep 17 00:00:00 2001 From: dblock Date: Thu, 8 Jul 2021 15:56:05 -0500 Subject: [PATCH 08/11] Quoted opensearch-extra. Signed-off-by: dblock --- TESTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TESTING.md b/TESTING.md index 33989ee349fe5..a184a9e4bdb83 100644 --- a/TESTING.md +++ b/TESTING.md @@ -446,7 +446,7 @@ Please read your IDE documentation for how to attach a debugger to a JVM process # Building with extra plugins -Additional plugins may be built alongside OpenSearch, where their dependency on OpenSearch will be substituted with the local OpenSearch build. To add your plugin, create a directory called opensearch-extra as a sibling of OpenSearch. Checkout your plugin underneath opensearch-extra and the build will automatically pick it up. You can verify the plugin is included as part of the build by checking the projects of the build. +Additional plugins may be built alongside OpenSearch, where their dependency on OpenSearch will be substituted with the local OpenSearch build. To add your plugin, create a directory called `opensearch-extra` as a sibling of OpenSearch. Checkout your plugin underneath `opensearch-extra` and the build will automatically pick it up. You can verify the plugin is included as part of the build by checking the projects of the build. ./gradlew projects From 06cd2e34eea8b82e872b8d4a5a411069540fbba7 Mon Sep 17 00:00:00 2001 From: dblock Date: Thu, 8 Jul 2021 15:57:16 -0500 Subject: [PATCH 09/11] Removed officially in favor of built-in. Signed-off-by: dblock --- DEVELOPER_GUIDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index efc3c276b8d18..5c02403c783c8 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -170,7 +170,7 @@ For example, reindex requires the `connect` permission so it can perform reindex ### `plugins` -Officially supported plugins to OpenSearch. We decide that a feature should be a plugin rather than shipped as a module because we feel that it is only important to a subset of users, especially if it requires extra dependencies. +Built-in plugins to OpenSearch. We decide that a feature should be a plugin rather than shipped as a module because we feel that it is only important to a subset of users, especially if it requires extra dependencies. The canonical example of this is the ICU analysis plugin. It is important for folks who want the fairly language neutral ICU analyzer but the library to implement the analyzer is 11MB so we don't ship it with OpenSearch by default. From 3b33b91f6819d5f93a75d5dd576827ecbc678703 Mon Sep 17 00:00:00 2001 From: dblock Date: Thu, 8 Jul 2021 15:58:57 -0500 Subject: [PATCH 10/11] Removed misleading benchmarking docs. Signed-off-by: dblock --- TESTING.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/TESTING.md b/TESTING.md index a184a9e4bdb83..6e72ceb063ac6 100644 --- a/TESTING.md +++ b/TESTING.md @@ -34,7 +34,6 @@ OpenSearch uses [jUnit](https://junit.org/junit5/) for testing, it also uses ran - [Test coverage analysis](#test-coverage-analysis) - [Building with extra plugins](#building-with-extra-plugins) - [Environment misc](#environment-misc) -- [Benchmarking](#benchmarking) # Requirements @@ -457,7 +456,3 @@ There is a known issue with macOS localhost resolve strategy that can cause some 127.0.0.1 localhost OpenSearchMBP.local 255.255.255.255 broadcasthost ::1 localhost OpenSearchMBP.local` - -# Benchmarking - -For changes that might affect the performance characteristics of OpenSearch you should also run macrobenchmarks. There is also a macrobenchmarking tool called [Rally](https://github.com/elastic/rally) which you can use to measure the performance impact. To get started, please see [Rally’s documentation](https://esrally.readthedocs.io/en/stable/). From c5c52cb3095c60622ea279236a29721c71492e13 Mon Sep 17 00:00:00 2001 From: dblock Date: Mon, 12 Jul 2021 13:01:01 -0500 Subject: [PATCH 11/11] Just say plugins. Signed-off-by: dblock --- DEVELOPER_GUIDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 5c02403c783c8..14b219ea07ed8 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -170,7 +170,7 @@ For example, reindex requires the `connect` permission so it can perform reindex ### `plugins` -Built-in plugins to OpenSearch. We decide that a feature should be a plugin rather than shipped as a module because we feel that it is only important to a subset of users, especially if it requires extra dependencies. +OpenSearch plugins. We decide that a feature should be a plugin rather than shipped as a module because we feel that it is only important to a subset of users, especially if it requires extra dependencies. The canonical example of this is the ICU analysis plugin. It is important for folks who want the fairly language neutral ICU analyzer but the library to implement the analyzer is 11MB so we don't ship it with OpenSearch by default.