Merge branch 'main' into 4967_prevent_deletion_of_snapshot_backing_index

Signed-off-by: Vishal Sarda <[email protected]>

.github/workflows/backport.yml

@@ -7,6 +7,7 @@ on:
 
 jobs:
   backport:
+    if: ${{ contains(github.event.label.name, 'backport') }}
     runs-on: ubuntu-latest
     permissions:
       contents: write
@@ -26,4 +27,3 @@ jobs:
         with:
           github_token: ${{ steps.github_app_token.outputs.token }}
           head_template: backport/backport-<%= number %>-to-<%= base %>
-          files_to_skip: 'CHANGELOG.md'

.github/workflows/changelog_verifier.yml

@@ -15,4 +15,4 @@ jobs:
 
       - uses: dangoslen/changelog-enforcer@v3
         with:
-          skipLabels: "autocut"
+          skipLabels: "autocut, skip-changelog"

CONTRIBUTING.md

@@ -119,20 +119,39 @@ You may type this line on your own when writing your commit messages. However, i
 
 ## Changelog
 
-OpenSearch maintains version specific changelog by enforcing a change to the ongoing [CHANGELOG](CHANGELOG.md) file adhering to the [Keep A Changelog](https://keepachangelog.com/en/1.0.0/) format.
+OpenSearch maintains version specific changelog by enforcing a change to the ongoing [CHANGELOG](CHANGELOG.md) file adhering to the [Keep A Changelog](https://keepachangelog.com/en/1.0.0/) format. The purpose of the changelog is for the contributors and maintainers to incrementally build the release notes throughout the development process to avoid a painful and error-prone process of attempting to compile the release notes at release time. On each release the "unreleased" entries of the changelog are moved to the appropriate release notes document in the `./release-notes` folder. Also, incrementally building the changelog provides a concise, human-readable list of significant features that have been added to the unreleased version under development.
 
-Briefly, the changes are curated by version, with the changes to the main branch added chronologically to `Unreleased` version. Further, each version has corresponding sections which list out the category of the change - `Added`, `Changed`, `Deprecated`, `Removed`, `Fixed`, `Security`.
+### Which changes require a CHANGELOG entry?
+Changelogs are intended for operators/administrators, developers integrating with libraries and APIs, and end-users interacting with OpenSearch Dashboards and/or the REST API (collectively referred to as "user"). In short, any change that a user of OpenSearch might want to be aware of should be included in the changelog. The changelog is _not_ intended to replace the git commit log that developers of OpenSearch itself rely upon. The following are some examples of changes that should be in the changelog:
 
+- A newly added feature
+- A fix for a user-facing bug
+- Dependency updates
+- Fixes for security issues
 
-### How to add my changes to [CHANGELOG](CHANGELOG.md)?
+The following are some examples where a changelog entry is not necessary:
 
-As a contributor, you must ensure that every pull request has the changes listed out within the corresponding version and appropriate section of [CHANGELOG](CHANGELOG.md) file.
+- Adding, modifying, or fixing tests
+- An incremental PR for a larger feature (such features should include _one_ changelog entry for the feature)
+- Documentation changes or code refactoring
+- Build-related changes
 
-Adding in the change is two step process -
-1. Add your changes to the corresponding section within the CHANGELOG file with dummy pull request information, publish the PR
+Any PR that does not include a changelog entry will result in a failure of the validation workflow in GitHub. If the contributor and maintainers agree that no changelog entry is required, then the `skip-changelog` label can be applied to the PR which will result in the workflow passing.
+
+### How to add my changes to [CHANGELOG](CHANGELOG.md)?
 
+Adding in the change is two step process:
+1. Add your changes to the corresponding section within the CHANGELOG file with dummy pull request information, publish the PR
 2. Update the entry for your change in [`CHANGELOG.md`](CHANGELOG.md) and make sure that you reference the pull request there.
 
+### Where should I put my CHANGELOG entry?
+Please review the [branching strategy](https://github.com/opensearch-project/.github/blob/main/RELEASING.md#opensearch-branching) document. The changelog on the `main` branch will contain sections for the _next major_ and _next minor_ releases. Your entry should go into the section it is intended to be released in. In practice, most changes to `main` will be backported to the next minor release so most entries will likely be in that section.
+
+The following examples assume the _next major_ release on main is 3.0, then _next minor_ release is 2.5, and the _current_ release is 2.4.
+
+- **Add a new feature to release in next minor:** Add a changelog entry to `[Unreleased 2.x]` on main, then backport to 2.x (including the changelog entry).
+- **Introduce a breaking API change to release in next major:** Add a changelog entry to `[Unreleased 3.0]` on main, do not backport.
+- **Upgrade a dependency to fix a CVE:** Add a changelog entry to `[Unreleased 2.x]` on main, then backport to 2.x (including the changelog entry), then backport to 2.4 and ensure the changelog entry is added to `[Unreleased 2.4.1]`.
 
 ## Review Process
 

build.gradle

@@ -316,6 +316,7 @@ allprojects {
     javadoc.options.encoding = 'UTF8'
     javadoc.options.addStringOption('Xdoclint:all,-missing', '-quiet')
     javadoc.options.tags = ["opensearch.internal", "opensearch.api", "opensearch.experimental"]
+    javadoc.options.addStringOption("-release", targetCompatibility.majorVersion)
   }
 
   // support for reproducible builds
@@ -412,6 +413,9 @@ gradle.projectsEvaluated {
         if (BuildParams.runtimeJavaVersion > JavaVersion.VERSION_17) {
           task.jvmArgs += ["-Djava.security.manager=allow"]
         }
+        if (BuildParams.runtimeJavaVersion >= JavaVersion.VERSION_19) {
+          task.jvmArgs += ["--enable-preview"]
+        }
       }
     }
 

buildSrc/build.gradle

@@ -106,15 +106,15 @@ dependencies {
   api 'org.apache.commons:commons-compress:1.21'
   api 'org.apache.ant:ant:1.10.12'
   api 'com.netflix.nebula:gradle-extra-configurations-plugin:7.0.0'
-  api 'com.netflix.nebula:nebula-publishing-plugin:4.4.4'
+  api 'com.netflix.nebula:nebula-publishing-plugin:4.6.0'
   api 'com.netflix.nebula:gradle-info-plugin:11.3.3'
   api 'org.apache.rat:apache-rat:0.13'
   api 'commons-io:commons-io:2.7'
   api "net.java.dev.jna:jna:5.11.0"
   api 'gradle.plugin.com.github.johnrengelman:shadow:7.1.2'
   api 'org.jdom:jdom2:2.0.6.1'
   api "org.jetbrains.kotlin:kotlin-stdlib-jdk8:${props.getProperty('kotlin')}"
-  api 'de.thetaphi:forbiddenapis:3.3'
+  api 'de.thetaphi:forbiddenapis:3.4'
   api 'com.avast.gradle:gradle-docker-compose-plugin:0.15.2'
   api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}"
   api 'org.apache.maven:maven-model:3.6.2'

buildSrc/src/main/java/org/opensearch/gradle/Jdk.java

@@ -37,6 +37,7 @@
 import org.gradle.api.model.ObjectFactory;
 import org.gradle.api.provider.Property;
 import org.gradle.api.tasks.TaskDependency;
+import org.gradle.internal.os.OperatingSystem;
 
 import java.io.File;
 import java.util.Arrays;
@@ -169,7 +170,7 @@ public Object getBinJavaPath() {
         return new Object() {
             @Override
             public String toString() {
-                return getHomeRoot() + "/bin/java";
+                return OperatingSystem.current().getExecutableName(getHomeRoot() + "/bin/java");
             }
         };
     }

buildSrc/src/main/java/org/opensearch/gradle/OpenSearchTestBasePlugin.java

@@ -111,6 +111,9 @@ public void execute(Task t) {
                         test.systemProperty("java.locale.providers", "SPI,COMPAT");
                         test.jvmArgs("--illegal-access=warn");
                     }
+                    if (test.getJavaVersion().compareTo(JavaVersion.VERSION_17) > 0) {
+                        test.jvmArgs("-Djava.security.manager=allow");
+                    }
                 }
             });
             test.getJvmArgumentProviders().add(nonInputProperties);

buildSrc/src/main/java/org/opensearch/gradle/info/GlobalBuildInfoPlugin.java

@@ -45,12 +45,15 @@
 import org.gradle.internal.jvm.Jvm;
 import org.gradle.internal.jvm.inspection.JvmInstallationMetadata;
 import org.gradle.internal.jvm.inspection.JvmMetadataDetector;
+import org.gradle.jvm.toolchain.internal.InstallationLocation;
 import org.gradle.util.GradleVersion;
 
 import javax.inject.Inject;
 import java.io.File;
 import java.io.FileInputStream;
 import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
 import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.nio.file.Path;
@@ -195,7 +198,29 @@ private JavaVersion determineJavaVersion(String description, File javaHome, Java
     }
 
     private JvmInstallationMetadata getJavaInstallation(File javaHome) {
-        return jvmMetadataDetector.getMetadata(javaHome);
+        final InstallationLocation location = new InstallationLocation(javaHome, "Java home");
+
+        try {
+            try {
+                // The getMetadata(File) is used by Gradle pre-7.6
+                return (JvmInstallationMetadata) MethodHandles.publicLookup()
+                    .findVirtual(JvmMetadataDetector.class, "getMetadata", MethodType.methodType(JvmInstallationMetadata.class, File.class))
+                    .bindTo(jvmMetadataDetector)
+                    .invokeExact(location.getLocation());
+            } catch (NoSuchMethodException | IllegalAccessException ex) {
+                // The getMetadata(InstallationLocation) is used by Gradle post-7.6
+                return (JvmInstallationMetadata) MethodHandles.publicLookup()
+                    .findVirtual(
+                        JvmMetadataDetector.class,
+                        "getMetadata",
+                        MethodType.methodType(JvmInstallationMetadata.class, InstallationLocation.class)
+                    )
+                    .bindTo(jvmMetadataDetector)
+                    .invokeExact(location);
+            }
+        } catch (Throwable ex) {
+            throw new IllegalStateException("Unable to find suitable JvmMetadataDetector::getMetadata", ex);
+        }
     }
 
     private List<JavaHome> getAvailableJavaVersions(JavaVersion minimumCompilerVersion) {
@@ -205,7 +230,7 @@ private List<JavaHome> getAvailableJavaVersions(JavaVersion minimumCompilerVersi
             String javaHomeEnvVarName = getJavaHomeEnvVarName(Integer.toString(version));
             if (System.getenv(javaHomeEnvVarName) != null) {
                 File javaHomeDirectory = new File(findJavaHome(Integer.toString(version)));
-                JvmInstallationMetadata javaInstallation = jvmMetadataDetector.getMetadata(javaHomeDirectory);
+                JvmInstallationMetadata javaInstallation = getJavaInstallation(javaHomeDirectory);
                 JavaHome javaHome = JavaHome.of(version, providers.provider(() -> {
                     int actualVersion = Integer.parseInt(javaInstallation.getLanguageVersion().getMajorVersion());
                     if (actualVersion != version) {

buildSrc/src/main/java/org/opensearch/gradle/precommit/ThirdPartyAuditPrecommitPlugin.java

@@ -51,7 +51,7 @@ public class ThirdPartyAuditPrecommitPlugin extends PrecommitPlugin {
     public TaskProvider<? extends Task> createTask(Project project) {
         project.getPlugins().apply(CompileOnlyResolvePlugin.class);
         project.getConfigurations().create("forbiddenApisCliJar");
-        project.getDependencies().add("forbiddenApisCliJar", "de.thetaphi:forbiddenapis:3.2");
+        project.getDependencies().add("forbiddenApisCliJar", "de.thetaphi:forbiddenapis:3.4");
 
         Configuration jdkJarHellConfig = project.getConfigurations().create(JDK_JAR_HELL_CONFIG_NAME);
         if (BuildParams.isInternal() && project.getPath().equals(":libs:opensearch-core") == false) {

buildSrc/version.properties

@@ -2,15 +2,14 @@ opensearch        = 3.0.0
 lucene            = 9.5.0-snapshot-a4ef70f
 
 bundled_jdk_vendor = adoptium
-bundled_jdk = 17.0.5+8
-
+bundled_jdk = 19.0.1+10
 
 
 # optional dependencies
 spatial4j         = 0.7
 jts               = 1.15.0
-jackson           = 2.13.4
-jackson_databind  = 2.13.4.2
+jackson           = 2.14.0
+jackson_databind  = 2.14.0
 snakeyaml         = 1.32
 icu4j             = 70.1
 supercsv          = 2.4.0

client/rest/src/main/java/org/opensearch/client/Response.java

@@ -147,7 +147,7 @@ public HttpEntity getEntity() {
      * @return {@code true} if the input string matches the specification
      */
     private static boolean matchWarningHeaderPatternByPrefix(final String s) {
-        return s.startsWith("299 OpenSearch-") || s.startsWith("299 Elasticsearch-");
+        return s.startsWith("299 OpenSearch-");
     }
 
     /**

client/sniffer/licenses/jackson-core-2.14.0.jar.sha1

@@ -0,0 +1 @@
+49d219171d6af643e061e9e1baaaf6a6a067918d

distribution/src/config/jvm.options

@@ -78,3 +78,6 @@ ${error.file}
 
 # Explicitly allow security manager (https://bugs.openjdk.java.net/browse/JDK-8270380)
 18-:-Djava.security.manager=allow
+
+# Allow mmap to use new JDK-19 preview APIs in Apache Lucene 9.4 (https://github.com/opensearch-project/OpenSearch/issues/4637)
+19-:--enable-preview

distribution/tools/launchers/src/main/java/org/opensearch/tools/launchers/SystemJvmOptions.java

@@ -77,12 +77,21 @@ static List<String> systemJvmOptions() {
                 // log4j 2
                 "-Dlog4j.shutdownHookEnabled=false",
                 "-Dlog4j2.disable.jmx=true",
-
+                // security manager
+                allowSecurityManagerOption(),
                 javaLocaleProviders()
             )
         ).stream().filter(e -> e.isEmpty() == false).collect(Collectors.toList());
     }
 
+    private static String allowSecurityManagerOption() {
+        if (Runtime.version().feature() > 17) {
+            return "-Djava.security.manager=allow";
+        } else {
+            return "";
+        }
+    }
+
     private static String maybeShowCodeDetailsInExceptionMessages() {
         if (Runtime.version().feature() >= 14) {
             return "-XX:+ShowCodeDetailsInExceptionMessages";

distribution/tools/plugin-cli/build.gradle

@@ -35,7 +35,7 @@ archivesBaseName = 'opensearch-plugin-cli'
 dependencies {
   compileOnly project(":server")
   compileOnly project(":libs:opensearch-cli")
-  api "org.bouncycastle:bcpg-fips:1.0.5.1"
+  api "org.bouncycastle:bcpg-fips:1.0.7.1"
   api "org.bouncycastle:bc-fips:1.0.2.3"
   testImplementation project(":test:framework")
   testImplementation 'com.google.jimfs:jimfs:1.2'

distribution/tools/plugin-cli/licenses/bcpg-fips-1.0.7.1.jar.sha1

@@ -0,0 +1 @@
+5e1952428655ea822066f86df2e3ecda8fa0ba2b

distribution/tools/upgrade-cli/licenses/jackson-annotations-2.14.0.jar.sha1

@@ -0,0 +1 @@
+fb7afb3c9c8ea363a9c88ea9c0a7177cf2fbd369

distribution/tools/upgrade-cli/licenses/jackson-databind-2.14.0.jar.sha1

@@ -0,0 +1 @@
+513b8ca3fea0352ceebe4d0bbeea527ab343dc1a

gradle/runtime-jdk-provision.gradle

@@ -20,7 +20,11 @@ if (BuildParams.getIsRuntimeJavaHomeSet()) {
   configure(allprojects - project(':build-tools')) {
     project.tasks.withType(Test).configureEach { Test test ->
       if (BuildParams.getIsRuntimeJavaHomeSet()) {
-        test.executable = "${BuildParams.runtimeJavaHome}/bin/java"
+        if (OS.current() == OS.WINDOWS) {
+          test.executable = "${BuildParams.runtimeJavaHome}/bin/java.exe"
+        } else {
+          test.executable = "${BuildParams.runtimeJavaHome}/bin/java"
+        }
       }
     }
   }

gradle/wrapper/gradle-wrapper.properties

@@ -11,7 +11,7 @@
 
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-7.5.1-all.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-7.6-rc-3-all.zip
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
-distributionSha256Sum=db9c8211ed63f61f60292c69e80d89196f9eb36665e369e7f00ac4cc841c2219
+distributionSha256Sum=abc6de2653ec9befb00cc0d064ce1ca4e4dab2c91955e830661505189c0b2f08

libs/x-content/licenses/jackson-core-2.14.0.jar.sha1

@@ -0,0 +1 @@
+49d219171d6af643e061e9e1baaaf6a6a067918d

libs/x-content/licenses/jackson-dataformat-cbor-2.14.0.jar.sha1

@@ -0,0 +1 @@
+68831fbd18bffd2ecb0eaf3ea75c95d94cfb940d

libs/x-content/licenses/jackson-dataformat-smile-2.14.0.jar.sha1

@@ -0,0 +1 @@
+7646180c97a3a2f6a4e63c0725dfb96d3d15353b

libs/x-content/licenses/jackson-dataformat-yaml-2.14.0.jar.sha1

@@ -0,0 +1 @@
+06c635ef06d3e4e72a7e9868da41ffa1a0f98d28

modules/analysis-common/src/main/java/org/opensearch/analysis/common/HyphenationCompoundWordTokenFilterFactory.java

@@ -32,12 +32,14 @@
 
 package org.opensearch.analysis.common;
 
+import org.apache.logging.log4j.LogManager;
 import org.apache.lucene.analysis.TokenStream;
 import org.apache.lucene.analysis.compound.HyphenationCompoundWordTokenFilter;
 import org.apache.lucene.analysis.compound.hyphenation.HyphenationTree;
 import org.opensearch.common.settings.Settings;
 import org.opensearch.env.Environment;
 import org.opensearch.index.IndexSettings;
+import org.opensearch.index.analysis.Analysis;
 import org.xml.sax.InputSource;
 
 import java.io.InputStream;
@@ -61,13 +63,15 @@ public class HyphenationCompoundWordTokenFilterFactory extends AbstractCompoundW
             throw new IllegalArgumentException("hyphenation_patterns_path is a required setting.");
         }
 
-        Path hyphenationPatternsFile = env.configDir().resolve(hyphenationPatternsPath);
+        Path hyphenationPatternsFile = Analysis.resolveAnalyzerPath(env, hyphenationPatternsPath);
 
         try {
             InputStream in = Files.newInputStream(hyphenationPatternsFile);
             hyphenationTree = HyphenationCompoundWordTokenFilter.getHyphenationTree(new InputSource(in));
         } catch (Exception e) {
-            throw new IllegalArgumentException("Exception while reading hyphenation_patterns_path.", e);
+            LogManager.getLogger(HyphenationCompoundWordTokenFilterFactory.class)
+                .error("Exception while reading hyphenation_patterns_path ", e);
+            throw new IllegalArgumentException("Exception while reading hyphenation_patterns_path.");
         }
     }