Merge branch 'main' into replicationexception

CHANGELOG.md

@@ -3,12 +3,7 @@
 Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ## [Unreleased]
-### Dependencies
-- Bumps `gson` from 2.9.0 to 2.9.1
-- Bumps `protobuf-java` from 3.21.2 to 3.21.7
-
 ### Added
-
 - Add support for s390x architecture ([#4001](https://github.com/opensearch-project/OpenSearch/pull/4001))
 - Github workflow for changelog verification ([#4085](https://github.com/opensearch-project/OpenSearch/pull/4085))
 - Point in time rest layer changes for create and delete PIT API ([#4064](https://github.com/opensearch-project/OpenSearch/pull/4064))
@@ -33,16 +28,18 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Added precommit support for MacOS ([#4682](https://github.com/opensearch-project/OpenSearch/pull/4682))
 - Recommission API changes for service layer ([#4320](https://github.com/opensearch-project/OpenSearch/pull/4320))
 - Update GeoGrid base class access modifier to support extensibility ([#4572](https://github.com/opensearch-project/OpenSearch/pull/4572))
-
+- Add a new node role 'search' which is dedicated to provide search capability ([#4689](https://github.com/opensearch-project/OpenSearch/pull/4689))
 ### Dependencies
 - Bumps `log4j-core` from 2.18.0 to 2.19.0
 - Bumps `reactor-netty-http` from 1.0.18 to 1.0.23
 - Bumps `jettison` from 1.5.0 to 1.5.1
 - Bumps `azure-storage-common` from 12.18.0 to 12.18.1
 - Bumps `forbiddenapis` from 3.3 to 3.4
-
-
-### Dependencies
+- Bumps `gson` from 2.9.0 to 2.9.1
+- Bumps `protobuf-java` from 3.21.2 to 3.21.7
+- Bumps `azure-core` from 1.31.0 to 1.33.0
+- Bumps `avro` from 1.11.0 to 1.11.1
+- Bumps `woodstox-core` from 6.3.0 to 6.3.1
 - Bumps `xmlbeans` from 5.1.0 to 5.1.1 ([#4354](https://github.com/opensearch-project/OpenSearch/pull/4354))
 - Bumps `azure-core-http-netty` from 1.12.0 to 1.12.4 ([#4160](https://github.com/opensearch-project/OpenSearch/pull/4160))
 - Bumps `azure-core` from 1.27.0 to 1.31.0 ([#4160](https://github.com/opensearch-project/OpenSearch/pull/4160))
@@ -55,9 +52,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Bumps `jempbox` from 1.8.16 to 1.8.17 ([#4550](https://github.com/opensearch-project/OpenSearch/pull/4550))
 - Bumps `hadoop-hdfs` from 3.3.3 to 3.3.4 ([#4644](https://github.com/opensearch-project/OpenSearch/pull/4644))
 - Bumps `jna` from 5.11.0 to 5.12.1 ([#4656](https://github.com/opensearch-project/OpenSearch/pull/4656))
-
 ### Changed
-
 - Dependency updates (httpcore, mockito, slf4j, httpasyncclient, commons-codec) ([#4308](https://github.com/opensearch-project/OpenSearch/pull/4308))
 - Use RemoteSegmentStoreDirectory instead of RemoteDirectory ([#4240](https://github.com/opensearch-project/OpenSearch/pull/4240))
 - Plugin ZIP publication groupId value is configurable ([#4156](https://github.com/opensearch-project/OpenSearch/pull/4156))
@@ -80,20 +75,18 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Controlling discovery for decommissioned nodes ([#4590](https://github.com/opensearch-project/OpenSearch/pull/4590))
 - Backport Apache Lucene version change for 2.4.0 ([#4677](https://github.com/opensearch-project/OpenSearch/pull/4677))
 - Use ReplicationFailedException instead of OpensearchException in ReplicationTarget ([#4725](https://github.com/opensearch-project/OpenSearch/pull/4725))
+- Fix weighted routing metadata deserialization error on process restart ([#4691](https://github.com/opensearch-project/OpenSearch/pull/4691))
 - Refactor Base Action class javadocs to OpenSearch.API ([#4732](https://github.com/opensearch-project/OpenSearch/pull/4732))
-
+- Migrate client transports to Apache HttpClient / Core 5.x ([#4459](https://github.com/opensearch-project/OpenSearch/pull/4459))
 ### Deprecated
-
 ### Removed
 - Remove deprecated code to add node name into log pattern of log4j property file ([#4568](https://github.com/opensearch-project/OpenSearch/pull/4568))
 - Unused object and import within TransportClusterAllocationExplainAction ([#4639](https://github.com/opensearch-project/OpenSearch/pull/4639))
 - Remove LegacyESVersion.V_7_0_* and V_7_1_* Constants ([#2768](https://https://github.com/opensearch-project/OpenSearch/pull/2768))
 - Remove LegacyESVersion.V_7_2_ and V_7_3_ Constants ([#4702](https://github.com/opensearch-project/OpenSearch/pull/4702))
 - Always auto release the flood stage block ([#4703](https://github.com/opensearch-project/OpenSearch/pull/4703))
 - Remove LegacyESVersion.V_7_4_ and V_7_5_ Constants ([#4704](https://github.com/opensearch-project/OpenSearch/pull/4704))
-
 ### Fixed
-
 - `opensearch-service.bat start` and `opensearch-service.bat manager` failing to run ([#4289](https://github.com/opensearch-project/OpenSearch/pull/4289))
 - PR reference to checkout code for changelog verifier ([#4296](https://github.com/opensearch-project/OpenSearch/pull/4296))
 - `opensearch.bat` and `opensearch-service.bat install` failing to run, missing logs directory ([#4305](https://github.com/opensearch-project/OpenSearch/pull/4305))
@@ -134,35 +127,27 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Fix new race condition in DecommissionControllerTests ([4688](https://github.com/opensearch-project/OpenSearch/pull/4688))
 - Fix SearchStats (de)serialization (caused by https://github.com/opensearch-project/OpenSearch/pull/4616) ([#4697](https://github.com/opensearch-project/OpenSearch/pull/4697))
 - Fixing Gradle warnings associated with publishPluginZipPublicationToXxx tasks ([#4696](https://github.com/opensearch-project/OpenSearch/pull/4696))
-
 ### Security
-
 - CVE-2022-25857 org.yaml:snakeyaml DOS vulnerability ([#4341](https://github.com/opensearch-project/OpenSearch/pull/4341))
 
 ## [2.x]
-
 ### Added
-
 - Github workflow for changelog verification ([#4085](https://github.com/opensearch-project/OpenSearch/pull/4085))
 - Label configuration for dependabot PRs ([#4348](https://github.com/opensearch-project/OpenSearch/pull/4348))
 - Added RestLayer Changes for PIT stats ([#4217](https://github.com/opensearch-project/OpenSearch/pull/4217))
 - Added GeoBounds aggregation on GeoShape field type.([#4266](https://github.com/opensearch-project/OpenSearch/pull/4266))
   - Addition of Doc values on the GeoShape Field
   - Addition of GeoShape ValueSource level code interfaces for accessing the DocValues.
   - Addition of Missing Value feature in the GeoShape Aggregations.
-
+- Install and configure Log4j JUL Adapter for Lucene 9.4 ([#4754](https://github.com/opensearch-project/OpenSearch/pull/4754))
 ### Changed
-
 ### Deprecated
-
 ### Removed
-
 ### Fixed
-
 - PR reference to checkout code for changelog verifier ([#4296](https://github.com/opensearch-project/OpenSearch/pull/4296))
 - Commit workflow for dependabot changelog helper ([#4331](https://github.com/opensearch-project/OpenSearch/pull/4331))
+- Better plural stemmer than minimal_english ([#4738](https://github.com/opensearch-project/OpenSearch/pull/4738))
 
 ### Security
-
 [Unreleased]: https://github.com/opensearch-project/OpenSearch/compare/2.2.0...HEAD
 [2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.2.0...2.x

buildSrc/src/main/resources/forbidden/http-signatures.txt

@@ -15,31 +15,14 @@
 # language governing permissions and limitations under the License.
 
 @defaultMessage Explicitly specify the ContentType of HTTP entities when creating
-org.apache.http.entity.StringEntity#<init>(java.lang.String)
-org.apache.http.entity.StringEntity#<init>(java.lang.String,java.lang.String)
-org.apache.http.entity.StringEntity#<init>(java.lang.String,java.nio.charset.Charset)
-org.apache.http.entity.ByteArrayEntity#<init>(byte[])
-org.apache.http.entity.ByteArrayEntity#<init>(byte[],int,int)
-org.apache.http.entity.FileEntity#<init>(java.io.File)
-org.apache.http.entity.InputStreamEntity#<init>(java.io.InputStream)
-org.apache.http.entity.InputStreamEntity#<init>(java.io.InputStream,long)
-org.apache.http.nio.entity.NByteArrayEntity#<init>(byte[])
-org.apache.http.nio.entity.NByteArrayEntity#<init>(byte[],int,int)
-org.apache.http.nio.entity.NFileEntity#<init>(java.io.File)
-org.apache.http.nio.entity.NStringEntity#<init>(java.lang.String)
-org.apache.http.nio.entity.NStringEntity#<init>(java.lang.String,java.lang.String)
+org.apache.hc.core5.http.io.entity.StringEntity#<init>(java.lang.String)
+org.apache.hc.core5.http.io.entity.StringEntity#<init>(java.lang.String,java.nio.charset.Charset)
 
 @defaultMessage Use non-deprecated constructors
-org.apache.http.nio.entity.NFileEntity#<init>(java.io.File,java.lang.String)
-org.apache.http.nio.entity.NFileEntity#<init>(java.io.File,java.lang.String,boolean)
-org.apache.http.entity.FileEntity#<init>(java.io.File,java.lang.String)
-org.apache.http.entity.StringEntity#<init>(java.lang.String,java.lang.String,java.lang.String)
+org.apache.hc.core5.http.io.entity.FileEntity#<init>(java.io.File,org.apache.hc.core5.http.ContentType)
 
 @defaultMessage BasicEntity is easy to mess up and forget to set content type
-org.apache.http.entity.BasicHttpEntity#<init>()
-
-@defaultMessage EntityTemplate is easy to mess up and forget to set content type
-org.apache.http.entity.EntityTemplate#<init>(org.apache.http.entity.ContentProducer)
+org.apache.hc.core5.http.io.entity.BasicHttpEntity#<init>(java.io.InputStream,org.apache.hc.core5.http.ContentType)
 
 @defaultMessage SerializableEntity uses java serialization and makes it easy to forget to set content type
-org.apache.http.entity.SerializableEntity#<init>(java.io.Serializable)
+org.apache.hc.core5.http.io.entity.SerializableEntity#<init>(java.io.Serializable,org.apache.hc.core5.http.ContentType)

buildSrc/version.properties

@@ -25,6 +25,8 @@ netty             = 4.1.79.Final
 joda              = 2.10.13
 
 # client dependencies
+httpclient5       = 5.1.3
+httpcore5         = 5.1.4
 httpclient        = 4.5.13
 httpcore          = 4.4.15
 httpasyncclient   = 4.1.5

client/benchmark/src/main/java/org/opensearch/client/benchmark/rest/RestClientBenchmark.java

@@ -31,10 +31,10 @@
 
 package org.opensearch.client.benchmark.rest;
 
-import org.apache.http.HttpHeaders;
-import org.apache.http.HttpHost;
-import org.apache.http.HttpStatus;
-import org.apache.http.message.BasicHeader;
+import org.apache.hc.core5.http.HttpHeaders;
+import org.apache.hc.core5.http.HttpHost;
+import org.apache.hc.core5.http.HttpStatus;
+import org.apache.hc.core5.http.message.BasicHeader;
 import org.opensearch.OpenSearchException;
 import org.opensearch.client.Request;
 import org.opensearch.client.Response;

client/rest-high-level/build.gradle

@@ -104,3 +104,9 @@ testClusters.all {
   extraConfigFile nodeTrustStore.name, nodeTrustStore
   extraConfigFile pkiTrustCert.name, pkiTrustCert
 }
+
+thirdPartyAudit.ignoreMissingClasses(
+  'org.conscrypt.Conscrypt',
+  'org.slf4j.Logger',
+  'org.slf4j.LoggerFactory'
+)

client/rest-high-level/src/main/java/org/opensearch/client/ClusterRequestConverters.java

@@ -32,10 +32,10 @@
 
 package org.opensearch.client;
 
-import org.apache.http.client.methods.HttpDelete;
-import org.apache.http.client.methods.HttpGet;
-import org.apache.http.client.methods.HttpHead;
-import org.apache.http.client.methods.HttpPut;
+import org.apache.hc.client5.http.classic.methods.HttpDelete;
+import org.apache.hc.client5.http.classic.methods.HttpGet;
+import org.apache.hc.client5.http.classic.methods.HttpHead;
+import org.apache.hc.client5.http.classic.methods.HttpPut;
 import org.opensearch.action.admin.cluster.health.ClusterHealthRequest;
 import org.opensearch.action.admin.cluster.settings.ClusterGetSettingsRequest;
 import org.opensearch.action.admin.cluster.settings.ClusterUpdateSettingsRequest;

client/rest-high-level/src/main/java/org/opensearch/client/IndicesRequestConverters.java

@@ -32,11 +32,11 @@
 
 package org.opensearch.client;
 
-import org.apache.http.client.methods.HttpDelete;
-import org.apache.http.client.methods.HttpGet;
-import org.apache.http.client.methods.HttpHead;
-import org.apache.http.client.methods.HttpPost;
-import org.apache.http.client.methods.HttpPut;
+import org.apache.hc.client5.http.classic.methods.HttpDelete;
+import org.apache.hc.client5.http.classic.methods.HttpGet;
+import org.apache.hc.client5.http.classic.methods.HttpHead;
+import org.apache.hc.client5.http.classic.methods.HttpPost;
+import org.apache.hc.client5.http.classic.methods.HttpPut;
 import org.opensearch.action.admin.indices.alias.IndicesAliasesRequest;
 import org.opensearch.action.admin.indices.alias.get.GetAliasesRequest;
 import org.opensearch.action.admin.indices.cache.clear.ClearIndicesCacheRequest;

client/rest-high-level/src/main/java/org/opensearch/client/IngestRequestConverters.java

@@ -32,10 +32,10 @@
 
 package org.opensearch.client;
 
-import org.apache.http.client.methods.HttpDelete;
-import org.apache.http.client.methods.HttpGet;
-import org.apache.http.client.methods.HttpPost;
-import org.apache.http.client.methods.HttpPut;
+import org.apache.hc.client5.http.classic.methods.HttpDelete;
+import org.apache.hc.client5.http.classic.methods.HttpGet;
+import org.apache.hc.client5.http.classic.methods.HttpPost;
+import org.apache.hc.client5.http.classic.methods.HttpPut;
 import org.opensearch.action.ingest.DeletePipelineRequest;
 import org.opensearch.action.ingest.GetPipelineRequest;
 import org.opensearch.action.ingest.PutPipelineRequest;

client/rest-high-level/src/main/java/org/opensearch/client/RequestConverters.java

@@ -32,14 +32,14 @@
 
 package org.opensearch.client;
 
-import org.apache.http.HttpEntity;
-import org.apache.http.client.methods.HttpDelete;
-import org.apache.http.client.methods.HttpGet;
-import org.apache.http.client.methods.HttpHead;
-import org.apache.http.client.methods.HttpPost;
-import org.apache.http.client.methods.HttpPut;
-import org.apache.http.entity.ContentType;
-import org.apache.http.nio.entity.NByteArrayEntity;
+import org.apache.hc.client5.http.classic.methods.HttpDelete;
+import org.apache.hc.client5.http.classic.methods.HttpGet;
+import org.apache.hc.client5.http.classic.methods.HttpHead;
+import org.apache.hc.client5.http.classic.methods.HttpPost;
+import org.apache.hc.client5.http.classic.methods.HttpPut;
+import org.apache.hc.core5.http.ContentType;
+import org.apache.hc.core5.http.HttpEntity;
+import org.apache.hc.core5.http.io.entity.ByteArrayEntity;
 import org.apache.lucene.util.BytesRef;
 import org.opensearch.action.DocWriteRequest;
 import org.opensearch.action.admin.cluster.health.ClusterHealthRequest;
@@ -269,7 +269,7 @@ static Request bulk(BulkRequest bulkRequest) throws IOException {
             }
         }
         request.addParameters(parameters.asMap());
-        request.setEntity(new NByteArrayEntity(content.toByteArray(), 0, content.size(), requestContentType));
+        request.setEntity(new ByteArrayEntity(content.toByteArray(), 0, content.size(), requestContentType));
         return request;
     }
 
@@ -358,7 +358,7 @@ static Request index(IndexRequest indexRequest) {
         BytesRef source = indexRequest.source().toBytesRef();
         ContentType contentType = createContentType(indexRequest.getContentType());
         request.addParameters(parameters.asMap());
-        request.setEntity(new NByteArrayEntity(source.bytes, source.offset, source.length, contentType));
+        request.setEntity(new ByteArrayEntity(source.bytes, source.offset, source.length, contentType));
         return request;
     }
 
@@ -514,7 +514,7 @@ static Request multiSearch(MultiSearchRequest multiSearchRequest) throws IOExcep
         XContent xContent = REQUEST_BODY_CONTENT_TYPE.xContent();
         byte[] source = MultiSearchRequest.writeMultiLineFormat(multiSearchRequest, xContent);
         request.addParameters(params.asMap());
-        request.setEntity(new NByteArrayEntity(source, createContentType(xContent.type())));
+        request.setEntity(new ByteArrayEntity(source, createContentType(xContent.type())));
         return request;
     }
 
@@ -549,7 +549,7 @@ static Request multiSearchTemplate(MultiSearchTemplateRequest multiSearchTemplat
 
         XContent xContent = REQUEST_BODY_CONTENT_TYPE.xContent();
         byte[] source = MultiSearchTemplateRequest.writeMultiLineFormat(multiSearchTemplateRequest, xContent);
-        request.setEntity(new NByteArrayEntity(source, createContentType(xContent.type())));
+        request.setEntity(new ByteArrayEntity(source, createContentType(xContent.type())));
         return request;
     }
 
@@ -817,7 +817,7 @@ static HttpEntity createEntity(ToXContent toXContent, XContentType xContentType)
     static HttpEntity createEntity(ToXContent toXContent, XContentType xContentType, ToXContent.Params toXContentParams)
         throws IOException {
         BytesRef source = XContentHelper.toXContent(toXContent, xContentType, toXContentParams, false).toBytesRef();
-        return new NByteArrayEntity(source.bytes, source.offset, source.length, createContentType(xContentType));
+        return new ByteArrayEntity(source.bytes, source.offset, source.length, createContentType(xContentType));
     }
 
     static String endpoint(String index, String id) {

client/rest-high-level/src/main/java/org/opensearch/client/RestHighLevelClient.java

@@ -32,7 +32,7 @@
 
 package org.opensearch.client;
 
-import org.apache.http.HttpEntity;
+import org.apache.hc.core5.http.HttpEntity;
 import org.opensearch.OpenSearchException;
 import org.opensearch.OpenSearchStatusException;
 import org.opensearch.action.ActionListener;
@@ -2220,9 +2220,9 @@ protected final <Resp> Resp parseEntity(final HttpEntity entity, final CheckedFu
         if (entity.getContentType() == null) {
             throw new IllegalStateException("OpenSearch didn't return the [Content-Type] header, unable to parse response body");
         }
-        XContentType xContentType = XContentType.fromMediaType(entity.getContentType().getValue());
+        XContentType xContentType = XContentType.fromMediaType(entity.getContentType());
         if (xContentType == null) {
-            throw new IllegalStateException("Unsupported Content-Type: " + entity.getContentType().getValue());
+            throw new IllegalStateException("Unsupported Content-Type: " + entity.getContentType());
         }
         try (XContentParser parser = xContentType.xContent().createParser(registry, DEPRECATION_HANDLER, entity.getContent())) {
             return entityParser.apply(parser);

client/rest-high-level/src/main/java/org/opensearch/client/SnapshotRequestConverters.java

@@ -32,10 +32,10 @@
 
 package org.opensearch.client;
 
-import org.apache.http.client.methods.HttpDelete;
-import org.apache.http.client.methods.HttpGet;
-import org.apache.http.client.methods.HttpPost;
-import org.apache.http.client.methods.HttpPut;
+import org.apache.hc.client5.http.classic.methods.HttpDelete;
+import org.apache.hc.client5.http.classic.methods.HttpGet;
+import org.apache.hc.client5.http.classic.methods.HttpPost;
+import org.apache.hc.client5.http.classic.methods.HttpPut;
 import org.opensearch.action.admin.cluster.repositories.cleanup.CleanupRepositoryRequest;
 import org.opensearch.action.admin.cluster.repositories.delete.DeleteRepositoryRequest;
 import org.opensearch.action.admin.cluster.repositories.get.GetRepositoriesRequest;

client/rest-high-level/src/main/java/org/opensearch/client/TasksRequestConverters.java

@@ -32,8 +32,8 @@
 
 package org.opensearch.client;
 
-import org.apache.http.client.methods.HttpGet;
-import org.apache.http.client.methods.HttpPost;
+import org.apache.hc.client5.http.classic.methods.HttpGet;
+import org.apache.hc.client5.http.classic.methods.HttpPost;
 import org.opensearch.action.admin.cluster.node.tasks.list.ListTasksRequest;
 import org.opensearch.client.RequestConverters.EndpointBuilder;
 import org.opensearch.client.tasks.CancelTasksRequest;

client/rest-high-level/src/main/resources/forbidden/rest-high-level-signatures.txt

@@ -15,10 +15,9 @@
 # language governing permissions and limitations under the License.
 
 @defaultMessage Use Request#createContentType(XContentType) to be sure to pass the right MIME type
-org.apache.http.entity.ContentType#create(java.lang.String)
-org.apache.http.entity.ContentType#create(java.lang.String,java.lang.String)
-org.apache.http.entity.ContentType#create(java.lang.String,java.nio.charset.Charset)
-org.apache.http.entity.ContentType#create(java.lang.String,org.apache.http.NameValuePair[])
+org.apache.hc.core5.http.ContentType#create(java.lang.String)
+org.apache.hc.core5.http.ContentType#create(java.lang.String,java.lang.String)
+org.apache.hc.core5.http.ContentType#create(java.lang.String,java.nio.charset.Charset)
 
 @defaultMessage ES's logging infrastructure uses log4j2 which we don't want to force on high level rest client users
 org.opensearch.common.logging.DeprecationLogger
@@ -30,7 +29,3 @@ org.opensearch.common.logging.PrefixLogger
 
 @defaultMessage We can't rely on log4j2 being on the classpath so don't log deprecations!
 org.opensearch.common.xcontent.LoggingDeprecationHandler
-
-@defaultMessage Use Nonblocking org.apache.http.nio.entity.NByteArrayEntity
-org.apache.http.entity.ByteArrayEntity
-org.apache.http.entity.StringEntity