[BUG] org.opensearch.client.util.MissingRequiredPropertyException: Missing required property 'OpenSearchVersionInfo.distribution'

[jhg023]

What is the bug?

Earlier today, I upgraded my AWS OpenSearch cluster from version 2.5 to 2.7, and now I'm seeing the following exception when fetching the cluster information on startup:

Caused by: org.opensearch.client.util.MissingRequiredPropertyException: Missing required property 'OpenSearchVersionInfo.distribution'
    at org.opensearch.client.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:88) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.opensearch._types.OpenSearchVersionInfo.<init>(OpenSearchVersionInfo.java:82) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.opensearch._types.OpenSearchVersionInfo$Builder.build(OpenSearchVersionInfo.java:324) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.opensearch._types.OpenSearchVersionInfo$Builder.build(OpenSearchVersionInfo.java:214) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:93) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:56) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:85) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:189) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:150) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:88) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:92) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:56) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.transport.aws.AwsSdk2Transport.parseResponse(AwsSdk2Transport.java:522) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.transport.aws.AwsSdk2Transport.executeSync(AwsSdk2Transport.java:393) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.transport.aws.AwsSdk2Transport.performRequest(AwsSdk2Transport.java:191) ~[opensearch-java-2.6.0.jar!/:na]
    at org.opensearch.client.opensearch.OpenSearchClient.info(OpenSearchClient.java:860) ~[opensearch-java-2.6.0.jar!/:na]

How can one reproduce the bug?

1. Create an AWS OpenSearch domain running version 2.7 (the latest version AWS supports currently).
2. Create your OpenSearchClient:

var client = new OpenSearchClient(
    new AwsSdk2Transport(
        ApacheHttpClient.builder().build(),
        opensearchHost,
        "es",
        Region.US_WEST_2,
        AwsSdk2TransportOptions.builder().build()
    )
);

3. Attempt to fetch information on the cluster:

var clusterName = client.info().clusterName();

What is the expected behavior?

I wouldn't expect this to result in an exception, but rather print the cluster information even if the distribution is missing.

What is your host/environment?

AWS OpenSearch 2.7 (the latest version AWS supports currently)

Do you have any screenshots?

N/A

Do you have any additional context?

If the fix for this issue is as simple as refactoring the following line, I'd be happy to PR it:

this.distribution = ApiTypeHelper.requireNonNull(builder.distribution, this, "distribution");

For now, I'm going to avoid fetching the cluster information on startup as a temporary solution.

[wbeckler]

What version of the java client are you using?

[jhg023]

Using version 2.6.0 of opensearch-java

[wbeckler]

Feel free to submit the PR to fix this. I think it makes sense.

[reta]

@jhg023 @wbeckler the distribution is a required property returned by the OpenSearch server, the issue looks to be AWS specific. The server/client API model is correct.

[jhg023]

In that case, I can open a support ticket to AWS to investigate.

[dblock]

I created a 2.7 domain and I cannot reproduce this, I see distribution below. What do you see @jhg023?

$ awscurl --region=us-west-2 --service es https://search-dblock-test-opensearch-27-....us-west-2.es.amazonaws.com 
{
  "name" : "fcecc570a4b1e0527475991424f81e8d",
  "cluster_name" : "013306242651:dblock-test-opensearch-27",
  "cluster_uuid" : "CDx1PKXnQnuym3oTrEPglA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.7.0",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2023-06-28T07:07:20.444435752Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Update: do you use OpenSearch in compatibility mode with compatibility.override_main_response_version? That would do it I believe, which exists to support ES 7.10 clients exclusively. (thanks @andrross for the tip).

[jhg023]

Unless AWS OpenSearch comes with that compatibility enabled by default, I never explicitly enabled it. Here's what I see using the same awscurl command:

{
  "name" : "50101469e0d4fa***caca4915",
  "cluster_name" : "924108133664:***-prod",
  "cluster_uuid" : "***-OZnA",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2023-07-04T04:44:03.664641392Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Seeing as how the number field in my response has a value of 7.10.2, I suspect there's some ES 7.10 compatibility happening here.

[dblock]

@jhg023 So yes, this is compatibility mode. When you create a new domain there's a checkbox to enable/disable it. If you don't need this, you can toggle it via API

$ awscurl --region=us-west-2 --service es https://search-dblock-test-opensearch-27-...us-west-2.es.amazonaws.com 
{
  "name" : "fcecc570a4b1e0527475991424f81e8d",
  "cluster_name" : "013306242651:dblock-test-opensearch-27",
  "cluster_uuid" : "CDx1PKXnQnuym3oTrEPglA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.7.0",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2023-06-28T07:07:20.444435752Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

$ awscurl -X PUT  --region=us-west-2 --service es https://search-dblock-test-opensearch-27-...us-west-2.es.amazonaws.com/_cluster/settings -d'{"persistent":{"compatibility.override_main_response_version" : true}}' 
{"acknowledged":true,"persistent":{"compatibility":{"override_main_response_version":"true"}},"transient":{}}
osx ~/source/opensearch-project/project-tools/dblock-project-tools (main)$ awscurl --region=us-west-2 --service es https://search-dblock-test-opensearch-27-...us-west-2.es.amazonaws.com 
{
  "name" : "fcecc570a4b1e0527475991424f81e8d",
  "cluster_name" : "013306242651:dblock-test-opensearch-27",
  "cluster_uuid" : "CDx1PKXnQnuym3oTrEPglA",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2023-06-28T07:07:20.444435752Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

I'm going to close this, but feel free to keep the conversation open as needed.

Docs for compatibility mode: https://opensearch.org/docs/latest/tools/index/#agents-and-ingestion-tools

[andrross]

Unless AWS OpenSearch comes with that compatibility enabled by default, I never explicitly enabled it.

@jhg023 It's not the default, but there is a check box to enable compatibility mode on an Amazon OpenSearch Service domain when creating it via the console UI.

[jhg023]

Thanks for letting me know! I created this domain through CloudFormation, so I suspect I may have unintentionally enabled it there, or didn't explicitly opt to disable it.

Edit: It looks like the override_main_response_version option is automatically set to true when an AWS OpenSearch domain is updated. I believe this can only be set to false within a CloudFormation template alongside a simultaneous version update. I tried setting it to false without a version update, but AWS rejected it claiming that the option doesn't exist, so I think it's context-specific.

[wbeckler]

I don't get it. I thought the issue was that the server response was causing an exception. I don't think compatibility mode should cause an error.

[reta]

I don't get it. I thought the issue was that the server response was causing an exception. I don't think compatibility mode should cause an error.

The compatibility mode (which if I am not mistaken phased out in 3.0.0) is only need when it is desired that OpenSearch cluster could be accessed by Elasticsearch clients (not OpenSearch clients).

[andrross]

@reta The compatibility.override_main_response_version setting was actually removed in opensearch-project/OpenSearch#3031 for the 2.0 release. However, Amazon OpenSearch Service has forward-ported this setting to the 2.x versions.

@wbeckler The question is whether opensearch-java should be compatible with OpenSearch running in a compatibility mode intended to make it compatible with Elasticsearch clients.

[dblock]

@wbeckler The question is whether opensearch-java should be compatible with OpenSearch running in a compatibility mode intended to make it compatible with Elasticsearch clients.

I believe the answer is no, there's no claims of opensearch-java working with Elasticsearch 7.10.

[wbeckler]

Wouldn't it be better if the client could be cross compatible? Maybe with a client compatibility mode flag as well?

…

On Tue, Aug 8, 2023, 11:06 AM Daniel (dB.) Doubrovkine < ***@***.***> wrote: @wbeckler <https://github.com/wbeckler> The question is whether opensearch-java should be compatible with OpenSearch running in a compatibility mode intended to make it compatible with Elasticsearch clients. I believe the answer is no, there's no claims of opensearch-java working with Elasticsearch 7.10. — Reply to this email directly, view it on GitHub <#573 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5PRLWHJSWMQXPIL2YOROTXUJIZHANCNFSM6AAAAAA2DJ723M> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[reta]

Wouldn't it be better if the client could be cross compatible?

We definitely shouldn't to take this route, the compatibility was interim measure, at this point the functionality between ES and OS diverged significantly, each has own ecosystem.

[amareshb]

Faced the same issue today and had to set compatibility.override_main_response_version to false.

PUT _cluster/settings
{
  "persistent": {
    "compatibility": {
      "override_main_response_version": false
    }
  }
}

This will add the property "distribution" : "opensearch" in the cluster info response.

If you are running a managed version of opensearch like on AWS, you need to contact AWS support to update this setting.