Forward compatibility of Opensearch 1.3 with Opensearch 2.5 cluster

[erezgong]

Is your feature request related to a problem? Please describe.
To have a smooth migration between Opensearch 1.3 to Opensearch 2.5, we must change the clusters first and then change to client.

Describe the solution you'd like
I would like to do the minimal changes to some request and responses, to allow empty type.

Describe alternatives you've considered
The alternative of taking a production system down for upgrading both the client and the clusters together is risky and hurtful for obvious reasons.

Additional context
Functions needed to be slightly changed:

1. org.opensearch.action.search.MultiSearchRequest#writeSearchRequestParams
2. org.opensearch.action.DocWriteResponse#DocWriteResponse(org.opensearch.index.shard.ShardId, java.lang.String, java.lang.String, long, long, long, org.opensearch.action.DocWriteResponse.Result)

[andrross]

@erezgong Can you be more specific about what exactly needs to change? I would expect you to be able to omit the type parameters in the requests using the 1.3 client and have it work against a 2.5 cluster. Is that not the case?

[erezgong]

Hey @andrross, thanks for the fast feedback.
Yes, you cannot work with 1.3 client against 2.5 clusters, which is what we want.
I found two places needed to be changed that I believe will support us to do that:

1. org.opensearch.action.search.MultiSearchRequest#writeSearchRequestParams:

Change this if to accept Strings.EMPTY_ARRAY :

        if (request.types() != null) {
            xContentBuilder.field("types", request.types());
        }

to:

        if (request.types() != null && request.types().length > 0) {
            xContentBuilder.field("types", request.types());
        }

For this I found a workaround without changing this code.

2. org.opensearch.action.DocWriteResponse#DocWriteResponse(org.opensearch.index.shard.ShardId, java.lang.String, java.lang.String, long, long, long, org.opensearch.action.DocWriteResponse.Result):
   The 2.5 cluster doesn't return type member, so this fails in any write operation when getting the response. So I suggest to change it to accept null, and use _doc as default which was the type before:
   this.type = Objects.requireNonNull(type);
   to:
   this.type = Optional.ofNullable(type).orElse("_doc");
   I couldn't find a workaround here. So that blocks us for upgrading to OS 2.5.

WDYT?

[dblock]

I recommend moving to https://github.com/opensearch-project/opensearch-java in which we are trying to deprecate this dependency altogether (opensearch-project/opensearch-java#326). You should be able to upgrade the client that supports multiple versions of OpenSearch.

[andrross]

@erezgong Big +1 to @dblock's suggestion to move to opensearch-java which will give you more independence between client and server and avoid these issues. Is that an option for you?

[erezgong]

I know it's the recommendation, but our usage of rest-high-level-client is huge, and cannot be replaced easily.
I've already checked my suggestion is enough for working with 1.3 client and 2.5 cluster, in our usage at least (all read and writes endpoints).

[dblock]

@erezgong Stepping back, are you proposing to add support for 2.x servers to 1.x high level REST clients or vice-versa? While it works for MultiSearchRequest I think we'd have to do this much more broadly, and add some serious testing before we can include this reliably 😓

[andrross]

@dblock I think @erezgong is asking to make changes to the 1.3 client so that it can work with 2.x, at least in the cases he has identified where it is too strict about expecting _type to be present to work with 2.x.

@erezgong We're really only doing security patches and critical bug fixes to the 1.3 release line now. You're of course welcome to apply the patches you've identified yourself and make your own distribution build. Is that an option for you?

[erezgong]

Correct.
We have created our own jars with those 2 changes for the transition period.

[dblock]

I think we all agree. Let's not add more features to 1.3.x, that would require bumping versions to 1.4 and we don't want to make any more 1.x's.

Should we close this?

[anasalkouz]

@erezgong closing this for now. please feel free to ask to re-open if you still have some topics to cover.