[FEATURE] Deprecate RestClientTransport

[reta]

Is your feature request related to a problem?

There are plans to deprecate RestClient / RHCL, as such the RestClientTransport should be deprecated as well.

What solution would you like?

Deprecate RestClientTransport, recommend using ApacheHttpClient5Transport (#262).

• Update documentation and user guide
• Mark rest-client dependency optional for 3.0.0

What alternatives have you considered?

Keep using deprecated client

Do you have any additional context?

N/A

[reta]

@dblock @andrross fyi guys

[dlvenable]

I'm not sure if a 2.3.0 is planned, but it might be nice to have the deprecations in the next 2.x release and remove in 3.x. But, if no such release is pending, it doesn't seem worth the effort.

[reta]

@dlvenable we discussed that here [1], we can deprecate now - not an issue, but we cannot remove it in 3.x. I mean technically we can but it would be not a gentle move, we should have given a time for people to migrate (we should have started 2.0 with deprecation right away). The optional dependency is the step towards that in 3.0, removal in the next 4.0.

[1] opensearch-project/OpenSearch#5424

[dlvenable]

I believe opensearch-project/OpenSearch#5424 is for OpenSearch 3.0, 4.0, etc.

For opensearch-java, do we want to remove in the upcoming 3.0 or in a future 4.0? I'm not against keeping it longer, I'm more looking for clarity since the version numbers can be a little confusing.

[reta]

For opensearch-java, do we want to remove in the upcoming 3.0 or in a future 4.0?

Removal in 4.0 I believe

[andrross]

opensearch-java is versioned from the server independently, right? I think we should definitely deprecate RestClientTransport in the next client release since an alternative is available. We also could remove it in 3.0 and release 3.0 now, but we'd need to continue supporting the 2.x client line until we've given folks enough time to move off.

[reta]

opensearch-java is versioned from the server independently, right?

Yes, but that's basically a trap: we could release whatever we want but how would people match OS core and OS client version? We still have to introduce new APIs to the client (fe PIT search, stats, etc...), aligning client and server version helps here a lot. Also, we do depend on the test framework from core - this is quite a big chunk we have to cut off (looks doable in general I think but I failed to do it quickly).

[dblock]

I feel like the sooner we break this implicit relationship of client 2.0 = server 2.0 the happier we'll be. IMO client needs to be compatible with multiple major versions of OpenSearch, and anything less than that is bad (TM).

[wbeckler]

Adding to @dblock's point, when the next version of the client is no longer compatible with application code that integrates the client, then it's the worst time to also jump server compatibility. You don't want to force people to do a release of their application at the same moment that they upgrade their server.

[andrross]

In my suggestion the server compatibility matrix stays the same... client 2.x and 3.0 both continue to support server 1.3-2.x. Client 3.0 will require application changes for most users who are upgrading (because they are likely using the now removed RestClientTransport) but it would fully break the dependency on the server (assuming we solve the test framework issue). The main burden would be on us to maintain both versions of the client for some period of time to give users plenty of time to upgrade.

Going forward there will be challenge in communicating server compatibility, and I don't have a great answer other than to try to keep the client compatible with as many server versions as possible, which should be easier when there is no code coupling complicating the issue.

[reta]

@dblock @andrross I am not questioning the compatibility target, I am questioning the need to independently version core and client. I do see the benefits but also the drawbacks: imagine the case OpenSearch 2.5.0, opensearch-java - 3.7.0. What does it mean?

In perspective of features. Take PIT as the best example here, introduced in 2.4. We should provide strongly typed client side API for it , would opensearch-java 3.7.0 support it? We would need to support pretty detailed matrix of what is the baseline for each OpenSearch version so for users to understand when to pick the right one.

Just my concerns.

[dblock]

@dblock @andrross I am not questioning the compatibility target, I am questioning the need to independently version core and client. I do see the benefits but also the drawbacks: imagine the case OpenSearch 2.5.0, opensearch-java - 3.7.0. What does it mean?

It means that I can upgrade any version of opensearch-java 3.x to 3.7.0 without breaking my application.

In perspective of features. Take PIT as the best example here, introduced in 2.4. We should provide strongly typed client side API for it , would opensearch-java 3.7.0 support it? We would need to support pretty detailed matrix of what is the baseline for each OpenSearch version so for users to understand when to pick the right one.

Great example. PIT is a new feature in opensearch-java 3.7.0 and requires server 2.4. When I try to use that API I get an error if the server is < 2.4, which is expected.

Users adopting a client pick the latest version compatible with their version of server and add code that they need (e.g. PIT). Then versioning becomes critical during upgrades, and semver ensures that you can reason about it. Tying server version into the client versioning is what makes it very complicated and scary.

[reta]

Users adopting a client pick the latest version compatible with their version of server and add code that they need (e.g. PIT)

That's they key question: how users would know what is the latest version compatible with their version of server? When versions are aligned, this is no brainer, 2.4 client must support 2.4+ server. Otherwise, we would have to support the matrix like that (hypothetical example):

opensearch-java	OpenSearch
5.1.x	3.0.x
3.7.x	2.4.x
3.2.x	2.2.x
2.2.x	2.1.x
...	...

Are we fine with supporting that? (I feel it would introduce a lot of confusion).

[dblock]

Yes, that's why we need to support multiple majors. The client increments major only when it drops support with a server major (or has other breaking changes). Because server now follows semver, it's possible! I think the matrix needs to look something like this:

opensearch-java	OpenSearch
4.x Java 11-17 breaking change	2.x - 3.x
3.x Java 8-17	2.x - 3.x
2.x	1.x - 2.x
1.x	1.x

In the middle of this example, the latest version of opensearch-java 2.x has all the APIs for the latest versions of OpenSearch 1.x and 2.x.

[dlvenable]

I see your point @reta about having a compatibility matrix being somewhat unpleasant. But, I think this is better than the alternative of having to update clients and server in lock-step. This is a big ask for users who want to migrate large applications.

Also, the client needs to support multiple versions so that tools like Data Prepper and spring-data-opensearch can continue to support multiple OpenSearch versions. These can support multiple versions of OpenSearch so that different users with different OpenSearch servers can continue to use the newer versions of the tools.

@reta , Is there an alternative solution which can allow users to update client and server independently without having a matrix?

@dblock , Why drop support for OpenSearch 1.x in the upcoming 3.x release of opensearch-java?

[reta]

Because server now follows semver, it's possible! I think the matrix needs to look something like this:

@dblock Yes, I think the matrix in your case is more clean (my version implied that each client supports OpenSearch up to baseline version, no dropped versions). But in this case it is even more maintenance because it is totally possible that we would need to support multiple release lines at the same time (referring to your example - 3.x and 4.x for sure).

But, I think this is better than the alternative of having to update clients and server in lock-step.

@dlvenable We do not require lock-step updates for client and server (this is covered by opensearch-project/opensearch-clients#17).

Is there an alternative solution which can allow users to update client and server independently without having a matrix?

We could update servers and clients independently even now, this not an issue. The issue is that we want the clients to be semantically detached from servers (having own version). Hence we need the matrix to at least make some sense over the clients and servers.

We do have live examples of both approaches (the clients over evolving APIs)

• Elasticsearch (surely) - matches client and server releases
• Cassandra - matches client and server releases
• HBase - matches client and server releases

One the other side (the clients over stable APIs - JDBC):

• Postgres - independent server and client releases
• MySQL - independent server and client releases
• Clickhouse - independent server and client releases

I am trying to find a balance here: evolving servers and clients but at the same time, not increasing the maintenance burden and not to introduce confusion.

[dblock]

@dblock , Why drop support for OpenSearch 1.x in the upcoming 3.x release of opensearch-java?

I only provided a matrix for illustration purposes. Ideally we don't want to.

[mnkugler]

Does this need to be backported to 2.x?

[VachaShah]

Does this need to be backported to 2.x?

No we are making RestClientTransport dependency optional in the next major version. In the 2.x, the RestClientTransport is not deprecated.