[PROPOSAL] Restore support for Java 8

[dlvenable]

What kind of business use case are you trying to solve? What are your requirements?

Many users still need Java 8 for their clients for maximum Java compatibility.

What is the problem? What is preventing you from meeting the requirements?

With the introduction of OpenSearch 2.0.0, the OpenSearch 2.0.0 build has updated to Java 11. This also includes the opensearch-rest-client and opensearch-rest-high-level-client clients.

Users of clients who wish to update their client version must now update to Java 11.

Additionally, the opensearch-java client relies on the opensearch-rest-client. Per opensearch-project/opensearch-clients#17, this is the client which is meant to have the most compatibility.

Here is an example of updating Data Prepper (which currently supports Java 8):

> No matching variant of org.opensearch.client:opensearch-rest-high-level-client:2.0.0-rc1 was found. The consumer was configured to find a runtime of a library compatible with Java 8, packaged as a jar, and its dependencies declared externally but:
    - Variant 'apiElements' capability org.opensearch.client:opensearch-rest-high-level-client:2.0.0-rc1 declares a library, packaged as a jar, and its dependencies declared externally:
        - Incompatible because this component declares an API of a component compatible with Java 11 and the consumer needed a runtime of a component compatible with Java 8
    - Variant 'runtimeElements' capability org.opensearch.client:opensearch-rest-high-level-client:2.0.0-rc1 declares a runtime of a library, packaged as a jar, and its dependencies declared externally:
        - Incompatible because this component declares a component compatible with Java 11 and the consumer needed a component compatible with Java 8

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?

I propose that the OpenSearch Core's opensearch-rest-client and opensearch-rest-high-level-client projects be made to use Java 8. The Gradle build can setup the sourceCompatibility and targetCompatibility to ensure that it uses only Java 8 language features.

An alternative solution would be to remove the dependency that opensearch-java has on opensearch-rest-client.

What are your assumptions or prerequisites?

None.

What are remaining open questions?

Can Gradle toolchains or another mechanism help ensure that the client projects do not use any Java 11 APIs?

[dlvenable]

Almost every action from the opensearch-rest-high-level-client comes from the server project which requires Lucene (and thus Java 11). Supporting Java 8 in this client would be a significant undertaking.

[dblock]

I recommend moving to https://github.com/opensearch-project/opensearch-java and restoring support for Java 8 there. Moved the issue here to say that. This may be challenging because it uses the low-level opensearch-rest-client.

Note that eventually we want to introduce a proper SDK in https://github.com/opensearch-project/opensearch-sdk that can mitigate this and similar differences.

[dlvenable]

@reta , @dblock , I'll try to move the discussion in here rather than opensearch-project/OpenSearch#3181.

It is reasonable to have OpenSearch on Java 11, but a client on Java 8. Thus, I think OpenSearch should support this, and created this issue for it.

Right now:

• A user on Java 8 can use the opensearch-java 1.0.0 and the low-level rest client from OpenSearch 1.x. I believe it works against OpenSearch 2.x, though I haven't tested it.
• A user on Java 8 will not be able to use the classic rest-high-level-client for 2.0. The classic rest-high-level-client for 1.x will not work against a 2.x cluster. So a Java 8 client using this library cannot work with an OpenSearch 2.0 cluster.

In the long-term, I think the project should encourage users to use the opensearch-java client. Per this comment, the opensearch-java client will be the one to attempt to support multiple versions. This client can also remain compatible with Java 8.

The opensearch-java client could support compiling on Java 8 and running tests on Java 11. The new Gradle toolchains should support this. That will remove the barrier of using the test framework which you pointed out @reta .

The opensearch-java client currently relies on the low-level client. There might be a few options here:

• Remove this dependency and replace it with similar internal functionality.
• Extract the low-level client into its own project.
• Update the low-level client in OpenSearch core to support Java 8.

[dlvenable]

@dblock , @reta ,

I have created a draft PR which restores support for Java 8 in #157. I have also pushed some changes to opensearch-project/OpenSearch#3181 which builds the opensearch-rest-client (low-level) using JDK 8.

These two PRs demonstrate a path forward for support JDK 8 for clients (low-level REST client and opensearch-java).

[wbeckler]

Any thoughts on the next steps on this? If someone's client has to be Java 8, what's the recommendation for them? Here's an example of someone who is stuck due to the java 11 requirement for opensearch-java: https://forum.opensearch.org/t/opensearch-1-3-x-gradle-build-problem-w-java-8/12400

[dblock]

@wbeckler I think #326 is next.

[dlvenable]

@wbeckler I think #326 is next.

Yes, this is blocking any 3.0.0 release.

[wbeckler]

I don't understand how that deprecation step blocks movement on creating a working java 8 client. Maybe a special build for java 8 that doesn't impact anything else? I think I'm missing something.

[reta]

@wbeckler we basically have 2 options:

a) migrate rest-client and opensearch-java to JDK-8, those have to go in lockstep
b) cut rest-client from opensearch-java and migrate the latter to JDK-8

The #326 is about b).

[dblock]

And a) was attempted in opensearch-project/OpenSearch#3181 that we decided not to merge.

Someone (wink wink) could create a branch that does b) in a hurry, but seems like we should do it properly.

[VachaShah]

@reta @dblock For option b) listed above, is the next step to bring in the changes from rest-client into opensearch-java so we can decouple them and remove the compileOnly dependency as well?

[reta]

b) listed above, is the next step to bring in the changes from rest-client into opensearch-java so we can decouple them and remove the compileOnly dependency as well?

@VachaShah we did that with Apache HttpClient 5 already: the rest-client is optional in opensearch-java 3.0.0 but still a default but deprecated for opensearch-java 2.x (to not break existing apps). From the non-user perspective, the opensearch-java still depends on OpenSearch test framework

[dblock]

I think we should 1) remove the dependency and downgrade JDK in a major version of this client and release 2) maintain the 2.x branch that has HLRC

[VachaShah]

Thanks @reta and @dblock! @dblock By remove the dependency, do you mean removing both optional rest-client and test framework from the client or just the test framework?

To restore JDK 8 though, I think we need to remove both the dependencies even though rest-client is optional for this client's next major version. @reta LMK if I am missing something here.

[reta]

Thanks @reta and @dblock! @dblock By remove the dependency, do you mean removing both optional rest-client and test framework from the client or just the test framework?

I think test framework may stay since this won't be brought to users (but we would need to build with JDK-11 keeping source/binary JDK-8).

To restore JDK 8 though, I think we need to remove both the dependencies even though rest-client is optional for this client's next major version. @reta LMK if I am missing something here.

To the previous point, getting rid of dependency on test framework would be just awesome from the decoupling perspective, but if user wants to use rest-client transport, they would need to use JDK-11, pretty messy to be fair.

[peerkar]

Is providing a Java 8 client for OpenSearch 2.x in the plans any more?

Meanwhile for those interested of it, I patched it here:

• Binary: https://github.com/peerkar/opensearch-java/releases
• Source: https://github.com/peerkar/opensearch-java/tree/2.8.0.jdk8

Tests discarded / fail, but the Java client compiled and seems to work so far. The REST client project was integrated from the OpenSearch into the opensearch-java.

[dblock]

@peerkar If you can make a PR that passes tests into this repo on Java 8, we'd gladly take it!

[rursprung]

i know that i'm a bit late to the party, but:

i think adding back support for a language version which was released nearly 10 years ago and superseded 5 years ago is the wrong way to go. there are so many newer language features in java 11 & newer which can help in making both the APIs and the code cleaner & nicer to work with that the trade-off is too big. and others are also moving away from supporting java 8 (as shown e.g. with the issues in #767) and this will just get worse over time.

for how long do you plan to support java 8? another 1-2 years (in which case i don't think that it's worth it)? or until 2030 (in which case i don't think that anyone will still want to work with this code base in a few years)?

i think the main reason why people are still stuck on java 8 is pure laziness (not even necessarily of the developers but sometimes also of the management which will say "oh, they still support it, so don't waste time on updates"). we shouldn't support that.
and the cases of a secluded server running somewhere where it's really hard to install a newer JDK does not really apply here, because then they also wouldn't have OpenSearch (at least not OpenSearch 2.x; and with 1.x it still worked with java 8).

[reta]

i know that i'm a bit late to the party, but:

@rursprung not at all, thank you for jumping in

i think adding back support for a language version which was released nearly 10 years ago and superseded 5 years ago is the wrong way to go.

Personally, I would prefer to forget about Java 8 and move on, the sad part is that many projects still claim support for Java 8 even if only some small percent of the users benefit from it. And for these projects the only way to deal with 2.x OpenSearch servers is to use 1.x clients (surely if JDK-11 not the runtime).

for how long do you plan to support java 8? another 1-2 years (in which case i don't think that it's worth it)? or until 2030 (in which case i don't think that anyone will still want to work with this code base in a few years)?

1-2 years sound reasonable, the question is obviously - is it worth it? We are listening to the community (even if it takes long time to be heard), but community asks for this option. That is the motivation.

[rursprung]

i think it's a shame that we have to be stuck on such an old language version (and with no CI support as well; i guarantee you that this will break as nobody will test against JDK8 anymore - why would you? you also don't test against Windows XP anymore 😉).

since you now merged it: can you please clearly specify the targeted support timeline for JDK8 to know by when you'll drop JDK8 support? will this happen with the next major release? in one year? in two? by 2030?

[dblock]

I am with you @rursprung. I added #778 to get some integration test working. If you have an idea of the best way to do it, please contribute!

I don't have any strong opinion of when to drop JDK8 support. There should be a reason other than "because it's old".

I am always amazed how many customers are still using JDK8, it's not easy to upgrade some enterprise applications or they don't want to because it's too expensive.

Speaking of Windows 98, I maintain https://github.com/dotnetinstaller/dotnetinstaller which only dropped support for Windows 98 in 2016. I tried to do it around 2012, but found a non-profit in Africa that was relying on the tool supporting Windows 98 and decided it was worth the effort to continue supporting that.

[reta]

I don't have any strong opinion of when to drop JDK8 support. There should be a reason other than "because it's old".

@rursprung so we have it restored in 3.x and plan to have it restored in 2.x (once 2.12.0 is out), so I believe targeting 4.0 as the target for dropping JDK-8 would be it.

[reta]

btw, interesting stats for 2023:

50% of the projects are still on JDK-8, want we or not, this is the reality.

[reta]

Backport to 2.x is pending #798