[BUG] Cluster goes down after upgrade to 2.1.0 (and then to 2.2.0)

[pietrogu]

Describe the bug
After upgrading my three nodes Docker cluster to version 2.1.0, the nodes started to go down with the following error:

[2022-08-12T10:04:22,455][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [elastic-01] fatal error in thread [opensearch[elastic-01][generic][T#78]], exiting
	at org.opensearch.enrich.EnrichPolicyMaintenanceService.cleanUpEnrichIndices(EnrichPolicyMaintenanceService.java:135) ~[?:?]
	at org.opensearch.enrich.EnrichPolicyMaintenanceService.maybeCleanUpEnrichIndices(EnrichPolicyMaintenanceService.java:123) ~[?:?]
	at org.opensearch.enrich.EnrichPolicyMaintenanceService.execute(EnrichPolicyMaintenanceService.java:114) ~[?:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.2.0.jar:2.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
fatal error in thread [opensearch[elastic-01][generic][T#78]], exiting
	at java.lang.Thread.run(Thread.java:833) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
java.lang.NoSuchMethodError: 'org.opensearch.action.support.master.info.ClusterInfoRequest org.opensearch.action.admin.indices.get.GetIndexRequest.indices(java.lang.String[])'

I then updated to 2.2.0 and the same error happened

To Reproduce
Steps to reproduce the behavior:

1. Deploy Opensearch cluster with three nodes and with at least version 2.1.0 (or 2.2.0)
2. Wait some time (from minutes up to one hour)
3. See error

Expected behavior
The cluster should not go down

Host/Environment (please complete the following information):
Docker Cluster with version 2.2.0
(First time error raised was on version 2.1.0)

[andrross]

Hey @pietrogu, what is org.opensearch.enrich.EnrichPolicyMaintenanceService? There has been some refactoring of internal classes in OpenSearch core that were never intended to be used/extended by plugins, but unfortunately there was no strict enforcement. This looks likes a case where org.opensearch.enrich.EnrichPolicyMaintenanceService is using something that has been renamed to use the clustermanager namespace.

[pietrogu]

I use some custom created plugins that leverage on the enrich plugin (developed in order to respect the license)
Is there a way to solve the problem?

[dblock]

@pietrogu What version are you upgrading from? Where's the code for org.opensearch.enrich.EnrichPolicyMaintenanceService?

We could restore org.opensearch.action.support.master.info.ClusterInfoRequest org.opensearch.action.admin.indices.get.GetIndexRequest.indices(java.lang.String[]) and mark it deprecated, or you could fix org.opensearch.enrich.* to work with OpenSearch 2.x. There will likely be more problems with the latter, so maybe that's a better route?

[pietrogu]

I upgraded from 2.0.1 to 2.1.0
At this point I received the error
Then when available I updated to 2.2.0

Honestly, I don't know this details about the EnrichPolicyMaintenanceService location :(

[dblock]

@tlfeng should we restore that interface in 2.x?

[tlfeng]

@dblock There was a deprecation made in issue #3542, the idea is to keep all backwards compatibility of the existing Java APIs, including org.opensearch.action.support.master.info.ClusterInfoRequest. If this problem shows that GetIndexRequest.indices(java.lang.String[]) is not accessible from the class, then I've made a mistake to break the compatibility, definitely I need to restore the API.

[andrross]

Honestly, I don't know this details about the EnrichPolicyMaintenanceService location :(

@pietrogu We can restore compatibility in 2.x, but this will break in 3.0 when we will fully remove the non-inclusive language from the code base, so you will have to update EnrichPolicyMaintenanceService eventually (though the "fix" should be a super simple find-and-replace).

[dblock]

@dblock There was a deprecation made in issue #3542, the idea is to keep all backwards compatibility of the existing Java APIs, including org.opensearch.action.support.master.info.ClusterInfoRequest. If this problem shows that GetIndexRequest.indices(java.lang.String[]) is not accessible from the class, then I've made a mistake to break the compatibility, definitely I need to restore the API.

@tlfeng Do you want to take it? thank you

[tlfeng]

I realized the backwards compatibility of class GetIndexRequest (

OpenSearch/server/src/main/java/org/opensearch/action/admin/indices/get/GetIndexRequest.java

Line 48 in 551f7c3

public class GetIndexRequest extends ClusterInfoRequest<GetIndexRequest> {

) is broken by mistake in version 2.1 . The return type of the method indices(java.lang.String[]) is not org.opensearch.action.support.master.info.ClusterInfoRequest any more.
The root cause for the problem is in the commit c0bf235, I changed the class GetIndexRequest to extend org.opensearch.action.support.clustermanager.info.ClusterInfoRequest from org.opensearch.action.support.master.info.ClusterInfoRequest.
On the other side, the class ClusterInfoRequest in package org.opensearch.action.support.master.info was changed to extend the same-name class in package org.opensearch.action.support.clustermanager.info, see the code

OpenSearch/server/src/main/java/org/opensearch/action/support/master/info/ClusterInfoRequest.java

Line 44 in 9d312d3

public abstract class ClusterInfoRequest<Request extends ClusterInfoRequest<Request>> extends

.
I plan to restore the inheritance relationship of the class GetIndexRequest, to extend org.opensearch.action.support.master.info.ClusterInfoRequest, and double check all the other similar situation of subclasses of the newly renamed classes.

[pietrogu]

Hi all,

thank you for the quick reply :)

So, If I understood well, after @tlfeng fixes the problem, the issue should be resolved for now, but some changes need to be done on my side in my custom plugin before the 3.0 release?

[dblock]

So, If I understood well, after @tlfeng fixes the problem, the issue should be resolved for now, but some changes need to be done on my side in my custom plugin before the 3.0 release?

Yes, these methods are deprecated in 2.x (2.x branch here). and will be removed in 3.0 (main branch here). Queue up work on the custom plugin to make it compatible with 3.0 by using the replacement methods. You can do this against 2.x that will have the fix and will be GA much earlier, and ensure that no deprecated methods are being called.

[pietrogu]

Ok. I'll wait for the fix by @tlfeng to close the issue.

Thank you

[tlfeng]

Hi @pietrogu I have made a fix for the backwards compatibility issue you pointed out. The fix have been merged into main branch (commit 65f966e) and 2.x branch (commit 7a0ee08), which means it will be available in version 2.3 and above.
According to the release schedule (https://www.opensearch.org/releases.html), version 2.3 will be released on Sept.14th, 2022. Please have a try with OpenSearch version 2.3, and re-open the issue if you still face the same problem. Thank you!