Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Remote Model returns connectorExecutor null bug #2754

Closed
yuye-aws opened this issue Jul 26, 2024 · 3 comments
Closed

[BUG] Remote Model returns connectorExecutor null bug #2754

yuye-aws opened this issue Jul 26, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@yuye-aws
Copy link
Member

yuye-aws commented Jul 26, 2024 •
edited
Loading

What is the bug?
Remote model returns connectorExecutor null bug

How can one reproduce the bug?

  1. Register a remote LLM via connector.

  2. Use model predict API to call the remote LLM

POST /_plugins/_ml/models/5Gww7ZABkZpCpT6HrObu/_predict/
{
  "parameters": {
    "prompt": """Human: Hello \n Assistant:"""
  }
}
  1. Receives the following error
{
  "error": {
    "root_cause": [
      {
        "type": "null_pointer_exception",
        "reason": "Cannot invoke \"org.opensearch.ml.engine.algorithms.remote.RemoteConnectorExecutor.setScriptService(org.opensearch.script.ScriptService)\" because \"this.connectorExecutor\" is null"
      }
    ],
    "type": "null_pointer_exception",
    "reason": "Cannot invoke \"org.opensearch.ml.engine.algorithms.remote.RemoteConnectorExecutor.setScriptService(org.opensearch.script.ScriptService)\" because \"this.connectorExecutor\" is null"
  },
  "status": 500
}

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

  • OS: [e.g. iOS]
  • Version [e.g. 22]
  • Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.
@yuye-aws yuye-aws added bug Something isn't working untriaged labels Jul 26, 2024
@yuye-aws
Copy link
Member Author

Luckily, this bug does not exist in 2.x branch.

@yuye-aws yuye-aws changed the title [BUG] Remote LLM returns connectorExecutor null bug [BUG] Remote Model returns connectorExecutor null bug Jul 26, 2024
@yuye-aws
Copy link
Member Author

The following are some error logs:

[2024-07-26T12:20:18,520][ERROR][o.o.m.e.MLEngineClassLoader] [integTest-0] Failed to init instance for type aws_sigv4
java.lang.reflect.InvocationTargetException: null
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
        at org.opensearch.ml.engine.MLEngineClassLoader.initInstance(MLEngineClassLoader.java:131) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.engine.MLEngineClassLoader.initInstance(MLEngineClassLoader.java:98) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.engine.algorithms.remote.RemoteModel.initModel(RemoteModel.java:105) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.engine.MLEngine.deploy(MLEngine.java:125) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.model.MLModelManager.setupParamsAndPredictable(MLModelManager.java:1164) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.ml.model.MLModelManager.lambda$deployRemoteOrBuiltInModel$56(MLModelManager.java:1153) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.ml.model.MLModelManager.lambda$getConnector$71(MLModelManager.java:1653) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:298) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:284) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1483) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1566) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1546) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:62) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:45) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.opensearch.ml.common.connector.ConnectorClientConfig
        at org.opensearch.ml.engine.algorithms.remote.AbstractConnectorExecutor.initialize(AbstractConnectorExecutor.java:23) ~[?:?]
        at org.opensearch.ml.engine.algorithms.remote.AwsConnectorExecutor.<init>(AwsConnectorExecutor.java:65) ~[?:?]
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
        ... 28 more
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoSuchFieldError: Class org.opensearch.Version does not have member field 'org.opensearch.Version V_2_15_0' [in thread "opensearch[integTest-0][system_read][T#1]"]
        at org.opensearch.ml.common.connector.ConnectorClientConfig.<clinit>(ConnectorClientConfig.java:45) ~[?:?]
        at org.opensearch.ml.engine.algorithms.remote.AbstractConnectorExecutor.initialize(AbstractConnectorExecutor.java:23) ~[?:?]
        at org.opensearch.ml.engine.algorithms.remote.AwsConnectorExecutor.<init>(AwsConnectorExecutor.java:65) ~[?:?]
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
        ... 28 more
[2024-07-26T12:20:18,522][ERROR][o.o.m.e.a.r.RemoteModel  ] [integTest-0] Failed to init remote model.
java.lang.NullPointerException: Cannot invoke "org.opensearch.ml.engine.algorithms.remote.RemoteConnectorExecutor.setScriptService(org.opensearch.script.ScriptService)" because "this.connectorExecutor" is null
        at org.opensearch.ml.engine.algorithms.remote.RemoteModel.initModel(RemoteModel.java:106) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.engine.MLEngine.deploy(MLEngine.java:125) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.model.MLModelManager.setupParamsAndPredictable(MLModelManager.java:1164) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.ml.model.MLModelManager.lambda$deployRemoteOrBuiltInModel$56(MLModelManager.java:1153) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.ml.model.MLModelManager.lambda$getConnector$71(MLModelManager.java:1653) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:298) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:284) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1483) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1566) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1546) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:62) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:45) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2024-07-26T12:20:18,522][INFO ][o.o.m.m.MLModelCacheHelper] [integTest-0] Remove model RYBE7ZABxFlvMXQgl3bb from auto deploy cache
[2024-07-26T12:20:18,522][ERROR][o.o.m.t.MLPredictTaskRunner] [integTest-0] Failed to auto deploy model RYBE7ZABxFlvMXQgl3bb
java.lang.NullPointerException: Cannot invoke "org.opensearch.ml.engine.algorithms.remote.RemoteConnectorExecutor.setScriptService(org.opensearch.script.ScriptService)" because "this.connectorExecutor" is null
        at org.opensearch.ml.engine.algorithms.remote.RemoteModel.initModel(RemoteModel.java:106) ~[?:?]
        at org.opensearch.ml.engine.MLEngine.deploy(MLEngine.java:125) ~[?:?]
        at org.opensearch.ml.model.MLModelManager.setupParamsAndPredictable(MLModelManager.java:1164) ~[?:?]
        at org.opensearch.ml.model.MLModelManager.lambda$deployRemoteOrBuiltInModel$56(MLModelManager.java:1153) ~[?:?]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.ml.model.MLModelManager.lambda$getConnector$71(MLModelManager.java:1653) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:298) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:284) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1483) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1566) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1546) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:62) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:45) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2024-07-26T12:20:18,523][WARN ][r.suppressed             ] [integTest-0] path: /_plugins/_ml/models/RYBE7ZABxFlvMXQgl3bb/_predict, params: {pretty=true, model_id=RYBE7ZABxFlvMXQgl3bb}
java.lang.NullPointerException: Cannot invoke "org.opensearch.ml.engine.algorithms.remote.RemoteConnectorExecutor.setScriptService(org.opensearch.script.ScriptService)" because "this.connectorExecutor" is null
        at org.opensearch.ml.engine.algorithms.remote.RemoteModel.initModel(RemoteModel.java:106) ~[?:?]
        at org.opensearch.ml.engine.MLEngine.deploy(MLEngine.java:125) ~[?:?]
        at org.opensearch.ml.model.MLModelManager.setupParamsAndPredictable(MLModelManager.java:1164) ~[?:?]
        at org.opensearch.ml.model.MLModelManager.lambda$deployRemoteOrBuiltInModel$56(MLModelManager.java:1153) ~[?:?]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.ml.model.MLModelManager.lambda$getConnector$71(MLModelManager.java:1653) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:298) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:284) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1483) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1566) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1546) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:62) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:45) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2024-07-26T12:20:18,588][WARN ][o.o.m.a.d.TransportDeployModelOnNodeAction] [integTest-0] Duplicate deploy model task
[2024-07-26T12:20:18,588][ERROR][o.o.m.a.f.TransportForwardAction] [integTest-0] deploy model failed on all nodes, model id: RYBE7ZABxFlvMXQgl3bb
[2024-07-26T12:20:18,592][INFO ][o.o.m.a.f.TransportForwardAction] [integTest-0] deploy model done with state: DEPLOY_FAILED, model id: RYBE7ZABxFlvMXQgl3bb
[2024-07-26T12:20:18,592][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [integTest-0] deploy model task done SYBE7ZABxFlvMXQgqXbO
[2024-07-26T12:20:18,592][INFO ][o.o.m.t.MLPredictTaskRunner] [integTest-0] Auto deployment action triggered for the model Model ID: RYBE7ZABxFlvMXQgl3bb
[2024-07-26T12:20:18,627][INFO ][stdout                   ] [integTest-0] path is /_plugins/_ml/models/RYBE7ZABxFlvMXQgl3bb/_predict
[2024-07-26T12:20:18,631][INFO ][stdout                   ] [integTest-0] actionType is PREDICT
[2024-07-26T12:20:18,632][INFO ][o.o.m.m.MLModelCacheHelper] [integTest-0] Add model RYBE7ZABxFlvMXQgl3bb to auto deploy cache
[2024-07-26T12:20:18,635][INFO ][o.o.m.t.MLPredictTaskRunner] [integTest-0] Automatically deploy model Model ID: RYBE7ZABxFlvMXQgl3bb
[2024-07-26T12:20:18,635][INFO ][o.o.m.m.MLModelCacheHelper] [integTest-0] Remove model RYBE7ZABxFlvMXQgl3bb from auto deploy cache
[2024-07-26T12:20:18,635][INFO ][o.o.m.t.MLPredictTaskRunner] [integTest-0] Auto deploy model RYBE7ZABxFlvMXQgl3bb to local node
[2024-07-26T12:20:18,636][INFO ][o.o.m.a.d.TransportDeployModelAction] [integTest-0] Will deploy model on these nodes: V6O3BY5pQXCNmd32jB0gBA
[2024-07-26T12:20:18,641][INFO ][o.o.m.m.MLModelManager   ] [integTest-0] Trying to deploy remote or built-in model without model controller configured.
[2024-07-26T12:20:18,645][INFO ][o.o.m.m.MLModelManager   ] [integTest-0] Set connector QoBE7ZABxFlvMXQge3b- for the model: RYBE7ZABxFlvMXQgl3bb
[2024-07-26T12:20:18,646][INFO ][o.o.m.m.MLModelManager   ] [integTest-0] Setting up basic ML predictor parameters.
[2024-07-26T12:20:18,652][ERROR][o.o.m.e.MLEngineClassLoader] [integTest-0] Failed to init instance for type aws_sigv4
java.lang.reflect.InvocationTargetException: null
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
        at org.opensearch.ml.engine.MLEngineClassLoader.initInstance(MLEngineClassLoader.java:131) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.engine.MLEngineClassLoader.initInstance(MLEngineClassLoader.java:98) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.engine.algorithms.remote.RemoteModel.initModel(RemoteModel.java:105) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.engine.MLEngine.deploy(MLEngine.java:125) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.model.MLModelManager.setupParamsAndPredictable(MLModelManager.java:1164) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.ml.model.MLModelManager.lambda$deployRemoteOrBuiltInModel$56(MLModelManager.java:1153) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.ml.model.MLModelManager.lambda$getConnector$71(MLModelManager.java:1653) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:298) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:284) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1483) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1566) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1546) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:62) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:45) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.opensearch.ml.common.connector.ConnectorClientConfig
        at org.opensearch.ml.engine.algorithms.remote.AbstractConnectorExecutor.initialize(AbstractConnectorExecutor.java:23) ~[?:?]
        at org.opensearch.ml.engine.algorithms.remote.AwsConnectorExecutor.<init>(AwsConnectorExecutor.java:65) ~[?:?]
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
        ... 28 more
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoSuchFieldError: Class org.opensearch.Version does not have member field 'org.opensearch.Version V_2_15_0' [in thread "opensearch[integTest-0][system_read][T#1]"]
        at org.opensearch.ml.common.connector.ConnectorClientConfig.<clinit>(ConnectorClientConfig.java:45) ~[?:?]
        at org.opensearch.ml.engine.algorithms.remote.AbstractConnectorExecutor.initialize(AbstractConnectorExecutor.java:23) ~[?:?]
        at org.opensearch.ml.engine.algorithms.remote.AwsConnectorExecutor.<init>(AwsConnectorExecutor.java:65) ~[?:?]
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
        ... 28 more
[2024-07-26T12:20:18,654][ERROR][o.o.m.e.a.r.RemoteModel  ] [integTest-0] Failed to init remote model.
java.lang.NullPointerException: Cannot invoke "org.opensearch.ml.engine.algorithms.remote.RemoteConnectorExecutor.setScriptService(org.opensearch.script.ScriptService)" because "this.connectorExecutor" is null
        at org.opensearch.ml.engine.algorithms.remote.RemoteModel.initModel(RemoteModel.java:106) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.engine.MLEngine.deploy(MLEngine.java:125) [opensearch-ml-algorithms-3.0.0.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.model.MLModelManager.setupParamsAndPredictable(MLModelManager.java:1164) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.ml.model.MLModelManager.lambda$deployRemoteOrBuiltInModel$56(MLModelManager.java:1153) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.ml.model.MLModelManager.lambda$getConnector$71(MLModelManager.java:1653) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:298) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:284) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1483) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1566) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1546) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:62) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:45) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2024-07-26T12:20:18,655][ERROR][o.o.m.t.MLPredictTaskRunner] [integTest-0] Failed to auto deploy model RYBE7ZABxFlvMXQgl3bb
java.lang.NullPointerException: Cannot invoke "org.opensearch.ml.engine.algorithms.remote.RemoteConnectorExecutor.setScriptService(org.opensearch.script.ScriptService)" because "this.connectorExecutor" is null
        at org.opensearch.ml.engine.algorithms.remote.RemoteModel.initModel(RemoteModel.java:106) ~[?:?]
        at org.opensearch.ml.engine.MLEngine.deploy(MLEngine.java:125) ~[?:?]
        at org.opensearch.ml.model.MLModelManager.setupParamsAndPredictable(MLModelManager.java:1164) ~[?:?]
        at org.opensearch.ml.model.MLModelManager.lambda$deployRemoteOrBuiltInModel$56(MLModelManager.java:1153) ~[?:?]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.ml.model.MLModelManager.lambda$getConnector$71(MLModelManager.java:1653) [opensearch-ml-3.0.0.0-SNAPSHOT.jar:3.0.0.0-SNAPSHOT]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:298) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:284) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1483) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1566) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1546) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:62) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:45) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

@yuye-aws
Copy link
Member Author

Seems the bug is related to --preserve-data option. The issue no longer persists after cleaning gradle cache. Closing the bug now.

@b4sjoo b4sjoo removed the untriaged label Jul 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuye-aws @b4sjoo