[BUG] Cannot execute-test. This test_execution ended with a fatal crash

[lijie123bes]

Describe the bug
Cannot execute-test. This test_execution ended with a fatal crash

To Reproduce
opensearch-benchmark execute-test --pipeline=benchmark-only --workload=http_logs --target-host=http://192.168.0.112:9200 --client-options=basic_auth_user:admin,basic_auth_password:admin,verify_certs:false --test-mode

Logs
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\benchmark.py", line 880, in dispatch_sub_command
execute_test(cfg, args.kill_running_processes)
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\benchmark.py", line 651, in execute_test
with_actor_system(test_execution_orchestrator.run, cfg)
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\benchmark.py", line 678, in with_actor_system
runnable(cfg)
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\test_execution_orchestrator.py", line 386, in run
raise exceptions.BenchmarkError("This test_execution ended with a fatal crash.").with_traceback(tb)
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\test_execution_orchestrator.py", line 378, in run
pipeline(cfg)
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\test_execution_orchestrator.py", line 69, in call
self.target(cfg)
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\test_execution_orchestrator.py", line 314, in benchmark_only
return execute_test(cfg, external=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\test_execution_orchestrator.py", line 263, in execute_test
benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\thespian\actors.py", line 702, in createActor
return self._systemBase.newPrimaryActor(actorClass,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\thespian\system\systemBase.py", line 205, in newPrimaryActor
raise NoCompatibleSystemForActor(
osbenchmark.exceptions.BenchmarkError: This test_execution ended with a fatal crash.
2024-02-01 02:17:39,430 -not-actor-/PID:6308 osbenchmark.test_execution_orchestrator INFO Telling benchmark actor to exit.
2024-02-01 02:17:42,433 -not-actor-/PID:6308 osbenchmark.benchmark INFO Attempting to shutdown internal actor system.
2024-02-01 02:17:42,458 -not-actor-/PID:6308 osbenchmark.benchmark INFO Actor system is still running. Waiting...
2024-02-01 02:17:45,498 -not-actor-/PID:6308 osbenchmark.benchmark INFO Shutdown completed.

[IanHoang]

Hi @lijie123bes, thanks for letting us know that you're experiencing this issue. Based on NoCompatibleSystemForActor, it looks like your cluster is undiscoverable and might not be in the same network as your machine running OSB. Are you able to curl the OpenSearch cluster in the same machine that is running OSB?

To better understand your setup, could you provide more insight into the following:

• What environment are you running OSB in (Linux, AL2, macOS, etc.)?
• How are you setting up your OpenSearch cluster and what OpenSearch version are you using?

It would also be helpful if you could provide the entire logs file related to this test run.

[gkamat]

@lijie123bes, do you have any updates on this issue. If it is not reproducible, we will close it for now.