[Bug]: Trying to create-workload in Amazon OpenSearch Serverless results in a HTTP 404

[jwoehrle]

Describe the bug

I'm trying to create a workload from an existing Amazon OpenSearch Serverless Collection. However the create-workload command fails with

[ERROR] Cannot create-workload. NotFoundError(404, '').

To reproduce

1. Create an Amazon OpenSearch Serverless Collection & Index some data
2. Verify that the data is there and the index exists:
   GET https://<COLLECTION_ID>.us-east-1.aoss.amazonaws.com:443/vehicles/_count returns the correct number of documents
3. Verify that environment is setup correctly:

❯ env | grep OSB
OSB_SERVICE=aoss
OSB_AWS_ACCESS_KEY_ID=<REDACTED>
OSB_AWS_SECRET_ACCESS_KEY=<REDACTED>
OSB_AWS_SESSION_TOKEN=<REDACTED>
OSB_REGION=us-east-1

4. try to create a workload:

  opensearch-benchmark create-workload \
--workload="vehicles-serverless" \
--target-hosts="https://<COLLECTION_ID>.us-east-1.aoss.amazonaws.com" \
--client-options="amazon_aws_log_in:environment" \
--indices="vehicles" \
--output-path="."

5. fails with

[ERROR] Cannot create-workload. NotFoundError(404, '').

Expected behavior

The workload gets created as it would be with a "non-serverless" OpenSearch Domain

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

❯ python --version
Python 3.11.7
❯ opensearch-benchmark --version
opensearch-benchmark 1.10.0
❯ sw_vers
ProductName:		macOS
ProductVersion:		14.7
BuildVersion:		23H124

Additional context

No response

Relevant log output

2024-10-16 14:33:00,308 -not-actor-/PID:12199 opensearch WARNING GET https://<COLLECTION_ID>.us-east-1.aoss.amazonaws.com:443/ [status:404 request:0.371s]
2024-10-16 14:33:00,308 -not-actor-/PID:12199 osbenchmark.benchmark ERROR A fatal error occurred while running subcommand [create-workload].
Traceback (most recent call last):
  File "<DIR>/.venv/lib/python3.11/site-packages/osbenchmark/benchmark.py", line 974, in dispatch_sub_command
    workload_generator.create_workload(cfg)
  File "<DIR>/.venv/lib/python3.11/site-packages/osbenchmark/workload_generator/workload_generator.py", line 39, in create_workload
    info = client.info()
           ^^^^^^^^^^^^^
  File "<DIR>/.venv/lib/python3.11/site-packages/opensearchpy/client/utils.py", line 176, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<DIR>/.venv/lib/python3.11/site-packages/opensearchpy/client/__init__.py", line 308, in info
    return self.transport.perform_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<DIR>/.venv/lib/python3.11/site-packages/opensearchpy/transport.py", line 458, in perform_request
    raise e
  File "<DIR>/.venv/lib/python3.11/site-packages/opensearchpy/transport.py", line 419, in perform_request
    status, headers_response, data = connection.perform_request(
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<DIR>/.venv/lib/python3.11/site-packages/opensearchpy/connection/http_urllib3.py", line 308, in perform_request
    self._raise_error(
  File "<DIR>/.venv/lib/python3.11/site-packages/opensearchpy/connection/base.py", line 315, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
opensearchpy.exceptions.NotFoundError: NotFoundError(404, '')

[IanHoang]

Will take a look into this