Enable run on macOS

[sharp-pixel]

Description

Enables run on macOS.

Note 1: due to OpenSearch 1.0.1 not being available for Apple Silicon, the integration tests fail on a Mac M1. There is not much that I can do here.

Note 2: this PR also fixes the templates to run on Amazon OpenSearch Service with OpenSearch 1.1, as the _doc is now an error.

Issues Resolved

#89

Check List

• New functionality includes testing
  • All unit and integration tests pass
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dblock]

Let's also add Darwin to CI?

[sharp-pixel]

Anything I can do from my side?

[dblock]

Yes @sharp-pixel! Add macos-latest to https://github.com/opensearch-project/opensearch-benchmark/blob/main/.github/workflows/main.yml as part of this PR?

[sharp-pixel]

Added

[dblock]

How about this?

[sharp-pixel]

Hum, time based unit tests... interesting.
Python time.sleep's implementation on macOS seems not to be accurate enough: https://stackoverflow.com/questions/1133857/how-accurate-is-pythons-time-sleep
Also these tests would be highly dependent of the underlying system.
Now the problem is that is the system's time.sleep is not accurate enough and this is used for the actual benchmarking then the results would be unreliable, and it would be impossible to compare benchmarks run from different environments.

A suggestion would be to remove these tests on the build system, but run them at start of an actual benchmark to validate that the system running it is within accuracy limits. Open for discussion.

[dblock]

Generally, I recommend disabling unpredictable/unreliable tests and opening an issue to replace/fix them to be 100% reliable.

[engechas]

Based on the mac CI unit tests, there's other tests that are time specific and will need to be disabled as well

FAILED tests/client_test.py::RequestContextManagerTests::test_propagates_nested_context
FAILED tests/worker_coordinator/runner_test.py::CompositeTests::test_adds_request_timings

Rest LGTM

Looking further into the ramifications of removing _doc

[sharp-pixel]

The _doc will be a problem for Elasticsearch < v6.7, and between 6.7 and 7.x, we would need to set include_type_name=false explicitely. This is the default after Elasticsearch 7.0 and for OpenSearch.

[sharp-pixel]

If we want to still support older versions, we would need two versions of the templates, check version and use the correct ones.

[engechas]

It looks like include_type_name based on version was accounted for here but doesn't work with OpenSearch metrics stores since the version is reported as 1.

I see two options:

1. Remove _doc, forcing metrics stores to be ES7+ or OpenSearch
2. Refactor the logic here to send include_type_name=true for OpenSearch metrics stores

Either option seems fine to me. Curious to hear others' thoughts

[sharp-pixel]

It looks like include_type_name based on version was accounted for here but doesn't work with OpenSearch metrics stores since the version is reported as 1.

I see two options:

1. Remove `_doc`, forcing metrics stores to be ES7+ or OpenSearch

2. Refactor the logic [here](https://github.com/opensearch-project/opensearch-benchmark/blob/ba96b9a262a303a739e39296ec676e6908c21905/osbenchmark/metrics.py#L68-L72) to send `include_type_name=true` for OpenSearch metrics stores

Either option seems fine to me. Curious to hear others' thoughts

I made some changes:

• I restored the _doc in the templates for now.
• I added a new _cluster_distribution field in OsClient that holds the distribution (either 'opensearch' or 'elasticsearch'), that way we can differentiate versions for OpenSearch from versions for Elasticsearch.

[engechas]

LGTM

Integration tests using an OpenSearch metrics store with these changes and the changes in #119 are passing

[engechas]

I have created followup issues to break the DCO out into it's own file and to re-evalutate the disabled unit tests: #120 #121

Merging this PR, thanks!

Changes were addressed, follow up issue created to break the DCO check into a separate file #120