Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Introduce Aggregate Subcommand] Enhance opensearch-benchmark compare command #630

Closed
OVI3D0 opened this issue Aug 28, 2024 · 0 comments
Closed
Assignees
Labels
enhancement New feature or request

Comments

@OVI3D0
Copy link
Member

OVI3D0 commented Aug 28, 2024

This is an issue based off one of the proposed priorities in this RFC: #627

Background

The existing compare subcommand in OpenSearch Benchmark (OSB) allows users to compare the results of two benchmark test executions by providing the unique IDs (UIDs) of the test executions. However, users have expressed interest in comparing aggregated results from two or more groups of test executions, rather than just two individual tests.

Proposed Design

To address this requirement, we propose enhancing the existing compare subcommand to support comparing two aggregated test results.

When comparing two aggregated test results, a validation step will be added to ensure that the underlying workload (the type of operations being performed, the data set being used, etc.) is consistent across the test executions being aggregated.
The enhanced compare subcommand will allow users to specify two test execution IDs, and OSB will perform the necessary validations before comparing the aggregated results. The output will display the performance differences between the two groups.

Example Usage:

$ opensearch-benchmark compare --baseline=729291a0-ee87-44e5-9b75-cc6d50c89702 --contender=a33845cc-c2e5-4488-a2db-b0670741ff9b

Proposed Priority

The ability to compare two aggregated test results is a highly requested feature from users. It will enable more accurate and representative performance comparisons by reducing the impact of variability and outliers, particularly when evaluating the impact of changes or optimizations across different configurations or workloads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants