[2.x] Optimize privilege evaluation for index permissions across '*' index pattern (i.e. all_access role)

[cwperks]

Description

Creating this backport to explore a small optimization for the all_access role when evaluating index permissions in clusters with a large number of indices.

This PR is not a strategic fix like #4898. #4898 takes a holistic approach to optimize privilege evaluation for any type of role definition.

• Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Performance improvement

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (55ff20e) to head (30d457c).
Report is 27 commits behind head on 2.x.

Additional details and impacted files

@@            Coverage Diff             @@
##              2.x   #4926       +/-   ##
==========================================
- Coverage   63.91%       0   -63.92%     
==========================================
  Files         330       0      -330     
  Lines       23118       0    -23118     
  Branches     3745       0     -3745     
==========================================
- Hits        14777       0    -14777     
+ Misses       6509       0     -6509     
+ Partials     1832       0     -1832     

see 330 files with indirect coverage changes

[shikharj05]

Thanks for the contribution, can we add some tests?

[cwperks]

Thanks for the contribution, can we add some tests?

Added unit tests.

[cwperks]

I setup OSB to test this change applied in a 2.18 cluster and pretty consistently saw an improvement when ingesting data with the all_access role.

1. Create a large number of indices (for testing I used both 500 or 1k)

Example script to create test indices:

from opensearchpy import OpenSearch

# OpenSearch connection details
opensearch_client = OpenSearch(
    hosts=[{'host': 'localhost', 'port': 9200}],
    http_auth=('admin', 'admin'),
    use_ssl=True,
    verify_certs=False,
)

def create_indices():
    """
    Create 30,000 indices in the OpenSearch cluster.
    """
    for i in range(1000):
        index_name = f"test-index-{i}"
        response = opensearch_client.indices.create(index=index_name, ignore=400)
        if 'acknowledged' in response and response['acknowledged']:
            print(f"Successfully created index: {index_name}")
        else:
            print(f"Failed to create index: {index_name}, Response: {response}")

if __name__ == "__main__":
    create_indices()

2. Run OSB workflow

opensearch-benchmark execute-test --pipeline=benchmark-only --workload=geonames --target-host=https://localhost:9200 --client-options=basic_auth_user:admin,basic_auth_password:admin,verify_certs:false --include-tasks=delete-index,create-index,check-cluster-health,index-append --kill-running-processes

Results for 3 node cluster, with 1k indices:

1. Without change applied

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                                                         Metric |         Task |     Value |   Unit |
|---------------------------------------------------------------:|-------------:|----------:|-------:|
|                     Cumulative indexing time of primary shards |              |   40.6637 |    min |

2. With change applied

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                                                         Metric |         Task |    Value |   Unit |
|---------------------------------------------------------------:|-------------:|---------:|-------:|
|                     Cumulative indexing time of primary shards |              |  33.3142 |    min |

On a separate runs, one result was close to 30s cumulative time for indexing and another 37.75s

Similar gains were seen with different # of indices as well.

[derek-ho]

LGTM

[derek-ho]

@cwperks can we create an issue to get this into main once the situation with OPE is resolved/backported?