[Bug] Ensure OpenSearch Dashboards stays available in large clusters

[boktorbb]

Problem Statement

For sufficiently large OpenSearch clusters, when Dashboards sends a health check out there can be a failure and Dashboards becomes unavailable.

Root Cause of the issue

The default Dashboards behavior is to fan out healthcheck requests across the entire cluster. For large clusters, if any nodes are processing or ingestion heavy and time out then it fails the healthcheck and Dashboards becomes unavailable.

Proposed Dashboards solution

The proposal is to create effectively a similar node attribute and use that for an optimized healthcheck

Dashboards Configuration: Create a setting that’s called optimized_healthcheck in opensearch_dashboards.yml that looks for the OpenSearch Node attribute cluster_id By default, optimized_healthcheck will default to null which lets Dashboards continue fanning out healthcheck requests across all nodes. If the value is cluster_id , then it will switch to the logic outlined in the below algorithm section.

OpenSearch Configuration: cluster_id → a new node attribute to be added in by customers that would differentiate cluster instances

cluster_id can take the form of an integer that is assigned during cluster creation to all OpenSearch nodes. It will increment when new instances of the cluster are spun up

Using this cluster_id we can follow a general algorithm:

Step 1: Aggregate all cluster_id for OpenSearch nodes
Step 2: if all the nodes have same cluster_id, retrieve nodes.info from _local node only.
- Using _cluster/state/nodes to retrieve from each master node
Step 3: if the nodes have different cluster_id, fan out the request to all the nodes.

• Design Doc
• Implementation