Reduce the amount of has_privileges call

[philippkahr]

Version: 8.8.1, everything on Elastic Cloud.

When performing a simple ES Query type alert rule that just checks if the amount of docs is above a threshold, like this:

The Kibana APM and Elasticsearch instrumentation then shows that a lot of time is spent on the has_privileges checks.
from a transaction point of view.

Looking at the spans:

Here is what the entire trace looks like

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[azasypkin]

Based on the very long and intricate APM trace, it appears that unwinding the issue won't be a straightforward task and will likely require collaboration between @elastic/response-ops, @elastic/kibana-security, and @elastic/kibana-core. We might also require assistance from the @elastic/apm-ui to access and interpret these traces (unfortunately, I couldn't even load them in our o11y cluster and had to give up after waiting for around 15 minutes for the timeline to show up).

Since the initial request originates from the Alerting UI and there are several requests to the .kibana_alerting_cases_* index in the trace, I'd ask @elastic/response-ops to assist with the initial investigation to determine if there are any optimizations that can be made at their level. If such optimizations are not feasible or not enough, then we need to identify the specific issues with the Security APIs and other functionalities that are causing the problem.

[dgieselaar]

@azasypkin happy to help out, feel free to put something on my calendar

[philippkahr]

I am not an expert in this topic and I definitely do not want to lead any of you to wrong conclusions. When looking through the APM, I see 5x a GET */_doc/strava%3Aconfig%3A8.8.1. To me, as a non kibana dev, it looks like every of this config pulls is "wrapped" in an has_privileges call. Could we automatically reduce the has_privileges call, when we reduce the amount of config pulls? Maybe even merge the config + rule setting call?

I am not entirely sure if this should be part of this investigation, so I opened a second issue #161382 close it if no needed :)

[rudolf]

This is very similar to #82218 To address that issue we added a 5s UiSettings cache to each request.

I suspect since these are server-side calls we don't use a UiClient from the request context but instead create one with UiSettingsServiceStart.asScopedToClient. In that case we won't benefit from the cache.

[mikecote]

Optimizations have been made in the Kibana Alerting and Task Manager framework to not call the has_privileges API as frequent. A downstream issue still exists whenever using search source / data view services (#192170) and some calls are still made by rule types (ex: security detection rules). Calls to such endpoint should be drastically reduced now. Perhaps sufficient to close this GitHub issue?