[Ingest Manager] KQL in agent checkin performance issue

[nchaulet]

Description

In the agent checkin we are doing a query to get AgentActions saved objects and this query use KQL. After some CPU profile (see screenshot below) it seems that is really inefficient as SO client parse the KQL query each time and this operation is CPU intensive.

The search query https://github.com/elastic/kibana/blob/master/x-pack/plugins/ingest_manager/server/services/agents/actions.ts/#L34

During a 20s, CPU profile of 2000 agents enrolled we passed 1.2 seconds for parsing KQL queries

Potential solutions

Fix KQL perfomance

Can we make the saved object filter option faster? implement some caching? (we use dynamic KQL, so it could be hard to cache) @kobelb curious to have your thoughts on that.

Change our data model

Currently we rely on a property on the AgentAction, sent_at to know if is an action has been sent to the agent, we can change our data model, to add the actions to send to the agent saved object something like

Agent {
  "actions_to_send": ["action1:id", "action2id"]
}

and the checkin will be

1. get the agent SO (this call is already done)
2. if there is actions do a bulkGet to get this actions.

Warning: we can potentially have race conditions when adding/removing actions actions_to_send

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[jfsiii]

@nchaulet can you upload a .cpuprofile like shown in the description? It'd be great to capture the current state.