Endpoint Telemetry: Agents Metrics + Policy Config / Response

[pjhampton]

Summary

This PR retrieves and transmits Endpoint agent telemetry if cluster permissions permit.
There have been auxiliary PRs / Issues opened:

• Granting Kbn System read permissions to read from EP Data stream elasticsearch#74309 - Grant kibana_system user to .ds-metrics-endpoint.policy-*
• Grant kbn system user permissions to read Endpoint metrics datastream elasticsearch#74340 - Grant kibana_system user to .ds-metrics-endpoint.metrics-*
• Open stack infra issue for telemetry channel

We are currently sharing the telemetry with the Endpoint team. We will be making changes to the final payload.

---

Implementation

The implementation is not that straightforward - here is a high level of how it works

• Every 24hr grab an available Kibana worker
• Aggregate the endpoint metrics datastream for the last 24 hours on unique endpoint id for latest metrics document

GET /.ds-metrics-endpoint.metrics*/_search?expand_wildcards=open,hidden
{
  "size": 0,
  "query": {
    "match_all": {}
  },
  "aggs": {
    "endpoint_agents": {
      "terms": { 
        "size": 10000,
        "field": "agent.id" 
      },
      "aggs": {
        "latest_metrics": {
          "top_hits": {
            "size": 1,
            "sort": [
              {
                "@timestamp": {
                  "order": "desc"
                }
              }
            ]
          }
        }
      }
    }
  }
}

• If there is no endpoint metrics we will assume there isn't any endpoint running and bail; otherwise
• Get the full list of fleet agents
• Iterate through the full list of agents, if it has a policy check it is of type endpoint, add to a cache associating the fleet agent id with the policy id
• Aggregate the endpoint policy datastream for the last 24 hours on a unique endpoint id for the latest failed policy response (if exists)

GET /.ds-metrics-endpoint.policy*/_search?expand_wildcards=open,hidden
{
  "size": 0,
  "query": {
    "match": {
      "Endpoint.policy.applied.status": "failure"
    }
  },
  "aggs": {
    "policy_responses": {
      "terms": {
        "field": "Endpoint.policy.applied.id",
        "size": 10
      },
      "aggs": {
        "latest_response": {
          "top_hits": {
            "size": 1,
            "sort": [
              {
                "@timestamp": {
                  "order": "desc"
                }
              }
            ]
          }
        }
      }
    }
  }
}

• Combine the metrics documents with the correct policy configuration and response failure (if applicable)
• Send to the endpoint-meta telemetry channel

Follow up

I'm hosting a call week beginning 28/Jun re this telemetry + design.
Let me know if you want me to swing you an invite.

Checklist

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[pjhampton]

@elasticmachine merge upstream

[pjhampton]

@elasticmachine merge upstream

[Bamieh]

LGTM

[michaelolo24]

Just a personal preference here, but might make it a bit easier to read intent if we store the boolean logic in a variable

const shouldCachePolicy = 
	policyInfo.policy_id !== null &&
	policyInfo.policy_id !== undefined &&
	!endpointPolicyCache.has(policyInfo.policy_id)

[pjhampton]

Yeah, that is slick. Thanks for the feedback!

[michaelolo24]

Looks good! Thanks for making these changes. We can work on performance improvements on a follow up PR 👍🏾

[pjhampton]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: c4316b0
• Storybooks Preview
• Documentation Changes

Metrics [docs]

✅ unchanged

History

• 💚 Build #135288 succeeded 99ceeef
• 💔 Build #135259 failed 4e5804b
• 💚 Build #135054 succeeded cbd499d
• 💚 Build #134989 succeeded eb75a93
• 💚 Build #134546 succeeded 0781c54

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @pjhampton @donaherc

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.