[Security_Solution][Telemetry] - Update endpoint usage to use agentService #93829

michaelolo24 · 2021-03-05T19:58:15Z

Summary

This PR addresses the first point made by @nchaulet in this issue: #92311

We were previously accessing the saved_objects for fleet directly to populate endpoint usage telemetry, but should have been using the agentService where possible. This PR adopts that change to continue to reliably provide us with the total_installed and os details information we currently have available for telemetry.

The outstanding issue remains that there is no way to currently reliably obtain the information we are getting from the fleet-agent-events savedObject which helps to populate the active_within_last_24_hours and policies.malware data. With the move to fleet server as detailed in the aforementioned issue, our use of that savedObject will break and we will no longer get this data. Ideally, we'll be able to determine a solution that allows us to retain this information and is better implemented than the current scenario (also detailed here: https://github.com/elastic/endpoint-dev/issues/8382)

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2021-03-05T19:58:17Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2021-03-05T19:58:18Z

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

michaelolo24 · 2021-03-05T19:58:42Z

x-pack/plugins/security_solution/server/usage/endpoints/fleet_saved_objects.ts

+};
+
+/*
+  TODO: AS OF 7.13, this access will no longer work due to the enabling of fleet server. An alternative route will have


This will break in 7.13

Should we also put this information in an issue?

When it breaks in 7.13 will it have an adverse affect on the plugin? Or will it just log and fail silently?

We currently don't log to prevent accidental leaking of PII, but yea if it stays as is for 7.13, we'll only get the total_installed and os details. Everything else will fail silently. Yea, I'm gonna add it into the issue that @nchaulet created that's referenced in the description here.

@jonathan-buttner actually went with a comment here to avoid splitting the discussion

michaelolo24 · 2021-03-05T19:58:58Z

x-pack/plugins/security_solution/server/usage/endpoints/index.ts

@@ -108,7 +109,7 @@ export const updateEndpointOSTelemetry = (
 * the same time span.
 */
 export const updateEndpointDailyActiveCount = (
-  latestEndpointEvent: SavedObject<NewAgentEvent>,
+  latestEndpointEvent: SavedObject<NewAgentEvent>, // TODO: This information will be lost in 7.13, need to find an alternative route.


This will break in 7.13

michaelolo24 · 2021-03-08T12:37:29Z

@elasticmachine merge upstream

nchaulet · 2021-03-08T12:50:39Z

x-pack/plugins/security_solution/server/usage/endpoints/fleet_saved_objects.ts

+  agentService: AgentService | undefined,
+  esClient: ElasticsearchClient
+) => {
+  const agentData = await agentService?.listAgents(savedObjectsClient, esClient, {


I just merged the change where we remove the support for saved object, I think this method is probably not accepting a savedObjectClient any more

Thanks for the note. I'll update this accordingly!

nchaulet

Code looks good you probably want to merge master, before merging

michaelolo24 · 2021-03-10T00:49:47Z

@elasticmachine merge upstream

michaelolo24 · 2021-03-10T13:40:42Z

@elasticmachine merge upstream

kibanamachine · 2021-03-10T15:47:14Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💔 Build #112341 failed 3572012
💔 Build #112285 failed 942f5ef
💔 Build #111760 failed 32b1012
💔 Build #111684 failed 18f30b369163a9a7776a20fff6de70c53311702e
💔 Build #111487 failed 81c5ea54d636777c06c1c3782a3f5ee1d0a6ac18

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @michaelolo24

michaelolo24 · 2021-03-10T20:09:41Z

Discussed with @achuguy who wrote the current endpoint_telemetry functional tests. We're deciding to skip them for now until we finalize how telemetry will be handled moving forward. I'll be creating an issue to update these tests

…rvice (elastic#93829) Co-authored-by: Kibana Machine <[email protected]>

…rvice (#93829) (#94366) Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

…-action * 'master' of github.com:elastic/kibana: (43 commits) [Console] Update copy when showing warnings in response headers (#94270) [TSVB] Type public code. Step 1 (#93231) [ML] Functional tests - stabilize slider value selection (#94313) skip another suite blocking es promotion (#94367) [Security Solution] Eliminates a redundant external link icon (#94194) skip another suite blocking es promotion (#94367) [App Search] Role mappings migration part 1 (#94346) [Security Solution][Detections] Fix flaky indicator enrichment tests (#94241) [Workplace Search] Deduplicate icons (#94359) [ML] Add latest transform to intro text (#94039) skip test failing es promotion (#94367) [Maps] convert elasticsearch_utils to TS (#93984) [Security_Solution][Telemetry] - Update endpoint usage to use agentService (#93829) [Security Solution][Exceptions] Fixes OS adding method for exception enrichment (#94343) [ILM] Add support for frozen phase (#93068) [App Search] Fixed 2 relevance tuning bugs (#94312) remove `try` auth mode (#94287) Removing resolver functional tests (#94331) migrate warning mixin to core (#94273) [App Search] Add routes for Role Mappings (#94221) ... # Conflicts: # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/phases/cold_phase/cold_phase.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/phases/phase/phase.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/phases/shared_fields/searchable_snapshot_field/searchable_snapshot_field.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/edit_policy.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/form/configuration_issues_context.tsx

michaelolo24 added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Threat Hunting Security Solution Threat Hunting Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v7.13.0 labels Mar 5, 2021

michaelolo24 requested a review from nchaulet March 5, 2021 19:58

michaelolo24 requested a review from a team as a code owner March 5, 2021 19:58

michaelolo24 self-assigned this Mar 5, 2021

michaelolo24 commented Mar 5, 2021

View reviewed changes

michaelolo24 requested a review from paul-tavares March 8, 2021 12:39

nchaulet reviewed Mar 8, 2021

View reviewed changes

nchaulet approved these changes Mar 8, 2021

View reviewed changes

michaelolo24 added 2 commits March 8, 2021 09:50

update telemetry to use agentService

9b665ba

update based on service api changes

32b1012

michaelolo24 force-pushed the update-endpoint-telemetry branch from 18f30b3 to 32b1012 Compare March 8, 2021 15:39

skip ftr test for now

942f5ef

Merge branch 'master' into update-endpoint-telemetry

3572012

Merge branch 'master' into update-endpoint-telemetry

64551ec

jonathan-buttner approved these changes Mar 10, 2021

View reviewed changes

michaelolo24 merged commit ebd92a6 into elastic:master Mar 10, 2021

michaelolo24 deleted the update-endpoint-telemetry branch March 10, 2021 20:10

michaelolo24 mentioned this pull request Mar 10, 2021

[7.x] [Security_Solution][Telemetry] - Update endpoint usage to use agentService (#93829) #94366

Merged

michaelolo24 added a commit to michaelolo24/kibana that referenced this pull request Mar 10, 2021

[Security_Solution][Telemetry] - Update endpoint usage to use agentSe…

5ce704f

…rvice (elastic#93829) Co-authored-by: Kibana Machine <[email protected]>

michaelolo24 added a commit that referenced this pull request Mar 10, 2021

[Security_Solution][Telemetry] - Update endpoint usage to use agentSe…

c9390c5

…rvice (#93829) (#94366) Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security_Solution][Telemetry] - Update endpoint usage to use agentService #93829

[Security_Solution][Telemetry] - Update endpoint usage to use agentService #93829

michaelolo24 commented Mar 5, 2021

elasticmachine commented Mar 5, 2021

elasticmachine commented Mar 5, 2021

michaelolo24 Mar 5, 2021

jonathan-buttner Mar 8, 2021

michaelolo24 Mar 8, 2021

michaelolo24 Mar 9, 2021

michaelolo24 Mar 5, 2021

michaelolo24 commented Mar 8, 2021

nchaulet Mar 8, 2021

michaelolo24 Mar 8, 2021

nchaulet left a comment

michaelolo24 commented Mar 10, 2021

michaelolo24 commented Mar 10, 2021

kibanamachine commented Mar 10, 2021

michaelolo24 commented Mar 10, 2021 •

edited

Loading

[Security_Solution][Telemetry] - Update endpoint usage to use agentService #93829

[Security_Solution][Telemetry] - Update endpoint usage to use agentService #93829

Conversation

michaelolo24 commented Mar 5, 2021

Summary

Checklist

elasticmachine commented Mar 5, 2021

elasticmachine commented Mar 5, 2021

michaelolo24 Mar 5, 2021

Choose a reason for hiding this comment

jonathan-buttner Mar 8, 2021

Choose a reason for hiding this comment

michaelolo24 Mar 8, 2021

Choose a reason for hiding this comment

michaelolo24 Mar 9, 2021

Choose a reason for hiding this comment

michaelolo24 Mar 5, 2021

Choose a reason for hiding this comment

michaelolo24 commented Mar 8, 2021

nchaulet Mar 8, 2021

Choose a reason for hiding this comment

michaelolo24 Mar 8, 2021

Choose a reason for hiding this comment

nchaulet left a comment

Choose a reason for hiding this comment

michaelolo24 commented Mar 10, 2021

michaelolo24 commented Mar 10, 2021

kibanamachine commented Mar 10, 2021

💚 Build Succeeded

Metrics [docs]

History

michaelolo24 commented Mar 10, 2021 • edited Loading

michaelolo24 commented Mar 10, 2021 •

edited

Loading