[Security_Solution][Endpoint] Leveraging msearch and ancestry array for resolver #70134

jonathan-buttner · 2020-06-26T22:10:38Z

This PR leverages msearch and the ancestry array to improve the resolver backend's performance. It relies on the generator changes in this PR: #70129

Notable changes:

Related alerts and related events no longer use aggregations, if ES doesn't return anymore data we know we've reached the end
The usage of the nextChild cursors for the children is different since we're leveraging the ancestry array. I've tried to outline the different scenarios in the code's comments
Removed generations since it can't be enforcing while using the ancestry array. The limit is instead base don the number of nodes you'd like to receive
Moved most of the fetcher's logic into individual classes so it can be used for msearch

elasticmachine · 2020-06-26T22:10:40Z

Pinging @elastic/endpoint-data-visibility-team (Team:Endpoint Data Visibility)

elasticmachine · 2020-06-26T22:10:41Z

Pinging @elastic/endpoint-app-team (Feature:Endpoint)

jonathan-buttner · 2020-06-26T22:12:03Z

x-pack/plugins/security_solution/common/endpoint/generate_data.test.ts

@@ -113,6 +113,8 @@ describe('data generator', () => {
        percentWithRelated: 100,
        relatedEvents: 0,
        relatedAlerts: 0,
+        useAncestryArray: true,


These changes are in this PR: #70129

they will be removed once the other PR gets merged.

jonathan-buttner · 2020-06-26T22:12:15Z

x-pack/plugins/security_solution/common/endpoint/generate_data.ts

@@ -17,6 +17,7 @@ import {
  EndpointStatus,
 } from './types';
 import { factory as policyFactory } from './models/policy_config';
+import { parentEntityId } from './models/event';


These changes are in this PR: #70129

they will be removed once the other PR gets merged.

jonathan-buttner · 2020-06-26T22:14:22Z

x-pack/plugins/security_solution/server/endpoint/routes/resolver/queries/base.ts

@@ -17,7 +17,7 @@ import { MSearchQuery } from './multi_searcher';
 * @param T the structured return type of a resolver query. This represents the type that is returned when translating
 * Elasticsearch's SearchResponse<ResolverEvent> response.
 */
-export abstract class ResolverQuery<T> implements MSearchQuery {
+export abstract class ResolverQuery<T, R = ResolverEvent> implements MSearchQuery {


The R here sets the stage for being able to limit the _source of a queries response. This will be in a follow up PR. Limit the source will be a performance improvement for querying for the start events for the children.

jonathan-buttner · 2020-06-26T22:27:08Z

x-pack/plugins/security_solution/server/endpoint/routes/resolver/queries/children.ts

-  formatResponse(response: SearchResponse<ResolverEvent>): PaginatedResults {
-    return {
-      results: ResolverQuery.getResults(response),
-      totals: PaginationBuilder.getTotals(response.aggregations),


Totals are inferred based on the response of ES while using the ancestry array.

jonathan-buttner · 2020-06-26T22:28:01Z

x-pack/plugins/security_solution/server/endpoint/routes/resolver/queries/multi_searcher.ts

+  /**
+   * a function to handle the response
+   */
+  handler: (response: SearchResponse<ResolverEvent>) => void;


This calls into the class (e.g. RelatedEventsQueryHandler etc) for it to handle the response from ES.

jonathan-buttner · 2020-06-26T22:34:29Z

x-pack/test/api_integration/apis/endpoint/resolver.ts

          const { body }: { body: ResolverChildren } = await supertest
-            .get(`/api/endpoint/resolver/${tree.origin.id}/children?generations=1`)


The generator creates nodes in depth first orderish. I don't want to have to rely on that though in the test. To get around that I move to a level of the tree where I know there aren't any other descendants before the immediate children.

jonathan-buttner · 2020-06-26T22:35:28Z

x-pack/test/api_integration/apis/endpoint/resolver.ts

@@ -16,6 +16,7 @@ import {
  LegacyEndpointEvent,
  ResolverNodeStats,
  ResolverRelatedAlerts,
+  ResolverChildNode,


TODO need to write more tests for the children's pagination.

…try-array

…actor

michaelolo24 · 2020-07-01T13:35:18Z

x-pack/plugins/security_solution/common/endpoint/types.ts

-   * 1. At the time of querying it could have no children in ES, in which case it will be marked as
-   *  null because we know it does not have children during this query.
-   * 2. If the max level was reached we do not know if this node has children or not so we'll mark it as null
+   * nextChild can have 3 different states:


Just a thought, I understand what this is saying, especially after reading through your write up, but one of @oatkiller's famous diagrams may be useful here, you could probs just copy paste one he's already done. Not necessary, but I think it could be helpful

Thanks @michaelolo24 do you mean an ascii diagram of a tree? Or a table?

ascii I think

I think what I'll probably do is put a link to the relevant section in the docs that I'll be adding as a follow up or add on to this PR.

michaelolo24 · 2020-07-01T13:48:00Z

x-pack/plugins/security_solution/server/endpoint/routes/resolver/utils/alerts_query_handler.ts

+
+    return {
+      query: this.query,
+      ids: this.entityID,


does the msearch query take strings and arrays or just strings? I know in your constructor it's looking for a string, I'm just confused by the pluralized id

It's both string | string[] https://github.com/jonathan-buttner/kibana/blob/msearch-ancestry-array/x-pack/plugins/security_solution/server/endpoint/routes/resolver/queries/multi_searcher.ts#L36

And some of the other *_query_handler files pass it string[] particularly the children one.

missed that, thanks!

michaelolo24 · 2020-07-01T13:50:50Z

x-pack/plugins/security_solution/server/endpoint/routes/resolver/utils/alerts_query_handler.ts

+  /**
+   * Get the results after an msearch.
+   */
+  getResults() {


Is this only used internally to the class?

No it's used by the fetch.ts file as well: https://github.com/jonathan-buttner/kibana/blob/msearch-ancestry-array/x-pack/plugins/security_solution/server/endpoint/routes/resolver/utils/fetch.ts#L178

michaelolo24 · 2020-07-01T14:05:34Z

...ck/plugins/security_solution/server/endpoint/routes/resolver/utils/ancestry_query_handler.ts

+    // bucket the start and end events together for a single node
+    const ancestryNodes = this.toMapOfNodes(results);
+
+    // the order of this array is going to be weird, it will look like this


not necessary, but I feel like a mini diagram could be helpful here too

I'll add the diagram in a follow up PR so I don't have to wait for ci again haha

michaelolo24 · 2020-07-01T14:12:54Z

x-pack/plugins/security_solution/server/endpoint/routes/resolver/utils/children_helper.test.ts

+  return children;
+}
+
+function getStartEventsFromLevels(levels: Array<Map<string, TreeNode>>) {


Wonder what the performance here is gonna look like at scale 😅.

This is just a test file :)

that'll do it 😂

jonathan-buttner · 2020-07-01T19:13:25Z

@elasticmachine merge upstream

elasticmachine · 2020-07-01T19:13:27Z

merge conflict between base and head

kibanamachine · 2020-07-01T20:54:39Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 200ce25

Build metrics

✅ unchanged

History

💚 Build #57300 succeeded d274a07
💚 Build #57252 succeeded c99773b
💚 Build #57220 succeeded caa8f8f
💚 Build #57113 succeeded 79464ed
💔 Build #56889 failed 6cdf961

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…or resolver (#70134) (#70583) * Refactor generator for ancestry support * Adding optional ancestry array * Refactor the pagination since the totals are not used anymore * Updating the queries to not use aggregations for determining the totals * Refactoring the children helper to handle pagination without totals * Pinning the seed for the resolver tree generator service * Splitting the fetcher into multiple classes for msearch * Updating tests and api for ancestry array and msearch * Adding more comments and fixing type errors * Fixing resolver test import * Fixing tests and type errors * Fixing type errors and tests * Removing useAncestry field * Fixing test * Removing useAncestry field from tests * An empty array will be returned because that's how ES will do it too

* master: Update known-plugins.asciidoc (elastic#69370) [ML] Anomaly Explorer swim lane pagination (elastic#70063) [Ingest Manager] Update asset paths to use _ instead of - (elastic#70320) Fix discover, tsvb and Lens chart theming issues (elastic#69695) Allow Saved Object type mappings to set a field's `doc_values` property (elastic#70433) [S&R] Support data streams (elastic#68078) [Maps] Add styling and tooltip support to mapbox mvt vector tile sources (elastic#64488) redirect to default app if hash can not be forwarded (elastic#70417) [APM] Don't fetch dynamic index pattern in setupRequest (elastic#70308) [Security_Solution][Endpoint] Leveraging msearch and ancestry array for resolver (elastic#70134) Update docs for api authentication usage (elastic#66819) chore(NA): disable alerts_detection_rules cypress suites (elastic#70577) add getVisibleTypes API to SO type registry (elastic#70559)

jonathan-buttner added 8 commits June 26, 2020 17:38

Refactor generator for ancestry support

1f0e448

Adding optional ancestry array

f0ee76a

Refactor the pagination since the totals are not used anymore

c95132f

Updating the queries to not use aggregations for determining the totals

e4b8a13

Refactoring the children helper to handle pagination without totals

407db62

Pinning the seed for the resolver tree generator service

8947b43

Splitting the fetcher into multiple classes for msearch

1287379

Updating tests and api for ancestry array and msearch

54f1a3f

jonathan-buttner added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Endpoint Data Visibility Team managing the endpoint resolver Feature:Endpoint Elastic Endpoint feature v7.9.0 labels Jun 26, 2020

jonathan-buttner requested review from oatkiller, bkimmel, michaelolo24 and kqualters-elastic June 26, 2020 22:10

jonathan-buttner requested review from a team as code owners June 26, 2020 22:10

jonathan-buttner added 2 commits June 26, 2020 18:25

Adding more comments and fixing type errors

92005dd

Fixing resolver test import

6cdf961

jonathan-buttner commented Jun 26, 2020

View reviewed changes

jonathan-buttner added 6 commits June 29, 2020 09:15

Fixing tests and type errors

40f5537

Fixing merge conflicts

f54a462

Fixing type errors and tests

1eac63f

Merge branch 'master' of github.com:elastic/kibana into msearch-ances…

79464ed

…try-array

Merge branch 'master' of github.com:elastic/kibana into generator-ref…

5a872da

…actor

Removing useAncestry field

99acb58

jonathan-buttner added 6 commits June 29, 2020 12:06

Fixing test

3f9f3eb

Merge branch 'generator-refactor' into msearch-ancestry-array

e75f7bf

Removing useAncestry field from tests

caa8f8f

An empty array will be returned because that's how ES will do it too

544e1f3

Merge branch 'generator-refactor' into msearch-ancestry-array

c99773b

Resolving conflicts

d274a07

michaelolo24 reviewed Jul 1, 2020

View reviewed changes

michaelolo24 approved these changes Jul 1, 2020

View reviewed changes

Resolving conflicts

200ce25

jonathan-buttner merged commit c081caa into elastic:master Jul 2, 2020

jonathan-buttner deleted the msearch-ancestry-array branch July 2, 2020 12:47

jonathan-buttner mentioned this pull request Jul 2, 2020

[7.x] [Security_Solution][Endpoint] Leveraging msearch and ancestry array for resolver (#70134) #70583

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security_Solution][Endpoint] Leveraging msearch and ancestry array for resolver #70134

[Security_Solution][Endpoint] Leveraging msearch and ancestry array for resolver #70134

jonathan-buttner commented Jun 26, 2020 •

edited

Loading

elasticmachine commented Jun 26, 2020

elasticmachine commented Jun 26, 2020

jonathan-buttner Jun 26, 2020

jonathan-buttner Jun 26, 2020

jonathan-buttner Jun 26, 2020

jonathan-buttner Jun 26, 2020

jonathan-buttner Jun 26, 2020

jonathan-buttner Jun 26, 2020

jonathan-buttner Jun 26, 2020

michaelolo24 Jul 1, 2020

jonathan-buttner Jul 1, 2020

michaelolo24 Jul 1, 2020

jonathan-buttner Jul 1, 2020

michaelolo24 Jul 1, 2020

jonathan-buttner Jul 1, 2020

michaelolo24 Jul 1, 2020

michaelolo24 Jul 1, 2020

jonathan-buttner Jul 1, 2020

michaelolo24 Jul 1, 2020

jonathan-buttner Jul 2, 2020

michaelolo24 Jul 1, 2020 •

edited

Loading

jonathan-buttner Jul 1, 2020 •

edited

Loading

michaelolo24 Jul 1, 2020

jonathan-buttner commented Jul 1, 2020

elasticmachine commented Jul 1, 2020

kibanamachine commented Jul 1, 2020

		const { body }: { body: ResolverChildren } = await supertest
		.get(`/api/endpoint/resolver/${tree.origin.id}/children?generations=1`)

[Security_Solution][Endpoint] Leveraging msearch and ancestry array for resolver #70134

[Security_Solution][Endpoint] Leveraging msearch and ancestry array for resolver #70134

Conversation

jonathan-buttner commented Jun 26, 2020 • edited Loading

elasticmachine commented Jun 26, 2020

elasticmachine commented Jun 26, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelolo24 Jul 1, 2020 • edited Loading

Choose a reason for hiding this comment

jonathan-buttner Jul 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathan-buttner commented Jul 1, 2020

elasticmachine commented Jul 1, 2020

kibanamachine commented Jul 1, 2020

💚 Build Succeeded

Build metrics

History

jonathan-buttner commented Jun 26, 2020 •

edited

Loading

michaelolo24 Jul 1, 2020 •

edited

Loading

jonathan-buttner Jul 1, 2020 •

edited

Loading