[Security Solutions] Exposes the search_after and point in time (pit) from saved objects to exception lists #125182

FrankHassanabad · 2022-02-10T02:38:09Z

Summary

Exposes the functionality of

search_after
point in time (pit)

From saved objects to the exception lists. This DOES NOT expose these to the REST API just yet. Rather this exposes it at the API level to start with and changes code that had hard limits of 10k and other limited loops. I use the batching of 1k for this at a time as I thought that would be a decent batch guess and I see other parts of the code changed to it. It's easy to change the 1k if we find we need to throttle back more as we get feedback from others.

See this PR where PIT and search_after were first introduced: #89915
See these 2 issues where we should be using more paging and PIT (Point in Time) with search_after: #93770 #103944

The new methods added to the exception_list_client.ts client class are:

openPointInTime
closePointInTime
findExceptionListItemPointInTimeFinder
findExceptionListPointInTimeFinder
findExceptionListsItemPointInTimeFinder
findValueListExceptionListItemsPointInTimeFinder

The areas of functionality that have been changed:

Exception list exports
Deletion of lists
Getting exception list items when generating signals

Note that currently we use our own ways of looping over the saved objects which you can see in the codebase such as this older way below which does work but had a limitation of 10k against saved objects and did not do point in time (PIT)

Older way example (deprecated):

  let page = 1;
  let ids: string[] = [];
  let foundExceptionListItems = await findExceptionListItem({
    filter: undefined,
    listId,
    namespaceType,
    page,
    perPage: PER_PAGE,
    pit: undefined,
    savedObjectsClient,
    searchAfter: undefined,
    sortField: 'tie_breaker_id',
    sortOrder: 'desc',
  });
  while (foundExceptionListItems != null && foundExceptionListItems.data.length > 0) {
    ids = [
      ...ids,
      ...foundExceptionListItems.data.map((exceptionListItem) => exceptionListItem.id),
    ];
    page += 1;
    foundExceptionListItems = await findExceptionListItem({
      filter: undefined,
      listId,
      namespaceType,
      page,
      perPage: PER_PAGE,
      pit: undefined,
      savedObjectsClient,
      searchAfter: undefined,
      sortField: 'tie_breaker_id',
      sortOrder: 'desc',
    });
  }
  return ids;

But now that is replaced with this newer way using PIT:

  // Stream the results from the Point In Time (PIT) finder into this array
  let ids: string[] = [];
  const executeFunctionOnStream = (response: FoundExceptionListItemSchema): void => {
    const responseIds = response.data.map((exceptionListItem) => exceptionListItem.id);
    ids = [...ids, ...responseIds];
  };

  await findExceptionListItemPointInTimeFinder({
    executeFunctionOnStream,
    filter: undefined,
    listId,
    maxSize: undefined, // NOTE: This is unbounded when it is "undefined"
    namespaceType,
    perPage: 1_000,
    savedObjectsClient,
    sortField: 'tie_breaker_id',
    sortOrder: 'desc',
  });
  return ids;

We also have areas of code that has perPage listed at 10k or a constant that represents 10k which this removes in most areas (but not all areas):

      const items = await client.findExceptionListsItem({
        listId: listIds,
        namespaceType: namespaceTypes,
        page: 1,
        pit: undefined,
        perPage: MAX_EXCEPTION_LIST_SIZE, // <--- Really bad to send in 10k per page at a time
        searchAfter: undefined,
        filter: [],
        sortOrder: undefined,
        sortField: undefined,
      });

That is now:

      // Stream the results from the Point In Time (PIT) finder into this array
      let items: ExceptionListItemSchema[] = [];
      const executeFunctionOnStream = (response: FoundExceptionListItemSchema): void => {
        items = [...items, ...response.data];
      };

      await client.findExceptionListsItemPointInTimeFinder({
        executeFunctionOnStream,
        listId: listIds,
        namespaceType: namespaceTypes,
        perPage: 1_000,
        filter: [],
        maxSize: undefined, // NOTE: This is unbounded when it is "undefined"
        sortOrder: undefined,
        sortField: undefined,
      });

Left over areas will be handled in separate PR's because they are in other people's code ownership areas.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

…o the exception lists

…ere.

packages/kbn-securitysolution-io-ts-list-types/src/common/pit/index.ts

packages/kbn-securitysolution-io-ts-list-types/src/common/pit/index.test.ts

packages/kbn-securitysolution-io-ts-list-types/src/common/search_after/index.test.ts

x-pack/plugins/lists/server/routes/delete_list_route.ts

yctercero

Code LGTM! It's super clean and easy to go through, especially with all the comments and tests you added that add lots of context. I really like the patterns you introduced too that will make it super easy to move the rest over to PIT.

Just fetching down now to play around with it and then will LGTM!

...gins/lists/server/services/exception_lists/find_exception_list_items_point_in_time_finder.ts

x-pack/plugins/lists/server/services/exception_lists/utils/get_exception_list_filter.test.ts

kibana-ci · 2022-02-15T21:49:47Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`lists`	293	296	+3
`securitySolution`	2847	2850	+3
total			+6

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/securitysolution-io-ts-list-types`	428	442	+14
`lists`	155	161	+6
total			+20

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`lists`	140.3KB	140.6KB	+285.0B
`securitySolution`	4.7MB	4.7MB	+280.0B
total			+565.0B

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`lists`	43	49	+6

Unknown metric groups

API count

id	before	after	diff
`@kbn/securitysolution-io-ts-list-types`	439	453	+14
`lists`	185	197	+12
total			+26

ESLint disabled in files

id	before	after	diff
`@kbn/securitysolution-io-ts-list-types`	12	14	+2

Total ESLint disabled count

id	before	after	diff
`@kbn/securitysolution-io-ts-list-types`	12	14	+2

History

💚 Build #24097 succeeded 61ff4b9
💚 Build #23721 succeeded 18e417a
💛 Build #23713 was flaky 211d36f
💔 Build #23687 failed 4a8a8aa
💚 Build #23352 succeeded b39c95a
💚 Build #23263 succeeded 6468a6c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @FrankHassanabad

kibanamachine · 2022-02-17T23:06:18Z