[Endpoint] Sample data generator CLI script #59952

marshallmain · 2020-03-11T20:06:02Z

Summary

Extends the work in #58936 and provides a configurable script that should generate everything necessary for a resolver tree (endpoint metadata document, process events with correlated entity_ids, an alert document, and related events for some of the process events).

The script takes a number of configuration options, including the --numEndpoints and --alertsPerEndpoint options which allow you to create a number of resolver trees at the same time.

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials

For maintainers

This was checked for breaking API changes and was labeled appropriately

jonathan-buttner · 2020-03-11T20:25:32Z

x-pack/plugins/endpoint/scripts/resolver_generator.ts

+
+async function main() {
+  const argv = yargs.help().options({
+    seed: {


👍 awesome

jonathan-buttner · 2020-03-11T20:30:54Z

x-pack/plugins/endpoint/common/generate_data.ts

-  public randomizeIPs() {
+  // This function will create new values for all the host fields, so documents from a different endpoint can be created
+  // This provides a convenient way to make documents from multiple endpoints that are all tied to a single seed value
+  public randomizeHostData() {


We could probably have the constructor call this function so the calls aren't duplicated.

Good idea. The compiler complained that the class variables weren't definitely assigned when I called it as it was so I pulled the fields into a new structure. It's now returned by the private function below and assigned in this function and the constructor.

elasticmachine · 2020-03-12T14:15:58Z

Pinging @elastic/endpoint-app-team (Feature:Endpoint)

elasticmachine · 2020-03-12T17:27:04Z

Pinging @elastic/endpoint-response (Team:Endpoint Response)

EricDavisX · 2020-03-13T01:16:23Z

can you add some Docs citing how to use this with a basic example please?

peluja1012 · 2020-03-13T08:02:15Z

x-pack/plugins/endpoint/scripts/resolver_generator.ts

+      );
+      const body = resolverDocs
+        .map(doc => [{ index: { _index: argv.eventIndex } }, doc])
+        .reduce((array, value) => (array.push(...value), array), []);


I think you could get rid of map() here if you did:

resolverDocs.reduce((array, doc) => (array.push({ index: { _index: argv.eventIndex } }, doc), array), []);

peluja1012 · 2020-03-13T08:18:01Z

x-pack/plugins/endpoint/tsconfig.json

@@ -0,0 +1,8 @@
+{


I would prefer if we renamed this configuration file so that it's not picked up automatically by IDEs as the configuration for the whole endpoint app code base. You could rename it to something like cli_tsconfig.json and then point ts-node to use it like:

ts-node --project cli_tsconfig.json scripts/resolver_generator.ts

peluja1012 · 2020-03-13T08:22:23Z

@marshallmain I agree with @EricDavisX's comment. Could you please add a scripts/README.md file that explains what it does, gives some usage examples, and lists any dependencies that the user needs to install before using it (I know ts-node is one of them)?

marshallmain · 2020-03-13T21:07:53Z

@elasticmachine merge upstream

peluja1012 · 2020-03-16T09:32:44Z

x-pack/plugins/endpoint/scripts/README.md

+  --numEndpoints, --ne        number of different endpoints to generate alerts
+                              for                          [number] [default: 1]
+  --alertsPerEndpoint, --ape  number of resolver trees to make for each endpoint
+                                                           [number] [default: 1]```


You have a markdown formatting issue here. The ``` appears in the readme.

…enerator-cli

marshallmain · 2020-03-16T16:07:11Z

@elasticmachine merge upstream

kibanamachine · 2020-03-16T17:54:23Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 7c49bfe

History

💔 Build #33193 failed f965e6b
💔 Build #33157 failed 9879a78
💔 Build #33121 failed 84940c7
💔 Build #32949 failed 5a840d5
💔 Build #32872 failed 1170303

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: (30 commits) [TSVB] fix text color when using custom background color (elastic#60261) Fix import to timefilter from in TSVB (elastic#60296) [NP] Get rid of usage redirectWhenMissing service (elastic#59777) [SIEM] Fix Timeline footer styling (elastic#59587) [ML] Fixes to error handling for analytics jobs and file data viz (elastic#60249) Give better stack traces for Unhandled Promise Rejection warnings (elastic#60235) resolves elastic#58905 (elastic#60120) Added variables button for text fields in Pagerduty component. (elastic#60189) adds test that action vars are rendered for alert action parms (elastic#60310) Closes 59786 by removing the update toast (elastic#60172) [EPM] Packages list tabs (elastic#60167) Added message variables button for Webhook body form field (elastic#60174) Revert "adds new test (elastic#60064)" [Maps] move MapSavedObject type out of telemetry (elastic#60127) [Reporting] Fix error handling for job handler in route (elastic#60161) [Endpoint] TEST: verify alerts page header says 'Alerts' (elastic#60206) EMT-248: implement ack resource to accept event payload to acknowledge agent actions (elastic#60218) Migrate dual validated range (elastic#59689) Embeddable triggers (elastic#58440) [Endpoint] Sample data generator CLI script (elastic#59952) ...

* start on cli * make it work * cleanup * remove failed attempt code * update package and tsconfig * remove empty file * generate resolver events from multiple endpoints * re-add child randomization * align index names with real plugin * remove duplication * better naming * add temporary mapping to sample data generator * error handling, move tsconfig * add readme * Update README.md * move mapping from common to scripts * make delete index option * remove unnecessary map call * fix import style Co-authored-by: Elastic Machine <[email protected]>

* start on cli * make it work * cleanup * remove failed attempt code * update package and tsconfig * remove empty file * generate resolver events from multiple endpoints * re-add child randomization * align index names with real plugin * remove duplication * better naming * add temporary mapping to sample data generator * error handling, move tsconfig * add readme * Update README.md * move mapping from common to scripts * make delete index option * remove unnecessary map call * fix import style Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

marshallmain requested a review from a team as a code owner March 11, 2020 20:06

marshallmain added release_note:skip Skip the PR/issue when compiling release notes v7.7.0 v8.0.0 labels Mar 11, 2020

jonathan-buttner approved these changes Mar 11, 2020

View reviewed changes

oatkiller added the Feature:Endpoint Elastic Endpoint feature label Mar 12, 2020

peluja1012 added the Team:Endpoint Response Endpoint Response Team label Mar 12, 2020

marshallmain added 11 commits March 12, 2020 13:50

start on cli

e42ee12

make it work

0bebe6a

cleanup

614b975

remove failed attempt code

a65fa34

update package and tsconfig

425ff62

remove empty file

2dd9bcd

generate resolver events from multiple endpoints

8df2995

re-add child randomization

080bd5f

align index names with real plugin

4ce88f0

remove duplication

3f61e1b

better naming

1170303

marshallmain force-pushed the generator-cli branch from 85dff7b to 1170303 Compare March 12, 2020 17:51

add temporary mapping to sample data generator

5a840d5

peluja1012 reviewed Mar 13, 2020

View reviewed changes

marshallmain and others added 3 commits March 13, 2020 11:57

error handling, move tsconfig

4dab2ac

add readme

692c362

Update README.md

84940c7

marshallmain force-pushed the generator-cli branch from c3879ce to 84940c7 Compare March 13, 2020 17:21

marshallmain added 3 commits March 13, 2020 13:25

move mapping from common to scripts

7809aa4

make delete index option

56a65ac

remove unnecessary map call

9879a78

Merge branch 'master' into generator-cli

f965e6b

peluja1012 approved these changes Mar 16, 2020

View reviewed changes

marshallmain added 2 commits March 16, 2020 12:04

fix import style

b722e9f

Merge branch 'generator-cli' of github.com:marshallmain/kibana into g…

cc8a890

…enerator-cli

Merge branch 'master' into generator-cli

7c49bfe

marshallmain merged commit 93914b6 into elastic:master Mar 16, 2020

marshallmain mentioned this pull request Mar 17, 2020

[7.x] [Endpoint] Sample data generator CLI script (#59952) #60374

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Endpoint] Sample data generator CLI script #59952

[Endpoint] Sample data generator CLI script #59952

marshallmain commented Mar 11, 2020

jonathan-buttner Mar 11, 2020

jonathan-buttner Mar 11, 2020

marshallmain Mar 11, 2020

elasticmachine commented Mar 12, 2020

elasticmachine commented Mar 12, 2020

EricDavisX commented Mar 13, 2020

peluja1012 Mar 13, 2020 •

edited

Loading

peluja1012 Mar 13, 2020

peluja1012 commented Mar 13, 2020

marshallmain commented Mar 13, 2020

peluja1012 Mar 16, 2020

marshallmain commented Mar 16, 2020

kibanamachine commented Mar 16, 2020

[Endpoint] Sample data generator CLI script #59952

[Endpoint] Sample data generator CLI script #59952

Conversation

marshallmain commented Mar 11, 2020

Summary

Checklist

For maintainers

jonathan-buttner Mar 11, 2020

Choose a reason for hiding this comment

jonathan-buttner Mar 11, 2020

Choose a reason for hiding this comment

marshallmain Mar 11, 2020

Choose a reason for hiding this comment

elasticmachine commented Mar 12, 2020

elasticmachine commented Mar 12, 2020

EricDavisX commented Mar 13, 2020

peluja1012 Mar 13, 2020 • edited Loading

Choose a reason for hiding this comment

peluja1012 Mar 13, 2020

Choose a reason for hiding this comment

peluja1012 commented Mar 13, 2020

marshallmain commented Mar 13, 2020

peluja1012 Mar 16, 2020

Choose a reason for hiding this comment

marshallmain commented Mar 16, 2020

kibanamachine commented Mar 16, 2020

💚 Build Succeeded

History

peluja1012 Mar 13, 2020 •

edited

Loading