New events resolver #82170

IgorGuz2000 · 2020-10-30T17:31:02Z

Summary

Added Alert Test for Resolver and fix for the APM enabled Run fail

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

…ents_resolver

elasticmachine · 2020-10-30T17:31:05Z

Pinging @elastic/endpoint-app-team (Feature:Resolver)

elasticmachine · 2020-10-30T17:31:05Z

Pinging @elastic/endpoint-data-visibility-team (Team:Endpoint Data Visibility)

achuguy · 2020-11-05T16:18:32Z

x-pack/test/security_solution_endpoint/page_objects/hosts_page.ts

@@ -121,6 +121,8 @@ export function SecurityHostsPageProvider({ getService, getPageObjects }: FtrPro
      for (let i = 0; i < NodeSubmenuItems.length; i++) {
        await (await testSubjects.findAll('resolver:map:node-submenu-item'))[i].click();
        const Events = await testSubjects.findAll('resolver:map:node-submenu-item');
+        // this sleep is for the AMP enabled run
+        await pageObjects.common.sleep(300);


How much does this sleep add to the duration of every test? Is there a way you can convert this to a wait?

I try to do the wait and different ways nothing works.

it just adds 2 sec per test only

I support a Sleep if we need the tests passing and to unblock the build. Always good to question it and follow up more if we can improve it

marshallmain

This PR adds more than 8MB of data to the kibana repo that every kibana dev will now be pulling and storing locally. Can the es_archive size be reduced?

EricDavisX · 2020-11-07T17:19:28Z

x-pack/test/security_solution_endpoint/page_objects/hosts_page.ts

@@ -121,6 +121,8 @@ export function SecurityHostsPageProvider({ getService, getPageObjects }: FtrPro
      for (let i = 0; i < NodeSubmenuItems.length; i++) {
        await (await testSubjects.findAll('resolver:map:node-submenu-item'))[i].click();
        const Events = await testSubjects.findAll('resolver:map:node-submenu-item');
+        // this sleep is for the AMP enabled run


Typo here, can you change it to APM in a subsequent PR please

EricDavisX · 2020-11-07T17:28:48Z

That's wise for us to take the time to review all of the ES-Archive datasets we have and see if we can safely combine them to reduce overall size. but I submit we can do that in a separate pr. Progress over perfection.

EricDavisX

ok to merge, needs some follow ups in subsequent pr

IgorGuz2000 · 2020-11-08T03:32:05Z

@elasticmachine merge upstream

marshallmain · 2020-11-08T04:27:06Z

@EricDavisX I'm primarily concerned that this new archive being added into the source tree for a single test is enormous. Looking at the uncompressed data, I see over 95000 documents and 10,000,000 lines of text.

charlie-pichette · 2020-11-09T14:24:33Z

@IgorGuz2000 95,000 documents in the archive file seems like a lot for a test checking a single node. Since all the documents are for the same host we should remove the extraneous documents and keep only the ones we need to validate the test. This would also improve the test run time.

IgorGuz2000 · 2020-11-09T16:37:17Z

This test is testing Alert event and all the data is the related events of this alert. I use Jeff script to pull the data only of one alert and this is what I get.

achuguy · 2020-11-09T17:59:01Z

Can you check what events show up in kibana and what indices you are storing? you should have ~800 events based on

const expectedAlertData = [
          '1 library',
          '157 file',
          '520 registry',
          '3 file',
          '5 library',
          '5 library',
        ];

But marshall mentioned you are storing ~95k docs

IgorGuz2000 · 2020-11-09T18:11:59Z

I am not counting all of them

achuguy · 2020-11-09T18:17:06Z

The process tree you're examining has 11k documents?

…ents_resolver

Data file is resized

kibanamachine · 2020-11-09T20:56:54Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 5918b80

Metrics [docs]

✅ unchanged

History

💚 Build #86604 succeeded 6810449
💚 Build #86004 succeeded a6ea8d3
💔 Build #85966 failed 91a640d
💔 Build #85924 failed 57f56c5
💔 Build #85915 failed 17e3cfe

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: (39 commits) Fix ilm navigation (elastic#81664) [Lens] Distinct icons for XY and pie chart value labels toolbar (elastic#82927) [data.search.aggs] Throw an error when trying to create an agg type that doesn't exist. (elastic#81509) Index patterns api - load field list on server (elastic#82629) New events resolver (elastic#82170) [App Search] Misc naming tech debt (elastic#82770) load empty_kibana in test to have clean starting point (elastic#82772) Remove data <--> expressions circular dependencies. (elastic#82685) Update 8.0 breaking change template to gather information on how to programmatically detect it. (elastic#82905) Add alerting as codeowners to related documentation folder (elastic#82777) Add captions to user and space grid pages (elastic#82713) add alternate path for x-pack/Cloud test for Lens (elastic#82634) Uses asCurrentUser in getClusterUuid (elastic#82908) [Alerting][Connectors] Add new executor subaction to get 3rd party case fields (elastic#82519) Fix test import objects (elastic#82767) [ML] Add option for anomaly charts for metric detector should plot min, mean or max as appropriate (elastic#81662) Update alert type selection layout to rows instead of grid (elastic#73665) Prevent Kerberos and PKI providers from initiating a new session for unauthenticated XHR/API requests. (elastic#82817) Update grunt and related packages (elastic#79327) Allow the repository to search across all namespaces (elastic#82863) ...

kibanamachine · 2020-11-11T22:44:39Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 82170 or prevent reminders by adding the backport:skip label.

kibanamachine · 2020-11-12T22:54:38Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 82170 or prevent reminders by adding the backport:skip label.

kibanamachine · 2020-11-13T22:54:38Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 82170 or prevent reminders by adding the backport:skip label.

* Added Test for event.library * renamed data directry and gzip data file * rename expectedData file * Changes per Charlie request * Changes for the enable_APM-ci branch * Update resolver.ts * Added comment per Charlie request * Update resolver.ts * Added Alert Test for Resolver and fix for the APM enabled Run fail * Added Alert Test for Resolver and fix for the APM enabled Run fail * removed commented out code * Fixing CI fail * Fixing CI fail * Removed Alert Resolver test * aAdding Alert test back * Adding Alert test back * Adding Alert test back * Adding info log for debuging * Adding info log for debuging * Adding info log for debuging * Adding info log for debuging * Adding info log for debuging * Adding info log for debuging * adding one more verification for Data * stripedd Data file Co-authored-by: Kibana Machine <[email protected]>

kibanamachine · 2020-11-16T23:04:38Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

* New events resolver (#82170) * Added Test for event.library * renamed data directry and gzip data file * rename expectedData file * Changes per Charlie request * Changes for the enable_APM-ci branch * Update resolver.ts * Added comment per Charlie request * Update resolver.ts * Added Alert Test for Resolver and fix for the APM enabled Run fail * Added Alert Test for Resolver and fix for the APM enabled Run fail * removed commented out code * Fixing CI fail * Fixing CI fail * Removed Alert Resolver test * aAdding Alert test back * Adding Alert test back * Adding Alert test back * Adding info log for debuging * Adding info log for debuging * Adding info log for debuging * Adding info log for debuging * Adding info log for debuging * Adding info log for debuging * adding one more verification for Data * stripedd Data file Co-authored-by: Kibana Machine <[email protected]> * Update hosts_page.ts * Update resolver.ts * Update resolver.ts * Update hosts_page.ts * Update resolver.ts Co-authored-by: Kibana Machine <[email protected]>

IgorGuz2000 and others added 18 commits October 15, 2020 13:00

Added Test for event.library

6407b75

renamed data directry and gzip data file

981618a

rename expectedData file

63eb262

Merge branch 'master' into New_events_resolver

6ed753a

Merge remote-tracking branch 'upstream/master' into New_events_resolver

00cc83b

Changes per Charlie request

4382daf

Merge remote-tracking branch 'origin/New_events_resolver' into New_ev…

332646d

…ents_resolver

Merge remote-tracking branch 'upstream/master' into New_events_resolver

9189098

Changes for the enable_APM-ci branch

ab8604b

Merge branch 'master' into New_events_resolver

1492602

Update resolver.ts

920b5fd

Added comment per Charlie request

b752c15

Update resolver.ts

4cbb232

Merge remote-tracking branch 'origin/New_events_resolver' into New_ev…

d7b25a1

…ents_resolver

Merge remote-tracking branch 'upstream/master' into New_events_resolver

383dec9

Merge remote-tracking branch 'upstream/master' into New_events_resolver

e514fce

Added Alert Test for Resolver and fix for the APM enabled Run fail

4a5982d

Added Alert Test for Resolver and fix for the APM enabled Run fail

dcdbe46

IgorGuz2000 added release_note:fix Team:Endpoint Data Visibility Team managing the endpoint resolver Feature:Resolver Security Solution Resolver feature v7.10.0 labels Oct 30, 2020

IgorGuz2000 requested review from EricDavisX, achuguy and charlie-pichette October 30, 2020 17:31

IgorGuz2000 requested review from a team as code owners October 30, 2020 17:31

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Oct 30, 2020

achuguy reviewed Nov 5, 2020

View reviewed changes

MindyRS approved these changes Nov 6, 2020

View reviewed changes

marshallmain previously requested changes Nov 6, 2020

View reviewed changes

EricDavisX reviewed Nov 7, 2020

View reviewed changes

EricDavisX approved these changes Nov 7, 2020

View reviewed changes

Merge branch 'master' into New_events_resolver

6810449

IgorGuz2000 added 2 commits November 9, 2020 11:10

stripedd Data file

e594ffd

Merge remote-tracking branch 'origin/New_events_resolver' into New_ev…

5918b80

…ents_resolver

IgorGuz2000 requested a review from marshallmain November 9, 2020 19:11

IgorGuz2000 merged commit bf75831 into elastic:master Nov 9, 2020

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 11, 2020

IgorGuz2000 mentioned this pull request Nov 16, 2020

[7.x] New events resolver (#82170) #83443

Merged

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New events resolver #82170

New events resolver #82170

IgorGuz2000 commented Oct 30, 2020 •

edited

Loading

elasticmachine commented Oct 30, 2020

elasticmachine commented Oct 30, 2020

achuguy Nov 5, 2020

IgorGuz2000 Nov 5, 2020

IgorGuz2000 Nov 5, 2020

EricDavisX Nov 7, 2020

marshallmain left a comment

EricDavisX Nov 7, 2020

EricDavisX commented Nov 7, 2020

EricDavisX left a comment

IgorGuz2000 commented Nov 8, 2020

marshallmain commented Nov 8, 2020 •

edited

Loading

charlie-pichette commented Nov 9, 2020 •

edited

Loading

IgorGuz2000 commented Nov 9, 2020

achuguy commented Nov 9, 2020

IgorGuz2000 commented Nov 9, 2020

achuguy commented Nov 9, 2020

kibanamachine commented Nov 9, 2020

kibanamachine commented Nov 11, 2020

kibanamachine commented Nov 12, 2020

kibanamachine commented Nov 13, 2020

kibanamachine commented Nov 16, 2020

New events resolver #82170

New events resolver #82170

Conversation

IgorGuz2000 commented Oct 30, 2020 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Oct 30, 2020

elasticmachine commented Oct 30, 2020

achuguy Nov 5, 2020

Choose a reason for hiding this comment

IgorGuz2000 Nov 5, 2020

Choose a reason for hiding this comment

IgorGuz2000 Nov 5, 2020

Choose a reason for hiding this comment

EricDavisX Nov 7, 2020

Choose a reason for hiding this comment

marshallmain left a comment

Choose a reason for hiding this comment

EricDavisX Nov 7, 2020

Choose a reason for hiding this comment

EricDavisX commented Nov 7, 2020

EricDavisX left a comment

Choose a reason for hiding this comment

IgorGuz2000 commented Nov 8, 2020

marshallmain commented Nov 8, 2020 • edited Loading

charlie-pichette commented Nov 9, 2020 • edited Loading

IgorGuz2000 commented Nov 9, 2020

achuguy commented Nov 9, 2020

IgorGuz2000 commented Nov 9, 2020

achuguy commented Nov 9, 2020

kibanamachine commented Nov 9, 2020

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Nov 11, 2020

kibanamachine commented Nov 12, 2020

kibanamachine commented Nov 13, 2020

kibanamachine commented Nov 16, 2020

IgorGuz2000 commented Oct 30, 2020 •

edited

Loading

marshallmain commented Nov 8, 2020 •

edited

Loading

charlie-pichette commented Nov 9, 2020 •

edited

Loading