[SIEM] Adds support for apm-* to the network map #54876

spong · 2020-01-15T05:23:30Z

Summary

To improve the display of APM data within SIEM (specifically the HTTP Table and Network Map), this PR adds apm-*-transaction* to siem:defaultIndex, and additional support for showing client/server layers on the Network Map when a matching apm-* index pattern is present.

The map now supports pattern matching when checking for available Kibana Index Patterns, and so matches apm-*-transaction* -> apm-* (if exists). Additionally, the map config was updated to generate layers for client/server geo fields (instead of the usual source/dest) since these are the fields Transactions use.

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

~~This was checked for cross-browser compatibility, including a check against IE11~~
Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
- Will work with @benskelker on updating the maps docs
Unit or functional tests were updated or added to match the most common scenarios
~~This was checked for keyboard-only and screenreader accessibility~~

For maintainers

~~This was checked for breaking API changes and was labeled appropriately~~
~~This includes a feature addition or change that requires a release note and was labeled appropriately~~

elasticmachine · 2020-01-15T05:23:33Z

Pinging @elastic/siem (Team:SIEM)

tsg · 2020-01-15T11:03:23Z

@elasticmachine merge upstream

elasticmachine · 2020-01-15T11:03:25Z

user doesn't have permission to update head repository

patrykkopycinski · 2020-01-15T11:05:29Z

@elasticmachine merge upstream

elasticmachine · 2020-01-15T11:05:31Z

user doesn't have permission to update head repository

mistic · 2020-01-15T13:41:21Z

@elasticmachine merge upstream

elasticmachine · 2020-01-15T13:41:23Z

user doesn't have permission to update head repository

andrew-goldstein · 2020-01-16T20:09:15Z

x-pack/legacy/plugins/siem/default_index_pattern.ts

@@ -6,6 +6,7 @@

 /** The comma-delimited list of Elasticsearch indices from which the SIEM app collects events */
 export const defaultIndexPattern = [
+  'apm-*-transaction*',


👍 this is the first 'non security' index pattern to be added to the defaults

andrew-goldstein · 2020-01-16T20:23:21Z

x-pack/legacy/plugins/siem/public/components/embeddables/map_config.ts

+      label: i18n.DESTINATION_LAYER,
+    },
+  },
+  'apm-*': {


the APM Kibana plugin creates this Kibana index pattern via this UI action:

...and also via the apm-server setup command's --index-management command line argument, documented here: https://www.elastic.co/guide/en/apm/server/current/command-line-options.html#setup-command

x-pack/legacy/plugins/siem/public/components/embeddables/map_tool_tip/line_tool_tip_content.tsx

x-pack/legacy/plugins/siem/public/components/embeddables/types.ts

andrew-goldstein

🗺 This is working mapsolutely perfectly after adding an apm-* index pattern to my local test environment.
LGTM 🚀

@benskelker

## Summary Resolves elastic#52297, elastic#52565 To improve the display of APM data within SIEM (specifically the `HTTP Table` and `Network Map`), this PR adds `apm-*-transcation*` to `siem:defaultIndex`, and additional support for showing `client`/`server` layers on the `Network Map` when a matching `apm-*` index pattern is present. The map now supports pattern matching when checking for available Kibana Index Patterns, and so matches `apm-*-transcation*` -> `apm-*` (if exists). Additionally, the map config was updated to generate layers for client/server geo fields (instead of the usual source/dest) since these are the fields Transactions use. ![image](https://user-images.githubusercontent.com/2946766/72573225-2a038880-3882-11ea-9590-a545d726dbf9.png) <img width="1214" alt="Screen Shot 2020-01-14 at 18 22 11" src="https://user-images.githubusercontent.com/2946766/72407120-bcd5e300-371b-11ea-90cc-a0714320a59c.png"> ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. - [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~ - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md) - [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials - Will work with @benskelker on updating the maps docs - [X] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios - [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~ ### For maintainers - [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~ - [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~

@benskelker

## Summary Resolves elastic#52297, elastic#52565 To improve the display of APM data within SIEM (specifically the `HTTP Table` and `Network Map`), this PR adds `apm-*-transcation*` to `siem:defaultIndex`, and additional support for showing `client`/`server` layers on the `Network Map` when a matching `apm-*` index pattern is present. The map now supports pattern matching when checking for available Kibana Index Patterns, and so matches `apm-*-transcation*` -> `apm-*` (if exists). Additionally, the map config was updated to generate layers for client/server geo fields (instead of the usual source/dest) since these are the fields Transactions use. ![image](https://user-images.githubusercontent.com/2946766/72573225-2a038880-3882-11ea-9590-a545d726dbf9.png) <img width="1214" alt="Screen Shot 2020-01-14 at 18 22 11" src="https://user-images.githubusercontent.com/2946766/72407120-bcd5e300-371b-11ea-90cc-a0714320a59c.png"> ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. - [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~ - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md) - [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials - Will work with @benskelker on updating the maps docs - [X] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios - [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~ ### For maintainers - [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~ - [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~

kibanamachine · 2020-01-17T01:34:50Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 3b7d15a

History

💚 Build #20971 succeeded 3b7d15a
💚 Build #20556 succeeded 57da3fd
💔 Build #20464 failed 7bc4305

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@benskelker

## Summary Resolves #52297, #52565 To improve the display of APM data within SIEM (specifically the `HTTP Table` and `Network Map`), this PR adds `apm-*-transcation*` to `siem:defaultIndex`, and additional support for showing `client`/`server` layers on the `Network Map` when a matching `apm-*` index pattern is present. The map now supports pattern matching when checking for available Kibana Index Patterns, and so matches `apm-*-transcation*` -> `apm-*` (if exists). Additionally, the map config was updated to generate layers for client/server geo fields (instead of the usual source/dest) since these are the fields Transactions use. ![image](https://user-images.githubusercontent.com/2946766/72573225-2a038880-3882-11ea-9590-a545d726dbf9.png) <img width="1214" alt="Screen Shot 2020-01-14 at 18 22 11" src="https://user-images.githubusercontent.com/2946766/72407120-bcd5e300-371b-11ea-90cc-a0714320a59c.png"> ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. - [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~ - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md) - [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials - Will work with @benskelker on updating the maps docs - [X] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios - [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~ ### For maintainers - [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~ - [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~

@benskelker

## Summary Resolves #52297, #52565 To improve the display of APM data within SIEM (specifically the `HTTP Table` and `Network Map`), this PR adds `apm-*-transcation*` to `siem:defaultIndex`, and additional support for showing `client`/`server` layers on the `Network Map` when a matching `apm-*` index pattern is present. The map now supports pattern matching when checking for available Kibana Index Patterns, and so matches `apm-*-transcation*` -> `apm-*` (if exists). Additionally, the map config was updated to generate layers for client/server geo fields (instead of the usual source/dest) since these are the fields Transactions use. ![image](https://user-images.githubusercontent.com/2946766/72573225-2a038880-3882-11ea-9590-a545d726dbf9.png) <img width="1214" alt="Screen Shot 2020-01-14 at 18 22 11" src="https://user-images.githubusercontent.com/2946766/72407120-bcd5e300-371b-11ea-90cc-a0714320a59c.png"> ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. - [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~ - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md) - [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials - Will work with @benskelker on updating the maps docs - [X] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios - [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~ ### For maintainers - [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~ - [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~

…t-of-legacy * 'master' of github.com:elastic/kibana: (142 commits) [Vis] Move Timelion Vis to vis_type_timelion (elastic#52069) Deprecate `chrome.navlinks.update` and add documentation (elastic#54893) [ML] Single Metric Viewer: Fix time bounds with custom strings. (elastic#55045) [Vis: Default editor] EUIficate and Reactify the sidebar (elastic#49864) [Mappings editor] Fix cannot set boolean value for "null_value" param (elastic#55015) [SIEM] Adds support for apm-* to the network map (elastic#54876) [Reporting] Define shims of legacy dependencies (elastic#54082) Resolver is overflow: hidden to prevent obscured elements from showing up (elastic#55076) Upgraded EUI to 18.2.1 (elastic#55090) [Maps] Support styles on agg fields with _of_ in name (elastic#54965) Remove xpack_main requirement, it's no longer in use (elastic#55060) Fix Snapshots Policies Alignment Issue in IE11 (elastic#54866) first rule cuts (elastic#54990) [DOCS] Adds geocentroid note to coordinate map (elastic#54389) [Canvas] Fixes the Copy Post Url link (elastic#54831) Fixes bugs with full screen filters (elastic#54792) [ML] Fix decoding in the URL state (elastic#54915) Remove redundant `x-pack/typings`. (elastic#55042) [SIEM][Detection Engine] Adds critical missing status route to prepackaged rules Generate legacy vars when rendering all applications (elastic#54768) ... # Conflicts: # x-pack/plugins/translations/translations/ja-JP.json # x-pack/plugins/translations/translations/zh-CN.json

@benskelker

## Summary Resolves elastic#52297, elastic#52565 To improve the display of APM data within SIEM (specifically the `HTTP Table` and `Network Map`), this PR adds `apm-*-transcation*` to `siem:defaultIndex`, and additional support for showing `client`/`server` layers on the `Network Map` when a matching `apm-*` index pattern is present. The map now supports pattern matching when checking for available Kibana Index Patterns, and so matches `apm-*-transcation*` -> `apm-*` (if exists). Additionally, the map config was updated to generate layers for client/server geo fields (instead of the usual source/dest) since these are the fields Transactions use. ![image](https://user-images.githubusercontent.com/2946766/72573225-2a038880-3882-11ea-9590-a545d726dbf9.png) <img width="1214" alt="Screen Shot 2020-01-14 at 18 22 11" src="https://user-images.githubusercontent.com/2946766/72407120-bcd5e300-371b-11ea-90cc-a0714320a59c.png"> ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. - [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~ - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md) - [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials - Will work with @benskelker on updating the maps docs - [X] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios - [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~ ### For maintainers - [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~ - [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~

* upstream/master: (83 commits) [Reporting] Fix map tiles not loading by using Chrome's Remote Protocol (elastic#55137) [Data Plugin] combine autocomplete provider and suggestions provider (elastic#54451) resolves elastic#53038 - remove references to specific license levels (elastic#53858) highlighting rules should still know about url parts when in sql state (elastic#55200) [Metric] convert mocha tests to jest (elastic#54054) [skip-ci] Update vector styling docs for 7.6 UI changes and new features (elastic#55087) Fix enable API to schedule task after alert is updated (elastic#55095) chore(NA): add 7.6 branch to the list of backport branches (elastic#54998) Convert tests to jest in vis_type_timeseries/public & common folders (elastic#55023) [ML] Accessibility fix for structural markup on table rows (elastic#55075) [Mappings editor] include/exclude fields only support custom options (elastic#54949) [Vis] Move Timelion Vis to vis_type_timelion (elastic#52069) Deprecate `chrome.navlinks.update` and add documentation (elastic#54893) [ML] Single Metric Viewer: Fix time bounds with custom strings. (elastic#55045) [Vis: Default editor] EUIficate and Reactify the sidebar (elastic#49864) [Mappings editor] Fix cannot set boolean value for "null_value" param (elastic#55015) [SIEM] Adds support for apm-* to the network map (elastic#54876) [Reporting] Define shims of legacy dependencies (elastic#54082) Resolver is overflow: hidden to prevent obscured elements from showing up (elastic#55076) Upgraded EUI to 18.2.1 (elastic#55090) ...

spong added 4 commits January 14, 2020 20:07

Adds support for apm-* to the network map

950572c

Merge branch 'master' of github.com:elastic/kibana into apm-map-config

2c9aa16

Updating new mocks with changes from master

d4c0c0b

Merge branch 'master' of github.com:elastic/kibana into apm-map-config

7bc4305

spong added release_note:enhancement Team:SIEM v8.0.0 labels Jan 15, 2020

spong self-assigned this Jan 15, 2020

elastic deleted a comment from elasticmachine Jan 15, 2020

spong added 3 commits January 15, 2020 07:04

Merge branch 'master' of github.com:elastic/kibana into apm-map-config

57da3fd

Merge branch 'master' of github.com:elastic/kibana into apm-map-config

7b12327

Merge branch 'master' of github.com:elastic/kibana into apm-map-config

525f7eb

andrew-goldstein reviewed Jan 16, 2020

View reviewed changes

x-pack/legacy/plugins/siem/public/components/embeddables/map_tool_tip/line_tool_tip_content.tsx Outdated Show resolved Hide resolved

andrew-goldstein reviewed Jan 16, 2020

View reviewed changes

x-pack/legacy/plugins/siem/public/components/embeddables/types.ts Outdated Show resolved Hide resolved

spong added 2 commits January 16, 2020 15:21

Comments from review

5c762dd

Merge branch 'master' of github.com:elastic/kibana into apm-map-config

3b7d15a

andrew-goldstein approved these changes Jan 17, 2020

View reviewed changes

spong merged commit 24df3a9 into elastic:master Jan 17, 2020

spong deleted the apm-map-config branch January 17, 2020 00:13

spong added v7.6.0 v7.7.0 labels Jan 17, 2020

spong mentioned this pull request Jan 17, 2020

[7.x] [SIEM] Adds support for apm-* to the network map (#54876) #55142

Merged

spong mentioned this pull request Jan 17, 2020

[7.6] [SIEM] Adds support for apm-* to the network map (#54876) #55143

Merged

spong mentioned this pull request Jan 17, 2020

[SIEM] [Map] Support pattern matching when validating which Kibana Index Patterns to create layers for #52565

Closed

spong mentioned this pull request Jul 7, 2020

[Security] [Maps] Network Map fails to create layers for Kibana Index Patterns containing multiple indices or exclusions #70914

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM] Adds support for apm-* to the network map #54876

[SIEM] Adds support for apm-* to the network map #54876

spong commented Jan 15, 2020 •

edited

Loading

elasticmachine commented Jan 15, 2020

tsg commented Jan 15, 2020

elasticmachine commented Jan 15, 2020

patrykkopycinski commented Jan 15, 2020

elasticmachine commented Jan 15, 2020

mistic commented Jan 15, 2020

elasticmachine commented Jan 15, 2020

andrew-goldstein Jan 16, 2020

andrew-goldstein Jan 16, 2020 •

edited

Loading

andrew-goldstein left a comment

kibanamachine commented Jan 17, 2020

[SIEM] Adds support for apm-* to the network map #54876

[SIEM] Adds support for apm-* to the network map #54876

Conversation

spong commented Jan 15, 2020 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Jan 15, 2020

tsg commented Jan 15, 2020

elasticmachine commented Jan 15, 2020

patrykkopycinski commented Jan 15, 2020

elasticmachine commented Jan 15, 2020

mistic commented Jan 15, 2020

elasticmachine commented Jan 15, 2020

andrew-goldstein Jan 16, 2020

Choose a reason for hiding this comment

andrew-goldstein Jan 16, 2020 • edited Loading

Choose a reason for hiding this comment

andrew-goldstein left a comment

Choose a reason for hiding this comment

kibanamachine commented Jan 17, 2020

💚 Build Succeeded

History

spong commented Jan 15, 2020 •

edited

Loading

andrew-goldstein Jan 16, 2020 •

edited

Loading