Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SIEM] Add apm-*-transcation* to siemDefaultIndex Kibana Advanced Setting #52297

Closed
spong opened this issue Dec 5, 2019 · 6 comments · Fixed by #54876
Closed

[SIEM] Add apm-*-transcation* to siemDefaultIndex Kibana Advanced Setting #52297

spong opened this issue Dec 5, 2019 · 6 comments · Fixed by #54876

Comments

@spong
Copy link
Member

spong commented Dec 5, 2019

In effort to augment the SIEM UI with APM data we'll need to include apm-* in the siem:defaultIndex Kibana Advanced Setting.

10-DEC-2019: Editing title to be apm-*-transcation* instead of apm-*

@elasticmachine
Copy link
Contributor

Pinging @elastic/siem (Team:SIEM)

@spong
Copy link
Member Author

spong commented Dec 5, 2019

cc @elastic/apm

@sorenlouv
Copy link
Member

sorenlouv commented Dec 5, 2019

fyi: Users can configure which index apm is written to. There are config options in kibana.dev.yml for this:

apm_oss.errorIndices: apm-*-error*
apm_oss.transactionIndices: apm-*-transaction*
apm_oss.spanIndices: apm-*-span*
apm_oss.metricsIndices: apm-*-metric*

We've recently introduced the possibility of overwriting these settings via the UI and thus stored in saved objects - which could make this a little more complicated for you. LMK if you need further details.

@spong
Copy link
Member Author

spong commented Dec 5, 2019

Thanks for the details @sqren -- the question of additional apm indices came up when we were discussing this, so I'll touch base with the team and detail feedback here before moving forward.

@tsg
Copy link
Contributor

tsg commented Dec 6, 2019

The question is if we add apm-* or just apm-*-transaction* as the default in SIEM. I'm leaning towards apm-*-transcation*, since I think that contains all the data that we might need for 7.6. In future versions we might query the spans as well, for example, but we can always add it later and start a bit more conservative.

@spong
Copy link
Member Author

spong commented Dec 6, 2019

Sounds good! Will move forward with adding apm-*-transcation* to siem:defaultIndex.

In effort to ensure the SIEM Map displays APM data without any additional configuration (since it requires a matching named Kibana Index Pattern), I'm going to make a slight enhancement to how the map layers are generated. When the map is checking for matching Kibana Index Patterns, it'll match apm-*-transcation* -> apm-* (if exists), and then create a corresponding map layer for client/server geo fields (instead of the usual source/dest) since these are the fields Transactions use.

The maps config generation is fairly well abstracted, so this'll be a pretty simple change and something we can expand upon further if/when we decide to add any other indices.

@spong spong changed the title [SIEM] Add apm-* to siemDefaultIndex Kibana Advanced Setting [SIEM] Add apm-*-transcation* to siemDefaultIndex Kibana Advanced Setting Dec 10, 2019
spong added a commit that referenced this issue Jan 17, 2020
## Summary

Resolves #52297, #52565

To improve the display of APM data within SIEM (specifically the `HTTP Table` and `Network Map`), this PR adds `apm-*-transcation*` to `siem:defaultIndex`, and additional support for showing `client`/`server` layers on the `Network Map` when a matching `apm-*` index pattern is present.

The map now supports pattern matching when checking for available Kibana Index Patterns, and so matches `apm-*-transcation*` -> `apm-*` (if exists). Additionally, the map config was updated to generate layers for client/server geo fields (instead of the usual source/dest) since these are the fields Transactions use.

![image](https://user-images.githubusercontent.com/2946766/72573225-2a038880-3882-11ea-9590-a545d726dbf9.png)

<img width="1214" alt="Screen Shot 2020-01-14 at 18 22 11" src="https://user-images.githubusercontent.com/2946766/72407120-bcd5e300-371b-11ea-90cc-a0714320a59c.png">


### Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

- [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~
- [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials
  - Will work with @benskelker on updating the maps docs
- [X] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
- [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~

### For maintainers

- [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
- [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
spong added a commit to spong/kibana that referenced this issue Jan 17, 2020
## Summary

Resolves elastic#52297, elastic#52565

To improve the display of APM data within SIEM (specifically the `HTTP Table` and `Network Map`), this PR adds `apm-*-transcation*` to `siem:defaultIndex`, and additional support for showing `client`/`server` layers on the `Network Map` when a matching `apm-*` index pattern is present.

The map now supports pattern matching when checking for available Kibana Index Patterns, and so matches `apm-*-transcation*` -> `apm-*` (if exists). Additionally, the map config was updated to generate layers for client/server geo fields (instead of the usual source/dest) since these are the fields Transactions use.

![image](https://user-images.githubusercontent.com/2946766/72573225-2a038880-3882-11ea-9590-a545d726dbf9.png)

<img width="1214" alt="Screen Shot 2020-01-14 at 18 22 11" src="https://user-images.githubusercontent.com/2946766/72407120-bcd5e300-371b-11ea-90cc-a0714320a59c.png">


### Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

- [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~
- [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials
  - Will work with @benskelker on updating the maps docs
- [X] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
- [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~

### For maintainers

- [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
- [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
spong added a commit to spong/kibana that referenced this issue Jan 17, 2020
## Summary

Resolves elastic#52297, elastic#52565

To improve the display of APM data within SIEM (specifically the `HTTP Table` and `Network Map`), this PR adds `apm-*-transcation*` to `siem:defaultIndex`, and additional support for showing `client`/`server` layers on the `Network Map` when a matching `apm-*` index pattern is present.

The map now supports pattern matching when checking for available Kibana Index Patterns, and so matches `apm-*-transcation*` -> `apm-*` (if exists). Additionally, the map config was updated to generate layers for client/server geo fields (instead of the usual source/dest) since these are the fields Transactions use.

![image](https://user-images.githubusercontent.com/2946766/72573225-2a038880-3882-11ea-9590-a545d726dbf9.png)

<img width="1214" alt="Screen Shot 2020-01-14 at 18 22 11" src="https://user-images.githubusercontent.com/2946766/72407120-bcd5e300-371b-11ea-90cc-a0714320a59c.png">


### Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

- [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~
- [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials
  - Will work with @benskelker on updating the maps docs
- [X] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
- [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~

### For maintainers

- [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
- [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
spong added a commit that referenced this issue Jan 17, 2020
## Summary

Resolves #52297, #52565

To improve the display of APM data within SIEM (specifically the `HTTP Table` and `Network Map`), this PR adds `apm-*-transcation*` to `siem:defaultIndex`, and additional support for showing `client`/`server` layers on the `Network Map` when a matching `apm-*` index pattern is present.

The map now supports pattern matching when checking for available Kibana Index Patterns, and so matches `apm-*-transcation*` -> `apm-*` (if exists). Additionally, the map config was updated to generate layers for client/server geo fields (instead of the usual source/dest) since these are the fields Transactions use.

![image](https://user-images.githubusercontent.com/2946766/72573225-2a038880-3882-11ea-9590-a545d726dbf9.png)

<img width="1214" alt="Screen Shot 2020-01-14 at 18 22 11" src="https://user-images.githubusercontent.com/2946766/72407120-bcd5e300-371b-11ea-90cc-a0714320a59c.png">


### Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

- [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~
- [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials
  - Will work with @benskelker on updating the maps docs
- [X] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
- [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~

### For maintainers

- [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
- [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
spong added a commit that referenced this issue Jan 17, 2020
## Summary

Resolves #52297, #52565

To improve the display of APM data within SIEM (specifically the `HTTP Table` and `Network Map`), this PR adds `apm-*-transcation*` to `siem:defaultIndex`, and additional support for showing `client`/`server` layers on the `Network Map` when a matching `apm-*` index pattern is present.

The map now supports pattern matching when checking for available Kibana Index Patterns, and so matches `apm-*-transcation*` -> `apm-*` (if exists). Additionally, the map config was updated to generate layers for client/server geo fields (instead of the usual source/dest) since these are the fields Transactions use.

![image](https://user-images.githubusercontent.com/2946766/72573225-2a038880-3882-11ea-9590-a545d726dbf9.png)

<img width="1214" alt="Screen Shot 2020-01-14 at 18 22 11" src="https://user-images.githubusercontent.com/2946766/72407120-bcd5e300-371b-11ea-90cc-a0714320a59c.png">


### Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

- [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~
- [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials
  - Will work with @benskelker on updating the maps docs
- [X] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
- [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~

### For maintainers

- [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
- [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
jkelastic pushed a commit to jkelastic/kibana that referenced this issue Jan 17, 2020
## Summary

Resolves elastic#52297, elastic#52565

To improve the display of APM data within SIEM (specifically the `HTTP Table` and `Network Map`), this PR adds `apm-*-transcation*` to `siem:defaultIndex`, and additional support for showing `client`/`server` layers on the `Network Map` when a matching `apm-*` index pattern is present.

The map now supports pattern matching when checking for available Kibana Index Patterns, and so matches `apm-*-transcation*` -> `apm-*` (if exists). Additionally, the map config was updated to generate layers for client/server geo fields (instead of the usual source/dest) since these are the fields Transactions use.

![image](https://user-images.githubusercontent.com/2946766/72573225-2a038880-3882-11ea-9590-a545d726dbf9.png)

<img width="1214" alt="Screen Shot 2020-01-14 at 18 22 11" src="https://user-images.githubusercontent.com/2946766/72407120-bcd5e300-371b-11ea-90cc-a0714320a59c.png">


### Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

- [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~
- [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials
  - Will work with @benskelker on updating the maps docs
- [X] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
- [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~

### For maintainers

- [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
- [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants