Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security solution] Sourcerer: Kibana index pattern selector for security views #74706

Merged
merged 13 commits into from
Aug 14, 2020
Merged

[Security solution] Sourcerer: Kibana index pattern selector for security views #74706

merged 13 commits into from
Aug 14, 2020

Conversation

stephmilovic
Copy link
Contributor

@stephmilovic stephmilovic commented Aug 10, 2020 •
edited
Loading

Summary

This PR introduces a Kibana index pattern selector to the security solution... I call it the Sourcerer. The Sourcerer finds all available Kibana index patterns and compares them against the recommended default index patterns for SIEM, and selects the available recommended patterns to pass to the source query. The component shows the available selections, unavailable selections, and allows the user to select which index patterns will go into the SIEM. I've connected one component to this data, overview_host. Hard to describe, check gifs below.

  • Available index patterns are all displayed, recommended index patterns are checked. When a pattern gets unchecked, we remove it from the query

step1

  • Recommended index patterns that are not configured as Kibana index patterns are shown as disabled with a note
    step2

  • A button directs the user to create new patterns
    step3

To test you'll need to turn on the feature flag. Set SOURCERER_FEATURE_FLAG_ON to true.

image
Sourcerer

Checklist

@stephmilovic stephmilovic added Team:SIEM v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.10.0 labels Aug 10, 2020
@stephmilovic stephmilovic requested a review from XavierM August 10, 2020 20:36
@stephmilovic stephmilovic self-assigned this Aug 10, 2020
@stephmilovic stephmilovic marked this pull request as ready for review August 10, 2020 20:39
@stephmilovic stephmilovic requested review from a team as code owners August 10, 2020 20:39
@elasticmachine
Copy link
Contributor

Pinging @elastic/siem (Team:SIEM)

@stephmilovic stephmilovic changed the title [Security solution] Sourcerer - Kibana index pattern selector for security views [Security solution] Sourcerer: Kibana index pattern selector for security views Aug 10, 2020
@angorayc
Copy link
Contributor

@marrasherrier @stephmilovic I played around with it and here is my thought about the experience:
https://docs.google.com/document/d/1-FmYsL2ZaitGoH5oh24ssqkjihsbTWB44rFsjDGl3xg/edit?usp=sharing

stephmilovic
stephmilovic commented Aug 12, 2020
View reviewed changes
x-pack/plugins/security_solution/public/overview/components/overview_host/index.tsx
@@ -107,7 +108,10 @@ const OverviewHostComponent: React.FC<OverviewHostProps> = ({
/>
}
>
{hostPageButton}
<>
<Sourcerer />
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be put anywhere in the app, putting it here for demo

stephmilovic
stephmilovic commented Aug 12, 2020
View reviewed changes
x-pack/plugins/security_solution/public/overview/containers/overview_host/index.tsx
@@ -38,6 +40,13 @@ export interface OverviewHostProps extends QueryTemplateProps {

const OverviewHostComponentQuery = React.memo<OverviewHostProps & PropsFromRedux>(
({ id = ID, children, filterQuery, isInspected, sourceId, startDate, endDate }) => {
const { activeSourceGroupId, getManageSourceGroupById } = useManageSource();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

activeSourceGroupId will be determined by which page/view the component is called in

angorayc
angorayc reviewed Aug 12, 2020
View reviewed changes
x-pack/plugins/security_solution/public/common/containers/sourcerer/index.tsx
id,
indexPattern: getIndexFields(defaultIndex.join(), []),
indexPatterns: defaultIndex,
indicesExist: indicesExistOrDataTemporarilyUnavailable(undefined),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we keep it as undefined or null as default value instead of converting into true?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As some components may be mislead by this, start fetching data, and cause some errors on the client side.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why this was like this before (in useWithSource), but I imagine it solved a bug somewhere. Afraid to create it again

Copy link
Contributor

@XavierM XavierM Aug 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that important to keep as null at the beginning because that 's the way we know that we did not ask yet for it. so we do not show the splash screen and then the data.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@XavierM im confused if you're saying to change it or not??

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to keep it as undefined

XavierM
XavierM reviewed Aug 12, 2020
View reviewed changes
x-pack/plugins/security_solution/public/common/components/sourcerer/index.tsx Outdated Show resolved Hide resolved
x-pack/plugins/security_solution/public/common/containers/sourcerer/constants.ts Outdated Show resolved Hide resolved
x-pack/plugins/security_solution/public/common/containers/sourcerer/index.tsx
id,
indexPattern: getIndexFields(defaultIndex.join(), []),
indexPatterns: defaultIndex,
indicesExist: indicesExistOrDataTemporarilyUnavailable(undefined),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to keep it as undefined

XavierM
XavierM reviewed Aug 12, 2020
View reviewed changes
Copy link
Contributor

@XavierM XavierM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stephmilovic, I still think that we still need to see if we can get back the indexPatterns and build our browserfields from the index pattern services and see what you need from our server-side to be able to do that.

@stephmilovic
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

id value diff baseline
securitySolution 1914 +5 1909

async chunks size

id value diff baseline
securitySolution 7.2MB +27.3KB 7.2MB

page load bundle size

id value diff baseline
securitySolution 806.1KB +150.0B 805.9KB

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@spong
Copy link
Member

spong commented Aug 14, 2020

Just stopping by to say I am 5000% excited about this feature and am super stoked for what this change will mean to the EmbeddableMap component as users will finally be able to granularly configure the index patterns used to create layers. This will yuuuuge for them and will make many happy users! 🙂

Many thanks @stephmilovic @angorayc @XavierM!! ❤️ 🎉 🚀

@stephmilovic stephmilovic merged commit 1626490 into elastic:master Aug 14, 2020
@stephmilovic stephmilovic mentioned this pull request Aug 17, 2020
Merged
stephmilovic added a commit to stephmilovic/kibana that referenced this pull request Aug 17, 2020
@stephmilovic
[Security solution] Sourcerer: Kibana index pattern selector for secu…
40c6f60 
…rity views (elastic#74706)
gmmorris added a commit to gmmorris/kibana that referenced this pull request Aug 17, 2020
@gmmorris
Merge branch 'master' into actions/webhook-remove-header
c913bee 
* master: (24 commits)
  [ML] Functional tests - skip regression and classification tests
  [Ingest Manager] fix removing ingest pipelines from elasticsearch (elastic#75092)
  move tests for placeholder indices to setup (elastic#75096)
  [jest] temporarily extend default test timeout (elastic#75118)
  [cli] remove reference to removed --optimize flag (elastic#75083)
  skip flaky suite (elastic#75044)
  Adding /etc/rc.d/init.d/functions to the init script when present to … (elastic#22985)
  [jenkins] add pipeline for hourly security solution cypress tests (elastic#75087)
  [Reporting/Flaky Test] Skip test for paging list of reports (elastic#75075)
  remove .kbn-optimizer-cache upload (elastic#75086)
  skip flaky suite (elastic#74814)
  Actions add proxy support (elastic#74289)
  [ILM] TS conversion of Edit policy components (elastic#74747)
  [Resolver] simulator tests select elements directly instead of using descendant selectors. (elastic#75058)
  [Enterprise Search] Add Workplace Search side navigation (elastic#74894)
  [Security solution] Sourcerer: Kibana index pattern selector for security views (elastic#74706)
  [Logs UI] Remove apollo deps from log link-to routes (elastic#74502)
  [Maps] add map configurations to docker list (elastic#75035)
  [functional test][saved objects] update tests for additional copy saved objects to space (elastic#74907)
  Make the alerts plugin support generics (elastic#72716)
  ...
stephmilovic added a commit that referenced this pull request Aug 17, 2020
@stephmilovic
[Security solution] Sourcerer: Kibana index pattern selector for secu…
8cf8944 
…rity views (#74706) (#75128)
@MindyRS MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@stephmilovic stephmilovic mentioned this pull request Nov 30, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@angorayc angorayc angorayc approved these changes

@XavierM XavierM XavierM approved these changes

Assignees

@stephmilovic stephmilovic

Labels
release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:SIEM v7.10.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@stephmilovic @elasticmachine @angorayc @kibanamachine @spong @XavierM @MindyRS