Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution]After upgrade to 8.8.1 from 8.7.1 duplicate index are present and selecting index .alerts-security.alerts-default not showing data. #159107

Closed
sukhwindersingh-qasource opened this issue Jun 6, 2023 · 13 comments
Closed

[Security Solution]After upgrade to 8.8.1 from 8.7.1 duplicate index are present and selecting index .alerts-security.alerts-default not showing data. #159107

sukhwindersingh-qasource opened this issue Jun 6, 2023 · 13 comments
Assignees
@angorayc
Labels
bug Fixes for quality problems that affect the customer experience fixed impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. QA:Validated Issue has been validated by QA Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Threat Hunting:Explore Team:Threat Hunting Security Solution Threat Hunting Team v8.8.2 v8.9.0

Comments

@sukhwindersingh-qasource
Copy link

sukhwindersingh-qasource commented Jun 6, 2023 •
edited
Loading

Describe the bug:

  • After upgrade to 8.8.1 from 8.7.1 duplicate index are present and selecting index .alerts-security.alerts-default not showing data.

Build Details:

VERSION: 8.8.1 BC2
BUILD: 63240
COMMIT: 0fda51d5cd9f9b724fd0ed4356221d49f2c7af27

Preconditions

  • Kibana should be running.
  • Deploy 8.7.1 Generate data on it.
  • i.e Alerts , Enable user risk , host risk , Configure ml
  • Upgrade the build to 8.8.1

Steps to Reproduce

  • Navigate to overview page
  • Select data view
  • Observe that there are duplicate indices present.
  • Now select .alerts-security.alerts-default this index.
  • And observe no data is showing.
  • To verify this index has data navigate to timeline and select index .alerts-security.alerts-default.
  • Type the desired query and observe data is present there.

Actual Result

  • After upgrade to 8.8.1 from 8.7.1 duplicate index are present and selecting index .alerts-security.alerts-default not showing data.
  • Data view button also vanishes.

Expected Result

  • Duplicate index should not be present.
  • After selecting index data should show up.
  • Data view button should not be vanished.

Screen-cast

Overview.-.Kibana.Mozilla.Firefox.2023-06-06.16-59-12.mp4

image
@sukhwindersingh-qasource sukhwindersingh-qasource added bug Fixes for quality problems that affect the customer experience triage_needed impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Jun 6, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@muskangulati-qasource
Copy link

Reviewed and assigned to @MadameSheema

@MadameSheema
Copy link
Member

thanks @sukhwindersingh-qasource can you please provide me the credentials of the environment where the issue can be reproduced?

@oatkiller
Copy link
Contributor

oatkiller commented Jun 6, 2023 •
edited
Loading

I tried to reproduce but couldn’t. My 8.7.1 had just an auditbeat integration and I created only 1 custom rule which had alerts. I did the upgrade. I don’t see the issue. Perhaps the 8.7.1 environment needs to be in a more specific state to reproduce. Any tips?

@oatkiller
Copy link
Contributor

oatkiller commented Jun 6, 2023 •
edited
Loading

I made another attempt at reproducing this. I created alerts from auditbeat and then I enabled user and host risk scoring. Lot's of data views show up in stack management:

image

However they don't show up in the data view selector in Timeline:
image

Or the data view selector in the Overview dashboard:
image

I expect to see all the data views in Stack Management represented in the Data View selector.

I did a hard refresh of the browser and reloaded the page:
image

Now I do see the data views in the selector.

After upgrading to 8.8.1 I do see the duplicate entries in the data view selector:

image

Looking at Stack Management, I also see the .alerts-security.alerts-default data view duplicated:

image

I selected .alerts-security.alerts-default data view in the selector and now I cannot see any dashboards. The app is acting like there is no data to be shown:

image

I deleted the two .alerts-security.alerts-default data views using stack management. Then I did a hard refresh. Then I navigated to the main Security app page. Then I did another hard refresh and the app shows the normal UI and both data views are gone. The 'Security Default Data View' item is available in the data view selector.

image

@MadameSheema MadameSheema added Team:Threat Hunting:Explore Team:Threat Hunting Security Solution Threat Hunting Team labels Jun 6, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

@MadameSheema
Copy link
Member

@stephmilovic @machadoum assigning this ticket to your team since looks like it is just happening when you have the host and user risk scores dashboards enabled before the upgrade.

This is also happening from 8.7.1 to 8.8.0 upgrade.

@angorayc angorayc self-assigned this Jun 7, 2023
@angorayc
Copy link
Contributor

angorayc commented Jun 8, 2023 •
edited
Loading

I can see that .alerts-security.alerts-default data view duplicated because we created one when installing user risk score module and created the other one when installing host risk score module. These data views are consumed by the dashboards each module installed.
Screenshot 2023-06-08 at 13 42 39

To fix this, we can check if the .alerts-security.alerts-default data view exists before installing. It's just if we are going to replace the current installation way of host or user risk score module, this might not be worthwhile to fix.

The data view in Security Solution page only refreshes on the page load to save requests. After having the two .alerts-security.alerts-default data views appears in the option, I can see data populating from Discover.

The reason why we see get started page component on dashboard > overview when selecting .alerts-security.alerts-default was because these existing logic:

  1. kibana/x-pack/plugins/security_solution/public/overview/pages/overview.tsx

    Line 71 in eb33c82

    {indicesExist ? (

  2. https://github.com/angorayc/kibana/blob/8fba39c2dadd9b916e50a02efb4de39f7391c7b2/x-pack/plugins/security_solution/public/common/store/sourcerer/helpers.ts#L115-L117.

We intentionally exclude alert index when checking if indices exists. Guess that's because most of the visualisations in dashboards or explore sections do not consume alert index. But this is annoying when it happens, I think we should change the logic not to exclude alert index when checking indicesExist. Otherwise whenever users create the same data view contains alert index manually, this will happen.

@angorayc angorayc mentioned this issue Jun 15, 2023
Merged
1 task
angorayc added a commit that referenced this issue Jun 15, 2023
@angorayc
[SecuritySolution] Update checkIndicesExists logic (#159806)
24bfa05 
## Summary

issue: #159107



**Steps to verify:**

1. Generate some alerts and enable host or user risk score module.
2. Hard refresh the page, select the alerts data view. 

<img width="639" alt="Screenshot 2023-06-15 at 14 54 54"
src="https://github.com/elastic/kibana/assets/6295984/412a2a9c-9125-4972-8c95-24dda90ad529">

3. Visit overview, host, network and users page. All should `Not`
display the get started page.



https://github.com/elastic/kibana/assets/6295984/4b942604-f98f-40fe-bbca-9cfd11cdf275


### Checklist

Delete any items that are not applicable to this PR.


- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
@angorayc
Copy link
Contributor

I had a simple fix in #159806
It doesn't change the logic of installing risk score modules, so if users installed both modules, they'll still have two alerts data views installed. I updated the logic of indicesExist, so the page will still display if users select the alerts data views.

kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Jun 15, 2023
@angorayc
[SecuritySolution] Update checkIndicesExists logic (elastic#159806)
1ae668a 
## Summary

issue: elastic#159107

**Steps to verify:**

1. Generate some alerts and enable host or user risk score module.
2. Hard refresh the page, select the alerts data view.

<img width="639" alt="Screenshot 2023-06-15 at 14 54 54"
src="https://github.com/elastic/kibana/assets/6295984/412a2a9c-9125-4972-8c95-24dda90ad529">

3. Visit overview, host, network and users page. All should `Not`
display the get started page.

https://github.com/elastic/kibana/assets/6295984/4b942604-f98f-40fe-bbca-9cfd11cdf275

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

(cherry picked from commit 24bfa05)
kibanamachine referenced this issue Jun 15, 2023
@kibanamachine @angorayc
[8.8] [SecuritySolution] Update checkIndicesExists logic (#159806) (#…
1bdfeba 
…159836)

# Backport

This will backport the following commits from `main` to `8.8`:
- [[SecuritySolution] Update checkIndicesExists logic
(#159806)](#159806)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Angela
Chuang","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-06-15T19:06:30Z","message":"[SecuritySolution]
Update checkIndicesExists logic (#159806)\n\n## Summary\r\n\r\nissue:
https://github.com/elastic/kibana/issues/159107\r\n\r\n\r\n\r\n**Steps
to verify:**\r\n\r\n1. Generate some alerts and enable host or user risk
score module.\r\n2. Hard refresh the page, select the alerts data view.
\r\n\r\n<img width=\"639\" alt=\"Screenshot 2023-06-15 at 14 54
54\"\r\nsrc=\"https://github.com/elastic/kibana/assets/6295984/412a2a9c-9125-4972-8c95-24dda90ad529\">\r\n\r\n3.
Visit overview, host, network and users page. All should
`Not`\r\ndisplay the get started
page.\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/6295984/4b942604-f98f-40fe-bbca-9cfd11cdf275\r\n\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"24bfa0514efb3eb16b3eb3276679dd53229d01ba","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","Team:Threat
Hunting","Team: SecuritySolution","Team:Threat
Hunting:Explore","v8.9.0","v8.8.2"],"number":159806,"url":"https://github.com/elastic/kibana/pull/159806","mergeCommit":{"message":"[SecuritySolution]
Update checkIndicesExists logic (#159806)\n\n## Summary\r\n\r\nissue:
https://github.com/elastic/kibana/issues/159107\r\n\r\n\r\n\r\n**Steps
to verify:**\r\n\r\n1. Generate some alerts and enable host or user risk
score module.\r\n2. Hard refresh the page, select the alerts data view.
\r\n\r\n<img width=\"639\" alt=\"Screenshot 2023-06-15 at 14 54
54\"\r\nsrc=\"https://github.com/elastic/kibana/assets/6295984/412a2a9c-9125-4972-8c95-24dda90ad529\">\r\n\r\n3.
Visit overview, host, network and users page. All should
`Not`\r\ndisplay the get started
page.\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/6295984/4b942604-f98f-40fe-bbca-9cfd11cdf275\r\n\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"24bfa0514efb3eb16b3eb3276679dd53229d01ba"}},"sourceBranch":"main","suggestedTargetBranches":["8.8"],"targetPullRequestStates":[{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/159806","number":159806,"mergeCommit":{"message":"[SecuritySolution]
Update checkIndicesExists logic (#159806)\n\n## Summary\r\n\r\nissue:
https://github.com/elastic/kibana/issues/159107\r\n\r\n\r\n\r\n**Steps
to verify:**\r\n\r\n1. Generate some alerts and enable host or user risk
score module.\r\n2. Hard refresh the page, select the alerts data view.
\r\n\r\n<img width=\"639\" alt=\"Screenshot 2023-06-15 at 14 54
54\"\r\nsrc=\"https://github.com/elastic/kibana/assets/6295984/412a2a9c-9125-4972-8c95-24dda90ad529\">\r\n\r\n3.
Visit overview, host, network and users page. All should
`Not`\r\ndisplay the get started
page.\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/6295984/4b942604-f98f-40fe-bbca-9cfd11cdf275\r\n\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"24bfa0514efb3eb16b3eb3276679dd53229d01ba"}},{"branch":"8.8","label":"v8.8.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Angela Chuang <[email protected]>
@stephmilovic
Copy link
Contributor

@sukhwindersingh-qasource can you please confirm the fix?

@sukhwindersingh-qasource
Copy link
Author

Hi @stephmilovic ,Sure we will validate this ticket once 8.8.2 BC1 Will be available at production.

@sukhwindersingh-qasource
Copy link
Author

Hi @MadameSheema @stephmilovic @angorayc

We have validated this issue on 8.8.2 BC1 build after upgrading it from 8.7.1 and observed that issue looks like , It is Fixed. ✔️

Please find the below Testing Details:

Build info

VERSION: 8.8.2 BC1
BUILD: 63328
COMMIT: db8521641e9f4c545859a80e19f34066c507a137

Screen-Cast

Overview.-.Kibana.Mozilla.Firefox.2023-06-23.17-04-20.mp4

As this issue needs to be validated on 8.9.0 also , so keeping this ticket opened. After availability of 8.9.0 BC1 we will validate it and close the ticket and add QA Validated label

Thanks!!

@sukhwindersingh-qasource
Copy link
Author

Hi @MadameSheema @stephmilovic @angorayc

We have validated this issue on 8.9.0 BC1 build after upgrading it from 8.7.1 and observed that issue looks like , It is Fixed. ✔️

Please find the below Testing Details:

Build info

VERSION: 8.9.0 BC1
BUILD: 64385
COMMIT: 313dac73d8d3bc5930447f732e3ae163fb1b7f70

Screen-Cast

Overview.-.Kibana.Mozilla.Firefox.2023-06-26.16-03-29.mp4

Hence we are closing this ticket and also marking it as QA Validated.

Thanks!!

@sukhwindersingh-qasource sukhwindersingh-qasource added the QA:Validated Issue has been validated by QA label Jun 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@angorayc angorayc

Labels
bug Fixes for quality problems that affect the customer experience fixed impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. QA:Validated Issue has been validated by QA Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Threat Hunting:Explore Team:Threat Hunting Security Solution Threat Hunting Team v8.8.2 v8.9.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@oatkiller @machadoum @angorayc @stephmilovic @elasticmachine @MadameSheema @muskangulati-qasource @sukhwindersingh-qasource