[Security Solution] [Sourcerer] Update available on legacy timelines #120022

stephmilovic · 2021-11-30T20:29:57Z

Summary

Resolves https://github.com/elastic/security-team/issues/1921

Implements an "Update available" workflow for pre-8.0 timelines and timelines that have indices that got deleted from the default data view

To test: save this json as a timelines.ndjson file and upload the timelines to the timelines page in security solution

Flow 1

Note: i paused the most in this flow so y'all can ready the copy, the rest of the flows go quicker
Legacy Timeline includes an active index pattern that is not included in the default data view, user decides to update data view. Page refresh is prompted

Flow 2

Legacy Timeline includes an active index pattern that is not included in the default data view, user decides to update to the new sourcerer with only the indices that already exist in the data view

Flow 3

Legacy Timeline includes an active index pattern that is not included in the default data view, user decides to reset to the new sourcerer and abandon their index pattern

Flow 4

Legacy Timeline index patterns are ALL included in the default data view, user updates to the new sourcerer without needing to update the data view

Flow 5

Legacy Timeline none of the index patterns in the legacy timeline match, don't allow user to upgrade data view. Forces them to reset to new sourcerer or keep a bad index pattern

Flow 6

Start with a valid non-legacy timeline. Delete one of the index patterns from advanced settings. Fallsback into temporary timeline and prompts user to re-add the deleted index pattern

Flow 7

Error state, not expected

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2021-12-06T14:11:41Z

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

elasticmachine · 2021-12-06T14:11:43Z

Pinging @elastic/security-solution (Team: SecuritySolution)

monina-n · 2021-12-06T16:57:19Z

from design, overall all these look great and thanks for accounting for all the possible flows! even flow 6 (which is a very rare edge case I don't expect a lot of users to come across) is very clear in what happens.

All good from design - please remember to update the width of the time selector so it is it's default width and have the lock and refresh button moved to be beside it. It can be in this PR or in a separate one, whatever's better for you

machadoum

It looks great! I played around and couldn't find any bug.

I still haven't taken a look at all files. I will finish it tomorrow. I left some minor comments for now.

x-pack/plugins/security_solution/public/app/home/template_wrapper/bottom_bar/index.tsx

machadoum · 2021-12-06T16:05:52Z

x-pack/plugins/security_solution/public/common/components/sourcerer/index.tsx

+              }}
+            />
+          </>
+        ) as unknown as string,


Curiosity: Why double casting?

TS2352: Conversion of type 'Element' to type 'string' may be a mistake because neither type sufficiently overlaps with the other. If this was intentional, convert the expression to 'unknown' first.

but i need the link here... maybe ill submit a ticket to core asking this prop be updated

x-pack/plugins/security_solution/public/common/components/sourcerer/temporary.tsx

stephmilovic · 2021-12-07T02:54:56Z

x-pack/plugins/security_solution/public/common/components/sourcerer/index.tsx

    setPopoverIsOpen(false);
  }, [onChangeDataView, dataViewId, selectedOptions]);

  const handleClosePopOver = useCallback(() => {
    setPopoverIsOpen(false);
    setExpandAdvancedOptions(false);
  }, []);
-  const trigger = useMemo(


moved into its own component

stephmilovic · 2021-12-07T02:55:34Z

x-pack/plugins/security_solution/public/common/components/sourcerer/index.tsx


-  const dataViewSelectOptions = useMemo(


moved into usePickIndexPatterns

stephmilovic · 2021-12-07T02:56:03Z

x-pack/plugins/security_solution/public/common/components/sourcerer/index.tsx

  }, [selectedDataViewId]);

-  const tooltipContent = useMemo(


moved into its own component along with buttonWithTooltip

kibana-ci · 2021-12-07T04:37:55Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	2800	2805	+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	4.6MB	4.6MB	+10.9KB
`timelines`	202.9KB	203.1KB	+148.0B
total			+11.1KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	243.7KB	244.0KB	+273.0B
`timelines`	164.5KB	164.6KB	+62.0B
total			+335.0B

History

💚 Build #11498 succeeded ead12fc
💔 Build #11420 failed 0afad2d
💔 Build #11172 failed 99e4803
💔 Build #11074 failed ef18962
💔 Build #10819 failed 8a0b8f9
💔 Build #10231 failed a6ecf07

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

angorayc · 2021-12-07T10:59:26Z

I found some inconstancy in the index patterns options, nothing todo with these flow though. In the first picture, the index pattern of security alert is not selectable, but we still can tick show only detection alerts to select that index pattern. Should we disable the show only detection alerts in this case?

machadoum

Awesome! : 🔥🔥 🚀

machadoum · 2021-12-07T10:55:27Z

x-pack/plugins/security_solution/public/common/store/sourcerer/helpers.ts

+export const ensurePatternFormat = (patternList: string[]): string[] =>
+  [
+    ...new Set(
+      patternList.reduce((acc: string[], pattern: string) => [...pattern.split(','), ...acc], [])


very nitpicking:

Suggested change

patternList.reduce((acc: string[], pattern: string) => [...pattern.split(','), ...acc], [])

patternList.flatMap((pattern) => pattern.split(','))

Or even more FP!

import { split } from "lodash/fp"; ... patternList.flatMap(split(','))

Oooor point-free style:

import { split, flatMap } from "lodash/fp"; ... flatMap(split(','), patternList)

machadoum · 2021-12-07T10:59:11Z

x-pack/plugins/security_solution/public/common/store/sourcerer/helpers.ts

-      ...(isEmpty(selectedPatterns)
+      missingPatterns,
+      // if in timeline, allow for empty in case pattern was deleted
+      // need flow for this


Is it a TODO?

whoops, nope thats flow 6. i need to delete this comment

stephmilovic · 2021-12-07T14:08:51Z

Should we disable the show only detection alerts in this case?

@angorayc yes ill do that in a follow up

kibanamachine · 2021-12-07T14:33:37Z

💔 Backport failed

Status	Branch	Result
❌	8.0	Commit could not be cherrypicked due to conflicts

To backport manually run:
node scripts/backport --pr 120022

…lastic#120022)

stephmilovic and others added 21 commits November 17, 2021 13:04

wip

ad5aecf

bugfix

0505853

add test

e04a0cd

Merge branch 'bugfix_tl_sourcerer' into temp_flow

90b640c

Fix type

ed2787f

Merge branch 'bugfix_tl_sourcerer' into temp_flow

44aecc8

here flow done

3cbf44e

Merge branch 'main' into temp_flow

142cdbd

dataViewId: string | null

18a8ad8

modal added

1c2b0d5

add comments

50dafcd

more pseudocode

79334df

updates

89f47fb

add index pattern

0beb4a7

unit tests

cdfe57b

types

7077b8d

add refresh button

ac7eb86

fix types

6fac4f3

merge

14a7ee7

Merge branch 'main' into temp_flow

9213eb5

merge master

235c6d0

stephmilovic added 2 commits November 30, 2021 20:10

fixing

a6ecf07

need a commit

67165d2

stephmilovic added 3 commits December 5, 2021 14:23

bug fixing

6b79399

fix tests

99e4803

Merge branch 'main' into temp_flow

432e57b

stephmilovic marked this pull request as ready for review December 6, 2021 14:11

stephmilovic requested review from a team as code owners December 6, 2021 14:11

stephmilovic added 3 commits December 6, 2021 08:35

fix err message

5caf6a1

undo whoops

948114c

fix timeline bug

0afad2d

machadoum reviewed Dec 6, 2021

View reviewed changes

stephmilovic added 2 commits December 6, 2021 12:18

pablo review and more tests

ead12fc

cleanups and tests

ccb42ed

stephmilovic commented Dec 7, 2021

View reviewed changes

machadoum approved these changes Dec 7, 2021

View reviewed changes

stephmilovic merged commit 268b290 into elastic:main Dec 7, 2021

stephmilovic mentioned this pull request Dec 7, 2021

[8.0] [Security Solution] [Sourcerer] Update available on legacy timelines (#120022) #120633

Merged

TinLe pushed a commit to TinLe/kibana that referenced this pull request Dec 22, 2021

[Security Solution] [Sourcerer] Update available on legacy timelines (e…

f717abe

…lastic#120022)

ghost mentioned this pull request Jun 24, 2022

[DOCS][New Page] Creates topic for data views in Elastic Security elastic/security-docs#2034

Merged

stephmilovic mentioned this pull request Nov 30, 2022

[Security Solution] Introduce ad hoc dataView for the legacy indices to unify the usage of the dataView plugin API across security_solution #142903

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] [Sourcerer] Update available on legacy timelines #120022

[Security Solution] [Sourcerer] Update available on legacy timelines #120022

stephmilovic commented Nov 30, 2021 •

edited

Loading

elasticmachine commented Dec 6, 2021

elasticmachine commented Dec 6, 2021

monina-n commented Dec 6, 2021

machadoum left a comment

machadoum Dec 6, 2021

stephmilovic Dec 6, 2021

stephmilovic Dec 6, 2021

stephmilovic Dec 7, 2021

stephmilovic Dec 7, 2021

stephmilovic Dec 7, 2021

kibana-ci commented Dec 7, 2021

angorayc commented Dec 7, 2021

machadoum left a comment

machadoum Dec 7, 2021

machadoum Dec 7, 2021

stephmilovic Dec 7, 2021 •

edited

Loading

stephmilovic commented Dec 7, 2021

kibanamachine commented Dec 7, 2021

	patternList.reduce((acc: string[], pattern: string) => [...pattern.split(','), ...acc], [])
	patternList.flatMap((pattern) => pattern.split(','))

[Security Solution] [Sourcerer] Update available on legacy timelines #120022

[Security Solution] [Sourcerer] Update available on legacy timelines #120022

Conversation

stephmilovic commented Nov 30, 2021 • edited Loading

Summary

Flow 1

Flow 2

Flow 3

Flow 4

Flow 5

Flow 6

Flow 7

Checklist

elasticmachine commented Dec 6, 2021

elasticmachine commented Dec 6, 2021

monina-n commented Dec 6, 2021

machadoum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented Dec 7, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

angorayc commented Dec 7, 2021

machadoum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephmilovic Dec 7, 2021 • edited Loading

Choose a reason for hiding this comment

stephmilovic commented Dec 7, 2021

kibanamachine commented Dec 7, 2021

💔 Backport failed

stephmilovic commented Nov 30, 2021 •

edited

Loading

stephmilovic Dec 7, 2021 •

edited

Loading