[SIEM] Import timeline fix #65448

angorayc · 2020-05-06T10:54:01Z

Summary

This PR is to allow template timeline updated via import,
and fix the case we had in #64439

How to verify this PR:

copy the json below and paste it onto a file

{"savedObjectId":null,"version":"WzM5LDFd","timelineType":"template","templateTimelineId":"49188240-6530-11ea-90dd-7d87cXavier","templateTimelineVersion":4,"columns":[{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"@timestamp","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"message","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"event.category","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"event.action","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"host.name","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"source.ip","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"destination.ip","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"user.name","searchable":null}],"dataProviders":[],"description":"is Super","eventType":"all","filters":[],"kqlMode":"filter","kqlQuery":{"filterQuery":{"serializedQuery":"{\"bool\":{\"should\":[{\"exists\":{\"field\":\"host.name\"}}],\"minimum_should_match\":1}}","kuery":{"expression":"host.name: *","kind":"kuery"}}},"title":"My template timeline updated","dateRange":{"start":1584020448645,"end":1584106848645},"savedQueryId":null,"sort":{"columnId":"@timestamp","sortDirection":"desc"},"favorite":[{"favoriteDate":1584480703444,"keySearch":"WGF2aWVyTQ==","fullName":"xavier mouligneau","userName":"XavierM"}],"created":1588693289719,"createdBy":"XavierM","updated":1588693315060,"updatedBy":"XavierM","eventNotes":[{"noteId":"e3cf0900-8ee6-11ea-bff9-5be90e832634","version":"WzM0LDFd","eventId":"Cfwg1HABO74GafNORxyF","note":"pinned and event note","timelineId":"e37d8da0-8ee6-11ea-bff9-5be90e832634","created":1588693290295,"createdBy":"XavierM","updated":1588693290295,"updatedBy":"XavierM"},{"noteId":"e3cfcc50-8ee6-11ea-bff9-5be90e832634","version":"WzM1LDFd","eventId":"Cvwg1HABO74GafNORxyF","note":"event note 1","timelineId":"e37d8da0-8ee6-11ea-bff9-5be90e832634","created":1588693290295,"createdBy":"XavierM","updated":1588693290295,"updatedBy":"XavierM"}],"globalNotes":[{"noteId":"e3cebae0-8ee6-11ea-bff9-5be90e832634","version":"WzI2LDFd","note":"Global","timelineId":"e37d8da0-8ee6-11ea-bff9-5be90e832634","created":1588693290295,"createdBy":"XavierM","updated":1588693290295,"updatedBy":"XavierM"},{"noteId":"e3cf3010-8ee6-11ea-bff9-5be90e832634","version":"WzM2LDFd","note":"Note","timelineId":"e37d8da0-8ee6-11ea-bff9-5be90e832634","created":1588693290295,"createdBy":"XavierM","updated":1588693290295,"updatedBy":"XavierM"}],"pinnedEventIds":["Yfwg1HABO74GafNObyFv","Cfwg1HABO74GafNORxyF","Cvwg1HABO74GafNORxyF"]}

save it as template_timeline.ndjson
import in to timeline
More tweak: [SIEM] Import timeline fix #65448 (comment)

Error cases:

Import a new timeline (Create timeline via import)
- input validation:
  - timelineId has to be null / empty or a timelineId that doesn't exist in SO - if given a timelineId that's already exist, therefor throwing error: savedObjectId: "${id}" already exists
  - title must have - if not given, throw error: Title cannot be empty
  - version: has to be null (this is to reserve the possibility of version control for updating timeline in the future)
  - timelineType has to be null / default or not specified - if set to template, I'll assume it is trying to create a template
  - status has to be null / 'active' or not specified - we don't expose this field to exported file. So far we do not block the 'immutable' status as we need that when installing Elastic template. If status is given as draft, throw error Cannot create a draft timeline
Import an existing timeline (Update existing timeline via import)
- We don't allow updating existing timeline atm: therefor throwing error: savedObjectId: "${id}" already exists
Import a new template (Create custom template via import)
- input validation:
  - templateTimelineId has to be given a uuid that doesn't exist in SO, if given a timelineId that's already exist, therefor throwing error: savedObjectId: "${id}" already exists; if not given a templateTimelineId, throw error: Create template timeline without a template timeline ID is not allowed
  - title must have - if not given, throw error: Title cannot be empty
  - timelineType has to be template - if set to others, I'll assume it is trying to create a default timeline
  - status has to be null / 'active' or not specified - we don't expose this field to exported file. So far we do not block the 'immutable' status as we need that when installing Elastic template. If status is given as draft, throw error Cannot create a draft timeline
Import an existing template
- input validation:
  - timelineType has to be the same as existing template timeline, otherwise throw error: 'Update timelineType is not allowed'
  - status has to be the same as existing template timeline. Otherwise throw error Update status is not allowed
  - templateTimelineId has to be given a uuid that matches an exist in SO.
    - If not given: template timeline without a template timeline ID is not allowed
    - if given a timelineId that's not exist, therefor throwing error: CREATE template timeline with PATCH is not allowed, please use POST instead (Given template timeline doesn't exist)
    - given timelineId doesn't match: Timeline id doesn't match with existing template timeline
    - given templateTimelineVersion doesn't match: TimelineVersion conflict: The given version doesn not match with existing timeline

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
~~Documentation was added for features that require explanation or tutorials~~
Unit or functional tests were updated or added to match the most common scenarios
~~This was checked for keyboard-only and screenreader accessibility~~
~~This renders correctly on smaller devices using a responsive layout. (You can test this in your browser~~
~~This was checked for cross-browser compatibility, including a check against IE11~~

For maintainers

~~This was checked for breaking API changes and was labeled appropriately~~

fix unit tests apply failure checker clean up error message fix update template

angorayc · 2020-05-12T16:11:23Z

@elasticmachine merge upstream

elasticmachine · 2020-05-12T16:11:25Z

user doesn't have permission to update head repository

patrykkopycinski · 2020-05-19T17:52:00Z

@elasticmachine merge upstream

elasticmachine · 2020-05-19T17:52:01Z

user doesn't have permission to update head repository

elasticmachine · 2020-05-19T17:52:22Z

Pinging @elastic/siem (Team:SIEM)

…ne-fix # Conflicts: # x-pack/plugins/siem/server/lib/timeline/routes/import_timelines_route.ts

x-pack/plugins/siem/server/lib/timeline/routes/utils/timeline_input.ts

patrykkopycinski · 2020-05-20T13:40:14Z

x-pack/plugins/siem/server/lib/timeline/routes/utils/timeline_input.ts

+  }
+
+  public getVersion() {
+    return this.version;


does it support template timelines?

yes, timeline_input is a generic input for both timeline and template.
When we init this object for template, we send templateTimelineVersion instead of version
https://github.com/angorayc/kibana/blob/af212242578580adf9e460ec7c4db6dd4d2dd01c/x-pack/plugins/siem/server/lib/timeline/routes/update_timelines_route.ts#L58

x-pack/plugins/siem/server/lib/timeline/routes/utils/timeline_input.ts

x-pack/plugins/siem/server/lib/timeline/routes/create_timelines_route.ts

x-pack/plugins/siem/server/lib/timeline/routes/import_timelines_route.ts

x-pack/plugins/siem/server/lib/timeline/routes/update_timelines_route.ts

x-pack/plugins/siem/server/lib/timeline/routes/import_timelines_route.ts

angorayc · 2020-06-25T09:00:32Z

@elasticmachine merge upstream

elasticmachine · 2020-06-25T09:00:34Z

user doesn't have permission to update head repository

XavierM · 2020-06-25T14:43:40Z

in this picture below, we are having two bugs

I should be able to add a global not in template timeline, only if it is immutable I cannot
The second msg is cut off

angorayc · 2020-06-25T15:51:30Z

in this picture below, we are having two bugs

I should be able to add a global note in template timeline, only if it is immutable I cannot

The second msg is cut off

Seems that the behaviour of add Note button is as expected. I checked on siem-dev, the add Note is disabled until we type something in the text box.

But sure, I'll fix item 2. in another PR.

angorayc · 2020-06-25T15:55:21Z

angorayc · 2020-06-25T18:52:01Z

@elasticmachine merge upstream

elasticmachine · 2020-06-25T18:52:03Z

user doesn't have permission to update head repository

…ne-fix

patrykkopycinski

I was able to import the timeline template smoothly now! LGTM 💪

… import-timeline-fix

kibanamachine · 2020-06-27T01:39:44Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: fed80c3

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
securitySolution	788	+1	787

History

💚 Build #56852 succeeded 3db3ffc
💚 Build #56742 succeeded 0e3003d
💚 Build #56606 succeeded c917ce8
💔 Build #56603 failed 005c8d4
💔 Build #56440 failed c1127aa

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* fix import timeline and clean up fix unit tests apply failure checker clean up error message fix update template * add unit tests * clean up common libs * rename variables * add unit tests * fix types * Fix imports * rename file * poc * fix unit test * review * cleanup fallback values * cleanup * check if title exists * fix unit test * add unit test * lint error * put the flag for disableTemplate into common * add immutiable * fix unit * check templateTimelineVersion only when update via import * update template timeline via import with response * add template filter * add filter count * add filter numbers * rename * enable pin events and note under active status * disable comment and pinnedEvents for template timelines * add timelineType for openTimeline * enable note icon for template * add timeline type for propertyLeft * fix types * duplicate elastic template * update schema * fix status check * fix import * add templateTimelineType * disable note for immutable timeline * fix unit * fix error message * fix update * fix types * rollback change * rollback change * fix create template timeline * add i18n for error message * fix unit test * fix wording and disable delete btn for immutable timeline * fix unit test provider * fix types * fix toaster * fix notes and pins * add i18n * fix selected items * set disableTemplateto true * move templateInfo to helper * review + imporvement * fix review * fix types * fix types Co-authored-by: Patryk Kopycinski <[email protected]> Co-authored-by: Xavier Mouligneau <[email protected]> # Conflicts: # x-pack/plugins/security_solution/public/timelines/store/timeline/epic.ts

* master: (59 commits) [Lens] Fix broken test (elastic#70117) [SIEM] Import timeline fix (elastic#65448) [SECURITY SOLUTION][INGEST] UX update for ingest manager edit/create datasource for endpoint (elastic#70079) [Telemetry] Collector Schema (elastic#64942) [Endpoint] Add Endpoint empty states for onboarding (elastic#69626) Hide unused resolver buttons (elastic#70112) [Security] `Investigate in Resolver` Timeline Integration (elastic#70111) [Discover] Improve styling of graphs in sidebar (elastic#69440) [Metrics UI] Fix EuiTheme type issue (elastic#69735) skip failing suite (elastic#70104) (elastic#70103) [ENDPOINT] Hide the Timeline Flyout while on the Management Pages (elastic#69998) [SIEM][CASE] Persist callout when dismissed (elastic#68372) [SIEM][Exceptions] - Cleaned up and updated exception list item comment structure (elastic#69532) [Maps] remove indexing state from redux (elastic#69765) Add API integration test for deleting data streams. (elastic#70020) renames SIEM to Security Solution (elastic#70070) Adding saved_objects_page in OSS (elastic#69900) [Lens] Use accordion menus in field list for available and empty fields (elastic#68871) Dynamic uiActions & license support (elastic#68507) [SIEM] Update readme for timeline apis (elastic#67038) ...

…bana into alerting/consumer-based-rbac * 'alerting/consumer-based-rbac' of github.com:gmmorris/kibana: (25 commits) [Lens] Fix broken test (elastic#70117) [SIEM] Import timeline fix (elastic#65448) [SECURITY SOLUTION][INGEST] UX update for ingest manager edit/create datasource for endpoint (elastic#70079) [Telemetry] Collector Schema (elastic#64942) [Endpoint] Add Endpoint empty states for onboarding (elastic#69626) Hide unused resolver buttons (elastic#70112) [Security] `Investigate in Resolver` Timeline Integration (elastic#70111) [Discover] Improve styling of graphs in sidebar (elastic#69440) [Metrics UI] Fix EuiTheme type issue (elastic#69735) skip failing suite (elastic#70104) (elastic#70103) [ENDPOINT] Hide the Timeline Flyout while on the Management Pages (elastic#69998) [SIEM][CASE] Persist callout when dismissed (elastic#68372) [SIEM][Exceptions] - Cleaned up and updated exception list item comment structure (elastic#69532) [Maps] remove indexing state from redux (elastic#69765) Add API integration test for deleting data streams. (elastic#70020) renames SIEM to Security Solution (elastic#70070) Adding saved_objects_page in OSS (elastic#69900) [Lens] Use accordion menus in field list for available and empty fields (elastic#68871) Dynamic uiActions & license support (elastic#68507) [SIEM] Update readme for timeline apis (elastic#67038) ...

elasticmachine · 2021-09-23T14:34:56Z

Pinging @elastic/security-solution (Team: SecuritySolution)

angorayc requested review from a team as code owners May 6, 2020 10:54

angorayc added v7.8.0 v8.0.0 release_note:skip Skip the PR/issue when compiling release notes labels May 6, 2020

fix import timeline and clean up

67e7406

fix unit tests apply failure checker clean up error message fix update template

angorayc force-pushed the import-timeline-fix branch from 86228e2 to 67e7406 Compare May 7, 2020 17:33

angorayc added 5 commits May 8, 2020 06:52

Merge remote-tracking branch 'upstream/master' into import-timeline-fix

f30c970

add unit tests

f5893e5

clean up common libs

9ee6cc0

rename variables

1668031

add unit tests

6aff660

angorayc added 3 commits May 12, 2020 17:12

Merge remote-tracking branch 'upstream/master' into import-timeline-fix

10880fa

Merge remote-tracking branch 'upstream/master' into import-timeline-fix

2b8a694

Merge remote-tracking branch 'upstream/master' into import-timeline-fix

d69bfb5

patrykkopycinski added bug Fixes for quality problems that affect the customer experience Team:SIEM labels May 19, 2020

angorayc and others added 3 commits May 19, 2020 19:13

fix types

c49b7d4

Merge branch 'master' of github.com:elastic/kibana into import-timeli…

09b03e0

…ne-fix # Conflicts: # x-pack/plugins/siem/server/lib/timeline/routes/import_timelines_route.ts

Fix imports

18f98a8

patrykkopycinski reviewed May 20, 2020

View reviewed changes

x-pack/plugins/siem/server/lib/timeline/routes/import_timelines_route.ts Outdated Show resolved Hide resolved

angorayc and others added 3 commits May 20, 2020 17:08

rename file

af21224

poc

5cf93cd

Merge remote-tracking branch 'upstream/master' into import-timeline-fix

2ec6f0c

move templateInfo to helper

635a6e5

XavierM added 2 commits June 25, 2020 09:01

Merge branch 'master' of github.com:elastic/kibana into pr/65448

07222f2

review + imporvement

f5c68e2

fix review

94f2a7d

angorayc mentioned this pull request Jun 25, 2020

Enhancement for timeline templates #69972

Closed

11 tasks

angorayc added 2 commits June 25, 2020 19:04

fix types

cd0bde5

Merge remote-tracking branch 'upstream/master' into import-timeline-fix

c1127aa

angorayc and others added 3 commits June 26, 2020 10:21

Merge remote-tracking branch 'upstream/master' into import-timeline-fix

005c8d4

fix types

c917ce8

Merge branch 'master' of github.com:elastic/kibana into import-timeli…

0e3003d

…ne-fix

patrykkopycinski approved these changes Jun 26, 2020

View reviewed changes

angorayc added 3 commits June 26, 2020 21:08

Merge remote-tracking branch 'upstream/master' into import-timeline-fix

71aeed2

Merge branch 'import-timeline-fix' of github.com:angorayc/kibana into…

3db3ffc

… import-timeline-fix

Merge remote-tracking branch 'upstream/master' into import-timeline-fix

fed80c3

angorayc merged commit f4e7f14 into elastic:master Jun 27, 2020

angorayc mentioned this pull request Jun 27, 2020

[7.x] [SIEM] Import timeline fix (#65448) #70145

Merged

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM] Import timeline fix #65448

[SIEM] Import timeline fix #65448

angorayc commented May 6, 2020 •

edited

Loading

angorayc commented May 12, 2020

elasticmachine commented May 12, 2020

patrykkopycinski commented May 19, 2020

elasticmachine commented May 19, 2020

elasticmachine commented May 19, 2020

patrykkopycinski May 20, 2020

angorayc May 21, 2020

angorayc commented Jun 25, 2020

elasticmachine commented Jun 25, 2020

XavierM commented Jun 25, 2020

angorayc commented Jun 25, 2020

angorayc commented Jun 25, 2020 •

edited

Loading

angorayc commented Jun 25, 2020

elasticmachine commented Jun 25, 2020

patrykkopycinski left a comment

kibanamachine commented Jun 27, 2020

elasticmachine commented Sep 23, 2021

[SIEM] Import timeline fix #65448

[SIEM] Import timeline fix #65448

Conversation

angorayc commented May 6, 2020 • edited Loading

Summary

Checklist

For maintainers

angorayc commented May 12, 2020

elasticmachine commented May 12, 2020

patrykkopycinski commented May 19, 2020

elasticmachine commented May 19, 2020

elasticmachine commented May 19, 2020

patrykkopycinski May 20, 2020

Choose a reason for hiding this comment

angorayc May 21, 2020

Choose a reason for hiding this comment

angorayc commented Jun 25, 2020

elasticmachine commented Jun 25, 2020

XavierM commented Jun 25, 2020

angorayc commented Jun 25, 2020

angorayc commented Jun 25, 2020 • edited Loading

angorayc commented Jun 25, 2020

elasticmachine commented Jun 25, 2020

patrykkopycinski left a comment

Choose a reason for hiding this comment

kibanamachine commented Jun 27, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

History

elasticmachine commented Sep 23, 2021

angorayc commented May 6, 2020 •

edited

Loading

angorayc commented Jun 25, 2020 •

edited

Loading