[RAM] ResponseOps saved object updates #152857

XavierM · 2023-03-07T21:40:24Z

Summary

This PR is just the first phase for response ops to go through their saved object attributes. The idea is to comment out all the attributes that we all agree that we do not need to filter/search/sort/aggregate on.

After, in a second phase/PR, we will create a new file who will represent all of attributes in our saved object as a source of truth. Then, we will generate our SO mappings from this source of truth to register our saved object.

Phase 3, we will try to generate also our type from our source of truth.

…object-updates

elasticmachine · 2023-03-07T21:40:29Z

Pinging @elastic/response-ops (Team:ResponseOps)

elasticmachine · 2023-03-07T21:40:32Z

Pinging @elastic/uptime (Team:uptime)

…object-updates

pmuellr

left comments on some additional fields we may be able to comment out

other than that, LGTM; going to pound on it for a while now ...

...core/server/integration_tests/saved_objects/migrations/group2/check_registered_types.test.ts

x-pack/plugins/alerting/server/saved_objects/mappings.ts

pmuellr · 2023-04-12T21:35:05Z

x-pack/plugins/task_manager/server/saved_objects/mappings.ts

+    // traceparent: {
+    //   type: 'text',
+    // },
+    params: {


params and state are JSON bags, so likely don't need to be indexed.

I think it was failing test because we are using _source to get this attributes and since they are not mapped anymore, source was not working anymore.

hmmm .... that doesn't seem right - the _source should always contain the JSON blob we index. It should be there regardless if it's indexed/mapped any more.

But ... not hard for me to believe that someone would use a runtime field on here, which would be a little easier to do with a field, than if it was in _source - certainly faster if for some reason this is done at runtime. Maybe telemetry?

let me re-check/search and give you the right answer here

ok You are right! we were able to not index these two, I tried multiple time and tests are not failing!

…object-updates

XavierM · 2023-04-13T15:51:55Z

@JiaweiWu Do you see any problem with that 0199f45 ?

JiaweiWu · 2023-04-13T16:49:56Z

@JiaweiWu Do you see any problem with that 0199f45 ?

Looks good, we don't do any sorting/filtering on rule settings at the moment.

…object-updates

…-fix'

…bana into alerting-saved-object-updates

…object-updates

afharo

LGTM

pmuellr

Woohoo! Got rid of the JSON bags!

LGTM!

…object-updates

…bana into alerting-saved-object-updates

kibana-ci · 2023-04-18T15:02:16Z

💚 Build Succeeded

Buildkite Build
Commit: 247d385

Metrics [docs]

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id	before	after	diff
`action`	6	4	-2
`action_task_params`	5	-	-5
`alert`	101	68	-33
`connector_token`	7	3	-4
`rules-settings`	9	-	-9
total			-53

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`securitySolution`	432	435	+3

Total ESLint disabled count

id	before	after	diff
`securitySolution`	512	515	+3

History

💚 Build #120425 succeeded 9993eb7
💛 Build #120381 was flaky 2336116
💔 Build #120259 failed 2b023c8
💔 Build #120099 failed 7c4d24b
💔 Build #119974 failed 446c22b
💔 Build #119961 failed 5b8fc18

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

banderror · 2023-04-19T13:25:39Z

x-pack/plugins/alerting/server/saved_objects/mappings.ts

    monitoring: {
      properties: {
        run: {
          properties: {


@XavierM Sorry I'm a bit late with this comment since the PR has been already merged, but:

Until it's too late and we can't introduce non-additive changes to the mappings anymore, can we consolidate and refactor the monitoring and lastRun objects? These two objects contain similar data, and it's confusing for a lot of devs, including me, @jpdjere, @maximpn etc.

You can see how our code looks like that has to deal with these two objects, it's more complex than should be:

https://github.com/elastic/kibana/blob/7e633bb063051f60e5c26a1ed9fa243c2cd5b1bb/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/create_rule_execution_summary.ts

XavierM added 13 commits February 8, 2023 14:22

bring tests

43eb8df

Merge branch 'main' of github.com:elastic/kibana

b0ecc29

Merge branch 'main' of github.com:elastic/kibana

a68aec7

Merge branch 'main' of github.com:elastic/kibana

561b6e0

Merge branch 'main' of github.com:elastic/kibana

db8a5f6

Merge branch 'main' of github.com:elastic/kibana

182ed1f

Merge branch 'main' of github.com:elastic/kibana

7bcac6c

Merge branch 'main' of github.com:elastic/kibana

bd02789

Merge branch 'main' of github.com:elastic/kibana

2a02b2c

fix rac privileges

c7a2993

Merge branch 'main' of github.com:elastic/kibana

3bd34da

first round on removing attributes from our saved object

1cc5f39

Merge branch 'main' of github.com:elastic/kibana into alerting-saved-…

fdf0ab5

…object-updates

XavierM added release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v8.8.0 labels Mar 7, 2023

XavierM requested review from a team as code owners March 7, 2023 21:40

botelastic bot added the Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability label Mar 7, 2023

XavierM marked this pull request as draft March 7, 2023 21:40

XavierM force-pushed the alerting-saved-object-updates branch from ec7fe69 to fdf0ab5 Compare March 7, 2023 21:41

XavierM requested review from a team and spong and removed request for a team March 7, 2023 21:41

This comment was marked as outdated.

Sign in to view

XavierM added 2 commits April 12, 2023 16:11

update core

e4c0842

Merge branch 'main' of github.com:elastic/kibana into alerting-saved-…

6577821

…object-updates

pmuellr requested changes Apr 12, 2023

View reviewed changes

XavierM added 3 commits April 13, 2023 09:25

remove flapping properties

0199f45

Merge branch 'main' of github.com:elastic/kibana into alerting-saved-…

2880938

…object-updates

update rule settings SO

04c4be6

XavierM and others added 9 commits April 13, 2023 17:04

Merge branch 'main' of github.com:elastic/kibana into alerting-saved-…

5b8fc18

…object-updates

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

c349928

…-fix'

test without params

8d6a817

Merge branch 'alerting-saved-object-updates' of github.com:XavierM/ki…

446c22b

…bana into alerting-saved-object-updates

remove state from task management so

d5c1f1e

Merge branch 'main' into alerting-saved-object-updates

7c4d24b

Merge branch 'main' of github.com:elastic/kibana into alerting-saved-…

2b023c8

…object-updates

update core

2336116

Merge branch 'main' into alerting-saved-object-updates

9993eb7

afharo approved these changes Apr 18, 2023

View reviewed changes

pmuellr approved these changes Apr 18, 2023

View reviewed changes

XavierM added 2 commits April 18, 2023 09:13

Merge branch 'main' of github.com:elastic/kibana into alerting-saved-…

20769ab

…object-updates

Merge branch 'alerting-saved-object-updates' of github.com:XavierM/ki…

247d385

…bana into alerting-saved-object-updates

XavierM enabled auto-merge (squash) April 18, 2023 13:25

pmuellr mentioned this pull request Apr 18, 2023

[ResponseOps][discuss] convert "JSON bags" in alerting Saved Objects to plain objects #155169

Open

XavierM merged commit ad2d5ad into elastic:main Apr 18, 2023

kibanamachine added the backport:skip This commit does not require backporting label Apr 18, 2023

banderror reviewed Apr 19, 2023

View reviewed changes

mikecote mentioned this pull request Oct 12, 2023

Task state shouldn't be analyzed in Elasticsearch #78850

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RAM] ResponseOps saved object updates #152857

[RAM] ResponseOps saved object updates #152857

XavierM commented Mar 7, 2023 •

edited

Loading

elasticmachine commented Mar 7, 2023

elasticmachine commented Mar 7, 2023

This comment was marked as outdated.

pmuellr left a comment

pmuellr Apr 12, 2023

XavierM Apr 13, 2023

pmuellr Apr 13, 2023

XavierM Apr 13, 2023

XavierM Apr 17, 2023

XavierM commented Apr 13, 2023 •

edited

Loading

JiaweiWu commented Apr 13, 2023

afharo left a comment

pmuellr left a comment

kibana-ci commented Apr 18, 2023

ESLint disabled line counts

Total ESLint disabled count

banderror Apr 19, 2023

[RAM] ResponseOps saved object updates #152857

[RAM] ResponseOps saved object updates #152857

Conversation

XavierM commented Mar 7, 2023 • edited Loading

Summary

elasticmachine commented Mar 7, 2023

elasticmachine commented Mar 7, 2023

This comment was marked as outdated.

pmuellr left a comment

Choose a reason for hiding this comment

pmuellr Apr 12, 2023

Choose a reason for hiding this comment

XavierM Apr 13, 2023

Choose a reason for hiding this comment

pmuellr Apr 13, 2023

Choose a reason for hiding this comment

XavierM Apr 13, 2023

Choose a reason for hiding this comment

XavierM Apr 17, 2023

Choose a reason for hiding this comment

XavierM commented Apr 13, 2023 • edited Loading

JiaweiWu commented Apr 13, 2023

afharo left a comment

Choose a reason for hiding this comment

pmuellr left a comment

Choose a reason for hiding this comment

kibana-ci commented Apr 18, 2023

💚 Build Succeeded

Metrics [docs]

Saved Objects .kibana field count

ESLint disabled line counts

Total ESLint disabled count

History

banderror Apr 19, 2023

Choose a reason for hiding this comment

XavierM commented Mar 7, 2023 •

edited

Loading

XavierM commented Apr 13, 2023 •

edited

Loading