[Fleet] Use default component templates from Elasticsearch #163731

joshdover · 2023-08-11T15:27:20Z

Summary

Blocked by:

Rename ES templates and pipelines according to the naming conventions elasticsearch#98535

This switches where integrations installed by EPM get their default index settings from to use the source-of-truth component templates supplied by Elasticsearch. This will help ensure that data streams configured by EPM always get the same defaults as data streams the user creates using the default logs-*-* and metrics-*-* templates. For now, no default mappings are sourced from Elasticsearch.

As part of this change the template format version was incremented to force EPM to reinstall all templates and rollover data streams on the Stack upgrade to the version including this change.

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

apmmachine · 2023-08-11T15:27:37Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
/oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

felixbarny · 2023-10-25T10:00:27Z

elastic/elasticsearch#99975 has just been merged so this should be unblocked now.

@jpountz last chance for objections.

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/default_settings.ts

joshdover · 2023-10-25T12:30:44Z

Thanks for the ping, @felixbarny. I'll get this updated soon

…-ref HEAD~1..HEAD --fix'

elasticmachine · 2023-10-30T13:05:00Z

Pinging @elastic/fleet (Team:Fleet)

kpollich

LGTM, one minor comment

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts

…mponents

…-fix'

kibana-ci · 2023-10-31T18:36:53Z

💚 Build Succeeded

Buildkite Build
Commit: f32dbe8

Metrics [docs]

✅ unchanged

History

💔 Build #172308 failed d84da6f
💔 Build #172136 failed fbbfa91
💔 Build #171880 failed c6a2c54
💔 Build #171852 failed b3b0064
💔 Build #171775 failed 5445274da672df0eddfb53ddebe9927cd7e24ba4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

joshdover · 2023-11-01T11:04:32Z

x-pack/plugins/cloud_security_posture/server/create_indices/create_indices.ts

@@ -242,7 +251,7 @@ const updateIndexTemplate = async (
        aliases: template?.aliases,
      },
      _meta,
-      composed_of: composedOf,
+      composed_of: composedOf.filter((ct) => ct !== STACK_COMPONENT_TEMPLATE_LOGS_SETTINGS),


@elastic/kibana-cloud-security-posture Not basing the CSP transform index on the logs@settings template was necessary due to the logs@settings template on serverless specifying a default lifecycle. I could not find a way to override this from the index template, so instead I decided to filter this out for now and just manually specify the settings that were used before.

I do think we should remove this special initialization of the transform and index template and use the dedicated feature provided by Fleet for setting up transforms which would not have this issue.

@joshdover, you right, we have task for that:

[Cloud Posture] Install Transforms using package assets #151860

CohenIdo · 2023-11-02T07:11:51Z

x-pack/plugins/cloud_security_posture/server/create_indices/create_indices.ts

+    codec?: string;
+    mapping?: {
+      ignore_malformed: boolean;
+    };


@joshdover, what those 2 new attributes are using for?

These attributes were already being supplied previously by Fleet in the index template you were copying. With this PR, they're not longer added directly to the index template and instead are sourced from the logs@settings component template.

Since I'm filtering that component template out as mentioned above, I added these settings back explicitly to ensure that this index gets the same settings as it did before this change. I'll leave it to your team to decide if you need these settings or not, but they generally should help reduce storage costs and not drop any data.

CohenIdo · 2023-11-02T07:12:27Z

x-pack/plugins/cloud_security_posture/server/create_indices/create_indices.ts

@@ -242,7 +251,7 @@ const updateIndexTemplate = async (
        aliases: template?.aliases,
      },
      _meta,
-      composed_of: composedOf,
+      composed_of: composedOf.filter((ct) => ct !== STACK_COMPONENT_TEMPLATE_LOGS_SETTINGS),


@joshdover, you right, we have task for that:

[Cloud Posture] Install Transforms using package assets #151860

…63731) ## Summary Fixes elastic#163141 Fixes elastic#160288 Blocked by: - elastic/elasticsearch#98535 This switches where integrations installed by EPM get their default index settings from to use the [source-of-truth component templates supplied by Elasticsearch](https://github.com/elastic/elasticsearch/tree/main/x-pack/plugin/core/template-resources/src/main/resources). This will help ensure that data streams configured by EPM always get the same defaults as data streams the user creates using the default `logs-*-*` and `metrics-*-*` templates. For now, no default mappings are sourced from Elasticsearch. As part of this change the template format version was incremented to force EPM to reinstall all templates and rollover data streams on the Stack upgrade to the version including this change. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]>

felixbarny reviewed Oct 25, 2023

View reviewed changes

joshdover and others added 7 commits October 30, 2023 12:42

[Fleet] Use default component templates from Elasticsearch

751ffa9

[CI] Auto-commit changed files from 'node scripts/precommit_hook.js -…

c3b1e99

…-ref HEAD~1..HEAD --fix'

Increment format version to force a reinstall of all templates

da4b3e3

Remove fallback

5e42adb

Update template names

df4b182

Update tests

4e2e3bb

Fix unit and api integration tests

b3b0064

joshdover force-pushed the fleet/es-default-components branch from 5445274 to b3b0064 Compare October 30, 2023 13:04

joshdover marked this pull request as ready for review October 30, 2023 13:04

joshdover requested a review from a team as a code owner October 30, 2023 13:04

joshdover added release_note:enhancement Team:Fleet Team label for Observability Data Collection Fleet team labels Oct 30, 2023

joshdover added 3 commits October 30, 2023 14:35

Fix lint

d324d59

Fix DLM

180bf1e

Fix DLM tests

b399b5a

kpollich approved these changes Oct 30, 2023

View reviewed changes

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts Outdated Show resolved Hide resolved

joshdover added 4 commits October 30, 2023 14:55

Add constant and simplify template logic

c6a2c54

Fix input package test

129635a

Merge remote-tracking branch 'upstream/main' into fleet/es-default-co…

fbbfa91

…mponents

Workaround for csp transform index template

d84da6f

joshdover requested a review from a team as a code owner October 31, 2023 16:45

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

f32dbe8

…-fix'

joshdover commented Nov 1, 2023

View reviewed changes

CohenIdo enabled auto-merge (squash) November 1, 2023 11:54

CohenIdo disabled auto-merge November 1, 2023 11:54

CohenIdo approved these changes Nov 2, 2023

View reviewed changes

joshdover merged commit ca90574 into elastic:main Nov 2, 2023

joshdover deleted the fleet/es-default-components branch November 2, 2023 08:55

kibanamachine added v8.12.0 backport:skip This commit does not require backporting labels Nov 2, 2023

kpollich mentioned this pull request Jan 29, 2024

Integrations - Assets displays No Assets found after upgrading from 8.10.4 to 8.12.0 #175173

Closed

felixbarny mentioned this pull request Jan 30, 2024

Add reference to ecs@mappings for each index template #174855

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Use default component templates from Elasticsearch #163731

[Fleet] Use default component templates from Elasticsearch #163731

joshdover commented Aug 11, 2023 •

edited

Loading

apmmachine commented Aug 11, 2023

felixbarny commented Oct 25, 2023

joshdover commented Oct 25, 2023

elasticmachine commented Oct 30, 2023

kpollich left a comment

kibana-ci commented Oct 31, 2023

joshdover Nov 1, 2023

joshdover Nov 1, 2023

CohenIdo Nov 2, 2023

CohenIdo Nov 2, 2023 •

edited

Loading

joshdover Nov 2, 2023

CohenIdo Nov 2, 2023

[Fleet] Use default component templates from Elasticsearch #163731

[Fleet] Use default component templates from Elasticsearch #163731

Conversation

joshdover commented Aug 11, 2023 • edited Loading

Summary

Checklist

apmmachine commented Aug 11, 2023

🤖 GitHub comments

felixbarny commented Oct 25, 2023

joshdover commented Oct 25, 2023

elasticmachine commented Oct 30, 2023

kpollich left a comment

Choose a reason for hiding this comment

kibana-ci commented Oct 31, 2023

💚 Build Succeeded

Metrics [docs]

History

joshdover Nov 1, 2023

Choose a reason for hiding this comment

joshdover Nov 1, 2023

Choose a reason for hiding this comment

CohenIdo Nov 2, 2023

Choose a reason for hiding this comment

CohenIdo Nov 2, 2023 • edited Loading

Choose a reason for hiding this comment

joshdover Nov 2, 2023

Choose a reason for hiding this comment

CohenIdo Nov 2, 2023

Choose a reason for hiding this comment

joshdover commented Aug 11, 2023 •

edited

Loading

CohenIdo Nov 2, 2023 •

edited

Loading