Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] Use default component templates from Elasticsearch #163731

Merged
merged 15 commits into from
Nov 2, 2023
Merged

[Fleet] Use default component templates from Elasticsearch #163731

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
751ffa9
[Fleet] Use default component templates from Elasticsearch
joshdover Aug 11, 2023
c3b1e99
[CI] Auto-commit changed files from 'node scripts/precommit_hook.js -…
kibanamachine Aug 11, 2023
da4b3e3
Increment format version to force a reinstall of all templates
joshdover Aug 11, 2023
5e42adb
Remove fallback
joshdover Aug 14, 2023
df4b182
Update template names
joshdover Oct 30, 2023
4e2e3bb
Update tests
joshdover Oct 30, 2023
b3b0064
Fix unit and api integration tests
joshdover Oct 30, 2023
d324d59
Fix lint
joshdover Oct 30, 2023
180bf1e
Fix DLM
joshdover Oct 30, 2023
b399b5a
Fix DLM tests
joshdover Oct 30, 2023
c6a2c54
Add constant and simplify template logic
joshdover Oct 30, 2023
129635a
Fix input package test
joshdover Oct 31, 2023
fbbfa91
Merge remote-tracking branch 'upstream/main' into fleet/es-default-co…
joshdover Oct 31, 2023
d84da6f
Workaround for csp transform index template
joshdover Oct 31, 2023
f32dbe8
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine Oct 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion x-pack/plugins/cloud_security_posture/server/create_indices/create_indices.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import { errors } from '@elastic/elasticsearch';
import type { MappingTypeMapping } from '@elastic/elasticsearch/lib/api/types';
import type { ElasticsearchClient, Logger } from '@kbn/core/server';
import { STACK_COMPONENT_TEMPLATE_LOGS_SETTINGS } from '@kbn/fleet-plugin/server/constants';
import {
BENCHMARK_SCORE_INDEX_DEFAULT_NS,
BENCHMARK_SCORE_INDEX_PATTERN,
Expand All @@ -24,6 +25,10 @@ import { CloudSecurityPostureConfig } from '../config';
interface IndexTemplateSettings {
index: {
default_pipeline: string;
codec?: string;
mapping?: {
ignore_malformed: boolean;
};
Copy link
Contributor

@CohenIdo CohenIdo Nov 2, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joshdover, what those 2 new attributes are using for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These attributes were already being supplied previously by Fleet in the index template you were copying. With this PR, they're not longer added directly to the index template and instead are sourced from the logs@settings component template.

Since I'm filtering that component template out as mentioned above, I added these settings back explicitly to ensure that this index gets the same settings as it did before this change. I'll leave it to your team to decide if you need these settings or not, but they generally should help reduce storage costs and not drop any data.

};
lifecycle?: { name: string };
}
Expand Down Expand Up @@ -226,6 +231,10 @@ const updateIndexTemplate = async (
...template?.settings, // nothing inside
index: {
default_pipeline: latestFindingsPipelineIngestConfig.id,
codec: 'best_compression',
mapping: {
ignore_malformed: true,
},
},
lifecycle: { name: '' },
};
Expand All @@ -242,7 +251,7 @@ const updateIndexTemplate = async (
aliases: template?.aliases,
},
_meta,
composed_of: composedOf,
composed_of: composedOf.filter((ct) => ct !== STACK_COMPONENT_TEMPLATE_LOGS_SETTINGS),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elastic/kibana-cloud-security-posture Not basing the CSP transform index on the logs@settings template was necessary due to the logs@settings template on serverless specifying a default lifecycle. I could not find a way to override this from the index template, so instead I decided to filter this out for now and just manually specify the settings that were used before.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think we should remove this special initialization of the transform and index template and use the dedicated feature provided by Fleet for setting up transforms which would not have this issue.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joshdover, you right, we have task for that:

});

logger.info(`Updated index template successfully [Name: ${indexTemplateName}]`);
Expand Down
1 change: 1 addition & 0 deletions x-pack/plugins/fleet/common/types/models/epm.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -615,6 +615,7 @@ export interface IndexTemplate {
};
data_stream: { hidden?: boolean };
composed_of: string[];
ignore_missing_component_templates?: string[];
_meta: object;
}

Expand Down
12 changes: 11 additions & 1 deletion x-pack/plugins/fleet/server/constants/fleet_es_assets.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import { getESAssetMetadata } from '../services/epm/elasticsearch/meta';

const meta = getESAssetMetadata();

export const FLEET_INSTALL_FORMAT_VERSION = '1.0.0';
export const FLEET_INSTALL_FORMAT_VERSION = '1.1.0';

export const FLEET_AGENT_POLICIES_SCHEMA_VERSION = '1.1.1';

Expand Down Expand Up @@ -81,6 +81,16 @@ export const FLEET_COMPONENT_TEMPLATES = [
},
];

export const STACK_COMPONENT_TEMPLATE_LOGS_SETTINGS = `logs@settings`;
export const STACK_COMPONENT_TEMPLATE_METRICS_SETTINGS = `metrics@settings`;
export const STACK_COMPONENT_TEMPLATE_METRICS_TSDB_SETTINGS = `metrics@tsdb-settings`;

export const STACK_COMPONENT_TEMPLATES = [
STACK_COMPONENT_TEMPLATE_LOGS_SETTINGS,
STACK_COMPONENT_TEMPLATE_METRICS_SETTINGS,
STACK_COMPONENT_TEMPLATE_METRICS_TSDB_SETTINGS,
];

export const FLEET_FINAL_PIPELINE_VERSION = 4;

// If the content is updated you probably need to update the FLEET_FINAL_PIPELINE_VERSION too to allow upgrade of the pipeline
Expand Down
4 changes: 4 additions & 0 deletions x-pack/plugins/fleet/server/constants/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,10 @@ export {
FLEET_FINAL_PIPELINE_VERSION,
FLEET_INSTALL_FORMAT_VERSION,
FLEET_AGENT_POLICIES_SCHEMA_VERSION,
STACK_COMPONENT_TEMPLATE_LOGS_SETTINGS,
STACK_COMPONENT_TEMPLATE_METRICS_SETTINGS,
STACK_COMPONENT_TEMPLATE_METRICS_TSDB_SETTINGS,
STACK_COMPONENT_TEMPLATES,
} from './fleet_es_assets';
export { FILE_STORAGE_DATA_AGENT_INDEX } from './fleet_es_assets';
export { FILE_STORAGE_METADATA_AGENT_INDEX } from './fleet_es_assets';
Expand Down
4 changes: 0 additions & 4 deletions x-pack/plugins/fleet/server/services/epm/elasticsearch/template/default_settings.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,13 +65,9 @@ describe('buildDefaultSettings', () => {
expect(settings).toMatchInlineSnapshot(`
Object {
"index": Object {
"codec": "best_compression",
"lifecycle": Object {
"name": "logs",
},
"mapping": Object {
"ignore_malformed": true,
},
"query": Object {
"default_field": Array [
"field1Keyword",
Expand Down
10 changes: 0 additions & 10 deletions x-pack/plugins/fleet/server/services/epm/elasticsearch/template/default_settings.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,16 +73,6 @@ export function buildDefaultSettings({
name: ilmPolicy ? ilmPolicy : type,
},
}),
// What should be our default for the compression?
codec: 'best_compression',
// setting `ignore_malformed` only for data_stream for logs
...(type === 'logs'
? {
mapping: {
ignore_malformed: true,
},
}
: {}),
// All the default fields which should be queried have to be added here.
// So far we add all keyword and text fields here if there are any, otherwise
// this setting is skipped.
Expand Down
28 changes: 26 additions & 2 deletions x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@
import { createAppContextStartContractMock } from '../../../../mocks';
import { appContextService } from '../../..';
import { loadFieldsFromYaml } from '../../fields/field';
import type { RegistryDataStream } from '../../../../types';
import type { ArchivePackage, RegistryDataStream } from '../../../../types';

import { prepareTemplate } from './install';
import { prepareTemplate, prepareToInstallTemplates } from './install';

jest.mock('../../fields/field', () => ({
...jest.requireActual('../../fields/field'),
Expand Down Expand Up @@ -455,4 +455,28 @@ describe('EPM index template install', () => {

expect(packageTemplate).not.toHaveProperty('lifecycle');
});

test('test prepareToInstallTemplates does not include stack component templates in tracked assets', () => {
const dataStreamDatasetIsPrefixUnset = {
type: 'logs',
dataset: 'package.dataset',
title: 'test data stream',
release: 'experimental',
package: 'package',
path: 'path',
ingest_pipeline: 'default',
} as RegistryDataStream;

const { assetsToAdd } = prepareToInstallTemplates(
{
name: 'package',
version: '0.0.1',
data_streams: [dataStreamDatasetIsPrefixUnset],
} as ArchivePackage,
[],
[]
);

expect(assetsToAdd).not.toContainEqual({ id: 'logs@settings', type: 'component_template' });
});
});
6 changes: 5 additions & 1 deletion x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ import {
FLEET_COMPONENT_TEMPLATES,
PACKAGE_TEMPLATE_SUFFIX,
USER_SETTINGS_TEMPLATE_SUFFIX,
STACK_COMPONENT_TEMPLATES,
} from '../../../../constants';

import { getESAssetMetadata } from '../meta';
Expand Down Expand Up @@ -530,7 +531,7 @@ export function prepareTemplate({

const isIndexModeTimeSeries =
dataStream.elasticsearch?.index_mode === 'time_series' ||
experimentalDataStreamFeature?.features.tsdb;
!!experimentalDataStreamFeature?.features.tsdb;

const validFields = processFields(fields);

Expand Down Expand Up @@ -572,6 +573,7 @@ export function prepareTemplate({
registryElasticsearch: dataStream.elasticsearch,
mappings,
isIndexModeTimeSeries,
type: dataStream.type,
});

return {
Expand Down Expand Up @@ -616,6 +618,8 @@ export function getAllTemplateRefs(installedTemplates: IndexTemplateEntry[]) {
.filter(
(componentTemplateId) => !FLEET_COMPONENT_TEMPLATE_NAMES.includes(componentTemplateId)
)
// Filter stack component templates shared between integrations
.filter((componentTemplateId) => !STACK_COMPONENT_TEMPLATES.includes(componentTemplateId))
.map((componentTemplateId) => ({
id: componentTemplateId,
type: ElasticsearchAssetType.componentTemplate,
Expand Down
54 changes: 50 additions & 4 deletions x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,12 @@ describe('EPM template', () => {

const template = getTemplate({
templateIndexPattern,
type: 'logs',
packageName: 'nginx',
composedOfTemplates: [],
templatePriority: 200,
mappings: { properties: [] },
isIndexModeTimeSeries: false,
});
expect(template.index_patterns).toStrictEqual([templateIndexPattern]);
});
Expand All @@ -69,13 +71,35 @@ describe('EPM template', () => {
const composedOfTemplates = ['component1', 'component2'];

const template = getTemplate({
templateIndexPattern: 'name-*',
templateIndexPattern: 'logs-*',
type: 'logs',
packageName: 'nginx',
composedOfTemplates,
templatePriority: 200,
mappings: { properties: [] },
isIndexModeTimeSeries: false,
});
expect(template.composed_of).toStrictEqual([
'logs@settings',
...composedOfTemplates,
...FLEET_COMPONENT_TEMPLATES_NAMES,
]);
});

it('supplies metrics@tsdb-settings for time series', () => {
const composedOfTemplates = ['component1', 'component2'];

const template = getTemplate({
templateIndexPattern: 'metrics-*',
type: 'metrics',
packageName: 'nginx',
composedOfTemplates,
templatePriority: 200,
mappings: { properties: [] },
isIndexModeTimeSeries: true,
});
expect(template.composed_of).toStrictEqual([
'metrics@tsdb-settings',
...composedOfTemplates,
...FLEET_COMPONENT_TEMPLATES_NAMES,
]);
Expand All @@ -90,13 +114,16 @@ describe('EPM template', () => {
const composedOfTemplates = ['component1', 'component2'];

const template = getTemplate({
templateIndexPattern: 'name-*',
templateIndexPattern: 'logs-*',
type: 'logs',
packageName: 'nginx',
composedOfTemplates,
templatePriority: 200,
mappings: { properties: [] },
isIndexModeTimeSeries: false,
});
expect(template.composed_of).toStrictEqual([
'logs@settings',
...composedOfTemplates,
FLEET_GLOBALS_COMPONENT_TEMPLATE_NAME,
]);
Expand All @@ -106,34 +133,43 @@ describe('EPM template', () => {
const composedOfTemplates: string[] = [];

const template = getTemplate({
templateIndexPattern: 'name-*',
templateIndexPattern: 'logs-*',
type: 'logs',
packageName: 'nginx',
composedOfTemplates,
templatePriority: 200,
mappings: { properties: [] },
isIndexModeTimeSeries: false,
});
expect(template.composed_of).toStrictEqual(FLEET_COMPONENT_TEMPLATES_NAMES);
expect(template.composed_of).toStrictEqual([
'logs@settings',
...FLEET_COMPONENT_TEMPLATES_NAMES,
]);
});

it('adds hidden field correctly', () => {
const templateIndexPattern = 'logs-nginx.access-abcd-*';

const templateWithHidden = getTemplate({
templateIndexPattern,
type: 'logs',
packageName: 'nginx',
composedOfTemplates: [],
templatePriority: 200,
hidden: true,
mappings: { properties: [] },
isIndexModeTimeSeries: false,
});
expect(templateWithHidden.data_stream.hidden).toEqual(true);

const templateWithoutHidden = getTemplate({
templateIndexPattern,
type: 'logs',
packageName: 'nginx',
composedOfTemplates: [],
templatePriority: 200,
mappings: { properties: [] },
isIndexModeTimeSeries: false,
});
expect(templateWithoutHidden.data_stream.hidden).toEqual(undefined);
});
Expand All @@ -143,6 +179,7 @@ describe('EPM template', () => {

const templateWithGlobalAndDataStreamHidden = getTemplate({
templateIndexPattern,
type: 'logs',
packageName: 'nginx',
composedOfTemplates: [],
templatePriority: 200,
Expand All @@ -153,11 +190,13 @@ describe('EPM template', () => {
hidden: true,
},
},
isIndexModeTimeSeries: false,
});
expect(templateWithGlobalAndDataStreamHidden.data_stream.hidden).toEqual(true);

const templateWithDataStreamHidden = getTemplate({
templateIndexPattern,
type: 'logs',
packageName: 'nginx',
composedOfTemplates: [],
templatePriority: 200,
Expand All @@ -167,21 +206,25 @@ describe('EPM template', () => {
hidden: true,
},
},
isIndexModeTimeSeries: false,
});
expect(templateWithDataStreamHidden.data_stream.hidden).toEqual(true);

const templateWithoutDataStreamHidden = getTemplate({
templateIndexPattern,
type: 'logs',
packageName: 'nginx',
composedOfTemplates: [],
templatePriority: 200,
hidden: true,
mappings: { properties: [] },
isIndexModeTimeSeries: false,
});
expect(templateWithoutDataStreamHidden.data_stream.hidden).toEqual(true);

const templateWithGlobalHiddenTrueAndDataStreamHiddenFalse = getTemplate({
templateIndexPattern,
type: 'logs',
packageName: 'nginx',
composedOfTemplates: [],
templatePriority: 200,
Expand All @@ -192,15 +235,18 @@ describe('EPM template', () => {
hidden: false,
},
},
isIndexModeTimeSeries: false,
});
expect(templateWithGlobalHiddenTrueAndDataStreamHiddenFalse.data_stream.hidden).toEqual(true);

const templateWithoutHidden = getTemplate({
templateIndexPattern,
type: 'logs',
packageName: 'nginx',
composedOfTemplates: [],
templatePriority: 200,
mappings: { properties: [] },
isIndexModeTimeSeries: false,
});
expect(templateWithoutHidden.data_stream.hidden).toEqual(undefined);
});
Expand Down
Loading
Loading