Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] Rollover data streams when package w/ TSDB setting changed is installed #157869

Merged
merged 5 commits into from
May 16, 2023
Merged

[Fleet] Rollover data streams when package w/ TSDB setting changed is installed #157869

Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0e5ac7d
WIP rollover data streams when package w/ TSDB changed is installed
kpollich May 16, 2023
cf016b5
Refactor per code review
kpollich May 16, 2023
b0b9281
Add integration test
kpollich May 16, 2023
de1610a
Merge branch 'main' into fleet/tsdb-rollovers
kibanamachine May 16, 2023
2663738
Revert clumsy delete of multiple_versions files 🙃
kpollich May 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 45 additions & 4 deletions x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@
*/

import type { ElasticsearchClient, Logger } from '@kbn/core/server';
import type { IndicesIndexSettings } from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
import type {
IndicesIndexSettings,
MappingTypeMapping,
} from '@elastic/elasticsearch/lib/api/typesWithBodyKey';

import type { Field, Fields } from '../../fields/field';
import type {
Expand Down Expand Up @@ -642,7 +645,20 @@ const updateExistingDataStream = async ({
esClient: ElasticsearchClient;
logger: Logger;
}) => {
const existingDs = await esClient.indices.get({
index: dataStreamName,
});

const existingDsConfig = Object.values(existingDs);
const currentBackingIndexConfig = existingDsConfig[existingDsConfig.length - 1];
kpollich marked this conversation as resolved.
Show resolved Hide resolved

const currentIndexMode = currentBackingIndexConfig?.settings?.index?.mode;
// @ts-expect-error Property 'mode' does not exist on type 'MappingSourceField'
const currentSourceType = currentBackingIndexConfig.mappings?._source?.mode;

let settings: IndicesIndexSettings;
let mappings: MappingTypeMapping;

try {
const simulateResult = await retryTransientEsErrors(() =>
esClient.indices.simulateTemplate({
Expand All @@ -651,14 +667,17 @@ const updateExistingDataStream = async ({
);

settings = simulateResult.template.settings;
const mappings = simulateResult.template.mappings;
mappings = simulateResult.template.mappings;

// for now, remove from object so as not to update stream or data stream properties of the index until type and name
// are added in https://github.com/elastic/kibana/issues/66551. namespace value we will continue
// to skip updating and assume the value in the index mapping is correct
if (mappings && mappings.properties) {
delete mappings.properties.stream;
delete mappings.properties.data_stream;
}

logger.debug(`Updating mappings for ${dataStreamName}`);
await retryTransientEsErrors(
() =>
esClient.indices.putMapping({
Expand All @@ -668,16 +687,38 @@ const updateExistingDataStream = async ({
}),
{ logger }
);
// if update fails, rollover data stream

// if update fails, rollover data stream and bail out
} catch (err) {
logger.error(`Mappings update for ${dataStreamName} failed`);
logger.error(err);

await rolloverDataStream(dataStreamName, esClient);
return;
}

// Trigger a rollover if the index mode or source type has changed
if (
currentIndexMode !== settings?.index?.mode ||
// @ts-expect-error Property 'mode' does not exist on type 'MappingSourceField'
currentSourceType !== mappings?._source?.mode
) {
logger.debug(
kpollich marked this conversation as resolved.
Show resolved Hide resolved
`Index mode or source type has changed for ${dataStreamName}, triggering a rollover`
);
await rolloverDataStream(dataStreamName, esClient);
}

// update settings after mappings was successful to ensure
// pointing to the new pipeline is safe
// for now, only update the pipeline
if (!settings?.index?.default_pipeline) return;
if (!settings?.index?.default_pipeline) {
return;
}

try {
logger.debug(`Updating settings for ${dataStreamName}`);

await retryTransientEsErrors(
() =>
esClient.indices.putSettings({
Expand Down