Added Time-series indexing (TSDB) to Integrations Experimental Indexi…

…ng settings (#144974)

## Summary

Closes #144530

Added a new option under integration / data streams / Index settings
(experimental) / TSDB

- [x] Add a toggle to the "Experimental indexing features" data streams
UI for TSDB
- Enabling this toggle adds `index.mode: time_series` to the data
stream's index template settings.
- Note: `index.routing_path` value was not needed to be added in this
logic, because there is an existing elasticsearch automation that
generates it from dimension fields, see test instructions.
- [x] If the current package's manifest contains `index_mode:
time_series` for a given data stream, the toggle should be in an
"enabled + readonly" state, e.g. it cannot be disabled
- Note: currently there is no package that has this setting enabled, I
tested locally by manually modifying the package info response. Will
need to figure out a way to create a mock package to test this.
- [x] Once the toggle is enabled and the policy is saved, the toggle
should _not_ be disable-able. Enabling TSDB for a data stream is an
irreversible operation
- [x] Add a tooltip to make the irreversible nature of this operation
clear

TSDB setting in System package:
<img width="915" alt="image"
src="https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png">
<img width="632" alt="image"
src="https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png">

Tooltip:
<img width="509" alt="image"
src="https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png">


## Test instructions

### Setup local registry

- For testing, I created a new version of `System` package, that has a
`dimension` field called `test_dimension`. This is needed so that the
elasticsearch automation runs to generate the `routing_path` setting.

- Download the test data zip:

[registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip)


- Start a registry:
```
docker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main
```
- Add this to your kibana config:
```
xpack.fleet.registryUrl: http://localhost:8080
```

- Open kibana, open Add System integration page (should be version
`1.20.5` coming from local registry)
- Enable only `Collect metrics from System instances / System cpu
metrics` stream, and enable `Time-series indexing (TSDB)` switch under
`Advanced options`
- Click on save integration. The integration should be added
successfully to an agent policy.
- Start a fleet server and enroll an agent to the previously created
agent policy. Wait for the agent to be healthy, this should trigger
system metrics flowing in, this way triggering the creation of the data
stream.
- Go to `Stack Management / Index Management / Index Templates`, and
open the details of `metrics-system.cpu`, check that it has this in
`Settings` tab: `index.mode: time_series`

<img width="2013" alt="image"
src="https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png">

- Go to `Indices` tab and look for
`data_stream="metrics-system.cpu-default"` including hidden indices. You
should see `routing_path` populated in Settings, and `test_dimension` in
Mappings.

<img width="1792" alt="image"
src="https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png">
<img width="774" alt="image"
src="https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png">

### Testing scenario of enabling TSDB in package spec by default

- Prepared a newer version of `system` package `v1.20.9` that has
`index_mode: "time_series"` in `system.cpu` data stream in package
manifest.
- Download and extract the below zip and restart the local epr container
to read from this folder.

[registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip)
- Open add system integration page
- Check that `tsdb:true` is set in `Preview API request` for
`system.cpu` data stream

<img width="1033" alt="image"
src="https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png">

- Open the system cpu metrics stream on the policy editor, and check
that the TSDB switch is turned on.
<img width="1093" alt="image"
src="https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png">
<img width="1051" alt="image"
src="https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png">

### Update 11/16:

- After latest changes, expect `routing_path` with dimension fields to
be generated in Index Template when enabling TSDB:

<img width="1163" alt="image"
src="https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png">

This was added to fix the issue mentioned here
#144974 (comment)

- After adding TSDB, modifying the parent component template should be
successful.

<img width="555" alt="image"
src="https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png">


### Checklist

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <[email protected]>

src/core/server/integration_tests/saved_objects/migrations/check_registered_types.test.ts

@@ -80,7 +80,7 @@ describe('checking migration metadata changes on all registered SO types', () =>
         "endpoint:user-artifact": "f94c250a52b30d0a2d32635f8b4c5bdabd1e25c0",
         "endpoint:user-artifact-manifest": "8c14d49a385d5d1307d956aa743ec78de0b2be88",
         "enterprise_search_telemetry": "fafcc8318528d34f721c42d1270787c52565bad5",
-        "epm-packages": "cb22b422398a785e7e0565a19c6d4d5c7af6f2fd",
+        "epm-packages": "fe3716a54188b3c71327fa060dd6780a674d3994",
         "epm-packages-assets": "9fd3d6726ac77369249e9a973902c2cd615fc771",
         "event_loop_delays_daily": "d2ed39cf669577d90921c176499908b4943fb7bd",
         "exception-list": "fe8cc004fd2742177cdb9300f4a67689463faf9c",

x-pack/plugins/fleet/common/services/simplified_package_policy_helper.test.ts

@@ -131,6 +131,7 @@ describe('toPackagePolicy', () => {
               data_stream: 'logs-nginx.access',
               features: {
                 synthetic_source: true,
+                tsdb: false,
               },
             },
           ],
@@ -142,6 +143,7 @@ describe('toPackagePolicy', () => {
           data_stream: 'logs-nginx.access',
           features: {
             synthetic_source: true,
+            tsdb: false,
           },
         },
       ]);

x-pack/plugins/fleet/common/types/models/epm.ts

@@ -352,6 +352,7 @@ export interface RegistryElasticsearch {
   'index_template.mappings'?: estypes.MappingTypeMapping;
   'ingest_pipeline.name'?: string;
   source_mode?: 'default' | 'synthetic';
+  index_mode?: 'time_series';
 }
 
 export interface RegistryDataStreamPrivileges {
@@ -444,7 +445,7 @@ export type PackageInfo =
   | Installable<Merge<ArchivePackage, EpmPackageAdditions>>;
 
 // TODO - Expand this with other experimental indexing types
-export type ExperimentalIndexingFeature = 'synthetic_source';
+export type ExperimentalIndexingFeature = 'synthetic_source' | 'tsdb';
 
 export interface ExperimentalDataStreamFeature {
   data_stream: string;

x-pack/plugins/fleet/common/types/models/package_policy.ts

@@ -33,6 +33,7 @@ export interface NewPackagePolicyInputStream {
       privileges?: {
         indices?: string[];
       };
+      index_mode?: string;
     };
   };
   release?: RegistryRelease;

x-pack/plugins/fleet/public/applications/fleet/sections/agent_policy/create_package_policy_page/components/steps/components/package_policy_input_panel.tsx

@@ -19,6 +19,7 @@ import {
   EuiButtonEmpty,
 } from '@elastic/eui';
 
+import { getRegistryDataStreamAssetBaseName } from '../../../../../../../../../common/services';
 import type {
   NewPackagePolicy,
   NewPackagePolicyInput,
@@ -125,6 +126,41 @@ export const PackagePolicyInputPanel: React.FunctionComponent<{
       [packageInputStreams, packagePolicyInput.streams]
     );
 
+    // setting Indexing setting: TSDB to enabled by default, if the data stream's index_mode is set to time_series
+    let isUpdated = false;
+    inputStreams.forEach(({ packagePolicyInputStream }) => {
+      const dataStreamInfo = packageInfo.data_streams?.find(
+        (ds) => ds.dataset === packagePolicyInputStream?.data_stream.dataset
+      );
+
+      if (dataStreamInfo?.elasticsearch?.index_mode === 'time_series') {
+        if (!packagePolicy.package) return;
+        if (!packagePolicy.package?.experimental_data_stream_features)
+          packagePolicy.package!.experimental_data_stream_features = [];
+
+        const dsName = getRegistryDataStreamAssetBaseName(packagePolicyInputStream!.data_stream);
+        const match = packagePolicy.package!.experimental_data_stream_features.find(
+          (feat) => feat.data_stream === dsName
+        );
+        if (match) {
+          if (!match.features.tsdb) {
+            match.features.tsdb = true;
+            isUpdated = true;
+          }
+        } else {
+          packagePolicy.package!.experimental_data_stream_features.push({
+            data_stream: dsName,
+            features: { tsdb: true, synthetic_source: false },
+          });
+          isUpdated = true;
+        }
+      }
+    });
+
+    if (isUpdated) {
+      updatePackagePolicy(packagePolicy);
+    }
+
     return (
       <>
         {/* Header / input-level toggle */}

x-pack/plugins/fleet/public/applications/fleet/sections/agent_policy/create_package_policy_page/components/steps/components/package_policy_input_stream.tsx

@@ -5,7 +5,7 @@
  * 2.0.
  */
 
-import React, { useState, Fragment, memo, useMemo, useEffect, useRef } from 'react';
+import React, { useState, Fragment, memo, useMemo, useEffect, useRef, useCallback } from 'react';
 import ReactMarkdown from 'react-markdown';
 import styled from 'styled-components';
 import { FormattedMessage } from '@kbn/i18n-react';
@@ -18,13 +18,14 @@ import {
   EuiSpacer,
   EuiButtonEmpty,
   EuiTitle,
+  EuiToolTip,
 } from '@elastic/eui';
 import { useRouteMatch } from 'react-router-dom';
 
 import { mapPackageReleaseToIntegrationCardRelease } from '../../../../../../../../services/package_prerelease';
 
 import { getRegistryDataStreamAssetBaseName } from '../../../../../../../../../common/services';
-
+import type { ExperimentalIndexingFeature } from '../../../../../../../../../common/types/models/epm';
 import type {
   NewPackagePolicy,
   NewPackagePolicyInputStream,
@@ -120,6 +121,62 @@ export const PackagePolicyInputStreamConfig = memo<Props>(
       [advancedVars, inputStreamValidationResults?.vars]
     );
 
+    const isFeatureEnabled = useCallback(
+      (feature: ExperimentalIndexingFeature) =>
+        packagePolicy.package?.experimental_data_stream_features?.some(
+          ({ data_stream: dataStream, features }) =>
+            dataStream ===
+              getRegistryDataStreamAssetBaseName(packagePolicyInputStream.data_stream) &&
+            features[feature]
+        ) ?? false,
+      [
+        packagePolicy.package?.experimental_data_stream_features,
+        packagePolicyInputStream.data_stream,
+      ]
+    );
+
+    const newExperimentalIndexingFeature = {
+      synthetic_source: isFeatureEnabled('synthetic_source'),
+      tsdb: isFeatureEnabled('tsdb'),
+    };
+
+    const onIndexingSettingChange = (
+      features: Partial<Record<ExperimentalIndexingFeature, boolean>>
+    ) => {
+      if (!packagePolicy.package) {
+        return;
+      }
+
+      const newExperimentalDataStreamFeatures = [
+        ...(packagePolicy.package.experimental_data_stream_features ?? []),
+      ];
+
+      const dataStream = getRegistryDataStreamAssetBaseName(packagePolicyInputStream.data_stream);
+
+      const existingSettingRecord = newExperimentalDataStreamFeatures.find(
+        (x) => x.data_stream === dataStream
+      );
+
+      if (existingSettingRecord) {
+        existingSettingRecord.features = {
+          ...existingSettingRecord.features,
+          ...features,
+        };
+      } else {
+        newExperimentalDataStreamFeatures.push({
+          data_stream: dataStream,
+          features: { ...newExperimentalIndexingFeature, ...features },
+        });
+      }
+
+      updatePackagePolicy({
+        package: {
+          ...packagePolicy.package,
+          experimental_data_stream_features: newExperimentalDataStreamFeatures,
+        },
+      });
+    };
+
     return (
       <>
         <EuiFlexGrid columns={2} id={isDefaultDatstream ? 'test123' : 'asas'}>
@@ -311,59 +368,46 @@ export const PackagePolicyInputStreamConfig = memo<Props>(
                         <EuiSpacer size="s" />
                         <EuiFlexItem>
                           <EuiSwitch
-                            checked={
-                              packagePolicy.package?.experimental_data_stream_features?.some(
-                                ({ data_stream: dataStream, features }) =>
-                                  dataStream ===
-                                    getRegistryDataStreamAssetBaseName(
-                                      packagePolicyInputStream.data_stream
-                                    ) && features.synthetic_source
-                              ) ?? false
-                            }
+                            checked={isFeatureEnabled('synthetic_source')}
                             label={
                               <FormattedMessage
                                 id="xpack.fleet.createPackagePolicy.experimentalFeatures.syntheticSourceLabel"
                                 defaultMessage="Synthetic source"
                               />
                             }
                             onChange={(e) => {
-                              if (!packagePolicy.package) {
-                                return;
-                              }
-
-                              const newExperimentalDataStreamFeatures = [
-                                ...(packagePolicy.package.experimental_data_stream_features ?? []),
-                              ];
-
-                              const dataStream = getRegistryDataStreamAssetBaseName(
-                                packagePolicyInputStream.data_stream
-                              );
-
-                              const existingSettingRecord = newExperimentalDataStreamFeatures.find(
-                                (x) => x.data_stream === dataStream
-                              );
-
-                              if (existingSettingRecord) {
-                                existingSettingRecord.features.synthetic_source = e.target.checked;
-                              } else {
-                                newExperimentalDataStreamFeatures.push({
-                                  data_stream: dataStream,
-                                  features: {
-                                    synthetic_source: e.target.checked,
-                                  },
-                                });
-                              }
-
-                              updatePackagePolicy({
-                                package: {
-                                  ...packagePolicy.package,
-                                  experimental_data_stream_features:
-                                    newExperimentalDataStreamFeatures,
-                                },
+                              onIndexingSettingChange({
+                                synthetic_source: e.target.checked,
                               });
                             }}
                           />
                         </EuiFlexItem>
+                        <EuiFlexItem>
+                          <EuiToolTip
+                            content={
+                              <FormattedMessage
+                                id="xpack.fleet.createPackagePolicy.experimentalFeatures.TSDBTooltip"
+                                defaultMessage="Enabling this feature is irreversible"
+                              />
+                            }
+                          >
+                            <EuiSwitch
+                              disabled={isFeatureEnabled('tsdb')}
+                              checked={isFeatureEnabled('tsdb')}
+                              label={
+                                <FormattedMessage
+                                  id="xpack.fleet.createPackagePolicy.experimentalFeatures.TSDBLabel"
+                                  defaultMessage="Time-series indexing (TSDB)"
+                                />
+                              }
+                              onChange={(e) => {
+                                onIndexingSettingChange({
+                                  tsdb: e.target.checked,
+                                });
+                              }}
+                            />
+                          </EuiToolTip>
+                        </EuiFlexItem>
                       </EuiFlexGroup>
                     </EuiFlexItem>
                   </>

x-pack/plugins/fleet/server/saved_objects/index.ts

@@ -284,6 +284,7 @@ const getSavedObjectTypes = (
               type: 'nested',
               properties: {
                 synthetic_source: { type: 'boolean' },
+                tsdb: { type: 'boolean' },
               },
             },
           },

x-pack/plugins/fleet/server/services/epm/archive/parse.test.ts

@@ -67,6 +67,11 @@ describe('parseDataStreamElasticsearchEntry', () => {
       source_mode: 'synthetic',
     });
   });
+  it('Should add index_mode', () => {
+    expect(parseDataStreamElasticsearchEntry({ index_mode: 'time_series' })).toEqual({
+      index_mode: 'time_series',
+    });
+  });
   it('Should add index_template mappings and expand dots', () => {
     expect(
       parseDataStreamElasticsearchEntry({

x-pack/plugins/fleet/server/services/epm/archive/parse.ts

@@ -539,6 +539,10 @@ export function parseDataStreamElasticsearchEntry(
     );
   }
 
+  if (expandedElasticsearch?.index_mode) {
+    parsedElasticsearchEntry.index_mode = expandedElasticsearch.index_mode;
+  }
+
   return parsedElasticsearchEntry;
 }