Added Time-series indexing (TSDB) to Integrations Experimental Indexing settings #144974
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
juliaElastic
added
the
release_note:feature
juliaElastic
changed the title
5 tasks
|
@elasticmachine merge upstream |
botelastic
bot
added
the
Team:Fleet
|
Pinging @elastic/fleet (Team:Fleet) |
nchaulet
reviewed
Nov 14, 2022
x-pack/plugins/fleet/common/services/simplified_package_policy_helper.test.ts
Outdated
Show resolved
Hide resolved
nchaulet
reviewed
Nov 14, 2022
x-pack/plugins/fleet/server/services/package_policies/experimental_datastream_features.ts
Show resolved
Hide resolved
nchaulet
reviewed
Nov 14, 2022
x-pack/plugins/fleet/server/services/package_policies/experimental_datastream_features.ts
Outdated
Show resolved
Hide resolved
juliaElastic
commented
Nov 15, 2022
| @@ -284,6 +284,7 @@ const getSavedObjectTypes = ( | |||
| type: 'nested', | |||
| properties: { | |||
| synthetic_source: { type: 'boolean' }, | |||
| tsdb: { type: 'boolean' }, | |||
There was a problem hiding this comment.
@elastic/kibana-core @pgayvallet hey, could you approve the change in core? It is due to a new property added to a fleet saved object.
pgayvallet
approved these changes
Nov 15, 2022
|
I ran into an issue, that the changes in this pr break the update of component templates. Looking for a solution. See details here: elastic/elasticsearch#74660 (comment) I raised an issue in elasticsearch repo: elastic/elasticsearch#91592 |
|
@elasticmachine merge upstream |
|
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]Public APIs missing comments
Async chunks
Public APIs missing exports
Saved Objects .kibana field count
Unknown metric groupsAPI count
ESLint disabled in files
ESLint disabled line counts
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: |
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kpollich
pushed a commit
to kpollich/kibana
that referenced
this pull request
Nov 29, 2022
…ng settings (elastic#144974) ## Summary Closes elastic#144530 Added a new option under integration / data streams / Index settings (experimental) / TSDB - [x] Add a toggle to the "Experimental indexing features" data streams UI for TSDB - Enabling this toggle adds `index.mode: time_series` to the data stream's index template settings. - Note: `index.routing_path` value was not needed to be added in this logic, because there is an existing elasticsearch automation that generates it from dimension fields, see test instructions. - [x] If the current package's manifest contains `index_mode: time_series` for a given data stream, the toggle should be in an "enabled + readonly" state, e.g. it cannot be disabled - Note: currently there is no package that has this setting enabled, I tested locally by manually modifying the package info response. Will need to figure out a way to create a mock package to test this. - [x] Once the toggle is enabled and the policy is saved, the toggle should _not_ be disable-able. Enabling TSDB for a data stream is an irreversible operation - [x] Add a tooltip to make the irreversible nature of this operation clear TSDB setting in System package: <img width="915" alt="image" src="https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png"> <img width="632" alt="image" src="https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png"> Tooltip: <img width="509" alt="image" src="https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png"> ## Test instructions ### Setup local registry - For testing, I created a new version of `System` package, that has a `dimension` field called `test_dimension`. This is needed so that the elasticsearch automation runs to generate the `routing_path` setting. - Download the test data zip: [registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip) - Start a registry: ``` docker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main ``` - Add this to your kibana config: ``` xpack.fleet.registryUrl: http://localhost:8080 ``` - Open kibana, open Add System integration page (should be version `1.20.5` coming from local registry) - Enable only `Collect metrics from System instances / System cpu metrics` stream, and enable `Time-series indexing (TSDB)` switch under `Advanced options` - Click on save integration. The integration should be added successfully to an agent policy. - Start a fleet server and enroll an agent to the previously created agent policy. Wait for the agent to be healthy, this should trigger system metrics flowing in, this way triggering the creation of the data stream. - Go to `Stack Management / Index Management / Index Templates`, and open the details of `metrics-system.cpu`, check that it has this in `Settings` tab: `index.mode: time_series` <img width="2013" alt="image" src="https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png"> - Go to `Indices` tab and look for `data_stream="metrics-system.cpu-default"` including hidden indices. You should see `routing_path` populated in Settings, and `test_dimension` in Mappings. <img width="1792" alt="image" src="https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png"> <img width="774" alt="image" src="https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png"> ### Testing scenario of enabling TSDB in package spec by default - Prepared a newer version of `system` package `v1.20.9` that has `index_mode: "time_series"` in `system.cpu` data stream in package manifest. - Download and extract the below zip and restart the local epr container to read from this folder. [registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip) - Open add system integration page - Check that `tsdb:true` is set in `Preview API request` for `system.cpu` data stream <img width="1033" alt="image" src="https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png"> - Open the system cpu metrics stream on the policy editor, and check that the TSDB switch is turned on. <img width="1093" alt="image" src="https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png"> <img width="1051" alt="image" src="https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png"> ### Update 11/16: - After latest changes, expect `routing_path` with dimension fields to be generated in Index Template when enabling TSDB: <img width="1163" alt="image" src="https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png"> This was added to fix the issue mentioned here elastic#144974 (comment) - After adding TSDB, modifying the parent component template should be successful. <img width="555" alt="image" src="https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png"> ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Kibana Machine <[email protected]> (cherry picked from commit 4608af4)
kpollich
pushed a commit
to kpollich/kibana
that referenced
this pull request
Nov 29, 2022
…ng settings (elastic#144974) ## Summary Closes elastic#144530 Added a new option under integration / data streams / Index settings (experimental) / TSDB - [x] Add a toggle to the "Experimental indexing features" data streams UI for TSDB - Enabling this toggle adds `index.mode: time_series` to the data stream's index template settings. - Note: `index.routing_path` value was not needed to be added in this logic, because there is an existing elasticsearch automation that generates it from dimension fields, see test instructions. - [x] If the current package's manifest contains `index_mode: time_series` for a given data stream, the toggle should be in an "enabled + readonly" state, e.g. it cannot be disabled - Note: currently there is no package that has this setting enabled, I tested locally by manually modifying the package info response. Will need to figure out a way to create a mock package to test this. - [x] Once the toggle is enabled and the policy is saved, the toggle should _not_ be disable-able. Enabling TSDB for a data stream is an irreversible operation - [x] Add a tooltip to make the irreversible nature of this operation clear TSDB setting in System package: <img width="915" alt="image" src="https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png"> <img width="632" alt="image" src="https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png"> Tooltip: <img width="509" alt="image" src="https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png"> ## Test instructions ### Setup local registry - For testing, I created a new version of `System` package, that has a `dimension` field called `test_dimension`. This is needed so that the elasticsearch automation runs to generate the `routing_path` setting. - Download the test data zip: [registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip) - Start a registry: ``` docker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main ``` - Add this to your kibana config: ``` xpack.fleet.registryUrl: http://localhost:8080 ``` - Open kibana, open Add System integration page (should be version `1.20.5` coming from local registry) - Enable only `Collect metrics from System instances / System cpu metrics` stream, and enable `Time-series indexing (TSDB)` switch under `Advanced options` - Click on save integration. The integration should be added successfully to an agent policy. - Start a fleet server and enroll an agent to the previously created agent policy. Wait for the agent to be healthy, this should trigger system metrics flowing in, this way triggering the creation of the data stream. - Go to `Stack Management / Index Management / Index Templates`, and open the details of `metrics-system.cpu`, check that it has this in `Settings` tab: `index.mode: time_series` <img width="2013" alt="image" src="https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png"> - Go to `Indices` tab and look for `data_stream="metrics-system.cpu-default"` including hidden indices. You should see `routing_path` populated in Settings, and `test_dimension` in Mappings. <img width="1792" alt="image" src="https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png"> <img width="774" alt="image" src="https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png"> ### Testing scenario of enabling TSDB in package spec by default - Prepared a newer version of `system` package `v1.20.9` that has `index_mode: "time_series"` in `system.cpu` data stream in package manifest. - Download and extract the below zip and restart the local epr container to read from this folder. [registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip) - Open add system integration page - Check that `tsdb:true` is set in `Preview API request` for `system.cpu` data stream <img width="1033" alt="image" src="https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png"> - Open the system cpu metrics stream on the policy editor, and check that the TSDB switch is turned on. <img width="1093" alt="image" src="https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png"> <img width="1051" alt="image" src="https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png"> ### Update 11/16: - After latest changes, expect `routing_path` with dimension fields to be generated in Index Template when enabling TSDB: <img width="1163" alt="image" src="https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png"> This was added to fix the issue mentioned here elastic#144974 (comment) - After adding TSDB, modifying the parent component template should be successful. <img width="555" alt="image" src="https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png"> ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Kibana Machine <[email protected]> (cherry picked from commit 4608af4)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kpollich
added a commit
that referenced
this pull request
Nov 29, 2022
…Indexing settings (#144974) (#146562) # Backport This will backport the following commits from `main` to `8.6`: - [Added Time-series indexing (TSDB) to Integrations Experimental Indexing settings (#144974)](#144974) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Julia Bardi","email":"[email protected]"},"sourceCommit":{"committedDate":"2022-11-17T09:13:20Z","message":"Added Time-series indexing (TSDB) to Integrations Experimental Indexing settings (#144974)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/144530\r\n\r\nAdded a new option under integration / data streams / Index settings\r\n(experimental) / TSDB\r\n\r\n- [x] Add a toggle to the \"Experimental indexing features\" data streams\r\nUI for TSDB\r\n- Enabling this toggle adds `index.mode: time_series` to the data\r\nstream's index template settings.\r\n- Note: `index.routing_path` value was not needed to be added in this\r\nlogic, because there is an existing elasticsearch automation that\r\ngenerates it from dimension fields, see test instructions.\r\n- [x] If the current package's manifest contains `index_mode:\r\ntime_series` for a given data stream, the toggle should be in an\r\n\"enabled + readonly\" state, e.g. it cannot be disabled\r\n- Note: currently there is no package that has this setting enabled, I\r\ntested locally by manually modifying the package info response. Will\r\nneed to figure out a way to create a mock package to test this.\r\n- [x] Once the toggle is enabled and the policy is saved, the toggle\r\nshould _not_ be disable-able. Enabling TSDB for a data stream is an\r\nirreversible operation\r\n- [x] Add a tooltip to make the irreversible nature of this operation\r\nclear\r\n\r\nTSDB setting in System package:\r\n<img width=\"915\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png\">\r\n<img width=\"632\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png\">\r\n\r\nTooltip:\r\n<img width=\"509\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png\">\r\n\r\n\r\n## Test instructions\r\n\r\n### Setup local registry\r\n\r\n- For testing, I created a new version of `System` package, that has a\r\n`dimension` field called `test_dimension`. This is needed so that the\r\nelasticsearch automation runs to generate the `routing_path` setting.\r\n\r\n- Download the test data zip:\r\n\r\n[registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip)\r\n\r\n\r\n- Start a registry:\r\n```\r\ndocker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main\r\n```\r\n- Add this to your kibana config:\r\n```\r\nxpack.fleet.registryUrl: http://localhost:8080\r\n```\r\n\r\n- Open kibana, open Add System integration page (should be version\r\n`1.20.5` coming from local registry)\r\n- Enable only `Collect metrics from System instances / System cpu\r\nmetrics` stream, and enable `Time-series indexing (TSDB)` switch under\r\n`Advanced options`\r\n- Click on save integration. The integration should be added\r\nsuccessfully to an agent policy.\r\n- Start a fleet server and enroll an agent to the previously created\r\nagent policy. Wait for the agent to be healthy, this should trigger\r\nsystem metrics flowing in, this way triggering the creation of the data\r\nstream.\r\n- Go to `Stack Management / Index Management / Index Templates`, and\r\nopen the details of `metrics-system.cpu`, check that it has this in\r\n`Settings` tab: `index.mode: time_series`\r\n\r\n<img width=\"2013\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png\">\r\n\r\n- Go to `Indices` tab and look for\r\n`data_stream=\"metrics-system.cpu-default\"` including hidden indices. You\r\nshould see `routing_path` populated in Settings, and `test_dimension` in\r\nMappings.\r\n\r\n<img width=\"1792\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png\">\r\n<img width=\"774\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png\">\r\n\r\n### Testing scenario of enabling TSDB in package spec by default\r\n\r\n- Prepared a newer version of `system` package `v1.20.9` that has\r\n`index_mode: \"time_series\"` in `system.cpu` data stream in package\r\nmanifest.\r\n- Download and extract the below zip and restart the local epr container\r\nto read from this folder.\r\n\r\n[registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip)\r\n- Open add system integration page\r\n- Check that `tsdb:true` is set in `Preview API request` for\r\n`system.cpu` data stream\r\n\r\n<img width=\"1033\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png\">\r\n\r\n- Open the system cpu metrics stream on the policy editor, and check\r\nthat the TSDB switch is turned on.\r\n<img width=\"1093\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png\">\r\n<img width=\"1051\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png\">\r\n\r\n### Update 11/16:\r\n\r\n- After latest changes, expect `routing_path` with dimension fields to\r\nbe generated in Index Template when enabling TSDB:\r\n\r\n<img width=\"1163\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png\">\r\n\r\nThis was added to fix the issue mentioned here\r\nhttps://github.com//pull/144974#issuecomment-1315270890\r\n\r\n- After adding TSDB, modifying the parent component template should be\r\nsuccessful.\r\n\r\n<img width=\"555\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png\">\r\n\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"4608af41da7dc98403bbdbfba8a3ac02475277e4","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["backport:skip","Team:Fleet","release_note:feature","v8.7.0"],"number":144974,"url":"https://github.com/elastic/kibana/pull/144974","mergeCommit":{"message":"Added Time-series indexing (TSDB) to Integrations Experimental Indexing settings (#144974)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/144530\r\n\r\nAdded a new option under integration / data streams / Index settings\r\n(experimental) / TSDB\r\n\r\n- [x] Add a toggle to the \"Experimental indexing features\" data streams\r\nUI for TSDB\r\n- Enabling this toggle adds `index.mode: time_series` to the data\r\nstream's index template settings.\r\n- Note: `index.routing_path` value was not needed to be added in this\r\nlogic, because there is an existing elasticsearch automation that\r\ngenerates it from dimension fields, see test instructions.\r\n- [x] If the current package's manifest contains `index_mode:\r\ntime_series` for a given data stream, the toggle should be in an\r\n\"enabled + readonly\" state, e.g. it cannot be disabled\r\n- Note: currently there is no package that has this setting enabled, I\r\ntested locally by manually modifying the package info response. Will\r\nneed to figure out a way to create a mock package to test this.\r\n- [x] Once the toggle is enabled and the policy is saved, the toggle\r\nshould _not_ be disable-able. Enabling TSDB for a data stream is an\r\nirreversible operation\r\n- [x] Add a tooltip to make the irreversible nature of this operation\r\nclear\r\n\r\nTSDB setting in System package:\r\n<img width=\"915\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png\">\r\n<img width=\"632\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png\">\r\n\r\nTooltip:\r\n<img width=\"509\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png\">\r\n\r\n\r\n## Test instructions\r\n\r\n### Setup local registry\r\n\r\n- For testing, I created a new version of `System` package, that has a\r\n`dimension` field called `test_dimension`. This is needed so that the\r\nelasticsearch automation runs to generate the `routing_path` setting.\r\n\r\n- Download the test data zip:\r\n\r\n[registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip)\r\n\r\n\r\n- Start a registry:\r\n```\r\ndocker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main\r\n```\r\n- Add this to your kibana config:\r\n```\r\nxpack.fleet.registryUrl: http://localhost:8080\r\n```\r\n\r\n- Open kibana, open Add System integration page (should be version\r\n`1.20.5` coming from local registry)\r\n- Enable only `Collect metrics from System instances / System cpu\r\nmetrics` stream, and enable `Time-series indexing (TSDB)` switch under\r\n`Advanced options`\r\n- Click on save integration. The integration should be added\r\nsuccessfully to an agent policy.\r\n- Start a fleet server and enroll an agent to the previously created\r\nagent policy. Wait for the agent to be healthy, this should trigger\r\nsystem metrics flowing in, this way triggering the creation of the data\r\nstream.\r\n- Go to `Stack Management / Index Management / Index Templates`, and\r\nopen the details of `metrics-system.cpu`, check that it has this in\r\n`Settings` tab: `index.mode: time_series`\r\n\r\n<img width=\"2013\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png\">\r\n\r\n- Go to `Indices` tab and look for\r\n`data_stream=\"metrics-system.cpu-default\"` including hidden indices. You\r\nshould see `routing_path` populated in Settings, and `test_dimension` in\r\nMappings.\r\n\r\n<img width=\"1792\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png\">\r\n<img width=\"774\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png\">\r\n\r\n### Testing scenario of enabling TSDB in package spec by default\r\n\r\n- Prepared a newer version of `system` package `v1.20.9` that has\r\n`index_mode: \"time_series\"` in `system.cpu` data stream in package\r\nmanifest.\r\n- Download and extract the below zip and restart the local epr container\r\nto read from this folder.\r\n\r\n[registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip)\r\n- Open add system integration page\r\n- Check that `tsdb:true` is set in `Preview API request` for\r\n`system.cpu` data stream\r\n\r\n<img width=\"1033\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png\">\r\n\r\n- Open the system cpu metrics stream on the policy editor, and check\r\nthat the TSDB switch is turned on.\r\n<img width=\"1093\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png\">\r\n<img width=\"1051\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png\">\r\n\r\n### Update 11/16:\r\n\r\n- After latest changes, expect `routing_path` with dimension fields to\r\nbe generated in Index Template when enabling TSDB:\r\n\r\n<img width=\"1163\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png\">\r\n\r\nThis was added to fix the issue mentioned here\r\nhttps://github.com//pull/144974#issuecomment-1315270890\r\n\r\n- After adding TSDB, modifying the parent component template should be\r\nsuccessful.\r\n\r\n<img width=\"555\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png\">\r\n\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"4608af41da7dc98403bbdbfba8a3ac02475277e4"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/144974","number":144974,"mergeCommit":{"message":"Added Time-series indexing (TSDB) to Integrations Experimental Indexing settings (#144974)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/144530\r\n\r\nAdded a new option under integration / data streams / Index settings\r\n(experimental) / TSDB\r\n\r\n- [x] Add a toggle to the \"Experimental indexing features\" data streams\r\nUI for TSDB\r\n- Enabling this toggle adds `index.mode: time_series` to the data\r\nstream's index template settings.\r\n- Note: `index.routing_path` value was not needed to be added in this\r\nlogic, because there is an existing elasticsearch automation that\r\ngenerates it from dimension fields, see test instructions.\r\n- [x] If the current package's manifest contains `index_mode:\r\ntime_series` for a given data stream, the toggle should be in an\r\n\"enabled + readonly\" state, e.g. it cannot be disabled\r\n- Note: currently there is no package that has this setting enabled, I\r\ntested locally by manually modifying the package info response. Will\r\nneed to figure out a way to create a mock package to test this.\r\n- [x] Once the toggle is enabled and the policy is saved, the toggle\r\nshould _not_ be disable-able. Enabling TSDB for a data stream is an\r\nirreversible operation\r\n- [x] Add a tooltip to make the irreversible nature of this operation\r\nclear\r\n\r\nTSDB setting in System package:\r\n<img width=\"915\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png\">\r\n<img width=\"632\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png\">\r\n\r\nTooltip:\r\n<img width=\"509\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png\">\r\n\r\n\r\n## Test instructions\r\n\r\n### Setup local registry\r\n\r\n- For testing, I created a new version of `System` package, that has a\r\n`dimension` field called `test_dimension`. This is needed so that the\r\nelasticsearch automation runs to generate the `routing_path` setting.\r\n\r\n- Download the test data zip:\r\n\r\n[registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip)\r\n\r\n\r\n- Start a registry:\r\n```\r\ndocker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main\r\n```\r\n- Add this to your kibana config:\r\n```\r\nxpack.fleet.registryUrl: http://localhost:8080\r\n```\r\n\r\n- Open kibana, open Add System integration page (should be version\r\n`1.20.5` coming from local registry)\r\n- Enable only `Collect metrics from System instances / System cpu\r\nmetrics` stream, and enable `Time-series indexing (TSDB)` switch under\r\n`Advanced options`\r\n- Click on save integration. The integration should be added\r\nsuccessfully to an agent policy.\r\n- Start a fleet server and enroll an agent to the previously created\r\nagent policy. Wait for the agent to be healthy, this should trigger\r\nsystem metrics flowing in, this way triggering the creation of the data\r\nstream.\r\n- Go to `Stack Management / Index Management / Index Templates`, and\r\nopen the details of `metrics-system.cpu`, check that it has this in\r\n`Settings` tab: `index.mode: time_series`\r\n\r\n<img width=\"2013\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png\">\r\n\r\n- Go to `Indices` tab and look for\r\n`data_stream=\"metrics-system.cpu-default\"` including hidden indices. You\r\nshould see `routing_path` populated in Settings, and `test_dimension` in\r\nMappings.\r\n\r\n<img width=\"1792\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png\">\r\n<img width=\"774\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png\">\r\n\r\n### Testing scenario of enabling TSDB in package spec by default\r\n\r\n- Prepared a newer version of `system` package `v1.20.9` that has\r\n`index_mode: \"time_series\"` in `system.cpu` data stream in package\r\nmanifest.\r\n- Download and extract the below zip and restart the local epr container\r\nto read from this folder.\r\n\r\n[registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip)\r\n- Open add system integration page\r\n- Check that `tsdb:true` is set in `Preview API request` for\r\n`system.cpu` data stream\r\n\r\n<img width=\"1033\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png\">\r\n\r\n- Open the system cpu metrics stream on the policy editor, and check\r\nthat the TSDB switch is turned on.\r\n<img width=\"1093\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png\">\r\n<img width=\"1051\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png\">\r\n\r\n### Update 11/16:\r\n\r\n- After latest changes, expect `routing_path` with dimension fields to\r\nbe generated in Index Template when enabling TSDB:\r\n\r\n<img width=\"1163\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png\">\r\n\r\nThis was added to fix the issue mentioned here\r\nhttps://github.com//pull/144974#issuecomment-1315270890\r\n\r\n- After adding TSDB, modifying the parent component template should be\r\nsuccessful.\r\n\r\n<img width=\"555\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png\">\r\n\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"4608af41da7dc98403bbdbfba8a3ac02475277e4"}}]}] BACKPORT--> Co-authored-by: Julia Bardi <[email protected]>
![https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png\">\r\n<img](https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png\">\r\n<img){kind=link}
![https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png\">\r\n\r\nTooltip:\r\n<img](https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png\">\r\n\r\nTooltip:\r\n<img){kind=link}
![https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png\">\r\n\r\n\r\n##](https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png\">\r\n\r\n\r\n##){kind=link}
![https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png\">\r\n\r\n-](https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png\">\r\n\r\n-){kind=link}
![https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png\">\r\n<img](https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png\">\r\n<img){kind=link}
![https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png\">\r\n\r\n###](https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png\">\r\n\r\n###){kind=link}
![https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png\">\r\n\r\n-](https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png\">\r\n\r\n-){kind=link}
![https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png\">\r\n<img](https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png\">\r\n<img){kind=link}
![https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png\">\r\n\r\n###](https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png\">\r\n\r\n###){kind=link}
![https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png\">\r\n\r\nThis](https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png\">\r\n\r\nThis){kind=link}
![https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png\">\r\n\r\n\r\n###](https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png\">\r\n\r\n\r\n###){kind=link}
jillguyonnet
added a commit
that referenced
this pull request
Apr 12, 2023
### Summary Disable `Time-series indexing (TSDB)` switch in experimental datastream settings if TSDB is enabled in the package itself. As part of this fix, the following changes are included in compliance with UX rules: * Rename toggle label from `Time-series indexing (TSDB)` to `Time-series database (TSDB) indexing`. * When the toggle is disabled, add a question mark icon with a tooltip with text `TSDB indexing is enabled by the integration`. Closes #152778 ### Screenshots #### Before  #### After When the integration does NOT enable TSDB:  When the integration DOES enable TSDB:  ### Steps to test locally Refer to the [original PR](#144974) for extended context. #### Testing a package without TSDB 1. Run Kibana on this branch. 2. Add the System integration (version 1.5.2 at the time of writing); under the `Collect metrics from System instances -> System cpu metrics -> Advanced options` settings, the `Time-series indexing (TSDB)` switch should be off and enabled (you should be able to toggle it on and off). #### Testing a package with TSDB 1. Update the system integration to use TSDB and run a local package registry: 1. In the integrations repo, bump the patch version of the system integration: add a changelog entry (1.25.3) and update the version accordingly in the package manifest. 2. In the manifest of the cpu data stream, add the following to enable TSDB (see [this doc](https://github.com/elastic/integrations/blob/2abe6f635a27996479fee83203149ba2236dec7a/docs/developer_tsdb_migration_guidelines.md) for context): ``` elasticsearch: index_mode: "time_series" ``` 3. Build the system integration: `cd packages/system && elastic-package build` 4. Run a local package registry: `elastic-package stack up -d -v --services package-registry` 5. Check at https://localhost:8080/search?package=system that the version is correct (1.5.3). 4. Run Kibana on this branch to test the change: 1. In your `kibana.dev.yml`, add `xpack.fleet.registryUrl: https://localhost:8080` if not there already. 2. Also in `kibana.dev.yml`, add `xpack.fleet.enableExperimental: ['experimentalDataStreamSettings']` if not there already. 6. Before running `yarn start`, run `export NODE_EXTRA_CA_CERTS=$HOME/.elastic-package/profiles/default/certs/kibana/ca-cert.pem` in the same shell. This is to set up the certificate needed to access EPR with https. 7. In Kibana, add the System integration (check that the version is correct); under the `Collect metrics from System instances -> System cpu metrics -> Advanced options` settings, the `Time-series indexing (TSDB)` switch should be on and disabled. There should be a question mark icon with a tooltip. --------- Co-authored-by: kibanamachine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Closes #144530
Added a new option under integration / data streams / Index settings (experimental) / TSDB
index.mode: time_seriesto the data stream's index template settings.index.routing_pathvalue was not needed to be added in this logic, because there is an existing elasticsearch automation that generates it from dimension fields, see test instructions.index_mode: time_seriesfor a given data stream, the toggle should be in an "enabled + readonly" state, e.g. it cannot be disabledTSDB setting in System package:
Tooltip:
Test instructions
Setup local registry
For testing, I created a new version of
Systempackage, that has adimensionfield calledtest_dimension. This is needed so that the elasticsearch automation runs to generate therouting_pathsetting.Download the test data zip:
registry-packages.zip
Start a registry:
1.20.5coming from local registry)Collect metrics from System instances / System cpu metricsstream, and enableTime-series indexing (TSDB)switch underAdvanced optionsStack Management / Index Management / Index Templates, and open the details ofmetrics-system.cpu, check that it has this inSettingstab:index.mode: time_seriesIndicestab and look fordata_stream="metrics-system.cpu-default"including hidden indices. You should seerouting_pathpopulated in Settings, andtest_dimensionin Mappings.Testing scenario of enabling TSDB in package spec by default
systempackagev1.20.9that hasindex_mode: "time_series"insystem.cpudata stream in package manifest.registry-packages.zip
tsdb:trueis set inPreview API requestforsystem.cpudata streamUpdate 11/16:
routing_pathwith dimension fields to be generated in Index Template when enabling TSDB:This was added to fix the issue mentioned here #144974 (comment)
Update 3/13/2023
routing_pathis being generated by elasticsearch, and the logic in kibana has been removed.Checklist