[Fleet] Add experimental toggle + support package spec field for TSDB #144530
Assignees
Labels
QA:Validated
Issue has been validated by QA
Team:Fleet
Team label for Observability Data Collection Fleet team
Comments
kpollich
added
the
Team:Fleet
|
Pinging @elastic/fleet (Team:Fleet) |
|
Is there any package having I also don’t find any examples for Or is it the same as EDIT: as discussed on slack, |
|
@jlind23 we discussed on the chat yesterday and I posted a question to the elasticsearch team about how the |
6 tasks
|
Hi @juliaElastic Could you please confirm:
CC: @kpollich |
|
@simranvaseer-qasource This is an extension of an existing feature, experimental indexing options that was introduced here with synthetic source. I'll document detailed steps how to verify the new TSDB toggle. |
|
@simranvaseer-qasource see detailed test instructions here #144974 |
@jsoriano I tried to add Added the setting below here: EDIT: found that the missing field was due to the package archive parse logic didn't map it: fc64c1b#diff-232dc9fd708e4f1495f59c12729834fecd84bd8af70d4c21363da14e79aa30c1 |
|
One more question:
cc @kpollich I have tested this scenario:
|
|
Thank you for providing the testing details. We will test this feature once it is available and will be creating test cases for the same. Thanks! |
I would expect a behaviour similar to the one when custom mappings are added. What happens in this case? Or, I would expect the user to be asked with a confirmation dialog if they want Fleet to make the rollover or they want to do it themselves. |
I checked and I found on the custom component template edit page there is a text saying "Datastreams using that template need to be updated with aditionnal requests."
Good idea, I'll leave this open for discussion, it can be changed in follow up issues/prs. |
juliaElastic
added a commit
that referenced
this issue
Nov 17, 2022
…ng settings (#144974) ## Summary Closes #144530 Added a new option under integration / data streams / Index settings (experimental) / TSDB - [x] Add a toggle to the "Experimental indexing features" data streams UI for TSDB - Enabling this toggle adds `index.mode: time_series` to the data stream's index template settings. - Note: `index.routing_path` value was not needed to be added in this logic, because there is an existing elasticsearch automation that generates it from dimension fields, see test instructions. - [x] If the current package's manifest contains `index_mode: time_series` for a given data stream, the toggle should be in an "enabled + readonly" state, e.g. it cannot be disabled - Note: currently there is no package that has this setting enabled, I tested locally by manually modifying the package info response. Will need to figure out a way to create a mock package to test this. - [x] Once the toggle is enabled and the policy is saved, the toggle should _not_ be disable-able. Enabling TSDB for a data stream is an irreversible operation - [x] Add a tooltip to make the irreversible nature of this operation clear TSDB setting in System package: <img width="915" alt="image" src="https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png"> <img width="632" alt="image" src="https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png"> Tooltip: <img width="509" alt="image" src="https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png"> ## Test instructions ### Setup local registry - For testing, I created a new version of `System` package, that has a `dimension` field called `test_dimension`. This is needed so that the elasticsearch automation runs to generate the `routing_path` setting. - Download the test data zip: [registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip) - Start a registry: ``` docker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main ``` - Add this to your kibana config: ``` xpack.fleet.registryUrl: http://localhost:8080 ``` - Open kibana, open Add System integration page (should be version `1.20.5` coming from local registry) - Enable only `Collect metrics from System instances / System cpu metrics` stream, and enable `Time-series indexing (TSDB)` switch under `Advanced options` - Click on save integration. The integration should be added successfully to an agent policy. - Start a fleet server and enroll an agent to the previously created agent policy. Wait for the agent to be healthy, this should trigger system metrics flowing in, this way triggering the creation of the data stream. - Go to `Stack Management / Index Management / Index Templates`, and open the details of `metrics-system.cpu`, check that it has this in `Settings` tab: `index.mode: time_series` <img width="2013" alt="image" src="https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png"> - Go to `Indices` tab and look for `data_stream="metrics-system.cpu-default"` including hidden indices. You should see `routing_path` populated in Settings, and `test_dimension` in Mappings. <img width="1792" alt="image" src="https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png"> <img width="774" alt="image" src="https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png"> ### Testing scenario of enabling TSDB in package spec by default - Prepared a newer version of `system` package `v1.20.9` that has `index_mode: "time_series"` in `system.cpu` data stream in package manifest. - Download and extract the below zip and restart the local epr container to read from this folder. [registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip) - Open add system integration page - Check that `tsdb:true` is set in `Preview API request` for `system.cpu` data stream <img width="1033" alt="image" src="https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png"> - Open the system cpu metrics stream on the policy editor, and check that the TSDB switch is turned on. <img width="1093" alt="image" src="https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png"> <img width="1051" alt="image" src="https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png"> ### Update 11/16: - After latest changes, expect `routing_path` with dimension fields to be generated in Index Template when enabling TSDB: <img width="1163" alt="image" src="https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png"> This was added to fix the issue mentioned here #144974 (comment) - After adding TSDB, modifying the parent component template should be successful. <img width="555" alt="image" src="https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png"> ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Kibana Machine <[email protected]>
kpollich
pushed a commit
to kpollich/kibana
that referenced
this issue
Nov 29, 2022
…ng settings (elastic#144974) ## Summary Closes elastic#144530 Added a new option under integration / data streams / Index settings (experimental) / TSDB - [x] Add a toggle to the "Experimental indexing features" data streams UI for TSDB - Enabling this toggle adds `index.mode: time_series` to the data stream's index template settings. - Note: `index.routing_path` value was not needed to be added in this logic, because there is an existing elasticsearch automation that generates it from dimension fields, see test instructions. - [x] If the current package's manifest contains `index_mode: time_series` for a given data stream, the toggle should be in an "enabled + readonly" state, e.g. it cannot be disabled - Note: currently there is no package that has this setting enabled, I tested locally by manually modifying the package info response. Will need to figure out a way to create a mock package to test this. - [x] Once the toggle is enabled and the policy is saved, the toggle should _not_ be disable-able. Enabling TSDB for a data stream is an irreversible operation - [x] Add a tooltip to make the irreversible nature of this operation clear TSDB setting in System package: <img width="915" alt="image" src="https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png"> <img width="632" alt="image" src="https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png"> Tooltip: <img width="509" alt="image" src="https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png"> ## Test instructions ### Setup local registry - For testing, I created a new version of `System` package, that has a `dimension` field called `test_dimension`. This is needed so that the elasticsearch automation runs to generate the `routing_path` setting. - Download the test data zip: [registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip) - Start a registry: ``` docker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main ``` - Add this to your kibana config: ``` xpack.fleet.registryUrl: http://localhost:8080 ``` - Open kibana, open Add System integration page (should be version `1.20.5` coming from local registry) - Enable only `Collect metrics from System instances / System cpu metrics` stream, and enable `Time-series indexing (TSDB)` switch under `Advanced options` - Click on save integration. The integration should be added successfully to an agent policy. - Start a fleet server and enroll an agent to the previously created agent policy. Wait for the agent to be healthy, this should trigger system metrics flowing in, this way triggering the creation of the data stream. - Go to `Stack Management / Index Management / Index Templates`, and open the details of `metrics-system.cpu`, check that it has this in `Settings` tab: `index.mode: time_series` <img width="2013" alt="image" src="https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png"> - Go to `Indices` tab and look for `data_stream="metrics-system.cpu-default"` including hidden indices. You should see `routing_path` populated in Settings, and `test_dimension` in Mappings. <img width="1792" alt="image" src="https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png"> <img width="774" alt="image" src="https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png"> ### Testing scenario of enabling TSDB in package spec by default - Prepared a newer version of `system` package `v1.20.9` that has `index_mode: "time_series"` in `system.cpu` data stream in package manifest. - Download and extract the below zip and restart the local epr container to read from this folder. [registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip) - Open add system integration page - Check that `tsdb:true` is set in `Preview API request` for `system.cpu` data stream <img width="1033" alt="image" src="https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png"> - Open the system cpu metrics stream on the policy editor, and check that the TSDB switch is turned on. <img width="1093" alt="image" src="https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png"> <img width="1051" alt="image" src="https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png"> ### Update 11/16: - After latest changes, expect `routing_path` with dimension fields to be generated in Index Template when enabling TSDB: <img width="1163" alt="image" src="https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png"> This was added to fix the issue mentioned here elastic#144974 (comment) - After adding TSDB, modifying the parent component template should be successful. <img width="555" alt="image" src="https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png"> ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Kibana Machine <[email protected]> (cherry picked from commit 4608af4)
kpollich
pushed a commit
to kpollich/kibana
that referenced
this issue
Nov 29, 2022
…ng settings (elastic#144974) ## Summary Closes elastic#144530 Added a new option under integration / data streams / Index settings (experimental) / TSDB - [x] Add a toggle to the "Experimental indexing features" data streams UI for TSDB - Enabling this toggle adds `index.mode: time_series` to the data stream's index template settings. - Note: `index.routing_path` value was not needed to be added in this logic, because there is an existing elasticsearch automation that generates it from dimension fields, see test instructions. - [x] If the current package's manifest contains `index_mode: time_series` for a given data stream, the toggle should be in an "enabled + readonly" state, e.g. it cannot be disabled - Note: currently there is no package that has this setting enabled, I tested locally by manually modifying the package info response. Will need to figure out a way to create a mock package to test this. - [x] Once the toggle is enabled and the policy is saved, the toggle should _not_ be disable-able. Enabling TSDB for a data stream is an irreversible operation - [x] Add a tooltip to make the irreversible nature of this operation clear TSDB setting in System package: <img width="915" alt="image" src="https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png"> <img width="632" alt="image" src="https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png"> Tooltip: <img width="509" alt="image" src="https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png"> ## Test instructions ### Setup local registry - For testing, I created a new version of `System` package, that has a `dimension` field called `test_dimension`. This is needed so that the elasticsearch automation runs to generate the `routing_path` setting. - Download the test data zip: [registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip) - Start a registry: ``` docker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main ``` - Add this to your kibana config: ``` xpack.fleet.registryUrl: http://localhost:8080 ``` - Open kibana, open Add System integration page (should be version `1.20.5` coming from local registry) - Enable only `Collect metrics from System instances / System cpu metrics` stream, and enable `Time-series indexing (TSDB)` switch under `Advanced options` - Click on save integration. The integration should be added successfully to an agent policy. - Start a fleet server and enroll an agent to the previously created agent policy. Wait for the agent to be healthy, this should trigger system metrics flowing in, this way triggering the creation of the data stream. - Go to `Stack Management / Index Management / Index Templates`, and open the details of `metrics-system.cpu`, check that it has this in `Settings` tab: `index.mode: time_series` <img width="2013" alt="image" src="https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png"> - Go to `Indices` tab and look for `data_stream="metrics-system.cpu-default"` including hidden indices. You should see `routing_path` populated in Settings, and `test_dimension` in Mappings. <img width="1792" alt="image" src="https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png"> <img width="774" alt="image" src="https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png"> ### Testing scenario of enabling TSDB in package spec by default - Prepared a newer version of `system` package `v1.20.9` that has `index_mode: "time_series"` in `system.cpu` data stream in package manifest. - Download and extract the below zip and restart the local epr container to read from this folder. [registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip) - Open add system integration page - Check that `tsdb:true` is set in `Preview API request` for `system.cpu` data stream <img width="1033" alt="image" src="https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png"> - Open the system cpu metrics stream on the policy editor, and check that the TSDB switch is turned on. <img width="1093" alt="image" src="https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png"> <img width="1051" alt="image" src="https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png"> ### Update 11/16: - After latest changes, expect `routing_path` with dimension fields to be generated in Index Template when enabling TSDB: <img width="1163" alt="image" src="https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png"> This was added to fix the issue mentioned here elastic#144974 (comment) - After adding TSDB, modifying the parent component template should be successful. <img width="555" alt="image" src="https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png"> ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Kibana Machine <[email protected]> (cherry picked from commit 4608af4)
kpollich
referenced
this issue
Nov 29, 2022
…Indexing settings (#144974) (#146562) # Backport This will backport the following commits from `main` to `8.6`: - [Added Time-series indexing (TSDB) to Integrations Experimental Indexing settings (#144974)](#144974) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Julia Bardi","email":"[email protected]"},"sourceCommit":{"committedDate":"2022-11-17T09:13:20Z","message":"Added Time-series indexing (TSDB) to Integrations Experimental Indexing settings (#144974)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/144530\r\n\r\nAdded a new option under integration / data streams / Index settings\r\n(experimental) / TSDB\r\n\r\n- [x] Add a toggle to the \"Experimental indexing features\" data streams\r\nUI for TSDB\r\n- Enabling this toggle adds `index.mode: time_series` to the data\r\nstream's index template settings.\r\n- Note: `index.routing_path` value was not needed to be added in this\r\nlogic, because there is an existing elasticsearch automation that\r\ngenerates it from dimension fields, see test instructions.\r\n- [x] If the current package's manifest contains `index_mode:\r\ntime_series` for a given data stream, the toggle should be in an\r\n\"enabled + readonly\" state, e.g. it cannot be disabled\r\n- Note: currently there is no package that has this setting enabled, I\r\ntested locally by manually modifying the package info response. Will\r\nneed to figure out a way to create a mock package to test this.\r\n- [x] Once the toggle is enabled and the policy is saved, the toggle\r\nshould _not_ be disable-able. Enabling TSDB for a data stream is an\r\nirreversible operation\r\n- [x] Add a tooltip to make the irreversible nature of this operation\r\nclear\r\n\r\nTSDB setting in System package:\r\n<img width=\"915\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png\">\r\n<img width=\"632\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png\">\r\n\r\nTooltip:\r\n<img width=\"509\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png\">\r\n\r\n\r\n## Test instructions\r\n\r\n### Setup local registry\r\n\r\n- For testing, I created a new version of `System` package, that has a\r\n`dimension` field called `test_dimension`. This is needed so that the\r\nelasticsearch automation runs to generate the `routing_path` setting.\r\n\r\n- Download the test data zip:\r\n\r\n[registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip)\r\n\r\n\r\n- Start a registry:\r\n```\r\ndocker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main\r\n```\r\n- Add this to your kibana config:\r\n```\r\nxpack.fleet.registryUrl: http://localhost:8080\r\n```\r\n\r\n- Open kibana, open Add System integration page (should be version\r\n`1.20.5` coming from local registry)\r\n- Enable only `Collect metrics from System instances / System cpu\r\nmetrics` stream, and enable `Time-series indexing (TSDB)` switch under\r\n`Advanced options`\r\n- Click on save integration. The integration should be added\r\nsuccessfully to an agent policy.\r\n- Start a fleet server and enroll an agent to the previously created\r\nagent policy. Wait for the agent to be healthy, this should trigger\r\nsystem metrics flowing in, this way triggering the creation of the data\r\nstream.\r\n- Go to `Stack Management / Index Management / Index Templates`, and\r\nopen the details of `metrics-system.cpu`, check that it has this in\r\n`Settings` tab: `index.mode: time_series`\r\n\r\n<img width=\"2013\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png\">\r\n\r\n- Go to `Indices` tab and look for\r\n`data_stream=\"metrics-system.cpu-default\"` including hidden indices. You\r\nshould see `routing_path` populated in Settings, and `test_dimension` in\r\nMappings.\r\n\r\n<img width=\"1792\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png\">\r\n<img width=\"774\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png\">\r\n\r\n### Testing scenario of enabling TSDB in package spec by default\r\n\r\n- Prepared a newer version of `system` package `v1.20.9` that has\r\n`index_mode: \"time_series\"` in `system.cpu` data stream in package\r\nmanifest.\r\n- Download and extract the below zip and restart the local epr container\r\nto read from this folder.\r\n\r\n[registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip)\r\n- Open add system integration page\r\n- Check that `tsdb:true` is set in `Preview API request` for\r\n`system.cpu` data stream\r\n\r\n<img width=\"1033\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png\">\r\n\r\n- Open the system cpu metrics stream on the policy editor, and check\r\nthat the TSDB switch is turned on.\r\n<img width=\"1093\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png\">\r\n<img width=\"1051\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png\">\r\n\r\n### Update 11/16:\r\n\r\n- After latest changes, expect `routing_path` with dimension fields to\r\nbe generated in Index Template when enabling TSDB:\r\n\r\n<img width=\"1163\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png\">\r\n\r\nThis was added to fix the issue mentioned here\r\nhttps://github.com//pull/144974#issuecomment-1315270890\r\n\r\n- After adding TSDB, modifying the parent component template should be\r\nsuccessful.\r\n\r\n<img width=\"555\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png\">\r\n\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"4608af41da7dc98403bbdbfba8a3ac02475277e4","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["backport:skip","Team:Fleet","release_note:feature","v8.7.0"],"number":144974,"url":"https://github.com/elastic/kibana/pull/144974","mergeCommit":{"message":"Added Time-series indexing (TSDB) to Integrations Experimental Indexing settings (#144974)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/144530\r\n\r\nAdded a new option under integration / data streams / Index settings\r\n(experimental) / TSDB\r\n\r\n- [x] Add a toggle to the \"Experimental indexing features\" data streams\r\nUI for TSDB\r\n- Enabling this toggle adds `index.mode: time_series` to the data\r\nstream's index template settings.\r\n- Note: `index.routing_path` value was not needed to be added in this\r\nlogic, because there is an existing elasticsearch automation that\r\ngenerates it from dimension fields, see test instructions.\r\n- [x] If the current package's manifest contains `index_mode:\r\ntime_series` for a given data stream, the toggle should be in an\r\n\"enabled + readonly\" state, e.g. it cannot be disabled\r\n- Note: currently there is no package that has this setting enabled, I\r\ntested locally by manually modifying the package info response. Will\r\nneed to figure out a way to create a mock package to test this.\r\n- [x] Once the toggle is enabled and the policy is saved, the toggle\r\nshould _not_ be disable-able. Enabling TSDB for a data stream is an\r\nirreversible operation\r\n- [x] Add a tooltip to make the irreversible nature of this operation\r\nclear\r\n\r\nTSDB setting in System package:\r\n<img width=\"915\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png\">\r\n<img width=\"632\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png\">\r\n\r\nTooltip:\r\n<img width=\"509\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png\">\r\n\r\n\r\n## Test instructions\r\n\r\n### Setup local registry\r\n\r\n- For testing, I created a new version of `System` package, that has a\r\n`dimension` field called `test_dimension`. This is needed so that the\r\nelasticsearch automation runs to generate the `routing_path` setting.\r\n\r\n- Download the test data zip:\r\n\r\n[registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip)\r\n\r\n\r\n- Start a registry:\r\n```\r\ndocker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main\r\n```\r\n- Add this to your kibana config:\r\n```\r\nxpack.fleet.registryUrl: http://localhost:8080\r\n```\r\n\r\n- Open kibana, open Add System integration page (should be version\r\n`1.20.5` coming from local registry)\r\n- Enable only `Collect metrics from System instances / System cpu\r\nmetrics` stream, and enable `Time-series indexing (TSDB)` switch under\r\n`Advanced options`\r\n- Click on save integration. The integration should be added\r\nsuccessfully to an agent policy.\r\n- Start a fleet server and enroll an agent to the previously created\r\nagent policy. Wait for the agent to be healthy, this should trigger\r\nsystem metrics flowing in, this way triggering the creation of the data\r\nstream.\r\n- Go to `Stack Management / Index Management / Index Templates`, and\r\nopen the details of `metrics-system.cpu`, check that it has this in\r\n`Settings` tab: `index.mode: time_series`\r\n\r\n<img width=\"2013\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png\">\r\n\r\n- Go to `Indices` tab and look for\r\n`data_stream=\"metrics-system.cpu-default\"` including hidden indices. You\r\nshould see `routing_path` populated in Settings, and `test_dimension` in\r\nMappings.\r\n\r\n<img width=\"1792\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png\">\r\n<img width=\"774\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png\">\r\n\r\n### Testing scenario of enabling TSDB in package spec by default\r\n\r\n- Prepared a newer version of `system` package `v1.20.9` that has\r\n`index_mode: \"time_series\"` in `system.cpu` data stream in package\r\nmanifest.\r\n- Download and extract the below zip and restart the local epr container\r\nto read from this folder.\r\n\r\n[registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip)\r\n- Open add system integration page\r\n- Check that `tsdb:true` is set in `Preview API request` for\r\n`system.cpu` data stream\r\n\r\n<img width=\"1033\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png\">\r\n\r\n- Open the system cpu metrics stream on the policy editor, and check\r\nthat the TSDB switch is turned on.\r\n<img width=\"1093\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png\">\r\n<img width=\"1051\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png\">\r\n\r\n### Update 11/16:\r\n\r\n- After latest changes, expect `routing_path` with dimension fields to\r\nbe generated in Index Template when enabling TSDB:\r\n\r\n<img width=\"1163\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png\">\r\n\r\nThis was added to fix the issue mentioned here\r\nhttps://github.com//pull/144974#issuecomment-1315270890\r\n\r\n- After adding TSDB, modifying the parent component template should be\r\nsuccessful.\r\n\r\n<img width=\"555\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png\">\r\n\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"4608af41da7dc98403bbdbfba8a3ac02475277e4"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/144974","number":144974,"mergeCommit":{"message":"Added Time-series indexing (TSDB) to Integrations Experimental Indexing settings (#144974)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/144530\r\n\r\nAdded a new option under integration / data streams / Index settings\r\n(experimental) / TSDB\r\n\r\n- [x] Add a toggle to the \"Experimental indexing features\" data streams\r\nUI for TSDB\r\n- Enabling this toggle adds `index.mode: time_series` to the data\r\nstream's index template settings.\r\n- Note: `index.routing_path` value was not needed to be added in this\r\nlogic, because there is an existing elasticsearch automation that\r\ngenerates it from dimension fields, see test instructions.\r\n- [x] If the current package's manifest contains `index_mode:\r\ntime_series` for a given data stream, the toggle should be in an\r\n\"enabled + readonly\" state, e.g. it cannot be disabled\r\n- Note: currently there is no package that has this setting enabled, I\r\ntested locally by manually modifying the package info response. Will\r\nneed to figure out a way to create a mock package to test this.\r\n- [x] Once the toggle is enabled and the policy is saved, the toggle\r\nshould _not_ be disable-able. Enabling TSDB for a data stream is an\r\nirreversible operation\r\n- [x] Add a tooltip to make the irreversible nature of this operation\r\nclear\r\n\r\nTSDB setting in System package:\r\n<img width=\"915\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png\">\r\n<img width=\"632\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png\">\r\n\r\nTooltip:\r\n<img width=\"509\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png\">\r\n\r\n\r\n## Test instructions\r\n\r\n### Setup local registry\r\n\r\n- For testing, I created a new version of `System` package, that has a\r\n`dimension` field called `test_dimension`. This is needed so that the\r\nelasticsearch automation runs to generate the `routing_path` setting.\r\n\r\n- Download the test data zip:\r\n\r\n[registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip)\r\n\r\n\r\n- Start a registry:\r\n```\r\ndocker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main\r\n```\r\n- Add this to your kibana config:\r\n```\r\nxpack.fleet.registryUrl: http://localhost:8080\r\n```\r\n\r\n- Open kibana, open Add System integration page (should be version\r\n`1.20.5` coming from local registry)\r\n- Enable only `Collect metrics from System instances / System cpu\r\nmetrics` stream, and enable `Time-series indexing (TSDB)` switch under\r\n`Advanced options`\r\n- Click on save integration. The integration should be added\r\nsuccessfully to an agent policy.\r\n- Start a fleet server and enroll an agent to the previously created\r\nagent policy. Wait for the agent to be healthy, this should trigger\r\nsystem metrics flowing in, this way triggering the creation of the data\r\nstream.\r\n- Go to `Stack Management / Index Management / Index Templates`, and\r\nopen the details of `metrics-system.cpu`, check that it has this in\r\n`Settings` tab: `index.mode: time_series`\r\n\r\n<img width=\"2013\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png\">\r\n\r\n- Go to `Indices` tab and look for\r\n`data_stream=\"metrics-system.cpu-default\"` including hidden indices. You\r\nshould see `routing_path` populated in Settings, and `test_dimension` in\r\nMappings.\r\n\r\n<img width=\"1792\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png\">\r\n<img width=\"774\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png\">\r\n\r\n### Testing scenario of enabling TSDB in package spec by default\r\n\r\n- Prepared a newer version of `system` package `v1.20.9` that has\r\n`index_mode: \"time_series\"` in `system.cpu` data stream in package\r\nmanifest.\r\n- Download and extract the below zip and restart the local epr container\r\nto read from this folder.\r\n\r\n[registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip)\r\n- Open add system integration page\r\n- Check that `tsdb:true` is set in `Preview API request` for\r\n`system.cpu` data stream\r\n\r\n<img width=\"1033\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png\">\r\n\r\n- Open the system cpu metrics stream on the policy editor, and check\r\nthat the TSDB switch is turned on.\r\n<img width=\"1093\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png\">\r\n<img width=\"1051\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png\">\r\n\r\n### Update 11/16:\r\n\r\n- After latest changes, expect `routing_path` with dimension fields to\r\nbe generated in Index Template when enabling TSDB:\r\n\r\n<img width=\"1163\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png\">\r\n\r\nThis was added to fix the issue mentioned here\r\nhttps://github.com//pull/144974#issuecomment-1315270890\r\n\r\n- After adding TSDB, modifying the parent component template should be\r\nsuccessful.\r\n\r\n<img width=\"555\" alt=\"image\"\r\nsrc=\"https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png\">\r\n\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"4608af41da7dc98403bbdbfba8a3ac02475277e4"}}]}] BACKPORT--> Co-authored-by: Julia Bardi <[email protected]>
This was referenced Dec 1, 2022
|
We have revalidated this feature on latest 8.7.0 BC4 self-managed environment and had below observations: On running first package:
Screenshots: ec2-52-207-251-0.compute-1.amazonaws.com.-.Remote.Desktop.Connection.2023-03-06.11-20-45.mp4
Agent policy with v1.20.5: On running second package:
Screenshots: ec2-44-192-51-97.compute-1.amazonaws.com.-.Remote.Desktop.Connection.2023-03-06.14-17-32.mp4ec2-44-192-51-97.compute-1.amazonaws.com.-.Remote.Desktop.Connection.2023-03-06.14-26-28.mp4
Agent policy with v1.20.9: Build details: Please let us know if we are missing anything here. Thanks |
|
@amolnater-qasource - You'll need to add the feature flag to |
|
Hi @kpollich Thank you for sharing the required xpack. Observations:
Screen Recordings: ec2-44-197-249-206.compute-1.amazonaws.com.-.Remote.Desktop.Connection.2023-03-07.11-06-54.mp4ec2-44-197-249-206.compute-1.amazonaws.com.-.Remote.Desktop.Connection.2023-03-07.11-15-04.mp4
Observations:
ec2-44-197-249-206.compute-1.amazonaws.com.-.Remote.Desktop.Connection.2023-03-07.11-58-44.mp4ec2-44-197-249-206.compute-1.amazonaws.com.-.Remote.Desktop.Connection.2023-03-07.11-56-58.mp4ec2-44-197-249-206.compute-1.amazonaws.com.-.Remote.Desktop.Connection.2023-03-07.11-52-05.mp4ec2-44-197-249-206.compute-1.amazonaws.com.-.Remote.Desktop.Connection.2023-03-07.14-06-51.mp4
Details for v1.20.9:
Could you please confirm if these are expected? Please let us know if anything else is required from our end. |
|
@amolnater-qasource Similarly to #152778 (comment) I'm confused where As for the |
|
Thank you for looking into this. Registery: Further, could you please update the testing details under #144974 (comment) cc: @kpollich |
|
@amolnater-qasource I see, I remembered now that the
I didn't see the When I make any change to the experimental switches (even toggle on and off another one), the preview request is updated with |
|
Thank you for the updates. Further, could you please confirm if we need to add this feature under our regression cycle. Please let us know if anything else is required from our end. cc: @kpollich |
|
@amolnater-qasource I think it would be good to add one testcase to regression, the one where the time series is enabled in package spec (this was in 1.20.9) |
|
Thank you for confirming.
Please let us know if any other validation needs to be covered from our end. |
![https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png\">\r\n<img](https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png\">\r\n<img){kind=link}
![https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png\">\r\n\r\nTooltip:\r\n<img](https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png\">\r\n\r\nTooltip:\r\n<img){kind=link}
![https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png\">\r\n\r\n\r\n##](https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png\">\r\n\r\n\r\n##){kind=link}
![https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png\">\r\n\r\n-](https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png\">\r\n\r\n-){kind=link}
![https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png\">\r\n<img](https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png\">\r\n<img){kind=link}
![https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png\">\r\n\r\n###](https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png\">\r\n\r\n###){kind=link}
![https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png\">\r\n\r\n-](https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png\">\r\n\r\n-){kind=link}
![https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png\">\r\n<img](https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png\">\r\n<img){kind=link}
![https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png\">\r\n\r\n###](https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png\">\r\n\r\n###){kind=link}
![https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png\">\r\n\r\nThis](https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png\">\r\n\r\nThis){kind=link}
![https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png\">\r\n\r\n\r\n###](https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png\">\r\n\r\n\r\n###){kind=link}
|
Hi Team, We have executed 02 testcases under Feature test run for 8.7.0 release at link: Status:
Open issue for failed testcase: #153191 Build details: As the testing is completed on this feature, we are marking this as QA:Validated. Please let us know if anything else is required from our end. |
amolnater-qasource
added
QA:Validated
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Child of #132818
Relates to elastic/package-spec#311
Since we've added support to the package spec for enabling TSDB on data streams, we need to add support in Fleet for this package manifest value.
We should also add a UI toggle (see synthetic source work: #140095) to enable package maintainers to opt in to TSDB functionality before committing to releasing a new version of their integrations.
https://github.com/elastic/integrations/blob/main/docs/how_to_test_new_indexing_features.md
Implementation
index.modeandindex.routing_pathvalues to the@packagecomponent template for the data streamindex_mode: time_seriesfor a given data stream, the toggle should be in an "enabled + readonly" state, e.g. it cannot be disabledThe text was updated successfully, but these errors were encountered: