[8.6] Added Time-series indexing (TSDB) to Integrations Experimental Indexing settings (#144974) #146562
+414
−103
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ng settings (elastic#144974) ## Summary Closes elastic#144530 Added a new option under integration / data streams / Index settings (experimental) / TSDB - [x] Add a toggle to the "Experimental indexing features" data streams UI for TSDB - Enabling this toggle adds `index.mode: time_series` to the data stream's index template settings. - Note: `index.routing_path` value was not needed to be added in this logic, because there is an existing elasticsearch automation that generates it from dimension fields, see test instructions. - [x] If the current package's manifest contains `index_mode: time_series` for a given data stream, the toggle should be in an "enabled + readonly" state, e.g. it cannot be disabled - Note: currently there is no package that has this setting enabled, I tested locally by manually modifying the package info response. Will need to figure out a way to create a mock package to test this. - [x] Once the toggle is enabled and the policy is saved, the toggle should _not_ be disable-able. Enabling TSDB for a data stream is an irreversible operation - [x] Add a tooltip to make the irreversible nature of this operation clear TSDB setting in System package: <img width="915" alt="image" src="https://user-images.githubusercontent.com/90178898/201656513-78bbe993-a900-4633-ad66-886f56ce2cf0.png"> <img width="632" alt="image" src="https://user-images.githubusercontent.com/90178898/201656559-e0c90ae2-4b21-4ee9-9bfc-a69c4ffaf708.png"> Tooltip: <img width="509" alt="image" src="https://user-images.githubusercontent.com/90178898/201657054-11daeac3-cab2-45e4-8794-17a74b6c4300.png"> ## Test instructions ### Setup local registry - For testing, I created a new version of `System` package, that has a `dimension` field called `test_dimension`. This is needed so that the elasticsearch automation runs to generate the `routing_path` setting. - Download the test data zip: [registry-packages.zip](https://github.com/elastic/kibana/files/10004220/registry-packages.zip) - Start a registry: ``` docker run -p 8080:8080 -v <path to unzipped dir>/input-packages/packages:/packages/test-packages -v <path to unzipped dir>/input-packages/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main ``` - Add this to your kibana config: ``` xpack.fleet.registryUrl: http://localhost:8080 ``` - Open kibana, open Add System integration page (should be version `1.20.5` coming from local registry) - Enable only `Collect metrics from System instances / System cpu metrics` stream, and enable `Time-series indexing (TSDB)` switch under `Advanced options` - Click on save integration. The integration should be added successfully to an agent policy. - Start a fleet server and enroll an agent to the previously created agent policy. Wait for the agent to be healthy, this should trigger system metrics flowing in, this way triggering the creation of the data stream. - Go to `Stack Management / Index Management / Index Templates`, and open the details of `metrics-system.cpu`, check that it has this in `Settings` tab: `index.mode: time_series` <img width="2013" alt="image" src="https://user-images.githubusercontent.com/90178898/201657409-1754d5a4-98fa-4646-9ccf-e47981404018.png"> - Go to `Indices` tab and look for `data_stream="metrics-system.cpu-default"` including hidden indices. You should see `routing_path` populated in Settings, and `test_dimension` in Mappings. <img width="1792" alt="image" src="https://user-images.githubusercontent.com/90178898/201659788-845fbca8-2cdb-4d1b-8af6-efc24b71f1d3.png"> <img width="774" alt="image" src="https://user-images.githubusercontent.com/90178898/201659882-e9cba197-d354-494c-974a-3c580d4d98a5.png"> ### Testing scenario of enabling TSDB in package spec by default - Prepared a newer version of `system` package `v1.20.9` that has `index_mode: "time_series"` in `system.cpu` data stream in package manifest. - Download and extract the below zip and restart the local epr container to read from this folder. [registry-packages.zip](https://github.com/elastic/kibana/files/10011702/registry-packages.zip) - Open add system integration page - Check that `tsdb:true` is set in `Preview API request` for `system.cpu` data stream <img width="1033" alt="image" src="https://user-images.githubusercontent.com/90178898/201897480-f9a79526-776f-4d9f-b168-474b44e0aa53.png"> - Open the system cpu metrics stream on the policy editor, and check that the TSDB switch is turned on. <img width="1093" alt="image" src="https://user-images.githubusercontent.com/90178898/201897787-f5a34aa6-c150-4c90-af97-bda6df92e0c4.png"> <img width="1051" alt="image" src="https://user-images.githubusercontent.com/90178898/201897943-85157b6b-caca-492d-8933-46ebe19b6930.png"> ### Update 11/16: - After latest changes, expect `routing_path` with dimension fields to be generated in Index Template when enabling TSDB: <img width="1163" alt="image" src="https://user-images.githubusercontent.com/90178898/202123899-388d3f74-014a-492c-91ad-8afc9e454186.png"> This was added to fix the issue mentioned here elastic#144974 (comment) - After adding TSDB, modifying the parent component template should be successful. <img width="555" alt="image" src="https://user-images.githubusercontent.com/90178898/202124250-14bcc310-0dd9-4246-8461-ef66c1f7ac43.png"> ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Kibana Machine <[email protected]> (cherry picked from commit 4608af4)
6 tasks
botelastic
bot
added
the
Team:Fleet
|
Pinging @elastic/fleet (Team:Fleet) |
💚 Build Succeeded
Metrics [docs]Public APIs missing comments
Async chunks
Public APIs missing exports
Saved Objects .kibana field count
Unknown metric groupsAPI count
ESLint disabled in files
ESLint disabled line counts
Total ESLint disabled count
To update your PR or re-run it, just comment with: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto8.6:Questions ?
Please refer to the Backport tool documentation