[ML] AIOps: 8.16 Log Rate Analysis improvements. #187684
Assignees
Labels
Feature:ML/AIOps
ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis
Meta
:ml
v8.16.0
Comments
walterra
added
Meta
:ml
Feature:ML/AIOps
|
Pinging @elastic/ml-ui (:ml) |
This was referenced Jul 5, 2024
walterra
added a commit
that referenced
this issue
Jul 17, 2024
…88137) ## Summary Part of #187684. So far we ran individual queries for each field candidate to get significant items. The historic reason is that we've been uber cautious not to run into issues with the `max_buckets` setting. But since we fetch the top 1k items and the `max_buckets` default is 65k it should be safe to change that. This PR updates fetching significant items to combine multiple field candidates within one query using multiple aggs. The current setting in this PR is now to add up to 50 field candidates into a single query. This will result in up to ~50k buckets (50 x 1k buckets for the sig terms agg plus 50 buckets for the cardinality aggs). If there's more field candidates, we'll still make use of the async queue where we do up to 5 queries in parallel. The result is that for example for 200 field candidates we'll just do 4 queries instead of 200 previously. Previous: <img width="1624" alt="image" src="https://github.com/user-attachments/assets/1e11ff1c-a0c2-4dcf-9399-27456439faad">  After: <img width="1554" alt="image" src="https://github.com/user-attachments/assets/33ccb9ef-fe5b-4945-a87f-77347ba097ea"> ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
2 tasks
walterra
added a commit
that referenced
this issue
Jul 23, 2024
#188648) ## Summary Part of #187684. This moves functions related to log rate changes to the `@kbn/aiops_log_rate_analysis` package. - `getLogRateAnalysisType` was renamed to `getLogRateAnalysisTypeForHistogram` to indicate its use with histogram data. - `getLogRateAnalysisTypeForCounts` was added for cases where we don't have the histogram data available but just the doc counts for baseline an deviation time ranges. This isn't used yet as of this PR but will be in a follow up in combination with the o11y AI assistant. - `getSwappedWindowParameters` is a helper to consolidate inline code that's used to swap baseline and deviation when we detected a dip in log rate. - Rounding for the log rate change messages was tweaked. Changes below `10x` will now be rounded to one digit to avoid messages like `1x increase`. - Tweaked/Shortened the message for 0 in baseline or deviation to just `45 up from 0 in baseline` / `down to 0 from 45 in baseline`. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
walterra
added a commit
that referenced
this issue
Jul 30, 2024
…from analysis (not just grouping) (#188913) ## Summary Part of #187684. So far the popover to filter fields was only available when grouping was enabled. This PR updates the behavior so it's available all the time and can be used to exclude field candidates from the analysis. If we detect the index to be based on an ECS schema, we auto-select a set of predefined fields. Changes in this PR: - Creates a new route `/internal/aiops/log_rate_analysis/field_candidates` to be able to fetch field candidates independent of the main streaming API call. - Fixes the code to consider "remaining" field candidates to also consider text field candidates. This was originally developed to allow to continue an analysis that errored for some reason. We use that option to also pass on the custom field list from the field selection popover. - Fetching the field candidates is done in a new redux slice `logRateAnalysisFieldCandidatesSlice` using an async thunk. - Filters the list of field candidates by a predefined field of allowed fields when an ECS schema gets detected. - Renames `fieldCandidates` to `keywordFieldCandidates` for clearer distinction against `textFieldCandidates`. - Refactors `getLogRateAnalysisTypeForCounts` args to a config object. - Bump the API version for the full log rate analysis to version 3. We missed bumping the version in #188648. This update manages proper versioning between v2 and v3, also the API integration tests cover both versions. [aiops-log-rate-analysis-fields-filter-0001.webm](https://github.com/user-attachments/assets/e3ed8d5b-f01c-42ef-8033-caa7135b8cc0) ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
walterra
added a commit
that referenced
this issue
Aug 5, 2024
…top items and histograms. (#189155) ## Summary Follow up to #188137. Part of #187684. - Groups chunks of terms aggregations for field candidates when running the fallback to get top terms instead of significant terms when either baseline or deviation time range contains no documents. - Groups chunks of histogram aggregations for the data for the mini histogram charts. Previously we reused the code for the transform/dfa data grid mini histograms for this, it's now refactored to an optimized version for log rate analysis. - Adds `withSpan` wrappers to group log rate analysis steps for APM (magenta bars in the "after" screenshot). - Removes some no longer used code from API version 1. - Disables support for `boolean` fields, it doesn't work properly with the `frequent_item_sets` aggregations. - Fixes the loading step sizes to correct the loading progress bar going from 0-100%. Before: <img width="480" alt="image" src="https://github.com/user-attachments/assets/dc316166-8f2b-4b0f-84a4-6813f69cd10a"> After: <img width="500" alt="image" src="https://github.com/user-attachments/assets/4c532c76-42a0-4321-a261-3b7cf9bbd361"> ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
3 tasks
walterra
added a commit
that referenced
this issue
Aug 7, 2024
…189863) ## Summary Follow up to #187669. Part of #187684. Fixes #176387. (Ran the flaky test runner on AIOps functional tests) - Fixes the `size: 0` option to be properly nested for `createCategoryRequest()`. - Changes the arguments structure for `fetchTopCategories` and `fetchTopTerms` from individual arguments to an options object to be more in line with the other functions used for log rate analysis. - Adds jest unit test for `fetchTopCategories` and `fetchTopTerms`. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
9 tasks
alvarezmelissa87
added a commit
that referenced
this issue
Oct 3, 2024
…nge (#193501) ## Summary This PR - updates the `LogRateAnalysisResultsTable` to use `EuiInMemoryTable` to simplify sorting and pagination - adds sorting to `Log rate change` column - persists columns selected for viewing in the result view Related meta issue: #187684 ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) --------- Co-authored-by: Elastic Machine <[email protected]>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Oct 3, 2024
…nge (elastic#193501) ## Summary This PR - updates the `LogRateAnalysisResultsTable` to use `EuiInMemoryTable` to simplify sorting and pagination - adds sorting to `Log rate change` column - persists columns selected for viewing in the result view Related meta issue: elastic#187684 ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) --------- Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit c18184a)
tiansivive
pushed a commit
to tiansivive/kibana
that referenced
this issue
Oct 7, 2024
…nge (elastic#193501) ## Summary This PR - updates the `LogRateAnalysisResultsTable` to use `EuiInMemoryTable` to simplify sorting and pagination - adds sorting to `Log rate change` column - persists columns selected for viewing in the result view Related meta issue: elastic#187684 ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) --------- Co-authored-by: Elastic Machine <[email protected]>
|
Closing, all work items for 8.16 have been completed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Break out from the full backlog in #187683.
Follow up to #181111.
API
UI
Analysis
Text field pattern support
Observability Alert Details Pages
Observability AI Assistant Context
Telemetry
Workflows
The text was updated successfully, but these errors were encountered: