Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[8.13] [ML] AIOps: Fix grouping for fields with large arrays. (#177438)…
… (#177765) # Backport This will backport the following commits from `main` to `8.13`: - [[ML] AIOps: Fix grouping for fields with large arrays. (#177438)](#177438) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Walter Rafelsberger","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-02-23T23:22:40Z","message":"[ML] AIOps: Fix grouping for fields with large arrays. (#177438)\n\n## Summary\r\n\r\nFixes edge cases for datasets with large arrays within single fields:\r\n\r\n- Deduplicates groups as a final step of creating groups.\r\n- Limits how many values (50) to use per field for the\r\n`frequent_item_sets` aggregations.\r\n- Fixes the `should` clauses for the query for `frequent_item_sets`, the\r\nprevious version of the query could be too narrow for fields with arrays\r\nand return no results.\r\n- For the fallback analysis when either deviation or baseline returns no\r\ndocs, increases the limit from 10 to 100 docs.\r\n- It turned out the grouping for array values of fields had another bug:\r\nBecause we treated the field/values of a group as a dictionary/record\r\nlike structure, this didn't hold multiple values for a single field. The\r\ncode was changed in this PR so it is an array of field/value pairs which\r\nnow supports multiple values per field.\r\n- On the client side, fixes unique keys for the group item badges if\r\nthere's multiple items for the same field.\r\n\r\nAdds API integration tests for a dataset with large arrays. This dataset\r\nalso triggers slowness of the `frequent_item_sets` agg and can be used\r\nfor a performance journey in a follow up. Without the new limit for how\r\nmany values per field to use, these new tests would fail because the agg\r\ncases a timeout. The assertions for chunk and action lengths were\r\nremoved because they are flaky for longer running requests (because of\r\nhow we implemented flush fix and keep alive behavior).\r\n\r\nDataset to test behavior:\r\n[aiops-lra-frequent-items-array.ndjson.zip](https://github.com/elastic/kibana/files/14362105/aiops-lra-frequent-items-array.ndjson.zip)\r\n\r\nWithout this PR, the dataset would cause the grouping part of log rate\r\nanalysis to time out. With this PR, it's still slow for just 18 docs,\r\nbut it is able to return results.\r\n\r\nVideo to replicate the test: Upload via ML File Upload, Adjust the date\r\npicker in Log Rate Analysis, then run the analysis.\r\n\r\n\r\n[aiops-log-rate-analysis-arrays-0001.webm](https://github.com/elastic/kibana/assets/230104/5d5ce34b-37ef-4e9f-81ae-f8002c194f88)\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"0d19e5eed93f358e6def5b14520b8d0b7f6aef20","branchLabelMapping":{"^v8.14.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix",":ml","Feature:ML/AIOps","v8.13.0","v8.14.0"],"title":"[ML] AIOps: Fix grouping for fields with large arrays.","number":177438,"url":"https://github.com/elastic/kibana/pull/177438","mergeCommit":{"message":"[ML] AIOps: Fix grouping for fields with large arrays. (#177438)\n\n## Summary\r\n\r\nFixes edge cases for datasets with large arrays within single fields:\r\n\r\n- Deduplicates groups as a final step of creating groups.\r\n- Limits how many values (50) to use per field for the\r\n`frequent_item_sets` aggregations.\r\n- Fixes the `should` clauses for the query for `frequent_item_sets`, the\r\nprevious version of the query could be too narrow for fields with arrays\r\nand return no results.\r\n- For the fallback analysis when either deviation or baseline returns no\r\ndocs, increases the limit from 10 to 100 docs.\r\n- It turned out the grouping for array values of fields had another bug:\r\nBecause we treated the field/values of a group as a dictionary/record\r\nlike structure, this didn't hold multiple values for a single field. The\r\ncode was changed in this PR so it is an array of field/value pairs which\r\nnow supports multiple values per field.\r\n- On the client side, fixes unique keys for the group item badges if\r\nthere's multiple items for the same field.\r\n\r\nAdds API integration tests for a dataset with large arrays. This dataset\r\nalso triggers slowness of the `frequent_item_sets` agg and can be used\r\nfor a performance journey in a follow up. Without the new limit for how\r\nmany values per field to use, these new tests would fail because the agg\r\ncases a timeout. The assertions for chunk and action lengths were\r\nremoved because they are flaky for longer running requests (because of\r\nhow we implemented flush fix and keep alive behavior).\r\n\r\nDataset to test behavior:\r\n[aiops-lra-frequent-items-array.ndjson.zip](https://github.com/elastic/kibana/files/14362105/aiops-lra-frequent-items-array.ndjson.zip)\r\n\r\nWithout this PR, the dataset would cause the grouping part of log rate\r\nanalysis to time out. With this PR, it's still slow for just 18 docs,\r\nbut it is able to return results.\r\n\r\nVideo to replicate the test: Upload via ML File Upload, Adjust the date\r\npicker in Log Rate Analysis, then run the analysis.\r\n\r\n\r\n[aiops-log-rate-analysis-arrays-0001.webm](https://github.com/elastic/kibana/assets/230104/5d5ce34b-37ef-4e9f-81ae-f8002c194f88)\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"0d19e5eed93f358e6def5b14520b8d0b7f6aef20"}},"sourceBranch":"main","suggestedTargetBranches":["8.13"],"targetPullRequestStates":[{"branch":"8.13","label":"v8.13.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.14.0","branchLabelMappingKey":"^v8.14.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/177438","number":177438,"mergeCommit":{"message":"[ML] AIOps: Fix grouping for fields with large arrays. (#177438)\n\n## Summary\r\n\r\nFixes edge cases for datasets with large arrays within single fields:\r\n\r\n- Deduplicates groups as a final step of creating groups.\r\n- Limits how many values (50) to use per field for the\r\n`frequent_item_sets` aggregations.\r\n- Fixes the `should` clauses for the query for `frequent_item_sets`, the\r\nprevious version of the query could be too narrow for fields with arrays\r\nand return no results.\r\n- For the fallback analysis when either deviation or baseline returns no\r\ndocs, increases the limit from 10 to 100 docs.\r\n- It turned out the grouping for array values of fields had another bug:\r\nBecause we treated the field/values of a group as a dictionary/record\r\nlike structure, this didn't hold multiple values for a single field. The\r\ncode was changed in this PR so it is an array of field/value pairs which\r\nnow supports multiple values per field.\r\n- On the client side, fixes unique keys for the group item badges if\r\nthere's multiple items for the same field.\r\n\r\nAdds API integration tests for a dataset with large arrays. This dataset\r\nalso triggers slowness of the `frequent_item_sets` agg and can be used\r\nfor a performance journey in a follow up. Without the new limit for how\r\nmany values per field to use, these new tests would fail because the agg\r\ncases a timeout. The assertions for chunk and action lengths were\r\nremoved because they are flaky for longer running requests (because of\r\nhow we implemented flush fix and keep alive behavior).\r\n\r\nDataset to test behavior:\r\n[aiops-lra-frequent-items-array.ndjson.zip](https://github.com/elastic/kibana/files/14362105/aiops-lra-frequent-items-array.ndjson.zip)\r\n\r\nWithout this PR, the dataset would cause the grouping part of log rate\r\nanalysis to time out. With this PR, it's still slow for just 18 docs,\r\nbut it is able to return results.\r\n\r\nVideo to replicate the test: Upload via ML File Upload, Adjust the date\r\npicker in Log Rate Analysis, then run the analysis.\r\n\r\n\r\n[aiops-log-rate-analysis-arrays-0001.webm](https://github.com/elastic/kibana/assets/230104/5d5ce34b-37ef-4e9f-81ae-f8002c194f88)\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"0d19e5eed93f358e6def5b14520b8d0b7f6aef20"}}]}] BACKPORT--> Co-authored-by: Walter Rafelsberger <[email protected]>
- Loading branch information
1 parent
4e4bca2
commit 1a61d3c