Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] AIOps Log Rate Analysis: Support for text fields. #167467

Closed
9 tasks done
walterra opened this issue Sep 28, 2023 · 2 comments
Closed
9 tasks done

[ML] AIOps Log Rate Analysis: Support for text fields. #167467

walterra opened this issue Sep 28, 2023 · 2 comments
Assignees
Labels
enhancement New value added to drive a business result Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis :ml v8.11.0

Comments

@walterra
Copy link
Contributor

walterra commented Sep 28, 2023

With the recent work done for the data drift view, we now have utilities in place to calculate p-values for any kind of data to be compared. We can make use of that to add support for identifying significant log categories. Within log rate analysis, we could categorize logs for the baseline and deviation time range and then use chi2test to identify significant changes. The results could then be part of the analysis results table. The trickier part will be to make this part of the grouping as the log categories cannot be used within the frequent_item_set aggregation. So this would have to be something on top, like augmenting the keyword based groups with matching log categories.

Tasks

@walterra walterra added enhancement New value added to drive a business result :ml Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis v8.11.0 labels Sep 28, 2023
@walterra walterra self-assigned this Sep 28, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@walterra walterra changed the title [ML] AIOPs Log Rate Analysis: Support for text fields. [ML] AIOps Log Rate Analysis: Support for text fields. Sep 28, 2023
walterra added a commit that referenced this issue Sep 28, 2023
…ailable via `server` (#167465)

Follow up to #167237.
Part of #167467.

We plan to reuse some of the queries log pattern analysis does to use
via log rate analysis too. Log pattern analysis mostly does queries from
the client side, late rate analysis has its own API endpoint and does ES
queries via Kibana server. In preparation for the use via log rate
analysis, this moves the code we need to have available server side for
log rate analysis to the `common` area of the plugin so it can be used
both on server/client.
walterra added a commit that referenced this issue Oct 4, 2023
Part of #167467.

Adds support for text fields in log pattern analysis. Text fields will
be analysed using log categorization, similar to log pattern analysis.
Significant log patterns will be identified using the `chi2test`
package, similar to how we detect data drifts.
@walterra
Copy link
Contributor Author

Moved follow up tasks to the meta issue here #168459. Closing as this covers the work done for 8.11.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New value added to drive a business result Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis :ml v8.11.0
Projects
None yet
Development

No branches or pull requests

2 participants