[Logs UI] Add dataset filter to ML module setup screen #64470

weltenwort · 2020-04-24T23:55:50Z

Summary

This adds the ability to filter the datasets to be processed by the ML jobs on the setup screen. When no dataset is selected for an index group, no filtering is applied. As soon as one dataset is selected, the datafeed applies a terms filter to the incoming documents.

Note: Even though the datasets are displayed for each index group, the datafeed can't apply the dataset filters selectively when it pushes the data to the ML job. The grouping only serves as an indication of occurrence of the dataset terms.

closes #60390

Previews

Testing

For performance reasons the dataset terms are fetched for the time interval selected in the preceding setup form section. When that time interval is changed the available datasets should be reloaded.
The resulting datafeed should be created with a corresponding filter query in its configuration.

Task breakdown

add route to query datasets for time interval
change index list to contain dataset selection
enhance datafeed definition to include dataset filter
fix conflicts after merging [Logs UI] Reimplement log source configuration routes in plain HTTP+JSON #64021

elasticmachine · 2020-04-24T23:55:52Z

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

weltenwort · 2020-05-04T06:46:18Z

@elasticmachine merge upstream

weltenwort · 2020-05-04T09:07:01Z

Test failures look unrelated...

@elasticmachine merge upstream

weltenwort · 2020-05-04T09:31:23Z

@elasticmachine merge upstream

Kerry350

Great job. Code looks good, and I was able to setup my jobs with "All" as well as a filter for the datasets 👍 One small comment.

Kerry350 · 2020-05-04T10:31:21Z

x-pack/plugins/infra/public/containers/logs/log_analysis/api/validate_datasets.ts

+  startTime: number,
+  endTime: number
+) => {
+  const response = await npStart.http.fetch(LOG_ANALYSIS_VALIDATE_DATASETS_PATH, {


Can this be changed so as to not use the legacy_singletons, e.g. callValidateDatasetsAPI taking fetch as an argument?

Absolutely - I just kept it in sync with the other API calls so far. Would it be ok to switch all of them over at once in a separate PR (that is not subject to a close deadline)?

Yep - of course 👍

weltenwort · 2020-05-04T10:42:19Z

...odels/data_recognizer/modules/logs_ui_categories/ml/datafeed_log_entry_categories_count.json

-  "query": {
-    "bool": {
-      "filter": [
-        {
-          "exists": {
-            "field": "message"
-          }
-        }
-      ]
-    }
-  }


ℹ️ Since the setup API doesn't merge this and any queries supplied at setup time, the query is now only defined via the API.

jgowdyelastic

ML changes LGTM

kibanamachine · 2020-05-04T11:24:57Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: ffab945

History

💔 Build #44841 failed 0ab8fa4
💚 Build #44560 succeeded d3cf524
💚 Build #44441 succeeded 0a63fc0
💚 Build #44186 succeeded 545bdcb
💔 Build #43773 failed 6a0e56f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

This adds the ability to filter the datasets to be processed by the ML jobs on the setup screen.

…ana into alerting/np-tests-migration * 'alerting/np-tests-migration' of github.com:gmmorris/kibana: [APM] Agent remote config: validation for Java agent configs (elastic#63956) [APM] Fix duplicate index patterns (elastic#64883) Drilldowns (elastic#61219) [Alerting] fix labels and links in PagerDuty action ui and docs (elastic#64032) [Event Log] Ensure sorting tests are less flaky (elastic#64781) update endpoint to restrict removing with datasources (elastic#64978) [Logs UI] [Alerting] Alerts management page enhancements (elastic#64654) Adjust kibana app owning files (elastic#65064) Migrate tutorial resources (elastic#64298) [Logs UI] Tweak copy in log alerts dialog (elastic#64645) [Logs UI] [Alerting] Documentation (elastic#64886) [Logs UI] Add dataset filter to ML module setup screen (elastic#64470) [TSVB] Fixing memory leak (elastic#64918) Bump backport to 5.4.1 (elastic#65041)

…#65063) Backports the following commits to 7.x: - [Logs UI] Add dataset filter to ML module setup screen (#64470)

Add route to fetch datasets for log indices

9e3cee9

weltenwort added release_note:enhancement Feature:Logs UI Logs UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.8.0 labels Apr 24, 2020

weltenwort self-assigned this Apr 24, 2020

weltenwort added 18 commits April 27, 2020 13:08

Merge branch 'master' into logs-ui-ml-setup-add-dataset-filter

70b7991

Remove unneeded adapter method

25f041c

Provide required lib deps

7dc774c

Add validation API call

55849ab

Factor out index selection row

77df085

Merge branch 'master' into logs-ui-ml-setup-add-dataset-filter

863eefe

Start adding dataset filter to setup screen (incomplete)

6a0e56f

WIP

27e131c

Only require timestamp field in dataset api

fca3b0c

Load datasets for valid indices

1e2112f

Avoid duplicate or empty dataset requests

545bdcb

Include dataset filter in module setup call

7a37b17

Ensure message check is not replaced by dataset filter

017ccb0

Merge branch 'master' into logs-ui-ml-setup-add-dataset-filter

0a63fc0

Disable the filter button when index not selected

2a0dec8

Disable the submission button while validating

d51fc2c

Add dataset filter to description text

162a9d2

Ensure disappering datasets get removed from filter

d3cf524

weltenwort added the v8.0.0 label Apr 30, 2020

weltenwort marked this pull request as ready for review April 30, 2020 23:04

weltenwort requested review from a team as code owners April 30, 2020 23:04

Merge branch 'master' into logs-ui-ml-setup-add-dataset-filter

0ab8fa4

Kerry350 self-requested a review May 4, 2020 08:08

Merge branch 'master' into logs-ui-ml-setup-add-dataset-filter

ffab945

Kerry350 approved these changes May 4, 2020

View reviewed changes

weltenwort commented May 4, 2020

View reviewed changes

jgowdyelastic approved these changes May 4, 2020

View reviewed changes

weltenwort merged commit 39e31d6 into elastic:master May 4, 2020

weltenwort deleted the logs-ui-ml-setup-add-dataset-filter branch May 4, 2020 11:29

weltenwort mentioned this pull request May 4, 2020

[7.x] [Logs UI] Add dataset filter to ML module setup screen (#64470) #65063

Merged

weltenwort added a commit to weltenwort/kibana that referenced this pull request May 4, 2020

[Logs UI] Add dataset filter to ML module setup screen (elastic#64470)

fcc783f

This adds the ability to filter the datasets to be processed by the ML jobs on the setup screen.

weltenwort added a commit that referenced this pull request May 4, 2020

[7.x] [Logs UI] Add dataset filter to ML module setup screen (#64470) (…

ddd0d1b

…#65063) Backports the following commits to 7.x: - [Logs UI] Add dataset filter to ML module setup screen (#64470)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Logs UI] Add dataset filter to ML module setup screen #64470

[Logs UI] Add dataset filter to ML module setup screen #64470

weltenwort commented Apr 24, 2020 •

edited

Loading

elasticmachine commented Apr 24, 2020

weltenwort commented May 4, 2020

weltenwort commented May 4, 2020

weltenwort commented May 4, 2020

Kerry350 left a comment

Kerry350 May 4, 2020

weltenwort May 4, 2020

Kerry350 May 4, 2020

weltenwort May 4, 2020 •

edited

Loading

jgowdyelastic left a comment •

edited

Loading

kibanamachine commented May 4, 2020

[Logs UI] Add dataset filter to ML module setup screen #64470

[Logs UI] Add dataset filter to ML module setup screen #64470

Conversation

weltenwort commented Apr 24, 2020 • edited Loading

Summary

Previews

Testing

Task breakdown

elasticmachine commented Apr 24, 2020

weltenwort commented May 4, 2020

weltenwort commented May 4, 2020

weltenwort commented May 4, 2020

Kerry350 left a comment

Choose a reason for hiding this comment

Kerry350 May 4, 2020

Choose a reason for hiding this comment

weltenwort May 4, 2020

Choose a reason for hiding this comment

Kerry350 May 4, 2020

Choose a reason for hiding this comment

weltenwort May 4, 2020 • edited Loading

Choose a reason for hiding this comment

jgowdyelastic left a comment • edited Loading

Choose a reason for hiding this comment

kibanamachine commented May 4, 2020

💚 Build Succeeded

History

weltenwort commented Apr 24, 2020 •

edited

Loading

weltenwort May 4, 2020 •

edited

Loading

jgowdyelastic left a comment •

edited

Loading