[ML] Updates APM Module to Work with Service Maps #70361

blaklaybul · 2020-06-30T19:35:32Z

Summary

This PR contains an updated ML job for APM's service map anomaly detection integration that allows users to enable machine learning on all services and transaction types in a specified environment.

The module id remains unchanged, but the job and datafeed ids have been updated, so there should be no issues for existing integration users looking to enable the service maps integration. Existing users can choose to run the old job alongside this new one, but we should provide guidance for those users looking to offload the existing job.

module: apm_transaction
job: high_mean_transaction_duration

elasticmachine · 2020-06-30T19:35:34Z

Pinging @elastic/ml-ui (:ml)

sorenlouv · 2020-07-01T06:59:18Z

x-pack/plugins/ml/server/models/data_recognizer/modules/apm_transaction/manifest.json

  "query": {
    "bool": {
      "filter": [
        { "term": { "processor.event": "transaction" } },
-        { "term": { "transaction.type": "request" } }


Was the previous job tied to transaction.type=request? 🤔

That change happened in #30820 - does that mean ML did not work for other transaction types for the past 1.5 years?

yes, if the jobs were created via the ML job creation process, then the datafeed filtered for "transaction.type": "request".

sorenlouv · 2020-07-01T07:06:45Z

...dels/data_recognizer/modules/apm_transaction/ml/datafeed_high_mean_transaction_duration.json

@@ -7,7 +7,7 @@
    "bool": {
      "filter": [
        { "term": { "processor.event": "transaction" } },
-        { "term": { "transaction.type": "request" } }
+        { "exists": { "field": "transaction.duration.us" } }


Given the following request to create a new ML job:

POST /api/ml/modules/setup/apm_transaction { // tagging the job with service.environment "custom_settings": { "service.environment": "production" }, // tagging the job with app name "groups": ["apm"], // specifying the indicies to query "indexPatternName": "apm-*-transaction-*", // create job and start immediately "startDatafeed": true, // limit job to specific environment "query": { "bool": { "filter": [{ "term": { "service.environment": "production" } }] } } }

Will the query specified by APM over the API and the query in the config file be merged? Or should we change the query to:

"query": { "bool": { "filter": [ { "term": { "processor.event": "transaction" } }, { "exists": { "field": "transaction.duration.us" } } { "term": { "service.environment": "production" } }] } }

?

Btw. overall does the request look correct?

the query will not get merged, it will overwrite the query in each datafeed in the module.

here’s a suggestion for the setup request.
i’ve moved the custom settings into the jobOverrides and added a job_tags property. l’m not sold on that name, so better suggestions are welcome.
i’ve also removed the groups override because it’s not needed, all APM modules already have this group set.

POST /api/ml/modules/setup/apm_transaction { // tagging the job with service.environment "jobOverrides": { "custom_settings": { "job_tags": { "service.environment": "production" } } }, // specifying the indicies to query "indexPatternName": "apm-*-transaction-*", // create job and start immediately "startDatafeed": true, // limit job to specific environment "query": { "bool": { "filter": [ { "term": { "processor.event": "transaction" } }, { "exists": { "field": "transaction.duration.us" } } { "term": { "service.environment": "production" } }] } } }

Thanks @jgowdyelastic !

Is jobOverrides requried? I thought this would do:

POST /api/ml/modules/setup/apm_transaction { "custom_settings": { "job_tags": { "service.environment": "production" } } //... }

yes, with it you can override any part of the job.

Okay, so do we then also need it for the other attributes like query?

{ "jobOverrides": { "query": { "bool": { "filter": [ { "term": { "processor.event": "transaction" } }, { "exists": { "field": "transaction.duration.us" } }, { "term": { "service.environment": "production" } } ] } } } }

no, only for custom_settings.
jobOverrides and datafeedOverrides can be thought of as additional or advanced overrides for the job.
#42946

Okay - that sounds a bit inconsistent to me. Is the intention to align this? Is it documented somewhere which fields need to be wrapped in jobOverrides and which don't?

query and groups are the only overrides which could potentially be moved into jobOverrides' and 'datafeedOverrides. For the sake of backwards compatibility I'm happy for them to stay where there are, unless others agree that they are confusing and should be moved.

sorenlouv

lgtm. Thanks for all the help @blaklaybul

peteharverson · 2020-07-01T11:52:44Z

x-pack/plugins/ml/server/models/data_recognizer/modules/apm_transaction/manifest.json

  "type": "Transaction data",
  "logoFile": "logo.json",
-  "defaultIndexPattern": "apm-*",
+  "defaultIndexPattern": "apm-*-transaction",


Just checking that this only has the * wildcard in the middle of the pattern, and not at the end too, as we have apm-*-transaction-* in the example provided to the setup endpoint.

there are two index patterns in the test data provided by @dgieselaar : apm-* and apm-*-transaction. I went with the latter since the module apm_transaction focuses only on transaction data. @dgieselaar will apm-*-transaction reliably exist? If not, @peteharverson are there any potential consequences of having a nonexistent index pattern here?

@blaklaybul it's configurable, so no guarantees (for both apm-* and apm-*-transaction). I'm assuming we set this when we create the job (have to check but can't find the code right now).

@blaklaybul the defaultIndexPattern supplied in the module manifest.json is just used as a fallback by our module endpoints if no indexPatternName is supplied to the endpoint. So for the common use case, where the index pattern is supplied to the setup endpoint to create the jobs, this value from the manifest won't be used.

thanks @peteharverson

ogupte · 2020-07-01T18:00:58Z

Is there an API within the ML plugin contract to do the job creation, or is only accessible on the REST endpoint POST /api/ml/modules/setup/apm_transaction?

jgowdyelastic · 2020-07-01T18:21:22Z

@ogupte yes, setupModuleItems is available in modulesProvider shared in our setup contract.

ogupte · 2020-07-02T08:37:33Z

In APM, we've hit a wall with the ML integration for 7.9 until this work is merged. I marked my PR (#70560) with the 'blocked' tag for now, while I paralleize on other integration areas. Here are the lines where the actual job creation code is: https://github.com/elastic/kibana/pull/70560/files#diff-e96da07e4982cbb34a6455843f012f49R84, if you could verify the jobOverrides looks correct on our end. Also are datafeedOverrides, start, and end values even necessary for this kind of job?

peteharverson

Tested the new module from within the ML plugin using a small data set. All worked fine. Just left a couple of minor comments on the description text.

x-pack/plugins/ml/server/models/data_recognizer/modules/apm_transaction/manifest.json

...server/models/data_recognizer/modules/apm_transaction/ml/high_mean_transaction_duration.json

peteharverson

Changes to the ML module files LGTM

blaklaybul · 2020-07-02T13:45:11Z

@elasticmachine merge upstream

kibanamachine · 2020-07-02T15:30:17Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 4a49cc5

Build metrics

✅ unchanged

History

💔 Build #58223 failed 96b1b35
💚 Build #57702 succeeded dc38f88
💔 Build #57661 failed 8df17b2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* updates apm integration job to work with service maps * rename apm job in setup_module test * modifies detector description Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

updates apm integration job to work with service maps

8df17b2

blaklaybul added release_note:enhancement :ml v7.9.0 apm:ml Integration between APM and ML labels Jun 30, 2020

blaklaybul requested review from sorenlouv, ogupte, peteharverson, jgowdyelastic and nehaduggal June 30, 2020 19:35

blaklaybul requested a review from a team as a code owner June 30, 2020 19:35

rename apm job in setup_module test

dc38f88

sorenlouv reviewed Jul 1, 2020

View reviewed changes

sorenlouv approved these changes Jul 1, 2020

View reviewed changes

peteharverson reviewed Jul 1, 2020

View reviewed changes

ogupte mentioned this pull request Jul 2, 2020

[APM] Adds 'Anomaly detection' settings page to create ML jobs per environment #70560

Merged

peteharverson reviewed Jul 2, 2020

View reviewed changes

x-pack/plugins/ml/server/models/data_recognizer/modules/apm_transaction/manifest.json Outdated Show resolved Hide resolved

...server/models/data_recognizer/modules/apm_transaction/ml/high_mean_transaction_duration.json Outdated Show resolved Hide resolved

modifies detector description

96b1b35

blaklaybul requested a review from peteharverson July 2, 2020 12:38

peteharverson approved these changes Jul 2, 2020

View reviewed changes

Merge branch 'master' into ml-apm-service-maps-module

4a49cc5

blaklaybul merged commit 8a09f24 into elastic:master Jul 2, 2020

blaklaybul mentioned this pull request Jul 2, 2020

[7.x] [ML] Updates APM Module to Work with Service Maps (#70361) #70616

Merged

lcawl mentioned this pull request Jul 14, 2020

[DOCS] Updates to APM ML module for service maps elastic/stack-docs#1272

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Updates APM Module to Work with Service Maps #70361

[ML] Updates APM Module to Work with Service Maps #70361

blaklaybul commented Jun 30, 2020

elasticmachine commented Jun 30, 2020

sorenlouv Jul 1, 2020

sorenlouv Jul 1, 2020

blaklaybul Jul 1, 2020

sorenlouv Jul 1, 2020 •

edited

Loading

sorenlouv Jul 1, 2020

jgowdyelastic Jul 1, 2020 •

edited

Loading

sorenlouv Jul 1, 2020

jgowdyelastic Jul 1, 2020

sorenlouv Jul 2, 2020 •

edited

Loading

jgowdyelastic Jul 2, 2020

sorenlouv Jul 5, 2020

jgowdyelastic Jul 6, 2020

sorenlouv left a comment

peteharverson Jul 1, 2020

blaklaybul Jul 1, 2020

dgieselaar Jul 1, 2020 •

edited

Loading

peteharverson Jul 1, 2020

blaklaybul Jul 1, 2020

ogupte commented Jul 1, 2020

jgowdyelastic commented Jul 1, 2020 •

edited

Loading

ogupte commented Jul 2, 2020 •

edited

Loading

peteharverson left a comment

peteharverson left a comment

blaklaybul commented Jul 2, 2020

kibanamachine commented Jul 2, 2020

[ML] Updates APM Module to Work with Service Maps #70361

[ML] Updates APM Module to Work with Service Maps #70361

Conversation

blaklaybul commented Jun 30, 2020

Summary

elasticmachine commented Jun 30, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sorenlouv Jul 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgowdyelastic Jul 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sorenlouv Jul 2, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sorenlouv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgieselaar Jul 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ogupte commented Jul 1, 2020

jgowdyelastic commented Jul 1, 2020 • edited Loading

ogupte commented Jul 2, 2020 • edited Loading

peteharverson left a comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

blaklaybul commented Jul 2, 2020

kibanamachine commented Jul 2, 2020

💚 Build Succeeded

Build metrics

History

sorenlouv Jul 1, 2020 •

edited

Loading

jgowdyelastic Jul 1, 2020 •

edited

Loading

sorenlouv Jul 2, 2020 •

edited

Loading

dgieselaar Jul 1, 2020 •

edited

Loading

jgowdyelastic commented Jul 1, 2020 •

edited

Loading

ogupte commented Jul 2, 2020 •

edited

Loading