[APM] Design flows for the new Machine Learning anomaly detection integration

[formgeist]

Summary

We're looking to introduce a new ML job to support only having to create a single job to support anomaly detection for all services detected in the APM index.

The current workflow is tied to setting up a job for each service for each transaction type. This means changes to the existing user flow of enabling anomaly detection and offering new options for users to manage the new anomaly detection integration.

Description of new and updated UX flows

Enable anomaly detection

We want to offer users a one-click "Enable anomaly detection" option that will create the new ML job. The default job configuration will be set to all services, all transaction types, and all environments.

• The option to enable anomaly detection should be available immediately upon landing in the APM app experience. This means displaying a message to users on the APM overview pages (services list, traces list and global service map).
• The message should be able to offer different messages depending on the license.

Manage anomaly detection integration

The default configuration is a great start, but we imagine some users would want to manage the services, environments, or even transaction types that are analyzed.

We propose to offer a configuration UI to manage rule-based exceptions to the job configuration.

This will allow users to...

• Configure to exclude a given service from the data feed.
• Configure to exclude a set of services based on service.environment variables
• Configure to exclude a set of service transactions based on the transaction.type field
• Ability to support more exclude rulesets?

Migrate existing anomaly detection jobs

Existing anomaly detection jobs should be stopped and deleted if the user enables the new anomaly detection job configuration.

Might need to show a confirmation modal or similar to manually confirm deleting the existing jobs (and their historic data).

[elasticmachine]

Pinging @elastic/observability-design (design)

[formgeist]

Design update, June 16 2020

I wanted to share some design mocks that I have put together based on our initial scope and approach. I've passed a few different options around in the team around the messaging and option to enable the new job, and we've landed on an approach that will also be consistent with the other Observability apps and SIEM.

I've put together a walkthrough video (which was shared in the Observability design weekly email), but I'll share it as well.

Finally, there's a Figma prototype where you can take the mocks for a spin.

Appreciate any feedback on the flow, or if anyone has any questions around the design, feel free to reply.

cc @blaklaybul @grabowskit @nehaduggal @elastic/apm-ui

[blaklaybul]

@formgeist lovely walkthrough! I want to confirm that your design for adding/removing environments is compatible with our datafeeds. That is, if you enable a job for a "forthcoming" environment that doesn't yet have any data, we should still be able to start the job and keep it open until the data arrives. This behavior is actually enabled by default, and the datafeeds on https://github.com/elastic/machine-learning-data/pull/423 reflect this.

[formgeist]

@blaklaybul Thanks for confirming - I think this might come in handy for those who are changing or adding new environments and wants to be sure that ML is working from the get-go.

[formgeist]

Design update, June 26 2020

Hi team,

We're fast approaching implementation (even more so with the current ML integration having been removed from the app), so I wanted to bring an update on the design flows. There are some minor tweaks to the setup flow due to the updated job specs.

I've yet again made a video presentation of the changes along with an updated Figma prototype. Let me know if you have any feedback or questions. The UI team is already starting to put together the larger pieces, so time is of the essence in order to make it in time for upcoming release.

Thanks!

[formgeist]

Design update, July 1 2020

Just another final design update, where mainly smaller improvements have been added to the enable/create jobs flow.

Updated Figma prototype

I will move over the designs to individual Kibana issues for the already in-progress implementation and close this initial design issue. If anything new or improvements need design, we'll open new Kibana issues to handle tracking those.

[formgeist]

Closing as I've opened two new design implementation issues for the UI team elastic/kibana#70440 + elastic/kibana#70422 to track the implementation of the new Settings view and the option added to the APM header. Any additional design issues will be created and tracked alongside the implementation in Kibana.