[APM] Shorten ML Job IDs

[graphaelli]

Job IDs have a 64 character limit. APM current generates very long job ids by default, like the 44-char apm-prod-6ed0-high_mean_transaction_duration. The ML team recommends we shorten them where possible.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[sorenlouv]

Currently a subset of the id is hardcoded on the ml side. The id prefix is specified on the apm side here:

kibana/x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts

Lines 84 to 86 in 1466099

	return ml.modules.setup({
	moduleId: ML_MODULE_ID_APM_TRANSACTION,
	prefix: `${APM_ML_JOB_GROUP}-${snakeCase(environment)}-${randomToken}-`,

The suffix is specified by ml here:

kibana/x-pack/plugins/ml/server/models/data_recognizer/modules/apm_transaction/manifest.json

Line 18 in 1466099

"id": "high_mean_transaction_duration",

@walterra @peteharverson It would be great if we could simply specify the full id when creating a job, instead of the id being a concatenation of different properties specified by apm and ml.

[peteharverson]

I am currently working on changes to the apm_transaction module, converting the job to work with metric (histogram) data and extending it for throughput and failed transactions. As part of that work, I will shorten the id which forms the suffix of the eventual job ID, from high_mean_transaction_duration to something like apm_metric, which would be 20 characters shorter. This provides a quick win, without requiring any more code changes.

You then need to ensure that whatever prefix you supply to the module setup endpoint is no more than (64 - len(suffix)) characters long (ideally shorter), and that it also starts with an alphanumeric character.

[sorenlouv]

You then need to ensure that whatever prefix you supply to the module setup endpoint is no more than (64 - len(suffix)) characters long (ideally shorter)

What about simply allowing the consumer to specify an id that must be less than 64 characters?

As I've mentioned before, the way ml modules work makes it quite hard for consumers to understand how the resulting ML job will look. Concatenating fields like in this case is a prime example of this.

[sophiec20]

Out of the box, the modules allow the user to specify a prefix and the job suffix is hardcoded, defined in the module - this helps users who create modules in the ML UI to have jobs with a meaningful name. Remember that most modules contain multiple jobs so its simpler to setup groups of jobs that all share the same prefix. This applies to all ML modules, not just APM. At this stage, I propose we keep current module prefix/suffix behaviour.

We could allow users to have a configurable APM job_id. The user could then create a job in the ML UI, using any name they wish, and then update the APM settings to use this per environment. We would then need to validate that the fields used in the job were valid (latency, throughput and error rate). This validation would be somewhat tricky as not everything can be perfectly validated. e.g Index patterns apm-* and trace-*, apm-* might search exactly the same indices. This would give us more flexibility. For example, this would have been a workaround for users with long environment names or an option for advanced users who want to configure additional influencers.

In summary:

• As minimum, supply a job_id prefix which is no more than (64 - len(suffix)) characters long, and which contains valid job_id characters job_id: (Required, string) Identifier for the anomaly detection job. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.
• Consider configurable APM job name and/or changing ML module prefix/suffix behaviour subject to requirement validation

[sorenlouv]

Fine to keep things as is for now. Just a bit of feedback for future api improvements:

this helps users who create modules in the ML UI to have jobs with a meaningful name

It sounds like the id is both a human readable label and a unique identifier. Separating these two would make it easy to both add a good label, and have a unique identifier within the 64 byte limit.

We could allow users to have a configurable APM job_id. The user could then create a job in the ML UI, using any name they wish

If we had a human readable label I agree that should be configurable. In that case the id doesn't need to be configurable.

[sorenlouv]

I think the real solution to APM's use of modules is for APM to not use modules at all.

Agree, going forward I think we should move away from the modules.