[APM] Add experimental support for data streams

[jalvz]

Right now APM pulls data from apm-* indices.

For the data streams feature coming with Fleet integration, we need a new set of additional defaults:

logs-apm*
metrics-apm*
traces-apm*

Notice there is no dash in the suffix (*, no -*)

We won't manage onboarding nor sourcemap indices (See #86462 and elastic/apm-server#3606 for details on those)

Let me know if something is unclear.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[sorenlouv]

Sounds like this is just a matter of updating the defaults in:

kibana/src/plugins/apm_oss/server/index.ts

Lines 27 to 33 in 2d564dd

	transactionIndices: schema.string({ defaultValue: 'apm-*' }),
	spanIndices: schema.string({ defaultValue: 'apm-*' }),
	errorIndices: schema.string({ defaultValue: 'apm-*' }),
	metricsIndices: schema.string({ defaultValue: 'apm-*' }),
	sourcemapIndices: schema.string({ defaultValue: 'apm-*' }),
	onboardingIndices: schema.string({ defaultValue: 'apm-*' }),
	indexPattern: schema.string({ defaultValue: 'apm-*' }),

@jalvz When is this needed for?

[jalvz]

7.12!

[simitt]

Just in case this isn't clear to everyone, the new indexing strategy will ingest like this:

errors -> logs-apm*
transactions -> traces-apm*
spans -> traces-apm*
metrics -> metrics-apm*

The new defaults for the events should therefore look like this:

transactionIndices: schema.string({ defaultValue: 'apm-*,traces-apm*' }), 
spanIndices: schema.string({ defaultValue: 'apm-*,traces-apm*' }), 
errorIndices: schema.string({ defaultValue: 'apm-*,logs-apm*' }), 
metricsIndices: schema.string({ defaultValue: 'apm-*,metrics-apm*' })

Not quite sure if/what we need to change for the index-pattern.

[sorenlouv]

Thanks for clarifying @simitt

Not quite sure if/what we need to change for the index-pattern.

Good question. We probably should change that to *apm* to capture everything

[dgieselaar]

The permissions of the kibana_system role need to be updated as well (in ES, and possibly cloud, depending on how far they are with migrating to using the kibana_system role). We also need to think about users that have specified custom indices.

[sorenlouv]

The permissions of the kibana_system role need to be updated as well

This, right?

We also need to think about users that have specified custom indices

True. Users who already specified custom indices and then moves to fleet won't see the updated default values presented here.
Perhaps we shouldn't solve this problem by updating the defaults. Perhaps we should hardcode these values so we are sure they will apply to everyone (they also shouldn't be configurable in the future anyway).

[simitt]

Perhaps we should hardcode these values so we are sure they will apply to everyone (they also shouldn't be configurable in the future anyway).

That would mean hardcode the new values + combine them with either the current default values or the user provided values? Sounds good to me. If they are not using the new indexing strategy, nothing changes; in case they do, they would still see old and new data, which sounds great.

[sorenlouv]

That would mean hardcode the new values + combine them with either the current default values or the user provided values?

Yes, that's what I mean. Sounds like we have a plan 👍

[sorenlouv]

fyi: created an issue to track the deprecation of UI indices in 8.0