[Upgrade Assistant] Handle migration of legacy APM indices

[alisonelizabeth]

This work is necessary for the 7.x-8 upgrade. This work has been deferred to 8.x. The below information might be partially or completely obsolete.

Background
APM Server is undergoing some fairly significant changes over the next few 7.x releases, moving from being a standalone service to one that runs under Fleet. As part of that effort we are migrating to use data streams instead of raw indices. We also anticipate some breaking changes to the schema, e.g. combining multiple numerical fields into a single aggregate_metric_double field.

For the remainder of 7.x there will be the option to use either indices or data streams, but in 8.0 we will require Fleet and data streams will be the only option. In 8.0, the APM app in Kibana will query only the new data streams. Because only the new data streams will be queried, we should provide an option for users to migrate their old data.

Requirements
The Upgrade Assistant will need to provide a means of reindexing old data (similar to the 6-7 upgrade):

1. Upgrade Assistant checks for legacy APM indices, and checks that the "apm" integration has been installed by Fleet (specifically, we need to make sure the index templates are in place).
2. Upgrade Assistant reindexes with a Painless script provided by the APM team. To keep things simple, I think we could just add this to our Fleet integration package and have it always installed. Given that in step 1 we've established the package is installed, Upgrade Assistant can just reference the installed pipeline.

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[axw]

Upgrade Assistant checks for legacy indices

By default apm-server indexes data into indices matching apm-*, but this can be customised. So I suppose this should be customisable in Upgrade Assistant too.

In addition to checking the index name, we should probably search on observer.type: apm-server

[axw]

@alisonelizabeth FYI, I've just merged a change in apm-server to introduce a new ingest pipeline called apm_data_stream_migration: elastic/apm-server#5768. This can be used to reindex docs in legacy APM indices to set data_stream.* fields and route them to the appropriate data streams.

[alisonelizabeth]

Great! Thanks @axw. I will likely follow-up with you with questions once I'm able to start working on this.

[cjcenizal]

@axw and I chatted about this over Slack on August 25. Andrew informed me that at the time the team wasn't sure if they'd need support for upgrading/reindexing indices to data streams in 8.0. APM UI will continue to read indices in 8.0 instead of introducing a hard break, but it's currently unclear how far into the future they'll extend that support. We agreed to defer working on this until they have more clarity on this.

[graphaelli]

no longer relevant