esArchiver datastream support

[klacabane]

Summary

Fixes #69061

Adds support for archiving/loading/unloading data streams.

When archiving indices we now verify whether we have a backing index of a data stream. When that's the case we build the data stream's backing index template resolving any component templates link, and save it as a data_stream record type in the mappings.json file:

{
  "type": "data_stream",
  "value": {
    "data_stream": "my-data-stream-one",
    "template": {
      "index_patterns": ["my-data-stream-*"],
      "mappings": { ... },
      "settings": { ... },
      ...
    }
  }
}

While multiple backing indices of the same data stream can be returned we'll only output a single entry containing the latest mappings and settings.

The documents associated with a data stream keep the same doc type but have an additional data_stream property used at load time to index it to the appropriate target (we can't directly write to a backing indice) and to pick the correct BULK operation (data streams can only use create).

Note:
The index template could have an ILM policy that we currently don't save or create when loading the archive. We can add it if they are uses case that could benefit from that

Testing

• the new data streams paths are unit tested
• functional test suites are green
• archived local and cloud data streams

Manual steps

• Create a data stream or target an existing one (see setup a data stream)
• Archive it
  node scripts/es_archiver.js save ~/my-data-stream my-data-stream --es-url=http://elastic:changeme@localhost:9200 --kibana-url=http://elastic:changeme@localhost:5601/pat
• Load the archive (while loading removes existing resources we should test against a clean cluster)
  node scripts/es_archiver.js load ~/my-data-stream --es-url=http://elastic:changeme@localhost:9200 --kibana-url=http://elastic:changeme@localhost:5601/pat
• Inspect the loaded data stream and template
• Unload the archive
  node scripts/es_archiver.js load ~/my-data-stream --es-url=http://elastic:changeme@localhost:9200 --kibana-url=http://elastic:changeme@localhost:5601/pat
• Verify the resources are gone

[klacabane]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[spalger]

LGTM, thank you so much for getting this in!

[spalger]

Nit: Part of me would prefer that docs either had an index or a data_stream, but I'm not opposed to keeping the index if there's some use for it.

[klacabane]

I mainly kept it for traceability when debugging or inspecting archived data, besides that there's no real use for it :)

[klacabane]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: dfa66dc
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

✅ unchanged

History

• 💚 Build #48042 succeeded e1f3fb0
• 💚 Build #48040 succeeded fc8acc1
• 💚 Build #48020 succeeded a2625b3
• 💔 Build #48019 failed d09c349
• 💚 Build #48005 succeeded 7764051
• 💔 Build #48003 failed 2501806

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @klacabane

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add the label auto-backport or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 132853 locally

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.3

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation