[FEATURE REQUEST] Migrate from ODFE to OpenSearch

[rafzei]

The issue goal

Provide a complete solution for migration path from Open Distro for Elasticsearch (ODFE) Epiphany component to the new one - OpenSearch.

The issue background

Open Distro for Elasticsearch (ODFE) is currently one of the Epiphany components.
However, the Elastic ceased making open source options available for Elasticsearch and Kibana, releasing them under the Elastic license (more info here), with source code available under the Elastic License or SSPL.

As stated in the Opensearch FAQ, these licensing standard is not open source and this is mainly why Open Distro development has moved to OpenSearch project which is aimed to be a fully open source project. This move was done as a code fork from the latest Apache 2.0 licensed version of Elastic&Kibana.

Based on this decisions Epiphany team have also decided to move to the OpenSearch.

The ODFE and OS changes were announced in the mid of 2021.

Additionally, on the official website for ODFE an announcement was made that all users should refer to the new OpenSearch site for news and updates as the previous site will be selectively decommissioned in time.

More details related to the new OpenSearch project can be found here and here.

Migration scenario

We can follow one of these scenarios:
a) a product replacement:

• for apply path: remove the ODFE role from the code + add a role that will install OpenSearch
• for upgrade: prepare a code which will uninstall ODFE (if exists) and install OpenSearch + remove the ODFE role from the code

b) a literal upgrade of software and data:

• for apply path: remove the ODFE role from the code + add a role that will install OpenSearch

• for upgrade: remove the ODFE role from the code + prepare a code which will run migration ODFE --> OpenSearch and here the OpenS docs suggest 2 approaches:
  b1) use a snapshot feature to migrate Elasticsearch data to a new OpenSearch cluster
  IMO: hard to implement as we can not predict the customer data structure
  b2) perform a rolling upgrade (shut down & upgrade one node at a time)
  or cluster restart upgrade (faster + fewer intermediate upgrades, but require downtime)

  plus a 3rd approach for Docker/k8s clusters:
  b3) Replace existing ODFE nodes with new OpenSearch nodes
  by updating configuration with new (OpenS) images, new settings, and new environment variables; then restarting the cluster

Sources: [1], [2], [3]

---

DoD checklist

• Changelog
  • updated
  • not needed
• COMPONENTS.md
  • updated
  • not needed
• Schema
  • updated
  • not needed
• Backport tasks
  • created
  • not needed
• Documentation
  • added
  • updated
  • not needed
• Feature has automated tests
• Automated tests passed (QA pipelines)
  • apply
  • upgrade
  • backup/restore
• Idempotency tested
• All conversations in PR resolved

[romsok24]

Scenario b2 can be achieved using 2 approaches:

b21) by running the upgrade tool prepared by OpenSearch team

( during tests that I've made it was hard to adopt this tool for Epiphany ODFE instance upgrade, ie. script was not able to find the Elastic installation nor the opened port of 9200)

b22) follow the steps described here

( preferred way )

Upgrade from ODFE version included in Epiphany v1.3 aligns with the general directives described by the official OpenSearch migration docs.