[Epic] Dashboards Metadata Storage Decoupling

[boktorbb]

Currently, Dashboards stores its metadata configuration inside OpenSearch index called .opensearch_dashboards. This approach is by design for Dashboards and is biased towards a product decision by the upstream which works out of the box for customers but it introduces challenges while operating at scale and providing high availability for Dashboards.

Dashboards depends on OpenSearch cluster’s availability and other cluster parameters such as cluster health, state, and versions because this metadata, which saves OSD configuration, and visualization metadata, is stored in the index.

The goal is to decouple this metadata from the .opensearch_dashboards index and OpenSearch entirely. This is a precursor for allowing multiple data sources to be used for Dashboards.

[ananzh]

Could you specify a bit 1) what are the Metadata 2) what they are used for 3) why we need to decouple? Thanks

[boktorbb]

Could you specify a bit 1) what are the Metadata 2) what they are used for 3) why we need to decouple? Thanks

Sure. I've updated the description of the issue for more context.

[seraphjiang]

Add one few requests, be able to allow admin who setup OSD to pick up whichever storage they want.

Save meta to opensearch is still a option

Provide tool/feature to migrate meta data from one storage to another.

A few thinking but nice to have, not only decouple the storage but also make it a separate meta service outside of dahsboards implementation

[boktorbb]

Add one few requests, be able to allow admin who setup OSD to pick up whichever storage they want.

Save meta to opensearch is still a option

Provide tool/feature to migrate meta data from one storage to another.

A few thinking but nice to have, not only decouple the storage but also make it a separate meta service outside of dahsboards implementation

Absolutely! Multiple storage options including OpenSearch is something we hope to work on.

[kavilla]

[Groom]:

Please feel free to update any questions. And then figure out some scope and timelines.

[boktorbb]

The current high level plan is to work on some PoCs corresponding to milestones in the project:

1. The first major hurdle is supporting rolling upgrades in Dashboards. This is a problem that needs to be solved regardless of metadata storage approach. There should be a plan in place for how we might approach this by either the end of June or first weeks of July. A PoC should follow.
2. There is a tight coupling of the data plugin and saved objects api to OpenSearch logic. We need to refactor the code so that OpenSearch can be slotted as a possible data source plugin. Currently, I'm working on a PoC that uses sqlite as different data source from OpenSearch
3. Create a new api/interface for current plugins/clients to make use of the new saved objects or refactor the current saved objects api. There are outstanding technical challenges that need to be discussed because of the extensive use of saved objects throughout the code by plugins and other components
4. We eventually want to have multiple data sources and have the ability to "plug-in" a new data source with minimal effort required on the core.

[AMoo-Miki]

Metadata storage decoupling shouldn't be just about being able to use something other than OpenSearch. We need a simple replicating standalone key-value store to completely separate OSD from OS.

[hdhalter]

Does this need documentation?