Revamping distribution build system

[gaiksaya]

Is your feature request related to a problem? Please describe

The current OpenSearch build system is fully automated, python based build system that uses manifests to build from the source. Read how to build distributions from source. However, with current set up there are few limitations causing the end to end process to build distributions especially during release cycles to be delayed by 2 to 3 hours. Along with this, the distribution build workflow for opensearch and opensearch-dashboards is tightly coupled for all distributions and architectures. Today, if we need to rebuild just one distribution or just one component of the distribution, it is not possible with this clunky workflow. Hence, we need to enhance our build system to allow decoupled, efficient builds as well as re-builds with minimal effort and time.

Describe the solution you'd like

The solution can be categorized into two:

• Introducing incremental builds #3922
• Refactor distribution job for OpenSearch and OpenSearch Dashboards #1906

Describe alternatives you've considered

No response

Additional context

No response

[zelinh]

Closing this issue as all sub tasks are done.