[BUG] functional test fail due to build on main fail

[ananzh]

What is the bug?

Currently the functional test repo is useless because any new features won't be able to be tested. For example, new funtional tests failed on this PR because tests fail (opensearch-project/opensearch-dashboards-functional-test#423). The reason is because that functional test repo is using main for test, but the current build failed on 3.0.0. So the build can't capture the latest changes in main. To reproduce:

1. download the latest main
   wget https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/3.0.0/latest/linux/x64/tar/dist/opensearch-dashboards/opensearch-dashboards-3.0.0-linux-x64.tar.gz

2. unzip the download and check the commit

cat manifest.yml 
---
schema-version: '1.1'
build:
  name: OpenSearch Dashboards
  version: 3.0.0
  platform: linux
  architecture: x64
  distribution: tar
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/3.0.0/5420/linux/x64/tar/dist/opensearch-dashboards/opensearch-dashboards-3.0.0-linux-x64.tar.gz
  id: '5420'
components:
  - name: OpenSearch-Dashboards
    repository: https://github.com/opensearch-project/OpenSearch-Dashboards.git
    ref: main
    commit_id: 5b8a378f38bdcd1223747fddfa9b791ec3071931
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/3.0.0/5420/linux/x64/tar/builds/opensearch-dashboards/dist/opensearch-dashboards-min-3.0.0-linux-x64.tar.gz
  - name: functionalTestDashboards
    repository: https://github.com/opensearch-project/opensearch-dashboards-functional-test.git
    ref: main
    commit_id: 6c3ead8260d8cc3db0ea511a7fba1216dd1db867
  - name: reportsDashboards
    repository: https://github.com/opensearch-project/dashboards-reporting.git
    ref: main
    commit_id: 7a60ae8a95968d8f977cfc5a48984789e240f20b
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/3.0.0/5420/linux/x64/tar/builds/opensearch-dashboards/plugins/reportsDashboards-3.0.0.zip
  - name: securityDashboards
    repository: https://github.com/opensearch-project/security-dashboards-plugin.git
    ref: main
    commit_id: 541fc2d3195097afb7bbd424daaa2e01b46d716e
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/3.0.0/5420/linux/x64/tar/builds/opensearch-dashboards/plugins/securityDashboards-3.0.0.zip
  - name: queryWorkbenchDashboards
    repository: https://github.com/opensearch-project/dashboards-query-workbench.git
    ref: main
    commit_id: 68d5009c6fb81d8140336f036246448ff40458e4
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/3.0.0/5420/linux/x64/tar/builds/opensearch-dashboards/plugins/queryWorkbenchDashboards-3.0.0.zip
  - name: searchRelevanceDashboards
    repository: https://github.com/opensearch-project/dashboards-search-relevance.git
    ref: main
    commit_id: 0c602c9039c5b0e3d446144650dcba49b24aa5eb
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/3.0.0/5420/linux/x64/tar/builds/opensearch-dashboards/plugins/searchRelevanceDashboards-3.0.0.zip
  - name: observabilityDashboards
    repository: https://github.com/opensearch-project/dashboards-observability.git
    ref: main
    commit_id: 3f9c0a5f62ff2a41e55efa6027e68cdb7d4e055e
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/3.0.0/5420/linux/x64/tar/builds/opensearch-dashboards/plugins/observabilityDashboards-3.0.0.zip
  - name: ganttChartDashboards
    repository: https://github.com/opensearch-project/dashboards-visualizations.git
    ref: main
    commit_id: 81e42ede979dff3edd71a919931cb7ec8af777a8
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/3.0.0/5420/linux/x64/tar/builds/opensearch-dashboards/plugins/ganttChartDashboards-3.0.0.zip
  - name: notificationsDashboards
    repository: https://github.com/opensearch-project/dashboards-notifications.git
    ref: main
    commit_id: 7b978963a440bb534bf4c543eaeaca5a6e97303f
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/3.0.0/5420/linux/x64/tar/builds/opensearch-dashboards/plugins/notificationsDashboards-3.0.0.zip

Could also check the context

~/opensearch-dashboards-3.0.0/src/plugins/vis_type_table/public$ ls
_table_vis.scss  agg_table  components  index.scss  paginated_table  table_vis.html

What is the expected behavior?

functional test should be able test the new features and changes

[tianleh]

We need infra team to check why latest OSD bundle build failed.

[tianleh]

@peterzhuamazon Could you please help check?

[rishabh6788]

@Divyaasm FYIA

[jordarlu]

Hi, @Divyaasm , @rishabh6788 , could one of you kindly provide a concise summary of the conclusion we've reached regarding the resolution of this issue? Are we planning to rely solely on the Validation mechanism, and does this mechanism encompass the FT repository? Just try to see how we can move on this issue,, thank you

[gaiksaya]

We have recently implemented --continue-on-error that will create a successful bundle and should be always up to date. Regarding a valid OS and OSD, see #3441

[jordarlu]

Hello, @ananzh , since we have deployed the --continue-on-error as mentioned above, the bundle build will continue even individual plugin component failed ; let me close this issue for now ; but if you think we still need to have it opened, please feel free to reopen it.. thanks!!