[Discuss] FTR test migration

[ashwin-pc]

Background:

The current system we are utilizing separates the tests and test infrastructure (Functiona test repository) from the main repository where our application code resides (OpenSearch Dashboards). This was done to consolidate the test architecture in a single place so that Dashboards and the other plugin repositories can share the benifits of an uptodate test library for all repositories. Unfortunately this has the unfortunate downside of the chicken and egg problem where tests and code changes canot happen at the same time since one will have to come in before the other breaking the others CI test.

Proposal

The proposal is to migrate to a new system in which the tests and code are colocated, while maintaining a separate test infrastructure. By implementing this change, we can ensure that our tests and code remain in sync and up to date, while also allowing the test infrastructure to be shared across multiple repositories.

Benefits:

1. Colocated code and tests: By migrating the tests to the main repository, we can ensure better synchronization and easier access for developers.
2. Common test infrastructure: A separate test infrastructure allows it to be shared among multiple repositories and maintained independently.
3. Tests and code will always be in sync: This ensures that any changes in the code will be immediately reflected in the tests.
4. Usability by external plugin owners: External plugins will be able to test their plugins using our common test infrastructure.

Risks and Tradeoffs:

1. Tests will be spread across multiple repositories: This could potentially make it more difficult to manage and maintain the tests for builds.

Implementation Details:

To achieve the proposed migration, the following changes are necessary:

On OSD:

1. Modify OpenSearch Dashboards's CI workflow script to use tests from Dashboards's repository. similar to the existing BWC workflow.
2. Move existing tests from Functional test's repository to the Dashboards's repository.
3. Update documentation in the repository to reflect the change.
4. Optional: For local development add commands for syncing scripts between remote and local environments, enabling users to write tests locally. This is essential as the functional test runner needs to run in an environment that supports the browser, while OSD does not support all desktop environments. Initially we can do this via documentation and in future add support scripts for syncing both Docker-based and SSH-based development.

On Functional test repo:

1. Update Functional test repositories's CI workflow to fetch code from a list of remote destinations and add OSD as one of its targets.
2. Remove existing tests from Functional test's repository to the Dashboards's repository. Keep any release specific tests in the functional test repo.
3. Update documentation in the repo repositories to reflect the change.
4. Create documentation detailing how to integrate the Functioanal test runner in a repository, allowing other repositories to adopt the architecture.

I'm open to any and all suggestions, so please let me know what you think in the comments below.

[seraphjiang]

The functional test repo is responsible for release testing (include OSD and Plugins) not used for testing for development. The repo name is confusion.

One approach is OSD could build its own functional test but reuse common test infrastructure and test library.

[kavilla]

The functional test repo is responsible for release testing (include OSD and Plugins) not used for testing for development. The repo name is confusion.

One approach is OSD could build its own functional test but reuse common test infrastructure and test library.

I think ideally that is the approach here since this takes leverage of a lot of great things. Another benefit is having only super generic testing for the core repo side in that repo for release testing that we know we can't test in our repo since the availability of plugins.

[ashwin-pc]

@seraphjiang yeah i think i wasnt clear about the release test. I'll update the proposal to mention that. This only changes the location of the tests that arent specific to releases. The Functional test repo will keep the test scripts specific to releases but other specific tests can be moved out. Additionally, the release process can also reuse existing test scripts from the repos using this process using a pointer that points to tests in the remote repositories. This can harden the release process beyond just the simple release scripts that we run today.

See attached diagram above

[seraphjiang]

Adding more to OSD repo make it big monolith repository.

opensearch project doesn't go that approach. E.g. not all plugin are in OSD repo. Oui is separate module. Build has its own repo. Recent there is new opensearch-benchmark repo focus on benchmark test.

We could still add cypress test in OSD repo. In the meantime we should consider move common test module into separate test lib/infra repo so other repo could reuse.

[ashwin-pc]

The problem with a common benchmark repo is that we again run into the chicken and egg problem. When we update one feature that breaks a test, we have to update the test in another place. Just to reiterate, only the test scripts specific to OSD will stay in OSD, but the rest of the test architecture will stay in place. Even if the fixtures need to be updated, they will happen in the functional test repo for now :)

Also another added benefit is we automatically have the right tests in the right release branches. Right now we have to manually make sure that tests for a 1.x or 2.x branch are correctly backported to the appropriate branch on both repos. This takes away that pain too.

[bbarani]

Note: We are actively working on parallelizing OSD integration tests using FT repo and its tracked here opensearch-project/opensearch-build#2144. FYI...We are very very close to completion and any change to the existing setup would require some heads up for us to incorporate the finalized design.

[ashwin-pc]

@bbarani Thanks for the headsup! Do you have a timeline for when that will be completed? Also i dont think that the design for the parallelization will change significantly for this. In fact this might actually make the job of moving the tests easier since the tests are already isolated. The main change here would be that we need to allow the --component flag to access remote destinations. This is still only in the discussion stage. Once we get enough feedback, I will come up with a proposal that accounts for all these changes.