[Testing] Migrate functional tests and remove selenium

[ashwin-pc]

Introduction

The testing ecosystem in OpenSearch Dashboards uses Selenium and an installation of Chrome, along with Jest’s Jasmin2. As of June 2021, OpenSearch Dashboards has around 700 functional tests developed in Jest and Selenium. Testing with Selenium is notorious for being flaky and we have had to patch Jasmin2 and other Jest components to partially fight this flakiness. Additionally tests take a long time to run and modern choices are shown to be 25% faster; that sums up to 1.5 hours faster tests.

The official functional testing repository for OpenSearch dashboards is now opensearch-dashboards-functional-test. We need to migrate all our existing Selenium based function tests to the new repository.

Pros of shifting from Selenium to Cypress:

• Faster test execution due to its architecture
• Automatic waiting for elements and assertions, reducing flakiness
• Real-time reloading and time-travel debugging for easier test development
• Simpler and more intuitive API
• Better support for modern JavaScript frameworks
• No need for a separate WebDriver process

Cons of shifting from Selenium to Cypress:

• Only supports JavaScript, whereas Selenium supports multiple languages
• Limited to testing on Chromium-based browsers
• No native support for multi-tab or multi-window testing

Tasks

Tasks are break down to three major parts, research, proposal and implementation.

Research

• [Research] Visualize Browser Interactions for Selenium Tests with Docker #3700
• Assess Selenium test suite #3701
• Understand Cypress test environment and document how to run cypress test locally #3702
• Research selenium tests migration strategies #3703

Proposal

• A migration proposal for moving Selenium tests to Cypress #3704

Implementation

• Migrate Selenium tests to Cypress #3705
• Update documentation and training materials #3706

[seanneumann]

We need to understand how much work is here and then where to prioritize it.

[kavilla]

Reducing priority as we will complete this as we go. Migrating tests whenever we are doing updates to that part of the repo.