Skip to content

Commit

Permalink
[skip CI] [ftr] add migration steps for deployment-agnostic tests (el…
Browse files Browse the repository at this point in the history
…astic#202913)

## Summary

This PR adds example of tests, that doesn't meet deployment-agnostic
criteria. I also added steps explaining how to migrate existing tests to
DA.
  • Loading branch information
dmlemeshko authored Dec 4, 2024
1 parent 6a4fe4f commit 72c906b
Showing 1 changed file with 18 additions and 0 deletions.
18 changes: 18 additions & 0 deletions x-pack/test/api_integration/deployment_agnostic/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ A deployment-agnostic API integration test is a test suite that fulfills the fol

A deployment-agnostic test should be loaded in stateful and at least 1 serverless FTR config files.

## Tests, that are not deployment-agnostic:
- tests verifying Kibana behavior under a basic license.
- tests dependent on ES/Kibana server arguments, that are not set in Elastic Cloud
- tests requiring a custom plugin to be loaded specifically for testing purposes.
- tests dependent on varying user privileges between serverless and stateful environments.

## Tests Design Requirements
A deployment-agnostic test is contained within a single test file and always utilizes the [DeploymentAgnosticFtrProviderContext](https://github.com/elastic/kibana/blob/main/x-pack/test/api_integration/deployment_agnostic/ftr_provider_context.d.ts) to load compatible FTR services. A compatible FTR service must support:

Expand Down Expand Up @@ -243,3 +249,15 @@ node scripts/functional_test_runner --config x-pack/test/api_integration/deploym
Since deployment-agnostic tests are designed to run both locally and on MKI/Cloud, we believe no extra tagging is required. If a test is not working on MKI/Cloud or both, there is most likely an issue with the FTR service or the configuration file it uses.

When a test fails on CI, automation will apply `.skip` to the top-level describe block. This means the test will be skipped in **both serverless and stateful environments**. If a test is unstable in a specific environment only, it is probably a sign that the test is not truly deployment-agnostic.

## Migrating existing tests
If your tests align with the outlined criteria and requirements, you can migrate them to deployment-agnostic by following these steps:

1. Move your tests to the `x-pack/test/api_integration/deployment_agnostic/apis/<plugin>` directory.
2. Update each test file to use the `DeploymentAgnosticFtrProviderContext` context and load the required FTR services it provides.
3. Ensure the `roleScopedSupertest` or `samlAuth` service is used instead for `supertest` for authentication and test API calls.
4. Remove all usage of the `supertest` service. It is authenticated as system index superuser and often causes test failures on Cloud, where priveleges are more strict.
5. Avoid modifying `config` files, as this could disrupt test runs in Cloud environments.
6. Include your tests in both the platform and at least one solution index file.
7. Execute your tests locally against a local environment.
8. Verify your tests by running them locally against a real MKI project and an ESS deployment to ensure full compatibility.

0 comments on commit 72c906b

Please sign in to comment.