From 72c906b25b8c8ebe430396a610ade0ad7f1d4baa Mon Sep 17 00:00:00 2001 From: Dzmitry Lemechko Date: Wed, 4 Dec 2024 16:25:09 +0100 Subject: [PATCH] [skip CI] [ftr] add migration steps for deployment-agnostic tests (#202913) ## Summary This PR adds example of tests, that doesn't meet deployment-agnostic criteria. I also added steps explaining how to migrate existing tests to DA. --- .../deployment_agnostic/README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/x-pack/test/api_integration/deployment_agnostic/README.md b/x-pack/test/api_integration/deployment_agnostic/README.md index 3bc1c70dda1ab..6064d52cdcf6b 100644 --- a/x-pack/test/api_integration/deployment_agnostic/README.md +++ b/x-pack/test/api_integration/deployment_agnostic/README.md @@ -9,6 +9,12 @@ A deployment-agnostic API integration test is a test suite that fulfills the fol A deployment-agnostic test should be loaded in stateful and at least 1 serverless FTR config files. +## Tests, that are not deployment-agnostic: +- tests verifying Kibana behavior under a basic license. +- tests dependent on ES/Kibana server arguments, that are not set in Elastic Cloud +- tests requiring a custom plugin to be loaded specifically for testing purposes. +- tests dependent on varying user privileges between serverless and stateful environments. + ## Tests Design Requirements A deployment-agnostic test is contained within a single test file and always utilizes the [DeploymentAgnosticFtrProviderContext](https://github.com/elastic/kibana/blob/main/x-pack/test/api_integration/deployment_agnostic/ftr_provider_context.d.ts) to load compatible FTR services. A compatible FTR service must support: @@ -243,3 +249,15 @@ node scripts/functional_test_runner --config x-pack/test/api_integration/deploym Since deployment-agnostic tests are designed to run both locally and on MKI/Cloud, we believe no extra tagging is required. If a test is not working on MKI/Cloud or both, there is most likely an issue with the FTR service or the configuration file it uses. When a test fails on CI, automation will apply `.skip` to the top-level describe block. This means the test will be skipped in **both serverless and stateful environments**. If a test is unstable in a specific environment only, it is probably a sign that the test is not truly deployment-agnostic. + +## Migrating existing tests +If your tests align with the outlined criteria and requirements, you can migrate them to deployment-agnostic by following these steps: + +1. Move your tests to the `x-pack/test/api_integration/deployment_agnostic/apis/` directory. +2. Update each test file to use the `DeploymentAgnosticFtrProviderContext` context and load the required FTR services it provides. +3. Ensure the `roleScopedSupertest` or `samlAuth` service is used instead for `supertest` for authentication and test API calls. +4. Remove all usage of the `supertest` service. It is authenticated as system index superuser and often causes test failures on Cloud, where priveleges are more strict. +5. Avoid modifying `config` files, as this could disrupt test runs in Cloud environments. +6. Include your tests in both the platform and at least one solution index file. +7. Execute your tests locally against a local environment. +8. Verify your tests by running them locally against a real MKI project and an ESS deployment to ensure full compatibility.