Skip to content

Commit

Permalink
Enhance test_operator role for flexible test workflows
Browse files Browse the repository at this point in the history
This commit adds key improvements to the test_operator role:

Support for defining test execution order.
Ability to run the same testing controller multiple times.
Pre-hooks and post-hooks for stage-specific customization.
Per-stage variable overrides via files or test_vars.
Refactored pre-set testing stages to enable more dynamic test
management and extend hook behavior for complex scenarios.

Jira: OSPRH-10106
  • Loading branch information
eshulman2 committed Nov 27, 2024
1 parent 924f1e8 commit d75e552
Show file tree
Hide file tree
Showing 17 changed files with 244 additions and 150 deletions.
2 changes: 0 additions & 2 deletions docs/source/usage/01_usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,6 @@ are shared among multiple roles:
- `cifmw_run_tests`: (Bool) Specifies whether tests should be executed.
Defaults to false.
- `cifmw_run_test_role`: (String) Specifies which ci-framework role will be used to run tests. Allowed options are `test_operator`, `tempest` and `shiftstack`. Defaults to `tempest`.
- `cifmw_run_tempest`: (Bool) Specifies whether tempest tests should be run. Notice tempest tests can be executed with either `test_operator` or `tempest` roles. Defaults to true.
- `cifmw_run_tobiko`: (Bool) Specifies whether tobiko tests should be run. Notice tobiko tests can only be executed with the `test_operator` role. Defaults to false.
- `cifmw_edpm_deploy_nfs`: (Bool) Specifies whether an nfs server should be deployed.
- `cifmw_nfs_target`: (String) The Ansible inventory group where the nfs server is deployed. Defaults to `computes`. Only has an effect if `cifmw_edpm_deploy_nfs` is set to `true`.
- `cifmw_nfs_network`: (String) The network the deployed nfs server will be accessible from. Defaults to `storage`. Only has an effect if `cifmw_edpm_deploy_nfs` is set to `true`.
Expand Down
4 changes: 2 additions & 2 deletions roles/shiftstack/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,12 @@ Role for triggering Openshift on Openstack QA automation (installation and tests
The role is imported in the test playbook, i.e. when:
```
cifmw_run_tests: true
cifmw_run_tempest: false
cifmw_test_operator_stages: []
cifmw_run_test_role: shiftstack
cifmw_run_test_shiftstack_testconfig:
- 4.17_ovnkubernetes_ipi.yaml
- 4.17_ovnkubernetes_upi.yaml
cifmw_shiftstack_qa_gerrithub_change: refs/changes/29/1188429/50 #optional
$ ansible-playbook deploy-edpm.yml -e 'cifmw_run_tests=true cifmw_run_tempest=false cifmw_run_test_role=shiftstack cifmw_openshift_kubeconfig={{ ansible_user_dir }}/.kube/config cifmw_shiftstack_qa_gerrithub_change=refs/changes/29/1188429/50' -e 'cifmw_run_test_shiftstack_testconfig=["4.17_ovnkubernetes_ipi.yaml","4.17_ovnkubernetes_upi.yaml"]'
$ ansible-playbook deploy-edpm.yml --extra-vars "cifmw_run_tests=true cifmw_run_test_role=shiftstack cifmw_test_operator_stages=[] cifmw_openshift_kubeconfig={{ ansible_user_dir }}/.kube/config cifmw_run_test_shiftstack_testconfig=4.15_ovnkubernetes_ipi_va1.yaml cifmw_shiftstack_qa_gerrithub_change=refs/changes/29/1188429/50"
```
49 changes: 45 additions & 4 deletions roles/test_operator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,12 @@
Execute tests via the [test-operator](https://openstack-k8s-operators.github.io/test-operator/).

## Parameters

* `cifmw_test_operator_artifacts_basedir`: (String) Directory where we will have all test-operator related files. Default value: `{{ cifmw_basedir }}/tests/test_operator` which defaults to `~/ci-framework-data/tests/test_operator`
* `cifmw_test_operator_namespace`: (String) Namespace inside which all the resources are created. Default value: `openstack`
* `cifmw_test_operator_index`: (String) Full name of container image with index that contains the test-operator. Default value: `quay.io/openstack-k8s-operators/test-operator-index:latest`
* `cifmw_test_operator_timeout`: (Integer) Timeout in seconds for the execution of the tests. Default value: `3600`
* `cifmw_test_operator_logs_image`: (String) Image that should be used to collect logs from the pods spawned by the test-operator. Default value: `quay.io/quay/busybox`
* `cifmw_test_operator_concurrency`: (Integer) Tempest concurrency value. Default value: `8`
* `cifmw_test_operator_concurrency`: (Integer) Tempest concurrency value. As of now this value can not be specified inside `test_vars`. Default value: `8`
* `cifmw_test_operator_cleanup`: (Bool) Delete all resources created by the role at the end of the testing. Default value: `false`
* `cifmw_test_operator_tempest_cleanup`: (Bool) Run tempest cleanup after test execution (tempest run) to delete any resources created by tempest that may have been left out.
* `cifmw_test_operator_default_groups`: (List) List of groups in the include list to search for tests to be executed. Default value: `[ 'default' ]`
Expand All @@ -25,6 +24,20 @@ Execute tests via the [test-operator](https://openstack-k8s-operators.github.io/
* `cifmw_test_operator_delete_logs_pod`: (Boolean) Delete tempest log pod created by the role at the end of the testing. Default value: `false`.
* `cifmw_test_operator_privileged`: (Boolean) Spawn the test pods with `allowPrivilegedEscalation: true` and default linux capabilities. This is required for certain test-operator functionalities to work properly (e.g.: `extraRPMs`, certain set of tobiko tests). Default value: `true`
* `cifmw_test_operator_selinux_level`: (String) Specify SELinux level that should be used for pods spawned with the test-operator. Note, that `cifmw_test_operator_privileged: true` must be set when this parameter has non-empty value. Default value: `s0:c478,c978`
* `cifmw_test_operator_stages`: (List) List of dictionaries defining the stages that should be used in the test operator role. List items options are:
* `name`: (String) The name of the stage. The name must be unique.
* `type`: (String) The framework name you would like to call, currently the options are: tempest, ansibletest, horizontest, tobiko.
* `test_vars_file`: (String) Path to the file used for testing, this file should contain the testing params for this stage. Only parameters specific for the controller can be used (Tempest, Ansibletest, Horizontest and Tobiko).
* `test_vars`: (String) Testing parameters for this specific stage if a `test_vars` is used the specified parameters would override the ones in the `test_vars_file`. Only parameters specific for the controller can be used (Tempest, Ansibletest, Horizontest and Tobiko).
> Important note! Only variables with the following structure can be used to override inside a stage: `cifmw_test_operator_[test-operator CR name]_[parameter name]` the only exception to that is `cifmw_test_operator_concurrency` which can not be override inside a stage.
* `pre_test_stage_hooks`: (List) List of pre hooks to run as described [hooks README](https://github.com/openstack-k8s-operators/ci-framework/tree/main/roles/run_hook#hooks-expected-format).
* `post_test_stage_hooks`: (List) List of post hooks to run as described [hooks README](https://github.com/openstack-k8s-operators/ci-framework/tree/main/roles/run_hook#hooks-expected-format).
Default value:
```
cifmw_test_operator_stages:
- name: tempest
type: tempest
```

## Tempest specific parameters
* `cifmw_test_operator_tempest_name`: (String) Value used in the `Tempest.Metadata.Name` field. The value specifies the name of some resources spawned by the test-operator role. Default value: `tempest-tests`
Expand All @@ -47,8 +60,16 @@ Execute tests via the [test-operator](https://openstack-k8s-operators.github.io/
* `cifmw_test_operator_tempest_network_attachments`: (List) List of network attachment definitions to attach to the tempest pods spawned by test-operator. Default value: `[]`.
* `cifmw_test_operator_tempest_extra_rpms`: (List) . A list of URLs that point to RPMs that should be installed before the execution of tempest. Note that this parameter has no effect when `cifmw_test_operator_tempest_external_plugin` is used. Default value: `[]`
* `cifmw_test_operator_tempest_extra_configmaps_mounts`: (List) A list of configmaps that should be mounted into the tempest test pods. Default value: `[]`
* `cifmw_test_operator_tempest_config`: (Object) Definition of Tempest CRD instance that is passed to the test-operator (see [the test-operator documentation](https://openstack-k8s-operators.github.io/test-operator/crds.html#tempest-custom-resource)). Default value:
* `cifmw_test_operator_tempest_debug`: (Bool) Run Tempest in debug mode, it keeps the operator pod sleeping infinity (it must only set to `true`only for debugging purposes). Default value: `false`
* `cifmw_tempest_tempestconf_config`: Deprecated, please use `cifmw_test_operator_tempest_tempestconf_config` instead
* `cifmw_test_operator_tempest_tempestconf_config`: (Dict) This parameter can be used to customize the execution of the `discover-tempest-config` run. Please consult the test-operator documentation. For example, to pass a custom configuration for `tempest.conf`, use the `overrides` section:
```
cifmw_test_operator_tempest_tempestconf_config:
overrides: |
identity.v3_endpoint_type public
Default value: {}
```
* `cifmw_test_operator_tempest_config`: (Object) Definition of Tempest CRD instance that is passed to the test-operator (see [the test-operator documentation](https://openstack-k8s-operators.github.io/test-operator/crds.html#tempest-custom-resource)). Default value:
```
apiVersion: test.openstack.org/v1beta1
kind: Tempest
Expand All @@ -70,7 +91,7 @@ Execute tests via the [test-operator](https://openstack-k8s-operators.github.io/
externalPlugin: "{{ cifmw_test_operator_tempest_external_plugin | default([]) }}"
extraRPMs: "{{ cifmw_test_operator_tempest_extra_rpms | default([]) }}"
extraImages: "{{ cifmw_test_operator_tempest_extra_images | default([]) }}"
tempestconfRun: "{{ cifmw_tempest_tempestconf_config | default(omit) }}"
tempestconfRun: "{{ cifmw_test_operator_tempest_tempestconf_config | default(omit) }}"
debug: "{{ cifmw_test_operator_tempest_debug }}"
```

Expand Down Expand Up @@ -217,3 +238,23 @@ other than the default one (e.g., `tempest-tests`, `tobiko-tests`, ...):
```
cifmw_test_operator_tempest_name: "post-update-tempest-tests"
```

### Use test operator stages
Test operator stages are meant to allow setting the order of stages,running stages
of the same controller multiple times and using different hooks,vars and names for stages.
```
cifmw_test_operator_stages:
- name: basic-functionallity
type: tempest
test_vars:
cifmw_test_operator_tempest_name: 'basic-functionality-tests'
- name: ansibletest
type: ansibletest
- name: advanced-tests
type: tempest
test_vars:
cifmw_test_operator_tempest_name: 'advanced-tests'
- name: tobiko
type: tobiko
test_vars_file: /path/to/tobiko/override/test/file
```
Loading

0 comments on commit d75e552

Please sign in to comment.