Allow setting order and hooks for test role #2374
Conversation
|
Thanks for the PR! ❤️ |
eshulman2
force-pushed
the
test_operator_stages
branch
from
d749fd6 to
9e7ab2d
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/e1ceb9f2cf3d47658a3de25c1ed70de8 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 39m 16s |
eshulman2
force-pushed
the
test_operator_stages
branch
3 times, most recently
from
f24ddba to
c4de17d
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/a25d6668f88443eeb5b2e354691eb8dc ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 39m 51s |
eshulman2
force-pushed
the
test_operator_stages
branch
2 times, most recently
from
c175381 to
5efbd8f
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/ea84a8ceb689406f85a63f9e9f8d1718 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 39m 40s |
eshulman2
force-pushed
the
test_operator_stages
branch
3 times, most recently
from
1c61cd8 to
34ffac9
Compare
eshulman2
force-pushed
the
test_operator_stages
branch
2 times, most recently
from
c00804c to
e6b67aa
Compare
eshulman2
requested review from
lpiwowar,
lewisdenny and
cjeanner
and removed request for
lpiwowar and
lewisdenny
eshulman2
force-pushed
the
test_operator_stages
branch
from
e6b67aa to
d2b1fed
Compare
eshulman2
force-pushed
the
test_operator_stages
branch
from
d2b1fed to
bbaa00e
Compare
I'd be fine with this change as long as we can merge it without touching a single job. That trick in the defaults will do the trick and I'm all good with it. |
|
@lpiwowar @pablintino updated the patch and related patch with changes in the compromise |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/5ab23c4b7d3f4804b46465355e232cf1 ✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 07m 37s |
|
@eshulman2 👍 the changes you've made look reasonable. Now it is about the testing. |
eshulman2
force-pushed
the
test_operator_stages
branch
from
d75e552 to
c511e9c
Compare
This commit adds key improvements to the test_operator role: Support for defining test execution order. Ability to run the same testing controller multiple times. Pre-hooks and post-hooks for stage-specific customization. Per-stage variable overrides via files or test_vars. Refactored pre-set testing stages to enable more dynamic test management and extend hook behavior for complex scenarios. Jira: OSPRH-10106
eshulman2
force-pushed
the
test_operator_stages
branch
from
c511e9c to
fda3ca9
Compare
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: pablintino The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| cifmw_run_test_role: shiftstack | ||
| cifmw_run_test_shiftstack_testconfig: | ||
| - 4.17_ovnkubernetes_ipi.yaml | ||
| - 4.17_ovnkubernetes_upi.yaml | ||
| cifmw_shiftstack_qa_gerrithub_change: refs/changes/29/1188429/50 #optional | ||
|
|
||
| $ ansible-playbook deploy-edpm.yml -e 'cifmw_run_tests=true cifmw_run_tempest=false cifmw_run_test_role=shiftstack cifmw_openshift_kubeconfig={{ ansible_user_dir }}/.kube/config cifmw_shiftstack_qa_gerrithub_change=refs/changes/29/1188429/50' -e 'cifmw_run_test_shiftstack_testconfig=["4.17_ovnkubernetes_ipi.yaml","4.17_ovnkubernetes_upi.yaml"]' | ||
| $ ansible-playbook deploy-edpm.yml --extra-vars "cifmw_run_tests=true cifmw_run_test_role=shiftstack cifmw_test_operator_stages=[] cifmw_openshift_kubeconfig={{ ansible_user_dir }}/.kube/config cifmw_run_test_shiftstack_testconfig=4.15_ovnkubernetes_ipi_va1.yaml cifmw_shiftstack_qa_gerrithub_change=refs/changes/29/1188429/50" |
There was a problem hiding this comment.
Hey Ella, thanks for adapting the shiftstack part too. I'm suggesting a change (even if it's a md file) just because cifmw_run_test_shiftstack_testconfig should be a list.
| $ ansible-playbook deploy-edpm.yml --extra-vars "cifmw_run_tests=true cifmw_run_test_role=shiftstack cifmw_test_operator_stages=[] cifmw_openshift_kubeconfig={{ ansible_user_dir }}/.kube/config cifmw_run_test_shiftstack_testconfig=4.15_ovnkubernetes_ipi_va1.yaml cifmw_shiftstack_qa_gerrithub_change=refs/changes/29/1188429/50" | |
| $ ansible-playbook deploy-edpm.yml --extra-vars 'cifmw_run_tests=true cifmw_run_test_role=shiftstack cifmw_test_operator_stages=[] cifmw_openshift_kubeconfig={{ ansible_user_dir }}/.kube/config cifmw_run_test_shiftstack_testconfig=["4.17_ovnkubernetes_ipi.yaml","4.17_ovnkubernetes_upi.yaml"] cifmw_shiftstack_qa_gerrithub_change=refs/changes/29/1188429/50' |
There was a problem hiding this comment.
We'll address this in a follow up patch, so ok from shiftstack side!
|
/lgtm Thank you @eshulman2 for sticking with this! I know it has been quite a lot of work. Kudos to you!:) We agreed with @eurijon and @eshulman2 that we are going to address the mild fix in another PR. If nobody proposes one till tomorrow then I'll probably propose it. I do not want this to be forgotten. Also, the README should be a source of truth. |
openshift-merge-bot
bot
merged commit 3b95d56
into
openstack-k8s-operators:main
With this PR in the test-operator role [1] we introduced the concept of stages. By default only tempest stage is enabled. In order for us to continue to test tobiko and ansibletest CR we have to explicitly enable the testing via the cifmw_test_operator_stages parameter. [1] openstack-k8s-operators/ci-framework#2374
With this PR in the test-operator role [1] we introduced the concept of stages. By default only tempest stage is enabled. In order for us to continue to test tobiko and ansibletest CR we have to explicitly enable the testing via the cifmw_test_operator_stages parameter. [1] openstack-k8s-operators/ci-framework#2374
This PR introduces improvements to the test_operator role, enabling ordering of testing stages and hooks for better control over test execution. Key updates include:
Customizable Testing Order:
Introduced the ability to define the testing order for the test_operator role, providing better control over test sequencing.
Repeatable Testing Controller:
Added support for running the same testing controller multiple times within a single test suite.
Stage-Specific Hooks:
Enabled configuration of pre-hooks and post-hooks for each testing stage, offering more granular customization.
Variable Overrides per Stage:
Allowed overriding variables for specific testing stages via either external files (
test_vars_file) or specific stage vars (test_vars) parameter.Refactoring for Enhanced Flexibility
The implementation refactors pre-set testing stage functionality, making test management more adaptable and significantly extending hook behavior when using the CI framework (cifmw). These changes enable the creation of more complex testing workflows, including scenarios requiring multiple and varied hooks or stage-specific configurations.
Impact and Use Cases
These updates allow users to:
Related Issue:
Jira: OSPRH-10106