Cleanup tempest related resources after testing #208
Conversation
There was a problem hiding this comment.
This will run tempest cleanup at the end of execution of each test pod. This means that we are going to be downloading and uploading images each time a new pod is spawned (after the necessary fix I proposed).
Meaning, if there are 3 workflow steps we are going to be downloading and deleting image 3 times. Considering that some of the images are big it would be waste of time (we are talking 15 mins per download).
Here are two things that we could do:
- Modify this patch so that the cleanup behavior can be turned on / off through env variable (default off)
- Allow setting this parameter through Tempest CR and test_operator role.
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
kopecmartin
force-pushed
the
tempest_cleanup
branch
from
e914064 to
68d76c2
Compare
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
kopecmartin
force-pushed
the
tempest_cleanup
branch
2 times, most recently
from
e0dff7b to
f37dab7
Compare
There was a problem hiding this comment.
tempet cleanup command is failing on missing credentials:
+ tempest cleanup --init-saved-state
Failure during cleanup
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/cmd/cleanup.py", line 153, in take_action
self.init(parsed_args)
File "/usr/lib/python3.9/site-packages/tempest/cmd/cleanup.py", line 183, in init
credentials.get_configured_admin_credentials())
File "/usr/lib/python3.9/site-packages/tempest/common/credentials_factory.py", line 257, in get_configured_admin_credentials
credentials = get_credentials(fill_in=fill_in,
File "/usr/lib/python3.9/site-packages/tempest/common/credentials_factory.py", line 300, in get_credentials
return auth.get_credentials(auth_url,
File "/usr/lib/python3.9/site-packages/tempest/lib/auth.py", line 649, in get_credentials
auth_provider = auth_provider_class(
File "/usr/lib/python3.9/site-packages/tempest/lib/auth.py", line 263, in __init__
super(KeystoneAuthProvider, self).__init__(credentials, scope)
File "/usr/lib/python3.9/site-packages/tempest/lib/auth.py", line 91, in __init__
raise exceptions.InvalidCredentials(message)
tempest.lib.exceptions.InvalidCredentials: Invalid Credentials
Details: Credentials are: {'domain_id': None, 'domain_name': 'Default', 'username': None, 'project_domain_id': None, 'project_domain_name': 'Default', 'project_id': None, 'project_name': None, 'tenant_id': None, 'tenant_name': None, 'user_domain_id': None, 'user_domain_name': 'Default', 'user_id': None, 'system': None} Password is not defined.
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tempest/cmd/cleanup.py", line 153, in take_action
self.init(parsed_args)
File "/usr/lib/python3.9/site-packages/tempest/cmd/cleanup.py", line 183, in init
credentials.get_configured_admin_credentials())
File "/usr/lib/python3.9/site-packages/tempest/common/credentials_factory.py", line 257, in get_configured_admin_credentials
credentials = get_credentials(fill_in=fill_in,
File "/usr/lib/python3.9/site-packages/tempest/common/credentials_factory.py", line 300, in get_credentials
return auth.get_credentials(auth_url,
File "/usr/lib/python3.9/site-packages/tempest/lib/auth.py", line 649, in get_credentials
auth_provider = auth_provider_class(
File "/usr/lib/python3.9/site-packages/tempest/lib/auth.py", line 263, in __init__
super(KeystoneAuthProvider, self).__init__(credentials, scope)
File "/usr/lib/python3.9/site-packages/tempest/lib/auth.py", line 91, in __init__
raise exceptions.InvalidCredentials(message)
tempest.lib.exceptions.InvalidCredentials: Invalid Credentials
and
+ '[' false == false
/var/lib/tempest/run_tempest.sh: line 368: [: missing `]'
f37dab7 to
397e41e
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/0f0d572ed4af4194a98fbeecbd311ec5 ✔️ openstack-meta-content-provider SUCCESS in 2h 28m 18s |
|
Missing brackets: (I missed the brackets in the comment above) (Note I'm not sure whether this PR contains fix for the missing credentials error mentioned in the previous comment). |
kopecmartin
force-pushed
the
tempest_cleanup
branch
from
397e41e to
fe746d7
Compare
|
oh, the creds error is related to the fact that now we run 'tempest cleanup' before we have a tempest.conf .. we need to run discover-tempest-config first |
We have to find either:
|
kopecmartin
force-pushed
the
tempest_cleanup
branch
2 times, most recently
from
09fecf1 to
90d23e1
Compare
|
done, the easier way will be to generate a simple tempest.conf containing basically the default settings |
kopecmartin
force-pushed
the
tempest_cleanup
branch
2 times, most recently
from
ce710bb to
30197fb
Compare
kopecmartin
force-pushed
the
tempest_cleanup
branch
from
30197fb to
261ac68
Compare
There was a problem hiding this comment.
Looks good to me! 👍
I've just added one comment. What do you think @kopecmartin?
kopecmartin
force-pushed
the
tempest_cleanup
branch
from
261ac68 to
5a973e2
Compare
Let's run tempest cleanup after the tempest run if TEMPEST_CLEANUP is set to true (false by default and debug mode is not enabled, to delete any leftover resources (from the tempest run) so that they won't disrupt any further testing - testing frameworks executed after tempest.
kopecmartin
force-pushed
the
tempest_cleanup
branch
from
5a973e2 to
97ad930
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED Approval requirements bypassed by manually added approval. This pull-request has been approved by: ashu-011, kopecmartin, lpiwowar The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-merge-bot
bot
merged commit 41545a5
into
openstack-k8s-operators:main
Allow users to enable tempest cleanup feature defined in the tempest image. openstack-k8s-operators/tcib#208 openstack-k8s-operators/test-operator#207
Allow users to enable tempest cleanup feature defined in the tempest image. openstack-k8s-operators/tcib#208 openstack-k8s-operators/test-operator#207
Let's run tempest cleanup after the tempest run, unless debug mode is enabled, to delete any leftover resources (from the tempest run) so that they won't disrupt any further testing - testing frameworks executed after tempest.