Cleanup created accounts and buckets after tests #2152

williamlardier · 2024-09-20T14:11:29Z

Without cleanup, the GetRolesForWebIdentity API ends up being very slow, eventually causing Vault to be unresponsive and thus causing timeouts, and tests failures. This API is not supposed to work with more than 100 accounts (as per requirements), and here, we end up with >1000, meaning at least this number of internal (and sequential) DB calls per "GetRolesForWebIdentity" API call.

The logic detects that we created an account (during cross-account tests), and delete all resources, before deleting the account.

Issue: ZENKO-4898

bert-e · 2024-09-20T14:11:33Z

Hello williamlardier,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2024-09-20T14:19:14Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

bert-e · 2024-09-23T07:28:56Z

Incorrect fix version

The Fix Version/s in issue ZENKO-4898 contains:

2.10.0
2.6.68
2.7.64
2.8.44
2.9.20

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

2.10.0
2.6.68
2.7.64
2.8.44
2.9.21

Please check the Fix Version/s of ZENKO-4898, or the target
branch of this pull request.

bert-e · 2024-09-23T08:47:52Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

williamlardier · 2024-09-24T08:54:49Z

/create_integration_branches

bert-e · 2024-09-24T08:57:12Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/ZENKO-4898 into
development/2.6
w/2.7/bugfix/ZENKO-4898 will be merged into development/2.7
w/2.8/bugfix/ZENKO-4898 will be merged into development/2.8
w/2.9/bugfix/ZENKO-4898 will be merged into development/2.9
w/2.10/bugfix/ZENKO-4898 will be merged into development/2.10

The following branches will NOT be impacted:

development/2.5

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-09-24T08:57:14Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

.github/actions/archive-artifacts/action.yaml

tests/ctst/common/hooks.ts

tests/ctst/world/Zenko.ts

tests/ctst/steps/utils/utils.ts

tests/ctst/steps/pra.ts

.github/actions/archive-artifacts/action.yaml

bert-e · 2024-09-25T09:17:08Z

Conflict

A conflict has been raised during the update of
integration branch w/2.7/bugfix/ZENKO-4898 with contents from bugfix/ZENKO-4898
and development/2.7.

Please resolve the conflict on the integration branch (w/2.7/bugfix/ZENKO-4898).

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout w/2.7/bugfix/ZENKO-4898
 $ git pull  # or "git reset --hard origin/w/2.7/bugfix/ZENKO-4898"
 $ git merge origin/development/2.7
 $ # <intense conflict resolution>
 $ git commit
 $ git merge origin/bugfix/ZENKO-4898
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.7/bugfix/ZENKO-4898

The following options are set: create_integration_branches

- Assume role tests create additional accounts that we must clean at the end of any (successful) scenario. - This ensures that the GetRolesForWebIdentity calls are not impacted by the high number of accounts. Issue: ZENKO-4898

- Quotas are not needed, plus, they cause errors when using the env variable in a kubectl command. Issue: ZENKO-4898

- Tests should cleanup resources, unless a scenario failed - Use of mongodump and bsondump for fast speed Issue: ZENKO-4898

bert-e · 2024-09-26T10:17:40Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/ZENKO-4898 into
development/2.6
w/2.7/bugfix/ZENKO-4898 will be merged into development/2.7
w/2.8/bugfix/ZENKO-4898 will be merged into development/2.8
w/2.9/bugfix/ZENKO-4898 will be merged into development/2.9
w/2.10/bugfix/ZENKO-4898 will be merged into development/2.10

The following branches will NOT be impacted:

development/2.5

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-09-26T10:17:41Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

tests/ctst/common/common.ts

tests/ctst/steps/utils/utils.ts

- Any bucket with Compliance retention is ignored - Governance is accepted as we delete the bucket with the account identity. Issue: ZENKO-4898

Issue: ZENKO-4898

williamlardier · 2024-10-08T12:19:46Z

/help

bert-e · 2024-10-08T12:19:51Z

Help page

The following options and commands are available at this time.

Options

name	description	privileged	authored
➡️ after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
➡️ bypass_author_approval	Bypass the pull request author's approval	⭐
➡️ bypass_build_status	Bypass the build and test status	⭐
➡️ bypass_commit_size	Bypass the check on the size of the changeset `TBA`	⭐
➡️ bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
➡️ bypass_jira_check	Bypass the Jira issue check	⭐
➡️ bypass_peer_approval	Bypass the pull request peers' approval	⭐
➡️ bypass_leader_approval	Bypass the pull request leaders' approval	⭐
➡️ approve	Instruct Bert-E that the author has approved the pull request.		✍️
➡️ create_pull_requests	Allow the creation of integration pull requests.
➡️ create_integration_branches	Allow the creation of integration branches.
➡️ no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
➡️ unanimity	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
➡️ wait	Instruct Bert-E not to run until further notice.

Commands

name	description	privileged
➡️ help	Print Bert-E's manual in the pull request.
➡️ status	Print Bert-E's current status in the pull request `TBA`
➡️ clear	Remove all comments from Bert-E from the history `TBA`
➡️ retry	Re-start a fresh build `TBA`
➡️ build	Re-start a fresh build `TBA`
➡️ force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
➡️ reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

The following options are set: create_integration_branches

bert-e · 2024-10-08T12:20:03Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

williamlardier · 2024-10-08T12:20:47Z

/after_pull_request=2160

bert-e · 2024-10-08T12:20:52Z

Waiting for other pull request(s)

The current pull request is locked by the after_pull_request option.

In order for me to merge this pull request, run the following actions first:

➡️ Merge the OPEN pull request:

pull request Bugfix/zenko 4912 #2160

Alternatively, delete all the after_pull_request comments from this pull request.

The following options are set: after_pull_request, create_integration_branches

williamlardier · 2024-10-08T12:21:05Z

/approve

bert-e · 2024-10-08T12:21:10Z

Waiting for other pull request(s)

The current pull request is locked by the after_pull_request option.

In order for me to merge this pull request, run the following actions first:

➡️ Merge the OPEN pull request:

pull request Bugfix/zenko 4912 #2160

Alternatively, delete all the after_pull_request comments from this pull request.

The following options are set: after_pull_request, approve, create_integration_branches

williamlardier · 2024-10-09T08:03:57Z

ping

bert-e · 2024-10-09T08:04:18Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.6
✔️ development/2.7
✔️ development/2.8
✔️ development/2.9
✔️ development/2.10

The following branches will NOT be impacted:

development/2.5

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: after_pull_request, approve, create_integration_branches

bert-e · 2024-10-09T09:12:57Z

Queue build failed

The corresponding build for the queue failed:

Checkout the status page.
Identify the failing build and review the logs.
If no issue is found, re-run the build.
If an issue is identified, checkout the steps below to remove
the pull request from the queue for further analysis and maybe rebase/merge.

Remove the pull request from the queue

Add a /wait comment on this pull request.
Click on login on the status page.
Go into the manage page.
Find the option called Rebuild the queue and click on it.
Bert-E will loop again on all pull requests to put the valid ones
in the queue again, while skipping the one with the /wait comment.
Wait for the new queue to merge, then merge/rebase your pull request
with the latest changes to then work on a proper fix.
Once the issue is fixed, delete the /wait comment and
follow the usual process to merge the pull request.

bert-e · 2024-10-09T12:33:34Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.6
✔️ development/2.7
✔️ development/2.8
✔️ development/2.9
✔️ development/2.10

The following branches have NOT changed:

development/2.5

Please check the status of the associated issue ZENKO-4898.

Goodbye williamlardier.

williamlardier force-pushed the bugfix/ZENKO-4898 branch from 6162a8c to 8f4cd25 Compare September 20, 2024 14:16

scality deleted a comment from bert-e Sep 20, 2024

williamlardier force-pushed the bugfix/ZENKO-4898 branch 5 times, most recently from 241c291 to eea904c Compare September 24, 2024 08:54

scality deleted a comment from bert-e Sep 24, 2024

williamlardier commented Sep 24, 2024

View reviewed changes

.github/actions/archive-artifacts/action.yaml Outdated Show resolved Hide resolved