Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CLDSRV-402: optimize API #5195

Merged
merged 5 commits into from
Jul 13, 2023
Merged

CLDSRV-402: optimize API #5195

merged 5 commits into from
Jul 13, 2023

Conversation

williamlardier
Copy link
Contributor

@williamlardier williamlardier commented Jun 8, 2023 •
edited
Loading

For Artesca, more changes will be introduced with a bump of arsenal, and conflicts resolution (related to code not specific to this PR).

Federation & Integration (green) builds: https://github.com/scality/Integration/actions/runs/5309192284 & https://github.com/scality/Federation/actions/runs/5309145071

Context: The Veeam Ready tests for object with/without immutability are expecting us to perform a data deletion of 4 VM of 100GB each, splitted into 1MB files, in less than 4h. Today Artesca completes this in 10h, and RIng, 6hours.

The objective of this PR is to introduce optimizations for the MultiObjectDelete API:

  • With a batching of the objects metadata reads, if the backends supports it (mongodb only for now).
  • With an increased parallelism (that won't scale perfectly, but it helps dividing the latency for free)
  • With a single call to the storage services, to avoid unecessary iowaits in the API.
  • And finally, for Artesca only, supporting the direct deletion of the object metadata, if the associated bucket permits it (i.e. no notification/lifecycle configurations).

Suggestion for the review: by commit. Changes in the lib/api/multiObjectDelete.js file are not huge, but the diff might display new code, even if only indentation changed...

With these changes:

  • Artesca completes the deletion in 39min instead of 10h.
  • RING to be tested...

PR for 8.5: #5205
8.6: #5206
8.7: #5207

Note 2: the integration changes are around all the breaking changes we have in this API across cloudserver versions, but the implementation is common between 7.x & 8.x.

@bert-e
Copy link
Contributor

bert-e commented Jun 8, 2023

Hello williamlardier,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

@bert-e
Copy link
Contributor

bert-e commented Jun 8, 2023

Incorrect fix version

The Fix Version/s in issue CLDSRV-402 contains:

  • 8.7.24

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 7.70.22

  • 8.5.21

  • 8.6.10

  • 8.7.24

Please check the Fix Version/s of CLDSRV-402, or the target
branch of this pull request.

@williamlardier williamlardier force-pushed the improvement/CLDSRV-402-optimize-multiobjectdelete-api branch 4 times, most recently from f90f036 to 52b73c4 Compare June 15, 2023 07:50
@scality scality deleted a comment from bert-e Jun 15, 2023
@williamlardier williamlardier force-pushed the improvement/CLDSRV-402-optimize-multiobjectdelete-api branch from 52b73c4 to 859f898 Compare June 15, 2023 08:52
@scality scality deleted a comment from bert-e Jun 15, 2023
@scality scality deleted a comment from bert-e Jun 15, 2023
@williamlardier williamlardier force-pushed the improvement/CLDSRV-402-optimize-multiobjectdelete-api branch from 859f898 to d0daff7 Compare June 15, 2023 11:55
@williamlardier
Copy link
Contributor Author

/force_reset

@bert-e
Copy link
Contributor

bert-e commented Jun 15, 2023

Reset complete

I have successfully deleted this pull request's integration branches.

@bert-e
Copy link
Contributor

bert-e commented Jun 15, 2023

Conflict

A conflict has been raised during the creation of
integration branch w/8.5/improvement/CLDSRV-402-optimize-multiobjectdelete-api with contents from improvement/CLDSRV-402-optimize-multiobjectdelete-api
and development/8.5.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/8.5/improvement/CLDSRV-402-optimize-multiobjectdelete-api origin/development/8.5
 $ git merge origin/improvement/CLDSRV-402-optimize-multiobjectdelete-api
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.5/improvement/CLDSRV-402-optimize-multiobjectdelete-api

@scality scality deleted a comment from bert-e Jun 15, 2023
@scality scality deleted a comment from bert-e Jun 15, 2023
@scality scality deleted a comment from bert-e Jun 15, 2023
@scality scality deleted a comment from bert-e Jun 15, 2023
@scality scality deleted a comment from bert-e Jun 15, 2023
@scality scality deleted a comment from bert-e Jun 15, 2023
@williamlardier
Copy link
Contributor Author

ping

@bert-e
Copy link
Contributor

bert-e commented Jun 15, 2023

Conflict

A conflict has been raised during the creation of
integration branch w/8.6/improvement/CLDSRV-402-optimize-multiobjectdelete-api with contents from w/8.5/improvement/CLDSRV-402-optimize-multiobjectdelete-api
and development/8.6.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/8.6/improvement/CLDSRV-402-optimize-multiobjectdelete-api origin/development/8.6
 $ git merge origin/w/8.5/improvement/CLDSRV-402-optimize-multiobjectdelete-api
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.6/improvement/CLDSRV-402-optimize-multiobjectdelete-api

@williamlardier
Copy link
Contributor Author

ping

@bert-e
Copy link
Contributor

bert-e commented Jun 15, 2023

Conflict

A conflict has been raised during the creation of
integration branch w/8.7/improvement/CLDSRV-402-optimize-multiobjectdelete-api with contents from w/8.6/improvement/CLDSRV-402-optimize-multiobjectdelete-api
and development/8.7.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/8.7/improvement/CLDSRV-402-optimize-multiobjectdelete-api origin/development/8.7
 $ git merge origin/w/8.6/improvement/CLDSRV-402-optimize-multiobjectdelete-api
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.7/improvement/CLDSRV-402-optimize-multiobjectdelete-api

@bert-e
Copy link
Contributor

bert-e commented Jun 15, 2023

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

@williamlardier
Copy link
Contributor Author

/create_pull_requests

This was referenced Jun 15, 2023
Closed
Closed
@bert-e
Copy link
Contributor

bert-e commented Jun 15, 2023

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

  • development/7.10
  • development/7.4

Follow integration pull requests if you would like to be notified of
build statuses by email.

The following options are set: create_pull_requests

@bert-e bert-e mentioned this pull request Jun 15, 2023
Closed
@williamlardier williamlardier force-pushed the improvement/CLDSRV-402-optimize-multiobjectdelete-api branch 2 times, most recently from fce82fa to 37288ac Compare July 12, 2023 15:50
@williamlardier
CLDSRV-402: optimize multideleteobject API
9774d31 
- Parallelism is increased to reduce the latency
- If the backend supports it, batching is used
- Batch the deletion of objects from storage
- Flag to disable or enable the optimization, as well as
  a way to tune it.
@williamlardier
CLDSRV-402: test multi object delete optimization
84068b2
@williamlardier
CLDSRV-402: bump arsenal dependency
f7593d3
@williamlardier
CLDSRV-402: bump project version
af0436f
@williamlardier williamlardier force-pushed the improvement/CLDSRV-402-optimize-multiobjectdelete-api branch from 37288ac to af0436f Compare July 13, 2023 07:55
@scality scality deleted a comment from bert-e Jul 13, 2023
@scality scality deleted a comment from bert-e Jul 13, 2023
@williamlardier
Copy link
Contributor Author

/force_reset

@scality scality deleted a comment from bert-e Jul 13, 2023
@scality scality deleted a comment from bert-e Jul 13, 2023
@bert-e
Copy link
Contributor

bert-e commented Jul 13, 2023

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_pull_requests

@bert-e
Copy link
Contributor

bert-e commented Jul 13, 2023

Conflict

A conflict has been raised during the creation of
integration branch w/8.6/improvement/CLDSRV-402-optimize-multiobjectdelete-api with contents from improvement/CLDSRV-402-optimize-multiobjectdelete-api
and development/8.6.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/8.6/improvement/CLDSRV-402-optimize-multiobjectdelete-api origin/development/8.6
 $ git merge origin/improvement/CLDSRV-402-optimize-multiobjectdelete-api
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.6/improvement/CLDSRV-402-optimize-multiobjectdelete-api

The following options are set: create_pull_requests

@scality scality deleted a comment from bert-e Jul 13, 2023
@williamlardier
Copy link
Contributor Author

ping

@bert-e
Copy link
Contributor

bert-e commented Jul 13, 2023

Conflict

A conflict has been raised during the creation of
integration branch w/8.7/improvement/CLDSRV-402-optimize-multiobjectdelete-api with contents from w/8.6/improvement/CLDSRV-402-optimize-multiobjectdelete-api
and development/8.7.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/8.7/improvement/CLDSRV-402-optimize-multiobjectdelete-api origin/development/8.7
 $ git merge origin/w/8.6/improvement/CLDSRV-402-optimize-multiobjectdelete-api
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.7/improvement/CLDSRV-402-optimize-multiobjectdelete-api

The following options are set: create_pull_requests

@williamlardier
Copy link
Contributor Author

ping

This was referenced Jul 13, 2023
Merged
Merged
@bert-e
Copy link
Contributor

bert-e commented Jul 13, 2023

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

  • development/7.10
  • development/7.4

Follow integration pull requests if you would like to be notified of
build statuses by email.

The following options are set: create_pull_requests

@bert-e bert-e mentioned this pull request Jul 13, 2023
Merged
@bert-e
Copy link
Contributor

bert-e commented Jul 13, 2023

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

The following options are set: create_pull_requests

@scality scality deleted a comment from bert-e Jul 13, 2023
@williamlardier
Copy link
Contributor Author

/approve

@bert-e
Copy link
Contributor

bert-e commented Jul 13, 2023

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

  • ✔️ development/7.70

  • ✔️ development/8.6

  • ✔️ development/8.7

  • ✔️ development/8.8

The following branches will NOT be impacted:

  • development/7.10
  • development/7.4

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

  • Any commit you add on the source branch will trigger a new cycle after the
    current queue is merged.
  • Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve, create_pull_requests

@bert-e
Copy link
Contributor

bert-e commented Jul 13, 2023

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/7.70

  • ✔️ development/8.6

  • ✔️ development/8.7

  • ✔️ development/8.8

The following branches have NOT changed:

  • development/7.10
  • development/7.4

Please check the status of the associated issue CLDSRV-402.

Goodbye williamlardier.

@bert-e bert-e merged commit 68e4b06 into development/7.70 Jul 13, 2023
15 checks passed
@bert-e bert-e deleted the improvement/CLDSRV-402-optimize-multiobjectdelete-api branch July 13, 2023 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@francoisferrand francoisferrand francoisferrand left review comments

@jonathan-gramain jonathan-gramain jonathan-gramain approved these changes

@KazToozs KazToozs KazToozs approved these changes

@Kerkesni Kerkesni Awaiting requested review from Kerkesni

Assignees

@KazToozs KazToozs

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@williamlardier @bert-e @francoisferrand @jonathan-gramain @KazToozs