Feature: ARSN-235 update object before deleting it #1944

Kerkesni · 2022-06-30T15:31:56Z

Note: The full changes are in the integration PR

Due to Armory work, any changes affecting the MongoClientInterface needs to be pushed both in 7.x and 8.x.
Some tests are incompatible with 7.x due to the ObjectMD model being different. These tests were skipped in 7.x.

Two new properties were added to the ObjectMD (deleted, isPHD). The commits are only applied to the 8.x branch for obvious reasons.

Bucket Notification as well as Cold Storage need object metadata when this one gets deleted. The MongoDB oplog's delete events do not contain any of the metadata of the deleted object. They only contain the "_id" of the object.

This issue was addressed by updating the metadata of the object while adding a new tag that shows that the object is being deleted, and then immediately deleting the object. This will add an update event to the oplog containing all of the needed metadata.

Cloudserver Build : https://eve.devsca.com/github/scality/cloudserver/#/builders/15/builds/9413

bert-e · 2022-06-30T15:31:58Z

Hello kerkesni,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

codecov-commenter · 2022-09-09T12:35:46Z

Codecov Report

Merging #1944 (5b2d50b) into development/8.1 (4e8a907) will increase coverage by 0.05%.
The diff coverage is 86.48%.

@@                 Coverage Diff                 @@
##           development/8.1    #1944      +/-   ##
===================================================
+ Coverage            63.83%   63.89%   +0.05%     
===================================================
  Files                  206      206              
  Lines                15593    15641      +48     
  Branches              3179     3186       +7     
===================================================
+ Hits                  9954     9994      +40     
- Misses                5623     5631       +8     
  Partials                16       16

Impacted Files	Coverage Δ
...orage/metadata/mongoclient/MongoClientInterface.js	`67.65% <83.87%> (+0.21%)`	⬆️
lib/constants.ts	`100.00% <100.00%> (ø)`
lib/models/ObjectMD.ts	`89.47% <100.00%> (+0.78%)`	⬆️
lib/storage/metadata/mongoclient/readStream.js	`68.29% <100.00%> (+0.39%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Kerkesni · 2022-09-12T15:24:42Z

/create_pull_requests

lib/storage/metadata/mongoclient/MongoClientInterface.js

Kerkesni · 2022-11-10T17:03:16Z

/force_reset

bert-e · 2022-11-10T17:03:21Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_pull_requests

bert-e · 2022-11-10T17:03:31Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge feature/ARSN-235-update-object-before-deleting-it into
development/7.10
w/8.1/feature/ARSN-235-update-object-before-deleting-it will be merged into development/8.1 (pull request INTEGRATION [PR#1944 > development/8.1] Feature: ARSN-235 update object before deleting it #2039)

The following branches will NOT be impacted:

development/6.4
development/7.4

Follow integration pull requests if you would like to be notified of
build statuses by email.

The following options are set: create_pull_requests

bert-e · 2022-11-10T17:03:33Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_pull_requests

Object deletion no longer directly deletes the object, it first updates its metadata by setting the deletion flag and originOp then proceeds to deleting the object. This is done to keep a trace of the latest object metadata before deletion in the oplog, as oplog delete events don't hold that information. This information is needed for both Cold Storage and Bucket Notification We also add all the object metadata to the placeholder (PHD) master which wasn't previously the case, again this is done to keep the metadata in the oplog as a PHD might get directly deleted in the repair phase.

…ject

…jects

Kerkesni · 2022-11-13T21:13:41Z

/force_reset

bert-e · 2022-11-13T21:13:47Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_pull_requests

bert-e · 2022-11-13T21:16:20Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge feature/ARSN-235-update-object-before-deleting-it into
development/7.10
w/8.1/feature/ARSN-235-update-object-before-deleting-it will be merged into development/8.1 (pull request INTEGRATION [PR#1944 > development/8.1] Feature: ARSN-235 update object before deleting it #2041)

The following branches will NOT be impacted:

development/6.4
development/7.4

Follow integration pull requests if you would like to be notified of
build statuses by email.

The following options are set: create_pull_requests

bert-e · 2022-11-13T21:16:22Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_pull_requests

Kerkesni · 2022-11-14T09:20:09Z

/approve

bert-e · 2022-11-14T09:20:19Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/7.10
✔️ development/8.1

The following branches will NOT be impacted:

development/6.4
development/7.4

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: create_pull_requests, approve

bert-e · 2022-11-14T09:27:12Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/7.10
✔️ development/8.1

The following branches have NOT changed:

development/6.4
development/7.4

Please check the status of the associated issue ARSN-235.

Goodbye kerkesni.

Kerkesni changed the base branch from development/7.10 to development/8.1 June 30, 2022 15:33

Kerkesni force-pushed the feature/ARSN-235-update-object-before-deleting-it branch 3 times, most recently from a958ad3 to e96f814 Compare September 9, 2022 12:29

Kerkesni force-pushed the feature/ARSN-235-update-object-before-deleting-it branch 3 times, most recently from 5b2d50b to a4ec247 Compare September 12, 2022 14:23

Kerkesni changed the base branch from development/8.1 to development/7.10 September 12, 2022 15:18

Kerkesni force-pushed the feature/ARSN-235-update-object-before-deleting-it branch from a4ec247 to a5128af Compare September 12, 2022 15:19

Kerkesni marked this pull request as ready for review September 12, 2022 15:21

bert-e mentioned this pull request Sep 12, 2022

INTEGRATION [PR#1944 > development/8.1] Feature: ARSN-235 update object before deleting it #2007

Closed

Kerkesni force-pushed the feature/ARSN-235-update-object-before-deleting-it branch from a5128af to 90e5231 Compare September 12, 2022 16:02

bert-e mentioned this pull request Sep 13, 2022

INTEGRATION [PR#1944 > development/8.1] Feature: ARSN-235 update object before deleting it #2008

Closed

Kerkesni requested review from alexanderchan-scality, francoisferrand and rachedbenmustapha September 13, 2022 10:47

Kerkesni force-pushed the feature/ARSN-235-update-object-before-deleting-it branch from 90e5231 to f8bd238 Compare September 13, 2022 13:50

bert-e mentioned this pull request Sep 13, 2022

INTEGRATION [PR#1944 > development/8.1] Feature: ARSN-235 update object before deleting it #2009

Closed

Kerkesni force-pushed the feature/ARSN-235-update-object-before-deleting-it branch from f8bd238 to bbfab35 Compare September 13, 2022 13:56

bert-e mentioned this pull request Sep 13, 2022

INTEGRATION [PR#1944 > development/8.1] Feature: ARSN-235 update object before deleting it #2010

Closed

Kerkesni mentioned this pull request Sep 13, 2022

Bugfix: BB-159 version delete not producing notification scality/backbeat#2165

Merged

francoisferrand reviewed Sep 15, 2022

View reviewed changes

lib/storage/metadata/mongoclient/MongoClientInterface.js Show resolved Hide resolved

Kerkesni requested review from francoisferrand and jbertran September 20, 2022 12:10

francoisferrand requested a review from nicolas2bert October 3, 2022 09:23

scality deleted a comment from bert-e Oct 6, 2022

Kerkesni force-pushed the feature/ARSN-235-update-object-before-deleting-it branch from abc9f56 to 7e25c9b Compare November 10, 2022 17:02

scality deleted a comment from bert-e Nov 10, 2022

bert-e mentioned this pull request Nov 10, 2022

INTEGRATION [PR#1944 > development/8.1] Feature: ARSN-235 update object before deleting it #2039

Closed

Kerkesni requested a review from nicolas2bert November 10, 2022 17:07

nicolas2bert approved these changes Nov 10, 2022

View reviewed changes

Kerkesni added 5 commits November 13, 2022 22:04

feature: ARSN-235 ignore objects flagged for deletion when getting ob…

d852eef

…ject

feature: ARSN-235 ignore objects flagged for deletion when listing ob…

b61138a

…jects

feature: ARSN-235 fix unit tests

ef024dd

feature: ARSN-235 document oplog

9a97572

Kerkesni force-pushed the feature/ARSN-235-update-object-before-deleting-it branch from 7e25c9b to 9a97572 Compare November 13, 2022 21:04

scality deleted a comment from bert-e Nov 13, 2022

bert-e mentioned this pull request Nov 13, 2022

INTEGRATION [PR#1944 > development/8.1] Feature: ARSN-235 update object before deleting it #2041

Merged

scality deleted a comment from bert-e Nov 13, 2022

bert-e merged commit 7c1bd45 into development/7.10 Nov 14, 2022

bert-e deleted the feature/ARSN-235-update-object-before-deleting-it branch November 14, 2022 09:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: ARSN-235 update object before deleting it #1944

Feature: ARSN-235 update object before deleting it #1944

Kerkesni commented Jun 30, 2022 •

edited

Loading

bert-e commented Jun 30, 2022

codecov-commenter commented Sep 9, 2022 •

edited

Loading

Kerkesni commented Sep 12, 2022

Kerkesni commented Nov 10, 2022

bert-e commented Nov 10, 2022

bert-e commented Nov 10, 2022

bert-e commented Nov 10, 2022

Kerkesni commented Nov 13, 2022

bert-e commented Nov 13, 2022

bert-e commented Nov 13, 2022

bert-e commented Nov 13, 2022

Kerkesni commented Nov 14, 2022

bert-e commented Nov 14, 2022

bert-e commented Nov 14, 2022

Feature: ARSN-235 update object before deleting it #1944

Feature: ARSN-235 update object before deleting it #1944

Conversation

Kerkesni commented Jun 30, 2022 • edited Loading

bert-e commented Jun 30, 2022

Hello kerkesni,

codecov-commenter commented Sep 9, 2022 • edited Loading

Codecov Report

Kerkesni commented Sep 12, 2022

Kerkesni commented Nov 10, 2022

bert-e commented Nov 10, 2022

Reset complete

bert-e commented Nov 10, 2022

Integration data created

bert-e commented Nov 10, 2022

Waiting for approval

Kerkesni commented Nov 13, 2022

bert-e commented Nov 13, 2022

Reset complete

bert-e commented Nov 13, 2022

Integration data created

bert-e commented Nov 13, 2022

Waiting for approval

Kerkesni commented Nov 14, 2022

bert-e commented Nov 14, 2022

In the queue

bert-e commented Nov 14, 2022

Kerkesni commented Jun 30, 2022 •

edited

Loading

codecov-commenter commented Sep 9, 2022 •

edited

Loading