feature: ZENKO-2089 add mongodb conditionals #913

alexanderchan-scality · 2019-11-22T01:41:23Z

add conditional put/delete operations to allow for correct blobserver behaviors

update:

adds dev dependency mongodb-memory-server

bert-e · 2019-11-22T01:41:25Z

Hello alexanderchan-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2019-11-22T01:41:30Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

rachedbenmustapha

This change seems to be breaking some abstraction here, the metadata interface is backend-agnostic. I suggest collecting which filters are going to be useful, and pass them as objects that are part of Arsenal's API and backend-agnostic, then doing the translation in the backend-specific impl (in this case MongoClientInterface).

Also the metadata wrapper should return an internal error if the backend does not support the translation, to avoid unexpected issues.

rachedbenmustapha · 2019-11-22T01:46:34Z

lib/storage/metadata/MetadataWrapper.js

@@ -356,6 +356,12 @@ class MetadataWrapper {
    notifyBucketChange(cb) {
        bucketNotificationHook = cb;
    }
+
+    close() {
+        if (this.client.close === 'function') {


Missing typeof?

rachedbenmustapha · 2019-11-22T02:01:44Z

lib/storage/metadata/mongoclient/MongoClientInterface.js

@@ -622,6 +662,10 @@ class MongoClientInterface {
            return this.putObjectVerCase4(c, bucketName, objName, objVal,
                                          params, log, cb);
        }
+        if (params && params.query) {
+            return this.putObjectNoVerConditional(


Is this only the NoVer variant because blobserver never supports versioning? What would happen if blobserver were to try updating/deleting a versioned object put by cloudserver?

bert-e · 2019-11-22T02:03:30Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

@rachedbenmustapha

vrancurel · 2019-12-02T18:36:22Z

This change seems to be breaking some abstraction here, the metadata interface is backend-agnostic. I suggest collecting which filters are going to be useful, and pass them as objects that are part of Arsenal's API and backend-agnostic, then doing the translation in the backend-specific impl (in this case MongoClientInterface).

Also the metadata wrapper should return an internal error if the backend does not support the translation, to avoid unexpected issues.

We made the abstraction mistake for search, can we fusion / factorize with search (whom parsing / execution logic lies in CS instead of Arsenal) with those conditional operators ?

bert-e · 2019-12-02T18:36:28Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

@rachedbenmustapha

rachedbenmustapha · 2019-12-02T18:52:13Z

lib/storage/metadata/mongoclient/MongoClientInterface.js

+                return cb(errors.InternalError);
+            }
+            if (!res.value) {
+                log.debug('object not found...inserting object', {


Since the action is already done, the log should use past tense to avoid confusion (can be surprising when troubleshooting that the DB does not reflect what the log says).

rachedbenmustapha · 2019-12-02T19:01:22Z

@vrancurel I'm not sure it would be appropriate to merge them. This particular abstraction is fine for programmatic use of the API, JSON is easy to build this way. But we can't expect users to type JSON in the search bar, and using a human-friendly search syntax for API use (especially implemented in javascript, in critical request path such as upsert/conditional writes) would be wasteful.

The search syntax definitely needs to be re-done, and the new parser could definitely translate to this data structure for convenience, but I don't think they are the same abstraction level.

For best impact, the human-friendly syntax would be like lucene/elasticsearch, as many users have used it one way or another, and its basic form is similar enough to a google search.

changed

bert-e · 2019-12-04T19:30:00Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

JianqinWang

other than the .only LGTM

tests/unit/storage/metadata/mongoclient/utils.js

vrancurel · 2019-12-10T23:19:38Z

I would prefer renaming Query to Cond, @rachedbenmustapha ?

add conditional put/delete operations to allow for correct blobserver behaviors

rachedbenmustapha · 2019-12-13T00:59:32Z

@vrancurel wouldn't these new cond primitives make the versioning code a lot cleaner?

alexanderchan-scality · 2019-12-13T17:37:57Z

/approve

bert-e · 2019-12-13T17:38:06Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/8.1

The following branches will NOT be impacted:

development/6.4
development/7.4
development/7.6
stabilization/7.4.5

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2019-12-13T17:38:13Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/8.1

The following branches have NOT changed:

development/6.4
development/7.4
development/7.6
stabilization/7.4.5

Please check the status of the associated issue ZENKO-2089.

Goodbye alexanderchan-scality.

alexanderchan-scality requested review from vrancurel and jonathan-gramain November 22, 2019 01:41

rachedbenmustapha requested changes Nov 22, 2019

View reviewed changes

alexanderchan-scality force-pushed the feature/ZENKO-2089/mongdbConditionals branch 3 times, most recently from a13d547 to f967b75 Compare November 23, 2019 02:37

rachedbenmustapha previously requested changes Dec 2, 2019

View reviewed changes

alexanderchan-scality force-pushed the feature/ZENKO-2089/mongdbConditionals branch 2 times, most recently from dbb19c0 to 2f771d3 Compare December 4, 2019 18:43

JianqinWang reviewed Dec 4, 2019

View reviewed changes

tests/unit/storage/metadata/mongoclient/utils.js Outdated Show resolved Hide resolved

JianqinWang approved these changes Dec 4, 2019

View reviewed changes

alexanderchan-scality force-pushed the feature/ZENKO-2089/mongdbConditionals branch from 4e0297f to 5125724 Compare December 11, 2019 19:41

vrancurel approved these changes Dec 11, 2019

View reviewed changes

feature: ZENKO-2089 add mongodb conditionals

ad58f66

add conditional put/delete operations to allow for correct blobserver behaviors

alexanderchan-scality force-pushed the feature/ZENKO-2089/mongdbConditionals branch from 5125724 to ad58f66 Compare December 12, 2019 00:03

rachedbenmustapha approved these changes Dec 13, 2019

View reviewed changes

bert-e merged commit ad58f66 into development/8.1 Dec 13, 2019

bert-e deleted the feature/ZENKO-2089/mongdbConditionals branch December 13, 2019 17:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: ZENKO-2089 add mongodb conditionals #913

feature: ZENKO-2089 add mongodb conditionals #913

alexanderchan-scality commented Nov 22, 2019 •

edited

Loading

bert-e commented Nov 22, 2019

bert-e commented Nov 22, 2019

rachedbenmustapha left a comment

rachedbenmustapha Nov 22, 2019

rachedbenmustapha Nov 22, 2019

bert-e commented Nov 22, 2019

vrancurel commented Dec 2, 2019

bert-e commented Dec 2, 2019

rachedbenmustapha Dec 2, 2019

rachedbenmustapha commented Dec 2, 2019

bert-e commented Dec 4, 2019

JianqinWang left a comment

vrancurel commented Dec 10, 2019

rachedbenmustapha commented Dec 13, 2019

alexanderchan-scality commented Dec 13, 2019

bert-e commented Dec 13, 2019

bert-e commented Dec 13, 2019

feature: ZENKO-2089 add mongodb conditionals #913

feature: ZENKO-2089 add mongodb conditionals #913

Conversation

alexanderchan-scality commented Nov 22, 2019 • edited Loading

bert-e commented Nov 22, 2019

Hello alexanderchan-scality,

bert-e commented Nov 22, 2019

Waiting for approval

rachedbenmustapha left a comment

Choose a reason for hiding this comment

rachedbenmustapha Nov 22, 2019

Choose a reason for hiding this comment

rachedbenmustapha Nov 22, 2019

Choose a reason for hiding this comment

bert-e commented Nov 22, 2019

Waiting for approval

vrancurel commented Dec 2, 2019

bert-e commented Dec 2, 2019

Waiting for approval

rachedbenmustapha Dec 2, 2019

Choose a reason for hiding this comment

rachedbenmustapha commented Dec 2, 2019

bert-e commented Dec 4, 2019

Waiting for approval

JianqinWang left a comment

Choose a reason for hiding this comment

vrancurel commented Dec 10, 2019

rachedbenmustapha commented Dec 13, 2019

alexanderchan-scality commented Dec 13, 2019

bert-e commented Dec 13, 2019

In the queue

bert-e commented Dec 13, 2019

alexanderchan-scality commented Nov 22, 2019 •

edited

Loading