Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZENKO-1124: mongo listing, avoid to loop #563

Merged
merged 1 commit into from
Sep 11, 2018

Conversation

jeremyds
Copy link

No description provided.

@bert-e
Copy link
Contributor

bert-e commented Sep 10, 2018

Hello jeremyds,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get
information on this process.

Status report is not available.

@bert-e
Copy link
Contributor

bert-e commented Sep 10, 2018

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • 2 peers

Peer approvals must include at least 1 approval from the following list:

Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a hard time understanding the issue here.
My guess is that it's related to the changes in #559
That being said, with what I read on the other PR, I fear that we are created new corner cases where NextMarker isn't properly updated, and we end up with a skipping() == SKIP_NONE; and I'm also unsure about the effect of that on this code.
Need an offline conversation to clarify this.

@rahulreddy
Copy link
Collaborator

@DavidPineauScality This is related to https://scality.atlassian.net/browse/ZENKO-1124 , observed at a customer.

@ghost
Copy link

ghost commented Sep 10, 2018

I understand that, but my issue is that I don't understand how the new/alternative fix in #559 works, and I'm worried whether the fix here should also land in MetaData.
So I'm a little lost.

@rahulreddy
Copy link
Collaborator

Oh I was only commenting on what this issue is related to. As for the changes, I don't understand them either, it's better to schedule a call with @jeremyds.

@vrancurel
Copy link
Contributor

In theory @DavidPineauScality you indeed have to change Metadata code.

We may factorize this function in Arsenal though since it is pure common code.

Copy link
Collaborator

@rahulreddy rahulreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will be discussing the solution offline with @jeremyds . I built a custom image and verified that the fix works.

@ghost
Copy link

ghost commented Sep 10, 2018

Ok, thanks for the heads up.
I'll leave that up to you guys, but if my feedback/understanding/review is still necessary, I'm available for a sync-up

@jeremyds
Copy link
Author

Sorry guys, I did not communicate well on the 2 PR. The other one, the fix in delimiterMaster, does not work. I tried to find a solution in the delimiterMaster code but each solution I found introduced (sometime subtle) regressions or behavior changes I was not confident with.
I finally fix the infinite loop on the same range bug in metadata in this PR. We don't use the skipping range returned by the skipping function when it is the same that the actual one, avoiding to loop on it.

Vianney closed the other PR, I should have done it yesterday.

@jeremyds jeremyds force-pushed the bugfix/ZENKO-1124-mongo-listing-loop branch 2 times, most recently from 566e3fd to b306e20 Compare September 10, 2018 22:00
@jeremyds jeremyds added the wip label Sep 10, 2018
@jeremyds jeremyds force-pushed the bugfix/ZENKO-1124-mongo-listing-loop branch 2 times, most recently from 758bca6 to be6263d Compare September 10, 2018 23:10
@jeremyds jeremyds removed the wip label Sep 10, 2018
class Skip {
/**
* @param {Object} params - skip parameters
* @param {Object} extension - delimiter extension used (required)
Copy link

@bennettbuchanan bennettbuchanan Sep 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: This looks like additional arguments to the class. Can you update to make it apparent they are properties of params?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

// eslint-disable-next-line no-param-reassign
params.gt = undefined;
// eslint-disable-next-line no-param-reassign
params.gte = range;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: Could be a bit cleaner to create a new object and reassign these properties rather than disabling the linter.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I saw it so much in the code than I thought it is a good practice :S

@jeremyds jeremyds force-pushed the bugfix/ZENKO-1124-mongo-listing-loop branch from be6263d to 5687a48 Compare September 11, 2018 00:07
Copy link
Contributor

@jonathan-gramain jonathan-gramain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm good with the logic, I have minor comments on the style though, feel free to ignore if time is short.

const stream = new MongoReadStream(c, params,
params.mongifiedSearch);

skip.setListingEndCb(() => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: if those are mandatory callbacks, maybe they should be set directly in the constructor params instead of setters?

}

setListingEndCb(cb) {
this.listingEndCb = cb;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

listingEndCb and skipRangeCb contracts should be documented explicitly, since they are essential to the behavior of the listing extension.

@bert-e
Copy link
Contributor

bert-e commented Sep 11, 2018

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

  • ✔️ development/8.0

The following branches will NOT be impacted:

  • development/6.4
  • development/7.4

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of Release Engineering will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

  • Any commit you add on the source branch will trigger a new cycle after the
    current queue is merged.
  • Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of Release Engineering now.

@bert-e
Copy link
Contributor

bert-e commented Sep 11, 2018

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/8.0

The following branches have NOT changed:

  • development/6.4
  • development/7.4

Please check the status of the associated issue ZENKO-1124.

Goodbye jeremyds.

@bert-e bert-e merged commit 5687a48 into development/8.0 Sep 11, 2018
@rahulreddy rahulreddy deleted the bugfix/ZENKO-1124-mongo-listing-loop branch September 11, 2018 02:49
@ghost
Copy link

ghost commented Sep 11, 2018

Note that I can still identify one corner-case (I don't see a solution ATM, tbh):
Lots of deleted-master-versions means that we'll continue listing. If the number is high enough, we might be able to hit a listing timeout, Thus answering with a 500 instead of answering with a truncated listing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants