Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

preserve maxPid in journal aggregation #1824

Conversation

alstanchev
Copy link
Contributor

@alstanchev alstanchev commented Nov 27, 2023

in the rare case where all or most of the elements in an aggregation are pruned for being deleted it is not possible to preserve maxPid which in turn makes it impossible to iterate the whole collection.

the aggregation pipeline:

[{$match: {
 pid: {
  $gt: ''
 }
}}, {$sort: {
 pid: 1,
 sn: -1
}}, {$limit: 200}, {$group: {
 _id: '$pid',
 sn: {
  $first: '$sn'
 },
 __lifecycle: {
  $first: '$s2.__lifecycle'
 },
 _revision: {
  $first: '$s2._revision'
 },
 policyId: {
  $first: '$s2.policyId'
 },
 _modified: {
  $first: '$s2._modified'
 }
}}, {$sort: {
 _id: 1
}}, {$group: {
 _id: null,
 m: {
  $max: '$_id'
 },
 i: {
  $push: '$$ROOT'
 }
}}, {$redact: {
 $cond: {
  'if': {
   $ne: [
    '$__lifecycle',
    'DELETED'
   ]
  },
  then: '$$DESCEND',
  'else': '$$PRUNE'
 }
}}]

Fixes: #1825

in the rare case where all or most of the elements in an aggregation are pruned for being deleted it is not possible to preserve maxPid which in turn maces it impossible to iterate the whole collection.

Signed-off-by: Aleksandar Stanchev <[email protected]>
@alstanchev alstanchev changed the title preserve maxPid in listNewestActiveSnapshotsByBatch aggregation preserve maxPid in journal aggregation Nov 27, 2023
@alstanchev
Copy link
Contributor Author

@thjaeckle would appreciate a review here.
Still waiting on the system test but locally looks good.
On an imported db only one device was added to the search index after the fix all (2k+) were added.

Copy link
Member

@thjaeckle thjaeckle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took a brief look.
Did not deeply look into the how and why this works and the former didn't, but I trust in you and the system tests

@thjaeckle thjaeckle added the bug label Nov 28, 2023
@thjaeckle thjaeckle added this to the 3.4.2 milestone Nov 28, 2023
@alstanchev
Copy link
Contributor Author

Tests are green

@alstanchev alstanchev merged commit b14d757 into eclipse-ditto:master Nov 28, 2023
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Background Sync not syncing entities
2 participants