Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dbnode] Integrate arrow iterators into tile aggregation #2477

Conversation

linasm
Copy link
Collaborator

@linasm linasm commented Jul 22, 2020

What this PR does / why we need it:
Uses arrow iterators for large tile aggregation.

Special notes for your reviewer:
This is relative to a branch with #2466 and #2451 merged in.

Does this PR introduce a user-facing and/or backwards incompatible change?:


Does this PR require updating code package or user-facing documentation?:


@codecov
Copy link

codecov bot commented Jul 22, 2020

Codecov Report

Merging #2477 into linasm/aggregate-tiles-plus-arrow-iterators will decrease coverage by 7.8%.
The diff coverage is 71.0%.

Impacted file tree graph

@@                              Coverage Diff                               @@
##           linasm/aggregate-tiles-plus-arrow-iterators   #2477      +/-   ##
==============================================================================
- Coverage                                         71.5%   63.6%    -7.9%     
==============================================================================
  Files                                             1066     319     -747     
  Lines                                            94340   29920   -64420     
==============================================================================
- Hits                                             67512   19058   -48454     
+ Misses                                           22286    9360   -12926     
+ Partials                                          4542    1502    -3040     
Flag Coverage Δ
#aggregator 63.1% <54.3%> (-13.4%) ⬇️
#cluster 100.0% <ø> (+15.1%) ⬆️
#collector 41.8% <ø> (-41.1%) ⬇️
#dbnode 62.7% <74.4%> (-16.3%) ⬇️
#m3em 44.8% <ø> (-29.6%) ⬇️
#m3ninx ?
#m3nsch ?
#metrics ?
#msg 71.5% <71.6%> (-3.7%) ⬇️
#query 69.4% <62.5%> (+1.8%) ⬆️
#x ?

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f6a4d43...d587253. Read the comment docs.

arnikola and others added 21 commits July 22, 2020 08:57
…com:m3db/m3 into linasm/aggregate-tiles-plus-arrow-iterators
…e file on open (#2468)

* Add schema.VersionChecker

VersionChecker allows us to make decisions around how to proceed
depending on the Major and Minor Version in the index info file.

* Add DecoderStreamWithDigest

* Wire up index entry checksum validation on decode
linasm and others added 28 commits August 5, 2020 13:55
Also removed concurrency from tile iterators and cleaned up utility
@linasm
Copy link
Collaborator Author

linasm commented Aug 19, 2020

Superseded by #2543

@linasm linasm closed this Aug 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants