Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make retrieval even faster #7746

Merged
merged 7 commits into from
Dec 10, 2021
Merged

Make retrieval even faster #7746

merged 7 commits into from
Dec 10, 2021

Conversation

magik6k
Copy link
Contributor

@magik6k magik6k commented Dec 8, 2021

#7693 made retrieval quite fast, but we can do better

  • Move piecereader closer to storage code (making it easier to add lower-level optimizations)
  • Reuse one storage lock between piece seeks
  • Reduce allocation pressure
  • Don't do redundant sector finding / alloc checks
  • (separate PR) mmaped reader for local sector access
  • (separate PR) Cache reads in a temp file

@codecov
Copy link

codecov bot commented Dec 9, 2021

Codecov Report

Merging #7746 (e8ef39e) into master (49c619d) will decrease coverage by 0.05%.
The diff coverage is 74.19%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7746      +/-   ##
==========================================
- Coverage   39.57%   39.52%   -0.06%     
==========================================
  Files         640      640              
  Lines       68410    68465      +55     
==========================================
- Hits        27074    27059      -15     
- Misses      36698    36751      +53     
- Partials     4638     4655      +17     
Impacted Files Coverage Δ
extern/sector-storage/fr32/fr32.go 100.00% <ø> (+8.13%) ⬆️
extern/sector-storage/stores/http_handler.go 75.14% <ø> (-0.15%) ⬇️
extern/sector-storage/piece_reader.go 58.75% <50.00%> (ø)
extern/sector-storage/piece_provider.go 55.31% <66.66%> (-0.38%) ⬇️
markets/dagstore/miner_api.go 63.09% <70.00%> (ø)
extern/sector-storage/stores/remote.go 57.80% <72.22%> (-0.45%) ⬇️
markets/sectoraccessor/sectoraccessor.go 72.30% <75.00%> (ø)
extern/sector-storage/fr32/readers.go 81.25% <100.00%> (+1.25%) ⬆️
extern/sector-storage/mock/mock.go 66.27% <100.00%> (+0.90%) ⬆️
markets/dagstore/mocks/mock_lotus_accessor.go 80.48% <100.00%> (-0.47%) ⬇️
... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 49c619d...e8ef39e. Read the comment docs.

@magik6k magik6k marked this pull request as ready for review December 9, 2021 20:07
@magik6k magik6k requested a review from a team as a code owner December 9, 2021 20:07
@magik6k magik6k changed the title [WIP] Make retrieval even faster Make retrieval even faster Dec 9, 2021
@magik6k magik6k merged commit 36f1a8f into master Dec 10, 2021
@magik6k magik6k deleted the feat/faster-retrieval branch December 10, 2021 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants