Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DSN optimizations (part 4) #1101

Merged
merged 8 commits into from
Jan 23, 2023
Merged

DSN optimizations (part 4) #1101

merged 8 commits into from
Jan 23, 2023

Conversation

nazar-pc
Copy link
Member

@nazar-pc nazar-pc commented Jan 22, 2023

This is where refactoring was going up until this point.

During plotting:

  • farmer will only announce pieces that were not plotted (and thus announced before)
  • pieces that satisfy caching policy will be cached and announced right away
  • cached pieces are returned from cache without reaching to the network at all

During announcements:

  • pieces that were previously announced are ignored
  • announced pieces are retrieved from announcer only (instead of making full network retrieval with lookup)

There is still one more optimization I came up with, but left it for later in #1100

Fixes #1076

Code contributor checklist:

@nazar-pc
Copy link
Member Author

@i1i1 please look carefully at the description of this PR, it describes some key changes that were partially true on the SDK side, but not entirely (with node and farmer you'll have two caches: recently archived segments on node and regular piece cache on the farmer for instance). It'll need to be corrected and I'll do further refactoring to expose more of the primitives that you can assemble the way you need it in SDK (right now there will be a bit more copy-paste required than I'd like to see).

Copy link
Contributor

@shamil-gadelshin shamil-gadelshin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Simplify cache size handling and make it explicit
Base automatically changed from dsn-optimizations-part-3 to main January 23, 2023 09:40
# Conflicts:
#	crates/subspace-farmer/src/bin/subspace-farmer/commands/farm/dsn.rs
@nazar-pc nazar-pc merged commit 0c4ba93 into main Jan 23, 2023
@nazar-pc nazar-pc deleted the dsn-optimizations-part-4 branch January 23, 2023 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants