Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not immediately abort the extraction if a seed chunk is invalid #220

Merged
merged 4 commits into from
Apr 18, 2022

Conversation

RyuzakiKK
Copy link
Contributor

  • Factor out writeChunk to a separate function

  • Do not immediately abort the extraction if a seed chunk is invalid

    If the seed points to a RW location it may happen that some files change
    while we are in the middle of an extraction.

    When using the "InvalidSeedActionRegenerate" option, we can try harder
    and attempt to take the invalid chunks from the self seed or the store.
    If both of those fail too, then we abort the entire operation.

  • Limit the maximum number of chunks is a single seed sequence

    If a seed is nearly completely equal to the output, we may end up with
    just a few SeedSequencer that are very long and others that are just
    one/a few chunks long.

    Because each SeedSequencer is handled by a goroutine (from a pool), we
    may reach a situation where the majority of the goroutines finish their
    operations and are just waiting for the longer SeedSequencer jobs to
    end.

    By limiting the maximum amount of chunks in the SeedSequencer, we will
    have jobs that are more balanced in term of amount of work.

  • When taking a chunk from the self seed, immediately return

    If we were able to take a chunk from the self seed, there is no need to
    continue looking into the existing file and the store.

@RyuzakiKK
Copy link
Contributor Author

I manually tested the situation where a seed changes while desync extract is in progress.

However I'd like to add some automated tests too. I'll probably look into it in the days.

assemble.go Show resolved Hide resolved
assemble.go Show resolved Hide resolved
sequencer.go Outdated Show resolved Hide resolved
If the seed points to a RW location it may happen that some files change
while we are in the middle of an extraction.

When using the "InvalidSeedActionRegenerate" option, we can try harder
and attempt to take the invalid chunks from the self seed or the store.
If both of those fail too, then we abort the entire operation.

Signed-off-by: Ludovico de Nittis <[email protected]>
If a seed is nearly completely equal to the output, we may end up with
just a few SeedSequencer that are very long and others that are just
one/a few chunks long.

Because each SeedSequencer is handled by a goroutine (from a pool), we
may reach a situation where the majority of the goroutines finish their
operations and are just waiting for the longer SeedSequencer jobs to
end.

By limiting the maximum amount of chunks in the SeedSequencer, we will
have jobs that are more balanced in term of amount of work.

Signed-off-by: Ludovico de Nittis <[email protected]>
If we were able to take a chunk from the self seed, there is no need to
continue looking into the existing file and the store.

Signed-off-by: Ludovico de Nittis <[email protected]>
@folbricht folbricht merged commit b545768 into folbricht:master Apr 18, 2022
@RyuzakiKK RyuzakiKK deleted the continue_with_invalid_seeds branch April 18, 2022 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants