Do not immediately abort the extraction if a seed chunk is invalid #220
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Factor out writeChunk to a separate function
Do not immediately abort the extraction if a seed chunk is invalid
If the seed points to a RW location it may happen that some files change
while we are in the middle of an extraction.
When using the "InvalidSeedActionRegenerate" option, we can try harder
and attempt to take the invalid chunks from the self seed or the store.
If both of those fail too, then we abort the entire operation.
Limit the maximum number of chunks is a single seed sequence
If a seed is nearly completely equal to the output, we may end up with
just a few SeedSequencer that are very long and others that are just
one/a few chunks long.
Because each SeedSequencer is handled by a goroutine (from a pool), we
may reach a situation where the majority of the goroutines finish their
operations and are just waiting for the longer SeedSequencer jobs to
end.
By limiting the maximum amount of chunks in the SeedSequencer, we will
have jobs that are more balanced in term of amount of work.
When taking a chunk from the self seed, immediately return
If we were able to take a chunk from the self seed, there is no need to
continue looking into the existing file and the store.