Filter BlobInfoCache candidates before prioritization, not in transports #2346
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mtrmac
force-pushed
the
bic-filter-first
branch
2 times, most recently
from
3d75b49 to
b940f89
Compare
Should not change behavior. Signed-off-by: Miloslav Trmač <[email protected]>
Make it more similar to testGenericCandidateLocations, and include both locations for the same digest on the same line, to make the relationships between the digests's entries a bit clearer. Should not change behavior. Signed-off-by: Miloslav Trmač <[email protected]>
We will add a long/unwieldy CandidateLocations2Options parameter, and adding that inline would make readability even more challenging. So, split that out; the new parameter will be added shortly. Should not change (test) behavior. Signed-off-by: Miloslav Trmač <[email protected]>
We will soon be looking up the actual compression alghorithms in the BIC code, so use real algorithm names in the tests. Should not change (test) behavior. Signed-off-by: Miloslav Trmač <[email protected]>
…internal/manifest We will want to call this from the pkg/blobinfocache code, and for that we need to make it dependent on neither internal/private nor internal/blobinfocache (otherwise the interface definitions would create a dependency loop). This only moves ~unchanged code, should not change behavior. Eventually CandidateMatchesTryReusingBlobOptions will be removed entirely. Signed-off-by: Miloslav Trmač <[email protected]>
... and move the canSubstitute parameter to it. We will add more options, and we well want to carry them around as a unit. Should not change behavior. Signed-off-by: Miloslav Trmač <[email protected]>
…ions ... into candidateLocations and appendReplacementCandidates. We will add more conditions there soon. Should not change behavior. Signed-off-by: Miloslav Trmač <[email protected]>
This is - more similar to the other BIC implementations - probably faster: we hold a transaction open anyway, this is all local, so making fewer queries is unlikely to be notably less costly, and we only read the value once instead of getting a column all containing the same value - necessary for future changes, were we need to do local checks on compressorName before consuming the other data, and those checks are in Go and can't be reasonably done by the SQLite query Should not change behavior. Signed-off-by: Miloslav Trmač <[email protected]>
… ones Append to candidates directly, avoid a separate allocation of a temporary array. Should not change behavior. Signed-off-by: Miloslav Trmač <[email protected]>
The primary benefit is that we now filter the algorithms _before_ collecting and trimming the list of candidates, i.e. if the most recent candidates are not matching the requirements, CandidateLocations2 will return older candidates instead of returning just some unwanted candidates which will be rejected. This requires us to look up the actual compression in the BIC code, so this also changes pkg/blobinfocache/internal/test to use real algorithm names. Signed-off-by: Miloslav Trmač <[email protected]>
Inline it into the only remaining caller. Should not change behavior. Signed-off-by: Miloslav Trmač <[email protected]>
mtrmac
force-pushed
the
bic-filter-first
branch
from
b940f89 to
75a01e9
Compare
…tions The implementations (sharing the code in pkg/blobinfocache/internal/prioritize) already need to look up the algorithm, so just return the value as a part of a BICReplacementCandidate2, instead of looking it up again in the transport. Should not change behavior. Signed-off-by: Miloslav Trmač <[email protected]>
... dropping various unreachable/redundant code paths, now that the semantics is clear and tested. Removes the unfortunate dependency of c/image/docker on c/image/pkg/compression, making it cheaper to use c/image/docker as a simple registry client. Should not change behavior. Signed-off-by: Miloslav Trmač <[email protected]>
…e field Should not change behavior. Signed-off-by: Miloslav Trmač <[email protected]>
mtrmac
force-pushed
the
bic-filter-first
branch
from
75a01e9 to
7482d25
Compare
|
LGTM |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This:
gzip/zstd, or when the user specifies av2s2manifest format and Zstd is not acceptable). Before, we would prioritize and return up to 5 candidates regardless of algorithm, and then the transport would filter unwanted algorithms — in the worst case, getting not a single acceptable algorithm. Now, we first filter out unacceptable candidates, and only afterwards prioritize and return up to 5 candidates to the transport. That should improve the chances of reusing a known blob.BlobInfoCache2implementation. This will, again, allow usingc/image/dockeras a ~light-weight registry client without dragging inc/storage/pkg/chunked.See individual commit messages for details.
Cc: @giuseppe