Less obnoxious OCI pulling on sync #6804
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Michael Nelson <[email protected]>
…ssarily. Signed-off-by: Michael Nelson <[email protected]>
Signed-off-by: Michael Nelson <[email protected]>
✅ Deploy Preview for kubeapps-dev canceled.
|
Signed-off-by: Michael Nelson <[email protected]>
Thanks for the review. Just note, I'm holding of landing this until I have the follow-up PRs also done and can check it works as expected locally. First one is at #6825 but there will be one more following that also. I'll land them once tested locally. |
absoludity
added a commit
that referenced
this pull request
Sep 19, 2023
### Description of the change Follows on from #6804, this PR updates so that the actual Syncing happens per chart+versions, rather than at the end of the process. Much of the change is refactoring functions that worked with slices of charts to work with a chart instead. The complexity is the actual refactoring of the sync, requiring channels to iterate on the results etc. ### Benefits We'll be able no have results in the UX much more quickly (important with OCI imports, which take longer without a big index.yaml to simply parse). ### Possible drawbacks Currently this will not work as expected, as two more changes are required: 1. Need to *update* the existing chart versions for a chart, rather than re-writing them (with only the chart versions that were synced, as it will do now) 2. Need to collect the charts to be deleted and pass those explicitly, rather than deleting all charts that are not in the current set being synced (which it currently does - as this worked when syncing everything). ### Applicable issues - ref #6706 ### Additional information Last PR to follow. --------- Signed-off-by: Michael Nelson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of the change
This PR update the way we pull OCI repos, so that we:
Benefits
We can pull large OCI catalogs from public repos (in particular, can pull the Bitnami catalog from Dockerhub) without hitting public pull limits.
Possible drawbacks
The initial time until the catalog can be used is slightly longer. But I have two things I want to improve in a subsequent PR:
This combination of changes will also remove another issue we have where a subsequent sync, running within 10m of the original (by default) will do the same work currently, rather than just doing the work that hasn't been done yet.
Applicable issues
Additional information