-
Notifications
You must be signed in to change notification settings - Fork 499
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proxy Cache Fill From Olympus #571
Comments
Ref #474 |
In case of storage miss currently the handlers fetch from the vcs synchronously and fill the storage Let's put down some possible approaches: We let the middleware check the storage and do the redirect in case of missIn case of hit we let the handlers serve the data from the storage. We let the handlers check the storage, redirect and trigger the asynchronous cache fillerThis has the advantage of having all the logic in the handlers and being more readable, it's probably faster than the worst case scenario of double miss (if the olympus storage is empty as well the whole operation would take 2x filling the storage and serving the data). We let the handlers check the storage, download & fill the cache from olympus synchronouslyAgain, all the code is in the handler. We'll have to have another download protocol (different from the one that serves from olympus) but probably the worst case scenario would be slower. We should take into account that filling the cache by the proxy generally would faster since the proxy is intended to work in proximity. Another advantage of this solution is that olympus will act as source of truth even for the first fetch. I'm torn between the last two, in any case the handlers will need to be specialized between the proxy and olympus and / or the public / private flag. Would love to hear your thoughts before I start messing up with the code. |
I'm torn between the same two. My first instinct is to go for the second option:
But I also really like the idea of Olympus being the source of truth from the first fetch. |
Haven't fully read the response yet, but my initial thought is this:
My current open PR (573) does a few changes in that we can pass a "Stasher" interface to the download Protocol and so it might make it even easier to have an Olympus Stasher (just one method implementation) instead of an Olympus download.Protocol (5 method implementation) Will read more about your suggestions and update on my thoughts when I get the time :) |
I'm on the edge if this is dup of #474 |
Ok then, I'll try to go for 3 trying to leverage @marwan-at-work PR for the stasher interface |
@fedepaol sorry for the late response. athens/pkg/download/goget/goget.go Line 28 in 7af04ff
We could use this: https://github.com/gomods/athens/blob/master/pkg/module/download.go Let me know what you think. |
Thanks @marpio for your opinion! I will try to collect all the pieces and start working. |
@marpio If I understood you correctly, I don't think we can swap the Olympus with the GoGetFetcher because how else is the download.Protocol gonna get private modules? See my comment for a suggestion where we can have the Stasher be able to stash from an upstream (olympus) or local (goget) |
@marwan-at-work my idea was to make it possible to provide a |
The download protocol would need to know if the module is public / private and have both the fetchers / stasher + fetcher . |
@fedepaol i don't intend to put both |
I would like to avoid using go to download bits from Olympus (in proxy). It is not an effective way to download modules. If we could use API of each VCS we would. We have both of them under control, Olympus provides a deterministic API, we can parallelize downloads of 3 pieces, without any additional go magic. Let's treat it as a network disk, which it is. |
I agree 💯 percent with you @michalpristas - we shouldn't use athens/pkg/download/goget/goget.go Lines 150 to 159 in e2cd124
It does it through the module.Fetcher .The implementation of the module.Fetcher used is here:https://github.com/gomods/athens/blob/master/pkg/module/go_get_fetcher.go This is the code which executes the go binary and provides module.Ref erence to the files on disk.What if we would provide another module.Fetcher , let say - pkg/module/olympus_fetcher.go which would return module.olympusRef instead of module.diskRef (https://github.com/gomods/athens/blob/master/pkg/module/disk_ref.go)?The implementation of the Read() function of the module.olympusRef would return module.Version using https://github.com/gomods/athens/blob/master/pkg/module/download.go
|
ok i get it now, Olympus implementation of Ref interface. |
The only thing that Lines 99 to 107 in e2cd124
|
Discussed on Slack. |
Discussed on slack, leaving this open in order to focus the work on the proxy. |
As of #772, we're not going to try and build a registry for the time being, so I'm closing this issue |
Issue #495 made it so that we can redirect to Olympus. We should also have the option to save an Olympus module into the proxy storage backend.
The text was updated successfully, but these errors were encountered: