Nuget.Server: Add read-through cache, fetching from other feeds (first-time) and caching locally

[maartenba]

From @NickCraver on April 24, 2016 12:43

Nuget.Server does most of what we need to build Stack Overflow locally for our packages (mostly produced by TeamCity today). However, it could be much more useful with read-through caching. Here's the flow:

• Client asks for a specific package version
  • Nuget.Server has it locally
    • Package returned
  • Nuget.Server doesn't have it
    • If no cache-miss feeds configured, 404 (same as today)
    • If cache-miss feeds configured (new)
      • Nuget.Server fetches (in order, same as nuget.exe) from a configured feed list. (new)
        • If found, Nuget.Server caches it in the local folder and returns it to the client.
        • If not found, 404 (same as today).

It's really far simpler to show this in code with a prototype. Here's a fully functional example I made using nuget.org as the cache-miss source: NickCraver/NuGet.Server@452af26

I'd love to upstream a fleshed out version of this feature this because I think it'd be useful to many others: it'd save the world a little bit of bandwidth, dependency depth, and some build failures when the internet goes boom.

A few questions to start with:

• Is there a fundamental problem with the approach?
  • Our uses are simple so maybe we're blind here.
• What's the simplest config mechanism we could use for multiple feed sources?
  • appSettings is a bit limited here, but we could make it work.
• Will this be confusing, by nature only working on authenticated feeds (as a cache-miss source)?
  • IMO, auth delegation is confusing, complex, and not worth it here.

Copied from original issue: NuGet/Home#2650

[NickCraver]

@maartenba Thanks for moving this, I was unsure on placement when filing it.

An update to Nuget.Server's README.md would be much appreciated. It's not at all clear where the issues are supposed to live. Current phrasing conveys only website issues should go in the Gallery and everything else goes in Home. I think "Issues for (current package name) live here: [link]" would be a welcome addition to every Nuget repo's README.md.

cc @yishaigalatzer

[yishaigalatzer]

Hey Nick, this is on our roadmap for later but heck we would love a PR to make Christmas come early :)

There is another feature here which is to define proxying rules. For example only allowing or rejecting certain globbed packages to come from a defined feed.

We did not flesh out a spec for proxy rules, but the current general thought is that this all lives in nuget.config along with the feed declarations. That way we can apply it directly to a client as well.

Last there are a couple of big missing items in the "spec"

1. Pass through search
2. V3 server upstream support

[maartenba]

Updated the README, good catch.

The change looks good, but will mostly accomodate package restores and not search nor older NuGet clients. Ideally the FindpackagesById() and Search() methods need proxying too, as wel as the Packages collection (the latter being a tricky one)

[eatdrinksleepcode]

What's the next step for this? Just had a need to stand up a new short-term internal mirror, and Nuget.Server would have been the simplest way to do it had this feature been available.

[yishaigalatzer]

You can use nexus, or klondike to achieve this today. We are not planning to work on this in the immediate future

[NickCraver]

Sorry, I haven't had time to get back to this - a lot is happening on many fronts. I think what the NuGet team wants is way more full featured than what we're after at Stack. We just want a simple caching store. We don't care about forwarded search at all...we simply want to cache packages. I suppose we can:

1. Consider this to be 2 separate features, and work on the caching store now.
2. Wait for someone to pick up work that's after all of the same functionality discussed above.
3. Maintain a fork of the code (which we have ready and working already for our purposes...we don't even need a config).

I'd love to be able to go with option 1 above, but that's not my call - and I'm not sure what the config would look like overall. If search forwarding is wanted, it makes sense they'd share a single config value. I certainly see the value of search forwarding, it's just not a value add for our use case. It's also somewhat cancelled out by this feed being the first in any NuGet.config, there the client takes the load here.

Personally, I'd rather have this in a client config, rather than an arbitrary server searching all feeds. If I want to pull pre-release packages for one project and not the other, then I can be share that pre-release feed config on the server for search forwards, that'll have very undesirable effects for things like -* package configs. Granted they can't be used on the server to cache for those cases, but that's the safe approach. It'd be nice to find an alternative that supports both use cases well, but I can't think of a solution there - thoughts?

[yishaigalatzer]

@NickCraver here is a strawman

What if you put your changes in another package that you can install on top of nuget.server? We can make extensiblity points in nuget.server. This will allow users to pick up immediately your (cool) addition, but not force the design in nuget.server.

I would also suggest that forwarding comes from using the nuget3 protocol instead of nuget.core and the V2 protocol for scalability purposes.

[NickCraver]

@yishaigalatzer Absolutely - both of those sound like solid approaches to me. Nuget.Core was me looking for just a few minutes to implement, I'll take a glance at v3 after kids are in bed - have any examples to point me at? It'd be much appreciated.

[yishaigalatzer]

Very embarrassed to say we don't have anything yet (other than nuget.exe and our client code base). But Dave Glick wrote a cool series of blogs - http://daveaglick.com/posts/exploring-the-nuget-v3-libraries-part-1 http://daveaglick.com/posts/exploring-the-nuget-v3-libraries-part-2

[ghuntley]

@NickCraver http://stackoverflow.com/a/34959690

[FlorianRappl]

I still would love to see this implemented. However, as the conversation stopped at the 2nd of June I believe not much is going on here.

[maartenba]

Recommending BaGet for this scenario.

[P0oOOOo0YA]

Hope this feature gets implemented in a near future. Not every body sitting behind a giga speed internet connection and having a caching package manager streamlines developing experience to a high degree. Something which is done on Python and Node environments using Devpi private pypi server and Verdaccio private npm server a long time ago. @maartenba thanks for mentioning BaGet seems like this project is the right private package manager for .Net