dotnet restore --no-cache --force completes without error with no internet connection

[livarcocc]

From @NickCraver on July 19, 2017 21:6

This is with .NET Core 2.0 Preview 2 tooling. I'd expect --no-cache restores instructed to bypass cache to fail when no upstream sources are available...but that doesn't happen. Looking at procmon, it seems to be completely ignoring the --no-cache argument.

Here's what dotnet restore --no-cache --force registers during a run:

Steps to reproduce

1. Download the MiniProfiler/dotnet repo
2. Restore the solution

git clone [email protected]:MiniProfiler/dotnet.git
cd dotnet
dotnet restore

3. Disconnect your internet (unplug a cable, turn on WiFi, disable the adapter...whatever)
4. Try restoring the package again, disabling cache (--no-cache) and forced (--force)

dotnet restore --no-cache --force

Expected behavior

The restore would fail, unable to connect to any package sources and bypassing local cache.

Actual behavior

The restore succeeds without issue:

λ dotnet restore --no-cache --force
  Restoring packages for C:\Temp\dotnet\samples\Samples.AspNetCore\Samples.AspNetCore.csproj...
  Restoring packages for C:\Temp\dotnet\samples\Samples.Console\Samples.Console.csproj...
  Restoring packages for C:\Temp\dotnet\benchmarks\MiniProfiler.Benchmarks\MiniProfiler.Benchmarks.csproj...
  Restore completed in 156.69 ms for C:\Temp\dotnet\samples\Samples.Console\Samples.Console.csproj.
  Restoring packages for C:\Temp\dotnet\samples\Samples.Mvc5\Samples.Mvc5.csproj...
  Restore completed in 37.47 ms for C:\Temp\dotnet\samples\Samples.Mvc5\Samples.Mvc5.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler.AspNetCore.Mvc\MiniProfiler.AspNetCore.Mvc.csproj...
  Restore completed in 1.43 sec for C:\Temp\dotnet\benchmarks\MiniProfiler.Benchmarks\MiniProfiler.Benchmarks.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler.AspNetCore\MiniProfiler.AspNetCore.csproj...
  Restore completed in 1.27 sec for C:\Temp\dotnet\src\MiniProfiler.AspNetCore.Mvc\MiniProfiler.AspNetCore.Mvc.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler.EF6\MiniProfiler.EF6.csproj...
  Restore completed in 5.78 ms for C:\Temp\dotnet\src\MiniProfiler.EF6\MiniProfiler.EF6.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler.EntityFrameworkCore\MiniProfiler.EntityFrameworkCore.csproj...
  Restore completed in 402.71 ms for C:\Temp\dotnet\src\MiniProfiler.AspNetCore\MiniProfiler.AspNetCore.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler.Mvc5\MiniProfiler.Mvc5.csproj...
  Restore completed in 11.53 ms for C:\Temp\dotnet\src\MiniProfiler.Mvc5\MiniProfiler.Mvc5.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler.Providers.MySql\MiniProfiler.Providers.MySql.csproj...
  Restore completed in 420.85 ms for C:\Temp\dotnet\src\MiniProfiler.EntityFrameworkCore\MiniProfiler.EntityFrameworkCore.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler.Providers.RavenDB\MiniProfiler.Providers.RavenDB.csproj...
  Restore completed in 276.22 ms for C:\Temp\dotnet\src\MiniProfiler.Providers.MySql\MiniProfiler.Providers.MySql.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler.Providers.Redis\MiniProfiler.Providers.Redis.csproj...
  Restore completed in 2.23 sec for C:\Temp\dotnet\samples\Samples.AspNetCore\Samples.AspNetCore.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler.Providers.SqlServerCe\MiniProfiler.Providers.SqlServerCe.csproj...
  Restore completed in 9.2 ms for C:\Temp\dotnet\src\MiniProfiler.Providers.SqlServerCe\MiniProfiler.Providers.SqlServerCe.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler.Providers.SqlServer\MiniProfiler.Providers.SqlServer.csproj...
  Restore completed in 286.82 ms for C:\Temp\dotnet\src\MiniProfiler.Providers.RavenDB\MiniProfiler.Providers.RavenDB.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler.Shared\MiniProfiler.Shared.csproj...
  Restore completed in 196.94 ms for C:\Temp\dotnet\src\MiniProfiler.Providers.Redis\MiniProfiler.Providers.Redis.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler\MiniProfiler.csproj...
  Restore completed in 7.85 ms for C:\Temp\dotnet\src\MiniProfiler\MiniProfiler.csproj.
  Restoring packages for C:\Temp\dotnet\tests\MiniProfiler.Providers.RavenDB.Tests\MiniProfiler.Providers.RavenDB.Tests.csproj...
  Restore completed in 19.07 ms for C:\Temp\dotnet\tests\MiniProfiler.Providers.RavenDB.Tests\MiniProfiler.Providers.RavenDB.Tests.csproj.
  Restore completed in 223.15 ms for C:\Temp\dotnet\src\MiniProfiler.Providers.SqlServer\MiniProfiler.Providers.SqlServer.csproj.
  Restoring packages for C:\Temp\dotnet\tests\MiniProfiler.Tests\MiniProfiler.Tests.csproj...
  Restoring packages for C:\Temp\dotnet\tests\MiniProfiler.Providers.Redis.Tests\MiniProfiler.Profilers.Redis.Tests.csproj...
  Restore completed in 52.73 ms for C:\Temp\dotnet\tests\MiniProfiler.Tests\MiniProfiler.Tests.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler.Shared\MiniProfiler.Shared.csproj...
  Restore completed in 381.32 ms for C:\Temp\dotnet\src\MiniProfiler.Shared\MiniProfiler.Shared.csproj.
  Restoring packages for C:\Temp\dotnet\src\MiniProfiler.Shared\MiniProfiler.Shared.csproj...
  Restore completed in 416.19 ms for C:\Temp\dotnet\tests\MiniProfiler.Providers.Redis.Tests\MiniProfiler.Profilers.Redis.Tests.csproj.
  Restoring packages for C:\Temp\dotnet\tests\MiniProfiler.Tests\MiniProfiler.Tests.csproj...
  Restore completed in 695.44 ms for C:\Temp\dotnet\src\MiniProfiler.Shared\MiniProfiler.Shared.csproj.
  Restore completed in 606.9 ms for C:\Temp\dotnet\src\MiniProfiler.Shared\MiniProfiler.Shared.csproj.
  Restore completed in 590.48 ms for C:\Temp\dotnet\tests\MiniProfiler.Tests\MiniProfiler.Tests.csproj.

Environment data

dotnet --info output:

λ dotnet --info
.NET Command Line Tools (2.0.0-preview2-006497)

Product Information:
 Version:            2.0.0-preview2-006497
 Commit SHA-1 hash:  06a2093335

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.15063
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.0.0-preview2-006497\

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.0-preview2-25407-01
  Build    : 40c565230930ead58a50719c0ec799df77bddee9

/cc @rrelyea @emgarten

Copied from original issue: dotnet/cli#7195

[livarcocc]

From @emgarten on July 19, 2017 21:53

The global packages folder (.nuget\packages) is considered a source not a cache.

--no-cache avoids using the http cache, if however the packages are already in the global folder then restore will not go online in the first place.

--force will cause the project to restore even if it hasn't changed. This skips the noop feature that has been added in CLI 2.0.0 for restore.

These flags aren't the most obvious, we're currently looking at improving this for NuGet. To force a restore to level you might be expecting here you would need to delete all packages and caches first, you can do this with: dotnet nuget locals --clear all

[NickCraver]

FWIW chasing these issues as they're moved between repos for what appears to the user to be a dotnet CLI problem is absolutely infuriating. What I was typing on the original issue while it was being closed:

The global packages folder (.nuget\packages) is considered a source not a cache.

...this is pretty much the opposite of everything I've ever been told about .nuget. Many talks refer to it as the "local NuGet cache". Additionally, the current help says:

--no-cache                           Do not cache packages and http requests.

That, to me, means both of them, which is consistent with messaging elsewhere.

Why would I care about HTTP cache most of the time? The .nuget folder is what I want to bypass. Additionally, the workaround for doing a build and forcing a remote fetch should not be "hose every other build on the machine". Clearing the entire local cache is very much a nuclear option in terms of builds. The same would be true on a build server which often uses the same user.

I believe this behavior should change.

[emgarten]

The .nuget folder is what I want to bypass.

Would you provide some examples of when this is needed? Are you reusing the same package id/versions?

The best way to bypass .nuget is to set the packages folder path so that the project or solution restores to its own folder and avoids the user wide folder.
dotnet restore --packages <path to an empty folder>

Additionally, the workaround for doing a build and forcing a remote fetch should not be "hose every other build on the machine".

Can you explain what you expect restore to do with the global folder when --no-cache is set? Replacing a package in the global folder would also "hose every other build on the machine" since builds could be using dlls from the global folder.

[terrajobst]

I think the problem is:

• We're effectively using the .nuget folder as a cache
• However, it's implemented as a package source, which means our tooling thinks of them as inputs, rather than cache locations

[NickCraver]

Would you provide some examples of when this is needed? Are you reusing the same package id/versions?

Sure! Right now I'm testing restore times. If we get a bad cache somehow on a build server we want to be able to bypass it without nuking the cache. --packages <path> works, but is rather janky as a workaround. Think of instructions to a user: "create a random folder, point at it"

Can you explain what you expect restore to do with the global folder when --no-cache is set?

I'd expect it to do nothing with the global folder at all.

Replacing a package in the global folder would also "hose every other build on the machine" since builds could be using dlls from the global folder.

I was referring to the proposed workaround: dotnet nuget locals --clear all, not the build.

In general there's a large disconnect between team and public terminology here, evidenced in talks by Microsoft folks as well. To a programmer, this is a cache. It's quite literally caching content locally and not going to the source. If we explained exactly how it works and asked 100 programmers, I'd fully expect no less than 90 of them to think of this as a cache.

I get that naming is hard, but is this case it's especially problematic because it's used elsewhere in the same product and in commands that don't match expectations.

[emgarten]

I agree on better messaging. For this repo which covers the restore part of this I don't see any actionable work here. I'm going to close this issue unless there is further input on that.

[NickCraver]

@emgarten Then where should this live? It's been moved and closed once already. There is an issue, closing this doesn't make it the unintuitive behavior and improper terminology go away.

[emgarten]

@NickCraver which places contain improper terminology?

As for the --no-cache command line help, this is accurate, packages are cached in the http cache. If you delete the packages folder and run restore again those cached packages will be installed to the packages folder and they will not be downloaded again. If you use the no cache flag they will be downloaded again, so in my opinion this is an accurate description.

The issue here seems to be a general misunderstanding of the packages folder, however in NuGet itself we have been very carefully to always message this correctly. Issues need to be filed on the docs which are not messaging this correctly.

[NickCraver]

@emgarten I asked others as a sanity check: https://twitter.com/Nick_Craver/status/890160768007770113

As of right now:

This is a problem. It is, in every behavioral way, a cache. It seems that overwhelmingly people expect --no-cache to leave it alone, as I do. Where do you suggest this problem live?

[emgarten]

It seems that overwhelmingly people expect --no-cache to leave it alone, as I do.

I understand the confusion and appreciate the feedback. Here is the problem: Restore's purpose is to put packages on disk for build/publish/run to consume. If this doesn't occur then build will be missing inputs and everything will be broken. Having a flag that puts users in this broken state would be confusing.

The packages need to go somewhere, and they need to be persisted for future operations to use them. This is why setting --packages is in my opinion the best way to control this.

If for example --no-cache did avoid the global packages folder location it would need to create a new temp directory and copy all packages there. This folder would then have to be cleaned up manually by the user since only the user really knows when they are done building and using the restore output. This would quickly become a problem when packages were duplicated again and again to temp folders that couldn't be cleaned up by restore.

Where do you suggest this problem live?

Would you define where you think the problem is at this point? Your poll indicates that is with the outputs of restore.

[bording]

@emgarten If I specify --no-cache for a restore, but the package already exists in the global packages folder, will it use that copy or will it re-download the package from my list of package feeds?

I would expect it to ignore the copy already sitting in the global packages folder and re-download a new copy of it to the global packages folder, overwriting the existing folder if necessary.

If this isn't the current behavior, I suspect that will come as a surprise to just about everyone using --no-cache.

[NickCraver]

What @bording describes is exactly the behavior I'd expect, don't use any cached copies. I'm not proposing an elaborate new system of storage and resolution here, only that a re-download is forced. And I'd really expect --no-cache --force to do this, like the issue itself...but I don't think --force should be required to get this behavior.

I honestly have yet to meet anyone who cares about the HTTP cache in practice. In fact I googled it and just found this doc as the first result: https://docs.microsoft.com/en-us/nuget/consume-packages/managing-the-nuget-cache A screenshot from that doc:

It'd been called a cache for a long time, let's treat it as users expect.

[emgarten]

If I specify --no-cache for a restore, but the package already exists in the global packages folder, will it use that copy or will it re-download the package from my list of package feeds?

The global packages folder could be shared between different restores, builds, and executing the app. Removing files, even temporarily would cause problems. For this reason packages are never replaced or removed from the global folder. The exception to this is the dotnet nuget locals command which allows clearing the global folder manually when the user knows it is safe to do.

@bording do you have a scenario where a package needs to be replaced in the global folder?

I'd like to understand if extra validations/replacing on the global packages folder are needed, or if the issue is that users do not trust restore and the packages to be correct and feel that they need to give it a push, in which case I'd like to find a way to improve stability without requiring additional downloads.

[bording]

@emgarten I do have a scenario where this would be useful. I have a project that produces packages, and as part of testing, those packages are consumed from another project by having a local folder configured as the package feed. During development, I could end up with different builds of those packages, but they won't have a different version, so the restore for the consuming test project will pick up the old copy in the global folder instead of always looking at the local folder to get the newly built package.

Wiping out the entire global folder every time before doing the restore seems like overkill, but currently I have to manually delete the package from the global folder, which is a pain.

[NickCraver]

We're also testing restores from remote repos, firewall rules, VPN configs, custom NuGet servers, multiple builds of the same package (fixing NuGet/.NET tooling issues - we've filed many), and dealing with local source packages between builds for library buildings (again, the version doesn't increment every time).

For example, when a bad cache from a source happens, I need to be able to force a re-download from a good source after removing the bad one. This can also happen when people do a bad push from MyGet to NuGet and the version doesn't differ. The global cache has no concept of feeds, AFAIK. I might also be testing that pulling from one of our other data centers is working as expected.

Common usage is why I came across this in the first place: we often need a way to force a redownload of packages. There are plenty of scenarios where we want to force a re-download. --no-cache intuitively should do that.

The objection seems to be if we replace the file at any point in time that may cause issues. How is that any different from the manual deletion we're already doing today to work around this? Also, I agree that emptying the entire cache (on a build server, for example) is total overkill to refresh bad packages specific to a certain build.

[emgarten]

I have a project that produces packages, and as part of testing, those packages are consumed from another project by having a local folder configured as the package feed. During development, I could end up with different builds of those packages, but they won't have a different version, so the restore for the consuming test project will pick up the old copy in the global folder instead of always looking at the local folder to get the newly built package.

This is a known pain point when developing packages, I think something should be done here to improve this work flow. Replacing packages in the global packages folder would help solve this.

In general, re-using the same id/version of a package is not a supported scenario for NuGet. Packages are expected to be unique on id/version and the same across all sources and the global packages folder so that they can be used interchangeably. This is because nuspecs references other packages using just the id and version, not a hash, which gives no way to differentiate and select between different variations of a package with the same id/version.

The recommended way to deal with this is to increase the version number, this is often done with git versioning or by using a time based prerelease label, anything to make it unique and of a higher version.

In summary, NuGet should look into replacing packages in the global folder when no cache is set. The down side of this is:

• Conflicts may occur when trying to delete from the global packages folder if an app is currently executing
• Different solutions may continually stomp on each other's packages if they are using sources with different hashes for the same package id/version. (The global folder currently doesn't store packages under the hash).
• Decreased performance, restore cannot automatically use the id/version install to the global packages folder without making network calls.
• Breaking change to how the global package folder is used, since it will not be treated in part as a cache, and the --no-cache flag.

[bording]

The recommended way to deal with this is to increase the version number, this is often done with git versioning or by using a time based prerelease label, anything to make it unique and of a higher version.

@emgarten I am using git versioning, but that only helps me if I'm making commits between package builds, which is not always going to be true when I'm in the middle of development.

• Decreased performance, restore cannot automatically use the id/version install to the global packages folder without making network calls.
• Breaking change to how the global package folder is used, since it will not be treated in part as a cache, and the --no-cache flag.

Can you explain these two a bit more? I'm not sure I understand them. Are you saying performance would always be decreased, or just when the --no-cache flag is used?

[evil-shrike]

I have similar concerns as @bording and @NickCraver already clearly explained. But let me describe my use-case.

On my local machine I have two projects both are libraries with different release cycles (e.g. LibA and LibB). LibB depends on LibA and consumes nuget-packages from it. It depends on some version of LibA in csproj:

<PackageReference Include="LibA" Version="3.0.0" />

Both libs are being developed in parallel. So at a time we have some dev (unreleased) version of LibA which is being consumed in LibB (at a moment of LibB releasing that LibA version should become released). To propagate new changes of LibA to LibB I build LibA packages and do restore in LibB.

LibA packages are built into a folder which is used as package source for LibB restore.

Despite @emgarten said:

In general, re-using the same id/version of a package is not a supported scenario for NuGet.

it worked pretty well with "old" nuget for years.

nuget.exe restore LibB.sln -Source "%~dp0..\..\..\LibA\Release" -NoCache

In LibB I had .nuget\nuget.config with repositoryPath:

<config>
	<add key="repositoryPath" value="../../../packages" />
</config>

So all packages for LibB are put inside that folder. And it's being used during build both via msbuild/nuget and VS. If I needed to refresh a package I just went and delete it inside "packages" folder in LibB, then run restore.

Now with "new" nuget and dotnet cli I have headache. It's simply not working at all.

repositoryPath is ignored. All restored packages are put inside a new global "cache" (which is not a cache as it turned out) - I mean C:\Users\UserName.nuget\packages.

dotnet restore --no-cache -f is simply useless, it does something but not what is expected - getting a package from a source and use it for building.

Overriding a .nuget folder with --packages flags works in some way:

dotnet restore --no-cache -f --packages ../../../packages

All packages are put inside the specified folder. But VS won't see that folder and continue to use global .nuget. So it's also useless.

So currently I have to go and delete manually my packages in global .nuget folder. Needless to say how tedious it is.

Moreover I have no idea what to do in CI scenario (on build-server). It seems that the only workaround is clearing the global .nuget "cache" on every build :(

If supporting repositoryPath isn't possible (btw why?) then please at least implement meaningful behavior for --no-cache option (override packages in .nuget "cache" with downloaded ones).

[emgarten]

@evil-shrike use globalPackagesFolder see: https://docs.microsoft.com/en-us/nuget/schema/nuget-config-file

Packages.config uses a different folder format than PackageReference which is why the same property cannot be applied for both project types. The new equivalent is globalPackagesFolder

[evil-shrike]

@emgarten Thank you, using globalPackagesFolder helps a lot. But I have to remove packages in that folder to get fresh versions. It'd nice to avoid it - if I tell "no cache please" and "force" (--no-cache -f) then just override packages from a source into globalPackagesFolder.

[Mpdreamz]

Even a dotnet nuget locals -c <PACKAGEID> <VERSION> would be somewhat of a step in allowing local packages to be used in tests without worrying whether nuget is taking the version that i point it to or one from the global-packages cache

[StefanBertels]

I just stumbled upon weird default restore behaviour:

1. I pushed a new version of package A to our own NuGet server (klondike), incrementing version number.
2. I manually entered the new version number in consuming project B (csproj, new SDK format)
3. Neither Visual Studio nor "dotnet restore" could find the new package and raised an error. My build script (calling msbuild with restore target) also failed.

Checking the NuGet server (with my browser) did show the new package immediately. I solved this by adding "-NoCache" to my build script after reading about dotnet/nuget restore caching http results.

Caching packages for some time might be ok dependent on use case, but caching http results when this will lead to a restore error isn't useful.

BTW: I tried to updating the package with Visual Studio's built-in NuGet dialog and VS couldn't find the new package, either. One can argue whether caching might be ok here -- but I would expect to use "-NoCache" here when explicitely hitting Reload button. (VS 15.6.0, NuGet 4.1.0.2450, dotnet 2.1.100)

[Svish]

How I ran into this annoyance:

We have switched package source from MyGet to Azure Artifacts. Wanted to test that nuget was able to still actually find (correct URL, authentication working, etc.) all the packages after the move.

Cleaning out the whole local cache of packages in this case is quite unwanted in case a package actually haven't been transferred correctly. So tried to run with no-cache and force, but yeah, as for others here, but yeah, nuget kept using the local "cache" still. I could tell, because nuget never got to a point where it asked me to authenticate with Azure. Not even after deleting the projects packages folder.

How --no-cache isn't no cache is very confusing to me... Had to do the "delete all cache" thing to actually get this to work, but that shouldn't have been necessary. --no-cache should've been enough for me to get nuget to try authenticate and download packages from Azure.

[veleek]

Similar to @Svish, I'm switching to a single NuGet feed, which has a bunch of additional feeds configured as upstream sources. I'd like to be able to validate that the feed is configured correctly but the global package cache gets in the way.

Honestly changing it from --no-cache to --do-not-cache would probably be enough to make it clear what the flag is intended to do. However, I would still expect --no-cache (when I'm using a nuget.config file which specifies a specific single package source) to only go to the package source specified and not anywhere else.

[alastairs]

I just hit this with .NET SDK 7.0.100 Preview 6 and @bording's scenario. --force --no-cache should ignore all caches and refetch the package(s) from source.

Deleting ~/.nuget/packages is a really painful workaround.

[jeffkl]

Team Triage: Unfortunately the --no-cache command line argument only applies to the HTTP cache. This can be useful if a package was pushed in the last 30 minutes when the HTTP cache would not be expired.

The easiest way to "disable" the global package folder is to set the environment variable NUGET_PACKAGES to an empty folder and then run restore. This will cause NuGet to pull down all of the packages. You can then delete that folder at the end.

For example, from the root directory of a repository:

SET NUGET_PACKAGES=%CD%\.nuget\packages
dotnet restore --no-cache --force

[dluc]

I'm developing a set of libraries and testing the resulting nugets locally before the release process. As you might guess, I'm building several times the same version number, and whenever one nupkg build is stored under ~/.nuget/packages/, that quickly becomes stale, and I have to delete it manually.

On projects consuming my local build, it would be useful having a csproj option, telling dotnet (core) not to restore packages from ~/.nuget/packages/. I've seen suggestions about command line approaches, but I'm looking for a no-scripting way.

<RestoreNoCache>true</RestoreNoCache> doesn't work because it was decided that global-packages is not a cache.... Any other suggestion? I'd prefer not having to deal with command line ways to build and launch, ie not having to change my (or anyone's) workflow, e.g. if one builds and runs their code pressing Ctrl+F5, let them keep doing that. Thanks

[dystopiandev]

I went through the pain of setting up a local nuget server, only to find there's no way to bypass global package cache, so back to powershell/bash recursive delete scripts. It's a wild decision to name a flag "no cache" but it precisely uses the one cache you would expect it to bypass.