Dotnet restore fails on OSX with too many open files error - need a generic solution for different projects

[livarcocc]

Just run build.sh on the Mac.

I got a bunch of these at first:

Warning: FindPackagesById: System.Reflection.Emit.Lightweight
Too many open files

And then it turned into an error.

[livarcocc]

@brthor Suggested a workaround, just run ulimit -n 512. We still need to prevent this from ever happening though.

[livarcocc]

Fixed with dotnet/cli#818

[vors]

Just hit it, thanks for the instruction to fix it 👍

[natemcmaster]

We still need to prevent this from ever happening though.

As a good sample case, try restore on EntityFramework. It always fails with this error.

[livarcocc]

I will re-open the issue. Seems like we need to find a generic solution to make dotnet restore as a command work, not only for the CLI restore step.

[moozzyk]

Related: NuGet/Home#2004

[natemcmaster]

FWIW ulimit -n 512 is not enough for big projects like EntityFramework. I have to do ulimit -n 2048 to avoid this error.

[wyoung]

Although this is Mac-specific insofar as OS X sets an unusually low limit on the number of open FDs, this is arguably a cross-platform design error in dotnet restore.

It seems that the problem is that it opens all of the NuGet HTTP connections needed to restore packages in parallel, blowing the default ulimit -n = 256 limit on OS X. But that just begs the question: why open that many parallel connections in the first place?

There are several good reasons to limit the number of simultaneous HTTP connections per server, quite aside from this client-side limit. (The C10K problem, TCP congestion issues, etc.) This is why all current web browsers limit themselves to at most 8 connections per server.

If dotnet restore behaved the same way, the Mac simultaneous open files limit would not be a problem.

The recommended workaround (ulimit -n 512) will only work as long as .NET remains the size it is now, and for relatively small projects. You can easily imagine this limit being blown again as .NET continues to grow, or on new development machines building preexisting large projects with many external dependencies.

There are several ways to fix this by making use of the rich parallel processing facilities in .NET. The recently-added Async stuff in C# provides one easy way, for example.

[agocke]

I just hit this again:

info :   NotFound https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-corefxtestdata/nuget/v3/flatcontainer/system.xml.xdocument/index.json 225ms
info :   GET https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-buildtools/nuget/v3/flatcontainer/microsoft.netcore.targets.dnxcore/index.json
log  : Installing System.Security.Cryptography.OpenSsl 4.0.0-beta-23504.
log  : Installing System.Security.Cryptography.Primitives 4.0.0-beta-23504.
log  : Installing System.IO.FileSystem.Watcher 4.0.0-beta-23504.
log  : Installing runtime.osx.10.10.System.Security.Cryptography.Algorithms 4.0.0-beta-23504.
log  : Installing runtime.unix.System.Security.Cryptography.Encoding 4.0.0-beta-23504.
log  : Installing runtime.osx.10.10.System.IO.FileSystem.Watcher 4.0.0-beta-23504.
info :   NotFound https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-coreclr/nuget/v3/flatcontainer/microsoft.netcore.platforms/index.json 224ms
info :   GET https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-coreclr/nuget/v3/flatcontainer/system.threading.overlapped/index.json
info :   NotFound https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-buildtools/nuget/v3/flatcontainer/system.xml.xdocument/index.json 235ms
info :   GET https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-corefxtestdata/nuget/v3/flatcontainer/system.threading.overlapped/index.json
info :   NotFound https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-coreclr/nuget/v3/flatcontainer/system.xml.readerwriter/index.json 309ms
info :   GET https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-buildtools/nuget/v3/flatcontainer/system.threading.overlapped/index.json
error: Too many open files

This is running with dotnet restore --disable-parallel.

Upping the ulimit is not an acceptable workaround -- we need to get this properly fixed ASAP.

[alexandru]

I just want to report that the error still happens on OS X, after a first dotnet restore after installing the .NET Core pkg:

$ dotnet --version
...
Product Information:
 Version:     1.0.0-beta-001598
 Commit Sha:  7582649f88

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.11
 OS Platform: Darwin
 Runtime Id:  osx.10.11-x64

[ffquintella]

Well maybe wrapping the command with the ulimit before it runs ?

Or just using this on your .bashrc
alias dotnet="ulimit -n 2048; dotnet"

[NickCraver]

Confirmed I'm hitting this with Dapper build scripts now. This should be reopened.

[davidfowl]

/cc @emgarten @yishaigalatzer

[yishaigalatzer]

We just checked in a file handle leak fix, it could be related. Nick - what build are you using?

[NickCraver]

@yishaigalatzer 1.0.0-rc2-002416

[TheRealPiotrP]

Since this is being tracked @nuget/home:2163, closing the CLI issue.

[NickCraver]

Link for others: NuGet/Home#2163

[yishaigalatzer]

Comments to others - At the moment this looks like a coreFX issue, we are discussing internally and will figure out if this will get fixed for RC2 and how.

[TyOverby]

Hit this today.