files not closed when installing dependencies causes Error: Too many open files for formulas with lots of dependencies

[timotheecour]

originally reported in https://github.com/Homebrew/linuxbrew-core/issues/21139 but reporting here as suggested here https://github.com/Homebrew/linuxbrew-core/issues/21139#issuecomment-726296817

Bug report

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

• ran brew update and can still reproduce the problem?
• ran brew doctor, fixed all issues and can still reproduce the problem?
• ran brew config and brew doctor and included their output with your issue?

What you were trying to do (and why)

brew install opencv

What happened (include command output)

Command output

...
==> Pouring libass-0.14.0_1.x86_64_linux.bottle.1.tar.gz                                                                   ==> Installing opencv dependency: adoptopenjdk                                                                             Error: Too many open files                                                                                                 
  

What you expected to happen

works

Step-by-step reproduction instructions (by running brew commands)

from a fresh linuxbrew install (under /home/linuxbrew/.linuxbrew/), run brew install opencv

it has many dependencies and eventually fails with Error: Too many open files

it looks like a regression.
maybe some file descriptors are not being closed?

this likely could happen with other formulas, shouldn't be specific to opencv.

I reproduced this on several systems (including fresh VM's in EC2 with 8CPU/32GB RAM)

Output of brew config and brew doctor commands

this is from a fresh VM and fresh brew install

HOMEBREW_VERSION: 2.5.10
ORIGIN: https://github.com/Homebrew/brew
HEAD: b43c0fed789b4cae33cd200284c44a095db57c3c
Last commit: 65 minutes ago
Core tap ORIGIN: https://github.com/Homebrew/linuxbrew-core
Core tap HEAD: ce692cf7ed773e724a5114087514eaf6fc4400aa
Core tap last commit: 5 hours ago
Core tap branch: master
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 8
Homebrew Ruby: 2.6.3 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.3_2/bin/ruby
CPU: octa-core 64-bit unknown_0x17_0x1
Clang: N/A
Git: 2.25.1 => /bin/git
Curl: 7.68.0 => /usr/bin/curl
Kernel: Linux 5.4.0-1029-aws x86_64 GNU/Linux
OS: Ubuntu 20.04.1 LTS (focal)
Host glibc: 2.31
/usr/bin/gcc: 9.3.0
/usr/bin/ruby: N/A
glibc: N/A
gcc: N/A
xorg: N/A

brew doctor
Your system is ready to brew.

workaround

keep running brew install opencv until it works as it makes progress each time by installing dependecies (but it adds a lot of overhead)

(note that it eventually fails for unrelated reasons, with qt: https://github.com/Homebrew/linuxbrew-core/issues/21130; EDIT: that one just got fixed)

EDIT

retried again on a fresh t3a.2xlarge AWS EC2 instance (8vCPU, 32GB memory):

==> Installing opencv dependency: open-mpi
==> Pouring open-mpi-4.0.5.x86_64_linux.bottle.tar.gz
Error: Too many open files @ rb_sysopen - /home/linuxbrew/.linuxbrew/Cellar/open-mpi/4.0.5/lib/libmpi_mpifh.so.40.20.2
Warning: Bottle installation failed: building from source.
==> Downloading https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.5.tar.bz2
######################################################################## 100.0%
==> ./configure --prefix=/home/linuxbrew/.linuxbrew/Cellar/open-mpi/4.0.5 --enable-ipv6 --with-libevent=/home/linuxbrew/.linuxbrew/opt/libevent --with-sge
==> make all
....
Error: Too many open files

note: it also affects semantics

here are logs from brew install opencv:

==> Installing opencv dependency: vtk
==> Pouring vtk-9.0.1.x86_64_linux.bottle.tar.gz
Error: Too many open files @ rb_sysopen - /home/linuxbrew/.linuxbrew/Cellar/vtk/9.0.1/lib/libvtkInfovisBoostGraphAlgorithms-9.0.so.9.0.1
Warning: Bottle installation failed: building from source.
==> Downloading https://www.vtk.org/files/release/9.0/VTK-9.0.1.tar.gz

=> it fails for unrelated reason (Too many open files) and then builds from source (which can be much slower) instead of building from bottle

This just seems like a homebrew/linuxbrew bug: files for a formula be closed once the corresponding formula was built.

[MikeMcQuaid]

[ ] ran brew config and brew doctor and included their output with your issue?

Closing until this is provided.

[timotheecour]

@MikeMcQuaid can you please re-open? I've edited OP and added brew config and brew doctor outputs

[iMichka]

Can you also provide:

cat /proc/sys/fs/file-max
ulimit -Hn
ulimit -Sn
cat /proc/1234/limits

Just to have a broad idea.

[timotheecour]

@iMichka
cat /proc/sys/fs/file-max
9223372036854775807

ulimit -Hn
1048576

ulimit -Sn
1024

cat /proc/1234/limits
not sure what you wanted here:
cat: /proc/1234/limits: No such file or directory

[gromgit]

Perhaps @iMichka wanted the output of cat /proc/$$/limits instead.

[timotheecour]

makes sense

cat /proc/$$/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             127364               127364               processes
Max open files            1024                 1048576              files
Max locked memory         67108864             67108864             bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       127364               127364               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

[iMichka]

I have the same 1048576 for the hard limit. But also 1048576 as the soft limit. And have no issue.

You have a soft limit set at 1024, which is really low. This means that only 1024 files can be opened at once.

Of course we could have a look in homebrew if we can close more files, I do not know our code enough to tell what is going on. But I would also expect your system to be able to keep more than 1024 files open.

[timotheecour]

You have a soft limit set at 1024, which is really low.

1024 might be low but note, this is default for any VM in AWS (and other VM's IIRC), so is likely to impact quite a few people (plus, it's always nice to lower resource usage either way).

when brew install foo must install pkg1, pkg2, ... dependencies, each dependency is installed, opening many bottle tar.gz files the process, but these are not released; instead those files should be closed right after each dependency is built.

[MikeMcQuaid]

1024 might be low but note, this is default for any VM in AWS (and other VM's IIRC), so is likely to impact quite a few people (plus, it's always nice to lower resource usage either way).

Will reopen this if we see it affect other people.

when brew install foo must install pkg1, pkg2, ... dependencies, each dependency is installed, opening many bottle tar.gz files the process, but these are not released; instead those files should be closed right after each dependency is built.

Will review a PR to address this. There's a trivial workaround for your current problem.

[iMichka]

Yes, same thoughts here: I think in 3-4 years there was only 1 or 2 users complaining about this. And the workaround is to increase the soft limit, which should be easy.

No maintainer will probably work on this, but we would review a PR, as Mike said.