Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

fix multiprocessing too many open files #8904

Merged
merged 2 commits into from
Dec 2, 2017
Merged

fix multiprocessing too many open files #8904

merged 2 commits into from
Dec 2, 2017

Conversation

piiswrong
Copy link
Contributor

Description

(Brief description on what this PR is about)

Checklist

Essentials

  • Passed code style checking (make lint)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage
  • For user-facing API changes, API doc string has been updated. For new C++ functions in header files, their functionalities and arguments are well-documented.
  • To my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

  • Feature1, tests, (and when applicable, API doc)
  • Feature2, tests, (and when applicable, API doc)

Comments

  • If this change is a backward incompatible change, why must this change be made.
  • Interesting edge cases to note here

@szha szha merged commit 1dc4aea into apache:master Dec 2, 2017
@bhavinthaker
Copy link
Contributor

I have documented the following in the MXNet 1.0 Release Notes here: https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.0+Release+Notes (see this Release notes for up-to-date information)

Problem: The data-loader has a file-descriptor leak when the number of files being opened exceeds the ulimit of the user session.
Workaround: Increase the ulimit of the user session or at the OS level using any of the following steps:
$ ulimit -n unlimited (to make the fd limit as unlimited), OR
$ ulimit -n 60000 (to make the fd limit as 60,000 files assuming you have around but less than 60,000 files to be opened) OR
$ echo 65535 > /proc/sys/fs/file-max (to increase the system-wide file limit) along with the ulimit command.
Further details: https://docs.oracle.com/cd/E19623-01/820-6168/file-descriptor-requirements.html

KellenSunderland pushed a commit to KellenSunderland/incubator-mxnet that referenced this pull request Dec 13, 2017
* fix

* Update cpu_shared_storage_manager.h
zhreshold pushed a commit to zhreshold/mxnet that referenced this pull request Dec 14, 2017
* fix

* Update cpu_shared_storage_manager.h
rahul003 pushed a commit to rahul003/mxnet that referenced this pull request Jun 4, 2018
* fix

* Update cpu_shared_storage_manager.h
zheng-da pushed a commit to zheng-da/incubator-mxnet that referenced this pull request Jun 28, 2018
* fix

* Update cpu_shared_storage_manager.h
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants