Error: EMFILE errno:20 in fs.readFile

[Somebi]

Sometimes i'm getting tons of EMFILE errors. This is the code from my sessions module:

for(var i in files) {
    var id=files[i];

    (function(id){

        var fullpath=path.join(self.path,id);

        self.log('Reading session file:',fullpath);

        Chain.busy();

        fs.readFile(fullpath,function(e,buffer){
            if(e){
                self.emit('error','Failed to read file: '+fullpath,e);
                Chain.ready();
                return;
            }

            // ... other stuff
        });
    })(id);
}

I have read, that this error is related to file descriptor. Doesn't fs.readFile close the descriptor?

Or i'm breaking the limit of allowed opened descriptors?

Not sure if have tons of sessions. But will check that the next time these errors appear.

Default limit is 1024 opened descriptors?

Thanks.

[Somebi]

Pardon me. I'm using node v0.11.0 on Ubuntu 12.10

[indutny]

Basically, you're running asynchronous action and before it'll end - it'll use one file descriptor per action, which is leading to obvious and inevitable EMFILE (considering that you've large files hashmap). I suggest you to try reading files in batches (like 10-50 at one time), this will be much faster and will help you avoiding EMFILE.

[indutny]

Also, please use mailing list for that kind of questions.

[Somebi]

I thought it's a bug. Guess i'll better use sync way for this. It's kinda becoming too complicated. It's already implemented via chainer... so i'll have to wrap it in another chainer to make it the way you suggested. It's not very handy i'd say, taking in consideration that node.js is created for high loads and "scalable" development...

[isaacs]

@Somebi You can use https://github.com/isaacs/node-graceful-fs to avoid this sort of EMFILE error. It'll add latency to your file open operations, but it lets you just throw a bunch of readFile's in there and not worry about it.

[Somebi]

Thanks i'll stick to sync file reading. It seems fixed my problem. Sessions reading is launched at the start of my app, before http server is starting to listen to the socket.

[Somebi]

"are put in a queue" yea that's a good idea. It sounds reasonable. Why isn't this implemented in native node.js fs module?

[indutny]

Because we're trying to keep core minimalistic and removing things that can be implemented in "user-land" modules.

[Somebi]

Then at least reconsider your documentation and add tips for situations like that. Create some wiki, where people will share their scalable solutions. Instead of sending everyone to irc or forums, where people repeat same questions again and again or spend hours finding what they are looking for.

Doesn't that sounds reasonable?

Like i said earlier in some post. Make node.js friendly for the beginners. It's kinda ok for me, but by doing so, you can attract more people to the node.js community.

Some comments inside docs would be great as well.

I'm sorry if I hurt someones feelings. :D

But there is always a small part of truth in every critique. Keep that in mind. ;)

There is such thing as usability in the web.

[Somebi]

Too bad that Ryan Dahl is not reading this.

[bnoordhuis]

You know we have a wiki, right?

[Somebi]

I'm talking about references in the docs. For example:

fs.readFile(filename, [options], callback)

tips (show/hide)

• (wiki) opening a lot of files %link%
• etc.

[bnoordhuis]

Right. People haven't exactly been clamoring for that, you're the first one to bring it up. Maybe if more people speak up.

[bradisbell]

I'm actually working on a separate site right now for that very reason. I've found the documentation for Node.js to be thin, and occasionally inaccurate. I intend to build a site to fill the gap, and the contribute documentation fixes back to the project. That way, those of us who prefer annotated documentation can have it, and the core contributors can continue on with the basics, focusing on keeping it clean and accurate.

[isaacs]

Too bad that Ryan Dahl is not reading this.

He might be. But it wouldn't matter so much. You're talking in this thread to me and @indutny and @bnoordhuis, and between the three of us, we've landed over 90% of the last 1000 commits ;)

$ git log --pretty=full -n1000 | egrep Commit: | egrep 'isaacs|Ben Noordhuis|Fedor Indutny' | wc -l
912

I've found the documentation for Node.js to be thin, and occasionally inaccurate.

Please send us patches! The docs are in doc/api/*.markdown. We do prefer the docs to be succinct, but they should always be accurate.

I'd be fine with a doc patch that pointed to some techniques or a wiki page with stuff.

[Somebi]

@isaacs sorry for being rude. @bradisbell would be nice to see it. Where you'll announce it?

[matthew-dean]

Right. People haven't exactly been clamoring for that, you're the first one to bring it up. Maybe if more people speak up.

Wha? EMFILE errors are like the most common errors encountered with the fs module. https://www.google.ca/#q=emfile+node