Memory leak in unzip()

[vflash]

когда съедает всю память процесс умирает. тест делал на виртуальной машине с 1гб памятью.
eats all memory when the process dies. test done on a virtual machine with 1GB of memory.

latest (0.7.0-pre) node
node --expose-gc

var zlib = require('zlib');
var buffer = new Buffer('eJzT0yMAAGTvBe8=', 'base64');
function nullfun(){};

function test() {
    console.log(process.memoryUsage());
    for (var i = 0; i < 100; i++) {
        zlib.unzip(buffer, nullfun);
    };
    process.nextTick(test);
};
process.nextTick(test);

....
{ rss: 870871040, heapTotal: 42841856, heapUsed: 8936624 }
{ rss: 871223296, heapTotal: 42841856, heapUsed: 9354648 }
{ rss: 871858176, heapTotal: 42841856, heapUsed: 9630272 }
{ rss: 872452096, heapTotal: 42841856, heapUsed: 10048328 }
{ rss: 872853504, heapTotal: 42841856, heapUsed: 10331552 }
Убито
vflash@vu:~/work$

[isaacs]

Fixed in 0.6.8. Not yet ported back to master.

[vflash]

в 0.6.8 повторяется
in 0.6.8 is repeated

[bnoordhuis]

@isaacs: Can this issue be closed?

[plievone]

I just tried this on 0.6.15 (ubuntu) and there seems to be a memory leak, even if I add gc() in there. Does this comment mean that this is fixed only in 0.7.x: #2504 (comment)

Also see #2933.

[vvo]

I too have seen this issue on zlib.unzip (that was causing our production servers to crash)

Using only zlib.gzunip convenience method on the same data does not trigger any memory leak (!?).

Memory drop on our servers

https://twitter.com/zeroload/status/210678245576818688

This graph shows the memory consumption when switching from zlib.unzip to zlib.gunzip on gzipped content.
Our product features are still ok. I know it doesn't prove anything for node and I'll try to make a good test file for this.

Old zlib version? (=> no)

UPDATE: I tested with latest zlib source code, recompiling nodejs with it but that did not change the behavior.

Leaving rest of the story for archive.
While I'm clearly not an expert, I did some searching about the zlib binding in node and found this:

Reading https://github.com/joyent/node/blob/v0.6.19-release/deps/zlib/README.chromium

Name: zlib
URL: http://zlib.net/
Version: 1.2.3

Then in the licence file we see zlib version 1.2.4, March 14th, 2010 and then in zlib.h we see version 1.2.3, July 18th, 2005.

Hard to tell which version is actually used.

So node use perhaps zlib 1.2.3. which seems old.

But old does not mean crappy, if it's used on the whole chromium project it must be customized and perhaps better than original?

Reading http://zlib.net/ChangeLog.txt and searching for leak :

Changes in 1.2.4.2 (9 Apr 2010)
- Fix memory leak on error in gz_open()
Changes in 1.2.4 (14 Mar 2010)
- Fix memory leaks in gzclose_r() and gzclose_w(), file leak in gz_open()

Also seems like theses leaks were related to using zlib with files not in memory data so I just lost myself here :)

Hope it helps.

[rrusso2007]

Just made the same change from unzip to gunzip and my prod memory munin graph now looks exactly like @vvo . this issue report was exactly what I was looking for. What exactly is the implementation difference between unzip and gunzip anyway?

[vvo]

AH! Great your adding some support to this i-m-going-completely-nuts issue. I do not have any more clue sorry.

[isaacs]

Fix will be included in 0.8.3.

[vflash]

Thank you very much

[OpakAlex]

Same issue on 6.2.1

[addaleax]

@OpakAlex This is an old issue that has been fixed for a long, long time. If you have trouble with current Node versions, please create an issue at https://github.com/nodejs/node and include as much detail as possible (esp. code to reproduce).