memory leak

[bferreira]

Hi,

We are using 0.9.6,

We've been noticing memory usage increasing until the server stops responding.
It usually would take a few days until this happened but now we increasing usage (about 220 peers) , the server stops responding in less than one day.

Should we be destroying some objects, releasing some connection? Anything we are not doing that we should do to stop this?

Our client setup:

var socket;
var subs = new pc.util.OrderedMap();

function addSubscription(subtopic, callback) {
    if(!PCAUser.loggedIn) return;
    subs.put(subtopic, callback);
    if(!socket) {
        initSockeIO();
    } else {
        socket.json.send({
            command : "subscribe",
            subscriptions: [subtopic]
        });
    }
}

function initSockeIO() {
    socket = io.connect(liveUpdateConfig.host,
        {port: liveUpdateConfig.port, rememberTransport: false});
    socket.on('message', function(obj) {
        if (obj.subtopic) {
            var key = obj.subtopic;
            if (subs.get(key)) {
                // execute callback
                window[subs.get(key)](obj); // FIXME bad approach
            }
        }
    });

    socket.on('connect', function subscribe(obj) {
        socket.json.send({
            command : "subscribe",
            subscriptions : subs.keys()
        });
    });

    socket.on('disconnect', function(obj) {
        setTimeout(function() {
            socket = io.connect(liveUpdateConfig.host,
                {port: liveUpdateConfig.port, rememberTransport: false});
        }, 1000);
    });
}

jQuery(document).unload(function() {
    if(socket) {
        socket.emit('disconnect');
    }
});

Our server setup:

//--------- Socket.IO server ---------


var socketio = io.listen(server, {
 transports: ['websocket','flashsocket','htmlfile','jsonp-polling']
});


socketio.sockets.on('connection', function(client) {
    stats.peersCount ++;

    var msg = "Client connected: " + client.id + ", active peers count: " + stats.peersCount;
    logger.info(msg);
    socketio.sockets.send(msg);

    client.on('message', function(message) {
        var decodedMessage = util.inspect(message, true, 3);
        logger.info("\n\Client " + client.id + " Message received : " + decodedMessage);

        if( decodedMessage.indexOf('monitor') > -1) {
            logger.info('monitor enabled');
            if(!subscriptions[monitor_subscription]) {
                subscriptions[monitor_subscription] = [client];
            } else {
                subscriptions[monitor_subscription].push(client);
            }
            logger.info("Client " + client.id + " added as monitor");
            client.json.send({ subscriptions : [monitor_subscription] });

        } else
        if (typeof message.command != undefined) {
            try {
                switch (message.command) {
                    case "subscribe":
                        var incomingSubscriptions = message.subscriptions;
                        //if the subscriptions are in a string, split them into an array
                        if(message.subscriptions.replace) {
                            incomingSubscriptions = message.subscriptions.replace('[','').replace(']','').replace(/[\"\s]/g, '').split(',');
                        }
                        for (var i = 0; i < incomingSubscriptions.length; i++) {
                            var subtopic = incomingSubscriptions[i];
                            if (!subscriptions[subtopic]) {
                                subscriptions[subtopic] = [client];
                            } else if(subscriptions[subtopic].indexOf(client) < 0) {
                                subscriptions[subtopic].push(client);
                            }
                        }
                        logger.info("Client " + client.id + " subscribed");
                        client.json.send({ subscriptions : message.subscriptions });
                        break;
                    default:
                        logger.info("Undefined message: " + decodedMessage);
                }
            } catch (e) {
                logger.error('Error while handling message ' + decodedMessage + ", " + e.message);
            }
        }
    });

    client.on('disconnect', function() {
        stats.peersCount --;

        var msg = "Client " + client.id + " disconnected, active peers count: " + stats.peersCount;
        socketio.sockets.send(msg);
        logger.debug(msg);

        for (var subtopic in subscriptions) {
            if (subscriptions[subtopic].length > 0) {
                subscriptions[subtopic] = subscriptions[subtopic].filter(function(element) {
                    return element.id != client.id
                });
            }
        }
    });
});

Thanks,
Barbara

[crickeys]

Hi, I had this issue long ago and spent a long time trying to figure it out. The closest I could get was to just disable "websockets" as a transport. Sad to do that, but that was the only thing keeping memory safe.

[bferreira]

Hi, Thank you. That is sad because that's the transport that we really would like to use, since our users will only use Chrome in the future.
But we are now seen the socket.io stop responding without any high memory consumption. We are logging everything but nothing is tell us much.

[paiboonpa]

Same with me on Socket.io 0.9.10, NodeJS 0.8.10. Websocket is leaking memory with 600 concurrent connection, rss went up to around 900MB. I have to restart every hours or else, it will be really hard to connect to server after consume this much memory.

[bferreira]

Heha,

you can probably install nodetime and try to figure out if you have a leak
https://nodetime.com/

We were having to many jsonp-polling requests. We were able to get down from 402 req/s to 42 req/s

And finnaly we got an error:
local/nodejs/live-update-nodejs/node_modules/socket.io/lib/transports/jsonp-pol

 ling.js:88

this.response.writeHead(200, {
^
TypeError: Cannot call method 'writeHead' of undefined
at JSONPPolling.doWrite (/local/nodejs/live-update-nodejs/node_modules/socke

 t.io/lib/transports/jsonp-polling.js:88:17)
at JSONPPolling.write (/local/nodejs/live-update-nodejs/node_modules/socket.                                            

 io/lib/transports/http-polling.js:132:8)
at JSONPPolling.packet (/local/nodejs/live-update-nodejs/node_modules/socket                                            

 .io/lib/transport.js:515:15)
at JSONPPolling.error (/local/nodejs/live-update-nodejs/node_modules/socket.                                            

 io/lib/transport.js:498:8)
at JSONPPolling.onData (/local/nodejs/live-update-nodejs/node_modules/socket                                            

 .io/lib/transports/jsonp-polling.js:69:10)
at IncomingMessage.<anonymous> (/local/nodejs/live-update-nodejs/node_module                                            

 s/socket.io/lib/transports/http.js:65:12)
at IncomingMessage.emit (events.js:64:17)
at HTTPParser.parserOnMessageComplete [as onMessageComplete] (http.js:130:23                                            

 )
at Socket.ondata (http.js:1506:22)
at TCP.onread (net.js:374:27)

So I guess our problem is with socket.io.

[paiboonpa]

I already got nodetime installed. I already used node-webkit-agent too. Nodetime cannot pinpoint the problem because sample object is quite random when use heap snapshot. For node-webkit-agent, it looks better, can drill down more but still, very hard to pinpoint a problem. There's no leak from one large object or else it would be caught very easy. I already tried using only xhr-polling and memory changed from 900MB/hour to 100 MB/hour but it's too slow for our real-time game. Not sure if jsonp is faster than xhr-polling or not.

[nodenstuff]

I can confirm. Our server will also eventually run out of memory and start to use swap during load when using socket.io and websockets.

I will try to do a baseline with only socket.io, without express or other requires.

I will run more tests but currently using:

node v0.8.9
socket.io v0.9.10
express v.3.0.0rc4

Disabling websockets significantly reduces memory load. I am currently using jsonp-polling and update this post with results at then end of the week.

[paiboonpa]

Real culprit is in
websockets/ws#43
Socket.io used this library as websocket transport. Short conclusion is Gabage Collector problem, v8 engine bug. Seem sockjs also has this issue with websocket:
sockjs/sockjs-node#62

One way to fix this problem is change node to version 0.4.12. Websocket will work smoothly with that version. However, lots of tools will be disable too like Nodetime, node-webkit-agent(can use node-inspector instead but I think it's harder to use.) including native cluster which introduced in Node 0.6 (may use other library instead). Another way is use XHR, JSON-P instead which is a lot slower.
I heard that in NodeJS 0.9, there will be a re-write on Garbage Collector but sadly, after tested 0.9.2 with Socket.io, it failed since cannot find socket.io.js.

[nodenstuff]

Thanks. I will give 0.4.12 a try on our system this week. I'm also not sure that a lot of npm stuff will still work.

[paiboonpa]

Still, cannot rely on NodeJS 0.4 much because engine.io (new core for Socket.io 1.0) will support only NodeJS 0.6 or higher. Really make me a hard decision to stay on 0.4 or try restart node 0.8 every x hour instead. :(

[nodenstuff]

I agree but for my purposes I'd rather not have to restart the server consistently versus the usual weekly reboot. As it stands right now in any high use situations it's a choice between potential service interruptions or using an old version.

I plan on giving engine.io and socket.io 1.0 some time to stabilize before going into production.

I've been testing my current setup using 4GB of ram and I can only get to about 200-300 users before memory is sucked up in a few hours. I'm really not doing much other than some redis pubsub and relaying messages.

[rauchg]

Actually making engine.io work on 0.4 again wouldn't be that hard. It used to be compatible until a few weeks ago.

[paiboonpa]

@guille Wow. Great to hear! :)

[anibalvy]

Dears, sorry my ignorance, but I have a very long message on my console finnish with this:
info - socket.io started
......
...
..
,"1048560":0,"1048561":0,"1048562":0,"1048563":0,"1048564":0,"1048565":0,"1048566":0,"1048567":0,"1048568":0,"1048569":0,"1048570":0,"1048571":0,"1048572":0,"1048573":0,"1048574":0,"1048575":0,"length":1048576},"offset":21456}]}

After a very long time I have the result on my client. Is this the same issue?

[nodenstuff]

Whatever is causing this bug pretty much renders socket.io and websockets on node useless is a large scale production environment.

I have been trying to track this down but I am not sure where its leaking. Sockjs has the same issues so I think its the websocket library not doing something correctly with the new versions of node. {Edit Sockjs not using the same websocket library}

It's eating 3 gigs of ram on one server will under 100 people connected.

[3rd-Eden]

Sockjs isn't using the same WebSocket library as Socket.IO does. So there is something else going on here.

[rauchg]

@nodenstuff did you try with socket.io master ? (uses ws)

[nodenstuff]

Thanks guys. I'll pull down the most recent from github and give it a try on node 0.8.12.

[crickeys]

Master still leaks.

Sent from my iPhone

On Oct 16, 2012, at 12:38 PM, Guillermo Rauch [email protected]
wrote:

@nodenstuff https://github.com/nodenstuff did you try with
socket.iomaster ? (uses
ws)

—
Reply to this email directly or view it on
GitHubhttps://github.com//issues/1015#issuecomment-9499274.

[nodenstuff]

updated to node 0.8.14 - will update this post accordingly if issue is fixed

Not going to do any charts or anything - newest node is no longer crashing our server during peak load - I'd say thats an improvement

6 hours - ram usage has been stable but increasing VIRT 671M RES 37096 SHR 4820
10 hours - ram usage has been stable but increasing VIRT 670M RES 45104 SHR 4820

Oct 28 - sever started swapping and crashed - looks like issue is still there just not as bad :S

[nodenstuff]

After some configuration with mysql and dropping all but jsonp and websockets - we've managed to stay stable. There is still a very slow memory leak - and some reconnection issues here and there

trying to track it down

[yurynix]

Same issue here, I've tried running 0.9.10 with node 0.9.3 and 0.8.14 on win2003server 32bit,
It leaks until it crashes when it reaches >=2GB memory with:
FATAL ERROR: Evacuation Allocation failed - process out of memory

Takes it about 6 hours on my test setup, with 2k concurrents, after seeing 200k connections.

Some stats before the crash:

Node version: v0.9.3
Uptime(seconds): 19442.942897800007
PID: 3988
Versions: { http_parser: '1.0',
node: '0.9.3',
v8: '3.13.7.4',
ares: '1.9.0-DEV',
uv: '0.9',
zlib: '1.2.3',
openssl: '1.0.1c' }
io.sockets.clients().length: 2163
Connections count: 2164
Max connections count: 2708
Connections seen: 196957
Memory: { rss: 1147138048, heapTotal: 83368576, heapUsed: 54788068 }
V8Statistics: { total_committed_bytes: 101298688,
new_space_live_bytes: 457036,
new_space_available_bytes: 591508,
new_space_commited_bytes: 2097152,
old_pointer_space_live_bytes: 34463120,
old_pointer_space_available_bytes: 12520304,
old_pointer_space_commited_bytes: 46983424,
old_data_space_live_bytes: 15699836,
old_data_space_available_bytes: 11920132,
old_data_space_commited_bytes: 27619968,
code_space_live_bytes: 2713728,
code_space_available_bytes: 1693568,
code_space_commited_bytes: 4407296,
cell_space_live_bytes: 43280,
cell_space_available_bytes: 55024,
cell_space_commited_bytes: 98304,
lo_space_live_bytes: 0,
lo_space_available_bytes: 665177472,
lo_space_commited_bytes: 0,
amount_of_external_allocated_memory: 1054993336 }

[rickgbw]

Node 0.4 is not needed...
I had the same problem and after try a lot of versions, now we use the node v0.6.2
I dont know why, but works fine. The server is up since 2 months

[paiboonpa]

@rickgbw It works! Thanks!

[yurynix]

@rickgbw Thanks for pointing that out =)
However I'm still having issues, not sure if it's the same issue manifasting in another way or completly unrelated ( maybe 0.6.2 bug? ).

After around 200K connections the server crashing in XHRpolling / JSONPpolling with "this.response" is undefined,
i've tracked it to https://github.com/LearnBoost/socket.io/blob/master/lib/transports/http-polling.js#L79
here self.response is already undefined when called from the setTimeout().
It happens after about 200K connections, so in my case it crashing after the same amount of time/connections less or more, just with different reason.

I've added checks if response is undefined all around the http-polling now, let's see if it helps.

[benkay]

nodejs/node-v0.x-archive@2433ec8 just landed which looks promising.

[paiboonpa]

I just setup stud to terminate ssl and now node do not need to process ssl anymore. Now, my memory consumption decrease about 3 times. Maybe this bug relate to using wss via https?

[eveiga]

@heha Dont think so, I'm not using https and still have these hungry memory issues... Downgrading to node v0.6.21 (last stable of v0.6) gave me some improvements on this. Will try 0.7 just in case...

[nodenstuff]

I upgraded to the latest 0.8 node and memory issues are gone - also no
issues with 0.9

On Tue, Feb 19, 2013 at 8:27 AM, eveiga [email protected] wrote:

@heha https://github.com/heha Dont think so, I'm not using https and
still have these hungry memory issues... Downgrading to node v0.6.21 (last
stable of v0.6) gave me some improvements on this. Will try 0.7 just in
case...

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1015#issuecomment-13778002.

[eveiga]

@nodenstuff v0.8.20?

[nodenstuff]

0.8.20 correct - complied form source on Debian 6 - and i use https - it
was crashing servers with memory usage on pre-0.8.14 or so ... I was using
0.9.x for a while but havent had the time to upgrade a few sections of code
in my app

i use redis to store my client data as well

it depends on what your specific implementation is using with regard to
memory

On Tue, Feb 19, 2013 at 9:06 AM, eveiga [email protected] wrote:

@nodenstuff https://github.com/nodenstuff v0.8.20?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1015#issuecomment-13780401.

[nodenstuff]

There is a leak somewhere in either node or the wss implementation that
only manifests when you reach a high number of connections

On Tue, Feb 19, 2013 at 9:18 AM, James Stanford [email protected]:

0.8.20 correct - complied form source on Debian 6 - and i use https - it
was crashing servers with memory usage on pre-0.8.14 or so ... I was using
0.9.x for a while but havent had the time to upgrade a few sections of code
in my app

i use redis to store my client data as well

it depends on what your specific implementation is using with regard to
memory

On Tue, Feb 19, 2013 at 9:06 AM, eveiga [email protected] wrote:

@nodenstuff https://github.com/nodenstuff v0.8.20?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1015#issuecomment-13780401.

[eveiga]

Ok! Gonna give a try on 0.8.20, makes me some "itch" still using 0.6.21 just because of this issue with memory consumption.

Either way I couldn't keep restarting node processes on a daily basis, it's just not maintainable!

I would love to use redis to store socket information, but it just doesn't scale :( #862

Best regards!

[nodenstuff]

You are right, I don't use the socket.io redis implementation - hiredis
seems to work for my app which is really just storing a complex online user
list.

Setting a cron/bash to restart it once a week is all I have done. Memory
usage is still high but it's not crashing the server anymore.

Let us know how your tests go and if it reduces the leaks from your app.

On Tue, Feb 19, 2013 at 9:25 AM, eveiga [email protected] wrote:

Ok! Gonna give a try on 0.8.20, makes me some "itch" still using 0.6.21
just because of this issue with memory consumption.

Either way I couldn't keep restarting node processes on a daily basis,
it's just not maintainable!

I would love to use redis to store socket information, but it just doesn't
scale :( #862 #862

Best regards!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1015#issuecomment-13781554.

[eveiga]

Well, came back to starting point. Only Node.js v0.6.2 works well for me :/ It's the only version that it's not memory hungry... v0.6.21 and 0.8.20 didn't solve the problem!

Either way, beside the fact that with 0.6.2 the memory consumption is uniform, I'm having @yurynix problem.
Did you find your leak?

[yurynix]

@eveiga
Switched to sockjs, it's less feature rich, however made the problems tracking (sockjs/sockjs-node#99) much easier for me, now my app is able to run smooth with >month uptime.

[eveiga]

Thanks for the quick reply. Are you still on node.js 0.6? or 0.8 fits well?

[eveiga]

Well got back to 0.8.19, gonna give a try on sockjs...

[jpallen]

I have also been experiencing significant memory leaks with Socket.io and Node 0.8. I've identified the problem and submitted a fix: #1177. Hopefully it will make it into the release version soon. This memory leak will only affect you if you are using Socket.io over HTTPS (note the S!) and Node 0.8, but if you are using these two things together then it will almost certainly affect you.

[eveiga]

@jpallen Hopefully this will be merged on 0.9! thanks a lot

[mons54]

Hello, same problem for me. I use version 0.8 node and socket.io 0.9 with SSL. Happens every day around 18:00 this error: Allocation failed - process out of memory. Do you think I should use the node with version 0.6?

[jpallen]

Another option to fix this memory leak is to set:

require("tls").SLAB_BUFFER_SIZE = 100 * 1024 # 100Kb

The default is 10Mb and it is these slab buffers that cause the leak. Making them smaller should help things significantly, although possibly at the risk of reducing the speed of HTTPS connections. This is only available in the most recent versions of Node (0.8.20 and 0.8.21 I think). See nodejs/node-v0.x-archive#4636 for more details.

[eveiga]

Hey everyone, this issue is solved with 0.9.14!

[yurynix]

My socket.io leaked without https, so i doubt the https fix had anything to do with my case.

[eveiga]

I hadn't https too, and 0.9.14 fixed!

[benkay]

Same here - I'm not using https but memory usage is a whole lot better with 0.9.14

[Neumann-Valle]

I been trying to find out why my simple script was leaking so much memory, my application don't have that many concurrent user and my memory would rise to 30% usage with merely 30 users connected..

Will get back with info if it fixed my memory issue.. I am not using https so lets see what happens..

[Nexum]

I currently use node v0.8.23 and socket.io 0.9.14 (no https) with about 2500 concurrent connections
I'm evaluating the possibility to use socket.io right now so this is my script:

require("tls").SLAB_BUFFER_SIZE = 100 * 1024; // 100Kb

var fs = require('fs');
var io = require('socket.io').listen(12031);
io.set('transports', [
    'websocket',
    'flashsocket',
    'htmlfile',
    'jsonp-polling',
    'xhr-polling'
]);
var stats = {
    connections: 0,
    max: 0,
    complete: 0
};
io.set('log level', 1);
io.sockets.on('connection', function (socket) {
    stats.connections++;
    stats.complete++;
    if(stats.connections > stats.max) {
        stats.max = stats.connections;
    }
    socket.on('getstats', function(data, cb) {
        cb(stats);
    });
    socket.on('connect_failed', function() {
        console.log('connect_failed');
        stats.connections--;
        delete socket;
    });
    socket.on('error', function() {
        console.log('error');
        stats.connections--;
        delete socket;
    });
    socket.on('reconnect_failed', function() {
        console.log('reconnect_failed');
        stats.connections--;
        delete socket;
    });
    socket.on('disconnect', function() {
        stats.connections--;
        delete socket;
    });
});

setInterval(function() {
    fs.writeFile('stats', JSON.stringify(stats));
}, 1000);

the client doesnt do anything just connect, noteven an event ist fired, the getstats is only used by one client (which is an company internal sats server and is only run once a second)

The memory consumption improved a lot with the node and socket.io update.
i'm still monitoring the memory usage and will update later today when i can confirm the fix or if the memory still leaks

[Nexum]

After 2 hours i have to say the memory is still going up :-( whith about 2200 connections all the time i went from 80mb to 134mb :-(

[Neumann-Valle]

As of a test of a whole day with same traffic, as I stated I don't have much traffic, my memory usage is day to night, start at 2.1 % grows to 3.7 during peak hour and returned to a healthy 2.5 % ..
This patch really helped a lot with my app..

FYI the socket.io Google group is a joke right now I joined and tried to post and told me it needed moderation but it seems there isn't a moderator available.

. Is there still a living person in charge of google groups?

[Nexum]

Good news! seems the memory just went to 200mb and now stays there, thats great!
Kepp up the good work! :-)

[3rd-Eden]

@utan I have access to google groups, but I can't moderate it because google groups keeps giving errors.

[Neumann-Valle]

@ 3rd-Eden thanks for your help.. github should have some forum group so we can come here and ask questions..
Don't you think is a good idea?

For the most part I'm only left putting questions in other forums,,

[vinothbtechit]

I just downgraded version from v0.10 to v0.6.9, memory leak reduced from 24,000K to 9,000K approximately.