High memory usage (or memleak?) #43
Comments
|
Update: the 60-minute version is still running, I'll post the results tomorrow. RSS seems to stabilize around 900 MB if I don't stop the clients. Come on, 900 MB (500 of which is garbage!)?! |
|
This is 60 minutes with 1 minute pause after each 60 minute run: Maybe the "leak" is no leak but very very high memory consumption... |
|
Great report, Nico. You are onto something when you say that each connection consumes a lot of memory. I pool rather aggressively, in order to ensure high speed. The 'ws' library is written to be as fast as possible, and will at present outperform all other node.js websocket libraries, for small data amounts as well as large. That said, I will deal with this, to push the memory use down. An alternative is to introduce a configuration option which allows the user to favor either speed or memory consumption. In your test above, are you sending any data for the 10k clients? How large would you say the largest packet is? |
|
This is WebSocket.IO 10-minute (like the very first chart) with Node 0.6. |
|
@einaros The client ist the one in the Gist – the client sends 10k at the beginning of each connection. A configuration option would be amazing! I'll observe the production server, which is running Node 0.6 right now, if memory usage rise ends somewhere ;) I understand that much memory is needed to ensure fast connection handling, however the memory should be freed after some (idle?) time... |
|
For 10k users sending an initial 10k (in a single payload), the consumption would be at least 10.000 * 10k. I agree that the pools should be released. Adding a set of timers to do that without negatively affecting the performance for high connection counts will require thorough consideration before being implemented. Thank you for your continued work on this. We'll get to the bottom of it :) |
|
Well, I open 10,000 clients at the startup, and then open and close 2,000 random clients every minute. So there should be 10,000 clients all the time, but about 20,000 "connection" events (and thus, 20,000 initial 10k messages received) during the 10 minutes. Isn't the pool released after a client disappers? |
|
Each pool is released as the client closes, yes. |
|
@einaros then it must be something different (or poor GC), because after each 10 minute period all clients are disconnected, so there should be no buffer left (so even the shared SlowBuffers should be freed by V8)... |
|
For comparison, this is WebSocket.IO without ws (backported the original Socket.IO hybi modules) and without client tracking (only clientsCount): GC seems to run from time to time, but does not catch everything. Maybe interesting for @guille |
|
@nicokaiser, I wrote the "old" websocket.io hybi parsers as well, so this is all my thing to fix in either case :) |
|
I've found a couple of issues now, which takes care of leaks visible through the v8 profiler's heap dump. With my recent changes, I can spawn up 5k clients to a server, send some data, then disconnect the users, and the resulting heap dump will look more or less as it did before any clients connected. I can't yet get the |
|
And you are not leaking any buffers? As they are shows in RSS and not in the V8 heap On Friday, March 30, 2012 at 10:29 AM, Einar Otto Stangvik wrote:
|
|
@3rd-Eden, for js Buffer/SlowBuffer instances, they should show up in the heap dump. I figured it may be due to allocations in my native extensions, but the results were exactly the same with those disabled. |
|
@einaros that sounds great! That's exactly the kind of issues I get – growing |
|
I can't help but think that the remaining issues are caused by something within node leaking native resources, probably triggered by something I am doing. And I'm guessing that something is the buffer pool. Now the question is how the buffers are handled / used wrong, and how they wind up leaking native allocation blocks. |
|
On the production system, I disabled BufferPools (by using So it must be either the buffertool (mask) (I cannot test the JS version from the Windows port, as I'm using Node 0.4) or something else... |
|
@nicokaiser, I'll have another look at the bufferutil and see what I can find. |
|
That could certainly be the issue here, as there also countless reports that socket.io on 0.4 leaks less memory than socket.io on node 0.6. On Friday, March 30, 2012 at 10:48 AM, Einar Otto Stangvik wrote:
|
|
@nicokaiser, I disabled my native extensions again, to no avail. Will go pick a fight with the buffers now. |
|
Another observation:
|
|
Well it is definitely the buffers. The question is how / why they are released by v8, but the native backing isn't. |
|
Is there a way to completely avoid the Buffers and e.g. use Strings for testing (I know this is highly inefficient)? |
|
Well I'm not saying it's my buffers, just node |
|
@einaros yes, but on a high-traffic websocket server, |
|
You can't avoid Buffer objects for binary network operations. |
|
Node will transform strings to Buffer automatically anyways, so even if we didn't do binary network operations it would still be impossible (if I remember correctly) On Friday, March 30, 2012 at 11:13 AM, Nico Kaiser wrote:
|
|
Hm, ok, none of the optimizations had any effect on the very high memory usage – I assume Node is failing to release unused Buffer memory: http://cl.ly/1R442g3t2d1T3S152s0i Do you think compiling node for ia32 (instead of x64) might change something? |
|
@crickeys can this make a different if I'm not using https? (only some crypt functions) I'm using the bundled libssl (0.9.8 I think). |
|
according to this: nodejs/node-v0.x-archive#2653 yes |
|
Actually, I may not have properly updated libcrypto eventhough I successfully built node with an openssl 1.0.0i. It appears the older 0.9.8 libcrypto is being used by my node. Let me keep playing with this, you may want to try it too as I really don't know what I'm doing here :) |
|
@nicokaiser Have you had any memory related crashes lately? I think it's time to summarize where we are at with this again. With the latest node version and latest ws version - are anyone seeing any actual crashes due to out of memory errors? |
|
@einaros We changed some things in the infrastructure (split the server, sticky sessions with HAproxy), which causes each server not to have more than ~10.000 connections. The last 2 weeks look like this for one of the servers (node 0.6.16, ws 0.4.13), memory and clients: Memory is at about 1.4 GB per server, with about 9k clients. I'll try to keep the servers running (needed to restart them because of an update after these screenshots) to see if memory rises above 2 GB (when swapping kicks in). |
|
The fact that it doesn't keep growing beyond the numbers seen there could suggest that there isn't actually any leakage anymore. At the same time, there could be a tiny leakage (anywhere), which over a greater extent of time than measured here could cause a crash. At some point I need to build a stress testing environment to see how it reacts to sustained loads from more than 50k clients.. |
|
Update: the installed version I mentioned adds socket.destroy to every socket.end, see #64. Don't know if this makes a difference; I can change one of the servers to use vanilla ws 0.4.14 next week. |
|
Since this hasn't been brought back up, I'm (temporarily) concluding that the issue has been covered by the fixes as of late. The lingering memory seen in graphs right now are due to free'd memory not being released immediately. Should anyone actually get a process crash due to running out of memory, we'll have to revisit it. Judging from the lack of talk here lately, that hasn't been the case in quite some time. |
|
Confirmed. The process takes lots of memory over time, but this seems like "node.js is lazy to clean up memory". A server that actually uses the memory it gets is ok as long as it does not take more than available. |
|
Confirmed that its happening again. Server eventually crashes after running out of RAM. node 0.8.9 |
|
I can confirm that our server uses more and more memory over time and eventually crashes when the memory use gets high. The more connections and data transferred the quicker the crash happens. node 0.8.9 |
|
@nodenstuff, thanks for reporting. Since nothing much has been changed in ws over the last few months, I'm not immediately jumping to the conclusion that this is caused by the ws parsers. If there wasn't a crashing leak four months ago, but there is today, that's a regression in another component - or a breaking change in node which ws will have to be updated to work with. @kanaka, are you using socket.io? @nicokaiser, how has your situation been lately? Any crashes? |
|
@einaros, no socket.io, it's a very simple server that just uses the ws module directly: https://github.com/n01se/1110/blob/6c90e0efc3a4afeb099f79d18d471a5936de1d3e/server.js |
|
@einaros No crashes, but this may be because we reduced the load from our servers and 2 GB memory seem to be enough. No I cannot check this at the moment, but I'm pretty sure, that, while Node allocates 1.5 GB memory, one process will crash if I start another process on this server that uses more than 512 MB. Fact is that Node, starting from 0.6.x, fails to return unused memory to the kernel, but keeps the RSS (!!) allocated. Which maybe no problem if this Node process is the only thing that runs on the server (and thus is allowed to eat all of its memory), but it's not nice and definitely a bug. |
|
No news on http://code.google.com/p/v8/issues/detail?id=2073 by the way ... |
|
@3rd-Eden seems to have similar problems: https://twitter.com/3rdEden/status/255327493572132865 |
|
I have just tried with NodeJS 0.9.2 (with a little modification to socket.io 0.9.10 because it cannot find socket.io.js client). This problem still going on. Seem like only way to solve this is use node 0.4.12. :( I want to use native cluster more but seem like I do not have other choice... |
|
I enabled flashsocket on 0.9.10 and I didn't run out of ram today. Might just be a fluke. But currently hold steady around 2GB total in use, that includes redis and everything else running on the server. I will update if that changes. |
|
Maybe related: https://groups.google.com/forum/#!topic/nodejs/KM0Yis-LNpg (thanks @3rd-Eden for the link) |
|
Good News on rewrite buffer: Branch: Bad news: |
|
I just setup stud to terminate ssl and now node do not need to process ssl anymore. Now, my memory consumption decrease about 3 times. Maybe this bug relate to using wss via https? |
|
@nicokaiser how did you solve this in the end? I see a very high memory consumption, and when sockets are closed memory is still used (at least most of it). |
|
@damianobarbati I simplified the application code to avoid retained references, this did already solve the problem. And I switched to uws, which also gave a huge memory advantage. |
|
@nicokaiser could you show a simplified example about "avoiding retained references"? Thanks for helping! |
|
Hi!
Yes, the GC got much better since Node 6 or 8. I‘m trying to remove event listeners on the connection objects etc., but associated data should not be a big problem. You could try to unset these properties in the close event to see if there is a difference, but I doubt there is much impact.
Also, disabling the compression made a huge difference back then in my case...
Nico
… Am 06.06.2018 um 10:59 schrieb damiano.barbati ***@***.***>:
@nicokaiser could you show a simplified example about "avoiding retained references"?
I typically associate data to ws object (i.e: ws.token = token) but shouldn't close event and terminate method be enough to have the gc clean up the empty references?
Thanks for helping!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
Hi all!
I created a small example for the memory leak my ws (or WebSocket.IO) server has.
This small server only accepts WebSocket connections and sends one message.
Client count and memory usage is displayed every 2 seconds.
The client open 10,000 connections, then every minute 2,000 new connections are openend and 2,000 are closed, so the count stays at about 10,000.
https://gist.github.com/2152933
The graph is the output of server.js (run on a EC2 "large" instance with node 0.6.13 and ws 0.4.9).
Questions:
The effect is the same (but slower) with only 1,000 client connections per minute, but things get even worse when I don't stop the client after 10 minutes. Our production server runs with about 30-40% CPU constantly (30k connections), 1-5% at night (1-2k connections), but is never completely idle. The growing of the RSS usage never seems to stop.
On the production server, RSS grows until node crashes (0.4.x crashes at 1 GB) or the process gets killed by the system (0.6 supports more than 1 GB).
I'll try two things tomorrow:
The setup is a default Amazon EC2 large instance, so this must be an issue for anyone who runs a WebSocket server using ws (or must likely also WebSocket.IO with ws receivers) with some traffic. I refuse to believe Node is not capable of serving this.
Or am I missing something?
Nico
The text was updated successfully, but these errors were encountered: