restart a hang process of node.js

[raitucarp]

Well, suppose I have website with 10K - 50K hits (include crawler)
My node.js server is hang. Node process doesn't exit, etc
How pm2 handle that?

Because it just happened to me. Indications of process hang is:

• server doesn't returning any response to client / user
• load balancer (nginx) doesn't return 502, because there is exist a process listening on
• server doesn't exit the process
• I had install toobusy-js modules, but it doesn't has an effect, doesn't response with 503 either
• server doesn't hit undhandledexception, thus doesn't exit the process
• pm2 logs myapp doesn't move (because I use default logger of express)

Thanks

[jshkurti]

Can you tail -f ~/.pm2/pm2.log please ?

[raitucarp]

I restart it manually, not so special at all

2015-03-05 17:10:44: App name:myapp id:2 exited with code null
2015-03-05 17:10:44: Process with pid 13322 killed
2015-03-05 17:10:44: Starting execution sequence in -fork mode- for app name:myapp id:2
2015-03-05 17:10:44: App name:myapp id:2 online
2015-03-05 19:21:46: App closed with code: null
2015-03-05 19:21:46: App name:myapp id:2 exited with code null
2015-03-05 19:21:46: Process with pid 31751 killed
2015-03-05 19:21:46: Starting execution sequence in -fork mode- for app name:myapp id:2
2015-03-05 19:21:46: App name:myapp id:2 online

[jshkurti]

So after the manual restart it still doesn't work ?
Is this your problem ?
Or are you saying you wish PM2 had a way to automatically detect this case scenario and restart your app if it happens again ?

[raitucarp]

After manual restart it works again.

Yes that I mean. Is pm2 can detect such scenario? because right now, PM2 Can't detect if node hang. I want pm2 to detect it, so I don't restart it manually. I'm tired, because user get my site down, most of the time.

This happens to me because of node 0.12.0. I think that is the problem, the node process is quickly becomes hang (maybe because of aggressive of GC, max sockets Infinity). Especially, when I activated pm2 keymetrics, with pmx. That node process hangs more quickly. Until now, I don't know what causes the problem.

I can't backward to node 0.10.34. Which is has uptime 1 - 2 months (if I am not updating my code). Because I set package.json, latest to all my package.

Still, I just want such feature, PM2 can detect hang node.js process. So I am not restarting it manually again, and not watching my realtime analytics that user is 0 (indicates that the site is down) every minutes.

[jshkurti]

Ok, I think the only way to do that is by using PM2 programmatically.
Have a look at this example and try understanding how it works and then you would be able to do what you need.

So there is file pm2.js :

var pm2 = require('pm2');

// Connect or launch PM2                                                                      
pm2.connect(function(err) {

    // Start a script on the current folder                                                   
  pm2.start('app.js', { name: 'myApp' }, function(err, proc) {
    if (err) throw new Error(err);

    setInterval(function() {
      // Get all processes running                                                            
      pm2.list(function(err, process_list) {
        if (err) throw new Error(err);

        process_list.forEach(function(proc) {
          console.log(proc.pm2_env.axm_monitor);
          if (proc.pm2_env.axm_monitor['User count'] > 60) {
            pm2.restart(proc.pm2_env.pm_id, function() {});
          }
        });
      });
    }, 2000);
  });
});

And then there is file app.js :

var pmx   = require('pmx');
var probe = pmx.probe();

var user_count = 0;

var user  = probe.metric({
  name  : 'User count',
  value : function() { return user_count; }
});

setInterval(function() {
  console.log('hey');
  user_count += 9;
}, 1000);

No need to say you have to npm i pm2 and npm i pmx either in the local folder or globally -g.
And finally do node pm2.js (don't forget to kill pm2 daemon before that pm2 kill) and see what happens.
Hint: you can also start pm2.js with pm2 itself.

Some further documentation on using pm2 programmatically here : https://github.com/Unitech/PM2/blob/development/ADVANCED_README.md#using-pm2-programmatically

If you have any questions I'll be glad to answer them :)

[raitucarp]

Hmm I read the code, it's restart when user more than 60, it will restart
but, what happens with my case? detect node.js server process is hang or not, isn't?
It's difficult I think

[jshkurti]

You said that when your 'real-time user' counter was 0 it means the server is down.
So you have to set a probe with pmx as I did in my code but for your user counter variable and then restart your app when this probe is equal to 0.

[raitucarp]

I count user manually via google analytics realtime by watching it. That user not visiting my site, thus my site down

It will be complicated it I count it via pmx

[jshkurti]

Ok then you can just ping your server and if it doesn't respond restart it :)
Here we go, I've done the work for you, you just have to replace the constants.

var request = require('request');
var pm2     = require('pm2');

var ping_timeout    = 3000;
var pm2_app_name    = 'app';
var worker_interval = 1000;
var hostname        = 'http://localhost:8000/';

var ping_server = function(timeout, callb) {
  var timer = setTimeout(function() {
    callb(false);
  }, timeout);

  request(hostname, function(err, res, body) {
    clearTimeout(timer);

    if (err || res.statusCode != 200)
      return callb(false);

    callb(true);
  });
};


pm2.connect(function(err) {
  if (err) throw err;

  setTimeout(function worker() {
    ping_server(ping_timeout, function(ok) {
      if (!ok) {
        console.log('server down, restarting it');
        pm2.restart(pm2_app_name, function() {});
      }
      else {
        console.log('server up');
      }
      setTimeout(worker, worker_interval);
    });
  }, worker_interval);
});

You can of course launch this worker with pm2 itself.

[raitucarp]

That would be double allocate resources, since node 0.12.0 use maxsocket infinity, and my node process is nearly 100% that is 99%~

Also I can't strace it (it's return empty logs)

Thanks I will try your code

[raitucarp]

[update]

Even I can't restart my node.js if hang, can't do pm2 restart myapps manually too..
When I restart it, I have 2 pid of myapps
What I did was kill -9 firstpid, myapps up

[jshkurti]

Which version of pm2 are you using already ?

[raitucarp]

0.12.0

[didasy]

I need this too,

PM2 version is 0.14.3

It doesn't restart when my bot hangs

[raghavgarg1257]

Is there any update for this problem?

[zkochan]

It is an old issue, has this been implemented since 2015? Can pm2 automatically ping the service and restart it if it doesn't respond?

[Technoblazed]

For processes that were having this issue, I've had to move over to using systemd, seems to resolve the hanging issue and still no idea what causes this issue at its core.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mwarhead]

I have a hunging process in PM2, when i debug with node i got this:

          break in timers.js:220
           218 };
           219
          >220 function processTimers(now) {
           221   if (this[owner_symbol])
           222     return unrefdHandle(this[owner_symbol], now);

in my app I use setTimout function in websocket heartbeat
the app run for several hours then it hung

[Technoblazed]

@mwarhead create a new issue for it, then no point in necroing an issue.