Default NodeJS max heap limit to 250MB. Override by setting NODE_OPTIONS #5451

mrdavidlaing · 2015-11-20T12:36:07Z

Relates to #5170 - this limits NodeJS to using only 250MB of heap before the garbage collector kicks in - useful in low memory environments.

To override, set the NODE_OPTIONS environment variable, eg, to allow Kibana to use 512MB of heap, set

NODE_OPTIONS="--max-old-space-size=512" bin/kibana

elasticsearch-release · 2015-11-20T12:36:10Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'.

elasticsearch-release · 2015-11-20T12:38:03Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'.

mrdavidlaing · 2015-11-20T12:52:41Z

I've just signed the CLA - not sure how to kick off the check aout

spalger · 2015-11-23T15:39:18Z

@mrdavidlaing I'm unable to find your CLA signature. Would you mind submitting another?

spalger · 2015-11-23T15:45:25Z

Can you add this to bin/kibana.bat too please?

mrdavidlaing · 2015-11-24T22:46:57Z

@spalger - Here is the signed CLA from my company.
Corporate Contributor License Agreement - signed (1).pdf

spalger · 2015-11-24T22:48:10Z

Ah, business signatures are a little more complex. Thanks for that!

…NS - Windows support

spalger · 2015-11-25T17:16:00Z

Okay, thinking about this a second time it feels wrong to set the value to anything by default. I'm thinking we should just inject NODE_OPTIONS so that we can use node's (probably intelligent) default but let others pick a value that's better for them.

rashidkpc · 2015-11-30T17:26:16Z

+1, I think we should keep node's default, and document the fact it exists and how to lower it.

nabheet · 2015-11-30T20:44:28Z

+1

nabheet · 2015-12-07T16:40:49Z

Can someone please resolve and merge this pull request? I have been quite anxiously waiting for this fix. It is quite annoying that I have to start my kibana docker container everyday after the OOM Killer kills it.

How can I help?

simianhacker · 2015-12-07T21:21:44Z

bin/kibana

@@ -21,5 +21,6 @@ if [ ! -x "$NODE" ]; then
  exit 1
 fi

-exec "${NODE}" "${DIR}/src/cli" ${@}
+NODE_OPTIONS="${NODE_OPTIONS:=--max-old-space-size=250}"


Remove this line, we decided not to set a default.

w33ble · 2015-12-08T19:27:52Z

Feature implemented in #5598

mrdavidlaing · 2015-12-08T22:33:44Z

@w33ble - Thanks for following up and merging a similar fix. Sorry for the radio silence.

Bargs · 2015-12-08T22:45:11Z

This is great, I was just looking for a clean way to run node in debug mode when using our npm scripts and grunt tasks that execute bin/kibana. This fix works great for that use case as well.

splitice · 2015-12-09T10:09:22Z

If you have to optimize resources then node runs out of memory (FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory) if you limit memory to 250mb.

ageis · 2017-05-10T00:37:52Z

Did you guys really just ship a broken systemd service in your release?

May 09 17:32:12 logs systemd[1]: Started Elasticsearch.
root@logs:/usr/share/logstash/bin# systemctl status kibana
● kibana.service - Kibana
   Loaded: error (Reason: Bad message)
   Active: inactive (dead) since Tue 2017-05-09 12:11:45 PDT; 5h 23min ago
 Main PID: 3692 (code=killed, signal=TERM)

May 09 12:11:38 logs systemd[1]: [/etc/systemd/system/kibana.service:17] Missing '='.
May 09 12:11:45 logs systemd[1]: Stopping Kibana...
May 09 12:11:45 logs systemd[1]: Stopped Kibana.
May 09 12:11:45 logs systemd[1]: [/etc/systemd/system/kibana.service:17] Missing '='.

[Unit]
Description=Kibana

[Service]
Type=simple
User=kibana
Group=kibana
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/kibana
EnvironmentFile=-/etc/sysconfig/kibana
ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"
Restart=always
WorkingDirectory=/
Environment=NODE_ENV=production
> # Workaround to prevent OOM errors with kibana/node. Should be fixed in v4.4.
> # Details: https://github.com/elastic/kibana/issues/5170
Environment=NODE_OPTIONS="--max-old-space-size=200"

[Install]
WantedBy=multi-user.target

ageis · 2017-05-10T00:56:55Z

Also, not only is that an invalid comment, the option doesn't work anyhow....

May 09 17:54:58 logs kibana[26665]: throw err;
May 09 17:54:58 logs kibana[26665]: ^
May 09 17:54:58 logs kibana[26665]: Error: Cannot find module '/"--max-old-space-size=200"'
May 09 17:54:58 logs kibana[26665]: at Function.Module._resolveFilename (module.js:469:15)
May 09 17:54:58 logs kibana[26665]: at Function.Module._load (module.js:417:25)
May 09 17:54:58 logs kibana[26665]: at Module.runMain (module.js:604:10)
May 09 17:54:58 logs kibana[26665]: at run (bootstrap_node.js:393:7)
May 09 17:54:58 logs kibana[26665]: at startup (bootstrap_node.js:150:9)
May 09 17:54:58 logs kibana[26665]: at bootstrap_node.js:508:3

w33ble · 2017-05-10T17:09:56Z

@ageis could you open a new issue for the problems you're having with the service? It'll help us track a fix.

I'm guessing that's a problem in 5.4.0, but please also note the version you're having problems with in the new issue.

tylersmalley · 2017-05-10T17:45:59Z

@ageis, our kibana.service file does not include those Environment settings. Are you downloading from our repository?

ageis · 2017-05-10T17:50:08Z

@tylersmalley Yes. I had a previous installation of Kibana 4 (when it was in /opt instead of /usr/share so there's also a file laying around at /etc/systemd/system/kibana.service.dpkg-old:

[Unit]
Description=Kibana 4

[Service]
Type=simple
User=kibana
Environment=CONFIG_PATH=/opt/kibana/config/kibana.yml
Environment=NODE_ENV=production
# Workaround to prevent OOM errors with kibana/node. Should be fixed in v4.4.
# Details: https://github.com/elastic/kibana/issues/5170
Environment=NODE_OPTIONS="--max-old-space-size=200"
ExecStart=/opt/kibana/node/bin/node /opt/kibana/src/cli

[Install]
WantedBy=multi-user.target

I removed everything related to that installation a long time ago though and had been running cleanly on version 5 for quite a while.

Default NodeJS max heap limit to 250M. Override by setting NODE_OPTIONS

50fc3a0

spalger self-assigned this Nov 23, 2015

rashidkpc added review updates_needed v4.4.0 v4.3.1 labels Nov 24, 2015

Default NodeJS max heap limit to 250M. Override by setting NODE_OPTIO…

37d0aeb

…NS - Windows support

rashidkpc added the v4.2.2 label Nov 30, 2015

simianhacker reviewed Dec 7, 2015
View reviewed changes

This was referenced Dec 8, 2015

Lets talk about v8's memory settings #5595

Closed

Fix/pr/5451 #5598

Closed

w33ble closed this Dec 8, 2015

mrdavidlaing deleted the i5170_limit_node_max_heap branch December 8, 2015 22:33

mrdavidlaing mentioned this pull request Dec 8, 2015

Memory leak in server? #5170

Closed

epixa removed v4.2.2 v4.3.1 v4.4.0 labels Dec 15, 2015

sadovnikov mentioned this pull request Feb 2, 2016

Executors (kibana) shuts down due to unknown error mesos/kibana#13

Closed

deviantony mentioned this pull request Apr 28, 2016

Kibana container shutting down in less than a day deviantony/docker-elk#51

Closed

cjcenizal mentioned this pull request Jun 7, 2017

Add --max-old-space-size=8192 argument to bin/kibana script to avoid running out of memory during optimize step. #12232

Closed

dobesv mentioned this pull request Feb 13, 2018

Limit Kibana server memory to 512mb by default #6030

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Default NodeJS max heap limit to 250MB. Override by setting NODE_OPTIONS #5451

Default NodeJS max heap limit to 250MB. Override by setting NODE_OPTIONS #5451

mrdavidlaing commented Nov 20, 2015

elasticsearch-release commented Nov 20, 2015

elasticsearch-release commented Nov 20, 2015

mrdavidlaing commented Nov 20, 2015

spalger commented Nov 23, 2015

spalger commented Nov 23, 2015

mrdavidlaing commented Nov 24, 2015

spalger commented Nov 24, 2015

spalger commented Nov 25, 2015

rashidkpc commented Nov 30, 2015

nabheet commented Nov 30, 2015

nabheet commented Dec 7, 2015

simianhacker Dec 7, 2015

w33ble commented Dec 8, 2015

mrdavidlaing commented Dec 8, 2015

Bargs commented Dec 8, 2015

splitice commented Dec 9, 2015

ageis commented May 10, 2017

ageis commented May 10, 2017

w33ble commented May 10, 2017

tylersmalley commented May 10, 2017

ageis commented May 10, 2017

Default NodeJS max heap limit to 250MB. Override by setting NODE_OPTIONS #5451

Default NodeJS max heap limit to 250MB. Override by setting NODE_OPTIONS #5451

Conversation

mrdavidlaing commented Nov 20, 2015

elasticsearch-release commented Nov 20, 2015

elasticsearch-release commented Nov 20, 2015

mrdavidlaing commented Nov 20, 2015

spalger commented Nov 23, 2015

spalger commented Nov 23, 2015

mrdavidlaing commented Nov 24, 2015

spalger commented Nov 24, 2015

spalger commented Nov 25, 2015

rashidkpc commented Nov 30, 2015

nabheet commented Nov 30, 2015

nabheet commented Dec 7, 2015

simianhacker Dec 7, 2015

Choose a reason for hiding this comment

w33ble commented Dec 8, 2015

mrdavidlaing commented Dec 8, 2015

Bargs commented Dec 8, 2015

splitice commented Dec 9, 2015

ageis commented May 10, 2017

ageis commented May 10, 2017

w33ble commented May 10, 2017

tylersmalley commented May 10, 2017

ageis commented May 10, 2017