Kibana container shutting down in less than a day

[rasmushjulskov]

When upping docker-compose on Azure, the Kibana container shuts down after ~1 day. Everything ells keeps running but that.

The containers logs is as follows:

{"type":"log","@timestamp":"2016-04-28T01:14:45+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2016-04-28T01:15:25+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2016-04-28T01:16:20+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2016-04-28T01:17:04+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2016-04-28T01:17:42+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2016-04-28T01:19:11+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2016-04-28T01:20:16+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}

[deviantony]

Hi, did you made any changes in the configuration?

What version of docker / docker-compose are you using?

[rasmushjulskov]

Docker version 1.10.3, build 20f81dd
docker-compose version: 1.5.1

With the original config. I've added a filter to Logstash config — that's all

[deviantony]

Weird, maybe the issue is related to your network?

I also found that post: https://discuss.elastic.co/t/kibana-flapping-between-red-and-green/36328/9

The issue here was the memory associated to the Node process. Seems that a way to solve this is to update the NODE_OPTIONS environment variable before starting the Kibana process.

NODE_OPTIONS="--max-old-space-size=250" bin/kibana

You could try to update the compose definition of the Kibana container like this:

kibana:
  build: kibana/
  environment:
    NODE_OPTIONS: "--max-old-space-size=250"
  volumes:
    - ./kibana/config/:/opt/kibana/config/
  ports:
    - "5601:5601"
  links:
    - elasticsearch

Of course, you may need to adjust the memory value according to your needs (the available memory on your system I guess).

See these issues for more info:
spujadas/elk-docker#17
elastic/kibana#5170
elastic/kibana#5451

Let me know if that solves your issue.

[deviantony]

@Rhjulskov any news?

[rasmushjulskov]

Sorry for the late response. I just restarted the compose file with the new changes. I'll response to the issue in a day or two.

Thanks a lot for your time :)

[deviantony]

@Rhjulskov can I assume that everything is fine? :)

[rasmushjulskov]

Yes it is. Thanks alot! :)

[bfleming-ciena]

I'm running a single node on a VM where I have the option to add a lot of RAM/CPU. Our elk stack crashes as well after awhile.

I just pulled the latest image and am trying it now. Is this fixed in the latest image?

tnx

[deviantony]

@stonefury please have a look at this comment: #51 (comment)

[bfleming-ciena]

Ah, ok thank you @deviantony. Appreciate it! Cheers.