New PID file issues

[alex-harvey-z3q]

I just tried the new module (using tip of master) and the RPM 4.4.2, and unfortunately a PID file issue has returned. The bug seems to have been introduced in 014c177. The problematic code is:

  # Generate the pidfile from here. If we instead made the forked process
  # generate it there will be a race condition between the pidfile writing
  # and a process possibly asking for status.
  echo $! > $pidfile

This line runs as root, and therefore $pidfile is owned by root. The application, running as kibana, can't write to it.

I may send in a fix tomorrow.

[yuval-adar]

I "fixed" it today by commenting that line out.

Kibana will write it's own pid file in any case so that line in the init template is 100% redundant (and harmful to begin with)

[alex-harvey-z3q]

Hi @yuval-adar

Is your issue before or after the merge of my patch #61 I also mentioned I am not convinced that the Init script should be writing a PID file, although I say that without knowing what the race-condition issue was that @lesaux was dealing with.

[yuval-adar]

Looks like i have the version that is older than that by some 10 days

In any case there should be no reason to write the pid file from init script as long as kibana writes it itself.

In my code that wraps your code I just make sure to create /var/run/kibana owned by my local kibana user and kibana writes the pid file there with no issues.

This might not be backwards compatible with kibana versions that don't write the pid file though (i've been using your module to install kibana for the best part of last 8 months)

[alex-harvey-z3q]

I propose we delete these lines from templates/kibana.init.erb:

  # Generate the pidfile from here. If we instead made the forked process
  # generate it there will be a race condition between the pidfile writing
  # and a process possibly asking for status.
  echo $! > $pidfile
  chown $user:$group $pidfile

@lesaux What do you think?

[lesaux]

Agreed, I am thinking of removing the whole init file template and just use the defaults provided by the package.

This probably means abandoning a few features, but keeping up with the fast evolving kibana package is just too much effort.

[lesaux]

I've removed many parts of the module. I am using the default init.d script provided by the package.

[yuval-adar]

It looks like your latest change effectivelly kills installation from .tar.gz package?

[alex-harvey-z3q]

The convention over at Elastic.co has been to publish a matrix of module versions mapping to product versions, i.e. if you're stuck with one of the early Kibana 4 releases for which there were no packages, you'd use module version X etc.

[yuval-adar]

can you give a pointer or two toward that matrix?

[yuval-adar]

Regarding the pid issue: kibana 4.5.0 RPM installs identical init.d file which creates the pid file owned by root so if kibana service doesn't run as root, you are back to square one.

My guess is that you should go back to managing the init templates.

[lesaux]

$user should be sourced from /etc/default/kibana. I'll be in front of a computer shortly to confirm, but what do you have in there. Are you sure the rpm overwrote the old files?

[yuval-adar]

rpm has it's own init script which is identical to the template you had. If you don't overwrite it you are back to the same issue

[lesaux]

If you don't specify

'pid.file' => '/var/run/kibana.pid',

the init script should work. process will run as kibana user but pid file will belong to root.

[yuval-adar]

Looks like it. I will have to test it in my environment. In any case I might fork your code and add all the goodies I need (systemd for starters since all I do is on RHEL7.2)

[alex-harvey-z3q]

I think we should just keep all the old hacks and support for archive installation until Elastic.co release an RPM that works.

[lesaux]

@alexharv074: I think I disagree, a puppet module shouldn't be correcting software bugs. If we keep on fixing kibana problems in the puppet module, we are just delaying them from being accounted for upstream.

@yuval-adar: systemd is the default on rhel7 since the last module release, although the rpm still installs an init file. Unfortunately kibana will log to /var/log/messages, and setting a pid file in kibana.yml still won't work.

[alex-harvey-z3q]

If the users of this module are blocked as a result of Elastic.co not fixing their RPM, they will just fork the module and implement their own workarounds. I don't see how that leads to more pressure on Elastic.co to fix their RPM. Do you know if there's an open issue about the PID file? I'll go and up vote it.

[alex-harvey-z3q]

@lesaux I understand the frustration of having to implement workarounds but this module is basically useless unless this PID file issue is fixed, and the previous behaviour is restored.

[lesaux]

"this module is basically useless"
I don't think that's the case, given the number of module downloads, and number of issues raised on the github in the last few weeks.
As I previously said I am not willing to solve software bugs with puppet anymore. I just don't have the time.

[alex-harvey-z3q]

Would you like an assistant maintainer here?

[chilcano]

I'm facing this issue too. Just disabled pid.file and logging.dest It worked.
My final configuration is:

  class { 'kibana4':
    ...
    config => {
      'server.port'  => $_port_kb_guest,
      'server.host'  => $elasticsearch_publish_host,
      'elasticsearch.url'  => "http://${_ipaddr}:${_port_es_guest}",
      'kibana.index' => '.kibana',
      'kibana.defaultAppId' => 'discover',
      #'pid.file' => '/var/run/kibana.pid',
      #'logging.dest' => '/var/log/kibana/kibana.log',    
    }
  }