Add distribution package formats & service definitions

[jordansissel]

This is work to solve #1811

This PR adds a new grunt task create_services. This task generates service definitions for launchd (osx), upstart (centos 6, ubuntu), systemd (fedora, centos 7), and sysv init (lsb 3.1-flavored, almost all other linuxes).

Additionally, the create_packages task was extended to create .rpm and .deb files for both 32bit and 64bit linux targets. It also adds a .pkg for OSX users.

Additionally, grunt build will run create_services before packaging and and rpm, deb, and osx packages all include service definitions for running Kibana. The OSX package ships with a launchd configuration and the Linux packages (rpm, deb) ship with a sysv init script.

The majority of the functionality is provided by fpm for packaging and by pleaserun for generating service definitions. These projects are implemented in Ruby (I think pleaserun requires Ruby 2.0.0 or newer).

Work still needing to be done:

• Create a kibana user during package installation
• Have the service definitions run as the kibana user
• Test it!
• Write logs to /var/log/kibana/

It is my intention that this solves nearly all the packaging needs for #1811.

I have littered the code with some TODO items I'd like guidance on, as well. I've never used the bluebird Promise library, so let me know if there's any style/correctness problems! :)

[jordansissel]

My plan is to squash this into a single commit once review has completed and before merging. If you don't want me to do that, let me know.

[jordansissel]

I attempted to rebase onto master, but something wild changed in how grunt tasks are defined, and I'm rather lost now.

[rashidkpc]

No worries, on the rebase stuff, but seems there's still some style issues. Can you jshint before pushing?

[jordansissel]

I've fixed all the jshint errors except these:

[L46:C30] W117: 'version' is not defined.
                       '-v', version, '-a', arch, join(distPath, name) + '/=/opt/kibana/',
[L49:C30] W117: 'version' is not defined.
                       '-v', version, '-a', arch, join(distPath, name) + '/=/opt/kibana/',
[L53:C30] W117: 'version' is not defined.
                       '-v', version, join(distPath, name) + '/=/opt/kibana/',

Code in 59d439b makes the createPackages funciton now only receive the computed artifact prefix name-version-platform, so the version isn't explicitly available within the createPackage function. Any thoughts on what the best approach is?

[jordansissel]

% grunt create_services create_packages
Running "create_services" task

Something about create_services is causing grunt to terminate silently after it runs ;(

[jordansissel]

I'm going to write an after-install script to create the Kibana user on package installation. Where should it go?

[jordansissel]

On second thought, I may just add user creation as a feature to fpm.

[rashidkpc]

Hah, I actually thought fpm already had user creation. That would be a solid feature

[jordansissel]

@rashidkpc +1, I'm going to add user creation to fpm and will update this PR once that's ready.

[rashidkpc]

Awesome, looking forward to it. I also referred #3334 to this ticket. It makes sense to have startup scripts outside of the packaged distribution and we should definitely share code.

[rashidkpc]

Hmm, rebase gone awry?

[ernetas]

#1811 is labeled for 4.2.0, this one for 4.1.0. Which is it?

[jordansissel]

@rashidkpc yeah, I think I had a rebase go very poorly. I'll work on it :P

[jordansissel]

I started over (I did some very bad git rebasing and ended up in Oz. There were witches and lollypop guildfolk. It was scary).

Still have to do the user creation, but at present (for users wishing to test and give feedback), the rpm and deb packages include sysv init scripts.

Some work left to do, but this is progress.

[rthorntn]

@jordansissel I haven't created any dashboards, it's the whole new integrated webserver thing that makes it complex for me as kibana 3 was setup with nginx

[jordansissel]

64bit packages are now available. Documentation will be published soon, but for now, here's how:

Debian-derived systems:

$ wget -qO - https://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -

$ echo "deb http://packages.elasticsearch.org/kibana/4.1/debian stable main" | sudo tee -a /etc/apt/sources.list

$ sudo apt-get update

$ sudo apt-get install kibana

On RPM-based systems:

$ rpm --import https://packages.elasticsearch.org/GPG-KEY-elasticsearch

Put the following in /etc/yum.repos.d/kibana.repo:

[kibana-4.1]
name=Kibana repository for 4.1.x packages
baseurl=http://packages.elasticsearch.org/kibana/4.1/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

Then run:

$ yum clean all

$ yum install kibana

Let me know if you have trouble.

[PSiAU]

I just tested the Debian package and it's working perfectly.
Thank you so so much for doing this Jordan. The effort is hugely appreciated!

[rthorntn]

Thanks @jordansissel

[jordansissel]

@PSiAU @rthorntn - Thank you both for testing! We'll announce this somewhere more appropriate soon :)

[Hinidu]

Hello @jordansissel! I've just tried to install provided debian package and found that kibana doesn't start because there is no /opt/kibana/config/kibana.yml after installation. Though as I can see you provide an explicit argument to fpm with that file. I have tried to reinstall the package and remove /opt/kibana after deinstallation but it didn't help. What am I doing wrong?
Another annoying thing is that /etc/init.d/kibana start says kibana started though it did not and subsequent /etc/init.d/kibana status says that kibana is not running.
Thank you for packaging kibana!
UPDATE
I'm almost sure that it is not relevant but I'm using this simple salt state for the installation:

kibana-repo:
  pkgrepo.managed:
    - humanname: Kibana Repository
    - name: deb http://packages.elastic.co/kibana/4.1/debian stable main
    - dist: stable
    - file: /etc/apt/sources.list.d/kibana.list
    - key_url: https://packages.elastic.co/GPG-KEY-elasticsearch

kibana:
  pkg.installed:
    - require_in:
      - service: kibana
  service.running:
    - enable: True

I'm using Vagrant with base image deb/wheezy-amd64.

[Hinidu]

I've just tried to use your instructions on the fresh virtual machine and found the way I've got reported behavior: if I remove kibana package, remove /opt/kibana/config manually and install kibana package again then the config file is not restored. Actually I wasn't aware of this behavior of apt-get. So the problem was solved by using apt-get purge kibana && apt-get install kibana. Sorry for the noise.

[sboulkour]

[edited post]

Tried the CentOS version (installed on a RHEL 7 box). Works so far.
Some questions :

• Is it possible to use systemd service files instead of init files for CentOS/RHEL 7 ?
• What is the path of config file that kibana wil use ? /etc/kibana.yml or /etc/kibana/kibana.yml ?
• Is there a plan to use a kibana user ? All files are owned by root.

[chandlermelton]

Tested on CentOS 6.6.

All worked well, but erasing the package left the directory structure in /opt behind. I'm not sure if this is intended.

Thanks Jordan.

[MikeN123]

The Debian package seems to work fine, thanks! I would appreciate a systemd unit file though(mostly because of issues with init.d scripts and config management tools), should be easy to add since it is already available.

[woolfie]

Thanks @jordansissel for providing the rpm repo info. Will this information be added to the download information at https://www.elastic.co/downloads/kibana soon?

Also, the latest rpm in kibana4.1 repo is kibana-4.1.1-1.x86_64.rpm, whereas kibana 4.1.2 has been available from the main download page since Sep 8th. Can we expect the repo to be maintained for future versions?

[jordansissel]

Yes it will be added soon.

[woolfie]

@jordansissel Please can you define "soon"? 😉

[jnovack]

Agree with MikeN123, I would appreciate a systemd unit file, rather than an init.d script, for exactly the same reasons.

[PSiAU]

cough kibana 4.2 cough 😉

[jnovack]

4.2 requires es2.0, CentOS 7 has only approved es1.6.3 in yum. They don't make it easy.

[stdweird]

for 4.2 rpms, apply #5306 and the new command seems to be
grunt clean build _build:pleaseRun _build:osPackages
( i also removed all but linux-x64 from ./tasks/config/platforms.js, but there's probably a better way to select linux-x64 target)

[stdweird]

@jordansissel fpm fails to pick up the symlinks in build dir
the fpm command issues also doesn't handle systemd files, see #5307
i now do (version=4.2.1 for current 4.2 branch)

grunt clean build
rm -Rf build/kibana-$version-linux-x64
tar xzf target/kibana-$version-linux-x64.tar.gz -C build
grunt _build:pleaseRun _build:osPackages 

[chandlermelton]

Is the 4.2 package available yet? I cant find it. We should add a repo link on the website like logstash and elasticsearch have.

[roman-parkhunovskyi]

The same question for Kibana 4.3 and the real plans to implement repositories for goodness' sake.

[chandlermelton]

What is going on with this? Why are there no 4.2 or 4.3 packages available?

[kresss]

Any movement on this?

[shoggeh]

+bump

[enp]

+bump

[shoggeh]

(just to note some progress: Kibana 4.4.0 RPM was released yesterday)

[chandlermelton]

I wish they'd put something on https://www.elastic.co/downloads/kibana for
it like their other products.

On Wed, Feb 3, 2016 at 7:52 AM, Maciej Anczura [email protected]
wrote:

(just to note some progress: Kibana 4.4.0 RPM was released yesterday)

—
Reply to this email directly or view it on GitHub
#3212 (comment).

[enp]

Binary packages for Kibana are available via repositories as described in https://www.elastic.co/guide/en/kibana/current/setup.html#install, but it will be good to publish direct urls for RPM/DEB in docs and in https://www.elastic.co/downloads/kibana too.