From d0384ef556fab1c8086c80d3ca37a732943278ca Mon Sep 17 00:00:00 2001 From: Jaime Soriano Pastor Date: Tue, 6 Nov 2018 13:11:42 +0100 Subject: [PATCH 1/6] Log by default to journald on systems with systemd --- CHANGELOG.asciidoc | 1 + dev-tools/packaging/templates/linux/systemd.unit.tmpl | 5 ++++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index d386e4abdc8..507e5bd1c26 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -14,6 +14,7 @@ https://github.com/elastic/beats/compare/v6.4.0...master[Check the HEAD diff] *Affecting all Beats* - Dissect syntax change, use * instead of ? when working with field reference. {issue}8054[8054] +- Add `-e` flag by default to systemd unit so logs are collected by journald, behaviour can be reverted by overriding BEAT_LOG_OPTS environment variable with an empty value {pull}[] *Auditbeat* diff --git a/dev-tools/packaging/templates/linux/systemd.unit.tmpl b/dev-tools/packaging/templates/linux/systemd.unit.tmpl index 5725ba3e2a3..2b91a480fee 100644 --- a/dev-tools/packaging/templates/linux/systemd.unit.tmpl +++ b/dev-tools/packaging/templates/linux/systemd.unit.tmpl @@ -5,7 +5,10 @@ Wants=network-online.target After=network-online.target [Service] -ExecStart=/usr/share/{{.BeatName}}/bin/{{.BeatName}} -c /etc/{{.BeatName}}/{{.BeatName}}.yml -path.home /usr/share/{{.BeatName}} -path.config /etc/{{.BeatName}} -path.data /var/lib/{{.BeatName}} -path.logs /var/log/{{.BeatName}} +Environment=BEAT_LOG_OPTS="-e" +Environment=BEAT_CONFIG_OPTS="-c /etc/{{.BeatName}}/{{.BeatName}}.yml" +Environment=BEAT_PATH_OPTS="-path.home /usr/share/{{.BeatName}} -path.config /etc/{{.BeatName}} -path.data /var/lib/{{.BeatName}} -path.logs /var/log/{{.BeatName}}" +ExecStart=/usr/share/{{.BeatName}}/bin/{{.BeatName}} $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS Restart=always [Install] From fa771b67ff4fd1111128e5e1603dfa8d42fd9f63 Mon Sep 17 00:00:00 2001 From: Jaime Soriano Pastor Date: Tue, 6 Nov 2018 18:11:22 +0100 Subject: [PATCH 2/6] Changelog updated --- CHANGELOG-developer.asciidoc | 1 + CHANGELOG.asciidoc | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG-developer.asciidoc b/CHANGELOG-developer.asciidoc index d90904eaac1..f3a5ca8277e 100644 --- a/CHANGELOG-developer.asciidoc +++ b/CHANGELOG-developer.asciidoc @@ -31,6 +31,7 @@ The list below covers the major changes between 6.3.0 and master only. used because the value is specified in magefile.go. {pull}7670[7670] - Outputs must implement String. {pull}6404[6404] - Renamed `-beat-name` CLI option used in `kibana_index_pattern.go` to `-beat` for consistency with other scripts in `dev-tools/cmd`. {pull}8615[8615] +- Systemd unit file template used on Linux packaging includes now environment variables to ease flag overriding. One of them includes the `-e` flag, making beats log to stderr by default on systemd uses. {pull}8942[8942] ==== Bugfixes diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 507e5bd1c26..7062c0354cf 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -14,7 +14,7 @@ https://github.com/elastic/beats/compare/v6.4.0...master[Check the HEAD diff] *Affecting all Beats* - Dissect syntax change, use * instead of ? when working with field reference. {issue}8054[8054] -- Add `-e` flag by default to systemd unit so logs are collected by journald, behaviour can be reverted by overriding BEAT_LOG_OPTS environment variable with an empty value {pull}[] +- On systems with systemd the Beats log is now written to journald by default rather than file, this behaviour can be reverted by overriding BEAT_LOG_OPTS with an empty value. {pull}8942[8942]. *Auditbeat* From 8eab223ea8504f74cbe0130edd0eed7e9ae20e88 Mon Sep 17 00:00:00 2001 From: Jaime Soriano Pastor Date: Tue, 6 Nov 2018 18:11:47 +0100 Subject: [PATCH 3/6] Add documentation for systemd --- auditbeat/docs/setting-up-running.asciidoc | 4 + filebeat/docs/setting-up-running.asciidoc | 4 + heartbeat/docs/setting-up-running.asciidoc | 4 + journalbeat/docs/setting-up-running.asciidoc | 4 + libbeat/docs/shared-systemd.asciidoc | 99 ++++++++++++++++++++ metricbeat/docs/setting-up-running.asciidoc | 4 + packetbeat/docs/setting-up-running.asciidoc | 4 + 7 files changed, 123 insertions(+) create mode 100644 libbeat/docs/shared-systemd.asciidoc diff --git a/auditbeat/docs/setting-up-running.asciidoc b/auditbeat/docs/setting-up-running.asciidoc index f76b2d13c31..eddc1c4e84c 100644 --- a/auditbeat/docs/setting-up-running.asciidoc +++ b/auditbeat/docs/setting-up-running.asciidoc @@ -22,6 +22,8 @@ This section includes additional information on how to set up and run * <> +* <> + //MAINTAINERS: If you add a new file to this section, make sure you update the bulleted list ^^ too. @@ -35,4 +37,6 @@ include::./running-on-docker.asciidoc[] include::./running-on-kubernetes.asciidoc[] +include::../../libbeat/docs/shared-systemd.asciidoc[] + include::../../libbeat/docs/shared-shutdown.asciidoc[] diff --git a/filebeat/docs/setting-up-running.asciidoc b/filebeat/docs/setting-up-running.asciidoc index f76b2d13c31..eddc1c4e84c 100644 --- a/filebeat/docs/setting-up-running.asciidoc +++ b/filebeat/docs/setting-up-running.asciidoc @@ -22,6 +22,8 @@ This section includes additional information on how to set up and run * <> +* <> + //MAINTAINERS: If you add a new file to this section, make sure you update the bulleted list ^^ too. @@ -35,4 +37,6 @@ include::./running-on-docker.asciidoc[] include::./running-on-kubernetes.asciidoc[] +include::../../libbeat/docs/shared-systemd.asciidoc[] + include::../../libbeat/docs/shared-shutdown.asciidoc[] diff --git a/heartbeat/docs/setting-up-running.asciidoc b/heartbeat/docs/setting-up-running.asciidoc index a9991ceffde..bacd7e60342 100644 --- a/heartbeat/docs/setting-up-running.asciidoc +++ b/heartbeat/docs/setting-up-running.asciidoc @@ -20,6 +20,8 @@ This section includes additional information on how to set up and run * <> +* <> + //MAINTAINERS: If you add a new file to this section, make sure you update the bulleted list ^^ too. include::../../libbeat/docs/shared-directory-layout.asciidoc[] @@ -30,4 +32,6 @@ include::../../libbeat/docs/command-reference.asciidoc[] include::./running-on-docker.asciidoc[] +include::../../libbeat/docs/shared-systemd.asciidoc[] + include::../../libbeat/docs/shared-shutdown.asciidoc[] diff --git a/journalbeat/docs/setting-up-running.asciidoc b/journalbeat/docs/setting-up-running.asciidoc index 8f7ccba3bb8..e7f2d51ecfa 100644 --- a/journalbeat/docs/setting-up-running.asciidoc +++ b/journalbeat/docs/setting-up-running.asciidoc @@ -26,6 +26,8 @@ This section includes additional information on how to set up and run //* <> +* <> + * <> @@ -41,4 +43,6 @@ include::../../libbeat/docs/command-reference.asciidoc[] //include::./running-on-kubernetes.asciidoc[] +include::../../libbeat/docs/shared-systemd.asciidoc[] + include::../../libbeat/docs/shared-shutdown.asciidoc[] diff --git a/libbeat/docs/shared-systemd.asciidoc b/libbeat/docs/shared-systemd.asciidoc new file mode 100644 index 00000000000..a0920fc3470 --- /dev/null +++ b/libbeat/docs/shared-systemd.asciidoc @@ -0,0 +1,99 @@ +[[running-with-systemd]] +=== { beatname_uc } and systemd + +DEB and RPM packages include a service unit for Linux systems with systemd. On +these systems { beatname_uc } can be managed with the usual systemd commands: + +==== Starting and stoping { beatname_uc} + +When using systemd, { beatname_uc } can be started or stopped with `systemctl`. + +["source", "sh", subs="attributes"] +------------------------------------------------ +systemctl start {beatname_lc} +------------------------------------------------ + +["source", "sh", subs="attributes"] +------------------------------------------------ +systemctl stop {beatname_lc} +------------------------------------------------ + +Service start on system boot can be enabled or disabled with `systemctl`, +{ beatname_uc } is enabled when installed. + +["source", "sh", subs="attributes"] +------------------------------------------------ +systemctl enable {beatname_lc} +------------------------------------------------ + +["source", "sh", subs="attributes"] +------------------------------------------------ +systemctl disable {beatname_lc} +------------------------------------------------ + + +==== { beatname_uc } status and logs + +Service status can be also obtained with `systemctl`. + +["source", "sh", subs="attributes"] +------------------------------------------------ +systemctl start {beatname_lc} +------------------------------------------------ + +Logs are stored by default in journald, they can be viewed with `journalctl`. + +["source", "sh", subs="attributes"] +------------------------------------------------ +journalctl -u {beatname_lc}.service +------------------------------------------------ + +NOTE: Unit file included in packages sets by default `-e` flag, that flag makes +{ beatname_uc } to log to stderr and disables other log outputs. Systemd stores +all output sent to stderr in journald. + +[float] +=== Customizing systemd unit for { beatname_uc } + +Systemd service unit file includes some environment variables that can be +overriden to change default options. +[cols="> +* <> + //MAINTAINERS: If you add a new file to this section, make sure you update the bulleted list ^^ too. include::../../libbeat/docs/shared-directory-layout.asciidoc[] @@ -34,4 +36,6 @@ include::./running-on-docker.asciidoc[] include::./running-on-kubernetes.asciidoc[] +include::../../libbeat/docs/shared-systemd.asciidoc[] + include::../../libbeat/docs/shared-shutdown.asciidoc[] diff --git a/packetbeat/docs/setting-up-running.asciidoc b/packetbeat/docs/setting-up-running.asciidoc index a9991ceffde..bacd7e60342 100644 --- a/packetbeat/docs/setting-up-running.asciidoc +++ b/packetbeat/docs/setting-up-running.asciidoc @@ -20,6 +20,8 @@ This section includes additional information on how to set up and run * <> +* <> + //MAINTAINERS: If you add a new file to this section, make sure you update the bulleted list ^^ too. include::../../libbeat/docs/shared-directory-layout.asciidoc[] @@ -30,4 +32,6 @@ include::../../libbeat/docs/command-reference.asciidoc[] include::./running-on-docker.asciidoc[] +include::../../libbeat/docs/shared-systemd.asciidoc[] + include::../../libbeat/docs/shared-shutdown.asciidoc[] From eb2d040575bf6589f4978dd333a873072e269c98 Mon Sep 17 00:00:00 2001 From: Jaime Soriano Pastor Date: Wed, 14 Nov 2018 16:41:08 +0100 Subject: [PATCH 4/6] Address comments from review --- CHANGELOG-developer.asciidoc | 2 +- libbeat/docs/shared-systemd.asciidoc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG-developer.asciidoc b/CHANGELOG-developer.asciidoc index 66dfbd2ebbc..77f3e1b3b98 100644 --- a/CHANGELOG-developer.asciidoc +++ b/CHANGELOG-developer.asciidoc @@ -31,7 +31,7 @@ The list below covers the major changes between 6.3.0 and master only. used because the value is specified in magefile.go. {pull}7670[7670] - Outputs must implement String. {pull}6404[6404] - Renamed `-beat-name` CLI option used in `kibana_index_pattern.go` to `-beat` for consistency with other scripts in `dev-tools/cmd`. {pull}8615[8615] -- Systemd unit file template used on Linux packaging includes now environment variables to ease flag overriding. One of them includes the `-e` flag, making beats log to stderr by default on systemd uses. {pull}8942[8942] +- Systemd unit file template used on Linux packaging now includes environment variables to ease flag overriding. One of them includes the `-e` flag, making beats log to stderr by default on systemd uses. {pull}8942[8942] ==== Bugfixes diff --git a/libbeat/docs/shared-systemd.asciidoc b/libbeat/docs/shared-systemd.asciidoc index a0920fc3470..46b9470ad3d 100644 --- a/libbeat/docs/shared-systemd.asciidoc +++ b/libbeat/docs/shared-systemd.asciidoc @@ -38,7 +38,7 @@ Service status can be also obtained with `systemctl`. ["source", "sh", subs="attributes"] ------------------------------------------------ -systemctl start {beatname_lc} +systemctl status {beatname_lc} ------------------------------------------------ Logs are stored by default in journald, they can be viewed with `journalctl`. From 0b32cdadd7b0e93c6e209d7d5dc407742fd99cc1 Mon Sep 17 00:00:00 2001 From: Jaime Soriano Pastor Date: Wed, 5 Dec 2018 14:25:26 +0100 Subject: [PATCH 5/6] Apply suggested changes to documentation --- CHANGELOG.asciidoc | 2 +- libbeat/docs/shared-systemd.asciidoc | 62 +++++++++++++++------------- 2 files changed, 34 insertions(+), 30 deletions(-) diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index be75965b2c0..0773b812d89 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -128,7 +128,7 @@ https://github.com/elastic/beats/compare/v6.5.0...v7.0.0-alpha1[View commits] *Affecting all Beats* - Dissect syntax change, use * instead of ? when working with field reference. {issue}8054[8054] -- On systems with systemd the Beats log is now written to journald by default rather than file, this behaviour can be reverted by overriding BEAT_LOG_OPTS with an empty value. {pull}8942[8942]. +- On systems with systemd, the Beats log is now written to journald by default rather than file. To revert this behaviour override BEAT_LOG_OPTS with an empty value. {pull}8942[8942]. - Removed dashboards and index patterns generation for Kibana 5. {pull}8927[8927] *Auditbeat* diff --git a/libbeat/docs/shared-systemd.asciidoc b/libbeat/docs/shared-systemd.asciidoc index 46b9470ad3d..19841187c1a 100644 --- a/libbeat/docs/shared-systemd.asciidoc +++ b/libbeat/docs/shared-systemd.asciidoc @@ -1,12 +1,13 @@ [[running-with-systemd]] -=== { beatname_uc } and systemd +=== {beatname_uc} and systemd -DEB and RPM packages include a service unit for Linux systems with systemd. On -these systems { beatname_uc } can be managed with the usual systemd commands: +The DEB and RPM packages include a service unit for Linux systems with +systemd. On these systems, you can manage {beatname_uc} by using the usual +systemd commands. -==== Starting and stoping { beatname_uc} +==== Start and stop {beatname_uc} -When using systemd, { beatname_uc } can be started or stopped with `systemctl`. +Use `systemctl` to start or stop {beatname_uc}: ["source", "sh", subs="attributes"] ------------------------------------------------ @@ -18,8 +19,8 @@ systemctl start {beatname_lc} systemctl stop {beatname_lc} ------------------------------------------------ -Service start on system boot can be enabled or disabled with `systemctl`, -{ beatname_uc } is enabled when installed. +By default, the {beatname_uc} service starts automatically when the system +boots. To enable or disable auto start use: ["source", "sh", subs="attributes"] ------------------------------------------------ @@ -32,42 +33,45 @@ systemctl disable {beatname_lc} ------------------------------------------------ -==== { beatname_uc } status and logs +==== {beatname_uc} status and logs -Service status can be also obtained with `systemctl`. +To get the service status, use `systemctl`: ["source", "sh", subs="attributes"] ------------------------------------------------ systemctl status {beatname_lc} ------------------------------------------------ -Logs are stored by default in journald, they can be viewed with `journalctl`. +Logs are stored by default in journald. To view the Logs, use `journalctl`: ["source", "sh", subs="attributes"] ------------------------------------------------ journalctl -u {beatname_lc}.service ------------------------------------------------ -NOTE: Unit file included in packages sets by default `-e` flag, that flag makes -{ beatname_uc } to log to stderr and disables other log outputs. Systemd stores -all output sent to stderr in journald. +NOTE: The unit file included in the packages sets the `-e` flag by default. +This flag makes {beatname_uc} log to stderr and disables other log outputs. +Systemd stores all output sent to stderr in journald. [float] -=== Customizing systemd unit for { beatname_uc } +=== Customize systemd unit for {beatname_uc} -Systemd service unit file includes some environment variables that can be -overriden to change default options. -[cols=" Date: Fri, 7 Dec 2018 22:32:29 +0100 Subject: [PATCH 6/6] Wrap commands with attributes with plus signs --- libbeat/docs/shared-systemd.asciidoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libbeat/docs/shared-systemd.asciidoc b/libbeat/docs/shared-systemd.asciidoc index 19841187c1a..7192ad71504 100644 --- a/libbeat/docs/shared-systemd.asciidoc +++ b/libbeat/docs/shared-systemd.asciidoc @@ -63,8 +63,8 @@ override to change the default options. |======================================= | Variable | Description | Default value | BEAT_LOG_OPTS | Log options | `-e` -| BEAT_CONFIG_OPTS | Flags for configuration file path | `-c /etc/{beatname_lc}/{beatname_lc}.yml` -| BEAT_PATH_OPTS | Other paths | -path.home /usr/share/{beatname_lc} -path.config /etc/{beatname_lc} -path.data /var/lib/{beatname_lc} -path.logs /var/log/{beatname_lc} +| BEAT_CONFIG_OPTS | Flags for configuration file path | +-c /etc/{beatname_lc}/{beatname_lc}.yml+ +| BEAT_PATH_OPTS | Other paths | +-path.home /usr/share/{beatname_lc} -path.config /etc/{beatname_lc} -path.data /var/lib/{beatname_lc} -path.logs /var/log/{beatname_lc}+ |======================================= To override these variables, create a drop-in unit file in the @@ -100,4 +100,4 @@ systemctl restart {beatname_lc} NOTE: It is recommended that you use a configuration management tool to include drop-in unit files. If you need to add a drop-in manually, use -`systemctl edit {beatname_lc}.service`. ++systemctl edit {beatname_lc}.service+.