[all beats] Remove BEAT_LOG_OPTS from the systemd unit file

[magnuslarsen]

OS Version: I've only tested DEB installations on Ubuntu 18.04, but I'd imagine it happens on all systemd enabled Linux systems
Beat Version: 7.0.0, 7.0.1

In a unit file for any beat BEAT_LOG_OPTS is defaulted to -e
This will make logging options in the configuration file obsolete/ignored.

Examples:
Filebeat systemd unit file (snippet):

[Service]
Environment="BEAT_LOG_OPTS=-e"

filebeat.yml (snippet):

logging:
  level: info
  to_files: true
  to_syslog: false
  json: true
  files:
    path: '/var/log/filebeat'
    name: 'filebeat'
    keepfiles: '7'
    permissions: '0644'

Expected result:
Logging to files in the /var/log/filebeat directory

Actual result:
Logging to /var/log/syslog

Why not keep all the configurations in the configuration file, instead of having it split up in 2 files: systemd drop in file for logging options, and a configuration file for all other options?

Blindly removing the environmental variable BEAT_LOG_OPTS results in the configuration file being used for everything, and for logging to happen in the expected result.
Is there ANY reason not to do this? It's both much simpler AND much cleaner for the user.

[eric-price]

Second this. I had to remove "Environment="BEAT_LOG_OPTS=-e"" from my service file to get logging to go where I specify in the config file. I'm using an rpm on Centos 7.

[jsoriano]

We wanted to make beats to log by default to stdout/stderr on systemd-based OSs, so logs are collected by journald, for coherence with other services, see #8942.
If having this setting in the unit file is being problematic, an alternative we could consider would be to set this in the default configuration file instead.

[magnuslarsen]

Wouldn't that leave duplicate logs? Making it send to stdout/stderr would log it to /var/log/syslog and journald. Journalbeat picks it up from journald, Filebeat's System module per default picks it up from /var/log/syslog

In #8942, i see this very issue was already discussed. The problem is, that if you don't have Journalbeat installed, your logs aren't going the place where you specify, leaving it in /var/log/syslog regardless of the configuration options, which means--in my case at least--that it didn't work as intended.

Having the default configuration file point to stdout/stderr would make a lot of sense. Then its clear for the user where it's going, and makes the change visible.

Alternatively, the beat could fail/log that the logging options in the configuration file (if any) are overridden by the systemd unit file. Just making it clearer would be ideal.

[watsonian]

I just ran into this problem. Spent about an hour trying to figure out why the logging config options were being ignored before realizing -e was overriding them. We have lots of customers who run just Filebeat to ship logs without any other beats and this behavior was very confusing. Particularly when you specify logging.to_syslog: false. The config file was clearly being read (-path.logs in the executing command was reflecting the config setting), but it was ignoring these other logging settings.

[inthecloud247]

This is incredibly confusing, and I ran into the same issue as @watsonian and similarly wasted a ton of time trying to debug. If there's a setting like this that overrides the other explicit config options, it would be helpful to at least update the docs to give people a heads-up.

[indreek]

Same here!

[indreek]

Another point is that if you collect metrics you have no good reason to keep duplicated metrics every x seconds in syslog + elastic. It's a waste of space. So i would like to keep my metrics in separate file and clear them in certain period.

[sanvila]

The problem with having -e in the systemd unit is that if you don't like it, you have to change it every time the package is upgraded, because /lib/systemd/system/filebeat.service is overwritten on upgrades.

In theory, it would also be possible to make /etc/systemd/system/multi-user.target.wants/filebeat.service a real file instead of a symlink, but then people would have to track changes to the original file, if any, to keep them in sync.

The packaging system already takes care of preserving changes in /etc/filebeat/filebeat.yml when the package is upgraded, so it would be really nice if that was the only file that had to be modified/customized.

[jsoriano]

The problem with having -e in the systemd unit is that if you don't like it, you have to change it every time the package is upgraded, because /lib/systemd/system/filebeat.service is overwritten on upgrades.

@sanvila systemd unit files shouldn't be directly modified, you can create a drop-in file to override values.

[sanvila]

Ok, didn't know, thanks a lot.

But even in such case, I still think it's slightly better to modify a single file than having to modify a file and create another one.

[gasperzupancic]

this is still an issue for all the beats in 7.4. i spent some time debugging why metricbeat flooded messages file and sucked up all my diskspace. Then i realized all the beats ignores the settings in yaml file.

[TheUniquePaulSmith]

I too wasted a ton of time on this expecting my files to be logged per config in filebeat.yml. I'm new to systemd

[archon810]

To save people some time in the future, I just updated to 7.7 and found that all logging which we previously directed to log files using

systemctl edit filebeat.service
[Service]
Environment="BEAT_LOG_OPTS="

stopped working (I'm using filebeat as an example here but it applies to all beats).

The solution is to remove the override as it's apparently no longer being used (even though docs like https://www.elastic.co/guide/en/beats/filebeat/7.7/running-with-systemd.html still actively recommend using systemctl edit) and to add something like this to the main /etc/filebeat/filebeat.yml:

#================================ Logging ======================================
# There are four options for the log output: file, stderr, syslog, eventlog
# The file output is the default.

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: info

# Enable debug output for selected components. To enable all selectors use ["*"]
# Other available selectors are "beat", "publish", "service"
# Multiple selectors can be chained.
#logging.selectors: [ ]

# Send all logging output to stderr. The default is false.
#logging.to_stderr: false

# Send all logging output to syslog. The default is false.
#logging.to_syslog: false

# Send all logging output to Windows Event Logs. The default is false.
#logging.to_eventlog: false

# If enabled, Filebeat periodically logs its internal metrics that have changed
# in the last period. For each metric that changed, the delta from the value at
# the beginning of the period is logged. Also, the total values for
# all non-zero internal metrics are logged on shutdown. The default is true.
#logging.metrics.enabled: true

# The period after which to log the internal metrics. The default is 30s.
#logging.metrics.period: 30s

# Logging to rotating files. Set logging.to_files to false to disable logging to
# files.
logging.to_files: true
logging.files:
  # Configure the path where the logs are written. The default is the logs directory
  # under the home path (the binary location).
  path: /var/log/filebeat

  # The name of the files where the logs are written to.
  name: filebeat

  # Configure log file size limit. If limit is reached, log file will be
  # automatically rotated
  rotateeverybytes: 10485760 # = 10MB

  # Number of rotated log files to keep. Oldest files will be deleted first.
  keepfiles: 7

  # The permissions mask to apply when rotating log files. The default value is 0600.
  # Must be a valid Unix-style file permissions mask expressed in octal notation.
  #permissions: 0600

  # Enable log file rotation on time intervals in addition to size-based rotation.
  # Intervals must be at least 1s. Values of 1m, 1h, 24h, 7*24h, 30*24h, and 365*24h
  # are boundary-aligned with minutes, hours, days, weeks, months, and years as
  # reported by the local system clock. All other intervals are calculated from the
  # Unix epoch. Defaults to disabled.
  #interval: 0

  # Rotate existing logs on startup rather than appending to the existing
  # file. Defaults to true.
  # rotateonstartup: true

# Set to true to log messages in JSON format.
#logging.json: false

Now my systemd-controlled beats are logging to the log file again.

[jf]

thanks, @archon810 . To save even more time:

• there is no need to remove or even edit BEAT_LOG_OPTS in the systemd unit file
• just refer to https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html to set up logging
• all you need (for logging to files; from the guide above):

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644