Forwarder service won't start due to incorrect permissions

[adoom42]

Summary

When using splunkforwarder 8.1.3, configured to start as the root user, it fails to start. The splunk enable boot-start ... command configures the systemd service to start as splunk, even though the Puppet module is configured to start it as root.

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet: 6.19.1
• Distribution: RHEL7
• Module version: 8.0.0

How to reproduce (e.g Puppet code you use)

Configure the module to run the forwarder as root. One way to do it is by adding this to Hiera:

splunk::forwarder::splunk_user: 'root'

Run Puppet on a host that doesn't have splunkforwarder installed yet.

The systemd file will be configured to run the service as the splunk user.

What are you seeing

Puppet runs this command to enable the forwarder:

/opt/splunkforwarder/bin/splunk enable boot-start  -systemd-managed 1 --accept-license --answer-yes --no-prompt

When running the command manually, it shows this output:

# /opt/splunkforwarder/bin/splunk enable boot-start  -systemd-managed 1 --accept-license --answer-yes --no-prompt

This appears to be your first time running this version of Splunk.

IMPORTANT: Because an admin password was not provided, the admin user
 will not be created. You will have to set up an admin username/password
 later using user-seed.conf.
Important: splunk will start under systemd as user: splunk
Systemd unit file installed at /etc/systemd/system/SplunkForwarder.service.
Configured as systemd managed service.

Note that by leaving out the -user parameter, it configures systemd to run the service as the splunk user. This is in contradiction to the Hiera setting where it was designated to run as root.

# grep -E '(User|Group)' /etc/systemd/system/SplunkForwarder.service
User=splunk
Group=splunk

The module correctly sets ownership on some files in /opt/splunkforwarder to root:root, but because the service is running as the splunk user, it cannot read those files and the service fails to start.

What behaviour did you expect instead

I expect the service to run as the root user, because that's what I set via splunk::forwarder::splunk_user.

Any additional information you'd like to impart

I suspect the issue is related to this code:

puppet-splunk/manifests/forwarder/service/nix.pp

Lines 22 to 26 in 14fef36

	if $splunk::params::supports_systemd and $splunk::forwarder::splunk_user == 'root' {
	$user_args = ''
	} else {
	$user_args = "-user ${splunk::forwarder::splunk_user}"
	}

Notice that if systemd is used and the user is root, it leaves out the -user parameter. I suspect this worked in the past because without the parameter, the "splunk enable" command used root by default, but in 8.x or some other version they changed it to default to the splunk user instead.

Also see #262 for history.

Anyway, I don't see any harm in always including the -user parameter, so the fix may be as easy as getting rid of that conditional.

[benohara]

Saw the same, had to set splunk::forwarder::splunk_user to splunk and the recursively chown /opt/splunkforwarder as var and etc not writeable to update configs or the pid file.

[davemcdonnell]

Any news on this one? It does look like a simple fix of removing the 'if' check for user root and just always passing the configured user to "splunk enable boot-start".

I manually fixed on my test system with:
/opt/splunk/bin/splunk disable boot-start
/opt/splunk/bin/splunk enable boot-start -user root -systemd-managed 1 --accept-license --answer-yes --no-prompt