Unable to write Kibana UUID file when using keystore

[LeeDr]

Kibana version: master

Elasticsearch version: master

Server OS version: CentOS 7

Browser version: N/A

Browser OS version: N/A

Original install method (e.g. download page, yum, from source, etc.): default dist rpms

Describe the bug: This could potentially be a couple of different issues.

1. the rpm install doesn't create a keystore file, but I think there's an issue to make it create it?
2. on an rpm install, you can't create the keystore as a non-root user (like vagrant user). And you can't log in as the kibana user. And you have to pass the --allow-root flag to create the keystore as the root user.
3. if you do create the keystore as the root user, @jbudz suggested this method;

++ sg kibana -c '/usr/share/kibana/bin/kibana-keystore --allow-root create'
Created Kibana keystore in /etc/kibana/kibana.keystore
++ sg kibana -c 'echo somepassword | /usr/share/kibana/bin/kibana-keystore --allow-root add --stdin elasticsearch.password'

4. the above steps do create the keystore file in /etc/kibana/ with the same ownership as the other files there;

[root@localhost vagrant]# ls -l /etc/kibana/
total 32
-rwxr-xr-x 1 root kibana 1220 Sep  4 18:48 ca.crt
-rwxr-xr-x 1 root kibana 1773 Sep  4 18:48 ca.key
-rwxr-xr-x 1 root kibana 1270 Sep  4 18:48 kibana.crt
-rwxr-xr-x 1 root kibana 1702 Sep  4 18:48 kibana.key
-rw-r--r-- 1 root kibana  174 Sep  4 18:49 kibana.keystore  <<<<<<<<<< this file
-rw-rw---- 1 root kibana 6008 Sep  4 18:48 kibana.yml
-rw-r--r-- 1 root kibana  216 Sep  4 00:18 node.options

5. and creating the keystore file that way also causes the creation of the data dir at /usr/share/kibana/data/
   but I think the permissions only allow kibana user to read, not write that dir;

drwxr-xr-x    3 root kibana      22 Sep  4 18:49 data

6. so now trying to start the kibana service fails with this error;
   "Error: Unable to write Kibana UUID file, please check the uuid.server configuration value in kibana.yml and ensure Kibana has sufficient permissions to read / write to this file. Error was: EACCES
7. I verified that opening it up wide open (chmod 777 /usr/share/kibana/data/) does allow kibana to start successfully.
8. when it starts in this scenario, that data dir gets a uuid file, and a optimize directory created in it.

So the simple bug is that the data dir is created with only owner:rw and group:r. The owner is root and group is kibana. And so the kibana user that the kibana service runs as can't write to the uuid file to that dir.

But what confuses me, is that when I don't create the keystore, no data directory is ever created.
And I only find an optimize directory at /usr/share/kibana/src/optimize.
And I don't find a uuid file anywhere. I think there must be a uuid file somewhere because /api/status shows

{
  name: "localhost.localdomain",
  uuid: "94ebd889-b6d2-4155-afc5-81f3c59948d6",
  version: {
    number: "8.0.0",

And restarting the server still shows the same uuid. So it must be persisted somewhere, but I haven't found it.

Steps to reproduce: (do all these as root user on CentOS 7)

1. install Elasticsearch master default dist master branch
2. yum install -y kibana-8.0.0-SNAPSHOT-x86_64.rpm
3. sg kibana -c '/usr/share/kibana/bin/kibana-keystore --allow-root create'
4. sg kibana -c 'echo somepassword | /usr/share/kibana/bin/kibana-keystore --allow-root add --stdin elasticsearch.password' (I'm thinking you really don't even need this step because creating the keystore creates the data dir)
5. service kibana start
6. on this OS, you probably need to check journalctl -u kibana to see the failure message

Expected behavior: the data dir should get created at some point with the permissions needed to allow the uuid file to be written there (if that's the right place for it)
and the uuid should always be in the same place regardless of whether you use a keystore file or not

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context: I'm not having this same problem on 7.x branch but I haven't dug into the details of the data dir creation or ownership or permissions.

/cc @jportner

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[LeeDr]

NOTE: This does NOT appear to be broken in 7.x, only master. In 7.x when I create the keystore in the same was as on master, it doesn't create the /usr/share/kibana/data dir.

[jbudz]

jon@debd1:~/kibana$ sudo dpkg -i target/kibana-8.0.0-SNAPSHOT-amd64.deb
Selecting previously unselected package kibana.
(Reading database ... 135296 files and directories currently installed.)
Preparing to unpack .../kibana-8.0.0-SNAPSHOT-amd64.deb ...
Unpacking kibana (8.0.0-SNAPSHOT) ...
Setting up kibana (8.0.0-SNAPSHOT) ...
Created Kibana keystore in /etc/kibana/kibana.keystore
jon@debd1:~/kibana$ sudo /usr/share/kibana/bin/kibana-keystore add elasticsearch.username
Enter value for elasticsearch.username: ***
jon@debd1:~/kibana$ sudo service kibana start
jon@debd1:~/kibana$ sudo ls -lah /var/lib/kibana/
total 16K
drwxr-s---  3 kibana kibana 4.0K Nov 23 17:14 .
drwxr-xr-x 44 root   root   4.0K Nov 23 17:11 ..
drwxrwxr-x  2 kibana kibana 4.0K Nov 23 16:00 optimize
-rw-r--r--  1 kibana kibana   36 Nov 23 17:14 uuid
jon@debd1:~/kibana$ sudo service kibana status
● kibana.service - Kibana
     Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: enabled)
     Active: active (running) since Mon 2020-11-23 17:14:10 UTC; 34s ago
       Docs: https://www.elastic.co
   Main PID: 46431 (node)
      Tasks: 11 (limit: 9333)
     Memory: 146.6M
     CGroup: /system.slice/kibana.service
             └─46431 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logg>

Nov 23 17:14:10 debd1 systemd[1]: Started Kibana.
jon@debd1:~/kibana$

This should be resolved now with the changes in #76465. Keystore creation at the old data path was setting up the data folder with the wrong permissions