Kibana 8.0.0 not starting

[graphaelli]

Plain old ./scripts/compose.py start 8.0.0 fails to bring up an 8.0.0-SNAPSHOT cluster as of at least 2021-08-24. Kibana logs:

localtesting_8.0.0_kibana | {"type":"log","@timestamp":"2021-08-26T00:52:06+00:00","tags":["info","savedobjects-service"],"pid":1195,"message":"[.kibana] INIT -> CREATE_NEW_TARGET. took: 232ms."}
localtesting_8.0.0_kibana | {"type":"log","@timestamp":"2021-08-26T00:52:06+00:00","tags":["error","savedobjects-service"],"pid":1195,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system_user] with roles [kibana_system] on indices [.kibana_8.0.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 1 in 2 seconds."}

The same message is logged for .kibana_task_manager and .kibana_task_manager_8.0.0_001.

There appears to be some change to permissions around the kibana system indices. elastic/elasticsearch#76449 looks suspicious but is 2 weeks old and doesn't seem to change this exact behavior.

I attempted to force the permissions with the following to no avail:

diff --git b/docker/elasticsearch/roles.yml a/docker/elasticsearch/roles.yml
index 7eb127d..59228f5 100644
--- b/docker/elasticsearch/roles.yml
+++ a/docker/elasticsearch/roles.yml
@@ -23,6 +23,10 @@ heartbeat:
   indices:
     - names: ['heartbeat-*', 'shrink-heartbeat-*']
       privileges: ['all']
+kibana_system_extra:
+  indices:
+    - names: ['.kibana', '.kibana_*', '.kibana_task_manager', '.kibana_task_manager_8.0.0_001']
+      privileges: ['all']
 metricbeat:
   cluster: ['manage_index_templates', 'monitor', 'manage_ingest_pipelines', 'manage_ilm']
   indices:
diff --git b/docker/elasticsearch/users_roles a/docker/elasticsearch/users_roles
index 629fe73..1b6a793 100644
--- b/docker/elasticsearch/users_roles
+++ a/docker/elasticsearch/users_roles
@@ -7,6 +7,7 @@ filebeat:filebeat_user
 heartbeat:heartbeat_user
 ingest_admin:apm_server_user
 kibana_system:kibana_system_user
+kibana_system_extra:kibana_system_user
 kibana_user:apm_server_user,apm_user_ro,beats_user,filebeat_user,heartbeat_user,metricbeat_user,opbeans_user
 metricbeat:metricbeat_user
 opbeans:opbeans_user

[graphaelli]

Another data point:

curl -XPUT -H 'Content-Type: application/json'  http://kibana_system_user:changeme@localhost:9200/.kibana1
{"acknowledged":true,"shards_acknowledged":true,"index":".kibana1"}

curl -XPUT -H 'Content-Type: application/json'  http://kibana_system_user:changeme@localhost:9200/.kibana_1
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [kibana_system_user] with roles [kibana_system] on indices [.kibana_1], this action is granted by the index privileges [create_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [kibana_system_user] with roles [kibana_system] on indices [.kibana_1], this action is granted by the index privileges [create_index,manage,all]"},"status":403}

[graphaelli]

elastic/elasticsearch#74212 is another one that looks related

[graphaelli]

Fixed by elastic/elasticsearch@8759c85, snapshot images have not been built yet

[graphaelli]

Kibana starts now but won't permit logins due to what seems to be a related issue:

localtesting_8.0.0_kibana | {"type":"log","@timestamp":"2021-08-26T20:38:48+00:00","tags":["error","plugins","security","session"],"pid":1196,"message":"Failed to retrieve session index cleanup task: action [indices:data/read/get] is unauthorized for user [kibana_system_user] with roles [kibana_system] on indices [.kibana_task_manager_8.0.0,.kibana_task_manager_8.0.0_001], this action is granted by the index privileges [read,all]: security_exception: [security_exception] Reason: action [indices:data/read/get] is unauthorized for user [kibana_system_user] with roles [kibana_system] on indices [.kibana_task_manager_8.0.0,.kibana_task_manager_8.0.0_001], this action is granted by the index privileges [read,all]"}
localtesting_8.0.0_kibana | {"type":"log","@timestamp":"2021-08-26T20:38:48+00:00","tags":["error","plugins","security","session","index"],"pid":1196,"message":"Failed to check if session index exists: Response Error"}

[graphaelli]

and now it's working after a full rebuild, I think this last issue was local to me