[BUG] Opensearch 2.9.0 - Failed to initialize LogType config index and builtin log types

[crazymind1337]

Describe the bug
I am trying to install the new version of opensearch on Ubuntu 20.04. But the cluster state stays at yellow.

To Reproduce
Steps to reproduce the behavior:

1. Install opensearch 2.9.0 on Ubuntu 20.04
2. configure a basic setting:

cluster.name: cluster
http.port: 9200
network.host: <local-ip>
node.name: test-node-1
path.data: "/var/lib/opensearch"
path.logs: "/var/log/opensearch"
plugins.security.disabled: true
transport.host: <local-ip>
transport.tcp.port: 9300
cluster.initial_cluster_manager_nodes: test-node-1
discovery.seed_hosts: test-node-1

3. start opensearch service
4. check logs

Expected behavior
Cluster state becomes green.

 ~# curl --silent --connect-timeout 3 --header 'Host: stats' --write-out '%{http_code}' "http://<local-ip>:9200/_cluster/health?pretty=true"
{
  "cluster_name" : "cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 4,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 3,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 57.14285714285714
}

Plugins
Basic installation.

Screenshots
%

Host/Environment (please complete the following information):

• OS: Ubuntu 20.04 (updates installed)
• Version 2.9.0

Additional context
In the logs of opensearch I am able to discover the following informations:

[2023-08-02T11:07:51,126][ERROR][o.o.s.u.SecurityAnalyticsException] [test-node-1] Security Analytics error:
org.opensearch.ResourceAlreadyExistsException: index [.opensearch-sap-pre-packaged-rules-config/CWaJHMTdTlmZSg9zl5rPaA] already exists

[2023-08-02T11:07:51,149][WARN ][o.o.s.SecurityAnalyticsPlugin] [test-node-1] Failed initializing prepackaged rules
org.opensearch.securityanalytics.util.SecurityAnalyticsException: index [.opensearch-sap-pre-packaged-rules-config/CWaJHMTdTlmZSg9zl5rPaA] already exists

[2023-08-02T11:07:51,196][WARN ][o.o.s.SecurityAnalyticsPlugin] [test-node-1] Failed to initialize LogType config index and builtin log types

[2023-08-02T11:07:52,152][INFO ][o.o.c.r.a.AllocationService] [test-node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.opensearch-sap-log-types-config][0]]]).

Complete log can be found at: https://justpaste.it/c9c32

[dbwiddis]

  "number_of_data_nodes" : 1,
  "active_primary_shards" : 4,
  "active_shards" : 4,
  "unassigned_shards" : 3,

From the docs:

A yellow cluster status means the primary shards for all indexes are allocated to nodes in a cluster, but the replica shards for at least one index aren't.

You only have one node, but the replica shard needs to be on a different node.

[rauldpm]

I have replicated this behavior with the RPM package (fresh install), this does not happen in 2.8.0, I also get the same log errors/warnings

OpenSearch 2.9.0 fresh install, cluster status and shards

[root@centos7 vagrant]# yum install opensearch-2.9.0-linux-x64.rpm -y
Loaded plugins: fastestmirror
Examining opensearch-2.9.0-linux-x64.rpm: opensearch-2.9.0-1.x86_64
Marking opensearch-2.9.0-linux-x64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package opensearch.x86_64 0:2.9.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================================================================================
 Package                                                Arch                                               Version                                             Repository                                                               Size
=============================================================================================================================================================================================================================================
Installing:
 opensearch                                             x86_64                                             2.9.0-1                                             /opensearch-2.9.0-linux-x64                                             993 M

Transaction Summary
=============================================================================================================================================================================================================================================
Install  1 Package

Total size: 993 M
Installed size: 993 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : opensearch-2.9.0-1.x86_64                                                                                                                                                                                                 1/1 
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
  Verifying  : opensearch-2.9.0-1.x86_64                                                                                                                                                                                                 1/1 

Installed:
  opensearch.x86_64 0:2.9.0-1                                                                                                                                                                                                                

Complete!
[root@centos7 vagrant]# sudo systemctl start opensearch
[root@centos7 vagrant]# sudo systemctl status opensearch
● opensearch.service - OpenSearch
   Loaded: loaded (/usr/lib/systemd/system/opensearch.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-10-11 15:33:17 UTC; 15s ago
     Docs: https://opensearch.org/
 Main PID: 5633 (java)
   CGroup: /system.slice/opensearch.service
           └─5633 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Dj...

Oct 11 15:33:07 centos7 systemd[1]: Starting OpenSearch...
Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: A terminally deprecated method in java.lang.System has been called
Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.9.0.jar)
Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: System::setSecurityManager will be removed in a future release
Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: A terminally deprecated method in java.lang.System has been called
Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.9.0.jar)
Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Oct 11 15:33:09 centos7 systemd-entrypoint[5633]: WARNING: System::setSecurityManager will be removed in a future release
Oct 11 15:33:17 centos7 systemd[1]: Started OpenSearch.

[root@centos7 vagrant]# curl -k -u admin:admin https://localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "opensearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 5,
  "active_shards" : 5,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 3,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 62.5
}
[root@centos7 vagrant]# curl -k -u admin:admin https://localhost:9200/_cat/shards
.opensearch-observability                 0 p STARTED     0   208b 127.0.0.1 centos7
.plugins-ml-config                        0 p STARTED     1  3.8kb 127.0.0.1 centos7
.plugins-ml-config                        0 r UNASSIGNED                     
.opensearch-sap-pre-packaged-rules-config 0 p STARTED              127.0.0.1 centos7
.opensearch-sap-pre-packaged-rules-config 0 r UNASSIGNED                     
.opensearch-sap-log-types-config          0 p STARTED              127.0.0.1 centos7
.opensearch-sap-log-types-config          0 r UNASSIGNED                     
.opendistro_security                      0 p STARTED    10 74.8kb 127.0.0.1 centos7

Apparently, the unassigned shards are duplicated:

.plugins-ml-config                        0 r UNASSIGNED           
.opensearch-sap-pre-packaged-rules-config 0 r UNASSIGNED            
.opensearch-sap-log-types-config          0 r UNASSIGNED     

---

Note: This appears to be solved in 2.10.0

OpenSearch 2.10.0 fresh install

[root@centos7 vagrant]# yum localinstall opensearch-2.10.0-linux-x64.rpm -y
Loaded plugins: fastestmirror
Examining opensearch-2.10.0-linux-x64.rpm: opensearch-2.10.0-1.x86_64
Marking opensearch-2.10.0-linux-x64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package opensearch.x86_64 0:2.10.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================================================================================
 Package                                               Arch                                              Version                                               Repository                                                               Size
=============================================================================================================================================================================================================================================
Installing:
 opensearch                                            x86_64                                            2.10.0-1                                              /opensearch-2.10.0-linux-x64                                            1.0 G

Transaction Summary
=============================================================================================================================================================================================================================================
Install  1 Package

Total size: 1.0 G
Installed size: 1.0 G
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : opensearch-2.10.0-1.x86_64                                                                                                                                                                                                1/1 
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
### Upcoming breaking change in packaging
 In a future release of OpenSearch, we plan to change the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043
  Verifying  : opensearch-2.10.0-1.x86_64                                                                                                                                                                                                1/1 

Installed:
  opensearch.x86_64 0:2.10.0-1                                                                                                                                                                                                               

Complete!
[root@centos7 vagrant]# sudo systemctl start opensearch
[root@centos7 vagrant]# systemctl  status opensearch
● opensearch.service - OpenSearch
   Loaded: loaded (/usr/lib/systemd/system/opensearch.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-10-11 15:51:34 UTC; 3s ago
     Docs: https://opensearch.org/
 Main PID: 24907 (java)
   CGroup: /system.slice/opensearch.service
           └─24907 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -D...

Oct 11 15:51:25 centos7 systemd[1]: Starting OpenSearch...
Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: A terminally deprecated method in java.lang.System has been called
Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.10.0.jar)
Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: System::setSecurityManager will be removed in a future release
Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: A terminally deprecated method in java.lang.System has been called
Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.10.0.jar)
Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Oct 11 15:51:26 centos7 systemd-entrypoint[24907]: WARNING: System::setSecurityManager will be removed in a future release
Oct 11 15:51:34 centos7 systemd[1]: Started OpenSearch.

[root@centos7 vagrant]# curl -k -u admin:admin https://localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "opensearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 4,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
[root@centos7 vagrant]# curl -k -u admin:admin https://localhost:9200/_cat/shards
.opensearch-observability        0 p STARTED  0   208b 127.0.0.1 centos7
.opensearch-sap-log-types-config 0 p STARTED           127.0.0.1 centos7
.opendistro_security             0 p STARTED 10 75.4kb 127.0.0.1 centos7

[root@centos7 vagrant]# curl -k -u admin:admin https://localhost:9200/_cat/shards
.opensearch-observability        0 p STARTED  0   208b 127.0.0.1 centos7
.opensearch-sap-log-types-config 0 p STARTED           127.0.0.1 centos7
.opendistro_security             0 p STARTED 10 75.4kb 127.0.0.1 centos7

[techsk8]

indeed, the problem is solved on opensearch 2.10 and 2.11

[HoffmannTom]

The warning "Failed to initialize LogType config index and builtin log types" as mentioned in the issue title is still present in version 2.11
It seems that the creation of the index is always triggered and produces the warning, if index already exists.
The cluster status is green though.