Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Infinite loop when running indexer-security-init.sh #1885

Closed
JcabreraC opened this issue Mar 31, 2022 · 9 comments
Closed

Infinite loop when running indexer-security-init.sh #1885

JcabreraC opened this issue Mar 31, 2022 · 9 comments
Assignees
@rauldpm
Labels
component: indexer/deb component: indexer/rpm type/bug Bug issue
Milestone
Release 4.4.0

Comments

@JcabreraC
Copy link
Member

Wazuh version Install type Platform
4.3 Wazuh-Indexer Linux

Description

When trying to execute the indexer-security-init.sh script and getting an error (probably due to configuration), if the configuration is modified and the wazuh-indexer service is restarted again, it remains in an infinite loop executing indexer-security-init.sh all the time, leaving the terminal unusable.

Steps to reproduce

  • Edit the /etc/wazuh-indexer/opensearch.yml file with an incorrect configuration (e.g. not setting the network.host correctly)
  • Execute the script indexer-security-init.sh and receive the following error: 
    Security Admin v7
Will connect to 0.0.0.0:9300 ... done
Connected as CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US
OpenSearch Version: 1.2.4
OpenSearch Security Version: 1.2.4.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
* Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
* Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
* If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
* Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
* Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
* If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
* Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
* Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
* If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
* Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
* Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
* If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
  • Edit the /etc/wazuh-indexer/opensearch.yml file with an correct configuration (e.g. setting the network.host correctly)
  • Restard the service: systemctl restart wazuh-indexer

Result

The following message is displayed in an infinite loop, making it impossible to use the terminal:

   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{sv4jb2n9R86jOA7_GRKlzw}{0.0.0.0}{0.0.0.0:9300}]. This is not an error, will keep on trying ...
  Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{sv4jb2n9R86jOA7_GRKlzw}{0.0.0.0}{0.0.0.0:9300}]] (org.opensearch.client.transport.NoNodeAvailableException/org.opensearch.client.transport.NoNodeAvailableException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
@JcabreraC JcabreraC mentioned this issue Mar 31, 2022
Closed
1 task
@fedepacher fedepacher mentioned this issue Mar 31, 2022
Closed
@jmv74211 jmv74211 mentioned this issue Apr 1, 2022
Closed
2 tasks
@okynos
Copy link
Contributor

okynos commented Apr 5, 2022

After some tests over wazuh-indexer package in CentOS 7 and Ubuntu 20 systems. It seems impossible to reproduce and such problem is related to Security admin and indexer deep code. We can't fix this problem at this development stage.
I will close the issue until we can perform fixes in the indexer code.

@okynos okynos closed this as completed Apr 5, 2022
@Rebits Rebits mentioned this issue Apr 18, 2022
Closed
@zbalkan
Copy link
Contributor

zbalkan commented Jul 28, 2022

This occurs on RHEL9 too. Just follow the assistant according to the docs, and create an Indexer cluster with at least 2 nodes. It is easy to reproduce.

But when you use just one node, the cluster start succeeds. The issue seems like somewhere in Indexer cluster discovery.

@c-bordon c-bordon self-assigned this Aug 4, 2022
@c-bordon
Copy link
Member

c-bordon commented Aug 4, 2022

At the moment I was trying to reproduce the problem without success in Centos 7 and in Redhat 9, both in single node and in cluster, I will continue testing

@zbalkan
Copy link
Contributor

zbalkan commented Aug 4, 2022

Hi. In my case, it was my problem. I did skip firewall configuration. And that blocked cluster discovery. But I do not know for the other use cases.

@vikman90 vikman90 added this to the Release 4.4.0 milestone Sep 26, 2022
@alberpilot alberpilot self-assigned this Sep 27, 2022
@alberpilot alberpilot transferred this issue from wazuh/wazuh Oct 14, 2022
@zbalkan
Copy link
Contributor

zbalkan commented Oct 14, 2022

A related discussion: #1776

@rauldpm rauldpm self-assigned this Nov 7, 2022
@rauldpm
Copy link
Member

rauldpm commented Nov 7, 2022 •
edited by alberpilot
Loading

Update report

Research

  • All Slack threads where this problem was reported are old, from the approximate date when this issue was opened.
  • The issue does not clearly reflect a way to reproduce the problem, what was the status of the Wazuh indexer before editing the file? Was it working correctly? What was the configuration used?
  • The indexer-security-init.sh script has undergone numerous changes since it was renamed on March 21 (10 days before this issue was opened)
  • The author of the issue has been asked if he remembers more information about the reported problem because it has not been possible to reproduce the problem with different installations and systems, It is possible that this problem has been fixed in some development carried out since the issue was reported.

Testing

[2022-11-07T22:50:33,595][INFO ][o.o.c.c.JoinHelper	  ] [node-1] failed to join {node-2}{L3Ap1SlzQMSYkwuHt10oBw}{eeKtFeY9Q6-rykKKkNXctA}{192.100.0.5}{192.100.0.5:9300}{dimr}{shard_indexing_pressure_enabled=$
org.opensearch.transport.NodeNotConnectedException: [node-2][192.100.0.5:9300] Node not connected
        at org.opensearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:204) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.transport.TransportService.getConnection(TransportService.java:784) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:701) [opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:372) [opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:293) [opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.cluster.coordination.JoinHelper.lambda$new$3(JoinHelper.java:195) [opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:139) [opensearch-index-management-1.2.4.0.jar:1.2.4.0]
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:193) [opensearch-security-1.2.4.0.jar:1.2.4.0]
        at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:336) [opensearch-security-1.2.4.0.jar:1.2.4.0]
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:153) [opensearch-security-1.2.4.0.jar:1.2.4.0]
        at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:647) [opensearch-security-1.2.4.0.jar:1.2.4.0]
        at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:64) [opensearch-performance-analyzer-1.2.4.0.ja$
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:91) [opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:373) [opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) [opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) [opensearch-1.2.4.jar:1.2.4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
  • In no case has it been possible to reproduce the problem. The modifications made in the configuration file to force the error reported, have only produced errors indicating a parsing problem or that the node is not reachable, this has been tried with 4.3.0, 4.3.9, and 4.4.0 versions.

Next steps

  • Try to reproduce the problem with the v4.3.0 Wazuh installation assistant.
  • Collect more information on how to reproduce the error.

@rauldpm
Copy link
Member

rauldpm commented Nov 8, 2022 •
edited
Loading

Update report

Testing multi-node deployment with and without firewall

Multi node deployment - CentOS 7 - CentOS 9 Stream - no firewall
CentOS 7 - Install Wazuh indexer - node-1 
[root@centos7 packages]# yum localinstall wazuh-indexer-4.4.0-wp.1885.x86_64.rpm -y
Loaded plugins: fastestmirror
Examining wazuh-indexer-4.4.0-wp.1885.x86_64.rpm: wazuh-indexer-4.4.0-wp.1885.x86_64
Marking wazuh-indexer-4.4.0-wp.1885.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-indexer.x86_64 0:4.4.0-wp.1885 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================
Package             Arch         Version               Repository                                 Size
========================================================================================================
Installing:
wazuh-indexer       x86_64       4.4.0-wp.1885         /wazuh-indexer-4.4.0-wp.1885.x86_64       644 M

Transaction Summary
========================================================================================================
Install  1 Package

Total size: 644 M
Installed size: 644 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wazuh-indexer-4.4.0-wp.1885.x86_64                                                   1/1 
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore
  Verifying  : wazuh-indexer-4.4.0-wp.1885.x86_64                                                   1/1 

Installed:
  wazuh-indexer.x86_64 0:4.4.0-wp.1885                                                                  

Complete!
[root@centos7 packages]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
[root@centos7 packages]# systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
  Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
  Active: inactive (dead)
    Docs: man:firewalld(1)
[root@centos7 packages]# curl -sO https://packages.wazuh.com/4.3/wazuh-certs-tool.sh
[root@centos7 packages]# curl -sO https://packages.wazuh.com/4.3/config.yml
[root@centos7 packages]# nano config.yml 
[root@centos7 packages]# cat config.yml 
nodes:
  # Wazuh indexer nodes
  indexer:
    - name: node-1
      ip: 192.100.0.2
    - name: node-2
      ip: 192.100.0.5
[root@centos7 packages]# bash ./wazuh-certs-tool.sh -A
08/11/2022 17:29:57 INFO: Admin certificates created.
08/11/2022 17:29:57 INFO: Wazuh indexer certificates created.
[root@centos7 packages]# tar -cvf ./wazuh-certificates.tar -C ./wazuh-certificates/ .
./
./root-ca.key
./root-ca.pem
./admin-key.pem
./admin.pem
./node-1-key.pem
./node-1.pem
./node-2-key.pem
./node-2.pem
[root@centos7 packages]# nano /etc/wazuh-indexer/opensearch.yml 
[root@centos7 packages]# NODE_NAME=node-1
[root@centos7 packages]# mkdir /etc/wazuh-indexer/certs
[root@centos7 packages]# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem
[root@centos7 packages]# mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem
[root@centos7 packages]# mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem
[root@centos7 packages]# chmod 500 /etc/wazuh-indexer/certs
[root@centos7 packages]# chmod 400 /etc/wazuh-indexer/certs/*
[root@centos7 packages]# chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs
[root@centos7 packages]# systemctl daemon-reload
[root@centos7 packages]# systemctl enable wazuh-indexer
Created symlink from /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service to /usr/lib/systemd/system/wazuh-indexer.service.
[root@centos7 packages]# systemctl start wazuh-indexer
CentOS 9 Stream - Install Wazuh indexer - node-2 
[root@centos9stream packages]# yum localinstall wazuh-indexer-4.4.0-wp.1885.x86_64.rpm -y
CentOS Stream 9 - BaseOS                                                4.4 MB/s | 5.9 MB     00:01    
CentOS Stream 9 - AppStream                                             7.8 MB/s |  15 MB     00:01    
CentOS Stream 9 - Extras packages                                       4.7 kB/s | 8.8 kB     00:01    
Extra Packages for Enterprise Linux 9 - x86_64                          8.6 MB/s |  11 MB     00:01    
Extra Packages for Enterprise Linux 9 - Next - x86_64                   1.6 MB/s | 1.4 MB     00:00    
Dependencies resolved.
========================================================================================================
Package                   Architecture       Version                    Repository                Size
========================================================================================================
Installing:
wazuh-indexer             x86_64             4.4.0-wp.1885              @commandline             397 M

Transaction Summary
========================================================================================================
Install  1 Package

Total size: 397 M
Installed size: 644 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                1/1 
  Running scriptlet: wazuh-indexer-4.4.0-wp.1885.x86_64                                             1/1 
  Installing       : wazuh-indexer-4.4.0-wp.1885.x86_64                                             1/1 
  Running scriptlet: wazuh-indexer-4.4.0-wp.1885.x86_64                                             1/1 
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore

Couldn't write '64' to 'kernel/random/read_wakeup_threshold', ignoring: No such file or directory

  Verifying        : wazuh-indexer-4.4.0-wp.1885.x86_64                                             1/1 

Installed:
  wazuh-indexer-4.4.0-wp.1885.x86_64                                                                    

Complete!
[root@centos9stream packages]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
[root@centos9stream packages]# systemctl status firewalld.service 
○ firewalld.service - firewalld - dynamic firewall daemon
    Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
    Active: inactive (dead) since Tue 2022-11-08 17:22:17 UTC; 6min ago
  Duration: 46.979s
      Docs: man:firewalld(1)
  Main PID: 719 (code=exited, status=0/SUCCESS)
        CPU: 416ms

Nov 08 17:21:29 centos9s.localdomain systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 08 17:21:30 centos9s.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.
Nov 08 17:22:17 centos9stream systemd[1]: Stopping firewalld - dynamic firewall daemon...
Nov 08 17:22:17 centos9stream systemd[1]: firewalld.service: Deactivated successfully.
Nov 08 17:22:17 centos9stream systemd[1]: Stopped firewalld - dynamic firewall daemon.
[root@centos9stream packages]# nano /etc/wazuh-indexer/opensearch.yml 
[root@centos9stream packages]# NODE_NAME=node-2
[root@centos9stream packages]# mkdir /etc/wazuh-indexer/certs
tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem
mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem
mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem
chmod 500 /etc/wazuh-indexer/certs
chmod 400 /etc/wazuh-indexer/certs/*
chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs
[root@centos9stream packages]# systemctl daemon-reload
systemctl enable wazuh-indexer
systemctl start wazuh-indexer
Created symlink /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service → /usr/lib/systemd/system/wazuh-indexer.service.
[root@centos9stream packages]#
Initialize cluster - node-1 
[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 2
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
  SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
  SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
  SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
  SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
  SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
  SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
  SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
  SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
  SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
  SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success
CentOS 9 Stream - Firewall enabled - 9200 
[root@centos9stream packages]# iptables -I INPUT -p tcp --dport 9200 -j REJECT
[root@centos9stream packages]# systemctl restart wazuh-indexer
[root@centos9stream packages]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
     Active: active (running) since Tue 2022-11-08 17:43:30 UTC; 1min 16s ago
       Docs: https://documentation.wazuh.com
   Main PID: 5954 (java)
      Tasks: 40 (limit: 17526)
     Memory: 1.2G
        CPU: 18.142s
     CGroup: /system.slice/wazuh-indexer.service
             └─5954 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t>

Nov 08 17:43:15 centos9stream systemd[1]: Starting Wazuh-indexer...
Nov 08 17:43:16 centos9stream systemd-entrypoint[5954]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 17:43:16 centos9stream systemd-entrypoint[5954]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 17:43:16 centos9stream systemd-entrypoint[5954]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Nov 08 17:43:16 centos9stream systemd-entrypoint[5954]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 17:43:18 centos9stream systemd-entrypoint[5954]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 17:43:18 centos9stream systemd-entrypoint[5954]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 17:43:18 centos9stream systemd-entrypoint[5954]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Nov 08 17:43:18 centos9stream systemd-entrypoint[5954]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 17:43:30 centos9stream systemd[1]: Started Wazuh-indexer.
[root@centos9stream packages]#

[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 2
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success
[root@centos7 packages]#
CentOS 9 Stream - Firewall enabled - 9200 and 9300 
[root@centos9stream packages]# iptables -I INPUT -p tcp --dport 9300 -j REJECT
[root@centos9stream packages]# systemctl restart wazuh-indexer
[root@centos9stream packages]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
     Active: active (running) since Tue 2022-11-08 17:46:27 UTC; 40s ago
       Docs: https://documentation.wazuh.com
   Main PID: 6221 (java)
      Tasks: 36 (limit: 17526)
     Memory: 1.2G
        CPU: 16.563s
     CGroup: /system.slice/wazuh-indexer.service
             └─6221 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t>

Nov 08 17:45:44 centos9stream systemd[1]: Starting Wazuh-indexer...
Nov 08 17:45:45 centos9stream systemd-entrypoint[6221]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 17:45:45 centos9stream systemd-entrypoint[6221]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 17:45:45 centos9stream systemd-entrypoint[6221]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Nov 08 17:45:45 centos9stream systemd-entrypoint[6221]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 17:45:47 centos9stream systemd-entrypoint[6221]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 17:45:47 centos9stream systemd-entrypoint[6221]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 17:45:47 centos9stream systemd-entrypoint[6221]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Nov 08 17:45:47 centos9stream systemd-entrypoint[6221]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 17:46:27 centos9stream systemd[1]: Started Wazuh-indexer.
[root@centos9stream packages]#
  • Only one node detected
[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success
CentOS 9 Stream - Firewall enabled - 9200 and 9400 
[root@centos9stream packages]# iptables -I INPUT -p tcp --dport 9400 -j REJECT
[root@centos9stream packages]# systemctl restart wazuh-indexer
[root@centos9stream packages]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
     Active: active (running) since Tue 2022-11-08 17:48:34 UTC; 8s ago
       Docs: https://documentation.wazuh.com
   Main PID: 6484 (java)
      Tasks: 41 (limit: 17526)
     Memory: 1.2G
        CPU: 16.392s
     CGroup: /system.slice/wazuh-indexer.service
             └─6484 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t>

Nov 08 17:48:19 centos9stream systemd[1]: Starting Wazuh-indexer...
Nov 08 17:48:21 centos9stream systemd-entrypoint[6484]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 17:48:21 centos9stream systemd-entrypoint[6484]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 17:48:21 centos9stream systemd-entrypoint[6484]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Nov 08 17:48:21 centos9stream systemd-entrypoint[6484]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 17:48:22 centos9stream systemd-entrypoint[6484]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 17:48:22 centos9stream systemd-entrypoint[6484]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 17:48:22 centos9stream systemd-entrypoint[6484]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Nov 08 17:48:22 centos9stream systemd-entrypoint[6484]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 17:48:34 centos9stream systemd[1]: Started Wazuh-indexer.
[root@centos9stream packages]#

[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 2
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success
[root@centos7 packages]#
CentOS 7 - Firewall enabled - 9200 
[root@centos7 packages]# iptables -I INPUT -p tcp --dport 9200 -j REJECT
[root@centos7 packages]# systemctl restart wazuh-indexer
[root@centos7 packages]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-11-08 17:51:31 UTC; 13s ago
     Docs: https://documentation.wazuh.com
 Main PID: 5249 (java)
   CGroup: /system.slice/wazuh-indexer.service
           └─5249 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t...

Nov 08 17:51:17 centos7 systemd[1]: Starting Wazuh-indexer...
Nov 08 17:51:19 centos7 systemd-entrypoint[5249]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 17:51:19 centos7 systemd-entrypoint[5249]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 17:51:19 centos7 systemd-entrypoint[5249]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Nov 08 17:51:19 centos7 systemd-entrypoint[5249]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 17:51:20 centos7 systemd-entrypoint[5249]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 17:51:20 centos7 systemd-entrypoint[5249]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 17:51:20 centos7 systemd-entrypoint[5249]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Nov 08 17:51:20 centos7 systemd-entrypoint[5249]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 17:51:31 centos7 systemd[1]: Started Wazuh-indexer.
[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200
ERR: Seems there is no OpenSearch running on 192.100.0.2:9200 - Will exit
[root@centos7 packages]#
CentOS 7 - Firewall enabled - 9300 
[root@centos7 packages]# iptables -I INPUT -p tcp --dport 9300 -j REJECT
[root@centos7 packages]# systemctl restart wazuh-indexer
[root@centos7 packages]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-11-08 17:54:11 UTC; 10s ago
     Docs: https://documentation.wazuh.com
 Main PID: 5870 (java)
   CGroup: /system.slice/wazuh-indexer.service
           └─5870 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t...

Nov 08 17:53:57 centos7 systemd[1]: Starting Wazuh-indexer...
Nov 08 17:53:58 centos7 systemd-entrypoint[5870]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 17:53:58 centos7 systemd-entrypoint[5870]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 17:53:58 centos7 systemd-entrypoint[5870]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Nov 08 17:53:58 centos7 systemd-entrypoint[5870]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 17:54:00 centos7 systemd-entrypoint[5870]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 17:54:00 centos7 systemd-entrypoint[5870]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 17:54:00 centos7 systemd-entrypoint[5870]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Nov 08 17:54:00 centos7 systemd-entrypoint[5870]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 17:54:11 centos7 systemd[1]: Started Wazuh-indexer
[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success
[root@centos7 packages]#
CentOS 7 - Firewall enabled - 9400 
[root@centos7 packages]# iptables -I INPUT -p tcp --dport 9400 -j REJECT
[root@centos7 packages]# systemctl restart wazuh-indexer
[root@centos7 packages]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-11-08 17:57:36 UTC; 28s ago
     Docs: https://documentation.wazuh.com
 Main PID: 6912 (java)
   CGroup: /system.slice/wazuh-indexer.service
           └─6912 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t...

Nov 08 17:57:21 centos7 systemd[1]: Starting Wazuh-indexer...
Nov 08 17:57:22 centos7 systemd-entrypoint[6912]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 17:57:22 centos7 systemd-entrypoint[6912]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 17:57:22 centos7 systemd-entrypoint[6912]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Nov 08 17:57:22 centos7 systemd-entrypoint[6912]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 17:57:24 centos7 systemd-entrypoint[6912]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 17:57:24 centos7 systemd-entrypoint[6912]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 17:57:24 centos7 systemd-entrypoint[6912]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Nov 08 17:57:24 centos7 systemd-entrypoint[6912]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 17:57:36 centos7 systemd[1]: Started Wazuh-indexer.
[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 2
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success
[root@centos7 packages]#

Testing multi-node deployment modifying opensearch.yml configuration

CentOS 7 - Node 1
CentOS 7 - Wrong IP 
[root@centos7 packages]# nano /etc/wazuh-indexer/opensearch.yml 
[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.200:9200ERR: An unexpected NoRouteToHostException occured: No route to host
Trace:
java.net.NoRouteToHostException: No route to host
  at java.base/sun.nio.ch.Net.connect0(Native Method)
  at java.base/sun.nio.ch.Net.connect(Net.java:579)
  at java.base/sun.nio.ch.Net.connect(Net.java:568)
  at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
  at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
  at java.base/java.net.Socket.connect(Socket.java:633)
  at java.base/java.net.Socket.connect(Socket.java:583)
  at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:420)
  at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159)
[root@centos7 packages]# systemctl restart wazuh-indexer
Job for wazuh-indexer.service failed because a timeout was exceeded. See "systemctl status wazuh-indexer.service" and "journalctl -xe" for details.
[root@centos7 packages]# journalctl -r -u wazuh-indexer.service
Nov 08 18:02:54 centos7 systemd[1]: wazuh-indexer.service failed.
Nov 08 18:02:54 centos7 systemd[1]: Unit wazuh-indexer.service entered failed state.
Nov 08 18:02:54 centos7 systemd[1]: Failed to start Wazuh-indexer.
Nov 08 18:02:53 centos7 systemd[1]: wazuh-indexer.service start operation timed out. Terminating.
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at java.base/java.lang.Thread.run(Thread.java:833)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:260)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:141)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:337)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at java.base/sun.nio.ch.Net.bind(Net.java:555)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: at java.base/sun.nio.ch.Net.bind0(Native Method)
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: Likely root cause: java.net.BindException: Cannot assign requested address
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: BindTransportException[Failed to bind to 192.100.0.200:[9300-9400]]; nested: BindException[Cannot assign requested address];
Nov 08 18:01:51 centos7 systemd-entrypoint[7833]: uncaught exception in thread [main]
[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.200:9200ERR: An unexpected NoRouteToHostException occured: No route to host
Trace:
java.net.NoRouteToHostException: No route to host
  at java.base/sun.nio.ch.Net.connect0(Native Method)
  at java.base/sun.nio.ch.Net.connect(Net.java:579)
  at java.base/sun.nio.ch.Net.connect(Net.java:568)
  at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
  at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
  at java.base/java.net.Socket.connect(Socket.java:633)
  at java.base/java.net.Socket.connect(Socket.java:583)
  at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:420)
  at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159)
CentOS 7 - Invalid IP 
[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
Device "192.100.0." does not exist.
ERROR: network host not valid, check /etc/wazuh-indexer/opensearch.yml
[root@centos7 packages]# systemctl restart wazuh-indexer
Job for wazuh-indexer.service failed because a timeout was exceeded. See "systemctl status wazuh-indexer.service" and "journalctl -xe" for details.
[root@centos7 packages]# journalctl -r -u wazuh-indexer.service
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.cli.Command.main(Command.java:101)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.node.Node.start(Node.java:1111)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:77)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.transport.TransportService.doStart(TransportService.java:283)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:77)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:163)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.transport.TcpTransport.bindServer(TcpTransport.java:416)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.common.network.NetworkService.resolveBindHostAddresses(NetworkService.java:158)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.common.network.NetworkService.resolveInetAddresses(NetworkService.java:249)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.common.network.NetworkService.resolveInternal(NetworkService.java:299)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1305)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1377)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1519)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1529)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:933)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: Likely root cause: java.net.UnknownHostException: 192.100.0.: Name or service not known
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: BindTransportException[Failed to resolve host [192.100.0.]]; nested: UnknownHostException[192.100.0.: Name or service not known];
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: uncaught exception in thread [main]
[root@centos7 packages]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
  Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
  Active: failed (Result: timeout) since Tue 2022-11-08 18:06:32 UTC; 47s ago
    Docs: https://documentation.wazuh.com
  Process: 8170 ExecStart=/usr/share/wazuh-indexer/bin/systemd-entrypoint -p ${PID_DIR}/wazuh-indexer.pid --quiet (code=exited, status=143)
Main PID: 8170 (code=exited, status=143)

Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.cli.Command.main(Command.java:101)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Nov 08 18:05:30 centos7 systemd-entrypoint[8170]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
Nov 08 18:06:32 centos7 systemd[1]: wazuh-indexer.service start operation timed out. Terminating.
Nov 08 18:06:32 centos7 systemd[1]: Failed to start Wazuh-indexer.
Nov 08 18:06:32 centos7 systemd[1]: Unit wazuh-indexer.service entered failed state.
Nov 08 18:06:32 centos7 systemd[1]: wazuh-indexer.service failed.
CentOS 7 - Correct configuration but failed service (no reboot) 
[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200
ERR: Seems there is no OpenSearch running on 192.100.0.2:9200 - Will exit
CentOS 7 - Correct configuration and running service 
[root@centos7 packages]# systemctl restart wazuh-indexer
[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 2
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
  SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
  SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
  SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
  SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
  SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
  SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
  SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
  SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
  SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
  SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success
[root@centos7 packages]#
CentOS 7 - typo in opensearch config (plugins.security.ssl.http.spemtrustedcas_filepath:) 
[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
ERROR: this tool try to find admin.pem and admin-key.pem in . but it couldn't. In this case, you must run manually the Indexer security initializer by running the command: JAVA_HOME=/usr/share/wazuh-indexer/jdk runuser wazuh-indexer --shell=/bin/bash --command=/usr/share/wazuh-indexer/plugins/opensearch-security/tools/securityadmin.sh -cd /etc/wazuh-indexer/opensearch-security -cacert /path/to/root-ca.pem -cert /path/to/admin.pem -key /path/to/admin-key.pem -h 192.100.0.2 -p 9200 -icl -nhnv replacing /path/to/ by your certificates path.
CentOS 9 Stream - Node 2
Wrong IP 
[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 2
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
  SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
  SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
  SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
  SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
  SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
  SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
  SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
  SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
  SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
  SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success
[root@centos7 packages]#

[root@centos9stream packages]# systemctl restart wazuh-indexer
[root@centos9stream packages]# journalctl -r -u wazuh-indexer.service
Nov 08 18:25:58 centos9stream systemd-entrypoint[11645]: BindTransportException[Failed to bind to 192.100.0.50:[9300-9400]]; nested: BindException[Cannot assign requested address];
Nov 08 18:25:58 centos9stream systemd-entrypoint[11645]: uncaught exception in thread [main]

[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
  SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
  SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
  SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
  SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
  SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
  SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
  SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
  SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
  SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
  SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success
Invalid IP 
[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 2
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
  SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
  SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
  SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
  SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
  SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
  SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
  SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
  SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
  SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
  SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success
[root@centos7 packages]#

[root@centos9stream packages]# journalctl -r -u wazuh-indexer.service
Nov 08 18:39:38 centos9stream systemd-entrypoint[12101]: Likely root cause: java.net.UnknownHostException: 192.100.0.: Name or service not known
Nov 08 18:39:38 centos9stream systemd-entrypoint[12101]: BindTransportException[Failed to resolve host [192.100.0.]]; nested: UnknownHostException[192.100.0.: Name or service not known];

[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
  SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
  SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
  SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
  SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
  SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
  SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
  SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
  SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
  SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
  SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success

Testing custom Wazuh installation assistant multi-node deployment with 4.4.0 Wazuh indexer - CentOS 7 and Red Hat 9

Wazuh indexer 4.4.0 - With firewalld and iptables disabled 
[root@centos7 packages]# bash wazuh-install.sh --wazuh-indexer node-1 -i
08/11/2022 20:40:29 INFO: Starting Wazuh installation assistant. Wazuh version: 4.4.0
08/11/2022 20:40:29 INFO: Verbose logging redirected to /var/log/wazuh-install.log
08/11/2022 20:40:30 WARNING: Hardware and system checks ignored.
08/11/2022 20:40:33 INFO: Wazuh development repository added.
08/11/2022 20:40:33 INFO: --- Wazuh indexer ---
08/11/2022 20:40:33 INFO: Starting Wazuh indexer installation.
Loaded plugins: fastestmirror
Examining wazuh-indexer-4.4.0-wp.1885.x86_64.rpm: wazuh-indexer-4.4.0-wp.1885.x86_64
Marking wazuh-indexer-4.4.0-wp.1885.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-indexer.x86_64 0:4.4.0-wp.1885 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================================
 Package                  Arch              Version                   Repository                                      Size
===========================================================================================================================
Installing:
 wazuh-indexer            x86_64            4.4.0-wp.1885             /wazuh-indexer-4.4.0-wp.1885.x86_64            644 M

Transaction Summary
===========================================================================================================================
Install  1 Package

Total size: 644 M
Installed size: 644 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                      1/1 
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore
  Verifying  : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                      1/1 

Installed:
  wazuh-indexer.x86_64 0:4.4.0-wp.1885                                                                                     

Complete!
08/11/2022 20:41:02 INFO: Wazuh indexer installation finished.
08/11/2022 20:41:02 INFO: Wazuh indexer post-install configuration finished.
08/11/2022 20:41:02 INFO: Starting service wazuh-indexer.
08/11/2022 20:41:47 INFO: wazuh-indexer service started.
08/11/2022 20:41:47 INFO: Initializing Wazuh indexer cluster security settings.
08/11/2022 20:41:47 INFO: Wazuh indexer cluster initialized.
08/11/2022 20:41:47 INFO: Installation finished.

[root@centos9stream packages]# bash wazuh-install.sh --wazuh-indexer node-2 -i
08/11/2022 20:42:06 INFO: Starting Wazuh installation assistant. Wazuh version: 4.4.0
08/11/2022 20:42:06 INFO: Verbose logging redirected to /var/log/wazuh-install.log
08/11/2022 20:42:07 WARNING: Hardware and system checks ignored.
08/11/2022 20:42:09 INFO: Wazuh development repository added.
08/11/2022 20:42:09 INFO: --- Wazuh indexer ---
08/11/2022 20:42:09 INFO: Starting Wazuh indexer installation.
Extra Packages for Enterprise Linux 9 - x86_64                                              52 kB/s |  32 kB     00:00    
Extra Packages for Enterprise Linux 9 - Next - x86_64                                       54 kB/s |  33 kB     00:00    
EL-9 - Wazuh                                                                               3.2 MB/s |  12 MB     00:03    
Dependencies resolved.
===========================================================================================================================
 Package                        Architecture            Version                        Repository                     Size
===========================================================================================================================
Installing:
 wazuh-indexer                  x86_64                  4.4.0-wp.1885                  @commandline                  397 M

Transaction Summary
===========================================================================================================================
Install  1 Package

Total size: 397 M
Installed size: 644 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                   1/1 
  Running scriptlet: wazuh-indexer-4.4.0-wp.1885.x86_64                                                                1/1 
  Installing       : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                1/1 
  Running scriptlet: wazuh-indexer-4.4.0-wp.1885.x86_64                                                                1/1 
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore

Couldn't write '64' to 'kernel/random/read_wakeup_threshold', ignoring: No such file or directory

  Verifying        : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                1/1 

Installed:
  wazuh-indexer-4.4.0-wp.1885.x86_64                                                                                       

Complete!
08/11/2022 20:42:47 INFO: Wazuh indexer installation finished.
08/11/2022 20:42:47 INFO: Wazuh indexer post-install configuration finished.
08/11/2022 20:42:47 INFO: Starting service wazuh-indexer.
08/11/2022 20:43:05 INFO: wazuh-indexer service started.
08/11/2022 20:43:05 INFO: Initializing Wazuh indexer cluster security settings.
08/11/2022 20:43:05 INFO: Wazuh indexer cluster initialized.
08/11/2022 20:43:05 INFO: Installation finished.

[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-indexer-cluster
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 2
.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success
[root@centos7 packages]#
Wazuh indexer 4.4.0 - With firewalld disabled and iptables enabled REJECT 9300 ⚠️ 
[root@centos7 packages]# bash wazuh-install.sh --wazuh-indexer node-1 -i
08/11/2022 21:05:23 INFO: Starting Wazuh installation assistant. Wazuh version: 4.4.0
08/11/2022 21:05:23 INFO: Verbose logging redirected to /var/log/wazuh-install.log
08/11/2022 21:05:24 WARNING: Hardware and system checks ignored.
08/11/2022 21:05:27 INFO: Wazuh development repository added.
08/11/2022 21:05:27 INFO: --- Wazuh indexer ---
08/11/2022 21:05:27 INFO: Starting Wazuh indexer installation.
Loaded plugins: fastestmirror
Examining wazuh-indexer-4.4.0-wp.1885.x86_64.rpm: wazuh-indexer-4.4.0-wp.1885.x86_64
Marking wazuh-indexer-4.4.0-wp.1885.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-indexer.x86_64 0:4.4.0-wp.1885 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================================
 Package                  Arch              Version                   Repository                                      Size
===========================================================================================================================
Installing:
 wazuh-indexer            x86_64            4.4.0-wp.1885             /wazuh-indexer-4.4.0-wp.1885.x86_64            644 M

Transaction Summary
===========================================================================================================================
Install  1 Package

Total size: 644 M
Installed size: 644 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                      1/1 
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore
  Verifying  : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                      1/1 

Installed:
  wazuh-indexer.x86_64 0:4.4.0-wp.1885                                                                                     

Complete!
08/11/2022 21:05:56 INFO: Wazuh indexer installation finished.
08/11/2022 21:05:56 INFO: Wazuh indexer post-install configuration finished.
08/11/2022 21:05:56 INFO: Starting service wazuh-indexer.
08/11/2022 21:06:41 INFO: wazuh-indexer service started.
08/11/2022 21:06:41 INFO: Initializing Wazuh indexer cluster security settings.
08/11/2022 21:06:41 INFO: Wazuh indexer cluster initialized.
08/11/2022 21:06:41 INFO: Installation finished.
[root@centos7 packages]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
[root@centos7 packages]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-11-08 21:06:41 UTC; 47s ago
     Docs: https://documentation.wazuh.com
 Main PID: 3879 (java)
   CGroup: /system.slice/wazuh-indexer.service
           └─3879 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t...

Nov 08 21:05:56 centos7 systemd[1]: Starting Wazuh-indexer...
Nov 08 21:05:57 centos7 systemd-entrypoint[3879]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 21:05:57 centos7 systemd-entrypoint[3879]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 21:05:57 centos7 systemd-entrypoint[3879]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Nov 08 21:05:57 centos7 systemd-entrypoint[3879]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 21:05:59 centos7 systemd-entrypoint[3879]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 21:05:59 centos7 systemd-entrypoint[3879]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 21:05:59 centos7 systemd-entrypoint[3879]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Nov 08 21:05:59 centos7 systemd-entrypoint[3879]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 21:06:41 centos7 systemd[1]: Started Wazuh-indexer.
[root@centos7 packages]# systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@centos7 packages]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

[root@redhat9 packages]# bash wazuh-install.sh --wazuh-indexer node-2 -i
08/11/2022 21:05:51 INFO: Starting Wazuh installation assistant. Wazuh version: 4.4.0
08/11/2022 21:05:51 INFO: Verbose logging redirected to /var/log/wazuh-install.log
08/11/2022 21:05:53 WARNING: Hardware and system checks ignored.
08/11/2022 21:05:55 INFO: Wazuh development repository added.
08/11/2022 21:05:56 INFO: --- Wazuh indexer ---
08/11/2022 21:05:56 INFO: Starting Wazuh indexer installation.
EL-9 - Wazuh                                                                               3.3 MB/s |  12 MB     00:03    
Last metadata expiration check: 0:00:04 ago on Tue 08 Nov 2022 09:05:58 PM UTC.
Dependencies resolved.
===========================================================================================================================
 Package                        Architecture            Version                        Repository                     Size
===========================================================================================================================
Installing:
 wazuh-indexer                  x86_64                  4.4.0-wp.1885                  @commandline                  397 M

Transaction Summary
===========================================================================================================================
Install  1 Package

Total size: 397 M
Installed size: 644 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                   1/1 
  Running scriptlet: wazuh-indexer-4.4.0-wp.1885.x86_64                                                                1/1 
  Installing       : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                1/1 
  Running scriptlet: wazuh-indexer-4.4.0-wp.1885.x86_64                                                                1/1 
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore

Couldn't write '64' to 'kernel/random/read_wakeup_threshold', ignoring: No such file or directory

  Verifying        : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                1/1 
Installed products updated.

Installed:
  wazuh-indexer-4.4.0-wp.1885.x86_64                                                                                       

Complete!
08/11/2022 21:06:34 INFO: Wazuh indexer installation finished.
08/11/2022 21:06:34 INFO: Wazuh indexer post-install configuration finished.
08/11/2022 21:06:34 INFO: Starting service wazuh-indexer.
08/11/2022 21:06:50 INFO: wazuh-indexer service started.
08/11/2022 21:06:50 INFO: Initializing Wazuh indexer cluster security settings.
08/11/2022 21:06:51 INFO: Wazuh indexer cluster initialized.
08/11/2022 21:06:51 INFO: Installation finished.
[root@redhat9 packages]# cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="9.0 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.0"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.0 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/9/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.0
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.0"
[root@redhat9 packages]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
     Active: active (running) since Tue 2022-11-08 21:06:50 UTC; 1min 3s ago
       Docs: https://documentation.wazuh.com
   Main PID: 4907 (java)
      Tasks: 41 (limit: 17550)
     Memory: 1.7G
        CPU: 17.672s
     CGroup: /system.slice/wazuh-indexer.service
             └─4907 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t>

Nov 08 21:06:34 redhat9 systemd[1]: Starting Wazuh-indexer...
Nov 08 21:06:36 redhat9 systemd-entrypoint[4907]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 21:06:36 redhat9 systemd-entrypoint[4907]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 21:06:36 redhat9 systemd-entrypoint[4907]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Nov 08 21:06:36 redhat9 systemd-entrypoint[4907]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 21:06:37 redhat9 systemd-entrypoint[4907]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 08 21:06:37 redhat9 systemd-entrypoint[4907]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
Nov 08 21:06:37 redhat9 systemd-entrypoint[4907]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Nov 08 21:06:37 redhat9 systemd-entrypoint[4907]: WARNING: System::setSecurityManager will be removed in a future release
Nov 08 21:06:50 redhat9 systemd[1]: Started Wazuh-indexer.
[root@redhat9 packages]# systemctl status firewalld.service 
○ firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Tue 2022-11-08 20:57:55 UTC; 10min ago
       Docs: man:firewalld(1)
   Main PID: 704 (code=exited, status=0/SUCCESS)
        CPU: 389ms

Nov 08 20:57:05 rhel9.localdomain systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 08 20:57:06 rhel9.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.
Nov 08 20:57:55 redhat9 systemd[1]: Stopping firewalld - dynamic firewall daemon...
Nov 08 20:57:55 redhat9 systemd[1]: firewalld.service: Deactivated successfully.
Nov 08 20:57:55 redhat9 systemd[1]: Stopped firewalld - dynamic firewall daemon.
[root@redhat9 packages]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

[root@centos7 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-indexer-cluster
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 2
.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success

[root@redhat9 packages]# iptables -I INPUT -p tcp -m tcp --dport 9300 -j REJECT

[root@centos7 vagrant]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Wazuh indexer 4.3.9 - CentOS 7 - Red Hat 9 ⚠️ 
[root@centos7 packages]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
[root@centos7 packages]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
[root@centos7 packages]# systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-11-09 02:09:41 UTC; 2min 16s ago
     Docs: man:firewalld(1)
 Main PID: 3870 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─3870 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid

Nov 09 02:09:41 centos7 systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 09 02:09:41 centos7 systemd[1]: Started firewalld - dynamic firewall daemon.
Nov 09 02:09:41 centos7 firewalld[3870]: WARNING: AllowZoneDrifting is enabled. This is considere...now.
Hint: Some lines were ellipsized, use -l to show in full.
[root@centos7 packages]# bash wazuh-install.sh --wazuh-indexer node-1 -i
09/11/2022 02:12:41 INFO: Starting Wazuh installation assistant. Wazuh version: 4.3.9
09/11/2022 02:12:41 INFO: Verbose logging redirected to /var/log/wazuh-install.log
09/11/2022 02:12:43 WARNING: Hardware and system checks ignored.
09/11/2022 02:12:46 INFO: Wazuh repository added.
09/11/2022 02:12:46 INFO: --- Wazuh indexer ---
09/11/2022 02:12:46 INFO: Starting Wazuh indexer installation.
09/11/2022 02:13:21 INFO: Wazuh indexer installation finished.
09/11/2022 02:13:21 INFO: Wazuh indexer post-install configuration finished.
09/11/2022 02:13:21 INFO: Starting service wazuh-indexer.
09/11/2022 02:14:06 INFO: wazuh-indexer service started.
09/11/2022 02:14:06 INFO: Initializing Wazuh indexer cluster security settings.
09/11/2022 02:14:06 INFO: Wazuh indexer cluster initialized.
09/11/2022 02:14:06 INFO: Installation finished.
[root@centos7 packages]#

[root@redhat9 vagrant]# cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="9.0 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.0"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.0 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/9/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.0
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.0"
[root@redhat9 packages]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
[root@redhat9 packages]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-11-09 02:09:49 UTC; 2min 17s ago
       Docs: man:firewalld(1)
   Main PID: 14015 (firewalld)
      Tasks: 2 (limit: 18797)
     Memory: 29.5M
     CGroup: /system.slice/firewalld.service
             └─14015 /usr/bin/python3 -s /usr/sbin/firewalld --nofork --nopid

Nov 09 02:09:49 redhat9 systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 09 02:09:49 redhat9 systemd[1]: Started firewalld - dynamic firewall daemon.
[root@redhat9 packages]# bash wazuh-install.sh --wazuh-indexer node-2 -i
09/11/2022 02:12:42 INFO: Starting Wazuh installation assistant. Wazuh version: 4.3.9
09/11/2022 02:12:42 INFO: Verbose logging redirected to /var/log/wazuh-install.log
09/11/2022 02:12:44 WARNING: Hardware and system checks ignored.
09/11/2022 02:12:46 INFO: Wazuh repository added.
09/11/2022 02:12:47 INFO: --- Wazuh indexer ---
09/11/2022 02:12:47 INFO: Starting Wazuh indexer installation.
09/11/2022 02:13:25 INFO: Wazuh indexer installation finished.
09/11/2022 02:13:25 INFO: Wazuh indexer post-install configuration finished.
09/11/2022 02:13:25 INFO: Starting service wazuh-indexer.
09/11/2022 02:14:10 INFO: wazuh-indexer service started.
09/11/2022 02:14:10 INFO: Initializing Wazuh indexer cluster security settings.
09/11/2022 02:14:10 INFO: Wazuh indexer cluster initialized.
09/11/2022 02:14:10 INFO: Installation finished.

After five minutes the reported behavior appeared, but without an immediate loop, this message repeated every five minutes.

[root@redhat9 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
Security Admin v7
Will connect to 192.100.0.5:9300 ... done
Connected as CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US
OpenSearch Version: 1.2.4
OpenSearch Security Version: 1.2.4.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
  Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
  Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

If the firewalld service stops while the loop messages appear, the cluster initializes successfully

[root@redhat9 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
Security Admin v7
Will connect to 192.100.0.5:9300 ... done
Connected as CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US
OpenSearch Version: 1.2.4
OpenSearch Security Version: 1.2.4.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
  Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
  Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Clustername: wazuh-indexer-cluster
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 2
.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/
Will update '_doc/config' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '_doc/nodesdn' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '_doc/whitelist' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '_doc/audit' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/audit.yml 
   SUCC: Configuration for 'audit' created or updated
Done with success

Firewalld configuration

[root@redhat9 vagrant]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0 eth1
  sources: 
  services: cockpit dhcpv6-client ssh
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:
[root@redhat9 vagrant]# firewall-cmd --list-services
cockpit dhcpv6-client ssh

Conclusion

  • The reported error has not been possible to reproduce in most cases, due to all the fixes made in the cluster initialization file.
  • It has only been possible to reproduce the error in a multi-node deployment by enabling the firewalld service as default on both nodes (CentOS 7 and Red Hat 9), showing the messages reported every 5 minutes, but at the moment the firewall has been disabled, the cluster has started correctly and the messages have stopped appearing, in addition, control of the terminal has never been lost.
  • Another behavior related to the use of the firewall has been observed when establishing a rule to reject inputs to port 9300, showing timeout messages in a loop, which is expected since the blocked port is used for communication between nodes.
  • Thus, this error is related to the management of the firewall by the end user and due to the impossibility of reproducing it due to a configuration error, no fix is ​​required.
  • Regarding the firewall, there is an old PR to incorporate firewall checks in the Wazuh installation assistant, but this does not prevent a user from reproducing this error if he performs a manual deployment, so it would be necessary to consider the possibility of adding some reference in the documentation.

Next steps

  • Check if the problem is reproduced using the OpenSearch package.

@rauldpm
Copy link
Member

rauldpm commented Nov 9, 2022 •
edited
Loading

Update report

Testing

Wazuh indexer 4.4.0 - CentOS 7 - Red Hat 7 - Same procedure as Wazuh indexer 4.3.9 
[root@centos7 packages]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[root@centos7 packages]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
[root@centos7 packages]# systemctl start firewalld.service
[root@centos7 packages]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-11-09 17:42:58 UTC; 11s ago
     Docs: man:firewalld(1)
 Main PID: 3195 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─3195 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid

Nov 09 17:42:58 centos7 systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 09 17:42:58 centos7 systemd[1]: Started firewalld - dynamic firewall daemon.
Nov 09 17:42:58 centos7 firewalld[3195]: WARNING: AllowZoneDrifting is enabled. This is considered an insecure configuration option. It will be removed in a future release. Please consider disabling it now.
[root@centos7 packages]# bash wazuh-install.sh --wazuh-indexer node-1 -i
09/11/2022 17:43:28 INFO: Starting Wazuh installation assistant. Wazuh version: 4.3.9
09/11/2022 17:43:28 INFO: Verbose logging redirected to /var/log/wazuh-install.log
09/11/2022 17:43:29 WARNING: Hardware and system checks ignored.
09/11/2022 17:43:31 INFO: Wazuh repository added.
09/11/2022 17:43:31 INFO: --- Wazuh indexer ---
09/11/2022 17:43:31 INFO: Starting Wazuh indexer installation.
Loaded plugins: fastestmirror
Examining wazuh-indexer-4.4.0-wp.1885.x86_64.rpm: wazuh-indexer-4.4.0-wp.1885.x86_64
Marking wazuh-indexer-4.4.0-wp.1885.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-indexer.x86_64 0:4.4.0-wp.1885 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
 Package                                        Arch                                    Version                                         Repository                                                            Size
===================================================================================================================================================================================================================
Installing:
 wazuh-indexer                                  x86_64                                  4.4.0-wp.1885                                   /wazuh-indexer-4.4.0-wp.1885.x86_64                                  644 M

Transaction Summary
===================================================================================================================================================================================================================
Install  1 Package

Total size: 644 M
Installed size: 644 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                                                                                                              1/1 
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore
  Verifying  : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                                                                                                              1/1 

Installed:
  wazuh-indexer.x86_64 0:4.4.0-wp.1885                                                                                                                                                                             

Complete!
09/11/2022 17:44:03 INFO: Wazuh indexer installation finished.
09/11/2022 17:44:03 INFO: Wazuh indexer post-install configuration finished.
09/11/2022 17:44:03 INFO: Starting service wazuh-indexer.
09/11/2022 17:44:56 INFO: wazuh-indexer service started.
09/11/2022 17:44:56 INFO: Initializing Wazuh indexer cluster security settings.
09/11/2022 17:44:56 INFO: Wazuh indexer cluster initialized.
09/11/2022 17:44:56 INFO: Installation finished.
[root@centos7 packages]# 

[root@redhat9 packages]# cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="9.0 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.0"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.0 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/9/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.0
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.0"
[root@redhat9 packages]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
[root@redhat9 packages]# systemctl start firewalld
[root@redhat9 packages]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-11-09 17:43:05 UTC; 10s ago
       Docs: man:firewalld(1)
   Main PID: 13280 (firewalld)
      Tasks: 2 (limit: 18797)
     Memory: 26.5M
     CGroup: /system.slice/firewalld.service
             └─13280 /usr/bin/python3 -s /usr/sbin/firewalld --nofork --nopid

Nov 09 17:43:05 redhat9 systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 09 17:43:05 redhat9 systemd[1]: Started firewalld - dynamic firewall daemon.
[root@redhat9 packages]# bash wazuh-install.sh --wazuh-indexer node-2 -i
09/11/2022 17:43:34 INFO: Starting Wazuh installation assistant. Wazuh version: 4.3.9
09/11/2022 17:43:34 INFO: Verbose logging redirected to /var/log/wazuh-install.log
09/11/2022 17:43:36 WARNING: Hardware and system checks ignored.
09/11/2022 17:43:38 INFO: Wazuh repository added.
09/11/2022 17:43:39 INFO: --- Wazuh indexer ---
09/11/2022 17:43:39 INFO: Starting Wazuh indexer installation.
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

EL-9 - Wazuh                                                                                                                                                                        22 MB/s |  11 MB     00:00    
Last metadata expiration check: 0:00:02 ago on Wed 09 Nov 2022 05:43:39 PM UTC.
Dependencies resolved.
===================================================================================================================================================================================================================
 Package                                              Architecture                                  Version                                              Repository                                           Size
===================================================================================================================================================================================================================
Installing:
 wazuh-indexer                                        x86_64                                        4.4.0-wp.1885                                        @commandline                                        397 M

Transaction Summary
===================================================================================================================================================================================================================
Install  1 Package

Total size: 397 M
Installed size: 644 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                           1/1 
  Running scriptlet: wazuh-indexer-4.4.0-wp.1885.x86_64                                                                                                                                                        1/1 
  Installing       : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                                                                                                        1/1 
  Running scriptlet: wazuh-indexer-4.4.0-wp.1885.x86_64                                                                                                                                                        1/1 
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore

  Verifying        : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                                                                                                        1/1 
Installed products updated.

Installed:
  wazuh-indexer-4.4.0-wp.1885.x86_64                                                                                                                                                                               

Complete!
09/11/2022 17:44:22 INFO: Wazuh indexer installation finished.
09/11/2022 17:44:22 INFO: Wazuh indexer post-install configuration finished.
09/11/2022 17:44:22 INFO: Starting service wazuh-indexer.
09/11/2022 17:45:17 INFO: wazuh-indexer service started.
09/11/2022 17:45:17 INFO: Initializing Wazuh indexer cluster security settings.
09/11/2022 17:45:18 INFO: Wazuh indexer cluster initialized.
09/11/2022 17:45:18 INFO: Installation finished.

Messages appear every 30 seconds of execution

[root@redhat9 packages]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.5:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
^C
Session terminated, killing shell... ...killed.
[root@redhat9 packages]#

The same procedure has been carried out in an OpenSearch multi-node deployment, where the same result has been obtained.

OpenSearch 2.3.0 - CentOS 7 - Red Hat 7 - Same procedure as Wazuh indexer 4.3.9 and 4.4.0
CentOS 7 - Node 1 configuration 
# ======================== OpenSearch Configuration =========================
#
# NOTE: OpenSearch comes with reasonable defaults for most settings.
#	Before you set out to tweak and tune the configuration, make sure you
#	understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.opensearch.org
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#
# Path to log files:
#
path.logs: /var/log/opensearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# OpenSearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.100.0.2
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["192.100.0.2", "192.100.0.5"]
#
# Bootstrap the cluster using an initial set of cluster-manager-eligible nodes:
#
#cluster.initial_cluster_manager_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: node-1.pem
plugins.security.ssl.transport.pemkey_filepath: node-1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node-1.pem
plugins.security.ssl.http.pemkey_filepath: node-1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=admin,OU=Wazuh,O=Wazuh,L=California, C=US

plugins.security.nodes_dn:
  - CN=node-1,OU=Wazuh,O=Wazuh,L=California,C=US
  - CN=node-2,OU=Wazuh,O=Wazuh,L=California,C=US

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########
node.master: true
Red Hat 9 - Node 2 configuration 
# ======================== OpenSearch Configuration =========================
#
# NOTE: OpenSearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.opensearch.org
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-2
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#
# Path to log files:
#
path.logs: /var/log/opensearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# OpenSearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.100.0.5
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["192.100.0.2", "192.100.0.5"]
#
# Bootstrap the cluster using an initial set of cluster-manager-eligible nodes:
#
#cluster.initial_cluster_manager_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: node-2.pem
plugins.security.ssl.transport.pemkey_filepath: node-2-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node-2.pem
plugins.security.ssl.http.pemkey_filepath: node-2-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=admin,OU=Wazuh,O=Wazuh,L=California, C=US

plugins.security.nodes_dn:
  - CN=node-1,OU=Wazuh,O=Wazuh,L=California,C=US
  - CN=node-2,OU=Wazuh,O=Wazuh,L=California,C=US

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########
node.master: false

[root@centos7 opensearch]# sudo -u opensearch JAVA_HOME=/usr/share/opensearch/jdk/ OPENSEARCH_PATH_CONF=/etc/opensearch /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd /usr/share/opensearch/plugins/opensearch-security/securityconfig -icl -p 9200 -cd /usr/share/opensearch/plugins/opensearch-security/securityconfig -nhnv -cacert /etc/opensearch/root-ca.pem -cert /etc/opensearch/admin.pem -key /etc/opensearch/admin-key.pem -h 192.100.0.2
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
  * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
  * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
  * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
  * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Wazuh indexer 4.4.0 - CentOS 7 - Red Hat 7 - Cluster start without Wazuh indexer wrapper 
[root@centos7 packages]# systemctl start firewalld
[root@centos7 packages]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-11-09 20:15:10 UTC; 3s ago
     Docs: man:firewalld(1)
 Main PID: 3237 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─3237 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid

Nov 09 20:15:09 centos7 systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 09 20:15:10 centos7 systemd[1]: Started firewalld - dynamic firewall daemon.
Nov 09 20:15:10 centos7 firewalld[3237]: WARNING: AllowZoneDrifting is enabled. This is considered an insecur... now.
Hint: Some lines were ellipsized, use -l to show in full.
[root@centos7 packages]# bash wazuh-install.sh --wazuh-indexer node-1 -i
09/11/2022 20:16:12 INFO: Starting Wazuh installation assistant. Wazuh version: 4.3.9
09/11/2022 20:16:12 INFO: Verbose logging redirected to /var/log/wazuh-install.log
09/11/2022 20:16:13 WARNING: Hardware and system checks ignored.
09/11/2022 20:16:15 INFO: Wazuh repository added.
09/11/2022 20:16:15 INFO: --- Wazuh indexer ---
09/11/2022 20:16:15 INFO: Starting Wazuh indexer installation.
Loaded plugins: fastestmirror
Examining wazuh-indexer-4.4.0-wp.1885.x86_64.rpm: wazuh-indexer-4.4.0-wp.1885.x86_64
Marking wazuh-indexer-4.4.0-wp.1885.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-indexer.x86_64 0:4.4.0-wp.1885 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================================
 Package                Arch            Version                   Repository                                    Size
=====================================================================================================================
Installing:
 wazuh-indexer          x86_64          4.4.0-wp.1885             /wazuh-indexer-4.4.0-wp.1885.x86_64          644 M

Transaction Summary
=====================================================================================================================
Install  1 Package

Total size: 644 M
Installed size: 644 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                1/1 
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore
  Verifying  : wazuh-indexer-4.4.0-wp.1885.x86_64                                                                1/1 

Installed:
  wazuh-indexer.x86_64 0:4.4.0-wp.1885                                                                               

Complete!
09/11/2022 20:16:46 INFO: Wazuh indexer installation finished.
09/11/2022 20:16:46 INFO: Wazuh indexer post-install configuration finished.
09/11/2022 20:16:46 INFO: Starting service wazuh-indexer.
09/11/2022 20:17:35 INFO: wazuh-indexer service started.
09/11/2022 20:17:35 INFO: Initializing Wazuh indexer cluster security settings.
09/11/2022 20:17:35 INFO: Wazuh indexer cluster initialized.
09/11/2022 20:17:35 INFO: Installation finished.

[root@redhat9 packages]# systemctl start firewalld
[root@redhat9 packages]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-11-09 20:16:19 UTC; 7s ago
       Docs: man:firewalld(1)
   Main PID: 13255 (firewalld)
      Tasks: 2 (limit: 18797)
     Memory: 27.5M
     CGroup: /system.slice/firewalld.service
             └─13255 /usr/bin/python3 -s /usr/sbin/firewalld --nofork --nopid

Nov 09 20:16:19 redhat9 systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 09 20:16:19 redhat9 systemd[1]: Started firewalld - dynamic firewall daemon.
[root@redhat9 packages]# bash wazuh-install.sh --wazuh-indexer node-2 -i
09/11/2022 20:16:30 INFO: Starting Wazuh installation assistant. Wazuh version: 4.3.9
09/11/2022 20:16:30 INFO: Verbose logging redirected to /var/log/wazuh-install.log
09/11/2022 20:16:33 WARNING: Hardware and system checks ignored.
09/11/2022 20:16:35 INFO: Wazuh repository added.
09/11/2022 20:16:35 INFO: --- Wazuh indexer ---
09/11/2022 20:16:35 INFO: Starting Wazuh indexer installation.
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

EL-9 - Wazuh                                                                          21 MB/s |  11 MB     00:00    
Last metadata expiration check: 0:00:03 ago on Wed 09 Nov 2022 08:16:36 PM UTC.
Dependencies resolved.
=====================================================================================================================
 Package                      Architecture          Version                        Repository                   Size
=====================================================================================================================
Installing:
 wazuh-indexer                x86_64                4.4.0-wp.1885                  @commandline                397 M

Transaction Summary
=====================================================================================================================
Install  1 Package

Total size: 397 M
Installed size: 644 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                             1/1 
  Running scriptlet: wazuh-indexer-4.4.0-wp.1885.x86_64                                                          1/1 
  Installing       : wazuh-indexer-4.4.0-wp.1885.x86_64                                                          1/1 
  Running scriptlet: wazuh-indexer-4.4.0-wp.1885.x86_64                                                          1/1 
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore

  Verifying        : wazuh-indexer-4.4.0-wp.1885.x86_64                                                          1/1 
Installed products updated.

Installed:
  wazuh-indexer-4.4.0-wp.1885.x86_64                                                                                 

Complete!
09/11/2022 20:17:17 INFO: Wazuh indexer installation finished.
09/11/2022 20:17:17 INFO: Wazuh indexer post-install configuration finished.
09/11/2022 20:17:17 INFO: Starting service wazuh-indexer.
09/11/2022 20:18:04 INFO: wazuh-indexer service started.
09/11/2022 20:18:04 INFO: Initializing Wazuh indexer cluster security settings.
09/11/2022 20:18:04 INFO: Wazuh indexer cluster initialized.
09/11/2022 20:18:04 INFO: Installation finished.

[root@centos7 packages]# sudo -u wazuh-indexer JAVA_HOME=/usr/share/wazuh-indexer/jdk/ OPENSEARCH_PATH_CONF=/etc/wazuh-indexer /usr/share/wazuh-indexer/plugins/opensearch-security/tools/securityadmin.sh -cd /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig -icl -p 9200 -cd /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig -nhnv -cacert /etc/wazuh-indexer/certs/root-ca.pem -cert /etc/wazuh-indexer/certs/admin.pem -key /etc/wazuh-indexer/certs/admin-key.pem -h 192.100.0.2
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.100.0.2:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-4 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-4 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

(disabled firewalld service)

Clustername: wazuh-indexer-cluster
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 2
.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/config.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/config.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/roles.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/roles.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/roles_mapping.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/roles_mapping.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/internal_users.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/internal_users.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/action_groups.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/action_groups.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/tenants.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/tenants.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/nodes_dn.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/nodes_dn.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/whitelist.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/whitelist.yml (No such file or directory)
ERR: cannot upload configuration, see errors above
OpenSearch 1.2.0 - CentOS 7 - Red Hat 7 - Cluster start ⚠️ 
[root@centos7 packages]# tar -xvf opensearch-1.2.0-linux-x64.tar.gz
...
opensearch-1.2.0/plugins/opensearch-sql/slf4j-api-1.7.30.jar
opensearch-1.2.0/plugins/opensearch-sql/spring-aop-5.2.5.RELEASE.jar
opensearch-1.2.0/plugins/opensearch-sql/spring-beans-5.2.5.RELEASE.jar
opensearch-1.2.0/plugins/opensearch-sql/spring-context-5.2.5.RELEASE.jar
opensearch-1.2.0/plugins/opensearch-sql/spring-core-5.2.5.RELEASE.jar
opensearch-1.2.0/plugins/opensearch-sql/spring-expression-5.2.5.RELEASE.jar
opensearch-1.2.0/plugins/opensearch-sql/spring-jcl-5.2.5.RELEASE.jar
opensearch-1.2.0/plugins/opensearch-sql/sql-1.2.0.0.jar
opensearch-1.2.0/plugins/opensearch-sql/vavr-0.10.2.jar
opensearch-1.2.0/plugins/opensearch-sql/vavr-match-0.10.2.jar
[root@centos7 packages]# cd opensearch-1.2.0/
[root@centos7 opensearch-1.2.0]# ls
bin  config  jdk  lib  LICENSE.txt  logs  manifest.yml  modules  NOTICE.txt  opensearch-tar-install.sh  performance-analyzer-rca  plugins  README.md
[root@centos7 opensearch-1.2.0]# systemctl start firewalld
[root@centos7 opensearch-1.2.0]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-11-09 21:14:21 UTC; 1min 55s ago
     Docs: man:firewalld(1)
 Main PID: 3170 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─3170 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid

Nov 09 21:14:21 centos7 systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 09 21:14:21 centos7 systemd[1]: Started firewalld - dynamic firewall daemon.
Nov 09 21:14:21 centos7 firewalld[3170]: WARNING: AllowZoneDrifting is enabled. This is considered an insecure configuration option. It will be removed in a future release. Please consider disabling it now.
[root@centos7 opensearch-1.2.0]# bash opensearch-tar-install.sh 
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /vagrant/packages/opensearch-1.2.0
OpenSearch install type: .tar.gz on NAME="Red Hat Enterprise Linux"
OpenSearch config dir: /vagrant/packages/opensearch-1.2.0/config
OpenSearch config file: /vagrant/packages/opensearch-1.2.0/config/opensearch.yml
OpenSearch bin dir: /vagrant/packages/opensearch-1.2.0/bin
OpenSearch plugins dir: /vagrant/packages/opensearch-1.2.0/plugins
OpenSearch lib dir: /vagrant/packages/opensearch-1.2.0/lib
Detected OpenSearch Version: x-content-1.2.0
Detected OpenSearch Security Version: 1.2.0.0

### Success
### Execute this script now on all your nodes and then start all nodes
### OpenSearch Security will be automatically initialized.
### If you like to change the runtime configuration 
### change the files in ../securityconfig and execute: 
"/vagrant/packages/opensearch-1.2.0/plugins/opensearch-security/tools/securityadmin.sh" -cd "/vagrant/packages/opensearch-1.2.0/plugins/opensearch-security/securityconfig" -icl -key "/vagrant/packages/opensearch-1.2.0/config/kirk-key.pem" -cert "/vagrant/packages/opensearch-1.2.0/config/kirk.pem" -cacert "/vagrant/packages/opensearch-1.2.0/config/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/admin.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
done security
done plugins
k-NN libraries not found in LD_LIBRARY_PATH. Updating path to: :/vagrant/packages/opensearch-1.2.0/plugins/opensearch-knn/knnlib.
(create certificates)
[root@centos7 opensearch-1.2.0]# chmod 644 config/*.pem

[root@redhat9 opensearch-1.2.0]# tar -xvf opensearch-1.2.0-linux-x64.tar.gz 
...
opensearch-1.2.0/plugins/opensearch-sql/spring-jcl-5.2.5.RELEASE.jar
opensearch-1.2.0/plugins/opensearch-sql/sql-1.2.0.0.jar
opensearch-1.2.0/plugins/opensearch-sql/vavr-0.10.2.jar
opensearch-1.2.0/plugins/opensearch-sql/vavr-match-0.10.2.jar
[root@redhat9 packages]# cd opensearch-1.2.0
[root@redhat9 opensearch-1.2.0]# ls
bin  config  jdk  lib  LICENSE.txt  logs  manifest.yml  modules  NOTICE.txt  opensearch-tar-install.sh  performance-analyzer-rca  plugins  README.md
[root@redhat9 opensearch-1.2.0]# systemctl start firewalld
[root@redhat9 opensearch-1.2.0]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-11-09 21:17:25 UTC; 4s ago
       Docs: man:firewalld(1)
   Main PID: 13236 (firewalld)
      Tasks: 2 (limit: 18797)
     Memory: 27.5M
     CGroup: /system.slice/firewalld.service
             └─13236 /usr/bin/python3 -s /usr/sbin/firewalld --nofork --nopid

Nov 09 21:17:25 redhat9 systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 09 21:17:25 redhat9 systemd[1]: Started firewalld - dynamic firewall daemon.
[root@redhat9 opensearch-1.2.0]# bash opensearch-tar-install.sh 
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /vagrant/packages/opensearch-1.2.0
OpenSearch install type: .tar.gz on NAME="Red Hat Enterprise Linux"
OpenSearch config dir: /vagrant/packages/opensearch-1.2.0/config
OpenSearch config file: /vagrant/packages/opensearch-1.2.0/config/opensearch.yml
OpenSearch bin dir: /vagrant/packages/opensearch-1.2.0/bin
OpenSearch plugins dir: /vagrant/packages/opensearch-1.2.0/plugins
OpenSearch lib dir: /vagrant/packages/opensearch-1.2.0/lib
Detected OpenSearch Version: x-content-1.2.0
Detected OpenSearch Security Version: 1.2.0.0

### Success
### Execute this script now on all your nodes and then start all nodes
### OpenSearch Security will be automatically initialized.
### If you like to change the runtime configuration 
### change the files in ../securityconfig and execute: 
"/vagrant/packages/opensearch-1.2.0/plugins/opensearch-security/tools/securityadmin.sh" -cd "/vagrant/packages/opensearch-1.2.0/plugins/opensearch-security/securityconfig" -icl -key "/vagrant/packages/opensearch-1.2.0/config/kirk-key.pem" -cert "/vagrant/packages/opensearch-1.2.0/config/kirk.pem" -cacert "/vagrant/packages/opensearch-1.2.0/config/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/admin.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
done security
done plugins
k-NN libraries not found in LD_LIBRARY_PATH. Updating path to: :/vagrant/packages/opensearch-1.2.0/plugins/opensearch-knn/knnlib.
(create certificates)
[root@redhat9 opensearch-1.2.0]# chmod 644 config/*.pem


[root@centos7 opensearch-1.2.0]# sudo -u vagrant JAVA_HOME=$(pwd)/jdk OPENSEARCH_PATH_CONF=$(pwd)/config $(pwd)/plugins/opensearch-security/tools/securityadmin.sh -cd $(pwd)/plugins/opensearch-security/securityconfig -icl -p 9300 -cd $(pwd)/plugins/opensearch-security/securityconfig -nhnv -cacert $(pwd)/config/root-ca.pem -cert $(pwd)/config/admin.pem -key $(pwd)/config/admin-key.pem -h 192.100.0.2
Security Admin v7
Will connect to 192.100.0.2:9300 ... done
Connected as CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US
OpenSearch Version: 1.2.0
OpenSearch Security Version: 1.2.0.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
  Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

When trying to reproduce the MasterNotDiscoveredException[null] message obtained in Wazuh indexer 4.3.9, a different message is obtained.

The difference in commits between the 4.3 and 4.4 branches:

Conclusion

  • The reported behavior has not been reproduced in its entirety.
  • The reported behavior has been reproduced in a multi-node deployment of Wazuh indexer 4.3.9 by enabling the firewalld service on both nodes.
  • The reported behavior has not been possible to reproduce in Wazuh indexer 4.4.0 or OpenSearch 2.3.0, even without using the custom wrapper for securityadmin.sh.
  • The reported behavior has been reproduced in a multi-node deployment of OpenSearch 1.2.0 by enabling the firewalld service on both nodes.
  • Because the reported behavior has only been reproduced by enabling the firewalld service in both Wazuh indexer 4.3.9 and OpenSearch 1.2.0, and Wazuh indexer 4.4.0 and OpenSearch 2.3.0 do not show the same behavior, I do not consider it necessary to perform any action on the Wazuh indexer code.

@alberpilot
Copy link
Contributor

Thanks, @rauldpm for the clarification.
I proceed to close this issue as no modifications in the Wazuh indexer (packaging or security init script) are required.

@vikman90 vikman90 added the type/bug Bug issue label Jun 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rauldpm rauldpm

Labels
component: indexer/deb component: indexer/rpm type/bug Bug issue
Projects
No open projects
Release 4.3.0
Status: Done
Release 4.4.0
Status: Done
Milestone
Release 4.4.0
Development

No branches or pull requests
8 participants
@alberpilot @vikman90 @okynos @rauldpm @JcabreraC @zbalkan @davidjiglesias @c-bordon