MX bean missing: G1 Concurrent GC error message

[rauldpm]

Description

The following log messages have been found in the v4.9.0-alpha3 AMI

/var/log/wazuh-indexer//var/log/wazuh-indexer/wazuh-cluster_server.json:{"type": "server", "timestamp": "2024-07-22T18:56:46,095Z", "level": "ERROR", "component": "o.o.p.c.j.GCMetrics", "cluster.name": "wazuh-cluster", "node.name": "node-1", "message": "MX bean missing: G1 Concurrent GC" }
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-22T18:56:46,095][ERROR][o.o.p.c.j.GCMetrics      ] [node-1] MX bean missing: G1 Concurrent GC

They have been found in the previous issues without a final issue reported

• https://github.com/wazuh/internal-devel-requests/issues/1261

We need to determine the origin of this error and what is causing it

Related

• Release 4.9.0 - Alpha 3 - Specific systems wazuh#24722 (comment)
• https://github.com/wazuh/internal-devel-requests/issues/1261

[f-galland]

I'm able to reproduce this issue by just installing the Indexer:

[root@amazonlinux2 ~]# grep -R 'MX bean missing: G1 Concurrent GC' /var/log/wazuh-indexer/
/var/log/wazuh-indexer/wazuh-cluster_server.json:{"type": "server", "timestamp": "2024-07-25T05:12:29,246-03:00", "level": "ERROR", "component": "o.o.p.c.j.GCMetrics", "cluster.name": "wazuh-cluster", "node.name": "node-1", "message": "MX bean missing: G1 Concurrent GC" }
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-25T05:12:29,246][ERROR][o.o.p.c.j.GCMetrics      ] [node-1] MX bean missing: G1 Concurrent GC

[f-galland]

The issue is also present upon installation of opensearch 2.13 (the version the indexer is currently using):

root@ubuntu:~# OPENSEARCH_INITIAL_ADMIN_PASSWORD=Secret.Password.1234 apt install ./opensearch-2.13.0-linux-x64.deb 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'opensearch' instead of './opensearch-2.13.0-linux-x64.deb'
opensearch is already the newest version (2.13.0).
0 upgraded, 0 newly installed, 0 to remove and 12 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up opensearch (2.13.0) ...
Running OpenSearch Post-Installation Script
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
### Breaking change in packaging since 2.13.0
 In 2.13.0 and later releases of OpenSearch, we have changed the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...

root@ubuntu:~# systemctl start opensearch.service

root@ubuntu:~# grep -R 'MX bean' /var/log/opensearch/
/var/log/opensearch/opensearch.log:[2024-07-25T12:24:29,960][ERROR][o.o.p.c.j.GCMetrics      ] [ubuntu] MX bean missing: G1 Concurrent GC
/var/log/opensearch/opensearch_server.json:{"type": "server", "timestamp": "2024-07-25T12:24:29,960Z", "level": "ERROR", "component": "o.o.p.c.j.GCMetrics", "cluster.name": "opensearch", "node.name": "ubuntu", "message": "MX bean missing: G1 Concurrent GC" }

[f-galland]

The issue is still present in version 2.15 of opensearch:

root@ubuntu:~# OPENSEARCH_INITIAL_ADMIN_PASSWORD=Secret.Password.1234 apt install ./opensearch-2.15.0-linux-x64.deb 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'opensearch' instead of './opensearch-2.15.0-linux-x64.deb'
The following NEW packages will be installed:
  opensearch
0 upgraded, 1 newly installed, 0 to remove and 12 not upgraded.
Need to get 0 B/893 MB of archives.
After this operation, 1171 MB of additional disk space will be used.
Get:1 /root/opensearch-2.15.0-linux-x64.deb opensearch amd64 2.15.0 [893 MB]
Selecting previously unselected package opensearch.
(Reading database ... 37700 files and directories currently installed.)
Preparing to unpack .../opensearch-2.15.0-linux-x64.deb ...
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.15.0) ...
Setting up opensearch (2.15.0) ...
Running OpenSearch Post-Installation Script
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
### Breaking change in packaging since 2.13.0
 In 2.13.0 and later releases of OpenSearch, we have changed the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
N: Download is performed unsandboxed as root as file '/root/opensearch-2.15.0-linux-x64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
root@ubuntu:~# systemctl start opensearch
root@ubuntu:~# grep -R 'MX bean' /var/log/opensearch/
/var/log/opensearch/opensearch.log:[2024-07-25T12:36:54,732][ERROR][o.o.p.c.j.GCMetrics      ] [ubuntu] MX bean missing: G1 Concurrent GC
/var/log/opensearch/opensearch_server.json:{"type": "server", "timestamp": "2024-07-25T12:36:54,732Z", "level": "ERROR", "component": "o.o.p.c.j.GCMetrics", "cluster.name": "opensearch", "node.name": "ubuntu", "message": "MX bean missing: G1 Concurrent GC" }

[f-galland]

I opened an issue to the opensearch team about this here.

Since the issue is present in their current packages, I'm not sure there is too much we can do on our own for now.

[gdiazlo]

Closing, as this would be solved eventually in upstream OpenSearch.