Issue upgrading on FIPS enabled system

[nmaludy]

Wazuh version	Component	Install type	Install method	Platform
4.3.10-1	wazuh-manager	yum	Rocky Linux 8

When upgrading via yum, i get the following error:

$ sudo yum -y upgrade
Last metadata expiration check: 1:13:43 ago on Thu 17 Nov 2022 12:26:52 PM UTC.
Dependencies resolved.
================================================================================================================
 Package                         Architecture           Version                     Repository             Size
================================================================================================================
Upgrading:
 wazuh-dashboard                 x86_64                 4.3.10-1                    wazuh                 150 M
 wazuh-indexer                   x86_64                 4.3.10-1                    wazuh                 361 M
 wazuh-manager                   x86_64                 4.3.10-1                    wazuh                 115 M

Transaction Summary
================================================================================================================
Upgrade  3 Packages

Total download size: 626 M
Downloading Packages:
(1/3): wazuh-manager-4.3.10-1.x86_64.rpm                                         17 MB/s | 115 MB     00:06    
(2/3): wazuh-dashboard-4.3.10-1.x86_64.rpm                                       17 MB/s | 150 MB     00:08    
(3/3): wazuh-indexer-4.3.10-1.x86_64.rpm                                         26 MB/s | 361 MB     00:13    
----------------------------------------------------------------------------------------------------------------
Total                                                                            45 MB/s | 626 MB     00:13     
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Transaction test error:
  package wazuh-dashboard-4.3.10-1.x86_64 does not verify: no digest

The unique thing about my system is that it is FIPS enabled. In RHEL8 this means that MD5 and SHA1 hashes are disabled.

[DFolchA]

Hello @nmaludy

Thank you for notifying this, we have found this old issue with a similar problem that was fixed for wazuh-manager and wazuh-agent packages:
#367

We will try to fix this issue as soon as possible.

Meanwhile, you can use the --nodigestcheck option to avoid this problem.

[nmaludy]

@DFolchA the flag --nodigestcheck does not exist in yum/dnf on EL8

The workaround i found is:

sudo bash -c 'echo "%_pkgverify_level none" >/etc/rpm/macros.verify'
sudo yum -y --setopt=tsflags=nocrypto upgrade

Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1830347

[verdx]

The error replicates on install:

[root@rocky vagrant]# yum install -y wazuh-dashboard
Last metadata expiration check: 0:02:39 ago on Tue 22 Nov 2022 07:37:45 AM EST.
Dependencies resolved.
========================================================================================================
 Package                       Architecture         Version                   Repository           Size
========================================================================================================
Installing:
 wazuh-dashboard               x86_64               4.3.10-1                  wazuh               150 M

Transaction Summary
========================================================================================================
Install  1 Package

Total download size: 150 M
Installed size: 587 M
Downloading Packages:
wazuh-dashboard-4.3.10-1.x86_64.rpm                                     2.2 MB/s | 150 MB     01:09    
--------------------------------------------------------------------------------------------------------
Total                                                                   2.2 MB/s | 150 MB     01:09     
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Transaction test error:
  package wazuh-dashboard-4.3.10-1.x86_64 does not verify: no digest

As commented previously, this is the same error seen for the Wazuh Manager RPM package in #367. In summary, CentOS machines come with package rpm installed in version 4.11, which doesn't include the tools to add a SHA256 digest in the package, needed for FIPS.
The error is only seen in the Wazuh Dashboard, as both the Manager and Indexer have it solved on creation:

Package differences

[root@rocky vagrant]# rpm --version
RPM version 4.14.3[root@rocky vagrant]# rpm -K -v wazuh-dashboard-4.3.10-1.x86_64.rpm 
wazuh-dashboard-4.3.10-1.x86_64.rpm:
    Header V3 RSA/SHA256 Signature, key ID 29111145: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: NOTFOUND
    V3 RSA/SHA256 Signature, key ID 29111145: OK
    MD5 digest: NOTFOUND
[root@rocky vagrant]# rpm -K -v wazuh-manager-4.3.10-1.x86_64.rpm 
wazuh-manager-4.3.10-1.x86_64.rpm:
    Header V3 RSA/SHA256 Signature, key ID 29111145: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V3 RSA/SHA256 Signature, key ID 29111145: OK
[root@rocky vagrant]# rpm -K -v wazuh-indexer-4.3.10-1.x86_64.rpm 
wazuh-indexer-4.3.10-1.x86_64.rpm:
    Header V3 RSA/SHA256 Signature, key ID 29111145: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V3 RSA/SHA256 Signature, key ID 29111145: OK

The solutions used in the Manager and Indexer could be used here:

• In the case of the Manager, rpm 4.14 is built from sources before creating our package. This means the installation of more packages and more building time.
• In the case of the Wazuh Indexer, the solution is simpler, the rpm package is not created on CentOS but on Rocky Linux 8.5, which, as seen earlier, comes with rpm 4.14. Changing the machine in which the Wazuh Dashboard package is created may prove a faster and cleaner solution.

[verdx]

After just changing the first line of the Dockerfile to create the Wazuh Dashboard RPM package from

FROM centos:7

FROM rockylinux:8.5

gives some errors and warnings, as seen in the output:

logrpmdash_rocky.txt

1 - /sbin/ldconfig: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf: No such file or directory
As seen in the official bug report from redhat (https://bugzilla.redhat.com/show_bug.cgi?id=1540971), it can be ignored
2 - Errors and warnings coming from line /usr/lib/rpm/redhat/brp-mangle-shebangs . The error finishes with:

error: Bad exit status from /var/tmp/rpm-tmp.lA1y2y (%install)

and in the middle there are warnings similar to

*** WARNING: ./usr/share/wazuh-dashboard/src/core/server/core_app/assets/fonts/roboto_mono/LICENSE.txt is executable but has no shebang, removing executable bit

and errors similar to:

*** ERROR: ambiguous python shebang in /usr/share/wazuh-dashboard/node_modules/node-gyp/gyp/pylib/gyp/MSVSSettings_test.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.

This seems to have to do with the version of python used by the executable files present in the wazuh-dashboard repository, with a solution being changing those lines to using python3, as seen in this issue: sudar/Arduino-Makefile#616

[verdx]

The first thing to prove has been why the new problems only arise when creating the Wazuh Dashboard RPM package using Rocky Linux . If it is created using CentOS 7 the problem doesn't show, and in the case of the Wazuh Indexer, it is also created with a Rocky Linux but no error is shown.

Most errors come from directory node_modules/node-gyp/gyp/ and as it is said in the error, all of them have the first line:

#!/usr/bin/env python

or

#!/usr/bin/python

As seen in this article, that itself can bring problems to the RPM build.

A test in both CentOS 7 and Rocky Linux 8 proved it had something to do with it:

[vagrant@centos7 ~]$ /usr/bin/env python --version
Python 2.7.5
[vagrant@centos7 ~]$ 

[vagrant@rocky ~]$ /usr/bin/env python -v
/usr/bin/env: ‘python’: No such file or directory
[vagrant@rocky ~]$ 

1- A first solution seemed to be creating /usr/bin/python in Rocky Linux as a link to one of the present versions of python in the OS:

[vagrant@rocky 1934]$ ls /usr/bin/python3
python3     python3.6   python3.6m  python3.9   
[vagrant@rocky 1934]$ ls /usr/bin/python3

The following line was added to the Dockerfile, with the same result as in the previous builds:

RUN ln -s /usr/bin/python3.9 /usr/bin/python

logrpmdash_rocky9_ln.txt

A couple other tests changing the version of python gave the same result.

2- The next solution was to try and do what the error asks for and change the line to #!/usr/bin/env python3 and #!/usr/bin/python3, respectively.

• The base package was extracted:

tar xf wazuh-dashboard-base-4.3.10-1-linux-x64.tar.xz

• All occurrences of the line were changed:

for i in $(find -type f); do                                        ✔ 
sed -i 's|#!/usr/bin/env python|#!/usr/bin/env python3|g' $i
done

• And the package was compressed again:

tar cJf wazuh-dashboard-base-4.3.10-changes-linux-x64.tar.xz wazuh-dashboard-base

• The build command was called using the local base:

sudo bash build_package.sh --base local | tee log.txt   

For the first times it kept exiting with an error, as seen in the following log, even when no changes were done to the files, which seemed to be an error related to the fact of being a local file, or having been extracted and compressed.
logrpmdash_rocky_localnochanges.txt

Finally, with the help of @c-bordon it was cleared the error came from using revision changes as it needed to download other sources and they didn't exist with that revision.

After solving that, the package was correctly created, although all the WARNINGS are still there.

The log of the build is:
logrpmdash_rocky_local_allchanges.txt

The installation on Rocky Linux 8 with FIPS enabled of the package went correctly:

[vagrant@rocky 1934]$ sudo yum install wazuh-dashboard-4.3.10-1.x86_64.rpm 
Last metadata expiration check: 4:40:17 ago on Wed 23 Nov 2022 08:05:37 AM EST.
Dependencies resolved.
===================================================================================================================================================================================================================
 Package                                                Architecture                                  Version                                            Repository                                           Size
===================================================================================================================================================================================================================
Installing:
 wazuh-dashboard                                        x86_64                                        4.3.10-1                                           @commandline                                        136 M

Transaction Summary
===================================================================================================================================================================================================================
Install  1 Package

Total size: 136 M
Installed size: 577 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                           1/1 
  Running scriptlet: wazuh-dashboard-4.3.10-1.x86_64                                                                                                                                                           1/1 
  Installing       : wazuh-dashboard-4.3.10-1.x86_64                                                                                                                                                           1/1 
  Running scriptlet: wazuh-dashboard-4.3.10-1.x86_64                                                                                                                                                           1/1 
  Verifying        : wazuh-dashboard-4.3.10-1.x86_64                                                                                                                                                           1/1 

Installed:
  wazuh-dashboard-4.3.10-1.x86_64                                                                                                                                                                                  

Complete!

On seeing the properties of the package we can see it has the needed SHA256 digest but it is missing some headers:

[vagrant@rocky 1934]$ rpm -K -v wazuh-dashboard-4.3.10-1.x86_64.rpm 
wazuh-dashboard-4.3.10-1.x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
[vagrant@rocky 1934]$ 

The next steps are to automatize the changing of the files if this solution is deemed appropriate, investigating on the missing headers in the package properties and trying to remove the Warnings.

[verdx]

The process explained in the last comment has been automatized. The base builder now changes, using sed the shebang of all files where it was #!/usr/bin/env python or #!/usr/bin/python for the python3 version. This removes errors and works on Rocky Linux 8.5 with FIPS enabled.

It isn't the best solution, as we depend on all systems having installed some version of Python 3.x. The best solution would be to try and remove that files, as they seem to be tests, which may not be used, more research is necessary on that.

For the warnings, they come from incorrect executable permissions on files where it is not needed and also needs to be changed.

[verdx]

As we can see here, node-gyp uses shebang #!/usr/bin/env python3 from version v.8.x.
Opensearch Dashboards uses a previous version, where shebang is still #!/usr/bin/env python , both in versions 1.2.0 and 2.3.0, the ones used for versions 4.3.x and 4.4.x of the Wazuh Dashboards.

The file node-modules isn't in Opensearch Dashboards in their sources so it must be done added during the creation of the downloadable tarball. This means there may be an option to fix this either compiling Opensearch Dashboards by sources or changing the tarball afterwards

[verdx]

A bug in Opensearch Dashboards was found with a similar problem: opensearch-project/OpenSearch-Dashboards#988
It is closed without changes as it may be solved by compiling the dashboards using a newer version of open-gyp. For that solution to work for us we should build the dashboard ourselves, which may bring a lot more problems than taking the image as it is done now. Furthermore, a new solution is mentioned in the issue, coming from the Fedora official documentation.:
https://fedoraproject.org/wiki/Changes/Make_ambiguous_python_shebangs_error#Using_pathfix.py_to_fix_shebangs

The solution is similar to the manual changing of shebangs, but this time using an specialized tool recommended by the official documentation of Fedora. This tool is pathfix.py, a python script downloadable by pip:

https://pypi.org/project/pathfix.py/

In the Fedora project documentation the recommended call is:

pathfix.py -pni '%{__python3} %{py3_shbang_opts}' .

but even installing pyhon3-devel, it has not been possible to make the macros work, so the call has been changed to

/usr/bin/pathfix.py -pni /usr/bin/python3 -s opensearch-dashboards-1.2.0-linux-x64/

In the Rocky Linux 8 version used by docker, pathfix.py must be called with a complete path, as it has been discovered by trial and error

Finally, with the solution implemented in branch 1934-error-upgrading-dashboard-on-fips-enabled-rl8, using pathfix.py, a working package has been created and tested on Rocky Linux 8 with FIPS enabled. More tests are needed to check if the dashboard correctly works on all systems.

[vagrant@rocky output]$ rpm -K -v wazuh-dashboard-4.3.10-1.x86_64.rpm
wazuh-dashboard-4.3.10-1.x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
[vagrant@rocky output]$ sudo yum install wazuh-dashboard-4.3.10-1.x86_64.rpm
Last metadata expiration check: 0:28:51 ago on Mon 28 Nov 2022 08:26:33 AM EST.
Dependencies resolved.
===================================================================================================================================================================================================================
 Package                                                Architecture                                  Version                                            Repository                                           Size
===================================================================================================================================================================================================================
Installing:
 wazuh-dashboard                                        x86_64                                        4.3.10-1                                           @commandline                                        136 M

Transaction Summary
===================================================================================================================================================================================================================
Install  1 Package

Total size: 136 M
Installed size: 577 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                           1/1 
  Running scriptlet: wazuh-dashboard-4.3.10-1.x86_64                                                                                                                                                           1/1 
  Installing       : wazuh-dashboard-4.3.10-1.x86_64                                                                                                                                                           1/1 
  Running scriptlet: wazuh-dashboard-4.3.10-1.x86_64                                                                                                                                                           1/1 
  Verifying        : wazuh-dashboard-4.3.10-1.x86_64                                                                                                                                                           1/1 

Installed:
  wazuh-dashboard-4.3.10-1.x86_64                                                                                                                                                                                  

Complete!
[vagrant@rocky output]$ 

[verdx]

Testing a complete cluster, the same error as the one seen for the dashboard is seen for filebeat in Rocky Linux 8 with FIPS enabled. The output of the installation is:

[root@rocky vagrant]# yum -y install filebeat
Last metadata expiration check: 2:08:08 ago on Tue 29 Nov 2022 04:00:17 AM EST.
Dependencies resolved.
========================================================================================================
 Package                  Architecture           Version                    Repository             Size
========================================================================================================
Installing:
 filebeat                 x86_64                 7.10.2-1                   wazuh                  21 M

Transaction Summary
========================================================================================================
Install  1 Package

Total download size: 21 M
Installed size: 70 M
Downloading Packages:
filebeat-oss-7.10.2-x86_64.rpm                                          3.4 MB/s |  21 MB     00:06    
--------------------------------------------------------------------------------------------------------
Total                                                                   3.4 MB/s |  21 MB     00:06     
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Transaction test error:
  package filebeat-7.10.2-1.x86_64 does not verify: no digest

[root@rocky vagrant]# 

[verdx]

Tests

🔴 CentOS 7:

Error coming from the different rpm versions, it is the same on all systems using rpm 4.11

[root@centos7 vagrant]# yum -y install /vagrant/wazuh-packages/stack/dashboard/rpm/output/wazuh-dashboard-4.3.10-1.x86_64.rpm
Loaded plugins: fastestmirror
Examining /vagrant/wazuh-packages/stack/dashboard/rpm/output/wazuh-dashboard-4.3.10-1.x86_64.rpm: wazuh-dashboard-4.3.10-1.x86_64
Marking /vagrant/wazuh-packages/stack/dashboard/rpm/output/wazuh-dashboard-4.3.10-1.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-dashboard.x86_64 0:4.3.10-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================
 Package                 Arch           Version          Repository                                Size
========================================================================================================
Installing:
 wazuh-dashboard         x86_64         4.3.10-1         /wazuh-dashboard-4.3.10-1.x86_64         577 M

Transaction Summary
========================================================================================================
Install  1 Package

Total size: 577 M
Installed size: 577 M
Downloading packages:
Running transaction check
ERROR You need to update rpm to handle:
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by wazuh-dashboard-4.3.10-1.x86_64
RPM needs to be updated
 You could try running: rpm -Va --nofiles --nodigest
Your transaction was saved, rerun it with:
 yum load-transaction /tmp/yum_save_tx.2022-11-29.11-45.0qVBd6.yumtx
[vagrant@centos7 ~]$ rpm --version
RPM version 4.11.3

🟡 CentOS 8:

This error comes from an insufficient space on the virtual machine where the test was done, in a posterior test the installation finished correctly.

[root@centos8 vagrant]# yum -y install /vagrant/wazuh-packages/stack/dashboard/rpm/output/wazuh-dashboard-4.3.10-1.x86_64.rpm
Last metadata expiration check: 3:01:32 ago on Tue 29 Nov 2022 08:43:36 AM UTC.
Dependencies resolved.
========================================================================================================
 Package                      Architecture        Version               Repository                 Size
========================================================================================================
Installing:
 wazuh-dashboard              x86_64              4.3.10-1              @commandline              136 M

Transaction Summary
========================================================================================================
Install  1 Package

Total size: 136 M
Installed size: 577 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Error: Transaction test error:
  installing package wazuh-dashboard-4.3.10-1.x86_64 needs 577MB on the / filesystem

Error Summary
-------------
Disk Requirements:
   At least 577MB more space needed on the / filesystem.

[vagrant@centos8 ~]$ rpm --version
RPM version 4.14.3

🔴 Red Hat 7:

[root@redhat7 vagrant]# yum -y install wazuh-dashboard-4.3.10-1.x86_64.rpm
Loaded plugins: product-id, search-disabled-repos
Examining wazuh-dashboard-4.3.10-1.x86_64.rpm: wazuh-dashboard-4.3.10-1.x86_64
Marking wazuh-dashboard-4.3.10-1.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-dashboard.x86_64 0:4.3.10-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================
 Package                 Arch           Version           Repository                                Size
=========================================================================================================
Installing:
 wazuh-dashboard         x86_64         4.3.10-1          /wazuh-dashboard-4.3.10-1.x86_64         577 M

Transaction Summary
=========================================================================================================
Install  1 Package

Total size: 577 M
Installed size: 577 M
Downloading packages:
Running transaction check
ERROR You need to update rpm to handle:
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by wazuh-dashboard-4.3.10-1.x86_64
RPM needs to be updated
 You could try running: rpm -Va --nofiles --nodigest
Your transaction was saved, rerun it with:
 yum load-transaction /tmp/yum_save_tx.2022-11-29.11-49.d3ojnF.yumtx
[root@redhat7 vagrant]# rpm --version
RPM version 4.11.3

🟢 Red Hat 8:

[root@redhat8 vagrant]# yum -y install wazuh-dashboard-4.3.10-1.x86_64.rpm
Last metadata expiration check: 0:38:32 ago on Tue 29 Nov 2022 11:16:45 AM UTC.
Dependencies resolved.
=========================================================================================================
 Package                      Architecture        Version                Repository                 Size
=========================================================================================================
Installing:
 wazuh-dashboard              x86_64              4.3.10-1               @commandline              136 M

Transaction Summary
=========================================================================================================
Install  1 Package

Total size: 136 M
Installed size: 577 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                 1/1 
  Running scriptlet: wazuh-dashboard-4.3.10-1.x86_64                                                 1/1 
The memcache was not invalidated by NSS responder.
The memcache was not invalidated by NSS responder.

  Installing       : wazuh-dashboard-4.3.10-1.x86_64                                                 1/1 
  Running scriptlet: wazuh-dashboard-4.3.10-1.x86_64                                                 1/1 
  Verifying        : wazuh-dashboard-4.3.10-1.x86_64                                                 1/1 
Installed products updated.

Installed:
  wazuh-dashboard-4.3.10-1.x86_64                                                                        

Complete!
[root@redhat8 vagrant]# rpm --version
RPM version 4.14.3

🟢 Red Hat 9:

[root@redhat9 vagrant]# yum install -y wazuh-dashboard-4.3.10-1.x86_64.rpm
Last metadata expiration check: 0:34:33 ago on Tue 29 Nov 2022 11:20:13 AM UTC.
Dependencies resolved.
=========================================================================================================
 Package                      Architecture        Version                Repository                 Size
=========================================================================================================
Installing:
 wazuh-dashboard              x86_64              4.3.10-1               @commandline              136 M

Transaction Summary
=========================================================================================================
Install  1 Package

Total size: 136 M
Installed size: 577 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                 1/1 
  Running scriptlet: wazuh-dashboard-4.3.10-1.x86_64                                                 1/1 
  Installing       : wazuh-dashboard-4.3.10-1.x86_64                                                 1/1 
  Running scriptlet: wazuh-dashboard-4.3.10-1.x86_64                                                 1/1 
  Verifying        : wazuh-dashboard-4.3.10-1.x86_64                                                 1/1 
Installed products updated.

Installed:
  wazuh-dashboard-4.3.10-1.x86_64                                                                        

Complete!
[root@redhat9 vagrant]# rpm --version
RPM version 4.16.1.3

Summary

The errors come from the difference of versions between the machine in which the package was created, with rpm 4.14 and the machines with rpm 4.11. On the machines where wazuh-dashboard is correctly installed, the cluster works without problem.

On CentOS 7 and Red Hat 7, the output of the recommended command doesn't give any information:

[root@centos7 vagrant]# rpm -Va --nofiles --nodigest wazuh-dashboard-4.3.10-1.x86_64.rpm
[root@centos7 vagrant]# rpm -i --nofiles --nodigest wazuh-dashboard-4.3.10-1.x86_64.rpm
error: Failed dependencies:
	rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by wazuh-dashboard-4.3.10-1.x86_64
[root@centos7 vagrant]# 

[verdx]

On hold because of wazuh/wazuh#15526

[verdx]

This issue has been on hold after the closing of wazuh/wazuh#15526 because of issues #1965, #1969 and #1984

[verdx]

There is an error building the packages with the branch created for this PR(sudo bash build_package.sh --base local):

+ /usr/lib/rpm/brp-strip /usr/bin/strip
/usr/bin/strip: Unable to recognise the format of the input file `/build/rpmbuild/BUILDROOT/wazuh-dashboard-4.3.11-1.x86_64/usr/share/wazuh-dashboard/plugins/reportsDashboards/.chromium/swiftshader/libGLESv2.so'
/usr/bin/strip: Unable to recognise the format of the input file `/build/rpmbuild/BUILDROOT/wazuh-dashboard-4.3.11-1.x86_64/usr/share/wazuh-dashboard/plugins/reportsDashboards/.chromium/swiftshader/libEGL.so'
error: Bad exit status from /var/tmp/rpm-tmp.zyeHWa (%install)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.zyeHWa (%install)

[verdx]

On hold for #1984 and problems with the Jenkins checks

[verdx]

After creating the package using the Jenkins pipeline, it has worked correctly on systems with rpm version 4.11

🟢 CentOS 7

[vagrant@centos7 ~]$ rpm --version
RPM version 4.11.3
[vagrant@centos7 ~]$ sudo yum install wazuh-dashboard-4.3.11-1.x86_64.rpm 
Loaded plugins: fastestmirror
Examining wazuh-dashboard-4.3.11-1.x86_64.rpm: wazuh-dashboard-4.3.11-1.x86_64
Marking wazuh-dashboard-4.3.11-1.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-dashboard.x86_64 0:4.3.11-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================
 Package                            Arch                      Version                       Repository                                           Size
======================================================================================================================================================
Installing:
 wazuh-dashboard                    x86_64                    4.3.11-1                      /wazuh-dashboard-4.3.11-1.x86_64                    840 M

Transaction Summary
======================================================================================================================================================
Install  1 Package

Total size: 840 M
Installed size: 840 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wazuh-dashboard-4.3.11-1.x86_64                                                                                                    1/1 
  Verifying  : wazuh-dashboard-4.3.11-1.x86_64                                                                                                    1/1 

Installed:
  wazuh-dashboard.x86_64 0:4.3.11-1                                                                                                                   

Complete!
[vagrant@centos7 ~]$ exit

🟢 CentOS 8

[vagrant@centos8 ~]$ sudo yum install ./wazuh-dashboard-4.3.11-1.x86_64.rpm 
CentOS Linux 8 - AppStream                                                                                            145 kB/s | 8.4 MB     00:59    
CentOS Linux 8 - BaseOS                                                                                               182 kB/s | 4.6 MB     00:25    
CentOS Linux 8 - Extras                                                                                               9.9 kB/s |  10 kB     00:01    
Dependencies resolved.
======================================================================================================================================================
 Package                                 Architecture                   Version                            Repository                            Size
======================================================================================================================================================
Installing:
 wazuh-dashboard                         x86_64                         4.3.11-1                           @commandline                         207 M

Transaction Summary
======================================================================================================================================================
Install  1 Package

Total size: 207 M
Installed size: 840 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                              1/1 
  Running scriptlet: wazuh-dashboard-4.3.11-1.x86_64                                                                                              1/1 
  Installing       : wazuh-dashboard-4.3.11-1.x86_64                                                                                              1/1 
  Running scriptlet: wazuh-dashboard-4.3.11-1.x86_64                                                                                              1/1 
  Verifying        : wazuh-dashboard-4.3.11-1.x86_64                                                                                              1/1 

Installed:
  wazuh-dashboard-4.3.11-1.x86_64                                                                                                                     

Complete!
[vagrant@centos8 ~]$ rpm --version
RPM version 4.14.3
[vagrant@centos8 ~]$ 

🟢 Red Hat Linux 7

[vagrant@redhat7 ~]$ rpm --version
RPM version 4.11.3
[vagrant@redhat7 ~]$ sudo yum install ./wazuh-dashboard-4.3.11-1.x86_64.rpm 
Loaded plugins: product-id, search-disabled-repos
Examining ./wazuh-dashboard-4.3.11-1.x86_64.rpm: wazuh-dashboard-4.3.11-1.x86_64
Marking ./wazuh-dashboard-4.3.11-1.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-dashboard.x86_64 0:4.3.11-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================
 Package                            Arch                      Version                       Repository                                           Size
======================================================================================================================================================
Installing:
 wazuh-dashboard                    x86_64                    4.3.11-1                      /wazuh-dashboard-4.3.11-1.x86_64                    840 M

Transaction Summary
======================================================================================================================================================
Install  1 Package

Total size: 840 M
Installed size: 840 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wazuh-dashboard-4.3.11-1.x86_64                                                                                                    1/1 
  Verifying  : wazuh-dashboard-4.3.11-1.x86_64                                                                                                    1/1 
epel/x86_64/metalink                                                                                                           |  21 kB  00:00:00     
epel/x86_64                                                                                                                    | 4.7 kB  00:00:00     
epel/x86_64/group_gz                                                                                                           |  98 kB  00:00:00     
epel/x86_64/updateinfo                                                                                                         | 1.0 MB  00:00:06     
epel/x86_64/primary_db                                                                                                         | 7.0 MB  00:00:40     

Installed:
  wazuh-dashboard.x86_64 0:4.3.11-1                                                                                                                   

Complete!

🟢 Red Hat Linux 8

[vagrant@redhat8 ~]$ sudo yum install ./wazuh-dashboard-4.3.11-1.x86_64.rpm 
Extra Packages for Enterprise Linux 8 - x86_64                                                                        115 kB/s |  13 MB     01:56    
Extra Packages for Enterprise Linux Modular 8 - x86_64                                                                132 kB/s | 733 kB     00:05    
Last metadata expiration check: 0:00:03 ago on Fri 16 Dec 2022 09:42:55 AM UTC.
Dependencies resolved.
======================================================================================================================================================
 Package                                 Architecture                   Version                            Repository                            Size
======================================================================================================================================================
Installing:
 wazuh-dashboard                         x86_64                         4.3.11-1                           @commandline                         207 M

Transaction Summary
======================================================================================================================================================
Install  1 Package

Total size: 207 M
Installed size: 840 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                              1/1 
  Running scriptlet: wazuh-dashboard-4.3.11-1.x86_64                                                                                              1/1 
  Installing       : wazuh-dashboard-4.3.11-1.x86_64                                                                                              1/1 
  Running scriptlet: wazuh-dashboard-4.3.11-1.x86_64                                                                                              1/1 
  Verifying        : wazuh-dashboard-4.3.11-1.x86_64                                                                                              1/1 
Installed products updated.

Installed:
  wazuh-dashboard-4.3.11-1.x86_64                                                                                                                     

Complete!
[vagrant@redhat8 ~]$ rpm --version
RPM version 4.14.3
[vagrant@redhat8 ~]$ 

[verdx]

After a test in a Rocky Linux 8 with FIPS enabled the first error of the issue appeared, making it clear the package of the last commentary must not have been created correctly, but just as a 4.3 package. The error is:

[vagrant@rocky vagrant]$ sudo yum install wazuh-dashboard-4.3.11-1.x86_64.rpm 
Last metadata expiration check: 4:38:46 ago on Fri 16 Dec 2022 06:57:50 AM EST.
Dependencies resolved.
======================================================================================================================================================
 Package                                 Architecture                   Version                            Repository                            Size
======================================================================================================================================================
Installing:
 wazuh-dashboard                         x86_64                         4.3.11-1                           @commandline                         207 M

Transaction Summary
======================================================================================================================================================
Install  1 Package

Total size: 207 M
Installed size: 840 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Error: Transaction test error:
  package wazuh-dashboard-4.3.11-1.x86_64 does not verify: no digest

[vagrant@rocky vagrant]$ 

When checking for digests, the output on Rocky Linux 8 is:

[vagrant@rocky vagrant]$ rpm -K -v wazuh-dashboard-4.3.11-1.x86_64.rpm 
wazuh-dashboard-4.3.11-1.x86_64.rpm:
    Header SHA1 digest: OK
    Payload SHA256 digest: NOTFOUND
    MD5 digest: NOTFOUND
[vagrant@rocky vagrant]$ 

On CentOS 8(rpm 4.14):

[vagrant@centos8 vagrant]$ rpm -K -v wazuh-dashboard-4.3.11-1.x86_64.rpm 
wazuh-dashboard-4.3.11-1.x86_64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
[vagrant@centos8 vagrant]$ 

Onc CentOS 7(rpm 4.11):

[vagrant@centos7 vagrant]$ rpm -K -v wazuh-dashboard-4.3.11-1.x86_64.rpm 
wazuh-dashboard-4.3.11-1.x86_64.rpm:
    Header SHA1 digest: OK (797975de20658cc878bbebc40c4b7156580cdefa)
    MD5 digest: OK (fb66c9008201c9590987637b7f4a56a4)
[vagrant@centos7 vagrant]$ 

[verdx]

When trying to build the package locally on CentOS 7 or 8 the build exits with error:

+ /usr/lib/rpm/brp-strip /usr/bin/strip
/usr/bin/strip: Unable to recognise the format of the input file `/build/rpmbuild/BUILDROOT/wazuh-dashboard-4.3.11-1.x86_64/usr/share/wazuh-dashboard/plugins/reportsDashboards/.chromium/swiftshader/libEGL.so'
/usr/bin/strip: Unable to recognise the format of the input file `/build/rpmbuild/BUILDROOT/wazuh-dashboard-4.3.11-1.x86_64/usr/share/wazuh-dashboard/plugins/reportsDashboards/.chromium/swiftshader/libGLESv2.so'
error: Bad exit status from /var/tmp/rpm-tmp.3gzCRE (%install)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.3gzCRE (%install)
[vagrant@centos7 rpm]$ exit
logout

Jenkins pipelines also give an error.
On Rocky Linux 8 the error the following, and it looks only dependant on the virtual machine:

Error Summary
-------------
Disk Requirements:
   At least 423MB more space needed on the / filesystem.

The command '/bin/sh -c yum install -y openssh-clients sudo gnupg     yum-utils epel-release redhat-rpm-config rpm-devel     zlib zlib-devel rpm-build autoconf automake     glibc-devel libtool perl' returned a non-zero code: 1

[verdx]

After some searching a solution has been found to the installation error rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by wazuh-dashboard-4.3.10-1.x86_64, seen when the package had been successfully created. As seen here, macro w0.gzdio can be used to change the compression algorithm used on the package.

At the same time, the package was not being built because of different errors in all systems tried, among them the errors seen here and here.
After a lot of testing, the error was found to be the new lines added in the base Dashboard builder.sh, as the uncompressed Opensearch Dashboards tar had a different name:

pip3 install pathfix.py
/usr/bin/pathfix.py -pni "/usr/bin/python3 -s" opensearch-dashboards-"${opensearch_version}"-linux-"${architecture}"

Once pathfix.py had a correct run, the base was correctly created and a package was built. It had the correct digests in both Rocky Linux 8(rpm 4.14) and CentOS 7 (rpm 4.11):

[vagrant@rocky output]$ rpm -K -v wazuh-dashboard-4.4.0-1.x86_64.rpm
wazuh-dashboard-4.4.0-1.x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
[vagrant@rocky output]$ 

[vagrant@centos7 output]$ rpm -K -v wazuh-dashboard-4.4.0-1.x86_64.rpm 
wazuh-dashboard-4.4.0-1.x86_64.rpm:
    Header SHA1 digest: OK (ba770cb479127c98f5e9b6cd3e23097a0ab36f06)
    MD5 digest: OK (a5f5031591a9c8de78cdc8128482dc72)

The installation was successful both in Rocky Linux 8 and CentOS 7:

Rocky Linux 8

[vagrant@rocky output]$ sudo yum install wazuh-dashboard-4.4.0-1.x86_64.rpm
Rocky Linux 8 - AppStream                                                                                                                                                          1.3 MB/s | 8.7 MB     00:06    
Rocky Linux 8 - BaseOS                                                                                                                                                             1.1 MB/s | 2.7 MB     00:02    
Rocky Linux 8 - Extras                                                                                                                                                              27 kB/s |  12 kB     00:00    
Dependencies resolved.
===================================================================================================================================================================================================================
 Package                                                 Architecture                                   Version                                         Repository                                            Size
===================================================================================================================================================================================================================
Installing:
 wazuh-dashboard                                         x86_64                                         4.4.0-1                                         @commandline                                         792 M

Transaction Summary
===================================================================================================================================================================================================================
Install  1 Package

Total size: 792 M
Installed size: 763 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                           1/1 
  Running scriptlet: wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                            1/1 
  Installing       : wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                            1/1 
  Running scriptlet: wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                            1/1 
  Verifying        : wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                            1/1 

Installed:
  wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                                                   

Complete!
[vagrant@rocky output]$ 

Signatures:

[vagrant@rocky output]$ rpm -K -v wazuh-dashboard-4.4.0-1.x86_64.rpm
wazuh-dashboard-4.4.0-1.x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
[vagrant@rocky output]$ 

CentOS 7

[vagrant@centos7 output]$ sudo yum install wazuh-dashboard-4.4.0-1.x86_64.rpm 
Loaded plugins: fastestmirror
Examining wazuh-dashboard-4.4.0-1.x86_64.rpm: wazuh-dashboard-4.4.0-1.x86_64
Marking wazuh-dashboard-4.4.0-1.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-dashboard.x86_64 0:4.4.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
 Package                                            Arch                                      Version                                     Repository                                                          Size
===================================================================================================================================================================================================================
Installing:
 wazuh-dashboard                                    x86_64                                    4.4.0-1                                     /wazuh-dashboard-4.4.0-1.x86_64                                    763 M

Transaction Summary
===================================================================================================================================================================================================================
Install  1 Package

Total size: 763 M
Installed size: 763 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                                  1/1 
  Verifying  : wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                                  1/1 

Installed:
  wazuh-dashboard.x86_64 0:4.4.0-1                                                                                                                                                                                 

Complete!
[vagrant@centos7 output]$ 

The only problem with this package is it doesn't use any compression and its size is too much:

-rw-r--r-- 1 verdx verdx 792M Dec 19 17:19 wazuh-dashboard-4.4.0-w0gz.x86_64.rpm

As seen in this article and in file /usr/lib/rpm/macros, there are 9 levels of compression, of which 9 is supposedly the default.

[vagrant@rocky output]$ cat /usr/lib/rpm/macros | grep gzdio
#		"w9.gzdio"	gzip level 9 (default).
#%_source_payload	w9.gzdio
#%_binary_payload	w9.gzdio

The problem is it appears as default on both Rocky Linux 8 and CentOS 7 but the default value is what gave the problem in rpm 4.11 systems here. It has been demonstrated level 0 of compression does not give any problem, and level 6 is the default for gzip.

              Regulate  the  speed of compression using the specified digit #, where -1 or --fast indicates the fastest compression method (less compression) and -9 or --best indicates the slowest compres‐
              sion method (best compression).  The default compression level is -6 (that is, biased towards high compression at expense of speed).

Other changes done today include changing the base of the branch used to 4.4, changing the building machine to Rocky Linux 8.5 and making the pathfix.py run silent, as it listed all files in Opensearch Dashboards source code.

[verdx]

Changing the _source_payload and _binary_payload to w9.gzdio creates the package successfully and the size is very reduced:

-rw-r--r-- 1 root root 227M Dec 19 18:27 output/wazuh-dashboard-4.4.0-1.x86_64.rpm

The package is also installed correctly on both CentOS 7 and Rocky Linux 8 with FIPS:

Rocky Linux 8

[vagrant@rocky vagrant]$ sudo yum install wazuh-dashboard-4.4.0-w9gz.x86_64.rpm 
Last metadata expiration check: 1:10:24 ago on Mon 19 Dec 2022 11:28:47 AM EST.
Dependencies resolved.
===================================================================================================================================================================================================================
 Package                                                 Architecture                                   Version                                         Repository                                            Size
===================================================================================================================================================================================================================
Installing:
 wazuh-dashboard                                         x86_64                                         4.4.0-1                                         @commandline                                         227 M

Transaction Summary
===================================================================================================================================================================================================================
Install  1 Package

Total size: 227 M
Installed size: 763 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                           1/1 
  Running scriptlet: wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                            1/1 
  Installing       : wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                            1/1 
  Running scriptlet: wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                            1/1 
  Verifying        : wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                            1/1 

Installed:
  wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                                                   

Complete!

CentOS 7

[vagrant@centos7 vagrant]$ sudo yum install wazuh-dashboard-4.4.0-w9gz.x86_64.rpm 
Loaded plugins: fastestmirror
Examining wazuh-dashboard-4.4.0-w9gz.x86_64.rpm: wazuh-dashboard-4.4.0-1.x86_64
Marking wazuh-dashboard-4.4.0-w9gz.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-dashboard.x86_64 0:4.4.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
 Package                                           Arch                                     Version                                     Repository                                                            Size
===================================================================================================================================================================================================================
Installing:
 wazuh-dashboard                                   x86_64                                   4.4.0-1                                     /wazuh-dashboard-4.4.0-w9gz.x86_64                                   763 M

Transaction Summary
===================================================================================================================================================================================================================
Install  1 Package

Total size: 763 M
Installed size: 763 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                                  1/1 
  Verifying  : wazuh-dashboard-4.4.0-1.x86_64                                                                                                                                                                  1/1 

Installed:
  wazuh-dashboard.x86_64 0:4.4.0-1                                                                                                                                                                                 

Complete!
[vagrant@centos7 vagrant]$ 

[verdx]

The issue has been rebased again to version 4.5.0, and all the tests repeated.

[verdx]

A package has been built correctly on Jenkins using the changes and uploading the images as dev:
https://ci.wazuh.info/job/Packages_builder/124141