Research about the generation of OS packages (rpm, deb)

[AlexRuiz7]

Description

We need to investigate how to produce production packages in rpm and deb formats.

Currently, we only generate production packages in tar.gz format. See #46.

Additional info

These are the tools used by CI/CD:

• https://github.com/wazuh/wazuh-packages/tree/master/stack/indexer/rpm
• https://github.com/wazuh/wazuh-packages/tree/master/stack/indexer/deb
• https://github.com/wazuh/wazuh-packages/tree/master/stack/dashboard/rpm
• https://github.com/wazuh/wazuh-packages/tree/master/stack/dashboard/deb

[Tostti]

Research

Currently the main repo of Opensearch Dashboards contains scripts to generate deb and rpm packages. Testing it in my local machine, it fails with some error.
Researching on that, I found several issues with the same error:

• [Build] Build amd64 for rpm opensearch-project/OpenSearch-Dashboards#1748

• RPM build for core dashboards failed in fpm assemble step opensearch-project/opensearch-build#1887

It seems to be an error related to a tool called fpm, that happens only to some users. In my research, I found comments suggesting different tools to replace it. I will continue researching and testing.

[AlexRuiz7]

I've tried building the packages using the command:

yarn build -v --skip-docker-ubi --release

which have failed due to the following error:

   │ERROR failure 0 sec
   │ERROR Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
   │          at new NodeError (internal/errors.js:322:7)
   │          at Socket.onclose (internal/streams/end-of-stream.js:117:38)
   │          at Socket.emit (events.js:412:35)
   │          at Pipe.<anonymous> (net.js:686:12)

The internal command failing is the following:

$ fpm       \
    --force \
    -t deb \
    --package /home/node/app/target/NAME-2.4.1-ARCH.TYPE \
    -s dir \
    --name opensearch-dashboards \
    --description Explore and visualize your OpenSearch data \
    --version 2.4.1 --url https://opensearch.org \
    --vendor OpenSearch \
    --maintainer OpenSearch Dashboards Team <[email protected]> \
    --license ASL-2.0 \
    --conflicts opensearch-dashboards \
    --after-install /home/node/app/src/dev/build/tasks/os_packages/package_scripts/post_install.sh \
    --before-install /home/node/app/src/dev/build/tasks/os_packages/package_scripts/pre_install.sh \
    --before-remove /home/node/app/src/dev/build/tasks/os_packages/package_scripts/pre_remove.sh \
    --after-remove /home/node/app/src/dev/build/tasks/os_packages/package_scripts/post_remove.sh \
    --config-files /etc/opensearch-dashboards/opensearch_dashboards.yml \
    --template-value user=opensearch-dashboards \
    --template-value group=opensearch-dashboards \
    --template-value optimizeDir=/usr/share/opensearch-dashboards/optimize \
    --template-value configDir=/etc/opensearch-dashboards \
    --template-value pluginsDir=/usr/share/opensearch-dashboards/plugins \
    --template-value dataDir=/var/lib/opensearch-dashboards \
    --exclude usr/share/opensearch-dashboards/config \
    --exclude usr/share/opensearch-dashboards/data \
    --exclude usr/share/opensearch-dashboards/assets \
    --architecture amd64 \
    --deb-priority optional /home/node/app/build/opensearch-dashboards-2.4.1-linux-x64/=/usr/share/opensearch-dashboards/ /home/node/app/build/opensearch-dashboards-2.4.1-linux-x64/config/=/etc/opensearch-dashboards/ /home/node/app/build/opensearch-dashboards-2.4.1-linux-x64/data/=/var/lib/opensearch-dashboards/ /home/node/app/src/dev/build/tasks/os_packages/service_templates/sysv/=/ /home/node/app/src/dev/build/tasks/os_packages/service_templates/systemd/=/

Raw

$ fpm --force -t deb --package /home/node/app/target/NAME-2.4.1-ARCH.TYPE -s dir --name opensearch-dashboards --description Explore and visualize your OpenSearch data --version 2.4.1 --url https://opensearch.org --vendor OpenSearch --maintainer OpenSearch Dashboards Team <[email protected]> --license ASL-2.0 --conflicts opensearch-dashboards --after-install /home/node/app/src/dev/build/tasks/os_packages/package_scripts/post_install.sh --before-install /home/node/app/src/dev/build/tasks/os_packages/package_scripts/pre_install.sh --before-remove /home/node/app/src/dev/build/tasks/os_packages/package_scripts/pre_remove.sh --after-remove /home/node/app/src/dev/build/tasks/os_packages/package_scripts/post_remove.sh --config-files /etc/opensearch-dashboards/opensearch_dashboards.yml --template-value user=opensearch-dashboards --template-value group=opensearch-dashboards --template-value optimizeDir=/usr/share/opensearch-dashboards/optimize --template-value configDir=/etc/opensearch-dashboards --template-value pluginsDir=/usr/share/opensearch-dashboards/plugins --template-value dataDir=/var/lib/opensearch-dashboards --exclude usr/share/opensearch-dashboards/config --exclude usr/share/opensearch-dashboards/data --exclude usr/share/opensearch-dashboards/assets --architecture amd64 --deb-priority optional /home/node/app/build/opensearch-dashboards-2.4.1-linux-x64/=/usr/share/opensearch-dashboards/ /home/node/app/build/opensearch-dashboards-2.4.1-linux-x64/config/=/etc/opensearch-dashboards/ /home/node/app/build/opensearch-dashboards-2.4.1-linux-x64/data/=/var/lib/opensearch-dashboards/ /home/node/app/src/dev/build/tasks/os_packages/service_templates/sysv/=/ /home/node/app/src/dev/build/tasks/os_packages/service_templates/systemd/=/

From this information, we now know that fpm is used internally to build the deb and rpm packages.

[Tostti]

I've been looking at the Opensearch Build repo and testing how it works, as it is mentioned in opensearch-project/opensearch-build#1545 that it uses rpmbuild instead of fpm due to some errors with fpm. I was not able to generate any other package than .tar.gz yet,

[Tostti]

I've been looking at the tools used by CI/CD and was able to generate packages (from the Opensearch repo). Then I started modifying the tools locally to make them match our requirements in order to generate packages of our own repository.

[Tostti]

I was able to generate a functional deb package based on our fork, and tested it in a VM.

The tools used by CI/CD consists in 3 parts:

1. Base: this generates a .tar.gz based on the Opensearch's own, but modifying the desired files for Wazuh.
2. Deb: this runs the base first, and then generates a deb package based on it.
3. RPM: this runs the base first, and then generates a deb package based on it.

To generate the deb package, I modified the base's builder.sh, Dockerfile and generate_base.sh files, to allow the script to read a local file and also to remove the unnecessary steps that are already done on the fork.
Then I generated a .tar.gz with the normal yarn build --release --skip-os-packages from our fork, and a build from our security plugin that the script will install.
After placing those build files in stack/dashboard/base, I ran the command ./build_package.sh --app-url https://packages-dev.wazuh.com/pre-release/ui/dashboard/wazuh-4.4.1-1.zip from stack/dashboard/deb
The URL at the end is the location of the wazuh plugin build (I used a previous version).

I was not able to generate a rpm package due to an error with some dependency of the Dockerfile:

ERROR [4/8] RUN yum install -y https://repo.ius.io/ius-release-el$(rpm -E '%{rhel}').rpm

[Tostti]

I was able to generate a functional rpm package based on our fork, and tested it in a VM.
Additionally to the previously modified files, I also modified rpm's wazuh-dashboard.spec and Dockerfile, then from stack/dashboard/rpm ran ./build_package.sh --app-url https://packages-dev.wazuh.com/pre-release/ui/dashboard/wazuh-4.4.1-1.zip

[AlexRuiz7]

Reviewed in #72