Research and document the changes from CI/CD team to the indexer

[AlexRuiz7]

Description

@wazuh/cicd team has managed the wazuh-indexer component until now.

We need to study and document the changes performed to OpenSearch to migrate them to this fork.

[AlexRuiz7]

Pre-fork packages builder

Issues:

• PoC - Wazuh Indexer package with Opensearch wazuh-packages#1099
• Indexer package v2 - update to 1.2.4 wazuh-packages#1225
• Indexer package v3 - Base generation wazuh-packages#1240

The tools used by the @wazuh/cicd team to generate the wazuh-indexer packages are stored in wazuh-packages/stack/indexer

In the base/builder.sh file, we can find these changes:

https://github.com/wazuh/wazuh-packages/blob/3d3684125348f32ba98836588c7eea8811fbde5f/stack/indexer/base/builder.sh#L54-L71

# Remove unnecessary files and set up configuration
mv opensearch-"${opensearch_version}" "${base_dir}"
cd "${base_dir}"
find -type l -exec rm -rf {} \;
find -name "*.bat" -exec rm -rf {} \;
rm -rf README.md manifest.yml opensearch-tar-install.sh logs
sed -i 's|OPENSEARCH_DISTRIBUTION_TYPE=tar|OPENSEARCH_DISTRIBUTION_TYPE=rpm|g' bin/opensearch-env
sed -i 's|"$OPENSEARCH_HOME"/config|/etc/wazuh-indexer|g' bin/opensearch-env 
cp -r /root/stack/indexer/base/files/systemd-entrypoint bin/
mkdir -p ./etc/wazuh-indexer/
cp -r ./config/* ./etc/wazuh-indexer/
rm -rf ./config
cp -r /root/stack/indexer/base/files/etc/wazuh-indexer/* ./etc/wazuh-indexer/
cp -r /root/stack/indexer/base/files/etc/sysconfig ./etc/
cp -r /root/stack/indexer/base/files/etc/init.d ./etc/
cp -r /root/stack/indexer/base/files/usr ./
rm -rf ./plugins/opensearch-security/tools/install_demo_configuration.sh
cp /root/VERSION .

List of changes

Let's do a line by line analysis:

A. find -type l -exec rm -rf {} \;: removes any symbolic link

B. find -name "*.bat" -exec rm -rf {} \: removes bat scripts (Windows)

C. rm -rf README.md manifest.yml opensearch-tar-install.sh logs: removes these files and the logs folder

D. sed -i 's|OPENSEARCH_DISTRIBUTION_TYPE=tar|OPENSEARCH_DISTRIBUTION_TYPE=rpm|g' bin/opensearch-env: change package type from tar to rpm

E. sed -i 's|"$OPENSEARCH_HOME"/config|/etc/wazuh-indexer|g' bin/opensearch-env: sets the config path to /etc/wazuh-indexer. This is used for a validation in this script.

[opensearch@1dd38f7f7640 ~]$ cat bin/opensearch-env | grep -n '"$OPENSEARCH_HOME"/config'
99:  if [ -z "$OPENSEARCH_PATH_CONF" ]; then OPENSEARCH_PATH_CONF="$OPENSEARCH_HOME"/config; fi

F. cp -r /root/stack/indexer/base/files/systemd-entrypoint bin/ copies the file to the bin folder within the OpenSearch files. I can't see the reason to do this.

G. The next lines copy the configuration and service files. These files can be compared with the original files hosted in the opensearch-build repository.

Finally, there are these changes. I've asked the @wazuh/cicd team about this. They don't remember the exact reason to include this.

# Compile systemD module
git clone https://github.com/opensearch-project/OpenSearch.git --branch="${opensearch_version}" --depth=1
cd OpenSearch/modules/systemd
export JAVA_HOME=/etc/alternatives/java_sdk_11
../../gradlew build || true
mkdir -p "${base_dir}"/modules/systemd
cp build/distributions/systemd-"${opensearch_version}"-SNAPSHOT.jar "${base_dir}"/modules/systemd/systemd-"${opensearch_version}".jar
cp build/resources/test/plugin-security.policy "${base_dir}"/modules/systemd/
cp build/generated-resources/plugin-descriptor.properties "${base_dir}"/modules/systemd/
sed -i 's|-SNAPSHOT||g' "${base_dir}"/modules/systemd/plugin-descriptor.properties
cd "${base_dir}"
rm -rf OpenSearch

Conclusions

• Changes to discuss / extend information: A, B, C, D, F.
• We need to perform a more in depth of the changes in the configuration and unit files (G).
• We need to clarify why re-compiling the systemD module is required.

[AlexRuiz7]

Service files comparison

In the base/files folder, we can find these files:

files
├── etc
│   ├── init.d
│   │   └── wazuh-indexer
│   ├── sysconfig
│   │   └── **wazuh-indexer**
│   └── wazuh-indexer
│       ├── jvm.options
│       └── opensearch.yml
├── systemd-entrypoint
└── usr
    └── lib
        ├── sysctl.d
        │   └── wazuh-indexer.conf
        ├── systemd
        │   └── system
        │       ├── **wazuh-indexer-performance-analyzer.service**
        │       └── **wazuh-indexer.service**
        └── tmpfiles.d
            └── wazuh-indexer.conf

From those, only these have their OpenSearch analog in the opensearch-build repo (wrapped between asterisks):

• etc/sysconfig/opensearch
• usr/lib/systemd/system/opensearch-performance-analyzer.service
• usr/lib/systemd/system/opensearch.service

We'll focus on comparing these first (against v2.9.0 files).

etc/sysconfig/wazuh-indexer

• OPENSEARCH_HOME is commented (disabled).
• LD_LIBRARY_PATH is missing (probably outdated).
• OPENSEARCH_JAVA_HOME has been renamed to JAVA_HOME (probably outdated. Might have unknown consequences). The variable is empty, while in OpenSearch contains the path to Java. In both versions, the setting is commented by default, but the changes are evident.
• PID_DIR is commented (disabled).
• OPENSEARCH_JAVA_OPTS renamed to WI_JAVA_OPTS.
• OPENSEARCH_STARTUP_SLEEP_TIME rename to WI_STARTUP_SLEEP_TIME.

usr/lib/systemd/system/opensearch-performance-analyzer.service

Lots of changes, although it's a short file. Better seen together.

[Unit]
Description=Wazuh-indexer Performance Analyzer
PartOf=wazuh-indexer.service
After=wazuh-indexer.service

[Service]
ExecStart=/usr/share/wazuh-indexer/bin/performance-analyzer-agent-cli
Restart=on-failure
User=wazuh-indexer
Group=wazuh-indexer
Environment=OPENSEARCH_HOME=/usr/share/wazuh-indexer
WorkingDirectory=/usr/share/wazuh-indexer

[Install]
WantedBy=wazuh-indexer.service

[Unit]
Description=OpenSearch Performance Analyzer

[Service]
Type=simple
ExecStart=/usr/share/opensearch/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli
Restart=on-failure
User=opensearch
Group=opensearch
EnvironmentFile=-/etc/sysconfig/opensearch
WorkingDirectory=/usr/share/opensearch

[Install]
WantedBy=multi-user.target

usr/lib/systemd/system/opensearch.service

• Changed description and documentation. These changes are expected.
• Added Wants=network-online.target After=network-online.target for no apparent reason.
• Changed workingDirectory, user, group, execStart and environmentFile. Expected changes.
• TimeoutStartSec increased from 75 to 180.
• Added several environment variables. The same that were commented in etc/sysconfig/wazuh-indexer. These changes are redundant and should be rolled back.

Environment=OPENSEARCH_HOME=/usr/share/wazuh-indexer
Environment=OPENSEARCH_PATH_CONF=/etc/wazuh-indexer
Environment=PID_DIR=/run/wazuh-indexer
Environment=OPENSEARCH_SD_NOTIFY=true

Other files

jvm.options

• Hard-coded Xms, Xmx, HeapDumpPath and ErrorFile settings, which are handles using variables in OpenSearch.
• Appended this chunk of code:

## OpenSearch Performance Analyzer
-Dclk.tck=100
-Djdk.attach.allowAttachSelf=true
-Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy
--add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED

opensearch.yml

Not going to dive into this one for obvious reasons.

init.d/wazuh-indexer

This file contains a typical init file with these operations: {start|stop|status|restart|condrestart|try-restart|reload|force-reload}

systemd-entrypoint

Wrapped file used to fetch keystore information.

# This wrapper script allows SystemD to feed a file containing a passphrase into
# the main Wazuh-indexer startup script

if [ -n "$WI_KEYSTORE_PASSPHRASE_FILE" ] ; then
  exec /usr/share/wazuh-indexer/bin/opensearch "$@" < "$WI_KEYSTORE_PASSPHRASE_FILE"
else
  exec /usr/share/wazuh-indexer/bin/opensearch "$@"
fi

sysctl.d/wazuh-indexer.conf

Contains vm.max_map_count=262144. This setting can be set in etc/sysconfig/wazuh-indexer (it's commented in fact), but I don't think it would hurt keeping it either.

tmpfiles.d/wazuh-indexer.conf

Contains d /run/wazuh-indexer 0750 wazuh-indexer wazuh-indexer - -

No idea, I don't speak gibberish (joke). It seems to store user, group and permissions for the wazuh-indexer process. More information here.