Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Filebeat template to Wazuh indexer #2697

Closed
rauldpm opened this issue Dec 18, 2023 · 4 comments · Fixed by #2712
Closed

Add Filebeat template to Wazuh indexer #2697

rauldpm opened this issue Dec 18, 2023 · 4 comments · Fixed by #2712
Assignees
@rauldpm
Labels
level/task Subtask issue type/enhancement Enhancement issue

Comments

@rauldpm
Copy link
Member

rauldpm commented Dec 18, 2023 •
edited
Loading

Description

As part of https://github.com/wazuh/internal-devel-requests/issues/600, it is necessary to modify the Wazuh indexer package to incorporate the Filebeat wazuh-template

For this, it is necessary to consider the following:

  • To facilitate DevOps deployments, the template must be incorporated into the base file
  • The template must be downloaded from the wazuh/wazuh repository
  • The template must be installed in /etc/wazuh-indexer/wazuh-template.json
  • If the template changes, the upgrade must modify the JSON
  • The management of the current Filebeat template should not be modified by this development, the current commands will be maintained

Working branch

  • wazuh-packages: enhancement/2697-add-filebeat-template-indexer
  • wazuh-jenkins: enhancement/wp2697-add-filebeat-template-indexer
@rauldpm rauldpm added level/task Subtask issue type/enhancement Enhancement issue labels Dec 18, 2023
@wazuhci wazuhci moved this to Triage in Release 4.8.0 Dec 18, 2023
@rauldpm rauldpm self-assigned this Dec 18, 2023
@wazuhci wazuhci moved this from Triage to Backlog in Release 4.8.0 Dec 18, 2023
@wazuhci wazuhci moved this from Backlog to In progress in Release 4.8.0 Dec 18, 2023
@rauldpm
Copy link
Member Author

rauldpm commented Dec 18, 2023 •
edited
Loading

Update report

  • Added new parameter filebeat_module_reference to generate_base.sh in indexer directory
    • If the parameter is empty, the version stored in the VERSION file will be used
    • Build log: indexer-base.log
+ filebeat_module_reference=v4.8.0-alpha1
....
+ curl -so ./etc/wazuh-indexer/wazuh-template.json https://github.com/wazuh/wazuh/blob/v4.8.0-alpha1/extensions/elasticsearch/7.x/wazuh-template.json
...
wazuh-indexer-base/etc/wazuh-indexer/wazuh-template.json

image

  • When building the RPM Wazuh indexer package, the following message was found
+ /root/unattended_installer/builder.sh -c
/root/unattended_installer/builder.sh: line 19: git: command not found
cat: /VERSION: No such file or directory
+ /root/unattended_installer/builder.sh -p
/root/unattended_installer/builder.sh: line 19: git: command not found
cat: /VERSION: No such file or directory
  • It is necessary to determine the impact of this problem, since although the package is generated, it may not be complete due to not having the certificate and password scripts
  • Apparently, the Docker image does not provide the git package anymore, the git package has been added to the BuildRequires section, also the git package has been added to the Dockerfile
  • Now, the following error is found:
+ /root/unattended_installer/builder.sh -c
fatal: not a git repository (or any of the parent directories): .git
cat: /VERSION: No such file or directory
+ /root/unattended_installer/builder.sh -p
fatal: not a git repository (or any of the parent directories): .git
cat: /VERSION: No such file or directory
  • It seems that it is necessary to initialize git inside the container, accessing the container, the following is observed:
bash-4.4# bash /root/unattended_installer/builder.sh -c
fatal: detected dubious ownership in repository at '/root'
To add an exception for this directory, call:

	git config --global --add safe.directory /root
  • After adding the safe directory, the error is no longer visible
  • Apparently, this behavior comes from this commit c45d90e as it is not returning the correct parent directory
bash-4.4# git rev-parse --show-toplevel
/
bash-4.4# pwd
/build/rpmbuild/BUILD
bash-4.4# ls -l /
total 72
lrwxrwxrwx   1 root root    7 Oct 11  2021 bin -> usr/bin
drwxr-xr-x   4 root root 4096 Dec  1 14:31 boot
drwxr-xr-x   4 root root 4096 Dec 18 23:06 build
drwxr-xr-x   5 root root  360 Dec 18 23:06 dev
drwxr-xr-x   1 root root 4096 Dec 18 23:06 etc
drwxr-xr-x   2 root root 4096 Oct 11  2021 home
lrwxrwxrwx   1 root root    7 Oct 11  2021 lib -> usr/lib
lrwxrwxrwx   1 root root    9 Oct 11  2021 lib64 -> usr/lib64
drwx------   2 root root 4096 Nov 14  2021 lost+found
drwxr-xr-x   2 root root 4096 Oct 11  2021 media
drwxr-xr-x   2 root root 4096 Oct 11  2021 mnt
drwxr-xr-x   2 root root 4096 Oct 11  2021 opt
dr-xr-xr-x 613 root root    0 Dec 18 23:06 proc
drwxrwxr-x  26 1000 1000 4096 Dec 18 23:05 root
drwxr-xr-x   1 root root 4096 Dec  1 14:31 run
lrwxrwxrwx   1 root root    8 Oct 11  2021 sbin -> usr/sbin
drwxr-xr-x   2 root root 4096 Oct 11  2021 srv
dr-xr-xr-x  13 root root    0 Dec 18 23:06 sys
drwxr-xr-x   2 root root 4096 Dec 18 23:06 tmp
drwxr-xr-x   1 root root 4096 Nov 14  2021 usr
drwxr-xr-x   1 root root 4096 Nov 14  2021 var
bash-4.4# ls -l /root/
total 120
-rw-rw-r-- 1 1000 1000  5406 Dec 18 20:06 CHANGELOG.md
-rw-rw-r-- 1 1000 1000 18092 Dec 18 20:06 LICENSE
-rw-rw-r-- 1 1000 1000  2627 Dec 18 20:06 README.md
-rw-rw-r-- 1 1000 1000     6 Dec 18 20:06 VERSION
drwxrwxr-x 3 1000 1000  4096 Dec 18 20:06 aix
drwxrwxr-x 4 1000 1000  4096 Dec 18 20:06 alpine
drwxrwxr-x 4 1000 1000  4096 Dec 18 20:06 arch
-rw-rw-r-- 1 1000 1000  8292 Dec 18 20:06 bump_version.py
drwxrwxr-x 4 1000 1000  4096 Dec 18 20:06 debs
drwxrwxr-x 5 1000 1000  4096 Dec 18 20:06 documentation-templates
drwxrwxr-x 2 1000 1000  4096 Dec 18 20:06 hp-ux
drwxrwxr-x 4 1000 1000  4096 Dec 18 20:06 macos
drwxrwxr-x 3 1000 1000  4096 Dec 18 20:06 ova
drwxrwxr-x 3 1000 1000  4096 Dec 18 20:06 puppet_module
drwxrwxr-x 4 1000 1000  4096 Dec 18 20:06 rpms
drwxrwxr-x 6 1000 1000  4096 Dec 18 20:06 solaris
drwxrwxr-x 3 1000 1000  4096 Dec 18 20:06 splunkapp
drwxrwxr-x 4 1000 1000  4096 Dec 18 20:25 stack
drwxrwxr-x 3 1000 1000  4096 Dec 18 20:06 tests
drwxrwxr-x 7 1000 1000  4096 Dec 18 22:20 unattended_installer
drwxrwxr-x 4 1000 1000  4096 Dec 18 20:06 wazuhapp
drwxrwxr-x 2 1000 1000  4096 Dec 18 20:06 windows
drwxrwxr-x 4 1000 1000  4096 Dec 18 20:06 wpk
  • This is also the reason why the build shows the git error, as the Wazuh unattended scripts never used the git command, this can severely impact user deployments
  • I do not consider that this development should be adapted to the approach used in the Wazuh unattended script as the git package should not be a dependency for the builder.sh script, I have opened the following issue The git package should not be a dependency to build the Wazuh unattended scripts #2702
  • This issue will be worked on until those scripts are necessary

@rauldpm rauldpm mentioned this issue Dec 18, 2023
Closed
@rauldpm
Copy link
Member Author

rauldpm commented Dec 19, 2023 •
edited
Loading

Update report

  • RPM install
[root@centos7 vagrant]# yum localinstall wazuh-indexer-4.8.0-1.x86_64.rpm 
Loaded plugins: fastestmirror
Examining wazuh-indexer-4.8.0-1.x86_64.rpm: wazuh-indexer-4.8.0-1.x86_64
Marking wazuh-indexer-4.8.0-1.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-indexer.x86_64 0:4.8.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================
 Package                Arch            Version            Repository                              Size
========================================================================================================
Installing:
 wazuh-indexer          x86_64          4.8.0-1            /wazuh-indexer-4.8.0-1.x86_64          1.0 G

Transaction Summary
========================================================================================================
Install  1 Package

Total size: 1.0 G
Installed size: 1.0 G
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wazuh-indexer-4.8.0-1.x86_64                                                         1/1 
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore
  Verifying  : wazuh-indexer-4.8.0-1.x86_64                                                         1/1 

Installed:
  wazuh-indexer.x86_64 0:4.8.0-1                                                                        

Complete!
[root@centos7 vagrant]# ls -l /etc/wazuh-indexer/wazuh-template.json 
-rw-r-----. 1 wazuh-indexer wazuh-indexer 247013 Dec 18 21:18 /etc/wazuh-indexer/wazuh-template.json
[root@centos7 vagrant]#
  • DEB install
root@debian12:/home/vagrant# apt install ./wazuh-indexer_4.8.0-1_amd64.deb 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'wazuh-indexer' instead of './wazuh-indexer_4.8.0-1_amd64.deb'
The following NEW packages will be installed:
  wazuh-indexer
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/754 MB of archives.
After this operation, 1,050 MB of additional disk space will be used.
Get:1 /home/vagrant/wazuh-indexer_4.8.0-1_amd64.deb wazuh-indexer amd64 4.8.0-1 [754 MB]
Selecting previously unselected package wazuh-indexer.
(Reading database ... 55564 files and directories currently installed.)
Preparing to unpack .../wazuh-indexer_4.8.0-1_amd64.deb ...
Creating wazuh-indexer group... OK
Creating wazuh-indexer user... OK
Unpacking wazuh-indexer (4.8.0-1) ...
Setting up wazuh-indexer (4.8.0-1) ...
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore...............................] 
N: Download is performed unsandboxed as root as file '/home/vagrant/wazuh-indexer_4.8.0-1_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
root@debian12:/home/vagrant# ls -l /etc/wazuh-indexer/wazuh-template.json 
-rw-r----- 1 wazuh-indexer wazuh-indexer 247014 Dec 19 21:41 /etc/wazuh-indexer/wazuh-template.json

@wazuhci wazuhci moved this from In progress to On hold in Release 4.8.0 Dec 19, 2023
@wazuhci wazuhci moved this from On hold to In progress in Release 4.8.0 Dec 19, 2023
@wazuhci wazuhci moved this from In progress to On hold in Release 4.8.0 Dec 20, 2023
@wazuhci wazuhci moved this from On hold to In progress in Release 4.8.0 Dec 20, 2023
@rauldpm rauldpm mentioned this issue Dec 20, 2023
Merged
11 tasks
@rauldpm rauldpm linked a pull request Dec 20, 2023 that will close this issue
Added wazuh-filebeat template to Wazuh indexer #2712
Merged
11 tasks
@rauldpm
Copy link
Member Author

rauldpm commented Dec 20, 2023 •
edited
Loading

Update report

[root@centos7 vagrant]# wget https://packages-dev.wazuh.com/warehouse/test/4.8/rpm/wazuh-indexer-4.8.0-wp2697.x86_64.rpm
--2023-12-20 21:27:02--  https://packages-dev.wazuh.com/warehouse/test/4.8/rpm/wazuh-indexer-4.8.0-wp2697.x86_64.rpm
Resolving packages-dev.wazuh.com (packages-dev.wazuh.com)... 52.84.66.16, 52.84.66.65, 52.84.66.124, ...
Connecting to packages-dev.wazuh.com (packages-dev.wazuh.com)|52.84.66.16|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 778834604 (743M) [application/x-rpm]
Saving to: ‘wazuh-indexer-4.8.0-wp2697.x86_64.rpm’

100%[==============================================================>] 778,834,604 24.6MB/s   in 32s    

2023-12-20 21:27:35 (23.2 MB/s) - ‘wazuh-indexer-4.8.0-wp2697.x86_64.rpm’ saved [778834604/778834604]

[root@centos7 vagrant]# yum localinstall wazuh-indexer-4.8.0-wp2697.x86_64.rpm -y
Loaded plugins: fastestmirror
Examining wazuh-indexer-4.8.0-wp2697.x86_64.rpm: wazuh-indexer-4.8.0-wp2697.x86_64
Marking wazuh-indexer-4.8.0-wp2697.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-indexer.x86_64 0:4.8.0-wp2697 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================
 Package              Arch          Version             Repository                                 Size
========================================================================================================
Installing:
 wazuh-indexer        x86_64        4.8.0-wp2697        /wazuh-indexer-4.8.0-wp2697.x86_64        1.0 G

Transaction Summary
========================================================================================================
Install  1 Package

Total size: 1.0 G
Installed size: 1.0 G
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wazuh-indexer-4.8.0-wp2697.x86_64                                                    1/1 
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore
  Verifying  : wazuh-indexer-4.8.0-wp2697.x86_64                                                    1/1 

Installed:
  wazuh-indexer.x86_64 0:4.8.0-wp2697                                                                   

Complete!
[root@centos7 vagrant]# ls -l /etc/wazuh-indexer/wazuh-template.json 
-rw-rw----. 1 wazuh-indexer wazuh-indexer 247014 Dec 20 20:39 /etc/wazuh-indexer/wazuh-template.json

root@ubuntu2204:/home/vagrant# wget https://packages-dev.wazuh.com/warehouse/test/4.8/deb/wazuh-indexer_4.8.0-wp2697_amd64.deb
--2023-12-20 21:58:06--  https://packages-dev.wazuh.com/warehouse/test/4.8/deb/wazuh-indexer_4.8.0-wp2697_amd64.deb
Resolving packages-dev.wazuh.com (packages-dev.wazuh.com)... 18.244.155.50, 18.244.155.11, 18.244.155.30, ...
Connecting to packages-dev.wazuh.com (packages-dev.wazuh.com)|18.244.155.50|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 759108564 (724M) [binary/octet-stream]
Saving to: ‘wazuh-indexer_4.8.0-wp2697_amd64.deb’

wazuh-indexer_4.8.0-wp2697_amd64.deb                 100%[=====================================================================================================================>] 723.94M  24.9MB/s    in 30s     

2023-12-20 21:58:37 (24.4 MB/s) - ‘wazuh-indexer_4.8.0-wp2697_amd64.deb’ saved [759108564/759108564]

root@ubuntu2204:/home/vagrant# apt install ./wazuh-indexer_4.8.0-wp2697_amd64.deb 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'wazuh-indexer' instead of './wazuh-indexer_4.8.0-wp2697_amd64.deb'
The following NEW packages will be installed:
  wazuh-indexer
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 0 B/759 MB of archives.
After this operation, 1,050 MB of additional disk space will be used.
Get:1 /home/vagrant/wazuh-indexer_4.8.0-wp2697_amd64.deb wazuh-indexer amd64 4.8.0-wp2697 [759 MB]
Selecting previously unselected package wazuh-indexer.
(Reading database ... 75812 files and directories currently installed.)
Preparing to unpack .../wazuh-indexer_4.8.0-wp2697_amd64.deb ...
Creating wazuh-indexer group... OK
Creating wazuh-indexer user... OK
Unpacking wazuh-indexer (4.8.0-wp2697) ...
Setting up wazuh-indexer (4.8.0-wp2697) ...
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
Scanning processes...                                                                                                                                                                                              
Scanning linux images...                                                                                                                                                                                           

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
N: Download is performed unsandboxed as root as file '/home/vagrant/wazuh-indexer_4.8.0-wp2697_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
root@ubuntu2204:/home/vagrant# ls -l /etc/wazuh-indexer/wazuh-template.json 
-rw-rw---- 1 wazuh-indexer wazuh-indexer 247013 Dec 20 21:29 /etc/wazuh-indexer/wazuh-template.json

@wazuhci wazuhci moved this from In progress to On hold in Release 4.8.0 Dec 20, 2023
@wazuhci wazuhci moved this from On hold to In progress in Release 4.8.0 Dec 20, 2023
This was referenced Dec 21, 2023
Closed
Closed
@wazuhci wazuhci moved this from In progress to Pending review in Release 4.8.0 Dec 21, 2023
@wazuhci wazuhci moved this from Pending review to Pending final review in Release 4.8.0 Dec 21, 2023
@davidjiglesias
Copy link
Member

LGTM!

@wazuhci wazuhci moved this from Pending final review to Done in Release 4.8.0 Dec 22, 2023
@c-bordon c-bordon mentioned this issue Dec 26, 2023
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rauldpm rauldpm

Labels
level/task Subtask issue type/enhancement Enhancement issue
Projects
No open projects
Release 4.8.0
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Added wazuh-filebeat template to Wazuh indexer wazuh/wazuh-packages
2 participants
@rauldpm @davidjiglesias