Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data Persistence Model Redesign MVP validation test - I #478

Open
15 tasks done
AlexRuiz7 opened this issue Oct 18, 2024 · 6 comments · May be fixed by #482
Open
15 tasks done

Data Persistence Model Redesign MVP validation test - I #478

AlexRuiz7 opened this issue Oct 18, 2024 · 6 comments · May be fixed by #482
Assignees
@QU3B1M
Labels
level/task Task issue type/test Test issue

Comments

@AlexRuiz7
Copy link
Member

AlexRuiz7 commented Oct 18, 2024 •
edited by QU3B1M
Loading

Description

For the delivery of the MVP phase of wazuh/wazuh#22887, we need to check that:

  • The packages can be built.
  • The packages can be installed.
  • The Wazuh Indexer service starts and runs (start, stop, restart).
  • A Wazuh Indexer cluster can be formed.
  • Wazuh Indexer plugins are installed (listed by the API).
  • The setup plugin
    • index templates and indices are created (listed by the API).
    • Every index template includes agent related fields (agent.id and agent.groups).
    • Every index template's index pattern allows a suffix (ends with *).
  • The Command Manager
    • API listens to POST HTTP request
    • index template is created
    • indexes data on the .commands index on a successful HTTP POST request
    • sends data to the Management API
  • Wazuh Indexer can be uninstalled.

Tasks

For each of the sections, provide instructions and evidences of their testing.
@AlexRuiz7 AlexRuiz7 added level/task Task issue type/test Test issue labels Oct 18, 2024
@QU3B1M QU3B1M self-assigned this Oct 18, 2024
@QU3B1M
Copy link
Member

QU3B1M commented Oct 18, 2024 •
edited
Loading

Validation process

  1. The packages can be built.
    Check the packages are built as result of the GHA Build workflow

    $ GITHUB_TOKEN=<MY_GITHUB_TOKEN> bash ./check_package.sh 11406244332 5.0.0

Fetching artifacts list...
Checking wazuh-indexer-5.0.0-0.x86_64.rpm package is generated for workflow run 11406244332
Wazuh indexer package built successfully.
[ Artifact ID: 2074797194 ]
    Test script 
    #!/bin/bash

## SPDX-License-Identifier: Apache-2.0
## The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.

# Check if the necessary arguments are provided
if [ "$#" -ne 2 ]; then
    echo "Usage: $0 <RUN_ID> <PKG_VERSION> <(Optional)PKG_REVISION>"
    echo
    echo "Parameters:"
    echo "    RUN_ID         The GHA workflow execution ID."
    echo "    PKG_VERSION    The version of the wazuh-indexer package."
    echo "    PKG_REVISION   (Optional) The revision of the package. Defaults to 'test' if not provided."
    echo
    echo "Please ensure you have the GITHUB_TOKEN environment variable set to access the GitHub repository."
    echo
    exit 1
fi

RUN_ID=$1
PKG_VERSION=$2
PKG_REVISION=${3:-"0"}
REPO="wazuh/wazuh-indexer"
URL="https://api.github.com/repos/$REPO/actions/artifacts"

# Detect OS and architecture
if [ -f /etc/os-release ]; then
    . /etc/os-release
    OS=$(echo $NAME | tr '[:upper:]' '[:lower:]')
else
    echo "Unsupported OS."
    exit 1
fi

ARCH=$(uname -m)
# Determine package type
case "$OS" in
    "ubuntu" | "debian")
        PKG_FORMAT="deb"
        [ "$ARCH" == "x86_64" ] && ARCH="amd64"
        PKG_NAME="wazuh-indexer_${PKG_VERSION}-${PKG_REVISION}_${ARCH}.${PKG_FORMAT}"
        ;;
    "centos" | "fedora" | "rhel" | "red hat enterprise linux")
        PKG_FORMAT="rpm"
        PKG_NAME="wazuh-indexer-${PKG_VERSION}-${PKG_REVISION}.${ARCH}.${PKG_FORMAT}"
        ;;
    *)
        echo "Unsupported OS. ${OS}"
        exit 1
        ;;
esac

# Fetch the list of artifacts
echo "Fetching artifacts list..."
RESPONSE=$(curl -s -L -H "Accept: application/vnd.github+json" -H "Authorization: Bearer $GITHUB_TOKEN" -H "X-GitHub-Api-Version: 2022-11-28" $URL?name=$PKG_NAME)

# Check if the curl command was successful
if [ $? -ne 0 ]; then
    echo "Error: Failed to fetch artifacts."
    exit 1
fi

# Check if the artifact from the specified workflow run ID exists
echo "Checking ${PKG_NAME} package is generated for workflow run ${RUN_ID}"
ARTIFACT=$(echo "$RESPONSE" | jq -e ".artifacts[] | select(.workflow_run.id == $RUN_ID)")

if [ -n "$ARTIFACT" ]; then
    ARTIFACT_ID=$(echo "$ARTIFACT" | jq -r '.id')
    echo "Wazuh indexer package built successfully."
    echo "[ Artifact ID: $ARTIFACT_ID ]"
else
    echo "Error: Wazuh indexer package not found."
fi

  2. The package can be installed
    Download and install the package generated on the GHA workflow

    $ GITHUB_TOKEN=<MY_GITHUB_TOKEN> bash ./check_installation.sh 2074797194 5.0.0

Downloading wazuh-indexer package from GitHub artifactory...
(It could take a couple minutes)
Package downloaded successfully
Decompressing wazuh-indexer package...
Archive:  ./package.zip
  inflating: wazuh-indexer-5.0.0-0.x86_64.rpm
Package decompressed
Installing wazuh-indexer package...
### NOT starting on installation, please execute the following statements to configure wazuh-indexer service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable wazuh-indexer.service
### You can start wazuh-indexer service by executing
 sudo systemctl start wazuh-indexer.service
Couldn't write '64' to 'kernel/random/read_wakeup_threshold', ignoring: No such file or directory
Package installed successfully.
    Testing script 
    #!/bin/bash

# SPDX-License-Identifier: Apache-2.0
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.

# Usage function to display help
usage() {
    echo "Usage: $0 <ARTIFACT_ID> <PKG_VERSION> <(Optional)PKG_REVISION>"
    echo
    echo "Parameters:"
    echo "    ARTIFACT_ID    The unique ID of the GHA artifact."
    echo "    PKG_VERSION    The version of the wazuh-indexer package."
    echo "    PKG_REVISION   (Optional) The revision of the package. Defaults to 'test' if not provided."
    echo
    echo "Please ensure you have the GITHUB_TOKEN environment variable set to access the GitHub repository."
    echo
    exit 1
}

# Check if GITHUB_TOKEN env var is set
if [ -z "$1" ]; then
    echo "Error: Environment variable GITHUB_TOKEN is not configured."
    usage
fi

# Check if ARTIFACT_ID is provided
if [ -z "$1" ]; then
    echo "Error: ARTIFACT_ID not provided."
    usage
fi

# Check if PKG_VERSION is provided
if [ -z "$2" ]; then
    echo "Error: PKG_VERSION not provided."
    usage
fi

ARTIFACT_ID=$1
PKG_VERSION=$2
PKG_REVISION=${3:-"0"}
REPO="wazuh/wazuh-indexer"
URL="https://api.github.com/repos/${REPO}/actions/artifacts/${ARTIFACT_ID}/zip"

# Detect OS and architecture
if [ -f /etc/os-release ]; then
    . /etc/os-release
    OS=$(echo $NAME | tr '[:upper:]' '[:lower:]')
else
    echo "Unsupported OS."
    exit 1
fi

ARCH=$(uname -m)
# Determine package type
case "$OS" in
    "ubuntu" | "debian")
        PKG_FORMAT="deb"
        [ "$ARCH" == "x86_64" ] && ARCH="amd64"
        # Construct package name
        PKG_NAME="wazuh-indexer_${PKG_VERSION}-${PKG_REVISION}_${ARCH}.${PKG_FORMAT}"
        ;;
    "centos" | "fedora" | "rhel" | "red hat enterprise linux")
        PKG_FORMAT="rpm"
        # Construct package name
        PKG_NAME="wazuh-indexer-${PKG_VERSION}-${PKG_REVISION}.${ARCH}.${PKG_FORMAT}"
        ;;
    *)
        echo "Unsupported OS."
        exit 1
        ;;
esac

# Download the package
echo "Downloading wazuh-indexer package from GitHub artifactory..."
echo "(It could take a couple minutes)"
curl -L -H "Accept: application/vnd.github+json" \
    -H "Authorization: Bearer $GITHUB_TOKEN" \
    -H "X-GitHub-Api-Version: 2022-11-28" \
    $URL -o package.zip > /dev/null 2>&1
if [ $? -ne 0 ]; then
    echo "Error downloading package."
    exit 1
fi
echo "Package downloaded successfully"

# Unzip the package
echo "Decompressing wazuh-indexer package..."
unzip ./package.zip
rm package.zip
if [ $? -ne 0 ]; then
    echo "Error unzipping package."
    exit 1
fi
echo "Package decompressed"

# Install the package
echo "Installing wazuh-indexer package..."
case "$PKG_FORMAT" in
    "deb")
        sudo dpkg -i $PKG_NAME
        ;;
    "rpm")
        sudo rpm -i $PKG_NAME
        ;;
esac
if [ $? -ne 0 ]; then
    echo "Error installing package."
    exit 1
fi

echo "Package installed successfully."

  3. The Wazuh Indexer service starts and runs (start, stop, restart).

    • Configure wazuh-indexer and deploy certificates

      sudo bash ./02_apply_certificates.sh node-1 node-2 192.168.56.10 192.168.56.11

Creating a backup of the original config file...
Updating configuration...
Configuration updated successfully. Backup created at ./opensearch.yml.bak
Creating certificates directory and extracting certificates...
Moving and setting permissions for certificates...
Certificates configured successfully.
      Test script 
      #!/bin/bash

# SPDX-License-Identifier: Apache-2.0
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.

# Function to display usage help
usage() {
    echo
    echo "Usage: $0 <CURRENT_NODE> <SECOND_NODE> <(Optional)CURRENT_NODE_IP> <(Optional)SECOND_NODE_IP>"
    echo
    echo "Parameters:"
    echo "  CURRENT_NODE       Name of the current node"
    echo "  SECOND_NODE        Name of the second node"
    echo "  CURRENT_NODE_IP    IP address of the current node (optional, defaults to CURRENT_NODE)"
    echo "  SECOND_NODE_IP     IP address of the second node (optional, defaults to SECOND_NODE)"
    echo
    exit 1
}

# Check if at least two arguments are provided
if [ $# -lt 2 ]; then
    usage
fi

# Assigning variables
CURRENT_NODE=$1
SECOND_NODE=$2
CURRENT_NODE_IP=${3:-$CURRENT_NODE}
SECOND_NODE_IP=${4:-$SECOND_NODE}
CONFIG_FILE="/etc/wazuh-indexer/opensearch.yml"
BACKUP_FILE="./opensearch.yml.bak"

# Backup the original config file
echo "Creating a backup of the original config file..."
cp $CONFIG_FILE $BACKUP_FILE

# Replace values in the config file
echo "Updating configuration..."
sed -i "s/network\.host: \"0\.0\.0\.0\"/network.host: \"${CURRENT_NODE_IP}\"/" $CONFIG_FILE
sed -i "s/node\.name: \"node-1\"/node.name: \"${CURRENT_NODE}\"/" $CONFIG_FILE
sed -i "s/#discovery\.seed_hosts:/discovery.seed_hosts:\n  - \"${CURRENT_NODE_IP}\"\n  - \"${SECOND_NODE_IP}\"/" $CONFIG_FILE
sed -i "s/cluster\.initial_master_nodes:\n-\"node-1\"/cluster.initial_master_nodes:\n- ${CURRENT_NODE}\n- ${SECOND_NODE}/" $CONFIG_FILE
sed -i ':a;N;$!ba;s/plugins\.security\.nodes_dn:\n- "CN=node-1,OU=Wazuh,O=Wazuh,L=California,C=US"/plugins.security.nodes_dn:\n- "CN='"${CURRENT_NODE}"',OU=Wazuh,O=Wazuh,L=California,C=US"\n- "CN='"${SECOND_NODE}"',OU=Wazuh,O=Wazuh,L=California,C=US"/' $CONFIG_FILE

if [ $? -eq 0 ]; then
    echo "Configuration updated successfully. Backup created at ${BACKUP_FILE}"
else
    echo "Error updating configuration."
fi

# Directory for certificates
CERT_DIR="/etc/wazuh-indexer/certs"

# Extract certificates
echo "Creating certificates directory and extracting certificates..."
mkdir -p $CERT_DIR
tar -xf ./wazuh-certificates.tar -C $CERT_DIR ./$CURRENT_NODE.pem ./$CURRENT_NODE-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem

if [ $? -ne 0 ]; then
    echo "Error extracting certificates."
    exit 1
fi

# Move and set permissions for certificates
echo "Moving and setting permissions for certificates..."
mv -n $CERT_DIR/$CURRENT_NODE.pem $CERT_DIR/indexer.pem
mv -n $CERT_DIR/$CURRENT_NODE-key.pem $CERT_DIR/indexer-key.pem
chmod 500 $CERT_DIR
chmod 400 $CERT_DIR/*
chown -R wazuh-indexer:wazuh-indexer $CERT_DIR

if [ $? -eq 0 ]; then
    echo "Certificates configured successfully."
else
    echo "Error configuring certificates."
fi

    • Check wazuh-indexer service

      sudo bash ./03_check_service.sh

Starting wazuh-indexer service...
Created symlink /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service → /usr/lib/systemd/system/wazuh-indexer.service.
wazuh-indexer service is running.
Stopping wazuh-indexer service...
wazuh-indexer service stopped successfully.
Restarting wazuh-indexer service...
wazuh-indexer service is running.
      Test script 
      #!/bin/bash

# SPDX-License-Identifier: Apache-2.0
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.

# Function to check the status of the wazuh-indexer service
check_service_is_running() {
    systemctl is-active --quiet wazuh-indexer
    if [ $? -eq 0 ]; then
        echo "wazuh-indexer service is running."
    else
        echo "Error: wazuh-indexer service is not running." >&2
        exit 1
    fi
}

# Start wazuh-indexer service
echo "Starting wazuh-indexer service..."
systemctl daemon-reload
systemctl enable wazuh-indexer
systemctl start wazuh-indexer

# Check if the service is running
check_service_is_running

# Stop wazuh-indexer service
echo "Stopping wazuh-indexer service..."
systemctl stop wazuh-indexer

# Check if the service is stopped
systemctl is-active --quiet wazuh-indexer
if [ $? -ne 0 ]; then
    echo "wazuh-indexer service stopped successfully."
else
    echo "Error: Failed to stop wazuh-indexer service." >&2
    exit 1
fi

# Restart wazuh-indexer service
echo "Restarting wazuh-indexer service..."
systemctl restart wazuh-indexer

# Check if the service is running after restart
check_service_is_running

    • Extra manual check

      systemctl status wazuh-indexer
● wazuh-indexer.service - wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2024-10-19 01:43:50 UTC; 4min 27s ago
       Docs: https://documentation.wazuh.com
   Main PID: 9869 (java)
      Tasks: 44 (limit: 2220)
     Memory: 1.2G
        CPU: 19.813s
     CGroup: /system.slice/wazuh-indexer.service
             └─9869 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Do>

  4. A Wazuh Indexer cluster can be formed.

    • Run all the previous scripts on the other indexer system 
      GITHUB_TOKEN=<MY_GH_TOKEN> bash ./00_check_package.sh 11406244332 5.0.0

Fetching artifacts list...
Checking wazuh-indexer_5.0.0-0_amd64.deb package is generated for workflow run 11406244332
Wazuh indexer package built successfully.
[ Artifact ID: 2074853568 ]
      GITHUB_TOKEN=<MY_GH_TOKEN> bash ./01_check_installation.sh 2074853568 5.0.0

Downloading wazuh-indexer package from GitHub artifactory...
(It could take a couple minutes)
Package downloaded successfully
Decompressing wazuh-indexer package...
Archive:  ./package.zip
  inflating: wazuh-indexer_5.0.0-0_amd64.deb
Package decompressed
Installing wazuh-indexer package...
Selecting previously unselected package wazuh-indexer.
(Reading database ... 76346 files and directories currently installed.)
Preparing to unpack wazuh-indexer_5.0.0-0_amd64.deb ...
Running Wazuh Indexer Pre-Installation Script
Unpacking wazuh-indexer (5.0.0-0) ...
Setting up wazuh-indexer (5.0.0-0) ...
Running Wazuh Indexer Post-Installation Script
### NOT starting on installation, please execute the following statements to configure wazuh-indexer service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable wazuh-indexer.service
### You can start wazuh-indexer service by executing
 sudo systemctl start wazuh-indexer.service
Package installed successfully.
      sudo bash ./02_apply_certificates.sh node-2 node -1 192.168.56.11 192.168.56.10

Creating a backup of the original config file...
Updating configuration...
Configuration updated successfully. Backup created at ./opensearch.yml.bak
Creating certificates directory and extracting certificates...
Moving and setting permissions for certificates...
Certificates configured successfully.
      sudo bash ./03_check_service.sh

Starting wazuh-indexer service...
Synchronizing state of wazuh-indexer.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable wazuh-indexer
Created symlink /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service → /lib/systemd/system/wazuh-indexer.service.
wazuh-indexer service is running.
Stopping wazuh-indexer service...
wazuh-indexer service stopped successfully.
Restarting wazuh-indexer service...
wazuh-indexer service is running.

    • Initialize the cluster

      sudo bash ./04_cluster_initialization.sh 192.168.56.10
Initializing wazuh-indexer cluster...
Checking cluster status...
Indexer Status:
  Node Name: node-1
  Cluster Name: wazuh-cluster
  Version Number: 7.10.2
Verifying the Wazuh indexer nodes...
Nodes:
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles                                        cluster_manager name
192.168.56.10           61          89   0    0.59    0.16     0.10 dimr      cluster_manager,data,ingest,remote_cluster_client *               node-1
192.168.56.11           57          97   0    0.19    0.23     0.15 dimr      cluster_manager,data,ingest,remote_cluster_client -               node-2
Initialization completed successfully.
      Test script 
      #!/bin/bash

# SPDX-License-Identifier: Apache-2.0
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.

# Function to display usage help
usage() {
    echo
    echo "Usage: $0 <CLUSTER_IP> <USER> <PASSWORD>"
    echo
    echo "Parameters:"
    echo "  CLUSTER_IP    IP address of the cluster (default: localhost)"
    echo "  USER          Username for authentication (default: admin)"
    echo "  PASSWORD      Password for authentication (default: admin)"
    echo
    exit 1
}

# Check if curl and jq are installed
if ! command -v curl &> /dev/null || ! command -v jq &> /dev/null; then
    echo "Error: curl and jq must be installed."
    exit 1
fi

# Assigning variables
CLUSTER_IP=${1:-"localhost"}
USER=${2:-"admin"}
PASSWORD=${3:-"admin"}

# Initialize cluster
echo "Initializing wazuh-indexer cluster..."
bash /usr/share/wazuh-indexer/bin/indexer-security-init.sh &> /dev/null

# Check if the initialization was successful
if [ $? -ne 0 ]; then
    echo "Error: Failed to initialize cluster."
    exit 1
fi

# Check the Wazuh indexer status
echo "Checking cluster status..."
RESPONSE=$(curl -s -k -u $USER:$PASSWORD https://$CLUSTER_IP:9200)

# Check if the request was successful
if [ $? -ne 0 ]; then
    echo "Error: Failed to connect to cluster."
    exit 1
fi

# Parse and print the response
INDEXER_NAME=$(echo $RESPONSE | jq -r '.name')
CLUSTER_NAME=$(echo $RESPONSE | jq -r '.cluster_name')
VERSION_NUMBER=$(echo $RESPONSE | jq -r '.version.number')

echo "Indexer Status:"
echo "  Node Name: $INDEXER_NAME"
echo "  Cluster Name: $CLUSTER_NAME"
echo "  Version Number: $VERSION_NUMBER"

# Verify the Wazuh indexer nodes
echo "Verifying the Wazuh indexer nodes..."
NODES_RESPONSE=$(curl -s -k -u $USER:$PASSWORD https://$CLUSTER_IP:9200/_cat/nodes?v)

if [ $? -ne 0 ]; then
    echo "Error: Failed to retrieve Wazuh indexer nodes."
    exit 1
fi

echo "Nodes:"
echo "$NODES_RESPONSE"

echo "Initialization completed successfully."

  5. Wazuh Indexer plugins are installed.

    bash ./05_check_plugins.sh 192.168.56.10 admin admin node-1 node-2

Checking installed plugins on Wazuh indexer nodes...
Checking node node-1...
  wazuh-indexer-command-manager is installed on node-1.
  wazuh-indexer-setup is installed on node-1.
Checking node node-2...
  wazuh-indexer-command-manager is installed on node-2.
  wazuh-indexer-setup is installed on node-2.
All required plugins are installed on all nodes.
    Test script 
    #!/bin/bash

# SPDX-License-Identifier: Apache-2.0
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.

# Function to display usage help
usage() {
    echo
    echo "Usage: $0 <CLUSTER_IP> <USER> <PASSWORD> <NODE_1> <NODE_2> [...]"
    echo
    echo "Parameters:"
    echo "  CLUSTER_IP    IP address of the cluster (default: localhost)"
    echo "  USER          Username for authentication (default: admin)"
    echo "  PASSWORD      Password for authentication (default: admin)"
    echo "  NODE_1        Name of the first node"
    echo "  NODE_2        Name of the second node (add more as needed)"
    echo
    exit 1
}

# Check if curl and jq are installed
if ! command -v curl &> /dev/null || ! command -v jq &> /dev/null; then
    echo "Error: curl and jq must be installed."
    exit 1
fi

# Check if at least four arguments are provided
if [ "$#" -lt 4 ]; then
    usage
fi

# Assigning variables
CLUSTER_IP=${1:-"localhost"}
USER=${2:-"admin"}
PASSWORD=${3:-"admin"}
NODES=${@:4}  # List of nodes passed as arguments starting from the 4th

# Check the installed plugins on each node
REQUIRED_PLUGINS=("wazuh-indexer-command-manager" "wazuh-indexer-setup")
ALL_MISSING_PLUGINS=()

echo "Checking installed plugins on Wazuh indexer nodes..."

for NODE in $NODES; do
    echo "Checking node $NODE..."
    RESPONSE=$(curl -s -k -u $USER:$PASSWORD https://$CLUSTER_IP:9200/_cat/plugins?v | grep $NODE)

    # Check if the request was successful
    if [ $? -ne 0 ]; then
        echo "Error: Failed to connect to Wazuh indexer."
        exit 1
    fi

    MISSING_PLUGINS=()
    for PLUGIN in "${REQUIRED_PLUGINS[@]}"; do
        if echo "$RESPONSE" | grep -q "$PLUGIN"; then
            echo "  $PLUGIN is installed on $NODE."
        else
            MISSING_PLUGINS+=("$PLUGIN")
        fi
    done

    if [ ${#MISSING_PLUGINS[@]} -ne 0 ]; then
        echo "Error: The following required plugins are missing on $NODE:"
        for PLUGIN in "${MISSING_PLUGINS[@]}"; do
            echo "  $PLUGIN"
        done
        ALL_MISSING_PLUGINS+=("${MISSING_PLUGINS[@]}")
    fi
done

if [ ${#ALL_MISSING_PLUGINS[@]} -ne 0 ]; then
    echo "Error: Some nodes are missing required plugins."
    exit 1
fi

echo "All required plugins are installed on all nodes."

  6. The index templates and indices generated by the Setup plugin are created.

    bash ./06_check_setup_plugin.sh 192.168.56.10

Fetching templates from Wazuh indexer cluster...
Validating templates...
  Template index-template-agent is created correctly.
  Error: Template index-template-alerts is missing required field 'agent.groups'.
  Template index-template-fim is created correctly.
  Template index-template-packages is created correctly.
  Template index-template-processes is created correctly.
  Template index-template-system is created correctly.
  Error: Template index-template-vulnerabilities is missing required field 'agent.groups'.
Some templates were not created correctly:
  index-template-alerts
  index-template-vulnerabilities

Fetching indices from Wazuh indexer cluster...
Fetching protected indices from Wazuh indexer cluster...
Validating index patterns...
  Index pattern wazuh-alerts-5.x-* is valid.
  Error: Index pattern wazuh-states-vulnerabilities does not end with '*'.
  Index pattern wazuh-states-inventory-system* is valid.
  Index pattern wazuh-states-inventory-packages* is valid.
  Index pattern .agents* is valid.
  Index pattern wazuh-states-fim* is valid.
  Index pattern wazuh-states-inventory-processes* is valid.
Errors on index-patterns detected:
  wazuh-states-vulnerabilities
    Test script 
    #!/bin/bash

# SPDX-License-Identifier: Apache-2.0
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.

# Function to display usage help
usage() {
    echo
    echo "Usage: $0 <CLUSTER_IP> <USER> <PASSWORD>"
    echo
    echo "Parameters:"
    echo "  CLUSTER_IP    (Optional) IP address of the cluster (default: localhost)"
    echo "  USER          (Optional) Username for authentication (default: admin)"
    echo "  PASSWORD      (Optional) Password for authentication (default: admin)"
    echo
    exit 1
}

# Check if curl and jq are installed
if ! command -v curl &> /dev/null || ! command -v jq &> /dev/null; then
    echo "Error: curl and jq must be installed."
    exit 1
fi

# Assigning variables
CLUSTER_IP=${1:-"localhost"}
USER=${2:-"admin"}
PASSWORD=${3:-"admin"}

# List of expected items
EXPECTED_TEMPLATES=("index-template-agent" "index-template-alerts" "index-template-fim" "index-template-packages"
    "index-template-processes" "index-template-system" "index-template-vulnerabilities")

# Fetch the templates
echo "Fetching templates from Wazuh indexer cluster..."
TEMPLATES_RESPONSE=$(curl -s -k -u $USER:$PASSWORD https://$CLUSTER_IP:9200/_cat/templates?v)
# Check if the request was successful
if [ $? -ne 0 ]; then
    echo "Error: Failed to fetch templates."
    exit 1
fi

# Validate the templates
MISSING_TEMPLATES=()
echo "Validating templates..."
for TEMPLATE in "${EXPECTED_TEMPLATES[@]}"; do
    if echo "$TEMPLATES_RESPONSE" | grep -q "$TEMPLATE"; then
        # Fetch the template info to check for required fields
        TEMPLATE_INFO=$(curl -s -k -u $USER:$PASSWORD https://$CLUSTER_IP:9200/_template/$TEMPLATE)
        if ! echo "$TEMPLATE_INFO" | jq -e '.[] | .mappings.properties.agent.properties.id' > /dev/null; then
            echo "  Error: Template $TEMPLATE is missing required field 'agent.id'."
            MISSING_TEMPLATES+=("$TEMPLATE")
        elif ! echo "$TEMPLATE_INFO" | jq -e '.[] | .mappings.properties.agent.properties.groups' > /dev/null; then
            echo "  Error: Template $TEMPLATE is missing required field 'agent.groups'."
            MISSING_TEMPLATES+=("$TEMPLATE")
        else
            echo "  Template $TEMPLATE is created correctly."
        fi
    else
        MISSING_TEMPLATES+=("$TEMPLATE")
        echo "  Error: Template $TEMPLATE is missing."
    fi
done

if [ ${#MISSING_TEMPLATES[@]} -ne 0 ]; then
    echo "Some templates were not created correctly:"
    for TEMPLATE in "${MISSING_TEMPLATES[@]}"; do
        echo "  $TEMPLATE"
    done
    echo
else
    echo "All templates are correctly created."
    echo
fi

# Fetch the indices
echo "Fetching indices from Wazuh indexer cluster..."
INDICES_RESPONSE=$(curl -s -k -u $USER:$PASSWORD https://$CLUSTER_IP:9200/_cat/indices?v)
# Check if the request was successful
if [ $? -ne 0 ]; then
    echo "Error: Failed to fetch indices."
    exit 1
fi

# Fetch the protected indices
echo "Fetching protected indices from Wazuh indexer cluster..."
PROTECTED_RESPONSE=$(curl -s -k -u $USER:$PASSWORD https://$CLUSTER_IP:9200/_cat/indices/.*?v)
# Check if the request was successful
if [ $? -ne 0 ]; then
    echo "Error: Failed to fetch indices."
    exit 1
fi

# Validate index patterns
echo "Validating index patterns..."
INVALID_PATTERNS=()
while read -r line; do
    TEMPLATE_NAME=$(echo $line | awk '{print $1}')
    INDEX_PATTERN=$(echo $line | awk '{print $2}' | tr -d '[]')

    if [[ $INDEX_PATTERN == .* ]]; then
        TO_MATCH=$PROTECTED_RESPONSE
    else
        TO_MATCH=$INDICES_RESPONSE
    fi

    # Check if index pattern ends with '*'
    if [[ $INDEX_PATTERN != *\* ]]; then
        echo "  Error: Index pattern $INDEX_PATTERN does not end with '*'."
        INVALID_PATTERNS+=("$INDEX_PATTERN")
        continue
    fi

    if echo "$TO_MATCH" | grep -q "$INDEX_PATTERN"; then
        echo "  Index pattern $INDEX_PATTERN is valid."
    else
        INVALID_PATTERNS+=("$INDEX_PATTERN")
        echo "  Error: Index pattern $INDEX_PATTERN not found in indices for template $TEMPLATE_NAME."
    fi
done <<< "$(echo "$TEMPLATES_RESPONSE" | tail -n +2)"  # Skip header line

if [ ${#INVALID_PATTERNS[@]} -ne 0 ]; then
    echo "Errors on index-patterns detected:"
    for PATTERN in "${INVALID_PATTERNS[@]}"; do
        echo "  $PATTERN"
    done
    echo
else
    echo "Index-patterns validated successfully."
fi

Caution

  • Index 'wazuh-states-vulnerabilities' does not end with '*'.
  • Template 'index-template-alerts' is missing required field 'agent.groups'.
  • Template 'index-template-vulnerabilities' is missing required field 'agent.groups'.

  1. Command manager API works, index creates, data indexed

    bash ./07_check_command_manager_plugin.sh 192.168.56.10
Command created successfully.
Validating .commands index is created...
Index created correctly.
Validate the command is created
Validation successful: The command was created and found in the search results.
    Test script 
    #!/bin/bash
# SPDX-License-Identifier: Apache-2.0
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.

# Assigning variables
CLUSTER_IP=${1:-"localhost"}
USERNAME=${2:-"admin"}
PASSWORD=${3:-"admin"}

# Check for curl command
if ! command -v curl &> /dev/null
then
    echo "curl command could not be found"
    exit
fi

COMMANDS_INDEX=".commands"
SRC="Engine"
USR="TestUser"
TRG_ID="TestTarget"
ARG="/test/path/fake/args"
BODY="{
  \"source\": \"$SRC\",
  \"user\": \"$USR\",
  \"target\": {
    \"id\": \"$TRG_ID\",
    \"type\": \"agent\"
  },
  \"action\": {
    \"name\": \"restart\",
    \"args\": [
      \"$ARG\"
    ],
    \"version\": \"v4\"
  },
  \"timeout\": 30
}"

# Send the POST request
RESPONSE=$(curl -s -k -u $USERNAME:$PASSWORD -X POST https://$CLUSTER_IP:9200/_plugins/_command_manager/commands -H 'accept: */*' -H 'Content-Type: application/json' -d "$BODY")

# Check if the request was successful
if [ $? -ne 0 ]; then
    echo "Error: Failed to create command."
    exit 1
fi
echo "Command created successfully."

# Fetch the indices
echo "Validating .commands index is created..."
INDICES_RESPONSE=$(curl -s -k -u $USERNAME:$PASSWORD https://$CLUSTER_IP:9200/_cat/indices/.*?v)
if [ $? -ne 0 ]; then
    echo "Error: Failed to fetch indices."
    exit 1
fi
if echo "$INDICES_RESPONSE" | grep -q "$COMMANDS_INDEX"; then
    echo "Index created correctly."
else
    echo "Error: Index is not created."
    exit 1
fi

echo "Validate the command is created"
# Validate the command was created
SEARCH_RESPONSE=$(curl -s -k -u $USERNAME:$PASSWORD https://$CLUSTER_IP:9200/.commands/_search)
# Check if the request was successful
if [ $? -ne 0 ]; then
    echo "Error: Failed to search for the command."
    exit 1
fi

# Extract and validate specific fields
COMMAND_FOUND=$(echo "$SEARCH_RESPONSE" | jq -r '.hits.hits[] | select(._source.command.source == "Engine" and ._source.command.user == "TestUser" and ._source.command.target.id == "TestTarget" and ._source.command.action.args[0] == "/test/path/fake/args")')

if [ -n "$COMMAND_FOUND" ]; then
    echo "Validation successful: The command was created and found in the search results."
else
    echo "Error: The command was not found in the search results."
    exit 1
fi

  2. Wazuh indexer can be uninstalled
    apt

    sudo bash 08_check_uninstall.sh 

Uninstalling Wazuh Indexer...
Validating Wazuh Indexer removal...
Wazuh Indexer packages removed.
Wazuh Indexer service removed.
Wazuh Indexer uninstallation and validation completed successfully.

    rpm

    sudo bash 08_check_uninstall.sh 

Uninstalling Wazuh Indexer...
Validating Wazuh Indexer removal...
Wazuh Indexer packages removed.
Wazuh Indexer service removed.
Wazuh Indexer uninstallation and validation completed successfully.
    Test script 
    #!/bin/bash
# SPDX-License-Identifier: Apache-2.0
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.

# Detect package manager
if command -v apt-get &> /dev/null; then
    PKG_MANAGER="apt-get"
elif command -v yum &> /dev/null; then
    PKG_MANAGER="yum"
else
    echo "Unsupported package manager. Please use a system with apt-get or yum."
    exit 1
fi

# Uninstall Wazuh Indexer
echo "Uninstalling Wazuh Indexer..."
sudo systemctl stop wazuh-indexer > /dev/null 2>&1
sudo systemctl disable wazuh-indexer > /dev/null 2>&1

if [ "$PKG_MANAGER" == "apt-get" ]; then
    sudo apt-get remove --purge wazuh-indexer -y > /dev/null 2>&1
elif [ "$PKG_MANAGER" == "yum" ]; then
    sudo yum remove wazuh-indexer -y > /dev/null 2>&1
fi

# Validate removal
echo "Validating Wazuh Indexer removal..."

# Check for remaining files and directories
if [ "$PKG_MANAGER" == "apt-get" ]; then
    if dpkg -l | grep wazuh-indexer > /dev/null 2>&1; then
        echo "Error: Wazuh Indexer packages still present."
        exit 1
    else
        echo "Wazuh Indexer packages removed."
    fi
elif [ "$PKG_MANAGER" == "yum" ]; then
    if rpm -qa | grep wazuh-indexer > /dev/null 2>&1; then
        echo "Error: Wazuh Indexer packages still present."
        exit 1
    else
        echo "Wazuh Indexer packages removed."
    fi
fi

# Check for remaining services
if systemctl list-units --full -all | grep wazuh-indexer.service > /dev/null 2>&1; then
    echo "Error: Wazuh Indexer service still present."
    exit 1
else
    echo "Wazuh Indexer service removed."
fi

echo "Wazuh Indexer uninstallation and validation completed successfully."

@QU3B1M QU3B1M linked a pull request Oct 21, 2024 that will close this issue
Open
3 tasks
@QU3B1M QU3B1M linked a pull request Oct 21, 2024 that will close this issue
Add bash scripts for MVP validation tests #482
Open
3 tasks
@QU3B1M QU3B1M mentioned this issue Oct 21, 2024
Closed
@QU3B1M
Copy link
Member

QU3B1M commented Oct 21, 2024

Created issue to track the detected bug:

@QU3B1M QU3B1M mentioned this issue Oct 21, 2024
Merged
@AlexRuiz7
Copy link
Member Author

Added a couple of tests more.

@QU3B1M
Copy link
Member

QU3B1M commented Oct 22, 2024

Added new validations to 06_check_setup_plugin.sh script, the comment #478 (comment) was updated with the new script execution.

@QU3B1M
Copy link
Member

QU3B1M commented Oct 22, 2024 •
edited
Loading

The scripts were executed on the package generated with the fix checking the index-pattern is fixed and working, but the agent.groups field is not correct.

bash ./06_check_setup_plugin.sh 192.168.56.10

Fetching templates from Wazuh indexer cluster...
Validating templates...
  Template index-template-agent is created correctly.
  Error: Template index-template-alerts is missing required field 'agent.groups'.
  Template index-template-fim is created correctly.
  Template index-template-packages is created correctly.
  Template index-template-processes is created correctly.
  Template index-template-system is created correctly.
  Error: Template index-template-vulnerabilities is missing required field 'agent.groups'.
Some templates were not created correctly:
  index-template-alerts
  index-template-vulnerabilities

Fetching indices from Wazuh indexer cluster...
Fetching protected indices from Wazuh indexer cluster...
Validating index patterns...
  Index pattern wazuh-alerts-5.x-* is valid.
  Index pattern wazuh-states-vulnerabilities* is valid.
  Index pattern wazuh-states-inventory-system* is valid.
  Index pattern wazuh-states-inventory-packages* is valid.
  Index pattern .agents* is valid.
  Index pattern wazuh-states-fim* is valid.
  Index pattern wazuh-states-inventory-processes* is valid.
Index-patterns validated successfully.

We can see the field on the vulnerabilities template the field is not present in the mappings

        "mappings": {
            "dynamic": "strict",
            "date_detection": false,
            "properties": {
                "agent": {
                    "properties": {
                        "build": {
                            "properties": {
                                "original": {
                                    "ignore_above": 1024,
                                    "type": "keyword"
                                }
                            }
                        },
                        "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                        },
                        "id": {
                            "ignore_above": 1024,
                            "type": "keyword"
                        },
                        "ephemeral_id": {
                            "ignore_above": 1024,
                            "type": "keyword"
                        },
                        "type": {
                            "ignore_above": 1024,
                            "type": "keyword"
                        },
                        "version": {
                            "ignore_above": 1024,
                            "type": "keyword"
                        }
                    }
                },

In the case of the index-template-alerts the body of the template is different so I'm not sure if the

@QU3B1M QU3B1M mentioned this issue Oct 23, 2024
Closed
@QU3B1M
Copy link
Member

QU3B1M commented Oct 23, 2024 •
edited
Loading

The Setup plugin validations executed on the package with the fix passed successfully

bash ./06_check_setup_plugin.sh 192.168.68.5
Fetching templates from Wazuh indexer cluster...
Validating templates...
  Template index-template-agent is created correctly.
  Template index-template-alerts is created correctly.
  Template index-template-fim is created correctly.
  Template index-template-packages is created correctly.
  Template index-template-processes is created correctly.
  Template index-template-system is created correctly.
  Template index-template-vulnerabilities is created correctly.
All templates are correctly created.

Fetching indices from Wazuh indexer cluster...
Fetching protected indices from Wazuh indexer cluster...
Validating index patterns...
  Index pattern wazuh-alerts-5.x-* is valid.
  Index pattern wazuh-states-vulnerabilities* is valid.
  Index pattern wazuh-states-inventory-system* is valid.
  Index pattern wazuh-states-inventory-packages* is valid.
  Index pattern .agents* is valid.
  Index pattern wazuh-states-fim* is valid.
  Index pattern wazuh-states-inventory-processes* is valid.
Index-patterns validated successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@QU3B1M QU3B1M

Labels
level/task Task issue type/test Test issue
Projects
Release 5.0.0
Status: In progress
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add bash scripts for MVP validation tests wazuh/wazuh-indexer
2 participants
@AlexRuiz7 @QU3B1M