Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a troubleshooting guide for vulnerability detector module #8014

Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions source/user-manual/capabilities/vulnerability-detection/FAQ.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
.. Copyright (C) 2015, Wazuh, Inc.

.. meta::
:description: Learn how to solve common issues with the Vulnerability Detection module in this section of the documentation.

Frequently Asked Questions
==========================

This guide presents the solution to common problems that might occur when using the Vulnerability Detection module.
These issues are related to the module's configuration, installation, and operation and aren't known issues that require an upgrade to be fixed.

Communication problem between the Wazuh manager and the Wazuh Indexer
---------------------------------------------------------------------

.. list-table:: Issue description
:widths: 15 50

* - Problem
- There are no vulnerabilities reported in Wazuh Dashboard or the `wazuh-states-vulnerabilities-*` index isn't created. It's also possible to find messages like `IndexerConnector initialization failed for index 'X', retrying until the connection is successful` in the manager's logs.
* - Cause
- A misconfiguration of the vulnerability detector, indexer connector or wazuh-keystore components is the most common cause.


Known configuration/environment problems
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- Make sure that the wazuh-keystore has been properly updated with the right admin credentials (username and password). See :ref:`wazuh-keystore <wazuh-keystore>` for more details.
- Verify the name and location of the certificates configured under the indexer connector section in the manager's configuration file. The host IP/FQDN must match the ones in the certificate used.

The related configuration block is the following, the `<host>` and `<ssl>` sections misconfiguration can cause the connection problem:

.. code-block:: xml
:emphasize-lines: 4,8,10,11

<indexer>
<enabled>yes</enabled>
<hosts>
<host>https://0.0.0.0:9200</host>
</hosts>
<ssl>
<certificate_authorities>
<ca>/etc/filebeat/certs/root-ca.pem</ca>
</certificate_authorities>
<certificate>/etc/filebeat/certs/filebeat.pem</certificate>
<key>/etc/filebeat/certs/filebeat-key.pem</key>
</ssl>
</indexer>


The connection with the Wazuh Indexer can be verified with a `curl` command from the Wazuh manager. Replace the credentials and certificate paths with the ones from the configuration:

.. code-block:: console

# curl --cacert <ROOT_CA> --cert CERTIFICATE_PEM<> --key <CERTIFICATE_KEY> -u <USER>:<PASS> -XGET https://<INDEXER_IP>:9200/_cluster/health

If this command isn't successful, the vulnerability detector module won't be able to connect to the Indexer either.
To verify if the certificates are the problem, we can skip the certificate verification in the request with the `-k` option:

.. code-block:: console

# curl -k -u <USER>:<PASS> -XGET https://<INDEXER_IP>:9200/_cluster/health

Once the connection is successful, the result is similar to this one:

.. code-block:: json

{
"cluster_name": "opensearch",
"status": "green",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"discovered_master": true,
"discovered_cluster_manager": true,
"active_primary_shards": 9,
"active_shards": 9,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100.0
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

.. meta::
:description: The Vulnerability Detection module detects vulnerabilities in applications installed on the endpoints. Learn more about this capability in this section.

Vulnerability detection
=======================

Expand All @@ -23,3 +23,4 @@ The Wazuh Vulnerability Detection module helps users discover vulnerabilities in
how-it-works
configuring-scans
offline-update
troubleshooting
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
.. Copyright (C) 2015, Wazuh, Inc.

.. meta::
:description: Learn how to solve common issues with the Vulnerability Detection module in this section of the documentation.

Known issues
============

This section covers known issues that might occur when using the Vulnerability Detection module.
It also describes the version that contains the fix, and the remediation plan when the upgrade isn't possible.

Dangling entries found in the vulnerabilities dashboard
-------------------------------------------------------

.. list-table:: Issue description
:widths: 15 50

* - Problem
- There are vulnerabilities reported from agents that were removed from the environment. This issue also presents as vulnerabilities reported about packages that are no longer installed (removed or upgraded).
* - Affected versions
- Wazuh 4.8.0 and later.
* - Cause
- The internal mechanism that handled the agent removal event misses the event and it isn't processed. This can happen for example when there are queued scan events and a content update triggers a re-scan.
* - Fix available
- Wazuh 4.10.0 has a fix that prevents this from happening again. Now an agent removal event won't be queued but be processed right away. This avoids the dangling entries in the vulnerabilities dashboard (https://github.com/wazuh/wazuh/pull/26232). Nevertheless, the upgrade to this version won't remove the prior dangling entries, so a cleanup should be applied manually.

Known configuration/environment issues that cause a similar behavior
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To confirm whether you are affected by this issue, first, some common causes must be discarded.

- If the problem is related to packages that are no longer installed, it's required to confirm the agent is active and reporting. Verify also in the agent's logs that there aren't error/warning messages related to the connection with the manager. Make sure the package isn't reported in the agents' inventory (if it is, there might be a problem with `syscollector` instead).
- The manager is in charge of updating the vulnerabilities inventory. Confirm the manager's logs don't show any error/warning messages related to the connection with Wazuh Indexer, for example: `No available server` or `Failed to sync agent 'X' with the indexer`.
- The Indexer cluster status is healthy. You can query this like `GET _cluster/health`, and it should be 'green' for Wazuh v4.8.0-v4.9.0 and 'green' or 'yellow' for Wazuh v4.9.1 and later (https://github.com/wazuh/wazuh/pull/25417).
- Some time has passed since the agent was removed or the package was upgraded. When an agent updates its inventory, it's required to wait for the next :doc:`Syscollector </user-manual/capabilities/system-inventory/configuration>` scan before the changes are reported.

First, correct any of the situations described above before proceeding with the cleanup.

Remediation plan
^^^^^^^^^^^^^^^^

If any of the known causes are confirmed, or the dangling entries still are present after the upgrade, it's required to follow these steps to reset the module's data.

.. note::

In the case of a cluster setup, the manager steps must be applied to each node.

1. Stop the Wazuh manager.
2. Disable the Vulnerability Detector module:

.. code-block:: xml

<vulnerability-detection>
<enabled>no</enabled>
<!-- Other configurations -->
</vulnerability-detection>

3. Remove the state DBs. They keep the agent's vulnerabilities data, queued events, etc:

.. code-block:: console

# rm -rf /var/ossec/queue/vd/inventory/
# rm -rf /var/ossec/queue/vd/delayed/
# rm -rf /var/ossec/queue/vd/event/
# rm -rf /var/ossec/queue/indexer/

4. Clean the vulnerabilities index

.. code-block:: console

DELETE wazuh-states-vulnerabilities-*

5. Make sure the index is clean:

.. code-block:: console

GET wazuh-states-vulnerabilities-*/_count

6. Start the Wazuh manager. The module updates its state to disabled.

7. Enable the Vulnerability Detector module:

.. code-block:: xml

<vulnerability-detection>
<enabled>yes</enabled>
<!-- Other configurations -->
</vulnerability-detection>

8. Restart the manager. Now the module will know that it was previously disabled and will trigger a re-scan of all agents. Only valid vulnerabilities will be indexed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
.. Copyright (C) 2015, Wazuh, Inc.

.. meta::
:description: Learn how to solve common issues with the Vulnerability Detection module in this section of the documentation.

Module troubleshooting
======================

This section covers common problems that might occur when using the Vulnerability Detection module.
It also describes known issues, the version that contains the fix, and the workaround to apply.

.. topic:: Contents

.. toctree::
:maxdepth: 2

FAQ
known-issues
2 changes: 2 additions & 0 deletions source/user-manual/reference/tools/wazuh-keystore.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
.. meta::
:description: Tool where sensitive configuration data can be securely stored, including any information that the Wazuh manager daemons/tools need for their work.

.. _wazuh-keystore:

wazuh-keystore
==============

Expand Down
Loading