Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tests fail during feed downloads and others finished with setup errors in the VD suite #4370

Closed
roronoasins opened this issue Jul 28, 2023 · 6 comments · Fixed by wazuh/wazuh-documentation#6435, #4496 or wazuh/wazuh#18783
Closed

Tests fail during feed downloads and others finished with setup errors in the VD suite #4370

roronoasins opened this issue Jul 28, 2023 · 6 comments · Fixed by wazuh/wazuh-documentation#6435, #4496 or wazuh/wazuh#18783
Assignees
@mauromalara @mateocervilla
Labels
level/task Task issue qa_known Issues that are already known by the QA team test/integration type/bug

Comments

@roronoasins
Copy link

roronoasins commented Jul 28, 2023 •
edited by mauromalara
Loading

Description

During the 4.5.0 Alpha 1 Integration testing, some Vulnerability tests failed:

  • test_vulnerability_detector/test_feeds/test_download_feeds.py
    • SUSE Linux Enterprise Server 12
    • SUSE Linux Enterprise Server 15
    • SUSE Linux Enterprise Desktop 15

The error reported is a TimeoutError when monitoring the log (.*) feed finished log, whereas the expected values are:

  • SUSE Linux Enterprise Server 12
  • SUSE Linux Enterprise Server 15
  • SUSE Linux Enterprise Desktop 15

This issue needs to be investigated and addressed to ensure the proper functioning of the Vulnerability tests for the mentioned distributions. It's possible that the timeout duration does not align with the download time required for the mentioned feeds.
@roronoasins roronoasins mentioned this issue Jul 28, 2023
Closed
2 tasks
@wazuhci wazuhci moved this to Triage in Release 4.5.0 Jul 28, 2023
@wazuhci wazuhci moved this from Triage to Backlog in Release 4.5.0 Jul 28, 2023
@mauromalara mauromalara self-assigned this Jul 28, 2023
@wazuhci wazuhci moved this from Backlog to In progress in Release 4.5.0 Jul 28, 2023
@mauromalara mauromalara changed the title SUSE Download Feed tests failing with TimeoutError Some tests fail with TimeoutError and other finished with errors in the VD suite Jul 28, 2023
@mauromalara
Copy link
Contributor

mauromalara commented Jul 28, 2023 •
edited
Loading

Evidence of the test

HTML-report.zip
TXT-report.txt

Analysis of errors in the test's setup

There are also some tests with errors:

ERROR test_vulnerability_detector/test_feeds/test_validate_feed_content.py::test_validate_xml_feed_content[Red Hat Enterprise Linux0]
ERROR test_vulnerability_detector/test_feeds/test_validate_feed_content.py::test_validate_xml_feed_content[Canonical Focal]
ERROR test_vulnerability_detector/test_feeds/test_validate_feed_content.py::test_validate_xml_feed_content[Canonical Bionic]
ERROR test_vulnerability_detector/test_feeds/test_validate_feed_content.py::test_validate_xml_feed_content[Canonical Xenial]
ERROR test_vulnerability_detector/test_feeds/test_validate_feed_content.py::test_validate_xml_feed_content[Canonical Trusty]

The test has a fixture to download the feed before the test starts and then it decompress the data from the feed. But, when the provider has changed the URL or the file is not available, the test will fail as it tries to decompress HTML data, for instance:

data = b'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>N...server.</p>\n<hr>\n<address>Apache/2.4.29 (Ubuntu) Server at people.canonical.com Port 443</address>\n</body></html>\n'

    def decompress(data):
        """Decompress a block of data.
    
        For incremental decompression, use a BZ2Decompressor object instead.
        """
        results = []
        while data:
            decomp = BZ2Decompressor()
            try:
>               res = decomp.decompress(data)
E               OSError: Invalid data stream

Conclusion

That test is flaky because it does not handle this kind of error at startup preventing the test from being executed. The test should control this kind of scenario and let the test continue its normal execution.

⚠️ Anyway, I don't think we should download the feed here. I think that this is not something related to our product and we are trying to verify Wazuh's capabilities.

@mauromalara
Copy link
Contributor

mauromalara commented Jul 28, 2023 •
edited
Loading

Evidence of the test

HTML-report.zip
TXT-report.txt

Analysis of failures in the test

FAILED test_vulnerability_detector/test_feeds/test_download_feeds.py::test_download_feeds[SUSE Linux Enterprise Server 12]
FAILED test_vulnerability_detector/test_feeds/test_download_feeds.py::test_download_feeds[SUSE Linux Enterprise Server 15]
FAILED test_vulnerability_detector/test_feeds/test_download_feeds.py::test_download_feeds[SUSE Linux Enterprise Desktop 15]
  • Test timeout: 6 minutes
  • Wazuh manager timeout: 5 minutes

So, when I tried to get the feed through wget:

$ wget https://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.12.xml
--2023-07-28 16:54:35--  https://ftp.suse.com/pub/projects/security/oval/suse.linux.enterprise.server.12.xml
Resolving ftp.suse.com (ftp.suse.com)... 195.135.221.132
Connecting to ftp.suse.com (ftp.suse.com)|195.135.221.132|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 185753599 (177M) [text/xml]
Saving to: ‘suse.linux.enterprise.server.12.xml’

      suse.linux.enterprise.s   0%[                                                 ] 447.49K  70.4KB/s    eta 42m 49s

The ETA is above the test and manager timeouts. So, the test is failing as expected. Also, the manager is detecting this timeout and reporting it in the logs:

DEBUG: CURL ERROR: Operation timed out after 300000 milliseconds with 72121841 out of 185753599 bytes received

Conclusion

The failures are not related to the test itself or the product but to SUSE's servers.

⚠️ Anyway, I don't think this test is well designed as we cannot infinitely wait for the vendor to return their vulnerability information. I think that this test should verify that Wazuh is able to download a feed, but not a production one, and also the test should verify that Wazuh can handle timeouts or format errors.

We cannot say that a test fails because a provider has a problem. Because we want to test our product.

@mauromalara mauromalara changed the title Some tests fail with TimeoutError and other finished with errors in the VD suite Tests fail with TimeoutError and other finished with errors in the VD suite Jul 28, 2023
@mauromalara mauromalara changed the title Tests fail with TimeoutError and other finished with errors in the VD suite Tests fail while downloading feeds and other finished with setup errors in the VD suite Jul 28, 2023
@wazuhci wazuhci moved this from In progress to Pending review in Release 4.5.0 Jul 28, 2023
@davidjiglesias davidjiglesias changed the title Tests fail while downloading feeds and other finished with setup errors in the VD suite Tests fail during feed downloads and others finished with setup errors in the VD suite Jul 31, 2023
@davidjiglesias
Copy link
Member

Based on the conclusions above from @mauromalara I will move this issue to @wazuh/core for them to fix or refactor accordingly.

@juliamagan juliamagan mentioned this issue Aug 2, 2023
Closed
@Deblintrake09 Deblintrake09 mentioned this issue Aug 4, 2023
Closed
1 task
@Deblintrake09
Copy link
Contributor

This issue errorss where present in 4.5.1 - Alpha 1 wazuh/wazuh#18286

@Deblintrake09 Deblintrake09 mentioned this issue Aug 7, 2023
Closed
1 task
@damarisg damarisg mentioned this issue Aug 9, 2023
Closed
@Rebits Rebits mentioned this issue Aug 14, 2023
Closed
1 task
@damarisg damarisg mentioned this issue Aug 15, 2023
Closed
@Deblintrake09 Deblintrake09 mentioned this issue Aug 18, 2023
Closed
1 task
@damarisg damarisg mentioned this issue Aug 18, 2023
Closed
@mateocervilla mateocervilla mentioned this issue Aug 24, 2023
Closed
@BelenValdivia BelenValdivia mentioned this issue Aug 28, 2023
Closed
1 task
@mauromalara mauromalara added the qa_known Issues that are already known by the QA team label Aug 29, 2023
@mateocervilla mateocervilla self-assigned this Sep 6, 2023
This was linked to pull requests Sep 6, 2023
Update SUSE OVAL URL for offline update wazuh/wazuh-documentation#6435
Merged
Update ITs URL for SUSE OVAL #4496
Merged
Update SUSE OVAL URL to download compressed feeds wazuh/wazuh#18783
Merged
@MarcelKemp
Copy link
Member

This issue is related to the following development:

So they should be fixed after completion.

@rauldpm rauldpm mentioned this issue Sep 11, 2023
Closed
10 tasks
@mauromalara mauromalara mentioned this issue Sep 12, 2023
Closed
2 tasks
@MarcelKemp
Copy link
Member

@fcaffieri fcaffieri mentioned this issue Sep 26, 2023
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mauromalara mauromalara

@mateocervilla mateocervilla

Labels
level/task Task issue qa_known Issues that are already known by the QA team test/integration type/bug
Projects
None yet
Milestone
No milestone
7 participants
@Deblintrake09 @roronoasins @MarcelKemp @mauromalara @davidjiglesias @mateocervilla @juliamagan