Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log all pytest output in the Jobflow testing module #5613

Closed
rauldpm opened this issue Jul 25, 2024 · 6 comments · Fixed by #5621
Closed

Log all pytest output in the Jobflow testing module #5613

rauldpm opened this issue Jul 25, 2024 · 6 comments · Fixed by #5621
Assignees
Labels

Comments

@rauldpm
Copy link
Member

rauldpm commented Jul 25, 2024

Description

Testing wazuh/wazuh#24935, it has been observed that the pytest output only appears if an error occurs, we need to change this so we can log the passed tests too

https://github.com/wazuh/wazuh-qa/blob/v4.9.0-alpha3/deployability/modules/testing/testing.py#L86-L102

We need to change the _run_tests method of the testing.py file to the following


    @classmethod
    def _run_tests(cls, test_list: list[str], ansible: Ansible, extra_vars: ExtraVars) -> None:
        """
        Execute the playbooks that runs the tests.

        Args:
            test_list (list[str]): The list of tests to run.
            ansible (Ansible): The Ansible object to run the tests.
            extra_vars (ExtraVars): The extra vars for the tests.
        """
        for test in test_list:
            rendering_var = {**extra_vars, 'test': test}
            template = str(cls._test_template)
            result = ansible.run_playbook(template, rendering_var)
            for event in result.events:
                logger.info(f"{event['stdout']}")
            if result.stats["failures"]:
                for event in result.events:
                    if "fatal" in event['stdout']:
                        raise Exception(f"Test {test} failed with error")

This has been tested with the mentioned testing issue and in the log, the pytest output can be observed

This is going to be treated as a bug as it is not desired to hide testing output

@joaquinsgi
Copy link
Member

joaquinsgi commented Jul 29, 2024

The changes to the _run_tests method in testing.py, in order to view the logs of passed tests, have been applied to a new branch: enhacement/5613-change_run_tests-method.

A new pull request was made to merge the branch with the specified changes with branch 4.9.0: #5621

@wazuhci wazuhci moved this from In progress to In review in Release 4.9.0 Jul 29, 2024
@wazuhci wazuhci moved this from In review to In progress in Release 4.9.0 Jul 29, 2024
@wazuhci wazuhci moved this from In progress to Pending review in Release 4.9.0 Jul 29, 2024
@joaquinsgi joaquinsgi linked a pull request Jul 29, 2024 that will close this issue
@fcaffieri
Copy link
Member

Good job, let's perform some execution from the branch where the changes were made, to ensure that it does not fail due to some indentation, and incorporate the executed test into the associated PR.
On the other hand, since it is a bug, the name of the branch should be bug/5613-change-run_tests-method instead of enhacement/5613-change-run_tests-method since it is not a new feature or enhancement.

@wazuhci wazuhci moved this from Pending review to On hold in Release 4.9.0 Jul 29, 2024
@rauldpm
Copy link
Member Author

rauldpm commented Jul 29, 2024

Testing has been done on the linked issue with the change proposed

@joaquinsgi joaquinsgi linked a pull request Jul 29, 2024 that will close this issue
@joaquinsgi joaquinsgi removed a link to a pull request Jul 29, 2024
@joaquinsgi
Copy link
Member

Updated the branch from enhacement/5613-change-run_tests-method to bug/5613-change-run_tests-method

@fcaffieri
Copy link
Member

LGTM

@wazuhci wazuhci moved this from On hold to Pending final review in Release 4.9.0 Jul 29, 2024
@rauldpm
Copy link
Member Author

rauldpm commented Jul 29, 2024

LGTM

@rauldpm rauldpm closed this as completed Jul 29, 2024
@wazuhci wazuhci moved this from Pending final review to Done in Release 4.9.0 Jul 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants