Research implementation of dependency and parallel tests in the current QA framework #4588
Comments
|
On hold in favor of wazuh/wazuh#19446 |
|
On hold in favor of #4597 |
|
Researched about pytest-xdist and pytest-dependency. |
|
On hold in favor of wazuh/wazuh#19477 |
DescriptionIn the context of the new System tests for Vulnerability Detector module, it is crucial that we explore efficient methods for implementing parallel testing within our testing framework. This necessity arises from our plan to execute these new tests across multiple agents. Given the considerable time required for the proposed test cases, it is imperative to conduct parallel tests for each agent, significantly reducing both the costs and time associated with the entire testing suite. Dependency & ParallelizationDependencyTests should ideally be independent; however, due to the product's requirements, most high-level tests cannot be fully isolated. In the case of Vulnerability Detector tests, the repetitive installation and uninstallation of packages would incur a high cost in terms of time and resources. Therefore, we propose minimizing these operations as much as possible. We have devised the following sequential workflow. To implement this new approach in our environment, we research the possibility of using the import pytest
@pytest.mark.dependency()
def test_one():
assert True
@pytest.mark.dependency(depends=["test_one"])
def test_two():
assert True
@pytest.mark.dependency(depends=["test_one", "test_two"])
def test_three():
assert TrueHowever, it's important to note that test run parallelization, using Example of useBased on the information provided, this approach may not be suitable for our use case. We should consider one of the following alternatives:
def test_example():
check_condition1()
check_condition2()ParallelizationOperation ParallelizationMostly implemented by the new framework, this approach involves not running multiple test cases simultaneously. Instead, we can perform multiple remote operations (log monitoring, service restarts, etc.) across all hosts in the environment simultaneously. This can be implemented concurrently with test parallelization. An example of this approach would be: class WazuhEnv:
...
def restart_agents(self, agent_list=None, parallel=True):
"""Restart a list of agents.
Args:
agent_list (list, optional): Agent list. Defaults to None.
parallel (bool, optional): Parallel execution. Defaults to True.
"""
self.logger.info(f'Restarting agents: {agent_list}')
if parallel:
agent_restart_tasks = self.pool.map(self.restart_agent, agent_list)
else:
for agent in agent_list:
self.restart_agent(agent)
self.logger.info(f'Agents restarted successfully: {agent_list}')
...
env = WazuhEnv('inventory')
env.restart_agents(agent_list=['agent1', 'agent2'])Tests ParallelizationInfrastructure DecouplingIt is desirable for tests not to be hardcoded to a single environment but to be dynamically launched based on the provided inventory. This is beneficial for our product's nature because it allows for more intensive testing on various operating systems during certain testing phases. This is viable because many test cases remain consistent across different versions of the same operating system (e.g., generating a syslog alert is the same for all versions of Windows Server). Additionally, this approach facilitates the integration of new environments into the testing process. This can be easily implemented using the current approach, as we obtain the inventory as a parameter and can launch tests dynamically: def pytest_addoption(parser):
parser.addoption("--inventory", action="store", default=None, help="Inventory")
@pytest.fixture(scope="session")
def environment(pytestconfig):
return WazuhEnvironmentHandler(pytestconfig.getoption("inventory"))
def test_e2e(environment):
# Here we can perform operations in the environment based on the data provided by the environmentHowever, there is an issue with parameterization. We need to parallelize tests for each agent, which means generating multiple test cases based on the provided inventory. After some investigation, we found a way to achieve this: def pytest_generate_tests(metafunc):
hosts = get_hosts_from_inventory(metafunc.config.option.inventory, 'agent')
metafunc.parametrize("agents", hosts, scope='session')This hook will parameterize each test for each agent, resulting in test cases like:
Pytest-XdistWe can implement parallelization using the pytest-xdist plugin here. This plugin provides an easy way to parallelize multiple tests: def test_addition():
assert 1 + 2 == 3
def test_subtraction():
assert 4 - 2 == 2
def test_multiplication():
assert 3 * 5 == 15
def test_division():
assert 8 / 4 == 2To run these tests in parallel, you can use the following command: However, the default behavior of pytest-xdist is to run all tests in parallel, which might make the framework less maintainable. To address this, we propose creating two categories of tests using a marker, e.g., parallel. When running tests, all parallel tests will be executed with the recommended number of workers, followed by sequential tests. This approach will maintain test structure and manage complexity effectively. import pytest
@pytest.mark.parallel
def test_vuln_package(configure_parallel_tests, agents, environment):
WazuhEnv.install_package(packages_vul[agent])
...
@pytest.mark.parallel
def test_vuln_package2(configure_parallel_tests, agents, environment):
WazuhEnv.install_package(packages_vul[agent])
# Sequential test
def test_vuln_package3(environment):
...You can then run parallel tests using the marker: python3 -m pytest -v test_examples --inventory=../Wazuh_QA_environment305_testing_inventory.yaml -m parallel -n auto
python3 -m pytest -v test_examples --inventory=../Wazuh_QA_environment305_testing_inventory.yaml -m "not parallel" This approach helps maintain the structure and manage the complexity of your test suite effectively. |
|
Based on the analysis presented in this comment, our testing strategy will involve the utilization of pytest-xdist for parallelization, implementing the hooks as specified. This approach allows us to eliminate the need for dependent tests in the case of VD tests, instead opting for tests that, despite repeating operations, will employ a parallel structure for more efficient execution. |
|
To move forward with the proposed approach, we must develop a launcher comprising a Python script. This script will handle the separate launch of sequential and parallel tests, taking charge of environment setup and appropriate cleanup based on the specific test type being executed. |
Currently working on cleanup and configuration functions of the launcher |
|
I have been in a meeting with @Deblintrake09 regarding the possible options of parallelization. The current approach detailed in this comment would imply to increase the number of vulnerable packages to use. In addition it would be difficult to handle the requirements of each tests case. For this reason, we are currently researching the possibility to launch the tests cases parallel and dependent. For now we have achieved parallelization encapsulated for each agent. In addition we have started to work in the launcher script, which is planned to be used in order to launch parallel and sequential tests correctly. In addition this launcher will be the responsible of the configuration and cleaning of the environment |
|
We have research possible packages to follow the parallel approach
|
|
Perform a little presentation in order to present current status of the development and possible approaches |
|
After a meeting with @davidjiglesias we have confirm that the approach to be followed for VD testing is going to be sequential using parallel operations among all the host of the environment. |
Description
In the context of the new System tests for Vulnerability Detector module, it is imperative that we explore efficient ways to implement parallel testing within our testing framework.
This necessity arises from our plan to launch these new tests across multiple agents. Given the considerable time required for the proposed test cases, it is crucial to execute parallel tests for each agent, thereby significantly reducing both the costs and time associated with the complete testing suite.
The text was updated successfully, but these errors were encountered: