Adding workflow as library function #4925

pro-akim · 2024-02-07T15:04:12Z

Description

Adding setup file and changing directories in order to create Workflow as a library option

Testing performed

(wf_test) akim@akim-PC:~/Desktop$ pip3 list
Package    Version
---------- -------
pip        22.0.2
setuptools 59.6.0

(wf_test) akim@akim-PC:~/Desktop/wazuh-qa/deployability/modules$ pip3 install .
Processing /home/akim/Desktop/wazuh-qa/deployability/modules
  Preparing metadata (setup.py) ... done
Using legacy 'setup.py install' for workflow-engine, since package 'wheel' is not installed.
Installing collected packages: workflow-engine
  Running setup.py install for workflow-engine ... done
Successfully installed workflow-engine-1.0


(wf_test) akim@akim-PC:~/Desktop/venvs/wf_test/bin$ python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from workflow_engine import models
>>> from workflow_engine import schema_validator
>>> from workflow_engine import task
>>> from workflow_engine import workflow_processor
>>> from workflow_engine import main
>>> from workflow_engine import logging


(wf_test) akim@akim-PC:~/Desktop/venvs/wf_test/lib/python3.10/site-packages/workflow_engine$ tree
.
├── examples
│   ├── dtt1-agents-poc.yaml
│   ├── dtt1-agents.yaml
│   └── dtt1-managers.yaml
├── __init__.py
├── logging
│   ├── config.yaml
│   ├── filter.py
│   ├── __init__.py
│   ├── logger.py
│   └── __pycache__
│       ├── filter.cpython-310.pyc
│       ├── __init__.cpython-310.pyc
│       └── logger.cpython-310.pyc
├── main.py
├── models.py
├── __pycache__
│   ├── __init__.cpython-310.pyc
│   ├── main.cpython-310.pyc
│   ├── models.cpython-310.pyc
│   ├── schema_validator.cpython-310.pyc
│   ├── task.cpython-310.pyc
│   ├── utils.cpython-310.pyc
│   └── workflow_processor.cpython-310.pyc
├── README.md
├── schemas
│   └── schema_v1.json
├── schema_validator.py
├── task.py
└── workflow_processor.py

5 directories, 25 files


(deplo_test) akim@akim-PC:~/Desktop/wazuh-qa/deployability$ python3 -m workflow_engine  /home/akim/Desktop/wazuh-qa/deployability/modules/workflow_engine/examples/dtt1-agents-poc.yaml
[2024-02-07 19:35:14] [INFO] [1271970] [MainThread] [workflow_engine]: Starting workflow execution.
[2024-02-07 19:35:14] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "allocate-manager".
[2024-02-07 19:38:14] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "allocate-manager" completed succesfully in 179.54 seconds.
[2024-02-07 19:38:14] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "allocate-linux-ubuntu-22.04-amd64".
[2024-02-07 19:41:05] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "allocate-linux-ubuntu-22.04-amd64" completed succesfully in 170.53 seconds.
[2024-02-07 19:41:05] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "provision-manager".
[2024-02-07 19:43:50] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "provision-manager" completed succesfully in 165.88 seconds.
[2024-02-07 19:43:50] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "provision-install-linux-ubuntu-22.04-amd64".
[2024-02-07 19:45:37] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "provision-install-linux-ubuntu-22.04-amd64" completed succesfully in 106.15 seconds.
[2024-02-07 19:45:37] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "run-agent-tests-linux-ubuntu-22.04-amd64".
[2024-02-07 19:45:38] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "run-agent-tests-linux-ubuntu-22.04-amd64" completed succesfully in 1.18 seconds.
[2024-02-07 19:45:38] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "provision-uninstall-linux-ubuntu-22.04-amd64".
[2024-02-07 19:45:57] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "provision-uninstall-linux-ubuntu-22.04-amd64" completed succesfully in 18.73 seconds.
[2024-02-07 19:45:57] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "run-agent-tests-uninstall-linux-ubuntu-22.04-amd64".
[2024-02-07 19:45:58] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "run-agent-tests-uninstall-linux-ubuntu-22.04-amd64" completed succesfully in 1.14 seconds.
[2024-02-07 19:45:58] [INFO] [1271970] [MainThread] [workflow_engine]: Executing cleanup tasks.

Paths of playbooks were added inside of each module and relative paths were fixed.

QU3B1M

LGTM

QU3B1M · 2024-02-07T19:43:23Z

deployability/modules/testing/testing.py

@@ -77,7 +77,7 @@ def _run_tests(cls, test_list: list[str], ansible: Ansible, extra_vars: ExtraVar
        """
        for test in test_list:
            rendering_var = {**extra_vars, 'test': test}
-            template = str(ansible.playbooks_path / cls._test_template)
+            template = str(ansible.testing_playbook_path / cls._test_template)


Now that the playbooks are inside the test module I think the playbooks path should be defined in the module itself. The same applies for the provision

Done af65f61

fcaffieri · 2024-02-07T19:55:17Z

deployability/modules/workflow_engine/examples/dtt1-agents-poc.yaml

@@ -19,7 +19,7 @@ tasks:
      with:
        path: python3
        args:
-          - test.py
+          - /home/akim/Desktop/wazuh-qa/deployability/modules/testing/main.py


Please remove personal paths from the example.

Done in 0c156f8

pro-akim added 2 commits February 7, 2024 15:59

enhacement(#4909): Changes in playbooks dir, setup.py and launchers

506d315

fix(#4909): Changes in script_list in setup

e40a2c9

pro-akim linked an issue Feb 7, 2024 that may be closed by this pull request

DTT1 - Iteration 3 - WorkFlow engine module - Install module as Python lib #4909

Closed

3 tasks

pro-akim added 2 commits February 7, 2024 17:52

enhancement(#4909): Chaning logging directory name

5767f63

fix(#4909): Fixing paths

b8c54da

pro-akim mentioned this pull request Feb 7, 2024

Adding workflow as library function #4918

Closed

QU3B1M approved these changes Feb 7, 2024

View reviewed changes

fcaffieri requested changes Feb 7, 2024

View reviewed changes

fcaffieri mentioned this pull request Feb 7, 2024

DTT1 - Iteration 3 - WorkFlow engine module - Install module as Python lib #4909

Closed

3 tasks

pro-akim added 3 commits February 8, 2024 10:52

fix(#4909): Fixing paths and restoring dtt-1-agents-poc-yaml

0c156f8

fix(#4909): Fixing path testing.py

af65f61

fix(#4909): Fixing path testing

b281a01

pro-akim requested a review from fcaffieri February 8, 2024 10:00

fix(#4909): Fixing path in dtt1-agents-poc.yaml

f658807

fcaffieri approved these changes Feb 8, 2024

View reviewed changes

fcaffieri merged commit 9b87195 into enhancement/4905-dtt1-iteration-3-workflow Feb 8, 2024

fcaffieri deleted the enhancement/4909-Install-module-as-Python-library branch February 8, 2024 12:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding workflow as library function #4925

Adding workflow as library function #4925

pro-akim commented Feb 7, 2024 •

edited

Loading

QU3B1M left a comment

QU3B1M Feb 7, 2024

pro-akim Feb 8, 2024

fcaffieri Feb 7, 2024

pro-akim Feb 8, 2024

Adding workflow as library function #4925

Adding workflow as library function #4925

Conversation

pro-akim commented Feb 7, 2024 • edited Loading

Description

Testing performed

QU3B1M left a comment

Choose a reason for hiding this comment

QU3B1M Feb 7, 2024

Choose a reason for hiding this comment

pro-akim Feb 8, 2024

Choose a reason for hiding this comment

fcaffieri Feb 7, 2024

Choose a reason for hiding this comment

pro-akim Feb 8, 2024

Choose a reason for hiding this comment

pro-akim commented Feb 7, 2024 •

edited

Loading