DTT1 - Iteration 3 - WorkFlow engine module - Install module as Python lib

[fcaffieri]

Epic: #4905

---

Description

The objective of this issue is to allow the Workflow engine module to be installable as a python library so that it can be used as needed.

Taks

• Generate the setup.py
• Implement installation with a setup.py
• Perform tests in a clean environment.

[pro-akim]

Update

Researching how to convert the python script to an installable library.

---

Done:

• setup.py created
• Install from setup.py

Research about whether the library should be public in the pip repository should be discussed and in case the result is positive, the user needs to be defined. After this, it should be fully tested.

Waiting for definitions. Moved to on hold.

[pro-akim]

Update

There were changes in the test structure, so the libraries will be modified. Initially, the focus was solely on generating packages from the workflow. However, after a meeting with @fcaffieri and @QU3B1M , it has been decided that:

There should be the possibility to install:

1. pip3 install deployability (install all in one)
2. pip3 install deployability/workflow
3. pip3 install deployability/allocator
4. pip3 install deployability/workflow
5. pip3 install deployability/test

Some additional research is required to fulfill the new requirement

[pro-akim]

Update

• Research 'poetry' as a solution to create multiple dependencies

It is possible to create dependencies for each module all in one moving through directories. There is not an elegant solution to create the modules I want selecting it from the command line, unless it is mediated by script.

• Change directory structure of current deployability app
• Fix the relative path to the new structure
• Test the new structure
• Create dependencies for each mentioned module

After knowing the performance and use of 'poetry', the decision is make only a library of workflow_engine

• Test the creation of new dependencies

(workflow_test) akim@akim-PC:~/Desktop/venvs$ pip3 list
Package    Version
---------- -------
pip        22.0.2
setuptools 59.6.0

(workflow_test) akim@akim-PC:~/Desktop/wazuh-qa/deployability/deps$ pip3 install .
Processing /home/akim/Desktop/wazuh-qa/deployability/deps
  Preparing metadata (setup.py) ... done
Using legacy 'setup.py install' for workflow, since package 'wheel' is not installed.
Installing collected packages: workflow
  Running setup.py install for workflow ... done
Successfully installed workflow-1.0


(testing_wf) akim@akim-PC:~/Desktop/wazuh-qa/deployability/deps$ python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from workflow_engine import schema_validator
>>> from workflow_engine import task
>>> from workflow_engine import utils
>>> from workflow_engine import workflow_processor


(workflow_test) akim@akim-PC:~/Desktop/wazuh-qa/deployability$ tree /home/akim/Desktop/venvs/workflow_test/lib/python3.10/site-packages/workflow
/home/akim/Desktop/venvs/workflow_test/lib/python3.10/site-packages/workflow
├── examples
│   ├── dtt1-agents-poc.yaml
│   ├── dtt1-agents.yaml
│   └── dtt1-managers.yaml
├── __init__.py
├── models.py
├── __pycache__
│   ├── __init__.cpython-310.pyc
│   ├── models.cpython-310.pyc
│   ├── schema_validator.cpython-310.pyc
│   ├── task.cpython-310.pyc
│   ├── utils.cpython-310.pyc
│   └── workflow_processor.cpython-310.pyc
├── README.md
├── schemas
│   └── schema_v1.json
├── schema_validator.py
├── task.py
├── utils.py
└── workflow_processor.py

3 directories, 17 files

[pro-akim]

Update

The requested changes have been made.

[pro-akim]

Update

Launchers .py will be renamed main.py and will be located each in each modules/{module}
On the other hand, setup.py will be moved to modules directory.

[pro-akim]

Update

Testing performed

(wf_test) akim@akim-PC:~/Desktop$ pip3 list
Package    Version
---------- -------
pip        22.0.2
setuptools 59.6.0

(wf_test) akim@akim-PC:~/Desktop/wazuh-qa/deployability/modules$ pip3 install .
Processing /home/akim/Desktop/wazuh-qa/deployability/modules
  Preparing metadata (setup.py) ... done
Using legacy 'setup.py install' for workflow-engine, since package 'wheel' is not installed.
Installing collected packages: workflow-engine
  Running setup.py install for workflow-engine ... done
Successfully installed workflow-engine-1.0


(wf_test) akim@akim-PC:~/Desktop/venvs/wf_test/bin$ python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from workflow_engine import models
>>> from workflow_engine import schema_validator
>>> from workflow_engine import task
>>> from workflow_engine import workflow_processor
>>> from workflow_engine import logging


(wf_test) akim@akim-PC:~/Desktop/venvs/wf_test/lib/python3.10/site-packages/workflow_engine$ tree
.
├── examples
│   ├── dtt1-agents-poc.yaml
│   ├── dtt1-agents.yaml
│   └── dtt1-managers.yaml
├── __init__.py
├── logging
│   ├── config.yaml
│   ├── filter.py
│   ├── __init__.py
│   ├── logger.py
│   └── __pycache__
│       ├── filter.cpython-310.pyc
│       ├── __init__.cpython-310.pyc
│       └── logger.cpython-310.pyc
├── main.py
├── models.py
├── __pycache__
│   ├── __init__.cpython-310.pyc
│   ├── main.cpython-310.pyc
│   ├── models.cpython-310.pyc
│   ├── schema_validator.cpython-310.pyc
│   ├── task.cpython-310.pyc
│   ├── utils.cpython-310.pyc
│   └── workflow_processor.cpython-310.pyc
├── README.md
├── schemas
│   └── schema_v1.json
├── schema_validator.py
├── task.py
└── workflow_processor.py

5 directories, 25 files


(deplo_test) akim@akim-PC:~/Desktop/wazuh-qa/deployability$ python3 -m workflow_engine  /home/akim/Desktop/wazuh-qa/deployability/modules/workflow_engine/examples/dtt1-agents-poc.yaml
[2024-02-07 19:35:14] [INFO] [1271970] [MainThread] [workflow_engine]: Starting workflow execution.
[2024-02-07 19:35:14] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "allocate-manager".
[2024-02-07 19:38:14] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "allocate-manager" completed succesfully in 179.54 seconds.
[2024-02-07 19:38:14] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "allocate-linux-ubuntu-22.04-amd64".
[2024-02-07 19:41:05] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "allocate-linux-ubuntu-22.04-amd64" completed succesfully in 170.53 seconds.
[2024-02-07 19:41:05] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "provision-manager".
[2024-02-07 19:43:50] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "provision-manager" completed succesfully in 165.88 seconds.
[2024-02-07 19:43:50] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "provision-install-linux-ubuntu-22.04-amd64".
[2024-02-07 19:45:37] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "provision-install-linux-ubuntu-22.04-amd64" completed succesfully in 106.15 seconds.
[2024-02-07 19:45:37] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "run-agent-tests-linux-ubuntu-22.04-amd64".
[2024-02-07 19:45:38] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "run-agent-tests-linux-ubuntu-22.04-amd64" completed succesfully in 1.18 seconds.
[2024-02-07 19:45:38] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "provision-uninstall-linux-ubuntu-22.04-amd64".
[2024-02-07 19:45:57] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "provision-uninstall-linux-ubuntu-22.04-amd64" completed succesfully in 18.73 seconds.
[2024-02-07 19:45:57] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Initiating execution of task "run-agent-tests-uninstall-linux-ubuntu-22.04-amd64".
[2024-02-07 19:45:58] [INFO] [1271970] [ThreadPoolExecutor-0_0] [workflow_engine]: Task "run-agent-tests-uninstall-linux-ubuntu-22.04-amd64" completed succesfully in 1.14 seconds.
[2024-02-07 19:45:58] [INFO] [1271970] [MainThread] [workflow_engine]: Executing cleanup tasks.

[QU3B1M]

LGTM!

[fcaffieri]

Review

GJ, a little change #4925 (comment)

[pro-akim]

Update

Requested changes done.

[fcaffieri]

LGTM