Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DTT1 - Workflow module - Bug with threads #5220

Closed
3 tasks done
fcaffieri opened this issue Apr 16, 2024 · 10 comments
Closed
3 tasks done

DTT1 - Workflow module - Bug with threads #5220

fcaffieri opened this issue Apr 16, 2024 · 10 comments
Assignees
@mhamra
Labels
level/task Task issue type/enhancement

Comments

@fcaffieri
Copy link
Member

fcaffieri commented Apr 16, 2024 •
edited
Loading

Description

The objective of this issue is to solve the bug when executing the Workflow with threads. Currently, when executing an execution with several VMs, random errors are found due to execution with threads. If the same execution is performed without the threads parameter, it works correctly.

Tasks

  • Investigate the cause of the bug when executing with threads.
  • Fix the bug
  • Perform integration tests.
@fcaffieri fcaffieri mentioned this issue Apr 16, 2024
Closed
1 task
@wazuhci wazuhci moved this to Triage in Release 4.9.0 Apr 16, 2024
@wazuhci wazuhci moved this from Triage to Backlog in Release 4.9.0 Apr 16, 2024
@rauldpm rauldpm mentioned this issue Apr 16, 2024
Closed
11 tasks
@mhamra mhamra self-assigned this Apr 16, 2024
@wazuhci wazuhci moved this from Backlog to In progress in Release 4.9.0 Apr 16, 2024
@mhamra
Copy link
Contributor

mhamra commented Apr 16, 2024 •
edited
Loading

UPDATE

I've tested the workflow's thread parameter with two workflows. I've used Visual Studio code to launch the workflow_engine.

🆗 TEST 1 - One manager, 2 agents; threads=3

workflow.yml file 
version: 0.1
description: This workflow is used to test agents deployment por DDT1 PoC
variables:
  agent-os:
    - linux-ubuntu-18.04-amd64
    - linux-ubuntu-20.04-amd64
  manager-os: linux-ubuntu-22.04-amd64
  infra-provider: aws
  working-dir: /tmp/dtt1-poc

tasks:
  # Unique manager allocate task
  - task: "allocate-manager-{manager-os}"
    description: "Allocate resources for the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: large
          - composite-name: "{manager-os}"
          - inventory-output: "{working-dir}/manager-{manager-os}/inventory.yaml"
          - track-output: "{working-dir}/manager-{manager-os}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    cleanup:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: delete
          - track-output: "{working-dir}/manager-{manager-os}/track.yaml"

  # Unique agent allocate task
  - task: "allocate-agent-{agent}"
    description: "Allocate resources for the agent."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: small
          - composite-name: "{agent}"
          - inventory-output: "{working-dir}/agent-{agent}/inventory.yaml"
          - track-output: "{working-dir}/agent-{agent}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    foreach:
      - variable: agent-os
        as: agent
    cleanup:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: delete
          - track-output: "{working-dir}/agent-{agent}/track.yaml"

  # Unique manager provision task
  - task: "provision-manager-{manager-os}"
    description: "Provision the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/provision/main.py
          - inventory: "{working-dir}/manager-{manager-os}/inventory.yaml"
          - install:
            - component: wazuh-manager
              type: assistant
              version: 4.7.3
              live: True
    depends-on:
      - "allocate-manager-{manager-os}"


  # Generic agent test task
  - task: "run-agent-{agent}-tests"
    description: "Run tests install for the agent {agent}."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/testing/main.py
          - targets:
            - wazuh-1: "{working-dir}/manager-{manager-os}/inventory.yaml"
            - agent: "{working-dir}/agent-{agent}/inventory.yaml"
          - tests: "install,registration,restart,stop,uninstall"
          - component: "agent"
          - wazuh-version: "4.7.3"
          - wazuh-revision: "40714"
          - live: "True"
    foreach:
      - variable: agent-os
        as: agent
    depends-on:
      - "provision-manager-{manager-os}"
      - "allocate-agent-{agent}"
Console output 
cd /home/marcelo/wazuh/wazuh-qa/deployability/ ; (dtt-test) marcelo@marcelo-B460-AORUS-PRO-AC:~/wazuh/wazuh-qa$  cd /home/marcelo/wazuh/wazuh-qa/deployability/ ; /usr/bin/env /home/marcelo/.pyenv/versions/dtt-test/bin/python /home/marcelo/.vscode/extensions/ms-python.debugpy-2024.4.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher 40967 -- /home/marcelo/wazuh/wazuh-qa/deployability/modules/workflow_engine/__main__.py --threads 3 /home/marcelo/wazuh/test.yml 
[2024-04-16 12:47:53] [INFO] [423209] [MainThread] [workflow_engine]: Executing DAG tasks.
[2024-04-16 12:47:53] [INFO] [423209] [MainThread] [workflow_engine]: Executing tasks in parallel.
[2024-04-16 12:48:02] [INFO] [423209] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Starting task.
[2024-04-16 12:48:02] [INFO] [423209] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-agent-linux-ubuntu-20.04-amd64] Starting task.
[2024-04-16 12:48:02] [INFO] [423209] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Starting task.
[2024-04-16 12:48:42] [INFO] [423209] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-agent-linux-ubuntu-20.04-amd64] Finished task in 40.65 seconds.
[2024-04-16 12:49:26] [INFO] [423209] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Finished task in 83.82 seconds.
[2024-04-16 12:49:28] [INFO] [423209] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Finished task in 86.75 seconds.
[2024-04-16 12:49:33] [INFO] [423209] [ThreadPoolExecutor-0_1] [workflow_engine]: [provision-manager-linux-ubuntu-22.04-amd64] Starting task.
[2024-04-16 12:54:26] [INFO] [423209] [ThreadPoolExecutor-0_1] [workflow_engine]: [provision-manager-linux-ubuntu-22.04-amd64] Finished task in 292.49 seconds.
[2024-04-16 12:54:26] [INFO] [423209] [ThreadPoolExecutor-0_0] [workflow_engine]: [run-agent-linux-ubuntu-20.04-amd64-tests] Starting task.
[2024-04-16 12:54:26] [INFO] [423209] [ThreadPoolExecutor-0_2] [workflow_engine]: [run-agent-linux-ubuntu-18.04-amd64-tests] Starting task.
[2024-04-16 13:01:30] [INFO] [423209] [ThreadPoolExecutor-0_0] [workflow_engine]: [run-agent-linux-ubuntu-20.04-amd64-tests] Finished task in 424.62 seconds.
[2024-04-16 13:01:31] [INFO] [423209] [ThreadPoolExecutor-0_2] [workflow_engine]: [run-agent-linux-ubuntu-18.04-amd64-tests] Finished task in 425.11 seconds.
[2024-04-16 13:01:31] [INFO] [423209] [MainThread] [workflow_engine]: Executing Reverse DAG tasks.
[2024-04-16 13:01:31] [INFO] [423209] [MainThread] [workflow_engine]: Executing tasks in parallel.
[2024-04-16 13:01:39] [INFO] [423209] [ThreadPoolExecutor-1_0] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Starting task.
[2024-04-16 13:01:39] [INFO] [423209] [ThreadPoolExecutor-1_1] [workflow_engine]: [allocate-agent-linux-ubuntu-20.04-amd64] Starting task.
[2024-04-16 13:01:39] [INFO] [423209] [ThreadPoolExecutor-1_2] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Starting task.
[2024-04-16 13:02:14] [INFO] [423209] [ThreadPoolExecutor-1_1] [workflow_engine]: [allocate-agent-linux-ubuntu-20.04-amd64] Finished task in 34.92 seconds.
[2024-04-16 13:02:45] [INFO] [423209] [ThreadPoolExecutor-1_2] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Finished task in 66.24 seconds.
[2024-04-16 13:03:00] [INFO] [423209] [ThreadPoolExecutor-1_0] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Finished task in 81.19 seconds.
worflow.log file 
[2024-04-16 12:43:23,384] [ERROR] [422316] [MainThread] [workflow_engine]: Error while validating schema [/home/marcelo/wazuh/wazuh-qa/deployability/modules/workflow_engine/schemas/schema_v1.json] with error: File "/home/marcelo/wazuh/test.yaml" not found.
[2024-04-16 12:44:53,600] [INFO] [422629] [MainThread] [workflow_engine]: Executing DAG tasks.
[2024-04-16 12:44:53,601] [INFO] [422629] [MainThread] [workflow_engine]: Executing tasks in parallel.
[2024-04-16 12:46:28,400] [INFO] [422629] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Starting task.
[2024-04-16 12:46:28,402] [INFO] [422629] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Starting task.
[2024-04-16 12:46:31,596] [INFO] [422629] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-agent-linux-ubuntu-20.04-amd64] Starting task.
[2024-04-16 12:47:53,669] [INFO] [423209] [MainThread] [workflow_engine]: Executing DAG tasks.
[2024-04-16 12:47:53,670] [INFO] [423209] [MainThread] [workflow_engine]: Executing tasks in parallel.
[2024-04-16 12:48:02,208] [INFO] [423209] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Starting task.
[2024-04-16 12:48:02,215] [INFO] [423209] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-agent-linux-ubuntu-20.04-amd64] Starting task.
[2024-04-16 12:48:02,217] [INFO] [423209] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Starting task.
[2024-04-16 12:48:20,139] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-04-16 12:48:20,270] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-04-16 12:48:20,271] [DEBUG] ALLOCATOR: Generating new key pair
[2024-04-16 12:48:20,271] [DEBUG] ALLOCATOR: Creating base directory: /tmp/wazuh-qa/AWS-A2967581-6C27-4CF1-AB37-B6920D7819A4
[2024-04-16 12:48:40,062] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-A2967581-6C27-4CF1-AB37-B6920D7819A4 directory to /tmp/wazuh-qa/i-06115b7d98a765e6e
[2024-04-16 12:48:40,085] [INFO] ALLOCATOR: Instance i-06115b7d98a765e6e created.
[2024-04-16 12:48:41,815] [INFO] ALLOCATOR: Instance i-06115b7d98a765e6e started.
[2024-04-16 12:48:42,225] [INFO] ALLOCATOR: Inventory file generated at /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/inventory.yaml
[2024-04-16 12:48:42,227] [INFO] ALLOCATOR: Track file generated at /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/track.yaml
[2024-04-16 12:48:42,879] [INFO] [423209] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-agent-linux-ubuntu-20.04-amd64] Finished task in 40.65 seconds.
[2024-04-16 12:49:03,301] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-04-16 12:49:03,426] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-04-16 12:49:03,426] [DEBUG] ALLOCATOR: Generating new key pair
[2024-04-16 12:49:03,427] [DEBUG] ALLOCATOR: Creating base directory: /tmp/wazuh-qa/AWS-6474989B-DBE9-47B8-BBE9-51438128949A
[2024-04-16 12:49:06,211] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-04-16 12:49:06,336] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-04-16 12:49:06,337] [DEBUG] ALLOCATOR: Generating new key pair
[2024-04-16 12:49:06,337] [DEBUG] ALLOCATOR: Creating base directory: /tmp/wazuh-qa/AWS-7948C8F1-07CA-43AA-90DA-C17976B756A6
[2024-04-16 12:49:23,330] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-6474989B-DBE9-47B8-BBE9-51438128949A directory to /tmp/wazuh-qa/i-08bcf22d47fdf26bf
[2024-04-16 12:49:23,352] [INFO] ALLOCATOR: Instance i-08bcf22d47fdf26bf created.
[2024-04-16 12:49:24,997] [INFO] ALLOCATOR: Instance i-08bcf22d47fdf26bf started.
[2024-04-16 12:49:25,405] [INFO] ALLOCATOR: Inventory file generated at /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/inventory.yaml
[2024-04-16 12:49:25,407] [INFO] ALLOCATOR: Track file generated at /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/track.yaml
[2024-04-16 12:49:26,035] [INFO] [423209] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Finished task in 83.82 seconds.
[2024-04-16 12:49:26,252] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-7948C8F1-07CA-43AA-90DA-C17976B756A6 directory to /tmp/wazuh-qa/i-05c2f0c68f28ab16e
[2024-04-16 12:49:26,278] [INFO] ALLOCATOR: Instance i-05c2f0c68f28ab16e created.
[2024-04-16 12:49:27,816] [INFO] ALLOCATOR: Instance i-05c2f0c68f28ab16e started.
[2024-04-16 12:49:28,224] [INFO] ALLOCATOR: Inventory file generated at /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml
[2024-04-16 12:49:28,226] [INFO] ALLOCATOR: Track file generated at /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/track.yaml
[2024-04-16 12:49:28,991] [INFO] [423209] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Finished task in 86.75 seconds.
[2024-04-16 12:49:33,808] [INFO] [423209] [ThreadPoolExecutor-0_1] [workflow_engine]: [provision-manager-linux-ubuntu-22.04-amd64] Starting task.
[2024-04-16 12:49:40,685] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:49:40,686] [DEBUG] ANSIBLE: Running playbook: {'hosts': 'ec2-100-24-22-95.compute-1.amazonaws.com', 'become': True, 'gather_facts': True, 'tasks': [{'name': 'Capture ansible_os_family', 'set_fact': {'ansible_os_family': "{{ ansible_facts['distribution_file_variety'] }}", 'cacheable': 'yes'}}]}
[2024-04-16 12:50:01,195] [DEBUG] ANSIBLE: Playbook {'hosts': 'ec2-100-24-22-95.compute-1.amazonaws.com', 'become': True, 'gather_facts': True, 'tasks': [{'name': 'Capture ansible_os_family', 'set_fact': {'ansible_os_family': "{{ ansible_facts['distribution_file_variety'] }}", 'cacheable': 'yes'}}]} finished with status {'skipped': {}, 'ok': {'ec2-100-24-22-95.compute-1.amazonaws.com': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'ec2-100-24-22-95.compute-1.amazonaws.com': 1}, 'changed': {}}
[2024-04-16 12:50:01,196] [DEBUG] ANSIBLE: Templates found: ['download.j2', 'install.j2']
[2024-04-16 12:50:01,215] [DEBUG] ANSIBLE: Rendering template download.j2
[2024-04-16 12:50:01,219] [DEBUG] ANSIBLE: Rendering template install.j2
[2024-04-16 12:50:01,220] [DEBUG] ANSIBLE: [{'name': 'Install the required packages', 'shell': '\nsudo apt-get update && apt-get -y install curl\n'}, {'name': 'Download the Wazuh installation assistant', 'shell': 'curl -sO https://packages.wazuh.com/4.7/wazuh-install.sh'}, {'name': 'Install wazuh-manager with assistant', 'shell': 'bash ./wazuh-install.sh -a -i'}]
[2024-04-16 12:50:01,221] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:50:01,221] [DEBUG] ANSIBLE: Running playbook: {'hosts': 'ec2-100-24-22-95.compute-1.amazonaws.com', 'become': True, 'gather_facts': True, 'tasks': [{'name': 'Install the required packages', 'shell': '\nsudo apt-get update && apt-get -y install curl\n'}, {'name': 'Download the Wazuh installation assistant', 'shell': 'curl -sO https://packages.wazuh.com/4.7/wazuh-install.sh'}, {'name': 'Install wazuh-manager with assistant', 'shell': 'bash ./wazuh-install.sh -a -i'}]}
[2024-04-16 12:54:25,667] [DEBUG] ANSIBLE: Playbook {'hosts': 'ec2-100-24-22-95.compute-1.amazonaws.com', 'become': True, 'gather_facts': True, 'tasks': [{'name': 'Install the required packages', 'shell': '\nsudo apt-get update && apt-get -y install curl\n'}, {'name': 'Download the Wazuh installation assistant', 'shell': 'curl -sO https://packages.wazuh.com/4.7/wazuh-install.sh'}, {'name': 'Install wazuh-manager with assistant', 'shell': 'bash ./wazuh-install.sh -a -i'}]} finished with status {'skipped': {}, 'ok': {'ec2-100-24-22-95.compute-1.amazonaws.com': 4}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'ec2-100-24-22-95.compute-1.amazonaws.com': 1}, 'changed': {'ec2-100-24-22-95.compute-1.amazonaws.com': 3}}
[2024-04-16 12:54:26,303] [INFO] [423209] [ThreadPoolExecutor-0_1] [workflow_engine]: [provision-manager-linux-ubuntu-22.04-amd64] Finished task in 292.49 seconds.
[2024-04-16 12:54:26,324] [INFO] [423209] [ThreadPoolExecutor-0_0] [workflow_engine]: [run-agent-linux-ubuntu-20.04-amd64-tests] Starting task.
[2024-04-16 12:54:26,329] [INFO] [423209] [ThreadPoolExecutor-0_2] [workflow_engine]: [run-agent-linux-ubuntu-18.04-amd64-tests] Starting task.
[2024-04-16 12:54:27,267] [INFO] TESTER: Running tests for ec2-100-24-22-95.compute-1.amazonaws.com
[2024-04-16 12:54:27,269] [INFO] TESTER: Running tests for ec2-18-205-238-133.compute-1.amazonaws.com
[2024-04-16 12:54:27,269] [DEBUG] TESTER: Using extra vars: {'component': 'agent', 'wazuh_version': '4.7.3', 'wazuh_revision': '40714', 'wazuh_branch': None, 'working_dir': '/tmp/tests', 'live': True, 'hosts_ip': ['ec2-100-24-22-95.compute-1.amazonaws.com', 'ec2-18-205-238-133.compute-1.amazonaws.com'], 'targets': '{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/inventory.yaml}', 'dependencies': '{}', 'local_host_path': '/home/marcelo/wazuh/wazuh-qa/deployability', 'current_user': 'marcelo'}
[2024-04-16 12:54:27,286] [DEBUG] ANSIBLE: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/setup.yml
[2024-04-16 12:54:27,290] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:54:27,291] [DEBUG] ANSIBLE: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Cleaning old key ssh-keygen registries', 'ansible.builtin.command': {'cmd': "ssh-keygen -f /home/marcelo/.ssh/known_hosts -R ''"}, 'loop': ['ec2-100-24-22-95.compute-1.amazonaws.com', 'ec2-18-205-238-133.compute-1.amazonaws.com']}]}]
[2024-04-16 12:54:27,682] [INFO] TESTER: Running tests for ec2-100-24-22-95.compute-1.amazonaws.com
[2024-04-16 12:54:27,684] [INFO] TESTER: Running tests for ec2-54-160-164-203.compute-1.amazonaws.com
[2024-04-16 12:54:27,684] [DEBUG] TESTER: Using extra vars: {'component': 'agent', 'wazuh_version': '4.7.3', 'wazuh_revision': '40714', 'wazuh_branch': None, 'working_dir': '/tmp/tests', 'live': True, 'hosts_ip': ['ec2-100-24-22-95.compute-1.amazonaws.com', 'ec2-54-160-164-203.compute-1.amazonaws.com'], 'targets': '{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/inventory.yaml}', 'dependencies': '{}', 'local_host_path': '/home/marcelo/wazuh/wazuh-qa/deployability', 'current_user': 'marcelo'}
[2024-04-16 12:54:27,700] [DEBUG] ANSIBLE: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/setup.yml
[2024-04-16 12:54:27,704] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:54:27,704] [DEBUG] ANSIBLE: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Cleaning old key ssh-keygen registries', 'ansible.builtin.command': {'cmd': "ssh-keygen -f /home/marcelo/.ssh/known_hosts -R ''"}, 'loop': ['ec2-100-24-22-95.compute-1.amazonaws.com', 'ec2-54-160-164-203.compute-1.amazonaws.com']}]}]
[2024-04-16 12:54:38,835] [DEBUG] ANSIBLE: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Cleaning old key ssh-keygen registries', 'ansible.builtin.command': {'cmd': "ssh-keygen -f /home/marcelo/.ssh/known_hosts -R ''"}, 'loop': ['ec2-100-24-22-95.compute-1.amazonaws.com', 'ec2-18-205-238-133.compute-1.amazonaws.com']}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}}
[2024-04-16 12:54:38,841] [DEBUG] ANSIBLE: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/test.yml
[2024-04-16 12:54:38,845] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:54:38,845] [DEBUG] ANSIBLE: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test install for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_install.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}]
[2024-04-16 12:54:39,274] [DEBUG] ANSIBLE: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Cleaning old key ssh-keygen registries', 'ansible.builtin.command': {'cmd': "ssh-keygen -f /home/marcelo/.ssh/known_hosts -R ''"}, 'loop': ['ec2-100-24-22-95.compute-1.amazonaws.com', 'ec2-54-160-164-203.compute-1.amazonaws.com']}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}}
[2024-04-16 12:54:39,280] [DEBUG] ANSIBLE: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/test.yml
[2024-04-16 12:54:39,285] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:54:39,285] [DEBUG] ANSIBLE: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test install for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_install.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}]
[2024-04-16 12:54:50,603] [INFO] [Testing]: Checking connection to ubuntu-18.04
[2024-04-16 12:54:51,038] [INFO] [Testing]: Checking connection to ubuntu-20.04
[2024-04-16 12:54:52,004] [INFO] [Testing]: Connection established successfully in ubuntu-18.04
[2024-04-16 12:54:52,745] [INFO] [Testing]: Connection established successfully in ubuntu-20.04
[2024-04-16 12:54:54,895] [INFO] [Testing]: Checking connection to ubuntu-22.04
[2024-04-16 12:54:55,720] [INFO] [Testing]: Checking connection to ubuntu-22.04
[2024-04-16 12:54:56,535] [INFO] [Testing]: Connection established successfully in ubuntu-22.04
[2024-04-16 12:54:57,111] [INFO] [Testing]: Connection established successfully in ubuntu-22.04
[2024-04-16 12:54:59,594] [INFO] [Testing]: No Firewall to disable on ubuntu-18.04
[2024-04-16 12:54:59,811] [INFO] [Testing]: No Firewall to disable on ubuntu-20.04
[2024-04-16 12:55:02,067] [INFO] [Testing]: Manager is already installed in ubuntu-22.04
[2024-04-16 12:55:02,270] [INFO] [Testing]: Manager is already installed in ubuntu-22.04
[2024-04-16 12:55:07,552] [INFO] [Testing]: Generating Snapshot for Checkfile in ubuntu-18.04
[2024-04-16 12:55:07,598] [INFO] [Testing]: Generating Snapshot for Checkfile in ubuntu-20.04
[2024-04-16 12:55:29,611] [INFO] [Testing]: Installing Agent in ubuntu-18.04
[2024-04-16 12:55:36,269] [INFO] [Testing]: Installing Agent in ubuntu-20.04
[2024-04-16 12:55:49,658] [INFO] [Testing]: Generating Snapshot for Checkfile in ubuntu-18.04
[2024-04-16 12:55:58,078] [INFO] [Testing]: Generating Snapshot for Checkfile in ubuntu-20.04
[2024-04-16 12:56:04,766] [INFO] [Testing]: Applying filters in checkfiles in ubuntu-18.04
[2024-04-16 12:56:04,766] [INFO] [Testing]: Pre and post install checkfile comparison in ubuntu-18.04: {'/boot': {'added': [], 'removed': [], 'modified': []}, '/usr/bin': {'added': [], 'removed': [], 'modified': []}, '/root': {'added': [], 'removed': [], 'modified': []}, '/usr/sbin': {'added': [], 'removed': [], 'modified': []}}
[2024-04-16 12:56:07,908] [INFO] [Testing]: Getting status of ubuntu-18.04
[2024-04-16 12:56:13,803] [INFO] [Testing]: Applying filters in checkfiles in ubuntu-20.04
[2024-04-16 12:56:13,803] [INFO] [Testing]: Pre and post install checkfile comparison in ubuntu-20.04: {'/boot': {'added': [], 'removed': [], 'modified': []}, '/usr/bin': {'added': [], 'removed': [], 'modified': []}, '/root': {'added': [], 'removed': [], 'modified': []}, '/usr/sbin': {'added': [], 'removed': [], 'modified': []}}
[2024-04-16 12:56:16,790] [INFO] [Testing]: Getting status of ubuntu-20.04
[2024-04-16 12:56:16,843] [DEBUG] ANSIBLE: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test install for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_install.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}}
[2024-04-16 12:56:16,848] [DEBUG] ANSIBLE: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/test.yml
[2024-04-16 12:56:16,852] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:56:16,853] [DEBUG] ANSIBLE: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test registration for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_registration.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}]
[2024-04-16 12:56:26,374] [DEBUG] ANSIBLE: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test install for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_install.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}}
[2024-04-16 12:56:26,380] [DEBUG] ANSIBLE: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/test.yml
[2024-04-16 12:56:26,384] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:56:26,384] [DEBUG] ANSIBLE: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test registration for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_registration.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}]
[2024-04-16 12:56:28,484] [INFO] [Testing]: Checking connection to ubuntu-18.04
[2024-04-16 12:56:30,026] [INFO] [Testing]: Connection established successfully in ubuntu-18.04
[2024-04-16 12:56:38,016] [INFO] [Testing]: Checking connection to ubuntu-20.04
[2024-04-16 12:56:39,651] [INFO] [Testing]: Connection established successfully in ubuntu-20.04
[2024-04-16 12:56:50,100] [INFO] [Testing]: Getting status of ubuntu-18.04
[2024-04-16 12:57:00,034] [INFO] [Testing]: Getting status of ubuntu-20.04
[2024-04-16 12:57:23,802] [DEBUG] ANSIBLE: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test registration for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_registration.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}}
[2024-04-16 12:57:23,808] [DEBUG] ANSIBLE: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/test.yml
[2024-04-16 12:57:23,812] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:57:23,813] [DEBUG] ANSIBLE: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test restart for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_restart.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}]
[2024-04-16 12:57:30,497] [DEBUG] ANSIBLE: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test registration for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_registration.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}}
[2024-04-16 12:57:30,503] [DEBUG] ANSIBLE: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/test.yml
[2024-04-16 12:57:30,507] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:57:30,507] [DEBUG] ANSIBLE: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test restart for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_restart.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}]
[2024-04-16 12:57:35,442] [INFO] [Testing]: Checking connection to ubuntu-18.04
[2024-04-16 12:57:36,995] [INFO] [Testing]: Connection established successfully in ubuntu-18.04
[2024-04-16 12:57:42,154] [INFO] [Testing]: Checking connection to ubuntu-20.04
[2024-04-16 12:57:43,581] [INFO] [Testing]: Connection established successfully in ubuntu-20.04
[2024-04-16 12:57:48,805] [INFO] [Testing]: Restarting wazuh-agent in ubuntu-18.04
[2024-04-16 12:57:55,225] [INFO] [Testing]: Restarting wazuh-agent in ubuntu-20.04
[2024-04-16 12:58:00,640] [INFO] [Testing]: Getting status of ubuntu-18.04
[2024-04-16 12:58:07,307] [INFO] [Testing]: Getting status of ubuntu-20.04
[2024-04-16 12:58:13,883] [DEBUG] ANSIBLE: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test restart for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_restart.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}}
[2024-04-16 12:58:13,889] [DEBUG] ANSIBLE: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/test.yml
[2024-04-16 12:58:13,893] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:58:13,893] [DEBUG] ANSIBLE: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test stop for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_stop.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}]
[2024-04-16 12:58:18,977] [DEBUG] ANSIBLE: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test restart for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_restart.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}}
[2024-04-16 12:58:18,983] [DEBUG] ANSIBLE: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/test.yml
[2024-04-16 12:58:18,986] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:58:18,987] [DEBUG] ANSIBLE: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test stop for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_stop.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}]
[2024-04-16 12:58:25,517] [INFO] [Testing]: Checking connection to ubuntu-18.04
[2024-04-16 12:58:27,069] [INFO] [Testing]: Connection established successfully in ubuntu-18.04
[2024-04-16 12:58:30,598] [INFO] [Testing]: Checking connection to ubuntu-20.04
[2024-04-16 12:58:32,189] [INFO] [Testing]: Connection established successfully in ubuntu-20.04
[2024-04-16 12:58:46,834] [INFO] [Testing]: Stopping wazuh-agent in ubuntu-18.04
[2024-04-16 12:58:50,542] [INFO] [Testing]: Getting status of ubuntu-18.04
[2024-04-16 12:58:51,595] [INFO] [Testing]: Stopping wazuh-agent in ubuntu-20.04
[2024-04-16 12:58:55,742] [INFO] [Testing]: Getting status of ubuntu-20.04
[2024-04-16 12:58:57,082] [INFO] [Testing]: Restarting wazuh-agent in ubuntu-18.04
[2024-04-16 12:59:02,706] [INFO] [Testing]: Restarting wazuh-agent in ubuntu-20.04
[2024-04-16 12:59:07,844] [DEBUG] ANSIBLE: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test stop for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_stop.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}}
[2024-04-16 12:59:07,849] [DEBUG] ANSIBLE: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/test.yml
[2024-04-16 12:59:07,853] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:59:07,853] [DEBUG] ANSIBLE: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test uninstall for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_uninstall.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}]
[2024-04-16 12:59:13,268] [DEBUG] ANSIBLE: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test stop for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_stop.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}}
[2024-04-16 12:59:13,274] [DEBUG] ANSIBLE: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/test.yml
[2024-04-16 12:59:13,278] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 12:59:13,278] [DEBUG] ANSIBLE: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test uninstall for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_uninstall.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}]
[2024-04-16 12:59:19,493] [INFO] [Testing]: Checking connection to ubuntu-18.04
[2024-04-16 12:59:21,035] [INFO] [Testing]: Connection established successfully in ubuntu-18.04
[2024-04-16 12:59:24,907] [INFO] [Testing]: Checking connection to ubuntu-20.04
[2024-04-16 12:59:26,364] [INFO] [Testing]: Connection established successfully in ubuntu-20.04
[2024-04-16 12:59:41,619] [INFO] [Testing]: Generating Snapshot for Checkfile in ubuntu-18.04
[2024-04-16 12:59:45,881] [INFO] [Testing]: Generating Snapshot for Checkfile in ubuntu-20.04
[2024-04-16 12:59:58,166] [INFO] [Testing]: Uninstalling Agent in ubuntu-18.04
[2024-04-16 13:00:03,306] [INFO] [Testing]: Uninstalling Agent in ubuntu-20.04
[2024-04-16 13:00:17,877] [INFO] [Testing]: Generating Snapshot for Checkfile in ubuntu-20.04
[2024-04-16 13:00:18,084] [INFO] [Testing]: Generating Snapshot for Checkfile in ubuntu-18.04
[2024-04-16 13:00:31,658] [INFO] [Testing]: Applying filters in checkfiles in ubuntu-20.04
[2024-04-16 13:00:31,658] [INFO] [Testing]: Pre and post uninstall checkfile comparison in ubuntu-20.04: {'/boot': {'added': [], 'removed': [], 'modified': []}, '/usr/bin': {'added': [], 'removed': [], 'modified': []}, '/root': {'added': [], 'removed': [], 'modified': []}, '/usr/sbin': {'added': [], 'removed': [], 'modified': []}}
[2024-04-16 13:00:31,925] [INFO] [Testing]: Applying filters in checkfiles in ubuntu-18.04
[2024-04-16 13:00:31,925] [INFO] [Testing]: Pre and post uninstall checkfile comparison in ubuntu-18.04: {'/boot': {'added': [], 'removed': [], 'modified': []}, '/usr/bin': {'added': [], 'removed': [], 'modified': []}, '/root': {'added': [], 'removed': [], 'modified': []}, '/usr/sbin': {'added': [], 'removed': [], 'modified': []}}
[2024-04-16 13:00:47,745] [DEBUG] ANSIBLE: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test uninstall for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_uninstall.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}}
[2024-04-16 13:00:47,749] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 13:00:47,749] [DEBUG] ANSIBLE: Running playbook: /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/cleanup.yml
[2024-04-16 13:00:48,622] [DEBUG] ANSIBLE: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test uninstall for agent', 'command': "python3 -m pytest modules/testing/tests/test_agent/test_uninstall.py  -v --wazuh_version=4.7.3 --wazuh_revision=40714  --component=agent --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, agent: /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}}
[2024-04-16 13:00:48,624] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 13:00:48,624] [DEBUG] ANSIBLE: Running playbook: /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/cleanup.yml
[2024-04-16 13:01:09,727] [DEBUG] ANSIBLE: Playbook /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/cleanup.yml finished with status {'skipped': {}, 'ok': {'ec2-100-24-22-95.compute-1.amazonaws.com': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'ec2-100-24-22-95.compute-1.amazonaws.com': 1}, 'changed': {}}
[2024-04-16 13:01:09,730] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 13:01:09,731] [DEBUG] ANSIBLE: Running playbook: /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/cleanup.yml
[2024-04-16 13:01:10,556] [DEBUG] ANSIBLE: Playbook /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/cleanup.yml finished with status {'skipped': {}, 'ok': {'ec2-100-24-22-95.compute-1.amazonaws.com': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'ec2-100-24-22-95.compute-1.amazonaws.com': 1}, 'changed': {}}
[2024-04-16 13:01:10,560] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'ec2-100-24-22-95.compute-1.amazonaws.com': {'ansible_port': 2200, 'ansible_user': 'ubuntu', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/i-05c2f0c68f28ab16e/ubuntu-22.04-amd64-key-8045'}}}}
[2024-04-16 13:01:10,560] [DEBUG] ANSIBLE: Running playbook: /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/cleanup.yml
[2024-04-16 13:01:30,291] [DEBUG] ANSIBLE: Playbook /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/cleanup.yml finished with status {'skipped': {}, 'ok': {'ec2-100-24-22-95.compute-1.amazonaws.com': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'ec2-100-24-22-95.compute-1.amazonaws.com': 1}, 'changed': {}}
[2024-04-16 13:01:30,771] [DEBUG] ANSIBLE: Playbook /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/cleanup.yml finished with status {'skipped': {}, 'ok': {'ec2-100-24-22-95.compute-1.amazonaws.com': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'ec2-100-24-22-95.compute-1.amazonaws.com': 1}, 'changed': {}}
[2024-04-16 13:01:30,953] [INFO] [423209] [ThreadPoolExecutor-0_0] [workflow_engine]: [run-agent-linux-ubuntu-20.04-amd64-tests] Finished task in 424.62 seconds.
[2024-04-16 13:01:31,445] [INFO] [423209] [ThreadPoolExecutor-0_2] [workflow_engine]: [run-agent-linux-ubuntu-18.04-amd64-tests] Finished task in 425.11 seconds.
[2024-04-16 13:01:31,451] [INFO] [423209] [MainThread] [workflow_engine]: Executing Reverse DAG tasks.
[2024-04-16 13:01:31,451] [INFO] [423209] [MainThread] [workflow_engine]: Executing tasks in parallel.
[2024-04-16 13:01:39,516] [INFO] [423209] [ThreadPoolExecutor-1_0] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Starting task.
[2024-04-16 13:01:39,516] [INFO] [423209] [ThreadPoolExecutor-1_1] [workflow_engine]: [allocate-agent-linux-ubuntu-20.04-amd64] Starting task.
[2024-04-16 13:01:39,518] [INFO] [423209] [ThreadPoolExecutor-1_2] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Starting task.
[2024-04-16 13:01:40,366] [INFO] ALLOCATOR: Deleting instance from trackfile /tmp/dtt1-poc/agent-linux-ubuntu-20.04-amd64/track.yaml
[2024-04-16 13:01:40,722] [INFO] ALLOCATOR: Deleting instance from trackfile /tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/track.yaml
[2024-04-16 13:01:41,119] [INFO] ALLOCATOR: Deleting instance from trackfile /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/track.yaml
[2024-04-16 13:01:41,383] [DEBUG] ALLOCATOR: Deleting credentials: ubuntu-20.04-amd64-key-8729
[2024-04-16 13:01:41,738] [DEBUG] ALLOCATOR: Deleting credentials: ubuntu-18.04-amd64-key-8935
[2024-04-16 13:01:42,072] [DEBUG] ALLOCATOR: Deleting credentials: ubuntu-22.04-amd64-key-8045
[2024-04-16 13:02:13,790] [INFO] ALLOCATOR: Instance i-06115b7d98a765e6e deleted.
[2024-04-16 13:02:14,437] [INFO] [423209] [ThreadPoolExecutor-1_1] [workflow_engine]: [allocate-agent-linux-ubuntu-20.04-amd64] Finished task in 34.92 seconds.
[2024-04-16 13:02:45,094] [INFO] ALLOCATOR: Instance i-05c2f0c68f28ab16e deleted.
[2024-04-16 13:02:45,760] [INFO] [423209] [ThreadPoolExecutor-1_2] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Finished task in 66.24 seconds.
[2024-04-16 13:03:00,109] [INFO] ALLOCATOR: Instance i-08bcf22d47fdf26bf deleted.
[2024-04-16 13:03:00,708] [INFO] [423209] [ThreadPoolExecutor-1_0] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Finished task in 81.19 seconds.

🆗 TEST 2 - One manager, 12 agents; threads=13

To run this workflow, I've used the branch fix/5197-logger-config to get a cleaner log.

workflow.yml file 
version: 0.1
description: This workflow is used to test agents deployment por DDT1 PoC
variables:
  agent-os:
    - linux-ubuntu-18.04-amd64
    - linux-ubuntu-20.04-amd64
    - linux-ubuntu-22.04-amd64
    - linux-debian-10-amd64
    - linux-debian-11-amd64
    - linux-debian-12-amd64
    - linux-oracle-9-amd64
    - linux-centos-7-amd64
    - linux-centos-8-amd64
    - linux-redhat-7-amd64
    - linux-redhat-8-amd64
    - linux-redhat-9-amd64
    - linux-amazon-2-amd64
  manager-os: linux-ubuntu-20.04-amd64
  infra-provider: aws
  working-dir: /tmp/dtt1-poc

tasks:
  # Unique manager allocate task
  - task: "allocate-manager-{manager-os}"
    description: "Allocate resources for the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: large
          - composite-name: "{manager-os}"
          - inventory-output: "{working-dir}/manager-{manager-os}/inventory.yaml"
          - track-output: "{working-dir}/manager-{manager-os}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    on-error: "abort-all"
    cleanup:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: delete
          - track-output: "{working-dir}/manager-{manager-os}/track.yaml"

  # Unique agent allocate task
  - task: "allocate-agent-{agent}"
    description: "Allocate resources for the agent."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: small
          - composite-name: "{agent}"
          - inventory-output: "{working-dir}/agent-{agent}/inventory.yaml"
          - track-output: "{working-dir}/agent-{agent}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    foreach:
      - variable: agent-os
        as: agent
    cleanup:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: delete
          - track-output: "{working-dir}/agent-{agent}/track.yaml"

  # Unique manager provision task
  - task: "provision-manager-{manager-os}"
    description: "Provision the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/provision/main.py
          - inventory: "{working-dir}/manager-{manager-os}/inventory.yaml"
          - install:
            - component: wazuh-manager
              type: assistant
              version: 4.7.3
              live: True
    depends-on:
      - "allocate-manager-{manager-os}"
    on-error: "abort-all"

  # Generic agent provision task
  - task: "provision-install-{agent}"
    description: "Provision resources for the {agent} agent."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/provision/main.py
          - inventory: "{working-dir}/agent-{agent}/inventory.yaml"
          - dependencies:
            - manager: "{working-dir}/manager-{manager-os}/inventory.yaml"
          - install:
            - component: wazuh-agent
              type: package
              version: 4.7.3
              live: True
    depends-on:
      - "allocate-agent-{agent}"
      - "provision-manager-{manager-os}"
    foreach:
      - variable: agent-os
        as: agent

  # Generic agent test task
  - task: "run-agent-{agent}-tests"
    description: "Run tests install for the agent {agent}."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/testing/main.py
          - targets:
            - wazuh-1: "{working-dir}/manager-{manager-os}/inventory.yaml"
            - agent: "{working-dir}/agent-{agent}/inventory.yaml"
          - tests: "stop"
          - component: "agent"
          - wazuh-version: "4.7.3"
          - wazuh-revision: "40714"
          - live: "True"
    foreach:
      - variable: agent-os
        as: agent
    depends-on:
      - "provision-install-{agent}"
Console output 
 cd /home/marcelo/wazuh/wazuh-qa/deployability/ ; (dtt-test) marcelo@marcelo-B460-AORUS-PRO-AC:~/wazuh/wazuh-qa/deployability$  cd /home/marcelo/wazuh/wazuh-qa/deployability/ ; /usr/bin/env /home/marcelo/.pyenv/versions/dtt-test/bin/python /home/marcelo/.vscode/extensions/ms-python.debugpy-2024.4.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher 42753 -- /home/marcelo/wazuh/wazuh-qa/deployability/modules/workflow_engine/__main__.py --threads 13 /home/marcelo/wazuh/test-2.yml 
[2024-04-16 16:54:48] [INFO] [456815] [MainThread] [workflow_engine]: Executing DAG tasks.
[2024-04-16 16:54:48] [INFO] [456815] [MainThread] [workflow_engine]: Executing tasks in parallel.
[2024-04-16 16:54:48] [INFO] [456815] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-20.04-amd64] Starting task.
[2024-04-16 16:54:48] [INFO] [456815] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Starting task.
[2024-04-16 16:54:48] [INFO] [456815] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-agent-linux-ubuntu-20.04-amd64] Starting task.
[2024-04-16 16:54:48] [INFO] [456815] [ThreadPoolExecutor-0_3] [workflow_engine]: [allocate-agent-linux-ubuntu-22.04-amd64] Starting task.
[2024-04-16 16:54:48] [INFO] [456815] [ThreadPoolExecutor-0_4] [workflow_engine]: [allocate-agent-linux-debian-10-amd64] Starting task.
[2024-04-16 16:54:48] [INFO] [456815] [ThreadPoolExecutor-0_5] [workflow_engine]: [allocate-agent-linux-debian-11-amd64] Starting task.
[2024-04-16 16:54:48] [INFO] [456815] [ThreadPoolExecutor-0_6] [workflow_engine]: [allocate-agent-linux-debian-12-amd64] Starting task.
[2024-04-16 16:54:48] [INFO] [456815] [ThreadPoolExecutor-0_7] [workflow_engine]: [allocate-agent-linux-oracle-9-amd64] Starting task.
[2024-04-16 16:54:48] [INFO] [456815] [ThreadPoolExecutor-0_8] [workflow_engine]: [allocate-agent-linux-centos-7-amd64] Starting task.
[2024-04-16 16:54:48] [INFO] [456815] [ThreadPoolExecutor-0_9] [workflow_engine]: [allocate-agent-linux-centos-8-amd64] Starting task.
[2024-04-16 16:54:48] [INFO] [456815] [ThreadPoolExecutor-0_10] [workflow_engine]: [allocate-agent-linux-redhat-7-amd64] Starting task.
[2024-04-16 16:54:48] [INFO] [456815] [ThreadPoolExecutor-0_11] [workflow_engine]: [allocate-agent-linux-redhat-8-amd64] Starting task.
[2024-04-16 16:54:48] [INFO] [456815] [ThreadPoolExecutor-0_12] [workflow_engine]: [allocate-agent-linux-redhat-9-amd64] Starting task.
[2024-04-16 16:55:12] [INFO] [456815] [ThreadPoolExecutor-0_11] [workflow_engine]: [allocate-agent-linux-redhat-8-amd64] Finished task in 23.51 seconds.
[2024-04-16 16:55:12] [INFO] [456815] [ThreadPoolExecutor-0_11] [workflow_engine]: [allocate-agent-linux-amazon-2-amd64] Starting task.
[2024-04-16 16:55:12] [INFO] [456815] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-20.04-amd64] Finished task in 23.85 seconds.
[2024-04-16 16:55:12] [INFO] [456815] [ThreadPoolExecutor-0_0] [workflow_engine]: [provision-manager-linux-ubuntu-20.04-amd64] Starting task.
[2024-04-16 16:55:12] [INFO] [456815] [ThreadPoolExecutor-0_8] [workflow_engine]: [allocate-agent-linux-centos-7-amd64] Finished task in 24.17 seconds.
[2024-04-16 16:55:13] [INFO] [456815] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-agent-linux-ubuntu-20.04-amd64] Finished task in 24.56 seconds.
[2024-04-16 16:55:13] [INFO] [456815] [ThreadPoolExecutor-0_7] [workflow_engine]: [allocate-agent-linux-oracle-9-amd64] Finished task in 24.57 seconds.
[2024-04-16 16:55:13] [INFO] [456815] [ThreadPoolExecutor-0_4] [workflow_engine]: [allocate-agent-linux-debian-10-amd64] Finished task in 25.12 seconds.
[2024-04-16 16:55:14] [INFO] [456815] [ThreadPoolExecutor-0_3] [workflow_engine]: [allocate-agent-linux-ubuntu-22.04-amd64] Finished task in 25.46 seconds.
[2024-04-16 16:55:14] [INFO] [456815] [ThreadPoolExecutor-0_6] [workflow_engine]: [allocate-agent-linux-debian-12-amd64] Finished task in 25.77 seconds.
[2024-04-16 16:55:14] [INFO] [456815] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Finished task in 26.09 seconds.
[2024-04-16 16:55:15] [INFO] [456815] [ThreadPoolExecutor-0_10] [workflow_engine]: [allocate-agent-linux-redhat-7-amd64] Finished task in 27.02 seconds.
[2024-04-16 16:55:15] [INFO] [456815] [ThreadPoolExecutor-0_12] [workflow_engine]: [allocate-agent-linux-redhat-9-amd64] Finished task in 27.31 seconds.
[2024-04-16 16:55:17] [INFO] [456815] [ThreadPoolExecutor-0_9] [workflow_engine]: [allocate-agent-linux-centos-8-amd64] Finished task in 28.81 seconds.
[2024-04-16 16:55:17] [INFO] [456815] [ThreadPoolExecutor-0_5] [workflow_engine]: [allocate-agent-linux-debian-11-amd64] Finished task in 28.85 seconds.
[2024-04-16 16:55:35] [INFO] [456815] [ThreadPoolExecutor-0_11] [workflow_engine]: [allocate-agent-linux-amazon-2-amd64] Finished task in 23.18 seconds.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_0] [workflow_engine]: [provision-manager-linux-ubuntu-20.04-amd64] Finished task in 292.66 seconds.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_8] [workflow_engine]: [provision-install-linux-ubuntu-18.04-amd64] Starting task.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_2] [workflow_engine]: [provision-install-linux-ubuntu-20.04-amd64] Starting task.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_7] [workflow_engine]: [provision-install-linux-ubuntu-22.04-amd64] Starting task.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_4] [workflow_engine]: [provision-install-linux-debian-10-amd64] Starting task.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_3] [workflow_engine]: [provision-install-linux-debian-11-amd64] Starting task.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_6] [workflow_engine]: [provision-install-linux-debian-12-amd64] Starting task.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_1] [workflow_engine]: [provision-install-linux-oracle-9-amd64] Starting task.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_10] [workflow_engine]: [provision-install-linux-centos-7-amd64] Starting task.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_12] [workflow_engine]: [provision-install-linux-centos-8-amd64] Starting task.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_9] [workflow_engine]: [provision-install-linux-redhat-7-amd64] Starting task.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_5] [workflow_engine]: [provision-install-linux-redhat-8-amd64] Starting task.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_11] [workflow_engine]: [provision-install-linux-redhat-9-amd64] Starting task.
[2024-04-16 17:00:05] [INFO] [456815] [ThreadPoolExecutor-0_0] [workflow_engine]: [provision-install-linux-amazon-2-amd64] Starting task.
[2024-04-16 17:00:56] [INFO] [456815] [ThreadPoolExecutor-0_4] [workflow_engine]: [provision-install-linux-debian-10-amd64] Finished task in 51.67 seconds.
[2024-04-16 17:00:56] [INFO] [456815] [ThreadPoolExecutor-0_4] [workflow_engine]: [run-agent-linux-debian-10-amd64-tests] Starting task.
[2024-04-16 17:01:44] [INFO] [456815] [ThreadPoolExecutor-0_0] [workflow_engine]: [provision-install-linux-amazon-2-amd64] Finished task in 99.12 seconds.
[2024-04-16 17:01:44] [INFO] [456815] [ThreadPoolExecutor-0_0] [workflow_engine]: [run-agent-linux-amazon-2-amd64-tests] Starting task.
[2024-04-16 17:01:59] [INFO] [456815] [ThreadPoolExecutor-0_3] [workflow_engine]: [provision-install-linux-debian-11-amd64] Finished task in 114.73 seconds.
[2024-04-16 17:01:59] [INFO] [456815] [ThreadPoolExecutor-0_3] [workflow_engine]: [run-agent-linux-debian-11-amd64-tests] Starting task.
[2024-04-16 17:02:05] [INFO] [456815] [ThreadPoolExecutor-0_4] [workflow_engine]: [run-agent-linux-debian-10-amd64-tests] Finished task in 68.48 seconds.
[2024-04-16 17:02:13] [INFO] [456815] [ThreadPoolExecutor-0_7] [workflow_engine]: [provision-install-linux-ubuntu-22.04-amd64] Finished task in 128.52 seconds.
[2024-04-16 17:02:13] [INFO] [456815] [ThreadPoolExecutor-0_4] [workflow_engine]: [run-agent-linux-ubuntu-22.04-amd64-tests] Starting task.
[2024-04-16 17:02:18] [INFO] [456815] [ThreadPoolExecutor-0_1] [workflow_engine]: [provision-install-linux-oracle-9-amd64] Finished task in 133.08 seconds.
[2024-04-16 17:02:18] [INFO] [456815] [ThreadPoolExecutor-0_7] [workflow_engine]: [run-agent-linux-oracle-9-amd64-tests] Starting task.
[2024-04-16 17:02:18] [INFO] [456815] [ThreadPoolExecutor-0_2] [workflow_engine]: [provision-install-linux-ubuntu-20.04-amd64] Finished task in 133.32 seconds.
[2024-04-16 17:02:18] [INFO] [456815] [ThreadPoolExecutor-0_1] [workflow_engine]: [run-agent-linux-ubuntu-20.04-amd64-tests] Starting task.
[2024-04-16 17:02:22] [INFO] [456815] [ThreadPoolExecutor-0_9] [workflow_engine]: [provision-install-linux-redhat-7-amd64] Finished task in 137.17 seconds.
[2024-04-16 17:02:22] [INFO] [456815] [ThreadPoolExecutor-0_2] [workflow_engine]: [run-agent-linux-redhat-7-amd64-tests] Starting task.
[2024-04-16 17:02:29] [INFO] [456815] [ThreadPoolExecutor-0_11] [workflow_engine]: [provision-install-linux-redhat-9-amd64] Finished task in 144.46 seconds.
[2024-04-16 17:02:29] [INFO] [456815] [ThreadPoolExecutor-0_9] [workflow_engine]: [run-agent-linux-redhat-9-amd64-tests] Starting task.
[2024-04-16 17:02:31] [INFO] [456815] [ThreadPoolExecutor-0_8] [workflow_engine]: [provision-install-linux-ubuntu-18.04-amd64] Finished task in 146.27 seconds.
[2024-04-16 17:02:31] [INFO] [456815] [ThreadPoolExecutor-0_11] [workflow_engine]: [run-agent-linux-ubuntu-18.04-amd64-tests] Starting task.
[2024-04-16 17:02:33] [INFO] [456815] [ThreadPoolExecutor-0_10] [workflow_engine]: [provision-install-linux-centos-7-amd64] Finished task in 147.76 seconds.
[2024-04-16 17:02:33] [INFO] [456815] [ThreadPoolExecutor-0_8] [workflow_engine]: [run-agent-linux-centos-7-amd64-tests] Starting task.
[2024-04-16 17:02:34] [INFO] [456815] [ThreadPoolExecutor-0_6] [workflow_engine]: [provision-install-linux-debian-12-amd64] Finished task in 149.23 seconds.
[2024-04-16 17:02:34] [INFO] [456815] [ThreadPoolExecutor-0_10] [workflow_engine]: [run-agent-linux-debian-12-amd64-tests] Starting task.
[2024-04-16 17:02:51] [INFO] [456815] [ThreadPoolExecutor-0_0] [workflow_engine]: [run-agent-linux-amazon-2-amd64-tests] Finished task in 66.99 seconds.
[2024-04-16 17:03:06] [INFO] [456815] [ThreadPoolExecutor-0_3] [workflow_engine]: [run-agent-linux-debian-11-amd64-tests] Finished task in 66.69 seconds.
[2024-04-16 17:03:08] [INFO] [456815] [ThreadPoolExecutor-0_5] [workflow_engine]: [provision-install-linux-redhat-8-amd64] Finished task in 183.17 seconds.
[2024-04-16 17:03:08] [INFO] [456815] [ThreadPoolExecutor-0_6] [workflow_engine]: [run-agent-linux-redhat-8-amd64-tests] Starting task.
[2024-04-16 17:03:10] [INFO] [456815] [ThreadPoolExecutor-0_12] [workflow_engine]: [provision-install-linux-centos-8-amd64] Finished task in 185.53 seconds.
[2024-04-16 17:03:10] [INFO] [456815] [ThreadPoolExecutor-0_3] [workflow_engine]: [run-agent-linux-centos-8-amd64-tests] Starting task.
[2024-04-16 17:03:19] [INFO] [456815] [ThreadPoolExecutor-0_4] [workflow_engine]: [run-agent-linux-ubuntu-22.04-amd64-tests] Finished task in 65.46 seconds.
[2024-04-16 17:03:23] [INFO] [456815] [ThreadPoolExecutor-0_7] [workflow_engine]: [run-agent-linux-oracle-9-amd64-tests] Finished task in 65.50 seconds.
[2024-04-16 17:03:24] [INFO] [456815] [ThreadPoolExecutor-0_1] [workflow_engine]: [run-agent-linux-ubuntu-20.04-amd64-tests] Finished task in 65.67 seconds.
[2024-04-16 17:03:28] [INFO] [456815] [ThreadPoolExecutor-0_2] [workflow_engine]: [run-agent-linux-redhat-7-amd64-tests] Finished task in 65.49 seconds.
[2024-04-16 17:03:35] [INFO] [456815] [ThreadPoolExecutor-0_9] [workflow_engine]: [run-agent-linux-redhat-9-amd64-tests] Finished task in 65.81 seconds.
[2024-04-16 17:03:36] [INFO] [456815] [ThreadPoolExecutor-0_11] [workflow_engine]: [run-agent-linux-ubuntu-18.04-amd64-tests] Finished task in 65.48 seconds.
[2024-04-16 17:03:38] [INFO] [456815] [ThreadPoolExecutor-0_8] [workflow_engine]: [run-agent-linux-centos-7-amd64-tests] Finished task in 65.15 seconds.
[2024-04-16 17:03:40] [INFO] [456815] [ThreadPoolExecutor-0_10] [workflow_engine]: [run-agent-linux-debian-12-amd64-tests] Finished task in 65.49 seconds.
[2024-04-16 17:04:16] [INFO] [456815] [ThreadPoolExecutor-0_6] [workflow_engine]: [run-agent-linux-redhat-8-amd64-tests] Finished task in 67.88 seconds.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-0_3] [workflow_engine]: [run-agent-linux-centos-8-amd64-tests] Finished task in 67.60 seconds.
[2024-04-16 17:04:18] [INFO] [456815] [MainThread] [workflow_engine]: Executing Reverse DAG tasks.
[2024-04-16 17:04:18] [INFO] [456815] [MainThread] [workflow_engine]: Executing tasks in parallel.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-1_0] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Starting task.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-1_1] [workflow_engine]: [allocate-agent-linux-ubuntu-20.04-amd64] Starting task.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-1_2] [workflow_engine]: [allocate-agent-linux-ubuntu-22.04-amd64] Starting task.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-1_3] [workflow_engine]: [allocate-agent-linux-debian-10-amd64] Starting task.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-1_4] [workflow_engine]: [allocate-agent-linux-debian-11-amd64] Starting task.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-1_5] [workflow_engine]: [allocate-agent-linux-debian-12-amd64] Starting task.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-1_6] [workflow_engine]: [allocate-agent-linux-oracle-9-amd64] Starting task.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-1_7] [workflow_engine]: [allocate-agent-linux-centos-7-amd64] Starting task.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-1_8] [workflow_engine]: [allocate-agent-linux-centos-8-amd64] Starting task.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-1_9] [workflow_engine]: [allocate-agent-linux-redhat-7-amd64] Starting task.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-1_10] [workflow_engine]: [allocate-agent-linux-redhat-8-amd64] Starting task.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-1_11] [workflow_engine]: [allocate-agent-linux-redhat-9-amd64] Starting task.
[2024-04-16 17:04:18] [INFO] [456815] [ThreadPoolExecutor-1_12] [workflow_engine]: [allocate-agent-linux-amazon-2-amd64] Starting task.
[2024-04-16 17:05:09] [INFO] [456815] [ThreadPoolExecutor-1_2] [workflow_engine]: [allocate-agent-linux-ubuntu-22.04-amd64] Finished task in 50.55 seconds.
[2024-04-16 17:05:09] [INFO] [456815] [ThreadPoolExecutor-1_2] [workflow_engine]: [allocate-manager-linux-ubuntu-20.04-amd64] Starting task.
[2024-04-16 17:05:09] [INFO] [456815] [ThreadPoolExecutor-1_1] [workflow_engine]: [allocate-agent-linux-ubuntu-20.04-amd64] Finished task in 50.57 seconds.
[2024-04-16 17:05:09] [INFO] [456815] [ThreadPoolExecutor-1_4] [workflow_engine]: [allocate-agent-linux-debian-11-amd64] Finished task in 51.19 seconds.
[2024-04-16 17:05:11] [INFO] [456815] [ThreadPoolExecutor-1_0] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Finished task in 52.82 seconds.
[2024-04-16 17:05:11] [INFO] [456815] [ThreadPoolExecutor-1_8] [workflow_engine]: [allocate-agent-linux-centos-8-amd64] Finished task in 52.81 seconds.
[2024-04-16 17:05:11] [INFO] [456815] [ThreadPoolExecutor-1_9] [workflow_engine]: [allocate-agent-linux-redhat-7-amd64] Finished task in 53.40 seconds.
[2024-04-16 17:05:12] [INFO] [456815] [ThreadPoolExecutor-1_11] [workflow_engine]: [allocate-agent-linux-redhat-9-amd64] Finished task in 54.02 seconds.
[2024-04-16 17:05:25] [INFO] [456815] [ThreadPoolExecutor-1_7] [workflow_engine]: [allocate-agent-linux-centos-7-amd64] Finished task in 67.13 seconds.
[2024-04-16 17:05:25] [INFO] [456815] [ThreadPoolExecutor-1_5] [workflow_engine]: [allocate-agent-linux-debian-12-amd64] Finished task in 67.14 seconds.
[2024-04-16 17:05:27] [INFO] [456815] [ThreadPoolExecutor-1_12] [workflow_engine]: [allocate-agent-linux-amazon-2-amd64] Finished task in 69.32 seconds.
[2024-04-16 17:05:39] [INFO] [456815] [ThreadPoolExecutor-1_3] [workflow_engine]: [allocate-agent-linux-debian-10-amd64] Finished task in 81.47 seconds.
[2024-04-16 17:05:42] [INFO] [456815] [ThreadPoolExecutor-1_10] [workflow_engine]: [allocate-agent-linux-redhat-8-amd64] Finished task in 83.99 seconds.
[2024-04-16 17:05:56] [INFO] [456815] [ThreadPoolExecutor-1_6] [workflow_engine]: [allocate-agent-linux-oracle-9-amd64] Finished task in 98.34 seconds.
[2024-04-16 17:05:59] [INFO] [456815] [ThreadPoolExecutor-1_2] [workflow_engine]: [allocate-manager-linux-ubuntu-20.04-amd64] Finished task in 50.52 seconds
worflow.log file

workflow.log

Execution time

1 Manager N Agent Allocation (Time) Provision (Time) Testing (Time) Cleanup (Time) Total (Time)
1 12 00:46 05:43 03:20 01:41 11:11

@mhamra
Copy link
Contributor

mhamra commented Apr 16, 2024

UPDATE - STRANGE BEHAVIOR

Sometimes, when I run workflows, whether I use the threads parameter or not, the allocation and clean-up tasks are executed OK, but the workflow does not perform the provision and the testing tasks. Here's a log file that shows the execution:

workflow-3.log

I could not find a pattern to reproduce the behavior.

@wazuhci wazuhci moved this from In progress to On hold in Release 4.9.0 Apr 17, 2024
@fcaffieri
Copy link
Member Author

@mhamra you could execute an execution with the following input and parameters:

workflow.yml file 
version: 0.1
description: This workflow is used to test agents' deployment for DDT1 PoC
variables:
  agent-os:
    - linux-ubuntu-18.04-amd64
    - linux-ubuntu-20.04-amd64
    - linux-ubuntu-22.04-amd64
    - linux-debian-10-amd64
    - linux-debian-11-amd64
    - linux-debian-12-amd64
    - linux-oracle-9-amd64
    - linux-centos-7-amd64
    - linux-centos-8-amd64
    - linux-redhat-7-amd64
    - linux-redhat-8-amd64
    - linux-redhat-9-amd64
    - linux-amazon-2-amd64
  manager-os: linux-ubuntu-22.04-amd64
  infra-provider: aws
  working-dir: /tmp/dtt1-poc

tasks:
  # Unique manager allocate task
  - task: "allocate-manager-{manager-os}"
    description: "Allocate resources for the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: large
          - composite-name: "{manager-os}"
          - inventory-output: "{working-dir}/manager-{manager-os}/inventory.yaml"
          - track-output: "{working-dir}/manager-{manager-os}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    on-error: "abort-all"
    cleanup:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: delete
          - track-output: "{working-dir}/manager-{manager-os}/track.yaml"

  # Unique agent allocate task
  - task: "allocate-agent-{agent}"
    description: "Allocate resources for the agent."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: small
          - composite-name: "{agent}"
          - inventory-output: "{working-dir}/agent-{agent}/inventory.yaml"
          - track-output: "{working-dir}/agent-{agent}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    on-error: "abort-all"
    foreach:
      - variable: agent-os
        as: agent
    cleanup:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: delete
          - track-output: "{working-dir}/agent-{agent}/track.yaml"
    depends-on:
      - "provision-manager-{manager-os}"

  # Unique manager provision task
  - task: "provision-manager-{manager-os}"
    description: "Provision the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/provision/main.py
          - inventory: "{working-dir}/manager-{manager-os}/inventory.yaml"
          - install:
            - component: wazuh-manager
              type: assistant
              version: 4.7.3
              live: True
    depends-on:
      - "allocate-manager-{manager-os}"
    on-error: "abort-all"

  # Generic agent test task
  - task: "run-agent-{agent}-tests"
    description: "Run tests install for the agent {agent}."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/testing/main.py
          - targets:
            - wazuh-1: "{working-dir}/manager-{manager-os}/inventory.yaml"
            - agent: "{working-dir}/agent-{agent}/inventory.yaml"
          - tests: "install,registration,restart,stop,uninstall"
          - component: "agent"
          - wazuh-version: "4.7.3"
          - wazuh-revision: "40714"
          - live: "True"
    foreach:
      - variable: agent-os
        as: agent
    depends-on:
      - "allocate-agent-{agent}"
Execution command:

Show dry-run:

python3 -m workflow_engine input_yaml.yaml --log-level DEBUG --dry-run

Execute test:

python3 -m workflow_engine input_yaml.yaml --log-level DEBUG

@wazuhci wazuhci moved this from On hold to In progress in Release 4.9.0 Apr 17, 2024
@mhamra
Copy link
Contributor

mhamra commented Apr 19, 2024 •
edited
Loading

UPDATE

I've run the workflow file provided by @fcaffieri with the option --threads and without it.

Here are the results

file threads 1 threads 12
command console output not saved console-log-threads-12.txt
workflow log file workflow-threads-1.log workflow-threads-12.log
dry-run log workflow-dry-run-1.log workflow-dry-run-12.log

Analysis

  1. The dry-run log file comparison shows that the dependencies are equivalent. All testing tasks have these dependencies:

    test agent -> allocate agent -> provision manager -> allocate manager. The files only differ in the order of execution of the test tasks.

  2. The manager provisioning task takes 1.67 seconds to finish and 2.13 seconds to run the workflow with 12 threads, while the provision of a manager takes more time.

  3. In the workflow with one thread, the first agent allocation started at 14:33:30,204, and the last ended at 14:38:23,721. So, the whole allocation process took 4:53 min. The workflow executed the first test 10 ms after at 14:38:23,731.

  4. In the workflow with 12 threads, the manager provision task ended at 20:06:00,730, and the first test started at 20:06:22,835, only 22 seconds later. The test failed because the manager had yet to finish the installation. The task run-agent-linux-debian-10-amd64-tests started at 20:10:42,948, 4:42 minutes after the manager provision task. All the other run-agent tasks began almost at the same time.

  5. The task run-agent-linux-oracle-9-amd64-tests does not appear in the 12-thread workflow log file but in the one-thread workflow log file because the Oracle Linux agent allocation failed.

Conclusion

The manager provisioning task runs the initialization script and exits without waiting for the manager installation to finish. While this approach is good because it allows the continuation of other tasks in parallel, the agent test should start after the manager installation finishes. To check the manager installation is finished and the manager is up and running, I think that we can implement one of these options:

  • Create a task with an estimated wait time (not good)
  • Create a task that periodically checks if the manager is up and running (better)
  • Create a task that waits for a message confirming that the manager is up and running. The manager can send the message to a queue. The workflow task ends when the message is received.

@mhamra
Copy link
Contributor

mhamra commented Apr 19, 2024 •
edited
Loading

UPDATE

I've modified the workflow, adding a task that waits 5 minutes after provisioning the manager. All the tests depend on that task and the agent allocation.

file threads 12 with wait
command console output console-log-threads-12-with-wait.txt
workflow log file workflow-threads-12-with-wait.log
dry-run log workflow-dry-run-12-with-wait.log

Analysis

After adding a delay task, some agent tests finished well, but others failed. The delay may have been shorter than needed to finish the manager installation.

@rauldpm
Copy link
Member

rauldpm commented Apr 19, 2024 •
edited
Loading

External test

@fcaffieri
Copy link
Member Author

fcaffieri commented Apr 19, 2024 •
edited
Loading

New analysis

Analyzing the executions and the logs, I notice that the problem occurs because the provision cannot connect to the VM that raised the allocation, throwing the error:

TASK [Gathering Facts] *********************************************************
�[1;31mfatal: [ec2-3-80-51-50.compute-1.amazonaws.com]: UNREACHABLE! => changed=false �[0m
�[1;31m  msg: 'Failed to connect to the host via ssh: ssh: connect to host ec2-3-80-51-50.compute-1.amazonaws.com port 2200: Connection refused'�[0m
�[1;31m  unreachable: true�[0m

PLAY RECAP *********************************************************************
�[0;31mec2-3-80-51-50.compute-1.amazonaws.com�[0m : ok=0    changed=0    �[1;31munreachable=1   �[0m failed=0    skipped=0    rescued=0    ignored=0   
�[37m[2024-04-18 20:06:00] [DEBUG] PROVISIONER: Playbook {'hosts': 'ec2-3-80-51-50.compute-1.amazonaws.com', 'become': True, 'gather_facts': True, 'tasks': [{'name': 'Install the required packages', 'shell': ''}, {'name': 'Download the Wazuh installation assistant', 'shell': 'curl -sO https://packages.wazuh.com/4.7/wazuh-install.sh'}, {'name': 'Install wazuh-manager with assistant', 'shell': 'bash ./wazuh-install.sh -a -i'}]} finished with status {'skipped': {}, 'ok': {}, 'dark': {'ec2-3-80-51-50.compute-1.amazonaws.com': 1}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'ec2-3-80-51-50.compute-1.amazonaws.com': 1}, 'changed': {}}�[0m
�[32m[2024-04-18 20:06:00] [INFO] PROVISIONER: Provision of "wazuh-manager" complete successfully.�[0m
�[32m[2024-04-18 20:06:00] [INFO] PROVISIONER: All components provisioned successfully.�[0m
�[37m[2024-04-18 20:06:00] [DEBUG] PROVISIONER: Provision summary: {'skipped': {}, 'ok': {}, 'dark': {'ec2-3-80-51-50.compute-1.amazonaws.com': 1}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'ec2-3-80-51-50.compute-1.amazonaws.com': 1}, 'changed': {}}�[0m

[2024-04-18 20:06:00,730] [INFO] [3958] [ThreadPoolExecutor-0_0] [workflow_engine]: [provision-manager-linux-ubuntu-22.04-amd64] Finished task in 2.15 seconds.
[2024-04-18 20:06:00,740] [INFO] [3958] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-agent-linux-ubuntu-18.04-amd64] Starting task.

The problem is that the VM, even though it was created by the allocator, is still not available in AWS to connect. This was fixed from the allocation module in this issue: #5198
A connection check was incorporated to the raised VM, to guarantee that said VM is available before completing the allocation execution.

That said, tests were carried out both with and without threads and satisfactory results were obtained:

file 3 threads
log workflow.log
input test_agent.yaml.txt
file 1 threads
log workflow.log
input test_agent.yaml.txt

This was referenced Apr 19, 2024
Open
Open
@mhamra
Copy link
Contributor

mhamra commented Apr 19, 2024

CONCLUSION

@wazuhci wazuhci moved this from In progress to Pending review in Release 4.9.0 Apr 19, 2024
@rauldpm
Copy link
Member

rauldpm commented Apr 19, 2024

LGTM!

@wazuhci wazuhci moved this from Pending review to Pending final review in Release 4.9.0 Apr 19, 2024
@fcaffieri
Copy link
Member Author

LGTM

@wazuhci wazuhci moved this from Pending final review to Done in Release 4.9.0 Apr 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mhamra mhamra

Labels
level/task Task issue type/enhancement
Projects
No open projects
Release 4.9.0
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mhamra @rauldpm @fcaffieri