DTT1-The allocator assigns the same IP address to many instances launching tasks in parallel #5237
Comments
|
This error only affects local deployments with The problem occurs because currently, the The fix that occurs to me for this is to directly remove this method and configure the |
Update reportI was testing changing the approach as we had thought, letting cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-20.04-amd64
[2024-04-22 14:40:08] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-04-22 14:40:08] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-04-22 14:40:08] [DEBUG] ALLOCATOR: Generating new key pair
[2024-04-22 14:40:09] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance.
[2024-04-22 14:40:09] [INFO] ALLOCATOR: Instance VAGRANT-74FAB66C-614B-430D-AC0C-8047C9229FE9 created.
[2024-04-22 14:40:50] [INFO] ALLOCATOR: Instance VAGRANT-74FAB66C-614B-430D-AC0C-8047C9229FE9 started.
[email protected]'s password:
[email protected]'s password:
I was looking for some alternatives but I can't find the solution, this happens with different boxes |
Update reportAfter validating and testing different alternatives, all of them present a possible solution but various problems, I will try to clarify here all the possible solutions that were addressed: Control fileWe were trying to solve this with a control file, the idea is to create a file and record the busy IPs, and have the threads block the reading and writing of this file to prevent the same IP from being assigned to two or more machines. The problem we find with this approach is where to keep this control file, since the working directory is configurable and variable, the VM directory is not valid since it maintains the VM's information, the option that is closest to What we need is the same directory where the module is located, the problem with this is that this does not prevent the module from being found in different directories and modules being executed from different directories, therefore, we also lose control. The same approach that we use in macStadiumAt first we thought that the solution was simple, let Vagrant and Virtualbox take care of assigning the IP, and then obtain it with the cbordon@cbordon-MS-7C88:/tmp/wazuh-qa/VAGRANT-A3AEAF0B-8EDA-4B11-8330-E553AEC34823$ vagrant ssh-config
Host default
HostName 127.0.0.1
User vagrant
Port 22
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
PasswordAuthentication no
IdentityFile /tmp/wazuh-qa/VAGRANT-A3AEAF0B-8EDA-4B11-8330-E553AEC34823/instance_key
IdentitiesOnly yes
LogLevel FATAL
ForwardAgent yesUpon detecting this, we tried to obtain the IP of the private network by accessing the VM, first, we tried using ssh -o 'StrictHostKeyChecking no' -i /tmp/wazuh-qa/VAGRANT-D0EE6388-19D0-45C9-94D3-E53A23CAD916/instance_key -p 2201 [email protected] ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:60:50:11:b6:59 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 86393sec preferred_lft 86393sec
inet6 fe80::60:50ff:fe11:b659/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:60:e3:39 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.40/24 brd 192.168.56.255 scope global dynamic enp0s8
valid_lft 593sec preferred_lft 593sec
inet6 fe80::a00:27ff:fe60:e339/64 scope link
valid_lft forever preferred_lft foreverRandom IPAfter discussing it with @fcaffieri and @teddytpc1 and understanding that this problem only occurs in local deployments with Vagrant, where due to the Hardware resources that each one normally manages in their Workstations, the possibility of deploying multiple virtual machines on the machine itself It is usually not possible (generally with a workstation with 32 GB of RAM you cannot create more than 5 VMs) we believe that random assignment of the last octet of the IP address can be a solution, although it greatly reduces the possibility of coincidences. In the assignment of IPs, the possibility still exists. But we understand that it is the solution that best approaches what we need, considering that we are not going to build many machines locally and that we have 254 allocation possibilities. |
|
The test is performed using this yaml: workflow.yamlversion: 0.1
description: This workflow is used to test manager deployment for DDT1 PoC
variables:
manager-os:
- linux-ubuntu-20.04-amd64
- linux-ubuntu-22.04-amd64
- linux-oracle-9-amd64
# - linux-amazon-2-amd64
# - linux-redhat-7-amd64
# - linux-redhat-8-amd64
# - linux-redhat-9-amd64
# - linux-centos-7-amd64
# - linux-centos-8-amd64
# - linux-debian-10-amd64
# - linux-debian-11-amd64
# - linux-debian-12-amd64
infra-provider: vagrant
working-dir: /tmp/dtt1-poc
tasks:
# Unique manager allocate task
- task: "allocate-manager-{manager}"
description: "Allocate resources for the manager."
do:
this: process
with:
path: python3
args:
- modules/allocation/main.py
- action: create
- provider: "{infra-provider}"
- size: large
- composite-name: "{manager}"
- inventory-output: "{working-dir}/manager-{manager}/inventory.yaml"
- track-output: "{working-dir}/manager-{manager}/track.yaml"
on-error: "abort-all"
foreach:
- variable: manager-os
as: manager
# Generic manager test task
- task: "run-manager-tests"
description: "Run tests install for the manager."
do:
this: process
with:
path: python3
args:
- modules/testing/main.py
- targets:
- wazuh-1: "{working-dir}/manager-linux-ubuntu-20.04-amd64/inventory.yaml"
- wazuh-2: "{working-dir}/manager-linux-ubuntu-22.04-amd64/inventory.yaml"
- wazuh-3: "{working-dir}/manager-linux-oracle-9-amd64/inventory.yaml"
# - wazuh-4: "{working-dir}/manager-linux-centos-7-amd64/inventory.yaml"
# - wazuh-5: "{working-dir}/manager-linux-amazon-2-amd64/inventory.yaml"
# - wazuh-6: "{working-dir}/manager-linux-redhat-7-amd64/inventory.yaml"
# - wazuh-7: "{working-dir}/manager-linux-redhat-8-amd64/inventory.yaml"
# - wazuh-8: "{working-dir}/manager-linux-redhat-9-amd64/inventory.yaml"
# - wazuh-9: "{working-dir}/manager-linux-centos-8-amd64/inventory.yaml"
# - wazuh-10: "{working-dir}/manager-linux-debian-10-amd64/inventory.yaml"
# - wazuh-11: "{working-dir}/manager-linux-debian-11-amd64/inventory.yaml"
# - wazuh-12: "{working-dir}/manager-linux-debian-12-amd64/inventory.yaml"
- tests: "install,restart,stop,uninstall"
- component: "manager"
- wazuh-version: "4.7.3"
- wazuh-revision: "40714"
- live: "True"
depends-on:
- "allocate-manager-linux-ubuntu-20.04-amd64"
- "allocate-manager-linux-ubuntu-22.04-amd64"
- "allocate-manager-linux-oracle-9-amd64"
Result:cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa/deployability$ python3 modules/workflow_engine/__main__.py modules/workflow_engine/examples/testing_threats.yaml --threads 3
[2024-04-23 16:14:28] [INFO] [615750] [MainThread] [workflow_engine]: Executing DAG tasks.
[2024-04-23 16:14:28] [INFO] [615750] [MainThread] [workflow_engine]: Executing tasks in parallel.
[2024-04-23 16:14:28] [INFO] [615750] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-20.04-amd64] Starting task.
[2024-04-23 16:14:28] [INFO] [615750] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Starting task.
[2024-04-23 16:14:28] [INFO] [615750] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-manager-linux-oracle-9-amd64] Starting task.
[2024-04-23 16:15:29] [INFO] [615750] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-20.04-amd64] Finished task in 61.48 seconds.
[2024-04-23 16:15:33] [INFO] [615750] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Finished task in 64.94 seconds.
[2024-04-23 16:15:39] [INFO] [615750] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-manager-linux-oracle-9-amd64] Finished task in 71.33 seconds.
[2024-04-23 16:15:39] [INFO] [615750] [ThreadPoolExecutor-0_0] [workflow_engine]: [run-manager-tests] Starting task.
[2024-04-23 16:15:57] [INFO] [615750] [ThreadPoolExecutor-0_0] [workflow_engine]: [run-manager-tests] Finished task in 17.64 seconds.
[2024-04-23 16:15:57] [INFO] [615750] [MainThread] [workflow_engine]: Executing Reverse DAG tasks.
[2024-04-23 16:15:57] [INFO] [615750] [MainThread] [workflow_engine]: Executing tasks in parallel.
Inventories:cbordon@cbordon-MS-7C88:~$ cat /tmp/dtt1-poc/manager-linux-oracle-9-amd64/inventory.yaml
ansible_connection: ssh
ansible_host: 192.168.57.230
ansible_port: 22
ansible_ssh_common_args: -o StrictHostKeyChecking=no
ansible_ssh_private_key_file: /tmp/wazuh-qa/VAGRANT-0EA9AC84-2CF4-483C-991F-A2947209675B/instance_key
ansible_user: vagrant
cbordon@cbordon-MS-7C88:~$ cat /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml
ansible_connection: ssh
ansible_host: 192.168.57.244
ansible_port: 22
ansible_ssh_common_args: -o StrictHostKeyChecking=no
ansible_ssh_private_key_file: /tmp/wazuh-qa/VAGRANT-1AA3392D-9D8B-42AD-83F8-1FE5024A552E/instance_key
ansible_user: vagrant
cbordon@cbordon-MS-7C88:~$ cat /tmp/dtt1-poc/manager-linux-ubuntu-20.04-amd64/inventory.yaml
ansible_connection: ssh
ansible_host: 192.168.57.59
ansible_port: 22
ansible_ssh_common_args: -o StrictHostKeyChecking=no
ansible_ssh_private_key_file: /tmp/wazuh-qa/VAGRANT-BB327E15-E57A-4B74-B926-2B3A8E611BE3/instance_key
ansible_user: vagrant |
|
New test with 5 threads: workflow.yamlversion: 0.1
description: This workflow is used to test manager deployment for DDT1 PoC
variables:
manager-os:
- linux-ubuntu-20.04-amd64
- linux-ubuntu-22.04-amd64
- linux-oracle-9-amd64
- linux-amazon-2-amd64
- linux-redhat-7-amd64
# - linux-redhat-8-amd64
# - linux-redhat-9-amd64
# - linux-centos-7-amd64
# - linux-centos-8-amd64
# - linux-debian-10-amd64
# - linux-debian-11-amd64
# - linux-debian-12-amd64
infra-provider: vagrant
working-dir: /tmp/dtt1-poc
tasks:
# Unique manager allocate task
- task: "allocate-manager-{manager}"
description: "Allocate resources for the manager."
do:
this: process
with:
path: python3
args:
- modules/allocation/main.py
- action: create
- provider: "{infra-provider}"
- size: large
- composite-name: "{manager}"
- inventory-output: "{working-dir}/manager-{manager}/inventory.yaml"
- track-output: "{working-dir}/manager-{manager}/track.yaml"
on-error: "abort-all"
foreach:
- variable: manager-os
as: manager
# Generic manager test task
- task: "run-manager-tests"
description: "Run tests install for the manager."
do:
this: process
with:
path: python3
args:
- modules/testing/main.py
- targets:
- wazuh-1: "{working-dir}/manager-linux-ubuntu-20.04-amd64/inventory.yaml"
- wazuh-2: "{working-dir}/manager-linux-ubuntu-22.04-amd64/inventory.yaml"
- wazuh-3: "{working-dir}/manager-linux-oracle-9-amd64/inventory.yaml"
- wazuh-4: "{working-dir}/manager-linux-amazon-2-amd64/inventory.yaml"
- wazuh-5: "{working-dir}/manager-linux-redhat-7-amd64/inventory.yaml"
# - wazuh-6: "{working-dir}/manager-linux-redhat-7-amd64/inventory.yaml"
# - wazuh-7: "{working-dir}/manager-linux-redhat-8-amd64/inventory.yaml"
# - wazuh-8: "{working-dir}/manager-linux-redhat-9-amd64/inventory.yaml"
# - wazuh-9: "{working-dir}/manager-linux-centos-8-amd64/inventory.yaml"
# - wazuh-10: "{working-dir}/manager-linux-debian-10-amd64/inventory.yaml"
# - wazuh-11: "{working-dir}/manager-linux-debian-11-amd64/inventory.yaml"
# - wazuh-12: "{working-dir}/manager-linux-debian-12-amd64/inventory.yaml"
- tests: "install,restart,stop,uninstall"
- component: "manager"
- wazuh-version: "4.7.3"
- wazuh-revision: "40714"
- live: "True"
depends-on:
- "allocate-manager-linux-ubuntu-20.04-amd64"
- "allocate-manager-linux-ubuntu-22.04-amd64"
- "allocate-manager-linux-oracle-9-amd64"
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa/deployability$ python3 modules/workflow_engine/__main__.py modules/workflow_engine/examples/testing_threats.yaml --threads 5
[2024-04-23 16:31:25] [INFO] [667823] [MainThread] [workflow_engine]: Executing DAG tasks.
[2024-04-23 16:31:25] [INFO] [667823] [MainThread] [workflow_engine]: Executing tasks in parallel.
[2024-04-23 16:31:25] [INFO] [667823] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-20.04-amd64] Starting task.
[2024-04-23 16:31:25] [INFO] [667823] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Starting task.
[2024-04-23 16:31:25] [INFO] [667823] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-manager-linux-oracle-9-amd64] Starting task.
[2024-04-23 16:31:25] [INFO] [667823] [ThreadPoolExecutor-0_3] [workflow_engine]: [allocate-manager-linux-amazon-2-amd64] Starting task.
[2024-04-23 16:31:25] [INFO] [667823] [ThreadPoolExecutor-0_4] [workflow_engine]: [allocate-manager-linux-redhat-7-amd64] Starting task.
[2024-04-23 16:32:35] [INFO] [667823] [ThreadPoolExecutor-0_4] [workflow_engine]: [allocate-manager-linux-redhat-7-amd64] Finished task in 69.69 seconds.
[2024-04-23 16:32:39] [INFO] [667823] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Finished task in 73.99 seconds.
[2024-04-23 16:32:40] [INFO] [667823] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-20.04-amd64] Finished task in 74.78 seconds.
[2024-04-23 16:32:44] [INFO] [667823] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-manager-linux-oracle-9-amd64] Finished task in 78.56 seconds.
[2024-04-23 16:32:44] [INFO] [667823] [ThreadPoolExecutor-0_4] [workflow_engine]: [run-manager-tests] Starting task.
[2024-04-23 16:32:44] [ERROR] [667823] [ThreadPoolExecutor-0_4] [workflow_engine]: [run-manager-tests] Task failed with error: Error executing process task Traceback (most recent call last):
File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/testing/main.py", line 30, in <module>
Tester.run(InputPayload(**vars(parse_arguments())))
File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/testing/testing.py", line 40, in run
inventory = Inventory(**Utils.load_from_yaml(', '.join(dictionary.values())))
File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/generic/utils.py", line 52, in load_from_yaml
raise FileNotFoundError(f'File "{file_path}" not found.')
FileNotFoundError: File "/tmp/dtt1-poc/manager-linux-amazon-2-amd64/inventory.yaml" not found.
.
[2024-04-23 16:33:07] [INFO] [667823] [ThreadPoolExecutor-0_3] [workflow_engine]: [allocate-manager-linux-amazon-2-amd64] Finished task in 101.82 seconds.
[2024-04-23 16:33:07] [INFO] [667823] [MainThread] [workflow_engine]: Executing Reverse DAG tasks.
[2024-04-23 16:33:07] [INFO] [667823] [MainThread] [workflow_engine]: Executing tasks in parallel.The error apparently occurs because the test is attempted to be executed before the machine is finished provisioning, since the provisioning is done correctly. Inventories.ymlcbordon@cbordon-MS-7C88:/tmp/dtt1-poc$ cat manager-linux-amazon-2-amd64/inventory.yaml
ansible_connection: ssh
ansible_host: 192.168.57.129
ansible_port: 22
ansible_ssh_common_args: -o StrictHostKeyChecking=no
ansible_ssh_private_key_file: /tmp/wazuh-qa/VAGRANT-714825EF-42CD-4B8A-BC13-147FFED5EA15/instance_key
ansible_user: vagrant
cbordon@cbordon-MS-7C88:/tmp/dtt1-poc$ cat manager-linux-oracle-9-amd64/inventory.yaml
ansible_connection: ssh
ansible_host: 192.168.57.39
ansible_port: 22
ansible_ssh_common_args: -o StrictHostKeyChecking=no
ansible_ssh_private_key_file: /tmp/wazuh-qa/VAGRANT-3EC1B0AF-5540-451B-AC8B-63B1B95A6021/instance_key
ansible_user: vagrant
cbordon@cbordon-MS-7C88:/tmp/dtt1-poc$ cat manager-linux-redhat-7-amd64/inventory.yaml
ansible_connection: ssh
ansible_host: 192.168.57.72
ansible_port: 22
ansible_ssh_common_args: -o StrictHostKeyChecking=no
ansible_ssh_private_key_file: /tmp/wazuh-qa/VAGRANT-017DABF6-327F-43BA-83E8-90AD75FC8CDD/instance_key
ansible_user: vagrant
cbordon@cbordon-MS-7C88:/tmp/dtt1-poc$ cat manager-linux-ubuntu-20.04-amd64/inventory.yaml
ansible_connection: ssh
ansible_host: 192.168.57.137
ansible_port: 22
ansible_ssh_common_args: -o StrictHostKeyChecking=no
ansible_ssh_private_key_file: /tmp/wazuh-qa/VAGRANT-FFB50910-A1AA-4F12-BC47-FC6A03E5CC8B/instance_key
ansible_user: vagrant
cbordon@cbordon-MS-7C88:/tmp/dtt1-poc$ cat manager-linux-ubuntu-22.04-amd64/inventory.yaml
ansible_connection: ssh
ansible_host: 192.168.57.64
ansible_port: 22
ansible_ssh_common_args: -o StrictHostKeyChecking=no
ansible_ssh_private_key_file: /tmp/wazuh-qa/VAGRANT-52293A82-347F-4C19-9EB3-2F6A010BE703/instance_key
ansible_user: vagrant |
|
LGTM |
Description
Running this workflow file with the
--threads 3workflow.yaml
the testing module logs that the instances have the same IP address, which causes the
sshlogin to fail.workflow.log
These are the inventory files generated by the allocator:
inventory.zip
All of them have the same IP address.
The text was updated successfully, but these errors were encountered: