Validation of system dependencies is added to the Allocation module #5336

c-bordon · 2024-05-03T15:51:50Z

Validation of system dependencies is added for the correct functioning of the Allocation module. With this we validate that the packages that are necessary for operation are installed prior to attempting to create the VMs. With this, errors were discovered when trying to run the module on a Centos 7, here the error was in the python dependencies.

Test Ubuntu 22.04

vagrant@ubuntu-jammy:~$ git clone https://github.com/wazuh/wazuh-qa.git
Cloning into 'wazuh-qa'...
remote: Enumerating objects: 105176, done.
remote: Counting objects: 100% (7687/7687), done.
remote: Compressing objects: 100% (2581/2581), done.
remote: Total 105176 (delta 5045), reused 7324 (delta 4773), pack-reused 97489
Receiving objects: 100% (105176/105176), 50.23 MiB | 8.17 MiB/s, done.
Resolving deltas: 100% (66414/66414), done.
vagrant@ubuntu-jammy:~$ cd wazuh-qa/
vagrant@ubuntu-jammy:~/wazuh-qa$ git checkout bug/5295-dtt1-allocator-module-the-allocation-fails-if-the-sshpass-utility-is-not-installed
Branch 'bug/5295-dtt1-allocator-module-the-allocation-fails-if-the-sshpass-utility-is-not-installed' set up to track remote branch 'bug/5295-dtt1-allocator-module-the-allocation-fails-if-the-sshpass-utility-is-not-installed' from 'origin'.
Switched to a new branch 'bug/5295-dtt1-allocator-module-the-allocation-fails-if-the-sshpass-utility-is-not-installed'
vagrant@ubuntu-jammy:~/wazuh-qa$ git pull
Already up to date.
vagrant@ubuntu-jammy:~/wazuh-qa$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
vagrant@ubuntu-jammy:~/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size small --composite-name linux-centos-7-amd64
[2024-05-03 14:47:24] [DEBUG] SPNEGO._GSS: Python gssapi not available, cannot use any GSSAPIProxy protocols: No module named 'gssapi'
[2024-05-03 14:47:24] [DEBUG] SPNEGO._GSS: Python gssapi IOV extension not available: No module named 'gssapi'
[2024-05-03 14:47:24] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 70, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 48, in _create_instance
    cls.validate_dependencies(params.composite_name)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 453, in validate_dependencies
    raise ValueError(f"Missing package: {package}")
ValueError: Missing package: vagrant
vagrant@ubuntu-jammy:~/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size small --composite-name linux-centos-7-amd64
[2024-05-03 14:51:49] [DEBUG] SPNEGO._GSS: Python gssapi not available, cannot use any GSSAPIProxy protocols: No module named 'gssapi'
[2024-05-03 14:51:49] [DEBUG] SPNEGO._GSS: Python gssapi IOV extension not available: No module named 'gssapi'
[2024-05-03 14:51:49] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 70, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 48, in _create_instance
    cls.validate_dependencies(params.composite_name)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 451, in validate_dependencies
    raise ValueError(f"Missing package: {package}")
ValueError: Missing package: virtualbox
sudo ./aws/install^C
vagrant@ubuntu-jammy:~/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size small --composite-name macos-sonoma-14.0-arm64
[2024-05-03 15:15:46] [DEBUG] SPNEGO._GSS: Python gssapi not available, cannot use any GSSAPIProxy protocols: No module named 'gssapi'
[2024-05-03 15:15:46] [DEBUG] SPNEGO._GSS: Python gssapi IOV extension not available: No module named 'gssapi'
[2024-05-03 15:15:46] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 70, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 48, in _create_instance
    cls.validate_dependencies(params.composite_name)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 448, in validate_dependencies
    raise ValueError(f"Missing package: {package}")
ValueError: Missing package: sshpass
vagrant@ubuntu-jammy:~/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size small --composite-name macos-sonoma-14.0-arm64
[2024-05-03 15:19:45] [DEBUG] SPNEGO._GSS: Python gssapi not available, cannot use any GSSAPIProxy protocols: No module named 'gssapi'
[2024-05-03 15:19:45] [DEBUG] SPNEGO._GSS: Python gssapi IOV extension not available: No module named 'gssapi'
[2024-05-03 15:19:45] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 70, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 48, in _create_instance
    cls.validate_dependencies(params.composite_name)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 450, in validate_dependencies
    raise ValueError(f"Missing package: {package}")
ValueError: Missing package: awscli

…g/5295-dtt1-allocator-module-the-allocation-fails-if-the-sshpass-utility-is-not-installed

deployability/modules/allocation/aws/provider.py

deployability/modules/allocation/vagrant/provider.py

c-bordon · 2024-05-03T18:28:50Z

New tests after PR review:

vagrant@ubuntu-jammy:~/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size small --composite-name linux-ubuntu-22.04-amd64
[2024-05-03 18:06:01] [DEBUG] SPNEGO._GSS: Python gssapi not available, cannot use any GSSAPIProxy protocols: No module named 'gssapi'
[2024-05-03 18:06:01] [DEBUG] SPNEGO._GSS: Python gssapi IOV extension not available: No module named 'gssapi'
[2024-05-03 18:06:01] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 70, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 48, in _create_instance
    cls.validate_dependencies(params.composite_name)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 445, in validate_dependencies
    raise ValueError(f"Missing package: {missing_dependecy}")
ValueError: Missing package: virtualbox
vagrant@ubuntu-jammy:~/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size small --composite-name linux-ubuntu-22.04-amd64
[2024-05-03 18:06:41] [DEBUG] SPNEGO._GSS: Python gssapi not available, cannot use any GSSAPIProxy protocols: No module named 'gssapi'
[2024-05-03 18:06:41] [DEBUG] SPNEGO._GSS: Python gssapi IOV extension not available: No module named 'gssapi'
[2024-05-03 18:06:41] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 70, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 48, in _create_instance
    cls.validate_dependencies(params.composite_name)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 445, in validate_dependencies
    raise ValueError(f"Missing package: {missing_dependecy}")
ValueError: Missing package: vagrant
vagrant@ubuntu-jammy:~/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size small --composite-name macos-sonoma-14.0-arm64
[2024-05-03 18:07:07] [DEBUG] SPNEGO._GSS: Python gssapi not available, cannot use any GSSAPIProxy protocols: No module named 'gssapi'
[2024-05-03 18:07:07] [DEBUG] SPNEGO._GSS: Python gssapi IOV extension not available: No module named 'gssapi'
[2024-05-03 18:07:07] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 70, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 48, in _create_instance
    cls.validate_dependencies(params.composite_name)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 445, in validate_dependencies
    raise ValueError(f"Missing package: {missing_dependecy}")
ValueError: Missing package: sshpass
vagrant@ubuntu-jammy:~/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size small --composite-name macos-sonoma-14.0-arm64
[2024-05-03 18:17:22] [DEBUG] SPNEGO._GSS: Python gssapi not available, cannot use any GSSAPIProxy protocols: No module named 'gssapi'
[2024-05-03 18:17:22] [DEBUG] SPNEGO._GSS: Python gssapi IOV extension not available: No module named 'gssapi'
[2024-05-03 18:17:22] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 70, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 48, in _create_instance
    cls.validate_dependencies(params.composite_name)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 445, in validate_dependencies
    raise ValueError(f"Missing package: {missing_dependecy}")
ValueError: Missing package: awscli

c-bordon · 2024-05-03T18:43:11Z

Improved missing dependencies message

vagrant@ubuntu-jammy:~/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size small --composite-name linux-ubuntu-22.04-amd64
[2024-05-03 18:41:19] [DEBUG] SPNEGO._GSS: Python gssapi not available, cannot use any GSSAPIProxy protocols: No module named 'gssapi'
[2024-05-03 18:41:19] [DEBUG] SPNEGO._GSS: Python gssapi IOV extension not available: No module named 'gssapi'
[2024-05-03 18:41:19] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 70, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 48, in _create_instance
    cls.validate_dependencies(params.composite_name)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 449, in validate_dependencies
    raise ValueError(f"Missing dependencies: {missing_dependencies}")
ValueError: Missing dependencies: ['vagrant', 'virtualbox']
vagrant@ubuntu-jammy:~/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size small --composite-name macos-sonoma-14.0-arm64
[2024-05-03 18:41:37] [DEBUG] SPNEGO._GSS: Python gssapi not available, cannot use any GSSAPIProxy protocols: No module named 'gssapi'
[2024-05-03 18:41:37] [DEBUG] SPNEGO._GSS: Python gssapi IOV extension not available: No module named 'gssapi'
[2024-05-03 18:41:37] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 70, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 48, in _create_instance
    cls.validate_dependencies(params.composite_name)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 449, in validate_dependencies
    raise ValueError(f"Missing dependencies: {missing_dependencies}")
ValueError: Missing dependencies: ['sshpass', 'awscli']
vagrant@ubuntu-jammy:~/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size small --composite-name macos-sonoma-14.0-arm64
[2024-05-03 18:42:13] [DEBUG] SPNEGO._GSS: Python gssapi not available, cannot use any GSSAPIProxy protocols: No module named 'gssapi'
[2024-05-03 18:42:13] [DEBUG] SPNEGO._GSS: Python gssapi IOV extension not available: No module named 'gssapi'
[2024-05-03 18:42:13] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 70, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 48, in _create_instance
    cls.validate_dependencies(params.composite_name)
  File "/home/vagrant/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 447, in validate_dependencies
    raise ValueError(f"Missing dependency: {missing_dependencies[0]}")
ValueError: Missing dependency: awscli

c-bordon added 5 commits May 3, 2024 11:16

Added validation for Allocation module dependencies

2caeab6

Added error message for yum sistems

bdac6c1

Added validation for awscli

0efab9a

Merge branch '4495-dtt1-release' of github.com:wazuh/wazuh-qa into bu…

8c442c4

…g/5295-dtt1-allocator-module-the-allocation-fails-if-the-sshpass-utility-is-not-installed

Fix validation for aws cli

97c3ee9

c-bordon requested review from mhamra, rauldpm and teddytpc1 May 3, 2024 15:51

c-bordon self-assigned this May 3, 2024

c-bordon linked an issue May 3, 2024 that may be closed by this pull request

DTT1 - Allocator module. The allocation fails if the sshpass utility is not installed #5295

Closed

3 tasks

rauldpm requested changes May 3, 2024

View reviewed changes

c-bordon added 3 commits May 3, 2024 14:03

Removed YUM validation and added error message for not apt systems

72f6ed8

The validation of dependencies according to the deploy is improved

d435b4e

Fixed aws cli binary validation

5c3676d

c-bordon requested a review from rauldpm May 3, 2024 18:27

enhancement in the missing dependencies print

76edd59

teddytpc1 approved these changes May 3, 2024

View reviewed changes

Removed installed_dependencies list

e9c00de

rauldpm approved these changes May 3, 2024

View reviewed changes

teddytpc1 merged commit 75ca9ac into 4495-dtt1-release May 3, 2024

teddytpc1 deleted the bug/5295-dtt1-allocator-module-the-allocation-fails-if-the-sshpass-utility-is-not-installed branch May 3, 2024 19:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Validation of system dependencies is added to the Allocation module #5336

Validation of system dependencies is added to the Allocation module #5336

c-bordon commented May 3, 2024

c-bordon commented May 3, 2024

c-bordon commented May 3, 2024

Validation of system dependencies is added to the Allocation module #5336

Validation of system dependencies is added to the Allocation module #5336

Conversation

c-bordon commented May 3, 2024

Test Ubuntu 22.04

c-bordon commented May 3, 2024

New tests after PR review:

c-bordon commented May 3, 2024

Improved missing dependencies message