Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cryptography python module not found when using community.crypto #762

Open
kwikmr2 opened this issue May 29, 2024 · 11 comments
Open

cryptography python module not found when using community.crypto #762

kwikmr2 opened this issue May 29, 2024 · 11 comments
Labels
question Further information is requested

Comments

@kwikmr2
Copy link

kwikmr2 commented May 29, 2024

SUMMARY

Built AWX EE from https://github.com/Frewx/awx-ee-builder.git.
Defined "quay.io/ansible/awx-ee:latest" in execution-environment.yml for base_image.
Included "cryptography" in requirements.txt.
Included "community.general" (latest) and "community.crypto" (latest) in requirements.yml.

When executing a simple task via AWX, the following error occurs:
"Cannot detect any of the required Python libraries cryptography (>= 1.6)"

ISSUE TYPE
  • Bug Report
COMPONENT NAME

community.crypto.x509_certificate_info

ANSIBLE VERSION
ansible [core 2.15.12]
  config file = None
  configured module search path = [‘/runner/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’]
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.4
  libyaml = True
COLLECTION VERSION
# /usr/local/lib/python3.9/site-packages/ansible_collections
Collection       Version
---------------- -------
community.crypto 2.16.1
# /usr/share/ansible/collections/ansible_collections
Collection       Version
---------------- -------
community.crypto 2.20.0
CONFIGURATION
CONFIG_FILE() = None
OS / ENVIRONMENT

quay.io/ansible/awx-ee:latest based on CentOS Stream release 9

STEPS TO REPRODUCE

Deploy EE based on the settings posted above. Setup EE in AWX (24.3.1) and execute simple playbook against target host.

---
  - name: Get information on generated certificate
    community.crypto.x509_certificate_info:
      path: /data/path/test-ca.crt
    register: result

  - name: Dump certificate information
    ansible.builtin.debug:
      var: result
EXPECTED RESULTS

That the information about a certificate would return

ACTUAL RESULTS
TASK [Get information on generated certificate] ********************************
task path: /runner/project/tasks/ssl-main.yml:2
<192.168.57.104> ESTABLISH SSH CONNECTION FOR USER: ansible_svc
<192.168.57.104> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="ansible_svc"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/30094d2be6"' 192.168.57.104 '/bin/sh -c '"'"'echo ~ansible_svc && sleep 0'"'"''
<192.168.57.104> (0, b'/home/ansible_svc\\n', b"Warning: Permanently added '192.168.57.104' (ECDSA) to the list of known hosts.\\r\\n<redacted>.\\n")
<192.168.57.104> ESTABLISH SSH CONNECTION FOR USER: ansible_svc
<192.168.57.104> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="ansible_svc"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/30094d2be6"' 192.168.57.104 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/ansible_svc/.ansible/tmp `"&& mkdir "` echo /home/ansible_svc/.ansible/tmp/ansible-tmp-1717007346.3548055-27-128022922135843 `" && echo ansible-tmp-1717007346.3548055-27-128022922135843="` echo /home/ansible_svc/.ansible/tmp/ansible-tmp-1717007346.3548055-27-128022922135843 `" ) && sleep 0'"'"''
<192.168.57.104> (0, b'ansible-tmp-1717007346.3548055-27-128022922135843=/home/ansible_svc/.ansible/tmp/ansible-tmp-1717007346.3548055-27-128022922135843\\n', b'')
<labymrepo01> Attempting python interpreter discovery
<192.168.57.104> ESTABLISH SSH CONNECTION FOR USER: ansible_svc
<192.168.57.104> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="ansible_svc"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/30094d2be6"' 192.168.57.104 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'python3.12'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.11'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.10'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.9'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.8'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<192.168.57.104> (0, b'PLATFORM\\nLinux\\nFOUND\\n/usr/libexec/platform-python\\n/usr/bin/python2.7\\n/usr/bin/python\\n/usr/bin/python\\nENDFOUND\\n', b'')
<192.168.57.104> ESTABLISH SSH CONNECTION FOR USER: ansible_svc
<192.168.57.104> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="ansible_svc"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/30094d2be6"' 192.168.57.104 '/bin/sh -c '"'"'/usr/libexec/platform-python && sleep 0'"'"''
<192.168.57.104> (0, b'{"osrelease_content": "NAME=\\\\"CentOS Linux\\\\"\\\\nVERSION=\\\\"7 (Core)\\\\"\\\\nID=\\\\"centos\\\\"\\\\nID_LIKE=\\\\"rhel fedora\\\\"\\\\nVERSION_ID=\\\\"7\\\\"\\\\nPRETTY_NAME=\\\\"CentOS Linux 7 (Core)\\\\"\\\\nANSI_COLOR=\\\\"0;31\\\\"\\\\nCPE_NAME=\\\\"cpe:/o:centos:centos:7\\\\"\\\\nHOME_URL=\\\\"https://www.centos.org/\\\\"\\\\nBUG_REPORT_URL=\\\\"https://bugs.centos.org/\\\\"\\\\n\\\\nCENTOS_MANTISBT_PROJECT=\\\\"CentOS-7\\\\"\\\\nCENTOS_MANTISBT_PROJECT_VERSION=\\\\"7\\\\"\\\\nREDHAT_SUPPORT_PRODUCT=\\\\"centos\\\\"\\\\nREDHAT_SUPPORT_PRODUCT_VERSION=\\\\"7\\\\"\\\\n\\\\n", "platform_dist_result": ["centos", "7.9.2009", "Core"]}\\n', b'')
Using module file /usr/share/ansible/collections/ansible_collections/community/crypto/plugins/modules/x509_certificate_info.py
<192.168.57.104> PUT /runner/.ansible/tmp/ansible-local-22yu_u7i7f/tmp5su6kh3e TO /home/ansible_svc/.ansible/tmp/ansible-tmp-1717007346.3548055-27-128022922135843/AnsiballZ_x509_certificate_info.py
<192.168.57.104> SSH: EXEC sshpass -d12 sftp -o BatchMode=no -b - -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="ansible_svc"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/30094d2be6"' '[192.168.57.104]'
<192.168.57.104> (0, b'sftp> put /runner/.ansible/tmp/ansible-local-22yu_u7i7f/tmp5su6kh3e /home/ansible_svc/.ansible/tmp/ansible-tmp-1717007346.3548055-27-128022922135843/AnsiballZ_x509_certificate_info.py\\n', b'')
<192.168.57.104> ESTABLISH SSH CONNECTION FOR USER: ansible_svc
<192.168.57.104> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="ansible_svc"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/30094d2be6"' 192.168.57.104 '/bin/sh -c '"'"'chmod u+x /home/ansible_svc/.ansible/tmp/ansible-tmp-1717007346.3548055-27-128022922135843/ /home/ansible_svc/.ansible/tmp/ansible-tmp-1717007346.3548055-27-128022922135843/AnsiballZ_x509_certificate_info.py && sleep 0'"'"''
<192.168.57.104> (0, b'', b'')
<192.168.57.104> ESTABLISH SSH CONNECTION FOR USER: ansible_svc
<192.168.57.104> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="ansible_svc"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/30094d2be6"' -tt 192.168.57.104 '/bin/sh -c '"'"'sudo -H -S -p "[sudo via ansible, key=etjodrossjnhiiejjjipkloplcelzydr] password:" -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-etjodrossjnhiiejjjipkloplcelzydr ; /usr/bin/python /home/ansible_svc/.ansible/tmp/ansible-tmp-1717007346.3548055-27-128022922135843/AnsiballZ_x509_certificate_info.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<192.168.57.104> (1, b'\\r\\n{"msg": "Cannot detect any of the required Python libraries cryptography (>= 1.6)", "failed": true, "invocation": {"module_args": {"content": null, "select_crypto_backend": "auto", "name_encoding": "ignore", "valid_at": null, "path": "/data/path/test-ca.crt"}}}\\r\\n', b'Shared connection to 192.168.57.104 closed.\\r\\n')
<192.168.57.104> Failed to connect to the host via ssh: Shared connection to 192.168.57.104 closed.
<192.168.57.104> ESTABLISH SSH CONNECTION FOR USER: ansible_svc
<192.168.57.104> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="ansible_svc"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/30094d2be6"' 192.168.57.104 '/bin/sh -c '"'"'rm -f -r /home/ansible_svc/.ansible/tmp/ansible-tmp-1717007346.3548055-27-128022922135843/ > /dev/null 2>&1 && sleep 0'"'"''
<192.168.57.104> (0, b'', b'')
fatal: [labymrepo01]: FAILED! => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "invocation": {
        "module_args": {
            "content": null,
            "name_encoding": "ignore",
            "path": "/data/path/test-ca.crt",
            "select_crypto_backend": "auto",
            "valid_at": null
        }
    },
    "msg": "Cannot detect any of the required Python libraries cryptography (>= 1.6)"
}
@felixfontein
Copy link
Contributor

You need to install the cryptography dependency for the Python where the module is executed. If you execute the module on a target node, you have to make sure it's also installed there. Ansible, AWX, the collection, requirements.txt, ... won't do that automatically for you.

@kwikmr2
Copy link
Author

kwikmr2 commented May 29, 2024

I just noticed in the debug output that Python2.7 is being used:
"04> (0, b'PLATFORM\nLinux\nFOUND\n/usr/libexec/platform-python\n/usr/bin/python2.7\n/usr/bin/python\n/usr/bin/python\nENDFOUND\n', b'')"

How is this possible when the EE is CentOS Stream 9 with Python3.9 installed? The whole purpose of the EE is to execute within that environment.

@felixfontein
Copy link
Contributor

If you ask Ansible to run the task on a remote target, then it won't be run inside the EE, but on the remote target.

If you want to run the module in the EE, you have to use hosts: localhost or delegate_to: localhost.

@felixfontein felixfontein added the question Further information is requested label May 29, 2024
@kwikmr2
Copy link
Author

kwikmr2 commented May 29, 2024

I made the adjustments and now it fails because the path to the certificate is not on the localhost (the EE)...this is seems to be a paradox.

---
  - name: Get information on generated certificate
    community.crypto.x509_certificate_info:
      path: /data/path/test-ca.crt
    register: result
    delegate_to: localhost

Error:

The full traceback is:
  File "/tmp/ansible_community.crypto.x509_certificate_info_payload_lcdbcu61/ansible_community.crypto.x509_certificate_info_payload.zip/ansible_collections/community/crypto/plugins/modules/x509_certificate_info.py", line 444, in main
fatal: [labymrepo01 -> localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "content": null,
            "name_encoding": "ignore",
            "path": "/data/path/test-ca.crt",
            "select_crypto_backend": "auto",
            "valid_at": null
        }
    },
    "msg": "Error while reading certificate file from disk: [Errno 2] No such file or directory: '/data/path/test-ca.crt'"
}

@felixfontein
Copy link
Contributor

Well, if you want to operate on files on the remote, you either first have to fetch them to the controller, or you have to run the module on the target - but then you have to make sure that the module's requirements are available.

@kwikmr2
Copy link
Author

kwikmr2 commented May 29, 2024

Okay, then assuming it is not feasible to install the cryptography python module on every single remote host in the inventory AND using delegate_to: localhost breaks since the certificate to be inspected is not on the EE...that would make this in a way only usable with CLI Ansible and not AWX/Tower.

@felixfontein
Copy link
Contributor

I don't see what the difference between CLI Ansible and AWX/Tower is. In both cases, installing on the controller is easier than installing on all remotes, and you can determine yourself whether to run on the controller or on the targets.

@mstyne
Copy link

mstyne commented Aug 13, 2024

I'm encountering an issue similar to this on Rocky Linux 9 (works fine on Ubuntu 22.04 and Debian 12) where cryptography is not discovered when Ansible is run in a Python venv.

ansible --version reports the correct Python interpreter path.

# ansible --version
ansible [core 2.15.12]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /root/.local/venv/lib64/python3.9/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /root/.local/venv/bin/ansible
  python version = 3.9.18 (main, Jul  3 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/root/.local/venv/bin/python3)
  jinja version = 3.1.4
  libyaml = True

Debug output from:

community.general.python_requirements_info:
  dependencies: cryptography
    "changed": false,
    "invocation": {
        "module_args": {
            "dependencies": [
                "cryptography"
            ]
        }
    },
    "mismatched": {},
    "not_found": [
        "cryptography"
    ],
    "python": "/usr/bin/python3",
    "python_system_path": [
        "/tmp/ansible_community.general.python_requirements_info_payload_wo1d5_v7/ansible_community.general.python_requirements_info_payload.zip",
        "/usr/lib64/python39.zip",
        "/usr/lib64/python3.9",
        "/usr/lib64/python3.9/lib-dynload",
        "/root/.local/lib/python3.9/site-packages",
        "/usr/lib64/python3.9/site-packages",
        "/usr/lib/python3.9/site-packages"
    ],
    "python_version": "3.9.18 (main, Jul  3 2024, 00:00:00) \n[GCC 11.4.1 20231218 (Red Hat 11.4.1-3)]",
    "python_version_info": {
        "major": 3,
        "micro": 18,
        "minor": 9,
        "releaselevel": "final",
        "serial": 0
    },
    "valid": {}
}

...indicates cryptography cannot be found, and the python interpreter uses the system python binary, and python_system_path shows the venv nowhere to be found. Yes, the venv has been activated prior to invoking Ansible.

(venv) # pip3 list | grep cryptography
cryptography        43.0.0

Overriding ansible_python_interpreter to use the Python binary from the venv resolves the issue, so it's possible this is an "Ansible on CentOS (and friends)" issue, rather than a community.crypto issue. I don't have a great solution for this, but I'm putting this out into the world in case someone else finds their way down this sad path.

@felixfontein
Copy link
Contributor

Overriding ansible_python_interpreter to use the Python binary from the venv resolves the issue, so it's possible this is an "Ansible on CentOS (and friends)" issue

Yes, that's exactly it. You need to point ansible_python_interpreter to the venv, otherwise Ansible won't use it to run modules when using connection: local.

@mstyne
Copy link

mstyne commented Aug 15, 2024

Overriding ansible_python_interpreter to use the Python binary from the venv resolves the issue, so it's possible this is an "Ansible on CentOS (and friends)" issue

Yes, that's exactly it. You need to point ansible_python_interpreter to the venv, otherwise Ansible won't use it to run modules when using connection: local.

I don't encounter this additional configuration requirement with Ubuntu or Debian, but I'll have to double check to verify. If that output is interesting, I can post it here.

@felixfontein
Copy link
Contributor

You probably already have cryptography installed on system level on Debian and Ubuntu, and thus don't notice that it does not use the cryptography you explicitly installed but the system one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants