unexpected output from Python interpreter discovery with aws_ssm connection plugin #1756

nmoseman · 2023-03-24T17:50:12Z

Summary

I had good luck with the aws_ssm plugin until attempting to use it against the latest Amazon Linux AMI. Simple commands that work well with a CentOS 7 host fails when trying to run them against AMI. It appears to be something to do with interpreting shell output.

A 'raw' command like this works fine:

ansible -i inventory.aws_ec2.yaml -m 'raw' -a 'whoami' tag_role_FAKEMX

However when running the equivalent 'command' module it fails for Amazon Linux, but works on CentOS 7.

❯ ansible -i inventory.aws_ec2.yaml -m 'command' -a 'whoami' tag_role_FAKEMX -l ec2-13-58-203-89.us-east-2.compute.amazonaws.com
[WARNING]: Unhandled error in Python interpreter discovery for host ec2-13-58-203-89.us-east-2.compute.amazonaws.com: unexpected output from Python interpreter discovery
ec2-13-58-203-89.us-east-2.compute.amazonaws.com | FAILED | rc=-1 >>
failed to transfer file to /Users/username/.ansible/tmp/ansible-local-78807y4966nip/tmpqn_vbdbk /AnsiballZ_command.py:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to open the file 2004h2004l/AnsiballZ_command.py: No such file
Warning: or directory
  1  129k    1  1531    0     0  19698      0  0:00:06 --:--:--  0:00:06 19883
curl: (23) Failure writing output to destination

Versus CentOS 7:

❯ ansible -i inventory.aws_ec2.yaml -m 'command' -a 'whoami' tag_role_FAKEMX -l ip-10-240-172-59.us-east-2.compute.internal
[WARNING]: Platform linux on host ip-10-240-172-59.us-east-2.compute.internal is using the discovered Python interpreter at /usr/libexec/platform-python, but future installation of another Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-
core/2.14/reference_appendices/interpreter_discovery.html for more information.
ip-10-240-172-59.us-east-2.compute.internal | CHANGED | rc=0 >>
root

From '-vvvv' output I see things like this:

<i-0cc859c89f4aaf5f4> ssm_retry: (success) (0, '\x1b[?2004h\x1b[?2004l\r\r\r\nPLATFORM\r\r\nLinux\r\r\nFOUND\r\r\n/usr/bin/python3.9\r\r\n/usr/bin/python3\r\r\nENDFOUND\r\r\n\x1b[?2004h\x1b[?2004l\r\r\r', '')
[WARNING]: Unhandled error in Python interpreter discovery for host ec2-13-58-203-89.us-east-2.compute.amazonaws.com: unexpected output from Python interpreter discovery
Using module file /Users/username/.asdf/installs/python/3.11.2/lib/python3.11/site-packages/ansible/modules/command.py

That's a failure to find the python version. Versus:

<i-03c9cbe64572b3eb0> ssm_retry: (success) (0, 'PLATFORM\r\r\nLinux\r\r\nFOUND\r\r\n/usr/libexec/platform-python\r\r\n/usr/bin/python2.7\r\r\n/usr/bin/python\r\r\n/usr/bin/python\r\r\nENDFOUND\r\r', '')
<ip-10-240-172-59.us-east-2.compute.internal> Python interpreter discovery fallback (pipelining support required for extended interpreter discovery)`

Where it didn't complain and seems to be successful. Notice the additional "x1b[?2004h\x1b[?2004" strings in the output.

I see the similar string in other problems, like "Warning: Failed to open the file 2004h2004l/AnsiballZ_command.py"

Note that if I set the python interpreter it will remove the warning, but it will still error out on the "Failed to open the file 2004h2004l/AnsiballZ_command.py: No such file Warning: or directory"

Tried this with community.aws collection versions 4.5.0, 5.2.0, and 5.3.0 and the errors is the same every time. Also tried a few different amazon.aws collection versions and had the same error.

These examples are taken from a Mac running python 3.11.2 and ansible 2.14.2, but the same errors occured in a Linux-based Execution Environment for AWX running in EKS.

Issue Type

Bug Report

Component Name

aws_ssm connection

Ansible Version

ansible [core 2.14.2]
  config file = None
  configured module search path = ['/Users/username/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/username/.asdf/installs/python/3.11.2/lib/python3.11/site-packages/ansible
  ansible collection location = /Users/username/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/username/.asdf/installs/python/3.11.2/bin/ansible
  python version = 3.11.2 (main, Feb 21 2023, 11:07:56) [Clang 13.1.6 (clang-1316.0.21.2.5)] (/Users/username/.asdf/installs/python/3.11.2/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True

Collection Versions

Collection                    Version
----------------------------- -------
ansible.netcommon             4.1.0
ansible.posix                 1.5.1
ansible.utils                 2.9.0
ansible.windows               1.13.0
arista.eos                    6.0.0
awx.awx                       21.11.0
azure.azcollection            1.14.0
check_point.mgmt              4.0.0
chocolatey.chocolatey         1.4.0
cisco.aci                     2.3.0
cisco.asa                     4.0.0
cisco.dnac                    6.6.3
cisco.intersight              1.0.23
cisco.ios                     4.3.1
cisco.iosxr                   4.1.0
cisco.ise                     2.5.12
cisco.meraki                  2.15.0
cisco.mso                     2.2.1
cisco.nso                     1.0.3
cisco.nxos                    4.0.1
cisco.ucs                     1.8.0
cloud.common                  2.1.2
cloudscale_ch.cloud           2.2.4
community.azure               2.0.0
community.ciscosmb            1.0.5
community.crypto              2.10.0
community.digitalocean        1.23.0
community.dns                 2.5.0
community.docker              3.4.0
community.fortios             1.0.0
community.general             6.3.0
community.google              1.0.0
community.grafana             1.5.3
community.hashi_vault         4.1.0
community.hrobot              1.7.0
community.libvirt             1.2.0
community.mongodb             1.4.2
community.mysql               3.5.1
community.network             5.0.0
community.okd                 2.2.0
community.postgresql          2.3.2
community.proxysql            1.5.1
community.rabbitmq            1.2.3
community.routeros            2.7.0
community.sap                 1.0.0
community.sap_libs            1.4.0
community.skydive             1.0.0
community.sops                1.6.0
community.vmware              3.3.0
community.windows             1.12.0
community.zabbix              1.9.1
containers.podman             1.10.1
cyberark.conjur               1.2.0
cyberark.pas                  1.0.17
dellemc.enterprise_sonic      2.0.0
dellemc.openmanage            6.3.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
dellemc.powerflex             1.5.0
dellemc.unity                 1.5.0
f5networks.f5_modules         1.22.0
fortinet.fortimanager         2.1.7
fortinet.fortios              2.2.2
frr.frr                       2.0.0
gluster.gluster               1.0.2
google.cloud                  1.1.2
grafana.grafana               1.1.0
hetzner.hcloud                1.9.1
hpe.nimble                    1.1.4
ibm.qradar                    2.1.0
ibm.spectrum_virtualize       1.11.0
infinidat.infinibox           1.3.12
infoblox.nios_modules         1.4.1
inspur.ispim                  1.2.0
inspur.sm                     2.3.0
junipernetworks.junos         4.1.0
kubernetes.core               2.3.2
lowlydba.sqlserver            1.3.1
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.22.0
netapp.elementsw              21.7.0
netapp.ontap                  22.2.0
netapp.storagegrid            21.11.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.4.0
netbox.netbox                 3.10.0
ngine_io.cloudstack           2.3.0
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.3
openstack.cloud               1.10.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   2.4.1
purestorage.flasharray        1.16.2
purestorage.flashblade        1.10.0
purestorage.fusion            1.3.0
sensu.sensu_go                1.13.2
splunk.es                     2.1.0
t_systems_mms.icinga_director 1.32.0
theforeman.foreman            3.8.0
vmware.vmware_rest            2.2.0
vultr.cloud                   1.7.0
vyos.vyos                     4.0.0
wti.remote                    1.0.4

# /Users/username/.ansible/collections/ansible_collections
Collection    Version
------------- -------
amazon.aws    4.3.0
community.aws 5.2.0

AWS SDK versions

❯ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.26.76
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/username/.asdf/installs/python/3.11.2/lib/python3.11/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.29.76
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/username/.asdf/installs/python/3.11.2/lib/python3.11/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

❯ ansible-config dump --only-changed
CONFIG_FILE() = None

OS / Environment

Darwin ENG-NMOSEMAN-MB 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:42:11 PST 2023; root:xnu-8792.81.3~2/RELEASE_X86_64 x86_64 i386 Darwin

and

modified version of quay.io/ansible/awx-ee:latest to include community.aws, and ssm stuff.

Steps to Reproduce

❯ ansible -i inventory.aws_ec2.yaml -m 'command' -a 'whoami' tag_role_FAKEMX -l ec2-13-58-203-89.us-east-2.compute.amazonaws.com
[WARNING]: Unhandled error in Python interpreter discovery for host ec2-13-58-203-89.us-east-2.compute.amazonaws.com: unexpected output from Python interpreter discovery
ec2-13-58-203-89.us-east-2.compute.amazonaws.com | FAILED | rc=-1 >>
failed to transfer file to /Users/username/.ansible/tmp/ansible-local-78807y4966nip/tmpqn_vbdbk /AnsiballZ_command.py:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to open the file 2004h2004l/AnsiballZ_command.py: No such file
Warning: or directory
  1  129k    1  1531    0     0  19698      0  0:00:06 --:--:--  0:00:06 19883
curl: (23) Failure writing output to destination

Expected Results

❯ ansible -i inventory.aws_ec2.yaml -m 'command' -a 'whoami' tag_role_FAKEMX -l ip-10-240-172-59.us-east-2.compute.internal
ip-10-240-172-59.us-east-2.compute.internal | CHANGED | rc=0 >>
root

Actual Results


  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to open the file 2004h2004l/AnsiballZ_command.py: No such file
Warning: or directory
  1  129k    1  1531    0     0  19698      0  0:00:06 --:--:--  0:00:06 19883
curl: (23) Failure writing output to destination

Code of Conduct

I agree to follow the Ansible Code of Conduct

The text was updated successfully, but these errors were encountered:

bwells-scripps · 2023-03-29T15:46:18Z

We just ran into this issue here. It seems that Amazon Linux outputs colorized text when Ansible runs any remote shell commands which causes parsing of the result to fail. Our solution was to not use aws_ssm connection - instead we setup SSH to make connections through Session Manager: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html

dennisjlee · 2023-06-01T20:25:57Z

I was facing this same issue with hosts running both Ubuntu 22.04 and Amazon Linux 2023, and I was finally able to trace the extra output to a root cause. This is due to newer versions of Bash/readline turning on the option enable-bracketed-paste by default (more details here).

I have a patch that will disable this option (will send a PR later today), which allows ansible -m ping to work on several hosts I have access to, including Ubuntu 18.04, Amazon Linux 2023, and Ubuntu 22.04.

adpavlov · 2023-07-10T12:59:48Z

#1839

Confirmed it's working with Amazon Linux

nnsense · 2023-09-07T13:18:08Z

I've ended up having the same issue switching from amazon Linux 2 to amazon Linux 2023, my error was:

service-use1-bh | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to create temporary directory. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\", for more error information use -vvv. Failed command was: ( umask 77 && mkdir -p \"` echo \u001b[?2004h\u001b[?2004l/.ansible/tmp `\"&& mkdir \"` echo \u001b[?2004h\u001b[?2004l/.ansible/tmp/ansible-tmp-1694091988.5261781-5952-41494157774678 `\" && echo ansible-tmp-1694091988.5261781-5952-41494157774678=\"` echo \u001b[?2004h\u001b[?2004l/.ansible/tmp/ansible-tmp-1694091988.5261781-5952-41494157774678 `\" ), exited with result 1, stdout output: \u001b[?2004h\u001b[?2004l\r\r\r\nmkdir: cannot create directory ‘2004h2004l’: Permission denied\r\r\n\u001b[?2004h\u001b[?2004l\r\r\r",
    "unreachable": true
}

#1839 fixes it, but it's taking ages to be merged for unknown reasons. In the meantime I've set set enable-bracketed-paste off into /etc/inputrc which is, needless to say, not a fix at all since you need to configure all servers this way, which is exactly what ansible is meant to do. In my case was just one so for now it's sorted, thanks to @dennisjlee !

…1839) SUMMARY aws_ssm - prevent escape codes from interfering with output Fixes #1756 ISSUE TYPE Bugfix Pull Request COMPONENT NAME aws_ssm ADDITIONAL INFORMATION This disables the Readline feature enable-bracketed-paste which is enabled by default on Bash 5.1 and above. This was causing escape sequences like \x1b[?2004h\x1b[?2004l to get into the output from some operating systems (e.g. Amazon Linux).

…1839) SUMMARY aws_ssm - prevent escape codes from interfering with output Fixes #1756 ISSUE TYPE Bugfix Pull Request COMPONENT NAME aws_ssm ADDITIONAL INFORMATION This disables the Readline feature enable-bracketed-paste which is enabled by default on Bash 5.1 and above. This was causing escape sequences like \x1b[?2004h\x1b[?2004l to get into the output from some operating systems (e.g. Amazon Linux). (cherry picked from commit af18bc6)

…1839) (#2030) [PR #1839/af18bc61 backport][stable-7] Eliminate escape codes in aws_ssm output from newer versions of Bash This is a backport of PR #1839 as merged into main (af18bc6). SUMMARY aws_ssm - prevent escape codes from interfering with output Fixes #1756 ISSUE TYPE Bugfix Pull Request COMPONENT NAME aws_ssm ADDITIONAL INFORMATION This disables the Readline feature enable-bracketed-paste which is enabled by default on Bash 5.1 and above. This was causing escape sequences like \x1b[?2004h\x1b[?2004l to get into the output from some operating systems (e.g. Amazon Linux). Reviewed-by: Mark Chappell

Hokwang · 2024-01-11T05:42:12Z

@tremble Hi, you released v7.1.0 and its release note mentioned this issue fixed, but it is not.

tremble · 2024-01-11T08:01:15Z

@Hokwang #1839 now results in the integration tests passing, which they previously didn't. This includes tests running against the latest Amazon Linux AMIs as exposed by Amazon as the SSM Parameter /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2.

As such I can only respond with "Unable to Reproduce" at this time

It would be helpful if you could provide details of exactly what error you're seeing against which AMI. I would strongly recommend opening a new issue (comments on closed issues are very easy to overlook), with much more detail than "still doesn't work for me".

richardsonky mentioned this issue Apr 11, 2023

Remove ansi escape sequences #1768

Closed

dennisjlee mentioned this issue Jun 1, 2023

Eliminate escape codes in aws_ssm output from newer versions of Bash #1839

Merged

ulyr mentioned this issue Jul 6, 2023

ssm connection plugin fails at gathering facts #113

Closed

tremble closed this as completed in #1839 Jan 3, 2024

patchback bot mentioned this issue Jan 3, 2024

[PR #1839/af18bc61 backport][stable-6] Eliminate escape codes in aws_ssm output from newer versions of Bash #2029

Closed

patchback bot mentioned this issue Jan 3, 2024

[PR #1839/af18bc61 backport][stable-7] Eliminate escape codes in aws_ssm output from newer versions of Bash #2030

Merged

tremble mentioned this issue Jan 19, 2024

SSM Connection: Failed to create temporary directory #2047

Closed

1 task

tremble mentioned this issue Feb 14, 2024

aws_ssm remote directory / #2054

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

unexpected output from Python interpreter discovery with aws_ssm connection plugin #1756

unexpected output from Python interpreter discovery with aws_ssm connection plugin #1756

nmoseman commented Mar 24, 2023

bwells-scripps commented Mar 29, 2023

dennisjlee commented Jun 1, 2023

adpavlov commented Jul 10, 2023

nnsense commented Sep 7, 2023

Hokwang commented Jan 11, 2024

tremble commented Jan 11, 2024

unexpected output from Python interpreter discovery with aws_ssm connection plugin #1756

unexpected output from Python interpreter discovery with aws_ssm connection plugin #1756

Comments

nmoseman commented Mar 24, 2023

Summary

Issue Type

Component Name

Ansible Version

Collection Versions

AWS SDK versions

Configuration

OS / Environment

Steps to Reproduce

Expected Results

Actual Results

Code of Conduct

bwells-scripps commented Mar 29, 2023

dennisjlee commented Jun 1, 2023

adpavlov commented Jul 10, 2023

nnsense commented Sep 7, 2023

Hokwang commented Jan 11, 2024

tremble commented Jan 11, 2024