ansible_connection=podman and async poor performance

[lucab85]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Hello,

When running the following playbook against a node with ansible_connection=local or ansible_connection=ssh, the async works as expected and perform well.

When running against a container, using ansible_connection=podman or ansible_connection=docker it took more time.

The ansible Engineer Team excluded any incompatibility between async and podman connector plugin with pipeline enable on #77046

Steps to reproduce the issue:

1. ansible_connection: local takes 0m10.455s

2. ansible_connection: podman takes 0m30.483s

Describe the results you received:

• ansible_connection: local

$ time ansible-playbook test.yml 
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] ******************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Async sleeping for batched_items] *******************************************************************************************************************************************************************************************************************************
changed: [localhost] => (item=1)
changed: [localhost] => (item=2)
changed: [localhost] => (item=3)
changed: [localhost] => (item=4)
changed: [localhost] => (item=5)

TASK [Check sync status] **********************************************************************************************************************************************************************************************************************************************
changed: [localhost] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '796033445730.3505116', 'results_file': '/opt/awxrpm/.ansible_async/796033445730.3505116', 'changed': True, 'failed': False, 'async_item': 1, 'ansible_loop_var': 'async_item'})
FAILED - RETRYING: Check sync status (30 retries left).
changed: [localhost] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '99334922661.3505135', 'results_file': '/opt/awxrpm/.ansible_async/99334922661.3505135', 'changed': True, 'failed': False, 'async_item': 2, 'ansible_loop_var': 'async_item'})
changed: [localhost] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '498307812124.3505155', 'results_file': '/opt/awxrpm/.ansible_async/498307812124.3505155', 'changed': True, 'failed': False, 'async_item': 3, 'ansible_loop_var': 'async_item'})
changed: [localhost] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '899582700725.3505175', 'results_file': '/opt/awxrpm/.ansible_async/899582700725.3505175', 'changed': True, 'failed': False, 'async_item': 4, 'ansible_loop_var': 'async_item'})
changed: [localhost] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '810355979454.3505195', 'results_file': '/opt/awxrpm/.ansible_async/810355979454.3505195', 'changed': True, 'failed': False, 'async_item': 5, 'ansible_loop_var': 'async_item'})

PLAY RECAP ************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


real    0m10.455s
user    0m4.929s
sys     0m1.227s

• ansible_connection: podman

$ time ansible-playbook -i ../container-dynamic test.yml 

PLAY [example] *****************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************************************************************************************************************************************
[WARNING]: Platform linux on host example is using the discovered Python interpreter at /usr/bin/python3.6, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
ok: [example]

TASK [Async sleeping for batched_items] *******************************************************************************************************************************************************************************************************************************
changed: [example] => (item=1)
changed: [example] => (item=2)
changed: [example] => (item=3)
changed: [example] => (item=4)
changed: [example] => (item=5)

TASK [Check sync status] **********************************************************************************************************************************************************************************************************************************************
changed: [example] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '30977658294.7303', 'results_file': '/root/.ansible_async/30977658294.7303', 'changed': True, 'failed': False, 'async_item': 1, 'ansible_loop_var': 'async_item'})
changed: [example] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '9649964613.7342', 'results_file': '/root/.ansible_async/9649964613.7342', 'changed': True, 'failed': False, 'async_item': 2, 'ansible_loop_var': 'async_item'})
changed: [example] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '286584213193.7383', 'results_file': '/root/.ansible_async/286584213193.7383', 'changed': True, 'failed': False, 'async_item': 3, 'ansible_loop_var': 'async_item'})
changed: [example] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '952408450077.7421', 'results_file': '/root/.ansible_async/952408450077.7421', 'changed': True, 'failed': False, 'async_item': 4, 'ansible_loop_var': 'async_item'})
changed: [example] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '50375873698.7463', 'results_file': '/root/.ansible_async/50375873698.7463', 'changed': True, 'failed': False, 'async_item': 5, 'ansible_loop_var': 'async_item'})

PLAY RECAP ************************************************************************************************************************************************************************************************************************************************************
example     : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


real    0m30.483s
user    0m14.705s
sys     0m8.448s

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Version of the containers.podman collection:
Either git commit if installed from git: git show --summary
Or version from ansible-galaxy if installed from galaxy: ansible-galaxy collection list | grep containers.podman

Installing 'containers.podman:1.9.1' to '/root/.ansible/collections/ansible_collections/containers/podman'

Output of ansible --version:

ansible --version
ansible 2.9.27
  config file = /opt/awxrpm/sclbuilder-podman/ansible.cfg
  configured module search path = ['/opt/awxrpm/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.8 (default, Mar 18 2021, 08:58:41) [GCC 8.4.1 20200928 (Red Hat 8.4.1-1)]

Output of podman version:

podman version 3.4.2

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.32-1.module+el8.5.0+13852+150547f7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.32, commit: 4b12bce835c3f8acc006a43620dd955a6a73bae0'
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "8.4"
  eventLogger: file
  hostname: host1.example.com
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-305.el8.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1225474048
  memTotal: 3918229504
  ociRuntime:
    name: runc
    package: runc-1.0.3-1.module+el8.5.0+13556+7f055e70.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.3
      spec: 1.0.2-dev
      go: go1.16.7
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.5.0+12582+56d94c81.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 2210394112
  swapTotal: 2210394112
  uptime: 2h 17m 33.37s (Approximately 0.08 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 1642068949
  BuiltTime: Thu Jan 13 10:15:49 2022
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.4.2-9.module+el8.5.0+13852+150547f7.x86_64

Playbok you run with ansible (e.g. content of playbook.yaml):

---
- hosts: example
  vars:
    durations:
      - 1
      - 2
      - 3
      - 4
      - 5
  tasks:
  - name: Async sleeping for batched_items
    ansible.builtin.command: sleep {{ async_item }}
    async: 45
    poll: 0
    loop: "{{ durations }}"
    loop_control:
      loop_var: "async_item"
    register: async_results

  - name: Check sync status
    async_status:
      jid: "{{ async_result_item.ansible_job_id }}"
    loop: "{{ async_results.results }}"
    loop_control:
      loop_var: "async_result_item"
    register: async_poll_results
    until: async_poll_results.finished
    retries: 30

Command line and output of ansible run with high verbosity

Please NOTE: if you submit a bug about idempotency, run the playbook with --diff option, like:

ansible-playbook -i inventory --diff -vv playbook.yml

(paste your output here)

Additional environment details (AWS, VirtualBox, physical, etc.):

[sshnaidm]

I think performance of podman connection can be improved, now it contains many hacks for various podman and ansible issues. Probably removing some of them and focusing on version 3 only would help to optimize the performance. But I'm not sure it will help so much as it's in local connection. Docker performance is out of scope here, but I wonder if both of connections have something in common in this performance issue.

[lucab85]

@sshnaidm I appreciate your message. This bug report is for ansible_connection=podman, I hope it could be added in the roadmap