[Bug] Ansible Collections not installing when ansible should install per pip

[avalor1]

Description

Ansible collections will not install when ansible is installed via pip

Reproduction steps

Configure DMC to install ansible via pip and also install ansible collections. After this build the container.

requirements:
  pip:
    packages:
      - ansible
    requirements:
      - https://raw.githubusercontent.com/ansible-collections/azure/v1.15.0/requirements-azure.txt

  ansible:
    collections:
      - azure.azcollection=1.15.0
      - community.crypto=2.11.0
      - t_systems_mms.acme=2.3.5
      - t_systems_mms.icinga_director=1.32.2

Current Behavior

Ansible collections are tried to be installed before ansible even though the collection part comes after the pip part.

#11 [ 8/12] RUN if [ -n "${REQUIREMENTS}" ]; then       IFS=';' &&       for REQUIREMENT in ${REQUIREMENTS[@]}; do IFS='' && declare -u REQ="${REQUIREMENT}" && REQUIREMENTS_REQ=$(eval "echo \$REQUIREMENTS_${REQ}") &&       IFS=';' &&       if [ "${REQ}" = "PIP" ]; then for REQUIREMENT_REQ in ${REQUIREMENTS_PIP_PACKAGES[@]}; do pip3 install --no-cache-dir "${REQUIREMENT_REQ//=/==}"; done &&       for REQUIREMENT_REQ in ${REQUIREMENTS_PIP_REQUIREMENTS[@]}; do pip3 install --no-cache-dir -r "${REQUIREMENT_REQ}"; done; fi &&       if [ "${REQ}" = "ANSIBLE" ]; then for REQUIREMENT_REQ in ${REQUIREMENTS_ANSIBLE_ROLES[@]}; do ansible-galaxy install "${REQUIREMENT_REQ//=/,}"; done &&       for REQUIREMENT_REQ in ${REQUIREMENTS_ANSIBLE_COLLECTIONS[@]}; do ansible-galaxy collection install "${REQUIREMENT_REQ//=/:}"; done&&       for REQUIREMENT_REQ in ${REQUIREMENTS_ANSIBLE_REQUIREMENTS[@]}; do ansible-galaxy install -r "${REQUIREMENT_REQ}"; done; fi; done     fi
#11 sha256:4578ff3140b72b84a7943f3c02b949c1493dd1b224b460c659c82ba1ec078985
#11 0.330 /bin/bash: line 1: ansible-galaxy: command not found
#11 0.331 /bin/bash: line 1: ansible-galaxy: command not found
#11 0.332 /bin/bash: line 1: ansible-galaxy: command not found
#11 0.333 /bin/bash: line 1: ansible-galaxy: command not found
#11 0.984 Collecting ansible
#11 1.054   Downloading ansible-7.3.0-py3-none-any.whl (43.1 MB)
#11 1.497      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.1/43.1 MB 106.1 MB/s eta 0:00:00
#11 1.794 Collecting ansible-core~=2.14.3
#11 1.804   Downloading ansible_core-2.14.3-py3-none-any.whl (2.2 MB)
#11 1.824      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 123.0 MB/s eta 0:00:00

Expected Behavior

It installs the ansible collections after installing ansible via pip.

Additional information

Wanted to use ansible from pip as it is more recent.
Versions which will get installed:
Pip:
ansible: 7.3.0
ansible_core: 2.14.3

Apt:
ansible_core: 2.10.7

[michaelamattes]

I couldn't reproduce the error. There may be a local caching problem with the build. Have you tried the build with the --no-cache option? Does the problem still persist?

[michaelamattes]

I couldn't reproduce the error. There may be a local caching problem with the build. Have you tried the build with the --no-cache option? Does the problem still persist?

@avalor1 Could you please give me an update on this ?

[avalor1]

I could not reproduce it either. Thank you for your time :)

[rndmh3ro]

This seems to happen at random. Here's an excerpt from a build:

#11 [ 8/12] RUN if [ -n "${REQUIREMENTS}" ]; then       IFS=';' &&       for REQUIREMENT in ${REQUIREMENTS[@]}; do IFS='' && declare -u REQ="${REQUIREMENT}" && REQUIREMENTS_REQ=$(eval "echo \$REQUIREMENTS_${REQ}") &&       IFS=';' &&       if [ "${REQ}" = "PIP" ]; then for REQUIREMENT_REQ in ${REQUIREMENTS_PIP_PACKAGES[@]}; do pip3 install --no-cache-dir "${REQUIREMENT_REQ//=/==}"; done &&       for REQUIREMENT_REQ in ${REQUIREMENTS_PIP_REQUIREMENTS[@]}; do pip3 install --no-cache-dir -r "${REQUIREMENT_REQ}"; done; fi &&       if [ "${REQ}" = "ANSIBLE" ]; then for REQUIREMENT_REQ in ${REQUIREMENTS_ANSIBLE_ROLES[@]}; do ansible-galaxy install "${REQUIREMENT_REQ//=/,}"; done &&       for REQUIREMENT_REQ in ${REQUIREMENTS_ANSIBLE_COLLECTIONS[@]}; do ansible-galaxy collection install "${REQUIREMENT_REQ//=/:}"; done&&       for REQUIREMENT_REQ in ${REQUIREMENTS_ANSIBLE_REQUIREMENTS[@]}; do ansible-galaxy install -r "${REQUIREMENT_REQ}"; done; fi; done     fi
#11 0.685 Starting galaxy collection install process
#11 0.685 Process install dependency map
#11 4.100 Starting collection install process
#11 4.100 Installing 'ansible.utils:2.10.3' to '/root/.ansible/collections/ansible_collections/ansible/utils'
#11 4.100 Downloading https://galaxy.ansible.com/download/ansible-utils-2.10.3.tar.gz to /root/.ansible/tmp/ansible-local-7zh7odfef/tmp8pc1w01g
#11 5.874 ansible.utils (2.10.3) was installed successfully
#11 6.318 Starting galaxy collection install process
#11 6.320 Process install dependency map
#11 8.274 Starting collection install process
#11 8.274 Installing 'azure.azcollection:1.15.0' to '/root/.ansible/collections/ansible_collections/azure/azcollection'
#11 8.274 Downloading https://galaxy.ansible.com/download/azure-azcollection-1.15.0.tar.gz to /root/.ansible/tmp/ansible-local-8mbx473dm/tmpij5bv0ag
#11 15.89 azure.azcollection (1.15.0) was installed successfully
#11 16.37 Starting galaxy collection install process
#11 16.37 Process install dependency map
#11 16.80 Starting collection install process
#11 16.80 Installing 'telekom_mms.icinga_director:1.33.2' to '/root/.ansible/collections/ansible_collections/telekom_mms/icinga_director'
#11 16.80 Downloading https://galaxy.ansible.com/download/telekom_mms-icinga_director-1.33.2.tar.gz to /root/.ansible/tmp/ansible-local-952p8s4e6/tmpvombzqe6
#11 21.44 telekom_mms.icinga_director (1.33.2) was installed successfully
#11 22.12 Collecting PyMySQL
#11 22.23   Downloading PyMySQL-1.1.0-py3-none-any.whl (44 kB)
#11 22.25      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.8/44.8 KB 2.2 MB/s eta 0:00:00
#11 22.32 Installing collected packages: PyMySQL
#11 22.36 Successfully installed PyMySQL-1.1.0
#11 22.36 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#11 23.10 Collecting ansible==8.1.0
#11 23.21   Downloading ansible-8.1.0-py3-none-any.whl (44.8 MB)
#11 24.04      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.8/44.8 MB 81.1 MB/s eta 0:00:00
#11 24.36 Collecting ansible-core~=2.15.1
#11 24.38   Downloading ansible_core-2.15.2-py3-none-any.whl (2.2 MB)
#11 24.40      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 151.5 MB/s eta 0:00:00
#11 24.43 Requirement already satisfied: packaging in /usr/lib/python3/dist-packages (from ansible-core~=2.15.1->ansible==8.1.0) (21.3)
#11 24.43 Requirement already satisfied: jinja2>=3.0.0 in /usr/lib/python3/dist-packages (from ansible-core~=2.15.1->ansible==8.1.0) (3.0.3)
#11 24.45 Collecting resolvelib<1.1.0,>=0.5.3
#11 24.47   Downloading resolvelib-1.0.1-py2.py3-none-any.whl (17 kB)
#11 24.47 Requirement already satisfied: cryptography in /usr/lib/python3/dist-packages (from ansible-core~=2.15.1->ansible==8.1.0) (3.4.8)
#11 24.47 Requirement already satisfied: PyYAML>=5.1 in /usr/lib/python3/dist-packages (from ansible-core~=2.15.1->ansible==8.1.0) (5.4.1)
#11 24.71 Installing collected packages: resolvelib, ansible-core, ansible
#11 46.97 Successfully installed ansible-8.1.0 ansible-core-2.15.2 resolvelib-1.0.1
#11 46.97 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#11 47.72 Collecting pyvmomi
#11 47.85   Downloading pyvmomi-8.0.1.0.2.tar.gz (909 kB)
#11 47.94      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 909.6/909.6 KB 10.2 MB/s eta 0:00:00
#11 48.06   Preparing metadata (setup.py): started
#11 48.25   Preparing metadata (setup.py): finished with status 'done'
#11 48.25 Requirement already satisfied: six>=1.7.3 in /usr/lib/python3/dist-packages (from pyvmomi) (1.16.0)
#11 48.26 Building wheels for collected packages: pyvmomi
#11 48.26   Building wheel for pyvmomi (setup.py): started
#11 48.59   Building wheel for pyvmomi (setup.py): finished with status 'done'
#11 48.60   Created wheel for pyvmomi: filename=pyvmomi-8.0.1.0.2-py2.py3-none-any.whl size=382066 sha256=43627fa3349ab44195a2a32d202f977b9e23dcc44634c4a0cc7ff2b93deb29cb
#11 48.60   Stored in directory: /tmp/pip-ephem-wheel-cache-tx0otz4s/wheels/cb/47/1c/98dc4e371551f03920ac6e9b9d9fcef3fb472079a62758c0ce
#11 48.60 Successfully built pyvmomi
#11 48.68 Installing collected packages: pyvmomi
#11 49.11 Successfully installed pyvmomi-8.0.1.0.2
#11 49.11 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
``

[rndmh3ro]

We found the problem and will look for a proper fix.

[rndmh3ro]

fixed by #100