module external_usergroup is not idempotent

[Thulium-Drake]

SUMMARY

Running the external_usergroup module multiple times always shows a change

ISSUE TYPE

• Bug Report

ANSIBLE VERSION

ansible 2.10.0 (base)
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.7.3 (default, Jul 25 2020, 13:03:44) [GCC 8.3.0]

Ansible controller is Debian 10 (Buster)

KATELLO/FOREMAN VERSION

tfm-rubygem-katello-3.15.3.1-1.el7.noarch
foreman-2.0.2-1.el7.noarch

Foreman/Katello is installed on CentOS7

APYPIE VERSION

Metadata-Version: 1.1
Version: 0.2.2

STEPS TO REPRODUCE

- name: 'Configure Foreman IPA admin group'
  theforeman.foreman.external_usergroup:
    username: admin
    password: changeme
    server_url: https://foreman.example.com
    validate_certs: false
    name: ipa_users
    usergroup: foreman_admins
    auth_source_ldap: 'External' # see #955 for my PR to rename this one

EXPECTED RESULTS

No changes after the first run

ACTUAL RESULTS

None of the info below is privileged, this is in my lab environment.

TASK [foreman : Configure Foreman IPA admin group] ******************************************************************
task path: /root/roles/foreman/tasks/configure_ipa_authentication.yml:29
<foreman.infra.htm.lab> ESTABLISH SSH CONNECTION FOR USER: ansible
<foreman.infra.htm.lab> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/73f35d548f foreman.infra.htm.lab '/bin/sh -c '"'"'echo ~ansible && sleep 0'"'"''
<foreman.infra.htm.lab> (0, b'/home/ansible\n', b'OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: /root/.ssh/config line 1: Applying options for *\r\ndebug1: /root/.ssh/config line 29: Applying options for *.lab\r\ndebug1: /root/.ssh/config line 33: Applying options for *.htm.lab\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 16886\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<foreman.infra.htm.lab> ESTABLISH SSH CONNECTION FOR USER: ansible
<foreman.infra.htm.lab> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/73f35d548f foreman.infra.htm.lab '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/ansible/.ansible/tmp `"&& mkdir "` echo /home/ansible/.ansible/tmp/ansible-tmp-1600174655.1453257-17048-71816907349924 `" && echo ansible-tmp-1600174655.1453257-17048-71816907349924="` echo /home/ansible/.ansible/tmp/ansible-tmp-1600174655.1453257-17048-71816907349924 `" ) && sleep 0'"'"''
<foreman.infra.htm.lab> (0, b'ansible-tmp-1600174655.1453257-17048-71816907349924=/home/ansible/.ansible/tmp/ansible-tmp-1600174655.1453257-17048-71816907349924\n', b'OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: /root/.ssh/config line 1: Applying options for *\r\ndebug1: /root/.ssh/config line 29: Applying options for *.lab\r\ndebug1: /root/.ssh/config line 33: Applying options for *.htm.lab\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 16886\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /root/projects/htm/collections/ansible_collections/theforeman/foreman/plugins/modules/external_usergroup.py
<foreman.infra.htm.lab> PUT /root/.ansible/tmp/ansible-local-168766l5hyxy7/tmpgkbd8j9l TO /home/ansible/.ansible/tmp/ansible-tmp-1600174655.1453257-17048-71816907349924/AnsiballZ_external_usergroup.py
<foreman.infra.htm.lab> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/73f35d548f '[foreman.infra.htm.lab]'
<foreman.infra.htm.lab> (0, b'sftp> put /root/.ansible/tmp/ansible-local-168766l5hyxy7/tmpgkbd8j9l /home/ansible/.ansible/tmp/ansible-tmp-1600174655.1453257-17048-71816907349924/AnsiballZ_external_usergroup.py\n', b'OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: /root/.ssh/config line 1: Applying options for *\r\ndebug1: /root/.ssh/config line 29: Applying options for *.lab\r\ndebug1: /root/.ssh/config line 33: Applying options for *.htm.lab\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 16886\r\ndebug3: mux_client_request_session: session request sent\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 2\r\ndebug2: Server supports extension "[email protected]" revision 2\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug3: Sent message fd 3 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/ansible size 0\r\ndebug3: Looking up /root/.ansible/tmp/ansible-local-168766l5hyxy7/tmpgkbd8j9l\r\ndebug3: Sent message fd 3 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/ansible/.ansible/tmp/ansible-tmp-1600174655.1453257-17048-71816907349924/AnsiballZ_external_usergroup.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:8 O:131072 S:1350\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 32768 bytes at 98304\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 8 1350 bytes at 131072\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<foreman.infra.htm.lab> ESTABLISH SSH CONNECTION FOR USER: ansible
<foreman.infra.htm.lab> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/73f35d548f foreman.infra.htm.lab '/bin/sh -c '"'"'chmod u+x /home/ansible/.ansible/tmp/ansible-tmp-1600174655.1453257-17048-71816907349924/ /home/ansible/.ansible/tmp/ansible-tmp-1600174655.1453257-17048-71816907349924/AnsiballZ_external_usergroup.py && sleep 0'"'"''
<foreman.infra.htm.lab> (0, b'', b'OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: /root/.ssh/config line 1: Applying options for *\r\ndebug1: /root/.ssh/config line 29: Applying options for *.lab\r\ndebug1: /root/.ssh/config line 33: Applying options for *.htm.lab\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 16886\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<foreman.infra.htm.lab> ESTABLISH SSH CONNECTION FOR USER: ansible
<foreman.infra.htm.lab> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/73f35d548f -tt foreman.infra.htm.lab '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-vhxkbbxhdviyywufvngjyjrgbdpwvhqp ; /usr/bin/python /home/ansible/.ansible/tmp/ansible-tmp-1600174655.1453257-17048-71816907349924/AnsiballZ_external_usergroup.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<foreman.infra.htm.lab> (0, b'/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html\r\n  InsecureRequestWarning)\r\n/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html\r\n  InsecureRequestWarning)\r\n/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html\r\n  InsecureRequestWarning)\r\n/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html\r\n  InsecureRequestWarning)\r\n\r\n{"invocation": {"module_args": {"username": "admin", "name": "foreman_admins", "auth_source_ldap": "External", "server_url": "https://foreman.infra.htm.lab", "state": "present", "usergroup": "ipa_foreman_admins", "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "validate_certs": false}}, "diff": {"after": {"external_usergroups": [{"id": 3, "name": "foreman_admins"}]}, "before": {"external_usergroups": [{"id": 3, "name": "foreman_admins"}]}}, "changed": true, "entity": {"external_usergroups": [{"auth_source_external": {"type": "AuthSourceExternal", "id": 3, "name": "External"}, "id": 3, "name": "foreman_admins"}]}}\r\n', b'OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: /root/.ssh/config line 1: Applying options for *\r\ndebug1: /root/.ssh/config line 29: Applying options for *.lab\r\ndebug1: /root/.ssh/config line 33: Applying options for *.htm.lab\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 16886\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to foreman.infra.htm.lab closed.\r\n')
<foreman.infra.htm.lab> ESTABLISH SSH CONNECTION FOR USER: ansible
<foreman.infra.htm.lab> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/73f35d548f foreman.infra.htm.lab '/bin/sh -c '"'"'rm -f -r /home/ansible/.ansible/tmp/ansible-tmp-1600174655.1453257-17048-71816907349924/ > /dev/null 2>&1 && sleep 0'"'"''
<foreman.infra.htm.lab> (0, b'', b'OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: /root/.ssh/config line 1: Applying options for *\r\ndebug1: /root/.ssh/config line 29: Applying options for *.lab\r\ndebug1: /root/.ssh/config line 33: Applying options for *.htm.lab\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 16886\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
changed: [foreman.infra.htm.lab] => changed=true 
  diff:
    after:
      external_usergroups:
      - id: 3
        name: foreman_admins
    before:
      external_usergroups:
      - id: 3
        name: foreman_admins
  entity:
    external_usergroups:
    - auth_source_external:
        id: 3
        name: External
        type: AuthSourceExternal
      id: 3
      name: foreman_admins
  invocation:
    module_args:
      auth_source_ldap: External
      name: foreman_admins
      password: VALUE_SPECIFIED_IN_NO_LOG_PARAMETER
      server_url: https://foreman.infra.htm.lab
      state: present
      usergroup: ipa_foreman_admins
      username: admin
      validate_certs: false
META: ran handlers
META: ran handlers

[evgeni]

I think the problem is that you're pointing at an "EXTERNAL" group, but the module only matches "LDAP" ones.

Can you please try the following patch and set the auth_source_external param instead of _ldap?

diff --git plugins/modules/external_usergroup.py plugins/modules/external_usergroup.py
index b3e52ff6..4b68aca1 100644
--- plugins/modules/external_usergroup.py
+++ plugins/modules/external_usergroup.py
@@ -83,7 +83,9 @@ def main():
             name=dict(required=True),
             usergroup=dict(required=True),
             auth_source_ldap=dict(required=True, type='entity', flat_name='auth_source_id', resource_type='auth_sources'),
+            auth_source_external=dict(required=True, type='entity', flat_name='auth_source_id', resource_type='auth_sources'),
         ),
+        mutually_exclusive=[['auth_source_ldap','auth_source_external']],
     )
 
     params = {"usergroup_id": module.foreman_params.pop('usergroup')}

That's obviously not how it should be in the final version, but the quickest for testing right now.

[evgeni]

err, obviously the required=True needs to be False.

diff --git plugins/modules/external_usergroup.py plugins/modules/external_usergroup.py
index b3e52ff6..758e51ba 100644
--- plugins/modules/external_usergroup.py
+++ plugins/modules/external_usergroup.py
@@ -82,8 +82,11 @@ def main():
         foreman_spec=dict(
             name=dict(required=True),
             usergroup=dict(required=True),
-            auth_source_ldap=dict(required=True, type='entity', flat_name='auth_source_id', resource_type='auth_sources'),
+            auth_source_ldap=dict(required=False, type='entity', flat_name='auth_source_id', resource_type='auth_sources'),
+            auth_source_external=dict(required=False, type='entity', flat_name='auth_source_id', resource_type='auth_sources'),
         ),
+        mutually_exclusive=[['auth_source_ldap','auth_source_external']],
+        required_one_of=[['auth_source_ldap','auth_source_external']],
     )
 
     params = {"usergroup_id": module.foreman_params.pop('usergroup')}

[evgeni]

Let's see if #959 fixes this. @Thulium-Drake can you give that PR a try in your env?

[Thulium-Drake]

I will, I'll get back to you tonight!

[Thulium-Drake]

Will be fixed with #959 :-)