Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

community.aws.rds_cluster module fails to manage cluster after creating cluster #1409

Closed
1 task done
imesias opened this issue Aug 12, 2022 · 5 comments · Fixed by #1417
Closed
1 task done

community.aws.rds_cluster module fails to manage cluster after creating cluster #1409

imesias opened this issue Aug 12, 2022 · 5 comments · Fixed by #1417
Labels
bug This issue/PR relates to a bug has_pr module module plugins plugin (any type) python3 traceback

Comments

@imesias
Copy link
Contributor

imesias commented Aug 12, 2022

Summary

When I try and create a cluster using community.aws.rds_cluster to create an rds cluster. Upon first execution of the playbook the cluster is created. When run again expecting to see 'ok' instead of changed the module fails with an error:

Traceback (most recent call last):
  File "/Users/USERNAME/.ansible/tmp/ansible-tmp-1660299591.748953-13510-114592878762279/AnsiballZ_rds_cluster.py", line 107, in <module>
    _ansiballz_main()
  File "/Users/USERNAME/.ansible/tmp/ansible-tmp-1660299591.748953-13510-114592878762279/AnsiballZ_rds_cluster.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/Users/USERNAME/.ansible/tmp/ansible-tmp-1660299591.748953-13510-114592878762279/AnsiballZ_rds_cluster.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.community.aws.plugins.modules.rds_cluster', init_globals=dict(_module_fqn='ansible_collections.community.aws.plugins.modules.rds_cluster', _modlib_path=modlib_path),
  File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 225, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 1026, in <module>
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 1010, in main
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 872, in ensure_present
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 846, in changing_cluster_options
KeyError: 'DBClusterParameterGroupName'

Issue Type

Bug Report

Component Name

rds_cluster

Ansible Version

$ ansible --version
ansible [core 2.13.2]
  config file = /Users/USERNAME/Development/proj/proj-iac-ansible-aws/ansible.cfg
  configured module search path = ['/Users/USERNAME/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/USERNAME/.local/share/virtualenvs/proj-iac-ansible-aws-m1mSr6qK/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/USERNAME/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/USERNAME/.local/share/virtualenvs/proj-iac-ansible-aws-m1mSr6qK/bin/ansible
  python version = 3.9.13 (main, May 24 2022, 21:28:44) [Clang 13.0.0 (clang-1300.0.29.30)]
  jinja version = 3.1.2
  libyaml = True

Collection Versions

$ ansible-galaxy collection list
Collection                    Version
----------------------------- -------
amazon.aws                    3.3.1  
ansible.netcommon             3.0.1  
ansible.posix                 1.4.0  
ansible.utils                 2.6.1  
ansible.windows               1.10.0 
arista.eos                    5.0.1  
awx.awx                       21.4.0 
azure.azcollection            1.13.0 
check_point.mgmt              2.3.0  
chocolatey.chocolatey         1.3.0  
cisco.aci                     2.2.0  
cisco.asa                     3.1.0  
cisco.dnac                    6.5.2  
cisco.intersight              1.0.19 
cisco.ios                     3.2.0  
cisco.iosxr                   3.2.0  
cisco.ise                     2.5.0  
cisco.meraki                  2.10.1 
cisco.mso                     2.0.0  
cisco.nso                     1.0.3  
cisco.nxos                    3.1.0  
cisco.ucs                     1.8.0  
cloud.common                  2.1.2  
cloudscale_ch.cloud           2.2.2  
community.aws                 3.4.0  
community.azure               1.1.0  
community.ciscosmb            1.0.5  
community.crypto              2.4.0  
community.digitalocean        1.21.0 
community.dns                 2.3.0  
community.docker              2.7.0  
community.fortios             1.0.0  
community.general             5.4.0  
community.google              1.0.0  
community.grafana             1.5.1  
community.hashi_vault         3.1.0  
community.hrobot              1.5.0  
community.libvirt             1.1.0  
community.mongodb             1.4.2  
community.mysql               3.3.0  
community.network             4.0.1  
community.okd                 2.2.0  
community.postgresql          2.2.0  
community.proxysql            1.4.0  
community.rabbitmq            1.2.2  
community.routeros            2.2.0  
community.sap                 1.0.0  
community.sap_libs            1.2.0  
community.skydive             1.0.0  
community.sops                1.2.3  
community.vmware              2.7.0  
community.windows             1.10.0 
community.zabbix              1.7.0  
containers.podman             1.9.4  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.14 
dellemc.enterprise_sonic      1.1.1  
dellemc.openmanage            5.5.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.18.0 
fortinet.fortimanager         2.1.5  
fortinet.fortios              2.1.6  
frr.frr                       2.0.0  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.8.1  
hpe.nimble                    1.1.4  
ibm.qradar                    2.0.0  
ibm.spectrum_virtualize       1.9.0  
infinidat.infinibox           1.3.3  
infoblox.nios_modules         1.3.0  
inspur.sm                     2.0.0  
junipernetworks.junos         3.1.0  
kubernetes.core               2.3.2  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.18.0
netapp.elementsw              21.7.0 
netapp.ontap                  21.21.0
netapp.storagegrid            21.10.0
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.3.0  
netbox.netbox                 3.7.1  
ngine_io.cloudstack           2.2.4  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.2  
openstack.cloud               1.8.0  
openvswitch.openvswitch       2.1.0  
ovirt.ovirt                   2.2.0  
purestorage.flasharray        1.13.0 
purestorage.flashblade        1.9.0  
purestorage.fusion            1.0.2  
sensu.sensu_go                1.13.1 
servicenow.servicenow         1.0.6  
splunk.es                     2.0.0  
t_systems_mms.icinga_director 1.30.1 
theforeman.foreman            3.4.0  
vmware.vmware_rest            2.2.0  
vyos.vyos                     3.0.1  
wti.remote                    1.0.4  

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.24.50
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /Users/USERNAME/.local/share/virtualenvs/sbsa-iac-ansible-aws-m1mSr6qK/lib/python3.9/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.27.50
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /Users/USERNAME/.local/share/virtualenvs/sbsa-iac-ansible-aws-m1mSr6qK/lib/python3.9/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed
HOST_KEY_CHECKING(/Users/USERNAME/Development/proj/proj-iac-ansible-aws/ansible.cfg) = False
INTERPRETER_PYTHON(/Users/USERNAME/Development/proj/proj-iac-ansible-aws/ansible.cfg) = /usr/bin/python3.9

OS / Environment

control node: macOS

Steps to Reproduce

---
- name: Provision RDS Cluster and Instances
  hosts: localhost
  connection: local
  gather_facts: False
  vars:
    # SUPPLY VPC INFO HERE
    aws_profile: "{{ lookup('env', 'AWS_PROFILE') | default('default', true) }}"
    db_username: 'my_cluster_username'
    db_password: 'my_cluster_password'
    subnet_group: 'my_cluster_subnet_group'
    parameter_group: 'my_cluster_parameter_group' 
    security_group: 'my_cluster_security_group'
    kms_key_arn: 'my_cluster_kms_key_arn'
  

  tasks:
    - name: Create Database Cluster
      local_action:
        module: community.aws.rds_cluster
        profile: "{{ aws_profile }}"
        region: "eu-west-1"
        db_cluster_identifier: empty-database-cluster
        engine: "aurora-postgresql"
        engine_version: "11.9"
        username: "{{ db_username }}"
        password: "{{ db_password }}"
        db_subnet_group_name: "{{ subnet_group }}"
        db_cluster_parameter_group_name: "{{ parameter_group }}"
        vpc_security_group_ids: "{{ security_group }}"
        storage_encrypted: yes
        availability_zones:
          - "eu-west-1a"
          - "eu-west-1b"
        kms_key_id: "{{ kms_key_arn }}"
        tags:
          Name: empty-database-cluster
        wait: yes

Expected Results

When you execute the play the first time the cluster is created, when you execute it a second time the playbook fails with an error. I expected it to say 'ok' as in the resource exists and is already created. I

Actual Results

Traceback (most recent call last):
  File "/Users/USERNAME/.ansible/tmp/ansible-tmp-1660299591.748953-13510-114592878762279/AnsiballZ_rds_cluster.py", line 107, in <module>
    _ansiballz_main()
  File "/Users/USERNAME/.ansible/tmp/ansible-tmp-1660299591.748953-13510-114592878762279/AnsiballZ_rds_cluster.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/Users/USERNAME/.ansible/tmp/ansible-tmp-1660299591.748953-13510-114592878762279/AnsiballZ_rds_cluster.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.community.aws.plugins.modules.rds_cluster', init_globals=dict(_module_fqn='ansible_collections.community.aws.plugins.modules.rds_cluster', _modlib_path=modlib_path),
  File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 225, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 1026, in <module>
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 1010, in main
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 872, in ensure_present
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 846, in changing_cluster_options
KeyError: 'DBClusterParameterGroupName'

I suspect that there is a difference being detected between what is defined and what has just been created in the environment so the code attempts a method with the wrong attributes.

Code of Conduct

  • I agree to follow the Ansible Code of Conduct
@ansibullbot
Copy link

Files identified in the description:

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

@ansibullbot
Copy link

@ansibullbot ansibullbot added bug This issue/PR relates to a bug module module needs_triage plugins plugin (any type) python3 traceback labels Aug 12, 2022
@alinabuzachis
Copy link
Contributor

@imesias Thank you for raising this issue. Will have a look and let you know.

@imesias
Copy link
Contributor Author

imesias commented Aug 18, 2022

@alinabuzachis @jillr @markuman @s-hertel @tremble I had a minute to have a quick look at this today. It appears the 'KeyError' is valid. get_cluster returns the cluster information returned by the describe_db_clusters boto call, if you have a look at the payload. You'll notice there is no DBClusterParameterGroupName there is however a DBClusterParameterGroup.

I noticed the way you tackled this previously was to pop the values off and compare before it reaches https://github.com/ansible-collections/community.aws/blob/main/plugins/modules/rds_cluster.py#L836 which is where the error occurs because the key DBClusterParameterGroupName doesn't exist in the current_cluster.

I suspect I could add the following to resolve?

desired_db_cluster_parameter_group = modify_params.pop("DBClusterParameterGroupName", None)
if desired_db_cluster_parameter_group:
    if desired_db_cluster_parameter_group != current_cluster["DBClusterParameterGroup"]:
        changing_params["DBClusterParameterGroupName"] = desired_db_cluster_parameter_group

If someone could guide me along the contribution process I'd be happy to submit a PR.

@imesias
Copy link
Contributor Author

imesias commented Aug 19, 2022

@alinabuzachis @jillr @markuman @s-hertel @tremble PR submitted #1417

softwarefactory-project-zuul bot pushed a commit that referenced this issue Sep 5, 2022
….py (#1417)

Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py

SUMMARY
Fix KeyError when comparing state.
Fixes: #1409 
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
rds_cluster.py

Reviewed-by: Alina Buzachis <None>
Reviewed-by: Markus Bergholz <[email protected]>
patchback bot pushed a commit that referenced this issue Sep 5, 2022
….py (#1417)

Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py

SUMMARY
Fix KeyError when comparing state.
Fixes: #1409
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
rds_cluster.py

Reviewed-by: Alina Buzachis <None>
Reviewed-by: Markus Bergholz <[email protected]>
(cherry picked from commit b3bc689)
patchback bot pushed a commit that referenced this issue Sep 5, 2022
….py (#1417)

Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py

SUMMARY
Fix KeyError when comparing state.
Fixes: #1409
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
rds_cluster.py

Reviewed-by: Alina Buzachis <None>
Reviewed-by: Markus Bergholz <[email protected]>
(cherry picked from commit b3bc689)
softwarefactory-project-zuul bot pushed a commit that referenced this issue Sep 28, 2022
….py (#1417) (#1433)

[PR #1417/b3bc6893 backport][stable-4] Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py

This is a backport of PR #1417 as merged into main (b3bc689).
SUMMARY
Fix KeyError when comparing state.
Fixes: #1409 
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
rds_cluster.py

Reviewed-by: Mark Chappell <None>
abikouo pushed a commit to abikouo/community.aws that referenced this issue Oct 24, 2023
….py (ansible-collections#1417)

Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py

SUMMARY
Fix KeyError when comparing state.
Fixes: ansible-collections#1409
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
rds_cluster.py

Reviewed-by: Alina Buzachis <None>
Reviewed-by: Markus Bergholz <[email protected]>

This commit was initially merged in https://github.com/ansible-collections/community.aws
See: ansible-collections@b3bc689
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue/PR relates to a bug has_pr module module plugins plugin (any type) python3 traceback
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants