Rds enhanced monitoring bug fix (#747)

Rds enhanced monitoring bug fix

SUMMARY
(a copy of #712 as I messed up my branch by accident)
This is a fix for an issue when an RDS instance already exists and you wish to enable enhanced monitoring, for the full details see the linked old reported issue:
ansible/ansible#51772
But in summary currently if you enable enhanced monitoring on an RDS instance that already exists where it isn't already enabled then the following is returned:
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'MonitoringRoleArn'
fatal: [localhost_eu-west-1-pdv-qa-1 -> 127.0.0.1]: FAILED! => changed=false 
  module_stderr: |-
    Traceback (most recent call last):
      File "master:/opt/mitogen/mitogen-0.2.9/ansible_mitogen/runner.py", line 975, in _run
        self._run_code(code, mod)
      File "master:/opt/mitogen/mitogen-0.2.9/ansible_mitogen/runner.py", line 939, in _run_code
        exec(code, vars(mod))
      File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 1245, in <module>
      File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 1210, in main
      File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 855, in get_parameters
      File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 885, in get_options_with_changing_values
      File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 983, in get_changing_options_with_consistent_keys
    KeyError: 'MonitoringRoleArn'
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error


Originally-Depends-On: mattclay/aws-terminator#164
Other changes
A load of issues have surfaced in the integration tests due to how slow RDS is to create / modify etc. I've condensed down the tests where possible reducing the number of inventory jobs to 6 and bumped serial to 6 so that hopefully all tests can run at once and finish within the 1 hr AWS session duration.
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
rds_instance

Reviewed-by: Mark Chappell <None>
Reviewed-by: Mark Woolley <[email protected]>
Reviewed-by: None <None>

changelogs/fragments/747-rds-enhanced-monitoring-bug-fix.yml

@@ -0,0 +1,2 @@
+bugfixes:
+- rds_instance - Fixed issue with enabling enhanced monitoring on a pre-existing RDS instance (https://github.com/ansible-collections/community.aws/pull/747).

plugins/modules/rds_instance.py

@@ -989,13 +989,12 @@ def get_changing_options_with_inconsistent_keys(modify_params, instance, purge_c
 
 
 def get_changing_options_with_consistent_keys(modify_params, instance):
-    inconsistent_parameters = list(modify_params.keys())
     changing_params = {}
 
     for param in modify_params:
-        current_option = instance.get('PendingModifiedValues', {}).get(param)
+        current_option = instance.get('PendingModifiedValues', {}).get(param, None)
         if current_option is None:
-            current_option = instance[param]
+            current_option = instance.get(param, None)
         if modify_params[param] != current_option:
             changing_params[param] = modify_params[param]
 

tests/integration/targets/rds_instance/aliases

@@ -1,5 +1,3 @@
-# reason: missing-policy
-# reason: slow
-unsupported
+slow
 
 cloud/aws

tests/integration/targets/rds_instance/inventory

@@ -1,15 +1,12 @@
 [tests]
 states
-tags
-modification
-bad_options
+modify
+modify_complex
 processor_features
-encryption
-final_snapshot
 read_replica
 vpc_security_groups
 restore_instance
-snapshot
+tagging
 
 # TODO: uncomment after adding rds_cluster module
 # aurora

tests/integration/targets/rds_instance/main.yml

@@ -6,6 +6,6 @@
 - hosts: all
   gather_facts: no
   strategy: free
-  serial: 5
+  serial: 8
   roles:
     - rds_instance

tests/integration/targets/rds_instance/roles/rds_instance/defaults/main.yml

@@ -8,6 +8,8 @@ storage_encrypted_db_instance_class: db.t3.small
 modified_db_instance_class: db.t3.medium
 allocated_storage: 20
 modified_allocated_storage: 30
+monitoring_interval: 60
+preferred_maintenance_window: "mon:06:20-mon:06:50"
 
 # For aurora tests
 cluster_id: "{{ resource_prefix }}-cluster"
@@ -16,13 +18,10 @@ aurora_db_instance_class: db.t3.medium
 # For oracle tests
 # Smallest instance that permits modification of the coreCount
 oracle_ee_db_instance_class: db.r5.2xlarge
-processor_features:
-  coreCount: 2
-  threadsPerCore: 1
 modified_processor_features:
   coreCount: 4
   threadsPerCore: 2
 
 # For mariadb tests
-mariadb_engine_version: 10.3.20
-mariadb_engine_version_2: 10.4.8
+mariadb_engine_version: 10.3.31
+mariadb_engine_version_2: 10.4.21

tests/integration/targets/rds_instance/roles/rds_instance/files/enhanced_monitoring_assume_policy.json

@@ -0,0 +1,13 @@
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "",
+            "Effect": "Allow",
+            "Principal": {
+                "Service": "monitoring.rds.amazonaws.com"
+            },
+            "Action": "sts:AssumeRole"
+        }
+    ]
+}

tests/integration/targets/rds_instance/roles/rds_instance/tasks/test_aurora.yml

@@ -113,6 +113,7 @@
           id: "{{ item }}"
           state: absent
           skip_final_snapshot: True
+          wait: false
         loop:
           - "{{ instance_id }}"
           - "{{ modified_instance_id }}"
@@ -123,4 +124,5 @@
           cluster_id: "{{ cluster_id }}"
           state: absent
           skip_final_snapshot: True
+          wait: false
         ignore_errors: yes