Merge branch 'main' into module_vpc_endpoint_service

.github/workflows/sanity.yml

@@ -60,5 +60,9 @@ jobs:
             {
               "ansible-version": "devel",
               "python-version": "3.8"
+            },
+            {
+              "ansible-version": "devel",
+              "python-version": "3.9"
             }
           ]

.github/workflows/units.yml

@@ -59,6 +59,10 @@ jobs:
             {
               "ansible-version": "devel",
               "python-version": "3.8"
+            },
+            {
+              "ansible-version": "devel",
+              "python-version": "3.9"
             }
           ]
       collection_pre_install: ''

CHANGELOG.rst

@@ -5,6 +5,51 @@ amazon.aws Release Notes
 .. contents:: Topics
 
 
+v6.1.0
+======
+
+Release Summary
+---------------
+
+This release brings some new features, several bugfixes, and deprecated features are also included.
+
+Minor Changes
+-------------
+
+- ec2_snapshot - Add support for modifying createVolumePermission (https://github.com/ansible-collections/amazon.aws/pull/1464).
+- ec2_snapshot_info - Add createVolumePermission to output result (https://github.com/ansible-collections/amazon.aws/pull/1464).
+
+Deprecated Features
+-------------------
+
+- s3_object - support for passing object keys with a leading ``/`` has been deprecated and will be removed in a release after 2025-12-01 (https://github.com/ansible-collections/amazon.aws/pull/1549).
+
+Bugfixes
+--------
+
+- autoscaling_group - fix ValidationError when describing an autoscaling group that has more than 20 target groups attached to it by breaking the request into chunks (https://github.com/ansible-collections/amazon.aws/pull/1593).
+- autoscaling_group_info - fix ValidationError when describing an autoscaling group that has more than 20 target groups attached to it by breaking the request into chunks (https://github.com/ansible-collections/amazon.aws/pull/1593).
+- ec2_instance - fix check_mode issue when adding network interfaces (https://github.com/ansible-collections/amazon.aws/issues/1403).
+- ec2_metadata_facts - Handle decompression when EC2 instance user-data is gzip compressed. The fetch_url method from ansible.module_utils.urls does not decompress the user-data unless the header explicitly contains ``Content-Encoding: gzip`` (https://github.com/ansible-collections/amazon.aws/pull/1575).
+- elb_application_lb - fix missing attributes on creation of ALB. The ``create_or_update_alb()`` was including ALB-specific attributes when updating an existing ALB but not when creating a new ALB (https://github.com/ansible-collections/amazon.aws/issues/1510).
+- module_utils.acm - fixes list_certificates returning only RSA_2048 certificates (https://github.com/ansible-collections/amazon.aws/issues/1567).
+- rds_instance - add support for CACertificateIdentifier to create/update rds instance (https://github.com/ansible-collections/amazon.aws/pull/1459)."
+
+v6.0.1
+======
+
+Release Summary
+---------------
+
+This is a patch release that includes some bug fixes for the aws_ec2 inventory plugin and the s3_bucket and s3_object modules.
+
+Bugfixes
+--------
+
+- aws_ec2 inventory plugin - fix ``NoRegionError`` when no regions are provided and region isn't specified (https://github.com/ansible-collections/amazon.aws/issues/1551).
+- s3_bucket - fixes issue when deleting a bucket with unversioned objects (https://github.com/ansible-collections/amazon.aws/issues/1533).
+- s3_object - fixes regression related to objects with a leading ``/`` (https://github.com/ansible-collections/amazon.aws/issues/1548).
+
 v6.0.0
 ======
 

changelogs/changelog.yaml

@@ -2114,3 +2114,63 @@ releases:
         name: aws_collection_constants
         namespace: null
     release_date: '2023-05-09'
+  6.0.1:
+    changes:
+      bugfixes:
+      - aws_ec2 inventory plugin - fix ``NoRegionError`` when no regions are provided
+        and region isn't specified (https://github.com/ansible-collections/amazon.aws/issues/1551).
+      - s3_bucket - fixes issue when deleting a bucket with unversioned objects (https://github.com/ansible-collections/amazon.aws/issues/1533).
+      - s3_object - fixes regression related to objects with a leading ``/`` (https://github.com/ansible-collections/amazon.aws/issues/1548).
+      release_summary: This is a patch release that includes some bug fixes for the
+        aws_ec2 inventory plugin and the s3_bucket and s3_object modules.
+    fragments:
+    - 1538-s3-null.yml
+    - 1548-s3_object-leading-slash.yml
+    - 1551-ec2_inventory-no-region.yml
+    - 1560-revert_1546.yml
+    - release_summary.yml
+    release_date: '2023-05-19'
+  6.1.0:
+    changes:
+      bugfixes:
+      - autoscaling_group - fix ValidationError when describing an autoscaling group
+        that has more than 20 target groups attached to it by breaking the request
+        into chunks (https://github.com/ansible-collections/amazon.aws/pull/1593).
+      - autoscaling_group_info - fix ValidationError when describing an autoscaling
+        group that has more than 20 target groups attached to it by breaking the request
+        into chunks (https://github.com/ansible-collections/amazon.aws/pull/1593).
+      - ec2_instance - fix check_mode issue when adding network interfaces (https://github.com/ansible-collections/amazon.aws/issues/1403).
+      - 'ec2_metadata_facts - Handle decompression when EC2 instance user-data is
+        gzip compressed. The fetch_url method from ansible.module_utils.urls does
+        not decompress the user-data unless the header explicitly contains ``Content-Encoding:
+        gzip`` (https://github.com/ansible-collections/amazon.aws/pull/1575).'
+      - elb_application_lb - fix missing attributes on creation of ALB. The ``create_or_update_alb()``
+        was including ALB-specific attributes when updating an existing ALB but not
+        when creating a new ALB (https://github.com/ansible-collections/amazon.aws/issues/1510).
+      - module_utils.acm - fixes list_certificates returning only RSA_2048 certificates
+        (https://github.com/ansible-collections/amazon.aws/issues/1567).
+      - rds_instance - add support for CACertificateIdentifier to create/update rds
+        instance (https://github.com/ansible-collections/amazon.aws/pull/1459)."
+      deprecated_features:
+      - s3_object - support for passing object keys with a leading ``/`` has been
+        deprecated and will be removed in a release after 2025-12-01 (https://github.com/ansible-collections/amazon.aws/pull/1549).
+      minor_changes:
+      - ec2_snapshot - Add support for modifying createVolumePermission (https://github.com/ansible-collections/amazon.aws/pull/1464).
+      - ec2_snapshot_info - Add createVolumePermission to output result (https://github.com/ansible-collections/amazon.aws/pull/1464).
+      release_summary: This release brings some new features, several bugfixes, and
+        deprecated features are also included.
+    fragments:
+    - 1459-rds_instance-add-support-for-ca_certificate_identifier-to-create-update-instance.yml
+    - 1464-ec2_snapshot-ec2_snapshot_info-support-modifying-create-volume-permissions.yml
+    - 1510-elb_application_lb-fix-alb-specific-attributes-not-added-on-create.yml
+    - 1548-s3_object-leading-slash-deprecate.yml
+    - 1567-list-certificate-all-key-types.yml
+    - 1578-rds-instance-docs.yml
+    - 1593-autoscaling_group_info-20-target-groups-per-call.yml
+    - 20230526-ec2_mertadata_facts-handle_compressed_user_data.yml
+    - 20230531-aws_ec2-use_ssm_inventory_documentation.yml
+    - ec2-inventory-hostnames-doc.yml
+    - ec2_instance-eni-attach-idempotency.yml
+    - release_summary.yml
+    - test-reqs.yml
+    release_date: '2023-06-07'

changelogs/fragments/1589-return_false_when_no_change..yml

@@ -0,0 +1,3 @@
+---
+bugfixes:
+- Fixes changed status to report False when no change has been made. The module had incorrectly always reported a change. (https://github.com/ansible-collections/amazon.aws/pull/1589)

changelogs/fragments/1604-c2_vpc_nat_gateway-fails-silently.yml

@@ -0,0 +1,3 @@
+---
+bugfixes:
+- ec2_vpc_nat_gateway - adding a boolean parameter called ``default_create`` to allow users to have the option to choose whether they want to display an error message or create a NAT gateway when an EIP address is not found. The module (ec2_vpc_nat_gateway) had incorrectly failed silently if EIP didn't exist (https://github.com/ansible-collections/amazon.aws/issues/1295).

changelogs/fragments/1615-no_formatted_with_extra_space.yml

@@ -0,0 +1,3 @@
+---
+bugfixes:
+- Fixes to the stack trace output, where it does not contain spaces between each character. The module had incorrectly always outputted extra spaces between each character. (https://github.com/ansible-collections/amazon.aws/pull/1615)

changelogs/fragments/1632-changes-to-no-allocate-eip-when-connectivity_type=private.yml

@@ -0,0 +1,2 @@
+bugfixes:
+- ec2_vpc_nat_gateway - fixes to nat gateway so that when the user creates a private NAT gateway, an Elastic IP address should not be allocated. The module had inncorrectly always allocate elastic IP address when creating private nat gateway (https://github.com/ansible-collections/amazon.aws/pull/1632).

changelogs/fragments/1633-backup-selection-conditions.yml

@@ -0,0 +1,5 @@
+minor_changes:
+- backup_selection - add validation and documentation for all conditions suboptions (https://github.com/ansible-collections/amazon.aws/pull/1633).
+
+bugfixes:
+- module_utils.backup - get_selection_details fix empty list returned when multiple backup selections exist (https://github.com/ansible-collections/amazon.aws/pull/1633).

changelogs/fragments/1843-iam_instance_profile.yml

@@ -0,0 +1,2 @@
+trivial:
+- new modules - iam_instance_profile, iam_instance_profile_info

changelogs/fragments/1846-arn-validation.yml

@@ -0,0 +1,5 @@
+minor_changes:
+- ec2_instance - refactored ARN validation handling (https://github.com/ansible-collections/amazon.aws/pull/1619).
+- iam_user - refactored ARN validation handling (https://github.com/ansible-collections/amazon.aws/pull/1619).
+- module_utils.arn - added ``validate_aws_arn`` function to handle common pattern matching for ARNs (https://github.com/ansible-collections/amazon.aws/pull/1619).
+- module_utils.arn - add ``resource_id`` and ``resource_type`` to ``parse_aws_arn`` return values (https://github.com/ansible-collections/amazon.aws/pull/1619).

changelogs/fragments/20230506-autoscaling_group-fix_sanity.yml

@@ -0,0 +1,2 @@
+trivial:
+  - "Fix sanity errors autoscaling_group and autoscaling_group_info."

changelogs/fragments/202306012-backup_plan-remove-none-from-nested-params.yml

@@ -0,0 +1,3 @@
+---
+bugfixes:
+  - backup_plan - Use existing `scrub_none_values` function from module_utils to remove None values from nested dicts in supplied params. Nested None values were being retained and causing an error when sent through to the boto3 client operation (https://github.com/ansible-collections/amazon.aws/pull/1611).

changelogs/fragments/20230612-backup_vault-fix-tag-update.yml

@@ -0,0 +1,3 @@
+---
+bugfixes:
+  - backup_vault - fix error when updating tags on a backup vault by using the correct boto3 client methods for tagging and untagging backup resources (https://github.com/ansible-collections/amazon.aws/pull/1610).

changelogs/fragments/20230612-s3_object-support-copy-objects-recursively.yml

@@ -0,0 +1,2 @@
+minor_changes:
+- s3_object - Allow recursive copy of objects in S3 bucket (https://github.com/ansible-collections/amazon.aws/issues/1379).

changelogs/fragments/20230627-ci-fixup.yml

@@ -0,0 +1,3 @@
+trivial:
+- CI fixup - ``include:`` should be ``include_tasks:``.
+- CI fixup - YAML ``Null`` no longer matches any explicit parameter type requirements.

changelogs/fragments/20230702-isort.yml

@@ -0,0 +1,2 @@
+trivial:
+- added isort configs to pyproject.toml

changelogs/fragments/ec2_instance_info-support-new-attribute.yml

@@ -0,0 +1,3 @@
+---
+minor_changes:
+  - ec2_instance_info - add new parameter `include_attributes` to describe instance attributes (https://github.com/ansible-collections/amazon.aws/pull/1577).

docs/docsite/rst/aws_ec2_guide.rst

@@ -484,7 +484,9 @@ Now the output of ``ansible-inventory -i demo.aws_ec2.yml --list``:
 ``use_ssm_inventory``
 ---------------------
 
-``use_ssm_inventory: True`` will include SSM inventory variables into hostvars for ssm-configured instances.
+``use_ssm_inventory: True`` enables fetching additional EC2 instance information from the AWS Systems Manager (SSM) inventory service into hostvars. By leveraging the SSM inventory data, the ``use_ssm_inventory`` option provides additional details and attributes about the EC2 instances in your inventory.
+These details can include operating system information, installed software, network configurations, and custom inventory attributes defined in SSM.
+
 
 ``cache``
 ---------

meta/runtime.yml

@@ -67,6 +67,8 @@ action_groups:
   - elb_application_lb_info
   - elb_classic_lb
   - execute_lambda
+  - iam_instance_profile
+  - iam_instance_profile_info
   - iam_policy
   - iam_policy_info
   - iam_user

plugins/inventory/aws_ec2.py

@@ -32,16 +32,17 @@
   hostnames:
     description:
       - A list in order of precedence for hostname variables.
+      - The elements of the list can be a dict with the keys mentioned below or a string.
+      - Can be one of the options specified in U(http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html#options).
+      - If value provided does not exist in the above options, it will be used as a literal string.
+      - To use tags as hostnames use the syntax tag:Name=Value to use the hostname Name_Value, or tag:Name to use the value of the Name tag.
     type: list
-    elements: dict
+    elements: raw
     default: []
     suboptions:
       name:
         description:
           - Name of the host.
-          - Can be one of the options specified in U(http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html#options).
-          - To use tags as hostnames use the syntax tag:Name=Value to use the hostname Name_Value, or tag:Name to use the value of the Name tag.
-          - If value provided does not exist in the above options, it will be used as a literal string.
         type: str
         required: True
       prefix:
@@ -137,7 +138,10 @@
     version_added: 3.1.0
   use_ssm_inventory:
     description:
-      - Add SSM inventory information into hostvars.
+      - Enables fetching additional EC2 instance information from the AWS Systems Manager (SSM) inventory service into hostvars.
+      - By leveraging the SSM inventory data, the I(use_ssm_inventory) option provides additional details and attributes
+        about the EC2 instances in your inventory. These details can include operating system information, installed software,
+        network configurations, and custom inventory attributes defined in SSM.
     type: bool
     default: False
     version_added: 6.0.0
@@ -773,6 +777,9 @@ def parse(self, inventory, loader, path, cache=True):
         use_contrib_script_compatible_ec2_tag_keys = self.get_option("use_contrib_script_compatible_ec2_tag_keys")
         use_ssm_inventory = self.get_option("use_ssm_inventory")
 
+        if not all(isinstance(element, (dict, str)) for element in hostnames):
+            self.fail_aws("Hostnames should be a list of dict and str.")
+
         if self.get_option("include_extra_api_calls"):
             self.display.deprecate(
                 "The include_extra_api_calls option has been deprecated and will be removed in release 6.0.0.",

plugins/module_utils/acm.py

@@ -63,7 +63,20 @@ def delete_certificate_with_backoff(self, arn):
     @AWSRetry.jittered_backoff(delay=5, catch_extra_error_codes=["RequestInProgressException"])
     def list_certificates_with_backoff(self, statuses=None):
         paginator = self.client.get_paginator("list_certificates")
-        kwargs = dict()
+        # `list_certificates` requires explicit key type filter, or it returns only RSA_2048 certificates
+        kwargs = {
+            "Includes": {
+                "keyTypes": [
+                    "RSA_1024",
+                    "RSA_2048",
+                    "RSA_3072",
+                    "RSA_4096",
+                    "EC_prime256v1",
+                    "EC_secp384r1",
+                    "EC_secp521r1",
+                ],
+            },
+        }
         if statuses:
             kwargs["CertificateStatuses"] = statuses
         return paginator.paginate(**kwargs).build_full_result()["CertificateSummaryList"]

plugins/module_utils/arn.py

@@ -6,14 +6,46 @@
 import re
 
 
+def validate_aws_arn(
+    arn, partition=None, service=None, region=None, account_id=None, resource=None, resource_type=None, resource_id=None
+):
+    details = parse_aws_arn(arn)
+
+    if not details:
+        return False
+
+    if partition and details.get("partition") != partition:
+        return False
+    if service and details.get("service") != service:
+        return False
+    if region and details.get("region") != region:
+        return False
+    if account_id and details.get("account_id") != account_id:
+        return False
+    if resource and details.get("resource") != resource:
+        return False
+    if resource_type and details.get("resource_type") != resource_type:
+        return False
+    if resource_id and details.get("resource_id") != resource_id:
+        return False
+
+    return True
+
+
 def parse_aws_arn(arn):
     """
+    Based on https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html
+
     The following are the general formats for ARNs.
         arn:partition:service:region:account-id:resource-id
         arn:partition:service:region:account-id:resource-type/resource-id
         arn:partition:service:region:account-id:resource-type:resource-id
     The specific formats depend on the resource.
     The ARNs for some resources omit the Region, the account ID, or both the Region and the account ID.
+
+    Note: resource_type handling is very naive, for complex cases it may be necessary to use
+    "resource" directly instead of resource_type, this will include the resource type and full ID,
+    including all paths.
     """
     m = re.search(r"arn:(aws(-([a-z\-]+))?):([\w-]+):([a-z0-9\-]*):(\d*|aws|aws-managed):(.*)", arn)
     if m is None:
@@ -25,6 +57,12 @@ def parse_aws_arn(arn):
     result.update(dict(account_id=m.group(6)))
     result.update(dict(resource=m.group(7)))
 
+    m2 = re.search(r"^(.*?)[:/](.+)$", m.group(7))
+    if m2 is None:
+        result.update(dict(resource_type=None, resource_id=m.group(7)))
+    else:
+        result.update(dict(resource_type=m2.group(1), resource_id=m2.group(2)))
+
     return result
 
 

plugins/module_utils/backup.py

@@ -144,7 +144,7 @@ def get_selection_details(module, client, plan_name: str, selection_name: Union[
                 if selection["SelectionName"] == selection_name:
                     selection_id = selection["SelectionId"]
                     result.append(_get_backup_selection(client, module, plan_id, selection_id))
-                break
+                    break
     else:
         for selection in selection_list:
             selection_id = selection["SelectionId"]