aws_ssm: ability to customize s3 endpoint for vpc interface endpoint (#…

…1619) (#1684)

[PR #1619/433f6c5e backport][stable-5] aws_ssm: ability to customize s3 endpoint for vpc interface endpoint

This is a backport of PR #1619 as merged into main (433f6c5).
Depends-On: ansible/ansible-zuul-jobs#1743
SUMMARY
Add a new variable for setting the s3 endpoint url
ISSUE TYPE

Feature Pull Request

COMPONENT NAME
connection aws_ssm.py
ADDITIONAL INFORMATION
If you try to running SSM commands on EC2 instances in private networks only with vpc interface endpoints. You are not able to access S3 service because the generated URL is wrong. For now this plugin only works for s3 vpc gateway endpoints. Not for s3 vpc interface endpoints.
To simply fix this. We need the possibility to set the interface endpoint url. So I added a new parameter to the connection plugin.
How to test
- name: test ssm on an EC2 instance
  hosts: router
  vars:
    ansible_python_interpreter: /usr/bin/python3
    ansible_connection: aws_ssm
    ansible_aws_ssm_region: 'eu-west-1'
    ansible_aws_ssm_bucket_name: testbucket
    ansible_aws_ssm_bucket_endpoint_url: "https://vpce-00000000000000-10ygvtbr.s3.eu-west-1.vpce.amazonaws.com"
  tasks:
    - name: list files in opt folder
      shell: echo "running on $(curl -s http://169.254.169.254/latest/meta-data/instance-id)"

Output before change
XEC curl 'https://testbucket.s3.amazonaws.com/i-0b5e11951ab6b12cd//home/ssm-user/.ansible/tmp/ansible-tmp-1610187779.9946203-6452-52757228453165/AnsiballZ_setup.py?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAZEY6F5MCGGCDZQGI%2F20210109%2F**eu-central-1**%2Fs3%2Faws4_request&X-Amz-Date=20210109T102300Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEKr%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDGV1LWNlbnRyYWwtMSJIMEYCIQDRoCbT6dS9geijC00Xhr4nKdDrfKSE0ULsEXNgjM3vUwIhAMoAiDJJSGehMKvcmUlZDHc17WcV3Wnw4lsCED4MH%2BkMKroDCHMQABoMNjI4NzM5MTQwMzU2IgylHaq9VXBqdPex8fsqlwMzbTC5nczwsbUzXkpdw1MWndywQnjxp%2BnZoYcHMba6TGM57osVwt6hQoYxKA04co63FOr%2FtvhmmLGdphxeEGBPRjyTCNB%2Bdtr%2BwfKmjyls7WmBQF4jRMm2xPMUSd3EBnitCOpRvHPtp4xsuIX59QKCZmUNKBYIn2USx18mcSrWpI1emQGkmgewn9EOxUT168X9unNnvmUerokKgD5f1dZvpnIEmUyPYhYFCkJAdmLa5E5CIWe4UFfULLwDwTqYe6akqSAhBUeMrzWvebp7oXkER%2BymsmdGdAl4nFKNDtJ5suSkcGooliKsFhrHKEb1gN4UH%2FldPSFZqCEOayiWByk6SK7yEkhqI7wbc5Ufwv68AimpRddA5dU95kXUL3tgBYq5QcSeXStdd%2B6nQ3vRDBJx%2BETvR2dGOeZv%2Bu6p1iLaT5wnMgMcSnPQWCTja%2Bnf7Lp%2Bkmd4pR9yfTYaPa%2FVdblsVAXtfDURQ7wHwV6DJJavt26oUXNOOjEXg4FDraLzGSNWGFjMkbxLSBFNEyKBB9g3Hk8hV4YOwjC4%2BeX%2FBTrqAfEXoF92NloBKePOvKXzFcp8YT8yC0p35rXYqa0GA5d9ZNaGewFw6ks9VMUTSht3SZ2ns2qCYF6p73ISe88pgrUWGwFaxZnbNxP1dvfpNH3X9zQ9oVyjKfD9dwPfnOYpx6j48dZZhdgZ6n2H13h3Ckf7hmebHo7po%2BWrXkc8K1Bo07YSFyFMffieXBk0NvrBPGNGtKTEJ3m%2FfF4vkM4lnEN2xWaS0umgwMQrCqfhKD3Gpf%2BdglVJ2oBRHb3ho7dfie48ohpAd%2B6j752PjR2SuaA3Gokns8scHLBB7dvkGlmqb3vrX8TYsc7CZg%3D%3D&X-Amz-Signature=a992947e6682f66239d42c08a6b8ba2da136a572bc15a2184763846fd5a39504' -o '/home/ssm-user/.ansible/tmp/ansible-tmp-1610187779.9946203-6452-52757228453165/AnsiballZ_setup.py'

Output after change
XEC curl 'https://testbucket.vpce-00000000000000-10ygvtbr.s3.eu-west-1.vpce.amazonaws.com/i-0b5e11951ab6b12cd//home/ssm-user/.ansible/tmp/ansible-tmp-1610187779.9946203-6452-52757228453165/AnsiballZ_setup.py?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAZEY6F5MCGGCDZQGI%2F20210109%2F**eu-central-1**%2Fs3%2Faws4_request&X-Amz-Date=20210109T102300Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEKr%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDGV1LWNlbnRyYWwtMSJIMEYCIQDRoCbT6dS9geijC00Xhr4nKdDrfKSE0ULsEXNgjM3vUwIhAMoAiDJJSGehMKvcmUlZDHc17WcV3Wnw4lsCED4MH%2BkMKroDCHMQABoMNjI4NzM5MTQwMzU2IgylHaq9VXBqdPex8fsqlwMzbTC5nczwsbUzXkpdw1MWndywQnjxp%2BnZoYcHMba6TGM57osVwt6hQoYxKA04co63FOr%2FtvhmmLGdphxeEGBPRjyTCNB%2Bdtr%2BwfKmjyls7WmBQF4jRMm2xPMUSd3EBnitCOpRvHPtp4xsuIX59QKCZmUNKBYIn2USx18mcSrWpI1emQGkmgewn9EOxUT168X9unNnvmUerokKgD5f1dZvpnIEmUyPYhYFCkJAdmLa5E5CIWe4UFfULLwDwTqYe6akqSAhBUeMrzWvebp7oXkER%2BymsmdGdAl4nFKNDtJ5suSkcGooliKsFhrHKEb1gN4UH%2FldPSFZqCEOayiWByk6SK7yEkhqI7wbc5Ufwv68AimpRddA5dU95kXUL3tgBYq5QcSeXStdd%2B6nQ3vRDBJx%2BETvR2dGOeZv%2Bu6p1iLaT5wnMgMcSnPQWCTja%2Bnf7Lp%2Bkmd4pR9yfTYaPa%2FVdblsVAXtfDURQ7wHwV6DJJavt26oUXNOOjEXg4FDraLzGSNWGFjMkbxLSBFNEyKBB9g3Hk8hV4YOwjC4%2BeX%2FBTrqAfEXoF92NloBKePOvKXzFcp8YT8yC0p35rXYqa0GA5d9ZNaGewFw6ks9VMUTSht3SZ2ns2qCYF6p73ISe88pgrUWGwFaxZnbNxP1dvfpNH3X9zQ9oVyjKfD9dwPfnOYpx6j48dZZhdgZ6n2H13h3Ckf7hmebHo7po%2BWrXkc8K1Bo07YSFyFMffieXBk0NvrBPGNGtKTEJ3m%2FfF4vkM4lnEN2xWaS0umgwMQrCqfhKD3Gpf%2BdglVJ2oBRHb3ho7dfie48ohpAd%2B6j752PjR2SuaA3Gokns8scHLBB7dvkGlmqb3vrX8TYsc7CZg%3D%3D&X-Amz-Signature=a992947e6682f66239d42c08a6b8ba2da136a572bc15a2184763846fd5a39504' -o '/home/ssm-user/.ansible/tmp/ansible-tmp-1610187779.9946203-6452-52757228453165/AnsiballZ_setup.py'

Reviewed-by: Mark Chappell <None>

changelogs/fragments/1619-add-s3-bucket-endpoint-url-var-for-private-network-vpc-interface-endpoints.yml

@@ -0,0 +1,2 @@
+minor_changes:
+- aws_ssm - added support for specifying the endpoint to use when connecting to the S3 API (https://github.com/ansible-collections/community.aws/pull/1619).

plugins/connection/aws_ssm.py

@@ -48,6 +48,11 @@
     description: The name of the S3 bucket used for file transfers.
     vars:
     - name: ansible_aws_ssm_bucket_name
+  bucket_endpoint_url:
+    description: The S3 endpoint URL of the bucket used for file transfers.
+    vars:
+    - name: ansible_aws_ssm_bucket_endpoint_url
+    version_added: 5.3.0
   plugin:
     description: This defines the location of the session-manager-plugin binary.
     vars:
@@ -351,10 +356,12 @@ def _vvvv(self, message):
         self._display(display.vvvv, message)
 
     def _get_bucket_endpoint(self):
-        # Fetch the correct S3 endpoint for use with our bucket.
-        # If we don't explicitly set the endpoint then some commands will use the global
-        # endpoint and fail
-        # (new AWS regions and new buckets in a region other than the one we're running in)
+        """
+        Fetches the correct S3 endpoint and region for use with our bucket.
+        If we don't explicitly set the endpoint then some commands will use the global
+        endpoint and fail
+        (new AWS regions and new buckets in a region other than the one we're running in)
+        """
 
         region_name = self.get_option('region') or 'us-east-1'
         profile_name = self.get_option('profile') or ''
@@ -368,6 +375,10 @@ def _get_bucket_endpoint(self):
             Bucket=(self.get_option('bucket_name')),
         )
         bucket_region = bucket_location['LocationConstraint']
+
+        if self.get_option("bucket_endpoint_url"):
+            return self.get_option("bucket_endpoint_url"), bucket_region
+
         # Create another client for the region the bucket lives in, so we can nab the endpoint URL
         self._vvvv(f"_get_bucket_endpoint: S3 (bucket region) - {bucket_region}")
         s3_bucket_client = self._get_boto_client(

tests/integration/targets/connection_aws_ssm_endpoint/aliases

@@ -0,0 +1,4 @@
+time=10m
+
+cloud/aws
+connection_aws_ssm

tests/integration/targets/connection_aws_ssm_endpoint/aws_ssm_integration_test_setup.yml

@@ -0,0 +1,7 @@
+- hosts: localhost
+  roles:
+    - role: ../setup_connection_aws_ssm
+      vars:
+        target_os: fedora
+        test_suffix: endpoint
+        endpoint_url: 'https://s3.dualstack.{{ aws_region }}.amazonaws.com'

tests/integration/targets/connection_aws_ssm_endpoint/aws_ssm_integration_test_teardown.yml

@@ -0,0 +1,5 @@
+- hosts: localhost
+  tasks:
+    - include_role:
+        name: ../setup_connection_aws_ssm
+        tasks_from: cleanup.yml

tests/integration/targets/connection_aws_ssm_endpoint/meta/main.yml

@@ -0,0 +1,3 @@
+dependencies:
+  - connection
+  - setup_connection_aws_ssm

tests/integration/targets/connection_aws_ssm_endpoint/runme.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+PLAYBOOK_DIR=$(pwd)
+set -eux
+
+CMD_ARGS=("$@")
+
+# Destroy Environment
+cleanup() {
+
+    cd "${PLAYBOOK_DIR}"
+    ansible-playbook -c local aws_ssm_integration_test_teardown.yml "${CMD_ARGS[@]}"
+
+}
+
+trap "cleanup" EXIT
+
+# Setup Environment
+ansible-playbook -c local aws_ssm_integration_test_setup.yml "$@"
+
+# Export the AWS Keys
+set +x
+. ./aws-env-vars.sh
+set -x
+
+cd ../connection
+
+# Execute Integration tests
+INVENTORY="${PLAYBOOK_DIR}/ssm_inventory" ./test.sh \
+    -e target_hosts=aws_ssm \
+    "$@"

tests/integration/targets/setup_connection_aws_ssm/templates/inventory-combined.aws_ssm.j2

@@ -44,6 +44,9 @@ ansible_aws_ssm_bucket_sse_kms_key_id=alias/{{ kms_key_name }}
 {% if use_ssm_document | default(False) %}
 ansible_aws_ssm_document={{ ssm_document_name }}
 {% endif %}
+{% if endpoint_url | default(False) %}
+ansible_aws_ssm_bucket_endpoint_url={{ endpoint_url }}
+{% endif %}
 
 # support tests that target testhost
 [testhost:children]