Scaleway s3 bucket: NoSuchPolicyConfiguration #422

psykotox · 2021-07-28T12:38:06Z

Summary

I have an error when I declare a s3 bucket hosted at Scaleway : NoSuchPolicyConfiguration

Issue Type

Bug Report

Component Name

s3_bucket

Ansible Version

ansible 2.10.8
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.7.3 (default, Dec 20 2019, 18:57:59) [GCC 8.3.0]

Collection Versions

amazon.aws                1.4.0

AWS SDK versions

Name: boto3
Version: 1.9.86
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: UNKNOWN
License: Apache License 2.0
Location: /usr/lib/python3/dist-packages
Requires: 
Required-by: 
---
Name: botocore
Version: 1.12.103
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: UNKNOWN
License: Apache License 2.0
Location: /usr/lib/python3/dist-packages
Requires: 
Required-by:

Configuration

Default configuration

OS / Environment

Debian 10.3

Steps to Reproduce

- name: Create bucket
  s3_bucket:
    aws_access_key: XXXXXXXXXXXX
    aws_secret_key: "{{ vault_scaleway_secret }}"
    s3_url: https://s3.fr-par.scw.cloud
    region: fr-par
    name: mybucket
    state: present
    requester_pays:

I use the same configuration: https://docs.ansible.com/ansible/latest/scenario_guides/guide_scaleway.html

Expected Results

Bucket created

Actual Results

The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_s3_bucket_payload_0avx_p99/ansible_s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 319, in create_or_update_bucket
  File "/tmp/ansible_s3_bucket_payload_0avx_p99/ansible_s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 154, in retry_func
    raise e
  File "/tmp/ansible_s3_bucket_payload_0avx_p99/ansible_s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 144, in retry_func
    return f(*args, **kwargs)
  File "/tmp/ansible_s3_bucket_payload_0avx_p99/ansible_s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 489, in get_bucket_policy
  File "/usr/lib/python3/dist-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3/dist-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (NoSuchPolicyConfiguration) when calling the GetBucketPolicy operation: The policy configuration does not exist.
fatal: [arena.office]: FAILED! => {
    "boto3_version": "1.9.86",
    "botocore_version": "1.12.103",
    "changed": false,
    "error": {
        "code": "NoSuchPolicyConfiguration",
        "message": "The policy configuration does not exist."
    },
    "invocation": {
        "module_args": {
            "aws_access_key": "XXXXXXXXXXXX",
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "ceph": false,
            "debug_botocore_endpoint_logs": false,
            "delete_public_access": false,
            "ec2_url": null,
            "encryption": null,
            "encryption_key_id": null,
            "force": false,
            "name": "mybucket",
            "policy": null,
            "profile": null,
            "public_access": null,
            "purge_tags": true,
            "region": "fr-par",
            "requester_pays": null,
            "s3_url": "https://s3.fr-par.scw.cloud",
            "security_token": null,
            "state": "present",
            "tags": null,
            "validate_certs": true,
            "versioning": null
        }
    },
    "msg": "Failed to get bucket policy: An error occurred (NoSuchPolicyConfiguration) when calling the GetBucketPolicy operation: The policy configuration does not exist.",
    "response_metadata": {
        "host_id": "txxxxxf15xx4ca4axdbbxxxxxx-xxxxxxxxxf1d",
        "http_headers": {
            "content-type": "application/xml",
            "date": "Wed, 28 Jul 2021 12:35:41 GMT",
            "transfer-encoding": "chunked",
            "x-amz-id-2": "txxxxxxxf15xx4ca4axdbbxxxxxx-xxxxxxxxxd",
            "x-amz-request-id": "txxxxxxxf15384ca4a9dbbxxxxxxx-xxxxxxxxxxf1d"
        },
        "http_status_code": 404,
        "request_id": "txxxxxxxf15384ca4a9dbbxxxxxxx-xxxxxxxxxxf1d",
        "retry_attempts": 0
    }
}

Code of Conduct

I agree to follow the Ansible Code of Conduct

The text was updated successfully, but these errors were encountered:

ansibullbot · 2021-07-28T12:49:52Z

Files identified in the description:
None

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot · 2021-07-28T13:49:58Z

Files identified in the description:

[plugins/modules/s3_bucket.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/s3_bucket.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot · 2021-07-28T13:49:59Z

cc @abikouo @jillr @s-hertel @tremble @wimnat
click here for bot help

jillr · 2021-08-24T19:59:35Z

Hi @psykotox, this sounds like the Scaleway object storage API is not implementing a supported aspect of the AWS S3 API. We don't have access to any Scaleway services to test with, I'd recommend reaching out to the vendor to ask about this. If you would like to open a PR that enables the necessary functionality while maintaining the existing S3 functionality we'd be open to that.

goneri · 2021-08-24T20:00:22Z

Hi Rémy (@remyleone)!

Can you help us troubleshoot this compatibility problem between s3 module and Scaleway?

angristan · 2021-08-30T10:12:35Z

Hello 👋

It looks like the issue happens here:

amazon.aws/plugins/modules/s3_bucket.py

Lines 359 to 366 in e8df917

    
           # Policy 
        
           try: 
        
               current_policy = get_bucket_policy(s3_client, name) 
        
           except is_boto3_error_code(['NotImplemented', 'XNotImplemented']) as e: 
        
               if policy is not None: 
        
                   module.fail_json_aws(e, msg="Failed to get bucket policy") 
        
           except (botocore.exceptions.BotoCoreError, botocore.exceptions.ClientError) as e:  # pylint: disable=duplicate-except 
        
               module.fail_json_aws(e, msg="Failed to get bucket policy")

Which calls:

amazon.aws/plugins/modules/s3_bucket.py

Lines 552 to 559 in e8df917

    
           @AWSRetry.exponential_backoff(max_delay=120, catch_extra_error_codes=['NoSuchBucket', 'OperationAborted']) 
        
           def get_bucket_policy(s3_client, bucket_name): 
        
               try: 
        
                   current_policy = json.loads(s3_client.get_bucket_policy(Bucket=bucket_name).get('Policy')) 
        
               except is_boto3_error_code('NoSuchBucketPolicy'): 
        
                   return None 
        
               return current_policy

It expects a NoSuchBucketPolicy code when there is no policy.

➜  ~ aws --profile aws s3api get-bucket-policy --bucket xxxx

An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist

➜  ~ aws --profile scw s3api get-bucket-policy --bucket xxxx

An error occurred (NoSuchPolicyConfiguration) when calling the GetBucketPolicy operation: The policy configuration does not exist.

^ We don't return the correct error code. Somehow, we missed this when implementing the bucket policy feature. We'll fix it and we'll update the issue once it's corrected 👍

angristan · 2021-09-08T20:48:07Z

The bug has been fixed and deployed.

There is now a new issue:

TASK [Create bucket] ***********************************************************************************************************************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: botocore.exceptions.ClientError: An error occurred (NotImplemented) when calling the GetBucketEncryption operation: The requested resource is not implemented
fatal: [localhost]: FAILED! => {"boto3_version": "1.17.94", "botocore_version": "1.20.94", "changed": false, "error": {"code": "NotImplemented", "message": "The requested resource is not implemented"}, "msg": "Failed to get bucket encryption: An error occurred (NotImplemented) when calling the GetBucketEncryption operation: The requested resource is not implemented", "response_metadata": {"host_id": "tx0f65a8f906344be3ba694-0061391b6e", "http_headers": {"content-type": "application/xml", "date": "Wed, 08 Sep 2021 20:22:06 GMT", "transfer-encoding": "chunked", "x-amz-id-2": "tx0f65a8f906344be3ba694-0061391b6e", "x-amz-request-id": "tx0f65a8f906344be3ba694-0061391b6e"}, "http_status_code": 501, "request_id": "tx0f65a8f906344be3ba694-0061391b6e", "retry_attempts": 0}}

In the full trace we can see that the GetBucketEncryption call returns NotImplemented, which is expected as Scaleway doesn't support it. We can also see that "encryption": null.

Traceback (most recent call last):
  File "/var/folders/3f/clc85fc139511_z9gyylkckr0000gn/T/ansible_s3_bucket_payload_39nrbzz6/ansible_s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 388, in create_or_update_bucket
  File "/var/folders/3f/clc85fc139511_z9gyylkckr0000gn/T/ansible_s3_bucket_payload_39nrbzz6/ansible_s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 154, in retry_func
    raise e
  File "/var/folders/3f/clc85fc139511_z9gyylkckr0000gn/T/ansible_s3_bucket_payload_39nrbzz6/ansible_s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 144, in retry_func
    return f(*args, **kwargs)
  File "/var/folders/3f/clc85fc139511_z9gyylkckr0000gn/T/ansible_s3_bucket_payload_39nrbzz6/ansible_s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 522, in get_bucket_encryption
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (NotImplemented) when calling the GetBucketEncryption operation: The requested resource is not implemented
fatal: [localhost]: FAILED! => {
    "boto3_version": "1.17.94",
    "botocore_version": "1.20.94",
    "changed": false,
    "error": {
        "code": "NotImplemented",
        "message": "The requested resource is not implemented"
    },
    "invocation": {
        "module_args": {
            "aws_access_key": "SCWxxxxxxx",
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "ceph": false,
            "debug_botocore_endpoint_logs": false,
            "delete_public_access": false,
            "ec2_url": null,
            "encryption": null,
            "encryption_key_id": null,
            "force": false,
            "name": "mybucketdsadssdsa",
            "policy": null,
            "profile": null,
            "public_access": null,
            "purge_tags": true,
            "region": "fr-par",
            "requester_pays": null,
            "s3_url": "https://s3.fr-par.scw.cloud",
            "security_token": null,
            "state": "present",
            "tags": null,
            "validate_certs": true,
            "versioning": null
        }
    },
    "msg": "Failed to get bucket encryption: An error occurred (NotImplemented) when calling the GetBucketEncryption operation: The requested resource is not implemented",
    "response_metadata": {
        "host_id": "txfa6ac2b3054f4423adc6a-0061391d85",
        "http_headers": {
            "content-type": "application/xml",
            "date": "Wed, 08 Sep 2021 20:31:01 GMT",
            "transfer-encoding": "chunked",
            "x-amz-id-2": "txfa6ac2b3054f4423adc6a-0061391d85",
            "x-amz-request-id": "txfa6ac2b3054f4423adc6a-0061391d85"
        },
        "http_status_code": 501,
        "request_id": "txfa6ac2b3054f4423adc6a-0061391d85",
        "retry_attempts": 0
    }
}

It looks like the exception is raised from boto and not catched: botocore.exceptions.ClientError: An error occurred (NotImplemented) is raised.

Looking at the code:

amazon.aws/plugins/modules/s3_bucket.py

Lines 428 to 437 in e8df917

    
           # Encryption 
        
           try: 
        
               current_encryption = get_bucket_encryption(s3_client, name) 
        
           except is_boto3_error_code(['NotImplemented', 'XNotImplemented']) as e: 
        
               if encryption is not None: 
        
                   module.fail_json_aws(e, msg="Failed to get bucket encryption settings") 
        
           except (botocore.exceptions.BotoCoreError, botocore.exceptions.ClientError) as e:  # pylint: disable=duplicate-except 
        
               module.fail_json_aws(e, msg="Failed to get bucket encryption settings") 
        
           else: 
        
               if encryption is not None:

I understand that we should fall in except is_boto3_error_code(['NotImplemented', 'XNotImplemented']) as e: but not in if encryption is not None:? Looking at the output, module.fail_json_aws(e, msg="Failed to get bucket encryption settings") doesn't seem to be called, but this is probably because the boto exception from get_bucket_encryption() doesn't seem to be catched.

Is this expected? What am I missing?

tremble · 2021-09-09T08:11:54Z

The code you're looking at there is our 'main' branch: the absolute latest code. Looking at the trace I suspect you're using a released version of the code.

Cleanly catching NotImplemented for encryption was added in #391 which, while merged, isn't released yet. I'd expect to see this available in version 2.0.0 of this collection which should be released by the end of the month.

angristan · 2021-09-09T20:08:16Z

Ah, thanks @tremble 👍

psykotox · 2021-09-10T08:02:32Z

Thx @angristan and @tremble

* Enable AWSRetry on aws_region_info * changelog

* Enable AWSRetry on aws_region_info * changelog This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections/community.aws@ea740a2

* Enable AWSRetry on aws_region_info * changelog

ansibullbot added bug This issue/PR relates to a bug needs_triage python3 traceback labels Jul 28, 2021

ansibullbot added module module plugins plugin (any type) labels Jul 28, 2021

jillr added waiting_on_contributor Needs help. Feel free to engage to get things unblocked and removed needs_triage labels Aug 24, 2021

tremble closed this as completed Sep 10, 2021

abikouo pushed a commit to abikouo/amazon.aws that referenced this issue Sep 18, 2023

Enable AWSRetry on aws_region_info (ansible-collections#422)

4982d14

* Enable AWSRetry on aws_region_info * changelog

abikouo pushed a commit to abikouo/amazon.aws that referenced this issue Sep 18, 2023

Enable AWSRetry on aws_region_info (ansible-collections#422)

dddb8d9

* Enable AWSRetry on aws_region_info * changelog

abikouo pushed a commit to abikouo/amazon.aws that referenced this issue Oct 24, 2023

Enable AWSRetry on aws_region_info (ansible-collections#422)

ea740a2

* Enable AWSRetry on aws_region_info * changelog

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scaleway s3 bucket: NoSuchPolicyConfiguration #422

Scaleway s3 bucket: NoSuchPolicyConfiguration #422

psykotox commented Jul 28, 2021 •

edited

Loading

ansibullbot commented Jul 28, 2021

ansibullbot commented Jul 28, 2021

ansibullbot commented Jul 28, 2021

jillr commented Aug 24, 2021

goneri commented Aug 24, 2021

angristan commented Aug 30, 2021

angristan commented Sep 8, 2021

tremble commented Sep 9, 2021

angristan commented Sep 9, 2021

psykotox commented Sep 10, 2021

Scaleway s3 bucket: NoSuchPolicyConfiguration #422

Scaleway s3 bucket: NoSuchPolicyConfiguration #422

Comments

psykotox commented Jul 28, 2021 • edited Loading

Summary

Issue Type

Component Name

Ansible Version

Collection Versions

AWS SDK versions

Configuration

OS / Environment

Steps to Reproduce

Expected Results

Actual Results

Code of Conduct

ansibullbot commented Jul 28, 2021

ansibullbot commented Jul 28, 2021

ansibullbot commented Jul 28, 2021

jillr commented Aug 24, 2021

goneri commented Aug 24, 2021

angristan commented Aug 30, 2021

angristan commented Sep 8, 2021

tremble commented Sep 9, 2021

angristan commented Sep 9, 2021

psykotox commented Sep 10, 2021

psykotox commented Jul 28, 2021 •

edited

Loading