s3_bucket -- DigitialOcean Spaces compatibility #390

eeshugerman · 2021-06-20T16:14:58Z

SUMMARY

I'm upgrading from Ansible v2.8 to v4.1. With v2.8, I was able to use s3_bucket with DO Spaces -- I just needed to set requester_pays: null. Now, with v4.1, it seems there's no way to prevent the module's get_bucket_encryption call, and DO Spaces doesn't implement encryption stuff, so the task fails. I tried all of the permitted values for encryption ('none', 'AES256' 'aws:kms'), as well as not setting it at all -- it made no difference.

See https://github.com/ansible-collections/amazon.aws/blob/main/plugins/modules/s3_bucket.py#L420-L426.
My naive assessment is that the get_bucket_encryption call (and its try/except block) would need to be moved into the if encryption is not None block.

ISSUE TYPE

Bug Report

COMPONENT NAME

s3_bucket

ANSIBLE VERSION

ansible [core 2.11.1] 
  config file = /home/elliott/devel/PikaNet/ansible.cfg
  configured module search path = ['/home/elliott/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/elliott/.virtualenvs/ansible-4/lib/python3.9/site-packages/ansible
  ansible collection location = /home/elliott/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/elliott/.virtualenvs/ansible-4/bin/ansible
  python version = 3.9.5 (default, May 24 2021, 12:50:35) [GCC 11.1.0]
  jinja version = 3.0.1
  libyaml = True

CONFIGURATION

ANSIBLE_FORCE_COLOR(/home/elliott/devel/PikaNet/ansible.cfg) = True
DEFAULT_STDOUT_CALLBACK(/home/elliott/devel/PikaNet/ansible.cfg) = yaml
DEFAULT_VAULT_PASSWORD_FILE(/home/elliott/devel/PikaNet/ansible.cfg) = /home/elliott/devel/PikaNet/.vault_pass
INTERPRETER_PYTHON(/home/elliott/devel/PikaNet/ansible.cfg) = auto

OS / ENVIRONMENT

Ubuntu 18.04

STEPS TO REPRODUCE

amazon.aws.s3_bucket:
  name: 'my-bucket'
  state: present
  aws_access_key: '{{ digitalocean_spaces.key }}'
  aws_secret_key: '{{ digitalocean_spaces.secret }}'
  s3_url: 'https://sfo2.digitaloceanspaces.com'

EXPECTED RESULTS

The task succeeds.

ACTUAL RESULTS

The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_amazon.aws.s3_bucket_payload_murbosko/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 388, in create_or_update_bucket
  File "/tmp/ansible_amazon.aws.s3_bucket_payload_murbosko/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 154, in retry_func
    raise e
  File "/tmp/ansible_amazon.aws.s3_bucket_payload_murbosko/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 144, in retry_func
    return f(*args, **kwargs)
  File "/tmp/ansible_amazon.aws.s3_bucket_payload_murbosko/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 522, in get_bucket_encryption
  File "/usr/lib/python3/dist-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3/dist-packages/botocore/client.py", line 635, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (NotImplemented) when calling the GetBucketEncryption operation: Server does not support the functionality required to fulfill the request. Please see https://developers.digitalocean.com/documentation/spaces/#aws-s3-compatibility
fatal: [default]: FAILED! => changed=false 
  boto3_version: 1.4.2
  botocore_version: 1.16.19
  error:
    code: NotImplemented
    message: Server does not support the functionality required to fulfill the request. Please see https://developers.digitalocean.com/documentation/spaces/#aws-s3-compatibility
  invocation:
    module_args:
      aws_access_key: <REDACTED>
      aws_ca_bundle: null
      aws_config: null
      aws_secret_key: VALUE_SPECIFIED_IN_NO_LOG_PARAMETER
      ceph: false
      debug_botocore_endpoint_logs: false
      delete_public_access: false
      ec2_url: null
      encryption: null
      encryption_key_id: null
      force: false
      name: my-bucket
      policy: null
      profile: null
      public_access: null
      purge_tags: true
      region: null
      requester_pays: null
      s3_url: https://sfo2.digitaloceanspaces.com
      security_token: null
      state: present
      tags: null
      validate_certs: true
      versioning: null
  msg: 'Failed to get bucket encryption: An error occurred (NotImplemented) when calling the GetBucketEncryption operation: Server does not support the functionality required to fulfill the request. Please see https://developers.digitalocean.com/documentation/spaces/#aws-s3-compatibility'
  response_metadata:
    http_headers:
      content-length: '268'
      content-type: application/xml
      strict-transport-security: max-age=15552000; includeSubDomains; preload
      x-amz-error-code: NotImplemented
      x-amz-error-message: Server does not support the functionality required to fulfill the request. Please see https://developers.digitalocean.com/documentation/spaces/#aws-s3-compatibility
      x-do-spaces-error: unsupported_operation_s3:GetBucketEncryptionConfiguration
    http_status_code: 501
    retry_attempts: 0

The text was updated successfully, but these errors were encountered:

ansibullbot · 2021-06-20T16:20:44Z

Files identified in the description:
None

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

eeshugerman · 2021-06-20T18:02:48Z

!component =plugins/modules/s3_bucket.py

ansibullbot · 2021-06-20T18:06:28Z

Files identified in the description:

[plugins/modules/s3_bucket.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/s3_bucket.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot · 2021-06-20T18:06:29Z

cc @abikouo @jillr @s-hertel @tremble @wimnat
click here for bot help

s3_bucket - handle "NotImplemented" for encryption SUMMARY fixes: #390 DigitalOcean doesn't support the encryption settings (and returns a NotImplemented error). Additionally: Slight logic tweak to avoid the line-too-long linting error Avoid importing specific exceptions for consistency with other modules ISSUE TYPE Bugfix Pull Request COMPONENT NAME s3_bucket ADDITIONAL INFORMATION Reviewed-by: Alina Buzachis <None>

tremble · 2021-06-30T09:06:02Z

The fix for this issue should be available when we release version 2.0.0 of the amazon.aws collection.

eeshugerman · 2021-06-30T20:09:42Z

thanks @tremble!

ansibullbot added bug This issue/PR relates to a bug needs_triage python3 traceback labels Jun 20, 2021

ansibullbot added module module plugins plugin (any type) labels Jun 20, 2021

tremble mentioned this issue Jun 21, 2021

s3_bucket - handle "NotImplemented" for encryption #391

Merged

gravesm removed the needs_triage label Jun 29, 2021

ansibullbot added the has_pr label Jun 29, 2021

ansible-zuul bot closed this as completed in #391 Jun 30, 2021

tremble mentioned this issue Oct 31, 2021

add object ownership controls options for s3 bucket #311

Merged

abikouo pushed a commit to abikouo/amazon.aws that referenced this issue Oct 24, 2023

Update boto3 versions for elbv2 tests (ansible-collections#390)

17fe59a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

s3_bucket -- DigitialOcean Spaces compatibility #390

s3_bucket -- DigitialOcean Spaces compatibility #390

eeshugerman commented Jun 20, 2021 •

edited

Loading

ansibullbot commented Jun 20, 2021

eeshugerman commented Jun 20, 2021

ansibullbot commented Jun 20, 2021

ansibullbot commented Jun 20, 2021

tremble commented Jun 30, 2021

eeshugerman commented Jun 30, 2021

s3_bucket -- DigitialOcean Spaces compatibility #390

s3_bucket -- DigitialOcean Spaces compatibility #390

Comments

eeshugerman commented Jun 20, 2021 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

ansibullbot commented Jun 20, 2021

eeshugerman commented Jun 20, 2021

ansibullbot commented Jun 20, 2021

ansibullbot commented Jun 20, 2021

tremble commented Jun 30, 2021

eeshugerman commented Jun 30, 2021

eeshugerman commented Jun 20, 2021 •

edited

Loading