5.0.0 : aws_s3 may fail on some S3 compatible provider that do not implement OwnershipControls #1115

VertPyDev · 2022-10-05T12:11:26Z

Summary

On the version 5.0.0, because of #921, we can have failure using aws_s3 on some non aws provider that are compatible with s3 but that dos not implement OwnershipControls (ie scaleway) :

module_stderr: |-
    Traceback (most recent call last):
      File "<stdin>", line 107, in <module>
      File "<stdin>", line 99, in _ansiballz_main
      File "<stdin>", line 47, in invoke_module
      File "/usr/lib/python3.10/runpy.py", line 224, in run_module
        return _run_module_code(code, init_globals, run_name, mod_spec)
      File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
        _run_code(code, mod_globals, init_globals,
      File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
        exec(code, run_globals)
      File "/tmp/ansible_amazon.aws.aws_s3_payload_r6y8tobh/ansible_amazon.aws.aws_s3_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 1274, in <module>
      File "/tmp/ansible_amazon.aws.aws_s3_payload_r6y8tobh/ansible_amazon.aws.aws_s3_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 1086, in main
    KeyError: 'Rules'
  module_stdout: ''

The problem is on s3_object.py. When calling s3.get_bucket_ownership_controls(Bucket=bucket)['OwnershipControls']['Rules'][0]['ObjectOwnership']. If OwnershipControls is null or empty, Rules does not exists.

This is working with version 4.2.0 of the collection.

Issue Type

Bug Report

Component Name

aws_s3

Ansible Version

ansible [core 2.13.4]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/guillaume/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/guillaume/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/guillaume/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/guillaume/.local/bin/ansible
  python version = 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
  jinja version = 3.0.2
  libyaml = True

Collection Versions

amazon.aws:5.0.0

AWS SDK versions

Name: boto3
Version: 1.24.26
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /home/guillaume/.local/lib/python3.8/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.27.26
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /home/guillaume/.local/lib/python3.8/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

No response

OS / Environment

Debian 11

Steps to Reproduce

Try to download a Scaleway bucket (You must have a Scaleway account) :

ansible -m amazon.aws.aws_s3 -a"bucket=test-bucket mode=get object=test.tar.gz s3_url=https://s3.fr-par.scw.cloud region=fr-par dest=/tmp/test.tar.gz" localhost

Expected Results

Bucket is downloaded on /tmp/test.tar.gz without errors

Actual Results

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'Rules'
localhost | FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 107, in <module>\n  File \"<stdin>\", line 99, in _ansiballz_main\n  File \"<stdin>\", line 47, in invoke_module\n  File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_amazon.aws.aws_s3_payload_vzfqrf8o/ansible_amazon.aws.aws_s3_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py\", line 1274, in <module>\n  File \"/tmp/ansible_amazon.aws.aws_s3_payload_vzfqrf8o/ansible_amazon.aws.aws_s3_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py\", line 1086, in main\nKeyError: 'Rules'\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

Code of Conduct

I agree to follow the Ansible Code of Conduct

The text was updated successfully, but these errors were encountered:

ansibullbot · 2022-10-05T12:15:21Z

Files identified in the description:
None

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

tremble · 2022-10-05T12:23:44Z

Please work with Scaleway

If they're not going to implement an API in a compatible manner then they should raise either 'XNotImplemented' or 'NotImplemented' errors.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketOwnershipControls.html

glitchsys · 2022-10-05T17:42:43Z

Ran into this same problem. What used to work, is now broken. It worked fine in 4.2.0 but broke when we used 5.0.0
Getting this error: The error was: AttributeError: 'S3' object has no attribute 'get_bucket_ownership_controls

  amazon.aws.aws_s3:
    bucket: somes3bucket
    object: "test.zip"
    dest: /tmp/test.zip
    mode: get
    aws_access_key: ABC123
    aws_secret_key: ABCDEF123456789ABCDEF

I've had to use ansible-galaxy collection install amazon.aws:4.2.0 to compensate for this error.
I used to just grab the latest version (ansible-galaxy collection install amazon.aws) but can't any longer.

This is from one of our own owned S3 buckets. We have about a dozen S3 buckets and we get this error across most of them. It broke our deployment until I was able to force version 4.2.0 and then things began to work again.

tremble · 2022-10-05T19:47:49Z

@glitchsys AttributeError: 'S3' object has no attribute 'get_bucket_ownership_controls' is a completely different issue. You're seeing that because you're using botocore < 1.18.11.

Please see the module/collection documentation:
https://github.com/ansible-collections/amazon.aws#aws-sdk-version-compatibility :

Starting with the 2.0.0 releases of amazon.aws and community.aws, it is generally the collection's policy to support the versions of botocore and boto3 that were released 12 months prior to the most recent major collection release, following semantic versioning (for example, 2.0.0, 3.0.0).

Version 5.0.0 of this collection supports boto3 >= 1.18.0 and botocore >= 1.21.0

While things may (mostly) work with older versions of botocore, we don't make any guarantees, and we don't actively try to code work-arounds for missing features. The rate with with Amazon AWS are developing their APIs means that if we don't set a limit somewhere the code becomes very unwieldy. We've drawn the line at botocore releases ~1 year old.

glitchsys · 2022-10-05T19:58:12Z

@tremble just prior to ansible running, the build system, Ubuntu 20.04, did a "apt-get install python3-boto3" which would have installed python3-boto3 1.9.253-1 and python3-botocore 1.16.19+repack-1ubuntu0.20.04.1
Are you saying that the amazon.aws collection isn't compatible with the python3 packaged with Ubuntu 20.04 (LTS)? 4.2.0 just happened to work for our specific S3 needs with those older versions of botocore but 5.0.0 won't?
So my options are to either use 4.2.0 of amazon.aws or grab botocore from pip and not the system package repository (apt). I can do that, thanks for the info.

tremble · 2022-10-05T20:09:20Z

Are you saying that the amazon.aws collection isn't compatible with the python3 packaged with Ubuntu 20.04 (LTS)? 4.2.0 just happened to work for our specific S3 needs with those older versions of botocore but 5.0.0 won't?

Pretty much. We don't actively go out of our way to break things, (we don't throw an error just because you're using an old version), however if someone adds a new feature we only check that it works with the oldest version we support.

So my options are to either use 4.2.0 of amazon.aws or grab botocore from pip and not the system package repository (apt). I can do that, thanks for the info.

Yeah we used to try to support older versions, unfortunately the rate at which AWS is moving makes this impractical to test. We now deliberately test against the older version we support, where someone needs an even newer feature, support has to be coded in such a way that as far as is practical, only the new feature won't work, everything else should work.

…_ownership_controls (#1117) s3_object - Be more defensive when checking the results of get_bucket_ownership_controls SUMMARY Scaleway have half-implemented get_bucket_ownership_controls, but don't return any rules. Be a little more defensive when checking the return value of get_bucket_ownership_controls. The API doesn't strictly say a Rule will always be returned. Fixes: #1115 ISSUE TYPE Bugfix Pull Request COMPONENT NAME s3_object ADDITIONAL INFORMATION Reviewed-by: Alina Buzachis <None>

…_ownership_controls (#1117) s3_object - Be more defensive when checking the results of get_bucket_ownership_controls SUMMARY Scaleway have half-implemented get_bucket_ownership_controls, but don't return any rules. Be a little more defensive when checking the return value of get_bucket_ownership_controls. The API doesn't strictly say a Rule will always be returned. Fixes: #1115 ISSUE TYPE Bugfix Pull Request COMPONENT NAME s3_object ADDITIONAL INFORMATION Reviewed-by: Alina Buzachis <None> (cherry picked from commit 0c3239b)

…_ownership_controls (#1117) (#1122) SUMMARY Scaleway have half-implemented get_bucket_ownership_controls, but don't return any rules. Be a little more defensive when checking the return value of get_bucket_ownership_controls. The API doesn't strictly say a Rule will always be returned. Fixes: #1115 ISSUE TYPE Bugfix Pull Request COMPONENT NAME s3_object ADDITIONAL INFORMATION Reviewed-by: Alina Buzachis <None> (cherry picked from commit 0c3239b)

tremble · 2022-10-06T14:52:42Z

@VertPyDev

We've just released 5.0.1 which should fix your issue.

VertPyDev · 2022-10-07T08:51:01Z

Its working fine ! Thinks for fixing it so fast !

…ctions#1115) execute_lambda - fix check mode and update RETURN docs Depends-On: ansible-collections#1116 SUMMARY check_mode fix update RETURN docs to match what is actually being returned require one of name, function_arn ISSUE TYPE Bugfix Pull Request COMPONENT NAME execute_lambda ADDITIONAL INFORMATION I noticed some modules in community.aws will return data directly, and others will return data nested in a dict. Example: let iam_group be the module, retrieving a key called group_arn, and registering the response as response. Some modules you would need to query result.iam_group.group_arn, meanwhile in others, you can result.group_arn (where iam_group is assumed, since its the name of the module). Do we have a preference for either method? Should we come to some sort of collection-wide consensus on which to use moving forward? Reviewed-by: Sloane Hertel <None> Reviewed-by: Markus Bergholz <[email protected]> This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections/community.aws@f2ad637

…ctions#1115) execute_lambda - fix check mode and update RETURN docs Depends-On: ansible-collections#1116 SUMMARY check_mode fix update RETURN docs to match what is actually being returned require one of name, function_arn ISSUE TYPE Bugfix Pull Request COMPONENT NAME execute_lambda ADDITIONAL INFORMATION I noticed some modules in community.aws will return data directly, and others will return data nested in a dict. Example: let iam_group be the module, retrieving a key called group_arn, and registering the response as response. Some modules you would need to query result.iam_group.group_arn, meanwhile in others, you can result.group_arn (where iam_group is assumed, since its the name of the module). Do we have a preference for either method? Should we come to some sort of collection-wide consensus on which to use moving forward? Reviewed-by: Sloane Hertel <None> Reviewed-by: Markus Bergholz <[email protected]>

ansibullbot added bug This issue/PR relates to a bug needs_triage python3 traceback labels Oct 5, 2022

tremble mentioned this issue Oct 5, 2022

s3_object - Be more defensive when checking the results of get_bucket_ownership_controls #1117

Merged

ansibullbot added the has_pr label Oct 5, 2022

softwarefactory-project-zuul bot closed this as completed in #1117 Oct 5, 2022

patchback bot mentioned this issue Oct 5, 2022

[PR #1117/0c3239be backport][stable-5] s3_object - Be more defensive when checking the results of get_bucket_ownership_controls #1122

Merged

renovate bot mentioned this issue Oct 12, 2022

Update dependency amazon.aws to v5 - autoclosed platan/metrics-shields-io-config#109

Closed

1 task

renovate bot mentioned this issue Oct 27, 2022

Update dependency amazon.aws to v5 - autoclosed jkroepke/homelab#27

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

5.0.0 : aws_s3 may fail on some S3 compatible provider that do not implement OwnershipControls #1115

5.0.0 : aws_s3 may fail on some S3 compatible provider that do not implement OwnershipControls #1115

VertPyDev commented Oct 5, 2022 •

edited

Loading

ansibullbot commented Oct 5, 2022

tremble commented Oct 5, 2022 •

edited

Loading

glitchsys commented Oct 5, 2022 •

edited

Loading

tremble commented Oct 5, 2022 •

edited

Loading

glitchsys commented Oct 5, 2022

tremble commented Oct 5, 2022 •

edited

Loading

tremble commented Oct 6, 2022

VertPyDev commented Oct 7, 2022

5.0.0 : aws_s3 may fail on some S3 compatible provider that do not implement OwnershipControls #1115

5.0.0 : aws_s3 may fail on some S3 compatible provider that do not implement OwnershipControls #1115

Comments

VertPyDev commented Oct 5, 2022 • edited Loading

Summary

Issue Type

Component Name

Ansible Version

Collection Versions

AWS SDK versions

Configuration

OS / Environment

Steps to Reproduce

Expected Results

Actual Results

Code of Conduct

ansibullbot commented Oct 5, 2022

tremble commented Oct 5, 2022 • edited Loading

glitchsys commented Oct 5, 2022 • edited Loading

tremble commented Oct 5, 2022 • edited Loading

glitchsys commented Oct 5, 2022

tremble commented Oct 5, 2022 • edited Loading

tremble commented Oct 6, 2022

VertPyDev commented Oct 7, 2022

VertPyDev commented Oct 5, 2022 •

edited

Loading

tremble commented Oct 5, 2022 •

edited

Loading

glitchsys commented Oct 5, 2022 •

edited

Loading

tremble commented Oct 5, 2022 •

edited

Loading

tremble commented Oct 5, 2022 •

edited

Loading