amazon.aws.s3_bucket: Setting policy to None/null does not remove bucket policy #385

2d6 · 2021-06-12T09:51:39Z

SUMMARY

When removing the policy attribute from an s3_bucket task, the module does not seem to remove the policy and reports no changes.

ISSUE TYPE

Bug Report

COMPONENT NAME

s3_bucket

ANSIBLE VERSION

ansible [core 2.11.1] 
  config file = /path/to/my/project/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.9/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.9.5 (default, May 24 2021, 12:50:35) [GCC 11.1.0]
  jinja version = 3.0.1
  libyaml = True

CONFIGURATION

(ansible-config dump --only-changed reported no changes)

Note that I used an S3-compatible third-party provider for running the module. I am aware that this might not be supported. However, I think that the issue also affects AWS, as the cause seems lie within the module itself (see below).

OS / ENVIRONMENT

Manjaro 21.0.6

STEPS TO REPRODUCE

- name: Create a bucket with a policy and then remove the policy
  hosts: localhost
  connection: local
  become: no
  vars:
    ansible_python_interpreter: "{{ansible_playbook_python}}"
   bucket_name: "my-unique-bucket-name"
  tasks:
    - name: Create a bucket with attached policy
      amazon.aws.s3_bucket:
        name: "{{ bucket_name }}"
        state: present
        policy: "{{ lookup('template', 'bucket-policy-public-access.json.j2') }}"
    - name: Try to remove the bucket's policy by omitting the policy key 
      amazon.aws.s3_bucket:
        name: "{{ bucket_name }}"
        state: present
    - name: Try to remove the bucket's policy by explicitly setting the policy to `null`
      amazon.aws.s3_bucket:
        name: "{{ bucket_name }}"
        state: present
        policy: null

The file bucket-policy-public-access.json.j2 contained

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowPublicRead",
    "Effect": "Allow",
    "Principal": "*",
    "Action": ["s3:GetObject"],
    "Resource": [ "arn:aws:s3:::{{ bucket_name }}/*" ]
  }]
}

EXPECTED RESULTS

I expected the first task to create a bucket with an attached policy. I then expected either of the second and third tasks to remove said policy

ACTUAL RESULTS

The first task created the bucket with attached policy, as expected. Neither of the second and third tasks removed the policy from the bucket. I attached a sanitized output below.

From looking at the module code, I think there might be a bug here:

amazon.aws/plugins/modules/s3_bucket.py

Lines 360 to 366 in 0cdc47b

    
           if policy is not None: 
        
               if isinstance(policy, string_types): 
        
                   policy = json.loads(policy) 
        
               if not policy and current_policy: 
        
                   try: 
        
                       delete_bucket_policy(s3_client, name)

(current main as of submitting this issue). Line 360 ensures that only non-None values of policy will enter line 364. ~~As far as I can tell, the predicate on line 364 always evaluates to False under this condition. Hence, Line 366 will never be reached, which means that bucket policies may not be deleted.~~ (EDIT: this is not true. Setting policy to the string value 'null' will lead to policy deletion. See my other comment below. Maybe this is simply an issue of documentation?)

ansible-playbook [core 2.11.1] 
  config file = /home/user/s3_bucket_example/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.9/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible-playbook
  python version = 3.9.5 (default, May 24 2021, 12:50:35) [GCC 11.1.0]
  jinja version = 3.0.1
  libyaml = True
Using /home/user/s3_bucket_example/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /home/user/s3_bucket_example/inventory as it did not pass its verify_file() method
script declined parsing /home/user/s3_bucket_example/inventory as it did not pass its verify_file() method
auto declined parsing /home/user/s3_bucket_example/inventory as it did not pass its verify_file() method
Set default localhost to localhost
Parsed /home/user/s3_bucket_example/inventory inventory source with ini plugin
Loading collection amazon.aws from /home/user/.ansible/collections/ansible_collections/amazon/aws
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: main.yml *************************************************************
Positional arguments: main.yml
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/home/user/s3_bucket_example/inventory',)
forks: 5
1 plays in main.yml
Read vars_file 'vars/main.yml'
Read vars_file 'vars/main.yml'
Read vars_file 'vars/main.yml'

PLAY [Do some things] **********************************************************
Read vars_file 'vars/main.yml'

TASK [Gathering Facts] *********************************************************
task path: /home/user/s3_bucket_example/main.yml:2
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: user
<localhost> EXEC /bin/sh -c 'echo ~user && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user/.ansible/tmp `"&& mkdir "` echo /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073 `" && echo ansible-tmp-1623491053.5475247-40397-239535370960073="` echo /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073 `" ) && sleep 0'
Using module file /usr/lib/python3.9/site-packages/ansible/modules/setup.py
<localhost> PUT /home/user/.ansible/tmp/ansible-local-40393dgnhyxmv/tmp6mg63skd TO /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073/AnsiballZ_setup.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073/ /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073/AnsiballZ_setup.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073/AnsiballZ_setup.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073/ > /dev/null 2>&1 && sleep 0'
ok: [localhost]
Read vars_file 'vars/main.yml'
META: ran handlers
Read vars_file 'vars/main.yml'

TASK [amazon.aws.s3_bucket] ****************************************************
task path: /home/user/s3_bucket_example/main.yml:11
File lookup using /home/user/s3_bucket_example/templates/bucket-policy-public-access.json.j2 as file
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: user
<localhost> EXEC /bin/sh -c 'echo ~user && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user/.ansible/tmp `"&& mkdir "` echo /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415 `" && echo ansible-tmp-1623491054.2478297-40459-110095007754415="` echo /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415 `" ) && sleep 0'
Using module file /home/user/.ansible/collections/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py
<localhost> PUT /home/user/.ansible/tmp/ansible-local-40393dgnhyxmv/tmpq_ku2_9n TO /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415/AnsiballZ_s3_bucket.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415/ /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415/AnsiballZ_s3_bucket.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415/AnsiballZ_s3_bucket.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => (item={'name': 'my-unique-bucket-name'}) => {
    "ansible_loop_var": "item",
    "changed": true,
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "ceph": false,
            "debug_botocore_endpoint_logs": false,
            "delete_public_access": false,
            "ec2_url": null,
            "encryption": null,
            "encryption_key_id": null,
            "force": false,
            "name": "my-unique-bucket-name",
            "policy": "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Sid\": \"AllowPublicRead\", \"Effect\": \"Allow\", \"Principal\": \"*\", \"Action\": [\"s3:GetObject\"], \"Resource\": [\"arn:aws:s3:::my-unique-bucket-name/*\"]}]}",
            "profile": null,
            "public_access": null,
            "purge_tags": true,
            "region": null,
            "requester_pays": null,
            "s3_url": "https://s3.hidrive.strato.com",
            "security_token": null,
            "state": "absent",
            "tags": null,
            "validate_certs": true,
            "versioning": null
        }
    },
    "item": {
        "name": "my-unique-bucket-name"
    }
}
Read vars_file 'vars/main.yml'

TASK [amazon.aws.s3_bucket] ****************************************************
task path: /home/user/s3_bucket_example/main.yml:19
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: user
<localhost> EXEC /bin/sh -c 'echo ~user && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user/.ansible/tmp `"&& mkdir "` echo /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489 `" && echo ansible-tmp-1623491055.092912-40475-1394715121489="` echo /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489 `" ) && sleep 0'
Using module file /home/user/.ansible/collections/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py
<localhost> PUT /home/user/.ansible/tmp/ansible-local-40393dgnhyxmv/tmp0s83nn1b TO /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489/AnsiballZ_s3_bucket.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489/ /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489/AnsiballZ_s3_bucket.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489/AnsiballZ_s3_bucket.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => (item={'name': 'my-unique-bucket-name'}) => {
    "ansible_loop_var": "item",
    "changed": true,
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "ceph": false,
            "debug_botocore_endpoint_logs": false,
            "delete_public_access": false,
            "ec2_url": null,
            "encryption": null,
            "encryption_key_id": null,
            "force": false,
            "name": "my-unique-bucket-name",
            "policy": null,
            "profile": null,
            "public_access": null,
            "purge_tags": true,
            "region": null,
            "requester_pays": null,
            "s3_url": "https://s3.hidrive.strato.com",
            "security_token": null,
            "state": "present",
            "tags": null,
            "validate_certs": true,
            "versioning": null
        }
    },
    "item": {
        "name": "my-unique-bucket-name"
    },
    "name": "my-unique-bucket-name",
    "policy": null
}
Read vars_file 'vars/main.yml'

TASK [amazon.aws.s3_bucket] ****************************************************
task path: /home/user/s3_bucket_example/main.yml:26
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: user
<localhost> EXEC /bin/sh -c 'echo ~user && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user/.ansible/tmp `"&& mkdir "` echo /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275 `" && echo ansible-tmp-1623491055.923643-40491-193397245992275="` echo /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275 `" ) && sleep 0'
Using module file /home/user/.ansible/collections/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py
<localhost> PUT /home/user/.ansible/tmp/ansible-local-40393dgnhyxmv/tmp8ldwx5gt TO /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275/AnsiballZ_s3_bucket.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275/ /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275/AnsiballZ_s3_bucket.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275/AnsiballZ_s3_bucket.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => (item={'name': 'my-unique-bucket-name'}) => {
    "ansible_loop_var": "item",
    "changed": false,
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "ceph": false,
            "debug_botocore_endpoint_logs": false,
            "delete_public_access": false,
            "ec2_url": null,
            "encryption": null,
            "encryption_key_id": null,
            "force": false,
            "name": "my-unique-bucket-name",
            "policy": null,
            "profile": null,
            "public_access": null,
            "purge_tags": true,
            "region": null,
            "requester_pays": null,
            "s3_url": "https://s3.hidrive.strato.com",
            "security_token": null,
            "state": "present",
            "tags": null,
            "validate_certs": true,
            "versioning": null
        }
    },
    "item": {
        "name": "my-unique-bucket-name"
    },
    "name": "my-unique-bucket-name",
    "policy": null
}
Read vars_file 'vars/main.yml'
META: ran handlers
Read vars_file 'vars/main.yml'
META: ran handlers

PLAY RECAP *********************************************************************
localhost                  : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The text was updated successfully, but these errors were encountered:

ansibullbot · 2021-06-12T09:57:22Z

Files identified in the description:

[plugins/modules/s3_bucket.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/s3_bucket.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot · 2021-06-12T09:57:23Z

cc @abikouo @jillr @s-hertel @tremble @wimnat
click here for bot help

2d6 · 2021-06-12T10:02:27Z

One addendum: I just realized that removing bucket policies works when setting the policy configuration key to 'null', i.e. the string value. Maybe this is the intended functionality, and it is simply a matter of adding a short example to the module documentation? I would be happy to provide one, e.g.

# Ensure no policy is attached to the bucket
        - amazon.aws.s3_bucket:
            name: mys3bucket
            state: present
            policy: 'null'

goneri · 2021-06-15T19:59:41Z

Hi @2d6,

Thank you for taking the time to report this. Would you like to submit a documentation fix?

2d6 · 2021-06-16T17:20:16Z

I just created a small PR to add this case to the documentation :)

mcandre · 2021-08-18T14:10:01Z

Validation rejects setting empty Action / Resource.

Validation rejects setting empty Statement.

Playbook does nothing when the policy field is removed.

Playbook does nothing when the delete_public_access field is configured as documented.

The user is forced to either manually delete the policy and then remove the Ansible policy field; Or else delete the S3 bucket and start from scratch.

s3_bucket - improve documentation of policy parameter SUMMARY This pull requests improves the documentation of the policy parameter in the s3_bucket module. It documents how to ensure the absence of a policy. Fixes #385 ISSUE TYPE Docs Pull Request COMPONENT NAME s3_bucket Reviewed-by: Jill R <None> Reviewed-by: Moritz Wagner <None> Reviewed-by: None <None> Reviewed-by: Mark Chappell <None>

* rds_instance: Add purge_security_groups feature for vpc_security_groups_ids. * Fixes: ansible-collections#385 This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections/community.aws@cd32e65

* rds_instance: Add purge_security_groups feature for vpc_security_groups_ids. * Fixes: ansible-collections#385

ansibullbot added bug This issue/PR relates to a bug module module needs_triage plugins plugin (any type) python3 labels Jun 12, 2021

goneri added waiting_on_contributor Needs help. Feel free to engage to get things unblocked and removed needs_triage labels Jun 15, 2021

2d6 mentioned this issue Jun 16, 2021

s3_bucket - improve documentation of policy parameter #387

Merged

ansibullbot added the has_pr label Jun 16, 2021

ansible-zuul bot closed this as completed in #387 Aug 20, 2021

abikouo pushed a commit to abikouo/amazon.aws that referenced this issue Sep 18, 2023

rds_instance: Add purge_security_groups (ansible-collections#500)

f91a00e

* rds_instance: Add purge_security_groups feature for vpc_security_groups_ids. * Fixes: ansible-collections#385

abikouo pushed a commit to abikouo/amazon.aws that referenced this issue Sep 18, 2023

rds_instance: Add purge_security_groups (ansible-collections#500)

879220d

* rds_instance: Add purge_security_groups feature for vpc_security_groups_ids. * Fixes: ansible-collections#385

abikouo pushed a commit to abikouo/amazon.aws that referenced this issue Oct 24, 2023

rds_instance: Add purge_security_groups (ansible-collections#500)

cd32e65

* rds_instance: Add purge_security_groups feature for vpc_security_groups_ids. * Fixes: ansible-collections#385

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

amazon.aws.s3_bucket: Setting policy to None/null does not remove bucket policy #385

amazon.aws.s3_bucket: Setting policy to None/null does not remove bucket policy #385

2d6 commented Jun 12, 2021 •

edited

Loading

ansibullbot commented Jun 12, 2021

ansibullbot commented Jun 12, 2021

2d6 commented Jun 12, 2021 •

edited

Loading

goneri commented Jun 15, 2021

2d6 commented Jun 16, 2021

mcandre commented Aug 18, 2021

amazon.aws.s3_bucket: Setting policy to None/null does not remove bucket policy #385

amazon.aws.s3_bucket: Setting policy to None/null does not remove bucket policy #385

Comments

2d6 commented Jun 12, 2021 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

ansibullbot commented Jun 12, 2021

ansibullbot commented Jun 12, 2021

2d6 commented Jun 12, 2021 • edited Loading

goneri commented Jun 15, 2021

2d6 commented Jun 16, 2021

mcandre commented Aug 18, 2021

2d6 commented Jun 12, 2021 •

edited

Loading

2d6 commented Jun 12, 2021 •

edited

Loading