S3 behavior changed when KMS encryption requested with no Key ID

[joshriverscambia2019]

Summary

As noted in https://github.com/ansible-collections/amazon.aws/blob/main/plugins/modules/s3_bucket.py#L64 AWS no longer supports disabling encryption for new S3 buckets.

Additionally, if encryption: aws:kms is requested without an explicit encryption_key_id, the bucket will be configured with AES256 encryption. This can cause deployment to fail in wait_encryption_is_applied as the state never aligns with the request.

This is a minor behavior bug, but can be a little tricky to comprehend the failure. It might be worth either;

1. Adding parameter validation to prevent encryption: aws:kms from being specified without encryption_key_id.
2. Updating the encryption variable to force the value to be "AES256" if it is set to aws:kms and no encryption_key_id is provided.

Issue Type

Bug Report

Component Name

amazon.aws.s3_bucket

Ansible Version

$ ansible --version
ansible [core 2.14.4]
config file = /etc/ansible/ansible.cfg
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /root/.local/pipx/venvs/ansible/lib/python3.9/site-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /root/.local/bin/ansible
python version = 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110] (/root/.local/pipx/venvs/ansible/bin/python)
jinja version = 3.1.2
libyaml = True

Collection Versions

$ansible-galaxy collection list

# /root/.local/pipx/venvs/ansible/lib/python3.9/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    5.4.0
ansible.netcommon             4.1.0
ansible.posix                 1.5.1
ansible.utils                 2.9.0
ansible.windows               1.13.0
arista.eos                    6.0.0
awx.awx                       21.14.0
azure.azcollection            1.15.0
check_point.mgmt              4.0.0
chocolatey.chocolatey         1.4.0
cisco.aci                     2.4.0
cisco.asa                     4.0.0
cisco.dnac                    6.6.4
cisco.intersight              1.0.24
cisco.ios                     4.4.0
cisco.iosxr                   4.1.0
cisco.ise                     2.5.12
cisco.meraki                  2.15.1
cisco.mso                     2.2.1
cisco.nso                     1.0.3
cisco.nxos                    4.1.0
cisco.ucs                     1.8.0
cloud.common                  2.1.3
cloudscale_ch.cloud           2.2.4
community.aws                 5.4.0
community.azure               2.0.0
community.ciscosmb            1.0.5
community.crypto              2.11.1
community.digitalocean        1.23.0
community.dns                 2.5.2
community.docker              3.4.3
community.fortios             1.0.0
community.general             6.5.0
community.google              1.0.0
community.grafana             1.5.4
community.hashi_vault         4.2.0
community.hrobot              1.8.0
community.libvirt             1.2.0
community.mongodb             1.5.1
community.mysql               3.6.0
community.network             5.0.0
community.okd                 2.3.0
community.postgresql          2.3.2
community.proxysql            1.5.1
community.rabbitmq            1.2.3
community.routeros            2.8.0
community.sap                 1.0.0
community.sap_libs            1.4.1
community.skydive             1.0.0
community.sops                1.6.1
community.vmware              3.5.0
community.windows             1.12.0
community.zabbix              1.9.2
containers.podman             1.10.1
cyberark.conjur               1.2.0
cyberark.pas                  1.0.17
dellemc.enterprise_sonic      2.0.0
dellemc.openmanage            6.3.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
dellemc.powerflex             1.5.0
dellemc.unity                 1.5.0
f5networks.f5_modules         1.23.0
fortinet.fortimanager         2.1.7
fortinet.fortios              2.2.3
frr.frr                       2.0.0
gluster.gluster               1.0.2
google.cloud                  1.1.3
grafana.grafana               1.1.1
hetzner.hcloud                1.10.0
hpe.nimble                    1.1.4
ibm.qradar                    2.1.0
ibm.spectrum_virtualize       1.11.0
infinidat.infinibox           1.3.12
infoblox.nios_modules         1.4.1
inspur.ispim                  1.3.0
inspur.sm                     2.3.0
junipernetworks.junos         4.1.0
kubernetes.core               2.4.0
lowlydba.sqlserver            1.3.1
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.22.0
netapp.elementsw              21.7.0
netapp.ontap                  22.4.1
netapp.storagegrid            21.11.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.4.0
netbox.netbox                 3.11.0
ngine_io.cloudstack           2.3.0
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.3
openstack.cloud               1.10.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   2.4.1
purestorage.flasharray        1.17.2
purestorage.flashblade        1.10.0
purestorage.fusion            1.4.1
sensu.sensu_go                1.13.2
splunk.es                     2.1.0
t_systems_mms.icinga_director 1.32.2
theforeman.foreman            3.9.0
vmware.vmware_rest            2.3.1
vultr.cloud                   1.7.0
vyos.vyos                     4.0.1
wti.remote                    1.0.4

# /root/.ansible/collections/ansible_collections
Collection           Version
-------------------- -------
community.kubernetes 2.0.1
kubernetes.core      2.4.0

AWS SDK versions

$ pip show boto boto3 botocore

Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: [email protected]
License: MIT
Location: /root/.local/pipx/venvs/ansible/lib/python3.9/site-packages
Requires:
Required-by:
---
Name: boto3
Version: 1.26.118
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /root/.local/pipx/venvs/ansible/lib/python3.9/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.29.118
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /root/.local/pipx/venvs/ansible/lib/python3.9/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed
ANSIBLE_FORCE_COLOR(/etc/ansible/ansible.cfg) = True
CONFIG_FILE() = /etc/ansible/ansible.cfg
DEFAULT_GATHERING(/etc/ansible/ansible.cfg) = implicit
DEFAULT_MANAGED_STR(/etc/ansible/ansible.cfg) = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}
DEFAULT_REMOTE_USER(/etc/ansible/ansible.cfg) = chp-auto
DEFAULT_TIMEOUT(/etc/ansible/ansible.cfg) = 10
HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = False

OS / Environment

Debialn Bullseye

Steps to Reproduce

  amazon.aws.s3_bucket:
    name: my_bucket_name
    encryption: aws:kms

Expected Results

Expected success.

Actual Results

fatal: [localhost]: FAILED! => {"changed": false, "live_encryption": {"SSEAlgorithm": "AES256"}, "msg": "Bucket encryption failed to apply in the expected time", "requested_encryption": {"SSEAlgorithm": "aws:kms"}}

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[hakbailey]

@joshriverscambia2019 it looks to me like if encryption: aws:kms is requested without an explicit encryption_key_id, the bucket is still configured with aws:kms encryption, it just uses the default AWS managed key instead of a customer managed key (see S3 User Guide). I'm not sure what's causing the error you're seeing, but the error message indicates that the requested bucket configuration is {"SSEAlgorithm": "aws:kms"}

[alinabuzachis]

The only thing I can add to what @hakbailey already said is that we can probably slightly increase the number of attempts here

amazon.aws/plugins/modules/s3_bucket.py

Line 819 in 68a36b2

max_retries = 3

The API may be slow which is why it may take longer to update. But this is only a supposition!

[hakbailey]

I think by the time this error is seen we're in the wait_encryption_is_applied() function which has 12 retries. I doubt we really need to increase that, and I suspect something else is going on here. I agree API slowness could be the cause, but not sure without more information. In any case I'm going to close this issue as I don't think we need to make any changes at the moment.

[joshriverscambia2019]

Sorry for the slow reply, as I was out for a few days.

Specifically, this issue was occurring while trying to "update" an existing bucket with kms set up with the account key. Running the module against that existing configuration failed as noted and we needed to re-code our role to allow it to function. I don't believe this is worth spending a lot of effort on, but I wanted to get it reported to improve the overall reliability.

[hakbailey]

@joshriverscambia2019 thank you for reporting the issue, we appreciate it! So just to make sure I understand the scenario: you have an existing bucket that uses kms encryption with a customer-managed key, and when you try to update it to use an AWS-managed key (passing no encryption_key_id), it never completes?

[joshriverscambia2019]

@hakbailey Yes. That is the failure I ran into. Modifying the code to set encryption: AES256 resolved my issue, but it had been working for some time and it seemed to me that behavior changed with the new defaults.

[hakbailey]

@joshriverscambia2019 hmm. I'm still not able to recreate this error. The following worked fine:

tasks:
  - name: Create an s3 bucket with customer managed key
    amazon.aws.s3_bucket:
      name: test-bucket
      state: present
      encryption: aws:kms
      encryption_key_id: <the existing kms key id>

  - name: Update bucket to use AWS managed key
    amazon.aws.s3_bucket:
      name: test-bucket
      state: present
      encryption: aws:kms

Are there other details about the existing bucket that I'm missing?

[joshriverscambia2019]

@hakbailey I can't reproduce it anymore either. Same versions and runtime environment, but I am not getting the "Bucket encryption failed to apply in the expected time" message anymore. My initial failures were spaced over 6 hours before I modified the code to encryption: AES256 which allowed it to work.

Running a test playbook now has the bucket moving between algorithms without a hitch:

% aws s3api get-bucket-encryption --bucket [MYBUCKETNAME]
{
    "ServerSideEncryptionConfiguration": {
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "AES256"
                },
                "BucketKeyEnabled": false
            }
        ]
    }
}

goes to

{
    "ServerSideEncryptionConfiguration": {
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "aws:kms"
                },
                "BucketKeyEnabled": false
            }
        ]
    }
}

and back.

There's some mystery here, but not an interesting one. Looks like things are functioning correctly now and there is no need to fix things in the Ansible module.

[hakbailey]

@joshriverscambia2019 well I'm glad your issue was solved! Thanks for confirming.