s3_object in copy mode cannot change metadata

[andreiboost]

Summary

The documentation suggests that metadata can be set when mode=copy but this doesn't appear to be the case. I cannot get metadata to be changed on the copy whatever I do.

This appears to be because the module doesn't override (or provide the option to override) the MetadataDirective parameter, which defaults to COPY (search for it in the boto3 docs).

I first observed this trying to copy an object while modifying its metadata (not shown here, but the copy always has the same metadata as its source). I suspect the reason is that objects have system defined metadata called Content-Type (visible in the console), which in essence is copied due to the directive, but this is speculation on my part.

Additionally, please consider updating the examples which show metadata as a comma separated string while the documentation lists it as a dict (which works as shown below).

Issue Type

Bug Report

Component Name

s3_object

Ansible Version

$ ansible --version
ansible [core 2.16.3]
  config file = /home/andrei/work/tools/andrei/playbooks/ansible.cfg
  configured module search path = ['/home/andrei/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.11/site-packages/ansible
  ansible collection location = /home/andrei/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.11.7 (main, Jan 29 2024, 16:03:57) [GCC 13.2.1 20230801] (/usr/bin/python)
  jinja version = 3.1.3
  libyaml = True

Collection Versions

$ ansible-galaxy collection list

# /home/andrei/.ansible/collections/ansible_collections
Collection                               Version
---------------------------------------- -------
amazon.aws                               7.2.0  
ansible.netcommon                        4.1.0  
ansible.posix                            1.5.1  
ansible.utils                            3.0.0  
cloud.common                             2.1.3  
community.aws                            7.1.0  
community.crypto                         2.10.0 
community.docker                         3.4.0  
community.general                        7.4.0  
community.postgresql                     3.3.0  
community.routeros                       2.5.0  
kubernetes.core                          2.4.0  

# /usr/lib/python3.11/site-packages/ansible_collections
Collection                               Version
---------------------------------------- -------
amazon.aws                               7.2.0  
ansible.netcommon                        5.3.0  
ansible.posix                            1.5.4  
ansible.utils                            2.12.0 
ansible.windows                          2.2.0  
arista.eos                               6.2.2  
awx.awx                                  23.6.0 
azure.azcollection                       1.19.0 
check_point.mgmt                         5.2.2  
chocolatey.chocolatey                    1.5.1  
cisco.aci                                2.8.0  
cisco.asa                                4.0.3  
cisco.dnac                               6.10.2 
cisco.intersight                         2.0.7  
cisco.ios                                5.3.0  
cisco.iosxr                              6.1.1  
cisco.ise                                2.7.0  
cisco.meraki                             2.17.2 
cisco.mso                                2.5.0  
cisco.nxos                               5.3.0  
cisco.ucs                                1.10.0 
cloud.common                             2.1.4  
cloudscale_ch.cloud                      2.3.1  
community.aws                            7.1.0  
community.azure                          2.0.0  
community.ciscosmb                       1.0.7  
community.crypto                         2.17.1 
community.digitalocean                   1.26.0 
community.dns                            2.8.0  
community.docker                         3.7.0  
community.general                        8.3.0  
community.grafana                        1.7.0  
community.hashi_vault                    6.1.0  
community.hrobot                         1.9.0  
community.library_inventory_filtering_v1 1.0.0  
community.libvirt                        1.3.0  
community.mongodb                        1.6.3  
community.mysql                          3.8.0  
community.network                        5.0.2  
community.okd                            2.3.0  
community.postgresql                     3.3.0  
community.proxysql                       1.5.1  
community.rabbitmq                       1.2.3  
community.routeros                       2.12.0 
community.sap                            2.0.0  
community.sap_libs                       1.4.2  
community.sops                           1.6.7  
community.vmware                         4.1.0  
community.windows                        2.1.0  
community.zabbix                         2.3.1  
containers.podman                        1.11.0 
cyberark.conjur                          1.2.2  
cyberark.pas                             1.0.25 
dellemc.enterprise_sonic                 2.4.0  
dellemc.openmanage                       8.7.0  
dellemc.powerflex                        2.1.0  
dellemc.unity                            1.7.1  
f5networks.f5_modules                    1.27.1 
fortinet.fortimanager                    2.3.1  
fortinet.fortios                         2.3.4  
frr.frr                                  2.0.2  
gluster.gluster                          1.0.2  
google.cloud                             1.3.0  
grafana.grafana                          2.2.4  
hetzner.hcloud                           2.4.1  
hpe.nimble                               1.1.4  
ibm.qradar                               2.1.0  
ibm.spectrum_virtualize                  2.0.0  
ibm.storage_virtualize                   2.2.0  
infinidat.infinibox                      1.3.12 
infoblox.nios_modules                    1.6.1  
inspur.ispim                             2.2.0  
inspur.sm                                2.3.0  
junipernetworks.junos                    5.3.1  
kubernetes.core                          2.4.0  
lowlydba.sqlserver                       2.2.2  
microsoft.ad                             1.4.1  
netapp.aws                               21.7.1 
netapp.azure                             21.10.1
netapp.cloudmanager                      21.22.1
netapp.elementsw                         21.7.0 
netapp.ontap                             22.9.0 
netapp.storagegrid                       21.11.1
netapp.um_info                           21.8.1 
netapp_eseries.santricity                1.4.0  
netbox.netbox                            3.16.0 
ngine_io.cloudstack                      2.3.0  
ngine_io.exoscale                        1.1.0  
openstack.cloud                          2.2.0  
openvswitch.openvswitch                  2.1.1  
ovirt.ovirt                              3.2.0  
purestorage.flasharray                   1.26.0 
purestorage.flashblade                   1.15.0 
purestorage.fusion                       1.6.0  
sensu.sensu_go                           1.14.0 
splunk.es                                2.1.2  
t_systems_mms.icinga_director            2.0.1  
telekom_mms.icinga_director              1.35.0 
theforeman.foreman                       3.15.0 
vmware.vmware_rest                       2.3.1  
vultr.cloud                              1.12.1 
vyos.vyos                                4.1.0  
wti.remote                               1.0.5

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.34.34
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/lib/python3.11/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.34.34
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/lib/python3.11/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed
CONFIG_FILE() = /home/andrei/work/tools/andrei/playbooks/ansible.cfg
DEFAULT_HOST_LIST(/home/andrei/work/tools/andrei/playbooks/ansible.cfg) = ...
DEFAULT_ROLES_PATH(/home/andrei/work/tools/andrei/playbooks/ansible.cfg) = ...
EDITOR(env: EDITOR) = vim
INVENTORY_ENABLED(/home/andrei/work/tools/andrei/playbooks/ansible.cfg) = ['ini', 'constructed', 'script']
PAGER(env: PAGER) = less

OS / Environment

Arch Linux

Steps to Reproduce

- name: Test playbook
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Create test bucket
      amazon.aws.s3_bucket:
        name: ansiblemetatest
        state: present

    - name: Create test object
      amazon.aws.s3_object:
        bucket: ansiblemetatest
        object: nometa
        mode: put
        content: "some content"

    - name: Copy and add metadata
      amazon.aws.s3_object:
        bucket: ansiblemetatest
        object: "metacopy"
        mode: copy
        copy_src:
          bucket: ansiblemetatest
          object: nometa
        metadata:
          something: exists
          version: "1.0.2"

    - name: Create test object with metadata
      amazon.aws.s3_object:
        bucket: ansiblemetatest
        object: withmeta
        mode: put
        content: "some content"
        metadata:
          something: exists
          version: "1.0.2"

    - name: Get object info
      amazon.aws.s3_object_info:
        bucket_name: ansiblemetatest
        object_name: "{{ item }}"
      loop:
        - nometa
        - metacopy
        - withmeta
      register: obj_info

    - debug:
        var: obj_info.results

Expected Results

metacopy should have its metadata set, like withmeta does.

Actual Results

PLAY [Test playbook] **********************************************************************************************************

TASK [Create test bucket] *****************************************************************************************************
changed: [localhost]

TASK [Create test object] *****************************************************************************************************
changed: [localhost]

TASK [Copy and add metadata] **************************************************************************************************
[WARNING]: PutObjectAcl operation : The bucket does not allow ACLs.
changed: [localhost]

TASK [Create test object with metadata] ***************************************************************************************
changed: [localhost]

TASK [Get object info] ********************************************************************************************************
ok: [localhost] => (item=nometa)
ok: [localhost] => (item=metacopy)
ok: [localhost] => (item=withmeta)

TASK [debug] ******************************************************************************************************************
ok: [localhost] => {
    "obj_info.results": [
        {
            "ansible_loop_var": "item",
            "changed": false,
            "failed": false,
            "invocation": {
                "module_args": {
                    "access_key": null,
                    "aws_ca_bundle": null,
                    "aws_config": null,
                    "bucket_name": "ansiblemetatest",
                    "ceph": false,
                    "debug_botocore_endpoint_logs": false,
                    "dualstack": false,
                    "endpoint_url": null,
                    "object_details": null,
                    "object_name": "nometa",
                    "profile": "sandbox",
                    "region": null,
                    "secret_key": null,
                    "session_token": null,
                    "validate_certs": true
                }
            },
            "item": "nometa",
            "object_info": [
                {
                    "object_data": {
                        "accept_ranges": "bytes",
                        "content_length": 12,
                        "content_type": "binary/octet-stream",
                        "e_tag": "\"9893532233caff98cd083a116b013c0b\"",
                        "last_modified": "2024-02-23T07:51:24+00:00",
                        "metadata": {},
                        "server_side_encryption": "AES256"
                    }
                }
            ]
        },
        {
            "ansible_loop_var": "item",
            "changed": false,
            "failed": false,
            "invocation": {
                "module_args": {
                    "access_key": null,
                    "aws_ca_bundle": null,
                    "aws_config": null,
                    "bucket_name": "ansiblemetatest",
                    "ceph": false,
                    "debug_botocore_endpoint_logs": false,
                    "dualstack": false,
                    "endpoint_url": null,
                    "object_details": null,
                    "object_name": "metacopy",
                    "profile": "sandbox",
                    "region": null,
                    "secret_key": null,
                    "session_token": null,
                    "validate_certs": true
                }
            },
            "item": "metacopy",
            "object_info": [
                {
                    "object_data": {
                        "accept_ranges": "bytes",
                        "content_length": 12,
                        "content_type": "binary/octet-stream",
                        "e_tag": "\"9893532233caff98cd083a116b013c0b\"",
                        "last_modified": "2024-02-23T07:51:26+00:00",
                        "metadata": {},
                        "server_side_encryption": "AES256"
                    }
                }
            ]
        },
        {
            "ansible_loop_var": "item",
            "changed": false,
            "failed": false,
            "invocation": {
                "module_args": {
                    "access_key": null,
                    "aws_ca_bundle": null,
                    "aws_config": null,
                    "bucket_name": "ansiblemetatest",
                    "ceph": false,
                    "debug_botocore_endpoint_logs": false,
                    "dualstack": false,
                    "endpoint_url": null,
                    "object_details": null,
                    "object_name": "withmeta",
                    "profile": "sandbox",
                    "region": null,
                    "secret_key": null,
                    "session_token": null,
                    "validate_certs": true
                }
            },
            "item": "withmeta",
            "object_info": [
                {
                    "object_data": {
                        "accept_ranges": "bytes",
                        "content_length": 12,
                        "content_type": "binary/octet-stream",
                        "e_tag": "\"9893532233caff98cd083a116b013c0b\"",
                        "last_modified": "2024-02-23T07:51:28+00:00",
                        "metadata": {
                            "something": "exists",
                            "version": "1.0.2"
                        },
                        "server_side_encryption": "AES256"
                    }
                }
            ]
        }
    ]
}

PLAY RECAP ********************************************************************************************************************
localhost                  : ok=6    changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Code of Conduct

• I agree to follow the Ansible Code of Conduct