ec2_launch_template doesn't process arguments correctly #230

niclonious · 2020-09-16T15:59:38Z

SUMMARY

ec2_launch_template module passes unset parameters as "None" to the boto3 calls instead of not setting them at all.

ISSUE TYPE

Bug Report

COMPONENT NAME

ec2_launch_template.py

ANSIBLE VERSION

ansible 2.10.0

CONFIGURATION

ANSIBLE_FORCE_COLOR(env: ANSIBLE_FORCE_COLOR) = True
DEFAULT_FORKS(env: ANSIBLE_FORKS) = 25
DEFAULT_HOST_LIST(env: ANSIBLE_INVENTORY) = ['/usr/local/ansible/inventory/x.aws_ec2.yaml']
DEFAULT_STRATEGY(env: ANSIBLE_STRATEGY) = linear
DEPRECATION_WARNINGS(env: ANSIBLE_DEPRECATION_WARNINGS) = False
HOST_KEY_CHECKING(env: ANSIBLE_HOST_KEY_CHECKING) = False
INTERPRETER_PYTHON(env: ANSIBLE_PYTHON_INTERPRETER) = auto_legacy_silent
RETRY_FILES_ENABLED(env: ANSIBLE_RETRY_FILES_ENABLED) = True

OS / ENVIRONMENT

Ansible is launched from debian:buster-slim docker image

STEPS TO REPRODUCE

Run this task

  - ec2_launch_template:
      name: "some-name"
      image_id: "ami-0c94855ba95c71c99"
      instance_type: "t3.micro"
      monitoring:
        enabled: yes
      network_interfaces:
        - device_index: 0
          associate_public_ip_address: false
          delete_on_termination: true
          groups: "{{ list_of_security_groups }}"
      state: present

EXPECTED RESULTS

Launch template gets created

ACTUAL RESULTS

Creation fails with the next error:

"Traceback (most recent call last):",
"  File \"/tmp/ansible_ec2_launch_template_payload_vx3l42ws/ansible_ec2_launch_template_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_launch_template.py\", line 495, in create_or_update",
"  File \"/tmp/ansible_ec2_launch_template_payload_vx3l42ws/ansible_ec2_launch_template_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/core.py\", line 294, in deciding_wrapper",
"    return retrying_wrapper(*args, **kwargs)",
"  File \"/tmp/ansible_ec2_launch_template_payload_vx3l42ws/ansible_ec2_launch_template_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py\", line 156, in retry_func",
"    raise e",
"  File \"/tmp/ansible_ec2_launch_template_payload_vx3l42ws/ansible_ec2_launch_template_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py\", line 143, in retry_func",
"    return f(*args, **kwargs)",
"  File \"/usr/local/python/python-3.7.0/lib/python3.7/site-packages/botocore/client.py\", line 316, in _api_call",
"    return self._make_api_call(operation_name, kwargs)",
"  File \"/usr/local/python/python-3.7.0/lib/python3.7/site-packages/botocore/client.py\", line 608, in _make_api_call",
"    api_params, operation_model, context=request_context)",
"  File \"/usr/local/python/python-3.7.0/lib/python3.7/site-packages/botocore/client.py\", line 656, in _convert_to_request_dict",
"    api_params, operation_model)",
"  File \"/usr/local/python/python-3.7.0/lib/python3.7/site-packages/botocore/validate.py\", line 297, in serialize_to_request",
"    raise ParamValidationError(report=report.generate_report())",
"botocore.exceptions.ParamValidationError: Parameter validation failed:",
"Invalid type for parameter LaunchTemplateData.NetworkInterfaces[0].Description, value: None, type: <class 'NoneType'>, valid types: <class 'str'>",
"Invalid type for parameter LaunchTemplateData.NetworkInterfaces[0].DeviceIndex, value: None, type: <class 'NoneType'>, valid types: <class 'int'>",
"Invalid type for parameter LaunchTemplateData.NetworkInterfaces[0].Ipv6AddressCount, value: None, type: <class 'NoneType'>, valid types: <class 'int'>",
"Invalid type for parameter LaunchTemplateData.NetworkInterfaces[0].Ipv6Addresses, value: None, type: <class 'NoneType'>, valid types: <class 'list'>, <class 'tuple'>",
"Invalid type for parameter LaunchTemplateData.NetworkInterfaces[0].NetworkInterfaceId, value: None, type: <class 'NoneType'>, valid types: <class 'str'>",
"Invalid type for parameter LaunchTemplateData.NetworkInterfaces[0].PrivateIpAddress, value: None, type: <class 'NoneType'>, valid types: <class 'str'>",
"Invalid type for parameter LaunchTemplateData.NetworkInterfaces[0].SubnetId, value: None, type: <class 'NoneType'>, valid types: <class 'str'>"

Additionaly, if I set Ipv6AddressCount to 0, it still tells about Invalid type of Ipv6Addresses parameter, considering that those two are mutually excluding.

The text was updated successfully, but these errors were encountered:

ansibullbot · 2020-09-18T18:36:22Z

Files identified in the description:

plugins/modules/ec2_launch_template.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot · 2020-09-18T18:36:23Z

cc @jillr @s-hertel @tremble @wimnat
click here for bot help

jaudriga · 2020-09-24T09:59:22Z

I am having related issue with the block_device_mappings. The following worked in ansible 2.9:

We specify:

- name: Create new version of launch template
  community.aws.ec2_launch_template:
...
    block_device_mappings:
      - device_name: "/dev/sda1"
        ebs:
          delete_on_termination: True
          encrypted: False
          volume_size: 20
          volume_type: "gp2"

However, it fails in 2.10.0 (ansible-base 2.10.0 - 2.10.2) with:

TASK [aws_update_ec2_asg : Create new version of launch template] ************************************************************************************************************************************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Invalid type for parameter LaunchTemplateData.BlockDeviceMappings[0].VirtualName, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
fatal: [localhost]: FAILED! => {"boto3_version": "1.14.54", "botocore_version": "1.17.54", "changed": false, "msg": "Couldn't create subsequent launch template version: Parameter validation failed:\nInvalid type for parameter LaunchTemplateData.BlockDeviceMappings[0].Ebs.Iops, value: None, type: <class 'NoneType'>, valid types: <class 'int'>\nInvalid type for parameter LaunchTemplateData.BlockDeviceMappings[0].Ebs.KmsKeyId, value: None, type: <class 'NoneType'>, valid types: <class 'str'>\nInvalid type for parameter LaunchTemplateData.BlockDeviceMappings[0].Ebs.SnapshotId, value: None, type: <class 'NoneType'>, valid types: <class 'str'>\nInvalid type for parameter LaunchTemplateData.BlockDeviceMappings[0].NoDevice, value: None, type: <class 'NoneType'>, valid types: <class 'str'>\nInvalid type for parameter LaunchTemplateData.BlockDeviceMappings[0].VirtualName, value: None, type: <class 'NoneType'>, valid types: <class 'str'>"}

Edit: I am also experiencing the same error on ansible 2.9.13 with community.aws:1.2.0 .

markuman · 2020-10-30T13:58:15Z

#251

markuman · 2020-10-30T13:59:08Z

ansible-collections/amazon.aws#184

kitsushadow · 2021-01-04T17:40:13Z

I am also having an issue with this module.
Specifically with the block_device_mappings. No matter what format its put in

    name: "ansible_development"
    default_version: 1
    block_device_mappings:
      device_name: "/dev/sdc"
      ebs:
        delete_on_termination: yes
        iops: 3000
        snapshot_id: "{{ testint_newest_snapshot[0].SnapshotId }}"
        volume_type: gp3

"msg": "argument block_device_mappings is of type <type 'dict'> and we were unable to convert to list: <type 'dict'> cannot be converted to a list"

    "_ansible_no_log": false,
    "warnings": [
        "The value 1 (type int) in a string field was converted to u'1' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change."
    ],
    "invocation": {
        "module_args": {
            "default_version": "1",
            "state": "present",
            "name": "ansible_development",
            "template_name": "ansible_development",
            "validate_certs": true,
            "debug_botocore_endpoint_logs": false,
            "block_device_mappings": {
                "ebs": {
                    "iops": 3000,
                    "delete_on_termination": true,
                    "volume_type": "gp3",
                    "snapshot_id": "snap-07116ff7006d7beb0"
                },
                "device_name": "/dev/sdc"
            }
        }
    },
    "changed": false,
    "msg": "argument block_device_mappings is of type <type 'dict'> and we were unable to convert to list: <type 'dict'> cannot be converted to a list"
}```

kepstin · 2021-01-27T22:03:29Z

The existing scrub_none_parameters added in ansible-collections/amazon.aws#184 won't help here, because the block_device_mappings parameter actually takes a list of dicts, rather than a dict.

The current implementation of the scrub helper function doesn't recursively apply to lists of dicts.

adding another conditional branch to scrub_none_parameters like this:

    elif isinstance(v, list):
        clean_parameters[k] = [scrub_none_parameters(vv) if isinstance(vv, dict) else vv for vv in v]

and then switching the cleanup code in params_to_launch_data to use this function would do the trick for fixing this issue.

markuman · 2021-02-22T06:14:05Z

#413

stefanhorning · 2021-02-22T17:24:34Z

Have the same issues with params that should be optional or not even allowed in combination.

For example
LaunchTemplateData.InstanceMarketOptions.SpotOptions.BlockDurationMinutes should be optional as I don't want my instance to stop running when I define it a spot instance

Also for volumes not all params should be mandatory, like

Invalid type for parameter LaunchTemplateData.BlockDeviceMappings[1].Ebs.Iops, value: None, type: <class 'NoneType'>, valid types: <class 'int'>
Invalid type for parameter LaunchTemplateData.BlockDeviceMappings[1].Ebs.KmsKeyId, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
Invalid type for parameter LaunchTemplateData.BlockDeviceMappings[1].Ebs.SnapshotId, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
Invalid type for parameter LaunchTemplateData.BlockDeviceMappings[1].NoDevice, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
Invalid type for parameter LaunchTemplateData.BlockDeviceMappings[1].VirtualName, value: None, type: <class 'NoneType'>, valid types: <class 'str'>"}

gp2 volumes have no iops param
KmsKeyId is not needed when enrypting with default AWS key
SnapshotId (not sure why needed when just defining a volume)
not even shure what the last two params do, but they are usually not needed in other modules or AWS tooling (unless you do s.th exotic maybe)

block device section defined in play:

[...]
        block_device_mappings:
          - device_name: /dev/sda1
            ebs:
              volume_type: gp2
              volume_size: 20
              delete_on_termination: true
              encrypted: true
          - device_name: /dev/sdb
            ebs:
              volume_type: gp2
              volume_size: 200
              delete_on_termination: true
              encrypted: true
[...]

Using Ansible 2.10.5

kitsushadow · 2021-02-22T20:00:10Z

It would be ideal if params that didn't need to be changed could be left blank AND a flag was set to inherit or not inherit the previously defined version template. Any configs provided by the user would overwrite the default.

…

On Mon, Feb 22, 2021 at 12:24 PM Stefan Horning ***@***.***> wrote: Have the same issues with params that should be optional or not even allowed in combination. For example LaunchTemplateData.InstanceMarketOptions.SpotOptions.BlockDurationMinutes should be optional as I don't want my instance to stop running when I define it a spot instance Also for volumes not all params should be mandatory, like Invalid type for parameter LaunchTemplateData.BlockDeviceMappings[1].Ebs.Iops, value: None, type: <class 'NoneType'>, valid types: <class 'int'> Invalid type for parameter LaunchTemplateData.BlockDeviceMappings[1].Ebs.KmsKeyId, value: None, type: <class 'NoneType'>, valid types: <class 'str'> Invalid type for parameter LaunchTemplateData.BlockDeviceMappings[1].Ebs.SnapshotId, value: None, type: <class 'NoneType'>, valid types: <class 'str'> Invalid type for parameter LaunchTemplateData.BlockDeviceMappings[1].NoDevice, value: None, type: <class 'NoneType'>, valid types: <class 'str'> Invalid type for parameter LaunchTemplateData.BlockDeviceMappings[1].VirtualName, value: None, type: <class 'NoneType'>, valid types: <class 'str'>"} - gp2 volumes have no iops param - KmsKeyId is not needed when enrypting with default AWS key - SnapshotId (not sure why needed when just defining a volume) - not even shure what the last two params do, but they are usually not needed in other modules or AWS tooling (unless you do s.th exotic maybe) block device section defined in play: [...] block_device_mappings: - device_name: /dev/sda1 ebs: volume_type: gp2 volume_size: 20 delete_on_termination: true encrypted: true - device_name: /dev/sdb ebs: volume_type: gp2 volume_size: 200 delete_on_termination: true encrypted: true[...] — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#230 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABI2ZYAKWEYB47JAOJS62ILTAKHOFANCNFSM4RPBR5BA> .

stefanhorning · 2021-02-23T12:22:03Z

Also the fix in #413 didn't work for me. As far I can see the None values are already excluded in https://github.com/ansible-collections/community.aws/blob/main/plugins/modules/ec2_launch_template.py#L441 anyhow. Probably None values need to be cleaned on all levels still.

ansible-collections#230

stefanhorning · 2021-02-23T12:39:18Z

Added a fix here #438 which I tested with nested params which all worked fine: #438

tremble · 2021-03-16T07:54:02Z

This issue should be fixed by #438 which was merged this morning.

ec2_launch_template doesn't process arguments correctly

ansibullbot added affects_2.10 bug This issue/PR relates to a bug module module needs_triage plugins plugin (any type) python3 traceback labels Sep 18, 2020

kepstin mentioned this issue Jan 27, 2021

elements='dict' set suboptions to None values #251

Closed

gravesm removed the needs_triage label Feb 12, 2021

stefanhorning added a commit to mediafellows/community.aws that referenced this issue Feb 23, 2021

Fix ec2_launch_template to correctly remove all None types, fixes issue

e3c1828

ansible-collections#230

stefanhorning mentioned this issue Feb 23, 2021

Fix ec2_launch_template to correctly remove all None types #438

Closed

ansibullbot added the has_pr label Feb 23, 2021

tremble closed this as completed Mar 16, 2021

falsedlah added a commit to falsedlah/community.aws that referenced this issue Sep 20, 2021

ansible-collections#230 :

ce77b4f

ec2_launch_template doesn't process arguments correctly

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ec2_launch_template doesn't process arguments correctly #230

ec2_launch_template doesn't process arguments correctly #230

niclonious commented Sep 16, 2020

ansibullbot commented Sep 18, 2020

ansibullbot commented Sep 18, 2020

jaudriga commented Sep 24, 2020 •

edited

Loading

markuman commented Oct 30, 2020

markuman commented Oct 30, 2020

kitsushadow commented Jan 4, 2021 •

edited

Loading

kepstin commented Jan 27, 2021 •

edited

Loading

markuman commented Feb 22, 2021

stefanhorning commented Feb 22, 2021 •

edited

Loading

kitsushadow commented Feb 22, 2021 via email

stefanhorning commented Feb 23, 2021

stefanhorning commented Feb 23, 2021

tremble commented Mar 16, 2021

ec2_launch_template doesn't process arguments correctly #230

ec2_launch_template doesn't process arguments correctly #230

Comments

niclonious commented Sep 16, 2020

SUMMARY

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

ansibullbot commented Sep 18, 2020

ansibullbot commented Sep 18, 2020

jaudriga commented Sep 24, 2020 • edited Loading

markuman commented Oct 30, 2020

markuman commented Oct 30, 2020

kitsushadow commented Jan 4, 2021 • edited Loading

kepstin commented Jan 27, 2021 • edited Loading

markuman commented Feb 22, 2021

stefanhorning commented Feb 22, 2021 • edited Loading

kitsushadow commented Feb 22, 2021 via email

stefanhorning commented Feb 23, 2021

stefanhorning commented Feb 23, 2021

tremble commented Mar 16, 2021

jaudriga commented Sep 24, 2020 •

edited

Loading

kitsushadow commented Jan 4, 2021 •

edited

Loading

kepstin commented Jan 27, 2021 •

edited

Loading

stefanhorning commented Feb 22, 2021 •

edited

Loading