ec2 inventory lost instances if customize dns is set via dhcp options #710

link89 · 2022-03-15T06:42:55Z

Summary

When I use the default dhpc that assign to my VPC, and use aws_ec2 inventory, everything works fine. But after I create a customize dhcp options that use dns servers (like 8.8.8.8)and set my VPC to use it, the output of the same inventory script will become empty.

Issue Type

Bug Report

Component Name

aws_ec2 inventory

Ansible Version

$ ansible --version
ansible [core 2.12.3]
  config file = None
  configured module search path = ['/Users/xxxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible
  ansible collection location = /Users/xxxx/.ansible/collections:/usr/share/ansible/collections
  executable location = /Library/Frameworks/Python.framework/Versions/3.8/bin/ansible
  python version = 3.8.7 (v3.8.7:6503f05dd5, Dec 21 2020, 12:45:15) [Clang 6.0 (clang-600.0.57)]
  jinja version = 3.0.1
  libyaml = False

Collection Versions

$ ansible-galaxy collection list
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0
ansible.netcommon             2.5.1
ansible.posix                 1.3.0
ansible.utils                 2.5.0
ansible.windows               1.9.0
arista.eos                    3.1.0
awx.awx                       19.4.0
azure.azcollection            1.11.0
check_point.mgmt              2.2.2
chocolatey.chocolatey         1.2.0
cisco.aci                     2.1.0
cisco.asa                     2.1.0
cisco.intersight              1.0.18
cisco.ios                     2.7.1
cisco.iosxr                   2.7.0
cisco.ise                     1.2.1
cisco.meraki                  2.6.0
cisco.mso                     1.3.0
cisco.nso                     1.0.3
cisco.nxos                    2.9.0
cisco.ucs                     1.6.0
cloud.common                  2.1.0
cloudscale_ch.cloud           2.2.0
community.aws                 2.3.0
community.azure               1.1.0
community.ciscosmb            1.0.4
community.crypto              2.2.2
community.digitalocean        1.15.1
community.dns                 2.0.7
community.docker              2.2.0
community.fortios             1.0.0
community.general             4.5.0
community.google              1.0.0
community.grafana             1.3.2
community.hashi_vault         2.3.0
community.hrobot              1.2.2
community.kubernetes          2.0.1
community.kubevirt            1.0.0
community.libvirt             1.0.2
community.mongodb             1.3.2
community.mysql               2.3.4
community.network             3.0.0
community.okd                 2.1.0
community.postgresql          1.7.0
community.proxysql            1.3.1
community.rabbitmq            1.1.0
community.routeros            2.0.0
community.skydive             1.0.0
community.sops                1.2.0
community.vmware              1.17.1
community.windows             1.9.0
community.zabbix              1.5.1
containers.podman             1.9.1
cyberark.conjur               1.1.0
cyberark.pas                  1.0.13
dellemc.enterprise_sonic      1.1.0
dellemc.openmanage            4.4.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
f5networks.f5_modules         1.14.0
fortinet.fortimanager         2.1.4
fortinet.fortios              2.1.4
frr.frr                       1.0.3
gluster.gluster               1.0.2
google.cloud                  1.0.2
hetzner.hcloud                1.6.0
hpe.nimble                    1.1.4
ibm.qradar                    1.0.3
infinidat.infinibox           1.3.3
infoblox.nios_modules         1.2.1
inspur.sm                     1.3.0
junipernetworks.junos         2.9.0
kubernetes.core               2.2.3
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.14.0
netapp.elementsw              21.7.0
netapp.ontap                  21.16.0
netapp.storagegrid            21.9.0
netapp.um_info                21.8.0
netapp_eseries.santricity     1.2.13
netbox.netbox                 3.5.1
ngine_io.cloudstack           2.2.3
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.0
openstack.cloud               1.7.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   1.6.6
purestorage.flasharray        1.12.1
purestorage.flashblade        1.9.0
sensu.sensu_go                1.13.0
servicenow.servicenow         1.0.6
splunk.es                     1.0.2
t_systems_mms.icinga_director 1.27.1
theforeman.foreman            2.2.0
vyos.vyos                     2.7.0
wti.remote                    1.0.3

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.21.19
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.24.19
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed

OS / Environment

MacOS, CentOS

Steps to Reproduce

plugin: aws_ec2
regions:
  - eu-central-1
keyed_groups:
 - key: tags
   prefix: tag
 - prefix: instance_type
   key: instance_type
 - key: placement.region
   prefix: aws_region
compose:
  ansible_host: private_ip_address

And run with
ansible-inventory -vvvv --graph -i ansible/aws_ec2.yml

Expected Results

List all instances.

Actual Results

@all:
plugin: aws_ec2
  |--@aws_ec2:
  |--@ungrouped:

Code of Conduct

I agree to follow the Ansible Code of Conduct

The text was updated successfully, but these errors were encountered:

ansibullbot · 2022-03-15T06:44:29Z

Files identified in the description:

[plugins/inventory/aws_ec2.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/inventory/aws_ec2.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

abikouo · 2022-03-28T14:39:21Z

@link89 Thanks for reporting this issue.
I tried to reproduce using this playbook following all steps provided above, but could not

- hosts: localhost
  gather_facts: no

  vars:
    ec2_ami_name: 'Fedora-Cloud-Base-*.x86_64*'

  module_defaults:
    group/aws:
      aws_access_key: "{{ aws_access_key }}"
      aws_secret_key: "{{ aws_secret_key }}"
      security_token: "{{ security_token | default(omit) }}"
      region: "{{ aws_region }}"
  
  collections:
    - community.aws
    - amazon.aws
  
  tasks:
    - debug:
        msg: "ec2 hosts: {{ groups.aws_ec2 }}"

    - name: get image ID to create an instance
      ec2_ami_info:
        filters:
          architecture: x86_64
          owner-id: '{{ aws_owner_id }}'
          virtualization-type: hvm
          root-device-type: ebs
          name: 'Fedora-Atomic-27*'
      register: _images
      no_log: true

    - set_fact:
        image_id: '{{ _images.images.0.image_id }}'
        vpc_cidr: '10.{{ 256 | random(seed=resource_prefix) }}.0.0/16'
        subnet_cidr: '10.{{ 256 | random(seed=resource_prefix) }}.0.0/24'

    - name: create a VPC to work in
      ec2_vpc_net:
        cidr_block: '{{ vpc_cidr }}'
        name: '{{ resource_prefix }}-vpc'
        resource_tags:
          Name: '{{ resource_prefix }}'
      register: _vpc
    
    - set_fact:
        vpc_id: '{{ _vpc.vpc.id }}'

    - name: create a subnet to use for creating an ec2 instance
      ec2_vpc_subnet:
        vpc_id: '{{ vpc_id }}'
        cidr: '{{ subnet_cidr }}'
        resource_tags:
          Name: '{{ resource_prefix }}'
      register: _subnet

    - set_fact:
        subnet_id: '{{ _subnet.subnet.id }}'

    - name: create a security group to use for creating an ec2 instance
      ec2_group:
        name: '{{ resource_prefix }}-sg'
        vpc_id: '{{ vpc_id }}'
        description: "create security group for testing"
      register: _secgroup

    - name: create a new host
      ec2:
        image: '{{ image_id }}'
        count_tag:
          Name: '{{ resource_prefix }}'
        instance_tags:
          Name: '{{ resource_prefix }}'
        instance_type: t2.micro
        wait: yes
        group_id: '{{ _secgroup.group_id }}'
        vpc_subnet_id: '{{ subnet_id }}'

    - meta: refresh_inventory

    - debug:
        msg: "ec2 hosts: {{ groups.aws_ec2 }}"

    - name: Create custom DHCP options
      ec2_vpc_dhcp_option:
        region: "{{ aws_region }}"
        dns_servers:
          - 8.8.8.8
        tags:
          Name: custom servers
          Environment: Test
      register: _dhcpoptions

    - name: update VPC with DHCP options
      ec2_vpc_net:
        cidr_block: '{{ vpc_cidr }}'
        name: '{{ resource_prefix }}-vpc'
        resource_tags:
          Name: '{{ resource_prefix }}'
        dhcp_opts_id: "{{ _dhcpoptions.dhcp_options_id }}"
      register: _vpc

    - meta: refresh_inventory

    - debug:
        msg: "ec2 hosts: {{ groups.aws_ec2 }}"

Here after the output

PLAY [localhost] ************************************************************************************************************************************************************

TASK [debug] ****************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "ec2 hosts: []"
}

TASK [get image ID to create an instance] ***********************************************************************************************************************************
ok: [localhost]

TASK [set_fact] *************************************************************************************************************************************************************
ok: [localhost]

TASK [create a VPC to work in] **********************************************************************************************************************************************
changed: [localhost]

TASK [set_fact] *************************************************************************************************************************************************************
ok: [localhost]

TASK [create a subnet to use for creating an ec2 instance] ******************************************************************************************************************
changed: [localhost]

TASK [set_fact] *************************************************************************************************************************************************************
ok: [localhost]

TASK [create a security group to use for creating an ec2 instance] **********************************************************************************************************
changed: [localhost]

TASK [create a new host] ****************************************************************************************************************************************************
changed: [localhost]

TASK [meta] *****************************************************************************************************************************************************************

TASK [debug] ****************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "ec2 hosts: ['ip-10-3-0-182.ec2.internal']"
}

TASK [Create custom DHCP options] *******************************************************************************************************************************************
ok: [localhost]

TASK [update VPC with DHCP options] *****************************************************************************************************************************************
changed: [localhost]

TASK [meta] *****************************************************************************************************************************************************************

TASK [debug] ****************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "ec2 hosts: ['ip-10-3-0-182.ec2.internal']"
}

Could you please provide a playbook to reproduce this issue ?

alinabuzachis · 2022-05-18T12:27:33Z

@link89 Could you please let us know if the issues still persist and in case have a look at what Aubin suggested?

ansibullbot · 2022-05-18T12:34:36Z

@link89 This issue is waiting for your response. Please respond or the issue will be closed.

click here for bot help

link89 · 2022-05-18T14:10:25Z

Hi @abikouo @alinabuzachis We figure out a fix, I think we are missing hostnames sections in our setup.

hostnames:
 - private-ip-address

I don't know why it works if we don't change the dhcp setting thought. But it works fine now.

Add waiter to the iam_role module SUMMARY This change adds the wait param used in other AWS modules, adding usage of a waiter for the iam_role creation / updates. Currently there is no waiting done to ensure the iam_role has actually created and is available before exiting. The tests have also been split up into separate files to make it a bit more manageable. Fixes: ansible-collections#710 ISSUE TYPE Feature Pull Request COMPONENT NAME iam_role ADDITIONAL INFORMATION Successful run completed of the iam_role integration test suite locally: ansible-test integration --docker centos8 -v iam_role --allow-unsupported PLAY RECAP ********************************************************************* testhost : ok=198 changed=46 unreachable=0 failed=0 skipped=0 rescued=0 ignored=7 AWS ACTIONS: ['iam:AddRoleToInstanceProfile', 'iam:AttachRolePolicy', 'iam:CreateInstanceProfile', 'iam:CreatePolicy', 'iam:CreateRole', 'iam:DeleteInstanceProfile', 'iam:DeletePolicy', 'iam:DeleteRole', 'iam:DeleteRolePermissionsBoundary', 'iam:DeleteRolePolicy', 'iam:DetachRolePolicy', 'iam:GetRole', 'iam:GetRolePolicy', 'iam:ListAttachedRolePolicies', 'iam:ListEntitiesForPolicy', 'iam:ListInstanceProfilesForRole', 'iam:ListPolicies', 'iam:ListPolicyVersions', 'iam:ListRolePolicies', 'iam:ListRoleTags', 'iam:ListRoles', 'iam:PutRolePermissionsBoundary', 'iam:PutRolePolicy', 'iam:RemoveRoleFromInstanceProfile', 'iam:TagRole', 'iam:UntagRole', 'iam:UpdateRole'] Run command: docker exec 56cb328c6d9af293d9e820e1f2a94fb8ca87e0769b2b9b6d46bad661f9edde65 tar czf /root/output.tgz --exclude .tmp -C /root/ansible/ansible_collections/community/aws/tests output Run command: docker exec -i 56cb328c6d9af293d9e820e1f2a94fb8ca87e0769b2b9b6d46bad661f9edde65 dd if=/root/output.tgz bs=65536 Run command: tar oxzf /tmp/ansible-result-k2lnga3v.tgz -C /mnt/c/Users/mark.woolley/Documents/GitHub/public/ansible_collections/community/aws/tests Run command: docker rm -f 56cb328c6d9af293d9e820e1f2a94fb8ca87e0769b2b9b6d46bad661f9edde65 Reviewed-by: Mark Chappell <None> Reviewed-by: None <None> This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections/community.aws@81d9abd

Add waiter to the iam_role module SUMMARY This change adds the wait param used in other AWS modules, adding usage of a waiter for the iam_role creation / updates. Currently there is no waiting done to ensure the iam_role has actually created and is available before exiting. The tests have also been split up into separate files to make it a bit more manageable. Fixes: ansible-collections#710 ISSUE TYPE Feature Pull Request COMPONENT NAME iam_role ADDITIONAL INFORMATION Successful run completed of the iam_role integration test suite locally: ansible-test integration --docker centos8 -v iam_role --allow-unsupported PLAY RECAP ********************************************************************* testhost : ok=198 changed=46 unreachable=0 failed=0 skipped=0 rescued=0 ignored=7 AWS ACTIONS: ['iam:AddRoleToInstanceProfile', 'iam:AttachRolePolicy', 'iam:CreateInstanceProfile', 'iam:CreatePolicy', 'iam:CreateRole', 'iam:DeleteInstanceProfile', 'iam:DeletePolicy', 'iam:DeleteRole', 'iam:DeleteRolePermissionsBoundary', 'iam:DeleteRolePolicy', 'iam:DetachRolePolicy', 'iam:GetRole', 'iam:GetRolePolicy', 'iam:ListAttachedRolePolicies', 'iam:ListEntitiesForPolicy', 'iam:ListInstanceProfilesForRole', 'iam:ListPolicies', 'iam:ListPolicyVersions', 'iam:ListRolePolicies', 'iam:ListRoleTags', 'iam:ListRoles', 'iam:PutRolePermissionsBoundary', 'iam:PutRolePolicy', 'iam:RemoveRoleFromInstanceProfile', 'iam:TagRole', 'iam:UntagRole', 'iam:UpdateRole'] Run command: docker exec 56cb328c6d9af293d9e820e1f2a94fb8ca87e0769b2b9b6d46bad661f9edde65 tar czf /root/output.tgz --exclude .tmp -C /root/ansible/ansible_collections/community/aws/tests output Run command: docker exec -i 56cb328c6d9af293d9e820e1f2a94fb8ca87e0769b2b9b6d46bad661f9edde65 dd if=/root/output.tgz bs=65536 Run command: tar oxzf /tmp/ansible-result-k2lnga3v.tgz -C /mnt/c/Users/mark.woolley/Documents/GitHub/public/ansible_collections/community/aws/tests Run command: docker rm -f 56cb328c6d9af293d9e820e1f2a94fb8ca87e0769b2b9b6d46bad661f9edde65 Reviewed-by: Mark Chappell <None> Reviewed-by: None <None>

Add waiter to the iam_role module SUMMARY This change adds the wait param used in other AWS modules, adding usage of a waiter for the iam_role creation / updates. Currently there is no waiting done to ensure the iam_role has actually created and is available before exiting. The tests have also been split up into separate files to make it a bit more manageable. Fixes: ansible-collections#710 ISSUE TYPE Feature Pull Request COMPONENT NAME iam_role ADDITIONAL INFORMATION Successful run completed of the iam_role integration test suite locally: ansible-test integration --docker centos8 -v iam_role --allow-unsupported PLAY RECAP ********************************************************************* testhost : ok=198 changed=46 unreachable=0 failed=0 skipped=0 rescued=0 ignored=7 AWS ACTIONS: ['iam:AddRoleToInstanceProfile', 'iam:AttachRolePolicy', 'iam:CreateInstanceProfile', 'iam:CreatePolicy', 'iam:CreateRole', 'iam:DeleteInstanceProfile', 'iam:DeletePolicy', 'iam:DeleteRole', 'iam:DeleteRolePermissionsBoundary', 'iam:DeleteRolePolicy', 'iam:DetachRolePolicy', 'iam:GetRole', 'iam:GetRolePolicy', 'iam:ListAttachedRolePolicies', 'iam:ListEntitiesForPolicy', 'iam:ListInstanceProfilesForRole', 'iam:ListPolicies', 'iam:ListPolicyVersions', 'iam:ListRolePolicies', 'iam:ListRoleTags', 'iam:ListRoles', 'iam:PutRolePermissionsBoundary', 'iam:PutRolePolicy', 'iam:RemoveRoleFromInstanceProfile', 'iam:TagRole', 'iam:UntagRole', 'iam:UpdateRole'] Run command: docker exec 56cb328c6d9af293d9e820e1f2a94fb8ca87e0769b2b9b6d46bad661f9edde65 tar czf /root/output.tgz --exclude .tmp -C /root/ansible/ansible_collections/community/aws/tests output Run command: docker exec -i 56cb328c6d9af293d9e820e1f2a94fb8ca87e0769b2b9b6d46bad661f9edde65 dd if=/root/output.tgz bs=65536 Run command: tar oxzf /tmp/ansible-result-k2lnga3v.tgz -C /mnt/c/Users/mark.woolley/Documents/GitHub/public/ansible_collections/community/aws/tests Run command: docker rm -f 56cb328c6d9af293d9e820e1f2a94fb8ca87e0769b2b9b6d46bad661f9edde65 Reviewed-by: Mark Chappell <None> Reviewed-by: None <None> This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections/community.aws@81d9abd

Add waiter to the iam_role module SUMMARY This change adds the wait param used in other AWS modules, adding usage of a waiter for the iam_role creation / updates. Currently there is no waiting done to ensure the iam_role has actually created and is available before exiting. The tests have also been split up into separate files to make it a bit more manageable. Fixes: ansible-collections#710 ISSUE TYPE Feature Pull Request COMPONENT NAME iam_role ADDITIONAL INFORMATION Successful run completed of the iam_role integration test suite locally: ansible-test integration --docker centos8 -v iam_role --allow-unsupported PLAY RECAP ********************************************************************* testhost : ok=198 changed=46 unreachable=0 failed=0 skipped=0 rescued=0 ignored=7 AWS ACTIONS: ['iam:AddRoleToInstanceProfile', 'iam:AttachRolePolicy', 'iam:CreateInstanceProfile', 'iam:CreatePolicy', 'iam:CreateRole', 'iam:DeleteInstanceProfile', 'iam:DeletePolicy', 'iam:DeleteRole', 'iam:DeleteRolePermissionsBoundary', 'iam:DeleteRolePolicy', 'iam:DetachRolePolicy', 'iam:GetRole', 'iam:GetRolePolicy', 'iam:ListAttachedRolePolicies', 'iam:ListEntitiesForPolicy', 'iam:ListInstanceProfilesForRole', 'iam:ListPolicies', 'iam:ListPolicyVersions', 'iam:ListRolePolicies', 'iam:ListRoleTags', 'iam:ListRoles', 'iam:PutRolePermissionsBoundary', 'iam:PutRolePolicy', 'iam:RemoveRoleFromInstanceProfile', 'iam:TagRole', 'iam:UntagRole', 'iam:UpdateRole'] Run command: docker exec 56cb328c6d9af293d9e820e1f2a94fb8ca87e0769b2b9b6d46bad661f9edde65 tar czf /root/output.tgz --exclude .tmp -C /root/ansible/ansible_collections/community/aws/tests output Run command: docker exec -i 56cb328c6d9af293d9e820e1f2a94fb8ca87e0769b2b9b6d46bad661f9edde65 dd if=/root/output.tgz bs=65536 Run command: tar oxzf /tmp/ansible-result-k2lnga3v.tgz -C /mnt/c/Users/mark.woolley/Documents/GitHub/public/ansible_collections/community/aws/tests Run command: docker rm -f 56cb328c6d9af293d9e820e1f2a94fb8ca87e0769b2b9b6d46bad661f9edde65 Reviewed-by: Mark Chappell <None> Reviewed-by: None <None>

ansibullbot added bug This issue/PR relates to a bug inventory inventory plugin needs_triage plugins plugin (any type) python3 labels Mar 15, 2022

alinabuzachis added needs_verified Some one might want to take a look at this and reproduce it to confirm and removed needs_triage labels Mar 15, 2022

abikouo self-assigned this Mar 18, 2022

abikouo added needs_info This issue requires further information. Please answer any outstanding questions and removed needs_verified Some one might want to take a look at this and reproduce it to confirm bug This issue/PR relates to a bug labels Mar 28, 2022

abikouo removed their assignment Mar 28, 2022

ansibullbot removed the needs_info This issue requires further information. Please answer any outstanding questions label May 18, 2022

alinabuzachis closed this as completed May 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ec2 inventory lost instances if customize dns is set via dhcp options #710

ec2 inventory lost instances if customize dns is set via dhcp options #710

link89 commented Mar 15, 2022

ansibullbot commented Mar 15, 2022

abikouo commented Mar 28, 2022

alinabuzachis commented May 18, 2022

ansibullbot commented May 18, 2022

link89 commented May 18, 2022

ec2 inventory lost instances if customize dns is set via dhcp options #710

ec2 inventory lost instances if customize dns is set via dhcp options #710

Comments

link89 commented Mar 15, 2022

Summary

Issue Type

Component Name

Ansible Version

Collection Versions

AWS SDK versions

Configuration

OS / Environment

Steps to Reproduce

Expected Results

Actual Results

Code of Conduct

ansibullbot commented Mar 15, 2022

abikouo commented Mar 28, 2022

alinabuzachis commented May 18, 2022

ansibullbot commented May 18, 2022

link89 commented May 18, 2022