From 926662ec79a9fbf5045084b6e0feaed808d3df65 Mon Sep 17 00:00:00 2001 From: "patchback[bot]" <45432694+patchback[bot]@users.noreply.github.com> Date: Fri, 5 Aug 2022 12:58:40 +0000 Subject: [PATCH] aws_ec2 doc guide (#904) (#957) [PR #904/4d195aa2 backport][stable-4] aws_ec2 doc guide This is a backport of PR #904 as merged into main (4d195aa). SUMMARY aws_ec2 doc guide ISSUE TYPE Docs Pull Request COMPONENT NAME aws_ec2 Reviewed-by: Mark Chappell --- docs/docsite/extra-docs.yml | 3 + docs/docsite/rst/aws_ec2_guide.rst | 582 +++++++++++++++++++++++++++++ docs/docsite/rst/guide_aws.rst | 4 +- 3 files changed, 587 insertions(+), 2 deletions(-) create mode 100644 docs/docsite/rst/aws_ec2_guide.rst diff --git a/docs/docsite/extra-docs.yml b/docs/docsite/extra-docs.yml index 8a402e3adaa..5c15da22ceb 100644 --- a/docs/docsite/extra-docs.yml +++ b/docs/docsite/extra-docs.yml @@ -6,3 +6,6 @@ sections: - title: Module Development Guidelines toctree: - dev_guidelines + - title: Dynamic Inventory Plugin Guide + toctree: + - aws_ec2_guide diff --git a/docs/docsite/rst/aws_ec2_guide.rst b/docs/docsite/rst/aws_ec2_guide.rst new file mode 100644 index 00000000000..fd330c1e6db --- /dev/null +++ b/docs/docsite/rst/aws_ec2_guide.rst @@ -0,0 +1,582 @@ +.. _ansible_collections.amazon.aws.docsite.dynamic_inventory: + + +Dynamic Inventory Plugin +======================== + +A dynamic inventory plugin allows users to point at data sources to compile the inventory of hosts that Ansible uses to target tasks, either via the ``-i /path/to/file`` and/or ``-i 'host1, host2'`` command line parameters or from other configuration sources. + +When using Ansible with AWS, inventory file maintenance will be a hectic task as AWS frequently changes IPs, autoscaling instances, and more. +Once your AWS EC2 hosts are spun up, you'll probably want to talk to them again. +With a cloud setup, it's best not to maintain a static list of cloud hostnames in text files. +Rather, the best way to handle this is to use the ``aws_ec2`` dynamic inventory plugin. + +The ``aws_ec2`` dynamic inventory plugin makes API calls to AWS to get a list of inventory hosts from Amazon Web Services EC2 in the run time. +It gives the EC2 instance details dynamically to manage the AWS infrastructure. + +The plugin will also return instances that were created outside of Ansible and allow Ansible to manage them. + +To start using the ``aws_ec2`` dynamic inventory plugin with a YAML configuration source, create a file with the accepted filename schema documented for the plugin (a YAML configuration file that ends with ``aws_ec2.(yml|yaml)``, e.g., ``demo.aws_ec2.yml``), then add ``plugin: amazon.aws.aws_ec2``. Use the fully qualified name if the plugin is in a collection. + +.. _ansible_collections.amazon.aws.docsite.using_inventory_plugin: + +Authentication +============== + +If your Ansible controller is not in AWS, authentication is handled by either +specifying your access and secret key as ENV variables or inventory plugin arguments. + +For environment variables: + +.. code-block:: bash + + export AWS_ACCESS_KEY_ID='AK123' + export AWS_SECRET_ACCESS_KEY='abc123' + +The ``AWS_SECURITY_TOKEN`` environment variable can also be used, but is only supported for backward compatibility. +The ``AWS_SECURITY_TOKEN`` is a replacement for ``AWS_SESSION_TOKEN`` and it is only needed when you are using temporary credentials. + +Or you can set ``aws_access_key``, ``aws_secret_key``, and ``security_token`` inside the inventory configuration file. + +.. code-block:: yaml + + # demo.aws_ec2.yml + plugin: amazon.aws.aws_ec2 + + # The access key for your AWS account. + aws_access_key: + # The secret access key for your AWS account. + aws_secret_key: + +If you use different credentials for different tools or applications, you can use profiles. + +The ``profile`` argument is mutually exclusive with the ``aws_access_key``, ``aws_secret_key`` and ``security_token`` options. +When no credentials are explicitly provided then the AWS SDK (boto3) which Ansible uses will fall back to its configuration files (typically ``~/.aws/credentials``). +The shared credentials file has a default location of ``~/.aws/credentials``. +You can change the location of the shared credentials file by setting the ``AWS_SHARED_CREDENTIALS_FILE`` environment variable. + +.. code-block:: yaml + + # demo.aws_ec2.yml + plugin: amazon.aws.aws_ec2 + + # Attach the default AWS profile + aws_profile: default + + # You could use Jinja2 to attach the AWS profile from the environment variable. + aws_profile: "{{ lookup('env', 'AWS_PROFILE') | default('dev-profile', true) }}" + +You can also set your AWS profile as an ENV variable: + +.. code-block:: bash + + export AWS_PROFILE='test-profile' + + +If your Ansible controller is running on an EC2 instance with an assigned IAM Role, the credential may be omitted. +See the documentation for the controller `for more details `_. + +You can also use the ARN of the IAM role to assume to perform the inventory lookup. +This can be useful for connecting across different accounts, or to limit user access. +To do so, you should specify the ``iam_role_arn``. +You should still provide AWS credentials with enough privilege to perform the AssumeRole action. + +.. code-block:: yaml + + # demo.aws_ec2.yml + plugin: amazon.aws.aws_ec2 + + iam_role_arn: arn:aws:iam::1234567890:role/assumed-ansible + + +Minimal Example +=============== + +Fetch all hosts in us-east-1, the hostname is the public DNS if it exists, otherwise the private IP address. + +.. code-block:: yaml + + # demo.aws_ec2.yml + plugin: amazon.aws.aws_ec2 + + # This sets the region. If empty (the default) default this will include all regions, except possibly + # restricted ones like us-gov-west-1 and cn-north-1. + regions: + - us-east-1 + +After providing any required options, you can view the populated inventory with ``ansible-inventory -i demo.aws_ec2.yml --graph``: + +.. code-block:: text + + @all: + |--@aws_ec2: + | |--ip-10-210-0-189.ec2.internal + | |--ip-10-210-0-195.ec2.internal + |--@ungrouped: + + +Allowed Options +=============== + +Some of the ``aws_ec2`` dynamic inventory plugin options are explained in detail below. For a full list see `the plugin documentation `_. + +``hostnames`` +------------- + +``hostnames`` option provides different settings to choose how the hostname will be displayed. + +Some examples are shown below: + +.. code-block:: yaml + + hostnames: + # This option allows displaying the public ip addresses. + - ip-address + + # This option allows displaying the private ip addresses using `tag:Name` as a prefix. + # `name` can be one of the options specified in http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html#options. + - name: 'private-ip-address' + separator: '_' + prefix: 'tag:Name' + + # Using literal values for hostname + # # Hostname will be aws-test_literal + - name: 'test_literal' + separator: '-' + prefix: 'aws' + + # To use tags as hostnames use the syntax `tag:Name=Value` to use the hostname `Name_Value`, or + # `tag:Name` to use the value of the Name tag. If value provided does not exist in the above options, + # it will be used as a literal string. + - name: 'tag:Tag1=Test1,Tag2=Test2' + + # Use dns-name attribute as hostname + - dns-name + + # You can also specify a list in order of precedence for hostname variables. + - ip-address + - dns-name + - tag:Name + - private-ip-address + + +``keyed_groups`` +---------------- + +You can create dynamic groups using host variables with the ``keyed_groups`` option. ``keyed_groups`` comes in a prefix and a key format. +The prefix will be the name of the host group that is to be concatenated with the key. + +Some examples are shown below: + +.. code-block:: yaml + + keyed_groups: + # This creates host groups based on architecture. + - prefix: arch + key: architecture + + # This creates host groups based on `x86_64` architecture. + - prefix: arch + key: architecture + value: + 'x86_64' + + # This creates host groups based on availability zone. + - prefix: az + key: placement.availability_zone + + # If the EC2 tag Name had the value `redhat` the tag variable would be: `tag_Name_redhat`. + # Similarly, if a tag existed for an AWS EC2 instance as `Applications` with the value of `nodejs` the + # variable would be: `tag_Applications_nodejs`. + - prefix: tag + key: tags + + # This creates host groups using instance_type, e.g., `instance_type_z3_tiny`. + - prefix: instance_type + key: instance_type + + # This creates host groups using security_groups id, e.g., `security_groups_sg_abcd1234` group for each security group. + - key: 'security_groups|json_query("[].group_id")' + prefix: 'security_groups' + + # This creates a host group for each value of the Application tag. + - key: tags.Application + separator: '' + + # This creates a host group per region e.g., `aws_region_us_east_2`. + - key: placement.region + prefix: aws_region + + # This creates host groups based on the value of a custom tag `Role` and adds them to a metagroup called `project`. + - key: tags['Role'] + prefix: foo + parent_group: "project" + + # This creates a common parent group for all EC2 availability zones. + - key: placement.availability_zone + parent_group: all_ec2_zones + + # This creates a group per distro (distro_CentOS, distro_Debian) and assigns the hosts that have matching values to it, + # using the default separator "_". + - prefix: distro + key: ansible_distribution + + +``groups`` +---------- + +It is also possible to create groups using the ``groups`` option. + +Some examples are shown below: + +.. code-block:: yaml + + groups: + # This created two groups - `Production` and `PreProduction` based on tags + # These conditionals are expressed using Jinja2 syntax. + redhat: "'Production' in tags.Environment" + ubuntu: "'PreProduction' in tags.Environment" + + # This created a libvpc group based on specific condition on `vpc_id`. + libvpc: vpc_id == 'vpc-####' + + +``compose`` +----------- + +``compose`` creates and modifies host variables from Jinja2 expressions. + +.. code-block:: yaml + + compose: + # This sets the ansible_host variable to connect with the private IP address without changing the hostname. + ansible_host: private_ip_address + + # This sets location_vars variable as a dictionary with location as a key. + location_vars: + location: "east_coast" + server_type: "ansible_hostname | regex_replace ('(.{6})(.{2}).*', '\\2')" + + # This sets location variable. + location: "'east_coast'" + + # This lets you connect over SSM to the instance id. + ansible_host: instance_id + ansible_connection: 'community.aws.aws_ssm' + + # This defines combinations of host servers, IP addresses, and related SSH private keys. + ansible_host: private_ip_address + ansible_user: centos + ansible_private_ssh_key_file: key_name + + # This sets the ec2_security_group_ids variable. + ec2_security_group_ids: security_groups | map(attribute='group_id') | list | join(',') + + +``include_filters`` and ``exclude_filters`` +------------------------------------------- + +``include_filters`` and ``exclude_filters`` options give you the ability to compose the inventory with several queries (see `available filters `_). + +.. code-block:: yaml + + include_filters: + # This includes everything in the inventory that has the following tags. + - tag:Project: + - 'planets' + - tag:Environment: + - 'demo' + + # This excludes everything from the inventory that has the following tag:Name. + exclude_filters: + - tag:Name: + - '{{ resource_prefix }}_3' + + +``filters`` +----------- + +``filters`` are used to filter out AWS EC2 instances based on conditions (see `available filters `_). + +.. code-block:: yaml + + filters: + # This selects only running instances with tag `Environment` tag set to `dev`. + tag:Environment: dev + instance-state-name : running + + # This selects only instances with tag `Environment` tag set to `dev` and `qa` and specific security group id. + tag:Environment: + - dev + - qa + instance.group-id: sg-xxxxxxxx + + # This selects only instances with tag `Name` fulfilling specific conditions. + - tag:Name: + - dev-* + - share-resource + - hotfix + + +``use_contrib_script_compatible_ec2_tag_keys`` and ``use_contrib_script_compatible_sanitization`` +------------------------------------------------------------------------------------------------- + +``use_contrib_script_compatible_ec2_tag_keys`` exposes the host tags with ec2_tag_TAGNAME keys like the old ec2.py inventory script when it's True. + +By default the ``aws_ec2`` plugin is using a general group name sanitization to create safe and usable group names for use in Ansible. + +``use_contrib_script_compatible_ec2_tag_keys`` allows you to override that, in efforts to allow migration from the old inventory script and matches the sanitization of groups when the script's replace_dash_in_groups option is set to False. +To replicate behavior of replace_dash_in_groups = True with constructed groups, you will need to replace hyphens with underscores via the regex_replace filter for those entries. + +For this to work you should also turn off the TRANSFORM_INVALID_GROUP_CHARS setting, otherwise the core engine will just use the standard sanitization on top. + +This is not the default as such names break certain functionality as not all characters are valid Python identifiers which group names end up being used as. + +The use of this feature is discouraged and we advise to migrate to the new tags structure. + +.. code-block:: yaml + + # demo.aws_ec2.yml + plugin: amazon.aws.aws_ec2 + regions: + - us-east-1 + filters: + tag:Name: + - 'instance-*' + hostnames: + - tag:Name + use_contrib_script_compatible_sanitization: True + use_contrib_script_compatible_ec2_tag_keys: True + +After providing any required options, you can view the populated inventory with ``ansible-inventory -i demo.aws_ec2.yml --list``: + +.. code-block:: text + + { + "_meta": { + "hostvars": { + "instance-01": { + "aws_ami_launch_index_ec2": 0, + "aws_architecture_ec2": "x86_64", + ... + "ebs_optimized": false, + "ec2_tag_Environment": "dev", + "ec2_tag_Name": "instance-01", + "ec2_tag_Tag1": "Test1", + "ec2_tag_Tag2": "Test2", + "ena_support": true, + "enclave_options": { + "enabled": false + }, + ... + }, + "instance-02": { + ... + "ebs_optimized": false, + "ec2_tag_Environment": "dev", + "ec2_tag_Name": "instance-02", + "ec2_tag_Tag1": "Test3", + "ec2_tag_Tag2": "Test4", + "ena_support": true, + "enclave_options": { + "enabled": false + }, + ... + } + } + }, + all": { + "children": [ + "aws_ec2", + "ungrouped" + ] + }, + "aws_ec2": { + "hosts": [ + "instance-01", + "instance-02" + ] + } + } + + +``hostvars_prefix`` and ``hostvars_suffix`` +------------------------------------------- + +``hostvars_prefix`` and ``hostvars_sufix`` allow to set up a prefix and suffix for host variables. + +.. code-block:: yaml + + # demo.aws_ec2.yml + plugin: amazon.aws.aws_ec2 + regions: + - us-east-1 + filters: + tag:Name: + - 'instance-*' + hostvars_prefix: 'aws_' + hostvars_suffix: '_ec2' + hostnames: + - tag:Name + +Now the output of ``ansible-inventory -i demo.aws_ec2.yml --list``: + +.. code-block:: text + + { + "_meta": { + "hostvars": { + "instance-01": { + "aws_ami_launch_index_ec2": 0, + "aws_architecture_ec2": "x86_64", + "aws_block_device_mappings_ec2": [ + { + "device_name": "/dev/sda1", + "ebs": { + "attach_time": "2022-06-27T09:04:57+00:00", + "delete_on_termination": true, + "status": "attached", + "volume_id": "vol-06e065bca44e6eae5" + } + } + ], + "aws_capacity_reservation_specification_ec2": { + "capacity_reservation_preference": "open" + } + ..., + }, + "instance-02": { + ..., + } + } + }, + all": { + "children": [ + "aws_ec2", + "ungrouped" + ] + }, + "aws_ec2": { + "hosts": [ + "instance-01", + "instance-02" + ] + } + } + + +``strict`` and ``strict_permissions`` +------------------------------------- + +``strict: False`` will skip instead of producing an error if there are missing facts. + +``strict_permissions: False`` will ignore 403 errors rather than failing. + + +``cache`` +--------- + +``aws_ec2`` inventory plugin support caching can use the general settings for the fact cache defined in the ``ansible.cfg`` file's ``[defaults]`` section or define inventory-specific settings in the ``[inventory]`` section. +You can can define plugin-specific cache settings in the config file: + +.. code-block:: yaml + + # demo.aws_ec2.yml + plugin: aws_ec2 + # This enables cache. + cache: yes + # Plugin to be used. + cache_plugin: jsonfile + cache_timeout: 7200 + # Location where files are stored in the cache. + cache_connection: /tmp/aws_inventory + cache_prefix: aws_ec2 + +Here is an example of setting inventory caching with some fact caching defaults for the cache plugin used and the timeout in an ``ansible.cfg`` file: + +.. code-block:: ini + + [defaults] + fact_caching = ansible.builtin.jsonfile + fact_caching_connection = /tmp/ansible_facts + cache_timeout = 3600 + + [inventory] + cache = yes + cache_connection = /tmp/ansible_inventory + + +Complex Example +=============== + +Here is an ``aws_ec2`` complex example utilizing some of the previously listed options: + +.. code-block:: yaml + + # demo.aws_ec2.yml + plugin: amazon.aws.aws_ec2 + regions: + - us-east-1 + - us-east-2 + keyed_groups: + # add hosts to tag_Name_value groups for each aws_ec2 host's tags.Name variable. + - key: tags.Name + prefix: tag_Name_ + separator: "" + groups: + # add hosts to the group dev if any of the dictionary's keys or values is the word 'dev'. + development: "'dev' in (tags|list)" + filters: + tag:Name: + - 'instance-01' + - 'instance-03' + include_filters: + - tag:Name: + - 'instance-02' + - 'instance-04' + exclude_filters: + - tag:Name: + - 'instance-03' + - 'instance-04' + hostnames: + # You can also specify a list in order of precedence for hostname variables. + - ip-address + - dns-name + - tag:Name + - private-ip-address + compose: + # This sets the `ansible_host` variable to connect with the private IP address without changing the hostname. + ansible_host: private_ip_address + +If a host does not have the variables in the configuration above (i.e. ``tags.Name``, ``tags``, ``private_ip_address``), the host will not be added to groups other than those that the inventory plugin creates and the ``ansible_host`` host variable will not be modified. + +Now the output of ``ansible-inventory -i demo.aws_ec2.yml --graph``: + +.. code-block:: text + + @all: + |--@aws_ec2: + | |--instance-01 + | |--instance-02 + |--@tag_Name_instance_01: + | |--instance-01 + |--@tag_Name_instance_02: + | |--instance-02 + |--@ungrouped: + + +Using Dynamic Inventory Inside Playbook +======================================= + +If you want to use dynamic inventory inside the playbook, you just need to mention the group name in the hosts variable as shown below. + +.. code-block:: yaml + + --- + - name: Ansible Test Playbook + gather_facts: false + hosts: tag_Name_instance_02 + + tasks: + - name: Run Shell Command + command: echo "Hello World" diff --git a/docs/docsite/rst/guide_aws.rst b/docs/docsite/rst/guide_aws.rst index 123a5f4e840..4657dafd188 100644 --- a/docs/docsite/rst/guide_aws.rst +++ b/docs/docsite/rst/guide_aws.rst @@ -202,7 +202,7 @@ When using the inventory plugin, you can configure extra inventory structure bas For instance, you might use ``keyed_groups`` to create groups from instance tags:: - plugin: aws_ec2 + plugin: amazon.aws.aws_ec2 keyed_groups: - prefix: tag key: tags @@ -214,7 +214,7 @@ You can then target all instances with a "class" tag where the value is "webserv tasks: - ping -You can also use these groups with 'group_vars' to set variables that are automatically applied to matching instances. See :ref:`splitting_out_vars`. +You can also use these groups with 'group_vars' to set variables that are automatically applied to matching instances. .. _ansible_collections.amazon.aws.docsite.aws_pull: