Ansible/EC2/Dynamic Inventory

From charlesreid1

{Main|Ansible/EC2}}

This page covers how to use a dynamic inventory script to manage Ansible nodes when using Amazon AWS EC2.

Static vs dynamic inventory

Ansible/EC2/Static Inventory - static inventory requires the hosts file (containing the list of machines that Ansible is managing) be kept up to date by hand. This can be a burden if details are changing often or if resources are allocated automatically.

Ansible/EC2/Dynamic Inventory - dynamic inventory uses the AWS API to get information about machines that AWS is managing.

Managing a dynamic inventory using EC2 nodes: example script

Here we walk through the dynamic EC2 inventory script provided by Ansible in their Github repository: https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py

This script also has an .ini configuration file associated with it: https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini

Let's run through an overview of how it works.

Environment variables

This dynamic inventory script uses lots of environment variables. The most important are:

  • AWS_ACCESS_KEY_ID to set your AWS API access key (for boto)
  • AWS_SECRET_ACCESS_KEY to set your AWS API access secret (for boto)
  • AWS_PROFILE to specify a boto profile
  • EC2_INSTANCE_FILTERS to filter the AWS instances returned on various criteria. Extremely detailed API reference for filtering is here

Note that each of these can also be specified in the .ini file.

Important

By default, the ec2.ini file is configured for all Amazon cloud services. You have to turn off the ones you don't want (elasticcache, rds, etc.)

Ref: https://docs.ansible.com/ansible/latest/user_guide/intro_dynamic_inventory.html#inventory-script-example-aws-ec2

Python script

The script proper defines a Python object that manages all of the information received from the AWS API.

The script uses the boto library to interact with the AWS API. The object defines methods for parsing user command line arguments, and implements a number of other methods to do things like ask for a list of nodes, filter nodes by attribute, and store/retrieve information from a cache on disk to prevent everything from going extremely slowly due to slow AWS API responses.

Key methods defined for the object:

The call order of the script, when run, is:

Alternative implementation that deals with auto scaling groups

An alternative dynamic inventory script is provided by @jszwedko: http://jesseszwedko.com/post/ansible-autoscaling-inventory/

This script is not the same as the EC2 script provided by Ansible. This script deals with ASGs (Auto-Scaling Groups) and uses their Name tag to return instances. The Ansible-provided EC2 script does not deal with ASGs.

Note: what is an ASG?


An Auto Scaling group contains a collection of EC2 instances that share similar characteristics and are treated as a logical grouping for the purposes of instance scaling and management.

- https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html


Inventory script

Here is the dynamic inventory script:

asg_inventory: 

#!/usr/bin/env python

import argparse
import boto.ec2.autoscale
import json
import os
import sys

def get_tag(tags, key):
    for tag in tags:
        if tag.key == key:
            return tag.value

    return None

region = os.environ.get('AWS_REGION', None)
if region is None:
    print("$AWS_REGION must be set")
    sys.exit(1)

parser = argparse.ArgumentParser(description='Dynamic inventory for autoscaling groups')
parser.add_argument('--list', help="list hosts", action="store_true")
parser.add_argument('--host', help="list host vars")
args = parser.parse_args()

if args.host:
  print("{}")

if not args.list:
  sys.exit(1)

autoscale = boto.ec2.autoscale.connect_to_region(region)
ec2 = boto.ec2.connect_to_region(region)

inventory = {"_meta": {"hostvars": {}}}
for autoscaling_group in autoscale.get_all_groups():
  instance_ids = [i.instance_id for i in autoscaling_group.instances]
  instance_dns_names = [i.public_dns_name for r in ec2.get_all_instances(instance_ids) for i in r.instances]
  name = get_tag(autoscaling_group.tags, 'Name')
  if name not in inventory:
      inventory[name] = { "hosts": [] }
  inventory[name]['hosts'] += instance_dns_names

print(json.dumps(inventory))

Playbook

In your playbook you should set the target host machines for each task as a variable, so that you can pass the target auto-scaling group via the command line.

Example playbook: 

---
- hosts: "{{ target }}"
  vars:
    target: 127.0.0.1
  tasks:

Now, you can run 

AWS_REGION=us-west-1 
ansible-playbook \
    -i ./asg_inventory \
    /path/to/playbook.yml \
    -e target=mycoolasg

This will run the local playbook against remote instances in the mycoolasg autoscaling group as you make changes locally.

Workflow

The final workflow with this script looks like this:

Tools:

  • Ansible
  • dynamic inventory script above
  • Terraform

Preparation:

For local development:

  • From your LOCAL machine, run the playbook with the desired target passed via command line

For production:

  • Launch one AWS instance (the Ansible mastermind instance) that has the Ansible playbook bundled up and unpacked as part of the cloud-init process
  • The cloud-init process starts up Ansible and runs the playbook automatically

Here's what the final cloud init script would look like (again, this can be automatically generated via scripts provided here: http://jesseszwedko.com/post/ansible-aws-launch-configuration/): 

#!/bin/bash

set -o errexit

mkdir -p /tmp/ansible
echo 'H4sIAAAAAAAAA+w9aXPbRpb+[...]+CV+qgxijv8ggqkPOrhZngzP99N/Tv0z+6bZbGw3u9QXa2v1ovOfw0nfP/yPxjlBAKBQCAQCAQCgUAgEAgEAoHg+eEnXr53cwCgAAA=' | base64 -d | tar xz -C /tmp/ansible
cd /tmp/ansible
/usr/local/bin/ansible-playbook playbook.yml --connection=local -i localhost, -e target=localhost

Flags


Defcon.png
Ansible
Ansible is a Python library for managing IT and compute infrastructure.

Ansible  · Category:Ansible


Ansible Configuration

Config file: Ansible/Configuration

Secrets and Sensitive Info: Ansible/Security  · Ansible/Vaults


Ansible Playbooks

Basics: Ansible/Playbooks

Examples: Ansible/Nginx Playbook (simple)  · Ansible/Full Stack Playbook (advanced)


Ansible Hosts and Inventory Control

All topics in Ansible Hosts: Ansible/Hosts

Splitting large host files into subfiles: Ansible/Splitting Hosts File

Automatically naming hosts: Ansible/Host Naming

Grouping Hosts: Ansible/Groups

Assigning Variables to Groups: Ansible/Group Variables

Roles: Ansible/Roles


Ansible and Vagrant (Local)

Ansible/Vagrant  · Ansible/Vagrant/Static Inventory  · Ansible/Vagrant/Dynamic Inventory


Ansible and EC2 (Cloud)

Ansible/EC2  · Ansible/EC2/Static Inventory  · Ansible/EC2/Dynamic Inventory


Best Practices

Ansible/Directory Layout  · Ansible/Directory Layout/Details

Ansible/Separate Playbooks by Role

Ansible/Tag Inventory with Environment

Ansible/Group by Distribution

Ansible/Variables and Vaults


Ansible Galaxy

Ansible/Galaxy


Flags  · Template:AnsibleFlag  · e



Retrieved from "https://charlesreid1.com/w/index.php?title=Ansible/EC2/Dynamic_Inventory&oldid=25522"