Ansible/Hosts/Dynamic Inventory

What is a dynamic inventory script

In order to remotely connect to machines and run commands to configure the machines, Ansible needs information about how to connect to each remote machine.

The There are two ways to provide this information:

• a static inventory file, in which details are given by hand (this can be streamlined using groups and variables)
• a dynamic inventory file, in which a script provides Ansible with details about nodes using a dynamic resource (API, database, etc.)

How to use dynamic inventory script

The dynamic inventory script must accept to command line flags, for the two ways Ansible will call this dynamic inventory script:

--host=<hostname>   show host details
--list              list groups

For example, Ansible will call the inventory script like so:

$ ./dynamic.py --host=vagrant2

Example: Vagrant Dynamic Inventory Script

Let's walk through an example script with Vagrant.

This example script is written in Python. It takes two input arguments, --host=X and --list, and interacts with Vagrant to return the requested information.

Getting List of Hosts

Start with how to get a list of hosts for the --list flag.

Vagrant provides a command to show info about all running vagrant machines:

vagrant status

This info can be used to create the list of hosts for the dynamic inventory file. A more convenient format for parsing is:

vagrant status --machine-readable

which prints everything in CSV format. The code:

def list_running_hosts():
    cmd = "vagrant status --machine-readable"
    status = subprocess.check_output(cmd.split()).rstrip().decode('utf-8')
    hosts = []
    for line in status.split('\n'):
        (_, host, key, value) = line.split(',')[:4]
        if key == 'state' and value == 'running':
            hosts.append(host)
    return hosts

Getting Machine Configuration Details

Now we need to get host-specific details for the --host=X flag.

Next we can use the vagrant ssh-config command to get info about the machine-specific configuration details of each machine, which is necessary for Ansible to connect to them. To parse SSH config files, we can use a Python library called pamiko.

def get_host_details(host):
    cmd = "vagrant ssh-config {}".format(host)
    p = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
    config = paramiko.SSHConfig()
    config.parse(p.stdout)
    c = config.lookup(host)
    return {'ansible_host': c['hostname'],
            'ansible_port': c['port'],
            'ansible_user': c['user'],
            'ansible_private_key_file': c['identityfile'][0]}

Parsing user arguments

A quick argparser tutorial for the flags we're interested in parsing:

def parse_args():
    parser = argparse.ArgumentParser(description="Vagrant inventory script")
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument('--list', action='store_true')
    group.add_argument('--host')
    return parser.parse_args()

Now we can call this function and use args.list or args.host to access the values of the flags.

Putting it all together

Here is the final dynamic inventory script:

(In the ansible repo under contrib/inventory: https://github.com/ansible/ansible/tree/devel/contrib/inventory)

dynamic_vagrant.py:

#!/usr/bin/env python
# Adapted from Mark Mandel's implementation
# https://github.com/ansible/ansible/blob/stable-2.1/contrib/inventory/vagrant.py
# License: GNU General Public License, Version 3 <http://www.gnu.org/licenses/>
import argparse
import json
import paramiko
import subprocess
import sys


def parse_args():
    parser = argparse.ArgumentParser(description="Vagrant inventory script")
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument('--list', action='store_true')
    group.add_argument('--host')
    return parser.parse_args()


def list_running_hosts():
    cmd = "vagrant status --machine-readable"
    status = subprocess.check_output(cmd.split()).rstrip().decode('utf-8')
    hosts = []
    for line in status.split('\n'):
        (_, host, key, value) = line.split(',')[:4]
        if key == 'state' and value == 'running':
            hosts.append(host)
    return hosts


def get_host_details(host):
    cmd = "vagrant ssh-config {}".format(host)
    p = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
    config = paramiko.SSHConfig()
    config.parse(p.stdout)
    c = config.lookup(host)
    return {'ansible_host': c['hostname'],
            'ansible_port': c['port'],
            'ansible_user': c['user'],
            'ansible_private_key_file': c['identityfile'][0]}


def main():
    args = parse_args()
    if args.list:
        hosts = list_running_hosts()
        json.dump({'vagrant': hosts}, sys.stdout)
    else:
        details = get_host_details(args.host)
        json.dump(details, sys.stdout)

if __name__ == '__main__':
    main()

Running the dynamic inventory script

Once we've written the dynamic inventory script and we're ready to run Ansible using it, we can pass the inventory script using the -i flag:

ansible -i dynamic_vagrant.py -u ubuntu myvagrantbox -m ping

Example: EC2 Dynamic Inventory Script

There is a very thorough EC2 dynamic inventory script in the Ansible Github repository: https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py

This script also has an .ini configuration file associated with it: https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini

Let's run through an overview of how it works.

Environment variables

This dynamic inventory script uses lots of environment variables. The most important are:

• AWS_ACCESS_KEY_ID to set your AWS API access key (for boto)
• AWS_SECRET_ACCESS_KEY to set your AWS API access secret (for boto)
• AWS_PROFILE to specify a boto profile
• EC2_INSTANCE_FILTERS to filter the AWS instances returned on various criteria. Extremely detailed API reference for filtering is here

Note that each of these can also be specified in the .ini file.

Important

By default, the ec2.ini file is configured for all Amazon cloud services. You have to turn off the ones you don't want (elasticcache, rds, etc.)

Ref: https://docs.ansible.com/ansible/latest/user_guide/intro_dynamic_inventory.html#inventory-script-example-aws-ec2

Python script

The script proper defines a Python object that manages all of the information received from the AWS API.

The script uses the boto library to interact with the AWS API. The object defines methods for parsing user command line arguments, and implements a number of other methods to do things like ask for a list of nodes, filter nodes by attribute, and store/retrieve information from a cache on disk to prevent everything from going extremely slowly due to slow AWS API responses.

Key bits:

• read input file: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L304
• parse cli: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L523
• get instances: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L593
• get route names for instances from route 53: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L1468
• get a dictionary with instance information: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L1489
• process API call returns: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L1542

Call order:

• object is created https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L1709
• init method is called: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L245
• credentials are set up
• cache is loaded (if present)
• data is printed (list of nodes)