Ansible/Hosts/Dynamic Inventory: Difference between revisions

From charlesreid1

No edit summary
 
(9 intermediate revisions by the same user not shown)
Line 25: Line 25:




==Example executable dynamic inventory script==
==Example: Vagrant Dynamic Inventory Script==


Let's walk through an example script with Vagrant.
See [[Ansible/Vagrant/Dynamic Inventory]] for a sample dynamic inventory script for use with Vagrant.


This example script is written in Python. It takes two input arguments, <code>--host=X</code> and <code>--list</code>, and interacts with Vagrant to return the requested information.
==Example: EC2 Dynamic Inventory Script==


===Vagrant===
There is a very thorough EC2 dynamic inventory script in the Ansible Github repository: https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py


====Getting List of Hosts====
This script also has an .ini configuration file associated with it: https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini


Start with how to get a list of hosts for the <code>--list</code> flag.
Let's run through an overview of how it works.


Vagrant provides a command to show info about all running vagrant machines:
===Environment variables===


<pre>
This dynamic inventory script uses lots of environment variables. The most important are:
vagrant status
</pre>


This info can be used to create the list of hosts for the dynamic inventory file. A more convenient format for parsing is:
* <code>AWS_ACCESS_KEY_ID</code> to set your AWS API access key (for boto)
* <code>AWS_SECRET_ACCESS_KEY</code> to set your AWS API access secret (for boto)
* <code>AWS_PROFILE</code> to specify a boto profile
* <code>EC2_INSTANCE_FILTERS</code> to filter the AWS instances returned on various criteria. Extremely detailed API reference for filtering is [https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html#query-DescribeInstances-filters here]


<pre>
Note that each of these can also be specified in the .ini file.
vagrant status --machine-readable
</pre>


which prints everything in CSV format. The code:
===Important===


<pre>
By default, the ec2.ini file is configured for all Amazon cloud services. You have to turn off the ones you don't want (elasticcache, rds, etc.)


def list_running_hosts():
Ref: https://docs.ansible.com/ansible/latest/user_guide/intro_dynamic_inventory.html#inventory-script-example-aws-ec2
    cmd = "vagrant status --machine-readable"
    status = subprocess.check_output(cmd.split()).rstrip().decode('utf-8')
    hosts = []
    for line in status.split('\n'):
        (_, host, key, value) = line.split(',')[:4]
        if key == 'state' and value == 'running':
            hosts.append(host)
    return hosts
</pre>


====Getting Machine Configuration Details====
===Python script===


Now we need to get host-specific details for the <code>--host=X</code> flag.
The script proper defines a Python object that manages all of the information received from the AWS API.


Next we can use the <code>vagrant ssh-config</code> command to get info about the machine-specific configuration details of each machine, which is necessary for Ansible to connect to them. To parse SSH config files, we can use a Python library called pamiko.
The script uses the boto library to interact with the AWS API. The object defines methods for parsing user command line arguments, and implements a number of other methods to do things like ask for a list of nodes, filter nodes by attribute, and store/retrieve information from a cache on disk to prevent everything from going extremely slowly due to slow AWS API responses.


<pre>
Key methods defined for the object:
def get_host_details(host):
    cmd = "vagrant ssh-config {}".format(host)
    p = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
    config = paramiko.SSHConfig()
    config.parse(p.stdout)
    c = config.lookup(host)
    return {'ansible_host': c['hostname'],
            'ansible_port': c['port'],
            'ansible_user': c['user'],
            'ansible_private_key_file': c['identityfile'][0]}
</pre>


====Parsing user arguments====
* read input file: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L304
* parse cli: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L523
* get instances: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L593
* get route names for instances from route 53: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L1468
* get a dictionary with instance information: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L1489
* process API call returns: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L1542


A quick argparser tutorial for the flags we're interested in parsing:
The call order of the script, when run, is:


<pre>
* object is created https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L1709
def parse_args():
* init method is called: https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py#L245
    parser = argparse.ArgumentParser(description="Vagrant inventory script")
* credentials are set up
    group = parser.add_mutually_exclusive_group(required=True)
* cache is loaded (if present)
    group.add_argument('--list', action='store_true')
* data is printed (list of nodes)
    group.add_argument('--host')
    return parser.parse_args()
</pre>
 
Now we can call this function and use args.list or args.host to access the values of the flags.
 
====Putting it all together====
 
Here is the final dynamic inventory script:
 
'''dynamic_vagrant.py''':
 
<pre>
#!/usr/bin/env python
# Adapted from Mark Mandel's implementation
# https://github.com/ansible/ansible/blob/stable-2.1/contrib/inventory/vagrant.py
# License: GNU General Public License, Version 3 <http://www.gnu.org/licenses/>
import argparse
import json
import paramiko
import subprocess
import sys
 
 
def parse_args():
    parser = argparse.ArgumentParser(description="Vagrant inventory script")
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument('--list', action='store_true')
    group.add_argument('--host')
    return parser.parse_args()


=Flags=


def list_running_hosts():
[[Category:Python]]
    cmd = "vagrant status --machine-readable"
    status = subprocess.check_output(cmd.split()).rstrip().decode('utf-8')
    hosts = []
    for line in status.split('\n'):
        (_, host, key, value) = line.split(',')[:4]
        if key == 'state' and value == 'running':
            hosts.append(host)
    return hosts
 
 
def get_host_details(host):
    cmd = "vagrant ssh-config {}".format(host)
    p = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
    config = paramiko.SSHConfig()
    config.parse(p.stdout)
    c = config.lookup(host)
    return {'ansible_host': c['hostname'],
            'ansible_port': c['port'],
            'ansible_user': c['user'],
            'ansible_private_key_file': c['identityfile'][0]}
 
 
def main():
    args = parse_args()
    if args.list:
        hosts = list_running_hosts()
        json.dump({'vagrant': hosts}, sys.stdout)
    else:
        details = get_host_details(args.host)
        json.dump(details, sys.stdout)
 
if __name__ == '__main__':
    main()
</pre>
 
====Running the dynamic inventory script====
 
Once we've written the dynamic inventory script and we're ready to run Ansible using it, we can pass the inventory script using the -i flag:
 
<pre>
ansible -i dynamic_vagrant.py -u ubuntu myvagrantbox -m ping
</pre>
 
 
===EC2===
 
There is a very thorough EC2 dynamic inventory script in the Ansible Github repository: https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py
 
This script also has an .ini configuration file associated with it: https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini
 
=Flags=


[[Category:Ansible]]
[[Category:Ansible]]
[[Category:Ansible Hosts]]
[[Category:Ansible Hosts]]
[[Category:Infrastructure]]
[[Category:Infrastructure]]
[[Category:Dynamic Inventory]]
[[Category:AWS]]
[[Category:Vagrant]]

Latest revision as of 01:44, 11 November 2018

What is a dynamic inventory script

In order to remotely connect to machines and run commands to configure the machines, Ansible needs information about how to connect to each remote machine.

The There are two ways to provide this information:

  • a static inventory file, in which details are given by hand (this can be streamlined using groups and variables)
  • a dynamic inventory file, in which a script provides Ansible with details about nodes using a dynamic resource (API, database, etc.)

How to use dynamic inventory script

The dynamic inventory script must accept to command line flags, for the two ways Ansible will call this dynamic inventory script: 

--host=<hostname>   show host details
--list              list groups

For example, Ansible will call the inventory script like so: 

$ ./dynamic.py --host=vagrant2


Example: Vagrant Dynamic Inventory Script

See Ansible/Vagrant/Dynamic Inventory for a sample dynamic inventory script for use with Vagrant.

Example: EC2 Dynamic Inventory Script

There is a very thorough EC2 dynamic inventory script in the Ansible Github repository: https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.py

This script also has an .ini configuration file associated with it: https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini

Let's run through an overview of how it works.

Environment variables

This dynamic inventory script uses lots of environment variables. The most important are:

  • AWS_ACCESS_KEY_ID to set your AWS API access key (for boto)
  • AWS_SECRET_ACCESS_KEY to set your AWS API access secret (for boto)
  • AWS_PROFILE to specify a boto profile
  • EC2_INSTANCE_FILTERS to filter the AWS instances returned on various criteria. Extremely detailed API reference for filtering is here

Note that each of these can also be specified in the .ini file.

Important

By default, the ec2.ini file is configured for all Amazon cloud services. You have to turn off the ones you don't want (elasticcache, rds, etc.)

Ref: https://docs.ansible.com/ansible/latest/user_guide/intro_dynamic_inventory.html#inventory-script-example-aws-ec2

Python script

The script proper defines a Python object that manages all of the information received from the AWS API.

The script uses the boto library to interact with the AWS API. The object defines methods for parsing user command line arguments, and implements a number of other methods to do things like ask for a list of nodes, filter nodes by attribute, and store/retrieve information from a cache on disk to prevent everything from going extremely slowly due to slow AWS API responses.

Key methods defined for the object:

The call order of the script, when run, is:

Flags

Retrieved from "https://charlesreid1.com/w/index.php?title=Ansible/Hosts/Dynamic_Inventory&oldid=25363"