Ansible/Full Stack Playbook: Difference between revisions

From charlesreid1
 
(5 intermediate revisions by the same user not shown)
Line 10: Line 10:
=Setup=
=Setup=


==Vagrant multi-machine setup==
See [[Ansible/Full Stack Playbook/Vagrant Setup]] for details of setting up Vagrant to test out this configuration
 
Here we walk through how to get set up with [[Vagrant]] before writing and testing the playbook.
 
Note: before running any vagrant boxes, destroy and clean up prior boxes via
 
<pre>
vagrant destroy ---force
</pre>
 
===Vagrantfile===
 
Create a Vagrantfile with 3 hosts:
 
'''<code>Vagrantfile</code>'''
 
<pre>
VAGRANTFILE_API_VERSION = "2"
 
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
 
  # Use the same key for each machine
  config.ssh.insert_key = false
 
  config.vm.define "vagrant1" do |vagrant1|
    vagrant1.vm.box = "ubuntu/xenial64"
    vagrant1.vm.network "forwarded_port", guest: 80, host: 8080
    vagrant1.vm.network "forwarded_port", guest: 443, host: 8443
  end
  config.vm.define "vagrant2" do |vagrant2|
    vagrant2.vm.box = "ubuntu/xenial64"
    vagrant2.vm.network "forwarded_port", guest: 80, host: 8081
    vagrant2.vm.network "forwarded_port", guest: 443, host: 8444
  end
  config.vm.define "vagrant3" do |vagrant3|
    vagrant3.vm.box = "ubuntu/xenial64"
    vagrant3.vm.network "forwarded_port", guest: 80, host: 8082
    vagrant3.vm.network "forwarded_port", guest: 443, host: 8445
  end
end
</pre>
 
Note that without <code>config.ssh.insert_key=false</code> each machine would use its own SSH key, which would be a bit of a headache. With this directive, we can define a single SSH key in our ansible config file.
 
===Ansible config file===
 
Now the <code>ansible.cfg</code> file should be modified to configure Ansible. Most important is the location of the private key:
 
<pre>
[defaults]
inventory = inventory
remote_user = vagrant
private_key_file = ~/.vagrant.d/insecure_private_key
host_key_checking = False
</pre>
 
===Run vagrant===
 
Run the vagrant machines with
 
<pre>
vagrant up
</pre>
 
See details about SSH ports using
 
<pre>
vagrant ssh-config
</pre>
 
which will output something like this
 
<pre>
Host vagrant1
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/lorin/.vagrant.d/insecure_private_key
  IdentitiesOnly yes
  LogLevel FATAL
 
Host vagrant2
  HostName 127.0.0.1
  User vagrant
  Port 2200
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/lorin/.vagrant.d/insecure_private_key
  IdentitiesOnly yes
  LogLevel FATAL
 
Host vagrant3
  HostName 127.0.0.1
  User vagrant
  Port 2201
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/lorin/.vagrant.d/insecure_private_key
  IdentitiesOnly yes
  LogLevel FATAL
</pre>
 
===Create Ansible inventory file===
 
Once we know the SSH port for each machine, we can create an inventory file.
 
A basic <code>playbook/hosts</code> file would contain:
 
<pre>
vagrant1 ansible_host=127.0.0.1 ansible_port=2222
vagrant2 ansible_host=127.0.0.1 ansible_port=2200
vagrant3 ansible_host=127.0.0.1 ansible_port=2201
</pre>
 
This, together with the playbook, tells Ansible how to reach and how to connect to each of the remote hosts. Now we can run a test command on the machines with Ansible:
 
<pre>
ansible vagrant2 -a "ip addr show dev eth0"
</pre>
 
===Group the Ansible hosts file===
 
As the hosts file gets more complicated, we may want to perform actions only on a group of machines. Toward this purpose we can group the machines under <code>[headings]</code> that indicate they are part of a group.
 
If we added many more hosts to the hosts file, we would want to group the Vagrant machines, so the host file would look like this:
 
<pre>
ontario.example.com
newhampshire.example.com
maryland.example.com
virginia.example.com
newyork.example.com
quebec.example.com
rhodeisland.example.com
 
[vagrant]
vagrant1 ansible_host=127.0.0.1 ansible_port=2222
vagrant2 ansible_host=127.0.0.1 ansible_port=2200
vagrant3 ansible_host=127.0.0.1 ansible_port=2201
</pre>
 
===Note on inventory files===
 
Inventory files can define a number of parameters for each host. Above, we define the host and port, but other variables are also available:
 
* ansible_host
* ansible_port
* ansible_user
* ansible_password
* ansible_private-key_file
* ansible_shell_type
* ansible_python_interpreter
 
(But apparently only ansible_port, ansible_user, ansible_private_key_file,and ansible_shell_type can be changed in Ansible config file???)
 
===Next steps===
 
We will modify this inventory file as needed for the actual full stack application, but this gets you going with a basic multi-machine Vagrantfile.


=Full Stack Playbook=
=Full Stack Playbook=
Line 286: Line 124:
* Postgres primary must have hostname, port, username, password of replica Postgres server (production only)
* Postgres primary must have hostname, port, username, password of replica Postgres server (production only)


It makes the most sense to provide this information for each different environment (vagrant, staging, and production), so here is what that looks like:
This information can be properly distributed by using groups, and assigning different variable values for each different group.
 
Specifically, things break down best by environment (vagrant, staging, and production).
 
Here's what the Vagrant hosts file looks like, grouping by environment:


<pre>
<pre>
Line 301: Line 143:
rabbitmq_host=pennsylvania.example.com
rabbitmq_host=pennsylvania.example.com
rabbitmq_port=5672
rabbitmq_port=5672


[staging:vars]
[staging:vars]
db_primary_host=quebec.example.com
db_primary_host=orangered.example.com
db_primary_port=5432
db_primary_port=5432
db_name=widget_staging
db_name=widget_staging
db_user=widgetuser
db_user=widgetuser
db_password=L@4Ryz8cRUXedj
db_password=L@4Ryz8cRUXedj
rabbitmq_host=quebec.example.com
rabbitmq_host=redblue.example.com
rabbitmq_port=5672
rabbitmq_port=5672


Line 322: Line 163:
</pre>
</pre>


This is a section of the Ansible hosts file.
=Flags=


=Flags=
{{AnsibleFlag}}
 
[[Category:HTTPS]]
[[Category:SSL]]


[[Category:Web Server]]
[[Category:Web Server]]
Line 332: Line 176:
[[Category:RabbitMQ]]
[[Category:RabbitMQ]]
[[Category:Postgres]]
[[Category:Postgres]]
[[Category:Ansible]]
[[Category:Infrastructure]]
[[Category:Python]]
[[Category:HTTPS]]
[[Category:SSL]]

Latest revision as of 19:14, 7 December 2018

Main article: Ansible/Playbooks

This page covers an Ansible playbook for a full stack example. This full stack example will run the following services:

  • Django web server
  • Celery task queue
  • RabbitMQ message queue
  • Postgresql for data storage

Setup

See Ansible/Full Stack Playbook/Vagrant Setup for details of setting up Vagrant to test out this configuration

Full Stack Playbook

Overview of Full Stack

Services

For this particular full stack deployment, we have the following services:

  • Django app + http server
  • Nginx web server
  • Celery task queue
  • RabbitMQ (Celery backend)
  • Postgres (persistent store)

Environments

We have three environments to deploy to:

  • vagrant (local testing)
  • staging (testing)
  • production

Architecture

Here is the architecture used for the above apps:

  • Web application is run on multiple hosts for better performance, with a load balancer in front
  • Task queue servers are run on multiple hosts for better performance
  • Celery, RabbitMQ, Postgres all on separate servers
  • 2 Postgres hosts - primary and replica

That is a total of 10 hosts (1 load balancer, 3 web servers, 3 task queues, 1 mq server, 2 database servers).

In the production environment: 10 separate hosts

In the staging environment: use fewer hosts (only 2 hosts: web server and task queue on one host, rabbitmq and postgres on another)

In the vagrant environment: use 3 servers (1 for web app, 1 for task queue, and 1 for postgres)

Ansible hosts file

Main article: Ansible/Hosts

Note that this initial hosts file (below) can be streamlined using suggestions from Ansible/Groups, Ansible/Group Variables, and Ansible/Host Naming, so consider it a first pass.

Using the information given above about the architecture, we have the following hosts file for our Django app: 

[production]
delaware.example.com
georgia.example.com
maryland.example.com
newhampshire.example.com
newjersey.example.com
newyork.example.com
northcarolina.example.com
pennsylvania.example.com
rhodeisland.example.com
virginia.example.com

[staging]
redblue.example.com
orangered.example.com

[vagrant]
vagrant1 ansible_host=127.0.0.1 ansible_port=2222
vagrant2 ansible_host=127.0.0.1 ansible_port=2200
vagrant3 ansible_host=127.0.0.1 ansible_port=2201

[lb]
delaware.example.com

[web]
georgia.example.com
newhampshire.example.com
newjersey.example.com
redblue.example.com
vagrant1

[task]
newyork.example.com
northcarolina.example.com
maryland.example.com
redblue.example.com
vagrant2

[rabbitmq]
pennsylvania.example.com
orangered.example.com
vagrant3

[db]
rhodeisland.example.com
virginia.example.com
orangered.example.com
vagrant3

Note that we make the vagrant machines part of the vagrant group when their behavioral inventory parameters are specified, but we can later add them to other groups by adding their short name (only) to the group.

In this way, we've defined which machines are in each of the 3 environments (production, staging, vagrant) and which one runs which service.

Passing credentials via the host file

Now that we're deploying these services, we have more details to take care of:

  • Web servers must have hostname, port, username, password of Postgres, and name of database
  • Task queues must have hostname, port, username, password of Postgres, and name of database
  • Web servers must have hostname, port of RabbitMQ server
  • task queues must have hostname, port of RabbitMQ server
  • Postgres primary must have hostname, port, username, password of replica Postgres server (production only)

This information can be properly distributed by using groups, and assigning different variable values for each different group.

Specifically, things break down best by environment (vagrant, staging, and production).

Here's what the Vagrant hosts file looks like, grouping by environment: 

[all:vars]
ntp_server=ntp.ubuntu.com

[production:vars]
db_primary_host=rhodeisland.example.com
db_primary_port=5432
db_replica_host=virginia.example.com
db_name=widget_production
db_user=widgetuser
db_password=pFmMxcyD;Fc6)6
rabbitmq_host=pennsylvania.example.com
rabbitmq_port=5672

[staging:vars]
db_primary_host=orangered.example.com
db_primary_port=5432
db_name=widget_staging
db_user=widgetuser
db_password=L@4Ryz8cRUXedj
rabbitmq_host=redblue.example.com
rabbitmq_port=5672

[vagrant:vars]
db_primary_host=vagrant3
db_primary_port=5432
db_name=widget_vagrant
db_user=widgetuser
db_password=password
rabbitmq_host=vagrant3
rabbitmq_port=5672

Flags


Defcon.png
Ansible
Ansible is a Python library for managing IT and compute infrastructure.

Ansible  · Category:Ansible


Ansible Configuration

Config file: Ansible/Configuration

Secrets and Sensitive Info: Ansible/Security  · Ansible/Vaults


Ansible Playbooks

Basics: Ansible/Playbooks

Examples: Ansible/Nginx Playbook (simple)  · Ansible/Full Stack Playbook (advanced)


Ansible Hosts and Inventory Control

All topics in Ansible Hosts: Ansible/Hosts

Splitting large host files into subfiles: Ansible/Splitting Hosts File

Automatically naming hosts: Ansible/Host Naming

Grouping Hosts: Ansible/Groups

Assigning Variables to Groups: Ansible/Group Variables

Roles: Ansible/Roles


Ansible and Vagrant (Local)

Ansible/Vagrant  · Ansible/Vagrant/Static Inventory  · Ansible/Vagrant/Dynamic Inventory


Ansible and EC2 (Cloud)

Ansible/EC2  · Ansible/EC2/Static Inventory  · Ansible/EC2/Dynamic Inventory


Best Practices

Ansible/Directory Layout  · Ansible/Directory Layout/Details

Ansible/Separate Playbooks by Role

Ansible/Tag Inventory with Environment

Ansible/Group by Distribution

Ansible/Variables and Vaults


Ansible Galaxy

Ansible/Galaxy


Flags  · Template:AnsibleFlag  · e



Retrieved from "https://charlesreid1.com/w/index.php?title=Ansible/Full_Stack_Playbook&oldid=25626"