@ -9,9 +9,18 @@ Tutorial: Rolling Upgrades for Continuous Delivery
Introduction
````````````
Continuous Delivery is the concept of frequently delivering updates to your software application. Some Ansible users are deploying updates to their customers on an hourly or even more frequent basis. To achieve this, you need tools to be able to quickly apply those updates in a zero-downtime way.
Continuous Delivery is the concept of frequently delivering updates to your software application.
This document describes in detail how to achieve this goal, using one of Ansible's most complete example playbooks as a template: lamp_haproxy. This example uses a lot of Ansible features: roles, templates, and group variables, and it also comes with an orchestration playbook that can do zero-downtime rolling upgrades of the web application stack.
The idea is that my updating more often, you not only have to wait for specific timed period, but your organization
gets better at the process of responding to change.
Some Ansible users are deploying updates to their end users on an hourly or even more frequent basis.
To achieve this, you need tools to be able to quickly apply those updates in a zero-downtime way.
This document describes in detail how to achieve this goal, using one of Ansible's most complete example
playbooks as a template: lamp_haproxy. This example uses a lot of Ansible features: roles, templates,
and group variables, and it also comes with an orchestration playbook that can do zero-downtime
rolling upgrades of the web application stack.
..note::
@ -20,14 +29,16 @@ This document describes in detail how to achieve this goal, using one of Ansible
The playbooks deploy Apache, PHP, MySQL, Nagios, and HAProxy to a CentOS-based set of servers.
We're not going to cover how to run these playbooks here. Read the README along with the example for that information. Instead, we're going to take a close look at every part of the playbook and describe what it does.
We're not going to cover how to run these playbooks here. Read the included README in the github project along with the
example for that information. Instead, we're going to take a close look at every part of the playbook and describe what it does.
.._lamp_deployment:
Site Deployment
```````````````
Let's start with ``site.yml``. This is our site-wide deployment playbook. It can be used to initially deploy the site, as well as push updates to all of the servers::
Let's start with ``site.yml``. This is our site-wide deployment playbook. It can be used to initially deploy the site, as well
as push updates to all of the servers::
---
# This playbook deploys the whole application stack in this site.
@ -71,33 +82,46 @@ Let's start with ``site.yml``. This is our site-wide deployment playbook. It can
If you're not familiar with terms like playbooks and plays, you should review :doc:`playbooks`.
In this playbook we have 5 plays. The first one targets ``all`` hosts and applies the ``common`` role to all of the hosts. This is for site-wide things like Yum repository configuration, firewall configuration, and anything else that needs to apply to all of the servers.
In this playbook we have 5 plays. The first one targets ``all`` hosts and applies the ``common`` role to all of the hosts.
This is for site-wide things like yum repository configuration, firewall configuration, and anything else that needs to apply to all of the servers.
The next four plays run against specific host groups and apply specific roles to those servers. Along with the roles for Nagios monitoring, the database, and the web application, We've implemented a ``base-apache`` role that installs and configures a basic Apache setup. This is used by both the sample web application and the Nagios hosts.
The next four plays run against specific host groups and apply specific roles to those servers.
Along with the roles for Nagios monitoring, the database, and the web application, we've implemented a
``base-apache`` role that installs and configures a basic Apache setup. This is used by both the
sample web application and the Nagios hosts.
.._lamp_roles:
Reusable Content: Roles
```````````````````````
By now you should have a bit of understanding about roles and how they work in Ansible. Roles are a way to organize content: tasks, handlers, templates, and files, into reusable components.
By now you should have a bit of understanding about roles and how they work in Ansible. Roles are a way to organize
content: tasks, handlers, templates, and files, into reusable components.
This example has six roles: ``common``, ``base-apache``, ``db``, ``haproxy``, ``nagios``, and ``web``. How you organize your roles is up to you and your application, but most sites will have one or more common roles that are applied to all systems, and then a series of application-specific roles that install and configure particular parts of the site.
This example has six roles: ``common``, ``base-apache``, ``db``, ``haproxy``, ``nagios``, and ``web``. How you organize
your roles is up to you and your application, but most sites will have one or more common roles that are applied to
all systems, and then a series of application-specific roles that install and configure particular parts of the site.
Roles can have variables and dependencies, and you can pass in parameters to roles to modify their behavior. You can read more about roles in the :doc:`playbooks_roles` section.
Roles can have variables and dependencies, and you can pass in parameters to roles to modify their behavior.
You can read more about roles in the :doc:`playbooks_roles` section.
.._lamp_group_variables:
Configuration: Group Variables
``````````````````````````````
Group variables are variables that are applied to groups of servers. They can be used in templates and in playbooks to customize behavior and to provide easily-changed settings and parameters. They are stored in a directory called ``group_vars`` in the same location as your inventory. Here is lamp_haproxy's ``group_vars/all`` file. As you might expect, these variables are applied to all of the machines in your inventory::
Group variables are variables that are applied to groups of servers. They can be used in templates and in
playbooks to customize behavior and to provide easily-changed settings and parameters. They are stored in
a directory called ``group_vars`` in the same location as your inventory.
Here is lamp_haproxy's ``group_vars/all`` file. As you might expect, these variables are applied to all of the machines in your inventory::
---
httpd_port: 80
ntpserver: 192.168.1.2
This is a YAML file, and you can create lists and dictionaries for more complex variable structures. In this case, we are just setting two variables, one for the port for the web server, and one for the NTP server that our machiens should use for time synchronization.
This is a YAML file, and you can create lists and dictionaries for more complex variable structures.
In this case, we are just setting two variables, one for the port for the web server, and one for the
NTP server that our machiens should use for time synchronization.
Here's another group variables file. This is ``group_vars/dbservers`` which applies to the hosts in the ``dbservers`` group::
@ -132,13 +156,17 @@ You can also use these variables in templates, like this, in ``roles/common/temp
keys /etc/ntp/keys
You can see that the variable substitution syntax of {{ and }} is the same for both templates and variables. The syntax inside the curly braces is Jinja2, and you can do all sorts of operations and apply different filters to the data inside. In templates, you can also use for loops and if statements to handle more complex situations, like this, in ``roles/common/templates/iptables.j2``::
You can see that the variable substitution syntax of {{ and }} is the same for both templates and variables. The syntax
inside the curly braces is Jinja2, and you can do all sorts of operations and apply different filters to the
data inside. In templates, you can also use for loops and if statements to handle more complex situations,
like this, in ``roles/common/templates/iptables.j2``::
{% if inventory_hostname in groups['dbservers'] %}
-A INPUT -p tcp --dport 3306 -j ACCEPT
{% endif %}
This is testing to see if the inventory name of the machine we're currently operating on (``inventory_hostname``) exists in the inventory group ``dbservers``. If so, that machine will get an iptables ACCEPT line for port 3306.
This is testing to see if the inventory name of the machine we're currently operating on (``inventory_hostname``)
exists in the inventory group ``dbservers``. If so, that machine will get an iptables ACCEPT line for port 3306.
Here's another example, from the same template::
@ -146,16 +174,22 @@ Here's another example, from the same template::
This loops over all of the hosts in the group called ``monitoring``, and adds an ACCEPT line for each monitoring hosts's default IPV4 address to the current machine's iptables configuration, so that Nagios can monitor those hosts.
This loops over all of the hosts in the group called ``monitoring``, and adds an ACCEPT line for
each monitoring hosts's default IPV4 address to the current machine's iptables configuration, so that Nagios can monitor those hosts.
You can learn a lot more about Jinja2 and its capabilities `here <http://jinja.pocoo.org/docs/>`_, and you can read more about Ansible variables in general in the :doc:`playbooks_variables` section.
You can learn a lot more about Jinja2 and its capabilities `here <http://jinja.pocoo.org/docs/>`_, and you
can read more about Ansible variables in general in the :doc:`playbooks_variables` section.
.._lamp_rolling_upgrade:
The Rolling Upgrade
```````````````````
Now you have a fully-deployed site with web servers, a load balancer, and monitoring. How do you update it? This is where Ansible's orchestration features come into play. Ansible has the capability to do operations on multi-tier applications in a coordinated way, making it easy to orchestrate a sophisticated zero-downtime rolling upgrade of our web application. This is implemented in a separate playbook, called ``rolling_upgrade.yml``.
Now you have a fully-deployed site with web servers, a load balancer, and monitoring. How do you update it? This is where Ansible's
orchestration features come into play. While some applications use the term 'orchestration' to mean basic ordering or command-blasting, Ansible
referes to orchestration as 'conducting machines like an orchestra', and has a pretty sophisticated engine for it.
Ansible has the capability to do operations on multi-tier applications in a coordinated way, making it easy to orchestrate a sophisticated zero-downtime rolling upgrade of our web application. This is implemented in a separate playbook, called ``rolling_upgrade.yml``.
Looking at the playbook, you can see it is made up of two plays. The first play is very simple and looks like this::
@ -189,6 +223,8 @@ The ``pre_tasks`` keyword just lets you list tasks to run before the roles are c
The ``delegate_to`` and ``with_items`` arguments, used together, cause Ansible to loop over each monitoring server and load balancer, and perform that operation (delegate that operation) on the monitoring or load balancing server, "on behalf" of the webserver. In programming terms, the outer loop is the list of web servers, and the inner loop is the list of monitoring servers.
Note that the HAProxy step looks a little complicated. We're using HAProxy in this example because it's freely available, though if you have (for instance) an F5 or Netscaler in your infrastructure (or maybe you have an AWS Elastic IP setup?), you can use modules included in core Ansible to communicate with them instead. You might also wish to use other monitoring modules instead of nagios, but this just shows the main goal of the 'pre tasks' section -- take the server out of monitoring, and take it out of rotation.
The next step simply re-applies the proper roles to the web servers. This will cause any configuration management declarations in ``web`` and ``base-apache`` roles to be applied to the web servers, including an update of the web application code itself. We don't have to do it this way--we could instead just purely update the web application, but this is a good example of how roles can be used to reuse tasks::
roles:
@ -209,14 +245,16 @@ Finally, in the ``post_tasks`` section, we reverse the changes to the Nagios con
delegate_to: "{{ item }}"
with_items: groups.monitoring
Again, if you were using a Netscaler or F5 or Elastic Load Balancer, you would just substitute in the appropriate modules instead.
.._lamp_end_notes:
Managing Load Balancers
```````````````````````
Managing Other Load Balancers
`````````````````````````````
In this example, we use the simple HAProxy load balancer to front-end the web servers. It's easy to configure and easy to manage. Ansible has built-in support for a variety of other load balancers like Citrix NetScaler, F5 BigIP, Amazon Elastic Load Balancers, and more. See the :doc:`modules` documentation for more information.
In this example, we use the simple HAProxy load balancer to front-end the web servers. It's easy to configure and easy to manage. As we have mentioned, Ansible has built-in support for a variety of other load balancers like Citrix NetScaler, F5 BigIP, Amazon Elastic Load Balancers, and more. See the :doc:`modules` documentation for more information.
For other load balancers, you may need to send shell commands to them (like we do for HAProxy above), or call an API, if your load balancer exposes one. For the load balancers for which Ansible has modules, you may want to run them as a ``local_action`` if they contact an API. You can read more about local actions in the :doc:`playbooks_delegation` section.
For other load balancers, you may need to send shell commands to them (like we do for HAProxy above), or call an API, if your load balancer exposes one. For the load balancers for which Ansible has modules, you may want to run them as a ``local_action`` if they contact an API. You can read more about local actions in the :doc:`playbooks_delegation` section. Should you develop anything interesting for some hardware where there is not a core module, it might make for a good module for core inclusion!
Now that you have an automated way to deploy updates to your application, how do you tie it all together? A lot of organizations use a continuous integration tool like `Jenkins <http://jenkins-ci.org/>`_ or `Atlassian Bamboo <https://www.atlassian.com/software/bamboo>`_ to tie the development, test, release, and deploy steps together. You may also want to use a tool like `Gerrit <https://code.google.com/p/gerrit/>`_ to add a code review step to commits to either the application code itself, or to your Ansible playbooks, or both.
You can easily trigger playbook runs using the ``ansible-playbook`` command line tool, or, if you're using AnsibleWorks AWX, the ``awx-cli`` or the built-in REST API.
Depending on your environment, you might be deploying continuously to a test environment, running an integration test battery against that environment, and then deploying automatically into production. Or you could keep it simple and just use the rolling-update for on-demand deployment into test or production specifically. This is all up to you.
For integration with Continuous Integration systems, you can easily trigger playbook runs using the ``ansible-playbook`` command line tool, or, if you're using AnsibleWorks AWX, the ``awx-cli`` or the built-in REST API. (The AWX-cli command 'joblaunch' will spawn a remote job over the REST API and is pretty slick).
That's it! This should give you a good idea of how to structure a multi-tier application with Ansible, and orchestrate operations upon that app, with the eventual goal of continuous delivery to your customers. You could extend the idea of the rolling upgrade to lots of different parts of the app; maybe add front-end web servers along with application servers, for instance, or replace the SQL database with something like MongoDB or Riak. Ansible gives you the capability to easily manage complicated environments and automate common operations.
This should give you a good idea of how to structure a multi-tier application with Ansible, and orchestrate operations upon that app, with the eventual goal of continuous delivery to your customers. You could extend the idea of the rolling upgrade to lots of different parts of the app; maybe add front-end web servers along with application servers, for instance, or replace the SQL database with something like MongoDB or Riak. Ansible gives you the capability to easily manage complicated environments and automate common operations.
If you need help or if you have questions, stop by the mailing list or the IRC channel, or email us at info@ansibleworks.com.
@ -242,4 +282,6 @@ If you need help or if you have questions, stop by the mailing list or the IRC c
Michael DeHaan
11 years ago