mirror of https://github.com/ansible/ansible.git
Core module program flow and glossary (#15355)
* Reformat glossary as a sphinx glossary so that :term: will work. * Add a document decribing program flow for executing modules * Feedback from @docschick * More feedback from docschick for the Program Flow: Modules doc * Changes to address docschick's feedback on the glossary * Add note section for async plugin * make singularpull/15397/head
Toshio Kuratomi
8 years ago
parent
40d0bb7aef
commit
8d60b298a4
@ -0,0 +1,401 @@
|
||||
.. _flow_modules:
|
||||
|
||||
=======
|
||||
Modules
|
||||
=======
|
||||
|
||||
This in-depth dive helps you understand Ansible's program flow to execute
|
||||
modules. It is written for people working on the portions of the Core Ansible
|
||||
Engine that execute a module. Those writing Ansible Modules may also find this
|
||||
in-depth dive to be of interest, but individuals simply using Ansible Modules
|
||||
will not likely find this to be helpful.
|
||||
|
||||
.. _flow_types_of_modules:
|
||||
|
||||
Types of Modules
|
||||
================
|
||||
|
||||
Ansible supports several different types of modules in its code base. Some of
|
||||
these are for backwards compatibility and others are to enable flexibility.
|
||||
|
||||
.. _flow_action_plugins:
|
||||
|
||||
Action Plugins
|
||||
--------------
|
||||
|
||||
Action Plugins look like modules to end users who are writing :term:`playbooks` but
|
||||
they're distinct entities for the purposes of this paper. Action Plugins
|
||||
always execute on the controller and are sometimes able to do all work there
|
||||
(for instance, the debug Action Plugin which prints some text for the user to
|
||||
see or the assert Action Plugin which can test whether several values in
|
||||
a playbook satisfy certain criteria.)
|
||||
|
||||
More often, Action Plugins set up some values on the controller, then invoke an
|
||||
actual module on the managed node that does something with these values. An
|
||||
easy to understand version of this is the :ref:`template Action Plugin
|
||||
<template>`. The :ref:`template Action Plugin <template>` takes values from
|
||||
the user to construct a file in a temporary location on the controller using
|
||||
variables from the playbook environment. It then transfers the temporary file
|
||||
to a temporary file on the remote system. After that, it invokes the
|
||||
:ref:`copy module <copy>` which operates on the remote system to move the file
|
||||
into its final location, sets file permissions, and so on.
|
||||
|
||||
.. _flow_new_style_modules:
|
||||
|
||||
New-style Modules
|
||||
-----------------
|
||||
|
||||
All of the modules that ship with Ansible fall into this category.
|
||||
|
||||
New-style modules have the arguments to the module embedded inside of them in
|
||||
some manner. Non-new-style modules must copy a separate file over to the
|
||||
managed node, which is less efficient as it requires two over-the-wire
|
||||
connections instead of only one.
|
||||
|
||||
.. _flow_python_modules:
|
||||
|
||||
Python
|
||||
^^^^^^
|
||||
|
||||
New-style Python modules use the :ref:`ziploader` framework for constructing
|
||||
modules. All official modules (shipped with Ansible) use either this or the
|
||||
:ref:`powershell module framework <flow_powershell_modules>`.
|
||||
|
||||
These modules use imports from :code:`ansible.module_utils` in order to pull in
|
||||
boilerplate module code, such as argument parsing, formatting of return
|
||||
values as :term:`JSON`, and various file operations.
|
||||
|
||||
.. note:: In Ansible, up to version 2.0.x, the official Python modules used the
|
||||
:ref:`module_replacer` framework. For module authors, :ref:`ziploader` is
|
||||
largely a superset of :ref:`module_replacer` functionality, so you usually
|
||||
do not need to know about one versus the other.
|
||||
|
||||
.. _flow_powershell_modules:
|
||||
|
||||
Powershell
|
||||
^^^^^^^^^^
|
||||
|
||||
New-style powershell modules use the :ref:`module_replacer` framework for
|
||||
constructing modules. These modules get a library of powershell code embedded
|
||||
in them before being sent to the managed node.
|
||||
|
||||
.. _flow_josnargs_modules:
|
||||
|
||||
JSONARGS
|
||||
^^^^^^^^
|
||||
|
||||
Scripts can arrange for an argument string to be placed within them by placing
|
||||
the string ``<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>`` somewhere inside of the
|
||||
file. The module typically sets a variable to that value like this::
|
||||
|
||||
json_arguments = """<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>"""
|
||||
|
||||
Which is expanded as::
|
||||
|
||||
json_arguments = """{"param1": "test's quotes", "param2": "\"To be or not to be\" - Hamlet"}"""
|
||||
|
||||
.. note:: Ansible outputs a :term:`JSON` string with bare quotes. Double quotes are
|
||||
used to quote string values, double quotes inside of string values are
|
||||
backslash escaped, and single quotes may appear unescaped inside of
|
||||
a string value. To use JSONARGS, your scripting language must have a way
|
||||
to handle this type of string. The example uses Python's triple quoted
|
||||
strings to do this. Other scripting languages may have a similar quote
|
||||
character that won't be confused by any quotes in the JSON or it may
|
||||
allow you to define your own start-of-quote and end-of-quote characters.
|
||||
If the language doesn't give you any of these then you'll need to write
|
||||
a :ref:`non-native JSON module <flow_want_json_modules>` or
|
||||
:ref:`Old-style module <flow_old_style_modules>` instead.
|
||||
|
||||
The module typically parses the contents of ``json_arguments`` using a JSON
|
||||
library and then use them as native variables throughout the rest of its code.
|
||||
|
||||
.. _flow_want_json_modules:
|
||||
|
||||
Non-native want JSON modules
|
||||
----------------------------
|
||||
|
||||
If a module has the string ``WANT_JSON`` in it anywhere, Ansible treats
|
||||
it as a non-native module that accepts a filename as its only command line
|
||||
parameter. The filename is for a temporary file containing a :term:`JSON`
|
||||
string containing the module's parameters. The module needs to open the file,
|
||||
read and parse the parameters, operate on the data, and print its return data
|
||||
as a JSON encoded dictionary to stdout before exiting.
|
||||
|
||||
These types of modules are self-contained entities. As of Ansible 2.1, Ansible
|
||||
only modifies them to change a shebang line if present.
|
||||
|
||||
.. seealso:: Examples of Non-native modules written in ruby are in the `Ansible
|
||||
for Rubyists <https://github.com/ansible/ansible-for-rubyists>`_ repository.
|
||||
|
||||
.. _flow_old_style_modules:
|
||||
|
||||
Old-style Modules
|
||||
-----------------
|
||||
|
||||
Old-style modules are similar to
|
||||
:ref:`want JSON modules <flow_want_json_modules>`, except that the file that
|
||||
they take contains ``key=value`` pairs for their parameters instead of
|
||||
:term:`JSON`.
|
||||
|
||||
Ansible decides that a module is old-style when it doesn't have any of the
|
||||
markers that would show that it is one of the other types.
|
||||
|
||||
.. _flow_how_modules_are_executed:
|
||||
|
||||
How modules are executed
|
||||
========================
|
||||
|
||||
When a user uses :program:`ansible` or :program:`ansible-playbook`, they
|
||||
specify a task to execute. The task is usually the name of a module along
|
||||
with several parameters to be passed to the module. Ansible takes these
|
||||
values and processes them in various ways before they are finally executed on
|
||||
the remote machine.
|
||||
|
||||
.. _flow_executor_task_executor:
|
||||
|
||||
executor/task_executor
|
||||
----------------------
|
||||
|
||||
The TaskExecutor receives the module name and parameters that were parsed from
|
||||
the :term:`playbook <playbooks>` (or from the command line in the case of
|
||||
:command:`/usr/bin/ansible`). It uses the name to decide whether it's looking
|
||||
at a module or an :ref:`Action Plugin <flow_action_plugins>`. If it's
|
||||
a module, it loads the :ref:`Normal Action Plugin <flow_normal_action_plugin>`
|
||||
and passes the name, variables, and other information about the task and play
|
||||
to that Action Plugin for further processing.
|
||||
|
||||
.. _flow_normal_action_plugin:
|
||||
|
||||
Normal Action Plugin
|
||||
--------------------
|
||||
|
||||
The ``normal`` Action Plugin executes the module on the remote host. It is
|
||||
the primary coordinator of much of the work to actually execute the module on
|
||||
the managed machine.
|
||||
|
||||
* It takes care of creating a connection to the managed machine by
|
||||
instantiating a Connection class according to the inventory configuration for
|
||||
that host.
|
||||
* It adds any internal Ansible variables to the module's parameters (for
|
||||
instance, the ones that pass along ``no_log`` to the module).
|
||||
* It takes care of creating any temporary files on the remote machine and
|
||||
cleans up afterwards.
|
||||
* It does the actual work of pushing the module and module parameters to the
|
||||
remote host, although the :ref:`module_common <flow_executor_module_common>`
|
||||
code described next does the work of deciding which format those will take.
|
||||
* It handles any special cases regarding modules (for instance, various
|
||||
complications around Windows modules that must have the same names as Python
|
||||
modules, so that internal calling of modules from other Action Plugins work.)
|
||||
|
||||
Much of this functionality comes from the :class:`BaseAction` class,
|
||||
which lives in :file:`plugins/action/__init__.py`. It makes use of Connection
|
||||
and Shell objects to do its work.
|
||||
|
||||
.. note::
|
||||
When :term:`tasks <tasks>` are run with the ``async:`` parameter, Ansible
|
||||
uses the ``async`` Action Plugin instead of the ``normal`` Action Plugin
|
||||
to invoke it. That program flow is currently not documented. Read the
|
||||
source for information on how that works.
|
||||
|
||||
.. _flow_executor_module_common:
|
||||
|
||||
executor/module_common.py
|
||||
-------------------------
|
||||
|
||||
Code in :file:`executor/module_common.py` takes care of assembling the module
|
||||
to be shipped to the managed node. The module is first read in, then examined
|
||||
to determine its type. :ref:`PowerShell <flow_powershell_modules>` and
|
||||
:ref:`JSON-args modules <flow_jsonargs_modules>` are passed through
|
||||
:ref:`Module Replacer <module_replacer>`. New-style
|
||||
:ref:`Python modules <flow_python_modules>` are assembled by :ref:`ziploader`.
|
||||
:ref:`Non-native-want-JSON <flow_want_json_modules>` and
|
||||
:ref:`Old-Style modules <flow_old_style_modules>` aren't touched by either of
|
||||
these and pass through unchanged. After the assembling step, one final
|
||||
modification is made to all modules that have a shebang line. Ansible checks
|
||||
whether the interpreter in the shebang line has a specific path configured via
|
||||
an ``ansible_$X_interpreter`` inventory variable. If it does, Ansible
|
||||
substitutes that path for the interpreter path given in the module. After
|
||||
this Ansible returns the complete module data and the module type to the
|
||||
:ref:`Normal Action <_flow_normal_action_plugin>` which continues execution of
|
||||
the module.
|
||||
|
||||
Next we'll go into some details of the two assembler frameworks.
|
||||
|
||||
.. _module_replacer:
|
||||
|
||||
Module Replacer
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
The Module Replacer is essentially a preprocessor (like the C Preprocessor for
|
||||
those familiar with that language). It does straight substitutions of
|
||||
specific substring patterns in the module file. There are two types of
|
||||
substitutions:
|
||||
|
||||
* Replacements that only happen in the module file. These are public
|
||||
replacement strings that modules can utilize to get helpful boilerplate or
|
||||
access to arguments.
|
||||
|
||||
- :code:`from ansible.module_utils.MOD_LIB_NAME import *` is replaced with the
|
||||
contents of the :file:`ansible/module_utils/MOD_LIB_NAME.py` These should
|
||||
only be used with :ref:`new-style Python modules <flow_python_modules>`.
|
||||
- :code:`#<<INCLUDE_ANSIBLE_MODULE_COMMON>>` is equivalent to
|
||||
:code:`from ansible.module_utils.basic import *` and should also only apply
|
||||
to new-style Python modules.
|
||||
- :code:`# POWERSHELL_COMMON` substitutes the contents of
|
||||
:file:`ansible/module_utils/powershell.ps1`. It should only be used with
|
||||
:ref:`new-style Powershell modules <flow_powershell_modules>`.
|
||||
|
||||
* Replacements that are used by ``ansible.module_utils`` code. These are internal
|
||||
replacement patterns. They may be used internally, in the above public
|
||||
replacements, but shouldn't be used directly by modules.
|
||||
|
||||
- :code:`"<<ANSIBLE_VERSION>>"` is substituted with the Ansible version. In
|
||||
a new-style Python module, it's better to use ``from ansible import
|
||||
__version__`` and then use ``__version__`` instead.
|
||||
- :code:`"<<INCLUDE_ANSIBLE_MODULE_COMPLEX_ARGS>>"` is substituted with
|
||||
a string which is the Python ``repr`` of the :term:`JSON` encoded module
|
||||
parameters. Using ``repr`` on the JSON string makes it safe to embed in
|
||||
a Python file. In :ref:`new-style Python modules <flow_python_modules>`
|
||||
under :ref:`ziploader` this is passed in via an environment variable
|
||||
instead.
|
||||
- :code:`<<SELINUX_SPECIAL_FILESYSTEMS>>` substitutes a string which is
|
||||
a comma separated list of file systems which have a file system dependent
|
||||
security context in SELinux. In new-style Python modules, this is found
|
||||
by looking up ``SELINUX_SPECIAL_FS`` from the
|
||||
:envvar:`ANSIBLE_MODULE_CONSTANTS` environment variable. See the
|
||||
:ref:`ziploader` documentation for details.
|
||||
- :code:`<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>` substitutes the module
|
||||
parameters as a JSON string. Care must be taken to properly quote the
|
||||
string as JSON data may contain quotes. This pattern is not substituted
|
||||
in new-style Python modules as they can get the module parameters via the
|
||||
environment variable.
|
||||
- the string :code:`syslog.LOG_USER` is replaced wherever it occurs with the
|
||||
value of ``syslog_facility`` from the :file:`ansible.cfg` or any
|
||||
``ansible_syslog_facility`` inventory variable that applies to this host. In
|
||||
new-style Python modules, you can get the value of the ``syslog_facility``
|
||||
by looking up ``SYSLOG_FACILITY`` in the :envvar:`ANSIBLE_MODULE_CONSTANTS`
|
||||
environment variable. See the :ref:`ziploader` documentation for details.
|
||||
|
||||
.. _ziploader:
|
||||
|
||||
ziploader
|
||||
^^^^^^^^^
|
||||
|
||||
Ziploader differs from :ref:`module_replacer` in that it uses real Python
|
||||
imports of things in module_utils instead of merely preprocessing the module.
|
||||
It does this by constructing a zipfile--which includes the module file, files
|
||||
in :file:`ansible/module_utils` that are imported by the module, and some
|
||||
boilerplate to pass in the constants. The zipfile is then Base64 encoded and
|
||||
wrapped in a small Python script which unzips the file on the managed node and
|
||||
then invokes Python on the file. (Ansible wraps the zipfile in the Python
|
||||
script so that pipelining will work.)
|
||||
|
||||
In ziploader, any imports of Python modules from the ``ansible.module_utils``
|
||||
package trigger inclusion of that Python file into the zipfile. Instances of
|
||||
:code:`#<<INCLUDE_ANSIBLE_MODULE_COMMON>>` in the module are turned into
|
||||
:code:`from ansible.module_utils.basic import *` and
|
||||
:file:`ansible/module-utils/basic.py` is then included in the zipfile. Files
|
||||
that are included from module_utils are themselves scanned for imports of other
|
||||
Python modules from module_utils to be included in the zipfile as well.
|
||||
|
||||
.. warning::
|
||||
At present, there are two caveats to how ziploader determines other files
|
||||
to import:
|
||||
|
||||
* Ziploader cannot determine whether an import should be included if it is
|
||||
a relative import. Always use an absolute import that has
|
||||
``ansible.module_utils`` in it to allow ziploader to determine that the
|
||||
file should be included.
|
||||
* Ziploader does not include Python packages (directories with
|
||||
:file:`__init__.py`` in them). Ziploader only works on :file:`*.py`
|
||||
files that are directly in the :file:`ansible/module_utils` directory.
|
||||
|
||||
.. _flow_passing_module_args:
|
||||
|
||||
Passing args
|
||||
~~~~~~~~~~~~
|
||||
|
||||
In :ref:`module_replacer`, module arguments are turned into a JSON-ified
|
||||
string and substituted into the combined module file. In :ref:`ziploader`,
|
||||
the JSON-ified string is placed in the the :envvar:`ANSIBLE_MODULE_ARGS`
|
||||
environment variable. When :code:`ansible.module_utils.basic` is imported,
|
||||
it places this string in the global variable
|
||||
``ansible.module_utils.basic.MODULE_COMPLEX_ARGS`` and removes it from the
|
||||
environment. Modules should not access this variable directly. Instead, they
|
||||
should instantiate an :class:`AnsibleModule()` and use
|
||||
:meth:`AnsibleModule.params` to access the parsed version of the arguments.
|
||||
|
||||
.. _flow_passing_module_constants:
|
||||
|
||||
Passing constants
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
Currently, there are three constants passed from the controller to the modules:
|
||||
``ANSIBLE_VERSION``, ``SELINUX_SPECIAL_FS``, and ``SYSLOG_FACILITY``. In
|
||||
:ref:`module_replacer`, ``ANSIBLE_VERSION`` and ``SELINUX_SPECIAL_FS`` were
|
||||
substituted into the global variables
|
||||
:code:`ansible.module_utils.basic.ANSIBLE_VERSION` and
|
||||
:code:`ansible.module_utils.basic.SELINUX_SPECIAL_FS`. ``SYSLOG_FACILITY`` didn't
|
||||
get placed into a variable. Instead, any occurrences of the string
|
||||
``syslog.LOG_USER`` in the combined module file were replaced with ``syslog.``
|
||||
followed by the string contained in ``SYSLOG_FACILITY``. All of these have
|
||||
changed in :ref:`ziploader`.
|
||||
|
||||
The Ansible verison can now be used by a module by importing ``__version__``
|
||||
from ansible::
|
||||
|
||||
from ansible import __version__
|
||||
module.exit_json({'msg': 'module invoked by ansible %s' % __version__})
|
||||
|
||||
For now, :code:`ANSIBLE_VERSION` is also available at its old location inside of
|
||||
``ansible.module_utils.basic``, but that will eventually be removed.
|
||||
|
||||
``SELINUX_SPECIAL_FS`` and ``SYSLOG_FACILITY`` have changed much more.
|
||||
:ref:`ziploader` passes these as another JSON-ified string inside of the
|
||||
:envvar:`ANSIBLE_MODULE_CONSTANTS` environment variable. When
|
||||
``ansible.module_utils.basic`` is imported, it places this string in the global
|
||||
variable :code:`ansible.module_utils.basic.MODULE_CONSTANTS` and removes it from
|
||||
the environment. The constants are parsed when an :class:`AnsibleModule` is
|
||||
instantiated. Modules shouldn't access any of those directly. Instead, they
|
||||
should instantiate an :class:`AnsibleModule` and use
|
||||
:attr:`AnsibleModule.constants` to access the parsed version of these values.
|
||||
|
||||
Unlike the ``ANSIBLE_ARGS`` and ``ANSIBLE_VERSION``, where some efforts were
|
||||
made to keep the old backwards compatible globals available, these two
|
||||
constants are not available at their old names. This is a combination of the
|
||||
degree to which these are internal to the needs of ``module_utils.basic`` and,
|
||||
in the case of ``SYSLOG_FACILITY``, how hacky and unsafe the previous
|
||||
implementation was.
|
||||
|
||||
Porting code from the :ref:`module_replacer` method of getting
|
||||
``SYSLOG_FACILITY`` to the new one is a little more tricky than the other
|
||||
constants and args, due to just how hacky the old way was. Here's an example
|
||||
of using it in the new way::
|
||||
|
||||
import syslog
|
||||
facility_name = module.constants.get('SYSLOG_FACILITY')
|
||||
facility = getattr(syslog, facility_name)
|
||||
syslog.openlog(str(module), 0, facility)
|
||||
|
||||
.. _flow_special_considerations:
|
||||
|
||||
Special Considerations
|
||||
----------------------
|
||||
|
||||
.. _flow_pipelining:
|
||||
|
||||
Pipelining
|
||||
^^^^^^^^^^
|
||||
|
||||
Ansible can transfer a module to a remote machine in one of two ways:
|
||||
|
||||
* it can write out the module to a temporary file on the remote host and then
|
||||
use a second connection to the remote host to execute it with the
|
||||
interpreter that the module needs
|
||||
* or it can use what's known as pipelining to execute the module by piping it
|
||||
into the remote interpreter's stdin.
|
||||
|
||||
Pipelining only works with modules written in Python at this time because
|
||||
Ansible only knows that Python supports this mode of operation. Supporting
|
||||
pipelining means that whatever format the module payload takes before being
|
||||
sent over the wire must be executable by Python via stdin.
|
||||
Loading…
Reference in New Issue