mirror of https://github.com/ansible/ansible.git
Core module program flow and glossary (#15355)
* Reformat glossary as a sphinx glossary so that :term: will work. * Add a document decribing program flow for executing modules * Feedback from @docschick * More feedback from docschick for the Program Flow: Modules doc * Changes to address docschick's feedback on the glossary * Add note section for async plugin * make singularpull/15397/head
parent
40d0bb7aef
commit
8d60b298a4
@ -0,0 +1,401 @@
|
|||||||
|
.. _flow_modules:
|
||||||
|
|
||||||
|
=======
|
||||||
|
Modules
|
||||||
|
=======
|
||||||
|
|
||||||
|
This in-depth dive helps you understand Ansible's program flow to execute
|
||||||
|
modules. It is written for people working on the portions of the Core Ansible
|
||||||
|
Engine that execute a module. Those writing Ansible Modules may also find this
|
||||||
|
in-depth dive to be of interest, but individuals simply using Ansible Modules
|
||||||
|
will not likely find this to be helpful.
|
||||||
|
|
||||||
|
.. _flow_types_of_modules:
|
||||||
|
|
||||||
|
Types of Modules
|
||||||
|
================
|
||||||
|
|
||||||
|
Ansible supports several different types of modules in its code base. Some of
|
||||||
|
these are for backwards compatibility and others are to enable flexibility.
|
||||||
|
|
||||||
|
.. _flow_action_plugins:
|
||||||
|
|
||||||
|
Action Plugins
|
||||||
|
--------------
|
||||||
|
|
||||||
|
Action Plugins look like modules to end users who are writing :term:`playbooks` but
|
||||||
|
they're distinct entities for the purposes of this paper. Action Plugins
|
||||||
|
always execute on the controller and are sometimes able to do all work there
|
||||||
|
(for instance, the debug Action Plugin which prints some text for the user to
|
||||||
|
see or the assert Action Plugin which can test whether several values in
|
||||||
|
a playbook satisfy certain criteria.)
|
||||||
|
|
||||||
|
More often, Action Plugins set up some values on the controller, then invoke an
|
||||||
|
actual module on the managed node that does something with these values. An
|
||||||
|
easy to understand version of this is the :ref:`template Action Plugin
|
||||||
|
<template>`. The :ref:`template Action Plugin <template>` takes values from
|
||||||
|
the user to construct a file in a temporary location on the controller using
|
||||||
|
variables from the playbook environment. It then transfers the temporary file
|
||||||
|
to a temporary file on the remote system. After that, it invokes the
|
||||||
|
:ref:`copy module <copy>` which operates on the remote system to move the file
|
||||||
|
into its final location, sets file permissions, and so on.
|
||||||
|
|
||||||
|
.. _flow_new_style_modules:
|
||||||
|
|
||||||
|
New-style Modules
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
All of the modules that ship with Ansible fall into this category.
|
||||||
|
|
||||||
|
New-style modules have the arguments to the module embedded inside of them in
|
||||||
|
some manner. Non-new-style modules must copy a separate file over to the
|
||||||
|
managed node, which is less efficient as it requires two over-the-wire
|
||||||
|
connections instead of only one.
|
||||||
|
|
||||||
|
.. _flow_python_modules:
|
||||||
|
|
||||||
|
Python
|
||||||
|
^^^^^^
|
||||||
|
|
||||||
|
New-style Python modules use the :ref:`ziploader` framework for constructing
|
||||||
|
modules. All official modules (shipped with Ansible) use either this or the
|
||||||
|
:ref:`powershell module framework <flow_powershell_modules>`.
|
||||||
|
|
||||||
|
These modules use imports from :code:`ansible.module_utils` in order to pull in
|
||||||
|
boilerplate module code, such as argument parsing, formatting of return
|
||||||
|
values as :term:`JSON`, and various file operations.
|
||||||
|
|
||||||
|
.. note:: In Ansible, up to version 2.0.x, the official Python modules used the
|
||||||
|
:ref:`module_replacer` framework. For module authors, :ref:`ziploader` is
|
||||||
|
largely a superset of :ref:`module_replacer` functionality, so you usually
|
||||||
|
do not need to know about one versus the other.
|
||||||
|
|
||||||
|
.. _flow_powershell_modules:
|
||||||
|
|
||||||
|
Powershell
|
||||||
|
^^^^^^^^^^
|
||||||
|
|
||||||
|
New-style powershell modules use the :ref:`module_replacer` framework for
|
||||||
|
constructing modules. These modules get a library of powershell code embedded
|
||||||
|
in them before being sent to the managed node.
|
||||||
|
|
||||||
|
.. _flow_josnargs_modules:
|
||||||
|
|
||||||
|
JSONARGS
|
||||||
|
^^^^^^^^
|
||||||
|
|
||||||
|
Scripts can arrange for an argument string to be placed within them by placing
|
||||||
|
the string ``<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>`` somewhere inside of the
|
||||||
|
file. The module typically sets a variable to that value like this::
|
||||||
|
|
||||||
|
json_arguments = """<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>"""
|
||||||
|
|
||||||
|
Which is expanded as::
|
||||||
|
|
||||||
|
json_arguments = """{"param1": "test's quotes", "param2": "\"To be or not to be\" - Hamlet"}"""
|
||||||
|
|
||||||
|
.. note:: Ansible outputs a :term:`JSON` string with bare quotes. Double quotes are
|
||||||
|
used to quote string values, double quotes inside of string values are
|
||||||
|
backslash escaped, and single quotes may appear unescaped inside of
|
||||||
|
a string value. To use JSONARGS, your scripting language must have a way
|
||||||
|
to handle this type of string. The example uses Python's triple quoted
|
||||||
|
strings to do this. Other scripting languages may have a similar quote
|
||||||
|
character that won't be confused by any quotes in the JSON or it may
|
||||||
|
allow you to define your own start-of-quote and end-of-quote characters.
|
||||||
|
If the language doesn't give you any of these then you'll need to write
|
||||||
|
a :ref:`non-native JSON module <flow_want_json_modules>` or
|
||||||
|
:ref:`Old-style module <flow_old_style_modules>` instead.
|
||||||
|
|
||||||
|
The module typically parses the contents of ``json_arguments`` using a JSON
|
||||||
|
library and then use them as native variables throughout the rest of its code.
|
||||||
|
|
||||||
|
.. _flow_want_json_modules:
|
||||||
|
|
||||||
|
Non-native want JSON modules
|
||||||
|
----------------------------
|
||||||
|
|
||||||
|
If a module has the string ``WANT_JSON`` in it anywhere, Ansible treats
|
||||||
|
it as a non-native module that accepts a filename as its only command line
|
||||||
|
parameter. The filename is for a temporary file containing a :term:`JSON`
|
||||||
|
string containing the module's parameters. The module needs to open the file,
|
||||||
|
read and parse the parameters, operate on the data, and print its return data
|
||||||
|
as a JSON encoded dictionary to stdout before exiting.
|
||||||
|
|
||||||
|
These types of modules are self-contained entities. As of Ansible 2.1, Ansible
|
||||||
|
only modifies them to change a shebang line if present.
|
||||||
|
|
||||||
|
.. seealso:: Examples of Non-native modules written in ruby are in the `Ansible
|
||||||
|
for Rubyists <https://github.com/ansible/ansible-for-rubyists>`_ repository.
|
||||||
|
|
||||||
|
.. _flow_old_style_modules:
|
||||||
|
|
||||||
|
Old-style Modules
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
Old-style modules are similar to
|
||||||
|
:ref:`want JSON modules <flow_want_json_modules>`, except that the file that
|
||||||
|
they take contains ``key=value`` pairs for their parameters instead of
|
||||||
|
:term:`JSON`.
|
||||||
|
|
||||||
|
Ansible decides that a module is old-style when it doesn't have any of the
|
||||||
|
markers that would show that it is one of the other types.
|
||||||
|
|
||||||
|
.. _flow_how_modules_are_executed:
|
||||||
|
|
||||||
|
How modules are executed
|
||||||
|
========================
|
||||||
|
|
||||||
|
When a user uses :program:`ansible` or :program:`ansible-playbook`, they
|
||||||
|
specify a task to execute. The task is usually the name of a module along
|
||||||
|
with several parameters to be passed to the module. Ansible takes these
|
||||||
|
values and processes them in various ways before they are finally executed on
|
||||||
|
the remote machine.
|
||||||
|
|
||||||
|
.. _flow_executor_task_executor:
|
||||||
|
|
||||||
|
executor/task_executor
|
||||||
|
----------------------
|
||||||
|
|
||||||
|
The TaskExecutor receives the module name and parameters that were parsed from
|
||||||
|
the :term:`playbook <playbooks>` (or from the command line in the case of
|
||||||
|
:command:`/usr/bin/ansible`). It uses the name to decide whether it's looking
|
||||||
|
at a module or an :ref:`Action Plugin <flow_action_plugins>`. If it's
|
||||||
|
a module, it loads the :ref:`Normal Action Plugin <flow_normal_action_plugin>`
|
||||||
|
and passes the name, variables, and other information about the task and play
|
||||||
|
to that Action Plugin for further processing.
|
||||||
|
|
||||||
|
.. _flow_normal_action_plugin:
|
||||||
|
|
||||||
|
Normal Action Plugin
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
The ``normal`` Action Plugin executes the module on the remote host. It is
|
||||||
|
the primary coordinator of much of the work to actually execute the module on
|
||||||
|
the managed machine.
|
||||||
|
|
||||||
|
* It takes care of creating a connection to the managed machine by
|
||||||
|
instantiating a Connection class according to the inventory configuration for
|
||||||
|
that host.
|
||||||
|
* It adds any internal Ansible variables to the module's parameters (for
|
||||||
|
instance, the ones that pass along ``no_log`` to the module).
|
||||||
|
* It takes care of creating any temporary files on the remote machine and
|
||||||
|
cleans up afterwards.
|
||||||
|
* It does the actual work of pushing the module and module parameters to the
|
||||||
|
remote host, although the :ref:`module_common <flow_executor_module_common>`
|
||||||
|
code described next does the work of deciding which format those will take.
|
||||||
|
* It handles any special cases regarding modules (for instance, various
|
||||||
|
complications around Windows modules that must have the same names as Python
|
||||||
|
modules, so that internal calling of modules from other Action Plugins work.)
|
||||||
|
|
||||||
|
Much of this functionality comes from the :class:`BaseAction` class,
|
||||||
|
which lives in :file:`plugins/action/__init__.py`. It makes use of Connection
|
||||||
|
and Shell objects to do its work.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
When :term:`tasks <tasks>` are run with the ``async:`` parameter, Ansible
|
||||||
|
uses the ``async`` Action Plugin instead of the ``normal`` Action Plugin
|
||||||
|
to invoke it. That program flow is currently not documented. Read the
|
||||||
|
source for information on how that works.
|
||||||
|
|
||||||
|
.. _flow_executor_module_common:
|
||||||
|
|
||||||
|
executor/module_common.py
|
||||||
|
-------------------------
|
||||||
|
|
||||||
|
Code in :file:`executor/module_common.py` takes care of assembling the module
|
||||||
|
to be shipped to the managed node. The module is first read in, then examined
|
||||||
|
to determine its type. :ref:`PowerShell <flow_powershell_modules>` and
|
||||||
|
:ref:`JSON-args modules <flow_jsonargs_modules>` are passed through
|
||||||
|
:ref:`Module Replacer <module_replacer>`. New-style
|
||||||
|
:ref:`Python modules <flow_python_modules>` are assembled by :ref:`ziploader`.
|
||||||
|
:ref:`Non-native-want-JSON <flow_want_json_modules>` and
|
||||||
|
:ref:`Old-Style modules <flow_old_style_modules>` aren't touched by either of
|
||||||
|
these and pass through unchanged. After the assembling step, one final
|
||||||
|
modification is made to all modules that have a shebang line. Ansible checks
|
||||||
|
whether the interpreter in the shebang line has a specific path configured via
|
||||||
|
an ``ansible_$X_interpreter`` inventory variable. If it does, Ansible
|
||||||
|
substitutes that path for the interpreter path given in the module. After
|
||||||
|
this Ansible returns the complete module data and the module type to the
|
||||||
|
:ref:`Normal Action <_flow_normal_action_plugin>` which continues execution of
|
||||||
|
the module.
|
||||||
|
|
||||||
|
Next we'll go into some details of the two assembler frameworks.
|
||||||
|
|
||||||
|
.. _module_replacer:
|
||||||
|
|
||||||
|
Module Replacer
|
||||||
|
^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
The Module Replacer is essentially a preprocessor (like the C Preprocessor for
|
||||||
|
those familiar with that language). It does straight substitutions of
|
||||||
|
specific substring patterns in the module file. There are two types of
|
||||||
|
substitutions:
|
||||||
|
|
||||||
|
* Replacements that only happen in the module file. These are public
|
||||||
|
replacement strings that modules can utilize to get helpful boilerplate or
|
||||||
|
access to arguments.
|
||||||
|
|
||||||
|
- :code:`from ansible.module_utils.MOD_LIB_NAME import *` is replaced with the
|
||||||
|
contents of the :file:`ansible/module_utils/MOD_LIB_NAME.py` These should
|
||||||
|
only be used with :ref:`new-style Python modules <flow_python_modules>`.
|
||||||
|
- :code:`#<<INCLUDE_ANSIBLE_MODULE_COMMON>>` is equivalent to
|
||||||
|
:code:`from ansible.module_utils.basic import *` and should also only apply
|
||||||
|
to new-style Python modules.
|
||||||
|
- :code:`# POWERSHELL_COMMON` substitutes the contents of
|
||||||
|
:file:`ansible/module_utils/powershell.ps1`. It should only be used with
|
||||||
|
:ref:`new-style Powershell modules <flow_powershell_modules>`.
|
||||||
|
|
||||||
|
* Replacements that are used by ``ansible.module_utils`` code. These are internal
|
||||||
|
replacement patterns. They may be used internally, in the above public
|
||||||
|
replacements, but shouldn't be used directly by modules.
|
||||||
|
|
||||||
|
- :code:`"<<ANSIBLE_VERSION>>"` is substituted with the Ansible version. In
|
||||||
|
a new-style Python module, it's better to use ``from ansible import
|
||||||
|
__version__`` and then use ``__version__`` instead.
|
||||||
|
- :code:`"<<INCLUDE_ANSIBLE_MODULE_COMPLEX_ARGS>>"` is substituted with
|
||||||
|
a string which is the Python ``repr`` of the :term:`JSON` encoded module
|
||||||
|
parameters. Using ``repr`` on the JSON string makes it safe to embed in
|
||||||
|
a Python file. In :ref:`new-style Python modules <flow_python_modules>`
|
||||||
|
under :ref:`ziploader` this is passed in via an environment variable
|
||||||
|
instead.
|
||||||
|
- :code:`<<SELINUX_SPECIAL_FILESYSTEMS>>` substitutes a string which is
|
||||||
|
a comma separated list of file systems which have a file system dependent
|
||||||
|
security context in SELinux. In new-style Python modules, this is found
|
||||||
|
by looking up ``SELINUX_SPECIAL_FS`` from the
|
||||||
|
:envvar:`ANSIBLE_MODULE_CONSTANTS` environment variable. See the
|
||||||
|
:ref:`ziploader` documentation for details.
|
||||||
|
- :code:`<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>` substitutes the module
|
||||||
|
parameters as a JSON string. Care must be taken to properly quote the
|
||||||
|
string as JSON data may contain quotes. This pattern is not substituted
|
||||||
|
in new-style Python modules as they can get the module parameters via the
|
||||||
|
environment variable.
|
||||||
|
- the string :code:`syslog.LOG_USER` is replaced wherever it occurs with the
|
||||||
|
value of ``syslog_facility`` from the :file:`ansible.cfg` or any
|
||||||
|
``ansible_syslog_facility`` inventory variable that applies to this host. In
|
||||||
|
new-style Python modules, you can get the value of the ``syslog_facility``
|
||||||
|
by looking up ``SYSLOG_FACILITY`` in the :envvar:`ANSIBLE_MODULE_CONSTANTS`
|
||||||
|
environment variable. See the :ref:`ziploader` documentation for details.
|
||||||
|
|
||||||
|
.. _ziploader:
|
||||||
|
|
||||||
|
ziploader
|
||||||
|
^^^^^^^^^
|
||||||
|
|
||||||
|
Ziploader differs from :ref:`module_replacer` in that it uses real Python
|
||||||
|
imports of things in module_utils instead of merely preprocessing the module.
|
||||||
|
It does this by constructing a zipfile--which includes the module file, files
|
||||||
|
in :file:`ansible/module_utils` that are imported by the module, and some
|
||||||
|
boilerplate to pass in the constants. The zipfile is then Base64 encoded and
|
||||||
|
wrapped in a small Python script which unzips the file on the managed node and
|
||||||
|
then invokes Python on the file. (Ansible wraps the zipfile in the Python
|
||||||
|
script so that pipelining will work.)
|
||||||
|
|
||||||
|
In ziploader, any imports of Python modules from the ``ansible.module_utils``
|
||||||
|
package trigger inclusion of that Python file into the zipfile. Instances of
|
||||||
|
:code:`#<<INCLUDE_ANSIBLE_MODULE_COMMON>>` in the module are turned into
|
||||||
|
:code:`from ansible.module_utils.basic import *` and
|
||||||
|
:file:`ansible/module-utils/basic.py` is then included in the zipfile. Files
|
||||||
|
that are included from module_utils are themselves scanned for imports of other
|
||||||
|
Python modules from module_utils to be included in the zipfile as well.
|
||||||
|
|
||||||
|
.. warning::
|
||||||
|
At present, there are two caveats to how ziploader determines other files
|
||||||
|
to import:
|
||||||
|
|
||||||
|
* Ziploader cannot determine whether an import should be included if it is
|
||||||
|
a relative import. Always use an absolute import that has
|
||||||
|
``ansible.module_utils`` in it to allow ziploader to determine that the
|
||||||
|
file should be included.
|
||||||
|
* Ziploader does not include Python packages (directories with
|
||||||
|
:file:`__init__.py`` in them). Ziploader only works on :file:`*.py`
|
||||||
|
files that are directly in the :file:`ansible/module_utils` directory.
|
||||||
|
|
||||||
|
.. _flow_passing_module_args:
|
||||||
|
|
||||||
|
Passing args
|
||||||
|
~~~~~~~~~~~~
|
||||||
|
|
||||||
|
In :ref:`module_replacer`, module arguments are turned into a JSON-ified
|
||||||
|
string and substituted into the combined module file. In :ref:`ziploader`,
|
||||||
|
the JSON-ified string is placed in the the :envvar:`ANSIBLE_MODULE_ARGS`
|
||||||
|
environment variable. When :code:`ansible.module_utils.basic` is imported,
|
||||||
|
it places this string in the global variable
|
||||||
|
``ansible.module_utils.basic.MODULE_COMPLEX_ARGS`` and removes it from the
|
||||||
|
environment. Modules should not access this variable directly. Instead, they
|
||||||
|
should instantiate an :class:`AnsibleModule()` and use
|
||||||
|
:meth:`AnsibleModule.params` to access the parsed version of the arguments.
|
||||||
|
|
||||||
|
.. _flow_passing_module_constants:
|
||||||
|
|
||||||
|
Passing constants
|
||||||
|
~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Currently, there are three constants passed from the controller to the modules:
|
||||||
|
``ANSIBLE_VERSION``, ``SELINUX_SPECIAL_FS``, and ``SYSLOG_FACILITY``. In
|
||||||
|
:ref:`module_replacer`, ``ANSIBLE_VERSION`` and ``SELINUX_SPECIAL_FS`` were
|
||||||
|
substituted into the global variables
|
||||||
|
:code:`ansible.module_utils.basic.ANSIBLE_VERSION` and
|
||||||
|
:code:`ansible.module_utils.basic.SELINUX_SPECIAL_FS`. ``SYSLOG_FACILITY`` didn't
|
||||||
|
get placed into a variable. Instead, any occurrences of the string
|
||||||
|
``syslog.LOG_USER`` in the combined module file were replaced with ``syslog.``
|
||||||
|
followed by the string contained in ``SYSLOG_FACILITY``. All of these have
|
||||||
|
changed in :ref:`ziploader`.
|
||||||
|
|
||||||
|
The Ansible verison can now be used by a module by importing ``__version__``
|
||||||
|
from ansible::
|
||||||
|
|
||||||
|
from ansible import __version__
|
||||||
|
module.exit_json({'msg': 'module invoked by ansible %s' % __version__})
|
||||||
|
|
||||||
|
For now, :code:`ANSIBLE_VERSION` is also available at its old location inside of
|
||||||
|
``ansible.module_utils.basic``, but that will eventually be removed.
|
||||||
|
|
||||||
|
``SELINUX_SPECIAL_FS`` and ``SYSLOG_FACILITY`` have changed much more.
|
||||||
|
:ref:`ziploader` passes these as another JSON-ified string inside of the
|
||||||
|
:envvar:`ANSIBLE_MODULE_CONSTANTS` environment variable. When
|
||||||
|
``ansible.module_utils.basic`` is imported, it places this string in the global
|
||||||
|
variable :code:`ansible.module_utils.basic.MODULE_CONSTANTS` and removes it from
|
||||||
|
the environment. The constants are parsed when an :class:`AnsibleModule` is
|
||||||
|
instantiated. Modules shouldn't access any of those directly. Instead, they
|
||||||
|
should instantiate an :class:`AnsibleModule` and use
|
||||||
|
:attr:`AnsibleModule.constants` to access the parsed version of these values.
|
||||||
|
|
||||||
|
Unlike the ``ANSIBLE_ARGS`` and ``ANSIBLE_VERSION``, where some efforts were
|
||||||
|
made to keep the old backwards compatible globals available, these two
|
||||||
|
constants are not available at their old names. This is a combination of the
|
||||||
|
degree to which these are internal to the needs of ``module_utils.basic`` and,
|
||||||
|
in the case of ``SYSLOG_FACILITY``, how hacky and unsafe the previous
|
||||||
|
implementation was.
|
||||||
|
|
||||||
|
Porting code from the :ref:`module_replacer` method of getting
|
||||||
|
``SYSLOG_FACILITY`` to the new one is a little more tricky than the other
|
||||||
|
constants and args, due to just how hacky the old way was. Here's an example
|
||||||
|
of using it in the new way::
|
||||||
|
|
||||||
|
import syslog
|
||||||
|
facility_name = module.constants.get('SYSLOG_FACILITY')
|
||||||
|
facility = getattr(syslog, facility_name)
|
||||||
|
syslog.openlog(str(module), 0, facility)
|
||||||
|
|
||||||
|
.. _flow_special_considerations:
|
||||||
|
|
||||||
|
Special Considerations
|
||||||
|
----------------------
|
||||||
|
|
||||||
|
.. _flow_pipelining:
|
||||||
|
|
||||||
|
Pipelining
|
||||||
|
^^^^^^^^^^
|
||||||
|
|
||||||
|
Ansible can transfer a module to a remote machine in one of two ways:
|
||||||
|
|
||||||
|
* it can write out the module to a temporary file on the remote host and then
|
||||||
|
use a second connection to the remote host to execute it with the
|
||||||
|
interpreter that the module needs
|
||||||
|
* or it can use what's known as pipelining to execute the module by piping it
|
||||||
|
into the remote interpreter's stdin.
|
||||||
|
|
||||||
|
Pipelining only works with modules written in Python at this time because
|
||||||
|
Ansible only knows that Python supports this mode of operation. Supporting
|
||||||
|
pipelining means that whatever format the module payload takes before being
|
||||||
|
sent over the wire must be executable by Python via stdin.
|
Loading…
Reference in New Issue