Docs: edits & expands module_utils & search path info in dev guide (#55931)

* Update docs/docsite/rst/dev_guide/developing_module_utilities.rst

docs/docsite/rst/dev_guide/developing_module_utilities.rst

@@ -1,87 +1,77 @@
-.. _appendix_module_utilities:
+.. _developing_module_utilities:
 
-**************************
-Appendix: Module Utilities
-**************************
+*************************************
+Using and Developing Module Utilities
+*************************************
 
-Ansible provides a number of module utilities that provide helper functions that you can use when developing your own modules. The ``basic.py`` module utility provides the main entry point for accessing the Ansible library, and all Python Ansible modules must, at minimum, import ``AnsibleModule``::
+Ansible provides a number of module utilities, or snippets of shared code, that
+provide helper functions you can use when developing your own modules. The
+``basic.py`` module utility provides the main entry point for accessing the
+Ansible library, and all Python Ansible modules must import something from
+``ansible.module_utils``. A common option is to import ``AnsibleModule``::
 
   from ansible.module_utils.basic import AnsibleModule
 
-If you need to share Python code between some of your own local modules, you can use Ansible's ``module_utils`` directories for this. When you run ``ansible-playbook``, Ansible will merge any files in the local ``module_utils`` directory into the ``ansible.module_utils`` namespace. For example, if you have your own custom modules that import a ``my_shared_code`` library, you can place that into a ``./module_utils/my_shared_code.py`` file in the root location where your playbook lives, and then import it in your modules like so::
+The ``ansible.module_utils`` namespace is not a plain Python package: it is
+constructed dynamically for each task invocation, by extracting imports and
+resolving those matching the namespace against a :ref:`search path <ansible_search_path>` derived from the
+active configuration.
+
+To reduce the maintenance burden on your own local modules, you can extract
+duplicated code into one or more module utilities and import them into your modules. For example, if you have your own custom modules that import a ``my_shared_code`` library, you can place that into a ``./module_utils/my_shared_code.py`` file like this::
 
   from ansible.module_utils.my_shared_code import MySharedCodeClient
 
-Your custom ``module_utils`` directories can live in the root directory of your playbook, or in the individual role directories, or in the directories specified by the ``ANSIBLE_MODULE_UTILS`` configuration setting.
+When you run ``ansible-playbook``, Ansible will merge any files in your local ``module_utils`` directories into the ``ansible.module_utils`` namespace in the order defined by the :ref:`Ansible search path <ansible_search_path>`.
+
+Naming and finding module utilities
+===================================
+
+You can generally tell what a module utility does from its name and/or its location. For example, ``openstack.py`` contains utilities for modules that work with OpenStack instances.
+Generic utilities (shared code used by many different kinds of modules) live in the ``common`` subdirectory or in the root directory. Utilities
+used by a particular set of modules generally live in a sub-directory that mirrors
+the directory for those modules. For example:
+
+* ``lib/ansible/module_utils/urls.py`` contains shared code for parsing URLs
+* ``lib/ansible/module_utils/storage/emc/`` contains shared code related to EMC
+*  ``lib/ansible/modules/storage/emc/`` contains modules related to EMC
+
+Following this pattern with your own module utilities makes everything easy to find and use.
+
+.. _standard_mod_utils:
+
+Standard module utilities
+=========================
 
-Ansible ships with the following list of ``module_utils`` files. The module utility source code lives in the ``./lib/ansible/module_utils`` directory under your main Ansible path - for more details on any specific module utility, please see the source code.
+Ansible ships with an extensive library of ``module_utils`` files.
+You can find the module
+utility source code in the ``lib/ansible/module_utils`` directory under
+your main Ansible path. We've described the most widely used utilities below. For more details on any specific module utility,
+please see the `source code for module_utils <https://github.com/ansible/ansible/tree/devel/lib/ansible/module_utils>`_.
 
 .. include:: shared_snippets/licensing.txt
 
-- alicloud_ecs.py - Definitions and utilities for modules working with Alibaba Cloud ECS.
-- api.py - Adds shared support for generic API modules.
-- azure_rm_common.py - Definitions and utilities for Microsoft Azure Resource Manager template deployments.
-- basic.py - General definitions and helper utilities for Ansible modules.
-- cloudstack.py  - Utilities for CloudStack modules.
-- database.py - Miscellaneous helper functions for PostGRES and MySQL
-- docker_common.py - Definitions and helper utilities for modules working with Docker.
-- ec2.py - Definitions and utilities for modules working with Amazon EC2
-- facts/- Folder containing helper functions for modules that return facts. See https://github.com/ansible/ansible/pull/23012 for more information.
-- gce.py - Definitions and helper functions for modules that work with Google Compute Engine resources.
-- ismount.py - Contains single helper function that fixes os.path.ismount
-- keycloak.py - Definitions and helper functions for modules working with the Keycloak API
-- known_hosts.py - utilities for working with known_hosts file
-- manageiq.py - Functions and utilities for modules that work with ManageIQ platform and its resources.
-- memset.py - Helper functions and utilities for interacting with Memset's API.
-- mysql.py - Allows modules to connect to a MySQL instance
-- netapp.py - Functions and utilities for modules that work with the NetApp storage platforms.
-- network/a10/a10.py - Utilities used by the a10_server module to manage A10 Networks devices.
-- network/aci/aci.py - Definitions and helper functions for modules that manage Cisco ACI Fabrics.
-- network/aireos/aireos.py - Definitions and helper functions for modules that manage Cisco WLC devices.
-- network/aos/aos.py - Module support utilities for managing Apstra AOS Server.
-- network/aruba/aruba.py - Helper functions for modules working with Aruba networking devices.
-- network/asa/asa.py - Module support utilities for managing Cisco ASA network devices.
-- network/avi/avi.py - Helper functions for modules working with AVI networking devices.
-- network/bigswitch/bigswitch_utils.py - Utilities used by the bigswitch module to manage Big Switch Networks devices.
-- network/cloudengine/ce.py - Module support utilities for managing Huawei Cloudengine switch.
-- network/cnos/cnos.py - Helper functions for modules working on devices running Lenovo CNOS.
-- network/common/config.py - Configuration utility functions for use by networking modules
-- network/common/netconf.py - Definitions and helper functions for modules that use Netconf transport.
-- network/common/parsing.py - Definitions and helper functions for Network modules.
-- network/common/network.py - Functions for running commands on networking devices
-- network/common/utils.py - Defines commands and comparison operators and other utilises for use in networking modules
-- network/dellos6/dellos6.py - Module support utilities for managing device running Dell OS6.
-- network/dellos9/dellos9.py - Module support utilities for managing device running Dell OS9.
-- network/dellos10/dellos10.py - Module support utilities for managing device running Dell OS10.
-- network/enos/enos.py - Helper functions for modules working with Lenovo ENOS devices.
-- network/eos/eos.py - Helper functions for modules working with EOS networking devices.
-- network/fortios/fortios.py - Module support utilities for managing FortiOS devices.
-- network/ios/ios.py - Definitions and helper functions for modules that manage Cisco IOS networking devices
-- network/iosxr/iosxr.py - Definitions and helper functions for modules that manage Cisco IOS-XR networking devices.
-- network/ironware/ironware.py - Module support utilities for managing Brocade IronWare devices.
-- network/junos/junos.py -  Definitions and helper functions for modules that manage Junos networking devices.
-- network/meraki/meraki.py - Utilities specifically for the Meraki network modules.
-- network/netscaler/netscaler.py - Utilities specifically for the netscaler network modules.
-- network/nso/nso.py - Utilities for modules that work with Cisco NSO.
-- network/nxos/nxos.py - Contains definitions and helper functions specific to Cisco NXOS networking devices.
-- network/onyx/onyx.py - Definitions and helper functions for modules that manage Mellanox ONYX networking devices.
-- network/ordance/ordance.py - Module support utilities for managing Ordnance devices.
-- network/sros/sros.py - Helper functions for modules working with Open vSwitch bridges.
-- network/vyos/vyos.py - Definitions and functions for working with VyOS networking
-- openstack.py - Utilities for modules that work with Openstack instances.
-- openswitch.py - Definitions and helper functions for modules that manage OpenSwitch devices
-- powershell.ps1 - Utilities for working with Microsoft Windows clients
-- pure.py - Functions and utilities for modules that work with the Pure Storage storage platforms.
-- pycompat24.py - Exception workaround for Python 2.4.
-- rax.py -  Definitions and helper functions for modules that work with Rackspace resources.
-- redhat.py - Functions for modules that manage Red Hat Network registration and subscriptions
-- service.py - Contains utilities to enable modules to work with Linux services (placeholder, not in use).
-- shell.py - Functions to allow modules to create shells and work with shell commands
-- six/__init__.py - Bundled copy of the `Six Python library <https://pythonhosted.org/six/>`_ to aid in writing code compatible with both Python 2 and Python 3.
-- splitter.py - String splitting and manipulation utilities for working with Jinja2 templates
-- urls.py - Utilities for working with http and https requests
-- utm_utils.py - Contains base class for creating new Sophos UTM Modules and helper functions for handling the rest interface of Sophos UTM
-- vca.py - Contains utilities for modules that work with VMware vCloud Air
-- vexata.py - Utilities for modules that work with Vexata storage platforms.
-- vmware.py - Contains utilities for modules that work with VMware vSphere VMs
-- xenserver.py - Contains utilities for modules that work with XenServer.
+- ``api.py`` - Supports generic API modules
+- ``basic.py`` - General definitions and helper utilities for Ansible modules
+- ``common/dict_transformations.py`` - Helper functions for dictionary transformations
+- ``common/file.py`` - Helper functions for working with files
+- ``common/text/`` - Helper functions for converting and formatting text.
+- ``common/parameters.py`` - Helper functions for dealing with module parameters
+- ``common/sys_info.py`` - Functions for getting distribution and platform information
+- ``common/validation.py`` - Helper functions for validating module parameters against a module argument spec
+- ``facts/`` - Directory of utilities for modules that return facts. See `PR 23012 <https://github.com/ansible/ansible/pull/23012>`_ for more information
+- ``ismount.py`` - Single helper function that fixes os.path.ismount
+- ``json_utils.py`` - Utilities for filtering unrelated output around module JSON output, like leading and trailing lines
+- ``known_hosts.py`` - utilities for working with known_hosts file
+- ``network/common/config.py`` - Configuration utility functions for use by networking modules
+- ``network/common/netconf.py`` - Definitions and helper functions for modules that use Netconf transport
+- ``network/common/parsing.py`` - Definitions and helper functions for Network modules
+- ``network/common/network.py`` - Functions for running commands on networking devices
+- ``network/common/utils.py`` - Defines commands and comparison operators and other utilises for use in networking modules
+- ``powershell/`` - Directory of definitions and helper functions for Windows PowerShell modules
+- ``pycompat24.py`` - Exception workaround for Python 2.4
+- ``service.py`` - Utilities to enable modules to work with Linux services (placeholder, not in use)
+- ``shell.py`` - Functions to allow modules to create shells and work with shell commands
+- ``six/__init__.py`` - Bundled copy of the `Six Python library <https://pythonhosted.org/six/>`_ to aid in writing code compatible with both Python 2 and Python 3
+- ``splitter.py`` - String splitting and manipulation utilities for working with Jinja2 templates
+- ``urls.py`` - Utilities for working with http and https requests

docs/docsite/rst/dev_guide/developing_modules_best_practices.rst

@@ -67,7 +67,7 @@ Python tips
 Importing and using shared code
 ===============================
 
-* Use shared code whenever possible - don't reinvent the wheel. Ansible offers the ``AnsibleModule`` common Python code, plus :ref:`utilities <appendix_module_utilities>` for many common use cases and patterns.
+* Use shared code whenever possible - don't reinvent the wheel. Ansible offers the ``AnsibleModule`` common Python code, plus :ref:`utilities <developing_module_utilities>` for many common use cases and patterns. You can also create documentation fragments for docs that apply to multiple modules.
 * Import ``ansible.module_utils`` code in the same place as you import other libraries.
 * Do NOT use wildcards (*) for importing other python modules; instead, list the function(s) you are importing (for example, ``from some.other_python_module.basic import otherFunction``).
 * Import custom packages in ``try``/``except``, capture any import errors, and handle them with ``fail_json()`` in ``main()``. For example:

docs/docsite/rst/dev_guide/developing_modules_in_groups.rst

@@ -91,7 +91,6 @@ The first PR should include the following files:
 * ``lib/ansible/modules/$category/$topic/$yourfirstmodule.py`` - A single module. *Required new file*
 * ``lib/ansible/plugins/doc_fragments/$topic.py`` - Code documentation, such as details regarding common arguments. *Optional new file*
 * ``lib/ansible/module_utils/$topic.py`` - Code shared between more than one module, such as common arguments. *Optional new file*
-*  ``docs/docsite/rst/dev_guide/developing_module_utilities.rst`` - Document your new `module_utils` file. *Optional update to existing file*
 
 And that's it.
 

docs/docsite/rst/dev_guide/overview_architecture.rst

@@ -1,5 +1,5 @@
 ********************
-Ansible Architecture
+Ansible architecture
 ********************
 
 Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.
@@ -10,26 +10,35 @@ It uses no agents and no additional custom security infrastructure, so it's easy
 
 In this section, we'll give you a really quick overview of how Ansible works so you can see how the pieces fit together.
 
+.. contents::
+   :local:
+
 Modules
 =======
 
-Ansible works by connecting to your nodes and pushing out small programs, called "Ansible Modules" to them. These programs are written to be resource models of the desired state of the system. Ansible then executes these modules (over SSH by default), and removes them when finished.
+Ansible works by connecting to your nodes and pushing out scripts called "Ansible modules" to them. Most modules accept parameters that describe the desired state of the system.
+Ansible then executes these modules (over SSH by default), and removes them when finished. Your library of modules can reside on any machine, and there are no servers, daemons, or databases required.
+
+You can :ref:`write your own modules <developing_modules_general>`, though you should first consider :ref:`whether you should <developing_modules>`. Typically you'll work with your favorite terminal program, a text editor, and probably a version control system to keep track of changes to your content. You may write specialized modules in any language that can return JSON (Ruby, Python, bash, etc).
+
+Module utilities
+================
 
-Your library of modules can reside on any machine, and there are no servers, daemons, or databases required. Typically you'll work with your favorite terminal program, a text editor, and probably a version control system to keep track of changes to your content.
+When multiple modules use the same code, Ansible stores those functions as module utilities to minimize duplication and maintenance. For example, the code that parses URLs is ``lib/ansible/module_utils/url.py``. You can :ref:`write your own module utilities <developing_module_utilities>` as well. Module utilities may only be written in Python or in PowerShell.
 
 Plugins
 =======
 
-Plugins are pieces of code that augment Ansible's core functionality. Ansible ships with a number of handy plugins, and you can easily write your own.
+:ref:`Plugins <plugins_lookup>` augment Ansible's core functionality. While modules execute on the target system in separate processes (usually that means on a remote system), plugins execute on the control node within the ``/usr/bin/ansible`` process. Plugins offer options and extensions for the core features of Ansible - transforming data, logging output, connecting to inventory, and more. Ansible ships with a number of handy plugins, and you can easily :ref:`write your own <developing_plugins>`. For example, you can write an :ref:`inventory plugin <developing_inventory>` to connect to any datasource that returns JSON. Plugins must be written in Python.
 
 Inventory
 =========
 
-By default, Ansible represents what machines it manages using a very simple INI file that puts all of your managed machines in groups of your own choosing.
+By default, Ansible represents the machines it manages in a file (INI, YAML, etc.) that puts all of your managed machines in groups of your own choosing.
 
 To add new machines, there is no additional SSL signing server involved, so there's never any hassle deciding why a particular machine didn't get linked up due to obscure NTP or DNS issues.
 
-If there's another source of truth in your infrastructure, Ansible can also plugin to that, such as drawing inventory, group, and variable information from sources like EC2, Rackspace, OpenStack, and more.
+If there's another source of truth in your infrastructure, Ansible can also connect to that. Ansible can draw inventory, group, and variable information from sources like EC2, Rackspace, OpenStack, and more.
 
 Here's what a plain text inventory file looks like::
 
@@ -67,8 +76,74 @@ Here's what a simple playbook looks like::
     - common
     - content
 
+.. _ansible_search_path:
+
+The Ansible search path
+=======================
+
+Modules, module utilities, plugins, playbooks, and roles can live in multiple locations. If you
+write your own code to extend Ansible's core features, you may have multiple files with similar or the same names in different locations on your Ansible control node. The search path determines which of these files Ansible will discover and use on any given playbook run.
+
+Ansible's search path grows incrementally over a run. As
+Ansible finds each playbook and role included in a given run, it appends
+any directories related to that playbook or role to the search path. Those
+directories remain in scope for the duration of the run, even after the playbook or role
+has finished executing. Ansible loads modules, module utilities, and plugins in this order:
+
+1. Directories adjacent to a playbook specified on the command line. If you run Ansible with ``ansible-playbook /path/to/play.yml``, Ansible appends these directories if they exist:
+
+   .. code-block:: bash
+
+      /path/to/modules
+      /path/to/module_utils
+      /path/to/plugins
+
+2. Directories adjacent to a playbook that is statically imported by a
+   playbook specified on the command line. If ``play.yml`` includes
+   ``- import_playbook: /path/to/subdir/play1.yml``, Ansible appends these directories if they exist:
+
+   .. code-block:: bash
+
+      /path/to/subdir/modules
+      /path/to/subdir/module_utils
+      /path/to/subdir/plugins
+
+3. Subdirectories of a role directory referenced by a playbook. If
+   ``play.yml`` runs ``myrole``, Ansible appends these directories if they exist:
+
+   .. code-block:: bash
+
+      /path/to/roles/myrole/modules
+      /path/to/roles/myrole/module_utils
+      /path/to/roles/myrole/plugins
+
+4. Directories specified as default paths in ``ansible.cfg`` or by the related
+   environment variables, including the paths for the various plugin types. See :ref:`ansible_configuration_settings` for more information.
+   Sample ``ansible.cfg`` fields:
+
+   .. code-block:: bash
+
+      DEFAULT_MODULE_PATH
+      DEFAULT_MODULE_UTILS_PATH
+      DEFAULT_CACHE_PLUGIN_PATH
+      DEFAULT_FILTER_PLUGIN_PATH
+
+   Sample environment variables:
+
+   .. code-block:: bash
+
+      ANSIBLE_LIBRARY
+      ANSIBLE_MODULE_UTILS
+      ANSIBLE_CACHE_PLUGINS
+      ANSIBLE_FILTER_PLUGINS
+
+5. The standard directories that ship as part of the Ansible distribution.
+
+.. caution::
 
-Extending Ansible with plug-ins and the API
-===========================================
+   Modules, module utilities, and plugins in user-specified directories will
+   override the standard versions. This includes some files with generic names.
+   For example, if you have a file named ``basic.py`` in a user-specified
+   directory, it will override the standard ``ansible.module_utils.basic``.
 
-Should you want to write your own, Ansible modules can be written in any language that can return JSON (Ruby, Python, bash, etc). Inventory can also plug in to any datasource by writing a program that speaks to that datasource and returns JSON. There's also various Python APIs for extending Ansible's connection types (SSH is not the only transport possible), callbacks (how Ansible logs, etc), and even for adding new server side behaviors.
+   If you have more than one module, module utility, or plugin with the same name in different user-specified directories, the order of commands at the command line and the order of includes and roles in each play will affect which one is found and used on that particular play.

docs/docsite/rst/porting_guides/porting_guide_2.5.rst

@@ -143,7 +143,7 @@ Ansible fact namespacing
 Ansible facts, which have historically been written to names like ``ansible_*``
 in the main facts namespace, have been placed in their own new namespace,
 ``ansible_facts.*`` For example, the fact ``ansible_distribution`` is now best
-queried through the variable structure ``ansible_facts.distribution``. 
+queried through the variable structure ``ansible_facts.distribution``.
 
 A new configuration variable, ``inject_facts_as_vars``, has been added to
 ansible.cfg. Its default setting, 'True', keeps the 2.4 behavior of facts
@@ -389,4 +389,4 @@ If your module uses shared module utilities, you must update all references. For
    from ansible.module_utils.network.vyos.vyos import get_config, load_config
 
 
-See the module utilities developer guide see :ref:`appendix_module_utilities` for more information.
+See the module utilities developer guide see :ref:`developing_module_utilities` for more information.