document adjacent docs (sidecar) (#79056)

* Update docs/docsite/rst/dev_guide/developing_locally.rst
Co-authored-by: Don Naro <dnaro@redhat.com>

docs/docsite/rst/dev_guide/ansible_index.rst

@@ -75,6 +75,7 @@ If you prefer to read the entire guide, here's a list of the pages in order.
    developing_python_3
    debugging
    developing_modules_documenting
+   adjacent_yaml_doc
    developing_modules_general_windows
    developing_modules_general_aci
    platforms/aws_guidelines

docs/docsite/rst/dev_guide/core_index.rst

@@ -71,6 +71,7 @@ If you prefer to read the entire guide, here's a list of the pages in order.
    developing_python_3
    debugging
    developing_modules_documenting
+   sidecar
    developing_modules_general_windows
    developing_modules_general_aci
    developing_modules_in_groups

docs/docsite/rst/dev_guide/developing_locally.rst

@@ -28,8 +28,8 @@ Modules and plugins: what is the difference?
 ============================================
 If you are looking to add functionality to Ansible, you might wonder whether you need a module or a plugin. Here is a quick overview to help you understand what you need:
 
-* Modules are reusable, standalone scripts that can be used by the Ansible API, the :command:`ansible` command, or the :command:`ansible-playbook` command. Modules provide a defined interface. Each module accepts arguments and returns information to Ansible by printing a JSON string to stdout before exiting. Modules execute on the target system (usually that means on a remote system) in separate processes. Modules are technically plugins, but for historical reasons we do not usually talk about "module plugins".
-* :ref:`Plugins <working_with_plugins>` extend Ansible's core functionality and execute on the control node within the ``/usr/bin/ansible`` process. Plugins offer options and extensions for the core features of Ansible - transforming data, logging output, connecting to inventory, and more.
+* :ref:`Plugins <working_with_plugins>` extend Ansible's core functionality. Most plugin types execute on the control node within the ``/usr/bin/ansible`` process. Plugins offer options and extensions for the core features of Ansible: transforming data, logging output, connecting to inventory, and more.
+* Modules are a type of plugin that execute automation tasks on a 'target' (usually a remote system). Modules work as standalone scripts that Ansible executes in their own process outside of the controller. Modules interface with Ansible mostly via JSON, accepting arguments and returning information by printing a JSON string to stdout before exiting. Unlike the other plugins (which must be written in Python), modules can be written in any language; although Ansible provides modules in Python and Powershell only.
 
 .. _use_collections:
 
@@ -42,10 +42,10 @@ The rest of this page describes other methods of using local, standalone modules
 
 .. _local_modules:
 
-Adding a module outside of a collection
-=======================================
+Adding a module or plugin outside of a collection
+==================================================
 
-You can configure Ansible to load standalone local modules in a specified location or locations and make them available to all playbooks and roles. Alternatively, you can make a non-collection local module available only to specific playbooks or roles.
+You can configure Ansible to load standalone local modules or plugins in specific locations and make them available to all playbooks and roles (using configured paths). Alternatively, you can make a non-collection local module or plugin available only to certain playbooks or roles (via adjacent paths).
 
 Adding standalone local modules for all playbooks and roles
 -----------------------------------------------------------
@@ -70,9 +70,10 @@ To confirm that ``my_local_module`` is available:
 * type ``ansible localhost -m my_local_module`` to see the output for that module, or
 * type ``ansible-doc -t module my_local_module`` to see the documentation for that module
 
+.. note:: This applies to all plugin types but requires specific configuration and/or adjacent directories for each plugin type, see below.
 .. note::
 
-   Currently, the ``ansible-doc`` command can parse module documentation only from modules written in Python. If you have a module written in a programming language other than Python, please write the documentation in a Python file adjacent to the module file.
+   The ``ansible-doc`` command can parse module documentation from modules written in Python or an adjacent YAML file. If you have a module written in a programming language other than Python, you should write the documentation in a Python or YAML file adjacent to the module file. :ref:`adjacent_yaml_doc`
 
 Adding standalone local modules for selected playbooks or a single role
 -----------------------------------------------------------------------
@@ -82,6 +83,8 @@ Ansible automatically loads all executable files from certain directories adjace
 * To use a standalone module only in a selected playbook or playbooks, store the module in a subdirectory called ``library`` in the directory that contains the playbook or playbooks.
 * To use a standalone module only in a single role, store the module in a subdirectory called ``library`` within that role.
 
+.. note:: This applies to all plugin types but requires specific configuration and/or adjacent directories for each plugin type, see below.
+
 .. warning::
 
    Roles contained in collections cannot contain any modules or other plugins. All plugins in a collection must live in the collection ``plugins`` directory tree. All plugins in that tree are accessible to all roles in the collection. If you are developing new modules, we recommend distributing them in :ref:`collections <developing_collections>`, not in roles.

docs/docsite/rst/dev_guide/developing_modules_documenting.rst

@@ -21,6 +21,10 @@ Every Ansible module written in Python must begin with seven standard sections i
 
   Some older Ansible modules have ``imports`` at the bottom of the file, ``Copyright`` notices with the full GPL prefix, and/or ``DOCUMENTATION`` fields in the wrong order. These are legacy files that need updating - do not copy them into new modules. Over time we are updating and correcting older modules. Please follow the guidelines on this page!
 
+.. note:: For non-Python modules you still create a ``.py`` file for documentation purposes. Starting at ansible-core 2.14 you can instead choose to create a ``.yml`` file that has the same data structure, but in pure YAML.
+          With YAML files, the examples below are easy to use by removing Python quoting and substituting ``=`` for ``:``, for example ``DOCUMENTATION = r''' ... '''` ` to ``DOCUMENTATION: ...`` and removing closing quotes. :ref:`adjacent_yaml_doc`
+
+
 .. _shebang:
 
 Python shebang & UTF-8 coding
@@ -28,6 +32,10 @@ Python shebang & UTF-8 coding
 
 Begin your Ansible module with ``#!/usr/bin/python`` - this "shebang" allows ``ansible_python_interpreter`` to work. Follow the shebang immediately with ``# -*- coding: utf-8 -*-`` to clarify that the file is UTF-8 encoded.
 
+.. note:: Using ``#!/usr/bin/env``, makes ``env`` the interpreter and bypasses ``ansible_<interpreter>_interpreter`` logic.
+.. note:: If you develop the module using a different scripting language, adjust the interpreter accordingly (``#!/usr/bin/<interpreter>``) so ``ansible_<interpreter>_interpreter`` can work for that specific language.
+.. note:: Binary modules do not require a shebang or an interpreter.
+
 .. _copyright:
 
 Copyright and license
@@ -48,7 +56,7 @@ Additions to the module (for instance, rewrites) are not permitted to add additi
 .. code-block:: python
 
     # Copyright: Contributors to the Ansible project
-    
+
 Any legal review will include the source control history, so an exhaustive copyright header is not necessary.
 Please do not include a copyright year. If the existing copyright statement includes a year, do not edit the existing copyright year. Any existing copyright header should not be modified without permission from the copyright author.
 
@@ -82,6 +90,7 @@ Each documentation field is described below. Before committing your module docum
 * As long as your module file is :ref:`available locally <local_modules>`, you can use ``ansible-doc -t module my_module_name`` to view your module documentation at the command line. Any parsing errors will be obvious - you can view details by adding ``-vvv`` to the command.
 * You should also :ref:`test the HTML output <testing_module_documentation>` of your module documentation.
 
+
 Documentation fields
 --------------------
 

docs/docsite/rst/dev_guide/developing_plugins.rst

@@ -81,6 +81,10 @@ Configuration sources follow the precedence rules for values in Ansible. When th
 
 Plugins that support embedded documentation (see :ref:`ansible-doc` for the list) should include well-formed doc strings. If you inherit from a plugin, you must document the options it takes, either through a documentation fragment or as a copy. See :ref:`module_documenting` for more information on correct documentation. Thorough documentation is a good idea even if you're developing a plugin for local use.
 
+In ansible-core 2.14 we added support for documenting filter and test plugins. You have two options for providing documentation:
+  - Define a Python file that includes inline documentation for each plugin.
+  - Define a Python file for multiple plugins and create adjacent documentation files in YAML format.
+
 Developing particular plugin types
 ==================================
 
@@ -313,7 +317,7 @@ Filter plugins
 
 Filter plugins manipulate data. They are a feature of Jinja2 and are also available in Jinja2 templates used by the ``template`` module. As with all plugins, they can be easily extended, but instead of having a file for each one you can have several per file. Most of the filter plugins shipped with Ansible reside in a ``core.py``.
 
-Filter plugins do not use the standard configuration and documentation system described above.
+Filter plugins do not use the standard configuration system described above, but since ansible-core 2.14 can use it as plain documentation.
 
 Since Ansible evaluates variables only when they are needed, filter plugins should propagate the exceptions ``jinja2.exceptions.UndefinedError`` and ``AnsibleUndefinedVariable`` to ensure undefined variables are only fatal when necessary.
 
@@ -356,7 +360,7 @@ Here's a simple lookup plugin implementation --- this lookup returns the content
     from __future__ import (absolute_import, division, print_function)
     __metaclass__ = type
 
-    DOCUMENTATION = """
+    DOCUMENTATION = r"""
       name: file
       author: Daniel Hokka Zakrisson (@dhozac) <daniel@hozac.com>
       version_added: "0.9"  # for collections, use the collection version, not the Ansible version
@@ -451,7 +455,7 @@ Test plugins
 
 Test plugins verify data. They are a feature of Jinja2 and are also available in Jinja2 templates used by the ``template`` module. As with all plugins, they can be easily extended, but instead of having a file for each one you can have several per file. Most of the test plugins shipped with Ansible reside in a ``core.py``. These are specially useful in conjunction with some filter plugins like ``map`` and ``select``; they are also available for conditional directives like ``when:``.
 
-Test plugins do not use the standard configuration and documentation system described above.
+Test plugins do not use the standard configuration system described above. Since ansible-core 2.14 test plugins can use plain documentation.
 
 Since Ansible evaluates variables only when they are needed, test plugins should propagate the exceptions ``jinja2.exceptions.UndefinedError`` and ``AnsibleUndefinedVariable`` to ensure undefined variables are only fatal when necessary.
 
@@ -547,3 +551,5 @@ For example vars plugins, see the source code for the `vars plugins included wit
        The development mailing list
    :ref:`communication_irc`
        How to join Ansible chat channels
+   :ref:`adjacent_yaml_doc`
+       Alternate YAML files as documentation

docs/docsite/rst/dev_guide/sidecar.rst

@@ -0,0 +1,100 @@
+.. _adjacent_yaml_doc:
+
+*********************************
+Adjacent YAML documentation files
+*********************************
+
+.. contents::
+   :local:
+
+YAML documentation for plugins
+------------------------------
+For most Ansible plugins, the documentation is in the same file as the code. This approach does not work for cases when:
+
+  * Multiple plugins are defined in the same file, such as tests and filters.
+  * Plugins are written in a language other than Python (modules).
+
+These cases require plugins to provide documentation in an adjacent ``.py`` file. As of ansible-core 2.14, you can provide documentation as adjacent YAML files instead.
+The format of a YAML documentation file is nearly identical to its Python equivalent, except it is pure YAML.
+
+
+YAML format
+-----------
+In Python each section is a variable ``DOCUMENTATION = r""" ... """`` while in YAML it is a mapping key ``DOCUMENTATION: ...``.
+
+Here is a longer example that shows documentation as embedded in a Python file:
+
+.. code-block:: python
+
+  DOCUMENTATION = r'''
+    description: something
+    options:
+      option_name:
+        description: describe this config option
+        default: default value for this config option
+        env:
+          - name: NAME_OF_ENV_VAR
+        ini:
+          - section: section_of_ansible.cfg_where_this_config_option_is_defined
+            key: key_used_in_ansible.cfg
+        vars:
+          - name: name_of_ansible_var
+          - name: name_of_second_var
+            version_added: X.x
+        required: True/False
+        type: boolean/float/integer/list/none/path/pathlist/pathspec/string/tmppath
+        version_added: X.x
+  '''
+
+  EXAMPLES = r'''
+    # TODO: write examples
+  '''
+
+This example shows the same documentation in YAML format:
+
+.. code-block:: YAML
+
+  DOCUMENTATION:
+    description: something
+    options:
+      option_name:
+        description: describe this config option
+        default: default value for this config option
+        env:
+          - name: NAME_OF_ENV_VAR
+        ini:
+          - section: section_of_ansible.cfg_where_this_config_option_is_defined
+            key: key_used_in_ansible.cfg
+        vars:
+          - name: name_of_ansible_var
+          - name: name_of_second_var
+            version_added: X.x
+        required: True/False
+        type: boolean/float/integer/list/none/path/pathlist/pathspec/string/tmppath
+        version_added: X.x
+
+  EXAMPLES: # TODO: write examples
+
+As the examples above show, Python variables already contain YAML. The main change to use YAML documentation is to simply move the YAML out of such variables.
+
+ Any adjacent YAML documentation files must be in the same directory as the plugin or module that they document. This means the documentation is available in any directory that contains the plugins or modules.
+
+
+Supported plugin types
+----------------------
+YAML documentation is mainly intended for filters, tests and modules. While it is possible to use with other plugin types, Ansible always recommends having documentation in the same file as the code for most cases.
+
+.. seealso::
+
+   :ref:`list_of_collections`
+       Browse existing collections, modules, and plugins
+   :ref:`developing_api`
+       Learn about the Python API for task execution
+   :ref:`developing_inventory`
+       Learn about how to develop dynamic inventory sources
+   :ref:`developing_modules_general`
+       Learn about how to write Ansible modules
+   `Mailing List <https://groups.google.com/group/ansible-devel>`_
+       The development mailing list
+   :ref:`communication_irc`
+       How to join Ansible chat channels

docs/docsite/rst/dev_guide/testing_documentation.rst

@@ -1,14 +1,18 @@
 :orphan:
 
 .. _testing_module_documentation:
+.. _testing_plugin_documentation:
 
 ****************************
-Testing module documentation
+Testing plugin documentation
 ****************************
 
-Before you submit a module for inclusion in the main Ansible repo, you must test your module documentation for correct HTML rendering and to ensure that the argspec matches the documentation in your Python file. The community pages offer more information on :ref:`testing reStructuredText documentation <testing_documentation_locally>`.
+A quick test while developing is to use ``ansible-doc -t <plugin_type> <name>`` to see if it renders, you might need to add ``-M /path/to/module`` if the module is not somewhere Ansible expects to find it.
 
-To check the HTML output of your module documentation:
+Before you submit a plugin for inclusion in Ansible, you must test your documentation for correct HTML rendering and for modules to ensure that the argspec matches the documentation in your Python file.
+The community pages offer more information on :ref:`testing reStructuredText documentation <testing_documentation_locally>`.
+
+For example, to check the HTML output of your module documentation:
 
 #. Ensure working :ref:`development environment <environment_setup>`.
 #. Install required Python packages (drop '--user' in venv/virtualenv):
@@ -18,7 +22,7 @@ To check the HTML output of your module documentation:
       pip install --user -r requirements.txt
       pip install --user -r docs/docsite/requirements.txt
 
-#. Ensure your module is in the correct directory: ``lib/ansible/modules/$CATEGORY/mymodule.py``.
+#. Ensure your module is in the correct directory: ``lib/ansible/modules/mymodule.py`` or in a configured path.
 #. Build HTML from your module documentation: ``MODULES=mymodule make webdocs``.
 #. To build the HTML documentation for multiple modules, use a comma-separated list of module names: ``MODULES=mymodule,mymodule2 make webdocs``.
 #. View the HTML page at ``file:///path/to/docs/docsite/_build/html/modules/mymodule_module.html``.
@@ -36,3 +40,5 @@ To ensure that your module documentation matches your ``argument_spec``:
    .. code-block:: bash
 
     ansible-test sanity --test validate-modules mymodule
+
+For other plugin types the steps are similar, just adjusting names and paths to the specific type.