starting metadata sunset (#69454)

* starting metadata sunset

 - purged metadata from any requirements
 - fix indent in generic handler for yaml content (whey metadata display was off)
 - make more resilient against bad formed docs
 - removed all metadata from docs template
 - remove metadata from schemas
 - removed mdata tests and from unrelated tests

Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Rick Elrod <rick@elrod.me>

docs/docsite/rst/dev_guide/developing_modules_documenting.rst

@@ -15,11 +15,11 @@ Every Ansible module written in Python must begin with seven standard sections i
 
 .. note:: Why don't the imports go first?
 
-  Keen Python programmers may notice that contrary to PEP 8's advice we don't put ``imports`` at the top of the file. This is because the ``ANSIBLE_METADATA`` through ``RETURN`` sections are not used by the module code itself; they are essentially extra docstrings for the file. The imports are placed after these special variables for the same reason as PEP 8 puts the imports after the introductory comments and docstrings. This keeps the active parts of the code together and the pieces which are purely informational apart. The decision to exclude E402 is based on readability (which is what PEP 8 is about). Documentation strings in a module are much more similar to module level docstrings, than code, and are never utilized by the module itself. Placing the imports below this documentation and closer to the code, consolidates and groups all related code in a congruent manner to improve readability, debugging and understanding.
+  Keen Python programmers may notice that contrary to PEP 8's advice we don't put ``imports`` at the top of the file. This is because the ``DOCUMENTATION`` through ``RETURN`` sections are not used by the module code itself; they are essentially extra docstrings for the file. The imports are placed after these special variables for the same reason as PEP 8 puts the imports after the introductory comments and docstrings. This keeps the active parts of the code together and the pieces which are purely informational apart. The decision to exclude E402 is based on readability (which is what PEP 8 is about). Documentation strings in a module are much more similar to module level docstrings, than code, and are never utilized by the module itself. Placing the imports below this documentation and closer to the code, consolidates and groups all related code in a congruent manner to improve readability, debugging and understanding.
 
 .. warning:: **Copy old modules with care!**
 
-  Some older modules in Ansible Core have ``imports`` at the bottom of the file, ``Copyright`` notices with the full GPL prefix, and/or ``ANSIBLE_METADATA`` fields in the wrong order. These are legacy files that need updating - do not copy them into new modules. Over time we're updating and correcting older modules. Please follow the guidelines on this page!
+  Some older modules in Ansible Core have ``imports`` at the bottom of the file, ``Copyright`` notices with the full GPL prefix, and/or ``DOCUMENTATION`` fields in the wrong order. These are legacy files that need updating - do not copy them into new modules. Over time we're updating and correcting older modules. Please follow the guidelines on this page!
 
 .. _shebang:
 
@@ -60,61 +60,15 @@ Major additions to the module (for instance, rewrites) may add additional copyri
 ANSIBLE_METADATA block
 ======================
 
-After the shebang, the UTF-8 coding, the copyright, and the license, your module file should contain an ``ANSIBLE_METADATA`` section. This section provides information about the module for use by other tools. For new modules, the following block can be simply added into your module:
+Since we moved to collections we have deprecated the METADATA functionality, it is no longer required for modules, but it will not break anything if present.
 
-.. code-block:: python
-
-   ANSIBLE_METADATA = {'metadata_version': '1.1',
-                       'status': ['preview'],
-                       'supported_by': 'community'}
-
-.. warning::
-
-   * ``metadata_version`` is the version of the ``ANSIBLE_METADATA`` schema, *not* the version of the module.
-   * Promoting a module's ``status`` or ``supported_by`` status should only be done by members of the Ansible Core Team.
-
-Ansible metadata fields
------------------------
-
-:metadata_version: An "X.Y" formatted string. X and Y are integers which
-   define the metadata format version. Modules shipped with Ansible are
-   tied to an Ansible release, so we will only ship with a single version
-   of the metadata. We'll increment Y if we add fields or legal values
-   to an existing field. We'll increment X if we remove fields or values
-   or change the type or meaning of a field.
-   Current metadata_version is "1.1"
-
-:supported_by: Who supports the module.
-   Default value is ``community``. For information on what the support level values entail, please see
-   :ref:`Modules Support <modules_support>`. Values are:
-
-   * core
-   * network
-   * certified
-   * community
-   * curated (*deprecated value - modules in this category should be core or
-     certified instead*)
-
-:status: List of strings describing how stable the module is likely to be. See also :ref:`module_lifecycle`.
-   The default value is a single element list ["preview"]. The following strings are valid
-   statuses and have the following meanings:
-
-   :stableinterface: The module's options (the parameters or arguments it accepts) are stable. Every effort will be made not to remove options or change
-      their meaning. **Not** a rating of the module's code quality.
-   :preview: The module is in tech preview. It may be
-      unstable, the options may change, or it may require libraries or
-      web services that are themselves subject to incompatible changes.
-   :deprecated: The module is deprecated and will be removed in a future release.
-   :removed: The module is not present in the release. A stub is
-      kept so that documentation can be built. The documentation helps
-      users port from the removed module to new modules.
 
 .. _documentation_block:
 
 DOCUMENTATION block
 ===================
 
-After the shebang, the UTF-8 coding, the copyright line, the license, and the ``ANSIBLE_METADATA`` section comes the ``DOCUMENTATION`` block. Ansible's online module documentation is generated from the ``DOCUMENTATION`` blocks in each module's source code. The ``DOCUMENTATION`` block must be valid YAML. You may find it easier to start writing your ``DOCUMENTATION`` string in an :ref:`editor with YAML syntax highlighting <other_tools_and_programs>` before you include it in your Python file. You can start by copying our `example documentation string <https://github.com/ansible/ansible/blob/devel/examples/DOCUMENTATION.yml>`_ into your module file and modifying it. If you run into syntax issues in your YAML, you can validate it on the `YAML Lint <http://www.yamllint.com/>`_ website.
+After the shebang, the UTF-8 coding, the copyright line, and the license section comes the ``DOCUMENTATION`` block. Ansible's online module documentation is generated from the ``DOCUMENTATION`` blocks in each module's source code. The ``DOCUMENTATION`` block must be valid YAML. You may find it easier to start writing your ``DOCUMENTATION`` string in an :ref:`editor with YAML syntax highlighting <other_tools_and_programs>` before you include it in your Python file. You can start by copying our `example documentation string <https://github.com/ansible/ansible/blob/devel/examples/DOCUMENTATION.yml>`_ into your module file and modifying it. If you run into syntax issues in your YAML, you can validate it on the `YAML Lint <http://www.yamllint.com/>`_ website.
 
 Module documentation should briefly and accurately define what each module and option does, and how it works with others in the underlying system. Documentation should be written for broad audience--readable both by experts and non-experts.
     * Descriptions should always start with a capital letter and end with a full stop. Consistency always helps.
@@ -353,7 +307,7 @@ For example, all AWS modules should include:
 EXAMPLES block
 ==============
 
-After the shebang, the UTF-8 coding, the copyright line, the license, the ``ANSIBLE_METADATA`` section, and the ``DOCUMENTATION`` block comes the ``EXAMPLES`` block. Here you show users how your module works with real-world examples in multi-line plain-text YAML format. The best examples are ready for the user to copy and paste into a playbook. Review and update your examples with every change to your module.
+After the shebang, the UTF-8 coding, the copyright line, the license section, and the ``DOCUMENTATION`` block comes the ``EXAMPLES`` block. Here you show users how your module works with real-world examples in multi-line plain-text YAML format. The best examples are ready for the user to copy and paste into a playbook. Review and update your examples with every change to your module.
 
 Per playbook best practices, each example should include a ``name:`` line::
 
@@ -375,7 +329,7 @@ If your module returns facts that are often needed, an example of how to use the
 RETURN block
 ============
 
-After the shebang, the UTF-8 coding, the copyright line, the license, the ``ANSIBLE_METADATA`` section, ``DOCUMENTATION`` and ``EXAMPLES`` blocks comes the ``RETURN`` block. This section documents the information the module returns for use by other modules.
+After the shebang, the UTF-8 coding, the copyright line, the license section, ``DOCUMENTATION`` and ``EXAMPLES`` blocks comes the ``RETURN`` block. This section documents the information the module returns for use by other modules.
 
 If your module doesn't return anything (apart from the standard returns), this section of your module should read: ``RETURN = r''' # '''``
 Otherwise, for each value returned, provide the following fields. All fields are required unless specified otherwise.
@@ -447,7 +401,7 @@ Here are two example ``RETURN`` sections, one with three simple fields and one w
 Python imports
 ==============
 
-After the shebang, the UTF-8 coding, the copyright line, the license, and the sections for ``ANSIBLE_METADATA``, ``DOCUMENTATION``, ``EXAMPLES``, and ``RETURN``, you can finally add the python imports. All modules must use Python imports in the form:
+After the shebang, the UTF-8 coding, the copyright line, the license, and the sections for ``DOCUMENTATION``, ``EXAMPLES``, and ``RETURN``, you can finally add the python imports. All modules must use Python imports in the form:
 
 .. code-block:: python
 

docs/docsite/rst/dev_guide/developing_modules_general.rst

@@ -63,12 +63,6 @@ To create a new module:
     # Copyright: (c) 2018, Terry Jones <terry.jones@example.org>
     # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
 
-    ANSIBLE_METADATA = {
-        'metadata_version': '1.1',
-        'status': ['preview'],
-        'supported_by': 'community'
-    }
-
     DOCUMENTATION = '''
     ---
     module: my_test

docs/docsite/rst/dev_guide/module_lifecycle.rst

@@ -4,15 +4,16 @@
 The lifecycle of an Ansible module
 **********************************
 
-Modules in the main Ansible repo have a defined life cycle, from first introduction to final removal. The module life cycle is tied to the `Ansible release cycle <release_cycle>` and reflected in the :ref:`ansible_metadata_block`. A module may move through these four states:
+Modules in the main Ansible repo have a defined life cycle, from first introduction to final removal. The module life cycle is tied to the `Ansible release cycle <release_cycle>`.
+A module may move through these four states:
 
-1. When a module is first accepted into Ansible, we consider it in tech preview and mark it ``preview``. Modules in ``preview`` are not stable. You may change the parameters or dependencies, expand or reduce the functionality of ``preview`` modules. Many modules remain ``preview`` for years.
+1. When a module is first accepted into Ansible, we consider it in tech preview and will mark it as such in the documentation.
 
-2. If a module matures, we may mark it ``stableinterface`` and commit to maintaining its parameters, dependencies, and functionality. We support (though we cannot guarantee) backwards compatibility for ``stableinterface`` modules, which means their parameters should be maintained with stable meanings.
+2. If a module matures, we will remove the 'preview' mark in the documentation. We support (though we cannot guarantee) backwards compatibility for these modules, which means their parameters should be maintained with stable meanings.
 
-3. If a module's target API changes radically, or if someone creates a better implementation of its functionality, we may mark it ``deprecated``. Modules that are ``deprecated`` are still available but they are reaching the end of their life cycle. We retain deprecated modules for 4 release cycles with deprecation warnings to help users update playbooks and roles that use them.
+3. If a module's target API changes radically, or if someone creates a better implementation of its functionality, we may mark it deprecated. Modules that are deprecated are still available but they are reaching the end of their life cycle. We retain deprecated modules for 4 release cycles with deprecation warnings to help users update playbooks and roles that use them.
 
-4. When a module has been deprecated for four release cycles, we remove the code and mark the stub file ``removed``. Modules that are ``removed`` are no longer shipped with Ansible. The stub file helps users find alternative modules.
+4. When a module has been deprecated for four release cycles, we remove the code and mark the stub file removed. Modules that are removed are no longer shipped with Ansible. The stub file helps users find alternative modules.
 
 .. _deprecating_modules:
 
@@ -24,14 +25,16 @@ To deprecate a module, you must:
 1. Rename the file so it starts with an ``_``, for example, rename ``old_cloud.py`` to ``_old_cloud.py``. This keeps the module available and marks it as deprecated on the module index pages.
 2. Mention the deprecation in the relevant ``CHANGELOG``.
 3. Reference the deprecation in the relevant ``porting_guide_x.y.rst``.
-4. Update ``ANSIBLE_METADATA`` to contain ``status: ['deprecated']``.
-5. Add ``deprecated:`` to the documentation with the following sub-values:
+4. Add ``deprecated:`` to the documentation with the following sub-values:
 
-  :removed_in: A ``string``, such as ``"2.9"``; the version of Ansible where the module will be replaced with a docs-only module stub. Usually current release +4.
+  :removed_in: A ``string``, such as ``"2.10"``; the version of Ansible where the module will be replaced with a docs-only module stub. Usually current release +4.
   :why: Optional string that used to detail why this has been removed.
   :alternative: Inform users they should do instead, i.e. ``Use M(whatmoduletouseinstead) instead.``.
 
+* note: with the advent of collections and ``routing.yml`` we might soon require another entry in this file to mark the deprecation.
+
 * For an example of documenting deprecation, see this `PR that deprecates multiple modules <https://github.com/ansible/ansible/pull/43781/files>`_.
+  Some of the elements in the PR might now be out of date.
 
 Changing a module name
 ======================

docs/docsite/rst/dev_guide/testing.rst

@@ -66,11 +66,10 @@ When Shippable detects an error and it can be linked back to a file that has bee
 
    lib/ansible/modules/network/foo/bar.py:509:17: E265 block comment should start with '# '
 
-   The test `ansible-test sanity --test validate-modules` failed with the following errors:
+   The test `ansible-test sanity --test validate-modules` failed with the following error:
    lib/ansible/modules/network/foo/bar.py:0:0: E307 version_added should be 2.4. Currently 2.3
-   lib/ansible/modules/network/foo/bar.py:0:0: E316 ANSIBLE_METADATA.metadata_version: required key not provided @ data['metadata_version']. Got None
 
-From the above example we can see that ``--test pep8`` and ``--test validate-modules`` have identified issues. The commands given allow you to run the same tests locally to ensure you've fixed the issues without having to push your changed to GitHub and wait for Shippable, for example:
+From the above example we can see that ``--test pep8`` and ``--test validate-modules`` have identified an issue. The commands given allow you to run the same tests locally to ensure you've fixed all issues without having to push your changes to GitHub and wait for Shippable, for example:
 
 If you haven't already got Ansible available, use the local checkout by running::
 

docs/docsite/rst/dev_guide/testing_validate-modules.rst

@@ -80,10 +80,10 @@ Codes
   documentation-error                                          Documentation        Error                  Unknown ``DOCUMENTATION`` error
   documentation-syntax-error                                   Documentation        Error                  Invalid ``DOCUMENTATION`` schema
   illegal-future-imports                                       Imports              Error                  Only the following ``from __future__`` imports are allowed: ``absolute_import``, ``division``, and ``print_function``.
-  import-before-documentation                                  Imports              Error                  Import found before documentation variables. All imports must appear below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``/``ANSIBLE_METADATA``
+  import-before-documentation                                  Imports              Error                  Import found before documentation variables. All imports must appear below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``
   import-error                                                 Documentation        Error                  ``Exception`` attempting to import module for ``argument_spec`` introspection
-  import-placement                                             Locations            Warning                Imports should be directly below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``/``ANSIBLE_METADATA`` for legacy modules
-  imports-improper-location                                    Imports              Error                  Imports should be directly below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``/``ANSIBLE_METADATA``
+  import-placement                                             Locations            Warning                Imports should be directly below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``
+  imports-improper-location                                    Imports              Error                  Imports should be directly below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``
   incompatible-choices                                         Documentation        Error                  Choices value from the argument_spec is not compatible with type defined in the argument_spec
   incompatible-default-type                                    Documentation        Error                  Default value from the argument_spec is not compatible with type defined in the argument_spec
   invalid-argument-name                                        Documentation        Error                  Argument in argument_spec must not be one of 'message', 'syslog_facility' as it is used internally by Ansible Core Engine
@@ -93,14 +93,10 @@ Codes
   invalid-documentation-options                                Documentation        Error                  ``DOCUMENTATION.options`` must be a dictionary/hash when used
   invalid-examples                                             Documentation        Error                  ``EXAMPLES`` is not valid YAML
   invalid-extension                                            Naming               Error                  Official Ansible modules must have a ``.py`` extension for python modules or a ``.ps1`` for powershell modules
-  invalid-metadata-status                                      Documentation        Error                  ``ANSIBLE_METADATA.status`` of deprecated or removed can't include other statuses
-  invalid-metadata-type                                        Documentation        Error                  ``ANSIBLE_METADATA`` was not provided as a dict, YAML not supported, Invalid ``ANSIBLE_METADATA`` schema
-  invalid-module-deprecation-source                            Documentation        Error                  The deprecated version for the module must not be from a documentation fragment from another collection or Ansible-base
   invalid-module-schema                                        Documentation        Error                  ``AnsibleModule`` schema validation error
   invalid-requires-extension                                   Naming               Error                  Module ``#AnsibleRequires -CSharpUtil`` should not end in .cs, Module ``#Requires`` should not end in .psm1
   invalid-tagged-version                                       Documentation        Error                  All version numbers specified in code have to be explicitly tagged with the collection name, i.e. ``community.general:1.2.3`` or ``ansible.builtin:2.10``
   last-line-main-call                                          Syntax               Error                  Call to ``main()`` not the last line (or ``removed_module()`` in the case of deprecated & docs only modules)
-  metadata-changed                                             Documentation        Error                  ``ANSIBLE_METADATA`` cannot be changed in a point release for a stable branch
   missing-doc-fragment                                         Documentation        Error                  ``DOCUMENTATION`` fragment missing
   missing-existing-doc-fragment                                Documentation        Warning                Pre-existing ``DOCUMENTATION`` fragment missing
   missing-documentation                                        Documentation        Error                  No ``DOCUMENTATION`` provided
@@ -108,7 +104,6 @@ Codes
   missing-gplv3-license                                        Documentation        Error                  GPLv3 license header not found
   missing-if-name-main                                         Syntax               Error                  Next to last line is not ``if __name__ == "__main__":``
   missing-main-call                                            Syntax               Error                  Did not find a call to ``main()`` (or ``removed_module()`` in the case of deprecated & docs only modules)
-  missing-metadata                                             Documentation        Error                  No ``ANSIBLE_METADATA`` provided
   missing-module-utils-basic-import                            Imports              Warning                Did not find ``ansible.module_utils.basic`` import
   missing-module-utils-import-csharp-requirements              Imports              Error                  No ``Ansible.ModuleUtils`` or C# Ansible util requirements/imports found
   missing-powershell-interpreter                               Syntax               Error                  Interpreter line is not ``#!powershell``

docs/docsite/rst/network/dev_guide/developing_resource_modules_network.rst

@@ -53,12 +53,6 @@ For example, the resource model builder includes the ``myos_interfaces.yml`` sam
 
   ---
   GENERATOR_VERSION: '1.0'
-  ANSIBLE_METADATA: |
-      {
-          'metadata_version': '1.1',
-          'status': ['preview'],
-          'supported_by': '<support_group>'
-      }
   NETWORK_OS: myos
   RESOURCE: interfaces
   COPYRIGHT: Copyright 2019 Red Hat

docs/templates/plugin.rst.j2

@@ -418,35 +418,6 @@ Status
 - This @{ plugin_type }@ will be removed in version @{ deprecated['removed_in'] | default('') | string | rst_ify }@. *[deprecated]*
 - For more information see `DEPRECATED`_.
 
-{% else %}
-
-{%   set support = { 'core': 'the Ansible Core Team', 'network': 'the Ansible Network Team', 'certified': 'an Ansible Partner', 'community': 'the Ansible Community', 'curated': 'a Third Party'} %}
-{%   set module_states = { 'preview': 'not guaranteed to have a backwards compatible interface', 'stableinterface': 'guaranteed to have backward compatible interface changes going forward'} %}
-
-{%   if metadata %}
-{%     if metadata.status %}
-
-{%       for cur_state in metadata.status %}
-- This @{ plugin_type }@ is @{ module_states[cur_state] }@. *[@{ cur_state }@]*
-{%       endfor %}
-
-{%     endif %}
-
-{%     if metadata.supported_by %}
-{%       set supported_by = support[metadata.supported_by] %}
-- This @{ plugin_type }@ is :ref:`maintained by @{ supported_by }@ <modules_support>`. *[@{ metadata.supported_by }@]*
-
-{%       if metadata.supported_by in ('core', 'network') %}
-Red Hat Support
-~~~~~~~~~~~~~~~
-
-More information about Red Hat's support of this @{ plugin_type }@ is available from this `Red Hat Knowledge Base article <https://access.redhat.com/articles/3166901>`_.
-{%       endif %}
-
-{%     endif %}
-
-{%   endif %}
-
 {% endif %}
 
 {% if author is defined -%}

hacking/metadata-tool.py

@@ -1,537 +0,0 @@
-#!/usr/bin/env python
-# (c) 2016-2017, Toshio Kuratomi <tkuratomi@ansible.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-# Make coding more python3-ish
-from __future__ import (absolute_import, division, print_function)
-__metaclass__ = type
-
-import ast
-import csv
-import os
-import sys
-from collections import defaultdict
-from distutils.version import StrictVersion
-from pprint import pformat, pprint
-
-from ansible.parsing.metadata import DEFAULT_METADATA, ParseError, extract_metadata
-from ansible.plugins.loader import module_loader
-
-
-# There's a few files that are not new-style modules.  Have to blacklist them
-NONMODULE_PY_FILES = frozenset(('async_wrapper.py',))
-NONMODULE_MODULE_NAMES = frozenset(os.path.splitext(p)[0] for p in NONMODULE_PY_FILES)
-
-
-class MissingModuleError(Exception):
-    """Thrown when unable to find a plugin"""
-    pass
-
-
-def usage():
-    print("""Usage:
-      metadata-tool.py report [--version X]
-      metadata-tool.py add [--version X] [--overwrite] CSVFILE
-      metadata-tool.py add-default [--version X] [--overwrite]
-      medatada-tool.py upgrade [--version X]""")
-    sys.exit(1)
-
-
-def parse_args(arg_string):
-    if len(arg_string) < 1:
-        usage()
-
-    action = arg_string[0]
-
-    version = None
-    if '--version' in arg_string:
-        version_location = arg_string.index('--version')
-        arg_string.pop(version_location)
-        version = arg_string.pop(version_location)
-
-    overwrite = False
-    if '--overwrite' in arg_string:
-        overwrite = True
-        arg_string.remove('--overwrite')
-
-    csvfile = None
-    if len(arg_string) == 2:
-        csvfile = arg_string[1]
-    elif len(arg_string) > 2:
-        usage()
-
-    return action, {'version': version, 'overwrite': overwrite, 'csvfile': csvfile}
-
-
-def find_documentation(module_data):
-    """Find the DOCUMENTATION metadata for a module file"""
-    start_line = -1
-    mod_ast_tree = ast.parse(module_data)
-    for child in mod_ast_tree.body:
-        if isinstance(child, ast.Assign):
-            for target in child.targets:
-                if target.id == 'DOCUMENTATION':
-                    start_line = child.lineno - 1
-                    break
-
-    return start_line
-
-
-def remove_metadata(module_data, start_line, start_col, end_line, end_col):
-    """Remove a section of a module file"""
-    lines = module_data.split('\n')
-    new_lines = lines[:start_line]
-    if start_col != 0:
-        new_lines.append(lines[start_line][:start_col])
-
-    next_line = lines[end_line]
-    if len(next_line) - 1 != end_col:
-        new_lines.append(next_line[end_col:])
-
-    if len(lines) > end_line:
-        new_lines.extend(lines[end_line + 1:])
-    return '\n'.join(new_lines)
-
-
-def insert_metadata(module_data, new_metadata, insertion_line, targets=('ANSIBLE_METADATA',)):
-    """Insert a new set of metadata at a specified line"""
-    assignments = ' = '.join(targets)
-    pretty_metadata = pformat(new_metadata, width=1).split('\n')
-
-    new_lines = []
-    new_lines.append('{0} = {1}'.format(assignments, pretty_metadata[0]))
-
-    if len(pretty_metadata) > 1:
-        for line in pretty_metadata[1:]:
-            new_lines.append('{0}{1}'.format(' ' * (len(assignments) - 1 + len(' = {')), line))
-
-    old_lines = module_data.split('\n')
-    lines = old_lines[:insertion_line] + new_lines + old_lines[insertion_line:]
-    return '\n'.join(lines)
-
-
-def parse_assigned_metadata_initial(csvfile):
-    """
-    Fields:
-        :0: Module name
-        :1: Core (x if so)
-        :2: Extras (x if so)
-        :3: Category
-        :4: Supported/SLA
-        :5: Curated
-        :6: Stable
-        :7: Deprecated
-        :8: Notes
-        :9: Team Notes
-        :10: Notes 2
-        :11: final supported_by field
-    """
-    with open(csvfile, 'rb') as f:
-        for record in csv.reader(f):
-            module = record[0]
-
-            if record[12] == 'core':
-                supported_by = 'core'
-            elif record[12] == 'curated':
-                supported_by = 'curated'
-            elif record[12] == 'community':
-                supported_by = 'community'
-            else:
-                print('Module %s has no supported_by field.  Using community' % record[0])
-                supported_by = 'community'
-                supported_by = DEFAULT_METADATA['supported_by']
-
-            status = []
-            if record[6]:
-                status.append('stableinterface')
-            if record[7]:
-                status.append('deprecated')
-            if not status:
-                status.extend(DEFAULT_METADATA['status'])
-
-            yield (module, {'version': DEFAULT_METADATA['metadata_version'], 'supported_by': supported_by, 'status': status})
-
-
-def parse_assigned_metadata(csvfile):
-    """
-    Fields:
-        :0: Module name
-        :1: supported_by  string.  One of the valid support fields
-            core, community, certified, network
-        :2: stableinterface
-        :3: preview
-        :4: deprecated
-        :5: removed
-
-        https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_documenting.html#ansible-metadata-block
-    """
-    with open(csvfile, 'rb') as f:
-        for record in csv.reader(f):
-            module = record[0]
-            supported_by = record[1]
-
-            status = []
-            if record[2]:
-                status.append('stableinterface')
-            if record[4]:
-                status.append('deprecated')
-            if record[5]:
-                status.append('removed')
-            if not status or record[3]:
-                status.append('preview')
-
-            yield (module, {'metadata_version': '1.1', 'supported_by': supported_by, 'status': status})
-
-
-def write_metadata(filename, new_metadata, version=None, overwrite=False):
-    with open(filename, 'rb') as f:
-        module_data = f.read()
-
-    try:
-        current_metadata, start_line, start_col, end_line, end_col, targets = \
-            extract_metadata(module_data=module_data, offsets=True)
-    except SyntaxError:
-        if filename.endswith('.py'):
-            raise
-        # Probably non-python modules.  These should all have python
-        # documentation files where we can place the data
-        raise ParseError('Could not add metadata to {0}'.format(filename))
-
-    if current_metadata is None:
-        # No current metadata so we can just add it
-        start_line = find_documentation(module_data)
-        if start_line < 0:
-            if os.path.basename(filename) in NONMODULE_PY_FILES:
-                # These aren't new-style modules
-                return
-
-            raise Exception('Module file {0} had no ANSIBLE_METADATA or DOCUMENTATION'.format(filename))
-
-        module_data = insert_metadata(module_data, new_metadata, start_line, targets=('ANSIBLE_METADATA',))
-
-    elif overwrite or (version is not None and ('metadata_version' not in current_metadata or
-                                                StrictVersion(current_metadata['metadata_version']) < StrictVersion(version))):
-        # Current metadata that we do not want.  Remove the current
-        # metadata and put the new version in its place
-        module_data = remove_metadata(module_data, start_line, start_col, end_line, end_col)
-        module_data = insert_metadata(module_data, new_metadata, start_line, targets=targets)
-
-    else:
-        # Current metadata and we don't want to overwrite it
-        return
-
-    # Save the new version of the module
-    with open(filename, 'wb') as f:
-        f.write(module_data)
-
-
-def return_metadata(plugins):
-    """Get the metadata for all modules
-
-    Handle duplicate module names
-
-    :arg plugins: List of plugins to look for
-    :returns: Mapping of plugin name to metadata dictionary
-    """
-    metadata = {}
-    for name, filename in plugins:
-        # There may be several files for a module (if it is written in another
-        # language, for instance) but only one of them (the .py file) should
-        # contain the metadata.
-        if name not in metadata or metadata[name] is not None:
-            with open(filename, 'rb') as f:
-                module_data = f.read()
-            metadata[name] = extract_metadata(module_data=module_data, offsets=True)[0]
-    return metadata
-
-
-def metadata_summary(plugins, version=None):
-    """Compile information about the metadata status for a list of modules
-
-    :arg plugins: List of plugins to look for.  Each entry in the list is
-        a tuple of (module name, full path to module)
-    :kwarg version: If given, make sure the modules have this version of
-        metadata or higher.
-    :returns: A tuple consisting of a list of modules with no metadata at the
-        required version and a list of files that have metadata at the
-        required version.
-    """
-    no_metadata = {}
-    has_metadata = {}
-    supported_by = defaultdict(set)
-    status = defaultdict(set)
-    requested_version = StrictVersion(version)
-
-    all_mods_metadata = return_metadata(plugins)
-    for name, filename in plugins:
-        # Does the module have metadata?
-        if name not in no_metadata and name not in has_metadata:
-            metadata = all_mods_metadata[name]
-            if metadata is None:
-                no_metadata[name] = filename
-            elif version is not None and ('metadata_version' not in metadata or StrictVersion(metadata['metadata_version']) < requested_version):
-                no_metadata[name] = filename
-            else:
-                has_metadata[name] = filename
-
-        # What categories does the plugin belong in?
-        if all_mods_metadata[name] is None:
-            # No metadata for this module.  Use the default metadata
-            supported_by[DEFAULT_METADATA['supported_by']].add(filename)
-            status[DEFAULT_METADATA['status'][0]].add(filename)
-        else:
-            supported_by[all_mods_metadata[name]['supported_by']].add(filename)
-            for one_status in all_mods_metadata[name]['status']:
-                status[one_status].add(filename)
-
-    return list(no_metadata.values()), list(has_metadata.values()), supported_by, status
-
-# Filters to convert between metadata versions
-
-
-def convert_metadata_pre_1_0_to_1_0(metadata):
-    """
-    Convert pre-1.0 to 1.0 metadata format
-
-    :arg metadata: The old metadata
-    :returns: The new metadata
-
-    Changes from pre-1.0 to 1.0:
-    * ``version`` field renamed to ``metadata_version``
-    * ``supported_by`` field value ``unmaintained`` has been removed (change to
-      ``community`` and let an external list track whether a module is unmaintained)
-    * ``supported_by`` field value ``committer`` has been renamed to ``curated``
-    """
-    new_metadata = {'metadata_version': '1.0',
-                    'supported_by': metadata['supported_by'],
-                    'status': metadata['status']
-                    }
-    if new_metadata['supported_by'] == 'unmaintained':
-        new_metadata['supported_by'] = 'community'
-    elif new_metadata['supported_by'] == 'committer':
-        new_metadata['supported_by'] = 'curated'
-
-    return new_metadata
-
-
-def convert_metadata_1_0_to_1_1(metadata):
-    """
-    Convert 1.0 to 1.1 metadata format
-
-    :arg metadata: The old metadata
-    :returns: The new metadata
-
-    Changes from 1.0 to 1.1:
-
-    * ``supported_by`` field value ``curated`` has been removed
-    * ``supported_by`` field value ``certified`` has been added
-    * ``supported_by`` field value ``network`` has been added
-    """
-    new_metadata = {'metadata_version': '1.1',
-                    'supported_by': metadata['supported_by'],
-                    'status': metadata['status']
-                    }
-    if new_metadata['supported_by'] == 'unmaintained':
-        new_metadata['supported_by'] = 'community'
-    elif new_metadata['supported_by'] == 'curated':
-        new_metadata['supported_by'] = 'certified'
-
-    return new_metadata
-
-# Subcommands
-
-
-def add_from_csv(csv_file, version=None, overwrite=False):
-    """Implement the subcommand to add metadata from a csv file
-    """
-    # Add metadata for everything from the CSV file
-    diagnostic_messages = []
-    for module_name, new_metadata in parse_assigned_metadata(csv_file):
-        filename = module_loader.find_plugin(module_name, mod_type='.py')
-        if filename is None:
-            diagnostic_messages.append('Unable to find the module file for {0}'.format(module_name))
-            continue
-
-        try:
-            write_metadata(filename, new_metadata, version, overwrite)
-        except ParseError as e:
-            diagnostic_messages.append(e.args[0])
-            continue
-
-    if diagnostic_messages:
-        pprint(diagnostic_messages)
-
-    return 0
-
-
-def add_default(version=None, overwrite=False):
-    """Implement the subcommand to add default metadata to modules
-
-    Add the default metadata to any plugin which lacks it.
-    :kwarg version: If given, the metadata must be at least this version.
-        Otherwise, treat the module as not having existing metadata.
-    :kwarg overwrite: If True, overwrite any existing metadata.  Otherwise,
-        do not modify files which have metadata at an appropriate version
-    """
-    # List of all plugins
-    plugins = module_loader.all(path_only=True)
-    plugins = ((os.path.splitext((os.path.basename(p)))[0], p) for p in plugins)
-    plugins = (p for p in plugins if p[0] not in NONMODULE_MODULE_NAMES)
-
-    # Iterate through each plugin
-    processed = set()
-    diagnostic_messages = []
-    for name, filename in (info for info in plugins if info[0] not in processed):
-        try:
-            write_metadata(filename, DEFAULT_METADATA, version, overwrite)
-        except ParseError as e:
-            diagnostic_messages.append(e.args[0])
-            continue
-        processed.add(name)
-
-    if diagnostic_messages:
-        pprint(diagnostic_messages)
-
-    return 0
-
-
-def upgrade_metadata(version=None):
-    """Implement the subcommand to upgrade the default metadata in modules.
-
-    :kwarg version: If given, the version of the metadata to upgrade to.  If
-        not given, upgrade to the latest format version.
-    """
-    if version is None:
-        # Number larger than any of the defined metadata formats.
-        version = 9999999
-    requested_version = StrictVersion(version)
-
-    # List all plugins
-    plugins = module_loader.all(path_only=True)
-    plugins = ((os.path.splitext((os.path.basename(p)))[0], p) for p in plugins)
-    plugins = (p for p in plugins if p[0] not in NONMODULE_MODULE_NAMES)
-
-    processed = set()
-    diagnostic_messages = []
-    for name, filename in (info for info in plugins if info[0] not in processed):
-        # For each plugin, read the existing metadata
-        with open(filename, 'rb') as f:
-            module_data = f.read()
-        metadata = extract_metadata(module_data=module_data, offsets=True)[0]
-
-        # If the metadata isn't the requested version, convert it to the new
-        # version
-        if 'metadata_version' not in metadata or metadata['metadata_version'] != version:
-            #
-            # With each iteration of metadata, add a new conditional to
-            # upgrade from the previous version
-            #
-
-            if 'metadata_version' not in metadata:
-                # First version, pre-1.0 final metadata
-                metadata = convert_metadata_pre_1_0_to_1_0(metadata)
-
-            if metadata['metadata_version'] == '1.0' and StrictVersion('1.0') < requested_version:
-                metadata = convert_metadata_1_0_to_1_1(metadata)
-
-            if metadata['metadata_version'] == '1.1' and StrictVersion('1.1') < requested_version:
-                # 1.1 version => XXX.  We don't yet have anything beyond 1.1
-                # so there's nothing here
-                pass
-
-            # Replace the existing metadata with the new format
-            try:
-                write_metadata(filename, metadata, version, overwrite=True)
-            except ParseError as e:
-                diagnostic_messages.append(e.args[0])
-                continue
-
-        processed.add(name)
-
-    if diagnostic_messages:
-        pprint(diagnostic_messages)
-
-    return 0
-
-
-def report(version=None):
-    """Implement the report subcommand
-
-    Print out all the modules that have metadata and all the ones that do not.
-
-    :kwarg version: If given, the metadata must be at least this version.
-        Otherwise return it as not having metadata
-    """
-    # List of all plugins
-    plugins = module_loader.all(path_only=True)
-    plugins = ((os.path.splitext((os.path.basename(p)))[0], p) for p in plugins)
-    plugins = (p for p in plugins if p[0] not in NONMODULE_MODULE_NAMES)
-    plugins = list(plugins)
-
-    no_metadata, has_metadata, support, status = metadata_summary(plugins, version=version)
-
-    print('== Has metadata ==')
-    pprint(sorted(has_metadata))
-    print('')
-
-    print('== Has no metadata ==')
-    pprint(sorted(no_metadata))
-    print('')
-
-    print('== Supported by core ==')
-    pprint(sorted(support['core']))
-    print('== Supported by value certified ==')
-    pprint(sorted(support['certified']))
-    print('== Supported by value network ==')
-    pprint(sorted(support['network']))
-    print('== Supported by community ==')
-    pprint(sorted(support['community']))
-    print('')
-
-    print('== Status: stableinterface ==')
-    pprint(sorted(status['stableinterface']))
-    print('== Status: preview ==')
-    pprint(sorted(status['preview']))
-    print('== Status: deprecated ==')
-    pprint(sorted(status['deprecated']))
-    print('== Status: removed ==')
-    pprint(sorted(status['removed']))
-    print('')
-
-    print('== Summary ==')
-    print('No Metadata: {0}             Has Metadata: {1}'.format(len(no_metadata), len(has_metadata)))
-    print('Support level: core: {0}   community: {1}   certified: {2}   network: {3}'.format(len(support['core']),
-          len(support['community']), len(support['certified']), len(support['network'])))
-    print('Status StableInterface: {0} Status Preview: {1}            Status Deprecated: {2}      Status Removed: {3}'.format(len(status['stableinterface']),
-          len(status['preview']), len(status['deprecated']), len(status['removed'])))
-
-    return 0
-
-
-if __name__ == '__main__':
-    action, args = parse_args(sys.argv[1:])
-
-    if action == 'report':
-        rc = report(version=args['version'])
-    elif action == 'add':
-        rc = add_from_csv(args['csvfile'], version=args['version'], overwrite=args['overwrite'])
-    elif action == 'add-default':
-        rc = add_default(version=args['version'], overwrite=args['overwrite'])
-    elif action == 'upgrade':
-        rc = upgrade_metadata(version=args['version'])
-
-    sys.exit(rc)

hacking/report.py

@@ -9,7 +9,6 @@ import json
 import os
 import sqlite3
 import sys
-import yaml
 
 DATABASE_PATH = os.path.expanduser('~/.ansible/report.db')
 BASE_PATH = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')) + '/'
@@ -81,7 +80,6 @@ def populate_modules():
     module_dir = os.path.join(BASE_PATH, 'lib/ansible/modules/')
 
     modules_rows = []
-    module_statuses_rows = []
 
     for root, dir_names, file_names in os.walk(module_dir):
         for file_name in file_names:
@@ -99,29 +97,15 @@ def populate_modules():
 
             result = read_docstring(path)
 
-            metadata = result['metadata']
             doc = result['doc']
 
-            if not metadata:
-                if module == 'async_wrapper':
-                    continue
-
-                raise Exception('no metadata for: %s' % path)
-
             modules_rows.append(dict(
                 module=module,
                 namespace=namespace,
                 path=path.replace(BASE_PATH, ''),
-                supported_by=metadata['supported_by'],
                 version_added=str(doc.get('version_added', '')) if doc else '',
             ))
 
-            for status in metadata['status']:
-                module_statuses_rows.append(dict(
-                    module=module,
-                    status=status,
-                ))
-
     populate_data(dict(
         modules=dict(
             rows=modules_rows,
@@ -129,15 +113,8 @@ def populate_modules():
                 ('module', 'TEXT'),
                 ('namespace', 'TEXT'),
                 ('path', 'TEXT'),
-                ('supported_by', 'TEXT'),
                 ('version_added', 'TEXT'),
             )),
-        module_statuses=dict(
-            rows=module_statuses_rows,
-            schema=(
-                ('module', 'TEXT'),
-                ('status', 'TEXT'),
-            )),
     ))
 
 

lib/ansible/cli/doc.py

@@ -24,7 +24,6 @@ from ansible.module_utils._text import to_native, to_text
 from ansible.module_utils.common._collections_compat import Container, Sequence
 from ansible.module_utils.common.json import AnsibleJSONEncoder
 from ansible.module_utils.six import string_types
-from ansible.parsing.metadata import extract_metadata
 from ansible.parsing.plugin_docs import read_docstub
 from ansible.parsing.yaml.dumper import AnsibleDumper
 from ansible.plugins.loader import action_loader, fragment_loader
@@ -54,10 +53,6 @@ def add_collection_plugins(plugin_list, plugin_type, coll_filter=None):
         plugin_list.update(DocCLI.find_plugins(os.path.join(path, 'plugins', ptype), plugin_type, collection=collname))
 
 
-class RemovedPlugin(Exception):
-    pass
-
-
 class PluginNotFound(Exception):
     pass
 
@@ -112,12 +107,45 @@ class DocCLI(CLI):
 
         return options
 
+    def display_plugin_list(self, results):
+
+        # format for user
+        displace = max(len(x) for x in self.plugin_list)
+        linelimit = display.columns - displace - 5
+        text = []
+
+        # format display per option
+        if context.CLIARGS['list_files']:
+            # list plugin file names
+            for plugin in results.keys():
+                filename = results[plugin]
+                text.append("%-*s %-*.*s" % (displace, plugin, linelimit, len(filename), filename))
+        else:
+            # list plugin names and short desc
+            deprecated = []
+            for plugin in results.keys():
+                desc = DocCLI.tty_ify(results[plugin])
+
+                if len(desc) > linelimit:
+                    desc = desc[:linelimit] + '...'
+
+                if plugin.startswith('_'):  # Handle deprecated # TODO: add mark for deprecated collection plugins
+                    deprecated.append("%-*s %-*.*s" % (displace, plugin[1:], linelimit, len(desc), desc))
+                else:
+                    text.append("%-*s %-*.*s" % (displace, plugin, linelimit, len(desc), desc))
+
+                if len(deprecated) > 0:
+                    text.append("\nDEPRECATED:")
+                    text.extend(deprecated)
+
+        # display results
+        DocCLI.pager("\n".join(text))
+
     def run(self):
 
         super(DocCLI, self).run()
 
         plugin_type = context.CLIARGS['type']
-
         do_json = context.CLIARGS['json_format']
 
         if plugin_type in C.DOCUMENTABLE_PLUGINS:
@@ -130,6 +158,7 @@ class DocCLI(CLI):
         if basedir:
             AnsibleCollectionConfig.playbook_paths = basedir
             loader.add_directory(basedir, with_subdir=True)
+
         if context.CLIARGS['module_path']:
             for path in context.CLIARGS['module_path']:
                 if path:
@@ -162,43 +191,10 @@ class DocCLI(CLI):
             if do_json:
                 jdump(results)
             elif self.plugin_list:
-                # format for user
-                displace = max(len(x) for x in self.plugin_list)
-                linelimit = display.columns - displace - 5
-                text = []
-
-                # format display per option
-                if context.CLIARGS['list_files']:
-                    # list files
-
-                    for plugin in results.keys():
-
-                        filename = results[plugin]
-                        text.append("%-*s %-*.*s" % (displace, plugin, linelimit, len(filename), filename))
-                else:
-                    # list plugins
-                    deprecated = []
-                    for plugin in results.keys():
-                        desc = DocCLI.tty_ify(results[plugin])
-
-                        if len(desc) > linelimit:
-                            desc = desc[:linelimit] + '...'
-
-                        if plugin.startswith('_'):  # Handle deprecated # TODO: add mark for deprecated collection plugins
-                            deprecated.append("%-*s %-*.*s" % (displace, plugin[1:], linelimit, len(desc), desc))
-                        else:
-                            text.append("%-*s %-*.*s" % (displace, plugin, linelimit, len(desc), desc))
-
-                        if len(deprecated) > 0:
-                            text.append("\nDEPRECATED:")
-                            text.extend(deprecated)
-
-                # display results
-                DocCLI.pager("\n".join(text))
+                self.display_plugin_list(results)
             else:
                 display.warning("No plugins found.")
-
-        # dump plugin desc/metadata as JSON
+        # dump plugin desc/data as JSON
         elif context.CLIARGS['dump']:
             plugin_data = {}
             plugin_names = DocCLI.get_all_plugins_of_type(plugin_type)
@@ -208,7 +204,6 @@ class DocCLI(CLI):
                     plugin_data[plugin_name] = plugin_info
 
             jdump(plugin_data)
-
         else:
             # display specific plugin docs
             if len(context.CLIARGS['args']) == 0:
@@ -222,9 +217,6 @@ class DocCLI(CLI):
                 except PluginNotFound:
                     display.warning("%s %s not found in:\n%s\n" % (plugin_type, plugin, search_paths))
                     continue
-                except RemovedPlugin:
-                    display.warning("%s %s has been removed\n" % (plugin_type, plugin))
-                    continue
                 except Exception as e:
                     display.vvv(traceback.format_exc())
                     raise AnsibleError("%s %s missing documentation (or could not parse"
@@ -235,8 +227,7 @@ class DocCLI(CLI):
                     # The doc section existed but was empty
                     continue
 
-                plugin_docs[plugin] = {'doc': doc, 'examples': plainexamples,
-                                       'return': returndocs, 'metadata': metadata}
+                plugin_docs[plugin] = {'doc': doc, 'examples': plainexamples, 'return': returndocs, 'metadata': metadata}
 
             if do_json:
                 # Some changes to how json docs are formatted
@@ -257,6 +248,8 @@ class DocCLI(CLI):
                                                        doc_data['return'], doc_data['metadata'])
                     if textret:
                         text.append(textret)
+                    else:
+                        display.warning("No valid documentation was retrieved from '%s'" % plugin)
 
                 if text:
                     DocCLI.pager(''.join(text))
@@ -286,20 +279,13 @@ class DocCLI(CLI):
             collection_name = '.'.join(plugin_name.split('.')[1:3])
 
         try:
-            doc, __, __, metadata = get_docstring(filename, fragment_loader, verbose=(context.CLIARGS['verbosity'] > 0),
+            doc, __, __, __ = get_docstring(filename, fragment_loader, verbose=(context.CLIARGS['verbosity'] > 0),
                                             collection_name=collection_name, is_module=(plugin_type == 'module'))
         except Exception:
             display.vvv(traceback.format_exc())
-            raise AnsibleError(
-                "%s %s at %s has a documentation error formatting or is missing documentation." %
-                (plugin_type, plugin_name, filename))
+            raise AnsibleError("%s %s at %s has a documentation formatting error or is missing documentation." % (plugin_type, plugin_name, filename))
 
         if doc is None:
-            if 'removed' not in metadata.get('status', []):
-                raise AnsibleError(
-                    "%s %s at %s has a documentation error formatting or is missing documentation." %
-                    (plugin_type, plugin_name, filename))
-
             # Removed plugins don't have any documentation
             return None
 
@@ -339,22 +325,8 @@ class DocCLI(CLI):
             filename, fragment_loader, verbose=(context.CLIARGS['verbosity'] > 0),
             collection_name=collection_name, is_module=(plugin_type == 'module'))
 
-        # If the plugin existed but did not have a DOCUMENTATION element and was not removed, it's
-        # an error
+        # If the plugin existed but did not have a DOCUMENTATION element and was not removed, it's an error
         if doc is None:
-            # doc may be None when the module has been removed.  Calling code may choose to
-            # handle that but we can't.
-            if 'status' in metadata and isinstance(metadata['status'], Container):
-                if 'removed' in metadata['status']:
-                    raise RemovedPlugin('%s has been removed' % plugin)
-
-                # Backwards compat: no documentation but valid metadata (or no metadata, which results in using the default metadata).
-                # Probably should make this an error in 2.10
-                return {}, {}, {}, metadata
-            else:
-                # If metadata is invalid, warn but don't error
-                display.warning(u'%s has an invalid ANSIBLE_METADATA field' % plugin)
-
             raise ValueError('%s did not contain a DOCUMENTATION attribute' % plugin)
 
         doc['filename'] = filename
@@ -383,7 +355,10 @@ class DocCLI(CLI):
         if context.CLIARGS['show_snippet'] and plugin_type == 'module':
             text = DocCLI.get_snippet_text(doc)
         else:
+            try:
                 text = DocCLI.get_man_text(doc)
+            except Exception as e:
+                raise AnsibleError("Unable to retrieve documentation from '%s' due to: %s" % (plugin, to_native(e)))
 
         return text
 
@@ -450,13 +425,6 @@ class DocCLI(CLI):
                 continue
 
             if not doc or not isinstance(doc, dict):
-                with open(filename) as f:
-                    metadata = extract_metadata(module_data=f.read())
-                if metadata[0]:
-                    if 'removed' not in metadata[0].get('status', []):
-                        display.warning("%s parsing did not produce documentation." % plugin)
-                    else:
-                        continue
                 desc = 'UNDOCUMENTED'
             else:
                 desc = doc.get('short_description', 'INVALID SHORT DESCRIPTION').strip()
@@ -564,16 +532,16 @@ class DocCLI(CLI):
             aliases = ''
             if 'aliases' in opt:
                 if len(opt['aliases']) > 0:
-                    aliases = "(Aliases: " + ", ".join(str(i) for i in opt['aliases']) + ")"
+                    aliases = "(Aliases: " + ", ".join(to_text(i) for i in opt['aliases']) + ")"
                 del opt['aliases']
             choices = ''
             if 'choices' in opt:
                 if len(opt['choices']) > 0:
-                    choices = "(Choices: " + ", ".join(str(i) for i in opt['choices']) + ")"
+                    choices = "(Choices: " + ", ".join(to_text(i) for i in opt['choices']) + ")"
                 del opt['choices']
             default = ''
             if 'default' in opt or not required:
-                default = "[Default: %s" % str(opt.pop('default', '(null)')) + "]"
+                default = "[Default: %s" % to_text(opt.pop('default', '(null)')) + "]"
 
             text.append(textwrap.fill(DocCLI.tty_ify(aliases + choices + default), limit,
                                       initial_indent=opt_indent, subsequent_indent=opt_indent))
@@ -612,31 +580,6 @@ class DocCLI(CLI):
                     text.append(DocCLI._dump_yaml({k: opt[k]}, opt_indent))
             text.append('')
 
-    @staticmethod
-    def get_support_block(doc):
-        # Note: 'curated' is deprecated and not used in any of the modules we ship
-        support_level_msg = {'core': 'The Ansible Core Team',
-                             'network': 'The Ansible Network Team',
-                             'certified': 'an Ansible Partner',
-                             'community': 'The Ansible Community',
-                             'curated': 'A Third Party',
-                             }
-        return ["  * This module is maintained by %s" % support_level_msg[doc['metadata']['supported_by']]]
-
-    @staticmethod
-    def get_metadata_block(doc):
-        text = []
-
-        text.append("METADATA:")
-        text.append('\tSUPPORT LEVEL: %s' % doc['metadata']['supported_by'])
-
-        for k in (m for m in doc['metadata'] if m != 'supported_by'):
-            if isinstance(k, list):
-                text.append("\t%s: %s" % (k.capitalize(), ", ".join(doc['metadata'][k])))
-            else:
-                text.append("\t%s: %s" % (k.capitalize(), doc['metadata'][k]))
-        return text
-
     @staticmethod
     def get_man_text(doc):
 
@@ -656,7 +599,7 @@ class DocCLI(CLI):
         text.append("%s\n" % textwrap.fill(DocCLI.tty_ify(desc), limit, initial_indent=opt_indent,
                                            subsequent_indent=opt_indent))
 
-        if 'deprecated' in doc and doc['deprecated'] is not None and len(doc['deprecated']) > 0:
+        if doc.get('deprecated', False):
             text.append("DEPRECATED: \n")
             if isinstance(doc['deprecated'], dict):
                 if 'version' in doc['deprecated'] and 'removed_in' not in doc['deprecated']:
@@ -666,22 +609,15 @@ class DocCLI(CLI):
                 text.append("%s" % doc.pop('deprecated'))
             text.append("\n")
 
-        try:
-            support_block = DocCLI.get_support_block(doc)
-            if support_block:
-                text.extend(support_block)
-        except Exception:
-            pass  # FIXME: not suported by plugins
-
         if doc.pop('action', False):
             text.append("  * note: %s\n" % "This module has a corresponding action plugin.")
 
-        if 'options' in doc and doc['options']:
+        if doc.get('options', False):
             text.append("OPTIONS (= is mandatory):\n")
             DocCLI.add_fields(text, doc.pop('options'), limit, opt_indent)
             text.append('')
 
-        if 'notes' in doc and doc['notes'] and len(doc['notes']) > 0:
+        if doc.get('notes', False):
             text.append("NOTES:")
             for note in doc['notes']:
                 text.append(textwrap.fill(DocCLI.tty_ify(note), limit - 6,
@@ -690,7 +626,7 @@ class DocCLI(CLI):
             text.append('')
             del doc['notes']
 
-        if 'seealso' in doc and doc['seealso']:
+        if doc.get('seealso', False):
             text.append("SEE ALSO:")
             for item in doc['seealso']:
                 if 'module' in item:
@@ -719,7 +655,7 @@ class DocCLI(CLI):
             text.append('')
             del doc['seealso']
 
-        if 'requirements' in doc and doc['requirements'] is not None and len(doc['requirements']) > 0:
+        if doc.get('requirements', False):
             req = ", ".join(doc.pop('requirements'))
             text.append("REQUIREMENTS:%s\n" % textwrap.fill(DocCLI.tty_ify(req), limit - 16, initial_indent="  ", subsequent_indent=opt_indent))
 
@@ -732,11 +668,12 @@ class DocCLI(CLI):
             elif isinstance(doc[k], (list, tuple)):
                 text.append('%s: %s' % (k.upper(), ', '.join(doc[k])))
             else:
-                text.append(DocCLI._dump_yaml({k.upper(): doc[k]}, opt_indent))
+                # use empty indent since this affects the start of the yaml doc, not it's keys
+                text.append(DocCLI._dump_yaml({k.upper(): doc[k]}, ''))
             del doc[k]
             text.append('')
 
-        if 'plainexamples' in doc and doc['plainexamples'] is not None:
+        if doc.get('plainexamples', False):
             text.append("EXAMPLES:")
             text.append('')
             if isinstance(doc['plainexamples'], string_types):
@@ -746,20 +683,11 @@ class DocCLI(CLI):
             text.append('')
             text.append('')
 
-        if 'returndocs' in doc and doc['returndocs'] is not None:
+        if doc.get('returndocs', False):
             text.append("RETURN VALUES:")
             if isinstance(doc['returndocs'], string_types):
                 text.append(doc.pop('returndocs'))
             else:
                 text.append(yaml.dump(doc.pop('returndocs'), indent=2, default_flow_style=False))
-        text.append('')
-
-        try:
-            metadata_block = DocCLI.get_metadata_block(doc)
-            if metadata_block:
-                text.extend(metadata_block)
-                text.append('')
-        except Exception:
-            pass  # metadata is optional
 
         return "\n".join(text)

lib/ansible/galaxy/data/network/library/example_command.py.j2

@@ -22,11 +22,6 @@ __metaclass__ = type
 
 
 ### Documentation
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'community'}
-
-
 DOCUMENTATION = """
     Examples:
         https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/iosxr/iosxr_command.py

lib/ansible/galaxy/data/network/library/example_config.py.j2

@@ -22,11 +22,6 @@ __metaclass__ = type
 
 
 ### Documentation
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'community'}
-
-
 DOCUMENTATION = """
     Examples:
         https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/iosxr/iosxr_config.py

lib/ansible/galaxy/data/network/library/example_facts.py.j2

@@ -22,11 +22,6 @@ __metaclass__ = type
 
 
 ### Documentation
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'community'}
-
-
 DOCUMENTATION = """
     Examples:
         https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/iosxr/iosxr_facts.py

lib/ansible/modules/add_host.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/apt.py

@@ -10,9 +10,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = '''
 ---

lib/ansible/modules/apt_key.py

@@ -9,9 +9,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = '''
 ---

lib/ansible/modules/apt_repository.py

@@ -10,9 +10,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = '''
 ---

lib/ansible/modules/assemble.py

@@ -9,9 +9,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/assert.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/async_status.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/blockinfile.py

@@ -8,9 +8,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/command.py

@@ -9,10 +9,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
-
 DOCUMENTATION = r'''
 ---
 module: command

lib/ansible/modules/copy.py

@@ -8,9 +8,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/cron.py

@@ -11,9 +11,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/debconf.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/debug.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/dnf.py

@@ -11,11 +11,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
-
-
 DOCUMENTATION = '''
 ---
 module: dnf

lib/ansible/modules/dpkg_selections.py

@@ -8,11 +8,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
-
-
 DOCUMENTATION = '''
 ---
 module: dpkg_selections

lib/ansible/modules/expect.py

@@ -8,11 +8,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
-
-
 DOCUMENTATION = r'''
 ---
 module: expect

lib/ansible/modules/fail.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/fetch.py

@@ -9,9 +9,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/file.py

@@ -8,9 +8,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/find.py

@@ -10,9 +10,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/gather_facts.py

@@ -7,11 +7,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
-
-
 DOCUMENTATION = '''
 ---
 module: gather_facts

lib/ansible/modules/get_url.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/getent.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = '''
 ---

lib/ansible/modules/git.py

@@ -8,11 +8,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
-
-
 DOCUMENTATION = '''
 ---
 module: git

lib/ansible/modules/group.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = '''
 ---

lib/ansible/modules/group_by.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/hostname.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = '''
 ---

lib/ansible/modules/import_playbook.py

@@ -7,11 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {
-    'metadata_version': '1.1',
-    'status': ['preview'],
-    'supported_by': 'core'
-}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/import_role.py

@@ -6,11 +6,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {
-    'metadata_version': '1.1',
-    'status': ['stableinterface'],
-    'supported_by': 'core'
-}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/import_tasks.py

@@ -7,11 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {
-    'metadata_version': '1.1',
-    'status': ['stableinterface'],
-    'supported_by': 'core'
-}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/include.py

@@ -7,11 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {
-    'metadata_version': '1.1',
-    'status': ['preview'],
-    'supported_by': 'core'
-}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/include_role.py

@@ -7,11 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {
-    'metadata_version': '1.1',
-    'status': ['stableinterface'],
-    'supported_by': 'core'
-}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/include_tasks.py

@@ -7,11 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {
-    'metadata_version': '1.1',
-    'status': ['stableinterface'],
-    'supported_by': 'core'
-}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/include_vars.py

@@ -6,11 +6,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {
-    'metadata_version': '1.1',
-    'status': ['stableinterface'],
-    'supported_by': 'core'
-}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/iptables.py

@@ -8,9 +8,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/known_hosts.py

@@ -7,11 +7,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
-
-
 DOCUMENTATION = '''
 ---
 module: known_hosts

lib/ansible/modules/lineinfile.py

@@ -9,10 +9,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
-
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/meta.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 module: meta

lib/ansible/modules/package.py

@@ -9,11 +9,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
-
-
 DOCUMENTATION = '''
 ---
 module: package

lib/ansible/modules/package_facts.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = '''
 module: package_facts

lib/ansible/modules/pause.py

@@ -6,11 +6,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
-
-
 DOCUMENTATION = '''
 ---
 module: pause

lib/ansible/modules/ping.py

@@ -9,10 +9,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
-
 DOCUMENTATION = '''
 ---
 module: ping

lib/ansible/modules/pip.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = '''
 ---

lib/ansible/modules/raw.py

@@ -7,11 +7,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
-
-
 DOCUMENTATION = r'''
 ---
 module: raw

lib/ansible/modules/reboot.py

@@ -6,9 +6,6 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 module: reboot

lib/ansible/modules/replace.py

@@ -8,10 +8,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
-
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/rpm_key.py

@@ -9,9 +9,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = '''
 ---

lib/ansible/modules/script.py

@@ -4,9 +4,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/service.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/service_facts.py

@@ -9,10 +9,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
-
 DOCUMENTATION = r'''
 ---
 module: service_facts

lib/ansible/modules/set_fact.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/set_stats.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/setup.py

@@ -8,11 +8,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
-
-
 DOCUMENTATION = '''
 ---
 module: setup

lib/ansible/modules/shell.py

@@ -12,11 +12,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
-
-
 DOCUMENTATION = r'''
 ---
 module: shell

lib/ansible/modules/slurp.py

@@ -8,10 +8,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
-
 DOCUMENTATION = r'''
 ---
 module: slurp

lib/ansible/modules/stat.py

@@ -6,9 +6,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/subversion.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = '''
 ---

lib/ansible/modules/systemd.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = '''
 module: systemd

lib/ansible/modules/sysvinit.py

@@ -8,11 +8,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {
-    'metadata_version': '1.1',
-    'status': ['preview'],
-    'supported_by': 'core'
-}
 
 DOCUMENTATION = '''
 module: sysvinit

lib/ansible/modules/tempfile.py

@@ -8,9 +8,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = '''
 ---

lib/ansible/modules/template.py

@@ -9,9 +9,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/unarchive.py

@@ -11,9 +11,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/uri.py

@@ -8,10 +8,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
-
 DOCUMENTATION = r'''
 ---
 module: uri

lib/ansible/modules/user.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 module: user

lib/ansible/modules/wait_for.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/wait_for_connection.py

@@ -7,9 +7,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = r'''
 ---

lib/ansible/modules/yum.py

@@ -10,9 +10,6 @@
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
 
 DOCUMENTATION = '''
 ---

lib/ansible/modules/yum_repository.py

@@ -9,12 +9,6 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {
-    'metadata_version': '1.1',
-    'status': ['stableinterface'],
-    'supported_by': 'core'
-}
-
 DOCUMENTATION = '''
 ---
 module: yum_repository

lib/ansible/parsing/metadata.py

@@ -1,245 +0,0 @@
-# (c) 2017, Toshio Kuratomi <tkuratomi@ansible.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-# Make coding more python3-ish
-from __future__ import (absolute_import, division, print_function)
-__metaclass__ = type
-
-import ast
-import sys
-
-import yaml
-
-from ansible.module_utils._text import to_text
-
-
-# There are currently defaults for all metadata fields so we can add it
-# automatically if a file doesn't specify it
-DEFAULT_METADATA = {'metadata_version': '1.1', 'status': ['preview'], 'supported_by': 'community'}
-
-
-class ParseError(Exception):
-    """Thrown when parsing a file fails"""
-    pass
-
-
-def _seek_end_of_dict(module_data, start_line, start_col, next_node_line, next_node_col):
-    """Look for the end of a dict in a set of lines
-
-    We know the starting position of the dict and we know the start of the
-    next code node but in between there may be multiple newlines and comments.
-    There may also be multiple python statements on the same line (separated
-    by semicolons)
-
-    Examples::
-        ANSIBLE_METADATA = {[..]}
-        DOCUMENTATION = [..]
-
-        ANSIBLE_METADATA = {[..]} # Optional comments with confusing junk => {}
-        # Optional comments {}
-        DOCUMENTATION = [..]
-
-        ANSIBLE_METADATA = {
-            [..]
-            }
-        # Optional comments {}
-        DOCUMENTATION = [..]
-
-        ANSIBLE_METADATA = {[..]} ; DOCUMENTATION = [..]
-
-        ANSIBLE_METADATA = {}EOF
-    """
-    if next_node_line is None:
-        # The dict is the last statement in the file
-        snippet = module_data.splitlines()[start_line:]
-        next_node_col = 0
-        # Include the last line in the file
-        last_line_offset = 0
-    else:
-        # It's somewhere in the middle so we need to separate it from the rest
-        snippet = module_data.splitlines()[start_line:next_node_line]
-        # Do not include the last line because that's where the next node
-        # starts
-        last_line_offset = 1
-
-    if next_node_col == 0:
-        # This handles all variants where there are only comments and blank
-        # lines between the dict and the next code node
-
-        # Step backwards through all the lines in the snippet
-        for line_idx, line in tuple(reversed(tuple(enumerate(snippet))))[last_line_offset:]:
-            end_col = None
-            # Step backwards through all the characters in the line
-            for col_idx, char in reversed(tuple(enumerate(c for c in line))):
-                if not isinstance(char, bytes):
-                    # Python3 wart.  slicing a byte string yields integers
-                    char = bytes((char,))
-                if char == b'}' and end_col is None:
-                    # Potentially found the end of the dict
-                    end_col = col_idx
-
-                elif char == b'#' and end_col is not None:
-                    # The previous '}' was part of a comment.  Keep trying
-                    end_col = None
-
-            if end_col is not None:
-                # Found the end!
-                end_line = start_line + line_idx
-                break
-        else:
-            raise ParseError('Unable to find the end of dictionary')
-    else:
-        # Harder cases involving multiple statements on one line
-        # Good Ansible Module style doesn't do this so we're just going to
-        # treat this as an error for now:
-        raise ParseError('Multiple statements per line confuses the module metadata parser.')
-
-    return end_line, end_col
-
-
-def _seek_end_of_string(module_data, start_line, start_col, next_node_line, next_node_col):
-    """
-    This is much trickier than finding the end of a dict.  A dict has only one
-    ending character, "}".  Strings have four potential ending characters.  We
-    have to parse the beginning of the string to determine what the ending
-    character will be.
-
-    Examples:
-        ANSIBLE_METADATA = '''[..]''' # Optional comment with confusing chars '''
-        # Optional comment with confusing chars '''
-        DOCUMENTATION = [..]
-
-        ANSIBLE_METADATA = '''
-            [..]
-            '''
-        DOCUMENTATIONS = [..]
-
-        ANSIBLE_METADATA = '''[..]''' ; DOCUMENTATION = [..]
-
-        SHORT_NAME = ANSIBLE_METADATA = '''[..]''' ; DOCUMENTATION = [..]
-
-    String marker variants:
-        * '[..]'
-        * "[..]"
-        * '''[..]'''
-        * \"\"\"[..]\"\"\"
-
-    Each of these come in u, r, and b variants:
-        * '[..]'
-        * u'[..]'
-        * b'[..]'
-        * r'[..]'
-        * ur'[..]'
-        * ru'[..]'
-        * br'[..]'
-        * b'[..]'
-        * rb'[..]'
-    """
-    raise NotImplementedError('Finding end of string not yet implemented')
-
-
-def extract_metadata(module_ast=None, module_data=None, offsets=False):
-    """Extract the metadata from a module
-
-    :kwarg module_ast: ast representation of the module.  At least one of this
-        or ``module_data`` must be given.  If the code calling
-        :func:`extract_metadata` has already parsed the module_data into an ast,
-        giving the ast here will save reparsing it.
-    :kwarg module_data: Byte string containing a module's code.  At least one
-        of this or ``module_ast`` must be given.
-    :kwarg offsets: If set to True, offests into the source code will be
-        returned.  This requires that ``module_data`` be set.
-    :returns: a tuple of metadata (a dict), line the metadata starts on,
-        column the metadata starts on, line the metadata ends on, column the
-        metadata ends on, and the names the metadata is assigned to.  One of
-        the names the metadata is assigned to will be ANSIBLE_METADATA.  If no
-        metadata is found, the tuple will be (None, -1, -1, -1, -1, None).
-        If ``offsets`` is False then the tuple will consist of
-        (metadata, -1, -1, -1, -1, None).
-    :raises ansible.parsing.metadata.ParseError: if ``module_data`` does not parse
-    :raises SyntaxError: if ``module_data`` is needed but does not parse correctly
-    """
-    if offsets and module_data is None:
-        raise TypeError('If offsets is True then module_data must also be given')
-
-    if module_ast is None and module_data is None:
-        raise TypeError('One of module_ast or module_data must be given')
-
-    metadata = None
-    start_line = -1
-    start_col = -1
-    end_line = -1
-    end_col = -1
-    targets = None
-    if module_ast is None:
-        module_ast = ast.parse(module_data)
-
-    for root_idx, child in reversed(list(enumerate(module_ast.body))):
-        if isinstance(child, ast.Assign):
-            for target in child.targets:
-                if isinstance(target, ast.Name) and target.id == 'ANSIBLE_METADATA':
-                    metadata = ast.literal_eval(child.value)
-                    if not offsets:
-                        continue
-
-                    try:
-                        # Determine where the next node starts
-                        next_node = module_ast.body[root_idx + 1]
-                        next_lineno = next_node.lineno
-                        next_col_offset = next_node.col_offset
-                    except IndexError:
-                        # Metadata is defined in the last node of the file
-                        next_lineno = None
-                        next_col_offset = None
-
-                    if isinstance(child.value, ast.Dict):
-                        # Determine where the current metadata ends
-                        end_line, end_col = _seek_end_of_dict(module_data,
-                                                              child.lineno - 1,
-                                                              child.col_offset,
-                                                              next_lineno,
-                                                              next_col_offset)
-
-                    elif isinstance(child.value, ast.Str):
-                        metadata = yaml.safe_load(child.value.s)
-                        end_line, end_col = _seek_end_of_string(module_data,
-                                                                child.lineno - 1,
-                                                                child.col_offset,
-                                                                next_lineno,
-                                                                next_col_offset)
-                    elif isinstance(child.value, ast.Bytes):
-                        metadata = yaml.safe_load(to_text(child.value.s, errors='surrogate_or_strict'))
-                        end_line, end_col = _seek_end_of_string(module_data,
-                                                                child.lineno - 1,
-                                                                child.col_offset,
-                                                                next_lineno,
-                                                                next_col_offset)
-                    else:
-                        raise ParseError('Ansible plugin metadata must be a dict')
-
-                    # Do these after the if-else so we don't pollute them in
-                    # case this was a false positive
-                    start_line = child.lineno - 1
-                    start_col = child.col_offset
-                    targets = [t.id for t in child.targets]
-                    break
-
-        if metadata is not None:
-            # Once we've found the metadata we're done
-            break
-
-    return metadata, start_line, start_col, end_line, end_col, targets

lib/ansible/parsing/plugin_docs.py

@@ -5,17 +5,18 @@ from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
 import ast
-import yaml
 
 from ansible.module_utils._text import to_text
-from ansible.parsing.metadata import extract_metadata
 from ansible.parsing.yaml.loader import AnsibleLoader
 from ansible.utils.display import Display
 
 display = Display()
 
 
+# NOTE: should move to just reading the variable as we do in plugin_loader since we already load as a 'module'
+# which is much faster than ast parsing ourselves.
 def read_docstring(filename, verbose=True, ignore_errors=True):
+
     """
     Search for assignment of the DOCUMENTATION and EXAMPLES variables in the given file.
     Parse DOCUMENTATION from YAML and return the YAML doc or None together with EXAMPLES, as plain text.
@@ -25,7 +26,7 @@ def read_docstring(filename, verbose=True, ignore_errors=True):
         'doc': None,
         'plainexamples': None,
         'returndocs': None,
-        'metadata': None,
+        'metadata': None,  # NOTE: not used anymore, kept for compat
         'seealso': None,
     }
 
@@ -33,6 +34,7 @@ def read_docstring(filename, verbose=True, ignore_errors=True):
         'DOCUMENTATION': 'doc',
         'EXAMPLES': 'plainexamples',
         'RETURN': 'returndocs',
+        'ANSIBLE_METADATA': 'metadata',  # NOTE: now unused, but kept for backwards compat
     }
 
     try:
@@ -54,33 +56,16 @@ def read_docstring(filename, verbose=True, ignore_errors=True):
                         if isinstance(child.value, ast.Dict):
                             data[varkey] = ast.literal_eval(child.value)
                         else:
-                            if theid == 'DOCUMENTATION':
-                                # string should be yaml
-                                data[varkey] = AnsibleLoader(child.value.s, file_name=filename).get_single_data()
-                            else:
-                                # not yaml, should be a simple string
+                            if theid in ['EXAMPLES', 'RETURN']:
+                                # examples 'can' be yaml, return must be, but even if so, we dont want to parse as such here
+                                # as it can create undesired 'objects' that don't display well as docs.
                                 data[varkey] = to_text(child.value.s)
-                        display.debug('assigned :%s' % varkey)
-
-        # Metadata is per-file and a dict rather than per-plugin/function and yaml
-        data['metadata'] = extract_metadata(module_ast=M)[0]
-
-        if data['metadata']:
-            # remove version
-            for field in ('version', 'metadata_version'):
-                if field in data['metadata']:
-                    del data['metadata'][field]
-
-            if 'supported_by' not in data['metadata']:
-                data['metadata']['supported_by'] = 'community'
+                            else:
+                                # string should be yaml if already not a dict
+                                data[varkey] = AnsibleLoader(child.value.s, file_name=filename).get_single_data()
 
-            if 'status' not in data['metadata']:
-                data['metadata']['status'] = ['preview']
+                        display.debug('assigned: %s' % varkey)
 
-        else:
-            # Add default metadata
-            data['metadata'] = {'supported_by': 'community',
-                                'status': ['preview']}
     except Exception:
         if verbose:
             display.error("unable to parse %s" % filename)

lib/ansible/plugins/filter/urlsplit.py

@@ -6,13 +6,6 @@ from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
 
-ANSIBLE_METADATA = {
-    'metadata_version': '1.1',
-    'status': ['preview'],
-    'supported_by': 'core'
-}
-
-
 from ansible.errors import AnsibleFilterError
 from ansible.module_utils.six.moves.urllib.parse import urlsplit
 from ansible.utils import helpers

lib/ansible/plugins/inventory/toml.py

@@ -4,10 +4,6 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['preview'],
-                    'supported_by': 'core'}
-
 DOCUMENTATION = '''
     inventory: toml
     version_added: "2.8"

test/integration/targets/ansible-doc/fakemodule.output

@@ -2,7 +2,6 @@
 
         this is a fake module
 
-  * This module is maintained by The Ansible Community
 OPTIONS (= is mandatory):
 
 - _notreal
@@ -11,10 +10,6 @@ OPTIONS (= is mandatory):
 
 
 AUTHOR: me
-        METADATA:
-          status:
-          - preview
-          supported_by: community
 
 SHORT_DESCIPTOIN: fake module
 

test/integration/targets/ansible-doc/test.yml

@@ -49,6 +49,7 @@
     - name: documented module with removed status
       command: ansible-doc test_docs_removed_status
       register: result
+
     - assert:
         that:
           - '"WARNING" not in result.stderr'
@@ -58,41 +59,17 @@
     - name: empty module
       command: ansible-doc test_empty
       register: result
+      ignore_errors: true
+
     - assert:
         that:
-          - 'result.stdout == ""'
-          - 'result.stderr == ""'
+          - result is failed
 
     - name: module with no documentation
       command: ansible-doc test_no_docs
       register: result
-    - assert:
-        that:
-          - 'result.stdout == ""'
-          - 'result.stderr == ""'
+      ignore_errors: true
 
-    - name: module with no documentation and no metadata
-      command: ansible-doc test_no_docs_no_metadata
-      register: result
-    - assert:
-        that:
-          - 'result.stdout == ""'
-          - 'result.stderr == ""'
-
-    - name: module with no documentation and no status in metadata
-      command: ansible-doc test_no_docs_no_status
-      ignore_errors: yes
-      register: result
-    - assert:
-        that:
-          - 'result.stdout == ""'
-          - 'result.stderr == ""'
-
-    - name: module with no documentation and non-iterable status in metadata
-      command: ansible-doc test_no_docs_non_iterable_status
-      ignore_errors: yes
-      register: result
     - assert:
         that:
-          - 'result is failed'
-          - '"ERROR! module test_no_docs_non_iterable_status missing documentation (or could not parse documentation): test_no_docs_non_iterable_status did not contain a DOCUMENTATION attribute" in result.stderr'
+          - result is failed

test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/main.py

@@ -41,7 +41,7 @@ import yaml
 from ansible import __version__ as ansible_version
 from ansible.executor.module_common import REPLACER_WINDOWS
 from ansible.module_utils.common._collections_compat import Mapping
-from ansible.module_utils._text import to_bytes, to_native
+from ansible.module_utils._text import to_native
 from ansible.plugins.loader import fragment_loader
 from ansible.utils.collection_loader._collection_finder import _AnsibleCollectionFinder
 from ansible.utils.plugin_docs import BLACKLIST, tag_versions_and_dates, add_fragments, get_docstring
@@ -49,7 +49,7 @@ from ansible.utils.version import SemanticVersion
 
 from .module_args import AnsibleModuleImportError, AnsibleModuleNotInitialized, get_argument_spec
 
-from .schema import ansible_module_kwargs_schema, doc_schema, metadata_1_1_schema, return_schema
+from .schema import ansible_module_kwargs_schema, doc_schema, return_schema
 
 from .utils import CaptureStd, NoArgsAnsibleModule, compare_unordered_lists, is_empty, parse_yaml, parse_isodate
 from voluptuous.humanize import humanize_error
@@ -696,7 +696,7 @@ class ModuleValidator(Validator):
                         code='import-before-documentation',
                         msg=('Import found before documentation variables. '
                              'All imports must appear below '
-                             'DOCUMENTATION/EXAMPLES/RETURN/ANSIBLE_METADATA.'),
+                             'DOCUMENTATION/EXAMPLES/RETURN.'),
                         line=child.lineno
                     )
                     break
@@ -713,8 +713,7 @@ class ModuleValidator(Validator):
                                 code='import-before-documentation',
                                 msg=('Import found before documentation '
                                      'variables. All imports must appear below '
-                                     'DOCUMENTATION/EXAMPLES/RETURN/'
-                                     'ANSIBLE_METADATA.'),
+                                     'DOCUMENTATION/EXAMPLES/RETURN.'),
                                 line=child.lineno
                             )
                             break
@@ -724,7 +723,7 @@ class ModuleValidator(Validator):
                 msg = (
                     'import-placement',
                     ('Imports should be directly below DOCUMENTATION/EXAMPLES/'
-                     'RETURN/ANSIBLE_METADATA.')
+                     'RETURN.')
                 )
                 if self._is_new_module():
                     self.reporter.error(
@@ -829,11 +828,6 @@ class ModuleValidator(Validator):
                 'lineno': 0,
                 'end_lineno': 0,
             },
-            'ANSIBLE_METADATA': {
-                'value': None,
-                'lineno': 0,
-                'end_lineno': 0,
-            }
         }
         for child in self.ast.body:
             if isinstance(child, ast.Assign):
@@ -859,17 +853,6 @@ class ModuleValidator(Validator):
                         docs['RETURN']['end_lineno'] = (
                             child.lineno + len(child.value.s.splitlines())
                         )
-                    elif grandchild.id == 'ANSIBLE_METADATA':
-                        docs['ANSIBLE_METADATA']['value'] = child.value
-                        docs['ANSIBLE_METADATA']['lineno'] = child.lineno
-                        try:
-                            docs['ANSIBLE_METADATA']['end_lineno'] = (
-                                child.lineno + len(child.value.s.splitlines())
-                            )
-                        except AttributeError:
-                            docs['ANSIBLE_METADATA']['end_lineno'] = (
-                                child.value.values[-1].lineno
-                            )
 
         return docs
 
@@ -939,44 +922,6 @@ class ModuleValidator(Validator):
         if self.object_name.startswith('_') and not os.path.islink(self.object_path):
             filename_deprecated_or_removed = True
 
-        # Have to check the metadata first so that we know if the module is removed or deprecated
-        metadata = None
-        if not self.collection:
-            if not bool(doc_info['ANSIBLE_METADATA']['value']):
-                self.reporter.error(
-                    path=self.object_path,
-                    code='missing-metadata',
-                    msg='No ANSIBLE_METADATA provided'
-                )
-            else:
-                if isinstance(doc_info['ANSIBLE_METADATA']['value'], ast.Dict):
-                    metadata = ast.literal_eval(
-                        doc_info['ANSIBLE_METADATA']['value']
-                    )
-                else:
-                    self.reporter.error(
-                        path=self.object_path,
-                        code='missing-metadata-format',
-                        msg='ANSIBLE_METADATA was not provided as a dict, YAML not supported'
-                    )
-
-            if metadata:
-                self._validate_docs_schema(metadata, metadata_1_1_schema(),
-                                           'ANSIBLE_METADATA', 'invalid-metadata-type')
-                # We could validate these via the schema if we knew what the values are ahead of
-                # time.  We can figure that out for deprecated but we can't for removed.  Only the
-                # metadata has that information.
-                if 'removed' in metadata['status']:
-                    removed = True
-                if 'deprecated' in metadata['status']:
-                    deprecated = True
-                if (deprecated or removed) and len(metadata['status']) > 1:
-                    self.reporter.error(
-                        path=self.object_path,
-                        code='missing-metadata-status',
-                        msg='ANSIBLE_METADATA.status must be exactly one of "deprecated" or "removed"'
-                    )
-        else:
         # We are testing a collection
         if self.routing:
             routing_deprecation = self.routing.get('plugin_routing', {}).get('modules', {}).get(self.name, {}).get('deprecation', {})
@@ -1080,7 +1025,7 @@ class ModuleValidator(Validator):
                         )
 
                     if not self.collection:
-                        existing_doc = self._check_for_new_args(doc, metadata)
+                        existing_doc = self._check_for_new_args(doc)
                         self._check_version_added(doc, existing_doc)
 
             if not bool(doc_info['EXAMPLES']['value']):
@@ -1152,7 +1097,7 @@ class ModuleValidator(Validator):
                 self.reporter.error(
                     path=self.object_path,
                     code='deprecation-mismatch',
-                    msg='Module deprecation/removed must agree in Metadata, by prepending filename with'
+                    msg='Module deprecation/removed must agree in documentaiton, by prepending filename with'
                         ' "_", and setting DOCUMENTATION.deprecated for deprecation or by removing all'
                         ' documentation for removed'
                 )
@@ -2003,7 +1948,7 @@ class ModuleValidator(Validator):
                     msg=msg
                 )
 
-    def _check_for_new_args(self, doc, metadata):
+    def _check_for_new_args(self, doc):
         if not self.base_branch or self._is_new_module():
             return
 
@@ -2038,16 +1983,6 @@ class ModuleValidator(Validator):
         except ValueError:
             mod_version_added = self._create_strict_version('0.0')
 
-        if self.base_branch and 'stable-' in self.base_branch:
-            metadata.pop('metadata_version', None)
-            metadata.pop('version', None)
-            if metadata != existing_metadata:
-                self.reporter.error(
-                    path=self.object_path,
-                    code='metadata-changed',
-                    msg=('ANSIBLE_METADATA cannot be changed in a point release for a stable branch')
-                )
-
         options = doc.get('options', {}) or {}
 
         should_be = '.'.join(ansible_version.split('.')[:2])
@@ -2146,24 +2081,10 @@ class ModuleValidator(Validator):
             doc_info, docs = self._validate_docs()
 
             # See if current version => deprecated.removed_in, ie, should be docs only
-            if isinstance(doc_info['ANSIBLE_METADATA']['value'], ast.Dict) and 'removed' in ast.literal_eval(doc_info['ANSIBLE_METADATA']['value'])['status']:
-                end_of_deprecation_should_be_removed_only = True
-            elif docs and 'deprecated' in docs and docs['deprecated'] is not None:
-                end_of_deprecation_should_be_removed_only = False
-                if 'removed_at_date' in docs['deprecated']:
-                    try:
-                        removed_at_date = docs['deprecated']['removed_at_date']
-                        if parse_isodate(removed_at_date) < datetime.date.today():
-                            msg = "Module's deprecated.removed_at_date date '%s' is before today" % removed_at_date
-                            self.reporter.error(
-                                path=self.object_path,
-                                code='deprecated-date',
-                                msg=msg,
-                            )
-                    except ValueError:
-                        # Already checked during schema validation
-                        pass
+            if docs and docs.get('deprecated', False):
+
                 if 'removed_in' in docs['deprecated']:
+                    removed_in = None
                     try:
                         collection_name, version = self._split_tagged_version(docs['deprecated']['removed_in'])
                         if collection_name != self.collection_name:
@@ -2172,20 +2093,31 @@ class ModuleValidator(Validator):
                                 code='invalid-module-deprecation-source',
                                 msg=('The deprecation version for a module must be added in this collection')
                             )
-                            # Treat the module as not to be removed:
-                            raise ValueError('')
-                        removed_in = self._create_strict_version(str(version))
-                    except ValueError:
-                        end_of_deprecation_should_be_removed_only = False
                         else:
+                            removed_in = self.StrictVersion(str(version))
+
+                    except ValueError:
+                        # ignore and hope we previouslly reported
+                        pass
+
+                    if removed_in:
                         if not self.collection:
-                            strict_ansible_version = self._create_strict_version('.'.join(ansible_version.split('.')[:2]))
+                            strict_ansible_version = self.StrictVersion('.'.join(ansible_version.split('.')[:2]))
                             end_of_deprecation_should_be_removed_only = strict_ansible_version >= removed_in
                         elif self.collection_version:
                             strict_ansible_version = self.collection_version
                             end_of_deprecation_should_be_removed_only = strict_ansible_version >= removed_in
-                        else:
-                            end_of_deprecation_should_be_removed_only = False
+
+                # handle deprecation by date
+                if 'removed_at_date' in docs['deprecated']:
+                    try:
+                        removed_at_date = docs['deprecated']['removed_at_date']
+                        if parse_isodate(removed_at_date) < datetime.date.today():
+                            msg = "Module's deprecated.removed_at_date date '%s' is before today" % removed_at_date
+                            self.reporter.error(path=self.object_path, code='deprecated-date', msg=msg)
+                    except ValueError:
+                        # ignore and hope we previouslly reported
+                        pass
 
         if self._python_module() and not self._just_docs() and not end_of_deprecation_should_be_removed_only:
             self._validate_ansible_module_call(docs)

test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/schema.py

@@ -442,32 +442,6 @@ def doc_schema(module_name, for_collection=False, deprecated_module=False):
     )
 
 
-def metadata_1_0_schema(deprecated):
-    valid_status = Any('stableinterface', 'preview', 'deprecated', 'removed')
-    if deprecated:
-        valid_status = Any('deprecated')
-
-    return Schema(
-        {
-            Required('status'): [valid_status],
-            Required('metadata_version'): '1.0',
-            Required('supported_by'): Any('core', 'community', 'curated')
-        }
-    )
-
-
-def metadata_1_1_schema():
-    valid_status = Any('stableinterface', 'preview', 'deprecated', 'removed')
-
-    return Schema(
-        {
-            Required('status'): [valid_status],
-            Required('metadata_version'): '1.1',
-            Required('supported_by'): Any('core', 'community', 'certified', 'network')
-        }
-    )
-
-
 # Things to add soon
 ####################
 # 1) Recursively validate `type: complex` fields

test/units/parsing/test_metadata.py

@@ -1,239 +0,0 @@
-# coding: utf-8
-# (c) 2017, Toshio Kuratomi <tkuratomi@ansible.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-# Make coding more python3-ish
-from __future__ import (absolute_import, division, print_function)
-__metaclass__ = type
-
-import ast
-
-import pytest
-
-from ansible.parsing import metadata as md
-
-
-LICENSE = b"""# some license text boilerplate
-# That we have at the top of files
-"""
-
-FUTURE_IMPORTS = b"""
-from __future__ import (absolute_import, division, print_function)
-"""
-
-REGULAR_IMPORTS = b"""
-import test
-from foo import bar
-"""
-
-STANDARD_METADATA = b"""
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'}
-"""
-
-TEXT_STD_METADATA = b"""
-ANSIBLE_METADATA = u'''
-metadata_version: '1.1'
-status:
-  - 'stableinterface'
-supported_by: 'core'
-'''
-"""
-
-BYTES_STD_METADATA = b"""
-ANSIBLE_METADATA = b'''
-metadata_version: '1.1'
-status:
-  - 'stableinterface'
-supported_by: 'core'
-'''
-"""
-
-TRAILING_COMMENT_METADATA = b"""
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'} # { Testing }
-"""
-
-MULTIPLE_STATEMENTS_METADATA = b"""
-DOCUMENTATION = "" ; ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core'} ; RETURNS = ""
-"""
-
-EMBEDDED_COMMENT_METADATA = b"""
-ANSIBLE_METADATA = {'metadata_version': '1.1',
-                    'status': ['stableinterface'],
-                    # { Testing }
-                    'supported_by': 'core'}
-"""
-
-HASH_SYMBOL_METADATA = b"""
-ANSIBLE_METADATA = {'metadata_version': '1.1 # 4',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core # Testing '}
-"""
-
-HASH_SYMBOL_METADATA = b"""
-ANSIBLE_METADATA = {'metadata_version': '1.1 # 4',
-                    'status': ['stableinterface'],
-                    'supported_by': 'core # Testing '}
-"""
-
-HASH_COMBO_METADATA = b"""
-ANSIBLE_METADATA = {'metadata_version': '1.1 # 4',
-                    'status': ['stableinterface'],
-                    # { Testing }
-                    'supported_by': 'core'} # { Testing }
-"""
-
-METADATA = {'metadata_version': '1.1', 'status': ['stableinterface'], 'supported_by': 'core'}
-HASH_SYMBOL_METADATA = {'metadata_version': '1.1 # 4', 'status': ['stableinterface'], 'supported_by': 'core'}
-
-METADATA_EXAMPLES = (
-    # Standard import
-    (LICENSE + FUTURE_IMPORTS + STANDARD_METADATA + REGULAR_IMPORTS,
-     (METADATA, 5, 0, 7, 42, ['ANSIBLE_METADATA'])),
-    # Metadata at end of file
-    (LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + STANDARD_METADATA.rstrip(),
-     (METADATA, 8, 0, 10, 42, ['ANSIBLE_METADATA'])),
-    # Metadata at beginning of file
-    (STANDARD_METADATA + LICENSE + REGULAR_IMPORTS,
-     (METADATA, 1, 0, 3, 42, ['ANSIBLE_METADATA'])),
-
-    # Standard import with a trailing comment
-    (LICENSE + FUTURE_IMPORTS + TRAILING_COMMENT_METADATA + REGULAR_IMPORTS,
-     (METADATA, 5, 0, 7, 42, ['ANSIBLE_METADATA'])),
-    # Metadata at end of file with a trailing comment
-    (LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + TRAILING_COMMENT_METADATA.rstrip(),
-     (METADATA, 8, 0, 10, 42, ['ANSIBLE_METADATA'])),
-    # Metadata at beginning of file with a trailing comment
-    (TRAILING_COMMENT_METADATA + LICENSE + REGULAR_IMPORTS,
-     (METADATA, 1, 0, 3, 42, ['ANSIBLE_METADATA'])),
-
-    # FIXME: Current code cannot handle multiple statements on the same line.
-    # This is bad style so we're just going to ignore it for now
-    # Standard import with other statements on the same line
-    # (LICENSE + FUTURE_IMPORTS + MULTIPLE_STATEMENTS_METADATA + REGULAR_IMPORTS,
-    #  (METADATA, 5, 0, 7, 42, ['ANSIBLE_METADATA'])),
-    # Metadata at end of file with other statements on the same line
-    # (LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + MULTIPLE_STATEMENTS_METADATA.rstrip(),
-    #  (METADATA, 8, 0, 10, 42, ['ANSIBLE_METADATA'])),
-    # Metadata at beginning of file with other statements on the same line
-    # (MULTIPLE_STATEMENTS_METADATA + LICENSE + REGULAR_IMPORTS,
-    #  (METADATA, 1, 0, 3, 42, ['ANSIBLE_METADATA'])),
-
-    # Standard import with comment inside the metadata
-    (LICENSE + FUTURE_IMPORTS + EMBEDDED_COMMENT_METADATA + REGULAR_IMPORTS,
-     (METADATA, 5, 0, 8, 42, ['ANSIBLE_METADATA'])),
-    # Metadata at end of file with comment inside the metadata
-    (LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + EMBEDDED_COMMENT_METADATA.rstrip(),
-     (METADATA, 8, 0, 11, 42, ['ANSIBLE_METADATA'])),
-    # Metadata at beginning of file with comment inside the metadata
-    (EMBEDDED_COMMENT_METADATA + LICENSE + REGULAR_IMPORTS,
-     (METADATA, 1, 0, 4, 42, ['ANSIBLE_METADATA'])),
-
-    # FIXME: Current code cannot handle hash symbols in the last element of
-    # the metadata.  Fortunately, the metadata currently fully specifies all
-    # the strings inside of metadata and none of them can contain a hash.
-    # Need to fix this to future-proof it against strings containing hashes
-    # Standard import with hash symbol in metadata
-    # (LICENSE + FUTURE_IMPORTS + HASH_SYMBOL_METADATA + REGULAR_IMPORTS,
-    #  (HASH_SYMBOL_METADATA, 5, 0, 7, 53, ['ANSIBLE_METADATA'])),
-    # Metadata at end of file with hash symbol in metadata
-    # (LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + HASH_SYMBOL_HASH_SYMBOL_METADATA.rstrip(),
-    #  (HASH_SYMBOL_METADATA, 8, 0, 10, 53, ['ANSIBLE_METADATA'])),
-    # Metadata at beginning of file with hash symbol in metadata
-    # (HASH_SYMBOL_HASH_SYMBOL_METADATA + LICENSE + REGULAR_IMPORTS,
-    #  (HASH_SYMBOL_METADATA, 1, 0, 3, 53, ['ANSIBLE_METADATA'])),
-
-    # Standard import with a bunch of hashes everywhere
-    (LICENSE + FUTURE_IMPORTS + HASH_COMBO_METADATA + REGULAR_IMPORTS,
-     (HASH_SYMBOL_METADATA, 5, 0, 8, 42, ['ANSIBLE_METADATA'])),
-    # Metadata at end of file with a bunch of hashes everywhere
-    (LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + HASH_COMBO_METADATA.rstrip(),
-     (HASH_SYMBOL_METADATA, 8, 0, 11, 42, ['ANSIBLE_METADATA'])),
-    # Metadata at beginning of file with a bunch of hashes everywhere
-    (HASH_COMBO_METADATA + LICENSE + REGULAR_IMPORTS,
-     (HASH_SYMBOL_METADATA, 1, 0, 4, 42, ['ANSIBLE_METADATA'])),
-
-    # Standard import with a junk ANSIBLE_METADATA as well
-    (LICENSE + FUTURE_IMPORTS + b"\nANSIBLE_METADATA = 10\n" + HASH_COMBO_METADATA + REGULAR_IMPORTS,
-     (HASH_SYMBOL_METADATA, 7, 0, 10, 42, ['ANSIBLE_METADATA'])),
-)
-
-# FIXME: String/yaml metadata is not implemented yet.  Need more test cases once it is implemented
-STRING_METADATA_EXAMPLES = (
-    # Standard import
-    (LICENSE + FUTURE_IMPORTS + TEXT_STD_METADATA + REGULAR_IMPORTS,
-     (METADATA, 5, 0, 10, 3, ['ANSIBLE_METADATA'])),
-    # Metadata at end of file
-    (LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + TEXT_STD_METADATA.rstrip(),
-     (METADATA, 8, 0, 13, 3, ['ANSIBLE_METADATA'])),
-    # Metadata at beginning of file
-    (TEXT_STD_METADATA + LICENSE + REGULAR_IMPORTS,
-     (METADATA, 1, 0, 6, 3, ['ANSIBLE_METADATA'])),
-
-    # Standard import
-    (LICENSE + FUTURE_IMPORTS + BYTES_STD_METADATA + REGULAR_IMPORTS,
-     (METADATA, 5, 0, 10, 3, ['ANSIBLE_METADATA'])),
-    # Metadata at end of file
-    (LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + BYTES_STD_METADATA.rstrip(),
-     (METADATA, 8, 0, 13, 3, ['ANSIBLE_METADATA'])),
-    # Metadata at beginning of file
-    (BYTES_STD_METADATA + LICENSE + REGULAR_IMPORTS,
-     (METADATA, 1, 0, 6, 3, ['ANSIBLE_METADATA'])),
-)
-
-
-@pytest.mark.parametrize("code, expected", METADATA_EXAMPLES)
-def test_dict_metadata(code, expected):
-    assert md.extract_metadata(module_data=code, offsets=True) == expected
-
-
-@pytest.mark.parametrize("code, expected", STRING_METADATA_EXAMPLES)
-def test_string_metadata(code, expected):
-    # FIXME: String/yaml metadata is not implemented yet.
-    with pytest.raises(NotImplementedError):
-        assert md.extract_metadata(module_data=code, offsets=True) == expected
-
-
-def test_required_params():
-    with pytest.raises(TypeError, match='One of module_ast or module_data must be given'):
-        assert md.extract_metadata()
-
-
-def test_module_data_param_given_with_offset():
-    with pytest.raises(TypeError, match='If offsets is True then module_data must also be given'):
-        assert md.extract_metadata(module_ast='something', offsets=True)
-
-
-def test_invalid_dict_metadata():
-    with pytest.raises(SyntaxError):
-        assert md.extract_metadata(module_data=LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.1",\n' + REGULAR_IMPORTS)
-
-    with pytest.raises(md.ParseError, match='Unable to find the end of dictionary'):
-        assert md.extract_metadata(module_ast=ast.parse(LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.1"}\n' + REGULAR_IMPORTS),
-                                   module_data=LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.1",\n' + REGULAR_IMPORTS,
-                                   offsets=True)
-
-
-def test_multiple_statements_limitation():
-    with pytest.raises(md.ParseError, match='Multiple statements per line confuses the module metadata parser.'):
-        assert md.extract_metadata(module_data=LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.1"}; a=b\n' + REGULAR_IMPORTS,
-                                   offsets=True)