Expand test documentation. (#23598)

* Replace test README.md with new README.rst.
* Add compile and sanity README.rst docs.

test/README.md

@@ -1,31 +0,0 @@
-Ansible Test System
-===================
-
-Folders
-=======
-
-units
------
-
-Unit tests that test small pieces of code not suited for the integration test layer, usually very API based, and should leverage
-mock interfaces rather than producing side effects.
-
-Playbook engine code is better suited for integration tests.
-
-Requirements: `pip install -r test/runner/requirements/units.txt`
-
-integration
------------
-
-Integration test layer, constructed using playbooks.
-
-Some tests may require cloud credentials, others will not, and destructive tests are separated from non-destructive so a subset
-can be run on development machines.
-
-learn more
-----------
-
-hop into a subdirectory and see the associated README.md for more info.
-
-
-

test/README.rst

@@ -0,0 +1,140 @@
+Testing Ansible
+===============
+
+How to run and create tests for the Ansible core engine and modules with ``ansible-test``.
+
+Requirements
+============
+
+There are no special requirements for running ``ansible-test`` on Python 2.7 or later.
+The ``argparse`` package is required for Python 2.6.
+The requirements for each ``ansible-test`` command are covered later.
+
+Setup
+=====
+
+#. Fork the `ansible/ansible <https://github.com/ansible/ansible/>`_ repository on Git Hub.
+#. Clone your fork: ``git clone git@github.com:USERNAME/ansible.git``
+#. Install the optional ``argcomplete`` package for tab completion (highly recommended):
+
+   #. ``pip install argcomplete``
+   #. ``activate-global-python-argcomplete``
+   #. Restart your shell to complete global activation.
+
+#. Configure your environment to run from your clone (once per shell): ``. hacking/env-setup``
+
+Test Environments
+=================
+
+Most ``ansible-test`` commands support running in one or more isolated test environments to simplify testing.
+
+Local
+-----
+
+The ``--local`` option runs tests locally without the use of an isolated test environment.
+This is the default behavior.
+
+    Recommended for ``compile`` tests.
+
+See the `command requirements directory <runner/requirements/>`_ for the requirements for each ``ansible-test`` command.
+Requirements files are named after their respective commands.
+See also the `constraints <runner/requirements/constraints.txt>`_ applicable to all commands.
+
+    Use the ``--requirements`` option to automatically install ``pip`` requirements relevant to the command being used.
+
+Docker
+------
+
+The ``--docker`` option runs tests in a docker container.
+
+    Recommended for ``integration`` tests.
+
+This option accepts an optional docker container image.
+See the `list of supported docker images <runner/completion/docker.txt>`_ for options.
+
+    Use the ``--docker-no-pull`` option to avoid pulling the latest container image.
+    This is required when using custom local images that are not available for download.
+
+Tox
+---
+
+The ``--tox`` option run tests in a ``tox`` managed Python virtual environment.
+
+    Recommended for ``windows-integration`` and ``units`` tests.
+
+The following Python versions are supported:
+
+* 2.6
+* 2.7
+* 3.5
+* 3.6
+
+By default, test commands will run against all supported Python versions when using ``tox``.
+
+    Use the ``--python`` option to specify a single Python version to use for test commands.
+
+Remote
+------
+
+The ``--remote`` option runs tests in a cloud hosted environment.
+An API key is required to use this feature.
+
+    Recommended for integration tests.
+
+See the `list of supported platforms and versions <runner/completion/remote.txt>`_ for additional details.
+
+General Usage
+=============
+
+Tests are run with the ``ansible-test`` command.
+Consult ``ansible-test --help`` for usage information not covered here.
+
+    Use the ``--explain`` option to see what commands will be executed without actually running them.
+
+Running Tests
+=============
+
+There are four main categories of tests, each in their own directory.
+
+* `compile <compile/>`_ - Python syntax checking for supported versions. Examples:
+
+  * ``ansible-test compile`` - Check syntax for all supported versions.
+  * ``ansible-test compile --python 3.5`` - Check only Python 3.5 syntax.
+
+* `sanity <sanity/>`_ - Static code analysis and general purpose script-based tests. Examples:
+
+  * ``ansible-test sanity --tox --python 2.7`` - Run all sanity tests on Python 2.7 using ``tox``.
+  * ``ansible-test sanity --test pep8`` - Run the ``pep8`` test without ``tox``.
+
+* `integration <integration/>`_ - Playbook based tests for modules and core engine functionality. Examples:
+
+  * ``ansible-test integration ping --docker`` - Run the ``ping`` module test using ``docker``.
+  * ``ansible-test windows-integration windows/ci/`` - Run all Windows tests covered by CI.
+
+* `units <units/>`_ - API oriented tests using mock interfaces for modules and core engine functionality. Examples:
+
+  * ``ansible-test units --tox`` - Run all unit tests on all supported Python versions using ``tox``.
+  * ``ansible-test units --tox --python 2.7 test/units/vars/`` - Run specific tests on Python 2.7 using ``tox``.
+
+Consult each of the test directories for additional details on usage and requirements.
+
+Interactive Shell
+=================
+
+Use the ``ansible-test shell`` command to get an interactive shell in the same environment used to run tests. Examples:
+
+* ``ansible-test shell --docker`` - Open a shell in the default docker container.
+* ``ansible-test shell --tox --python 3.6`` - Open a shell in the Python 3.6 ``tox`` environment.
+
+Code Coverage
+=============
+
+Add the ``--coverage`` option to any test command to collect code coverage data.
+
+Reports can be generated in several different formats:
+
+* ``ansible-test coverage report`` - Console report.
+* ``ansible-test coverage html`` - HTML report.
+* ``ansible-test coverage xml`` - XML report.
+
+To clear data between test runs, use the ``ansible-test coverage erase`` command.

test/compile/README.rst

@@ -0,0 +1,13 @@
+Compile Tests
+=============
+
+Compile tests check source files for valid syntax on all supported python versions:
+
+- 2.6
+- 2.7
+- 3.5
+- 3.6
+
+Tests are run with ``ansible-test compile``.
+All versions are tested unless the ``--python`` option is used.
+All ``*.py`` files are tested unless specific files are specified.

test/sanity/README.rst

@@ -0,0 +1,70 @@
+Sanity Tests
+============
+
+Sanity tests are made up of scripts and tools used to perform static code analysis.
+The primary purpose of these tests is to enforce Ansible coding standards and requirements.
+
+Tests are run with ``ansible-test sanity``.
+All available tests are run unless the ``--test`` option is used.
+
+Available Tests
+===============
+
+Tests can be listed with ``ansible-test sanity --list-tests``.
+
+This list is a combination of two different categories of tests.
+
+Code Smell Tests
+----------------
+
+Miscellaneous `scripts <code-smell/>`_ used for enforcing coding standards and requirements, identifying trip hazards, etc.
+
+    These tests are listed and accessed by script name. There is no actual test named ``code-smell``.
+
+All executable scripts added to the ``code-smell`` directory are automatically detected and executed by ``ansible-test``.
+
+Scripts in the directory which fail can be skipped by adding them to `skip.txt <code-smell/skip.txt>`_.
+This is useful for scripts which identify issues that have not yet been resolved in the code base.
+
+Files tested are specific to the individual test scripts and are not affected by command line arguments.
+
+Built-in Tests
+--------------
+
+These tests are integrated directly into ``ansible-test``.
+All files relevant to each test are tested unless specific files are specified.
+
+ansible-doc
+~~~~~~~~~~~
+
+Verifies that ``ansible-doc`` can parse module documentation on all supported python versions.
+
+pep8
+~~~~
+
+Python static analysis for PEP 8 style guideline compliance.
+
+pylint
+~~~~~~
+
+Python static analysis for common programming errors.
+
+rstcheck
+~~~~~~~~
+
+Check reStructuredText files for syntax and formatting issues.
+
+shellcheck
+~~~~~~~~~~
+
+Static code analysis for shell scripts using the excellent `shellcheck <https://www.shellcheck.net/>`_ tool.
+
+validate-modules
+~~~~~~~~~~~~~~~~
+
+Analyze modules for common issues in code and documentation.
+
+yamllint
+~~~~~~~~
+
+Check YAML files for syntax and formatting issues.