ansible/docs/docsite/rst/dev_guide/testing_running_locally.rst

:orphan:

.. _testing_running_locally:

***************
Testing Ansible
***************

.. contents:: Topics

This document describes how to:

* Run tests locally using ``ansible-test``
* Extend

Requirements
============

There are no special requirements for running ``ansible-test`` on Python 2.7 or later.
The ``argparse`` package is required for Python 2.6.
The requirements for each ``ansible-test`` command are covered later.


Test Environments
=================

Most ``ansible-test`` commands support running in one or more isolated test environments to simplify testing.


Remote
------

The ``--remote`` option runs tests in a cloud hosted environment.
An API key is required to use this feature.

    Recommended for integration tests.

See the `list of supported platforms and versions <https://github.com/ansible/ansible/blob/devel/test/runner/completion/remote.txt>`_ for additional details.

Environment Variables
---------------------

When using environment variables to manipulate tests there some limitations to keep in mind. Environment variables are:

* Not propagated from the host to the test environment when using the ``--docker`` or ``--remote`` options.
* Not exposed to the test environment unless whitelisted in ``test/runner/lib/util.py`` in the ``common_environment`` function.
* Not exposed to the test environment when using the ``--tox`` option unless whitelisted in ``test/runner/tox.ini`` by the ``passenv`` definition.

    Example: ``ANSIBLE_KEEP_REMOTE_FILES=1`` can be set when running ``ansible-test integration --tox``. However, using the ``--docker`` option would
    require running ``ansible-test shell`` to gain access to the Docker environment. Once at the shell prompt, the environment variable could be set
    and the tests executed. This is useful for debugging tests inside a container by following the
    :ref:`Debugging AnsibleModule-based modules <debugging_ansiblemodule_based_modules>` instructions.

Interactive Shell
=================

Use the ``ansible-test shell`` command to get an interactive shell in the same environment used to run tests. Examples:

* ``ansible-test shell --docker`` - Open a shell in the default docker container.
* ``ansible-test shell --tox 3.6`` - Open a shell in the Python 3.6 ``tox`` environment.


Code Coverage
=============

Code coverage reports make it easy to identify untested code for which more tests should
be written.  Online reports are available but only cover the ``devel`` branch (see
:doc:`testing`).  For new code local reports are needed.

Add the ``--coverage`` option to any test command to collect code coverage data.  If you
aren't using the ``--tox`` or ``--docker`` options which create an isolated python
environment then you may have to use the ``--requirements`` option to ensure that the
correct version of the coverage module is installed::

   ansible-test units --coverage apt
   ansible-test integration --coverage aws_lambda --tox --requirements
   ansible-test coverage html


Reports can be generated in several different formats:

* ``ansible-test coverage report`` - Console report.
* ``ansible-test coverage html`` - HTML report.
* ``ansible-test coverage xml`` - XML report.

To clear data between test runs, use the ``ansible-test coverage erase`` command. For a full list of features see the online help::

   ansible-test coverage --help
Remove more docs build errors (#45364) * orphans testing pages to avoid not-in-toctree errors * orphans various pages pending reorg * adds module_utils and special_vars to main TOC * uses a glob for scenario_guide TOC * normalize and Sentence-case headings on community pages, typos * re-orgs community TOC, adds all pages to toctree * removes scenario guides index page * adds style guide to community index * basic update to style guide * fix typo that created a new error * removes not-in-toctree from ignore errors list * leave removing files for future cleanup task 6 years ago			`:orphan:`

Reduce warnings (#39254) * removes FAQ links; no entries exist for linked config settings * fixes various anchors and links * addresses abadger comments, thanks * marks orphan pages, avoids TOC errors * adds links for remote_tmp setting to FAQ 7 years ago			`.. _testing_running_locally:`

Docs how to test (2nd) (#24094) * Big testing doc refactor * Combine all the testing documentation in to one place to make it easier to find * Convert everything to RST * Create testing_network guide * Create testing landing page * For each section detail "how to run" and "how to extend testing" * More examples * Lots more detail 8 years ago			`***************`
Expand test documentation. (#23598) * Replace test README.md with new README.rst. * Add compile and sanity README.rst docs. 8 years ago			`Testing Ansible`
Docs how to test (2nd) (#24094) * Big testing doc refactor * Combine all the testing documentation in to one place to make it easier to find * Convert everything to RST * Create testing_network guide * Create testing landing page * For each section detail "how to run" and "how to extend testing" * More examples * Lots more detail 8 years ago			`***************`
Expand test documentation. (#23598) * Replace test README.md with new README.rst. * Add compile and sanity README.rst docs. 8 years ago
Docs how to test (2nd) (#24094) * Big testing doc refactor * Combine all the testing documentation in to one place to make it easier to find * Convert everything to RST * Create testing_network guide * Create testing landing page * For each section detail "how to run" and "how to extend testing" * More examples * Lots more detail 8 years ago			`.. contents:: Topics`

Minor testing docs improvements (#24103) Fix a few formatting issues spotted post review. Also reapply missing commit 8 years ago			`This document describes how to:`
Docs how to test (2nd) (#24094) * Big testing doc refactor * Combine all the testing documentation in to one place to make it easier to find * Convert everything to RST * Create testing_network guide * Create testing landing page * For each section detail "how to run" and "how to extend testing" * More examples * Lots more detail 8 years ago
			* Run tests locally using ``ansible-test``
			`* Extend`
Expand test documentation. (#23598) * Replace test README.md with new README.rst. * Add compile and sanity README.rst docs. 8 years ago
			`Requirements`
			`============`

			There are no special requirements for running ``ansible-test`` on Python 2.7 or later.
			The ``argparse`` package is required for Python 2.6.
			The requirements for each ``ansible-test`` command are covered later.


			`Test Environments`
			`=================`

			Most ``ansible-test`` commands support running in one or more isolated test environments to simplify testing.


			`Remote`
			`------`

			The ``--remote`` option runs tests in a cloud hosted environment.
			`An API key is required to use this feature.`

			`Recommended for integration tests.`

Docs how to test (2nd) (#24094) * Big testing doc refactor * Combine all the testing documentation in to one place to make it easier to find * Convert everything to RST * Create testing_network guide * Create testing landing page * For each section detail "how to run" and "how to extend testing" * More examples * Lots more detail 8 years ago			See the `list of supported platforms and versions <https://github.com/ansible/ansible/blob/devel/test/runner/completion/remote.txt>`_ for additional details.
Expand test documentation. (#23598) * Replace test README.md with new README.rst. * Add compile and sanity README.rst docs. 8 years ago
allow ANSIBLE_KEEP_REMOTE_FILES for local test runner (#33357) * allow ANSIBLE_KEEP_REMOTE_FILES for local test runner * add ANSIBLE_KEEP_REMOTE_FILES to tox.ini, update docs * Clarify handling of environment variables. 7 years ago			`Environment Variables`
			`---------------------`

			`When using environment variables to manipulate tests there some limitations to keep in mind. Environment variables are:`

Remove more docs build errors (#45364) * orphans testing pages to avoid not-in-toctree errors * orphans various pages pending reorg * adds module_utils and special_vars to main TOC * uses a glob for scenario_guide TOC * normalize and Sentence-case headings on community pages, typos * re-orgs community TOC, adds all pages to toctree * removes scenario guides index page * adds style guide to community index * basic update to style guide * fix typo that created a new error * removes not-in-toctree from ignore errors list * leave removing files for future cleanup task 6 years ago			* Not propagated from the host to the test environment when using the ``--docker`` or ``--remote`` options.
allow ANSIBLE_KEEP_REMOTE_FILES for local test runner (#33357) * allow ANSIBLE_KEEP_REMOTE_FILES for local test runner * add ANSIBLE_KEEP_REMOTE_FILES to tox.ini, update docs * Clarify handling of environment variables. 7 years ago			* Not exposed to the test environment unless whitelisted in ``test/runner/lib/util.py`` in the ``common_environment`` function.
			* Not exposed to the test environment when using the ``--tox`` option unless whitelisted in ``test/runner/tox.ini`` by the ``passenv`` definition.

Remove more docs build errors (#45364) * orphans testing pages to avoid not-in-toctree errors * orphans various pages pending reorg * adds module_utils and special_vars to main TOC * uses a glob for scenario_guide TOC * normalize and Sentence-case headings on community pages, typos * re-orgs community TOC, adds all pages to toctree * removes scenario guides index page * adds style guide to community index * basic update to style guide * fix typo that created a new error * removes not-in-toctree from ignore errors list * leave removing files for future cleanup task 6 years ago			Example: ``ANSIBLE_KEEP_REMOTE_FILES=1`` can be set when running ``ansible-test integration --tox``. However, using the ``--docker`` option would
			require running ``ansible-test shell`` to gain access to the Docker environment. Once at the shell prompt, the environment variable could be set
			`and the tests executed. This is useful for debugging tests inside a container by following the`
allow ANSIBLE_KEEP_REMOTE_FILES for local test runner (#33357) * allow ANSIBLE_KEEP_REMOTE_FILES for local test runner * add ANSIBLE_KEEP_REMOTE_FILES to tox.ini, update docs * Clarify handling of environment variables. 7 years ago			:ref:`Debugging AnsibleModule-based modules <debugging_ansiblemodule_based_modules>` instructions.
Expand test documentation. (#23598) * Replace test README.md with new README.rst. * Add compile and sanity README.rst docs. 8 years ago
			`Interactive Shell`
			`=================`

			Use the ``ansible-test shell`` command to get an interactive shell in the same environment used to run tests. Examples:

			* ``ansible-test shell --docker`` - Open a shell in the default docker container.
Local test docs: correct tox shell command usage (#49350) The ansible-test shell command doesn't have a --python option; you have to specify the tox environment directly to the --tox command Partially fixes: #49349 6 years ago			* ``ansible-test shell --tox 3.6`` - Open a shell in the Python 3.6 ``tox`` environment.
Expand test documentation. (#23598) * Replace test README.md with new README.rst. * Add compile and sanity README.rst docs. 8 years ago
allow ANSIBLE_KEEP_REMOTE_FILES for local test runner (#33357) * allow ANSIBLE_KEEP_REMOTE_FILES for local test runner * add ANSIBLE_KEEP_REMOTE_FILES to tox.ini, update docs * Clarify handling of environment variables. 7 years ago
Expand test documentation. (#23598) * Replace test README.md with new README.rst. * Add compile and sanity README.rst docs. 8 years ago			`Code Coverage`
			`=============`

Mdd module unit test docs (#31373) * new documentation for unit testing - especially module unit testing * unit test documentation reformatting and further fixes * unit test documentation - point to online coverage reports & fix bad spaces * Small copy edits. * First pass copy edit / rewrite. More info needed. * testing documentation - clean up structure, especially code coverage - reduce repetition * module unit test documentation - improved introduction * testing documentation - more fixes from and inspired by review from dharmabumstead * testing documentation - fixes from mattclay + some other minor tweaks * More copy edits. * testing documentation - further fixes from review * Copy edits * Copy edits * More copy edits. 7 years ago			`Code coverage reports make it easy to identify untested code for which more tests should`
			be written. Online reports are available but only cover the ``devel`` branch (see
			:doc:`testing`). For new code local reports are needed.

			Add the ``--coverage`` option to any test command to collect code coverage data. If you
			aren't using the ``--tox`` or ``--docker`` options which create an isolated python
			environment then you may have to use the ``--requirements`` option to ensure that the
doc: preseve block-quote structure (#53293) Without the `::`, the indentation of the example block is lost. 6 years ago			`correct version of the coverage module is installed::`
Mdd module unit test docs (#31373) * new documentation for unit testing - especially module unit testing * unit test documentation reformatting and further fixes * unit test documentation - point to online coverage reports & fix bad spaces * Small copy edits. * First pass copy edit / rewrite. More info needed. * testing documentation - clean up structure, especially code coverage - reduce repetition * module unit test documentation - improved introduction * testing documentation - more fixes from and inspired by review from dharmabumstead * testing documentation - fixes from mattclay + some other minor tweaks * More copy edits. * testing documentation - further fixes from review * Copy edits * Copy edits * More copy edits. 7 years ago
			`ansible-test units --coverage apt`
			`ansible-test integration --coverage aws_lambda --tox --requirements`
			`ansible-test coverage html`

Expand test documentation. (#23598) * Replace test README.md with new README.rst. * Add compile and sanity README.rst docs. 8 years ago
			`Reports can be generated in several different formats:`

			* ``ansible-test coverage report`` - Console report.
			* ``ansible-test coverage html`` - HTML report.
			* ``ansible-test coverage xml`` - XML report.

Mdd module unit test docs (#31373) * new documentation for unit testing - especially module unit testing * unit test documentation reformatting and further fixes * unit test documentation - point to online coverage reports & fix bad spaces * Small copy edits. * First pass copy edit / rewrite. More info needed. * testing documentation - clean up structure, especially code coverage - reduce repetition * module unit test documentation - improved introduction * testing documentation - more fixes from and inspired by review from dharmabumstead * testing documentation - fixes from mattclay + some other minor tweaks * More copy edits. * testing documentation - further fixes from review * Copy edits * Copy edits * More copy edits. 7 years ago			To clear data between test runs, use the ``ansible-test coverage erase`` command. For a full list of features see the online help::

			`ansible-test coverage --help`