From 8733253a76376736ce80e1f66105e2972a5c354c Mon Sep 17 00:00:00 2001 From: John R Barker Date: Fri, 28 Apr 2017 11:58:38 +0100 Subject: [PATCH] Minor testing docs improvements (#24103) Fix a few formatting issues spotted post review. Also reapply missing commit --- docs/docsite/rst/dev_guide/testing.rst | 12 +-- .../docsite/rst/dev_guide/testing_compile.rst | 6 +- .../rst/dev_guide/testing_integration.rst | 22 +++-- .../dev_guide/testing_integration_legacy.rst | 9 +- .../rst/dev_guide/testing_running_locally.rst | 95 +------------------ docs/docsite/rst/dev_guide/testing_units.rst | 18 +++- 6 files changed, 42 insertions(+), 120 deletions(-) diff --git a/docs/docsite/rst/dev_guide/testing.rst b/docs/docsite/rst/dev_guide/testing.rst index 4920a1bd169..64e5ee239e9 100644 --- a/docs/docsite/rst/dev_guide/testing.rst +++ b/docs/docsite/rst/dev_guide/testing.rst @@ -33,9 +33,6 @@ At a high level we have the following classifications of tests: * Tests directly against individual parts of the code base. - - -Link to Manual testing of PRs (testing_pullrequests.rst) If you're a developer, one of the most valuable things you can do is look at the GitHub issues list and help fix bugs. We almost always prioritize bug fixing over feature development, so helping to fix bugs is one of the best things you can do. Even if you're not a developer, helping to test pull requests for bug fixes and features is still immensely valuable. @@ -62,8 +59,6 @@ When Shippable detects an error and it can be linked back to a file that has bee lib/ansible/modules/network/foo/bar.py:0:0: E316 ANSIBLE_METADATA.metadata_version: required key not provided @ data['metadata_version']. Got None From the above example we can see that ``--test pep8`` and ``--test validate-modules`` have identified issues. The commands given allow you to run the same tests locally to ensure you've fixed the issues without having to push your changed to GitHub and wait for Shippable, for example: - - TBD If you haven't already got Ansible available, use the local checkout by running:: @@ -137,7 +132,8 @@ and destination repositories. It will look something like this:: Someuser wants to merge 1 commit into ansible:devel from someuser:feature_branch_name .. note:: Only test ``ansible:devel`` - It is important that the PR request target be ansible:devel, as we do not accept pull requests into any other branch. Dot releases are cherry-picked manually by Ansible staff. + + It is important that the PR request target be ``ansible:devel``, as we do not accept pull requests into any other branch. Dot releases are cherry-picked manually by Ansible staff. The username and branch at the end are the important parts, which will be turned into git commands as follows:: @@ -187,9 +183,9 @@ If the PR does not resolve the issue, or if you see any failures from the unit/i | When I ran this Ubuntu 16.04 it failed with the following: | | \``` - | BLARG + | some output | StrackTrace - | RRRARRGGG + | some other output | \``` Want to know more about testing? diff --git a/docs/docsite/rst/dev_guide/testing_compile.rst b/docs/docsite/rst/dev_guide/testing_compile.rst index 51b1375c275..de129975903 100644 --- a/docs/docsite/rst/dev_guide/testing_compile.rst +++ b/docs/docsite/rst/dev_guide/testing_compile.rst @@ -38,21 +38,19 @@ Or against a specific Python version by doing: ansible-test compile --python 2.7 lineinfile -For advanced usage see the online help: +For advanced usage see the help: .. code:: shell ansible-test units --help -For advanced options see ``ansible-test compile --help`` - Installing dependencies ======================= ``ansible-test`` has a number of dependencies , for ``compile`` tests we suggest running the tests with ``--local``, which is the default -The dependencies can be installed using the ``-requirements`` argument. For example: +The dependencies can be installed using the ``--requirements`` argument. For example: .. code:: shell diff --git a/docs/docsite/rst/dev_guide/testing_integration.rst b/docs/docsite/rst/dev_guide/testing_integration.rst index 6b55c89c37a..bb10cd2e4bc 100644 --- a/docs/docsite/rst/dev_guide/testing_integration.rst +++ b/docs/docsite/rst/dev_guide/testing_integration.rst @@ -18,9 +18,6 @@ Quick Start It is highly recommended that you install and activate the ``argcomplete`` python package. It provides tab completion in ``bash`` for the ``ansible-test`` test runner. -To get started quickly using Docker containers for testing, -see [Tests in Docker containers](#tests-in-docker-containers). - Configuration ============= @@ -80,7 +77,8 @@ for testing, and enable PowerShell Remoting to continue. Running these tests may result in changes to your Windows host, so don't run them against a production/critical Windows environment. -Enable PowerShell Remoting (run on the Windows host via Remote Desktop): +Enable PowerShell Remoting (run on the Windows host via Remote Desktop):: + Enable-PSRemoting -Force Define Windows inventory:: @@ -141,6 +139,12 @@ To test with Python 3 use the following images: - ubuntu1604py3 +Cloud Tests +=========== + +See the :doc:`testing_integration_legacy` page for more information. + + Network Tests ============= @@ -183,9 +187,9 @@ Writing network integration tests --------------------------------- Test cases are added to roles based on the module being testing. Test cases -should include both `cli` and `eapi` test cases. Cli test cases should be -added to `test/integration/targets/modulename/tests/cli` and eapi tests should be added to -`test/integration/targets/modulename/tests/eapi`. +should include both cli and API test cases. Cli test cases should be +added to ``test/integration/targets/modulename/tests/cli`` and API tests should be added to +``test/integration/targets/modulename/tests/eapi``, or ``nxapi``. In addition to positive testing, negative tests are required to ensure user friendly warnings & errors are generated, rather than backtraces, for example: @@ -229,7 +233,9 @@ Conventions Adding a new Network Platform ````````````````````````````` -A top level playbook is required such as `ansible/test/integration/eos.yaml` which needs to be references by `ansible/test/integration/network-all.yaml` +A top level playbook is required such as ``ansible/test/integration/eos.yaml`` which needs to be references by ``ansible/test/integration/network-all.yaml`` Where to find out more ====================== + +If you'd like to know more about the plans for improving testing Ansible then why not join the `Testing Working Group `_. diff --git a/docs/docsite/rst/dev_guide/testing_integration_legacy.rst b/docs/docsite/rst/dev_guide/testing_integration_legacy.rst index deb97e2cf49..0c7358b394e 100644 --- a/docs/docsite/rst/dev_guide/testing_integration_legacy.rst +++ b/docs/docsite/rst/dev_guide/testing_integration_legacy.rst @@ -29,7 +29,7 @@ Cloud tests exercise capabilities of cloud modules (e.g. ec2_key). These are not 'tests run in the cloud' so much as tests that leverage the cloud modules and are organized by cloud provider. -Some AWS tests may use environment variables. It is recommended to either unset any AWS environment variables( such as ``AWS_DEFAULT_PROFILE``, ``AWS_SECRET_ACCESS_KEY``, etc) or be sure that the environment variables match the credentials provided in ``credentials.yml`` to ensure the tests run with consistency to their full capability on the expected account . See `AWS CLI docs `_ for information on creating a profile. +Some AWS tests may use environment variables. It is recommended to either unset any AWS environment variables( such as ``AWS_DEFAULT_PROFILE``, ``AWS_SECRET_ACCESS_KEY``, etc) or be sure that the environment variables match the credentials provided in ``credentials.yml`` to ensure the tests run with consistency to their full capability on the expected account. See `AWS CLI docs `_ for information on creating a profile. Subsets of tests may be run by ``#commenting`` out unnecessary roles in the appropriate playbook, such as ``test/integration/amazon.yml``. @@ -46,6 +46,7 @@ Provide cloud credentials:: Other configuration =================== + In order to run some tests, you must provide access credentials in a file named ``credentials.yml``. A sample credentials file named ``credentials.template`` is available for syntax help. @@ -53,12 +54,14 @@ named ``credentials.yml``. A sample credentials file named Running Tests ============= -The tests are invoked via a ``Makefile``:: +The tests are invoked via a ``Makefile``. - # If you haven't already got Ansible available use the local checkout by doing:: +If you haven't already got Ansible available use the local checkout by doing:: source hacking/env-setup +Run the tests by doing:: + cd test/integration/ # TARGET is the name of the test from the list at the top of this page #make TARGET diff --git a/docs/docsite/rst/dev_guide/testing_running_locally.rst b/docs/docsite/rst/dev_guide/testing_running_locally.rst index 8d5c72cf908..c22500904b9 100644 --- a/docs/docsite/rst/dev_guide/testing_running_locally.rst +++ b/docs/docsite/rst/dev_guide/testing_running_locally.rst @@ -4,7 +4,7 @@ Testing Ansible .. contents:: Topics -This page describes how to: +This document describes how to: * Run tests locally using ``ansible-test`` * Extend @@ -16,71 +16,12 @@ There are no special requirements for running ``ansible-test`` on Python 2.7 or The ``argparse`` package is required for Python 2.6. The requirements for each ``ansible-test`` command are covered later. -Setup -===== - -The code and tests are in the same GitHub repository, to get a local copy do - -#. Fork the `ansible/ansible `_ repository on GitHub. -#. Clone your fork: ``git clone git@github.com:USERNAME/ansible.git`` -#. Install the optional ``argcomplete`` package for tab completion (highly recommended):: - - pip install argcomplete - activate-global-python-argcomplete - # Restart your shell to complete global activation. - -#. Configure your environment to run from your clone (once per shell): ``. hacking/env-setup`` -#. ``ansible``, ``ansible-playbook`` and ``ansible-test`` will now be in your ``PATH`` Test Environments ================= Most ``ansible-test`` commands support running in one or more isolated test environments to simplify testing. -Local ------ - -The ``--local`` option runs tests locally without the use of an isolated test environment. -This is the default behavior. - - Recommended for ``compile`` tests. - -See the `command requirements directory `_ for the requirements for each ``ansible-test`` command. -Requirements files are named after their respective commands. -See also the `constraints `_ applicable to all commands. - - Use the ``--requirements`` option to automatically install ``pip`` requirements relevant to the command being used. - -Docker ------- - -The ``--docker`` option runs tests in a docker container. - - Recommended for ``integration`` tests. - -This option accepts an optional docker container image. -See the `list of supported docker images `_ for options. - - Use the ``--docker-no-pull`` option to avoid pulling the latest container image. - This is required when using custom local images that are not available for download. - -Tox ---- - -The ``--tox`` option run tests in a ``tox`` managed Python virtual environment. - - Recommended for ``windows-integration`` and ``units`` tests. - -The following Python versions are supported: - -* 2.6 -* 2.7 -* 3.5 -* 3.6 - -By default, test commands will run against all supported Python versions when using ``tox``. - - Use the ``--python`` option to specify a single Python version to use for test commands. Remote ------ @@ -92,40 +33,6 @@ An API key is required to use this feature. See the `list of supported platforms and versions `_ for additional details. -General Usage -============= - -Tests are run with the ``ansible-test`` command. -Consult ``ansible-test --help`` for usage information not covered here. - - Use the ``--explain`` option to see what commands will be executed without actually running them. - -Running Tests -============= - -There are four main categories of tests, each in their own directory. - -* `compile `_ - Python syntax checking for supported versions. Examples: - - * ``ansible-test compile`` - Check syntax for all supported versions. - * ``ansible-test compile --python 3.5`` - Check only Python 3.5 syntax. - -* `sanity `_ - Static code analysis and general purpose script-based tests. Examples: - - * ``ansible-test sanity --tox --python 2.7`` - Run all sanity tests on Python 2.7 using ``tox``. - * ``ansible-test sanity --test pep8`` - Run the ``pep8`` test without ``tox``. - -* `integration `_ - Playbook based tests for modules and core engine functionality. Examples: - - * ``ansible-test integration ping --docker`` - Run the ``ping`` module test using ``docker``. - * ``ansible-test windows-integration windows/ci/`` - Run all Windows tests covered by CI. - -* `units `_ - API oriented tests using mock interfaces for modules and core engine functionality. Examples: - - * ``ansible-test units --tox`` - Run all unit tests on all supported Python versions using ``tox``. - * ``ansible-test units --tox --python 2.7 test/units/vars/`` - Run specific tests on Python 2.7 using ``tox``. - -Consult each of the test directories for additional details on usage and requirements. Interactive Shell ================= diff --git a/docs/docsite/rst/dev_guide/testing_units.rst b/docs/docsite/rst/dev_guide/testing_units.rst index c7167cb6af2..4f929d92c80 100644 --- a/docs/docsite/rst/dev_guide/testing_units.rst +++ b/docs/docsite/rst/dev_guide/testing_units.rst @@ -82,12 +82,24 @@ See `eos_banner test