From c05a12472cead4ddf33280f45168cab70b67570e Mon Sep 17 00:00:00 2001 From: Michael DeHaan Date: Wed, 10 Sep 2014 15:55:36 -0400 Subject: [PATCH] Add PR testing guide (95% from James). --- docsite/rst/developing.rst | 1 + docsite/rst/developing_test_pr.rst | 170 +++++++++++++++++++++++++++++ 2 files changed, 171 insertions(+) create mode 100644 docsite/rst/developing_test_pr.rst diff --git a/docsite/rst/developing.rst b/docsite/rst/developing.rst index eeb0ebb0c82..2a258993019 100644 --- a/docsite/rst/developing.rst +++ b/docsite/rst/developing.rst @@ -10,6 +10,7 @@ Learn how to build modules of your own in any language, and also how to extend A developing_inventory developing_modules developing_plugins + developing_test_pr Developers will also likely be interested in the fully-discoverable in :doc:`tower`. It's great for embedding Ansible in all manner of applications. diff --git a/docsite/rst/developing_test_pr.rst b/docsite/rst/developing_test_pr.rst new file mode 100644 index 00000000000..634d37dd1b8 --- /dev/null +++ b/docsite/rst/developing_test_pr.rst @@ -0,0 +1,170 @@ +Helping Testing PRs +``````````````````` + +If you're a developer, one of the most valuable things you can do is look at the github issues list and help fix bugs. We almost always prioritize bug fixing over +feature development, so clearing bugs out of the way is one of the best things you can do. + +Even if you're not a developer, helping test pull requests for bug fixes and features is still immensely valuable. + +This goes for testing new features as well as testing bugfixes. + +In many cases, code should add tests that prove it works but that's not ALWAYS possible and tests are not always comprehensive, especially when a user doesn't have access +to a wide variety of platforms, or that is using an API or web service. + +In these cases, live testing against real equipment can be more valuable than automation that runs against simulated interfaces. +In any case, things should always be tested manually the first time too. + +Thankfully helping test ansible is pretty straightforward, assuming you are already used to how ansible works. + +Get Started with A Source Checkout +++++++++++++++++++++++++++++++++++ + +You can do this by checking out ansible, making a test branch off the main one, merging a GitHub issue, testing, +and then commenting on that particular issue on GitHub. Here's how: + +.. note:: + Testing source code from GitHub pull requests sent to us does have some inherent risk, as the source code + sent may have mistakes or malicious code that could have a negative impact on your system. We recommend + doing all testing on a virtual machine, whether a cloud instance, or locally. Some users like Vagrant + or Docker for this, but they are optional. It is also useful to have virtual machines of different Linux or + other flavors, since some features (apt vs. yum, for example) are specific to those OS versions. + +First, you will need to configure your testing environment with the neccessary tools required to run our test +suites. You will need at least:: + + git + python-nosetests + +Second, if you haven't already, clone the Ansible source code from GitHub:: + + git clone https://github.com/ansible/ansible.git + cd ansible/ + +.. note:: + If you have previously forked the repository on GitHub, you could also clone it from there. + +Activating The Source Checkout +++++++++++++++++++++++++++++++ + +The Ansible source includes a script that allows you to use Ansible directly from source without requiring a +full installation, that is frequently used by developers on Ansible. + +Simply source it (to use the Linux/Unix terminology) to begin using it immediately:: + + source ./hacking/env-setup + +This script modifies the PYTHONPATH enviromnent variables (along with a few other things), which will be temporarily +set as long as your shell session is open. + +If you'd like your testing environment to always use the latest source, you could call the command from startup scripts (for example, +`.bash_profile`). + +Finding A Pull Request and Checking It Out On A Branch +++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +Next, find the pull request you'd like to test and make note of the line at the top which describes the source +and destination repositories. It will look something like this:: + + Someuser wants to merge 1 commit into ansible:devel from someuser:feature_branch_name + +.. note:: + It is important that the PR request target be ansible:devel, as we do not accept pull requests into any other branch. + Dot releases are cherry-picked manually by ansible staff. + +At the bottom of the GitHub page, there is a link that says "You can also merge this request on the command line". Click this link +to expand the GitHub interface, and it will show instructions that look somewhat like this:: + + Step 1: From your project repository, check out a new branch and test the changes. + + git checkout -b username-branchname branchname + git pull git://github.com/username/branchname.git branchname + +Do not follow step 2, as you don't want to merge the user features back into your branch. + +.. note:: + If the GitHub user interface shows that the pull request will not merge cleanly, we do not recommend proceeding if you + are not somewhat familiar with git and coding, as you will have to resolve a merge conflict. This is the responsibility of + the original pull request contributor. + +.. note:: + Some users do not create feature branches, which can cause problems when they have multiple, un-related commits in + their version of `devel`. If the source looks like `someuser:devel`, make sure there is only one commit listed on + the pull request. + +For Those About To Test, We Salute You +++++++++++++++++++++++++++++++++++++++ + +At this point, you should be ready to begin testing! + +If the PR is a bug-fix pull request, the first things to do are to run the suite of unit and integration tests, to ensure +the pull request does not break current functionality:: + + # Unit Tests + make tests + + # Integration Tests + cd test/integration + make + +.. note:: + Ansible does provide integration tests for cloud-based modules as well, however we do not recommend using them for some users + due to the associated costs from the cloud providers. As such, typically it's better to run specific parts of the integration battery + and skip these tests. + +Integration tests aren't the end all beat all - in many cases what is fixed might not *HAVE* a test, so determining if it works means +checking the functionality of the system and making sure it does what it said it would do. + +Pull requests for bug-fixes should reference the bug issue number they are fixing. + +We encourage users to provide playbook examples for bugs that show how to reproduce the error, and these playbooks should be used to verify the bugfix does resolve +the issue if available. You may wish to also do your own review to poke the corners of the change. + +Since some reproducers can be quite involved, you might wish to create a testing directory with the issue # as a sub- +directory to keep things organized:: + + mkdir -p testing/XXXX # where XXXX is again the issue # for the original issue or PR + cd testing/XXXX + + +While it should go without saying, be sure to read any playbooks before you run them. VMs help with running untrusted content greatly, +though a playbook could still do something to your computing resources that you'd rather not like. + +Once the files are in place, you can run the provided playbook (if there is one) to test the functionality:: + + ansible-playbook -vvv playbook_name.yml + +If there's not a playbook, you may have to copy and paste playbook snippets or run a ad-hoc command that was pasted in. + +Our issue template also included sections for "Expected Output" and "Actual Output", which should be used to gauge the output +from the provided examples. + +If the pull request resolves the issue, please leave a comment on the pull request, showing the following information: + + * "Works for me!" + * The output from `ansible --version`. + +In some cases, you may wish to share playbook output from the test run as well. + +Example!:: + + Works for me! Tested on `Ansible 1.7.1`. I verified this on CentOS 6.5 and also Ubuntu 14.04. + +If the PR does not resolve the issue, or if you see any failures from the unit/integration tests, just include that output instead:: + + This doesn't work for me. + + When I ran this my toaster started making loud noises! + + Output from the toaster looked like this: + + ``` + BLARG + StrackTrace + RRRARRGGG + ``` + +We understand some users may be inexperienced with git, or other aspects of the above procedure, so feel free to stop by ansible-devel +list for questions and we'd be happy to help answer them. + + +