Commit Graph

62 Commits (540d53327d41d42f62a7dce845f0e5037b16c36d)

Author SHA1 Message Date
Matt Clay a1453a7917 Improve handling of first CI run for new branch. (#29070) 7 years ago
Matt Clay f71d861563 Collect data from integration test runs. (#28650) 7 years ago
Matt Clay 5ea8a5e34b Add support for setup targets to ansible-test. (#28544)
* Add support for setup targets to ansible-test.
* Code cleanup.
7 years ago
Ricardo Carrillo Cruz 746b433c29 Remove 'net' hardcoding on ansible-test executor (#28334)
We need to run network-integration against platform agnostic modules,
which they all are named as net_* .
There is a hardcoding in executor where if prefix is 'net' the hosts is set
to all, removing these lines to have desired behaviour.
7 years ago
Matt Clay f6d7fc548e Fix ansible-test type hints. 7 years ago
Matt Clay e241e15899 Add support for `non_local` alias in ansible-test. 7 years ago
Matt Clay 27033915cc Add --list-targets and speed up --explain. (#26838)
* Add ansible-test --list-targets option.
* Speed up ansible-test integration --explain.
7 years ago
Matt Clay e63f69464f Add ansible-test options and docs. (#26823)
* All integration commands support --continue-on-error
* The network-integration command supports --inventory
* Add landing page for compile test docs.
* Add bot documentation links.
7 years ago
Matt Clay 7931e11437 Use subset of Windows tests for "all" tests. (#26830) 7 years ago
Matt Clay 0b784c65b1 Add sanity import test to ansible-test. (#26730)
* Add sanity import test to ansible-test.
* Run sanity import test on all python versions.
7 years ago
Matt Clay 20d49af475 Add ANSIBLE_TEST_CI env var to integration tests. 7 years ago
Matt Clay ab5dd24430 Handle pip/python errors in ansible-test better. 7 years ago
David Newswanger 294c0f91c0 added option to ansible-test to run test playbooks using the debug strategy (#26499) 7 years ago
Matt Clay 45e377566c Refactor ansible-test config classes. (#26505)
* Move Config classes from executor.py to config.py.
* Move Environment and Test config to config.py.
* Move Coverage/CoverageReport Config to config.py.
* Clean up type hints.
7 years ago
Matt Clay c26a556929 Add check for known_hosts changing during tests. (#26470)
* Add check for known_hosts changing during tests.
* Don't use known hosts file for delegate_to test.
7 years ago
David Newswanger 0a6dcf4169 changed ansible-test so that it replaces . with - instead of . with _ so that the hostname is valid 7 years ago
René Moser 77d0542985 test, integration: add diff support (#26296) 7 years ago
Pierre-Louis Bonicoli d08ff86169 ansible-test: pass --[skip-]tags to ansible-playbook
Thanks to Olaf Klischat for the idea.
7 years ago
Matt Clay 53180ebb57 Fix Shippable BASE_BRANCH lookup on merges. 8 years ago
Matt Clay 5babe2daea Increase python version coverage for tests. (#24762)
* Improve ansible-test inventory handling.
* Fix python 3 re-raise of exception from thread.
* Fix python 3 encoding for windows-integration.
* Run network tests on multiple python versions.
* Run windows tests on multiple python versions.
* Support Shippable delegation using --tox.
* Skip vyos_command on python 3 tests until fixed.
* Add python 3 filtering to local and tox.
* Fix tests to support back to back runs.
* Temporarily test networking with python 2.7 only.

Running the tests back to back causes intermittent test failures
which need to be addressed before we can test multiple versions
in a single test run.
8 years ago
Matt Clay 6a2a7a2392 Enable codecov.io and add coverage grouping. 8 years ago
Matt Clay fdcad0f44d Improve on-demand coverage for Shippable. 8 years ago
John R Barker e9e2d67035 Run vyos_command Network tests in Shippable (#24514)
* Run VyOS Network tests in Shippable

* net_command no longer exists

* network/ci

* Disable other network tests

* pep8 fix

* Fix PEP 8 issue.
8 years ago
Matt Clay dfd19a812f Miscellaneous bug fixes for ansible-test.
- Overhauled coverage injector to fix issues with non-local tests.
- Updated integration tests to work with the new coverage injector.
- Fix concurrency issue by using random temp files for delegation.
- Fix handling of coverage files from root user.
- Fix handling of coverage files without arcs.
- Make sure temp copy of injector is world readable and executable.
8 years ago
Matt Clay b8cb3f519b Detect and fix environment tampering in tests. 8 years ago
Matt Clay a07d42e16d Add support for cloud tests to ansible-test. (#24315)
* Split out ansible-test docker functions.
* Add cloud support to ansible-test.
8 years ago
Matt Clay 6522d703a1 Fix ansible-test pylint and pep8 issues. (#24245)
* Fix pylint useless-super-delegation issues.
* Fix miscellaneous pylint issues.
* Fix PEP 8 issues.
* Fix warnings reported by PyCharm.
8 years ago
Matt Clay 991bf0a1c3 Fix ansible-test base branch handling on merge. (#22759) 8 years ago
Matt Clay 47887a2edd Fix ansible-test base branch usage on Shippable. 8 years ago
Matt Clay 3a22467b38 Remove python 2.4 compile test. 8 years ago
Matt Clay 869449e288 Add test verification to ansible-test. (#22636)
* Add unified git diff parser.
* Add metadata and diff handling.
* Add test confidence/verification to bot output.
8 years ago
Matt Clay 89559f78de Add `--failure-ok` option to `ansible-test`. (#22623) 8 years ago
Matt Clay 3ab344e8bd Use lint/junit output for compile test. (#22395)
* Refactor sanity classes for use in all tests.
* Use lint/junit output for compile test.
* Add missing options for compile test.
* Fix early bailout on requirements install.
8 years ago
Matt Clay fbf55886ba Use lint friendly output for python syntax checks. (#22390) 8 years ago
Matt Clay d66ce40ecb Overhaul `ansible-test sanity` implementation. (#22177)
- Tests are run to completion instead of stopping on first failure.
- Test results are now parsed instead of passing through to the console.
- Test results can be saved in junit xml format.
- Test results will show up on the Shippable "Tests" result tab.
- Added an experimental --lint option for easier integration with other tools.
- Code smell tests are now usable with the --list-tests, --test and --skip-test options.
- Code split out from executor.py into sanity.py.
- Rename download-logs to download.py and add support for test and coverage results.
- Miscellaneous improvements.
8 years ago
Matt Clay 28b882e737 Support ansible commands in sanity tests. (#22172) 8 years ago
Matt Clay 45cec93e99 Disable newer shellcheck tests which don't pass. 8 years ago
Matt Clay fcac261eef Run unit tests in isolation w/ coverage support. 8 years ago
Matt Clay 2797cff773 Run destructive win tests when creating instances. 8 years ago
Matt Clay bc8186543a Pass base branch to module validator on delegate. 8 years ago
Matt Martz 9b1bd2eb7e Re-enable module comparisons in validate-modules (#21582)
* Re-enable module comparisons, specifically for new module detection and for finding new options/arguments
* Only do new module checks in shippable, local will display warning
8 years ago
Matt Clay 2192c1eb02 Fix ansible-test selection of inventory file. 8 years ago
John R Barker 5e28d3a15c network_cli shouldn't be specified anymore (#21458) 8 years ago
Matt Clay d54bc09fae Consider module_utils deps when running tests. (#21382)
* Skip pep8 analysis when --explain is used.
* Fix return type annotations.
* Match line length requirement of PEP 8 config.
* Consider module_utils deps when running tests.
8 years ago
Matt Clay 4189592701 Revert "Enable process isolation for unit tests."
This reverts commit 91526cd9f2.

Removing this feature primarily because it interferes with
collecting proper code coverage results. I may restore the
feature later if that can be resolved.
8 years ago
Matt Clay 91526cd9f2 Enable process isolation for unit tests. 8 years ago
Matt Clay 6ef1a6aeb6 Add `pep8` to `ansible-test`. (#20745)
The rule sets and legacy file list are a first draft.
It is likely that they will need to be revised.
8 years ago
Matt Clay fa7f84217b Add startup checks to ansible-test. 8 years ago
Matt Clay 61e6e7868c Improve jinja2 test coverage. (#20533)
* Run `template` test on latest jinja2 version.
* Update jinja2 install for groupby_filter.
* Clean test output dir for each test.
8 years ago
Matt Clay 4ba9f63afe Use multiple AWS regions in ansible-test. (#20429)
* Move ansible-test EnvironmentConfig to util.
* Use multiple AWS regions in ansible-test.
8 years ago