Commit Graph

166 Commits (bf3c291606ccd92272038d7df00b10ba791e4592)

Author SHA1 Message Date
Matt Clay 5babe2daea Increase python version coverage for tests. (#24762)
* Improve ansible-test inventory handling.
* Fix python 3 re-raise of exception from thread.
* Fix python 3 encoding for windows-integration.
* Run network tests on multiple python versions.
* Run windows tests on multiple python versions.
* Support Shippable delegation using --tox.
* Skip vyos_command on python 3 tests until fixed.
* Add python 3 filtering to local and tox.
* Fix tests to support back to back runs.
* Temporarily test networking with python 2.7 only.

Running the tests back to back causes intermittent test failures
which need to be addressed before we can test multiple versions
in a single test run.
8 years ago
Matt Clay 036ba7eeec Clean up Shippable tools and update job handling.
- Tools are now in a tools subdirectory.
- Removed obsolete ansible-core-ci tool.
- Added run.py for starting new CI runs.
- Improved handling of run IDs and URLs.
- General code cleanup and docs updates.
- Nightly CI runs use complete coverage.
8 years ago
Matt Clay 6e160650e9 Fix run filter on Shippable downloader. 8 years ago
Matt Clay 4daa16f9ce Add --all and --stub coverage options.
ci_coverage
ci_complete
8 years ago
Matt Clay 6a2a7a2392 Enable codecov.io and add coverage grouping. 8 years ago
Matt Clay fdcad0f44d Improve on-demand coverage for Shippable. 8 years ago
Matt Clay 5cb5228cde Implement on-demand coverage on Shippable. (#24535) 8 years ago
Matt Clay af5db3e4a9 Revert "Add support for on-demand coverage on Shippable."
This reverts commit 7b3f0b8870.
8 years ago
Matt Clay 7b3f0b8870 Add support for on-demand coverage on Shippable.
Triggered by setting the COVERAGE environment var to a non-empty value.
Alternatively the string `ci_coverage` can be added to a commit message.
8 years ago
Matt Clay fb9efd3c7e Add `--remote-terminate` option to ansible-test.
Set `--remote-terminate success` for osx tests.
8 years ago
Matt Clay 17e07a27b2 Enable cloud tests for use with ansible-test. 8 years ago
Matt Clay 0587d59266 Use Shippable image: drydock/u16pytall:master (#23997)
* Use Shippable image: drydock/u16pytall:master
* Do not install python 3.6 on Shippable.
8 years ago
Matt Clay e7bb508ad6 Fix var precedence check to support python 3. (#23552)
* Fix var precedence check to support python 3.
* Run CI sanity tests using python 3.5.
* Disable pylint non-iterator-returned test to pass on python 3.5.
8 years ago
Matt Clay eec01e2137 Set IO encoding for ansible-test on Shippable. 8 years ago
Matt Clay dfe8eca3d2 Update Shippable downloader to support new API.
- Download run/jobs by URL (old or new).
- Download run metadata.
- Download job metadata.
- Format downloaded JSON.
- Handle and save coverage/testresults download errors.
8 years ago
Matt Clay 12f6beaebd Remove python2.4 package install from other test. 8 years ago
Matt Clay 869449e288 Add test verification to ansible-test. (#22636)
* Add unified git diff parser.
* Add metadata and diff handling.
* Add test confidence/verification to bot output.
8 years ago
Matt Clay 89559f78de Add `--failure-ok` option to `ansible-test`. (#22623) 8 years ago
Matt Clay a64665ae91 Fix error handling in CI other script. 8 years ago
Matt Clay 3ab344e8bd Use lint/junit output for compile test. (#22395)
* Refactor sanity classes for use in all tests.
* Use lint/junit output for compile test.
* Add missing options for compile test.
* Fix early bailout on requirements install.
8 years ago
Matt Clay 9acf551674 Run all `other` tests on Shippable before failing. 8 years ago
Matt Clay 8f463fcdd2 Create bot friendly sanity output. (#22381) 8 years ago
Matt Clay 72c96b3ac3 Fix encoding issue in shippable downloader. 8 years ago
Matt Clay d66ce40ecb Overhaul `ansible-test sanity` implementation. (#22177)
- Tests are run to completion instead of stopping on first failure.
- Test results are now parsed instead of passing through to the console.
- Test results can be saved in junit xml format.
- Test results will show up on the Shippable "Tests" result tab.
- Added an experimental --lint option for easier integration with other tools.
- Code smell tests are now usable with the --list-tests, --test and --skip-test options.
- Code split out from executor.py into sanity.py.
- Rename download-logs to download.py and add support for test and coverage results.
- Miscellaneous improvements.
8 years ago
Matt Clay ae2d2860f5 Reduce CI output processing code coverage. 8 years ago
Matt Clay 2804d2e848 Split up unit tests on Shippable. (#21931)
* Add more verbose output to coverage combine.
* Run unit tests on CI separately by version.
8 years ago
Matt Clay b2bd75a408 Restore 2008-SP2 and 2008-R2_SP1 CI testing. 8 years ago
Matt Davis 8527013fbe Complete rewrite of Windows exec wrapper (#21510)
* supports pipelining for faster execution
* supports become (runas), creates interactive subsession under WinRM batch logon
* supports usage of arbitrary module_utils files
* modular exec wrapper payload supports easier extension
* integrates async wrapper behavior for pipelined/become'd async
* module_utils are loaded as true Powershell modules, no more runtime modifications to module code
8 years ago
Matt Clay f16ee2841b Add retries to Shippable CI scripts. (#21108)
* Add retry.py for auto-retry in CI scripts.
* Add retries to other and units CI scripts.
8 years ago
Matt Clay f42fa8879b Temporarily remove Windows Server 2016 from CI. 8 years ago
Matt Clay 7ed67746cf Enable Windows 2016 on Shippable.
Also launch Windows instances on us-east-2 and all others on us-east-1.
8 years ago
Matt Clay 3c51ce56fd Enable more network support in ansible-test. (#20372)
* Only start platform instances with tests selected.
* Enable ios on Shippable.
* Show inventory in explain mode.
* Fix indentation of generated network inventory.
* Update classification of network module_utils.
8 years ago
Matt Clay 203e375449 Fix python 3.6 install. 8 years ago
Matt Clay f534573dcf Enable first network tests on Shippable. (#20208) 8 years ago
Matt Clay cd3fdca540 Switch tests to pytest and ansible-test.
- Replace nose usage with pytest.
- Remove legacy Shippable integration.sh.
- Update Makefile to use pytest and ansible-test.
- Convert most yield unit tests to pytest parametrize.
8 years ago
Matt Clay f340b8d14f Add test support for python 3.6. 8 years ago
Matt Clay 422857166b Remove legacy test alias migration script. (#19629)
* Require code-smell tests to be files.
* Add sanity check for integration test aliases.
* Remove migration script for test aliases.
8 years ago
Matt Clay f789a20f29 Update log download url. 8 years ago
Toshio Kuratomi c6b42028c4 Fixes for uri under python3 and local (non-httptester) testing 8 years ago
Matt Clay dab2bf7e13 Skip test_uri on python 3. 8 years ago
Lumír 'Frenzy' Balhar 84544ee8fd Fix AST nodes for Python 3 and enable dependent test_uri (#18597)
* Enable tests on python 3 for uri

* Added one more node type to SAFE_NODES into safe_eval module.

ast.USub represents unary operators. This is necessary for
parsing some unusual but still valid JSON files during testing
with Python 3.
8 years ago
Matt Clay f9c4158ff8 Remove core/extras dirs from old PRs during CI. 8 years ago
Matt Clay 580401c02b Remove legacy test code. 8 years ago
Matt Clay 4da7a7ce0e Update coverage exclusions. (#18675)
* Update coverage exclusions.
* Fix logic error in Windows change detection.
8 years ago
Matt Clay 6bbd92e422 Initial ansible-test implementation. (#18556) 8 years ago
Adrien Vergé 527d8307c1 Lint YAML files under test/
This commit extends YAML linting by enabling standard rules from the
`yamllint` tool [1]. Since syntax errors and key duplicates are already
checked since 4d48711, this change only adds detection for cosmetic
problems. It also narrows checks to the test/ dir only.

The main goal is to prevent future problems to enter the code base
without being noticed. While it would be a huge effort to be PEP8
compliant, it is relatively easy to have correct YAML style *now* and
prevent future errors by enabling linting.

Note: for those (like me) caring about code attribution: use `git blame
-w` to ignore whitespace-only changes.

Note: I disabled some linting checks (such as indentation), they can be
enforced in the future if needed. Similarly, current checks can also be
disabled. See the `.yamllint` file.

[1]: https://yamllint.readthedocs.io/
8 years ago
Matt Clay c955688772 Move ansible ad-hoc tests to integration targets. 8 years ago
Matt Clay 912d6ed8cc Clean up code-smell sanity scripts. (#18407)
- Replace find ';' with '+' for faster execution.
- Replace grep -R with -r to avoid recursive warnings.
- Exclude .git and .tox directories from recursive grep.
- Improve messaging on failed sanity checks.
- Add no-basestring check to Shippable.
8 years ago
Matt Clay 8c270ac75f Add empty-init code-smell script. (#18406)
Also removed boilerplate from otherwise empty __init__.py files
which should not contain any code (checked by empty-init script).
8 years ago
Matt Clay ecb7f13119 Update log url for Shippable download-logs script.
Also improved error handling.
8 years ago
Matt Clay 469c4a106b Fix and create code-smell test for line endings. 8 years ago
Matt Clay 0d46805979 Clean up shebangs for various files.
- Remove shebangs from:
  - ini files
  - unit tests
  - module_utils
  - plugins
  - module_docs_fragments
  - non-executable Makefiles
- Change non-modules from '/usr/bin/python' to '/usr/bin/env python'.
- Change '/bin/env' to '/usr/bin/env'.

Also removed main functions from unit tests (since they no longer
have a shebang) and fixed a python 3 compatibility issue with
update_bundled.py so it does not need to specify a python 2 shebang.

A script was added to check for unexpected shebangs in files.
This script is run during CI on Shippable.
8 years ago
Matt Clay 71819c0a60 Test reorganization and cleanup. (#18270)
- Correct directory name in test/README.md
- Move code-smell tests to test/sanity/code-smell
- Update code-smell.sh to use new script paths
- Add test/integration/target-prefixes.win for ansible-test
- Move module unit tests to match module directory layout
8 years ago
Toshio Kuratomi 2ac12432ef Enable the git test on py3 8 years ago
Michael Scherer 2804e64ed5 Enable filters test for python 3 8 years ago
Toshio Kuratomi 67ac375188 pip tests now pass on python3. Enable them. 8 years ago
Michael Scherer 578da9a615 Enable test for lookups on python 3
Since passlib algo sometime takes a bytes, and sometime
not, depending on a internal variable, we have to convert
bnased on it, or it fail with "TypeError: salt must be bytes,
not str" (or unicode instead of bytes)

However, that's not great to use internal structure for that.
8 years ago
Michael Scherer e93a8814a3 Let authorized_keys be tested on python 3 8 years ago
Toshio Kuratomi 60acfd1e87 Fix ansible-pull on python3
On python3, we can't write bytes directly to sys.stdout.
8 years ago
Toshio Kuratomi fd14048d46 Thanks to mscherer, these tests are now passing 8 years ago
Toshio Kuratomi 91c1fdec3d unarchive tests now pass on python3 8 years ago
John R Barker 0ef3ff6a9d Install dependencies for validate-modules (#18028)
Currently (pre-repomerge) we aren't running sanity.sh from
ansible/ansible, after the merge we will. Therefore I've added the
requirements here, rather than in ansible-modules-*/test/utils/shippable
8 years ago
Matt Clay 656b82417e Fix role path for generate-tests. 8 years ago
Matt Clay 80a5c70ad7 Split integration tests out from Makefile. (#17976) 8 years ago
Toshio Kuratomi 5bc3cb278c Remove unicode escape (#17866)
* Remove unicode-escape which is not present on python3

Alternative fix for #17305

* Enable the assemble test on python3

* Fix other problems with assemble on python3
8 years ago
Matt Clay 21857e8618 Disable privileged mode on Shippable. (#17844) 8 years ago
Toshio Kuratomi 4452ee86bd Turn mount test back on (#17797)
* Turn mount test back on

* Mount tests need PRIVILEGED so turn that back on

Revert "Revert "Set PRIVILEGED=true for non_destructive tests. (#17733)" (#17738)"

This reverts commit dc0fb1c212.

* Add a needs_privileged tag so that we can skip mount tests on centos6

Some containers timeout on shippable tests when run with privileged.
Unfortunately, some tests require that in order to run.  Tagging those
allows us to skip those tests on the platforms that timeout when we get
ready to run the integration test in shippable.

* Centos6 times out with PRIVILEGED set so remove that (will disable the mount tests on centos6)

* Remove false start
8 years ago
Matt Clay a265d2d77d Run test_template tests on OS X. (#17791) 8 years ago
Matt Clay 0dc3db9e75 Enable unarchive tests for OS X and FreeBSD. (#17789) 8 years ago
Matt Clay d05df5fc61 Disable expect continue in ansible-core-ci. (#17737)
This will allow use of larger SSH keys.
8 years ago
Matt Clay 2e0bec293c Improve error handling for ansible-core-ci. (#17734) 8 years ago
Matt Clay 50b0d011fe Update integration.sh to copy source by default. (#17717)
Also add more documentation to the script.
8 years ago
Matt Clay 40ffd8269d Add shellcheck to code-smell checks. (#17715) 8 years ago
Adrian Likins e11f7d2339 add a defaults for integration.sh (#17709)
* add a default test image (ansible/ansible:centos7)
* add a default test target (all)
8 years ago
Matt Clay ada2432165 Add keep on failure option to integration.sh. (#17711) 8 years ago
Matt Clay bcb49fecb5 Add tool for downloading Shippable logs. 8 years ago
Allen Sanabria b510abce17 CloudRetry/AWSRetry backoff decorator with unit tests (#17039)
* Added aws_retry decorator function with unit tests

* Restructured the code to be used with a base class.

This base class CloudRetry can be reused by any other cloud provider.
This decorator should be used in situations, where you need to implement
a backoff algorithm and want to retry based on the status code from the
exception.

* updated documentation

* fixed tabs

* added botocore and boto3 to requirements.txt

* removed cloud.py from py24 tests, as it depends on boto3

* fix relative imports

* updated test to be 2.6 compat

* updated method name from retry to backoff

* readded lxd

* Updated default backoff from 2 seconds to 1.1s.

This will be about a total of 48 seconds in 10 tries. This is
configurable.
8 years ago
Matt Clay c81fe70fbf Enable more integration tests for python 3. (#17473)
* Enable more integration tests for python 3.
* Split out python 3 integration tests.

Now that we're running more integration tests on python 3, the
tests are taking long enough that they warrant splitting out in
the same way the python 2 tests are split.
8 years ago
Matt Clay a1947401ba Enable more tests that pass on python 3. (#17471)
* Enable more tests that pass on python 3.
* Disable test_apt_repository for python 3.
8 years ago
Matt Clay 7cd988f60c Enable more integration tests for python 3. (#17375)
The test_async test target was updated to accommodate changes in
output buffering behavior in python 3. This change in behavior
may need to be addressed in the future.
8 years ago
Toshio Kuratomi f7b22a5eaa Fix paramiko's exec_command() to return bytes on python3 (#17372)
* Fix paramiko's exec_command() to return bytes on python3

* Run test_connection for python3 now too

* Fix atomic_move for problem in shippable's testing

* Python-2.4 needs to use b()
8 years ago
Matt Clay 75cb27ffe3 Enable more python 3 tests that now pass. (#17359) 8 years ago
Matt Clay 2ffb136b3f Add support for testing module PRs with python3. (#17339) 8 years ago
Matt Clay 6e65ebda26 Put integration.sh pip reqs in separate file. (#17338) 8 years ago
Matt Davis 1c33b5a9f0 fix remote shippable target failures (#17287)
cryptography upgrade caused fatal error when pycrypto was not installed
8 years ago
Toshio Kuratomi 4a3a9c0f2d Fix for run_command on py3 and enable lineinfile test on py3 (#17257)
* run_command needed a bit of tweaking to its string handling of
  arguments.
* The run_command change fixes the last bit of lineinfile so we can
  enable its tests
8 years ago
Matt Clay 178292d2cd Fix file and copy modules on py3 and enable tests. (#17239)
- Fix octal formatting of file mode in module response on py3.
- Convert file path to unicode in copy action.
- Enable file and copy module tests for py3 now that they pass.
8 years ago
Matt Clay acc8a13eff Enable more tests that pass on python 3. (#17233) 8 years ago
Toshio Kuratomi 380a5801e9 Enable integration tests on python3 that are already working (#17212) 8 years ago
Matt Clay e2e7390adc Add Python 3 CI support. (#17209) 8 years ago
Matt Clay a695e18615 Add support for OS X CI on Shippable. (#17160) 8 years ago
Matt Clay 04bd6a3619 Make tests more cross platform. (#17154) 8 years ago
Brian Coca 4fb09d5693 moved from extras repo where it incorrectly exists (#17124)
* moved from extras repo where it incorrectly exists

* added to sanity exclusion for 2.4

* changed license with author's consent
8 years ago
Matt Clay b44eb402bd Update ansible-core-ci endpoint. (#17071) 8 years ago
Matt Clay 901042f458 Generate test script to file instead of stdout. 8 years ago
Matt Clay 2b0d63b0d8 Flush stderr to avoid lost output. 8 years ago
Matt Clay 80385a47bd Remove FreeBSD interpreter test hacks. (#17007)
* Add TEST_FLAGS to no_log target.
8 years ago
Filipe Niero Felisbino e54a9d3a51 Add generic data structures querying (#13684)
* Query lookup plugin

* Add license and docstrings

* Add python3-ish imports

* Change query plugin type from lookup to filter

* Switch from dq to jsonpath_rw

* Add integration test for query filter

* Rename query filter to json_query

* Add jsonpath-rw

* Rename query filter to json_query

* Switch query implementation from jsonpath-rw to jmespath
8 years ago
Matt Clay d2fb845955 Enable more FreeBSD integration tests. (#16991) 8 years ago
Matt Clay 72cca01cd4 Use file list, not recursion, in _fixup_perms. (#16924)
Run setfacl/chown/chmod on each temp dir and file.

This fixes temp file permissions handling on platforms such as FreeBSD
which always return success when using find -exec. This is done by
eliminating the use of find when setting up temp files and directories.

Additionally, tests that now pass on FreeBSD have been enabled for CI.
8 years ago