Commit Graph

462 Commits (73701004b0a2e734d9b5e69c2dfeafb7be630ca0)

Author SHA1 Message Date
Matt Clay 2b463ef197
ansible-test - Upgrade pylint to 2.9.3. (#75480)
* ansible-test - Upgrade `pylint` to 2.9.3.
* Update pylint ignores due to rule name change.
* Disable pylint deprecated-class for compat code.
* Add pylint ignores for test support content.
* Add ignores for arguments-renamed in lib/ansible/
* Add pylint ignores for collection_loader.
* ansible-test - Ignore deprecations in legacy collection loader.
* ansible-test - Suppress pylint consider-using-with
* ansible-test - Suppress pylint false positive.
* ansible-test - Suppress pylint consider-using-with.
* ansible-test - Suppress pylint deprecated-module
* Disable some of the new pylint suggestions.
* Remove unnecessary six usage from string_format pylint plugin.
* Remove obsolete ignore entry.
3 years ago
Matt Clay 7450e87615
Fix up string splitting to prepare for pylint update. (#75479)
* Use rsplit with maxsplit in BaseCacheModule.
* Use maxsplit for accessing first item.
* ansible-test - More efficient splitting.
3 years ago
Matt Clay ca2d2c5f38
ansible-test - Cleanup to prepare for pylint update. (#75469)
* ansible-test - Fix use of abstractproperty
* ansible-test - Use dict.items() where possible.
* ansible-test - Remove unused code.
* ansible-test - Cleanup issues reported by pylint.
* ansible-test - Use dict.items() where possible.
* ansible-test - Use generator.
3 years ago
Sam Doran 1d2af0b89e
ansible-test - allow for the same listening port on multiple interfaces (#75387) 3 years ago
Sam Doran d8dcfe737a
ansible-test - add ssh debugging logging (#75374) 3 years ago
Martin Krizek 8ab418f41b
Disable jinja2_native warning in CI (#75214) 3 years ago
Gonéri Le Bouder 5b8fb4dcd3 ansible-test: exposes tiny_prefix variable
ansible-test aws provider now creates and exposes a new tiny_prefix
variable to provide a shorter prefix for the AWS tests.
3 years ago
Matt Clay adc9e4a023 Updated ansible-test azure plugin help message. 3 years ago
Matt Clay 397d3763ea ansible-test - Do not downgrade cryptography. 3 years ago
Matt Clay 787f722d94 Fix combined coverage file detection.
Resolves an issue introduced with https://github.com/ansible/ansible/pull/74719
3 years ago
Matt Clay ce04056797 Expand delegation options for coverage commands. 3 years ago
Jordan Borean e875e91363
ansible-test - use pwsh to generate stubs (#74271) 3 years ago
Mark Mercado 1906d75907
Adding DigitalOcean cloud support to ansible-test (#74222) 3 years ago
Matt Clay 51fd05e76b Add support for testing with Python 3.10. 3 years ago
Matt Clay e6d7aecbe4
Reorganize more ansible-test code. (#74611)
* Split out shell command.
* Relocate ansible-test integration code.
3 years ago
Matt Clay 065fc3ca17 Remove check for legacy core and extras dirs. 3 years ago
Matt Clay 07fb6f7547 Fix imports for the previous commit.
Also add a changelog entry.
3 years ago
Matt Clay 92c40f6365 Relocate change classification files. 3 years ago
Matt Clay 5dfc7ee078 Remove Shippable CI provider from ansible-test. 3 years ago
Matt Clay df2ae06d46 Fix ansible-test imports and paths after refactor.
Also include a changelog entry covering this commit and the previous.
3 years ago
Matt Clay bb3904d299 Reorganize command implementations.
This commit only renames existing files.

Changes to adapt imports will occur in a follow-up commit.
3 years ago
Matt Clay fb0d5609cb
Code cleanup in ansible-test. (#74529)
* Fix get_available_python_versions calls.
* Make run_playbook vars optional.
* Use ansible_pipelining in inventory.
* Fix type hint.
* Fix order of conditional evaluation.
* Remove unused ibmi platform.
* Add changelog fragment.
3 years ago
Matt Clay 877e5b3dfe Fix ansible-test AZP build lookup. 3 years ago
Matt Clay c4e76a7f80 Add collection config support to ansible-test. 3 years ago
Matt Clay cb7f4f1971 Fix pypi-test-container port conflict. 3 years ago
Matt Clay 6cfe36fa1f
Set ansible-test min controller Python to 3.8. (#74395)
- Version neutral sanity tests now require Python 3.8 or later.
- Unit tests for controller-only code now require Python 3.8 or later.
3 years ago
Matt Clay 14ff5e213c Fix ansible-test docker container IP detection. 3 years ago
Matt Clay e6af2d6827 Fix ansible-test coverage traceback when no data.
If no coverage directory exists, ansible-test coverage would traceback.

Now it silently continues just as if the directory was present but empty.
3 years ago
Matt Clay 7786dbbdc1 Split out module/module_utils unit test execution.
- Unit tests for `modules` and `module_utils` are now limited to importing only `ansible.module_utils` from the `ansible` module.
- Unit tests other than `modules` and `module_utils` are now run only on Python versions supported by the controller (Python 3.8+).
- Unit tests are now run in separate contexts (`controller`, `modules`, `module_utils`), each using separate invocations of `pytest`.
3 years ago
Matt Clay 277a06f641
Clean up ansible-test cloud plugins. (#74322)
- Improve code reuse.
- Add missing type hints, fix existing ones and convert them to PEP 484 style.
- Add missing imports and clean up existing ones.
- Add missing docstrings and clean up existing ones.
3 years ago
Matt Clay b752d07163 Overhaul ansible-test container management.
This brings ansible-test closer to being able to support split controller/remote testing.
3 years ago
Matt Clay becf941673 Add PyPI proxy container for tests on Python 2.6. 3 years ago
Sam Doran 4df9ea86c2 ansible-test - change diff mathching regexp so prefixes are optional
The `a/` and `b/` prefixes can be disabled in the `git diff` output by setting
`diff.noprefix` to `true`. The output is still a valid diff, but `ansible-test` would
raise an exception since without the prefixes, it thought the diff line was invalid.
3 years ago
Sam Doran 7b79b69369
ansible-test - ensure trailing separator is added for connection unit test target (#74176)
For connection units tests, if the `__init__.py` file is modified, the trailing separator was
not added, making the target invalid.
3 years ago
Matt Clay b75844af45 Fix ansible-test coverage exporting.
ci_complete
ci_coverage
3 years ago
Mark Chappell ed18fcac3b
Ensure unit test paths for connection and inventory plugins are based on the context (#73877) 3 years ago
Matt Clay 15064c7a42 Update `resource_prefix` syntax for ansible-test. 3 years ago
Felix Fontein 29aef842d7
ansible-test: make sure tests are also run for the new plugin types (#73599)
* Make sure tests are also run for the new plugin types.
* New plugin types are not documentable.
3 years ago
Matt Clay d092356fc5 Fix ansible-test handling of egg-info.
Resolves https://github.com/ansible/ansible/issues/67990
3 years ago
Felix Fontein 1f3a90270b
Import sanity test for plugins (#72497) 3 years ago
Matt Clay f533d46572
Fix ansible-test constraints issues. (#73578)
* Silence Python 3.5 EOL notice in pip.
* Skip import test on compat/selinux.py utils.
* Improve Python version skip warning.
* Use Python 3.6 as minimum Python for sanity tests.
* Improve min Python handling for code-smell tests.
* Overhaul test-constraints sanity test.
* Merge sanity test constraints with requirements.
* Remove legacy content specific constraints.
* Add changelog fragment.
3 years ago
Matt Clay 8a175f59c9
Upgrade pylint used by ansible-test. (#70155)
* Upgrade pylint and deps in ansible-test.
* Enable pylint on Python 3.9.
* Update pylint config.
* Add ignore for vendored six.
* Add ignores for support plugins.
* Fix issue reported by pylint.
3 years ago
Matt Clay 1a2da990a4
Temporary fix for cryptography issues. (#73530) 3 years ago
Florian Heiderich 8f887b7adb
add self-signed https endpoint for ansible-test (#73463)
* introduce self-signed.ansible.http.tests

* forwarding of port 444

* forward port 8444 to port 444 on http test container

* Fix port forwarding for Windows under docker

* add changelog fragment

Co-authored-by: Jordan Borean <jborean93@gmail.com>
3 years ago
Matt Clay a9b5bebab3
Overhaul ansible-test SSH key management. (#73451)
* Pass remote.sh to shell over stdin.
* Pass docker.sh to shell over stdin.
* Standardize SSH key management.
* Update docker containers.
3 years ago
Matt Clay c7cb944315 Always use python exec wrapper in ansible-test. 3 years ago
Matt Clay ded2d0b90e Use `sh` for ansible-test raw remote shell. 3 years ago
Matt Clay 418b9b781d No longer warn about missing pip/python pair.
This warning was intended to help improve test environment configuration.
However, it has ended up producing mostly non-actionable warning noise instead.

In most situations a missing pip or python should result in test failures.
Reporting a missing pip also implies that it should be used by tests, which is not the case.
Tests should be invoking pip as a python module with the appropriate python interpreter instead.
3 years ago
Matt Clay 672941fb0d
Add test support for Python 3.8 on RHEL and FreeBSD. (#73393)
* Support Python 3.8 on RHEL 8.2+ in ansible-test.

* Support Python 3.8 on FreeBSD in ansible-test.

* Use libyaml with PyYAML on FreeBSD.
3 years ago
Matt Clay 1b157ef42f
Limit rstcheck sanity test to ansible-core. (#73391) 3 years ago
Matt Clay 73fadc5e97
Clean up code in ansible-test. (#73379)
* Relocate code to fix type dependencies.

* Fix missing and unused imports.

* Fix type hints.

* Suppress PyCharm false positives.

* Avoid shadowing `file` built-in.

* Use json.JSONEncoder directly instead of super().

This matches the recommended usage and avoids a PyCharm warning.

* Remove redundant regex escape.

* Remove redundant find_python call.

* Use tarfile.open directly.

* Add changelog fragment.
3 years ago
Matt Clay 4164cb26f0 Remove obsolete `tower` plugin from ansible-test. 4 years ago
Matt Clay 21f1811ddf
Cleanup provisioning code in ansible-test. (#73207)
* Remove unused code in ansible-test.

* Remove obsolete endpoint logic from ansible-test.

* Remove obsolete region selection in ansible-test.

* Remove obsolete port logic in ansible-test.

* Clean up ansible-test remote providers.
4 years ago
Matt Clay 77942acefc Refactor ansible-test cryptography install code. 4 years ago
Matt Clay 37d09f2488
Update ansible-test pylint Python support. (#72972)
* Update ansible-test pylint Python support.

* Python 3.8 is now officially supported.
* Python 3.9 is now skipped with a warning.
4 years ago
Matt Clay 36ab3d1189
Fix cryptography constraints in ansible-test. (#72914) 4 years ago
Matt Martz 6bc1e9f5dd
Address additional ansible_core rename issues (#72906)
* Update __requires__ to reference the correct ansible_core package name

* ansible-test updates to handle the correct egg_info for ansible_core
4 years ago
Felix Fontein f94ba68d8f
ansible-test sanity: fix UnicodeDecodeError for `--python 2.7 --docker` (#72623)
* Store target paths as unicode.

* Add changelog fragment.
4 years ago
Matt Clay 221c50b57c Use AZP config for integration-aliases test.
No changelog entry since this test is limited to the ansible/ansible repo.
4 years ago
Matt Clay 08842cd6bb
Fix Azure Pipelines change detection. (#72824) 4 years ago
Paul Belanger a95213d2f5
Switch to hashlib.sha256() for ansible-test (#72411)
When FIPs mode is enable on centos-8, we are not able to load md5
functions.

Signed-off-by: Paul Belanger <pabelanger@redhat.com>
4 years ago
Jordan Borean fb092a82a1
ansible-galaxy - source deps from all servers and not just parent (#72576)
* ansible-galaxy - source deps from all servers and not just parent

* Added integration tests for this scenario
4 years ago
Rick Elrod 6894ae7d1d
Rename to ansible-core (#72594)
Change:
- Initial set of changes for renaming to ansible-core
- Includes changelog fragment changes from base -> core
- Does NOT include docs changes
- Modifies detection stuff in setup.py to support ansible<2.9 and ansible-base

Test Plan:
- ci_complete
4 years ago
Jordan Borean 03a395cba4
Rollback pulp container to stabilise CI (#72580) 4 years ago
Matt Clay fa2be89cd4 Add coverage exporting to ansible-test
A new `--export` option for `ansible-test coverage combine` allows multi-step aggregation of code coverage for CI pipelines.
4 years ago
Matt Clay f022dedd0a Fix container discovery for the acme test plugin. 4 years ago
Matt Martz 96ad5b799e
Emit warning when running on the controller with a Python older than 3.8 (#72467)
* Emit warning when running on the controller with a Python older than 3.8

* Add spaces

Co-authored-by: Matt Clay <matt@mystile.com>

* and more spaces

Co-authored-by: Matt Clay <matt@mystile.com>

* s/Py/Python/

Co-authored-by: Matt Clay <matt@mystile.com>

* Add note to Control node requirements about Py3.8 requirement

* Add collection_name to deprecated call

* more spaces

Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>

* Clarify that we are only packaging for py3.8+

Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
4 years ago
Jordan Borean 1eae2bb890
Update Galaxy NG version used in CI (#72469) 4 years ago
Matt Clay 79021df2e4 Restore ansible-test coverage config.
Reverts a portion of the changes made in https://github.com/ansible/ansible/pull/70028
4 years ago
Felix Fontein 6b30efa454 Fix fallout from #70028. 4 years ago
Felix Fontein 64a809d2b6
Improve ansible-test classifications for collections (#72353) 4 years ago
Brian Coca 4b673484f0
rethink wording (#70028)
* rethink wording
* removed unrequired requirement
* fix tests
* fixed versions
Co-authored-by: Sloane Hertel <shertel@redhat.com>
4 years ago
Jordan Borean 4856ab0e68
Galaxy server update (#72286)
* Update galaxy server image used in CI

* Fix port and migration issue

* Add delete step, still need to deal with pagination

* Make cleanup more efficient

* Remove testing code
4 years ago
Matt Clay 3c2e8b99be
Fix ansible-test handling of user-defined docker networks. (#72256)
* Fix ansible-test docker container detection.

* Attach test containers to the correct network.

* Do not assume `localhost` for accesing Docker.

* Look for containers on current network.

* Always map /var/run/docker.sock into containers.

This fixes issues when using a remote Docker host.

* Support container IP lookup from networks list.

* Fix container network attachment.

* Remove redundant container detection messages.

* Limit DOCKER_HOST parsing to TCP.

* Restore docker socket existence check.

The check is skipped if the docker hostname is not localhost.

* Correct changelog entry.
4 years ago
Matt Clay 2ef4b7e07e Fix ansible-test Azure Pipelines container auth. 4 years ago
Jordan Borean caba47dd3f
Add support for GSSAPI/Kerberos to urls.py (#72113)
* Add support for GSSAPI/Kerberos to urls.py

* Test out changes with the latest test container

* Get remote hosts working

* Fix up httptester_krb5_password reader

* Fix tests for opensuse and macOS

* Hopefully last lot of testing changes

* Dont do CBT on macOS

* Fixes from review
4 years ago
Matt Clay 5f76bd2af7 Support collection constraints in ansible-test.
This allows collections to specify requirements and constraints for packages that ansible-test has requirements or constraints for.
4 years ago
Rick Elrod 0332046699
[ansible-test] attempt to work around podman (#72096)
Change:
- podman > 2 && < 2.2 does not support "images --format {{json .}}"
- podman also now outputs images JSON differently than docker
- Work around both of the above.

Test Plan:
- Tested with podman 2.0.6 in Fedora 31.

Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
4 years ago
Jordan Borean 672df07a33
Bump httptester version (#72036) 4 years ago
Alicia Cozine 27826827e9
updates intersphinx references for docs links (#71921)
* DOCS: updates intersphinx references for docs links
* TESTS: Raise the number of bytes scanned to determine if a file is binary. The newest ansible-2.10.inv file has its first null byte at position 2261. 4096 is still a cheap chunksize to read so it still makes sense to raise this.

Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>
Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
4 years ago
Matt Clay 92b66e3e31
CI provider fixes for ansible-test. (#71929)
* Make Azure Pipelines resource_prefix lowercase.

* Make classification of CI files consistent.

* Update package-data sanity test for AZP.
4 years ago
René Moser 5348e7bf61
test: cloudstack: Use test-container 1.4.0 (#71905) 4 years ago
Matt Clay 98febab975 Use new endpoint for Parallels based instances. 4 years ago
Matt Clay 2bbcbe99fd Support macOS 10.15 for ansible-test --remote.
Use of this new version is experimental, so it is not enabled in CI yet.
4 years ago
Matt Clay f5b6df14ab
Fix ansible-test coverage traceback. (#71446)
* Add integration test for ansible-test coverage.

* Fix ansible-test coverage traceback.

* Fix coverage reporting on Python 2.6.
4 years ago
Matt Clay d099591964
Update ansible-test remote endpoint handling. (#71413)
* Request ansible-core-ci resources by provider.
* Remove obsolete us-east-2 CI endpoint.
* Add new --remote-endpoint option.
* Add warning for --remote-aws-region option.
* Update service endpoints.
* Allow non-standard remote stages.
* Add changelog fragment.
4 years ago
Felix Fontein 050841324c
ansible-test: bump acme test container version to 2.0.0 (#71097) 4 years ago
Matt Clay b53e7f8720 Fix ansible-test self-test change classification.
Changes to sanity and unit tests now trigger the ansible-test self-test integration tests.

No changelog entry since this only affects tests for ansible itself and not collections.
4 years ago
Toshio Kuratomi e10902d744
Add the ansible_2_10 intersphinx reference. (#71028)
There are links from the scenario guides to collections but collections
docs aren't built in testing or locally when we're on the devel branch.
Due to that we need to make sure those references resolve to the
production docsite.  We can use intersphinx to make sure that happens.

* The test for binary files wasn't reading enough of the file.

Checking for null bytes in the first 1024 bytes failed to diagnose the
ansible_2_10.inv file as binary
4 years ago
Matt Clay 2e0097ada3 Fix ansible-test relative import analysis. 4 years ago
Matt Clay 1e02a201a6 Fix ansible-test virtualenv management. 4 years ago
Matt Clay 09f02980a4 Update ansible-test change detection for plugins. 4 years ago
Matt Martz ce570691e7
Test against galaxy_ng (#70303)
* Test against galaxy_ng

* Switch container image

* Remove redundant |default

* Re-enable

* Update image

* Update wording

* Don't use pulp as the container name
4 years ago
Matt Clay 74494a1908 Code cleanup for ansible-test coverage support. 4 years ago
Matt Clay 8d92df4537 Test netconf plugins in ansible-doc sanity test. 4 years ago
Matt Martz fa1fb2d13b
Test ansible-galaxy against pulp/pulp_ansible (#69605)
* Test galaxy cli against pulp

* linting fix

* Renames and small fixes

* Better handling for resetting pulp

* Clean up some things, add a comment

* I can't spell

* Bump fallaxy, use alternate pulp image

* Only reset pulp when we're are executing against pulp

* Update for updated pulp container

* Update some comments with correct URLs and typos

* Linting fix

* Pin pulp-fedora31 to a digest

* Address review comments for documentation
4 years ago
Matt Clay 9f49db1f99 Avoid use of deprecated junit-xml method. 4 years ago
Matt Clay 8ffaed00f8 Add Azure Pipelines support to ansible-test. 4 years ago
Matt Martz 4a4a11d282
Add mccabe complexity testing (#64623)
* Add mccabe complexity testing

* Make mccabe complexity an optional error

* Add mccabe to new sanity pylint requirements

* Add a changelog fragment.

Co-authored-by: Matt Clay <matt@mystile.com>
4 years ago
James Cassell 47d14a33bd
config: singular ANSIBLE_COLLECTIONS_PATH (#70007)
* config: singular ANSIBLE_COLLECTIONS_PATH

Every other *_PATH setting in ansible is singular, and the traditional
$PATH variable is also singular despite containing a list of
directories.  Let's be consistent both internally and with POSIX
tradition.

* update all ANSIBLE_COLLECTIONS_PATHS env references to be singular

* deprecate plural ANSIBLE_COLLECTIONS_PATHS setting
4 years ago
Felix Fontein a114da80ee
Use antsibull-changelog instead of packaged changelog generator (#69313)
Replace the ansible-base changelog linting and generation tool with antsibull-changelog and make it available for linting collections. Previously changelog linting was limited to ansible-base.
4 years ago
Gonéri Le Bouder 4869874337
ansible-test: vcenter initialize group/vmware (#70011)
Ensure the vcenter provider initialize the `module_defaults` of all
the vmware modules, not just `vmware_guest`.
The VMware CI relies on this for the authentication of the different
VMware modules.

The commit adjust `incidental_vmware_prepare_tests/tasks/init_vcsim.yml`.
The test-suite uses a copy of `vmware_guest` that is not in the
`group/vmware` group. As a result, we need to manually pass the
authentification parameter.
4 years ago
Matt Clay 6c2be04b6f Intercept code-smell tests using subprocess.
Some code-smell sanity tests for ansible-base use subprocess to invoke ansible commands.
Intercept these commands to make sure the correct script and python version are used.
4 years ago
Matt Clay a5679d4dd1
Additional ansible-test sanity adjustments. (#69974)
* Fix ansible-test check_pyyaml usage.
* Avoid unnecessary pip commands.
* Avoid pointless pip check.
* Only install ansible-test requirements as needed.
4 years ago
Matt Clay a81dd4f06a
Update ansible-test default containers. (#69819)
* Move ansible-test completion code.

* Fix a few type hints.

* Change docker completion based on context.

Collections now use version 2.0 of the default-test-container.
This is an updated version of the earlier 1.x default-test-container with ansible-base and cloud specific requirements removed.

Testing of ansible-base now uses version 1.0 of a new ansible-base-test-container.
This container is similar to the earlier 1.x default-test-container, but with unnecessary duplication of requirements across Python versions removed.

Collections which have tests that depend on requirements no longer present in the default test container should specify them in their test requirements files:

* tests/integration/requirements.txt
* tests/unit/requirements.txt

* Bump test container versions

Co-authored-by: Jordan Borean <jborean93@gmail.com>
4 years ago
Sloane Hertel 51f6d129cb
support hard coded module_defaults.yml groups for collections (#69919)
* Only allow groups which were hardcoded in module_defaults.yml

only load action groups from the collection if module_defaults contains a potential group for the action

* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml

Third party modules should not be using group/ - use the action name instead

* add externalized module_defaults tests

add the missing group and collections

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>

* changelog

ci_complete

* Fix import in tests

ci_complete

* Update with requested changes

ci_complete

* don't traceback since we don't validate the contents of module_defaults

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>
4 years ago
Matt Clay 7bff3d312f
Split out and install sanity test requirements. (#69971)
* Split out sanity test requirements.

* Run each --venv test separately.

This provides verification that the requirements for each test are properly specified.

* Use a separate requirements file per sanity test.

* Skip setuptools/cryptography setup for sanity.

* Eliminate pyyaml missing warning.

* Eliminate more pip noise.

* Fix conflicting generate_pip_install commands.

* Add changelog fragment.
4 years ago
Matt Clay 06efc41fb0
Reduce pip output noise in ansible-test. (#69925) 4 years ago
Jordan Borean 112251c7f4
Bump fallaxy version (#69917) 4 years ago
Felix Fontein 0b82d4499e
ansible-test ansible-doc sanity test: also run with --json parameter (#69288) 4 years ago
Felix Fontein 40f21dfd3c
Version source tagging (automatic and manual) for version_added and deprecation versions (#69680)
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.

* Make tagging/untagging functions more flexible.

* Tag all versions in doc fragments.

* Tag all deprecation versions issued by code.

* Make Display.deprecated() understand tagged versions.

* Extend validation to enforce tagged version numbers.

* Tag versions in tests.

* Lint and fix test.

* Mention collection name in collection loader's deprecation/removal messages.

* Fix error IDs.

* Handle tagged dates in Display.deprecated().

* Also require that removed_at_date and deprecated_aliases.date are tagged.

* Also automatically tag/untag removed_at_date; fix sanity module removal version check.

* Improve error message when invalid version number is used (like '2.14' in collections).
4 years ago
Jordan Borean f5f3ba7ab5
ansible-test - fix up relative util import for powershell validate-modules (#69753)
* ansible-test - fix up relative util import for powershell validate-modules

* Use different tactic for generic group

* Use python 2 and 3
4 years ago
Felix Fontein ea04e0048d
Allow to deprecate options and aliases by date (#68177)
* Allow to deprecate options and aliases by date instead of only by version.

* Update display.deprecate().

* Adjust behavior to conform to tested behavior, extend tests, and improve C# style.

* Parse date and fail on invalid date.

This is mainly to make sure that people start using invalid dates, and we eventually have a mess to clean up.

* C# code: improve validation and update/extend tests.

* Make sure that deprecate() is not called with both date and version.

* Forgot to remove no longer necessary formatting.

* Adjust order of warnings in C# code.

* Adjust unrelated test.

* Fix grammar (and make that test pass).

* Don't parse date, and adjust message to be same as in #67684.

* Sanity tests: disable date in past test.

* Validate-modules: validate ISO 8601 date format.

* Validate-modules: switch schema declaration for deprecated_aliases to improve error messages for invalid dates.

* Use DateTime instead of string for date deprecation.

* Validate that date in deprecated_aliases is actually a DateTime.

* Fix tests.

* Fix rebasing error.

* Adjust error codes for pylint, and add removed_at_date and deprecated_aliases.date checks to validate-modules.

* Make deprecation date in the past error codes optional.

* Make sure not both version and date are specified for AnsibleModule.deprecate() calls.

* Stop using Python 3.7+ API.

* Make sure errors are actually reported. Re-add 'ansible-' prefix.

* Avoid crashing when 'name' isn't there.

* Linting.

* Update lib/ansible/module_utils/csharp/Ansible.Basic.cs

Co-authored-by: Jordan Borean <jborean93@gmail.com>

* Adjust test to latest change.

* Prefer date over version if both end up in Display.deprecated().

Co-authored-by: Jordan Borean <jborean93@gmail.com>
4 years ago
Matt Davis f7dfa817ae
collection routing (#67684)
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
4 years ago
Matt Martz b748edea45
Add multipart/form-data functionality (#69376)
* Add multipart/form-data functionality

* Fix some linting issues

* Fix error message

* Allow filename to be provided with content

* Add integration test

* Update examples

* General improvements to multipart handling

* Use prepare_multipart for galaxy collection publish

* Properly account for py2 vs py3, ensuring no max header length

* Address test assumptions

* Add unit tests

* Add changelog

* Ensure to use CRLF instead of NL

* Ignore line-endings in fixture

* Consolidate code, add comment

* Bump fallaxy container version

* ci_complete
4 years ago
Felix Fontein 43acd61901
ansible-test local change detection: use --base-branch if specified (#69508) 4 years ago
Matt Clay c495c92a6e Code cleanup in ansible-test. 4 years ago
Matt Clay e7c2eb519b
Add Shippable request signing to ansible-test. (#69526) 4 years ago
Matt Clay d8e0aadc0d Update ansible-test support for CI providers.
Refactored CI provider code to simplify multiple provider support and addition of new providers.
4 years ago
Felix Fontein 0e15375ffe
Add deprecated removed_in_version and deprecated_aliases version tests (#66920) 4 years ago
Matt Davis 776f4840fc
fix `ansible-test units` to work(ish) under podman (#69462)
* ignore the missing `Networks` key, issue a warning that network disconnection won't function
4 years ago
Matt Clay 1cf26896c5 Default ansible-test provisioning to us-east-1.
Previously windows instances on Shippable would be automatically directed to us-east-2.
4 years ago
Matt Clay a050d892d8 Remove obsolete vcenter setup from ansible-test. 4 years ago
Matt Clay 5a3ca0ae5f Fix ansible-test delegation from pip install. 4 years ago
Matt Clay 122743ae30
Update ansible-test handling of delegation paths. (#69056) 4 years ago
Yanis Guenane 46d82179d8
Testing: Add support for CentOS Linux On Power platform (#68130)
* Testing: Add CentOS Linux On Power platform

* Add arch designation to remotes.

This avoids overloading the provider with the arch.

Also add a changelog entry.

Co-authored-by: Matt Clay <matt@mystile.com>
4 years ago
Matt Clay a095172130 Fix ansible-test --docker without tests dir. 4 years ago
Felix Fontein bae5f18f5c
ansible-test: don't trigger full CI run for changes to changelogs/ and docs/ in collections (#68550)
* Don't trigger full CI run for changes to changelogs/ and docs/ in collections.

* Add changelog fragment.

* Update changelogs/fragments/68550-ansible-test-docs-changelogs.yml

Co-Authored-By: Matt Clay <matt@mystile.com>

Co-authored-by: Matt Clay <matt@mystile.com>
4 years ago
Matt Clay 148e83f832
Fix ansible-test submodule handling. (#68759)
* Refactor ansible-test integration test.
* Add env --list-files option.
* Add tests for collection files tracked by git.
* Fix ansible-test submodule usage on older git.
* Fix submodule directory detection as files.
* Improve handling of nested source control.
4 years ago
Rick Elrod 13aef3c2e5
Add --docker-terminate flag to ansible-test (#68688)
* Add --docker-terminate flag to ansible-test

Signed-off-by: Rick Elrod <rick@elrod.me>
4 years ago
Jordan Borean 6db66bcadd
Add relative module_util support for powershell (#68321)
* Add relative module_util support for powershell

* Added ansible-test classification support
4 years ago
Matt Clay 7323d5dd0d Fix references to old egg-info directory. 4 years ago
Matt Clay 735885d57c Fix ansible-test requirements installation. 4 years ago
Matt Clay 53a3d1ffdb Fix ansible-test change detection traceback. 4 years ago
Matt Clay d8b5c11a63 Report error for coverage 5+ in ansible-test. 4 years ago
Matt Clay 27fc049993
Fix ansible-test code coverage collection. (#68502)
The last task in a play should now properly report code coverage.

This change should also eliminate empty coverage files, as well as incomplete coverage files resulting from early worker termination.
4 years ago
Jordan Borean e12cea2c7c
ansible-test - fix up powershell module_util analysis for collections (#68422) 4 years ago
Yanis Guenane fb2a7ce923 Testing: Move from ibmcloud to ibmps 4 years ago
Matt Clay f1b3e8364e Fix ansible-test PATH handling. 4 years ago
Matt Clay 9765a80bd4
Fix ansible-test coverage analysis option usage. (#68406)
* Fix ansible-test coverage analysis option usage.

The `--input-dir` option for `coverage analyze targets generate` was being ignored.

No changelog entry since this feature has not yet been released.

* Move coverage config to fix type annotations.

Declaring the types before referencing them makes sure they're recognized by tools such as PyCharm.
4 years ago
Matt Clay 70827c7923 Fix ansible-test import analysis warning.
Fix overlooked in https://github.com/ansible/ansible/pull/68372/
4 years ago
Matt Clay 10fe54de58 Fix ansible-test module_utils import analysis.
Now empty `*.py` files are ignored during module_utils import analysis for change detection.
This eliminates "No imports found" warnings for files which should have no imports.
4 years ago
Matt Clay cf75e3f526 Fix ansible-test import analysis for collections. (#68352)
* Fix ansible-test import analysis for collections.

* Ignore plugins/module_utils/__init__.py
4 years ago
Matt Clay 2c7f8bf586 Fix ansible-test windows-integration module refs. 4 years ago
Matt Clay 8715bc400a Add coverage filtering to ansible-test. (#68158)
* Relocate expand_indexes so it can be reused.

* Add generate_indexes function.

* Simplify type annotations.

* Add `coverage analyze targets filter` command.

* Add changelog entry.
4 years ago
Matt Clay 27ce6dbc1f Remove test support option from ansible-test.
This option was added to facilitate migration and is no longer needed.
4 years ago
Matt Clay 50442f9fdd Fix ansible-test handling of no Python coverage. 4 years ago
Matt Clay 1a8fdaadc7 Fix powershell coverage path rewriting.
This applies the same rewrite logic to PowerShell coverage as is used for Python coverage.
4 years ago
Matt Clay ebb255f0d3 Add Python 3.9 to ansible-test.
ci_complete
ci_coverage
4 years ago
Matt Clay c06f442409 Fix ansible-test import sanity test venv setup. 4 years ago
Matt Clay 99e657162f Fix ansible-test module_utils analysis. 4 years ago
Matt Clay 7c8b046b5f
Fourth batch of incidental integration tests. (#67873)
* Copy in incidental posix tests.

* Update incidental test aliases.

* Update target names.

* Add support plugins.

* Fix paths.

* Update ignores.

* Update integration-aliases sanity test.

* Add incidental tests to CI.
4 years ago
Jordan Borean 26129fcb80
ansible-galaxy - optimise some paths and use fake galaxy int tests (#67685)
* ansible-galaxy - optimise some paths and use fake galaxy int tests

* Added init, built, and publish tests

* Test against both mocked Galaxy and AH server

* Finish off writing the install tests

* Fix up broken tests

* Rename test target and add migrated tests

* Use cloud provider for Galaxy implementation

* Added blank static config

* Use correct alias group

* Set release version and fix copy typo

* Remove reset step as it is no longer needed

* Use sane env var names for test container name
4 years ago
Matt Clay f735fd672a
Third batch of incidental integration tests. (#67830)
* Copy in incidental windows tests.

* Update incidental test aliases.

* Add support plugins.

* Update target references.

* Update sanity ignores.

* Update integration-aliases test.

* Add to CI.
4 years ago
Matt Clay b68e55da26
Remove `test/legacy/` directory. (#67786)
* Remove `test/legacy/` directory.

* Remove references to `test/legacy/` directory.

* Update the remaining valid legacy reference.

* Remove outdated legacy references.
4 years ago
Matt Clay e3591223a0
Second batch of incidental integration tests. (#67765)
* Update incidental test aliases.

* Rewrite target references for renamed targets.

* Add incidental tests to CI.

* Update sanity tests for incidental cloud tests.

* Initial copy of incidental tests.

* Copy contrib files into test.

* Update paths in test.

* Add support plugins.

* Update plugin to work around missing deps.

* Update sanity ignores.

* Fix matrix entries.

* Remove debug echo.
4 years ago
Jill R 96244c4cde
Make AWS test plugin to refer to the aws-permissions-for-integration-tests (#67665)
Make AWS test plugin to refer to the aws-permissions-for-integration-tests section of the AWS devel docs now that we have them, rather than suggesting to flag down specific users.
4 years ago
Matt Clay 482885e0c8 Fix network setup for collections in ansible-test. 4 years ago
Matt Clay 542d1b98ff
Support private test plugins in ansible-test. (#67660)
* Support private test plugins in ansible-test.

This feature is exclusively used for testing Ansible itself and is not available to collections.

Content in the newly supported directories will be added in follow-up PRs.

* Add new directory to path classification.

* Support new paths in shebang test.

* Add new directory to manifest.
4 years ago
Yanis Guenane 34800fe027 Test: Bump Tower version to latest release 4 years ago
Matt Martz c61c0f7ad5
Add warning when running devel (#67502)
* Add warning when running devel. Addresses #67362

* Add config entry to disable devel warning, so it doesn't impact CI

* Update warning about using devel
4 years ago
Matt Clay 19f6fc68d4
Relocate Shippable tools. (#67556)
* Move Shippable tools to hacking directory.

These limits the `test/utils/shippable/` directory to scripts required for CI.

* Fix `test/utils/shippable/` file classification.

* Update package-data sanity test.
4 years ago
Yanis Guenane b9bd2c25ca
Test: Bump TOWER_VERSION to supported version (#67378)
* Revert "Remove Tower module tests from CI. (#67368)"

This reverts commit b041d96762.

* Test: Bump TOWER_VERSION to 3.6.2
4 years ago
Matt Clay 5e68bb3d93
Add code coverage target analysis to ansible-test. (#67141)
* Refactor coverage file enumeration.
* Relocate sanitize_filename function.
* Support sets when writing JSON files.
* Generalize setting of info_stderr mode.
* Split out coverage path checking.
* Split out collection regex logic.
* Improve sanitize_filename type hints and docs.
* Clean up coverage erase command.
* Fix docs and type hints for initialize_coverage.
* Update type hints on CoverageConfig.
* Split out logic for finding modules.
* Split out arc enumeration.
* Split out powershell coverage enumeration.
* Raise verbosity level of empty coverage warnings.
* Add code coverage target analysis to ansible-test.
4 years ago
Matt Clay f4a80bb600
Code cleanup and refactoring in ansible-test. (#67063)
* Code cleanup in ansible-test.
* Split out encoding functions.
* Consoldate loading of JSON files.
* Split out disk IO functions.
* Simplify file access.
* Add functions for opening files.
* Replace open calls with appropriate functions.
* Expose more types from typing module.
* Support writing compact JSON.
* Add verbosity argument to display.warning.
* Add changelog entry.
* Update files overlooked during rebase.
* Use `io.open` instead of `open`.
* Fix file opening for imp.load_module.
* Remove use of `r+` mode to access files.
* Add missing import.
* Fix httptester on Python 2.x.
* Clarify changelog fragment.
* Consolidate imports. Remove extra newlines.
* Fix indirect imports.
4 years ago
Matt Clay d584584474 Reorganize ansible-test coverage code.
This change moves all code for the `ansible-test coverage` command into the `coverage` directory.

Each subcommand is split into a separate file.

Only minor spelling changes were made aside from code relocation.
4 years ago
Matt Clay 49d8d5ae33 Fix ansible-test tar format handling. 4 years ago
Yanis Guenane 4fd2dce7f3
Testing: Add support for AIX platform (#65802) 4 years ago
Matt Clay 3c3cf50e7a Fix ansible-test coverage of ansible-connection. 4 years ago
Matt Clay efd2dd8929 Add more ansible-test args to delegation config. 4 years ago
Matt Clay ee50792416 Add a `--no-pip-check` option to ansible-test. 4 years ago
Matt Clay 8b63da7b43 Fix ansible-test egg-info directory detection. 5 years ago
Matt Clay ebf21bb48d Fix ansible-test inventory delegation. 5 years ago
Matt Clay 934d6b235b Add `--venv-system-site-packages` to ansible-test. 5 years ago
Gonéri Le Bouder 68a8cc2b1f vcenter: don't bind container on <1024 port
Bind the govcsim service on port 1443, this way we don't need root privilege
to start the container. This allow us to use ansible-test with podman
seamlessly.
The commit also avoid the mapping of port 80. We don't need it.
5 years ago
Gonéri Le Bouder 16d3f6bcbb vcenter provider: do not set env var
Some secrets are exposed because of the environment variables. The
Ansible variables are now enough, we don't need to expose the envvar
anymore.
5 years ago
Toshio Kuratomi 2b6d94c4c8 Change the check_pyyaml() function to return the features of pyyaml that we check 5 years ago
Toshio Kuratomi ece306b201 Add a sanity test for yaml to make sure we can parse all yaml
Check that all yaml we ship is parsable by the pyyaml c backend.  Since
Ansible uses Pyyaml for docs and playbooks, if the yaml files aren't
parsable, they will error out if they were used.

Warn and skip yamllint if libyaml backend is not present

Ignore new errors in examples until someone can fix them
5 years ago
Matt Clay fb69d68821
Fix plugin paths for ansible-test pylint test. (#65526)
* Fix plugin paths for ansible-test pylint test.
5 years ago
Matt Clay ba273c72d8
Fix ansible-test handling of git submodules. (#65027)
* Revert most of PR #61605 commit e218c9814c

This removes the git error handling that converted all git errors into warnings.

* Fix ansible-test handling of git submodules.
5 years ago
Matt Clay 4f51b7d323 Remove ansible-test --tox delegation option.
Use the --venv option instead.

This option was only available when running from source to test the ansible/ansible repository.

This will have no effect on testing collections or running from an installed version of Ansible.

Also update docs to reference the --venv option instead of the --tox option.
5 years ago
Felix Fontein f4ff45dcd4 ansible-test: bumping ACME test container version (#64424)
* Bumping ACME test container version.

* Add changelog.
5 years ago
Matt Clay e9f8a34dce
Fixes for validate-modules import handling. (#63932)
* Fix validate-modules support for collections.

- Relative imports now work correctly.
- The collection loader is now used.
- Modules are invoked as `__main__`.

* Remove obsolete validate-modules code ignores.

* Handle sys.exit in validate-modules.

* Add check for AnsibleModule initialization.

* Remove `missing-module-utils-import` check.

This check does not support relative imports or collections.

Instead of trying to overhaul the test, we can rely on the `ansible-module-not-initialized` test instead.

* Fix badly named error codes with `c#` in the name.

The `#` conflicts with comments in the sanity test ignore files.

* Add changelog entries.
5 years ago
Matt Clay 92ccdeac31 Fix various import sanity test issues.
- Relative imports are now properly recognized.
- Correct script invocation of Ansible modules is used.
- Warnings are now consistently reported as errors.
- Errors are now consistently reported with the file tested.

Resolves https://github.com/ansible/ansible/issues/62723
Resolves https://github.com/ansible/ansible/issues/61884
5 years ago
Matt Clay 6be4741f72 Fix ansible-test env var mixing running commands. 5 years ago
Gonéri Le Bouder 73febd4ea6 prepare_vmware_tests: use module_defaults (#63209) 5 years ago
Matt Davis 6d52bdf4db
fix default collection resolution in adhoc (#63298)
* fix default collection resolution in adhoc

* if an adhoc command is run with a playbook-dir under a configured collection, default collection resolution is used to resolve unqualified module/action names

* Set ANSIBLE_PLAYBOOK_DIR in integration tests.

* Fix config conflict in ansible integration test.

* add adhoc default collection test

* text-ify warning string
5 years ago
Matt Clay 022335669c
Work around ssh-keygen issue in ansible-test. (#63211)
Newer versions of ssh-keygen create PEM keys that are not recognized by Paramiko.

Now ansible-test compensates for this by updating they keys it generates so Paramiko will recognize them.
5 years ago
Matt Clay 4c79f1ec4d Fix ansible-test integration temp dir location.
Previously the temporary directory used to run integration tests resided under the user's home directory. This prevented ansible-playbook from detecting the default collection when running tests.

Now the temporary directory is created within the collection to facilitate default collection detection.
5 years ago
Matt Clay 32979430d0 Add missing var for format string in ansible-test. 5 years ago
Matt Clay b91f452f4f Fix ansible-test virtualenv real python search.
ansible-test now properly searches for `pythonX.Y` instead of `python` when looking for the real python that created a `virtualenv`.
5 years ago
Matt Clay 3f2380ccce Exclude tmp dir from ansible-test results copy.
This fixes test errors related to failures copying temporary test results files from a remote system back to the local system.

It also speeds up processing of test results and reduces network utilization by avoiding the temporary files.
5 years ago
Matt Clay 831e1bf2e0 Fix ansible-test PYTHONPATH handling.
Running from an installed version of ansible-test now results in tests using a dedicated directory for PYTHONPATH instead of using the site-packages directory where ansible is installed.

This provides consistency with tests running from source, which already used a dedicated directory.

Resolves https://github.com/ansible/ansible/issues/62716
5 years ago
Gonéri Le Bouder 5ecbe9cbbb vcenter provider: Only rely on VMWARE_TEST_PLATFORM
Until now, the vcenter provider was switching between `static` and
`govcsim` depending on the presence of the following configuration file:
`test/integration/cloud-config-vcenter.ini`.

This was not consistent with Worldstream, which we enable with the
`VMWARE_TEST_PLATFORM` environment variable.

We now only rely on `VMWARE_TEST_PLATFORM` to know which platform should be
used. `govcsim` is still the default, this to preserve the original
behaviour.

This commit also rename the following variables to be consistent with the rest
of the code base. It also ensures they are alway defined, even with `govcsim`:

- `VCENTER_HOSTNAME`
- `VCENTER_USERNAME`
- `VCENTER_PASSWORD`
5 years ago
Matt Clay 9f7b124a6f Mark ansible-test cloud credentials as sensitive. 5 years ago
Matt Clay 0631e057e9 Redact sensitive values by default in ansible-test 5 years ago
Felix Fontein d8d3790d6a Bump acme-test-container version. (#62381) 5 years ago
Matt Clay b73e7721df Mark AWS credentials in ansible-test as sensitive.
This avoids displaying the credentials in CI when retrying tests at maximum verbosity.
5 years ago
Matt Clay 32d965e2c6 Fix display of sanity test doc links.
The documentation links are now displayed when running from an install.

Previously the links were only displayed when running from source.
This was due to ansible-test checking for the presence of documentation files locally, which are only present when running from source.
The check is no longer necessary since there is a sanity test in place to enforce the presence of documentation for all sanity tests.
5 years ago
Matt Clay cd4882e229 Fix ansible-test vcenter test filtering.
The default behavior of the ansible-test vcenter plugin is to use the govcsim container to run tests.

However, unless the govcsim mode was specified using the VMWARE_TEST_PLATFORM environment variable, the filter code would skip the tests unless the tests ran on Shippable or the user had an ansible-core-ci key.

Now the filter correctly recognizes that govcsim is the default.
5 years ago
Matt Clay cdc4926340
Fix ansible-test collections requirements installation. (#62181)
* Fix location of unit test requirements.

* Preserve ansible-test unit test requirements.

* Remove redundant unit test requirements.

* Fix location of network test requirements.

* Preserve ansible-test network test requirements.

* Remove redundant network test requirements.

* Add missing ordereddict requirements.

* Load collection requirements correctly.

* Add changelog fragment.
5 years ago
Matt Clay aaa6d2ecb0
Fix ansible-test pytest plugin loading. (#62119)
* Avoid assertion rewriting in pytest plugins.

Adding PYTEST_DONT_REWRITE to the ansible-test pytest plugin docstrings disables assertion rewriting in pytest for those plugins.

This avoids warnings during test execution if the plugins are loaded multiple times (such as being imported within tests).

* Run ansible-test pytest plugins early.

The ansible-test pytest plugins need to load and run earlier than conftest modules.

To facilitate this, the pytest_configure function is run during loading, which works since they are loaded (but not always run) before conftest modules are loaded.

A check has also been added to the pytest_configure functions to prevent them from running multiple times in the same process.

* Load pytest plugins using an env var.

The -p command line option loads plugins before conftest, but only during collection.
The PYTEST_PLUGINS environment variable loads plugins before confest, both during collection and test execution.
5 years ago
Jordan Borean 6fb1d56fdc
Fix ansible-test coverage --all (#62115) 5 years ago
Matt Clay a7bc11ce67
Work around virtualenv/venv issue in ansible-test. (#62111)
Creating a virtual environment using `venv` when running in a virtual environment created by `virtualenv` results in a copy of the original virtual environment instead of creation of a new one.

To work around this, `ansible-test` now identifies when it is running in a `virtualenv` created virtual environment and uses the real Python interpreter to create the `venv` virtual environment.
5 years ago
Matt Clay 983f0ad5a0 Properly exclude tests/output/ from code coverage. 5 years ago
Matt Clay f110abb806
Fix ansible-test to ignore `tests/output/`. (#62084)
The `test/results/` directory for Ansible test output was already ignored when not using git.

When Ansible Collections were switched to `tests/output/` the ignore entry was previously overlooked.
5 years ago
Matt Clay 064e8e1ef4
Fix ansible-doc traceback and sanity test. (#62040)
* Fix ansible-doc traceback for removed modules.

This avoids tracebacks with errors like the following when a module has been removed:

module module_name missing documentation (or could not parse documentation): 'NoneType' object does not support item assignment

* Fix ansible-doc sanity test warning handling.

Warnings about removed modules/plugins on stderr are now properly ignored.

Previously an ansible-doc error could result in unrelated errors going undetected because tests were stopped early and the underlying error was ignored.
5 years ago
Matt Clay c77ab11051
Fix issues with ansible-test --venv option. (#62033)
* Fix ansible-test venv activation.

When using the ansible-test --venv option, an execv wrapper for each python interpreter is now used instead of a symbolic link.

* Fix ansible-test execv wrapper generation.

Use the currently running Python interpreter for the shebang in the execv wrapper instead of the selected interpreter.

This allows the wrapper to work when the selected interpreter is a script instead of a binary.

* Fix ansible-test sanity requirements install.

When running sanity tests on multiple Python versions, install requirements for all versions used instead of only the default version.

* Fix ansible-test --venv when installed.

When running ansible-test from an install, the --venv delegation option needs to make sure the ansible-test code is available in the created virtual environment.

Exposing system site packages does not work because the virtual environment may be for a different Python version than the one on which ansible-test is installed.
5 years ago
Gonéri Le Bouder 8e22a60f97 ansible-test: vcenter+Worldstream: hide password
Hide the temporary password when `ansible-test` is called with the
`--redact` parameter.
5 years ago
Sam Doran e218c9814c ansible-test - Continue if the git command returns an error (#61605)
* ansible-test - Contiune if the git command returns an error

* Just return stdout

* Use to_text() when displaying exception

* Add a message property to SubprocessError
5 years ago
Matt Clay e4e5005640
Fix ansible-test coverage path handling. (#61528)
* Fix ansible-test coverage path handling.
* Split CI unit tests into two groups.
5 years ago
Matt Clay 4063d58339 Use relative submodule status in ansible-test.
The `git submodule status` command is relative to the current git repository by default.
When running from a repository subdirectory paths can be returned above the current directory.
Specifying the current directory with `git submodule status` avoids listing submodules above that directory.

This will fix issues when testing a collection that is rooted below the repository root when that repository uses submodules.
5 years ago
Matt Clay 2215a62057 Fix ansible-test handling of submodules.
Resolves https://github.com/ansible/ansible/issues/61550
5 years ago
Matt Clay f90587069c
Update collections integration targets path. (#61523)
* Update collections integration targets path.

* Fix integration path handling.

* Add collections test target fallback.

Also add warnings and errors for common path mistakes.

* Improve role target detection.
5 years ago
Matt Clay 032bd3a030 Move the ansible-test output_dir.
This directory is currently a fixed location to make troubleshooting easier.
It is cleared before each test target runs, but is preserved when a test target finishes.
This allows the contents to be inspected when a test fails.

The previous location was `~/ansible_testing/`.
The new location is within the content root:

- `test/results/.tmp/output_dir` for Ansible
- `tests/output/.tmp/output_dir` for Ansible Collections

Moving the directory reduces the number of places on the filesystem where tests create output.
It also enables the results to be returned from delegated systems.
5 years ago
Matt Clay d92e2a6b0e
Simplify ansible-test target processing. (#61506)
* Clean up layout paths for integration tests.

* Remove "special" integration test target type.

* Remove unnecessary role detection logic.

* Remove support for non-sh runme scripts.

* Simplify reading of aliases.
5 years ago
Matt Clay 81ca44b7e5 Change location of collections tests.
The directories used for collections tests are changing as follows:

`test/` -> `tests/`
`test/results/` -> `tests/output/`

This is a breaking change for collections tests executed by ansible-test.
All collections will need to be updated to use the new directory.

The `tests/output/` directory should be added to the `.gitignore` or equivalent in each collection.

This change is being made before the first pre-release of Ansible 2.9 since ansible-test has not yet been shipped.
Using the `tests/` directory matches the other collections directories `plugins/` and `roles/`.

This resolves https://github.com/ansible/ansible/issues/60218
5 years ago
Matt Clay 460b583dff Improve ansible-test --venv error handling.
This allows it to fall back to virtualenv when venv is broken.
5 years ago
Sam Doran 8dd79fbbd2 Show doc URL for sanity test failures 5 years ago
Matt Clay 4afe757f93
More ansible-test path handling updates. (#61484)
* Move test results vars into layout.

* Add test_path to layout.

* Add sanity_path to layout.

* Clean up layout.

* Update change classification.

* Improve classification layout.

* Relocate common classification code.

* Use is_subdir for relocated code.

* Relocate ansible test/units/compat/ hack.

* Fix variable shadowing.

* Remove unused code.

* Fix ordering of path tests.

* Clean up ansible classification.

* Fix hard-coded plugin paths in classification.

* Relocate more common classification.

* Fix logic.

* Fix pylint issue.

* Add missing licenses classification.
5 years ago
Sam Doran 73248bf27d validate-modules - Use error codes instead of numbers (#60711) 5 years ago
Matt Clay 1db78d4ba4 Attempt quick fix for CI failures. 5 years ago
Matt Clay 830f995ed4
Add a --venv option to ansible-test. (#61422)
* Add --venv delegation to ansible-test.

* Update import sanity test venv creation.

* Fix import test when using --venv on Python 2.x.

* Improve virtualenv setup overhead.

* Hide pip noise for import sanity test.

* Raise verbosity on venv info messages.

* Get rid of base branch noise for collections.

* Add missing --requirements check.
5 years ago
Matt Clay f5d829392a
Overhaul ansible-test test path handling. (#61416)
* Remove .keep files from test/results/ dirs.

* Remove classification of test/results/ dir.

* Add results_relative to data context.

* Use variables in delegation paths.

* Standardize file writing and results paths.

* Fix issues reported by PyCharm.

* Clean up invocation of coverage command.

It now runs through the injector.

* Hack to allow intercept_command in cover.py.

* Simplify git ignore for test results.

* Use test result tmp dir instead of cache dir.

* Remove old .pytest_cache reference.

* Fix unit test docker delegation.

* Show HTML report link.

* Clean up more results references.

* Move import sanity test output to .tmp dir.

* Exclude test results dir from coverage.

* Fix import sanity test lib paths.

* Fix hard-coded import test paths.

* Fix most hard-coded integration test paths.

* Fix PyCharm warnings.

* Fix import placement.

* Fix integration test dir path.

* Fix Shippable scripts.

* Fix Shippable matrix check.

* Overhaul key pair management.
5 years ago
Jordan Borean 66f52b74b1 Change collection PS util import pattern (#61307)
* Change collection PS util import pattern

* Add changes for py2 compat

* fix up regex and doc errors

* fix up import analysis

* Sanity fix for 2.6 CI workers

* Get collection util path for coverage collection
5 years ago
Jordan Borean faaa669764
PowerShell - Added coverage collector (#59009)
* Added coverage collection for PowerShell - ci_complete ci_coverage

* uncomment out coverage uploader call

* Generate XML for PowerShell coverage

* Use whitelist to exclude coverage run on non content plugins

* Remove uneeded ignore entry

* Try to reduce diff in cover.py

* Fix up coverage report package - ci_complete ci_coverage
5 years ago
Gonéri Le Bouder e7a9d71ac0
ansible-test: async deployment w/ vcenter provider (#59984)
ansible-test: async deployment w/ vcenter provider

So far, `ansible-test`'s vcenter provider was trigger by a blocking POST
call. Since the depoyment take a lot of time (> 8m), we decided to use
an async call instead and poll the API server until the deployment
is ready.
5 years ago
Matt Clay 8a7b89c66f Support delegation testing collections with deps.
All collections in the same collection root as the collection being tested will be sent to the remote system/container.
5 years ago
Matt Clay e258ba703e
Improve ansible-test env display. (#61339)
* Improve ansible-test env display.
* Fix PyCharm warnings.
* Fix ansible-test ansible-doc sanity test with -e.
5 years ago
Matt Clay 02c38ed4fd
Relocate ansible-test self tests outside package. (#61255)
* Relocate ansible-test self tests outside package.

We don't want to include the tests for verifying ansible-test within the ansible-test package.

* Add `test/ansible_test/` to classification.py.

* Fix test invocation.

* Relocate tests in MANIFEST.in.

* Improve package-data sanity test error checking.

* Only use includes for ansible-test in MANIFEST.in.

* Improve readability of MANIFEST.in.
5 years ago
Matt Clay 2a9b996bd2 Use `.cfg` extension for pylint configs. 5 years ago
Matt Clay 6ec0b4ec86
Sanity test updates for collections support. (#61248)
* Run no-unwanted-files sanity test only on Ansible.

Since collections should be able to use binary modules there is not really any limit on what could exist in a collection `plugins` directory.

* Add support for symlinks in sanity target lists:

- Sanity tests that need to analyze symlinks can do so using the supplied target list.
- Tests that analyze directories will now only look at symlinks if requested.
- Directory symlinks will now be seen as directories instead of files.

* Enable symlinks on filename based sanity tests.

Sanity tests that evalulate filenames instead of content should include symlinks.

* Update symlinks sanity test.

Use the sanity test target list now that it can include symlinks.
5 years ago
Felix Fontein 4551965af1 ACME modules: bump test container version (#61192)
* Fix root cert path.

* Bump testing container version.

* Improve check (needed due to letsencrypt/pebble#257).
5 years ago
Matt Clay 2eff25aef8 Update ansible-test network inventory handling.
DCI uses an inventory directory instead of a file, along with --no-temp-workdir.

This changes the inventory check to keep that scenario working until all DCI jobs can be migrated to Zuul.
5 years ago
Matt Martz adb886e4ce
Speed up units using time.sleep by mocking or shortening sleep time (#61064)
* Speed up units using time.sleep by mocking or shortening sleep time

* Only use durations when not a collection
5 years ago
Matt Clay 3551d83289 Fix ansible-test payload bin creation. 5 years ago
Matt Clay 29ac0273d4
Update ansible-test collection inventory handling. (#61031)
* Update ansible-test collection inventory handling.

- The `windows-integration` command now supports the `--inventory` option.
- The incomplete support for host_vars and group_vars directories has been removed.
- The incomplete support for an inventory directory has been removed.
- The inventory specified by `--inventory` can now reside outside the install and content roots.
- Using `ansible_ssh_private_key_file` with `--docker` or `--remote` results in a warning about the combination being unsupported and likely to fail.

* Fix config handling.

* Fix payload handling of ssh keys.

* Disable pylint no-self-use rule for ansible-test.

* De-duplicate payload paths.
5 years ago
Matt Clay 32b0a72547
Support windows-integration on collections. (#60984)
* Support windows-integration on collections.

* Update configure-remoting-ps1 sanity test.

* Add copy of ConfigureRemotingForAnsible.ps1.

* Add ignore entry for copied file.
5 years ago
Matt Clay 92380d7050
Update ansible-test --windows inventory creation. (#60911)
Host specific settings are created as part of the generated inventory instead of using a host_vars directory.

Also update connection tests to work properly with the generated inventory changes.
5 years ago
Matt Clay ebe55e83c3 Fix ansible-test bin creation.
Use symlink instead of link.
5 years ago
Matt Martz def3d1f815
validate-modules: support collections (#60247)
* Start of work to support collections

* remove version_added from base schema

* If a collection, pass that to validate-modules

* clean ups

* Allow version_added in a collection, just make it optional

* Don't traceback on missing doc_fragment

* Don't validate metadata in a collection
5 years ago
Matt Clay d9c44897a6 Fix ansible-test cloud vars overwrite. 5 years ago
Matt Clay 7968dc4c00 Load integration config test vars from content.
The `test/integration/integration_config.yml` vars file will now be loaded from the content under test and is now optional.

The `output_dir` and `win_output_dir` vars are now provided by ansible-test.
5 years ago
Matt Clay ba1b4565d3 Clean up ansible-test results path handling.
This brings us closer to being able to use `tests/` instead of `test/` in collections.
5 years ago
Matt Clay 1b8aa798df
Update ansible-test integration config handling. (#60448)
* Move Ansible test config into env vars.

This allows ansible-test to use an empty Ansible config file, leaving open the option for users to customize the one used to run tests (although such usage is discouraged).

* Use config from content under test when present.
5 years ago
Matt Clay 25d396fdfc Fix testing of Windows binary modules. 5 years ago
Gonéri Le Bouder a3407682ed ansible-test: fix the vcenter+vcsim support
The vcenter provider allow one to use the `VMWARE_TEST_PLATFORM` to
manually select a test platform. The test platforms are:

- `govcsim`
- `static`
- and `worldstream`.

Before this commit, the `govcsim` value was ignored.

Example:

```shell
$ VMWARE_TEST_PLATFORM=govcsim ansible-test integration --python 3.7 needs/target/prepare_vmware_tests/
```
5 years ago
Matt Clay 3fac6a0adb Fix ansible-test TypeVar type annotations. 5 years ago
Matt Clay cb68110296
Clean up ansible-test inventory templates. (#60321)
* Move ansible-test inventory templates.
* Remove obsolete inventoy.remote.template file.
5 years ago
Matt Clay 2a9964ede8
Code cleanup in ansible-test. (#60318)
* Move ansible-test units code into separate dir.
* Fix --explain on "code smell" sanity tests.
* Add --strict-markers to pytest invoctaion.
* Fix classification of units code changes.
5 years ago