Commit Graph

11937 Commits (ec82eba5ba9015650c46ad698eb155906a35a9f8)

Author SHA1 Message Date
Sloane Hertel 2464e1e91c
Fix vars plugin code and documentation mismatch (#78562)
* Add a warning for collections that are attempting to be autoloaded to no effect
* Deprecate REQUIRES_WHITELIST and add support for REQUIRES_ENABLED so the docs are accurate
* Fix documentation
* add more vars plugin tests
* Simplify code and add a FIXME for another bug
* fix precedence
* Make setting the class attr at all a warning, even if it's True
* Add fun parsing for _load_name
* include _load_name in messages
2 years ago
Martin Krizek 19e7c5b0c1
_get_next_task_from_state should not mutate anyting outside of state (#78657)
Fixes #78612
2 years ago
Matt Martz d8cb9117ac
Skip test if no there are no problematic chars for wcswidth (#78645) 2 years ago
Sloane Hertel 12cec69545
Fix validate-modules version_added test (#78554) 2 years ago
Sviatoslav Sydorenko f1c56e988d Fail fast in stuck `ansible-galaxy-collection`
This specific integration test gets stuck periodically causing the
Galaxy jobs to be killed on timeout wasting an hour of runtime. The
module that gets stuck waiting on Pulp is an in-test one, called
`setup_collections`. When it works, the task is complete in around 70
seconds but when it doesn't, it just freezes the whole play.

This patch attempts to make it fail faster by putting a reasonable
timeout value of 2 minutes.
2 years ago
Matt Clay 38db9cf417
ansible-test - Update test containers. (#78615)
The test containers no longer use the `VOLUME` instruction.
2 years ago
Matt Clay 4187707f03
ansible-test - Use --tmpfs to run containers. (#78605) 2 years ago
Matt Clay 2e536c0afb
Fix file integration test chattr/lsattr check. (#78614)
On BusyBox systems such as Alpine, chattr on a tmpfs fails with a status of 0 and output only on stderr.

This change updates the test to not assume output on stdout.
2 years ago
Matt Clay ece85abbc4
ansible-test - Verify executables are executable. (#78606) 2 years ago
Sviatoslav Sydorenko beb70daf14 Fix pytest collection during test_host partial run
Sometimes pytest errors out with an `ImportError` during its tests
collection stage when a Python package/directory containing the test
module does not have an `__init__.py` in it. This is being observed
under Python 3.9 and higher.

The patch provides a workaround for this problem but does not address
the root cause which is currently unknown.

Ref:
https://github.com/ansible/ansible/pull/78585#issuecomment-1220885431
2 years ago
Martin Krizek 42d8a9daa8
Prevent double failing hosts for includes in loops (#76928)
Fixes #23161
2 years ago
Matt Martz deb54e4c5b
Relax minimal config to enable manifest functionality (#78574)
* Revert "Fix incorrect docs about how to enable manifest functionality (#78572)"

This reverts commit ac1ca40fb3.
2 years ago
Matt Martz d2f8099118
Use MANIFEST.in style directives to build collections (#78422) 2 years ago
Elijah DeLee f9a450551d
add required file to collection skeleton (#77418)
* add required file to collection skeleton

This file is required to be able to upload a collection.

It is present in
https://github.com/ansible-collections/collection_template/blob/main/meta/runtime.yml
but that does not get used by default.

Without this, if you use the "ansible-galaxy collection init" command
and you try and publish that collection without adding this file, you
get the error:

"ERROR! Galaxy import process failed: 'requires_ansible' in
meta/runtime.yml is mandatory, but no meta/runtime.yml found (Code:
UNKNOWN)"

Also updates relevant test and adds a changelog fragment
2 years ago
Kate Case bf1ef5a1f3
Replace get_persistent_connection_options in task_executor with get_options (#74446)
Replace get_persistent_connection_options with get_options
Remove special case for network sub_plugin in _set_plugin_options
Try to avoid mock connection pretending to be persistent
Rename variables->options to reflect what they actually are
Gather options for ssh_type_conn on network_cli
Drop reliance on sub_plugin["type"]
2 years ago
Martin Krizek 811093f022
Move handler processing into new PlayIterator state (#77955)
Fixes #46447
Fixes #52561
Fixes #54991
Fixes #64611
Fixes #64999
Fixes #65067
Fixes #72725
Fixes #72781
Fixes #77616
2 years ago
Matt Clay d683c5bd21
ansible-test - Fix Ubuntu 20.04 bootstrapping. (#78551) 2 years ago
Matt Clay 8e96c27888
ansible-test - Update base and default containers. (#78532) 2 years ago
Matt Clay 69ceb31600
ansible-test - Update sanity test requirements. (#78528) 2 years ago
Matt Clay 135f95fb2f
Split up ansible-test integration tests. (#78520)
* Relocate venv-pythons.py script.

* Split out unsupported-directory test.

* Split out sanity-import test.

* Split out sanity-validate-modules test.

* Split out units test.

* Split out integration test.

* Split out units-constraints test.

* Split out integration-constraints test.

* Split out coverage test.

* Split out sanity test.

* Split out git test.

* Update test groups.
2 years ago
Brian Coca 0de4480467
allow multiple types for null representation (#78451)
* allow multiple types for null representation

  '' is not Nonetype
  fixes #76493
2 years ago
Matt Clay 99016653a2
ansible-test - Fix self-test change handling. (#78521) 2 years ago
Matt Martz 86298b7a47
Fix --role-file arg detection (#78475)
* Fix --role-file arg detection. Fixes #78204

* Do not traceback, give better error

* Add coverage for compound shortopts to match -r. Fixes #78491
2 years ago
Sloane Hertel 3add5e9321
Fix apt making changes in check mode (#78496)
* Don't actually update the cache in check mode

Add tests for updating the cache in check mode

* Don't mark packages as manually installed in check mode and add a test for it

* changelog

* Update test/integration/targets/apt/tasks/apt.yml

* fix test
2 years ago
Matt Clay 2b63fdd1b8
ansible-test - Fix ansible-doc sanity test FQCN. (#78518) 2 years ago
Matt Clay 2fedde6356
ansible-test - Update distro test containers. (#78506) 2 years ago
Matt Clay 8367af9cdd
ansible-test - No locale warning with delegation. (#78505)
The locale warning for the origin host is not relevant when delegation is used.
It is also not relevant when non-test commands are in use.
2 years ago
Matt Clay 47804fcdf5
ansible-test - Update coverage to 6.4.3. (#78504) 2 years ago
Matt Clay 9ee183c4b9
Clean up unit tests. (#78498) 2 years ago
Matt Clay 89862fda3b
ansible-test - Sanity test code cleanup. (#78497) 2 years ago
Matthew Donoughe 3f1838bf91
Update attributes of files that are links without specifying link target (#76167)
* update attributes of files that are symlinks

* update attributes of files that are hard links

* fix default state in documentation

* remove unnecessary suppression

* add to changelog
2 years ago
Matt Clay 650befed37
Add missing space after keywords. (#78480)
Also remove unnecessary parenthesis.
2 years ago
Patrick Hemmer 04e8927579
apt: include apt preferences (e.g. pinning) when selecting packages (#78327)
Fixes #77969
2 years ago
Matt Clay 85acf4d1e5
ansible-test - Avoid use of deprecated type hints. (#78456)
* ansible-test - Avoid use of deprecated type hints.

PEP 585 deprecated many container types in the `typing` module in favor of the actual types, which support subscripting as of Python 3.9.

Conversion of `t.Type` was skipped since PyCharm does not currently recognize it.

* ansible-test - Fix `t` and `c` imports/shadowing.
2 years ago
Matt Clay 5bee66fc5d
ansible-test - More type hint updates. (#78455)
* Simple regex replace of multi-line function arg annotations on the first line.

* Manually fix up ArgumentParser type annotations.

* Manual type hint conversions.

* Manual conversion of function type hints.

* Remove unnecessary type hints on for statements.
2 years ago
Matt Clay b993b5cd49
ansible-test - Convert more type hints. (#78449)
* Simple regex replace of multi-line function arg annotations.

* Simple regex replace of multi-line function arg annotations with default values.

* Simple regex replace of multi-line function arg return annotations.

* Simple regex replace of assignment annotations.
2 years ago
Matt Clay 02e98810a9
Re-organize CI test groups for POSIX tests. (#77420)
* `context/target` tests must be in groups 1 - 2.
* `context/controller` tests must be in groups 3 - 5.

This makes it easier to efficiently organize groups and balance test runtimes.
2 years ago
Matt Clay 3eb0485dd9
ansible-test - Use more native type hints. (#78435)
* ansible-test - Use more native type hints.

Simple search and replace to switch from comments to native type hints for return types of functions with no arguments.

* ansible-test - Use more native type hints.

Conversion of simple single-line function annotation type comments to native type hints.

* ansible-test - Use more native type hints.

Conversion of single-line function annotation type comments with default values to native type hints.

* ansible-test - Use more native type hints.

Manual conversion of type annotation comments for functions which have pylint directives.
2 years ago
Matt Clay f2abfc4b3d
ansible-test - Parse content config only once. (#78418) 2 years ago
Jordan Borean ad79c1e0d0
Add support for json in adhoc -a (#78114)
Co-authored-by: Sandra McCann <samccann@redhat.com>
2 years ago
Matt Clay d8fefba20e
ansible-test - Update locale logic to match core. (#78389)
Now that core requires UTF-8 filesystem encoding, ansible-test does as well.

Additionally, the `en_US.UTF-8` or `C.UTF-8` encoding must be available.
Previously the `en_US.UTF-8` encoding was requested, but its availability was never verified.
The fallback to `C.UTF-8` maintains UTF-8 encoding while allowing more flexibility in the running environment.
2 years ago
Sloane Hertel 05608b20e8
Fix listing collections that are missing the metadata required by build (#76596)
* Rethread pr/70185 through the dependency resolver

Hang optional metadata toggle on the ConcreteArtifactsManager instead of threading it through whole list codepath

Don't error while listing collections if a collection's metadata is missing keys required for building a collection.

Give an informative warning if metadata has been badly formatted.

Co-authored-by: Sam Doran <sdoran@redhat.com>
2 years ago
Brian Coca 9b79d6ba35
config error origin (#78407) 2 years ago
Sloane Hertel a2174fc6d9
Fix removing existing dir/files with `ansible-galaxy collection init --force` (#78403)
* Remove collection contents when re-initializing with --force


Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
2 years ago
Martin Krizek ce6c9befb8
Do not allow handlers from dynamic includes to be notified (#78399)
* Do not allow handlers from dynamic includes to be notified
2 years ago
Matt Martz f6d2b18322
Correctly get the filename from a url in fetch_file. Fixes #29680 (#78383) 2 years ago
Matt Clay b0b58fa7c2
ansible-test - Update base and default containers. (#78377) 2 years ago
Matt Clay 234938cd76
Change the language used in the unarchive test. (#78382)
The current language pack fails to install on Ubuntu 22.04.
However, since the langauge only needs to be non-English, changing it preserves the test functionality.
2 years ago
Matt Clay cc5c706681
Update unit test controller requirements. (#78376) 2 years ago
Matt Clay f5c01a8b6e
Remove unit test Python 2.6 compat. (#78369) 2 years ago
Matt Clay b86a18bd27
Remove unit test pytz requirement. (#78368) 2 years ago
Matt Davis 1368bfa348
PluginLoader now installs module-to-be-imported in sys.modules before exec (as Python import does). (#78364) 2 years ago
Ruediger Pluem 0ef1376966
Make distro.id() report newer versions of OpenSuSE (at least >=15) also report as opensuse (#76121)
* Report OpenSuSE >= 15 as opensuse

Make distro.id() report newer versions of OpenSuSE (at least >=15) also report
as opensuse. They report themselves as opensuse-leap.

* Add a test
2 years ago
Matt Clay a3c90dd0bc
ansible-test - Fix TTY and output handling. (#78350) 2 years ago
Sloane Hertel 5728d72cda
Fix KeyError for ansible-galaxy when caching paginated responses from v3 (#78325)
* Fix KeyError for ansible-galaxy when caching paginated responses from v3

* changelog

* generate responses in loop for test

Co-authored-by: Matt Martz <matt@sivel.net>
2 years ago
Matt Martz 1429672213
Add pep440 version_type for version test (#78308) 2 years ago
Matt Martz 9d4ced1237
wait_for - Read file and perform comparisons using bytes to avoid decode errors (#78317)
* wait_for - Read file and perform comparisons using bytes to avoid decode errors. Fixes #78214

* Write non-ascii via script instead of static file

* Use contexlib.closing to support py27 context manager

* Use executable from task, instead of shebang

* Update encoded bytes to utf16
2 years ago
Matt Clay 0012263c7a
ansible-test - Fix handling of args after `--` (#78328) 2 years ago
Matt Clay 61af59c808
ansible-test - Update distro containers to 4.4.0. (#78322) 2 years ago
Matt Clay 4cd3425dfe
ansible-test - Update coverage to 6.4.2 (#78320) 2 years ago
Martin Krizek b56b38b3ba
Use classproperty to access field attributes of a class (#78180)
* Use classproperty to access field attributes of a class, consolidate into one class
2 years ago
Matt Clay 90ef914f77
ansible-test - Drop Python 3.8 controller support. (#78237) 2 years ago
Matt Clay 681c32ec65
Fix fetch integration test on macOS. (#78314)
The test can now run split.
2 years ago
Matt Clay 61ea6c4a5b
Fix expect integration test. (#78315)
The test can now run split on macOS.
2 years ago
Matt Clay 27d39be580
Fix become integration test. (#78313)
* Don't set become_flags in become integration test.

Setting the flags in the test overrides the default value in the become plugins, which was not intentional.

* Remove hack for broken implicit tilde support.

It was only broken because the tests were overriding the default become flags for sudo.

* Simplify assertions.
2 years ago
Sviatoslav Sydorenko 619a1fa7e2
Drop custom repo from libuser install @ Alpine CI (#78305)
`group` and `setup_cron` integration tests currently install `libuser` and `faketime`
packages respectively, from the non-default edge/testing repositories. It is no
longer necessary to do so for the former so this patch drops the repo option. As
for the latter, this patch includes a note explaining when the same could be done
for it too.

This patch also adds a note on installing `faketime` on Alpine in CI.

Co-Authored-By: Matt Clay <matt@mystile.com>
2 years ago
Matt Clay de2163b160
Overhaul become integration tests. (#78312) 2 years ago
Matt Clay f3f4ad9332
ansible-test - Use sudo for all remotes. (#78310) 2 years ago
Matt Clay 8565deaae3
ansible-test - Install `acl` on Alpine remotes. (#78303) 2 years ago
Matt Clay 8bb20fe06b
ansible-test - Enable ACL support on more remotes. (#78299) 2 years ago
Matt Clay 9b42f9befe
Revert "Re-enable strategy units (#78293)" (#78300)
This reverts commit c6c9d90ca4.
2 years ago
Matt Martz c6c9d90ca4
Re-enable strategy units (#78293) 2 years ago
Matt Clay ee4588abca
Split integration test fixes. (#78281)
* Add setup_test_user integration target.

This integration target creates a temporary test user and removes
it when the current integration target finishes. This user is
configured with a password so it can be used with become, even
when the current user is unprivileged.

* Fix slurp integration test.

* Fix unarchive integration test.

* Fix module_utils integration test.
2 years ago
Matt Clay b63812bc08 ansible-test - Enable ACLs on FreeBSD remotes.
This allows integration tests to switch from one unprivileged user
to another, sharing files between the users using ACLs.
2 years ago
Martin Krizek d070b03ad8
Fix templating nested vars with convert_data=False (#78273)
Regression introduced in #78259.
2 years ago
Martin Krizek 9afdb7fec1
template module/lookup: fix convert_data for macros (#78259)
Fixes #78141
2 years ago
James Livulpi d635b871d1
Cleanup write_file method in uri (#76412) 2 years ago
Matt Martz d58e69c82d
Add support for gzip decoding responses (#41925) 2 years ago
zorun a6e671db25
Fix AIX processor facts and add unit test (#78223)
- `processor_count` was erroneously set to the number of cores
- `processor_cores` was erroneously set to the number of threads per core
- `processor_vcpus` and `processor_threads_per_core` were not set
- `processor` was a string, while it's supposed to be a list

Before:

```
"ansible_processor": "PowerPC_POWER7",
"ansible_processor_cores": 4,
"ansible_processor_count": 12,
```

After:

```
"ansible_processor": [
    "PowerPC_POWER7"
],
"ansible_processor_cores": 12,
"ansible_processor_count": 1,
"ansible_processor_threads_per_core": 4,
"ansible_processor_vcpus": 48,
```

Also add a unit test.

Co-authored-by: Baptiste Jonglez <git@bitsofnetworks.org>
2 years ago
Matt Clay b85bcd7abf
Fix module_utils_common.respawn test. (#78248)
The respawn failure can be returned on either stdout or stderr.
2 years ago
Kay Yan 2e4b0fefbf
Added Kylin Linux Advanced Server Support (#78232) 2 years ago
Jordan Borean 75de4a37bc
validate-modules - allow raw module return type (#78231) 2 years ago
Martin Krizek 2bc2153c01
yum: fix releasever for latest (#78066)
Fixes #78058
2 years ago
Matt Martz b1dd2af4ca
Preflight utf8 encodings check (#78175) 2 years ago
Brian Coca d26801e994
Fix wording for :ref|term: substitution (#78216)
* Fix workding for :ref|term: subsitution

 now matches 'seealso' and makes more sense:

  ```
  delay:
  applies_to:
  - Task
  description: Number of seconds to delay between retries. This setting is only used
    in combination with `until`.
  ```
 vs
  ```
  delay:
  applies_to:
  - Task
  description: Number of seconds to delay between retries. This setting is only used
    in combination with website for `until`.
  ```

* updated unit tests

* match see also

* more sanity
2 years ago
Martin Krizek 17d52c8d64
Move undefined check from concat to finalize (#78165)
* Move undefined check from concat to finalize

In the classic Jinja2's Environment str() is called on the return value of the
finalize method to potentially trigger the undefined error. That is not
the case in NativeEnvironment where string conversion of the return value is
not desired. We workaround that by checking for Undefined in all of our concat
functions. It seems simpler to do it earlier in the finalize method(s) instead.
As a side-effect it fixes an undefined variable detection in imported templates.

Fixes #78156

ci_complete

* Fix sanity

* ...

* sigh
2 years ago
Sloane Hertel 1562672bd1
Make unit test for missing git executable more generic (#78173)
* Make unit test for missing git executable more generic

* use MagicMock side_effect to raise exception instead
2 years ago
LRitzdorf 6bcb494f83
[dnf] Fix skip_broken, add test coverage (v2) (#78158)
* [dnf] Fix skip_broken, add test coverage

Change:
- skip_broken was set in config but not actually used in calls to
  base.install()
- added a lot of test cases with specialized repo
- got rid of external (docker repo) nobest test cases since the
  specialized repo works well for those too
- Slight cleanup and adding comments in dnf module

Test Plan:
- ci_complete

Tickets:
- Fixes #73072

Original-author: Rick Elrod <rick@elrod.me>

* Use a better test for checking results list

ci_complete

Signed-off-by: Rick Elrod <rick@elrod.me>

Co-authored-by: Rick Elrod <rick@elrod.me>
2 years ago
Matt Clay f70cc2fb7e
Refactor ansible-test integration tests. (#78168)
* Relocate update-ignore.py for easier re-use.

* Add script to ease collection testing.

* Skip ignore rewrite if file does not exist.

* Add integration test for the shebang sanity test.

* Fix ansible-test-no-tty integration test.

Previously the test only verified a TTY was not used if a TTY already existed.
This prevented the test from verifying behavior when run in CI.
Now the test creates a PTY before invoking ansible-test.

* Clean up ansible-test-docker integration test.
2 years ago
Matt Martz bcdc2e167a
Support for Python 3.11+ tomllib for inventory (#77435) 2 years ago
Martin Krizek 43153c5831
`FieldAttribute`s as descriptors (#73908) 2 years ago
Matt Clay 4594c0c609 Fix HTTP tests on FreeBSD when running split. 2 years ago
Matt Clay b9d13d222c ansible-test - Improve pip bootstrap download. 2 years ago
Brian Coca fa35aa4865
Galaxy server config updates (#77106)
* ansible-galaxy configurable timeouts

  - also fixed issues with precedence,
    so --ignore-certs now overrides config
  - made galaxy_timeout generic setting,
    if set, it becomes default for server configs,
    but now specific servers can override
  - updated tests or added notes (some tests ignore/override precedence)

Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
2 years ago
Sviatoslav Sydorenko e481b35e23
Delete core-ci `query` param from ansible-test (#78136)
Ref: https://github.com/ansible/ansible-core-ci/pull/343
2 years ago
Matt Clay ca04dc28a2 ansible-test - Remove deprecated test platforms. 2 years ago
Brian Coca cea18bf60a
password lookup argument parsing fix (#78080)
fixes #78079
2 years ago
Matt Clay cf9fd2d7da ansible-test - Use shlex.join where appropriate. 2 years ago
Matt Clay 24d91f552c ansible-test - Add support for more remotes. 2 years ago
Matt Clay 5666c6d6a3 ansible-test - More flexible become support. 2 years ago
Chris James ea351f0ae2
user - Use -n instead of -N for luseradd on all distros (#75042)
* Use -n instead of -N for luseradd on all distros

Co-authored-by: Chris James <git@etcet.net>
2 years ago
Don Naro 3cd2c494bd
issue #72449 winrm script rm (#77931) 2 years ago
Matt Martz 18992b7947
Add new `loop_control.extended_allitems` option (#75760)
* Add new `loop_control.extended_allitems` option. Fixes #75216

* Add test for extended_allitems

* docs code block fix
2 years ago
Matt Martz a90f666ab3
Use ctypes to access shadow, instead of spwd which is deprecated (#78050)
* Use ctypes to access shadow, instead of spwd which is deprecated

* Remove sanity ignore

* cast to str to satisfy mypy

* Add typing.cast polyfill

* Appease mypy redefine

* Add clog frag
2 years ago
David Moreau Simard d81a519f25
Remove .travis.yml files from galaxy init skeletons (#75740)
* Remove .travis.yml files from galaxy init skeletons

These files have existed for many years but in light of the recent
circumstances around Travis, we should not include them by default in
the galaxy init skeletons.

* Add breaking_changes changelog fragment with recommendation to use a custom skeleton if the .travis.yml file is still needed.

Co-authored-by: s-hertel <19572925+s-hertel@users.noreply.github.com>
2 years ago
Matt Clay 95df5cb740 Fix invalid-ansiblemodule-schema sanity errors.
The validate-modules sanity test expects a list not a dict_keys iterable.
2 years ago
Matt Clay 2f0530396b ansible-test - Clean up type hints and comments. 3 years ago
Matt Clay 5f5c4ef2ef Remove incidental_inventory_foreman test. 3 years ago
Matt Clay a4e4413cde ansible-test - Update distro containers to 4.3.0.
The primary change in this update is removal of `/etc/ansible/hosts` from the containers.
3 years ago
Matt Clay 5b3557f8ba ansible-test - Allow docstring in docs-only module 3 years ago
Matt Clay c2b36ba04d ansible-test - Update base and default containers. 3 years ago
Matt Clay 23914d3f0b
ansible-test - Adjust unit test mock usage. (#77961) 3 years ago
Sloane Hertel e059dde9fb
Add missing inventory cache test (#77990)
Fix intentional coverage to replace incidental_inventory_foreman

ci_complete
ci_coverage
3 years ago
Sloane Hertel 9d6cc7b576
unarchive - fall back to unzip -Z if zipinfo is not available (#76971)
Add a new handler class ZipZArchive to use unzip -Z as an alternative to zipinfo

Run 'unzip -Z' in can_handle_archive so we fall back to the next handler if it's not available (failing in is_unarchived is too late)

* Add a test for unzip -Z when zipinfo is not available

* Update test for missing binary altogether by removing /usr/bin from the PATH
3 years ago
Sloane Hertel a43112290a
fix ansible-galaxy-collection test with revoked key (#77989)
* Remove unnecessary gpg passphrase
3 years ago
Sloane Hertel e89176caac
Pin PyYAML version compatible with Python 3.8+ (#77936)
Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
Co-authored-by: Matt Clay <matt@mystile.com>
3 years ago
Matt Martz 5e369604e1
Forked display via queue (#77056)
* Forked Display via queue

* Docs and simple code cleanup

* Only proxy Display.display

* Remove unused import

* comment

* Update deadlock comment, remove py3 check

* Don't flush display, and don't lock from forks

* clog frag

* ci_complete ci_coverage

* Add units for queue proxying

* Cleanup flush

* ci_complete

* Only lock the write, switch to RLock

* Remove unused import
3 years ago
Sloane Hertel 143e7fb45e
ansible-galaxy - support resolvelib versions >= 0.5.3, < 0.9.0 (#77649)
* ansible-galaxy - support resolvelib versions >= 0.5.3, <= 0.8.1

Test incompatibilities are removed for resolvelib >= 0.6.0

Test against the latest 0.8.x version and fix requirements

* Fix tests - use a venv for testing the range of resolvelib versions

* Update temporary hardcoded fallback for ansible-test

* Update hardcoded upperbound for sanity tests

* Make error check more flexible
3 years ago
Matt Clay df765c0ecf ansible-test - Update Alpine, openSUSE containers 3 years ago
Matt Clay fe2d8b7066 Skip libuser tests on openSUSE 15.4+
The libuser package is not available.
3 years ago
Matt Clay e31e918e04 Don't install lgroupadd for SUSE tests.
It's not used by the tests and is not available as of openSUSE 15.4.
3 years ago
Brian Coca 89c6547892
preserve add_host/group_by on refresh (#77944)
* preserve add_host/group_by on meta: refresh_inventory

Co-authored-by: Jordan Borean <jborean93@gmail.com>
3 years ago
Matt Clay 52c8613a04 ansible-test - Skip mypy runs under Python 3.11. 3 years ago
Matt Clay 11b6032e73 ansible-test - Ignore vendored code in mypy test. 3 years ago
Sloane Hertel 6c6a7851ed
Add inventory cache integration tests (#77895)
* Add intentional coverage for incidental_inventory_foreman coverage cache tests

ci_complete
ci_coverage
3 years ago
Matt Clay 5391150b30 Integration tests for invalid group name. 3 years ago
Sloane Hertel 1706d35fc4
ansible-galaxy-collection - use --pinentry-mode loopback for gpg key revocation (#77967) 3 years ago
Sloane Hertel c908d782fb
get_url - permit to have a checksum only file (#77948)
checksum can also accept a checksum only file (no filename beside the checksum).
fixes #54390

Co-authored-by: Baptiste Mille-Mathias <baptiste.millemathias@gmail.com>
3 years ago
Sloane Hertel f270b4e224
Fix items2dict errors for incorrect list item type and missing keys (#77946)
* Fix items2dict errors for incorrect list item type and missing keys

* keep dict comprehension
3 years ago
Martin Krizek 058a69e6b0
Properly send a skipped message for empty templated loop (#77945)
Fixes #77934
3 years ago
Matt Clay 81351022d8 ansible-test - Prefer unittest.mock for core. 3 years ago
Matt Clay f933314101 ansible-test - Fix shell target options handling. 3 years ago
Matt Clay 86779cc903 ansible-test - Code cleanup.
This helps prepare for a future pylint upgrade.
3 years ago
Sloane Hertel e6075109d0
Remove incidental_inventory_aws_ec2 and supporting plugins (#77877)
* Remove aws/2.7 and aws/3.6 from incidental test matrix entries

* Remove aws.sh symlink
3 years ago
Matt Clay bd849b3076 Fix ansible-runner integration tests. 3 years ago
Sandra McCann f3c684564c
Remove platform dev guides from core docs (#77915) 3 years ago
Julien Lecomte 33beeace10
Fix lusermod using a group name and not id (#61965) (#77914)
* Fix lusermod using a group name and not id (#61965)
3 years ago
Matt Clay e259317c3e
Remove incidental_inventory_docker_swarm tests. (#77924)
ci_complete
ci_coverage
3 years ago
Sloane Hertel 1214b63f4f
Fix 'ansible-config dump --only-changed -t all' verbosity (#77898)
* Fix 'ansible-config dump --only-changed -t all' to only display headers if plugin options are changed

* changelog

* add a test
3 years ago
Matt Clay 43d650f924
Add RHEL 9.0 to CI. (#77853)
* Add RHEL 9.0 to CI.

* Restrict network manager inspection to RHEL8

* Skip module tests when astream_name is undefined, undefine it for RHEL9 until 9.1

* Remove redundant test.

Co-authored-by: Matt Martz <matt@sivel.net>
3 years ago
Wouter Schoot 711b51fad6
Remove double spaces in errormessage (#77839) 3 years ago
Matt Clay 289cba333b ansible-test - More multi-arch containers. 3 years ago
OscarBell 74eb8b779d
Blockinfile multiline search (#75090) 3 years ago
Matt Martz e7e1d592a6
Incidental coverage 1 (#77846)
* Add find test to assert patterns with regex

* Add test to assert that nested includes with relative path work

* Add test to assert symmetric_difference supports unhashable types

* ci_complete ci_coverage

* Add test to assert JSON values are converted to Python

* Add test to assert how unexpected module failures behave

* Add test to verify remote_is_local

* ci_complete ci_coverage

* Get real path of local tmp

* ci_complete ci_coverage

* Expanded symmetric_difference tests

* ci_complete ci_coverage

* I don't even know

* ci_complete ci_coverage

* Revert mathstuff changes

* ci_complete ci_coverage
3 years ago
Brian Coca f7d7604454
Add uri, urn and url test plugins (#77423)
and docs!

Co-authored-by: Tabah Baridule <dulemartins07@gmail.com>
Co-authored-by: Felix Fontein <felix@fontein.de>
3 years ago
Ian Wienand e4c0bbf885
apt: fix virtual package install version detection (#76781)
* apt: fix virtual package install version detection

Change 4a62c4e3e4 introduced version
matching in installation.

The problem stems from

 if version_installable or version:
     pkg_list.append("'%s=%s'" % (name, version_installable or version))

When the package is a virtual-package, package_status() is returning
the "version_installable" of the package *satisfying* the
virtual-package; but then this is trying to install the
virtual-package with this version pin.

For example, "yaml-mode" is a virtual package satisifed by
"elpa-yaml-mode" (currently 0.0.14-1) and trying to install it fails
with

 $ usr/bin/apt-get -y ... install 'yaml-mode=0.0.14-1'
 ... failed: E: Version '0.0.14-1' for 'yaml-mode' was not found ...

In the case of a virtual-package with nothing installed to satisfy it,
we should just return blank values to allow apt-get to do it's thing.

The tests are updated to install and remove this package.

Fixes: #76779
3 years ago
Sloane Hertel 82f3a57bee
Fix ansible-galaxy traceback when unexpected version of resolvelib is installed (#77630)
* Fix traceback when a supported version of resolvelib is not installed

Try to read the supported version range from the package distribution info and fall back to a hardcoded lowerbound/upperbound (>=0.5.3,<0.6.0).

* Add tests for unsupported resolvelib versions

* Resolve remaining import sanity test issues.

Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Martz <matt@sivel.net>
3 years ago
Sloane Hertel 621e782ed0
Add toggle to fix module_defaults with module-as-redirected-action on a per-module basis (#77265)
* If there is a platform specific handler, prefer the resolved module over the resolved action when loading module_defaults

Add a toggle for action plugins to prefer the resolved module when loading module_defaults

Allow moving away from modules intercepted as actions pattern

Fixes #77059
3 years ago
Matt Clay 2595b42aca
Prefer unittest.mock over mock. (#77886) 3 years ago
Matt Clay 3a9a23fb1a
Use unittest.mock instead of mock. (#77883)
This can only be done for controller tests.
3 years ago
Sloane Hertel 1eb7f71f0e
Replace incidental_inventory_aws_ec2 with intentional coverage (#77743)
* Add intentional test coverage for lib/ansible/plugins/inventory/__init__.py

* Add intentional (integration) test coverage for lib/ansible/module_utils/common/network.py

* Add ansible_release info test from https://github.com/ansible/ansible/pull/74673

ci_complete
ci_coverage

Co-authored-by: Rick Elrod <rick@elrod.me>
3 years ago
Sloane Hertel 400475acc0
validate meta/main.yml dependencies and meta/requirements.yml are both lists before concatenating them together (#77821) 3 years ago
Matt Clay 13d03c3c22
ansible-test - Add RHEL 9 / FreeBSD 13.1 remotes. (#77851) 3 years ago
Matt Clay 3d2c2e60aa
ansible-test - Update distro containers. (#77847)
* ansible-test - Add a Ubuntu 22.04 container.
* ansible-test - Add a Fedora 36 container.
* ansible-test - Update distro containers.
* Fix dnf test on Fedora 36.
* Work around scp test issues.
3 years ago
Matt Martz ef3ea4cdc3
Incidental docker chip away (#77830)
* Add new templating target, test splitter

* Add test for host:port parsing in parse_address via add_host

* Test already notified listening handler

* Add test for gathering bridge network facts

* Add veth to bridge

* ci_complete ci_coverage
3 years ago
Matt Clay 76ead1e768 ansible-test - Fix remote args restriction.
The platform-specific and global fallbacks were not working with the `--remote` option.

This regression was introduced by https://github.com/ansible/ansible/pull/77711
3 years ago
Matt Clay 185340e667 ansible-test - Update default containers.
These containers include Python 3.11 sanity test environments.
3 years ago
Matt Clay dfde4be444 Add Python 3.11 support.
ci_complete
ci_coverage
3 years ago
Matt Clay 8ca28acd0d Fix unit tests for Python 3.11. 3 years ago
Matt Clay 5f74350fd5 Avoid deprecated method in unit tests. 3 years ago
Matt Clay 9600e3064a
ansible-test - Improve requirements handling. (#77825)
The requirements for virtualenv and coverage are now kept in a requirements file for easier container builds.

The test-constraints sanity test has been updated to make sure the requirements file is kept up-to-date.
3 years ago
Sloane Hertel fecc78b5fb
remove extra validate-modules ignore entry (#77826) 3 years ago
Brian Coca 4706091a38
rename systemd module to services only (#77644)
* rename systemd module to services only

 disambiguates what it handles since systemd is now much more
 that a service manager, but the module is specific to services
3 years ago
Joseph Torcasso c9ce7d08a2
template/__init__.py - fix KeyError when wantlist=False with non-list 3 years ago
Brian Coca 5e50284693
Restrict role loading .. to role ... (#77683)
* Ansible will now error out if you try to use the `*_from` to load files from outside the role being loaded by `{import,include}_role`
3 years ago
Matt Clay 290fb5e0ca
ansible-test - Upgrade to coverage 6.3.3. (#77817) 3 years ago
Martin Krizek e9af6efee6
Raise a proper error when include/import_role is used as a handler (#77807) 3 years ago
Matt Clay 1e3cf9f732
ansible-test - Make distro containers multi-arch. (#77814) 3 years ago
Abhijeet Kasurde 524d30b8b0
distribution: add distribution_minor_version for Debian Distro (#74721)
* Added a fact about minor version details for Debian Distro

Fixes: #74481

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
3 years ago
Matt Clay 449d82a521 ansible-test - Add RHEL 8.6 support. 3 years ago
Matt Clay fe349a1ccd
ansible-test - Enhance the shell command. (#77734)
* ansible-test - Add shell --export option.

* ansible-test - Support cmd args for shell command.

Also allow shell to be used without a valid layout if no delegation is required.

* ansible-test - Improve stderr/stdout consistency.

By default all output goes to stdout only, with the exception of a fatal error.

When using any of the following, all output defaults to stderr instead:

* sanity with the `--lint` option -- sanity messages to stdout
* coverage analyze -- output to stdout if the output file is `/dev/stdout`
* shell -- shell output to stdout

This fixes issues two main issues:

* Unpredictable output order when using both info and error/warning messages.
* Mixing of lint/command/shell output with bootstrapping messages on stdout.

* ansible-test - Add changelog fragment.
3 years ago
s-hertel fbc5b3f9c5 fix copy test for split controller/remote 3 years ago
s-hertel 3f4cacd355 Remove MacOSX-specific cleanup for unarchive test 3 years ago
Matt Clay cc5ac88c4c Update mypy test. Fix type hints for Python 3.9.
This also bumps the minimum controller version (for mypy only) to Python 3.9.
3 years ago
Matt Clay 2cc74b04c4 ansible-test - Add multi-arch remote support. 3 years ago
Matt Clay de5d6820f8 ansible-test - Remove obsolete code. 3 years ago
Matt Clay a115a7fce2 Install acl on Ubuntu for copy integration test. 3 years ago
Matt Clay e2200e8dfc ansible-test - Fix remote completion validation. 3 years ago
Matt Davis 4ff10fcd31 fix SELinux facts tests
* make the pre-flight sniffing more robust to different failure conditions (was failing on Ubuntu 22.04 VMs)
* remove skip aliases (the test needs to function everywhere to assert that the selinux facts bits behave properly when it's N/A)
3 years ago
Matt Clay 6513453310 ansible-test - Add support for remote Ubuntu VMs. 3 years ago
Matt Clay dcda1e7239 Trim message in split integration test. 3 years ago
Martin Krizek dbdbfe845a
Remove deprecated subclassing DefaultCallback without doc_fragment (#77677) 3 years ago
Brian Coca a3cc6a581e
dont rely on path to set config defs for plugins (#77659) 3 years ago
Martin Krizek ed30fc9a06
Remove deprecated loading cache plugins directly without cache_loader (#77655) 3 years ago
Martin Krizek 572bc1354a Fix lazy eval version in integration tests 3 years ago
Brian Coca 749a105956
ansible-doc stay 3.8 python compatible (#77682)
pathlib with_stem will wait till 3.9
  also add tests by listing filters
3 years ago
Matt Clay cc872a57f2 Reduce output from ansible-doc integration test.
This removes about 20K lines of output which are not evaluated as part of the test.
3 years ago
Matt Clay 5c2d830dea
ansible-test - Fix subprocess management. (#77641)
* Run code-smell sanity tests in UTF-8 Mode.
* Update subprocess use in sanity test programs.
* Use raw_command instead of run_command with always=True set.
* Add more capture=True usage.
* Don't expose stdin to subprocesses.
* Capture more output. Warn on retry.
* Add more captures.
* Capture coverage cli output.
* Capture windows and network host checks.
* Be explicit about interactive usage.
* Use a shell for non-captured, non-interactive subprocesses.
* Add integration test to assert no TTY.
* Add unit test to assert no TTY.
* Require blocking stdin/stdout/stderr.
* Use subprocess.run in ansible-core sanity tests.
* Remove unused arg.
* Be explicit with subprocess.run check=False.
* Add changelog.
* Use a Python subprocess instead of a shell.
* Use InternalError instead of Exception.
* Require capture argument.
* Check for invalid raw_command arguments.
* Removed pointless communicate=True usage.
* Relocate stdout w/o capture check.
* Use threads instead of a subprocess for IO.
3 years ago
Brian Coca b439e41a91
expand ansible-doc coverage (#74963)
* Expand ansible-doc to tests/filters and fix existing issues

  enable filter/test docs if in single file or companion yaml
  add docs for several filters/tests plugins
  allow .yml companion for docs for other plugins, must be colocated
  verify plugins are valid (not modules, cannot)
  fix 'per collection' filtering
  limit old style deprecation (_ prefix) to builtin/legacy
  start move to pathlib for saner path handling
  moved some funcitons, kept backwards compat shims with deprecation notice

  Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
  Co-authored-by: Felix Fontein <felix@fontein.de>
  Co-authored-by: Sandra McCann <samccann@redhat.com>
3 years ago
Martin Krizek 3980eb8c09
Prevent losing unsafe from lookups (#77609)
* Prevent losing unsafe from lookups

This patch fixes a bug which under certain conditions results in data
returned from lookups not being marked as unsafe.

Each time Templar.do_template is invoked a new AnsibleContext is
created and stored effectively at two places:
1) as an instance variable in templar_obj.cur_context
2) as a local variable called new_context in do_template method of Templar

Due to custom functionality in Ansible's Context that allows for nested
templating it is possible that during resolving variable's value
template/do_template method is called recursively again, again creating
a new context. At that point the problem manifests itself because as
mentioned in 1) above the context is overwriten on the templar object
which means that any subsequent calls to _lookup will use the new
context to mark it as unsafe which is now different to the local
new_context which is used for testing for unsafe property.

The solution to the problem appears to be to restore the original
context inside do_template and also to eliminate the local variable
new_context to prevent problems in the future.

It appears that we don't have a better way of storing the context other
than as some form of global variable and so this appears to be the
"best" solution possible at this point. Hopefully data tagging will be
the solution here.

For more examples see unit and integration tests included in this patch.

Fixes #77535
3 years ago
Matt Clay 26fd5a8c3a Revert "ansible-test - Fix subprocess management. (#77638)"
This reverts commit 62d03c8e75.
3 years ago
Matt Clay 62d03c8e75
ansible-test - Fix subprocess management. (#77638)
* Run code-smell sanity tests in UTF-8 Mode.
* Update subprocess use in sanity test programs.
* Use raw_command instead of run_command with always=True set.
* Add more capture=True usage.
* Don't expose stdin to subprocesses.
* Capture more output. Warn on retry.
* Add more captures.
* Capture coverage cli output.
* Capture windows and network host checks.
* Be explicit about interactive usage.
* Use a shell for non-captured, non-interactive subprocesses.
* Add integration test to assert no TTY.
* Add unit test to assert no TTY.
* Require blocking stdin/stdout/stderr.
* Use subprocess.run in ansible-core sanity tests.
* Remove unused arg.
* Be explicit with subprocess.run check=False.
* Add changelog.
3 years ago
Felix Fontein 841bdb74eb
Replace antsibull with antsibull-docs (#77504) 3 years ago
Sam Doran 1b947eaf92
arg_spec - Return aliases in validation result and update aliases (#77576)
When looking up the `no_log` setting for a parameter that is an alias in
`AnsibleModule._log_invocation()`, the alias value will always be an
empty dictionary since `self.aliases` on the `AnsibleModule` instance is
never updated after initialization. Since the `no_log` setting is on the
canonical parameter not the alias, an incorrect warning is issued if the
parameter matches `PASSWORD_MATCH`.

This PR returns the aliases dictionary as an attribute of the
`ValidationResult` and updates the `aliases` attribute on the
`AnsibleModule` instance.
3 years ago
Matt Clay b960641759 ansible-test - Support multiple coverage versions.
ci_complete
ci_coverage
3 years ago
Jack 4d69c09695
ansible-galaxy collection - ensure dependencies are a dict (#77561)
* fix traceback when installing collection with dependencies set to None
3 years ago
Sloane Hertel 4faa576ee9
Add a test and changelog for non-string host names in yaml inventory files (#77569) 3 years ago
Brian Coca abdd237de7
Yolo (#77554)
* Revert "Revert "Config, ensure templating happens at functions (#77483)""

This reverts commit 94c9106153.

* removed update configdata, which is unused
* removed test for action we don't perform anymore
* removed unused configdata
3 years ago
Matt Clay 191d9a771a ansible-test - Remove more Python 2.7 support. 3 years ago
Sloane Hertel 76273a6b80
Clean up integration tests (#77516) 3 years ago
Martin Krizek cbe42bff7f
Allow for lazy evaluation of Jinja2 expressions (#56116) 3 years ago
Brian Coca f2ab920822
Better info sourcing (#77511)
Task is  authoritative
   also includes latest per loop info
   and fix tests
3 years ago
Brian Coca 3b9592fcaf
Move to display as single source of verbosity (#77498)
* Move to display as single source of verbosity

Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
3 years ago
htol d7fbde4ea9
Make "~/.ansible" path configurable (#76114)
* replace hardcoded '~/.ansible' to C.ANSIBLE_HOME
* rename previously existing env ANSIBLE_HOME in env-setup script
* modify cache dir monkeypatching in galaxy api unit tests
* update "version_added" to 2.14 for ANSIBLE_HOME
* fix description of collections with proper use of ANSIBLE_HOME

Co-authored-by: htol <github@h-tol.net>
Co-authored-by: stefanwascoding <stefan@syntaxhelden.de>
3 years ago
Corubba 94138cf608
template override colon bugfixes (#77495)
* Proper error on missing jinja2 override separator

Properly catch the case when no (or a wrong) separator is used in a
jinja2 override, and return a useful error message to the user.

* Support colons in jinja2 override value

By limiting the split to 1, any colons in the value are preserved and
passed on.
3 years ago
Martin Krizek 94eff70030
PlayIterator - remove deprecated states (#77444) 3 years ago
Sloane Hertel 477c55b0d2
Get git executable for collections in git repos (#77493)
* Fix traceback installing collections from git repos if git is not installed
3 years ago
Matt Martz 66c5844978
ansible-test - Don't fail if network cannot be disconnected (#77472)
* Don't fail if network cannot be disconnected

* add clog frag
3 years ago
h4rr21 1d9c68d27e
honor use_proxy parameter (#77312)
* honor use_proxy parameter
* fix uri test with "use_proxy: no"
* fix urls.py module

Co-authored-by: Carlos <Juan.Carlos.Cardenas.Viera@ibm.com>
3 years ago
Felix Fontein 6216c9fc93 Fix sanity test crash. 3 years ago
Matt Martz 2797dc644a Add test to assert that missing libyaml doesn't result in an error. Fixes #77437 3 years ago
Martin Krizek 9f82fc204b
apt_key: remove deprecated key module param (#77409) 3 years ago
Martin Krizek f933f9a178
get_url: remove deprecated sha256sum module param (#77408) 3 years ago
Martin Krizek 26149c45df
import_playbook: remove deprecated free form params (#77431) 3 years ago
Martin Krizek d4dd4a82c0
Remove deprecated default.plugin_filters_cfg config option (#77429)
* Remove deprecated default.plugin_filters_cfg config option

Fixes #77398

* Fix tests
3 years ago
Sloane Hertel 87a8fedd94
Fix type for ansible-galaxy server config definitions (#77424)
* Fix type for ansible-galaxy server config definitions

* changelog

* check attr without api call

* pep8
3 years ago
Matt Clay 4b51e61645 ansible-test - Fix origin host target filtering. 3 years ago
Martin Krizek 6d260ad967
Remove COMMAND_WARNINGS and warn (#77411)
* Remove COMMAND_WARNINGS and warn

Fixes #77394
3 years ago
Martin Krizek e080bae766
Remove deprecated ALLOW_WORLD_READABLE_TMPFILES config option (#77410)
Fixes #77393
3 years ago
Jordan Borean 60b4200bc6
winrm - ensure callers PATH for kinit is set (#77401)
* winrm - ensure callers PATH for kinit is set

* Fix unit test expectations

* Fix type annotation
3 years ago
Matt Clay 353511a900
Add script to handle more deprecations. (#77400)
* Add script to handle more deprecations.

This script currently supports deprecations from the following sanity tests:

* deprecated-config
* update-bundled

* Ignore script in package-data test.
3 years ago
Sloane Hertel f8ec660e62
[ansible-galaxy] Fix listing collections with null namespace/name/version metadata (#77014)
* Fall back to implicit namespace.name from the path if the metadata is invalid

* Test listing a collection with null namespace/name/version fields in its galaxy.yml
3 years ago
Matt Clay 4ccc4c5aeb Report relative path in deprecated-config test. 3 years ago
Sloane Hertel 87d52e0ce0
Only require a keyring if collections with signatures will be installed (#77355)
Fixes #77349
3 years ago
Matt Clay 50d4cf931c
Start of 2.14 development cycle. (#77378)
* Delete old changelog fragments.
* Add 2.14 changelog stub.
* Add porting guide.
* Update version and codename.
* Add temporary deprecated version ignores.
3 years ago
Matt Clay 708e4c206a ansible-test - Update default test containers. 3 years ago
Felix Fontein 7075fa36a1
Bump requirements for antsibull to 0.42.0 (which depends on antsibull-changelog >= 0.14.0). (#77362) 3 years ago
Matt Martz 4723eb9caa
Add family fallback for RHEL to register as RedHat (#77371)
* Add family fallback for RHEL to register as RedHat

* Update tests to properly assert RHEL behavior
3 years ago
Felix Fontein babc26adc1
Handle errors during ansible-doc --metadata-dump more gracefully (#77035)
* Add option --no-fail-on-errors to return errors for ansible-doc --metadata-dump in JSON result instead of failing.

* Adjust changelog fragment.

* Add basic tests.
3 years ago
Matt Clay 55f90ba31f ansible-test - Remove RHEL 9.0b support.
Support can be restored once RHEL 9 has been released.
3 years ago
Matt Martz afecc6400e
Action Plugin argspec validation (#77013) 3 years ago
James Milligan ea7f24a1d5
Support ignoring of certificates for ansible-galaxy during SCM cloning (#67616)
* Support ignoring of certificates for ansible-galaxy during SCM cloning

* Add integration tests installing a role from an untrusted repository

Test installing the role without --ignore-certs fails
Test installing the role with --ignore-certs is successful
3 years ago
Sloane Hertel 2769f5621b
Add find_spec and exec_module to RestrictedModuleLoader (#76427)
* Add find_spec and exec_module to RestrictedModuleLoader

* Fix getting new loader with correct path

Fix pep8 errors

* Use convert_ansible_name_to_absolute_paths instead of the loader path

* Apply suggestions from code review

* Fix type hints and test them in CI

* Fix error message for ansible.module_utils.basic if it's missing

Add mypy ignored missing imports for controller sanity tests

* Add mypy attr-defined ignore entries for python 3.8, 3.9, 3.10 for vendored six

Add mypy attr-defined ignore for python 2.7 in lib/ansible/utils/collection_loader/_collection_finder.py

* Just test controller python versions to simplify ignoring mypy errors
3 years ago
Sloane Hertel f96a661ada
ansible-galaxy - add configuration options for more flexible collection signature verification (#77026)
* Add a toggle to control the number of signatures required to verify the authenticity of a collection

* Make the default number of required valid signatures 1

* Add option to make signature verification strict and fail if there are no valid signatures (e.g. "+1")

* Use a regex to validate --required-valid-signature-count

* Add a toggle to limit the gpg status codes that are considered a failure

* Update documentation and changelog

* Add unit and integration tests for the new options

* Fixes #77146

Fix using user-provided signatures when running 'ansible-galaxy collection verify ns.coll --offline'

Add a test for a user-provided signature when running ansible-galaxy collection verify with --offline

Fix displaying overall gpg failure without extra verbosity

Add a test for displaying gpg failure without verbosity

Improve documentation to be more clear that signature verification only currently applies to collections directly sourced from Galaxy servers
3 years ago
Matt Martz c1a34d5a63
Update systemd strategy to not confuse NetworkManager (#77243)
* Update systemd strategy to not confuse NetworkManager. Fixes #76958

* Add tests
3 years ago
Abhijeet Kasurde 1100289a45
docker_util: Handle error in JSON parsing (#77298)
While getting hostname from container, podman command
fails to return JSON so wrap exception and return
hostname as 'None'

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
3 years ago
ChanthMiao 34e60c0a7a
ansible_distribution: Add support for Uos/Deepin (#77275)
* ansible_distribution: Add support for Uos/Deepin

* Add changelog fragment.
3 years ago
Felix Fontein 4d984613f5
validate-modules for plugins: do some more schema validations so that some issues that are currently reported for modules are also reported for plugins (#77268)
* Add more sanity tests on schema level (so they also work for plugins).

* Fix various issues the sanity test reported.

* Add changelog fragment.

* Fix function name.
3 years ago
Felix Fontein e3c72230cd
unarchive: fix io_buffer_size option, remove ignore.txt entry (#77271)
* Fix io_buffer_size option.

* Remove ignore.txt entry by adding action plugin only options to module's argument spec.

* Add changelog fragment.

* Adjust unit tests.
3 years ago
Matt Clay e8afdac06e ansible-test - Fix delegation inventory path. 3 years ago
Matt Martz 7cb581ed2c
Support podman-remote in ansible-test (#75753) 3 years ago
Matt Clay 0c514bcf54 ansible-test - Fix PS coverage `--all` generation. 3 years ago
Matt Clay f68c66a3ef Remove collections compat from controller code. 3 years ago
Matt Clay 822fddd627 Fix yaml loader and compat. 3 years ago
Matt Clay f43c54d890 ansible-test - Update default containers to 5.7.0. 3 years ago
Matt Clay 3d5637beec Initial mypy sanity test support for core. 3 years ago
Sloane Hertel 8063643b4c
Fix collection redirects for filter and test plugins (#77210)
* Fix collection redirects for jinja2 filters/tests

* Handle recursive redirects

Co-authored-by: Matt Martz <matt@sivel.net>
3 years ago
Felix Fontein 0990c4ca7c
Extend validate-modules to also validate plugins (#71734)
* Let validate-modules also validate plugins.

* Support 'option' in 'cli'.

* Use DOCUMENTABLE_PLUGINS instead of UNDOCUMENTED_PLUGIN_TYPES.

* Support 'keyword', clean up error codes.

* Call settings.process_errors only once; remove __version__.

* Add changelog fragment.
3 years ago
Matt Clay af2b1361ff ansible-test - Simplify type hints. 3 years ago
Matt Clay 5a1b891917
Additional type hinting cleanup. (#77188) 3 years ago
Martin Krizek 74a204e6f1
first_found: allow spaces in names (#77141)
* first_found: allow spaces in names

Fixes #77136
3 years ago
saeedhosseini b667409d6d
ansible-test - Correct a typo in comments (#77148) 3 years ago
Matt Clay a06fa496d3
ansible-test - Code cleanup and refactoring. (#77169)
* Remove unnecessary PyCharm ignores.
* Ignore intentional undefined attribute usage.
* Add missing type hints. Fix existing type hints.
* Fix docstrings and comments.
* Use function to register completion handler.
* Pass strings to display functions.
* Fix CompositeAction handling of dest argument.
* Use consistent types in expressions/assignments.
* Use custom function to keep linters happy.
* Add missing raise for custom exception.
* Clean up key/value type handling in cloud plugins.
* Use dataclass instead of dict for results.
* Add custom type_guard function to check lists.
* Ignore return type that can't be checked (yet).
* Avoid changing types on local variables.
3 years ago
Matt Clay 27fe26edbf Update setup_pexpect to prefer pip user installs.
This works around issues on RHEL 7.9 when an old version of pexpect is installed from an OS package.
3 years ago
Sloane Hertel 7657caa072
Fix 'ansible-galaxy collection verify' to display new files/dirs (#76690)
* Fix 'ansible-galaxy collection verify' to report files/directories not listed in the FILES.json

* changelog

* Fix bug in 2.13+
3 years ago
Matt Clay dc6b0d4857 ansible-test - Remove obsolete display of MAXFD.
Use of ``subprocess.MAXFD`` only worked on Python 2.x, which is no longer supported.
3 years ago
Abhijeet Kasurde 4f48f375a0
parameters: handle blank values when argument is a list (#77119)
Fixes: #77108

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
3 years ago
Matt Martz bdf37336c8
Update pylint deprecated plugin to use ansible-invalid* symbols (#77086) 3 years ago
Matt Martz 8cbe1435c2
Use full python package for ansiballz cache filenames (#77090)
* Use full python package for ansiballz cache filenames

* Be a little more explicit about test goals
3 years ago
Matt Clay 4f9c7fd509 ansible-test - Remove validate-modules symlink. 3 years ago
Matt Clay 1a5853d794
Remove obsolete units.compat.mock compat layer. (#77118)
* Remove obsolete units.compat.mock compat layer.
* Update remaining units.compat.mock references.
3 years ago
Matt Clay 5a6e05affb More strict pylint checking of ansible-test code. 3 years ago
Matt Clay b613808277 Remove obsolete unit test builtins compat. 3 years ago
Matt Clay 2cd6cdc6a7
Fix duplicate unit test function names. (#77115)
* Fix duplicate unit test function names.
* Disable faulty tests.
3 years ago
Matt Clay 0bd8106d15 Remove unused mock compat code.
The code is unreachable since Python 3 versions before 3.5 are not supported.
3 years ago
Matt Clay 92800a04c2 Add missing integration test dependency. 3 years ago
Sloane Hertel 88bfb7193e
Use --no-deps when installing amazon.aws and not testing dependencies (#77091) 3 years ago
Matt Clay de9a3bda2c Fix warning in unit tests for _yaml import. 3 years ago
Matt Clay d3d5244912 ansible-test - Fail sanity test on distutils use. 3 years ago
Matt Clay c1a271c792 Remove unused task from template integration test. 3 years ago
Matt Clay d286c2e8b3 ansible-test - Fix CParser import in yamllint. 3 years ago
Matt Clay c27fd777f4 ansible-test - Use quay.io containers in plugins. 3 years ago
Matt Clay addb9baec2 ansible-test - Clean up venv code. 3 years ago
Pablo Escobar Lopez f2612ce169
apply owner/group permissions to top folder with unarchive module (#73024)
* apply owner/group permissions to top folder
* remove unused var unarchive30
* fix permissions for top folders if the tarball include multiple top folders
* added test for top folder ownership
3 years ago
Matt Clay 13d5381e56
ansible-test - Update base and default containers. (#77033) 3 years ago
Matt Martz 494cb4e211 Ensure tests that need to validate higher verbosity messages run with vvvv 3 years ago
Matt Clay 68fb3bf90e
ansible-test - Fix consistency of managed venvs. (#77028) 3 years ago
Matt Martz 3779c1f278
restore inadvertently deleted default for convert_data. Fixes #77004 (#77016) 3 years ago
Brian Coca c9d3518d2f
Fix final fact delegation (#77008)
* fix facts delegation loop overwrite

 partial revert of change to allow facts to be present in each loop iteration
 was not needed in final results as result processing alreayd had the disctiontion
 and ended up breaking the assumptions in the calling code.

 fixes #76676
3 years ago
Matt Clay 3a662ef2c1 Fix package-data sanity test for newer setuptools. 3 years ago
Matt Clay ef4c5cd61b ansible-test - Use newer pip to bootstrap FreeBSD. 3 years ago
Sloane Hertel 43e55db208
ansible-galaxy - add signature verification of the MANIFEST.json (#76681)
* ansible-galaxy collection install|verify:

  - Support verifying the origin of the MANIFEST.json when the Galaxy server has provided signatures.
  - Allow supplemental signatures to use during verification on the CLI/requirements file.

* ansible-galaxy collection install:

  - Support disabling signature verification. This silences the warning provided by ansible-galaxy if the Galaxy server provided signatures it cannot use because no keyring is configured.
  - Store Galaxy server metadata alongside installed collections for provenance. This is used by 'ansible-galaxy collection verify --offline'.

* Add unit tests for method that gets signatures from a Galaxy server

* Add integration tests for user-provided signature sources

- Test CLI option combinations
- Test installing collections with valid/invalid signature sources
- Test disabling GPG verification when installing collections
- Test verifying collections with valid/invalid signature sources

* Make signature verification advisory-by-default if signatures are provided by the Galaxy server

- Make the default keyring None
- Warn if the keyring is None but the Galaxy server provided signatures
- Error if the keyring is None but the user supplied signatures
- Error if the keyring is not None but is invalid

* changelog

* add ansible-galaxy user documentation for new options

Co-authored-by: Matt Martz <matt@sivel.net>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Martin Krizek <martin.krizek@gmail.com>
Co-authored-by: Sandra McCann <samccann@redhat.com>
Co-authored-by: Andy Mott <amott@redhat.com>
Co-authored-by: John R Barker <john@johnrbarker.com>
3 years ago
Jack Scheible d35bef68f5
replace - always return rc (#71963)
* Return rc=0 on success.

Error handling in playbooks generally expects `rc` to be set to 0 when a module has not failed.  Playbook authors should not have to check for the existence of `rc` first.

* Use single definition and added changelog

* Fix up tests with new return value

Co-authored-by: Jordan Borean <jborean93@gmail.com>
3 years ago
Thomas Sjögren b145732973
Resubmit: Use SystemdStrategy for Debian in the hostname module (#76929)
* add DebianStrategy tests
* ensure hostname can be changed by using become
* use Systemd strat for debian and Base for generic.
* add test to ensure all strategies are available

Signed-off-by: Thomas Sjögren <konstruktoid@users.noreply.github.com>
3 years ago
Stephen Ryan 522f9d1050
Make builtin.file modification times support check_mode (#76973)
* Make builtin.file modification times support check_mode
* Add integration test
3 years ago
Matt Clay 0d40423f1c
ansible-test - Fix collection loader import. (#76986)
* ansible-test - Fix collection loader import.

Resolves https://github.com/ansible/ansible/issues/76960
3 years ago
Matt Clay 0d5401d950 ansible-test - Remove unused code.
This was overlooked in a previous commit that removed FreeBSD 12.2 support.
3 years ago
Matt Clay 00a2b7788e ansible-test - Remove cap on cryptography version. 3 years ago
Matt Clay e27b94c467 ansible-test - Update alpine3 container to 3.3.0.
This updates the base image from 3.14.2 to 3.15.0, which includes support for installing binary wheels using pip.
3 years ago
Brian Coca be19863e44
ssh connection: use 'correct' host in all cases (#76017)
ssh plugin, use 'correct' information source in all cases
  * still fallback to pc
  * added inventory to new test
  * undef var can still show as parser error on pc
    now task_exectuer has a  more accurate error handling
3 years ago
Martin Krizek 394d216922
add_host/group_by: fix using changed_when in a loop (#71719)
Fixes #71627
Fixes #75971
3 years ago
Jordan Borean 2749d9fbf9
incidental - remove win_lineinfile coverage (#76952) 3 years ago
Matt Clay 17dd9ff924 ansible-test - Update default containers to 5.5.0. 3 years ago
Felix Fontein bc1e29d722
Bump antsibull version from 0.40.1 to 0.41.0 and ansible sphinx theme version to fix HTML generation in return value tables (#76759)
* Use Jinja2 comment instead of HTML comment to avoid W3C validator warning (-- inside comment).

(For those interested in the history of this: http://www.howtocreate.co.uk/SGMLComments.html#doubledash)
3 years ago
Matt Clay b568e0db6f ansible-test - Remove outdated remote support. 3 years ago
Matt Clay 7e64c4fe55 ansible-test - Update bootstrap URL to not use S3. 3 years ago
Matt Clay 6c29028f05 Update ci-files URL in integration tests.
This uses the CloudFront URL instead of going directly to S3.
3 years ago
Matt Clay 995d7e4db3 ansible-test - Limit installed pyopenssl version. 3 years ago
Matt Clay fbb5d56bd2
ansible-test - Use relative paths in junit output. (#76871)
* ansible-test - Use relative paths in junit output.

Also fix a traceback in the junit callback during automatic fact gathering.

* ansible-test - Handle out-of-tree JUnit paths.
3 years ago
Brian Coca b1d6750e8b
fix block var inheritance (#75287)
* updated tests to conform to new block inheritance

Co-authored-by: Brian Scholer <1260690+briantist@users.noreply.github.com>
Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
3 years ago