* ansible-galaxy - support `resolvelib >= 0.5.3, < 1.1.0`
<https://pypi.org/project/resolvelib/1.0.1> released on 2023-03-09:
- <https://github.com/sarugaku/resolvelib/blob/main/CHANGELOG.rst#101-2023-03-09>
- <https://github.com/sarugaku/resolvelib/releases/tag/1.0.1>
Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>
* Trigger CI by pinning resolvelib with latest version
Since resolvelib is pre-installed in our test containers, we should temporarily pin the latest version allowed to force the tests to run with that version. Once the tests have passed that commit can be reverted.
Please make those changes without force pushing, so that we keep the reference to the passing CI run. We can squash the commits when merging the PR so the temporary commits won't be in the final commit merged to the devel branch.
Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>
* https://github.com/ansible/ansible/pull/80196#discussion_r1136003637
Also test resolvelib with multiple supported versions.
Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>
* Revert "Trigger CI by pinning resolvelib with latest version"
This reverts commit 5518e5dbca.
---------
Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>
* Pin setuptools to lowest supported @ PEP 517 test
This allows catching the behavior of builds under old setuptools.
* Stop invoking `setup.py install` in tests
This is not the part we care about since it involves dealing with the
external runtime dependencies rather than building our source
distribution.
This patch modifies the in-tree build backend to build sdists that swap
out pointers to it in the `pyproject.toml`'s `[build-system]` section.
The effect of this is that the first build from source (for example,
from a Git checkout) uses our PEP 517 in-tree build backend. But the
produced tarball has `build-backend` set to `setuptools.build_meta`
which is the native build backend of `setuptools`. So any following
builds from that sdist will skip using the in-tree build backend,
calling the setuptools' one.
The good news is that if the first build generated the manpages, they
will be included and won't go anywhere even though, a different build
system is in place.
Combined with #80253, this will make sure not to modify the current
source checkout on that first build.
Co-authored-by: Matt Clay <matt@mystile.com>
* 🧪 Switch macOS 13.2 to 12.0 in CI
The former revealed unexpected flakiness while the latter is the
previous value that was used to be stable. This is a temporary revert.
* Skip lookup_url integration test under macOS 12.0
* ✨ Add macOS 13.2 to `ansible-test`
* 🧪 Replace macOS 12.0 with 13.2 in the CI matrix
* Skip `lookup_url` under macOS 13.2
This is due to https://wefearchange.org/2018/11/forkmacos.rst.html
that manifests itself as follows:
TASK [lookup_url : Test that retrieving a url works] ***************************
objc[15394]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[15394]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in t
he fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state
* 📝 Extend ansible-test change note w/ macOS 13.2
* 🐛 Make integration tests compatible w/ modern Git
This patch makes use of the `init.defaultBranch` setting to unify
the test across new and old Git versions since one defaults to
`master` and the other uses `main` for the default branch.
Where possible, it uses the `HEAD` committish to avoid having to
normalize the branch name.
The change fixes the following integration tests:
* `ansible-galaxy`
* `ansible-galaxy-collection-scm` (recursive collection)
* `git`
* 🐛Replace `git-symbolic-ref` with a repo template
This custom Git repository template emulates the `init.defaultBranch` setting
on Git versions below 2.28. Ref: https://superuser.com/a/1559582.
Other workarounds mentioned there, like invoking
`git symbolic-ref HEAD refs/heads/main` after each `git init` turned
out to have mysterious side effects that break the tests in surprising ways.
* 🎨 Make Git integration test non-destructive
This patch makes use of the `$HOME` environment variable to trick Git
into using a user-global config generated in the temporary directory.
* unarchive - properly handle relative path for dest
* Add integration test
* Return output of underlying commands with increased verbosity
* Revert "Return output of underlying commands with increased verbosity"
This reverts commit a2790c8275cdc5697b65670a0beffdc74b741bf6.
* Warn when a relative destination path was provided
* Create a queue per WorkerProcess to receive intra-task updates
* Update `pause` action to use the worker queue
* Deprecate ConnectionBase()._new_stdin
* Add new `Display` convenience method `prompt_until` to manage both controller- and worker-sourced prompting without cross-fork stdin sharing, in-worker mechanism to handle request-response over new worker queue.
This patch creates a thin wrapper around the `setuptools`' PEP 517
build backend in-tree. It features an ability to request generating
the manpage files in the process of building a source distribution.
This toggle is implemented using the `config_settings` mechanism of
PEP 517.
One must explicitly pass it a CLI option to the build front-end to
trigger said behavior. The packagers are expected to use the
following call:
python -m build --config-setting=--build-manpages
This option has no effect on building wheels.
🧪 The change includes integration tests
This test runs building and re-building sdists and wheels with and
without the `--build-manpages` config setting under the
oldest-supported and new `setuptools` pinned.
It is intended to preserve the interoperability of the packaging setup
across Python runtimes.
An extra smoke test also verifies that non PEP 517 interfaces remain functional.
PR #79606
Co-authored-by: Matt Clay <matt@mystile.com>
The previous change to the default mode of operation for the plugin is now limited to collections.
This enables easier testing of the plugin in core, so the previously removed tests have been restored.
* Add condition that causes a when to skip a task
* Fix up tests
* Use false_condition instead of failed_condition
* Remove formatting accidentially added
* Fix sanity
* ansible-inventory, add --limit option
* also graph
* optimize the when not providing limit
* added tests and clog
* avoid empty groups, fix tests
* i swear there as an ignore_errors there already!!?!?!?
* Fix stdout test fallback
Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
* simpler approach using subset against inventory directly
* tyupose
* renamed funciton to what it actuall does
got yaml in line with others about removing empty groups
* have graph ignore limts, also note same for --host
* fixed typo long line
* better test
* cause hosvars are not a thing
---------
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>
Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
* Change ansible_job_id format
...to be something that does not resemble float or other type which
value could be changed by literal_eval that jinja2_native uses.
Specifically the format of '%d.%d' is converted from str to float and
then back to float which may result in truncating the number resulting
in the job not being found because the job id does not exist.
* Fix detection of available hashlib algorithms
Detection of hashlib algorithms now works on Python 3.x.
The new implementation works on Python 2.7 and later.
Test coverage is provided by both integration and unit tests.
* Add additional details about hashlib in docs
* quiet default ansible-doc integration test output
* typical non-verbose output was exceeding 27k lines per fun
* disables `set -x` unless `-v` is passed to the script (eg when ansible-test is called with `-v` or `--retry-on-error` adds it on the second try
* added simple progress echoes
* suppress some grep output
* only enable `set -x` for >= `-vvv`
* fix shellcheck default complaint
* Improving the documentation on how we generate the default value of the filename parameter
* fix pep8
* removing unnecessary documentation and improving the module's return
* making the RETURN docs
* pep8
* version_added and changelog
* module._diff
* module._diff fix
* add rudimentary tests for new outputs
---------
Co-authored-by: Matt Davis <mrd@redhat.com>
* Update `collections.abc` imports
- Use `six.moves` for modules and module_utils
- Use `collections.abc` for controller code
This avoids using `ansible.module_utils.common._collections_compat`,
which was added before the vendored `six` was updated to provide these
imports.
* Update _collections_compat to use six.moves
Also update the custom pylint rule to reflect this change.
* add null 'manifest' key to metadata for git repo collections containing MANIFEST.json
changelog
* set to Sentinel instead of None
* Test installing a collection in a git repo that contains a MANIFEST.json
* fix test
* Update changelogs/fragments/ansible-galaxy-install-git-src-manifest.yml
* validate-modules - Remove `__future__` limits
Limits on specific `__future__` imports are handled by other sanity tests.
* Add integration test for module/plugin imports.
* Normalize deprecation records.
* Fix alias deprecations in suboptions.
* Report in which option an alias warning happened for suboptions.
* Add deprecation tests for suboptions.
* Also test deprecation in list of dicts.
* Adjust unit tests for toplevel alias deprecation field name change.
* Add support for importlib.resources
* Remove the importlib.resources imports
* return the correct data
* Some code comments, and re-order for consistency
* Disallow traversing packages below an individual collection
* Add a traversable class for namespaces
* Re-use variable
* Utilize itertools.chain.from_iterable
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Simplify logic to check for packages from ansible loaders
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Just a generator expression, instead of a generator
* docstrings
* Add comment about find_spec for our namespaces
* Add some initial unit tests for importlib.resources
* normalize
* Utilize importlib.resources for listing collections
* collections_path is already in config, just use config
* install uses a different default for collections_path
* Remove unused import
* Remove duplicate __truediv__
* Bring back TraversableResources
* Apply some small suggestions from code review
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>
* Remove cross contamination between plugin loader code and CLI code
* Remove unused import
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>
* validate-modules: don't fail on invalid YAML
When validate-modules encounters invalid YAML (e.g. in the EXAMPLES
section), it tries to reformat the exception to include the line number
in the Python file instead of the line number of the embedded YAML
document. However, PyYAML doesn't allow modification of the Mark object
(anymore) which leads to a new exception being raised, instead of
reporting the original exception.
As the original exception is not needed in other places anymore, we
don't have to modify it at all and can just compute the right line
number when reporting the error via ansible-test.
Fixes: #75837
* Add test for invalid module doc YAML syntax.
Co-authored-by: Matt Clay <matt@mystile.com>
Also improve the ansible-test-container integration test:
- Add coverage for the no-probe code path.
- Add work-arounds for centos6 containers (to support backporting).
- Avoid systemd debug when the container doesn't use cgroup.
* background threads writing to stdout/stderr can cause children to deadlock if a thread in the parent holds the internal lock on the BufferedWriter wrapper
* prevent writes to std handles during fork by monkeypatching stdout/stderr during display startup to require a mutex lock with fork(); this ensures no background threads can hold the lock during a fork operation
* add integration test that fails reliably on Linux without this fix
* galaxy: Add license_file to manifest directives
* ag collection build: Test license handling
This adds tests to ensure that
- REUSE licensing files: .reuse/dep5, LICENSES/*, anyfile.license
- galaxy.yml license_file
are always included in the manifest.
This to match the structure with the tests for the user module. When having the tests available as stand-alone tests, it will be easier to add more integration tests in the future.
* Log `runme.sh` execution in integration tests
This patch adds `set -x` where it's missing in the integration tests.
It also enables `pipefail` in `runme.sh` scripts that use pipes.
* Add a change note for PR #79263
* Restrict `wheel` below v0.38.0 under Pythons < 3.7
* Add a change note for PR #79187
* Update changelogs/fragments/79187--wheel-0.38.0.yml
Co-authored-by: Matt Clay <matt@mystile.com>
* Use constraints file when installing wheel.
Co-authored-by: Matt Clay <matt@mystile.com>
Use GalaxyAPI for isinstance check instead of RoleDistributionServer, since the latter is defined in __main__ sometimes (when running integration tests or ansible-galaxy from source) and importing from ansible.cli.galaxy won't reference the same object.
* copy module - fix copying directories containing modified subdirs with remote_src=True. Previously, the first changed subdir would prevent recursively checking for changes for in subdirs at the same level.
* Fix reporting changed for copying empty directories with remote_src=True. If a directory is created on the remote but nothing else, changed is True.
* Custom salt for ansible-vault encrypt
add VAULT_ENCRYPT_SALT config
add salt testing
Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
Co-authored-by: Piotr Stawarski <p@stawarski.pl>
* Fix installation from source transforms symlinks of dirs to empty dirs
* Add test to check symlinks to dirs are respected when installing from source
* Add changelog for collection install from source symlink to dirs issue
* Ensure that we do not squash keywords in validate. Fixes#79021
* become_user: nobody should only apply to the test tasks, not the setup_test_user role
* Update how become_user is specified
* Add test to ensure keyword inheritance is working for become
* Add clog frag
* Cache fattributes to prevent re-calculation
* ci_complete
* Remove unnecessary getattr
* Do not crash templating when filter/test name is not a valid Ansible plugin name.
* Store and re-raise KeyError if there was one.
Co-authored-by: s-hertel <19572925+s-hertel@users.noreply.github.com>
* omit keyword should reset to context
ensure we use context/inheritance when calculating value,
using default only when context is unavailable.
fixes#75692
- correct 'vars:' precedence to allow phasing out of include_params
- actually merge vars and always include role_vars
- avoided dupe deps from giving wrong vars
- use 'first' instance of dep as others are from previous instances/invocations
and can have diff values for vars
- ensured deps only provide exportable vars themselves
- added COMMENTS
- added tests
- apply export restrictions setting to defaults
- use 'public' as cutoff
Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
ci_complete
* Ensure that meta/runtime.yml redirects are FQCRs.
* Avoid crash when YAML errors without context mark happen, for example if file starts with 'foo---' instead of '---'.
- Allow disabled, unsupported, unstable and destructive integration test targets to be selected using their respective prefixes.
- Allow unstable tests to run when targeted changes are made and the ``--allow-unstable-changed`` option is specified (resolves https://github.com/ansible/ansible/issues/74213).
* Add --offline option to 'ansible-galaxy collection install' to prevent querying distribution servers
This allows installing/upgrading individual tarfiles to have dependency resolution.
Previously needed to be done manually with --no-deps or else all collections and dependencies needed to be included in the requirements.
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* new _fqcn attribute to plugin objects
* unbreak plugins in subdirs
* Fix inadventent changes to _load_name and use existing vars
* add plugin aliases and name property, and replace plugin._load_name where incorrect
* Fix listing plugin names
Fix listing legacy and builtin together
test deprecated plugin documentation
fix doc extensions
remove sometimes inaccurate _load_name handling from plugin.name
* Add tests for REJECT_EXTS and doc extensions
Fix unpredictable collection redirects so non-fqcns in the redirect list are guaranteed to be legacy (instead of determined by the collections keyword)
Move aliases and name properties to _update_object so all plugin types, including doc fragments, can use them
* make legacy plugin names internally consistent
* rename attributes to ansible_name and ansible_aliases
* Recover unix load average into an ansible_loadavg fact
The average load is available out of the box with os.getloadavg().
It adds useful context to the existing ansible_memory_mb and ansible_mounts
facts that provide size and utilization or memory and disk space.
* Add an integration test
* Limit Galaxy API calls during ansible-galaxy collection dependency resolution when possible
Installing a tarfile with a dependency from a Galaxy server (e.g. dependencies: {'ns.coll': '>=1.0.0'}) does not get the available versions of the dependency from the galaxy server if a sufficient version is already installed.
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* refactor and remove redundant code in documentation
allow location and building api to be more accessible
fix issues with displaying ansible.legacy and ansible.builtin
ensure we don't x2 process tokens (some modules reference them also) fixes#77764
move to constants vs hardcoded
more informative errors and comments
now have actual filter/test plugins, which expose the filter/test functions
moved filter/test loading/finding logic into jinja2pluginloader, removed dupe implementations
added tests for case in which we unique by basename when listing
Update lib/ansible/utils/plugin_docs.py
Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
* Add a warning for collections that are attempting to be autoloaded to no effect
* Deprecate REQUIRES_WHITELIST and add support for REQUIRES_ENABLED so the docs are accurate
* Fix documentation
* add more vars plugin tests
* Simplify code and add a FIXME for another bug
* fix precedence
* Make setting the class attr at all a warning, even if it's True
* Add fun parsing for _load_name
* include _load_name in messages
This specific integration test gets stuck periodically causing the
Galaxy jobs to be killed on timeout wasting an hour of runtime. The
module that gets stuck waiting on Pulp is an in-test one, called
`setup_collections`. When it works, the task is complete in around 70
seconds but when it doesn't, it just freezes the whole play.
This patch attempts to make it fail faster by putting a reasonable
timeout value of 2 minutes.
On BusyBox systems such as Alpine, chattr on a tmpfs fails with a status of 0 and output only on stderr.
This change updates the test to not assume output on stdout.
* add required file to collection skeleton
This file is required to be able to upload a collection.
It is present in
https://github.com/ansible-collections/collection_template/blob/main/meta/runtime.yml
but that does not get used by default.
Without this, if you use the "ansible-galaxy collection init" command
and you try and publish that collection without adding this file, you
get the error:
"ERROR! Galaxy import process failed: 'requires_ansible' in
meta/runtime.yml is mandatory, but no meta/runtime.yml found (Code:
UNKNOWN)"
Also updates relevant test and adds a changelog fragment
* Relocate venv-pythons.py script.
* Split out unsupported-directory test.
* Split out sanity-import test.
* Split out sanity-validate-modules test.
* Split out units test.
* Split out integration test.
* Split out units-constraints test.
* Split out integration-constraints test.
* Split out coverage test.
* Split out sanity test.
* Split out git test.
* Update test groups.
* Don't actually update the cache in check mode
Add tests for updating the cache in check mode
* Don't mark packages as manually installed in check mode and add a test for it
* changelog
* Update test/integration/targets/apt/tasks/apt.yml
* fix test
* update attributes of files that are symlinks
* update attributes of files that are hard links
* fix default state in documentation
* remove unnecessary suppression
* add to changelog
* `context/target` tests must be in groups 1 - 2.
* `context/controller` tests must be in groups 3 - 5.
This makes it easier to efficiently organize groups and balance test runtimes.
* Rethread pr/70185 through the dependency resolver
Hang optional metadata toggle on the ConcreteArtifactsManager instead of threading it through whole list codepath
Don't error while listing collections if a collection's metadata is missing keys required for building a collection.
Give an informative warning if metadata has been badly formatted.
Co-authored-by: Sam Doran <sdoran@redhat.com>
The current language pack fails to install on Ubuntu 22.04.
However, since the langauge only needs to be non-English, changing it preserves the test functionality.
* wait_for - Read file and perform comparisons using bytes to avoid decode errors. Fixes#78214
* Write non-ascii via script instead of static file
* Use contexlib.closing to support py27 context manager
* Use executable from task, instead of shebang
* Update encoded bytes to utf16
* Don't set become_flags in become integration test.
Setting the flags in the test overrides the default value in the become plugins, which was not intentional.
* Remove hack for broken implicit tilde support.
It was only broken because the tests were overriding the default become flags for sudo.
* Simplify assertions.
`group` and `setup_cron` integration tests currently install `libuser` and `faketime`
packages respectively, from the non-default edge/testing repositories. It is no
longer necessary to do so for the former so this patch drops the repo option. As
for the latter, this patch includes a note explaining when the same could be done
for it too.
This patch also adds a note on installing `faketime` on Alpine in CI.
Co-Authored-By: Matt Clay <matt@mystile.com>
* Add setup_test_user integration target.
This integration target creates a temporary test user and removes
it when the current integration target finishes. This user is
configured with a password so it can be used with become, even
when the current user is unprivileged.
* Fix slurp integration test.
* Fix unarchive integration test.
* Fix module_utils integration test.
* Move undefined check from concat to finalize
In the classic Jinja2's Environment str() is called on the return value of the
finalize method to potentially trigger the undefined error. That is not
the case in NativeEnvironment where string conversion of the return value is
not desired. We workaround that by checking for Undefined in all of our concat
functions. It seems simpler to do it earlier in the finalize method(s) instead.
As a side-effect it fixes an undefined variable detection in imported templates.
Fixes#78156
ci_complete
* Fix sanity
* ...
* sigh
* [dnf] Fix skip_broken, add test coverage
Change:
- skip_broken was set in config but not actually used in calls to
base.install()
- added a lot of test cases with specialized repo
- got rid of external (docker repo) nobest test cases since the
specialized repo works well for those too
- Slight cleanup and adding comments in dnf module
Test Plan:
- ci_complete
Tickets:
- Fixes#73072
Original-author: Rick Elrod <rick@elrod.me>
* Use a better test for checking results list
ci_complete
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Rick Elrod <rick@elrod.me>
* Relocate update-ignore.py for easier re-use.
* Add script to ease collection testing.
* Skip ignore rewrite if file does not exist.
* Add integration test for the shebang sanity test.
* Fix ansible-test-no-tty integration test.
Previously the test only verified a TTY was not used if a TTY already existed.
This prevented the test from verifying behavior when run in CI.
Now the test creates a PTY before invoking ansible-test.
* Clean up ansible-test-docker integration test.
Add a new handler class ZipZArchive to use unzip -Z as an alternative to zipinfo
Run 'unzip -Z' in can_handle_archive so we fall back to the next handler if it's not available (failing in is_unarchived is too late)
* Add a test for unzip -Z when zipinfo is not available
* Update test for missing binary altogether by removing /usr/bin from the PATH
* ansible-galaxy - support resolvelib versions >= 0.5.3, <= 0.8.1
Test incompatibilities are removed for resolvelib >= 0.6.0
Test against the latest 0.8.x version and fix requirements
* Fix tests - use a venv for testing the range of resolvelib versions
* Update temporary hardcoded fallback for ansible-test
* Update hardcoded upperbound for sanity tests
* Make error check more flexible
checksum can also accept a checksum only file (no filename beside the checksum).
fixes#54390
Co-authored-by: Baptiste Mille-Mathias <baptiste.millemathias@gmail.com>
Wong Hoi Sing Edison