* Improve host_group_vars efficiency:
* normalize the basedir with `os.path.realpath()` once and cache it
* cache missing paths/files
* reduce the calls to `isinstance`
Add a couple more general improvements in vars/plugins.py get_vars_from_path():
* call `PluginLoader.all()` once for vars plugins and reload specific
plugins subsequently
* don't reload legacy/builtin vars plugins that are not enabled
Add a test for host_group_vars and legacy plugin loading
Co-authored-by: Matt Davis <mrd@redhat.com>
* changelog
* Add a new is_stateless attribute to the vars plugin baseclass
update integration tests to be quieter and use the same test pattern
Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False)
move realpath cache to host_group_vars (do not smuggle call state as instance data)
refactor under a single 'if cache:' statement
Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory.
remove call to super(VarsModule, self).get_vars()
use the entity name as the cache key instead of variable location
Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group
Replace startswith by checking index 0 of the name instead, since host/group names are required
* rename is_stateless to cache_instance to make it more clear what it does
* add plugin instance cache using the path to plugin loader
reduce loading stage option if a new instance isn't created
don't require a known subdir on PluginLoader instantiation for backwards
compatibility
rename attribute again
contain reading from/initializing cached instances to a plugin loader method
* Deprecate v2 vars plugins
* Refactor to use the cache in existing plugin loader methods
Rename the attribute again
Refactor host_group_vars with requested changes
Make changelog a bugfixes fragment
Add a deprecation fragment for v2 vars plugins.
Add type hints
* unbreak group_vars
* Apply suggestions from code review
* misc tweaks
* always cache instance by both requested and resolved FQ name
* add lru_cache to stage calculation to avoid repeated config consultation
* handle KeyError from missing stage option
---------
Co-authored-by: Matt Davis <mrd@redhat.com>
When templating tags (which happens outside of standard `post_validate`) we
need to template each object in the inheritance chain and set the templated
values on those objects individually. That way when `task.tags` is called the
`extend` functionality properly picks up the templated values of all
parents into one flatten list.
Fixes#81053
revert to previous behavior to push vars to play at compile time
add `public` parameter to allow per import control of exporting (vs just the global config)
Co-authored-by: tchernomax <maxime.deroucy@gmail.com>
Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
Provide more information about plugin usage which needs to be
used instead of the deprecated plugin in the deprecation message.
Fixes: #80561
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Test heuristic_log_sanitize. See #81689
* Add note about what this test is doing
* grammar
Co-authored-by: Matt Clay <matt@mystile.com>
---------
Co-authored-by: Matt Clay <matt@mystile.com>
* Make sure paths are correct when building collection files manifest
This commit makes sure the path of the files part of
the collection build manifest are correct.
This commit uses os.path.commonprefix instead of
dealing with strings.
Signed-off-by: Carlos Camacho <ccamacho@redhat.com>
Bugfix Pull Request
Fixes: #81618
* Revert the change note type to `minor_changes`
* Clarify the change note with user-oriented details
---------
Signed-off-by: Carlos Camacho <ccamacho@redhat.com>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
PR #81606.
Prior to this patch, when `--pre` CLI flag was not passed, the
dependency resolver would treat concrete collection dependency
candidates (Git repositories, subdirs, tarball URLs, or local dirs or
files etc) as not meeting the requirements.
This patch makes it so pre-releases in any concrete artifact
references, and the ones being specifically pinned dependencies or
user requests, met anywhere in the dependency tree, are allowed
unconditionally.
This is achieved by moving the pre-release check from
`is_satisfied_by()` to the `find_matches()` hook, following the
Pip's example.
As a bonus, this change also fixes the situation when a collection
pre-releases weren't considered if it didn't have any stable releases.
This now works even if `--pre` wasn't requested explicitly.
Finally, this patch partially reverts commit
6f4b4c345b, except for the tests. And it
also improves the `--pre` hint warning to explain that it mostly
affects Galaxy/Automation Hub-hosted collection releases.
Ref #73416
Ref #79112Fixes#79168Fixes#80048Resolves#81605
Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
* Symlinks in the collection might be pointing to non-existent
targets. Check and report the failure to the user.
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* when --diff is used with content parameter, print destination
path instead of temporary file path.
Fixes: #79749
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* If all tasks in the role are skipped or unreachable, the role is not marked as complete for the host.
* Only mark the role as complete if a task in the role succeeds or fails for the host.
* Before performing shredding the original file, check if the
destination file location is writable or not. This will
prevent corruption of original file.
Fixes: #81455
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Python 3.11.4 introduces a new parameter 'filter' in extract and
extractall in tarfile. Handle deprecation warning message emitted
in Python 3.12.
* added probing mechanism in ansible-galaxy code to detect broken
data filter implementation in tarfile.
Fixes: #80832
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Co-authored-by: Matt Clay <matt@mystile.com>
As of coverage 7.1.0, files without arcs are not generated.
To work around this, an empty (0, 0) arc is used for each file instead.
This is a follow-up to https://github.com/ansible/ansible/pull/81077
ci_coverage
ci_complete
Improves the error checking when running PowerShell modules using
Ansible.ModuleUtils.Legacy. It will only return an rc of 1 if both the
PowerShell module runner signalled an error occurred and those error
records were present in the output. This should reduce some false
positive errors when using the older module style.
* apt: ignore fail_on_autoremove and allow_downgrade when using aptitude
* fail_on_autoremove (--no-remove) and allow_downgrade (--allow-downgrades)
parameters are only valid for apt-get and not for aptitude. Ignore them when
aptitude is detected and used.
Fixes: #77868
* Gather value of password using debconf-get-selections command,
use this information for idempotency.
Fixes: #47676
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Enable mypy for the entire packaging directory
* Return CompletedProcess only when capturing output
This allows stdout/stderr on CompletedProcess to be `str` instead of `str | None`.
The unused args on CompletedProcess have been removed.
Overload type hints have been added to reflect these changes.
* Relax return type on ensure_venv
This improves consistency with its usage, since `run` accepts `env` of `dict[str, t.Any]`.
Also removed unnecssary `str()` usage when updating `env`.
* Fix type hint on suppress_when
* Fix callable annotation
* Add type hint for command_parser
PyCharm complains about using a protected member, and also that it can't find the type in the type stubs.
However, mypy properly recognizes the type.
* Avoid unnecessary TypeVar usage
* Back out use of communicate, add better comments, add bufsize, and align with subprocess._communicate
* tests
* re-order logic slightly
* more comments
* loopty loop
* yet another comment
* Revert "yet another comment"
This reverts commit 96cd8ada5fa0441b92f2298bdaa6cb40594847d2.
* Revert "loopty loop"
This reverts commit 96ea066f6a7d18902c04a14f18dd79b38e56f5e7.
* ci_complete
* Copy in comment too
* Wording updates
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>
* Back out bufsize
---------
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>
* a recent optimization lost the unsafe lookup disable behavior when templating conditionals with inline templates that referred to untrusted values
* added regression test to catch this case
* iptables chain creation does not populate with a rule
fixes#80256
* Add changelog fragment
* Add rules and flush chain during integration tests
* Check chain rule on comment
* Update test/integration/targets/iptables/tasks/chain_management.yml
* User-provided collection type might differ from collection
source. Cross-check the type before proceeding
Fixes: #79463
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* ansible-galaxy - improve ignoring multiple signature status codes when using --ignore-signature-status-code
* fix backwards compatibility by adding a new plural option instead, and hide the singular from --help
Since man pages aren't accessible to users after a `pip install`, there's no need to include them in the sdist.
This change makes it trivial to build man pages from source, which makes them much easier to iterate on.
It also simplifies creation and testing of the sdist, since it no longer requires building man pages.
The new `packaging/cli-doc/build.py` script can generate both man pages and RST documentation.
This supports inclusion on the docs site without a dependency on `ansible-core` internals.
Having a single implementation for both simplifies keeping the two formats in sync.
* Remove datetime.datetime.utcfromtimestamp and datetime.datetime.uctnow
from controller code since they are deprecated in Python 3.12.
* Update target side code to use new utcfromtimestamp and utcnow utils in ansible.module_utils.compat.datetime that return aware datetime objects on Python 2.7 and 3.
Co-authored-by: Matt Clay <matt@mystile.com>
* Raise the minimum setuptools version to 66.1.0
This is the first version to support Python 3.12.
While Python 3.10 and 3.11 could use an older version, a consistent minimum is easier to work with and test against.
* Fix PEP 517 integration test
* update docker containers versions to use newer ansible-test ref in the pre-built venvs
* Allow invoking ansible-test with Python 3.12
* Add python3.12 to the INTERPRETER_PYTHON_FALLBACK
* changelog
* add Python 3.12 as a non-default Python version for the test containers
* Update mypy ignores for Python 3.12
* Add Python 3.12 to CI matrix for unit tests, generic tests, and galaxy
* Update unit test for using the Python 2 collection loader path with Python 3.
Skip the existing test on Python 3.12, since find_module is removed.
Suppress the pre-existing deprecation warnings using the Python 2
codepath with Python 3.
Add a test for Python >= 3.12, which doesn't call find_module.
* Ignore sanity test errors on systems without libselinux present.
* Only install collections which can't be satisfied by a collection in any of the configured paths.
* Improve warning for unexpected collection install path
Fix warning when path is configured, but is a pip-managed path
Normalize the path before validating to fix warning consistency
* Add test for 256-color configuration values
See #78607.
* color is not restricted to 16 choices
currently supports up to 256, not listing them all
TOOD: create examples and point to/list the basic 16
---------
Co-authored-by: Brian Coca <brian.coca+git@gmail.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* ansible-test - Limit scope of replace-urlopen test
Only ansible-core code and plugins in collections can be expected to rely on module_utils.
* ansible-test - Limit scope of use-compat-six test
Only ansible-core code and plugins in collections can be expected to rely on module_utils.
* ansible-test - Limit scope of no-get-exception test
Only ansible-core code and plugins in collections should be checked for usage of outdated module_utils functions.
* Add integration tests
* replace: handle exception while parsing escape char
* Fail early when bad escape character is provided in replace module
Fixes: #79364
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Run tests in Python 3.6 or greater env
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
---------
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Remove docs dir
* Updates to reflect docs removal
* Fix integration test
* Remove examples dir
* Updates to reflect examples removal
* Remove build_library and build-ansible.py
* Remove refs to build_library and build-ansible.py
* Remove obsolete template
* Remove obsolete template reference
* Remove the now obsolete rstcheck sanity test
* Rename README.rst to README.md
* Change README format from reStructuredText to Markdown
* Fix whitespace in README.md
* Update setup.cfg to use README.md
* Replace changelog placeholder with README.md
* Update package-data sanity test
* first_found lookup, let lookup handle templating errors
Avoids case in which TE was not sending valid and templatable entries to the lookup
The lookup already handles the case TE was attempting to itself, so no need for this code anymore.
Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
It adds exception treatment when execute a inventory based on script with the --host argument
---------
Co-authored-by: Everson Leal <everson.leal@sonda.com>
* Improve readability of unit test output
This drops the trailing `-expectedXXX` suffixes from test names generated by parametrize.
* Add more splitter unit tests
This fills in code coverage gaps in the exising unit tests.
* Bug fixes and code cleanup
- Fix IndexError exceptions caused by parsing a leading newline, space or escaped space.
- Fix an AttributeError exception in `parse_args` when parsing `None`.
- Fix incorrect parsing of multi-line Jinja2 blocks, which resulted in doubling newlines.
- Remove unreachable exception handlers in the `parse_kv` function.
The unreachable code was verified through analysis of the code as well as use of the `atheris` fuzzer.
- Remove unnecessary code in the `split_args` function.
- Add an optimization to `split_args` for the empty args case.
* Add unit tests for bug fixes
The splitter code is now fully covered by unit tests.
* Add another issue ref in changelog
This fixes the issue where handlers notifying other handlers are
not properly run because the notification is not registered unless
another flush_handlers occurs. Instead, if the current host state
is iterating handlers we immediately register the handler to be
run so the notification is not lost.
Fixes#80880
* Check if skeleton is /dev/null while creating home folder
* Add test for linux
Co-authored-by: Holger Dörner <h.doerner@bitexpert.de>
Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
* Remove unnecessary MANIFEST.in commands
This eliminates more setuptools warnings during build.
* Sort MANIFEST.in
Now that only include commands are used, the order of the commands no longer effects the build output.
* Fix remote_tmp tests so that they actually use the remote_tmp
* Isolate remote_tmp dir during remote_tmp tests. Fixes#80965
* Do everything without become
* replace deprecated ast.value.s with ast.value.value
the s attribute is deprecated since Python 3.8 and emits a warning in
3.12 causing some test failures
* Improve coverage of validate-modules unit tests
* Remove unused galaxy unit test code
* Fix galaxy unit test teardown logic
* Improve coverage of galaxy unit test code
* Improve coverage of galaxy unit tests
* Remove unused code in galaxy API tests
* Remove unused galaxy collection unit test code
* Improve coverage of galaxy collection unit tests
* Remove unused galaxy unit test code
* Bump antsibull-docs-parser version to 1.0.0.
There are no functional changes that affect the functionality used in validate-modules.
* Bump antsibull-docs to 2.0.0.
* Add test for symbolic to octal when others is omitted
Add case when there should be no permissions for other.
And specific permissions for owner and group.
* Fix permissions test by explicitly setting no permissions for others
* Add additional cases where multiple permissions are specified
* Remove test_real_path_symlink test case
* Check if we do not get `-` return when some value other `-` is passed
Fixes: #80444
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Instead of using Templar.environment in Templar.do_template for
accessing/mutating the environment, myenv local variable should be used
because it is the environment used for actual templating. It can either
point to Templar.environment or newly created environment overlay.
Fixes#80605
Atthe moment if a users wants to protect virtualenv_command by using
quotes around 'venv', module will fail out as literal parsing is used
at the moment. In order to imrpove things, argparse is leveraged to
parse out passed value to the virtualenv_command
Closes-Bug: #76372
Signed-off-by: Dmitriy Rabotyagov <noonedeadpunk@gmail.com>
* Add TTY check and argument to disable it (#50603)
* Fix formatting
* add changelog
* rename flag and updated help description
* add tests for tty check
* replace deprecated uses of assertRaisesRegexp to assertRaisesRegex
* fix yaml syntax
* shorten line 79
* Revert "replace deprecated uses of assertRaisesRegexp to assertRaisesRegex"
This reverts commit cea5fe1655.
* change back to assertRaisesRegexp
* fix meaning of parallel in gather_facts
* Update docs with note about parallel not always being faster
* add 'smarter' usage of gahter_timeout for parallel tasks
* restore async when needed, not always
* added typing
* parallelism tests
* Only bypass type validation for null parameters if the default is None. A default is mutually exclusive with required.
* Prevent coercing None to str type. Fail the type check instead.
- Unit tests now report warnings generated during test runs.
- Python 3.12 warnings about `os.fork` usage with threads (due to `pytest-xdist`) are suppressed.
- Added integration tests to verify forked test behavior.
Fixes#73643
* clear_notification method and simplify ifs
* Deduplicate code
* Limit number of Templar creations
* Fix sanity
* Preserve handler callbacks order as they were notified
* Symbolic modes with X or =[ugo] always use original mode (Fixes#80128)
Here's what's happening, by way of this mode example: u=,u=rX
At the first step in the loop, the "u" bits of are set to 0. On the next
step in the loop, the current stat of the filesystem object is used to
determine X, not the "new_mode" in the previous iteration of the loop. So
while most operations kind of operate left to right, "X" is always going
back to the original file to determine whether to set x bit.
The Linux "chmod" (the only one I've tested) doesn't operate this way. In
it, "X" operates on the current state the loop understands it is in,
based on previous operations (and starting with the file permissions).
This is an issue with "X" and any of the "=[ugo]" settings, because
they are lookups. For example, if a file is 755 and you do "ug=rx,o=u",
file module produces 0557 and chmod produces 0555.
This really becomes a problem when you want to recursively change a
directory of files, and the files are currently 755, but you want to
change the directory to 750 and the files to 640. In chmod you can do
"a=,ug=rX,u+w" (or "a=,u=rwX,g=rX"), and have it apply equally to the
directory and the files. I can't come up with a single way in the ansible
file module to deterministically, recursively, set a directory to 750
and the contents to 640 no matter what the current permissions are,
as the code currently is.
The fix is to pass in "new_mode" to _get_octal_mode_from_symbolic_perms
in lib/ansible/module_utils/basic.py inside _symbolic_mode_to_octal. And
then take "new_mode" as an argument and use it instead of the filesystem
object stat.st_mode value.
* Fixing my new unit test, fixing bug in test comments
Most of the bad_parsing tests were no longer running, with several of them no longer being valid.
The invalid tests have been removed and the valid ones rewritten.
Reduce the number of Galaxy API calls made during dependency resolution by fetching remote signatures afterwards, since these are not used in backtracking.
Reduce the verbosity to `-vvvv` (to match other Galaxy API calls) to see this activity.
Co-authored-by: Sviatoslav Sydorenko <webknjaz@redhat.com>
* fix using templated values for include/import role options 'public', 'allow_duplicates', and 'rolespec_validate'
* pass templated values without changing the instance
* Fix templating by setting always_post_validate to True and calling IncludeRole.post_validate() instead
ci_complete
* add changelog
* password lookup, handle ident properly when saved
Currently we format and save ident when present but we didn't account for this when reading the saved file
Also added some more robust error handling.
* ansible-galaxy - support `resolvelib >= 0.5.3, < 1.1.0`
<https://pypi.org/project/resolvelib/1.0.1> released on 2023-03-09:
- <https://github.com/sarugaku/resolvelib/blob/main/CHANGELOG.rst#101-2023-03-09>
- <https://github.com/sarugaku/resolvelib/releases/tag/1.0.1>
Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>
* Trigger CI by pinning resolvelib with latest version
Since resolvelib is pre-installed in our test containers, we should temporarily pin the latest version allowed to force the tests to run with that version. Once the tests have passed that commit can be reverted.
Please make those changes without force pushing, so that we keep the reference to the passing CI run. We can squash the commits when merging the PR so the temporary commits won't be in the final commit merged to the devel branch.
Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>
* https://github.com/ansible/ansible/pull/80196#discussion_r1136003637
Also test resolvelib with multiple supported versions.
Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>
* Revert "Trigger CI by pinning resolvelib with latest version"
This reverts commit 5518e5dbca.
---------
Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>
* Pin setuptools to lowest supported @ PEP 517 test
This allows catching the behavior of builds under old setuptools.
* Stop invoking `setup.py install` in tests
This is not the part we care about since it involves dealing with the
external runtime dependencies rather than building our source
distribution.
Matt Davis