allow extra vars when templating j2 files in the skeleton, for example:
ansible-galaxy init --role-skeleton /path/to/skeleton --extra-vars @/path/to/vars_file.yml newrole
ansible-galaxy init --extra-vars "min_ansible_version=2.17.0" newrole
Extend the wordings in warning if the reserved keyword _ansible_
is used as a module parameter.
Fixes: #82514
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Co-authored-by: Jordan Borean <jborean93@gmail.com>
The ``validate-modules`` sanity test no longer attempts to process files with unrecognized extensions as Python.
Integration tests have been added to verify Python-specific checks do not apply to these files.
The `invalid-extension` and `missing-gplv3-license` checks still apply to these files. This may change in the future.
* Allow subdirectories with import_role _from options
Add tests that tasks_from is restricted to the role
Note that a task like:
- import_role:
name: role
tasks_from: tasks/entrypoint.yml
will now load tasks from "{{ role_path }}/tasks/tasks/entrypoint.yml"
instead of "{{ role_path }}/tasks/entrypoint.yml". This change in
behavior matches include_role.
* better test case (filename doesn't match one in tasks/)
Fixes#82584
best/nobest options are one of the options whose default values are set
by an OS distribution. For example in our CI, both Fedora and RHEL set
the best option to different default values. As such we should defer to
the distributions for the default value and not change it by default but
if users wish to change it they can do so explicitly.
Currently the dnf module sets the nobest option inconsistenly and not for
all cases. This patch fixes that to reflect the behavior described
above. In addition adding the best option for both dnf and dnf5 modules
since the best option is prefer to nobest in dnf while in dnf5 nobest is
completely removed in favor of best.
Fixes#82616
* prettify ansibile-doc output
delimiters when no color
avoid triggering color mode for existing tests
all use _format to observe nocolor
more v more info
imporoved conditional display
updated version on -v
normalize role errors
expand role data, dedupe code, fix formatting on warning
fix bug with galaxy info
role list improvements:
lists all roles (even w/o meta/argspec)
specs now indented under role
role only listed once
Updated tests to reflect format changes
Co-authored-by: Felix Fontein <felix@fontein.de>
Jinja may generate an invalid Python source code from a template. Trying
to compile such source code into a Python code object results in
SyntaxError being thrown. An example of such a template is providing the
same keyword argument into a lookup twice, resulting in:
`SyntaxError: keyword argument repeated`.
Since `jinja2.exceptions.TemplateSyntaxError` does not cover such a
case, as it is not a Jinja parsing error, we need to catch SyntaxError
explicitly ourselves.
Fixes#82606
* Fix loading legacy vars plugins when the plugin loader cache is reset
* Remove extra cache layer by ensuring vars plugin names are cached (stateless or not) so that the plugin loader cache can double as the load order
Prior to this commit, it was impossible to use a module like dnf with a
URL that contains a username with an @ such as an email address
username, because:
dnf:
name: https://foo@example.com:bar@example.com/some.rpm
Would cause netloc parsing to fail. However, the following:
dnf:
name: https://foo%40example.com:bar@example.com/some.rpm
Would also fail because ansible would *not* URL-decode the credentials,
causing the following to be base64 encoded in the Authorization header:
Zm9vJTQwZXhhbXBsZS5jb206YmFyCg==
Which decodes to:
foo%40example.com:foo
Which is *not* the authorized username, and as such, *won't* pass basic
auth.
With this commit, Ansible's url lib behaves like curl, chromium, wget,
etc, and encodes the above to:
Zm9vQGV4YW1wbGUuY29tOmJhcgo=
Which decodes to:
foo@example.com:bar
Which will actually pass the HTTP Basic Auth, and is the same behaviour
that you will find ie. with:
curl -vvI https://foo%40bar:test@example.com 2>&1 |grep Auth | awk '{ print $4 }'
This moves handling of callbacks and play recap stats from
_load_included_file to individual strategies so include_role tasks are
accounted for, not just include_tasks.
Fixes#77336
* uri: Two tests that demonstrate missing handling of the "force" parameter
Add unit and integration tests that demonstrate that the uri module is not
handling the "force" parameter.
The unit test demonstrates that when "force" is present in the module parameters,
it is not being passed through to fetch_url().
The integration test demonstrates that "force" does not disable caching as
documented, and calls with a "dest" parameter that points to an existing file
can result in a "304 Not Modified" response.
* uri: Handle the "force" parameter properly
The uri module documents a "force" parameter that can be used to disable caching.
The module accepted the parameter but didn't pass it through to the fetch_url() method
which implements the logic to handle setting the appropriate headers for disabling
caching. This change passes the "force" parameter through as expected, allowing caching
to be disabled when requested by the module caller.
* when doing a 'contains' search, determine the encoding of the files to be searched
* set default encoding to None for backwards compatibility
* changelog, error handling, tests added
* add sanity ignore.txt for non-utf-8 test
This patch removes an import fallback that was only executed under
Python 2. Now that we don't run tests against that runtime, it
generates an uncovered line. Dropping it will slightly increase the
coverage metric as a side effect.
* Changes as suggested by sivel
* Add changelog fragment and tests
Co-authored-by: Matt Martz <matt@sivel.net>
Co-authored-by: s-hertel <19572925+s-hertel@users.noreply.github.com>
* default svn URL to 127.0.0.1 for subversion integration tests
* svn client gives up before trying IPv4 addresses when localhost->(::1,127.0.0.1) in dual-stack envs (eg podman, most real hosts)
* svn client also requires legacy CN match on cert
* IPv6 works, but setup playbook would need a bunch more templating exceptions to conditionally manage `[::1]`
* explain IPv4 defaults
The timing of the async tasks was a little unpredictable, meaning that
sometimes we would get an unexpected number of v2_runner_on_async_poll
callbacks, and fail the test. This change fixes the issue by increasing
the poll interval to 2 seconds and the sleep duration to 3 seconds, such
that on a reasonably responsive system we will poll twice per task, with
the sleep ending in the middle of the two polls.
The include_me.yml file does not exist in this integration test. It has
been added.
The remote_tmp_dir.path expression is invalid - the setup_remote_tmp_dir
role uses set_fact to set remote_tmp_dir to remote_tmp_dir.path.
The integration tests run with ANSIBLE_HOST_PATTERN_MISMATCH=error,
meaning that the final play was never reached. Set
ANSIBLE_HOST_PATTERN_MISMATCH=warning to continue past the play and
trigger the v2_playbook_on_no_hosts_matched callback.
When ansible-test installs requirements, it now instructs pip to allow installs on externally managed environments as defined by PEP 668.
This only occurs in ephemeral environments managed by ansible-test, such as containers,
or when the `--requirements` option is used.
* deb822_repository: handle idempotency
Sort the parameters in order to handle idempotency
Fixes: #82454
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Add the task info for tombstoned plugins
* Fix deprecation for 'include' by removing it from BUILTIN_TASKS which skip the plugin loader lookup
* changelog
remove obsolete unit test using 'include'
* Update changelogs/fragments/improve-tombstone-error.yml
* move incorrect, unused 'others' option
this was to document 'file' options are usable .. but we already
import the file fragment to document those options.
* removed now bad ignore
* Various fixes to roles
- static property is now properly set
- role_names and other magic vars now have full list
- role public/private var loading is now done when adding to play.roles instead of on each var query
- added tests
Co-authored-by: Felix Fontein <felix@fontein.de>
* plugin config, ensure set_option correctly casts
until now we relied on plugin author getting the right type, now
the config system itself will process as it would for set_options
* lookups, make file searching use better is_role
The dwim function will internally try by detecting tasks/main['','.yml','.yaml]
but this is far from optimial, the existince of role path in vars is much better
indicator that we can use to pass a hint
* updated test to avoid main.yml
* Run svn integration test locally with TLS
This patch uses a `trustme` to make an ephemeral CA, and server, and
client TLS artifacts for testing. These are integrated into the Apache
web server via it's `mod_ssl`.
Resolves#82207
* Replace GitHub SVN HTTPS URL w/ localhost over TLS
This change gets rid of the need to use GitHub, which is just about to
drop support for SVN [[1]]. Moreover, it eliminates the need to use
external network for any SVN commands in the test.
[1]: https://github.blog/2023-01-20-sunsetting-subversion-support/
Set the tarfile attribute to a normalized value from unfrackpath instead
of validating path parts and omiting potentially invald parts
Allow tarfile paths/links containing '..', '$', '~' as long as the
normalized realpath is in the tarfile's role directory
* apt_repository: Do not convert symlink repo to a normal file
* module manually writes to repo files. If the file is a symlink
module used to re-write and convert a symlink to a normal file.
This fix tracks the symlink and keeps the changes intact.
Fixes: #49809
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Enable use of vendored copy of distutils by removing
the environment variable SETUPTOOLS_USE_DISTUTILS
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
This replaces the regex based check previously used within the validate-modules sanity test.
NOTE: The pylint check overlaps with a similar check for pep8.
Both are used, since the pep8 based check is more nuanced than the one in pylint.
For example, allowing `is` checks, but not `==` comparisons.
If the pylint check is ignored, the pep8 check would still be relevant.
* Remove unused argument in call to build_vault_ids
* Remove obsolete build_vault_ids tests
* Change tests to account for argument removal
* Remove redundant test
* dont warn about using a yescrypt hash as password
Signed-off-by: Thomas Sjögren <konstruktoid@users.noreply.github.com>
* add changelog
Signed-off-by: Thomas Sjögren <konstruktoid@users.noreply.github.com>
* add yescrypt test
Signed-off-by: Thomas Sjögren <konstruktoid@users.noreply.github.com>
---------
Signed-off-by: Thomas Sjögren <konstruktoid@users.noreply.github.com>
Also update tests to support the format on modules/plugins
Co-authored-by: Brian Scholer <1260690+briantist@users.noreply.github.com>
Co-authored-by: Felix Fontein <felix@fontein.de>
- Update `data2_vaulted_string_with_id` to match the documented plaintext.
- Add a comment explaining how `data2_vaulted_string_with_id` was derived.
- Add assertions for unvaulted values to ensure they match their plaintext.
- Add round-trip tests for vault+unvault when no salt is used.
Python 3.10 is the minimum version on the controller, and it requires openssl 1.1.1 or later.
As a result, there's no need to check the openssl version any longer.
Adds an option that can have an action plugin tell the module to ignore
options that do not fit its arg spec. This is to enable support for core
running modules that exist outside of the collection that may not be new
enough to support some of the options supplied to it.
* Improve host_group_vars efficiency:
* normalize the basedir with `os.path.realpath()` once and cache it
* cache missing paths/files
* reduce the calls to `isinstance`
Add a couple more general improvements in vars/plugins.py get_vars_from_path():
* call `PluginLoader.all()` once for vars plugins and reload specific
plugins subsequently
* don't reload legacy/builtin vars plugins that are not enabled
Add a test for host_group_vars and legacy plugin loading
Co-authored-by: Matt Davis <mrd@redhat.com>
* changelog
* Add a new is_stateless attribute to the vars plugin baseclass
update integration tests to be quieter and use the same test pattern
Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False)
move realpath cache to host_group_vars (do not smuggle call state as instance data)
refactor under a single 'if cache:' statement
Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory.
remove call to super(VarsModule, self).get_vars()
use the entity name as the cache key instead of variable location
Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group
Replace startswith by checking index 0 of the name instead, since host/group names are required
* rename is_stateless to cache_instance to make it more clear what it does
* add plugin instance cache using the path to plugin loader
reduce loading stage option if a new instance isn't created
don't require a known subdir on PluginLoader instantiation for backwards
compatibility
rename attribute again
contain reading from/initializing cached instances to a plugin loader method
* Deprecate v2 vars plugins
* Refactor to use the cache in existing plugin loader methods
Rename the attribute again
Refactor host_group_vars with requested changes
Make changelog a bugfixes fragment
Add a deprecation fragment for v2 vars plugins.
Add type hints
* unbreak group_vars
* Apply suggestions from code review
* misc tweaks
* always cache instance by both requested and resolved FQ name
* add lru_cache to stage calculation to avoid repeated config consultation
* handle KeyError from missing stage option
---------
Co-authored-by: Matt Davis <mrd@redhat.com>
When templating tags (which happens outside of standard `post_validate`) we
need to template each object in the inheritance chain and set the templated
values on those objects individually. That way when `task.tags` is called the
`extend` functionality properly picks up the templated values of all
parents into one flatten list.
Fixes#81053
revert to previous behavior to push vars to play at compile time
add `public` parameter to allow per import control of exporting (vs just the global config)
Co-authored-by: tchernomax <maxime.deroucy@gmail.com>
Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
Provide more information about plugin usage which needs to be
used instead of the deprecated plugin in the deprecation message.
Fixes: #80561
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Test heuristic_log_sanitize. See #81689
* Add note about what this test is doing
* grammar
Co-authored-by: Matt Clay <matt@mystile.com>
---------
Co-authored-by: Matt Clay <matt@mystile.com>
* Make sure paths are correct when building collection files manifest
This commit makes sure the path of the files part of
the collection build manifest are correct.
This commit uses os.path.commonprefix instead of
dealing with strings.
Signed-off-by: Carlos Camacho <ccamacho@redhat.com>
Bugfix Pull Request
Fixes: #81618
* Revert the change note type to `minor_changes`
* Clarify the change note with user-oriented details
---------
Signed-off-by: Carlos Camacho <ccamacho@redhat.com>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
PR #81606.
Prior to this patch, when `--pre` CLI flag was not passed, the
dependency resolver would treat concrete collection dependency
candidates (Git repositories, subdirs, tarball URLs, or local dirs or
files etc) as not meeting the requirements.
This patch makes it so pre-releases in any concrete artifact
references, and the ones being specifically pinned dependencies or
user requests, met anywhere in the dependency tree, are allowed
unconditionally.
This is achieved by moving the pre-release check from
`is_satisfied_by()` to the `find_matches()` hook, following the
Pip's example.
As a bonus, this change also fixes the situation when a collection
pre-releases weren't considered if it didn't have any stable releases.
This now works even if `--pre` wasn't requested explicitly.
Finally, this patch partially reverts commit
6f4b4c345b, except for the tests. And it
also improves the `--pre` hint warning to explain that it mostly
affects Galaxy/Automation Hub-hosted collection releases.
Ref #73416
Ref #79112Fixes#79168Fixes#80048Resolves#81605
Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
* Symlinks in the collection might be pointing to non-existent
targets. Check and report the failure to the user.
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* when --diff is used with content parameter, print destination
path instead of temporary file path.
Fixes: #79749
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* If all tasks in the role are skipped or unreachable, the role is not marked as complete for the host.
* Only mark the role as complete if a task in the role succeeds or fails for the host.
* Before performing shredding the original file, check if the
destination file location is writable or not. This will
prevent corruption of original file.
Fixes: #81455
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Python 3.11.4 introduces a new parameter 'filter' in extract and
extractall in tarfile. Handle deprecation warning message emitted
in Python 3.12.
* added probing mechanism in ansible-galaxy code to detect broken
data filter implementation in tarfile.
Fixes: #80832
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Co-authored-by: Matt Clay <matt@mystile.com>
As of coverage 7.1.0, files without arcs are not generated.
To work around this, an empty (0, 0) arc is used for each file instead.
This is a follow-up to https://github.com/ansible/ansible/pull/81077
ci_coverage
ci_complete
Improves the error checking when running PowerShell modules using
Ansible.ModuleUtils.Legacy. It will only return an rc of 1 if both the
PowerShell module runner signalled an error occurred and those error
records were present in the output. This should reduce some false
positive errors when using the older module style.
* apt: ignore fail_on_autoremove and allow_downgrade when using aptitude
* fail_on_autoremove (--no-remove) and allow_downgrade (--allow-downgrades)
parameters are only valid for apt-get and not for aptitude. Ignore them when
aptitude is detected and used.
Fixes: #77868
* Gather value of password using debconf-get-selections command,
use this information for idempotency.
Fixes: #47676
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Enable mypy for the entire packaging directory
* Return CompletedProcess only when capturing output
This allows stdout/stderr on CompletedProcess to be `str` instead of `str | None`.
The unused args on CompletedProcess have been removed.
Overload type hints have been added to reflect these changes.
* Relax return type on ensure_venv
This improves consistency with its usage, since `run` accepts `env` of `dict[str, t.Any]`.
Also removed unnecssary `str()` usage when updating `env`.
* Fix type hint on suppress_when
* Fix callable annotation
* Add type hint for command_parser
PyCharm complains about using a protected member, and also that it can't find the type in the type stubs.
However, mypy properly recognizes the type.
* Avoid unnecessary TypeVar usage
* Back out use of communicate, add better comments, add bufsize, and align with subprocess._communicate
* tests
* re-order logic slightly
* more comments
* loopty loop
* yet another comment
* Revert "yet another comment"
This reverts commit 96cd8ada5fa0441b92f2298bdaa6cb40594847d2.
* Revert "loopty loop"
This reverts commit 96ea066f6a7d18902c04a14f18dd79b38e56f5e7.
* ci_complete
* Copy in comment too
* Wording updates
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>
* Back out bufsize
---------
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>
* a recent optimization lost the unsafe lookup disable behavior when templating conditionals with inline templates that referred to untrusted values
* added regression test to catch this case
* iptables chain creation does not populate with a rule
fixes#80256
* Add changelog fragment
* Add rules and flush chain during integration tests
* Check chain rule on comment
* Update test/integration/targets/iptables/tasks/chain_management.yml
* User-provided collection type might differ from collection
source. Cross-check the type before proceeding
Fixes: #79463
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* ansible-galaxy - improve ignoring multiple signature status codes when using --ignore-signature-status-code
* fix backwards compatibility by adding a new plural option instead, and hide the singular from --help
Since man pages aren't accessible to users after a `pip install`, there's no need to include them in the sdist.
This change makes it trivial to build man pages from source, which makes them much easier to iterate on.
It also simplifies creation and testing of the sdist, since it no longer requires building man pages.
The new `packaging/cli-doc/build.py` script can generate both man pages and RST documentation.
This supports inclusion on the docs site without a dependency on `ansible-core` internals.
Having a single implementation for both simplifies keeping the two formats in sync.
* Remove datetime.datetime.utcfromtimestamp and datetime.datetime.uctnow
from controller code since they are deprecated in Python 3.12.
* Update target side code to use new utcfromtimestamp and utcnow utils in ansible.module_utils.compat.datetime that return aware datetime objects on Python 2.7 and 3.
Co-authored-by: Matt Clay <matt@mystile.com>
* Raise the minimum setuptools version to 66.1.0
This is the first version to support Python 3.12.
While Python 3.10 and 3.11 could use an older version, a consistent minimum is easier to work with and test against.
* Fix PEP 517 integration test
* update docker containers versions to use newer ansible-test ref in the pre-built venvs
* Allow invoking ansible-test with Python 3.12
* Add python3.12 to the INTERPRETER_PYTHON_FALLBACK
* changelog
* add Python 3.12 as a non-default Python version for the test containers
* Update mypy ignores for Python 3.12
* Add Python 3.12 to CI matrix for unit tests, generic tests, and galaxy
* Update unit test for using the Python 2 collection loader path with Python 3.
Skip the existing test on Python 3.12, since find_module is removed.
Suppress the pre-existing deprecation warnings using the Python 2
codepath with Python 3.
Add a test for Python >= 3.12, which doesn't call find_module.
* Ignore sanity test errors on systems without libselinux present.
* Only install collections which can't be satisfied by a collection in any of the configured paths.
* Improve warning for unexpected collection install path
Fix warning when path is configured, but is a pip-managed path
Normalize the path before validating to fix warning consistency
Sloane Hertel