* password lookup, handle ident properly when saved
Currently we format and save ident when present but we didn't account for this when reading the saved file
Also added some more robust error handling.
* Create a queue per WorkerProcess to receive intra-task updates
* Update `pause` action to use the worker queue
* Deprecate ConnectionBase()._new_stdin
* Add new `Display` convenience method `prompt_until` to manage both controller- and worker-sourced prompting without cross-fork stdin sharing, in-worker mechanism to handle request-response over new worker queue.
This provides the same test coverage as the previous tests, without the dependency on git.
It also includes many more specific test cases with assertions, instead of simply relying on the code to not raise an exception.
* Add condition that causes a when to skip a task
* Fix up tests
* Use false_condition instead of failed_condition
* Remove formatting accidentially added
* Fix sanity
* Fix detection of available hashlib algorithms
Detection of hashlib algorithms now works on Python 3.x.
The new implementation works on Python 2.7 and later.
Test coverage is provided by both integration and unit tests.
* Add additional details about hashlib in docs
* Update `collections.abc` imports
- Use `six.moves` for modules and module_utils
- Use `collections.abc` for controller code
This avoids using `ansible.module_utils.common._collections_compat`,
which was added before the vendored `six` was updated to provide these
imports.
* Update _collections_compat to use six.moves
Also update the custom pylint rule to reflect this change.
* Remove unused test fixtures
* Removed unused _old_dump_load_cycle method
* Remove Python 2.x compat
* Remove unused code
* Remove unused context manager
* Fix cowsay test
- The test no longer depends on another test to initialize config.
- Also remove unreachable code.
* Remove Python 2.x compat
* Normalize deprecation records.
* Fix alias deprecations in suboptions.
* Report in which option an alias warning happened for suboptions.
* Add deprecation tests for suboptions.
* Also test deprecation in list of dicts.
* Adjust unit tests for toplevel alias deprecation field name change.
* Add support for importlib.resources
* Remove the importlib.resources imports
* return the correct data
* Some code comments, and re-order for consistency
* Disallow traversing packages below an individual collection
* Add a traversable class for namespaces
* Re-use variable
* Utilize itertools.chain.from_iterable
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Simplify logic to check for packages from ansible loaders
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Just a generator expression, instead of a generator
* docstrings
* Add comment about find_spec for our namespaces
* Add some initial unit tests for importlib.resources
* normalize
* Utilize importlib.resources for listing collections
* collections_path is already in config, just use config
* install uses a different default for collections_path
* Remove unused import
* Remove duplicate __truediv__
* Bring back TraversableResources
* Apply some small suggestions from code review
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>
* Remove cross contamination between plugin loader code and CLI code
* Remove unused import
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>
Fix ignoring certs when downloading tarballs
Fix ignoring certs when downloading a collection from a specific source that isn't in the configured servers list
* Remove unused mock from test_password_already_created_encrypt
The _get_paths mock is never used in the
test_password_already_created_encrypt test case.
* Add test to assert the password file is not rewritten
If the password file already contains the salt and the hasing algorithm
does not use the ident parameter, the password lookup should not write
to the password file.
* Fix "changed" if using "encrypt" in password lookup
When using the "encrypt" parameter to the password lookup without the
ident parameter, the password file was always marked as "changed". This
caused the file to be rewritten with the same content. This is fixed by
only marking the file as changed, if an "ident" value needs to be added
to the file.
Fixes#79430.
Add changelog entry
* galaxy: Add license_file to manifest directives
* ag collection build: Test license handling
This adds tests to ensure that
- REUSE licensing files: .reuse/dep5, LICENSES/*, anyfile.license
- galaxy.yml license_file
are always included in the manifest.
After changes:
```
"ansible_locally_reachable_ips": {
"ipv4": [
"127.0.0.0/8",
"127.0.0.1",
"192.168.0.1",
"192.168.1.0/24"
],
"ipv6": [
"::1",
"fe80::2eea:7fff:feca:fe68",
...
]
},
```
192.168.1.0/24 is a local prefix, where any IP address inside this range
is reachable locally (or outside this host if this prefix is announced via
EGP/IGP).
Signed-off-by: Donatas Abraitis <donatas.abraitis@hostinger.com>
When running the unit tests locally, in a git clone, `ansible --test`
prints additional information about the git revision, e.g.
```
~/src/ansible$ ansible --version
ansible 2.10.8
...
~/src/ansible$ source hacking/env-setup
...
Setting up Ansible to run out of checkout...
...
~/src/ansible$ ansible --version
[WARNING]: You are running the development version of Ansible. You
should only run Ansible from "devel" if you are modifying the Ansible
engine, or trying out features under development. This is a rapidly
changing source of code and can become unstable at any point.
ansible [core 2.14.0.dev0] (test_ansible_version-devel df497ea13b) last
updated 2022/08/25 10:37:07 (GMT +100)
...
```
which causes `test_ansible_version()` to fail.
Also removed an unused argument from previous parameterisation.
Co-authored-by: Alex Willmer <alex.willmer@cgi.com>
* Add --offline option to 'ansible-galaxy collection install' to prevent querying distribution servers
This allows installing/upgrading individual tarfiles to have dependency resolution.
Previously needed to be done manually with --no-deps or else all collections and dependencies needed to be included in the requirements.
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Add OSMC to Debian OS_FAMILY_MAP
- os_family fact of the Debian-based OSMC distribution was not detected correctly
* tweak changelog
Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com>
* new _fqcn attribute to plugin objects
* unbreak plugins in subdirs
* Fix inadventent changes to _load_name and use existing vars
* add plugin aliases and name property, and replace plugin._load_name where incorrect
* Fix listing plugin names
Fix listing legacy and builtin together
test deprecated plugin documentation
fix doc extensions
remove sometimes inaccurate _load_name handling from plugin.name
* Add tests for REJECT_EXTS and doc extensions
Fix unpredictable collection redirects so non-fqcns in the redirect list are guaranteed to be legacy (instead of determined by the collections keyword)
Move aliases and name properties to _update_object so all plugin types, including doc fragments, can use them
* make legacy plugin names internally consistent
* rename attributes to ansible_name and ansible_aliases
* Fix distro fact handling for Flatcar
The existence of the file /etc/flatcar/update.conf depends on
bootstrap configuration typically provided by the user. For that
reason this file is unsuitable for determining distro facts for
Flatcar Container Linux.
The distribution_release fact is meaningless in the case of Flatcar
since Flatcar doesn't have named releases. The distribution_version
fact, however, IS meaningful and should contain a number such as
"3139.2.0".
- Use /etc/os-release instead of /etc/flatcar/update.conf.
- Drop the distribution_release fact.
- Set the distribution_version fact.
- Update distro test fixture for Flatcar
- Generate the fixture using gen_distribution_version_testcase.py.
- Override result.distribution and result.os_family manually as the
generator script gives wrong values.
- Use a recent Flatcar version.
Signed-off-by: Johanan Liebermann <jliebermann@microsoft.com>
* refactor and remove redundant code in documentation
allow location and building api to be more accessible
fix issues with displaying ansible.legacy and ansible.builtin
ensure we don't x2 process tokens (some modules reference them also) fixes#77764
move to constants vs hardcoded
more informative errors and comments
now have actual filter/test plugins, which expose the filter/test functions
moved filter/test loading/finding logic into jinja2pluginloader, removed dupe implementations
added tests for case in which we unique by basename when listing
Update lib/ansible/utils/plugin_docs.py
Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
Sometimes pytest errors out with an `ImportError` during its tests
collection stage when a Python package/directory containing the test
module does not have an `__init__.py` in it. This is being observed
under Python 3.9 and higher.
The patch provides a workaround for this problem but does not address
the root cause which is currently unknown.
Ref:
https://github.com/ansible/ansible/pull/78585#issuecomment-1220885431
* add required file to collection skeleton
This file is required to be able to upload a collection.
It is present in
https://github.com/ansible-collections/collection_template/blob/main/meta/runtime.yml
but that does not get used by default.
Without this, if you use the "ansible-galaxy collection init" command
and you try and publish that collection without adding this file, you
get the error:
"ERROR! Galaxy import process failed: 'requires_ansible' in
meta/runtime.yml is mandatory, but no meta/runtime.yml found (Code:
UNKNOWN)"
Also updates relevant test and adds a changelog fragment
Replace get_persistent_connection_options with get_options
Remove special case for network sub_plugin in _set_plugin_options
Try to avoid mock connection pretending to be persistent
Rename variables->options to reflect what they actually are
Gather options for ssh_type_conn on network_cli
Drop reliance on sub_plugin["type"]
* Rethread pr/70185 through the dependency resolver
Hang optional metadata toggle on the ConcreteArtifactsManager instead of threading it through whole list codepath
Don't error while listing collections if a collection's metadata is missing keys required for building a collection.
Give an informative warning if metadata has been badly formatted.
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Report OpenSuSE >= 15 as opensuse
Make distro.id() report newer versions of OpenSuSE (at least >=15) also report
as opensuse. They report themselves as opensuse-leap.
* Add a test
* Fix KeyError for ansible-galaxy when caching paginated responses from v3
* changelog
* generate responses in loop for test
Co-authored-by: Matt Martz <matt@sivel.net>
- `processor_count` was erroneously set to the number of cores
- `processor_cores` was erroneously set to the number of threads per core
- `processor_vcpus` and `processor_threads_per_core` were not set
- `processor` was a string, while it's supposed to be a list
Before:
```
"ansible_processor": "PowerPC_POWER7",
"ansible_processor_cores": 4,
"ansible_processor_count": 12,
```
After:
```
"ansible_processor": [
"PowerPC_POWER7"
],
"ansible_processor_cores": 12,
"ansible_processor_count": 1,
"ansible_processor_threads_per_core": 4,
"ansible_processor_vcpus": 48,
```
Also add a unit test.
Co-authored-by: Baptiste Jonglez <git@bitsofnetworks.org>
* Fix workding for :ref|term: subsitution
now matches 'seealso' and makes more sense:
```
delay:
applies_to:
- Task
description: Number of seconds to delay between retries. This setting is only used
in combination with `until`.
```
vs
```
delay:
applies_to:
- Task
description: Number of seconds to delay between retries. This setting is only used
in combination with website for `until`.
```
* updated unit tests
* match see also
* more sanity
* ansible-galaxy configurable timeouts
- also fixed issues with precedence,
so --ignore-certs now overrides config
- made galaxy_timeout generic setting,
if set, it becomes default for server configs,
but now specific servers can override
- updated tests or added notes (some tests ignore/override precedence)
Co-authored-by: Sloane Hertel <19572925+s-hertel@users.noreply.github.com>
* Remove .travis.yml files from galaxy init skeletons
These files have existed for many years but in light of the recent
circumstances around Travis, we should not include them by default in
the galaxy init skeletons.
* Add breaking_changes changelog fragment with recommendation to use a custom skeleton if the .travis.yml file is still needed.
Co-authored-by: s-hertel <19572925+s-hertel@users.noreply.github.com>
* If there is a platform specific handler, prefer the resolved module over the resolved action when loading module_defaults
Add a toggle for action plugins to prefer the resolved module when loading module_defaults
Allow moving away from modules intercepted as actions pattern
Fixes#77059
* Run code-smell sanity tests in UTF-8 Mode.
* Update subprocess use in sanity test programs.
* Use raw_command instead of run_command with always=True set.
* Add more capture=True usage.
* Don't expose stdin to subprocesses.
* Capture more output. Warn on retry.
* Add more captures.
* Capture coverage cli output.
* Capture windows and network host checks.
* Be explicit about interactive usage.
* Use a shell for non-captured, non-interactive subprocesses.
* Add integration test to assert no TTY.
* Add unit test to assert no TTY.
* Require blocking stdin/stdout/stderr.
* Use subprocess.run in ansible-core sanity tests.
* Remove unused arg.
* Be explicit with subprocess.run check=False.
* Add changelog.
* Use a Python subprocess instead of a shell.
* Use InternalError instead of Exception.
* Require capture argument.
* Check for invalid raw_command arguments.
* Removed pointless communicate=True usage.
* Relocate stdout w/o capture check.
* Use threads instead of a subprocess for IO.
* Expand ansible-doc to tests/filters and fix existing issues
enable filter/test docs if in single file or companion yaml
add docs for several filters/tests plugins
allow .yml companion for docs for other plugins, must be colocated
verify plugins are valid (not modules, cannot)
fix 'per collection' filtering
limit old style deprecation (_ prefix) to builtin/legacy
start move to pathlib for saner path handling
moved some funcitons, kept backwards compat shims with deprecation notice
Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Sandra McCann <samccann@redhat.com>
* Prevent losing unsafe from lookups
This patch fixes a bug which under certain conditions results in data
returned from lookups not being marked as unsafe.
Each time Templar.do_template is invoked a new AnsibleContext is
created and stored effectively at two places:
1) as an instance variable in templar_obj.cur_context
2) as a local variable called new_context in do_template method of Templar
Due to custom functionality in Ansible's Context that allows for nested
templating it is possible that during resolving variable's value
template/do_template method is called recursively again, again creating
a new context. At that point the problem manifests itself because as
mentioned in 1) above the context is overwriten on the templar object
which means that any subsequent calls to _lookup will use the new
context to mark it as unsafe which is now different to the local
new_context which is used for testing for unsafe property.
The solution to the problem appears to be to restore the original
context inside do_template and also to eliminate the local variable
new_context to prevent problems in the future.
It appears that we don't have a better way of storing the context other
than as some form of global variable and so this appears to be the
"best" solution possible at this point. Hopefully data tagging will be
the solution here.
For more examples see unit and integration tests included in this patch.
Fixes#77535
* Run code-smell sanity tests in UTF-8 Mode.
* Update subprocess use in sanity test programs.
* Use raw_command instead of run_command with always=True set.
* Add more capture=True usage.
* Don't expose stdin to subprocesses.
* Capture more output. Warn on retry.
* Add more captures.
* Capture coverage cli output.
* Capture windows and network host checks.
* Be explicit about interactive usage.
* Use a shell for non-captured, non-interactive subprocesses.
* Add integration test to assert no TTY.
* Add unit test to assert no TTY.
* Require blocking stdin/stdout/stderr.
* Use subprocess.run in ansible-core sanity tests.
* Remove unused arg.
* Be explicit with subprocess.run check=False.
* Add changelog.
When looking up the `no_log` setting for a parameter that is an alias in
`AnsibleModule._log_invocation()`, the alias value will always be an
empty dictionary since `self.aliases` on the `AnsibleModule` instance is
never updated after initialization. Since the `no_log` setting is on the
canonical parameter not the alias, an incorrect warning is issued if the
parameter matches `PASSWORD_MATCH`.
This PR returns the aliases dictionary as an attribute of the
`ValidationResult` and updates the `aliases` attribute on the
`AnsibleModule` instance.
* Revert "Revert "Config, ensure templating happens at functions (#77483)""
This reverts commit 94c9106153.
* removed update configdata, which is unused
* removed test for action we don't perform anymore
* removed unused configdata
* replace hardcoded '~/.ansible' to C.ANSIBLE_HOME
* rename previously existing env ANSIBLE_HOME in env-setup script
* modify cache dir monkeypatching in galaxy api unit tests
* update "version_added" to 2.14 for ANSIBLE_HOME
* fix description of collections with proper use of ANSIBLE_HOME
Co-authored-by: htol <github@h-tol.net>
Co-authored-by: stefanwascoding <stefan@syntaxhelden.de>
* Add a toggle to control the number of signatures required to verify the authenticity of a collection
* Make the default number of required valid signatures 1
* Add option to make signature verification strict and fail if there are no valid signatures (e.g. "+1")
* Use a regex to validate --required-valid-signature-count
* Add a toggle to limit the gpg status codes that are considered a failure
* Update documentation and changelog
* Add unit and integration tests for the new options
* Fixes#77146
Fix using user-provided signatures when running 'ansible-galaxy collection verify ns.coll --offline'
Add a test for a user-provided signature when running ansible-galaxy collection verify with --offline
Fix displaying overall gpg failure without extra verbosity
Add a test for displaying gpg failure without verbosity
Improve documentation to be more clear that signature verification only currently applies to collections directly sourced from Galaxy servers
* ansible-galaxy collection install|verify:
- Support verifying the origin of the MANIFEST.json when the Galaxy server has provided signatures.
- Allow supplemental signatures to use during verification on the CLI/requirements file.
* ansible-galaxy collection install:
- Support disabling signature verification. This silences the warning provided by ansible-galaxy if the Galaxy server provided signatures it cannot use because no keyring is configured.
- Store Galaxy server metadata alongside installed collections for provenance. This is used by 'ansible-galaxy collection verify --offline'.
* Add unit tests for method that gets signatures from a Galaxy server
* Add integration tests for user-provided signature sources
- Test CLI option combinations
- Test installing collections with valid/invalid signature sources
- Test disabling GPG verification when installing collections
- Test verifying collections with valid/invalid signature sources
* Make signature verification advisory-by-default if signatures are provided by the Galaxy server
- Make the default keyring None
- Warn if the keyring is None but the Galaxy server provided signatures
- Error if the keyring is None but the user supplied signatures
- Error if the keyring is not None but is invalid
* changelog
* add ansible-galaxy user documentation for new options
Co-authored-by: Matt Martz <matt@sivel.net>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Co-authored-by: Martin Krizek <martin.krizek@gmail.com>
Co-authored-by: Sandra McCann <samccann@redhat.com>
Co-authored-by: Andy Mott <amott@redhat.com>
Co-authored-by: John R Barker <john@johnrbarker.com>
* add DebianStrategy tests
* ensure hostname can be changed by using become
* use Systemd strat for debian and Base for generic.
* add test to ensure all strategies are available
Signed-off-by: Thomas Sjögren <konstruktoid@users.noreply.github.com>
- Avoids false positives on become strings being echoed back
by ssh cli itself
- added test for debug lines
- also simplified some of existing test code
modules with python were always normalized to /usr/bin/python,
while other interpreters could have specific versions.
* now shebang is always constructed by get_shebang and args are preserved
* only update shebang if interpreter changed
* updated test expectation
* added python shebang test
* ansible-galaxy - fix the --ignore-certs flag for the implicit galaxy server
* changelog
* Add a test without the server config
* Fix respecting --ignore-certs for individual --server URLs also
* Update changelogs/fragments/76735-ansible-galaxy-fix-ignore-certs.yaml
* check finder type before passing path
ci_complete
* Reduce nesting
* Test find_module does not cause a traceback with Python 3 FileFinder
* Update lib/ansible/utils/collection_loader/_collection_finder.py
* catch the case that cowsay is broken
fixes https://github.com/ansible/ansible/issues/72582
add changelog
raise Exception for broken cowsay
add test for broken cowsay
Co-authored-by: Matthias Bernt <m.bernt@ufz.de>
- On Python 3, decode bytes to str
- Tolerate whitespace around the line.
- Report a missing HOSTNAME line as an error.
- Don't clobber line separators when setting a new hostname.
- Add some tests for this strategy to verify correct operation.
* Use NativeEnvironment for all templating
ci_complete
* Keep Templar.copy_with_new_env for backwards compat
* Mention that AnsibleUndefined.__repr__ changed in the porting guide
* Templar.copy_with_new_env backwards compat
* ci_complete
* download role from api response location
* include changelog fragment
* add unit test for role download url
Co-authored-by: Sam Doran <sdoran@redhat.com>
* ansible-galaxy: Clone git collections using shallow clones
This ensures the collection obtained via git url is a result of a
shallow git clone (git clone --depth=1). The git history of the
collection is not used by ansible, and as such, cloning the entire
history of the repo is unnecessary.
Signed-off-by: Tomas Babej <tomas@tbabej.com>
* ansible-galaxy: Only perform shallow clones on non-versioned git urls
In general, the version can be anything we can checkout (branch, tag,
but also a commit hash). In particular for commit hashes we cannot
perform a shallow clone.
Err on the safe side and only perform shallow clones for non-versioned
git urls (cloning HEAD).
* galaxy-install: Make shallow cloning compatible with older Python versions
Signed-off-by: Tomas Babej <tomas@tbabej.com>
* Pass args as a tuple
Add tests for the git command created from different repo and version combinations
* changelog
Co-authored-by: Tomas Babej <tomas@tbabej.com>
* add tests for fail filter
also tests that fail does not block inspectability
* add fail filter
fallback message is a bit clunky,
since you can't invoke a filter without specifying an input.
That is, "{{ fail }}" doesn't work,
so you have to do "{{ None | fail }}"
* document 'fail' filter
* add changelog fragment
* fail filter uses default message on Undefined or emptystring
makes it slightly easier to use the default message:
```diff
- "{{ None | fail }}"
+ "{{ '' | fail }}"
```
and the user sees a slightly more relevant message
if the message itself is undefined:
```diff
- The error was: {{ failmsg | fail }}: 'failmsg' is undefined
+ The error was: {{ failmsg | fail }}: Mandatory variable has not been overridden
```
* rebuild as the builtin `Undefined`
* harmonise `hint` parameter for make_undefined with jinja
* use code block for documentation item
[ref](https://github.com/ansible/ansible/pull/75435#discussion_r707661035)
* rename to `undef` to expose less Python into the Jinja
[ref](https://github.com/ansible/ansible/pull/75435#pullrequestreview-757799031)
* explicitly instantiate undefined value now that it's possible
see I knew we would break something with reflection
* preserve test coverage of undefined variable
Co-authored-by: Matt Davis <nitzmahone@users.noreply.github.com>
* Enable ansible-galaxy to specify client id override with Keycloak Token
* Specify ability to provide override of client_id
* Test client_id can be configured for individual servers
* Add issue link to changelog
* Document client_id as a config option and add an example
Co-authored-by: s-hertel <19572925+s-hertel@users.noreply.github.com>
* pip - Use pip from the current Python interpreter.
If `executable` and `virtualenv` were not specified, and
the `pip` Python module is available for the current interpreter,
use that `pip` module instead of searching for a `pip` command.
* Add comment about needing `__main__` to run `pip`.
* Fix unit test.
* Add porting guide entry.
* Update changelog to match porting guide description.
ci_complete
* Test a passlib wrapped algo with a password lookup
* Fix error when passlib is used with a wrapped algo
The exception was:
An unhandled exception occurred while running the lookup plugin 'password'.
Error was a <class 'TypeError'>, original message: issubclass() arg 1 must be a class
and can be reproduced using the following command:
ansible localhost -i localhost, -mdebug -amsg="{{ lookup('password', '/dev/null encrypt=ldap_sha512_crypt') }}"
The concerned algo are: bsd_nthash, django_argon2, django_bcrypt, ldap_bcrypt,
ldap_bsdi_crypt, ldap_des_crypt, ldap_hex_md5, ldap_hex_sha1, ldap_md5_crypt,
ldap_pbkdf2_sha1, ldap_pbkdf2_sha256, ldap_pbkdf2_sha512, ldap_sha1_crypt,
ldap_sha256_crypt, ldap_sha512_crypt, roundup_plaintext
Rather than adding further complexity to the regex, preprocess the output to
remove any empty lines. Now the only purpose of the regex is to fix
wrapped lines.
Fixes#70949
Matt Martz